diff --git a/.github/workflows/cpp-curl.yml b/.github/workflows/cpp-curl.yml new file mode 100644 index 0000000..feaa901 --- /dev/null +++ b/.github/workflows/cpp-curl.yml @@ -0,0 +1,51 @@ +name: C++ libcurl transport + +# Builds the optional libcurl transport for HttpEncoder against a real libcurl +# and runs its tests. ci.yml builds the C++ port without libcurl (the transport +# is behind -DISONGRAPH_VECTOR_CURL=ON and needs the library installed), so +# nothing there compiles or exercises include/isongraph_vector_http.hpp's +# CurlTransport. The tests talk to a small HTTP server started inside the test +# process on 127.0.0.1 - no internet is used. + +on: + push: + paths: + - 'isongraph-vector-cpp/**' + - '.github/workflows/cpp-curl.yml' + pull_request: + paths: + - 'isongraph-vector-cpp/**' + - '.github/workflows/cpp-curl.yml' + workflow_dispatch: + +permissions: + contents: read + +jobs: + curl: + name: libcurl transport on ${{ matrix.os }} + runs-on: ${{ matrix.os }} + strategy: + fail-fast: false + matrix: + os: [ubuntu-latest, macos-latest] + steps: + - uses: actions/checkout@v4 + + # Ubuntu's runner image ships the curl command but not the development + # package (headers and libcurl.so for find_package(CURL)). macOS has + # libcurl and its headers in the SDK, so nothing to install there. + - name: Install libcurl headers (Linux) + if: runner.os == 'Linux' + run: | + sudo apt-get update + sudo apt-get install -y libcurl4-openssl-dev + + - name: Configure + run: > + cmake -S isongraph-vector-cpp -B isongraph-vector-cpp/build-curl + -DISONGRAPH_VECTOR_CURL=ON -DCMAKE_BUILD_TYPE=Release + - name: Build + run: cmake --build isongraph-vector-cpp/build-curl --config Release + - name: Test + run: ctest --test-dir isongraph-vector-cpp/build-curl -C Release --output-on-failure diff --git a/CHANGELOG.md b/CHANGELOG.md index d557fd9..a56134a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -173,18 +173,84 @@ JavaScript, Rust and C++. - A cross-port CI check (`scripts/cross_port/graph_files.py`) of graph files, sidecars and the graph digest: every port writes, every port reads, and a tampered file must raise each reader's mismatch error (72 checks). +- **Approximate search in every port (opt-in)** - the same lazily built, + never-persisted IVF-flat index (seeded k-means, nlist about 4*sqrt(N), nprobe + about 5%, minimum size 50,000, exact override, `recall_check`) in Rust + (`set_ann`), C++ (`enableAnn`), C# (`AnnMode.Ivf`) and TypeScript / + JavaScript (`ann: 'ivf'`). Graph files, sidecars and the digest are + unchanged. Seeds do not reproduce across ports (each uses its own generator), + so indexes differ while recall is comparable. The standalone SQLite stores + have no index. See `benchmarks/competitive/` for measured speed and recall. +- **Rust: HNSW, int8 scan, binary snapshot** (all opt-in; design in + `isongraph-vector-rs/DESIGN.md` sections 10-12, measured in + `benchmarks/rust-stage-c/`). `AnnConfig::hnsw()` adds an HNSW index that + holds links only (about 1.2x raw memory at 1M x 384); at recall@10 0.95 its + single-thread speed is level with faiss HNSW and hnswlib and ahead of + usearch. It builds about 8-20x slower than IVF (about 150 s at 1M) and is + not persisted. `ScanPrecision::Int8` adds an int8 first pass with exact + rerank: 3-3.6x faster single-query scans at 1M, recall 1.000 against the + f32 scan from rerank factor 2 on real embeddings, no gain for batches, +0.25x + memory, and not used by HNSW graph search. `save_with_snapshot` writes a + binary `.snap` beside the JSON sidecar: 3x smaller and about 6x faster to + load at 1M (JSON stays the portable default; graph file and digest bytes are + unchanged; a stale or corrupt snapshot falls back to JSON). +- **Rust engine rework** (design in `isongraph-vector-rs/DESIGN.md`, measured + in `benchmarks/rust-stage-a/`): exact search uses AVX2/FMA kernels and + threads; searches take `&self` on a persistent pool, with batch search + (`similarity_search_vector_batch` and friends) that returns bit-for-bit the + same answers as one query at a time; `SharedEmbeddingStore` for threads that + read and write together; each vector is held once in a chunked matrix + (1.12x raw at 1M x 384, was 2.41x); pending rows are folded into the index + in groups of 256 at write time, so the first query after 30k writes into 1M + takes about 8 ms (was about 600 ms). Equal scores are ordered by arrival. + **Breaking:** `EmbeddingStore::get` and `SemanticGraph::get_embedding` now + return an owned `Option`; use `get_vector` to borrow. + Costs: single-thread exact scan is 0-13% slower, ingest is slower, and + updates at 1M are 2-3x slower because index upkeep moved from the first + search to the write. +- **Async encoders in TypeScript / JavaScript** - `AsyncEncoder` and + `...Async` methods (`addNodeAsync`, `updateNodeAsync`, `embedAllNodesAsync`, + `reembedStaleAsync`, `similaritySearchAsync`, `semanticMultiHopAsync`, ...) + let `HttpEncoder` auto-embed without a prefetch. Failures are atomic. The + synchronous interface and `prefetch` still work. +- **C++ libcurl transport is built and tested against real libcurl** (8.11.1, + loopback server, no internet). Two bugs fixed: `Expect: 100-continue` + stalling large bodies, and a misleading error for over-cap responses. A new + `cpp-curl.yml` workflow runs it on Linux and macOS. +- **`isongraph-vector-server` (alpha)** - a separate package that serves the + library over HTTP: per-tenant API keys (salted hashes), isolated graphs, + quotas and rate limits, search / multi-hop / path / subgraph / similar, + atomic checksummed snapshots with digest verification, a tenant-sharding + router and digest-verified read replicas. Not provided: TLS, failover, + resharding, a write-ahead log, cross-tenant queries. +- **Real-data benchmarks** - `benchmarks/multihop/` (HotpotQA, 2Wiki, + MuSiQue; two encoders; graphs built without the gold annotations; negative + controls; BM25, hybrid, cross-encoder and personalised-PageRank baselines) + and `benchmarks/competitive/` (faiss, hnswlib, usearch, chromadb, lancedb, + sqlite-vec, numpy on text, GloVe and SIFT vectors at 100k and 1M). ### Known limits - `C#` copies: nodes and edges read from the C# read-only graph view are copies, so editing one's `Properties` silently changes nothing. -- The JavaScript / TypeScript encoder interface is synchronous, so an - `HttpEncoder` only serves texts that were prefetched; auto-embedding in - `SemanticGraph` needs a `prefetch` first. -- The libcurl transport for the C++ `HttpEncoder` has not been built against a - real libcurl; it is off by default. -- Approximate search is Python-only, uses a numpy IVF index rather than HNSW, - and recall was measured on synthetic data, not real embeddings. +- The synchronous JavaScript / TypeScript encoder interface still only serves + prefetched texts; use the `...Async` methods to embed through a network + encoder. `loadGraph({onMismatch: 'reembed'})` has no async form. +- Approximate search is IVF-flat in every port; only Rust also has HNSW. Recall on real embeddings was + measured at 100k and 1M (`benchmarks/competitive/`); the index is not + persisted and is rebuilt after load. JavaScript builds are slow (about 8 + minutes at 1M vectors). +- **Not a vector-database replacement.** Against faiss, hnswlib and usearch + the ports are 2-4x slower per query at 100k and about 14x slower at 1M text + vectors at equal recall; only Rust has batched and concurrent search (the other ports serialise on one lock); vectors + persist as JSON only; and memory is 2-4x higher (Rust is now 1.1x raw; the other ports still store vectors twice). Graph-aware retrieval + beating plain vector search on multi-hop questions is the supported claim, + and it depends on the graph carrying real structure + (`benchmarks/multihop/README.md`). +- Fast filtered search was only measured for node-type filters, and only Rust + is fast there (Python's filtered search is no faster than its unfiltered + scan). The C++ exact scan is scalar and about 6x slower than numpy; the C# + and JS exact scans are also slower than numpy. - The digest cannot tell `1` from `1.0` in JavaScript / TypeScript, and integers beyond 64 bits are not supported in C# / C++ / Rust. Where a graph contains such values the loader warns instead of raising a false mismatch. @@ -212,8 +278,9 @@ JavaScript, Rust and C++. prerelease dependency (NU5104), so a stable 1.0.0 was not possible against it. The redistribution carries the same upstream C at the version the Rust, Node and C++ ports already use. -- Similarity search is brute force. There is no ANN index: this is built for an - application's own knowledge graph, not as a vector-database replacement. +- Similarity search is exact (brute force) by default, with an opt-in IVF + index; this is built for an application's own knowledge graph, not as a + vector-database replacement. - Payload vectors are float32 in every port, but the JSON text differs: Rust writes the shortest form that round-trips as f32 (`0.33590567`), Python widens to float64 first (`0.33590567111968994`). Same number, and importing diff --git a/DESCRIBE.md b/DESCRIBE.md index a3bb833..eb13fbd 100644 --- a/DESCRIBE.md +++ b/DESCRIBE.md @@ -272,10 +272,16 @@ the default `threshold` of 0.3, so she never seeds the search either. linear in the number of nodes. In Python the numpy path is vectorised and patches its cache on writes, which moved the point where a search passes 50 ms from roughly 90,000 vectors to (extrapolated) around a million; - see `benchmarks/`. Python has an opt-in IVF approximate index - (`ann="ivf"`) that pays off from about a million vectors; the other ports are - exact only. This stays an application's - own knowledge graph rather than a vector-database replacement. + see `benchmarks/`. Every in-memory store has an opt-in IVF approximate index + (`ann="ivf"` in Python; `set_ann`, `enableAnn`, `AnnMode.Ivf`, `ann: 'ivf'` + elsewhere) that pays off from roughly a million vectors; the standalone + SQLite stores have none. Measured against faiss, hnswlib and usearch + (`benchmarks/competitive/`), the ports are 2-4x slower per query at 100k and + about 14x slower at 1M vectors at equal recall, have no batched or + concurrent search, and persist vectors as JSON only. This stays an + application's own knowledge graph rather than a vector-database replacement; + its measured strength is graph-aware retrieval on multi-hop questions + (`benchmarks/multihop/`). - One embedding per node, generated from a single text field/concat - no multi-vector-per-node, no edge embeddings (that's RudraDB's domain, not this package's). @@ -304,8 +310,10 @@ the default `threshold` of 0.3, so she never seeds the search either. (or errors). Use `stale_embeddings()` and `reembed_stale()` to refresh them. An encoder `version` is recorded in the sidecar and warns on a change, but a version-only change is not treated as stale. -- Distributed or multi-tenant retrieval, access control and any server layer - are out of scope for an embedded library. +- The library itself stays embedded. A separate package, + `isongraph-vector-server` (alpha), adds multi-tenant access control, quotas, + a router that shards tenants across nodes and snapshot-based read replicas; + it has no automatic failover, no resharding and no TLS. - C++ remains the one port where the base graph library is vendored rather than declared as a dependency, because there is no package registry to declare it in. diff --git a/README.md b/README.md index 24d6795..e9c2fd6 100644 --- a/README.md +++ b/README.md @@ -342,11 +342,25 @@ every suite asserts that set. ## Honest limits -- **Search is brute force in every backend**, sqlite-vec included — all of them - linear in the number of nodes. This is built for an application's own - knowledge graph, not as a vector-database replacement. For vectors on both - nodes *and* edges, ANN indexing and link prediction, RudraDB is the right - tool; these are not really competitors despite the surface similarity. +- **Search is exact (brute force) by default**, sqlite-vec included — linear in + the number of nodes. Every in-memory store has an opt-in IVF index for large + stores. This is built for an application's own knowledge graph, not as a + vector-database replacement: measured against faiss, hnswlib and usearch + (`benchmarks/competitive/`) the ports are 2-4x slower per query at 100k and + about 14x slower at 1M vectors at equal recall, have no batched or concurrent + search, and persist vectors as JSON only. They do build indexes 8-25x faster + and take inserts and updates far faster than HNSW. For vectors on both + nodes *and* edges and link prediction, RudraDB is the right tool; these are + not really competitors despite the surface similarity. +- **Where it does help: multi-hop questions.** On HotpotQA, 2WikiMultihopQA and + MuSiQue, graph-aware retrieval found every supporting passage more often than + plain vector search (+9 to +51 points at k=10, paired p<0.001), but only when + the graph carries real structure: on a randomly relabelled graph the gain + disappears, and an independently written personalised-PageRank re-ranker + matches it. See `benchmarks/multihop/README.md`. +- **A separate server package exists** (`isongraph-vector-server`, alpha): + multi-tenant HTTP API, tenant-sharding router and read replicas. It has no + TLS, failover or write-ahead log. - **One embedding per node**, from one text field or a concatenation. No multi-vector nodes, no edge embeddings. - **`MockEncoder` has no semantic structure.** It exists so tests and demos diff --git a/ROADMAP.md b/ROADMAP.md index 8f0d901..8e3dbf4 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -6,30 +6,39 @@ What is deliberately not done yet, in rough priority order. Items marked ## Next 1. **Cut 1.0.0.** Nothing is tagged or published. Follow `RELEASING.md`. -2. **Run the new CI step once on real runners.** `graph_files.py` passes - locally on Windows only; the Linux paths (g++, TS symlink, dotnet) are - unproven. Also build the C++ libcurl transport once against real libcurl. -3. **Retrieval-quality benchmark on real data.** `benchmarks/` uses a - synthetic graph and synthetic vectors. Run it against a real knowledge - graph and a real encoder, add answer-groundedness, and measure ANN recall on - real embeddings. -4. **Per-record encoder version.** Today a version-only change warns on +2. **Run the new CI jobs once on real runners.** The cross-port graph-file + check and `cpp-curl.yml` have run on Linux only through the first PR; the + server package's tests and its Dockerfile have only run on Windows (the + Dockerfile has never been built). +3. **Make Rust the reference engine.** Done and measured in Rust + (`isongraph-vector-rs/DESIGN.md`, `benchmarks/rust-stage-c/`): concurrent + and batch search, one copy of the vectors, no post-write stall, opt-in HNSW + (level with faiss HNSW / hnswlib single-thread), int8 scan, binary snapshot. + Still open: port the proven pieces to the other ports; persist the ANN + index; faster HNSW single-call inserts (1.4k/s at 1M); exact scan is at + parity with faiss Flat, not ahead. +5. **Fix the slow ports.** The C++ exact scan has no scan cache and is scalar + (6x slower than numpy); JavaScript k-means takes about 8 minutes at 1M. +6. **Per-record encoder version.** Today a version-only change warns on load but cannot be fixed by `reembed_stale`. ## Later -- **Approximate search beyond Python**, and HNSW as an optional extra if IVF's - recall/latency trade-off proves too weak on real data. -- **Async encoders in TypeScript / JavaScript**, so an `HttpEncoder` works with - auto-embedding without a `prefetch`. +- **Binary persistence** for vectors and the index, as an addition to the + portable JSON sidecar (which stays the default and keeps the digest). +- **Approximate search for the standalone SQLite stores.** - **Relationship-type weights** (per type rather than a per-edge property), and richer node predicates than a callback. - **Multi-vector and edge embeddings.** - **C# loading Python's undirected graph files** - blocked on the IsonGraph NuGet package accepting the `meta.graph` block. +- **Server package:** TLS, a write-ahead log, failover and resharding, only if + the service is to be run beyond a trusted network. ## Out of scope -- Distributed indexing and retrieval, replication, multi-node availability. -- Multi-tenant access control and a service layer. Put those in front of the - library. +- Distributed indexing and retrieval inside the library, and multi-node + availability. The server package offers per-tenant sharding and + bounded-staleness snapshot replicas only. +- Competing with dedicated vector databases on raw query throughput. The + supported claim is graph-aware retrieval over up to about a million vectors. diff --git a/benchmarks/competitive/.gitignore b/benchmarks/competitive/.gitignore new file mode 100644 index 0000000..c0d02bb --- /dev/null +++ b/benchmarks/competitive/.gitignore @@ -0,0 +1,4 @@ +# Large captured console output and the discarded first (unpinned) run stay local +results/raw/*.stdout.txt +results/archive_unpinned/ +__pycache__/ diff --git a/benchmarks/competitive/README.md b/benchmarks/competitive/README.md new file mode 100644 index 0000000..124deba --- /dev/null +++ b/benchmarks/competitive/README.md @@ -0,0 +1,2240 @@ +# Competitive benchmark: ISONGraph Vector against the established vector-search libraries + +> **Update, 2026-11-16.** Everything below was measured before the Rust engine +> rework. For Rust, the "no batch or concurrent search" loss, the 2-4x memory +> loss and the slow first query after writes are addressed; see +> `benchmarks/rust-stage-a/RESULTS.md` for old-versus-new numbers. The Rust +> single-thread exact scan is now 0-13% slower than the kernel measured here. +> `benchmarks/rust-stage-c/README.md` re-ran numpy and faiss against the +> current build: the "Rust exact beats numpy / faiss Flat by 6-33%" win is +> **refuted** (parity: 0.91-1.2x faiss Flat, 1.0-1.45x numpy), while Rust now +> has an opt-in HNSW index that matches faiss HNSW and hnswlib single-thread, +> int8 scanning, and a binary snapshot. The other ports are unchanged. + +**One-paragraph answer.** On this machine, with identical vectors and an exact float64 ground truth, ISONGraph Vector is **not** the fastest vector search and the evidence does **not** support calling it the "best vector store". Its approximate index is a plain IVF-flat. At equal recall@10 its IVF is 1.4-2.3x slower than faiss IVFFlat at the same nlist and **1.9x (GloVe 100k) to 4.3x (text 100k) and 14-15x (text 1M) slower than the HNSW libraries** (hnswlib, faiss HNSW) in single-thread QPS; it has no batch or concurrent-query path, so its all-core throughput is 10-50x below faiss/hnswlib; and its only persistence is a JSON file (15-120x slower to save and load than faiss/hnswlib index files, 3-5x larger on disk). What it does well: the Rust exact scan is the fastest single-thread exact scan measured (6-33% ahead of numpy and faiss `IndexFlatIP` at 100k and 1M on text and GloVe, equal within noise at 10k); index build is 8-25x faster than HNSW builds at 100k-1M; the IVF index is maintained incrementally through insert/update/delete (faiss HNSW cannot update or delete; hnswlib/usearch can); and a node-type filter on a 1% subset is among the fastest filtered searches measured (see the filter table). Losses are listed as plainly as wins below. + +Everything here was measured on one laptop (i9-14900HX, 24 cores / 32 threads, 47.7 GB, Windows 11), see `results/environment.json`. Nothing in the libraries was changed. All numbers are in `results/raw/*.json`; the tables are generated from those files by `report.py`. + +## Read this first: what we can and cannot claim + +| claim | supported? | evidence | +|---|---|---| +| "ISONGraph is the best / fastest vector store" | **No** | HNSW libraries are 1.9-15x faster in single-thread QPS at equal recall; faiss IVFFlat is ~1.4-2.3x faster than our IVF at the same nlist; batch/all-core throughput is 10-50x behind. | +| "ISONGraph's Rust exact search is competitive with faiss Flat / numpy" | **Yes (single thread)** | text384: Rust 154 QPS vs numpy 144, faiss Flat 131 at 100k; 15.1 vs 14.2 / 13.4 at 1M; glove 100k 700 vs 528 / 548. Not true for batched/all-core search (faiss Flat batch is ~35x ours at 100k, 540 vs 15 QPS at 1M) or for the C++/C#/JS ports. | +| "The opt-in IVF index gives a large speed-up over our exact scan with a recall you can choose" | **Yes** | e.g. Rust text384 100k: 154 -> 828 QPS at recall 0.95 (5x); at 1M 15 -> 103 (7x). Python 131 -> 530. | +| "ISONGraph Python is competitive with Rust" | partly | After pinning, Python IVF (numpy/BLAS) is within 1.5-2x of the Rust IVF, and Python exact is within 15% of Rust at 100k text. | +| "We are competitive with other embedded stores (lancedb, chromadb, sqlite-vec, qdrant local mode)" | **Yes, for exact and IVF; not vs chroma's HNSW** | sqlite-vec, lancedb flat and qdrant local mode are 10-100x slower than our exact scans; lancedb IVF_FLAT is 2.5-4x slower than our IVF; chromadb (HNSW) is faster than our IVF at 100k text (906 vs 828 QPS at 0.95) and 5x faster at 1M. | +| "Incremental updates are a strength" | **Yes versus HNSW, no edge versus faiss IVF** | The IVF index stays consistent through insert/update/delete (Rust 1M text: 35k inserts/s, 430k updates/s, 590k deletes/s, recall 0.951 -> 0.952) versus 3-5k inserts/s and updates/s for hnswlib/usearch at 1M; faiss HNSW cannot update or delete at all; faiss IVFFlat does the same job at 72-80k/s inserts. The first search after 30k writes at 1M costs 0.1-0.7 s (Rust 0.6 s) and 5.7 s in JS. | +| "Filtered search is a strength" | **Only for node-type equality** | Our only filter is `node_type`; arbitrary metadata predicates do not exist. On a 1% subset Rust gave 15,288 QPS at recall 1.0 (100k text) against hnswlib 259 (recall 0.98) and chromadb 21; see the filter table for faiss with a bitmap selector. Python/C++ filtered search is 25-120x slower than Rust (widening to an exact scan). | +| "Persistence is competitive" | **No** | JSON sidecar only: 100k x 384 takes 1.9-16 s to save and 1.3-11 s to load per port (450-810 MB), and the IVF index is rebuilt on the first query after load (Python: 5.1 s more). Rust at 1M: 4.5 GB, 21 s save, 20 s to ready. faiss/hnswlib/usearch: 0.06-0.2 s save, 150-180 MB. | +| "Memory efficiency" | **No** | 1M x 384: faiss HNSW 1.6 GB, hnswlib 1.8 GB, faiss IVF 1.9 GB vs Rust 3.5 GB, C++ 3.3 GB, C# 6.7 GB, JS 7.0 GB, Python 4.2 GB (records plus a second contiguous copy). | + +## Headline results (single thread, QPS at recall@10; higher is better) + +QPS at recall@10 = 0.95, copied from `results/tables_cells.md` (interpolated between measured sweep points; `*` = conservative measured point because the bracket was wide). + +| system | text384 100k | text384 1M | glove 100k | glove 1M | sift 100k | +|---|---|---|---|---|---| +| hnswlib | 3,490 | 1,486 | 3,762 | not reached (ef<=640) | 22,439 | +| faiss HNSWFlat | 3,542 | 1,572 | 2,152 | not reached | 17,392 | +| usearch | 2,230 | 706 | 1,947 | not reached | 13,215 | +| chromadb | 906 | 605 | 933 | not reached | 1,163* | +| faiss IVFFlat | 1,193 | 190 | 2,893 | 43.8* | 8,764* | +| **ISONGraph Rust IVF** | 828 | 103 | 2,029 | 181 | 4,873 | +| **ISONGraph C++ IVF** (default SSE2 / AVX2 build) | 681 / 957 | 94 / 128 | 1,919 / 2,167 | 175 / 185 | 4,054* / 4,693* | +| **ISONGraph C# IVF** | 615 | not reached | 1,268 | not reached | 2,805* | +| **ISONGraph Python IVF** | 530 | 69 | 813 | 156 | 2,622 | +| **ISONGraph JS IVF** | 201 | not reached | 331 | not run | 795 | +| lancedb IVF_FLAT | 222 | 63 | 238 | 109 | 247 | +| *exact:* **ISONGraph Rust** | 154 | 15.1 | 700 | 43.5 | 344 | +| *exact:* numpy | 144 | 14.2 | 528 | 34.4 | 282 | +| *exact:* faiss Flat | 131 | 13.4 | 548 | 40.7 | 283 | +| *exact:* **ISONGraph Python** | 131 | 12.7 | 471 | 26.5 | 318 | +| *exact:* **ISONGraph C#** | 39 | 3.7 | 108 | 5.8 | 55 | +| *exact:* **ISONGraph C++** | 22.6 | 2.1 | 40.1 | 2.5 | 22.8 | +| *exact:* **ISONGraph JS** | 19.7 | 1.7 | 48.1 | not run | 29 | +| *exact:* sqlite-vec / lancedb / qdrant-local | 10 / 7 / 7 | 1.05 / 0.7 / - | 40 / 34 / - | - | 22 / 15 / - | + +SIFT 1M (reduced set, QPS @0.95, 1T): hnswlib 7,311, faiss HNSW 5,603, faiss IVFFlat 1,270*, **Rust IVF 546***, **C++ IVF 562***; exact: **Rust 33.4**, numpy 26.9, **C++ 2.2**. + +"not reached": the sweep (HNSW ef up to 640, IVF nprobe up to 256 at 1M) never got to recall 0.95 at M=16 / nlist=4000. HNSW with larger M would; we held M=16 for everyone. + +Build time and memory (text384; ready = ingest + index + first query): + +| system | 100k ready s | 1M ready s | 1M steady RSS (MB over baseline) | +|---|---|---|---| +| ISONGraph Rust IVF | 0.62 | 7.6 | 3,532 | +| ISONGraph C++ IVF (default / AVX2) | 0.97 / 0.61 | 12.4 / 7.1 | 3,279 | +| ISONGraph Python IVF | 3.6 | 42 | 4,251 | +| ISONGraph JS IVF | 57 | 522 | 7,002 | +| faiss IVFFlat | 1.7 | 18.6 | 1,885 | +| faiss HNSWFlat | 8.9 | 221 | 1,640 | +| hnswlib | 6.2 | 173 | 1,830 | +| usearch | 5.9 | 174 | 1,715 | +| chromadb | 14.7 | 345 | 4,023 | + +## Wins and losses, ranked + +Wins (measured): +1. Rust exact scan: 1-thread QPS 6-33% above numpy and faiss Flat in every 100k/1M cell measured (text, glove, sift), equal within noise at 10k. Single thread only; ours has no batch path (faiss Flat batch is 35x faster at text 100k). +2. Build time versus HNSW: 8-25x faster (text 1M: Rust 7.6 s vs 173-221 s for HNSW; 100k: 0.6 s vs 6-9 s); versus faiss IVFFlat 2.4x faster at text 1M (7.6 s vs 18.6 s) but not at 100k (0.6 vs 1.7 s, i.e. 2.8x). +3. Index maintenance: 7-100x faster writes than the HNSW libraries (Rust 1M text: 35k inserts/s, 430k updates/s, 590k deletes/s vs hnswlib 4.8k/3.2k/-), recall unchanged (0.951 -> 0.952). +4. Filtered search on a node-type subset, Rust only: 1% of 100k text 15,288 QPS at recall 1.0 vs faiss Flat (bitmap selector) 2,491, faiss IVF 1,630, hnswlib 259 @0.98; 10%: 3,094 vs 464 / 388 / 1,779 @0.91; at text 1M 10%: 650 QPS @0.94 vs faiss Flat 40, hnswlib 406 @0.955; 1%: 1,944 @0.906 vs 172 / 135 @0.945. Python and C++ filtered search is 25-120x slower than Rust (widening to an exact scan). +5. Python IVF on numpy/BLAS is within 1.5-2x of the native ports and is the only port besides Rust/C++/C# with realistic speed at 1M. + +Losses (measured), worst first: +1. All-core / batched throughput: no batch API and one lock per store. text 100k @0.95: faiss HNSW 19,900 QPS, hnswlib 19,900, faiss Flat batch 5,437 vs ours <=1,518 (Rust auto-threads loop, 1.0x its own 1T loop). The "8 concurrent clients" probe confirms no scaling (`concurrent_clients` in the JSON). +2. Persistence: JSON sidecar, 3-5x larger and 15-120x slower than faiss/hnswlib index files; IVF rebuilt on load. Python 100k: save 15.5 s, load 11 s, then 5.1 s for the first query (index rebuild). +3. QPS at equal recall vs HNSW: 4-6x slower at 100k, 12-15x slower at 1M (text384). +4. QPS vs faiss IVFFlat at the same nlist: Rust 0.69x (text 100k), 0.54x (text 1M), 0.70x (glove 100k); C++ default build 0.57x; Python 0.44x. +5. Memory: 1.8-4x the HNSW/IVF libraries at 1M (the store keeps two copies of every vector). +6. C++ exact scan: 22.6 QPS vs numpy 144 at text 100k (6.4x slower), JS 7x slower, C# 3.7x slower (see library issues). +7. JS IVF build: 522 s at 1M (30x slower than Rust). +8. First search after heavy writes on a 1M index costs 0.1-0.7 s in Rust/C++/C#/Python and 5.7 s in JS (lazy fold / rebuild), versus about 1 ms for hnswlib/usearch. + +## Cross-process variation: why the 1-thread loops are pinned + +A first full run of the whole matrix (kept in `results/archive_unpinned/`) used unpinned single-thread loops. A second run of 12 cells hours later differed by **-7% to +227%** per system (for example faiss HNSW 1,083 vs 3,259 QPS at the same recall, Rust IVF 229 vs 748; `archive_unpinned/tables_repeat.md`), while the three timed runs inside each process agreed to 5%. On this hybrid CPU (8 P-cores + 16 E-cores) an unpinned thread may land on an E-core (measured 2.0x slower) and the machine as a whole spent some hours in a slower state. **All numbers in this README come from a second, complete run in which every single-thread loop is pinned to one P-core (logical CPU 8) at HIGH priority** (`bench_py.py: pinned`, `competitive_bench.{rs,cpp}: Pin`, C# `Point`, node pinned to P-cores 8,10,12,14 by `run_all.py`). Three timed runs per point agree to 1-5% (min-max in the full sweep tables). A second independent pinned process repeat of the final data set was not possible in the time available; cross-process stability of the pinned numbers is established only by a Rust spot-check (775 / 761 / 779 QPS in three separate processes at nprobe=128, text384 100k) and by the agreement of the three in-process runs. Treat differences below ~10% between systems as noise. + + + +## How this was measured + +### Data (identical normalised float32 vectors and queries for every system) + +| dataset | vectors | dim | source | queries | +|---|---|---|---|---| +| `text384` | 1,060,000 passages (first 100k / 1M used) | 384 | **real text**: seeded random sample of the BeIR/hotpotqa Wikipedia-abstract corpus (5.2M passages) embedded with `sentence-transformers/all-MiniLM-L6-v2` (fp16 on GPU, `max_seq_length` 128, L2-normalised) | 2,000 real HotpotQA questions | +| `glove` | 1,183,514 (first 100k / 1M used) | 100 | ann-benchmarks `glove-100-angular` | first 2,000 of its test queries | +| `sift` | 1,000,000 (100k / 1M used) | 128 | ann-benchmarks `sift-128-euclidean`, **L2-normalised, scored as cosine** | first 2,000 of its test queries | + +* The HDF5 files' own neighbour lists are **not** used. Ground truth is the exact **float64 cosine** top-100 over the same normalised float32 vectors (`prep_gt.py`), so one definition of "right" applies to all systems. Recall@10 = |returned ∩ true top-10| / 10, averaged over the queries. (On vectors with exact score ties the id sets can differ trivially; none of the datasets have duplicate vectors in the first 100k.) +* Because the vectors are unit length, cosine = inner product, so every system is configured with the metric its documentation recommends for normalised vectors (`ip`/`dot`/`cosine`) and returns the same ranking. +* Streaming-mutation and filter tests use deterministic plans written once by `prep_gt.py` (seeded): 10k inserted ids, 10k updated ids (a different vector is stored under an existing id), 10k deleted ids, all disjoint; a `grp` column splitting the ids into a random **10%** subset, a random **1%** subset and the remaining 89%. Ground truth after mutation / inside each subset is recomputed exactly. + +### What is measured, and how + +* **Recall versus queries/second** (`bench_py.py`, `competitive_bench.{rs,cpp}`, `...CompetitiveBench`, `bench/competitive.mjs`): every approximate system is swept over its accuracy knob (`nprobe` for IVF, `efSearch`/`ef`/`expansion_search`/`ef_search` for HNSW). At every knob value: warm-up (up to 50 queries or 3 s), then **three timed runs** of a strictly **single-threaded, one-query-at-a-time loop** (each query individually timed with a monotonic clock: median/p95/p99 latency from the pooled samples; QPS = queries / loop time; the min-max of the three runs is the run-to-run variation), then, where the system supports it, **three timed runs of a batched call using all 32 logical CPUs**. Up to 2,000 queries per run; systems slower than ~10 s per run are given fewer queries (the count is recorded as `nq`, never below 30). +* **Thread counts**: single-thread loops pin every library to one thread (`faiss.omp_set_num_threads(1)`, `hnswlib num_threads=1`, `usearch threads=1`, BLAS limited with `threadpoolctl`, ISONGraph Rust `set_scan_threads(1)`). The "all-core" column uses 32 threads (24 cores / 32 logical CPUs, hybrid P/E). The ISONGraph ports have **no batched search API and serialise searches on one lock**, so their all-core column is the *same one-query-at-a-time loop* with the library's automatic scan-thread choice (marked †), and a separate "N concurrent clients" probe shows the lock. +* **Build time**: ingest (all vectors added through the library's own add path from pre-parsed arrays) + index build/training + first query (some systems build lazily on the first query). At 100k the build is repeated three times in fresh instances. +* **Memory**: process working set (RSS, `psutil`/`GetProcessMemoryInfo`) minus the baseline measured after imports and before any data is touched. The harness streams vectors from disk in 20k-row chunks and keeps none, so the number is the system's own footprint. "Peak" is the peak working set after the build. File-backed engines (lancedb, chromadb, sqlite-vec) memory-map or cache data and are flattered by RSS; see the caveats. +* **Streaming mutation** (on the already-built index at the recall >= 0.95 operating point): insert 10k new vectors, update 10k existing ids, delete 10k ids, in batches of 1,000 through each system's natural API; recall@10 is re-measured against the exact post-mutation ground truth with the same knob value. `unsupported` means the library offers no such operation on that index type (it would need a rebuild). +* **Filtered search**: a separate build that carries the metadata column (ISONGraph: the node *type*, its only filter; others: an integer payload/metadata column), then restricted search over the 10% and 1% subsets at five accuracy settings. Reported: best single-thread QPS with recall >= 0.90 and the best recall reached. +* **Save/load**: each system's native persistence (faiss/hnswlib/usearch index files; lancedb/chroma/sqlite-vec are on-disk stores, so "save" is ~0 and "load" is reopening; ISONGraph's persistence is the JSON embeddings sidecar and the IVF index is rebuilt after load, which is why its time-to-ready is long). +* **Exact search** (`exact` tables): ISONGraph exact scans (Python numpy, Rust scan cache, C++ hash-map scan, C#, JS) against `faiss.IndexFlatIP`, plain numpy and the other exact engines, at 10k/100k/1M. + +### Parameters that are held equal + +| | value | +|---|---| +| HNSW (faiss, hnswlib, usearch, chromadb) | M (connectivity) = 16, efConstruction (expansion_add) = 200 | +| IVF (ISONGraph, faiss, lancedb) | nlist = round(4·sqrt(N)) (4,000 at 1M: ISONGraph's automatic choice, imposed on the others); k-means on a sample of ~40 points per cell for faiss, ISONGraph uses its own sample/8 iterations; lancedb its default sampling | +| k | 10 | +| Everything else | library defaults | + +### Versions and hardware + +See `results/environment.json` (written by `collect_env.py`). + + +## Results tables (generated by `report.py` from `results/raw/*.json`) + + +Plots (recall vs QPS, single thread and batched): `results/plots/__{1t,batch}.png`. + + +### Recall versus QPS, build time and memory, per dataset and size + +### text384, N = 100,000 + +Queries/second at recall@10 = 0.90 / 0.95 / 0.99, log-linearly interpolated between the two measured sweep points that bracket the target (no extrapolation; `*` = the two bracketing points are more than 0.05 recall apart, so the measured point above the target is shown instead, a conservative value; `-` = never reached; exact scans appear at their single recall-1.0 point). 1T = one pinned P-core, one query at a time. All-core = batched call on 32 logical CPUs (`†` = the ISONGraph ports have no batch API, so this is the same one-query loop with automatic scan threads). Latency columns are for the first measured sweep point with recall >= 0.95. + +| system | QPS 1T @0.90 | @0.95 | @0.99 | all-core QPS @0.90 | @0.95 | 1T latency ms median / p95 / p99 | +|---|---|---|---|---|---|---| +| **ISONGraph C++, /arch:AVX2 (exact)** | 23.0 | 23.0 | 23.0 | - | - | 43.5 / 45.4 / 47.6 (recall 1.000) | +| **ISONGraph C++, /arch:AVX2 (IVF)** | 1697* | 957 | 114 | - | - | 1.10 / 1.30 / 1.48 (recall 0.954) | +| **ISONGraph C++ (exact)** | 22.6 | 22.6 | 22.6 | - | - | 44.2 / 47.2 / 57.7 (recall 1.000) | +| **ISONGraph C++ (IVF)** | 1232* | 681 | 93.3 | - | - | 1.55 / 1.81 / 2.00 (recall 0.954) | +| **ISONGraph C# (exact)** | 39.1 | 39.1 | 39.1 | - | - | 23.6 / 26.7 / 78.4 (recall 1.000) | +| **ISONGraph C# (IVF)** | 1134* | 615 | 112 | - | - | 1.66 / 2.00 / 2.22 (recall 0.952) | +| **ISONGraph Python (exact)** | 131 | 131 | 131 | - | - | 7.37 / 8.92 / 14.3 (recall 1.000) | +| **ISONGraph Python (IVF)** | 937* | 530 | 190 | - | - | 1.93 / 2.34 / 2.83 (recall 0.954) | +| **ISONGraph JS (exact)** | 19.7 | 19.7 | 19.7 | - | - | 50.8 / 55.8 / 58.1 (recall 1.000) | +| **ISONGraph JS (IVF)** | 358* | 201 | 51.3 | - | - | 5.11 / 6.06 / 6.71 (recall 0.953) | +| **ISONGraph Rust (exact)** | 154 | 154 | 154 | - | - | 6.41 / 6.99 / 7.86 (recall 1.000) | +| **ISONGraph Rust (IVF)** | 1562* | 828 | 253 | 1667* † | 1518 † | 1.25 / 1.51 / 1.63 (recall 0.953) | +| **ISONGraph Rust, target-cpu=native (exact)** | 163 | 163 | 163 | - | - | 6.06 / 6.79 / 7.39 (recall 1.000) | +| **ISONGraph Rust, target-cpu=native (IVF)** | 1506* | 802 | 254 | 1676* † | 1484 † | 1.30 / 1.57 / 1.70 (recall 0.953) | +| chromadb (HNSW) | 1029* | 906 | 602 | 4130* | 2890 | 1.14 / 1.34 / 1.54 (recall 0.965) | +| faiss IndexFlatIP | 131 | 131 | 131 | 5437 | 5437 | 7.29 / 8.84 / 15.0 (recall 1.000) | +| faiss HNSWFlat | 5913* | 3542 | 1329 | 30747* | 19930 | 0.296 / 0.377 / 0.553 (recall 0.957) | +| faiss IVFFlat | 2078* | 1193 | 317 | 11708* | 6189 | 0.866 / 1.07 / 1.26 (recall 0.955) | +| hnswlib | 5400 | 3490 | 1275 | 29766 | 19875 | 0.354 / 0.440 / 0.587 (recall 0.964) | +| lancedb (no index) | 7.19 | 7.19 | 7.19 | - | - | 137 / 151 / 173 (recall 1.000) | +| lancedb IVF_FLAT | 256* | 222 | 120 | - | - | 4.52 / 5.16 / 6.06 (recall 0.956) | +| numpy brute force | 144 | 144 | 144 | 1553 | 1553 | 6.75 / 7.66 / 10.1 (recall 1.000) | +| qdrant-client local mode (exact numpy) | 7.24 | 7.24 | 7.24 | - | - | 136 / 152 / 158 (recall 1.000) | +| sqlite-vec (vec0, exact) | 10.2 | 10.2 | 10.2 | - | - | 95.5 / 115 / 122 (recall 1.000) | +| usearch (HNSW) | 3624 | 2230 | 743 | 29559 | 19546 | 0.540 / 0.681 / 0.848 (recall 0.964) | + +Build and memory (RSS growth over the process baseline after imports; the raw vectors are streamed in from disk and are not held by the harness). + +| system | ingest s | index s | total build s | 3 builds: median (min-max) s | ready s (incl. first query) | steady RSS +MB | peak RSS +MB (build) | +|---|---|---|---|---|---|---|---| +| **ISONGraph C++, /arch:AVX2 (exact)** | 0.081 | 0.000 | 0.081 | - | 0.125 | 192 | 206 | +| **ISONGraph C++, /arch:AVX2 (IVF)** | 0.081 | 0.525 | 0.607 | - | 0.607 | 342 | 417 | +| **ISONGraph C++ (exact)** | 0.077 | 0.000 | 0.077 | - | 0.122 | 192 | 206 | +| **ISONGraph C++ (IVF)** | 0.077 | 0.891 | 0.968 | - | 0.968 | 342 | 417 | +| **ISONGraph C# (exact)** | 0.018 | 0.000 | 0.018 | - | 0.065 | 291 | 291 | +| **ISONGraph C# (IVF)** | 0.016 | 1.27 | 1.28 | - | 1.28 | 788 | 788 | +| **ISONGraph Python (exact)** | 2.01 | 0.000 | 2.01 | 1.96 (1.90-2.01) | 2.31 | 454 | 571 | +| **ISONGraph Python (IVF)** | 1.97 | 1.60 | 3.56 | 3.56 (3.48-3.57) | 3.57 | 481 | 693 | +| **ISONGraph JS (exact)** | 0.030 | 0.000 | 0.030 | - | 0.095 | 518 | 518 | +| **ISONGraph JS (IVF)** | 0.027 | 56.7 | 56.7 | - | 56.7 | 1,216 | 1,216 | +| **ISONGraph Rust (exact)** | 0.056 | 0.000 | 0.056 | - | 0.136 | 339 | 339 | +| **ISONGraph Rust (IVF)** | 0.056 | 0.480 | 0.616 | - | 0.616 | 342 | 420 | +| **ISONGraph Rust, target-cpu=native (exact)** | 0.058 | 0.000 | 0.058 | - | 0.137 | 339 | 339 | +| **ISONGraph Rust, target-cpu=native (IVF)** | 0.058 | 0.471 | 0.609 | - | 0.609 | 342 | 420 | +| chromadb (HNSW) | 14.6 | 0.000 | 14.6 | 14.8 (14.6-15.0) | 14.7 | 486 | 385 | +| faiss IndexFlatIP | 0.108 | 0.000 | 0.108 | 0.108 (0.108-0.111) | 0.117 | 172 | 317 | +| faiss HNSWFlat | 8.90 | 0.000 | 8.90 | 8.67 (8.41-8.90) | 8.90 | 184 | 328 | +| faiss IVFFlat | 0.000 | 1.72 | 1.72 | 1.83 (1.72-1.85) | 1.74 | 238 | 459 | +| hnswlib | 6.21 | 0.000 | 6.21 | 6.21 (6.09-6.21) | 6.21 | 194 | 226 | +| lancedb (no index) | 0.219 | 0.000 | 0.219 | 0.212 (0.211-0.219) | 0.284 | 149 | 224 | +| lancedb IVF_FLAT | 0.234 | 1.66 | 1.89 | 1.80 (1.77-1.89) | 1.98 | 339 | 487 | +| numpy brute force | 0.030 | 0.000 | 0.030 | 0.030 (0.030-0.030) | 0.037 | 159 | 191 | +| qdrant-client local mode (exact numpy) | 284 | 0.000 | 284 | - | 284 | 399 | 567 | +| sqlite-vec (vec0, exact) | 0.967 | 0.000 | 0.967 | 0.964 (0.952-0.967) | 1.07 | 4 | 62 | +| usearch (HNSW) | 5.86 | 0.000 | 5.86 | 5.86 (5.73-6.01) | 5.87 | 182 | 213 | + +
Full accuracy sweeps (recall@10, single-thread QPS, latency, all-core QPS) + +**ISONGraph C++, /arch:AVX2 (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 23.0 (23.0-23.1) | 43.5 | 45.4 | 47.6 | - | 68 | + +**ISONGraph C++, /arch:AVX2 (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2808 | 17264 (17032-17852) | 0.056 | 0.069 | 0.080 | - | 2000 | +| 2 | 0.4022 | 14310 (14254-14358) | 0.068 | 0.086 | 0.102 | - | 2000 | +| 4 | 0.5291 | 11488 (11327-11627) | 0.085 | 0.110 | 0.132 | - | 2000 | +| 8 | 0.6479 | 8130 (7995-8407) | 0.118 | 0.158 | 0.212 | - | 2000 | +| 16 | 0.7548 | 5301 (5296-5315) | 0.183 | 0.243 | 0.305 | - | 2000 | +| 32 | 0.8427 | 3053 (3021-3054) | 0.319 | 0.414 | 0.486 | - | 2000 | +| 64 | 0.9091 | 1697 (1694-1723) | 0.576 | 0.717 | 0.869 | - | 2000 | +| 128 | 0.9543 | 902 (900-906) | 1.10 | 1.30 | 1.48 | - | 2000 | +| 256 | 0.9813 | 455 (452-462) | 2.15 | 2.59 | 2.98 | - | 985 | +| 1265 | 1.0000 | 23.0 (23.0-23.2) | 43.4 | 45.5 | 46.5 | - | 65 | + +**ISONGraph C++ (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 22.6 (22.5-23.3) | 44.2 | 47.2 | 57.7 | - | 66 | + +**ISONGraph C++ (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2808 | 16276 (16014-17282) | 0.059 | 0.077 | 0.090 | - | 2000 | +| 2 | 0.4022 | 14590 (14424-14658) | 0.067 | 0.089 | 0.110 | - | 2000 | +| 4 | 0.5291 | 10815 (10091-10998) | 0.091 | 0.127 | 0.151 | - | 2000 | +| 8 | 0.6479 | 6399 (6091-6902) | 0.152 | 0.206 | 0.247 | - | 2000 | +| 16 | 0.7548 | 3908 (3831-3928) | 0.250 | 0.336 | 0.409 | - | 2000 | +| 32 | 0.8427 | 2255 (2200-2287) | 0.436 | 0.570 | 0.649 | - | 2000 | +| 64 | 0.9091 | 1232 (1225-1234) | 0.806 | 0.993 | 1.10 | - | 2000 | +| 128 | 0.9542 | 641 (639-645) | 1.55 | 1.81 | 2.00 | - | 1869 | +| 256 | 0.9811 | 331 (330-332) | 3.01 | 3.38 | 3.66 | - | 1016 | +| 1265 | 1.0000 | 22.5 (22.4-22.6) | 44.5 | 46.8 | 48.1 | - | 67 | + +**ISONGraph C# (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 39.1 (38.6-39.9) | 23.6 | 26.7 | 78.4 | - | 128 | + +**ISONGraph C# (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2776 | 9706 (5430-9894) | 0.104 | 0.212 | 0.231 | - | 2000 | +| 2 | 0.3992 | 8524 (8519-8604) | 0.113 | 0.136 | 0.157 | - | 2000 | +| 4 | 0.5201 | 6881 (6817-6932) | 0.143 | 0.176 | 0.200 | - | 2000 | +| 8 | 0.6376 | 4982 (4893-5001) | 0.197 | 0.248 | 0.277 | - | 2000 | +| 16 | 0.7452 | 3315 (3315-3349) | 0.296 | 0.372 | 0.411 | - | 2000 | +| 32 | 0.8359 | 1975 (1971-2019) | 0.494 | 0.620 | 0.690 | - | 2000 | +| 64 | 0.9060 | 1134 (1127-1135) | 0.878 | 1.07 | 1.16 | - | 2000 | +| 128 | 0.9521 | 598 (591-600) | 1.66 | 2.00 | 2.22 | - | 1848 | +| 256 | 0.9805 | 306 (305-306) | 3.27 | 3.69 | 3.90 | - | 897 | +| 1265 | 1.0000 | 38.6 (37.4-39.4) | 23.7 | 25.1 | 95.1 | - | 122 | + +**ISONGraph Python (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 0.9999 | 131 (129-131) | 7.37 | 8.92 | 14.3 | - | 411 | + +**ISONGraph Python (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2833 | 8545 (8514-8573) | 0.114 | 0.139 | 0.175 | - | 2000 | +| 2 | 0.4083 | 7156 (6794-7226) | 0.135 | 0.175 | 0.216 | - | 2000 | +| 4 | 0.5305 | 5400 (5019-5451) | 0.180 | 0.230 | 0.324 | - | 2000 | +| 8 | 0.6501 | 3877 (3854-4006) | 0.245 | 0.302 | 0.351 | - | 2000 | +| 16 | 0.7542 | 2640 (2575-2669) | 0.369 | 0.458 | 0.593 | - | 2000 | +| 32 | 0.8428 | 1581 (1579-1603) | 0.591 | 0.741 | 1.08 | - | 2000 | +| 64 | 0.9091 | 937 (927-947) | 1.03 | 1.26 | 1.68 | - | 2000 | +| 128 | 0.9540 | 501 (500-506) | 1.93 | 2.34 | 2.83 | - | 1546 | +| 256 | 0.9809 | 268 (268-271) | 3.59 | 4.14 | 5.35 | - | 814 | +| 1265 | 0.9999 | 131 (130-132) | 7.36 | 8.47 | 14.2 | - | 422 | + +**ISONGraph JS (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 19.7 (19.1-20.3) | 50.8 | 55.8 | 58.1 | - | 56 | + +**ISONGraph JS (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2654 | 2125 (2089-2145) | 0.467 | 0.532 | 0.590 | - | 2000 | +| 2 | 0.3867 | 1927 (1927-1931) | 0.513 | 0.597 | 0.674 | - | 2000 | +| 4 | 0.5155 | 1637 (1632-1653) | 0.602 | 0.718 | 0.794 | - | 2000 | +| 8 | 0.6369 | 1304 (1294-1307) | 0.763 | 0.897 | 0.968 | - | 2000 | +| 16 | 0.7420 | 933 (929-937) | 1.07 | 1.27 | 1.40 | - | 2000 | +| 32 | 0.8353 | 607 (605-609) | 1.65 | 1.95 | 2.11 | - | 1779 | +| 64 | 0.9045 | 358 (357-359) | 2.77 | 3.29 | 3.55 | - | 1172 | +| 128 | 0.9527 | 194 (193-195) | 5.11 | 6.06 | 6.71 | - | 607 | +| 256 | 0.9828 | 103 (100-103) | 9.70 | 11.3 | 12.3 | - | 319 | +| 1265 | 1.0000 | 19.5 (19.4-20.0) | 50.4 | 54.3 | 59.1 | - | 59 | + +**ISONGraph Rust (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 154 (154-154) | 6.41 | 6.99 | 7.86 | - | 476 | + +**ISONGraph Rust (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2730 | 24266 (23139-24374) | 0.041 | 0.053 | 0.061 | 22169 † | 2000 | +| 2 | 0.3918 | 17640 (17404-17743) | 0.056 | 0.071 | 0.079 | 17989 † | 2000 | +| 4 | 0.5168 | 12425 (12047-12885) | 0.079 | 0.102 | 0.116 | 12368 † | 2000 | +| 8 | 0.6402 | 7664 (7441-8108) | 0.126 | 0.169 | 0.201 | 8336 † | 2000 | +| 16 | 0.7450 | 5069 (4964-5091) | 0.196 | 0.251 | 0.285 | 5069 † | 2000 | +| 32 | 0.8346 | 2840 (2774-2848) | 0.349 | 0.452 | 0.509 | 2891 † | 2000 | +| 64 | 0.9071 | 1562 (1537-1569) | 0.634 | 0.800 | 0.870 | 1667 † | 2000 | +| 128 | 0.9529 | 793 (791-796) | 1.25 | 1.51 | 1.63 | 1509 † | 2000 | +| 256 | 0.9807 | 401 (393-402) | 2.50 | 2.88 | 3.20 | 1084 † | 1148 | +| 1265 | 1.0000 | 154 (153-157) | 6.32 | 7.30 | 8.65 | 300 † | 478 | + +**ISONGraph Rust, target-cpu=native (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 163 (161-164) | 6.06 | 6.79 | 7.39 | - | 501 | + +**ISONGraph Rust, target-cpu=native (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2730 | 24669 (23997-24957) | 0.040 | 0.051 | 0.060 | 24453 † | 2000 | +| 2 | 0.3918 | 17732 (17481-17842) | 0.056 | 0.071 | 0.081 | 18452 † | 2000 | +| 4 | 0.5168 | 13134 (12835-13139) | 0.075 | 0.099 | 0.111 | 12737 † | 2000 | +| 8 | 0.6402 | 8489 (8363-8498) | 0.116 | 0.153 | 0.177 | 8381 † | 2000 | +| 16 | 0.7450 | 4821 (4660-4993) | 0.203 | 0.269 | 0.310 | 5055 † | 2000 | +| 32 | 0.8346 | 2789 (2748-2792) | 0.355 | 0.458 | 0.508 | 2838 † | 2000 | +| 64 | 0.9071 | 1506 (1480-1529) | 0.656 | 0.830 | 0.914 | 1676 † | 2000 | +| 128 | 0.9529 | 769 (761-774) | 1.30 | 1.57 | 1.70 | 1472 † | 2000 | +| 256 | 0.9807 | 387 (385-389) | 2.58 | 2.94 | 3.20 | 1095 † | 1127 | +| 1265 | 1.0000 | 161 (161-163) | 6.05 | 6.99 | 8.32 | 310 † | 470 | + +**chromadb (HNSW)** (ef_search) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.6149 | 1361 (1337-1385) | 0.702 | 0.850 | 0.974 | 10714 | 2000 | +| 16 | 0.7140 | 1254 (1229-1255) | 0.787 | 0.910 | 1.07 | 8882 | 2000 | +| 32 | 0.8313 | 1170 (1131-1187) | 0.848 | 0.988 | 1.28 | 6354 | 2000 | +| 64 | 0.9152 | 1029 (1013-1034) | 0.964 | 1.12 | 1.39 | 4130 | 2000 | +| 128 | 0.9646 | 858 (851-898) | 1.14 | 1.34 | 1.54 | 2489 | 2000 | +| 256 | 0.9870 | 680 (676-703) | 1.44 | 1.66 | 1.96 | 1445 | 2000 | +| 512 | 0.9955 | 483 (466-490) | 2.06 | 2.41 | 2.86 | 821 | 1479 | + +**faiss IndexFlatIP** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 0.9999 | 131 (130-135) | 7.29 | 8.84 | 15.0 | 5437 | 410 | + +**faiss HNSWFlat** (efSearch) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.6361 | 15713 (15200-17057) | 0.058 | 0.088 | 0.118 | 96809 | 2000 | +| 16 | 0.7315 | 12783 (12779-12881) | 0.073 | 0.105 | 0.136 | 70072 | 2000 | +| 24 | 0.8014 | 9414 (9092-9648) | 0.097 | 0.135 | 0.217 | 50335 | 2000 | +| 32 | 0.8492 | 7879 (7578-8243) | 0.121 | 0.164 | 0.210 | 44864 | 2000 | +| 48 | 0.9004 | 5913 (5763-5919) | 0.165 | 0.209 | 0.261 | 30747 | 2000 | +| 64 | 0.9279 | 4554 (3996-4704) | 0.217 | 0.305 | 0.395 | 24862 | 2000 | +| 96 | 0.9569 | 3273 (3103-3312) | 0.296 | 0.377 | 0.553 | 18591 | 2000 | +| 128 | 0.9714 | 2370 (2277-2432) | 0.386 | 0.519 | 0.929 | 14274 | 2000 | +| 192 | 0.9842 | 1701 (1642-1724) | 0.558 | 0.691 | 1.04 | 10206 | 2000 | +| 256 | 0.9909 | 1282 (1249-1318) | 0.742 | 0.967 | 1.59 | 7844 | 2000 | +| 400 | 0.9958 | 823 (781-839) | 1.17 | 1.50 | 2.42 | 5285 | 2000 | +| 640 | 0.9983 | 508 (492-515) | 1.89 | 2.44 | 4.01 | 3847 | 1585 | + +**faiss IVFFlat** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2834 | 18763 (18006-18898) | 0.052 | 0.065 | 0.082 | 192,036 | 2000 | +| 2 | 0.4099 | 16033 (16022-16181) | 0.060 | 0.076 | 0.095 | 172,377 | 2000 | +| 4 | 0.5287 | 12664 (11616-12949) | 0.077 | 0.101 | 0.141 | 116,179 | 2000 | +| 8 | 0.6503 | 9036 (8998-9414) | 0.107 | 0.134 | 0.171 | 84704 | 2000 | +| 16 | 0.7551 | 6072 (5829-6281) | 0.157 | 0.199 | 0.249 | 38405 | 2000 | +| 32 | 0.8439 | 3761 (3701-3838) | 0.255 | 0.324 | 0.385 | 23327 | 2000 | +| 64 | 0.9096 | 2078 (2072-2081) | 0.458 | 0.571 | 0.689 | 11708 | 2000 | +| 128 | 0.9549 | 1116 (1103-1154) | 0.866 | 1.07 | 1.26 | 5728 | 2000 | +| 256 | 0.9841 | 567 (566-570) | 1.72 | 2.00 | 2.43 | 2721 | 1702 | +| 1265 | 0.9999 | 120 (118-121) | 8.10 | 9.46 | 12.5 | 505 | 377 | + +**hnswlib** (ef) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.6185 | 16855 (16407-17226) | 0.055 | 0.082 | 0.104 | 93453 | 2000 | +| 16 | 0.7160 | 13172 (13035-13312) | 0.072 | 0.105 | 0.134 | 75207 | 2000 | +| 24 | 0.7870 | 9860 (9650-9987) | 0.096 | 0.132 | 0.176 | 52454 | 2000 | +| 32 | 0.8309 | 7740 (7450-8230) | 0.116 | 0.161 | 0.244 | 46850 | 2000 | +| 48 | 0.8855 | 6085 (5802-6165) | 0.157 | 0.206 | 0.266 | 33393 | 2000 | +| 64 | 0.9162 | 4726 (4337-4839) | 0.202 | 0.273 | 0.377 | 26177 | 2000 | +| 96 | 0.9488 | 3563 (3414-3576) | 0.280 | 0.344 | 0.428 | 20267 | 2000 | +| 128 | 0.9645 | 2715 (2704-2728) | 0.354 | 0.440 | 0.587 | 15705 | 2000 | +| 192 | 0.9809 | 1959 (1930-1962) | 0.504 | 0.615 | 0.777 | 10961 | 2000 | +| 256 | 0.9869 | 1558 (1553-1565) | 0.634 | 0.759 | 0.862 | 8428 | 2000 | +| 400 | 0.9933 | 1026 (1017-1037) | 0.949 | 1.17 | 1.46 | 6018 | 2000 | +| 640 | 0.9970 | 696 (690-697) | 1.41 | 1.74 | 2.27 | 3946 | 1874 | + +**lancedb (no index)** (nprobes) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 7.19 (7.16-7.22) | 137 | 151 | 173 | - | 30 | + +**lancedb IVF_FLAT** (nprobes) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2415 | 243 (198-288) | 3.28 | 4.56 | 12.3 | - | 924 | +| 2 | 0.3584 | 300 (298-301) | 3.24 | 4.06 | 5.79 | - | 912 | +| 4 | 0.4898 | 285 (282-300) | 3.30 | 4.06 | 6.32 | - | 933 | +| 8 | 0.6160 | 294 (286-299) | 3.33 | 3.94 | 5.10 | - | 843 | +| 16 | 0.7339 | 286 (278-288) | 3.41 | 4.03 | 5.67 | - | 903 | +| 32 | 0.8351 | 266 (265-278) | 3.59 | 4.31 | 5.36 | - | 786 | +| 64 | 0.9090 | 256 (251-257) | 3.86 | 4.43 | 5.06 | - | 790 | +| 128 | 0.9556 | 218 (212-220) | 4.52 | 5.16 | 6.06 | - | 652 | +| 256 | 0.9828 | 177 (168-178) | 5.61 | 6.48 | 7.70 | - | 500 | +| 1265 | 0.9999 | 70.3 (69.0-71.1) | 14.0 | 15.6 | 17.4 | - | 210 | + +**numpy brute force** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 0.9999 | 144 (143-146) | 6.75 | 7.66 | 10.1 | 1553 | 448 | + +**qdrant-client local mode (exact numpy)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 7.24 (7.14-7.25) | 136 | 152 | 158 | - | 30 | + +**sqlite-vec (vec0, exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 10.2 (10.2-10.3) | 95.5 | 115 | 122 | - | 30 | + +**usearch (HNSW)** (expansion_search) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.6119 | 11865 (10700-12258) | 0.080 | 0.127 | 0.172 | 97939 | 2000 | +| 16 | 0.7104 | 8530 (7906-8812) | 0.109 | 0.166 | 0.218 | 79790 | 2000 | +| 24 | 0.7848 | 6731 (6269-6756) | 0.140 | 0.198 | 0.268 | 58748 | 2000 | +| 32 | 0.8294 | 5372 (5330-5488) | 0.172 | 0.229 | 0.280 | 48489 | 2000 | +| 48 | 0.8844 | 4092 (4055-4107) | 0.235 | 0.302 | 0.368 | 33537 | 2000 | +| 64 | 0.9149 | 3228 (3164-3248) | 0.299 | 0.378 | 0.453 | 26211 | 2000 | +| 96 | 0.9468 | 2334 (2290-2353) | 0.418 | 0.520 | 0.618 | 20803 | 2000 | +| 128 | 0.9643 | 1820 (1775-1833) | 0.540 | 0.681 | 0.848 | 14806 | 2000 | +| 192 | 0.9790 | 1297 (1266-1307) | 0.765 | 0.927 | 1.06 | 11370 | 2000 | +| 256 | 0.9854 | 1020 (1001-1029) | 0.974 | 1.19 | 1.38 | 8865 | 2000 | +| 400 | 0.9913 | 679 (656-692) | 1.46 | 1.81 | 2.26 | 5836 | 1759 | +| 640 | 0.9951 | 451 (443-460) | 2.20 | 2.72 | 3.28 | 3704 | 1355 | + +
+ +### glove, N = 100,000 + +Queries/second at recall@10 = 0.90 / 0.95 / 0.99, log-linearly interpolated between the two measured sweep points that bracket the target (no extrapolation; `*` = the two bracketing points are more than 0.05 recall apart, so the measured point above the target is shown instead, a conservative value; `-` = never reached; exact scans appear at their single recall-1.0 point). 1T = one pinned P-core, one query at a time. All-core = batched call on 32 logical CPUs (`†` = the ISONGraph ports have no batch API, so this is the same one-query loop with automatic scan threads). Latency columns are for the first measured sweep point with recall >= 0.95. + +| system | QPS 1T @0.90 | @0.95 | @0.99 | all-core QPS @0.90 | @0.95 | 1T latency ms median / p95 / p99 | +|---|---|---|---|---|---|---| +| **ISONGraph C++, /arch:AVX2 (exact)** | 40.2 | 40.2 | 40.2 | - | - | 24.4 / 28.5 / 31.7 (recall 1.000) | +| **ISONGraph C++, /arch:AVX2 (IVF)** | 2655* | 2167 | 162 | - | - | 0.708 / 0.882 / 1.03 (recall 0.974) | +| **ISONGraph C++ (exact)** | 40.1 | 40.1 | 40.1 | - | - | 24.5 / 28.4 / 29.7 (recall 1.000) | +| **ISONGraph C++ (IVF)** | 2361* | 1919 | 150 | - | - | 0.781 / 1.00 / 1.20 (recall 0.974) | +| **ISONGraph C# (exact)** | 108 | 108 | 108 | - | - | 7.79 / 26.4 / 35.6 (recall 1.000) | +| **ISONGraph C# (IVF)** | 1677* | 1268 | 220 | - | - | 1.14 / 1.43 / 1.62 (recall 0.973) | +| **ISONGraph Python (exact)** | 471 | 471 | 471 | - | - | 1.98 / 2.72 / 4.89 (recall 1.000) | +| **ISONGraph Python (IVF)** | 1035* | 813 | 477 | - | - | 1.73 / 2.31 / 3.53 (recall 0.973) | +| **ISONGraph JS (exact)** | 48.1 | 48.1 | 48.1 | - | - | 20.1 / 24.6 / 27.1 (recall 1.000) | +| **ISONGraph JS (IVF)** | 432* | 331 | 91.8 | - | - | 4.37 / 5.46 / 6.29 (recall 0.975) | +| **ISONGraph Rust (exact)** | 700 | 700 | 700 | - | - | 1.42 / 1.79 / 2.14 (recall 1.000) | +| **ISONGraph Rust (IVF)** | 2595* | 2029 | 888 | 2638* † | 2381 † | 0.733 / 0.912 / 1.01 (recall 0.974) | +| **ISONGraph Rust, target-cpu=native (exact)** | 621 | 621 | 621 | - | - | 1.57 / 1.94 / 2.25 (recall 1.000) | +| **ISONGraph Rust, target-cpu=native (IVF)** | 2584* | 2028 | 851 | 2610* † | 2334 † | 0.725 / 0.909 / 1.02 (recall 0.974) | +| chromadb (HNSW) | 956* | 933 | - | 3684* | 3468 | 1.29 / 1.50 / 1.92 (recall 0.977) | +| faiss IndexFlatIP | 548 | 548 | 548 | 16430 | 16430 | 1.72 / 2.23 / 3.01 (recall 1.000) | +| faiss HNSWFlat | 4756 | 2152 | - | 94888 | 51124 | 0.679 / 0.908 / 1.71 (recall 0.970) | +| faiss IVFFlat | 3719* | 2893 | 785 | 53383* | 40104 | 0.457 / 0.603 / 0.990 (recall 0.974) | +| hnswlib | 7183 | 3762 | - | 70103 | 48987 | 0.366 / 0.452 / 0.573 (recall 0.969) | +| lancedb (no index) | 34.3 | 34.3 | 34.3 | - | - | 29.5 / 34.2 / 37.4 (recall 1.000) | +| lancedb IVF_FLAT | 253* | 238 | 145 | - | - | 4.45 / 5.51 / 7.19 (recall 0.974) | +| numpy brute force | 528 | 528 | 528 | 1733 | 1733 | 1.82 / 2.53 / 3.70 (recall 1.000) | +| sqlite-vec (vec0, exact) | 40.1 | 40.1 | 40.1 | - | - | 24.3 / 27.4 / 34.3 (recall 1.000) | +| usearch (HNSW) | 3763 | 1947 | - | 58683 | 27876 | 0.720 / 0.949 / 1.18 (recall 0.967) | + +Build and memory (RSS growth over the process baseline after imports; the raw vectors are streamed in from disk and are not held by the harness). + +| system | ingest s | index s | total build s | 3 builds: median (min-max) s | ready s (incl. first query) | steady RSS +MB | peak RSS +MB (build) | +|---|---|---|---|---|---|---|---| +| **ISONGraph C++, /arch:AVX2 (exact)** | 0.042 | 0.000 | 0.042 | - | 0.069 | 72 | 84 | +| **ISONGraph C++, /arch:AVX2 (IVF)** | 0.042 | 0.253 | 0.295 | - | 0.295 | 111 | 131 | +| **ISONGraph C++ (exact)** | 0.038 | 0.000 | 0.038 | - | 0.064 | 72 | 84 | +| **ISONGraph C++ (IVF)** | 0.038 | 0.290 | 0.328 | - | 0.328 | 112 | 132 | +| **ISONGraph C# (exact)** | 0.017 | 0.000 | 0.017 | - | 0.035 | 87 | 87 | +| **ISONGraph C# (IVF)** | 0.011 | 0.602 | 0.613 | - | 0.613 | 249 | 249 | +| **ISONGraph Python (exact)** | 0.934 | 0.000 | 0.934 | 0.931 (0.914-0.934) | 1.11 | 159 | 164 | +| **ISONGraph Python (IVF)** | 0.929 | 0.848 | 1.78 | 1.78 (1.73-1.78) | 1.78 | 176 | 278 | +| **ISONGraph JS (exact)** | 0.029 | 0.000 | 0.029 | - | 0.053 | 308 | 308 | +| **ISONGraph JS (IVF)** | 0.027 | 20.6 | 20.7 | - | 20.7 | 642 | 645 | +| **ISONGraph Rust (exact)** | 0.054 | 0.000 | 0.054 | - | 0.094 | 121 | 121 | +| **ISONGraph Rust (IVF)** | 0.054 | 0.212 | 0.307 | - | 0.307 | 125 | 144 | +| **ISONGraph Rust, target-cpu=native (exact)** | 0.053 | 0.000 | 0.053 | - | 0.095 | 121 | 121 | +| **ISONGraph Rust, target-cpu=native (IVF)** | 0.053 | 0.176 | 0.270 | - | 0.270 | 125 | 144 | +| chromadb (HNSW) | 9.46 | 0.000 | 9.46 | 9.40 (9.34-9.46) | 9.55 | 264 | 202 | +| faiss IndexFlatIP | 0.031 | 0.000 | 0.031 | 0.035 (0.031-0.047) | 0.033 | 63 | 101 | +| faiss HNSWFlat | 2.25 | 0.000 | 2.25 | 2.28 (2.25-2.41) | 2.26 | 76 | 112 | +| faiss IVFFlat | 0.000 | 0.973 | 0.973 | 1.08 (0.973-1.09) | 0.976 | 91 | 155 | +| hnswlib | 1.33 | 0.000 | 1.33 | 1.31 (1.28-1.33) | 1.34 | 88 | 96 | +| lancedb (no index) | 0.083 | 0.000 | 0.083 | 0.080 (0.077-0.083) | 0.105 | 100 | 122 | +| lancedb IVF_FLAT | 0.085 | 0.643 | 0.729 | 0.697 (0.693-0.729) | 0.797 | 173 | 201 | +| numpy brute force | 0.012 | 0.000 | 0.012 | 0.012 (0.012-0.012) | 0.014 | 51 | 61 | +| sqlite-vec (vec0, exact) | 0.529 | 0.000 | 0.529 | 0.521 (0.493-0.529) | 0.554 | 4 | 19 | +| usearch (HNSW) | 1.72 | 0.000 | 1.72 | 1.72 (1.62-1.72) | 1.73 | 75 | 84 | + +
Full accuracy sweeps (recall@10, single-thread QPS, latency, all-core QPS) + +**ISONGraph C++, /arch:AVX2 (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 40.2 (39.7-41.2) | 24.4 | 28.5 | 31.7 | - | 126 | + +**ISONGraph C++, /arch:AVX2 (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2848 | 35648 (33711-35885) | 0.028 | 0.034 | 0.039 | - | 2000 | +| 2 | 0.4074 | 30432 (29216-30692) | 0.033 | 0.041 | 0.046 | - | 2000 | +| 4 | 0.5255 | 25325 (24998-25645) | 0.039 | 0.049 | 0.055 | - | 2000 | +| 8 | 0.6348 | 19291 (19117-19436) | 0.051 | 0.066 | 0.075 | - | 2000 | +| 16 | 0.7357 | 13080 (13064-13094) | 0.075 | 0.098 | 0.120 | - | 2000 | +| 32 | 0.8185 | 8058 (7722-8187) | 0.123 | 0.161 | 0.185 | - | 2000 | +| 64 | 0.8845 | 4787 (4719-4842) | 0.205 | 0.265 | 0.309 | - | 2000 | +| 128 | 0.9386 | 2655 (2628-2693) | 0.371 | 0.464 | 0.524 | - | 2000 | +| 256 | 0.9741 | 1408 (1345-1414) | 0.708 | 0.882 | 1.03 | - | 2000 | +| 1265 | 1.0000 | 41.6 (41.5-41.7) | 23.5 | 26.8 | 27.9 | - | 125 | + +**ISONGraph C++ (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 40.1 (39.7-40.1) | 24.5 | 28.4 | 29.7 | - | 123 | + +**ISONGraph C++ (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2848 | 31630 (24601-32671) | 0.032 | 0.047 | 0.058 | - | 2000 | +| 2 | 0.4074 | 27775 (27261-27919) | 0.035 | 0.044 | 0.060 | - | 2000 | +| 4 | 0.5255 | 22908 (22737-23047) | 0.042 | 0.055 | 0.074 | - | 2000 | +| 8 | 0.6348 | 16744 (16490-17301) | 0.057 | 0.077 | 0.099 | - | 2000 | +| 16 | 0.7357 | 11815 (11676-12044) | 0.082 | 0.110 | 0.136 | - | 2000 | +| 32 | 0.8185 | 7116 (6877-7441) | 0.136 | 0.187 | 0.224 | - | 2000 | +| 64 | 0.8845 | 4281 (4218-4369) | 0.227 | 0.304 | 0.353 | - | 2000 | +| 128 | 0.9386 | 2361 (2314-2397) | 0.411 | 0.547 | 0.648 | - | 2000 | +| 256 | 0.9741 | 1235 (1234-1247) | 0.781 | 1.00 | 1.20 | - | 2000 | +| 1265 | 1.0000 | 39.7 (39.7-40.3) | 24.6 | 27.8 | 29.0 | - | 116 | + +**ISONGraph C# (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 108 (107-108) | 7.79 | 26.4 | 35.6 | - | 387 | + +**ISONGraph C# (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2868 | 14066 (6337-14394) | 0.069 | 0.212 | 0.232 | - | 2000 | +| 2 | 0.4092 | 13476 (13083-13665) | 0.073 | 0.084 | 0.091 | - | 2000 | +| 4 | 0.5273 | 12150 (11932-12195) | 0.081 | 0.096 | 0.106 | - | 2000 | +| 8 | 0.6336 | 9682 (9672-9773) | 0.100 | 0.123 | 0.139 | - | 2000 | +| 16 | 0.7249 | 7268 (7001-7362) | 0.136 | 0.175 | 0.195 | - | 2000 | +| 32 | 0.8096 | 4612 (4436-4695) | 0.215 | 0.276 | 0.309 | - | 2000 | +| 64 | 0.8780 | 2983 (2918-2984) | 0.334 | 0.423 | 0.482 | - | 2000 | +| 128 | 0.9323 | 1677 (1632-1700) | 0.594 | 0.748 | 0.829 | - | 2000 | +| 256 | 0.9727 | 886 (826-886) | 1.14 | 1.43 | 1.62 | - | 2000 | +| 1265 | 1.0000 | 98.7 (95.5-101) | 7.90 | 38.7 | 49.3 | - | 329 | + +**ISONGraph Python (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 471 (468-487) | 1.98 | 2.72 | 4.89 | - | 1462 | + +**ISONGraph Python (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2853 | 14708 (14670-16768) | 0.059 | 0.083 | 0.117 | - | 2000 | +| 2 | 0.4022 | 12857 (12777-14101) | 0.069 | 0.092 | 0.130 | - | 2000 | +| 4 | 0.5192 | 9844 (9812-10234) | 0.094 | 0.127 | 0.166 | - | 2000 | +| 8 | 0.6309 | 6339 (6092-6998) | 0.133 | 0.184 | 0.248 | - | 2000 | +| 16 | 0.7282 | 4918 (4775-4947) | 0.198 | 0.261 | 0.303 | - | 2000 | +| 32 | 0.8134 | 3161 (3152-3178) | 0.313 | 0.396 | 0.454 | - | 2000 | +| 64 | 0.8796 | 1876 (1867-1876) | 0.521 | 0.658 | 0.771 | - | 2000 | +| 128 | 0.9362 | 1035 (1011-1071) | 0.930 | 1.18 | 1.70 | - | 2000 | +| 256 | 0.9728 | 547 (535-558) | 1.73 | 2.31 | 3.53 | - | 1803 | +| 1265 | 1.0000 | 440 (435-446) | 2.09 | 3.11 | 5.46 | - | 1414 | + +**ISONGraph JS (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 48.1 (47.8-48.9) | 20.1 | 24.6 | 27.1 | - | 167 | + +**ISONGraph JS (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2795 | 6775 (6501-6778) | 0.145 | 0.184 | 0.233 | - | 2000 | +| 2 | 0.3983 | 5710 (5550-5862) | 0.171 | 0.224 | 0.250 | - | 2000 | +| 4 | 0.5165 | 4523 (4349-4604) | 0.218 | 0.290 | 0.328 | - | 2000 | +| 8 | 0.6258 | 3533 (3512-3553) | 0.279 | 0.371 | 0.415 | - | 2000 | +| 16 | 0.7241 | 2301 (2293-2328) | 0.426 | 0.580 | 0.676 | - | 2000 | +| 32 | 0.8079 | 1401 (1350-1412) | 0.709 | 0.968 | 1.13 | - | 2000 | +| 64 | 0.8763 | 779 (764-799) | 1.27 | 1.69 | 1.90 | - | 2000 | +| 128 | 0.9321 | 432 (431-447) | 2.28 | 2.96 | 3.26 | - | 1366 | +| 256 | 0.9745 | 230 (221-232) | 4.37 | 5.46 | 6.29 | - | 651 | +| 1265 | 1.0000 | 50.8 (49.5-51.3) | 19.1 | 23.1 | 24.9 | - | 154 | + +**ISONGraph Rust (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 700 (660-705) | 1.42 | 1.79 | 2.14 | - | 2000 | + +**ISONGraph Rust (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2851 | 52956 (49100-55464) | 0.018 | 0.025 | 0.037 | 53680 † | 2000 | +| 2 | 0.4016 | 40791 (40768-41483) | 0.024 | 0.031 | 0.038 | 42250 † | 2000 | +| 4 | 0.5209 | 32168 (32032-32233) | 0.030 | 0.041 | 0.050 | 32288 † | 2000 | +| 8 | 0.6351 | 21635 (21002-22195) | 0.045 | 0.062 | 0.076 | 21857 † | 2000 | +| 16 | 0.7308 | 13906 (13495-14167) | 0.070 | 0.097 | 0.120 | 14291 † | 2000 | +| 32 | 0.8134 | 8542 (8508-8573) | 0.115 | 0.154 | 0.180 | 8303 † | 2000 | +| 64 | 0.8817 | 4864 (4744-4875) | 0.205 | 0.267 | 0.301 | 4573 † | 2000 | +| 128 | 0.9355 | 2595 (2484-2619) | 0.384 | 0.507 | 0.570 | 2638 † | 2000 | +| 256 | 0.9740 | 1350 (1347-1359) | 0.733 | 0.912 | 1.01 | 2010 † | 2000 | +| 1265 | 1.0000 | 684 (673-693) | 1.43 | 1.72 | 2.13 | 1703 † | 1978 | + +**ISONGraph Rust, target-cpu=native (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 621 (616-627) | 1.57 | 1.94 | 2.25 | - | 1749 | + +**ISONGraph Rust, target-cpu=native (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2851 | 52724 (50519-53187) | 0.019 | 0.025 | 0.029 | 52173 † | 2000 | +| 2 | 0.4016 | 40953 (40918-43267) | 0.024 | 0.030 | 0.033 | 42748 † | 2000 | +| 4 | 0.5209 | 31286 (27148-32777) | 0.032 | 0.045 | 0.055 | 32611 † | 2000 | +| 8 | 0.6351 | 21679 (21020-22277) | 0.045 | 0.059 | 0.068 | 20838 † | 2000 | +| 16 | 0.7308 | 13851 (13535-13940) | 0.072 | 0.095 | 0.108 | 14078 † | 2000 | +| 32 | 0.8134 | 8218 (8199-8559) | 0.118 | 0.154 | 0.171 | 7957 † | 2000 | +| 64 | 0.8817 | 4859 (4614-4861) | 0.207 | 0.267 | 0.297 | 4645 † | 2000 | +| 128 | 0.9355 | 2584 (2419-2716) | 0.382 | 0.512 | 0.589 | 2610 † | 2000 | +| 256 | 0.9740 | 1358 (1352-1383) | 0.725 | 0.909 | 1.02 | 1940 † | 2000 | +| 1265 | 1.0000 | 636 (627-640) | 1.55 | 1.84 | 2.04 | 1625 † | 2000 | + +**chromadb (HNSW)** (ef_search) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.5396 | 1412 (1390-1486) | 0.717 | 0.801 | 0.950 | 23630 | 2000 | +| 16 | 0.6245 | 1404 (1357-1450) | 0.714 | 0.815 | 0.942 | 20762 | 2000 | +| 32 | 0.7422 | 1367 (1291-1381) | 0.753 | 0.860 | 0.953 | 15763 | 2000 | +| 64 | 0.8288 | 1153 (1132-1160) | 0.861 | 0.966 | 1.12 | 9913 | 2000 | +| 128 | 0.8957 | 1042 (1034-1051) | 0.940 | 1.09 | 1.35 | 6356 | 2000 | +| 256 | 0.9470 | 956 (933-981) | 1.03 | 1.21 | 1.43 | 3684 | 2000 | +| 512 | 0.9767 | 753 (749-776) | 1.29 | 1.50 | 1.92 | 2029 | 2000 | + +**faiss IndexFlatIP** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 548 (543-570) | 1.72 | 2.23 | 3.01 | 16430 | 1809 | + +**faiss HNSWFlat** (efSearch) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.5377 | 32429 (29418-34917) | 0.028 | 0.043 | 0.063 | 823,215 | 2000 | +| 16 | 0.6264 | 25691 (25351-27102) | 0.036 | 0.053 | 0.068 | 502,740 | 2000 | +| 24 | 0.6956 | 20186 (19218-20931) | 0.047 | 0.068 | 0.088 | 499,650 | 2000 | +| 32 | 0.7423 | 17280 (15500-17482) | 0.057 | 0.079 | 0.108 | 381,767 | 2000 | +| 48 | 0.7966 | 11435 (11370-12845) | 0.078 | 0.108 | 0.177 | 257,689 | 2000 | +| 64 | 0.8334 | 9621 (8971-9751) | 0.100 | 0.137 | 0.240 | 211,013 | 2000 | +| 96 | 0.8744 | 6390 (4882-6712) | 0.149 | 0.208 | 0.355 | 120,766 | 2000 | +| 128 | 0.9023 | 4629 (4219-4853) | 0.200 | 0.274 | 0.430 | 92811 | 2000 | +| 192 | 0.9305 | 3206 (3076-3243) | 0.296 | 0.401 | 0.833 | 66248 | 2000 | +| 256 | 0.9483 | 2237 (2230-2250) | 0.408 | 0.539 | 1.09 | 54004 | 2000 | +| 400 | 0.9696 | 1381 (1379-1435) | 0.679 | 0.908 | 1.71 | 27177 | 2000 | +| 640 | 0.9842 | 785 (780-789) | 1.23 | 1.58 | 2.59 | 17016 | 2000 | + +**faiss IVFFlat** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2827 | 51821 (50764-51839) | 0.019 | 0.022 | 0.029 | 310,318 | 2000 | +| 2 | 0.4049 | 44673 (41408-45531) | 0.021 | 0.026 | 0.042 | 382,168 | 2000 | +| 4 | 0.5251 | 35842 (34299-39054) | 0.026 | 0.034 | 0.042 | 275,217 | 2000 | +| 8 | 0.6316 | 27988 (27219-28595) | 0.034 | 0.043 | 0.056 | 246,831 | 2000 | +| 16 | 0.7287 | 19426 (11660-19775) | 0.049 | 0.066 | 0.157 | 134,086 | 2000 | +| 32 | 0.8119 | 10764 (10512-12410) | 0.083 | 0.118 | 0.165 | 153,105 | 2000 | +| 64 | 0.8796 | 6348 (5918-6871) | 0.149 | 0.203 | 0.298 | 93354 | 2000 | +| 128 | 0.9332 | 3719 (3637-3728) | 0.256 | 0.334 | 0.498 | 53383 | 2000 | +| 256 | 0.9738 | 2031 (1979-2051) | 0.457 | 0.603 | 0.990 | 26799 | 2000 | +| 1265 | 1.0000 | 438 (428-438) | 2.18 | 2.75 | 4.53 | 6400 | 1387 | + +**hnswlib** (ef) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.5421 | 44724 (41330-46904) | 0.022 | 0.031 | 0.040 | 373,518 | 2000 | +| 16 | 0.6244 | 36871 (34289-36877) | 0.027 | 0.038 | 0.044 | 328,116 | 2000 | +| 24 | 0.6949 | 27706 (27313-29071) | 0.035 | 0.048 | 0.058 | 212,972 | 2000 | +| 32 | 0.7392 | 23477 (23317-23597) | 0.042 | 0.055 | 0.065 | 171,278 | 2000 | +| 48 | 0.7937 | 17284 (15501-17469) | 0.059 | 0.078 | 0.090 | 132,860 | 2000 | +| 64 | 0.8287 | 12918 (12807-14084) | 0.073 | 0.095 | 0.112 | 150,099 | 2000 | +| 96 | 0.8706 | 9092 (7599-9613) | 0.103 | 0.134 | 0.216 | 103,974 | 2000 | +| 128 | 0.8950 | 7641 (7197-7770) | 0.131 | 0.164 | 0.195 | 71297 | 2000 | +| 192 | 0.9278 | 5112 (5038-5168) | 0.190 | 0.238 | 0.320 | 63883 | 2000 | +| 256 | 0.9466 | 3983 (3913-4050) | 0.243 | 0.310 | 0.412 | 51910 | 2000 | +| 400 | 0.9692 | 2709 (2670-2731) | 0.366 | 0.452 | 0.573 | 35142 | 2000 | +| 640 | 0.9843 | 1769 (1750-1809) | 0.559 | 0.693 | 0.853 | 23069 | 2000 | + +**lancedb (no index)** (nprobes) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 34.3 (31.2-34.5) | 29.5 | 34.2 | 37.4 | - | 86 | + +**lancedb IVF_FLAT** (nprobes) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2556 | 308 (307-310) | 3.20 | 3.67 | 4.23 | - | 950 | +| 2 | 0.3718 | 309 (304-314) | 3.18 | 3.74 | 4.39 | - | 922 | +| 4 | 0.4942 | 309 (307-309) | 3.20 | 3.71 | 4.47 | - | 987 | +| 8 | 0.6105 | 302 (297-304) | 3.25 | 3.85 | 4.72 | - | 959 | +| 16 | 0.7110 | 305 (304-310) | 3.24 | 3.66 | 4.37 | - | 922 | +| 32 | 0.8030 | 298 (297-300) | 3.33 | 3.74 | 4.41 | - | 887 | +| 64 | 0.8755 | 283 (278-284) | 3.50 | 3.90 | 4.58 | - | 880 | +| 128 | 0.9348 | 253 (247-256) | 3.84 | 4.68 | 5.99 | - | 786 | +| 256 | 0.9740 | 217 (207-219) | 4.45 | 5.51 | 7.19 | - | 670 | +| 1265 | 1.0000 | 113 (112-114) | 8.54 | 10.2 | 16.3 | - | 352 | + +**numpy brute force** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 528 (479-537) | 1.82 | 2.53 | 3.70 | 1733 | 1607 | + +**sqlite-vec (vec0, exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 40.1 (40.0-40.4) | 24.3 | 27.4 | 34.3 | - | 125 | + +**usearch (HNSW)** (expansion_search) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.5376 | 27475 (26481-27781) | 0.035 | 0.050 | 0.059 | 293,591 | 2000 | +| 16 | 0.6238 | 21039 (19802-21251) | 0.046 | 0.067 | 0.095 | 282,602 | 2000 | +| 24 | 0.6930 | 15794 (15665-16627) | 0.060 | 0.086 | 0.108 | 227,663 | 2000 | +| 32 | 0.7346 | 12833 (11838-13645) | 0.073 | 0.102 | 0.130 | 190,170 | 2000 | +| 48 | 0.7900 | 9549 (9154-10111) | 0.101 | 0.141 | 0.171 | 121,533 | 2000 | +| 64 | 0.8295 | 7238 (7210-7919) | 0.130 | 0.181 | 0.227 | 108,436 | 2000 | +| 96 | 0.8690 | 5186 (5094-5409) | 0.181 | 0.241 | 0.329 | 80683 | 2000 | +| 128 | 0.8950 | 3966 (3948-4152) | 0.240 | 0.316 | 0.449 | 60907 | 2000 | +| 192 | 0.9264 | 2848 (2796-2901) | 0.345 | 0.447 | 0.549 | 48221 | 2000 | +| 256 | 0.9456 | 2135 (1926-2190) | 0.471 | 0.637 | 0.736 | 29190 | 2000 | +| 400 | 0.9667 | 1375 (1372-1376) | 0.720 | 0.949 | 1.18 | 23417 | 2000 | +| 640 | 0.9819 | 876 (853-896) | 1.14 | 1.47 | 1.80 | 15709 | 2000 | + +
+ +### sift, N = 100,000 + +Queries/second at recall@10 = 0.90 / 0.95 / 0.99, log-linearly interpolated between the two measured sweep points that bracket the target (no extrapolation; `*` = the two bracketing points are more than 0.05 recall apart, so the measured point above the target is shown instead, a conservative value; `-` = never reached; exact scans appear at their single recall-1.0 point). 1T = one pinned P-core, one query at a time. All-core = batched call on 32 logical CPUs (`†` = the ISONGraph ports have no batch API, so this is the same one-query loop with automatic scan threads). Latency columns are for the first measured sweep point with recall >= 0.95. + +| system | QPS 1T @0.90 | @0.95 | @0.99 | all-core QPS @0.90 | @0.95 | 1T latency ms median / p95 / p99 | +|---|---|---|---|---|---|---| +| **ISONGraph C++, /arch:AVX2 (exact)** | 23.4 | 23.4 | 23.4 | - | - | 42.3 / 45.4 / 47.4 (recall 1.000) | +| **ISONGraph C++, /arch:AVX2 (IVF)** | 4693* | 4693* | 2451 | - | - | 0.211 / 0.293 / 0.336 (recall 0.952) | +| **ISONGraph C++ (exact)** | 22.8 | 22.8 | 22.8 | - | - | 44.2 / 47.5 / 49.5 (recall 1.000) | +| **ISONGraph C++ (IVF)** | 4054* | 4054* | 2083 | - | - | 0.241 / 0.336 / 0.393 (recall 0.952) | +| **ISONGraph C# (exact)** | 55.0 | 55.0 | 55.0 | - | - | 14.6 / 42.8 / 80.8 (recall 1.000) | +| **ISONGraph C# (IVF)** | 2805* | 2805* | 1635 | - | - | 0.362 / 0.482 / 0.555 (recall 0.953) | +| **ISONGraph Python (exact)** | 318 | 318 | 318 | - | - | 3.22 / 4.42 / 5.68 (recall 1.000) | +| **ISONGraph Python (IVF)** | 2722* | 2622 | 1395 | - | - | 0.570 / 0.778 / 1.08 (recall 0.987) | +| **ISONGraph JS (exact)** | 29.0 | 29.0 | 29.0 | - | - | 33.7 / 41.6 / 48.7 (recall 1.000) | +| **ISONGraph JS (IVF)** | 810* | 795 | 416 | - | - | 2.10 / 2.73 / 3.12 (recall 0.988) | +| **ISONGraph Rust (exact)** | 344 | 344 | 344 | - | - | 2.86 / 3.36 / 3.84 (recall 1.000) | +| **ISONGraph Rust (IVF)** | 4980* | 4873 | 2247 | 2648* † | 2592 † | 0.365 / 0.508 / 0.626 (recall 0.986) | +| **ISONGraph Rust, target-cpu=native (exact)** | 364 | 364 | 364 | - | - | 2.62 / 3.14 / 3.42 (recall 1.000) | +| **ISONGraph Rust, target-cpu=native (IVF)** | 4741* | 4650 | 2335 | 2523* † | 2470 † | 0.357 / 0.476 / 0.545 (recall 0.986) | +| chromadb (HNSW) | 1163* | 1163* | 1107 | 12847* | 12847* | 0.818 / 1.11 / 1.83 (recall 0.952) | +| faiss IndexFlatIP | 283 | 283 | 283 | 4537 | 4537 | 3.30 / 4.26 / 6.04 (recall 1.000) | +| faiss HNSWFlat | 21197* | 17392 | 8549 | 354,906* | 331,856 | 0.056 / 0.078 / 0.121 (recall 0.953) | +| faiss IVFFlat | 8764* | 8764* | 4401 | 112,537* | 112,537* | 0.105 / 0.160 / 0.248 (recall 0.950) | +| hnswlib | 26532* | 22439 | 11479 | 281,005* | 193,015 | 0.044 / 0.061 / 0.079 (recall 0.950) | +| lancedb (no index) | 15.1 | 15.1 | 15.1 | - | - | 66.9 / 78.5 / 82.0 (recall 1.000) | +| lancedb IVF_FLAT | 247* | 247 | 231 | - | - | 4.06 / 5.08 / 6.78 (recall 0.987) | +| numpy brute force | 282 | 282 | 282 | 1543 | 1543 | 3.34 / 4.87 / 6.69 (recall 1.000) | +| sqlite-vec (vec0, exact) | 21.6 | 21.6 | 21.6 | - | - | 47.0 / 52.6 / 57.7 (recall 1.000) | +| usearch (HNSW) | 16246* | 13215 | 5873 | 160,195* | 106,020 | 0.072 / 0.101 / 0.133 (recall 0.952) | + +Build and memory (RSS growth over the process baseline after imports; the raw vectors are streamed in from disk and are not held by the harness). + +| system | ingest s | index s | total build s | 3 builds: median (min-max) s | ready s (incl. first query) | steady RSS +MB | peak RSS +MB (build) | +|---|---|---|---|---|---|---|---| +| **ISONGraph C++, /arch:AVX2 (exact)** | 0.049 | 0.000 | 0.049 | - | 0.079 | 84 | 102 | +| **ISONGraph C++, /arch:AVX2 (IVF)** | 0.049 | 0.638 | 0.687 | - | 0.687 | 135 | 160 | +| **ISONGraph C++ (exact)** | 0.050 | 0.000 | 0.050 | - | 0.081 | 84 | 102 | +| **ISONGraph C++ (IVF)** | 0.050 | 0.970 | 1.02 | - | 1.02 | 134 | 160 | +| **ISONGraph C# (exact)** | 0.018 | 0.000 | 0.018 | - | 0.041 | 108 | 108 | +| **ISONGraph C# (IVF)** | 0.028 | 1.48 | 1.51 | - | 1.51 | 311 | 311 | +| **ISONGraph Python (exact)** | 1.26 | 0.000 | 1.26 | 4.28 (1.26-4.49) | 1.48 | 170 | 195 | +| **ISONGraph Python (IVF)** | 1.11 | 0.965 | 2.08 | 2.08 (2.02-2.08) | 2.08 | 192 | 303 | +| **ISONGraph JS (exact)** | 0.035 | 0.000 | 0.035 | - | 0.063 | 317 | 317 | +| **ISONGraph JS (IVF)** | 0.044 | 35.4 | 35.5 | - | 35.5 | 651 | 651 | +| **ISONGraph Rust (exact)** | 0.055 | 0.000 | 0.055 | - | 0.106 | 143 | 143 | +| **ISONGraph Rust (IVF)** | 0.055 | 0.565 | 0.671 | - | 0.671 | 146 | 172 | +| **ISONGraph Rust, target-cpu=native (exact)** | 0.057 | 0.000 | 0.057 | - | 0.115 | 143 | 143 | +| **ISONGraph Rust, target-cpu=native (IVF)** | 0.057 | 0.575 | 0.690 | - | 0.690 | 146 | 172 | +| chromadb (HNSW) | 9.06 | 0.000 | 9.06 | 9.15 (9.06-9.23) | 9.16 | 289 | 219 | +| faiss IndexFlatIP | 0.051 | 0.000 | 0.051 | 0.082 (0.051-0.088) | 0.054 | 74 | 122 | +| faiss HNSWFlat | 1.98 | 0.000 | 1.98 | 2.00 (1.98-2.09) | 1.98 | 85 | 132 | +| faiss IVFFlat | 0.000 | 1.29 | 1.29 | 1.26 (1.25-1.29) | 1.30 | 105 | 181 | +| hnswlib | 1.13 | 0.000 | 1.13 | 1.13 (1.09-1.20) | 1.13 | 97 | 108 | +| lancedb (no index) | 0.103 | 0.000 | 0.103 | 0.223 (0.103-0.247) | 0.137 | 107 | 125 | +| lancedb IVF_FLAT | 0.113 | 0.858 | 0.971 | 0.869 (0.830-0.971) | 1.06 | 189 | 230 | +| numpy brute force | 0.015 | 0.000 | 0.015 | 0.015 (0.015-0.018) | 0.018 | 62 | 74 | +| sqlite-vec (vec0, exact) | 0.556 | 0.000 | 0.556 | 1.99 (0.556-2.16) | 0.588 | 4 | 23 | +| usearch (HNSW) | 1.57 | 0.000 | 1.57 | 1.56 (1.34-1.57) | 1.57 | 85 | 96 | + +
Full accuracy sweeps (recall@10, single-thread QPS, latency, all-core QPS) + +**ISONGraph C++, /arch:AVX2 (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 23.4 (23.3-27.3) | 42.3 | 45.4 | 47.4 | - | 97 | + +**ISONGraph C++, /arch:AVX2 (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2796 | 23360 (21573-23763) | 0.045 | 0.056 | 0.066 | - | 2000 | +| 2 | 0.4219 | 19563 (18794-20324) | 0.051 | 0.066 | 0.077 | - | 2000 | +| 4 | 0.5793 | 16494 (16250-17277) | 0.059 | 0.078 | 0.091 | - | 2000 | +| 8 | 0.7401 | 11995 (11642-12084) | 0.082 | 0.112 | 0.127 | - | 2000 | +| 16 | 0.8696 | 7631 (7570-7635) | 0.128 | 0.177 | 0.203 | - | 2000 | +| 32 | 0.9522 | 4693 (4524-4768) | 0.211 | 0.293 | 0.336 | - | 2000 | +| 64 | 0.9882 | 2719 (2711-2761) | 0.361 | 0.484 | 0.553 | - | 2000 | +| 128 | 0.9984 | 1488 (1403-1494) | 0.675 | 0.885 | 1.02 | - | 2000 | +| 256 | 0.9999 | 787 (783-833) | 1.24 | 1.52 | 1.67 | - | 2000 | +| 1265 | 1.0000 | 23.0 (22.9-23.3) | 43.3 | 46.3 | 48.4 | - | 69 | + +**ISONGraph C++ (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 22.8 (22.4-25.2) | 44.2 | 47.5 | 49.5 | - | 92 | + +**ISONGraph C++ (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2796 | 19362 (18833-19708) | 0.052 | 0.065 | 0.072 | - | 2000 | +| 2 | 0.4219 | 16480 (16363-16797) | 0.059 | 0.075 | 0.086 | - | 2000 | +| 4 | 0.5793 | 13533 (13443-13864) | 0.071 | 0.097 | 0.118 | - | 2000 | +| 8 | 0.7401 | 10563 (10319-10609) | 0.092 | 0.129 | 0.154 | - | 2000 | +| 16 | 0.8696 | 6672 (6639-6802) | 0.144 | 0.204 | 0.242 | - | 2000 | +| 32 | 0.9522 | 4054 (3966-4060) | 0.241 | 0.336 | 0.393 | - | 2000 | +| 64 | 0.9882 | 2290 (2202-2307) | 0.430 | 0.605 | 0.739 | - | 2000 | +| 128 | 0.9984 | 1321 (1305-1339) | 0.737 | 1.02 | 1.22 | - | 2000 | +| 256 | 0.9999 | 679 (677-688) | 1.45 | 1.84 | 2.06 | - | 2000 | +| 1265 | 1.0000 | 22.3 (22.3-23.5) | 44.8 | 47.5 | 49.3 | - | 66 | + +**ISONGraph C# (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 55.0 (55.0-72.9) | 14.6 | 42.8 | 80.8 | - | 314 | + +**ISONGraph C# (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2850 | 8528 (5455-9395) | 0.116 | 0.295 | 0.324 | - | 2000 | +| 2 | 0.4301 | 7905 (7740-8001) | 0.124 | 0.142 | 0.156 | - | 2000 | +| 4 | 0.5908 | 6901 (6828-7039) | 0.143 | 0.170 | 0.187 | - | 2000 | +| 8 | 0.7513 | 5823 (5639-5949) | 0.172 | 0.212 | 0.236 | - | 2000 | +| 16 | 0.8749 | 4284 (4243-4379) | 0.231 | 0.294 | 0.331 | - | 2000 | +| 32 | 0.9526 | 2805 (2528-2838) | 0.362 | 0.482 | 0.555 | - | 2000 | +| 64 | 0.9889 | 1727 (1719-1855) | 0.556 | 0.748 | 0.870 | - | 2000 | +| 128 | 0.9984 | 1056 (1029-1103) | 0.915 | 1.28 | 1.76 | - | 2000 | +| 256 | 0.9999 | 622 (597-652) | 1.58 | 2.12 | 2.44 | - | 1547 | +| 1265 | 1.0000 | 53.6 (53.0-53.7) | 15.4 | 40.9 | 87.7 | - | 198 | + +**ISONGraph Python (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 318 (265-323) | 3.22 | 4.42 | 5.68 | - | 1127 | + +**ISONGraph Python (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2856 | 15392 (15349-15420) | 0.061 | 0.084 | 0.121 | - | 2000 | +| 2 | 0.4253 | 13796 (13604-13991) | 0.069 | 0.091 | 0.119 | - | 2000 | +| 4 | 0.5855 | 10215 (10168-10222) | 0.093 | 0.130 | 0.163 | - | 2000 | +| 8 | 0.7427 | 7098 (6537-7152) | 0.136 | 0.186 | 0.230 | - | 2000 | +| 16 | 0.8690 | 4348 (4293-4513) | 0.212 | 0.298 | 0.458 | - | 2000 | +| 32 | 0.9468 | 2722 (2610-2761) | 0.339 | 0.516 | 0.929 | - | 2000 | +| 64 | 0.9869 | 1688 (1663-1711) | 0.570 | 0.778 | 1.08 | - | 2000 | +| 128 | 0.9976 | 880 (876-894) | 1.07 | 1.58 | 2.25 | - | 2000 | +| 256 | 0.9998 | 499 (496-509) | 1.90 | 2.53 | 3.37 | - | 1542 | +| 1265 | 1.0000 | 338 (337-354) | 2.79 | 3.71 | 4.72 | - | 1068 | + +**ISONGraph JS (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 29.0 (27.6-32.5) | 33.7 | 41.6 | 48.7 | - | 106 | + +**ISONGraph JS (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2886 | 3499 (3416-3525) | 0.289 | 0.350 | 0.424 | - | 2000 | +| 2 | 0.4338 | 3127 (3108-3164) | 0.317 | 0.380 | 0.426 | - | 2000 | +| 4 | 0.5894 | 2439 (2407-2486) | 0.399 | 0.524 | 0.617 | - | 2000 | +| 8 | 0.7425 | 1900 (1884-1909) | 0.516 | 0.683 | 0.797 | - | 2000 | +| 16 | 0.8685 | 1319 (1308-1363) | 0.739 | 1.02 | 1.19 | - | 2000 | +| 32 | 0.9486 | 810 (763-820) | 1.24 | 1.70 | 1.93 | - | 2000 | +| 64 | 0.9877 | 480 (464-482) | 2.10 | 2.73 | 3.12 | - | 1328 | +| 128 | 0.9984 | 248 (244-256) | 3.99 | 4.99 | 5.65 | - | 761 | +| 256 | 1.0000 | 142 (141-145) | 7.06 | 8.09 | 8.66 | - | 424 | +| 1265 | 1.0000 | 27.8 (26.8-28.0) | 35.2 | 42.4 | 45.9 | - | 79 | + +**ISONGraph Rust (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 344 (333-398) | 2.86 | 3.36 | 3.84 | - | 1231 | + +**ISONGraph Rust (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2912 | 33756 (33679-34626) | 0.029 | 0.038 | 0.045 | 16984 † | 2000 | +| 2 | 0.4356 | 25889 (25153-26284) | 0.038 | 0.051 | 0.060 | 14003 † | 2000 | +| 4 | 0.5955 | 20026 (19055-22804) | 0.048 | 0.066 | 0.079 | 10165 † | 2000 | +| 8 | 0.7474 | 13525 (13509-13581) | 0.072 | 0.100 | 0.117 | 6768 † | 2000 | +| 16 | 0.8710 | 8413 (8216-8451) | 0.116 | 0.162 | 0.191 | 4405 † | 2000 | +| 32 | 0.9486 | 4980 (4685-5131) | 0.199 | 0.276 | 0.332 | 2648 † | 2000 | +| 64 | 0.9860 | 2725 (2580-2747) | 0.365 | 0.508 | 0.626 | 1464 † | 2000 | +| 128 | 0.9984 | 1490 (1475-1517) | 0.666 | 0.856 | 0.989 | 738 † | 2000 | +| 256 | 0.9998 | 827 (800-837) | 1.20 | 1.50 | 1.66 | 468 † | 2000 | +| 1265 | 1.0000 | 335 (333-339) | 2.96 | 3.43 | 3.81 | 378 † | 1030 | + +**ISONGraph Rust, target-cpu=native (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 364 (362-465) | 2.62 | 3.14 | 3.42 | - | 1261 | + +**ISONGraph Rust, target-cpu=native (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2912 | 36185 (33982-42399) | 0.027 | 0.035 | 0.042 | 16215 † | 2000 | +| 2 | 0.4356 | 27078 (25967-27602) | 0.036 | 0.049 | 0.056 | 14669 † | 2000 | +| 4 | 0.5955 | 20258 (20173-22787) | 0.047 | 0.063 | 0.073 | 10146 † | 2000 | +| 8 | 0.7474 | 13568 (13551-14109) | 0.071 | 0.098 | 0.114 | 7176 † | 2000 | +| 16 | 0.8710 | 8586 (8391-9422) | 0.112 | 0.157 | 0.179 | 4303 † | 2000 | +| 32 | 0.9486 | 4741 (4733-4973) | 0.204 | 0.276 | 0.317 | 2523 † | 2000 | +| 64 | 0.9860 | 2780 (2727-2801) | 0.357 | 0.476 | 0.545 | 1399 † | 2000 | +| 128 | 0.9984 | 1612 (1463-1643) | 0.628 | 0.829 | 0.976 | 744 † | 2000 | +| 256 | 0.9998 | 839 (799-840) | 1.19 | 1.55 | 1.90 | 486 † | 1908 | +| 1265 | 1.0000 | 360 (352-360) | 2.79 | 3.26 | 3.65 | 316 † | 1103 | + +**chromadb (HNSW)** (ef_search) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.7916 | 1279 (1260-1281) | 0.751 | 0.990 | 1.55 | 21363 | 2000 | +| 16 | 0.8714 | 1258 (1240-1307) | 0.766 | 0.934 | 1.69 | 18336 | 2000 | +| 32 | 0.9524 | 1163 (1116-1185) | 0.818 | 1.11 | 1.83 | 12847 | 2000 | +| 64 | 0.9888 | 1129 (1112-1154) | 0.841 | 1.09 | 1.71 | 8413 | 2000 | +| 128 | 0.9978 | 982 (965-1008) | 0.952 | 1.35 | 2.21 | 5786 | 2000 | +| 256 | 0.9998 | 812 (779-818) | 1.19 | 1.66 | 2.62 | 3652 | 2000 | +| 512 | 0.9999 | 764 (757-783) | 1.24 | 1.72 | 2.46 | 2105 | 2000 | + +**faiss IndexFlatIP** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 283 (280-332) | 3.30 | 4.26 | 6.04 | 4537 | 920 | + +**faiss HNSWFlat** (efSearch) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.7926 | 35472 (34557-35531) | 0.027 | 0.038 | 0.053 | 579,257 | 2000 | +| 16 | 0.8737 | 28109 (27242-28189) | 0.034 | 0.048 | 0.061 | 407,672 | 2000 | +| 24 | 0.9257 | 21197 (21007-21337) | 0.045 | 0.063 | 0.084 | 354,906 | 2000 | +| 32 | 0.9533 | 16937 (16706-17368) | 0.056 | 0.078 | 0.121 | 328,882 | 2000 | +| 48 | 0.9778 | 12453 (12093-12677) | 0.078 | 0.109 | 0.149 | 191,734 | 2000 | +| 64 | 0.9889 | 9172 (7963-9527) | 0.101 | 0.148 | 0.263 | 140,578 | 2000 | +| 96 | 0.9958 | 6013 (5876-6248) | 0.150 | 0.228 | 0.409 | 99528 | 2000 | +| 128 | 0.9981 | 4997 (4910-5057) | 0.191 | 0.264 | 0.397 | 90958 | 2000 | +| 192 | 0.9992 | 3318 (3279-3386) | 0.289 | 0.393 | 0.566 | 70793 | 2000 | +| 256 | 0.9997 | 2528 (2456-2663) | 0.380 | 0.508 | 0.801 | 48638 | 2000 | +| 400 | 0.9999 | 1537 (1491-1552) | 0.627 | 0.836 | 1.33 | 29966 | 2000 | + +**faiss IVFFlat** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2837 | 46449 (45779-47166) | 0.021 | 0.025 | 0.031 | 260,845 | 2000 | +| 2 | 0.4257 | 39480 (39424-40291) | 0.024 | 0.031 | 0.041 | 250,401 | 2000 | +| 4 | 0.5849 | 30978 (28532-32812) | 0.030 | 0.042 | 0.062 | 221,614 | 2000 | +| 8 | 0.7447 | 22575 (21736-24142) | 0.041 | 0.058 | 0.083 | 207,689 | 2000 | +| 16 | 0.8686 | 14664 (14010-15618) | 0.064 | 0.095 | 0.132 | 144,067 | 2000 | +| 32 | 0.9505 | 8764 (8547-8823) | 0.105 | 0.160 | 0.248 | 112,537 | 2000 | +| 64 | 0.9876 | 4991 (4987-5213) | 0.181 | 0.279 | 0.610 | 51491 | 2000 | +| 128 | 0.9986 | 2804 (2624-2813) | 0.340 | 0.506 | 0.854 | 29165 | 2000 | +| 256 | 1.0000 | 1438 (1423-1475) | 0.654 | 1.01 | 1.51 | 15785 | 2000 | +| 1265 | 1.0000 | 308 (300-311) | 3.13 | 3.91 | 4.82 | 3462 | 988 | + +**hnswlib** (ef) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.7877 | 44466 (42336-45863) | 0.021 | 0.031 | 0.043 | 392,472 | 2000 | +| 16 | 0.8690 | 35361 (34241-36359) | 0.028 | 0.038 | 0.049 | 310,299 | 2000 | +| 24 | 0.9228 | 26532 (26390-27774) | 0.036 | 0.051 | 0.068 | 281,005 | 2000 | +| 32 | 0.9503 | 22404 (21713-22520) | 0.044 | 0.061 | 0.079 | 192,350 | 2000 | +| 48 | 0.9762 | 16412 (15860-16465) | 0.060 | 0.081 | 0.103 | 175,231 | 2000 | +| 64 | 0.9879 | 12748 (12557-13235) | 0.075 | 0.105 | 0.137 | 130,878 | 2000 | +| 96 | 0.9954 | 8801 (8414-8850) | 0.107 | 0.157 | 0.246 | 99854 | 2000 | +| 128 | 0.9978 | 6789 (6288-7469) | 0.137 | 0.210 | 0.373 | 80265 | 2000 | +| 192 | 0.9990 | 4893 (4496-4938) | 0.193 | 0.308 | 0.601 | 53647 | 2000 | +| 256 | 0.9997 | 4274 (4261-4377) | 0.230 | 0.315 | 0.412 | 47855 | 2000 | +| 400 | 1.0000 | 2940 (2766-2973) | 0.329 | 0.474 | 0.840 | 36822 | 2000 | + +**lancedb (no index)** (nprobes) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 15.1 (13.7-16.0) | 66.9 | 78.5 | 82.0 | - | 43 | + +**lancedb IVF_FLAT** (nprobes) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2785 | 277 (270-279) | 3.55 | 4.48 | 5.87 | - | 835 | +| 2 | 0.4209 | 287 (268-289) | 3.47 | 4.27 | 5.94 | - | 780 | +| 4 | 0.5785 | 276 (270-283) | 3.53 | 4.43 | 5.90 | - | 832 | +| 8 | 0.7388 | 269 (264-281) | 3.59 | 4.54 | 6.01 | - | 825 | +| 16 | 0.8673 | 263 (263-270) | 3.68 | 4.55 | 6.20 | - | 800 | +| 32 | 0.9486 | 247 (247-251) | 3.90 | 4.89 | 7.06 | - | 766 | +| 64 | 0.9868 | 238 (234-244) | 4.06 | 5.08 | 6.78 | - | 731 | +| 128 | 0.9977 | 213 (203-215) | 4.59 | 5.78 | 8.62 | - | 624 | +| 256 | 0.9999 | 186 (185-189) | 5.21 | 6.41 | 8.26 | - | 550 | +| 1265 | 1.0000 | 92.3 (90.6-92.3) | 10.6 | 12.9 | 17.3 | - | 300 | + +**numpy brute force** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 282 (267-297) | 3.34 | 4.87 | 6.69 | 1543 | 842 | + +**sqlite-vec (vec0, exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 21.6 (21.0-22.4) | 47.0 | 52.6 | 57.7 | - | 93 | + +**usearch (HNSW)** (expansion_search) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.7861 | 27069 (26424-27812) | 0.035 | 0.052 | 0.072 | 360,458 | 2000 | +| 16 | 0.8702 | 19975 (19615-21362) | 0.045 | 0.066 | 0.089 | 235,977 | 2000 | +| 24 | 0.9224 | 16246 (15886-16879) | 0.058 | 0.080 | 0.103 | 160,195 | 2000 | +| 32 | 0.9517 | 13048 (12862-13737) | 0.072 | 0.101 | 0.133 | 103,358 | 2000 | +| 48 | 0.9768 | 9040 (8229-9296) | 0.105 | 0.167 | 0.257 | 133,914 | 2000 | +| 64 | 0.9868 | 6439 (6418-7550) | 0.134 | 0.222 | 0.319 | 91078 | 2000 | +| 96 | 0.9939 | 5250 (5048-5360) | 0.181 | 0.274 | 0.393 | 77948 | 2000 | +| 128 | 0.9958 | 4190 (4062-4302) | 0.228 | 0.336 | 0.633 | 32827 | 2000 | +| 192 | 0.9970 | 2769 (2674-3016) | 0.336 | 0.519 | 0.885 | 47475 | 2000 | +| 256 | 0.9975 | 2330 (1993-2437) | 0.422 | 0.630 | 0.957 | 37724 | 2000 | +| 400 | 0.9976 | 1592 (1525-1656) | 0.607 | 0.875 | 1.32 | 25792 | 2000 | +| 640 | 0.9977 | 1064 (1030-1108) | 0.909 | 1.29 | 1.82 | 13865 | 2000 | + +
+ +### text384, N = 1,000,000 + +Queries/second at recall@10 = 0.90 / 0.95 / 0.99, log-linearly interpolated between the two measured sweep points that bracket the target (no extrapolation; `*` = the two bracketing points are more than 0.05 recall apart, so the measured point above the target is shown instead, a conservative value; `-` = never reached; exact scans appear at their single recall-1.0 point). 1T = one pinned P-core, one query at a time. All-core = batched call on 32 logical CPUs (`†` = the ISONGraph ports have no batch API, so this is the same one-query loop with automatic scan threads). Latency columns are for the first measured sweep point with recall >= 0.95. + +| system | QPS 1T @0.90 | @0.95 | @0.99 | all-core QPS @0.90 | @0.95 | 1T latency ms median / p95 / p99 | +|---|---|---|---|---|---|---| +| **ISONGraph C++, /arch:AVX2 (exact)** | 2.13 | 2.13 | 2.13 | - | - | 466 / 491 / 493 (recall 1.000) | +| **ISONGraph C++, /arch:AVX2 (IVF)** | 221* | 128 | - | - | - | 8.38 / 9.75 / 11.4 (recall 0.955) | +| **ISONGraph C++ (exact)** | 2.11 | 2.11 | 2.11 | - | - | 469 / 500 / 516 (recall 1.000) | +| **ISONGraph C++ (IVF)** | 173* | 94.0 | - | - | - | 11.1 / 12.9 / 14.1 (recall 0.953) | +| **ISONGraph C# (exact)** | 3.72 | 3.72 | 3.72 | - | - | 245 / 394 / 398 (recall 1.000) | +| **ISONGraph C# (IVF)** | 141* | - | - | - | - | - | +| **ISONGraph Python (exact)** | 12.7 | 12.7 | 12.7 | - | - | 77.2 / 86.1 / 93.0 (recall 1.000) | +| **ISONGraph Python (IVF)** | 125* | 69.2 | 16.7 | - | - | 14.6 / 18.1 / 23.7 (recall 0.953) | +| **ISONGraph JS (exact)** | 1.67 | 1.67 | 1.67 | - | - | 545 / 838 / 872 (recall 1.000) | +| **ISONGraph JS (IVF)** | 52.0* | - | - | - | - | - | +| **ISONGraph Rust (exact)** | 15.1 | 15.1 | 15.1 | - | - | 65.3 / 70.0 / 80.5 (recall 1.000) | +| **ISONGraph Rust (IVF)** | 192* | 103 | - | 579* † | 356 † | 9.89 / 11.4 / 12.3 (recall 0.951) | +| **ISONGraph Rust, target-cpu=native (exact)** | 15.8 | 15.8 | 15.8 | - | - | 62.7 / 68.9 / 71.3 (recall 1.000) | +| **ISONGraph Rust, target-cpu=native (IVF)** | 190* | 103 | - | 548* † | 342 † | 9.91 / 11.7 / 12.5 (recall 0.952) | +| chromadb (HNSW) | 687* | 605 | - | 1939* | 1464 | 1.83 / 2.15 / 2.93 (recall 0.967) | +| faiss IndexFlatIP | 13.4 | 13.4 | 13.4 | 540 | 540 | 73.4 / 83.8 / 87.6 (recall 1.000) | +| faiss HNSWFlat | 2880 | 1572 | 406 | 10413 | 5757 | 0.796 / 0.979 / 1.24 (recall 0.965) | +| faiss IVFFlat | 321* | 190 | 22.5 | 910* | 520 | 5.63 / 7.03 / 13.2 (recall 0.956) | +| hnswlib | 2663 | 1486 | 474 | 10424 | 5894 | 0.718 / 0.905 / 1.29 (recall 0.957) | +| lancedb (no index) | 0.725 | 0.725 | 0.725 | - | - | 1372 / 1438 / 1463 (recall 1.000) | +| lancedb IVF_FLAT | 95.4* | 11.4* | 11.4* | - | - | 87.4 / 90.4 / 92.8 (recall 1.000) | +| numpy brute force | 14.2 | 14.2 | 14.2 | 142 | 142 | 69.3 / 77.1 / 84.0 (recall 1.000) | +| sqlite-vec (vec0, exact) | 1.05 | 1.05 | 1.05 | - | - | 947 / 988 / 991 (recall 1.000) | +| usearch (HNSW) | 1274 | 706 | 214 | 10254 | 5827 | 1.52 / 1.93 / 2.44 (recall 0.957) | + +Build and memory (RSS growth over the process baseline after imports; the raw vectors are streamed in from disk and are not held by the harness). + +| system | ingest s | index s | total build s | 3 builds: median (min-max) s | ready s (incl. first query) | steady RSS +MB | peak RSS +MB (build) | +|---|---|---|---|---|---|---|---| +| **ISONGraph C++, /arch:AVX2 (exact)** | 0.922 | 0.000 | 0.922 | - | 1.43 | 1,783 | 1,922 | +| **ISONGraph C++, /arch:AVX2 (IVF)** | 0.922 | 6.20 | 7.12 | - | 7.12 | 3,279 | 3,514 | +| **ISONGraph C++ (exact)** | 0.917 | 0.000 | 0.917 | - | 1.43 | 1,783 | 1,922 | +| **ISONGraph C++ (IVF)** | 0.917 | 11.4 | 12.4 | - | 12.4 | 3,279 | 3,514 | +| **ISONGraph C# (exact)** | 0.217 | 0.000 | 0.217 | - | 0.515 | 2,447 | 2,447 | +| **ISONGraph C# (IVF)** | 0.177 | 10.7 | 10.9 | - | 10.9 | 6,656 | 6,664 | +| **ISONGraph Python (exact)** | 19.6 | 0.000 | 19.6 | - | 22.8 | 4,223 | 2,682 | +| **ISONGraph Python (IVF)** | 20.8 | 21.4 | 42.2 | - | 42.2 | 4,251 | 4,795 | +| **ISONGraph JS (exact)** | 0.470 | 0.000 | 0.470 | - | 1.03 | 3,620 | 3,620 | +| **ISONGraph JS (IVF)** | 0.691 | 521 | 522 | - | 522 | 7,002 | 7,002 | +| **ISONGraph Rust (exact)** | 0.812 | 0.000 | 0.812 | - | 1.84 | 3,517 | 3,524 | +| **ISONGraph Rust (IVF)** | 0.812 | 5.71 | 7.55 | - | 7.55 | 3,532 | 3,779 | +| **ISONGraph Rust, target-cpu=native (exact)** | 0.763 | 0.000 | 0.763 | - | 1.83 | 3,517 | 3,525 | +| **ISONGraph Rust, target-cpu=native (IVF)** | 0.763 | 5.56 | 7.40 | - | 7.40 | 3,532 | 3,779 | +| chromadb (HNSW) | 343 | 0.000 | 343 | - | 345 | 4,023 | 2,147 | +| faiss IndexFlatIP | 1.10 | 0.000 | 1.10 | - | 1.18 | 1,553 | 2,122 | +| faiss HNSWFlat | 221 | 0.000 | 221 | - | 221 | 1,640 | 2,168 | +| faiss IVFFlat | 0.000 | 18.5 | 18.5 | - | 18.6 | 1,885 | 3,592 | +| hnswlib | 173 | 0.000 | 173 | - | 173 | 1,830 | 1,860 | +| lancedb (no index) | 2.04 | 0.000 | 2.04 | - | 2.69 | 152 | 226 | +| lancedb IVF_FLAT | 2.06 | 10.7 | 12.8 | - | 13.4 | 1,736 | 3,081 | +| numpy brute force | 0.298 | 0.000 | 0.298 | - | 0.368 | 1,576 | 1,605 | +| sqlite-vec (vec0, exact) | 9.60 | 0.000 | 9.60 | - | 10.5 | 4 | 62 | +| usearch (HNSW) | 174 | 0.000 | 174 | - | 174 | 1,715 | 1,745 | + +
Full accuracy sweeps (recall@10, single-thread QPS, latency, all-core QPS) + +**ISONGraph C++, /arch:AVX2 (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 2.13 (2.13-2.13) | 466 | 491 | 493 | - | 30 | + +**ISONGraph C++, /arch:AVX2 (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2666 | 4780 (4475-4797) | 0.206 | 0.281 | 0.373 | - | 2000 | +| 2 | 0.3725 | 3920 (3909-3942) | 0.246 | 0.330 | 0.381 | - | 2000 | +| 4 | 0.4895 | 3016 (2956-3039) | 0.323 | 0.432 | 0.499 | - | 2000 | +| 8 | 0.6020 | 1989 (1984-2021) | 0.489 | 0.658 | 0.772 | - | 2000 | +| 16 | 0.7007 | 1243 (1230-1245) | 0.795 | 1.02 | 1.15 | - | 2000 | +| 32 | 0.7890 | 713 (696-718) | 1.39 | 1.74 | 2.00 | - | 2000 | +| 64 | 0.8603 | 404 (397-406) | 2.45 | 3.04 | 3.41 | - | 1301 | +| 128 | 0.9141 | 221 (216-223) | 4.50 | 5.33 | 5.77 | - | 714 | +| 256 | 0.9547 | 119 (115-120) | 8.38 | 9.75 | 11.4 | - | 360 | + +**ISONGraph C++ (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 2.11 (2.11-2.11) | 469 | 500 | 516 | - | 30 | + +**ISONGraph C++ (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2669 | 4177 (3635-4194) | 0.239 | 0.334 | 0.550 | - | 2000 | +| 2 | 0.3726 | 3308 (3222-3337) | 0.297 | 0.399 | 0.457 | - | 2000 | +| 4 | 0.4897 | 2387 (2353-2421) | 0.409 | 0.556 | 0.668 | - | 2000 | +| 8 | 0.6026 | 1589 (1576-1602) | 0.613 | 0.819 | 0.954 | - | 2000 | +| 16 | 0.7010 | 961 (954-985) | 1.02 | 1.32 | 1.49 | - | 2000 | +| 32 | 0.7849 | 559 (554-559) | 1.78 | 2.21 | 2.50 | - | 1627 | +| 64 | 0.8613 | 311 (303-313) | 3.16 | 3.87 | 4.28 | - | 969 | +| 128 | 0.9152 | 173 (171-173) | 5.78 | 6.77 | 7.48 | - | 499 | +| 256 | 0.9528 | 89.5 (89.5-90.5) | 11.1 | 12.9 | 14.1 | - | 286 | + +**ISONGraph C# (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 3.72 (3.72-3.72) | 245 | 394 | 398 | - | 30 | + +**ISONGraph C# (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2741 | 2529 (2179-2620) | 0.384 | 0.630 | 0.733 | - | 2000 | +| 2 | 0.3823 | 2129 (2066-2179) | 0.455 | 0.612 | 0.754 | - | 2000 | +| 4 | 0.5001 | 1685 (1601-1685) | 0.591 | 0.783 | 0.916 | - | 2000 | +| 8 | 0.6082 | 1171 (1136-1175) | 0.840 | 1.11 | 1.26 | - | 2000 | +| 16 | 0.7054 | 747 (734-756) | 1.32 | 1.74 | 2.02 | - | 2000 | +| 32 | 0.7822 | 445 (438-446) | 2.23 | 2.83 | 3.18 | - | 1183 | +| 64 | 0.8550 | 253 (250-258) | 3.88 | 4.87 | 5.62 | - | 857 | +| 128 | 0.9096 | 141 (141-145) | 6.93 | 8.40 | 9.48 | - | 426 | +| 256 | 0.9462 | 74.5 (72.8-77.0) | 13.1 | 16.2 | 19.2 | - | 234 | + +**ISONGraph Python (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 12.7 (12.7-12.8) | 77.2 | 86.1 | 93.0 | - | 36 | + +**ISONGraph Python (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2745 | 3293 (3160-3317) | 0.290 | 0.423 | 0.519 | - | 2000 | +| 2 | 0.3764 | 2389 (2356-2438) | 0.393 | 0.569 | 0.712 | - | 2000 | +| 4 | 0.4940 | 1711 (1689-1742) | 0.557 | 0.786 | 1.03 | - | 2000 | +| 8 | 0.6007 | 1166 (1077-1175) | 0.857 | 1.13 | 1.33 | - | 2000 | +| 16 | 0.7006 | 668 (658-687) | 1.44 | 1.88 | 2.25 | - | 2000 | +| 32 | 0.7813 | 396 (385-399) | 2.50 | 3.08 | 3.62 | - | 1191 | +| 64 | 0.8540 | 217 (213-219) | 4.47 | 5.41 | 6.55 | - | 689 | +| 128 | 0.9119 | 125 (125-125) | 7.76 | 9.83 | 15.3 | - | 404 | +| 256 | 0.9532 | 65.8 (65.0-67.4) | 14.6 | 18.1 | 23.7 | - | 211 | +| 4000 | 1.0000 | 11.5 (11.4-12.1) | 83.0 | 99.3 | 139 | - | 37 | + +**ISONGraph JS (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 1.67 (1.67-1.67) | 545 | 838 | 872 | - | 30 | + +**ISONGraph JS (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2670 | 666 (660-668) | 1.47 | 1.74 | 1.99 | - | 1907 | +| 2 | 0.3726 | 594 (587-594) | 1.66 | 2.02 | 2.28 | - | 1634 | +| 4 | 0.4833 | 493 (488-493) | 2.00 | 2.55 | 2.80 | - | 1452 | +| 8 | 0.5923 | 368 (366-368) | 2.70 | 3.40 | 3.68 | - | 1071 | +| 16 | 0.6892 | 245 (244-249) | 4.03 | 5.05 | 5.61 | - | 647 | +| 32 | 0.7796 | 158 (158-161) | 6.23 | 7.67 | 8.31 | - | 465 | +| 64 | 0.8444 | 95.9 (94.8-96.2) | 10.3 | 12.4 | 13.5 | - | 252 | +| 128 | 0.9132 | 52.0 (50.9-54.3) | 18.7 | 23.3 | 25.6 | - | 167 | +| 256 | 0.9488 | 29.0 (28.3-29.4) | 34.3 | 40.4 | 42.1 | - | 84 | + +**ISONGraph Rust (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 15.1 (15.1-15.1) | 65.3 | 70.0 | 80.5 | - | 46 | + +**ISONGraph Rust (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2757 | 5323 (5238-5829) | 0.175 | 0.248 | 0.324 | 5692 † | 2000 | +| 2 | 0.3806 | 4242 (4037-4392) | 0.229 | 0.330 | 0.408 | 4320 † | 2000 | +| 4 | 0.4961 | 3080 (2980-3091) | 0.323 | 0.436 | 0.509 | 2997 † | 2000 | +| 8 | 0.6046 | 1936 (1889-1993) | 0.507 | 0.673 | 0.791 | 1970 † | 2000 | +| 16 | 0.7022 | 1147 (1140-1157) | 0.867 | 1.09 | 1.22 | 1233 † | 2000 | +| 32 | 0.7902 | 648 (647-652) | 1.53 | 1.88 | 2.05 | 1131 † | 1961 | +| 64 | 0.8606 | 356 (353-356) | 2.77 | 3.37 | 4.00 | 867 † | 1083 | +| 128 | 0.9149 | 192 (190-193) | 5.13 | 6.23 | 7.19 | 579 † | 598 | +| 256 | 0.9512 | 100 (100-100) | 9.89 | 11.4 | 12.3 | 350 † | 287 | + +**ISONGraph Rust, target-cpu=native (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 15.8 (15.6-15.8) | 62.7 | 68.9 | 71.3 | - | 49 | + +**ISONGraph Rust, target-cpu=native (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2757 | 5805 (5410-5872) | 0.171 | 0.243 | 0.283 | 6133 † | 2000 | +| 2 | 0.3806 | 4753 (4604-4781) | 0.206 | 0.294 | 0.342 | 4672 † | 2000 | +| 4 | 0.4961 | 3050 (2960-3122) | 0.315 | 0.444 | 0.555 | 3007 † | 2000 | +| 8 | 0.6046 | 1935 (1921-1950) | 0.510 | 0.678 | 0.774 | 1956 † | 2000 | +| 16 | 0.7022 | 1143 (1127-1148) | 0.869 | 1.12 | 1.26 | 1200 † | 2000 | +| 32 | 0.7908 | 647 (639-650) | 1.54 | 1.89 | 2.06 | 1086 † | 1986 | +| 64 | 0.8593 | 354 (352-361) | 2.79 | 3.36 | 3.61 | 872 † | 1149 | +| 128 | 0.9149 | 190 (188-191) | 5.21 | 6.19 | 6.68 | 548 † | 570 | +| 256 | 0.9521 | 99.8 (98.9-100) | 9.91 | 11.7 | 12.5 | 333 † | 292 | + +**chromadb (HNSW)** (ef_search) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.5578 | 1205 (1198-1215) | 0.821 | 0.960 | 1.14 | 8619 | 2000 | +| 16 | 0.6533 | 1110 (1108-1170) | 0.866 | 1.02 | 1.36 | 6744 | 2000 | +| 32 | 0.7740 | 992 (991-1007) | 0.968 | 1.15 | 1.61 | 5012 | 2000 | +| 64 | 0.8717 | 878 (851-887) | 1.09 | 1.29 | 1.89 | 3256 | 2000 | +| 128 | 0.9323 | 687 (646-713) | 1.42 | 1.70 | 2.43 | 1939 | 2000 | +| 256 | 0.9674 | 535 (535-540) | 1.83 | 2.15 | 2.93 | 1112 | 1640 | +| 512 | 0.9859 | 349 (348-365) | 2.77 | 3.50 | 4.82 | 614 | 1103 | + +**faiss IndexFlatIP** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 13.4 (13.3-13.5) | 73.4 | 83.8 | 87.6 | 540 | 40 | + +**faiss HNSWFlat** (efSearch) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.5647 | 11088 (10192-11826) | 0.084 | 0.137 | 0.175 | 44533 | 2000 | +| 16 | 0.6714 | 8786 (8337-8862) | 0.110 | 0.168 | 0.203 | 31150 | 2000 | +| 24 | 0.7482 | 6566 (5968-6666) | 0.146 | 0.225 | 0.286 | 24367 | 2000 | +| 32 | 0.7935 | 5568 (5413-5569) | 0.173 | 0.247 | 0.321 | 19413 | 2000 | +| 48 | 0.8516 | 4032 (3892-4187) | 0.238 | 0.331 | 0.409 | 14764 | 2000 | +| 64 | 0.8875 | 3259 (3140-3290) | 0.299 | 0.401 | 0.510 | 11597 | 2000 | +| 96 | 0.9255 | 2238 (2124-2263) | 0.426 | 0.566 | 0.792 | 8359 | 2000 | +| 128 | 0.9445 | 1721 (1665-1743) | 0.558 | 0.717 | 0.982 | 6400 | 2000 | +| 192 | 0.9648 | 1233 (1191-1284) | 0.796 | 0.979 | 1.24 | 4330 | 2000 | +| 256 | 0.9742 | 913 (853-926) | 1.09 | 1.43 | 1.89 | 3621 | 2000 | +| 400 | 0.9843 | 594 (578-602) | 1.65 | 2.10 | 3.03 | 2364 | 1626 | +| 640 | 0.9913 | 371 (370-377) | 2.65 | 3.14 | 3.91 | 1590 | 1086 | + +**faiss IVFFlat** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2711 | 5665 (5464-5733) | 0.168 | 0.219 | 0.344 | 48473 | 2000 | +| 2 | 0.3827 | 4593 (4539-4850) | 0.197 | 0.260 | 0.462 | 33894 | 2000 | +| 4 | 0.4944 | 3674 (3513-3767) | 0.255 | 0.350 | 0.654 | 17040 | 2000 | +| 8 | 0.6087 | 2710 (2700-2774) | 0.351 | 0.475 | 0.670 | 11517 | 2000 | +| 16 | 0.7099 | 1693 (1644-1827) | 0.537 | 0.750 | 1.51 | 6263 | 2000 | +| 32 | 0.7952 | 1079 (1069-1087) | 0.874 | 1.16 | 1.62 | 3435 | 2000 | +| 64 | 0.8620 | 579 (578-598) | 1.59 | 2.10 | 3.37 | 1784 | 1690 | +| 128 | 0.9183 | 321 (315-324) | 3.00 | 3.76 | 5.05 | 910 | 986 | +| 256 | 0.9561 | 171 (169-171) | 5.63 | 7.03 | 13.2 | 467 | 541 | +| 4000 | 1.0000 | 12.3 (12.2-12.6) | 78.6 | 89.4 | 100 | 30.4 | 38 | + +**hnswlib** (ef) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.5589 | 9990 (7940-11885) | 0.092 | 0.170 | 0.251 | 45715 | 2000 | +| 16 | 0.6512 | 8804 (8071-8872) | 0.109 | 0.164 | 0.208 | 36683 | 2000 | +| 24 | 0.7278 | 6992 (6565-7080) | 0.136 | 0.197 | 0.249 | 27873 | 2000 | +| 32 | 0.7763 | 5558 (5293-5665) | 0.173 | 0.247 | 0.312 | 22939 | 2000 | +| 48 | 0.8354 | 4276 (4196-4387) | 0.224 | 0.306 | 0.392 | 17165 | 2000 | +| 64 | 0.8727 | 3223 (3211-3270) | 0.296 | 0.406 | 0.557 | 13525 | 2000 | +| 96 | 0.9101 | 2482 (2352-2494) | 0.398 | 0.523 | 0.663 | 9472 | 2000 | +| 128 | 0.9315 | 1908 (1862-1987) | 0.510 | 0.654 | 0.816 | 7534 | 2000 | +| 192 | 0.9572 | 1348 (1332-1385) | 0.718 | 0.905 | 1.29 | 5355 | 2000 | +| 256 | 0.9685 | 1043 (1037-1047) | 0.944 | 1.20 | 1.53 | 4059 | 2000 | +| 400 | 0.9817 | 700 (685-721) | 1.41 | 1.75 | 2.37 | 2681 | 2000 | +| 640 | 0.9901 | 472 (461-472) | 2.15 | 2.57 | 2.99 | 1848 | 1224 | + +**lancedb (no index)** (nprobes) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 0.725 (0.725-0.725) | 1372 | 1438 | 1463 | - | 30 | + +**lancedb IVF_FLAT** (nprobes) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2147 | 228 (225-231) | 4.26 | 5.05 | 5.76 | - | 688 | +| 2 | 0.3104 | 229 (223-229) | 4.29 | 5.03 | 6.11 | - | 703 | +| 4 | 0.4113 | 240 (234-243) | 4.13 | 4.66 | 5.25 | - | 728 | +| 8 | 0.5227 | 225 (223-234) | 4.31 | 4.95 | 5.88 | - | 630 | +| 16 | 0.6346 | 211 (211-220) | 4.57 | 5.35 | 6.32 | - | 689 | +| 32 | 0.7313 | 185 (185-192) | 5.16 | 6.38 | 8.70 | - | 593 | +| 64 | 0.8150 | 162 (159-165) | 5.95 | 7.13 | 9.41 | - | 525 | +| 128 | 0.8876 | 134 (127-135) | 7.29 | 8.52 | 10.2 | - | 389 | +| 256 | 0.9378 | 95.4 (95.0-96.7) | 10.2 | 11.7 | 13.3 | - | 299 | +| 4000 | 1.0000 | 11.4 (11.3-11.5) | 87.4 | 90.4 | 92.8 | - | 33 | + +**numpy brute force** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 14.2 (14.1-14.4) | 69.3 | 77.1 | 84.0 | 142 | 42 | + +**sqlite-vec (vec0, exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 1.05 (1.05-1.05) | 947 | 988 | 991 | - | 30 | + +**usearch (HNSW)** (expansion_search) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.5583 | 5894 (5335-6034) | 0.166 | 0.252 | 0.307 | 47685 | 2000 | +| 16 | 0.6489 | 4309 (4212-4677) | 0.217 | 0.329 | 0.396 | 33250 | 2000 | +| 24 | 0.7299 | 3584 (3020-3614) | 0.271 | 0.407 | 0.564 | 27185 | 2000 | +| 32 | 0.7786 | 2935 (2833-2957) | 0.328 | 0.473 | 0.566 | 22420 | 2000 | +| 48 | 0.8359 | 2082 (2039-2155) | 0.450 | 0.631 | 0.811 | 15681 | 2000 | +| 64 | 0.8685 | 1651 (1638-1655) | 0.574 | 0.780 | 1.15 | 13218 | 2000 | +| 96 | 0.9090 | 1183 (1141-1229) | 0.804 | 1.08 | 1.64 | 9541 | 2000 | +| 128 | 0.9307 | 920 (908-926) | 1.07 | 1.35 | 1.70 | 7204 | 2000 | +| 192 | 0.9569 | 642 (627-669) | 1.52 | 1.93 | 2.44 | 5398 | 2000 | +| 256 | 0.9673 | 522 (511-541) | 1.90 | 2.33 | 2.79 | 4125 | 1639 | +| 400 | 0.9822 | 335 (333-339) | 2.97 | 3.69 | 4.92 | 2616 | 999 | +| 640 | 0.9900 | 214 (208-220) | 4.61 | 5.83 | 11.1 | 1810 | 634 | + +
+ +### glove, N = 1,000,000 + +Queries/second at recall@10 = 0.90 / 0.95 / 0.99, log-linearly interpolated between the two measured sweep points that bracket the target (no extrapolation; `*` = the two bracketing points are more than 0.05 recall apart, so the measured point above the target is shown instead, a conservative value; `-` = never reached; exact scans appear at their single recall-1.0 point). 1T = one pinned P-core, one query at a time. All-core = batched call on 32 logical CPUs (`†` = the ISONGraph ports have no batch API, so this is the same one-query loop with automatic scan threads). Latency columns are for the first measured sweep point with recall >= 0.95. + +| system | QPS 1T @0.90 | @0.95 | @0.99 | all-core QPS @0.90 | @0.95 | 1T latency ms median / p95 / p99 | +|---|---|---|---|---|---|---| +| **ISONGraph C++, /arch:AVX2 (exact)** | 2.58 | 2.58 | 2.58 | - | - | 387 / 405 / 407 (recall 1.000) | +| **ISONGraph C++, /arch:AVX2 (IVF)** | 325* | 185 | - | - | - | 5.63 / 7.38 / 8.03 (recall 0.953) | +| **ISONGraph C++ (exact)** | 2.48 | 2.48 | 2.48 | - | - | 406 / 415 / 420 (recall 1.000) | +| **ISONGraph C++ (IVF)** | 322* | 175 | - | - | - | 5.80 / 7.43 / 8.02 (recall 0.951) | +| **ISONGraph C# (exact)** | 5.81 | 5.81 | 5.81 | - | - | 126 / 370 / 399 (recall 1.000) | +| **ISONGraph C# (IVF)** | 248* | - | - | - | - | - | +| **ISONGraph Python (exact)** | 26.5 | 26.5 | 26.5 | - | - | 37.1 / 42.5 / 47.8 (recall 1.000) | +| **ISONGraph Python (IVF)** | 255* | 156 | 47.9 | - | - | 6.95 / 9.14 / 14.9 (recall 0.957) | +| **ISONGraph Rust (exact)** | 43.5 | 43.5 | 43.5 | - | - | 22.5 / 24.6 / 25.9 (recall 1.000) | +| **ISONGraph Rust (IVF)** | 314* | 181 | - | 273* † | 176 † | 6.01 / 7.67 / 8.42 (recall 0.954) | +| **ISONGraph Rust, target-cpu=native (exact)** | 40.1 | 40.1 | 40.1 | - | - | 25.7 / 26.9 / 28.1 (recall 1.000) | +| **ISONGraph Rust, target-cpu=native (IVF)** | 325* | 171 | - | 292* † | 175 † | 6.36 / 8.01 / 8.54 (recall 0.954) | +| faiss IndexFlatIP | 40.7 | 40.7 | 40.7 | 639 | 639 | 24.2 / 27.1 / 33.2 (recall 1.000) | +| faiss HNSWFlat | 1405 | - | - | 11921 | - | - | +| faiss IVFFlat | 989* | 43.8* | 43.8* | 3993* | 159* | 22.2 / 28.9 / 31.4 (recall 1.000) | +| hnswlib | 1934 | - | - | 9075 | - | - | +| lancedb (no index) | 2.89 | 2.89 | 2.89 | - | - | 346 / 385 / 393 (recall 1.000) | +| lancedb IVF_FLAT | 133* | 109 | 29.3 | - | - | 9.32 / 10.9 / 11.4 (recall 0.951) | +| numpy brute force | 34.4 | 34.4 | 34.4 | 69.2 | 69.2 | 28.8 / 33.2 / 36.0 (recall 1.000) | +| usearch (HNSW) | 840 | - | - | 10396 | - | - | + +Build and memory (RSS growth over the process baseline after imports; the raw vectors are streamed in from disk and are not held by the harness). + +| system | ingest s | index s | total build s | 3 builds: median (min-max) s | ready s (incl. first query) | steady RSS +MB | peak RSS +MB (build) | +|---|---|---|---|---|---|---|---| +| **ISONGraph C++, /arch:AVX2 (exact)** | 0.556 | 0.000 | 0.556 | - | 0.871 | 671 | 810 | +| **ISONGraph C++, /arch:AVX2 (IVF)** | 0.556 | 6.73 | 7.29 | - | 7.29 | 1,079 | 1,141 | +| **ISONGraph C++ (exact)** | 0.533 | 0.000 | 0.533 | - | 0.815 | 672 | 810 | +| **ISONGraph C++ (IVF)** | 0.533 | 8.72 | 9.25 | - | 9.25 | 1,080 | 1,141 | +| **ISONGraph C# (exact)** | 0.198 | 0.000 | 0.198 | - | 0.320 | 810 | 810 | +| **ISONGraph C# (IVF)** | 0.339 | 11.3 | 11.6 | - | 11.6 | 2,073 | 2,073 | +| **ISONGraph Python (exact)** | 27.9 | 0.000 | 27.9 | - | 32.8 | 1,424 | 764 | +| **ISONGraph Python (IVF)** | 10.1 | 11.8 | 21.9 | - | 22.0 | 1,444 | 1,634 | +| **ISONGraph Rust (exact)** | 0.786 | 0.000 | 0.786 | - | 1.45 | 1,345 | 1,353 | +| **ISONGraph Rust (IVF)** | 0.786 | 5.69 | 7.14 | - | 7.14 | 1,356 | 1,422 | +| **ISONGraph Rust, target-cpu=native (exact)** | 0.750 | 0.000 | 0.750 | - | 1.41 | 1,345 | 1,353 | +| **ISONGraph Rust, target-cpu=native (IVF)** | 0.750 | 5.69 | 7.10 | - | 7.10 | 1,356 | 1,422 | +| faiss IndexFlatIP | 0.375 | 0.000 | 0.375 | - | 0.404 | 470 | 605 | +| faiss HNSWFlat | 60.1 | 0.000 | 60.1 | - | 60.2 | 546 | 666 | +| faiss IVFFlat | 0.000 | 11.9 | 11.9 | - | 11.9 | 534 | 983 | +| hnswlib | 51.0 | 0.000 | 51.0 | - | 51.0 | 749 | 757 | +| lancedb (no index) | 0.801 | 0.000 | 0.801 | - | 0.958 | 107 | 124 | +| lancedb IVF_FLAT | 0.765 | 8.92 | 9.68 | - | 9.96 | 555 | 866 | +| numpy brute force | 0.122 | 0.000 | 0.122 | - | 0.149 | 493 | 500 | +| usearch (HNSW) | 105 | 0.000 | 105 | - | 105 | 631 | 639 | + +
Full accuracy sweeps (recall@10, single-thread QPS, latency, all-core QPS) + +**ISONGraph C++, /arch:AVX2 (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 2.58 (2.58-2.58) | 387 | 405 | 407 | - | 30 | + +**ISONGraph C++, /arch:AVX2 (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2955 | 7425 (7276-7598) | 0.137 | 0.190 | 0.229 | - | 2000 | +| 2 | 0.4190 | 6023 (5907-6306) | 0.165 | 0.232 | 0.292 | - | 2000 | +| 4 | 0.5330 | 4520 (4502-4652) | 0.218 | 0.318 | 0.386 | - | 2000 | +| 8 | 0.6322 | 2864 (2804-3030) | 0.339 | 0.494 | 0.556 | - | 2000 | +| 16 | 0.7206 | 1825 (1797-1840) | 0.538 | 0.785 | 0.872 | - | 2000 | +| 32 | 0.7965 | 1111 (1061-1130) | 0.890 | 1.32 | 1.50 | - | 2000 | +| 64 | 0.8616 | 610 (601-633) | 1.60 | 2.33 | 2.64 | - | 1991 | +| 128 | 0.9203 | 325 (323-343) | 3.02 | 4.14 | 4.55 | - | 943 | +| 256 | 0.9527 | 176 (170-183) | 5.63 | 7.38 | 8.03 | - | 552 | + +**ISONGraph C++ (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 2.48 (2.48-2.48) | 406 | 415 | 420 | - | 30 | + +**ISONGraph C++ (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2955 | 6471 (6388-6694) | 0.153 | 0.198 | 0.223 | - | 2000 | +| 2 | 0.4190 | 5429 (5367-5468) | 0.183 | 0.237 | 0.260 | - | 2000 | +| 4 | 0.5330 | 4324 (4301-4344) | 0.228 | 0.309 | 0.346 | - | 2000 | +| 8 | 0.6322 | 2985 (2982-3000) | 0.327 | 0.459 | 0.511 | - | 2000 | +| 16 | 0.7206 | 1908 (1818-1972) | 0.514 | 0.745 | 0.848 | - | 2000 | +| 32 | 0.7965 | 1086 (1065-1086) | 0.905 | 1.29 | 1.43 | - | 2000 | +| 64 | 0.8636 | 589 (588-594) | 1.67 | 2.30 | 2.51 | - | 1797 | +| 128 | 0.9202 | 322 (313-326) | 3.10 | 4.07 | 4.39 | - | 924 | +| 256 | 0.9508 | 172 (167-175) | 5.80 | 7.43 | 8.02 | - | 494 | + +**ISONGraph C# (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 5.81 (5.81-5.81) | 126 | 370 | 399 | - | 30 | + +**ISONGraph C# (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2969 | 2483 (2139-2665) | 0.411 | 0.500 | 0.945 | - | 2000 | +| 2 | 0.4143 | 2367 (2353-2403) | 0.431 | 0.501 | 0.540 | - | 2000 | +| 4 | 0.5305 | 2036 (2021-2060) | 0.499 | 0.601 | 0.652 | - | 2000 | +| 8 | 0.6330 | 1558 (1547-1575) | 0.637 | 0.814 | 0.892 | - | 2000 | +| 16 | 0.7188 | 1108 (1092-1115) | 0.893 | 1.19 | 1.32 | - | 2000 | +| 32 | 0.7955 | 713 (707-719) | 1.38 | 1.88 | 2.04 | - | 2000 | +| 64 | 0.8603 | 423 (417-425) | 2.35 | 3.13 | 3.45 | - | 1483 | +| 128 | 0.9160 | 248 (246-273) | 3.86 | 5.20 | 5.63 | - | 674 | +| 256 | 0.9485 | 131 (129-145) | 7.37 | 9.61 | 10.5 | - | 336 | + +**ISONGraph Python (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 26.5 (26.2-27.9) | 37.1 | 42.5 | 47.8 | - | 84 | + +**ISONGraph Python (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.3014 | 7447 (7436-7674) | 0.127 | 0.183 | 0.227 | - | 2000 | +| 2 | 0.4195 | 5473 (5429-5495) | 0.170 | 0.246 | 0.320 | - | 2000 | +| 4 | 0.5359 | 3762 (3631-3937) | 0.248 | 0.360 | 0.445 | - | 2000 | +| 8 | 0.6390 | 2438 (2412-2542) | 0.383 | 0.570 | 0.689 | - | 2000 | +| 16 | 0.7264 | 1503 (1497-1512) | 0.636 | 0.942 | 1.10 | - | 2000 | +| 32 | 0.7996 | 881 (854-891) | 1.10 | 1.58 | 1.90 | - | 2000 | +| 64 | 0.8634 | 492 (492-498) | 1.98 | 2.70 | 3.07 | - | 1635 | +| 128 | 0.9207 | 255 (251-256) | 3.78 | 5.02 | 11.5 | - | 774 | +| 256 | 0.9574 | 138 (137-140) | 6.95 | 9.14 | 14.9 | - | 418 | +| 4000 | 1.0000 | 34.6 (32.8-35.7) | 28.0 | 35.0 | 53.5 | - | 111 | + +**ISONGraph Rust (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 43.5 (42.8-48.6) | 22.5 | 24.6 | 25.9 | - | 170 | + +**ISONGraph Rust (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.3088 | 10525 (10503-10566) | 0.092 | 0.134 | 0.167 | 5474 † | 2000 | +| 2 | 0.4230 | 7216 (6962-7478) | 0.135 | 0.197 | 0.241 | 4107 † | 2000 | +| 4 | 0.5377 | 5054 (5004-5225) | 0.192 | 0.282 | 0.345 | 2611 † | 2000 | +| 8 | 0.6346 | 3277 (3218-3329) | 0.297 | 0.441 | 0.522 | 1647 † | 2000 | +| 16 | 0.7284 | 1923 (1908-1938) | 0.508 | 0.751 | 0.857 | 1028 † | 2000 | +| 32 | 0.8013 | 1168 (1096-1174) | 0.854 | 1.21 | 1.39 | 564 † | 2000 | +| 64 | 0.8640 | 644 (588-644) | 1.58 | 2.19 | 2.47 | 363 † | 1587 | +| 128 | 0.9204 | 314 (312-321) | 3.19 | 4.24 | 4.63 | 273 † | 921 | +| 256 | 0.9537 | 169 (162-170) | 6.01 | 7.67 | 8.42 | 166 † | 475 | + +**ISONGraph Rust, target-cpu=native (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 40.1 (38.9-40.2) | 25.7 | 26.9 | 28.1 | - | 159 | + +**ISONGraph Rust, target-cpu=native (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.3088 | 9787 (9759-10042) | 0.099 | 0.140 | 0.165 | 5548 † | 2000 | +| 2 | 0.4230 | 7586 (7559-7781) | 0.129 | 0.178 | 0.211 | 3924 † | 2000 | +| 4 | 0.5377 | 5136 (5043-5309) | 0.190 | 0.270 | 0.322 | 2675 † | 2000 | +| 8 | 0.6346 | 3220 (3142-3377) | 0.300 | 0.441 | 0.520 | 1596 † | 2000 | +| 16 | 0.7284 | 1903 (1795-1965) | 0.519 | 0.765 | 0.875 | 1049 † | 2000 | +| 32 | 0.8013 | 1010 (1005-1016) | 0.971 | 1.39 | 1.54 | 634 † | 2000 | +| 64 | 0.8626 | 576 (554-581) | 1.73 | 2.39 | 2.64 | 336 † | 1848 | +| 128 | 0.9166 | 325 (308-344) | 3.05 | 4.08 | 4.53 | 292 † | 1032 | +| 256 | 0.9544 | 157 (156-158) | 6.36 | 8.01 | 8.54 | 164 † | 548 | + +**faiss IndexFlatIP** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 40.7 (39.8-42.6) | 24.2 | 27.1 | 33.2 | 639 | 149 | + +**faiss HNSWFlat** (efSearch) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.4765 | 22888 (18723-24051) | 0.041 | 0.066 | 0.086 | 166,477 | 2000 | +| 16 | 0.5605 | 17882 (14705-18039) | 0.052 | 0.083 | 0.117 | 134,983 | 2000 | +| 24 | 0.6299 | 13430 (13067-13770) | 0.071 | 0.108 | 0.134 | 103,151 | 2000 | +| 32 | 0.6724 | 10956 (10906-11267) | 0.086 | 0.129 | 0.157 | 73210 | 2000 | +| 48 | 0.7281 | 7966 (7703-8175) | 0.118 | 0.179 | 0.224 | 57418 | 2000 | +| 64 | 0.7658 | 6097 (6026-6304) | 0.151 | 0.224 | 0.295 | 48203 | 2000 | +| 96 | 0.8093 | 4504 (4414-4594) | 0.213 | 0.301 | 0.391 | 33683 | 2000 | +| 128 | 0.8340 | 3283 (3280-3489) | 0.282 | 0.392 | 0.503 | 27685 | 2000 | +| 192 | 0.8646 | 2297 (2103-2329) | 0.419 | 0.551 | 0.717 | 19009 | 2000 | +| 256 | 0.8850 | 1782 (1671-1796) | 0.553 | 0.701 | 0.980 | 14590 | 2000 | +| 400 | 0.9119 | 1164 (1152-1232) | 0.803 | 1.16 | 1.87 | 10161 | 2000 | +| 640 | 0.9347 | 653 (650-654) | 1.52 | 1.79 | 2.02 | 6535 | 1892 | + +**faiss IVFFlat** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.3046 | 19282 (18048-19407) | 0.050 | 0.071 | 0.091 | 170,065 | 2000 | +| 2 | 0.4281 | 15642 (13255-16241) | 0.061 | 0.091 | 0.135 | 117,092 | 2000 | +| 4 | 0.5371 | 12355 (11603-12379) | 0.078 | 0.106 | 0.134 | 83273 | 2000 | +| 8 | 0.6348 | 8461 (8222-8862) | 0.112 | 0.163 | 0.216 | 49956 | 2000 | +| 16 | 0.7203 | 5748 (5576-5760) | 0.170 | 0.239 | 0.288 | 26400 | 2000 | +| 32 | 0.7933 | 3221 (3128-3365) | 0.286 | 0.406 | 0.527 | 14622 | 2000 | +| 64 | 0.8588 | 1796 (1793-1819) | 0.522 | 0.721 | 0.882 | 7627 | 2000 | +| 128 | 0.9123 | 989 (977-997) | 0.973 | 1.28 | 1.64 | 3993 | 2000 | +| 256 | 0.9488 | 525 (519-534) | 1.85 | 2.34 | 3.19 | 2064 | 1649 | +| 4000 | 1.0000 | 43.8 (43.0-43.8) | 22.2 | 28.9 | 31.4 | 159 | 136 | + +**hnswlib** (ef) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.4829 | 27767 (24854-28305) | 0.035 | 0.057 | 0.075 | 141,005 | 2000 | +| 16 | 0.5657 | 22942 (21474-23347) | 0.042 | 0.065 | 0.079 | 105,029 | 2000 | +| 24 | 0.6309 | 17335 (15586-17788) | 0.054 | 0.080 | 0.097 | 82464 | 2000 | +| 32 | 0.6724 | 13832 (13632-14170) | 0.067 | 0.101 | 0.125 | 67729 | 2000 | +| 48 | 0.7266 | 10275 (10124-10618) | 0.090 | 0.127 | 0.158 | 54141 | 2000 | +| 64 | 0.7621 | 8681 (8081-8779) | 0.111 | 0.154 | 0.189 | 44182 | 2000 | +| 96 | 0.8022 | 6005 (5703-6227) | 0.158 | 0.224 | 0.449 | 31593 | 2000 | +| 128 | 0.8283 | 4941 (4752-5057) | 0.198 | 0.260 | 0.311 | 25814 | 2000 | +| 192 | 0.8605 | 3292 (2942-3346) | 0.289 | 0.384 | 0.822 | 17875 | 2000 | +| 256 | 0.8807 | 2481 (2466-2493) | 0.380 | 0.539 | 1.12 | 10936 | 2000 | +| 400 | 0.9119 | 1660 (1584-1692) | 0.585 | 0.832 | 1.32 | 8093 | 2000 | +| 640 | 0.9367 | 1086 (1071-1101) | 0.893 | 1.24 | 1.82 | 5013 | 2000 | + +**lancedb (no index)** (nprobes) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 2.89 (2.89-2.89) | 346 | 385 | 393 | - | 30 | + +**lancedb IVF_FLAT** (nprobes) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2826 | 232 (165-257) | 4.14 | 7.38 | 9.20 | - | 710 | +| 2 | 0.3981 | 212 (158-234) | 4.48 | 7.54 | 10.2 | - | 499 | +| 4 | 0.5174 | 260 (220-261) | 3.81 | 6.06 | 8.14 | - | 787 | +| 8 | 0.6209 | 166 (159-166) | 6.23 | 7.77 | 8.86 | - | 638 | +| 16 | 0.7151 | 168 (167-177) | 5.81 | 7.56 | 8.25 | - | 481 | +| 32 | 0.7928 | 166 (160-171) | 5.93 | 7.75 | 8.37 | - | 557 | +| 64 | 0.8552 | 150 (146-152) | 6.70 | 8.27 | 9.45 | - | 384 | +| 128 | 0.9126 | 133 (124-134) | 7.72 | 9.20 | 11.3 | - | 338 | +| 256 | 0.9513 | 109 (107-111) | 9.32 | 10.9 | 11.4 | - | 323 | +| 4000 | 1.0000 | 20.9 (20.2-21.2) | 48.0 | 52.1 | 54.9 | - | 59 | + +**numpy brute force** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 34.4 (33.3-36.6) | 28.8 | 33.2 | 36.0 | 69.2 | 121 | + +**usearch (HNSW)** (expansion_search) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.4733 | 10942 (10420-11107) | 0.087 | 0.135 | 0.163 | 60767 | 2000 | +| 16 | 0.5670 | 7685 (6804-8557) | 0.124 | 0.194 | 0.240 | 54147 | 2000 | +| 24 | 0.6345 | 6417 (6102-6460) | 0.150 | 0.230 | 0.281 | 45841 | 2000 | +| 32 | 0.6755 | 5292 (5219-5339) | 0.180 | 0.271 | 0.333 | 32467 | 2000 | +| 48 | 0.7307 | 3777 (3637-3788) | 0.248 | 0.369 | 0.470 | 22370 | 2000 | +| 64 | 0.7647 | 2879 (2823-2890) | 0.340 | 0.512 | 0.646 | 19236 | 2000 | +| 96 | 0.8055 | 2037 (1988-2059) | 0.484 | 0.710 | 0.853 | 13595 | 2000 | +| 128 | 0.8315 | 2194 (1636-2462) | 0.443 | 0.798 | 1.14 | 11891 | 2000 | +| 192 | 0.8649 | 1400 (1137-1461) | 0.712 | 1.15 | 1.61 | 16330 | 2000 | +| 256 | 0.8839 | 1041 (1019-1070) | 0.908 | 1.36 | 2.44 | 12443 | 2000 | +| 400 | 0.9121 | 716 (712-726) | 1.33 | 1.91 | 3.53 | 9088 | 2000 | +| 640 | 0.9394 | 437 (404-451) | 2.21 | 3.61 | 5.66 | 2503 | 1353 | + +
+ +### sift, N = 1,000,000 + +Queries/second at recall@10 = 0.90 / 0.95 / 0.99, log-linearly interpolated between the two measured sweep points that bracket the target (no extrapolation; `*` = the two bracketing points are more than 0.05 recall apart, so the measured point above the target is shown instead, a conservative value; `-` = never reached; exact scans appear at their single recall-1.0 point). 1T = one pinned P-core, one query at a time. All-core = batched call on 32 logical CPUs (`†` = the ISONGraph ports have no batch API, so this is the same one-query loop with automatic scan threads). Latency columns are for the first measured sweep point with recall >= 0.95. + +| system | QPS 1T @0.90 | @0.95 | @0.99 | all-core QPS @0.90 | @0.95 | 1T latency ms median / p95 / p99 | +|---|---|---|---|---|---|---| +| **ISONGraph C++ (exact)** | 2.19 | 2.19 | 2.19 | - | - | 459 / 472 / 474 (recall 1.000) | +| **ISONGraph C++ (IVF)** | 979* | 562* | 314 | - | - | 1.75 / 2.54 / 2.83 (recall 0.970) | +| **ISONGraph Rust (exact)** | 33.4 | 33.4 | 33.4 | - | - | 30.3 / 31.9 / 32.3 (recall 0.999) | +| **ISONGraph Rust (IVF)** | 1078* | 546* | 314 | 600* † | 382* † | 1.76 / 2.62 / 2.89 (recall 0.970) | +| faiss HNSWFlat | 8350 | 5603 | 2175 | 56008 | 37075 | 0.190 / 0.255 / 0.324 (recall 0.957) | +| faiss IVFFlat | 2209* | 1270* | 714 | 7504* | 3254* | 0.761 / 1.14 / 1.30 (recall 0.970) | +| hnswlib | 10660 | 7311 | 3266 | 51989 | 37313 | 0.151 / 0.193 / 0.226 (recall 0.958) | +| numpy brute force | 26.9 | 26.9 | 26.9 | 67.4 | 67.4 | 37.4 / 41.8 / 46.8 (recall 0.998) | + +Build and memory (RSS growth over the process baseline after imports; the raw vectors are streamed in from disk and are not held by the harness). + +| system | ingest s | index s | total build s | 3 builds: median (min-max) s | ready s (incl. first query) | steady RSS +MB | peak RSS +MB (build) | +|---|---|---|---|---|---|---|---| +| **ISONGraph C++ (exact)** | 0.615 | 0.000 | 0.615 | - | 0.944 | 783 | 922 | +| **ISONGraph C++ (IVF)** | 0.615 | 10.9 | 11.5 | - | 11.5 | 1,299 | 1,377 | +| **ISONGraph Rust (exact)** | 0.827 | 0.000 | 0.827 | - | 1.67 | 1,560 | 1,567 | +| **ISONGraph Rust (IVF)** | 0.827 | 6.37 | 8.04 | - | 8.04 | 1,570 | 1,654 | +| faiss HNSWFlat | 99.7 | 0.000 | 99.7 | - | 99.7 | 663 | 814 | +| faiss IVFFlat | 0.000 | 26.8 | 26.8 | - | 26.9 | 684 | 1,258 | +| hnswlib | 60.4 | 0.000 | 60.4 | - | 60.4 | 854 | 864 | +| numpy brute force | 0.137 | 0.000 | 0.137 | - | 0.166 | 599 | 609 | + +
Full accuracy sweeps (recall@10, single-thread QPS, latency, all-core QPS) + +**ISONGraph C++ (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 2.19 (2.19-2.19) | 459 | 472 | 474 | - | 30 | + +**ISONGraph C++ (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2549 | 5716 (5606-5905) | 0.174 | 0.222 | 0.277 | - | 2000 | +| 2 | 0.3890 | 4864 (4720-4877) | 0.204 | 0.259 | 0.301 | - | 2000 | +| 4 | 0.5391 | 3921 (3791-3942) | 0.251 | 0.344 | 0.417 | - | 2000 | +| 8 | 0.6881 | 2732 (2601-2735) | 0.356 | 0.526 | 0.640 | - | 2000 | +| 16 | 0.8212 | 1686 (1661-1687) | 0.566 | 0.879 | 1.05 | - | 2000 | +| 32 | 0.9147 | 979 (944-989) | 0.984 | 1.51 | 1.74 | - | 2000 | +| 64 | 0.9700 | 562 (525-579) | 1.75 | 2.54 | 2.83 | - | 1522 | +| 128 | 0.9918 | 297 (292-302) | 3.30 | 4.47 | 5.00 | - | 870 | +| 256 | 0.9978 | 155 (154-174) | 6.18 | 7.79 | 8.36 | - | 411 | + +**ISONGraph Rust (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 0.9990 | 33.4 (32.8-33.9) | 30.3 | 31.9 | 32.3 | - | 105 | + +**ISONGraph Rust (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2633 | 10887 (10198-10960) | 0.092 | 0.122 | 0.149 | 6064 † | 2000 | +| 2 | 0.3933 | 7937 (7581-8142) | 0.123 | 0.168 | 0.205 | 4494 † | 2000 | +| 4 | 0.5365 | 5763 (5718-5832) | 0.165 | 0.245 | 0.308 | 3000 † | 2000 | +| 8 | 0.6876 | 3349 (3312-3668) | 0.277 | 0.442 | 0.545 | 1974 † | 2000 | +| 16 | 0.8216 | 2089 (2066-2111) | 0.453 | 0.734 | 0.862 | 1224 † | 2000 | +| 32 | 0.9163 | 1078 (1046-1088) | 0.891 | 1.41 | 1.60 | 600 † | 2000 | +| 64 | 0.9701 | 546 (541-565) | 1.76 | 2.62 | 2.89 | 382 † | 1518 | +| 128 | 0.9924 | 294 (294-295) | 3.34 | 4.57 | 5.02 | 290 † | 1014 | +| 256 | 0.9988 | 157 (155-162) | 6.33 | 7.87 | 8.50 | 239 † | 502 | + +**faiss HNSWFlat** (efSearch) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.6877 | 17412 (15222-17554) | 0.055 | 0.092 | 0.165 | 135,453 | 2000 | +| 16 | 0.7802 | 13831 (12997-13850) | 0.070 | 0.104 | 0.201 | 92820 | 2000 | +| 24 | 0.8484 | 11228 (11083-12040) | 0.085 | 0.124 | 0.169 | 72882 | 2000 | +| 32 | 0.8891 | 9050 (8957-9248) | 0.107 | 0.153 | 0.248 | 60222 | 2000 | +| 48 | 0.9339 | 6490 (6449-7113) | 0.148 | 0.201 | 0.275 | 44633 | 2000 | +| 64 | 0.9573 | 5243 (5201-5347) | 0.190 | 0.255 | 0.324 | 34095 | 2000 | +| 96 | 0.9778 | 3716 (3678-4041) | 0.265 | 0.344 | 0.410 | 24544 | 2000 | +| 128 | 0.9866 | 2622 (2368-2680) | 0.390 | 0.512 | 0.609 | 16650 | 2000 | +| 192 | 0.9936 | 1790 (1707-1800) | 0.584 | 0.723 | 0.835 | 11858 | 2000 | +| 256 | 0.9962 | 1384 (1364-1415) | 0.742 | 0.918 | 1.03 | 8866 | 2000 | +| 400 | 0.9984 | 809 (803-836) | 1.27 | 1.50 | 1.65 | 5686 | 2000 | +| 640 | 0.9990 | 482 (482-488) | 2.14 | 2.51 | 2.66 | 3978 | 1572 | + +**faiss IVFFlat** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2635 | 9853 (9769-10714) | 0.097 | 0.138 | 0.178 | 47015 | 2000 | +| 2 | 0.3934 | 9352 (9319-9919) | 0.107 | 0.146 | 0.202 | 53015 | 2000 | +| 4 | 0.5409 | 7229 (6865-7675) | 0.135 | 0.197 | 0.253 | 38314 | 2000 | +| 8 | 0.6917 | 5382 (4759-5567) | 0.187 | 0.280 | 0.349 | 23059 | 2000 | +| 16 | 0.8229 | 3576 (3541-3728) | 0.267 | 0.409 | 0.483 | 13783 | 2000 | +| 32 | 0.9173 | 2209 (2168-2230) | 0.434 | 0.664 | 0.777 | 7504 | 2000 | +| 64 | 0.9700 | 1270 (1266-1270) | 0.761 | 1.14 | 1.30 | 3254 | 2000 | +| 128 | 0.9923 | 668 (663-678) | 1.46 | 2.02 | 2.26 | 2326 | 2000 | +| 256 | 0.9984 | 358 (346-363) | 2.80 | 3.63 | 4.04 | 1115 | 1035 | +| 4000 | 0.9988 | 27.4 (26.9-28.1) | 36.2 | 39.0 | 40.8 | 79.8 | 86 | + +**hnswlib** (ef) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.6795 | 22475 (21929-22477) | 0.044 | 0.062 | 0.073 | 80773 | 2000 | +| 16 | 0.7752 | 17160 (15591-17310) | 0.058 | 0.088 | 0.111 | 72843 | 2000 | +| 24 | 0.8472 | 13361 (13153-14935) | 0.073 | 0.098 | 0.115 | 52927 | 2000 | +| 32 | 0.8900 | 11346 (11208-11457) | 0.089 | 0.117 | 0.136 | 55624 | 2000 | +| 48 | 0.9339 | 8629 (8446-8866) | 0.117 | 0.155 | 0.180 | 41344 | 2000 | +| 64 | 0.9576 | 6765 (6690-6876) | 0.151 | 0.193 | 0.226 | 35561 | 2000 | +| 96 | 0.9768 | 4866 (4847-4880) | 0.210 | 0.267 | 0.312 | 25048 | 2000 | +| 128 | 0.9862 | 3957 (3729-4163) | 0.257 | 0.328 | 0.374 | 24450 | 2000 | +| 192 | 0.9937 | 2717 (2703-2840) | 0.374 | 0.468 | 0.540 | 14951 | 2000 | +| 256 | 0.9966 | 2159 (1950-2229) | 0.488 | 0.627 | 0.725 | 11572 | 2000 | +| 400 | 0.9987 | 1475 (1412-1490) | 0.705 | 0.896 | 0.994 | 8898 | 2000 | +| 640 | 0.9993 | 973 (950-976) | 1.06 | 1.35 | 1.50 | 5333 | 2000 | + +**numpy brute force** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 0.9982 | 26.9 (26.7-28.1) | 37.4 | 41.8 | 46.8 | 67.4 | 110 | + +
+ + + +### Exact search latency (10k / 100k / 1M) + +`median ms (1T)` is the pinned single-thread loop; `all-threads single query` lets the library use every core for one query at a time; `batch QPS` is a batched call on 32 logical CPUs. + +#### text384 + +| N | system | median ms (1T) | p95 | p99 | QPS 1T (min-max) | all-threads single query | batch QPS | recall | first query ms | +|---|---|---|---|---|---|---|---|---|---| +| 10,000 | **ISONGraph C++, /arch:AVX2 (exact)** | 3.08 | 3.48 | 3.84 | 328 (308-328) | - | - | 1.000 | 3.58 | +| 10,000 | **ISONGraph C++ (exact)** | 3.23 | 3.63 | 4.30 | 311 (289-311) | - | - | 1.000 | 4.08 | +| 10,000 | **ISONGraph C# (exact)** | 2.33 | 2.76 | 4.29 | 422 (382-423) | - | - | 1.000 | 24.2 | +| 10,000 | **ISONGraph Python (exact)** | 0.421 | 0.503 | 0.873 | 2206 (2202-2337) | 0.283 ms (3334 QPS) | - | 1.000 | 32.4 | +| 10,000 | **ISONGraph JS (exact)** | 5.00 | 5.64 | 6.04 | 198 (192-198) | - | - | 1.000 | 5.88 | +| 10,000 | **ISONGraph Rust (exact)** | 0.273 | 0.310 | 0.363 | 3728 (3617-3796) | 0.258 ms (3896 QPS) | - | 1.000 | 7.19 | +| 10,000 | **ISONGraph Rust, target-cpu=native (exact)** | 0.274 | 0.308 | 0.351 | 3617 (3469-3630) | 0.261 ms (3836 QPS) | - | 1.000 | 7.38 | +| 10,000 | faiss IndexFlatIP | 0.336 | 0.441 | 0.590 | 2845 (2753-2860) | 3.48 ms (293 QPS) | 55693 | 1.000 | 0.873 | +| 10,000 | lancedb (no index) | 16.0 | 18.9 | 23.2 | 60.9 (60.4-61.1) | 13.3 ms (74.6 QPS) | - | 1.000 | 16.3 | +| 10,000 | numpy brute force | 0.304 | 0.350 | 0.453 | 3364 (3126-3518) | 0.151 ms (6581 QPS) | 14690 | 1.000 | 0.842 | +| 10,000 | qdrant-client local mode (exact numpy) | 13.2 | 15.9 | 22.1 | 73.0 (72.4-73.4) | 12.5 ms (79.2 QPS) | - | 1.000 | 39.9 | +| 10,000 | sqlite-vec (vec0, exact) | 9.36 | 11.2 | 14.0 | 104 (103-105) | 9.57 ms (103 QPS) | - | 1.000 | 9.76 | +| 100,000 | **ISONGraph C++, /arch:AVX2 (exact)** | 43.5 | 45.4 | 47.6 | 23.0 (23.0-23.1) | - | - | 1.000 | 43.8 | +| 100,000 | **ISONGraph C++ (exact)** | 44.2 | 47.2 | 57.7 | 22.6 (22.5-23.3) | - | - | 1.000 | 45.0 | +| 100,000 | **ISONGraph C# (exact)** | 23.6 | 26.7 | 78.4 | 39.1 (38.6-39.9) | - | - | 1.000 | 46.9 | +| 100,000 | **ISONGraph Python (exact)** | 7.37 | 8.92 | 14.3 | 131 (129-131) | 5.30 ms (183 QPS) | - | 1.000 | 303 | +| 100,000 | **ISONGraph JS (exact)** | 50.8 | 55.8 | 58.1 | 19.7 (19.1-20.3) | - | - | 1.000 | 65.7 | +| 100,000 | **ISONGraph Rust (exact)** | 6.41 | 6.99 | 7.86 | 154 (154-154) | 3.21 ms (299 QPS) | - | 1.000 | 80.0 | +| 100,000 | **ISONGraph Rust, target-cpu=native (exact)** | 6.06 | 6.79 | 7.39 | 163 (161-164) | 3.24 ms (293 QPS) | - | 1.000 | 79.1 | +| 100,000 | faiss IndexFlatIP | 7.29 | 8.84 | 15.0 | 131 (130-135) | 4.97 ms (198 QPS) | 5437 | 1.000 | 8.38 | +| 100,000 | lancedb (no index) | 137 | 151 | 173 | 7.19 (7.16-7.22) | 72.1 ms (13.7 QPS) | - | 1.000 | 64.9 | +| 100,000 | numpy brute force | 6.75 | 7.66 | 10.1 | 144 (143-146) | 4.54 ms (217 QPS) | 1553 | 1.000 | 7.36 | +| 100,000 | qdrant-client local mode (exact numpy) | 136 | 152 | 158 | 7.24 (7.14-7.25) | 132 ms (7.55 QPS) | - | 1.000 | 127 | +| 100,000 | sqlite-vec (vec0, exact) | 95.5 | 115 | 122 | 10.2 (10.2-10.3) | 95.3 ms (10.4 QPS) | - | 1.000 | 98.5 | +| 1,000,000 | **ISONGraph C++, /arch:AVX2 (exact)** | 466 | 491 | 493 | 2.13 (2.13-2.13) | - | - | 1.000 | 510 | +| 1,000,000 | **ISONGraph C++ (exact)** | 469 | 500 | 516 | 2.11 (2.11-2.11) | - | - | 1.000 | 509 | +| 1,000,000 | **ISONGraph C# (exact)** | 245 | 394 | 398 | 3.72 (3.72-3.72) | - | - | 1.000 | 299 | +| 1,000,000 | **ISONGraph Python (exact)** | 77.2 | 86.1 | 93.0 | 12.7 (12.7-12.8) | 46.6 ms (21.3 QPS) | - | 1.000 | 3159 | +| 1,000,000 | **ISONGraph JS (exact)** | 545 | 838 | 872 | 1.67 (1.67-1.67) | - | - | 1.000 | 558 | +| 1,000,000 | **ISONGraph Rust (exact)** | 65.3 | 70.0 | 80.5 | 15.1 (15.1-15.1) | 28.7 ms (34.2 QPS) | - | 1.000 | 1030 | +| 1,000,000 | **ISONGraph Rust, target-cpu=native (exact)** | 62.7 | 68.9 | 71.3 | 15.8 (15.6-15.8) | 30.9 ms (31.9 QPS) | - | 1.000 | 1070 | +| 1,000,000 | faiss IndexFlatIP | 73.4 | 83.8 | 87.6 | 13.4 (13.3-13.5) | 43.1 ms (22.8 QPS) | 540 | 1.000 | 77.6 | +| 1,000,000 | lancedb (no index) | 1372 | 1438 | 1463 | 0.725 (0.725-0.725) | 576 ms (1.73 QPS) | - | 1.000 | 654 | +| 1,000,000 | numpy brute force | 69.3 | 77.1 | 84.0 | 14.2 (14.1-14.4) | 35.1 ms (28.0 QPS) | 142 | 1.000 | 70.3 | +| 1,000,000 | sqlite-vec (vec0, exact) | 947 | 988 | 991 | 1.05 (1.05-1.05) | 938 ms (1.06 QPS) | - | 1.000 | 943 | + +#### glove + +| N | system | median ms (1T) | p95 | p99 | QPS 1T (min-max) | all-threads single query | batch QPS | recall | first query ms | +|---|---|---|---|---|---|---|---|---|---| +| 100,000 | **ISONGraph C++, /arch:AVX2 (exact)** | 24.4 | 28.5 | 31.7 | 40.2 (39.7-41.2) | - | - | 1.000 | 27.1 | +| 100,000 | **ISONGraph C++ (exact)** | 24.5 | 28.4 | 29.7 | 40.1 (39.7-40.1) | - | - | 1.000 | 25.5 | +| 100,000 | **ISONGraph C# (exact)** | 7.79 | 26.4 | 35.6 | 108 (107-108) | - | - | 1.000 | 18.3 | +| 100,000 | **ISONGraph Python (exact)** | 1.98 | 2.72 | 4.89 | 471 (468-487) | 1.28 ms (734 QPS) | - | 1.000 | 179 | +| 100,000 | **ISONGraph JS (exact)** | 20.1 | 24.6 | 27.1 | 48.1 (47.8-48.9) | - | - | 1.000 | 24.7 | +| 100,000 | **ISONGraph Rust (exact)** | 1.42 | 1.79 | 2.14 | 700 (660-705) | 0.578 ms (1670 QPS) | - | 1.000 | 40.8 | +| 100,000 | **ISONGraph Rust, target-cpu=native (exact)** | 1.57 | 1.94 | 2.25 | 621 (616-627) | 0.611 ms (1594 QPS) | - | 1.000 | 41.8 | +| 100,000 | faiss IndexFlatIP | 1.72 | 2.23 | 3.01 | 548 (543-570) | 0.413 ms (2335 QPS) | 16430 | 1.000 | 2.19 | +| 100,000 | lancedb (no index) | 29.5 | 34.2 | 37.4 | 34.3 (31.2-34.5) | 11.4 ms (86.5 QPS) | - | 1.000 | 21.9 | +| 100,000 | numpy brute force | 1.82 | 2.53 | 3.70 | 528 (479-537) | 0.907 ms (1018 QPS) | 1733 | 1.000 | 2.37 | +| 100,000 | sqlite-vec (vec0, exact) | 24.3 | 27.4 | 34.3 | 40.1 (40.0-40.4) | 24.5 ms (40.5 QPS) | - | 1.000 | 25.3 | +| 1,000,000 | **ISONGraph C++, /arch:AVX2 (exact)** | 387 | 405 | 407 | 2.58 (2.58-2.58) | - | - | 1.000 | 316 | +| 1,000,000 | **ISONGraph C++ (exact)** | 406 | 415 | 420 | 2.48 (2.48-2.48) | - | - | 1.000 | 281 | +| 1,000,000 | **ISONGraph C# (exact)** | 126 | 370 | 399 | 5.81 (5.81-5.81) | - | - | 1.000 | 123 | +| 1,000,000 | **ISONGraph Python (exact)** | 37.1 | 42.5 | 47.8 | 26.5 (26.2-27.9) | 35.6 ms (27.7 QPS) | - | 1.000 | 4900 | +| 1,000,000 | **ISONGraph Rust (exact)** | 22.5 | 24.6 | 25.9 | 43.5 (42.8-48.6) | 13.3 ms (70.9 QPS) | - | 1.000 | 664 | +| 1,000,000 | **ISONGraph Rust, target-cpu=native (exact)** | 25.7 | 26.9 | 28.1 | 40.1 (38.9-40.2) | 12.8 ms (68.8 QPS) | - | 1.000 | 658 | +| 1,000,000 | faiss IndexFlatIP | 24.2 | 27.1 | 33.2 | 40.7 (39.8-42.6) | 20.4 ms (49.7 QPS) | 639 | 1.000 | 28.8 | +| 1,000,000 | lancedb (no index) | 346 | 385 | 393 | 2.89 (2.89-2.89) | 105 ms (9.45 QPS) | - | 1.000 | 157 | +| 1,000,000 | numpy brute force | 28.8 | 33.2 | 36.0 | 34.4 (33.3-36.6) | 24.7 ms (39.4 QPS) | 69.2 | 1.000 | 26.9 | + +#### sift + +| N | system | median ms (1T) | p95 | p99 | QPS 1T (min-max) | all-threads single query | batch QPS | recall | first query ms | +|---|---|---|---|---|---|---|---|---|---| +| 100,000 | **ISONGraph C++, /arch:AVX2 (exact)** | 42.3 | 45.4 | 47.4 | 23.4 (23.3-27.3) | - | - | 1.000 | 29.6 | +| 100,000 | **ISONGraph C++ (exact)** | 44.2 | 47.5 | 49.5 | 22.8 (22.4-25.2) | - | - | 1.000 | 31.6 | +| 100,000 | **ISONGraph C# (exact)** | 14.6 | 42.8 | 80.8 | 55.0 (55.0-72.9) | - | - | 1.000 | 22.9 | +| 100,000 | **ISONGraph Python (exact)** | 3.22 | 4.42 | 5.68 | 318 (265-323) | 3.99 ms (222 QPS) | - | 1.000 | 218 | +| 100,000 | **ISONGraph JS (exact)** | 33.7 | 41.6 | 48.7 | 29.0 (27.6-32.5) | - | - | 1.000 | 28.8 | +| 100,000 | **ISONGraph Rust (exact)** | 2.86 | 3.36 | 3.84 | 344 (333-398) | 2.86 ms (336 QPS) | - | 1.000 | 50.6 | +| 100,000 | **ISONGraph Rust, target-cpu=native (exact)** | 2.62 | 3.14 | 3.42 | 364 (362-465) | 2.88 ms (328 QPS) | - | 1.000 | 58.6 | +| 100,000 | faiss IndexFlatIP | 3.30 | 4.26 | 6.04 | 283 (280-332) | 3.44 ms (285 QPS) | 4537 | 1.000 | 3.51 | +| 100,000 | lancedb (no index) | 66.9 | 78.5 | 82.0 | 15.1 (13.7-16.0) | 46.0 ms (21.1 QPS) | - | 1.000 | 33.7 | +| 100,000 | numpy brute force | 3.34 | 4.87 | 6.69 | 282 (267-297) | 2.15 ms (428 QPS) | 1543 | 1.000 | 3.02 | +| 100,000 | sqlite-vec (vec0, exact) | 47.0 | 52.6 | 57.7 | 21.6 (21.0-22.4) | 89.9 ms (11.0 QPS) | - | 1.000 | 32.2 | +| 1,000,000 | **ISONGraph C++ (exact)** | 459 | 472 | 474 | 2.19 (2.19-2.19) | - | - | 1.000 | 330 | +| 1,000,000 | **ISONGraph Rust (exact)** | 30.3 | 31.9 | 32.3 | 33.4 (32.8-33.9) | 16.2 ms (61.3 QPS) | - | 0.999 | 841 | +| 1,000,000 | numpy brute force | 37.4 | 41.8 | 46.8 | 26.9 (26.7-28.1) | 32.0 ms (31.0 QPS) | 67.4 | 0.998 | 29.7 | + + + +### Streaming mutation (insert 10k / update 10k / delete 10k on the built index) + +Batches of 1,000 through each system's own API. ISONGraph rows include index maintenance; numpy/faiss Flat have no index to maintain. `recall before` is 0.000 where no sweep point reached 0.95 (the highest nprobe was used). + +| dataset | N | system | insert 10k | update 10k | delete 10k | recall@10 before | after | first query after ms | +|---|---|---|---|---|---|---|---|---| +| glove | 100,000 | **ISONGraph C++, /arch:AVX2 (IVF)** | 0.012 s (851,332/s) | 0.006 s (1,540,002/s) | 0.004 s (2,662,903/s) | 0.974 | 0.973 | 19.9 | +| glove | 100,000 | **ISONGraph C++ (IVF)** | 0.013 s (797,925/s) | 0.007 s (1,488,427/s) | 0.004 s (2,524,870/s) | 0.974 | 0.973 | 22.0 | +| glove | 100,000 | **ISONGraph C# (IVF)** | 0.049 s (202,219/s) | 0.008 s (1,294,080/s) | 0.003 s (2,900,316/s) | 0.973 | 0.972 | 33.1 | +| glove | 100,000 | **ISONGraph Python (exact)** | 0.130 s (77055/s) | 0.126 s (79264/s) | 0.116 s (86540/s) | 1.00 | 1.000 | 3.41 | +| glove | 100,000 | **ISONGraph Python (IVF)** | 0.137 s (73067/s) | 0.140 s (71216/s) | 0.118 s (84936/s) | 0.973 | 0.972 | 78.5 | +| glove | 100,000 | **ISONGraph JS (IVF)** | 0.018 s (550,973/s) | 0.009 s (1,170,412/s) | 0.006 s (1,739,615/s) | 0.975 | 0.974 | 1174 | +| glove | 100,000 | **ISONGraph Rust (IVF)** | 0.015 s (689,579/s) | 0.014 s (702,020/s) | 0.009 s (1,127,218/s) | 0.974 | 0.974 | 83.7 | +| glove | 100,000 | **ISONGraph Rust, target-cpu=native (IVF)** | 0.014 s (708,567/s) | 0.014 s (732,220/s) | 0.009 s (1,092,383/s) | 0.974 | 0.974 | 83.6 | +| glove | 100,000 | chromadb (HNSW) | 2.42 s (4125/s) | 3.46 s (2892/s) | 1.95 s (5121/s) | 0.977 | 0.976 | 89.0 | +| glove | 100,000 | faiss IndexFlatIP | 0.002 s (4,972,403/s) | 0.102 s (97716/s) | 0.099 s (101,166/s) | 1.00 | 1.000 | 2.72 | +| glove | 100,000 | faiss HNSWFlat | 0.989 s (10112/s) | unsupported | unsupported | 0.970 | n/a | - | +| glove | 100,000 | faiss IVFFlat | 0.015 s (668,717/s) | 0.017 s (600,017/s) | 0.002 s (5,319,149/s) | 0.974 | 0.974 | 1.32 | +| glove | 100,000 | hnswlib | 0.226 s (44217/s) | 0.934 s (10703/s) | 0.008 s (1,242,329/s) | 0.969 | 0.969 | 0.644 | +| glove | 100,000 | lancedb (no index) | 0.029 s (349,167/s) | 0.134 s (74457/s) | 0.108 s (92786/s) | 1.00 | 1.000 | 20.8 | +| glove | 100,000 | lancedb IVF_FLAT | 0.031 s (319,488/s) | 0.138 s (72668/s) | 0.103 s (97019/s) | 0.974 | 0.977 | 9.27 | +| glove | 100,000 | numpy brute force | 0.001 s (8,836,264/s) | 0.006 s (1,624,643/s) | 0.002 s (4,615,314/s) | 1.00 | 1.000 | 2.93 | +| glove | 100,000 | sqlite-vec (vec0, exact) | 0.080 s (124,613/s) | 2.47 s (4042/s) | 2.07 s (4838/s) | 1.00 | 1.000 | 28.8 | +| glove | 100,000 | usearch (HNSW) | 0.261 s (38339/s) | 0.281 s (35606/s) | 0.003 s (3,713,882/s) | 0.967 | 0.964 | 3.67 | +| glove | 1,000,000 | **ISONGraph C++, /arch:AVX2 (IVF)** | 0.159 s (63000/s) | 0.018 s (571,167/s) | 0.013 s (783,687/s) | 0.953 | 0.951 | 86.5 | +| glove | 1,000,000 | **ISONGraph C++ (IVF)** | 0.155 s (64395/s) | 0.016 s (620,409/s) | 0.012 s (803,006/s) | 0.951 | 0.951 | 92.5 | +| glove | 1,000,000 | **ISONGraph C# (IVF)** | 0.029 s (350,733/s) | 0.016 s (616,922/s) | 0.039 s (258,405/s) | 0.000 | 0.949 | 136 | +| glove | 1,000,000 | **ISONGraph Python (exact)** | 0.881 s (11357/s) | 0.602 s (16617/s) | 0.509 s (19659/s) | 1.00 | 1.000 | 100 | +| glove | 1,000,000 | **ISONGraph Python (IVF)** | 0.167 s (59736/s) | 0.156 s (64253/s) | 0.148 s (67731/s) | 0.957 | 0.957 | 294 | +| glove | 1,000,000 | **ISONGraph Rust (IVF)** | 0.156 s (64014/s) | 0.042 s (238,538/s) | 0.033 s (303,255/s) | 0.954 | 0.954 | 746 | +| glove | 1,000,000 | **ISONGraph Rust, target-cpu=native (IVF)** | 0.164 s (61106/s) | 0.039 s (253,556/s) | 0.034 s (297,217/s) | 0.954 | 0.953 | 750 | +| glove | 1,000,000 | faiss IndexFlatIP | 0.004 s (2,325,527/s) | 3.94 s (2536/s) | 4.19 s (2387/s) | 1.00 | 1.000 | 65.7 | +| glove | 1,000,000 | faiss HNSWFlat | 1.79 s (5594/s) | unsupported | unsupported | 0.935 | n/a | - | +| glove | 1,000,000 | faiss IVFFlat | 0.044 s (225,461/s) | 0.083 s (119,972/s) | 0.005 s (2,143,898/s) | 1.00 | 1.000 | 31.1 | +| glove | 1,000,000 | hnswlib | 1.07 s (9386/s) | 2.66 s (3766/s) | 0.015 s (671,768/s) | 0.937 | 0.935 | 2.27 | +| glove | 1,000,000 | lancedb (no index) | 0.070 s (141,898/s) | 1.50 s (6686/s) | 1.40 s (7153/s) | 1.00 | 1.000 | 273 | +| glove | 1,000,000 | lancedb IVF_FLAT | 0.105 s (95045/s) | 1.29 s (7723/s) | 1.27 s (7897/s) | 0.951 | 0.939 | 41.9 | +| glove | 1,000,000 | numpy brute force | 0.004 s (2,843,494/s) | 0.151 s (66231/s) | 0.024 s (419,856/s) | 1.00 | 1.000 | 68.9 | +| glove | 1,000,000 | usearch (HNSW) | 1.37 s (7313/s) | 1.49 s (6723/s) | 0.004 s (2,399,635/s) | 0.939 | 0.939 | 25.1 | +| sift | 100,000 | **ISONGraph C++, /arch:AVX2 (IVF)** | 0.030 s (333,745/s) | 0.011 s (876,263/s) | 0.006 s (1,582,804/s) | 0.952 | 0.949 | 30.4 | +| sift | 100,000 | **ISONGraph C++ (IVF)** | 0.025 s (400,699/s) | 0.012 s (835,939/s) | 0.007 s (1,374,004/s) | 0.952 | 0.949 | 37.1 | +| sift | 100,000 | **ISONGraph C# (IVF)** | 0.030 s (332,383/s) | 0.015 s (655,476/s) | 0.009 s (1,169,946/s) | 0.953 | 0.949 | 51.8 | +| sift | 100,000 | **ISONGraph Python (exact)** | 0.810 s (12346/s) | 0.676 s (14783/s) | 0.426 s (23490/s) | 1.00 | 1.000 | 10.1 | +| sift | 100,000 | **ISONGraph Python (IVF)** | 0.151 s (66209/s) | 0.157 s (63624/s) | 0.135 s (74040/s) | 0.987 | 0.986 | 96.4 | +| sift | 100,000 | **ISONGraph JS (IVF)** | 0.044 s (225,056/s) | 0.012 s (845,101/s) | 0.008 s (1,271,407/s) | 0.988 | 0.987 | 2043 | +| sift | 100,000 | **ISONGraph Rust (IVF)** | 0.039 s (253,924/s) | 0.033 s (304,480/s) | 0.022 s (463,420/s) | 0.986 | 0.985 | 261 | +| sift | 100,000 | **ISONGraph Rust, target-cpu=native (IVF)** | 0.028 s (354,949/s) | 0.029 s (342,604/s) | 0.020 s (506,183/s) | 0.986 | 0.985 | 277 | +| sift | 100,000 | chromadb (HNSW) | 2.33 s (4284/s) | 3.34 s (2991/s) | 2.33 s (4296/s) | 0.952 | 0.942 | 111 | +| sift | 100,000 | faiss IndexFlatIP | 0.005 s (2,211,215/s) | 0.520 s (19235/s) | 0.361 s (27688/s) | 1.00 | 1.000 | 7.96 | +| sift | 100,000 | faiss HNSWFlat | 0.995 s (10053/s) | unsupported | unsupported | 0.953 | n/a | - | +| sift | 100,000 | faiss IVFFlat | 0.035 s (284,106/s) | 0.063 s (157,573/s) | 0.004 s (2,790,957/s) | 0.950 | 0.947 | 0.438 | +| sift | 100,000 | hnswlib | 0.238 s (42037/s) | 0.570 s (17557/s) | 0.008 s (1,250,750/s) | 0.950 | 0.941 | 0.101 | +| sift | 100,000 | lancedb (no index) | 0.120 s (83312/s) | 0.366 s (27354/s) | 0.249 s (40241/s) | 1.00 | 1.000 | 77.4 | +| sift | 100,000 | lancedb IVF_FLAT | 0.040 s (252,179/s) | 0.149 s (67089/s) | 0.120 s (83050/s) | 0.987 | 0.989 | 12.1 | +| sift | 100,000 | numpy brute force | 0.002 s (6,123,324/s) | 0.006 s (1,562,402/s) | 0.003 s (3,611,412/s) | 1.00 | 1.000 | 3.07 | +| sift | 100,000 | sqlite-vec (vec0, exact) | 0.307 s (32571/s) | 11.0 s (912/s) | 8.30 s (1205/s) | 1.00 | 1.000 | 109 | +| sift | 100,000 | usearch (HNSW) | 0.216 s (46402/s) | 0.258 s (38686/s) | 0.003 s (3,433,948/s) | 0.952 | 0.945 | 2.84 | +| text384 | 100,000 | **ISONGraph C++, /arch:AVX2 (IVF)** | 0.038 s (262,907/s) | 0.014 s (708,602/s) | 0.005 s (2,130,243/s) | 0.954 | 0.951 | 43.8 | +| text384 | 100,000 | **ISONGraph C++ (IVF)** | 0.038 s (265,111/s) | 0.013 s (764,427/s) | 0.004 s (2,329,482/s) | 0.954 | 0.950 | 64.3 | +| text384 | 100,000 | **ISONGraph C# (IVF)** | 0.026 s (386,698/s) | 0.009 s (1,148,805/s) | 0.004 s (2,422,891/s) | 0.952 | 0.949 | 44.8 | +| text384 | 100,000 | **ISONGraph Python (exact)** | 0.206 s (48551/s) | 0.195 s (51276/s) | 0.128 s (78119/s) | 1.000 | 1.000 | 9.15 | +| text384 | 100,000 | **ISONGraph Python (IVF)** | 0.211 s (47387/s) | 0.204 s (49074/s) | 0.126 s (79668/s) | 0.954 | 0.948 | 203 | +| text384 | 100,000 | **ISONGraph JS (IVF)** | 0.063 s (158,336/s) | 0.014 s (729,001/s) | 0.008 s (1,216,382/s) | 0.953 | 0.952 | 3284 | +| text384 | 100,000 | **ISONGraph Rust (IVF)** | 0.044 s (228,710/s) | 0.017 s (579,979/s) | 0.010 s (1,037,086/s) | 0.953 | 0.949 | 192 | +| text384 | 100,000 | **ISONGraph Rust, target-cpu=native (IVF)** | 0.039 s (256,465/s) | 0.016 s (606,160/s) | 0.010 s (1,032,983/s) | 0.953 | 0.949 | 192 | +| text384 | 100,000 | chromadb (HNSW) | 3.49 s (2866/s) | 5.23 s (1912/s) | 1.98 s (5063/s) | 0.965 | 0.965 | 171 | +| text384 | 100,000 | faiss IndexFlatIP | 0.005 s (1,992,310/s) | 0.225 s (44475/s) | 0.196 s (51042/s) | 1.000 | 1.000 | 9.09 | +| text384 | 100,000 | faiss HNSWFlat | 2.84 s (3524/s) | unsupported | unsupported | 0.957 | n/a | - | +| text384 | 100,000 | faiss IVFFlat | 0.049 s (202,182/s) | 0.047 s (211,541/s) | 0.002 s (4,849,425/s) | 0.955 | 0.951 | 1.23 | +| text384 | 100,000 | hnswlib | 1.11 s (9049/s) | 2.97 s (3372/s) | 0.011 s (876,186/s) | 0.964 | 0.968 | 0.687 | +| text384 | 100,000 | lancedb (no index) | 0.053 s (190,268/s) | 0.165 s (60618/s) | 0.111 s (89958/s) | 1.00 | 1.000 | 103 | +| text384 | 100,000 | lancedb IVF_FLAT | 0.056 s (178,739/s) | 0.207 s (48245/s) | 0.135 s (74346/s) | 0.956 | 0.969 | 15.5 | +| text384 | 100,000 | numpy brute force | 0.003 s (3,145,346/s) | 0.007 s (1,341,526/s) | 0.002 s (4,841,208/s) | 1.000 | 1.000 | 7.10 | +| text384 | 100,000 | qdrant-client local mode (exact numpy) | 25.7 s (390/s) | 26.4 s (379/s) | 31.8 s (315/s) | 1.00 | 1.000 | 136 | +| text384 | 100,000 | sqlite-vec (vec0, exact) | 0.170 s (58976/s) | 9.85 s (1016/s) | 7.37 s (1356/s) | 1.00 | 1.000 | 110 | +| text384 | 100,000 | usearch (HNSW) | 1.18 s (8452/s) | 1.19 s (8434/s) | 0.003 s (3,381,920/s) | 0.964 | 0.958 | 3.58 | +| text384 | 1,000,000 | **ISONGraph C++, /arch:AVX2 (IVF)** | 0.288 s (34735/s) | 0.016 s (615,483/s) | 0.007 s (1,382,361/s) | 0.955 | 0.956 | 81.2 | +| text384 | 1,000,000 | **ISONGraph C++ (IVF)** | 0.292 s (34211/s) | 0.018 s (569,256/s) | 0.007 s (1,378,607/s) | 0.953 | 0.953 | 149 | +| text384 | 1,000,000 | **ISONGraph C# (IVF)** | 0.015 s (677,952/s) | 0.009 s (1,124,227/s) | 0.006 s (1,552,650/s) | 0.000 | 0.946 | 111 | +| text384 | 1,000,000 | **ISONGraph Python (exact)** | 0.284 s (35183/s) | 0.212 s (47127/s) | 0.148 s (67540/s) | 1.00 | 1.000 | 76.1 | +| text384 | 1,000,000 | **ISONGraph Python (IVF)** | 0.269 s (37230/s) | 0.228 s (43813/s) | 0.163 s (61533/s) | 0.953 | 0.954 | 665 | +| text384 | 1,000,000 | **ISONGraph JS (IVF)** | 0.017 s (589,240/s) | 0.014 s (703,408/s) | 0.011 s (935,559/s) | 0.000 | 0.946 | 5743 | +| text384 | 1,000,000 | **ISONGraph Rust (IVF)** | 0.283 s (35306/s) | 0.023 s (428,754/s) | 0.017 s (591,195/s) | 0.951 | 0.952 | 611 | +| text384 | 1,000,000 | **ISONGraph Rust, target-cpu=native (IVF)** | 0.277 s (36069/s) | 0.021 s (479,409/s) | 0.015 s (645,561/s) | 0.952 | 0.954 | 602 | +| text384 | 1,000,000 | chromadb (HNSW) | 12.7 s (789/s) | 14.4 s (694/s) | 10.7 s (937/s) | 0.967 | 0.968 | 2631 | +| text384 | 1,000,000 | faiss IndexFlatIP | 0.007 s (1,357,958/s) | 2.31 s (4331/s) | 2.25 s (4445/s) | 1.00 | 1.000 | 72.1 | +| text384 | 1,000,000 | faiss HNSWFlat | 4.32 s (2315/s) | unsupported | unsupported | 0.965 | n/a | - | +| text384 | 1,000,000 | faiss IVFFlat | 0.125 s (79926/s) | 0.139 s (71689/s) | 0.005 s (2,116,626/s) | 0.956 | 0.957 | 8.73 | +| text384 | 1,000,000 | hnswlib | 2.09 s (4774/s) | 3.09 s (3236/s) | 0.009 s (1,138,771/s) | 0.957 | 0.956 | 0.775 | +| text384 | 1,000,000 | lancedb (no index) | 0.056 s (179,892/s) | 0.635 s (15755/s) | 0.358 s (27946/s) | 1.00 | 1.000 | 940 | +| text384 | 1,000,000 | lancedb IVF_FLAT | 0.055 s (181,022/s) | 0.603 s (16597/s) | 0.356 s (28076/s) | 1.00 | 1.000 | 108 | +| text384 | 1,000,000 | numpy brute force | 0.004 s (2,584,647/s) | 0.087 s (115,119/s) | 0.015 s (670,628/s) | 1.00 | 1.000 | 73.8 | +| text384 | 1,000,000 | sqlite-vec (vec0, exact) | 0.194 s (51559/s) | 10.5 s (949/s) | 8.07 s (1240/s) | 1.00 | 1.000 | 958 | +| text384 | 1,000,000 | usearch (HNSW) | 2.25 s (4440/s) | 2.37 s (4225/s) | 0.005 s (2,170,940/s) | 0.957 | 0.956 | 23.9 | + + +### Filtered search (random 10% and 1% subsets) + +#### glove, N = 100,000 + +| system | build with metadata s | 10% subset: QPS 1T @ recall>=0.90 | max recall | 1% subset: QPS 1T @ recall>=0.90 | max recall | +|---|---|---|---|---|---| +| **ISONGraph C++ (IVF)** | 0.354 | 152 @ 1.000 | 1.000 | 316 @ 1.000 | 1.000 | +| **ISONGraph Python (exact)** | 0.952 | 368 @ 1.000 | 1.000 | 409 @ 1.000 | 1.000 | +| **ISONGraph Python (IVF)** | 1.78 | 366 @ 1.000 | 1.000 | 400 @ 1.000 | 1.000 | +| **ISONGraph Rust (IVF)** | 2.13 | 8377 @ 1.000 | 1.000 | 18699 @ 1.000 | 1.000 | +| chromadb (HNSW) | 11.1 | 4.23 @ 0.912 | 1.000 | 28.0 @ 0.936 | 1.000 | +| faiss IndexFlatIP | 0.036 | 1813 @ 1.000 | 1.000 | 2913 @ 1.000 | 1.000 | +| faiss HNSWFlat | 3.85 | 534 @ 0.957 | 0.957 | - (max recall 0.835 at 541 QPS) | 0.835 | +| faiss IVFFlat | 1.03 | 1083 @ 1.000 | 1.000 | 1910 @ 1.000 | 1.000 | +| hnswlib | 1.31 | 1354 @ 0.935 | 1.000 | 449 @ 0.951 | 1.000 | +| lancedb (no index) | 0.081 | 33.8 @ 1.000 | 1.000 | 37.6 @ 1.000 | 1.000 | +| lancedb IVF_FLAT | 0.740 | 88.3 @ 1.000 | 1.000 | 99.0 @ 1.000 | 1.000 | +| numpy brute force | 0.012 | 552 @ 1.000 | 1.000 | 624 @ 1.000 | 1.000 | +| sqlite-vec (vec0, exact) | 0.579 | 46.1 @ 1.000 | 1.000 | 48.0 @ 1.000 | 1.000 | +| usearch (HNSW) | 1.57 | unsupported | - | unsupported | - | + +#### text384, N = 100,000 + +| system | build with metadata s | 10% subset: QPS 1T @ recall>=0.90 | max recall | 1% subset: QPS 1T @ recall>=0.90 | max recall | +|---|---|---|---|---|---| +| **ISONGraph C++ (IVF)** | 0.986 | 116 @ 1.000 | 1.000 | 345 @ 1.000 | 1.000 | +| **ISONGraph Python (exact)** | 1.97 | 121 @ 1.000 | 1.000 | 131 @ 1.000 | 1.000 | +| **ISONGraph Python (IVF)** | 3.62 | 122 @ 1.000 | 1.000 | 130 @ 1.000 | 1.000 | +| **ISONGraph Rust (IVF)** | 4.77 | 3094 @ 1.000 | 1.000 | 15288 @ 1.000 | 1.000 | +| chromadb (HNSW) | 16.5 | 2.24 @ 0.953 | 1.000 | 20.9 @ 0.971 | 1.000 | +| faiss IndexFlatIP | 0.109 | 464 @ 1.000 | 1.000 | 2491 @ 1.000 | 1.000 | +| faiss HNSWFlat | 13.7 | 1279 @ 0.944 | 0.992 | 377 @ 0.928 | 0.928 | +| faiss IVFFlat | 1.69 | 388 @ 1.000 | 1.000 | 1630 @ 1.000 | 1.000 | +| hnswlib | 6.44 | 1779 @ 0.912 | 1.000 | 259 @ 0.981 | 1.000 | +| lancedb (no index) | 0.217 | 7.09 @ 1.000 | 1.000 | 7.28 @ 1.000 | 1.000 | +| lancedb IVF_FLAT | 1.94 | 73.1 @ 1.000 | 1.000 | 97.8 @ 1.000 | 1.000 | +| numpy brute force | 0.030 | 141 @ 1.000 | 1.000 | 146 @ 1.000 | 1.000 | +| sqlite-vec (vec0, exact) | 1.01 | 13.0 @ 1.000 | 1.000 | 13.3 @ 1.000 | 1.000 | +| usearch (HNSW) | 5.68 | unsupported | - | unsupported | - | + +#### text384, N = 1,000,000 + +| system | build with metadata s | 10% subset: QPS 1T @ recall>=0.90 | max recall | 1% subset: QPS 1T @ recall>=0.90 | max recall | +|---|---|---|---|---|---| +| **ISONGraph C++ (IVF)** | 12.6 | 308 @ 0.943 | 1.000 | 506 @ 0.905 | 1.000 | +| **ISONGraph Python (IVF)** | 117 | 8.64 @ 1.000 | 1.000 | 9.52 @ 1.000 | 1.000 | +| **ISONGraph Rust (IVF)** | 69.5 | 650 @ 0.940 | 1.000 | 1944 @ 0.906 | 1.000 | +| faiss IndexFlatIP | 1.13 | 39.7 @ 1.000 | 1.000 | 172 @ 1.000 | 1.000 | +| faiss HNSWFlat | 268 | 267 @ 0.973 | 0.973 | - (max recall 0.854 at 275 QPS) | 0.854 | +| faiss IVFFlat | 47.0 | 38.1 @ 1.000 | 1.000 | 140 @ 1.000 | 1.000 | +| hnswlib | 151 | 406 @ 0.955 | 0.999 | 135 @ 0.945 | 1.000 | +| lancedb IVF_FLAT | 25.3 | 8.73 @ 1.000 | 1.000 | 13.1 @ 1.000 | 1.000 | +| numpy brute force | 0.300 | 12.9 @ 1.000 | 1.000 | 14.0 @ 1.000 | 1.000 | + + + +### Save / load and on-disk size + +| dataset | N | system | save s | on-disk MB | load s | first query after load ms | time to ready s | +|---|---|---|---|---|---|---|---| +| glove | 100,000 | **ISONGraph C++, /arch:AVX2 (IVF)** | 4.26 | 133 | 2.02 | 236 | 2.25 | +| glove | 100,000 | **ISONGraph C++ (IVF)** | 4.40 | 133 | 2.03 | 302 | 2.33 | +| glove | 100,000 | **ISONGraph C# (IVF)** | 1.36 | 121 | 1.27 | 333 | 1.61 | +| glove | 100,000 | **ISONGraph Python (exact)** | 4.35 | 215 | 3.37 | 173 | 3.54 | +| glove | 100,000 | **ISONGraph Python (IVF)** | 4.42 | 215 | 3.41 | 2017 | 5.43 | +| glove | 100,000 | **ISONGraph Rust (IVF)** | 0.754 | 121 | 0.414 | 237 | 0.651 | +| glove | 100,000 | chromadb (HNSW) | 0.000 | 76.3 | 0.012 | 87.3 | 0.099 | +| glove | 100,000 | faiss IndexFlatIP | 0.045 | 38.9 | 0.025 | 2.13 | 0.027 | +| glove | 100,000 | faiss HNSWFlat | 0.052 | 57.1 | 0.029 | 1.18 | 0.031 | +| glove | 100,000 | faiss IVFFlat | 0.020 | 39.4 | 0.027 | 0.713 | 0.028 | +| glove | 100,000 | hnswlib | 0.036 | 57.5 | 0.142 | 0.708 | 0.142 | +| glove | 100,000 | lancedb (no index) | 0.000 | 46.8 | 0.002 | 16.9 | 0.018 | +| glove | 100,000 | lancedb IVF_FLAT | 0.000 | 86.1 | 0.002 | 26.8 | 0.028 | +| glove | 100,000 | numpy brute force | 0.025 | 42.8 | 0.017 | 2.38 | 0.020 | +| glove | 100,000 | sqlite-vec (vec0, exact) | 0.000 | 48.9 | 0.000 | 30.7 | 0.031 | +| glove | 100,000 | usearch (HNSW) | 0.097 | 57.5 | 0.052 | 1.56 | 0.053 | +| glove | 1,000,000 | **ISONGraph Rust (IVF)** | 23.1 | 1216 | 10.8 | 7592 | 18.3 | +| glove | 1,000,000 | faiss IndexFlatIP | 0.655 | 389 | 0.973 | 66.8 | 1.04 | +| glove | 1,000,000 | faiss HNSWFlat | 0.307 | 524 | 0.228 | 2.15 | 0.230 | +| glove | 1,000,000 | faiss IVFFlat | 0.149 | 391 | 0.246 | 23.8 | 0.270 | +| glove | 1,000,000 | hnswlib | 0.777 | 528 | 3.66 | 3.27 | 3.66 | +| glove | 1,000,000 | usearch (HNSW) | 0.585 | 528 | 0.499 | 3.37 | 0.503 | +| sift | 100,000 | **ISONGraph C++, /arch:AVX2 (IVF)** | 12.2 | 134 | 5.14 | 636 | 5.78 | +| sift | 100,000 | **ISONGraph C++ (IVF)** | 12.4 | 134 | 4.97 | 826 | 5.80 | +| sift | 100,000 | **ISONGraph C# (IVF)** | 3.51 | 122 | 3.06 | 1133 | 4.20 | +| sift | 100,000 | **ISONGraph Python (exact)** | 11.0 | 222 | 9.70 | 528 | 10.2 | +| sift | 100,000 | **ISONGraph Python (IVF)** | 4.97 | 222 | 3.66 | 2380 | 6.04 | +| sift | 100,000 | **ISONGraph Rust (IVF)** | 1.67 | 128 | 0.904 | 853 | 1.76 | +| sift | 100,000 | chromadb (HNSW) | 0.000 | 88.5 | 0.015 | 108 | 0.123 | +| sift | 100,000 | faiss IndexFlatIP | 0.060 | 49.6 | 0.087 | 12.9 | 0.100 | +| sift | 100,000 | faiss HNSWFlat | 0.044 | 68.9 | 0.039 | 0.340 | 0.039 | +| sift | 100,000 | faiss IVFFlat | 0.025 | 50.2 | 0.035 | 0.216 | 0.036 | +| sift | 100,000 | hnswlib | 0.067 | 69.3 | 0.158 | 0.214 | 0.158 | +| sift | 100,000 | lancedb (no index) | 0.000 | 59.6 | 0.005 | 74.0 | 0.079 | +| sift | 100,000 | lancedb IVF_FLAT | 0.000 | 110 | 0.008 | 18.6 | 0.027 | +| sift | 100,000 | numpy brute force | 0.031 | 54.6 | 0.026 | 4.71 | 0.030 | +| sift | 100,000 | sqlite-vec (vec0, exact) | 0.001 | 61.8 | 0.001 | 105 | 0.106 | +| sift | 100,000 | usearch (HNSW) | 0.080 | 69.3 | 0.061 | 0.200 | 0.062 | +| sift | 1,000,000 | **ISONGraph Rust (IVF)** | 22.7 | 1285 | 11.2 | 8390 | 19.6 | +| sift | 1,000,000 | faiss HNSWFlat | 1.12 | 626 | 1.10 | 0.567 | 1.10 | +| sift | 1,000,000 | faiss IVFFlat | 0.353 | 498 | 0.723 | 1.63 | 0.725 | +| sift | 1,000,000 | hnswlib | 1.39 | 630 | 4.90 | 0.936 | 4.90 | +| text384 | 100,000 | **ISONGraph C++, /arch:AVX2 (IVF)** | 16.4 | 500 | 6.95 | 531 | 7.48 | +| text384 | 100,000 | **ISONGraph C++ (IVF)** | 16.5 | 500 | 7.34 | 903 | 8.24 | +| text384 | 100,000 | **ISONGraph C# (IVF)** | 4.41 | 453 | 3.95 | 911 | 4.86 | +| text384 | 100,000 | **ISONGraph Python (exact)** | 15.5 | 807 | 10.8 | 291 | 11.1 | +| text384 | 100,000 | **ISONGraph Python (IVF)** | 16.0 | 807 | 10.9 | 5129 | 16.0 | +| text384 | 100,000 | **ISONGraph Rust (IVF)** | 1.94 | 453 | 1.25 | 546 | 1.80 | +| text384 | 100,000 | chromadb (HNSW) | 0.000 | 202 | 0.012 | 175 | 0.187 | +| text384 | 100,000 | faiss IndexFlatIP | 0.128 | 147 | 0.069 | 7.27 | 0.076 | +| text384 | 100,000 | faiss HNSWFlat | 0.134 | 176 | 0.074 | 0.496 | 0.075 | +| text384 | 100,000 | faiss IVFFlat | 0.055 | 149 | 0.091 | 1.29 | 0.092 | +| text384 | 100,000 | hnswlib | 0.147 | 177 | 0.225 | 0.615 | 0.225 | +| text384 | 100,000 | lancedb (no index) | 0.000 | 177 | 0.003 | 110 | 0.113 | +| text384 | 100,000 | lancedb IVF_FLAT | 0.000 | 326 | 0.009 | 23.8 | 0.033 | +| text384 | 100,000 | numpy brute force | 0.110 | 162 | 0.068 | 9.16 | 0.077 | +| text384 | 100,000 | sqlite-vec (vec0, exact) | 0.000 | 180 | 0.000 | 109 | 0.110 | +| text384 | 100,000 | usearch (HNSW) | 0.178 | 177 | 0.136 | 0.949 | 0.137 | +| text384 | 1,000,000 | **ISONGraph Rust (IVF)** | 21.5 | 4534 | 13.0 | 6601 | 19.6 | +| text384 | 1,000,000 | faiss IndexFlatIP | 0.968 | 1472 | 0.736 | 74.1 | 0.811 | +| text384 | 1,000,000 | faiss HNSWFlat | 0.994 | 1618 | 0.805 | 1.21 | 0.806 | +| text384 | 1,000,000 | faiss IVFFlat | 0.417 | 1478 | 0.670 | 6.22 | 0.676 | +| text384 | 1,000,000 | hnswlib | 1.07 | 1623 | 2.12 | 1.58 | 2.12 | +| text384 | 1,000,000 | usearch (HNSW) | 1.54 | 1623 | 1.30 | 1.72 | 1.30 | + + +### Jobs that failed or timed out + +None in the final run. + + +## Library issues found (described, not fixed) + +1. **C++ exact search has no scan cache and is scalar** (`isongraph-vector-cpp/include/isongraph_vector.hpp`, `similaritySearchVector`, the loop over `embeddings_` that calls `cosineSimilarity` at ~line 1246/1672). It iterates an `unordered_map` of records, and for each one runs a scalar loop with three dependent accumulators and recomputes both vector norms. Measured: 22.6 QPS at 100k x 384 (numpy 144, Rust 154), 2.1 QPS at 1M. `nprobe >= nlist` falls into the same path, which is why the last point of every C++ IVF sweep collapses to exact speed. Suggested fix: reuse the contiguous vector copy the IVF index already keeps (or build one lazily like the Rust `ScanCache`), cache inverse norms, and score with the existing `simd::dot` kernels. +2. **No batched or concurrent search in any port.** Rust `search_vector` (`isongraph-vector-rs/src/ann.rs` ~line 1722) holds the `scan` mutex for the whole scan; C++ takes `ann_.mutex`; C# `lock (_lock)`; the Python store holds an RLock. The `concurrent_clients` probe in the Rust/C++ JSON shows N client threads do not scale. When a large probe is spread over threads, Rust spawns fresh threads per query (`std::thread::scope` in `scan_probed`). Suggested fix: search against an immutable snapshot (`Arc` / read lock) and add `search_batch(&[Query])` backed by a persistent pool; this alone is worth 10-50x on the all-core column. +3. **First search after writes pays a lazy rebuild/fold.** After 10k inserts + 10k updates + 10k deletes on a 1M text index the first query took 0.6 s (Rust), 0.15 s (C++), 0.11 s (C#), 0.67 s (Python), 5.7 s (JS) versus ~1 ms for hnswlib/usearch (`first_query_after_ms` in the mutation table). Suggested fix: fold pending rows incrementally at write time or on a background thread. +4. **JS IVF build is 522 s at 1M** (Rust 7.6 s, C++ 12 s default / 6 s AVX2, C# 11 s): single-threaded k-means over plain JS arrays of doubles; the JS port also needs ~7 GB for 1M x 384. +5. **Persistence is the JSON sidecar only; the IVF index is never persisted.** 100k x 384 = 0.45-0.8 GB, 1M = 4.5 GB; load then rebuilds the index. A binary vector file (and an index file) would close most of the 20-100x gap to faiss/hnswlib. +6. **Two copies of every vector** (record map + contiguous cache/index copy) give 2-4x the RSS of faiss/hnswlib at 1M. +7. Build flags matter for C++: the default bench build is SSE2; `/arch:AVX2` makes the IVF 1.1-1.4x faster (text 100k 681 -> 957 at 0.95). Rust `target-cpu=native` made no measurable difference to the IVF (828 vs 828) and +5% to the exact scan. + +## What did not run, and why + +* **Qdrant server (HNSW)**: not run. Docker Desktop's daemon was not running, and downloading and executing the official Windows binary was not permitted in this session. Only `qdrant-client` **local mode** ran, which is a pure-numpy brute-force scan (the client itself warns against more than 20,000 points). It was run at 10k and 100k only (100k ingest alone took 284 s) and is not a Qdrant performance number. +* **chromadb build repeats** after the first build failed ("Collection already exists": chromadb 1.5.9 shares one system object per process). The `3 builds` column for chromadb therefore has 1-2 samples at most; everything else for chromadb ran. chromadb ignores a changed `ef_search` until the index is reloaded, so the harness reloads the collection for every knob value (verified separately). +* **chromadb at 1M** ran for text384 only; **sqlite-vec at 1M** for text384 only (and 100k for all three datasets); **qdrant local** only text384 10k and 100k; **JS at 1M** text384 only (a glove/sift 1M JS run would add 30 min each). +* **SIFT 1M** ran for a reduced set only (ISONGraph Rust and C++, faiss HNSW, faiss IVFFlat, hnswlib, numpy brute force) because of the time budget; no streaming-mutation test is possible at SIFT 1M (the file has exactly 1,000,000 vectors, no spare rows for inserts). SIFT 100k ran for every system. +* **usearch filtered search**: the Python binding has no filter predicate: recorded as `unsupported`. hnswlib filtering is a Python callback per visited node, which is the only API it has. +* **faiss HNSW update/delete**: unsupported by that index type (recorded as `unsupported`; it would need a rebuild). +* **Streaming mutation for ISONGraph "exact" and numpy/faiss Flat** measures only cache patching, which is not an index; the like-for-like comparisons are the IVF/HNSW rows. hnswlib `delete` is `mark_deleted` (memory is not freed). +* **Save/load**: not run for ISONGraph at 1M in Python, C++, C#, JS (JSON of 4.5 GB at 1M; Python's `json.dumps` would need tens of GB). Rust did run it. JS has no save/load call for vectors. +* **Cross-process repeat of the pinned run** (second independent process per headline cell) was not done for the final data set because of the time budget; in-process 3 runs per point and the Rust spot-check are the only repeatability evidence (see above). +* **numpy BLAS** is the PyPI wheel (OpenBLAS, `MAX_THREADS=24`), so "all-core" numpy uses up to 24 threads, not 32. + +## Unfairness that could not be removed + +* **Language/runtime**: Python-callable libraries pay Python call overhead per query (20-100 us; invisible for ms-scale queries, material for the 0.3 ms HNSW queries); ISONGraph Python additionally builds result objects. Rust/C++ harnesses call the library directly. +* **SIMD**: faiss, numpy/OpenBLAS and usearch dispatch AVX2/AVX-512 at run time; hnswlib's wheel and our Rust/C++/C#/JS builds do not necessarily. We report both baseline and `target-cpu=native` (Rust) / `/arch:AVX2` (C++) builds. The 14900HX has no AVX-512. +* **Defaults vs tuning**: HNSW M=16, efConstruction=200 for all (M=32 would reach higher recall on GloVe, which is why HNSW rows are missing at glove 1M); IVF nlist = 4*sqrt(N) for all (not tuned per library). faiss IVF is trained on a 40-points-per-cell sample like ours but with its default 25 k-means iterations and non-spherical centroids; ISONGraph runs 8 spherical iterations. Nobody was tuned further than anyone else. +* **Knob grid**: nprobe grid 1,2,4,...,256 (plus nlist at 100k); the interpolation is log-linear between bracketing points, flagged `*` when the bracket is wider than 0.05 recall. +* **Hybrid CPU**: single-thread loops are pinned to one P-core, all-core runs use all 32 logical CPUs including E-cores; the process-wide affinity switch also affects any helper threads. +* **Memory** is process RSS: file-backed engines (lancedb, sqlite-vec, chromadb) look small because pages are cached by the OS or mapped, and chromadb/lancedb hold index data off-heap. +* **Filters**: ISONGraph supports node-type equality only; the others ran an integer metadata column. faiss was given an `IDSelectorBitmap` (the cheapest selector) after an earlier run with `IDSelectorBatch`; hnswlib uses a Python callback. +* **Ground truth ties**: SIFT has many near-tied float32 scores; faiss/numpy score 0.998-0.999 recall on exact search against the float64 ground truth for that reason. +* **Datasets**: text384 is a random sample of Wikipedia abstracts embedded with MiniLM (fp16, max 128 tokens) and 2,000 HotpotQA questions that are mostly not answerable from that sample, so query-to-neighbour similarity is lower than for in-corpus queries; GloVe/SIFT are classic ANN sets (SIFT is Euclidean data scored as cosine after normalisation). +* **Machine state**: other users' background processes (VS Code, Windows services) kept CPU at 3-17% during runs (`load_before_pct` in every JSON); the orchestrator waited for < 25%. The earlier unpinned run showed the machine can also change speed over hours; pinned numbers within one run are consistent to 1-5%, but absolute QPS from a different day may differ. + +## How to reproduce + +```powershell +# one-off setup (venv outside the repo, data outside the repo) +python -m venv D:\Goldmine\ISON\bench_env +D:\Goldmine\ISON\bench_env\Scripts\pip install numpy psutil h5py pyarrow faiss-cpu hnswlib usearch chromadb qdrant-client lancedb sqlite-vec threadpoolctl matplotlib +# data: ann-benchmarks files + MiniLM embeddings of BeIR/hotpotqa, then float64 ground truth, filter groups and mutation plans +curl -O http://ann-benchmarks.com/glove-100-angular.hdf5 ; curl -O http://ann-benchmarks.com/sift-128-euclidean.hdf5 +python benchmarks/competitive/prep_ann.py glove-100-angular glove-100-angular.hdf5 # and sift-128-euclidean +python benchmarks/competitive/prep_text.py --out D:\Goldmine\ISON\bench_data\prep\text384 # needs sentence-transformers (GPU: ~6 min) +python benchmarks/competitive/prep_gt.py text384 10000,100000,1000000 # and glove, sift +# native harnesses +cd isongraph-vector-rs && cargo build --release --example competitive_bench # + RUSTFLAGS="-C target-cpu=native" in a second target dir +cl /std:c++17 /O2 /EHsc /MD /DNDEBUG [/arch:AVX2] /I include /I vendor/ison-graph-cpp bench/competitive_bench.cpp /link psapi.lib # from a vcvars64 prompt +dotnet build isongraph-vector-csharp/bench/IsonGraph.Vector.CompetitiveBench -c Release -o +# (JS needs no build: node --max-old-space-size=24000 isongraph-vector-js/bench/competitive.mjs ...) +# run everything (resumable; waits for CPU load < 25% before each job), then the tables and plots +python benchmarks/competitive/run_all.py --out benchmarks/competitive/results/raw +python benchmarks/competitive/report.py && python benchmarks/competitive/collect_env.py +``` +Paths for the native executables are the constants at the top of `run_all.py`. One cell by hand: `python bench_py.py --system faiss_hnsw --ds text384 --n 100000 --exp core,mutation,saveload --queries 2000 --budget 3 --out x.json`. + +Files: `adapters.py` (one adapter per Python-callable system), `bench_py.py` (runner), `common.py` (data layout, float64 ground truth), `prep_*.py`, `run_all.py`, `report.py`, `compare_runs.py`, `collect_env.py`; native harnesses `isongraph-vector-rs/examples/competitive_bench.rs`, `isongraph-vector-cpp/bench/competitive_bench.cpp`, `isongraph-vector-csharp/bench/IsonGraph.Vector.CompetitiveBench/`, `isongraph-vector-js/bench/competitive.mjs`. Plots: `results/plots/`. diff --git a/benchmarks/competitive/README_method.md b/benchmarks/competitive/README_method.md new file mode 100644 index 0000000..eaab906 --- /dev/null +++ b/benchmarks/competitive/README_method.md @@ -0,0 +1,37 @@ +## How this was measured + +### Data (identical normalised float32 vectors and queries for every system) + +| dataset | vectors | dim | source | queries | +|---|---|---|---|---| +| `text384` | 1,060,000 passages (first 100k / 1M used) | 384 | **real text**: seeded random sample of the BeIR/hotpotqa Wikipedia-abstract corpus (5.2M passages) embedded with `sentence-transformers/all-MiniLM-L6-v2` (fp16 on GPU, `max_seq_length` 128, L2-normalised) | 2,000 real HotpotQA questions | +| `glove` | 1,183,514 (first 100k / 1M used) | 100 | ann-benchmarks `glove-100-angular` | first 2,000 of its test queries | +| `sift` | 1,000,000 (100k / 1M used) | 128 | ann-benchmarks `sift-128-euclidean`, **L2-normalised, scored as cosine** | first 2,000 of its test queries | + +* The HDF5 files' own neighbour lists are **not** used. Ground truth is the exact **float64 cosine** top-100 over the same normalised float32 vectors (`prep_gt.py`), so one definition of "right" applies to all systems. Recall@10 = |returned ∩ true top-10| / 10, averaged over the queries. (On vectors with exact score ties the id sets can differ trivially; none of the datasets have duplicate vectors in the first 100k.) +* Because the vectors are unit length, cosine = inner product, so every system is configured with the metric its documentation recommends for normalised vectors (`ip`/`dot`/`cosine`) and returns the same ranking. +* Streaming-mutation and filter tests use deterministic plans written once by `prep_gt.py` (seeded): 10k inserted ids, 10k updated ids (a different vector is stored under an existing id), 10k deleted ids, all disjoint; a `grp` column splitting the ids into a random **10%** subset, a random **1%** subset and the remaining 89%. Ground truth after mutation / inside each subset is recomputed exactly. + +### What is measured, and how + +* **Recall versus queries/second** (`bench_py.py`, `competitive_bench.{rs,cpp}`, `...CompetitiveBench`, `bench/competitive.mjs`): every approximate system is swept over its accuracy knob (`nprobe` for IVF, `efSearch`/`ef`/`expansion_search`/`ef_search` for HNSW). At every knob value: warm-up (up to 50 queries or 3 s), then **three timed runs** of a strictly **single-threaded, one-query-at-a-time loop** (each query individually timed with a monotonic clock: median/p95/p99 latency from the pooled samples; QPS = queries / loop time; the min-max of the three runs is the run-to-run variation), then, where the system supports it, **three timed runs of a batched call using all 32 logical CPUs**. Up to 2,000 queries per run; systems slower than ~10 s per run are given fewer queries (the count is recorded as `nq`, never below 30). +* **Thread counts**: single-thread loops pin every library to one thread (`faiss.omp_set_num_threads(1)`, `hnswlib num_threads=1`, `usearch threads=1`, BLAS limited with `threadpoolctl`, ISONGraph Rust `set_scan_threads(1)`). The "all-core" column uses 32 threads (24 cores / 32 logical CPUs, hybrid P/E). The ISONGraph ports have **no batched search API and serialise searches on one lock**, so their all-core column is the *same one-query-at-a-time loop* with the library's automatic scan-thread choice (marked †), and a separate "N concurrent clients" probe shows the lock. +* **Build time**: ingest (all vectors added through the library's own add path from pre-parsed arrays) + index build/training + first query (some systems build lazily on the first query). At 100k the build is repeated three times in fresh instances. +* **Memory**: process working set (RSS, `psutil`/`GetProcessMemoryInfo`) minus the baseline measured after imports and before any data is touched. The harness streams vectors from disk in 20k-row chunks and keeps none, so the number is the system's own footprint. "Peak" is the peak working set after the build. File-backed engines (lancedb, chromadb, sqlite-vec) memory-map or cache data and are flattered by RSS; see the caveats. +* **Streaming mutation** (on the already-built index at the recall >= 0.95 operating point): insert 10k new vectors, update 10k existing ids, delete 10k ids, in batches of 1,000 through each system's natural API; recall@10 is re-measured against the exact post-mutation ground truth with the same knob value. `unsupported` means the library offers no such operation on that index type (it would need a rebuild). +* **Filtered search**: a separate build that carries the metadata column (ISONGraph: the node *type*, its only filter; others: an integer payload/metadata column), then restricted search over the 10% and 1% subsets at five accuracy settings. Reported: best single-thread QPS with recall >= 0.90 and the best recall reached. +* **Save/load**: each system's native persistence (faiss/hnswlib/usearch index files; lancedb/chroma/sqlite-vec are on-disk stores, so "save" is ~0 and "load" is reopening; ISONGraph's persistence is the JSON embeddings sidecar and the IVF index is rebuilt after load, which is why its time-to-ready is long). +* **Exact search** (`exact` tables): ISONGraph exact scans (Python numpy, Rust scan cache, C++ hash-map scan, C#, JS) against `faiss.IndexFlatIP`, plain numpy and the other exact engines, at 10k/100k/1M. + +### Parameters that are held equal + +| | value | +|---|---| +| HNSW (faiss, hnswlib, usearch, chromadb) | M (connectivity) = 16, efConstruction (expansion_add) = 200 | +| IVF (ISONGraph, faiss, lancedb) | nlist = round(4·sqrt(N)) (4,000 at 1M: ISONGraph's automatic choice, imposed on the others); k-means on a sample of ~40 points per cell for faiss, ISONGraph uses its own sample/8 iterations; lancedb its default sampling | +| k | 10 | +| Everything else | library defaults | + +### Versions and hardware + +See `results/environment.json` (written by `collect_env.py`). diff --git a/benchmarks/competitive/README_tail.md b/benchmarks/competitive/README_tail.md new file mode 100644 index 0000000..42ea961 --- /dev/null +++ b/benchmarks/competitive/README_tail.md @@ -0,0 +1,59 @@ +## Library issues found (described, not fixed) + +1. **C++ exact search has no scan cache and is scalar** (`isongraph-vector-cpp/include/isongraph_vector.hpp`, `similaritySearchVector`, the loop over `embeddings_` that calls `cosineSimilarity` at ~line 1246/1672). It iterates an `unordered_map` of records, and for each one runs a scalar loop with three dependent accumulators and recomputes both vector norms. Measured: 22.6 QPS at 100k x 384 (numpy 144, Rust 154), 2.1 QPS at 1M. `nprobe >= nlist` falls into the same path, which is why the last point of every C++ IVF sweep collapses to exact speed. Suggested fix: reuse the contiguous vector copy the IVF index already keeps (or build one lazily like the Rust `ScanCache`), cache inverse norms, and score with the existing `simd::dot` kernels. +2. **No batched or concurrent search in any port.** Rust `search_vector` (`isongraph-vector-rs/src/ann.rs` ~line 1722) holds the `scan` mutex for the whole scan; C++ takes `ann_.mutex`; C# `lock (_lock)`; the Python store holds an RLock. The `concurrent_clients` probe in the Rust/C++ JSON shows N client threads do not scale. When a large probe is spread over threads, Rust spawns fresh threads per query (`std::thread::scope` in `scan_probed`). Suggested fix: search against an immutable snapshot (`Arc` / read lock) and add `search_batch(&[Query])` backed by a persistent pool; this alone is worth 10-50x on the all-core column. +3. **First search after writes pays a lazy rebuild/fold.** After 10k inserts + 10k updates + 10k deletes on a 1M text index the first query took 0.6 s (Rust), 0.15 s (C++), 0.11 s (C#), 0.67 s (Python), 5.7 s (JS) versus ~1 ms for hnswlib/usearch (`first_query_after_ms` in the mutation table). Suggested fix: fold pending rows incrementally at write time or on a background thread. +4. **JS IVF build is 522 s at 1M** (Rust 7.6 s, C++ 12 s default / 6 s AVX2, C# 11 s): single-threaded k-means over plain JS arrays of doubles; the JS port also needs ~7 GB for 1M x 384. +5. **Persistence is the JSON sidecar only; the IVF index is never persisted.** 100k x 384 = 0.45-0.8 GB, 1M = 4.5 GB; load then rebuilds the index. A binary vector file (and an index file) would close most of the 20-100x gap to faiss/hnswlib. +6. **Two copies of every vector** (record map + contiguous cache/index copy) give 2-4x the RSS of faiss/hnswlib at 1M. +7. Build flags matter for C++: the default bench build is SSE2; `/arch:AVX2` makes the IVF 1.1-1.4x faster (text 100k 681 -> 957 at 0.95). Rust `target-cpu=native` made no measurable difference to the IVF (828 vs 828) and +5% to the exact scan. + +## What did not run, and why + +* **Qdrant server (HNSW)**: not run. Docker Desktop's daemon was not running, and downloading and executing the official Windows binary was not permitted in this session. Only `qdrant-client` **local mode** ran, which is a pure-numpy brute-force scan (the client itself warns against more than 20,000 points). It was run at 10k and 100k only (100k ingest alone took 284 s) and is not a Qdrant performance number. +* **chromadb build repeats** after the first build failed ("Collection already exists": chromadb 1.5.9 shares one system object per process). The `3 builds` column for chromadb therefore has 1-2 samples at most; everything else for chromadb ran. chromadb ignores a changed `ef_search` until the index is reloaded, so the harness reloads the collection for every knob value (verified separately). +* **chromadb at 1M** ran for text384 only; **sqlite-vec at 1M** for text384 only (and 100k for all three datasets); **qdrant local** only text384 10k and 100k; **JS at 1M** text384 only (a glove/sift 1M JS run would add 30 min each). +* **SIFT 1M** ran for a reduced set only (ISONGraph Rust and C++, faiss HNSW, faiss IVFFlat, hnswlib, numpy brute force) because of the time budget; no streaming-mutation test is possible at SIFT 1M (the file has exactly 1,000,000 vectors, no spare rows for inserts). SIFT 100k ran for every system. +* **usearch filtered search**: the Python binding has no filter predicate: recorded as `unsupported`. hnswlib filtering is a Python callback per visited node, which is the only API it has. +* **faiss HNSW update/delete**: unsupported by that index type (recorded as `unsupported`; it would need a rebuild). +* **Streaming mutation for ISONGraph "exact" and numpy/faiss Flat** measures only cache patching, which is not an index; the like-for-like comparisons are the IVF/HNSW rows. hnswlib `delete` is `mark_deleted` (memory is not freed). +* **Save/load**: not run for ISONGraph at 1M in Python, C++, C#, JS (JSON of 4.5 GB at 1M; Python's `json.dumps` would need tens of GB). Rust did run it. JS has no save/load call for vectors. +* **Cross-process repeat of the pinned run** (second independent process per headline cell) was not done for the final data set because of the time budget; in-process 3 runs per point and the Rust spot-check are the only repeatability evidence (see above). +* **numpy BLAS** is the PyPI wheel (OpenBLAS, `MAX_THREADS=24`), so "all-core" numpy uses up to 24 threads, not 32. + +## Unfairness that could not be removed + +* **Language/runtime**: Python-callable libraries pay Python call overhead per query (20-100 us; invisible for ms-scale queries, material for the 0.3 ms HNSW queries); ISONGraph Python additionally builds result objects. Rust/C++ harnesses call the library directly. +* **SIMD**: faiss, numpy/OpenBLAS and usearch dispatch AVX2/AVX-512 at run time; hnswlib's wheel and our Rust/C++/C#/JS builds do not necessarily. We report both baseline and `target-cpu=native` (Rust) / `/arch:AVX2` (C++) builds. The 14900HX has no AVX-512. +* **Defaults vs tuning**: HNSW M=16, efConstruction=200 for all (M=32 would reach higher recall on GloVe, which is why HNSW rows are missing at glove 1M); IVF nlist = 4*sqrt(N) for all (not tuned per library). faiss IVF is trained on a 40-points-per-cell sample like ours but with its default 25 k-means iterations and non-spherical centroids; ISONGraph runs 8 spherical iterations. Nobody was tuned further than anyone else. +* **Knob grid**: nprobe grid 1,2,4,...,256 (plus nlist at 100k); the interpolation is log-linear between bracketing points, flagged `*` when the bracket is wider than 0.05 recall. +* **Hybrid CPU**: single-thread loops are pinned to one P-core, all-core runs use all 32 logical CPUs including E-cores; the process-wide affinity switch also affects any helper threads. +* **Memory** is process RSS: file-backed engines (lancedb, sqlite-vec, chromadb) look small because pages are cached by the OS or mapped, and chromadb/lancedb hold index data off-heap. +* **Filters**: ISONGraph supports node-type equality only; the others ran an integer metadata column. faiss was given an `IDSelectorBitmap` (the cheapest selector) after an earlier run with `IDSelectorBatch`; hnswlib uses a Python callback. +* **Ground truth ties**: SIFT has many near-tied float32 scores; faiss/numpy score 0.998-0.999 recall on exact search against the float64 ground truth for that reason. +* **Datasets**: text384 is a random sample of Wikipedia abstracts embedded with MiniLM (fp16, max 128 tokens) and 2,000 HotpotQA questions that are mostly not answerable from that sample, so query-to-neighbour similarity is lower than for in-corpus queries; GloVe/SIFT are classic ANN sets (SIFT is Euclidean data scored as cosine after normalisation). +* **Machine state**: other users' background processes (VS Code, Windows services) kept CPU at 3-17% during runs (`load_before_pct` in every JSON); the orchestrator waited for < 25%. The earlier unpinned run showed the machine can also change speed over hours; pinned numbers within one run are consistent to 1-5%, but absolute QPS from a different day may differ. + +## How to reproduce + +```powershell +# one-off setup (venv outside the repo, data outside the repo) +python -m venv D:\Goldmine\ISON\bench_env +D:\Goldmine\ISON\bench_env\Scripts\pip install numpy psutil h5py pyarrow faiss-cpu hnswlib usearch chromadb qdrant-client lancedb sqlite-vec threadpoolctl matplotlib +# data: ann-benchmarks files + MiniLM embeddings of BeIR/hotpotqa, then float64 ground truth, filter groups and mutation plans +curl -O http://ann-benchmarks.com/glove-100-angular.hdf5 ; curl -O http://ann-benchmarks.com/sift-128-euclidean.hdf5 +python benchmarks/competitive/prep_ann.py glove-100-angular glove-100-angular.hdf5 # and sift-128-euclidean +python benchmarks/competitive/prep_text.py --out D:\Goldmine\ISON\bench_data\prep\text384 # needs sentence-transformers (GPU: ~6 min) +python benchmarks/competitive/prep_gt.py text384 10000,100000,1000000 # and glove, sift +# native harnesses +cd isongraph-vector-rs && cargo build --release --example competitive_bench # + RUSTFLAGS="-C target-cpu=native" in a second target dir +cl /std:c++17 /O2 /EHsc /MD /DNDEBUG [/arch:AVX2] /I include /I vendor/ison-graph-cpp bench/competitive_bench.cpp /link psapi.lib # from a vcvars64 prompt +dotnet build isongraph-vector-csharp/bench/IsonGraph.Vector.CompetitiveBench -c Release -o +# (JS needs no build: node --max-old-space-size=24000 isongraph-vector-js/bench/competitive.mjs ...) +# run everything (resumable; waits for CPU load < 25% before each job), then the tables and plots +python benchmarks/competitive/run_all.py --out benchmarks/competitive/results/raw +python benchmarks/competitive/report.py && python benchmarks/competitive/collect_env.py +``` +Paths for the native executables are the constants at the top of `run_all.py`. One cell by hand: `python bench_py.py --system faiss_hnsw --ds text384 --n 100000 --exp core,mutation,saveload --queries 2000 --budget 3 --out x.json`. + +Files: `adapters.py` (one adapter per Python-callable system), `bench_py.py` (runner), `common.py` (data layout, float64 ground truth), `prep_*.py`, `run_all.py`, `report.py`, `compare_runs.py`, `collect_env.py`; native harnesses `isongraph-vector-rs/examples/competitive_bench.rs`, `isongraph-vector-cpp/bench/competitive_bench.cpp`, `isongraph-vector-csharp/bench/IsonGraph.Vector.CompetitiveBench/`, `isongraph-vector-js/bench/competitive.mjs`. Plots: `results/plots/`. diff --git a/benchmarks/competitive/README_top.md b/benchmarks/competitive/README_top.md new file mode 100644 index 0000000..6c8553f --- /dev/null +++ b/benchmarks/competitive/README_top.md @@ -0,0 +1,87 @@ +# Competitive benchmark: ISONGraph Vector against the established vector-search libraries + +**One-paragraph answer.** On this machine, with identical vectors and an exact float64 ground truth, ISONGraph Vector is **not** the fastest vector search and the evidence does **not** support calling it the "best vector store". Its approximate index is a plain IVF-flat. At equal recall@10 its IVF is 1.4-2.3x slower than faiss IVFFlat at the same nlist and **1.9x (GloVe 100k) to 4.3x (text 100k) and 14-15x (text 1M) slower than the HNSW libraries** (hnswlib, faiss HNSW) in single-thread QPS; it has no batch or concurrent-query path, so its all-core throughput is 10-50x below faiss/hnswlib; and its only persistence is a JSON file (15-120x slower to save and load than faiss/hnswlib index files, 3-5x larger on disk). What it does well: the Rust exact scan is the fastest single-thread exact scan measured (6-33% ahead of numpy and faiss `IndexFlatIP` at 100k and 1M on text and GloVe, equal within noise at 10k); index build is 8-25x faster than HNSW builds at 100k-1M; the IVF index is maintained incrementally through insert/update/delete (faiss HNSW cannot update or delete; hnswlib/usearch can); and a node-type filter on a 1% subset is among the fastest filtered searches measured (see the filter table). Losses are listed as plainly as wins below. + +Everything here was measured on one laptop (i9-14900HX, 24 cores / 32 threads, 47.7 GB, Windows 11), see `results/environment.json`. Nothing in the libraries was changed. All numbers are in `results/raw/*.json`; the tables are generated from those files by `report.py`. + +## Read this first: what we can and cannot claim + +| claim | supported? | evidence | +|---|---|---| +| "ISONGraph is the best / fastest vector store" | **No** | HNSW libraries are 1.9-15x faster in single-thread QPS at equal recall; faiss IVFFlat is ~1.4-2.3x faster than our IVF at the same nlist; batch/all-core throughput is 10-50x behind. | +| "ISONGraph's Rust exact search is competitive with faiss Flat / numpy" | **Yes (single thread)** | text384: Rust 154 QPS vs numpy 144, faiss Flat 131 at 100k; 15.1 vs 14.2 / 13.4 at 1M; glove 100k 700 vs 528 / 548. Not true for batched/all-core search (faiss Flat batch is ~35x ours at 100k, 540 vs 15 QPS at 1M) or for the C++/C#/JS ports. | +| "The opt-in IVF index gives a large speed-up over our exact scan with a recall you can choose" | **Yes** | e.g. Rust text384 100k: 154 -> 828 QPS at recall 0.95 (5x); at 1M 15 -> 103 (7x). Python 131 -> 530. | +| "ISONGraph Python is competitive with Rust" | partly | After pinning, Python IVF (numpy/BLAS) is within 1.5-2x of the Rust IVF, and Python exact is within 15% of Rust at 100k text. | +| "We are competitive with other embedded stores (lancedb, chromadb, sqlite-vec, qdrant local mode)" | **Yes, for exact and IVF; not vs chroma's HNSW** | sqlite-vec, lancedb flat and qdrant local mode are 10-100x slower than our exact scans; lancedb IVF_FLAT is 2.5-4x slower than our IVF; chromadb (HNSW) is faster than our IVF at 100k text (906 vs 828 QPS at 0.95) and 5x faster at 1M. | +| "Incremental updates are a strength" | **Yes versus HNSW, no edge versus faiss IVF** | The IVF index stays consistent through insert/update/delete (Rust 1M text: 35k inserts/s, 430k updates/s, 590k deletes/s, recall 0.951 -> 0.952) versus 3-5k inserts/s and updates/s for hnswlib/usearch at 1M; faiss HNSW cannot update or delete at all; faiss IVFFlat does the same job at 72-80k/s inserts. The first search after 30k writes at 1M costs 0.1-0.7 s (Rust 0.6 s) and 5.7 s in JS. | +| "Filtered search is a strength" | **Only for node-type equality** | Our only filter is `node_type`; arbitrary metadata predicates do not exist. On a 1% subset Rust gave 15,288 QPS at recall 1.0 (100k text) against hnswlib 259 (recall 0.98) and chromadb 21; see the filter table for faiss with a bitmap selector. Python/C++ filtered search is 25-120x slower than Rust (widening to an exact scan). | +| "Persistence is competitive" | **No** | JSON sidecar only: 100k x 384 takes 1.9-16 s to save and 1.3-11 s to load per port (450-810 MB), and the IVF index is rebuilt on the first query after load (Python: 5.1 s more). Rust at 1M: 4.5 GB, 21 s save, 20 s to ready. faiss/hnswlib/usearch: 0.06-0.2 s save, 150-180 MB. | +| "Memory efficiency" | **No** | 1M x 384: faiss HNSW 1.6 GB, hnswlib 1.8 GB, faiss IVF 1.9 GB vs Rust 3.5 GB, C++ 3.3 GB, C# 6.7 GB, JS 7.0 GB, Python 4.2 GB (records plus a second contiguous copy). | + +## Headline results (single thread, QPS at recall@10; higher is better) + +QPS at recall@10 = 0.95, copied from `results/tables_cells.md` (interpolated between measured sweep points; `*` = conservative measured point because the bracket was wide). + +| system | text384 100k | text384 1M | glove 100k | glove 1M | sift 100k | +|---|---|---|---|---|---| +| hnswlib | 3,490 | 1,486 | 3,762 | not reached (ef<=640) | 22,439 | +| faiss HNSWFlat | 3,542 | 1,572 | 2,152 | not reached | 17,392 | +| usearch | 2,230 | 706 | 1,947 | not reached | 13,215 | +| chromadb | 906 | 605 | 933 | not reached | 1,163* | +| faiss IVFFlat | 1,193 | 190 | 2,893 | 43.8* | 8,764* | +| **ISONGraph Rust IVF** | 828 | 103 | 2,029 | 181 | 4,873 | +| **ISONGraph C++ IVF** (default SSE2 / AVX2 build) | 681 / 957 | 94 / 128 | 1,919 / 2,167 | 175 / 185 | 4,054* / 4,693* | +| **ISONGraph C# IVF** | 615 | not reached | 1,268 | not reached | 2,805* | +| **ISONGraph Python IVF** | 530 | 69 | 813 | 156 | 2,622 | +| **ISONGraph JS IVF** | 201 | not reached | 331 | not run | 795 | +| lancedb IVF_FLAT | 222 | 63 | 238 | 109 | 247 | +| *exact:* **ISONGraph Rust** | 154 | 15.1 | 700 | 43.5 | 344 | +| *exact:* numpy | 144 | 14.2 | 528 | 34.4 | 282 | +| *exact:* faiss Flat | 131 | 13.4 | 548 | 40.7 | 283 | +| *exact:* **ISONGraph Python** | 131 | 12.7 | 471 | 26.5 | 318 | +| *exact:* **ISONGraph C#** | 39 | 3.7 | 108 | 5.8 | 55 | +| *exact:* **ISONGraph C++** | 22.6 | 2.1 | 40.1 | 2.5 | 22.8 | +| *exact:* **ISONGraph JS** | 19.7 | 1.7 | 48.1 | not run | 29 | +| *exact:* sqlite-vec / lancedb / qdrant-local | 10 / 7 / 7 | 1.05 / 0.7 / - | 40 / 34 / - | - | 22 / 15 / - | + +SIFT 1M (reduced set, QPS @0.95, 1T): hnswlib 7,311, faiss HNSW 5,603, faiss IVFFlat 1,270*, **Rust IVF 546***, **C++ IVF 562***; exact: **Rust 33.4**, numpy 26.9, **C++ 2.2**. + +"not reached": the sweep (HNSW ef up to 640, IVF nprobe up to 256 at 1M) never got to recall 0.95 at M=16 / nlist=4000. HNSW with larger M would; we held M=16 for everyone. + +Build time and memory (text384; ready = ingest + index + first query): + +| system | 100k ready s | 1M ready s | 1M steady RSS (MB over baseline) | +|---|---|---|---| +| ISONGraph Rust IVF | 0.62 | 7.6 | 3,532 | +| ISONGraph C++ IVF (default / AVX2) | 0.97 / 0.61 | 12.4 / 7.1 | 3,279 | +| ISONGraph Python IVF | 3.6 | 42 | 4,251 | +| ISONGraph JS IVF | 57 | 522 | 7,002 | +| faiss IVFFlat | 1.7 | 18.6 | 1,885 | +| faiss HNSWFlat | 8.9 | 221 | 1,640 | +| hnswlib | 6.2 | 173 | 1,830 | +| usearch | 5.9 | 174 | 1,715 | +| chromadb | 14.7 | 345 | 4,023 | + +## Wins and losses, ranked + +Wins (measured): +1. Rust exact scan: 1-thread QPS 6-33% above numpy and faiss Flat in every 100k/1M cell measured (text, glove, sift), equal within noise at 10k. Single thread only; ours has no batch path (faiss Flat batch is 35x faster at text 100k). +2. Build time versus HNSW: 8-25x faster (text 1M: Rust 7.6 s vs 173-221 s for HNSW; 100k: 0.6 s vs 6-9 s); versus faiss IVFFlat 2.4x faster at text 1M (7.6 s vs 18.6 s) but not at 100k (0.6 vs 1.7 s, i.e. 2.8x). +3. Index maintenance: 7-100x faster writes than the HNSW libraries (Rust 1M text: 35k inserts/s, 430k updates/s, 590k deletes/s vs hnswlib 4.8k/3.2k/-), recall unchanged (0.951 -> 0.952). +4. Filtered search on a node-type subset, Rust only: 1% of 100k text 15,288 QPS at recall 1.0 vs faiss Flat (bitmap selector) 2,491, faiss IVF 1,630, hnswlib 259 @0.98; 10%: 3,094 vs 464 / 388 / 1,779 @0.91; at text 1M 10%: 650 QPS @0.94 vs faiss Flat 40, hnswlib 406 @0.955; 1%: 1,944 @0.906 vs 172 / 135 @0.945. Python and C++ filtered search is 25-120x slower than Rust (widening to an exact scan). +5. Python IVF on numpy/BLAS is within 1.5-2x of the native ports and is the only port besides Rust/C++/C# with realistic speed at 1M. + +Losses (measured), worst first: +1. All-core / batched throughput: no batch API and one lock per store. text 100k @0.95: faiss HNSW 19,900 QPS, hnswlib 19,900, faiss Flat batch 5,437 vs ours <=1,518 (Rust auto-threads loop, 1.0x its own 1T loop). The "8 concurrent clients" probe confirms no scaling (`concurrent_clients` in the JSON). +2. Persistence: JSON sidecar, 3-5x larger and 15-120x slower than faiss/hnswlib index files; IVF rebuilt on load. Python 100k: save 15.5 s, load 11 s, then 5.1 s for the first query (index rebuild). +3. QPS at equal recall vs HNSW: 4-6x slower at 100k, 12-15x slower at 1M (text384). +4. QPS vs faiss IVFFlat at the same nlist: Rust 0.69x (text 100k), 0.54x (text 1M), 0.70x (glove 100k); C++ default build 0.57x; Python 0.44x. +5. Memory: 1.8-4x the HNSW/IVF libraries at 1M (the store keeps two copies of every vector). +6. C++ exact scan: 22.6 QPS vs numpy 144 at text 100k (6.4x slower), JS 7x slower, C# 3.7x slower (see library issues). +7. JS IVF build: 522 s at 1M (30x slower than Rust). +8. First search after heavy writes on a 1M index costs 0.1-0.7 s in Rust/C++/C#/Python and 5.7 s in JS (lazy fold / rebuild), versus about 1 ms for hnswlib/usearch. + +## Cross-process variation: why the 1-thread loops are pinned + +A first full run of the whole matrix (kept in `results/archive_unpinned/`) used unpinned single-thread loops. A second run of 12 cells hours later differed by **-7% to +227%** per system (for example faiss HNSW 1,083 vs 3,259 QPS at the same recall, Rust IVF 229 vs 748; `archive_unpinned/tables_repeat.md`), while the three timed runs inside each process agreed to 5%. On this hybrid CPU (8 P-cores + 16 E-cores) an unpinned thread may land on an E-core (measured 2.0x slower) and the machine as a whole spent some hours in a slower state. **All numbers in this README come from a second, complete run in which every single-thread loop is pinned to one P-core (logical CPU 8) at HIGH priority** (`bench_py.py: pinned`, `competitive_bench.{rs,cpp}: Pin`, C# `Point`, node pinned to P-cores 8,10,12,14 by `run_all.py`). Three timed runs per point agree to 1-5% (min-max in the full sweep tables). A second independent pinned process repeat of the final data set was not possible in the time available; cross-process stability of the pinned numbers is established only by a Rust spot-check (775 / 761 / 779 QPS in three separate processes at nprobe=128, text384 100k) and by the agreement of the three in-process runs. Treat differences below ~10% between systems as noise. + diff --git a/benchmarks/competitive/adapters.py b/benchmarks/competitive/adapters.py new file mode 100644 index 0000000..4c30836 --- /dev/null +++ b/benchmarks/competitive/adapters.py @@ -0,0 +1,843 @@ +"""One adapter per Python-callable system. Each exposes the same small surface so the runner is identical. + + A = ADAPTERS[name](d=..., n_cap=..., workdir=...) + A.add(ids, X, grp=None) ingest a chunk (ids int64, X float32 (m,d), grp uint8 or None) + A.finalize() build / train / index (the "build" cost beyond ingest) + A.set_knob(v) accuracy knob (nprobe / ef_search); knobs() lists the sweep + A.set_threads(t) library thread count (1 = strictly single thread) + A.search1(q, k) one query -> ids (k,) + A.searchN(Q, k) many queries at once (uses the thread count) -> ids (Q,k) or None + A.upsert(ids, X) / A.delete(ids) streaming mutation, or raise NotImplementedError + A.search_filtered1(q, k, g) restrict to grp==g (needs add(..., grp)), or NotImplementedError + A.save(path) / A.load(path) native persistence (returns None) ; A.disk_bytes(path) + A.close() +Every library is used with its own documented defaults except where the knob is the point of the sweep; +common HNSW parameters are identical across libraries (M=16, efConstruction=200); the IVF cell count is the +same for every IVF system (nlist = round(4*sqrt(N)), what isongraph-vector picks automatically). +""" +import math +import os +import shutil +import sys + +import numpy as np + +HERE = os.path.dirname(os.path.abspath(__file__)) + + +def nlist_for(n): + return int(min(max(round(4 * math.sqrt(n)), 16), 65536)) + + +def dir_bytes(path): + if os.path.isfile(path): + return os.path.getsize(path) + tot = 0 + for r, _, fs in os.walk(path): + for f in fs: + tot += os.path.getsize(os.path.join(r, f)) + return tot + + +class Base: + name = "base" + exact = False + knob_name = None + version = "" + supports_batch = False + + def __init__(self, d, n_cap, workdir): + self.d, self.n_cap, self.workdir = d, n_cap, workdir + os.makedirs(workdir, exist_ok=True) + self.threads = 1 + + def knobs(self, n): + return [None] + + def finalize(self): + pass + + def set_knob(self, v): + pass + + def set_threads(self, t): + self.threads = t + + def searchN(self, Q, k): + return None + + def upsert(self, ids, X): + raise NotImplementedError + + def delete(self, ids): + raise NotImplementedError + + def search_filtered1(self, q, k, g): + raise NotImplementedError + + def save(self, path): + raise NotImplementedError + + def load(self, path): + raise NotImplementedError + + def disk_bytes(self, path): + return dir_bytes(path) + + def close(self): + pass + + +# ----------------------------------------------------------------------------- numpy brute force +class NumpyBF(Base): + name = "numpy_bf" + exact = True + supports_batch = True + + def __init__(self, d, n_cap, workdir): + super().__init__(d, n_cap, workdir) + import numpy + self.version = "numpy " + numpy.__version__ + self.M = np.empty((n_cap, d), dtype=np.float32) + self.ids = np.full(n_cap, -1, dtype=np.int64) + self.alive = np.zeros(n_cap, dtype=bool) + self.cnt = 0 + self.slot = {} + self.grp = None + + def add(self, ids, X, grp=None): + m = len(ids) + self.M[self.cnt:self.cnt + m] = X + self.ids[self.cnt:self.cnt + m] = ids + self.alive[self.cnt:self.cnt + m] = True + if grp is not None: + if self.grp is None: + self.grp = np.full(self.n_cap, 255, dtype=np.uint8) + self.grp[self.cnt:self.cnt + m] = grp + self.slot.update(zip(ids.tolist(), range(self.cnt, self.cnt + m))) + self.cnt += m + + def set_threads(self, t): + from threadpoolctl import threadpool_limits + self._lim = threadpool_limits(limits=t) + self.threads = t + + def _top(self, s, k): + s = np.where(self.alive[:self.cnt], s, -np.inf) + p = np.argpartition(-s, k - 1)[:k] + p = p[np.argsort(-s[p])] + return self.ids[p] + + def search1(self, q, k): + return self._top(self.M[:self.cnt] @ q, k) + + def searchN(self, Q, k): + out = np.empty((len(Q), k), dtype=np.int64) + step = max(1, 4_000_000_000 // (4 * max(self.cnt, 1))) # bound the score matrix at ~4 GB + for s in range(0, len(Q), step): + S = Q[s:s + step] @ self.M[:self.cnt].T + S = np.where(self.alive[:self.cnt][None, :], S, -np.inf) + P = np.argpartition(-S, k - 1, axis=1)[:, :k] + ps = np.take_along_axis(S, P, axis=1) + o = np.argsort(-ps, axis=1) + out[s:s + step] = self.ids[np.take_along_axis(P, o, axis=1)] + return out + + def upsert(self, ids, X): + new_ids, new_X = [], [] + for i, row in zip(ids.tolist(), X): + s = self.slot.get(i) + if s is not None and self.alive[s]: + self.M[s] = row + else: + new_ids.append(i) + new_X.append(row) + if new_ids: + self.add(np.array(new_ids, dtype=np.int64), np.stack(new_X)) + + def delete(self, ids): + for i in ids.tolist(): + self.alive[self.slot[i]] = False + + def search_filtered1(self, q, k, g): + s = self.M[:self.cnt] @ q + s = np.where(self.grp[:self.cnt] == g, s, -np.inf) + p = np.argpartition(-s, k - 1)[:k] + return self.ids[p[np.argsort(-s[p])]] + + def save(self, path): + np.savez(path, M=self.M[:self.cnt], ids=self.ids[:self.cnt]) if False else None + np.save(path + ".M.npy", self.M[:self.cnt]) + np.save(path + ".ids.npy", self.ids[:self.cnt]) + + def load(self, path): + M = np.load(path + ".M.npy") + self.M[:len(M)] = M + self.ids[:len(M)] = np.load(path + ".ids.npy") + self.alive[:len(M)] = True + self.cnt = len(M) + + def disk_bytes(self, path): + return sum(os.path.getsize(path + s) for s in (".M.npy", ".ids.npy")) + + +# ----------------------------------------------------------------------------- isongraph-vector (Python) +class IsonPy(Base): + name = "isong_py" + exact = True + ann = None + + def __init__(self, d, n_cap, workdir): + super().__init__(d, n_cap, workdir) + sys.path.insert(0, os.path.join(HERE, "..")) + # isongraph-vector depends on ison-graph, which is installed in the user's global site-packages. It is + # appended LAST so that the venv's numpy etc. still win; only ison_graph/ison_parser are taken from there. + gs = os.path.join(os.environ.get("APPDATA", ""), "Python", "Python312", "site-packages") + if os.path.isdir(gs) and gs not in sys.path: + sys.path.append(gs) + from _common import import_package + self.mod = import_package() + self.version = "isongraph-vector (python, repo HEAD)" + self._new() + + def _new(self, db_path=":memory:"): + kw = {} + if self.ann: + kw["ann"] = self.ann + self.s = self.mod.EmbeddingStore(db_path, dimension=self.d, **kw) + self.typ = None + + def set_threads(self, t): + from threadpoolctl import threadpool_limits + self._lim = threadpool_limits(limits=t) + self.threads = t + + def add(self, ids, X, grp=None): + rows = X.tolist() + il = ids.tolist() + if grp is None: + ent = [(("doc", i), "", v) for i, v in zip(il, rows)] + else: + tn = ("t10", "t01", "rest") + ent = [((tn[g], i), "", v) for i, v, g in zip(il, rows, grp.tolist())] + self.s.add_batch(ent) + + def search1(self, q, k): + return np.fromiter((r.node_ref[1] for r in self.s.similarity_search(q, top_k=k)), dtype=np.int64, count=-1) + + def upsert(self, ids, X): + rows = X.tolist() + self.s.add_batch([(("doc", i), "", v) for i, v in zip(ids.tolist(), rows)]) + + def delete(self, ids): + for i in ids.tolist(): + self.s.remove(("doc", i)) + + def search_filtered1(self, q, k, g): + t = ("t10", "t01")[g] + return np.fromiter((r.node_ref[1] for r in self.s.similarity_search(q, top_k=k, node_type=t)), dtype=np.int64) + + def save(self, path): + self.s.save_embeddings(path) + + def load(self, path): + self.s.close() + self._new() + self.s.load_embeddings(path) + + def close(self): + self.s.close() + + +class IsonPyIVF(IsonPy): + name = "isong_py_ivf" + exact = False + ann = "ivf" + knob_name = "nprobe" + + def knobs(self, n): + nl = nlist_for(n) + base = [1, 2, 4, 8, 16, 32, 64, 128, 256] + return [b for b in base if b < nl] + [nl] + + def finalize(self): + self.s.build_ann_index() + + def set_knob(self, v): + self.s.ann_nprobe = int(v) + + def stats(self): + return self.s.ann_stats() + + +# ----------------------------------------------------------------------------- faiss +class FaissBase(Base): + def __init__(self, d, n_cap, workdir): + super().__init__(d, n_cap, workdir) + import faiss + self.faiss = faiss + self.version = "faiss-cpu " + faiss.__version__ + self.supports_batch = True + + def set_threads(self, t): + self.faiss.omp_set_num_threads(t) + self.threads = t + + def search1(self, q, k): + return self.idx.search(q[None, :], k, params=self._params())[1][0] + + def searchN(self, Q, k): + return self.idx.search(Q, k, params=self._params())[1] + + def _params(self): + return None + + def save(self, path): + self.faiss.write_index(self.idx, path) + + def load(self, path): + self.idx = self.faiss.read_index(path) + + def _sel_params(self, g): + return self._params_with_sel(self._sels[g]) + + def prepare_filter(self, grp_all): + # IDSelectorBitmap: a packed bit per id, the cheapest selector faiss offers (IDSelectorBatch hashes every id) + self._bits, self._sels = {}, {} + for g in (0, 1): + self._bits[g] = np.packbits(grp_all == g, bitorder="little") + self._sels[g] = self.faiss.IDSelectorBitmap(len(grp_all), self.faiss.swig_ptr(self._bits[g])) + + def search_filtered1(self, q, k, g): + return self.idx.search(q[None, :], k, params=self._sel_params(g))[1][0] + + +class FaissFlat(FaissBase): + name = "faiss_flat" + exact = True + + def __init__(self, d, n_cap, workdir): + super().__init__(d, n_cap, workdir) + self.idx = self.faiss.IndexIDMap2(self.faiss.IndexFlatIP(d)) + + def add(self, ids, X, grp=None): + self.idx.add_with_ids(X, ids) + + def upsert(self, ids, X): + self.idx.remove_ids(self.faiss.IDSelectorBatch(ids)) + self.idx.add_with_ids(X, ids) + + def delete(self, ids): + self.idx.remove_ids(self.faiss.IDSelectorBatch(ids)) + + def _params_with_sel(self, sel): + return self.faiss.SearchParameters(sel=sel) + + +class FaissIVF(FaissBase): + name = "faiss_ivfflat" + knob_name = "nprobe" + + def __init__(self, d, n_cap, workdir): + super().__init__(d, n_cap, workdir) + self.nlist = nlist_for(n_cap if n_cap < 1_000_000 else 1_000_000) + self.nprobe = 1 + self._train_buf = [] + self._pending = [] + self.idx = None + + def configure(self, n): + self.nlist = nlist_for(n) + + def knobs(self, n): + nl = nlist_for(n) + return [b for b in [1, 2, 4, 8, 16, 32, 64, 128, 256] if b < nl] + [nl] + + def add(self, ids, X, grp=None): + # buffer until finalize(): IVF must be trained on data first (40 points per cell, the same sample + # size isongraph-vector's k-means uses). Training is part of the build cost. + if self.idx is not None: # already built: streaming insert + self.idx.add_with_ids(X, ids) + else: + self._pending.append((ids, X)) + + def finalize(self): + f = self.faiss + n = sum(len(i) for i, _ in self._pending) + self.nlist = nlist_for(n) + q = f.IndexFlatIP(self.d) + self.idx = f.IndexIVFFlat(q, self.d, self.nlist, f.METRIC_INNER_PRODUCT) + want = min(n, 40 * self.nlist) + rng = np.random.default_rng(0) + allX = np.concatenate([x for _, x in self._pending]) if len(self._pending) > 1 else self._pending[0][1] + sample = allX[np.sort(rng.choice(n, want, replace=False))] if want < n else allX + self.idx.train(sample) + del allX + for ids, X in self._pending: + self.idx.add_with_ids(X, ids) + self._pending = [] + + def set_knob(self, v): + self.nprobe = int(v) + + def _params(self): + return self.faiss.SearchParametersIVF(nprobe=self.nprobe) + + def _params_with_sel(self, sel): + return self.faiss.SearchParametersIVF(sel=sel, nprobe=self.nprobe) + + def upsert(self, ids, X): + self.idx.remove_ids(self.faiss.IDSelectorBatch(ids)) + self.idx.add_with_ids(X, ids) + + def delete(self, ids): + self.idx.remove_ids(self.faiss.IDSelectorBatch(ids)) + + def load(self, path): + self.idx = self.faiss.read_index(path) + + +class FaissHNSW(FaissBase): + name = "faiss_hnsw" + knob_name = "efSearch" + + def __init__(self, d, n_cap, workdir): + super().__init__(d, n_cap, workdir) + f = self.faiss + self.idx = f.IndexHNSWFlat(d, 16, f.METRIC_INNER_PRODUCT) + self.idx.hnsw.efConstruction = 200 + self.ef = 16 + self._next = 0 + + def knobs(self, n): + return [10, 16, 24, 32, 48, 64, 96, 128, 192, 256, 400, 640] + + def add(self, ids, X, grp=None): + assert ids[0] == self._next, "faiss HNSW assigns sequential ids" + self.idx.add(X) + self._next += len(ids) + + def set_knob(self, v): + self.ef = int(v) + + def _params(self): + return self.faiss.SearchParametersHNSW(efSearch=self.ef) + + def _params_with_sel(self, sel): + return self.faiss.SearchParametersHNSW(sel=sel, efSearch=self.ef) + + def load(self, path): + self.idx = self.faiss.read_index(path) + + +# ----------------------------------------------------------------------------- hnswlib +class Hnswlib(Base): + name = "hnswlib" + knob_name = "ef" + supports_batch = True + + def __init__(self, d, n_cap, workdir): + super().__init__(d, n_cap, workdir) + import hnswlib + self.h = hnswlib + self.version = "hnswlib " + getattr(hnswlib, "__version__", "0.8.0") + self.idx = hnswlib.Index(space="ip", dim=d) + self.idx.init_index(max_elements=n_cap, ef_construction=200, M=16, allow_replace_deleted=True) + self.idx.set_num_threads(32) + self.threads = 32 + self.grp = None + + def knobs(self, n): + return [10, 16, 24, 32, 48, 64, 96, 128, 192, 256, 400, 640] + + def set_threads(self, t): + self.idx.set_num_threads(t) + self.threads = t + + def add(self, ids, X, grp=None): + self.idx.add_items(X, ids, num_threads=self.threads) + + def set_knob(self, v): + self.idx.set_ef(int(v)) + + def search1(self, q, k): + return self.idx.knn_query(q[None, :], k=k, num_threads=1)[0][0] + + def searchN(self, Q, k): + return self.idx.knn_query(Q, k=k, num_threads=self.threads)[0] + + def upsert(self, ids, X): + self.idx.add_items(X, ids, num_threads=self.threads, replace_deleted=False) + + def delete(self, ids): + for i in ids.tolist(): + self.idx.mark_deleted(i) + + def prepare_filter(self, grp_all): + self.grp = grp_all + + def search_filtered1(self, q, k, g): + grp = self.grp + return self.idx.knn_query(q[None, :], k=k, num_threads=1, filter=lambda i: grp[i] == g)[0][0] + + def save(self, path): + self.idx.save_index(path) + + def load(self, path): + self.idx = self.h.Index(space="ip", dim=self.d) + self.idx.load_index(path, max_elements=self.n_cap) + self.idx.set_num_threads(self.threads) + + +# ----------------------------------------------------------------------------- usearch +class Usearch(Base): + name = "usearch" + knob_name = "expansion_search" + supports_batch = True + + def __init__(self, d, n_cap, workdir): + super().__init__(d, n_cap, workdir) + import usearch + from usearch.index import Index + self.Index = Index + self.version = "usearch " + usearch.__version__ + self.idx = Index(ndim=d, metric="ip", dtype="f32", connectivity=16, expansion_add=200, expansion_search=16) + self.threads = 32 + + def knobs(self, n): + return [10, 16, 24, 32, 48, 64, 96, 128, 192, 256, 400, 640] + + def add(self, ids, X, grp=None): + self.idx.add(ids.astype(np.uint64), X, threads=self.threads, copy=True) + + def set_knob(self, v): + self.idx.expansion_search = int(v) + + def search1(self, q, k): + return self.idx.search(q, k, threads=1).keys.astype(np.int64) + + def searchN(self, Q, k): + return self.idx.search(Q, k, threads=self.threads).keys.astype(np.int64) + + def upsert(self, ids, X): + self.idx.remove(ids.astype(np.uint64)) + self.idx.add(ids.astype(np.uint64), X, threads=self.threads, copy=True) + + def delete(self, ids): + self.idx.remove(ids.astype(np.uint64)) + + def save(self, path): + self.idx.save(path) + + def load(self, path): + self.idx = self.Index(ndim=self.d, metric="ip", dtype="f32", connectivity=16, expansion_add=200, expansion_search=16) + self.idx.load(path) + + +# ----------------------------------------------------------------------------- chromadb +class Chroma(Base): + name = "chromadb" + knob_name = "ef_search" + supports_batch = True + + def __init__(self, d, n_cap, workdir): + super().__init__(d, n_cap, workdir) + import chromadb + self.chromadb = chromadb + self.version = "chromadb " + chromadb.__version__ + self.path = os.path.join(workdir, "chroma") + from chromadb.api.shared_system_client import SharedSystemClient + SharedSystemClient.clear_system_cache() + shutil.rmtree(self.path, ignore_errors=True) + self.client = chromadb.PersistentClient(path=self.path) + self.col = self.client.create_collection( + "bench", configuration={"hnsw": {"space": "ip", "ef_construction": 200, "max_neighbors": 16, "ef_search": 16}}) + self.has_meta = False + + def knobs(self, n): + return [10, 16, 32, 64, 128, 256, 512] + + def add(self, ids, X, grp=None): + step = 5000 + for s in range(0, len(ids), step): + kw = {} + if grp is not None: + kw["metadatas"] = [{"grp": int(g)} for g in grp[s:s + step]] + self.col.add(ids=[str(i) for i in ids[s:s + step].tolist()], embeddings=X[s:s + step], **kw) + + def _reopen(self): + # chromadb 1.5.9 only applies a changed ef_search when the collection's HNSW index is reloaded + # (verified: same-handle recall is unchanged after modify(), correct after reopening the client). + from chromadb.api.shared_system_client import SharedSystemClient + self.col = None + self.client = None + SharedSystemClient.clear_system_cache() + self.client = self.chromadb.PersistentClient(path=self.path) + self.col = self.client.get_collection("bench") + + def set_knob(self, v): + self.col.modify(configuration={"hnsw": {"ef_search": int(v)}}) + self._reopen() + + def _ids(self, r): + return np.array([[int(x) for x in row] for row in r["ids"]], dtype=np.int64) + + def search1(self, q, k): + r = self.col.query(query_embeddings=[q], n_results=k, include=[]) + return self._ids(r)[0] + + def searchN(self, Q, k): + out = [] + for s in range(0, len(Q), 500): + out.append(self._ids(self.col.query(query_embeddings=Q[s:s + 500], n_results=k, include=[]))) + return np.concatenate(out) + + def upsert(self, ids, X): + step = 5000 + for s in range(0, len(ids), step): + self.col.upsert(ids=[str(i) for i in ids[s:s + step].tolist()], embeddings=X[s:s + step]) + + def delete(self, ids): + step = 5000 + for s in range(0, len(ids), step): + self.col.delete(ids=[str(i) for i in ids[s:s + step].tolist()]) + + def search_filtered1(self, q, k, g): + r = self.col.query(query_embeddings=[q], n_results=k, where={"grp": int(g)}, include=[]) + return self._ids(r)[0] + + def save(self, path): + pass # PersistentClient writes as it goes; "save" cost is whatever add() already paid + + def load(self, path): + self._reopen() + + def disk_bytes(self, path): + return dir_bytes(self.path) + + def close(self): + self.client = None + + +# ----------------------------------------------------------------------------- qdrant (local mode) +class QdrantLocal(Base): + name = "qdrant_local" + exact = True # local mode is a brute-force numpy scan, not HNSW + + def __init__(self, d, n_cap, workdir): + super().__init__(d, n_cap, workdir) + import qdrant_client + from qdrant_client import QdrantClient, models + self.models = models + self.version = "qdrant-client " + __import__("importlib.metadata", fromlist=["version"]).version("qdrant-client") + " (local mode)" + self.path = os.path.join(workdir, "qdrant") + shutil.rmtree(self.path, ignore_errors=True) + self.QC = QdrantClient + import warnings + warnings.filterwarnings("ignore") + self.c = QdrantClient(path=self.path) + self.c.create_collection("bench", vectors_config=models.VectorParams(size=d, distance=models.Distance.COSINE)) + + def add(self, ids, X, grp=None): + step = 2000 + for s in range(0, len(ids), step): + payload = None if grp is None else [{"grp": int(g)} for g in grp[s:s + step]] + self.c.upload_collection("bench", vectors=X[s:s + step], ids=ids[s:s + step].tolist(), payload=payload, batch_size=step) + + def search1(self, q, k): + r = self.c.query_points("bench", query=q, limit=k) + return np.array([p.id for p in r.points], dtype=np.int64) + + def upsert(self, ids, X): + step = 2000 + for s in range(0, len(ids), step): + self.c.upload_collection("bench", vectors=X[s:s + step], ids=ids[s:s + step].tolist(), batch_size=step) + + def delete(self, ids): + self.c.delete("bench", points_selector=self.models.PointIdsList(points=ids.tolist())) + + def search_filtered1(self, q, k, g): + m = self.models + f = m.Filter(must=[m.FieldCondition(key="grp", match=m.MatchValue(value=int(g)))]) + r = self.c.query_points("bench", query=q, limit=k, query_filter=f) + return np.array([p.id for p in r.points], dtype=np.int64) + + def save(self, path): + self.c.close() + + def load(self, path): + self.c = self.QC(path=self.path) + + def disk_bytes(self, path): + return dir_bytes(self.path) + + def close(self): + try: + self.c.close() + except Exception: + pass + + +# ----------------------------------------------------------------------------- lancedb +class LanceBase(Base): + knob_name = "nprobes" + index = True + + def __init__(self, d, n_cap, workdir): + super().__init__(d, n_cap, workdir) + import lancedb + import pyarrow as pa + self.lancedb, self.pa = lancedb, pa + self.version = "lancedb " + lancedb.__version__ + self.path = os.path.join(workdir, "lance") + shutil.rmtree(self.path, ignore_errors=True) + self.db = lancedb.connect(self.path) + self.tbl = None + self.nprobes = 1 + self.has_grp = False + + def _tab(self, ids, X, grp): + pa = self.pa + cols = {"id": pa.array(ids, pa.int64()), + "vector": pa.FixedSizeListArray.from_arrays(pa.array(X.reshape(-1)), self.d)} + if grp is not None: + cols["grp"] = pa.array(grp.astype(np.int8), pa.int8()) + return pa.table(cols) + + def add(self, ids, X, grp=None): + t = self._tab(ids, X, grp) + if self.tbl is None: + self.tbl = self.db.create_table("bench", data=t) + else: + self.tbl.add(t) + + def set_knob(self, v): + self.nprobes = int(v) + + def _q(self, q): + s = self.tbl.search(q, vector_column_name="vector") + if self.index: + s = s.nprobes(self.nprobes) + return s + + def search1(self, q, k): + return np.asarray(self._q(q).limit(k).select(["id"]).to_arrow()["id"].to_numpy()) + + def upsert(self, ids, X): + self.tbl.merge_insert("id").when_matched_update_all().when_not_matched_insert_all().execute(self._tab(ids, X, None)) + + def delete(self, ids): + self.tbl.delete("id IN (" + ",".join(map(str, ids.tolist())) + ")") + + def search_filtered1(self, q, k, g): + return np.asarray(self._q(q).where(f"grp = {int(g)}", prefilter=True).limit(k).select(["id"]).to_arrow()["id"].to_numpy()) + + def save(self, path): + pass # lance writes datasets as it goes + + def load(self, path): + self.db = self.lancedb.connect(self.path) + self.tbl = self.db.open_table("bench") + + def disk_bytes(self, path): + return dir_bytes(self.path) + + +class LanceIVF(LanceBase): + name = "lancedb_ivfflat" + + def knobs(self, n): + nl = nlist_for(n) + return [b for b in [1, 2, 4, 8, 16, 32, 64, 128, 256] if b < nl] + [nl] + + def finalize(self): + n = self.tbl.count_rows() + self.tbl.create_index(metric="dot", index_type="IVF_FLAT", num_partitions=nlist_for(n), vector_column_name="vector") + + +class LanceFlat(LanceBase): + name = "lancedb_flat" + exact = True + index = False + + def _q(self, q): + return self.tbl.search(q, vector_column_name="vector").metric("dot") + + +# ----------------------------------------------------------------------------- sqlite-vec +class SqliteVec(Base): + name = "sqlite_vec" + exact = True + + def __init__(self, d, n_cap, workdir): + super().__init__(d, n_cap, workdir) + import sqlite3 + import sqlite_vec + self.sv = sqlite_vec + self.version = "sqlite-vec " + sqlite_vec.__version__ + " / sqlite " + sqlite3.sqlite_version + self.path = os.path.join(workdir, "sv.db") + for s in ("", "-wal", "-shm"): + if os.path.exists(self.path + s): + os.remove(self.path + s) + self.sqlite3 = sqlite3 + self._open() + self.has_grp = False + + def _open(self): + self.db = self.sqlite3.connect(self.path) + self.db.enable_load_extension(True) + self.sv.load(self.db) + self.db.enable_load_extension(False) + + def _create(self, grp): + self.has_grp = grp is not None + meta = ", grp integer" if self.has_grp else "" + self.db.execute(f"CREATE VIRTUAL TABLE IF NOT EXISTS v USING vec0(embedding float[{self.d}] distance_metric=cosine{meta})") + + def add(self, ids, X, grp=None): + self._create(grp) + cur = self.db.cursor() + if grp is None: + cur.executemany("INSERT INTO v(rowid, embedding) VALUES (?, ?)", ((int(i), x.tobytes()) for i, x in zip(ids, X))) + else: + cur.executemany("INSERT INTO v(rowid, embedding, grp) VALUES (?, ?, ?)", ((int(i), x.tobytes(), int(g)) for i, x, g in zip(ids, X, grp))) + self.db.commit() + + def search1(self, q, k): + rows = self.db.execute("SELECT rowid FROM v WHERE embedding MATCH ? AND k = ?", (q.tobytes(), k)).fetchall() + return np.array([r[0] for r in rows], dtype=np.int64) + + def upsert(self, ids, X): + cur = self.db.cursor() + for i, x in zip(ids.tolist(), X): + cur.execute("DELETE FROM v WHERE rowid = ?", (i,)) + cur.execute("INSERT INTO v(rowid, embedding) VALUES (?, ?)", (i, x.tobytes())) + self.db.commit() + + def delete(self, ids): + self.db.executemany("DELETE FROM v WHERE rowid = ?", ((i,) for i in ids.tolist())) + self.db.commit() + + def search_filtered1(self, q, k, g): + rows = self.db.execute("SELECT rowid FROM v WHERE embedding MATCH ? AND k = ? AND grp = ?", (q.tobytes(), k, int(g))).fetchall() + return np.array([r[0] for r in rows], dtype=np.int64) + + def save(self, path): + self.db.commit() + self.db.close() + + def load(self, path): + self._open() + + def disk_bytes(self, path): + return sum(os.path.getsize(self.path + s) for s in ("", "-wal") if os.path.exists(self.path + s)) + + def close(self): + try: + self.db.close() + except Exception: + pass + + +ADAPTERS = {c.name: c for c in (NumpyBF, IsonPy, IsonPyIVF, FaissFlat, FaissIVF, FaissHNSW, Hnswlib, Usearch, Chroma, + QdrantLocal, LanceIVF, LanceFlat, SqliteVec)} diff --git a/benchmarks/competitive/bench_py.py b/benchmarks/competitive/bench_py.py new file mode 100644 index 0000000..1b8d1a9 --- /dev/null +++ b/benchmarks/competitive/bench_py.py @@ -0,0 +1,463 @@ +"""Run one (system, dataset, n) cell of the competitive benchmark in a fresh process and write JSON. + + python bench_py.py --system faiss_hnsw --ds text384 --n 100000 --exp core,mutation,saveload --out result.json + python bench_py.py --system faiss_hnsw --ds text384 --n 100000 --exp filter --out result_filter.json + +Experiments (each is its own block in the JSON): + core build time, memory, exact/ANN accuracy sweep (1 thread query loop and all-cores batch), x3 runs + mutation insert 10k / update 10k / delete 10k into the built index, then recall again + saveload native save + load, on-disk size, time to first query after load + filter a separate build with a metadata column; search restricted to a random 10% and 1% subset +Memory is the process RSS (working set) and its peak, read with psutil. +""" +import argparse +import gc +import json +import os +import statistics +import sys +import time + +import numpy as np +import psutil + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, HERE) +import common as C # noqa: E402 +from adapters import ADAPTERS # noqa: E402 + +PROC = psutil.Process() +K = 10 +ALL_THREADS = os.cpu_count() or 1 + + +PIN_CPU = int(os.environ.get("BENCH_PIN_CPU", "8")) # a P-core logical CPU on the i9-14900HX (0-15 = P-cores, 16-31 = E-cores) +ALL_CPUS = list(range(os.cpu_count() or 1)) + + +class pinned: + """Pin the process to one P-core for a strictly single-threaded timed loop (and raise priority); restore after. + Unpinned single-thread loops on this hybrid CPU were measured to vary up to 3x between processes (E-core placement, + power state), so every 1-thread number in the report is taken pinned.""" + + def __enter__(self): + try: + PROC.cpu_affinity([PIN_CPU]) + except Exception: # noqa: BLE001 + pass + + def __exit__(self, *a): + try: + PROC.cpu_affinity(ALL_CPUS) + except Exception: # noqa: BLE001 + pass + + +def mem(): + mi = PROC.memory_info() + return {"rss_mb": mi.rss / 2**20, "peak_mb": getattr(mi, "peak_wset", mi.rss) / 2**20} + + +def pct(a, p): + return float(np.percentile(a, p)) + + +def lat_stats(lat_s): + a = np.asarray(lat_s) * 1000.0 + return {"median_ms": pct(a, 50), "p95_ms": pct(a, 95), "p99_ms": pct(a, 99), "mean_ms": float(a.mean())} + + +def loop(fn, Q, k, collect=True): + lat = np.empty(len(Q)) + res = np.empty((len(Q), k), dtype=np.int64) if collect else None + for i in range(len(Q)): + t = time.perf_counter() + r = fn(Q[i], k) + lat[i] = time.perf_counter() - t + if collect: + m = min(len(r), k) + res[i, :m] = r[:m] + res[i, m:] = -1 + return lat, res + + +def ingest(A, ds, n, with_grp=None, chunk=20000, start=0): + secs = 0.0 + for s in range(start, start + n, chunk): + m = min(chunk, start + n - s) + X = np.ascontiguousarray(C.base_rows(ds, s, m)) + ids = np.arange(s, s + m, dtype=np.int64) + g = None if with_grp is None else with_grp[s:s + m] + t = time.perf_counter() + A.add(ids, X, g) + secs += time.perf_counter() - t + return secs + + +def build_once(name, ds, n, d, workdir, n_cap): + A = ADAPTERS[name](d, n_cap, workdir) + A.set_threads(ALL_THREADS) + t_ing = ingest(A, ds, n) + t = time.perf_counter() + A.finalize() + t_idx = time.perf_counter() - t + return A, t_ing, t_idx + + +def warm(fn, Q, k, max_q=50, max_s=3.0): + """warm-up: up to max_q queries or max_s seconds, whichever first.""" + t0 = time.perf_counter() + for i in range(min(max_q, len(Q))): + fn(Q[i], k) + if time.perf_counter() - t0 > max_s: + break + + +def sweep_point(A, Q, truth, k, budget_s, repeats=3): + with pinned(): + return _sweep_point(A, Q, truth, k, budget_s, repeats) + + +def _sweep_point(A, Q, truth, k, budget_s, repeats=3): + """1-thread loop at the current knob. returns dict.""" + A.set_threads(1) + warm(A.search1, Q, k) + # probe latency to size the query count + probe, _ = loop(A.search1, Q[:5], k, collect=False) + est = float(np.median(probe)) + nq = int(min(len(Q), max(30, budget_s / max(est, 1e-6)))) + q = Q[:nq] + qps_runs, lat_all = [], [] + rec = None + for r in range(repeats): + lat, res = loop(A.search1, q, k, collect=(r == 0)) + if r == 0: + rec = C.recall_at(res, truth[:nq], k) + qps_runs.append(nq / lat.sum()) + lat_all.append(lat) + if lat.sum() > budget_s * 1.5: # too slow to repeat more + break + lat_cat = np.concatenate(lat_all) + rec_q = nq + want = min(len(Q), 1000) + if nq < want and est * (want - nq) < 20: + # recall over at least 1000 queries even when the timed loop was shortened (untimed extra pass) + _, res2 = loop(A.search1, Q[nq:want], k, collect=True) + rec = (rec * nq + C.recall_at(res2, truth[nq:want], k) * (want - nq)) / want + rec_q = want + out = {"nq": nq, "recall_queries": rec_q, "recall": rec, "qps_1t_runs": qps_runs, "qps_1t": statistics.median(qps_runs), + "qps_1t_min": min(qps_runs), "qps_1t_max": max(qps_runs)} + out.update(lat_stats(lat_cat)) + return out + + +def batch_point(A, Q, truth, k, repeats=3): + A.set_threads(ALL_THREADS) + try: + r0 = A.searchN(Q[:min(64, len(Q))], k) + except NotImplementedError: + return None + if r0 is None: + return None + runs = [] + rec = None + for r in range(repeats): + t = time.perf_counter() + res = A.searchN(Q, k) + dt = time.perf_counter() - t + runs.append(len(Q) / dt) + if r == 0: + rec = C.recall_at(res, truth[:len(Q)], k) + if dt > 90: + break + return {"recall": rec, "qps_batch_runs": runs, "qps_batch": statistics.median(runs), + "qps_batch_min": min(runs), "qps_batch_max": max(runs), "threads": ALL_THREADS} + + +def core(args, name, ds, n, d, workdir, out): + Qall = C.queries(ds, args.queries) + truth = C.gt(ds, n, "", args.queries) + n_cap = n + 2 * C.MUT + gc.collect() + out["mem_baseline"] = mem() + A, t_ing, t_idx = build_once(name, ds, n, d, workdir, n_cap) + gc.collect() + out["version"] = A.version + out["build"] = {"ingest_s": t_ing, "index_s": t_idx, "total_s": t_ing + t_idx} + out["mem_after_build"] = mem() + # first query (some systems build lazily here) + A.set_threads(1) + knobs = A.knobs(n) + if knobs[0] is not None: + A.set_knob(knobs[-1]) + t = time.perf_counter() + A.search1(Qall[0], K) + out["build"]["first_query_ms"] = (time.perf_counter() - t) * 1000 + out["build"]["ready_s"] = out["build"]["total_s"] + out["build"]["first_query_ms"] / 1000 + out["mem_after_first_query"] = mem() + if hasattr(A, "stats"): + try: + out["index_stats"] = {k_: v for k_, v in A.stats().items() if isinstance(v, (int, float, str, type(None)))} + except Exception as e: # noqa: BLE001 + out["index_stats"] = {"error": str(e)} + pts = [] + budget = args.budget + for kn in knobs: + if kn is not None: + A.set_knob(kn) + p = sweep_point(A, Qall, truth, K, budget) + p["knob"] = kn + b = batch_point(A, Qall, truth, K) + if b: + p["batch"] = b + pts.append(p) + print(f" {name} {ds} {n} knob={kn} recall={p['recall']:.4f} qps1t={p['qps_1t']:.0f} med={p['median_ms']:.3f}ms" + + (f" batch={b['qps_batch']:.0f}" if b else ""), flush=True) + if p["recall"] >= 0.9995 and len(pts) > 2 and kn is not None and pts[-2]["recall"] >= 0.9995: + break + out["sweep"] = pts + out["knob_name"] = A.knob_name + # exact systems: also "one query at a time, library may use all cores" (latency mode) + if A.exact or name in ("isong_py",): + A.set_threads(ALL_THREADS) + warm(A.search1, Qall, K, 20) + probe, _ = loop(A.search1, Qall[:5], K, collect=False) + nq = int(min(len(Qall), max(30, budget / max(float(np.median(probe)), 1e-6)))) + runs = [] + for r in range(3): + lat, _ = loop(A.search1, Qall[:nq], K, collect=False) + runs.append(lat) + if lat.sum() > budget * 1.5: + break + cat = np.concatenate(runs) + out["loop_allthreads"] = dict(nq=nq, threads=ALL_THREADS, qps=statistics.median([nq / r_.sum() for r_ in runs]), **lat_stats(cat)) + out["mem_after_sweep"] = mem() + # reference knob: first with recall >= 0.95 else best + ref = None + for p in pts: + if p["recall"] >= 0.95: + ref = p["knob"] + break + if ref is None and pts and pts[0]["knob"] is not None: + ref = pts[-1]["knob"] + out["ref_knob"] = ref + out["ref_recall_pre"] = next((p["recall"] for p in pts if p["knob"] == ref), None) + # build repeats (fresh adapters) at small sizes only + if args.build_repeats > 1: + rep = [out["build"]["total_s"]] + try: + for _ in range(args.build_repeats - 1): + A2, a, b = build_once(name, ds, n, d, workdir + "_rep", n_cap) + rep.append(a + b) + A2.close() + del A2 + gc.collect() + except Exception as e: # noqa: BLE001 + out["build"]["repeat_error"] = f"{type(e).__name__}: {e}" + out["build"]["total_s_runs"] = rep + return A + + +def mutation(args, A, name, ds, n, out): + if not os.path.exists(os.path.join(C.ds_dir(ds), f"gt_mut_{n}.i32")): + out["mutation"] = {"skipped": "dataset has no spare rows for this n"} + return + Qall = C.queries(ds, args.queries) + truth = C.gt(ds, n, "mut_", args.queries) + upd, dele = C.mut_plan(ds, n) + ref = out.get("ref_knob") + res = {"ref_knob": ref, "recall_before": out.get("ref_recall_pre")} + A.set_threads(ALL_THREADS if name not in ("isong_py", "isong_py_ivf") else 1) + ok = True + + def timed(label, fn, ids_all, X_all=None, batch=1000): + nonlocal ok + t0 = time.perf_counter() + try: + for s in range(0, len(ids_all), batch): + if X_all is None: + fn(ids_all[s:s + batch]) + else: + fn(ids_all[s:s + batch], X_all[s:s + batch]) + except NotImplementedError: + res[label] = {"unsupported": True} + ok = False + return + except Exception as e: # noqa: BLE001 + res[label] = {"error": f"{type(e).__name__}: {e}"} + ok = False + return + dt = time.perf_counter() - t0 + res[label] = {"seconds": dt, "per_s": len(ids_all) / dt, "batch": batch} + + Xi = np.ascontiguousarray(C.base_rows(ds, n, C.MUT)) + timed("insert_10k", A_add_wrapper(A), np.arange(n, n + C.MUT, dtype=np.int64), Xi) + Xu = np.ascontiguousarray(C.base_rows(ds, n + C.MUT, C.MUT)) + timed("update_10k", A.upsert, upd.astype(np.int64), Xu) + timed("delete_10k", A.delete, dele.astype(np.int64)) + res["mem_after"] = mem() + if ok: + A.set_threads(1) + if ref is not None: + A.set_knob(ref) + t = time.perf_counter() + A.search1(Qall[0], K) + res["first_query_after_ms"] = (time.perf_counter() - t) * 1000 + p = sweep_point(A, Qall, truth, K, args.budget) + res["after"] = p + res["recall_after"] = p["recall"] + if hasattr(A, "stats"): + try: + res["index_stats_after"] = {k_: v for k_, v in A.stats().items() if isinstance(v, (int, float, str, type(None)))} + except Exception: # noqa: BLE001 + pass + out["mutation"] = res + + +def A_add_wrapper(A): + def f(ids, X): + A.add(ids, X, None) + return f + + +def saveload(args, A, name, ds, n, out): + Qall = C.queries(ds, args.queries) + path = os.path.join(args.workdir, f"{name}_{ds}_{n}.saved") + res = {} + ref = out.get("ref_knob") + try: + A.set_threads(ALL_THREADS) + t = time.perf_counter() + A.save(path) + res["save_s"] = time.perf_counter() - t + res["disk_mb"] = A.disk_bytes(path) / 2**20 + t = time.perf_counter() + A.load(path) + res["load_s"] = time.perf_counter() - t + A.set_threads(1) + if ref is not None: + A.set_knob(ref) + t = time.perf_counter() + A.finalize() if name in ("isong_py_ivf",) else None + A.search1(Qall[0], K) + res["first_query_after_load_ms"] = (time.perf_counter() - t) * 1000 + res["time_to_ready_s"] = res["load_s"] + res["first_query_after_load_ms"] / 1000 + except NotImplementedError: + res["unsupported"] = True + except Exception as e: # noqa: BLE001 + res["error"] = f"{type(e).__name__}: {e}" + out["saveload"] = res + + +def filt(args, name, ds, n, d, workdir, out): + Qall = C.queries(ds, args.queries) + g = C.grp(ds, n) + n_cap = n + 2 * C.MUT + A = ADAPTERS[name](d, n_cap, workdir) + A.set_threads(ALL_THREADS) + t_ing = ingest(A, ds, n, with_grp=g) + t = time.perf_counter() + A.finalize() + t_idx = time.perf_counter() - t + if hasattr(A, "prepare_filter"): + A.prepare_filter(g) + res = {"build_with_metadata_s": t_ing + t_idx, "mem": mem(), "subsets": {}} + knobs = A.knobs(n) + if knobs[0] is not None and len(knobs) > 5: + pick = [knobs[i] for i in sorted(set([1, len(knobs) // 3, len(knobs) // 2, 2 * len(knobs) // 3, len(knobs) - 1]))] + else: + pick = knobs + A.set_threads(1) + _pin = pinned() + _pin.__enter__() + for gi, label in ((0, "f10"), (1, "f01")): + truth = C.gt(ds, n, label + "_", args.queries) + pts = [] + for kn in pick: + if kn is not None: + A.set_knob(kn) + fn = (lambda q, k, gi=gi: A.search_filtered1(q, k, gi)) + try: + warm(fn, Qall, K, 10, 2.0) + probe, _ = loop(fn, Qall[:5], K, collect=False) + except NotImplementedError: + res["subsets"][label] = {"unsupported": True} + pts = None + break + nq = int(min(len(Qall), max(30, args.budget / max(float(np.median(probe)), 1e-6)))) + runs = [] + rec = None + lats = [] + for r in range(3): + lat, rs = loop(fn, Qall[:nq], K, collect=(r == 0)) + if r == 0: + rec = C.recall_at(rs, truth[:nq], K) + runs.append(nq / lat.sum()) + lats.append(lat) + if lat.sum() > args.budget * 1.5: + break + p = {"knob": kn, "nq": nq, "recall": rec, "qps_1t": statistics.median(runs), "qps_1t_runs": runs} + p.update(lat_stats(np.concatenate(lats))) + pts.append(p) + print(f" filter {label} {name} knob={kn} recall={rec:.4f} qps={p['qps_1t']:.0f}", flush=True) + if pts is not None: + res["subsets"][label] = pts + _pin.__exit__() + out["filter"] = res + A.close() + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--system", required=True) + ap.add_argument("--ds", required=True) + ap.add_argument("--n", type=int, required=True) + ap.add_argument("--exp", default="core,mutation,saveload") + ap.add_argument("--queries", type=int, default=1000) + ap.add_argument("--budget", type=float, default=15.0, help="target seconds per timed 1-thread loop; fewer queries if slower") + ap.add_argument("--build-repeats", type=int, default=1) + ap.add_argument("--workdir", default=r"D:\Goldmine\ISON\bench_data\work") + ap.add_argument("--out", required=True) + args = ap.parse_args() + exps = args.exp.split(",") + m = C.meta(args.ds) + d = m["d"] + os.makedirs(args.workdir, exist_ok=True) + wd = os.path.join(args.workdir, f"{args.system}_{args.ds}_{args.n}") + os.makedirs(wd, exist_ok=True) + out = {"system": args.system, "ds": args.ds, "n": args.n, "d": d, "k": K, "queries": args.queries, + "all_threads": ALL_THREADS, "exps": exps, "started": time.strftime("%Y-%m-%d %H:%M:%S")} + A = None + try: + if "core" in exps: + A = core(args, args.system, args.ds, args.n, d, wd, out) + if "mutation" in exps and A is not None: + mutation(args, A, args.system, args.ds, args.n, out) + if "saveload" in exps and A is not None: + saveload(args, A, args.system, args.ds, args.n, out) + if "filter" in exps: + if A is not None: + A.close() + A = None + gc.collect() + filt(args, args.system, args.ds, args.n, d, wd, out) + out["mem_end"] = mem() + out["status"] = "ok" + except Exception as e: # noqa: BLE001 + import traceback + out["status"] = "error" + out["error"] = f"{type(e).__name__}: {e}" + out["traceback"] = traceback.format_exc() + print(out["traceback"], flush=True) + out["finished"] = time.strftime("%Y-%m-%d %H:%M:%S") + os.makedirs(os.path.dirname(os.path.abspath(args.out)), exist_ok=True) + json.dump(out, open(args.out, "w"), indent=1, default=float) + if A is not None: + try: + A.close() + except Exception: # noqa: BLE001 + pass + + +if __name__ == "__main__": + main() diff --git a/benchmarks/competitive/collect_env.py b/benchmarks/competitive/collect_env.py new file mode 100644 index 0000000..56cc0a7 --- /dev/null +++ b/benchmarks/competitive/collect_env.py @@ -0,0 +1,53 @@ +"""Write results/environment.json: hardware, OS, toolchains and exact library versions used.""" +import json +import os +import platform +import subprocess +import sys +from importlib import metadata + +import psutil + +HERE = os.path.dirname(os.path.abspath(__file__)) + + +def run(cmd): + try: + return subprocess.run(cmd, capture_output=True, text=True, timeout=60, shell=isinstance(cmd, str)).stdout.strip() or "n/a" + except Exception as e: # noqa: BLE001 + return f"n/a ({e})" + + +def main(): + env = { + "python": sys.version.split()[0], "platform": platform.platform(), + "cpu_model": run(["powershell", "-NoProfile", "-Command", "(Get-CimInstance Win32_Processor).Name"]), + "cpu_cores": psutil.cpu_count(logical=False), "cpu_logical": psutil.cpu_count(logical=True), + "ram_gb": round(psutil.virtual_memory().total / 2**30, 1), + "note": "i9-14900HX is a hybrid CPU (8 P-cores with SMT + 16 E-cores); single-thread loops are not pinned to a core type", + "packages": {p: metadata.version(p) for p in ("numpy", "faiss-cpu", "hnswlib", "usearch", "chromadb", "qdrant-client", "lancedb", "sqlite-vec", "pyarrow", "threadpoolctl", "psutil")}, + "rustc": run(["rustc", "--version"]), "cargo": run(["cargo", "--version"]), + "rust_build": "cargo build --release --example competitive_bench (default flags = baseline x86-64) and RUSTFLAGS='-C target-cpu=native'", + "msvc": run('cmd /c ""C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Auxiliary\\Build\\vcvars64.bat" >nul && cl 2>&1"').splitlines()[0:1], + "cpp_build": "cl /std:c++17 /O2 /EHsc /MD /DNDEBUG (default SSE2) and the same with /arch:AVX2", + "dotnet": run(["dotnet", "--version"]), "node": run(["node", "--version"]), + } + try: + import sqlite3 + env["sqlite3"] = sqlite3.sqlite_version + except Exception: # noqa: BLE001 + pass + try: + import numpy + import faiss + env["numpy_blas"] = str(numpy.show_config(mode="dicts").get("Build Dependencies", {}).get("blas", "")) + env["faiss_avx"] = [m for m in ("avx512", "avx2", "sse4") if hasattr(faiss, "supported_instruction_sets") and m in str(faiss.supported_instruction_sets()).lower()] + except Exception as e: # noqa: BLE001 + env["numpy_blas"] = f"n/a ({e})" + os.makedirs(os.path.join(HERE, "results"), exist_ok=True) + json.dump(env, open(os.path.join(HERE, "results", "environment.json"), "w"), indent=1) + print(json.dumps(env, indent=1)) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/competitive/common.py b/benchmarks/competitive/common.py new file mode 100644 index 0000000..1e26432 --- /dev/null +++ b/benchmarks/competitive/common.py @@ -0,0 +1,127 @@ +"""Shared data layout, deterministic plans and float64 ground truth for the competitive benchmark. + +Layout (all raw little-endian, row-major): + //base.f32 N_total x d float32, L2-normalised + //queries.f32 Q x d float32, L2-normalised + //meta.json + //gt_.i32 Q x 100 exact float64 cosine top-100 ids over base[0:n] + //grp_.u8 n 0 = 10% subset, 1 = 1% subset (disjoint), 2 = the rest + //gt_f10_.i32, gt_f01_.i32 Q x 100 top-100 ids inside the 10% / 1% subset + //mut__upd.i32, mut__del.i32 10k ids updated / 10k ids deleted (disjoint) + //gt_mut_.i32 Q x 100 top-100 ids after: insert ids n..n+10k-1 (rows n..n+10k-1), + update (row n+10k+j supplies the new vector of id upd[j]), + delete . +""" +import json +import os + +import numpy as np + +PREP = os.environ.get("BENCH_PREP", r"D:\Goldmine\ISON\bench_data\prep") +K_GT = 100 +MUT = 10_000 +SEED = 20261115 + + +def ds_dir(ds): + return os.path.join(PREP, ds) + + +def meta(ds): + return json.load(open(os.path.join(ds_dir(ds), "meta.json"))) + + +def read_f32(path, start, count, d): + """rows [start, start+count) of a raw float32 matrix with d columns.""" + a = np.fromfile(path, dtype=np.float32, count=count * d, offset=start * d * 4) + return a.reshape(-1, d) + + +def base_rows(ds, start, count): + m = meta(ds) + return read_f32(os.path.join(ds_dir(ds), "base.f32"), start, count, m["d"]) + + +def queries(ds, q=None): + m = meta(ds) + a = read_f32(os.path.join(ds_dir(ds), "queries.f32"), 0, m["q"], m["d"]) + return a if q is None else a[:q] + + +def gt(ds, n, kind="", q=None): + """kind in '', 'f10_', 'f01_', 'mut_' -> (Q, 100) int32""" + a = np.fromfile(os.path.join(ds_dir(ds), f"gt_{kind}{n}.i32"), dtype=np.int32).reshape(-1, K_GT) + return a if q is None else a[:q] + + +def grp(ds, n): + return np.fromfile(os.path.join(ds_dir(ds), f"grp_{n}.u8"), dtype=np.uint8) + + +def mut_plan(ds, n): + d = ds_dir(ds) + return (np.fromfile(os.path.join(d, f"mut_{n}_upd.i32"), dtype=np.int32), + np.fromfile(os.path.join(d, f"mut_{n}_del.i32"), dtype=np.int32)) + + +def make_grp(n): + rng = np.random.default_rng([SEED, 11, n]) + perm = rng.permutation(n) + g = np.full(n, 2, dtype=np.uint8) + g[perm[: n // 10]] = 0 + g[perm[n // 10: n // 10 + n // 100]] = 1 + return g + + +def make_mut(n): + rng = np.random.default_rng([SEED, 12, n]) + ids = rng.choice(n, size=2 * MUT, replace=False).astype(np.int32) + return ids[:MUT], ids[MUT:] + + +def exact_topk(chunks, queries32, k=K_GT): + """Exact float64 cosine top-k. `chunks` yields (first_id_or_id_array, matrix32). Returns ids (Q,k).""" + q = queries32.astype(np.float64) + q /= np.linalg.norm(q, axis=1, keepdims=True) + nq = q.shape[0] + best_s = np.full((nq, k), -np.inf) + best_i = np.zeros((nq, k), dtype=np.int64) + def pieces(): + for ids, mat in chunks: + if len(mat) <= 50000: + yield ids, mat + else: + for s0 in range(0, len(mat), 50000): + sub = ids[s0:s0 + 50000] if not isinstance(ids, (int, np.integer)) else ids + s0 + yield sub, mat[s0:s0 + 50000] + + for ids, mat in pieces(): + m = mat.astype(np.float64) + nrm = np.linalg.norm(m, axis=1, keepdims=True) + nrm[nrm == 0] = 1.0 + m /= nrm + s = (m @ q.T).T # (nq, c) + c = s.shape[1] + if isinstance(ids, (int, np.integer)): + ids_arr = np.arange(ids, ids + c, dtype=np.int64) + else: + ids_arr = np.asarray(ids, dtype=np.int64) + kk = min(k, c) + part = np.argpartition(-s, kk - 1, axis=1)[:, :kk] + ps = np.take_along_axis(s, part, axis=1) + pi = ids_arr[part] + cs = np.concatenate([best_s, ps], axis=1) + ci = np.concatenate([best_i, pi], axis=1) + sel = np.argpartition(-cs, k - 1, axis=1)[:, :k] + best_s = np.take_along_axis(cs, sel, axis=1) + best_i = np.take_along_axis(ci, sel, axis=1) + order = np.argsort(-best_s, axis=1, kind="stable") + return np.take_along_axis(best_i, order, axis=1).astype(np.int32) + + +def recall_at(found, truth, k=10): + """mean recall@k; found (Q,>=k) ids (-1 padded ok), truth (Q,>=k).""" + hit = 0 + for f, t in zip(found, truth): + hit += len(set(int(x) for x in f[:k] if x >= 0) & set(int(x) for x in t[:k])) + return hit / (k * len(truth)) diff --git a/benchmarks/competitive/compare_runs.py b/benchmarks/competitive/compare_runs.py new file mode 100644 index 0000000..68877b7 --- /dev/null +++ b/benchmarks/competitive/compare_runs.py @@ -0,0 +1,25 @@ +"""Cross-process repeatability: compare two independent runs (results/raw vs results/raw_rerun) of the same cells. + + python compare_runs.py > results/tables_repeat.md +""" +import glob, json, os, sys +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import report as R + +HERE = os.path.dirname(os.path.abspath(__file__)) +a, _ = R.load(os.path.join(HERE, "results", "raw")) +b, _ = R.load(os.path.join(HERE, "results", "raw_rerun")) +rows = [] +for k in sorted(b, key=lambda k: (not R.OURS(k[0]), k[0])): + if k not in a or "sweep" not in a[k]: + continue + ra, rb = a[k], b[k] + for thr in (0.95,): + pa, pb = R.best_at(ra, thr), R.best_at(rb, thr) + if not pa or not pb: + continue + d = (pb["qps_1t"] / pa["qps_1t"] - 1) * 100 + ba, bb = ra["build"]["total_s"], rb["build"]["total_s"] + rows.append([R.name(k[0]), f"{k[1]} {k[2]:,}", f"{R.fmt(pa['qps_1t'])} @ {pa['recall']:.3f}", f"{R.fmt(pb['qps_1t'])} @ {pb['recall']:.3f}", f"{d:+.0f}%", + f"{R.fmt(ba)} / {R.fmt(bb)}"]) +print(R.table(["system", "cell", "run 1: QPS 1T @ recall>=0.95", "run 2 (fresh process, later)", "difference", "build s run1 / run2"], rows)) diff --git a/benchmarks/competitive/make_readme.py b/benchmarks/competitive/make_readme.py new file mode 100644 index 0000000..3c259ec --- /dev/null +++ b/benchmarks/competitive/make_readme.py @@ -0,0 +1,25 @@ +"""Assemble README.md from the hand-written parts and the generated tables. python make_readme.py""" +import os + +H = os.path.dirname(os.path.abspath(__file__)) +R = os.path.join(H, "results") + + +def rd(p): + f = os.path.join(H, p) + return open(f, encoding="utf-8").read() if os.path.exists(f) else "" + + +parts = [rd("README_top.md"), rd("README_method.md")] +parts.append("## Results tables (generated by `report.py` from `results/raw/*.json`)\n") +parts.append("Plots (recall vs QPS, single thread and batched): `results/plots/__{1t,batch}.png`.\n") +parts.append("### Recall versus QPS, build time and memory, per dataset and size\n\n" + rd("results/tables_cells.md")) +parts.append("### Exact search latency (10k / 100k / 1M)\n\n`median ms (1T)` is the pinned single-thread loop; `all-threads single query` lets the library use every core for one query at a time; `batch QPS` is a batched call on 32 logical CPUs.\n\n" + rd("results/tables_exact.md")) +parts.append("### Streaming mutation (insert 10k / update 10k / delete 10k on the built index)\n\nBatches of 1,000 through each system's own API. ISONGraph rows include index maintenance; numpy/faiss Flat have no index to maintain. `recall before` is 0.000 where no sweep point reached 0.95 (the highest nprobe was used).\n\n" + rd("results/tables_mutation.md")) +parts.append("### Filtered search (random 10% and 1% subsets)\n\n" + rd("results/tables_filter.md")) +parts.append("### Save / load and on-disk size\n\n" + rd("results/tables_saveload.md")) +f = rd("results/tables_failures.md") +parts.append("### Jobs that failed or timed out\n\n" + (f if f.count("\n") > 2 else "None in the final run.\n")) +parts.append(rd("README_tail.md")) +open(os.path.join(H, "README.md"), "w", encoding="utf-8").write("\n\n".join(parts)) +print("README.md written,", sum(len(p) for p in parts), "chars") diff --git a/benchmarks/competitive/prep_ann.py b/benchmarks/competitive/prep_ann.py new file mode 100644 index 0000000..9d05a65 --- /dev/null +++ b/benchmarks/competitive/prep_ann.py @@ -0,0 +1,42 @@ +"""Convert an ann-benchmarks HDF5 file into the benchmark's raw layout (see common.py). + +The file's own neighbour lists are NOT used: ground truth is recomputed by prep_gt.py in float64 cosine +on the normalised vectors that every system is fed. + + python prep_ann.py glove-100-angular D:/Goldmine/ISON/bench_data/raw/glove-100-angular.hdf5 +""" +import json +import os +import sys + +import h5py +import numpy as np + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import common # noqa: E402 + +name, path = sys.argv[1], sys.argv[2] +nq = 2000 +short = name.split("-")[0] +out = common.ds_dir(short) +os.makedirs(out, exist_ok=True) +f = h5py.File(path, "r") +train, test = f["train"], f["test"] +d = train.shape[1] + + +def norm(x): + x = np.asarray(x, dtype=np.float32) + n = np.linalg.norm(x.astype(np.float64), axis=1, keepdims=True) + n[n == 0] = 1.0 + return (x / n).astype(np.float32) + + +with open(os.path.join(out, "base.f32"), "wb") as o: + for s in range(0, train.shape[0], 100000): + norm(train[s:s + 100000]).tofile(o) +norm(test[:nq]).tofile(os.path.join(out, "queries.f32")) +json.dump({"name": short, "source": f"ann-benchmarks {name}.hdf5", "n": int(train.shape[0]), "d": int(d), "q": nq, + "metric": "cosine on L2-normalised vectors (original metric: " + name.split("-")[-1] + ")"}, + open(os.path.join(out, "meta.json"), "w"), indent=1) +print(name, train.shape, "->", out) diff --git a/benchmarks/competitive/prep_gt.py b/benchmarks/competitive/prep_gt.py new file mode 100644 index 0000000..8f19704 --- /dev/null +++ b/benchmarks/competitive/prep_gt.py @@ -0,0 +1,56 @@ +"""Write ground truth, filter groups and mutation plans for a dataset (see common.py). + + python prep_gt.py glove 10000,100000,1000000 +""" +import os +import sys +import time + +import numpy as np + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import common as C # noqa: E402 + +ds = sys.argv[1] +ns = [int(x) for x in sys.argv[2].split(",")] +m = C.meta(ds) +d = m["d"] +Q = C.queries(ds) +out = C.ds_dir(ds) +base_path = os.path.join(out, "base.f32") + + +def chunks_of(n, step=50000): + for s in range(0, n, step): + yield s, C.read_f32(base_path, s, min(step, n - s), d) + + +for n in ns: + t0 = time.time() + if n > m["n"]: + print("skip", n) + continue + C.exact_topk(chunks_of(n), Q).tofile(os.path.join(out, f"gt_{n}.i32")) + print(ds, n, "gt", round(time.time() - t0, 1), "s", flush=True) + if n < 100000: + continue + g = C.make_grp(n) + g.tofile(os.path.join(out, f"grp_{n}.u8")) + X = C.read_f32(base_path, 0, n, d) + for kind, code in (("f10_", 0), ("f01_", 1)): + ids = np.nonzero(g == code)[0] + C.exact_topk([(ids, X[ids])], Q).tofile(os.path.join(out, f"gt_{kind}{n}.i32")) + del X + if m["n"] >= n + 2 * C.MUT: + upd, dele = C.make_mut(n) + upd.tofile(os.path.join(out, f"mut_{n}_upd.i32")) + dele.tofile(os.path.join(out, f"mut_{n}_del.i32")) + Y = C.read_f32(base_path, 0, n + 2 * C.MUT, d) # rows n..n+10k-1 inserts, n+10k.. update vectors + row_of = np.arange(n + C.MUT, dtype=np.int64) + row_of[upd] = n + C.MUT + np.arange(C.MUT) + alive = np.ones(n + C.MUT, dtype=bool) + alive[dele] = False + ids = np.nonzero(alive)[0] + C.exact_topk([(ids, Y[row_of[ids]])], Q).tofile(os.path.join(out, f"gt_mut_{n}.i32")) + del Y + print(ds, n, "filters+mutation", round(time.time() - t0, 1), "s", flush=True) diff --git a/benchmarks/competitive/prep_text.py b/benchmarks/competitive/prep_text.py new file mode 100644 index 0000000..1ebdaa9 --- /dev/null +++ b/benchmarks/competitive/prep_text.py @@ -0,0 +1,66 @@ +"""Embed a real passage corpus with all-MiniLM-L6-v2 (384-d) for the competitive benchmark. + +Corpus: BeIR/hotpotqa (Wikipedia abstracts, 5.2M passages). We take a seeded random +sample of N passages (default 1,060,000: 1M for the scale tests + headroom for the +streaming-mutation inserts) and 2,000 real HotpotQA questions as queries. +Output (raw little-endian float32, row-major, L2-normalised): base.f32, queries.f32, meta.json + +Run with any Python that has sentence-transformers + torch (a CUDA GPU makes it ~10 minutes): + python prep_text.py --out D:/Goldmine/ISON/bench_data/prep/text384 +""" +import argparse, json, time, os +import numpy as np + +ap = argparse.ArgumentParser() +ap.add_argument("--out", required=True) +ap.add_argument("--n", type=int, default=1_060_000) +ap.add_argument("--queries", type=int, default=2000) +ap.add_argument("--max-seq", type=int, default=128) +ap.add_argument("--seed", type=int, default=20261115) +a = ap.parse_args() +os.makedirs(a.out, exist_ok=True) + +import pandas as pd, torch +from huggingface_hub import hf_hub_download +from sentence_transformers import SentenceTransformer + +cp = hf_hub_download("BeIR/hotpotqa", "corpus/corpus-00000-of-00001.parquet", repo_type="dataset") +qp = hf_hub_download("BeIR/hotpotqa", "queries/queries-00000-of-00001.parquet", repo_type="dataset") +corpus = pd.read_parquet(cp) +queries = pd.read_parquet(qp) +print("corpus", len(corpus), "queries", len(queries), corpus.columns.tolist(), queries.columns.tolist()) +rng = np.random.default_rng(a.seed) +sel = np.sort(rng.choice(len(corpus), size=a.n, replace=False)) +rng.shuffle(sel) # random order: prefixes of the base file are random samples +texts = [(corpus.iloc[i]["title"] + ". " + corpus.iloc[i]["text"]).strip() for i in sel] +qsel = rng.choice(len(queries), size=a.queries, replace=False) +qtexts = [queries.iloc[i]["text"] for i in qsel] + +dev = "cuda" if torch.cuda.is_available() else "cpu" +model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", device=dev) +model.max_seq_length = a.max_seq +if dev == "cuda": + model.half() +def enc(ts, bs=512): + return model.encode(ts, batch_size=bs, convert_to_numpy=True, normalize_embeddings=False, show_progress_bar=False).astype(np.float32) + +t0 = time.time() +qv = enc(qtexts) +qv /= np.linalg.norm(qv, axis=1, keepdims=True) +qv.tofile(os.path.join(a.out, "queries.f32")) +# stream in slices so a crash does not lose everything +out = open(os.path.join(a.out, "base.f32"), "wb") +step = 20000 +for s in range(0, a.n, step): + v = enc(texts[s:s + step]) + nrm = np.linalg.norm(v, axis=1, keepdims=True) + nrm[nrm == 0] = 1 + (v / nrm).astype(np.float32).tofile(out) + if (s // step) % 5 == 0: + print(s + step, "done", round(time.time() - t0), "s", flush=True) +out.close() +json.dump({"name": "text384", "n": a.n, "d": 384, "q": a.queries, "source": "BeIR/hotpotqa corpus (seeded random sample) + queries", + "model": "sentence-transformers/all-MiniLM-L6-v2", "device": dev, "max_seq_length": a.max_seq, + "fp16": dev == "cuda", "seed": a.seed, "metric": "cosine (vectors L2-normalised)"}, + open(os.path.join(a.out, "meta.json"), "w"), indent=1) +print("done", round(time.time() - t0), "s") diff --git a/benchmarks/competitive/report.py b/benchmarks/competitive/report.py new file mode 100644 index 0000000..875ae3a --- /dev/null +++ b/benchmarks/competitive/report.py @@ -0,0 +1,367 @@ +"""Turn results/raw/*.json into results/*.md tables, results/summary.json and results/plots/*.png. + + python report.py [--raw results/raw] [--out results] +Nothing here measures anything; every number in the tables is read from the raw JSON files. +""" +import argparse +import glob +import json +import os +import statistics +from collections import defaultdict + +HERE = os.path.dirname(os.path.abspath(__file__)) + +DISPLAY = { + "isong_py": "ISONGraph Python (exact)", "isong_py_ivf": "ISONGraph Python (IVF)", + "rust_default_exact": "ISONGraph Rust (exact)", "rust_default_ivf": "ISONGraph Rust (IVF)", + "rust_native_exact": "ISONGraph Rust, target-cpu=native (exact)", "rust_native_ivf": "ISONGraph Rust, target-cpu=native (IVF)", + "cpp_default_exact": "ISONGraph C++ (exact)", "cpp_default_ivf": "ISONGraph C++ (IVF)", + "cpp_avx2_exact": "ISONGraph C++, /arch:AVX2 (exact)", "cpp_avx2_ivf": "ISONGraph C++, /arch:AVX2 (IVF)", + "csharp_exact": "ISONGraph C# (exact)", "csharp_ivf": "ISONGraph C# (IVF)", + "js_exact": "ISONGraph JS (exact)", "js_ivf": "ISONGraph JS (IVF)", + "numpy_bf": "numpy brute force", "faiss_flat": "faiss IndexFlatIP", "faiss_ivfflat": "faiss IVFFlat", + "faiss_hnsw": "faiss HNSWFlat", "hnswlib": "hnswlib", "usearch": "usearch (HNSW)", "chromadb": "chromadb (HNSW)", + "qdrant_local": "qdrant-client local mode (exact numpy)", "lancedb_ivfflat": "lancedb IVF_FLAT", "lancedb_flat": "lancedb (no index)", + "sqlite_vec": "sqlite-vec (vec0, exact)", +} +OURS = lambda s: s.startswith(("isong", "rust", "cpp", "csharp", "js_")) # noqa: E731 +EXACT_SYSTEMS = {"numpy_bf", "isong_py", "faiss_flat", "lancedb_flat", "sqlite_vec", "qdrant_local"} + + +def load(raw): + recs = defaultdict(dict) # (system, ds, n) -> merged dict + status = {} + for f in sorted(glob.glob(os.path.join(raw, "*.json"))): + if f.endswith(".meta.json"): + continue + try: + d = json.load(open(f)) + except Exception: # noqa: BLE001 + continue + runs = d["runs"] if "runs" in d else [d] + for r in runs: + sysname = r.get("system") or d.get("system") + if not sysname: + continue + key = (sysname, r.get("ds", d.get("ds")), r.get("n", d.get("n"))) + st = d.get("status", "ok") + if st != "ok": + status[key] = st + continue + m = recs[key] + for k, v in r.items(): + if k in ("filter", "mutation", "saveload", "sweep", "build") and k in m and k != "filter": + continue + m[k] = v + m.setdefault("_files", []).append(os.path.basename(f)) + for kk in ("load_before_pct", "load_contaminated", "wall_s"): + if kk in d: + m.setdefault(kk, d[kk]) + return recs, status + + +def fmt(x, nd=1): + if x is None: + return "-" + if isinstance(x, str): + return x + if x >= 100000: + return f"{x:,.0f}" + if x >= 100: + return f"{x:.0f}" + if x >= 10: + return f"{x:.1f}" + if x >= 1: + return f"{x:.2f}" + return f"{x:.3f}" + + +def multi_qps(p): + if "batch" in p and p["batch"]: + return p["batch"]["qps_batch"], "batch" + if "allthreads" in p: + return p["allthreads"]["qps"], "auto-threads loop" + return None, None + + +def best_at(rec, thr): + """highest-QPS sweep point with recall >= thr; returns the point or None.""" + ok = [p for p in rec.get("sweep", []) if p.get("recall") is not None and p["recall"] >= thr] + return max(ok, key=lambda p: p["qps_1t"]) if ok else None + + +def interp_at(rec, thr, getter): + """QPS at recall == thr by log-linear interpolation between the two measured points that bracket it. + No extrapolation: None when the sweep never reaches thr. Exact systems (one point at recall ~1.0) return that point.""" + import math + pts = [(p["recall"], getter(p)) for p in rec.get("sweep", []) if p.get("recall") is not None and getter(p)] + if not pts: + return None + pts.sort() + if len(pts) == 1: + return pts[0][1] if pts[0][0] >= thr else None + best = None + for (r1, q1), (r2, q2) in zip(pts, pts[1:]): + if r1 <= thr <= r2: + if r2 == r1: + v = max(q1, q2) + elif r2 - r1 > 0.05: + v = q2 # bracket too wide to interpolate honestly: report the measured point above (conservative) + v = (v, "*") + else: + f = (thr - r1) / (r2 - r1) + v = math.exp(math.log(q1) * (1 - f) + math.log(q2) * f) + if isinstance(v, tuple): + best = v if best is None else best + else: + best = v if best is None or isinstance(best, tuple) else max(best, v) + if best is None: + above = [q for r, q in pts if r >= thr] + if above and pts[0][0] >= thr: # lowest measured point already above thr: report it (a lower bound) + return max(above) + return best + + +def name(s): + return DISPLAY.get(s, s) + + +def table(headers, rows): + out = ["| " + " | ".join(headers) + " |", "|" + "|".join("---" for _ in headers) + "|"] + for r in rows: + out.append("| " + " | ".join(str(c) for c in r) + " |") + return "\n".join(out) + + +def build_median(rec): + b = rec.get("build", {}) + runs = b.get("total_s_runs") + return statistics.median(runs) if runs else b.get("total_s") + + +def section_cell(recs, ds, n): + keys = sorted([k for k in recs if k[1] == ds and k[2] == n and "sweep" in recs[k]], key=lambda k: (not OURS(k[0]), k[0])) + if not keys: + return "" + md = [f"### {ds}, N = {n:,}\n"] + # headline: qps at recall thresholds + rows = [] + for k in keys: + rec = recs[k] + row = [("**" + name(k[0]) + "**") if OURS(k[0]) else name(k[0])] + def _f(v): + if not v: + return "-" + return fmt(v[0]) + "*" if isinstance(v, tuple) else fmt(v) + for thr in (0.90, 0.95, 0.99): + v = interp_at(rec, thr, lambda p: p["qps_1t"]) + row.append(_f(v)) + for thr in (0.90, 0.95): + v = interp_at(rec, thr, lambda p: multi_qps(p)[0]) + mode = next((multi_qps(p)[1] for p in rec.get("sweep", []) if multi_qps(p)[1]), None) + row.append((_f(v) + (" †" if mode and mode != "batch" else "")) if v else "-") + p95 = best_at(rec, 0.95) + row.append(f"{fmt(p95['median_ms'])} / {fmt(p95['p95_ms'])} / {fmt(p95['p99_ms'])} (recall {p95['recall']:.3f})" if p95 else "-") + rows.append(row) + md.append("Queries/second at recall@10 = 0.90 / 0.95 / 0.99, log-linearly interpolated between the two measured sweep points that bracket the target (no extrapolation; `*` = the two bracketing points are more than 0.05 recall apart, so the measured point above the target is shown instead, a conservative value; `-` = never reached; exact scans appear at their single recall-1.0 point). 1T = one pinned P-core, one query at a time. All-core = batched call on 32 logical CPUs (`†` = the ISONGraph ports have no batch API, so this is the same one-query loop with automatic scan threads). Latency columns are for the first measured sweep point with recall >= 0.95.\n") + md.append(table(["system", "QPS 1T @0.90", "@0.95", "@0.99", "all-core QPS @0.90", "@0.95", "1T latency ms median / p95 / p99"], rows)) + md.append("") + # build + memory + rows = [] + for k in keys: + rec = recs[k] + b = rec.get("build", {}) + mb = rec.get("mem_baseline", {}) + ma = rec.get("mem_after_build", {}) + mq = rec.get("mem_after_first_query", ma) + base = mb.get("rss_mb", 0) + srv = "" + if mq.get("server_rss_mb"): + srv = f" (+{mq['server_rss_mb']:.0f} server)" + reps = b.get("total_s_runs") + rows.append([("**" + name(k[0]) + "**") if OURS(k[0]) else name(k[0]), fmt(b.get("ingest_s")), fmt(b.get("index_s")), fmt(b.get("total_s")), + (fmt(statistics.median(reps)) + f" ({fmt(min(reps))}-{fmt(max(reps))})") if reps else "-", fmt(b.get("ready_s")), + f"{mq.get('rss_mb', 0) - base:,.0f}{srv}", f"{ma.get('peak_mb', 0) - base:,.0f}"]) + md.append("Build and memory (RSS growth over the process baseline after imports; the raw vectors are streamed in from disk and are not held by the harness).\n") + md.append(table(["system", "ingest s", "index s", "total build s", "3 builds: median (min-max) s", "ready s (incl. first query)", "steady RSS +MB", "peak RSS +MB (build)"], rows)) + md.append("") + # full sweeps + md.append("
Full accuracy sweeps (recall@10, single-thread QPS, latency, all-core QPS)\n") + for k in keys: + rec = recs[k] + rows = [] + for p in rec["sweep"]: + q, mode = multi_qps(p) + rows.append([p.get("knob") if p.get("knob") is not None else "-", f"{p['recall']:.4f}", f"{fmt(p['qps_1t'])} ({fmt(p.get('qps_1t_min'))}-{fmt(p.get('qps_1t_max'))})", + fmt(p["median_ms"]), fmt(p["p95_ms"]), fmt(p["p99_ms"]), (fmt(q) + ("" if mode == "batch" else " †")) if q else "-", p.get("nq")]) + md.append(f"**{name(k[0])}** ({rec.get('knob_name') or 'exact'})\n") + md.append(table(["knob", "recall@10", "QPS 1T (min-max)", "median ms", "p95 ms", "p99 ms", "all-core QPS", "queries/run"], rows)) + md.append("") + md.append("
\n") + return "\n".join(md) + + +def section_exact(recs): + md = [] + for ds in ("text384", "glove", "sift"): + rows = [] + ns = sorted({k[2] for k in recs if k[1] == ds}) + for n in ns: + for k in sorted([k for k in recs if k[1] == ds and k[2] == n], key=lambda k: (not OURS(k[0]), k[0])): + rec = recs[k] + sw = rec.get("sweep", []) + ex = None + if k[0].endswith("_exact") or k[0] in EXACT_SYSTEMS: + ex = sw[0] if sw else None + else: + continue + if not ex: + continue + la = rec.get("loop_allthreads") + b = ex.get("batch") + rows.append([f"{n:,}", ("**" + name(k[0]) + "**") if OURS(k[0]) else name(k[0]), fmt(ex["median_ms"]), fmt(ex["p95_ms"]), fmt(ex["p99_ms"]), + f"{fmt(ex['qps_1t'])} ({fmt(ex.get('qps_1t_min'))}-{fmt(ex.get('qps_1t_max'))})", + (fmt(la["median_ms"]) + f" ms ({fmt(la['qps'])} QPS)") if la else "-", fmt(b["qps_batch"]) if b else "-", f"{ex['recall']:.3f}", fmt(rec.get("build", {}).get("first_query_ms"))]) + if rows: + md.append(f"#### {ds}\n") + md.append(table(["N", "system", "median ms (1T)", "p95", "p99", "QPS 1T (min-max)", "all-threads single query", "batch QPS", "recall", "first query ms"], rows)) + md.append("") + return "\n".join(md) + + +def section_mutation(recs): + rows = [] + for k in sorted(recs, key=lambda k: (k[1], k[2], not OURS(k[0]), k[0])): + m = recs[k].get("mutation") + if not m: + continue + if "skipped" in m: + continue + + def cell(key): + v = m.get(key) + if v is None: + return "-" + if v.get("unsupported"): + return "unsupported" + if v.get("error"): + return "error" + return f"{fmt(v['seconds'])} s ({fmt(v['per_s'])}/s)" + rows.append([k[1], f"{k[2]:,}", ("**" + name(k[0]) + "**") if OURS(k[0]) else name(k[0]), cell("insert_10k"), cell("update_10k"), cell("delete_10k"), + fmt(m.get("recall_before")) if m.get("recall_before") is not None else "-", f"{m['recall_after']:.3f}" if m.get("recall_after") is not None else "n/a", + fmt(m.get("first_query_after_ms"))]) + return table(["dataset", "N", "system", "insert 10k", "update 10k", "delete 10k", "recall@10 before", "after", "first query after ms"], rows) + + +def section_filter(recs): + md = [] + for ds, n in sorted({(k[1], k[2]) for k in recs if recs[k].get("filter")}): + rows = [] + for k in sorted([k for k in recs if k[1] == ds and k[2] == n and recs[k].get("filter")], key=lambda k: (not OURS(k[0]), k[0])): + f = recs[k]["filter"] + row = [("**" + name(k[0]) + "**") if OURS(k[0]) else name(k[0]), fmt(f.get("build_with_metadata_s"))] + for lab in ("f10", "f01"): + pts = f.get("subsets", {}).get(lab) + if pts is None: + row += ["-", "-"] + continue + if isinstance(pts, dict) and pts.get("unsupported"): + row += ["unsupported", "-"] + continue + ok = [p for p in pts if p["recall"] >= 0.9] + best = max(ok, key=lambda p: p["qps_1t"]) if ok else None + top = max(pts, key=lambda p: p["recall"]) + row.append(f"{fmt(best['qps_1t'])} @ {best['recall']:.3f}" if best else f"- (max recall {top['recall']:.3f} at {fmt(top['qps_1t'])} QPS)") + row.append(f"{top['recall']:.3f}") + rows.append(row) + md.append(f"#### {ds}, N = {n:,}\n") + md.append(table(["system", "build with metadata s", "10% subset: QPS 1T @ recall>=0.90", "max recall", "1% subset: QPS 1T @ recall>=0.90", "max recall"], rows)) + md.append("") + return "\n".join(md) + + +def section_saveload(recs): + rows = [] + for k in sorted(recs, key=lambda k: (k[1], k[2], not OURS(k[0]), k[0])): + s = recs[k].get("saveload") + if not s: + continue + if s.get("unsupported") or s.get("error"): + rows.append([k[1], f"{k[2]:,}", name(k[0]), s.get("error", "unsupported")[:60], "-", "-", "-", "-"]) + continue + rows.append([k[1], f"{k[2]:,}", ("**" + name(k[0]) + "**") if OURS(k[0]) else name(k[0]), fmt(s.get("save_s")), fmt(s.get("disk_mb")), fmt(s.get("load_s")), + fmt(s.get("first_query_after_load_ms")), fmt(s.get("time_to_ready_s"))]) + return table(["dataset", "N", "system", "save s", "on-disk MB", "load s", "first query after load ms", "time to ready s"], rows) + + +def plots(recs, out): + try: + import matplotlib + matplotlib.use("Agg") + import matplotlib.pyplot as plt + except Exception: # noqa: BLE001 + return [] + os.makedirs(out, exist_ok=True) + made = [] + cells = sorted({(k[1], k[2]) for k in recs if "sweep" in recs[k] and len(recs[k]["sweep"]) > 2}) + for ds, n in cells: + for mode in ("1t", "batch"): + fig, ax = plt.subplots(figsize=(8, 5.2)) + drew = False + for k in sorted([k for k in recs if k[1] == ds and k[2] == n and "sweep" in recs[k]], key=lambda k: k[0]): + pts = recs[k]["sweep"] + if len(pts) < 3 and not OURS(k[0]): + continue + xs, ys = [], [] + for p in pts: + q = p["qps_1t"] if mode == "1t" else (multi_qps(p)[0] if multi_qps(p)[1] == "batch" else None) + if q: + xs.append(p["recall"]) + ys.append(q) + if len(xs) < 2: + continue + ax.plot(xs, ys, marker="o" if OURS(k[0]) else "s", ms=3, lw=2.4 if OURS(k[0]) else 1.2, ls="-" if OURS(k[0]) else "--", label=name(k[0])) + drew = True + if not drew: + plt.close(fig) + continue + ax.set_yscale("log") + ax.set_xlim(0.5, 1.005) + ax.set_xlabel("recall@10 (vs exact float64 cosine)") + ax.set_ylabel("queries / second (" + ("1 thread" if mode == "1t" else "batched, all cores") + ")") + ax.set_title(f"{ds}, N={n:,}") + ax.grid(True, alpha=0.3) + ax.legend(fontsize=6, loc="lower left") + f = os.path.join(out, f"{ds}_{n}_{mode}.png") + fig.savefig(f, dpi=120, bbox_inches="tight") + plt.close(fig) + made.append(os.path.relpath(f, os.path.dirname(out))) + return made + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--raw", default=os.path.join(HERE, "results", "raw")) + ap.add_argument("--out", default=os.path.join(HERE, "results")) + a = ap.parse_args() + recs, status = load(a.raw) + os.makedirs(a.out, exist_ok=True) + parts = {} + parts["cells"] = "\n".join(section_cell(recs, ds, n) for ds, n in [("text384", 100000), ("glove", 100000), ("sift", 100000), ("text384", 1000000), ("glove", 1000000), ("sift", 1000000)]) + parts["exact"] = section_exact(recs) + parts["mutation"] = section_mutation(recs) + parts["filter"] = section_filter(recs) + parts["saveload"] = section_saveload(recs) + parts["failures"] = table(["system", "dataset", "N", "status"], [[name(k[0]), k[1], f"{k[2]:,}", v] for k, v in sorted(status.items())]) + for k, v in parts.items(): + open(os.path.join(a.out, f"tables_{k}.md"), "w", encoding="utf-8").write(v + "\n") + made = plots(recs, os.path.join(a.out, "plots")) + summary = {f"{k[0]}|{k[1]}|{k[2]}": {kk: vv for kk, vv in v.items() if kk in ("build", "mem_baseline", "mem_after_build", "mem_after_first_query", "sweep", "mutation", "saveload", "filter", "loop_allthreads", "version", "load_before_pct", "load_contaminated")} + for k, v in recs.items()} + json.dump(summary, open(os.path.join(a.out, "summary.json"), "w"), indent=1) + print("systems/cells:", len(recs), "failures:", len(status), "plots:", len(made)) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/competitive/results/environment.json b/benchmarks/competitive/results/environment.json new file mode 100644 index 0000000..46eac41 --- /dev/null +++ b/benchmarks/competitive/results/environment.json @@ -0,0 +1,37 @@ +{ + "python": "3.12.7", + "platform": "Windows-11-10.0.26300-SP0", + "cpu_model": "Intel(R) Core(TM) i9-14900HX", + "cpu_cores": 24, + "cpu_logical": 32, + "ram_gb": 47.7, + "note": "i9-14900HX is a hybrid CPU (8 P-cores with SMT + 16 E-cores); single-thread loops are not pinned to a core type", + "packages": { + "numpy": "2.5.5", + "faiss-cpu": "1.16.0", + "hnswlib": "0.8.0", + "usearch": "2.26.6", + "chromadb": "1.5.9", + "qdrant-client": "1.20.0", + "lancedb": "0.41.0", + "sqlite-vec": "0.1.9", + "pyarrow": "26.0.1", + "threadpoolctl": "3.8.0", + "psutil": "7.2.2" + }, + "rustc": "rustc 1.97.1 (8bab26f4f 2026-07-14)", + "cargo": "cargo 1.97.1 (c980f4866 2026-06-30)", + "rust_build": "cargo build --release --example competitive_bench (default flags = baseline x86-64) and RUSTFLAGS='-C target-cpu=native'", + "msvc": [ + "Microsoft (R) C/C++ Optimizing Compiler Version 19.41.34123 for x64" + ], + "cpp_build": "cl /std:c++17 /O2 /EHsc /MD /DNDEBUG (default SSE2) and the same with /arch:AVX2", + "dotnet": "8.0.403", + "node": "v24.18.0", + "sqlite3": "3.45.3", + "numpy_blas": "{'name': 'scipy-openblas', 'found': True, 'version': '0.3.34.237.2', 'detection method': 'pkgconfig', 'include directory': 'C:/Users/runneradmin/AppData/Local/Temp/cibw-run-4tkbs59h/cp312-win_amd64/build/venv/Lib/site-packages/scipy_openblas64/include', 'lib directory': 'C:/Users/runneradmin/AppData/Local/Temp/cibw-run-4tkbs59h/cp312-win_amd64/build/venv/Lib/site-packages/scipy_openblas64/lib', 'openblas configuration': 'OpenBLAS 0.3.34.237.2 USE64BITINT DYNAMIC_ARCH NO_AFFINITY SkylakeX MAX_THREADS=24', 'pc file directory': 'D:/a/numpy-release/numpy-release/.openblas'}", + "faiss_avx": [ + "avx2", + "sse4" + ] +} \ No newline at end of file diff --git a/benchmarks/competitive/results/plots/glove_1000000_1t.png b/benchmarks/competitive/results/plots/glove_1000000_1t.png new file mode 100644 index 0000000..647fc42 Binary files /dev/null and b/benchmarks/competitive/results/plots/glove_1000000_1t.png differ diff --git a/benchmarks/competitive/results/plots/glove_1000000_batch.png b/benchmarks/competitive/results/plots/glove_1000000_batch.png new file mode 100644 index 0000000..721b0e9 Binary files /dev/null and b/benchmarks/competitive/results/plots/glove_1000000_batch.png differ diff --git a/benchmarks/competitive/results/plots/glove_100000_1t.png b/benchmarks/competitive/results/plots/glove_100000_1t.png new file mode 100644 index 0000000..5172890 Binary files /dev/null and b/benchmarks/competitive/results/plots/glove_100000_1t.png differ diff --git a/benchmarks/competitive/results/plots/glove_100000_batch.png b/benchmarks/competitive/results/plots/glove_100000_batch.png new file mode 100644 index 0000000..c3ffe72 Binary files /dev/null and b/benchmarks/competitive/results/plots/glove_100000_batch.png differ diff --git a/benchmarks/competitive/results/plots/sift_1000000_1t.png b/benchmarks/competitive/results/plots/sift_1000000_1t.png new file mode 100644 index 0000000..019bd7a Binary files /dev/null and b/benchmarks/competitive/results/plots/sift_1000000_1t.png differ diff --git a/benchmarks/competitive/results/plots/sift_1000000_batch.png b/benchmarks/competitive/results/plots/sift_1000000_batch.png new file mode 100644 index 0000000..da18281 Binary files /dev/null and b/benchmarks/competitive/results/plots/sift_1000000_batch.png differ diff --git a/benchmarks/competitive/results/plots/sift_100000_1t.png b/benchmarks/competitive/results/plots/sift_100000_1t.png new file mode 100644 index 0000000..a8c2483 Binary files /dev/null and b/benchmarks/competitive/results/plots/sift_100000_1t.png differ diff --git a/benchmarks/competitive/results/plots/sift_100000_batch.png b/benchmarks/competitive/results/plots/sift_100000_batch.png new file mode 100644 index 0000000..f5ace2b Binary files /dev/null and b/benchmarks/competitive/results/plots/sift_100000_batch.png differ diff --git a/benchmarks/competitive/results/plots/text384_1000000_1t.png b/benchmarks/competitive/results/plots/text384_1000000_1t.png new file mode 100644 index 0000000..3615781 Binary files /dev/null and b/benchmarks/competitive/results/plots/text384_1000000_1t.png differ diff --git a/benchmarks/competitive/results/plots/text384_1000000_batch.png b/benchmarks/competitive/results/plots/text384_1000000_batch.png new file mode 100644 index 0000000..2479665 Binary files /dev/null and b/benchmarks/competitive/results/plots/text384_1000000_batch.png differ diff --git a/benchmarks/competitive/results/plots/text384_100000_1t.png b/benchmarks/competitive/results/plots/text384_100000_1t.png new file mode 100644 index 0000000..5e605d6 Binary files /dev/null and b/benchmarks/competitive/results/plots/text384_100000_1t.png differ diff --git a/benchmarks/competitive/results/plots/text384_100000_batch.png b/benchmarks/competitive/results/plots/text384_100000_batch.png new file mode 100644 index 0000000..86bbd21 Binary files /dev/null and b/benchmarks/competitive/results/plots/text384_100000_batch.png differ diff --git a/benchmarks/competitive/results/plots/text384_10000_1t.png b/benchmarks/competitive/results/plots/text384_10000_1t.png new file mode 100644 index 0000000..3ee9443 Binary files /dev/null and b/benchmarks/competitive/results/plots/text384_10000_1t.png differ diff --git a/benchmarks/competitive/results/raw/chromadb__glove__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/chromadb__glove__100000__core+mutation+saveload.json new file mode 100644 index 0000000..1123c45 --- /dev/null +++ b/benchmarks/competitive/results/raw/chromadb__glove__100000__core+mutation+saveload.json @@ -0,0 +1,318 @@ +{ + "system": "chromadb", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 08:21:01", + "mem_baseline": { + "rss_mb": 35.1640625, + "peak_mb": 35.1640625 + }, + "version": "chromadb 1.5.9", + "build": { + "ingest_s": 9.46120769996196, + "index_s": 2.3999600671231747e-06, + "total_s": 9.461210099922027, + "first_query_ms": 87.71570003591478, + "ready_s": 9.548925799957942, + "repeat_error": "InternalError: Collection [bench] already exists", + "total_s_runs": [ + 9.461210099922027, + 9.34063710004557 + ] + }, + "mem_after_build": { + "rss_mb": 212.01171875, + "peak_mb": 237.36328125 + }, + "mem_after_first_query": { + "rss_mb": 299.6171875, + "peak_mb": 299.6171875 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.5396, + "qps_1t_runs": [ + 1411.6182952327244, + 1485.575176072503, + 1390.2194440882397 + ], + "qps_1t": 1411.6182952327244, + "qps_1t_min": 1390.2194440882397, + "qps_1t_max": 1485.575176072503, + "median_ms": 0.7173999911174178, + "p95_ms": 0.801014990429394, + "p99_ms": 0.9504890191601608, + "mean_ms": 0.7002858836591864, + "knob": 10, + "batch": { + "recall": 0.5396, + "qps_batch_runs": [ + 23630.43884378117, + 23243.81365924262, + 23763.55265438207 + ], + "qps_batch": 23630.43884378117, + "qps_batch_min": 23243.81365924262, + "qps_batch_max": 23763.55265438207, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6245, + "qps_1t_runs": [ + 1357.176113461374, + 1403.578605862074, + 1449.7641853361804 + ], + "qps_1t": 1403.578605862074, + "qps_1t_min": 1357.176113461374, + "qps_1t_max": 1449.7641853361804, + "median_ms": 0.714199966751039, + "p95_ms": 0.8147100306814536, + "p99_ms": 0.9422010148409754, + "mean_ms": 0.7130186498689, + "knob": 16, + "batch": { + "recall": 0.6245, + "qps_batch_runs": [ + 20451.75889459749, + 20916.43257128759, + 20762.114948411614 + ], + "qps_batch": 20762.114948411614, + "qps_batch_min": 20451.75889459749, + "qps_batch_max": 20916.43257128759, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7422, + "qps_1t_runs": [ + 1291.2050983481727, + 1367.012231102707, + 1381.4532169504039 + ], + "qps_1t": 1367.012231102707, + "qps_1t_min": 1291.2050983481727, + "qps_1t_max": 1381.4532169504039, + "median_ms": 0.7527499983552843, + "p95_ms": 0.8599049731856212, + "p99_ms": 0.9532359824515887, + "mean_ms": 0.7432893500857366, + "knob": 32, + "batch": { + "recall": 0.7422, + "qps_batch_runs": [ + 15781.184411407197, + 15763.149219952671, + 14749.839042658305 + ], + "qps_batch": 15763.149219952671, + "qps_batch_min": 14749.839042658305, + "qps_batch_max": 15781.184411407197, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8288, + "qps_1t_runs": [ + 1159.9888509017046, + 1131.9797228966868, + 1153.0134124115882 + ], + "qps_1t": 1153.0134124115882, + "qps_1t_min": 1131.9797228966868, + "qps_1t_max": 1159.9888509017046, + "median_ms": 0.8609999786131084, + "p95_ms": 0.966024966328405, + "p99_ms": 1.1247230076696728, + "mean_ms": 0.870925967086805, + "knob": 64, + "batch": { + "recall": 0.8288, + "qps_batch_runs": [ + 9511.412984422934, + 9913.391654460556, + 10549.051773822624 + ], + "qps_batch": 9913.391654460556, + "qps_batch_min": 9511.412984422934, + "qps_batch_max": 10549.051773822624, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8957, + "qps_1t_runs": [ + 1050.9036703939914, + 1034.3695134095262, + 1041.6991649771956 + ], + "qps_1t": 1041.6991649771956, + "qps_1t_min": 1034.3695134095262, + "qps_1t_max": 1050.9036703939914, + "median_ms": 0.9402499999850988, + "p95_ms": 1.0893999744439498, + "p99_ms": 1.3479710184037723, + "mean_ms": 0.9594348505391584, + "knob": 128, + "batch": { + "recall": 0.8957, + "qps_batch_runs": [ + 5832.417157561866, + 6356.371897821059, + 6482.120690483515 + ], + "qps_batch": 6356.371897821059, + "qps_batch_min": 5832.417157561866, + "qps_batch_max": 6482.120690483515, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.947, + "qps_1t_runs": [ + 980.5784194517371, + 933.3874082171179, + 955.7744524309082 + ], + "qps_1t": 955.7744524309082, + "qps_1t_min": 933.3874082171179, + "qps_1t_max": 980.5784194517371, + "median_ms": 1.0254000080749393, + "p95_ms": 1.205914985621348, + "p99_ms": 1.4304100058507199, + "mean_ms": 1.0458148994560663, + "knob": 256, + "batch": { + "recall": 0.947, + "qps_batch_runs": [ + 3683.8267298157884, + 3685.999303002262, + 3558.720760679231 + ], + "qps_batch": 3683.8267298157884, + "qps_batch_min": 3558.720760679231, + "qps_batch_max": 3685.999303002262, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.97665, + "qps_1t_runs": [ + 748.958142970622, + 776.0253786375672, + 752.5215490626313 + ], + "qps_1t": 752.5215490626313, + "qps_1t_min": 748.958142970622, + "qps_1t_max": 776.0253786375672, + "median_ms": 1.2944499612785876, + "p95_ms": 1.5026050299638882, + "p99_ms": 1.9172779883956554, + "mean_ms": 1.3175571169025109, + "knob": 512, + "batch": { + "recall": 0.97665, + "qps_batch_runs": [ + 2012.4269376767793, + 2069.4773203670516, + 2028.615858073334 + ], + "qps_batch": 2028.615858073334, + "qps_batch_min": 2012.4269376767793, + "qps_batch_max": 2069.4773203670516, + "threads": 32 + } + } + ], + "knob_name": "ef_search", + "mem_after_sweep": { + "rss_mb": 856.9453125, + "peak_mb": 856.9609375 + }, + "ref_knob": 512, + "ref_recall_pre": 0.97665, + "mutation": { + "ref_knob": 512, + "recall_before": 0.97665, + "insert_10k": { + "seconds": 2.4239940000115894, + "per_s": 4125.422752676858, + "batch": 1000 + }, + "update_10k": { + "seconds": 3.4582213000394404, + "per_s": 2891.659940873637, + "batch": 1000 + }, + "delete_10k": { + "seconds": 1.952752600016538, + "per_s": 5120.976410383634, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 358.84765625, + "peak_mb": 990.73828125 + }, + "first_query_after_ms": 89.03720002854243, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9762, + "qps_1t_runs": [ + 737.2386117471513, + 732.425701778321, + 784.6157871231162 + ], + "qps_1t": 737.2386117471513, + "qps_1t_min": 732.425701778321, + "qps_1t_max": 784.6157871231162, + "median_ms": 1.318100024946034, + "p95_ms": 1.5458050125744194, + "p99_ms": 1.7699079727754, + "mean_ms": 1.3320827665156685 + }, + "recall_after": 0.9762 + }, + "saveload": { + "save_s": 1.100008375942707e-06, + "disk_mb": 76.30125045776367, + "load_s": 0.011646299972198904, + "first_query_after_load_ms": 87.3268999857828, + "time_to_ready_s": 0.0989731999579817 + }, + "mem_end": { + "rss_mb": 537.671875, + "peak_mb": 990.73828125 + }, + "status": "ok", + "finished": "2026-11-16 08:22:24", + "load_before_pct": 3.9437499999999974, + "load_contaminated": false, + "wall_s": 83.35406756401062, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/chromadb__glove__100000__filter.json b/benchmarks/competitive/results/raw/chromadb__glove__100000__filter.json new file mode 100644 index 0000000..eeddfce --- /dev/null +++ b/benchmarks/competitive/results/raw/chromadb__glove__100000__filter.json @@ -0,0 +1,170 @@ +{ + "system": "chromadb", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 08:49:57", + "filter": { + "build_with_metadata_s": 11.126682000001892, + "mem": { + "rss_mb": 216.97265625, + "peak_mb": 246.265625 + }, + "subsets": { + "f10": [ + { + "knob": 16, + "nq": 76, + "recall": 0.825, + "qps_1t": 5.4099673236627455, + "qps_1t_runs": [ + 5.4099673236627455 + ], + "median_ms": 39.79289997369051, + "p95_ms": 568.1091250007739, + "p99_ms": 686.7677500267746, + "mean_ms": 184.84400000459956 + }, + { + "knob": 32, + "nq": 86, + "recall": 0.9116279069767442, + "qps_1t": 4.229212082488843, + "qps_1t_runs": [ + 4.229212082488843 + ], + "median_ms": 39.50435001752339, + "p95_ms": 685.3640749905026, + "p99_ms": 1006.4198650361511, + "mean_ms": 236.45066279379196 + }, + { + "knob": 64, + "nq": 30, + "recall": 0.95, + "qps_1t": 1.8577230019097792, + "qps_1t_runs": [ + 1.8577230019097792 + ], + "median_ms": 540.5647500010673, + "p95_ms": 557.7135399827966, + "p99_ms": 565.5559779866599, + "mean_ms": 538.2933833364708 + }, + { + "knob": 128, + "nq": 30, + "recall": 0.9666666666666667, + "qps_1t": 2.5395738405206187, + "qps_1t_runs": [ + 2.5395738405206187 + ], + "median_ms": 415.0752999994438, + "p95_ms": 886.5208649687697, + "p99_ms": 968.2058709819103, + "mean_ms": 393.76685333748657 + }, + { + "knob": 512, + "nq": 30, + "recall": 1.0, + "qps_1t": 1.7990575744727344, + "qps_1t_runs": [ + 1.7990575744727344 + ], + "median_ms": 675.4120500118006, + "p95_ms": 1033.7916499964192, + "p99_ms": 1099.1252370225266, + "mean_ms": 555.8465800034659 + } + ], + "f01": [ + { + "knob": 16, + "nq": 146, + "recall": 0.9363013698630137, + "qps_1t": 28.028240179833855, + "qps_1t_runs": [ + 28.028240179833855 + ], + "median_ms": 16.321150003932416, + "p95_ms": 94.01652500673663, + "p99_ms": 111.20478500670292, + "mean_ms": 35.67830136975542 + }, + { + "knob": 32, + "nq": 133, + "recall": 0.9827067669172932, + "qps_1t": 21.191251716703626, + "qps_1t_runs": [ + 21.191251716703626 + ], + "median_ms": 18.92760000191629, + "p95_ms": 139.8339799954556, + "p99_ms": 146.48944400949404, + "mean_ms": 47.18928420881188 + }, + { + "knob": 64, + "nq": 121, + "recall": 0.9991735537190083, + "qps_1t": 17.934054968587994, + "qps_1t_runs": [ + 17.934054968587994 + ], + "median_ms": 51.08679999830201, + "p95_ms": 130.09569997666404, + "p99_ms": 155.6541200377978, + "mean_ms": 55.759838014968075 + }, + { + "knob": 128, + "nq": 30, + "recall": 1.0, + "qps_1t": 12.792725270408223, + "qps_1t_runs": [ + 12.792725270408223, + 11.853814024425251, + 12.904388589706025 + ], + "median_ms": 85.76095002354123, + "p95_ms": 132.44221999775618, + "p99_ms": 164.50325596379116, + "mean_ms": 80.00782555577138 + }, + { + "knob": 512, + "nq": 30, + "recall": 1.0, + "qps_1t": 8.812886554855199, + "qps_1t_runs": [ + 8.812886554855199, + 8.328435750887204, + 8.848317974363184 + ], + "median_ms": 111.08534998493269, + "p95_ms": 136.53670500498265, + "p99_ms": 137.85027002100833, + "mean_ms": 115.51886555583526 + } + ] + } + }, + "mem_end": { + "rss_mb": 1205.97265625, + "peak_mb": 1206.0078125 + }, + "status": "ok", + "finished": "2026-11-16 08:52:26", + "load_before_pct": 3.146874999999998, + "load_contaminated": false, + "wall_s": 149.75718474388123, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/chromadb__sift__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/chromadb__sift__100000__core+mutation+saveload.json new file mode 100644 index 0000000..eeeea37 --- /dev/null +++ b/benchmarks/competitive/results/raw/chromadb__sift__100000__core+mutation+saveload.json @@ -0,0 +1,318 @@ +{ + "system": "chromadb", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 11:10:11", + "mem_baseline": { + "rss_mb": 35.38671875, + "peak_mb": 35.38671875 + }, + "version": "chromadb 1.5.9", + "build": { + "ingest_s": 9.058855700073764, + "index_s": 2.500019036233425e-06, + "total_s": 9.0588582000928, + "first_query_ms": 101.34489997290075, + "ready_s": 9.1602031000657, + "repeat_error": "InternalError: Collection [bench] already exists", + "total_s_runs": [ + 9.0588582000928, + 9.233196799992584 + ] + }, + "mem_after_build": { + "rss_mb": 224.796875, + "peak_mb": 254.6953125 + }, + "mem_after_first_query": { + "rss_mb": 323.96484375, + "peak_mb": 323.96484375 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7916, + "qps_1t_runs": [ + 1279.0723550829064, + 1280.9432647181459, + 1260.4226716388887 + ], + "qps_1t": 1279.0723550829064, + "qps_1t_min": 1260.4226716388887, + "qps_1t_max": 1280.9432647181459, + "median_ms": 0.7512000156566501, + "p95_ms": 0.990025029750541, + "p99_ms": 1.545555950142451, + "mean_ms": 0.7852919836198756, + "knob": 10, + "batch": { + "recall": 0.7916, + "qps_batch_runs": [ + 20821.68619806102, + 21362.728446020592, + 22498.35199659069 + ], + "qps_batch": 21362.728446020592, + "qps_batch_min": 20821.68619806102, + "qps_batch_max": 22498.35199659069, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.87145, + "qps_1t_runs": [ + 1240.157873965033, + 1307.064046966063, + 1258.3539772092918 + ], + "qps_1t": 1258.3539772092918, + "qps_1t_min": 1240.157873965033, + "qps_1t_max": 1307.064046966063, + "median_ms": 0.7659999828319997, + "p95_ms": 0.9340500109829011, + "p99_ms": 1.688574992585931, + "mean_ms": 0.7887037832891414, + "knob": 16, + "batch": { + "recall": 0.87145, + "qps_batch_runs": [ + 18336.259499059895, + 17760.191247496085, + 18543.67266745581 + ], + "qps_batch": 18336.259499059895, + "qps_batch_min": 17760.191247496085, + "qps_batch_max": 18543.67266745581, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.95235, + "qps_1t_runs": [ + 1116.0513738894722, + 1185.4238152440125, + 1162.8415392632346 + ], + "qps_1t": 1162.8415392632346, + "qps_1t_min": 1116.0513738894722, + "qps_1t_max": 1185.4238152440125, + "median_ms": 0.8177999989129603, + "p95_ms": 1.1075449932832273, + "p99_ms": 1.8296040198765695, + "mean_ms": 0.8665195499391606, + "knob": 32, + "batch": { + "recall": 0.95235, + "qps_batch_runs": [ + 13719.481874934498, + 12846.68381780063, + 10800.115777494071 + ], + "qps_batch": 12846.68381780063, + "qps_batch_min": 10800.115777494071, + "qps_batch_max": 13719.481874934498, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.98875, + "qps_1t_runs": [ + 1111.5342352092764, + 1128.7277940424103, + 1153.7014751985846 + ], + "qps_1t": 1128.7277940424103, + "qps_1t_min": 1111.5342352092764, + "qps_1t_max": 1153.7014751985846, + "median_ms": 0.8407999994233251, + "p95_ms": 1.0921349923592063, + "p99_ms": 1.7142300138948492, + "mean_ms": 0.8841286496802544, + "knob": 64, + "batch": { + "recall": 0.98875, + "qps_batch_runs": [ + 8209.87393334354, + 9026.484608662204, + 8413.49288739725 + ], + "qps_batch": 8413.49288739725, + "qps_batch_min": 8209.87393334354, + "qps_batch_max": 9026.484608662204, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99775, + "qps_1t_runs": [ + 1007.5157653753861, + 982.1403670349492, + 965.3772273263502 + ], + "qps_1t": 982.1403670349492, + "qps_1t_min": 965.3772273263502, + "qps_1t_max": 1007.5157653753861, + "median_ms": 0.952300033532083, + "p95_ms": 1.345434997347184, + "p99_ms": 2.211042952258152, + "mean_ms": 1.0155297335974562, + "knob": 128, + "batch": { + "recall": 0.99775, + "qps_batch_runs": [ + 5786.047582972032, + 5633.006261984994, + 5806.427076032027 + ], + "qps_batch": 5786.047582972032, + "qps_batch_min": 5633.006261984994, + "qps_batch_max": 5806.427076032027, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99975, + "qps_1t_runs": [ + 811.5561708338621, + 778.9418373912421, + 817.6880979812471 + ], + "qps_1t": 811.5561708338621, + "qps_1t_min": 778.9418373912421, + "qps_1t_max": 817.6880979812471, + "median_ms": 1.1875999625772238, + "p95_ms": 1.6578999930061404, + "p99_ms": 2.6186080311890714, + "mean_ms": 1.246317899650118, + "knob": 256, + "batch": { + "recall": 0.99975, + "qps_batch_runs": [ + 3651.678110394038, + 3650.97083880668, + 3659.8210930532878 + ], + "qps_batch": 3651.678110394038, + "qps_batch_min": 3650.97083880668, + "qps_batch_max": 3659.8210930532878, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9999, + "qps_1t_runs": [ + 764.3267603929414, + 783.3136339750433, + 757.292174812171 + ], + "qps_1t": 764.3267603929414, + "qps_1t_min": 757.292174812171, + "qps_1t_max": 783.3136339750433, + "median_ms": 1.235899981111288, + "p95_ms": 1.7222000140463938, + "p99_ms": 2.461191997281278, + "mean_ms": 1.3018210331210867, + "knob": 512, + "batch": { + "recall": 0.9999, + "qps_batch_runs": [ + 2148.9991358904995, + 2105.278892049367, + 2069.9918048063064 + ], + "qps_batch": 2105.278892049367, + "qps_batch_min": 2069.9918048063064, + "qps_batch_max": 2148.9991358904995, + "threads": 32 + } + } + ], + "knob_name": "ef_search", + "mem_after_sweep": { + "rss_mb": 943.78515625, + "peak_mb": 943.82421875 + }, + "ref_knob": 32, + "ref_recall_pre": 0.95235, + "mutation": { + "ref_knob": 32, + "recall_before": 0.95235, + "insert_10k": { + "seconds": 2.3344285000348464, + "per_s": 4283.703698721434, + "batch": 1000 + }, + "update_10k": { + "seconds": 3.343258799985051, + "per_s": 2991.093600066113, + "batch": 1000 + }, + "delete_10k": { + "seconds": 2.3275026999763213, + "per_s": 4296.450440251577, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 385.21484375, + "peak_mb": 1096.796875 + }, + "first_query_after_ms": 111.04819999309257, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.94235, + "qps_1t_runs": [ + 1222.9589997704738, + 1240.6607708217812, + 1198.929930712384 + ], + "qps_1t": 1222.9589997704738, + "qps_1t_min": 1198.929930712384, + "qps_1t_max": 1240.6607708217812, + "median_ms": 0.7790999952703714, + "p95_ms": 1.0326200048439205, + "p99_ms": 1.587519038585015, + "mean_ms": 0.8192627003105978 + }, + "recall_after": 0.94235 + }, + "saveload": { + "save_s": 1.00000761449337e-06, + "disk_mb": 88.51536178588867, + "load_s": 0.015021700004581362, + "first_query_after_load_ms": 108.12930000247434, + "time_to_ready_s": 0.1231510000070557 + }, + "mem_end": { + "rss_mb": 586.13671875, + "peak_mb": 1096.796875 + }, + "status": "ok", + "finished": "2026-11-16 11:11:34", + "load_before_pct": 6.4125000000000005, + "load_contaminated": false, + "wall_s": 83.56264781951904, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/chromadb__text384__1000000__core+mutation.json b/benchmarks/competitive/results/raw/chromadb__text384__1000000__core+mutation.json new file mode 100644 index 0000000..c23caf2 --- /dev/null +++ b/benchmarks/competitive/results/raw/chromadb__text384__1000000__core+mutation.json @@ -0,0 +1,305 @@ +{ + "system": "chromadb", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation" + ], + "started": "2026-11-16 09:32:18", + "mem_baseline": { + "rss_mb": 37.3671875, + "peak_mb": 37.3671875 + }, + "version": "chromadb 1.5.9", + "build": { + "ingest_s": 342.8886743001058, + "index_s": 3.89997148886323e-06, + "total_s": 342.88867820007727, + "first_query_ms": 2027.104299981147, + "ready_s": 344.9157825000584 + }, + "mem_after_build": { + "rss_mb": 2132.7109375, + "peak_mb": 2184.09765625 + }, + "mem_after_first_query": { + "rss_mb": 4059.90234375, + "peak_mb": 4059.90234375 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.5578, + "qps_1t_runs": [ + 1205.0106272812911, + 1214.7080983704996, + 1198.4069340504425 + ], + "qps_1t": 1205.0106272812911, + "qps_1t_min": 1198.4069340504425, + "qps_1t_max": 1214.7080983704996, + "median_ms": 0.8210500236600637, + "p95_ms": 0.9600149758625776, + "p99_ms": 1.1412200162885715, + "mean_ms": 0.829184116582231, + "knob": 10, + "batch": { + "recall": 0.5578, + "qps_batch_runs": [ + 8618.531307650717, + 8350.375057569907, + 8636.510850235009 + ], + "qps_batch": 8618.531307650717, + "qps_batch_min": 8350.375057569907, + "qps_batch_max": 8636.510850235009, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6533, + "qps_1t_runs": [ + 1110.331474827692, + 1169.655623428848, + 1108.2821588331865 + ], + "qps_1t": 1110.331474827692, + "qps_1t_min": 1108.2821588331865, + "qps_1t_max": 1169.655623428848, + "median_ms": 0.8663999906275421, + "p95_ms": 1.0182100115343928, + "p99_ms": 1.3616110070142897, + "mean_ms": 0.8859605832452265, + "knob": 16, + "batch": { + "recall": 0.6533, + "qps_batch_runs": [ + 6626.2882747541325, + 6801.490342747759, + 6743.865948723625 + ], + "qps_batch": 6743.865948723625, + "qps_batch_min": 6626.2882747541325, + "qps_batch_max": 6801.490342747759, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.774, + "qps_1t_runs": [ + 991.1341563778788, + 991.759126275454, + 1006.5020535074455 + ], + "qps_1t": 991.759126275454, + "qps_1t_min": 991.1341563778788, + "qps_1t_max": 1006.5020535074455, + "median_ms": 0.9679999784566462, + "p95_ms": 1.150999974925071, + "p99_ms": 1.607160955318251, + "mean_ms": 1.0035981500113849, + "knob": 32, + "batch": { + "recall": 0.774, + "qps_batch_runs": [ + 5012.023845300445, + 5071.014487157763, + 5010.317496562268 + ], + "qps_batch": 5012.023845300445, + "qps_batch_min": 5010.317496562268, + "qps_batch_max": 5071.014487157763, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8717, + "qps_1t_runs": [ + 877.905510738874, + 850.9272830327142, + 887.2437024025633 + ], + "qps_1t": 877.905510738874, + "qps_1t_min": 850.9272830327142, + "qps_1t_max": 887.2437024025633, + "median_ms": 1.0912999860011041, + "p95_ms": 1.2907249911222607, + "p99_ms": 1.8880139989778402, + "mean_ms": 1.1471164500010975, + "knob": 64, + "batch": { + "recall": 0.8717, + "qps_batch_runs": [ + 3255.834700004518, + 3225.2680400956924, + 3293.421456575566 + ], + "qps_batch": 3255.834700004518, + "qps_batch_min": 3225.2680400956924, + "qps_batch_max": 3293.421456575566, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.93225, + "qps_1t_runs": [ + 645.9276522218141, + 687.0113390423369, + 712.9068758236804 + ], + "qps_1t": 687.0113390423369, + "qps_1t_min": 645.9276522218141, + "qps_1t_max": 712.9068758236804, + "median_ms": 1.4240500167943537, + "p95_ms": 1.7009250266710296, + "p99_ms": 2.4345310265198377, + "mean_ms": 1.4688162671615526, + "knob": 128, + "batch": { + "recall": 0.93225, + "qps_batch_runs": [ + 1882.2105132494921, + 1942.3080135793114, + 1939.0945902166181 + ], + "qps_batch": 1939.0945902166181, + "qps_batch_min": 1882.2105132494921, + "qps_batch_max": 1942.3080135793114, + "threads": 32 + } + }, + { + "nq": 1640, + "recall_queries": 1640, + "recall": 0.9673780487804878, + "qps_1t_runs": [ + 534.9663292883475, + 540.1008053611605, + 534.5814373422849 + ], + "qps_1t": 534.9663292883475, + "qps_1t_min": 534.5814373422849, + "qps_1t_max": 540.1008053611605, + "median_ms": 1.8252500158268958, + "p95_ms": 2.1456449816469103, + "p99_ms": 2.930787008954225, + "mean_ms": 1.863801706976511, + "knob": 256, + "batch": { + "recall": 0.96775, + "qps_batch_runs": [ + 1082.7923612422842, + 1113.5097347824662, + 1112.2864889623786 + ], + "qps_batch": 1112.2864889623786, + "qps_batch_min": 1082.7923612422842, + "qps_batch_max": 1113.5097347824662, + "threads": 32 + } + }, + { + "nq": 1103, + "recall_queries": 1103, + "recall": 0.985947416137806, + "qps_1t_runs": [ + 349.04625885335355, + 365.23727873695873, + 348.2116961656387 + ], + "qps_1t": 349.04625885335355, + "qps_1t_min": 348.2116961656387, + "qps_1t_max": 365.23727873695873, + "median_ms": 2.7698000194504857, + "p95_ms": 3.4996600123122334, + "p99_ms": 4.818207963835448, + "mean_ms": 2.824904049379257, + "knob": 512, + "batch": { + "recall": 0.98555, + "qps_batch_runs": [ + 613.7440303862056, + 617.1228701147598, + 613.5853377323836 + ], + "qps_batch": 613.7440303862056, + "qps_batch_min": 613.5853377323836, + "qps_batch_max": 617.1228701147598, + "threads": 32 + } + } + ], + "knob_name": "ef_search", + "mem_after_sweep": { + "rss_mb": 15707.23828125, + "peak_mb": 15707.23828125 + }, + "ref_knob": 256, + "ref_recall_pre": 0.9673780487804878, + "mutation": { + "ref_knob": 256, + "recall_before": 0.9673780487804878, + "insert_10k": { + "seconds": 12.669519199989736, + "per_s": 789.2959347666564, + "batch": 1000 + }, + "update_10k": { + "seconds": 14.411285399983171, + "per_s": 693.9006287400066, + "batch": 1000 + }, + "delete_10k": { + "seconds": 10.66952009999659, + "per_s": 937.2492770319817, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 15833.31640625, + "peak_mb": 15838.484375 + }, + "first_query_after_ms": 2631.103899970185, + "after": { + "nq": 1550, + "recall_queries": 1550, + "recall": 0.9681290322580646, + "qps_1t_runs": [ + 449.7962395113203, + 493.41251320443797, + 498.79109127194823 + ], + "qps_1t": 493.41251320443797, + "qps_1t_min": 449.7962395113203, + "qps_1t_max": 498.79109127194823, + "median_ms": 2.0158999832347035, + "p95_ms": 2.6002750091720377, + "p99_ms": 3.740299014607452, + "mean_ms": 2.084925999847411 + }, + "recall_after": 0.9681290322580646 + }, + "mem_end": { + "rss_mb": 17743.62890625, + "peak_mb": 17775.8671875 + }, + "status": "ok", + "finished": "2026-11-16 09:40:25", + "load_before_pct": 7.481250000000001, + "load_contaminated": false, + "wall_s": 490.5667505264282, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/chromadb__text384__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/chromadb__text384__100000__core+mutation+saveload.json new file mode 100644 index 0000000..9bdd203 --- /dev/null +++ b/benchmarks/competitive/results/raw/chromadb__text384__100000__core+mutation+saveload.json @@ -0,0 +1,318 @@ +{ + "system": "chromadb", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 07:58:05", + "mem_baseline": { + "rss_mb": 37.375, + "peak_mb": 37.375 + }, + "version": "chromadb 1.5.9", + "build": { + "ingest_s": 14.582159700046759, + "index_s": 2.7999631129205227e-06, + "total_s": 14.582162500009872, + "first_query_ms": 160.1226999773644, + "ready_s": 14.742285199987236, + "repeat_error": "InternalError: Collection [bench] already exists", + "total_s_runs": [ + 14.582162500009872, + 14.994863499887288 + ] + }, + "mem_after_build": { + "rss_mb": 325.97265625, + "peak_mb": 422.72265625 + }, + "mem_after_first_query": { + "rss_mb": 523.046875, + "peak_mb": 523.046875 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.61495, + "qps_1t_runs": [ + 1337.002336154772, + 1385.425998043155, + 1360.9946411944622 + ], + "qps_1t": 1360.9946411944622, + "qps_1t_min": 1337.002336154772, + "qps_1t_max": 1385.425998043155, + "median_ms": 0.7024999940767884, + "p95_ms": 0.8498000097461045, + "p99_ms": 0.9739159652963284, + "mean_ms": 0.7348327498572568, + "knob": 10, + "batch": { + "recall": 0.61495, + "qps_batch_runs": [ + 10713.732808321422, + 10772.605906865701, + 10647.257427143082 + ], + "qps_batch": 10713.732808321422, + "qps_batch_min": 10647.257427143082, + "qps_batch_max": 10772.605906865701, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.714, + "qps_1t_runs": [ + 1228.689756800662, + 1254.1955198401554, + 1254.5199561508741 + ], + "qps_1t": 1254.1955198401554, + "qps_1t_min": 1228.689756800662, + "qps_1t_max": 1254.5199561508741, + "median_ms": 0.7868499960750341, + "p95_ms": 0.9096049994695932, + "p99_ms": 1.0655050224158913, + "mean_ms": 0.8027722003947323, + "knob": 16, + "batch": { + "recall": 0.714, + "qps_batch_runs": [ + 8783.247887836216, + 8997.5270315521, + 8881.673024101685 + ], + "qps_batch": 8881.673024101685, + "qps_batch_min": 8783.247887836216, + "qps_batch_max": 8997.5270315521, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8313, + "qps_1t_runs": [ + 1186.6349543538458, + 1169.5553513053485, + 1131.201437733578 + ], + "qps_1t": 1169.5553513053485, + "qps_1t_min": 1131.201437733578, + "qps_1t_max": 1186.6349543538458, + "median_ms": 0.8484999998472631, + "p95_ms": 0.9878099575871602, + "p99_ms": 1.2817659811116768, + "mean_ms": 0.8605869331901582, + "knob": 32, + "batch": { + "recall": 0.8313, + "qps_batch_runs": [ + 6354.186821656716, + 6419.288678425107, + 6345.107208914823 + ], + "qps_batch": 6354.186821656716, + "qps_batch_min": 6345.107208914823, + "qps_batch_max": 6419.288678425107, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.91515, + "qps_1t_runs": [ + 1034.0508830241047, + 1029.0595079459292, + 1013.0796692854713 + ], + "qps_1t": 1029.0595079459292, + "qps_1t_min": 1013.0796692854713, + "qps_1t_max": 1034.0508830241047, + "median_ms": 0.9639000054448843, + "p95_ms": 1.1214050377020612, + "p99_ms": 1.3939080201089384, + "mean_ms": 0.9753068997524679, + "knob": 64, + "batch": { + "recall": 0.91515, + "qps_batch_runs": [ + 4130.46230402775, + 4137.2148684830645, + 4117.868212674134 + ], + "qps_batch": 4130.46230402775, + "qps_batch_min": 4117.868212674134, + "qps_batch_max": 4137.2148684830645, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9646, + "qps_1t_runs": [ + 858.4653721230983, + 850.5723441459293, + 898.1049223542241 + ], + "qps_1t": 858.4653721230983, + "qps_1t_min": 850.5723441459293, + "qps_1t_max": 898.1049223542241, + "median_ms": 1.1433499748818576, + "p95_ms": 1.337224995950237, + "p99_ms": 1.5410279761999852, + "mean_ms": 1.1513346497764967, + "knob": 128, + "batch": { + "recall": 0.9646, + "qps_batch_runs": [ + 2488.79606234071, + 2437.0123691111703, + 2500.2631525539473 + ], + "qps_batch": 2488.79606234071, + "qps_batch_min": 2437.0123691111703, + "qps_batch_max": 2500.2631525539473, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.987, + "qps_1t_runs": [ + 676.416672335202, + 680.0762977359431, + 703.2256083691258 + ], + "qps_1t": 680.0762977359431, + "qps_1t_min": 676.416672335202, + "qps_1t_max": 703.2256083691258, + "median_ms": 1.439899962861091, + "p95_ms": 1.6598050075117499, + "p99_ms": 1.9601509661879402, + "mean_ms": 1.456940232727599, + "knob": 256, + "batch": { + "recall": 0.987, + "qps_batch_runs": [ + 1430.8218047038515, + 1447.4542969800739, + 1444.6764755532777 + ], + "qps_batch": 1444.6764755532777, + "qps_batch_min": 1430.8218047038515, + "qps_batch_max": 1447.4542969800739, + "threads": 32 + } + }, + { + "nq": 1479, + "recall_queries": 1479, + "recall": 0.9954699121027721, + "qps_1t_runs": [ + 465.5699821358445, + 482.8548968397274, + 489.8952398501875 + ], + "qps_1t": 482.8548968397274, + "qps_1t_min": 465.5699821358445, + "qps_1t_max": 489.8952398501875, + "median_ms": 2.0620000432245433, + "p95_ms": 2.4147999938577414, + "p99_ms": 2.86400000564755, + "mean_ms": 2.086724363513948, + "knob": 512, + "batch": { + "recall": 0.99565, + "qps_batch_runs": [ + 813.3006207436933, + 827.8510964198846, + 821.2085989769488 + ], + "qps_batch": 821.2085989769488, + "qps_batch_min": 813.3006207436933, + "qps_batch_max": 827.8510964198846, + "threads": 32 + } + } + ], + "knob_name": "ef_search", + "mem_after_sweep": { + "rss_mb": 1730.21875, + "peak_mb": 1730.21875 + }, + "ref_knob": 128, + "ref_recall_pre": 0.9646, + "mutation": { + "ref_knob": 128, + "recall_before": 0.9646, + "insert_10k": { + "seconds": 3.489183099998627, + "per_s": 2866.0003540668117, + "batch": 1000 + }, + "update_10k": { + "seconds": 5.230450800037943, + "per_s": 1911.8810944416987, + "batch": 1000 + }, + "delete_10k": { + "seconds": 1.9751236999873072, + "per_s": 5062.974030469212, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 609.703125, + "peak_mb": 2046.71484375 + }, + "first_query_after_ms": 170.83680001087487, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.96495, + "qps_1t_runs": [ + 793.8609780147654, + 762.4688145663962, + 819.8130784180476 + ], + "qps_1t": 793.8609780147654, + "qps_1t_min": 762.4688145663962, + "qps_1t_max": 819.8130784180476, + "median_ms": 1.1723999923560768, + "p95_ms": 1.594814978307113, + "p99_ms": 2.7335589757422, + "mean_ms": 1.2636618999240454 + }, + "recall_after": 0.96495 + }, + "saveload": { + "save_s": 1.00000761449337e-06, + "disk_mb": 202.46568298339844, + "load_s": 0.012246400001458824, + "first_query_after_load_ms": 174.5604999596253, + "time_to_ready_s": 0.18680689996108413 + }, + "mem_end": { + "rss_mb": 1003.40234375, + "peak_mb": 2046.71484375 + }, + "status": "ok", + "finished": "2026-11-16 07:59:59", + "load_before_pct": 10.881250000000014, + "load_contaminated": false, + "wall_s": 114.69789218902588, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/chromadb__text384__100000__filter.json b/benchmarks/competitive/results/raw/chromadb__text384__100000__filter.json new file mode 100644 index 0000000..6284f90 --- /dev/null +++ b/benchmarks/competitive/results/raw/chromadb__text384__100000__filter.json @@ -0,0 +1,174 @@ +{ + "system": "chromadb", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 08:38:28", + "filter": { + "build_with_metadata_s": 16.517546100076288, + "mem": { + "rss_mb": 327.57421875, + "peak_mb": 429.1484375 + }, + "subsets": { + "f10": [ + { + "knob": 16, + "nq": 30, + "recall": 0.8966666666666666, + "qps_1t": 2.4678706480474504, + "qps_1t_runs": [ + 2.4678706480474504 + ], + "median_ms": 534.566399990581, + "p95_ms": 677.5786500045797, + "p99_ms": 683.3730260015, + "mean_ms": 405.2076233376283 + }, + { + "knob": 32, + "nq": 30, + "recall": 0.9533333333333334, + "qps_1t": 2.2375069471046465, + "qps_1t_runs": [ + 2.2375069471046465 + ], + "median_ms": 535.4829999851063, + "p95_ms": 689.375394990202, + "p99_ms": 795.0477479700932, + "mean_ms": 446.9259866629727 + }, + { + "knob": 64, + "nq": 30, + "recall": 0.9633333333333334, + "qps_1t": 1.8540625123915617, + "qps_1t_runs": [ + 1.8540625123915617 + ], + "median_ms": 608.4011499769986, + "p95_ms": 760.4019399877867, + "p99_ms": 911.2748709850713, + "mean_ms": 539.3561399988054 + }, + { + "knob": 128, + "nq": 30, + "recall": 0.9933333333333333, + "qps_1t": 1.8555886523058287, + "qps_1t_runs": [ + 1.8555886523058287 + ], + "median_ms": 537.2666000039317, + "p95_ms": 695.2215550118126, + "p99_ms": 783.1240159901791, + "mean_ms": 538.91254333624 + }, + { + "knob": 512, + "nq": 30, + "recall": 1.0, + "qps_1t": 1.9525564958746813, + "qps_1t_runs": [ + 1.9525564958746813 + ], + "median_ms": 550.6684000429232, + "p95_ms": 831.4975300163496, + "p99_ms": 835.723816017271, + "mean_ms": 512.1490733368167 + } + ], + "f01": [ + { + "knob": 16, + "nq": 38, + "recall": 0.9710526315789474, + "qps_1t": 20.928856808842635, + "qps_1t_runs": [ + 20.659439519756404, + 20.928856808842635, + 21.418056855482735 + ], + "median_ms": 19.827849988359958, + "p95_ms": 96.35248502308967, + "p99_ms": 100.01419603417162, + "mean_ms": 47.62483947795995 + }, + { + "knob": 32, + "nq": 128, + "recall": 0.996875, + "qps_1t": 19.661152628726846, + "qps_1t_runs": [ + 19.661152628726846 + ], + "median_ms": 22.473700024420395, + "p95_ms": 115.68687499966472, + "p99_ms": 142.1754209982464, + "mean_ms": 50.86171797165662 + }, + { + "knob": 64, + "nq": 30, + "recall": 1.0, + "qps_1t": 12.431096504342186, + "qps_1t_runs": [ + 10.769348624111169, + 12.431096504342186, + 14.973055984967703 + ], + "median_ms": 100.47009997651912, + "p95_ms": 134.63836998853367, + "p99_ms": 138.30098800593987, + "mean_ms": 80.02872889046557 + }, + { + "knob": 128, + "nq": 30, + "recall": 1.0, + "qps_1t": 9.500454090140131, + "qps_1t_runs": [ + 9.500454090140131, + 9.337574128170521, + 10.338529281931722 + ], + "median_ms": 103.30894999788143, + "p95_ms": 139.7542299790075, + "p99_ms": 171.637331971433, + "mean_ms": 103.02595999997317 + }, + { + "knob": 512, + "nq": 30, + "recall": 1.0, + "qps_1t": 7.901943622219323, + "qps_1t_runs": [ + 7.901943622219323, + 8.154924649097785, + 7.889694704501344 + ], + "median_ms": 124.30065000080504, + "p95_ms": 156.18595496634953, + "p99_ms": 172.3463370121317, + "mean_ms": 125.30801666693554 + } + ] + } + }, + "mem_end": { + "rss_mb": 2400.40625, + "peak_mb": 2403.33984375 + }, + "status": "ok", + "finished": "2026-11-16 08:41:09", + "load_before_pct": 4.80625, + "load_contaminated": false, + "wall_s": 161.30152487754822, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/cpp_avx2__glove__1000000__core+mutation.json b/benchmarks/competitive/results/raw/cpp_avx2__glove__1000000__core+mutation.json new file mode 100644 index 0000000..749db8d --- /dev/null +++ b/benchmarks/competitive/results/raw/cpp_avx2__glove__1000000__core+mutation.json @@ -0,0 +1,310 @@ +{ + "status": "ok", + "label": "cpp_avx2", + "runs": [ + { + "system": "cpp_avx2_exact", + "ds": "glove", + "n": 1000000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.5555575, + "index_s": 0, + "total_s": 0.5555575, + "first_query_ms": 315.8191, + "ready_s": 0.8713766 + }, + "mem_baseline": { + "rss_mb": 6.75, + "peak_mb": 6.75 + }, + "mem_after_ingest": { + "rss_mb": 675.6875, + "peak_mb": 675.6875 + }, + "mem_after_build": { + "rss_mb": 677.777344, + "peak_mb": 816.609375 + }, + "mem_after_first_query": { + "rss_mb": 677.777344, + "peak_mb": 816.609375 + }, + "sweep": [ + { + "nq": 30, + "recall": 1, + "qps_1t": 2.58211371, + "qps_1t_min": 2.58211371, + "qps_1t_max": 2.58211371, + "qps_1t_runs": [ + 2.58211371 + ], + "median_ms": 387.4538, + "p95_ms": 405.094, + "p99_ms": 407.2581, + "mean_ms": 387.279613, + "knob": null + } + ], + "mem_after_sweep": { + "rss_mb": 678, + "peak_mb": 816.84375 + } + }, + { + "system": "cpp_avx2_ivf", + "ds": "glove", + "n": 1000000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "knob_name": "nprobe", + "build": { + "ingest_s": 0.5555575, + "index_s": 6.7325823, + "total_s": 7.2881398, + "first_query_ms": 0, + "ready_s": 7.2881398 + }, + "mem_baseline": { + "rss_mb": 6.75, + "peak_mb": 6.75 + }, + "mem_after_ingest": { + "rss_mb": 675.6875, + "peak_mb": 675.6875 + }, + "mem_after_build": { + "rss_mb": 1085.95312, + "peak_mb": 1147.46875 + }, + "index_stats": { + "nlist": 4000, + "trained_n": 1000000, + "index_bytes": 19616004, + "vector_bytes": 400000000, + "build_seconds": 5.6838201 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.2955, + "qps_1t": 7425.39612, + "qps_1t_min": 7276.11133, + "qps_1t_max": 7598.33959, + "qps_1t_runs": [ + 7425.39612, + 7276.11133, + 7598.33959 + ], + "median_ms": 0.137299998, + "p95_ms": 0.189699989, + "p99_ms": 0.229400001, + "mean_ms": 0.134572233, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.41905, + "qps_1t": 6022.5055, + "qps_1t_min": 5906.95715, + "qps_1t_max": 6306.30546, + "qps_1t_runs": [ + 6022.5055, + 5906.95715, + 6306.30546 + ], + "median_ms": 0.16450003, + "p95_ms": 0.232499966, + "p99_ms": 0.292000012, + "mean_ms": 0.164635733, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.533, + "qps_1t": 4520.19078, + "qps_1t_min": 4501.84801, + "qps_1t_max": 4652.21766, + "qps_1t_runs": [ + 4501.84801, + 4520.19078, + 4652.21766 + ], + "median_ms": 0.218299974, + "p95_ms": 0.317699974, + "p99_ms": 0.386000029, + "mean_ms": 0.219437283, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.63225, + "qps_1t": 2864.23963, + "qps_1t_min": 2803.91965, + "qps_1t_max": 3030.39348, + "qps_1t_runs": [ + 3030.39348, + 2803.91965, + 2864.23963 + ], + "median_ms": 0.339200022, + "p95_ms": 0.493599975, + "p99_ms": 0.55560004, + "mean_ms": 0.345255517, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.72055, + "qps_1t": 1824.70547, + "qps_1t_min": 1796.67981, + "qps_1t_max": 1840.2799, + "qps_1t_runs": [ + 1796.67981, + 1840.2799, + 1824.70547 + ], + "median_ms": 0.538499968, + "p95_ms": 0.784899981, + "p99_ms": 0.8715, + "mean_ms": 0.54933715, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.7965, + "qps_1t": 1111.30948, + "qps_1t_min": 1060.77498, + "qps_1t_max": 1129.52756, + "qps_1t_runs": [ + 1060.77498, + 1111.30948, + 1129.52756 + ], + "median_ms": 0.890400028, + "p95_ms": 1.31700002, + "p99_ms": 1.49649999, + "mean_ms": 0.90929075, + "knob": 32 + }, + { + "nq": 1991, + "recall": 0.861627323, + "qps_1t": 609.506209, + "qps_1t_min": 600.547956, + "qps_1t_max": 632.533153, + "qps_1t_runs": [ + 609.506209, + 600.547956, + 632.533153 + ], + "median_ms": 1.60229998, + "p95_ms": 2.33380002, + "p99_ms": 2.6371, + "mean_ms": 1.62892104, + "knob": 64 + }, + { + "nq": 943, + "recall": 0.920254507, + "qps_1t": 324.841053, + "qps_1t_min": 322.540097, + "qps_1t_max": 343.126833, + "qps_1t_runs": [ + 322.540097, + 324.841053, + 343.126833 + ], + "median_ms": 3.02300003, + "p95_ms": 4.13880002, + "p99_ms": 4.55409999, + "mean_ms": 3.03106419, + "knob": 128 + }, + { + "nq": 552, + "recall": 0.952717391, + "qps_1t": 176.162091, + "qps_1t_min": 170.0925, + "qps_1t_max": 182.710829, + "qps_1t_runs": [ + 182.710829, + 176.162091, + 170.0925 + ], + "median_ms": 5.63159998, + "p95_ms": 7.37599999, + "p99_ms": 8.03039997, + "mean_ms": 5.67629118, + "knob": 256 + } + ], + "ref_knob": 256, + "ref_recall_pre": 0.952717391, + "mem_after_sweep": { + "rss_mb": 1085.94922, + "peak_mb": 1147.46875 + }, + "concurrent_clients": { + "clients": 8, + "qps": 81.1762438 + }, + "mutation": { + "ref_knob": 256, + "recall_before": 0.952717391, + "insert_10k": { + "seconds": 0.1587293, + "per_s": 63000.3408 + }, + "update_10k": { + "seconds": 0.017508, + "per_s": 571167.466 + }, + "delete_10k": { + "seconds": 0.0127602, + "per_s": 783686.775 + }, + "first_query_after_ms": 86.5016, + "recall_after": 0.950809717, + "after": { + "nq": 494, + "recall": 0.950809717, + "qps_1t": 162.19649, + "qps_1t_min": 158.138802, + "qps_1t_max": 165.651104, + "qps_1t_runs": [ + 158.138802, + 162.19649, + 165.651104 + ], + "median_ms": 6.19879999, + "p95_ms": 7.9502, + "p99_ms": 8.59309995, + "mean_ms": 6.17523489 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 1000000, + "builds": 1 + }, + "mem_after": { + "rss_mb": 1102.50781, + "peak_mb": 1467.42188 + } + } + } + ], + "load_before_pct": 5.715624999999997, + "load_contaminated": false, + "wall_s": 82.98990225791931, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/cpp_avx2__glove__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/cpp_avx2__glove__100000__core+mutation+saveload.json new file mode 100644 index 0000000..9a64b46 --- /dev/null +++ b/benchmarks/competitive/results/raw/cpp_avx2__glove__100000__core+mutation+saveload.json @@ -0,0 +1,337 @@ +{ + "status": "ok", + "label": "cpp_avx2", + "runs": [ + { + "system": "cpp_avx2_exact", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.0418406, + "index_s": 0, + "total_s": 0.0418406, + "first_query_ms": 27.0559, + "ready_s": 0.0688965 + }, + "mem_baseline": { + "rss_mb": 6.734375, + "peak_mb": 6.734375 + }, + "mem_after_ingest": { + "rss_mb": 76.0117188, + "peak_mb": 81.5859375 + }, + "mem_after_build": { + "rss_mb": 78.3359375, + "peak_mb": 90.5195312 + }, + "mem_after_first_query": { + "rss_mb": 78.3359375, + "peak_mb": 90.5195312 + }, + "sweep": [ + { + "nq": 126, + "recall": 1, + "qps_1t": 40.179886, + "qps_1t_min": 39.6986257, + "qps_1t_max": 41.1854439, + "qps_1t_runs": [ + 39.6986257, + 40.179886, + 41.1854439 + ], + "median_ms": 24.3726, + "p95_ms": 28.5183, + "p99_ms": 31.7103, + "mean_ms": 24.7860955, + "knob": null + } + ], + "mem_after_sweep": { + "rss_mb": 78.578125, + "peak_mb": 96.8632812 + } + }, + { + "system": "cpp_avx2_ivf", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "knob_name": "nprobe", + "build": { + "ingest_s": 0.0418406, + "index_s": 0.2526785, + "total_s": 0.2945191, + "first_query_ms": 0, + "ready_s": 0.2945191 + }, + "mem_baseline": { + "rss_mb": 6.734375, + "peak_mb": 6.734375 + }, + "mem_after_ingest": { + "rss_mb": 76.0117188, + "peak_mb": 81.5859375 + }, + "mem_after_build": { + "rss_mb": 117.976562, + "peak_mb": 137.757812 + }, + "index_stats": { + "nlist": 1265, + "trained_n": 100000, + "index_bytes": 2311064, + "vector_bytes": 40000000, + "build_seconds": 0.2206831 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.28485, + "qps_1t": 35648.0186, + "qps_1t_min": 33710.5544, + "qps_1t_max": 35884.867, + "qps_1t_runs": [ + 33710.5544, + 35884.867, + 35648.0186 + ], + "median_ms": 0.0282999827, + "p95_ms": 0.0342000276, + "p99_ms": 0.0385000021, + "mean_ms": 0.02852775, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.4074, + "qps_1t": 30431.7196, + "qps_1t_min": 29216.2307, + "qps_1t_max": 30692.1227, + "qps_1t_runs": [ + 30692.1227, + 29216.2307, + 30431.7196 + ], + "median_ms": 0.0328000169, + "p95_ms": 0.0408999622, + "p99_ms": 0.045599998, + "mean_ms": 0.0332232167, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.5255, + "qps_1t": 25324.6942, + "qps_1t_min": 24997.8127, + "qps_1t_max": 25644.8724, + "qps_1t_runs": [ + 25324.6942, + 24997.8127, + 25644.8724 + ], + "median_ms": 0.0390000059, + "p95_ms": 0.0490000239, + "p99_ms": 0.0549000106, + "mean_ms": 0.0394949333, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.63475, + "qps_1t": 19291.3329, + "qps_1t_min": 19116.8036, + "qps_1t_max": 19436.3271, + "qps_1t_runs": [ + 19291.3329, + 19116.8036, + 19436.3271 + ], + "median_ms": 0.0508999801, + "p95_ms": 0.0657999772, + "p99_ms": 0.074599986, + "mean_ms": 0.0518656, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.7357, + "qps_1t": 13080.3592, + "qps_1t_min": 13064.4581, + "qps_1t_max": 13093.6154, + "qps_1t_runs": [ + 13093.6154, + 13080.3592, + 13064.4581 + ], + "median_ms": 0.0749000465, + "p95_ms": 0.098199991, + "p99_ms": 0.120099983, + "mean_ms": 0.0764557166, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.81855, + "qps_1t": 8057.7158, + "qps_1t_min": 7722.42515, + "qps_1t_max": 8186.69596, + "qps_1t_runs": [ + 7722.42515, + 8186.69596, + 8057.7158 + ], + "median_ms": 0.122899946, + "p95_ms": 0.161000004, + "p99_ms": 0.184700009, + "mean_ms": 0.125249017, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.88455, + "qps_1t": 4786.74454, + "qps_1t_min": 4719.04905, + "qps_1t_max": 4841.50955, + "qps_1t_runs": [ + 4841.50955, + 4786.74454, + 4719.04905 + ], + "median_ms": 0.205399992, + "p95_ms": 0.265499984, + "p99_ms": 0.308600022, + "mean_ms": 0.2091215, + "knob": 64 + }, + { + "nq": 2000, + "recall": 0.93865, + "qps_1t": 2655.36402, + "qps_1t_min": 2627.55291, + "qps_1t_max": 2692.88532, + "qps_1t_runs": [ + 2627.55291, + 2655.36402, + 2692.88532 + ], + "median_ms": 0.371000031, + "p95_ms": 0.464400044, + "p99_ms": 0.523900031, + "mean_ms": 0.376175783, + "knob": 128 + }, + { + "nq": 2000, + "recall": 0.9741, + "qps_1t": 1407.55641, + "qps_1t_min": 1344.76988, + "qps_1t_max": 1414.43706, + "qps_1t_runs": [ + 1344.76988, + 1414.43706, + 1407.55641 + ], + "median_ms": 0.70829998, + "p95_ms": 0.882099965, + "p99_ms": 1.03340001, + "mean_ms": 0.720355934, + "knob": 256 + }, + { + "nq": 125, + "recall": 1, + "qps_1t": 41.6154464, + "qps_1t_min": 41.4861122, + "qps_1t_max": 41.6784255, + "qps_1t_runs": [ + 41.6784255, + 41.4861122, + 41.6154464 + ], + "median_ms": 23.537, + "p95_ms": 26.7577, + "p99_ms": 27.8641, + "mean_ms": 24.0424067, + "knob": 1265 + } + ], + "ref_knob": 256, + "ref_recall_pre": 0.9741, + "mem_after_sweep": { + "rss_mb": 120.015625, + "peak_mb": 138.335938 + }, + "concurrent_clients": { + "clients": 8, + "qps": 1436.84251 + }, + "mutation": { + "ref_knob": 256, + "recall_before": 0.9741, + "insert_10k": { + "seconds": 0.0117463, + "per_s": 851331.91 + }, + "update_10k": { + "seconds": 0.00649349997, + "per_s": 1540001.55 + }, + "delete_10k": { + "seconds": 0.00375530001, + "per_s": 2662903.09 + }, + "first_query_after_ms": 19.8733, + "recall_after": 0.9725, + "after": { + "nq": 2000, + "recall": 0.9725, + "qps_1t": 1347.34954, + "qps_1t_min": 1331.85143, + "qps_1t_max": 1380.28235, + "qps_1t_runs": [ + 1331.85143, + 1380.28235, + 1347.34954 + ], + "median_ms": 0.723399979, + "p95_ms": 0.913900032, + "p99_ms": 1.14820001, + "mean_ms": 0.73917395 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 100000, + "builds": 1 + }, + "mem_after": { + "rss_mb": 137.554688, + "peak_mb": 160.539062 + } + }, + "saveload": { + "save_s": 4.2604119, + "disk_mb": 133.363812, + "load_s": 2.0163434, + "first_query_after_load_ms": 235.5158, + "time_to_ready_s": 2.2518592, + "note": "load = JSON parse + insert; first query builds the IVF index (never persisted)" + } + } + ], + "load_before_pct": 3.134374999999998, + "load_contaminated": false, + "wall_s": 42.93906879425049, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/cpp_avx2__sift__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/cpp_avx2__sift__100000__core+mutation+saveload.json new file mode 100644 index 0000000..3ad6507 --- /dev/null +++ b/benchmarks/competitive/results/raw/cpp_avx2__sift__100000__core+mutation+saveload.json @@ -0,0 +1,337 @@ +{ + "status": "ok", + "label": "cpp_avx2", + "runs": [ + { + "system": "cpp_avx2_exact", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.0491732, + "index_s": 0, + "total_s": 0.0491732, + "first_query_ms": 29.6095, + "ready_s": 0.0787827 + }, + "mem_baseline": { + "rss_mb": 7.12890625, + "peak_mb": 7.12890625 + }, + "mem_after_ingest": { + "rss_mb": 87.4882812, + "peak_mb": 95.109375 + }, + "mem_after_build": { + "rss_mb": 91.1835938, + "peak_mb": 109.460938 + }, + "mem_after_first_query": { + "rss_mb": 91.1835938, + "peak_mb": 109.460938 + }, + "sweep": [ + { + "nq": 97, + "recall": 1, + "qps_1t": 23.4311232, + "qps_1t_min": 23.333474, + "qps_1t_max": 27.2832601, + "qps_1t_runs": [ + 27.2832601, + 23.4311232, + 23.333474 + ], + "median_ms": 42.3488, + "p95_ms": 45.4001, + "p99_ms": 47.3759, + "mean_ms": 40.7292247, + "knob": null + } + ], + "mem_after_sweep": { + "rss_mb": 91.3125, + "peak_mb": 109.597656 + } + }, + { + "system": "cpp_avx2_ivf", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "knob_name": "nprobe", + "build": { + "ingest_s": 0.0491732, + "index_s": 0.6379454, + "total_s": 0.6871186, + "first_query_ms": 0, + "ready_s": 0.6871186 + }, + "mem_baseline": { + "rss_mb": 7.12890625, + "peak_mb": 7.12890625 + }, + "mem_after_ingest": { + "rss_mb": 87.4882812, + "peak_mb": 95.109375 + }, + "mem_after_build": { + "rss_mb": 141.824219, + "peak_mb": 167 + }, + "index_stats": { + "nlist": 1265, + "trained_n": 100000, + "index_bytes": 2452744, + "vector_bytes": 51200000, + "build_seconds": 0.5497378 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.27955, + "qps_1t": 23359.5194, + "qps_1t_min": 21573.1811, + "qps_1t_max": 23763.3551, + "qps_1t_runs": [ + 23359.5194, + 21573.1811, + 23763.3551 + ], + "median_ms": 0.0445999904, + "p95_ms": 0.0557000167, + "p99_ms": 0.0663000392, + "mean_ms": 0.0437481833, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.42195, + "qps_1t": 19562.7147, + "qps_1t_min": 18794.0427, + "qps_1t_max": 20323.6748, + "qps_1t_runs": [ + 19562.7147, + 18794.0427, + 20323.6748 + ], + "median_ms": 0.0508999801, + "p95_ms": 0.0659999787, + "p99_ms": 0.0765000004, + "mean_ms": 0.0511765666, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.5793, + "qps_1t": 16494.2196, + "qps_1t_min": 16249.7786, + "qps_1t_max": 17277.4236, + "qps_1t_runs": [ + 16494.2196, + 17277.4236, + 16249.7786 + ], + "median_ms": 0.0594999874, + "p95_ms": 0.0784000149, + "p99_ms": 0.0907999929, + "mean_ms": 0.0600152001, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.7401, + "qps_1t": 11994.6216, + "qps_1t_min": 11642.4668, + "qps_1t_max": 12084.3512, + "qps_1t_runs": [ + 11642.4668, + 11994.6216, + 12084.3512 + ], + "median_ms": 0.0819000416, + "p95_ms": 0.111699977, + "p99_ms": 0.126700033, + "mean_ms": 0.0840049333, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.86955, + "qps_1t": 7631.03633, + "qps_1t_min": 7570.06569, + "qps_1t_max": 7635.26639, + "qps_1t_runs": [ + 7570.06569, + 7631.03633, + 7635.26639 + ], + "median_ms": 0.127800042, + "p95_ms": 0.176700007, + "p99_ms": 0.203299976, + "mean_ms": 0.131371417, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.95215, + "qps_1t": 4693.14571, + "qps_1t_min": 4524.2277, + "qps_1t_max": 4768.34202, + "qps_1t_runs": [ + 4768.34202, + 4524.2277, + 4693.14571 + ], + "median_ms": 0.210799975, + "p95_ms": 0.293300021, + "p99_ms": 0.3363, + "mean_ms": 0.214608467, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.98825, + "qps_1t": 2719.49448, + "qps_1t_min": 2710.51007, + "qps_1t_max": 2761.44612, + "qps_1t_runs": [ + 2710.51007, + 2761.44612, + 2719.49448 + ], + "median_ms": 0.361000013, + "p95_ms": 0.484400021, + "p99_ms": 0.55279996, + "mean_ms": 0.366259583, + "knob": 64 + }, + { + "nq": 2000, + "recall": 0.9984, + "qps_1t": 1488.20043, + "qps_1t_min": 1403.35967, + "qps_1t_max": 1494.32984, + "qps_1t_runs": [ + 1488.20043, + 1403.35967, + 1494.32984 + ], + "median_ms": 0.675299962, + "p95_ms": 0.884700043, + "p99_ms": 1.01939996, + "mean_ms": 0.684574833, + "knob": 128 + }, + { + "nq": 2000, + "recall": 0.9999, + "qps_1t": 787.236813, + "qps_1t_min": 782.572394, + "qps_1t_max": 832.727455, + "qps_1t_runs": [ + 832.727455, + 787.236813, + 782.572394 + ], + "median_ms": 1.24119996, + "p95_ms": 1.51699997, + "p99_ms": 1.666, + "mean_ms": 1.24965865, + "knob": 256 + }, + { + "nq": 69, + "recall": 1, + "qps_1t": 23.0380589, + "qps_1t_min": 22.9235987, + "qps_1t_max": 23.2509506, + "qps_1t_runs": [ + 22.9235987, + 23.0380589, + 23.2509506 + ], + "median_ms": 43.3466, + "p95_ms": 46.3013, + "p99_ms": 48.4077, + "mean_ms": 43.346199, + "knob": 1265 + } + ], + "ref_knob": 32, + "ref_recall_pre": 0.95215, + "mem_after_sweep": { + "rss_mb": 143.105469, + "peak_mb": 167 + }, + "concurrent_clients": { + "clients": 8, + "qps": 2420.08135 + }, + "mutation": { + "ref_knob": 32, + "recall_before": 0.95215, + "insert_10k": { + "seconds": 0.029963, + "per_s": 333744.952 + }, + "update_10k": { + "seconds": 0.0114121, + "per_s": 876262.914 + }, + "delete_10k": { + "seconds": 0.00631789997, + "per_s": 1582804.42 + }, + "first_query_after_ms": 30.3824, + "recall_after": 0.949, + "after": { + "nq": 2000, + "recall": 0.949, + "qps_1t": 4870.65133, + "qps_1t_min": 4730.30744, + "qps_1t_max": 5086.68732, + "qps_1t_runs": [ + 5086.68732, + 4730.30744, + 4870.65133 + ], + "median_ms": 0.19749999, + "p95_ms": 0.274699996, + "p99_ms": 0.335499994, + "mean_ms": 0.204435233 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 100000, + "builds": 1 + }, + "mem_after": { + "rss_mb": 163.820312, + "peak_mb": 194.328125 + } + }, + "saveload": { + "save_s": 12.2242788, + "disk_mb": 133.678411, + "load_s": 5.1421373, + "first_query_after_load_ms": 635.6938, + "time_to_ready_s": 5.7778311, + "note": "load = JSON parse + insert; first query builds the IVF index (never persisted)" + } + } + ], + "load_before_pct": 14.274999999999986, + "load_contaminated": false, + "wall_s": 62.85570001602173, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/cpp_avx2__text384__1000000__core+mutation.json b/benchmarks/competitive/results/raw/cpp_avx2__text384__1000000__core+mutation.json new file mode 100644 index 0000000..059a839 --- /dev/null +++ b/benchmarks/competitive/results/raw/cpp_avx2__text384__1000000__core+mutation.json @@ -0,0 +1,310 @@ +{ + "status": "ok", + "label": "cpp_avx2", + "runs": [ + { + "system": "cpp_avx2_exact", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.9221926, + "index_s": 0, + "total_s": 0.9221926, + "first_query_ms": 510.0265, + "ready_s": 1.4322191 + }, + "mem_baseline": { + "rss_mb": 11.0585938, + "peak_mb": 11.0585938 + }, + "mem_after_ingest": { + "rss_mb": 1791.33203, + "peak_mb": 1791.33203 + }, + "mem_after_build": { + "rss_mb": 1794.36328, + "peak_mb": 1933.19531 + }, + "mem_after_first_query": { + "rss_mb": 1794.36328, + "peak_mb": 1933.19531 + }, + "sweep": [ + { + "nq": 30, + "recall": 1, + "qps_1t": 2.1345438, + "qps_1t_min": 2.1345438, + "qps_1t_max": 2.1345438, + "qps_1t_runs": [ + 2.1345438 + ], + "median_ms": 466.0438, + "p95_ms": 491.3723, + "p99_ms": 493.2179, + "mean_ms": 468.48418, + "knob": null + } + ], + "mem_after_sweep": { + "rss_mb": 1794.39453, + "peak_mb": 1933.23828 + } + }, + { + "system": "cpp_avx2_ivf", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "knob_name": "nprobe", + "build": { + "ingest_s": 0.9221926, + "index_s": 6.2006516, + "total_s": 7.1228442, + "first_query_ms": 0, + "ready_s": 7.1228442 + }, + "mem_baseline": { + "rss_mb": 11.0585938, + "peak_mb": 11.0585938 + }, + "mem_after_ingest": { + "rss_mb": 1791.33203, + "peak_mb": 1791.33203 + }, + "mem_after_build": { + "rss_mb": 3290.5, + "peak_mb": 3525.36328 + }, + "index_stats": { + "nlist": 4000, + "trained_n": 1000000, + "index_bytes": 24160004, + "vector_bytes": 1536000000, + "build_seconds": 5.4602875 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.26665, + "qps_1t": 4780.26896, + "qps_1t_min": 4475.10388, + "qps_1t_max": 4797.09948, + "qps_1t_runs": [ + 4475.10388, + 4780.26896, + 4797.09948 + ], + "median_ms": 0.205699995, + "p95_ms": 0.280899985, + "p99_ms": 0.372799987, + "mean_ms": 0.213703683, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.3725, + "qps_1t": 3919.84927, + "qps_1t_min": 3908.62418, + "qps_1t_max": 3942.15906, + "qps_1t_runs": [ + 3942.15906, + 3919.84927, + 3908.62418 + ], + "median_ms": 0.245500007, + "p95_ms": 0.330200011, + "p99_ms": 0.380599988, + "mean_ms": 0.254874817, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.4895, + "qps_1t": 3015.91483, + "qps_1t_min": 2955.93528, + "qps_1t_max": 3038.65473, + "qps_1t_runs": [ + 3038.65473, + 3015.91483, + 2955.93528 + ], + "median_ms": 0.322799955, + "p95_ms": 0.431899971, + "p99_ms": 0.499300018, + "mean_ms": 0.332989917, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.60205, + "qps_1t": 1989.41096, + "qps_1t_min": 1984.49356, + "qps_1t_max": 2021.33172, + "qps_1t_runs": [ + 1984.49356, + 1989.41096, + 2021.33172 + ], + "median_ms": 0.488699996, + "p95_ms": 0.657500001, + "p99_ms": 0.771599996, + "mean_ms": 0.500430533, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.7007, + "qps_1t": 1243.44951, + "qps_1t_min": 1230.49587, + "qps_1t_max": 1245.15557, + "qps_1t_runs": [ + 1230.49587, + 1243.44951, + 1245.15557 + ], + "median_ms": 0.795300002, + "p95_ms": 1.0168, + "p99_ms": 1.15140004, + "mean_ms": 0.806669133, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.789, + "qps_1t": 712.564997, + "qps_1t_min": 695.861126, + "qps_1t_max": 718.010986, + "qps_1t_runs": [ + 695.861126, + 712.564997, + 718.010986 + ], + "median_ms": 1.39469997, + "p95_ms": 1.7433, + "p99_ms": 1.995, + "mean_ms": 1.41106182, + "knob": 32 + }, + { + "nq": 1301, + "recall": 0.860261337, + "qps_1t": 403.978489, + "qps_1t_min": 397.137242, + "qps_1t_max": 406.292684, + "qps_1t_runs": [ + 406.292684, + 403.978489, + 397.137242 + ], + "median_ms": 2.4466, + "p95_ms": 3.03749996, + "p99_ms": 3.41240002, + "mean_ms": 2.48489347, + "knob": 64 + }, + { + "nq": 714, + "recall": 0.914145658, + "qps_1t": 220.598281, + "qps_1t_min": 216.367715, + "qps_1t_max": 223.351688, + "qps_1t_runs": [ + 223.351688, + 216.367715, + 220.598281 + ], + "median_ms": 4.5022, + "p95_ms": 5.33060002, + "p99_ms": 5.77290001, + "mean_ms": 4.54404416, + "knob": 128 + }, + { + "nq": 360, + "recall": 0.954722222, + "qps_1t": 119.292006, + "qps_1t_min": 114.767574, + "qps_1t_max": 119.952711, + "qps_1t_runs": [ + 119.292006, + 114.767574, + 119.952711 + ], + "median_ms": 8.37580004, + "p95_ms": 9.74719995, + "p99_ms": 11.3932, + "mean_ms": 8.4775575, + "knob": 256 + } + ], + "ref_knob": 256, + "ref_recall_pre": 0.954722222, + "mem_after_sweep": { + "rss_mb": 3290.58203, + "peak_mb": 3525.36328 + }, + "concurrent_clients": { + "clients": 8, + "qps": 118.483693 + }, + "mutation": { + "ref_knob": 256, + "recall_before": 0.954722222, + "insert_10k": { + "seconds": 0.2878924, + "per_s": 34735.1997 + }, + "update_10k": { + "seconds": 0.0162474, + "per_s": 615483.092 + }, + "delete_10k": { + "seconds": 0.00723399996, + "per_s": 1382361.08 + }, + "first_query_after_ms": 81.231, + "recall_after": 0.956056338, + "after": { + "nq": 355, + "recall": 0.956056338, + "qps_1t": 117.493688, + "qps_1t_min": 115.307031, + "qps_1t_max": 118.805164, + "qps_1t_runs": [ + 118.805164, + 115.307031, + 117.493688 + ], + "median_ms": 8.4082, + "p95_ms": 9.98079998, + "p99_ms": 11.7942, + "mean_ms": 8.53357869 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 1000000, + "builds": 1 + }, + "mem_after": { + "rss_mb": 3338.42969, + "peak_mb": 4755.55859 + } + } + } + ], + "load_before_pct": 3.8312499999999994, + "load_contaminated": false, + "wall_s": 93.26696515083313, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/cpp_avx2__text384__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/cpp_avx2__text384__100000__core+mutation+saveload.json new file mode 100644 index 0000000..a2fe6f2 --- /dev/null +++ b/benchmarks/competitive/results/raw/cpp_avx2__text384__100000__core+mutation+saveload.json @@ -0,0 +1,337 @@ +{ + "status": "ok", + "label": "cpp_avx2", + "runs": [ + { + "system": "cpp_avx2_exact", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.0811739, + "index_s": 0, + "total_s": 0.0811739, + "first_query_ms": 43.8393, + "ready_s": 0.1250132 + }, + "mem_baseline": { + "rss_mb": 11.0507812, + "peak_mb": 11.0507812 + }, + "mem_after_ingest": { + "rss_mb": 200.125, + "peak_mb": 217.082031 + }, + "mem_after_build": { + "rss_mb": 203.308594, + "peak_mb": 217.082031 + }, + "mem_after_first_query": { + "rss_mb": 203.308594, + "peak_mb": 217.082031 + }, + "sweep": [ + { + "nq": 68, + "recall": 1, + "qps_1t": 23.016217, + "qps_1t_min": 22.9903773, + "qps_1t_max": 23.1046586, + "qps_1t_runs": [ + 23.1046586, + 22.9903773, + 23.016217 + ], + "median_ms": 43.5436, + "p95_ms": 45.3769, + "p99_ms": 47.5699, + "mean_ms": 43.4084667, + "knob": null + } + ], + "mem_after_sweep": { + "rss_mb": 203.5, + "peak_mb": 217.082031 + } + }, + { + "system": "cpp_avx2_ivf", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "knob_name": "nprobe", + "build": { + "ingest_s": 0.0811739, + "index_s": 0.5254779, + "total_s": 0.6066518, + "first_query_ms": 0, + "ready_s": 0.6066518 + }, + "mem_baseline": { + "rss_mb": 11.0507812, + "peak_mb": 11.0507812 + }, + "mem_after_ingest": { + "rss_mb": 200.125, + "peak_mb": 217.082031 + }, + "mem_after_build": { + "rss_mb": 353.316406, + "peak_mb": 427.917969 + }, + "index_stats": { + "nlist": 1265, + "trained_n": 100000, + "index_bytes": 3748104, + "vector_bytes": 153600000, + "build_seconds": 0.4632085 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.28085, + "qps_1t": 17263.8521, + "qps_1t_min": 17031.8291, + "qps_1t_max": 17851.9147, + "qps_1t_runs": [ + 17031.8291, + 17263.8521, + 17851.9147 + ], + "median_ms": 0.056000019, + "p95_ms": 0.0690000015, + "p99_ms": 0.0801000278, + "mean_ms": 0.0575515001, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.40215, + "qps_1t": 14309.5397, + "qps_1t_min": 14254.2023, + "qps_1t_max": 14358.2955, + "qps_1t_runs": [ + 14309.5397, + 14358.2955, + 14254.2023 + ], + "median_ms": 0.0679999939, + "p95_ms": 0.0863000168, + "p99_ms": 0.10179996, + "mean_ms": 0.0698947833, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.5291, + "qps_1t": 11488.1193, + "qps_1t_min": 11326.8242, + "qps_1t_max": 11627.1498, + "qps_1t_runs": [ + 11627.1498, + 11326.8242, + 11488.1193 + ], + "median_ms": 0.0849000062, + "p95_ms": 0.109800021, + "p99_ms": 0.131800014, + "mean_ms": 0.0871126834, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.6479, + "qps_1t": 8129.60211, + "qps_1t_min": 7994.6532, + "qps_1t_max": 8406.75231, + "qps_1t_runs": [ + 8129.60211, + 7994.6532, + 8406.75231 + ], + "median_ms": 0.117900025, + "p95_ms": 0.15789998, + "p99_ms": 0.212399988, + "mean_ms": 0.122347617, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.75475, + "qps_1t": 5301.21082, + "qps_1t_min": 5296.1894, + "qps_1t_max": 5315.19646, + "qps_1t_runs": [ + 5296.1894, + 5315.19646, + 5301.21082 + ], + "median_ms": 0.182699994, + "p95_ms": 0.242899987, + "p99_ms": 0.305099995, + "mean_ms": 0.188530317, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.84275, + "qps_1t": 3053.08552, + "qps_1t_min": 3021.49309, + "qps_1t_max": 3054.45031, + "qps_1t_runs": [ + 3053.08552, + 3054.45031, + 3021.49309 + ], + "median_ms": 0.319499988, + "p95_ms": 0.414400012, + "p99_ms": 0.485800032, + "mean_ms": 0.328630283, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.90905, + "qps_1t": 1696.79697, + "qps_1t_min": 1694.25263, + "qps_1t_max": 1723.32544, + "qps_1t_runs": [ + 1694.25263, + 1723.32544, + 1696.79697 + ], + "median_ms": 0.575600017, + "p95_ms": 0.716999988, + "p99_ms": 0.868900039, + "mean_ms": 0.586616633, + "knob": 64 + }, + { + "nq": 2000, + "recall": 0.95425, + "qps_1t": 901.545912, + "qps_1t_min": 900.405678, + "qps_1t_max": 905.743159, + "qps_1t_runs": [ + 900.405678, + 905.743159, + 901.545912 + ], + "median_ms": 1.09629997, + "p95_ms": 1.2994, + "p99_ms": 1.47660001, + "mean_ms": 1.1079607, + "knob": 128 + }, + { + "nq": 985, + "recall": 0.981319797, + "qps_1t": 454.985197, + "qps_1t_min": 452.333217, + "qps_1t_max": 461.837435, + "qps_1t_runs": [ + 452.333217, + 454.985197, + 461.837435 + ], + "median_ms": 2.15499999, + "p95_ms": 2.58510001, + "p99_ms": 2.97590002, + "mean_ms": 2.19129912, + "knob": 256 + }, + { + "nq": 65, + "recall": 1, + "qps_1t": 23.0463919, + "qps_1t_min": 23.0153493, + "qps_1t_max": 23.2042002, + "qps_1t_runs": [ + 23.0463919, + 23.2042002, + 23.0153493 + ], + "median_ms": 43.4106, + "p95_ms": 45.4693999, + "p99_ms": 46.4893, + "mean_ms": 43.3118836, + "knob": 1265 + } + ], + "ref_knob": 128, + "ref_recall_pre": 0.95425, + "mem_after_sweep": { + "rss_mb": 355.894531, + "peak_mb": 427.917969 + }, + "concurrent_clients": { + "clients": 8, + "qps": 892.541167 + }, + "mutation": { + "ref_knob": 128, + "recall_before": 0.95425, + "insert_10k": { + "seconds": 0.0380363, + "per_s": 262906.749 + }, + "update_10k": { + "seconds": 0.0141123, + "per_s": 708601.716 + }, + "delete_10k": { + "seconds": 0.00469430001, + "per_s": 2130243.06 + }, + "first_query_after_ms": 43.7709, + "recall_after": 0.95065, + "after": { + "nq": 2000, + "recall": 0.95065, + "qps_1t": 895.144397, + "qps_1t_min": 889.471908, + "qps_1t_max": 895.624222, + "qps_1t_runs": [ + 895.144397, + 895.624222, + 889.471908 + ], + "median_ms": 1.10910001, + "p95_ms": 1.30970002, + "p99_ms": 1.43160002, + "mean_ms": 1.1193135 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 100000, + "builds": 1 + }, + "mem_after": { + "rss_mb": 419.621094, + "peak_mb": 503.675781 + } + }, + "saveload": { + "save_s": 16.37915, + "disk_mb": 499.571589, + "load_s": 6.9462202, + "first_query_after_load_ms": 531.1766, + "time_to_ready_s": 7.4773968, + "note": "load = JSON parse + insert; first query builds the IVF index (never persisted)" + } + } + ], + "load_before_pct": 6.212500000000001, + "load_contaminated": false, + "wall_s": 76.45933747291565, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/cpp_avx2__text384__10000__core.json b/benchmarks/competitive/results/raw/cpp_avx2__text384__10000__core.json new file mode 100644 index 0000000..fdc4f76 --- /dev/null +++ b/benchmarks/competitive/results/raw/cpp_avx2__text384__10000__core.json @@ -0,0 +1,285 @@ +{ + "status": "ok", + "label": "cpp_avx2", + "runs": [ + { + "system": "cpp_avx2_exact", + "ds": "text384", + "n": 10000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.00655609998, + "index_s": 0, + "total_s": 0.00655609998, + "first_query_ms": 3.57950001, + "ready_s": 0.0101356 + }, + "mem_baseline": { + "rss_mb": 11.0507812, + "peak_mb": 11.0507812 + }, + "mem_after_ingest": { + "rss_mb": 29.53125, + "peak_mb": 44.1601562 + }, + "mem_after_build": { + "rss_mb": 32.7460938, + "peak_mb": 44.1601562 + }, + "mem_after_first_query": { + "rss_mb": 32.7460938, + "peak_mb": 44.1601562 + }, + "sweep": [ + { + "nq": 999, + "recall": 1, + "qps_1t": 327.830065, + "qps_1t_min": 307.938716, + "qps_1t_max": 328.352164, + "qps_1t_runs": [ + 327.830065, + 328.352164, + 307.938716 + ], + "median_ms": 3.08160001, + "p95_ms": 3.48419999, + "p99_ms": 3.84210004, + "mean_ms": 3.11442362, + "knob": null + } + ], + "mem_after_sweep": { + "rss_mb": 32.1875, + "peak_mb": 44.1601562 + } + }, + { + "system": "cpp_avx2_ivf", + "ds": "text384", + "n": 10000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "knob_name": "nprobe", + "build": { + "ingest_s": 0.00655609998, + "index_s": 0.0559779, + "total_s": 0.062534, + "first_query_ms": 0, + "ready_s": 0.062534 + }, + "mem_baseline": { + "rss_mb": 11.0507812, + "peak_mb": 11.0507812 + }, + "mem_after_ingest": { + "rss_mb": 29.53125, + "peak_mb": 44.1601562 + }, + "mem_after_build": { + "rss_mb": 47.2226562, + "peak_mb": 62.328125 + }, + "index_stats": { + "nlist": 400, + "trained_n": 10000, + "index_bytes": 796004, + "vector_bytes": 15360000, + "build_seconds": 0.0508875 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.2963, + "qps_1t": 59166.6962, + "qps_1t_min": 57900.0244, + "qps_1t_max": 61641.4498, + "qps_1t_runs": [ + 57900.0244, + 59166.6962, + 61641.4498 + ], + "median_ms": 0.0164000085, + "p95_ms": 0.0199000351, + "p99_ms": 0.0280000386, + "mean_ms": 0.0167984667, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.41875, + "qps_1t": 56484.889, + "qps_1t_min": 56146.4974, + "qps_1t_max": 56753.3674, + "qps_1t_runs": [ + 56146.4974, + 56484.889, + 56753.3674 + ], + "median_ms": 0.0175000168, + "p95_ms": 0.020699983, + "p99_ms": 0.022799999, + "mean_ms": 0.0177114999, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.55125, + "qps_1t": 48555.2388, + "qps_1t_min": 46373.0479, + "qps_1t_max": 50370.9824, + "qps_1t_runs": [ + 50370.9824, + 48555.2388, + 46373.0479 + ], + "median_ms": 0.0196999754, + "p95_ms": 0.0236000051, + "p99_ms": 0.0265000272, + "mean_ms": 0.0206706833, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.6798, + "qps_1t": 39167.1495, + "qps_1t_min": 38511.3811, + "qps_1t_max": 39623.2623, + "qps_1t_runs": [ + 39623.2623, + 38511.3811, + 39167.1495 + ], + "median_ms": 0.024399953, + "p95_ms": 0.0299999956, + "p99_ms": 0.03559998, + "mean_ms": 0.02557855, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.8005, + "qps_1t": 29515.7207, + "qps_1t_min": 28252.6579, + "qps_1t_max": 30075.6856, + "qps_1t_runs": [ + 29515.7207, + 30075.6856, + 28252.6579 + ], + "median_ms": 0.0335999648, + "p95_ms": 0.0416000257, + "p99_ms": 0.046000001, + "mean_ms": 0.0341748667, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.8922, + "qps_1t": 18121.6378, + "qps_1t_min": 18109.1839, + "qps_1t_max": 19142.2731, + "qps_1t_runs": [ + 18109.1839, + 19142.2731, + 18121.6378 + ], + "median_ms": 0.0537000014, + "p95_ms": 0.0656999764, + "p99_ms": 0.0714000198, + "mean_ms": 0.05421455, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.9563, + "qps_1t": 10819.8644, + "qps_1t_min": 10349.7169, + "qps_1t_max": 10930.0547, + "qps_1t_runs": [ + 10819.8644, + 10349.7169, + 10930.0547 + ], + "median_ms": 0.0922000036, + "p95_ms": 0.112999987, + "p99_ms": 0.132900022, + "mean_ms": 0.0935114834, + "knob": 64 + }, + { + "nq": 2000, + "recall": 0.9874, + "qps_1t": 6556.78372, + "qps_1t_min": 6235.10978, + "qps_1t_max": 6684.27974, + "qps_1t_runs": [ + 6684.27974, + 6235.10978, + 6556.78372 + ], + "median_ms": 0.155000016, + "p95_ms": 0.177900016, + "p99_ms": 0.206099998, + "mean_ms": 0.154166883, + "knob": 128 + }, + { + "nq": 2000, + "recall": 0.99845, + "qps_1t": 3471.99498, + "qps_1t_min": 3423.98094, + "qps_1t_max": 3594.77277, + "qps_1t_runs": [ + 3471.99498, + 3594.77277, + 3423.98094 + ], + "median_ms": 0.28810004, + "p95_ms": 0.332999974, + "p99_ms": 0.385999971, + "mean_ms": 0.286086083, + "knob": 256 + }, + { + "nq": 977, + "recall": 1, + "qps_1t": 329.317869, + "qps_1t_min": 323.319657, + "qps_1t_max": 329.820046, + "qps_1t_runs": [ + 323.319657, + 329.820046, + 329.317869 + ], + "median_ms": 3.0353, + "p95_ms": 3.33409995, + "p99_ms": 3.56660003, + "mean_ms": 3.05381685, + "knob": 400 + } + ], + "ref_knob": 64, + "ref_recall_pre": 0.9563, + "mem_after_sweep": { + "rss_mb": 49.6171875, + "peak_mb": 62.328125 + }, + "concurrent_clients": { + "clients": 8, + "qps": 9803.29684 + } + } + ], + "load_before_pct": 6.406250000000001, + "load_contaminated": false, + "wall_s": 23.05611538887024, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/cpp_default__glove__1000000__core+mutation.json b/benchmarks/competitive/results/raw/cpp_default__glove__1000000__core+mutation.json new file mode 100644 index 0000000..9dbe006 --- /dev/null +++ b/benchmarks/competitive/results/raw/cpp_default__glove__1000000__core+mutation.json @@ -0,0 +1,310 @@ +{ + "status": "ok", + "label": "cpp_default", + "runs": [ + { + "system": "cpp_default_exact", + "ds": "glove", + "n": 1000000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.5334006, + "index_s": 0, + "total_s": 0.5334006, + "first_query_ms": 281.1621, + "ready_s": 0.8145627 + }, + "mem_baseline": { + "rss_mb": 6.75, + "peak_mb": 6.75 + }, + "mem_after_ingest": { + "rss_mb": 676.207031, + "peak_mb": 676.207031 + }, + "mem_after_build": { + "rss_mb": 678.277344, + "peak_mb": 817.109375 + }, + "mem_after_first_query": { + "rss_mb": 678.277344, + "peak_mb": 817.109375 + }, + "sweep": [ + { + "nq": 30, + "recall": 1, + "qps_1t": 2.47707451, + "qps_1t_min": 2.47707451, + "qps_1t_max": 2.47707451, + "qps_1t_runs": [ + 2.47707451 + ], + "median_ms": 406.4044, + "p95_ms": 414.879, + "p99_ms": 419.6363, + "mean_ms": 403.702027, + "knob": null + } + ], + "mem_after_sweep": { + "rss_mb": 678.511719, + "peak_mb": 817.355469 + } + }, + { + "system": "cpp_default_ivf", + "ds": "glove", + "n": 1000000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "knob_name": "nprobe", + "build": { + "ingest_s": 0.5334006, + "index_s": 8.7187386, + "total_s": 9.2521392, + "first_query_ms": 0, + "ready_s": 9.2521392 + }, + "mem_baseline": { + "rss_mb": 6.75, + "peak_mb": 6.75 + }, + "mem_after_ingest": { + "rss_mb": 676.207031, + "peak_mb": 676.207031 + }, + "mem_after_build": { + "rss_mb": 1086.50391, + "peak_mb": 1147.94141 + }, + "index_stats": { + "nlist": 4000, + "trained_n": 1000000, + "index_bytes": 19616004, + "vector_bytes": 400000000, + "build_seconds": 7.6935123 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.2955, + "qps_1t": 6470.72241, + "qps_1t_min": 6388.37623, + "qps_1t_max": 6694.39933, + "qps_1t_runs": [ + 6388.37623, + 6694.39933, + 6470.72241 + ], + "median_ms": 0.153200002, + "p95_ms": 0.198399997, + "p99_ms": 0.223100011, + "mean_ms": 0.15348505, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.41905, + "qps_1t": 5428.91702, + "qps_1t_min": 5367.33504, + "qps_1t_max": 5467.80937, + "qps_1t_runs": [ + 5367.33504, + 5428.91702, + 5467.80937 + ], + "median_ms": 0.182500051, + "p95_ms": 0.237400003, + "p99_ms": 0.259799999, + "mean_ms": 0.184466533, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.533, + "qps_1t": 4323.66619, + "qps_1t_min": 4300.91156, + "qps_1t_max": 4344.25436, + "qps_1t_runs": [ + 4300.91156, + 4344.25436, + 4323.66619 + ], + "median_ms": 0.228299992, + "p95_ms": 0.309300027, + "p99_ms": 0.34639996, + "mean_ms": 0.231327717, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.63225, + "qps_1t": 2984.86702, + "qps_1t_min": 2981.55995, + "qps_1t_max": 3000.01515, + "qps_1t_runs": [ + 2981.55995, + 3000.01515, + 2984.86702 + ], + "median_ms": 0.326899986, + "p95_ms": 0.459200004, + "p99_ms": 0.510800048, + "mean_ms": 0.334583283, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.72055, + "qps_1t": 1907.67083, + "qps_1t_min": 1818.11356, + "qps_1t_max": 1972.01183, + "qps_1t_runs": [ + 1972.01183, + 1907.67083, + 1818.11356 + ], + "median_ms": 0.514300016, + "p95_ms": 0.744500023, + "p99_ms": 0.84750005, + "mean_ms": 0.527105483, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.7965, + "qps_1t": 1085.6769, + "qps_1t_min": 1064.86707, + "qps_1t_max": 1085.74745, + "qps_1t_runs": [ + 1064.86707, + 1085.6769, + 1085.74745 + ], + "median_ms": 0.904699962, + "p95_ms": 1.28909998, + "p99_ms": 1.43200002, + "mean_ms": 0.9270644, + "knob": 32 + }, + { + "nq": 1797, + "recall": 0.86360601, + "qps_1t": 589.219043, + "qps_1t_min": 587.792598, + "qps_1t_max": 594.365455, + "qps_1t_runs": [ + 587.792598, + 589.219043, + 594.365455 + ], + "median_ms": 1.6747, + "p95_ms": 2.29819998, + "p99_ms": 2.51219998, + "mean_ms": 1.69363621, + "knob": 64 + }, + { + "nq": 924, + "recall": 0.920238095, + "qps_1t": 321.655711, + "qps_1t_min": 313.351118, + "qps_1t_max": 325.964363, + "qps_1t_runs": [ + 313.351118, + 321.655711, + 325.964363 + ], + "median_ms": 3.10430001, + "p95_ms": 4.07359999, + "p99_ms": 4.39140003, + "mean_ms": 3.12268081, + "knob": 128 + }, + { + "nq": 494, + "recall": 0.950809717, + "qps_1t": 171.933643, + "qps_1t_min": 167.389519, + "qps_1t_max": 174.846145, + "qps_1t_runs": [ + 174.846145, + 171.933643, + 167.389519 + ], + "median_ms": 5.80150005, + "p95_ms": 7.4308, + "p99_ms": 8.0184, + "mean_ms": 5.83653367, + "knob": 256 + } + ], + "ref_knob": 256, + "ref_recall_pre": 0.950809717, + "mem_after_sweep": { + "rss_mb": 1086.53516, + "peak_mb": 1147.94141 + }, + "concurrent_clients": { + "clients": 8, + "qps": 86.5493204 + }, + "mutation": { + "ref_knob": 256, + "recall_before": 0.950809717, + "insert_10k": { + "seconds": 0.1552916, + "per_s": 64394.9834 + }, + "update_10k": { + "seconds": 0.0161184, + "per_s": 620408.975 + }, + "delete_10k": { + "seconds": 0.0124532, + "per_s": 803006.456 + }, + "first_query_after_ms": 92.4923, + "recall_after": 0.950687623, + "after": { + "nq": 509, + "recall": 0.950687623, + "qps_1t": 170.170001, + "qps_1t_min": 168.283266, + "qps_1t_max": 173.872967, + "qps_1t_runs": [ + 168.283266, + 170.170001, + 173.872967 + ], + "median_ms": 5.83350001, + "p95_ms": 7.47169997, + "p99_ms": 7.98280002, + "mean_ms": 5.85672109 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 1000000, + "builds": 1 + }, + "mem_after": { + "rss_mb": 1102.64062, + "peak_mb": 1468.00781 + } + } + } + ], + "load_before_pct": 4.415625000000001, + "load_contaminated": false, + "wall_s": 83.35169792175293, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/cpp_default__glove__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/cpp_default__glove__100000__core+mutation+saveload.json new file mode 100644 index 0000000..5390d90 --- /dev/null +++ b/benchmarks/competitive/results/raw/cpp_default__glove__100000__core+mutation+saveload.json @@ -0,0 +1,337 @@ +{ + "status": "ok", + "label": "cpp_default", + "runs": [ + { + "system": "cpp_default_exact", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.0382684001, + "index_s": 0, + "total_s": 0.0382684001, + "first_query_ms": 25.487, + "ready_s": 0.0637554001 + }, + "mem_baseline": { + "rss_mb": 6.7421875, + "peak_mb": 6.7421875 + }, + "mem_after_ingest": { + "rss_mb": 76.28125, + "peak_mb": 81.6015625 + }, + "mem_after_build": { + "rss_mb": 78.6054688, + "peak_mb": 90.7890625 + }, + "mem_after_first_query": { + "rss_mb": 78.6054688, + "peak_mb": 90.7890625 + }, + "sweep": [ + { + "nq": 123, + "recall": 1, + "qps_1t": 40.1078791, + "qps_1t_min": 39.6911589, + "qps_1t_max": 40.1278362, + "qps_1t_runs": [ + 40.1278362, + 40.1078791, + 39.6911589 + ], + "median_ms": 24.4744, + "p95_ms": 28.4097, + "p99_ms": 29.7178, + "mean_ms": 25.0158805, + "knob": null + } + ], + "mem_after_sweep": { + "rss_mb": 78.8632812, + "peak_mb": 97.1484375 + } + }, + { + "system": "cpp_default_ivf", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "knob_name": "nprobe", + "build": { + "ingest_s": 0.0382684001, + "index_s": 0.2896739, + "total_s": 0.3279423, + "first_query_ms": 0, + "ready_s": 0.3279423 + }, + "mem_baseline": { + "rss_mb": 6.7421875, + "peak_mb": 6.7421875 + }, + "mem_after_ingest": { + "rss_mb": 76.28125, + "peak_mb": 81.6015625 + }, + "mem_after_build": { + "rss_mb": 118.519531, + "peak_mb": 138.285156 + }, + "index_stats": { + "nlist": 1265, + "trained_n": 100000, + "index_bytes": 2311064, + "vector_bytes": 40000000, + "build_seconds": 0.2581773 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.28485, + "qps_1t": 31629.5548, + "qps_1t_min": 24600.5788, + "qps_1t_max": 32671.3038, + "qps_1t_runs": [ + 24600.5788, + 31629.5548, + 32671.3038 + ], + "median_ms": 0.0317000085, + "p95_ms": 0.0468000071, + "p99_ms": 0.057999976, + "mean_ms": 0.0342911166, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.4074, + "qps_1t": 27774.6533, + "qps_1t_min": 27260.8068, + "qps_1t_max": 27919.1962, + "qps_1t_runs": [ + 27774.6533, + 27919.1962, + 27260.8068 + ], + "median_ms": 0.0347000314, + "p95_ms": 0.0441999873, + "p99_ms": 0.0598999904, + "mean_ms": 0.0361681333, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.5255, + "qps_1t": 22907.592, + "qps_1t_min": 22737.0911, + "qps_1t_max": 23047.3958, + "qps_1t_runs": [ + 22737.0911, + 22907.592, + 23047.3958 + ], + "median_ms": 0.041900028, + "p95_ms": 0.0548999524, + "p99_ms": 0.0744999852, + "mean_ms": 0.0436744999, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.63475, + "qps_1t": 16744.2623, + "qps_1t_min": 16489.6095, + "qps_1t_max": 17300.649, + "qps_1t_runs": [ + 17300.649, + 16744.2623, + 16489.6095 + ], + "median_ms": 0.0569000258, + "p95_ms": 0.0772000058, + "p99_ms": 0.0990999979, + "mean_ms": 0.0593891667, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.7357, + "qps_1t": 11814.7658, + "qps_1t_min": 11675.58, + "qps_1t_max": 12043.564, + "qps_1t_runs": [ + 12043.564, + 11814.7658, + 11675.58 + ], + "median_ms": 0.0818999833, + "p95_ms": 0.10970002, + "p99_ms": 0.136199989, + "mean_ms": 0.0844402, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.81855, + "qps_1t": 7115.70059, + "qps_1t_min": 6877.48922, + "qps_1t_max": 7441.4036, + "qps_1t_runs": [ + 6877.48922, + 7115.70059, + 7441.4036 + ], + "median_ms": 0.136399991, + "p95_ms": 0.187200028, + "p99_ms": 0.223700015, + "mean_ms": 0.140106483, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.88455, + "qps_1t": 4280.83566, + "qps_1t_min": 4217.9935, + "qps_1t_max": 4368.74382, + "qps_1t_runs": [ + 4217.9935, + 4368.74382, + 4280.83566 + ], + "median_ms": 0.227399985, + "p95_ms": 0.30449999, + "p99_ms": 0.352500007, + "mean_ms": 0.233192517, + "knob": 64 + }, + { + "nq": 2000, + "recall": 0.93865, + "qps_1t": 2360.84751, + "qps_1t_min": 2313.6529, + "qps_1t_max": 2396.98191, + "qps_1t_runs": [ + 2396.98191, + 2360.84751, + 2313.6529 + ], + "median_ms": 0.410900044, + "p95_ms": 0.546699972, + "p99_ms": 0.647700042, + "mean_ms": 0.424328317, + "knob": 128 + }, + { + "nq": 2000, + "recall": 0.9741, + "qps_1t": 1235.22005, + "qps_1t_min": 1233.54239, + "qps_1t_max": 1247.2061, + "qps_1t_runs": [ + 1247.2061, + 1233.54239, + 1235.22005 + ], + "median_ms": 0.78100001, + "p95_ms": 1.00029999, + "p99_ms": 1.20139995, + "mean_ms": 0.80734595, + "knob": 256 + }, + { + "nq": 116, + "recall": 1, + "qps_1t": 39.7482892, + "qps_1t_min": 39.6881103, + "qps_1t_max": 40.324651, + "qps_1t_runs": [ + 39.6881103, + 39.7482892, + 40.324651 + ], + "median_ms": 24.6437, + "p95_ms": 27.7731, + "p99_ms": 28.9565, + "mean_ms": 25.0511684, + "knob": 1265 + } + ], + "ref_knob": 256, + "ref_recall_pre": 0.9741, + "mem_after_sweep": { + "rss_mb": 120.101562, + "peak_mb": 138.421875 + }, + "concurrent_clients": { + "clients": 8, + "qps": 1125.05175 + }, + "mutation": { + "ref_knob": 256, + "recall_before": 0.9741, + "insert_10k": { + "seconds": 0.0125325, + "per_s": 797925.392 + }, + "update_10k": { + "seconds": 0.00671849999, + "per_s": 1488427.48 + }, + "delete_10k": { + "seconds": 0.00396060001, + "per_s": 2524869.97 + }, + "first_query_after_ms": 22.0308, + "recall_after": 0.9725, + "after": { + "nq": 2000, + "recall": 0.9725, + "qps_1t": 1211.13131, + "qps_1t_min": 1133.34437, + "qps_1t_max": 1228.4226, + "qps_1t_runs": [ + 1211.13131, + 1133.34437, + 1228.4226 + ], + "median_ms": 0.818499946, + "p95_ms": 1.0772, + "p99_ms": 1.2539, + "mean_ms": 0.84069025 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 100000, + "builds": 1 + }, + "mem_after": { + "rss_mb": 137.675781, + "peak_mb": 160.726562 + } + }, + "saveload": { + "save_s": 4.4033457, + "disk_mb": 133.363812, + "load_s": 2.0282403, + "first_query_after_load_ms": 302.2448, + "time_to_ready_s": 2.3304851, + "note": "load = JSON parse + insert; first query builds the IVF index (never persisted)" + } + } + ], + "load_before_pct": 3.4374999999999982, + "load_contaminated": false, + "wall_s": 44.69797420501709, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/cpp_default__glove__100000__filter.json b/benchmarks/competitive/results/raw/cpp_default__glove__100000__filter.json new file mode 100644 index 0000000..b9d0dfd --- /dev/null +++ b/benchmarks/competitive/results/raw/cpp_default__glove__100000__filter.json @@ -0,0 +1,237 @@ +{ + "status": "ok", + "label": "cpp_default", + "runs": [ + { + "system": "cpp_default_ivf", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "filter": { + "build_with_metadata_s": 0.3537934, + "mem": { + "rss_mb": 117.808594, + "peak_mb": 137.585938 + }, + "subsets": { + "f10": [ + { + "nq": 2000, + "recall": 0.26255, + "qps_1t": 33753.2805, + "qps_1t_min": 33367.8693, + "qps_1t_max": 34441.6662, + "qps_1t_runs": [ + 33367.8693, + 33753.2805, + 34441.6662 + ], + "median_ms": 0.0278000371, + "p95_ms": 0.0478999573, + "p99_ms": 0.0546999509, + "mean_ms": 0.0295434332, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.4765, + "qps_1t": 31251.758, + "qps_1t_min": 31157.2584, + "qps_1t_max": 31479.4557, + "qps_1t_runs": [ + 31157.2584, + 31251.758, + 31479.4557 + ], + "median_ms": 0.0318000093, + "p95_ms": 0.0360999838, + "p99_ms": 0.0399999553, + "mean_ms": 0.0319534, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.6976, + "qps_1t": 20020.8818, + "qps_1t_min": 19130.6459, + "qps_1t_max": 20067.4668, + "qps_1t_runs": [ + 20067.4668, + 19130.6459, + 20020.8818 + ], + "median_ms": 0.0501999748, + "p95_ms": 0.0591999851, + "p99_ms": 0.0674999901, + "mean_ms": 0.0506839668, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.79485, + "qps_1t": 14290.0013, + "qps_1t_min": 14008.2116, + "qps_1t_max": 14508.3238, + "qps_1t_runs": [ + 14008.2116, + 14508.3238, + 14290.0013 + ], + "median_ms": 0.0694000046, + "p95_ms": 0.0826000469, + "p99_ms": 0.0914999982, + "mean_ms": 0.0700972166, + "knob": 64 + }, + { + "nq": 476, + "recall": 1, + "qps_1t": 151.949656, + "qps_1t_min": 150.413969, + "qps_1t_max": 151.984651, + "qps_1t_runs": [ + 151.949656, + 150.413969, + 151.984651 + ], + "median_ms": 6.43150002, + "p95_ms": 8.29480001, + "p99_ms": 9.02339997, + "mean_ms": 6.60301919, + "knob": 1265 + }, + { + "nq": 462, + "recall": 1, + "qps_1t": 151.847621, + "qps_1t_min": 148.011494, + "qps_1t_max": 152.322437, + "qps_1t_runs": [ + 152.322437, + 148.011494, + 151.847621 + ], + "median_ms": 6.44499995, + "p95_ms": 8.38429999, + "p99_ms": 9.42820002, + "mean_ms": 6.63560079, + "knob": "exact" + } + ], + "f01": [ + { + "nq": 2000, + "recall": 0.3655, + "qps_1t": 15167.1764, + "qps_1t_min": 14900.6537, + "qps_1t_max": 15774.6621, + "qps_1t_runs": [ + 14900.6537, + 15774.6621, + 15167.1764 + ], + "median_ms": 0.0676999916, + "p95_ms": 0.0848000054, + "p99_ms": 0.100200006, + "mean_ms": 0.0654785999, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.3655, + "qps_1t": 21750.6204, + "qps_1t_min": 21094.799, + "qps_1t_max": 21798.175, + "qps_1t_runs": [ + 21094.799, + 21798.175, + 21750.6204 + ], + "median_ms": 0.0523999915, + "p95_ms": 0.0633999589, + "p99_ms": 0.0682999962, + "mean_ms": 0.0464187167, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.45215, + "qps_1t": 26973.481, + "qps_1t_min": 26932.7623, + "qps_1t_max": 27140.7247, + "qps_1t_runs": [ + 26932.7623, + 26973.481, + 27140.7247 + ], + "median_ms": 0.0369999907, + "p95_ms": 0.0421999721, + "p99_ms": 0.0462000025, + "mean_ms": 0.0370159833, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.59105, + "qps_1t": 19639.0538, + "qps_1t_min": 19081.1837, + "qps_1t_max": 20065.5945, + "qps_1t_runs": [ + 20065.5945, + 19639.0538, + 19081.1837 + ], + "median_ms": 0.0499999733, + "p95_ms": 0.0601999927, + "p99_ms": 0.0703000114, + "mean_ms": 0.0510543833, + "knob": 64 + }, + { + "nq": 880, + "recall": 1, + "qps_1t": 315.801086, + "qps_1t_min": 272.336549, + "qps_1t_max": 334.744217, + "qps_1t_runs": [ + 272.336549, + 315.801086, + 334.744217 + ], + "median_ms": 3.0573, + "p95_ms": 5.1674, + "p99_ms": 6.98509999, + "mean_ms": 3.27527769, + "knob": 1265 + }, + { + "nq": 1161, + "recall": 1, + "qps_1t": 312.327972, + "qps_1t_min": 309.935259, + "qps_1t_max": 322.174201, + "qps_1t_runs": [ + 312.327972, + 322.174201, + 309.935259 + ], + "median_ms": 3.02359997, + "p95_ms": 4.52809996, + "p99_ms": 6.52709999, + "mean_ms": 3.17738455, + "knob": "exact" + } + ] + } + } + } + ], + "load_before_pct": 3.331249999999997, + "load_contaminated": false, + "wall_s": 42.20636296272278, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/cpp_default__sift__1000000__core.json b/benchmarks/competitive/results/raw/cpp_default__sift__1000000__core.json new file mode 100644 index 0000000..997a89f --- /dev/null +++ b/benchmarks/competitive/results/raw/cpp_default__sift__1000000__core.json @@ -0,0 +1,266 @@ +{ + "status": "ok", + "label": "cpp_default", + "runs": [ + { + "system": "cpp_default_exact", + "ds": "sift", + "n": 1000000, + "d": 128, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.6148045, + "index_s": 0, + "total_s": 0.6148045, + "first_query_ms": 329.5979, + "ready_s": 0.9444024 + }, + "mem_baseline": { + "rss_mb": 7.0859375, + "peak_mb": 7.0859375 + }, + "mem_after_ingest": { + "rss_mb": 787.199219, + "peak_mb": 787.199219 + }, + "mem_after_build": { + "rss_mb": 790.34375, + "peak_mb": 929.175781 + }, + "mem_after_first_query": { + "rss_mb": 790.34375, + "peak_mb": 929.175781 + }, + "sweep": [ + { + "nq": 30, + "recall": 1, + "qps_1t": 2.18706115, + "qps_1t_min": 2.18706115, + "qps_1t_max": 2.18706115, + "qps_1t_runs": [ + 2.18706115 + ], + "median_ms": 458.9676, + "p95_ms": 471.8328, + "p99_ms": 473.9323, + "mean_ms": 457.234587, + "knob": null + } + ], + "mem_after_sweep": { + "rss_mb": 790.472656, + "peak_mb": 929.316406 + } + }, + { + "system": "cpp_default_ivf", + "ds": "sift", + "n": 1000000, + "d": 128, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "knob_name": "nprobe", + "build": { + "ingest_s": 0.6148045, + "index_s": 10.9004137, + "total_s": 11.5152182, + "first_query_ms": 0, + "ready_s": 11.5152182 + }, + "mem_baseline": { + "rss_mb": 7.0859375, + "peak_mb": 7.0859375 + }, + "mem_after_ingest": { + "rss_mb": 787.199219, + "peak_mb": 787.199219 + }, + "mem_after_build": { + "rss_mb": 1305.70312, + "peak_mb": 1384.32812 + }, + "index_stats": { + "nlist": 4000, + "trained_n": 1000000, + "index_bytes": 20064004, + "vector_bytes": 512000000, + "build_seconds": 9.8352777 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.25485, + "qps_1t": 5716.30765, + "qps_1t_min": 5605.84466, + "qps_1t_max": 5905.34209, + "qps_1t_runs": [ + 5605.84466, + 5905.34209, + 5716.30765 + ], + "median_ms": 0.174399989, + "p95_ms": 0.222200004, + "p99_ms": 0.276899955, + "mean_ms": 0.174220517, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.38895, + "qps_1t": 4863.64882, + "qps_1t_min": 4720.03914, + "qps_1t_max": 4876.94849, + "qps_1t_runs": [ + 4876.94849, + 4720.03914, + 4863.64882 + ], + "median_ms": 0.204399985, + "p95_ms": 0.25859999, + "p99_ms": 0.300999964, + "mean_ms": 0.207505283, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.5391, + "qps_1t": 3920.99816, + "qps_1t_min": 3790.9338, + "qps_1t_max": 3942.3883, + "qps_1t_runs": [ + 3942.3883, + 3920.99816, + 3790.9338 + ], + "median_ms": 0.251100049, + "p95_ms": 0.343799999, + "p99_ms": 0.416600029, + "mean_ms": 0.257492567, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.6881, + "qps_1t": 2732.08181, + "qps_1t_min": 2600.88068, + "qps_1t_max": 2734.66914, + "qps_1t_runs": [ + 2734.66914, + 2732.08181, + 2600.88068 + ], + "median_ms": 0.356399978, + "p95_ms": 0.525599986, + "p99_ms": 0.640000042, + "mean_ms": 0.37206045, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.8212, + "qps_1t": 1685.54647, + "qps_1t_min": 1660.67385, + "qps_1t_max": 1687.27875, + "qps_1t_runs": [ + 1660.67385, + 1687.27875, + 1685.54647 + ], + "median_ms": 0.5657, + "p95_ms": 0.879300002, + "p99_ms": 1.04960002, + "mean_ms": 0.5960383, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.9147, + "qps_1t": 979.311554, + "qps_1t_min": 943.580269, + "qps_1t_max": 988.607436, + "qps_1t_runs": [ + 988.607436, + 979.311554, + 943.580269 + ], + "median_ms": 0.984299986, + "p95_ms": 1.51490001, + "p99_ms": 1.7423, + "mean_ms": 1.0308142, + "knob": 32 + }, + { + "nq": 1522, + "recall": 0.969973719, + "qps_1t": 562.161819, + "qps_1t_min": 525.069372, + "qps_1t_max": 579.249538, + "qps_1t_runs": [ + 525.069372, + 562.161819, + 579.249538 + ], + "median_ms": 1.74880004, + "p95_ms": 2.54020002, + "p99_ms": 2.8281, + "mean_ms": 1.80324306, + "knob": 64 + }, + { + "nq": 870, + "recall": 0.99183908, + "qps_1t": 297.31564, + "qps_1t_min": 291.993145, + "qps_1t_max": 301.956522, + "qps_1t_runs": [ + 301.956522, + 297.31564, + 291.993145 + ], + "median_ms": 3.30139999, + "p95_ms": 4.47340001, + "p99_ms": 4.99910006, + "mean_ms": 3.36663395, + "knob": 128 + }, + { + "nq": 411, + "recall": 0.997810219, + "qps_1t": 155.294277, + "qps_1t_min": 153.956001, + "qps_1t_max": 174.005036, + "qps_1t_runs": [ + 153.956001, + 155.294277, + 174.005036 + ], + "median_ms": 6.18129998, + "p95_ms": 7.79280002, + "p99_ms": 8.3553, + "mean_ms": 6.22723658, + "knob": 256 + } + ], + "ref_knob": 64, + "ref_recall_pre": 0.969973719, + "mem_after_sweep": { + "rss_mb": 1305.69141, + "peak_mb": 1384.32812 + }, + "concurrent_clients": { + "clients": 8, + "qps": 312.653982 + } + } + ], + "load_before_pct": 11.962499999999993, + "load_contaminated": false, + "wall_s": 74.68000960350037, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/cpp_default__sift__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/cpp_default__sift__100000__core+mutation+saveload.json new file mode 100644 index 0000000..4dbd6d9 --- /dev/null +++ b/benchmarks/competitive/results/raw/cpp_default__sift__100000__core+mutation+saveload.json @@ -0,0 +1,337 @@ +{ + "status": "ok", + "label": "cpp_default", + "runs": [ + { + "system": "cpp_default_exact", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.0495777, + "index_s": 0, + "total_s": 0.0495777, + "first_query_ms": 31.582, + "ready_s": 0.0811597 + }, + "mem_baseline": { + "rss_mb": 7.12890625, + "peak_mb": 7.12890625 + }, + "mem_after_ingest": { + "rss_mb": 87.4921875, + "peak_mb": 95.109375 + }, + "mem_after_build": { + "rss_mb": 91.2265625, + "peak_mb": 109.503906 + }, + "mem_after_first_query": { + "rss_mb": 91.2265625, + "peak_mb": 109.503906 + }, + "sweep": [ + { + "nq": 92, + "recall": 1, + "qps_1t": 22.7960828, + "qps_1t_min": 22.4102038, + "qps_1t_max": 25.158072, + "qps_1t_runs": [ + 25.158072, + 22.4102038, + 22.7960828 + ], + "median_ms": 44.2155, + "p95_ms": 47.4825, + "p99_ms": 49.4587, + "mean_ms": 42.7461301, + "knob": null + } + ], + "mem_after_sweep": { + "rss_mb": 91.2578125, + "peak_mb": 109.542969 + } + }, + { + "system": "cpp_default_ivf", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "knob_name": "nprobe", + "build": { + "ingest_s": 0.0495777, + "index_s": 0.9699913, + "total_s": 1.019569, + "first_query_ms": 0, + "ready_s": 1.019569 + }, + "mem_baseline": { + "rss_mb": 7.12890625, + "peak_mb": 7.12890625 + }, + "mem_after_ingest": { + "rss_mb": 87.4921875, + "peak_mb": 95.109375 + }, + "mem_after_build": { + "rss_mb": 141.617188, + "peak_mb": 166.804688 + }, + "index_stats": { + "nlist": 1265, + "trained_n": 100000, + "index_bytes": 2452744, + "vector_bytes": 51200000, + "build_seconds": 0.8808714 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.27955, + "qps_1t": 19362.4338, + "qps_1t_min": 18832.906, + "qps_1t_max": 19708.0838, + "qps_1t_runs": [ + 19362.4338, + 19708.0838, + 18832.906 + ], + "median_ms": 0.0520999893, + "p95_ms": 0.064599968, + "p99_ms": 0.0718000229, + "mean_ms": 0.0518285167, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.42195, + "qps_1t": 16480.071, + "qps_1t_min": 16362.729, + "qps_1t_max": 16797.3077, + "qps_1t_runs": [ + 16797.3077, + 16362.729, + 16480.071 + ], + "median_ms": 0.0592999859, + "p95_ms": 0.0746999867, + "p99_ms": 0.0862000161, + "mean_ms": 0.0604424, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.5793, + "qps_1t": 13532.697, + "qps_1t_min": 13443.1007, + "qps_1t_max": 13864.0972, + "qps_1t_runs": [ + 13443.1007, + 13532.697, + 13864.0972 + ], + "median_ms": 0.0711999601, + "p95_ms": 0.0971000409, + "p99_ms": 0.11839997, + "mean_ms": 0.0734704834, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.7401, + "qps_1t": 10562.7015, + "qps_1t_min": 10319.2891, + "qps_1t_max": 10608.515, + "qps_1t_runs": [ + 10608.515, + 10319.2891, + 10562.7015 + ], + "median_ms": 0.0921000028, + "p95_ms": 0.129299995, + "p99_ms": 0.153700006, + "mean_ms": 0.09528085, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.86955, + "qps_1t": 6672.37154, + "qps_1t_min": 6639.22894, + "qps_1t_max": 6802.39485, + "qps_1t_runs": [ + 6802.39485, + 6639.22894, + 6672.37154 + ], + "median_ms": 0.144199992, + "p95_ms": 0.203599979, + "p99_ms": 0.242499984, + "mean_ms": 0.149166233, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.95215, + "qps_1t": 4053.93107, + "qps_1t_min": 3966.45725, + "qps_1t_max": 4059.59156, + "qps_1t_runs": [ + 4059.59156, + 4053.93107, + 3966.45725 + ], + "median_ms": 0.241100031, + "p95_ms": 0.336199999, + "p99_ms": 0.392999966, + "mean_ms": 0.248372833, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.98825, + "qps_1t": 2290.30143, + "qps_1t_min": 2201.50241, + "qps_1t_max": 2307.09699, + "qps_1t_runs": [ + 2307.09699, + 2201.50241, + 2290.30143 + ], + "median_ms": 0.429699954, + "p95_ms": 0.605000008, + "p99_ms": 0.738800038, + "mean_ms": 0.441434717, + "knob": 64 + }, + { + "nq": 2000, + "recall": 0.9984, + "qps_1t": 1321.13643, + "qps_1t_min": 1304.80848, + "qps_1t_max": 1338.60333, + "qps_1t_runs": [ + 1304.80848, + 1321.13643, + 1338.60333 + ], + "median_ms": 0.736700022, + "p95_ms": 1.01829995, + "p99_ms": 1.21630001, + "mean_ms": 0.756789133, + "knob": 128 + }, + { + "nq": 2000, + "recall": 0.9999, + "qps_1t": 678.918751, + "qps_1t_min": 677.351434, + "qps_1t_max": 687.975468, + "qps_1t_runs": [ + 678.918751, + 687.975468, + 677.351434 + ], + "median_ms": 1.45109999, + "p95_ms": 1.84019998, + "p99_ms": 2.0563, + "mean_ms": 1.467603, + "knob": 256 + }, + { + "nq": 66, + "recall": 1, + "qps_1t": 22.339829, + "qps_1t_min": 22.2738571, + "qps_1t_max": 23.5006731, + "qps_1t_runs": [ + 22.339829, + 22.2738571, + 23.5006731 + ], + "median_ms": 44.7927, + "p95_ms": 47.5427, + "p99_ms": 49.3286, + "mean_ms": 44.0702515, + "knob": 1265 + } + ], + "ref_knob": 32, + "ref_recall_pre": 0.95215, + "mem_after_sweep": { + "rss_mb": 142.863281, + "peak_mb": 166.804688 + }, + "concurrent_clients": { + "clients": 8, + "qps": 2806.07825 + }, + "mutation": { + "ref_knob": 32, + "recall_before": 0.95215, + "insert_10k": { + "seconds": 0.0249564, + "per_s": 400698.819 + }, + "update_10k": { + "seconds": 0.0119626001, + "per_s": 835938.672 + }, + "delete_10k": { + "seconds": 0.007278, + "per_s": 1374003.85 + }, + "first_query_after_ms": 37.1066, + "recall_after": 0.949, + "after": { + "nq": 2000, + "recall": 0.949, + "qps_1t": 4125.9768, + "qps_1t_min": 4119.45514, + "qps_1t_max": 4230.51036, + "qps_1t_runs": [ + 4230.51036, + 4119.45514, + 4125.9768 + ], + "median_ms": 0.233200029, + "p95_ms": 0.327699992, + "p99_ms": 0.402999984, + "mean_ms": 0.2404985 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 100000, + "builds": 1 + }, + "mem_after": { + "rss_mb": 163.992188, + "peak_mb": 194.105469 + } + }, + "saveload": { + "save_s": 12.4136456, + "disk_mb": 133.678411, + "load_s": 4.9743937, + "first_query_after_load_ms": 826.4113, + "time_to_ready_s": 5.800805, + "note": "load = JSON parse + insert; first query builds the IVF index (never persisted)" + } + } + ], + "load_before_pct": 13.984374999999982, + "load_contaminated": false, + "wall_s": 66.20936942100525, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/cpp_default__text384__1000000__core+mutation.json b/benchmarks/competitive/results/raw/cpp_default__text384__1000000__core+mutation.json new file mode 100644 index 0000000..cd5eb36 --- /dev/null +++ b/benchmarks/competitive/results/raw/cpp_default__text384__1000000__core+mutation.json @@ -0,0 +1,310 @@ +{ + "status": "ok", + "label": "cpp_default", + "runs": [ + { + "system": "cpp_default_exact", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.9167631, + "index_s": 0, + "total_s": 0.9167631, + "first_query_ms": 508.9527, + "ready_s": 1.4257158 + }, + "mem_baseline": { + "rss_mb": 11.0585938, + "peak_mb": 11.0585938 + }, + "mem_after_ingest": { + "rss_mb": 1791.36719, + "peak_mb": 1791.36719 + }, + "mem_after_build": { + "rss_mb": 1794.48047, + "peak_mb": 1933.3125 + }, + "mem_after_first_query": { + "rss_mb": 1794.48047, + "peak_mb": 1933.3125 + }, + "sweep": [ + { + "nq": 30, + "recall": 1, + "qps_1t": 2.1147531, + "qps_1t_min": 2.1147531, + "qps_1t_max": 2.1147531, + "qps_1t_runs": [ + 2.1147531 + ], + "median_ms": 469.0725, + "p95_ms": 499.9009, + "p99_ms": 516.0898, + "mean_ms": 472.86844, + "knob": null + } + ], + "mem_after_sweep": { + "rss_mb": 1794.50781, + "peak_mb": 1933.35156 + } + }, + { + "system": "cpp_default_ivf", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "knob_name": "nprobe", + "build": { + "ingest_s": 0.9167631, + "index_s": 11.4398135, + "total_s": 12.3565766, + "first_query_ms": 0, + "ready_s": 12.3565766 + }, + "mem_baseline": { + "rss_mb": 11.0585938, + "peak_mb": 11.0585938 + }, + "mem_after_ingest": { + "rss_mb": 1791.36719, + "peak_mb": 1791.36719 + }, + "mem_after_build": { + "rss_mb": 3290.40234, + "peak_mb": 3525.29688 + }, + "index_stats": { + "nlist": 4000, + "trained_n": 1000000, + "index_bytes": 24160004, + "vector_bytes": 1536000000, + "build_seconds": 10.7134662 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.2669, + "qps_1t": 4176.96637, + "qps_1t_min": 3634.68904, + "qps_1t_max": 4193.69151, + "qps_1t_runs": [ + 3634.68904, + 4176.96637, + 4193.69151 + ], + "median_ms": 0.238600012, + "p95_ms": 0.333599979, + "p99_ms": 0.549699995, + "mean_ms": 0.2509961, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.3726, + "qps_1t": 3308.03787, + "qps_1t_min": 3221.80466, + "qps_1t_max": 3336.83646, + "qps_1t_runs": [ + 3221.80466, + 3308.03787, + 3336.83646 + ], + "median_ms": 0.297199993, + "p95_ms": 0.398600008, + "p99_ms": 0.456799986, + "mean_ms": 0.304121367, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.4897, + "qps_1t": 2387.27326, + "qps_1t_min": 2353.06464, + "qps_1t_max": 2421.47519, + "qps_1t_runs": [ + 2421.47519, + 2353.06464, + 2387.27326 + ], + "median_ms": 0.409199973, + "p95_ms": 0.556299987, + "p99_ms": 0.667600019, + "mean_ms": 0.418945683, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.6026, + "qps_1t": 1588.828, + "qps_1t_min": 1576.05543, + "qps_1t_max": 1601.92244, + "qps_1t_runs": [ + 1601.92244, + 1588.828, + 1576.05543 + ], + "median_ms": 0.61280001, + "p95_ms": 0.819000008, + "p99_ms": 0.953999988, + "mean_ms": 0.62938005, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.701, + "qps_1t": 960.889538, + "qps_1t_min": 953.515454, + "qps_1t_max": 984.54743, + "qps_1t_runs": [ + 984.54743, + 960.889538, + 953.515454 + ], + "median_ms": 1.02059997, + "p95_ms": 1.32340001, + "p99_ms": 1.49370002, + "mean_ms": 1.03504938, + "knob": 16 + }, + { + "nq": 1627, + "recall": 0.78494161, + "qps_1t": 558.61326, + "qps_1t_min": 554.357325, + "qps_1t_max": 559.146092, + "qps_1t_runs": [ + 554.357325, + 559.146092, + 558.61326 + ], + "median_ms": 1.77579996, + "p95_ms": 2.2098, + "p99_ms": 2.50400003, + "mean_ms": 1.79415976, + "knob": 32 + }, + { + "nq": 969, + "recall": 0.86130031, + "qps_1t": 311.111696, + "qps_1t_min": 302.544644, + "qps_1t_max": 312.582914, + "qps_1t_runs": [ + 302.544644, + 311.111696, + 312.582914 + ], + "median_ms": 3.15619999, + "p95_ms": 3.87209997, + "p99_ms": 4.28369996, + "mean_ms": 3.23957606, + "knob": 64 + }, + { + "nq": 499, + "recall": 0.915230461, + "qps_1t": 173.003759, + "qps_1t_min": 170.55657, + "qps_1t_max": 173.031546, + "qps_1t_runs": [ + 173.031546, + 173.003759, + 170.55657 + ], + "median_ms": 5.78070001, + "p95_ms": 6.77370001, + "p99_ms": 7.47860002, + "mean_ms": 5.80755718, + "knob": 128 + }, + { + "nq": 286, + "recall": 0.952797203, + "qps_1t": 89.5362156, + "qps_1t_min": 89.4603815, + "qps_1t_max": 90.4557111, + "qps_1t_runs": [ + 89.5362156, + 89.4603815, + 90.4557111 + ], + "median_ms": 11.051, + "p95_ms": 12.9159, + "p99_ms": 14.1046, + "mean_ms": 11.1339772, + "knob": 256 + } + ], + "ref_knob": 256, + "ref_recall_pre": 0.952797203, + "mem_after_sweep": { + "rss_mb": 3290.59766, + "peak_mb": 3525.29688 + }, + "concurrent_clients": { + "clients": 8, + "qps": 91.1796603 + }, + "mutation": { + "ref_knob": 256, + "recall_before": 0.952797203, + "insert_10k": { + "seconds": 0.2923063, + "per_s": 34210.6893 + }, + "update_10k": { + "seconds": 0.0175668, + "per_s": 569255.641 + }, + "delete_10k": { + "seconds": 0.00725369999, + "per_s": 1378606.78 + }, + "first_query_after_ms": 149.3498, + "recall_after": 0.952895753, + "after": { + "nq": 259, + "recall": 0.952895753, + "qps_1t": 90.246679, + "qps_1t_min": 88.8601914, + "qps_1t_max": 90.7865402, + "qps_1t_runs": [ + 88.8601914, + 90.7865402, + 90.246679 + ], + "median_ms": 11.0211, + "p95_ms": 12.958, + "p99_ms": 13.6072999, + "mean_ms": 11.1164073 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 1000000, + "builds": 1 + }, + "mem_after": { + "rss_mb": 3338.33594, + "peak_mb": 4755.44922 + } + } + } + ], + "load_before_pct": 5.6687500000000055, + "load_contaminated": false, + "wall_s": 102.26529550552368, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/cpp_default__text384__1000000__filter.json b/benchmarks/competitive/results/raw/cpp_default__text384__1000000__filter.json new file mode 100644 index 0000000..9045427 --- /dev/null +++ b/benchmarks/competitive/results/raw/cpp_default__text384__1000000__filter.json @@ -0,0 +1,237 @@ +{ + "status": "ok", + "label": "cpp_default", + "runs": [ + { + "system": "cpp_default_ivf", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "filter": { + "build_with_metadata_s": 12.6140878, + "mem": { + "rss_mb": 3289.16016, + "peak_mb": 3524.01172 + }, + "subsets": { + "f10": [ + { + "nq": 2000, + "recall": 0.31395, + "qps_1t": 4559.66828, + "qps_1t_min": 3495.87705, + "qps_1t_max": 4560.78396, + "qps_1t_runs": [ + 3495.87705, + 4559.66828, + 4560.78396 + ], + "median_ms": 0.221399998, + "p95_ms": 0.370400026, + "p99_ms": 0.655800046, + "mean_ms": 0.241542, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.5416, + "qps_1t": 2946.32648, + "qps_1t_min": 2618.97924, + "qps_1t_max": 3099.19676, + "qps_1t_runs": [ + 3099.19676, + 2618.97924, + 2946.32648 + ], + "median_ms": 0.32500003, + "p95_ms": 0.500399969, + "p99_ms": 0.764900004, + "mean_ms": 0.347966033, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.648, + "qps_1t": 2171.86229, + "qps_1t_min": 2166.75193, + "qps_1t_max": 2376.16159, + "qps_1t_runs": [ + 2166.75193, + 2171.86229, + 2376.16159 + ], + "median_ms": 0.426300045, + "p95_ms": 0.613900018, + "p99_ms": 0.782200019, + "mean_ms": 0.447600483, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.8244, + "qps_1t": 961.102361, + "qps_1t_min": 957.682225, + "qps_1t_max": 964.846261, + "qps_1t_runs": [ + 961.102361, + 957.682225, + 964.846261 + ], + "median_ms": 1.01569999, + "p95_ms": 1.33850001, + "p99_ms": 1.54590001, + "mean_ms": 1.04036472, + "knob": 64 + }, + { + "nq": 929, + "recall": 0.942518837, + "qps_1t": 307.878202, + "qps_1t_min": 306.789746, + "qps_1t_max": 310.602782, + "qps_1t_runs": [ + 310.602782, + 307.878202, + 306.789746 + ], + "median_ms": 3.20199999, + "p95_ms": 3.86150001, + "p99_ms": 4.32079996, + "mean_ms": 3.24238174, + "knob": 256 + }, + { + "nq": 30, + "recall": 1, + "qps_1t": 7.46364682, + "qps_1t_min": 7.45864591, + "qps_1t_max": 7.49316773, + "qps_1t_runs": [ + 7.46364682, + 7.49316773, + 7.45864591 + ], + "median_ms": 132.9719, + "p95_ms": 142.3555, + "p99_ms": 149.1464, + "mean_ms": 133.836753, + "knob": "exact" + } + ], + "f01": [ + { + "nq": 2000, + "recall": 0.32855, + "qps_1t": 2516.98746, + "qps_1t_min": 2503.04057, + "qps_1t_max": 2540.33867, + "qps_1t_runs": [ + 2503.04057, + 2516.98746, + 2540.33867 + ], + "median_ms": 0.432599976, + "p95_ms": 0.510699989, + "p99_ms": 0.589700008, + "mean_ms": 0.396820917, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.3727, + "qps_1t": 3646.24031, + "qps_1t_min": 3606.5857, + "qps_1t_max": 3648.82271, + "qps_1t_runs": [ + 3606.5857, + 3648.82271, + 3646.24031 + ], + "median_ms": 0.267299998, + "p95_ms": 0.330500014, + "p99_ms": 0.429000007, + "mean_ms": 0.27519555, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.49245, + "qps_1t": 2901.21395, + "qps_1t_min": 2899.20326, + "qps_1t_max": 2929.89851, + "qps_1t_runs": [ + 2899.20326, + 2929.89851, + 2901.21395 + ], + "median_ms": 0.335500052, + "p95_ms": 0.425299979, + "p99_ms": 0.496099994, + "mean_ms": 0.343638133, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.7289, + "qps_1t": 1428.29087, + "qps_1t_min": 1427.75771, + "qps_1t_max": 1445.06447, + "qps_1t_runs": [ + 1428.29087, + 1445.06447, + 1427.75771 + ], + "median_ms": 0.675400021, + "p95_ms": 0.912100018, + "p99_ms": 1.0545, + "mean_ms": 0.6975157, + "knob": 64 + }, + { + "nq": 1513, + "recall": 0.90548579, + "qps_1t": 505.774234, + "qps_1t_min": 493.870651, + "qps_1t_max": 507.859623, + "qps_1t_runs": [ + 493.870651, + 507.859623, + 505.774234 + ], + "median_ms": 1.9569, + "p95_ms": 2.4612, + "p99_ms": 2.81969999, + "mean_ms": 1.99034549, + "knob": 256 + }, + { + "nq": 34, + "recall": 1, + "qps_1t": 11.2739486, + "qps_1t_min": 11.2538943, + "qps_1t_max": 11.305939, + "qps_1t_runs": [ + 11.2739486, + 11.305939, + 11.2538943 + ], + "median_ms": 87.8238, + "p95_ms": 91.8591, + "p99_ms": 105.5235, + "mean_ms": 88.6690951, + "knob": "exact" + } + ] + } + } + } + ], + "load_before_pct": 7.228125000000002, + "load_contaminated": false, + "wall_s": 84.08570837974548, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/cpp_default__text384__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/cpp_default__text384__100000__core+mutation+saveload.json new file mode 100644 index 0000000..d661f41 --- /dev/null +++ b/benchmarks/competitive/results/raw/cpp_default__text384__100000__core+mutation+saveload.json @@ -0,0 +1,337 @@ +{ + "status": "ok", + "label": "cpp_default", + "runs": [ + { + "system": "cpp_default_exact", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.0772106, + "index_s": 0, + "total_s": 0.0772106, + "first_query_ms": 44.9507, + "ready_s": 0.1221613 + }, + "mem_baseline": { + "rss_mb": 11.0585938, + "peak_mb": 11.0585938 + }, + "mem_after_ingest": { + "rss_mb": 200.128906, + "peak_mb": 217.085938 + }, + "mem_after_build": { + "rss_mb": 203.3125, + "peak_mb": 217.085938 + }, + "mem_after_first_query": { + "rss_mb": 203.3125, + "peak_mb": 217.085938 + }, + "sweep": [ + { + "nq": 66, + "recall": 1, + "qps_1t": 22.6002529, + "qps_1t_min": 22.4759446, + "qps_1t_max": 23.2881655, + "qps_1t_runs": [ + 22.4759446, + 22.6002529, + 23.2881655 + ], + "median_ms": 44.1773, + "p95_ms": 47.1510999, + "p99_ms": 57.6838, + "mean_ms": 43.8931899, + "knob": null + } + ], + "mem_after_sweep": { + "rss_mb": 203.503906, + "peak_mb": 217.085938 + } + }, + { + "system": "cpp_default_ivf", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "knob_name": "nprobe", + "build": { + "ingest_s": 0.0772106, + "index_s": 0.891219, + "total_s": 0.9684296, + "first_query_ms": 0, + "ready_s": 0.9684296 + }, + "mem_baseline": { + "rss_mb": 11.0585938, + "peak_mb": 11.0585938 + }, + "mem_after_ingest": { + "rss_mb": 200.128906, + "peak_mb": 217.085938 + }, + "mem_after_build": { + "rss_mb": 353.148438, + "peak_mb": 427.757812 + }, + "index_stats": { + "nlist": 1265, + "trained_n": 100000, + "index_bytes": 3748104, + "vector_bytes": 153600000, + "build_seconds": 0.8328012 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.28085, + "qps_1t": 16275.7636, + "qps_1t_min": 16013.5923, + "qps_1t_max": 17282.216, + "qps_1t_runs": [ + 16013.5923, + 16275.7636, + 17282.216 + ], + "median_ms": 0.0590000418, + "p95_ms": 0.0765000004, + "p99_ms": 0.0903000473, + "mean_ms": 0.06058365, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.40215, + "qps_1t": 14590.3504, + "qps_1t_min": 14423.6039, + "qps_1t_max": 14658.4687, + "qps_1t_runs": [ + 14423.6039, + 14590.3504, + 14658.4687 + ], + "median_ms": 0.0665000407, + "p95_ms": 0.0887000351, + "p99_ms": 0.109800021, + "mean_ms": 0.0686964, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.5291, + "qps_1t": 10815.4644, + "qps_1t_min": 10090.7308, + "qps_1t_max": 10997.6097, + "qps_1t_runs": [ + 10997.6097, + 10090.7308, + 10815.4644 + ], + "median_ms": 0.0913999975, + "p95_ms": 0.127099978, + "p99_ms": 0.151499989, + "mean_ms": 0.0941633, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.6479, + "qps_1t": 6398.64861, + "qps_1t_min": 6091.14299, + "qps_1t_max": 6901.58751, + "qps_1t_runs": [ + 6901.58751, + 6398.64861, + 6091.14299 + ], + "median_ms": 0.151699991, + "p95_ms": 0.206299999, + "p99_ms": 0.246800017, + "mean_ms": 0.155116667, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.75475, + "qps_1t": 3908.01471, + "qps_1t_min": 3831.05801, + "qps_1t_max": 3928.3917, + "qps_1t_runs": [ + 3908.01471, + 3928.3917, + 3831.05801 + ], + "median_ms": 0.249500037, + "p95_ms": 0.335699995, + "p99_ms": 0.409299973, + "mean_ms": 0.257155333, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.84275, + "qps_1t": 2254.86083, + "qps_1t_min": 2199.54419, + "qps_1t_max": 2286.80228, + "qps_1t_runs": [ + 2199.54419, + 2286.80228, + 2254.86083 + ], + "median_ms": 0.4357, + "p95_ms": 0.569800031, + "p99_ms": 0.649199996, + "mean_ms": 0.445139283, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.90905, + "qps_1t": 1232.11015, + "qps_1t_min": 1224.88563, + "qps_1t_max": 1233.79411, + "qps_1t_runs": [ + 1224.88563, + 1233.79411, + 1232.11015 + ], + "median_ms": 0.805900025, + "p95_ms": 0.992999994, + "p99_ms": 1.10200001, + "mean_ms": 0.812842167, + "knob": 64 + }, + { + "nq": 1869, + "recall": 0.954200107, + "qps_1t": 641.321552, + "qps_1t_min": 638.691566, + "qps_1t_max": 645.133053, + "qps_1t_runs": [ + 641.321552, + 645.133053, + 638.691566 + ], + "median_ms": 1.54650002, + "p95_ms": 1.812, + "p99_ms": 2.00480002, + "mean_ms": 1.55834967, + "knob": 128 + }, + { + "nq": 1016, + "recall": 0.981102362, + "qps_1t": 331.200059, + "qps_1t_min": 329.795016, + "qps_1t_max": 331.934005, + "qps_1t_runs": [ + 329.795016, + 331.200059, + 331.934005 + ], + "median_ms": 3.00600001, + "p95_ms": 3.38459999, + "p99_ms": 3.66450002, + "mean_ms": 3.02138556, + "knob": 256 + }, + { + "nq": 67, + "recall": 1, + "qps_1t": 22.478133, + "qps_1t_min": 22.3918544, + "qps_1t_max": 22.6263751, + "qps_1t_runs": [ + 22.3918544, + 22.478133, + 22.6263751 + ], + "median_ms": 44.5362, + "p95_ms": 46.7511, + "p99_ms": 48.0529999, + "mean_ms": 44.4476622, + "knob": 1265 + } + ], + "ref_knob": 128, + "ref_recall_pre": 0.954200107, + "mem_after_sweep": { + "rss_mb": 355.832031, + "peak_mb": 427.757812 + }, + "concurrent_clients": { + "clients": 8, + "qps": 648.631889 + }, + "mutation": { + "ref_knob": 128, + "recall_before": 0.954200107, + "insert_10k": { + "seconds": 0.03772, + "per_s": 265111.347 + }, + "update_10k": { + "seconds": 0.0130817, + "per_s": 764426.641 + }, + "delete_10k": { + "seconds": 0.00429279997, + "per_s": 2329481.94 + }, + "first_query_after_ms": 64.303, + "recall_after": 0.950125628, + "after": { + "nq": 1592, + "recall": 0.950125628, + "qps_1t": 640.604009, + "qps_1t_min": 635.628587, + "qps_1t_max": 644.542633, + "qps_1t_runs": [ + 644.542633, + 640.604009, + 635.628587 + ], + "median_ms": 1.54610001, + "p95_ms": 1.84119999, + "p99_ms": 2.0489, + "mean_ms": 1.5619201 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 100000, + "builds": 1 + }, + "mem_after": { + "rss_mb": 419.304688, + "peak_mb": 503.359375 + } + }, + "saveload": { + "save_s": 16.5449248, + "disk_mb": 499.571589, + "load_s": 7.3371524, + "first_query_after_load_ms": 902.5092, + "time_to_ready_s": 8.2396616, + "note": "load = JSON parse + insert; first query builds the IVF index (never persisted)" + } + } + ], + "load_before_pct": 6.487499999999999, + "load_contaminated": false, + "wall_s": 86.76119828224182, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/cpp_default__text384__100000__filter.json b/benchmarks/competitive/results/raw/cpp_default__text384__100000__filter.json new file mode 100644 index 0000000..25b704a --- /dev/null +++ b/benchmarks/competitive/results/raw/cpp_default__text384__100000__filter.json @@ -0,0 +1,237 @@ +{ + "status": "ok", + "label": "cpp_default", + "runs": [ + { + "system": "cpp_default_ivf", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "filter": { + "build_with_metadata_s": 0.9856867, + "mem": { + "rss_mb": 352.316406, + "peak_mb": 426.910156 + }, + "subsets": { + "f10": [ + { + "nq": 2000, + "recall": 0.30525, + "qps_1t": 15841.3207, + "qps_1t_min": 15522.1175, + "qps_1t_max": 16119.8283, + "qps_1t_runs": [ + 15522.1175, + 16119.8283, + 15841.3207 + ], + "median_ms": 0.0581999775, + "p95_ms": 0.108399952, + "p99_ms": 0.121500052, + "mean_ms": 0.0631952167, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.5293, + "qps_1t": 14227.4731, + "qps_1t_min": 13453.5724, + "qps_1t_max": 14282.4905, + "qps_1t_runs": [ + 14227.4731, + 14282.4905, + 13453.5724 + ], + "median_ms": 0.0703000114, + "p95_ms": 0.0849000062, + "p99_ms": 0.0939000165, + "mean_ms": 0.0715440168, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.7753, + "qps_1t": 9219.89967, + "qps_1t_min": 8565.50125, + "qps_1t_max": 9457.60628, + "qps_1t_runs": [ + 8565.50125, + 9457.60628, + 9219.89967 + ], + "median_ms": 0.108100008, + "p95_ms": 0.135900045, + "p99_ms": 0.151000044, + "mean_ms": 0.110314483, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.86635, + "qps_1t": 6586.23772, + "qps_1t_min": 6576.98726, + "qps_1t_max": 6670.57785, + "qps_1t_runs": [ + 6670.57785, + 6586.23772, + 6576.98726 + ], + "median_ms": 0.149299973, + "p95_ms": 0.179300027, + "p99_ms": 0.194899971, + "mean_ms": 0.151263033, + "knob": 64 + }, + { + "nq": 350, + "recall": 1, + "qps_1t": 112.936181, + "qps_1t_min": 111.427505, + "qps_1t_max": 118.53174, + "qps_1t_runs": [ + 112.936181, + 118.53174, + 111.427505 + ], + "median_ms": 8.48670001, + "p95_ms": 10.6079, + "p99_ms": 11.4348, + "mean_ms": 8.75518733, + "knob": 1265 + }, + { + "nq": 369, + "recall": 1, + "qps_1t": 116.163134, + "qps_1t_min": 114.132039, + "qps_1t_max": 118.580394, + "qps_1t_runs": [ + 118.580394, + 116.163134, + 114.132039 + ], + "median_ms": 8.39119998, + "p95_ms": 10.2264, + "p99_ms": 11.0627, + "mean_ms": 8.60115402, + "knob": "exact" + } + ], + "f01": [ + { + "nq": 2000, + "recall": 0.4965, + "qps_1t": 6217.71258, + "qps_1t_min": 6114.36242, + "qps_1t_max": 6251.55312, + "qps_1t_runs": [ + 6251.55312, + 6114.36242, + 6217.71258 + ], + "median_ms": 0.169800012, + "p95_ms": 0.203900039, + "p99_ms": 0.233199971, + "mean_ms": 0.161446817, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.49675, + "qps_1t": 8560.59942, + "qps_1t_min": 8502.15424, + "qps_1t_max": 8666.37115, + "qps_1t_runs": [ + 8502.15424, + 8560.59942, + 8666.37115 + ], + "median_ms": 0.128499989, + "p95_ms": 0.15659997, + "p99_ms": 0.177400012, + "mean_ms": 0.116606683, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.5704, + "qps_1t": 11771.478, + "qps_1t_min": 11758.1144, + "qps_1t_max": 11802.8708, + "qps_1t_runs": [ + 11758.1144, + 11771.478, + 11802.8708 + ], + "median_ms": 0.082899991, + "p95_ms": 0.101500016, + "p99_ms": 0.115200004, + "mean_ms": 0.0849079665, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.7197, + "qps_1t": 9267.99948, + "qps_1t_min": 9224.47105, + "qps_1t_max": 9271.76754, + "qps_1t_runs": [ + 9267.99948, + 9271.76754, + 9224.47105 + ], + "median_ms": 0.106099993, + "p95_ms": 0.128099986, + "p99_ms": 0.140500022, + "mean_ms": 0.10805325, + "knob": 64 + }, + { + "nq": 775, + "recall": 1, + "qps_1t": 320.493018, + "qps_1t_min": 306.2757, + "qps_1t_max": 332.351332, + "qps_1t_runs": [ + 306.2757, + 320.493018, + 332.351332 + ], + "median_ms": 2.84999999, + "p95_ms": 5.02549997, + "p99_ms": 6.95160002, + "mean_ms": 3.13136301, + "knob": 1265 + }, + { + "nq": 1084, + "recall": 1, + "qps_1t": 345.496103, + "qps_1t_min": 339.178934, + "qps_1t_max": 364.635055, + "qps_1t_runs": [ + 364.635055, + 339.178934, + 345.496103 + ], + "median_ms": 2.65839999, + "p95_ms": 4.19960002, + "p99_ms": 5.83909999, + "mean_ms": 2.86171768, + "knob": "exact" + } + ] + } + } + } + ], + "load_before_pct": 4.284374999999999, + "load_contaminated": false, + "wall_s": 43.15490365028381, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/cpp_default__text384__10000__core.json b/benchmarks/competitive/results/raw/cpp_default__text384__10000__core.json new file mode 100644 index 0000000..cffd2cf --- /dev/null +++ b/benchmarks/competitive/results/raw/cpp_default__text384__10000__core.json @@ -0,0 +1,285 @@ +{ + "status": "ok", + "label": "cpp_default", + "runs": [ + { + "system": "cpp_default_exact", + "ds": "text384", + "n": 10000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.00753950002, + "index_s": 0, + "total_s": 0.00753950002, + "first_query_ms": 4.08300001, + "ready_s": 0.0116225 + }, + "mem_baseline": { + "rss_mb": 11.0507812, + "peak_mb": 11.0507812 + }, + "mem_after_ingest": { + "rss_mb": 29.53125, + "peak_mb": 44.1601562 + }, + "mem_after_build": { + "rss_mb": 32.7460938, + "peak_mb": 44.1601562 + }, + "mem_after_first_query": { + "rss_mb": 32.7460938, + "peak_mb": 44.1601562 + }, + "sweep": [ + { + "nq": 952, + "recall": 1, + "qps_1t": 310.7407, + "qps_1t_min": 288.55097, + "qps_1t_max": 311.217139, + "qps_1t_runs": [ + 310.7407, + 311.217139, + 288.55097 + ], + "median_ms": 3.22829996, + "p95_ms": 3.62990005, + "p99_ms": 4.30209999, + "mean_ms": 3.2989667, + "knob": null + } + ], + "mem_after_sweep": { + "rss_mb": 32.6875, + "peak_mb": 44.1601562 + } + }, + { + "system": "cpp_default_ivf", + "ds": "text384", + "n": 10000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "knob_name": "nprobe", + "build": { + "ingest_s": 0.00753950002, + "index_s": 0.0880093, + "total_s": 0.0955488, + "first_query_ms": 0, + "ready_s": 0.0955488 + }, + "mem_baseline": { + "rss_mb": 11.0507812, + "peak_mb": 11.0507812 + }, + "mem_after_ingest": { + "rss_mb": 29.53125, + "peak_mb": 44.1601562 + }, + "mem_after_build": { + "rss_mb": 47.7226562, + "peak_mb": 62.859375 + }, + "index_stats": { + "nlist": 400, + "trained_n": 10000, + "index_bytes": 796004, + "vector_bytes": 15360000, + "build_seconds": 0.0829413 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.2963, + "qps_1t": 44365.6711, + "qps_1t_min": 43354.9819, + "qps_1t_max": 46078.5966, + "qps_1t_runs": [ + 43354.9819, + 44365.6711, + 46078.5966 + ], + "median_ms": 0.0211999868, + "p95_ms": 0.0316000078, + "p99_ms": 0.0389000052, + "mean_ms": 0.0224357999, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.41875, + "qps_1t": 44228.6086, + "qps_1t_min": 44186.2008, + "qps_1t_max": 45433.7902, + "qps_1t_runs": [ + 44228.6086, + 45433.7902, + 44186.2008 + ], + "median_ms": 0.0219999929, + "p95_ms": 0.0261999667, + "p99_ms": 0.0290000462, + "mean_ms": 0.0224171167, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.55125, + "qps_1t": 39332.3721, + "qps_1t_min": 39285.3985, + "qps_1t_max": 39407.7024, + "qps_1t_runs": [ + 39407.7024, + 39332.3721, + 39285.3985 + ], + "median_ms": 0.024900015, + "p95_ms": 0.0298999948, + "p99_ms": 0.0334000215, + "mean_ms": 0.0254182833, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.6798, + "qps_1t": 31354.7937, + "qps_1t_min": 31278.5415, + "qps_1t_max": 31627.7039, + "qps_1t_runs": [ + 31278.5415, + 31354.7937, + 31627.7039 + ], + "median_ms": 0.0311999465, + "p95_ms": 0.0381999998, + "p99_ms": 0.0419999706, + "mean_ms": 0.0318272334, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.8005, + "qps_1t": 23507.1768, + "qps_1t_min": 23091.5685, + "qps_1t_max": 23638.2031, + "qps_1t_runs": [ + 23091.5685, + 23638.2031, + 23507.1768 + ], + "median_ms": 0.0418000272, + "p95_ms": 0.0513000414, + "p99_ms": 0.0569000258, + "mean_ms": 0.0427168166, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.8922, + "qps_1t": 15302.2189, + "qps_1t_min": 14886.9005, + "qps_1t_max": 15408.7122, + "qps_1t_runs": [ + 14886.9005, + 15408.7122, + 15302.2189 + ], + "median_ms": 0.0648999703, + "p95_ms": 0.0795000233, + "p99_ms": 0.086999964, + "mean_ms": 0.0658071666, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.9563, + "qps_1t": 9372.79592, + "qps_1t_min": 9323.58336, + "qps_1t_max": 9429.51883, + "qps_1t_runs": [ + 9429.51883, + 9372.79592, + 9323.58336 + ], + "median_ms": 0.105899991, + "p95_ms": 0.123400008, + "p99_ms": 0.138600008, + "mean_ms": 0.106665533, + "knob": 64 + }, + { + "nq": 2000, + "recall": 0.9874, + "qps_1t": 4953.5984, + "qps_1t_min": 4682.45465, + "qps_1t_max": 5307.43306, + "qps_1t_runs": [ + 4682.45465, + 4953.5984, + 5307.43306 + ], + "median_ms": 0.192300009, + "p95_ms": 0.269600016, + "p99_ms": 0.330599956, + "mean_ms": 0.201283883, + "knob": 128 + }, + { + "nq": 2000, + "recall": 0.99845, + "qps_1t": 2772.07462, + "qps_1t_min": 2745.69773, + "qps_1t_max": 2842.35288, + "qps_1t_runs": [ + 2842.35288, + 2745.69773, + 2772.07462 + ], + "median_ms": 0.349299982, + "p95_ms": 0.420299999, + "p99_ms": 0.473699998, + "mean_ms": 0.358922667, + "knob": 256 + }, + { + "nq": 875, + "recall": 1, + "qps_1t": 311.48558, + "qps_1t_min": 308.851014, + "qps_1t_max": 314.30099, + "qps_1t_runs": [ + 308.851014, + 314.30099, + 311.48558 + ], + "median_ms": 3.17690003, + "p95_ms": 3.5012, + "p99_ms": 3.66749999, + "mean_ms": 3.20996404, + "knob": 400 + } + ], + "ref_knob": 64, + "ref_recall_pre": 0.9563, + "mem_after_sweep": { + "rss_mb": 49.21875, + "peak_mb": 62.859375 + }, + "concurrent_clients": { + "clients": 8, + "qps": 7640.63058 + } + } + ], + "load_before_pct": 4.415624999999998, + "load_contaminated": false, + "wall_s": 23.74683690071106, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/csharp__glove__1000000__core+mutation.json b/benchmarks/competitive/results/raw/csharp__glove__1000000__core+mutation.json new file mode 100644 index 0000000..342ecc7 --- /dev/null +++ b/benchmarks/competitive/results/raw/csharp__glove__1000000__core+mutation.json @@ -0,0 +1,305 @@ +{ + "status": "ok", + "label": "csharp", + "runs": [ + { + "system": "csharp_exact", + "ds": "glove", + "n": 1000000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-csharp (repo HEAD, .NET 8 Release)", + "build": { + "ingest_s": 0.19757949997438118, + "index_s": 0, + "total_s": 0.19757949997438118, + "first_query_ms": 122.77479999465868, + "ready_s": 0.32035429996903986 + }, + "mem_baseline": { + "rss_mb": 27.96484375, + "peak_mb": 27.96484375 + }, + "mem_after_ingest": { + "rss_mb": 789.078125, + "peak_mb": 789.078125 + }, + "mem_after_build": { + "rss_mb": 838.33203125, + "peak_mb": 838.33203125 + }, + "mem_after_first_query": { + "rss_mb": 838.33984375, + "peak_mb": 838.33984375 + }, + "sweep": [ + { + "nq": 30, + "recall": 1, + "qps_1t": 5.808417818488097, + "qps_1t_min": 5.808417818488097, + "qps_1t_max": 5.808417818488097, + "qps_1t_runs": [ + 5.808417818488097 + ], + "median_ms": 126.38750002952293, + "p95_ms": 369.5667000138201, + "p99_ms": 399.25610000500455, + "mean_ms": 172.1639233350288, + "knob": null + } + ], + "mem_after_sweep": { + "rss_mb": 1614.75390625, + "peak_mb": 2000.53125 + } + }, + { + "system": "csharp_ivf", + "ds": "glove", + "n": 1000000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-csharp (repo HEAD, .NET 8 Release)", + "knob_name": "nprobe", + "build": { + "ingest_s": 0.33852740010479465, + "index_s": 11.303626800014172, + "total_s": 11.642154200118966, + "first_query_ms": 0, + "ready_s": 11.642154200118966 + }, + "mem_baseline": { + "rss_mb": 27.96484375, + "peak_mb": 27.96484375 + }, + "mem_after_ingest": { + "rss_mb": 1649.703125, + "peak_mb": 2000.53125 + }, + "mem_after_build": { + "rss_mb": 2101.2890625, + "peak_mb": 2101.2890625 + }, + "index_stats": { + "nlist": 4000, + "trained_n": 1000000, + "index_bytes": 10680132, + "vector_bytes": 472506720, + "build_seconds": 10.5247032 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.2969, + "qps_1t": 2482.6610944296576, + "qps_1t_min": 2138.7119363346815, + "qps_1t_max": 2664.7097044690763, + "qps_1t_runs": [ + 2138.7119363346815, + 2482.6610944296576, + 2664.7097044690763 + ], + "median_ms": 0.41089998558163643, + "p95_ms": 0.5002000252716243, + "p99_ms": 0.9454999817535281, + "mean_ms": 0.41521338339468156, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.41435, + "qps_1t": 2367.0765230664338, + "qps_1t_min": 2352.7452077845332, + "qps_1t_max": 2402.815138448941, + "qps_1t_runs": [ + 2352.7452077845332, + 2367.0765230664338, + 2402.815138448941 + ], + "median_ms": 0.43129996629431844, + "p95_ms": 0.5011000321246684, + "p99_ms": 0.539699976798147, + "mean_ms": 0.42122531652178924, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.53055, + "qps_1t": 2035.6451631667, + "qps_1t_min": 2020.5320398783858, + "qps_1t_max": 2060.001244889335, + "qps_1t_runs": [ + 2035.6451631667, + 2060.001244889335, + 2020.5320398783858 + ], + "median_ms": 0.4993999609723687, + "p95_ms": 0.6009999779053032, + "p99_ms": 0.6523000192828476, + "mean_ms": 0.4905335001143006, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.633, + "qps_1t": 1558.0563745986215, + "qps_1t_min": 1547.043583584809, + "qps_1t_max": 1574.5244464732775, + "qps_1t_runs": [ + 1547.043583584809, + 1574.5244464732775, + 1558.0563745986215 + ], + "median_ms": 0.6372000207193196, + "p95_ms": 0.8135999669320881, + "p99_ms": 0.8918999810703099, + "mean_ms": 0.6411106334029075, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.71885, + "qps_1t": 1107.506485028945, + "qps_1t_min": 1092.3245849112725, + "qps_1t_max": 1115.1629317074523, + "qps_1t_runs": [ + 1107.506485028945, + 1092.3245849112725, + 1115.1629317074523 + ], + "median_ms": 0.8931999909691513, + "p95_ms": 1.194199954625219, + "p99_ms": 1.323700009379536, + "mean_ms": 0.9050459999707527, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.7955, + "qps_1t": 713.1717656520507, + "qps_1t_min": 706.7175018085878, + "qps_1t_max": 718.5526596060247, + "qps_1t_runs": [ + 713.1717656520507, + 706.7175018085878, + 718.5526596060247 + ], + "median_ms": 1.3818999868817627, + "p95_ms": 1.8756999634206295, + "p99_ms": 2.0440000225789845, + "mean_ms": 1.4029552498735334, + "knob": 32 + }, + { + "nq": 1483, + "recall": 0.8602832097100472, + "qps_1t": 422.7358574928612, + "qps_1t_min": 417.2103342087863, + "qps_1t_max": 425.44809616995457, + "qps_1t_runs": [ + 425.44809616995457, + 422.7358574928612, + 417.2103342087863 + ], + "median_ms": 2.3467999999411404, + "p95_ms": 3.132499987259507, + "p99_ms": 3.4493000130169094, + "mean_ms": 2.370959631344029, + "knob": 64 + }, + { + "nq": 674, + "recall": 0.9160237388724035, + "qps_1t": 248.45892797274553, + "qps_1t_min": 246.3038885978749, + "qps_1t_max": 273.0388920877229, + "qps_1t_runs": [ + 246.3038885978749, + 248.45892797274553, + 273.0388920877229 + ], + "median_ms": 3.863200021442026, + "p95_ms": 5.196599988266826, + "p99_ms": 5.62569999601692, + "mean_ms": 3.915772403352715, + "knob": 128 + }, + { + "nq": 336, + "recall": 0.9485119047619047, + "qps_1t": 131.04159778558738, + "qps_1t_min": 128.50672770607466, + "qps_1t_max": 144.59467039493364, + "qps_1t_runs": [ + 144.59467039493364, + 131.04159778558738, + 128.50672770607466 + ], + "median_ms": 7.374399981927127, + "p95_ms": 9.607499989215285, + "p99_ms": 10.472499998286366, + "mean_ms": 7.442914186651164, + "knob": 256 + } + ], + "ref_knob": 256, + "ref_recall_pre": 0, + "mem_after_sweep": { + "rss_mb": 2127.41015625, + "peak_mb": 2127.41015625 + }, + "mutation": { + "ref_knob": 256, + "recall_before": 0, + "insert_10k": { + "seconds": 0.028511700045783073, + "per_s": 350733.20720765006 + }, + "update_10k": { + "seconds": 0.016209499968681484, + "per_s": 616922.1764595506 + }, + "delete_10k": { + "seconds": 0.038698900025337934, + "per_s": 258405.2775002014 + }, + "first_query_after_ms": 135.6134000234306, + "recall_after": 0.9493297587131367, + "after": { + "nq": 373, + "recall": 0.9493297587131367, + "qps_1t": 127.7292802347123, + "qps_1t_min": 127.68586185220663, + "qps_1t_max": 128.30734099394698, + "qps_1t_runs": [ + 127.68586185220663, + 127.7292802347123, + 128.30734099394698 + ], + "median_ms": 7.819599995855242, + "p95_ms": 9.831800009123981, + "p99_ms": 10.671299998648465, + "mean_ms": 7.818188471469938 + }, + "mem_after": { + "rss_mb": 2136.64453125, + "peak_mb": 2136.64453125 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 1000000 + } + } + } + ], + "load_before_pct": 5.187500000000002, + "load_contaminated": false, + "wall_s": 85.68259334564209, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/csharp__glove__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/csharp__glove__100000__core+mutation+saveload.json new file mode 100644 index 0000000..dccc4ae --- /dev/null +++ b/benchmarks/competitive/results/raw/csharp__glove__100000__core+mutation+saveload.json @@ -0,0 +1,331 @@ +{ + "status": "ok", + "label": "csharp", + "runs": [ + { + "system": "csharp_exact", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-csharp (repo HEAD, .NET 8 Release)", + "build": { + "ingest_s": 0.017087700020056218, + "index_s": 0, + "total_s": 0.017087700020056218, + "first_query_ms": 18.324600008782, + "ready_s": 0.03541230002883822 + }, + "mem_baseline": { + "rss_mb": 27.90234375, + "peak_mb": 27.90234375 + }, + "mem_after_ingest": { + "rss_mb": 109.70703125, + "peak_mb": 112.12109375 + }, + "mem_after_build": { + "rss_mb": 115.22265625, + "peak_mb": 115.22265625 + }, + "mem_after_first_query": { + "rss_mb": 115.23046875, + "peak_mb": 115.23046875 + }, + "sweep": [ + { + "nq": 387, + "recall": 1, + "qps_1t": 107.91965457129174, + "qps_1t_min": 107.40336680013168, + "qps_1t_max": 108.34074222690144, + "qps_1t_runs": [ + 107.40336680013168, + 107.91965457129174, + 108.34074222690144 + ], + "median_ms": 7.786600035615265, + "p95_ms": 26.426199998240918, + "p99_ms": 35.582799988333136, + "mean_ms": 9.26899526248791, + "knob": null + } + ], + "mem_after_sweep": { + "rss_mb": 219.5234375, + "peak_mb": 235.2109375 + } + }, + { + "system": "csharp_ivf", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-csharp (repo HEAD, .NET 8 Release)", + "knob_name": "nprobe", + "build": { + "ingest_s": 0.011220699991099536, + "index_s": 0.6018785000196658, + "total_s": 0.6130992000107653, + "first_query_ms": 0, + "ready_s": 0.6130992000107653 + }, + "mem_baseline": { + "rss_mb": 27.90234375, + "peak_mb": 27.90234375 + }, + "mem_after_ingest": { + "rss_mb": 241.8984375, + "peak_mb": 244.04296875 + }, + "mem_after_build": { + "rss_mb": 277.21875, + "peak_mb": 277.21875 + }, + "index_stats": { + "nlist": 1265, + "trained_n": 100000, + "index_bytes": 1431432, + "vector_bytes": 47256720, + "build_seconds": 0.5615482 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.28675, + "qps_1t": 14066.11213269189, + "qps_1t_min": 6337.005098092103, + "qps_1t_max": 14394.164016536084, + "qps_1t_runs": [ + 6337.005098092103, + 14394.164016536084, + 14066.11213269189 + ], + "median_ms": 0.0686999992467463, + "p95_ms": 0.21190004190430045, + "p99_ms": 0.23200002033263445, + "mean_ms": 0.09945623337019546, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.4092, + "qps_1t": 13475.581571006189, + "qps_1t_min": 13083.345468361927, + "qps_1t_max": 13665.057155795492, + "qps_1t_runs": [ + 13665.057155795492, + 13475.581571006189, + 13083.345468361927 + ], + "median_ms": 0.07269997149705887, + "p95_ms": 0.08359999628737569, + "p99_ms": 0.09129999671131372, + "mean_ms": 0.07460690003548127, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.52735, + "qps_1t": 12150.188474798064, + "qps_1t_min": 11931.764631294978, + "qps_1t_max": 12194.720437087037, + "qps_1t_runs": [ + 12194.720437087037, + 11931.764631294978, + 12150.188474798064 + ], + "median_ms": 0.08149998029693961, + "p95_ms": 0.096399977337569, + "p99_ms": 0.1058999914675951, + "mean_ms": 0.08270528328527386, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.6336, + "qps_1t": 9682.402673244265, + "qps_1t_min": 9671.914175416765, + "qps_1t_max": 9773.47527511135, + "qps_1t_runs": [ + 9682.402673244265, + 9671.914175416765, + 9773.47527511135 + ], + "median_ms": 0.10040000779554248, + "p95_ms": 0.12330000754445791, + "p99_ms": 0.13899995246902108, + "mean_ms": 0.10299668328176873, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.7249, + "qps_1t": 7268.405875377789, + "qps_1t_min": 7000.624444767023, + "qps_1t_max": 7361.59376170538, + "qps_1t_runs": [ + 7000.624444767023, + 7268.405875377789, + 7361.59376170538 + ], + "median_ms": 0.13639999087899923, + "p95_ms": 0.17499999376013875, + "p99_ms": 0.1947000273503363, + "mean_ms": 0.13875543336810855, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.8096, + "qps_1t": 4611.594239951243, + "qps_1t_min": 4435.917624805918, + "qps_1t_max": 4694.782781633785, + "qps_1t_runs": [ + 4611.594239951243, + 4694.782781633785, + 4435.917624805918 + ], + "median_ms": 0.2148000057786703, + "p95_ms": 0.27600000612437725, + "p99_ms": 0.30890002381056547, + "mean_ms": 0.21842655000121644, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.878, + "qps_1t": 2982.8050219600977, + "qps_1t_min": 2918.132237454387, + "qps_1t_max": 2983.527200916418, + "qps_1t_runs": [ + 2918.132237454387, + 2982.8050219600977, + 2983.527200916418 + ], + "median_ms": 0.33359997905790806, + "p95_ms": 0.42259995825588703, + "p99_ms": 0.4817000008188188, + "mean_ms": 0.33770453340063494, + "knob": 64 + }, + { + "nq": 2000, + "recall": 0.9323, + "qps_1t": 1677.137173299409, + "qps_1t_min": 1631.7020117415975, + "qps_1t_max": 1699.974406339156, + "qps_1t_runs": [ + 1677.137173299409, + 1699.974406339156, + 1631.7020117415975 + ], + "median_ms": 0.5938000394962728, + "p95_ms": 0.747899990528822, + "p99_ms": 0.8287999662570655, + "mean_ms": 0.5991184334949745, + "knob": 128 + }, + { + "nq": 2000, + "recall": 0.97265, + "qps_1t": 886.2994766147127, + "qps_1t_min": 825.7559218151433, + "qps_1t_max": 886.3834571301405, + "qps_1t_runs": [ + 886.2994766147127, + 886.3834571301405, + 825.7559218151433 + ], + "median_ms": 1.1380999931134284, + "p95_ms": 1.4329000259749591, + "p99_ms": 1.62390002515167, + "mean_ms": 1.1558260998378196, + "knob": 256 + }, + { + "nq": 329, + "recall": 1, + "qps_1t": 98.65570062707044, + "qps_1t_min": 95.54740662194229, + "qps_1t_max": 100.95501603998943, + "qps_1t_runs": [ + 95.54740662194229, + 98.65570062707044, + 100.95501603998943 + ], + "median_ms": 7.89549999171868, + "p95_ms": 38.67310000350699, + "p99_ms": 49.266899994108826, + "mean_ms": 10.169224113909095, + "knob": 1265 + } + ], + "ref_knob": 256, + "ref_recall_pre": 0.97265, + "mem_after_sweep": { + "rss_mb": 384.39453125, + "peak_mb": 821.53515625 + }, + "mutation": { + "ref_knob": 256, + "recall_before": 0.97265, + "insert_10k": { + "seconds": 0.04945140000199899, + "per_s": 202218.74405165005 + }, + "update_10k": { + "seconds": 0.007727499993052334, + "per_s": 1294079.587058018 + }, + "delete_10k": { + "seconds": 0.0034479000023566186, + "per_s": 2900316.1324763075 + }, + "first_query_after_ms": 33.11979997670278, + "recall_after": 0.9723, + "after": { + "nq": 2000, + "recall": 0.9723, + "qps_1t": 862.0308059308751, + "qps_1t_min": 856.138545681082, + "qps_1t_max": 870.4372438566799, + "qps_1t_runs": [ + 870.4372438566799, + 856.138545681082, + 862.0308059308751 + ], + "median_ms": 1.1377999908290803, + "p95_ms": 1.4328999677672982, + "p99_ms": 1.6453000134788454, + "mean_ms": 1.1589781666310348 + }, + "mem_after": { + "rss_mb": 371.09765625, + "peak_mb": 821.53515625 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 100000 + } + }, + "saveload": { + "save_s": 1.3590893999789841, + "disk_mb": 121.46593761444092, + "load_s": 1.273968099965714, + "first_query_after_load_ms": 332.74230000097305, + "time_to_ready_s": 1.606710399966687 + } + } + ], + "load_before_pct": 4.287499999999999, + "load_contaminated": false, + "wall_s": 50.09799122810364, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/csharp__sift__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/csharp__sift__100000__core+mutation+saveload.json new file mode 100644 index 0000000..7fbb564 --- /dev/null +++ b/benchmarks/competitive/results/raw/csharp__sift__100000__core+mutation+saveload.json @@ -0,0 +1,330 @@ +{ + "status": "ok", + "label": "csharp", + "runs": [ + { + "system": "csharp_exact", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-csharp (repo HEAD, .NET 8 Release)", + "build": { + "ingest_s": 0.017743100004736334, + "index_s": 0, + "total_s": 0.017743100004736334, + "first_query_ms": 22.882099961861968, + "ready_s": 0.0406251999665983 + }, + "mem_baseline": { + "rss_mb": 28.546875, + "peak_mb": 28.546875 + }, + "mem_after_ingest": { + "rss_mb": 130.45703125, + "peak_mb": 135.4296875 + }, + "mem_after_build": { + "rss_mb": 136.390625, + "peak_mb": 136.390625 + }, + "mem_after_first_query": { + "rss_mb": 136.3984375, + "peak_mb": 136.3984375 + }, + "sweep": [ + { + "nq": 314, + "recall": 1, + "qps_1t": 54.95953245815263, + "qps_1t_min": 54.95953245815263, + "qps_1t_max": 72.93859954981716, + "qps_1t_runs": [ + 72.93859954981716, + 54.95953245815263 + ], + "median_ms": 14.593200001399964, + "p95_ms": 42.7979999803938, + "p99_ms": 80.84619999863207, + "mean_ms": 15.952683757037033, + "knob": null + } + ], + "mem_after_sweep": { + "rss_mb": 281.0546875, + "peak_mb": 287.01953125 + } + }, + { + "system": "csharp_ivf", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-csharp (repo HEAD, .NET 8 Release)", + "knob_name": "nprobe", + "build": { + "ingest_s": 0.02837600006023422, + "index_s": 1.4822395000373945, + "total_s": 1.5106155000976287, + "first_query_ms": 0, + "ready_s": 1.5106155000976287 + }, + "mem_baseline": { + "rss_mb": 28.546875, + "peak_mb": 28.546875 + }, + "mem_after_ingest": { + "rss_mb": 282.859375, + "peak_mb": 302.578125 + }, + "mem_after_build": { + "rss_mb": 339.109375, + "peak_mb": 339.109375 + }, + "index_stats": { + "nlist": 1265, + "trained_n": 100000, + "index_bytes": 1552872, + "vector_bytes": 58058256, + "build_seconds": 1.3865745 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.285, + "qps_1t": 8528.420952993927, + "qps_1t_min": 5455.372193484733, + "qps_1t_max": 9394.955467120839, + "qps_1t_runs": [ + 5455.372193484733, + 9394.955467120839, + 8528.420952993927 + ], + "median_ms": 0.11610001092776656, + "p95_ms": 0.29450003057718277, + "p99_ms": 0.32409996492788196, + "mean_ms": 0.13566688337596133, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.43015, + "qps_1t": 7905.097725173819, + "qps_1t_min": 7739.644545359886, + "qps_1t_max": 8000.614444416882, + "qps_1t_runs": [ + 7905.097725173819, + 7739.644545359886, + 8000.614444416882 + ], + "median_ms": 0.12390001211315393, + "p95_ms": 0.14220003504306078, + "p99_ms": 0.15599996550008655, + "mean_ms": 0.12689865001205664, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.5908, + "qps_1t": 6901.389843260633, + "qps_1t_min": 6827.7701788934755, + "qps_1t_max": 7039.360938641796, + "qps_1t_runs": [ + 6901.389843260633, + 7039.360938641796, + 6827.7701788934755 + ], + "median_ms": 0.14349998673424125, + "p95_ms": 0.16960001084953547, + "p99_ms": 0.18700002692639828, + "mean_ms": 0.144472466626515, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.7513, + "qps_1t": 5823.219857140829, + "qps_1t_min": 5638.512164968473, + "qps_1t_max": 5948.9496826447075, + "qps_1t_runs": [ + 5823.219857140829, + 5638.512164968473, + 5948.9496826447075 + ], + "median_ms": 0.17209997167810798, + "p95_ms": 0.21239998750388622, + "p99_ms": 0.23639999562874436, + "mean_ms": 0.1723916500050109, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.8749, + "qps_1t": 4284.10835744266, + "qps_1t_min": 4242.892892065133, + "qps_1t_max": 4379.495554297895, + "qps_1t_runs": [ + 4379.495554297895, + 4284.10835744266, + 4242.892892065133 + ], + "median_ms": 0.23070001043379307, + "p95_ms": 0.2940999693237245, + "p99_ms": 0.3305000136606395, + "mean_ms": 0.2324819499044679, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.95255, + "qps_1t": 2805.371331108233, + "qps_1t_min": 2527.946767907984, + "qps_1t_max": 2838.346919403495, + "qps_1t_runs": [ + 2838.346919403495, + 2805.371331108233, + 2527.946767907984 + ], + "median_ms": 0.36180001916363835, + "p95_ms": 0.48160000005736947, + "p99_ms": 0.5549000343307853, + "mean_ms": 0.3681182499858551, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.98895, + "qps_1t": 1726.807485746021, + "qps_1t_min": 1719.4581886181063, + "qps_1t_max": 1855.0279008457028, + "qps_1t_runs": [ + 1855.0279008457028, + 1726.807485746021, + 1719.4581886181063 + ], + "median_ms": 0.5561999860219657, + "p95_ms": 0.7480999920517206, + "p99_ms": 0.8698999881744385, + "mean_ms": 0.566585783274301, + "knob": 64 + }, + { + "nq": 2000, + "recall": 0.9984, + "qps_1t": 1055.6210977180679, + "qps_1t_min": 1028.940071454162, + "qps_1t_max": 1102.7663556178188, + "qps_1t_runs": [ + 1028.940071454162, + 1102.7663556178188, + 1055.6210977180679 + ], + "median_ms": 0.9150999831035733, + "p95_ms": 1.2823999859392643, + "p99_ms": 1.7554000369273126, + "mean_ms": 0.9419979666708969, + "knob": 128 + }, + { + "nq": 1547, + "recall": 0.9999353587588882, + "qps_1t": 621.7537531149608, + "qps_1t_min": 596.8806329730581, + "qps_1t_max": 651.9730127543297, + "qps_1t_runs": [ + 621.7537531149608, + 651.9730127543297, + 596.8806329730581 + ], + "median_ms": 1.5802000416442752, + "p95_ms": 2.12459999602288, + "p99_ms": 2.443400037009269, + "mean_ms": 1.6058454860645561, + "knob": 256 + }, + { + "nq": 198, + "recall": 1, + "qps_1t": 53.60233838210898, + "qps_1t_min": 52.992020471916135, + "qps_1t_max": 53.6696923184099, + "qps_1t_runs": [ + 53.60233838210898, + 53.6696923184099, + 52.992020471916135 + ], + "median_ms": 15.355499985162169, + "p95_ms": 40.92530003981665, + "p99_ms": 87.66459999606013, + "mean_ms": 18.71971936039203, + "knob": 1265 + } + ], + "ref_knob": 32, + "ref_recall_pre": 0.95255, + "mem_after_sweep": { + "rss_mb": 471.0390625, + "peak_mb": 1027.79296875 + }, + "mutation": { + "ref_knob": 32, + "recall_before": 0.95255, + "insert_10k": { + "seconds": 0.030085799982771277, + "per_s": 332382.7189480263 + }, + "update_10k": { + "seconds": 0.01525609998498112, + "per_s": 655475.5153574314 + }, + "delete_10k": { + "seconds": 0.00854740000795573, + "per_s": 1169946.4153651663 + }, + "first_query_after_ms": 51.75270000472665, + "recall_after": 0.9487, + "after": { + "nq": 2000, + "recall": 0.9487, + "qps_1t": 2633.470889560701, + "qps_1t_min": 2580.2196747197495, + "qps_1t_max": 2896.9888688641504, + "qps_1t_runs": [ + 2633.470889560701, + 2580.2196747197495, + 2896.9888688641504 + ], + "median_ms": 0.35989994648844004, + "p95_ms": 0.485299970023334, + "p99_ms": 0.597700010985136, + "mean_ms": 0.37082563330962637 + }, + "mem_after": { + "rss_mb": 479.72265625, + "peak_mb": 1027.79296875 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 100000 + } + }, + "saveload": { + "save_s": 3.507927500002552, + "disk_mb": 121.89836692810059, + "load_s": 3.0640739999944344, + "first_query_after_load_ms": 1132.6488999766298, + "time_to_ready_s": 4.196722899971064 + } + } + ], + "load_before_pct": 9.890625, + "load_contaminated": false, + "wall_s": 58.76704430580139, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/csharp__text384__1000000__core+mutation.json b/benchmarks/competitive/results/raw/csharp__text384__1000000__core+mutation.json new file mode 100644 index 0000000..b4bc559 --- /dev/null +++ b/benchmarks/competitive/results/raw/csharp__text384__1000000__core+mutation.json @@ -0,0 +1,305 @@ +{ + "status": "ok", + "label": "csharp", + "runs": [ + { + "system": "csharp_exact", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-csharp (repo HEAD, .NET 8 Release)", + "build": { + "ingest_s": 0.21663540008012205, + "index_s": 0, + "total_s": 0.21663540008012205, + "first_query_ms": 298.69659995893016, + "ready_s": 0.5153320000390522 + }, + "mem_baseline": { + "rss_mb": 34.35546875, + "peak_mb": 34.35546875 + }, + "mem_after_ingest": { + "rss_mb": 2432.55859375, + "peak_mb": 2432.55859375 + }, + "mem_after_build": { + "rss_mb": 2480.88671875, + "peak_mb": 2480.88671875 + }, + "mem_after_first_query": { + "rss_mb": 2480.890625, + "peak_mb": 2480.89453125 + }, + "sweep": [ + { + "nq": 30, + "recall": 1, + "qps_1t": 3.717775299591297, + "qps_1t_min": 3.717775299591297, + "qps_1t_max": 3.717775299591297, + "qps_1t_runs": [ + 3.717775299591297 + ], + "median_ms": 245.34419999690726, + "p95_ms": 394.20649997191504, + "p99_ms": 397.59900001809, + "mean_ms": 268.97806333533174, + "knob": null + } + ], + "mem_after_sweep": { + "rss_mb": 3668.90625, + "peak_mb": 3668.90625 + } + }, + { + "system": "csharp_ivf", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-csharp (repo HEAD, .NET 8 Release)", + "knob_name": "nprobe", + "build": { + "ingest_s": 0.17699659999925643, + "index_s": 10.716460700030439, + "total_s": 10.893457300029695, + "first_query_ms": 0, + "ready_s": 10.893457300029695 + }, + "mem_baseline": { + "rss_mb": 34.35546875, + "peak_mb": 34.35546875 + }, + "mem_after_ingest": { + "rss_mb": 4968.04296875, + "peak_mb": 4968.04296875 + }, + "mem_after_build": { + "rss_mb": 6689.90625, + "peak_mb": 6698.26171875 + }, + "index_stats": { + "nlist": 4000, + "trained_n": 1000000, + "index_bytes": 15160132, + "vector_bytes": 1732524640, + "build_seconds": 10.2765374 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.27405, + "qps_1t": 2528.7109857144565, + "qps_1t_min": 2179.0743696895534, + "qps_1t_max": 2619.7744516893154, + "qps_1t_runs": [ + 2179.0743696895534, + 2528.7109857144565, + 2619.7744516893154 + ], + "median_ms": 0.3837000112980604, + "p95_ms": 0.6302999681793153, + "p99_ms": 0.7325999904423952, + "mean_ms": 0.4120270332787186, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.3823, + "qps_1t": 2128.6749987375856, + "qps_1t_min": 2065.8463738630026, + "qps_1t_max": 2179.2206301267274, + "qps_1t_runs": [ + 2128.6749987375856, + 2179.2206301267274, + 2065.8463738630026 + ], + "median_ms": 0.45469996985048056, + "p95_ms": 0.6121999467723072, + "p99_ms": 0.7542999810539186, + "mean_ms": 0.470906183278809, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.50015, + "qps_1t": 1684.8468663471908, + "qps_1t_min": 1600.9149550235593, + "qps_1t_max": 1685.3750328611393, + "qps_1t_runs": [ + 1685.3750328611393, + 1600.9149550235593, + 1684.8468663471908 + ], + "median_ms": 0.5908000166527927, + "p95_ms": 0.7831000257283449, + "p99_ms": 0.9157000458799303, + "mean_ms": 0.6038360998597152, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.6082, + "qps_1t": 1170.855166345593, + "qps_1t_min": 1136.27686613993, + "qps_1t_max": 1174.9780894480484, + "qps_1t_runs": [ + 1170.855166345593, + 1136.27686613993, + 1174.9780894480484 + ], + "median_ms": 0.8402999956160784, + "p95_ms": 1.1089000036008656, + "p99_ms": 1.2582999770529568, + "mean_ms": 0.8617411668237764, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.70535, + "qps_1t": 746.7836868029974, + "qps_1t_min": 734.0894822449857, + "qps_1t_max": 756.2227775822607, + "qps_1t_runs": [ + 746.7836868029974, + 756.2227775822607, + 734.0894822449857 + ], + "median_ms": 1.317800022661686, + "p95_ms": 1.7407999839633703, + "p99_ms": 2.0183000015094876, + "mean_ms": 1.3412230832618661, + "knob": 16 + }, + { + "nq": 1183, + "recall": 0.7822485207100591, + "qps_1t": 444.68901292113406, + "qps_1t_min": 438.26828241652515, + "qps_1t_max": 445.86765300693673, + "qps_1t_runs": [ + 438.26828241652515, + 444.68901292113406, + 445.86765300693673 + ], + "median_ms": 2.233400009572506, + "p95_ms": 2.8313000220805407, + "p99_ms": 3.177399979904294, + "mean_ms": 2.2577626654642344, + "knob": 32 + }, + { + "nq": 857, + "recall": 0.8549591598599766, + "qps_1t": 252.84508998112273, + "qps_1t_min": 250.30883350087203, + "qps_1t_max": 257.58553330033277, + "qps_1t_runs": [ + 257.58553330033277, + 252.84508998112273, + 250.30883350087203 + ], + "median_ms": 3.882699995301664, + "p95_ms": 4.866900038905442, + "p99_ms": 5.616399983409792, + "mean_ms": 3.9440870475392025, + "knob": 64 + }, + { + "nq": 426, + "recall": 0.9096244131455399, + "qps_1t": 141.02315610854967, + "qps_1t_min": 140.94946606200546, + "qps_1t_max": 145.00826903551325, + "qps_1t_runs": [ + 140.94946606200546, + 141.02315610854967, + 145.00826903551325 + ], + "median_ms": 6.930499977897853, + "p95_ms": 8.40059999609366, + "p99_ms": 9.481300017796457, + "mean_ms": 7.027311267328328, + "knob": 128 + }, + { + "nq": 234, + "recall": 0.9461538461538461, + "qps_1t": 74.5134605231173, + "qps_1t_min": 72.82908632251784, + "qps_1t_max": 77.02682257961665, + "qps_1t_runs": [ + 74.5134605231173, + 77.02682257961665, + 72.82908632251784 + ], + "median_ms": 13.105199963320047, + "p95_ms": 16.224100021645427, + "p99_ms": 19.192199979443103, + "mean_ms": 13.377887465240848, + "knob": 256 + } + ], + "ref_knob": 256, + "ref_recall_pre": 0, + "mem_after_sweep": { + "rss_mb": 6741.375, + "peak_mb": 6741.375 + }, + "mutation": { + "ref_knob": 256, + "recall_before": 0, + "insert_10k": { + "seconds": 0.014750300033483654, + "per_s": 677952.3112953417 + }, + "update_10k": { + "seconds": 0.008894999977201223, + "per_s": 1124227.096754469 + }, + "delete_10k": { + "seconds": 0.00644060003105551, + "per_s": 1552650.3667021163 + }, + "first_query_after_ms": 110.59449997264892, + "recall_after": 0.9464435146443515, + "after": { + "nq": 239, + "recall": 0.9464435146443515, + "qps_1t": 76.2746320776492, + "qps_1t_min": 75.29371558113026, + "qps_1t_max": 76.36682558516418, + "qps_1t_runs": [ + 76.2746320776492, + 75.29371558113026, + 76.36682558516418 + ], + "median_ms": 12.906700023449957, + "p95_ms": 15.67679998697713, + "p99_ms": 17.723999975714833, + "mean_ms": 13.162176987881066 + }, + "mem_after": { + "rss_mb": 6774.3828125, + "peak_mb": 6774.3828125 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 1000000 + } + } + } + ], + "load_before_pct": 2.6718749999999987, + "load_contaminated": false, + "wall_s": 98.52357268333435, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/csharp__text384__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/csharp__text384__100000__core+mutation+saveload.json new file mode 100644 index 0000000..2257870 --- /dev/null +++ b/benchmarks/competitive/results/raw/csharp__text384__100000__core+mutation+saveload.json @@ -0,0 +1,331 @@ +{ + "status": "ok", + "label": "csharp", + "runs": [ + { + "system": "csharp_exact", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-csharp (repo HEAD, .NET 8 Release)", + "build": { + "ingest_s": 0.018145900103263557, + "index_s": 0, + "total_s": 0.018145900103263557, + "first_query_ms": 46.90479999408126, + "ready_s": 0.06505070009734482 + }, + "mem_baseline": { + "rss_mb": 34.3671875, + "peak_mb": 34.3671875 + }, + "mem_after_ingest": { + "rss_mb": 320.6875, + "peak_mb": 320.6875 + }, + "mem_after_build": { + "rss_mb": 325.40625, + "peak_mb": 325.40625 + }, + "mem_after_first_query": { + "rss_mb": 325.40625, + "peak_mb": 325.40625 + }, + "sweep": [ + { + "nq": 128, + "recall": 1, + "qps_1t": 39.143486688959456, + "qps_1t_min": 38.5773911913353, + "qps_1t_max": 39.927237596704984, + "qps_1t_runs": [ + 39.927237596704984, + 39.143486688959456, + 38.5773911913353 + ], + "median_ms": 23.59890000661835, + "p95_ms": 26.737500040326267, + "p99_ms": 78.39999999850988, + "mean_ms": 25.504837500344973, + "knob": null + } + ], + "mem_after_sweep": { + "rss_mb": 557.86328125, + "peak_mb": 658.625 + } + }, + { + "system": "csharp_ivf", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-csharp (repo HEAD, .NET 8 Release)", + "knob_name": "nprobe", + "build": { + "ingest_s": 0.015840800013393164, + "index_s": 1.2653368000173941, + "total_s": 1.2811776000307873, + "first_query_ms": 0, + "ready_s": 1.2811776000307873 + }, + "mem_baseline": { + "rss_mb": 34.3671875, + "peak_mb": 34.3671875 + }, + "mem_after_ingest": { + "rss_mb": 584.828125, + "peak_mb": 658.625 + }, + "mem_after_build": { + "rss_mb": 821.87109375, + "peak_mb": 821.87109375 + }, + "index_stats": { + "nlist": 1265, + "trained_n": 100000, + "index_bytes": 2848232, + "vector_bytes": 173274640, + "build_seconds": 1.1980652 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.27755, + "qps_1t": 9706.066360519993, + "qps_1t_min": 5430.374857466292, + "qps_1t_max": 9894.171933994656, + "qps_1t_runs": [ + 5430.374857466292, + 9706.066360519993, + 9894.171933994656 + ], + "median_ms": 0.10350003140047193, + "p95_ms": 0.21210004342719913, + "p99_ms": 0.23100001271814108, + "mean_ms": 0.12941576658825701, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.39915, + "qps_1t": 8523.895674038828, + "qps_1t_min": 8518.765561809434, + "qps_1t_max": 8603.634434426238, + "qps_1t_runs": [ + 8603.634434426238, + 8518.765561809434, + 8523.895674038828 + ], + "median_ms": 0.11339999036863446, + "p95_ms": 0.13609998859465122, + "p99_ms": 0.1566999708302319, + "mean_ms": 0.11697836667493296, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.52005, + "qps_1t": 6881.152180755526, + "qps_1t_min": 6817.441201758321, + "qps_1t_max": 6932.4138244126725, + "qps_1t_runs": [ + 6817.441201758321, + 6932.4138244126725, + 6881.152180755526 + ], + "median_ms": 0.14270003885030746, + "p95_ms": 0.17590000061318278, + "p99_ms": 0.20030001178383827, + "mean_ms": 0.14541899991066506, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.6376, + "qps_1t": 4982.163855045766, + "qps_1t_min": 4892.782018353975, + "qps_1t_max": 5000.908921098519, + "qps_1t_runs": [ + 4892.782018353975, + 4982.163855045766, + 5000.908921098519 + ], + "median_ms": 0.1970999874174595, + "p95_ms": 0.24769996525719762, + "p99_ms": 0.27650000993162394, + "mean_ms": 0.20168744993861765, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.74525, + "qps_1t": 3315.4728823879177, + "qps_1t_min": 3315.4014311213896, + "qps_1t_max": 3348.955653799659, + "qps_1t_runs": [ + 3315.4014311213896, + 3348.955653799659, + 3315.4728823879177 + ], + "median_ms": 0.2958999830298126, + "p95_ms": 0.3722999827004969, + "p99_ms": 0.4113000468350947, + "mean_ms": 0.30061308330429404, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.8359, + "qps_1t": 1974.545151249959, + "qps_1t_min": 1971.2368919733783, + "qps_1t_max": 2018.693504697093, + "qps_1t_runs": [ + 1974.545151249959, + 1971.2368919733783, + 2018.693504697093 + ], + "median_ms": 0.49399997806176543, + "p95_ms": 0.6204000092111528, + "p99_ms": 0.6898000137880445, + "mean_ms": 0.5030371169074594, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.90595, + "qps_1t": 1134.2566903835593, + "qps_1t_min": 1127.0211435259314, + "qps_1t_max": 1134.5587846774165, + "qps_1t_runs": [ + 1134.5587846774165, + 1127.0211435259314, + 1134.2566903835593 + ], + "median_ms": 0.8779999916441739, + "p95_ms": 1.0717000113800168, + "p99_ms": 1.1555000091902912, + "mean_ms": 0.8834431167439712, + "knob": 64 + }, + { + "nq": 1848, + "recall": 0.952056277056277, + "qps_1t": 598.0226033292612, + "qps_1t_min": 591.1868058939422, + "qps_1t_max": 599.513946038056, + "qps_1t_runs": [ + 598.0226033292612, + 599.513946038056, + 591.1868058939422 + ], + "median_ms": 1.664099982008338, + "p95_ms": 1.999199972487986, + "p99_ms": 2.2201999672688544, + "mean_ms": 1.6772360750175859, + "knob": 128 + }, + { + "nq": 897, + "recall": 0.9804905239687849, + "qps_1t": 305.6254191463579, + "qps_1t_min": 304.8535022490456, + "qps_1t_max": 306.37391987381085, + "qps_1t_runs": [ + 305.6254191463579, + 306.37391987381085, + 304.8535022490456 + ], + "median_ms": 3.2719000009819865, + "p95_ms": 3.690499986987561, + "p99_ms": 3.9022000273689628, + "mean_ms": 3.2720762169425, + "knob": 256 + }, + { + "nq": 122, + "recall": 1, + "qps_1t": 38.63550419867212, + "qps_1t_min": 37.41275238642007, + "qps_1t_max": 39.41333445668147, + "qps_1t_runs": [ + 38.63550419867212, + 37.41275238642007, + 39.41333445668147 + ], + "median_ms": 23.669400019571185, + "p95_ms": 25.08359996136278, + "p99_ms": 95.14760004822165, + "mean_ms": 25.99463551900034, + "knob": 1265 + } + ], + "ref_knob": 128, + "ref_recall_pre": 0.952056277056277, + "mem_after_sweep": { + "rss_mb": 1151.58984375, + "peak_mb": 1370.30859375 + }, + "mutation": { + "ref_knob": 128, + "recall_before": 0.952056277056277, + "insert_10k": { + "seconds": 0.02585999999428168, + "per_s": 386697.6025603736 + }, + "update_10k": { + "seconds": 0.008704700041562319, + "per_s": 1148804.6632569777 + }, + "delete_10k": { + "seconds": 0.004127299995161593, + "per_s": 2422891.4815310095 + }, + "first_query_after_ms": 44.82020001159981, + "recall_after": 0.9494189263973437, + "after": { + "nq": 1807, + "recall": 0.9494189263973437, + "qps_1t": 591.1852739081532, + "qps_1t_min": 590.4691545739432, + "qps_1t_max": 592.5683598025627, + "qps_1t_runs": [ + 591.1852739081532, + 592.5683598025627, + 590.4691545739432 + ], + "median_ms": 1.6832000110298395, + "p95_ms": 1.9957000040449202, + "p99_ms": 2.1343999542295933, + "mean_ms": 1.6908848921292827 + }, + "mem_after": { + "rss_mb": 1170.19140625, + "peak_mb": 1370.30859375 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 100000 + } + }, + "saveload": { + "save_s": 4.408426200039685, + "disk_mb": 452.6868906021118, + "load_s": 3.953144599974621, + "first_query_after_load_ms": 910.6310000061058, + "time_to_ready_s": 4.863775599980727 + } + } + ], + "load_before_pct": 4.203124999999998, + "load_contaminated": false, + "wall_s": 75.05336785316467, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/csharp__text384__10000__core.json b/benchmarks/competitive/results/raw/csharp__text384__10000__core.json new file mode 100644 index 0000000..7ef43bd --- /dev/null +++ b/benchmarks/competitive/results/raw/csharp__text384__10000__core.json @@ -0,0 +1,281 @@ +{ + "status": "ok", + "label": "csharp", + "runs": [ + { + "system": "csharp_exact", + "ds": "text384", + "n": 10000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-csharp (repo HEAD, .NET 8 Release)", + "build": { + "ingest_s": 0.0047627000021748245, + "index_s": 0, + "total_s": 0.0047627000021748245, + "first_query_ms": 24.22700001625344, + "ready_s": 0.028989700018428266 + }, + "mem_baseline": { + "rss_mb": 34.35546875, + "peak_mb": 34.35546875 + }, + "mem_after_ingest": { + "rss_mb": 81.01171875, + "peak_mb": 81.01171875 + }, + "mem_after_build": { + "rss_mb": 82.64453125, + "peak_mb": 82.64453125 + }, + "mem_after_first_query": { + "rss_mb": 82.64453125, + "peak_mb": 82.64453125 + }, + "sweep": [ + { + "nq": 706, + "recall": 1, + "qps_1t": 421.6285659999388, + "qps_1t_min": 382.4527638614905, + "qps_1t_max": 422.5427136924413, + "qps_1t_runs": [ + 382.4527638614905, + 421.6285659999388, + 422.5427136924413 + ], + "median_ms": 2.332200005184859, + "p95_ms": 2.7593999984674156, + "p99_ms": 4.294199985451996, + "mean_ms": 2.4510274785883004, + "knob": null + } + ], + "mem_after_sweep": { + "rss_mb": 96.01171875, + "peak_mb": 214.0546875 + } + }, + { + "system": "csharp_ivf", + "ds": "text384", + "n": 10000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-csharp (repo HEAD, .NET 8 Release)", + "knob_name": "nprobe", + "build": { + "ingest_s": 0.0010065999813377857, + "index_s": 0.21723320003366098, + "total_s": 0.21823980001499876, + "first_query_ms": 0, + "ready_s": 0.21823980001499876 + }, + "mem_baseline": { + "rss_mb": 34.35546875, + "peak_mb": 34.35546875 + }, + "mem_after_ingest": { + "rss_mb": 109.47265625, + "peak_mb": 214.0546875 + }, + "mem_after_build": { + "rss_mb": 127.7890625, + "peak_mb": 214.0546875 + }, + "index_stats": { + "nlist": 400, + "trained_n": 10000, + "index_bytes": 706132, + "vector_bytes": 17349640, + "build_seconds": 0.208709 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.2871, + "qps_1t": 13263.742914285021, + "qps_1t_min": 11380.364206647917, + "qps_1t_max": 38014.57456031691, + "qps_1t_runs": [ + 11380.364206647917, + 13263.742914285021, + 38014.57456031691 + ], + "median_ms": 0.06509997183457017, + "p95_ms": 0.09330001194030046, + "p99_ms": 0.10390003444626927, + "mean_ms": 0.06318995001493022, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.4162, + "qps_1t": 35917.47603515432, + "qps_1t_min": 35410.13789991665, + "qps_1t_max": 36260.26163904982, + "qps_1t_runs": [ + 35410.13789991665, + 35917.47603515432, + 36260.26163904982 + ], + "median_ms": 0.027399975806474686, + "p95_ms": 0.031800009310245514, + "p99_ms": 0.034999975468963385, + "mean_ms": 0.027886833335893847, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.55765, + "qps_1t": 32061.71243596814, + "qps_1t_min": 28122.63154126515, + "qps_1t_max": 32333.892692292353, + "qps_1t_runs": [ + 32333.892692292353, + 32061.71243596814, + 28122.63154126515 + ], + "median_ms": 0.030700000934302807, + "p95_ms": 0.03619998460635543, + "p99_ms": 0.0399000127799809, + "mean_ms": 0.032558566657826304, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.69015, + "qps_1t": 27379.78225036018, + "qps_1t_min": 26392.15318134635, + "qps_1t_max": 27465.359339925908, + "qps_1t_runs": [ + 26392.15318134635, + 27379.78225036018, + 27465.359339925908 + ], + "median_ms": 0.03619998460635543, + "p95_ms": 0.04250003257766366, + "p99_ms": 0.047200010158121586, + "mean_ms": 0.03694094993018855, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.80795, + "qps_1t": 20211.75860021833, + "qps_1t_min": 19990.38461461821, + "qps_1t_max": 21096.24530544721, + "qps_1t_runs": [ + 21096.24530544721, + 20211.75860021833, + 19990.38461461821 + ], + "median_ms": 0.04769995575770736, + "p95_ms": 0.057699973694980145, + "p99_ms": 0.06610003765672445, + "mean_ms": 0.04896733332619382, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.8959, + "qps_1t": 14154.37181271068, + "qps_1t_min": 13804.870893174471, + "qps_1t_max": 14311.393029373858, + "qps_1t_runs": [ + 13804.870893174471, + 14311.393029373858, + 14154.37181271068 + ], + "median_ms": 0.06890000076964498, + "p95_ms": 0.08410000009462237, + "p99_ms": 0.09869999485090375, + "mean_ms": 0.07098738335965511, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.95285, + "qps_1t": 8624.912887402754, + "qps_1t_min": 8440.670504909185, + "qps_1t_max": 8767.780517236995, + "qps_1t_runs": [ + 8624.912887402754, + 8767.780517236995, + 8440.670504909185 + ], + "median_ms": 0.11319998884573579, + "p95_ms": 0.139000010676682, + "p99_ms": 0.16009999671950936, + "mean_ms": 0.11615705007958847, + "knob": 64 + }, + { + "nq": 2000, + "recall": 0.9878, + "qps_1t": 4986.246684553161, + "qps_1t_min": 4766.138679841505, + "qps_1t_max": 5007.739458761456, + "qps_1t_runs": [ + 4986.246684553161, + 5007.739458761456, + 4766.138679841505 + ], + "median_ms": 0.19910000264644623, + "p95_ms": 0.24100003065541387, + "p99_ms": 0.2741999924182892, + "mean_ms": 0.20335200006957166, + "knob": 128 + }, + { + "nq": 2000, + "recall": 0.99895, + "qps_1t": 2565.8934268456387, + "qps_1t_min": 2479.8115446371407, + "qps_1t_max": 2636.376458691203, + "qps_1t_runs": [ + 2636.376458691203, + 2479.8115446371407, + 2565.8934268456387 + ], + "median_ms": 0.3795999800786376, + "p95_ms": 0.46389998169615865, + "p99_ms": 0.542099995072931, + "mean_ms": 0.3907642499446714, + "knob": 256 + }, + { + "nq": 1333, + "recall": 1, + "qps_1t": 420.6626462141402, + "qps_1t_min": 417.6507673578117, + "qps_1t_max": 421.22623790768824, + "qps_1t_runs": [ + 417.6507673578117, + 421.22623790768824, + 420.6626462141402 + ], + "median_ms": 2.3204999743029475, + "p95_ms": 2.628199988976121, + "p99_ms": 3.1295999651774764, + "mean_ms": 2.3818559641059034, + "knob": 400 + } + ], + "ref_knob": 64, + "ref_recall_pre": 0.95285, + "mem_after_sweep": { + "rss_mb": 142.5390625, + "peak_mb": 214.0546875 + } + } + ], + "load_before_pct": 5.240625000000002, + "load_contaminated": false, + "wall_s": 21.63262128829956, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_flat__glove__1000000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/faiss_flat__glove__1000000__core+mutation+saveload.json new file mode 100644 index 0000000..debbc4f --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_flat__glove__1000000__core+mutation+saveload.json @@ -0,0 +1,142 @@ +{ + "system": "faiss_flat", + "ds": "glove", + "n": 1000000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 10:31:28", + "mem_baseline": { + "rss_mb": 34.97265625, + "peak_mb": 34.97265625 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 0.3754827000084333, + "index_s": 2.3999600671231747e-06, + "total_s": 0.37548509996850044, + "first_query_ms": 28.816199977882206, + "ready_s": 0.40430129994638264 + }, + "mem_after_build": { + "rss_mb": 504.5234375, + "peak_mb": 639.9921875 + }, + "mem_after_first_query": { + "rss_mb": 504.6015625, + "peak_mb": 639.9921875 + }, + "sweep": [ + { + "nq": 149, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 42.621622221620804, + 40.715336211406004, + 39.77757448303346 + ], + "qps_1t": 40.715336211406004, + "qps_1t_min": 39.77757448303346, + "qps_1t_max": 42.621622221620804, + "median_ms": 24.24780000001192, + "p95_ms": 27.131040033418685, + "p99_ms": 33.16403796430677, + "mean_ms": 24.387610963466567, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 638.9404629218408, + 642.4085076945206, + 596.8994654130149 + ], + "qps_batch": 638.9404629218408, + "qps_batch_min": 596.8994654130149, + "qps_batch_max": 642.4085076945206, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 85, + "threads": 32, + "qps": 49.72282741778036, + "median_ms": 20.405200019013137, + "p95_ms": 25.612299994099867, + "p99_ms": 29.875211997423346, + "mean_ms": 20.80622627240076 + }, + "mem_after_sweep": { + "rss_mb": 517.64453125, + "peak_mb": 639.9921875 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.004300100030377507, + "per_s": 2325527.2968899044, + "batch": 1000 + }, + "update_10k": { + "seconds": 3.943905799998902, + "per_s": 2535.5575176270145, + "batch": 1000 + }, + "delete_10k": { + "seconds": 4.188923900015652, + "per_s": 2387.247951666688, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 529.20703125, + "peak_mb": 639.9921875 + }, + "first_query_after_ms": 65.74649998219684, + "after": { + "nq": 119, + "recall_queries": 119, + "recall": 1.0, + "qps_1t_runs": [ + 37.5394677020512, + 37.76714183374294, + 37.35729474198767 + ], + "qps_1t": 37.5394677020512, + "qps_1t_min": 37.35729474198767, + "qps_1t_max": 37.76714183374294, + "median_ms": 26.818200014531612, + "p95_ms": 27.768639964051545, + "p99_ms": 29.021992003545165, + "mean_ms": 26.628402240459575 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 0.6554340000147931, + "disk_mb": 389.0992069244385, + "load_s": 0.9732154000084847, + "first_query_after_load_ms": 66.75289996201172, + "time_to_ready_s": 1.0399682999704964 + }, + "mem_end": { + "rss_mb": 531.0625, + "peak_mb": 970.30078125 + }, + "status": "ok", + "finished": "2026-11-16 10:32:40", + "load_before_pct": 9.45937499999999, + "load_contaminated": false, + "wall_s": 72.60048651695251, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_flat__glove__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/faiss_flat__glove__100000__core+mutation+saveload.json new file mode 100644 index 0000000..7509143 --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_flat__glove__100000__core+mutation+saveload.json @@ -0,0 +1,147 @@ +{ + "system": "faiss_flat", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 08:26:11", + "mem_baseline": { + "rss_mb": 35.1953125, + "peak_mb": 35.1953125 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 0.03100400004768744, + "index_s": 1.400010660290718e-06, + "total_s": 0.031005400058347732, + "first_query_ms": 2.1878000115975738, + "ready_s": 0.033193200069945306, + "total_s_runs": [ + 0.031005400058347732, + 0.04659990005893633, + 0.03541469993069768 + ] + }, + "mem_after_build": { + "rss_mb": 98.4375, + "peak_mb": 135.70703125 + }, + "mem_after_first_query": { + "rss_mb": 98.515625, + "peak_mb": 135.70703125 + }, + "sweep": [ + { + "nq": 1809, + "recall_queries": 1809, + "recall": 1.0, + "qps_1t_runs": [ + 548.4301740392543, + 569.6022816084964, + 543.4661298597612 + ], + "qps_1t": 548.4301740392543, + "qps_1t_min": 543.4661298597612, + "qps_1t_max": 569.6022816084964, + "median_ms": 1.7164000310003757, + "p95_ms": 2.2275400231592357, + "p99_ms": 3.014491976937277, + "mean_ms": 1.8063460655963646, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 17355.988679941973, + 16184.59502077011, + 16430.47853314927 + ], + "qps_batch": 16430.47853314927, + "qps_batch_min": 16184.59502077011, + "qps_batch_max": 17355.988679941973, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 2000, + "threads": 32, + "qps": 2335.268685500277, + "median_ms": 0.41310000233352184, + "p95_ms": 0.613204954424873, + "p99_ms": 0.7323019881732762, + "mean_ms": 0.4299485333710133 + }, + "mem_after_sweep": { + "rss_mb": 112.1640625, + "peak_mb": 135.70703125 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.0020111000048927963, + "per_s": 4972403.150351074, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.10233710001921281, + "per_s": 97716.27296574357, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.09884759999113157, + "per_s": 101165.8350925787, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 123.79296875, + "peak_mb": 195.296875 + }, + "first_query_after_ms": 2.7194999856874347, + "after": { + "nq": 1344, + "recall_queries": 1344, + "recall": 1.0, + "qps_1t_runs": [ + 556.5854873373852, + 552.7321026899448, + 562.1154042453633 + ], + "qps_1t": 556.5854873373852, + "qps_1t_min": 552.7321026899448, + "qps_1t_max": 562.1154042453633, + "median_ms": 1.7166000325232744, + "p95_ms": 2.0978749991627406, + "p99_ms": 3.052488010143864, + "mean_ms": 1.7949526782248295 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 0.04537860001437366, + "disk_mb": 38.90999794006348, + "load_s": 0.024948099977336824, + "first_query_after_load_ms": 2.1277000196278095, + "time_to_ready_s": 0.027075799996964633 + }, + "mem_end": { + "rss_mb": 109.06640625, + "peak_mb": 195.296875 + }, + "status": "ok", + "finished": "2026-11-16 08:26:32", + "load_before_pct": 7.184375000000008, + "load_contaminated": false, + "wall_s": 21.293954372406006, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_flat__glove__100000__filter.json b/benchmarks/competitive/results/raw/faiss_flat__glove__100000__filter.json new file mode 100644 index 0000000..6cb3b04 --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_flat__glove__100000__filter.json @@ -0,0 +1,66 @@ +{ + "system": "faiss_flat", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 11:46:59", + "filter": { + "build_with_metadata_s": 0.035980099986772984, + "mem": { + "rss_mb": 97.71484375, + "peak_mb": 134.9140625 + }, + "subsets": { + "f10": [ + { + "knob": null, + "nq": 2000, + "recall": 1.0, + "qps_1t": 1813.4391806365081, + "qps_1t_runs": [ + 1813.4391806365081, + 1816.32883306855, + 1348.9110076033562 + ], + "median_ms": 0.5298000178299844, + "p95_ms": 0.9326049999799579, + "p99_ms": 1.0319079994224014, + "mean_ms": 0.6144460831710603 + } + ], + "f01": [ + { + "knob": null, + "nq": 2000, + "recall": 1.0, + "qps_1t": 2913.2868925627654, + "qps_1t_runs": [ + 2836.3367024385, + 2989.4910407770653, + 2913.2868925627654 + ], + "median_ms": 0.37949997931718826, + "p95_ms": 0.41820498590823263, + "p99_ms": 0.47071303299162565, + "mean_ms": 0.34344248344617273 + } + ] + } + }, + "mem_end": { + "rss_mb": 52.56640625, + "peak_mb": 134.9140625 + }, + "status": "ok", + "finished": "2026-11-16 11:47:05", + "load_before_pct": 8.865624999999993, + "load_contaminated": false, + "wall_s": 6.567217111587524, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_flat__sift__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/faiss_flat__sift__100000__core+mutation+saveload.json new file mode 100644 index 0000000..d336ba5 --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_flat__sift__100000__core+mutation+saveload.json @@ -0,0 +1,147 @@ +{ + "system": "faiss_flat", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 11:16:08", + "mem_baseline": { + "rss_mb": 35.37109375, + "peak_mb": 35.37109375 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 0.050621099944692105, + "index_s": 2.2999593056738377e-06, + "total_s": 0.05062339990399778, + "first_query_ms": 3.5072999889962375, + "ready_s": 0.054130699892994016, + "total_s_runs": [ + 0.05062339990399778, + 0.08835479995468631, + 0.08185000001685694 + ] + }, + "mem_after_build": { + "rss_mb": 109.171875, + "peak_mb": 157.1015625 + }, + "mem_after_first_query": { + "rss_mb": 109.25, + "peak_mb": 157.1015625 + }, + "sweep": [ + { + "nq": 920, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 332.22724533304824, + 283.06582681202025, + 279.60595504025054 + ], + "qps_1t": 283.06582681202025, + "qps_1t_min": 279.60595504025054, + "qps_1t_max": 332.22724533304824, + "median_ms": 3.2985500001814216, + "p95_ms": 4.2591150093358, + "p99_ms": 6.042718980461356, + "mean_ms": 3.373065615955241, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 3003.8428160889043, + 4965.600800532972, + 4537.410953264215 + ], + "qps_batch": 4537.410953264215, + "qps_batch_min": 3003.8428160889043, + "qps_batch_max": 4965.600800532972, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 835, + "threads": 32, + "qps": 285.1063871587876, + "median_ms": 3.4351000213064253, + "p95_ms": 4.934740008320654, + "p99_ms": 6.255139959976077, + "mean_ms": 3.515263473728445 + }, + "mem_after_sweep": { + "rss_mb": 123.875, + "peak_mb": 157.1015625 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.004522399976849556, + "per_s": 2211215.29523939, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.5198939999681897, + "per_s": 19234.690149553295, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.3611703999922611, + "per_s": 27687.76178838098, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 138.796875, + "peak_mb": 228.41796875 + }, + "first_query_after_ms": 7.9619999742135406, + "after": { + "nq": 932, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 261.7900976180128, + 278.3804708982491, + 278.04389681098286 + ], + "qps_1t": 278.04389681098286, + "qps_1t_min": 261.7900976180128, + "qps_1t_max": 278.3804708982491, + "median_ms": 3.473200020380318, + "p95_ms": 4.544775016256608, + "p99_ms": 5.785115031176253, + "mean_ms": 3.669538197048656 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 0.06047320005018264, + "disk_mb": 49.59115028381348, + "load_s": 0.08740339998621494, + "first_query_after_load_ms": 12.896700005512685, + "time_to_ready_s": 0.10030009999172762 + }, + "mem_end": { + "rss_mb": 121.13671875, + "peak_mb": 228.41796875 + }, + "status": "ok", + "finished": "2026-11-16 11:16:41", + "load_before_pct": 16.834375, + "load_contaminated": false, + "wall_s": 33.22057032585144, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_flat__text384__1000000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/faiss_flat__text384__1000000__core+mutation+saveload.json new file mode 100644 index 0000000..f094e8a --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_flat__text384__1000000__core+mutation+saveload.json @@ -0,0 +1,142 @@ +{ + "system": "faiss_flat", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 09:45:53", + "mem_baseline": { + "rss_mb": 37.29296875, + "peak_mb": 37.29296875 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 1.1041080001741648, + "index_s": 2.600019797682762e-06, + "total_s": 1.1041106001939625, + "first_query_ms": 77.55009998800233, + "ready_s": 1.1816607001819648 + }, + "mem_after_build": { + "rss_mb": 1590.109375, + "peak_mb": 2158.8671875 + }, + "mem_after_first_query": { + "rss_mb": 1590.1875, + "peak_mb": 2158.8671875 + }, + "sweep": [ + { + "nq": 40, + "recall_queries": 40, + "recall": 1.0, + "qps_1t_runs": [ + 13.406619277544596, + 13.30779700113377, + 13.498657238072042 + ], + "qps_1t": 13.406619277544596, + "qps_1t_min": 13.30779700113377, + "qps_1t_max": 13.498657238072042, + "median_ms": 73.35145000251941, + "p95_ms": 83.81378499907441, + "p99_ms": 87.61429797159508, + "mean_ms": 74.60512666633197, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 536.9377607260415, + 539.9866601723231, + 544.7127902417595 + ], + "qps_batch": 539.9866601723231, + "qps_batch_min": 536.9377607260415, + "qps_batch_max": 544.7127902417595, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 71, + "threads": 32, + "qps": 22.764519389133255, + "median_ms": 43.10910002095625, + "p95_ms": 48.87395998230203, + "p99_ms": 52.566392018925384, + "mean_ms": 43.705510796745145 + }, + "mem_after_sweep": { + "rss_mb": 1609.8515625, + "peak_mb": 2158.8671875 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.0073639999609440565, + "per_s": 1357957.6389239973, + "batch": 1000 + }, + "update_10k": { + "seconds": 2.3090925000142306, + "per_s": 4330.705677636722, + "batch": 1000 + }, + "delete_10k": { + "seconds": 2.249738300044555, + "per_s": 4444.9614427606775, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 1654.01953125, + "peak_mb": 2158.8671875 + }, + "first_query_after_ms": 72.11409998126328, + "after": { + "nq": 41, + "recall_queries": 41, + "recall": 1.0, + "qps_1t_runs": [ + 13.200426289242136, + 13.317282868129901, + 13.340324097395301 + ], + "qps_1t": 13.317282868129901, + "qps_1t_min": 13.200426289242136, + "qps_1t_max": 13.340324097395301, + "median_ms": 73.87199997901917, + "p95_ms": 82.18046997790225, + "p99_ms": 88.66506198071875, + "mean_ms": 75.26873983726723 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 0.9680129000335, + "disk_mb": 1472.4732303619385, + "load_s": 0.7364274999708869, + "first_query_after_load_ms": 74.09990002634004, + "time_to_ready_s": 0.8105273999972269 + }, + "mem_end": { + "rss_mb": 1620.57421875, + "peak_mb": 3156.95703125 + }, + "status": "ok", + "finished": "2026-11-16 09:46:49", + "load_before_pct": 3.487499999999998, + "load_contaminated": false, + "wall_s": 56.37593460083008, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_flat__text384__1000000__filter.json b/benchmarks/competitive/results/raw/faiss_flat__text384__1000000__filter.json new file mode 100644 index 0000000..596b19d --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_flat__text384__1000000__filter.json @@ -0,0 +1,66 @@ +{ + "system": "faiss_flat", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 11:54:31", + "filter": { + "build_with_metadata_s": 1.132167800096795, + "mem": { + "rss_mb": 1591.390625, + "peak_mb": 2158.99609375 + }, + "subsets": { + "f10": [ + { + "knob": null, + "nq": 148, + "recall": 1.0, + "qps_1t": 39.68689719300676, + "qps_1t_runs": [ + 41.11284013059143, + 39.68689719300676, + 38.95106774889948 + ], + "median_ms": 24.96424998389557, + "p95_ms": 27.92759498115629, + "p99_ms": 32.722414002055274, + "mean_ms": 25.064590541742373 + } + ], + "f01": [ + { + "knob": null, + "nq": 512, + "recall": 1.0, + "qps_1t": 171.8712664963945, + "qps_1t_runs": [ + 173.52706364024056, + 171.8712664963945, + 169.55055494698 + ], + "median_ms": 5.966450000414625, + "p95_ms": 6.443374993978068, + "p99_ms": 7.135140005266295, + "mean_ms": 5.826347982103168 + } + ] + } + }, + "mem_end": { + "rss_mb": 52.94140625, + "peak_mb": 2158.99609375 + }, + "status": "ok", + "finished": "2026-11-16 11:54:54", + "load_before_pct": 13.424999999999994, + "load_contaminated": false, + "wall_s": 23.033551454544067, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_flat__text384__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/faiss_flat__text384__100000__core+mutation+saveload.json new file mode 100644 index 0000000..525b0ee --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_flat__text384__100000__core+mutation+saveload.json @@ -0,0 +1,147 @@ +{ + "system": "faiss_flat", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 08:04:31", + "mem_baseline": { + "rss_mb": 37.18359375, + "peak_mb": 37.18359375 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 0.10819310002261773, + "index_s": 2.500019036233425e-06, + "total_s": 0.10819560004165396, + "first_query_ms": 8.38010001461953, + "ready_s": 0.11657570005627349, + "total_s_runs": [ + 0.10819560004165396, + 0.11093499994603917, + 0.10809980001067743 + ] + }, + "mem_after_build": { + "rss_mb": 208.7734375, + "peak_mb": 354.43359375 + }, + "mem_after_first_query": { + "rss_mb": 208.8515625, + "peak_mb": 354.43359375 + }, + "sweep": [ + { + "nq": 410, + "recall_queries": 1000, + "recall": 0.9999, + "qps_1t_runs": [ + 134.87535559755298, + 130.04684857724365, + 131.07833414999007 + ], + "qps_1t": 131.07833414999007, + "qps_1t_min": 130.04684857724365, + "qps_1t_max": 134.87535559755298, + "median_ms": 7.291549991350621, + "p95_ms": 8.84210500807967, + "p99_ms": 14.971388985286474, + "mean_ms": 7.577605121882587, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 5290.98009611987, + 5437.3254450555505, + 5464.146456969809 + ], + "qps_batch": 5437.3254450555505, + "qps_batch_min": 5290.98009611987, + "qps_batch_max": 5464.146456969809, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 592, + "threads": 32, + "qps": 197.76314517220155, + "median_ms": 4.9707500147633255, + "p95_ms": 5.844175000675023, + "p99_ms": 7.938650014693849, + "mean_ms": 5.062252589997517 + }, + "mem_after_sweep": { + "rss_mb": 228.80859375, + "peak_mb": 354.43359375 + }, + "ref_knob": null, + "ref_recall_pre": 0.9999, + "mutation": { + "ref_knob": null, + "recall_before": 0.9999, + "insert_10k": { + "seconds": 0.005019299976993352, + "per_s": 1992309.6937494008, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.22484439995605499, + "per_s": 44475.20152583059, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.19591890001902357, + "per_s": 51041.5278925566, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 273.23046875, + "peak_mb": 528.78515625 + }, + "first_query_after_ms": 9.092299966141582, + "after": { + "nq": 417, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 135.23524022774401, + 135.12611445120822, + 129.72071409544358 + ], + "qps_1t": 135.12611445120822, + "qps_1t_min": 129.72071409544358, + "qps_1t_max": 135.23524022774401, + "median_ms": 7.211500022094697, + "p95_ms": 8.32735002040863, + "p99_ms": 15.417250018799677, + "mean_ms": 7.501295124555452 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 0.12820340000325814, + "disk_mb": 147.24740028381348, + "load_s": 0.0691456000204198, + "first_query_after_load_ms": 7.26719998056069, + "time_to_ready_s": 0.0764128000009805 + }, + "mem_end": { + "rss_mb": 223.47265625, + "peak_mb": 528.78515625 + }, + "status": "ok", + "finished": "2026-11-16 08:05:11", + "load_before_pct": 3.718749999999997, + "load_contaminated": false, + "wall_s": 40.304704427719116, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_flat__text384__100000__filter.json b/benchmarks/competitive/results/raw/faiss_flat__text384__100000__filter.json new file mode 100644 index 0000000..63fc139 --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_flat__text384__100000__filter.json @@ -0,0 +1,66 @@ +{ + "system": "faiss_flat", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 11:45:12", + "filter": { + "build_with_metadata_s": 0.10917459987103939, + "mem": { + "rss_mb": 208.296875, + "peak_mb": 353.9453125 + }, + "subsets": { + "f10": [ + { + "knob": null, + "nq": 2000, + "recall": 1.0, + "qps_1t": 464.01631223229913, + "qps_1t_runs": [ + 621.7209465948339, + 464.01631223229913, + 454.2665659081649 + ], + "median_ms": 2.0099000248592347, + "p95_ms": 2.5670049886684865, + "p99_ms": 2.95190300908871, + "mean_ms": 1.9882953165312454 + } + ], + "f01": [ + { + "knob": null, + "nq": 2000, + "recall": 1.0, + "qps_1t": 2490.833729894517, + "qps_1t_runs": [ + 2490.833729894517, + 2629.590836015668, + 2349.4957100081424 + ], + "median_ms": 0.4429499967955053, + "p95_ms": 0.4827000084333122, + "p99_ms": 0.5441209528362383, + "mean_ms": 0.4024608502222691 + } + ] + } + }, + "mem_end": { + "rss_mb": 52.5, + "peak_mb": 353.9453125 + }, + "status": "ok", + "finished": "2026-11-16 11:45:27", + "load_before_pct": 8.765624999999998, + "load_contaminated": false, + "wall_s": 15.378841876983643, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_flat__text384__10000__core.json b/benchmarks/competitive/results/raw/faiss_flat__text384__10000__core.json new file mode 100644 index 0000000..ca8fa13 --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_flat__text384__10000__core.json @@ -0,0 +1,91 @@ +{ + "system": "faiss_flat", + "ds": "text384", + "n": 10000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 08:57:31", + "mem_baseline": { + "rss_mb": 37.26171875, + "peak_mb": 37.26171875 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 0.004413699964061379, + "index_s": 1.200009137392044e-06, + "total_s": 0.0044148999731987715, + "first_query_ms": 0.8730000117793679, + "ready_s": 0.005287899984978139 + }, + "mem_after_build": { + "rss_mb": 69.9921875, + "peak_mb": 84.6484375 + }, + "mem_after_first_query": { + "rss_mb": 70.0703125, + "peak_mb": 84.6484375 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 1.0, + "qps_1t_runs": [ + 2845.028324334414, + 2752.5226171264526, + 2860.3783559199533 + ], + "qps_1t": 2845.028324334414, + "qps_1t_min": 2752.5226171264526, + "qps_1t_max": 2860.3783559199533, + "median_ms": 0.3364000003784895, + "p95_ms": 0.4412049864185974, + "p99_ms": 0.5895130062708634, + "mean_ms": 0.35479918311466463, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 57217.23959163985, + 55693.241558925154, + 50884.62930485946 + ], + "qps_batch": 55693.241558925154, + "qps_batch_min": 50884.62930485946, + "qps_batch_max": 57217.23959163985, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 723, + "threads": 32, + "qps": 292.60018974727126, + "median_ms": 3.4824999747797847, + "p95_ms": 4.258919972926377, + "p99_ms": 5.689883986487994, + "mean_ms": 3.5678609504045404 + }, + "mem_after_sweep": { + "rss_mb": 81.8671875, + "peak_mb": 89.44140625 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 78.93359375, + "peak_mb": 89.44140625 + }, + "status": "ok", + "finished": "2026-11-16 08:57:41", + "load_before_pct": 4.443749999999998, + "load_contaminated": false, + "wall_s": 10.524060010910034, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_hnsw__glove__1000000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/faiss_hnsw__glove__1000000__core+mutation+saveload.json new file mode 100644 index 0000000..324e645 --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_hnsw__glove__1000000__core+mutation+saveload.json @@ -0,0 +1,440 @@ +{ + "system": "faiss_hnsw", + "ds": "glove", + "n": 1000000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 10:18:49", + "mem_baseline": { + "rss_mb": 35.26953125, + "peak_mb": 35.26953125 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 60.1476027003373, + "index_s": 4.700035788118839e-06, + "total_s": 60.14760740037309, + "first_query_ms": 7.465199974831194, + "ready_s": 60.15507260034792 + }, + "mem_after_build": { + "rss_mb": 580.34375, + "peak_mb": 701.4375 + }, + "mem_after_first_query": { + "rss_mb": 581.390625, + "peak_mb": 701.4375 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.4765, + "qps_1t_runs": [ + 22888.21810888945, + 24051.18092754045, + 18722.752523644373 + ], + "qps_1t": 22888.21810888945, + "qps_1t_min": 18722.752523644373, + "qps_1t_max": 24051.18092754045, + "median_ms": 0.04120002267882228, + "p95_ms": 0.06629998388234526, + "p99_ms": 0.08560201153159146, + "mean_ms": 0.04622651693838028, + "knob": 10, + "batch": { + "recall": 0.4765, + "qps_batch_runs": [ + 166476.60621035923, + 139533.26121217277, + 198532.84249354433 + ], + "qps_batch": 166476.60621035923, + "qps_batch_min": 139533.26121217277, + "qps_batch_max": 198532.84249354433, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.56045, + "qps_1t_runs": [ + 17882.401761741836, + 18038.949655426582, + 14705.276824495266 + ], + "qps_1t": 17882.401761741836, + "qps_1t_min": 14705.276824495266, + "qps_1t_max": 18038.949655426582, + "median_ms": 0.0523999915458262, + "p95_ms": 0.08330499404110016, + "p99_ms": 0.11730202008038763, + "mean_ms": 0.059786433392825224, + "knob": 16, + "batch": { + "recall": 0.56045, + "qps_batch_runs": [ + 113286.20646650107, + 136314.06797283728, + 134982.82329708757 + ], + "qps_batch": 134982.82329708757, + "qps_batch_min": 113286.20646650107, + "qps_batch_max": 136314.06797283728, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.62985, + "qps_1t_runs": [ + 13066.77979409951, + 13769.74239294838, + 13430.471511793348 + ], + "qps_1t": 13430.471511793348, + "qps_1t_min": 13066.77979409951, + "qps_1t_max": 13769.74239294838, + "median_ms": 0.07050001295283437, + "p95_ms": 0.10820000898092985, + "p99_ms": 0.1342109747929501, + "mean_ms": 0.07453683307782437, + "knob": 24, + "batch": { + "recall": 0.62985, + "qps_batch_runs": [ + 106855.87264693242, + 88955.70470287952, + 103150.7395173291 + ], + "qps_batch": 103150.7395173291, + "qps_batch_min": 88955.70470287952, + "qps_batch_max": 106855.87264693242, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.67245, + "qps_1t_runs": [ + 11266.590013464034, + 10956.010486430781, + 10905.60107655429 + ], + "qps_1t": 10956.010486430781, + "qps_1t_min": 10905.60107655429, + "qps_1t_max": 11266.590013464034, + "median_ms": 0.08570001227781177, + "p95_ms": 0.1294999965466559, + "p99_ms": 0.15721003233920805, + "mean_ms": 0.09057603365120788, + "knob": 32, + "batch": { + "recall": 0.67245, + "qps_batch_runs": [ + 73210.19383262319, + 76873.40480569516, + 68137.52882005414 + ], + "qps_batch": 73210.19383262319, + "qps_batch_min": 68137.52882005414, + "qps_batch_max": 76873.40480569516, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7281, + "qps_1t_runs": [ + 7702.842789337466, + 8175.124250343606, + 7966.242259985236 + ], + "qps_1t": 7966.242259985236, + "qps_1t_min": 7702.842789337466, + "qps_1t_max": 8175.124250343606, + "median_ms": 0.11789996642619371, + "p95_ms": 0.17869996663648635, + "p99_ms": 0.22449996380601078, + "mean_ms": 0.12589140003547072, + "knob": 48, + "batch": { + "recall": 0.7281, + "qps_batch_runs": [ + 57418.30097382197, + 55036.8334217464, + 60086.22380713183 + ], + "qps_batch": 57418.30097382197, + "qps_batch_min": 55036.8334217464, + "qps_batch_max": 60086.22380713183, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.76575, + "qps_1t_runs": [ + 6303.590397279359, + 6097.133442825387, + 6025.640913166395 + ], + "qps_1t": 6097.133442825387, + "qps_1t_min": 6025.640913166395, + "qps_1t_max": 6303.590397279359, + "median_ms": 0.15070001245476305, + "p95_ms": 0.22401501773856586, + "p99_ms": 0.29450003057718277, + "mean_ms": 0.1628695662948303, + "knob": 64, + "batch": { + "recall": 0.76575, + "qps_batch_runs": [ + 49314.40643675709, + 48203.45717490607, + 45455.88848543174 + ], + "qps_batch": 48203.45717490607, + "qps_batch_min": 45455.88848543174, + "qps_batch_max": 49314.40643675709, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.80925, + "qps_1t_runs": [ + 4593.628113187744, + 4503.60176557051, + 4413.581736931383 + ], + "qps_1t": 4503.60176557051, + "qps_1t_min": 4413.581736931383, + "qps_1t_max": 4593.628113187744, + "median_ms": 0.2125999890267849, + "p95_ms": 0.3007050196174532, + "p99_ms": 0.3912020102143288, + "mean_ms": 0.22210356683353893, + "knob": 96, + "batch": { + "recall": 0.80925, + "qps_batch_runs": [ + 34172.08720355805, + 33341.335251327335, + 33682.96250082914 + ], + "qps_batch": 33682.96250082914, + "qps_batch_min": 33341.335251327335, + "qps_batch_max": 34172.08720355805, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.83405, + "qps_1t_runs": [ + 3280.262503932642, + 3488.773908215054, + 3282.8599139606063 + ], + "qps_1t": 3282.8599139606063, + "qps_1t_min": 3280.262503932642, + "qps_1t_max": 3488.773908215054, + "median_ms": 0.2818999928422272, + "p95_ms": 0.39241501654032623, + "p99_ms": 0.5027200438780715, + "mean_ms": 0.29869991686427966, + "knob": 128, + "batch": { + "recall": 0.83405, + "qps_batch_runs": [ + 28110.891842939778, + 27684.688004903946, + 26828.568132643573 + ], + "qps_batch": 27684.688004903946, + "qps_batch_min": 26828.568132643573, + "qps_batch_max": 28110.891842939778, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8646, + "qps_1t_runs": [ + 2102.5641247943013, + 2296.977350121448, + 2329.227450934512 + ], + "qps_1t": 2296.977350121448, + "qps_1t_min": 2102.5641247943013, + "qps_1t_max": 2329.227450934512, + "median_ms": 0.4186999867670238, + "p95_ms": 0.5511350056622178, + "p99_ms": 0.7170399877941245, + "mean_ms": 0.4467638002242893, + "knob": 192, + "batch": { + "recall": 0.8646, + "qps_batch_runs": [ + 19009.36496026901, + 19025.477009259394, + 18690.662985797964 + ], + "qps_batch": 19009.36496026901, + "qps_batch_min": 18690.662985797964, + "qps_batch_max": 19025.477009259394, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.88495, + "qps_1t_runs": [ + 1796.1655601101736, + 1671.270598800572, + 1782.0778922326388 + ], + "qps_1t": 1782.0778922326388, + "qps_1t_min": 1671.270598800572, + "qps_1t_max": 1796.1655601101736, + "median_ms": 0.5533999647013843, + "p95_ms": 0.7011099834926426, + "p99_ms": 0.9799850109266136, + "mean_ms": 0.5720771337607099, + "knob": 256, + "batch": { + "recall": 0.88495, + "qps_batch_runs": [ + 15758.09434197008, + 13896.754066554366, + 14589.690489100063 + ], + "qps_batch": 14589.690489100063, + "qps_batch_min": 13896.754066554366, + "qps_batch_max": 15758.09434197008, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9119, + "qps_1t_runs": [ + 1151.5785406704854, + 1164.3932271955393, + 1231.6779437558573 + ], + "qps_1t": 1164.3932271955393, + "qps_1t_min": 1151.5785406704854, + "qps_1t_max": 1231.6779437558573, + "median_ms": 0.8033500052988529, + "p95_ms": 1.155210006982088, + "p99_ms": 1.8656600022222882, + "mean_ms": 0.8463634001382161, + "knob": 400, + "batch": { + "recall": 0.9119, + "qps_batch_runs": [ + 10160.851357343035, + 9946.398856004893, + 10451.25924438675 + ], + "qps_batch": 10160.851357343035, + "qps_batch_min": 9946.398856004893, + "qps_batch_max": 10451.25924438675, + "threads": 32 + } + }, + { + "nq": 1892, + "recall_queries": 1892, + "recall": 0.9346723044397462, + "qps_1t_runs": [ + 654.3569622523773, + 649.9139104051824, + 653.4912648642223 + ], + "qps_1t": 653.4912648642223, + "qps_1t_min": 649.9139104051824, + "qps_1t_max": 654.3569622523773, + "median_ms": 1.5218500047922134, + "p95_ms": 1.7911749746417627, + "p99_ms": 2.020375002757646, + "mean_ms": 1.5323751765226032, + "knob": 640, + "batch": { + "recall": 0.935, + "qps_batch_runs": [ + 6535.266421735189, + 6461.448895639118, + 6777.232700617452 + ], + "qps_batch": 6535.266421735189, + "qps_batch_min": 6461.448895639118, + "qps_batch_max": 6777.232700617452, + "threads": 32 + } + } + ], + "knob_name": "efSearch", + "mem_after_sweep": { + "rss_mb": 612.9296875, + "peak_mb": 701.4375 + }, + "ref_knob": 640, + "ref_recall_pre": 0.9346723044397462, + "mutation": { + "ref_knob": 640, + "recall_before": 0.9346723044397462, + "insert_10k": { + "seconds": 1.7875868999981321, + "per_s": 5594.13363345326, + "batch": 1000 + }, + "update_10k": { + "unsupported": true + }, + "delete_10k": { + "unsupported": true + }, + "mem_after": { + "rss_mb": 643.96484375, + "peak_mb": 701.4375 + } + }, + "saveload": { + "save_s": 0.3067853999673389, + "disk_mb": 524.2364444732666, + "load_s": 0.22791910002706572, + "first_query_after_load_ms": 2.1547000505961478, + "time_to_ready_s": 0.23007380007766187 + }, + "mem_end": { + "rss_mb": 618.96484375, + "peak_mb": 1142.30859375 + }, + "status": "ok", + "finished": "2026-11-16 10:20:22", + "load_before_pct": 14.475000000000016, + "load_contaminated": false, + "wall_s": 93.24940872192383, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_hnsw__glove__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/faiss_hnsw__glove__100000__core+mutation+saveload.json new file mode 100644 index 0000000..9aae663 --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_hnsw__glove__100000__core+mutation+saveload.json @@ -0,0 +1,445 @@ +{ + "system": "faiss_hnsw", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 08:19:14", + "mem_baseline": { + "rss_mb": 35.1875, + "peak_mb": 35.1875 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 2.2544728000066243, + "index_s": 2.100015990436077e-06, + "total_s": 2.2544749000226147, + "first_query_ms": 2.0253999973647296, + "ready_s": 2.2565003000199795, + "total_s_runs": [ + 2.2544749000226147, + 2.279344000038691, + 2.4062576998840086 + ] + }, + "mem_after_build": { + "rss_mb": 111.03125, + "peak_mb": 146.97265625 + }, + "mem_after_first_query": { + "rss_mb": 111.1484375, + "peak_mb": 146.97265625 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.53775, + "qps_1t_runs": [ + 32428.786327140377, + 34916.6278201828, + 29418.297361020104 + ], + "qps_1t": 32428.786327140377, + "qps_1t_min": 29418.297361020104, + "qps_1t_max": 34916.6278201828, + "median_ms": 0.027799978852272034, + "p95_ms": 0.04289998032618314, + "p99_ms": 0.06301101413555467, + "mean_ms": 0.03115629999471518, + "knob": 10, + "batch": { + "recall": 0.53775, + "qps_batch_runs": [ + 823214.656820855, + 696694.1926771529, + 892817.283993283 + ], + "qps_batch": 823214.656820855, + "qps_batch_min": 696694.1926771529, + "qps_batch_max": 892817.283993283, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6264, + "qps_1t_runs": [ + 25690.95805872654, + 25351.048714172917, + 27102.07049303316 + ], + "qps_1t": 25690.95805872654, + "qps_1t_min": 25351.048714172917, + "qps_1t_max": 27102.07049303316, + "median_ms": 0.036299985367804766, + "p95_ms": 0.053099996875971556, + "p99_ms": 0.06750604778062565, + "mean_ms": 0.03842261675163172, + "knob": 16, + "batch": { + "recall": 0.6264, + "qps_batch_runs": [ + 631273.2845665496, + 400248.1553731371, + 502739.93695657566 + ], + "qps_batch": 502739.93695657566, + "qps_batch_min": 400248.1553731371, + "qps_batch_max": 631273.2845665496, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6956, + "qps_1t_runs": [ + 19218.166529280712, + 20186.075205835587, + 20931.252264240684 + ], + "qps_1t": 20186.075205835587, + "qps_1t_min": 19218.166529280712, + "qps_1t_max": 20931.252264240684, + "median_ms": 0.04740001168102026, + "p95_ms": 0.0678999931551516, + "p99_ms": 0.08822003088425888, + "mean_ms": 0.04978288344379204, + "knob": 24, + "batch": { + "recall": 0.6956, + "qps_batch_runs": [ + 499650.24751489, + 455404.5160487642, + 511299.72618173336 + ], + "qps_batch": 499650.24751489, + "qps_batch_min": 455404.5160487642, + "qps_batch_max": 511299.72618173336, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7423, + "qps_1t_runs": [ + 17280.140673566646, + 15500.487560586498, + 17481.92150026605 + ], + "qps_1t": 17280.140673566646, + "qps_1t_min": 15500.487560586498, + "qps_1t_max": 17481.92150026605, + "median_ms": 0.056699966080486774, + "p95_ms": 0.0792999635450542, + "p99_ms": 0.10830400977283725, + "mean_ms": 0.05986198299797252, + "knob": 32, + "batch": { + "recall": 0.7423, + "qps_batch_runs": [ + 371367.5630120495, + 381766.81971751305, + 420291.68499289773 + ], + "qps_batch": 381766.81971751305, + "qps_batch_min": 371367.5630120495, + "qps_batch_max": 420291.68499289773, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7966, + "qps_1t_runs": [ + 12844.89349042002, + 11435.308453872674, + 11369.644192816184 + ], + "qps_1t": 11435.308453872674, + "qps_1t_min": 11369.644192816184, + "qps_1t_max": 12844.89349042002, + "median_ms": 0.0784999574534595, + "p95_ms": 0.10840001050382853, + "p99_ms": 0.17650300520472234, + "mean_ms": 0.08441796651459299, + "knob": 48, + "batch": { + "recall": 0.7966, + "qps_batch_runs": [ + 227409.68919318425, + 257688.78866515282, + 263862.68677815323 + ], + "qps_batch": 257688.78866515282, + "qps_batch_min": 227409.68919318425, + "qps_batch_max": 263862.68677815323, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8334, + "qps_1t_runs": [ + 9750.679962644133, + 8971.128209204435, + 9621.194343860849 + ], + "qps_1t": 9621.194343860849, + "qps_1t_min": 8971.128209204435, + "qps_1t_max": 9750.679962644133, + "median_ms": 0.10040000779554248, + "p95_ms": 0.1369049947243184, + "p99_ms": 0.23951401934027702, + "mean_ms": 0.10598761676616657, + "knob": 64, + "batch": { + "recall": 0.8334, + "qps_batch_runs": [ + 227445.89584921583, + 173130.19461554143, + 211012.75586735076 + ], + "qps_batch": 211012.75586735076, + "qps_batch_min": 173130.19461554143, + "qps_batch_max": 227445.89584921583, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8744, + "qps_1t_runs": [ + 4882.339302033936, + 6711.634618795114, + 6389.811079287228 + ], + "qps_1t": 6389.811079287228, + "qps_1t_min": 4882.339302033936, + "qps_1t_max": 6711.634618795114, + "median_ms": 0.14860002556815743, + "p95_ms": 0.20760000916197896, + "p99_ms": 0.35470302274916327, + "mean_ms": 0.17010466629290022, + "knob": 96, + "batch": { + "recall": 0.8744, + "qps_batch_runs": [ + 120765.65448402686, + 109461.28621737093, + 142056.55227585943 + ], + "qps_batch": 120765.65448402686, + "qps_batch_min": 109461.28621737093, + "qps_batch_max": 142056.55227585943, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.90235, + "qps_1t_runs": [ + 4219.419088777758, + 4629.238495423772, + 4853.100287566249 + ], + "qps_1t": 4629.238495423772, + "qps_1t_min": 4219.419088777758, + "qps_1t_max": 4853.100287566249, + "median_ms": 0.2001000102609396, + "p95_ms": 0.2741049916949123, + "p99_ms": 0.4297169542405758, + "mean_ms": 0.21969051659107208, + "knob": 128, + "batch": { + "recall": 0.90235, + "qps_batch_runs": [ + 92810.87001407248, + 111843.68716628088, + 72496.86452998761 + ], + "qps_batch": 92810.87001407248, + "qps_batch_min": 72496.86452998761, + "qps_batch_max": 111843.68716628088, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.93055, + "qps_1t_runs": [ + 3242.6730589140698, + 3075.502672615211, + 3206.340859890137 + ], + "qps_1t": 3206.340859890137, + "qps_1t_min": 3075.502672615211, + "qps_1t_max": 3242.6730589140698, + "median_ms": 0.29639998683705926, + "p95_ms": 0.40111502457875764, + "p99_ms": 0.8328649960458314, + "mean_ms": 0.31513988314933766, + "knob": 192, + "batch": { + "recall": 0.93055, + "qps_batch_runs": [ + 81515.53693931492, + 62786.46323704987, + 66248.1988426525 + ], + "qps_batch": 66248.1988426525, + "qps_batch_min": 62786.46323704987, + "qps_batch_max": 81515.53693931492, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9483, + "qps_1t_runs": [ + 2236.6380989326058, + 2249.8331771518483, + 2230.111117860299 + ], + "qps_1t": 2236.6380989326058, + "qps_1t_min": 2230.111117860299, + "qps_1t_max": 2249.8331771518483, + "median_ms": 0.4082999948877841, + "p95_ms": 0.5389650264987724, + "p99_ms": 1.0940319456858654, + "mean_ms": 0.44666171651139547, + "knob": 256, + "batch": { + "recall": 0.9483, + "qps_batch_runs": [ + 59144.82501820361, + 54004.13666035046, + 53987.372339161186 + ], + "qps_batch": 54004.13666035046, + "qps_batch_min": 53987.372339161186, + "qps_batch_max": 59144.82501820361, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9696, + "qps_1t_runs": [ + 1378.5129920775269, + 1380.774727899608, + 1435.2644762819937 + ], + "qps_1t": 1380.774727899608, + "qps_1t_min": 1378.5129920775269, + "qps_1t_max": 1435.2644762819937, + "median_ms": 0.6793999928049743, + "p95_ms": 0.9083049895707518, + "p99_ms": 1.7064569529611695, + "mean_ms": 0.7154620504103756, + "knob": 400, + "batch": { + "recall": 0.9696, + "qps_batch_runs": [ + 34420.268027078215, + 27177.236349409926, + 25610.621241390967 + ], + "qps_batch": 27177.236349409926, + "qps_batch_min": 25610.621241390967, + "qps_batch_max": 34420.268027078215, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.98425, + "qps_1t_runs": [ + 780.2768788632924, + 784.8356567527835, + 788.6883477062906 + ], + "qps_1t": 784.8356567527835, + "qps_1t_min": 780.2768788632924, + "qps_1t_max": 788.6883477062906, + "median_ms": 1.2322499824222177, + "p95_ms": 1.575945006334223, + "p99_ms": 2.5888749805744755, + "mean_ms": 1.2745587999816053, + "knob": 640, + "batch": { + "recall": 0.98425, + "qps_batch_runs": [ + 17514.714550991266, + 15584.80824323615, + 17015.78810035338 + ], + "qps_batch": 17015.78810035338, + "qps_batch_min": 15584.80824323615, + "qps_batch_max": 17514.714550991266, + "threads": 32 + } + } + ], + "knob_name": "efSearch", + "mem_after_sweep": { + "rss_mb": 116.3203125, + "peak_mb": 146.97265625 + }, + "ref_knob": 400, + "ref_recall_pre": 0.9696, + "mutation": { + "ref_knob": 400, + "recall_before": 0.9696, + "insert_10k": { + "seconds": 0.988941400020849, + "per_s": 10111.822601206886, + "batch": 1000 + }, + "update_10k": { + "unsupported": true + }, + "delete_10k": { + "unsupported": true + }, + "mem_after": { + "rss_mb": 129.8359375, + "peak_mb": 204.36328125 + } + }, + "saveload": { + "save_s": 0.052062700036913157, + "disk_mb": 57.1041202545166, + "load_s": 0.02949290000833571, + "first_query_after_load_ms": 1.1775000020861626, + "time_to_ready_s": 0.030670400010421872 + }, + "mem_end": { + "rss_mb": 121.05078125, + "peak_mb": 204.36328125 + }, + "status": "ok", + "finished": "2026-11-16 08:19:45", + "load_before_pct": 12.387500000000014, + "load_contaminated": false, + "wall_s": 31.09557795524597, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_hnsw__glove__100000__filter.json b/benchmarks/competitive/results/raw/faiss_hnsw__glove__100000__filter.json new file mode 100644 index 0000000..3d6a7a5 --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_hnsw__glove__100000__filter.json @@ -0,0 +1,186 @@ +{ + "system": "faiss_hnsw", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 11:46:08", + "filter": { + "build_with_metadata_s": 3.84950030001346, + "mem": { + "rss_mb": 110.53125, + "peak_mb": 146.8046875 + }, + "subsets": { + "f10": [ + { + "knob": 16, + "nq": 2000, + "recall": 0.42795, + "qps_1t": 15136.722441572838, + "qps_1t_runs": [ + 14725.924809037962, + 15819.455791392835, + 15136.722441572838 + ], + "median_ms": 0.062400009483098984, + "p95_ms": 0.09089999366551638, + "p99_ms": 0.11690201703459029, + "mean_ms": 0.06572841651116808 + }, + { + "knob": 48, + "nq": 2000, + "recall": 0.63885, + "qps_1t": 7889.832685214196, + "qps_1t_runs": [ + 7929.5066659141485, + 7538.149602062232, + 7889.832685214196 + ], + "median_ms": 0.12579996837303042, + "p95_ms": 0.17231002857442948, + "p99_ms": 0.21080103295389566, + "mean_ms": 0.12850506696850061 + }, + { + "knob": 96, + "nq": 2000, + "recall": 0.75445, + "qps_1t": 4396.9921076126675, + "qps_1t_runs": [ + 4314.771214582915, + 4407.980642087073, + 4396.9921076126675 + ], + "median_ms": 0.2294000005349517, + "p95_ms": 0.29679999279323965, + "p99_ms": 0.349700985243544, + "mean_ms": 0.2286838162690401 + }, + { + "knob": 192, + "nq": 2000, + "recall": 0.8501, + "qps_1t": 2198.3355960583854, + "qps_1t_runs": [ + 2209.443777625927, + 2198.30756849026, + 2198.3355960583854 + ], + "median_ms": 0.46309997560456395, + "p95_ms": 0.575800018850714, + "p99_ms": 0.6611119705485182, + "mean_ms": 0.4541292001570885 + }, + { + "knob": 640, + "nq": 1872, + "recall": 0.9574786324786325, + "qps_1t": 533.7559736979956, + "qps_1t_runs": [ + 544.5763527583243, + 519.7980729753436, + 533.7559736979956 + ], + "median_ms": 1.9596499914769083, + "p95_ms": 2.2726000170223415, + "p99_ms": 2.4317800067365196, + "mean_ms": 1.877876370913081 + } + ], + "f01": [ + { + "knob": 16, + "nq": 2000, + "recall": 0.15125, + "qps_1t": 17209.20111428282, + "qps_1t_runs": [ + 18250.0069504321, + 15523.575273736153, + 17209.20111428282 + ], + "median_ms": 0.057299970649182796, + "p95_ms": 0.08540000999346375, + "p99_ms": 0.10970202041789893, + "mean_ms": 0.059107033167189606 + }, + { + "knob": 48, + "nq": 2000, + "recall": 0.29555, + "qps_1t": 8272.51313010353, + "qps_1t_runs": [ + 8272.51313010353, + 7559.8903719100435, + 8736.27203433525 + ], + "median_ms": 0.12099999003112316, + "p95_ms": 0.17010501469485464, + "p99_ms": 0.21370299742557114, + "mean_ms": 0.12254153349203989 + }, + { + "knob": 96, + "nq": 2000, + "recall": 0.42335, + "qps_1t": 4567.178743169348, + "qps_1t_runs": [ + 4610.583226204444, + 4567.178743169348, + 4124.817809605718 + ], + "median_ms": 0.2243000199086964, + "p95_ms": 0.29360502085182816, + "p99_ms": 0.35770098795183003, + "mean_ms": 0.22609360007724413 + }, + { + "knob": 192, + "nq": 2000, + "recall": 0.583, + "qps_1t": 2097.7287790037176, + "qps_1t_runs": [ + 2082.618519497406, + 2097.7287790037176, + 2167.555508147719 + ], + "median_ms": 0.4835000145249069, + "p95_ms": 0.5848099710419774, + "p99_ms": 0.6378999684238806, + "mean_ms": 0.47273999990041676 + }, + { + "knob": 640, + "nq": 1501, + "recall": 0.835176548967355, + "qps_1t": 541.4554097121783, + "qps_1t_runs": [ + 541.4554097121783, + 537.8322333921916, + 547.5736231871928 + ], + "median_ms": 1.9076999742537737, + "p95_ms": 2.2440199798438694, + "p99_ms": 2.4566780216991897, + "mean_ms": 1.8441428163266933 + } + ] + } + }, + "mem_end": { + "rss_mb": 58.66015625, + "peak_mb": 146.8046875 + }, + "status": "ok", + "finished": "2026-11-16 11:46:42", + "load_before_pct": 7.721874999999998, + "load_contaminated": false, + "wall_s": 34.76664400100708, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_hnsw__sift__1000000__core+saveload.json b/benchmarks/competitive/results/raw/faiss_hnsw__sift__1000000__core+saveload.json new file mode 100644 index 0000000..2fde18a --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_hnsw__sift__1000000__core+saveload.json @@ -0,0 +1,420 @@ +{ + "system": "faiss_hnsw", + "ds": "sift", + "n": 1000000, + "d": 128, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "saveload" + ], + "started": "2026-11-16 11:31:45", + "mem_baseline": { + "rss_mb": 35.41015625, + "peak_mb": 35.41015625 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 99.66364149999572, + "index_s": 4.800036549568176e-06, + "total_s": 99.66364630003227, + "first_query_ms": 4.775700042955577, + "ready_s": 99.66842200007522 + }, + "mem_after_build": { + "rss_mb": 698.6015625, + "peak_mb": 849.3046875 + }, + "mem_after_first_query": { + "rss_mb": 698.69921875, + "peak_mb": 849.3046875 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6877, + "qps_1t_runs": [ + 15221.60368264573, + 17411.713858712134, + 17554.26899415655 + ], + "qps_1t": 17411.713858712134, + "qps_1t_min": 15221.60368264573, + "qps_1t_max": 17554.26899415655, + "median_ms": 0.05490001058205962, + "p95_ms": 0.09160499903373422, + "p99_ms": 0.16472397372126632, + "mean_ms": 0.06003163346516279, + "knob": 10, + "batch": { + "recall": 0.6877, + "qps_batch_runs": [ + 135452.71655586155, + 139613.13202342496, + 133927.9732603827 + ], + "qps_batch": 135452.71655586155, + "qps_batch_min": 133927.9732603827, + "qps_batch_max": 139613.13202342496, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.78015, + "qps_1t_runs": [ + 12996.523927629638, + 13849.993583143503, + 13831.392592896178 + ], + "qps_1t": 13831.392592896178, + "qps_1t_min": 12996.523927629638, + "qps_1t_max": 13849.993583143503, + "median_ms": 0.06969997775740921, + "p95_ms": 0.10400003520771861, + "p99_ms": 0.2012190181994815, + "mean_ms": 0.073815050078944, + "knob": 16, + "batch": { + "recall": 0.78015, + "qps_batch_runs": [ + 85890.36092934318, + 93999.53931667774, + 92819.915264173 + ], + "qps_batch": 92819.915264173, + "qps_batch_min": 85890.36092934318, + "qps_batch_max": 93999.53931667774, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8484, + "qps_1t_runs": [ + 11082.579713142077, + 12039.917093270498, + 11228.347557247738 + ], + "qps_1t": 11228.347557247738, + "qps_1t_min": 11082.579713142077, + "qps_1t_max": 12039.917093270498, + "median_ms": 0.08460000390186906, + "p95_ms": 0.1236050098668784, + "p99_ms": 0.1687040040269495, + "mean_ms": 0.08744968314810346, + "knob": 24, + "batch": { + "recall": 0.8484, + "qps_batch_runs": [ + 75110.88255893446, + 72881.78213417878, + 68304.84447250169 + ], + "qps_batch": 72881.78213417878, + "qps_batch_min": 68304.84447250169, + "qps_batch_max": 75110.88255893446, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.88915, + "qps_1t_runs": [ + 8957.322889695202, + 9247.622692760386, + 9049.540365327706 + ], + "qps_1t": 9049.540365327706, + "qps_1t_min": 8957.322889695202, + "qps_1t_max": 9247.622692760386, + "median_ms": 0.10679999832063913, + "p95_ms": 0.15270499861799183, + "p99_ms": 0.2480120252585041, + "mean_ms": 0.1100930829513042, + "knob": 32, + "batch": { + "recall": 0.88915, + "qps_batch_runs": [ + 61178.35637065269, + 57189.58922444197, + 60222.09905381038 + ], + "qps_batch": 60222.09905381038, + "qps_batch_min": 57189.58922444197, + "qps_batch_max": 61178.35637065269, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.93395, + "qps_1t_runs": [ + 6490.191817015184, + 6449.413851717736, + 7112.552572912351 + ], + "qps_1t": 6490.191817015184, + "qps_1t_min": 6449.413851717736, + "qps_1t_max": 7112.552572912351, + "median_ms": 0.1482999650761485, + "p95_ms": 0.2014049649005756, + "p99_ms": 0.2750010561430827, + "mean_ms": 0.14990933384979144, + "knob": 48, + "batch": { + "recall": 0.93395, + "qps_batch_runs": [ + 44632.79484573443, + 42181.10030571881, + 45456.92163567897 + ], + "qps_batch": 44632.79484573443, + "qps_batch_min": 42181.10030571881, + "qps_batch_max": 45456.92163567897, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.95725, + "qps_1t_runs": [ + 5242.815771186856, + 5346.925925907517, + 5201.269644431854 + ], + "qps_1t": 5242.815771186856, + "qps_1t_min": 5201.269644431854, + "qps_1t_max": 5346.925925907517, + "median_ms": 0.18960004672408104, + "p95_ms": 0.25480001932010055, + "p99_ms": 0.3242010233225301, + "mean_ms": 0.19000709987206696, + "knob": 64, + "batch": { + "recall": 0.95725, + "qps_batch_runs": [ + 34094.899722281334, + 34115.487748806205, + 32618.818094757655 + ], + "qps_batch": 34094.899722281334, + "qps_batch_min": 32618.818094757655, + "qps_batch_max": 34115.487748806205, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9778, + "qps_1t_runs": [ + 3677.5474920151423, + 3715.6895513562404, + 4040.752599524377 + ], + "qps_1t": 3715.6895513562404, + "qps_1t_min": 3677.5474920151423, + "qps_1t_max": 4040.752599524377, + "median_ms": 0.2654999843798578, + "p95_ms": 0.3440100001171232, + "p99_ms": 0.41020498028956365, + "mean_ms": 0.26284268345140543, + "knob": 96, + "batch": { + "recall": 0.9778, + "qps_batch_runs": [ + 26176.94830262621, + 24544.213953662176, + 22960.030030076927 + ], + "qps_batch": 24544.213953662176, + "qps_batch_min": 22960.030030076927, + "qps_batch_max": 26176.94830262621, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.98655, + "qps_1t_runs": [ + 2680.3439188855923, + 2621.8817786382783, + 2367.6991835381095 + ], + "qps_1t": 2621.8817786382783, + "qps_1t_min": 2367.6991835381095, + "qps_1t_max": 2680.3439188855923, + "median_ms": 0.38979999953880906, + "p95_ms": 0.5123100010678172, + "p99_ms": 0.6094059837050737, + "mean_ms": 0.39228095043411787, + "knob": 128, + "batch": { + "recall": 0.98655, + "qps_batch_runs": [ + 15782.6165510127, + 16773.317006425434, + 16650.155266836067 + ], + "qps_batch": 16650.155266836067, + "qps_batch_min": 15782.6165510127, + "qps_batch_max": 16773.317006425434, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9936, + "qps_1t_runs": [ + 1707.4400499424014, + 1789.5660063661223, + 1799.9969418573462 + ], + "qps_1t": 1789.5660063661223, + "qps_1t_min": 1707.4400499424014, + "qps_1t_max": 1799.9969418573462, + "median_ms": 0.5837000207975507, + "p95_ms": 0.7234250340843574, + "p99_ms": 0.8352020144229755, + "mean_ms": 0.5666744332702365, + "knob": 192, + "batch": { + "recall": 0.9936, + "qps_batch_runs": [ + 12020.162620381148, + 11858.471514675508, + 11390.248125827971 + ], + "qps_batch": 11858.471514675508, + "qps_batch_min": 11390.248125827971, + "qps_batch_max": 12020.162620381148, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99625, + "qps_1t_runs": [ + 1364.3275830496264, + 1384.428817197928, + 1414.5979295631594 + ], + "qps_1t": 1384.428817197928, + "qps_1t_min": 1364.3275830496264, + "qps_1t_max": 1414.5979295631594, + "median_ms": 0.742000003810972, + "p95_ms": 0.9175000013783574, + "p99_ms": 1.0287189751397823, + "mean_ms": 0.7207319996183893, + "knob": 256, + "batch": { + "recall": 0.99625, + "qps_batch_runs": [ + 8889.177294124036, + 8639.61114871948, + 8866.152134119544 + ], + "qps_batch": 8866.152134119544, + "qps_batch_min": 8639.61114871948, + "qps_batch_max": 8889.177294124036, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9984, + "qps_1t_runs": [ + 809.4912201268561, + 836.4593911104172, + 802.845927782934 + ], + "qps_1t": 809.4912201268561, + "qps_1t_min": 802.845927782934, + "qps_1t_max": 836.4593911104172, + "median_ms": 1.2682999949902296, + "p95_ms": 1.4987050031777471, + "p99_ms": 1.6465010220417753, + "mean_ms": 1.2254760501091368, + "knob": 400, + "batch": { + "recall": 0.9984, + "qps_batch_runs": [ + 5540.9536037766475, + 5685.58164945011, + 6897.1605779151405 + ], + "qps_batch": 5685.58164945011, + "qps_batch_min": 5540.9536037766475, + "qps_batch_max": 6897.1605779151405, + "threads": 32 + } + }, + { + "nq": 1572, + "recall_queries": 1572, + "recall": 0.9989821882951654, + "qps_1t_runs": [ + 482.46921135496234, + 482.44073784558555, + 488.31076602188097 + ], + "qps_1t": 482.46921135496234, + "qps_1t_min": 482.44073784558555, + "qps_1t_max": 488.31076602188097, + "median_ms": 2.1400999976322055, + "p95_ms": 2.510875026928261, + "p99_ms": 2.6561249687802047, + "mean_ms": 2.0644469246648622, + "knob": 640, + "batch": { + "recall": 0.99905, + "qps_batch_runs": [ + 3593.852284451932, + 3978.1543630467136, + 4054.1874583677513 + ], + "qps_batch": 3978.1543630467136, + "qps_batch_min": 3593.852284451932, + "qps_batch_max": 4054.1874583677513, + "threads": 32 + } + } + ], + "knob_name": "efSearch", + "mem_after_sweep": { + "rss_mb": 718.9296875, + "peak_mb": 849.3046875 + }, + "ref_knob": 64, + "ref_recall_pre": 0.95725, + "saveload": { + "save_s": 1.122972299985122, + "disk_mb": 625.8567447662354, + "load_s": 1.0980615000007674, + "first_query_after_load_ms": 0.566599948797375, + "time_to_ready_s": 1.0986280999495648 + }, + "mem_end": { + "rss_mb": 718.22265625, + "peak_mb": 1344.23828125 + }, + "status": "ok", + "finished": "2026-11-16 11:34:06", + "load_before_pct": 8.231249999999996, + "load_contaminated": false, + "wall_s": 141.98465943336487, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_hnsw__sift__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/faiss_hnsw__sift__100000__core+mutation+saveload.json new file mode 100644 index 0000000..4d4d8f8 --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_hnsw__sift__100000__core+mutation+saveload.json @@ -0,0 +1,415 @@ +{ + "system": "faiss_hnsw", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 11:08:43", + "mem_baseline": { + "rss_mb": 35.94921875, + "peak_mb": 35.94921875 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 1.9758728001033887, + "index_s": 3.300025127828121e-06, + "total_s": 1.9758761001285166, + "first_query_ms": 1.9087999826297164, + "ready_s": 1.9777849001111463, + "total_s_runs": [ + 1.9758761001285166, + 1.9990646000951529, + 2.0857574000256136 + ] + }, + "mem_after_build": { + "rss_mb": 121.0078125, + "peak_mb": 167.9765625 + }, + "mem_after_first_query": { + "rss_mb": 121.20703125, + "peak_mb": 167.9765625 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7926, + "qps_1t_runs": [ + 35531.23656587686, + 35472.44006732308, + 34556.57872236042 + ], + "qps_1t": 35472.44006732308, + "qps_1t_min": 34556.57872236042, + "qps_1t_max": 35531.23656587686, + "median_ms": 0.026799971237778664, + "p95_ms": 0.03809999907389283, + "p99_ms": 0.052900995360687396, + "mean_ms": 0.028424399827296536, + "knob": 10, + "batch": { + "recall": 0.7926, + "qps_batch_runs": [ + 579256.806981008, + 665335.991676808, + 480746.1164904281 + ], + "qps_batch": 579256.806981008, + "qps_batch_min": 480746.1164904281, + "qps_batch_max": 665335.991676808, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.87365, + "qps_1t_runs": [ + 28188.626704953174, + 27241.980895966684, + 28108.837376632495 + ], + "qps_1t": 28108.837376632495, + "qps_1t_min": 27241.980895966684, + "qps_1t_max": 28188.626704953174, + "median_ms": 0.03430002834647894, + "p95_ms": 0.047900015488266945, + "p99_ms": 0.061102999607101145, + "mean_ms": 0.03591978352051228, + "knob": 16, + "batch": { + "recall": 0.87365, + "qps_batch_runs": [ + 407672.3955752733, + 396518.56779264717, + 484954.28964742157 + ], + "qps_batch": 407672.3955752733, + "qps_batch_min": 396518.56779264717, + "qps_batch_max": 484954.28964742157, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.92575, + "qps_1t_runs": [ + 21336.997793962426, + 21006.527781138193, + 21197.331364720452 + ], + "qps_1t": 21197.331364720452, + "qps_1t_min": 21006.527781138193, + "qps_1t_max": 21336.997793962426, + "median_ms": 0.04529999569058418, + "p95_ms": 0.06290001329034567, + "p99_ms": 0.08430500165559363, + "mean_ms": 0.04721564978050689, + "knob": 24, + "batch": { + "recall": 0.92575, + "qps_batch_runs": [ + 376208.5723213932, + 289775.27956675127, + 354905.68312752625 + ], + "qps_batch": 354905.68312752625, + "qps_batch_min": 289775.27956675127, + "qps_batch_max": 376208.5723213932, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.95325, + "qps_1t_runs": [ + 17367.88044096626, + 16936.98087835346, + 16705.911637072553 + ], + "qps_1t": 16936.98087835346, + "qps_1t_min": 16705.911637072553, + "qps_1t_max": 17367.88044096626, + "median_ms": 0.055500015150755644, + "p95_ms": 0.07790501113049687, + "p99_ms": 0.12070598779246224, + "mean_ms": 0.05882633325139371, + "knob": 32, + "batch": { + "recall": 0.95325, + "qps_batch_runs": [ + 338833.7337702731, + 328882.4557782537, + 194992.5905542895 + ], + "qps_batch": 328882.4557782537, + "qps_batch_min": 194992.5905542895, + "qps_batch_max": 338833.7337702731, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.97775, + "qps_1t_runs": [ + 12452.920234205405, + 12092.541804703415, + 12677.283845810938 + ], + "qps_1t": 12452.920234205405, + "qps_1t_min": 12092.541804703415, + "qps_1t_max": 12677.283845810938, + "median_ms": 0.07830001413822174, + "p95_ms": 0.10891001438722019, + "p99_ms": 0.14870496874209505, + "mean_ms": 0.08062643330777064, + "knob": 48, + "batch": { + "recall": 0.97775, + "qps_batch_runs": [ + 182715.1471712587, + 214167.15796670993, + 191734.33231475647 + ], + "qps_batch": 191734.33231475647, + "qps_batch_min": 182715.1471712587, + "qps_batch_max": 214167.15796670993, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.98885, + "qps_1t_runs": [ + 9172.05680832665, + 9527.325560037729, + 7962.528332573384 + ], + "qps_1t": 9172.05680832665, + "qps_1t_min": 7962.528332573384, + "qps_1t_max": 9527.325560037729, + "median_ms": 0.10090001160278916, + "p95_ms": 0.14780501951463523, + "p99_ms": 0.2626010205131024, + "mean_ms": 0.11319209993234836, + "knob": 64, + "batch": { + "recall": 0.98885, + "qps_batch_runs": [ + 140577.7751604037, + 153848.52034566354, + 126686.51459788925 + ], + "qps_batch": 140577.7751604037, + "qps_batch_min": 126686.51459788925, + "qps_batch_max": 153848.52034566354, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99575, + "qps_1t_runs": [ + 5876.443678501833, + 6013.169443623567, + 6247.579067269414 + ], + "qps_1t": 6013.169443623567, + "qps_1t_min": 5876.443678501833, + "qps_1t_max": 6247.579067269414, + "median_ms": 0.14990003546699882, + "p95_ms": 0.22800998995080593, + "p99_ms": 0.40872402663808366, + "mean_ms": 0.16551153309410438, + "knob": 96, + "batch": { + "recall": 0.99575, + "qps_batch_runs": [ + 98180.22949951982, + 108692.69854053286, + 99527.74065783831 + ], + "qps_batch": 99527.74065783831, + "qps_batch_min": 98180.22949951982, + "qps_batch_max": 108692.69854053286, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99815, + "qps_1t_runs": [ + 4910.015393694244, + 4996.54362960391, + 5057.248053808577 + ], + "qps_1t": 4996.54362960391, + "qps_1t_min": 4910.015393694244, + "qps_1t_max": 5057.248053808577, + "median_ms": 0.19130000146105886, + "p95_ms": 0.26441003137733793, + "p99_ms": 0.39741199929267196, + "mean_ms": 0.20051323315904787, + "knob": 128, + "batch": { + "recall": 0.99815, + "qps_batch_runs": [ + 90957.87754942862, + 90236.01242785207, + 94692.48616042655 + ], + "qps_batch": 90957.87754942862, + "qps_batch_min": 90236.01242785207, + "qps_batch_max": 94692.48616042655, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9992, + "qps_1t_runs": [ + 3317.5286232630474, + 3279.131472997281, + 3385.8346492080973 + ], + "qps_1t": 3317.5286232630474, + "qps_1t_min": 3279.131472997281, + "qps_1t_max": 3385.8346492080973, + "median_ms": 0.2888999879360199, + "p95_ms": 0.39281002245843416, + "p99_ms": 0.5657670000800877, + "mean_ms": 0.30057871689011034, + "knob": 192, + "batch": { + "recall": 0.9992, + "qps_batch_runs": [ + 63597.04910850862, + 70887.54756771697, + 70792.95189234504 + ], + "qps_batch": 70792.95189234504, + "qps_batch_min": 63597.04910850862, + "qps_batch_max": 70887.54756771697, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99965, + "qps_1t_runs": [ + 2662.9708237718537, + 2528.408561910027, + 2456.427580322251 + ], + "qps_1t": 2528.408561910027, + "qps_1t_min": 2456.427580322251, + "qps_1t_max": 2662.9708237718537, + "median_ms": 0.3800999838858843, + "p95_ms": 0.5081150273326784, + "p99_ms": 0.8009799866704288, + "mean_ms": 0.3927071333261362, + "knob": 256, + "batch": { + "recall": 0.99965, + "qps_batch_runs": [ + 53417.09134018126, + 44524.58869696301, + 48638.48714060023 + ], + "qps_batch": 48638.48714060023, + "qps_batch_min": 44524.58869696301, + "qps_batch_max": 53417.09134018126, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9999, + "qps_1t_runs": [ + 1537.3599611503926, + 1490.9581216083075, + 1551.6797010265384 + ], + "qps_1t": 1537.3599611503926, + "qps_1t_min": 1490.9581216083075, + "qps_1t_max": 1551.6797010265384, + "median_ms": 0.6274500046856701, + "p95_ms": 0.8356150181498379, + "p99_ms": 1.329462994472125, + "mean_ms": 0.6552127666557984, + "knob": 400, + "batch": { + "recall": 0.9999, + "qps_batch_runs": [ + 31586.24545067845, + 29965.958671634555, + 26070.45280001906 + ], + "qps_batch": 29965.958671634555, + "qps_batch_min": 26070.45280001906, + "qps_batch_max": 31586.24545067845, + "threads": 32 + } + } + ], + "knob_name": "efSearch", + "mem_after_sweep": { + "rss_mb": 126.41796875, + "peak_mb": 167.9765625 + }, + "ref_knob": 32, + "ref_recall_pre": 0.95325, + "mutation": { + "ref_knob": 32, + "recall_before": 0.95325, + "insert_10k": { + "seconds": 0.9947571000084281, + "per_s": 10052.705328683027, + "batch": 1000 + }, + "update_10k": { + "unsupported": true + }, + "delete_10k": { + "unsupported": true + }, + "mem_after": { + "rss_mb": 143.51953125, + "peak_mb": 235.96875 + } + }, + "saveload": { + "save_s": 0.043917100003454834, + "disk_mb": 68.8533878326416, + "load_s": 0.039114899991545826, + "first_query_after_load_ms": 0.340400030836463, + "time_to_ready_s": 0.03945530002238229 + }, + "mem_end": { + "rss_mb": 132.15625, + "peak_mb": 235.96875 + }, + "status": "ok", + "finished": "2026-11-16 11:09:04", + "load_before_pct": 8.315625000000004, + "load_contaminated": false, + "wall_s": 21.31098699569702, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_hnsw__text384__1000000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/faiss_hnsw__text384__1000000__core+mutation+saveload.json new file mode 100644 index 0000000..00c3896 --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_hnsw__text384__1000000__core+mutation+saveload.json @@ -0,0 +1,440 @@ +{ + "system": "faiss_hnsw", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 09:18:47", + "mem_baseline": { + "rss_mb": 37.30078125, + "peak_mb": 37.30078125 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 220.9135291997809, + "index_s": 5.800044164061546e-06, + "total_s": 220.91353499982506, + "first_query_ms": 3.805199987255037, + "ready_s": 220.91734019981232 + }, + "mem_after_build": { + "rss_mb": 1677.4140625, + "peak_mb": 2205.703125 + }, + "mem_after_first_query": { + "rss_mb": 1677.51171875, + "peak_mb": 2205.703125 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.56475, + "qps_1t_runs": [ + 10192.261730123666, + 11088.195530166031, + 11826.237086049165 + ], + "qps_1t": 11088.195530166031, + "qps_1t_min": 10192.261730123666, + "qps_1t_max": 11826.237086049165, + "median_ms": 0.08379999781027436, + "p95_ms": 0.1366999960737303, + "p99_ms": 0.17521299538202612, + "mean_ms": 0.09095246663006644, + "knob": 10, + "batch": { + "recall": 0.56475, + "qps_batch_runs": [ + 44237.70688109123, + 44630.40445190999, + 44533.11482424754 + ], + "qps_batch": 44533.11482424754, + "qps_batch_min": 44237.70688109123, + "qps_batch_max": 44630.40445190999, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.67145, + "qps_1t_runs": [ + 8336.70622747123, + 8861.622680781658, + 8785.690236420953 + ], + "qps_1t": 8785.690236420953, + "qps_1t_min": 8336.70622747123, + "qps_1t_max": 8861.622680781658, + "median_ms": 0.10959996143355966, + "p95_ms": 0.16750499489717188, + "p99_ms": 0.2025009709177539, + "mean_ms": 0.11553968316487347, + "knob": 16, + "batch": { + "recall": 0.67145, + "qps_batch_runs": [ + 31232.57808054674, + 30551.749328506827, + 31149.97938836586 + ], + "qps_batch": 31149.97938836586, + "qps_batch_min": 30551.749328506827, + "qps_batch_max": 31232.57808054674, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7482, + "qps_1t_runs": [ + 6565.699329251801, + 5968.191327827954, + 6666.446668629778 + ], + "qps_1t": 6565.699329251801, + "qps_1t_min": 5968.191327827954, + "qps_1t_max": 6666.446668629778, + "median_ms": 0.1463499793317169, + "p95_ms": 0.224510018597357, + "p99_ms": 0.28600202349480247, + "mean_ms": 0.1566222001662633, + "knob": 24, + "batch": { + "recall": 0.7482, + "qps_batch_runs": [ + 24787.81629498942, + 23401.392395765477, + 24366.649848515193 + ], + "qps_batch": 24366.649848515193, + "qps_batch_min": 23401.392395765477, + "qps_batch_max": 24787.81629498942, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.79355, + "qps_1t_runs": [ + 5567.81554088785, + 5568.8790942069445, + 5413.383555844972 + ], + "qps_1t": 5567.81554088785, + "qps_1t_min": 5413.383555844972, + "qps_1t_max": 5568.8790942069445, + "median_ms": 0.17315000877715647, + "p95_ms": 0.24710501602385196, + "p99_ms": 0.3212120011448863, + "mean_ms": 0.18130011716857553, + "knob": 32, + "batch": { + "recall": 0.79355, + "qps_batch_runs": [ + 19412.631989268186, + 19416.702834836407, + 19348.759455371794 + ], + "qps_batch": 19412.631989268186, + "qps_batch_min": 19348.759455371794, + "qps_batch_max": 19416.702834836407, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8516, + "qps_1t_runs": [ + 3891.822891329091, + 4031.562294452521, + 4186.749983768726 + ], + "qps_1t": 4031.562294452521, + "qps_1t_min": 3891.822891329091, + "qps_1t_max": 4186.749983768726, + "median_ms": 0.23800000781193376, + "p95_ms": 0.33120496082119644, + "p99_ms": 0.40860502573195856, + "mean_ms": 0.24794684998535862, + "knob": 48, + "batch": { + "recall": 0.8516, + "qps_batch_runs": [ + 14630.941807717025, + 14764.048361847717, + 14767.907569448676 + ], + "qps_batch": 14764.048361847717, + "qps_batch_min": 14630.941807717025, + "qps_batch_max": 14767.907569448676, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8875, + "qps_1t_runs": [ + 3259.3328942954463, + 3139.5611861162934, + 3290.2773097408053 + ], + "qps_1t": 3259.3328942954463, + "qps_1t_min": 3139.5611861162934, + "qps_1t_max": 3290.2773097408053, + "median_ms": 0.29880000511184335, + "p95_ms": 0.4010000266134739, + "p99_ms": 0.5098089820239695, + "mean_ms": 0.3097509501676541, + "knob": 64, + "batch": { + "recall": 0.8875, + "qps_batch_runs": [ + 11896.42732243021, + 11597.255158434835, + 11569.588181406783 + ], + "qps_batch": 11597.255158434835, + "qps_batch_min": 11569.588181406783, + "qps_batch_max": 11896.42732243021, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9255, + "qps_1t_runs": [ + 2124.4096816103493, + 2238.0321778978687, + 2263.3523633250843 + ], + "qps_1t": 2238.0321778978687, + "qps_1t_min": 2124.4096816103493, + "qps_1t_max": 2263.3523633250843, + "median_ms": 0.42630001553334296, + "p95_ms": 0.5663250049110503, + "p99_ms": 0.7917200319934663, + "mean_ms": 0.453120866654596, + "knob": 96, + "batch": { + "recall": 0.9255, + "qps_batch_runs": [ + 8492.082195002055, + 8358.869680357435, + 7487.19876206358 + ], + "qps_batch": 8358.869680357435, + "qps_batch_min": 7487.19876206358, + "qps_batch_max": 8492.082195002055, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9445, + "qps_1t_runs": [ + 1665.2790724754075, + 1742.5921323283403, + 1720.937430948472 + ], + "qps_1t": 1720.937430948472, + "qps_1t_min": 1665.2790724754075, + "qps_1t_max": 1742.5921323283403, + "median_ms": 0.5580500001087785, + "p95_ms": 0.7169050426455215, + "p99_ms": 0.9819570259423939, + "mean_ms": 0.5851454501777577, + "knob": 128, + "batch": { + "recall": 0.9445, + "qps_batch_runs": [ + 6658.22847238178, + 6312.125118810746, + 6399.639573130844 + ], + "qps_batch": 6399.639573130844, + "qps_batch_min": 6312.125118810746, + "qps_batch_max": 6658.22847238178, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9648, + "qps_1t_runs": [ + 1232.91989873788, + 1191.4535375292921, + 1283.9168301239383 + ], + "qps_1t": 1232.91989873788, + "qps_1t_min": 1191.4535375292921, + "qps_1t_max": 1283.9168301239383, + "median_ms": 0.7961000083014369, + "p95_ms": 0.979115001973696, + "p99_ms": 1.2354869773844277, + "mean_ms": 0.8097534330154303, + "knob": 192, + "batch": { + "recall": 0.9648, + "qps_batch_runs": [ + 4678.933895546764, + 4329.592853323451, + 4243.057615628609 + ], + "qps_batch": 4329.592853323451, + "qps_batch_min": 4243.057615628609, + "qps_batch_max": 4678.933895546764, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9742, + "qps_1t_runs": [ + 852.976347304822, + 925.788772123046, + 912.5564250682239 + ], + "qps_1t": 912.5564250682239, + "qps_1t_min": 852.976347304822, + "qps_1t_max": 925.788772123046, + "median_ms": 1.0916500177700073, + "p95_ms": 1.429825002560392, + "p99_ms": 1.8939099856652322, + "mean_ms": 1.1161160335953657, + "knob": 256, + "batch": { + "recall": 0.9742, + "qps_batch_runs": [ + 3591.7946888631536, + 3634.7663220055592, + 3621.346242645393 + ], + "qps_batch": 3621.346242645393, + "qps_batch_min": 3591.7946888631536, + "qps_batch_max": 3634.7663220055592, + "threads": 32 + } + }, + { + "nq": 1626, + "recall_queries": 1626, + "recall": 0.9843173431734318, + "qps_1t_runs": [ + 602.3082854523135, + 593.8931567535175, + 578.4804627686716 + ], + "qps_1t": 593.8931567535175, + "qps_1t_min": 578.4804627686716, + "qps_1t_max": 602.3082854523135, + "median_ms": 1.654949999647215, + "p95_ms": 2.1025050024036314, + "p99_ms": 3.031699963030401, + "mean_ms": 1.6909169124860535, + "knob": 400, + "batch": { + "recall": 0.98445, + "qps_batch_runs": [ + 2356.491438736329, + 2369.8403687891728, + 2364.0044390906096 + ], + "qps_batch": 2364.0044390906096, + "qps_batch_min": 2356.491438736329, + "qps_batch_max": 2369.8403687891728, + "threads": 32 + } + }, + { + "nq": 1086, + "recall_queries": 1086, + "recall": 0.9913443830570903, + "qps_1t_runs": [ + 376.68850383676767, + 369.90787830462375, + 371.48460884239967 + ], + "qps_1t": 371.48460884239967, + "qps_1t_min": 369.90787830462375, + "qps_1t_max": 376.68850383676767, + "median_ms": 2.6522499974817038, + "p95_ms": 3.1428549991687746, + "p99_ms": 3.9103779970900914, + "mean_ms": 2.6833302326715667, + "knob": 640, + "batch": { + "recall": 0.9909, + "qps_batch_runs": [ + 1590.1997895392503, + 1596.7146000049204, + 1471.514098992545 + ], + "qps_batch": 1590.1997895392503, + "qps_batch_min": 1471.514098992545, + "qps_batch_max": 1596.7146000049204, + "threads": 32 + } + } + ], + "knob_name": "efSearch", + "mem_after_sweep": { + "rss_mb": 1698.67578125, + "peak_mb": 2205.703125 + }, + "ref_knob": 192, + "ref_recall_pre": 0.9648, + "mutation": { + "ref_knob": 192, + "recall_before": 0.9648, + "insert_10k": { + "seconds": 4.320585099980235, + "per_s": 2314.5013391926354, + "batch": 1000 + }, + "update_10k": { + "unsupported": true + }, + "delete_10k": { + "unsupported": true + }, + "mem_after": { + "rss_mb": 1751.83203125, + "peak_mb": 2205.703125 + } + }, + "saveload": { + "save_s": 0.9943152000196278, + "disk_mb": 1618.4442081451416, + "load_s": 0.8049297000397928, + "first_query_after_load_ms": 1.2053000391460955, + "time_to_ready_s": 0.8061350000789389 + }, + "mem_end": { + "rss_mb": 1720.69140625, + "peak_mb": 3341.18359375 + }, + "status": "ok", + "finished": "2026-11-16 09:23:30", + "load_before_pct": 15.743750000000004, + "load_contaminated": false, + "wall_s": 283.74309730529785, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_hnsw__text384__1000000__filter.json b/benchmarks/competitive/results/raw/faiss_hnsw__text384__1000000__filter.json new file mode 100644 index 0000000..7d54ab3 --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_hnsw__text384__1000000__filter.json @@ -0,0 +1,186 @@ +{ + "system": "faiss_hnsw", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 11:49:02", + "filter": { + "build_with_metadata_s": 268.1877243000781, + "mem": { + "rss_mb": 1678.1875, + "peak_mb": 2206.24609375 + }, + "subsets": { + "f10": [ + { + "knob": 16, + "nq": 2000, + "recall": 0.44155, + "qps_1t": 5466.842380570978, + "qps_1t_runs": [ + 5466.842380570978, + 5953.92436367418, + 4664.952613533861 + ], + "median_ms": 0.16110000433400273, + "p95_ms": 0.2903049986343831, + "p99_ms": 0.4365100053837547, + "mean_ms": 0.18841394981912649 + }, + { + "knob": 48, + "nq": 2000, + "recall": 0.6739, + "qps_1t": 2877.110826475339, + "qps_1t_runs": [ + 2870.8482890137743, + 2877.110826475339, + 2918.275728731884 + ], + "median_ms": 0.3234999894630164, + "p95_ms": 0.5018999829189852, + "p99_ms": 0.7591719599440709, + "mean_ms": 0.3461893667408731 + }, + { + "knob": 96, + "nq": 2000, + "recall": 0.8077, + "qps_1t": 1695.5023579818276, + "qps_1t_runs": [ + 1699.9336612510183, + 1695.5023579818276, + 1678.5396781116997 + ], + "median_ms": 0.5709000397473574, + "p95_ms": 0.7965450116898866, + "p99_ms": 1.0679200402228166, + "mean_ms": 0.5912699664089208 + }, + { + "knob": 192, + "nq": 2000, + "recall": 0.89915, + "qps_1t": 874.4338363345231, + "qps_1t_runs": [ + 874.4338363345231, + 846.3365538837717, + 876.6589896918307 + ], + "median_ms": 1.1131999781355262, + "p95_ms": 1.5285150526324287, + "p99_ms": 2.5221679959213374, + "mean_ms": 1.1552848501693613 + }, + { + "knob": 640, + "nq": 776, + "recall": 0.9730670103092783, + "qps_1t": 266.70005458656505, + "qps_1t_runs": [ + 266.70005458656505, + 267.1959442475345, + 255.4049497183635 + ], + "median_ms": 3.818400000454858, + "p95_ms": 4.594715003622696, + "p99_ms": 5.634321011020805, + "mean_ms": 3.8024844065479795 + } + ], + "f01": [ + { + "knob": 16, + "nq": 2000, + "recall": 0.1634, + "qps_1t": 6291.692266024276, + "qps_1t_runs": [ + 6291.692266024276, + 6300.593752965803, + 6159.548938791261 + ], + "median_ms": 0.1503999810665846, + "p95_ms": 0.24009996850509197, + "p99_ms": 0.30784095753915697, + "mean_ms": 0.1600015004320691 + }, + { + "knob": 48, + "nq": 2000, + "recall": 0.314, + "qps_1t": 2850.207893216705, + "qps_1t_runs": [ + 2889.053276076182, + 2850.207893216705, + 2846.8223126044777 + ], + "median_ms": 0.33660000190138817, + "p95_ms": 0.4846100200666115, + "p99_ms": 0.6095239846035844, + "mean_ms": 0.3494182000479971 + }, + { + "knob": 96, + "nq": 2000, + "recall": 0.4525, + "qps_1t": 1677.5708767278245, + "qps_1t_runs": [ + 1677.5708767278245, + 1671.4556662592422, + 1681.7205881031446 + ], + "median_ms": 0.5917000235058367, + "p95_ms": 0.763929996173829, + "p99_ms": 0.8816130191553387, + "mean_ms": 0.5963366667274386 + }, + { + "knob": 192, + "nq": 2000, + "recall": 0.61315, + "qps_1t": 882.2221907440756, + "qps_1t_runs": [ + 882.2221907440756, + 862.7187933461809, + 906.4746583933228 + ], + "median_ms": 1.1019499797839671, + "p95_ms": 1.42611003248021, + "p99_ms": 1.838933972758246, + "mean_ms": 1.1319341167400125 + }, + { + "knob": 640, + "nq": 765, + "recall": 0.8538562091503268, + "qps_1t": 274.6660034651382, + "qps_1t_runs": [ + 266.67058946641896, + 274.6660034651382, + 280.1825561204588 + ], + "median_ms": 3.65949998376891, + "p95_ms": 4.610290005803108, + "p99_ms": 5.334977973252532, + "mean_ms": 3.6532772976848293 + } + ] + } + }, + "mem_end": { + "rss_mb": 59.9765625, + "peak_mb": 2206.24609375 + }, + "status": "ok", + "finished": "2026-11-16 11:54:16", + "load_before_pct": 8.909374999999995, + "load_contaminated": false, + "wall_s": 314.92415142059326, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_hnsw__text384__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/faiss_hnsw__text384__100000__core+mutation+saveload.json new file mode 100644 index 0000000..9b0d2eb --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_hnsw__text384__100000__core+mutation+saveload.json @@ -0,0 +1,445 @@ +{ + "system": "faiss_hnsw", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 07:54:21", + "mem_baseline": { + "rss_mb": 37.3515625, + "peak_mb": 37.3515625 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 8.898113100090995, + "index_s": 3.400025889277458e-06, + "total_s": 8.898116500116885, + "first_query_ms": 3.1128000118769705, + "ready_s": 8.901229300128762, + "total_s_runs": [ + 8.898116500116885, + 8.407066700048745, + 8.672152799903415 + ] + }, + "mem_after_build": { + "rss_mb": 221.4609375, + "peak_mb": 365.76953125 + }, + "mem_after_first_query": { + "rss_mb": 221.65625, + "peak_mb": 365.76953125 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.63615, + "qps_1t_runs": [ + 15712.745621311362, + 15199.693601217044, + 17056.58023085163 + ], + "qps_1t": 15712.745621311362, + "qps_1t_min": 15199.693601217044, + "qps_1t_max": 17056.58023085163, + "median_ms": 0.058399979025125504, + "p95_ms": 0.08829997386783361, + "p99_ms": 0.11770196550060068, + "mean_ms": 0.06268726679263636, + "knob": 10, + "batch": { + "recall": 0.63615, + "qps_batch_runs": [ + 90515.34926644799, + 97671.98825100507, + 96808.70128416846 + ], + "qps_batch": 96808.70128416846, + "qps_batch_min": 90515.34926644799, + "qps_batch_max": 97671.98825100507, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.73145, + "qps_1t_runs": [ + 12778.55657739224, + 12881.062004773192, + 12783.195732363236 + ], + "qps_1t": 12783.195732363236, + "qps_1t_min": 12778.55657739224, + "qps_1t_max": 12881.062004773192, + "median_ms": 0.07349997758865356, + "p95_ms": 0.10460503690410407, + "p99_ms": 0.13550298404879874, + "mean_ms": 0.07803905007313006, + "knob": 16, + "batch": { + "recall": 0.73145, + "qps_batch_runs": [ + 72568.677135575, + 66867.0449477335, + 70071.68336399632 + ], + "qps_batch": 70071.68336399632, + "qps_batch_min": 66867.0449477335, + "qps_batch_max": 72568.677135575, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8014, + "qps_1t_runs": [ + 9414.094982830675, + 9092.02076580962, + 9647.909209724548 + ], + "qps_1t": 9414.094982830675, + "qps_1t_min": 9092.02076580962, + "qps_1t_max": 9647.909209724548, + "median_ms": 0.0971999834291637, + "p95_ms": 0.13510503631550821, + "p99_ms": 0.21710802335292118, + "mean_ms": 0.10661988338688388, + "knob": 24, + "batch": { + "recall": 0.8014, + "qps_batch_runs": [ + 50166.42710141613, + 52264.21652425431, + 50335.359347950674 + ], + "qps_batch": 50335.359347950674, + "qps_batch_min": 50166.42710141613, + "qps_batch_max": 52264.21652425431, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.84925, + "qps_1t_runs": [ + 7577.838667905151, + 7879.373105523631, + 8242.793130391423 + ], + "qps_1t": 7879.373105523631, + "qps_1t_min": 7577.838667905151, + "qps_1t_max": 8242.793130391423, + "median_ms": 0.12129999231547117, + "p95_ms": 0.1642999704927206, + "p99_ms": 0.2095230232225736, + "mean_ms": 0.12673183292887794, + "knob": 32, + "batch": { + "recall": 0.84925, + "qps_batch_runs": [ + 42663.9361539615, + 44864.47565823431, + 45496.21587440867 + ], + "qps_batch": 44864.47565823431, + "qps_batch_min": 42663.9361539615, + "qps_batch_max": 45496.21587440867, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.90045, + "qps_1t_runs": [ + 5918.579847600545, + 5913.146525522864, + 5762.908202399839 + ], + "qps_1t": 5913.146525522864, + "qps_1t_min": 5762.908202399839, + "qps_1t_max": 5918.579847600545, + "median_ms": 0.16529997810721397, + "p95_ms": 0.20900001982226968, + "p99_ms": 0.2608020068146289, + "mean_ms": 0.1705325501097832, + "knob": 48, + "batch": { + "recall": 0.90045, + "qps_batch_runs": [ + 30514.87753081251, + 30793.439167646644, + 30746.714733704695 + ], + "qps_batch": 30746.714733704695, + "qps_batch_min": 30514.87753081251, + "qps_batch_max": 30793.439167646644, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9279, + "qps_1t_runs": [ + 3995.9880323559482, + 4554.424680026955, + 4703.849190498367 + ], + "qps_1t": 4554.424680026955, + "qps_1t_min": 3995.9880323559482, + "qps_1t_max": 4703.849190498367, + "median_ms": 0.21744999685324728, + "p95_ms": 0.304805047926493, + "p99_ms": 0.39480297942645853, + "mean_ms": 0.22746984999200018, + "knob": 64, + "batch": { + "recall": 0.9279, + "qps_batch_runs": [ + 24861.95399342435, + 25427.014940882702, + 24381.205018600456 + ], + "qps_batch": 24861.95399342435, + "qps_batch_min": 24381.205018600456, + "qps_batch_max": 25427.014940882702, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.95695, + "qps_1t_runs": [ + 3312.498820793018, + 3103.066172948751, + 3272.6775254334375 + ], + "qps_1t": 3272.6775254334375, + "qps_1t_min": 3103.066172948751, + "qps_1t_max": 3312.498820793018, + "median_ms": 0.29639998683705926, + "p95_ms": 0.37670496094506234, + "p99_ms": 0.5525669583585128, + "mean_ms": 0.3099029999769603, + "knob": 96, + "batch": { + "recall": 0.95695, + "qps_batch_runs": [ + 18590.660606781523, + 18644.489660895728, + 18068.185716811222 + ], + "qps_batch": 18590.660606781523, + "qps_batch_min": 18068.185716811222, + "qps_batch_max": 18644.489660895728, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9714, + "qps_1t_runs": [ + 2277.247095234753, + 2431.755218577571, + 2369.900745145576 + ], + "qps_1t": 2369.900745145576, + "qps_1t_min": 2277.247095234753, + "qps_1t_max": 2431.755218577571, + "median_ms": 0.38599997060373425, + "p95_ms": 0.5194200522964821, + "p99_ms": 0.9286969696404433, + "mean_ms": 0.42410363312228583, + "knob": 128, + "batch": { + "recall": 0.9714, + "qps_batch_runs": [ + 14389.545131548419, + 14274.223535869021, + 13814.692479831621 + ], + "qps_batch": 14274.223535869021, + "qps_batch_min": 13814.692479831621, + "qps_batch_max": 14389.545131548419, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9842, + "qps_1t_runs": [ + 1700.5611198062475, + 1641.858591965703, + 1724.1346603584495 + ], + "qps_1t": 1700.5611198062475, + "qps_1t_min": 1641.858591965703, + "qps_1t_max": 1724.1346603584495, + "median_ms": 0.5584499740507454, + "p95_ms": 0.6910200201673434, + "p99_ms": 1.0444640362402438, + "mean_ms": 0.592369383336821, + "knob": 192, + "batch": { + "recall": 0.9842, + "qps_batch_runs": [ + 10424.827351795268, + 10206.466613284618, + 9914.345015554363 + ], + "qps_batch": 10206.466613284618, + "qps_batch_min": 9914.345015554363, + "qps_batch_max": 10424.827351795268, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99085, + "qps_1t_runs": [ + 1281.855964406836, + 1317.9759256987318, + 1248.615909055167 + ], + "qps_1t": 1281.855964406836, + "qps_1t_min": 1248.615909055167, + "qps_1t_max": 1317.9759256987318, + "median_ms": 0.7422000053338706, + "p95_ms": 0.967204972403124, + "p99_ms": 1.591091007576326, + "mean_ms": 0.7799149499624036, + "knob": 256, + "batch": { + "recall": 0.99085, + "qps_batch_runs": [ + 7492.407382294504, + 7844.496967176118, + 8105.389515179969 + ], + "qps_batch": 7844.496967176118, + "qps_batch_min": 7492.407382294504, + "qps_batch_max": 8105.389515179969, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9958, + "qps_1t_runs": [ + 780.9628438166951, + 838.6397214042684, + 823.4041092042909 + ], + "qps_1t": 823.4041092042909, + "qps_1t_min": 780.9628438166951, + "qps_1t_max": 838.6397214042684, + "median_ms": 1.1656999704428017, + "p95_ms": 1.496334985131398, + "p99_ms": 2.4191169685218488, + "mean_ms": 1.2291160997798822, + "knob": 400, + "batch": { + "recall": 0.9958, + "qps_batch_runs": [ + 5363.90041428864, + 5285.262810873027, + 4851.04855408874 + ], + "qps_batch": 5285.262810873027, + "qps_batch_min": 4851.04855408874, + "qps_batch_max": 5363.90041428864, + "threads": 32 + } + }, + { + "nq": 1585, + "recall_queries": 1585, + "recall": 0.9982965299684543, + "qps_1t_runs": [ + 515.1336179804398, + 508.02444649877924, + 492.43010663020107 + ], + "qps_1t": 508.02444649877924, + "qps_1t_min": 492.43010663020107, + "qps_1t_max": 515.1336179804398, + "median_ms": 1.8851000349968672, + "p95_ms": 2.443619980476797, + "p99_ms": 4.007778021041305, + "mean_ms": 1.980132723523874, + "knob": 640, + "batch": { + "recall": 0.99855, + "qps_batch_runs": [ + 3847.067525050958, + 3804.6900415597056, + 3885.931587008845 + ], + "qps_batch": 3847.067525050958, + "qps_batch_min": 3804.6900415597056, + "qps_batch_max": 3885.931587008845, + "threads": 32 + } + } + ], + "knob_name": "efSearch", + "mem_after_sweep": { + "rss_mb": 226.6875, + "peak_mb": 365.76953125 + }, + "ref_knob": 96, + "ref_recall_pre": 0.95695, + "mutation": { + "ref_knob": 96, + "recall_before": 0.95695, + "insert_10k": { + "seconds": 2.837573600001633, + "per_s": 3524.1376646562558, + "batch": 1000 + }, + "update_10k": { + "unsupported": true + }, + "delete_10k": { + "unsupported": true + }, + "mem_after": { + "rss_mb": 271.62109375, + "peak_mb": 531.39453125 + } + }, + "saveload": { + "save_s": 0.13437949999934062, + "disk_mb": 176.2752628326416, + "load_s": 0.07410839997464791, + "first_query_after_load_ms": 0.4959999932907522, + "time_to_ready_s": 0.07460439996793866 + }, + "mem_end": { + "rss_mb": 240.05078125, + "peak_mb": 531.39453125 + }, + "status": "ok", + "finished": "2026-11-16 07:55:31", + "load_before_pct": 7.343750000000005, + "load_contaminated": false, + "wall_s": 70.31420993804932, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_hnsw__text384__100000__filter.json b/benchmarks/competitive/results/raw/faiss_hnsw__text384__100000__filter.json new file mode 100644 index 0000000..1f0b358 --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_hnsw__text384__100000__filter.json @@ -0,0 +1,186 @@ +{ + "system": "faiss_hnsw", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 11:44:08", + "filter": { + "build_with_metadata_s": 13.72953139996389, + "mem": { + "rss_mb": 221.21875, + "peak_mb": 365.46875 + }, + "subsets": { + "f10": [ + { + "knob": 16, + "nq": 2000, + "recall": 0.49035, + "qps_1t": 9177.78514624439, + "qps_1t_runs": [ + 8415.447028125576, + 9177.78514624439, + 9200.334540849128 + ], + "median_ms": 0.10820000898092985, + "p95_ms": 0.15601002087350938, + "p99_ms": 0.21310405049007394, + "mean_ms": 0.11215985017285372 + }, + { + "knob": 48, + "nq": 2000, + "recall": 0.7286, + "qps_1t": 4095.5240055168656, + "qps_1t_runs": [ + 4311.129940938423, + 3866.2965105164344, + 4095.5240055168656 + ], + "median_ms": 0.24279998615384102, + "p95_ms": 0.3362099989317358, + "p99_ms": 0.42862000409513756, + "mean_ms": 0.24492406644276343 + }, + { + "knob": 96, + "nq": 2000, + "recall": 0.85835, + "qps_1t": 2346.184168661592, + "qps_1t_runs": [ + 2350.0958986093956, + 2346.184168661592, + 2327.5741798904346 + ], + "median_ms": 0.43130002450197935, + "p95_ms": 0.5379999667638913, + "p99_ms": 0.6185499951243412, + "mean_ms": 0.42712346655510675 + }, + { + "knob": 192, + "nq": 2000, + "recall": 0.9435, + "qps_1t": 1279.2286969116685, + "qps_1t_runs": [ + 1292.1180423004619, + 1279.2286969116685, + 1244.0417399722471 + ], + "median_ms": 0.7985500269569457, + "p95_ms": 1.0106150089995936, + "p99_ms": 1.1330060119507837, + "mean_ms": 0.7864918997123217 + }, + { + "knob": 640, + "nq": 1009, + "recall": 0.9924677898909812, + "qps_1t": 376.1127896790685, + "qps_1t_runs": [ + 384.2244376254349, + 376.1127896790685, + 361.92478129769273 + ], + "median_ms": 2.741199976298958, + "p95_ms": 3.2588100177235897, + "p99_ms": 3.5108140157535668, + "mean_ms": 2.6748091509940455 + } + ], + "f01": [ + { + "knob": 16, + "nq": 2000, + "recall": 0.1894, + "qps_1t": 9230.882866141921, + "qps_1t_runs": [ + 9378.47446222709, + 9222.369791819785, + 9230.882866141921 + ], + "median_ms": 0.10490004206076264, + "p95_ms": 0.15209999401122332, + "p99_ms": 0.18720602791290744, + "mean_ms": 0.10779704974265769 + }, + { + "knob": 48, + "nq": 2000, + "recall": 0.3687, + "qps_1t": 4099.546843558394, + "qps_1t_runs": [ + 4202.854130580572, + 4099.546843558394, + 4070.13658853396 + ], + "median_ms": 0.2410999732092023, + "p95_ms": 0.31101503700483596, + "p99_ms": 0.3716159774921838, + "mean_ms": 0.24251831635289514 + }, + { + "knob": 96, + "nq": 2000, + "recall": 0.52705, + "qps_1t": 2491.699210748766, + "qps_1t_runs": [ + 2509.0860180991162, + 2491.699210748766, + 2277.0158222760533 + ], + "median_ms": 0.41310000233352184, + "p95_ms": 0.5335149879101664, + "p99_ms": 0.6408009899314493, + "mean_ms": 0.413018450946159 + }, + { + "knob": 192, + "nq": 2000, + "recall": 0.7021, + "qps_1t": 1285.794344449443, + "qps_1t_runs": [ + 1303.0277738859409, + 1266.3730959873144, + 1285.794344449443 + ], + "median_ms": 0.7897000177763402, + "p95_ms": 0.9795050049433485, + "p99_ms": 1.0780070006148892, + "mean_ms": 0.7782764667645097 + }, + { + "knob": 640, + "nq": 1013, + "recall": 0.9277393879565646, + "qps_1t": 377.32581029791163, + "qps_1t_runs": [ + 377.32581029791163, + 380.2200878508526, + 370.64568175764583 + ], + "median_ms": 2.72910000057891, + "p95_ms": 3.215020033530891, + "p99_ms": 3.4220020379871126, + "mean_ms": 2.659426554453698 + } + ] + } + }, + "mem_end": { + "rss_mb": 60.00390625, + "peak_mb": 365.46875 + }, + "status": "ok", + "finished": "2026-11-16 11:44:57", + "load_before_pct": 8.999999999999996, + "load_contaminated": false, + "wall_s": 49.68212890625, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_ivfflat__glove__1000000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/faiss_ivfflat__glove__1000000__core+mutation+saveload.json new file mode 100644 index 0000000..1502344 --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_ivfflat__glove__1000000__core+mutation+saveload.json @@ -0,0 +1,403 @@ +{ + "system": "faiss_ivfflat", + "ds": "glove", + "n": 1000000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 10:16:38", + "mem_baseline": { + "rss_mb": 35.08984375, + "peak_mb": 35.08984375 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 3.609998384490609e-05, + "index_s": 11.909425000019837, + "total_s": 11.909461100003682, + "first_query_ms": 31.58200002508238, + "ready_s": 11.941043100028764 + }, + "mem_after_build": { + "rss_mb": 569.12890625, + "peak_mb": 1018.16796875 + }, + "mem_after_first_query": { + "rss_mb": 569.30859375, + "peak_mb": 1018.16796875 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.3046, + "qps_1t_runs": [ + 19407.451816610996, + 18048.29362201203, + 19281.977684321428 + ], + "qps_1t": 19281.977684321428, + "qps_1t_min": 18048.29362201203, + "qps_1t_max": 19407.451816610996, + "median_ms": 0.050350005039945245, + "p95_ms": 0.07059995841700584, + "p99_ms": 0.0911029952112586, + "mean_ms": 0.05293179991228195, + "knob": 1, + "batch": { + "recall": 0.3046, + "qps_batch_runs": [ + 174596.24662671867, + 170065.134591196, + 169922.09095888014 + ], + "qps_batch": 170065.134591196, + "qps_batch_min": 169922.09095888014, + "qps_batch_max": 174596.24662671867, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.42815, + "qps_1t_runs": [ + 16240.607886157788, + 15642.084130302677, + 13254.864542624031 + ], + "qps_1t": 15642.084130302677, + "qps_1t_min": 13254.864542624031, + "qps_1t_max": 16240.607886157788, + "median_ms": 0.060699996538460255, + "p95_ms": 0.09070499218069018, + "p99_ms": 0.13461197726428534, + "mean_ms": 0.06698271655477583, + "knob": 2, + "batch": { + "recall": 0.42815, + "qps_batch_runs": [ + 117091.90527067326, + 130277.0339776239, + 96350.71649683625 + ], + "qps_batch": 117091.90527067326, + "qps_batch_min": 96350.71649683625, + "qps_batch_max": 130277.0339776239, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.5371, + "qps_1t_runs": [ + 11602.590427840798, + 12379.079562858198, + 12354.578805281451 + ], + "qps_1t": 12354.578805281451, + "qps_1t_min": 11602.590427840798, + "qps_1t_max": 12379.079562858198, + "median_ms": 0.07765000918880105, + "p95_ms": 0.10619999666232616, + "p99_ms": 0.13381302880588947, + "mean_ms": 0.08263691684502798, + "knob": 4, + "batch": { + "recall": 0.5371, + "qps_batch_runs": [ + 82463.34510119348, + 83272.61374473717, + 83461.65567814918 + ], + "qps_batch": 83272.61374473717, + "qps_batch_min": 82463.34510119348, + "qps_batch_max": 83461.65567814918, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.63475, + "qps_1t_runs": [ + 8461.346462714502, + 8221.824846522439, + 8861.799353882217 + ], + "qps_1t": 8461.346462714502, + "qps_1t_min": 8221.824846522439, + "qps_1t_max": 8861.799353882217, + "median_ms": 0.11230004020035267, + "p95_ms": 0.16260495758615437, + "p99_ms": 0.21591701370198316, + "mean_ms": 0.11755196656061646, + "knob": 8, + "batch": { + "recall": 0.63475, + "qps_batch_runs": [ + 49956.038668181696, + 49251.500312317956, + 49977.13545806013 + ], + "qps_batch": 49956.038668181696, + "qps_batch_min": 49251.500312317956, + "qps_batch_max": 49977.13545806013, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7203, + "qps_1t_runs": [ + 5748.170372087822, + 5760.302320134666, + 5575.506562798144 + ], + "qps_1t": 5748.170372087822, + "qps_1t_min": 5575.506562798144, + "qps_1t_max": 5760.302320134666, + "median_ms": 0.1700000138953328, + "p95_ms": 0.23919996165204793, + "p99_ms": 0.2879049803595991, + "mean_ms": 0.1756420998038569, + "knob": 16, + "batch": { + "recall": 0.7203, + "qps_batch_runs": [ + 26552.596724555224, + 25542.784165509365, + 26399.643067961286 + ], + "qps_batch": 26399.643067961286, + "qps_batch_min": 25542.784165509365, + "qps_batch_max": 26552.596724555224, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.79335, + "qps_1t_runs": [ + 3221.387438729675, + 3127.584654353327, + 3364.702453708167 + ], + "qps_1t": 3221.387438729675, + "qps_1t_min": 3127.584654353327, + "qps_1t_max": 3364.702453708167, + "median_ms": 0.28600002406165004, + "p95_ms": 0.40630495350342244, + "p99_ms": 0.5269339960068472, + "mean_ms": 0.309121299952191, + "knob": 32, + "batch": { + "recall": 0.79335, + "qps_batch_runs": [ + 14470.17804969104, + 14622.43769506232, + 14831.349025807702 + ], + "qps_batch": 14622.43769506232, + "qps_batch_min": 14470.17804969104, + "qps_batch_max": 14831.349025807702, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.85875, + "qps_1t_runs": [ + 1792.8785955925964, + 1819.1575486770291, + 1796.05442919011 + ], + "qps_1t": 1796.05442919011, + "qps_1t_min": 1792.8785955925964, + "qps_1t_max": 1819.1575486770291, + "median_ms": 0.5222000181674957, + "p95_ms": 0.7214249664684759, + "p99_ms": 0.8821380225708716, + "mean_ms": 0.5547477498863979, + "knob": 64, + "batch": { + "recall": 0.85875, + "qps_batch_runs": [ + 7866.236225995296, + 7626.802262251877, + 7595.762628754349 + ], + "qps_batch": 7626.802262251877, + "qps_batch_min": 7595.762628754349, + "qps_batch_max": 7866.236225995296, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.91225, + "qps_1t_runs": [ + 988.6746336637101, + 997.061311137976, + 976.702376154054 + ], + "qps_1t": 988.6746336637101, + "qps_1t_min": 976.702376154054, + "qps_1t_max": 997.061311137976, + "median_ms": 0.9730500169098377, + "p95_ms": 1.2777250085491687, + "p99_ms": 1.6372900089481888, + "mean_ms": 1.0127519333230641, + "knob": 128, + "batch": { + "recall": 0.91225, + "qps_batch_runs": [ + 3992.9667886658317, + 3977.6693640971093, + 4103.244189362854 + ], + "qps_batch": 3992.9667886658317, + "qps_batch_min": 3977.6693640971093, + "qps_batch_max": 4103.244189362854, + "threads": 32 + } + }, + { + "nq": 1649, + "recall_queries": 1649, + "recall": 0.9487568223165554, + "qps_1t_runs": [ + 534.3025477109227, + 524.7253180281099, + 518.5877216780946 + ], + "qps_1t": 524.7253180281099, + "qps_1t_min": 518.5877216780946, + "qps_1t_max": 534.3025477109227, + "median_ms": 1.8497999990358949, + "p95_ms": 2.341410017106682, + "p99_ms": 3.1939419882837647, + "mean_ms": 1.9018906204056085, + "knob": 256, + "batch": { + "recall": 0.9484, + "qps_batch_runs": [ + 2015.9313003813513, + 2098.031354166346, + 2064.2278296885274 + ], + "qps_batch": 2064.2278296885274, + "qps_batch_min": 2015.9313003813513, + "qps_batch_max": 2098.031354166346, + "threads": 32 + } + }, + { + "nq": 136, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 43.766656276399935, + 43.78137552342066, + 43.020590821539116 + ], + "qps_1t": 43.766656276399935, + "qps_1t_min": 43.020590821539116, + "qps_1t_max": 43.78137552342066, + "median_ms": 22.24394999211654, + "p95_ms": 28.908635003608637, + "p99_ms": 31.374543039128188, + "mean_ms": 22.977963232894098, + "knob": 4000, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 157.31589507569646, + 159.5238862848845, + 159.04147737307707 + ], + "qps_batch": 159.04147737307707, + "qps_batch_min": 157.31589507569646, + "qps_batch_max": 159.5238862848845, + "threads": 32 + } + } + ], + "knob_name": "nprobe", + "mem_after_sweep": { + "rss_mb": 579.46484375, + "peak_mb": 1018.16796875 + }, + "ref_knob": 4000, + "ref_recall_pre": 1.0, + "mutation": { + "ref_knob": 4000, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.04435360000934452, + "per_s": 225460.84191346762, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.08335299999453127, + "per_s": 119971.6866898143, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.004664399952162057, + "per_s": 2143898.486956456, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 587.8359375, + "peak_mb": 1018.16796875 + }, + "first_query_after_ms": 31.066800002008677, + "after": { + "nq": 130, + "recall_queries": 130, + "recall": 1.0, + "qps_1t_runs": [ + 42.239189092513755, + 41.53314956842764, + 42.63354490640715 + ], + "qps_1t": 42.239189092513755, + "qps_1t_min": 41.53314956842764, + "qps_1t_max": 42.63354490640715, + "median_ms": 22.592400026042014, + "p95_ms": 31.08705000195185, + "p99_ms": 38.17615797044714, + "mean_ms": 23.735852820727114 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 0.14943550003226846, + "disk_mb": 390.655650138855, + "load_s": 0.24632809998001903, + "first_query_after_load_ms": 23.79170001950115, + "time_to_ready_s": 0.2701197999995202 + }, + "mem_end": { + "rss_mb": 495.75390625, + "peak_mb": 1018.16796875 + }, + "status": "ok", + "finished": "2026-11-16 10:18:41", + "load_before_pct": 6.406250000000003, + "load_contaminated": false, + "wall_s": 123.08152794837952, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_ivfflat__glove__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/faiss_ivfflat__glove__100000__core+mutation+saveload.json new file mode 100644 index 0000000..f7ad631 --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_ivfflat__glove__100000__core+mutation+saveload.json @@ -0,0 +1,408 @@ +{ + "system": "faiss_ivfflat", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 08:18:41", + "mem_baseline": { + "rss_mb": 35.11328125, + "peak_mb": 35.11328125 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 5.600042641162872e-06, + "index_s": 0.9731661999830976, + "total_s": 0.9731718000257388, + "first_query_ms": 3.0707999831065536, + "ready_s": 0.9762426000088453, + "total_s_runs": [ + 0.9731718000257388, + 1.0775378000107594, + 1.0854720000061207 + ] + }, + "mem_after_build": { + "rss_mb": 125.71875, + "peak_mb": 189.9453125 + }, + "mem_after_first_query": { + "rss_mb": 125.85546875, + "peak_mb": 189.9453125 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.28265, + "qps_1t_runs": [ + 50764.12710632269, + 51839.256755303286, + 51820.72081536658 + ], + "qps_1t": 51820.72081536658, + "qps_1t_min": 50764.12710632269, + "qps_1t_max": 51839.256755303286, + "median_ms": 0.018899969290941954, + "p95_ms": 0.022299995180219412, + "p99_ms": 0.029099989333190024, + "mean_ms": 0.019428883359069005, + "knob": 1, + "batch": { + "recall": 0.28265, + "qps_batch_runs": [ + 367329.6957723643, + 310318.0757221067, + 292056.0751934181 + ], + "qps_batch": 310318.0757221067, + "qps_batch_min": 292056.0751934181, + "qps_batch_max": 367329.6957723643, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.40495, + "qps_1t_runs": [ + 45530.70516741316, + 44673.171046688614, + 41407.953405237175 + ], + "qps_1t": 44673.171046688614, + "qps_1t_min": 41407.953405237175, + "qps_1t_max": 45530.70516741316, + "median_ms": 0.021199986804276705, + "p95_ms": 0.025899967295117676, + "p99_ms": 0.041602025157772046, + "mean_ms": 0.022832650089791667, + "knob": 2, + "batch": { + "recall": 0.40495, + "qps_batch_runs": [ + 382168.03673912975, + 441783.9222497578, + 311827.6216124506 + ], + "qps_batch": 382168.03673912975, + "qps_batch_min": 311827.6216124506, + "qps_batch_max": 441783.9222497578, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.5251, + "qps_1t_runs": [ + 39053.72819832607, + 34299.02230730964, + 35841.715715522325 + ], + "qps_1t": 35841.715715522325, + "qps_1t_min": 34299.02230730964, + "qps_1t_max": 39053.72819832607, + "median_ms": 0.02629996743053198, + "p95_ms": 0.03350002225488424, + "p99_ms": 0.04200602823402746, + "mean_ms": 0.027553850018496934, + "knob": 4, + "batch": { + "recall": 0.5251, + "qps_batch_runs": [ + 266481.906291436, + 275216.73397126055, + 306306.8578009863 + ], + "qps_batch": 275216.73397126055, + "qps_batch_min": 266481.906291436, + "qps_batch_max": 306306.8578009863, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6316, + "qps_1t_runs": [ + 27987.72467294657, + 28594.91742960397, + 27218.698087547822 + ], + "qps_1t": 27987.72467294657, + "qps_1t_min": 27218.698087547822, + "qps_1t_max": 28594.91742960397, + "median_ms": 0.033800024539232254, + "p95_ms": 0.04349998198449612, + "p99_ms": 0.055702016688883346, + "mean_ms": 0.03581354992153744, + "knob": 8, + "batch": { + "recall": 0.6316, + "qps_batch_runs": [ + 219440.2076507734, + 246831.30321182124, + 260718.80074205264 + ], + "qps_batch": 246831.30321182124, + "qps_batch_min": 219440.2076507734, + "qps_batch_max": 260718.80074205264, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7287, + "qps_1t_runs": [ + 19426.07604238988, + 19775.449641076913, + 11660.128260928292 + ], + "qps_1t": 19426.07604238988, + "qps_1t_min": 11660.128260928292, + "qps_1t_max": 19775.449641076913, + "median_ms": 0.04929996794089675, + "p95_ms": 0.06619998021051288, + "p99_ms": 0.15700697316788154, + "mean_ms": 0.06260243338571551, + "knob": 16, + "batch": { + "recall": 0.7287, + "qps_batch_runs": [ + 134086.0031675445, + 187015.51193360728, + 62529.11510069719 + ], + "qps_batch": 134086.0031675445, + "qps_batch_min": 62529.11510069719, + "qps_batch_max": 187015.51193360728, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.81195, + "qps_1t_runs": [ + 12410.15821948934, + 10512.317795088215, + 10764.054056587604 + ], + "qps_1t": 10764.054056587604, + "qps_1t_min": 10512.317795088215, + "qps_1t_max": 12410.15821948934, + "median_ms": 0.083099992480129, + "p95_ms": 0.11841002560686324, + "p99_ms": 0.16481203201692576, + "mean_ms": 0.08953581677633338, + "knob": 32, + "batch": { + "recall": 0.81195, + "qps_batch_runs": [ + 137203.38349474253, + 153105.35888254055, + 156825.84540283465 + ], + "qps_batch": 153105.35888254055, + "qps_batch_min": 137203.38349474253, + "qps_batch_max": 156825.84540283465, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.87955, + "qps_1t_runs": [ + 5917.858332807343, + 6870.736706255522, + 6347.765518460254 + ], + "qps_1t": 6347.765518460254, + "qps_1t_min": 5917.858332807343, + "qps_1t_max": 6870.736706255522, + "median_ms": 0.14929997269064188, + "p95_ms": 0.20280497265048328, + "p99_ms": 0.2977019967511297, + "mean_ms": 0.15735353342218636, + "knob": 64, + "batch": { + "recall": 0.87955, + "qps_batch_runs": [ + 84246.70815372968, + 93353.6842871729, + 94936.5586810349 + ], + "qps_batch": 93353.6842871729, + "qps_batch_min": 84246.70815372968, + "qps_batch_max": 94936.5586810349, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.93315, + "qps_1t_runs": [ + 3637.090404540063, + 3718.7875250712545, + 3728.4120285061463 + ], + "qps_1t": 3718.7875250712545, + "qps_1t_min": 3637.090404540063, + "qps_1t_max": 3728.4120285061463, + "median_ms": 0.25590002769604325, + "p95_ms": 0.33351997844874864, + "p99_ms": 0.4978390067117297, + "mean_ms": 0.2706868662304866, + "knob": 128, + "batch": { + "recall": 0.93315, + "qps_batch_runs": [ + 53001.473415159344, + 53712.039015396615, + 53383.015123214165 + ], + "qps_batch": 53383.015123214165, + "qps_batch_min": 53001.473415159344, + "qps_batch_max": 53712.039015396615, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.97375, + "qps_1t_runs": [ + 2050.8910114333194, + 2030.7749735870777, + 1979.474824344494 + ], + "qps_1t": 2030.7749735870777, + "qps_1t_min": 1979.474824344494, + "qps_1t_max": 2050.8910114333194, + "median_ms": 0.457499991171062, + "p95_ms": 0.603009996120818, + "p99_ms": 0.990019025630329, + "mean_ms": 0.4950667669957814, + "knob": 256, + "batch": { + "recall": 0.97375, + "qps_batch_runs": [ + 24478.9053023146, + 26799.05399898174, + 27518.71959977751 + ], + "qps_batch": 26799.05399898174, + "qps_batch_min": 24478.9053023146, + "qps_batch_max": 27518.71959977751, + "threads": 32 + } + }, + { + "nq": 1387, + "recall_queries": 1387, + "recall": 1.0, + "qps_1t_runs": [ + 437.65144063976624, + 428.20064626510197, + 438.13187792093464 + ], + "qps_1t": 437.65144063976624, + "qps_1t_min": 428.20064626510197, + "qps_1t_max": 438.13187792093464, + "median_ms": 2.1810000180266798, + "p95_ms": 2.7548999642021954, + "p99_ms": 4.526239994447587, + "mean_ms": 2.300898317157317, + "knob": 1265, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 6205.656704038925, + 6444.906042742331, + 6399.543328686767 + ], + "qps_batch": 6399.543328686767, + "qps_batch_min": 6205.656704038925, + "qps_batch_max": 6444.906042742331, + "threads": 32 + } + } + ], + "knob_name": "nprobe", + "mem_after_sweep": { + "rss_mb": 133.41015625, + "peak_mb": 189.9453125 + }, + "ref_knob": 256, + "ref_recall_pre": 0.97375, + "mutation": { + "ref_knob": 256, + "recall_before": 0.97375, + "insert_10k": { + "seconds": 0.014954000012949109, + "per_s": 668717.3994476866, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.016666199953760952, + "per_s": 600016.8021351121, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.001879999996162951, + "per_s": 5319148.947026508, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 148.92578125, + "peak_mb": 249.453125 + }, + "first_query_after_ms": 1.3157000066712499, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.97375, + "qps_1t_runs": [ + 1999.9248001595286, + 2063.5051990951074, + 1877.156090388782 + ], + "qps_1t": 1999.9248001595286, + "qps_1t_min": 1877.156090388782, + "qps_1t_max": 2063.5051990951074, + "median_ms": 0.4807999939657748, + "p95_ms": 0.6394049793016165, + "p99_ms": 0.8056510222377267, + "mean_ms": 0.5057839500659611 + }, + "recall_after": 0.97375 + }, + "saveload": { + "save_s": 0.01961500005563721, + "disk_mb": 39.402255058288574, + "load_s": 0.02694720000727102, + "first_query_after_load_ms": 0.7131000165827572, + "time_to_ready_s": 0.02766030002385378 + }, + "mem_end": { + "rss_mb": 127.7890625, + "peak_mb": 249.453125 + }, + "status": "ok", + "finished": "2026-11-16 08:19:06", + "load_before_pct": 4.343749999999999, + "load_contaminated": false, + "wall_s": 26.014530420303345, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_ivfflat__glove__100000__filter.json b/benchmarks/competitive/results/raw/faiss_ivfflat__glove__100000__filter.json new file mode 100644 index 0000000..8131ed7 --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_ivfflat__glove__100000__filter.json @@ -0,0 +1,186 @@ +{ + "system": "faiss_ivfflat", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 11:45:41", + "filter": { + "build_with_metadata_s": 1.0329756001010537, + "mem": { + "rss_mb": 125.35546875, + "peak_mb": 189.16796875 + }, + "subsets": { + "f10": [ + { + "knob": 2, + "nq": 2000, + "recall": 0.25245, + "qps_1t": 46568.26834281507, + "qps_1t_runs": [ + 46836.43299188914, + 46568.26834281507, + 46301.44129870635 + ], + "median_ms": 0.02099998528137803, + "p95_ms": 0.023800006601959467, + "p99_ms": 0.027801036485470852, + "mean_ms": 0.021474116559450824 + }, + { + "knob": 8, + "nq": 2000, + "recall": 0.4757, + "qps_1t": 37922.02518435879, + "qps_1t_runs": [ + 38069.49521159052, + 37502.8359749178, + 37922.02518435879 + ], + "median_ms": 0.02600002335384488, + "p95_ms": 0.029600050766021013, + "p99_ms": 0.03320101939607414, + "mean_ms": 0.026434100087499246 + }, + { + "knob": 32, + "nq": 2000, + "recall": 0.6965, + "qps_1t": 22912.31583810015, + "qps_1t_runs": [ + 22912.31583810015, + 23068.529704173736, + 22905.466859203912 + ], + "median_ms": 0.042700034100562334, + "p95_ms": 0.04989997250959277, + "p99_ms": 0.057500030379742384, + "mean_ms": 0.04355048323244167 + }, + { + "knob": 64, + "nq": 2000, + "recall": 0.79565, + "qps_1t": 15191.946480572195, + "qps_1t_runs": [ + 14187.154178696659, + 15191.946480572195, + 15580.449698923905 + ], + "median_ms": 0.06549997488036752, + "p95_ms": 0.07910501735750587, + "p99_ms": 0.09650303574744619, + "mean_ms": 0.06683121641981415 + }, + { + "knob": 1265, + "nq": 2000, + "recall": 1.0, + "qps_1t": 1083.0819719056162, + "qps_1t_runs": [ + 1295.036848554639, + 1083.0819719056162, + 1057.6836882883838 + ], + "median_ms": 0.9545999928377569, + "p95_ms": 1.1356250324752182, + "p99_ms": 1.2982409901451328, + "mean_ms": 0.8803107336959025 + } + ], + "f01": [ + { + "knob": 2, + "nq": 2000, + "recall": 0.07965, + "qps_1t": 31984.903219243304, + "qps_1t_runs": [ + 36245.67266438837, + 31984.903219243304, + 31314.145038388862 + ], + "median_ms": 0.032400013878941536, + "p95_ms": 0.034400029107928276, + "p99_ms": 0.044000985799357316, + "mean_ms": 0.030262900108937174 + }, + { + "knob": 8, + "nq": 2000, + "recall": 0.22195, + "qps_1t": 26685.4177381236, + "qps_1t_runs": [ + 26685.4177381236, + 29704.92586738681, + 26623.678403054608 + ], + "median_ms": 0.037299992982298136, + "p95_ms": 0.04129996814299375, + "p99_ms": 0.05170204443857078, + "mean_ms": 0.03623288332407052 + }, + { + "knob": 32, + "nq": 2000, + "recall": 0.45625, + "qps_1t": 19772.986221011946, + "qps_1t_runs": [ + 19635.564119728824, + 20179.27265375653, + 19772.986221011946 + ], + "median_ms": 0.053499999921768904, + "p95_ms": 0.06029999349266291, + "p99_ms": 0.07230202609207485, + "mean_ms": 0.05035261661396362 + }, + { + "knob": 64, + "nq": 2000, + "recall": 0.5928, + "qps_1t": 14100.305376488077, + "qps_1t_runs": [ + 14100.305376488077, + 14085.002939840428, + 15017.044303743827 + ], + "median_ms": 0.07399998139590025, + "p95_ms": 0.08390000148210675, + "p99_ms": 0.09510102507192644, + "mean_ms": 0.0695029834265976 + }, + { + "knob": 1265, + "nq": 2000, + "recall": 1.0, + "qps_1t": 1909.7279259837674, + "qps_1t_runs": [ + 1959.5918473009788, + 1907.8686436107294, + 1909.7279259837674 + ], + "median_ms": 0.5600000149570405, + "p95_ms": 0.609705041279085, + "p99_ms": 0.6907009630231187, + "mean_ms": 0.5193634161454005 + } + ] + } + }, + "mem_end": { + "rss_mb": 80.4375, + "peak_mb": 189.16796875 + }, + "status": "ok", + "finished": "2026-11-16 11:45:53", + "load_before_pct": 9.865624999999998, + "load_contaminated": false, + "wall_s": 12.359814882278442, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_ivfflat__sift__1000000__core+saveload.json b/benchmarks/competitive/results/raw/faiss_ivfflat__sift__1000000__core+saveload.json new file mode 100644 index 0000000..3c5e94b --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_ivfflat__sift__1000000__core+saveload.json @@ -0,0 +1,360 @@ +{ + "system": "faiss_ivfflat", + "ds": "sift", + "n": 1000000, + "d": 128, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "saveload" + ], + "started": "2026-11-16 11:28:51", + "mem_baseline": { + "rss_mb": 35.5703125, + "peak_mb": 35.5703125 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 6.360013503581285e-05, + "index_s": 26.820929800043814, + "total_s": 26.82099340017885, + "first_query_ms": 77.96510000480339, + "ready_s": 26.898958500183653 + }, + "mem_after_build": { + "rss_mb": 719.046875, + "peak_mb": 1293.51171875 + }, + "mem_after_first_query": { + "rss_mb": 719.1953125, + "peak_mb": 1293.51171875 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.2635, + "qps_1t_runs": [ + 9853.396307648662, + 10714.077144156134, + 9769.207334896133 + ], + "qps_1t": 9853.396307648662, + "qps_1t_min": 9769.207334896133, + "qps_1t_max": 10714.077144156134, + "median_ms": 0.0974999857135117, + "p95_ms": 0.13760500005446377, + "p99_ms": 0.17810201679822063, + "mean_ms": 0.09906181662033002, + "knob": 1, + "batch": { + "recall": 0.2635, + "qps_batch_runs": [ + 35564.913062274325, + 47615.193074488714, + 47014.90608411602 + ], + "qps_batch": 47014.90608411602, + "qps_batch_min": 35564.913062274325, + "qps_batch_max": 47615.193074488714, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.3934, + "qps_1t_runs": [ + 9319.021735079097, + 9352.19235245332, + 9919.242489558554 + ], + "qps_1t": 9352.19235245332, + "qps_1t_min": 9319.021735079097, + "qps_1t_max": 9919.242489558554, + "median_ms": 0.10659999679774046, + "p95_ms": 0.14620000729337335, + "p99_ms": 0.20221102633513535, + "mean_ms": 0.10501611686777323, + "knob": 2, + "batch": { + "recall": 0.3934, + "qps_batch_runs": [ + 53014.96081705539, + 58774.26280339925, + 52214.004336904516 + ], + "qps_batch": 53014.96081705539, + "qps_batch_min": 52214.004336904516, + "qps_batch_max": 58774.26280339925, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.54085, + "qps_1t_runs": [ + 6865.368706107131, + 7229.084632254392, + 7675.100713065397 + ], + "qps_1t": 7229.084632254392, + "qps_1t_min": 6865.368706107131, + "qps_1t_max": 7675.100713065397, + "median_ms": 0.13499998021870852, + "p95_ms": 0.19749999046325684, + "p99_ms": 0.25320000713691115, + "mean_ms": 0.13809338357532397, + "knob": 4, + "batch": { + "recall": 0.54085, + "qps_batch_runs": [ + 38314.24961928604, + 41490.50490655643, + 37140.54914806244 + ], + "qps_batch": 38314.24961928604, + "qps_batch_min": 37140.54914806244, + "qps_batch_max": 41490.50490655643, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.69175, + "qps_1t_runs": [ + 4758.556786293455, + 5567.311851330581, + 5382.155942135849 + ], + "qps_1t": 5382.155942135849, + "qps_1t_min": 4758.556786293455, + "qps_1t_max": 5567.311851330581, + "median_ms": 0.18660002388060093, + "p95_ms": 0.28009997913613915, + "p99_ms": 0.3491160389967266, + "mean_ms": 0.191855600035827, + "knob": 8, + "batch": { + "recall": 0.69175, + "qps_batch_runs": [ + 24214.921968571074, + 23059.140929699992, + 22450.7458757803 + ], + "qps_batch": 23059.140929699992, + "qps_batch_min": 22450.7458757803, + "qps_batch_max": 24214.921968571074, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.82295, + "qps_1t_runs": [ + 3540.9768010642715, + 3727.9762848837395, + 3576.300002495727 + ], + "qps_1t": 3576.300002495727, + "qps_1t_min": 3540.9768010642715, + "qps_1t_max": 3727.9762848837395, + "median_ms": 0.26699999580159783, + "p95_ms": 0.40949997783172876, + "p99_ms": 0.48300495254807185, + "mean_ms": 0.2767561999789905, + "knob": 16, + "batch": { + "recall": 0.82295, + "qps_batch_runs": [ + 13486.567713335664, + 14027.380040111697, + 13782.505177573274 + ], + "qps_batch": 13782.505177573274, + "qps_batch_min": 13486.567713335664, + "qps_batch_max": 14027.380040111697, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9173, + "qps_1t_runs": [ + 2230.347132253294, + 2167.904181056244, + 2208.673237858 + ], + "qps_1t": 2208.673237858, + "qps_1t_min": 2167.904181056244, + "qps_1t_max": 2230.347132253294, + "median_ms": 0.4336500132922083, + "p95_ms": 0.663514988264069, + "p99_ms": 0.7766079762950542, + "mean_ms": 0.4541320663508183, + "knob": 32, + "batch": { + "recall": 0.9173, + "qps_batch_runs": [ + 7504.127269866864, + 7912.563014781055, + 7423.519194051266 + ], + "qps_batch": 7504.127269866864, + "qps_batch_min": 7423.519194051266, + "qps_batch_max": 7912.563014781055, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.97005, + "qps_1t_runs": [ + 1269.8222424623443, + 1270.4119468765543, + 1265.9179692673463 + ], + "qps_1t": 1269.8222424623443, + "qps_1t_min": 1265.9179692673463, + "qps_1t_max": 1270.4119468765543, + "median_ms": 0.7607000297866762, + "p95_ms": 1.1436299770139158, + "p99_ms": 1.302521022735164, + "mean_ms": 0.7881995500065386, + "knob": 64, + "batch": { + "recall": 0.97005, + "qps_batch_runs": [ + 3831.8756874844803, + 3253.9298524527885, + 3108.588595018012 + ], + "qps_batch": 3253.9298524527885, + "qps_batch_min": 3108.588595018012, + "qps_batch_max": 3831.8756874844803, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9923, + "qps_1t_runs": [ + 678.4205974514858, + 662.9495085604967, + 668.0849664744334 + ], + "qps_1t": 668.0849664744334, + "qps_1t_min": 662.9495085604967, + "qps_1t_max": 678.4205974514858, + "median_ms": 1.4595000247936696, + "p95_ms": 2.0172150485450406, + "p99_ms": 2.264212010777556, + "mean_ms": 1.4930793168896344, + "knob": 128, + "batch": { + "recall": 0.9923, + "qps_batch_runs": [ + 1990.1527242880686, + 2325.5202828837096, + 2590.068666695632 + ], + "qps_batch": 2325.5202828837096, + "qps_batch_min": 1990.1527242880686, + "qps_batch_max": 2590.068666695632, + "threads": 32 + } + }, + { + "nq": 1035, + "recall_queries": 1035, + "recall": 0.9983574879227053, + "qps_1t_runs": [ + 345.9760561921172, + 362.5302466304616, + 358.0662182325903 + ], + "qps_1t": 358.0662182325903, + "qps_1t_min": 345.9760561921172, + "qps_1t_max": 362.5302466304616, + "median_ms": 2.798899949993938, + "p95_ms": 3.625900007318705, + "p99_ms": 4.040848035365343, + "mean_ms": 2.813847826219399, + "knob": 256, + "batch": { + "recall": 0.9982, + "qps_batch_runs": [ + 1224.0601757720042, + 1106.9108150222319, + 1114.6970295162796 + ], + "qps_batch": 1114.6970295162796, + "qps_batch_min": 1106.9108150222319, + "qps_batch_max": 1224.0601757720042, + "threads": 32 + } + }, + { + "nq": 86, + "recall_queries": 86, + "recall": 0.9988372093023256, + "qps_1t_runs": [ + 26.94771588663001, + 27.40591159928293, + 28.092552287989317 + ], + "qps_1t": 27.40591159928293, + "qps_1t_min": 26.94771588663001, + "qps_1t_max": 28.092552287989317, + "median_ms": 36.22710003401153, + "p95_ms": 39.043385011609644, + "p99_ms": 40.78133401402738, + "mean_ms": 36.39799922405987, + "knob": 4000, + "batch": { + "recall": 0.9994, + "qps_batch_runs": [ + 78.89019513958107, + 81.98307717623237, + 79.83002750199307 + ], + "qps_batch": 79.83002750199307, + "qps_batch_min": 78.89019513958107, + "qps_batch_max": 81.98307717623237, + "threads": 32 + } + } + ], + "knob_name": "nprobe", + "mem_after_sweep": { + "rss_mb": 728.58203125, + "peak_mb": 1293.51171875 + }, + "ref_knob": 64, + "ref_recall_pre": 0.97005, + "saveload": { + "save_s": 0.35315360000822693, + "disk_mb": 497.894419670105, + "load_s": 0.7231700000120327, + "first_query_after_load_ms": 1.6272999928332865, + "time_to_ready_s": 0.724797300004866 + }, + "mem_end": { + "rss_mb": 609.5625, + "peak_mb": 1293.51171875 + }, + "status": "ok", + "finished": "2026-11-16 11:31:29", + "load_before_pct": 12.340624999999987, + "load_contaminated": false, + "wall_s": 158.71269822120667, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_ivfflat__sift__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/faiss_ivfflat__sift__100000__core+mutation+saveload.json new file mode 100644 index 0000000..0ef8343 --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_ivfflat__sift__100000__core+mutation+saveload.json @@ -0,0 +1,408 @@ +{ + "system": "faiss_ivfflat", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 11:08:08", + "mem_baseline": { + "rss_mb": 35.3671875, + "peak_mb": 35.3671875 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 7.90000194683671e-06, + "index_s": 1.2938719000085257, + "total_s": 1.2938798000104725, + "first_query_ms": 6.0953000211156905, + "ready_s": 1.2999751000315882, + "total_s_runs": [ + 1.2938798000104725, + 1.2565003000781871, + 1.2489298000000417 + ] + }, + "mem_after_build": { + "rss_mb": 140.640625, + "peak_mb": 216.1875 + }, + "mem_after_first_query": { + "rss_mb": 140.765625, + "peak_mb": 216.1875 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.28365, + "qps_1t_runs": [ + 45779.16058838538, + 47166.47381183704, + 46449.19107151231 + ], + "qps_1t": 46449.19107151231, + "qps_1t_min": 45779.16058838538, + "qps_1t_max": 47166.47381183704, + "median_ms": 0.020799983758479357, + "p95_ms": 0.02540001878514886, + "p99_ms": 0.030800001695752144, + "mean_ms": 0.02152480023020568, + "knob": 1, + "batch": { + "recall": 0.28365, + "qps_batch_runs": [ + 260844.61482609718, + 260023.9218380907, + 269030.5494736345 + ], + "qps_batch": 260844.61482609718, + "qps_batch_min": 260023.9218380907, + "qps_batch_max": 269030.5494736345, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.4257, + "qps_1t_runs": [ + 40290.73925960876, + 39424.326068370654, + 39479.97042904718 + ], + "qps_1t": 39479.97042904718, + "qps_1t_min": 39424.326068370654, + "qps_1t_max": 40290.73925960876, + "median_ms": 0.023900007363408804, + "p95_ms": 0.03130000550299883, + "p99_ms": 0.040902020409703296, + "mean_ms": 0.025171316257910803, + "knob": 2, + "batch": { + "recall": 0.4257, + "qps_batch_runs": [ + 299056.47718483285, + 250400.64129287703, + 242045.77099248278 + ], + "qps_batch": 250400.64129287703, + "qps_batch_min": 242045.77099248278, + "qps_batch_max": 299056.47718483285, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.5849, + "qps_1t_runs": [ + 32811.57505688234, + 30978.31057355483, + 28531.891573655146 + ], + "qps_1t": 30978.31057355483, + "qps_1t_min": 28531.891573655146, + "qps_1t_max": 32811.57505688234, + "median_ms": 0.029799994081258774, + "p95_ms": 0.041605025762692115, + "p99_ms": 0.06180000491440296, + "mean_ms": 0.03260206649429165, + "knob": 4, + "batch": { + "recall": 0.5849, + "qps_batch_runs": [ + 228287.0495511245, + 215375.66797053098, + 221614.01404819093 + ], + "qps_batch": 221614.01404819093, + "qps_batch_min": 215375.66797053098, + "qps_batch_max": 228287.0495511245, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7447, + "qps_1t_runs": [ + 22574.71396180766, + 24141.672987691236, + 21736.295353263977 + ], + "qps_1t": 22574.71396180766, + "qps_1t_min": 21736.295353263977, + "qps_1t_max": 24141.672987691236, + "median_ms": 0.040599959902465343, + "p95_ms": 0.05829998117405921, + "p99_ms": 0.08330805168952811, + "mean_ms": 0.043908499860359974, + "knob": 8, + "batch": { + "recall": 0.7447, + "qps_batch_runs": [ + 179953.21226942915, + 207688.6335376515, + 212357.05710580616 + ], + "qps_batch": 207688.6335376515, + "qps_batch_min": 179953.21226942915, + "qps_batch_max": 212357.05710580616, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.86855, + "qps_1t_runs": [ + 15617.532835481019, + 14663.971379752347, + 14009.840427012357 + ], + "qps_1t": 14663.971379752347, + "qps_1t_min": 14009.840427012357, + "qps_1t_max": 15617.532835481019, + "median_ms": 0.06360001862049103, + "p95_ms": 0.09540497267153115, + "p99_ms": 0.13230801792815344, + "mean_ms": 0.06786778359673917, + "knob": 16, + "batch": { + "recall": 0.86855, + "qps_batch_runs": [ + 144067.3088262884, + 142585.21253684963, + 146522.29360921946 + ], + "qps_batch": 144067.3088262884, + "qps_batch_min": 142585.21253684963, + "qps_batch_max": 146522.29360921946, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.95045, + "qps_1t_runs": [ + 8823.154425692188, + 8763.631273373754, + 8546.840522530436 + ], + "qps_1t": 8763.631273373754, + "qps_1t_min": 8546.840522530436, + "qps_1t_max": 8823.154425692188, + "median_ms": 0.10529998689889908, + "p95_ms": 0.15952499234117576, + "p99_ms": 0.24826096952892973, + "mean_ms": 0.11481613341796522, + "knob": 32, + "batch": { + "recall": 0.95045, + "qps_batch_runs": [ + 112537.20766997218, + 120023.5246781843, + 108717.51366544623 + ], + "qps_batch": 112537.20766997218, + "qps_batch_min": 108717.51366544623, + "qps_batch_max": 120023.5246781843, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9876, + "qps_1t_runs": [ + 4987.015040976421, + 5212.723412740188, + 4990.789484955886 + ], + "qps_1t": 4990.789484955886, + "qps_1t_min": 4987.015040976421, + "qps_1t_max": 5212.723412740188, + "median_ms": 0.18099998123943806, + "p95_ms": 0.278900028206408, + "p99_ms": 0.6100069882813842, + "mean_ms": 0.19757605055929162, + "knob": 64, + "batch": { + "recall": 0.9876, + "qps_batch_runs": [ + 51455.54878984149, + 51942.520410659665, + 51490.52186920943 + ], + "qps_batch": 51490.52186920943, + "qps_batch_min": 51455.54878984149, + "qps_batch_max": 51942.520410659665, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9986, + "qps_1t_runs": [ + 2803.8677723626506, + 2623.92950416551, + 2812.937541746591 + ], + "qps_1t": 2803.8677723626506, + "qps_1t_min": 2623.92950416551, + "qps_1t_max": 2812.937541746591, + "median_ms": 0.34009997034445405, + "p95_ms": 0.5055150075349957, + "p99_ms": 0.8544079866260292, + "mean_ms": 0.36441941644685966, + "knob": 128, + "batch": { + "recall": 0.9986, + "qps_batch_runs": [ + 30326.142518890818, + 28045.41675089572, + 29165.105100498964 + ], + "qps_batch": 29165.105100498964, + "qps_batch_min": 28045.41675089572, + "qps_batch_max": 30326.142518890818, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99995, + "qps_1t_runs": [ + 1475.1417608256018, + 1423.1979422029708, + 1438.301331700387 + ], + "qps_1t": 1438.301331700387, + "qps_1t_min": 1423.1979422029708, + "qps_1t_max": 1475.1417608256018, + "median_ms": 0.6542000046465546, + "p95_ms": 1.0054100275738165, + "p99_ms": 1.5070700080832482, + "mean_ms": 0.6919361662876327, + "knob": 256, + "batch": { + "recall": 0.99995, + "qps_batch_runs": [ + 15345.661012639717, + 16373.727046861477, + 15784.821316471978 + ], + "qps_batch": 15784.821316471978, + "qps_batch_min": 15345.661012639717, + "qps_batch_max": 16373.727046861477, + "threads": 32 + } + }, + { + "nq": 988, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 307.81225947274436, + 299.73582896198633, + 310.6211038313147 + ], + "qps_1t": 307.81225947274436, + "qps_1t_min": 299.73582896198633, + "qps_1t_max": 310.6211038313147, + "median_ms": 3.133050020551309, + "p95_ms": 3.9111950376536693, + "p99_ms": 4.8217820015270245, + "mean_ms": 3.2681203109505064, + "knob": 1265, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 3785.6625977814692, + 3415.4416901498516, + 3461.6845182099487 + ], + "qps_batch": 3461.6845182099487, + "qps_batch_min": 3415.4416901498516, + "qps_batch_max": 3785.6625977814692, + "threads": 32 + } + } + ], + "knob_name": "nprobe", + "mem_after_sweep": { + "rss_mb": 146.26953125, + "peak_mb": 216.1875 + }, + "ref_knob": 32, + "ref_recall_pre": 0.95045, + "mutation": { + "ref_knob": 32, + "recall_before": 0.95045, + "insert_10k": { + "seconds": 0.03519810002762824, + "per_s": 284106.2441481399, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.0634624999947846, + "per_s": 157573.37011340255, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.0035829999833367765, + "per_s": 2790957.3113330575, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 168.4140625, + "peak_mb": 290.11328125 + }, + "first_query_after_ms": 0.43760001426562667, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.94675, + "qps_1t_runs": [ + 8355.81043831318, + 9084.48800273748, + 8754.639393979862 + ], + "qps_1t": 8754.639393979862, + "qps_1t_min": 8355.81043831318, + "qps_1t_max": 9084.48800273748, + "median_ms": 0.10555001790635288, + "p95_ms": 0.15769998135510832, + "p99_ms": 0.22260300698690122, + "mean_ms": 0.11466003355841774 + }, + "recall_after": 0.94675 + }, + "saveload": { + "save_s": 0.02542570000514388, + "disk_mb": 50.21852397918701, + "load_s": 0.03533669997705147, + "first_query_after_load_ms": 0.2161000156775117, + "time_to_ready_s": 0.03555279999272898 + }, + "mem_end": { + "rss_mb": 137.6015625, + "peak_mb": 290.11328125 + }, + "status": "ok", + "finished": "2026-11-16 11:08:35", + "load_before_pct": 9.115625000000003, + "load_contaminated": false, + "wall_s": 27.559601306915283, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_ivfflat__text384__1000000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/faiss_ivfflat__text384__1000000__core+mutation+saveload.json new file mode 100644 index 0000000..c9321f3 --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_ivfflat__text384__1000000__core+mutation+saveload.json @@ -0,0 +1,403 @@ +{ + "system": "faiss_ivfflat", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 09:13:13", + "mem_baseline": { + "rss_mb": 37.30859375, + "peak_mb": 37.30859375 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 6.080017192289233e-05, + "index_s": 18.49423830001615, + "total_s": 18.494299100188073, + "first_query_ms": 80.72970004286617, + "ready_s": 18.57502880023094 + }, + "mem_after_build": { + "rss_mb": 1922.3203125, + "peak_mb": 3629.421875 + }, + "mem_after_first_query": { + "rss_mb": 1922.4453125, + "peak_mb": 3629.421875 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.27105, + "qps_1t_runs": [ + 5732.901260874405, + 5665.234498165314, + 5464.012098871559 + ], + "qps_1t": 5665.234498165314, + "qps_1t_min": 5464.012098871559, + "qps_1t_max": 5732.901260874405, + "median_ms": 0.16820000018924475, + "p95_ms": 0.21871003555133942, + "p99_ms": 0.3441310004564009, + "mean_ms": 0.17798755005545294, + "knob": 1, + "batch": { + "recall": 0.27105, + "qps_batch_runs": [ + 53177.48784421624, + 43160.468418000586, + 48472.74502099718 + ], + "qps_batch": 48472.74502099718, + "qps_batch_min": 43160.468418000586, + "qps_batch_max": 53177.48784421624, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.38275, + "qps_1t_runs": [ + 4593.111188797486, + 4539.455240808732, + 4849.87925672094 + ], + "qps_1t": 4593.111188797486, + "qps_1t_min": 4539.455240808732, + "qps_1t_max": 4849.87925672094, + "median_ms": 0.1965999836102128, + "p95_ms": 0.2596049977000803, + "p99_ms": 0.4619120241841304, + "mean_ms": 0.21473293371188143, + "knob": 2, + "batch": { + "recall": 0.38275, + "qps_batch_runs": [ + 29571.49425121084, + 34184.644924359236, + 33893.99652168202 + ], + "qps_batch": 33893.99652168202, + "qps_batch_min": 29571.49425121084, + "qps_batch_max": 34184.644924359236, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.4944, + "qps_1t_runs": [ + 3766.98700524271, + 3673.866454891245, + 3513.13561316215 + ], + "qps_1t": 3673.866454891245, + "qps_1t_min": 3513.13561316215, + "qps_1t_max": 3766.98700524271, + "median_ms": 0.25450001703575253, + "p95_ms": 0.3500999911921099, + "p99_ms": 0.653701971168632, + "mean_ms": 0.274100983311655, + "knob": 4, + "batch": { + "recall": 0.4944, + "qps_batch_runs": [ + 17023.66715560851, + 17039.54878939169, + 17988.992535419085 + ], + "qps_batch": 17039.54878939169, + "qps_batch_min": 17023.66715560851, + "qps_batch_max": 17988.992535419085, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.60865, + "qps_1t_runs": [ + 2710.319803372828, + 2699.503793986944, + 2773.988587727927 + ], + "qps_1t": 2710.319803372828, + "qps_1t_min": 2699.503793986944, + "qps_1t_max": 2773.988587727927, + "median_ms": 0.3511499962769449, + "p95_ms": 0.4747050057630986, + "p99_ms": 0.6702169805066686, + "mean_ms": 0.366630116312687, + "knob": 8, + "batch": { + "recall": 0.60865, + "qps_batch_runs": [ + 11569.300401222521, + 11014.374310297293, + 11517.08847674131 + ], + "qps_batch": 11517.08847674131, + "qps_batch_min": 11014.374310297293, + "qps_batch_max": 11569.300401222521, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7099, + "qps_1t_runs": [ + 1643.6749417390806, + 1693.4195893400558, + 1826.5639197918922 + ], + "qps_1t": 1693.4195893400558, + "qps_1t_min": 1643.6749417390806, + "qps_1t_max": 1826.5639197918922, + "median_ms": 0.5374000174924731, + "p95_ms": 0.7504300097934903, + "p99_ms": 1.50892302277498, + "mean_ms": 0.5821299836000738, + "knob": 16, + "batch": { + "recall": 0.7099, + "qps_batch_runs": [ + 6113.211169174637, + 6334.993840232643, + 6263.064361043098 + ], + "qps_batch": 6263.064361043098, + "qps_batch_min": 6113.211169174637, + "qps_batch_max": 6334.993840232643, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7952, + "qps_1t_runs": [ + 1086.8689221971993, + 1069.1211977486498, + 1079.1109751208205 + ], + "qps_1t": 1079.1109751208205, + "qps_1t_min": 1069.1211977486498, + "qps_1t_max": 1086.8689221971993, + "median_ms": 0.8744499937165529, + "p95_ms": 1.1628400068730118, + "p99_ms": 1.6152060148306258, + "mean_ms": 0.927370183247452, + "knob": 32, + "batch": { + "recall": 0.7952, + "qps_batch_runs": [ + 3435.3677637225355, + 3438.737515409886, + 3173.5594264085485 + ], + "qps_batch": 3435.3677637225355, + "qps_batch_min": 3173.5594264085485, + "qps_batch_max": 3438.737515409886, + "threads": 32 + } + }, + { + "nq": 1690, + "recall_queries": 1690, + "recall": 0.8620118343195267, + "qps_1t_runs": [ + 597.9830632198161, + 577.9825118404891, + 578.6310854243291 + ], + "qps_1t": 578.6310854243291, + "qps_1t_min": 577.9825118404891, + "qps_1t_max": 597.9830632198161, + "median_ms": 1.5928999637253582, + "p95_ms": 2.0974849903723225, + "p99_ms": 3.3699640113627956, + "mean_ms": 1.7102203948291925, + "knob": 64, + "batch": { + "recall": 0.8651, + "qps_batch_runs": [ + 1747.707270206291, + 1818.0234848510586, + 1783.6145572898147 + ], + "qps_batch": 1783.6145572898147, + "qps_batch_min": 1747.707270206291, + "qps_batch_max": 1818.0234848510586, + "threads": 32 + } + }, + { + "nq": 986, + "recall_queries": 1000, + "recall": 0.9183, + "qps_1t_runs": [ + 314.76592156322806, + 324.40771256497766, + 321.21529615018915 + ], + "qps_1t": 321.21529615018915, + "qps_1t_min": 314.76592156322806, + "qps_1t_max": 324.40771256497766, + "median_ms": 2.9966000001877546, + "p95_ms": 3.7562450132099916, + "p99_ms": 5.053043008665543, + "mean_ms": 3.1242271805343615, + "knob": 128, + "batch": { + "recall": 0.91875, + "qps_batch_runs": [ + 910.3805044678606, + 907.4954907645792, + 913.484812563989 + ], + "qps_batch": 910.3805044678606, + "qps_batch_min": 907.4954907645792, + "qps_batch_max": 913.484812563989, + "threads": 32 + } + }, + { + "nq": 541, + "recall_queries": 1000, + "recall": 0.9561, + "qps_1t_runs": [ + 171.32075374782696, + 171.2566129190882, + 169.00607316506571 + ], + "qps_1t": 171.2566129190882, + "qps_1t_min": 169.00607316506571, + "qps_1t_max": 171.32075374782696, + "median_ms": 5.629299965221435, + "p95_ms": 7.034130021929739, + "p99_ms": 13.249214024981484, + "mean_ms": 5.864380715166415, + "knob": 256, + "batch": { + "recall": 0.9572, + "qps_batch_runs": [ + 466.90232160621355, + 468.0216267183672, + 456.78724174622664 + ], + "qps_batch": 466.90232160621355, + "qps_batch_min": 456.78724174622664, + "qps_batch_max": 468.0216267183672, + "threads": 32 + } + }, + { + "nq": 38, + "recall_queries": 38, + "recall": 1.0, + "qps_1t_runs": [ + 12.34720496393977, + 12.22943029109082, + 12.599246838612803 + ], + "qps_1t": 12.34720496393977, + "qps_1t_min": 12.22943029109082, + "qps_1t_max": 12.599246838612803, + "median_ms": 78.59804999316111, + "p95_ms": 89.40011499798857, + "p99_ms": 100.34008598478978, + "mean_ms": 80.70992367962997, + "knob": 4000, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 30.929301084455542, + 30.449447254460516, + 27.25492746363667 + ], + "qps_batch": 30.449447254460516, + "qps_batch_min": 27.25492746363667, + "qps_batch_max": 30.929301084455542, + "threads": 32 + } + } + ], + "knob_name": "nprobe", + "mem_after_sweep": { + "rss_mb": 1927.41015625, + "peak_mb": 3629.421875 + }, + "ref_knob": 256, + "ref_recall_pre": 0.9561, + "mutation": { + "ref_knob": 256, + "recall_before": 0.9561, + "insert_10k": { + "seconds": 0.12511570000788197, + "per_s": 79926.0204704128, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.13949179998598993, + "per_s": 71688.80178622948, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.004724500002339482, + "per_s": 2116626.0969516756, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 1976.7578125, + "peak_mb": 3629.421875 + }, + "first_query_after_ms": 8.726300031412393, + "after": { + "nq": 502, + "recall_queries": 1000, + "recall": 0.9571000000000001, + "qps_1t_runs": [ + 159.49925111018794, + 157.23342836365086, + 157.79887793672094 + ], + "qps_1t": 157.79887793672094, + "qps_1t_min": 157.23342836365086, + "qps_1t_max": 159.49925111018794, + "median_ms": 6.059849984012544, + "p95_ms": 7.609975000377744, + "p99_ms": 13.724885028204888, + "mean_ms": 6.3222577687706005 + }, + "recall_after": 0.9571000000000001 + }, + "saveload": { + "save_s": 0.41686230001505464, + "disk_mb": 1478.363169670105, + "load_s": 0.6695059000048786, + "first_query_after_load_ms": 6.217199959792197, + "time_to_ready_s": 0.6757230999646708 + }, + "mem_end": { + "rss_mb": 1594.6796875, + "peak_mb": 3629.421875 + }, + "status": "ok", + "finished": "2026-11-16 09:18:39", + "load_before_pct": 10.125000000000012, + "load_contaminated": false, + "wall_s": 326.23267245292664, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_ivfflat__text384__1000000__filter.json b/benchmarks/competitive/results/raw/faiss_ivfflat__text384__1000000__filter.json new file mode 100644 index 0000000..2b1eb85 --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_ivfflat__text384__1000000__filter.json @@ -0,0 +1,186 @@ +{ + "system": "faiss_ivfflat", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 11:47:20", + "filter": { + "build_with_metadata_s": 46.97382059984375, + "mem": { + "rss_mb": 1913.45703125, + "peak_mb": 3619.25 + }, + "subsets": { + "f10": [ + { + "knob": 2, + "nq": 2000, + "recall": 0.31405, + "qps_1t": 3902.1382789327786, + "qps_1t_runs": [ + 3902.1382789327786, + 4053.5489980521975, + 3787.378114240684 + ], + "median_ms": 0.26890001026913524, + "p95_ms": 0.31049997778609395, + "p99_ms": 0.3722069825744258, + "mean_ms": 0.2556673498669018 + }, + { + "knob": 8, + "nq": 2000, + "recall": 0.54505, + "qps_1t": 3212.525763997164, + "qps_1t_runs": [ + 3199.3596261992006, + 3315.013461573325, + 3212.525763997164 + ], + "median_ms": 0.32130000181496143, + "p95_ms": 0.37839997094124556, + "p99_ms": 0.4318020283244551, + "mean_ms": 0.30850066649145447 + }, + { + "knob": 32, + "nq": 2000, + "recall": 0.7427, + "qps_1t": 2010.0108607920495, + "qps_1t_runs": [ + 2010.0108607920495, + 2052.8251439403884, + 2006.1228889847303 + ], + "median_ms": 0.5012999754399061, + "p95_ms": 0.6028999923728406, + "p99_ms": 0.738600977929309, + "mean_ms": 0.49437241643317975 + }, + { + "knob": 64, + "nq": 2000, + "recall": 0.82635, + "qps_1t": 1419.4566932721766, + "qps_1t_runs": [ + 1461.192342100759, + 1419.4566932721766, + 1410.3788514493797 + ], + "median_ms": 0.7134999905247241, + "p95_ms": 0.8912149758543819, + "p99_ms": 1.0548009991180152, + "mean_ms": 0.6992989491360883 + }, + { + "knob": 4000, + "nq": 118, + "recall": 1.0, + "qps_1t": 38.09917777729757, + "qps_1t_runs": [ + 38.09917777729757, + 36.752284114688536, + 38.722345062547184 + ], + "median_ms": 26.602250029100105, + "p95_ms": 29.446885021752678, + "p99_ms": 31.17722800117917, + "mean_ms": 26.427120342258743 + } + ], + "f01": [ + { + "knob": 2, + "nq": 2000, + "recall": 0.1722, + "qps_1t": 4159.5322092282795, + "qps_1t_runs": [ + 4159.5322092282795, + 4096.676660594222, + 4263.111564658021 + ], + "median_ms": 0.2539000124670565, + "p95_ms": 0.2825999981723726, + "p99_ms": 0.3076009562937543, + "mean_ms": 0.23969413319719024 + }, + { + "knob": 8, + "nq": 2000, + "recall": 0.3754, + "qps_1t": 3921.391784770546, + "qps_1t_runs": [ + 3937.841930935292, + 3776.852233549111, + 3921.391784770546 + ], + "median_ms": 0.2696499868761748, + "p95_ms": 0.29510497697629035, + "p99_ms": 0.31650102289859205, + "mean_ms": 0.25790948382928036 + }, + { + "knob": 32, + "nq": 2000, + "recall": 0.61645, + "qps_1t": 3325.8003924184354, + "qps_1t_runs": [ + 3325.8003924184354, + 3180.073404658829, + 3413.454508696176 + ], + "median_ms": 0.32049999572336674, + "p95_ms": 0.3630099701695145, + "p99_ms": 0.4293040093034507, + "mean_ms": 0.3026986667246092 + }, + { + "knob": 64, + "nq": 2000, + "recall": 0.7293, + "qps_1t": 2711.3313291631353, + "qps_1t_runs": [ + 2748.871477922609, + 2697.958672405094, + 2711.3313291631353 + ], + "median_ms": 0.38700003642588854, + "p95_ms": 0.4525050084339455, + "p99_ms": 0.6172019854420796, + "mean_ms": 0.3677529170042059 + }, + { + "knob": 4000, + "nq": 414, + "recall": 1.0, + "qps_1t": 140.23270566994586, + "qps_1t_runs": [ + 143.38686778958785, + 138.5265347887052, + 140.23270566994586 + ], + "median_ms": 7.258399971760809, + "p95_ms": 8.107459978782572, + "p99_ms": 8.53707101137843, + "mean_ms": 7.1079923501185736 + } + ] + } + }, + "mem_end": { + "rss_mb": 80.5625, + "peak_mb": 3619.25 + }, + "status": "ok", + "finished": "2026-11-16 11:48:45", + "load_before_pct": 8.796874999999998, + "load_contaminated": false, + "wall_s": 85.26953911781311, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_ivfflat__text384__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/faiss_ivfflat__text384__100000__core+mutation+saveload.json new file mode 100644 index 0000000..ce9faff --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_ivfflat__text384__100000__core+mutation+saveload.json @@ -0,0 +1,408 @@ +{ + "system": "faiss_ivfflat", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 07:53:08", + "mem_baseline": { + "rss_mb": 37.5546875, + "peak_mb": 37.5546875 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 8.200004231184721e-06, + "index_s": 1.7210487000411376, + "total_s": 1.7210569000453688, + "first_query_ms": 15.245300019159913, + "ready_s": 1.7363022000645287, + "total_s_runs": [ + 1.7210569000453688, + 1.854149300022982, + 1.829295700066723 + ] + }, + "mem_after_build": { + "rss_mb": 275.8515625, + "peak_mb": 496.47265625 + }, + "mem_after_first_query": { + "rss_mb": 275.98046875, + "peak_mb": 496.47265625 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.2834, + "qps_1t_runs": [ + 18898.46518103432, + 18005.803256934512, + 18763.310235400597 + ], + "qps_1t": 18763.310235400597, + "qps_1t_min": 18005.803256934512, + "qps_1t_max": 18898.46518103432, + "median_ms": 0.052099989261478186, + "p95_ms": 0.06479996955022216, + "p99_ms": 0.08200098411180082, + "mean_ms": 0.05391583331705382, + "knob": 1, + "batch": { + "recall": 0.2834, + "qps_batch_runs": [ + 212442.7725172613, + 181796.6966135266, + 192036.2570791064 + ], + "qps_batch": 192036.2570791064, + "qps_batch_min": 181796.6966135266, + "qps_batch_max": 212442.7725172613, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.40995, + "qps_1t_runs": [ + 16180.58840526749, + 16032.848155309437, + 16022.110509262971 + ], + "qps_1t": 16032.848155309437, + "qps_1t_min": 16022.110509262971, + "qps_1t_max": 16180.58840526749, + "median_ms": 0.060399994254112244, + "p95_ms": 0.07639999967068434, + "p99_ms": 0.09510102507192644, + "mean_ms": 0.062196049810154364, + "knob": 2, + "batch": { + "recall": 0.40995, + "qps_batch_runs": [ + 168357.25406953352, + 172376.6431441996, + 176973.9225600307 + ], + "qps_batch": 172376.6431441996, + "qps_batch_min": 168357.25406953352, + "qps_batch_max": 176973.9225600307, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.5287, + "qps_1t_runs": [ + 12948.629586596435, + 12664.199290334738, + 11615.966841498226 + ], + "qps_1t": 12664.199290334738, + "qps_1t_min": 11615.966841498226, + "qps_1t_max": 12948.629586596435, + "median_ms": 0.07720000576227903, + "p95_ms": 0.10081000800710176, + "p99_ms": 0.14101596781984008, + "mean_ms": 0.0807597998646088, + "knob": 4, + "batch": { + "recall": 0.5287, + "qps_batch_runs": [ + 124651.7542054606, + 116179.10171446901, + 99298.94919005745 + ], + "qps_batch": 116179.10171446901, + "qps_batch_min": 99298.94919005745, + "qps_batch_max": 124651.7542054606, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6503, + "qps_1t_runs": [ + 8997.951993443592, + 9414.086148625704, + 9036.0928464684 + ], + "qps_1t": 9036.0928464684, + "qps_1t_min": 8997.951993443592, + "qps_1t_max": 9414.086148625704, + "median_ms": 0.1066999975591898, + "p95_ms": 0.13449997641146183, + "p99_ms": 0.17090502078644942, + "mean_ms": 0.10934250022789153, + "knob": 8, + "batch": { + "recall": 0.6503, + "qps_batch_runs": [ + 76763.05559563964, + 86381.52109212214, + 84703.76978550976 + ], + "qps_batch": 84703.76978550976, + "qps_batch_min": 76763.05559563964, + "qps_batch_max": 86381.52109212214, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.75515, + "qps_1t_runs": [ + 6280.884320968016, + 5829.1329518186985, + 6072.218710602774 + ], + "qps_1t": 6072.218710602774, + "qps_1t_min": 5829.1329518186985, + "qps_1t_max": 6280.884320968016, + "median_ms": 0.15670002903789282, + "p95_ms": 0.19890000112354755, + "p99_ms": 0.2494169783312831, + "mean_ms": 0.16514993316377513, + "knob": 16, + "batch": { + "recall": 0.75515, + "qps_batch_runs": [ + 38405.26001015603, + 35027.8646929996, + 40084.417782680204 + ], + "qps_batch": 38405.26001015603, + "qps_batch_min": 35027.8646929996, + "qps_batch_max": 40084.417782680204, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8439, + "qps_1t_runs": [ + 3760.506864140523, + 3838.3295611262834, + 3701.0252573141947 + ], + "qps_1t": 3760.506864140523, + "qps_1t_min": 3701.0252573141947, + "qps_1t_max": 3838.3295611262834, + "median_ms": 0.25549999554641545, + "p95_ms": 0.324005022412166, + "p99_ms": 0.3853259654715663, + "mean_ms": 0.26554899978024576, + "knob": 32, + "batch": { + "recall": 0.8439, + "qps_batch_runs": [ + 23326.60749431601, + 24435.65846811203, + 22952.889184927895 + ], + "qps_batch": 23326.60749431601, + "qps_batch_min": 22952.889184927895, + "qps_batch_max": 24435.65846811203, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.90965, + "qps_1t_runs": [ + 2080.7086007328244, + 2078.0254932820503, + 2072.0381349543754 + ], + "qps_1t": 2078.0254932820503, + "qps_1t_min": 2072.0381349543754, + "qps_1t_max": 2080.7086007328244, + "median_ms": 0.458099995739758, + "p95_ms": 0.5707149801310152, + "p99_ms": 0.6885090039577337, + "mean_ms": 0.4814827171309541, + "knob": 64, + "batch": { + "recall": 0.90965, + "qps_batch_runs": [ + 10561.976356621335, + 11828.524250734354, + 11707.936167779182 + ], + "qps_batch": 11707.936167779182, + "qps_batch_min": 10561.976356621335, + "qps_batch_max": 11828.524250734354, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9549, + "qps_1t_runs": [ + 1102.7866649101884, + 1153.5756408268974, + 1115.5847295945691 + ], + "qps_1t": 1115.5847295945691, + "qps_1t_min": 1102.7866649101884, + "qps_1t_max": 1153.5756408268974, + "median_ms": 0.8658000151626766, + "p95_ms": 1.0673050332115963, + "p99_ms": 1.2555330136092386, + "mean_ms": 0.890018166557032, + "knob": 128, + "batch": { + "recall": 0.9549, + "qps_batch_runs": [ + 5728.257753342597, + 5564.328647498335, + 5727.952609826657 + ], + "qps_batch": 5727.952609826657, + "qps_batch_min": 5564.328647498335, + "qps_batch_max": 5728.257753342597, + "threads": 32 + } + }, + { + "nq": 1702, + "recall_queries": 1702, + "recall": 0.9840775558166862, + "qps_1t_runs": [ + 566.3842996186253, + 567.0691171290053, + 570.2379160019698 + ], + "qps_1t": 567.0691171290053, + "qps_1t_min": 566.3842996186253, + "qps_1t_max": 570.2379160019698, + "median_ms": 1.7172999796457589, + "p95_ms": 2.000599983148277, + "p99_ms": 2.4252599629107845, + "mean_ms": 1.7608977087543196, + "knob": 256, + "batch": { + "recall": 0.9838, + "qps_batch_runs": [ + 2529.3963278486594, + 2808.9942873922123, + 2721.057707816942 + ], + "qps_batch": 2721.057707816942, + "qps_batch_min": 2529.3963278486594, + "qps_batch_max": 2808.9942873922123, + "threads": 32 + } + }, + { + "nq": 377, + "recall_queries": 1000, + "recall": 0.9999, + "qps_1t_runs": [ + 120.67298137777473, + 119.72903636238806, + 117.61160101239489 + ], + "qps_1t": 119.72903636238806, + "qps_1t_min": 117.61160101239489, + "qps_1t_max": 120.67298137777473, + "median_ms": 8.104599954094738, + "p95_ms": 9.462000016355887, + "p99_ms": 12.528770003700634, + "mean_ms": 8.380538196023801, + "knob": 1265, + "batch": { + "recall": 0.99995, + "qps_batch_runs": [ + 502.46965090218185, + 508.4982128193498, + 505.0018922478272 + ], + "qps_batch": 505.0018922478272, + "qps_batch_min": 502.46965090218185, + "qps_batch_max": 508.4982128193498, + "threads": 32 + } + } + ], + "knob_name": "nprobe", + "mem_after_sweep": { + "rss_mb": 273.546875, + "peak_mb": 496.47265625 + }, + "ref_knob": 128, + "ref_recall_pre": 0.9549, + "mutation": { + "ref_knob": 128, + "recall_before": 0.9549, + "insert_10k": { + "seconds": 0.04946050001308322, + "per_s": 202181.53875021107, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.04727209999691695, + "per_s": 211541.2685421674, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.0020620999857783318, + "per_s": 4849425.376541835, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 329.33203125, + "peak_mb": 681.0625 + }, + "first_query_after_ms": 1.2323000119067729, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.951, + "qps_1t_runs": [ + 1061.7292582537762, + 1121.1331069947676, + 1059.470853151706 + ], + "qps_1t": 1061.7292582537762, + "qps_1t_min": 1059.470853151706, + "qps_1t_max": 1121.1331069947676, + "median_ms": 0.8800000068731606, + "p95_ms": 1.1297499877400698, + "p99_ms": 1.6815099975792716, + "mean_ms": 0.9258939498880258 + }, + "recall_after": 0.951 + }, + "saveload": { + "save_s": 0.05474939994746819, + "disk_mb": 149.110125541687, + "load_s": 0.09070970001630485, + "first_query_after_load_ms": 1.2865000171586871, + "time_to_ready_s": 0.09199620003346354 + }, + "mem_end": { + "rss_mb": 246.12890625, + "peak_mb": 681.0625 + }, + "status": "ok", + "finished": "2026-11-16 07:54:14", + "load_before_pct": 9.534375000000008, + "load_contaminated": false, + "wall_s": 65.90399694442749, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/faiss_ivfflat__text384__100000__filter.json b/benchmarks/competitive/results/raw/faiss_ivfflat__text384__100000__filter.json new file mode 100644 index 0000000..660c3f4 --- /dev/null +++ b/benchmarks/competitive/results/raw/faiss_ivfflat__text384__100000__filter.json @@ -0,0 +1,186 @@ +{ + "system": "faiss_ivfflat", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 11:43:33", + "filter": { + "build_with_metadata_s": 1.6864017000771128, + "mem": { + "rss_mb": 273.50390625, + "peak_mb": 494.09375 + }, + "subsets": { + "f10": [ + { + "knob": 2, + "nq": 2000, + "recall": 0.28885, + "qps_1t": 18864.489932928645, + "qps_1t_runs": [ + 18864.489932928645, + 18109.16743247217, + 19321.095468209747 + ], + "median_ms": 0.05059997783973813, + "p95_ms": 0.0674049893859774, + "p99_ms": 0.08729996683541685, + "mean_ms": 0.05332906647042061 + }, + { + "knob": 8, + "nq": 2000, + "recall": 0.52915, + "qps_1t": 15522.997005674655, + "qps_1t_runs": [ + 15518.347649691634, + 15659.19331034095, + 15522.997005674655 + ], + "median_ms": 0.0619000056758523, + "p95_ms": 0.07750000804662704, + "p99_ms": 0.10620199376717214, + "mean_ms": 0.06424021668499336 + }, + { + "knob": 32, + "nq": 2000, + "recall": 0.77835, + "qps_1t": 6741.713155971033, + "qps_1t_runs": [ + 7703.9821735060095, + 6741.713155971033, + 6637.891430592654 + ], + "median_ms": 0.152900000102818, + "p95_ms": 0.19241500995121902, + "p99_ms": 0.22960000205785036, + "mean_ms": 0.14292783334773654 + }, + { + "knob": 64, + "nq": 2000, + "recall": 0.8687, + "qps_1t": 4476.739638094907, + "qps_1t_runs": [ + 4476.739638094907, + 4428.985318577982, + 4478.646372956246 + ], + "median_ms": 0.2300000051036477, + "p95_ms": 0.2772050123894587, + "p99_ms": 0.322203008690849, + "mean_ms": 0.22414798342894451 + }, + { + "knob": 1265, + "nq": 1084, + "recall": 1.0, + "qps_1t": 388.2701160434854, + "qps_1t_runs": [ + 401.68952670004603, + 369.6673733555414, + 388.2701160434854 + ], + "median_ms": 2.6224000030197203, + "p95_ms": 3.2083049823995675, + "p99_ms": 3.5304290201747786, + "mean_ms": 2.5900486785783015 + } + ], + "f01": [ + { + "knob": 2, + "nq": 2000, + "recall": 0.1001, + "qps_1t": 13077.237460110497, + "qps_1t_runs": [ + 13077.237460110497, + 14393.65645248317, + 12968.150237898188 + ], + "median_ms": 0.07670000195503235, + "p95_ms": 0.08709996764082462, + "p99_ms": 0.0955999712459743, + "mean_ms": 0.07435193321240756 + }, + { + "knob": 8, + "nq": 2000, + "recall": 0.26595, + "qps_1t": 12459.40565021944, + "qps_1t_runs": [ + 12457.131851567365, + 12517.320862792621, + 12459.40565021944 + ], + "median_ms": 0.08190004155039787, + "p95_ms": 0.09240000508725643, + "p99_ms": 0.10210102074779573, + "mean_ms": 0.08014175015462872 + }, + { + "knob": 32, + "nq": 2000, + "recall": 0.55835, + "qps_1t": 9902.176417425559, + "qps_1t_runs": [ + 9793.134730328331, + 9902.176417425559, + 10347.48938370298 + ], + "median_ms": 0.10190001921728253, + "p95_ms": 0.11519994877744466, + "p99_ms": 0.12640303059015429, + "mean_ms": 0.09991401658044197 + }, + { + "knob": 64, + "nq": 2000, + "recall": 0.7189, + "qps_1t": 8412.063329895049, + "qps_1t_runs": [ + 8412.063329895049, + 8220.196039263788, + 8874.271951516337 + ], + "median_ms": 0.12469995999708772, + "p95_ms": 0.1466000103391707, + "p99_ms": 0.17181002709548943, + "mean_ms": 0.11773793297470547 + }, + { + "knob": 1265, + "nq": 2000, + "recall": 1.0, + "qps_1t": 1629.7196985702874, + "qps_1t_runs": [ + 1606.3573184628947, + 1634.6548127756025, + 1629.7196985702874 + ], + "median_ms": 0.6629499839618802, + "p95_ms": 0.7409099955111742, + "p99_ms": 0.8447539713233722, + "mean_ms": 0.6159596335298071 + } + ] + } + }, + "mem_end": { + "rss_mb": 82.65625, + "peak_mb": 494.09375 + }, + "status": "ok", + "finished": "2026-11-16 11:43:52", + "load_before_pct": 7.553124999999996, + "load_contaminated": false, + "wall_s": 20.08293342590332, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/hnswlib__glove__1000000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/hnswlib__glove__1000000__core+mutation+saveload.json new file mode 100644 index 0000000..ac9fbc2 --- /dev/null +++ b/benchmarks/competitive/results/raw/hnswlib__glove__1000000__core+mutation+saveload.json @@ -0,0 +1,463 @@ +{ + "system": "hnswlib", + "ds": "glove", + "n": 1000000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 10:20:29", + "mem_baseline": { + "rss_mb": 35.22265625, + "peak_mb": 35.22265625 + }, + "version": "hnswlib 0.8.0", + "build": { + "ingest_s": 50.97159750020364, + "index_s": 3.900029696524143e-06, + "total_s": 50.97160140023334, + "first_query_ms": 2.3651000228710473, + "ready_s": 50.97396650025621 + }, + "mem_after_build": { + "rss_mb": 783.9765625, + "peak_mb": 792.1015625 + }, + "mem_after_first_query": { + "rss_mb": 783.9921875, + "peak_mb": 792.1015625 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.4829, + "qps_1t_runs": [ + 24853.73573143744, + 28305.478216910997, + 27767.287816735425 + ], + "qps_1t": 27767.287816735425, + "qps_1t_min": 24853.73573143744, + "qps_1t_max": 28305.478216910997, + "median_ms": 0.03459997242316604, + "p95_ms": 0.0565050228033215, + "p99_ms": 0.07450298522599048, + "mean_ms": 0.03719261679604339, + "knob": 10, + "batch": { + "recall": 0.4829, + "qps_batch_runs": [ + 146679.18347670152, + 141004.94194485634, + 138433.2129237236 + ], + "qps_batch": 141004.94194485634, + "qps_batch_min": 138433.2129237236, + "qps_batch_max": 146679.18347670152, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.56575, + "qps_1t_runs": [ + 21474.296079339165, + 23346.51252732373, + 22942.33110313481 + ], + "qps_1t": 22942.33110313481, + "qps_1t_min": 21474.296079339165, + "qps_1t_max": 23346.51252732373, + "median_ms": 0.04160002572461963, + "p95_ms": 0.06470502412412317, + "p99_ms": 0.07860101643018427, + "mean_ms": 0.044329266742958374, + "knob": 16, + "batch": { + "recall": 0.56575, + "qps_batch_runs": [ + 105028.77786889613, + 82217.57232420969, + 113747.52594929886 + ], + "qps_batch": 105028.77786889613, + "qps_batch_min": 82217.57232420969, + "qps_batch_max": 113747.52594929886, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.63085, + "qps_1t_runs": [ + 17787.580990259372, + 15586.42355286752, + 17334.537539932342 + ], + "qps_1t": 17334.537539932342, + "qps_1t_min": 15586.42355286752, + "qps_1t_max": 17787.580990259372, + "median_ms": 0.05400000372901559, + "p95_ms": 0.08020002860575914, + "p99_ms": 0.09729998419061303, + "mean_ms": 0.059355233068345115, + "knob": 24, + "batch": { + "recall": 0.63085, + "qps_batch_runs": [ + 82463.68491743511, + 85007.26817182369, + 78122.86383128668 + ], + "qps_batch": 82463.68491743511, + "qps_batch_min": 78122.86383128668, + "qps_batch_max": 85007.26817182369, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6724, + "qps_1t_runs": [ + 13831.669807020386, + 13632.075732251442, + 14170.357475793706 + ], + "qps_1t": 13831.669807020386, + "qps_1t_min": 13632.075732251442, + "qps_1t_max": 14170.357475793706, + "median_ms": 0.06749999010935426, + "p95_ms": 0.10120501101482661, + "p99_ms": 0.12520202202722436, + "mean_ms": 0.07207470022452374, + "knob": 32, + "batch": { + "recall": 0.6724, + "qps_batch_runs": [ + 54492.79467104369, + 67729.3400241416, + 69469.04818270568 + ], + "qps_batch": 67729.3400241416, + "qps_batch_min": 54492.79467104369, + "qps_batch_max": 69469.04818270568, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7266, + "qps_1t_runs": [ + 10123.547759460356, + 10275.34853224859, + 10617.638690352958 + ], + "qps_1t": 10275.34853224859, + "qps_1t_min": 10123.547759460356, + "qps_1t_max": 10617.638690352958, + "median_ms": 0.08999998681247234, + "p95_ms": 0.12709998118225485, + "p99_ms": 0.15799998072907329, + "mean_ms": 0.09676093323893535, + "knob": 48, + "batch": { + "recall": 0.7266, + "qps_batch_runs": [ + 51812.531929188146, + 55714.18622761616, + 54141.118876686385 + ], + "qps_batch": 54141.118876686385, + "qps_batch_min": 51812.531929188146, + "qps_batch_max": 55714.18622761616, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.76205, + "qps_1t_runs": [ + 8779.36147075755, + 8681.222448936891, + 8081.444813229237 + ], + "qps_1t": 8681.222448936891, + "qps_1t_min": 8081.444813229237, + "qps_1t_max": 8779.36147075755, + "median_ms": 0.11070002801716328, + "p95_ms": 0.15420500712934884, + "p99_ms": 0.1892010431038216, + "mean_ms": 0.11761163340027754, + "knob": 64, + "batch": { + "recall": 0.76205, + "qps_batch_runs": [ + 35241.98025144597, + 45377.09502003826, + 44181.710554651 + ], + "qps_batch": 44181.710554651, + "qps_batch_min": 35241.98025144597, + "qps_batch_max": 45377.09502003826, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8022, + "qps_1t_runs": [ + 6226.710156250394, + 5702.889708318204, + 6005.077287115498 + ], + "qps_1t": 6005.077287115498, + "qps_1t_min": 5702.889708318204, + "qps_1t_max": 6226.710156250394, + "median_ms": 0.15820004045963287, + "p95_ms": 0.2236050117062405, + "p99_ms": 0.44870404002722364, + "mean_ms": 0.1674913000994517, + "knob": 96, + "batch": { + "recall": 0.8022, + "qps_batch_runs": [ + 29464.97497352799, + 31593.23082658677, + 32031.929431727487 + ], + "qps_batch": 31593.23082658677, + "qps_batch_min": 29464.97497352799, + "qps_batch_max": 32031.929431727487, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.82825, + "qps_1t_runs": [ + 4751.732664296486, + 5057.079258155335, + 4940.955583864194 + ], + "qps_1t": 4940.955583864194, + "qps_1t_min": 4751.732664296486, + "qps_1t_max": 5057.079258155335, + "median_ms": 0.1984999980777502, + "p95_ms": 0.26020000223070383, + "p99_ms": 0.31050097779370844, + "mean_ms": 0.20352738316675337, + "knob": 128, + "batch": { + "recall": 0.82825, + "qps_batch_runs": [ + 24803.740402503612, + 26478.142292399258, + 25813.646117851957 + ], + "qps_batch": 25813.646117851957, + "qps_batch_min": 24803.740402503612, + "qps_batch_max": 26478.142292399258, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8605, + "qps_1t_runs": [ + 3291.709662069511, + 3346.307526538395, + 2942.3109896850424 + ], + "qps_1t": 3291.709662069511, + "qps_1t_min": 2942.3109896850424, + "qps_1t_max": 3346.307526538395, + "median_ms": 0.2888999879360199, + "p95_ms": 0.3837100113742054, + "p99_ms": 0.8220430277288057, + "mean_ms": 0.314166416763328, + "knob": 192, + "batch": { + "recall": 0.8605, + "qps_batch_runs": [ + 17874.985925617406, + 17880.53122846641, + 17010.52015956229 + ], + "qps_batch": 17874.985925617406, + "qps_batch_min": 17010.52015956229, + "qps_batch_max": 17880.53122846641, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8807, + "qps_1t_runs": [ + 2492.540446495755, + 2466.2606301818614, + 2481.26766831609 + ], + "qps_1t": 2481.26766831609, + "qps_1t_min": 2466.2606301818614, + "qps_1t_max": 2492.540446495755, + "median_ms": 0.3797000099439174, + "p95_ms": 0.5393049737904221, + "p99_ms": 1.1234569968655739, + "mean_ms": 0.4032296835503075, + "knob": 256, + "batch": { + "recall": 0.8807, + "qps_batch_runs": [ + 10539.545693398934, + 10936.414048208235, + 11943.321775795503 + ], + "qps_batch": 10936.414048208235, + "qps_batch_min": 10539.545693398934, + "qps_batch_max": 11943.321775795503, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.91185, + "qps_1t_runs": [ + 1583.6276972738062, + 1659.9601510429572, + 1692.3552089859636 + ], + "qps_1t": 1659.9601510429572, + "qps_1t_min": 1583.6276972738062, + "qps_1t_max": 1692.3552089859636, + "median_ms": 0.5854500050190836, + "p95_ms": 0.8317649888340386, + "p99_ms": 1.3181159668602054, + "mean_ms": 0.6082593831912769, + "knob": 400, + "batch": { + "recall": 0.91185, + "qps_batch_runs": [ + 8483.617075655224, + 8092.674067316121, + 7735.920430702874 + ], + "qps_batch": 8092.674067316121, + "qps_batch_min": 7735.920430702874, + "qps_batch_max": 8483.617075655224, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9367, + "qps_1t_runs": [ + 1100.9520757753453, + 1071.3724203800332, + 1085.8803198148994 + ], + "qps_1t": 1085.8803198148994, + "qps_1t_min": 1071.3724203800332, + "qps_1t_max": 1100.9520757753453, + "median_ms": 0.8930000476539135, + "p95_ms": 1.2401150132063776, + "p99_ms": 1.8217029579682342, + "mean_ms": 0.920866266610877, + "knob": 640, + "batch": { + "recall": 0.9367, + "qps_batch_runs": [ + 4815.955839557642, + 5044.858885123081, + 5013.180906027894 + ], + "qps_batch": 5013.180906027894, + "qps_batch_min": 4815.955839557642, + "qps_batch_max": 5044.858885123081, + "threads": 32 + } + } + ], + "knob_name": "ef", + "mem_after_sweep": { + "rss_mb": 787.6796875, + "peak_mb": 792.1015625 + }, + "ref_knob": 640, + "ref_recall_pre": 0.9367, + "mutation": { + "ref_knob": 640, + "recall_before": 0.9367, + "insert_10k": { + "seconds": 1.0653810999938287, + "per_s": 9386.312559944912, + "batch": 1000 + }, + "update_10k": { + "seconds": 2.6550877999980003, + "per_s": 3766.3537906383103, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.014886100019793957, + "per_s": 671767.6212509025, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 801.5234375, + "peak_mb": 801.60546875 + }, + "first_query_after_ms": 2.2681999835185707, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9355, + "qps_1t_runs": [ + 1032.29434678509, + 1038.462307205691, + 1001.8454492622539 + ], + "qps_1t": 1032.29434678509, + "qps_1t_min": 1001.8454492622539, + "qps_1t_max": 1038.462307205691, + "median_ms": 0.9595500014256686, + "p95_ms": 1.350234978599474, + "p99_ms": 1.802758990088479, + "mean_ms": 0.9766120504646096 + }, + "recall_after": 0.9355 + }, + "saveload": { + "save_s": 0.7770867999643087, + "disk_mb": 528.3318481445312, + "load_s": 3.656790999986697, + "first_query_after_load_ms": 3.2733999541960657, + "time_to_ready_s": 3.660064399940893 + }, + "mem_end": { + "rss_mb": 729.72265625, + "peak_mb": 801.67578125 + }, + "status": "ok", + "finished": "2026-11-16 10:21:57", + "load_before_pct": 6.612500000000003, + "load_contaminated": false, + "wall_s": 88.61154007911682, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/hnswlib__glove__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/hnswlib__glove__100000__core+mutation+saveload.json new file mode 100644 index 0000000..1cecdc4 --- /dev/null +++ b/benchmarks/competitive/results/raw/hnswlib__glove__100000__core+mutation+saveload.json @@ -0,0 +1,468 @@ +{ + "system": "hnswlib", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 08:19:52", + "mem_baseline": { + "rss_mb": 35.20703125, + "peak_mb": 35.20703125 + }, + "version": "hnswlib 0.8.0", + "build": { + "ingest_s": 1.3345952999661677, + "index_s": 2.0999577827751637e-06, + "total_s": 1.3345973999239504, + "first_query_ms": 0.8077000384218991, + "ready_s": 1.3354050999623723, + "total_s_runs": [ + 1.3345973999239504, + 1.2821055001113564, + 1.3087505000294186 + ] + }, + "mem_after_build": { + "rss_mb": 122.84765625, + "peak_mb": 130.97265625 + }, + "mem_after_first_query": { + "rss_mb": 122.8671875, + "peak_mb": 130.97265625 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.54205, + "qps_1t_runs": [ + 44723.819005460275, + 46903.54526299682, + 41330.25599087344 + ], + "qps_1t": 44723.819005460275, + "qps_1t_min": 41330.25599087344, + "qps_1t_max": 46903.54526299682, + "median_ms": 0.021599989850074053, + "p95_ms": 0.031200004741549492, + "p99_ms": 0.0396010105032474, + "mean_ms": 0.022625049954513088, + "knob": 10, + "batch": { + "recall": 0.54205, + "qps_batch_runs": [ + 271506.7273731674, + 373517.6042328868, + 387574.3656137073 + ], + "qps_batch": 373517.6042328868, + "qps_batch_min": 271506.7273731674, + "qps_batch_max": 387574.3656137073, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6244, + "qps_1t_runs": [ + 34289.31990045052, + 36876.69168737768, + 36870.777463584236 + ], + "qps_1t": 36870.777463584236, + "qps_1t_min": 34289.31990045052, + "qps_1t_max": 36876.69168737768, + "median_ms": 0.026900030206888914, + "p95_ms": 0.03789999755099416, + "p99_ms": 0.04439998883754015, + "mean_ms": 0.027800916500079136, + "knob": 16, + "batch": { + "recall": 0.6244, + "qps_batch_runs": [ + 278067.4315904601, + 345172.7590557858, + 328116.28323354566 + ], + "qps_batch": 328116.28323354566, + "qps_batch_min": 278067.4315904601, + "qps_batch_max": 345172.7590557858, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.69495, + "qps_1t_runs": [ + 29071.035227971708, + 27705.858117625008, + 27312.591215687415 + ], + "qps_1t": 27705.858117625008, + "qps_1t_min": 27312.591215687415, + "qps_1t_max": 29071.035227971708, + "median_ms": 0.03460003063082695, + "p95_ms": 0.0475000124424696, + "p99_ms": 0.057500972761772594, + "mean_ms": 0.03570169989446489, + "knob": 24, + "batch": { + "recall": 0.69495, + "qps_batch_runs": [ + 212972.133282918, + 210768.14537390825, + 294399.05868354416 + ], + "qps_batch": 212972.133282918, + "qps_batch_min": 210768.14537390825, + "qps_batch_max": 294399.05868354416, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7392, + "qps_1t_runs": [ + 23316.89868393719, + 23596.84328586468, + 23476.823851824578 + ], + "qps_1t": 23476.823851824578, + "qps_1t_min": 23316.89868393719, + "qps_1t_max": 23596.84328586468, + "median_ms": 0.04160002572461963, + "p95_ms": 0.05520001286640763, + "p99_ms": 0.06500197108834986, + "mean_ms": 0.042620366594443716, + "knob": 32, + "batch": { + "recall": 0.7392, + "qps_batch_runs": [ + 162867.77625371164, + 179768.81694323747, + 171278.3354111458 + ], + "qps_batch": 171278.3354111458, + "qps_batch_min": 162867.77625371164, + "qps_batch_max": 179768.81694323747, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.79375, + "qps_1t_runs": [ + 15501.160239324272, + 17468.88136227897, + 17283.799087645886 + ], + "qps_1t": 17283.799087645886, + "qps_1t_min": 15501.160239324272, + "qps_1t_max": 17468.88136227897, + "median_ms": 0.05879998207092285, + "p95_ms": 0.07819995807949454, + "p99_ms": 0.08970198454335336, + "mean_ms": 0.05987120007436412, + "knob": 48, + "batch": { + "recall": 0.79375, + "qps_batch_runs": [ + 132860.35036971865, + 153449.54578700237, + 94064.08575280246 + ], + "qps_batch": 132860.35036971865, + "qps_batch_min": 94064.08575280246, + "qps_batch_max": 153449.54578700237, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8287, + "qps_1t_runs": [ + 14084.21943420152, + 12806.950650724291, + 12917.760342862528 + ], + "qps_1t": 12917.760342862528, + "qps_1t_min": 12806.950650724291, + "qps_1t_max": 14084.21943420152, + "median_ms": 0.07289997301995754, + "p95_ms": 0.0947999651543796, + "p99_ms": 0.1116060349158944, + "mean_ms": 0.07549894988187589, + "knob": 64, + "batch": { + "recall": 0.8287, + "qps_batch_runs": [ + 132275.13230674472, + 155902.87238800808, + 150099.44084168825 + ], + "qps_batch": 150099.44084168825, + "qps_batch_min": 132275.13230674472, + "qps_batch_max": 155902.87238800808, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.87055, + "qps_1t_runs": [ + 9613.332494154269, + 7599.156672345472, + 9091.917473638105 + ], + "qps_1t": 9091.917473638105, + "qps_1t_min": 7599.156672345472, + "qps_1t_max": 9613.332494154269, + "median_ms": 0.10270002530887723, + "p95_ms": 0.1342049741651863, + "p99_ms": 0.2162590157240664, + "mean_ms": 0.11520118331342626, + "knob": 96, + "batch": { + "recall": 0.87055, + "qps_batch_runs": [ + 103974.42219421266, + 100314.98903498605, + 108647.82348704936 + ], + "qps_batch": 103974.42219421266, + "qps_batch_min": 100314.98903498605, + "qps_batch_max": 108647.82348704936, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.89495, + "qps_1t_runs": [ + 7640.566370589797, + 7197.195987622838, + 7769.594247664829 + ], + "qps_1t": 7640.566370589797, + "qps_1t_min": 7197.195987622838, + "qps_1t_max": 7769.594247664829, + "median_ms": 0.13080000644549727, + "p95_ms": 0.16399997111875564, + "p99_ms": 0.19450502586551022, + "mean_ms": 0.13284339980843166, + "knob": 128, + "batch": { + "recall": 0.89495, + "qps_batch_runs": [ + 59020.379746301645, + 71296.67263987656, + 88748.46908499993 + ], + "qps_batch": 71296.67263987656, + "qps_batch_min": 59020.379746301645, + "qps_batch_max": 88748.46908499993, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9278, + "qps_1t_runs": [ + 5038.43696484007, + 5168.247122880161, + 5112.2235319167785 + ], + "qps_1t": 5112.2235319167785, + "qps_1t_min": 5038.43696484007, + "qps_1t_max": 5168.247122880161, + "median_ms": 0.18969998927786946, + "p95_ms": 0.23820500937290492, + "p99_ms": 0.3203219943679874, + "mean_ms": 0.1958576833906894, + "knob": 192, + "batch": { + "recall": 0.9278, + "qps_batch_runs": [ + 56815.59929728225, + 63882.66026907072, + 65531.65828765434 + ], + "qps_batch": 63882.66026907072, + "qps_batch_min": 56815.59929728225, + "qps_batch_max": 65531.65828765434, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.94665, + "qps_1t_runs": [ + 4050.3326978571886, + 3983.4114815115527, + 3913.2579445526244 + ], + "qps_1t": 3983.4114815115527, + "qps_1t_min": 3913.2579445526244, + "qps_1t_max": 4050.3326978571886, + "median_ms": 0.24289998691529036, + "p95_ms": 0.30960002914071083, + "p99_ms": 0.41153204801958126, + "mean_ms": 0.25115864945109934, + "knob": 256, + "batch": { + "recall": 0.94665, + "qps_batch_runs": [ + 42867.21663636747, + 51909.76006280174, + 52287.03484600862 + ], + "qps_batch": 51909.76006280174, + "qps_batch_min": 42867.21663636747, + "qps_batch_max": 52287.03484600862, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9692, + "qps_1t_runs": [ + 2708.985725087262, + 2730.634882965438, + 2669.608217338934 + ], + "qps_1t": 2708.985725087262, + "qps_1t_min": 2669.608217338934, + "qps_1t_max": 2730.634882965438, + "median_ms": 0.36604999331757426, + "p95_ms": 0.4523000097833574, + "p99_ms": 0.5730289977509535, + "mean_ms": 0.3699812831667562, + "knob": 400, + "batch": { + "recall": 0.9692, + "qps_batch_runs": [ + 33451.36087631815, + 35141.60308682183, + 35275.48386166561 + ], + "qps_batch": 35141.60308682183, + "qps_batch_min": 33451.36087631815, + "qps_batch_max": 35275.48386166561, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9843, + "qps_1t_runs": [ + 1808.9008385495708, + 1750.1215026648167, + 1769.4494125945832 + ], + "qps_1t": 1769.4494125945832, + "qps_1t_min": 1750.1215026648167, + "qps_1t_max": 1808.9008385495708, + "median_ms": 0.5594499816652387, + "p95_ms": 0.6926250352989883, + "p99_ms": 0.8525169722270224, + "mean_ms": 0.5631194501571978, + "knob": 640, + "batch": { + "recall": 0.9843, + "qps_batch_runs": [ + 21716.422601259816, + 23110.726962525463, + 23068.928816427026 + ], + "qps_batch": 23068.928816427026, + "qps_batch_min": 21716.422601259816, + "qps_batch_max": 23110.726962525463, + "threads": 32 + } + } + ], + "knob_name": "ef", + "mem_after_sweep": { + "rss_mb": 128.578125, + "peak_mb": 130.97265625 + }, + "ref_knob": 400, + "ref_recall_pre": 0.9692, + "mutation": { + "ref_knob": 400, + "recall_before": 0.9692, + "insert_10k": { + "seconds": 0.22615729999961331, + "per_s": 44217.011787888776, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.9343183999881148, + "per_s": 10702.989473531943, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.008049399999435991, + "per_s": 1242328.6208538134, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 141.1171875, + "peak_mb": 218.20703125 + }, + "first_query_after_ms": 0.6444000173360109, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9686, + "qps_1t_runs": [ + 2307.1608589658426, + 2289.887379307556, + 2229.330072772871 + ], + "qps_1t": 2289.887379307556, + "qps_1t_min": 2229.330072772871, + "qps_1t_max": 2307.1608589658426, + "median_ms": 0.43440001900307834, + "p95_ms": 0.5435300030512736, + "p99_ms": 0.6905010191258043, + "mean_ms": 0.4395670331335471 + }, + "recall_after": 0.9686 + }, + "saveload": { + "save_s": 0.03637069999240339, + "disk_mb": 57.546424865722656, + "load_s": 0.141510899993591, + "first_query_after_load_ms": 0.7082999800331891, + "time_to_ready_s": 0.1422191999736242 + }, + "mem_end": { + "rss_mb": 125.56640625, + "peak_mb": 218.20703125 + }, + "status": "ok", + "finished": "2026-11-16 08:20:13", + "load_before_pct": 4.259374999999998, + "load_contaminated": false, + "wall_s": 21.079107999801636, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/hnswlib__glove__100000__filter.json b/benchmarks/competitive/results/raw/hnswlib__glove__100000__filter.json new file mode 100644 index 0000000..f1c8989 --- /dev/null +++ b/benchmarks/competitive/results/raw/hnswlib__glove__100000__filter.json @@ -0,0 +1,186 @@ +{ + "system": "hnswlib", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 08:48:22", + "filter": { + "build_with_metadata_s": 1.3072742999647744, + "mem": { + "rss_mb": 122.0703125, + "peak_mb": 130.19921875 + }, + "subsets": { + "f10": [ + { + "knob": 16, + "nq": 2000, + "recall": 0.8166, + "qps_1t": 3677.6387820146956, + "qps_1t_runs": [ + 3759.364577268687, + 3532.096783489829, + 3677.6387820146956 + ], + "median_ms": 0.2583499881438911, + "p95_ms": 0.394509980105795, + "p99_ms": 0.5736070021521301, + "mean_ms": 0.27367798321453546 + }, + { + "knob": 48, + "nq": 2000, + "recall": 0.93485, + "qps_1t": 1354.4558920117986, + "qps_1t_runs": [ + 1316.3519541187577, + 1383.4948914627462, + 1354.4558920117986 + ], + "median_ms": 0.6934000120963901, + "p95_ms": 1.02491500438191, + "p99_ms": 2.0306119776796594, + "mean_ms": 0.7402620835540196 + }, + { + "knob": 96, + "nq": 2000, + "recall": 0.975, + "qps_1t": 728.8365452804516, + "qps_1t_runs": [ + 728.8365452804516, + 739.2378702419824, + 721.7520067027885 + ], + "median_ms": 1.3130000152159482, + "p95_ms": 1.8889250030042608, + "p99_ms": 3.229097022558567, + "mean_ms": 1.370103916793596 + }, + { + "knob": 192, + "nq": 1268, + "recall": 0.9927444794952681, + "qps_1t": 384.4068922023777, + "qps_1t_runs": [ + 384.4068922023777, + 375.6272536580646, + 389.81977657218965 + ], + "median_ms": 2.4999500019475818, + "p95_ms": 3.4383249963866547, + "p99_ms": 5.298062009387644, + "mean_ms": 2.609637276877034 + }, + { + "knob": 640, + "nq": 430, + "recall": 1.0, + "qps_1t": 137.21710496936936, + "qps_1t_runs": [ + 136.7346556167165, + 137.78870658602034, + 137.21710496936936 + ], + "median_ms": 7.10880002588965, + "p95_ms": 8.725929964566603, + "p99_ms": 12.637714000302358, + "mean_ms": 7.286214961421265 + } + ], + "f01": [ + { + "knob": 16, + "nq": 1186, + "recall": 0.9507588532883643, + "qps_1t": 448.8249489720564, + "qps_1t_runs": [ + 464.19604221110353, + 448.8249489720564, + 441.90245786025764 + ], + "median_ms": 2.122300007613376, + "p95_ms": 2.9921900160843506, + "p99_ms": 3.8339270139112958, + "mean_ms": 2.2150817320624783 + }, + { + "knob": 48, + "nq": 455, + "recall": 0.9958241758241758, + "qps_1t": 166.91723210228446, + "qps_1t_runs": [ + 166.91723210228446, + 162.06699750473518, + 174.38353408205953 + ], + "median_ms": 5.707699980121106, + "p95_ms": 7.627739978488534, + "p99_ms": 14.033827986568172, + "mean_ms": 5.965255750317618 + }, + { + "knob": 96, + "nq": 286, + "recall": 0.9996503496503496, + "qps_1t": 95.20224302845789, + "qps_1t_runs": [ + 95.04263593748968, + 96.66358540018246, + 95.20224302845789 + ], + "median_ms": 10.192600020673126, + "p95_ms": 12.572064984124152, + "p99_ms": 19.371890987385974, + "mean_ms": 10.45690174655509 + }, + { + "knob": 192, + "nq": 172, + "recall": 1.0, + "qps_1t": 57.15175459003141, + "qps_1t_runs": [ + 57.013561371532646, + 57.15175459003141, + 57.40438466791265 + ], + "median_ms": 16.920199996093288, + "p95_ms": 21.02540002670139, + "p99_ms": 27.633194957161344, + "mean_ms": 17.48574476588929 + }, + { + "knob": 640, + "nq": 79, + "recall": 1.0, + "qps_1t": 25.329869448785068, + "qps_1t_runs": [ + 25.4083418025337, + 24.806453922089396, + 25.329869448785068 + ], + "median_ms": 38.42579998308793, + "p95_ms": 47.238399996422224, + "p99_ms": 59.03204800793901, + "mean_ms": 39.71610843883788 + } + ] + } + }, + "mem_end": { + "rss_mb": 40.16796875, + "peak_mb": 130.19921875 + }, + "status": "ok", + "finished": "2026-11-16 08:49:42", + "load_before_pct": 3.2156249999999984, + "load_contaminated": false, + "wall_s": 80.00535774230957, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/hnswlib__sift__1000000__core+saveload.json b/benchmarks/competitive/results/raw/hnswlib__sift__1000000__core+saveload.json new file mode 100644 index 0000000..1f48b03 --- /dev/null +++ b/benchmarks/competitive/results/raw/hnswlib__sift__1000000__core+saveload.json @@ -0,0 +1,420 @@ +{ + "system": "hnswlib", + "ds": "sift", + "n": 1000000, + "d": 128, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "saveload" + ], + "started": "2026-11-16 11:34:21", + "mem_baseline": { + "rss_mb": 35.75, + "peak_mb": 35.75 + }, + "version": "hnswlib 0.8.0", + "build": { + "ingest_s": 60.422680200252216, + "index_s": 3.3999676816165447e-06, + "total_s": 60.4226836002199, + "first_query_ms": 1.896000001579523, + "ready_s": 60.42457960022148 + }, + "mem_after_build": { + "rss_mb": 889.25390625, + "peak_mb": 899.5234375 + }, + "mem_after_first_query": { + "rss_mb": 889.26953125, + "peak_mb": 899.5234375 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6795, + "qps_1t_runs": [ + 22475.091832658785, + 22476.68311615447, + 21929.031050822185 + ], + "qps_1t": 22475.091832658785, + "qps_1t_min": 21929.031050822185, + "qps_1t_max": 22476.68311615447, + "median_ms": 0.04379998426884413, + "p95_ms": 0.062399954185821116, + "p99_ms": 0.07270702917594478, + "mean_ms": 0.04486196687018188, + "knob": 10, + "batch": { + "recall": 0.6795, + "qps_batch_runs": [ + 89007.96166477687, + 80772.50824594033, + 73767.80114057181 + ], + "qps_batch": 80772.50824594033, + "qps_batch_min": 73767.80114057181, + "qps_batch_max": 89007.96166477687, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7752, + "qps_1t_runs": [ + 15590.821951365464, + 17309.543255140707, + 17159.634345843588 + ], + "qps_1t": 17159.634345843588, + "qps_1t_min": 15590.821951365464, + "qps_1t_max": 17309.543255140707, + "median_ms": 0.05849997978657484, + "p95_ms": 0.08780002826824784, + "p99_ms": 0.11110297287814326, + "mean_ms": 0.06006273323631225, + "knob": 16, + "batch": { + "recall": 0.7752, + "qps_batch_runs": [ + 79260.65661241273, + 72842.76165548181, + 70652.90347222987 + ], + "qps_batch": 72842.76165548181, + "qps_batch_min": 70652.90347222987, + "qps_batch_max": 79260.65661241273, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8472, + "qps_1t_runs": [ + 14934.701728235763, + 13360.67822834269, + 13153.07489103741 + ], + "qps_1t": 13360.67822834269, + "qps_1t_min": 13153.07489103741, + "qps_1t_max": 14934.701728235763, + "median_ms": 0.07319997530430555, + "p95_ms": 0.09820499108172955, + "p99_ms": 0.11510100332088771, + "mean_ms": 0.07261083316795218, + "knob": 24, + "batch": { + "recall": 0.8472, + "qps_batch_runs": [ + 58473.71895197593, + 52926.715000554956, + 51003.103559514704 + ], + "qps_batch": 52926.715000554956, + "qps_batch_min": 51003.103559514704, + "qps_batch_max": 58473.71895197593, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.89, + "qps_1t_runs": [ + 11346.005010787292, + 11456.853503939941, + 11208.469128958717 + ], + "qps_1t": 11346.005010787292, + "qps_1t_min": 11208.469128958717, + "qps_1t_max": 11456.853503939941, + "median_ms": 0.08900003740563989, + "p95_ms": 0.11650501401163639, + "p99_ms": 0.13640099088661375, + "mean_ms": 0.08821299972866352, + "knob": 32, + "batch": { + "recall": 0.89, + "qps_batch_runs": [ + 55623.84931818337, + 58617.506080550585, + 52415.571658493 + ], + "qps_batch": 55623.84931818337, + "qps_batch_min": 52415.571658493, + "qps_batch_max": 58617.506080550585, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9339, + "qps_1t_runs": [ + 8629.08440927331, + 8445.81041408436, + 8865.507562237099 + ], + "qps_1t": 8629.08440927331, + "qps_1t_min": 8445.81041408436, + "qps_1t_max": 8865.507562237099, + "median_ms": 0.11729996185749769, + "p95_ms": 0.15490501536987725, + "p99_ms": 0.18040103430394086, + "mean_ms": 0.11569525014298658, + "knob": 48, + "batch": { + "recall": 0.9339, + "qps_batch_runs": [ + 44422.8253599564, + 35113.34587798847, + 41343.58377407923 + ], + "qps_batch": 41343.58377407923, + "qps_batch_min": 35113.34587798847, + "qps_batch_max": 44422.8253599564, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.95755, + "qps_1t_runs": [ + 6689.793275359951, + 6875.940507499319, + 6764.684698092554 + ], + "qps_1t": 6764.684698092554, + "qps_1t_min": 6689.793275359951, + "qps_1t_max": 6875.940507499319, + "median_ms": 0.15119998715817928, + "p95_ms": 0.1933050167281181, + "p99_ms": 0.22629997751209885, + "mean_ms": 0.14758088329108432, + "knob": 64, + "batch": { + "recall": 0.95755, + "qps_batch_runs": [ + 34527.823403220515, + 35560.67628041415, + 35637.66471861058 + ], + "qps_batch": 35560.67628041415, + "qps_batch_min": 34527.823403220515, + "qps_batch_max": 35637.66471861058, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9768, + "qps_1t_runs": [ + 4866.216761466154, + 4879.756700329306, + 4847.488307564959 + ], + "qps_1t": 4866.216761466154, + "qps_1t_min": 4847.488307564959, + "qps_1t_max": 4879.756700329306, + "median_ms": 0.21044997265562415, + "p95_ms": 0.2673999988473952, + "p99_ms": 0.3121029905742035, + "mean_ms": 0.20557303334741542, + "knob": 96, + "batch": { + "recall": 0.9768, + "qps_batch_runs": [ + 25048.092333879147, + 24020.15771325015, + 27240.645233337054 + ], + "qps_batch": 25048.092333879147, + "qps_batch_min": 24020.15771325015, + "qps_batch_max": 27240.645233337054, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9862, + "qps_1t_runs": [ + 3729.2608961709225, + 3956.6250975409653, + 4162.767545359252 + ], + "qps_1t": 3956.6250975409653, + "qps_1t_min": 3729.2608961709225, + "qps_1t_max": 4162.767545359252, + "median_ms": 0.25670003378763795, + "p95_ms": 0.3282999969087541, + "p99_ms": 0.37360305024776613, + "mean_ms": 0.25370503297502484, + "knob": 128, + "batch": { + "recall": 0.9862, + "qps_batch_runs": [ + 24450.146768424864, + 22507.56816715879, + 24599.429052640924 + ], + "qps_batch": 24450.146768424864, + "qps_batch_min": 22507.56816715879, + "qps_batch_max": 24599.429052640924, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99365, + "qps_1t_runs": [ + 2702.800223433291, + 2717.1328775732827, + 2840.061109111305 + ], + "qps_1t": 2717.1328775732827, + "qps_1t_min": 2702.800223433291, + "qps_1t_max": 2840.061109111305, + "median_ms": 0.374099996406585, + "p95_ms": 0.4678050085203722, + "p99_ms": 0.5398049775976689, + "mean_ms": 0.3633755833725445, + "knob": 192, + "batch": { + "recall": 0.99365, + "qps_batch_runs": [ + 14539.131344918409, + 14951.270073134194, + 15398.462157050204 + ], + "qps_batch": 14951.270073134194, + "qps_batch_min": 14539.131344918409, + "qps_batch_max": 15398.462157050204, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99655, + "qps_1t_runs": [ + 2229.0212344468423, + 2159.2796956749307, + 1950.4551245233356 + ], + "qps_1t": 2159.2796956749307, + "qps_1t_min": 1950.4551245233356, + "qps_1t_max": 2229.0212344468423, + "median_ms": 0.48809999134391546, + "p95_ms": 0.6267049990128726, + "p99_ms": 0.7245240447809925, + "mean_ms": 0.47481521671094623, + "knob": 256, + "batch": { + "recall": 0.99655, + "qps_batch_runs": [ + 11588.170330215526, + 11572.165471434459, + 11233.317120954891 + ], + "qps_batch": 11572.165471434459, + "qps_batch_min": 11233.317120954891, + "qps_batch_max": 11588.170330215526, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99865, + "qps_1t_runs": [ + 1411.8767911897073, + 1490.2587747883554, + 1475.1483973311501 + ], + "qps_1t": 1475.1483973311501, + "qps_1t_min": 1411.8767911897073, + "qps_1t_max": 1490.2587747883554, + "median_ms": 0.7050499843899161, + "p95_ms": 0.8956050092820078, + "p99_ms": 0.994307946530171, + "mean_ms": 0.685733133306106, + "knob": 400, + "batch": { + "recall": 0.99865, + "qps_batch_runs": [ + 9635.517287900113, + 8898.06773955238, + 8766.408525530082 + ], + "qps_batch": 8898.06773955238, + "qps_batch_min": 8766.408525530082, + "qps_batch_max": 9635.517287900113, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99935, + "qps_1t_runs": [ + 950.0436192333061, + 975.9859629121419, + 972.9978032743089 + ], + "qps_1t": 972.9978032743089, + "qps_1t_min": 950.0436192333061, + "qps_1t_max": 975.9859629121419, + "median_ms": 1.0636000079102814, + "p95_ms": 1.3488999713445082, + "p99_ms": 1.4980329980608083, + "mean_ms": 1.0349798997340258, + "knob": 640, + "batch": { + "recall": 0.99935, + "qps_batch_runs": [ + 5242.292192309696, + 5333.336177800954, + 5401.266867138309 + ], + "qps_batch": 5333.336177800954, + "qps_batch_min": 5242.292192309696, + "qps_batch_max": 5401.266867138309, + "threads": 32 + } + } + ], + "knob_name": "ef", + "mem_after_sweep": { + "rss_mb": 893.484375, + "peak_mb": 899.5234375 + }, + "ref_knob": 64, + "ref_recall_pre": 0.95755, + "saveload": { + "save_s": 1.3896516999811865, + "disk_mb": 629.9174461364746, + "load_s": 4.90050660003908, + "first_query_after_load_ms": 0.935999967623502, + "time_to_ready_s": 4.901442600006703 + }, + "mem_end": { + "rss_mb": 829.953125, + "peak_mb": 899.5234375 + }, + "status": "ok", + "finished": "2026-11-16 11:35:54", + "load_before_pct": 8.974999999999998, + "load_contaminated": false, + "wall_s": 93.83252835273743, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/hnswlib__sift__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/hnswlib__sift__100000__core+mutation+saveload.json new file mode 100644 index 0000000..a0af1b6 --- /dev/null +++ b/benchmarks/competitive/results/raw/hnswlib__sift__100000__core+mutation+saveload.json @@ -0,0 +1,438 @@ +{ + "system": "hnswlib", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 11:09:13", + "mem_baseline": { + "rss_mb": 35.44921875, + "peak_mb": 35.44921875 + }, + "version": "hnswlib 0.8.0", + "build": { + "ingest_s": 1.1284188999561593, + "index_s": 4.099973011761904e-06, + "total_s": 1.128422999929171, + "first_query_ms": 0.7225999725051224, + "ready_s": 1.1291455999016762, + "total_s_runs": [ + 1.128422999929171, + 1.088209499954246, + 1.1993346998933703 + ] + }, + "mem_after_build": { + "rss_mb": 132.87890625, + "peak_mb": 143.14453125 + }, + "mem_after_first_query": { + "rss_mb": 132.89453125, + "peak_mb": 143.14453125 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7877, + "qps_1t_runs": [ + 44466.479261374996, + 45863.24908288654, + 42336.37541892785 + ], + "qps_1t": 44466.479261374996, + "qps_1t_min": 42336.37541892785, + "qps_1t_max": 45863.24908288654, + "median_ms": 0.021199986804276705, + "p95_ms": 0.030800001695752144, + "p99_ms": 0.043000978766940555, + "mean_ms": 0.022637716834045325, + "knob": 10, + "batch": { + "recall": 0.7877, + "qps_batch_runs": [ + 359725.1687996511, + 422448.94000532664, + 392472.37695326173 + ], + "qps_batch": 392472.37695326173, + "qps_batch_min": 359725.1687996511, + "qps_batch_max": 422448.94000532664, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.86895, + "qps_1t_runs": [ + 34241.06380580334, + 35361.24168269553, + 36359.27349157056 + ], + "qps_1t": 35361.24168269553, + "qps_1t_min": 34241.06380580334, + "qps_1t_max": 36359.27349157056, + "median_ms": 0.027500034775584936, + "p95_ms": 0.03840000135824084, + "p99_ms": 0.04859996261075139, + "mean_ms": 0.02832918328931555, + "knob": 16, + "batch": { + "recall": 0.86895, + "qps_batch_runs": [ + 275546.6144899087, + 342941.40878269455, + 310298.8173090875 + ], + "qps_batch": 310298.8173090875, + "qps_batch_min": 275546.6144899087, + "qps_batch_max": 342941.40878269455, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9228, + "qps_1t_runs": [ + 26531.70817352751, + 27773.573253201233, + 26390.16789665563 + ], + "qps_1t": 26531.70817352751, + "qps_1t_min": 26390.16789665563, + "qps_1t_max": 27773.573253201233, + "median_ms": 0.035999983083456755, + "p95_ms": 0.05080498231109234, + "p99_ms": 0.06810599472373736, + "mean_ms": 0.03719636697011689, + "knob": 24, + "batch": { + "recall": 0.9228, + "qps_batch_runs": [ + 281781.4233230866, + 262598.14541322534, + 281005.43732487067 + ], + "qps_batch": 281005.43732487067, + "qps_batch_min": 262598.14541322534, + "qps_batch_max": 281781.4233230866, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.95025, + "qps_1t_runs": [ + 22404.244233921945, + 22519.707471806014, + 21712.768402343423 + ], + "qps_1t": 22404.244233921945, + "qps_1t_min": 21712.768402343423, + "qps_1t_max": 22519.707471806014, + "median_ms": 0.04369998350739479, + "p95_ms": 0.061400001868605614, + "p99_ms": 0.07870601722970617, + "mean_ms": 0.04503193333706198, + "knob": 32, + "batch": { + "recall": 0.95025, + "qps_batch_runs": [ + 183135.0889854097, + 246035.7483069214, + 192350.2311013815 + ], + "qps_batch": 192350.2311013815, + "qps_batch_min": 183135.0889854097, + "qps_batch_max": 246035.7483069214, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.97625, + "qps_1t_runs": [ + 16464.618393411292, + 15859.736493281327, + 16412.006981227994 + ], + "qps_1t": 16412.006981227994, + "qps_1t_min": 15859.736493281327, + "qps_1t_max": 16464.618393411292, + "median_ms": 0.060250022215768695, + "p95_ms": 0.08149998320732266, + "p99_ms": 0.10349997377488762, + "mean_ms": 0.061573350006559245, + "knob": 48, + "batch": { + "recall": 0.97625, + "qps_batch_runs": [ + 178550.7036492056, + 157970.39619589678, + 175231.08682717508 + ], + "qps_batch": 175231.08682717508, + "qps_batch_min": 157970.39619589678, + "qps_batch_max": 178550.7036492056, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.98785, + "qps_1t_runs": [ + 13235.373926141097, + 12557.442419530878, + 12747.898839214908 + ], + "qps_1t": 12747.898839214908, + "qps_1t_min": 12557.442419530878, + "qps_1t_max": 13235.373926141097, + "median_ms": 0.07499998901039362, + "p95_ms": 0.10520504147280009, + "p99_ms": 0.1367020508041606, + "mean_ms": 0.07787781669564235, + "knob": 64, + "batch": { + "recall": 0.98785, + "qps_batch_runs": [ + 132068.99282907954, + 130072.84107308216, + 130878.06088491467 + ], + "qps_batch": 130878.06088491467, + "qps_batch_min": 130072.84107308216, + "qps_batch_max": 132068.99282907954, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99545, + "qps_1t_runs": [ + 8413.740697985422, + 8801.104723281625, + 8850.32898238257 + ], + "qps_1t": 8801.104723281625, + "qps_1t_min": 8413.740697985422, + "qps_1t_max": 8850.32898238257, + "median_ms": 0.10740000288933516, + "p95_ms": 0.15650002751499414, + "p99_ms": 0.24602401070296817, + "mean_ms": 0.11515514975568901, + "knob": 96, + "batch": { + "recall": 0.99545, + "qps_batch_runs": [ + 90674.98473305057, + 101372.07100820688, + 99853.71426416578 + ], + "qps_batch": 99853.71426416578, + "qps_batch_min": 90674.98473305057, + "qps_batch_max": 101372.07100820688, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99775, + "qps_1t_runs": [ + 7469.088230636029, + 6788.983787934223, + 6287.8627539312665 + ], + "qps_1t": 6788.983787934223, + "qps_1t_min": 6287.8627539312665, + "qps_1t_max": 7469.088230636029, + "median_ms": 0.13689999468624592, + "p95_ms": 0.21040502761024985, + "p99_ms": 0.3733939904486781, + "mean_ms": 0.14673971678712405, + "knob": 128, + "batch": { + "recall": 0.99775, + "qps_batch_runs": [ + 77868.58121009485, + 80264.55203437756, + 83760.86262273634 + ], + "qps_batch": 80264.55203437756, + "qps_batch_min": 77868.58121009485, + "qps_batch_max": 83760.86262273634, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.999, + "qps_1t_runs": [ + 4496.088517143993, + 4893.009454854954, + 4937.852192321759 + ], + "qps_1t": 4893.009454854954, + "qps_1t_min": 4496.088517143993, + "qps_1t_max": 4937.852192321759, + "median_ms": 0.19324998720549047, + "p95_ms": 0.3084249619860203, + "p99_ms": 0.6013419787632355, + "mean_ms": 0.2097686499764677, + "knob": 192, + "batch": { + "recall": 0.999, + "qps_batch_runs": [ + 62395.68220625229, + 53647.20525730502, + 49378.936394904355 + ], + "qps_batch": 53647.20525730502, + "qps_batch_min": 49378.936394904355, + "qps_batch_max": 62395.68220625229, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9997, + "qps_1t_runs": [ + 4377.404015145242, + 4260.5614547100195, + 4274.168237013549 + ], + "qps_1t": 4274.168237013549, + "qps_1t_min": 4260.5614547100195, + "qps_1t_max": 4377.404015145242, + "median_ms": 0.22960000205785036, + "p95_ms": 0.31460000900551677, + "p99_ms": 0.41180399246513855, + "mean_ms": 0.2323734665211911, + "knob": 256, + "batch": { + "recall": 0.9997, + "qps_batch_runs": [ + 47855.13299708468, + 46291.47427771338, + 51960.468458886695 + ], + "qps_batch": 47855.13299708468, + "qps_batch_min": 46291.47427771338, + "qps_batch_max": 51960.468458886695, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99995, + "qps_1t_runs": [ + 2973.0152739995783, + 2940.223492592194, + 2765.8056811963256 + ], + "qps_1t": 2940.223492592194, + "qps_1t_min": 2765.8056811963256, + "qps_1t_max": 2973.0152739995783, + "median_ms": 0.32920000376179814, + "p95_ms": 0.4738099989481271, + "p99_ms": 0.8397219912149017, + "mean_ms": 0.34600911678474705, + "knob": 400, + "batch": { + "recall": 0.99995, + "qps_batch_runs": [ + 33657.341359052145, + 36979.923600510476, + 36822.44404639132 + ], + "qps_batch": 36822.44404639132, + "qps_batch_min": 33657.341359052145, + "qps_batch_max": 36979.923600510476, + "threads": 32 + } + } + ], + "knob_name": "ef", + "mem_after_sweep": { + "rss_mb": 137.08984375, + "peak_mb": 143.14453125 + }, + "ref_knob": 32, + "ref_recall_pre": 0.95025, + "mutation": { + "ref_knob": 32, + "recall_before": 0.95025, + "insert_10k": { + "seconds": 0.23788480000803247, + "per_s": 42037.15411687648, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.5695672999718226, + "per_s": 17557.1877116097, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.007995199994184077, + "per_s": 1250750.4511799915, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 152.32421875, + "peak_mb": 239.421875 + }, + "first_query_after_ms": 0.10080001084133983, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.94095, + "qps_1t_runs": [ + 20147.236095205542, + 20960.932950558163, + 20559.82347651121 + ], + "qps_1t": 20559.82347651121, + "qps_1t_min": 20147.236095205542, + "qps_1t_max": 20960.932950558163, + "median_ms": 0.04689994966611266, + "p95_ms": 0.06659998325631022, + "p99_ms": 0.08729996683541685, + "mean_ms": 0.048660316601550825 + }, + "recall_after": 0.94095 + }, + "saveload": { + "save_s": 0.06717080000089481, + "disk_mb": 69.29621124267578, + "load_s": 0.15757999999914318, + "first_query_after_load_ms": 0.2136999974027276, + "time_to_ready_s": 0.1577936999965459 + }, + "mem_end": { + "rss_mb": 134.9140625, + "peak_mb": 239.421875 + }, + "status": "ok", + "finished": "2026-11-16 11:09:27", + "load_before_pct": 8.856250000000006, + "load_contaminated": false, + "wall_s": 14.174603939056396, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/hnswlib__text384__1000000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/hnswlib__text384__1000000__core+mutation+saveload.json new file mode 100644 index 0000000..508ad00 --- /dev/null +++ b/benchmarks/competitive/results/raw/hnswlib__text384__1000000__core+mutation+saveload.json @@ -0,0 +1,463 @@ +{ + "system": "hnswlib", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 09:23:37", + "mem_baseline": { + "rss_mb": 37.37890625, + "peak_mb": 37.37890625 + }, + "version": "hnswlib 0.8.0", + "build": { + "ingest_s": 172.9470293999184, + "index_s": 4.800036549568176e-06, + "total_s": 172.94703419995494, + "first_query_ms": 3.6525000468827784, + "ready_s": 172.95068670000182 + }, + "mem_after_build": { + "rss_mb": 1867.3984375, + "peak_mb": 1897.1953125 + }, + "mem_after_first_query": { + "rss_mb": 1867.4140625, + "peak_mb": 1897.1953125 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.5589, + "qps_1t_runs": [ + 7939.9454134163125, + 9989.71059184505, + 11884.57227022613 + ], + "qps_1t": 9989.71059184505, + "qps_1t_min": 7939.9454134163125, + "qps_1t_max": 11884.57227022613, + "median_ms": 0.09200000204145908, + "p95_ms": 0.16950501012615862, + "p99_ms": 0.25110899121500574, + "mean_ms": 0.1033970501254468, + "knob": 10, + "batch": { + "recall": 0.5589, + "qps_batch_runs": [ + 45334.923072481906, + 47940.01741762274, + 45715.2261976148 + ], + "qps_batch": 45715.2261976148, + "qps_batch_min": 45334.923072481906, + "qps_batch_max": 47940.01741762274, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6512, + "qps_1t_runs": [ + 8071.373527158176, + 8872.409890884685, + 8803.514351564298 + ], + "qps_1t": 8803.514351564298, + "qps_1t_min": 8071.373527158176, + "qps_1t_max": 8872.409890884685, + "median_ms": 0.10870001278817654, + "p95_ms": 0.16351502272300428, + "p99_ms": 0.20780301070772117, + "mean_ms": 0.11673153335383783, + "knob": 16, + "batch": { + "recall": 0.6512, + "qps_batch_runs": [ + 34565.835815985214, + 36683.112920381, + 37715.14130693933 + ], + "qps_batch": 36683.112920381, + "qps_batch_min": 34565.835815985214, + "qps_batch_max": 37715.14130693933, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.72775, + "qps_1t_runs": [ + 6565.330768994968, + 6992.124406289892, + 7079.8840931398545 + ], + "qps_1t": 6992.124406289892, + "qps_1t_min": 6565.330768994968, + "qps_1t_max": 7079.8840931398545, + "median_ms": 0.13610004680231214, + "p95_ms": 0.19749999046325684, + "p99_ms": 0.2486010303255171, + "mean_ms": 0.1455261836235877, + "knob": 24, + "batch": { + "recall": 0.72775, + "qps_batch_runs": [ + 27745.061706597382, + 27873.28247232849, + 28350.095336296166 + ], + "qps_batch": 27873.28247232849, + "qps_batch_min": 27745.061706597382, + "qps_batch_max": 28350.095336296166, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.77635, + "qps_1t_runs": [ + 5293.43493108574, + 5557.560899104856, + 5665.078842947724 + ], + "qps_1t": 5557.560899104856, + "qps_1t_min": 5293.43493108574, + "qps_1t_max": 5665.078842947724, + "median_ms": 0.1729000359773636, + "p95_ms": 0.2474049630109221, + "p99_ms": 0.31220499076880526, + "mean_ms": 0.1817894502892159, + "knob": 32, + "batch": { + "recall": 0.77635, + "qps_batch_runs": [ + 22528.205305215724, + 23128.63336314208, + 22938.568208679393 + ], + "qps_batch": 22938.568208679393, + "qps_batch_min": 22528.205305215724, + "qps_batch_max": 23128.63336314208, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8354, + "qps_1t_runs": [ + 4386.61424591747, + 4275.635692782341, + 4196.184188634086 + ], + "qps_1t": 4275.635692782341, + "qps_1t_min": 4196.184188634086, + "qps_1t_max": 4386.61424591747, + "median_ms": 0.22429999080486596, + "p95_ms": 0.30629994871560484, + "p99_ms": 0.39223196043167335, + "mean_ms": 0.2333871169539634, + "knob": 48, + "batch": { + "recall": 0.8354, + "qps_batch_runs": [ + 16080.531299932483, + 17164.77410402789, + 17655.83480798102 + ], + "qps_batch": 17164.77410402789, + "qps_batch_min": 16080.531299932483, + "qps_batch_max": 17655.83480798102, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8727, + "qps_1t_runs": [ + 3223.399826829237, + 3211.24500821457, + 3269.922989365925 + ], + "qps_1t": 3223.399826829237, + "qps_1t_min": 3211.24500821457, + "qps_1t_max": 3269.922989365925, + "median_ms": 0.29559998074546456, + "p95_ms": 0.40610500727780163, + "p99_ms": 0.5568379908800133, + "mean_ms": 0.30915158306015655, + "knob": 64, + "batch": { + "recall": 0.8727, + "qps_batch_runs": [ + 13480.168315181072, + 13764.767012079901, + 13524.698806255925 + ], + "qps_batch": 13524.698806255925, + "qps_batch_min": 13480.168315181072, + "qps_batch_max": 13764.767012079901, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.91005, + "qps_1t_runs": [ + 2481.8662463574624, + 2351.890293282098, + 2493.6673315274925 + ], + "qps_1t": 2481.8662463574624, + "qps_1t_min": 2351.890293282098, + "qps_1t_max": 2493.6673315274925, + "median_ms": 0.39840000681579113, + "p95_ms": 0.5230050242971629, + "p99_ms": 0.663223986048252, + "mean_ms": 0.4097094334332117, + "knob": 96, + "batch": { + "recall": 0.91005, + "qps_batch_runs": [ + 9534.210895498365, + 9471.589494207263, + 9318.40090669534 + ], + "qps_batch": 9471.589494207263, + "qps_batch_min": 9318.40090669534, + "qps_batch_max": 9534.210895498365, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.93155, + "qps_1t_runs": [ + 1862.1670481633507, + 1987.39672440239, + 1908.1160091121474 + ], + "qps_1t": 1908.1160091121474, + "qps_1t_min": 1862.1670481633507, + "qps_1t_max": 1987.39672440239, + "median_ms": 0.5098499823361635, + "p95_ms": 0.6543450319441037, + "p99_ms": 0.8159069844987245, + "mean_ms": 0.5214189002678419, + "knob": 128, + "batch": { + "recall": 0.93155, + "qps_batch_runs": [ + 7621.167028207698, + 7534.069061083572, + 7173.608867673934 + ], + "qps_batch": 7534.069061083572, + "qps_batch_min": 7173.608867673934, + "qps_batch_max": 7621.167028207698, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9572, + "qps_1t_runs": [ + 1385.3156401395422, + 1348.0865230657812, + 1332.3331063397723 + ], + "qps_1t": 1348.0865230657812, + "qps_1t_min": 1332.3331063397723, + "qps_1t_max": 1385.3156401395422, + "median_ms": 0.7179499953053892, + "p95_ms": 0.9054200229002164, + "p99_ms": 1.2947230215650056, + "mean_ms": 0.7380707833120445, + "knob": 192, + "batch": { + "recall": 0.9572, + "qps_batch_runs": [ + 5355.257042319188, + 5337.06358438381, + 5443.139603460202 + ], + "qps_batch": 5355.257042319188, + "qps_batch_min": 5337.06358438381, + "qps_batch_max": 5443.139603460202, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.96845, + "qps_1t_runs": [ + 1042.9634193553227, + 1036.723278516595, + 1047.174587481958 + ], + "qps_1t": 1042.9634193553227, + "qps_1t_min": 1036.723278516595, + "qps_1t_max": 1047.174587481958, + "median_ms": 0.9442000009585172, + "p95_ms": 1.1992049927357584, + "p99_ms": 1.527310988167301, + "mean_ms": 0.9594448493347348, + "knob": 256, + "batch": { + "recall": 0.96845, + "qps_batch_runs": [ + 4286.852138612777, + 4059.25373856563, + 3975.8886207427563 + ], + "qps_batch": 4059.25373856563, + "qps_batch_min": 3975.8886207427563, + "qps_batch_max": 4286.852138612777, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9817, + "qps_1t_runs": [ + 685.2266474192253, + 720.9815992185978, + 700.4394766196394 + ], + "qps_1t": 700.4394766196394, + "qps_1t_min": 685.2266474192253, + "qps_1t_max": 720.9815992185978, + "median_ms": 1.406650000717491, + "p95_ms": 1.7451050167437643, + "p99_ms": 2.3742699756985544, + "mean_ms": 1.4246813998130772, + "knob": 400, + "batch": { + "recall": 0.9817, + "qps_batch_runs": [ + 2649.38034302476, + 2766.009177006968, + 2680.8576976003537 + ], + "qps_batch": 2680.8576976003537, + "qps_batch_min": 2649.38034302476, + "qps_batch_max": 2766.009177006968, + "threads": 32 + } + }, + { + "nq": 1224, + "recall_queries": 1224, + "recall": 0.9901143790849674, + "qps_1t_runs": [ + 460.69300640039273, + 471.90366917754915, + 472.23481150639606 + ], + "qps_1t": 471.90366917754915, + "qps_1t_min": 460.69300640039273, + "qps_1t_max": 472.23481150639606, + "median_ms": 2.146399987395853, + "p95_ms": 2.574079984333366, + "p99_ms": 2.989146026084199, + "mean_ms": 2.135770016271608, + "knob": 640, + "batch": { + "recall": 0.98985, + "qps_batch_runs": [ + 1842.6957016255428, + 1863.4264910216637, + 1848.0949144102544 + ], + "qps_batch": 1848.0949144102544, + "qps_batch_min": 1842.6957016255428, + "qps_batch_max": 1863.4264910216637, + "threads": 32 + } + } + ], + "knob_name": "ef", + "mem_after_sweep": { + "rss_mb": 1872.6953125, + "peak_mb": 1897.1953125 + }, + "ref_knob": 192, + "ref_recall_pre": 0.9572, + "mutation": { + "ref_knob": 192, + "recall_before": 0.9572, + "insert_10k": { + "seconds": 2.094509400019888, + "per_s": 4774.3877396325115, + "batch": 1000 + }, + "update_10k": { + "seconds": 3.0900770999724045, + "per_s": 3236.1652076866635, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.008781400043517351, + "per_s": 1138770.5776349695, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 1919.6796875, + "peak_mb": 1919.8125 + }, + "first_query_after_ms": 0.7746999617666006, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.95635, + "qps_1t_runs": [ + 1319.6338612899997, + 1335.8710433239014, + 1320.6679515979642 + ], + "qps_1t": 1320.6679515979642, + "qps_1t_min": 1319.6338612899997, + "qps_1t_max": 1335.8710433239014, + "median_ms": 0.7574500050395727, + "p95_ms": 0.916799996048212, + "p99_ms": 1.09075798187405, + "mean_ms": 0.754517933470197 + }, + "recall_after": 0.95635 + }, + "saveload": { + "save_s": 1.0720006999908946, + "disk_mb": 1622.5444107055664, + "load_s": 2.1154646999784745, + "first_query_after_load_ms": 1.5783000271767378, + "time_to_ready_s": 2.1170430000056513 + }, + "mem_end": { + "rss_mb": 1822.51953125, + "peak_mb": 1919.83203125 + }, + "status": "ok", + "finished": "2026-11-16 09:27:35", + "load_before_pct": 6.412500000000005, + "load_contaminated": false, + "wall_s": 238.22834992408752, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/hnswlib__text384__1000000__filter.json b/benchmarks/competitive/results/raw/hnswlib__text384__1000000__filter.json new file mode 100644 index 0000000..06982c3 --- /dev/null +++ b/benchmarks/competitive/results/raw/hnswlib__text384__1000000__filter.json @@ -0,0 +1,182 @@ +{ + "system": "hnswlib", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 10:55:59", + "filter": { + "build_with_metadata_s": 150.55821330007166, + "mem": { + "rss_mb": 1868.0, + "peak_mb": 1897.7734375 + }, + "subsets": { + "f10": [ + { + "knob": 16, + "nq": 2000, + "recall": 0.8597, + "qps_1t": 965.5264101051243, + "qps_1t_runs": [ + 815.9749286167372, + 965.5264101051243, + 1311.292860013389 + ], + "median_ms": 0.8855000196490437, + "p95_ms": 1.792909973300994, + "p99_ms": 3.1010729796253185, + "mean_ms": 1.007946150319185 + }, + { + "knob": 48, + "nq": 1915, + "recall": 0.9550913838120104, + "qps_1t": 406.06719944948367, + "qps_1t_runs": [ + 406.06719944948367 + ], + "median_ms": 2.4363999837078154, + "p95_ms": 3.3190599933732297, + "p99_ms": 3.7982520007062694, + "mean_ms": 2.4626465800629234 + }, + { + "knob": 96, + "nq": 743, + "recall": 0.9826379542395693, + "qps_1t": 206.41749494898892, + "qps_1t_runs": [ + 206.41749494898892, + 206.1225098979031, + 207.06033962903118 + ], + "median_ms": 4.820000031031668, + "p95_ms": 6.258079991675912, + "p99_ms": 7.112372014671559, + "mean_ms": 4.841848137342868 + }, + { + "knob": 192, + "nq": 342, + "recall": 0.9953216374269006, + "qps_1t": 111.41473254946023, + "qps_1t_runs": [ + 110.39197802661296, + 111.41473254946023, + 113.79916722395953 + ], + "median_ms": 8.90740001341328, + "p95_ms": 11.198549997061491, + "p99_ms": 12.305099968216382, + "mean_ms": 8.940504483522785 + }, + { + "knob": 640, + "nq": 120, + "recall": 0.9991666666666666, + "qps_1t": 40.45463865958441, + "qps_1t_runs": [ + 40.41673561092942, + 40.77178405057407, + 40.45463865958441 + ], + "median_ms": 24.713349994271994, + "p95_ms": 29.000344980158843, + "p99_ms": 31.51762301393323, + "mean_ms": 24.662678613417988 + } + ], + "f01": [ + { + "knob": 16, + "nq": 486, + "recall": 0.9446502057613169, + "qps_1t": 135.0047551718261, + "qps_1t_runs": [ + 135.0047551718261, + 132.9490940372456, + 138.19347852491904 + ], + "median_ms": 7.16199999442324, + "p95_ms": 10.532300028717144, + "p99_ms": 12.1265950112138, + "mean_ms": 7.388351303483749 + }, + { + "knob": 48, + "nq": 168, + "recall": 0.9910714285714286, + "qps_1t": 52.143865538030056, + "qps_1t_runs": [ + 51.028790542712166, + 52.702241560200754, + 52.143865538030056 + ], + "median_ms": 19.08254998852499, + "p95_ms": 25.072974979411807, + "p99_ms": 29.955342992907365, + "mean_ms": 19.249672221901296 + }, + { + "knob": 96, + "nq": 84, + "recall": 0.9940476190476191, + "qps_1t": 28.00126939239545, + "qps_1t_runs": [ + 27.301230465099742, + 28.00126939239545, + 28.041923611467304 + ], + "median_ms": 36.020000028656796, + "p95_ms": 44.82396502280608, + "p99_ms": 48.251943037030294, + "mean_ms": 36.00064801543744 + }, + { + "knob": 192, + "nq": 49, + "recall": 1.0, + "qps_1t": 16.3899491341929, + "qps_1t_runs": [ + 16.52350009199954, + 15.453822305968659, + 16.3899491341929 + ], + "median_ms": 62.02630000188947, + "p95_ms": 77.42504997877401, + "p99_ms": 82.25276600336656, + "mean_ms": 62.08059252049698 + }, + { + "knob": 640, + "nq": 30, + "recall": 1.0, + "qps_1t": 6.01650013147678, + "qps_1t_runs": [ + 6.01650013147678 + ], + "median_ms": 167.38199998508207, + "p95_ms": 190.8775699936086, + "p99_ms": 201.73997197591234, + "mean_ms": 166.20958666123138 + } + ] + } + }, + "mem_end": { + "rss_mb": 41.7421875, + "peak_mb": 1897.7734375 + }, + "status": "ok", + "finished": "2026-11-16 10:59:59", + "load_before_pct": 5.88125, + "load_contaminated": false, + "wall_s": 240.25848054885864, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/hnswlib__text384__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/hnswlib__text384__100000__core+mutation+saveload.json new file mode 100644 index 0000000..0adff52 --- /dev/null +++ b/benchmarks/competitive/results/raw/hnswlib__text384__100000__core+mutation+saveload.json @@ -0,0 +1,468 @@ +{ + "system": "hnswlib", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 07:55:38", + "mem_baseline": { + "rss_mb": 37.33984375, + "peak_mb": 37.33984375 + }, + "version": "hnswlib 0.8.0", + "build": { + "ingest_s": 6.211588800011668, + "index_s": 2.6999623514711857e-06, + "total_s": 6.21159149997402, + "first_query_ms": 1.7135999514721334, + "ready_s": 6.213305099925492, + "total_s_runs": [ + 6.21159149997402, + 6.21282560005784, + 6.091282200010028 + ] + }, + "mem_after_build": { + "rss_mb": 231.16796875, + "peak_mb": 262.8984375 + }, + "mem_after_first_query": { + "rss_mb": 231.19921875, + "peak_mb": 262.8984375 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.61845, + "qps_1t_runs": [ + 17225.623834351933, + 16406.823625073837, + 16855.447770790626 + ], + "qps_1t": 16855.447770790626, + "qps_1t_min": 16406.823625073837, + "qps_1t_max": 17225.623834351933, + "median_ms": 0.05549995694309473, + "p95_ms": 0.08239998714998364, + "p99_ms": 0.10400803468655813, + "mean_ms": 0.059443766493738316, + "knob": 10, + "batch": { + "recall": 0.61845, + "qps_batch_runs": [ + 93453.14035921605, + 95652.13252624813, + 89512.69293528692 + ], + "qps_batch": 93453.14035921605, + "qps_batch_min": 89512.69293528692, + "qps_batch_max": 95652.13252624813, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.71605, + "qps_1t_runs": [ + 13312.096264093334, + 13172.479852289463, + 13035.277370789938 + ], + "qps_1t": 13172.479852289463, + "qps_1t_min": 13035.277370789938, + "qps_1t_max": 13312.096264093334, + "median_ms": 0.07240002742037177, + "p95_ms": 0.10459998447913677, + "p99_ms": 0.13390297186560934, + "mean_ms": 0.07591679958083357, + "knob": 16, + "batch": { + "recall": 0.71605, + "qps_batch_runs": [ + 63072.381851184095, + 75401.98687064977, + 75207.19569016236 + ], + "qps_batch": 75207.19569016236, + "qps_batch_min": 63072.381851184095, + "qps_batch_max": 75401.98687064977, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.787, + "qps_1t_runs": [ + 9859.793670188494, + 9987.146542771392, + 9649.864291725911 + ], + "qps_1t": 9859.793670188494, + "qps_1t_min": 9649.864291725911, + "qps_1t_max": 9987.146542771392, + "median_ms": 0.09560002945363522, + "p95_ms": 0.13160501257516447, + "p99_ms": 0.1758019998669625, + "mean_ms": 0.10172636699280702, + "knob": 24, + "batch": { + "recall": 0.787, + "qps_batch_runs": [ + 51953.044843418575, + 52453.925795430834, + 57681.10426298905 + ], + "qps_batch": 52453.925795430834, + "qps_batch_min": 51953.044843418575, + "qps_batch_max": 57681.10426298905, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8309, + "qps_1t_runs": [ + 8229.501019281879, + 7449.621947561647, + 7739.848228119891 + ], + "qps_1t": 7739.848228119891, + "qps_1t_min": 7449.621947561647, + "qps_1t_max": 8229.501019281879, + "median_ms": 0.11600001016631722, + "p95_ms": 0.16080500208772722, + "p99_ms": 0.24430599703919156, + "mean_ms": 0.12831684993579984, + "knob": 32, + "batch": { + "recall": 0.8309, + "qps_batch_runs": [ + 44188.348425563265, + 48013.67435386259, + 46850.147496390935 + ], + "qps_batch": 46850.147496390935, + "qps_batch_min": 44188.348425563265, + "qps_batch_max": 48013.67435386259, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8855, + "qps_1t_runs": [ + 6085.143953116595, + 5802.094611550192, + 6165.389660777488 + ], + "qps_1t": 6085.143953116595, + "qps_1t_min": 5802.094611550192, + "qps_1t_max": 6165.389660777488, + "median_ms": 0.1572000328451395, + "p95_ms": 0.20570499473251405, + "p99_ms": 0.26632199063897183, + "mean_ms": 0.16629398326040246, + "knob": 48, + "batch": { + "recall": 0.8855, + "qps_batch_runs": [ + 33392.716714494665, + 31074.52604981799, + 33902.15501498349 + ], + "qps_batch": 33392.716714494665, + "qps_batch_min": 31074.52604981799, + "qps_batch_max": 33902.15501498349, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9162, + "qps_1t_runs": [ + 4838.660895718044, + 4725.756108459523, + 4337.287420002313 + ], + "qps_1t": 4725.756108459523, + "qps_1t_min": 4337.287420002313, + "qps_1t_max": 4838.660895718044, + "median_ms": 0.20209999638609588, + "p95_ms": 0.27279998175799847, + "p99_ms": 0.3774740215158106, + "mean_ms": 0.2162779833500584, + "knob": 64, + "batch": { + "recall": 0.9162, + "qps_batch_runs": [ + 26177.153875592645, + 27882.76412342768, + 26037.767778750873 + ], + "qps_batch": 26177.153875592645, + "qps_batch_min": 26037.767778750873, + "qps_batch_max": 27882.76412342768, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9488, + "qps_1t_runs": [ + 3413.8297308579226, + 3563.313753701257, + 3575.553866166564 + ], + "qps_1t": 3563.313753701257, + "qps_1t_min": 3413.8297308579226, + "qps_1t_max": 3575.553866166564, + "median_ms": 0.2798999776132405, + "p95_ms": 0.343804998556152, + "p99_ms": 0.4275569960009319, + "mean_ms": 0.2844135835475754, + "knob": 96, + "batch": { + "recall": 0.9488, + "qps_batch_runs": [ + 20021.002020894895, + 20348.05345627372, + 20267.120647963467 + ], + "qps_batch": 20267.120647963467, + "qps_batch_min": 20021.002020894895, + "qps_batch_max": 20348.05345627372, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.96445, + "qps_1t_runs": [ + 2715.3986587391196, + 2704.1121273724907, + 2727.729249379135 + ], + "qps_1t": 2715.3986587391196, + "qps_1t_min": 2704.1121273724907, + "qps_1t_max": 2727.729249379135, + "median_ms": 0.3541000187397003, + "p95_ms": 0.439604974235408, + "p99_ms": 0.5871070461580531, + "mean_ms": 0.36822750015805167, + "knob": 128, + "batch": { + "recall": 0.96445, + "qps_batch_runs": [ + 15528.251714547954, + 15857.259293311603, + 15704.73835204133 + ], + "qps_batch": 15704.73835204133, + "qps_batch_min": 15528.251714547954, + "qps_batch_max": 15857.259293311603, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.98095, + "qps_1t_runs": [ + 1959.1762580813638, + 1962.101230154518, + 1929.5917084535288 + ], + "qps_1t": 1959.1762580813638, + "qps_1t_min": 1929.5917084535288, + "qps_1t_max": 1962.101230154518, + "median_ms": 0.5038999952375889, + "p95_ms": 0.6147000240162015, + "p99_ms": 0.7766069762874396, + "mean_ms": 0.5127735493782287, + "knob": 192, + "batch": { + "recall": 0.98095, + "qps_batch_runs": [ + 11013.14308374829, + 10961.066295289133, + 10596.655271550744 + ], + "qps_batch": 10961.066295289133, + "qps_batch_min": 10596.655271550744, + "qps_batch_max": 11013.14308374829, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.98695, + "qps_1t_runs": [ + 1553.4706954621568, + 1558.0544324838709, + 1565.162120047441 + ], + "qps_1t": 1558.0544324838709, + "qps_1t_min": 1553.4706954621568, + "qps_1t_max": 1565.162120047441, + "median_ms": 0.6339999963529408, + "p95_ms": 0.758910016156733, + "p99_ms": 0.8619120437651875, + "mean_ms": 0.6414858165953774, + "knob": 256, + "batch": { + "recall": 0.98695, + "qps_batch_runs": [ + 8700.441199111725, + 8390.595653448469, + 8427.677672575583 + ], + "qps_batch": 8427.677672575583, + "qps_batch_min": 8390.595653448469, + "qps_batch_max": 8700.441199111725, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9933, + "qps_1t_runs": [ + 1026.0785023027886, + 1037.470314984736, + 1017.1449962061066 + ], + "qps_1t": 1026.0785023027886, + "qps_1t_min": 1017.1449962061066, + "qps_1t_max": 1037.470314984736, + "median_ms": 0.9490000084042549, + "p95_ms": 1.169709998066537, + "p99_ms": 1.4647279772907502, + "mean_ms": 0.9738704337602636, + "knob": 400, + "batch": { + "recall": 0.9933, + "qps_batch_runs": [ + 6018.400658157129, + 5967.642843979703, + 6069.327500040121 + ], + "qps_batch": 6018.400658157129, + "qps_batch_min": 5967.642843979703, + "qps_batch_max": 6069.327500040121, + "threads": 32 + } + }, + { + "nq": 1874, + "recall_queries": 1874, + "recall": 0.9969583778014941, + "qps_1t_runs": [ + 689.6798930975095, + 697.2763284482164, + 695.8861615719146 + ], + "qps_1t": 695.8861615719146, + "qps_1t_min": 689.6798930975095, + "qps_1t_max": 697.2763284482164, + "median_ms": 1.4125999878160655, + "p95_ms": 1.7367000109516084, + "p99_ms": 2.2665380290709436, + "mean_ms": 1.440372110005867, + "knob": 640, + "batch": { + "recall": 0.997, + "qps_batch_runs": [ + 4124.082932811876, + 3945.9242645267195, + 3492.5963072247396 + ], + "qps_batch": 3945.9242645267195, + "qps_batch_min": 3492.5963072247396, + "qps_batch_max": 4124.082932811876, + "threads": 32 + } + } + ], + "knob_name": "ef", + "mem_after_sweep": { + "rss_mb": 239.28515625, + "peak_mb": 262.8984375 + }, + "ref_knob": 128, + "ref_recall_pre": 0.96445, + "mutation": { + "ref_knob": 128, + "recall_before": 0.96445, + "insert_10k": { + "seconds": 1.105076999985613, + "per_s": 9049.143181995636, + "batch": 1000 + }, + "update_10k": { + "seconds": 2.965886700025294, + "per_s": 3371.672963742923, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.011413100000936538, + "per_s": 876186.1369110424, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 284.1796875, + "peak_mb": 458.48046875 + }, + "first_query_after_ms": 0.6867000483907759, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9683, + "qps_1t_runs": [ + 2083.5219350971756, + 2218.385892404205, + 2238.180451599162 + ], + "qps_1t": 2218.385892404205, + "qps_1t_min": 2083.5219350971756, + "qps_1t_max": 2238.180451599162, + "median_ms": 0.4464000230655074, + "p95_ms": 0.5581100005656481, + "p99_ms": 0.6933049822691829, + "mean_ms": 0.45917541658855043 + }, + "recall_after": 0.9683 + }, + "saveload": { + "save_s": 0.14747150003677234, + "disk_mb": 176.71808624267578, + "load_s": 0.22453179996227846, + "first_query_after_load_ms": 0.6153000285848975, + "time_to_ready_s": 0.22514709999086335 + }, + "mem_end": { + "rss_mb": 243.109375, + "peak_mb": 458.48046875 + }, + "status": "ok", + "finished": "2026-11-16 07:56:40", + "load_before_pct": 4.643749999999999, + "load_contaminated": false, + "wall_s": 61.72725200653076, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/hnswlib__text384__100000__filter.json b/benchmarks/competitive/results/raw/hnswlib__text384__100000__filter.json new file mode 100644 index 0000000..cbc2232 --- /dev/null +++ b/benchmarks/competitive/results/raw/hnswlib__text384__100000__filter.json @@ -0,0 +1,186 @@ +{ + "system": "hnswlib", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 08:36:32", + "filter": { + "build_with_metadata_s": 6.4425217001116835, + "mem": { + "rss_mb": 230.86328125, + "peak_mb": 262.44140625 + }, + "subsets": { + "f10": [ + { + "knob": 16, + "nq": 2000, + "recall": 0.91235, + "qps_1t": 1778.6843554561513, + "qps_1t_runs": [ + 1714.9114129067339, + 1778.6843554561513, + 1781.5343607501627 + ], + "median_ms": 0.5446000141091645, + "p95_ms": 0.8071250311331825, + "p99_ms": 1.042873024125585, + "mean_ms": 0.5688825663431393 + }, + { + "knob": 48, + "nq": 2000, + "recall": 0.9833, + "qps_1t": 681.4732403313656, + "qps_1t_runs": [ + 697.3406114780528, + 647.9030068889845, + 681.4732403313656 + ], + "median_ms": 1.3814999838359654, + "p95_ms": 1.894315041135997, + "p99_ms": 3.1411739939358125, + "mean_ms": 1.4816231331399952 + }, + { + "knob": 96, + "nq": 1244, + "recall": 0.9949356913183279, + "qps_1t": 385.2228753884162, + "qps_1t_runs": [ + 372.54788090474375, + 387.9784442935935, + 385.2228753884162 + ], + "median_ms": 2.520250011002645, + "p95_ms": 3.426375013077631, + "p99_ms": 5.214121007011277, + "mean_ms": 2.6191937833898904 + }, + { + "knob": 192, + "nq": 701, + "recall": 0.998716119828816, + "qps_1t": 217.47272683683482, + "qps_1t_runs": [ + 217.47272683683482, + 216.6985222890845, + 228.98169362459495 + ], + "median_ms": 4.44779999088496, + "p95_ms": 5.740439996588975, + "p99_ms": 7.4728159874212015, + "mean_ms": 4.526715073059169 + }, + { + "knob": 640, + "nq": 265, + "recall": 1.0, + "qps_1t": 85.00550387736226, + "qps_1t_runs": [ + 85.00550387736226, + 86.72104458735168, + 83.45960930995703 + ], + "median_ms": 11.389799998141825, + "p95_ms": 13.891609973506998, + "p99_ms": 21.025283990893488, + "mean_ms": 11.759004778369277 + } + ], + "f01": [ + { + "knob": 16, + "nq": 762, + "recall": 0.9813648293963254, + "qps_1t": 258.7360249980111, + "qps_1t_runs": [ + 241.37487880135672, + 258.7360249980111, + 270.94914233767344 + ], + "median_ms": 3.725250018760562, + "p95_ms": 5.672749975929037, + "p99_ms": 8.250459967530347, + "mean_ms": 3.899535258347046 + }, + { + "knob": 48, + "nq": 358, + "recall": 0.9986033519553073, + "qps_1t": 107.85546934809844, + "qps_1t_runs": [ + 107.82210844960854, + 109.81509869096007, + 107.85546934809844 + ], + "median_ms": 8.957550016930327, + "p95_ms": 12.161819997709244, + "p99_ms": 18.01534997648559, + "mean_ms": 9.217472904410778 + }, + { + "knob": 96, + "nq": 200, + "recall": 1.0, + "qps_1t": 63.843694279518886, + "qps_1t_runs": [ + 63.18820047455781, + 63.843694279518886, + 64.12194248917788 + ], + "median_ms": 15.221199981169775, + "p95_ms": 19.741240015719086, + "p99_ms": 24.973395992419682, + "mean_ms": 15.694759667094331 + }, + { + "knob": 192, + "nq": 120, + "recall": 1.0, + "qps_1t": 39.407314004695856, + "qps_1t_runs": [ + 39.01131416105161, + 39.51068258689269, + 39.407314004695856 + ], + "median_ms": 25.014050013851374, + "p95_ms": 29.422394998255186, + "p99_ms": 39.64057695528036, + "mean_ms": 25.43973333102056 + }, + { + "knob": 640, + "nq": 60, + "recall": 1.0, + "qps_1t": 19.460341381440205, + "qps_1t_runs": [ + 19.56792010367712, + 19.460341381440205, + 19.355191474301996 + ], + "median_ms": 50.41320002055727, + "p95_ms": 58.380130023579106, + "p99_ms": 65.33958000713028, + "mean_ms": 51.38544555488301 + } + ] + } + }, + "mem_end": { + "rss_mb": 40.3515625, + "peak_mb": 262.44140625 + }, + "status": "ok", + "finished": "2026-11-16 08:38:08", + "load_before_pct": 3.5593749999999975, + "load_contaminated": false, + "wall_s": 96.23951411247253, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/isong_py__glove__1000000__core+mutation.json b/benchmarks/competitive/results/raw/isong_py__glove__1000000__core+mutation.json new file mode 100644 index 0000000..fcf107c --- /dev/null +++ b/benchmarks/competitive/results/raw/isong_py__glove__1000000__core+mutation.json @@ -0,0 +1,122 @@ +{ + "system": "isong_py", + "ds": "glove", + "n": 1000000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation" + ], + "started": "2026-11-16 10:30:04", + "mem_baseline": { + "rss_mb": 35.0078125, + "peak_mb": 35.0078125 + }, + "version": "isongraph-vector (python, repo HEAD)", + "build": { + "ingest_s": 27.908122700056992, + "index_s": 4.200031980872154e-06, + "total_s": 27.908126900088973, + "first_query_ms": 4899.777499958873, + "ready_s": 32.807904400047846 + }, + "mem_after_build": { + "rss_mb": 729.9296875, + "peak_mb": 798.8125 + }, + "mem_after_first_query": { + "rss_mb": 1459.21875, + "peak_mb": 1470.53515625 + }, + "sweep": [ + { + "nq": 84, + "recall_queries": 84, + "recall": 1.0, + "qps_1t_runs": [ + 27.887184253289014, + 26.46472568856041, + 26.159219335618886 + ], + "qps_1t": 26.46472568856041, + "qps_1t_min": 26.159219335618886, + "qps_1t_max": 27.887184253289014, + "median_ms": 37.09220001474023, + "p95_ms": 42.527489989879534, + "p99_ms": 47.820610992494046, + "mean_ms": 37.29078412635274, + "knob": null + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 93, + "threads": 32, + "qps": 27.714460003367048, + "median_ms": 35.600699950009584, + "p95_ms": 49.929859989788376, + "p99_ms": 52.247072028694596, + "mean_ms": 36.24233978739556 + }, + "mem_after_sweep": { + "rss_mb": 1460.0703125, + "peak_mb": 1471.51953125 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.8805213999585249, + "per_s": 11356.907396539174, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.6018026000238024, + "per_s": 16616.74442683445, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.5086630000150762, + "per_s": 19659.381554592353, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 1472.03515625, + "peak_mb": 1472.03515625 + }, + "first_query_after_ms": 100.25179997319356, + "after": { + "nq": 77, + "recall_queries": 77, + "recall": 1.0, + "qps_1t_runs": [ + 26.34213520958877, + 27.546546775197086, + 27.93582075127861 + ], + "qps_1t": 27.546546775197086, + "qps_1t_min": 26.34213520958877, + "qps_1t_max": 27.93582075127861, + "median_ms": 36.20279999449849, + "p95_ms": 42.87535001640208, + "p99_ms": 47.78328002430498, + "mean_ms": 36.68684026179029 + }, + "recall_after": 1.0 + }, + "mem_end": { + "rss_mb": 1464.3515625, + "peak_mb": 1483.6015625 + }, + "status": "ok", + "finished": "2026-11-16 10:31:13", + "load_before_pct": 7.521874999999993, + "load_contaminated": false, + "wall_s": 69.04397058486938, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/isong_py__glove__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/isong_py__glove__100000__core+mutation+saveload.json new file mode 100644 index 0000000..88eb535 --- /dev/null +++ b/benchmarks/competitive/results/raw/isong_py__glove__100000__core+mutation+saveload.json @@ -0,0 +1,135 @@ +{ + "system": "isong_py", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 08:25:25", + "mem_baseline": { + "rss_mb": 35.02734375, + "peak_mb": 35.02734375 + }, + "version": "isongraph-vector (python, repo HEAD)", + "build": { + "ingest_s": 0.9344128000084311, + "index_s": 2.400018274784088e-06, + "total_s": 0.9344152000267059, + "first_query_ms": 179.07509999349713, + "ready_s": 1.113490300020203, + "total_s_runs": [ + 0.9344152000267059, + 0.9137653000070713, + 0.9312418999616057 + ] + }, + "mem_after_build": { + "rss_mb": 111.9140625, + "peak_mb": 199.09765625 + }, + "mem_after_first_query": { + "rss_mb": 194.46484375, + "peak_mb": 199.09765625 + }, + "sweep": [ + { + "nq": 1462, + "recall_queries": 1462, + "recall": 1.0, + "qps_1t_runs": [ + 470.57277024090325, + 487.232330096314, + 468.2267177290404 + ], + "qps_1t": 470.57277024090325, + "qps_1t_min": 468.2267177290404, + "qps_1t_max": 487.232330096314, + "median_ms": 1.9831500248983502, + "p95_ms": 2.7151750109624118, + "p99_ms": 4.885184983140761, + "mean_ms": 2.1043987691772332, + "knob": null + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 2000, + "threads": 32, + "qps": 733.8303687431866, + "median_ms": 1.276099996175617, + "p95_ms": 1.7859099782072008, + "p99_ms": 3.2207130180904646, + "mean_ms": 1.3543596997818288 + }, + "mem_after_sweep": { + "rss_mb": 195.34375, + "peak_mb": 199.09765625 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.12977739999769256, + "per_s": 77055.01882591115, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.1261600000434555, + "per_s": 79264.42609825241, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.11555340001359582, + "per_s": 86540.07583354031, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 217.94921875, + "peak_mb": 339.4375 + }, + "first_query_after_ms": 3.411400015465915, + "after": { + "nq": 1133, + "recall_queries": 1133, + "recall": 1.0, + "qps_1t_runs": [ + 368.6149218486817, + 362.15402827025093, + 359.13217803847925 + ], + "qps_1t": 362.15402827025093, + "qps_1t_min": 359.13217803847925, + "qps_1t_max": 368.6149218486817, + "median_ms": 2.634999982547015, + "p95_ms": 3.3076899824663992, + "p99_ms": 4.197101996978744, + "mean_ms": 2.7528681083551927 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 4.347207100014202, + "disk_mb": 215.12163829803467, + "load_s": 3.3704436999978498, + "first_query_after_load_ms": 173.42160001862794, + "time_to_ready_s": 3.5438653000164777 + }, + "mem_end": { + "rss_mb": 205.3046875, + "peak_mb": 1124.09765625 + }, + "status": "ok", + "finished": "2026-11-16 08:26:04", + "load_before_pct": 4.871874999999999, + "load_contaminated": false, + "wall_s": 38.56782865524292, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/isong_py__glove__100000__filter.json b/benchmarks/competitive/results/raw/isong_py__glove__100000__filter.json new file mode 100644 index 0000000..4480c19 --- /dev/null +++ b/benchmarks/competitive/results/raw/isong_py__glove__100000__filter.json @@ -0,0 +1,66 @@ +{ + "system": "isong_py", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 08:54:36", + "filter": { + "build_with_metadata_s": 0.951974900031928, + "mem": { + "rss_mb": 119.0625, + "peak_mb": 198.6328125 + }, + "subsets": { + "f10": [ + { + "knob": null, + "nq": 806, + "recall": 1.0, + "qps_1t": 368.30133653177387, + "qps_1t_runs": [ + 364.21885098908365, + 368.30133653177387, + 374.0330828163496 + ], + "median_ms": 2.841649984475225, + "p95_ms": 3.657465026481077, + "p99_ms": 5.016813997644929, + "mean_ms": 2.711443424126059 + } + ], + "f01": [ + { + "knob": null, + "nq": 1224, + "recall": 1.0, + "qps_1t": 409.2525993129238, + "qps_1t_runs": [ + 402.3647606355641, + 409.2525993129238, + 410.3109945700142 + ], + "median_ms": 2.3779500043019652, + "p95_ms": 2.875359982135705, + "p99_ms": 3.5022469749674197, + "mean_ms": 2.4553205072302786 + } + ] + } + }, + "mem_end": { + "rss_mb": 70.609375, + "peak_mb": 200.77734375 + }, + "status": "ok", + "finished": "2026-11-16 08:54:53", + "load_before_pct": 3.5187499999999963, + "load_contaminated": false, + "wall_s": 17.066850662231445, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/isong_py__sift__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/isong_py__sift__100000__core+mutation+saveload.json new file mode 100644 index 0000000..f16578f --- /dev/null +++ b/benchmarks/competitive/results/raw/isong_py__sift__100000__core+mutation+saveload.json @@ -0,0 +1,135 @@ +{ + "system": "isong_py", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 11:14:48", + "mem_baseline": { + "rss_mb": 35.4453125, + "peak_mb": 35.4453125 + }, + "version": "isongraph-vector (python, repo HEAD)", + "build": { + "ingest_s": 1.2639303999603726, + "index_s": 2.800021320581436e-06, + "total_s": 1.2639331999816932, + "first_query_ms": 217.70449995528907, + "ready_s": 1.4816376999369822, + "total_s_runs": [ + 1.2639331999816932, + 4.486591300053988, + 4.277331799909007 + ] + }, + "mem_after_build": { + "rss_mb": 120.2890625, + "peak_mb": 230.625 + }, + "mem_after_first_query": { + "rss_mb": 205.3359375, + "peak_mb": 230.625 + }, + "sweep": [ + { + "nq": 1127, + "recall_queries": 1127, + "recall": 1.0, + "qps_1t_runs": [ + 323.34627532053725, + 318.4160723281366, + 265.48087410898836 + ], + "qps_1t": 318.4160723281366, + "qps_1t_min": 265.48087410898836, + "qps_1t_max": 323.34627532053725, + "median_ms": 3.2240000437013805, + "p95_ms": 4.41530003445223, + "p99_ms": 5.68031995790078, + "mean_ms": 3.3333181309646243, + "knob": null + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 827, + "threads": 32, + "qps": 221.62838441176189, + "median_ms": 3.9862999692559242, + "p95_ms": 6.613600009586662, + "p99_ms": 9.214140009134965, + "mean_ms": 4.304634542628621 + }, + "mem_after_sweep": { + "rss_mb": 207.0234375, + "peak_mb": 230.625 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.809980700025335, + "per_s": 12345.973181443971, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.6764652999700047, + "per_s": 14782.724258647726, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.4257113999919966, + "per_s": 23490.09211448883, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 250.2421875, + "peak_mb": 391.02734375 + }, + "first_query_after_ms": 10.10459999088198, + "after": { + "nq": 669, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 202.95658902539296, + 215.85714318386917, + 211.1228945033651 + ], + "qps_1t": 211.1228945033651, + "qps_1t_min": 202.95658902539296, + "qps_1t_max": 215.85714318386917, + "median_ms": 4.807600053027272, + "p95_ms": 5.517460004193706, + "p99_ms": 6.0025979694910365, + "mean_ms": 4.765477778511164 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 10.997030099970289, + "disk_mb": 222.15946578979492, + "load_s": 9.703692900016904, + "first_query_after_load_ms": 528.2480000169016, + "time_to_ready_s": 10.231940900033806 + }, + "mem_end": { + "rss_mb": 235.609375, + "peak_mb": 1276.5859375 + }, + "status": "ok", + "finished": "2026-11-16 11:15:55", + "load_before_pct": 15.33437500000001, + "load_contaminated": false, + "wall_s": 67.6143126487732, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/isong_py__text384__1000000__core+mutation.json b/benchmarks/competitive/results/raw/isong_py__text384__1000000__core+mutation.json new file mode 100644 index 0000000..36c9095 --- /dev/null +++ b/benchmarks/competitive/results/raw/isong_py__text384__1000000__core+mutation.json @@ -0,0 +1,122 @@ +{ + "system": "isong_py", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation" + ], + "started": "2026-11-16 09:44:46", + "mem_baseline": { + "rss_mb": 37.44140625, + "peak_mb": 37.44140625 + }, + "version": "isongraph-vector (python, repo HEAD)", + "build": { + "ingest_s": 19.603895099658985, + "index_s": 2.8999638743698597e-06, + "total_s": 19.60389799962286, + "first_query_ms": 3159.0028000064194, + "ready_s": 22.76290079962928 + }, + "mem_after_build": { + "rss_mb": 2458.02734375, + "peak_mb": 2719.56640625 + }, + "mem_after_first_query": { + "rss_mb": 4260.78125, + "peak_mb": 4282.1875 + }, + "sweep": [ + { + "nq": 36, + "recall_queries": 36, + "recall": 1.0, + "qps_1t_runs": [ + 12.691703331185803, + 12.691139131289624, + 12.803293120965305 + ], + "qps_1t": 12.691703331185803, + "qps_1t_min": 12.691139131289624, + "qps_1t_max": 12.803293120965305, + "median_ms": 77.24439998855814, + "p95_ms": 86.11922999843955, + "p99_ms": 92.96392500284126, + "mean_ms": 78.56388981464853, + "knob": null + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 64, + "threads": 32, + "qps": 21.255942570846003, + "median_ms": 46.58279998693615, + "p95_ms": 51.560625023557805, + "p99_ms": 56.12793797801715, + "mean_ms": 47.07301093882658 + }, + "mem_after_sweep": { + "rss_mb": 4261.796875, + "peak_mb": 4282.1875 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.28422920004231855, + "per_s": 35182.873534848324, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.21219099999871105, + "per_s": 47127.352244255155, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.1480594999738969, + "per_s": 67540.41450743124, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 4319.8125, + "peak_mb": 4319.8125 + }, + "first_query_after_ms": 76.06789999408647, + "after": { + "nq": 37, + "recall_queries": 37, + "recall": 1.0, + "qps_1t_runs": [ + 12.633175004505267, + 12.728609263827707, + 12.417905062494837 + ], + "qps_1t": 12.633175004505267, + "qps_1t_min": 12.417905062494837, + "qps_1t_max": 12.728609263827707, + "median_ms": 78.01810000091791, + "p95_ms": 86.07399999164045, + "p99_ms": 98.09642998152417, + "mean_ms": 79.41624143702053 + }, + "recall_after": 1.0 + }, + "mem_end": { + "rss_mb": 4287.57421875, + "peak_mb": 4331.37890625 + }, + "status": "ok", + "finished": "2026-11-16 09:45:45", + "load_before_pct": 4.121874999999999, + "load_contaminated": false, + "wall_s": 59.9491012096405, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/isong_py__text384__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/isong_py__text384__100000__core+mutation+saveload.json new file mode 100644 index 0000000..5eef2c0 --- /dev/null +++ b/benchmarks/competitive/results/raw/isong_py__text384__100000__core+mutation+saveload.json @@ -0,0 +1,135 @@ +{ + "system": "isong_py", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 08:03:11", + "mem_baseline": { + "rss_mb": 37.32421875, + "peak_mb": 37.32421875 + }, + "version": "isongraph-vector (python, repo HEAD)", + "build": { + "ingest_s": 2.0098258999641985, + "index_s": 2.6999623514711857e-06, + "total_s": 2.00982859992655, + "first_query_ms": 302.6059999829158, + "ready_s": 2.3124345999094658, + "total_s_runs": [ + 2.00982859992655, + 1.9037308999104425, + 1.9565701999817975 + ] + }, + "mem_after_build": { + "rss_mb": 301.0703125, + "peak_mb": 608.5 + }, + "mem_after_first_query": { + "rss_mb": 491.3671875, + "peak_mb": 608.5 + }, + "sweep": [ + { + "nq": 411, + "recall_queries": 1000, + "recall": 0.9999, + "qps_1t_runs": [ + 130.79180149272884, + 129.21784657734457, + 130.55264871905612 + ], + "qps_1t": 130.55264871905612, + "qps_1t_min": 129.21784657734457, + "qps_1t_max": 130.79180149272884, + "median_ms": 7.368899998255074, + "p95_ms": 8.921960019506512, + "p99_ms": 14.31764797540383, + "mean_ms": 7.681451094858364, + "knob": null + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 592, + "threads": 32, + "qps": 182.78046255038845, + "median_ms": 5.295849987305701, + "p95_ms": 5.986950010992587, + "p99_ms": 11.848275025840849, + "mean_ms": 5.468754278898904 + }, + "mem_after_sweep": { + "rss_mb": 492.27734375, + "peak_mb": 608.5 + }, + "ref_knob": null, + "ref_recall_pre": 0.9999, + "mutation": { + "ref_knob": null, + "recall_before": 0.9999, + "insert_10k": { + "seconds": 0.20596729998942465, + "per_s": 48551.39626782236, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.19502439995994791, + "per_s": 51275.63526437562, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.1280096999835223, + "per_s": 78119.0800485215, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 607.47265625, + "peak_mb": 1030.41015625 + }, + "first_query_after_ms": 9.15479997638613, + "after": { + "nq": 335, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 117.71077355262582, + 111.40207559022471, + 113.48448766836302 + ], + "qps_1t": 113.48448766836302, + "qps_1t_min": 111.40207559022471, + "qps_1t_max": 117.71077355262582, + "median_ms": 8.436200034338981, + "p95_ms": 10.36600002553314, + "p99_ms": 13.535275987815114, + "mean_ms": 8.761223183657547 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 15.478363099973649, + "disk_mb": 807.4883146286011, + "load_s": 10.803367899963632, + "first_query_after_load_ms": 290.74789996957406, + "time_to_ready_s": 11.094115799933206 + }, + "mem_end": { + "rss_mb": 554.36328125, + "peak_mb": 3815.515625 + }, + "status": "ok", + "finished": "2026-11-16 08:04:24", + "load_before_pct": 5.115624999999998, + "load_contaminated": false, + "wall_s": 73.05485653877258, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/isong_py__text384__100000__filter.json b/benchmarks/competitive/results/raw/isong_py__text384__100000__filter.json new file mode 100644 index 0000000..6fd1ad7 --- /dev/null +++ b/benchmarks/competitive/results/raw/isong_py__text384__100000__filter.json @@ -0,0 +1,66 @@ +{ + "system": "isong_py", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 08:43:25", + "filter": { + "build_with_metadata_s": 1.971411700011231, + "mem": { + "rss_mb": 305.13671875, + "peak_mb": 608.5234375 + }, + "subsets": { + "f10": [ + { + "knob": null, + "nq": 373, + "recall": 1.0, + "qps_1t": 120.92616604587073, + "qps_1t_runs": [ + 120.56053215637243, + 120.92616604587073, + 121.09234516207893 + ], + "median_ms": 8.045599970500916, + "p95_ms": 9.732029965380201, + "p99_ms": 14.989941958338022, + "mean_ms": 8.274085879593445 + } + ], + "f01": [ + { + "knob": null, + "nq": 413, + "recall": 1.0, + "qps_1t": 130.75389562362255, + "qps_1t_runs": [ + 130.40696316614958, + 132.61044267594022, + 130.75389562362255 + ], + "median_ms": 7.410000020172447, + "p95_ms": 8.720459992764518, + "p99_ms": 10.668384009040883, + "mean_ms": 7.619047377057843 + } + ] + } + }, + "mem_end": { + "rss_mb": 67.578125, + "peak_mb": 608.5234375 + }, + "status": "ok", + "finished": "2026-11-16 08:43:46", + "load_before_pct": 4.221874999999997, + "load_contaminated": false, + "wall_s": 21.843071699142456, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/isong_py__text384__10000__core.json b/benchmarks/competitive/results/raw/isong_py__text384__10000__core.json new file mode 100644 index 0000000..8bf2217 --- /dev/null +++ b/benchmarks/competitive/results/raw/isong_py__text384__10000__core.json @@ -0,0 +1,79 @@ +{ + "system": "isong_py", + "ds": "text384", + "n": 10000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 08:57:19", + "mem_baseline": { + "rss_mb": 37.21875, + "peak_mb": 37.21875 + }, + "version": "isongraph-vector (python, repo HEAD)", + "build": { + "ingest_s": 0.18982440000399947, + "index_s": 2.700020559132099e-06, + "total_s": 0.1898271000245586, + "first_query_ms": 32.44519996223971, + "ready_s": 0.22227229998679832 + }, + "mem_after_build": { + "rss_mb": 71.58203125, + "peak_mb": 232.5390625 + }, + "mem_after_first_query": { + "rss_mb": 89.04296875, + "peak_mb": 232.5390625 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 1.0, + "qps_1t_runs": [ + 2336.9320918560566, + 2206.460009760146, + 2201.8797907733897 + ], + "qps_1t": 2206.460009760146, + "qps_1t_min": 2201.8797907733897, + "qps_1t_max": 2336.9320918560566, + "median_ms": 0.42050000047311187, + "p95_ms": 0.5032299901358784, + "p99_ms": 0.8731300116051017, + "mean_ms": 0.4450944997079205, + "knob": null + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 2000, + "threads": 32, + "qps": 3333.930658197846, + "median_ms": 0.2832000027410686, + "p95_ms": 0.400005019037053, + "p99_ms": 0.6143000209704041, + "mean_ms": 0.3009677502947549 + }, + "mem_after_sweep": { + "rss_mb": 90.64453125, + "peak_mb": 232.5390625 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 85.28125, + "peak_mb": 232.5390625 + }, + "status": "ok", + "finished": "2026-11-16 08:57:24", + "load_before_pct": 5.9937500000000075, + "load_contaminated": false, + "wall_s": 4.94828724861145, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/isong_py_ivf__glove__1000000__core+mutation.json b/benchmarks/competitive/results/raw/isong_py_ivf__glove__1000000__core+mutation.json new file mode 100644 index 0000000..61de5d3 --- /dev/null +++ b/benchmarks/competitive/results/raw/isong_py_ivf__glove__1000000__core+mutation.json @@ -0,0 +1,301 @@ +{ + "system": "isong_py_ivf", + "ds": "glove", + "n": 1000000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation" + ], + "started": "2026-11-16 10:14:52", + "mem_baseline": { + "rss_mb": 35.23828125, + "peak_mb": 35.23828125 + }, + "version": "isongraph-vector (python, repo HEAD)", + "build": { + "ingest_s": 10.126719099993352, + "index_s": 11.820377000025474, + "total_s": 21.947096100018825, + "first_query_ms": 28.81889999844134, + "ready_s": 21.975915000017267 + }, + "mem_after_build": { + "rss_mb": 1479.4296875, + "peak_mb": 1668.84375 + }, + "mem_after_first_query": { + "rss_mb": 1479.52734375, + "peak_mb": 1668.84375 + }, + "index_stats": { + "enabled": true, + "mode": "ivf", + "active": true, + "nlist": 4000, + "nprobe": 4000, + "trained_n": 1000000, + "pending": 0, + "churn": 0, + "build_seconds": 9.951588800002355, + "index_bytes": 10132072, + "min_size": 50000 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.3014, + "qps_1t_runs": [ + 7673.6312150786935, + 7436.414915525494, + 7446.640160591303 + ], + "qps_1t": 7446.640160591303, + "qps_1t_min": 7436.414915525494, + "qps_1t_max": 7673.6312150786935, + "median_ms": 0.12725000851787627, + "p95_ms": 0.18259999342262745, + "p99_ms": 0.22690398152917632, + "mean_ms": 0.13302618371866023, + "knob": 1 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.41955, + "qps_1t_runs": [ + 5473.482212912351, + 5429.38126627681, + 5494.671545272573 + ], + "qps_1t": 5473.482212912351, + "qps_1t_min": 5429.38126627681, + "qps_1t_max": 5494.671545272573, + "median_ms": 0.17030001617968082, + "p95_ms": 0.24620500917080793, + "p99_ms": 0.3202259936369961, + "mean_ms": 0.18295886652776971, + "knob": 2 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.53585, + "qps_1t_runs": [ + 3936.734298016734, + 3631.4954561523077, + 3761.8734120014833 + ], + "qps_1t": 3761.8734120014833, + "qps_1t_min": 3631.4954561523077, + "qps_1t_max": 3936.734298016734, + "median_ms": 0.24750002194195986, + "p95_ms": 0.3604999539675191, + "p99_ms": 0.4447320097824566, + "mean_ms": 0.26507043366048794, + "knob": 4 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.639, + "qps_1t_runs": [ + 2411.959267253749, + 2437.508973631519, + 2541.966274453487 + ], + "qps_1t": 2437.508973631519, + "qps_1t_min": 2411.959267253749, + "qps_1t_max": 2541.966274453487, + "median_ms": 0.3833499795291573, + "p95_ms": 0.5698150314856321, + "p99_ms": 0.6890139501774687, + "mean_ms": 0.4060839498900653, + "knob": 8 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7264, + "qps_1t_runs": [ + 1512.3768765919078, + 1502.6607620354866, + 1496.5012562519441 + ], + "qps_1t": 1502.6607620354866, + "qps_1t_min": 1496.5012562519441, + "qps_1t_max": 1512.3768765919078, + "median_ms": 0.6358500104397535, + "p95_ms": 0.9423200157471002, + "p99_ms": 1.0954140167450535, + "mean_ms": 0.6649741163516106, + "knob": 16 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7996, + "qps_1t_runs": [ + 881.0629739591949, + 853.6949537055935, + 890.8565734356268 + ], + "qps_1t": 881.0629739591949, + "qps_1t_min": 853.6949537055935, + "qps_1t_max": 890.8565734356268, + "median_ms": 1.0962999949697405, + "p95_ms": 1.5826100017875433, + "p99_ms": 1.8951249937526908, + "mean_ms": 1.1429621335313034, + "knob": 32 + }, + { + "nq": 1635, + "recall_queries": 1635, + "recall": 0.8634250764525994, + "qps_1t_runs": [ + 498.4330757686888, + 492.3623903698503, + 492.0661185138374 + ], + "qps_1t": 492.3623903698503, + "qps_1t_min": 492.0661185138374, + "qps_1t_max": 498.4330757686888, + "median_ms": 1.9783999887295067, + "p95_ms": 2.7038399945013225, + "p99_ms": 3.073048002552241, + "mean_ms": 2.0231863199927815, + "knob": 64 + }, + { + "nq": 774, + "recall_queries": 1000, + "recall": 0.9207000000000001, + "qps_1t_runs": [ + 251.2547479487054, + 256.49987753534083, + 254.68469296302763 + ], + "qps_1t": 254.68469296302763, + "qps_1t_min": 251.2547479487054, + "qps_1t_max": 256.49987753534083, + "median_ms": 3.775549994315952, + "p95_ms": 5.023495008936152, + "p99_ms": 11.496879992191678, + "mean_ms": 3.9350284242885816, + "knob": 128 + }, + { + "nq": 418, + "recall_queries": 1000, + "recall": 0.9574, + "qps_1t_runs": [ + 138.35026249925076, + 136.85363665981453, + 140.22167975817968 + ], + "qps_1t": 138.35026249925076, + "qps_1t_min": 136.85363665981453, + "qps_1t_max": 140.22167975817968, + "median_ms": 6.950399983907118, + "p95_ms": 9.135000038077123, + "p99_ms": 14.947907001478601, + "mean_ms": 7.222224162364946, + "knob": 256 + }, + { + "nq": 111, + "recall_queries": 111, + "recall": 1.0, + "qps_1t_runs": [ + 35.67577928921252, + 32.765002232607976, + 34.60663402601248 + ], + "qps_1t": 34.60663402601248, + "qps_1t_min": 32.765002232607976, + "qps_1t_max": 35.67577928921252, + "median_ms": 27.95209997566417, + "p95_ms": 34.97580002294853, + "p99_ms": 53.4547840058804, + "mean_ms": 29.148928528999793, + "knob": 4000 + } + ], + "knob_name": "nprobe", + "mem_after_sweep": { + "rss_mb": 1482.39453125, + "peak_mb": 1668.84375 + }, + "ref_knob": 256, + "ref_recall_pre": 0.9574, + "mutation": { + "ref_knob": 256, + "recall_before": 0.9574, + "insert_10k": { + "seconds": 0.16740310000022873, + "per_s": 59736.05028811495, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.15563410002505407, + "per_s": 64253.27096304855, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.1476425000000745, + "per_s": 67731.17496652355, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 1496.15234375, + "peak_mb": 1668.84375 + }, + "first_query_after_ms": 293.5758999665268, + "after": { + "nq": 423, + "recall_queries": 1000, + "recall": 0.9566999999999999, + "qps_1t_runs": [ + 141.2508143568193, + 135.43345478895924, + 138.8761622412545 + ], + "qps_1t": 138.8761622412545, + "qps_1t_min": 135.43345478895924, + "qps_1t_max": 141.2508143568193, + "median_ms": 7.083099975716323, + "p95_ms": 9.031679993495345, + "p99_ms": 10.92401604168117, + "mean_ms": 7.221321670062931 + }, + "recall_after": 0.9566999999999999, + "index_stats_after": { + "enabled": true, + "mode": "ivf", + "active": true, + "nlist": 4000, + "nprobe": 256, + "trained_n": 1000000, + "pending": 0, + "churn": 30000, + "build_seconds": 9.951588800002355, + "index_bytes": 10132072, + "min_size": 50000 + } + }, + "mem_end": { + "rss_mb": 1488.04296875, + "peak_mb": 1668.84375 + }, + "status": "ok", + "finished": "2026-11-16 10:16:31", + "load_before_pct": 5.75625, + "load_contaminated": false, + "wall_s": 99.30513596534729, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/isong_py_ivf__glove__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/isong_py_ivf__glove__100000__core+mutation+saveload.json new file mode 100644 index 0000000..6a79383 --- /dev/null +++ b/benchmarks/competitive/results/raw/isong_py_ivf__glove__100000__core+mutation+saveload.json @@ -0,0 +1,314 @@ +{ + "system": "isong_py_ivf", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 08:17:34", + "mem_baseline": { + "rss_mb": 35.4765625, + "peak_mb": 35.4765625 + }, + "version": "isongraph-vector (python, repo HEAD)", + "build": { + "ingest_s": 0.9291130999918096, + "index_s": 0.8479947000159882, + "total_s": 1.7771078000077978, + "first_query_ms": 3.174100012984127, + "ready_s": 1.780281900020782, + "total_s_runs": [ + 1.7771078000077978, + 1.7793858999502845, + 1.7270096999127418 + ] + }, + "mem_after_build": { + "rss_mb": 211.15234375, + "peak_mb": 313.45703125 + }, + "mem_after_first_query": { + "rss_mb": 211.2578125, + "peak_mb": 313.45703125 + }, + "index_stats": { + "enabled": true, + "mode": "ivf", + "active": true, + "nlist": 1265, + "nprobe": 1265, + "trained_n": 100000, + "pending": 0, + "churn": 0, + "build_seconds": 0.6770595000125468, + "index_bytes": 1366192, + "min_size": 50000 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.28525, + "qps_1t_runs": [ + 14707.947768870014, + 14669.736592289886, + 16767.593652751257 + ], + "qps_1t": 14707.947768870014, + "qps_1t_min": 14669.736592289886, + "qps_1t_max": 16767.593652751257, + "median_ms": 0.05859998054802418, + "p95_ms": 0.08270498947240414, + "p99_ms": 0.11720601934939635, + "mean_ms": 0.06526561683858745, + "knob": 1 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.40225, + "qps_1t_runs": [ + 14100.911595623664, + 12857.081349663158, + 12777.454446384369 + ], + "qps_1t": 12857.081349663158, + "qps_1t_min": 12777.454446384369, + "qps_1t_max": 14100.911595623664, + "median_ms": 0.06910000229254365, + "p95_ms": 0.09180000051856041, + "p99_ms": 0.12980999890714906, + "mean_ms": 0.07565280021905589, + "knob": 2 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.51915, + "qps_1t_runs": [ + 9812.348654203588, + 10234.48222592726, + 9844.026336901727 + ], + "qps_1t": 9844.026336901727, + "qps_1t_min": 9812.348654203588, + "qps_1t_max": 10234.48222592726, + "median_ms": 0.0944999628700316, + "p95_ms": 0.12749998131766915, + "p99_ms": 0.16630298574455088, + "mean_ms": 0.10040191657996426, + "knob": 4 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6309, + "qps_1t_runs": [ + 6997.952063673539, + 6339.397393306068, + 6091.877696571202 + ], + "qps_1t": 6339.397393306068, + "qps_1t_min": 6091.877696571202, + "qps_1t_max": 6997.952063673539, + "median_ms": 0.13280002167448401, + "p95_ms": 0.1836050010751933, + "p99_ms": 0.24820502672810119, + "mean_ms": 0.15493188323065019, + "knob": 8 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7282, + "qps_1t_runs": [ + 4917.7917231284255, + 4775.358762802328, + 4946.8277885201705 + ], + "qps_1t": 4917.7917231284255, + "qps_1t_min": 4775.358762802328, + "qps_1t_max": 4946.8277885201705, + "median_ms": 0.19829999655485153, + "p95_ms": 0.26099995302502066, + "p99_ms": 0.30341198202222613, + "mean_ms": 0.2049671334971208, + "knob": 16 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.81335, + "qps_1t_runs": [ + 3151.535437876544, + 3178.276355214525, + 3160.883939295367 + ], + "qps_1t": 3160.883939295367, + "qps_1t_min": 3151.535437876544, + "qps_1t_max": 3178.276355214525, + "median_ms": 0.31274999491870403, + "p95_ms": 0.39591004315298056, + "p99_ms": 0.45373502012807954, + "mean_ms": 0.31610293308040127, + "knob": 32 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.87955, + "qps_1t_runs": [ + 1876.346863706935, + 1876.4220349860614, + 1866.647627425756 + ], + "qps_1t": 1876.346863706935, + "qps_1t_min": 1866.647627425756, + "qps_1t_max": 1876.4220349860614, + "median_ms": 0.5209000082686543, + "p95_ms": 0.6576200015842916, + "p99_ms": 0.7712239929242065, + "mean_ms": 0.5338664667312211, + "knob": 64 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.93615, + "qps_1t_runs": [ + 1071.4241894386548, + 1035.3602444476332, + 1010.8230846182003 + ], + "qps_1t": 1035.3602444476332, + "qps_1t_min": 1010.8230846182003, + "qps_1t_max": 1071.4241894386548, + "median_ms": 0.9303499828092754, + "p95_ms": 1.1795050173532218, + "p99_ms": 1.7031010455684736, + "mean_ms": 0.9628257835671926, + "knob": 128 + }, + { + "nq": 1803, + "recall_queries": 1803, + "recall": 0.9728230726566833, + "qps_1t_runs": [ + 546.5862462760346, + 558.3354306974928, + 534.8742457788961 + ], + "qps_1t": 546.5862462760346, + "qps_1t_min": 534.8742457788961, + "qps_1t_max": 558.3354306974928, + "median_ms": 1.7321999766863883, + "p95_ms": 2.310219977516681, + "p99_ms": 3.5273239715024833, + "mean_ms": 1.8300579403861799, + "knob": 256 + }, + { + "nq": 1414, + "recall_queries": 1414, + "recall": 1.0, + "qps_1t_runs": [ + 435.42610792602176, + 440.0730658606763, + 446.3582924195241 + ], + "qps_1t": 440.0730658606763, + "qps_1t_min": 435.42610792602176, + "qps_1t_max": 446.3582924195241, + "median_ms": 2.0861999946646392, + "p95_ms": 3.1089449854334807, + "p99_ms": 5.4630870069377195, + "mean_ms": 2.2697678454782775, + "knob": 1265 + } + ], + "knob_name": "nprobe", + "mem_after_sweep": { + "rss_mb": 212.9453125, + "peak_mb": 313.45703125 + }, + "ref_knob": 256, + "ref_recall_pre": 0.9728230726566833, + "mutation": { + "ref_knob": 256, + "recall_before": 0.9728230726566833, + "insert_10k": { + "seconds": 0.13686079997569323, + "per_s": 73066.94102165135, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.14041749999159947, + "per_s": 71216.19456690409, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.11773510003695264, + "per_s": 84936.43779010145, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 238.40625, + "peak_mb": 461.671875 + }, + "first_query_after_ms": 78.54189997306094, + "after": { + "nq": 1711, + "recall_queries": 1711, + "recall": 0.9715955581531268, + "qps_1t_runs": [ + 532.9944762334069, + 531.9560717725213, + 527.8145138522884 + ], + "qps_1t": 531.9560717725213, + "qps_1t_min": 527.8145138522884, + "qps_1t_max": 532.9944762334069, + "median_ms": 1.7930999747477472, + "p95_ms": 2.330859994981438, + "p99_ms": 3.5363199957646447, + "mean_ms": 1.8835504969410322 + }, + "recall_after": 0.9715955581531268, + "index_stats_after": { + "enabled": true, + "mode": "ivf", + "active": true, + "nlist": 1265, + "nprobe": 256, + "trained_n": 100000, + "pending": 0, + "churn": 30000, + "build_seconds": 0.6770595000125468, + "index_bytes": 1366192, + "min_size": 50000 + } + }, + "saveload": { + "save_s": 4.42302980000386, + "disk_mb": 215.12163829803467, + "load_s": 3.408074300037697, + "first_query_after_load_ms": 2017.010500014294, + "time_to_ready_s": 5.425084800051991 + }, + "mem_end": { + "rss_mb": 233.03515625, + "peak_mb": 1146.86328125 + }, + "status": "ok", + "finished": "2026-11-16 08:18:34", + "load_before_pct": 3.8406249999999966, + "load_contaminated": false, + "wall_s": 60.26763033866882, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/isong_py_ivf__glove__100000__filter.json b/benchmarks/competitive/results/raw/isong_py_ivf__glove__100000__filter.json new file mode 100644 index 0000000..d417187 --- /dev/null +++ b/benchmarks/competitive/results/raw/isong_py_ivf__glove__100000__filter.json @@ -0,0 +1,186 @@ +{ + "system": "isong_py_ivf", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 08:46:28", + "filter": { + "build_with_metadata_s": 1.7763816000078805, + "mem": { + "rss_mb": 208.26171875, + "peak_mb": 310.90625 + }, + "subsets": { + "f10": [ + { + "knob": 2, + "nq": 2000, + "recall": 0.26685, + "qps_1t": 16284.563057732048, + "qps_1t_runs": [ + 15431.634240173593, + 16284.563057732048, + 16703.372267074268 + ], + "median_ms": 0.05570001667365432, + "p95_ms": 0.1017050148220733, + "p99_ms": 0.11810496798716497, + "mean_ms": 0.06202598349773325 + }, + { + "knob": 8, + "nq": 2000, + "recall": 0.47395, + "qps_1t": 12530.668764699758, + "qps_1t_runs": [ + 12530.668764699758, + 11967.649048568923, + 12809.616446447559 + ], + "median_ms": 0.07450004341080785, + "p95_ms": 0.1108050288166851, + "p99_ms": 0.17210397170856603, + "mean_ms": 0.08047638341668062 + }, + { + "knob": 32, + "nq": 2000, + "recall": 0.699, + "qps_1t": 7019.023997824779, + "qps_1t_runs": [ + 7019.023997824779, + 7024.553625664872, + 6576.78613658603 + ], + "median_ms": 0.1403999631293118, + "p95_ms": 0.17070496105588975, + "p99_ms": 0.20820995618123583, + "mean_ms": 0.14562589992419817 + }, + { + "knob": 64, + "nq": 2000, + "recall": 0.79565, + "qps_1t": 4564.469702070527, + "qps_1t_runs": [ + 4564.469702070527, + 4598.897913853672, + 4419.855325873309 + ], + "median_ms": 0.21739999647252262, + "p95_ms": 0.2603050001198426, + "p99_ms": 0.32729998929426074, + "mean_ms": 0.22092621674528345 + }, + { + "knob": 1265, + "nq": 844, + "recall": 1.0, + "qps_1t": 366.3635385348709, + "qps_1t_runs": [ + 382.9506904201276, + 366.3635385348709, + 366.235165687388 + ], + "median_ms": 2.8377499838825315, + "p95_ms": 3.57357996981591, + "p99_ms": 4.024586987798103, + "mean_ms": 2.6904391787381985 + } + ], + "f01": [ + { + "knob": 2, + "nq": 2000, + "recall": 0.3628, + "qps_1t": 6464.186786614411, + "qps_1t_runs": [ + 6481.589195167696, + 6074.568384296385, + 6464.186786614411 + ], + "median_ms": 0.17140002455562353, + "p95_ms": 0.2121049852576107, + "p99_ms": 0.26980401657056075, + "mean_ms": 0.15786746660402665 + }, + { + "knob": 8, + "nq": 2000, + "recall": 0.36295, + "qps_1t": 7849.228862823635, + "qps_1t_runs": [ + 7511.838698306427, + 8149.676974218535, + 7849.228862823635 + ], + "median_ms": 0.14084999565966427, + "p95_ms": 0.18860503914766016, + "p99_ms": 0.3476340271299712, + "mean_ms": 0.1277428331086412 + }, + { + "knob": 32, + "nq": 2000, + "recall": 0.4572, + "qps_1t": 8799.702935747748, + "qps_1t_runs": [ + 8682.390738034337, + 8799.702935747748, + 9079.621469865477 + ], + "median_ms": 0.10870001278817654, + "p95_ms": 0.13619998935610056, + "p99_ms": 0.2081040129996837, + "mean_ms": 0.11298420006642118 + }, + { + "knob": 64, + "nq": 2000, + "recall": 0.5944, + "qps_1t": 5803.310142003998, + "qps_1t_runs": [ + 5844.977769729354, + 5725.799470926135, + 5803.310142003998 + ], + "median_ms": 0.16659998800605536, + "p95_ms": 0.20750500843860212, + "p99_ms": 0.283606005832553, + "mean_ms": 0.1726835332713866 + }, + { + "knob": 1265, + "nq": 1296, + "recall": 1.0, + "qps_1t": 399.70808973227605, + "qps_1t_runs": [ + 392.5772843516169, + 418.222874125607, + 399.70808973227605 + ], + "median_ms": 2.395149989752099, + "p95_ms": 2.927895021275617, + "p99_ms": 3.7017560144886374, + "mean_ms": 2.480054835647818 + } + ] + } + }, + "mem_end": { + "rss_mb": 89.140625, + "peak_mb": 310.90625 + }, + "status": "ok", + "finished": "2026-11-16 08:46:53", + "load_before_pct": 5.421875000000004, + "load_contaminated": false, + "wall_s": 25.205848455429077, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/isong_py_ivf__sift__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/isong_py_ivf__sift__100000__core+mutation+saveload.json new file mode 100644 index 0000000..faffae4 --- /dev/null +++ b/benchmarks/competitive/results/raw/isong_py_ivf__sift__100000__core+mutation+saveload.json @@ -0,0 +1,314 @@ +{ + "system": "isong_py_ivf", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 11:07:02", + "mem_baseline": { + "rss_mb": 35.49609375, + "peak_mb": 35.49609375 + }, + "version": "isongraph-vector (python, repo HEAD)", + "build": { + "ingest_s": 1.1146168000414036, + "index_s": 0.9648579999920912, + "total_s": 2.079474800033495, + "first_query_ms": 3.1757999677211046, + "ready_s": 2.082650600001216, + "total_s_runs": [ + 2.079474800033495, + 2.0232003999408334, + 2.079858400102239 + ] + }, + "mem_after_build": { + "rss_mb": 226.88671875, + "peak_mb": 338.74609375 + }, + "mem_after_first_query": { + "rss_mb": 227.01953125, + "peak_mb": 338.74609375 + }, + "index_stats": { + "enabled": true, + "mode": "ivf", + "active": true, + "nlist": 1265, + "nprobe": 1265, + "trained_n": 100000, + "pending": 0, + "churn": 0, + "build_seconds": 0.7610118000302464, + "index_bytes": 1507872, + "min_size": 50000 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.28555, + "qps_1t_runs": [ + 15392.27607357927, + 15419.58226987032, + 15348.958528000368 + ], + "qps_1t": 15392.27607357927, + "qps_1t_min": 15348.958528000368, + "qps_1t_max": 15419.58226987032, + "median_ms": 0.06079999729990959, + "p95_ms": 0.08440000237897038, + "p99_ms": 0.12070198776200418, + "mean_ms": 0.06499041660572402, + "knob": 1 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.4253, + "qps_1t_runs": [ + 13796.071975524428, + 13991.003778776161, + 13604.211294384017 + ], + "qps_1t": 13796.071975524428, + "qps_1t_min": 13604.211294384017, + "qps_1t_max": 13991.003778776161, + "median_ms": 0.06940000457689166, + "p95_ms": 0.09139999747276306, + "p99_ms": 0.11860903003253062, + "mean_ms": 0.07248851673405927, + "knob": 2 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.5855, + "qps_1t_runs": [ + 10214.968975221796, + 10221.510350151724, + 10168.264424150182 + ], + "qps_1t": 10214.968975221796, + "qps_1t_min": 10168.264424150182, + "qps_1t_max": 10221.510350151724, + "median_ms": 0.09300000965595245, + "p95_ms": 0.13000000035390258, + "p99_ms": 0.162600958137773, + "mean_ms": 0.09802454985522975, + "knob": 4 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.74265, + "qps_1t_runs": [ + 7152.30843273225, + 7097.662790220524, + 6537.391899606096 + ], + "qps_1t": 7097.662790220524, + "qps_1t_min": 6537.391899606096, + "qps_1t_max": 7152.30843273225, + "median_ms": 0.13615001807920635, + "p95_ms": 0.18560501630418005, + "p99_ms": 0.2301119477488103, + "mean_ms": 0.1445575498995216, + "knob": 8 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.86895, + "qps_1t_runs": [ + 4347.524612886644, + 4513.194431744968, + 4293.412687223287 + ], + "qps_1t": 4347.524612886644, + "qps_1t_min": 4293.412687223287, + "qps_1t_max": 4513.194431744968, + "median_ms": 0.2115000388585031, + "p95_ms": 0.2977049967739731, + "p99_ms": 0.45842899824492694, + "mean_ms": 0.22816781633688757, + "knob": 16 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.94685, + "qps_1t_runs": [ + 2760.6593944371084, + 2610.4272750694454, + 2722.420756329228 + ], + "qps_1t": 2722.420756329228, + "qps_1t_min": 2610.4272750694454, + "qps_1t_max": 2760.6593944371084, + "median_ms": 0.33924999297596514, + "p95_ms": 0.5158350249985234, + "p99_ms": 0.9291180310538043, + "mean_ms": 0.3708771662786603, + "knob": 32 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.98685, + "qps_1t_runs": [ + 1663.2730355330273, + 1711.4863581881468, + 1687.841766628338 + ], + "qps_1t": 1687.841766628338, + "qps_1t_min": 1663.2730355330273, + "qps_1t_max": 1711.4863581881468, + "median_ms": 0.5703000060748309, + "p95_ms": 0.7775450387271122, + "p99_ms": 1.0802220174809922, + "mean_ms": 0.5926614166722478, + "knob": 64 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9976, + "qps_1t_runs": [ + 893.8901448134695, + 875.8977675382142, + 880.1468017721687 + ], + "qps_1t": 880.1468017721687, + "qps_1t_min": 875.8977675382142, + "qps_1t_max": 893.8901448134695, + "median_ms": 1.065549993654713, + "p95_ms": 1.5799250395502895, + "p99_ms": 2.2450089815538377, + "mean_ms": 1.1321885165913652, + "knob": 128 + }, + { + "nq": 1542, + "recall_queries": 1542, + "recall": 0.9998054474708171, + "qps_1t_runs": [ + 509.2492979560646, + 495.6974616079729, + 499.1166249830159 + ], + "qps_1t": 499.1166249830159, + "qps_1t_min": 495.6974616079729, + "qps_1t_max": 509.2492979560646, + "median_ms": 1.900399976875633, + "p95_ms": 2.5255000218749046, + "p99_ms": 3.3655000297585502, + "mean_ms": 1.994858020490782, + "knob": 256 + }, + { + "nq": 1068, + "recall_queries": 1068, + "recall": 1.0, + "qps_1t_runs": [ + 338.395514854715, + 337.1322796044335, + 354.2290672561122 + ], + "qps_1t": 338.395514854715, + "qps_1t_min": 337.1322796044335, + "qps_1t_max": 354.2290672561122, + "median_ms": 2.7924499881919473, + "p95_ms": 3.714610004681162, + "p99_ms": 4.719091969891444, + "mean_ms": 2.914782895979247, + "knob": 1265 + } + ], + "knob_name": "nprobe", + "mem_after_sweep": { + "rss_mb": 231.66796875, + "peak_mb": 338.74609375 + }, + "ref_knob": 64, + "ref_recall_pre": 0.98685, + "mutation": { + "ref_knob": 64, + "recall_before": 0.98685, + "insert_10k": { + "seconds": 0.15103710000403225, + "per_s": 66208.89834175199, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.1571733999880962, + "per_s": 63623.99744967894, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.13506170001346618, + "per_s": 74040.23493709143, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 273.59375, + "peak_mb": 522.51171875 + }, + "first_query_after_ms": 96.44670004490763, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.98565, + "qps_1t_runs": [ + 1594.6595487628774, + 1523.5822339139527, + 1296.6704087350088 + ], + "qps_1t": 1523.5822339139527, + "qps_1t_min": 1296.6704087350088, + "qps_1t_max": 1594.6595487628774, + "median_ms": 0.6274499755818397, + "p95_ms": 0.9622199926525355, + "p99_ms": 1.872863000025974, + "mean_ms": 0.6848823336185887 + }, + "recall_after": 0.98565, + "index_stats_after": { + "enabled": true, + "mode": "ivf", + "active": true, + "nlist": 1265, + "nprobe": 64, + "trained_n": 100000, + "pending": 0, + "churn": 30000, + "build_seconds": 0.7610118000302464, + "index_bytes": 1507872, + "min_size": 50000 + } + }, + "saveload": { + "save_s": 4.966320100007579, + "disk_mb": 222.15946578979492, + "load_s": 3.661656900017988, + "first_query_after_load_ms": 2379.858599975705, + "time_to_ready_s": 6.041515499993693 + }, + "mem_end": { + "rss_mb": 270.921875, + "peak_mb": 1303.83984375 + }, + "status": "ok", + "finished": "2026-11-16 11:07:59", + "load_before_pct": 8.184375000000001, + "load_contaminated": false, + "wall_s": 57.658135414123535, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/isong_py_ivf__text384__1000000__core+mutation.json b/benchmarks/competitive/results/raw/isong_py_ivf__text384__1000000__core+mutation.json new file mode 100644 index 0000000..dbf69e3 --- /dev/null +++ b/benchmarks/competitive/results/raw/isong_py_ivf__text384__1000000__core+mutation.json @@ -0,0 +1,301 @@ +{ + "system": "isong_py_ivf", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation" + ], + "started": "2026-11-16 09:10:26", + "mem_baseline": { + "rss_mb": 37.265625, + "peak_mb": 37.265625 + }, + "version": "isongraph-vector (python, repo HEAD)", + "build": { + "ingest_s": 20.75935379974544, + "index_s": 21.402648299990688, + "total_s": 42.16200209973613, + "first_query_ms": 78.90149997547269, + "ready_s": 42.2409035997116 + }, + "mem_after_build": { + "rss_mb": 4287.9765625, + "peak_mb": 4832.640625 + }, + "mem_after_first_query": { + "rss_mb": 4288.07421875, + "peak_mb": 4832.640625 + }, + "index_stats": { + "enabled": true, + "mode": "ivf", + "active": true, + "nlist": 4000, + "nprobe": 4000, + "trained_n": 1000000, + "pending": 0, + "churn": 0, + "build_seconds": 17.958885200030636, + "index_bytes": 14676072, + "min_size": 50000 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.2745, + "qps_1t_runs": [ + 3293.372653744326, + 3317.069190745804, + 3159.7134000636274 + ], + "qps_1t": 3293.372653744326, + "qps_1t_min": 3159.7134000636274, + "qps_1t_max": 3317.069190745804, + "median_ms": 0.28979999478906393, + "p95_ms": 0.42301001667510724, + "p99_ms": 0.5193099961616101, + "mean_ms": 0.30719848316706094, + "knob": 1 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.37635, + "qps_1t_runs": [ + 2355.801677791145, + 2438.017073734718, + 2389.2213614623442 + ], + "qps_1t": 2389.2213614623442, + "qps_1t_min": 2355.801677791145, + "qps_1t_max": 2438.017073734718, + "median_ms": 0.39329996798187494, + "p95_ms": 0.5690999707439914, + "p99_ms": 0.7120090076932686, + "mean_ms": 0.41773324977839366, + "knob": 2 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.494, + "qps_1t_runs": [ + 1688.896628288766, + 1741.5275347085244, + 1711.4826977091589 + ], + "qps_1t": 1711.4826977091589, + "qps_1t_min": 1688.896628288766, + "qps_1t_max": 1741.5275347085244, + "median_ms": 0.5573999951593578, + "p95_ms": 0.7863999926485121, + "p99_ms": 1.025820011273027, + "mean_ms": 0.583533266413724, + "knob": 4 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6007, + "qps_1t_runs": [ + 1077.1719936220263, + 1165.851736510476, + 1175.020958467421 + ], + "qps_1t": 1165.851736510476, + "qps_1t_min": 1077.1719936220263, + "qps_1t_max": 1175.020958467421, + "median_ms": 0.85704997763969, + "p95_ms": 1.131199998781085, + "p99_ms": 1.326803975389339, + "mean_ms": 0.8790491668332834, + "knob": 8 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.70055, + "qps_1t_runs": [ + 657.8049600662753, + 667.5915257092452, + 686.6548691486038 + ], + "qps_1t": 667.5915257092452, + "qps_1t_min": 657.8049600662753, + "qps_1t_max": 686.6548691486038, + "median_ms": 1.444549998268485, + "p95_ms": 1.8761549668852242, + "p99_ms": 2.2502150211948906, + "mean_ms": 1.4914883664265897, + "knob": 16 + }, + { + "nq": 1191, + "recall_queries": 1191, + "recall": 0.7812762384550798, + "qps_1t_runs": [ + 395.66668252476427, + 399.41108508059085, + 384.71149400228575 + ], + "qps_1t": 395.66668252476427, + "qps_1t_min": 384.71149400228575, + "qps_1t_max": 399.41108508059085, + "median_ms": 2.495800028555095, + "p95_ms": 3.082580014597624, + "p99_ms": 3.6152119957841924, + "mean_ms": 2.5434721521071397, + "knob": 32 + }, + { + "nq": 689, + "recall_queries": 1000, + "recall": 0.854, + "qps_1t_runs": [ + 217.04706556251222, + 213.03474582024813, + 219.32285258958467 + ], + "qps_1t": 217.04706556251222, + "qps_1t_min": 213.03474582024813, + "qps_1t_max": 219.32285258958467, + "median_ms": 4.465100006200373, + "p95_ms": 5.410000000847503, + "p99_ms": 6.545270028291263, + "mean_ms": 4.620284663808875, + "knob": 64 + }, + { + "nq": 404, + "recall_queries": 1000, + "recall": 0.9118999999999999, + "qps_1t_runs": [ + 124.9592336207618, + 124.59377876794932, + 124.61836394926401 + ], + "qps_1t": 124.61836394926401, + "qps_1t_min": 124.59377876794932, + "qps_1t_max": 124.9592336207618, + "median_ms": 7.763800007523969, + "p95_ms": 9.83091500820592, + "p99_ms": 15.338851984124673, + "mean_ms": 8.017730776064187, + "knob": 128 + }, + { + "nq": 211, + "recall_queries": 1000, + "recall": 0.9531999999999999, + "qps_1t_runs": [ + 65.01708763627119, + 67.3822125431702, + 65.83238830865464 + ], + "qps_1t": 65.83238830865464, + "qps_1t_min": 65.01708763627119, + "qps_1t_max": 67.3822125431702, + "median_ms": 14.624100003857166, + "p95_ms": 18.05828002979979, + "p99_ms": 23.72098397929221, + "mean_ms": 15.137125117543167, + "knob": 256 + }, + { + "nq": 37, + "recall_queries": 37, + "recall": 1.0, + "qps_1t_runs": [ + 12.055673098901895, + 11.484309793851029, + 11.445846791054567 + ], + "qps_1t": 11.484309793851029, + "qps_1t_min": 11.445846791054567, + "qps_1t_max": 12.055673098901895, + "median_ms": 83.045499981381, + "p95_ms": 99.29074998944998, + "p99_ms": 139.0585399756675, + "mean_ms": 85.7972531534337, + "knob": 4000 + } + ], + "knob_name": "nprobe", + "mem_after_sweep": { + "rss_mb": 4292.02734375, + "peak_mb": 4832.640625 + }, + "ref_knob": 256, + "ref_recall_pre": 0.9531999999999999, + "mutation": { + "ref_knob": 256, + "recall_before": 0.9531999999999999, + "insert_10k": { + "seconds": 0.268599400005769, + "per_s": 37230.16507030626, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.22824040002888069, + "per_s": 43813.4528275215, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.16251339996233582, + "per_s": 61533.38741493073, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 4352.0078125, + "peak_mb": 4832.640625 + }, + "first_query_after_ms": 664.8137000156567, + "after": { + "nq": 212, + "recall_queries": 1000, + "recall": 0.9540000000000001, + "qps_1t_runs": [ + 66.50885036239598, + 68.82862351398286, + 65.08190988094326 + ], + "qps_1t": 66.50885036239598, + "qps_1t_min": 65.08190988094326, + "qps_1t_max": 68.82862351398286, + "median_ms": 14.517750008963048, + "p95_ms": 17.528975018649362, + "p99_ms": 23.239050002302967, + "mean_ms": 14.976561635548876 + }, + "recall_after": 0.9540000000000001, + "index_stats_after": { + "enabled": true, + "mode": "ivf", + "active": true, + "nlist": 4000, + "nprobe": 256, + "trained_n": 1000000, + "pending": 0, + "churn": 30000, + "build_seconds": 17.958885200030636, + "index_bytes": 14676072, + "min_size": 50000 + } + }, + "mem_end": { + "rss_mb": 4320.7421875, + "peak_mb": 4832.640625 + }, + "status": "ok", + "finished": "2026-11-16 09:13:05", + "load_before_pct": 5.971875000000003, + "load_contaminated": false, + "wall_s": 160.93938040733337, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/isong_py_ivf__text384__1000000__filter.json b/benchmarks/competitive/results/raw/isong_py_ivf__text384__1000000__filter.json new file mode 100644 index 0000000..4955a50 --- /dev/null +++ b/benchmarks/competitive/results/raw/isong_py_ivf__text384__1000000__filter.json @@ -0,0 +1,186 @@ +{ + "system": "isong_py_ivf", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 10:45:19", + "filter": { + "build_with_metadata_s": 116.88233719981508, + "mem": { + "rss_mb": 4288.765625, + "peak_mb": 4833.14453125 + }, + "subsets": { + "f10": [ + { + "knob": 2, + "nq": 2000, + "recall": 0.30765, + "qps_1t": 2767.7964137097997, + "qps_1t_runs": [ + 2521.6521614777334, + 2824.470190279404, + 2767.7964137097997 + ], + "median_ms": 0.3793000360019505, + "p95_ms": 0.4487999831326306, + "p99_ms": 0.5065029574325309, + "mean_ms": 0.3706374502896021 + }, + { + "knob": 8, + "nq": 2000, + "recall": 0.5351, + "qps_1t": 1946.4960742901098, + "qps_1t_runs": [ + 1946.4960742901098, + 1937.1118584742012, + 2003.7612600927503 + ], + "median_ms": 0.5242999759502709, + "p95_ms": 0.6293050159001723, + "p99_ms": 0.6993030279409141, + "mean_ms": 0.5096791835967451 + }, + { + "knob": 32, + "nq": 2000, + "recall": 0.7416, + "qps_1t": 1040.4644423938623, + "qps_1t_runs": [ + 1026.1317259853643, + 1042.7509102610582, + 1040.4644423938623 + ], + "median_ms": 0.9917500137817115, + "p95_ms": 1.1970100313192233, + "p99_ms": 1.3191239745356147, + "mean_ms": 0.9648816004240265 + }, + { + "knob": 64, + "nq": 1909, + "recall": 0.8242011524358303, + "qps_1t": 649.2092520951015, + "qps_1t_runs": [ + 659.6261847780818, + 633.7770360220844, + 649.2092520951015 + ], + "median_ms": 1.553500012960285, + "p95_ms": 1.8928999779745936, + "p99_ms": 2.099613980390131, + "mean_ms": 1.544729177592059 + }, + { + "knob": 4000, + "nq": 30, + "recall": 1.0, + "qps_1t": 8.642143777309547, + "qps_1t_runs": [ + 8.386630369495343, + 8.642143777309547, + 8.651004464746551 + ], + "median_ms": 116.28655000822619, + "p95_ms": 129.68221000337508, + "p99_ms": 135.23232901701704, + "mean_ms": 116.84764777827594 + } + ], + "f01": [ + { + "knob": 2, + "nq": 2000, + "recall": 0.33035, + "qps_1t": 1627.7126539663636, + "qps_1t_runs": [ + 1595.1886566576757, + 1627.7126539663636, + 1672.4455245077775 + ], + "median_ms": 0.680800003465265, + "p95_ms": 0.7686050317715853, + "p99_ms": 1.1919149948516863, + "mean_ms": 0.6130569831390554 + }, + { + "knob": 8, + "nq": 2000, + "recall": 0.3679, + "qps_1t": 2595.411206890395, + "qps_1t_runs": [ + 2543.8188644426073, + 2805.059317766705, + 2595.411206890395 + ], + "median_ms": 0.3963999915868044, + "p95_ms": 0.4947999870637432, + "p99_ms": 0.7482999935746193, + "mean_ms": 0.3783012837035737 + }, + { + "knob": 32, + "nq": 2000, + "recall": 0.60955, + "qps_1t": 1654.3760308882959, + "qps_1t_runs": [ + 1671.0643516605112, + 1648.7714509287766, + 1654.3760308882959 + ], + "median_ms": 0.6339500250760466, + "p95_ms": 0.8098299964331092, + "p99_ms": 1.0052180284401406, + "mean_ms": 0.6031302333188554 + }, + { + "knob": 64, + "nq": 2000, + "recall": 0.72475, + "qps_1t": 1119.139256835492, + "qps_1t_runs": [ + 1146.3782935707418, + 1068.1510258643498, + 1119.139256835492 + ], + "median_ms": 0.9490499796811491, + "p95_ms": 1.1690999963320792, + "p99_ms": 1.3412139686988669, + "mean_ms": 0.9006844832232067 + }, + { + "knob": 4000, + "nq": 30, + "recall": 1.0, + "qps_1t": 9.524133344978656, + "qps_1t_runs": [ + 9.325284287417265, + 9.529939846115665, + 9.524133344978656 + ], + "median_ms": 105.76075001154095, + "p95_ms": 115.39345996570773, + "p99_ms": 116.6629690042464, + "mean_ms": 105.72140777496517 + } + ] + } + }, + "mem_end": { + "rss_mb": 236.66015625, + "peak_mb": 4833.14453125 + }, + "status": "ok", + "finished": "2026-11-16 10:48:17", + "load_before_pct": 6.046874999999994, + "load_contaminated": false, + "wall_s": 178.48971605300903, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/isong_py_ivf__text384__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/isong_py_ivf__text384__100000__core+mutation+saveload.json new file mode 100644 index 0000000..acc9859 --- /dev/null +++ b/benchmarks/competitive/results/raw/isong_py_ivf__text384__100000__core+mutation+saveload.json @@ -0,0 +1,314 @@ +{ + "system": "isong_py_ivf", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 07:51:18", + "mem_baseline": { + "rss_mb": 37.4296875, + "peak_mb": 37.4296875 + }, + "version": "isongraph-vector (python, repo HEAD)", + "build": { + "ingest_s": 1.9661553999758326, + "index_s": 1.598260400001891, + "total_s": 3.5644157999777235, + "first_query_ms": 7.300400000531226, + "ready_s": 3.5717161999782547, + "total_s_runs": [ + 3.5644157999777235, + 3.5697892999160103, + 3.4769746000529267 + ] + }, + "mem_after_build": { + "rss_mb": 518.12890625, + "peak_mb": 730.7890625 + }, + "mem_after_first_query": { + "rss_mb": 518.26171875, + "peak_mb": 730.7890625 + }, + "index_stats": { + "enabled": true, + "mode": "ivf", + "active": true, + "nlist": 1265, + "nprobe": 1265, + "trained_n": 100000, + "pending": 0, + "churn": 0, + "build_seconds": 1.2876625999924727, + "index_bytes": 2803232, + "min_size": 50000 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.2833, + "qps_1t_runs": [ + 8545.117014877103, + 8513.975024681084, + 8572.634849849783 + ], + "qps_1t": 8545.117014877103, + "qps_1t_min": 8513.975024681084, + "qps_1t_max": 8572.634849849783, + "median_ms": 0.11399999493733048, + "p95_ms": 0.1388999546179548, + "p99_ms": 0.17499999434221536, + "mean_ms": 0.11704336649078566, + "knob": 1 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.40835, + "qps_1t_runs": [ + 7155.804035074119, + 7225.869465126067, + 6793.505962437482 + ], + "qps_1t": 7155.804035074119, + "qps_1t_min": 6793.505962437482, + "qps_1t_max": 7225.869465126067, + "median_ms": 0.13539998326450586, + "p95_ms": 0.1750050490954891, + "p99_ms": 0.21620101644657555, + "mean_ms": 0.141779250028776, + "knob": 2 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.53055, + "qps_1t_runs": [ + 5451.192477884488, + 5400.288278604827, + 5018.829396046299 + ], + "qps_1t": 5400.288278604827, + "qps_1t_min": 5018.829396046299, + "qps_1t_max": 5451.192477884488, + "median_ms": 0.18009997438639402, + "p95_ms": 0.2297000028192997, + "p99_ms": 0.32366996165365136, + "mean_ms": 0.18929034980828874, + "knob": 4 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.65005, + "qps_1t_runs": [ + 3854.1423096422295, + 4005.6471591323034, + 3877.4155091854054 + ], + "qps_1t": 3877.4155091854054, + "qps_1t_min": 3854.1423096422295, + "qps_1t_max": 4005.6471591323034, + "median_ms": 0.24500000290572643, + "p95_ms": 0.3024050296517089, + "p99_ms": 0.3510289953555918, + "mean_ms": 0.25567079975735396, + "knob": 8 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.75425, + "qps_1t_runs": [ + 2640.2807580120398, + 2668.5303656896854, + 2574.986832836467 + ], + "qps_1t": 2640.2807580120398, + "qps_1t_min": 2574.986832836467, + "qps_1t_max": 2668.5303656896854, + "median_ms": 0.36914998781867325, + "p95_ms": 0.4582199966534973, + "p99_ms": 0.5930029758019374, + "mean_ms": 0.38061240019548376, + "knob": 16 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.84285, + "qps_1t_runs": [ + 1578.517514672899, + 1603.0987263170168, + 1580.8420906025253 + ], + "qps_1t": 1580.8420906025253, + "qps_1t_min": 1578.517514672899, + "qps_1t_max": 1603.0987263170168, + "median_ms": 0.5910000181756914, + "p95_ms": 0.740724994102493, + "p99_ms": 1.075628983089701, + "mean_ms": 0.6299573164239215, + "knob": 32 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.90905, + "qps_1t_runs": [ + 947.1340107756721, + 937.1362879959365, + 927.44732622626 + ], + "qps_1t": 937.1362879959365, + "qps_1t_min": 927.44732622626, + "qps_1t_max": 947.1340107756721, + "median_ms": 1.0270499915350229, + "p95_ms": 1.264340023044497, + "p99_ms": 1.6760009556310274, + "mean_ms": 1.0670419333231014, + "knob": 64 + }, + { + "nq": 1546, + "recall_queries": 1546, + "recall": 0.9540103492884864, + "qps_1t_runs": [ + 506.33548171855824, + 500.7509319772026, + 499.6443347609951 + ], + "qps_1t": 500.7509319772026, + "qps_1t_min": 499.6443347609951, + "qps_1t_max": 506.33548171855824, + "median_ms": 1.9275000086054206, + "p95_ms": 2.336295036366209, + "p99_ms": 2.831875989795663, + "mean_ms": 1.991133204036412, + "knob": 128 + }, + { + "nq": 814, + "recall_queries": 1000, + "recall": 0.9809000000000001, + "qps_1t_runs": [ + 270.967193538073, + 267.5820998714532, + 267.91264258534295 + ], + "qps_1t": 267.91264258534295, + "qps_1t_min": 267.5820998714532, + "qps_1t_max": 270.967193538073, + "median_ms": 3.5941500100307167, + "p95_ms": 4.1410549863940105, + "p99_ms": 5.3537270141532955, + "mean_ms": 3.720071457291516, + "knob": 256 + }, + { + "nq": 422, + "recall_queries": 1000, + "recall": 0.9999, + "qps_1t_runs": [ + 132.1166454887518, + 131.20162407022406, + 129.7549899662039 + ], + "qps_1t": 131.20162407022406, + "qps_1t_min": 129.7549899662039, + "qps_1t_max": 132.1166454887518, + "median_ms": 7.360049989074469, + "p95_ms": 8.474500005831942, + "p99_ms": 14.194564969511685, + "mean_ms": 7.63258617693164, + "knob": 1265 + } + ], + "knob_name": "nprobe", + "mem_after_sweep": { + "rss_mb": 520.328125, + "peak_mb": 730.7890625 + }, + "ref_knob": 128, + "ref_recall_pre": 0.9540103492884864, + "mutation": { + "ref_knob": 128, + "recall_before": 0.9540103492884864, + "insert_10k": { + "seconds": 0.21102709998376667, + "per_s": 47387.278699130366, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.2037728000432253, + "per_s": 49074.2630904554, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.125521200010553, + "per_s": 79667.8170632472, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 626.26171875, + "peak_mb": 1189.75390625 + }, + "first_query_after_ms": 202.5108000379987, + "after": { + "nq": 1348, + "recall_queries": 1348, + "recall": 0.9481454005934719, + "qps_1t_runs": [ + 480.78613822988945, + 485.09851486509643, + 491.1899543299876 + ], + "qps_1t": 485.09851486509643, + "qps_1t_min": 480.78613822988945, + "qps_1t_max": 491.1899543299876, + "median_ms": 1.9911999988835305, + "p95_ms": 2.4066649901214987, + "p99_ms": 2.9588390036951755, + "mean_ms": 2.0590786843932105 + }, + "recall_after": 0.9481454005934719, + "index_stats_after": { + "enabled": true, + "mode": "ivf", + "active": true, + "nlist": 1265, + "nprobe": 128, + "trained_n": 100000, + "pending": 0, + "churn": 30000, + "build_seconds": 1.2876625999924727, + "index_bytes": 2803232, + "min_size": 50000 + } + }, + "saveload": { + "save_s": 15.985765799996443, + "disk_mb": 807.4883146286011, + "load_s": 10.858780499955174, + "first_query_after_load_ms": 5128.686800017022, + "time_to_ready_s": 15.987467299972195 + }, + "mem_end": { + "rss_mb": 601.13671875, + "peak_mb": 3852.5078125 + }, + "status": "ok", + "finished": "2026-11-16 07:53:00", + "load_before_pct": 6.809375000000003, + "load_contaminated": false, + "wall_s": 103.10571908950806, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/isong_py_ivf__text384__100000__filter.json b/benchmarks/competitive/results/raw/isong_py_ivf__text384__100000__filter.json new file mode 100644 index 0000000..0b40f81 --- /dev/null +++ b/benchmarks/competitive/results/raw/isong_py_ivf__text384__100000__filter.json @@ -0,0 +1,186 @@ +{ + "system": "isong_py_ivf", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 08:34:13", + "filter": { + "build_with_metadata_s": 3.6155435000546277, + "mem": { + "rss_mb": 516.1875, + "peak_mb": 728.88671875 + }, + "subsets": { + "f10": [ + { + "knob": 2, + "nq": 2000, + "recall": 0.3022, + "qps_1t": 8861.818984661579, + "qps_1t_runs": [ + 8900.08316833595, + 8637.521665845838, + 8861.818984661579 + ], + "median_ms": 0.10250002378597856, + "p95_ms": 0.18759997619781643, + "p99_ms": 0.21991198649629976, + "mean_ms": 0.11365870014803174 + }, + { + "knob": 8, + "nq": 2000, + "recall": 0.53195, + "qps_1t": 7773.371979587393, + "qps_1t_runs": [ + 7606.6509746269, + 7820.546622743308, + 7773.371979587393 + ], + "median_ms": 0.12510002125054598, + "p95_ms": 0.1470999588491395, + "p99_ms": 0.19650604052003487, + "mean_ms": 0.12932549992304607 + }, + { + "knob": 32, + "nq": 2000, + "recall": 0.76625, + "qps_1t": 4816.732941998437, + "qps_1t_runs": [ + 4925.221581385811, + 4816.732941998437, + 4764.162068792711 + ], + "median_ms": 0.2001000102609396, + "p95_ms": 0.2452050044666976, + "p99_ms": 0.3167250246042391, + "mean_ms": 0.20684888349690786 + }, + { + "knob": 64, + "nq": 2000, + "recall": 0.86115, + "qps_1t": 3413.984740720125, + "qps_1t_runs": [ + 3503.986221983041, + 3413.2453730892416, + 3413.984740720125 + ], + "median_ms": 0.2775000175461173, + "p95_ms": 0.3369000041857362, + "p99_ms": 0.4359090013895186, + "mean_ms": 0.29042613327813643 + }, + { + "knob": 1265, + "nq": 348, + "recall": 1.0, + "qps_1t": 121.55768630020441, + "qps_1t_runs": [ + 123.99780021377535, + 121.55768630020441, + 121.52248393576335 + ], + "median_ms": 7.936599984532222, + "p95_ms": 9.63313004467636, + "p99_ms": 13.48838001489628, + "mean_ms": 8.173378641289833 + } + ], + "f01": [ + { + "knob": 2, + "nq": 2000, + "recall": 0.4917, + "qps_1t": 3663.1022882635857, + "qps_1t_runs": [ + 3663.587424894048, + 3566.1220518706914, + 3663.1022882635857 + ], + "median_ms": 0.2905000001192093, + "p95_ms": 0.34620501392055303, + "p99_ms": 0.5144060170277954, + "mean_ms": 0.27545526656710234 + }, + { + "knob": 8, + "nq": 2000, + "recall": 0.49185, + "qps_1t": 4826.572783158953, + "qps_1t_runs": [ + 4826.572783158953, + 4862.180286632508, + 4607.146321359404 + ], + "median_ms": 0.22950000129640102, + "p95_ms": 0.27199997566640377, + "p99_ms": 0.41464995592832676, + "mean_ms": 0.20996983356114166 + }, + { + "knob": 32, + "nq": 2000, + "recall": 0.5694, + "qps_1t": 6221.664220722495, + "qps_1t_runs": [ + 5961.758304437987, + 6221.664220722495, + 6352.509660554177 + ], + "median_ms": 0.15339997480623424, + "p95_ms": 0.18230499117635193, + "p99_ms": 0.2765170094789941, + "mean_ms": 0.16196084981008121 + }, + { + "knob": 64, + "nq": 2000, + "recall": 0.7223, + "qps_1t": 4839.23692163944, + "qps_1t_runs": [ + 4810.332213747565, + 4839.23692163944, + 4856.693546769946 + ], + "median_ms": 0.20220002625137568, + "p95_ms": 0.2329100243514404, + "p99_ms": 0.28094104316551327, + "mean_ms": 0.20681046652801646 + }, + { + "knob": 1265, + "nq": 422, + "recall": 1.0, + "qps_1t": 129.51690195616382, + "qps_1t_runs": [ + 129.7527478603694, + 128.89391337834743, + 129.51690195616382 + ], + "median_ms": 7.393149979179725, + "p95_ms": 9.319625023636036, + "p99_ms": 13.083250002819108, + "mean_ms": 7.728761373892188 + } + ] + } + }, + "mem_end": { + "rss_mb": 92.921875, + "peak_mb": 728.88671875 + }, + "status": "ok", + "finished": "2026-11-16 08:34:45", + "load_before_pct": 6.862500000000005, + "load_contaminated": false, + "wall_s": 32.22874355316162, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/js__glove__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/js__glove__100000__core+mutation+saveload.json new file mode 100644 index 0000000..280bb80 --- /dev/null +++ b/benchmarks/competitive/results/raw/js__glove__100000__core+mutation+saveload.json @@ -0,0 +1,333 @@ +{ + "status": "ok", + "label": "js", + "runs": [ + { + "system": "js_exact", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-js (repo HEAD, Node v24.18.0)", + "build": { + "ingest_s": 0.028693200030829757, + "index_s": 0, + "total_s": 0.028693200030829757, + "first_query_ms": 24.664999975357205, + "ready_s": 0.05335820000618696 + }, + "mem_baseline": { + "rss_mb": 52.39453125, + "peak_mb": 52.39453125, + "heap_mb": 10.311317443847656 + }, + "mem_after_ingest": { + "rss_mb": 356.1328125, + "peak_mb": 356.1328125, + "heap_mb": 185.67807006835938 + }, + "mem_after_build": { + "rss_mb": 360.25, + "peak_mb": 360.25, + "heap_mb": 204.57239532470703 + }, + "mem_after_first_query": { + "rss_mb": 360.25, + "peak_mb": 360.25, + "heap_mb": 204.5730438232422 + }, + "sweep": [ + { + "nq": 167, + "recall": 1, + "qps_1t": 48.0650022974727, + "qps_1t_min": 47.828708381835746, + "qps_1t_max": 48.86487053063795, + "qps_1t_runs": [ + 47.828708381835746, + 48.0650022974727, + 48.86487053063795 + ], + "median_ms": 20.08340001339093, + "p95_ms": 24.64339998550713, + "p99_ms": 27.087900030892342, + "mean_ms": 20.72590099765683, + "knob": null + } + ], + "mem_after_sweep": { + "rss_mb": 509.25390625, + "peak_mb": 509.25390625, + "heap_mb": 345.2169876098633 + } + }, + { + "system": "js_ivf", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-js (repo HEAD, Node v24.18.0)", + "knob_name": "nprobe", + "build": { + "ingest_s": 0.02707650000229478, + "index_s": 20.63768899999559, + "total_s": 20.664765499997884, + "first_query_ms": 0, + "ready_s": 20.664765499997884 + }, + "mem_baseline": { + "rss_mb": 52.39453125, + "peak_mb": 52.39453125, + "heap_mb": 10.311317443847656 + }, + "mem_after_ingest": { + "rss_mb": 697.76171875, + "peak_mb": 697.76171875, + "heap_mb": 500.4320526123047 + }, + "mem_after_build": { + "rss_mb": 694.3125, + "peak_mb": 697.76171875, + "heap_mb": 215.28913116455078 + }, + "index_stats": { + "nlist": 1265, + "trained_n": 100000, + "index_bytes": 1336064, + "build_seconds": 20.578255600000002 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.2795, + "qps_1t": 6774.909387662348, + "qps_1t_min": 6500.906542238136, + "qps_1t_max": 6777.76785109088, + "qps_1t_runs": [ + 6500.906542238136, + 6774.909387662348, + 6777.76785109088 + ], + "median_ms": 0.14459999511018395, + "p95_ms": 0.1841000048443675, + "p99_ms": 0.23309997050091624, + "mean_ms": 0.14965645001696734, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.3983, + "qps_1t": 5710.070046012471, + "qps_1t_min": 5550.393691614603, + "qps_1t_max": 5862.431026587508, + "qps_1t_runs": [ + 5550.393691614603, + 5862.431026587508, + 5710.070046012471 + ], + "median_ms": 0.1710999640636146, + "p95_ms": 0.2237000153400004, + "p99_ms": 0.25049998657777905, + "mean_ms": 0.17529143330951533, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.51655, + "qps_1t": 4523.073553428299, + "qps_1t_min": 4348.697717073671, + "qps_1t_max": 4603.7071359589545, + "qps_1t_runs": [ + 4603.7071359589545, + 4348.697717073671, + 4523.073553428299 + ], + "median_ms": 0.21810003090649843, + "p95_ms": 0.28979999478906393, + "p99_ms": 0.3281999961473048, + "mean_ms": 0.22275291668483987, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.62575, + "qps_1t": 3532.9784113280625, + "qps_1t_min": 3512.2102015833266, + "qps_1t_max": 3552.699505303434, + "qps_1t_runs": [ + 3512.2102015833266, + 3532.9784113280625, + 3552.699505303434 + ], + "median_ms": 0.2793999738059938, + "p95_ms": 0.3706999705173075, + "p99_ms": 0.4146000137552619, + "mean_ms": 0.2830814664484933, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.72415, + "qps_1t": 2300.654341422209, + "qps_1t_min": 2292.8893274212364, + "qps_1t_max": 2327.541402666945, + "qps_1t_runs": [ + 2300.654341422209, + 2327.541402666945, + 2292.8893274212364 + ], + "median_ms": 0.42610004311427474, + "p95_ms": 0.5797999911010265, + "p99_ms": 0.6764999707229435, + "mean_ms": 0.43347593313471106, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.8079, + "qps_1t": 1400.9757097129889, + "qps_1t_min": 1350.2549073477542, + "qps_1t_max": 1412.358661277198, + "qps_1t_runs": [ + 1412.358661277198, + 1350.2549073477542, + 1400.9757097129889 + ], + "median_ms": 0.7085999823175371, + "p95_ms": 0.9676999761722982, + "p99_ms": 1.130899996496737, + "mean_ms": 0.720808199994887, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.87635, + "qps_1t": 778.8949381767982, + "qps_1t_min": 764.2925575667001, + "qps_1t_max": 799.1614241173585, + "qps_1t_runs": [ + 799.1614241173585, + 778.8949381767982, + 764.2925575667001 + ], + "median_ms": 1.2710000155493617, + "p95_ms": 1.6903000068850815, + "p99_ms": 1.9036000012420118, + "mean_ms": 1.2811937831381024, + "knob": 64 + }, + { + "nq": 1366, + "recall": 0.9320644216691069, + "qps_1t": 432.3019019324809, + "qps_1t_min": 431.0375566428076, + "qps_1t_max": 446.930620647938, + "qps_1t_runs": [ + 446.930620647938, + 431.0375566428076, + 432.3019019324809 + ], + "median_ms": 2.28159996913746, + "p95_ms": 2.958500001113862, + "p99_ms": 3.2582999556325376, + "mean_ms": 2.290221815281118, + "knob": 128 + }, + { + "nq": 651, + "recall": 0.9745007680491552, + "qps_1t": 229.55479554178177, + "qps_1t_min": 221.3337592742448, + "qps_1t_max": 232.24724033020672, + "qps_1t_runs": [ + 229.55479554178177, + 221.3337592742448, + 232.24724033020672 + ], + "median_ms": 4.37069998588413, + "p95_ms": 5.45820000115782, + "p99_ms": 6.29390001995489, + "mean_ms": 4.393359396143645, + "knob": 256 + }, + { + "nq": 154, + "recall": 1, + "qps_1t": 50.813091960134905, + "qps_1t_min": 49.47061145795657, + "qps_1t_max": 51.25979091027489, + "qps_1t_runs": [ + 51.25979091027489, + 50.813091960134905, + 49.47061145795657 + ], + "median_ms": 19.09049996174872, + "p95_ms": 23.098700039554387, + "p99_ms": 24.940599978435785, + "mean_ms": 19.800819047107826, + "knob": 1265 + } + ], + "ref_knob": 256, + "ref_recall_pre": 0.9745007680491552, + "mem_after_sweep": { + "rss_mb": 710.4453125, + "peak_mb": 710.4453125, + "heap_mb": 335.0654525756836 + }, + "mutation": { + "ref_knob": 256, + "recall_before": 0.9745007680491552, + "insert_10k": { + "seconds": 0.01814970001578331, + "per_s": 550973.2938452876 + }, + "update_10k": { + "seconds": 0.008544000040274113, + "per_s": 1170411.97950172 + }, + "delete_10k": { + "seconds": 0.0057483999989926815, + "per_s": 1739614.5017313245 + }, + "first_query_after_ms": 1173.6763999797404, + "recall_after": 0.9744485294117647, + "after": { + "nq": 544, + "recall": 0.9744485294117647, + "qps_1t": 224.84353221535994, + "qps_1t_min": 222.28020758946403, + "qps_1t_max": 226.69062031863888, + "qps_1t_runs": [ + 224.84353221535994, + 226.69062031863888, + 222.28020758946403 + ], + "median_ms": 4.454899986740202, + "p95_ms": 5.3301999578252435, + "p99_ms": 6.0882000252604485, + "mean_ms": 4.452553982927758 + }, + "mem_after": { + "rss_mb": 680.4375, + "peak_mb": 710.4453125, + "heap_mb": 161.70631408691406 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 100000 + } + } + } + ], + "load_before_pct": 3.3781249999999976, + "load_contaminated": false, + "wall_s": 90.08911609649658, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/js__sift__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/js__sift__100000__core+mutation+saveload.json new file mode 100644 index 0000000..18fe836 --- /dev/null +++ b/benchmarks/competitive/results/raw/js__sift__100000__core+mutation+saveload.json @@ -0,0 +1,333 @@ +{ + "status": "ok", + "label": "js", + "runs": [ + { + "system": "js_exact", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-js (repo HEAD, Node v24.18.0)", + "build": { + "ingest_s": 0.034534100035671145, + "index_s": 0, + "total_s": 0.034534100035671145, + "first_query_ms": 28.81559997331351, + "ready_s": 0.06334970000898466 + }, + "mem_baseline": { + "rss_mb": 52.86328125, + "peak_mb": 52.86328125, + "heap_mb": 9.740829467773438 + }, + "mem_after_ingest": { + "rss_mb": 365.77734375, + "peak_mb": 365.77734375, + "heap_mb": 187.73895263671875 + }, + "mem_after_build": { + "rss_mb": 369.75390625, + "peak_mb": 369.75390625, + "heap_mb": 207.31876373291016 + }, + "mem_after_first_query": { + "rss_mb": 369.75390625, + "peak_mb": 369.75390625, + "heap_mb": 207.3194122314453 + }, + "sweep": [ + { + "nq": 106, + "recall": 1, + "qps_1t": 29.015488331823676, + "qps_1t_min": 27.56688246071011, + "qps_1t_max": 32.53022648455043, + "qps_1t_runs": [ + 32.53022648455043, + 29.015488331823676, + 27.56688246071011 + ], + "median_ms": 33.672999998088926, + "p95_ms": 41.62020003423095, + "p99_ms": 48.66359999869019, + "mean_ms": 33.82680125742471, + "knob": null + } + ], + "mem_after_sweep": { + "rss_mb": 507.2109375, + "peak_mb": 507.2109375, + "heap_mb": 374.7082290649414 + } + }, + { + "system": "js_ivf", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-js (repo HEAD, Node v24.18.0)", + "knob_name": "nprobe", + "build": { + "ingest_s": 0.04355299996677786, + "index_s": 35.42172629997367, + "total_s": 35.46527929994045, + "first_query_ms": 0, + "ready_s": 35.46527929994045 + }, + "mem_baseline": { + "rss_mb": 52.86328125, + "peak_mb": 52.86328125, + "heap_mb": 9.740829467773438 + }, + "mem_after_ingest": { + "rss_mb": 703.01171875, + "peak_mb": 703.01171875, + "heap_mb": 486.50797271728516 + }, + "mem_after_build": { + "rss_mb": 703.66796875, + "peak_mb": 703.66796875, + "heap_mb": 213.23184967041016 + }, + "index_stats": { + "nlist": 1265, + "trained_n": 100000, + "index_bytes": 1477744, + "build_seconds": 35.29808839999999 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.2886, + "qps_1t": 3499.20865791667, + "qps_1t_min": 3416.21702191343, + "qps_1t_max": 3524.8470916818233, + "qps_1t_runs": [ + 3524.8470916818233, + 3416.21702191343, + 3499.20865791667 + ], + "median_ms": 0.28899998869746923, + "p95_ms": 0.3502999898046255, + "p99_ms": 0.424400030169636, + "mean_ms": 0.2874001997697633, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.4338, + "qps_1t": 3126.918173166386, + "qps_1t_min": 3107.9813753619237, + "qps_1t_max": 3163.5022934552576, + "qps_1t_runs": [ + 3107.9813753619237, + 3163.5022934552576, + 3126.918173166386 + ], + "median_ms": 0.3170999698340893, + "p95_ms": 0.3800999838858843, + "p99_ms": 0.42619998566806316, + "mean_ms": 0.31922043316687143, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.5894, + "qps_1t": 2439.1082697930437, + "qps_1t_min": 2407.44255962951, + "qps_1t_max": 2486.2732852665545, + "qps_1t_runs": [ + 2486.2732852665545, + 2407.44255962951, + 2439.1082697930437 + ], + "median_ms": 0.3988000098615885, + "p95_ms": 0.5239000311121345, + "p99_ms": 0.6170999840833247, + "mean_ms": 0.4091909501585178, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.7425, + "qps_1t": 1899.5105716300106, + "qps_1t_min": 1884.0905609592085, + "qps_1t_max": 1909.3722194016998, + "qps_1t_runs": [ + 1909.3722194016998, + 1899.5105716300106, + 1884.0905609592085 + ], + "median_ms": 0.5158000276423991, + "p95_ms": 0.6834000232629478, + "p99_ms": 0.7966000121086836, + "mean_ms": 0.5269812667393126, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.8685, + "qps_1t": 1318.562966176771, + "qps_1t_min": 1308.1881259520942, + "qps_1t_max": 1363.2442757955162, + "qps_1t_runs": [ + 1308.1881259520942, + 1318.562966176771, + 1363.2442757955162 + ], + "median_ms": 0.7391000399366021, + "p95_ms": 1.0248000035062432, + "p99_ms": 1.1904999846592546, + "mean_ms": 0.7521205668454058, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.9486, + "qps_1t": 809.7634476612682, + "qps_1t_min": 763.2318549516632, + "qps_1t_max": 820.0141509468529, + "qps_1t_runs": [ + 820.0141509468529, + 763.2318549516632, + 809.7634476612682 + ], + "median_ms": 1.2365999864414334, + "p95_ms": 1.7020999803207815, + "p99_ms": 1.9302999717183411, + "mean_ms": 1.2548791832232382, + "knob": 32 + }, + { + "nq": 1328, + "recall": 0.9876506024096385, + "qps_1t": 480.1718147239224, + "qps_1t_min": 463.69302783714807, + "qps_1t_max": 481.6145634518156, + "qps_1t_runs": [ + 480.1718147239224, + 481.6145634518156, + 463.69302783714807 + ], + "median_ms": 2.1003999863751233, + "p95_ms": 2.732599969021976, + "p99_ms": 3.1218999647535384, + "mean_ms": 2.105178740222697, + "knob": 64 + }, + { + "nq": 761, + "recall": 0.9984231274638633, + "qps_1t": 248.22625874677325, + "qps_1t_min": 244.3492156576467, + "qps_1t_max": 255.56793874842808, + "qps_1t_runs": [ + 255.56793874842808, + 244.3492156576467, + 248.22625874677325 + ], + "median_ms": 3.9916999521665275, + "p95_ms": 4.9854000099003315, + "p99_ms": 5.646600038744509, + "mean_ms": 4.011313315671455, + "knob": 128 + }, + { + "nq": 424, + "recall": 1, + "qps_1t": 142.33386449958948, + "qps_1t_min": 140.8073427916259, + "qps_1t_max": 145.45854247428872, + "qps_1t_runs": [ + 140.8073427916259, + 145.45854247428872, + 142.33386449958948 + ], + "median_ms": 7.057500013615936, + "p95_ms": 8.092699979897588, + "p99_ms": 8.661899948492646, + "mean_ms": 7.000816116053113, + "knob": 256 + }, + { + "nq": 79, + "recall": 1, + "qps_1t": 27.81044429292253, + "qps_1t_min": 26.752337358303926, + "qps_1t_max": 27.959403795720092, + "qps_1t_runs": [ + 26.752337358303926, + 27.959403795720092, + 27.81044429292253 + ], + "median_ms": 35.214200033806264, + "p95_ms": 42.42409998551011, + "p99_ms": 45.87979998905212, + "mean_ms": 36.36792236285771, + "knob": 1265 + } + ], + "ref_knob": 64, + "ref_recall_pre": 0.9876506024096385, + "mem_after_sweep": { + "rss_mb": 717.83984375, + "peak_mb": 717.83984375, + "heap_mb": 370.5705337524414 + }, + "mutation": { + "ref_knob": 64, + "recall_before": 0.9876506024096385, + "insert_10k": { + "seconds": 0.04443340003490448, + "per_s": 225055.92622091805 + }, + "update_10k": { + "seconds": 0.011832899996079504, + "per_s": 845101.370189321 + }, + "delete_10k": { + "seconds": 0.007865299994591624, + "per_s": 1271407.3216376044 + }, + "first_query_after_ms": 2042.7271999651566, + "recall_after": 0.9873373807458803, + "after": { + "nq": 1153, + "recall": 0.9873373807458803, + "qps_1t": 447.9219278040597, + "qps_1t_min": 441.951051784361, + "qps_1t_max": 457.85640285476217, + "qps_1t_runs": [ + 441.951051784361, + 457.85640285476217, + 447.9219278040597 + ], + "median_ms": 2.2129000280983746, + "p95_ms": 2.8608000138774514, + "p99_ms": 3.22979997145012, + "mean_ms": 2.2264389418742434 + }, + "mem_after": { + "rss_mb": 745.4140625, + "peak_mb": 745.4140625, + "heap_mb": 200.06658935546875 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 100000 + } + } + } + ], + "load_before_pct": 11.05312499999999, + "load_contaminated": false, + "wall_s": 118.04112315177917, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/js__text384__1000000__core+mutation.json b/benchmarks/competitive/results/raw/js__text384__1000000__core+mutation.json new file mode 100644 index 0000000..f8f5908 --- /dev/null +++ b/benchmarks/competitive/results/raw/js__text384__1000000__core+mutation.json @@ -0,0 +1,314 @@ +{ + "status": "ok", + "label": "js", + "runs": [ + { + "system": "js_exact", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-js (repo HEAD, Node v24.18.0)", + "build": { + "ingest_s": 0.4697339002741501, + "index_s": 0, + "total_s": 0.4697339002741501, + "first_query_ms": 557.7046000398695, + "ready_s": 1.0274385003140196 + }, + "mem_baseline": { + "rss_mb": 66.1796875, + "peak_mb": 66.1796875, + "heap_mb": 16.08751678466797 + }, + "mem_after_ingest": { + "rss_mb": 3620.58984375, + "peak_mb": 3620.58984375, + "heap_mb": 3363.2532272338867 + }, + "mem_after_build": { + "rss_mb": 3686.65234375, + "peak_mb": 3686.65234375, + "heap_mb": 3380.0526733398438 + }, + "mem_after_first_query": { + "rss_mb": 3686.65234375, + "peak_mb": 3686.65234375, + "heap_mb": 3380.053321838379 + }, + "sweep": [ + { + "nq": 30, + "recall": 1, + "qps_1t": 1.668134513829699, + "qps_1t_min": 1.668134513829699, + "qps_1t_max": 1.668134513829699, + "qps_1t_runs": [ + 1.668134513829699 + ], + "median_ms": 544.5717999828048, + "p95_ms": 838.1194999674335, + "p99_ms": 871.8156000250019, + "mean_ms": 599.4720400000612, + "knob": null + } + ], + "mem_after_sweep": { + "rss_mb": 5653.24609375, + "peak_mb": 5653.24609375, + "heap_mb": 5297.903793334961 + } + }, + { + "system": "js_ivf", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-js (repo HEAD, Node v24.18.0)", + "knob_name": "nprobe", + "build": { + "ingest_s": 0.690798299969174, + "index_s": 521.2741673999699, + "total_s": 521.9649656999391, + "first_query_ms": 0, + "ready_s": 521.9649656999391 + }, + "mem_baseline": { + "rss_mb": 66.1796875, + "peak_mb": 66.1796875, + "heap_mb": 16.08751678466797 + }, + "mem_after_ingest": { + "rss_mb": 5401.671875, + "peak_mb": 5653.24609375, + "heap_mb": 3359.64591217041 + }, + "mem_after_build": { + "rss_mb": 7068.578125, + "peak_mb": 7068.578125, + "heap_mb": 3351.2668838500977 + }, + "index_stats": { + "nlist": 4000, + "trained_n": 1000000, + "index_bytes": 14410004, + "build_seconds": 519.3788293 + }, + "sweep": [ + { + "nq": 1907, + "recall": 0.2670162558993183, + "qps_1t": 666.1068387526135, + "qps_1t_min": 659.6800313784976, + "qps_1t_max": 668.1428406805679, + "qps_1t_runs": [ + 659.6800313784976, + 668.1428406805679, + 666.1068387526135 + ], + "median_ms": 1.4743999927304685, + "p95_ms": 1.7371000139974058, + "p99_ms": 1.9876000005751848, + "mean_ms": 1.5046110121358927, + "knob": 1 + }, + { + "nq": 1634, + "recall": 0.3726438188494492, + "qps_1t": 593.6938882283083, + "qps_1t_min": 586.5755120760934, + "qps_1t_max": 594.3714982716806, + "qps_1t_runs": [ + 594.3714982716806, + 586.5755120760934, + 593.6938882283083 + ], + "median_ms": 1.659700006712228, + "p95_ms": 2.0185000030323863, + "p99_ms": 2.2839000448584557, + "mean_ms": 1.6905431661824952, + "knob": 2 + }, + { + "nq": 1452, + "recall": 0.48326446280991736, + "qps_1t": 492.7275015615987, + "qps_1t_min": 487.9994975615684, + "qps_1t_max": 493.328278641302, + "qps_1t_runs": [ + 492.7275015615987, + 487.9994975615684, + 493.328278641302 + ], + "median_ms": 2.0038000075146556, + "p95_ms": 2.546000003349036, + "p99_ms": 2.801199967507273, + "mean_ms": 2.0352498621597954, + "knob": 4 + }, + { + "nq": 1071, + "recall": 0.592250233426704, + "qps_1t": 368.0081163474449, + "qps_1t_min": 365.7176232817451, + "qps_1t_max": 368.12085748602084, + "qps_1t_runs": [ + 368.12085748602084, + 365.7176232817451, + 368.0081163474449 + ], + "median_ms": 2.697399992030114, + "p95_ms": 3.3989999792538583, + "p99_ms": 3.6818000371567905, + "mean_ms": 2.7227268594379748, + "knob": 8 + }, + { + "nq": 647, + "recall": 0.6891808346213292, + "qps_1t": 245.12318526070712, + "qps_1t_min": 243.61615537122444, + "qps_1t_max": 248.90575720084718, + "qps_1t_runs": [ + 248.90575720084718, + 243.61615537122444, + 245.12318526070712 + ], + "median_ms": 4.0283999987877905, + "p95_ms": 5.053000000771135, + "p99_ms": 5.611600005067885, + "mean_ms": 4.067328129499184, + "knob": 16 + }, + { + "nq": 465, + "recall": 0.7795698924731183, + "qps_1t": 158.0498349946911, + "qps_1t_min": 157.87083910584533, + "qps_1t_max": 161.07181691948702, + "qps_1t_runs": [ + 158.0498349946911, + 161.07181691948702, + 157.87083910584533 + ], + "median_ms": 6.228200043551624, + "p95_ms": 7.667800004128367, + "p99_ms": 8.3147999830544, + "mean_ms": 6.289940358074483, + "knob": 32 + }, + { + "nq": 252, + "recall": 0.8444444444444444, + "qps_1t": 95.89655226979065, + "qps_1t_min": 94.75926015701279, + "qps_1t_max": 96.216059115072, + "qps_1t_runs": [ + 94.75926015701279, + 96.216059115072, + 95.89655226979065 + ], + "median_ms": 10.337699961382896, + "p95_ms": 12.434499978553504, + "p99_ms": 13.532700017094612, + "mean_ms": 10.458079100726296, + "knob": 64 + }, + { + "nq": 167, + "recall": 0.9131736526946108, + "qps_1t": 52.037531084342305, + "qps_1t_min": 50.88291299377793, + "qps_1t_max": 54.306735500045434, + "qps_1t_runs": [ + 52.037531084342305, + 50.88291299377793, + 54.306735500045434 + ], + "median_ms": 18.741499981842935, + "p95_ms": 23.26260000700131, + "p99_ms": 25.570300000254065, + "mean_ms": 19.094594810627395, + "knob": 128 + }, + { + "nq": 84, + "recall": 0.9488095238095238, + "qps_1t": 28.994268559004134, + "qps_1t_min": 28.34453443266689, + "qps_1t_max": 29.417070282163937, + "qps_1t_runs": [ + 28.994268559004134, + 29.417070282163937, + 28.34453443266689 + ], + "median_ms": 34.329500049352646, + "p95_ms": 40.43839999940246, + "p99_ms": 42.12669999105856, + "mean_ms": 34.58787103105957, + "knob": 256 + } + ], + "ref_knob": 256, + "ref_recall_pre": 0, + "mem_after_sweep": { + "rss_mb": 7069.29296875, + "peak_mb": 7069.29296875, + "heap_mb": 3353.831642150879 + }, + "mutation": { + "ref_knob": 256, + "recall_before": 0, + "insert_10k": { + "seconds": 0.016971000004559755, + "per_s": 589240.4688770967 + }, + "update_10k": { + "seconds": 0.014216500043403357, + "per_s": 703408.009669731 + }, + "delete_10k": { + "seconds": 0.010688800015486777, + "per_s": 935558.7143094839 + }, + "first_query_after_ms": 5743.163299979642, + "recall_after": 0.9459770114942528, + "after": { + "nq": 87, + "recall": 0.9459770114942528, + "qps_1t": 28.75972781281632, + "qps_1t_min": 28.535344715200925, + "qps_1t_max": 28.786808735306945, + "qps_1t_runs": [ + 28.75972781281632, + 28.535344715200925, + 28.786808735306945 + ], + "median_ms": 34.423000004608184, + "p95_ms": 41.20670002885163, + "p99_ms": 43.94539998611435, + "mean_ms": 34.85107854262023 + }, + "mem_after": { + "rss_mb": 6926.3984375, + "peak_mb": 7069.29296875, + "heap_mb": 3353.5948333740234 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 1000000 + } + } + } + ], + "load_before_pct": 2.968749999999999, + "load_contaminated": false, + "wall_s": 665.79034948349, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/js__text384__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/js__text384__100000__core+mutation+saveload.json new file mode 100644 index 0000000..1bec86e --- /dev/null +++ b/benchmarks/competitive/results/raw/js__text384__100000__core+mutation+saveload.json @@ -0,0 +1,333 @@ +{ + "status": "ok", + "label": "js", + "runs": [ + { + "system": "js_exact", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-js (repo HEAD, Node v24.18.0)", + "build": { + "ingest_s": 0.029746399959549308, + "index_s": 0, + "total_s": 0.029746399959549308, + "first_query_ms": 65.6650000018999, + "ready_s": 0.0954113999614492 + }, + "mem_baseline": { + "rss_mb": 66.21875, + "peak_mb": 66.21875, + "heap_mb": 16.054977416992188 + }, + "mem_after_ingest": { + "rss_mb": 572.375, + "peak_mb": 572.375, + "heap_mb": 395.76651763916016 + }, + "mem_after_build": { + "rss_mb": 583.72265625, + "peak_mb": 583.72265625, + "heap_mb": 362.2503356933594 + }, + "mem_after_first_query": { + "rss_mb": 583.72265625, + "peak_mb": 583.72265625, + "heap_mb": 362.25098419189453 + }, + "sweep": [ + { + "nq": 56, + "recall": 1, + "qps_1t": 19.70100419273891, + "qps_1t_min": 19.095522829155946, + "qps_1t_max": 20.27124885418181, + "qps_1t_runs": [ + 19.095522829155946, + 19.70100419273891, + 20.27124885418181 + ], + "median_ms": 50.79419998219237, + "p95_ms": 55.78120000427589, + "p99_ms": 58.090100006666034, + "mean_ms": 50.819360716212984, + "knob": null + } + ], + "mem_after_sweep": { + "rss_mb": 1091.96484375, + "peak_mb": 1091.96484375, + "heap_mb": 777.9389190673828 + } + }, + { + "system": "js_ivf", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-js (repo HEAD, Node v24.18.0)", + "knob_name": "nprobe", + "build": { + "ingest_s": 0.027143400046043098, + "index_s": 56.6768264999846, + "total_s": 56.70396990003064, + "first_query_ms": 0, + "ready_s": 56.70396990003064 + }, + "mem_baseline": { + "rss_mb": 66.21875, + "peak_mb": 66.21875, + "heap_mb": 16.054977416992188 + }, + "mem_after_ingest": { + "rss_mb": 1159.66796875, + "peak_mb": 1159.66796875, + "heap_mb": 396.57994079589844 + }, + "mem_after_build": { + "rss_mb": 1282.41796875, + "peak_mb": 1282.41796875, + "heap_mb": 410.84786224365234 + }, + "index_stats": { + "nlist": 1265, + "trained_n": 100000, + "index_bytes": 2773104, + "build_seconds": 56.57028999999999 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.2654, + "qps_1t": 2125.2072196805125, + "qps_1t_min": 2089.4242216666457, + "qps_1t_max": 2145.2915898579677, + "qps_1t_runs": [ + 2089.4242216666457, + 2125.2072196805125, + 2145.2915898579677 + ], + "median_ms": 0.46749995090067387, + "p95_ms": 0.5320000345818698, + "p99_ms": 0.5902000120840967, + "mean_ms": 0.47176006677909754, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.38675, + "qps_1t": 1927.4951502629824, + "qps_1t_min": 1927.2314023746917, + "qps_1t_max": 1931.0118954294092, + "qps_1t_runs": [ + 1927.4951502629824, + 1931.0118954294092, + 1927.2314023746917 + ], + "median_ms": 0.5130000063218176, + "p95_ms": 0.596800004132092, + "p99_ms": 0.6743999547325075, + "mean_ms": 0.5185167668581319, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.5155, + "qps_1t": 1636.652640164911, + "qps_1t_min": 1631.503683179281, + "qps_1t_max": 1653.452960212079, + "qps_1t_runs": [ + 1636.652640164911, + 1653.452960212079, + 1631.503683179281 + ], + "median_ms": 0.6022999878041446, + "p95_ms": 0.7175999926403165, + "p99_ms": 0.7940000505186617, + "mean_ms": 0.6095765498757828, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.6369, + "qps_1t": 1303.9286126904303, + "qps_1t_min": 1294.4784402719067, + "qps_1t_max": 1306.6609854318106, + "qps_1t_runs": [ + 1303.9286126904303, + 1306.6609854318106, + 1294.4784402719067 + ], + "median_ms": 0.762699986808002, + "p95_ms": 0.8972999639809132, + "p99_ms": 0.9675000328570604, + "mean_ms": 0.7682448335302373, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.74205, + "qps_1t": 932.5690992060768, + "qps_1t_min": 929.0743700854791, + "qps_1t_max": 936.7351994916859, + "qps_1t_runs": [ + 936.7351994916859, + 929.0743700854791, + 932.5690992060768 + ], + "median_ms": 1.0694999946281314, + "p95_ms": 1.272300025448203, + "p99_ms": 1.3978999922983348, + "mean_ms": 1.0720614165572138, + "knob": 16 + }, + { + "nq": 1779, + "recall": 0.835300730747611, + "qps_1t": 607.3269644619176, + "qps_1t_min": 604.7278088714621, + "qps_1t_max": 608.8654713724035, + "qps_1t_runs": [ + 604.7278088714621, + 607.3269644619176, + 608.8654713724035 + ], + "median_ms": 1.6472999705001712, + "p95_ms": 1.9453000277280807, + "p99_ms": 2.113300026394427, + "mean_ms": 1.647531665914481, + "knob": 32 + }, + { + "nq": 1172, + "recall": 0.9045221843003413, + "qps_1t": 358.0915880647533, + "qps_1t_min": 357.00586855232217, + "qps_1t_max": 358.6797500413248, + "qps_1t_runs": [ + 358.0915880647533, + 358.6797500413248, + 357.00586855232217 + ], + "median_ms": 2.773099986370653, + "p95_ms": 3.2890000147745013, + "p99_ms": 3.549800021573901, + "mean_ms": 2.793886149220012, + "knob": 64 + }, + { + "nq": 607, + "recall": 0.9527182866556837, + "qps_1t": 193.73100507323974, + "qps_1t_min": 193.129416216683, + "qps_1t_max": 194.75259506946915, + "qps_1t_runs": [ + 193.73100507323974, + 194.75259506946915, + 193.129416216683 + ], + "median_ms": 5.108300014398992, + "p95_ms": 6.063799955882132, + "p99_ms": 6.7069000215269625, + "mean_ms": 5.1581304233197995, + "knob": 128 + }, + { + "nq": 319, + "recall": 0.9827586206896551, + "qps_1t": 103.04111809385445, + "qps_1t_min": 100.37806026264553, + "qps_1t_max": 103.14827949154343, + "qps_1t_runs": [ + 103.04111809385445, + 100.37806026264553, + 103.14827949154343 + ], + "median_ms": 9.70460003009066, + "p95_ms": 11.326299980282784, + "p99_ms": 12.252999993506819, + "mean_ms": 9.787327064312663, + "knob": 256 + }, + { + "nq": 59, + "recall": 1, + "qps_1t": 19.541129888221302, + "qps_1t_min": 19.409294888010816, + "qps_1t_max": 19.96571717127789, + "qps_1t_runs": [ + 19.96571717127789, + 19.541129888221302, + 19.409294888010816 + ], + "median_ms": 50.44710001675412, + "p95_ms": 54.28759998176247, + "p99_ms": 59.094499971251935, + "mean_ms": 50.927224859074386, + "knob": 1265 + } + ], + "ref_knob": 128, + "ref_recall_pre": 0.9527182866556837, + "mem_after_sweep": { + "rss_mb": 1557.93359375, + "peak_mb": 1557.93359375, + "heap_mb": 1251.347770690918 + }, + "mutation": { + "ref_knob": 128, + "recall_before": 0.9527182866556837, + "insert_10k": { + "seconds": 0.0631569999968633, + "per_s": 158335.5764285297 + }, + "update_10k": { + "seconds": 0.013717400026507676, + "per_s": 729001.121252998 + }, + "delete_10k": { + "seconds": 0.008221099968068302, + "per_s": 1216382.2406783947 + }, + "first_query_after_ms": 3283.551600005012, + "recall_after": 0.9515044247787611, + "after": { + "nq": 565, + "recall": 0.9515044247787611, + "qps_1t": 193.6929463448165, + "qps_1t_min": 186.21395780264191, + "qps_1t_max": 194.44508590031936, + "qps_1t_runs": [ + 193.6929463448165, + 194.44508590031936, + 186.21395780264191 + ], + "median_ms": 5.142899986822158, + "p95_ms": 6.2559000216424465, + "p99_ms": 6.848499993793666, + "mean_ms": 5.22527250679001 + }, + "mem_after": { + "rss_mb": 1745.265625, + "peak_mb": 1745.265625, + "heap_mb": 441.2213897705078 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 100000 + } + } + } + ], + "load_before_pct": 5.646875000000006, + "load_contaminated": false, + "wall_s": 153.53388023376465, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/js__text384__10000__core.json b/benchmarks/competitive/results/raw/js__text384__10000__core.json new file mode 100644 index 0000000..c09736f --- /dev/null +++ b/benchmarks/competitive/results/raw/js__text384__10000__core.json @@ -0,0 +1,289 @@ +{ + "status": "ok", + "label": "js", + "runs": [ + { + "system": "js_exact", + "ds": "text384", + "n": 10000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-js (repo HEAD, Node v24.18.0)", + "build": { + "ingest_s": 0.002464399964082986, + "index_s": 0, + "total_s": 0.002464399964082986, + "first_query_ms": 5.883199977688491, + "ready_s": 0.008347599941771477 + }, + "mem_baseline": { + "rss_mb": 66.22265625, + "peak_mb": 66.22265625, + "heap_mb": 16.078323364257812 + }, + "mem_after_ingest": { + "rss_mb": 159.578125, + "peak_mb": 159.578125, + "heap_mb": 54.90699005126953 + }, + "mem_after_build": { + "rss_mb": 161.3046875, + "peak_mb": 161.3046875, + "heap_mb": 58.83563995361328 + }, + "mem_after_first_query": { + "rss_mb": 161.3046875, + "peak_mb": 161.3046875, + "heap_mb": 58.83628845214844 + }, + "sweep": [ + { + "nq": 565, + "recall": 1, + "qps_1t": 197.694587196636, + "qps_1t_min": 192.32202157720317, + "qps_1t_max": 198.39146314010614, + "qps_1t_runs": [ + 197.694587196636, + 198.39146314010614, + 192.32202157720317 + ], + "median_ms": 4.995900031644851, + "p95_ms": 5.642500007525086, + "p99_ms": 6.035899976268411, + "mean_ms": 5.099486488887386, + "knob": null + } + ], + "mem_after_sweep": { + "rss_mb": 230.453125, + "peak_mb": 230.453125, + "heap_mb": 62.894752502441406 + } + }, + { + "system": "js_ivf", + "ds": "text384", + "n": 10000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "version": "isongraph-vector-js (repo HEAD, Node v24.18.0)", + "knob_name": "nprobe", + "build": { + "ingest_s": 0.0027119999867863953, + "index_s": 8.043748299998697, + "total_s": 8.046460299985483, + "first_query_ms": 0, + "ready_s": 8.046460299985483 + }, + "mem_baseline": { + "rss_mb": 66.22265625, + "peak_mb": 66.22265625, + "heap_mb": 16.078323364257812 + }, + "mem_after_ingest": { + "rss_mb": 231.33203125, + "peak_mb": 231.33203125, + "heap_mb": 48.58556365966797 + }, + "mem_after_build": { + "rss_mb": 265.80859375, + "peak_mb": 265.80859375, + "heap_mb": 84.01072692871094 + }, + "index_stats": { + "nlist": 400, + "trained_n": 10000, + "index_bytes": 700100, + "build_seconds": 8.0312465 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.2792, + "qps_1t": 6642.333585191237, + "qps_1t_min": 6415.460750179452, + "qps_1t_max": 6646.735318534214, + "qps_1t_runs": [ + 6415.460750179452, + 6646.735318534214, + 6642.333585191237 + ], + "median_ms": 0.14650000957772136, + "p95_ms": 0.17690000822767615, + "p99_ms": 0.30099996365606785, + "mean_ms": 0.15229091667182124, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.40335, + "qps_1t": 6277.737793039991, + "qps_1t_min": 6191.871864416425, + "qps_1t_max": 6289.440687618349, + "qps_1t_runs": [ + 6277.737793039991, + 6191.871864416425, + 6289.440687618349 + ], + "median_ms": 0.15750003512948751, + "p95_ms": 0.17559999832883477, + "p99_ms": 0.19120000069960952, + "mean_ms": 0.1599305833709271, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.54455, + "qps_1t": 5561.6455653699395, + "qps_1t_min": 5467.792918603301, + "qps_1t_max": 5578.946843235145, + "qps_1t_runs": [ + 5561.6455653699395, + 5467.792918603301, + 5578.946843235145 + ], + "median_ms": 0.17870002193376422, + "p95_ms": 0.20180002320557833, + "p99_ms": 0.21959998412057757, + "mean_ms": 0.18064578325720504, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.6832, + "qps_1t": 4575.091777079824, + "qps_1t_min": 4523.33543844947, + "qps_1t_max": 4588.684123994372, + "qps_1t_runs": [ + 4588.684123994372, + 4575.091777079824, + 4523.33543844947 + ], + "median_ms": 0.21769996965304017, + "p95_ms": 0.24750002194195986, + "p99_ms": 0.26700005400925875, + "mean_ms": 0.21919268331839703, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.8017, + "qps_1t": 3366.767581514926, + "qps_1t_min": 3339.9631596225513, + "qps_1t_max": 3380.7654704598326, + "qps_1t_runs": [ + 3339.9631596225513, + 3366.767581514926, + 3380.7654704598326 + ], + "median_ms": 0.2946000313386321, + "p95_ms": 0.3440000000409782, + "p99_ms": 0.37679995875805616, + "mean_ms": 0.2974054333753884, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.8925, + "qps_1t": 2256.327872437399, + "qps_1t_min": 2255.5736366301467, + "qps_1t_max": 2262.2937290449036, + "qps_1t_runs": [ + 2262.2937290449036, + 2256.327872437399, + 2255.5736366301467 + ], + "median_ms": 0.43839996214956045, + "p95_ms": 0.5108999903313816, + "p99_ms": 0.5484999855980277, + "mean_ms": 0.4428578164661303, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.9516, + "qps_1t": 1361.3603853694185, + "qps_1t_min": 1359.5598775289182, + "qps_1t_max": 1367.0311987711439, + "qps_1t_runs": [ + 1367.0311987711439, + 1359.5598775289182, + 1361.3603853694185 + ], + "median_ms": 0.7284999592229724, + "p95_ms": 0.830999983008951, + "p99_ms": 0.8974000229500234, + "mean_ms": 0.7338679001841228, + "knob": 64 + }, + { + "nq": 2000, + "recall": 0.9873, + "qps_1t": 765.1226492834795, + "qps_1t_min": 762.8747318258785, + "qps_1t_max": 767.6268424457456, + "qps_1t_runs": [ + 762.8747318258785, + 765.1226492834795, + 767.6268424457456 + ], + "median_ms": 1.3051999849267304, + "p95_ms": 1.4507999876514077, + "p99_ms": 1.5788000309839845, + "mean_ms": 1.3068424998685562, + "knob": 128 + }, + { + "nq": 1244, + "recall": 0.9986334405144695, + "qps_1t": 407.8588366045327, + "qps_1t_min": 407.7402604716675, + "qps_1t_max": 408.5740646426448, + "qps_1t_runs": [ + 407.8588366045327, + 408.5740646426448, + 407.7402604716675 + ], + "median_ms": 2.430399996228516, + "p95_ms": 2.711000037379563, + "p99_ms": 2.9160999692976475, + "mean_ms": 2.450635691370769, + "knob": 256 + }, + { + "nq": 606, + "recall": 1, + "qps_1t": 198.14701802026764, + "qps_1t_min": 196.64134634792745, + "qps_1t_max": 201.90120298376974, + "qps_1t_runs": [ + 198.14701802026764, + 201.90120298376974, + 196.64134634792745 + ], + "median_ms": 4.968799999915063, + "p95_ms": 5.559799959883094, + "p99_ms": 5.925300007220358, + "mean_ms": 5.0283585810110205, + "knob": 400 + } + ], + "ref_knob": 64, + "ref_recall_pre": 0.9516, + "mem_after_sweep": { + "rss_mb": 248.32421875, + "peak_mb": 265.80859375, + "heap_mb": 72.15968322753906 + } + } + ], + "load_before_pct": 4.290624999999999, + "load_contaminated": false, + "wall_s": 57.13064169883728, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/lancedb_flat__glove__1000000__core+mutation.json b/benchmarks/competitive/results/raw/lancedb_flat__glove__1000000__core+mutation.json new file mode 100644 index 0000000..f2957a4 --- /dev/null +++ b/benchmarks/competitive/results/raw/lancedb_flat__glove__1000000__core+mutation.json @@ -0,0 +1,118 @@ +{ + "system": "lancedb_flat", + "ds": "glove", + "n": 1000000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation" + ], + "started": "2026-11-16 10:32:54", + "mem_baseline": { + "rss_mb": 35.12109375, + "peak_mb": 35.12109375 + }, + "version": "lancedb 0.41.0", + "build": { + "ingest_s": 0.8012462998158298, + "index_s": 1.800013706088066e-06, + "total_s": 0.8012480998295359, + "first_query_ms": 156.952099991031, + "ready_s": 0.9582001998205669 + }, + "mem_after_build": { + "rss_mb": 128.203125, + "peak_mb": 158.734375 + }, + "mem_after_first_query": { + "rss_mb": 141.90625, + "peak_mb": 286.6875 + }, + "sweep": [ + { + "nq": 30, + "recall_queries": 30, + "recall": 1.0, + "qps_1t_runs": [ + 2.8897626910166934 + ], + "qps_1t": 2.8897626910166934, + "qps_1t_min": 2.8897626910166934, + "qps_1t_max": 2.8897626910166934, + "median_ms": 345.8859999955166, + "p95_ms": 384.75180996756535, + "p99_ms": 392.7767049934482, + "mean_ms": 346.0491766706885, + "knob": null + } + ], + "knob_name": "nprobes", + "loop_allthreads": { + "nq": 30, + "threads": 32, + "qps": 9.45213711840502, + "median_ms": 105.48774999915622, + "p95_ms": 124.36967001412992, + "p99_ms": 137.11867800157052, + "mean_ms": 106.23351222214599 + }, + "mem_after_sweep": { + "rss_mb": 170.53515625, + "peak_mb": 305.52734375 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.0704729999997653, + "per_s": 141898.3156674656, + "batch": 1000 + }, + "update_10k": { + "seconds": 1.495726499997545, + "per_s": 6685.7142666232185, + "batch": 1000 + }, + "delete_10k": { + "seconds": 1.3979403999983333, + "per_s": 7153.380787916225, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 176.37890625, + "peak_mb": 305.52734375 + }, + "first_query_after_ms": 272.6897999527864, + "after": { + "nq": 30, + "recall_queries": 30, + "recall": 1.0, + "qps_1t_runs": [ + 2.267650244375897 + ], + "qps_1t": 2.267650244375897, + "qps_1t_min": 2.267650244375897, + "qps_1t_max": 2.267650244375897, + "median_ms": 440.3876999858767, + "p95_ms": 467.7804049686529, + "p99_ms": 483.53027496719733, + "mean_ms": 440.98511332607205 + }, + "recall_after": 1.0 + }, + "mem_end": { + "rss_mb": 194.81640625, + "peak_mb": 336.40625 + }, + "status": "ok", + "finished": "2026-11-16 10:33:46", + "load_before_pct": 12.109374999999995, + "load_contaminated": false, + "wall_s": 52.0783486366272, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/lancedb_flat__glove__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/lancedb_flat__glove__100000__core+mutation+saveload.json new file mode 100644 index 0000000..41eade6 --- /dev/null +++ b/benchmarks/competitive/results/raw/lancedb_flat__glove__100000__core+mutation+saveload.json @@ -0,0 +1,135 @@ +{ + "system": "lancedb_flat", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 08:26:39", + "mem_baseline": { + "rss_mb": 35.25, + "peak_mb": 35.25 + }, + "version": "lancedb 0.41.0", + "build": { + "ingest_s": 0.0828034000005573, + "index_s": 2.600019797682762e-06, + "total_s": 0.08280600002035499, + "first_query_ms": 21.85830002417788, + "ready_s": 0.10466430004453287, + "total_s_runs": [ + 0.08280600002035499, + 0.07991450000554323, + 0.07684410008369014 + ] + }, + "mem_after_build": { + "rss_mb": 126.44140625, + "peak_mb": 156.97265625 + }, + "mem_after_first_query": { + "rss_mb": 134.87890625, + "peak_mb": 179.64453125 + }, + "sweep": [ + { + "nq": 86, + "recall_queries": 86, + "recall": 1.0, + "qps_1t_runs": [ + 31.198842740640725, + 34.47822143451636, + 34.250152835093274 + ], + "qps_1t": 34.250152835093274, + "qps_1t_min": 31.198842740640725, + "qps_1t_max": 34.47822143451636, + "median_ms": 29.508699983125553, + "p95_ms": 34.16231001028791, + "p99_ms": 37.37298197462224, + "mean_ms": 30.084412403733204, + "knob": null + } + ], + "knob_name": "nprobes", + "loop_allthreads": { + "nq": 255, + "threads": 32, + "qps": 86.46535722215252, + "median_ms": 11.355900031048805, + "p95_ms": 13.478979992214589, + "p99_ms": 15.230768015608191, + "mean_ms": 11.593433857052373 + }, + "mem_after_sweep": { + "rss_mb": 167.51953125, + "peak_mb": 206.03515625 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.02863959997193888, + "per_s": 349166.88814780983, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.134306599968113, + "per_s": 74456.50476130133, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.1077742999768816, + "per_s": 92786.4992131248, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 200.8046875, + "peak_mb": 222.77734375 + }, + "first_query_after_ms": 20.78080002684146, + "after": { + "nq": 71, + "recall_queries": 71, + "recall": 1.0, + "qps_1t_runs": [ + 24.111326952514904, + 23.934592088038578, + 22.96083974063845 + ], + "qps_1t": 23.934592088038578, + "qps_1t_min": 22.96083974063845, + "qps_1t_max": 24.111326952514904, + "median_ms": 41.49259999394417, + "p95_ms": 46.80060000391676, + "p99_ms": 51.528455985244356, + "mean_ms": 42.269076524300175 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 6.00004568696022e-07, + "disk_mb": 46.77094554901123, + "load_s": 0.0015787999727763236, + "first_query_after_load_ms": 16.915499989409, + "time_to_ready_s": 0.018494299962185323 + }, + "mem_end": { + "rss_mb": 218.07421875, + "peak_mb": 256.359375 + }, + "status": "ok", + "finished": "2026-11-16 08:27:10", + "load_before_pct": 6.821875000000004, + "load_contaminated": false, + "wall_s": 31.756428241729736, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/lancedb_flat__glove__100000__filter.json b/benchmarks/competitive/results/raw/lancedb_flat__glove__100000__filter.json new file mode 100644 index 0000000..a165265 --- /dev/null +++ b/benchmarks/competitive/results/raw/lancedb_flat__glove__100000__filter.json @@ -0,0 +1,66 @@ +{ + "system": "lancedb_flat", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 08:55:16", + "filter": { + "build_with_metadata_s": 0.08135599992237985, + "mem": { + "rss_mb": 126.08984375, + "peak_mb": 156.6171875 + }, + "subsets": { + "f10": [ + { + "knob": null, + "nq": 86, + "recall": 1.0, + "qps_1t": 33.84476313064172, + "qps_1t_runs": [ + 31.98960858190996, + 35.077751262317385, + 33.84476313064172 + ], + "median_ms": 28.858350007794797, + "p95_ms": 37.26772001537029, + "p99_ms": 40.7329980126815, + "mean_ms": 29.77163953388059 + } + ], + "f01": [ + { + "knob": null, + "nq": 109, + "recall": 1.0, + "qps_1t": 37.5524200749648, + "qps_1t_runs": [ + 36.17342388999656, + 37.5524200749648, + 37.58643111291594 + ], + "median_ms": 26.721499976702034, + "p95_ms": 30.50053000333719, + "p99_ms": 33.50158200482839, + "mean_ms": 26.959797555604943 + } + ] + } + }, + "mem_end": { + "rss_mb": 154.9375, + "peak_mb": 196.2890625 + }, + "status": "ok", + "finished": "2026-11-16 08:55:35", + "load_before_pct": 10.987500000000008, + "load_contaminated": false, + "wall_s": 18.42353391647339, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/lancedb_flat__sift__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/lancedb_flat__sift__100000__core+mutation+saveload.json new file mode 100644 index 0000000..d90185a --- /dev/null +++ b/benchmarks/competitive/results/raw/lancedb_flat__sift__100000__core+mutation+saveload.json @@ -0,0 +1,135 @@ +{ + "system": "lancedb_flat", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 11:16:55", + "mem_baseline": { + "rss_mb": 35.3671875, + "peak_mb": 35.3671875 + }, + "version": "lancedb 0.41.0", + "build": { + "ingest_s": 0.10293530009221286, + "index_s": 1.9000144675374031e-06, + "total_s": 0.1029372001066804, + "first_query_ms": 33.70490000816062, + "ready_s": 0.13664210011484101, + "total_s_runs": [ + 0.1029372001066804, + 0.22343769995495677, + 0.247466099972371 + ] + }, + "mem_after_build": { + "rss_mb": 131.12109375, + "peak_mb": 160.4296875 + }, + "mem_after_first_query": { + "rss_mb": 141.91015625, + "peak_mb": 204.19140625 + }, + "sweep": [ + { + "nq": 43, + "recall_queries": 43, + "recall": 1.0, + "qps_1t_runs": [ + 13.65146235646176, + 15.077235594120301, + 15.965725523477303 + ], + "qps_1t": 15.077235594120301, + "qps_1t_min": 13.65146235646176, + "qps_1t_max": 15.965725523477303, + "median_ms": 66.86670001363382, + "p95_ms": 78.52365997387096, + "p99_ms": 81.96539199445397, + "mean_ms": 67.40385116043667, + "knob": null + } + ], + "knob_name": "nprobes", + "loop_allthreads": { + "nq": 56, + "threads": 32, + "qps": 21.055871491797273, + "median_ms": 45.960849965922534, + "p95_ms": 56.49838999379426, + "p99_ms": 64.64650301611984, + "mean_ms": 46.83740595161604 + }, + "mem_after_sweep": { + "rss_mb": 169.6875, + "peak_mb": 233.9453125 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.12003140000160784, + "per_s": 83311.53348095622, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.36558129999320954, + "per_s": 27353.696702171976, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.24850059999153018, + "per_s": 40241.35153130752, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 186.97265625, + "peak_mb": 233.9453125 + }, + "first_query_after_ms": 77.36070000100881, + "after": { + "nq": 35, + "recall_queries": 35, + "recall": 1.0, + "qps_1t_runs": [ + 11.753031862202603, + 11.860933737857877, + 12.405521322502707 + ], + "qps_1t": 11.860933737857877, + "qps_1t_min": 11.753031862202603, + "qps_1t_max": 12.405521322502707, + "median_ms": 82.52759999595582, + "p95_ms": 94.37460000626743, + "p99_ms": 99.6326959808357, + "mean_ms": 83.33469714104596 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 7.00005330145359e-07, + "disk_mb": 59.58809280395508, + "load_s": 0.00497519999044016, + "first_query_after_load_ms": 74.04520001728088, + "time_to_ready_s": 0.07902040000772104 + }, + "mem_end": { + "rss_mb": 216.6796875, + "peak_mb": 275.69921875 + }, + "status": "ok", + "finished": "2026-11-16 11:17:31", + "load_before_pct": 12.684374999999996, + "load_contaminated": false, + "wall_s": 36.34006118774414, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/lancedb_flat__text384__1000000__core+mutation.json b/benchmarks/competitive/results/raw/lancedb_flat__text384__1000000__core+mutation.json new file mode 100644 index 0000000..47181b8 --- /dev/null +++ b/benchmarks/competitive/results/raw/lancedb_flat__text384__1000000__core+mutation.json @@ -0,0 +1,118 @@ +{ + "system": "lancedb_flat", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation" + ], + "started": "2026-11-16 09:46:56", + "mem_baseline": { + "rss_mb": 37.4765625, + "peak_mb": 37.4765625 + }, + "version": "lancedb 0.41.0", + "build": { + "ingest_s": 2.0412177999387495, + "index_s": 2.400018274784088e-06, + "total_s": 2.0412201999570243, + "first_query_ms": 653.5677000065334, + "ready_s": 2.6947878999635577 + }, + "mem_after_build": { + "rss_mb": 175.48046875, + "peak_mb": 263.3125 + }, + "mem_after_first_query": { + "rss_mb": 189.359375, + "peak_mb": 601.17578125 + }, + "sweep": [ + { + "nq": 30, + "recall_queries": 30, + "recall": 1.0, + "qps_1t_runs": [ + 0.7249557861538458 + ], + "qps_1t": 0.7249557861538458, + "qps_1t_min": 0.7249557861538458, + "qps_1t_max": 0.7249557861538458, + "median_ms": 1372.2370000032242, + "p95_ms": 1437.713464992703, + "p99_ms": 1462.75608900527, + "mean_ms": 1379.3944666686002, + "knob": null + } + ], + "knob_name": "nprobes", + "loop_allthreads": { + "nq": 30, + "threads": 32, + "qps": 1.7299119097310964, + "median_ms": 575.7210999727249, + "p95_ms": 602.9495449736714, + "p99_ms": 608.2479439658346, + "mean_ms": 578.0641166609712 + }, + "mem_after_sweep": { + "rss_mb": 192.671875, + "peak_mb": 628.53125 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.055588999995961785, + "per_s": 179891.7052065416, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.6347060999833047, + "per_s": 15755.323606095859, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.3578336000209674, + "per_s": 27945.950294813134, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 205.18359375, + "peak_mb": 628.53125 + }, + "first_query_after_ms": 939.8463999968953, + "after": { + "nq": 30, + "recall_queries": 30, + "recall": 1.0, + "qps_1t_runs": [ + 0.5560314855568307 + ], + "qps_1t": 0.5560314855568307, + "qps_1t_min": 0.5560314855568307, + "qps_1t_max": 0.5560314855568307, + "median_ms": 1796.5057000110392, + "p95_ms": 1852.6564899861114, + "p99_ms": 1873.9484239957528, + "mean_ms": 1798.4593066677917 + }, + "recall_after": 1.0 + }, + "mem_end": { + "rss_mb": 208.078125, + "peak_mb": 628.53125 + }, + "status": "ok", + "finished": "2026-11-16 09:49:25", + "load_before_pct": 3.549999999999999, + "load_contaminated": false, + "wall_s": 148.90827870368958, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/lancedb_flat__text384__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/lancedb_flat__text384__100000__core+mutation+saveload.json new file mode 100644 index 0000000..314c1b2 --- /dev/null +++ b/benchmarks/competitive/results/raw/lancedb_flat__text384__100000__core+mutation+saveload.json @@ -0,0 +1,133 @@ +{ + "system": "lancedb_flat", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 08:05:18", + "mem_baseline": { + "rss_mb": 37.3515625, + "peak_mb": 37.3515625 + }, + "version": "lancedb 0.41.0", + "build": { + "ingest_s": 0.21901230001822114, + "index_s": 2.400018274784088e-06, + "total_s": 0.21901470003649592, + "first_query_ms": 64.86110005062073, + "ready_s": 0.28387580008711666, + "total_s_runs": [ + 0.21901470003649592, + 0.21189370000502095, + 0.21135139995021746 + ] + }, + "mem_after_build": { + "rss_mb": 173.78515625, + "peak_mb": 261.6875 + }, + "mem_after_first_query": { + "rss_mb": 186.32421875, + "peak_mb": 361.078125 + }, + "sweep": [ + { + "nq": 30, + "recall_queries": 30, + "recall": 1.0, + "qps_1t_runs": [ + 7.163113575546817, + 7.216794403598273, + 7.188175853432592 + ], + "qps_1t": 7.188175853432592, + "qps_1t_min": 7.163113575546817, + "qps_1t_max": 7.216794403598273, + "median_ms": 136.85725000686944, + "p95_ms": 150.51211501995567, + "p99_ms": 172.52110999659635, + "mean_ms": 139.09570889081806, + "knob": null + } + ], + "knob_name": "nprobes", + "loop_allthreads": { + "nq": 42, + "threads": 32, + "qps": 13.747629719819061, + "median_ms": 72.05929997144267, + "p95_ms": 80.92444996873382, + "p99_ms": 90.0290749996202, + "mean_ms": 72.53502539473601 + }, + "mem_after_sweep": { + "rss_mb": 194.8984375, + "peak_mb": 395.10546875 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.052557399962097406, + "per_s": 190268.1640874864, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.16496710001956671, + "per_s": 60618.14749009895, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.11116339999716729, + "per_s": 89957.66592470925, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 252.078125, + "peak_mb": 395.10546875 + }, + "first_query_after_ms": 103.32120000384748, + "after": { + "nq": 30, + "recall_queries": 30, + "recall": 1.0, + "qps_1t_runs": [ + 4.803872766892777 + ], + "qps_1t": 4.803872766892777, + "qps_1t_min": 4.803872766892777, + "qps_1t_max": 4.803872766892777, + "median_ms": 202.50729998224415, + "p95_ms": 225.15360498800874, + "p99_ms": 309.4573300110643, + "mean_ms": 208.165380001689 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 1.100008375942707e-06, + "disk_mb": 176.77569484710693, + "load_s": 0.002555200015194714, + "first_query_after_load_ms": 110.22700002649799, + "time_to_ready_s": 0.1127822000416927 + }, + "mem_end": { + "rss_mb": 224.91796875, + "peak_mb": 464.95703125 + }, + "status": "ok", + "finished": "2026-11-16 08:05:58", + "load_before_pct": 4.240624999999999, + "load_contaminated": false, + "wall_s": 40.120402097702026, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/lancedb_flat__text384__100000__filter.json b/benchmarks/competitive/results/raw/lancedb_flat__text384__100000__filter.json new file mode 100644 index 0000000..59fee22 --- /dev/null +++ b/benchmarks/competitive/results/raw/lancedb_flat__text384__100000__filter.json @@ -0,0 +1,66 @@ +{ + "system": "lancedb_flat", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 08:44:13", + "filter": { + "build_with_metadata_s": 0.2166323999990709, + "mem": { + "rss_mb": 173.4296875, + "peak_mb": 261.3203125 + }, + "subsets": { + "f10": [ + { + "knob": null, + "nq": 30, + "recall": 1.0, + "qps_1t": 7.091514744945411, + "qps_1t_runs": [ + 7.091514744945411, + 7.249012370459008, + 6.987596248668594 + ], + "median_ms": 140.09994998923503, + "p95_ms": 149.52533498581033, + "p99_ms": 157.88445995829534, + "mean_ms": 140.69138443964118 + } + ], + "f01": [ + { + "knob": null, + "nq": 30, + "recall": 1.0, + "qps_1t": 7.280470869889407, + "qps_1t_runs": [ + 7.294221087910168, + 7.2739716952257645, + 7.280470869889407 + ], + "median_ms": 134.84800001606345, + "p95_ms": 152.26042499416508, + "p99_ms": 163.86655697773676, + "mean_ms": 137.30835332987934 + } + ] + } + }, + "mem_end": { + "rss_mb": 191.03125, + "peak_mb": 391.84765625 + }, + "status": "ok", + "finished": "2026-11-16 08:44:43", + "load_before_pct": 4.387499999999998, + "load_contaminated": false, + "wall_s": 30.38770031929016, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/lancedb_flat__text384__10000__core.json b/benchmarks/competitive/results/raw/lancedb_flat__text384__10000__core.json new file mode 100644 index 0000000..55f96db --- /dev/null +++ b/benchmarks/competitive/results/raw/lancedb_flat__text384__10000__core.json @@ -0,0 +1,79 @@ +{ + "system": "lancedb_flat", + "ds": "text384", + "n": 10000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 08:57:49", + "mem_baseline": { + "rss_mb": 37.515625, + "peak_mb": 37.515625 + }, + "version": "lancedb 0.41.0", + "build": { + "ingest_s": 0.03669500001706183, + "index_s": 1.300009898841381e-06, + "total_s": 0.03669630002696067, + "first_query_ms": 16.3079000194557, + "ready_s": 0.05300420004641637 + }, + "mem_after_build": { + "rss_mb": 141.32421875, + "peak_mb": 183.375 + }, + "mem_after_first_query": { + "rss_mb": 149.83203125, + "peak_mb": 183.375 + }, + "sweep": [ + { + "nq": 181, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 60.40533849721971, + 60.93997452638205, + 61.141338538648114 + ], + "qps_1t": 60.93997452638205, + "qps_1t_min": 60.40533849721971, + "qps_1t_max": 61.141338538648114, + "median_ms": 16.04149997001514, + "p95_ms": 18.860060034785416, + "p99_ms": 23.160338011803127, + "mean_ms": 16.43998821524982, + "knob": null + } + ], + "knob_name": "nprobes", + "loop_allthreads": { + "nq": 219, + "threads": 32, + "qps": 74.59007392256773, + "median_ms": 13.29600001918152, + "p95_ms": 14.838460006285459, + "p99_ms": 16.455072010867294, + "mean_ms": 13.422500151028798 + }, + "mem_after_sweep": { + "rss_mb": 152.68359375, + "peak_mb": 183.375 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 152.68359375, + "peak_mb": 183.375 + }, + "status": "ok", + "finished": "2026-11-16 08:58:22", + "load_before_pct": 4.124999999999997, + "load_contaminated": false, + "wall_s": 33.28580975532532, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/lancedb_ivfflat__glove__1000000__core+mutation.json b/benchmarks/competitive/results/raw/lancedb_ivfflat__glove__1000000__core+mutation.json new file mode 100644 index 0000000..922bf8a --- /dev/null +++ b/benchmarks/competitive/results/raw/lancedb_ivfflat__glove__1000000__core+mutation.json @@ -0,0 +1,275 @@ +{ + "system": "lancedb_ivfflat", + "ds": "glove", + "n": 1000000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation" + ], + "started": "2026-11-16 10:25:05", + "mem_baseline": { + "rss_mb": 35.19921875, + "peak_mb": 35.19921875 + }, + "version": "lancedb 0.41.0", + "build": { + "ingest_s": 0.7647575001465157, + "index_s": 8.916056599991862, + "total_s": 9.680814100138377, + "first_query_ms": 281.7618999979459, + "ready_s": 9.962576000136323 + }, + "mem_after_build": { + "rss_mb": 185.77734375, + "peak_mb": 901.3671875 + }, + "mem_after_first_query": { + "rss_mb": 589.79296875, + "peak_mb": 901.3671875 + }, + "sweep": [ + { + "nq": 710, + "recall_queries": 1000, + "recall": 0.2826, + "qps_1t_runs": [ + 256.52459469079184, + 232.00775005017073, + 165.08332383338188 + ], + "qps_1t": 232.00775005017073, + "qps_1t_min": 165.08332383338188, + "qps_1t_max": 256.52459469079184, + "median_ms": 4.1416500171180815, + "p95_ms": 7.376519998069853, + "p99_ms": 9.202730014221748, + "mean_ms": 4.755336525243327, + "knob": 1 + }, + { + "nq": 499, + "recall_queries": 1000, + "recall": 0.3981, + "qps_1t_runs": [ + 158.2638400017644, + 212.1748929719422, + 233.77547819396054 + ], + "qps_1t": 212.1748929719422, + "qps_1t_min": 158.2638400017644, + "qps_1t_max": 233.77547819396054, + "median_ms": 4.477099981158972, + "p95_ms": 7.542659982573241, + "p99_ms": 10.214051986113187, + "mean_ms": 5.103088109267213, + "knob": 2 + }, + { + "nq": 787, + "recall_queries": 1000, + "recall": 0.5174, + "qps_1t_runs": [ + 260.2982694323632, + 261.4322200443843, + 220.0374488931418 + ], + "qps_1t": 260.2982694323632, + "qps_1t_min": 220.0374488931418, + "qps_1t_max": 261.4322200443843, + "median_ms": 3.812399983871728, + "p95_ms": 6.056600017473102, + "p99_ms": 8.138520014472324, + "mean_ms": 4.070503599819448, + "knob": 4 + }, + { + "nq": 638, + "recall_queries": 1000, + "recall": 0.6209, + "qps_1t_runs": [ + 158.831494116633, + 165.95902570030233, + 166.3314315144138 + ], + "qps_1t": 165.95902570030233, + "qps_1t_min": 158.831494116633, + "qps_1t_max": 166.3314315144138, + "median_ms": 6.228350015589967, + "p95_ms": 7.774985005380585, + "p99_ms": 8.855420989566479, + "mean_ms": 6.111219017616792, + "knob": 8 + }, + { + "nq": 481, + "recall_queries": 1000, + "recall": 0.7151000000000001, + "qps_1t_runs": [ + 167.83595288354198, + 166.7475963982963, + 177.41365562357663 + ], + "qps_1t": 167.83595288354198, + "qps_1t_min": 166.7475963982963, + "qps_1t_max": 177.41365562357663, + "median_ms": 5.806700035464019, + "p95_ms": 7.55847999243997, + "p99_ms": 8.25189002789557, + "mean_ms": 5.863943867466386, + "knob": 16 + }, + { + "nq": 557, + "recall_queries": 1000, + "recall": 0.7928, + "qps_1t_runs": [ + 171.24781734170855, + 160.0558109168945, + 166.48933210881273 + ], + "qps_1t": 166.48933210881273, + "qps_1t_min": 160.0558109168945, + "qps_1t_max": 171.24781734170855, + "median_ms": 5.927999969571829, + "p95_ms": 7.746899995254353, + "p99_ms": 8.367049985099582, + "mean_ms": 6.031233991857424, + "knob": 32 + }, + { + "nq": 384, + "recall_queries": 1000, + "recall": 0.8552000000000001, + "qps_1t_runs": [ + 145.92519288300105, + 149.77800676363572, + 151.59595006369514 + ], + "qps_1t": 149.77800676363572, + "qps_1t_min": 145.92519288300105, + "qps_1t_max": 151.59595006369514, + "median_ms": 6.700850004563108, + "p95_ms": 8.267495036125183, + "p99_ms": 9.446597989881411, + "mean_ms": 6.708618749952519, + "knob": 64 + }, + { + "nq": 338, + "recall_queries": 1000, + "recall": 0.9125999999999999, + "qps_1t_runs": [ + 124.44170591030655, + 133.33127156472432, + 134.13723604082497 + ], + "qps_1t": 133.33127156472432, + "qps_1t_min": 124.44170591030655, + "qps_1t_max": 134.13723604082497, + "median_ms": 7.718499982729554, + "p95_ms": 9.20445000519976, + "p99_ms": 11.316130001214333, + "mean_ms": 7.663686193307197, + "knob": 128 + }, + { + "nq": 323, + "recall_queries": 1000, + "recall": 0.9512999999999999, + "qps_1t_runs": [ + 108.56475778048762, + 106.81718873717033, + 111.20980363393076 + ], + "qps_1t": 108.56475778048762, + "qps_1t_min": 106.81718873717033, + "qps_1t_max": 111.20980363393076, + "median_ms": 9.323399979621172, + "p95_ms": 10.863739997148514, + "p99_ms": 11.396468023303887, + "mean_ms": 9.188298140861521, + "knob": 256 + }, + { + "nq": 59, + "recall_queries": 59, + "recall": 1.0, + "qps_1t_runs": [ + 20.22210243508479, + 21.213294755503018, + 20.885383071862652 + ], + "qps_1t": 20.885383071862652, + "qps_1t_min": 20.22210243508479, + "qps_1t_max": 21.213294755503018, + "median_ms": 48.00289997365326, + "p95_ms": 52.13980001863092, + "p99_ms": 54.887856019195205, + "mean_ms": 48.15715593009619, + "knob": 4000 + } + ], + "knob_name": "nprobes", + "mem_after_sweep": { + "rss_mb": 590.63671875, + "peak_mb": 901.3671875 + }, + "ref_knob": 256, + "ref_recall_pre": 0.9512999999999999, + "mutation": { + "ref_knob": 256, + "recall_before": 0.9512999999999999, + "insert_10k": { + "seconds": 0.10521310003241524, + "per_s": 95045.19871498023, + "batch": 1000 + }, + "update_10k": { + "seconds": 1.294876700034365, + "per_s": 7722.7430223546435, + "batch": 1000 + }, + "delete_10k": { + "seconds": 1.2662446000031196, + "per_s": 7897.36832834301, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 614.109375, + "peak_mb": 901.3671875 + }, + "first_query_after_ms": 41.94409999763593, + "after": { + "nq": 117, + "recall_queries": 117, + "recall": 0.9393162393162393, + "qps_1t_runs": [ + 43.460760270633635, + 41.13110394804053, + 40.475915119885435 + ], + "qps_1t": 41.13110394804053, + "qps_1t_min": 40.475915119885435, + "qps_1t_max": 43.460760270633635, + "median_ms": 23.94420001655817, + "p95_ms": 29.041250032605603, + "p99_ms": 30.291050003143027, + "mean_ms": 24.009270942769945 + }, + "recall_after": 0.9393162393162393 + }, + "mem_end": { + "rss_mb": 622.328125, + "peak_mb": 901.3671875 + }, + "status": "ok", + "finished": "2026-11-16 10:27:34", + "load_before_pct": 8.259375000000002, + "load_contaminated": false, + "wall_s": 148.9761447906494, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/lancedb_ivfflat__glove__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/lancedb_ivfflat__glove__100000__core+mutation+saveload.json new file mode 100644 index 0000000..a1d4963 --- /dev/null +++ b/benchmarks/competitive/results/raw/lancedb_ivfflat__glove__100000__core+mutation+saveload.json @@ -0,0 +1,288 @@ +{ + "system": "lancedb_ivfflat", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 08:22:32", + "mem_baseline": { + "rss_mb": 35.12109375, + "peak_mb": 35.12109375 + }, + "version": "lancedb 0.41.0", + "build": { + "ingest_s": 0.08521079999627545, + "index_s": 0.6434958000318147, + "total_s": 0.7287066000280902, + "first_query_ms": 68.63890000386164, + "ready_s": 0.7973455000319518, + "total_s_runs": [ + 0.7287066000280902, + 0.6973931000684388, + 0.6925771000678651 + ] + }, + "mem_after_build": { + "rss_mb": 166.86328125, + "peak_mb": 235.97265625 + }, + "mem_after_first_query": { + "rss_mb": 208.12109375, + "peak_mb": 235.97265625 + }, + "sweep": [ + { + "nq": 950, + "recall_queries": 1000, + "recall": 0.25559999999999994, + "qps_1t_runs": [ + 308.46440287274817, + 306.93527368700455, + 309.8845134375193 + ], + "qps_1t": 308.46440287274817, + "qps_1t_min": 306.93527368700455, + "qps_1t_max": 309.8845134375193, + "median_ms": 3.2040500082075596, + "p95_ms": 3.6748650163644925, + "p99_ms": 4.23184000479523, + "mean_ms": 3.242296561284252, + "knob": 1 + }, + { + "nq": 922, + "recall_queries": 1000, + "recall": 0.37179999999999996, + "qps_1t_runs": [ + 308.6557069689085, + 313.83323811540305, + 303.77133096482214 + ], + "qps_1t": 308.6557069689085, + "qps_1t_min": 303.77133096482214, + "qps_1t_max": 313.83323811540305, + "median_ms": 3.181600011885166, + "p95_ms": 3.741824970347807, + "p99_ms": 4.388720021233891, + "mean_ms": 3.2394037961866236, + "knob": 2 + }, + { + "nq": 987, + "recall_queries": 1000, + "recall": 0.49420000000000003, + "qps_1t_runs": [ + 308.8301099786555, + 308.8613543381432, + 306.5842270858264 + ], + "qps_1t": 308.8301099786555, + "qps_1t_min": 306.5842270858264, + "qps_1t_max": 308.8613543381432, + "median_ms": 3.202899999450892, + "p95_ms": 3.709100012201816, + "p99_ms": 4.47353997733444, + "mean_ms": 3.2458237754453094, + "knob": 4 + }, + { + "nq": 959, + "recall_queries": 1000, + "recall": 0.6105, + "qps_1t_runs": [ + 304.3762834280036, + 296.55246770715104, + 301.6791860492452 + ], + "qps_1t": 301.6791860492452, + "qps_1t_min": 296.55246770715104, + "qps_1t_max": 304.3762834280036, + "median_ms": 3.252300026360899, + "p95_ms": 3.8504400290548797, + "p99_ms": 4.720644003245979, + "mean_ms": 3.3240904069324677, + "knob": 8 + }, + { + "nq": 922, + "recall_queries": 1000, + "recall": 0.711, + "qps_1t_runs": [ + 304.32276962406553, + 310.17596160351616, + 304.7157035284071 + ], + "qps_1t": 304.7157035284071, + "qps_1t_min": 304.32276962406553, + "qps_1t_max": 310.17596160351616, + "median_ms": 3.2418500049971044, + "p95_ms": 3.6611749965231866, + "p99_ms": 4.371875012293456, + "mean_ms": 3.2639029285012233, + "knob": 16 + }, + { + "nq": 887, + "recall_queries": 1000, + "recall": 0.803, + "qps_1t_runs": [ + 297.3240102081176, + 297.86377947374024, + 299.8711096058121 + ], + "qps_1t": 297.86377947374024, + "qps_1t_min": 297.3240102081176, + "qps_1t_max": 299.8711096058121, + "median_ms": 3.3300999784842134, + "p95_ms": 3.740200016181916, + "p99_ms": 4.408160003367828, + "mean_ms": 3.351779856434885, + "knob": 32 + }, + { + "nq": 880, + "recall_queries": 1000, + "recall": 0.8755, + "qps_1t_runs": [ + 283.58769103631505, + 278.4807922801689, + 282.8018671293463 + ], + "qps_1t": 282.8018671293463, + "qps_1t_min": 278.4807922801689, + "qps_1t_max": 283.58769103631505, + "median_ms": 3.4990000422112644, + "p95_ms": 3.8997000083327293, + "p99_ms": 4.581024025683294, + "mean_ms": 3.551067538677616, + "knob": 64 + }, + { + "nq": 786, + "recall_queries": 1000, + "recall": 0.9348, + "qps_1t_runs": [ + 256.25471802676833, + 247.43556332202778, + 253.02816926525713 + ], + "qps_1t": 253.02816926525713, + "qps_1t_min": 247.43556332202778, + "qps_1t_max": 256.25471802676833, + "median_ms": 3.837800002656877, + "p95_ms": 4.676574986660853, + "p99_ms": 5.990969007834784, + "mean_ms": 3.9653175150704247, + "knob": 128 + }, + { + "nq": 670, + "recall_queries": 1000, + "recall": 0.974, + "qps_1t_runs": [ + 207.08804896050407, + 217.03634918144752, + 219.11147158043107 + ], + "qps_1t": 217.03634918144752, + "qps_1t_min": 207.08804896050407, + "qps_1t_max": 219.11147158043107, + "median_ms": 4.454350011656061, + "p95_ms": 5.512090030242689, + "p99_ms": 7.193224999355153, + "mean_ms": 4.666758010677057, + "knob": 256 + }, + { + "nq": 352, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 112.78721528209984, + 113.7208833336949, + 112.3670039183191 + ], + "qps_1t": 112.78721528209984, + "qps_1t_min": 112.3670039183191, + "qps_1t_max": 113.7208833336949, + "median_ms": 8.53935000486672, + "p95_ms": 10.213950008619577, + "p99_ms": 16.33733499620577, + "mean_ms": 8.853040814453076, + "knob": 1265 + } + ], + "knob_name": "nprobes", + "mem_after_sweep": { + "rss_mb": 199.1015625, + "peak_mb": 235.97265625 + }, + "ref_knob": 256, + "ref_recall_pre": 0.974, + "mutation": { + "ref_knob": 256, + "recall_before": 0.974, + "insert_10k": { + "seconds": 0.03130010003224015, + "per_s": 319487.7968345042, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.13761199999134988, + "per_s": 72668.08127654993, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.10307250003097579, + "per_s": 97019.08847650689, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 253.96875, + "peak_mb": 323.70703125 + }, + "first_query_after_ms": 9.27480001701042, + "after": { + "nq": 181, + "recall_queries": 1000, + "recall": 0.9773999999999998, + "qps_1t_runs": [ + 64.06556688330947, + 62.784884231407126, + 66.1675875753147 + ], + "qps_1t": 64.06556688330947, + "qps_1t_min": 62.784884231407126, + "qps_1t_max": 66.1675875753147, + "median_ms": 14.918000029865652, + "p95_ms": 18.37568997289054, + "p99_ms": 27.89207602036186, + "mean_ms": 15.54984972652727 + }, + "recall_after": 0.9773999999999998 + }, + "saveload": { + "save_s": 1.200009137392044e-06, + "disk_mb": 86.09145164489746, + "load_s": 0.0016540000215172768, + "first_query_after_load_ms": 26.843000028748065, + "time_to_ready_s": 0.028497000050265342 + }, + "mem_end": { + "rss_mb": 263.1328125, + "peak_mb": 323.70703125 + }, + "status": "ok", + "finished": "2026-11-16 08:24:42", + "load_before_pct": 3.7687499999999967, + "load_contaminated": false, + "wall_s": 130.37158751487732, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/lancedb_ivfflat__glove__100000__filter.json b/benchmarks/competitive/results/raw/lancedb_ivfflat__glove__100000__filter.json new file mode 100644 index 0000000..ecb3fd4 --- /dev/null +++ b/benchmarks/competitive/results/raw/lancedb_ivfflat__glove__100000__filter.json @@ -0,0 +1,186 @@ +{ + "system": "lancedb_ivfflat", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 08:52:33", + "filter": { + "build_with_metadata_s": 0.7403945000260137, + "mem": { + "rss_mb": 166.5546875, + "peak_mb": 237.5625 + }, + "subsets": { + "f10": [ + { + "knob": 2, + "nq": 451, + "recall": 0.21263858093126387, + "qps_1t": 157.1920084239539, + "qps_1t_runs": [ + 145.87104093734274, + 157.1920084239539, + 160.48983061336196 + ], + "median_ms": 6.462000019382685, + "p95_ms": 7.775840000249441, + "p99_ms": 13.47486401209608, + "mean_ms": 6.4826470804304375 + }, + { + "knob": 8, + "nq": 473, + "recall": 0.42959830866807613, + "qps_1t": 156.44470831367647, + "qps_1t_runs": [ + 154.21222544456882, + 159.12505303808453, + 156.44470831367647 + ], + "median_ms": 6.462799967266619, + "p95_ms": 7.3917799978517, + "p99_ms": 9.57161603029816, + "mean_ms": 6.386990134618714 + }, + { + "knob": 32, + "nq": 455, + "recall": 0.6760439560439561, + "qps_1t": 160.103913433019, + "qps_1t_runs": [ + 153.81023270379276, + 160.103913433019, + 161.71620772535354 + ], + "median_ms": 6.4830000046640635, + "p95_ms": 7.28233999107033, + "p99_ms": 8.925828018691382, + "mean_ms": 6.310377875812038 + }, + { + "knob": 64, + "nq": 428, + "recall": 0.789018691588785, + "qps_1t": 150.59075202851295, + "qps_1t_runs": [ + 150.59075202851295, + 148.46389470852142, + 152.68430962905364 + ], + "median_ms": 6.660499988356605, + "p95_ms": 8.035800003563054, + "p99_ms": 11.029024035087787, + "mean_ms": 6.641873287623081 + }, + { + "knob": 1265, + "nq": 229, + "recall": 1.0, + "qps_1t": 88.26139619078172, + "qps_1t_runs": [ + 88.26139619078172, + 89.62518434181486, + 87.52938636075316 + ], + "median_ms": 11.270400020293891, + "p95_ms": 12.537630024598911, + "p99_ms": 13.556226032087578, + "mean_ms": 11.304098107967949 + } + ], + "f01": [ + { + "knob": 2, + "nq": 568, + "recall": 0.06654929577464788, + "qps_1t": 188.47488048163868, + "qps_1t_runs": [ + 188.47488048163868, + 189.30004805282664, + 174.68660135026875 + ], + "median_ms": 5.444300011731684, + "p95_ms": 6.220150031731464, + "p99_ms": 7.358800988295116, + "mean_ms": 5.437634389828661 + }, + { + "knob": 8, + "nq": 485, + "recall": 0.18639175257731957, + "qps_1t": 169.46758145277738, + "qps_1t_runs": [ + 169.46758145277738, + 169.18342417652553, + 172.9895537865669 + ], + "median_ms": 6.002800015266985, + "p95_ms": 6.690100039122624, + "p99_ms": 7.8513219871092605, + "mean_ms": 5.864091339413896 + }, + { + "knob": 32, + "nq": 475, + "recall": 0.43115789473684213, + "qps_1t": 161.83975345934857, + "qps_1t_runs": [ + 161.83975345934857, + 163.79662226719077, + 158.62341327720793 + ], + "median_ms": 6.294700026046485, + "p95_ms": 7.222119998186827, + "p99_ms": 8.880116001237182, + "mean_ms": 6.196107718472679 + }, + { + "knob": 64, + "nq": 428, + "recall": 0.572429906542056, + "qps_1t": 154.17383698209585, + "qps_1t_runs": [ + 149.86519133864815, + 154.17383698209585, + 157.73335111207084 + ], + "median_ms": 6.517199974041432, + "p95_ms": 7.923950001713818, + "p99_ms": 9.571372971986433, + "mean_ms": 6.4995538942806705 + }, + { + "knob": 1265, + "nq": 318, + "recall": 1.0, + "qps_1t": 98.98701729962468, + "qps_1t_runs": [ + 98.98701729962468, + 97.31415978439246, + 100.58018000632342 + ], + "median_ms": 9.800450003240258, + "p95_ms": 11.970265029231085, + "p99_ms": 15.246728970087144, + "mean_ms": 10.106882809190786 + } + ] + } + }, + "mem_end": { + "rss_mb": 210.7734375, + "peak_mb": 237.5625 + }, + "status": "ok", + "finished": "2026-11-16 08:54:03", + "load_before_pct": 3.506249999999997, + "load_contaminated": false, + "wall_s": 90.19842267036438, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/lancedb_ivfflat__sift__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/lancedb_ivfflat__sift__100000__core+mutation+saveload.json new file mode 100644 index 0000000..9c714c8 --- /dev/null +++ b/benchmarks/competitive/results/raw/lancedb_ivfflat__sift__100000__core+mutation+saveload.json @@ -0,0 +1,288 @@ +{ + "system": "lancedb_ivfflat", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 11:11:43", + "mem_baseline": { + "rss_mb": 35.49609375, + "peak_mb": 35.49609375 + }, + "version": "lancedb 0.41.0", + "build": { + "ingest_s": 0.11340080003719777, + "index_s": 0.8575120000168681, + "total_s": 0.9709128000540659, + "first_query_ms": 91.44479996757582, + "ready_s": 1.0623576000216417, + "total_s_runs": [ + 0.9709128000540659, + 0.8295959000242874, + 0.8694884999422356 + ] + }, + "mem_after_build": { + "rss_mb": 173.86328125, + "peak_mb": 265.78515625 + }, + "mem_after_first_query": { + "rss_mb": 224.69921875, + "peak_mb": 265.78515625 + }, + "sweep": [ + { + "nq": 835, + "recall_queries": 1000, + "recall": 0.2785, + "qps_1t_runs": [ + 270.3759264629438, + 279.07354817999885, + 277.26289523624047 + ], + "qps_1t": 277.26289523624047, + "qps_1t_min": 270.3759264629438, + "qps_1t_max": 279.07354817999885, + "median_ms": 3.546499996446073, + "p95_ms": 4.483719985000789, + "p99_ms": 5.873563967179508, + "mean_ms": 3.6295080632055012, + "knob": 1 + }, + { + "nq": 780, + "recall_queries": 1000, + "recall": 0.4209, + "qps_1t_runs": [ + 286.7034863200027, + 288.6847032611311, + 268.40922516003894 + ], + "qps_1t": 286.7034863200027, + "qps_1t_min": 268.40922516003894, + "qps_1t_max": 288.6847032611311, + "median_ms": 3.4685999853536487, + "p95_ms": 4.268470022361726, + "p99_ms": 5.938413013354875, + "mean_ms": 3.559188418932116, + "knob": 2 + }, + { + "nq": 832, + "recall_queries": 1000, + "recall": 0.5785, + "qps_1t_runs": [ + 270.26455484794405, + 275.54052895406556, + 283.07718645067973 + ], + "qps_1t": 275.54052895406556, + "qps_1t_min": 270.26455484794405, + "qps_1t_max": 283.07718645067973, + "median_ms": 3.530949994456023, + "p95_ms": 4.427474981639534, + "p99_ms": 5.89647502347364, + "mean_ms": 3.620637940679071, + "knob": 4 + }, + { + "nq": 825, + "recall_queries": 1000, + "recall": 0.7387999999999999, + "qps_1t_runs": [ + 280.6730955796045, + 264.3531079845733, + 268.7196631886525 + ], + "qps_1t": 268.7196631886525, + "qps_1t_min": 264.3531079845733, + "qps_1t_max": 280.6730955796045, + "median_ms": 3.587900020647794, + "p95_ms": 4.5391900290269405, + "p99_ms": 6.010548011399791, + "mean_ms": 3.689011071145422, + "knob": 8 + }, + { + "nq": 800, + "recall_queries": 1000, + "recall": 0.8673, + "qps_1t_runs": [ + 270.4215785601032, + 263.2281178825422, + 263.3481377813385 + ], + "qps_1t": 263.3481377813385, + "qps_1t_min": 263.2281178825422, + "qps_1t_max": 270.4215785601032, + "median_ms": 3.683699993416667, + "p95_ms": 4.553804980241692, + "p99_ms": 6.2035610264865575, + "mean_ms": 3.7647236239232975, + "knob": 16 + }, + { + "nq": 766, + "recall_queries": 1000, + "recall": 0.9486, + "qps_1t_runs": [ + 251.3068283345626, + 246.7051894232069, + 247.05439241022614 + ], + "qps_1t": 247.05439241022614, + "qps_1t_min": 246.7051894232069, + "qps_1t_max": 251.3068283345626, + "median_ms": 3.8989500026218593, + "p95_ms": 4.889725011889825, + "p99_ms": 7.055723987868997, + "mean_ms": 4.02677071339441, + "knob": 32 + }, + { + "nq": 731, + "recall_queries": 1000, + "recall": 0.9868000000000001, + "qps_1t_runs": [ + 243.86317977526355, + 238.109422861669, + 234.43039803763216 + ], + "qps_1t": 238.109422861669, + "qps_1t_min": 234.43039803763216, + "qps_1t_max": 243.86317977526355, + "median_ms": 4.064999986439943, + "p95_ms": 5.078340030740948, + "p99_ms": 6.776079996488985, + "mean_ms": 4.188689421444054, + "knob": 64 + }, + { + "nq": 624, + "recall_queries": 1000, + "recall": 0.9976999999999999, + "qps_1t_runs": [ + 213.42368631321537, + 215.44596330083124, + 202.54623994690402 + ], + "qps_1t": 213.42368631321537, + "qps_1t_min": 202.54623994690402, + "qps_1t_max": 215.44596330083124, + "median_ms": 4.58559999242425, + "p95_ms": 5.782204965362324, + "p99_ms": 8.624416001257483, + "mean_ms": 4.754731624136464, + "knob": 128 + }, + { + "nq": 550, + "recall_queries": 1000, + "recall": 0.9999, + "qps_1t_runs": [ + 186.37544032037258, + 188.95889145778986, + 185.44762167997942 + ], + "qps_1t": 186.37544032037258, + "qps_1t_min": 185.44762167997942, + "qps_1t_max": 188.95889145778986, + "median_ms": 5.207700014580041, + "p95_ms": 6.414425006369129, + "p99_ms": 8.255622963770293, + "mean_ms": 5.350009453885088, + "knob": 256 + }, + { + "nq": 300, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 92.33298018754584, + 92.32238435926035, + 90.62980007498007 + ], + "qps_1t": 92.32238435926035, + "qps_1t_min": 90.62980007498007, + "qps_1t_max": 92.33298018754584, + "median_ms": 10.567049990640953, + "p95_ms": 12.906670029042282, + "p99_ms": 17.296480967779644, + "mean_ms": 10.89862466602224, + "knob": 1265 + } + ], + "knob_name": "nprobes", + "mem_after_sweep": { + "rss_mb": 211.484375, + "peak_mb": 265.78515625 + }, + "ref_knob": 64, + "ref_recall_pre": 0.9868000000000001, + "mutation": { + "ref_knob": 64, + "recall_before": 0.9868000000000001, + "insert_10k": { + "seconds": 0.039654400025028735, + "per_s": 252178.8248892502, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.14905529998941347, + "per_s": 67089.1944178452, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.12040869996417314, + "per_s": 83050.47727427867, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 272.796875, + "peak_mb": 371.64453125 + }, + "first_query_after_ms": 12.059500033501536, + "after": { + "nq": 168, + "recall_queries": 1000, + "recall": 0.9886000000000001, + "qps_1t_runs": [ + 57.838612370683805, + 60.023188248452946, + 58.993988151833705 + ], + "qps_1t": 58.993988151833705, + "qps_1t_min": 57.838612370683805, + "qps_1t_max": 60.023188248452946, + "median_ms": 16.647149983327836, + "p95_ms": 19.76132999407127, + "p99_ms": 26.17706401797472, + "mean_ms": 16.966865278109687 + }, + "recall_after": 0.9886000000000001 + }, + "saveload": { + "save_s": 1.400010660290718e-06, + "disk_mb": 109.67045783996582, + "load_s": 0.00807640003040433, + "first_query_after_load_ms": 18.645100004505366, + "time_to_ready_s": 0.026721500034909695 + }, + "mem_end": { + "rss_mb": 280.14453125, + "peak_mb": 371.64453125 + }, + "status": "ok", + "finished": "2026-11-16 11:14:01", + "load_before_pct": 8.309375000000001, + "load_contaminated": false, + "wall_s": 138.38206505775452, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/lancedb_ivfflat__text384__1000000__core+mutation.json b/benchmarks/competitive/results/raw/lancedb_ivfflat__text384__1000000__core+mutation.json new file mode 100644 index 0000000..1722389 --- /dev/null +++ b/benchmarks/competitive/results/raw/lancedb_ivfflat__text384__1000000__core+mutation.json @@ -0,0 +1,275 @@ +{ + "system": "lancedb_ivfflat", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation" + ], + "started": "2026-11-16 09:40:36", + "mem_baseline": { + "rss_mb": 37.3125, + "peak_mb": 37.3125 + }, + "version": "lancedb 0.41.0", + "build": { + "ingest_s": 2.0603686998365447, + "index_s": 10.740709300036542, + "total_s": 12.801077999873087, + "first_query_ms": 586.2553999759257, + "ready_s": 13.387333399849012 + }, + "mem_after_build": { + "rss_mb": 278.03125, + "peak_mb": 3117.9375 + }, + "mem_after_first_query": { + "rss_mb": 1773.515625, + "peak_mb": 3117.9375 + }, + "sweep": [ + { + "nq": 688, + "recall_queries": 1000, + "recall": 0.21470000000000003, + "qps_1t_runs": [ + 225.31000264479792, + 228.49204847797822, + 231.2099721997163 + ], + "qps_1t": 228.49204847797822, + "qps_1t_min": 225.31000264479792, + "qps_1t_max": 231.2099721997163, + "median_ms": 4.262500006007031, + "p95_ms": 5.053334953845479, + "p99_ms": 5.759006004664102, + "mean_ms": 4.379974079386931, + "knob": 1 + }, + { + "nq": 703, + "recall_queries": 1000, + "recall": 0.31039999999999995, + "qps_1t_runs": [ + 228.53095520185963, + 229.49241275427826, + 222.8880822844825 + ], + "qps_1t": 228.53095520185963, + "qps_1t_min": 222.8880822844825, + "qps_1t_max": 229.49241275427826, + "median_ms": 4.291099961847067, + "p95_ms": 5.032539984676987, + "p99_ms": 6.111732022836811, + "mean_ms": 4.406591323140307, + "knob": 2 + }, + { + "nq": 728, + "recall_queries": 1000, + "recall": 0.41129999999999994, + "qps_1t_runs": [ + 234.35203613647982, + 239.50827115438165, + 242.6207628877579 + ], + "qps_1t": 239.50827115438165, + "qps_1t_min": 234.35203613647982, + "qps_1t_max": 242.6207628877579, + "median_ms": 4.12694999249652, + "p95_ms": 4.658775025745854, + "p99_ms": 5.246869000839074, + "mean_ms": 4.1879882316204, + "knob": 4 + }, + { + "nq": 630, + "recall_queries": 1000, + "recall": 0.5227, + "qps_1t_runs": [ + 222.50830435735676, + 225.0725536951612, + 234.27616476960497 + ], + "qps_1t": 225.0725536951612, + "qps_1t_min": 222.50830435735676, + "qps_1t_max": 234.27616476960497, + "median_ms": 4.310999996960163, + "p95_ms": 4.950819979421794, + "p99_ms": 5.880025012302205, + "mean_ms": 4.401897566107215, + "knob": 8 + }, + { + "nq": 689, + "recall_queries": 1000, + "recall": 0.6346, + "qps_1t_runs": [ + 220.4087638911061, + 211.2752155297306, + 210.9917968788648 + ], + "qps_1t": 211.2752155297306, + "qps_1t_min": 210.9917968788648, + "qps_1t_max": 220.4087638911061, + "median_ms": 4.570600052829832, + "p95_ms": 5.352519970620051, + "p99_ms": 6.3183380104601365, + "mean_ms": 4.669902756751389, + "knob": 16 + }, + { + "nq": 593, + "recall_queries": 1000, + "recall": 0.7313, + "qps_1t_runs": [ + 192.08556304150832, + 184.5083262103106, + 184.9112253169848 + ], + "qps_1t": 184.9112253169848, + "qps_1t_min": 184.5083262103106, + "qps_1t_max": 192.08556304150832, + "median_ms": 5.15889999223873, + "p95_ms": 6.382949993712827, + "p99_ms": 8.698084019124511, + "mean_ms": 5.344607813117325, + "knob": 32 + }, + { + "nq": 525, + "recall_queries": 1000, + "recall": 0.815, + "qps_1t_runs": [ + 164.5230823711992, + 162.30969726152478, + 158.68173142225018 + ], + "qps_1t": 162.30969726152478, + "qps_1t_min": 158.68173142225018, + "qps_1t_max": 164.5230823711992, + "median_ms": 5.95159997465089, + "p95_ms": 7.128359994385391, + "p99_ms": 9.413218001136554, + "mean_ms": 6.180386158992492, + "knob": 64 + }, + { + "nq": 389, + "recall_queries": 1000, + "recall": 0.8876000000000001, + "qps_1t_runs": [ + 134.58084127274407, + 133.98940608014908, + 127.46982619661769 + ], + "qps_1t": 133.98940608014908, + "qps_1t_min": 127.46982619661769, + "qps_1t_max": 134.58084127274407, + "median_ms": 7.287700020242482, + "p95_ms": 8.524770010262728, + "p99_ms": 10.185072009917329, + "mean_ms": 7.57958286110872, + "knob": 128 + }, + { + "nq": 299, + "recall_queries": 1000, + "recall": 0.9378, + "qps_1t_runs": [ + 96.74604859295353, + 94.9675308270553, + 95.42913893748785 + ], + "qps_1t": 95.42913893748785, + "qps_1t_min": 94.9675308270553, + "qps_1t_max": 96.74604859295353, + "median_ms": 10.187000036239624, + "p95_ms": 11.666379997041075, + "p99_ms": 13.311188004445283, + "mean_ms": 10.44841125780621, + "knob": 256 + }, + { + "nq": 33, + "recall_queries": 33, + "recall": 1.0, + "qps_1t_runs": [ + 11.258384809990291, + 11.495695541636517, + 11.43679443504118 + ], + "qps_1t": 11.43679443504118, + "qps_1t_min": 11.258384809990291, + "qps_1t_max": 11.495695541636517, + "median_ms": 87.44259999366477, + "p95_ms": 90.38233000319451, + "p99_ms": 92.83751600421965, + "mean_ms": 87.74961918919357, + "knob": 4000 + } + ], + "knob_name": "nprobes", + "mem_after_sweep": { + "rss_mb": 1717.53515625, + "peak_mb": 3117.9375 + }, + "ref_knob": 4000, + "ref_recall_pre": 1.0, + "mutation": { + "ref_knob": 4000, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.05524190003052354, + "per_s": 181022.01398711064, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.6025168999913149, + "per_s": 16597.04483001912, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.35617240000283346, + "per_s": 28076.291144177503, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 1792.80859375, + "peak_mb": 3117.9375 + }, + "first_query_after_ms": 108.42829995090142, + "after": { + "nq": 30, + "recall_queries": 30, + "recall": 1.0, + "qps_1t_runs": [ + 8.02267657690412, + 7.867899850251763, + 7.969609011448342 + ], + "qps_1t": 7.969609011448342, + "qps_1t_min": 7.867899850251763, + "qps_1t_max": 8.02267657690412, + "median_ms": 123.97595003130846, + "p95_ms": 134.71618500188924, + "p99_ms": 143.3858879870968, + "mean_ms": 125.74068999836325 + }, + "recall_after": 1.0 + }, + "mem_end": { + "rss_mb": 1793.51171875, + "peak_mb": 3117.9375 + }, + "status": "ok", + "finished": "2026-11-16 09:43:09", + "load_before_pct": 8.15625, + "load_contaminated": false, + "wall_s": 153.78905963897705, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/lancedb_ivfflat__text384__1000000__filter.json b/benchmarks/competitive/results/raw/lancedb_ivfflat__text384__1000000__filter.json new file mode 100644 index 0000000..bb644d0 --- /dev/null +++ b/benchmarks/competitive/results/raw/lancedb_ivfflat__text384__1000000__filter.json @@ -0,0 +1,184 @@ +{ + "system": "lancedb_ivfflat", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 11:00:13", + "filter": { + "build_with_metadata_s": 25.306510100199375, + "mem": { + "rss_mb": 214.53515625, + "peak_mb": 3118.65625 + }, + "subsets": { + "f10": [ + { + "knob": 2, + "nq": 82, + "recall": 0.22073170731707317, + "qps_1t": 27.01546220061804, + "qps_1t_runs": [ + 25.754254973711564, + 27.385659116378317, + 27.01546220061804 + ], + "median_ms": 37.50959996250458, + "p95_ms": 43.866074993275106, + "p99_ms": 47.37958000041547, + "mean_ms": 37.45327885920462 + }, + { + "knob": 8, + "nq": 80, + "recall": 0.45875, + "qps_1t": 26.275066574724107, + "qps_1t_runs": [ + 25.07087614678705, + 26.275066574724107, + 26.49751448280967 + ], + "median_ms": 38.82885002531111, + "p95_ms": 45.00488999765365, + "p99_ms": 49.018052007886574, + "mean_ms": 38.56173417031338 + }, + { + "knob": 32, + "nq": 72, + "recall": 0.6847222222222222, + "qps_1t": 14.408384814559135, + "qps_1t_runs": [ + 14.408384814559135 + ], + "median_ms": 46.742299979086965, + "p95_ms": 223.27411498408765, + "p99_ms": 236.85879700351518, + "mean_ms": 69.40403194878147 + }, + { + "knob": 64, + "nq": 80, + "recall": 0.785, + "qps_1t": 26.082502933587808, + "qps_1t_runs": [ + 22.607663417416827, + 26.45873015012876, + 26.082502933587808 + ], + "median_ms": 39.41040000063367, + "p95_ms": 50.009849999332786, + "p99_ms": 56.919133985065834, + "mean_ms": 40.12245875031416 + }, + { + "knob": 4000, + "nq": 30, + "recall": 1.0, + "qps_1t": 8.726117706820212, + "qps_1t_runs": [ + 8.726117706820212, + 9.161802843296455, + 8.506619794499755 + ], + "median_ms": 113.79905001376756, + "p95_ms": 125.15862001455389, + "p99_ms": 132.09589600854088, + "mean_ms": 113.76761000202451 + } + ], + "f01": [ + { + "knob": 2, + "nq": 101, + "recall": 0.12673267326732673, + "qps_1t": 30.090289338055868, + "qps_1t_runs": [ + 29.935552314106573, + 30.090289338055868, + 30.23800271517329 + ], + "median_ms": 33.27429998898879, + "p95_ms": 39.68294995720498, + "p99_ms": 40.48900998313911, + "mean_ms": 33.23645874774913 + }, + { + "knob": 8, + "nq": 88, + "recall": 0.3125, + "qps_1t": 30.311280312820497, + "qps_1t_runs": [ + 30.221107580814248, + 30.311280312820497, + 30.720493673177607 + ], + "median_ms": 32.935950002865866, + "p95_ms": 39.694530025008135, + "p99_ms": 42.30855902133044, + "mean_ms": 32.87734469830122 + }, + { + "knob": 32, + "nq": 94, + "recall": 0.5457446808510639, + "qps_1t": 28.900017373680924, + "qps_1t_runs": [ + 28.606340480035, + 28.900017373680924, + 29.398921532514578 + ], + "median_ms": 34.50405001058243, + "p95_ms": 40.28495498932898, + "p99_ms": 42.53876401053276, + "mean_ms": 34.52473120777934 + }, + { + "knob": 64, + "nq": 88, + "recall": 0.6875, + "qps_1t": 29.43461293085843, + "qps_1t_runs": [ + 28.7293776505382, + 29.658033438730197, + 29.43461293085843 + ], + "median_ms": 34.2530999914743, + "p95_ms": 41.02235503378324, + "p99_ms": 42.667963021667674, + "mean_ms": 34.16628712169988 + }, + { + "knob": 4000, + "nq": 40, + "recall": 1.0, + "qps_1t": 13.069751302577982, + "qps_1t_runs": [ + 13.069751302577982, + 13.096706407126604, + 12.932709046552187 + ], + "median_ms": 77.22004997776821, + "p95_ms": 85.2545350062428, + "p99_ms": 87.36987999291159, + "mean_ms": 76.7303149982278 + } + ] + } + }, + "mem_end": { + "rss_mb": 1702.63671875, + "peak_mb": 3118.65625 + }, + "status": "ok", + "finished": "2026-11-16 11:02:18", + "load_before_pct": 8.137499999999992, + "load_contaminated": false, + "wall_s": 126.40920782089233, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/lancedb_ivfflat__text384__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/lancedb_ivfflat__text384__100000__core+mutation+saveload.json new file mode 100644 index 0000000..fd8df64 --- /dev/null +++ b/benchmarks/competitive/results/raw/lancedb_ivfflat__text384__100000__core+mutation+saveload.json @@ -0,0 +1,287 @@ +{ + "system": "lancedb_ivfflat", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 08:00:07", + "mem_baseline": { + "rss_mb": 37.25390625, + "peak_mb": 37.25390625 + }, + "version": "lancedb 0.41.0", + "build": { + "ingest_s": 0.23378770000999793, + "index_s": 1.660392599995248, + "total_s": 1.894180300005246, + "first_query_ms": 89.4075000542216, + "ready_s": 1.9835878000594676, + "total_s_runs": [ + 1.894180300005246, + 1.7727591000148095, + 1.800324699957855 + ] + }, + "mem_after_build": { + "rss_mb": 236.921875, + "peak_mb": 524.640625 + }, + "mem_after_first_query": { + "rss_mb": 375.859375, + "peak_mb": 524.640625 + }, + "sweep": [ + { + "nq": 924, + "recall_queries": 1000, + "recall": 0.2415, + "qps_1t_runs": [ + 287.87555996423646, + 198.49865843185904 + ], + "qps_1t": 243.18710919804775, + "qps_1t_min": 198.49865843185904, + "qps_1t_max": 287.87555996423646, + "median_ms": 3.284099977463484, + "p95_ms": 4.56145500356797, + "p99_ms": 12.32441597850993, + "mean_ms": 4.255770292044647, + "knob": 1 + }, + { + "nq": 912, + "recall_queries": 1000, + "recall": 0.3584, + "qps_1t_runs": [ + 298.3341766201878, + 300.05830418776014, + 301.058636997805 + ], + "qps_1t": 300.05830418776014, + "qps_1t_min": 298.3341766201878, + "qps_1t_max": 301.058636997805, + "median_ms": 3.2394500158261508, + "p95_ms": 4.0596499893581495, + "p99_ms": 5.787509997026071, + "mean_ms": 3.335414509761606, + "knob": 2 + }, + { + "nq": 933, + "recall_queries": 1000, + "recall": 0.4898, + "qps_1t_runs": [ + 282.24981717587826, + 284.96744500341686, + 300.0064825058929 + ], + "qps_1t": 284.96744500341686, + "qps_1t_min": 282.24981717587826, + "qps_1t_max": 300.0064825058929, + "median_ms": 3.2953000045381486, + "p95_ms": 4.062800022074952, + "p99_ms": 6.324091986753043, + "mean_ms": 3.461798249337777, + "knob": 4 + }, + { + "nq": 843, + "recall_queries": 1000, + "recall": 0.616, + "qps_1t_runs": [ + 285.8630448873732, + 299.1199698851492, + 294.169723509095 + ], + "qps_1t": 294.169723509095, + "qps_1t_min": 285.8630448873732, + "qps_1t_max": 299.1199698851492, + "median_ms": 3.3267999533563852, + "p95_ms": 3.944119997322559, + "p99_ms": 5.098276003263885, + "mean_ms": 3.413572321596569, + "knob": 8 + }, + { + "nq": 903, + "recall_queries": 1000, + "recall": 0.7339, + "qps_1t_runs": [ + 277.7639783679517, + 287.77295722676075, + 285.5820983997752 + ], + "qps_1t": 285.5820983997752, + "qps_1t_min": 277.7639783679517, + "qps_1t_max": 287.77295722676075, + "median_ms": 3.4129000268876553, + "p95_ms": 4.028900002595037, + "p99_ms": 5.674276025965811, + "mean_ms": 3.525586859057508, + "knob": 16 + }, + { + "nq": 786, + "recall_queries": 1000, + "recall": 0.8351000000000001, + "qps_1t_runs": [ + 264.6997165339855, + 266.3056113975814, + 277.96179883554464 + ], + "qps_1t": 266.3056113975814, + "qps_1t_min": 264.6997165339855, + "qps_1t_max": 277.96179883554464, + "median_ms": 3.5851499997079372, + "p95_ms": 4.306825014646166, + "p99_ms": 5.363074974738984, + "mean_ms": 3.710188888498377, + "knob": 32 + }, + { + "nq": 790, + "recall_queries": 1000, + "recall": 0.9089999999999999, + "qps_1t_runs": [ + 255.77491176843697, + 256.77042674030986, + 251.2046295598156 + ], + "qps_1t": 255.77491176843697, + "qps_1t_min": 251.2046295598156, + "qps_1t_max": 256.77042674030986, + "median_ms": 3.863599995383993, + "p95_ms": 4.430014971876517, + "p99_ms": 5.061538993613793, + "mean_ms": 3.928345148004631, + "knob": 64 + }, + { + "nq": 652, + "recall_queries": 1000, + "recall": 0.9556, + "qps_1t_runs": [ + 218.04052600866663, + 220.33545525956112, + 212.13579422114074 + ], + "qps_1t": 218.04052600866663, + "qps_1t_min": 212.13579422114074, + "qps_1t_max": 220.33545525956112, + "median_ms": 4.518050001934171, + "p95_ms": 5.162800007383339, + "p99_ms": 6.057295014034028, + "mean_ms": 4.612933078046092, + "knob": 128 + }, + { + "nq": 500, + "recall_queries": 1000, + "recall": 0.9828, + "qps_1t_runs": [ + 177.89543227601519, + 177.232167352002, + 168.22906608561948 + ], + "qps_1t": 177.232167352002, + "qps_1t_min": 168.22906608561948, + "qps_1t_max": 177.89543227601519, + "median_ms": 5.611349974060431, + "p95_ms": 6.478485025581904, + "p99_ms": 7.699603011715226, + "mean_ms": 5.7359574668419855, + "knob": 256 + }, + { + "nq": 210, + "recall_queries": 1000, + "recall": 0.9999, + "qps_1t_runs": [ + 68.98402397160703, + 70.27806452146946, + 71.13250053012611 + ], + "qps_1t": 70.27806452146946, + "qps_1t_min": 68.98402397160703, + "qps_1t_max": 71.13250053012611, + "median_ms": 14.031900005647913, + "p95_ms": 15.610205003758892, + "p99_ms": 17.41453402442858, + "mean_ms": 14.2611907964151, + "knob": 1265 + } + ], + "knob_name": "nprobes", + "mem_after_sweep": { + "rss_mb": 319.98828125, + "peak_mb": 524.640625 + }, + "ref_knob": 128, + "ref_recall_pre": 0.9556, + "mutation": { + "ref_knob": 128, + "recall_before": 0.9556, + "insert_10k": { + "seconds": 0.055947599990759045, + "per_s": 178738.67693434065, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.2072751999949105, + "per_s": 48245.03848142731, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.13450540002668276, + "per_s": 74346.4574508996, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 412.0625, + "peak_mb": 711.171875 + }, + "first_query_after_ms": 15.473199950065464, + "after": { + "nq": 97, + "recall_queries": 97, + "recall": 0.9690721649484536, + "qps_1t_runs": [ + 31.710146511916395, + 31.651814891227882, + 31.762547646825023 + ], + "qps_1t": 31.710146511916395, + "qps_1t_min": 31.651814891227882, + "qps_1t_max": 31.762547646825023, + "median_ms": 31.298300018534064, + "p95_ms": 33.82444998715073, + "p99_ms": 39.01119997608477, + "mean_ms": 31.53767766346163 + }, + "recall_after": 0.9690721649484536 + }, + "saveload": { + "save_s": 1.100008375942707e-06, + "disk_mb": 325.56323432922363, + "load_s": 0.008934999990742654, + "first_query_after_load_ms": 23.7724999897182, + "time_to_ready_s": 0.03270749998046085 + }, + "mem_end": { + "rss_mb": 411.375, + "peak_mb": 711.171875 + }, + "status": "ok", + "finished": "2026-11-16 08:02:16", + "load_before_pct": 4.453124999999999, + "load_contaminated": false, + "wall_s": 129.6781566143036, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/lancedb_ivfflat__text384__100000__filter.json b/benchmarks/competitive/results/raw/lancedb_ivfflat__text384__100000__filter.json new file mode 100644 index 0000000..bd32a83 --- /dev/null +++ b/benchmarks/competitive/results/raw/lancedb_ivfflat__text384__100000__filter.json @@ -0,0 +1,186 @@ +{ + "system": "lancedb_ivfflat", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 08:41:16", + "filter": { + "build_with_metadata_s": 1.9351456998847425, + "mem": { + "rss_mb": 229.95703125, + "peak_mb": 517.33203125 + }, + "subsets": { + "f10": [ + { + "knob": 2, + "nq": 477, + "recall": 0.23542976939203356, + "qps_1t": 156.29965769121, + "qps_1t_runs": [ + 156.29965769121, + 160.60083160604404, + 156.20551976064894 + ], + "median_ms": 6.506800011266023, + "p95_ms": 7.435499981511384, + "p99_ms": 8.532039989950142, + "mean_ms": 6.342135638891505 + }, + { + "knob": 8, + "nq": 447, + "recall": 0.48456375838926175, + "qps_1t": 158.70324330305317, + "qps_1t_runs": [ + 150.53946166259996, + 160.732543442604, + 158.70324330305317 + ], + "median_ms": 6.571299978531897, + "p95_ms": 7.721600006334484, + "p99_ms": 8.746199985034762, + "mean_ms": 6.388453467157694 + }, + { + "knob": 32, + "nq": 414, + "recall": 0.7618357487922706, + "qps_1t": 149.94613886216828, + "qps_1t_runs": [ + 147.76194720490454, + 154.06413372590055, + 149.94613886216828 + ], + "median_ms": 6.751450011506677, + "p95_ms": 7.889020003494807, + "p99_ms": 9.078151986468571, + "mean_ms": 6.6425023355724475 + }, + { + "knob": 64, + "nq": 437, + "recall": 0.862929061784897, + "qps_1t": 143.90365977640565, + "qps_1t_runs": [ + 146.35898860294085, + 140.1701781747389, + 143.90365977640565 + ], + "median_ms": 7.028100022580475, + "p95_ms": 8.533449989045039, + "p99_ms": 10.071499965852132, + "mean_ms": 6.971931272844516 + }, + { + "knob": 1265, + "nq": 215, + "recall": 1.0, + "qps_1t": 73.06069393557968, + "qps_1t_runs": [ + 73.06069393557968, + 74.25336684646925, + 71.87058989742357 + ], + "median_ms": 13.177199987694621, + "p95_ms": 16.332939988933493, + "p99_ms": 23.4779840195551, + "mean_ms": 13.689516743468193 + } + ], + "f01": [ + { + "knob": 2, + "nq": 671, + "recall": 0.08614008941877795, + "qps_1t": 172.60083425363078, + "qps_1t_runs": [ + 169.5663395174669, + 172.7977238919334, + 172.60083425363078 + ], + "median_ms": 5.83899999037385, + "p95_ms": 7.211340020876378, + "p99_ms": 9.6610160032287, + "mean_ms": 5.826074963017291 + }, + { + "knob": 8, + "nq": 420, + "recall": 0.25476190476190474, + "qps_1t": 155.09228343919395, + "qps_1t_runs": [ + 155.09228343919395, + 157.13828141319163, + 152.37187500238755 + ], + "median_ms": 6.434749986510724, + "p95_ms": 8.244884968735276, + "p99_ms": 10.475211981683973, + "mean_ms": 6.458162222406827 + }, + { + "knob": 32, + "nq": 447, + "recall": 0.5715883668903803, + "qps_1t": 152.91142323495305, + "qps_1t_runs": [ + 152.91142323495305, + 151.34390334826222, + 156.0606412691341 + ], + "median_ms": 6.626600050367415, + "p95_ms": 7.770700030960143, + "p99_ms": 10.051760007627282, + "mean_ms": 6.518322445409154 + }, + { + "knob": 64, + "nq": 451, + "recall": 0.7354767184035477, + "qps_1t": 154.99109996789653, + "qps_1t_runs": [ + 158.87435512382578, + 154.99109996789653, + 149.35043820710197 + ], + "median_ms": 6.613400008063763, + "p95_ms": 7.6994200120680025, + "p99_ms": 9.462751988321546, + "mean_ms": 6.480642350897171 + }, + { + "knob": 1265, + "nq": 319, + "recall": 1.0, + "qps_1t": 97.76160253340882, + "qps_1t_runs": [ + 95.7891049534074, + 97.76160253340882, + 98.02989111359042 + ], + "median_ms": 10.096799989696592, + "p95_ms": 12.153680028859526, + "p99_ms": 16.170268012210638, + "mean_ms": 10.28984524580906 + } + ] + } + }, + "mem_end": { + "rss_mb": 322.19921875, + "peak_mb": 517.33203125 + }, + "status": "ok", + "finished": "2026-11-16 08:42:51", + "load_before_pct": 5.153124999999999, + "load_contaminated": false, + "wall_s": 95.23642349243164, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/numpy_bf__glove__1000000__core+mutation.json b/benchmarks/competitive/results/raw/numpy_bf__glove__1000000__core+mutation.json new file mode 100644 index 0000000..0ce9f33 --- /dev/null +++ b/benchmarks/competitive/results/raw/numpy_bf__glove__1000000__core+mutation.json @@ -0,0 +1,134 @@ +{ + "system": "numpy_bf", + "ds": "glove", + "n": 1000000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation" + ], + "started": "2026-11-16 10:27:47", + "mem_baseline": { + "rss_mb": 35.03125, + "peak_mb": 35.03125 + }, + "version": "numpy 2.5.5", + "build": { + "ingest_s": 0.1222563999472186, + "index_s": 1.600012183189392e-06, + "total_s": 0.12225799995940179, + "first_query_ms": 26.943999982904643, + "ready_s": 0.14920199994230643 + }, + "mem_after_build": { + "rss_mb": 527.4140625, + "peak_mb": 535.046875 + }, + "mem_after_first_query": { + "rss_mb": 527.6328125, + "peak_mb": 542.87890625 + }, + "sweep": [ + { + "nq": 121, + "recall_queries": 121, + "recall": 1.0, + "qps_1t_runs": [ + 36.63080399859705, + 34.43434249912364, + 33.251017371399705 + ], + "qps_1t": 34.43434249912364, + "qps_1t_min": 33.251017371399705, + "qps_1t_max": 36.63080399859705, + "median_ms": 28.821000014431775, + "p95_ms": 33.2050500146579, + "p99_ms": 35.98132596584037, + "mean_ms": 28.804823692558337, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 59.1624292069753, + 69.20876433070468, + 72.77909382401293 + ], + "qps_batch": 69.20876433070468, + "qps_batch_min": 59.1624292069753, + "qps_batch_max": 72.77909382401293, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 114, + "threads": 32, + "qps": 39.3697456266803, + "median_ms": 24.71554998192005, + "p95_ms": 29.404569999314845, + "p99_ms": 37.80972401495081, + "mean_ms": 25.274040059687582 + }, + "mem_after_sweep": { + "rss_mb": 438.2421875, + "peak_mb": 23324.5078125 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.0035168000031262636, + "per_s": 2843494.0830045748, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.15098780003609136, + "per_s": 66230.51662193668, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.023817700042854995, + "per_s": 419855.82075545, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 511.70703125, + "peak_mb": 23324.5078125 + }, + "first_query_after_ms": 68.85280000278726, + "after": { + "nq": 101, + "recall_queries": 101, + "recall": 1.0, + "qps_1t_runs": [ + 31.983570703500607, + 33.1606212116313, + 32.34572753361104 + ], + "qps_1t": 32.34572753361104, + "qps_1t_min": 31.983570703500607, + "qps_1t_max": 33.1606212116313, + "median_ms": 30.406599980778992, + "p95_ms": 34.835900011239566, + "p99_ms": 44.31360597838653, + "mean_ms": 30.779429044798878 + }, + "recall_after": 1.0 + }, + "mem_end": { + "rss_mb": 502.62109375, + "peak_mb": 23324.5078125 + }, + "status": "ok", + "finished": "2026-11-16 10:29:51", + "load_before_pct": 11.518749999999995, + "load_contaminated": false, + "wall_s": 124.18437433242798, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/numpy_bf__glove__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/numpy_bf__glove__100000__core+mutation+saveload.json new file mode 100644 index 0000000..f91afd0 --- /dev/null +++ b/benchmarks/competitive/results/raw/numpy_bf__glove__100000__core+mutation+saveload.json @@ -0,0 +1,147 @@ +{ + "system": "numpy_bf", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 08:24:49", + "mem_baseline": { + "rss_mb": 35.04296875, + "peak_mb": 35.04296875 + }, + "version": "numpy 2.5.5", + "build": { + "ingest_s": 0.011630199966020882, + "index_s": 2.100015990436077e-06, + "total_s": 0.011632299982011318, + "first_query_ms": 2.3740999749861658, + "ready_s": 0.014006399956997484, + "total_s_runs": [ + 0.011632299982011318, + 0.011848699941765517, + 0.011565599997993559 + ] + }, + "mem_after_build": { + "rss_mb": 86.27734375, + "peak_mb": 96.0390625 + }, + "mem_after_first_query": { + "rss_mb": 86.12890625, + "peak_mb": 96.0390625 + }, + "sweep": [ + { + "nq": 1607, + "recall_queries": 1607, + "recall": 1.0, + "qps_1t_runs": [ + 478.8508532046889, + 528.2931208601073, + 537.2645987466597 + ], + "qps_1t": 528.2931208601073, + "qps_1t_min": 478.8508532046889, + "qps_1t_max": 537.2645987466597, + "median_ms": 1.816099975258112, + "p95_ms": 2.5314000085927546, + "p99_ms": 3.69705999037251, + "mean_ms": 1.9475005815455486, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 1687.3489242631317, + 1732.7897802056275, + 1745.4020437750667 + ], + "qps_batch": 1732.7897802056275, + "qps_batch_min": 1687.3489242631317, + "qps_batch_max": 1745.4020437750667, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 2000, + "threads": 32, + "qps": 1017.9281058967854, + "median_ms": 0.9073999826796353, + "p95_ms": 1.29273000638932, + "p99_ms": 1.6822410031454647, + "mean_ms": 0.95744280005844 + }, + "mem_after_sweep": { + "rss_mb": 92.2421875, + "peak_mb": 3144.34375 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.0011317000025883317, + "per_s": 8836264.007359562, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.006155200011562556, + "per_s": 1624642.5755808062, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.0021666999673470855, + "per_s": 4615313.680114203, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 105.03515625, + "peak_mb": 3144.34375 + }, + "first_query_after_ms": 2.9347999952733517, + "after": { + "nq": 1677, + "recall_queries": 1677, + "recall": 1.0, + "qps_1t_runs": [ + 486.3989849075593, + 484.53277203690453, + 548.716773449077 + ], + "qps_1t": 486.3989849075593, + "qps_1t_min": 484.53277203690453, + "qps_1t_max": 548.716773449077, + "median_ms": 1.908699981868267, + "p95_ms": 2.492800005711615, + "p99_ms": 3.2826900191139394, + "mean_ms": 1.9807343469810312 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 0.024996799998916686, + "disk_mb": 42.8011474609375, + "load_s": 0.01716030004899949, + "first_query_after_load_ms": 2.3825999815016985, + "time_to_ready_s": 0.019542900030501187 + }, + "mem_end": { + "rss_mb": 95.765625, + "peak_mb": 3144.34375 + }, + "status": "ok", + "finished": "2026-11-16 08:25:18", + "load_before_pct": 3.6312499999999983, + "load_contaminated": false, + "wall_s": 29.33121418952942, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/numpy_bf__glove__100000__filter.json b/benchmarks/competitive/results/raw/numpy_bf__glove__100000__filter.json new file mode 100644 index 0000000..af285db --- /dev/null +++ b/benchmarks/competitive/results/raw/numpy_bf__glove__100000__filter.json @@ -0,0 +1,66 @@ +{ + "system": "numpy_bf", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 08:54:11", + "filter": { + "build_with_metadata_s": 0.012173999974038452, + "mem": { + "rss_mb": 85.78515625, + "peak_mb": 95.54296875 + }, + "subsets": { + "f10": [ + { + "knob": null, + "nq": 1659, + "recall": 1.0, + "qps_1t": 552.0243702974032, + "qps_1t_runs": [ + 528.7185643145853, + 554.6350455179166, + 552.0243702974032 + ], + "median_ms": 1.7308000242337584, + "p95_ms": 2.2066600271500647, + "p99_ms": 3.2523079845122935, + "mean_ms": 1.8352890102372774 + } + ], + "f01": [ + { + "knob": null, + "nq": 1751, + "recall": 1.0, + "qps_1t": 623.9321931919695, + "qps_1t_runs": [ + 574.6952337204558, + 623.9321931919695, + 648.6188142113433 + ], + "median_ms": 1.518300035968423, + "p95_ms": 2.1711000008508563, + "p99_ms": 3.3314000186510375, + "mean_ms": 1.6281761847415686 + } + ] + } + }, + "mem_end": { + "rss_mb": 40.23046875, + "peak_mb": 95.54296875 + }, + "status": "ok", + "finished": "2026-11-16 08:54:29", + "load_before_pct": 4.5249999999999995, + "load_contaminated": false, + "wall_s": 17.96711754798889, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/numpy_bf__sift__1000000__core.json b/benchmarks/competitive/results/raw/numpy_bf__sift__1000000__core.json new file mode 100644 index 0000000..f9b4459 --- /dev/null +++ b/benchmarks/competitive/results/raw/numpy_bf__sift__1000000__core.json @@ -0,0 +1,91 @@ +{ + "system": "numpy_bf", + "ds": "sift", + "n": 1000000, + "d": 128, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 11:36:09", + "mem_baseline": { + "rss_mb": 35.53125, + "peak_mb": 35.53125 + }, + "version": "numpy 2.5.5", + "build": { + "ingest_s": 0.13666780025232583, + "index_s": 2.100015990436077e-06, + "total_s": 0.13666990026831627, + "first_query_ms": 29.749699984677136, + "ready_s": 0.1664196002529934 + }, + "mem_after_build": { + "rss_mb": 634.73046875, + "peak_mb": 644.5 + }, + "mem_after_first_query": { + "rss_mb": 634.94921875, + "peak_mb": 650.1953125 + }, + "sweep": [ + { + "nq": 110, + "recall_queries": 110, + "recall": 0.9981818181818182, + "qps_1t_runs": [ + 28.12480880033291, + 26.871127609015293, + 26.663160701938843 + ], + "qps_1t": 26.871127609015293, + "qps_1t_min": 26.663160701938843, + "qps_1t_max": 28.12480880033291, + "median_ms": 37.35805000178516, + "p95_ms": 41.83341998432297, + "p99_ms": 46.82670901587697, + "mean_ms": 36.75846424355934, + "knob": null, + "batch": { + "recall": 0.99925, + "qps_batch_runs": [ + 48.02491809377273, + 67.38187995185218, + 68.93683072445786 + ], + "qps_batch": 67.38187995185218, + "qps_batch_min": 48.02491809377273, + "qps_batch_max": 68.93683072445786, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 98, + "threads": 32, + "qps": 31.04580650055689, + "median_ms": 32.03544998541474, + "p95_ms": 39.03979499591514, + "p99_ms": 40.99516700254752, + "mean_ms": 32.78946632865443 + }, + "mem_after_sweep": { + "rss_mb": 532.609375, + "peak_mb": 23417.640625 + }, + "ref_knob": null, + "ref_recall_pre": 0.9981818181818182, + "mem_end": { + "rss_mb": 530.40625, + "peak_mb": 23417.640625 + }, + "status": "ok", + "finished": "2026-11-16 11:38:15", + "load_before_pct": 11.124999999999995, + "load_contaminated": false, + "wall_s": 126.72024130821228, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/numpy_bf__sift__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/numpy_bf__sift__100000__core+mutation+saveload.json new file mode 100644 index 0000000..702192d --- /dev/null +++ b/benchmarks/competitive/results/raw/numpy_bf__sift__100000__core+mutation+saveload.json @@ -0,0 +1,147 @@ +{ + "system": "numpy_bf", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 11:14:10", + "mem_baseline": { + "rss_mb": 35.43359375, + "peak_mb": 35.43359375 + }, + "version": "numpy 2.5.5", + "build": { + "ingest_s": 0.015353899914771318, + "index_s": 2.00001522898674e-06, + "total_s": 0.015355899930000305, + "first_query_ms": 3.0196000006981194, + "ready_s": 0.018375499930698425, + "total_s_runs": [ + 0.015355899930000305, + 0.017753299965988845, + 0.015054100076667964 + ] + }, + "mem_after_build": { + "rss_mb": 97.43359375, + "peak_mb": 109.296875 + }, + "mem_after_first_query": { + "rss_mb": 97.27734375, + "peak_mb": 109.296875 + }, + "sweep": [ + { + "nq": 842, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 282.134764185857, + 297.3362380747439, + 266.56199198650955 + ], + "qps_1t": 282.134764185857, + "qps_1t_min": 266.56199198650955, + "qps_1t_max": 297.3362380747439, + "median_ms": 3.343299962580204, + "p95_ms": 4.870125005254522, + "p99_ms": 6.689900023047812, + "mean_ms": 3.553024624666621, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 1543.385293765154, + 1542.770258454746, + 1529.395751041704 + ], + "qps_batch": 1542.770258454746, + "qps_batch_min": 1529.395751041704, + "qps_batch_max": 1543.385293765154, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 830, + "threads": 32, + "qps": 427.9685161615656, + "median_ms": 2.1524999756366014, + "p95_ms": 4.043454964994452, + "p99_ms": 5.958811984746724, + "mean_ms": 2.3629968275267434 + }, + "mem_after_sweep": { + "rss_mb": 104.4765625, + "peak_mb": 3156.97265625 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.0016330999787896872, + "per_s": 6123323.819654408, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.006400400015991181, + "per_s": 1562402.3459495252, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.002769000013358891, + "per_s": 3611412.044693225, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 120.86328125, + "peak_mb": 3156.97265625 + }, + "first_query_after_ms": 3.0731000006198883, + "after": { + "nq": 856, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 286.2934374403579, + 261.30254062037386, + 274.0523431570449 + ], + "qps_1t": 274.0523431570449, + "qps_1t_min": 261.30254062037386, + "qps_1t_max": 286.2934374403579, + "median_ms": 3.4618499921634793, + "p95_ms": 4.889120016014205, + "p99_ms": 6.9433079991722435, + "mean_ms": 3.656279751098921 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 0.030838600010611117, + "disk_mb": 54.5504150390625, + "load_s": 0.025650800031144172, + "first_query_after_load_ms": 4.709500004537404, + "time_to_ready_s": 0.030360300035681576 + }, + "mem_end": { + "rss_mb": 109.3125, + "peak_mb": 3156.97265625 + }, + "status": "ok", + "finished": "2026-11-16 11:14:40", + "load_before_pct": 12.63750000000001, + "load_contaminated": false, + "wall_s": 30.189536809921265, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/numpy_bf__text384__1000000__core+mutation.json b/benchmarks/competitive/results/raw/numpy_bf__text384__1000000__core+mutation.json new file mode 100644 index 0000000..f4d1477 --- /dev/null +++ b/benchmarks/competitive/results/raw/numpy_bf__text384__1000000__core+mutation.json @@ -0,0 +1,134 @@ +{ + "system": "numpy_bf", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation" + ], + "started": "2026-11-16 09:43:17", + "mem_baseline": { + "rss_mb": 37.40625, + "peak_mb": 37.40625 + }, + "version": "numpy 2.5.5", + "build": { + "ingest_s": 0.29768730007344857, + "index_s": 2.100015990436077e-06, + "total_s": 0.297689400089439, + "first_query_ms": 70.26820001192391, + "ready_s": 0.3679576001013629 + }, + "mem_after_build": { + "rss_mb": 1613.10546875, + "peak_mb": 1642.40625 + }, + "mem_after_first_query": { + "rss_mb": 1613.32421875, + "peak_mb": 1642.40625 + }, + "sweep": [ + { + "nq": 42, + "recall_queries": 42, + "recall": 1.0, + "qps_1t_runs": [ + 14.425729730066768, + 14.061476305081415, + 14.181329092005516 + ], + "qps_1t": 14.181329092005516, + "qps_1t_min": 14.061476305081415, + "qps_1t_max": 14.425729730066768, + "median_ms": 69.28120003431104, + "p95_ms": 77.11289999133442, + "p99_ms": 83.95727499737404, + "mean_ms": 70.31737460610678, + "knob": null, + "batch": { + "recall": 0.99995, + "qps_batch_runs": [ + 112.22853446621616, + 142.438233031093, + 145.18666707814978 + ], + "qps_batch": 142.438233031093, + "qps_batch_min": 112.22853446621616, + "qps_batch_max": 145.18666707814978, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 77, + "threads": 32, + "qps": 27.952094173654178, + "median_ms": 35.07129999343306, + "p95_ms": 39.29985000286251, + "p99_ms": 41.545520001091056, + "mean_ms": 35.583145023440515 + }, + "mem_after_sweep": { + "rss_mb": 1507.59375, + "peak_mb": 24393.6484375 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.0038690000073984265, + "per_s": 2584647.190715347, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.08686679997481406, + "per_s": 115118.7795901239, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.014911399979609996, + "per_s": 670627.8427025031, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 1631.4375, + "peak_mb": 24393.6484375 + }, + "first_query_after_ms": 73.75340000726283, + "after": { + "nq": 40, + "recall_queries": 40, + "recall": 1.0, + "qps_1t_runs": [ + 14.081808691877153, + 14.042345887366727, + 14.327790915131303 + ], + "qps_1t": 14.081808691877153, + "qps_1t_min": 14.042345887366727, + "qps_1t_max": 14.327790915131303, + "median_ms": 68.76729999203235, + "p95_ms": 81.97205997421406, + "p99_ms": 88.8461139757419, + "mean_ms": 70.67373582879857 + }, + "recall_after": 1.0 + }, + "mem_end": { + "rss_mb": 1598.6640625, + "peak_mb": 24393.6484375 + }, + "status": "ok", + "finished": "2026-11-16 09:44:38", + "load_before_pct": 5.881250000000006, + "load_contaminated": false, + "wall_s": 80.94404435157776, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/numpy_bf__text384__1000000__filter.json b/benchmarks/competitive/results/raw/numpy_bf__text384__1000000__filter.json new file mode 100644 index 0000000..7685055 --- /dev/null +++ b/benchmarks/competitive/results/raw/numpy_bf__text384__1000000__filter.json @@ -0,0 +1,66 @@ +{ + "system": "numpy_bf", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 11:02:34", + "filter": { + "build_with_metadata_s": 0.3003055000444874, + "mem": { + "rss_mb": 1614.1484375, + "peak_mb": 1643.4453125 + }, + "subsets": { + "f10": [ + { + "knob": null, + "nq": 41, + "recall": 1.0, + "qps_1t": 12.887030124605031, + "qps_1t_runs": [ + 12.887030124605031, + 12.594177956844689, + 13.63567323591825 + ], + "median_ms": 74.57850000355393, + "p95_ms": 93.15719002042897, + "p99_ms": 100.24626799160615, + "mean_ms": 76.77873740033284 + } + ], + "f01": [ + { + "knob": null, + "nq": 43, + "recall": 1.0, + "qps_1t": 13.966856066036692, + "qps_1t_runs": [ + 13.814059608766382, + 14.032562660715573, + 13.966856066036692 + ], + "median_ms": 69.72070003394037, + "p95_ms": 83.76633997540921, + "p99_ms": 93.87779998360202, + "mean_ms": 71.75030387475971 + } + ] + } + }, + "mem_end": { + "rss_mb": 39.05859375, + "peak_mb": 1643.4453125 + }, + "status": "ok", + "finished": "2026-11-16 11:02:56", + "load_before_pct": 9.706249999999995, + "load_contaminated": false, + "wall_s": 21.85072112083435, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/numpy_bf__text384__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/numpy_bf__text384__100000__core+mutation+saveload.json new file mode 100644 index 0000000..e4a7d82 --- /dev/null +++ b/benchmarks/competitive/results/raw/numpy_bf__text384__100000__core+mutation+saveload.json @@ -0,0 +1,147 @@ +{ + "system": "numpy_bf", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 08:02:23", + "mem_baseline": { + "rss_mb": 37.5234375, + "peak_mb": 37.5234375 + }, + "version": "numpy 2.5.5", + "build": { + "ingest_s": 0.02955939999083057, + "index_s": 1.9999570213258266e-06, + "total_s": 0.029561399947851896, + "first_query_ms": 7.355800014920533, + "ready_s": 0.03691719996277243, + "total_s_runs": [ + 0.029561399947851896, + 0.029899799963459373, + 0.029745299951173365 + ] + }, + "mem_after_build": { + "rss_mb": 197.0625, + "peak_mb": 228.515625 + }, + "mem_after_first_query": { + "rss_mb": 196.8828125, + "peak_mb": 228.515625 + }, + "sweep": [ + { + "nq": 448, + "recall_queries": 1000, + "recall": 0.9999, + "qps_1t_runs": [ + 143.21663674351632, + 145.64401038113556, + 144.26549799463044 + ], + "qps_1t": 144.26549799463044, + "qps_1t_min": 143.21663674351632, + "qps_1t_max": 145.64401038113556, + "median_ms": 6.752399989636615, + "p95_ms": 7.655209978111083, + "p99_ms": 10.1361830078531, + "mean_ms": 6.9267165908535215, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 1552.8583269017806, + 1563.6238546363275, + 1543.6353291333107 + ], + "qps_batch": 1552.8583269017806, + "qps_batch_min": 1543.6353291333107, + "qps_batch_max": 1563.6238546363275, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 582, + "threads": 32, + "qps": 216.94630995397839, + "median_ms": 4.542099981335923, + "p95_ms": 5.10002500959672, + "p99_ms": 6.665445011458358, + "mean_ms": 4.615391064897971 + }, + "mem_after_sweep": { + "rss_mb": 206.20703125, + "peak_mb": 3258.85546875 + }, + "ref_knob": null, + "ref_recall_pre": 0.9999, + "mutation": { + "ref_knob": null, + "recall_before": 0.9999, + "insert_10k": { + "seconds": 0.0031792999943718314, + "per_s": 3145346.4654806214, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.0074541999492794275, + "per_s": 1341525.592021001, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.0020655999542213976, + "per_s": 4841208.472900734, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 251.890625, + "peak_mb": 3258.85546875 + }, + "first_query_after_ms": 7.095300010405481, + "after": { + "nq": 421, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 132.06745992427594, + 134.18411802835428, + 134.63707648157887 + ], + "qps_1t": 134.18411802835428, + "qps_1t_min": 132.06745992427594, + "qps_1t_max": 134.63707648157887, + "median_ms": 7.252100040204823, + "p95_ms": 8.472410024842246, + "p99_ms": 12.065381995634917, + "mean_ms": 7.483903087840747 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 0.11037170002236962, + "disk_mb": 161.9722900390625, + "load_s": 0.0681961000082083, + "first_query_after_load_ms": 9.157199994660914, + "time_to_ready_s": 0.07735330000286922 + }, + "mem_end": { + "rss_mb": 220.57421875, + "peak_mb": 3258.85546875 + }, + "status": "ok", + "finished": "2026-11-16 08:03:04", + "load_before_pct": 5.221875, + "load_contaminated": false, + "wall_s": 40.683335065841675, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/numpy_bf__text384__100000__filter.json b/benchmarks/competitive/results/raw/numpy_bf__text384__100000__filter.json new file mode 100644 index 0000000..daed859 --- /dev/null +++ b/benchmarks/competitive/results/raw/numpy_bf__text384__100000__filter.json @@ -0,0 +1,66 @@ +{ + "system": "numpy_bf", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 08:42:59", + "filter": { + "build_with_metadata_s": 0.029936499951872975, + "mem": { + "rss_mb": 196.30078125, + "peak_mb": 227.7265625 + }, + "subsets": { + "f10": [ + { + "knob": null, + "nq": 415, + "recall": 1.0, + "qps_1t": 141.22187135997544, + "qps_1t_runs": [ + 138.51827369505224, + 141.22187135997544, + 141.79592363184014 + ], + "median_ms": 6.85239996528253, + "p95_ms": 8.506780001334846, + "p99_ms": 13.748928005807068, + "mean_ms": 7.117569718807262 + } + ], + "f01": [ + { + "knob": null, + "nq": 441, + "recall": 1.0, + "qps_1t": 146.07631563739315, + "qps_1t_runs": [ + 142.94661528395815, + 146.2720236747061, + 146.07631563739315 + ], + "median_ms": 6.655600038357079, + "p95_ms": 7.938750035827979, + "p99_ms": 12.056917967274762, + "mean_ms": 6.892644217465206 + } + ] + } + }, + "mem_end": { + "rss_mb": 40.203125, + "peak_mb": 227.7265625 + }, + "status": "ok", + "finished": "2026-11-16 08:43:17", + "load_before_pct": 5.184375000000003, + "load_contaminated": false, + "wall_s": 18.457741260528564, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/numpy_bf__text384__10000__core.json b/benchmarks/competitive/results/raw/numpy_bf__text384__10000__core.json new file mode 100644 index 0000000..fe6bfce --- /dev/null +++ b/benchmarks/competitive/results/raw/numpy_bf__text384__10000__core.json @@ -0,0 +1,91 @@ +{ + "system": "numpy_bf", + "ds": "text384", + "n": 10000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 08:57:09", + "mem_baseline": { + "rss_mb": 37.44140625, + "peak_mb": 37.44140625 + }, + "version": "numpy 2.5.5", + "build": { + "ingest_s": 0.0029279000009410083, + "index_s": 1.200009137392044e-06, + "total_s": 0.0029291000100784004, + "first_query_ms": 0.8420000085607171, + "ready_s": 0.0037711000186391175 + }, + "mem_after_build": { + "rss_mb": 54.22265625, + "peak_mb": 68.875 + }, + "mem_after_first_query": { + "rss_mb": 54.44140625, + "peak_mb": 68.875 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 1.0, + "qps_1t_runs": [ + 3364.167051224355, + 3126.0887583616272, + 3518.481704649937 + ], + "qps_1t": 3364.167051224355, + "qps_1t_min": 3126.0887583616272, + "qps_1t_max": 3518.481704649937, + "median_ms": 0.3035000408999622, + "p95_ms": 0.3500199876725674, + "p99_ms": 0.4527010128367692, + "mean_ms": 0.30045081649829325, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 14581.978423595207, + 14956.87186371766, + 14689.982611883277 + ], + "qps_batch": 14689.982611883277, + "qps_batch_min": 14581.978423595207, + "qps_batch_max": 14956.87186371766, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 2000, + "threads": 32, + "qps": 6581.068879255676, + "median_ms": 0.15139998868107796, + "p95_ms": 0.1896050438517705, + "p99_ms": 0.23000400455202916, + "mean_ms": 0.15553231681891097 + }, + "mem_after_sweep": { + "rss_mb": 63.44140625, + "peak_mb": 369.37890625 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 59.63671875, + "peak_mb": 369.37890625 + }, + "status": "ok", + "finished": "2026-11-16 08:57:12", + "load_before_pct": 3.9468749999999986, + "load_contaminated": false, + "wall_s": 3.463876724243164, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/qdrant_local__text384__100000__core+mutation.json b/benchmarks/competitive/results/raw/qdrant_local__text384__100000__core+mutation.json new file mode 100644 index 0000000..fa261bd --- /dev/null +++ b/benchmarks/competitive/results/raw/qdrant_local__text384__100000__core+mutation.json @@ -0,0 +1,120 @@ +{ + "system": "qdrant_local", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation" + ], + "started": "2026-11-16 09:03:29", + "mem_baseline": { + "rss_mb": 37.4296875, + "peak_mb": 37.4296875 + }, + "version": "qdrant-client 1.20.0 (local mode)", + "build": { + "ingest_s": 284.33503050002037, + "index_s": 2.300017513334751e-06, + "total_s": 284.3350328000379, + "first_query_ms": 126.82280002627522, + "ready_s": 284.46185560006415 + }, + "mem_after_build": { + "rss_mb": 431.7421875, + "peak_mb": 603.9453125 + }, + "mem_after_first_query": { + "rss_mb": 436.0, + "peak_mb": 726.62109375 + }, + "sweep": [ + { + "nq": 30, + "recall_queries": 30, + "recall": 1.0, + "qps_1t_runs": [ + 7.250915113595278, + 7.141317168949704, + 7.241052014743276 + ], + "qps_1t": 7.241052014743276, + "qps_1t_min": 7.141317168949704, + "qps_1t_max": 7.250915113595278, + "median_ms": 136.17230000090785, + "p95_ms": 152.33883499749936, + "p99_ms": 157.8155379614327, + "mean_ms": 138.68176555649066, + "knob": null + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 30, + "threads": 32, + "qps": 7.554045037370992, + "median_ms": 132.17250001616776, + "p95_ms": 139.86650497536175, + "p99_ms": 144.6756109787384, + "mean_ms": 132.34423555420815 + }, + "mem_after_sweep": { + "rss_mb": 436.7109375, + "peak_mb": 730.24609375 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 25.664128399977926, + "per_s": 389.6489233590571, + "batch": 1000 + }, + "update_10k": { + "seconds": 26.403800099971704, + "per_s": 378.73336270299654, + "batch": 1000 + }, + "delete_10k": { + "seconds": 31.76803919998929, + "per_s": 314.7817823142, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 466.4453125, + "peak_mb": 730.24609375 + }, + "first_query_after_ms": 136.2393000163138, + "after": { + "nq": 30, + "recall_queries": 30, + "recall": 1.0, + "qps_1t_runs": [ + 6.493147683826058 + ], + "qps_1t": 6.493147683826058, + "qps_1t_min": 6.493147683826058, + "qps_1t_max": 6.493147683826058, + "median_ms": 153.4129999927245, + "p95_ms": 163.65067503065802, + "p99_ms": 169.4600670260843, + "mean_ms": 154.0085100006157 + }, + "recall_after": 1.0 + }, + "mem_end": { + "rss_mb": 429.82421875, + "peak_mb": 790.421875 + }, + "status": "ok", + "finished": "2026-11-16 09:10:18", + "load_before_pct": 5.025000000000002, + "load_contaminated": false, + "wall_s": 409.1669487953186, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/qdrant_local__text384__10000__core.json b/benchmarks/competitive/results/raw/qdrant_local__text384__10000__core.json new file mode 100644 index 0000000..dd741d6 --- /dev/null +++ b/benchmarks/competitive/results/raw/qdrant_local__text384__10000__core.json @@ -0,0 +1,79 @@ +{ + "system": "qdrant_local", + "ds": "text384", + "n": 10000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 08:59:01", + "mem_baseline": { + "rss_mb": 37.36328125, + "peak_mb": 37.36328125 + }, + "version": "qdrant-client 1.20.0 (local mode)", + "build": { + "ingest_s": 33.30397010000888, + "index_s": 1.300009898841381e-06, + "total_s": 33.30397140001878, + "first_query_ms": 39.89770001498982, + "ready_s": 33.34386910003377 + }, + "mem_after_build": { + "rss_mb": 147.74609375, + "peak_mb": 198.59375 + }, + "mem_after_first_query": { + "rss_mb": 148.328125, + "peak_mb": 198.59375 + }, + "sweep": [ + { + "nq": 231, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 72.99020291564169, + 73.43907393726488, + 72.42674590726222 + ], + "qps_1t": 72.99020291564169, + "qps_1t_min": 72.42674590726222, + "qps_1t_max": 73.43907393726488, + "median_ms": 13.216899998951703, + "p95_ms": 15.94205998117104, + "p99_ms": 22.059979962650694, + "mean_ms": 13.7080841274928, + "knob": null + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 210, + "threads": 32, + "qps": 79.2029707126397, + "median_ms": 12.475349998567253, + "p95_ms": 14.004170024418272, + "p99_ms": 15.141900998423829, + "mean_ms": 12.645105237606913 + }, + "mem_after_sweep": { + "rss_mb": 149.15625, + "peak_mb": 198.59375 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 146.22265625, + "peak_mb": 198.59375 + }, + "status": "ok", + "finished": "2026-11-16 09:00:05", + "load_before_pct": 3.565624999999999, + "load_contaminated": false, + "wall_s": 63.546509981155396, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/rust_default__glove__1000000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/rust_default__glove__1000000__core+mutation+saveload.json new file mode 100644 index 0000000..d9d4fb6 --- /dev/null +++ b/benchmarks/competitive/results/raw/rust_default__glove__1000000__core+mutation+saveload.json @@ -0,0 +1,380 @@ +{ + "label": "rust_default", + "runs": [ + { + "all_threads": 32, + "build": { + "first_query_ms": 663.5181, + "index_s": 0.0, + "ingest_s": 0.7856887000000002, + "ready_s": 1.4492068000000002, + "total_s": 0.7856887000000002 + }, + "concurrent_clients": { + "clients": 8, + "qps": 15.293718053693148, + "scan_threads": 1 + }, + "d": 100, + "ds": "glove", + "k": 10, + "loop_allthreads": { + "mean_ms": 14.153318148148127, + "median_ms": 13.2813, + "nq": 180, + "p95_ms": 19.4111, + "p99_ms": 24.898400000000002, + "qps": 70.89076929717626, + "qps_runs": [ + 67.65923262101195, + 73.67135552101732, + 70.89076929717626 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 1359.53125, + "rss_mb": 1351.91015625 + }, + "mem_after_first_query": { + "peak_mb": 1359.53125, + "rss_mb": 1351.91015625 + }, + "mem_after_ingest": { + "peak_mb": 978.4296875, + "rss_mb": 850.29296875 + }, + "mem_after_sweep": { + "peak_mb": 1359.53125, + "rss_mb": 1352.21875 + }, + "mem_baseline": { + "peak_mb": 6.58984375, + "rss_mb": 6.58984375 + }, + "n": 1000000, + "queries": 2000, + "sweep": [ + { + "knob": null, + "mean_ms": 22.32282999999998, + "median_ms": 22.503700000000002, + "nq": 170, + "p95_ms": 24.6228, + "p99_ms": 25.876199999999997, + "qps_1t": 43.495671374749, + "qps_1t_max": 48.56086450221941, + "qps_1t_min": 42.76248683481381, + "qps_1t_runs": [ + 48.56086450221941, + 43.495671374749, + 42.76248683481381 + ], + "recall": 1.0 + } + ], + "system": "rust_default_exact", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + }, + { + "all_threads": 32, + "build": { + "cache_s": 0.6635181, + "first_query_ms": 0.0, + "index_s": 5.6935397, + "ingest_s": 0.7856887000000002, + "ready_s": 7.142746499999999, + "total_s": 7.142746499999999 + }, + "d": 100, + "ds": "glove", + "index_stats": { + "build_seconds": 5.6926484, + "index_bytes": 9632008, + "nlist": 4000, + "trained_n": 1000000 + }, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 1428.23046875, + "rss_mb": 1362.37890625 + }, + "mem_after_ingest": { + "peak_mb": 978.4296875, + "rss_mb": 850.29296875 + }, + "mem_after_sweep": { + "peak_mb": 1428.23046875, + "rss_mb": 1362.41796875 + }, + "mem_baseline": { + "peak_mb": 6.58984375, + "rss_mb": 6.58984375 + }, + "mutation": { + "after": { + "mean_ms": 6.04390242165242, + "median_ms": 6.0344999999999995, + "nq": 468, + "p95_ms": 7.765000000000001, + "p99_ms": 8.5421, + "qps_1t": 164.65186263475061, + "qps_1t_max": 167.33848778783036, + "qps_1t_min": 164.40945598878318, + "qps_1t_runs": [ + 164.40945598878318, + 167.33848778783036, + 164.65186263475061 + ], + "recall": 0.9540598290598291 + }, + "delete_10k": { + "per_s": 303254.5275900969, + "seconds": 0.0329756 + }, + "first_query_after_ms": 745.865, + "index_stats_after": { + "builds": 90140, + "churn": 30000, + "pending": 0, + "trained_n": 1000000 + }, + "insert_10k": { + "per_s": 64013.9294310442, + "seconds": 0.156216 + }, + "mem_after": { + "peak_mb": 1748.73046875, + "rss_mb": 1378.09765625 + }, + "recall_after": 0.9540598290598291, + "recall_before": 0.9536842105263158, + "ref_knob": 256, + "update_10k": { + "per_s": 238537.66867594898, + "seconds": 0.0419221 + } + }, + "n": 1000000, + "queries": 2000, + "ref_knob": 256, + "ref_recall_pre": 0.9536842105263158, + "saveload": { + "disk_mb": 1215.590708732605, + "first_query_after_load_ms": 7592.225799999999, + "load_s": 10.7524962, + "note": "load = JSON parse + insert; first query includes the scan-cache build and the IVF index build (the index is never persisted)", + "save_s": 23.1342869, + "time_to_ready_s": 18.344721999999997 + }, + "sweep": [ + { + "allthreads": { + "median_ms": 0.1735, + "qps": 5473.557108768614, + "recall": 0.30875 + }, + "knob": 1, + "mean_ms": 0.09495421666666651, + "median_ms": 0.0922, + "nq": 2000, + "p95_ms": 0.134, + "p99_ms": 0.1669, + "qps_1t": 10525.25218504233, + "qps_1t_max": 10565.854328566365, + "qps_1t_min": 10503.25837332886, + "qps_1t_runs": [ + 10565.854328566365, + 10525.25218504233, + 10503.25837332886 + ], + "recall": 0.30875 + }, + { + "allthreads": { + "median_ms": 0.2337, + "qps": 4107.446698177634, + "recall": 0.42305 + }, + "knob": 2, + "mean_ms": 0.13864956666666656, + "median_ms": 0.13540000000000002, + "nq": 2000, + "p95_ms": 0.1972, + "p99_ms": 0.2407, + "qps_1t": 7216.137014236721, + "qps_1t_max": 7478.044461461132, + "qps_1t_min": 6961.577660575742, + "qps_1t_runs": [ + 6961.577660575742, + 7216.137014236721, + 7478.044461461132 + ], + "recall": 0.42305 + }, + { + "allthreads": { + "median_ms": 0.3635, + "qps": 2611.1378607678635, + "recall": 0.5377 + }, + "knob": 4, + "mean_ms": 0.19636315000000049, + "median_ms": 0.1918, + "nq": 2000, + "p95_ms": 0.2825, + "p99_ms": 0.3448, + "qps_1t": 5053.718500804303, + "qps_1t_max": 5225.215951643762, + "qps_1t_min": 5004.110877085523, + "qps_1t_runs": [ + 5004.110877085523, + 5225.215951643762, + 5053.718500804303 + ], + "recall": 0.5377 + }, + { + "allthreads": { + "median_ms": 0.589, + "qps": 1646.895412753984, + "recall": 0.6346 + }, + "knob": 8, + "mean_ms": 0.30542776666666693, + "median_ms": 0.2967, + "nq": 2000, + "p95_ms": 0.4406, + "p99_ms": 0.5218, + "qps_1t": 3277.4307433819245, + "qps_1t_max": 3328.959635199287, + "qps_1t_min": 3217.792073355365, + "qps_1t_runs": [ + 3217.792073355365, + 3277.4307433819245, + 3328.959635199287 + ], + "recall": 0.6346 + }, + { + "allthreads": { + "median_ms": 0.9058999999999999, + "qps": 1028.4823796284809, + "recall": 0.72835 + }, + "knob": 16, + "mean_ms": 0.5200717500000002, + "median_ms": 0.5075000000000001, + "nq": 2000, + "p95_ms": 0.7512000000000001, + "p99_ms": 0.8566999999999999, + "qps_1t": 1922.5306655656982, + "qps_1t_max": 1938.2230177550957, + "qps_1t_min": 1907.9199665732406, + "qps_1t_runs": [ + 1938.2230177550957, + 1907.9199665732406, + 1922.5306655656982 + ], + "recall": 0.72835 + }, + { + "allthreads": { + "median_ms": 1.747, + "qps": 563.9185987796458, + "recall": 0.8031725888324873 + }, + "knob": 32, + "mean_ms": 0.8734791833333301, + "median_ms": 0.8537, + "nq": 2000, + "p95_ms": 1.2114, + "p99_ms": 1.39, + "qps_1t": 1167.9571425134307, + "qps_1t_max": 1174.0134471500257, + "qps_1t_min": 1095.9351491334842, + "qps_1t_runs": [ + 1174.0134471500257, + 1167.9571425134307, + 1095.9351491334842 + ], + "recall": 0.8013 + }, + { + "allthreads": { + "median_ms": 2.6516, + "qps": 363.3975807228954, + "recall": 0.8654895666131621 + }, + "knob": 64, + "mean_ms": 1.6018584961142626, + "median_ms": 1.5775, + "nq": 1587, + "p95_ms": 2.1917999999999997, + "p99_ms": 2.4734000000000003, + "qps_1t": 644.0744923489667, + "qps_1t_max": 644.3474246606822, + "qps_1t_min": 587.8886927778146, + "qps_1t_runs": [ + 587.8886927778146, + 644.3474246606822, + 644.0744923489667 + ], + "recall": 0.8640201638311279 + }, + { + "allthreads": { + "median_ms": 3.3215, + "qps": 272.8295808389948, + "recall": 0.9198317307692307 + }, + "knob": 128, + "mean_ms": 3.1683282663771237, + "median_ms": 3.1863, + "nq": 921, + "p95_ms": 4.2356, + "p99_ms": 4.6311, + "qps_1t": 313.6266915452204, + "qps_1t_max": 321.3772505174752, + "qps_1t_min": 312.02486929681385, + "qps_1t_runs": [ + 313.6266915452204, + 321.3772505174752, + 312.02486929681385 + ], + "recall": 0.9204125950054289 + }, + { + "allthreads": { + "median_ms": 5.0913, + "qps": 166.43642960571202, + "recall": 0.9556492411467117 + }, + "knob": 256, + "mean_ms": 5.9860054035087655, + "median_ms": 6.014, + "nq": 475, + "p95_ms": 7.6666, + "p99_ms": 8.415799999999999, + "qps_1t": 168.77358877387718, + "qps_1t_max": 170.49829580671812, + "qps_1t_min": 162.13348828971255, + "qps_1t_runs": [ + 170.49829580671812, + 168.77358877387718, + 162.13348828971255 + ], + "recall": 0.9536842105263158 + } + ], + "system": "rust_default_ivf", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + } + ], + "status": "ok", + "load_before_pct": 8.693749999999996, + "load_contaminated": false, + "wall_s": 200.92344903945923, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/rust_default__glove__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/rust_default__glove__100000__core+mutation+saveload.json new file mode 100644 index 0000000..e215a36 --- /dev/null +++ b/benchmarks/competitive/results/raw/rust_default__glove__100000__core+mutation+saveload.json @@ -0,0 +1,402 @@ +{ + "label": "rust_default", + "runs": [ + { + "all_threads": 32, + "build": { + "first_query_ms": 40.8248, + "index_s": 0.0, + "ingest_s": 0.053535200000000005, + "ready_s": 0.09436, + "total_s": 0.053535200000000005 + }, + "concurrent_clients": { + "clients": 8, + "qps": 694.9927164763313, + "scan_threads": 1 + }, + "d": 100, + "ds": "glove", + "k": 10, + "loop_allthreads": { + "mean_ms": 0.6032887499999995, + "median_ms": 0.5783, + "nq": 2000, + "p95_ms": 0.7379, + "p99_ms": 0.9842, + "qps": 1669.8171942579288, + "qps_runs": [ + 1669.8171942579288, + 1693.179601655797, + 1611.8802018396402 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 127.77734375, + "rss_mb": 127.77734375 + }, + "mem_after_first_query": { + "peak_mb": 127.77734375, + "rss_mb": 127.77734375 + }, + "mem_after_ingest": { + "peak_mb": 83.8984375, + "rss_mb": 78.44140625 + }, + "mem_after_sweep": { + "peak_mb": 128.08203125, + "rss_mb": 127.9609375 + }, + "mem_baseline": { + "peak_mb": 6.5703125, + "rss_mb": 6.5703125 + }, + "n": 100000, + "queries": 2000, + "sweep": [ + { + "knob": null, + "mean_ms": 1.4547141333333304, + "median_ms": 1.4215, + "nq": 2000, + "p95_ms": 1.7947, + "p99_ms": 2.1357, + "qps_1t": 699.5854990897174, + "qps_1t_max": 704.9802519169391, + "qps_1t_min": 659.5239780164837, + "qps_1t_runs": [ + 704.9802519169391, + 659.5239780164837, + 699.5854990897174 + ], + "recall": 1.0 + } + ], + "system": "rust_default_exact", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + }, + { + "all_threads": 32, + "build": { + "cache_s": 0.0408248, + "first_query_ms": 0.0, + "index_s": 0.2123974, + "ingest_s": 0.053535200000000005, + "ready_s": 0.30675739999999996, + "total_s": 0.30675739999999996 + }, + "d": 100, + "ds": "glove", + "index_stats": { + "build_seconds": 0.2123836, + "index_bytes": 1316128, + "nlist": 1265, + "trained_n": 100000 + }, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 150.78515625, + "rss_mb": 131.36328125 + }, + "mem_after_ingest": { + "peak_mb": 83.8984375, + "rss_mb": 78.44140625 + }, + "mem_after_sweep": { + "peak_mb": 150.78515625, + "rss_mb": 131.35546875 + }, + "mem_baseline": { + "peak_mb": 6.5703125, + "rss_mb": 6.5703125 + }, + "mutation": { + "after": { + "mean_ms": 0.7657480499999967, + "median_ms": 0.7532, + "nq": 2000, + "p95_ms": 0.9539000000000001, + "p99_ms": 1.1238000000000001, + "qps_1t": 1314.1168277456336, + "qps_1t_max": 1333.9626079609309, + "qps_1t_min": 1271.2447240165836, + "qps_1t_runs": [ + 1314.1168277456336, + 1333.9626079609309, + 1271.2447240165836 + ], + "recall": 0.9743 + }, + "delete_10k": { + "per_s": 1127217.8010235138, + "seconds": 0.0088714 + }, + "first_query_after_ms": 83.6679, + "index_stats_after": { + "builds": 115046, + "churn": 30000, + "pending": 0, + "trained_n": 100000 + }, + "insert_10k": { + "per_s": 689579.080929001, + "seconds": 0.0145016 + }, + "mem_after": { + "peak_mb": 172.7265625, + "rss_mb": 145.47265625 + }, + "recall_after": 0.9743, + "recall_before": 0.974, + "ref_knob": 256, + "update_10k": { + "per_s": 702020.414753661, + "seconds": 0.0142446 + } + }, + "n": 100000, + "queries": 2000, + "ref_knob": 256, + "ref_recall_pre": 0.974, + "saveload": { + "disk_mb": 121.47405910491943, + "first_query_after_load_ms": 237.37449999999998, + "load_s": 0.4137497, + "note": "load = JSON parse + insert; first query includes the scan-cache build and the IVF index build (the index is never persisted)", + "save_s": 0.7539028, + "time_to_ready_s": 0.6511241999999999 + }, + "sweep": [ + { + "allthreads": { + "median_ms": 0.018199999999999997, + "qps": 53680.17886235612, + "recall": 0.28505 + }, + "knob": 1, + "mean_ms": 0.019093250000000127, + "median_ms": 0.018199999999999997, + "nq": 2000, + "p95_ms": 0.0248, + "p99_ms": 0.0368, + "qps_1t": 52956.284587073395, + "qps_1t_max": 55463.578454618284, + "qps_1t_min": 49100.47922067721, + "qps_1t_runs": [ + 49100.47922067721, + 55463.578454618284, + 52956.284587073395 + ], + "recall": 0.28505 + }, + { + "allthreads": { + "median_ms": 0.0229, + "qps": 42249.89120652995, + "recall": 0.4016 + }, + "knob": 2, + "mean_ms": 0.024383399999999965, + "median_ms": 0.0237, + "nq": 2000, + "p95_ms": 0.0312, + "p99_ms": 0.0383, + "qps_1t": 40791.43543021715, + "qps_1t_max": 41482.75976504172, + "qps_1t_min": 40768.31975408543, + "qps_1t_runs": [ + 41482.75976504172, + 40768.31975408543, + 40791.43543021715 + ], + "recall": 0.4016 + }, + { + "allthreads": { + "median_ms": 0.030000000000000002, + "qps": 32287.59203981703, + "recall": 0.52085 + }, + "knob": 4, + "mean_ms": 0.03110996666666654, + "median_ms": 0.030199999999999998, + "nq": 2000, + "p95_ms": 0.0406, + "p99_ms": 0.050300000000000004, + "qps_1t": 32167.7354396767, + "qps_1t_max": 32233.005953436226, + "qps_1t_min": 32032.032032032013, + "qps_1t_runs": [ + 32167.7354396767, + 32233.005953436226, + 32032.032032032013 + ], + "recall": 0.52085 + }, + { + "allthreads": { + "median_ms": 0.0449, + "qps": 21856.609711984496, + "recall": 0.63505 + }, + "knob": 8, + "mean_ms": 0.04629701666666656, + "median_ms": 0.0451, + "nq": 2000, + "p95_ms": 0.061900000000000004, + "p99_ms": 0.076, + "qps_1t": 21634.90662374299, + "qps_1t_max": 22195.415093104213, + "qps_1t_min": 21001.741044332583, + "qps_1t_runs": [ + 22195.415093104213, + 21001.741044332583, + 21634.90662374299 + ], + "recall": 0.63505 + }, + { + "allthreads": { + "median_ms": 0.0687, + "qps": 14291.34916052615, + "recall": 0.73085 + }, + "knob": 16, + "mean_ms": 0.07220015000000007, + "median_ms": 0.0704, + "nq": 2000, + "p95_ms": 0.09720000000000001, + "p99_ms": 0.1202, + "qps_1t": 13906.494124158584, + "qps_1t_max": 14166.814237648312, + "qps_1t_min": 13494.529992267668, + "qps_1t_runs": [ + 14166.814237648312, + 13906.494124158584, + 13494.529992267668 + ], + "recall": 0.73085 + }, + { + "allthreads": { + "median_ms": 0.118, + "qps": 8302.59298281447, + "recall": 0.8134 + }, + "knob": 32, + "mean_ms": 0.11708581666666681, + "median_ms": 0.1147, + "nq": 2000, + "p95_ms": 0.1538, + "p99_ms": 0.17980000000000002, + "qps_1t": 8541.744788040876, + "qps_1t_max": 8572.763473169398, + "qps_1t_min": 8507.970053647017, + "qps_1t_runs": [ + 8507.970053647017, + 8572.763473169398, + 8541.744788040876 + ], + "recall": 0.8134 + }, + { + "allthreads": { + "median_ms": 0.2125, + "qps": 4573.128207191982, + "recall": 0.88165 + }, + "knob": 64, + "mean_ms": 0.20716886666666656, + "median_ms": 0.205, + "nq": 2000, + "p95_ms": 0.26699999999999996, + "p99_ms": 0.3014, + "qps_1t": 4863.969368666501, + "qps_1t_max": 4874.662673342992, + "qps_1t_min": 4744.490223503442, + "qps_1t_runs": [ + 4874.662673342992, + 4863.969368666501, + 4744.490223503442 + ], + "recall": 0.88165 + }, + { + "allthreads": { + "median_ms": 0.37579999999999997, + "qps": 2638.3887676402624, + "recall": 0.9355 + }, + "knob": 128, + "mean_ms": 0.38995593333333417, + "median_ms": 0.38430000000000003, + "nq": 2000, + "p95_ms": 0.507, + "p99_ms": 0.5695, + "qps_1t": 2594.9842069261135, + "qps_1t_max": 2618.5636523725793, + "qps_1t_min": 2483.730016063527, + "qps_1t_runs": [ + 2618.5636523725793, + 2594.9842069261135, + 2483.730016063527 + ], + "recall": 0.9355 + }, + { + "allthreads": { + "median_ms": 0.48950000000000005, + "qps": 2009.531812245019, + "recall": 0.974 + }, + "knob": 256, + "mean_ms": 0.7395931666666631, + "median_ms": 0.7328, + "nq": 2000, + "p95_ms": 0.9119, + "p99_ms": 1.0108000000000001, + "qps_1t": 1350.3063237410697, + "qps_1t_max": 1358.6289206039664, + "qps_1t_min": 1347.3990112651325, + "qps_1t_runs": [ + 1350.3063237410697, + 1347.3990112651325, + 1358.6289206039664 + ], + "recall": 0.974 + }, + { + "allthreads": { + "median_ms": 0.5703, + "qps": 1703.3327493740856, + "recall": 1.0 + }, + "knob": 1265, + "mean_ms": 1.4638045163464755, + "median_ms": 1.4307, + "nq": 1978, + "p95_ms": 1.7236, + "p99_ms": 2.1279, + "qps_1t": 683.7856683847615, + "qps_1t_max": 693.3252686074593, + "qps_1t_min": 672.6566663455278, + "qps_1t_runs": [ + 672.6566663455278, + 683.7856683847615, + 693.3252686074593 + ], + "recall": 1.0 + } + ], + "system": "rust_default_ivf", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + } + ], + "status": "ok", + "load_before_pct": 5.503125000000001, + "load_contaminated": false, + "wall_s": 50.568493366241455, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/rust_default__glove__100000__filter.json b/benchmarks/competitive/results/raw/rust_default__glove__100000__filter.json new file mode 100644 index 0000000..7b881c4 --- /dev/null +++ b/benchmarks/competitive/results/raw/rust_default__glove__100000__filter.json @@ -0,0 +1,233 @@ +{ + "label": "rust_default", + "runs": [ + { + "d": 100, + "ds": "glove", + "filter": { + "build_with_metadata_s": 2.1289925, + "mem": { + "peak_mb": 149.01953125, + "rss_mb": 130.48828125 + }, + "subsets": { + "f01": [ + { + "knob": 2, + "mean_ms": 0.032043500000000086, + "median_ms": 0.0347, + "nq": 2000, + "p95_ms": 0.039799999999999995, + "p99_ms": 0.0443, + "qps_1t": 31466.677575114914, + "qps_1t_max": 32388.768870506432, + "qps_1t_min": 29872.191827267052, + "qps_1t_runs": [ + 29872.191827267052, + 31466.677575114914, + 32388.768870506432 + ], + "recall": 0.3727 + }, + { + "knob": 8, + "mean_ms": 0.021686716666666755, + "median_ms": 0.0255, + "nq": 2000, + "p95_ms": 0.0294, + "p99_ms": 0.032799999999999996, + "qps_1t": 45984.411284574424, + "qps_1t_max": 46678.69421020812, + "qps_1t_min": 45681.70813043051, + "qps_1t_runs": [ + 45984.411284574424, + 45681.70813043051, + 46678.69421020812 + ], + "recall": 0.3727 + }, + { + "knob": 32, + "mean_ms": 0.016466833333333136, + "median_ms": 0.016399999999999998, + "nq": 2000, + "p95_ms": 0.018199999999999997, + "p99_ms": 0.0209, + "qps_1t": 60890.21494245865, + "qps_1t_max": 61049.688341340894, + "qps_1t_min": 60250.40066516447, + "qps_1t_runs": [ + 60250.40066516447, + 60890.21494245865, + 61049.688341340894 + ], + "recall": 0.4615 + }, + { + "knob": 64, + "mean_ms": 0.02137643333333331, + "median_ms": 0.0212, + "nq": 2000, + "p95_ms": 0.024, + "p99_ms": 0.026699999999999998, + "qps_1t": 46963.34980181477, + "qps_1t_max": 47052.0701734575, + "qps_1t_min": 46332.654559944094, + "qps_1t_runs": [ + 47052.0701734575, + 46332.654559944094, + 46963.34980181477 + ], + "recall": 0.5994 + }, + { + "knob": 1265, + "mean_ms": 0.05480208333333471, + "median_ms": 0.0531, + "nq": 2000, + "p95_ms": 0.0562, + "p99_ms": 0.0608, + "qps_1t": 18088.579775158898, + "qps_1t_max": 18702.96788045809, + "qps_1t_min": 17967.740718314293, + "qps_1t_runs": [ + 18088.579775158898, + 18702.96788045809, + 17967.740718314293 + ], + "recall": 1.0 + }, + { + "knob": "exact", + "mean_ms": 0.053449250000001405, + "median_ms": 0.053, + "nq": 2000, + "p95_ms": 0.055099999999999996, + "p99_ms": 0.0579, + "qps_1t": 18699.435557537647, + "qps_1t_max": 18756.570661159716, + "qps_1t_min": 18672.201107074772, + "qps_1t_runs": [ + 18699.435557537647, + 18672.201107074772, + 18756.570661159716 + ], + "recall": 1.0 + } + ], + "f10": [ + { + "knob": 2, + "mean_ms": 0.014949683333333517, + "median_ms": 0.0138, + "nq": 2000, + "p95_ms": 0.025, + "p99_ms": 0.0285, + "qps_1t": 67053.5657409922, + "qps_1t_max": 70436.42408362214, + "qps_1t_min": 63538.86036700058, + "qps_1t_runs": [ + 63538.86036700058, + 67053.5657409922, + 70436.42408362214 + ], + "recall": 0.266 + }, + { + "knob": 8, + "mean_ms": 0.01640416666666663, + "median_ms": 0.0162, + "nq": 2000, + "p95_ms": 0.018600000000000002, + "p99_ms": 0.020999999999999998, + "qps_1t": 62135.922330096895, + "qps_1t_max": 62710.66865250436, + "qps_1t_min": 58232.63939438042, + "qps_1t_runs": [ + 62135.922330096895, + 58232.63939438042, + 62710.66865250436 + ], + "recall": 0.47545 + }, + { + "knob": 32, + "mean_ms": 0.023570316666666476, + "median_ms": 0.0234, + "nq": 2000, + "p95_ms": 0.0269, + "p99_ms": 0.0294, + "qps_1t": 43076.15433324578, + "qps_1t_max": 43110.788259207904, + "qps_1t_min": 41151.92467551716, + "qps_1t_runs": [ + 43110.788259207904, + 43076.15433324578, + 41151.92467551716 + ], + "recall": 0.7007 + }, + { + "knob": 64, + "mean_ms": 0.03252056666666669, + "median_ms": 0.032, + "nq": 2000, + "p95_ms": 0.0387, + "p99_ms": 0.0458, + "qps_1t": 30830.227187944096, + "qps_1t_max": 31068.39551931605, + "qps_1t_min": 30359.19481343518, + "qps_1t_runs": [ + 30830.227187944096, + 31068.39551931605, + 30359.19481343518 + ], + "recall": 0.797 + }, + { + "knob": 1265, + "mean_ms": 0.12008315000000111, + "median_ms": 0.1185, + "nq": 2000, + "p95_ms": 0.12769999999999998, + "p99_ms": 0.1515, + "qps_1t": 8310.507140387728, + "qps_1t_max": 8379.844964488313, + "qps_1t_min": 8292.843524823385, + "qps_1t_runs": [ + 8292.843524823385, + 8379.844964488313, + 8310.507140387728 + ], + "recall": 1.0 + }, + { + "knob": "exact", + "mean_ms": 0.11908006666666739, + "median_ms": 0.1182, + "nq": 2000, + "p95_ms": 0.127, + "p99_ms": 0.1471, + "qps_1t": 8377.475648772612, + "qps_1t_max": 8441.443605036664, + "qps_1t_min": 8374.553583953015, + "qps_1t_runs": [ + 8377.475648772612, + 8441.443605036664, + 8374.553583953015 + ], + "recall": 1.0 + } + ] + } + }, + "n": 100000, + "system": "rust_default_ivf" + } + ], + "status": "ok", + "load_before_pct": 4.181249999999997, + "load_contaminated": false, + "wall_s": 5.435130596160889, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/rust_default__sift__1000000__core+saveload.json b/benchmarks/competitive/results/raw/rust_default__sift__1000000__core+saveload.json new file mode 100644 index 0000000..d295d3f --- /dev/null +++ b/benchmarks/competitive/results/raw/rust_default__sift__1000000__core+saveload.json @@ -0,0 +1,336 @@ +{ + "label": "rust_default", + "runs": [ + { + "all_threads": 32, + "build": { + "first_query_ms": 840.6548, + "index_s": 0.0, + "ingest_s": 0.8266650999999998, + "ready_s": 1.6673198999999999, + "total_s": 0.8266650999999998 + }, + "concurrent_clients": { + "clients": 8, + "qps": 13.89962490124751, + "scan_threads": 1 + }, + "d": 128, + "ds": "sift", + "k": 10, + "loop_allthreads": { + "mean_ms": 16.292859021406727, + "median_ms": 16.191, + "nq": 218, + "p95_ms": 18.8101, + "p99_ms": 23.867099999999997, + "qps": 61.30097714038762, + "qps_runs": [ + 61.30097714038762, + 60.87214991637757, + 61.96650886430912 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 1574.12890625, + "rss_mb": 1566.5078125 + }, + "mem_after_first_query": { + "peak_mb": 1574.12890625, + "rss_mb": 1566.5078125 + }, + "mem_after_ingest": { + "peak_mb": 1077.75390625, + "rss_mb": 958.04296875 + }, + "mem_after_sweep": { + "peak_mb": 1574.12890625, + "rss_mb": 1566.80859375 + }, + "mem_baseline": { + "peak_mb": 6.99609375, + "rss_mb": 6.99609375 + }, + "n": 1000000, + "queries": 2000, + "sweep": [ + { + "knob": null, + "mean_ms": 30.004278730158727, + "median_ms": 30.2601, + "nq": 105, + "p95_ms": 31.850499999999997, + "p99_ms": 32.3378, + "qps_1t": 33.386588651854005, + "qps_1t_max": 33.850588595646414, + "qps_1t_min": 32.76635984559368, + "qps_1t_runs": [ + 33.850588595646414, + 32.76635984559368, + 33.386588651854005 + ], + "recall": 0.9990476190476191 + } + ], + "system": "rust_default_exact", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + }, + { + "all_threads": 32, + "build": { + "cache_s": 0.8406548, + "first_query_ms": 0.0, + "index_s": 6.3692511, + "ingest_s": 0.8266650999999998, + "ready_s": 8.036570999999999, + "total_s": 8.036570999999999 + }, + "d": 128, + "ds": "sift", + "index_stats": { + "build_seconds": 6.3685774, + "index_bytes": 10080008, + "nlist": 4000, + "trained_n": 1000000 + }, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 1661.16796875, + "rss_mb": 1577.37890625 + }, + "mem_after_ingest": { + "peak_mb": 1077.75390625, + "rss_mb": 958.04296875 + }, + "mem_after_sweep": { + "peak_mb": 1661.16796875, + "rss_mb": 1577.43359375 + }, + "mem_baseline": { + "peak_mb": 6.99609375, + "rss_mb": 6.99609375 + }, + "n": 1000000, + "queries": 2000, + "ref_knob": 64, + "ref_recall_pre": 0.9700922266139658, + "saveload": { + "disk_mb": 1285.1193675994873, + "first_query_after_load_ms": 8389.5167, + "load_s": 11.2341163, + "note": "load = JSON parse + insert; first query includes the scan-cache build and the IVF index build (the index is never persisted)", + "save_s": 22.7026778, + "time_to_ready_s": 19.623632999999998 + }, + "sweep": [ + { + "allthreads": { + "median_ms": 0.1628, + "qps": 6063.900780029878, + "recall": 0.2633 + }, + "knob": 1, + "mean_ms": 0.09371651666666696, + "median_ms": 0.0919, + "nq": 2000, + "p95_ms": 0.1224, + "p99_ms": 0.1487, + "qps_1t": 10886.972539789167, + "qps_1t_max": 10960.039149259854, + "qps_1t_min": 10198.243658477139, + "qps_1t_runs": [ + 10198.243658477139, + 10886.972539789167, + 10960.039149259854 + ], + "recall": 0.2633 + }, + { + "allthreads": { + "median_ms": 0.21350000000000002, + "qps": 4494.123484132142, + "recall": 0.39335 + }, + "knob": 2, + "mean_ms": 0.12690718333333284, + "median_ms": 0.12329999999999999, + "nq": 2000, + "p95_ms": 0.1676, + "p99_ms": 0.2053, + "qps_1t": 7936.621316812469, + "qps_1t_max": 8142.414079211031, + "qps_1t_min": 7580.946503913854, + "qps_1t_runs": [ + 8142.414079211031, + 7936.621316812469, + 7580.946503913854 + ], + "recall": 0.39335 + }, + { + "allthreads": { + "median_ms": 0.314, + "qps": 3000.2271171927664, + "recall": 0.53655 + }, + "knob": 4, + "mean_ms": 0.17329533333333352, + "median_ms": 0.1653, + "nq": 2000, + "p95_ms": 0.2448, + "p99_ms": 0.3079, + "qps_1t": 5762.821846396589, + "qps_1t_max": 5831.696084657568, + "qps_1t_min": 5718.102139027636, + "qps_1t_runs": [ + 5831.696084657568, + 5718.102139027636, + 5762.821846396589 + ], + "recall": 0.53655 + }, + { + "allthreads": { + "median_ms": 0.49229999999999996, + "qps": 1973.826274075255, + "recall": 0.68765 + }, + "knob": 8, + "mean_ms": 0.29107343333333235, + "median_ms": 0.2767, + "nq": 2000, + "p95_ms": 0.4417, + "p99_ms": 0.545, + "qps_1t": 3348.591749739646, + "qps_1t_max": 3667.8444115072025, + "qps_1t_min": 3311.8329140552855, + "qps_1t_runs": [ + 3667.8444115072025, + 3348.591749739646, + 3311.8329140552855 + ], + "recall": 0.68765 + }, + { + "allthreads": { + "median_ms": 0.7952, + "qps": 1223.8839983164276, + "recall": 0.82165 + }, + "knob": 16, + "mean_ms": 0.47877408333333255, + "median_ms": 0.4527, + "nq": 2000, + "p95_ms": 0.7337, + "p99_ms": 0.8621, + "qps_1t": 2089.486653874136, + "qps_1t_max": 2111.0962278827087, + "qps_1t_min": 2065.909751765446, + "qps_1t_runs": [ + 2065.909751765446, + 2111.0962278827087, + 2089.486653874136 + ], + "recall": 0.82165 + }, + { + "allthreads": { + "median_ms": 1.5816999999999999, + "qps": 600.1781508805259, + "recall": 0.9163 + }, + "knob": 32, + "mean_ms": 0.9342941666666669, + "median_ms": 0.8911, + "nq": 2000, + "p95_ms": 1.407, + "p99_ms": 1.6033, + "qps_1t": 1077.7035581299028, + "qps_1t_max": 1088.24579728995, + "qps_1t_min": 1045.9446893988809, + "qps_1t_runs": [ + 1077.7035581299028, + 1088.24579728995, + 1045.9446893988809 + ], + "recall": 0.9163 + }, + { + "allthreads": { + "median_ms": 2.5244999999999997, + "qps": 381.9570133889485, + "recall": 0.9689274447949526 + }, + "knob": 64, + "mean_ms": 1.8168356609573981, + "median_ms": 1.7577, + "nq": 1518, + "p95_ms": 2.6174000000000004, + "p99_ms": 2.8911000000000002, + "qps_1t": 546.4091219943318, + "qps_1t_max": 564.8487900161236, + "qps_1t_min": 540.5431197798689, + "qps_1t_runs": [ + 546.4091219943318, + 540.5431197798689, + 564.8487900161236 + ], + "recall": 0.9700922266139658 + }, + { + "allthreads": { + "median_ms": 3.0686, + "qps": 289.5903006952332, + "recall": 0.9923736075407027 + }, + "knob": 128, + "mean_ms": 3.3970247534516775, + "median_ms": 3.3392, + "nq": 1014, + "p95_ms": 4.5659, + "p99_ms": 5.0150999999999994, + "qps_1t": 293.9971096082981, + "qps_1t_max": 295.42758510499584, + "qps_1t_min": 293.7068017107755, + "qps_1t_runs": [ + 295.42758510499584, + 293.7068017107755, + 293.9971096082981 + ], + "recall": 0.9924063116370808 + }, + { + "allthreads": { + "median_ms": 3.9513, + "qps": 239.0854404798855, + "recall": 0.9989942528735632 + }, + "knob": 256, + "mean_ms": 6.336291766268254, + "median_ms": 6.3290999999999995, + "nq": 502, + "p95_ms": 7.865, + "p99_ms": 8.5034, + "qps_1t": 157.06441969014625, + "qps_1t_max": 161.9820876851634, + "qps_1t_min": 154.59441831001496, + "qps_1t_runs": [ + 154.59441831001496, + 157.06441969014625, + 161.9820876851634 + ], + "recall": 0.998804780876494 + } + ], + "system": "rust_default_ivf", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + } + ], + "status": "ok", + "load_before_pct": 7.599999999999997, + "load_contaminated": false, + "wall_s": 201.8282823562622, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/rust_default__sift__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/rust_default__sift__100000__core+mutation+saveload.json new file mode 100644 index 0000000..eb4ce16 --- /dev/null +++ b/benchmarks/competitive/results/raw/rust_default__sift__100000__core+mutation+saveload.json @@ -0,0 +1,402 @@ +{ + "label": "rust_default", + "runs": [ + { + "all_threads": 32, + "build": { + "first_query_ms": 50.649899999999995, + "index_s": 0.0, + "ingest_s": 0.0554142, + "ready_s": 0.1060641, + "total_s": 0.0554142 + }, + "concurrent_clients": { + "clients": 8, + "qps": 133.15930300159712, + "scan_threads": 1 + }, + "d": 128, + "ds": "sift", + "k": 10, + "loop_allthreads": { + "mean_ms": 3.08890448999512, + "median_ms": 2.8602, + "nq": 683, + "p95_ms": 4.7961, + "p99_ms": 6.5978, + "qps": 335.9577553044066, + "qps_runs": [ + 335.9577553044066, + 299.33044643799354, + 339.05668968280753 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 149.80078125, + "rss_mb": 149.80078125 + }, + "mem_after_first_query": { + "peak_mb": 149.80078125, + "rss_mb": 149.80078125 + }, + "mem_after_ingest": { + "peak_mb": 97.25, + "rss_mb": 89.6328125 + }, + "mem_after_sweep": { + "peak_mb": 150.11328125, + "rss_mb": 149.99609375 + }, + "mem_baseline": { + "peak_mb": 7.01171875, + "rss_mb": 7.01171875 + }, + "n": 100000, + "queries": 2000, + "sweep": [ + { + "knob": null, + "mean_ms": 2.807940888166804, + "median_ms": 2.8586, + "nq": 1231, + "p95_ms": 3.3598999999999997, + "p99_ms": 3.8371, + "qps_1t": 343.6908042496044, + "qps_1t_max": 397.7487999521281, + "qps_1t_min": 333.32432554354506, + "qps_1t_runs": [ + 397.7487999521281, + 343.6908042496044, + 333.32432554354506 + ], + "recall": 1.0 + } + ], + "system": "rust_default_exact", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + }, + { + "all_threads": 32, + "build": { + "cache_s": 0.0506499, + "first_query_ms": 0.0, + "index_s": 0.5648412, + "ingest_s": 0.0554142, + "ready_s": 0.6709053, + "total_s": 0.6709053 + }, + "d": 128, + "ds": "sift", + "index_stats": { + "build_seconds": 0.5646897, + "index_bytes": 1457808, + "nlist": 1265, + "trained_n": 100000 + }, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 178.62890625, + "rss_mb": 152.56640625 + }, + "mem_after_ingest": { + "peak_mb": 97.25, + "rss_mb": 89.6328125 + }, + "mem_after_sweep": { + "peak_mb": 178.62890625, + "rss_mb": 152.55859375 + }, + "mem_baseline": { + "peak_mb": 7.01171875, + "rss_mb": 7.01171875 + }, + "mutation": { + "after": { + "mean_ms": 0.3573184499999988, + "median_ms": 0.3478, + "nq": 2000, + "p95_ms": 0.4643, + "p99_ms": 0.5712999999999999, + "qps_1t": 2802.712577340857, + "qps_1t_max": 2807.7371126972325, + "qps_1t_min": 2785.519587843383, + "qps_1t_runs": [ + 2802.712577340857, + 2807.7371126972325, + 2785.519587843383 + ], + "recall": 0.98545 + }, + "delete_10k": { + "per_s": 463419.9465213382, + "seconds": 0.0215787 + }, + "first_query_after_ms": 260.6058, + "index_stats_after": { + "builds": 114242, + "churn": 30000, + "pending": 0, + "trained_n": 100000 + }, + "insert_10k": { + "per_s": 253924.40162714757, + "seconds": 0.0393818 + }, + "mem_after": { + "peak_mb": 206.8515625, + "rss_mb": 170.859375 + }, + "recall_after": 0.98545, + "recall_before": 0.98605, + "ref_knob": 64, + "update_10k": { + "per_s": 304479.81146610074, + "seconds": 0.0328429 + } + }, + "n": 100000, + "queries": 2000, + "ref_knob": 64, + "ref_recall_pre": 0.98605, + "saveload": { + "disk_mb": 127.68917655944824, + "first_query_after_load_ms": 852.5371, + "load_s": 0.9036185, + "note": "load = JSON parse + insert; first query includes the scan-cache build and the IVF index build (the index is never persisted)", + "save_s": 1.6734498, + "time_to_ready_s": 1.7561556 + }, + "sweep": [ + { + "allthreads": { + "median_ms": 0.0582, + "qps": 16984.315833543515, + "recall": 0.2912 + }, + "knob": 1, + "mean_ms": 0.029398816666666765, + "median_ms": 0.0286, + "nq": 2000, + "p95_ms": 0.037700000000000004, + "p99_ms": 0.0449, + "qps_1t": 33756.41371860651, + "qps_1t_max": 34626.09872939535, + "qps_1t_min": 33678.53835143551, + "qps_1t_runs": [ + 34626.09872939535, + 33756.41371860651, + 33678.53835143551 + ], + "recall": 0.2912 + }, + { + "allthreads": { + "median_ms": 0.06949999999999999, + "qps": 14002.89579885121, + "recall": 0.43565 + }, + "knob": 2, + "mean_ms": 0.03880953333333346, + "median_ms": 0.0376, + "nq": 2000, + "p95_ms": 0.050899999999999994, + "p99_ms": 0.0599, + "qps_1t": 25888.76117100045, + "qps_1t_max": 26284.282918764464, + "qps_1t_min": 25153.27778651157, + "qps_1t_runs": [ + 25153.27778651157, + 25888.76117100045, + 26284.282918764464 + ], + "recall": 0.43565 + }, + { + "allthreads": { + "median_ms": 0.09519999999999999, + "qps": 10165.375409855225, + "recall": 0.5955 + }, + "knob": 4, + "mean_ms": 0.04875518333333353, + "median_ms": 0.0478, + "nq": 2000, + "p95_ms": 0.066, + "p99_ms": 0.0786, + "qps_1t": 20026.434894060145, + "qps_1t_max": 22804.185024035596, + "qps_1t_min": 19054.896203216656, + "qps_1t_runs": [ + 22804.185024035596, + 20026.434894060145, + 19054.896203216656 + ], + "recall": 0.5955 + }, + { + "allthreads": { + "median_ms": 0.1365, + "qps": 6767.7451971004275, + "recall": 0.74745 + }, + "knob": 8, + "mean_ms": 0.07386681666666645, + "median_ms": 0.07200000000000001, + "nq": 2000, + "p95_ms": 0.1002, + "p99_ms": 0.1166, + "qps_1t": 13524.515889953715, + "qps_1t_max": 13580.70128025269, + "qps_1t_min": 13508.630326199658, + "qps_1t_runs": [ + 13580.70128025269, + 13508.630326199658, + 13524.515889953715 + ], + "recall": 0.74745 + }, + { + "allthreads": { + "median_ms": 0.22, + "qps": 4404.844888893299, + "recall": 0.87095 + }, + "knob": 16, + "mean_ms": 0.11963656666666658, + "median_ms": 0.1164, + "nq": 2000, + "p95_ms": 0.16210000000000002, + "p99_ms": 0.1908, + "qps_1t": 8412.816926587659, + "qps_1t_max": 8451.160111876434, + "qps_1t_min": 8215.81297523342, + "qps_1t_runs": [ + 8215.81297523342, + 8412.816926587659, + 8451.160111876434 + ], + "recall": 0.87095 + }, + { + "allthreads": { + "median_ms": 0.3655, + "qps": 2647.667709229952, + "recall": 0.94865 + }, + "knob": 32, + "mean_ms": 0.203042766666666, + "median_ms": 0.1993, + "nq": 2000, + "p95_ms": 0.2758, + "p99_ms": 0.332, + "qps_1t": 4980.120603580655, + "qps_1t_max": 5131.027193161364, + "qps_1t_min": 4685.218884055826, + "qps_1t_runs": [ + 5131.027193161364, + 4685.218884055826, + 4980.120603580655 + ], + "recall": 0.94865 + }, + { + "allthreads": { + "median_ms": 0.6529, + "qps": 1464.4384552165918, + "recall": 0.98605 + }, + "knob": 64, + "mean_ms": 0.3729039833333329, + "median_ms": 0.3651, + "nq": 2000, + "p95_ms": 0.5085, + "p99_ms": 0.6264000000000001, + "qps_1t": 2725.124585883258, + "qps_1t_max": 2746.5966577489226, + "qps_1t_min": 2579.5181614845446, + "qps_1t_runs": [ + 2746.5966577489226, + 2579.5181614845446, + 2725.124585883258 + ], + "recall": 0.98605 + }, + { + "allthreads": { + "median_ms": 1.3495, + "qps": 738.048748931721, + "recall": 0.9984 + }, + "knob": 128, + "mean_ms": 0.6693367500000026, + "median_ms": 0.6657, + "nq": 2000, + "p95_ms": 0.8558, + "p99_ms": 0.9892, + "qps_1t": 1490.0364984440298, + "qps_1t_max": 1517.2565937316394, + "qps_1t_min": 1475.358307362724, + "qps_1t_runs": [ + 1490.0364984440298, + 1517.2565937316394, + 1475.358307362724 + ], + "recall": 0.9984 + }, + { + "allthreads": { + "median_ms": 1.8943, + "qps": 468.1828655235003, + "recall": 0.9998267898383372 + }, + "knob": 256, + "mean_ms": 1.2183222833333338, + "median_ms": 1.2026000000000001, + "nq": 2000, + "p95_ms": 1.502, + "p99_ms": 1.6647, + "qps_1t": 826.939198261772, + "qps_1t_max": 836.6879640009979, + "qps_1t_min": 799.6805116419888, + "qps_1t_runs": [ + 799.6805116419888, + 826.939198261772, + 836.6879640009979 + ], + "recall": 0.9998 + }, + { + "allthreads": { + "median_ms": 2.5709, + "qps": 378.46356182338485, + "recall": 1.0 + }, + "knob": 1265, + "mean_ms": 2.9812179288025864, + "median_ms": 2.9638, + "nq": 1030, + "p95_ms": 3.4282000000000004, + "p99_ms": 3.8087, + "qps_1t": 334.57601796134, + "qps_1t_max": 339.145519242327, + "qps_1t_min": 332.6448016547239, + "qps_1t_runs": [ + 334.57601796134, + 339.145519242327, + 332.6448016547239 + ], + "recall": 1.0 + } + ], + "system": "rust_default_ivf", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + } + ], + "status": "ok", + "load_before_pct": 10.334375, + "load_contaminated": false, + "wall_s": 91.82787585258484, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/rust_default__text384__1000000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/rust_default__text384__1000000__core+mutation+saveload.json new file mode 100644 index 0000000..8c8e30f --- /dev/null +++ b/benchmarks/competitive/results/raw/rust_default__text384__1000000__core+mutation+saveload.json @@ -0,0 +1,380 @@ +{ + "label": "rust_default", + "runs": [ + { + "all_threads": 32, + "build": { + "first_query_ms": 1029.9592, + "index_s": 0.0, + "ingest_s": 0.8120639999999997, + "ready_s": 1.8420231999999996, + "total_s": 0.8120639999999997 + }, + "concurrent_clients": { + "clients": 8, + "qps": 14.981631115724205, + "scan_threads": 1 + }, + "d": 384, + "ds": "text384", + "k": 10, + "loop_allthreads": { + "mean_ms": 29.137532362459545, + "median_ms": 28.7247, + "nq": 103, + "p95_ms": 31.866699999999998, + "p99_ms": 33.9555, + "qps": 34.190669346418446, + "qps_runs": [ + 34.72379312737248, + 34.190669346418446, + 34.05280744073018 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 3535.3359375, + "rss_mb": 3527.71484375 + }, + "mem_after_first_query": { + "peak_mb": 3535.3359375, + "rss_mb": 3527.71484375 + }, + "mem_after_ingest": { + "peak_mb": 1983.859375, + "rss_mb": 1942.61328125 + }, + "mem_after_sweep": { + "peak_mb": 3535.3359375, + "rss_mb": 3527.93359375 + }, + "mem_baseline": { + "peak_mb": 10.8515625, + "rss_mb": 10.8515625 + }, + "n": 1000000, + "queries": 2000, + "sweep": [ + { + "knob": null, + "mean_ms": 66.36352753623184, + "median_ms": 65.3193, + "nq": 46, + "p95_ms": 70.0012, + "p99_ms": 80.529, + "qps_1t": 15.0615107186387, + "qps_1t_max": 15.089622846686218, + "qps_1t_min": 15.05446591222192, + "qps_1t_runs": [ + 15.05446591222192, + 15.0615107186387, + 15.089622846686218 + ], + "recall": 1.0 + } + ], + "system": "rust_default_exact", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + }, + { + "all_threads": 32, + "build": { + "cache_s": 1.0299592, + "first_query_ms": 0.0, + "index_s": 5.7057285, + "ingest_s": 0.8120639999999997, + "ready_s": 7.5477517, + "total_s": 7.5477517 + }, + "d": 384, + "ds": "text384", + "index_stats": { + "build_seconds": 5.7046481, + "index_bytes": 14176008, + "nlist": 4000, + "trained_n": 1000000 + }, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 3790.33203125, + "rss_mb": 3542.54296875 + }, + "mem_after_ingest": { + "peak_mb": 1983.859375, + "rss_mb": 1942.61328125 + }, + "mem_after_sweep": { + "peak_mb": 3790.33203125, + "rss_mb": 3542.91796875 + }, + "mem_baseline": { + "peak_mb": 10.8515625, + "rss_mb": 10.8515625 + }, + "mutation": { + "after": { + "mean_ms": 9.852475945017186, + "median_ms": 9.7885, + "nq": 291, + "p95_ms": 11.424999999999999, + "p99_ms": 11.9871, + "qps_1t": 100.85102321507371, + "qps_1t_max": 102.88985425422318, + "qps_1t_min": 100.77921728903141, + "qps_1t_runs": [ + 100.77921728903141, + 100.85102321507371, + 102.88985425422318 + ], + "recall": 0.952233676975945 + }, + "delete_10k": { + "per_s": 591194.7454611023, + "seconds": 0.0169149 + }, + "first_query_after_ms": 611.4287, + "index_stats_after": { + "builds": 93416, + "churn": 30000, + "pending": 0, + "trained_n": 1000000 + }, + "insert_10k": { + "per_s": 35306.14665890873, + "seconds": 0.2832368 + }, + "mem_after": { + "peak_mb": 5023.3984375, + "rss_mb": 3591.1953125 + }, + "recall_after": 0.952233676975945, + "recall_before": 0.9512195121951219, + "ref_knob": 256, + "update_10k": { + "per_s": 428753.95525523723, + "seconds": 0.0233234 + } + }, + "n": 1000000, + "queries": 2000, + "ref_knob": 256, + "ref_recall_pre": 0.9512195121951219, + "saveload": { + "disk_mb": 4534.105839729309, + "first_query_after_load_ms": 6601.3625999999995, + "load_s": 13.0069672, + "note": "load = JSON parse + insert; first query includes the scan-cache build and the IVF index build (the index is never persisted)", + "save_s": 21.4794608, + "time_to_ready_s": 19.6083298 + }, + "sweep": [ + { + "allthreads": { + "median_ms": 0.1698, + "qps": 5692.130316769877, + "recall": 0.27565 + }, + "knob": 1, + "mean_ms": 0.1834446999999998, + "median_ms": 0.1754, + "nq": 2000, + "p95_ms": 0.24830000000000002, + "p99_ms": 0.3236, + "qps_1t": 5323.396326856535, + "qps_1t_max": 5828.839055553209, + "qps_1t_min": 5237.704031486981, + "qps_1t_runs": [ + 5323.396326856535, + 5237.704031486981, + 5828.839055553209 + ], + "recall": 0.27565 + }, + { + "allthreads": { + "median_ms": 0.2248, + "qps": 4320.463464756792, + "recall": 0.3806 + }, + "knob": 2, + "mean_ms": 0.23702303333333366, + "median_ms": 0.2286, + "nq": 2000, + "p95_ms": 0.3297, + "p99_ms": 0.4081, + "qps_1t": 4242.414986246092, + "qps_1t_max": 4392.364664656133, + "qps_1t_min": 4037.3617455937197, + "qps_1t_runs": [ + 4242.414986246092, + 4037.3617455937197, + 4392.364664656133 + ], + "recall": 0.3806 + }, + { + "allthreads": { + "median_ms": 0.3305, + "qps": 2997.2870056668235, + "recall": 0.49615 + }, + "knob": 4, + "mean_ms": 0.3279156166666667, + "median_ms": 0.3226, + "nq": 2000, + "p95_ms": 0.4365, + "p99_ms": 0.5088999999999999, + "qps_1t": 3079.6042277422785, + "qps_1t_max": 3091.231197199843, + "qps_1t_min": 2980.323013368833, + "qps_1t_runs": [ + 3091.231197199843, + 3079.6042277422785, + 2980.323013368833 + ], + "recall": 0.49615 + }, + { + "allthreads": { + "median_ms": 0.5065999999999999, + "qps": 1970.3886059520344, + "recall": 0.60455 + }, + "knob": 8, + "mean_ms": 0.515896800000001, + "median_ms": 0.5069, + "nq": 2000, + "p95_ms": 0.6728, + "p99_ms": 0.791, + "qps_1t": 1935.9401283412142, + "qps_1t_max": 1992.9153850975156, + "qps_1t_min": 1889.044871521808, + "qps_1t_runs": [ + 1889.044871521808, + 1992.9153850975156, + 1935.9401283412142 + ], + "recall": 0.60455 + }, + { + "allthreads": { + "median_ms": 0.8159000000000001, + "qps": 1232.5509305451378, + "recall": 0.70215 + }, + "knob": 16, + "mean_ms": 0.8710186166666669, + "median_ms": 0.8673, + "nq": 2000, + "p95_ms": 1.0901999999999998, + "p99_ms": 1.2177, + "qps_1t": 1147.0627994514973, + "qps_1t_max": 1157.0312383392954, + "qps_1t_min": 1140.2728285185901, + "qps_1t_runs": [ + 1147.0627994514973, + 1157.0312383392954, + 1140.2728285185901 + ], + "recall": 0.70215 + }, + { + "allthreads": { + "median_ms": 0.88, + "qps": 1130.7853309777547, + "recall": 0.79095 + }, + "knob": 32, + "mean_ms": 1.5408190379058335, + "median_ms": 1.5315, + "nq": 1961, + "p95_ms": 1.8802999999999999, + "p99_ms": 2.0486, + "qps_1t": 648.3679016030798, + "qps_1t_max": 651.969589824354, + "qps_1t_min": 646.7012629307287, + "qps_1t_runs": [ + 648.3679016030798, + 646.7012629307287, + 651.969589824354 + ], + "recall": 0.7901580826109128 + }, + { + "allthreads": { + "median_ms": 1.1032, + "qps": 866.8276249989277, + "recall": 0.8621 + }, + "knob": 64, + "mean_ms": 2.8165679593721156, + "median_ms": 2.7748, + "nq": 1083, + "p95_ms": 3.3748, + "p99_ms": 3.9991999999999996, + "qps_1t": 355.70602719722376, + "qps_1t_max": 356.3128235767884, + "qps_1t_min": 353.1234288982718, + "qps_1t_runs": [ + 356.3128235767884, + 355.70602719722376, + 353.1234288982718 + ], + "recall": 0.8605724838411819 + }, + { + "allthreads": { + "median_ms": 1.6937, + "qps": 579.1982003109285, + "recall": 0.9145584725536993 + }, + "knob": 128, + "mean_ms": 5.220931326644365, + "median_ms": 5.1327, + "nq": 598, + "p95_ms": 6.2302, + "p99_ms": 7.188899999999999, + "qps_1t": 191.55693278654812, + "qps_1t_max": 193.13226821640987, + "qps_1t_min": 189.94740521521246, + "qps_1t_runs": [ + 189.94740521521246, + 191.55693278654812, + 193.13226821640987 + ], + "recall": 0.9148829431438127 + }, + { + "allthreads": { + "median_ms": 2.7998, + "qps": 350.32285542751816, + "recall": 0.9523489932885906 + }, + "knob": 256, + "mean_ms": 9.970505691056907, + "median_ms": 9.888, + "nq": 287, + "p95_ms": 11.433499999999999, + "p99_ms": 12.3086, + "qps_1t": 100.31053976122804, + "qps_1t_max": 100.39728640477311, + "qps_1t_min": 100.17985949896871, + "qps_1t_runs": [ + 100.31053976122804, + 100.17985949896871, + 100.39728640477311 + ], + "recall": 0.9512195121951219 + } + ], + "system": "rust_default_ivf", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + } + ], + "status": "ok", + "load_before_pct": 5.3343750000000005, + "load_contaminated": false, + "wall_s": 204.11415839195251, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/rust_default__text384__1000000__filter.json b/benchmarks/competitive/results/raw/rust_default__text384__1000000__filter.json new file mode 100644 index 0000000..2bc0a51 --- /dev/null +++ b/benchmarks/competitive/results/raw/rust_default__text384__1000000__filter.json @@ -0,0 +1,233 @@ +{ + "label": "rust_default", + "runs": [ + { + "d": 384, + "ds": "text384", + "filter": { + "build_with_metadata_s": 69.5115023, + "mem": { + "peak_mb": 3789.87109375, + "rss_mb": 3542.55859375 + }, + "subsets": { + "f01": [ + { + "knob": 2, + "mean_ms": 0.22849560000000035, + "median_ms": 0.2512, + "nq": 2000, + "p95_ms": 0.2949, + "p99_ms": 0.3957, + "qps_1t": 4343.88637088276, + "qps_1t_max": 4532.608720285908, + "qps_1t_min": 4261.580952778712, + "qps_1t_runs": [ + 4343.88637088276, + 4261.580952778712, + 4532.608720285908 + ], + "recall": 0.3293 + }, + { + "knob": 8, + "mean_ms": 0.14312516666666655, + "median_ms": 0.1407, + "nq": 2000, + "p95_ms": 0.1632, + "p99_ms": 0.2797, + "qps_1t": 7008.936393902229, + "qps_1t_max": 7067.087865103409, + "qps_1t_min": 6887.0760574244505, + "qps_1t_runs": [ + 7008.936393902229, + 6887.0760574244505, + 7067.087865103409 + ], + "recall": 0.37265 + }, + { + "knob": 16, + "mean_ms": 0.15801510000000052, + "median_ms": 0.1568, + "nq": 2000, + "p95_ms": 0.18159999999999998, + "p99_ms": 0.20359999999999998, + "qps_1t": 6308.2169572441635, + "qps_1t_max": 6388.04566686086, + "qps_1t_min": 6290.119166307609, + "qps_1t_runs": [ + 6388.04566686086, + 6308.2169572441635, + 6290.119166307609 + ], + "recall": 0.49165 + }, + { + "knob": 64, + "mean_ms": 0.2480489333333339, + "median_ms": 0.2409, + "nq": 2000, + "p95_ms": 0.3098, + "p99_ms": 0.375, + "qps_1t": 4115.572689141949, + "qps_1t_max": 4164.387532073587, + "qps_1t_min": 3830.8906935789278, + "qps_1t_runs": [ + 4164.387532073587, + 4115.572689141949, + 3830.8906935789278 + ], + "recall": 0.7302 + }, + { + "knob": 256, + "mean_ms": 0.5232255500000014, + "median_ms": 0.5025000000000001, + "nq": 2000, + "p95_ms": 0.6904, + "p99_ms": 0.9068999999999999, + "qps_1t": 1944.2501814471507, + "qps_1t_max": 2027.054077241906, + "qps_1t_min": 1779.3189051921954, + "qps_1t_runs": [ + 2027.054077241906, + 1944.2501814471507, + 1779.3189051921954 + ], + "recall": 0.9062 + }, + { + "knob": "exact", + "mean_ms": 0.9152129166666687, + "median_ms": 0.8466, + "nq": 2000, + "p95_ms": 1.3227, + "p99_ms": 1.6739000000000002, + "qps_1t": 1079.5255722605523, + "qps_1t_max": 1192.1108014537072, + "qps_1t_min": 1019.9318134785411, + "qps_1t_runs": [ + 1019.9318134785411, + 1079.5255722605523, + 1192.1108014537072 + ], + "recall": 1.0 + } + ], + "f10": [ + { + "knob": 2, + "mean_ms": 0.13679368333333372, + "median_ms": 0.136, + "nq": 2000, + "p95_ms": 0.1553, + "p99_ms": 0.1747, + "qps_1t": 7286.338152395938, + "qps_1t_max": 7503.403731517722, + "qps_1t_min": 7149.748990187306, + "qps_1t_runs": [ + 7149.748990187306, + 7286.338152395938, + 7503.403731517722 + ], + "recall": 0.31865 + }, + { + "knob": 8, + "mean_ms": 0.1785827333333338, + "median_ms": 0.1762, + "nq": 2000, + "p95_ms": 0.2102, + "p99_ms": 0.2475, + "qps_1t": 5638.922322999166, + "qps_1t_max": 5778.755157900156, + "qps_1t_min": 5394.857729509385, + "qps_1t_runs": [ + 5394.857729509385, + 5778.755157900156, + 5638.922322999166 + ], + "recall": 0.5444 + }, + { + "knob": 16, + "mean_ms": 0.2260592333333335, + "median_ms": 0.2223, + "nq": 2000, + "p95_ms": 0.2692, + "p99_ms": 0.3067, + "qps_1t": 4413.919737283497, + "qps_1t_max": 4551.118619445478, + "qps_1t_min": 4312.287215577705, + "qps_1t_runs": [ + 4551.118619445478, + 4413.919737283497, + 4312.287215577705 + ], + "recall": 0.6526 + }, + { + "knob": 64, + "mean_ms": 0.5247319166666653, + "median_ms": 0.5024, + "nq": 2000, + "p95_ms": 0.6893999999999999, + "p99_ms": 0.9543, + "qps_1t": 1900.8400192212903, + "qps_1t_max": 2024.7858080332992, + "qps_1t_min": 1804.2948170819445, + "qps_1t_runs": [ + 1804.2948170819445, + 1900.8400192212903, + 2024.7858080332992 + ], + "recall": 0.828 + }, + { + "knob": 256, + "mean_ms": 1.5236947868116706, + "median_ms": 1.5024000000000002, + "nq": 1931, + "p95_ms": 1.8276, + "p99_ms": 2.1075999999999997, + "qps_1t": 649.9776681675847, + "qps_1t_max": 674.8989425336649, + "qps_1t_min": 644.8007637432971, + "qps_1t_runs": [ + 674.8989425336649, + 644.8007637432971, + 649.9776681675847 + ], + "recall": 0.9400310719834283 + }, + { + "knob": "exact", + "mean_ms": 7.103582568807335, + "median_ms": 6.963699999999999, + "nq": 436, + "p95_ms": 7.9895000000000005, + "p99_ms": 8.9993, + "qps_1t": 140.65354605655205, + "qps_1t_max": 141.26098628161424, + "qps_1t_min": 140.41030467361486, + "qps_1t_runs": [ + 140.41030467361486, + 140.65354605655205, + 141.26098628161424 + ], + "recall": 1.0 + } + ] + } + }, + "n": 1000000, + "system": "rust_default_ivf" + } + ], + "status": "ok", + "load_before_pct": 8.109375000000007, + "load_contaminated": false, + "wall_s": 110.20223021507263, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/rust_default__text384__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/rust_default__text384__100000__core+mutation+saveload.json new file mode 100644 index 0000000..3617310 --- /dev/null +++ b/benchmarks/competitive/results/raw/rust_default__text384__100000__core+mutation+saveload.json @@ -0,0 +1,402 @@ +{ + "label": "rust_default", + "runs": [ + { + "all_threads": 32, + "build": { + "first_query_ms": 79.9521, + "index_s": 0.0, + "ingest_s": 0.0558613, + "ready_s": 0.1358134, + "total_s": 0.0558613 + }, + "concurrent_clients": { + "clients": 8, + "qps": 153.12487092051896, + "scan_threads": 1 + }, + "d": 384, + "ds": "text384", + "k": 10, + "loop_allthreads": { + "mean_ms": 3.3514406727828745, + "median_ms": 3.21, + "nq": 981, + "p95_ms": 4.2322999999999995, + "p99_ms": 5.5107, + "qps": 298.7845954337795, + "qps_runs": [ + 296.9535651803086, + 298.7845954337795, + 299.41021681053775 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 349.43359375, + "rss_mb": 349.43359375 + }, + "mem_after_first_query": { + "peak_mb": 349.43359375, + "rss_mb": 349.43359375 + }, + "mem_after_ingest": { + "peak_mb": 218.80078125, + "rss_mb": 191.67578125 + }, + "mem_after_sweep": { + "peak_mb": 349.7421875, + "rss_mb": 349.6171875 + }, + "mem_baseline": { + "peak_mb": 10.84375, + "rss_mb": 10.84375 + }, + "n": 100000, + "queries": 2000, + "sweep": [ + { + "knob": null, + "mean_ms": 6.48268382352941, + "median_ms": 6.4139, + "nq": 476, + "p95_ms": 6.9882, + "p99_ms": 7.859, + "qps_1t": 154.3898916915987, + "qps_1t_max": 154.44166773899806, + "qps_1t_min": 153.9407224082359, + "qps_1t_runs": [ + 154.44166773899806, + 153.9407224082359, + 154.3898916915987 + ], + "recall": 1.0 + } + ], + "system": "rust_default_exact", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + }, + { + "all_threads": 32, + "build": { + "cache_s": 0.0799521, + "first_query_ms": 0.0, + "index_s": 0.4803373, + "ingest_s": 0.0558613, + "ready_s": 0.6161507, + "total_s": 0.6161507 + }, + "d": 384, + "ds": "text384", + "index_stats": { + "build_seconds": 0.4801689, + "index_bytes": 2753168, + "nlist": 1265, + "trained_n": 100000 + }, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 430.984375, + "rss_mb": 353.2890625 + }, + "mem_after_ingest": { + "peak_mb": 218.80078125, + "rss_mb": 191.67578125 + }, + "mem_after_sweep": { + "peak_mb": 430.984375, + "rss_mb": 353.4296875 + }, + "mem_baseline": { + "peak_mb": 10.84375, + "rss_mb": 10.84375 + }, + "mutation": { + "after": { + "mean_ms": 1.2855967499999983, + "median_ms": 1.2834, + "nq": 2000, + "p95_ms": 1.5433000000000001, + "p99_ms": 1.6485, + "qps_1t": 777.8958710142853, + "qps_1t_max": 781.6779992388792, + "qps_1t_min": 774.0105822726825, + "qps_1t_runs": [ + 777.8958710142853, + 781.6779992388792, + 774.0105822726825 + ], + "recall": 0.94895 + }, + "delete_10k": { + "per_s": 1037086.2026051604, + "seconds": 0.0096424 + }, + "first_query_after_ms": 191.6616, + "index_stats_after": { + "builds": 110999, + "churn": 30000, + "pending": 0, + "trained_n": 100000 + }, + "insert_10k": { + "per_s": 228709.9614623715, + "seconds": 0.0437235 + }, + "mem_after": { + "peak_mb": 515.1328125, + "rss_mb": 400.9921875 + }, + "recall_after": 0.94895, + "recall_before": 0.9529, + "ref_knob": 128, + "update_10k": { + "per_s": 579979.120751653, + "seconds": 0.017242 + } + }, + "n": 100000, + "queries": 2000, + "ref_knob": 128, + "ref_recall_pre": 0.9529, + "saveload": { + "disk_mb": 453.31724739074707, + "first_query_after_load_ms": 546.3891, + "load_s": 1.2547969, + "note": "load = JSON parse + insert; first query includes the scan-cache build and the IVF index build (the index is never persisted)", + "save_s": 1.9371505999999998, + "time_to_ready_s": 1.801186 + }, + "sweep": [ + { + "allthreads": { + "median_ms": 0.0424, + "qps": 22169.041155716477, + "recall": 0.273 + }, + "knob": 1, + "mean_ms": 0.04181829999999983, + "median_ms": 0.0408, + "nq": 2000, + "p95_ms": 0.052500000000000005, + "p99_ms": 0.0609, + "qps_1t": 24266.219844671956, + "qps_1t_max": 24373.68762925675, + "qps_1t_min": 23138.774801874253, + "qps_1t_runs": [ + 23138.774801874253, + 24266.219844671956, + 24373.68762925675 + ], + "recall": 0.273 + }, + { + "allthreads": { + "median_ms": 0.05450000000000001, + "qps": 17989.073436794522, + "recall": 0.39185 + }, + "knob": 2, + "mean_ms": 0.05683525000000003, + "median_ms": 0.0559, + "nq": 2000, + "p95_ms": 0.0712, + "p99_ms": 0.0787, + "qps_1t": 17640.215457591614, + "qps_1t_max": 17742.89131059643, + "qps_1t_min": 17404.47120865353, + "qps_1t_runs": [ + 17404.47120865353, + 17742.89131059643, + 17640.215457591614 + ], + "recall": 0.39185 + }, + { + "allthreads": { + "median_ms": 0.0797, + "qps": 12368.285489712995, + "recall": 0.51675 + }, + "knob": 4, + "mean_ms": 0.08036593333333353, + "median_ms": 0.07880000000000001, + "nq": 2000, + "p95_ms": 0.1023, + "p99_ms": 0.11649999999999999, + "qps_1t": 12425.223452112215, + "qps_1t_max": 12885.352575459821, + "qps_1t_min": 12046.9082513491, + "qps_1t_runs": [ + 12885.352575459821, + 12046.9082513491, + 12425.223452112215 + ], + "recall": 0.51675 + }, + { + "allthreads": { + "median_ms": 0.1171, + "qps": 8335.879249787527, + "recall": 0.64025 + }, + "knob": 8, + "mean_ms": 0.12940155000000014, + "median_ms": 0.1262, + "nq": 2000, + "p95_ms": 0.1685, + "p99_ms": 0.201, + "qps_1t": 7663.9363893279515, + "qps_1t_max": 8108.156318767302, + "qps_1t_min": 7440.985543653292, + "qps_1t_runs": [ + 7440.985543653292, + 7663.9363893279515, + 8108.156318767302 + ], + "recall": 0.64025 + }, + { + "allthreads": { + "median_ms": 0.1966, + "qps": 5069.301148678294, + "recall": 0.74505 + }, + "knob": 16, + "mean_ms": 0.19838436666666703, + "median_ms": 0.19560000000000002, + "nq": 2000, + "p95_ms": 0.251, + "p99_ms": 0.2852, + "qps_1t": 5069.430925961983, + "qps_1t_max": 5090.874658084133, + "qps_1t_min": 4963.705386216, + "qps_1t_runs": [ + 5090.874658084133, + 5069.430925961983, + 4963.705386216 + ], + "recall": 0.74505 + }, + { + "allthreads": { + "median_ms": 0.3455, + "qps": 2890.5585397360746, + "recall": 0.83455 + }, + "knob": 32, + "mean_ms": 0.3545716166666663, + "median_ms": 0.34909999999999997, + "nq": 2000, + "p95_ms": 0.45189999999999997, + "p99_ms": 0.5088999999999999, + "qps_1t": 2839.9215727258534, + "qps_1t_max": 2847.863703521438, + "qps_1t_min": 2774.29411207399, + "qps_1t_runs": [ + 2839.9215727258534, + 2774.29411207399, + 2847.863703521438 + ], + "recall": 0.83455 + }, + { + "allthreads": { + "median_ms": 0.5929000000000001, + "qps": 1667.319700215919, + "recall": 0.90705 + }, + "knob": 64, + "mean_ms": 0.6426998166666651, + "median_ms": 0.6338999999999999, + "nq": 2000, + "p95_ms": 0.7996, + "p99_ms": 0.87, + "qps_1t": 1562.1560815751616, + "qps_1t_max": 1568.613917840707, + "qps_1t_min": 1537.3899776471187, + "qps_1t_runs": [ + 1562.1560815751616, + 1568.613917840707, + 1537.3899776471187 + ], + "recall": 0.90705 + }, + { + "allthreads": { + "median_ms": 0.6632, + "qps": 1508.873572313256, + "recall": 0.9529 + }, + "knob": 128, + "mean_ms": 1.2603591499999984, + "median_ms": 1.2541, + "nq": 2000, + "p95_ms": 1.5132999999999999, + "p99_ms": 1.6332, + "qps_1t": 793.386520378886, + "qps_1t_max": 796.1783756440444, + "qps_1t_min": 790.7277367838166, + "qps_1t_runs": [ + 790.7277367838166, + 796.1783756440444, + 793.386520378886 + ], + "recall": 0.9529 + }, + { + "allthreads": { + "median_ms": 0.8999, + "qps": 1083.9053824880696, + "recall": 0.9808383233532935 + }, + "knob": 256, + "mean_ms": 2.5085312427410043, + "median_ms": 2.5017, + "nq": 1148, + "p95_ms": 2.8765, + "p99_ms": 3.198, + "qps_1t": 400.75782326922064, + "qps_1t_max": 402.1308451101971, + "qps_1t_min": 393.1484633734097, + "qps_1t_runs": [ + 393.1484633734097, + 400.75782326922064, + 402.1308451101971 + ], + "recall": 0.9806620209059234 + }, + { + "allthreads": { + "median_ms": 3.2567999999999997, + "qps": 299.54243283765044, + "recall": 0.9998976458546571 + }, + "knob": 1265, + "mean_ms": 6.4665334728033494, + "median_ms": 6.3221, + "nq": 478, + "p95_ms": 7.301, + "p99_ms": 8.6458, + "qps_1t": 154.1176827148959, + "qps_1t_max": 157.30762536208323, + "qps_1t_min": 152.5766834291411, + "qps_1t_runs": [ + 157.30762536208323, + 152.5766834291411, + 154.1176827148959 + ], + "recall": 1.0 + } + ], + "system": "rust_default_ivf", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + } + ], + "status": "ok", + "load_before_pct": 5.584375000000002, + "load_contaminated": false, + "wall_s": 97.45267415046692, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/rust_default__text384__100000__filter.json b/benchmarks/competitive/results/raw/rust_default__text384__100000__filter.json new file mode 100644 index 0000000..2c7ab54 --- /dev/null +++ b/benchmarks/competitive/results/raw/rust_default__text384__100000__filter.json @@ -0,0 +1,233 @@ +{ + "label": "rust_default", + "runs": [ + { + "d": 384, + "ds": "text384", + "filter": { + "build_with_metadata_s": 4.7653895, + "mem": { + "peak_mb": 429.875, + "rss_mb": 352.44140625 + }, + "subsets": { + "f01": [ + { + "knob": 2, + "mean_ms": 0.08471788333333351, + "median_ms": 0.09079999999999999, + "nq": 2000, + "p95_ms": 0.1007, + "p99_ms": 0.107, + "qps_1t": 11832.485141356823, + "qps_1t_max": 11890.981107014688, + "qps_1t_min": 11689.999906480021, + "qps_1t_runs": [ + 11689.999906480021, + 11890.981107014688, + 11832.485141356823 + ], + "recall": 0.5042 + }, + { + "knob": 8, + "mean_ms": 0.06029223333333303, + "median_ms": 0.0669, + "nq": 2000, + "p95_ms": 0.0752, + "p99_ms": 0.0849, + "qps_1t": 16608.55058009517, + "qps_1t_max": 16694.281791129502, + "qps_1t_min": 16456.57110884376, + "qps_1t_runs": [ + 16608.55058009517, + 16694.281791129502, + 16456.57110884376 + ], + "recall": 0.5042 + }, + { + "knob": 32, + "mean_ms": 0.04070648333333328, + "median_ms": 0.040299999999999996, + "nq": 2000, + "p95_ms": 0.0448, + "p99_ms": 0.0524, + "qps_1t": 24552.680845839874, + "qps_1t_max": 24690.93126785461, + "qps_1t_min": 24455.857177794118, + "qps_1t_runs": [ + 24455.857177794118, + 24552.680845839874, + 24690.93126785461 + ], + "recall": 0.57315 + }, + { + "knob": 64, + "mean_ms": 0.04964830000000025, + "median_ms": 0.0482, + "nq": 2000, + "p95_ms": 0.0575, + "p99_ms": 0.07100000000000001, + "qps_1t": 20376.641847916915, + "qps_1t_max": 20493.819064170217, + "qps_1t_min": 19579.47209827325, + "qps_1t_runs": [ + 20376.641847916915, + 20493.819064170217, + 19579.47209827325 + ], + "recall": 0.72055 + }, + { + "knob": 1265, + "mean_ms": 0.06680314999999908, + "median_ms": 0.0653, + "nq": 2000, + "p95_ms": 0.0731, + "p99_ms": 0.08009999999999999, + "qps_1t": 15041.699351025905, + "qps_1t_max": 15208.685984739608, + "qps_1t_min": 14667.982873663208, + "qps_1t_runs": [ + 14667.982873663208, + 15041.699351025905, + 15208.685984739608 + ], + "recall": 1.0 + }, + { + "knob": "exact", + "mean_ms": 0.06590931666666716, + "median_ms": 0.06499999999999999, + "nq": 2000, + "p95_ms": 0.0698, + "p99_ms": 0.0733, + "qps_1t": 15288.018626921888, + "qps_1t_max": 15408.403589233587, + "qps_1t_min": 14832.921966964139, + "qps_1t_runs": [ + 14832.921966964139, + 15408.403589233587, + 15288.018626921888 + ], + "recall": 1.0 + } + ], + "f10": [ + { + "knob": 2, + "mean_ms": 0.03395223333333317, + "median_ms": 0.0316, + "nq": 2000, + "p95_ms": 0.0597, + "p99_ms": 0.0652, + "qps_1t": 29580.547831745884, + "qps_1t_max": 29633.58077373282, + "qps_1t_min": 29150.099693340973, + "qps_1t_runs": [ + 29150.099693340973, + 29633.58077373282, + 29580.547831745884 + ], + "recall": 0.3079 + }, + { + "knob": 8, + "mean_ms": 0.041291883333333286, + "median_ms": 0.0408, + "nq": 2000, + "p95_ms": 0.0479, + "p99_ms": 0.0517, + "qps_1t": 24351.13360614719, + "qps_1t_max": 24935.10638563146, + "qps_1t_min": 23416.077947440273, + "qps_1t_runs": [ + 24351.13360614719, + 23416.077947440273, + 24935.10638563146 + ], + "recall": 0.5295 + }, + { + "knob": 32, + "mean_ms": 0.06929940000000015, + "median_ms": 0.068, + "nq": 2000, + "p95_ms": 0.0807, + "p99_ms": 0.0863, + "qps_1t": 14452.109682841197, + "qps_1t_max": 14461.912023850613, + "qps_1t_min": 14376.698247480468, + "qps_1t_runs": [ + 14461.912023850613, + 14452.109682841197, + 14376.698247480468 + ], + "recall": 0.7686 + }, + { + "knob": 64, + "mean_ms": 0.09254078333333327, + "median_ms": 0.0893, + "nq": 2000, + "p95_ms": 0.1181, + "p99_ms": 0.1393, + "qps_1t": 11118.622358215353, + "qps_1t_max": 11504.309226628562, + "qps_1t_min": 9924.652041699432, + "qps_1t_runs": [ + 9924.652041699432, + 11118.622358215353, + 11504.309226628562 + ], + "recall": 0.86425 + }, + { + "knob": 1265, + "mean_ms": 0.32901843333333386, + "median_ms": 0.32220000000000004, + "nq": 2000, + "p95_ms": 0.3634, + "p99_ms": 0.42069999999999996, + "qps_1t": 3051.619517365011, + "qps_1t_max": 3082.655393190381, + "qps_1t_min": 2985.388315428961, + "qps_1t_runs": [ + 3051.619517365011, + 3082.655393190381, + 2985.388315428961 + ], + "recall": 1.0 + }, + { + "knob": "exact", + "mean_ms": 0.3240598166666661, + "median_ms": 0.32239999999999996, + "nq": 2000, + "p95_ms": 0.3554, + "p99_ms": 0.3848, + "qps_1t": 3093.7669722122528, + "qps_1t_max": 3108.4470338498995, + "qps_1t_min": 3055.815851647473, + "qps_1t_runs": [ + 3055.815851647473, + 3108.4470338498995, + 3093.7669722122528 + ], + "recall": 1.0 + } + ] + } + }, + "n": 100000, + "system": "rust_default_ivf" + } + ], + "status": "ok", + "load_before_pct": 5.8937500000000025, + "load_contaminated": false, + "wall_s": 12.65131139755249, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/rust_default__text384__10000__core.json b/benchmarks/competitive/results/raw/rust_default__text384__10000__core.json new file mode 100644 index 0000000..dc2b48b --- /dev/null +++ b/benchmarks/competitive/results/raw/rust_default__text384__10000__core.json @@ -0,0 +1,350 @@ +{ + "label": "rust_default", + "runs": [ + { + "all_threads": 32, + "build": { + "first_query_ms": 7.188899999999999, + "index_s": 0.0, + "ingest_s": 0.0058946, + "ready_s": 0.0130835, + "total_s": 0.0058946 + }, + "concurrent_clients": { + "clients": 8, + "qps": 3616.286670276926, + "scan_threads": 1 + }, + "d": 384, + "ds": "text384", + "k": 10, + "loop_allthreads": { + "mean_ms": 0.26278995000000105, + "median_ms": 0.2576, + "nq": 2000, + "p95_ms": 0.3039, + "p99_ms": 0.4231, + "qps": 3896.271891203626, + "qps_runs": [ + 3621.769223400328, + 3912.269699353802, + 3896.271891203626 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 46.09765625, + "rss_mb": 46.09765625 + }, + "mem_after_first_query": { + "peak_mb": 46.09765625, + "rss_mb": 46.09765625 + }, + "mem_after_ingest": { + "peak_mb": 40.765625, + "rss_mb": 30.12890625 + }, + "mem_after_sweep": { + "peak_mb": 46.6953125, + "rss_mb": 46.5703125 + }, + "mem_baseline": { + "peak_mb": 10.8515625, + "rss_mb": 10.8515625 + }, + "n": 10000, + "queries": 2000, + "sweep": [ + { + "knob": null, + "mean_ms": 0.2693790666666671, + "median_ms": 0.27349999999999997, + "nq": 2000, + "p95_ms": 0.3096, + "p99_ms": 0.3631, + "qps_1t": 3727.6705544406655, + "qps_1t_max": 3796.0813431902357, + "qps_1t_min": 3617.374393908926, + "qps_1t_runs": [ + 3617.374393908926, + 3727.6705544406655, + 3796.0813431902357 + ], + "recall": 1.0 + } + ], + "system": "rust_default_exact", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + }, + { + "all_threads": 32, + "build": { + "cache_s": 0.0071889, + "first_query_ms": 0.0, + "index_s": 0.0527886, + "ingest_s": 0.0058946, + "ready_s": 0.0658721, + "total_s": 0.0658721 + }, + "d": 384, + "ds": "text384", + "index_stats": { + "build_seconds": 0.0527274, + "index_bytes": 697608, + "nlist": 400, + "trained_n": 10000 + }, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 63.515625, + "rss_mb": 47.203125 + }, + "mem_after_ingest": { + "peak_mb": 40.765625, + "rss_mb": 30.12890625 + }, + "mem_after_sweep": { + "peak_mb": 63.515625, + "rss_mb": 47.765625 + }, + "mem_baseline": { + "peak_mb": 10.8515625, + "rss_mb": 10.8515625 + }, + "n": 10000, + "queries": 2000, + "ref_knob": 64, + "ref_recall_pre": 0.9531, + "sweep": [ + { + "allthreads": { + "median_ms": 0.0101, + "qps": 94795.71523367145, + "recall": 0.28205 + }, + "knob": 1, + "mean_ms": 0.0103977833333331, + "median_ms": 0.01, + "nq": 2000, + "p95_ms": 0.012700000000000001, + "p99_ms": 0.02, + "qps_1t": 97584.30063771328, + "qps_1t_max": 98118.57629945799, + "qps_1t_min": 92988.22304155189, + "qps_1t_runs": [ + 92988.22304155189, + 97584.30063771328, + 98118.57629945799 + ], + "recall": 0.28205 + }, + { + "allthreads": { + "median_ms": 0.0122, + "qps": 77811.62583501612, + "recall": 0.41255 + }, + "knob": 2, + "mean_ms": 0.012802383333333433, + "median_ms": 0.0121, + "nq": 2000, + "p95_ms": 0.015400000000000002, + "p99_ms": 0.0198, + "qps_1t": 77013.41958836328, + "qps_1t_max": 81272.40070544426, + "qps_1t_min": 76230.55168050241, + "qps_1t_runs": [ + 77013.41958836328, + 76230.55168050241, + 81272.40070544426 + ], + "recall": 0.41255 + }, + { + "allthreads": { + "median_ms": 0.0167, + "qps": 58913.1119967717, + "recall": 0.54975 + }, + "knob": 4, + "mean_ms": 0.017383666666666648, + "median_ms": 0.0166, + "nq": 2000, + "p95_ms": 0.022799999999999997, + "p99_ms": 0.028, + "qps_1t": 56952.148804574426, + "qps_1t_max": 60021.6678220838, + "qps_1t_min": 55767.005451224824, + "qps_1t_runs": [ + 55767.005451224824, + 56952.148804574426, + 60021.6678220838 + ], + "recall": 0.54975 + }, + { + "allthreads": { + "median_ms": 0.0209, + "qps": 45763.134591666996, + "recall": 0.6894 + }, + "knob": 8, + "mean_ms": 0.02615191666666659, + "median_ms": 0.0245, + "nq": 2000, + "p95_ms": 0.0389, + "p99_ms": 0.0521, + "qps_1t": 38850.41618508339, + "qps_1t_max": 42739.243600866714, + "qps_1t_min": 34108.38964059989, + "qps_1t_runs": [ + 42739.243600866714, + 38850.41618508339, + 34108.38964059989 + ], + "recall": 0.6894 + }, + { + "allthreads": { + "median_ms": 0.0313, + "qps": 31228.042782418597, + "recall": 0.80415 + }, + "knob": 16, + "mean_ms": 0.033053599999999995, + "median_ms": 0.032100000000000004, + "nq": 2000, + "p95_ms": 0.043699999999999996, + "p99_ms": 0.054599999999999996, + "qps_1t": 30262.954814382112, + "qps_1t_max": 32184.662720827047, + "qps_1t_min": 28533.601168736335, + "qps_1t_runs": [ + 30262.954814382112, + 28533.601168736335, + 32184.662720827047 + ], + "recall": 0.80415 + }, + { + "allthreads": { + "median_ms": 0.048600000000000004, + "qps": 20209.67538208919, + "recall": 0.8948 + }, + "knob": 32, + "mean_ms": 0.050496883333333346, + "median_ms": 0.0492, + "nq": 2000, + "p95_ms": 0.060000000000000005, + "p99_ms": 0.0781, + "qps_1t": 19591.57444749312, + "qps_1t_max": 20393.617205687005, + "qps_1t_min": 19450.20116370554, + "qps_1t_runs": [ + 20393.617205687005, + 19450.20116370554, + 19591.57444749312 + ], + "recall": 0.8948 + }, + { + "allthreads": { + "median_ms": 0.0821, + "qps": 11971.947333009288, + "recall": 0.9531 + }, + "knob": 64, + "mean_ms": 0.0849742833333333, + "median_ms": 0.0818, + "nq": 2000, + "p95_ms": 0.1002, + "p99_ms": 0.12760000000000002, + "qps_1t": 11870.344970030317, + "qps_1t_max": 12313.185882932386, + "qps_1t_min": 11177.486753281008, + "qps_1t_runs": [ + 11177.486753281008, + 11870.344970030317, + 12313.185882932386 + ], + "recall": 0.9531 + }, + { + "allthreads": { + "median_ms": 0.1516, + "qps": 6451.17588808501, + "recall": 0.98575 + }, + "knob": 128, + "mean_ms": 0.16351716666666705, + "median_ms": 0.158, + "nq": 2000, + "p95_ms": 0.1936, + "p99_ms": 0.2422, + "qps_1t": 6232.9198518185685, + "qps_1t_max": 6255.802256592984, + "qps_1t_min": 5873.320120320836, + "qps_1t_runs": [ + 5873.320120320836, + 6255.802256592984, + 6232.9198518185685 + ], + "recall": 0.98575 + }, + { + "allthreads": { + "median_ms": 0.24649999999999997, + "qps": 3737.7350895206196, + "recall": 0.9979 + }, + "knob": 256, + "mean_ms": 0.2847525000000006, + "median_ms": 0.2816, + "nq": 2000, + "p95_ms": 0.3328, + "p99_ms": 0.39189999999999997, + "qps_1t": 3569.374269059082, + "qps_1t_max": 3591.080760892564, + "qps_1t_min": 3382.6220834820947, + "qps_1t_runs": [ + 3591.080760892564, + 3382.6220834820947, + 3569.374269059082 + ], + "recall": 0.9979 + }, + { + "allthreads": { + "median_ms": 0.25470000000000004, + "qps": 3821.5796193630345, + "recall": 1.0 + }, + "knob": 400, + "mean_ms": 0.2501355666666668, + "median_ms": 0.24449999999999997, + "nq": 2000, + "p95_ms": 0.2837, + "p99_ms": 0.3343, + "qps_1t": 4006.9352034501276, + "qps_1t_max": 4048.6739681802546, + "qps_1t_min": 3939.412621759858, + "qps_1t_runs": [ + 4048.6739681802546, + 3939.412621759858, + 4006.9352034501276 + ], + "recall": 1.0 + } + ], + "system": "rust_default_ivf", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + } + ], + "status": "ok", + "load_before_pct": 4.58125, + "load_contaminated": false, + "wall_s": 14.721854448318481, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/rust_native__glove__1000000__core+mutation.json b/benchmarks/competitive/results/raw/rust_native__glove__1000000__core+mutation.json new file mode 100644 index 0000000..436d4c9 --- /dev/null +++ b/benchmarks/competitive/results/raw/rust_native__glove__1000000__core+mutation.json @@ -0,0 +1,372 @@ +{ + "label": "rust_native", + "runs": [ + { + "all_threads": 32, + "build": { + "first_query_ms": 658.0114, + "index_s": 0.0, + "ingest_s": 0.7495724000000001, + "ready_s": 1.4075838000000003, + "total_s": 0.7495724000000001 + }, + "concurrent_clients": { + "clients": 8, + "qps": 16.95327326531491, + "scan_threads": 1 + }, + "d": 100, + "ds": "glove", + "k": 10, + "loop_allthreads": { + "mean_ms": 14.124988545688547, + "median_ms": 12.790700000000001, + "nq": 259, + "p95_ms": 22.253700000000002, + "p99_ms": 27.1099, + "qps": 68.82732194883368, + "qps_runs": [ + 64.35732538807778, + 68.82732194883368, + 81.250576243831 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 1359.609375, + "rss_mb": 1351.9921875 + }, + "mem_after_first_query": { + "peak_mb": 1359.609375, + "rss_mb": 1351.9921875 + }, + "mem_after_ingest": { + "peak_mb": 978.484375, + "rss_mb": 850.34765625 + }, + "mem_after_sweep": { + "peak_mb": 1359.609375, + "rss_mb": 1352.390625 + }, + "mem_baseline": { + "peak_mb": 6.58984375, + "rss_mb": 6.58984375 + }, + "n": 1000000, + "queries": 2000, + "sweep": [ + { + "knob": null, + "mean_ms": 25.172556394129973, + "median_ms": 25.696900000000003, + "nq": 159, + "p95_ms": 26.857499999999998, + "p99_ms": 28.1042, + "qps_1t": 40.13650146474251, + "qps_1t_max": 40.184567973435136, + "qps_1t_min": 38.884001759513325, + "qps_1t_runs": [ + 40.184567973435136, + 38.884001759513325, + 40.13650146474251 + ], + "recall": 1.0 + } + ], + "system": "rust_native_exact", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + }, + { + "all_threads": 32, + "build": { + "cache_s": 0.6580114, + "first_query_ms": 0.0, + "index_s": 5.6914566, + "ingest_s": 0.7495724000000001, + "ready_s": 7.099040400000001, + "total_s": 7.099040400000001 + }, + "d": 100, + "ds": "glove", + "index_stats": { + "build_seconds": 5.6907636, + "index_bytes": 9632008, + "nlist": 4000, + "trained_n": 1000000 + }, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 1428.37109375, + "rss_mb": 1362.56640625 + }, + "mem_after_ingest": { + "peak_mb": 978.484375, + "rss_mb": 850.34765625 + }, + "mem_after_sweep": { + "peak_mb": 1428.37109375, + "rss_mb": 1362.60546875 + }, + "mem_baseline": { + "peak_mb": 6.58984375, + "rss_mb": 6.58984375 + }, + "mutation": { + "after": { + "mean_ms": 6.415073264604813, + "median_ms": 6.4448, + "nq": 485, + "p95_ms": 7.9916, + "p99_ms": 8.532100000000002, + "qps_1t": 155.55716921617923, + "qps_1t_max": 157.51122137546145, + "qps_1t_min": 154.60822896401254, + "qps_1t_runs": [ + 157.51122137546145, + 154.60822896401254, + 155.55716921617923 + ], + "recall": 0.9531958762886598 + }, + "delete_10k": { + "per_s": 297216.5668514363, + "seconds": 0.0336455 + }, + "first_query_after_ms": 749.9970999999999, + "index_stats_after": { + "builds": 89609, + "churn": 30000, + "pending": 0, + "trained_n": 1000000 + }, + "insert_10k": { + "per_s": 61105.53353380022, + "seconds": 0.1636513 + }, + "mem_after": { + "peak_mb": 1748.8046875, + "rss_mb": 1378.28515625 + }, + "recall_after": 0.9531958762886598, + "recall_before": 0.9543795620437956, + "ref_knob": 256, + "update_10k": { + "per_s": 253556.12464819086, + "seconds": 0.039439 + } + }, + "n": 1000000, + "queries": 2000, + "ref_knob": 256, + "ref_recall_pre": 0.9543795620437956, + "sweep": [ + { + "allthreads": { + "median_ms": 0.1747, + "qps": 5548.384268888027, + "recall": 0.30875 + }, + "knob": 1, + "mean_ms": 0.10140630000000009, + "median_ms": 0.0991, + "nq": 2000, + "p95_ms": 0.13970000000000002, + "p99_ms": 0.16469999999999999, + "qps_1t": 9787.46040360551, + "qps_1t_max": 10042.277990339335, + "qps_1t_min": 9759.110984025803, + "qps_1t_runs": [ + 9759.110984025803, + 9787.46040360551, + 10042.277990339335 + ], + "recall": 0.30875 + }, + { + "allthreads": { + "median_ms": 0.2436, + "qps": 3924.265598219951, + "recall": 0.42305 + }, + "knob": 2, + "mean_ms": 0.1308827333333332, + "median_ms": 0.1285, + "nq": 2000, + "p95_ms": 0.1777, + "p99_ms": 0.211, + "qps_1t": 7585.6188803019595, + "qps_1t_max": 7780.729700173469, + "qps_1t_min": 7558.74181718956, + "qps_1t_runs": [ + 7780.729700173469, + 7585.6188803019595, + 7558.74181718956 + ], + "recall": 0.42305 + }, + { + "allthreads": { + "median_ms": 0.3667, + "qps": 2675.3193629048014, + "recall": 0.5377 + }, + "knob": 4, + "mean_ms": 0.19379598333333323, + "median_ms": 0.19039999999999999, + "nq": 2000, + "p95_ms": 0.2703, + "p99_ms": 0.32239999999999996, + "qps_1t": 5135.5463258375785, + "qps_1t_max": 5308.778463347533, + "qps_1t_min": 5042.878333752326, + "qps_1t_runs": [ + 5135.5463258375785, + 5308.778463347533, + 5042.878333752326 + ], + "recall": 0.5377 + }, + { + "allthreads": { + "median_ms": 0.5979, + "qps": 1595.88478765117, + "recall": 0.6346 + }, + "knob": 8, + "mean_ms": 0.30831666666666685, + "median_ms": 0.3004, + "nq": 2000, + "p95_ms": 0.4412, + "p99_ms": 0.5203, + "qps_1t": 3220.1027148363946, + "qps_1t_max": 3376.5184414463597, + "qps_1t_min": 3142.3037831609236, + "qps_1t_runs": [ + 3220.1027148363946, + 3142.3037831609236, + 3376.5184414463597 + ], + "recall": 0.6346 + }, + { + "allthreads": { + "median_ms": 0.9238000000000001, + "qps": 1049.2226545427593, + "recall": 0.72835 + }, + "knob": 16, + "mean_ms": 0.5305095166666661, + "median_ms": 0.5186000000000001, + "nq": 2000, + "p95_ms": 0.7647, + "p99_ms": 0.8749, + "qps_1t": 1902.5639903616077, + "qps_1t_max": 1965.0307065523355, + "qps_1t_min": 1795.2542991179096, + "qps_1t_runs": [ + 1965.0307065523355, + 1795.2542991179096, + 1902.5639903616077 + ], + "recall": 0.72835 + }, + { + "allthreads": { + "median_ms": 1.5516, + "qps": 633.6315663725784, + "recall": 0.8026859504132231 + }, + "knob": 32, + "mean_ms": 0.9899157500000009, + "median_ms": 0.971, + "nq": 2000, + "p95_ms": 1.3912, + "p99_ms": 1.5382, + "qps_1t": 1009.7905250143275, + "qps_1t_max": 1016.0905560225331, + "qps_1t_min": 1004.7437973648804, + "qps_1t_runs": [ + 1016.0905560225331, + 1009.7905250143275, + 1004.7437973648804 + ], + "recall": 0.8013 + }, + { + "allthreads": { + "median_ms": 2.7607999999999997, + "qps": 335.76141020546766, + "recall": 0.8657303370786517 + }, + "knob": 64, + "mean_ms": 1.7541737734487721, + "median_ms": 1.7332999999999998, + "nq": 1848, + "p95_ms": 2.3914, + "p99_ms": 2.6355, + "qps_1t": 575.5939207314974, + "qps_1t_max": 581.3493867975117, + "qps_1t_min": 554.0014568919268, + "qps_1t_runs": [ + 581.3493867975117, + 575.5939207314974, + 554.0014568919268 + ], + "recall": 0.8625541125541125 + }, + { + "allthreads": { + "median_ms": 3.0884, + "qps": 292.19850478467004, + "recall": 0.9199261992619926 + }, + "knob": 128, + "mean_ms": 3.078618540051677, + "median_ms": 3.0542, + "nq": 1032, + "p95_ms": 4.0786, + "p99_ms": 4.5251, + "qps_1t": 324.82901387860284, + "qps_1t_max": 343.6943640585216, + "qps_1t_min": 307.9053719671433, + "qps_1t_runs": [ + 343.6943640585216, + 324.82901387860284, + 307.9053719671433 + ], + "recall": 0.9165697674418605 + }, + { + "allthreads": { + "median_ms": 5.473999999999999, + "qps": 163.77652385618495, + "recall": 0.9553939393939394 + }, + "knob": 256, + "mean_ms": 6.364565754257904, + "median_ms": 6.3609, + "nq": 548, + "p95_ms": 8.0134, + "p99_ms": 8.5396, + "qps_1t": 157.21381770604563, + "qps_1t_max": 158.38667105640525, + "qps_1t_min": 155.7809382293614, + "qps_1t_runs": [ + 158.38667105640525, + 157.21381770604563, + 155.7809382293614 + ], + "recall": 0.9543795620437956 + } + ], + "system": "rust_native_ivf", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + } + ], + "status": "ok", + "load_before_pct": 5.837499999999997, + "load_contaminated": false, + "wall_s": 156.92534399032593, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/rust_native__glove__100000__core+mutation.json b/benchmarks/competitive/results/raw/rust_native__glove__100000__core+mutation.json new file mode 100644 index 0000000..6b21f04 --- /dev/null +++ b/benchmarks/competitive/results/raw/rust_native__glove__100000__core+mutation.json @@ -0,0 +1,394 @@ +{ + "label": "rust_native", + "runs": [ + { + "all_threads": 32, + "build": { + "first_query_ms": 41.7821, + "index_s": 0.0, + "ingest_s": 0.052907600000000006, + "ready_s": 0.09468970000000002, + "total_s": 0.052907600000000006 + }, + "concurrent_clients": { + "clients": 8, + "qps": 624.2763271763561, + "scan_threads": 1 + }, + "d": 100, + "ds": "glove", + "k": 10, + "loop_allthreads": { + "mean_ms": 0.6440888666666664, + "median_ms": 0.6114999999999999, + "nq": 2000, + "p95_ms": 0.8467, + "p99_ms": 1.141, + "qps": 1593.736107103529, + "qps_runs": [ + 1462.1382587702526, + 1610.616410280693, + 1593.736107103529 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 127.73828125, + "rss_mb": 127.73828125 + }, + "mem_after_first_query": { + "peak_mb": 127.73828125, + "rss_mb": 127.73828125 + }, + "mem_after_ingest": { + "peak_mb": 83.91796875, + "rss_mb": 78.4609375 + }, + "mem_after_sweep": { + "peak_mb": 128.04296875, + "rss_mb": 127.921875 + }, + "mem_baseline": { + "peak_mb": 6.58203125, + "rss_mb": 6.58203125 + }, + "n": 100000, + "queries": 2000, + "sweep": [ + { + "knob": null, + "mean_ms": 1.6096058700209652, + "median_ms": 1.5729, + "nq": 1749, + "p95_ms": 1.9405, + "p99_ms": 2.2458, + "qps_1t": 621.1011829225351, + "qps_1t_max": 627.0951800331468, + "qps_1t_min": 615.718154433941, + "qps_1t_runs": [ + 627.0951800331468, + 621.1011829225351, + 615.718154433941 + ], + "recall": 1.0 + } + ], + "system": "rust_native_exact", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + }, + { + "all_threads": 32, + "build": { + "cache_s": 0.0417821, + "first_query_ms": 0.0, + "index_s": 0.1755758, + "ingest_s": 0.052907600000000006, + "ready_s": 0.27026550000000005, + "total_s": 0.27026550000000005 + }, + "d": 100, + "ds": "glove", + "index_stats": { + "build_seconds": 0.1755668, + "index_bytes": 1316128, + "nlist": 1265, + "trained_n": 100000 + }, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 150.78125, + "rss_mb": 131.375 + }, + "mem_after_ingest": { + "peak_mb": 83.91796875, + "rss_mb": 78.4609375 + }, + "mem_after_sweep": { + "peak_mb": 150.78125, + "rss_mb": 131.34765625 + }, + "mem_baseline": { + "peak_mb": 6.58203125, + "rss_mb": 6.58203125 + }, + "mutation": { + "after": { + "mean_ms": 0.7658948000000008, + "median_ms": 0.7534, + "nq": 2000, + "p95_ms": 0.929, + "p99_ms": 1.0816000000000001, + "qps_1t": 1301.8391537368536, + "qps_1t_max": 1315.7182344666462, + "qps_1t_min": 1299.546510249782, + "qps_1t_runs": [ + 1315.7182344666462, + 1299.546510249782, + 1301.8391537368536 + ], + "recall": 0.9743 + }, + "delete_10k": { + "per_s": 1092382.8146335601, + "seconds": 0.0091543 + }, + "first_query_after_ms": 83.5681, + "index_stats_after": { + "builds": 115046, + "churn": 30000, + "pending": 0, + "trained_n": 100000 + }, + "insert_10k": { + "per_s": 708566.5698292354, + "seconds": 0.014113 + }, + "mem_after": { + "peak_mb": 173.14453125, + "rss_mb": 145.70703125 + }, + "recall_after": 0.9743, + "recall_before": 0.974, + "ref_knob": 256, + "update_10k": { + "per_s": 732219.8709828587, + "seconds": 0.0136571 + } + }, + "n": 100000, + "queries": 2000, + "ref_knob": 256, + "ref_recall_pre": 0.974, + "sweep": [ + { + "allthreads": { + "median_ms": 0.0188, + "qps": 52173.005686857614, + "recall": 0.28505 + }, + "knob": 1, + "mean_ms": 0.01918759999999997, + "median_ms": 0.0187, + "nq": 2000, + "p95_ms": 0.0245, + "p99_ms": 0.0285, + "qps_1t": 52724.123659159646, + "qps_1t_max": 53186.82136940121, + "qps_1t_min": 50519.08358382383, + "qps_1t_runs": [ + 50519.08358382383, + 52724.123659159646, + 53186.82136940121 + ], + "recall": 0.28505 + }, + { + "allthreads": { + "median_ms": 0.023, + "qps": 42747.55644280486, + "recall": 0.4016 + }, + "knob": 2, + "mean_ms": 0.023989816666666594, + "median_ms": 0.0236, + "nq": 2000, + "p95_ms": 0.0304, + "p99_ms": 0.0334, + "qps_1t": 40953.14346090927, + "qps_1t_max": 43266.81781208359, + "qps_1t_min": 40918.371939817254, + "qps_1t_runs": [ + 40953.14346090927, + 40918.371939817254, + 43266.81781208359 + ], + "recall": 0.4016 + }, + { + "allthreads": { + "median_ms": 0.0303, + "qps": 32611.106038272366, + "recall": 0.52085 + }, + "knob": 4, + "mean_ms": 0.03310233333333326, + "median_ms": 0.032, + "nq": 2000, + "p95_ms": 0.0455, + "p99_ms": 0.054799999999999995, + "qps_1t": 31286.37040559653, + "qps_1t_max": 32777.3744339757, + "qps_1t_min": 27147.83489229774, + "qps_1t_runs": [ + 32777.3744339757, + 31286.37040559653, + 27147.83489229774 + ], + "recall": 0.52085 + }, + { + "allthreads": { + "median_ms": 0.0468, + "qps": 20838.28242540935, + "recall": 0.63505 + }, + "knob": 8, + "mean_ms": 0.04619751666666673, + "median_ms": 0.0446, + "nq": 2000, + "p95_ms": 0.0593, + "p99_ms": 0.06820000000000001, + "qps_1t": 21678.52481974306, + "qps_1t_max": 22276.850008855068, + "qps_1t_min": 21019.75226119985, + "qps_1t_runs": [ + 22276.850008855068, + 21019.75226119985, + 21678.52481974306 + ], + "recall": 0.63505 + }, + { + "allthreads": { + "median_ms": 0.0709, + "qps": 14078.09258739152, + "recall": 0.73085 + }, + "knob": 16, + "mean_ms": 0.07260376666666657, + "median_ms": 0.07150000000000001, + "nq": 2000, + "p95_ms": 0.09519999999999999, + "p99_ms": 0.1077, + "qps_1t": 13851.202076572197, + "qps_1t_max": 13940.371455137816, + "qps_1t_min": 13535.224745639289, + "qps_1t_runs": [ + 13851.202076572197, + 13940.371455137816, + 13535.224745639289 + ], + "recall": 0.73085 + }, + { + "allthreads": { + "median_ms": 0.1227, + "qps": 7956.737626178936, + "recall": 0.8134 + }, + "knob": 32, + "mean_ms": 0.12016353333333346, + "median_ms": 0.1177, + "nq": 2000, + "p95_ms": 0.1536, + "p99_ms": 0.1707, + "qps_1t": 8218.07821062851, + "qps_1t_max": 8558.738407723744, + "qps_1t_min": 8198.871835235475, + "qps_1t_runs": [ + 8198.871835235475, + 8218.07821062851, + 8558.738407723744 + ], + "recall": 0.8134 + }, + { + "allthreads": { + "median_ms": 0.2099, + "qps": 4644.946124431138, + "recall": 0.88165 + }, + "knob": 64, + "mean_ms": 0.20941661666666714, + "median_ms": 0.2074, + "nq": 2000, + "p95_ms": 0.2671, + "p99_ms": 0.2966, + "qps_1t": 4858.765407145102, + "qps_1t_max": 4861.232473130136, + "qps_1t_min": 4614.097637074454, + "qps_1t_runs": [ + 4861.232473130136, + 4614.097637074454, + 4858.765407145102 + ], + "recall": 0.88165 + }, + { + "allthreads": { + "median_ms": 0.3767, + "qps": 2610.207870429288, + "recall": 0.9355 + }, + "knob": 128, + "mean_ms": 0.3895535166666657, + "median_ms": 0.3824, + "nq": 2000, + "p95_ms": 0.512, + "p99_ms": 0.5889, + "qps_1t": 2583.9018778765258, + "qps_1t_max": 2715.61656037291, + "qps_1t_min": 2418.915239879592, + "qps_1t_runs": [ + 2715.61656037291, + 2418.915239879592, + 2583.9018778765258 + ], + "recall": 0.9355 + }, + { + "allthreads": { + "median_ms": 0.49660000000000004, + "qps": 1940.3932536795403, + "recall": 0.974 + }, + "knob": 256, + "mean_ms": 0.7330722166666664, + "median_ms": 0.7251, + "nq": 2000, + "p95_ms": 0.9089, + "p99_ms": 1.0217, + "qps_1t": 1357.672693760426, + "qps_1t_max": 1382.6379252190575, + "qps_1t_min": 1352.4352185220912, + "qps_1t_runs": [ + 1357.672693760426, + 1382.6379252190575, + 1352.4352185220912 + ], + "recall": 0.974 + }, + { + "allthreads": { + "median_ms": 0.5972, + "qps": 1624.7220303696256, + "recall": 1.0 + }, + "knob": 1265, + "mean_ms": 1.5772921, + "median_ms": 1.5485, + "nq": 2000, + "p95_ms": 1.8404999999999998, + "p99_ms": 2.0385, + "qps_1t": 635.5073999117142, + "qps_1t_max": 639.6698893981976, + "qps_1t_min": 626.9497549613916, + "qps_1t_runs": [ + 639.6698893981976, + 626.9497549613916, + 635.5073999117142 + ], + "recall": 1.0 + } + ], + "system": "rust_native_ivf", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + } + ], + "status": "ok", + "load_before_pct": 4.156249999999997, + "load_contaminated": false, + "wall_s": 50.27099394798279, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/rust_native__sift__100000__core+mutation.json b/benchmarks/competitive/results/raw/rust_native__sift__100000__core+mutation.json new file mode 100644 index 0000000..c90ea52 --- /dev/null +++ b/benchmarks/competitive/results/raw/rust_native__sift__100000__core+mutation.json @@ -0,0 +1,394 @@ +{ + "label": "rust_native", + "runs": [ + { + "all_threads": 32, + "build": { + "first_query_ms": 58.5794, + "index_s": 0.0, + "ingest_s": 0.05671, + "ready_s": 0.1152894, + "total_s": 0.05671 + }, + "concurrent_clients": { + "clients": 8, + "qps": 134.76631705925612, + "scan_threads": 1 + }, + "d": 128, + "ds": "sift", + "k": 10, + "loop_allthreads": { + "mean_ms": 3.0940368769806974, + "median_ms": 2.8814, + "nq": 1157, + "p95_ms": 4.8353, + "p99_ms": 6.1524, + "qps": 328.0837290093046, + "qps_runs": [ + 337.99778060783245, + 328.0837290093046, + 305.2960714860383 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 149.76171875, + "rss_mb": 149.76171875 + }, + "mem_after_first_query": { + "peak_mb": 149.76171875, + "rss_mb": 149.76171875 + }, + "mem_after_ingest": { + "peak_mb": 97.24609375, + "rss_mb": 89.66015625 + }, + "mem_after_sweep": { + "peak_mb": 150.08203125, + "rss_mb": 149.97265625 + }, + "mem_baseline": { + "peak_mb": 7.0078125, + "rss_mb": 7.0078125 + }, + "n": 100000, + "queries": 2000, + "sweep": [ + { + "knob": null, + "mean_ms": 2.5556045202220448, + "median_ms": 2.6197999999999997, + "nq": 1261, + "p95_ms": 3.1369000000000002, + "p99_ms": 3.4226, + "qps_1t": 363.7788119814834, + "qps_1t_max": 464.61190057880094, + "qps_1t_min": 361.59089210752825, + "qps_1t_runs": [ + 464.61190057880094, + 361.59089210752825, + 363.7788119814834 + ], + "recall": 1.0 + } + ], + "system": "rust_native_exact", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + }, + { + "all_threads": 32, + "build": { + "cache_s": 0.0585794, + "first_query_ms": 0.0, + "index_s": 0.5745037, + "ingest_s": 0.05671, + "ready_s": 0.6897930999999999, + "total_s": 0.6897930999999999 + }, + "d": 128, + "ds": "sift", + "index_stats": { + "build_seconds": 0.5743599, + "index_bytes": 1457808, + "nlist": 1265, + "trained_n": 100000 + }, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 178.640625, + "rss_mb": 152.59765625 + }, + "mem_after_ingest": { + "peak_mb": 97.24609375, + "rss_mb": 89.66015625 + }, + "mem_after_sweep": { + "peak_mb": 178.640625, + "rss_mb": 152.5625 + }, + "mem_baseline": { + "peak_mb": 7.0078125, + "rss_mb": 7.0078125 + }, + "mutation": { + "after": { + "mean_ms": 0.37663301666666676, + "median_ms": 0.3698, + "nq": 2000, + "p95_ms": 0.505, + "p99_ms": 0.5996, + "qps_1t": 2637.741196703611, + "qps_1t_max": 2708.0804381132552, + "qps_1t_min": 2621.0839204233707, + "qps_1t_runs": [ + 2621.0839204233707, + 2637.741196703611, + 2708.0804381132552 + ], + "recall": 0.98545 + }, + "delete_10k": { + "per_s": 506183.02565841755, + "seconds": 0.0197557 + }, + "first_query_after_ms": 276.5351, + "index_stats_after": { + "builds": 113339, + "churn": 30000, + "pending": 0, + "trained_n": 100000 + }, + "insert_10k": { + "per_s": 354948.5147179402, + "seconds": 0.0281731 + }, + "mem_after": { + "peak_mb": 206.8046875, + "rss_mb": 170.87109375 + }, + "recall_after": 0.98545, + "recall_before": 0.98605, + "ref_knob": 64, + "update_10k": { + "per_s": 342604.20306836325, + "seconds": 0.0291882 + } + }, + "n": 100000, + "queries": 2000, + "ref_knob": 64, + "ref_recall_pre": 0.98605, + "sweep": [ + { + "allthreads": { + "median_ms": 0.059, + "qps": 16215.076778388535, + "recall": 0.2912 + }, + "knob": 1, + "mean_ms": 0.02688296666666659, + "median_ms": 0.027, + "nq": 2000, + "p95_ms": 0.0348, + "p99_ms": 0.0422, + "qps_1t": 36184.817574242225, + "qps_1t_max": 42398.66189823047, + "qps_1t_min": 33981.992941940065, + "qps_1t_runs": [ + 42398.66189823047, + 36184.817574242225, + 33981.992941940065 + ], + "recall": 0.2912 + }, + { + "allthreads": { + "median_ms": 0.0678, + "qps": 14669.047942115954, + "recall": 0.43565 + }, + "knob": 2, + "mean_ms": 0.037223333333333365, + "median_ms": 0.0365, + "nq": 2000, + "p95_ms": 0.048600000000000004, + "p99_ms": 0.056, + "qps_1t": 27078.07285355503, + "qps_1t_max": 27601.968572398593, + "qps_1t_min": 25966.97779433897, + "qps_1t_runs": [ + 25966.97779433897, + 27078.07285355503, + 27601.968572398593 + ], + "recall": 0.43565 + }, + { + "allthreads": { + "median_ms": 0.0958, + "qps": 10145.779634680906, + "recall": 0.5955 + }, + "knob": 4, + "mean_ms": 0.047606150000000014, + "median_ms": 0.047, + "nq": 2000, + "p95_ms": 0.0633, + "p99_ms": 0.07279999999999999, + "qps_1t": 20257.616103994493, + "qps_1t_max": 22787.40039057601, + "qps_1t_min": 20173.32924487196, + "qps_1t_runs": [ + 22787.40039057601, + 20173.32924487196, + 20257.616103994493 + ], + "recall": 0.5955 + }, + { + "allthreads": { + "median_ms": 0.13620000000000002, + "qps": 7175.915045776964, + "recall": 0.74745 + }, + "knob": 8, + "mean_ms": 0.07279359999999997, + "median_ms": 0.07050000000000001, + "nq": 2000, + "p95_ms": 0.09849999999999999, + "p99_ms": 0.1144, + "qps_1t": 13567.56375398205, + "qps_1t_max": 14108.819917138895, + "qps_1t_min": 13550.511904463468, + "qps_1t_runs": [ + 13550.511904463468, + 14108.819917138895, + 13567.56375398205 + ], + "recall": 0.74745 + }, + { + "allthreads": { + "median_ms": 0.2271, + "qps": 4303.130592052672, + "recall": 0.87095 + }, + "knob": 16, + "mean_ms": 0.11392756666666629, + "median_ms": 0.1121, + "nq": 2000, + "p95_ms": 0.1571, + "p99_ms": 0.1795, + "qps_1t": 8585.732401609643, + "qps_1t_max": 9421.647459853168, + "qps_1t_min": 8391.243401860416, + "qps_1t_runs": [ + 9421.647459853168, + 8585.732401609643, + 8391.243401860416 + ], + "recall": 0.87095 + }, + { + "allthreads": { + "median_ms": 0.384, + "qps": 2522.9496965459157, + "recall": 0.94865 + }, + "knob": 32, + "mean_ms": 0.20777173333333251, + "median_ms": 0.2044, + "nq": 2000, + "p95_ms": 0.2761, + "p99_ms": 0.317, + "qps_1t": 4740.742988204084, + "qps_1t_max": 4973.279810896004, + "qps_1t_min": 4732.534875824197, + "qps_1t_runs": [ + 4973.279810896004, + 4732.534875824197, + 4740.742988204084 + ], + "recall": 0.94865 + }, + { + "allthreads": { + "median_ms": 0.7008, + "qps": 1399.2686442577065, + "recall": 0.98605 + }, + "knob": 64, + "mean_ms": 0.3611575166666662, + "median_ms": 0.3566, + "nq": 2000, + "p95_ms": 0.47590000000000005, + "p99_ms": 0.5452, + "qps_1t": 2780.4095126150673, + "qps_1t_max": 2800.678380317286, + "qps_1t_min": 2726.6013636551397, + "qps_1t_runs": [ + 2800.678380317286, + 2780.4095126150673, + 2726.6013636551397 + ], + "recall": 0.98605 + }, + { + "allthreads": { + "median_ms": 1.3581999999999999, + "qps": 743.5913272451731, + "recall": 0.9984118581259925 + }, + "knob": 128, + "mean_ms": 0.6374388666666685, + "median_ms": 0.6277, + "nq": 2000, + "p95_ms": 0.8291999999999999, + "p99_ms": 0.9763999999999999, + "qps_1t": 1611.6373106930503, + "qps_1t_max": 1643.4874672576204, + "qps_1t_min": 1463.3416812098444, + "qps_1t_runs": [ + 1643.4874672576204, + 1611.6373106930503, + 1463.3416812098444 + ], + "recall": 0.9984 + }, + { + "allthreads": { + "median_ms": 1.9338, + "qps": 485.74485986662154, + "recall": 0.9998776009791922 + }, + "knob": 256, + "mean_ms": 1.2109834905660386, + "median_ms": 1.1862000000000001, + "nq": 1908, + "p95_ms": 1.5536, + "p99_ms": 1.9023, + "qps_1t": 839.4112928799261, + "qps_1t_max": 840.4635447175514, + "qps_1t_min": 798.8370473324763, + "qps_1t_runs": [ + 798.8370473324763, + 839.4112928799261, + 840.4635447175514 + ], + "recall": 0.9997903563941299 + }, + { + "allthreads": { + "median_ms": 2.9061999999999997, + "qps": 316.4481758992458, + "recall": 1.0 + }, + "knob": 1265, + "mean_ms": 2.8010908129344116, + "median_ms": 2.7870999999999997, + "nq": 1103, + "p95_ms": 3.2592, + "p99_ms": 3.6491000000000002, + "qps_1t": 359.53671461478245, + "qps_1t_max": 359.69023594277206, + "qps_1t_min": 351.8964170085482, + "qps_1t_runs": [ + 359.69023594277206, + 351.8964170085482, + 359.53671461478245 + ], + "recall": 1.0 + } + ], + "system": "rust_native_ivf", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + } + ], + "status": "ok", + "load_before_pct": 12.721874999999995, + "load_contaminated": false, + "wall_s": 91.23237705230713, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/rust_native__text384__1000000__core+mutation.json b/benchmarks/competitive/results/raw/rust_native__text384__1000000__core+mutation.json new file mode 100644 index 0000000..dbde4c2 --- /dev/null +++ b/benchmarks/competitive/results/raw/rust_native__text384__1000000__core+mutation.json @@ -0,0 +1,372 @@ +{ + "label": "rust_native", + "runs": [ + { + "all_threads": 32, + "build": { + "first_query_ms": 1069.9223, + "index_s": 0.0, + "ingest_s": 0.7628488999999998, + "ready_s": 1.8327711999999998, + "total_s": 0.7628488999999998 + }, + "concurrent_clients": { + "clients": 8, + "qps": 15.43103788157774, + "scan_threads": 1 + }, + "d": 384, + "ds": "text384", + "k": 10, + "loop_allthreads": { + "mean_ms": 31.419985947712394, + "median_ms": 30.944699999999997, + "nq": 102, + "p95_ms": 35.4849, + "p99_ms": 39.5858, + "qps": 31.93727669150855, + "qps_runs": [ + 31.34563588079623, + 31.93727669150855, + 32.210043843238005 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 3535.34765625, + "rss_mb": 3527.73046875 + }, + "mem_after_first_query": { + "peak_mb": 3535.34765625, + "rss_mb": 3527.73046875 + }, + "mem_after_ingest": { + "peak_mb": 1983.83203125, + "rss_mb": 1942.6171875 + }, + "mem_after_sweep": { + "peak_mb": 3535.34765625, + "rss_mb": 3527.98046875 + }, + "mem_baseline": { + "peak_mb": 10.83984375, + "rss_mb": 10.83984375 + }, + "n": 1000000, + "queries": 2000, + "sweep": [ + { + "knob": null, + "mean_ms": 63.51418027210884, + "median_ms": 62.73819999999999, + "nq": 49, + "p95_ms": 68.93730000000001, + "p99_ms": 71.3337, + "qps_1t": 15.79984116322945, + "qps_1t_max": 15.816851958397413, + "qps_1t_min": 15.61839654592467, + "qps_1t_runs": [ + 15.61839654592467, + 15.816851958397413, + 15.79984116322945 + ], + "recall": 1.0 + } + ], + "system": "rust_native_exact", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + }, + { + "all_threads": 32, + "build": { + "cache_s": 1.0699223, + "first_query_ms": 0.0, + "index_s": 5.5632862, + "ingest_s": 0.7628488999999998, + "ready_s": 7.3960574, + "total_s": 7.3960574 + }, + "d": 384, + "ds": "text384", + "index_stats": { + "build_seconds": 5.5621877, + "index_bytes": 14176008, + "nlist": 4000, + "trained_n": 1000000 + }, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 3790.25390625, + "rss_mb": 3542.5 + }, + "mem_after_ingest": { + "peak_mb": 1983.83203125, + "rss_mb": 1942.6171875 + }, + "mem_after_sweep": { + "peak_mb": 3790.25390625, + "rss_mb": 3542.8515625 + }, + "mem_baseline": { + "peak_mb": 10.83984375, + "rss_mb": 10.83984375 + }, + "mutation": { + "after": { + "mean_ms": 10.060370670995669, + "median_ms": 10.0075, + "nq": 308, + "p95_ms": 11.6226, + "p99_ms": 12.583499999999999, + "qps_1t": 99.10852844010927, + "qps_1t_max": 101.6050530436107, + "qps_1t_min": 97.56923051677892, + "qps_1t_runs": [ + 101.6050530436107, + 97.56923051677892, + 99.10852844010927 + ], + "recall": 0.9538961038961039 + }, + "delete_10k": { + "per_s": 645561.1217270051, + "seconds": 0.0154904 + }, + "first_query_after_ms": 601.5160000000001, + "index_stats_after": { + "builds": 93479, + "churn": 30000, + "pending": 0, + "trained_n": 1000000 + }, + "insert_10k": { + "per_s": 36068.959522331555, + "seconds": 0.2772467 + }, + "mem_after": { + "peak_mb": 5023.4375, + "rss_mb": 3590.8828125 + }, + "recall_after": 0.9538961038961039, + "recall_before": 0.952054794520548, + "ref_knob": 256, + "update_10k": { + "per_s": 479409.3676590441, + "seconds": 0.020859 + } + }, + "n": 1000000, + "queries": 2000, + "ref_knob": 256, + "ref_recall_pre": 0.952054794520548, + "sweep": [ + { + "allthreads": { + "median_ms": 0.158, + "qps": 6133.458537666965, + "recall": 0.27565 + }, + "knob": 1, + "mean_ms": 0.17581004999999988, + "median_ms": 0.17129999999999998, + "nq": 2000, + "p95_ms": 0.2428, + "p99_ms": 0.2827, + "qps_1t": 5804.502610720169, + "qps_1t_max": 5871.959572732735, + "qps_1t_min": 5409.815135797173, + "qps_1t_runs": [ + 5409.815135797173, + 5871.959572732735, + 5804.502610720169 + ], + "recall": 0.27565 + }, + { + "allthreads": { + "median_ms": 0.20729999999999998, + "qps": 4672.436502756033, + "recall": 0.3806 + }, + "knob": 2, + "mean_ms": 0.2122564666666669, + "median_ms": 0.2063, + "nq": 2000, + "p95_ms": 0.2942, + "p99_ms": 0.3421, + "qps_1t": 4753.362290816402, + "qps_1t_max": 4780.507766412916, + "qps_1t_min": 4603.856558561967, + "qps_1t_runs": [ + 4603.856558561967, + 4753.362290816402, + 4780.507766412916 + ], + "recall": 0.3806 + }, + { + "allthreads": { + "median_ms": 0.32699999999999996, + "qps": 3006.9847745836405, + "recall": 0.49615 + }, + "knob": 4, + "mean_ms": 0.32868351666666656, + "median_ms": 0.3151, + "nq": 2000, + "p95_ms": 0.4435, + "p99_ms": 0.5546, + "qps_1t": 3049.643313718026, + "qps_1t_max": 3121.599894864514, + "qps_1t_min": 2960.377128363144, + "qps_1t_runs": [ + 3049.643313718026, + 2960.377128363144, + 3121.599894864514 + ], + "recall": 0.49615 + }, + { + "allthreads": { + "median_ms": 0.5069, + "qps": 1956.3372905471977, + "recall": 0.60455 + }, + "knob": 8, + "mean_ms": 0.5167608833333333, + "median_ms": 0.5096999999999999, + "nq": 2000, + "p95_ms": 0.6784, + "p99_ms": 0.7736999999999999, + "qps_1t": 1935.0837455868034, + "qps_1t_max": 1949.9591727298232, + "qps_1t_min": 1920.5731604894327, + "qps_1t_runs": [ + 1949.9591727298232, + 1920.5731604894327, + 1935.0837455868034 + ], + "recall": 0.60455 + }, + { + "allthreads": { + "median_ms": 0.825, + "qps": 1200.3922161527014, + "recall": 0.70215 + }, + "knob": 16, + "mean_ms": 0.8775537833333348, + "median_ms": 0.8694, + "nq": 2000, + "p95_ms": 1.1192000000000002, + "p99_ms": 1.2642, + "qps_1t": 1143.451803134876, + "qps_1t_max": 1148.3809207075149, + "qps_1t_min": 1126.9825313199697, + "qps_1t_runs": [ + 1126.9825313199697, + 1143.451803134876, + 1148.3809207075149 + ], + "recall": 0.70215 + }, + { + "allthreads": { + "median_ms": 0.8964, + "qps": 1085.6667614272806, + "recall": 0.79095 + }, + "knob": 32, + "mean_ms": 1.5492485733467605, + "median_ms": 1.5375, + "nq": 1986, + "p95_ms": 1.887, + "p99_ms": 2.0597, + "qps_1t": 647.4727404521726, + "qps_1t_max": 649.5527930410323, + "qps_1t_min": 639.48497052541, + "qps_1t_runs": [ + 647.4727404521726, + 639.48497052541, + 649.5527930410323 + ], + "recall": 0.790785498489426 + }, + { + "allthreads": { + "median_ms": 1.1093, + "qps": 872.4742742235514, + "recall": 0.8621 + }, + "knob": 64, + "mean_ms": 2.8110195242239597, + "median_ms": 2.7883, + "nq": 1149, + "p95_ms": 3.3551, + "p99_ms": 3.6066000000000003, + "qps_1t": 354.0230373790096, + "qps_1t_max": 360.8796531057676, + "qps_1t_min": 352.4382144983147, + "qps_1t_runs": [ + 352.4382144983147, + 354.0230373790096, + 360.8796531057676 + ], + "recall": 0.8592689295039164 + }, + { + "allthreads": { + "median_ms": 1.7229, + "qps": 548.307555633623, + "recall": 0.914856429463171 + }, + "knob": 128, + "mean_ms": 5.274658128654975, + "median_ms": 5.2091, + "nq": 570, + "p95_ms": 6.1895999999999995, + "p99_ms": 6.6829, + "qps_1t": 189.59317727996515, + "qps_1t_max": 191.19237122313282, + "qps_1t_min": 187.99859205545658, + "qps_1t_runs": [ + 191.19237122313282, + 187.99859205545658, + 189.59317727996515 + ], + "recall": 0.9149122807017543 + }, + { + "allthreads": { + "median_ms": 2.944, + "qps": 332.76858381862684, + "recall": 0.9527654867256637 + }, + "knob": 256, + "mean_ms": 10.030933675799082, + "median_ms": 9.9133, + "nq": 292, + "p95_ms": 11.668, + "p99_ms": 12.490599999999999, + "qps_1t": 99.81722167719664, + "qps_1t_max": 100.33586054854786, + "qps_1t_min": 98.93189802574274, + "qps_1t_runs": [ + 99.81722167719664, + 100.33586054854786, + 98.93189802574274 + ], + "recall": 0.952054794520548 + } + ], + "system": "rust_native_ivf", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + } + ], + "status": "ok", + "load_before_pct": 3.4124999999999983, + "load_contaminated": false, + "wall_s": 163.09614300727844, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/rust_native__text384__100000__core+mutation.json b/benchmarks/competitive/results/raw/rust_native__text384__100000__core+mutation.json new file mode 100644 index 0000000..792cbc3 --- /dev/null +++ b/benchmarks/competitive/results/raw/rust_native__text384__100000__core+mutation.json @@ -0,0 +1,394 @@ +{ + "label": "rust_native", + "runs": [ + { + "all_threads": 32, + "build": { + "first_query_ms": 79.10849999999999, + "index_s": 0.0, + "ingest_s": 0.058094299999999995, + "ready_s": 0.13720279999999999, + "total_s": 0.058094299999999995 + }, + "concurrent_clients": { + "clients": 8, + "qps": 160.3264631573395, + "scan_threads": 1 + }, + "d": 384, + "ds": "text384", + "k": 10, + "loop_allthreads": { + "mean_ms": 3.4037624275362415, + "median_ms": 3.2439999999999998, + "nq": 920, + "p95_ms": 4.3783, + "p99_ms": 5.5847, + "qps": 293.33622673385435, + "qps_runs": [ + 283.30055617749935, + 293.33622673385435, + 305.5851904913447 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 349.41796875, + "rss_mb": 349.41796875 + }, + "mem_after_first_query": { + "peak_mb": 349.41796875, + "rss_mb": 349.41796875 + }, + "mem_after_ingest": { + "peak_mb": 218.80859375, + "rss_mb": 191.68359375 + }, + "mem_after_sweep": { + "peak_mb": 349.7265625, + "rss_mb": 349.6015625 + }, + "mem_baseline": { + "peak_mb": 10.8515625, + "rss_mb": 10.8515625 + }, + "n": 100000, + "queries": 2000, + "sweep": [ + { + "knob": null, + "mean_ms": 6.148305056553553, + "median_ms": 6.061000000000001, + "nq": 501, + "p95_ms": 6.7933, + "p99_ms": 7.3915, + "qps_1t": 163.27361712134723, + "qps_1t_max": 163.96568921406532, + "qps_1t_min": 160.735783609724, + "qps_1t_runs": [ + 163.27361712134723, + 160.735783609724, + 163.96568921406532 + ], + "recall": 1.0 + } + ], + "system": "rust_native_exact", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + }, + { + "all_threads": 32, + "build": { + "cache_s": 0.0791085, + "first_query_ms": 0.0, + "index_s": 0.4714389, + "ingest_s": 0.058094299999999995, + "ready_s": 0.6086417, + "total_s": 0.6086417 + }, + "d": 384, + "ds": "text384", + "index_stats": { + "build_seconds": 0.4712903, + "index_bytes": 2753168, + "nlist": 1265, + "trained_n": 100000 + }, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 431.03515625, + "rss_mb": 353.34375 + }, + "mem_after_ingest": { + "peak_mb": 218.80859375, + "rss_mb": 191.68359375 + }, + "mem_after_sweep": { + "peak_mb": 431.03515625, + "rss_mb": 353.45703125 + }, + "mem_baseline": { + "peak_mb": 10.8515625, + "rss_mb": 10.8515625 + }, + "mutation": { + "after": { + "mean_ms": 1.300234283333337, + "median_ms": 1.2962, + "nq": 2000, + "p95_ms": 1.5609, + "p99_ms": 1.6765, + "qps_1t": 768.3209294962226, + "qps_1t_max": 772.804066618648, + "qps_1t_min": 766.1811619267572, + "qps_1t_runs": [ + 766.1811619267572, + 768.3209294962226, + 772.804066618648 + ], + "recall": 0.94895 + }, + "delete_10k": { + "per_s": 1032983.1520447901, + "seconds": 0.0096807 + }, + "first_query_after_ms": 192.1449, + "index_stats_after": { + "builds": 112427, + "churn": 30000, + "pending": 0, + "trained_n": 100000 + }, + "insert_10k": { + "per_s": 256464.83738846987, + "seconds": 0.0389917 + }, + "mem_after": { + "peak_mb": 515.2578125, + "rss_mb": 400.87109375 + }, + "recall_after": 0.94895, + "recall_before": 0.9529, + "ref_knob": 128, + "update_10k": { + "per_s": 606159.7958453808, + "seconds": 0.0164973 + } + }, + "n": 100000, + "queries": 2000, + "ref_knob": 128, + "ref_recall_pre": 0.9529, + "sweep": [ + { + "allthreads": { + "median_ms": 0.039799999999999995, + "qps": 24453.1959715805, + "recall": 0.273 + }, + "knob": 1, + "mean_ms": 0.04075911666666657, + "median_ms": 0.039700000000000006, + "nq": 2000, + "p95_ms": 0.0514, + "p99_ms": 0.0601, + "qps_1t": 24669.18621288522, + "qps_1t_max": 24957.4164082534, + "qps_1t_min": 23996.52530313614, + "qps_1t_runs": [ + 23996.52530313614, + 24957.4164082534, + 24669.18621288522 + ], + "recall": 0.273 + }, + { + "allthreads": { + "median_ms": 0.0538, + "qps": 18451.64837800783, + "recall": 0.39185 + }, + "knob": 2, + "mean_ms": 0.056549433333333364, + "median_ms": 0.0555, + "nq": 2000, + "p95_ms": 0.0708, + "p99_ms": 0.0813, + "qps_1t": 17732.178495655146, + "qps_1t_max": 17841.897592860394, + "qps_1t_min": 17480.74495942719, + "qps_1t_runs": [ + 17732.178495655146, + 17480.74495942719, + 17841.897592860394 + ], + "recall": 0.39185 + }, + { + "allthreads": { + "median_ms": 0.0772, + "qps": 12737.263373489703, + "recall": 0.51675 + }, + "knob": 4, + "mean_ms": 0.07671965000000035, + "median_ms": 0.0751, + "nq": 2000, + "p95_ms": 0.0992, + "p99_ms": 0.1108, + "qps_1t": 13134.201328918492, + "qps_1t_max": 13139.25773705193, + "qps_1t_min": 12834.655623728966, + "qps_1t_runs": [ + 12834.655623728966, + 13134.201328918492, + 13139.25773705193 + ], + "recall": 0.51675 + }, + { + "allthreads": { + "median_ms": 0.1171, + "qps": 8381.091586892628, + "recall": 0.64025 + }, + "knob": 8, + "mean_ms": 0.11834945000000005, + "median_ms": 0.11620000000000001, + "nq": 2000, + "p95_ms": 0.15330000000000002, + "p99_ms": 0.177, + "qps_1t": 8489.24900283158, + "qps_1t_max": 8498.266990903881, + "qps_1t_min": 8362.515026394198, + "qps_1t_runs": [ + 8362.515026394198, + 8498.266990903881, + 8489.24900283158 + ], + "recall": 0.64025 + }, + { + "allthreads": { + "median_ms": 0.196, + "qps": 5054.506534845138, + "recall": 0.74505 + }, + "knob": 16, + "mean_ms": 0.20743258333333306, + "median_ms": 0.2029, + "nq": 2000, + "p95_ms": 0.2688, + "p99_ms": 0.3103, + "qps_1t": 4821.191644296347, + "qps_1t_max": 4992.871427818932, + "qps_1t_min": 4659.949504787166, + "qps_1t_runs": [ + 4659.949504787166, + 4821.191644296347, + 4992.871427818932 + ], + "recall": 0.74505 + }, + { + "allthreads": { + "median_ms": 0.35100000000000003, + "qps": 2837.647811300961, + "recall": 0.83455 + }, + "knob": 32, + "mean_ms": 0.36021103333333326, + "median_ms": 0.3549, + "nq": 2000, + "p95_ms": 0.45790000000000003, + "p99_ms": 0.5084, + "qps_1t": 2788.6764676560247, + "qps_1t_max": 2792.409616332693, + "qps_1t_min": 2747.808348061581, + "qps_1t_runs": [ + 2788.6764676560247, + 2747.808348061581, + 2792.409616332693 + ], + "recall": 0.83455 + }, + { + "allthreads": { + "median_ms": 0.5975, + "qps": 1676.1664798718773, + "recall": 0.90705 + }, + "knob": 64, + "mean_ms": 0.6645801333333337, + "median_ms": 0.656, + "nq": 2000, + "p95_ms": 0.8297, + "p99_ms": 0.9142, + "qps_1t": 1506.1073405713832, + "qps_1t_max": 1528.7722972395031, + "qps_1t_min": 1480.0399225968717, + "qps_1t_runs": [ + 1506.1073405713832, + 1528.7722972395031, + 1480.0399225968717 + ], + "recall": 0.90705 + }, + { + "allthreads": { + "median_ms": 0.6681, + "qps": 1471.7698717080295, + "recall": 0.9529 + }, + "knob": 128, + "mean_ms": 1.3025767833333324, + "median_ms": 1.297, + "nq": 2000, + "p95_ms": 1.5675000000000001, + "p99_ms": 1.7002, + "qps_1t": 768.5030854822505, + "qps_1t_max": 774.0316409428224, + "qps_1t_min": 760.7093325456456, + "qps_1t_runs": [ + 768.5030854822505, + 760.7093325456456, + 774.0316409428224 + ], + "recall": 0.9529 + }, + { + "allthreads": { + "median_ms": 0.8917, + "qps": 1094.7782362465746, + "recall": 0.9814 + }, + "knob": 256, + "mean_ms": 2.5845094350783757, + "median_ms": 2.5845, + "nq": 1127, + "p95_ms": 2.9424, + "p99_ms": 3.1992000000000003, + "qps_1t": 386.975323953629, + "qps_1t_max": 388.54662462962256, + "qps_1t_min": 385.2539421699303, + "qps_1t_runs": [ + 388.54662462962256, + 385.2539421699303, + 386.975323953629 + ], + "recall": 0.9807453416149068 + }, + { + "allthreads": { + "median_ms": 3.1554, + "qps": 310.45320705826, + "recall": 0.9998734177215189 + }, + "knob": 1265, + "mean_ms": 6.188903120567375, + "median_ms": 6.0545, + "nq": 470, + "p95_ms": 6.985, + "p99_ms": 8.324, + "qps_1t": 161.0875093156563, + "qps_1t_max": 162.91040336823824, + "qps_1t_min": 160.75723086024405, + "qps_1t_runs": [ + 162.91040336823824, + 160.75723086024405, + 161.0875093156563 + ], + "recall": 1.0 + } + ], + "system": "rust_native_ivf", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + } + ], + "status": "ok", + "load_before_pct": 4.084374999999999, + "load_contaminated": false, + "wall_s": 92.50139880180359, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/rust_native__text384__10000__core.json b/benchmarks/competitive/results/raw/rust_native__text384__10000__core.json new file mode 100644 index 0000000..245d143 --- /dev/null +++ b/benchmarks/competitive/results/raw/rust_native__text384__10000__core.json @@ -0,0 +1,350 @@ +{ + "label": "rust_native", + "runs": [ + { + "all_threads": 32, + "build": { + "first_query_ms": 7.3756, + "index_s": 0.0, + "ingest_s": 0.0043495, + "ready_s": 0.0117251, + "total_s": 0.0043495 + }, + "concurrent_clients": { + "clients": 8, + "qps": 3423.40306805383, + "scan_threads": 1 + }, + "d": 384, + "ds": "text384", + "k": 10, + "loop_allthreads": { + "mean_ms": 0.2615493333333339, + "median_ms": 0.2612, + "nq": 2000, + "p95_ms": 0.2938, + "p99_ms": 0.3441, + "qps": 3836.2003816252136, + "qps_runs": [ + 3724.243596814881, + 3836.2003816252136, + 3914.468855507168 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 46.09765625, + "rss_mb": 46.09765625 + }, + "mem_after_first_query": { + "peak_mb": 46.09765625, + "rss_mb": 46.09765625 + }, + "mem_after_ingest": { + "peak_mb": 40.7578125, + "rss_mb": 30.12890625 + }, + "mem_after_sweep": { + "peak_mb": 46.625, + "rss_mb": 46.5 + }, + "mem_baseline": { + "peak_mb": 10.84765625, + "rss_mb": 10.84765625 + }, + "n": 10000, + "queries": 2000, + "sweep": [ + { + "knob": null, + "mean_ms": 0.28007021666666715, + "median_ms": 0.2744, + "nq": 2000, + "p95_ms": 0.3085, + "p99_ms": 0.3507, + "qps_1t": 3617.4830059692117, + "qps_1t_max": 3629.814131737575, + "qps_1t_min": 3468.859957440568, + "qps_1t_runs": [ + 3468.859957440568, + 3617.4830059692117, + 3629.814131737575 + ], + "recall": 1.0 + } + ], + "system": "rust_native_exact", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + }, + { + "all_threads": 32, + "build": { + "cache_s": 0.0073756, + "first_query_ms": 0.0, + "index_s": 0.0535153, + "ingest_s": 0.0043495, + "ready_s": 0.0652404, + "total_s": 0.0652404 + }, + "d": 384, + "ds": "text384", + "index_stats": { + "build_seconds": 0.0534555, + "index_bytes": 697608, + "nlist": 400, + "trained_n": 10000 + }, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 63.44921875, + "rss_mb": 47.1484375 + }, + "mem_after_ingest": { + "peak_mb": 40.7578125, + "rss_mb": 30.12890625 + }, + "mem_after_sweep": { + "peak_mb": 63.44921875, + "rss_mb": 47.609375 + }, + "mem_baseline": { + "peak_mb": 10.84765625, + "rss_mb": 10.84765625 + }, + "n": 10000, + "queries": 2000, + "ref_knob": 64, + "ref_recall_pre": 0.9531, + "sweep": [ + { + "allthreads": { + "median_ms": 0.013, + "qps": 74975.16447676686, + "recall": 0.28205 + }, + "knob": 1, + "mean_ms": 0.013493199999999768, + "median_ms": 0.0129, + "nq": 2000, + "p95_ms": 0.0165, + "p99_ms": 0.0264, + "qps_1t": 73753.11145938959, + "qps_1t_max": 74847.49822237175, + "qps_1t_min": 73744.40925197362, + "qps_1t_runs": [ + 73744.40925197362, + 74847.49822237175, + 73753.11145938959 + ], + "recall": 0.28205 + }, + { + "allthreads": { + "median_ms": 0.0149, + "qps": 64299.17118368329, + "recall": 0.41255 + }, + "knob": 2, + "mean_ms": 0.014960416666666485, + "median_ms": 0.0146, + "nq": 2000, + "p95_ms": 0.018000000000000002, + "p99_ms": 0.0219, + "qps_1t": 68011.48033788093, + "qps_1t_max": 68450.48634070536, + "qps_1t_min": 64231.23243677243, + "qps_1t_runs": [ + 64231.23243677243, + 68450.48634070536, + 68011.48033788093 + ], + "recall": 0.41255 + }, + { + "allthreads": { + "median_ms": 0.0157, + "qps": 62532.24318789372, + "recall": 0.54975 + }, + "knob": 4, + "mean_ms": 0.01879223333333348, + "median_ms": 0.018099999999999998, + "nq": 2000, + "p95_ms": 0.0247, + "p99_ms": 0.0304, + "qps_1t": 53059.827608620035, + "qps_1t_max": 57683.26694950694, + "qps_1t_min": 49519.65930474398, + "qps_1t_runs": [ + 53059.827608620035, + 49519.65930474398, + 57683.26694950694 + ], + "recall": 0.54975 + }, + { + "allthreads": { + "median_ms": 0.0218, + "qps": 44628.11392664926, + "recall": 0.6894 + }, + "knob": 8, + "mean_ms": 0.02193473333333346, + "median_ms": 0.0214, + "nq": 2000, + "p95_ms": 0.026699999999999998, + "p99_ms": 0.032, + "qps_1t": 45586.01967948471, + "qps_1t_max": 45604.21018068391, + "qps_1t_min": 45579.16302982923, + "qps_1t_runs": [ + 45586.01967948471, + 45579.16302982923, + 45604.21018068391 + ], + "recall": 0.6894 + }, + { + "allthreads": { + "median_ms": 0.0307, + "qps": 32141.52556536941, + "recall": 0.80415 + }, + "knob": 16, + "mean_ms": 0.0311168166666667, + "median_ms": 0.030800000000000004, + "nq": 2000, + "p95_ms": 0.0371, + "p99_ms": 0.0435, + "qps_1t": 32066.955803718098, + "qps_1t_max": 32367.95895742802, + "qps_1t_min": 31978.561572321916, + "qps_1t_runs": [ + 32367.95895742802, + 32066.955803718098, + 31978.561572321916 + ], + "recall": 0.80415 + }, + { + "allthreads": { + "median_ms": 0.0506, + "qps": 19418.90857966218, + "recall": 0.8948 + }, + "knob": 32, + "mean_ms": 0.054750150000000164, + "median_ms": 0.0535, + "nq": 2000, + "p95_ms": 0.0688, + "p99_ms": 0.0818, + "qps_1t": 18089.675950589823, + "qps_1t_max": 19270.86745882779, + "qps_1t_min": 17519.731597711925, + "qps_1t_runs": [ + 18089.675950589823, + 17519.731597711925, + 19270.86745882779 + ], + "recall": 0.8948 + }, + { + "allthreads": { + "median_ms": 0.0885, + "qps": 11124.924559105326, + "recall": 0.9531 + }, + "knob": 64, + "mean_ms": 0.09016785000000026, + "median_ms": 0.0891, + "nq": 2000, + "p95_ms": 0.1038, + "p99_ms": 0.1254, + "qps_1t": 11118.338030831155, + "qps_1t_max": 11142.110939770182, + "qps_1t_min": 11011.70599405702, + "qps_1t_runs": [ + 11118.338030831155, + 11142.110939770182, + 11011.70599405702 + ], + "recall": 0.9531 + }, + { + "allthreads": { + "median_ms": 0.1587, + "qps": 6285.841080108644, + "recall": 0.98575 + }, + "knob": 128, + "mean_ms": 0.1609034833333329, + "median_ms": 0.1587, + "nq": 2000, + "p95_ms": 0.1858, + "p99_ms": 0.2132, + "qps_1t": 6288.175118131226, + "qps_1t_max": 6391.391562532255, + "qps_1t_min": 5980.097637054107, + "qps_1t_runs": [ + 6391.391562532255, + 5980.097637054107, + 6288.175118131226 + ], + "recall": 0.98575 + }, + { + "allthreads": { + "median_ms": 0.2221, + "qps": 4352.253085257807, + "recall": 0.9979 + }, + "knob": 256, + "mean_ms": 0.29612661666666823, + "median_ms": 0.2924, + "nq": 2000, + "p95_ms": 0.3412, + "p99_ms": 0.3877, + "qps_1t": 3438.407632164651, + "qps_1t_max": 3449.474257004806, + "qps_1t_min": 3250.464897742001, + "qps_1t_runs": [ + 3449.474257004806, + 3250.464897742001, + 3438.407632164651 + ], + "recall": 0.9979 + }, + { + "allthreads": { + "median_ms": 0.2497, + "qps": 3937.1241276809405, + "recall": 1.0 + }, + "knob": 400, + "mean_ms": 0.2279436166666678, + "median_ms": 0.2276, + "nq": 2000, + "p95_ms": 0.24649999999999997, + "p99_ms": 0.28350000000000003, + "qps_1t": 4401.856967380271, + "qps_1t_max": 4409.43937505897, + "qps_1t_min": 4350.326579016282, + "qps_1t_runs": [ + 4401.856967380271, + 4409.43937505897, + 4350.326579016282 + ], + "recall": 1.0 + } + ], + "system": "rust_native_ivf", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + } + ], + "status": "ok", + "load_before_pct": 6.084375, + "load_contaminated": false, + "wall_s": 14.545634746551514, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/sqlite_vec__glove__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/sqlite_vec__glove__100000__core+mutation+saveload.json new file mode 100644 index 0000000..f7af81d --- /dev/null +++ b/benchmarks/competitive/results/raw/sqlite_vec__glove__100000__core+mutation+saveload.json @@ -0,0 +1,135 @@ +{ + "system": "sqlite_vec", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 08:27:18", + "mem_baseline": { + "rss_mb": 35.01953125, + "peak_mb": 35.01953125 + }, + "version": "sqlite-vec 0.1.9 / sqlite 3.45.3", + "build": { + "ingest_s": 0.5290911999763921, + "index_s": 1.6999547369778156e-06, + "total_s": 0.5290928999311291, + "first_query_ms": 25.250599952414632, + "ready_s": 0.5543434998835437, + "total_s_runs": [ + 0.5290928999311291, + 0.5210702000767924, + 0.4929345999844372 + ] + }, + "mem_after_build": { + "rss_mb": 38.76171875, + "peak_mb": 54.02734375 + }, + "mem_after_first_query": { + "rss_mb": 39.03125, + "peak_mb": 54.02734375 + }, + "sweep": [ + { + "nq": 125, + "recall_queries": 125, + "recall": 1.0, + "qps_1t_runs": [ + 39.98742922997561, + 40.43658834708476, + 40.07162112720647 + ], + "qps_1t": 40.07162112720647, + "qps_1t_min": 39.98742922997561, + "qps_1t_max": 40.43658834708476, + "median_ms": 24.331900000106543, + "p95_ms": 27.37560999230482, + "p99_ms": 34.3428380077239, + "mean_ms": 24.89775146730244, + "knob": null + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 125, + "threads": 32, + "qps": 40.51256103382289, + "median_ms": 24.473500030580908, + "p95_ms": 25.93897999613546, + "p99_ms": 27.342113993363455, + "mean_ms": 24.618752799462527 + }, + "mem_after_sweep": { + "rss_mb": 40.65234375, + "peak_mb": 54.02734375 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.08024839998688549, + "per_s": 124613.07641814963, + "batch": 1000 + }, + "update_10k": { + "seconds": 2.47424809995573, + "per_s": 4041.6318800765866, + "batch": 1000 + }, + "delete_10k": { + "seconds": 2.067099599982612, + "per_s": 4837.69625812134, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 48.06640625, + "peak_mb": 57.9609375 + }, + "first_query_after_ms": 28.84109999286011, + "after": { + "nq": 99, + "recall_queries": 99, + "recall": 1.0, + "qps_1t_runs": [ + 33.202292713679284, + 33.191395969538874, + 33.58924378676408 + ], + "qps_1t": 33.202292713679284, + "qps_1t_min": 33.191395969538874, + "qps_1t_max": 33.58924378676408, + "median_ms": 29.00550002232194, + "p95_ms": 36.38529997551814, + "p99_ms": 43.65457602776593, + "mean_ms": 30.006042425873492 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 0.0003665999975055456, + "disk_mb": 48.875, + "load_s": 0.00038629997288808227, + "first_query_after_load_ms": 30.68840003106743, + "time_to_ready_s": 0.031074700003955513 + }, + "mem_end": { + "rss_mb": 36.85546875, + "peak_mb": 57.9609375 + }, + "status": "ok", + "finished": "2026-11-16 08:27:55", + "load_before_pct": 4.434374999999998, + "load_contaminated": false, + "wall_s": 37.547940254211426, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/sqlite_vec__glove__100000__filter.json b/benchmarks/competitive/results/raw/sqlite_vec__glove__100000__filter.json new file mode 100644 index 0000000..0334676 --- /dev/null +++ b/benchmarks/competitive/results/raw/sqlite_vec__glove__100000__filter.json @@ -0,0 +1,66 @@ +{ + "system": "sqlite_vec", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 08:55:42", + "filter": { + "build_with_metadata_s": 0.5791347998892888, + "mem": { + "rss_mb": 38.34375, + "peak_mb": 53.60546875 + }, + "subsets": { + "f10": [ + { + "knob": null, + "nq": 142, + "recall": 1.0, + "qps_1t": 46.12656550824415, + "qps_1t_runs": [ + 45.99653750272907, + 46.89542075728957, + 46.12656550824415 + ], + "median_ms": 21.14765002625063, + "p95_ms": 24.147375021129847, + "p99_ms": 28.752274985890836, + "mean_ms": 21.581430518026835 + } + ], + "f01": [ + { + "knob": null, + "nq": 135, + "recall": 1.0, + "qps_1t": 47.97134170944395, + "qps_1t_runs": [ + 47.366501457312545, + 48.01380236947476, + 47.97134170944395 + ], + "median_ms": 20.34819999244064, + "p95_ms": 24.34583998983725, + "p99_ms": 29.691936005838212, + "mean_ms": 20.928363457040426 + } + ] + } + }, + "mem_end": { + "rss_mb": 36.28515625, + "peak_mb": 53.60546875 + }, + "status": "ok", + "finished": "2026-11-16 08:56:00", + "load_before_pct": 4.171875, + "load_contaminated": false, + "wall_s": 19.067617893218994, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/sqlite_vec__sift__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/sqlite_vec__sift__100000__core+mutation+saveload.json new file mode 100644 index 0000000..9524381 --- /dev/null +++ b/benchmarks/competitive/results/raw/sqlite_vec__sift__100000__core+mutation+saveload.json @@ -0,0 +1,135 @@ +{ + "system": "sqlite_vec", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 11:17:45", + "mem_baseline": { + "rss_mb": 35.421875, + "peak_mb": 35.421875 + }, + "version": "sqlite-vec 0.1.9 / sqlite 3.45.3", + "build": { + "ingest_s": 0.5556321000331081, + "index_s": 2.400018274784088e-06, + "total_s": 0.5556345000513829, + "first_query_ms": 32.17329998733476, + "ready_s": 0.5878078000387177, + "total_s_runs": [ + 0.5556345000513829, + 1.9912856000592, + 2.163471699866932 + ] + }, + "mem_after_build": { + "rss_mb": 39.12890625, + "peak_mb": 58.66796875 + }, + "mem_after_first_query": { + "rss_mb": 39.51953125, + "peak_mb": 58.66796875 + }, + "sweep": [ + { + "nq": 93, + "recall_queries": 93, + "recall": 1.0, + "qps_1t_runs": [ + 22.39173649564215, + 21.024955742055827, + 21.553401877047236 + ], + "qps_1t": 21.553401877047236, + "qps_1t_min": 21.024955742055827, + "qps_1t_max": 22.39173649564215, + "median_ms": 46.99310002615675, + "p95_ms": 52.64740001875907, + "p99_ms": 57.69439799711084, + "mean_ms": 46.2060820792825, + "knob": null + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 35, + "threads": 32, + "qps": 11.035035922920613, + "median_ms": 89.86820001155138, + "p95_ms": 108.1220000050962, + "p99_ms": 113.00987200811505, + "mean_ms": 90.91911619263036 + }, + "mem_after_sweep": { + "rss_mb": 41.1015625, + "peak_mb": 58.66796875 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.307022099965252, + "per_s": 32570.945222287828, + "batch": 1000 + }, + "update_10k": { + "seconds": 10.963700100022834, + "per_s": 912.1008335478981, + "batch": 1000 + }, + "delete_10k": { + "seconds": 8.295781599998008, + "per_s": 1205.43192699316, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 50.640625, + "peak_mb": 62.6171875 + }, + "first_query_after_ms": 109.39749999670312, + "after": { + "nq": 56, + "recall_queries": 56, + "recall": 1.0, + "qps_1t_runs": [ + 18.043468713921857, + 18.005915586202843, + 17.695198498405215 + ], + "qps_1t": 18.005915586202843, + "qps_1t_min": 17.695198498405215, + "qps_1t_max": 18.043468713921857, + "median_ms": 55.82765000872314, + "p95_ms": 66.20149501541165, + "p99_ms": 83.73340703430587, + "mean_ms": 55.82384166787268 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 0.0011999999987892807, + "disk_mb": 61.7890625, + "load_s": 0.0007896000170148909, + "first_query_after_load_ms": 104.93339999811724, + "time_to_ready_s": 0.10572300001513213 + }, + "mem_end": { + "rss_mb": 36.95703125, + "peak_mb": 62.6171875 + }, + "status": "ok", + "finished": "2026-11-16 11:18:48", + "load_before_pct": 12.049999999999994, + "load_contaminated": false, + "wall_s": 63.71775794029236, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/sqlite_vec__text384__1000000__core+mutation.json b/benchmarks/competitive/results/raw/sqlite_vec__text384__1000000__core+mutation.json new file mode 100644 index 0000000..6522999 --- /dev/null +++ b/benchmarks/competitive/results/raw/sqlite_vec__text384__1000000__core+mutation.json @@ -0,0 +1,118 @@ +{ + "system": "sqlite_vec", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation" + ], + "started": "2026-11-16 09:49:32", + "mem_baseline": { + "rss_mb": 37.296875, + "peak_mb": 37.296875 + }, + "version": "sqlite-vec 0.1.9 / sqlite 3.45.3", + "build": { + "ingest_s": 9.600988200050779, + "index_s": 1.9999570213258266e-06, + "total_s": 9.6009902000078, + "first_query_ms": 942.8271999931894, + "ready_s": 10.54381740000099 + }, + "mem_after_build": { + "rss_mb": 41.10546875, + "peak_mb": 99.70703125 + }, + "mem_after_first_query": { + "rss_mb": 41.140625, + "peak_mb": 99.70703125 + }, + "sweep": [ + { + "nq": 30, + "recall_queries": 30, + "recall": 1.0, + "qps_1t_runs": [ + 1.0509534128538331 + ], + "qps_1t": 1.0509534128538331, + "qps_1t_min": 1.0509534128538331, + "qps_1t_max": 1.0509534128538331, + "median_ms": 946.5698999993037, + "p95_ms": 988.4569849993568, + "p99_ms": 990.9356639964972, + "mean_ms": 951.5169633300198, + "knob": null + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 30, + "threads": 32, + "qps": 1.0638727873009715, + "median_ms": 937.6317000424024, + "p95_ms": 963.8271150091896, + "p99_ms": 971.6743419924751, + "mean_ms": 939.9620066765541 + }, + "mem_after_sweep": { + "rss_mb": 42.5234375, + "peak_mb": 99.70703125 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.1939520000014454, + "per_s": 51559.14865495316, + "batch": 1000 + }, + "update_10k": { + "seconds": 10.536050499998964, + "per_s": 949.122254112296, + "batch": 1000 + }, + "delete_10k": { + "seconds": 8.067454600008205, + "per_s": 1239.5483452723527, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 72.34765625, + "peak_mb": 99.70703125 + }, + "first_query_after_ms": 957.5068000121973, + "after": { + "nq": 30, + "recall_queries": 30, + "recall": 1.0, + "qps_1t_runs": [ + 1.0352260402141704 + ], + "qps_1t": 1.0352260402141704, + "qps_1t_min": 1.0352260402141704, + "qps_1t_max": 1.0352260402141704, + "median_ms": 963.0893999710679, + "p95_ms": 991.10578498221, + "p99_ms": 993.1055910210125, + "mean_ms": 965.972609994545 + }, + "recall_after": 1.0 + }, + "mem_end": { + "rss_mb": 38.8984375, + "peak_mb": 99.70703125 + }, + "status": "ok", + "finished": "2026-11-16 09:51:55", + "load_before_pct": 4.396875, + "load_contaminated": false, + "wall_s": 142.72977542877197, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/sqlite_vec__text384__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/sqlite_vec__text384__100000__core+mutation+saveload.json new file mode 100644 index 0000000..71cdac1 --- /dev/null +++ b/benchmarks/competitive/results/raw/sqlite_vec__text384__100000__core+mutation+saveload.json @@ -0,0 +1,135 @@ +{ + "system": "sqlite_vec", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 08:06:05", + "mem_baseline": { + "rss_mb": 37.41796875, + "peak_mb": 37.41796875 + }, + "version": "sqlite-vec 0.1.9 / sqlite 3.45.3", + "build": { + "ingest_s": 0.966977899952326, + "index_s": 2.100015990436077e-06, + "total_s": 0.9669799999683164, + "first_query_ms": 98.54739997535944, + "ready_s": 1.0655273999436758, + "total_s_runs": [ + 0.9669799999683164, + 0.9519995000446215, + 0.9635596000007354 + ] + }, + "mem_after_build": { + "rss_mb": 41.12890625, + "peak_mb": 99.73046875 + }, + "mem_after_first_query": { + "rss_mb": 41.1640625, + "peak_mb": 99.73046875 + }, + "sweep": [ + { + "nq": 30, + "recall_queries": 30, + "recall": 1.0, + "qps_1t_runs": [ + 10.174947023143666, + 10.198895459094988, + 10.27547099182717 + ], + "qps_1t": 10.198895459094988, + "qps_1t_min": 10.174947023143666, + "qps_1t_max": 10.27547099182717, + "median_ms": 95.47854997799732, + "p95_ms": 115.11505498201586, + "p99_ms": 121.78105900005903, + "mean_ms": 97.88319444051012, + "knob": null + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 31, + "threads": 32, + "qps": 10.361603934464103, + "median_ms": 95.295500010252, + "p95_ms": 99.65811998117715, + "p99_ms": 111.8514999793842, + "mean_ms": 96.14113225634159 + }, + "mem_after_sweep": { + "rss_mb": 42.49609375, + "peak_mb": 99.73046875 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.16956050001317635, + "per_s": 58975.99971233225, + "batch": 1000 + }, + "update_10k": { + "seconds": 9.847085700021125, + "per_s": 1015.5288889156867, + "batch": 1000 + }, + "delete_10k": { + "seconds": 7.372649600030854, + "per_s": 1356.3644744432381, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 72.2734375, + "peak_mb": 103.69140625 + }, + "first_query_after_ms": 110.41389999445528, + "after": { + "nq": 30, + "recall_queries": 30, + "recall": 1.0, + "qps_1t_runs": [ + 9.00273548151586, + 8.826417987026353, + 9.040549395469546 + ], + "qps_1t": 9.00273548151586, + "qps_1t_min": 8.826417987026353, + "qps_1t_max": 9.040549395469546, + "median_ms": 109.6875499933958, + "p95_ms": 122.77052999706939, + "p99_ms": 127.13630900194403, + "mean_ms": 111.66211333087024 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 0.0004209000035189092, + "disk_mb": 179.953125, + "load_s": 0.00042400002712383866, + "first_query_after_load_ms": 109.11669995402917, + "time_to_ready_s": 0.10954069998115301 + }, + "mem_end": { + "rss_mb": 37.5625, + "peak_mb": 103.69140625 + }, + "status": "ok", + "finished": "2026-11-16 08:07:03", + "load_before_pct": 3.171874999999996, + "load_contaminated": false, + "wall_s": 58.392027854919434, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/sqlite_vec__text384__100000__filter.json b/benchmarks/competitive/results/raw/sqlite_vec__text384__100000__filter.json new file mode 100644 index 0000000..d6db12f --- /dev/null +++ b/benchmarks/competitive/results/raw/sqlite_vec__text384__100000__filter.json @@ -0,0 +1,66 @@ +{ + "system": "sqlite_vec", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 08:44:51", + "filter": { + "build_with_metadata_s": 1.007789200055413, + "mem": { + "rss_mb": 40.53515625, + "peak_mb": 99.1328125 + }, + "subsets": { + "f10": [ + { + "knob": null, + "nq": 37, + "recall": 1.0, + "qps_1t": 12.954744350426363, + "qps_1t_runs": [ + 12.510534631107083, + 13.056111391789301, + 12.954744350426363 + ], + "median_ms": 75.81910002045333, + "p95_ms": 85.41519998107105, + "p99_ms": 88.92741004237907, + "mean_ms": 77.90563873860667 + } + ], + "f01": [ + { + "knob": null, + "nq": 40, + "recall": 1.0, + "qps_1t": 13.286640975914697, + "qps_1t_runs": [ + 13.404244466664046, + 13.286640975914697, + 12.986777707918678 + ], + "median_ms": 73.68184998631477, + "p95_ms": 84.5282300200779, + "p99_ms": 94.23895399377216, + "mean_ms": 75.62273250126357 + } + ] + } + }, + "mem_end": { + "rss_mb": 36.25390625, + "peak_mb": 99.1328125 + }, + "status": "ok", + "finished": "2026-11-16 08:45:12", + "load_before_pct": 3.9343749999999957, + "load_contaminated": false, + "wall_s": 21.27250075340271, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/sqlite_vec__text384__10000__core.json b/benchmarks/competitive/results/raw/sqlite_vec__text384__10000__core.json new file mode 100644 index 0000000..878e6c4 --- /dev/null +++ b/benchmarks/competitive/results/raw/sqlite_vec__text384__10000__core.json @@ -0,0 +1,79 @@ +{ + "system": "sqlite_vec", + "ds": "text384", + "n": 10000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 08:58:29", + "mem_baseline": { + "rss_mb": 37.29296875, + "peak_mb": 37.29296875 + }, + "version": "sqlite-vec 0.1.9 / sqlite 3.45.3", + "build": { + "ingest_s": 0.12505999999120831, + "index_s": 2.400018274784088e-06, + "total_s": 0.1250624000094831, + "first_query_ms": 9.759800042957067, + "ready_s": 0.13482220005244017 + }, + "mem_after_build": { + "rss_mb": 40.8359375, + "peak_mb": 55.48828125 + }, + "mem_after_first_query": { + "rss_mb": 40.87890625, + "peak_mb": 55.48828125 + }, + "sweep": [ + { + "nq": 292, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 103.71207908579683, + 102.51495514706376, + 104.71020944458607 + ], + "qps_1t": 103.71207908579683, + "qps_1t_min": 102.51495514706376, + "qps_1t_max": 104.71020944458607, + "median_ms": 9.35500001651235, + "p95_ms": 11.23795002058614, + "p99_ms": 13.979025025037117, + "mean_ms": 9.648973288556208, + "knob": null + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 293, + "threads": 32, + "qps": 102.81119130601117, + "median_ms": 9.56530001712963, + "p95_ms": 10.552820021985099, + "p99_ms": 11.41818599542603, + "mean_ms": 9.673479749713584 + }, + "mem_after_sweep": { + "rss_mb": 42.3671875, + "peak_mb": 55.48828125 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 38.59375, + "peak_mb": 55.48828125 + }, + "status": "ok", + "finished": "2026-11-16 08:58:54", + "load_before_pct": 4.631250000000001, + "load_contaminated": false, + "wall_s": 24.769706964492798, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/usearch__glove__1000000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/usearch__glove__1000000__core+mutation+saveload.json new file mode 100644 index 0000000..c868069 --- /dev/null +++ b/benchmarks/competitive/results/raw/usearch__glove__1000000__core+mutation+saveload.json @@ -0,0 +1,463 @@ +{ + "system": "usearch", + "ds": "glove", + "n": 1000000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 10:22:12", + "mem_baseline": { + "rss_mb": 35.15625, + "peak_mb": 35.15625 + }, + "version": "usearch 2.26.6", + "build": { + "ingest_s": 104.53917879989604, + "index_s": 4.199973773211241e-06, + "total_s": 104.53918299986981, + "first_query_ms": 5.705899966415018, + "ready_s": 104.54488889983622 + }, + "mem_after_build": { + "rss_mb": 665.5546875, + "peak_mb": 674.57421875 + }, + "mem_after_first_query": { + "rss_mb": 665.6875, + "peak_mb": 674.57421875 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.47335, + "qps_1t_runs": [ + 10941.613329159567, + 11107.192721715397, + 10420.118326591039 + ], + "qps_1t": 10941.613329159567, + "qps_1t_min": 10420.118326591039, + "qps_1t_max": 11107.192721715397, + "median_ms": 0.0874000252224505, + "p95_ms": 0.13460503250826153, + "p99_ms": 0.16270201595034456, + "mean_ms": 0.09246471682369399, + "knob": 10, + "batch": { + "recall": 0.47335, + "qps_batch_runs": [ + 69913.93591626002, + 60766.81650976942, + 58869.41289680278 + ], + "qps_batch": 60766.81650976942, + "qps_batch_min": 58869.41289680278, + "qps_batch_max": 69913.93591626002, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.567, + "qps_1t_runs": [ + 6804.033276685068, + 7684.77658632071, + 8556.592660722596 + ], + "qps_1t": 7684.77658632071, + "qps_1t_min": 6804.033276685068, + "qps_1t_max": 8556.592660722596, + "median_ms": 0.12359998072497547, + "p95_ms": 0.1940149639267475, + "p99_ms": 0.24041302560362993, + "mean_ms": 0.13132266694447026, + "knob": 16, + "batch": { + "recall": 0.567, + "qps_batch_runs": [ + 56015.6619157394, + 54146.83539688662, + 40165.48179190745 + ], + "qps_batch": 54146.83539688662, + "qps_batch_min": 40165.48179190745, + "qps_batch_max": 56015.6619157394, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6345, + "qps_1t_runs": [ + 6416.786274578042, + 6102.139437938941, + 6459.837737367857 + ], + "qps_1t": 6416.786274578042, + "qps_1t_min": 6102.139437938941, + "qps_1t_max": 6459.837737367857, + "median_ms": 0.14970003394410014, + "p95_ms": 0.22990500146988782, + "p99_ms": 0.2809019858250395, + "mean_ms": 0.15817361698524715, + "knob": 24, + "batch": { + "recall": 0.6345, + "qps_batch_runs": [ + 41150.56993937414, + 45841.17406224451, + 47479.21001112127 + ], + "qps_batch": 45841.17406224451, + "qps_batch_min": 41150.56993937414, + "qps_batch_max": 47479.21001112127, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6755, + "qps_1t_runs": [ + 5292.066487605985, + 5218.5924821847475, + 5339.289132981858 + ], + "qps_1t": 5292.066487605985, + "qps_1t_min": 5218.5924821847475, + "qps_1t_max": 5339.289132981858, + "median_ms": 0.18039997667074203, + "p95_ms": 0.27090496732853353, + "p99_ms": 0.33311797538772264, + "mean_ms": 0.18929183359917565, + "knob": 32, + "batch": { + "recall": 0.6755, + "qps_batch_runs": [ + 33522.00636620724, + 32467.216217857058, + 28126.94252091266 + ], + "qps_batch": 32467.216217857058, + "qps_batch_min": 28126.94252091266, + "qps_batch_max": 33522.00636620724, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.73065, + "qps_1t_runs": [ + 3636.9660498550597, + 3787.6033175220136, + 3776.656109626694 + ], + "qps_1t": 3776.656109626694, + "qps_1t_min": 3636.9660498550597, + "qps_1t_max": 3787.6033175220136, + "median_ms": 0.24810002651065588, + "p95_ms": 0.3694050159538165, + "p99_ms": 0.4698229691712191, + "mean_ms": 0.2679193835647311, + "knob": 48, + "batch": { + "recall": 0.73065, + "qps_batch_runs": [ + 20223.551134285553, + 22369.988428338755, + 27052.06845977377 + ], + "qps_batch": 22369.988428338755, + "qps_batch_min": 20223.551134285553, + "qps_batch_max": 27052.06845977377, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.76465, + "qps_1t_runs": [ + 2889.8426668563043, + 2822.5178219471654, + 2878.642614136965 + ], + "qps_1t": 2878.642614136965, + "qps_1t_min": 2822.5178219471654, + "qps_1t_max": 2889.8426668563043, + "median_ms": 0.33980002626776695, + "p95_ms": 0.5118049972224981, + "p99_ms": 0.6458090274827557, + "mean_ms": 0.3492397166749773, + "knob": 64, + "batch": { + "recall": 0.76465, + "qps_batch_runs": [ + 17938.764236955303, + 19235.744637154483, + 23485.011288594756 + ], + "qps_batch": 19235.744637154483, + "qps_batch_min": 17938.764236955303, + "qps_batch_max": 23485.011288594756, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8055, + "qps_1t_runs": [ + 1988.0881682927268, + 2059.058323372624, + 2036.819380828179 + ], + "qps_1t": 2036.819380828179, + "qps_1t_min": 1988.0881682927268, + "qps_1t_max": 2059.058323372624, + "median_ms": 0.4835499858018011, + "p95_ms": 0.7098999922163785, + "p99_ms": 0.853150977054612, + "mean_ms": 0.49320541678268154, + "knob": 96, + "batch": { + "recall": 0.8055, + "qps_batch_runs": [ + 14093.369984391786, + 13278.343284263068, + 13595.09326114477 + ], + "qps_batch": 13595.09326114477, + "qps_batch_min": 13278.343284263068, + "qps_batch_max": 14093.369984391786, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.83145, + "qps_1t_runs": [ + 1635.506718806674, + 2462.2912408801676, + 2194.1447283036414 + ], + "qps_1t": 2194.1447283036414, + "qps_1t_min": 1635.506718806674, + "qps_1t_max": 2462.2912408801676, + "median_ms": 0.44319999869912863, + "p95_ms": 0.7982150214957073, + "p99_ms": 1.1405300110345713, + "mean_ms": 0.4911051834157358, + "knob": 128, + "batch": { + "recall": 0.83145, + "qps_batch_runs": [ + 9926.50909024146, + 11891.023523822738, + 11994.492130560624 + ], + "qps_batch": 11891.023523822738, + "qps_batch_min": 9926.50909024146, + "qps_batch_max": 11994.492130560624, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8649, + "qps_1t_runs": [ + 1137.1260912095722, + 1399.6818943829937, + 1461.412392621064 + ], + "qps_1t": 1399.6818943829937, + "qps_1t_min": 1137.1260912095722, + "qps_1t_max": 1461.412392621064, + "median_ms": 0.7115000043995678, + "p95_ms": 1.146019995212555, + "p99_ms": 1.611593045527122, + "mean_ms": 0.7593758502140796, + "knob": 192, + "batch": { + "recall": 0.8649, + "qps_batch_runs": [ + 15057.531058562437, + 16330.276575524489, + 16744.739006367203 + ], + "qps_batch": 16330.276575524489, + "qps_batch_min": 15057.531058562437, + "qps_batch_max": 16744.739006367203, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8839, + "qps_1t_runs": [ + 1018.8648417119123, + 1070.3990194754808, + 1041.0692448105042 + ], + "qps_1t": 1041.0692448105042, + "qps_1t_min": 1018.8648417119123, + "qps_1t_max": 1070.3990194754808, + "median_ms": 0.9078500152099878, + "p95_ms": 1.3639299664646387, + "p99_ms": 2.44202402594965, + "mean_ms": 0.9587554670773292, + "knob": 256, + "batch": { + "recall": 0.8839, + "qps_batch_runs": [ + 12800.688165467176, + 12406.539982445247, + 12443.049715220604 + ], + "qps_batch": 12443.049715220604, + "qps_batch_min": 12406.539982445247, + "qps_batch_max": 12800.688165467176, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.91205, + "qps_1t_runs": [ + 712.4709743073782, + 715.8785440638719, + 726.031266484282 + ], + "qps_1t": 715.8785440638719, + "qps_1t_min": 712.4709743073782, + "qps_1t_max": 726.031266484282, + "median_ms": 1.3280999846756458, + "p95_ms": 1.9126050086924806, + "p99_ms": 3.533559009083561, + "mean_ms": 1.3926007001136895, + "knob": 400, + "batch": { + "recall": 0.91205, + "qps_batch_runs": [ + 9105.859257960406, + 9088.137210748393, + 8353.887377893667 + ], + "qps_batch": 9088.137210748393, + "qps_batch_min": 8353.887377893667, + "qps_batch_max": 9105.859257960406, + "threads": 32 + } + }, + { + "nq": 1353, + "recall_queries": 1353, + "recall": 0.9393939393939394, + "qps_1t_runs": [ + 436.7728524609268, + 451.0766077970741, + 403.8289429781995 + ], + "qps_1t": 436.7728524609268, + "qps_1t_min": 403.8289429781995, + "qps_1t_max": 451.0766077970741, + "median_ms": 2.2125000250525773, + "p95_ms": 3.6095300049055368, + "p99_ms": 5.660058009671049, + "mean_ms": 2.3275779748469017, + "knob": 640, + "batch": { + "recall": 0.93665, + "qps_batch_runs": [ + 2154.7479655729303, + 2503.090377849747, + 2683.6671613000744 + ], + "qps_batch": 2503.090377849747, + "qps_batch_min": 2154.7479655729303, + "qps_batch_max": 2683.6671613000744, + "threads": 32 + } + } + ], + "knob_name": "expansion_search", + "mem_after_sweep": { + "rss_mb": 668.90625, + "peak_mb": 674.57421875 + }, + "ref_knob": 640, + "ref_recall_pre": 0.9393939393939394, + "mutation": { + "ref_knob": 640, + "recall_before": 0.9393939393939394, + "insert_10k": { + "seconds": 1.367447299999185, + "per_s": 7312.89608016774, + "batch": 1000 + }, + "update_10k": { + "seconds": 1.4873334000003524, + "per_s": 6723.442101143987, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.004167300008703023, + "per_s": 2399635.250429755, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 682.38671875, + "peak_mb": 716.30859375 + }, + "first_query_after_ms": 25.090099952649325, + "after": { + "nq": 1240, + "recall_queries": 1240, + "recall": 0.9389516129032258, + "qps_1t_runs": [ + 453.52336445809095, + 470.42636226205815, + 459.54450033204455 + ], + "qps_1t": 459.54450033204455, + "qps_1t_min": 453.52336445809095, + "qps_1t_max": 470.42636226205815, + "median_ms": 2.134649985237047, + "p95_ms": 2.9372099554166193, + "p99_ms": 4.37878997530788, + "mean_ms": 2.1689190320865883 + }, + "recall_after": 0.9389516129032258 + }, + "saveload": { + "save_s": 0.5854958000127226, + "disk_mb": 528.3713569641113, + "load_s": 0.4993411999894306, + "first_query_after_load_ms": 3.3715000026859343, + "time_to_ready_s": 0.5027126999921165 + }, + "mem_end": { + "rss_mb": 669.0234375, + "peak_mb": 716.30859375 + }, + "status": "ok", + "finished": "2026-11-16 10:24:58", + "load_before_pct": 9.062499999999982, + "load_contaminated": false, + "wall_s": 165.6321086883545, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/usearch__glove__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/usearch__glove__100000__core+mutation+saveload.json new file mode 100644 index 0000000..805bf30 --- /dev/null +++ b/benchmarks/competitive/results/raw/usearch__glove__100000__core+mutation+saveload.json @@ -0,0 +1,468 @@ +{ + "system": "usearch", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 08:20:20", + "mem_baseline": { + "rss_mb": 35.14453125, + "peak_mb": 35.14453125 + }, + "version": "usearch 2.26.6", + "build": { + "ingest_s": 1.7242632998968475, + "index_s": 2.800021320581436e-06, + "total_s": 1.724266099918168, + "first_query_ms": 2.073699957691133, + "ready_s": 1.7263397998758592, + "total_s_runs": [ + 1.724266099918168, + 1.6228081999579445, + 1.7237644000560977 + ] + }, + "mem_after_build": { + "rss_mb": 110.3203125, + "peak_mb": 119.50390625 + }, + "mem_after_first_query": { + "rss_mb": 110.55859375, + "peak_mb": 119.50390625 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.53765, + "qps_1t_runs": [ + 26481.40271492198, + 27475.05602791111, + 27781.095971433166 + ], + "qps_1t": 27475.05602791111, + "qps_1t_min": 26481.40271492198, + "qps_1t_max": 27781.095971433166, + "median_ms": 0.03539997851476073, + "p95_ms": 0.05009997694287449, + "p99_ms": 0.05940098664723339, + "mean_ms": 0.03671823343029246, + "knob": 10, + "batch": { + "recall": 0.53765, + "qps_batch_runs": [ + 293590.91006227705, + 287220.1400897643, + 349919.51960857597 + ], + "qps_batch": 293590.91006227705, + "qps_batch_min": 287220.1400897643, + "qps_batch_max": 349919.51960857597, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.62375, + "qps_1t_runs": [ + 21251.004326648366, + 21039.23276226155, + 19802.03909154337 + ], + "qps_1t": 21039.23276226155, + "qps_1t_min": 19802.03909154337, + "qps_1t_max": 21251.004326648366, + "median_ms": 0.045900000259280205, + "p95_ms": 0.06699998630210757, + "p99_ms": 0.09479996573645622, + "mean_ms": 0.04836223322005632, + "knob": 16, + "batch": { + "recall": 0.62375, + "qps_batch_runs": [ + 264872.5947182476, + 282601.62875678693, + 282829.4254517652 + ], + "qps_batch": 282601.62875678693, + "qps_batch_min": 264872.5947182476, + "qps_batch_max": 282829.4254517652, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.693, + "qps_1t_runs": [ + 15664.736939501921, + 15794.03309058259, + 16626.55510416158 + ], + "qps_1t": 15794.03309058259, + "qps_1t_min": 15664.736939501921, + "qps_1t_max": 16626.55510416158, + "median_ms": 0.05989999044686556, + "p95_ms": 0.08620001608505845, + "p99_ms": 0.1082030090037734, + "mean_ms": 0.06243248356622644, + "knob": 24, + "batch": { + "recall": 0.693, + "qps_batch_runs": [ + 227663.37812938154, + 178469.44605269586, + 250884.36769002103 + ], + "qps_batch": 227663.37812938154, + "qps_batch_min": 178469.44605269586, + "qps_batch_max": 250884.36769002103, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.73455, + "qps_1t_runs": [ + 13645.124448001072, + 12832.67092027009, + 11837.709675190305 + ], + "qps_1t": 12832.67092027009, + "qps_1t_min": 11837.709675190305, + "qps_1t_max": 13645.124448001072, + "median_ms": 0.07349997758865356, + "p95_ms": 0.10190501634497197, + "p99_ms": 0.13021300139371333, + "mean_ms": 0.07856271683704108, + "knob": 32, + "batch": { + "recall": 0.73455, + "qps_batch_runs": [ + 146126.19430047876, + 193687.7175194879, + 190170.1063411548 + ], + "qps_batch": 190170.1063411548, + "qps_batch_min": 146126.19430047876, + "qps_batch_max": 193687.7175194879, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.78995, + "qps_1t_runs": [ + 9549.251407179689, + 9153.506599548024, + 10111.095688205 + ], + "qps_1t": 9549.251407179689, + "qps_1t_min": 9153.506599548024, + "qps_1t_max": 10111.095688205, + "median_ms": 0.10119998478330672, + "p95_ms": 0.14110997144598517, + "p99_ms": 0.17130702384747581, + "mean_ms": 0.10428975008350487, + "knob": 48, + "batch": { + "recall": 0.78995, + "qps_batch_runs": [ + 119156.84586972021, + 121533.26332737689, + 150440.4142423235 + ], + "qps_batch": 121533.26332737689, + "qps_batch_min": 119156.84586972021, + "qps_batch_max": 150440.4142423235, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8295, + "qps_1t_runs": [ + 7919.067096959967, + 7238.341203368562, + 7210.338193967392 + ], + "qps_1t": 7238.341203368562, + "qps_1t_min": 7210.338193967392, + "qps_1t_max": 7919.067096959967, + "median_ms": 0.1297999988310039, + "p95_ms": 0.1813049864722416, + "p99_ms": 0.22730398457497367, + "mean_ms": 0.13437348346148306, + "knob": 64, + "batch": { + "recall": 0.8295, + "qps_batch_runs": [ + 80585.04732029412, + 108435.76001904267, + 126035.85726901238 + ], + "qps_batch": 108435.76001904267, + "qps_batch_min": 80585.04732029412, + "qps_batch_max": 126035.85726901238, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.86895, + "qps_1t_runs": [ + 5185.8330213365425, + 5093.542919099803, + 5408.940193440393 + ], + "qps_1t": 5185.8330213365425, + "qps_1t_min": 5093.542919099803, + "qps_1t_max": 5408.940193440393, + "median_ms": 0.1812000118661672, + "p95_ms": 0.2411050314549357, + "p99_ms": 0.32941500539891455, + "mean_ms": 0.1913463837117888, + "knob": 96, + "batch": { + "recall": 0.86895, + "qps_batch_runs": [ + 59542.06199390638, + 86920.61974165458, + 80682.57463165042 + ], + "qps_batch": 80682.57463165042, + "qps_batch_min": 59542.06199390638, + "qps_batch_max": 86920.61974165458, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.895, + "qps_1t_runs": [ + 4152.0893783467445, + 3966.425789650838, + 3947.660343681471 + ], + "qps_1t": 3966.425789650838, + "qps_1t_min": 3947.660343681471, + "qps_1t_max": 4152.0893783467445, + "median_ms": 0.23969996254891157, + "p95_ms": 0.31560501374769956, + "p99_ms": 0.44911798555403987, + "mean_ms": 0.24875778320711106, + "knob": 128, + "batch": { + "recall": 0.895, + "qps_batch_runs": [ + 60906.90384019251, + 67475.47260757248, + 48475.79965746959 + ], + "qps_batch": 60906.90384019251, + "qps_batch_min": 48475.79965746959, + "qps_batch_max": 67475.47260757248, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9264, + "qps_1t_runs": [ + 2900.5407477842627, + 2795.5239185552487, + 2848.318359748414 + ], + "qps_1t": 2848.318359748414, + "qps_1t_min": 2795.5239185552487, + "qps_1t_max": 2900.5407477842627, + "median_ms": 0.3449000068940222, + "p95_ms": 0.44660502462647855, + "p99_ms": 0.5491049902047963, + "mean_ms": 0.3511874500254635, + "knob": 192, + "batch": { + "recall": 0.9264, + "qps_batch_runs": [ + 38682.54974045804, + 48220.54148622443, + 51362.12352558602 + ], + "qps_batch": 48220.54148622443, + "qps_batch_min": 38682.54974045804, + "qps_batch_max": 51362.12352558602, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9456, + "qps_1t_runs": [ + 2134.5277566378336, + 2189.6346202453196, + 1925.888534938385 + ], + "qps_1t": 2134.5277566378336, + "qps_1t_min": 1925.888534938385, + "qps_1t_max": 2189.6346202453196, + "median_ms": 0.47119997907429934, + "p95_ms": 0.6373050186084583, + "p99_ms": 0.7360079581849279, + "mean_ms": 0.4814752501163942, + "knob": 256, + "batch": { + "recall": 0.9456, + "qps_batch_runs": [ + 27037.25734654794, + 29189.963904610235, + 33315.17655442926 + ], + "qps_batch": 29189.963904610235, + "qps_batch_min": 27037.25734654794, + "qps_batch_max": 33315.17655442926, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.96665, + "qps_1t_runs": [ + 1375.5997352042907, + 1372.137190512153, + 1374.707445329708 + ], + "qps_1t": 1374.707445329708, + "qps_1t_min": 1372.137190512153, + "qps_1t_max": 1375.5997352042907, + "median_ms": 0.7197500090114772, + "p95_ms": 0.9486000053584576, + "p99_ms": 1.1774120014160874, + "mean_ms": 0.7277244166374052, + "knob": 400, + "batch": { + "recall": 0.96665, + "qps_batch_runs": [ + 20147.56072548454, + 23417.44880109996, + 25174.902625504725 + ], + "qps_batch": 23417.44880109996, + "qps_batch_min": 20147.56072548454, + "qps_batch_max": 25174.902625504725, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.98195, + "qps_1t_runs": [ + 852.5408659493986, + 895.8063451111749, + 875.7549389770719 + ], + "qps_1t": 875.7549389770719, + "qps_1t_min": 852.5408659493986, + "qps_1t_max": 895.8063451111749, + "median_ms": 1.1377999908290803, + "p95_ms": 1.473804988199845, + "p99_ms": 1.8019379832549052, + "mean_ms": 1.1437163167671922, + "knob": 640, + "batch": { + "recall": 0.98195, + "qps_batch_runs": [ + 15709.401209119207, + 15940.255918684243, + 15402.565295445569 + ], + "qps_batch": 15709.401209119207, + "qps_batch_min": 15402.565295445569, + "qps_batch_max": 15940.255918684243, + "threads": 32 + } + } + ], + "knob_name": "expansion_search", + "mem_after_sweep": { + "rss_mb": 114.296875, + "peak_mb": 119.50390625 + }, + "ref_knob": 400, + "ref_recall_pre": 0.96665, + "mutation": { + "ref_knob": 400, + "recall_before": 0.96665, + "insert_10k": { + "seconds": 0.26082840003073215, + "per_s": 38339.383283498835, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.2808492000331171, + "per_s": 35606.296898195986, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.0026926000136882067, + "per_s": 3713882.473877891, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 130.9609375, + "peak_mb": 191.05078125 + }, + "first_query_after_ms": 3.6702000070363283, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9636, + "qps_1t_runs": [ + 1237.7031883711013, + 1299.8645938257835, + 1271.7753373789055 + ], + "qps_1t": 1271.7753373789055, + "qps_1t_min": 1237.7031883711013, + "qps_1t_max": 1299.8645938257835, + "median_ms": 0.7844500069040805, + "p95_ms": 1.0358100262237713, + "p99_ms": 1.2599719897843913, + "mean_ms": 0.7878538161652008 + }, + "recall_after": 0.9636 + }, + "saveload": { + "save_s": 0.09747639996930957, + "disk_mb": 57.543521881103516, + "load_s": 0.051671900029759854, + "first_query_after_load_ms": 1.5568999806419015, + "time_to_ready_s": 0.053228800010401756 + }, + "mem_end": { + "rss_mb": 115.38671875, + "peak_mb": 191.05078125 + }, + "status": "ok", + "finished": "2026-11-16 08:20:54", + "load_before_pct": 2.856249999999997, + "load_contaminated": false, + "wall_s": 34.40745282173157, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/usearch__glove__100000__filter.json b/benchmarks/competitive/results/raw/usearch__glove__100000__filter.json new file mode 100644 index 0000000..90736ec --- /dev/null +++ b/benchmarks/competitive/results/raw/usearch__glove__100000__filter.json @@ -0,0 +1,38 @@ +{ + "system": "usearch", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 08:49:49", + "filter": { + "build_with_metadata_s": 1.5688168000197038, + "mem": { + "rss_mb": 108.9609375, + "peak_mb": 117.99609375 + }, + "subsets": { + "f10": { + "unsupported": true + }, + "f01": { + "unsupported": true + } + } + }, + "mem_end": { + "rss_mb": 42.11328125, + "peak_mb": 117.99609375 + }, + "status": "ok", + "finished": "2026-11-16 08:49:50", + "load_before_pct": 3.081249999999999, + "load_contaminated": false, + "wall_s": 1.7668533325195312, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/usearch__sift__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/usearch__sift__100000__core+mutation+saveload.json new file mode 100644 index 0000000..2b1066b --- /dev/null +++ b/benchmarks/competitive/results/raw/usearch__sift__100000__core+mutation+saveload.json @@ -0,0 +1,468 @@ +{ + "system": "usearch", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 11:09:36", + "mem_baseline": { + "rss_mb": 35.41015625, + "peak_mb": 35.41015625 + }, + "version": "usearch 2.26.6", + "build": { + "ingest_s": 1.5662795999087393, + "index_s": 3.8000289350748062e-06, + "total_s": 1.5662833999376744, + "first_query_ms": 1.3413000269792974, + "ready_s": 1.5676246999646537, + "total_s_runs": [ + 1.5662833999376744, + 1.5631513000116684, + 1.3388760999660008 + ] + }, + "mem_after_build": { + "rss_mb": 120.14453125, + "peak_mb": 130.94921875 + }, + "mem_after_first_query": { + "rss_mb": 120.25390625, + "peak_mb": 130.94921875 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7861, + "qps_1t_runs": [ + 27069.130961110088, + 27812.311509041658, + 26424.23333457998 + ], + "qps_1t": 27069.130961110088, + "qps_1t_min": 26424.23333457998, + "qps_1t_max": 27812.311509041658, + "median_ms": 0.03459997242316604, + "p95_ms": 0.0515049876412377, + "p99_ms": 0.07180096523370597, + "mean_ms": 0.036913932980193444, + "knob": 10, + "batch": { + "recall": 0.7861, + "qps_batch_runs": [ + 357110.9698777513, + 392803.83348365396, + 360457.78201546194 + ], + "qps_batch": 360457.78201546194, + "qps_batch_min": 357110.9698777513, + "qps_batch_max": 392803.83348365396, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.87025, + "qps_1t_runs": [ + 21361.975675150676, + 19975.210776979882, + 19615.45851640123 + ], + "qps_1t": 19975.210776979882, + "qps_1t_min": 19615.45851640123, + "qps_1t_max": 21361.975675150676, + "median_ms": 0.044899992644786835, + "p95_ms": 0.06560003384947777, + "p99_ms": 0.0894059822894634, + "mean_ms": 0.04928479986847378, + "knob": 16, + "batch": { + "recall": 0.87025, + "qps_batch_runs": [ + 235977.06348661665, + 117907.84324861944, + 273119.57188006246 + ], + "qps_batch": 235977.06348661665, + "qps_batch_min": 117907.84324861944, + "qps_batch_max": 273119.57188006246, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9224, + "qps_1t_runs": [ + 16878.91906143038, + 16246.372866444264, + 15886.027276176243 + ], + "qps_1t": 16246.372866444264, + "qps_1t_min": 15886.027276176243, + "qps_1t_max": 16878.91906143038, + "median_ms": 0.057800032664090395, + "p95_ms": 0.0799999717855826, + "p99_ms": 0.1031009707367048, + "mean_ms": 0.06124870019266382, + "knob": 24, + "batch": { + "recall": 0.9224, + "qps_batch_runs": [ + 160194.79671496156, + 126416.65684440518, + 240589.92740892884 + ], + "qps_batch": 160194.79671496156, + "qps_batch_min": 126416.65684440518, + "qps_batch_max": 240589.92740892884, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9517, + "qps_1t_runs": [ + 13737.197886268134, + 12861.678421829287, + 13047.565576531997 + ], + "qps_1t": 13047.565576531997, + "qps_1t_min": 12861.678421829287, + "qps_1t_max": 13737.197886268134, + "median_ms": 0.07189996540546417, + "p95_ms": 0.10080001084133983, + "p99_ms": 0.1329100219300019, + "mean_ms": 0.07572934981241512, + "knob": 32, + "batch": { + "recall": 0.9517, + "qps_batch_runs": [ + 110714.38452159616, + 103358.10472761495, + 74285.37453718168 + ], + "qps_batch": 103358.10472761495, + "qps_batch_min": 74285.37453718168, + "qps_batch_max": 110714.38452159616, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9768, + "qps_1t_runs": [ + 8228.539437715186, + 9040.23857087095, + 9296.406150467108 + ], + "qps_1t": 9040.23857087095, + "qps_1t_min": 8228.539437715186, + "qps_1t_max": 9296.406150467108, + "median_ms": 0.10459998156875372, + "p95_ms": 0.16680498956702652, + "p99_ms": 0.2566010324517265, + "mean_ms": 0.11323775000831422, + "knob": 48, + "batch": { + "recall": 0.9768, + "qps_batch_runs": [ + 106356.38964060324, + 142566.9172567634, + 133913.62584870064 + ], + "qps_batch": 133913.62584870064, + "qps_batch_min": 106356.38964060324, + "qps_batch_max": 142566.9172567634, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9868, + "qps_1t_runs": [ + 6439.467281229915, + 7549.511578564022, + 6417.66344014736 + ], + "qps_1t": 6439.467281229915, + "qps_1t_min": 6417.66344014736, + "qps_1t_max": 7549.511578564022, + "median_ms": 0.13369997031986713, + "p95_ms": 0.22220500395633283, + "p99_ms": 0.31881104048807196, + "mean_ms": 0.14785706653492525, + "knob": 64, + "batch": { + "recall": 0.9868, + "qps_batch_runs": [ + 90733.8098028361, + 91077.58428874474, + 101010.10096353963 + ], + "qps_batch": 91077.58428874474, + "qps_batch_min": 90733.8098028361, + "qps_batch_max": 101010.10096353963, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9939, + "qps_1t_runs": [ + 5047.802693810271, + 5249.557399203695, + 5360.190025936672 + ], + "qps_1t": 5249.557399203695, + "qps_1t_min": 5047.802693810271, + "qps_1t_max": 5360.190025936672, + "median_ms": 0.18119998276233673, + "p95_ms": 0.2735149872023613, + "p99_ms": 0.3934120270423594, + "mean_ms": 0.19171959998008484, + "knob": 96, + "batch": { + "recall": 0.9939, + "qps_batch_runs": [ + 62887.93994649447, + 85289.66511592029, + 77948.09445414573 + ], + "qps_batch": 77948.09445414573, + "qps_batch_min": 62887.93994649447, + "qps_batch_max": 85289.66511592029, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99575, + "qps_1t_runs": [ + 4190.171792421206, + 4301.8681119205285, + 4062.1913385550233 + ], + "qps_1t": 4190.171792421206, + "qps_1t_min": 4062.1913385550233, + "qps_1t_max": 4301.8681119205285, + "median_ms": 0.22819999139755964, + "p95_ms": 0.33569999504834414, + "p99_ms": 0.6329029879998417, + "mean_ms": 0.2390944664075505, + "knob": 128, + "batch": { + "recall": 0.99575, + "qps_batch_runs": [ + 32942.19961171052, + 16449.830485719525, + 32827.13885251534 + ], + "qps_batch": 32827.13885251534, + "qps_batch_min": 16449.830485719525, + "qps_batch_max": 32942.19961171052, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99695, + "qps_1t_runs": [ + 2674.003067122305, + 2769.1319326123185, + 3016.1545197719274 + ], + "qps_1t": 2769.1319326123185, + "qps_1t_min": 2674.003067122305, + "qps_1t_max": 3016.1545197719274, + "median_ms": 0.3363499999977648, + "p95_ms": 0.5188199924305082, + "p99_ms": 0.8846420416375631, + "mean_ms": 0.35554771654036205, + "knob": 192, + "batch": { + "recall": 0.99695, + "qps_batch_runs": [ + 41251.91304261989, + 53736.57250767222, + 47475.37808868633 + ], + "qps_batch": 47475.37808868633, + "qps_batch_min": 41251.91304261989, + "qps_batch_max": 53736.57250767222, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9975, + "qps_1t_runs": [ + 2330.3325577074784, + 2436.9066586890453, + 1993.29932409281 + ], + "qps_1t": 2330.3325577074784, + "qps_1t_min": 1993.29932409281, + "qps_1t_max": 2436.9066586890453, + "median_ms": 0.4217000096105039, + "p95_ms": 0.6297750194789844, + "p99_ms": 0.9572960122022799, + "mean_ms": 0.4470534671466642, + "knob": 256, + "batch": { + "recall": 0.9975, + "qps_batch_runs": [ + 35042.9626709708, + 38197.6806189497, + 37724.31821204922 + ], + "qps_batch": 37724.31821204922, + "qps_batch_min": 35042.9626709708, + "qps_batch_max": 38197.6806189497, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9976, + "qps_1t_runs": [ + 1656.308939527239, + 1591.9763119441568, + 1525.4890147096621 + ], + "qps_1t": 1591.9763119441568, + "qps_1t_min": 1525.4890147096621, + "qps_1t_max": 1656.308939527239, + "median_ms": 0.6066000205464661, + "p95_ms": 0.8747950225370008, + "p99_ms": 1.322138996911236, + "mean_ms": 0.6291432165598962, + "knob": 400, + "batch": { + "recall": 0.9976, + "qps_batch_runs": [ + 25792.2741858166, + 23588.187984603996, + 27727.32944475074 + ], + "qps_batch": 25792.2741858166, + "qps_batch_min": 23588.187984603996, + "qps_batch_max": 27727.32944475074, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99765, + "qps_1t_runs": [ + 1108.3640302381805, + 1063.633015502327, + 1029.843209069429 + ], + "qps_1t": 1063.633015502327, + "qps_1t_min": 1029.843209069429, + "qps_1t_max": 1108.3640302381805, + "median_ms": 0.9094499982893467, + "p95_ms": 1.2913049955386668, + "p99_ms": 1.822889968752863, + "mean_ms": 0.9378087165241595, + "knob": 640, + "batch": { + "recall": 0.99765, + "qps_batch_runs": [ + 13320.456894933395, + 13864.885311145908, + 16800.087358381366 + ], + "qps_batch": 13864.885311145908, + "qps_batch_min": 13320.456894933395, + "qps_batch_max": 16800.087358381366, + "threads": 32 + } + } + ], + "knob_name": "expansion_search", + "mem_after_sweep": { + "rss_mb": 123.3046875, + "peak_mb": 130.94921875 + }, + "ref_knob": 32, + "ref_recall_pre": 0.9517, + "mutation": { + "ref_knob": 32, + "recall_before": 0.9517, + "insert_10k": { + "seconds": 0.2155102000106126, + "per_s": 46401.51602804674, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.25849350000498816, + "per_s": 38685.69228938843, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.002912099997047335, + "per_s": 3433948.013508911, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 141.046875, + "peak_mb": 210.8359375 + }, + "first_query_after_ms": 2.8415999840945005, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.945, + "qps_1t_runs": [ + 12260.807078531554, + 10577.39346901822, + 10986.155814392707 + ], + "qps_1t": 10986.155814392707, + "qps_1t_min": 10577.39346901822, + "qps_1t_max": 12260.807078531554, + "median_ms": 0.08269998943433166, + "p95_ms": 0.1221999991685152, + "p99_ms": 0.18990499083884071, + "mean_ms": 0.08904186639119871 + }, + "recall_after": 0.945 + }, + "saveload": { + "save_s": 0.07970520004164428, + "disk_mb": 69.29278945922852, + "load_s": 0.06134930002735928, + "first_query_after_load_ms": 0.20030001178383827, + "time_to_ready_s": 0.061549600039143115 + }, + "mem_end": { + "rss_mb": 128.8125, + "peak_mb": 210.8359375 + }, + "status": "ok", + "finished": "2026-11-16 11:10:03", + "load_before_pct": 12.12187500000001, + "load_contaminated": false, + "wall_s": 27.736467123031616, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/usearch__text384__1000000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/usearch__text384__1000000__core+mutation+saveload.json new file mode 100644 index 0000000..8194acf --- /dev/null +++ b/benchmarks/competitive/results/raw/usearch__text384__1000000__core+mutation+saveload.json @@ -0,0 +1,463 @@ +{ + "system": "usearch", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 09:27:43", + "mem_baseline": { + "rss_mb": 37.3671875, + "peak_mb": 37.3671875 + }, + "version": "usearch 2.26.6", + "build": { + "ingest_s": 173.75032389973057, + "index_s": 3.100023604929447e-06, + "total_s": 173.75032699975418, + "first_query_ms": 5.395699990913272, + "ready_s": 173.7557226997451 + }, + "mem_after_build": { + "rss_mb": 1751.875, + "peak_mb": 1782.6640625 + }, + "mem_after_first_query": { + "rss_mb": 1751.9375, + "peak_mb": 1782.6640625 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.5583, + "qps_1t_runs": [ + 5335.188558767594, + 6033.5301370247835, + 5894.473018927222 + ], + "qps_1t": 5894.473018927222, + "qps_1t_min": 5335.188558767594, + "qps_1t_max": 6033.5301370247835, + "median_ms": 0.16639998648315668, + "p95_ms": 0.2515100495656953, + "p99_ms": 0.30662000644952103, + "mean_ms": 0.1742752332938835, + "knob": 10, + "batch": { + "recall": 0.5583, + "qps_batch_runs": [ + 47578.37943490698, + 47684.89814743232, + 47737.482062584415 + ], + "qps_batch": 47684.89814743232, + "qps_batch_min": 47578.37943490698, + "qps_batch_max": 47737.482062584415, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.64885, + "qps_1t_runs": [ + 4309.308358560452, + 4676.670220743963, + 4212.3921812005165 + ], + "qps_1t": 4309.308358560452, + "qps_1t_min": 4212.3921812005165, + "qps_1t_max": 4676.670220743963, + "median_ms": 0.21650001872330904, + "p95_ms": 0.3289099986432121, + "p99_ms": 0.3963029908481986, + "mean_ms": 0.22775931681583947, + "knob": 16, + "batch": { + "recall": 0.64885, + "qps_batch_runs": [ + 33250.37366577372, + 34043.48032530006, + 32971.960645154984 + ], + "qps_batch": 33250.37366577372, + "qps_batch_min": 32971.960645154984, + "qps_batch_max": 34043.48032530006, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7299, + "qps_1t_runs": [ + 3584.134954498596, + 3614.3058536737617, + 3019.6938427585874 + ], + "qps_1t": 3584.134954498596, + "qps_1t_min": 3019.6938427585874, + "qps_1t_max": 3614.3058536737617, + "median_ms": 0.2711999695748091, + "p95_ms": 0.4070000140927732, + "p99_ms": 0.5640459869755442, + "mean_ms": 0.2956150170066394, + "knob": 24, + "batch": { + "recall": 0.7299, + "qps_batch_runs": [ + 27184.661342805575, + 27095.387959379364, + 27939.516552147372 + ], + "qps_batch": 27184.661342805575, + "qps_batch_min": 27095.387959379364, + "qps_batch_max": 27939.516552147372, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7786, + "qps_1t_runs": [ + 2935.3084432497985, + 2957.1052653075467, + 2832.7524548543925 + ], + "qps_1t": 2935.3084432497985, + "qps_1t_min": 2832.7524548543925, + "qps_1t_max": 2957.1052653075467, + "median_ms": 0.3276999923400581, + "p95_ms": 0.4734049958642572, + "p99_ms": 0.5659370019566276, + "mean_ms": 0.34395393333397806, + "knob": 32, + "batch": { + "recall": 0.7786, + "qps_batch_runs": [ + 22419.865799794956, + 21227.366317750708, + 22714.882789895833 + ], + "qps_batch": 22419.865799794956, + "qps_batch_min": 21227.366317750708, + "qps_batch_max": 22714.882789895833, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.83595, + "qps_1t_runs": [ + 2154.607525434639, + 2082.1634167633856, + 2038.7987484718174 + ], + "qps_1t": 2082.1634167633856, + "qps_1t_min": 2038.7987484718174, + "qps_1t_max": 2154.607525434639, + "median_ms": 0.45035002403892577, + "p95_ms": 0.6310049764579162, + "p99_ms": 0.8108210033969958, + "mean_ms": 0.4782920833968092, + "knob": 48, + "batch": { + "recall": 0.83595, + "qps_batch_runs": [ + 15680.531129883453, + 15943.737740466944, + 15618.167053521367 + ], + "qps_batch": 15680.531129883453, + "qps_batch_min": 15618.167053521367, + "qps_batch_max": 15943.737740466944, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8685, + "qps_1t_runs": [ + 1655.0363123926268, + 1637.7821947676293, + 1650.9278216153132 + ], + "qps_1t": 1650.9278216153132, + "qps_1t_min": 1637.7821947676293, + "qps_1t_max": 1655.0363123926268, + "median_ms": 0.5744999798480421, + "p95_ms": 0.7804250053595753, + "p99_ms": 1.1520349816419213, + "mean_ms": 0.6068393832538277, + "knob": 64, + "batch": { + "recall": 0.8685, + "qps_batch_runs": [ + 13217.634967754377, + 13011.650629423228, + 13608.052699849144 + ], + "qps_batch": 13217.634967754377, + "qps_batch_min": 13011.650629423228, + "qps_batch_max": 13608.052699849144, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.90895, + "qps_1t_runs": [ + 1140.7990103440372, + 1228.939585185393, + 1183.1633253678037 + ], + "qps_1t": 1183.1633253678037, + "qps_1t_min": 1140.7990103440372, + "qps_1t_max": 1228.939585185393, + "median_ms": 0.8043000125326216, + "p95_ms": 1.0800250165630132, + "p99_ms": 1.640918980701827, + "mean_ms": 0.8451600336411502, + "knob": 96, + "batch": { + "recall": 0.90895, + "qps_batch_runs": [ + 9784.099174065563, + 9540.546367565983, + 9227.25435599558 + ], + "qps_batch": 9540.546367565983, + "qps_batch_min": 9227.25435599558, + "qps_batch_max": 9784.099174065563, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.93075, + "qps_1t_runs": [ + 920.4036317979316, + 926.2288373562499, + 907.8523096341713 + ], + "qps_1t": 920.4036317979316, + "qps_1t_min": 907.8523096341713, + "qps_1t_max": 926.2288373562499, + "median_ms": 1.0709500056691468, + "p95_ms": 1.3511150405975059, + "p99_ms": 1.702712042606436, + "mean_ms": 1.089209133303181, + "knob": 128, + "batch": { + "recall": 0.93075, + "qps_batch_runs": [ + 7677.392879812757, + 7010.113139607297, + 7203.585944043157 + ], + "qps_batch": 7203.585944043157, + "qps_batch_min": 7010.113139607297, + "qps_batch_max": 7677.392879812757, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.95695, + "qps_1t_runs": [ + 641.868525528768, + 626.6844889436964, + 669.4238224782664 + ], + "qps_1t": 641.868525528768, + "qps_1t_min": 626.6844889436964, + "qps_1t_max": 669.4238224782664, + "median_ms": 1.5242999943438917, + "p95_ms": 1.9273800076916816, + "p99_ms": 2.4350569728994755, + "mean_ms": 1.5491575665267494, + "knob": 192, + "batch": { + "recall": 0.95695, + "qps_batch_runs": [ + 5397.919803609916, + 5506.814132166697, + 5107.8222979705815 + ], + "qps_batch": 5397.919803609916, + "qps_batch_min": 5107.8222979705815, + "qps_batch_max": 5506.814132166697, + "threads": 32 + } + }, + { + "nq": 1639, + "recall_queries": 1639, + "recall": 0.9672971323978036, + "qps_1t_runs": [ + 521.8354546542845, + 540.8268862508924, + 510.652637746035 + ], + "qps_1t": 521.8354546542845, + "qps_1t_min": 510.652637746035, + "qps_1t_max": 540.8268862508924, + "median_ms": 1.901899988297373, + "p95_ms": 2.3345600231550634, + "p99_ms": 2.785200020298362, + "mean_ms": 1.9078705718044555, + "knob": 256, + "batch": { + "recall": 0.96785, + "qps_batch_runs": [ + 4125.263630185285, + 3963.3375423748707, + 4146.053009424596 + ], + "qps_batch": 4125.263630185285, + "qps_batch_min": 3963.3375423748707, + "qps_batch_max": 4146.053009424596, + "threads": 32 + } + }, + { + "nq": 999, + "recall_queries": 1000, + "recall": 0.9822000000000001, + "qps_1t_runs": [ + 334.5629441679244, + 332.55070197854235, + 338.70559736581873 + ], + "qps_1t": 334.5629441679244, + "qps_1t_min": 332.55070197854235, + "qps_1t_max": 338.70559736581873, + "median_ms": 2.9717000434175134, + "p95_ms": 3.692380001302808, + "p99_ms": 4.919920025859027, + "mean_ms": 2.9828169835524116, + "knob": 400, + "batch": { + "recall": 0.9813, + "qps_batch_runs": [ + 2615.9428373802743, + 2836.315388149292, + 2549.444565422957 + ], + "qps_batch": 2615.9428373802743, + "qps_batch_min": 2549.444565422957, + "qps_batch_max": 2836.315388149292, + "threads": 32 + } + }, + { + "nq": 634, + "recall_queries": 1000, + "recall": 0.99, + "qps_1t_runs": [ + 213.65551236378653, + 208.08420252920422, + 219.82433400068635 + ], + "qps_1t": 213.65551236378653, + "qps_1t_min": 208.08420252920422, + "qps_1t_max": 219.82433400068635, + "median_ms": 4.6139500045683235, + "p95_ms": 5.832479998935014, + "p99_ms": 11.148029983160088, + "mean_ms": 4.678421766794904, + "knob": 640, + "batch": { + "recall": 0.9898, + "qps_batch_runs": [ + 1850.4987602462766, + 1807.3735599749343, + 1810.2565151027852 + ], + "qps_batch": 1810.2565151027852, + "qps_batch_min": 1807.3735599749343, + "qps_batch_max": 1850.4987602462766, + "threads": 32 + } + } + ], + "knob_name": "expansion_search", + "mem_after_sweep": { + "rss_mb": 1754.8046875, + "peak_mb": 1782.6640625 + }, + "ref_knob": 192, + "ref_recall_pre": 0.95695, + "mutation": { + "ref_knob": 192, + "recall_before": 0.95695, + "insert_10k": { + "seconds": 2.2522861000034027, + "per_s": 4439.933274900064, + "batch": 1000 + }, + "update_10k": { + "seconds": 2.3669153000228107, + "per_s": 4224.908259245114, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.00460629997542128, + "per_s": 2170939.8114232514, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 1800.89453125, + "peak_mb": 1834.61328125 + }, + "first_query_after_ms": 23.924000037368387, + "after": { + "nq": 1961, + "recall_queries": 1961, + "recall": 0.955583885772565, + "qps_1t_runs": [ + 623.3032590172232, + 624.2202221970825, + 641.6424581996398 + ], + "qps_1t": 624.2202221970825, + "qps_1t_min": 623.3032590172232, + "qps_1t_max": 641.6424581996398, + "median_ms": 1.5801999834366143, + "p95_ms": 1.9869599957019086, + "p99_ms": 2.5361139862798163, + "mean_ms": 1.588284854240337 + }, + "recall_after": 0.955583885772565 + }, + "saveload": { + "save_s": 1.5433163999696262, + "disk_mb": 1622.5791206359863, + "load_s": 1.296950799995102, + "first_query_after_load_ms": 1.7228000215254724, + "time_to_ready_s": 1.2986736000166275 + }, + "mem_end": { + "rss_mb": 1762.80859375, + "peak_mb": 1834.61328125 + }, + "status": "ok", + "finished": "2026-11-16 09:32:10", + "load_before_pct": 6.6625000000000005, + "load_contaminated": false, + "wall_s": 266.9826946258545, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/usearch__text384__100000__core+mutation+saveload.json b/benchmarks/competitive/results/raw/usearch__text384__100000__core+mutation+saveload.json new file mode 100644 index 0000000..484d83e --- /dev/null +++ b/benchmarks/competitive/results/raw/usearch__text384__100000__core+mutation+saveload.json @@ -0,0 +1,468 @@ +{ + "system": "usearch", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 07:56:47", + "mem_baseline": { + "rss_mb": 37.51171875, + "peak_mb": 37.51171875 + }, + "version": "usearch 2.26.6", + "build": { + "ingest_s": 5.863001300080214, + "index_s": 2.5999615900218487e-06, + "total_s": 5.863003900041804, + "first_query_ms": 2.5531999999657273, + "ready_s": 5.8655571000417694, + "total_s_runs": [ + 5.863003900041804, + 5.733341399987694, + 6.0116672000149265 + ] + }, + "mem_after_build": { + "rss_mb": 219.58203125, + "peak_mb": 250.08203125 + }, + "mem_after_first_query": { + "rss_mb": 219.703125, + "peak_mb": 250.08203125 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.61185, + "qps_1t_runs": [ + 10699.678717746468, + 12257.680972262828, + 11865.471602766822 + ], + "qps_1t": 11865.471602766822, + "qps_1t_min": 10699.678717746468, + "qps_1t_max": 12257.680972262828, + "median_ms": 0.07960002403706312, + "p95_ms": 0.12690503499470654, + "p99_ms": 0.172304030857049, + "mean_ms": 0.08644013352265272, + "knob": 10, + "batch": { + "recall": 0.61185, + "qps_batch_runs": [ + 100641.08380517158, + 97938.87652022688, + 96350.71649683625 + ], + "qps_batch": 97938.87652022688, + "qps_batch_min": 96350.71649683625, + "qps_batch_max": 100641.08380517158, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7104, + "qps_1t_runs": [ + 8530.32344209426, + 7906.328968693185, + 8812.443860092553 + ], + "qps_1t": 8530.32344209426, + "qps_1t_min": 7906.328968693185, + "qps_1t_max": 8812.443860092553, + "median_ms": 0.10880001354962587, + "p95_ms": 0.16560498042963448, + "p99_ms": 0.21780097100418064, + "mean_ms": 0.11906189982740518, + "knob": 16, + "batch": { + "recall": 0.7104, + "qps_batch_runs": [ + 71225.57848736992, + 79789.67441438432, + 81063.22535712857 + ], + "qps_batch": 79789.67441438432, + "qps_batch_min": 71225.57848736992, + "qps_batch_max": 81063.22535712857, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7848, + "qps_1t_runs": [ + 6731.395754631413, + 6755.9944317212585, + 6269.462777002373 + ], + "qps_1t": 6731.395754631413, + "qps_1t_min": 6269.462777002373, + "qps_1t_max": 6755.9944317212585, + "median_ms": 0.14044999261386693, + "p95_ms": 0.19750499050132933, + "p99_ms": 0.26801900297869036, + "mean_ms": 0.1520258666035564, + "knob": 24, + "batch": { + "recall": 0.7848, + "qps_batch_runs": [ + 54533.65547796089, + 58747.675846622864, + 60987.323791175004 + ], + "qps_batch": 58747.675846622864, + "qps_batch_min": 54533.65547796089, + "qps_batch_max": 60987.323791175004, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8294, + "qps_1t_runs": [ + 5330.226176255889, + 5371.659770193802, + 5488.245271587127 + ], + "qps_1t": 5371.659770193802, + "qps_1t_min": 5330.226176255889, + "qps_1t_max": 5488.245271587127, + "median_ms": 0.17180002760142088, + "p95_ms": 0.22860999451950195, + "p99_ms": 0.2799119782866913, + "mean_ms": 0.1853263665786168, + "knob": 32, + "batch": { + "recall": 0.8294, + "qps_batch_runs": [ + 45147.74599771415, + 48488.84516066182, + 49929.84856335997 + ], + "qps_batch": 48488.84516066182, + "qps_batch_min": 45147.74599771415, + "qps_batch_max": 49929.84856335997, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8844, + "qps_1t_runs": [ + 4106.810754089227, + 4055.057951099779, + 4092.170409055547 + ], + "qps_1t": 4092.170409055547, + "qps_1t_min": 4055.057951099779, + "qps_1t_max": 4106.810754089227, + "median_ms": 0.23519998649135232, + "p95_ms": 0.3022099728696049, + "p99_ms": 0.3680390078807258, + "mean_ms": 0.24482421677870056, + "knob": 48, + "batch": { + "recall": 0.8844, + "qps_batch_runs": [ + 33536.733643833795, + 33181.30764467409, + 36037.26974287856 + ], + "qps_batch": 33536.733643833795, + "qps_batch_min": 33181.30764467409, + "qps_batch_max": 36037.26974287856, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.91485, + "qps_1t_runs": [ + 3227.9514774261716, + 3248.1331783857645, + 3164.0993669390855 + ], + "qps_1t": 3227.9514774261716, + "qps_1t_min": 3164.0993669390855, + "qps_1t_max": 3248.1331783857645, + "median_ms": 0.2987499756272882, + "p95_ms": 0.3779999678954482, + "p99_ms": 0.45251301082316814, + "mean_ms": 0.31123628333443776, + "knob": 64, + "batch": { + "recall": 0.91485, + "qps_batch_runs": [ + 26210.94569523811, + 25761.11205407629, + 28458.28656619501 + ], + "qps_batch": 26210.94569523811, + "qps_batch_min": 25761.11205407629, + "qps_batch_max": 28458.28656619501, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9468, + "qps_1t_runs": [ + 2290.17397480596, + 2333.795269558051, + 2352.842082347129 + ], + "qps_1t": 2333.795269558051, + "qps_1t_min": 2290.17397480596, + "qps_1t_max": 2352.842082347129, + "median_ms": 0.4180999821983278, + "p95_ms": 0.5196149984840304, + "p99_ms": 0.6176080455770718, + "mean_ms": 0.430050849817538, + "knob": 96, + "batch": { + "recall": 0.9468, + "qps_batch_runs": [ + 19483.720374828627, + 20839.80234978227, + 20803.363482419667 + ], + "qps_batch": 20803.363482419667, + "qps_batch_min": 19483.720374828627, + "qps_batch_max": 20839.80234978227, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.96425, + "qps_1t_runs": [ + 1833.2582024325486, + 1775.4250651327163, + 1820.0640949159829 + ], + "qps_1t": 1820.0640949159829, + "qps_1t_min": 1775.4250651327163, + "qps_1t_max": 1833.2582024325486, + "median_ms": 0.539999979082495, + "p95_ms": 0.6812099512899294, + "p99_ms": 0.8476079930551352, + "mean_ms": 0.5527178329842476, + "knob": 128, + "batch": { + "recall": 0.96425, + "qps_batch_runs": [ + 14550.862177947769, + 14806.336814294576, + 16095.905848644685 + ], + "qps_batch": 14806.336814294576, + "qps_batch_min": 14550.862177947769, + "qps_batch_max": 16095.905848644685, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.979, + "qps_1t_runs": [ + 1266.1291410883366, + 1296.8990303849139, + 1306.9749596369795 + ], + "qps_1t": 1296.8990303849139, + "qps_1t_min": 1266.1291410883366, + "qps_1t_max": 1306.9749596369795, + "median_ms": 0.7648000027984381, + "p95_ms": 0.9273150149965659, + "p99_ms": 1.0611080465605485, + "mean_ms": 0.7753348330055209, + "knob": 192, + "batch": { + "recall": 0.979, + "qps_batch_runs": [ + 10812.751259208284, + 11567.453581478621, + 11369.592481371807 + ], + "qps_batch": 11369.592481371807, + "qps_batch_min": 10812.751259208284, + "qps_batch_max": 11567.453581478621, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9854, + "qps_1t_runs": [ + 1000.5676723145737, + 1028.7305937045494, + 1019.9870019656166 + ], + "qps_1t": 1019.9870019656166, + "qps_1t_min": 1000.5676723145737, + "qps_1t_max": 1028.7305937045494, + "median_ms": 0.9744999988470227, + "p95_ms": 1.1896099778823555, + "p99_ms": 1.3780320150544874, + "mean_ms": 0.9839697000473583, + "knob": 256, + "batch": { + "recall": 0.9854, + "qps_batch_runs": [ + 8865.224648840747, + 8830.080550985269, + 9006.915509959501 + ], + "qps_batch": 8865.224648840747, + "qps_batch_min": 8830.080550985269, + "qps_batch_max": 9006.915509959501, + "threads": 32 + } + }, + { + "nq": 1759, + "recall_queries": 1759, + "recall": 0.9913018760659466, + "qps_1t_runs": [ + 656.1471496655383, + 679.2696743304351, + 691.571198001111 + ], + "qps_1t": 679.2696743304351, + "qps_1t_min": 656.1471496655383, + "qps_1t_max": 691.571198001111, + "median_ms": 1.455599965993315, + "p95_ms": 1.805500010959804, + "p99_ms": 2.2556999907828836, + "mean_ms": 1.4807334846264728, + "knob": 400, + "batch": { + "recall": 0.99115, + "qps_batch_runs": [ + 5737.157301596366, + 6206.819929218143, + 5836.416906433218 + ], + "qps_batch": 5836.416906433218, + "qps_batch_min": 5737.157301596366, + "qps_batch_max": 6206.819929218143, + "threads": 32 + } + }, + { + "nq": 1355, + "recall_queries": 1355, + "recall": 0.9950553505535056, + "qps_1t_runs": [ + 451.45550569803294, + 459.5257985769428, + 443.3981628480064 + ], + "qps_1t": 451.45550569803294, + "qps_1t_min": 443.3981628480064, + "qps_1t_max": 459.5257985769428, + "median_ms": 2.1974999690428376, + "p95_ms": 2.7249800274148583, + "p99_ms": 3.2800079789012715, + "mean_ms": 2.2155077986792646, + "knob": 640, + "batch": { + "recall": 0.9947, + "qps_batch_runs": [ + 4035.423756900103, + 3628.4345173594993, + 3704.1193883276096 + ], + "qps_batch": 3704.1193883276096, + "qps_batch_min": 3628.4345173594993, + "qps_batch_max": 4035.423756900103, + "threads": 32 + } + } + ], + "knob_name": "expansion_search", + "mem_after_sweep": { + "rss_mb": 224.46484375, + "peak_mb": 250.08203125 + }, + "ref_knob": 128, + "ref_recall_pre": 0.96425, + "mutation": { + "ref_knob": 128, + "recall_before": 0.96425, + "insert_10k": { + "seconds": 1.1832042000023648, + "per_s": 8451.626523959274, + "batch": 1000 + }, + "update_10k": { + "seconds": 1.1856134999543428, + "per_s": 8434.451868492635, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.0029568999889306724, + "per_s": 3381920.266980819, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 274.125, + "peak_mb": 430.4609375 + }, + "first_query_after_ms": 3.576699993573129, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.95805, + "qps_1t_runs": [ + 1604.565823716765, + 1642.7042660657337, + 1636.1476065261588 + ], + "qps_1t": 1636.1476065261588, + "qps_1t_min": 1604.565823716765, + "qps_1t_max": 1642.7042660657337, + "median_ms": 0.6064499903004616, + "p95_ms": 0.755919993389398, + "p99_ms": 0.9202139644185084, + "mean_ms": 0.6143885498750024 + }, + "recall_after": 0.95805 + }, + "saveload": { + "save_s": 0.1780498999869451, + "disk_mb": 176.71466445922852, + "load_s": 0.1356891999603249, + "first_query_after_load_ms": 0.9485000045970082, + "time_to_ready_s": 0.13663769996492192 + }, + "mem_end": { + "rss_mb": 235.62890625, + "peak_mb": 430.4609375 + }, + "status": "ok", + "finished": "2026-11-16 07:57:57", + "load_before_pct": 3.256249999999997, + "load_contaminated": false, + "wall_s": 70.40881705284119, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/raw/usearch__text384__100000__filter.json b/benchmarks/competitive/results/raw/usearch__text384__100000__filter.json new file mode 100644 index 0000000..3ab41f5 --- /dev/null +++ b/benchmarks/competitive/results/raw/usearch__text384__100000__filter.json @@ -0,0 +1,38 @@ +{ + "system": "usearch", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 2000, + "all_threads": 32, + "exps": [ + "filter" + ], + "started": "2026-11-16 08:38:15", + "filter": { + "build_with_metadata_s": 5.679011999978684, + "mem": { + "rss_mb": 220.6875, + "peak_mb": 251.28515625 + }, + "subsets": { + "f10": { + "unsupported": true + }, + "f01": { + "unsupported": true + } + } + }, + "mem_end": { + "rss_mb": 44.49609375, + "peak_mb": 251.28515625 + }, + "status": "ok", + "finished": "2026-11-16 08:38:21", + "load_before_pct": 5.878125000000004, + "load_contaminated": false, + "wall_s": 5.924612522125244, + "flag": null +} \ No newline at end of file diff --git a/benchmarks/competitive/results/run_log.txt b/benchmarks/competitive/results/run_log.txt new file mode 100644 index 0000000..d16b865 --- /dev/null +++ b/benchmarks/competitive/results/run_log.txt @@ -0,0 +1,308 @@ +[07:51:17] START isong_py_ivf__text384__100000__core+mutation+saveload load_before=6.8% +[07:53:01] END isong_py_ivf__text384__100000__core+mutation+saveload ok 103s +[07:53:08] START faiss_ivfflat__text384__100000__core+mutation+saveload load_before=9.5% +[07:54:14] END faiss_ivfflat__text384__100000__core+mutation+saveload ok 66s +[07:54:21] START faiss_hnsw__text384__100000__core+mutation+saveload load_before=7.3% +[07:55:32] END faiss_hnsw__text384__100000__core+mutation+saveload ok 70s +[07:55:38] START hnswlib__text384__100000__core+mutation+saveload load_before=4.6% +[07:56:40] END hnswlib__text384__100000__core+mutation+saveload ok 62s +[07:56:47] START usearch__text384__100000__core+mutation+saveload load_before=3.3% +[07:57:57] END usearch__text384__100000__core+mutation+saveload ok 70s +[07:58:05] START chromadb__text384__100000__core+mutation+saveload load_before=10.9% +[07:59:59] END chromadb__text384__100000__core+mutation+saveload ok 115s +[08:00:06] START lancedb_ivfflat__text384__100000__core+mutation+saveload load_before=4.5% +[08:02:16] END lancedb_ivfflat__text384__100000__core+mutation+saveload ok 130s +[08:02:23] START numpy_bf__text384__100000__core+mutation+saveload load_before=5.2% +[08:03:04] END numpy_bf__text384__100000__core+mutation+saveload ok 41s +[08:03:11] START isong_py__text384__100000__core+mutation+saveload load_before=5.1% +[08:04:24] END isong_py__text384__100000__core+mutation+saveload ok 73s +[08:04:31] START faiss_flat__text384__100000__core+mutation+saveload load_before=3.7% +[08:05:11] END faiss_flat__text384__100000__core+mutation+saveload ok 40s +[08:05:18] START lancedb_flat__text384__100000__core+mutation+saveload load_before=4.2% +[08:05:58] END lancedb_flat__text384__100000__core+mutation+saveload ok 40s +[08:06:05] START sqlite_vec__text384__100000__core+mutation+saveload load_before=3.2% +[08:07:03] END sqlite_vec__text384__100000__core+mutation+saveload ok 58s +[08:07:10] START rust_default__text384__100000__core+mutation+saveload load_before=5.6% +[08:08:47] END rust_default__text384__100000__core+mutation+saveload ok 97s +[08:08:54] START rust_native__text384__100000__core+mutation load_before=4.1% +[08:10:27] END rust_native__text384__100000__core+mutation ok 93s +[08:10:34] START cpp_default__text384__100000__core+mutation+saveload load_before=6.5% +[08:12:01] END cpp_default__text384__100000__core+mutation+saveload ok 87s +[08:12:08] START cpp_avx2__text384__100000__core+mutation+saveload load_before=6.2% +[08:13:24] END cpp_avx2__text384__100000__core+mutation+saveload ok 76s +[08:13:31] START csharp__text384__100000__core+mutation+saveload load_before=4.2% +[08:14:46] END csharp__text384__100000__core+mutation+saveload ok 75s +[08:14:53] START js__text384__100000__core+mutation+saveload load_before=5.6% +[08:17:27] END js__text384__100000__core+mutation+saveload ok 154s +[08:17:34] START isong_py_ivf__glove__100000__core+mutation+saveload load_before=3.8% +[08:18:34] END isong_py_ivf__glove__100000__core+mutation+saveload ok 60s +[08:18:41] START faiss_ivfflat__glove__100000__core+mutation+saveload load_before=4.3% +[08:19:07] END faiss_ivfflat__glove__100000__core+mutation+saveload ok 26s +[08:19:14] START faiss_hnsw__glove__100000__core+mutation+saveload load_before=12.4% +[08:19:45] END faiss_hnsw__glove__100000__core+mutation+saveload ok 31s +[08:19:52] START hnswlib__glove__100000__core+mutation+saveload load_before=4.3% +[08:20:13] END hnswlib__glove__100000__core+mutation+saveload ok 21s +[08:20:20] START usearch__glove__100000__core+mutation+saveload load_before=2.9% +[08:20:54] END usearch__glove__100000__core+mutation+saveload ok 34s +[08:21:01] START chromadb__glove__100000__core+mutation+saveload load_before=3.9% +[08:22:25] END chromadb__glove__100000__core+mutation+saveload ok 83s +[08:22:32] START lancedb_ivfflat__glove__100000__core+mutation+saveload load_before=3.8% +[08:24:42] END lancedb_ivfflat__glove__100000__core+mutation+saveload ok 130s +[08:24:49] START numpy_bf__glove__100000__core+mutation+saveload load_before=3.6% +[08:25:18] END numpy_bf__glove__100000__core+mutation+saveload ok 29s +[08:25:25] START isong_py__glove__100000__core+mutation+saveload load_before=4.9% +[08:26:04] END isong_py__glove__100000__core+mutation+saveload ok 39s +[08:26:11] START faiss_flat__glove__100000__core+mutation+saveload load_before=7.2% +[08:26:32] END faiss_flat__glove__100000__core+mutation+saveload ok 21s +[08:26:39] START lancedb_flat__glove__100000__core+mutation+saveload load_before=6.8% +[08:27:10] END lancedb_flat__glove__100000__core+mutation+saveload ok 32s +[08:27:17] START sqlite_vec__glove__100000__core+mutation+saveload load_before=4.4% +[08:27:55] END sqlite_vec__glove__100000__core+mutation+saveload ok 38s +[08:28:02] START rust_default__glove__100000__core+mutation+saveload load_before=5.5% +[08:28:53] END rust_default__glove__100000__core+mutation+saveload ok 51s +[08:29:00] START rust_native__glove__100000__core+mutation load_before=4.2% +[08:29:50] END rust_native__glove__100000__core+mutation ok 50s +[08:29:57] START cpp_default__glove__100000__core+mutation+saveload load_before=3.4% +[08:30:42] END cpp_default__glove__100000__core+mutation+saveload ok 45s +[08:30:48] START cpp_avx2__glove__100000__core+mutation+saveload load_before=3.1% +[08:31:31] END cpp_avx2__glove__100000__core+mutation+saveload ok 43s +[08:31:38] START csharp__glove__100000__core+mutation+saveload load_before=4.3% +[08:32:28] END csharp__glove__100000__core+mutation+saveload ok 50s +[08:32:35] START js__glove__100000__core+mutation+saveload load_before=3.4% +[08:34:05] END js__glove__100000__core+mutation+saveload ok 90s +[08:34:12] START isong_py_ivf__text384__100000__filter load_before=6.9% +[08:34:45] END isong_py_ivf__text384__100000__filter ok 32s +[08:34:52] START faiss_ivfflat__text384__100000__filter load_before=6.9% +[08:35:21] END faiss_ivfflat__text384__100000__filter ok 30s +[08:35:28] START faiss_hnsw__text384__100000__filter load_before=3.2% +[08:36:25] END faiss_hnsw__text384__100000__filter ok 57s +[08:36:32] START hnswlib__text384__100000__filter load_before=3.6% +[08:38:08] END hnswlib__text384__100000__filter ok 96s +[08:38:15] START usearch__text384__100000__filter load_before=5.9% +[08:38:21] END usearch__text384__100000__filter ok 6s +[08:38:28] START chromadb__text384__100000__filter load_before=4.8% +[08:41:10] END chromadb__text384__100000__filter ok 161s +[08:41:16] START lancedb_ivfflat__text384__100000__filter load_before=5.2% +[08:42:51] END lancedb_ivfflat__text384__100000__filter ok 95s +[08:42:59] START numpy_bf__text384__100000__filter load_before=5.2% +[08:43:17] END numpy_bf__text384__100000__filter ok 18s +[08:43:24] START isong_py__text384__100000__filter load_before=4.2% +[08:43:46] END isong_py__text384__100000__filter ok 22s +[08:43:53] START faiss_flat__text384__100000__filter load_before=4.7% +[08:44:06] END faiss_flat__text384__100000__filter ok 13s +[08:44:13] START lancedb_flat__text384__100000__filter load_before=4.4% +[08:44:43] END lancedb_flat__text384__100000__filter ok 30s +[08:44:51] START sqlite_vec__text384__100000__filter load_before=3.9% +[08:45:12] END sqlite_vec__text384__100000__filter ok 21s +[08:45:19] START rust_default__text384__100000__filter load_before=5.9% +[08:45:32] END rust_default__text384__100000__filter ok 13s +[08:45:38] START cpp_default__text384__100000__filter load_before=4.3% +[08:46:21] END cpp_default__text384__100000__filter ok 43s +[08:46:28] START isong_py_ivf__glove__100000__filter load_before=5.4% +[08:46:54] END isong_py_ivf__glove__100000__filter ok 25s +[08:47:00] START faiss_ivfflat__glove__100000__filter load_before=3.2% +[08:47:28] END faiss_ivfflat__glove__100000__filter ok 27s +[08:47:34] START faiss_hnsw__glove__100000__filter load_before=6.2% +[08:48:15] END faiss_hnsw__glove__100000__filter ok 41s +[08:48:22] START hnswlib__glove__100000__filter load_before=3.2% +[08:49:42] END hnswlib__glove__100000__filter ok 80s +[08:49:49] START usearch__glove__100000__filter load_before=3.1% +[08:49:50] END usearch__glove__100000__filter ok 2s +[08:49:57] START chromadb__glove__100000__filter load_before=3.1% +[08:52:27] END chromadb__glove__100000__filter ok 150s +[08:52:33] START lancedb_ivfflat__glove__100000__filter load_before=3.5% +[08:54:04] END lancedb_ivfflat__glove__100000__filter ok 90s +[08:54:11] START numpy_bf__glove__100000__filter load_before=4.5% +[08:54:29] END numpy_bf__glove__100000__filter ok 18s +[08:54:36] START isong_py__glove__100000__filter load_before=3.5% +[08:54:53] END isong_py__glove__100000__filter ok 17s +[08:55:00] START faiss_flat__glove__100000__filter load_before=2.8% +[08:55:09] END faiss_flat__glove__100000__filter ok 10s +[08:55:16] START lancedb_flat__glove__100000__filter load_before=11.0% +[08:55:35] END lancedb_flat__glove__100000__filter ok 18s +[08:55:41] START sqlite_vec__glove__100000__filter load_before=4.2% +[08:56:00] END sqlite_vec__glove__100000__filter ok 19s +[08:56:07] START rust_default__glove__100000__filter load_before=4.2% +[08:56:13] END rust_default__glove__100000__filter ok 5s +[08:56:20] START cpp_default__glove__100000__filter load_before=3.3% +[08:57:02] END cpp_default__glove__100000__filter ok 42s +[08:57:08] START numpy_bf__text384__10000__core load_before=3.9% +[08:57:12] END numpy_bf__text384__10000__core ok 3s +[08:57:19] START isong_py__text384__10000__core load_before=6.0% +[08:57:24] END isong_py__text384__10000__core ok 5s +[08:57:31] START faiss_flat__text384__10000__core load_before=4.4% +[08:57:41] END faiss_flat__text384__10000__core ok 11s +[08:57:49] START lancedb_flat__text384__10000__core load_before=4.1% +[08:58:22] END lancedb_flat__text384__10000__core ok 33s +[08:58:29] START sqlite_vec__text384__10000__core load_before=4.6% +[08:58:54] END sqlite_vec__text384__10000__core ok 25s +[08:59:01] START qdrant_local__text384__10000__core load_before=3.6% +[09:00:05] END qdrant_local__text384__10000__core ok 64s +[09:00:11] START rust_default__text384__10000__core load_before=4.6% +[09:00:26] END rust_default__text384__10000__core ok 15s +[09:00:33] START rust_native__text384__10000__core load_before=6.1% +[09:00:48] END rust_native__text384__10000__core ok 15s +[09:00:55] START cpp_default__text384__10000__core load_before=4.4% +[09:01:19] END cpp_default__text384__10000__core ok 24s +[09:01:26] START cpp_avx2__text384__10000__core load_before=6.4% +[09:01:49] END cpp_avx2__text384__10000__core ok 23s +[09:01:56] START csharp__text384__10000__core load_before=5.2% +[09:02:18] END csharp__text384__10000__core ok 22s +[09:02:24] START js__text384__10000__core load_before=4.3% +[09:03:21] END js__text384__10000__core ok 57s +[09:03:29] START qdrant_local__text384__100000__core+mutation load_before=5.0% +[09:10:18] END qdrant_local__text384__100000__core+mutation ok 409s +[09:10:25] START isong_py_ivf__text384__1000000__core+mutation load_before=6.0% +[09:13:06] END isong_py_ivf__text384__1000000__core+mutation ok 161s +[09:13:13] START faiss_ivfflat__text384__1000000__core+mutation+saveload load_before=10.1% +[09:18:40] END faiss_ivfflat__text384__1000000__core+mutation+saveload ok 326s +[09:18:47] START faiss_hnsw__text384__1000000__core+mutation+saveload load_before=15.7% +[09:23:30] END faiss_hnsw__text384__1000000__core+mutation+saveload ok 284s +[09:23:37] START hnswlib__text384__1000000__core+mutation+saveload load_before=6.4% +[09:27:36] END hnswlib__text384__1000000__core+mutation+saveload ok 238s +[09:27:43] START usearch__text384__1000000__core+mutation+saveload load_before=6.7% +[09:32:10] END usearch__text384__1000000__core+mutation+saveload ok 267s +[09:32:18] START chromadb__text384__1000000__core+mutation load_before=7.5% +[09:40:28] END chromadb__text384__1000000__core+mutation ok 491s +[09:40:36] START lancedb_ivfflat__text384__1000000__core+mutation load_before=8.2% +[09:43:09] END lancedb_ivfflat__text384__1000000__core+mutation ok 154s +[09:43:17] START numpy_bf__text384__1000000__core+mutation load_before=5.9% +[09:44:38] END numpy_bf__text384__1000000__core+mutation ok 81s +[09:44:46] START isong_py__text384__1000000__core+mutation load_before=4.1% +[09:45:45] END isong_py__text384__1000000__core+mutation ok 60s +[09:45:52] START faiss_flat__text384__1000000__core+mutation+saveload load_before=3.5% +[09:46:49] END faiss_flat__text384__1000000__core+mutation+saveload ok 56s +[09:46:56] START lancedb_flat__text384__1000000__core+mutation load_before=3.5% +[09:49:25] END lancedb_flat__text384__1000000__core+mutation ok 149s +[09:49:32] START sqlite_vec__text384__1000000__core+mutation load_before=4.4% +[09:51:55] END sqlite_vec__text384__1000000__core+mutation ok 143s +[09:52:02] START rust_default__text384__1000000__core+mutation+saveload load_before=5.3% +[09:55:26] END rust_default__text384__1000000__core+mutation+saveload ok 204s +[09:55:33] START rust_native__text384__1000000__core+mutation load_before=3.4% +[09:58:16] END rust_native__text384__1000000__core+mutation ok 163s +[09:58:24] START cpp_default__text384__1000000__core+mutation load_before=5.7% +[10:00:06] END cpp_default__text384__1000000__core+mutation ok 102s +[10:00:13] START cpp_avx2__text384__1000000__core+mutation load_before=3.8% +[10:01:46] END cpp_avx2__text384__1000000__core+mutation ok 93s +[10:01:53] START csharp__text384__1000000__core+mutation load_before=2.7% +[10:03:32] END csharp__text384__1000000__core+mutation ok 99s +[10:03:39] START js__text384__1000000__core+mutation load_before=3.0% +[10:14:45] END js__text384__1000000__core+mutation ok 666s +[10:14:52] START isong_py_ivf__glove__1000000__core+mutation load_before=5.8% +[10:16:31] END isong_py_ivf__glove__1000000__core+mutation ok 99s +[10:16:38] START faiss_ivfflat__glove__1000000__core+mutation+saveload load_before=6.4% +[10:18:41] END faiss_ivfflat__glove__1000000__core+mutation+saveload ok 123s +[10:18:48] START faiss_hnsw__glove__1000000__core+mutation+saveload load_before=14.5% +[10:20:22] END faiss_hnsw__glove__1000000__core+mutation+saveload ok 93s +[10:20:29] START hnswlib__glove__1000000__core+mutation+saveload load_before=6.6% +[10:21:57] END hnswlib__glove__1000000__core+mutation+saveload ok 89s +[10:22:12] START usearch__glove__1000000__core+mutation+saveload load_before=9.1% +[10:24:58] END usearch__glove__1000000__core+mutation+saveload ok 166s +[10:25:05] START lancedb_ivfflat__glove__1000000__core+mutation load_before=8.3% +[10:27:34] END lancedb_ivfflat__glove__1000000__core+mutation ok 149s +[10:27:47] START numpy_bf__glove__1000000__core+mutation load_before=11.5% +[10:29:51] END numpy_bf__glove__1000000__core+mutation ok 124s +[10:30:04] START isong_py__glove__1000000__core+mutation load_before=7.5% +[10:31:13] END isong_py__glove__1000000__core+mutation ok 69s +[10:31:28] START faiss_flat__glove__1000000__core+mutation+saveload load_before=9.5% +[10:32:40] END faiss_flat__glove__1000000__core+mutation+saveload ok 73s +[10:32:54] START lancedb_flat__glove__1000000__core+mutation load_before=12.1% +[10:33:46] END lancedb_flat__glove__1000000__core+mutation ok 52s +[10:34:00] START rust_default__glove__1000000__core+mutation+saveload load_before=8.7% +[10:37:21] END rust_default__glove__1000000__core+mutation+saveload ok 201s +[10:37:35] START rust_native__glove__1000000__core+mutation load_before=5.8% +[10:40:12] END rust_native__glove__1000000__core+mutation ok 157s +[10:40:25] START cpp_default__glove__1000000__core+mutation load_before=4.4% +[10:41:49] END cpp_default__glove__1000000__core+mutation ok 83s +[10:42:02] START cpp_avx2__glove__1000000__core+mutation load_before=5.7% +[10:43:25] END cpp_avx2__glove__1000000__core+mutation ok 83s +[10:43:39] START csharp__glove__1000000__core+mutation load_before=5.2% +[10:45:05] END csharp__glove__1000000__core+mutation ok 86s +[10:45:19] START isong_py_ivf__text384__1000000__filter load_before=6.0% +[10:48:17] END isong_py_ivf__text384__1000000__filter ok 178s +[10:48:31] START faiss_ivfflat__text384__1000000__filter load_before=5.1% +[10:50:41] END faiss_ivfflat__text384__1000000__filter ok 130s +[10:50:55] START faiss_hnsw__text384__1000000__filter load_before=7.3% +[10:55:52] END faiss_hnsw__text384__1000000__filter ok 297s +[10:55:59] START hnswlib__text384__1000000__filter load_before=5.9% +[10:59:59] END hnswlib__text384__1000000__filter ok 240s +[11:00:13] START lancedb_ivfflat__text384__1000000__filter load_before=8.1% +[11:02:19] END lancedb_ivfflat__text384__1000000__filter ok 126s +[11:02:34] START numpy_bf__text384__1000000__filter load_before=9.7% +[11:02:56] END numpy_bf__text384__1000000__filter ok 22s +[11:03:04] START faiss_flat__text384__1000000__filter load_before=8.1% +[11:03:26] END faiss_flat__text384__1000000__filter ok 22s +[11:03:33] START rust_default__text384__1000000__filter load_before=8.1% +[11:05:23] END rust_default__text384__1000000__filter ok 110s +[11:05:30] START cpp_default__text384__1000000__filter load_before=7.2% +[11:06:54] END cpp_default__text384__1000000__filter ok 84s +[11:07:02] START isong_py_ivf__sift__100000__core+mutation+saveload load_before=8.2% +[11:07:59] END isong_py_ivf__sift__100000__core+mutation+saveload ok 58s +[11:08:08] START faiss_ivfflat__sift__100000__core+mutation+saveload load_before=9.1% +[11:08:35] END faiss_ivfflat__sift__100000__core+mutation+saveload ok 28s +[11:08:43] START faiss_hnsw__sift__100000__core+mutation+saveload load_before=8.3% +[11:09:04] END faiss_hnsw__sift__100000__core+mutation+saveload ok 21s +[11:09:13] START hnswlib__sift__100000__core+mutation+saveload load_before=8.9% +[11:09:27] END hnswlib__sift__100000__core+mutation+saveload ok 14s +[11:09:35] START usearch__sift__100000__core+mutation+saveload load_before=12.1% +[11:10:03] END usearch__sift__100000__core+mutation+saveload ok 28s +[11:10:11] START chromadb__sift__100000__core+mutation+saveload load_before=6.4% +[11:11:34] END chromadb__sift__100000__core+mutation+saveload ok 84s +[11:11:43] START lancedb_ivfflat__sift__100000__core+mutation+saveload load_before=8.3% +[11:14:01] END lancedb_ivfflat__sift__100000__core+mutation+saveload ok 138s +[11:14:10] START numpy_bf__sift__100000__core+mutation+saveload load_before=12.6% +[11:14:40] END numpy_bf__sift__100000__core+mutation+saveload ok 30s +[11:14:48] START isong_py__sift__100000__core+mutation+saveload load_before=15.3% +[11:15:55] END isong_py__sift__100000__core+mutation+saveload ok 68s +[11:16:08] START faiss_flat__sift__100000__core+mutation+saveload load_before=16.8% +[11:16:41] END faiss_flat__sift__100000__core+mutation+saveload ok 33s +[11:16:55] START lancedb_flat__sift__100000__core+mutation+saveload load_before=12.7% +[11:17:31] END lancedb_flat__sift__100000__core+mutation+saveload ok 36s +[11:17:45] START sqlite_vec__sift__100000__core+mutation+saveload load_before=12.0% +[11:18:48] END sqlite_vec__sift__100000__core+mutation+saveload ok 64s +[11:19:03] START rust_default__sift__100000__core+mutation+saveload load_before=10.3% +[11:20:35] END rust_default__sift__100000__core+mutation+saveload ok 92s +[11:20:49] START rust_native__sift__100000__core+mutation load_before=12.7% +[11:22:20] END rust_native__sift__100000__core+mutation ok 91s +[11:22:35] START cpp_default__sift__100000__core+mutation+saveload load_before=14.0% +[11:23:41] END cpp_default__sift__100000__core+mutation+saveload ok 66s +[11:23:56] START cpp_avx2__sift__100000__core+mutation+saveload load_before=14.3% +[11:24:59] END cpp_avx2__sift__100000__core+mutation+saveload ok 63s +[11:25:14] START csharp__sift__100000__core+mutation+saveload load_before=9.9% +[11:26:13] END csharp__sift__100000__core+mutation+saveload ok 59s +[11:26:27] START js__sift__100000__core+mutation+saveload load_before=11.1% +[11:28:25] END js__sift__100000__core+mutation+saveload ok 118s +[11:28:39] START isong_py_ivf__sift__1000000__core load_before=8.8% +QUEUE KILLED at sift 1M; starting reduced sift 1M +[11:28:51] START faiss_ivfflat__sift__1000000__core+saveload load_before=12.3% +[11:31:30] END faiss_ivfflat__sift__1000000__core+saveload ok 159s +[11:31:45] START faiss_hnsw__sift__1000000__core+saveload load_before=8.2% +[11:34:07] END faiss_hnsw__sift__1000000__core+saveload ok 142s +[11:34:21] START hnswlib__sift__1000000__core+saveload load_before=9.0% +[11:35:54] END hnswlib__sift__1000000__core+saveload ok 94s +[11:36:09] START numpy_bf__sift__1000000__core load_before=11.1% +[11:38:16] END numpy_bf__sift__1000000__core ok 127s +[11:38:32] START rust_default__sift__1000000__core+saveload load_before=7.6% +[11:41:53] END rust_default__sift__1000000__core+saveload ok 202s +[11:42:06] START cpp_default__sift__1000000__core load_before=12.0% +[11:43:21] END cpp_default__sift__1000000__core ok 75s +REDUCED SIFT 1M DONE +[11:43:32] START faiss_ivfflat__text384__100000__filter load_before=7.6% +[11:43:53] END faiss_ivfflat__text384__100000__filter ok 20s +[11:44:08] START faiss_hnsw__text384__100000__filter load_before=9.0% +[11:44:58] END faiss_hnsw__text384__100000__filter ok 50s +[11:45:12] START faiss_flat__text384__100000__filter load_before=8.8% +[11:45:27] END faiss_flat__text384__100000__filter ok 15s +[11:45:41] START faiss_ivfflat__glove__100000__filter load_before=9.9% +[11:45:53] END faiss_ivfflat__glove__100000__filter ok 12s +[11:46:07] START faiss_hnsw__glove__100000__filter load_before=7.7% +[11:46:42] END faiss_hnsw__glove__100000__filter ok 35s +[11:46:59] START faiss_flat__glove__100000__filter load_before=8.9% +[11:47:05] END faiss_flat__glove__100000__filter ok 7s +[11:47:20] START faiss_ivfflat__text384__1000000__filter load_before=8.8% +[11:48:45] END faiss_ivfflat__text384__1000000__filter ok 85s +[11:49:02] START faiss_hnsw__text384__1000000__filter load_before=8.9% +[11:54:17] END faiss_hnsw__text384__1000000__filter ok 315s +[11:54:31] START faiss_flat__text384__1000000__filter load_before=13.4% +[11:54:54] END faiss_flat__text384__1000000__filter ok 23s +FAISS FILTER RERUN DONE diff --git a/benchmarks/competitive/results/run_stdout.txt b/benchmarks/competitive/results/run_stdout.txt new file mode 100644 index 0000000..4a37139 --- /dev/null +++ b/benchmarks/competitive/results/run_stdout.txt @@ -0,0 +1,275 @@ +[07:51:17] START isong_py_ivf__text384__100000__core+mutation+saveload load_before=6.8% +[07:53:01] END isong_py_ivf__text384__100000__core+mutation+saveload ok 103s +[07:53:08] START faiss_ivfflat__text384__100000__core+mutation+saveload load_before=9.5% +[07:54:14] END faiss_ivfflat__text384__100000__core+mutation+saveload ok 66s +[07:54:21] START faiss_hnsw__text384__100000__core+mutation+saveload load_before=7.3% +[07:55:32] END faiss_hnsw__text384__100000__core+mutation+saveload ok 70s +[07:55:38] START hnswlib__text384__100000__core+mutation+saveload load_before=4.6% +[07:56:40] END hnswlib__text384__100000__core+mutation+saveload ok 62s +[07:56:47] START usearch__text384__100000__core+mutation+saveload load_before=3.3% +[07:57:57] END usearch__text384__100000__core+mutation+saveload ok 70s +[07:58:05] START chromadb__text384__100000__core+mutation+saveload load_before=10.9% +[07:59:59] END chromadb__text384__100000__core+mutation+saveload ok 115s +[08:00:06] START lancedb_ivfflat__text384__100000__core+mutation+saveload load_before=4.5% +[08:02:16] END lancedb_ivfflat__text384__100000__core+mutation+saveload ok 130s +[08:02:23] START numpy_bf__text384__100000__core+mutation+saveload load_before=5.2% +[08:03:04] END numpy_bf__text384__100000__core+mutation+saveload ok 41s +[08:03:11] START isong_py__text384__100000__core+mutation+saveload load_before=5.1% +[08:04:24] END isong_py__text384__100000__core+mutation+saveload ok 73s +[08:04:31] START faiss_flat__text384__100000__core+mutation+saveload load_before=3.7% +[08:05:11] END faiss_flat__text384__100000__core+mutation+saveload ok 40s +[08:05:18] START lancedb_flat__text384__100000__core+mutation+saveload load_before=4.2% +[08:05:58] END lancedb_flat__text384__100000__core+mutation+saveload ok 40s +[08:06:05] START sqlite_vec__text384__100000__core+mutation+saveload load_before=3.2% +[08:07:03] END sqlite_vec__text384__100000__core+mutation+saveload ok 58s +[08:07:10] START rust_default__text384__100000__core+mutation+saveload load_before=5.6% +[08:08:47] END rust_default__text384__100000__core+mutation+saveload ok 97s +[08:08:54] START rust_native__text384__100000__core+mutation load_before=4.1% +[08:10:27] END rust_native__text384__100000__core+mutation ok 93s +[08:10:34] START cpp_default__text384__100000__core+mutation+saveload load_before=6.5% +[08:12:01] END cpp_default__text384__100000__core+mutation+saveload ok 87s +[08:12:08] START cpp_avx2__text384__100000__core+mutation+saveload load_before=6.2% +[08:13:24] END cpp_avx2__text384__100000__core+mutation+saveload ok 76s +[08:13:31] START csharp__text384__100000__core+mutation+saveload load_before=4.2% +[08:14:46] END csharp__text384__100000__core+mutation+saveload ok 75s +[08:14:53] START js__text384__100000__core+mutation+saveload load_before=5.6% +[08:17:27] END js__text384__100000__core+mutation+saveload ok 154s +[08:17:34] START isong_py_ivf__glove__100000__core+mutation+saveload load_before=3.8% +[08:18:34] END isong_py_ivf__glove__100000__core+mutation+saveload ok 60s +[08:18:41] START faiss_ivfflat__glove__100000__core+mutation+saveload load_before=4.3% +[08:19:07] END faiss_ivfflat__glove__100000__core+mutation+saveload ok 26s +[08:19:14] START faiss_hnsw__glove__100000__core+mutation+saveload load_before=12.4% +[08:19:45] END faiss_hnsw__glove__100000__core+mutation+saveload ok 31s +[08:19:52] START hnswlib__glove__100000__core+mutation+saveload load_before=4.3% +[08:20:13] END hnswlib__glove__100000__core+mutation+saveload ok 21s +[08:20:20] START usearch__glove__100000__core+mutation+saveload load_before=2.9% +[08:20:54] END usearch__glove__100000__core+mutation+saveload ok 34s +[08:21:01] START chromadb__glove__100000__core+mutation+saveload load_before=3.9% +[08:22:25] END chromadb__glove__100000__core+mutation+saveload ok 83s +[08:22:32] START lancedb_ivfflat__glove__100000__core+mutation+saveload load_before=3.8% +[08:24:42] END lancedb_ivfflat__glove__100000__core+mutation+saveload ok 130s +[08:24:49] START numpy_bf__glove__100000__core+mutation+saveload load_before=3.6% +[08:25:18] END numpy_bf__glove__100000__core+mutation+saveload ok 29s +[08:25:25] START isong_py__glove__100000__core+mutation+saveload load_before=4.9% +[08:26:04] END isong_py__glove__100000__core+mutation+saveload ok 39s +[08:26:11] START faiss_flat__glove__100000__core+mutation+saveload load_before=7.2% +[08:26:32] END faiss_flat__glove__100000__core+mutation+saveload ok 21s +[08:26:39] START lancedb_flat__glove__100000__core+mutation+saveload load_before=6.8% +[08:27:10] END lancedb_flat__glove__100000__core+mutation+saveload ok 32s +[08:27:17] START sqlite_vec__glove__100000__core+mutation+saveload load_before=4.4% +[08:27:55] END sqlite_vec__glove__100000__core+mutation+saveload ok 38s +[08:28:02] START rust_default__glove__100000__core+mutation+saveload load_before=5.5% +[08:28:53] END rust_default__glove__100000__core+mutation+saveload ok 51s +[08:29:00] START rust_native__glove__100000__core+mutation load_before=4.2% +[08:29:50] END rust_native__glove__100000__core+mutation ok 50s +[08:29:57] START cpp_default__glove__100000__core+mutation+saveload load_before=3.4% +[08:30:42] END cpp_default__glove__100000__core+mutation+saveload ok 45s +[08:30:48] START cpp_avx2__glove__100000__core+mutation+saveload load_before=3.1% +[08:31:31] END cpp_avx2__glove__100000__core+mutation+saveload ok 43s +[08:31:38] START csharp__glove__100000__core+mutation+saveload load_before=4.3% +[08:32:28] END csharp__glove__100000__core+mutation+saveload ok 50s +[08:32:35] START js__glove__100000__core+mutation+saveload load_before=3.4% +[08:34:05] END js__glove__100000__core+mutation+saveload ok 90s +[08:34:12] START isong_py_ivf__text384__100000__filter load_before=6.9% +[08:34:45] END isong_py_ivf__text384__100000__filter ok 32s +[08:34:52] START faiss_ivfflat__text384__100000__filter load_before=6.9% +[08:35:21] END faiss_ivfflat__text384__100000__filter ok 30s +[08:35:28] START faiss_hnsw__text384__100000__filter load_before=3.2% +[08:36:25] END faiss_hnsw__text384__100000__filter ok 57s +[08:36:32] START hnswlib__text384__100000__filter load_before=3.6% +[08:38:08] END hnswlib__text384__100000__filter ok 96s +[08:38:15] START usearch__text384__100000__filter load_before=5.9% +[08:38:21] END usearch__text384__100000__filter ok 6s +[08:38:28] START chromadb__text384__100000__filter load_before=4.8% +[08:41:10] END chromadb__text384__100000__filter ok 161s +[08:41:16] START lancedb_ivfflat__text384__100000__filter load_before=5.2% +[08:42:51] END lancedb_ivfflat__text384__100000__filter ok 95s +[08:42:59] START numpy_bf__text384__100000__filter load_before=5.2% +[08:43:17] END numpy_bf__text384__100000__filter ok 18s +[08:43:24] START isong_py__text384__100000__filter load_before=4.2% +[08:43:46] END isong_py__text384__100000__filter ok 22s +[08:43:53] START faiss_flat__text384__100000__filter load_before=4.7% +[08:44:06] END faiss_flat__text384__100000__filter ok 13s +[08:44:13] START lancedb_flat__text384__100000__filter load_before=4.4% +[08:44:43] END lancedb_flat__text384__100000__filter ok 30s +[08:44:51] START sqlite_vec__text384__100000__filter load_before=3.9% +[08:45:12] END sqlite_vec__text384__100000__filter ok 21s +[08:45:19] START rust_default__text384__100000__filter load_before=5.9% +[08:45:32] END rust_default__text384__100000__filter ok 13s +[08:45:38] START cpp_default__text384__100000__filter load_before=4.3% +[08:46:21] END cpp_default__text384__100000__filter ok 43s +[08:46:28] START isong_py_ivf__glove__100000__filter load_before=5.4% +[08:46:54] END isong_py_ivf__glove__100000__filter ok 25s +[08:47:00] START faiss_ivfflat__glove__100000__filter load_before=3.2% +[08:47:28] END faiss_ivfflat__glove__100000__filter ok 27s +[08:47:34] START faiss_hnsw__glove__100000__filter load_before=6.2% +[08:48:15] END faiss_hnsw__glove__100000__filter ok 41s +[08:48:22] START hnswlib__glove__100000__filter load_before=3.2% +[08:49:42] END hnswlib__glove__100000__filter ok 80s +[08:49:49] START usearch__glove__100000__filter load_before=3.1% +[08:49:50] END usearch__glove__100000__filter ok 2s +[08:49:57] START chromadb__glove__100000__filter load_before=3.1% +[08:52:27] END chromadb__glove__100000__filter ok 150s +[08:52:33] START lancedb_ivfflat__glove__100000__filter load_before=3.5% +[08:54:04] END lancedb_ivfflat__glove__100000__filter ok 90s +[08:54:11] START numpy_bf__glove__100000__filter load_before=4.5% +[08:54:29] END numpy_bf__glove__100000__filter ok 18s +[08:54:36] START isong_py__glove__100000__filter load_before=3.5% +[08:54:53] END isong_py__glove__100000__filter ok 17s +[08:55:00] START faiss_flat__glove__100000__filter load_before=2.8% +[08:55:09] END faiss_flat__glove__100000__filter ok 10s +[08:55:16] START lancedb_flat__glove__100000__filter load_before=11.0% +[08:55:35] END lancedb_flat__glove__100000__filter ok 18s +[08:55:41] START sqlite_vec__glove__100000__filter load_before=4.2% +[08:56:00] END sqlite_vec__glove__100000__filter ok 19s +[08:56:07] START rust_default__glove__100000__filter load_before=4.2% +[08:56:13] END rust_default__glove__100000__filter ok 5s +[08:56:20] START cpp_default__glove__100000__filter load_before=3.3% +[08:57:02] END cpp_default__glove__100000__filter ok 42s +[08:57:08] START numpy_bf__text384__10000__core load_before=3.9% +[08:57:12] END numpy_bf__text384__10000__core ok 3s +[08:57:19] START isong_py__text384__10000__core load_before=6.0% +[08:57:24] END isong_py__text384__10000__core ok 5s +[08:57:31] START faiss_flat__text384__10000__core load_before=4.4% +[08:57:41] END faiss_flat__text384__10000__core ok 11s +[08:57:49] START lancedb_flat__text384__10000__core load_before=4.1% +[08:58:22] END lancedb_flat__text384__10000__core ok 33s +[08:58:29] START sqlite_vec__text384__10000__core load_before=4.6% +[08:58:54] END sqlite_vec__text384__10000__core ok 25s +[08:59:01] START qdrant_local__text384__10000__core load_before=3.6% +[09:00:05] END qdrant_local__text384__10000__core ok 64s +[09:00:11] START rust_default__text384__10000__core load_before=4.6% +[09:00:26] END rust_default__text384__10000__core ok 15s +[09:00:33] START rust_native__text384__10000__core load_before=6.1% +[09:00:48] END rust_native__text384__10000__core ok 15s +[09:00:55] START cpp_default__text384__10000__core load_before=4.4% +[09:01:19] END cpp_default__text384__10000__core ok 24s +[09:01:26] START cpp_avx2__text384__10000__core load_before=6.4% +[09:01:49] END cpp_avx2__text384__10000__core ok 23s +[09:01:56] START csharp__text384__10000__core load_before=5.2% +[09:02:18] END csharp__text384__10000__core ok 22s +[09:02:24] START js__text384__10000__core load_before=4.3% +[09:03:21] END js__text384__10000__core ok 57s +[09:03:29] START qdrant_local__text384__100000__core+mutation load_before=5.0% +[09:10:18] END qdrant_local__text384__100000__core+mutation ok 409s +[09:10:25] START isong_py_ivf__text384__1000000__core+mutation load_before=6.0% +[09:13:06] END isong_py_ivf__text384__1000000__core+mutation ok 161s +[09:13:13] START faiss_ivfflat__text384__1000000__core+mutation+saveload load_before=10.1% +[09:18:40] END faiss_ivfflat__text384__1000000__core+mutation+saveload ok 326s +[09:18:47] START faiss_hnsw__text384__1000000__core+mutation+saveload load_before=15.7% +[09:23:30] END faiss_hnsw__text384__1000000__core+mutation+saveload ok 284s +[09:23:37] START hnswlib__text384__1000000__core+mutation+saveload load_before=6.4% +[09:27:36] END hnswlib__text384__1000000__core+mutation+saveload ok 238s +[09:27:43] START usearch__text384__1000000__core+mutation+saveload load_before=6.7% +[09:32:10] END usearch__text384__1000000__core+mutation+saveload ok 267s +[09:32:18] START chromadb__text384__1000000__core+mutation load_before=7.5% +[09:40:28] END chromadb__text384__1000000__core+mutation ok 491s +[09:40:36] START lancedb_ivfflat__text384__1000000__core+mutation load_before=8.2% +[09:43:09] END lancedb_ivfflat__text384__1000000__core+mutation ok 154s +[09:43:17] START numpy_bf__text384__1000000__core+mutation load_before=5.9% +[09:44:38] END numpy_bf__text384__1000000__core+mutation ok 81s +[09:44:46] START isong_py__text384__1000000__core+mutation load_before=4.1% +[09:45:45] END isong_py__text384__1000000__core+mutation ok 60s +[09:45:52] START faiss_flat__text384__1000000__core+mutation+saveload load_before=3.5% +[09:46:49] END faiss_flat__text384__1000000__core+mutation+saveload ok 56s +[09:46:56] START lancedb_flat__text384__1000000__core+mutation load_before=3.5% +[09:49:25] END lancedb_flat__text384__1000000__core+mutation ok 149s +[09:49:32] START sqlite_vec__text384__1000000__core+mutation load_before=4.4% +[09:51:55] END sqlite_vec__text384__1000000__core+mutation ok 143s +[09:52:02] START rust_default__text384__1000000__core+mutation+saveload load_before=5.3% +[09:55:26] END rust_default__text384__1000000__core+mutation+saveload ok 204s +[09:55:33] START rust_native__text384__1000000__core+mutation load_before=3.4% +[09:58:16] END rust_native__text384__1000000__core+mutation ok 163s +[09:58:24] START cpp_default__text384__1000000__core+mutation load_before=5.7% +[10:00:06] END cpp_default__text384__1000000__core+mutation ok 102s +[10:00:13] START cpp_avx2__text384__1000000__core+mutation load_before=3.8% +[10:01:46] END cpp_avx2__text384__1000000__core+mutation ok 93s +[10:01:53] START csharp__text384__1000000__core+mutation load_before=2.7% +[10:03:32] END csharp__text384__1000000__core+mutation ok 99s +[10:03:39] START js__text384__1000000__core+mutation load_before=3.0% +[10:14:45] END js__text384__1000000__core+mutation ok 666s +[10:14:52] START isong_py_ivf__glove__1000000__core+mutation load_before=5.8% +[10:16:31] END isong_py_ivf__glove__1000000__core+mutation ok 99s +[10:16:38] START faiss_ivfflat__glove__1000000__core+mutation+saveload load_before=6.4% +[10:18:41] END faiss_ivfflat__glove__1000000__core+mutation+saveload ok 123s +[10:18:48] START faiss_hnsw__glove__1000000__core+mutation+saveload load_before=14.5% +[10:20:22] END faiss_hnsw__glove__1000000__core+mutation+saveload ok 93s +[10:20:29] START hnswlib__glove__1000000__core+mutation+saveload load_before=6.6% +[10:21:57] END hnswlib__glove__1000000__core+mutation+saveload ok 89s +[10:22:12] START usearch__glove__1000000__core+mutation+saveload load_before=9.1% +[10:24:58] END usearch__glove__1000000__core+mutation+saveload ok 166s +[10:25:05] START lancedb_ivfflat__glove__1000000__core+mutation load_before=8.3% +[10:27:34] END lancedb_ivfflat__glove__1000000__core+mutation ok 149s +[10:27:47] START numpy_bf__glove__1000000__core+mutation load_before=11.5% +[10:29:51] END numpy_bf__glove__1000000__core+mutation ok 124s +[10:30:04] START isong_py__glove__1000000__core+mutation load_before=7.5% +[10:31:13] END isong_py__glove__1000000__core+mutation ok 69s +[10:31:28] START faiss_flat__glove__1000000__core+mutation+saveload load_before=9.5% +[10:32:40] END faiss_flat__glove__1000000__core+mutation+saveload ok 73s +[10:32:54] START lancedb_flat__glove__1000000__core+mutation load_before=12.1% +[10:33:46] END lancedb_flat__glove__1000000__core+mutation ok 52s +[10:34:00] START rust_default__glove__1000000__core+mutation+saveload load_before=8.7% +[10:37:21] END rust_default__glove__1000000__core+mutation+saveload ok 201s +[10:37:35] START rust_native__glove__1000000__core+mutation load_before=5.8% +[10:40:12] END rust_native__glove__1000000__core+mutation ok 157s +[10:40:25] START cpp_default__glove__1000000__core+mutation load_before=4.4% +[10:41:49] END cpp_default__glove__1000000__core+mutation ok 83s +[10:42:02] START cpp_avx2__glove__1000000__core+mutation load_before=5.7% +[10:43:25] END cpp_avx2__glove__1000000__core+mutation ok 83s +[10:43:39] START csharp__glove__1000000__core+mutation load_before=5.2% +[10:45:05] END csharp__glove__1000000__core+mutation ok 86s +[10:45:19] START isong_py_ivf__text384__1000000__filter load_before=6.0% +[10:48:17] END isong_py_ivf__text384__1000000__filter ok 178s +[10:48:31] START faiss_ivfflat__text384__1000000__filter load_before=5.1% +[10:50:41] END faiss_ivfflat__text384__1000000__filter ok 130s +[10:50:55] START faiss_hnsw__text384__1000000__filter load_before=7.3% +[10:55:52] END faiss_hnsw__text384__1000000__filter ok 297s +[10:55:59] START hnswlib__text384__1000000__filter load_before=5.9% +[10:59:59] END hnswlib__text384__1000000__filter ok 240s +[11:00:13] START lancedb_ivfflat__text384__1000000__filter load_before=8.1% +[11:02:19] END lancedb_ivfflat__text384__1000000__filter ok 126s +[11:02:34] START numpy_bf__text384__1000000__filter load_before=9.7% +[11:02:56] END numpy_bf__text384__1000000__filter ok 22s +[11:03:04] START faiss_flat__text384__1000000__filter load_before=8.1% +[11:03:26] END faiss_flat__text384__1000000__filter ok 22s +[11:03:33] START rust_default__text384__1000000__filter load_before=8.1% +[11:05:23] END rust_default__text384__1000000__filter ok 110s +[11:05:30] START cpp_default__text384__1000000__filter load_before=7.2% +[11:06:54] END cpp_default__text384__1000000__filter ok 84s +[11:07:02] START isong_py_ivf__sift__100000__core+mutation+saveload load_before=8.2% +[11:07:59] END isong_py_ivf__sift__100000__core+mutation+saveload ok 58s +[11:08:08] START faiss_ivfflat__sift__100000__core+mutation+saveload load_before=9.1% +[11:08:35] END faiss_ivfflat__sift__100000__core+mutation+saveload ok 28s +[11:08:43] START faiss_hnsw__sift__100000__core+mutation+saveload load_before=8.3% +[11:09:04] END faiss_hnsw__sift__100000__core+mutation+saveload ok 21s +[11:09:13] START hnswlib__sift__100000__core+mutation+saveload load_before=8.9% +[11:09:27] END hnswlib__sift__100000__core+mutation+saveload ok 14s +[11:09:35] START usearch__sift__100000__core+mutation+saveload load_before=12.1% +[11:10:03] END usearch__sift__100000__core+mutation+saveload ok 28s +[11:10:11] START chromadb__sift__100000__core+mutation+saveload load_before=6.4% +[11:11:34] END chromadb__sift__100000__core+mutation+saveload ok 84s +[11:11:43] START lancedb_ivfflat__sift__100000__core+mutation+saveload load_before=8.3% +[11:14:01] END lancedb_ivfflat__sift__100000__core+mutation+saveload ok 138s +[11:14:10] START numpy_bf__sift__100000__core+mutation+saveload load_before=12.6% +[11:14:40] END numpy_bf__sift__100000__core+mutation+saveload ok 30s +[11:14:48] START isong_py__sift__100000__core+mutation+saveload load_before=15.3% +[11:15:55] END isong_py__sift__100000__core+mutation+saveload ok 68s +[11:16:08] START faiss_flat__sift__100000__core+mutation+saveload load_before=16.8% +[11:16:41] END faiss_flat__sift__100000__core+mutation+saveload ok 33s +[11:16:55] START lancedb_flat__sift__100000__core+mutation+saveload load_before=12.7% +[11:17:31] END lancedb_flat__sift__100000__core+mutation+saveload ok 36s +[11:17:45] START sqlite_vec__sift__100000__core+mutation+saveload load_before=12.0% +[11:18:48] END sqlite_vec__sift__100000__core+mutation+saveload ok 64s +[11:19:03] START rust_default__sift__100000__core+mutation+saveload load_before=10.3% +[11:20:35] END rust_default__sift__100000__core+mutation+saveload ok 92s +[11:20:49] START rust_native__sift__100000__core+mutation load_before=12.7% +[11:22:20] END rust_native__sift__100000__core+mutation ok 91s +[11:22:35] START cpp_default__sift__100000__core+mutation+saveload load_before=14.0% +[11:23:41] END cpp_default__sift__100000__core+mutation+saveload ok 66s +[11:23:56] START cpp_avx2__sift__100000__core+mutation+saveload load_before=14.3% +[11:24:59] END cpp_avx2__sift__100000__core+mutation+saveload ok 63s +[11:25:14] START csharp__sift__100000__core+mutation+saveload load_before=9.9% +[11:26:13] END csharp__sift__100000__core+mutation+saveload ok 59s +[11:26:27] START js__sift__100000__core+mutation+saveload load_before=11.1% +[11:28:25] END js__sift__100000__core+mutation+saveload ok 118s +[11:28:39] START isong_py_ivf__sift__1000000__core load_before=8.8% diff --git a/benchmarks/competitive/results/run_stdout_filter.txt b/benchmarks/competitive/results/run_stdout_filter.txt new file mode 100644 index 0000000..7da6929 --- /dev/null +++ b/benchmarks/competitive/results/run_stdout_filter.txt @@ -0,0 +1,18 @@ +[11:43:32] START faiss_ivfflat__text384__100000__filter load_before=7.6% +[11:43:53] END faiss_ivfflat__text384__100000__filter ok 20s +[11:44:08] START faiss_hnsw__text384__100000__filter load_before=9.0% +[11:44:58] END faiss_hnsw__text384__100000__filter ok 50s +[11:45:12] START faiss_flat__text384__100000__filter load_before=8.8% +[11:45:27] END faiss_flat__text384__100000__filter ok 15s +[11:45:41] START faiss_ivfflat__glove__100000__filter load_before=9.9% +[11:45:53] END faiss_ivfflat__glove__100000__filter ok 12s +[11:46:07] START faiss_hnsw__glove__100000__filter load_before=7.7% +[11:46:42] END faiss_hnsw__glove__100000__filter ok 35s +[11:46:59] START faiss_flat__glove__100000__filter load_before=8.9% +[11:47:05] END faiss_flat__glove__100000__filter ok 7s +[11:47:20] START faiss_ivfflat__text384__1000000__filter load_before=8.8% +[11:48:45] END faiss_ivfflat__text384__1000000__filter ok 85s +[11:49:02] START faiss_hnsw__text384__1000000__filter load_before=8.9% +[11:54:17] END faiss_hnsw__text384__1000000__filter ok 315s +[11:54:31] START faiss_flat__text384__1000000__filter load_before=13.4% +[11:54:54] END faiss_flat__text384__1000000__filter ok 23s diff --git a/benchmarks/competitive/results/run_stdout_sift.txt b/benchmarks/competitive/results/run_stdout_sift.txt new file mode 100644 index 0000000..171a743 --- /dev/null +++ b/benchmarks/competitive/results/run_stdout_sift.txt @@ -0,0 +1,12 @@ +[11:28:51] START faiss_ivfflat__sift__1000000__core+saveload load_before=12.3% +[11:31:30] END faiss_ivfflat__sift__1000000__core+saveload ok 159s +[11:31:45] START faiss_hnsw__sift__1000000__core+saveload load_before=8.2% +[11:34:07] END faiss_hnsw__sift__1000000__core+saveload ok 142s +[11:34:21] START hnswlib__sift__1000000__core+saveload load_before=9.0% +[11:35:54] END hnswlib__sift__1000000__core+saveload ok 94s +[11:36:09] START numpy_bf__sift__1000000__core load_before=11.1% +[11:38:16] END numpy_bf__sift__1000000__core ok 127s +[11:38:32] START rust_default__sift__1000000__core+saveload load_before=7.6% +[11:41:53] END rust_default__sift__1000000__core+saveload ok 202s +[11:42:06] START cpp_default__sift__1000000__core load_before=12.0% +[11:43:21] END cpp_default__sift__1000000__core ok 75s diff --git a/benchmarks/competitive/results/summary.json b/benchmarks/competitive/results/summary.json new file mode 100644 index 0000000..6289acd --- /dev/null +++ b/benchmarks/competitive/results/summary.json @@ -0,0 +1,31525 @@ +{ + "chromadb|glove|100000": { + "mem_baseline": { + "rss_mb": 35.1640625, + "peak_mb": 35.1640625 + }, + "version": "chromadb 1.5.9", + "build": { + "ingest_s": 9.46120769996196, + "index_s": 2.3999600671231747e-06, + "total_s": 9.461210099922027, + "first_query_ms": 87.71570003591478, + "ready_s": 9.548925799957942, + "repeat_error": "InternalError: Collection [bench] already exists", + "total_s_runs": [ + 9.461210099922027, + 9.34063710004557 + ] + }, + "mem_after_build": { + "rss_mb": 212.01171875, + "peak_mb": 237.36328125 + }, + "mem_after_first_query": { + "rss_mb": 299.6171875, + "peak_mb": 299.6171875 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.5396, + "qps_1t_runs": [ + 1411.6182952327244, + 1485.575176072503, + 1390.2194440882397 + ], + "qps_1t": 1411.6182952327244, + "qps_1t_min": 1390.2194440882397, + "qps_1t_max": 1485.575176072503, + "median_ms": 0.7173999911174178, + "p95_ms": 0.801014990429394, + "p99_ms": 0.9504890191601608, + "mean_ms": 0.7002858836591864, + "knob": 10, + "batch": { + "recall": 0.5396, + "qps_batch_runs": [ + 23630.43884378117, + 23243.81365924262, + 23763.55265438207 + ], + "qps_batch": 23630.43884378117, + "qps_batch_min": 23243.81365924262, + "qps_batch_max": 23763.55265438207, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6245, + "qps_1t_runs": [ + 1357.176113461374, + 1403.578605862074, + 1449.7641853361804 + ], + "qps_1t": 1403.578605862074, + "qps_1t_min": 1357.176113461374, + "qps_1t_max": 1449.7641853361804, + "median_ms": 0.714199966751039, + "p95_ms": 0.8147100306814536, + "p99_ms": 0.9422010148409754, + "mean_ms": 0.7130186498689, + "knob": 16, + "batch": { + "recall": 0.6245, + "qps_batch_runs": [ + 20451.75889459749, + 20916.43257128759, + 20762.114948411614 + ], + "qps_batch": 20762.114948411614, + "qps_batch_min": 20451.75889459749, + "qps_batch_max": 20916.43257128759, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7422, + "qps_1t_runs": [ + 1291.2050983481727, + 1367.012231102707, + 1381.4532169504039 + ], + "qps_1t": 1367.012231102707, + "qps_1t_min": 1291.2050983481727, + "qps_1t_max": 1381.4532169504039, + "median_ms": 0.7527499983552843, + "p95_ms": 0.8599049731856212, + "p99_ms": 0.9532359824515887, + "mean_ms": 0.7432893500857366, + "knob": 32, + "batch": { + "recall": 0.7422, + "qps_batch_runs": [ + 15781.184411407197, + 15763.149219952671, + 14749.839042658305 + ], + "qps_batch": 15763.149219952671, + "qps_batch_min": 14749.839042658305, + "qps_batch_max": 15781.184411407197, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8288, + "qps_1t_runs": [ + 1159.9888509017046, + 1131.9797228966868, + 1153.0134124115882 + ], + "qps_1t": 1153.0134124115882, + "qps_1t_min": 1131.9797228966868, + "qps_1t_max": 1159.9888509017046, + "median_ms": 0.8609999786131084, + "p95_ms": 0.966024966328405, + "p99_ms": 1.1247230076696728, + "mean_ms": 0.870925967086805, + "knob": 64, + "batch": { + "recall": 0.8288, + "qps_batch_runs": [ + 9511.412984422934, + 9913.391654460556, + 10549.051773822624 + ], + "qps_batch": 9913.391654460556, + "qps_batch_min": 9511.412984422934, + "qps_batch_max": 10549.051773822624, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8957, + "qps_1t_runs": [ + 1050.9036703939914, + 1034.3695134095262, + 1041.6991649771956 + ], + "qps_1t": 1041.6991649771956, + "qps_1t_min": 1034.3695134095262, + "qps_1t_max": 1050.9036703939914, + "median_ms": 0.9402499999850988, + "p95_ms": 1.0893999744439498, + "p99_ms": 1.3479710184037723, + "mean_ms": 0.9594348505391584, + "knob": 128, + "batch": { + "recall": 0.8957, + "qps_batch_runs": [ + 5832.417157561866, + 6356.371897821059, + 6482.120690483515 + ], + "qps_batch": 6356.371897821059, + "qps_batch_min": 5832.417157561866, + "qps_batch_max": 6482.120690483515, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.947, + "qps_1t_runs": [ + 980.5784194517371, + 933.3874082171179, + 955.7744524309082 + ], + "qps_1t": 955.7744524309082, + "qps_1t_min": 933.3874082171179, + "qps_1t_max": 980.5784194517371, + "median_ms": 1.0254000080749393, + "p95_ms": 1.205914985621348, + "p99_ms": 1.4304100058507199, + "mean_ms": 1.0458148994560663, + "knob": 256, + "batch": { + "recall": 0.947, + "qps_batch_runs": [ + 3683.8267298157884, + 3685.999303002262, + 3558.720760679231 + ], + "qps_batch": 3683.8267298157884, + "qps_batch_min": 3558.720760679231, + "qps_batch_max": 3685.999303002262, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.97665, + "qps_1t_runs": [ + 748.958142970622, + 776.0253786375672, + 752.5215490626313 + ], + "qps_1t": 752.5215490626313, + "qps_1t_min": 748.958142970622, + "qps_1t_max": 776.0253786375672, + "median_ms": 1.2944499612785876, + "p95_ms": 1.5026050299638882, + "p99_ms": 1.9172779883956554, + "mean_ms": 1.3175571169025109, + "knob": 512, + "batch": { + "recall": 0.97665, + "qps_batch_runs": [ + 2012.4269376767793, + 2069.4773203670516, + 2028.615858073334 + ], + "qps_batch": 2028.615858073334, + "qps_batch_min": 2012.4269376767793, + "qps_batch_max": 2069.4773203670516, + "threads": 32 + } + } + ], + "mutation": { + "ref_knob": 512, + "recall_before": 0.97665, + "insert_10k": { + "seconds": 2.4239940000115894, + "per_s": 4125.422752676858, + "batch": 1000 + }, + "update_10k": { + "seconds": 3.4582213000394404, + "per_s": 2891.659940873637, + "batch": 1000 + }, + "delete_10k": { + "seconds": 1.952752600016538, + "per_s": 5120.976410383634, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 358.84765625, + "peak_mb": 990.73828125 + }, + "first_query_after_ms": 89.03720002854243, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9762, + "qps_1t_runs": [ + 737.2386117471513, + 732.425701778321, + 784.6157871231162 + ], + "qps_1t": 737.2386117471513, + "qps_1t_min": 732.425701778321, + "qps_1t_max": 784.6157871231162, + "median_ms": 1.318100024946034, + "p95_ms": 1.5458050125744194, + "p99_ms": 1.7699079727754, + "mean_ms": 1.3320827665156685 + }, + "recall_after": 0.9762 + }, + "saveload": { + "save_s": 1.100008375942707e-06, + "disk_mb": 76.30125045776367, + "load_s": 0.011646299972198904, + "first_query_after_load_ms": 87.3268999857828, + "time_to_ready_s": 0.0989731999579817 + }, + "load_before_pct": 3.146874999999998, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 11.126682000001892, + "mem": { + "rss_mb": 216.97265625, + "peak_mb": 246.265625 + }, + "subsets": { + "f10": [ + { + "knob": 16, + "nq": 76, + "recall": 0.825, + "qps_1t": 5.4099673236627455, + "qps_1t_runs": [ + 5.4099673236627455 + ], + "median_ms": 39.79289997369051, + "p95_ms": 568.1091250007739, + "p99_ms": 686.7677500267746, + "mean_ms": 184.84400000459956 + }, + { + "knob": 32, + "nq": 86, + "recall": 0.9116279069767442, + "qps_1t": 4.229212082488843, + "qps_1t_runs": [ + 4.229212082488843 + ], + "median_ms": 39.50435001752339, + "p95_ms": 685.3640749905026, + "p99_ms": 1006.4198650361511, + "mean_ms": 236.45066279379196 + }, + { + "knob": 64, + "nq": 30, + "recall": 0.95, + "qps_1t": 1.8577230019097792, + "qps_1t_runs": [ + 1.8577230019097792 + ], + "median_ms": 540.5647500010673, + "p95_ms": 557.7135399827966, + "p99_ms": 565.5559779866599, + "mean_ms": 538.2933833364708 + }, + { + "knob": 128, + "nq": 30, + "recall": 0.9666666666666667, + "qps_1t": 2.5395738405206187, + "qps_1t_runs": [ + 2.5395738405206187 + ], + "median_ms": 415.0752999994438, + "p95_ms": 886.5208649687697, + "p99_ms": 968.2058709819103, + "mean_ms": 393.76685333748657 + }, + { + "knob": 512, + "nq": 30, + "recall": 1.0, + "qps_1t": 1.7990575744727344, + "qps_1t_runs": [ + 1.7990575744727344 + ], + "median_ms": 675.4120500118006, + "p95_ms": 1033.7916499964192, + "p99_ms": 1099.1252370225266, + "mean_ms": 555.8465800034659 + } + ], + "f01": [ + { + "knob": 16, + "nq": 146, + "recall": 0.9363013698630137, + "qps_1t": 28.028240179833855, + "qps_1t_runs": [ + 28.028240179833855 + ], + "median_ms": 16.321150003932416, + "p95_ms": 94.01652500673663, + "p99_ms": 111.20478500670292, + "mean_ms": 35.67830136975542 + }, + { + "knob": 32, + "nq": 133, + "recall": 0.9827067669172932, + "qps_1t": 21.191251716703626, + "qps_1t_runs": [ + 21.191251716703626 + ], + "median_ms": 18.92760000191629, + "p95_ms": 139.8339799954556, + "p99_ms": 146.48944400949404, + "mean_ms": 47.18928420881188 + }, + { + "knob": 64, + "nq": 121, + "recall": 0.9991735537190083, + "qps_1t": 17.934054968587994, + "qps_1t_runs": [ + 17.934054968587994 + ], + "median_ms": 51.08679999830201, + "p95_ms": 130.09569997666404, + "p99_ms": 155.6541200377978, + "mean_ms": 55.759838014968075 + }, + { + "knob": 128, + "nq": 30, + "recall": 1.0, + "qps_1t": 12.792725270408223, + "qps_1t_runs": [ + 12.792725270408223, + 11.853814024425251, + 12.904388589706025 + ], + "median_ms": 85.76095002354123, + "p95_ms": 132.44221999775618, + "p99_ms": 164.50325596379116, + "mean_ms": 80.00782555577138 + }, + { + "knob": 512, + "nq": 30, + "recall": 1.0, + "qps_1t": 8.812886554855199, + "qps_1t_runs": [ + 8.812886554855199, + 8.328435750887204, + 8.848317974363184 + ], + "median_ms": 111.08534998493269, + "p95_ms": 136.53670500498265, + "p99_ms": 137.85027002100833, + "mean_ms": 115.51886555583526 + } + ] + } + } + }, + "chromadb|sift|100000": { + "mem_baseline": { + "rss_mb": 35.38671875, + "peak_mb": 35.38671875 + }, + "version": "chromadb 1.5.9", + "build": { + "ingest_s": 9.058855700073764, + "index_s": 2.500019036233425e-06, + "total_s": 9.0588582000928, + "first_query_ms": 101.34489997290075, + "ready_s": 9.1602031000657, + "repeat_error": "InternalError: Collection [bench] already exists", + "total_s_runs": [ + 9.0588582000928, + 9.233196799992584 + ] + }, + "mem_after_build": { + "rss_mb": 224.796875, + "peak_mb": 254.6953125 + }, + "mem_after_first_query": { + "rss_mb": 323.96484375, + "peak_mb": 323.96484375 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7916, + "qps_1t_runs": [ + 1279.0723550829064, + 1280.9432647181459, + 1260.4226716388887 + ], + "qps_1t": 1279.0723550829064, + "qps_1t_min": 1260.4226716388887, + "qps_1t_max": 1280.9432647181459, + "median_ms": 0.7512000156566501, + "p95_ms": 0.990025029750541, + "p99_ms": 1.545555950142451, + "mean_ms": 0.7852919836198756, + "knob": 10, + "batch": { + "recall": 0.7916, + "qps_batch_runs": [ + 20821.68619806102, + 21362.728446020592, + 22498.35199659069 + ], + "qps_batch": 21362.728446020592, + "qps_batch_min": 20821.68619806102, + "qps_batch_max": 22498.35199659069, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.87145, + "qps_1t_runs": [ + 1240.157873965033, + 1307.064046966063, + 1258.3539772092918 + ], + "qps_1t": 1258.3539772092918, + "qps_1t_min": 1240.157873965033, + "qps_1t_max": 1307.064046966063, + "median_ms": 0.7659999828319997, + "p95_ms": 0.9340500109829011, + "p99_ms": 1.688574992585931, + "mean_ms": 0.7887037832891414, + "knob": 16, + "batch": { + "recall": 0.87145, + "qps_batch_runs": [ + 18336.259499059895, + 17760.191247496085, + 18543.67266745581 + ], + "qps_batch": 18336.259499059895, + "qps_batch_min": 17760.191247496085, + "qps_batch_max": 18543.67266745581, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.95235, + "qps_1t_runs": [ + 1116.0513738894722, + 1185.4238152440125, + 1162.8415392632346 + ], + "qps_1t": 1162.8415392632346, + "qps_1t_min": 1116.0513738894722, + "qps_1t_max": 1185.4238152440125, + "median_ms": 0.8177999989129603, + "p95_ms": 1.1075449932832273, + "p99_ms": 1.8296040198765695, + "mean_ms": 0.8665195499391606, + "knob": 32, + "batch": { + "recall": 0.95235, + "qps_batch_runs": [ + 13719.481874934498, + 12846.68381780063, + 10800.115777494071 + ], + "qps_batch": 12846.68381780063, + "qps_batch_min": 10800.115777494071, + "qps_batch_max": 13719.481874934498, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.98875, + "qps_1t_runs": [ + 1111.5342352092764, + 1128.7277940424103, + 1153.7014751985846 + ], + "qps_1t": 1128.7277940424103, + "qps_1t_min": 1111.5342352092764, + "qps_1t_max": 1153.7014751985846, + "median_ms": 0.8407999994233251, + "p95_ms": 1.0921349923592063, + "p99_ms": 1.7142300138948492, + "mean_ms": 0.8841286496802544, + "knob": 64, + "batch": { + "recall": 0.98875, + "qps_batch_runs": [ + 8209.87393334354, + 9026.484608662204, + 8413.49288739725 + ], + "qps_batch": 8413.49288739725, + "qps_batch_min": 8209.87393334354, + "qps_batch_max": 9026.484608662204, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99775, + "qps_1t_runs": [ + 1007.5157653753861, + 982.1403670349492, + 965.3772273263502 + ], + "qps_1t": 982.1403670349492, + "qps_1t_min": 965.3772273263502, + "qps_1t_max": 1007.5157653753861, + "median_ms": 0.952300033532083, + "p95_ms": 1.345434997347184, + "p99_ms": 2.211042952258152, + "mean_ms": 1.0155297335974562, + "knob": 128, + "batch": { + "recall": 0.99775, + "qps_batch_runs": [ + 5786.047582972032, + 5633.006261984994, + 5806.427076032027 + ], + "qps_batch": 5786.047582972032, + "qps_batch_min": 5633.006261984994, + "qps_batch_max": 5806.427076032027, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99975, + "qps_1t_runs": [ + 811.5561708338621, + 778.9418373912421, + 817.6880979812471 + ], + "qps_1t": 811.5561708338621, + "qps_1t_min": 778.9418373912421, + "qps_1t_max": 817.6880979812471, + "median_ms": 1.1875999625772238, + "p95_ms": 1.6578999930061404, + "p99_ms": 2.6186080311890714, + "mean_ms": 1.246317899650118, + "knob": 256, + "batch": { + "recall": 0.99975, + "qps_batch_runs": [ + 3651.678110394038, + 3650.97083880668, + 3659.8210930532878 + ], + "qps_batch": 3651.678110394038, + "qps_batch_min": 3650.97083880668, + "qps_batch_max": 3659.8210930532878, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9999, + "qps_1t_runs": [ + 764.3267603929414, + 783.3136339750433, + 757.292174812171 + ], + "qps_1t": 764.3267603929414, + "qps_1t_min": 757.292174812171, + "qps_1t_max": 783.3136339750433, + "median_ms": 1.235899981111288, + "p95_ms": 1.7222000140463938, + "p99_ms": 2.461191997281278, + "mean_ms": 1.3018210331210867, + "knob": 512, + "batch": { + "recall": 0.9999, + "qps_batch_runs": [ + 2148.9991358904995, + 2105.278892049367, + 2069.9918048063064 + ], + "qps_batch": 2105.278892049367, + "qps_batch_min": 2069.9918048063064, + "qps_batch_max": 2148.9991358904995, + "threads": 32 + } + } + ], + "mutation": { + "ref_knob": 32, + "recall_before": 0.95235, + "insert_10k": { + "seconds": 2.3344285000348464, + "per_s": 4283.703698721434, + "batch": 1000 + }, + "update_10k": { + "seconds": 3.343258799985051, + "per_s": 2991.093600066113, + "batch": 1000 + }, + "delete_10k": { + "seconds": 2.3275026999763213, + "per_s": 4296.450440251577, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 385.21484375, + "peak_mb": 1096.796875 + }, + "first_query_after_ms": 111.04819999309257, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.94235, + "qps_1t_runs": [ + 1222.9589997704738, + 1240.6607708217812, + 1198.929930712384 + ], + "qps_1t": 1222.9589997704738, + "qps_1t_min": 1198.929930712384, + "qps_1t_max": 1240.6607708217812, + "median_ms": 0.7790999952703714, + "p95_ms": 1.0326200048439205, + "p99_ms": 1.587519038585015, + "mean_ms": 0.8192627003105978 + }, + "recall_after": 0.94235 + }, + "saveload": { + "save_s": 1.00000761449337e-06, + "disk_mb": 88.51536178588867, + "load_s": 0.015021700004581362, + "first_query_after_load_ms": 108.12930000247434, + "time_to_ready_s": 0.1231510000070557 + }, + "load_before_pct": 6.4125000000000005, + "load_contaminated": false + }, + "chromadb|text384|1000000": { + "mem_baseline": { + "rss_mb": 37.3671875, + "peak_mb": 37.3671875 + }, + "version": "chromadb 1.5.9", + "build": { + "ingest_s": 342.8886743001058, + "index_s": 3.89997148886323e-06, + "total_s": 342.88867820007727, + "first_query_ms": 2027.104299981147, + "ready_s": 344.9157825000584 + }, + "mem_after_build": { + "rss_mb": 2132.7109375, + "peak_mb": 2184.09765625 + }, + "mem_after_first_query": { + "rss_mb": 4059.90234375, + "peak_mb": 4059.90234375 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.5578, + "qps_1t_runs": [ + 1205.0106272812911, + 1214.7080983704996, + 1198.4069340504425 + ], + "qps_1t": 1205.0106272812911, + "qps_1t_min": 1198.4069340504425, + "qps_1t_max": 1214.7080983704996, + "median_ms": 0.8210500236600637, + "p95_ms": 0.9600149758625776, + "p99_ms": 1.1412200162885715, + "mean_ms": 0.829184116582231, + "knob": 10, + "batch": { + "recall": 0.5578, + "qps_batch_runs": [ + 8618.531307650717, + 8350.375057569907, + 8636.510850235009 + ], + "qps_batch": 8618.531307650717, + "qps_batch_min": 8350.375057569907, + "qps_batch_max": 8636.510850235009, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6533, + "qps_1t_runs": [ + 1110.331474827692, + 1169.655623428848, + 1108.2821588331865 + ], + "qps_1t": 1110.331474827692, + "qps_1t_min": 1108.2821588331865, + "qps_1t_max": 1169.655623428848, + "median_ms": 0.8663999906275421, + "p95_ms": 1.0182100115343928, + "p99_ms": 1.3616110070142897, + "mean_ms": 0.8859605832452265, + "knob": 16, + "batch": { + "recall": 0.6533, + "qps_batch_runs": [ + 6626.2882747541325, + 6801.490342747759, + 6743.865948723625 + ], + "qps_batch": 6743.865948723625, + "qps_batch_min": 6626.2882747541325, + "qps_batch_max": 6801.490342747759, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.774, + "qps_1t_runs": [ + 991.1341563778788, + 991.759126275454, + 1006.5020535074455 + ], + "qps_1t": 991.759126275454, + "qps_1t_min": 991.1341563778788, + "qps_1t_max": 1006.5020535074455, + "median_ms": 0.9679999784566462, + "p95_ms": 1.150999974925071, + "p99_ms": 1.607160955318251, + "mean_ms": 1.0035981500113849, + "knob": 32, + "batch": { + "recall": 0.774, + "qps_batch_runs": [ + 5012.023845300445, + 5071.014487157763, + 5010.317496562268 + ], + "qps_batch": 5012.023845300445, + "qps_batch_min": 5010.317496562268, + "qps_batch_max": 5071.014487157763, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8717, + "qps_1t_runs": [ + 877.905510738874, + 850.9272830327142, + 887.2437024025633 + ], + "qps_1t": 877.905510738874, + "qps_1t_min": 850.9272830327142, + "qps_1t_max": 887.2437024025633, + "median_ms": 1.0912999860011041, + "p95_ms": 1.2907249911222607, + "p99_ms": 1.8880139989778402, + "mean_ms": 1.1471164500010975, + "knob": 64, + "batch": { + "recall": 0.8717, + "qps_batch_runs": [ + 3255.834700004518, + 3225.2680400956924, + 3293.421456575566 + ], + "qps_batch": 3255.834700004518, + "qps_batch_min": 3225.2680400956924, + "qps_batch_max": 3293.421456575566, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.93225, + "qps_1t_runs": [ + 645.9276522218141, + 687.0113390423369, + 712.9068758236804 + ], + "qps_1t": 687.0113390423369, + "qps_1t_min": 645.9276522218141, + "qps_1t_max": 712.9068758236804, + "median_ms": 1.4240500167943537, + "p95_ms": 1.7009250266710296, + "p99_ms": 2.4345310265198377, + "mean_ms": 1.4688162671615526, + "knob": 128, + "batch": { + "recall": 0.93225, + "qps_batch_runs": [ + 1882.2105132494921, + 1942.3080135793114, + 1939.0945902166181 + ], + "qps_batch": 1939.0945902166181, + "qps_batch_min": 1882.2105132494921, + "qps_batch_max": 1942.3080135793114, + "threads": 32 + } + }, + { + "nq": 1640, + "recall_queries": 1640, + "recall": 0.9673780487804878, + "qps_1t_runs": [ + 534.9663292883475, + 540.1008053611605, + 534.5814373422849 + ], + "qps_1t": 534.9663292883475, + "qps_1t_min": 534.5814373422849, + "qps_1t_max": 540.1008053611605, + "median_ms": 1.8252500158268958, + "p95_ms": 2.1456449816469103, + "p99_ms": 2.930787008954225, + "mean_ms": 1.863801706976511, + "knob": 256, + "batch": { + "recall": 0.96775, + "qps_batch_runs": [ + 1082.7923612422842, + 1113.5097347824662, + 1112.2864889623786 + ], + "qps_batch": 1112.2864889623786, + "qps_batch_min": 1082.7923612422842, + "qps_batch_max": 1113.5097347824662, + "threads": 32 + } + }, + { + "nq": 1103, + "recall_queries": 1103, + "recall": 0.985947416137806, + "qps_1t_runs": [ + 349.04625885335355, + 365.23727873695873, + 348.2116961656387 + ], + "qps_1t": 349.04625885335355, + "qps_1t_min": 348.2116961656387, + "qps_1t_max": 365.23727873695873, + "median_ms": 2.7698000194504857, + "p95_ms": 3.4996600123122334, + "p99_ms": 4.818207963835448, + "mean_ms": 2.824904049379257, + "knob": 512, + "batch": { + "recall": 0.98555, + "qps_batch_runs": [ + 613.7440303862056, + 617.1228701147598, + 613.5853377323836 + ], + "qps_batch": 613.7440303862056, + "qps_batch_min": 613.5853377323836, + "qps_batch_max": 617.1228701147598, + "threads": 32 + } + } + ], + "mutation": { + "ref_knob": 256, + "recall_before": 0.9673780487804878, + "insert_10k": { + "seconds": 12.669519199989736, + "per_s": 789.2959347666564, + "batch": 1000 + }, + "update_10k": { + "seconds": 14.411285399983171, + "per_s": 693.9006287400066, + "batch": 1000 + }, + "delete_10k": { + "seconds": 10.66952009999659, + "per_s": 937.2492770319817, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 15833.31640625, + "peak_mb": 15838.484375 + }, + "first_query_after_ms": 2631.103899970185, + "after": { + "nq": 1550, + "recall_queries": 1550, + "recall": 0.9681290322580646, + "qps_1t_runs": [ + 449.7962395113203, + 493.41251320443797, + 498.79109127194823 + ], + "qps_1t": 493.41251320443797, + "qps_1t_min": 449.7962395113203, + "qps_1t_max": 498.79109127194823, + "median_ms": 2.0158999832347035, + "p95_ms": 2.6002750091720377, + "p99_ms": 3.740299014607452, + "mean_ms": 2.084925999847411 + }, + "recall_after": 0.9681290322580646 + }, + "load_before_pct": 7.481250000000001, + "load_contaminated": false + }, + "chromadb|text384|100000": { + "mem_baseline": { + "rss_mb": 37.375, + "peak_mb": 37.375 + }, + "version": "chromadb 1.5.9", + "build": { + "ingest_s": 14.582159700046759, + "index_s": 2.7999631129205227e-06, + "total_s": 14.582162500009872, + "first_query_ms": 160.1226999773644, + "ready_s": 14.742285199987236, + "repeat_error": "InternalError: Collection [bench] already exists", + "total_s_runs": [ + 14.582162500009872, + 14.994863499887288 + ] + }, + "mem_after_build": { + "rss_mb": 325.97265625, + "peak_mb": 422.72265625 + }, + "mem_after_first_query": { + "rss_mb": 523.046875, + "peak_mb": 523.046875 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.61495, + "qps_1t_runs": [ + 1337.002336154772, + 1385.425998043155, + 1360.9946411944622 + ], + "qps_1t": 1360.9946411944622, + "qps_1t_min": 1337.002336154772, + "qps_1t_max": 1385.425998043155, + "median_ms": 0.7024999940767884, + "p95_ms": 0.8498000097461045, + "p99_ms": 0.9739159652963284, + "mean_ms": 0.7348327498572568, + "knob": 10, + "batch": { + "recall": 0.61495, + "qps_batch_runs": [ + 10713.732808321422, + 10772.605906865701, + 10647.257427143082 + ], + "qps_batch": 10713.732808321422, + "qps_batch_min": 10647.257427143082, + "qps_batch_max": 10772.605906865701, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.714, + "qps_1t_runs": [ + 1228.689756800662, + 1254.1955198401554, + 1254.5199561508741 + ], + "qps_1t": 1254.1955198401554, + "qps_1t_min": 1228.689756800662, + "qps_1t_max": 1254.5199561508741, + "median_ms": 0.7868499960750341, + "p95_ms": 0.9096049994695932, + "p99_ms": 1.0655050224158913, + "mean_ms": 0.8027722003947323, + "knob": 16, + "batch": { + "recall": 0.714, + "qps_batch_runs": [ + 8783.247887836216, + 8997.5270315521, + 8881.673024101685 + ], + "qps_batch": 8881.673024101685, + "qps_batch_min": 8783.247887836216, + "qps_batch_max": 8997.5270315521, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8313, + "qps_1t_runs": [ + 1186.6349543538458, + 1169.5553513053485, + 1131.201437733578 + ], + "qps_1t": 1169.5553513053485, + "qps_1t_min": 1131.201437733578, + "qps_1t_max": 1186.6349543538458, + "median_ms": 0.8484999998472631, + "p95_ms": 0.9878099575871602, + "p99_ms": 1.2817659811116768, + "mean_ms": 0.8605869331901582, + "knob": 32, + "batch": { + "recall": 0.8313, + "qps_batch_runs": [ + 6354.186821656716, + 6419.288678425107, + 6345.107208914823 + ], + "qps_batch": 6354.186821656716, + "qps_batch_min": 6345.107208914823, + "qps_batch_max": 6419.288678425107, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.91515, + "qps_1t_runs": [ + 1034.0508830241047, + 1029.0595079459292, + 1013.0796692854713 + ], + "qps_1t": 1029.0595079459292, + "qps_1t_min": 1013.0796692854713, + "qps_1t_max": 1034.0508830241047, + "median_ms": 0.9639000054448843, + "p95_ms": 1.1214050377020612, + "p99_ms": 1.3939080201089384, + "mean_ms": 0.9753068997524679, + "knob": 64, + "batch": { + "recall": 0.91515, + "qps_batch_runs": [ + 4130.46230402775, + 4137.2148684830645, + 4117.868212674134 + ], + "qps_batch": 4130.46230402775, + "qps_batch_min": 4117.868212674134, + "qps_batch_max": 4137.2148684830645, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9646, + "qps_1t_runs": [ + 858.4653721230983, + 850.5723441459293, + 898.1049223542241 + ], + "qps_1t": 858.4653721230983, + "qps_1t_min": 850.5723441459293, + "qps_1t_max": 898.1049223542241, + "median_ms": 1.1433499748818576, + "p95_ms": 1.337224995950237, + "p99_ms": 1.5410279761999852, + "mean_ms": 1.1513346497764967, + "knob": 128, + "batch": { + "recall": 0.9646, + "qps_batch_runs": [ + 2488.79606234071, + 2437.0123691111703, + 2500.2631525539473 + ], + "qps_batch": 2488.79606234071, + "qps_batch_min": 2437.0123691111703, + "qps_batch_max": 2500.2631525539473, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.987, + "qps_1t_runs": [ + 676.416672335202, + 680.0762977359431, + 703.2256083691258 + ], + "qps_1t": 680.0762977359431, + "qps_1t_min": 676.416672335202, + "qps_1t_max": 703.2256083691258, + "median_ms": 1.439899962861091, + "p95_ms": 1.6598050075117499, + "p99_ms": 1.9601509661879402, + "mean_ms": 1.456940232727599, + "knob": 256, + "batch": { + "recall": 0.987, + "qps_batch_runs": [ + 1430.8218047038515, + 1447.4542969800739, + 1444.6764755532777 + ], + "qps_batch": 1444.6764755532777, + "qps_batch_min": 1430.8218047038515, + "qps_batch_max": 1447.4542969800739, + "threads": 32 + } + }, + { + "nq": 1479, + "recall_queries": 1479, + "recall": 0.9954699121027721, + "qps_1t_runs": [ + 465.5699821358445, + 482.8548968397274, + 489.8952398501875 + ], + "qps_1t": 482.8548968397274, + "qps_1t_min": 465.5699821358445, + "qps_1t_max": 489.8952398501875, + "median_ms": 2.0620000432245433, + "p95_ms": 2.4147999938577414, + "p99_ms": 2.86400000564755, + "mean_ms": 2.086724363513948, + "knob": 512, + "batch": { + "recall": 0.99565, + "qps_batch_runs": [ + 813.3006207436933, + 827.8510964198846, + 821.2085989769488 + ], + "qps_batch": 821.2085989769488, + "qps_batch_min": 813.3006207436933, + "qps_batch_max": 827.8510964198846, + "threads": 32 + } + } + ], + "mutation": { + "ref_knob": 128, + "recall_before": 0.9646, + "insert_10k": { + "seconds": 3.489183099998627, + "per_s": 2866.0003540668117, + "batch": 1000 + }, + "update_10k": { + "seconds": 5.230450800037943, + "per_s": 1911.8810944416987, + "batch": 1000 + }, + "delete_10k": { + "seconds": 1.9751236999873072, + "per_s": 5062.974030469212, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 609.703125, + "peak_mb": 2046.71484375 + }, + "first_query_after_ms": 170.83680001087487, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.96495, + "qps_1t_runs": [ + 793.8609780147654, + 762.4688145663962, + 819.8130784180476 + ], + "qps_1t": 793.8609780147654, + "qps_1t_min": 762.4688145663962, + "qps_1t_max": 819.8130784180476, + "median_ms": 1.1723999923560768, + "p95_ms": 1.594814978307113, + "p99_ms": 2.7335589757422, + "mean_ms": 1.2636618999240454 + }, + "recall_after": 0.96495 + }, + "saveload": { + "save_s": 1.00000761449337e-06, + "disk_mb": 202.46568298339844, + "load_s": 0.012246400001458824, + "first_query_after_load_ms": 174.5604999596253, + "time_to_ready_s": 0.18680689996108413 + }, + "load_before_pct": 4.80625, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 16.517546100076288, + "mem": { + "rss_mb": 327.57421875, + "peak_mb": 429.1484375 + }, + "subsets": { + "f10": [ + { + "knob": 16, + "nq": 30, + "recall": 0.8966666666666666, + "qps_1t": 2.4678706480474504, + "qps_1t_runs": [ + 2.4678706480474504 + ], + "median_ms": 534.566399990581, + "p95_ms": 677.5786500045797, + "p99_ms": 683.3730260015, + "mean_ms": 405.2076233376283 + }, + { + "knob": 32, + "nq": 30, + "recall": 0.9533333333333334, + "qps_1t": 2.2375069471046465, + "qps_1t_runs": [ + 2.2375069471046465 + ], + "median_ms": 535.4829999851063, + "p95_ms": 689.375394990202, + "p99_ms": 795.0477479700932, + "mean_ms": 446.9259866629727 + }, + { + "knob": 64, + "nq": 30, + "recall": 0.9633333333333334, + "qps_1t": 1.8540625123915617, + "qps_1t_runs": [ + 1.8540625123915617 + ], + "median_ms": 608.4011499769986, + "p95_ms": 760.4019399877867, + "p99_ms": 911.2748709850713, + "mean_ms": 539.3561399988054 + }, + { + "knob": 128, + "nq": 30, + "recall": 0.9933333333333333, + "qps_1t": 1.8555886523058287, + "qps_1t_runs": [ + 1.8555886523058287 + ], + "median_ms": 537.2666000039317, + "p95_ms": 695.2215550118126, + "p99_ms": 783.1240159901791, + "mean_ms": 538.91254333624 + }, + { + "knob": 512, + "nq": 30, + "recall": 1.0, + "qps_1t": 1.9525564958746813, + "qps_1t_runs": [ + 1.9525564958746813 + ], + "median_ms": 550.6684000429232, + "p95_ms": 831.4975300163496, + "p99_ms": 835.723816017271, + "mean_ms": 512.1490733368167 + } + ], + "f01": [ + { + "knob": 16, + "nq": 38, + "recall": 0.9710526315789474, + "qps_1t": 20.928856808842635, + "qps_1t_runs": [ + 20.659439519756404, + 20.928856808842635, + 21.418056855482735 + ], + "median_ms": 19.827849988359958, + "p95_ms": 96.35248502308967, + "p99_ms": 100.01419603417162, + "mean_ms": 47.62483947795995 + }, + { + "knob": 32, + "nq": 128, + "recall": 0.996875, + "qps_1t": 19.661152628726846, + "qps_1t_runs": [ + 19.661152628726846 + ], + "median_ms": 22.473700024420395, + "p95_ms": 115.68687499966472, + "p99_ms": 142.1754209982464, + "mean_ms": 50.86171797165662 + }, + { + "knob": 64, + "nq": 30, + "recall": 1.0, + "qps_1t": 12.431096504342186, + "qps_1t_runs": [ + 10.769348624111169, + 12.431096504342186, + 14.973055984967703 + ], + "median_ms": 100.47009997651912, + "p95_ms": 134.63836998853367, + "p99_ms": 138.30098800593987, + "mean_ms": 80.02872889046557 + }, + { + "knob": 128, + "nq": 30, + "recall": 1.0, + "qps_1t": 9.500454090140131, + "qps_1t_runs": [ + 9.500454090140131, + 9.337574128170521, + 10.338529281931722 + ], + "median_ms": 103.30894999788143, + "p95_ms": 139.7542299790075, + "p99_ms": 171.637331971433, + "mean_ms": 103.02595999997317 + }, + { + "knob": 512, + "nq": 30, + "recall": 1.0, + "qps_1t": 7.901943622219323, + "qps_1t_runs": [ + 7.901943622219323, + 8.154924649097785, + 7.889694704501344 + ], + "median_ms": 124.30065000080504, + "p95_ms": 156.18595496634953, + "p99_ms": 172.3463370121317, + "mean_ms": 125.30801666693554 + } + ] + } + } + }, + "cpp_avx2_exact|glove|1000000": { + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.5555575, + "index_s": 0, + "total_s": 0.5555575, + "first_query_ms": 315.8191, + "ready_s": 0.8713766 + }, + "mem_baseline": { + "rss_mb": 6.75, + "peak_mb": 6.75 + }, + "mem_after_build": { + "rss_mb": 677.777344, + "peak_mb": 816.609375 + }, + "mem_after_first_query": { + "rss_mb": 677.777344, + "peak_mb": 816.609375 + }, + "sweep": [ + { + "nq": 30, + "recall": 1, + "qps_1t": 2.58211371, + "qps_1t_min": 2.58211371, + "qps_1t_max": 2.58211371, + "qps_1t_runs": [ + 2.58211371 + ], + "median_ms": 387.4538, + "p95_ms": 405.094, + "p99_ms": 407.2581, + "mean_ms": 387.279613, + "knob": null + } + ], + "load_before_pct": 5.715624999999997, + "load_contaminated": false + }, + "cpp_avx2_ivf|glove|1000000": { + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.5555575, + "index_s": 6.7325823, + "total_s": 7.2881398, + "first_query_ms": 0, + "ready_s": 7.2881398 + }, + "mem_baseline": { + "rss_mb": 6.75, + "peak_mb": 6.75 + }, + "mem_after_build": { + "rss_mb": 1085.95312, + "peak_mb": 1147.46875 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.2955, + "qps_1t": 7425.39612, + "qps_1t_min": 7276.11133, + "qps_1t_max": 7598.33959, + "qps_1t_runs": [ + 7425.39612, + 7276.11133, + 7598.33959 + ], + "median_ms": 0.137299998, + "p95_ms": 0.189699989, + "p99_ms": 0.229400001, + "mean_ms": 0.134572233, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.41905, + "qps_1t": 6022.5055, + "qps_1t_min": 5906.95715, + "qps_1t_max": 6306.30546, + "qps_1t_runs": [ + 6022.5055, + 5906.95715, + 6306.30546 + ], + "median_ms": 0.16450003, + "p95_ms": 0.232499966, + "p99_ms": 0.292000012, + "mean_ms": 0.164635733, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.533, + "qps_1t": 4520.19078, + "qps_1t_min": 4501.84801, + "qps_1t_max": 4652.21766, + "qps_1t_runs": [ + 4501.84801, + 4520.19078, + 4652.21766 + ], + "median_ms": 0.218299974, + "p95_ms": 0.317699974, + "p99_ms": 0.386000029, + "mean_ms": 0.219437283, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.63225, + "qps_1t": 2864.23963, + "qps_1t_min": 2803.91965, + "qps_1t_max": 3030.39348, + "qps_1t_runs": [ + 3030.39348, + 2803.91965, + 2864.23963 + ], + "median_ms": 0.339200022, + "p95_ms": 0.493599975, + "p99_ms": 0.55560004, + "mean_ms": 0.345255517, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.72055, + "qps_1t": 1824.70547, + "qps_1t_min": 1796.67981, + "qps_1t_max": 1840.2799, + "qps_1t_runs": [ + 1796.67981, + 1840.2799, + 1824.70547 + ], + "median_ms": 0.538499968, + "p95_ms": 0.784899981, + "p99_ms": 0.8715, + "mean_ms": 0.54933715, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.7965, + "qps_1t": 1111.30948, + "qps_1t_min": 1060.77498, + "qps_1t_max": 1129.52756, + "qps_1t_runs": [ + 1060.77498, + 1111.30948, + 1129.52756 + ], + "median_ms": 0.890400028, + "p95_ms": 1.31700002, + "p99_ms": 1.49649999, + "mean_ms": 0.90929075, + "knob": 32 + }, + { + "nq": 1991, + "recall": 0.861627323, + "qps_1t": 609.506209, + "qps_1t_min": 600.547956, + "qps_1t_max": 632.533153, + "qps_1t_runs": [ + 609.506209, + 600.547956, + 632.533153 + ], + "median_ms": 1.60229998, + "p95_ms": 2.33380002, + "p99_ms": 2.6371, + "mean_ms": 1.62892104, + "knob": 64 + }, + { + "nq": 943, + "recall": 0.920254507, + "qps_1t": 324.841053, + "qps_1t_min": 322.540097, + "qps_1t_max": 343.126833, + "qps_1t_runs": [ + 322.540097, + 324.841053, + 343.126833 + ], + "median_ms": 3.02300003, + "p95_ms": 4.13880002, + "p99_ms": 4.55409999, + "mean_ms": 3.03106419, + "knob": 128 + }, + { + "nq": 552, + "recall": 0.952717391, + "qps_1t": 176.162091, + "qps_1t_min": 170.0925, + "qps_1t_max": 182.710829, + "qps_1t_runs": [ + 182.710829, + 176.162091, + 170.0925 + ], + "median_ms": 5.63159998, + "p95_ms": 7.37599999, + "p99_ms": 8.03039997, + "mean_ms": 5.67629118, + "knob": 256 + } + ], + "mutation": { + "ref_knob": 256, + "recall_before": 0.952717391, + "insert_10k": { + "seconds": 0.1587293, + "per_s": 63000.3408 + }, + "update_10k": { + "seconds": 0.017508, + "per_s": 571167.466 + }, + "delete_10k": { + "seconds": 0.0127602, + "per_s": 783686.775 + }, + "first_query_after_ms": 86.5016, + "recall_after": 0.950809717, + "after": { + "nq": 494, + "recall": 0.950809717, + "qps_1t": 162.19649, + "qps_1t_min": 158.138802, + "qps_1t_max": 165.651104, + "qps_1t_runs": [ + 158.138802, + 162.19649, + 165.651104 + ], + "median_ms": 6.19879999, + "p95_ms": 7.9502, + "p99_ms": 8.59309995, + "mean_ms": 6.17523489 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 1000000, + "builds": 1 + }, + "mem_after": { + "rss_mb": 1102.50781, + "peak_mb": 1467.42188 + } + }, + "load_before_pct": 5.715624999999997, + "load_contaminated": false + }, + "cpp_avx2_exact|glove|100000": { + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.0418406, + "index_s": 0, + "total_s": 0.0418406, + "first_query_ms": 27.0559, + "ready_s": 0.0688965 + }, + "mem_baseline": { + "rss_mb": 6.734375, + "peak_mb": 6.734375 + }, + "mem_after_build": { + "rss_mb": 78.3359375, + "peak_mb": 90.5195312 + }, + "mem_after_first_query": { + "rss_mb": 78.3359375, + "peak_mb": 90.5195312 + }, + "sweep": [ + { + "nq": 126, + "recall": 1, + "qps_1t": 40.179886, + "qps_1t_min": 39.6986257, + "qps_1t_max": 41.1854439, + "qps_1t_runs": [ + 39.6986257, + 40.179886, + 41.1854439 + ], + "median_ms": 24.3726, + "p95_ms": 28.5183, + "p99_ms": 31.7103, + "mean_ms": 24.7860955, + "knob": null + } + ], + "load_before_pct": 3.134374999999998, + "load_contaminated": false + }, + "cpp_avx2_ivf|glove|100000": { + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.0418406, + "index_s": 0.2526785, + "total_s": 0.2945191, + "first_query_ms": 0, + "ready_s": 0.2945191 + }, + "mem_baseline": { + "rss_mb": 6.734375, + "peak_mb": 6.734375 + }, + "mem_after_build": { + "rss_mb": 117.976562, + "peak_mb": 137.757812 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.28485, + "qps_1t": 35648.0186, + "qps_1t_min": 33710.5544, + "qps_1t_max": 35884.867, + "qps_1t_runs": [ + 33710.5544, + 35884.867, + 35648.0186 + ], + "median_ms": 0.0282999827, + "p95_ms": 0.0342000276, + "p99_ms": 0.0385000021, + "mean_ms": 0.02852775, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.4074, + "qps_1t": 30431.7196, + "qps_1t_min": 29216.2307, + "qps_1t_max": 30692.1227, + "qps_1t_runs": [ + 30692.1227, + 29216.2307, + 30431.7196 + ], + "median_ms": 0.0328000169, + "p95_ms": 0.0408999622, + "p99_ms": 0.045599998, + "mean_ms": 0.0332232167, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.5255, + "qps_1t": 25324.6942, + "qps_1t_min": 24997.8127, + "qps_1t_max": 25644.8724, + "qps_1t_runs": [ + 25324.6942, + 24997.8127, + 25644.8724 + ], + "median_ms": 0.0390000059, + "p95_ms": 0.0490000239, + "p99_ms": 0.0549000106, + "mean_ms": 0.0394949333, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.63475, + "qps_1t": 19291.3329, + "qps_1t_min": 19116.8036, + "qps_1t_max": 19436.3271, + "qps_1t_runs": [ + 19291.3329, + 19116.8036, + 19436.3271 + ], + "median_ms": 0.0508999801, + "p95_ms": 0.0657999772, + "p99_ms": 0.074599986, + "mean_ms": 0.0518656, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.7357, + "qps_1t": 13080.3592, + "qps_1t_min": 13064.4581, + "qps_1t_max": 13093.6154, + "qps_1t_runs": [ + 13093.6154, + 13080.3592, + 13064.4581 + ], + "median_ms": 0.0749000465, + "p95_ms": 0.098199991, + "p99_ms": 0.120099983, + "mean_ms": 0.0764557166, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.81855, + "qps_1t": 8057.7158, + "qps_1t_min": 7722.42515, + "qps_1t_max": 8186.69596, + "qps_1t_runs": [ + 7722.42515, + 8186.69596, + 8057.7158 + ], + "median_ms": 0.122899946, + "p95_ms": 0.161000004, + "p99_ms": 0.184700009, + "mean_ms": 0.125249017, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.88455, + "qps_1t": 4786.74454, + "qps_1t_min": 4719.04905, + "qps_1t_max": 4841.50955, + "qps_1t_runs": [ + 4841.50955, + 4786.74454, + 4719.04905 + ], + "median_ms": 0.205399992, + "p95_ms": 0.265499984, + "p99_ms": 0.308600022, + "mean_ms": 0.2091215, + "knob": 64 + }, + { + "nq": 2000, + "recall": 0.93865, + "qps_1t": 2655.36402, + "qps_1t_min": 2627.55291, + "qps_1t_max": 2692.88532, + "qps_1t_runs": [ + 2627.55291, + 2655.36402, + 2692.88532 + ], + "median_ms": 0.371000031, + "p95_ms": 0.464400044, + "p99_ms": 0.523900031, + "mean_ms": 0.376175783, + "knob": 128 + }, + { + "nq": 2000, + "recall": 0.9741, + "qps_1t": 1407.55641, + "qps_1t_min": 1344.76988, + "qps_1t_max": 1414.43706, + "qps_1t_runs": [ + 1344.76988, + 1414.43706, + 1407.55641 + ], + "median_ms": 0.70829998, + "p95_ms": 0.882099965, + "p99_ms": 1.03340001, + "mean_ms": 0.720355934, + "knob": 256 + }, + { + "nq": 125, + "recall": 1, + "qps_1t": 41.6154464, + "qps_1t_min": 41.4861122, + "qps_1t_max": 41.6784255, + "qps_1t_runs": [ + 41.6784255, + 41.4861122, + 41.6154464 + ], + "median_ms": 23.537, + "p95_ms": 26.7577, + "p99_ms": 27.8641, + "mean_ms": 24.0424067, + "knob": 1265 + } + ], + "mutation": { + "ref_knob": 256, + "recall_before": 0.9741, + "insert_10k": { + "seconds": 0.0117463, + "per_s": 851331.91 + }, + "update_10k": { + "seconds": 0.00649349997, + "per_s": 1540001.55 + }, + "delete_10k": { + "seconds": 0.00375530001, + "per_s": 2662903.09 + }, + "first_query_after_ms": 19.8733, + "recall_after": 0.9725, + "after": { + "nq": 2000, + "recall": 0.9725, + "qps_1t": 1347.34954, + "qps_1t_min": 1331.85143, + "qps_1t_max": 1380.28235, + "qps_1t_runs": [ + 1331.85143, + 1380.28235, + 1347.34954 + ], + "median_ms": 0.723399979, + "p95_ms": 0.913900032, + "p99_ms": 1.14820001, + "mean_ms": 0.73917395 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 100000, + "builds": 1 + }, + "mem_after": { + "rss_mb": 137.554688, + "peak_mb": 160.539062 + } + }, + "saveload": { + "save_s": 4.2604119, + "disk_mb": 133.363812, + "load_s": 2.0163434, + "first_query_after_load_ms": 235.5158, + "time_to_ready_s": 2.2518592, + "note": "load = JSON parse + insert; first query builds the IVF index (never persisted)" + }, + "load_before_pct": 3.134374999999998, + "load_contaminated": false + }, + "cpp_avx2_exact|sift|100000": { + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.0491732, + "index_s": 0, + "total_s": 0.0491732, + "first_query_ms": 29.6095, + "ready_s": 0.0787827 + }, + "mem_baseline": { + "rss_mb": 7.12890625, + "peak_mb": 7.12890625 + }, + "mem_after_build": { + "rss_mb": 91.1835938, + "peak_mb": 109.460938 + }, + "mem_after_first_query": { + "rss_mb": 91.1835938, + "peak_mb": 109.460938 + }, + "sweep": [ + { + "nq": 97, + "recall": 1, + "qps_1t": 23.4311232, + "qps_1t_min": 23.333474, + "qps_1t_max": 27.2832601, + "qps_1t_runs": [ + 27.2832601, + 23.4311232, + 23.333474 + ], + "median_ms": 42.3488, + "p95_ms": 45.4001, + "p99_ms": 47.3759, + "mean_ms": 40.7292247, + "knob": null + } + ], + "load_before_pct": 14.274999999999986, + "load_contaminated": false + }, + "cpp_avx2_ivf|sift|100000": { + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.0491732, + "index_s": 0.6379454, + "total_s": 0.6871186, + "first_query_ms": 0, + "ready_s": 0.6871186 + }, + "mem_baseline": { + "rss_mb": 7.12890625, + "peak_mb": 7.12890625 + }, + "mem_after_build": { + "rss_mb": 141.824219, + "peak_mb": 167 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.27955, + "qps_1t": 23359.5194, + "qps_1t_min": 21573.1811, + "qps_1t_max": 23763.3551, + "qps_1t_runs": [ + 23359.5194, + 21573.1811, + 23763.3551 + ], + "median_ms": 0.0445999904, + "p95_ms": 0.0557000167, + "p99_ms": 0.0663000392, + "mean_ms": 0.0437481833, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.42195, + "qps_1t": 19562.7147, + "qps_1t_min": 18794.0427, + "qps_1t_max": 20323.6748, + "qps_1t_runs": [ + 19562.7147, + 18794.0427, + 20323.6748 + ], + "median_ms": 0.0508999801, + "p95_ms": 0.0659999787, + "p99_ms": 0.0765000004, + "mean_ms": 0.0511765666, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.5793, + "qps_1t": 16494.2196, + "qps_1t_min": 16249.7786, + "qps_1t_max": 17277.4236, + "qps_1t_runs": [ + 16494.2196, + 17277.4236, + 16249.7786 + ], + "median_ms": 0.0594999874, + "p95_ms": 0.0784000149, + "p99_ms": 0.0907999929, + "mean_ms": 0.0600152001, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.7401, + "qps_1t": 11994.6216, + "qps_1t_min": 11642.4668, + "qps_1t_max": 12084.3512, + "qps_1t_runs": [ + 11642.4668, + 11994.6216, + 12084.3512 + ], + "median_ms": 0.0819000416, + "p95_ms": 0.111699977, + "p99_ms": 0.126700033, + "mean_ms": 0.0840049333, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.86955, + "qps_1t": 7631.03633, + "qps_1t_min": 7570.06569, + "qps_1t_max": 7635.26639, + "qps_1t_runs": [ + 7570.06569, + 7631.03633, + 7635.26639 + ], + "median_ms": 0.127800042, + "p95_ms": 0.176700007, + "p99_ms": 0.203299976, + "mean_ms": 0.131371417, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.95215, + "qps_1t": 4693.14571, + "qps_1t_min": 4524.2277, + "qps_1t_max": 4768.34202, + "qps_1t_runs": [ + 4768.34202, + 4524.2277, + 4693.14571 + ], + "median_ms": 0.210799975, + "p95_ms": 0.293300021, + "p99_ms": 0.3363, + "mean_ms": 0.214608467, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.98825, + "qps_1t": 2719.49448, + "qps_1t_min": 2710.51007, + "qps_1t_max": 2761.44612, + "qps_1t_runs": [ + 2710.51007, + 2761.44612, + 2719.49448 + ], + "median_ms": 0.361000013, + "p95_ms": 0.484400021, + "p99_ms": 0.55279996, + "mean_ms": 0.366259583, + "knob": 64 + }, + { + "nq": 2000, + "recall": 0.9984, + "qps_1t": 1488.20043, + "qps_1t_min": 1403.35967, + "qps_1t_max": 1494.32984, + "qps_1t_runs": [ + 1488.20043, + 1403.35967, + 1494.32984 + ], + "median_ms": 0.675299962, + "p95_ms": 0.884700043, + "p99_ms": 1.01939996, + "mean_ms": 0.684574833, + "knob": 128 + }, + { + "nq": 2000, + "recall": 0.9999, + "qps_1t": 787.236813, + "qps_1t_min": 782.572394, + "qps_1t_max": 832.727455, + "qps_1t_runs": [ + 832.727455, + 787.236813, + 782.572394 + ], + "median_ms": 1.24119996, + "p95_ms": 1.51699997, + "p99_ms": 1.666, + "mean_ms": 1.24965865, + "knob": 256 + }, + { + "nq": 69, + "recall": 1, + "qps_1t": 23.0380589, + "qps_1t_min": 22.9235987, + "qps_1t_max": 23.2509506, + "qps_1t_runs": [ + 22.9235987, + 23.0380589, + 23.2509506 + ], + "median_ms": 43.3466, + "p95_ms": 46.3013, + "p99_ms": 48.4077, + "mean_ms": 43.346199, + "knob": 1265 + } + ], + "mutation": { + "ref_knob": 32, + "recall_before": 0.95215, + "insert_10k": { + "seconds": 0.029963, + "per_s": 333744.952 + }, + "update_10k": { + "seconds": 0.0114121, + "per_s": 876262.914 + }, + "delete_10k": { + "seconds": 0.00631789997, + "per_s": 1582804.42 + }, + "first_query_after_ms": 30.3824, + "recall_after": 0.949, + "after": { + "nq": 2000, + "recall": 0.949, + "qps_1t": 4870.65133, + "qps_1t_min": 4730.30744, + "qps_1t_max": 5086.68732, + "qps_1t_runs": [ + 5086.68732, + 4730.30744, + 4870.65133 + ], + "median_ms": 0.19749999, + "p95_ms": 0.274699996, + "p99_ms": 0.335499994, + "mean_ms": 0.204435233 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 100000, + "builds": 1 + }, + "mem_after": { + "rss_mb": 163.820312, + "peak_mb": 194.328125 + } + }, + "saveload": { + "save_s": 12.2242788, + "disk_mb": 133.678411, + "load_s": 5.1421373, + "first_query_after_load_ms": 635.6938, + "time_to_ready_s": 5.7778311, + "note": "load = JSON parse + insert; first query builds the IVF index (never persisted)" + }, + "load_before_pct": 14.274999999999986, + "load_contaminated": false + }, + "cpp_avx2_exact|text384|1000000": { + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.9221926, + "index_s": 0, + "total_s": 0.9221926, + "first_query_ms": 510.0265, + "ready_s": 1.4322191 + }, + "mem_baseline": { + "rss_mb": 11.0585938, + "peak_mb": 11.0585938 + }, + "mem_after_build": { + "rss_mb": 1794.36328, + "peak_mb": 1933.19531 + }, + "mem_after_first_query": { + "rss_mb": 1794.36328, + "peak_mb": 1933.19531 + }, + "sweep": [ + { + "nq": 30, + "recall": 1, + "qps_1t": 2.1345438, + "qps_1t_min": 2.1345438, + "qps_1t_max": 2.1345438, + "qps_1t_runs": [ + 2.1345438 + ], + "median_ms": 466.0438, + "p95_ms": 491.3723, + "p99_ms": 493.2179, + "mean_ms": 468.48418, + "knob": null + } + ], + "load_before_pct": 3.8312499999999994, + "load_contaminated": false + }, + "cpp_avx2_ivf|text384|1000000": { + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.9221926, + "index_s": 6.2006516, + "total_s": 7.1228442, + "first_query_ms": 0, + "ready_s": 7.1228442 + }, + "mem_baseline": { + "rss_mb": 11.0585938, + "peak_mb": 11.0585938 + }, + "mem_after_build": { + "rss_mb": 3290.5, + "peak_mb": 3525.36328 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.26665, + "qps_1t": 4780.26896, + "qps_1t_min": 4475.10388, + "qps_1t_max": 4797.09948, + "qps_1t_runs": [ + 4475.10388, + 4780.26896, + 4797.09948 + ], + "median_ms": 0.205699995, + "p95_ms": 0.280899985, + "p99_ms": 0.372799987, + "mean_ms": 0.213703683, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.3725, + "qps_1t": 3919.84927, + "qps_1t_min": 3908.62418, + "qps_1t_max": 3942.15906, + "qps_1t_runs": [ + 3942.15906, + 3919.84927, + 3908.62418 + ], + "median_ms": 0.245500007, + "p95_ms": 0.330200011, + "p99_ms": 0.380599988, + "mean_ms": 0.254874817, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.4895, + "qps_1t": 3015.91483, + "qps_1t_min": 2955.93528, + "qps_1t_max": 3038.65473, + "qps_1t_runs": [ + 3038.65473, + 3015.91483, + 2955.93528 + ], + "median_ms": 0.322799955, + "p95_ms": 0.431899971, + "p99_ms": 0.499300018, + "mean_ms": 0.332989917, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.60205, + "qps_1t": 1989.41096, + "qps_1t_min": 1984.49356, + "qps_1t_max": 2021.33172, + "qps_1t_runs": [ + 1984.49356, + 1989.41096, + 2021.33172 + ], + "median_ms": 0.488699996, + "p95_ms": 0.657500001, + "p99_ms": 0.771599996, + "mean_ms": 0.500430533, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.7007, + "qps_1t": 1243.44951, + "qps_1t_min": 1230.49587, + "qps_1t_max": 1245.15557, + "qps_1t_runs": [ + 1230.49587, + 1243.44951, + 1245.15557 + ], + "median_ms": 0.795300002, + "p95_ms": 1.0168, + "p99_ms": 1.15140004, + "mean_ms": 0.806669133, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.789, + "qps_1t": 712.564997, + "qps_1t_min": 695.861126, + "qps_1t_max": 718.010986, + "qps_1t_runs": [ + 695.861126, + 712.564997, + 718.010986 + ], + "median_ms": 1.39469997, + "p95_ms": 1.7433, + "p99_ms": 1.995, + "mean_ms": 1.41106182, + "knob": 32 + }, + { + "nq": 1301, + "recall": 0.860261337, + "qps_1t": 403.978489, + "qps_1t_min": 397.137242, + "qps_1t_max": 406.292684, + "qps_1t_runs": [ + 406.292684, + 403.978489, + 397.137242 + ], + "median_ms": 2.4466, + "p95_ms": 3.03749996, + "p99_ms": 3.41240002, + "mean_ms": 2.48489347, + "knob": 64 + }, + { + "nq": 714, + "recall": 0.914145658, + "qps_1t": 220.598281, + "qps_1t_min": 216.367715, + "qps_1t_max": 223.351688, + "qps_1t_runs": [ + 223.351688, + 216.367715, + 220.598281 + ], + "median_ms": 4.5022, + "p95_ms": 5.33060002, + "p99_ms": 5.77290001, + "mean_ms": 4.54404416, + "knob": 128 + }, + { + "nq": 360, + "recall": 0.954722222, + "qps_1t": 119.292006, + "qps_1t_min": 114.767574, + "qps_1t_max": 119.952711, + "qps_1t_runs": [ + 119.292006, + 114.767574, + 119.952711 + ], + "median_ms": 8.37580004, + "p95_ms": 9.74719995, + "p99_ms": 11.3932, + "mean_ms": 8.4775575, + "knob": 256 + } + ], + "mutation": { + "ref_knob": 256, + "recall_before": 0.954722222, + "insert_10k": { + "seconds": 0.2878924, + "per_s": 34735.1997 + }, + "update_10k": { + "seconds": 0.0162474, + "per_s": 615483.092 + }, + "delete_10k": { + "seconds": 0.00723399996, + "per_s": 1382361.08 + }, + "first_query_after_ms": 81.231, + "recall_after": 0.956056338, + "after": { + "nq": 355, + "recall": 0.956056338, + "qps_1t": 117.493688, + "qps_1t_min": 115.307031, + "qps_1t_max": 118.805164, + "qps_1t_runs": [ + 118.805164, + 115.307031, + 117.493688 + ], + "median_ms": 8.4082, + "p95_ms": 9.98079998, + "p99_ms": 11.7942, + "mean_ms": 8.53357869 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 1000000, + "builds": 1 + }, + "mem_after": { + "rss_mb": 3338.42969, + "peak_mb": 4755.55859 + } + }, + "load_before_pct": 3.8312499999999994, + "load_contaminated": false + }, + "cpp_avx2_exact|text384|100000": { + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.0811739, + "index_s": 0, + "total_s": 0.0811739, + "first_query_ms": 43.8393, + "ready_s": 0.1250132 + }, + "mem_baseline": { + "rss_mb": 11.0507812, + "peak_mb": 11.0507812 + }, + "mem_after_build": { + "rss_mb": 203.308594, + "peak_mb": 217.082031 + }, + "mem_after_first_query": { + "rss_mb": 203.308594, + "peak_mb": 217.082031 + }, + "sweep": [ + { + "nq": 68, + "recall": 1, + "qps_1t": 23.016217, + "qps_1t_min": 22.9903773, + "qps_1t_max": 23.1046586, + "qps_1t_runs": [ + 23.1046586, + 22.9903773, + 23.016217 + ], + "median_ms": 43.5436, + "p95_ms": 45.3769, + "p99_ms": 47.5699, + "mean_ms": 43.4084667, + "knob": null + } + ], + "load_before_pct": 6.212500000000001, + "load_contaminated": false + }, + "cpp_avx2_ivf|text384|100000": { + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.0811739, + "index_s": 0.5254779, + "total_s": 0.6066518, + "first_query_ms": 0, + "ready_s": 0.6066518 + }, + "mem_baseline": { + "rss_mb": 11.0507812, + "peak_mb": 11.0507812 + }, + "mem_after_build": { + "rss_mb": 353.316406, + "peak_mb": 427.917969 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.28085, + "qps_1t": 17263.8521, + "qps_1t_min": 17031.8291, + "qps_1t_max": 17851.9147, + "qps_1t_runs": [ + 17031.8291, + 17263.8521, + 17851.9147 + ], + "median_ms": 0.056000019, + "p95_ms": 0.0690000015, + "p99_ms": 0.0801000278, + "mean_ms": 0.0575515001, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.40215, + "qps_1t": 14309.5397, + "qps_1t_min": 14254.2023, + "qps_1t_max": 14358.2955, + "qps_1t_runs": [ + 14309.5397, + 14358.2955, + 14254.2023 + ], + "median_ms": 0.0679999939, + "p95_ms": 0.0863000168, + "p99_ms": 0.10179996, + "mean_ms": 0.0698947833, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.5291, + "qps_1t": 11488.1193, + "qps_1t_min": 11326.8242, + "qps_1t_max": 11627.1498, + "qps_1t_runs": [ + 11627.1498, + 11326.8242, + 11488.1193 + ], + "median_ms": 0.0849000062, + "p95_ms": 0.109800021, + "p99_ms": 0.131800014, + "mean_ms": 0.0871126834, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.6479, + "qps_1t": 8129.60211, + "qps_1t_min": 7994.6532, + "qps_1t_max": 8406.75231, + "qps_1t_runs": [ + 8129.60211, + 7994.6532, + 8406.75231 + ], + "median_ms": 0.117900025, + "p95_ms": 0.15789998, + "p99_ms": 0.212399988, + "mean_ms": 0.122347617, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.75475, + "qps_1t": 5301.21082, + "qps_1t_min": 5296.1894, + "qps_1t_max": 5315.19646, + "qps_1t_runs": [ + 5296.1894, + 5315.19646, + 5301.21082 + ], + "median_ms": 0.182699994, + "p95_ms": 0.242899987, + "p99_ms": 0.305099995, + "mean_ms": 0.188530317, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.84275, + "qps_1t": 3053.08552, + "qps_1t_min": 3021.49309, + "qps_1t_max": 3054.45031, + "qps_1t_runs": [ + 3053.08552, + 3054.45031, + 3021.49309 + ], + "median_ms": 0.319499988, + "p95_ms": 0.414400012, + "p99_ms": 0.485800032, + "mean_ms": 0.328630283, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.90905, + "qps_1t": 1696.79697, + "qps_1t_min": 1694.25263, + "qps_1t_max": 1723.32544, + "qps_1t_runs": [ + 1694.25263, + 1723.32544, + 1696.79697 + ], + "median_ms": 0.575600017, + "p95_ms": 0.716999988, + "p99_ms": 0.868900039, + "mean_ms": 0.586616633, + "knob": 64 + }, + { + "nq": 2000, + "recall": 0.95425, + "qps_1t": 901.545912, + "qps_1t_min": 900.405678, + "qps_1t_max": 905.743159, + "qps_1t_runs": [ + 900.405678, + 905.743159, + 901.545912 + ], + "median_ms": 1.09629997, + "p95_ms": 1.2994, + "p99_ms": 1.47660001, + "mean_ms": 1.1079607, + "knob": 128 + }, + { + "nq": 985, + "recall": 0.981319797, + "qps_1t": 454.985197, + "qps_1t_min": 452.333217, + "qps_1t_max": 461.837435, + "qps_1t_runs": [ + 452.333217, + 454.985197, + 461.837435 + ], + "median_ms": 2.15499999, + "p95_ms": 2.58510001, + "p99_ms": 2.97590002, + "mean_ms": 2.19129912, + "knob": 256 + }, + { + "nq": 65, + "recall": 1, + "qps_1t": 23.0463919, + "qps_1t_min": 23.0153493, + "qps_1t_max": 23.2042002, + "qps_1t_runs": [ + 23.0463919, + 23.2042002, + 23.0153493 + ], + "median_ms": 43.4106, + "p95_ms": 45.4693999, + "p99_ms": 46.4893, + "mean_ms": 43.3118836, + "knob": 1265 + } + ], + "mutation": { + "ref_knob": 128, + "recall_before": 0.95425, + "insert_10k": { + "seconds": 0.0380363, + "per_s": 262906.749 + }, + "update_10k": { + "seconds": 0.0141123, + "per_s": 708601.716 + }, + "delete_10k": { + "seconds": 0.00469430001, + "per_s": 2130243.06 + }, + "first_query_after_ms": 43.7709, + "recall_after": 0.95065, + "after": { + "nq": 2000, + "recall": 0.95065, + "qps_1t": 895.144397, + "qps_1t_min": 889.471908, + "qps_1t_max": 895.624222, + "qps_1t_runs": [ + 895.144397, + 895.624222, + 889.471908 + ], + "median_ms": 1.10910001, + "p95_ms": 1.30970002, + "p99_ms": 1.43160002, + "mean_ms": 1.1193135 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 100000, + "builds": 1 + }, + "mem_after": { + "rss_mb": 419.621094, + "peak_mb": 503.675781 + } + }, + "saveload": { + "save_s": 16.37915, + "disk_mb": 499.571589, + "load_s": 6.9462202, + "first_query_after_load_ms": 531.1766, + "time_to_ready_s": 7.4773968, + "note": "load = JSON parse + insert; first query builds the IVF index (never persisted)" + }, + "load_before_pct": 6.212500000000001, + "load_contaminated": false + }, + "cpp_avx2_exact|text384|10000": { + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.00655609998, + "index_s": 0, + "total_s": 0.00655609998, + "first_query_ms": 3.57950001, + "ready_s": 0.0101356 + }, + "mem_baseline": { + "rss_mb": 11.0507812, + "peak_mb": 11.0507812 + }, + "mem_after_build": { + "rss_mb": 32.7460938, + "peak_mb": 44.1601562 + }, + "mem_after_first_query": { + "rss_mb": 32.7460938, + "peak_mb": 44.1601562 + }, + "sweep": [ + { + "nq": 999, + "recall": 1, + "qps_1t": 327.830065, + "qps_1t_min": 307.938716, + "qps_1t_max": 328.352164, + "qps_1t_runs": [ + 327.830065, + 328.352164, + 307.938716 + ], + "median_ms": 3.08160001, + "p95_ms": 3.48419999, + "p99_ms": 3.84210004, + "mean_ms": 3.11442362, + "knob": null + } + ], + "load_before_pct": 6.406250000000001, + "load_contaminated": false + }, + "cpp_avx2_ivf|text384|10000": { + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.00655609998, + "index_s": 0.0559779, + "total_s": 0.062534, + "first_query_ms": 0, + "ready_s": 0.062534 + }, + "mem_baseline": { + "rss_mb": 11.0507812, + "peak_mb": 11.0507812 + }, + "mem_after_build": { + "rss_mb": 47.2226562, + "peak_mb": 62.328125 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.2963, + "qps_1t": 59166.6962, + "qps_1t_min": 57900.0244, + "qps_1t_max": 61641.4498, + "qps_1t_runs": [ + 57900.0244, + 59166.6962, + 61641.4498 + ], + "median_ms": 0.0164000085, + "p95_ms": 0.0199000351, + "p99_ms": 0.0280000386, + "mean_ms": 0.0167984667, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.41875, + "qps_1t": 56484.889, + "qps_1t_min": 56146.4974, + "qps_1t_max": 56753.3674, + "qps_1t_runs": [ + 56146.4974, + 56484.889, + 56753.3674 + ], + "median_ms": 0.0175000168, + "p95_ms": 0.020699983, + "p99_ms": 0.022799999, + "mean_ms": 0.0177114999, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.55125, + "qps_1t": 48555.2388, + "qps_1t_min": 46373.0479, + "qps_1t_max": 50370.9824, + "qps_1t_runs": [ + 50370.9824, + 48555.2388, + 46373.0479 + ], + "median_ms": 0.0196999754, + "p95_ms": 0.0236000051, + "p99_ms": 0.0265000272, + "mean_ms": 0.0206706833, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.6798, + "qps_1t": 39167.1495, + "qps_1t_min": 38511.3811, + "qps_1t_max": 39623.2623, + "qps_1t_runs": [ + 39623.2623, + 38511.3811, + 39167.1495 + ], + "median_ms": 0.024399953, + "p95_ms": 0.0299999956, + "p99_ms": 0.03559998, + "mean_ms": 0.02557855, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.8005, + "qps_1t": 29515.7207, + "qps_1t_min": 28252.6579, + "qps_1t_max": 30075.6856, + "qps_1t_runs": [ + 29515.7207, + 30075.6856, + 28252.6579 + ], + "median_ms": 0.0335999648, + "p95_ms": 0.0416000257, + "p99_ms": 0.046000001, + "mean_ms": 0.0341748667, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.8922, + "qps_1t": 18121.6378, + "qps_1t_min": 18109.1839, + "qps_1t_max": 19142.2731, + "qps_1t_runs": [ + 18109.1839, + 19142.2731, + 18121.6378 + ], + "median_ms": 0.0537000014, + "p95_ms": 0.0656999764, + "p99_ms": 0.0714000198, + "mean_ms": 0.05421455, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.9563, + "qps_1t": 10819.8644, + "qps_1t_min": 10349.7169, + "qps_1t_max": 10930.0547, + "qps_1t_runs": [ + 10819.8644, + 10349.7169, + 10930.0547 + ], + "median_ms": 0.0922000036, + "p95_ms": 0.112999987, + "p99_ms": 0.132900022, + "mean_ms": 0.0935114834, + "knob": 64 + }, + { + "nq": 2000, + "recall": 0.9874, + "qps_1t": 6556.78372, + "qps_1t_min": 6235.10978, + "qps_1t_max": 6684.27974, + "qps_1t_runs": [ + 6684.27974, + 6235.10978, + 6556.78372 + ], + "median_ms": 0.155000016, + "p95_ms": 0.177900016, + "p99_ms": 0.206099998, + "mean_ms": 0.154166883, + "knob": 128 + }, + { + "nq": 2000, + "recall": 0.99845, + "qps_1t": 3471.99498, + "qps_1t_min": 3423.98094, + "qps_1t_max": 3594.77277, + "qps_1t_runs": [ + 3471.99498, + 3594.77277, + 3423.98094 + ], + "median_ms": 0.28810004, + "p95_ms": 0.332999974, + "p99_ms": 0.385999971, + "mean_ms": 0.286086083, + "knob": 256 + }, + { + "nq": 977, + "recall": 1, + "qps_1t": 329.317869, + "qps_1t_min": 323.319657, + "qps_1t_max": 329.820046, + "qps_1t_runs": [ + 323.319657, + 329.820046, + 329.317869 + ], + "median_ms": 3.0353, + "p95_ms": 3.33409995, + "p99_ms": 3.56660003, + "mean_ms": 3.05381685, + "knob": 400 + } + ], + "load_before_pct": 6.406250000000001, + "load_contaminated": false + }, + "cpp_default_exact|glove|1000000": { + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.5334006, + "index_s": 0, + "total_s": 0.5334006, + "first_query_ms": 281.1621, + "ready_s": 0.8145627 + }, + "mem_baseline": { + "rss_mb": 6.75, + "peak_mb": 6.75 + }, + "mem_after_build": { + "rss_mb": 678.277344, + "peak_mb": 817.109375 + }, + "mem_after_first_query": { + "rss_mb": 678.277344, + "peak_mb": 817.109375 + }, + "sweep": [ + { + "nq": 30, + "recall": 1, + "qps_1t": 2.47707451, + "qps_1t_min": 2.47707451, + "qps_1t_max": 2.47707451, + "qps_1t_runs": [ + 2.47707451 + ], + "median_ms": 406.4044, + "p95_ms": 414.879, + "p99_ms": 419.6363, + "mean_ms": 403.702027, + "knob": null + } + ], + "load_before_pct": 4.415625000000001, + "load_contaminated": false + }, + "cpp_default_ivf|glove|1000000": { + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.5334006, + "index_s": 8.7187386, + "total_s": 9.2521392, + "first_query_ms": 0, + "ready_s": 9.2521392 + }, + "mem_baseline": { + "rss_mb": 6.75, + "peak_mb": 6.75 + }, + "mem_after_build": { + "rss_mb": 1086.50391, + "peak_mb": 1147.94141 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.2955, + "qps_1t": 6470.72241, + "qps_1t_min": 6388.37623, + "qps_1t_max": 6694.39933, + "qps_1t_runs": [ + 6388.37623, + 6694.39933, + 6470.72241 + ], + "median_ms": 0.153200002, + "p95_ms": 0.198399997, + "p99_ms": 0.223100011, + "mean_ms": 0.15348505, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.41905, + "qps_1t": 5428.91702, + "qps_1t_min": 5367.33504, + "qps_1t_max": 5467.80937, + "qps_1t_runs": [ + 5367.33504, + 5428.91702, + 5467.80937 + ], + "median_ms": 0.182500051, + "p95_ms": 0.237400003, + "p99_ms": 0.259799999, + "mean_ms": 0.184466533, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.533, + "qps_1t": 4323.66619, + "qps_1t_min": 4300.91156, + "qps_1t_max": 4344.25436, + "qps_1t_runs": [ + 4300.91156, + 4344.25436, + 4323.66619 + ], + "median_ms": 0.228299992, + "p95_ms": 0.309300027, + "p99_ms": 0.34639996, + "mean_ms": 0.231327717, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.63225, + "qps_1t": 2984.86702, + "qps_1t_min": 2981.55995, + "qps_1t_max": 3000.01515, + "qps_1t_runs": [ + 2981.55995, + 3000.01515, + 2984.86702 + ], + "median_ms": 0.326899986, + "p95_ms": 0.459200004, + "p99_ms": 0.510800048, + "mean_ms": 0.334583283, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.72055, + "qps_1t": 1907.67083, + "qps_1t_min": 1818.11356, + "qps_1t_max": 1972.01183, + "qps_1t_runs": [ + 1972.01183, + 1907.67083, + 1818.11356 + ], + "median_ms": 0.514300016, + "p95_ms": 0.744500023, + "p99_ms": 0.84750005, + "mean_ms": 0.527105483, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.7965, + "qps_1t": 1085.6769, + "qps_1t_min": 1064.86707, + "qps_1t_max": 1085.74745, + "qps_1t_runs": [ + 1064.86707, + 1085.6769, + 1085.74745 + ], + "median_ms": 0.904699962, + "p95_ms": 1.28909998, + "p99_ms": 1.43200002, + "mean_ms": 0.9270644, + "knob": 32 + }, + { + "nq": 1797, + "recall": 0.86360601, + "qps_1t": 589.219043, + "qps_1t_min": 587.792598, + "qps_1t_max": 594.365455, + "qps_1t_runs": [ + 587.792598, + 589.219043, + 594.365455 + ], + "median_ms": 1.6747, + "p95_ms": 2.29819998, + "p99_ms": 2.51219998, + "mean_ms": 1.69363621, + "knob": 64 + }, + { + "nq": 924, + "recall": 0.920238095, + "qps_1t": 321.655711, + "qps_1t_min": 313.351118, + "qps_1t_max": 325.964363, + "qps_1t_runs": [ + 313.351118, + 321.655711, + 325.964363 + ], + "median_ms": 3.10430001, + "p95_ms": 4.07359999, + "p99_ms": 4.39140003, + "mean_ms": 3.12268081, + "knob": 128 + }, + { + "nq": 494, + "recall": 0.950809717, + "qps_1t": 171.933643, + "qps_1t_min": 167.389519, + "qps_1t_max": 174.846145, + "qps_1t_runs": [ + 174.846145, + 171.933643, + 167.389519 + ], + "median_ms": 5.80150005, + "p95_ms": 7.4308, + "p99_ms": 8.0184, + "mean_ms": 5.83653367, + "knob": 256 + } + ], + "mutation": { + "ref_knob": 256, + "recall_before": 0.950809717, + "insert_10k": { + "seconds": 0.1552916, + "per_s": 64394.9834 + }, + "update_10k": { + "seconds": 0.0161184, + "per_s": 620408.975 + }, + "delete_10k": { + "seconds": 0.0124532, + "per_s": 803006.456 + }, + "first_query_after_ms": 92.4923, + "recall_after": 0.950687623, + "after": { + "nq": 509, + "recall": 0.950687623, + "qps_1t": 170.170001, + "qps_1t_min": 168.283266, + "qps_1t_max": 173.872967, + "qps_1t_runs": [ + 168.283266, + 170.170001, + 173.872967 + ], + "median_ms": 5.83350001, + "p95_ms": 7.47169997, + "p99_ms": 7.98280002, + "mean_ms": 5.85672109 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 1000000, + "builds": 1 + }, + "mem_after": { + "rss_mb": 1102.64062, + "peak_mb": 1468.00781 + } + }, + "load_before_pct": 4.415625000000001, + "load_contaminated": false + }, + "cpp_default_exact|glove|100000": { + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.0382684001, + "index_s": 0, + "total_s": 0.0382684001, + "first_query_ms": 25.487, + "ready_s": 0.0637554001 + }, + "mem_baseline": { + "rss_mb": 6.7421875, + "peak_mb": 6.7421875 + }, + "mem_after_build": { + "rss_mb": 78.6054688, + "peak_mb": 90.7890625 + }, + "mem_after_first_query": { + "rss_mb": 78.6054688, + "peak_mb": 90.7890625 + }, + "sweep": [ + { + "nq": 123, + "recall": 1, + "qps_1t": 40.1078791, + "qps_1t_min": 39.6911589, + "qps_1t_max": 40.1278362, + "qps_1t_runs": [ + 40.1278362, + 40.1078791, + 39.6911589 + ], + "median_ms": 24.4744, + "p95_ms": 28.4097, + "p99_ms": 29.7178, + "mean_ms": 25.0158805, + "knob": null + } + ], + "load_before_pct": 3.4374999999999982, + "load_contaminated": false + }, + "cpp_default_ivf|glove|100000": { + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.0382684001, + "index_s": 0.2896739, + "total_s": 0.3279423, + "first_query_ms": 0, + "ready_s": 0.3279423 + }, + "mem_baseline": { + "rss_mb": 6.7421875, + "peak_mb": 6.7421875 + }, + "mem_after_build": { + "rss_mb": 118.519531, + "peak_mb": 138.285156 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.28485, + "qps_1t": 31629.5548, + "qps_1t_min": 24600.5788, + "qps_1t_max": 32671.3038, + "qps_1t_runs": [ + 24600.5788, + 31629.5548, + 32671.3038 + ], + "median_ms": 0.0317000085, + "p95_ms": 0.0468000071, + "p99_ms": 0.057999976, + "mean_ms": 0.0342911166, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.4074, + "qps_1t": 27774.6533, + "qps_1t_min": 27260.8068, + "qps_1t_max": 27919.1962, + "qps_1t_runs": [ + 27774.6533, + 27919.1962, + 27260.8068 + ], + "median_ms": 0.0347000314, + "p95_ms": 0.0441999873, + "p99_ms": 0.0598999904, + "mean_ms": 0.0361681333, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.5255, + "qps_1t": 22907.592, + "qps_1t_min": 22737.0911, + "qps_1t_max": 23047.3958, + "qps_1t_runs": [ + 22737.0911, + 22907.592, + 23047.3958 + ], + "median_ms": 0.041900028, + "p95_ms": 0.0548999524, + "p99_ms": 0.0744999852, + "mean_ms": 0.0436744999, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.63475, + "qps_1t": 16744.2623, + "qps_1t_min": 16489.6095, + "qps_1t_max": 17300.649, + "qps_1t_runs": [ + 17300.649, + 16744.2623, + 16489.6095 + ], + "median_ms": 0.0569000258, + "p95_ms": 0.0772000058, + "p99_ms": 0.0990999979, + "mean_ms": 0.0593891667, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.7357, + "qps_1t": 11814.7658, + "qps_1t_min": 11675.58, + "qps_1t_max": 12043.564, + "qps_1t_runs": [ + 12043.564, + 11814.7658, + 11675.58 + ], + "median_ms": 0.0818999833, + "p95_ms": 0.10970002, + "p99_ms": 0.136199989, + "mean_ms": 0.0844402, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.81855, + "qps_1t": 7115.70059, + "qps_1t_min": 6877.48922, + "qps_1t_max": 7441.4036, + "qps_1t_runs": [ + 6877.48922, + 7115.70059, + 7441.4036 + ], + "median_ms": 0.136399991, + "p95_ms": 0.187200028, + "p99_ms": 0.223700015, + "mean_ms": 0.140106483, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.88455, + "qps_1t": 4280.83566, + "qps_1t_min": 4217.9935, + "qps_1t_max": 4368.74382, + "qps_1t_runs": [ + 4217.9935, + 4368.74382, + 4280.83566 + ], + "median_ms": 0.227399985, + "p95_ms": 0.30449999, + "p99_ms": 0.352500007, + "mean_ms": 0.233192517, + "knob": 64 + }, + { + "nq": 2000, + "recall": 0.93865, + "qps_1t": 2360.84751, + "qps_1t_min": 2313.6529, + "qps_1t_max": 2396.98191, + "qps_1t_runs": [ + 2396.98191, + 2360.84751, + 2313.6529 + ], + "median_ms": 0.410900044, + "p95_ms": 0.546699972, + "p99_ms": 0.647700042, + "mean_ms": 0.424328317, + "knob": 128 + }, + { + "nq": 2000, + "recall": 0.9741, + "qps_1t": 1235.22005, + "qps_1t_min": 1233.54239, + "qps_1t_max": 1247.2061, + "qps_1t_runs": [ + 1247.2061, + 1233.54239, + 1235.22005 + ], + "median_ms": 0.78100001, + "p95_ms": 1.00029999, + "p99_ms": 1.20139995, + "mean_ms": 0.80734595, + "knob": 256 + }, + { + "nq": 116, + "recall": 1, + "qps_1t": 39.7482892, + "qps_1t_min": 39.6881103, + "qps_1t_max": 40.324651, + "qps_1t_runs": [ + 39.6881103, + 39.7482892, + 40.324651 + ], + "median_ms": 24.6437, + "p95_ms": 27.7731, + "p99_ms": 28.9565, + "mean_ms": 25.0511684, + "knob": 1265 + } + ], + "mutation": { + "ref_knob": 256, + "recall_before": 0.9741, + "insert_10k": { + "seconds": 0.0125325, + "per_s": 797925.392 + }, + "update_10k": { + "seconds": 0.00671849999, + "per_s": 1488427.48 + }, + "delete_10k": { + "seconds": 0.00396060001, + "per_s": 2524869.97 + }, + "first_query_after_ms": 22.0308, + "recall_after": 0.9725, + "after": { + "nq": 2000, + "recall": 0.9725, + "qps_1t": 1211.13131, + "qps_1t_min": 1133.34437, + "qps_1t_max": 1228.4226, + "qps_1t_runs": [ + 1211.13131, + 1133.34437, + 1228.4226 + ], + "median_ms": 0.818499946, + "p95_ms": 1.0772, + "p99_ms": 1.2539, + "mean_ms": 0.84069025 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 100000, + "builds": 1 + }, + "mem_after": { + "rss_mb": 137.675781, + "peak_mb": 160.726562 + } + }, + "saveload": { + "save_s": 4.4033457, + "disk_mb": 133.363812, + "load_s": 2.0282403, + "first_query_after_load_ms": 302.2448, + "time_to_ready_s": 2.3304851, + "note": "load = JSON parse + insert; first query builds the IVF index (never persisted)" + }, + "load_before_pct": 3.4374999999999982, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 0.3537934, + "mem": { + "rss_mb": 117.808594, + "peak_mb": 137.585938 + }, + "subsets": { + "f10": [ + { + "nq": 2000, + "recall": 0.26255, + "qps_1t": 33753.2805, + "qps_1t_min": 33367.8693, + "qps_1t_max": 34441.6662, + "qps_1t_runs": [ + 33367.8693, + 33753.2805, + 34441.6662 + ], + "median_ms": 0.0278000371, + "p95_ms": 0.0478999573, + "p99_ms": 0.0546999509, + "mean_ms": 0.0295434332, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.4765, + "qps_1t": 31251.758, + "qps_1t_min": 31157.2584, + "qps_1t_max": 31479.4557, + "qps_1t_runs": [ + 31157.2584, + 31251.758, + 31479.4557 + ], + "median_ms": 0.0318000093, + "p95_ms": 0.0360999838, + "p99_ms": 0.0399999553, + "mean_ms": 0.0319534, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.6976, + "qps_1t": 20020.8818, + "qps_1t_min": 19130.6459, + "qps_1t_max": 20067.4668, + "qps_1t_runs": [ + 20067.4668, + 19130.6459, + 20020.8818 + ], + "median_ms": 0.0501999748, + "p95_ms": 0.0591999851, + "p99_ms": 0.0674999901, + "mean_ms": 0.0506839668, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.79485, + "qps_1t": 14290.0013, + "qps_1t_min": 14008.2116, + "qps_1t_max": 14508.3238, + "qps_1t_runs": [ + 14008.2116, + 14508.3238, + 14290.0013 + ], + "median_ms": 0.0694000046, + "p95_ms": 0.0826000469, + "p99_ms": 0.0914999982, + "mean_ms": 0.0700972166, + "knob": 64 + }, + { + "nq": 476, + "recall": 1, + "qps_1t": 151.949656, + "qps_1t_min": 150.413969, + "qps_1t_max": 151.984651, + "qps_1t_runs": [ + 151.949656, + 150.413969, + 151.984651 + ], + "median_ms": 6.43150002, + "p95_ms": 8.29480001, + "p99_ms": 9.02339997, + "mean_ms": 6.60301919, + "knob": 1265 + }, + { + "nq": 462, + "recall": 1, + "qps_1t": 151.847621, + "qps_1t_min": 148.011494, + "qps_1t_max": 152.322437, + "qps_1t_runs": [ + 152.322437, + 148.011494, + 151.847621 + ], + "median_ms": 6.44499995, + "p95_ms": 8.38429999, + "p99_ms": 9.42820002, + "mean_ms": 6.63560079, + "knob": "exact" + } + ], + "f01": [ + { + "nq": 2000, + "recall": 0.3655, + "qps_1t": 15167.1764, + "qps_1t_min": 14900.6537, + "qps_1t_max": 15774.6621, + "qps_1t_runs": [ + 14900.6537, + 15774.6621, + 15167.1764 + ], + "median_ms": 0.0676999916, + "p95_ms": 0.0848000054, + "p99_ms": 0.100200006, + "mean_ms": 0.0654785999, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.3655, + "qps_1t": 21750.6204, + "qps_1t_min": 21094.799, + "qps_1t_max": 21798.175, + "qps_1t_runs": [ + 21094.799, + 21798.175, + 21750.6204 + ], + "median_ms": 0.0523999915, + "p95_ms": 0.0633999589, + "p99_ms": 0.0682999962, + "mean_ms": 0.0464187167, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.45215, + "qps_1t": 26973.481, + "qps_1t_min": 26932.7623, + "qps_1t_max": 27140.7247, + "qps_1t_runs": [ + 26932.7623, + 26973.481, + 27140.7247 + ], + "median_ms": 0.0369999907, + "p95_ms": 0.0421999721, + "p99_ms": 0.0462000025, + "mean_ms": 0.0370159833, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.59105, + "qps_1t": 19639.0538, + "qps_1t_min": 19081.1837, + "qps_1t_max": 20065.5945, + "qps_1t_runs": [ + 20065.5945, + 19639.0538, + 19081.1837 + ], + "median_ms": 0.0499999733, + "p95_ms": 0.0601999927, + "p99_ms": 0.0703000114, + "mean_ms": 0.0510543833, + "knob": 64 + }, + { + "nq": 880, + "recall": 1, + "qps_1t": 315.801086, + "qps_1t_min": 272.336549, + "qps_1t_max": 334.744217, + "qps_1t_runs": [ + 272.336549, + 315.801086, + 334.744217 + ], + "median_ms": 3.0573, + "p95_ms": 5.1674, + "p99_ms": 6.98509999, + "mean_ms": 3.27527769, + "knob": 1265 + }, + { + "nq": 1161, + "recall": 1, + "qps_1t": 312.327972, + "qps_1t_min": 309.935259, + "qps_1t_max": 322.174201, + "qps_1t_runs": [ + 312.327972, + 322.174201, + 309.935259 + ], + "median_ms": 3.02359997, + "p95_ms": 4.52809996, + "p99_ms": 6.52709999, + "mean_ms": 3.17738455, + "knob": "exact" + } + ] + } + } + }, + "cpp_default_exact|sift|1000000": { + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.6148045, + "index_s": 0, + "total_s": 0.6148045, + "first_query_ms": 329.5979, + "ready_s": 0.9444024 + }, + "mem_baseline": { + "rss_mb": 7.0859375, + "peak_mb": 7.0859375 + }, + "mem_after_build": { + "rss_mb": 790.34375, + "peak_mb": 929.175781 + }, + "mem_after_first_query": { + "rss_mb": 790.34375, + "peak_mb": 929.175781 + }, + "sweep": [ + { + "nq": 30, + "recall": 1, + "qps_1t": 2.18706115, + "qps_1t_min": 2.18706115, + "qps_1t_max": 2.18706115, + "qps_1t_runs": [ + 2.18706115 + ], + "median_ms": 458.9676, + "p95_ms": 471.8328, + "p99_ms": 473.9323, + "mean_ms": 457.234587, + "knob": null + } + ], + "load_before_pct": 11.962499999999993, + "load_contaminated": false + }, + "cpp_default_ivf|sift|1000000": { + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.6148045, + "index_s": 10.9004137, + "total_s": 11.5152182, + "first_query_ms": 0, + "ready_s": 11.5152182 + }, + "mem_baseline": { + "rss_mb": 7.0859375, + "peak_mb": 7.0859375 + }, + "mem_after_build": { + "rss_mb": 1305.70312, + "peak_mb": 1384.32812 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.25485, + "qps_1t": 5716.30765, + "qps_1t_min": 5605.84466, + "qps_1t_max": 5905.34209, + "qps_1t_runs": [ + 5605.84466, + 5905.34209, + 5716.30765 + ], + "median_ms": 0.174399989, + "p95_ms": 0.222200004, + "p99_ms": 0.276899955, + "mean_ms": 0.174220517, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.38895, + "qps_1t": 4863.64882, + "qps_1t_min": 4720.03914, + "qps_1t_max": 4876.94849, + "qps_1t_runs": [ + 4876.94849, + 4720.03914, + 4863.64882 + ], + "median_ms": 0.204399985, + "p95_ms": 0.25859999, + "p99_ms": 0.300999964, + "mean_ms": 0.207505283, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.5391, + "qps_1t": 3920.99816, + "qps_1t_min": 3790.9338, + "qps_1t_max": 3942.3883, + "qps_1t_runs": [ + 3942.3883, + 3920.99816, + 3790.9338 + ], + "median_ms": 0.251100049, + "p95_ms": 0.343799999, + "p99_ms": 0.416600029, + "mean_ms": 0.257492567, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.6881, + "qps_1t": 2732.08181, + "qps_1t_min": 2600.88068, + "qps_1t_max": 2734.66914, + "qps_1t_runs": [ + 2734.66914, + 2732.08181, + 2600.88068 + ], + "median_ms": 0.356399978, + "p95_ms": 0.525599986, + "p99_ms": 0.640000042, + "mean_ms": 0.37206045, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.8212, + "qps_1t": 1685.54647, + "qps_1t_min": 1660.67385, + "qps_1t_max": 1687.27875, + "qps_1t_runs": [ + 1660.67385, + 1687.27875, + 1685.54647 + ], + "median_ms": 0.5657, + "p95_ms": 0.879300002, + "p99_ms": 1.04960002, + "mean_ms": 0.5960383, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.9147, + "qps_1t": 979.311554, + "qps_1t_min": 943.580269, + "qps_1t_max": 988.607436, + "qps_1t_runs": [ + 988.607436, + 979.311554, + 943.580269 + ], + "median_ms": 0.984299986, + "p95_ms": 1.51490001, + "p99_ms": 1.7423, + "mean_ms": 1.0308142, + "knob": 32 + }, + { + "nq": 1522, + "recall": 0.969973719, + "qps_1t": 562.161819, + "qps_1t_min": 525.069372, + "qps_1t_max": 579.249538, + "qps_1t_runs": [ + 525.069372, + 562.161819, + 579.249538 + ], + "median_ms": 1.74880004, + "p95_ms": 2.54020002, + "p99_ms": 2.8281, + "mean_ms": 1.80324306, + "knob": 64 + }, + { + "nq": 870, + "recall": 0.99183908, + "qps_1t": 297.31564, + "qps_1t_min": 291.993145, + "qps_1t_max": 301.956522, + "qps_1t_runs": [ + 301.956522, + 297.31564, + 291.993145 + ], + "median_ms": 3.30139999, + "p95_ms": 4.47340001, + "p99_ms": 4.99910006, + "mean_ms": 3.36663395, + "knob": 128 + }, + { + "nq": 411, + "recall": 0.997810219, + "qps_1t": 155.294277, + "qps_1t_min": 153.956001, + "qps_1t_max": 174.005036, + "qps_1t_runs": [ + 153.956001, + 155.294277, + 174.005036 + ], + "median_ms": 6.18129998, + "p95_ms": 7.79280002, + "p99_ms": 8.3553, + "mean_ms": 6.22723658, + "knob": 256 + } + ], + "load_before_pct": 11.962499999999993, + "load_contaminated": false + }, + "cpp_default_exact|sift|100000": { + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.0495777, + "index_s": 0, + "total_s": 0.0495777, + "first_query_ms": 31.582, + "ready_s": 0.0811597 + }, + "mem_baseline": { + "rss_mb": 7.12890625, + "peak_mb": 7.12890625 + }, + "mem_after_build": { + "rss_mb": 91.2265625, + "peak_mb": 109.503906 + }, + "mem_after_first_query": { + "rss_mb": 91.2265625, + "peak_mb": 109.503906 + }, + "sweep": [ + { + "nq": 92, + "recall": 1, + "qps_1t": 22.7960828, + "qps_1t_min": 22.4102038, + "qps_1t_max": 25.158072, + "qps_1t_runs": [ + 25.158072, + 22.4102038, + 22.7960828 + ], + "median_ms": 44.2155, + "p95_ms": 47.4825, + "p99_ms": 49.4587, + "mean_ms": 42.7461301, + "knob": null + } + ], + "load_before_pct": 13.984374999999982, + "load_contaminated": false + }, + "cpp_default_ivf|sift|100000": { + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.0495777, + "index_s": 0.9699913, + "total_s": 1.019569, + "first_query_ms": 0, + "ready_s": 1.019569 + }, + "mem_baseline": { + "rss_mb": 7.12890625, + "peak_mb": 7.12890625 + }, + "mem_after_build": { + "rss_mb": 141.617188, + "peak_mb": 166.804688 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.27955, + "qps_1t": 19362.4338, + "qps_1t_min": 18832.906, + "qps_1t_max": 19708.0838, + "qps_1t_runs": [ + 19362.4338, + 19708.0838, + 18832.906 + ], + "median_ms": 0.0520999893, + "p95_ms": 0.064599968, + "p99_ms": 0.0718000229, + "mean_ms": 0.0518285167, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.42195, + "qps_1t": 16480.071, + "qps_1t_min": 16362.729, + "qps_1t_max": 16797.3077, + "qps_1t_runs": [ + 16797.3077, + 16362.729, + 16480.071 + ], + "median_ms": 0.0592999859, + "p95_ms": 0.0746999867, + "p99_ms": 0.0862000161, + "mean_ms": 0.0604424, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.5793, + "qps_1t": 13532.697, + "qps_1t_min": 13443.1007, + "qps_1t_max": 13864.0972, + "qps_1t_runs": [ + 13443.1007, + 13532.697, + 13864.0972 + ], + "median_ms": 0.0711999601, + "p95_ms": 0.0971000409, + "p99_ms": 0.11839997, + "mean_ms": 0.0734704834, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.7401, + "qps_1t": 10562.7015, + "qps_1t_min": 10319.2891, + "qps_1t_max": 10608.515, + "qps_1t_runs": [ + 10608.515, + 10319.2891, + 10562.7015 + ], + "median_ms": 0.0921000028, + "p95_ms": 0.129299995, + "p99_ms": 0.153700006, + "mean_ms": 0.09528085, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.86955, + "qps_1t": 6672.37154, + "qps_1t_min": 6639.22894, + "qps_1t_max": 6802.39485, + "qps_1t_runs": [ + 6802.39485, + 6639.22894, + 6672.37154 + ], + "median_ms": 0.144199992, + "p95_ms": 0.203599979, + "p99_ms": 0.242499984, + "mean_ms": 0.149166233, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.95215, + "qps_1t": 4053.93107, + "qps_1t_min": 3966.45725, + "qps_1t_max": 4059.59156, + "qps_1t_runs": [ + 4059.59156, + 4053.93107, + 3966.45725 + ], + "median_ms": 0.241100031, + "p95_ms": 0.336199999, + "p99_ms": 0.392999966, + "mean_ms": 0.248372833, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.98825, + "qps_1t": 2290.30143, + "qps_1t_min": 2201.50241, + "qps_1t_max": 2307.09699, + "qps_1t_runs": [ + 2307.09699, + 2201.50241, + 2290.30143 + ], + "median_ms": 0.429699954, + "p95_ms": 0.605000008, + "p99_ms": 0.738800038, + "mean_ms": 0.441434717, + "knob": 64 + }, + { + "nq": 2000, + "recall": 0.9984, + "qps_1t": 1321.13643, + "qps_1t_min": 1304.80848, + "qps_1t_max": 1338.60333, + "qps_1t_runs": [ + 1304.80848, + 1321.13643, + 1338.60333 + ], + "median_ms": 0.736700022, + "p95_ms": 1.01829995, + "p99_ms": 1.21630001, + "mean_ms": 0.756789133, + "knob": 128 + }, + { + "nq": 2000, + "recall": 0.9999, + "qps_1t": 678.918751, + "qps_1t_min": 677.351434, + "qps_1t_max": 687.975468, + "qps_1t_runs": [ + 678.918751, + 687.975468, + 677.351434 + ], + "median_ms": 1.45109999, + "p95_ms": 1.84019998, + "p99_ms": 2.0563, + "mean_ms": 1.467603, + "knob": 256 + }, + { + "nq": 66, + "recall": 1, + "qps_1t": 22.339829, + "qps_1t_min": 22.2738571, + "qps_1t_max": 23.5006731, + "qps_1t_runs": [ + 22.339829, + 22.2738571, + 23.5006731 + ], + "median_ms": 44.7927, + "p95_ms": 47.5427, + "p99_ms": 49.3286, + "mean_ms": 44.0702515, + "knob": 1265 + } + ], + "mutation": { + "ref_knob": 32, + "recall_before": 0.95215, + "insert_10k": { + "seconds": 0.0249564, + "per_s": 400698.819 + }, + "update_10k": { + "seconds": 0.0119626001, + "per_s": 835938.672 + }, + "delete_10k": { + "seconds": 0.007278, + "per_s": 1374003.85 + }, + "first_query_after_ms": 37.1066, + "recall_after": 0.949, + "after": { + "nq": 2000, + "recall": 0.949, + "qps_1t": 4125.9768, + "qps_1t_min": 4119.45514, + "qps_1t_max": 4230.51036, + "qps_1t_runs": [ + 4230.51036, + 4119.45514, + 4125.9768 + ], + "median_ms": 0.233200029, + "p95_ms": 0.327699992, + "p99_ms": 0.402999984, + "mean_ms": 0.2404985 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 100000, + "builds": 1 + }, + "mem_after": { + "rss_mb": 163.992188, + "peak_mb": 194.105469 + } + }, + "saveload": { + "save_s": 12.4136456, + "disk_mb": 133.678411, + "load_s": 4.9743937, + "first_query_after_load_ms": 826.4113, + "time_to_ready_s": 5.800805, + "note": "load = JSON parse + insert; first query builds the IVF index (never persisted)" + }, + "load_before_pct": 13.984374999999982, + "load_contaminated": false + }, + "cpp_default_exact|text384|1000000": { + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.9167631, + "index_s": 0, + "total_s": 0.9167631, + "first_query_ms": 508.9527, + "ready_s": 1.4257158 + }, + "mem_baseline": { + "rss_mb": 11.0585938, + "peak_mb": 11.0585938 + }, + "mem_after_build": { + "rss_mb": 1794.48047, + "peak_mb": 1933.3125 + }, + "mem_after_first_query": { + "rss_mb": 1794.48047, + "peak_mb": 1933.3125 + }, + "sweep": [ + { + "nq": 30, + "recall": 1, + "qps_1t": 2.1147531, + "qps_1t_min": 2.1147531, + "qps_1t_max": 2.1147531, + "qps_1t_runs": [ + 2.1147531 + ], + "median_ms": 469.0725, + "p95_ms": 499.9009, + "p99_ms": 516.0898, + "mean_ms": 472.86844, + "knob": null + } + ], + "load_before_pct": 5.6687500000000055, + "load_contaminated": false + }, + "cpp_default_ivf|text384|1000000": { + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.9167631, + "index_s": 11.4398135, + "total_s": 12.3565766, + "first_query_ms": 0, + "ready_s": 12.3565766 + }, + "mem_baseline": { + "rss_mb": 11.0585938, + "peak_mb": 11.0585938 + }, + "mem_after_build": { + "rss_mb": 3290.40234, + "peak_mb": 3525.29688 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.2669, + "qps_1t": 4176.96637, + "qps_1t_min": 3634.68904, + "qps_1t_max": 4193.69151, + "qps_1t_runs": [ + 3634.68904, + 4176.96637, + 4193.69151 + ], + "median_ms": 0.238600012, + "p95_ms": 0.333599979, + "p99_ms": 0.549699995, + "mean_ms": 0.2509961, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.3726, + "qps_1t": 3308.03787, + "qps_1t_min": 3221.80466, + "qps_1t_max": 3336.83646, + "qps_1t_runs": [ + 3221.80466, + 3308.03787, + 3336.83646 + ], + "median_ms": 0.297199993, + "p95_ms": 0.398600008, + "p99_ms": 0.456799986, + "mean_ms": 0.304121367, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.4897, + "qps_1t": 2387.27326, + "qps_1t_min": 2353.06464, + "qps_1t_max": 2421.47519, + "qps_1t_runs": [ + 2421.47519, + 2353.06464, + 2387.27326 + ], + "median_ms": 0.409199973, + "p95_ms": 0.556299987, + "p99_ms": 0.667600019, + "mean_ms": 0.418945683, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.6026, + "qps_1t": 1588.828, + "qps_1t_min": 1576.05543, + "qps_1t_max": 1601.92244, + "qps_1t_runs": [ + 1601.92244, + 1588.828, + 1576.05543 + ], + "median_ms": 0.61280001, + "p95_ms": 0.819000008, + "p99_ms": 0.953999988, + "mean_ms": 0.62938005, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.701, + "qps_1t": 960.889538, + "qps_1t_min": 953.515454, + "qps_1t_max": 984.54743, + "qps_1t_runs": [ + 984.54743, + 960.889538, + 953.515454 + ], + "median_ms": 1.02059997, + "p95_ms": 1.32340001, + "p99_ms": 1.49370002, + "mean_ms": 1.03504938, + "knob": 16 + }, + { + "nq": 1627, + "recall": 0.78494161, + "qps_1t": 558.61326, + "qps_1t_min": 554.357325, + "qps_1t_max": 559.146092, + "qps_1t_runs": [ + 554.357325, + 559.146092, + 558.61326 + ], + "median_ms": 1.77579996, + "p95_ms": 2.2098, + "p99_ms": 2.50400003, + "mean_ms": 1.79415976, + "knob": 32 + }, + { + "nq": 969, + "recall": 0.86130031, + "qps_1t": 311.111696, + "qps_1t_min": 302.544644, + "qps_1t_max": 312.582914, + "qps_1t_runs": [ + 302.544644, + 311.111696, + 312.582914 + ], + "median_ms": 3.15619999, + "p95_ms": 3.87209997, + "p99_ms": 4.28369996, + "mean_ms": 3.23957606, + "knob": 64 + }, + { + "nq": 499, + "recall": 0.915230461, + "qps_1t": 173.003759, + "qps_1t_min": 170.55657, + "qps_1t_max": 173.031546, + "qps_1t_runs": [ + 173.031546, + 173.003759, + 170.55657 + ], + "median_ms": 5.78070001, + "p95_ms": 6.77370001, + "p99_ms": 7.47860002, + "mean_ms": 5.80755718, + "knob": 128 + }, + { + "nq": 286, + "recall": 0.952797203, + "qps_1t": 89.5362156, + "qps_1t_min": 89.4603815, + "qps_1t_max": 90.4557111, + "qps_1t_runs": [ + 89.5362156, + 89.4603815, + 90.4557111 + ], + "median_ms": 11.051, + "p95_ms": 12.9159, + "p99_ms": 14.1046, + "mean_ms": 11.1339772, + "knob": 256 + } + ], + "mutation": { + "ref_knob": 256, + "recall_before": 0.952797203, + "insert_10k": { + "seconds": 0.2923063, + "per_s": 34210.6893 + }, + "update_10k": { + "seconds": 0.0175668, + "per_s": 569255.641 + }, + "delete_10k": { + "seconds": 0.00725369999, + "per_s": 1378606.78 + }, + "first_query_after_ms": 149.3498, + "recall_after": 0.952895753, + "after": { + "nq": 259, + "recall": 0.952895753, + "qps_1t": 90.246679, + "qps_1t_min": 88.8601914, + "qps_1t_max": 90.7865402, + "qps_1t_runs": [ + 88.8601914, + 90.7865402, + 90.246679 + ], + "median_ms": 11.0211, + "p95_ms": 12.958, + "p99_ms": 13.6072999, + "mean_ms": 11.1164073 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 1000000, + "builds": 1 + }, + "mem_after": { + "rss_mb": 3338.33594, + "peak_mb": 4755.44922 + } + }, + "load_before_pct": 5.6687500000000055, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 12.6140878, + "mem": { + "rss_mb": 3289.16016, + "peak_mb": 3524.01172 + }, + "subsets": { + "f10": [ + { + "nq": 2000, + "recall": 0.31395, + "qps_1t": 4559.66828, + "qps_1t_min": 3495.87705, + "qps_1t_max": 4560.78396, + "qps_1t_runs": [ + 3495.87705, + 4559.66828, + 4560.78396 + ], + "median_ms": 0.221399998, + "p95_ms": 0.370400026, + "p99_ms": 0.655800046, + "mean_ms": 0.241542, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.5416, + "qps_1t": 2946.32648, + "qps_1t_min": 2618.97924, + "qps_1t_max": 3099.19676, + "qps_1t_runs": [ + 3099.19676, + 2618.97924, + 2946.32648 + ], + "median_ms": 0.32500003, + "p95_ms": 0.500399969, + "p99_ms": 0.764900004, + "mean_ms": 0.347966033, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.648, + "qps_1t": 2171.86229, + "qps_1t_min": 2166.75193, + "qps_1t_max": 2376.16159, + "qps_1t_runs": [ + 2166.75193, + 2171.86229, + 2376.16159 + ], + "median_ms": 0.426300045, + "p95_ms": 0.613900018, + "p99_ms": 0.782200019, + "mean_ms": 0.447600483, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.8244, + "qps_1t": 961.102361, + "qps_1t_min": 957.682225, + "qps_1t_max": 964.846261, + "qps_1t_runs": [ + 961.102361, + 957.682225, + 964.846261 + ], + "median_ms": 1.01569999, + "p95_ms": 1.33850001, + "p99_ms": 1.54590001, + "mean_ms": 1.04036472, + "knob": 64 + }, + { + "nq": 929, + "recall": 0.942518837, + "qps_1t": 307.878202, + "qps_1t_min": 306.789746, + "qps_1t_max": 310.602782, + "qps_1t_runs": [ + 310.602782, + 307.878202, + 306.789746 + ], + "median_ms": 3.20199999, + "p95_ms": 3.86150001, + "p99_ms": 4.32079996, + "mean_ms": 3.24238174, + "knob": 256 + }, + { + "nq": 30, + "recall": 1, + "qps_1t": 7.46364682, + "qps_1t_min": 7.45864591, + "qps_1t_max": 7.49316773, + "qps_1t_runs": [ + 7.46364682, + 7.49316773, + 7.45864591 + ], + "median_ms": 132.9719, + "p95_ms": 142.3555, + "p99_ms": 149.1464, + "mean_ms": 133.836753, + "knob": "exact" + } + ], + "f01": [ + { + "nq": 2000, + "recall": 0.32855, + "qps_1t": 2516.98746, + "qps_1t_min": 2503.04057, + "qps_1t_max": 2540.33867, + "qps_1t_runs": [ + 2503.04057, + 2516.98746, + 2540.33867 + ], + "median_ms": 0.432599976, + "p95_ms": 0.510699989, + "p99_ms": 0.589700008, + "mean_ms": 0.396820917, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.3727, + "qps_1t": 3646.24031, + "qps_1t_min": 3606.5857, + "qps_1t_max": 3648.82271, + "qps_1t_runs": [ + 3606.5857, + 3648.82271, + 3646.24031 + ], + "median_ms": 0.267299998, + "p95_ms": 0.330500014, + "p99_ms": 0.429000007, + "mean_ms": 0.27519555, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.49245, + "qps_1t": 2901.21395, + "qps_1t_min": 2899.20326, + "qps_1t_max": 2929.89851, + "qps_1t_runs": [ + 2899.20326, + 2929.89851, + 2901.21395 + ], + "median_ms": 0.335500052, + "p95_ms": 0.425299979, + "p99_ms": 0.496099994, + "mean_ms": 0.343638133, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.7289, + "qps_1t": 1428.29087, + "qps_1t_min": 1427.75771, + "qps_1t_max": 1445.06447, + "qps_1t_runs": [ + 1428.29087, + 1445.06447, + 1427.75771 + ], + "median_ms": 0.675400021, + "p95_ms": 0.912100018, + "p99_ms": 1.0545, + "mean_ms": 0.6975157, + "knob": 64 + }, + { + "nq": 1513, + "recall": 0.90548579, + "qps_1t": 505.774234, + "qps_1t_min": 493.870651, + "qps_1t_max": 507.859623, + "qps_1t_runs": [ + 493.870651, + 507.859623, + 505.774234 + ], + "median_ms": 1.9569, + "p95_ms": 2.4612, + "p99_ms": 2.81969999, + "mean_ms": 1.99034549, + "knob": 256 + }, + { + "nq": 34, + "recall": 1, + "qps_1t": 11.2739486, + "qps_1t_min": 11.2538943, + "qps_1t_max": 11.305939, + "qps_1t_runs": [ + 11.2739486, + 11.305939, + 11.2538943 + ], + "median_ms": 87.8238, + "p95_ms": 91.8591, + "p99_ms": 105.5235, + "mean_ms": 88.6690951, + "knob": "exact" + } + ] + } + } + }, + "cpp_default_exact|text384|100000": { + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.0772106, + "index_s": 0, + "total_s": 0.0772106, + "first_query_ms": 44.9507, + "ready_s": 0.1221613 + }, + "mem_baseline": { + "rss_mb": 11.0585938, + "peak_mb": 11.0585938 + }, + "mem_after_build": { + "rss_mb": 203.3125, + "peak_mb": 217.085938 + }, + "mem_after_first_query": { + "rss_mb": 203.3125, + "peak_mb": 217.085938 + }, + "sweep": [ + { + "nq": 66, + "recall": 1, + "qps_1t": 22.6002529, + "qps_1t_min": 22.4759446, + "qps_1t_max": 23.2881655, + "qps_1t_runs": [ + 22.4759446, + 22.6002529, + 23.2881655 + ], + "median_ms": 44.1773, + "p95_ms": 47.1510999, + "p99_ms": 57.6838, + "mean_ms": 43.8931899, + "knob": null + } + ], + "load_before_pct": 6.487499999999999, + "load_contaminated": false + }, + "cpp_default_ivf|text384|100000": { + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.0772106, + "index_s": 0.891219, + "total_s": 0.9684296, + "first_query_ms": 0, + "ready_s": 0.9684296 + }, + "mem_baseline": { + "rss_mb": 11.0585938, + "peak_mb": 11.0585938 + }, + "mem_after_build": { + "rss_mb": 353.148438, + "peak_mb": 427.757812 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.28085, + "qps_1t": 16275.7636, + "qps_1t_min": 16013.5923, + "qps_1t_max": 17282.216, + "qps_1t_runs": [ + 16013.5923, + 16275.7636, + 17282.216 + ], + "median_ms": 0.0590000418, + "p95_ms": 0.0765000004, + "p99_ms": 0.0903000473, + "mean_ms": 0.06058365, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.40215, + "qps_1t": 14590.3504, + "qps_1t_min": 14423.6039, + "qps_1t_max": 14658.4687, + "qps_1t_runs": [ + 14423.6039, + 14590.3504, + 14658.4687 + ], + "median_ms": 0.0665000407, + "p95_ms": 0.0887000351, + "p99_ms": 0.109800021, + "mean_ms": 0.0686964, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.5291, + "qps_1t": 10815.4644, + "qps_1t_min": 10090.7308, + "qps_1t_max": 10997.6097, + "qps_1t_runs": [ + 10997.6097, + 10090.7308, + 10815.4644 + ], + "median_ms": 0.0913999975, + "p95_ms": 0.127099978, + "p99_ms": 0.151499989, + "mean_ms": 0.0941633, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.6479, + "qps_1t": 6398.64861, + "qps_1t_min": 6091.14299, + "qps_1t_max": 6901.58751, + "qps_1t_runs": [ + 6901.58751, + 6398.64861, + 6091.14299 + ], + "median_ms": 0.151699991, + "p95_ms": 0.206299999, + "p99_ms": 0.246800017, + "mean_ms": 0.155116667, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.75475, + "qps_1t": 3908.01471, + "qps_1t_min": 3831.05801, + "qps_1t_max": 3928.3917, + "qps_1t_runs": [ + 3908.01471, + 3928.3917, + 3831.05801 + ], + "median_ms": 0.249500037, + "p95_ms": 0.335699995, + "p99_ms": 0.409299973, + "mean_ms": 0.257155333, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.84275, + "qps_1t": 2254.86083, + "qps_1t_min": 2199.54419, + "qps_1t_max": 2286.80228, + "qps_1t_runs": [ + 2199.54419, + 2286.80228, + 2254.86083 + ], + "median_ms": 0.4357, + "p95_ms": 0.569800031, + "p99_ms": 0.649199996, + "mean_ms": 0.445139283, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.90905, + "qps_1t": 1232.11015, + "qps_1t_min": 1224.88563, + "qps_1t_max": 1233.79411, + "qps_1t_runs": [ + 1224.88563, + 1233.79411, + 1232.11015 + ], + "median_ms": 0.805900025, + "p95_ms": 0.992999994, + "p99_ms": 1.10200001, + "mean_ms": 0.812842167, + "knob": 64 + }, + { + "nq": 1869, + "recall": 0.954200107, + "qps_1t": 641.321552, + "qps_1t_min": 638.691566, + "qps_1t_max": 645.133053, + "qps_1t_runs": [ + 641.321552, + 645.133053, + 638.691566 + ], + "median_ms": 1.54650002, + "p95_ms": 1.812, + "p99_ms": 2.00480002, + "mean_ms": 1.55834967, + "knob": 128 + }, + { + "nq": 1016, + "recall": 0.981102362, + "qps_1t": 331.200059, + "qps_1t_min": 329.795016, + "qps_1t_max": 331.934005, + "qps_1t_runs": [ + 329.795016, + 331.200059, + 331.934005 + ], + "median_ms": 3.00600001, + "p95_ms": 3.38459999, + "p99_ms": 3.66450002, + "mean_ms": 3.02138556, + "knob": 256 + }, + { + "nq": 67, + "recall": 1, + "qps_1t": 22.478133, + "qps_1t_min": 22.3918544, + "qps_1t_max": 22.6263751, + "qps_1t_runs": [ + 22.3918544, + 22.478133, + 22.6263751 + ], + "median_ms": 44.5362, + "p95_ms": 46.7511, + "p99_ms": 48.0529999, + "mean_ms": 44.4476622, + "knob": 1265 + } + ], + "mutation": { + "ref_knob": 128, + "recall_before": 0.954200107, + "insert_10k": { + "seconds": 0.03772, + "per_s": 265111.347 + }, + "update_10k": { + "seconds": 0.0130817, + "per_s": 764426.641 + }, + "delete_10k": { + "seconds": 0.00429279997, + "per_s": 2329481.94 + }, + "first_query_after_ms": 64.303, + "recall_after": 0.950125628, + "after": { + "nq": 1592, + "recall": 0.950125628, + "qps_1t": 640.604009, + "qps_1t_min": 635.628587, + "qps_1t_max": 644.542633, + "qps_1t_runs": [ + 644.542633, + 640.604009, + 635.628587 + ], + "median_ms": 1.54610001, + "p95_ms": 1.84119999, + "p99_ms": 2.0489, + "mean_ms": 1.5619201 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 100000, + "builds": 1 + }, + "mem_after": { + "rss_mb": 419.304688, + "peak_mb": 503.359375 + } + }, + "saveload": { + "save_s": 16.5449248, + "disk_mb": 499.571589, + "load_s": 7.3371524, + "first_query_after_load_ms": 902.5092, + "time_to_ready_s": 8.2396616, + "note": "load = JSON parse + insert; first query builds the IVF index (never persisted)" + }, + "load_before_pct": 6.487499999999999, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 0.9856867, + "mem": { + "rss_mb": 352.316406, + "peak_mb": 426.910156 + }, + "subsets": { + "f10": [ + { + "nq": 2000, + "recall": 0.30525, + "qps_1t": 15841.3207, + "qps_1t_min": 15522.1175, + "qps_1t_max": 16119.8283, + "qps_1t_runs": [ + 15522.1175, + 16119.8283, + 15841.3207 + ], + "median_ms": 0.0581999775, + "p95_ms": 0.108399952, + "p99_ms": 0.121500052, + "mean_ms": 0.0631952167, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.5293, + "qps_1t": 14227.4731, + "qps_1t_min": 13453.5724, + "qps_1t_max": 14282.4905, + "qps_1t_runs": [ + 14227.4731, + 14282.4905, + 13453.5724 + ], + "median_ms": 0.0703000114, + "p95_ms": 0.0849000062, + "p99_ms": 0.0939000165, + "mean_ms": 0.0715440168, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.7753, + "qps_1t": 9219.89967, + "qps_1t_min": 8565.50125, + "qps_1t_max": 9457.60628, + "qps_1t_runs": [ + 8565.50125, + 9457.60628, + 9219.89967 + ], + "median_ms": 0.108100008, + "p95_ms": 0.135900045, + "p99_ms": 0.151000044, + "mean_ms": 0.110314483, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.86635, + "qps_1t": 6586.23772, + "qps_1t_min": 6576.98726, + "qps_1t_max": 6670.57785, + "qps_1t_runs": [ + 6670.57785, + 6586.23772, + 6576.98726 + ], + "median_ms": 0.149299973, + "p95_ms": 0.179300027, + "p99_ms": 0.194899971, + "mean_ms": 0.151263033, + "knob": 64 + }, + { + "nq": 350, + "recall": 1, + "qps_1t": 112.936181, + "qps_1t_min": 111.427505, + "qps_1t_max": 118.53174, + "qps_1t_runs": [ + 112.936181, + 118.53174, + 111.427505 + ], + "median_ms": 8.48670001, + "p95_ms": 10.6079, + "p99_ms": 11.4348, + "mean_ms": 8.75518733, + "knob": 1265 + }, + { + "nq": 369, + "recall": 1, + "qps_1t": 116.163134, + "qps_1t_min": 114.132039, + "qps_1t_max": 118.580394, + "qps_1t_runs": [ + 118.580394, + 116.163134, + 114.132039 + ], + "median_ms": 8.39119998, + "p95_ms": 10.2264, + "p99_ms": 11.0627, + "mean_ms": 8.60115402, + "knob": "exact" + } + ], + "f01": [ + { + "nq": 2000, + "recall": 0.4965, + "qps_1t": 6217.71258, + "qps_1t_min": 6114.36242, + "qps_1t_max": 6251.55312, + "qps_1t_runs": [ + 6251.55312, + 6114.36242, + 6217.71258 + ], + "median_ms": 0.169800012, + "p95_ms": 0.203900039, + "p99_ms": 0.233199971, + "mean_ms": 0.161446817, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.49675, + "qps_1t": 8560.59942, + "qps_1t_min": 8502.15424, + "qps_1t_max": 8666.37115, + "qps_1t_runs": [ + 8502.15424, + 8560.59942, + 8666.37115 + ], + "median_ms": 0.128499989, + "p95_ms": 0.15659997, + "p99_ms": 0.177400012, + "mean_ms": 0.116606683, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.5704, + "qps_1t": 11771.478, + "qps_1t_min": 11758.1144, + "qps_1t_max": 11802.8708, + "qps_1t_runs": [ + 11758.1144, + 11771.478, + 11802.8708 + ], + "median_ms": 0.082899991, + "p95_ms": 0.101500016, + "p99_ms": 0.115200004, + "mean_ms": 0.0849079665, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.7197, + "qps_1t": 9267.99948, + "qps_1t_min": 9224.47105, + "qps_1t_max": 9271.76754, + "qps_1t_runs": [ + 9267.99948, + 9271.76754, + 9224.47105 + ], + "median_ms": 0.106099993, + "p95_ms": 0.128099986, + "p99_ms": 0.140500022, + "mean_ms": 0.10805325, + "knob": 64 + }, + { + "nq": 775, + "recall": 1, + "qps_1t": 320.493018, + "qps_1t_min": 306.2757, + "qps_1t_max": 332.351332, + "qps_1t_runs": [ + 306.2757, + 320.493018, + 332.351332 + ], + "median_ms": 2.84999999, + "p95_ms": 5.02549997, + "p99_ms": 6.95160002, + "mean_ms": 3.13136301, + "knob": 1265 + }, + { + "nq": 1084, + "recall": 1, + "qps_1t": 345.496103, + "qps_1t_min": 339.178934, + "qps_1t_max": 364.635055, + "qps_1t_runs": [ + 364.635055, + 339.178934, + 345.496103 + ], + "median_ms": 2.65839999, + "p95_ms": 4.19960002, + "p99_ms": 5.83909999, + "mean_ms": 2.86171768, + "knob": "exact" + } + ] + } + } + }, + "cpp_default_exact|text384|10000": { + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.00753950002, + "index_s": 0, + "total_s": 0.00753950002, + "first_query_ms": 4.08300001, + "ready_s": 0.0116225 + }, + "mem_baseline": { + "rss_mb": 11.0507812, + "peak_mb": 11.0507812 + }, + "mem_after_build": { + "rss_mb": 32.7460938, + "peak_mb": 44.1601562 + }, + "mem_after_first_query": { + "rss_mb": 32.7460938, + "peak_mb": 44.1601562 + }, + "sweep": [ + { + "nq": 952, + "recall": 1, + "qps_1t": 310.7407, + "qps_1t_min": 288.55097, + "qps_1t_max": 311.217139, + "qps_1t_runs": [ + 310.7407, + 311.217139, + 288.55097 + ], + "median_ms": 3.22829996, + "p95_ms": 3.62990005, + "p99_ms": 4.30209999, + "mean_ms": 3.2989667, + "knob": null + } + ], + "load_before_pct": 4.415624999999998, + "load_contaminated": false + }, + "cpp_default_ivf|text384|10000": { + "version": "isongraph-vector-cpp (repo HEAD, MSVC /O2)", + "build": { + "ingest_s": 0.00753950002, + "index_s": 0.0880093, + "total_s": 0.0955488, + "first_query_ms": 0, + "ready_s": 0.0955488 + }, + "mem_baseline": { + "rss_mb": 11.0507812, + "peak_mb": 11.0507812 + }, + "mem_after_build": { + "rss_mb": 47.7226562, + "peak_mb": 62.859375 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.2963, + "qps_1t": 44365.6711, + "qps_1t_min": 43354.9819, + "qps_1t_max": 46078.5966, + "qps_1t_runs": [ + 43354.9819, + 44365.6711, + 46078.5966 + ], + "median_ms": 0.0211999868, + "p95_ms": 0.0316000078, + "p99_ms": 0.0389000052, + "mean_ms": 0.0224357999, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.41875, + "qps_1t": 44228.6086, + "qps_1t_min": 44186.2008, + "qps_1t_max": 45433.7902, + "qps_1t_runs": [ + 44228.6086, + 45433.7902, + 44186.2008 + ], + "median_ms": 0.0219999929, + "p95_ms": 0.0261999667, + "p99_ms": 0.0290000462, + "mean_ms": 0.0224171167, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.55125, + "qps_1t": 39332.3721, + "qps_1t_min": 39285.3985, + "qps_1t_max": 39407.7024, + "qps_1t_runs": [ + 39407.7024, + 39332.3721, + 39285.3985 + ], + "median_ms": 0.024900015, + "p95_ms": 0.0298999948, + "p99_ms": 0.0334000215, + "mean_ms": 0.0254182833, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.6798, + "qps_1t": 31354.7937, + "qps_1t_min": 31278.5415, + "qps_1t_max": 31627.7039, + "qps_1t_runs": [ + 31278.5415, + 31354.7937, + 31627.7039 + ], + "median_ms": 0.0311999465, + "p95_ms": 0.0381999998, + "p99_ms": 0.0419999706, + "mean_ms": 0.0318272334, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.8005, + "qps_1t": 23507.1768, + "qps_1t_min": 23091.5685, + "qps_1t_max": 23638.2031, + "qps_1t_runs": [ + 23091.5685, + 23638.2031, + 23507.1768 + ], + "median_ms": 0.0418000272, + "p95_ms": 0.0513000414, + "p99_ms": 0.0569000258, + "mean_ms": 0.0427168166, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.8922, + "qps_1t": 15302.2189, + "qps_1t_min": 14886.9005, + "qps_1t_max": 15408.7122, + "qps_1t_runs": [ + 14886.9005, + 15408.7122, + 15302.2189 + ], + "median_ms": 0.0648999703, + "p95_ms": 0.0795000233, + "p99_ms": 0.086999964, + "mean_ms": 0.0658071666, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.9563, + "qps_1t": 9372.79592, + "qps_1t_min": 9323.58336, + "qps_1t_max": 9429.51883, + "qps_1t_runs": [ + 9429.51883, + 9372.79592, + 9323.58336 + ], + "median_ms": 0.105899991, + "p95_ms": 0.123400008, + "p99_ms": 0.138600008, + "mean_ms": 0.106665533, + "knob": 64 + }, + { + "nq": 2000, + "recall": 0.9874, + "qps_1t": 4953.5984, + "qps_1t_min": 4682.45465, + "qps_1t_max": 5307.43306, + "qps_1t_runs": [ + 4682.45465, + 4953.5984, + 5307.43306 + ], + "median_ms": 0.192300009, + "p95_ms": 0.269600016, + "p99_ms": 0.330599956, + "mean_ms": 0.201283883, + "knob": 128 + }, + { + "nq": 2000, + "recall": 0.99845, + "qps_1t": 2772.07462, + "qps_1t_min": 2745.69773, + "qps_1t_max": 2842.35288, + "qps_1t_runs": [ + 2842.35288, + 2745.69773, + 2772.07462 + ], + "median_ms": 0.349299982, + "p95_ms": 0.420299999, + "p99_ms": 0.473699998, + "mean_ms": 0.358922667, + "knob": 256 + }, + { + "nq": 875, + "recall": 1, + "qps_1t": 311.48558, + "qps_1t_min": 308.851014, + "qps_1t_max": 314.30099, + "qps_1t_runs": [ + 308.851014, + 314.30099, + 311.48558 + ], + "median_ms": 3.17690003, + "p95_ms": 3.5012, + "p99_ms": 3.66749999, + "mean_ms": 3.20996404, + "knob": 400 + } + ], + "load_before_pct": 4.415624999999998, + "load_contaminated": false + }, + "csharp_exact|glove|1000000": { + "version": "isongraph-vector-csharp (repo HEAD, .NET 8 Release)", + "build": { + "ingest_s": 0.19757949997438118, + "index_s": 0, + "total_s": 0.19757949997438118, + "first_query_ms": 122.77479999465868, + "ready_s": 0.32035429996903986 + }, + "mem_baseline": { + "rss_mb": 27.96484375, + "peak_mb": 27.96484375 + }, + "mem_after_build": { + "rss_mb": 838.33203125, + "peak_mb": 838.33203125 + }, + "mem_after_first_query": { + "rss_mb": 838.33984375, + "peak_mb": 838.33984375 + }, + "sweep": [ + { + "nq": 30, + "recall": 1, + "qps_1t": 5.808417818488097, + "qps_1t_min": 5.808417818488097, + "qps_1t_max": 5.808417818488097, + "qps_1t_runs": [ + 5.808417818488097 + ], + "median_ms": 126.38750002952293, + "p95_ms": 369.5667000138201, + "p99_ms": 399.25610000500455, + "mean_ms": 172.1639233350288, + "knob": null + } + ], + "load_before_pct": 5.187500000000002, + "load_contaminated": false + }, + "csharp_ivf|glove|1000000": { + "version": "isongraph-vector-csharp (repo HEAD, .NET 8 Release)", + "build": { + "ingest_s": 0.33852740010479465, + "index_s": 11.303626800014172, + "total_s": 11.642154200118966, + "first_query_ms": 0, + "ready_s": 11.642154200118966 + }, + "mem_baseline": { + "rss_mb": 27.96484375, + "peak_mb": 27.96484375 + }, + "mem_after_build": { + "rss_mb": 2101.2890625, + "peak_mb": 2101.2890625 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.2969, + "qps_1t": 2482.6610944296576, + "qps_1t_min": 2138.7119363346815, + "qps_1t_max": 2664.7097044690763, + "qps_1t_runs": [ + 2138.7119363346815, + 2482.6610944296576, + 2664.7097044690763 + ], + "median_ms": 0.41089998558163643, + "p95_ms": 0.5002000252716243, + "p99_ms": 0.9454999817535281, + "mean_ms": 0.41521338339468156, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.41435, + "qps_1t": 2367.0765230664338, + "qps_1t_min": 2352.7452077845332, + "qps_1t_max": 2402.815138448941, + "qps_1t_runs": [ + 2352.7452077845332, + 2367.0765230664338, + 2402.815138448941 + ], + "median_ms": 0.43129996629431844, + "p95_ms": 0.5011000321246684, + "p99_ms": 0.539699976798147, + "mean_ms": 0.42122531652178924, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.53055, + "qps_1t": 2035.6451631667, + "qps_1t_min": 2020.5320398783858, + "qps_1t_max": 2060.001244889335, + "qps_1t_runs": [ + 2035.6451631667, + 2060.001244889335, + 2020.5320398783858 + ], + "median_ms": 0.4993999609723687, + "p95_ms": 0.6009999779053032, + "p99_ms": 0.6523000192828476, + "mean_ms": 0.4905335001143006, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.633, + "qps_1t": 1558.0563745986215, + "qps_1t_min": 1547.043583584809, + "qps_1t_max": 1574.5244464732775, + "qps_1t_runs": [ + 1547.043583584809, + 1574.5244464732775, + 1558.0563745986215 + ], + "median_ms": 0.6372000207193196, + "p95_ms": 0.8135999669320881, + "p99_ms": 0.8918999810703099, + "mean_ms": 0.6411106334029075, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.71885, + "qps_1t": 1107.506485028945, + "qps_1t_min": 1092.3245849112725, + "qps_1t_max": 1115.1629317074523, + "qps_1t_runs": [ + 1107.506485028945, + 1092.3245849112725, + 1115.1629317074523 + ], + "median_ms": 0.8931999909691513, + "p95_ms": 1.194199954625219, + "p99_ms": 1.323700009379536, + "mean_ms": 0.9050459999707527, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.7955, + "qps_1t": 713.1717656520507, + "qps_1t_min": 706.7175018085878, + "qps_1t_max": 718.5526596060247, + "qps_1t_runs": [ + 713.1717656520507, + 706.7175018085878, + 718.5526596060247 + ], + "median_ms": 1.3818999868817627, + "p95_ms": 1.8756999634206295, + "p99_ms": 2.0440000225789845, + "mean_ms": 1.4029552498735334, + "knob": 32 + }, + { + "nq": 1483, + "recall": 0.8602832097100472, + "qps_1t": 422.7358574928612, + "qps_1t_min": 417.2103342087863, + "qps_1t_max": 425.44809616995457, + "qps_1t_runs": [ + 425.44809616995457, + 422.7358574928612, + 417.2103342087863 + ], + "median_ms": 2.3467999999411404, + "p95_ms": 3.132499987259507, + "p99_ms": 3.4493000130169094, + "mean_ms": 2.370959631344029, + "knob": 64 + }, + { + "nq": 674, + "recall": 0.9160237388724035, + "qps_1t": 248.45892797274553, + "qps_1t_min": 246.3038885978749, + "qps_1t_max": 273.0388920877229, + "qps_1t_runs": [ + 246.3038885978749, + 248.45892797274553, + 273.0388920877229 + ], + "median_ms": 3.863200021442026, + "p95_ms": 5.196599988266826, + "p99_ms": 5.62569999601692, + "mean_ms": 3.915772403352715, + "knob": 128 + }, + { + "nq": 336, + "recall": 0.9485119047619047, + "qps_1t": 131.04159778558738, + "qps_1t_min": 128.50672770607466, + "qps_1t_max": 144.59467039493364, + "qps_1t_runs": [ + 144.59467039493364, + 131.04159778558738, + 128.50672770607466 + ], + "median_ms": 7.374399981927127, + "p95_ms": 9.607499989215285, + "p99_ms": 10.472499998286366, + "mean_ms": 7.442914186651164, + "knob": 256 + } + ], + "mutation": { + "ref_knob": 256, + "recall_before": 0, + "insert_10k": { + "seconds": 0.028511700045783073, + "per_s": 350733.20720765006 + }, + "update_10k": { + "seconds": 0.016209499968681484, + "per_s": 616922.1764595506 + }, + "delete_10k": { + "seconds": 0.038698900025337934, + "per_s": 258405.2775002014 + }, + "first_query_after_ms": 135.6134000234306, + "recall_after": 0.9493297587131367, + "after": { + "nq": 373, + "recall": 0.9493297587131367, + "qps_1t": 127.7292802347123, + "qps_1t_min": 127.68586185220663, + "qps_1t_max": 128.30734099394698, + "qps_1t_runs": [ + 127.68586185220663, + 127.7292802347123, + 128.30734099394698 + ], + "median_ms": 7.819599995855242, + "p95_ms": 9.831800009123981, + "p99_ms": 10.671299998648465, + "mean_ms": 7.818188471469938 + }, + "mem_after": { + "rss_mb": 2136.64453125, + "peak_mb": 2136.64453125 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 1000000 + } + }, + "load_before_pct": 5.187500000000002, + "load_contaminated": false + }, + "csharp_exact|glove|100000": { + "version": "isongraph-vector-csharp (repo HEAD, .NET 8 Release)", + "build": { + "ingest_s": 0.017087700020056218, + "index_s": 0, + "total_s": 0.017087700020056218, + "first_query_ms": 18.324600008782, + "ready_s": 0.03541230002883822 + }, + "mem_baseline": { + "rss_mb": 27.90234375, + "peak_mb": 27.90234375 + }, + "mem_after_build": { + "rss_mb": 115.22265625, + "peak_mb": 115.22265625 + }, + "mem_after_first_query": { + "rss_mb": 115.23046875, + "peak_mb": 115.23046875 + }, + "sweep": [ + { + "nq": 387, + "recall": 1, + "qps_1t": 107.91965457129174, + "qps_1t_min": 107.40336680013168, + "qps_1t_max": 108.34074222690144, + "qps_1t_runs": [ + 107.40336680013168, + 107.91965457129174, + 108.34074222690144 + ], + "median_ms": 7.786600035615265, + "p95_ms": 26.426199998240918, + "p99_ms": 35.582799988333136, + "mean_ms": 9.26899526248791, + "knob": null + } + ], + "load_before_pct": 4.287499999999999, + "load_contaminated": false + }, + "csharp_ivf|glove|100000": { + "version": "isongraph-vector-csharp (repo HEAD, .NET 8 Release)", + "build": { + "ingest_s": 0.011220699991099536, + "index_s": 0.6018785000196658, + "total_s": 0.6130992000107653, + "first_query_ms": 0, + "ready_s": 0.6130992000107653 + }, + "mem_baseline": { + "rss_mb": 27.90234375, + "peak_mb": 27.90234375 + }, + "mem_after_build": { + "rss_mb": 277.21875, + "peak_mb": 277.21875 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.28675, + "qps_1t": 14066.11213269189, + "qps_1t_min": 6337.005098092103, + "qps_1t_max": 14394.164016536084, + "qps_1t_runs": [ + 6337.005098092103, + 14394.164016536084, + 14066.11213269189 + ], + "median_ms": 0.0686999992467463, + "p95_ms": 0.21190004190430045, + "p99_ms": 0.23200002033263445, + "mean_ms": 0.09945623337019546, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.4092, + "qps_1t": 13475.581571006189, + "qps_1t_min": 13083.345468361927, + "qps_1t_max": 13665.057155795492, + "qps_1t_runs": [ + 13665.057155795492, + 13475.581571006189, + 13083.345468361927 + ], + "median_ms": 0.07269997149705887, + "p95_ms": 0.08359999628737569, + "p99_ms": 0.09129999671131372, + "mean_ms": 0.07460690003548127, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.52735, + "qps_1t": 12150.188474798064, + "qps_1t_min": 11931.764631294978, + "qps_1t_max": 12194.720437087037, + "qps_1t_runs": [ + 12194.720437087037, + 11931.764631294978, + 12150.188474798064 + ], + "median_ms": 0.08149998029693961, + "p95_ms": 0.096399977337569, + "p99_ms": 0.1058999914675951, + "mean_ms": 0.08270528328527386, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.6336, + "qps_1t": 9682.402673244265, + "qps_1t_min": 9671.914175416765, + "qps_1t_max": 9773.47527511135, + "qps_1t_runs": [ + 9682.402673244265, + 9671.914175416765, + 9773.47527511135 + ], + "median_ms": 0.10040000779554248, + "p95_ms": 0.12330000754445791, + "p99_ms": 0.13899995246902108, + "mean_ms": 0.10299668328176873, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.7249, + "qps_1t": 7268.405875377789, + "qps_1t_min": 7000.624444767023, + "qps_1t_max": 7361.59376170538, + "qps_1t_runs": [ + 7000.624444767023, + 7268.405875377789, + 7361.59376170538 + ], + "median_ms": 0.13639999087899923, + "p95_ms": 0.17499999376013875, + "p99_ms": 0.1947000273503363, + "mean_ms": 0.13875543336810855, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.8096, + "qps_1t": 4611.594239951243, + "qps_1t_min": 4435.917624805918, + "qps_1t_max": 4694.782781633785, + "qps_1t_runs": [ + 4611.594239951243, + 4694.782781633785, + 4435.917624805918 + ], + "median_ms": 0.2148000057786703, + "p95_ms": 0.27600000612437725, + "p99_ms": 0.30890002381056547, + "mean_ms": 0.21842655000121644, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.878, + "qps_1t": 2982.8050219600977, + "qps_1t_min": 2918.132237454387, + "qps_1t_max": 2983.527200916418, + "qps_1t_runs": [ + 2918.132237454387, + 2982.8050219600977, + 2983.527200916418 + ], + "median_ms": 0.33359997905790806, + "p95_ms": 0.42259995825588703, + "p99_ms": 0.4817000008188188, + "mean_ms": 0.33770453340063494, + "knob": 64 + }, + { + "nq": 2000, + "recall": 0.9323, + "qps_1t": 1677.137173299409, + "qps_1t_min": 1631.7020117415975, + "qps_1t_max": 1699.974406339156, + "qps_1t_runs": [ + 1677.137173299409, + 1699.974406339156, + 1631.7020117415975 + ], + "median_ms": 0.5938000394962728, + "p95_ms": 0.747899990528822, + "p99_ms": 0.8287999662570655, + "mean_ms": 0.5991184334949745, + "knob": 128 + }, + { + "nq": 2000, + "recall": 0.97265, + "qps_1t": 886.2994766147127, + "qps_1t_min": 825.7559218151433, + "qps_1t_max": 886.3834571301405, + "qps_1t_runs": [ + 886.2994766147127, + 886.3834571301405, + 825.7559218151433 + ], + "median_ms": 1.1380999931134284, + "p95_ms": 1.4329000259749591, + "p99_ms": 1.62390002515167, + "mean_ms": 1.1558260998378196, + "knob": 256 + }, + { + "nq": 329, + "recall": 1, + "qps_1t": 98.65570062707044, + "qps_1t_min": 95.54740662194229, + "qps_1t_max": 100.95501603998943, + "qps_1t_runs": [ + 95.54740662194229, + 98.65570062707044, + 100.95501603998943 + ], + "median_ms": 7.89549999171868, + "p95_ms": 38.67310000350699, + "p99_ms": 49.266899994108826, + "mean_ms": 10.169224113909095, + "knob": 1265 + } + ], + "mutation": { + "ref_knob": 256, + "recall_before": 0.97265, + "insert_10k": { + "seconds": 0.04945140000199899, + "per_s": 202218.74405165005 + }, + "update_10k": { + "seconds": 0.007727499993052334, + "per_s": 1294079.587058018 + }, + "delete_10k": { + "seconds": 0.0034479000023566186, + "per_s": 2900316.1324763075 + }, + "first_query_after_ms": 33.11979997670278, + "recall_after": 0.9723, + "after": { + "nq": 2000, + "recall": 0.9723, + "qps_1t": 862.0308059308751, + "qps_1t_min": 856.138545681082, + "qps_1t_max": 870.4372438566799, + "qps_1t_runs": [ + 870.4372438566799, + 856.138545681082, + 862.0308059308751 + ], + "median_ms": 1.1377999908290803, + "p95_ms": 1.4328999677672982, + "p99_ms": 1.6453000134788454, + "mean_ms": 1.1589781666310348 + }, + "mem_after": { + "rss_mb": 371.09765625, + "peak_mb": 821.53515625 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 100000 + } + }, + "saveload": { + "save_s": 1.3590893999789841, + "disk_mb": 121.46593761444092, + "load_s": 1.273968099965714, + "first_query_after_load_ms": 332.74230000097305, + "time_to_ready_s": 1.606710399966687 + }, + "load_before_pct": 4.287499999999999, + "load_contaminated": false + }, + "csharp_exact|sift|100000": { + "version": "isongraph-vector-csharp (repo HEAD, .NET 8 Release)", + "build": { + "ingest_s": 0.017743100004736334, + "index_s": 0, + "total_s": 0.017743100004736334, + "first_query_ms": 22.882099961861968, + "ready_s": 0.0406251999665983 + }, + "mem_baseline": { + "rss_mb": 28.546875, + "peak_mb": 28.546875 + }, + "mem_after_build": { + "rss_mb": 136.390625, + "peak_mb": 136.390625 + }, + "mem_after_first_query": { + "rss_mb": 136.3984375, + "peak_mb": 136.3984375 + }, + "sweep": [ + { + "nq": 314, + "recall": 1, + "qps_1t": 54.95953245815263, + "qps_1t_min": 54.95953245815263, + "qps_1t_max": 72.93859954981716, + "qps_1t_runs": [ + 72.93859954981716, + 54.95953245815263 + ], + "median_ms": 14.593200001399964, + "p95_ms": 42.7979999803938, + "p99_ms": 80.84619999863207, + "mean_ms": 15.952683757037033, + "knob": null + } + ], + "load_before_pct": 9.890625, + "load_contaminated": false + }, + "csharp_ivf|sift|100000": { + "version": "isongraph-vector-csharp (repo HEAD, .NET 8 Release)", + "build": { + "ingest_s": 0.02837600006023422, + "index_s": 1.4822395000373945, + "total_s": 1.5106155000976287, + "first_query_ms": 0, + "ready_s": 1.5106155000976287 + }, + "mem_baseline": { + "rss_mb": 28.546875, + "peak_mb": 28.546875 + }, + "mem_after_build": { + "rss_mb": 339.109375, + "peak_mb": 339.109375 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.285, + "qps_1t": 8528.420952993927, + "qps_1t_min": 5455.372193484733, + "qps_1t_max": 9394.955467120839, + "qps_1t_runs": [ + 5455.372193484733, + 9394.955467120839, + 8528.420952993927 + ], + "median_ms": 0.11610001092776656, + "p95_ms": 0.29450003057718277, + "p99_ms": 0.32409996492788196, + "mean_ms": 0.13566688337596133, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.43015, + "qps_1t": 7905.097725173819, + "qps_1t_min": 7739.644545359886, + "qps_1t_max": 8000.614444416882, + "qps_1t_runs": [ + 7905.097725173819, + 7739.644545359886, + 8000.614444416882 + ], + "median_ms": 0.12390001211315393, + "p95_ms": 0.14220003504306078, + "p99_ms": 0.15599996550008655, + "mean_ms": 0.12689865001205664, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.5908, + "qps_1t": 6901.389843260633, + "qps_1t_min": 6827.7701788934755, + "qps_1t_max": 7039.360938641796, + "qps_1t_runs": [ + 6901.389843260633, + 7039.360938641796, + 6827.7701788934755 + ], + "median_ms": 0.14349998673424125, + "p95_ms": 0.16960001084953547, + "p99_ms": 0.18700002692639828, + "mean_ms": 0.144472466626515, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.7513, + "qps_1t": 5823.219857140829, + "qps_1t_min": 5638.512164968473, + "qps_1t_max": 5948.9496826447075, + "qps_1t_runs": [ + 5823.219857140829, + 5638.512164968473, + 5948.9496826447075 + ], + "median_ms": 0.17209997167810798, + "p95_ms": 0.21239998750388622, + "p99_ms": 0.23639999562874436, + "mean_ms": 0.1723916500050109, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.8749, + "qps_1t": 4284.10835744266, + "qps_1t_min": 4242.892892065133, + "qps_1t_max": 4379.495554297895, + "qps_1t_runs": [ + 4379.495554297895, + 4284.10835744266, + 4242.892892065133 + ], + "median_ms": 0.23070001043379307, + "p95_ms": 0.2940999693237245, + "p99_ms": 0.3305000136606395, + "mean_ms": 0.2324819499044679, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.95255, + "qps_1t": 2805.371331108233, + "qps_1t_min": 2527.946767907984, + "qps_1t_max": 2838.346919403495, + "qps_1t_runs": [ + 2838.346919403495, + 2805.371331108233, + 2527.946767907984 + ], + "median_ms": 0.36180001916363835, + "p95_ms": 0.48160000005736947, + "p99_ms": 0.5549000343307853, + "mean_ms": 0.3681182499858551, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.98895, + "qps_1t": 1726.807485746021, + "qps_1t_min": 1719.4581886181063, + "qps_1t_max": 1855.0279008457028, + "qps_1t_runs": [ + 1855.0279008457028, + 1726.807485746021, + 1719.4581886181063 + ], + "median_ms": 0.5561999860219657, + "p95_ms": 0.7480999920517206, + "p99_ms": 0.8698999881744385, + "mean_ms": 0.566585783274301, + "knob": 64 + }, + { + "nq": 2000, + "recall": 0.9984, + "qps_1t": 1055.6210977180679, + "qps_1t_min": 1028.940071454162, + "qps_1t_max": 1102.7663556178188, + "qps_1t_runs": [ + 1028.940071454162, + 1102.7663556178188, + 1055.6210977180679 + ], + "median_ms": 0.9150999831035733, + "p95_ms": 1.2823999859392643, + "p99_ms": 1.7554000369273126, + "mean_ms": 0.9419979666708969, + "knob": 128 + }, + { + "nq": 1547, + "recall": 0.9999353587588882, + "qps_1t": 621.7537531149608, + "qps_1t_min": 596.8806329730581, + "qps_1t_max": 651.9730127543297, + "qps_1t_runs": [ + 621.7537531149608, + 651.9730127543297, + 596.8806329730581 + ], + "median_ms": 1.5802000416442752, + "p95_ms": 2.12459999602288, + "p99_ms": 2.443400037009269, + "mean_ms": 1.6058454860645561, + "knob": 256 + }, + { + "nq": 198, + "recall": 1, + "qps_1t": 53.60233838210898, + "qps_1t_min": 52.992020471916135, + "qps_1t_max": 53.6696923184099, + "qps_1t_runs": [ + 53.60233838210898, + 53.6696923184099, + 52.992020471916135 + ], + "median_ms": 15.355499985162169, + "p95_ms": 40.92530003981665, + "p99_ms": 87.66459999606013, + "mean_ms": 18.71971936039203, + "knob": 1265 + } + ], + "mutation": { + "ref_knob": 32, + "recall_before": 0.95255, + "insert_10k": { + "seconds": 0.030085799982771277, + "per_s": 332382.7189480263 + }, + "update_10k": { + "seconds": 0.01525609998498112, + "per_s": 655475.5153574314 + }, + "delete_10k": { + "seconds": 0.00854740000795573, + "per_s": 1169946.4153651663 + }, + "first_query_after_ms": 51.75270000472665, + "recall_after": 0.9487, + "after": { + "nq": 2000, + "recall": 0.9487, + "qps_1t": 2633.470889560701, + "qps_1t_min": 2580.2196747197495, + "qps_1t_max": 2896.9888688641504, + "qps_1t_runs": [ + 2633.470889560701, + 2580.2196747197495, + 2896.9888688641504 + ], + "median_ms": 0.35989994648844004, + "p95_ms": 0.485299970023334, + "p99_ms": 0.597700010985136, + "mean_ms": 0.37082563330962637 + }, + "mem_after": { + "rss_mb": 479.72265625, + "peak_mb": 1027.79296875 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 100000 + } + }, + "saveload": { + "save_s": 3.507927500002552, + "disk_mb": 121.89836692810059, + "load_s": 3.0640739999944344, + "first_query_after_load_ms": 1132.6488999766298, + "time_to_ready_s": 4.196722899971064 + }, + "load_before_pct": 9.890625, + "load_contaminated": false + }, + "csharp_exact|text384|1000000": { + "version": "isongraph-vector-csharp (repo HEAD, .NET 8 Release)", + "build": { + "ingest_s": 0.21663540008012205, + "index_s": 0, + "total_s": 0.21663540008012205, + "first_query_ms": 298.69659995893016, + "ready_s": 0.5153320000390522 + }, + "mem_baseline": { + "rss_mb": 34.35546875, + "peak_mb": 34.35546875 + }, + "mem_after_build": { + "rss_mb": 2480.88671875, + "peak_mb": 2480.88671875 + }, + "mem_after_first_query": { + "rss_mb": 2480.890625, + "peak_mb": 2480.89453125 + }, + "sweep": [ + { + "nq": 30, + "recall": 1, + "qps_1t": 3.717775299591297, + "qps_1t_min": 3.717775299591297, + "qps_1t_max": 3.717775299591297, + "qps_1t_runs": [ + 3.717775299591297 + ], + "median_ms": 245.34419999690726, + "p95_ms": 394.20649997191504, + "p99_ms": 397.59900001809, + "mean_ms": 268.97806333533174, + "knob": null + } + ], + "load_before_pct": 2.6718749999999987, + "load_contaminated": false + }, + "csharp_ivf|text384|1000000": { + "version": "isongraph-vector-csharp (repo HEAD, .NET 8 Release)", + "build": { + "ingest_s": 0.17699659999925643, + "index_s": 10.716460700030439, + "total_s": 10.893457300029695, + "first_query_ms": 0, + "ready_s": 10.893457300029695 + }, + "mem_baseline": { + "rss_mb": 34.35546875, + "peak_mb": 34.35546875 + }, + "mem_after_build": { + "rss_mb": 6689.90625, + "peak_mb": 6698.26171875 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.27405, + "qps_1t": 2528.7109857144565, + "qps_1t_min": 2179.0743696895534, + "qps_1t_max": 2619.7744516893154, + "qps_1t_runs": [ + 2179.0743696895534, + 2528.7109857144565, + 2619.7744516893154 + ], + "median_ms": 0.3837000112980604, + "p95_ms": 0.6302999681793153, + "p99_ms": 0.7325999904423952, + "mean_ms": 0.4120270332787186, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.3823, + "qps_1t": 2128.6749987375856, + "qps_1t_min": 2065.8463738630026, + "qps_1t_max": 2179.2206301267274, + "qps_1t_runs": [ + 2128.6749987375856, + 2179.2206301267274, + 2065.8463738630026 + ], + "median_ms": 0.45469996985048056, + "p95_ms": 0.6121999467723072, + "p99_ms": 0.7542999810539186, + "mean_ms": 0.470906183278809, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.50015, + "qps_1t": 1684.8468663471908, + "qps_1t_min": 1600.9149550235593, + "qps_1t_max": 1685.3750328611393, + "qps_1t_runs": [ + 1685.3750328611393, + 1600.9149550235593, + 1684.8468663471908 + ], + "median_ms": 0.5908000166527927, + "p95_ms": 0.7831000257283449, + "p99_ms": 0.9157000458799303, + "mean_ms": 0.6038360998597152, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.6082, + "qps_1t": 1170.855166345593, + "qps_1t_min": 1136.27686613993, + "qps_1t_max": 1174.9780894480484, + "qps_1t_runs": [ + 1170.855166345593, + 1136.27686613993, + 1174.9780894480484 + ], + "median_ms": 0.8402999956160784, + "p95_ms": 1.1089000036008656, + "p99_ms": 1.2582999770529568, + "mean_ms": 0.8617411668237764, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.70535, + "qps_1t": 746.7836868029974, + "qps_1t_min": 734.0894822449857, + "qps_1t_max": 756.2227775822607, + "qps_1t_runs": [ + 746.7836868029974, + 756.2227775822607, + 734.0894822449857 + ], + "median_ms": 1.317800022661686, + "p95_ms": 1.7407999839633703, + "p99_ms": 2.0183000015094876, + "mean_ms": 1.3412230832618661, + "knob": 16 + }, + { + "nq": 1183, + "recall": 0.7822485207100591, + "qps_1t": 444.68901292113406, + "qps_1t_min": 438.26828241652515, + "qps_1t_max": 445.86765300693673, + "qps_1t_runs": [ + 438.26828241652515, + 444.68901292113406, + 445.86765300693673 + ], + "median_ms": 2.233400009572506, + "p95_ms": 2.8313000220805407, + "p99_ms": 3.177399979904294, + "mean_ms": 2.2577626654642344, + "knob": 32 + }, + { + "nq": 857, + "recall": 0.8549591598599766, + "qps_1t": 252.84508998112273, + "qps_1t_min": 250.30883350087203, + "qps_1t_max": 257.58553330033277, + "qps_1t_runs": [ + 257.58553330033277, + 252.84508998112273, + 250.30883350087203 + ], + "median_ms": 3.882699995301664, + "p95_ms": 4.866900038905442, + "p99_ms": 5.616399983409792, + "mean_ms": 3.9440870475392025, + "knob": 64 + }, + { + "nq": 426, + "recall": 0.9096244131455399, + "qps_1t": 141.02315610854967, + "qps_1t_min": 140.94946606200546, + "qps_1t_max": 145.00826903551325, + "qps_1t_runs": [ + 140.94946606200546, + 141.02315610854967, + 145.00826903551325 + ], + "median_ms": 6.930499977897853, + "p95_ms": 8.40059999609366, + "p99_ms": 9.481300017796457, + "mean_ms": 7.027311267328328, + "knob": 128 + }, + { + "nq": 234, + "recall": 0.9461538461538461, + "qps_1t": 74.5134605231173, + "qps_1t_min": 72.82908632251784, + "qps_1t_max": 77.02682257961665, + "qps_1t_runs": [ + 74.5134605231173, + 77.02682257961665, + 72.82908632251784 + ], + "median_ms": 13.105199963320047, + "p95_ms": 16.224100021645427, + "p99_ms": 19.192199979443103, + "mean_ms": 13.377887465240848, + "knob": 256 + } + ], + "mutation": { + "ref_knob": 256, + "recall_before": 0, + "insert_10k": { + "seconds": 0.014750300033483654, + "per_s": 677952.3112953417 + }, + "update_10k": { + "seconds": 0.008894999977201223, + "per_s": 1124227.096754469 + }, + "delete_10k": { + "seconds": 0.00644060003105551, + "per_s": 1552650.3667021163 + }, + "first_query_after_ms": 110.59449997264892, + "recall_after": 0.9464435146443515, + "after": { + "nq": 239, + "recall": 0.9464435146443515, + "qps_1t": 76.2746320776492, + "qps_1t_min": 75.29371558113026, + "qps_1t_max": 76.36682558516418, + "qps_1t_runs": [ + 76.2746320776492, + 75.29371558113026, + 76.36682558516418 + ], + "median_ms": 12.906700023449957, + "p95_ms": 15.67679998697713, + "p99_ms": 17.723999975714833, + "mean_ms": 13.162176987881066 + }, + "mem_after": { + "rss_mb": 6774.3828125, + "peak_mb": 6774.3828125 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 1000000 + } + }, + "load_before_pct": 2.6718749999999987, + "load_contaminated": false + }, + "csharp_exact|text384|100000": { + "version": "isongraph-vector-csharp (repo HEAD, .NET 8 Release)", + "build": { + "ingest_s": 0.018145900103263557, + "index_s": 0, + "total_s": 0.018145900103263557, + "first_query_ms": 46.90479999408126, + "ready_s": 0.06505070009734482 + }, + "mem_baseline": { + "rss_mb": 34.3671875, + "peak_mb": 34.3671875 + }, + "mem_after_build": { + "rss_mb": 325.40625, + "peak_mb": 325.40625 + }, + "mem_after_first_query": { + "rss_mb": 325.40625, + "peak_mb": 325.40625 + }, + "sweep": [ + { + "nq": 128, + "recall": 1, + "qps_1t": 39.143486688959456, + "qps_1t_min": 38.5773911913353, + "qps_1t_max": 39.927237596704984, + "qps_1t_runs": [ + 39.927237596704984, + 39.143486688959456, + 38.5773911913353 + ], + "median_ms": 23.59890000661835, + "p95_ms": 26.737500040326267, + "p99_ms": 78.39999999850988, + "mean_ms": 25.504837500344973, + "knob": null + } + ], + "load_before_pct": 4.203124999999998, + "load_contaminated": false + }, + "csharp_ivf|text384|100000": { + "version": "isongraph-vector-csharp (repo HEAD, .NET 8 Release)", + "build": { + "ingest_s": 0.015840800013393164, + "index_s": 1.2653368000173941, + "total_s": 1.2811776000307873, + "first_query_ms": 0, + "ready_s": 1.2811776000307873 + }, + "mem_baseline": { + "rss_mb": 34.3671875, + "peak_mb": 34.3671875 + }, + "mem_after_build": { + "rss_mb": 821.87109375, + "peak_mb": 821.87109375 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.27755, + "qps_1t": 9706.066360519993, + "qps_1t_min": 5430.374857466292, + "qps_1t_max": 9894.171933994656, + "qps_1t_runs": [ + 5430.374857466292, + 9706.066360519993, + 9894.171933994656 + ], + "median_ms": 0.10350003140047193, + "p95_ms": 0.21210004342719913, + "p99_ms": 0.23100001271814108, + "mean_ms": 0.12941576658825701, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.39915, + "qps_1t": 8523.895674038828, + "qps_1t_min": 8518.765561809434, + "qps_1t_max": 8603.634434426238, + "qps_1t_runs": [ + 8603.634434426238, + 8518.765561809434, + 8523.895674038828 + ], + "median_ms": 0.11339999036863446, + "p95_ms": 0.13609998859465122, + "p99_ms": 0.1566999708302319, + "mean_ms": 0.11697836667493296, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.52005, + "qps_1t": 6881.152180755526, + "qps_1t_min": 6817.441201758321, + "qps_1t_max": 6932.4138244126725, + "qps_1t_runs": [ + 6817.441201758321, + 6932.4138244126725, + 6881.152180755526 + ], + "median_ms": 0.14270003885030746, + "p95_ms": 0.17590000061318278, + "p99_ms": 0.20030001178383827, + "mean_ms": 0.14541899991066506, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.6376, + "qps_1t": 4982.163855045766, + "qps_1t_min": 4892.782018353975, + "qps_1t_max": 5000.908921098519, + "qps_1t_runs": [ + 4892.782018353975, + 4982.163855045766, + 5000.908921098519 + ], + "median_ms": 0.1970999874174595, + "p95_ms": 0.24769996525719762, + "p99_ms": 0.27650000993162394, + "mean_ms": 0.20168744993861765, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.74525, + "qps_1t": 3315.4728823879177, + "qps_1t_min": 3315.4014311213896, + "qps_1t_max": 3348.955653799659, + "qps_1t_runs": [ + 3315.4014311213896, + 3348.955653799659, + 3315.4728823879177 + ], + "median_ms": 0.2958999830298126, + "p95_ms": 0.3722999827004969, + "p99_ms": 0.4113000468350947, + "mean_ms": 0.30061308330429404, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.8359, + "qps_1t": 1974.545151249959, + "qps_1t_min": 1971.2368919733783, + "qps_1t_max": 2018.693504697093, + "qps_1t_runs": [ + 1974.545151249959, + 1971.2368919733783, + 2018.693504697093 + ], + "median_ms": 0.49399997806176543, + "p95_ms": 0.6204000092111528, + "p99_ms": 0.6898000137880445, + "mean_ms": 0.5030371169074594, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.90595, + "qps_1t": 1134.2566903835593, + "qps_1t_min": 1127.0211435259314, + "qps_1t_max": 1134.5587846774165, + "qps_1t_runs": [ + 1134.5587846774165, + 1127.0211435259314, + 1134.2566903835593 + ], + "median_ms": 0.8779999916441739, + "p95_ms": 1.0717000113800168, + "p99_ms": 1.1555000091902912, + "mean_ms": 0.8834431167439712, + "knob": 64 + }, + { + "nq": 1848, + "recall": 0.952056277056277, + "qps_1t": 598.0226033292612, + "qps_1t_min": 591.1868058939422, + "qps_1t_max": 599.513946038056, + "qps_1t_runs": [ + 598.0226033292612, + 599.513946038056, + 591.1868058939422 + ], + "median_ms": 1.664099982008338, + "p95_ms": 1.999199972487986, + "p99_ms": 2.2201999672688544, + "mean_ms": 1.6772360750175859, + "knob": 128 + }, + { + "nq": 897, + "recall": 0.9804905239687849, + "qps_1t": 305.6254191463579, + "qps_1t_min": 304.8535022490456, + "qps_1t_max": 306.37391987381085, + "qps_1t_runs": [ + 305.6254191463579, + 306.37391987381085, + 304.8535022490456 + ], + "median_ms": 3.2719000009819865, + "p95_ms": 3.690499986987561, + "p99_ms": 3.9022000273689628, + "mean_ms": 3.2720762169425, + "knob": 256 + }, + { + "nq": 122, + "recall": 1, + "qps_1t": 38.63550419867212, + "qps_1t_min": 37.41275238642007, + "qps_1t_max": 39.41333445668147, + "qps_1t_runs": [ + 38.63550419867212, + 37.41275238642007, + 39.41333445668147 + ], + "median_ms": 23.669400019571185, + "p95_ms": 25.08359996136278, + "p99_ms": 95.14760004822165, + "mean_ms": 25.99463551900034, + "knob": 1265 + } + ], + "mutation": { + "ref_knob": 128, + "recall_before": 0.952056277056277, + "insert_10k": { + "seconds": 0.02585999999428168, + "per_s": 386697.6025603736 + }, + "update_10k": { + "seconds": 0.008704700041562319, + "per_s": 1148804.6632569777 + }, + "delete_10k": { + "seconds": 0.004127299995161593, + "per_s": 2422891.4815310095 + }, + "first_query_after_ms": 44.82020001159981, + "recall_after": 0.9494189263973437, + "after": { + "nq": 1807, + "recall": 0.9494189263973437, + "qps_1t": 591.1852739081532, + "qps_1t_min": 590.4691545739432, + "qps_1t_max": 592.5683598025627, + "qps_1t_runs": [ + 591.1852739081532, + 592.5683598025627, + 590.4691545739432 + ], + "median_ms": 1.6832000110298395, + "p95_ms": 1.9957000040449202, + "p99_ms": 2.1343999542295933, + "mean_ms": 1.6908848921292827 + }, + "mem_after": { + "rss_mb": 1170.19140625, + "peak_mb": 1370.30859375 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 100000 + } + }, + "saveload": { + "save_s": 4.408426200039685, + "disk_mb": 452.6868906021118, + "load_s": 3.953144599974621, + "first_query_after_load_ms": 910.6310000061058, + "time_to_ready_s": 4.863775599980727 + }, + "load_before_pct": 4.203124999999998, + "load_contaminated": false + }, + "csharp_exact|text384|10000": { + "version": "isongraph-vector-csharp (repo HEAD, .NET 8 Release)", + "build": { + "ingest_s": 0.0047627000021748245, + "index_s": 0, + "total_s": 0.0047627000021748245, + "first_query_ms": 24.22700001625344, + "ready_s": 0.028989700018428266 + }, + "mem_baseline": { + "rss_mb": 34.35546875, + "peak_mb": 34.35546875 + }, + "mem_after_build": { + "rss_mb": 82.64453125, + "peak_mb": 82.64453125 + }, + "mem_after_first_query": { + "rss_mb": 82.64453125, + "peak_mb": 82.64453125 + }, + "sweep": [ + { + "nq": 706, + "recall": 1, + "qps_1t": 421.6285659999388, + "qps_1t_min": 382.4527638614905, + "qps_1t_max": 422.5427136924413, + "qps_1t_runs": [ + 382.4527638614905, + 421.6285659999388, + 422.5427136924413 + ], + "median_ms": 2.332200005184859, + "p95_ms": 2.7593999984674156, + "p99_ms": 4.294199985451996, + "mean_ms": 2.4510274785883004, + "knob": null + } + ], + "load_before_pct": 5.240625000000002, + "load_contaminated": false + }, + "csharp_ivf|text384|10000": { + "version": "isongraph-vector-csharp (repo HEAD, .NET 8 Release)", + "build": { + "ingest_s": 0.0010065999813377857, + "index_s": 0.21723320003366098, + "total_s": 0.21823980001499876, + "first_query_ms": 0, + "ready_s": 0.21823980001499876 + }, + "mem_baseline": { + "rss_mb": 34.35546875, + "peak_mb": 34.35546875 + }, + "mem_after_build": { + "rss_mb": 127.7890625, + "peak_mb": 214.0546875 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.2871, + "qps_1t": 13263.742914285021, + "qps_1t_min": 11380.364206647917, + "qps_1t_max": 38014.57456031691, + "qps_1t_runs": [ + 11380.364206647917, + 13263.742914285021, + 38014.57456031691 + ], + "median_ms": 0.06509997183457017, + "p95_ms": 0.09330001194030046, + "p99_ms": 0.10390003444626927, + "mean_ms": 0.06318995001493022, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.4162, + "qps_1t": 35917.47603515432, + "qps_1t_min": 35410.13789991665, + "qps_1t_max": 36260.26163904982, + "qps_1t_runs": [ + 35410.13789991665, + 35917.47603515432, + 36260.26163904982 + ], + "median_ms": 0.027399975806474686, + "p95_ms": 0.031800009310245514, + "p99_ms": 0.034999975468963385, + "mean_ms": 0.027886833335893847, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.55765, + "qps_1t": 32061.71243596814, + "qps_1t_min": 28122.63154126515, + "qps_1t_max": 32333.892692292353, + "qps_1t_runs": [ + 32333.892692292353, + 32061.71243596814, + 28122.63154126515 + ], + "median_ms": 0.030700000934302807, + "p95_ms": 0.03619998460635543, + "p99_ms": 0.0399000127799809, + "mean_ms": 0.032558566657826304, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.69015, + "qps_1t": 27379.78225036018, + "qps_1t_min": 26392.15318134635, + "qps_1t_max": 27465.359339925908, + "qps_1t_runs": [ + 26392.15318134635, + 27379.78225036018, + 27465.359339925908 + ], + "median_ms": 0.03619998460635543, + "p95_ms": 0.04250003257766366, + "p99_ms": 0.047200010158121586, + "mean_ms": 0.03694094993018855, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.80795, + "qps_1t": 20211.75860021833, + "qps_1t_min": 19990.38461461821, + "qps_1t_max": 21096.24530544721, + "qps_1t_runs": [ + 21096.24530544721, + 20211.75860021833, + 19990.38461461821 + ], + "median_ms": 0.04769995575770736, + "p95_ms": 0.057699973694980145, + "p99_ms": 0.06610003765672445, + "mean_ms": 0.04896733332619382, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.8959, + "qps_1t": 14154.37181271068, + "qps_1t_min": 13804.870893174471, + "qps_1t_max": 14311.393029373858, + "qps_1t_runs": [ + 13804.870893174471, + 14311.393029373858, + 14154.37181271068 + ], + "median_ms": 0.06890000076964498, + "p95_ms": 0.08410000009462237, + "p99_ms": 0.09869999485090375, + "mean_ms": 0.07098738335965511, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.95285, + "qps_1t": 8624.912887402754, + "qps_1t_min": 8440.670504909185, + "qps_1t_max": 8767.780517236995, + "qps_1t_runs": [ + 8624.912887402754, + 8767.780517236995, + 8440.670504909185 + ], + "median_ms": 0.11319998884573579, + "p95_ms": 0.139000010676682, + "p99_ms": 0.16009999671950936, + "mean_ms": 0.11615705007958847, + "knob": 64 + }, + { + "nq": 2000, + "recall": 0.9878, + "qps_1t": 4986.246684553161, + "qps_1t_min": 4766.138679841505, + "qps_1t_max": 5007.739458761456, + "qps_1t_runs": [ + 4986.246684553161, + 5007.739458761456, + 4766.138679841505 + ], + "median_ms": 0.19910000264644623, + "p95_ms": 0.24100003065541387, + "p99_ms": 0.2741999924182892, + "mean_ms": 0.20335200006957166, + "knob": 128 + }, + { + "nq": 2000, + "recall": 0.99895, + "qps_1t": 2565.8934268456387, + "qps_1t_min": 2479.8115446371407, + "qps_1t_max": 2636.376458691203, + "qps_1t_runs": [ + 2636.376458691203, + 2479.8115446371407, + 2565.8934268456387 + ], + "median_ms": 0.3795999800786376, + "p95_ms": 0.46389998169615865, + "p99_ms": 0.542099995072931, + "mean_ms": 0.3907642499446714, + "knob": 256 + }, + { + "nq": 1333, + "recall": 1, + "qps_1t": 420.6626462141402, + "qps_1t_min": 417.6507673578117, + "qps_1t_max": 421.22623790768824, + "qps_1t_runs": [ + 417.6507673578117, + 421.22623790768824, + 420.6626462141402 + ], + "median_ms": 2.3204999743029475, + "p95_ms": 2.628199988976121, + "p99_ms": 3.1295999651774764, + "mean_ms": 2.3818559641059034, + "knob": 400 + } + ], + "load_before_pct": 5.240625000000002, + "load_contaminated": false + }, + "faiss_flat|glove|1000000": { + "mem_baseline": { + "rss_mb": 34.97265625, + "peak_mb": 34.97265625 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 0.3754827000084333, + "index_s": 2.3999600671231747e-06, + "total_s": 0.37548509996850044, + "first_query_ms": 28.816199977882206, + "ready_s": 0.40430129994638264 + }, + "mem_after_build": { + "rss_mb": 504.5234375, + "peak_mb": 639.9921875 + }, + "mem_after_first_query": { + "rss_mb": 504.6015625, + "peak_mb": 639.9921875 + }, + "sweep": [ + { + "nq": 149, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 42.621622221620804, + 40.715336211406004, + 39.77757448303346 + ], + "qps_1t": 40.715336211406004, + "qps_1t_min": 39.77757448303346, + "qps_1t_max": 42.621622221620804, + "median_ms": 24.24780000001192, + "p95_ms": 27.131040033418685, + "p99_ms": 33.16403796430677, + "mean_ms": 24.387610963466567, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 638.9404629218408, + 642.4085076945206, + 596.8994654130149 + ], + "qps_batch": 638.9404629218408, + "qps_batch_min": 596.8994654130149, + "qps_batch_max": 642.4085076945206, + "threads": 32 + } + } + ], + "loop_allthreads": { + "nq": 85, + "threads": 32, + "qps": 49.72282741778036, + "median_ms": 20.405200019013137, + "p95_ms": 25.612299994099867, + "p99_ms": 29.875211997423346, + "mean_ms": 20.80622627240076 + }, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.004300100030377507, + "per_s": 2325527.2968899044, + "batch": 1000 + }, + "update_10k": { + "seconds": 3.943905799998902, + "per_s": 2535.5575176270145, + "batch": 1000 + }, + "delete_10k": { + "seconds": 4.188923900015652, + "per_s": 2387.247951666688, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 529.20703125, + "peak_mb": 639.9921875 + }, + "first_query_after_ms": 65.74649998219684, + "after": { + "nq": 119, + "recall_queries": 119, + "recall": 1.0, + "qps_1t_runs": [ + 37.5394677020512, + 37.76714183374294, + 37.35729474198767 + ], + "qps_1t": 37.5394677020512, + "qps_1t_min": 37.35729474198767, + "qps_1t_max": 37.76714183374294, + "median_ms": 26.818200014531612, + "p95_ms": 27.768639964051545, + "p99_ms": 29.021992003545165, + "mean_ms": 26.628402240459575 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 0.6554340000147931, + "disk_mb": 389.0992069244385, + "load_s": 0.9732154000084847, + "first_query_after_load_ms": 66.75289996201172, + "time_to_ready_s": 1.0399682999704964 + }, + "load_before_pct": 9.45937499999999, + "load_contaminated": false + }, + "faiss_flat|glove|100000": { + "mem_baseline": { + "rss_mb": 35.1953125, + "peak_mb": 35.1953125 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 0.03100400004768744, + "index_s": 1.400010660290718e-06, + "total_s": 0.031005400058347732, + "first_query_ms": 2.1878000115975738, + "ready_s": 0.033193200069945306, + "total_s_runs": [ + 0.031005400058347732, + 0.04659990005893633, + 0.03541469993069768 + ] + }, + "mem_after_build": { + "rss_mb": 98.4375, + "peak_mb": 135.70703125 + }, + "mem_after_first_query": { + "rss_mb": 98.515625, + "peak_mb": 135.70703125 + }, + "sweep": [ + { + "nq": 1809, + "recall_queries": 1809, + "recall": 1.0, + "qps_1t_runs": [ + 548.4301740392543, + 569.6022816084964, + 543.4661298597612 + ], + "qps_1t": 548.4301740392543, + "qps_1t_min": 543.4661298597612, + "qps_1t_max": 569.6022816084964, + "median_ms": 1.7164000310003757, + "p95_ms": 2.2275400231592357, + "p99_ms": 3.014491976937277, + "mean_ms": 1.8063460655963646, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 17355.988679941973, + 16184.59502077011, + 16430.47853314927 + ], + "qps_batch": 16430.47853314927, + "qps_batch_min": 16184.59502077011, + "qps_batch_max": 17355.988679941973, + "threads": 32 + } + } + ], + "loop_allthreads": { + "nq": 2000, + "threads": 32, + "qps": 2335.268685500277, + "median_ms": 0.41310000233352184, + "p95_ms": 0.613204954424873, + "p99_ms": 0.7323019881732762, + "mean_ms": 0.4299485333710133 + }, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.0020111000048927963, + "per_s": 4972403.150351074, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.10233710001921281, + "per_s": 97716.27296574357, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.09884759999113157, + "per_s": 101165.8350925787, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 123.79296875, + "peak_mb": 195.296875 + }, + "first_query_after_ms": 2.7194999856874347, + "after": { + "nq": 1344, + "recall_queries": 1344, + "recall": 1.0, + "qps_1t_runs": [ + 556.5854873373852, + 552.7321026899448, + 562.1154042453633 + ], + "qps_1t": 556.5854873373852, + "qps_1t_min": 552.7321026899448, + "qps_1t_max": 562.1154042453633, + "median_ms": 1.7166000325232744, + "p95_ms": 2.0978749991627406, + "p99_ms": 3.052488010143864, + "mean_ms": 1.7949526782248295 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 0.04537860001437366, + "disk_mb": 38.90999794006348, + "load_s": 0.024948099977336824, + "first_query_after_load_ms": 2.1277000196278095, + "time_to_ready_s": 0.027075799996964633 + }, + "load_before_pct": 8.865624999999993, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 0.035980099986772984, + "mem": { + "rss_mb": 97.71484375, + "peak_mb": 134.9140625 + }, + "subsets": { + "f10": [ + { + "knob": null, + "nq": 2000, + "recall": 1.0, + "qps_1t": 1813.4391806365081, + "qps_1t_runs": [ + 1813.4391806365081, + 1816.32883306855, + 1348.9110076033562 + ], + "median_ms": 0.5298000178299844, + "p95_ms": 0.9326049999799579, + "p99_ms": 1.0319079994224014, + "mean_ms": 0.6144460831710603 + } + ], + "f01": [ + { + "knob": null, + "nq": 2000, + "recall": 1.0, + "qps_1t": 2913.2868925627654, + "qps_1t_runs": [ + 2836.3367024385, + 2989.4910407770653, + 2913.2868925627654 + ], + "median_ms": 0.37949997931718826, + "p95_ms": 0.41820498590823263, + "p99_ms": 0.47071303299162565, + "mean_ms": 0.34344248344617273 + } + ] + } + } + }, + "faiss_flat|sift|100000": { + "mem_baseline": { + "rss_mb": 35.37109375, + "peak_mb": 35.37109375 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 0.050621099944692105, + "index_s": 2.2999593056738377e-06, + "total_s": 0.05062339990399778, + "first_query_ms": 3.5072999889962375, + "ready_s": 0.054130699892994016, + "total_s_runs": [ + 0.05062339990399778, + 0.08835479995468631, + 0.08185000001685694 + ] + }, + "mem_after_build": { + "rss_mb": 109.171875, + "peak_mb": 157.1015625 + }, + "mem_after_first_query": { + "rss_mb": 109.25, + "peak_mb": 157.1015625 + }, + "sweep": [ + { + "nq": 920, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 332.22724533304824, + 283.06582681202025, + 279.60595504025054 + ], + "qps_1t": 283.06582681202025, + "qps_1t_min": 279.60595504025054, + "qps_1t_max": 332.22724533304824, + "median_ms": 3.2985500001814216, + "p95_ms": 4.2591150093358, + "p99_ms": 6.042718980461356, + "mean_ms": 3.373065615955241, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 3003.8428160889043, + 4965.600800532972, + 4537.410953264215 + ], + "qps_batch": 4537.410953264215, + "qps_batch_min": 3003.8428160889043, + "qps_batch_max": 4965.600800532972, + "threads": 32 + } + } + ], + "loop_allthreads": { + "nq": 835, + "threads": 32, + "qps": 285.1063871587876, + "median_ms": 3.4351000213064253, + "p95_ms": 4.934740008320654, + "p99_ms": 6.255139959976077, + "mean_ms": 3.515263473728445 + }, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.004522399976849556, + "per_s": 2211215.29523939, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.5198939999681897, + "per_s": 19234.690149553295, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.3611703999922611, + "per_s": 27687.76178838098, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 138.796875, + "peak_mb": 228.41796875 + }, + "first_query_after_ms": 7.9619999742135406, + "after": { + "nq": 932, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 261.7900976180128, + 278.3804708982491, + 278.04389681098286 + ], + "qps_1t": 278.04389681098286, + "qps_1t_min": 261.7900976180128, + "qps_1t_max": 278.3804708982491, + "median_ms": 3.473200020380318, + "p95_ms": 4.544775016256608, + "p99_ms": 5.785115031176253, + "mean_ms": 3.669538197048656 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 0.06047320005018264, + "disk_mb": 49.59115028381348, + "load_s": 0.08740339998621494, + "first_query_after_load_ms": 12.896700005512685, + "time_to_ready_s": 0.10030009999172762 + }, + "load_before_pct": 16.834375, + "load_contaminated": false + }, + "faiss_flat|text384|1000000": { + "mem_baseline": { + "rss_mb": 37.29296875, + "peak_mb": 37.29296875 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 1.1041080001741648, + "index_s": 2.600019797682762e-06, + "total_s": 1.1041106001939625, + "first_query_ms": 77.55009998800233, + "ready_s": 1.1816607001819648 + }, + "mem_after_build": { + "rss_mb": 1590.109375, + "peak_mb": 2158.8671875 + }, + "mem_after_first_query": { + "rss_mb": 1590.1875, + "peak_mb": 2158.8671875 + }, + "sweep": [ + { + "nq": 40, + "recall_queries": 40, + "recall": 1.0, + "qps_1t_runs": [ + 13.406619277544596, + 13.30779700113377, + 13.498657238072042 + ], + "qps_1t": 13.406619277544596, + "qps_1t_min": 13.30779700113377, + "qps_1t_max": 13.498657238072042, + "median_ms": 73.35145000251941, + "p95_ms": 83.81378499907441, + "p99_ms": 87.61429797159508, + "mean_ms": 74.60512666633197, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 536.9377607260415, + 539.9866601723231, + 544.7127902417595 + ], + "qps_batch": 539.9866601723231, + "qps_batch_min": 536.9377607260415, + "qps_batch_max": 544.7127902417595, + "threads": 32 + } + } + ], + "loop_allthreads": { + "nq": 71, + "threads": 32, + "qps": 22.764519389133255, + "median_ms": 43.10910002095625, + "p95_ms": 48.87395998230203, + "p99_ms": 52.566392018925384, + "mean_ms": 43.705510796745145 + }, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.0073639999609440565, + "per_s": 1357957.6389239973, + "batch": 1000 + }, + "update_10k": { + "seconds": 2.3090925000142306, + "per_s": 4330.705677636722, + "batch": 1000 + }, + "delete_10k": { + "seconds": 2.249738300044555, + "per_s": 4444.9614427606775, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 1654.01953125, + "peak_mb": 2158.8671875 + }, + "first_query_after_ms": 72.11409998126328, + "after": { + "nq": 41, + "recall_queries": 41, + "recall": 1.0, + "qps_1t_runs": [ + 13.200426289242136, + 13.317282868129901, + 13.340324097395301 + ], + "qps_1t": 13.317282868129901, + "qps_1t_min": 13.200426289242136, + "qps_1t_max": 13.340324097395301, + "median_ms": 73.87199997901917, + "p95_ms": 82.18046997790225, + "p99_ms": 88.66506198071875, + "mean_ms": 75.26873983726723 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 0.9680129000335, + "disk_mb": 1472.4732303619385, + "load_s": 0.7364274999708869, + "first_query_after_load_ms": 74.09990002634004, + "time_to_ready_s": 0.8105273999972269 + }, + "load_before_pct": 13.424999999999994, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 1.132167800096795, + "mem": { + "rss_mb": 1591.390625, + "peak_mb": 2158.99609375 + }, + "subsets": { + "f10": [ + { + "knob": null, + "nq": 148, + "recall": 1.0, + "qps_1t": 39.68689719300676, + "qps_1t_runs": [ + 41.11284013059143, + 39.68689719300676, + 38.95106774889948 + ], + "median_ms": 24.96424998389557, + "p95_ms": 27.92759498115629, + "p99_ms": 32.722414002055274, + "mean_ms": 25.064590541742373 + } + ], + "f01": [ + { + "knob": null, + "nq": 512, + "recall": 1.0, + "qps_1t": 171.8712664963945, + "qps_1t_runs": [ + 173.52706364024056, + 171.8712664963945, + 169.55055494698 + ], + "median_ms": 5.966450000414625, + "p95_ms": 6.443374993978068, + "p99_ms": 7.135140005266295, + "mean_ms": 5.826347982103168 + } + ] + } + } + }, + "faiss_flat|text384|100000": { + "mem_baseline": { + "rss_mb": 37.18359375, + "peak_mb": 37.18359375 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 0.10819310002261773, + "index_s": 2.500019036233425e-06, + "total_s": 0.10819560004165396, + "first_query_ms": 8.38010001461953, + "ready_s": 0.11657570005627349, + "total_s_runs": [ + 0.10819560004165396, + 0.11093499994603917, + 0.10809980001067743 + ] + }, + "mem_after_build": { + "rss_mb": 208.7734375, + "peak_mb": 354.43359375 + }, + "mem_after_first_query": { + "rss_mb": 208.8515625, + "peak_mb": 354.43359375 + }, + "sweep": [ + { + "nq": 410, + "recall_queries": 1000, + "recall": 0.9999, + "qps_1t_runs": [ + 134.87535559755298, + 130.04684857724365, + 131.07833414999007 + ], + "qps_1t": 131.07833414999007, + "qps_1t_min": 130.04684857724365, + "qps_1t_max": 134.87535559755298, + "median_ms": 7.291549991350621, + "p95_ms": 8.84210500807967, + "p99_ms": 14.971388985286474, + "mean_ms": 7.577605121882587, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 5290.98009611987, + 5437.3254450555505, + 5464.146456969809 + ], + "qps_batch": 5437.3254450555505, + "qps_batch_min": 5290.98009611987, + "qps_batch_max": 5464.146456969809, + "threads": 32 + } + } + ], + "loop_allthreads": { + "nq": 592, + "threads": 32, + "qps": 197.76314517220155, + "median_ms": 4.9707500147633255, + "p95_ms": 5.844175000675023, + "p99_ms": 7.938650014693849, + "mean_ms": 5.062252589997517 + }, + "mutation": { + "ref_knob": null, + "recall_before": 0.9999, + "insert_10k": { + "seconds": 0.005019299976993352, + "per_s": 1992309.6937494008, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.22484439995605499, + "per_s": 44475.20152583059, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.19591890001902357, + "per_s": 51041.5278925566, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 273.23046875, + "peak_mb": 528.78515625 + }, + "first_query_after_ms": 9.092299966141582, + "after": { + "nq": 417, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 135.23524022774401, + 135.12611445120822, + 129.72071409544358 + ], + "qps_1t": 135.12611445120822, + "qps_1t_min": 129.72071409544358, + "qps_1t_max": 135.23524022774401, + "median_ms": 7.211500022094697, + "p95_ms": 8.32735002040863, + "p99_ms": 15.417250018799677, + "mean_ms": 7.501295124555452 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 0.12820340000325814, + "disk_mb": 147.24740028381348, + "load_s": 0.0691456000204198, + "first_query_after_load_ms": 7.26719998056069, + "time_to_ready_s": 0.0764128000009805 + }, + "load_before_pct": 8.765624999999998, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 0.10917459987103939, + "mem": { + "rss_mb": 208.296875, + "peak_mb": 353.9453125 + }, + "subsets": { + "f10": [ + { + "knob": null, + "nq": 2000, + "recall": 1.0, + "qps_1t": 464.01631223229913, + "qps_1t_runs": [ + 621.7209465948339, + 464.01631223229913, + 454.2665659081649 + ], + "median_ms": 2.0099000248592347, + "p95_ms": 2.5670049886684865, + "p99_ms": 2.95190300908871, + "mean_ms": 1.9882953165312454 + } + ], + "f01": [ + { + "knob": null, + "nq": 2000, + "recall": 1.0, + "qps_1t": 2490.833729894517, + "qps_1t_runs": [ + 2490.833729894517, + 2629.590836015668, + 2349.4957100081424 + ], + "median_ms": 0.4429499967955053, + "p95_ms": 0.4827000084333122, + "p99_ms": 0.5441209528362383, + "mean_ms": 0.4024608502222691 + } + ] + } + } + }, + "faiss_flat|text384|10000": { + "mem_baseline": { + "rss_mb": 37.26171875, + "peak_mb": 37.26171875 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 0.004413699964061379, + "index_s": 1.200009137392044e-06, + "total_s": 0.0044148999731987715, + "first_query_ms": 0.8730000117793679, + "ready_s": 0.005287899984978139 + }, + "mem_after_build": { + "rss_mb": 69.9921875, + "peak_mb": 84.6484375 + }, + "mem_after_first_query": { + "rss_mb": 70.0703125, + "peak_mb": 84.6484375 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 1.0, + "qps_1t_runs": [ + 2845.028324334414, + 2752.5226171264526, + 2860.3783559199533 + ], + "qps_1t": 2845.028324334414, + "qps_1t_min": 2752.5226171264526, + "qps_1t_max": 2860.3783559199533, + "median_ms": 0.3364000003784895, + "p95_ms": 0.4412049864185974, + "p99_ms": 0.5895130062708634, + "mean_ms": 0.35479918311466463, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 57217.23959163985, + 55693.241558925154, + 50884.62930485946 + ], + "qps_batch": 55693.241558925154, + "qps_batch_min": 50884.62930485946, + "qps_batch_max": 57217.23959163985, + "threads": 32 + } + } + ], + "loop_allthreads": { + "nq": 723, + "threads": 32, + "qps": 292.60018974727126, + "median_ms": 3.4824999747797847, + "p95_ms": 4.258919972926377, + "p99_ms": 5.689883986487994, + "mean_ms": 3.5678609504045404 + }, + "load_before_pct": 4.443749999999998, + "load_contaminated": false + }, + "faiss_hnsw|glove|1000000": { + "mem_baseline": { + "rss_mb": 35.26953125, + "peak_mb": 35.26953125 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 60.1476027003373, + "index_s": 4.700035788118839e-06, + "total_s": 60.14760740037309, + "first_query_ms": 7.465199974831194, + "ready_s": 60.15507260034792 + }, + "mem_after_build": { + "rss_mb": 580.34375, + "peak_mb": 701.4375 + }, + "mem_after_first_query": { + "rss_mb": 581.390625, + "peak_mb": 701.4375 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.4765, + "qps_1t_runs": [ + 22888.21810888945, + 24051.18092754045, + 18722.752523644373 + ], + "qps_1t": 22888.21810888945, + "qps_1t_min": 18722.752523644373, + "qps_1t_max": 24051.18092754045, + "median_ms": 0.04120002267882228, + "p95_ms": 0.06629998388234526, + "p99_ms": 0.08560201153159146, + "mean_ms": 0.04622651693838028, + "knob": 10, + "batch": { + "recall": 0.4765, + "qps_batch_runs": [ + 166476.60621035923, + 139533.26121217277, + 198532.84249354433 + ], + "qps_batch": 166476.60621035923, + "qps_batch_min": 139533.26121217277, + "qps_batch_max": 198532.84249354433, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.56045, + "qps_1t_runs": [ + 17882.401761741836, + 18038.949655426582, + 14705.276824495266 + ], + "qps_1t": 17882.401761741836, + "qps_1t_min": 14705.276824495266, + "qps_1t_max": 18038.949655426582, + "median_ms": 0.0523999915458262, + "p95_ms": 0.08330499404110016, + "p99_ms": 0.11730202008038763, + "mean_ms": 0.059786433392825224, + "knob": 16, + "batch": { + "recall": 0.56045, + "qps_batch_runs": [ + 113286.20646650107, + 136314.06797283728, + 134982.82329708757 + ], + "qps_batch": 134982.82329708757, + "qps_batch_min": 113286.20646650107, + "qps_batch_max": 136314.06797283728, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.62985, + "qps_1t_runs": [ + 13066.77979409951, + 13769.74239294838, + 13430.471511793348 + ], + "qps_1t": 13430.471511793348, + "qps_1t_min": 13066.77979409951, + "qps_1t_max": 13769.74239294838, + "median_ms": 0.07050001295283437, + "p95_ms": 0.10820000898092985, + "p99_ms": 0.1342109747929501, + "mean_ms": 0.07453683307782437, + "knob": 24, + "batch": { + "recall": 0.62985, + "qps_batch_runs": [ + 106855.87264693242, + 88955.70470287952, + 103150.7395173291 + ], + "qps_batch": 103150.7395173291, + "qps_batch_min": 88955.70470287952, + "qps_batch_max": 106855.87264693242, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.67245, + "qps_1t_runs": [ + 11266.590013464034, + 10956.010486430781, + 10905.60107655429 + ], + "qps_1t": 10956.010486430781, + "qps_1t_min": 10905.60107655429, + "qps_1t_max": 11266.590013464034, + "median_ms": 0.08570001227781177, + "p95_ms": 0.1294999965466559, + "p99_ms": 0.15721003233920805, + "mean_ms": 0.09057603365120788, + "knob": 32, + "batch": { + "recall": 0.67245, + "qps_batch_runs": [ + 73210.19383262319, + 76873.40480569516, + 68137.52882005414 + ], + "qps_batch": 73210.19383262319, + "qps_batch_min": 68137.52882005414, + "qps_batch_max": 76873.40480569516, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7281, + "qps_1t_runs": [ + 7702.842789337466, + 8175.124250343606, + 7966.242259985236 + ], + "qps_1t": 7966.242259985236, + "qps_1t_min": 7702.842789337466, + "qps_1t_max": 8175.124250343606, + "median_ms": 0.11789996642619371, + "p95_ms": 0.17869996663648635, + "p99_ms": 0.22449996380601078, + "mean_ms": 0.12589140003547072, + "knob": 48, + "batch": { + "recall": 0.7281, + "qps_batch_runs": [ + 57418.30097382197, + 55036.8334217464, + 60086.22380713183 + ], + "qps_batch": 57418.30097382197, + "qps_batch_min": 55036.8334217464, + "qps_batch_max": 60086.22380713183, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.76575, + "qps_1t_runs": [ + 6303.590397279359, + 6097.133442825387, + 6025.640913166395 + ], + "qps_1t": 6097.133442825387, + "qps_1t_min": 6025.640913166395, + "qps_1t_max": 6303.590397279359, + "median_ms": 0.15070001245476305, + "p95_ms": 0.22401501773856586, + "p99_ms": 0.29450003057718277, + "mean_ms": 0.1628695662948303, + "knob": 64, + "batch": { + "recall": 0.76575, + "qps_batch_runs": [ + 49314.40643675709, + 48203.45717490607, + 45455.88848543174 + ], + "qps_batch": 48203.45717490607, + "qps_batch_min": 45455.88848543174, + "qps_batch_max": 49314.40643675709, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.80925, + "qps_1t_runs": [ + 4593.628113187744, + 4503.60176557051, + 4413.581736931383 + ], + "qps_1t": 4503.60176557051, + "qps_1t_min": 4413.581736931383, + "qps_1t_max": 4593.628113187744, + "median_ms": 0.2125999890267849, + "p95_ms": 0.3007050196174532, + "p99_ms": 0.3912020102143288, + "mean_ms": 0.22210356683353893, + "knob": 96, + "batch": { + "recall": 0.80925, + "qps_batch_runs": [ + 34172.08720355805, + 33341.335251327335, + 33682.96250082914 + ], + "qps_batch": 33682.96250082914, + "qps_batch_min": 33341.335251327335, + "qps_batch_max": 34172.08720355805, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.83405, + "qps_1t_runs": [ + 3280.262503932642, + 3488.773908215054, + 3282.8599139606063 + ], + "qps_1t": 3282.8599139606063, + "qps_1t_min": 3280.262503932642, + "qps_1t_max": 3488.773908215054, + "median_ms": 0.2818999928422272, + "p95_ms": 0.39241501654032623, + "p99_ms": 0.5027200438780715, + "mean_ms": 0.29869991686427966, + "knob": 128, + "batch": { + "recall": 0.83405, + "qps_batch_runs": [ + 28110.891842939778, + 27684.688004903946, + 26828.568132643573 + ], + "qps_batch": 27684.688004903946, + "qps_batch_min": 26828.568132643573, + "qps_batch_max": 28110.891842939778, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8646, + "qps_1t_runs": [ + 2102.5641247943013, + 2296.977350121448, + 2329.227450934512 + ], + "qps_1t": 2296.977350121448, + "qps_1t_min": 2102.5641247943013, + "qps_1t_max": 2329.227450934512, + "median_ms": 0.4186999867670238, + "p95_ms": 0.5511350056622178, + "p99_ms": 0.7170399877941245, + "mean_ms": 0.4467638002242893, + "knob": 192, + "batch": { + "recall": 0.8646, + "qps_batch_runs": [ + 19009.36496026901, + 19025.477009259394, + 18690.662985797964 + ], + "qps_batch": 19009.36496026901, + "qps_batch_min": 18690.662985797964, + "qps_batch_max": 19025.477009259394, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.88495, + "qps_1t_runs": [ + 1796.1655601101736, + 1671.270598800572, + 1782.0778922326388 + ], + "qps_1t": 1782.0778922326388, + "qps_1t_min": 1671.270598800572, + "qps_1t_max": 1796.1655601101736, + "median_ms": 0.5533999647013843, + "p95_ms": 0.7011099834926426, + "p99_ms": 0.9799850109266136, + "mean_ms": 0.5720771337607099, + "knob": 256, + "batch": { + "recall": 0.88495, + "qps_batch_runs": [ + 15758.09434197008, + 13896.754066554366, + 14589.690489100063 + ], + "qps_batch": 14589.690489100063, + "qps_batch_min": 13896.754066554366, + "qps_batch_max": 15758.09434197008, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9119, + "qps_1t_runs": [ + 1151.5785406704854, + 1164.3932271955393, + 1231.6779437558573 + ], + "qps_1t": 1164.3932271955393, + "qps_1t_min": 1151.5785406704854, + "qps_1t_max": 1231.6779437558573, + "median_ms": 0.8033500052988529, + "p95_ms": 1.155210006982088, + "p99_ms": 1.8656600022222882, + "mean_ms": 0.8463634001382161, + "knob": 400, + "batch": { + "recall": 0.9119, + "qps_batch_runs": [ + 10160.851357343035, + 9946.398856004893, + 10451.25924438675 + ], + "qps_batch": 10160.851357343035, + "qps_batch_min": 9946.398856004893, + "qps_batch_max": 10451.25924438675, + "threads": 32 + } + }, + { + "nq": 1892, + "recall_queries": 1892, + "recall": 0.9346723044397462, + "qps_1t_runs": [ + 654.3569622523773, + 649.9139104051824, + 653.4912648642223 + ], + "qps_1t": 653.4912648642223, + "qps_1t_min": 649.9139104051824, + "qps_1t_max": 654.3569622523773, + "median_ms": 1.5218500047922134, + "p95_ms": 1.7911749746417627, + "p99_ms": 2.020375002757646, + "mean_ms": 1.5323751765226032, + "knob": 640, + "batch": { + "recall": 0.935, + "qps_batch_runs": [ + 6535.266421735189, + 6461.448895639118, + 6777.232700617452 + ], + "qps_batch": 6535.266421735189, + "qps_batch_min": 6461.448895639118, + "qps_batch_max": 6777.232700617452, + "threads": 32 + } + } + ], + "mutation": { + "ref_knob": 640, + "recall_before": 0.9346723044397462, + "insert_10k": { + "seconds": 1.7875868999981321, + "per_s": 5594.13363345326, + "batch": 1000 + }, + "update_10k": { + "unsupported": true + }, + "delete_10k": { + "unsupported": true + }, + "mem_after": { + "rss_mb": 643.96484375, + "peak_mb": 701.4375 + } + }, + "saveload": { + "save_s": 0.3067853999673389, + "disk_mb": 524.2364444732666, + "load_s": 0.22791910002706572, + "first_query_after_load_ms": 2.1547000505961478, + "time_to_ready_s": 0.23007380007766187 + }, + "load_before_pct": 14.475000000000016, + "load_contaminated": false + }, + "faiss_hnsw|glove|100000": { + "mem_baseline": { + "rss_mb": 35.1875, + "peak_mb": 35.1875 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 2.2544728000066243, + "index_s": 2.100015990436077e-06, + "total_s": 2.2544749000226147, + "first_query_ms": 2.0253999973647296, + "ready_s": 2.2565003000199795, + "total_s_runs": [ + 2.2544749000226147, + 2.279344000038691, + 2.4062576998840086 + ] + }, + "mem_after_build": { + "rss_mb": 111.03125, + "peak_mb": 146.97265625 + }, + "mem_after_first_query": { + "rss_mb": 111.1484375, + "peak_mb": 146.97265625 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.53775, + "qps_1t_runs": [ + 32428.786327140377, + 34916.6278201828, + 29418.297361020104 + ], + "qps_1t": 32428.786327140377, + "qps_1t_min": 29418.297361020104, + "qps_1t_max": 34916.6278201828, + "median_ms": 0.027799978852272034, + "p95_ms": 0.04289998032618314, + "p99_ms": 0.06301101413555467, + "mean_ms": 0.03115629999471518, + "knob": 10, + "batch": { + "recall": 0.53775, + "qps_batch_runs": [ + 823214.656820855, + 696694.1926771529, + 892817.283993283 + ], + "qps_batch": 823214.656820855, + "qps_batch_min": 696694.1926771529, + "qps_batch_max": 892817.283993283, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6264, + "qps_1t_runs": [ + 25690.95805872654, + 25351.048714172917, + 27102.07049303316 + ], + "qps_1t": 25690.95805872654, + "qps_1t_min": 25351.048714172917, + "qps_1t_max": 27102.07049303316, + "median_ms": 0.036299985367804766, + "p95_ms": 0.053099996875971556, + "p99_ms": 0.06750604778062565, + "mean_ms": 0.03842261675163172, + "knob": 16, + "batch": { + "recall": 0.6264, + "qps_batch_runs": [ + 631273.2845665496, + 400248.1553731371, + 502739.93695657566 + ], + "qps_batch": 502739.93695657566, + "qps_batch_min": 400248.1553731371, + "qps_batch_max": 631273.2845665496, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6956, + "qps_1t_runs": [ + 19218.166529280712, + 20186.075205835587, + 20931.252264240684 + ], + "qps_1t": 20186.075205835587, + "qps_1t_min": 19218.166529280712, + "qps_1t_max": 20931.252264240684, + "median_ms": 0.04740001168102026, + "p95_ms": 0.0678999931551516, + "p99_ms": 0.08822003088425888, + "mean_ms": 0.04978288344379204, + "knob": 24, + "batch": { + "recall": 0.6956, + "qps_batch_runs": [ + 499650.24751489, + 455404.5160487642, + 511299.72618173336 + ], + "qps_batch": 499650.24751489, + "qps_batch_min": 455404.5160487642, + "qps_batch_max": 511299.72618173336, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7423, + "qps_1t_runs": [ + 17280.140673566646, + 15500.487560586498, + 17481.92150026605 + ], + "qps_1t": 17280.140673566646, + "qps_1t_min": 15500.487560586498, + "qps_1t_max": 17481.92150026605, + "median_ms": 0.056699966080486774, + "p95_ms": 0.0792999635450542, + "p99_ms": 0.10830400977283725, + "mean_ms": 0.05986198299797252, + "knob": 32, + "batch": { + "recall": 0.7423, + "qps_batch_runs": [ + 371367.5630120495, + 381766.81971751305, + 420291.68499289773 + ], + "qps_batch": 381766.81971751305, + "qps_batch_min": 371367.5630120495, + "qps_batch_max": 420291.68499289773, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7966, + "qps_1t_runs": [ + 12844.89349042002, + 11435.308453872674, + 11369.644192816184 + ], + "qps_1t": 11435.308453872674, + "qps_1t_min": 11369.644192816184, + "qps_1t_max": 12844.89349042002, + "median_ms": 0.0784999574534595, + "p95_ms": 0.10840001050382853, + "p99_ms": 0.17650300520472234, + "mean_ms": 0.08441796651459299, + "knob": 48, + "batch": { + "recall": 0.7966, + "qps_batch_runs": [ + 227409.68919318425, + 257688.78866515282, + 263862.68677815323 + ], + "qps_batch": 257688.78866515282, + "qps_batch_min": 227409.68919318425, + "qps_batch_max": 263862.68677815323, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8334, + "qps_1t_runs": [ + 9750.679962644133, + 8971.128209204435, + 9621.194343860849 + ], + "qps_1t": 9621.194343860849, + "qps_1t_min": 8971.128209204435, + "qps_1t_max": 9750.679962644133, + "median_ms": 0.10040000779554248, + "p95_ms": 0.1369049947243184, + "p99_ms": 0.23951401934027702, + "mean_ms": 0.10598761676616657, + "knob": 64, + "batch": { + "recall": 0.8334, + "qps_batch_runs": [ + 227445.89584921583, + 173130.19461554143, + 211012.75586735076 + ], + "qps_batch": 211012.75586735076, + "qps_batch_min": 173130.19461554143, + "qps_batch_max": 227445.89584921583, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8744, + "qps_1t_runs": [ + 4882.339302033936, + 6711.634618795114, + 6389.811079287228 + ], + "qps_1t": 6389.811079287228, + "qps_1t_min": 4882.339302033936, + "qps_1t_max": 6711.634618795114, + "median_ms": 0.14860002556815743, + "p95_ms": 0.20760000916197896, + "p99_ms": 0.35470302274916327, + "mean_ms": 0.17010466629290022, + "knob": 96, + "batch": { + "recall": 0.8744, + "qps_batch_runs": [ + 120765.65448402686, + 109461.28621737093, + 142056.55227585943 + ], + "qps_batch": 120765.65448402686, + "qps_batch_min": 109461.28621737093, + "qps_batch_max": 142056.55227585943, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.90235, + "qps_1t_runs": [ + 4219.419088777758, + 4629.238495423772, + 4853.100287566249 + ], + "qps_1t": 4629.238495423772, + "qps_1t_min": 4219.419088777758, + "qps_1t_max": 4853.100287566249, + "median_ms": 0.2001000102609396, + "p95_ms": 0.2741049916949123, + "p99_ms": 0.4297169542405758, + "mean_ms": 0.21969051659107208, + "knob": 128, + "batch": { + "recall": 0.90235, + "qps_batch_runs": [ + 92810.87001407248, + 111843.68716628088, + 72496.86452998761 + ], + "qps_batch": 92810.87001407248, + "qps_batch_min": 72496.86452998761, + "qps_batch_max": 111843.68716628088, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.93055, + "qps_1t_runs": [ + 3242.6730589140698, + 3075.502672615211, + 3206.340859890137 + ], + "qps_1t": 3206.340859890137, + "qps_1t_min": 3075.502672615211, + "qps_1t_max": 3242.6730589140698, + "median_ms": 0.29639998683705926, + "p95_ms": 0.40111502457875764, + "p99_ms": 0.8328649960458314, + "mean_ms": 0.31513988314933766, + "knob": 192, + "batch": { + "recall": 0.93055, + "qps_batch_runs": [ + 81515.53693931492, + 62786.46323704987, + 66248.1988426525 + ], + "qps_batch": 66248.1988426525, + "qps_batch_min": 62786.46323704987, + "qps_batch_max": 81515.53693931492, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9483, + "qps_1t_runs": [ + 2236.6380989326058, + 2249.8331771518483, + 2230.111117860299 + ], + "qps_1t": 2236.6380989326058, + "qps_1t_min": 2230.111117860299, + "qps_1t_max": 2249.8331771518483, + "median_ms": 0.4082999948877841, + "p95_ms": 0.5389650264987724, + "p99_ms": 1.0940319456858654, + "mean_ms": 0.44666171651139547, + "knob": 256, + "batch": { + "recall": 0.9483, + "qps_batch_runs": [ + 59144.82501820361, + 54004.13666035046, + 53987.372339161186 + ], + "qps_batch": 54004.13666035046, + "qps_batch_min": 53987.372339161186, + "qps_batch_max": 59144.82501820361, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9696, + "qps_1t_runs": [ + 1378.5129920775269, + 1380.774727899608, + 1435.2644762819937 + ], + "qps_1t": 1380.774727899608, + "qps_1t_min": 1378.5129920775269, + "qps_1t_max": 1435.2644762819937, + "median_ms": 0.6793999928049743, + "p95_ms": 0.9083049895707518, + "p99_ms": 1.7064569529611695, + "mean_ms": 0.7154620504103756, + "knob": 400, + "batch": { + "recall": 0.9696, + "qps_batch_runs": [ + 34420.268027078215, + 27177.236349409926, + 25610.621241390967 + ], + "qps_batch": 27177.236349409926, + "qps_batch_min": 25610.621241390967, + "qps_batch_max": 34420.268027078215, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.98425, + "qps_1t_runs": [ + 780.2768788632924, + 784.8356567527835, + 788.6883477062906 + ], + "qps_1t": 784.8356567527835, + "qps_1t_min": 780.2768788632924, + "qps_1t_max": 788.6883477062906, + "median_ms": 1.2322499824222177, + "p95_ms": 1.575945006334223, + "p99_ms": 2.5888749805744755, + "mean_ms": 1.2745587999816053, + "knob": 640, + "batch": { + "recall": 0.98425, + "qps_batch_runs": [ + 17514.714550991266, + 15584.80824323615, + 17015.78810035338 + ], + "qps_batch": 17015.78810035338, + "qps_batch_min": 15584.80824323615, + "qps_batch_max": 17514.714550991266, + "threads": 32 + } + } + ], + "mutation": { + "ref_knob": 400, + "recall_before": 0.9696, + "insert_10k": { + "seconds": 0.988941400020849, + "per_s": 10111.822601206886, + "batch": 1000 + }, + "update_10k": { + "unsupported": true + }, + "delete_10k": { + "unsupported": true + }, + "mem_after": { + "rss_mb": 129.8359375, + "peak_mb": 204.36328125 + } + }, + "saveload": { + "save_s": 0.052062700036913157, + "disk_mb": 57.1041202545166, + "load_s": 0.02949290000833571, + "first_query_after_load_ms": 1.1775000020861626, + "time_to_ready_s": 0.030670400010421872 + }, + "load_before_pct": 7.721874999999998, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 3.84950030001346, + "mem": { + "rss_mb": 110.53125, + "peak_mb": 146.8046875 + }, + "subsets": { + "f10": [ + { + "knob": 16, + "nq": 2000, + "recall": 0.42795, + "qps_1t": 15136.722441572838, + "qps_1t_runs": [ + 14725.924809037962, + 15819.455791392835, + 15136.722441572838 + ], + "median_ms": 0.062400009483098984, + "p95_ms": 0.09089999366551638, + "p99_ms": 0.11690201703459029, + "mean_ms": 0.06572841651116808 + }, + { + "knob": 48, + "nq": 2000, + "recall": 0.63885, + "qps_1t": 7889.832685214196, + "qps_1t_runs": [ + 7929.5066659141485, + 7538.149602062232, + 7889.832685214196 + ], + "median_ms": 0.12579996837303042, + "p95_ms": 0.17231002857442948, + "p99_ms": 0.21080103295389566, + "mean_ms": 0.12850506696850061 + }, + { + "knob": 96, + "nq": 2000, + "recall": 0.75445, + "qps_1t": 4396.9921076126675, + "qps_1t_runs": [ + 4314.771214582915, + 4407.980642087073, + 4396.9921076126675 + ], + "median_ms": 0.2294000005349517, + "p95_ms": 0.29679999279323965, + "p99_ms": 0.349700985243544, + "mean_ms": 0.2286838162690401 + }, + { + "knob": 192, + "nq": 2000, + "recall": 0.8501, + "qps_1t": 2198.3355960583854, + "qps_1t_runs": [ + 2209.443777625927, + 2198.30756849026, + 2198.3355960583854 + ], + "median_ms": 0.46309997560456395, + "p95_ms": 0.575800018850714, + "p99_ms": 0.6611119705485182, + "mean_ms": 0.4541292001570885 + }, + { + "knob": 640, + "nq": 1872, + "recall": 0.9574786324786325, + "qps_1t": 533.7559736979956, + "qps_1t_runs": [ + 544.5763527583243, + 519.7980729753436, + 533.7559736979956 + ], + "median_ms": 1.9596499914769083, + "p95_ms": 2.2726000170223415, + "p99_ms": 2.4317800067365196, + "mean_ms": 1.877876370913081 + } + ], + "f01": [ + { + "knob": 16, + "nq": 2000, + "recall": 0.15125, + "qps_1t": 17209.20111428282, + "qps_1t_runs": [ + 18250.0069504321, + 15523.575273736153, + 17209.20111428282 + ], + "median_ms": 0.057299970649182796, + "p95_ms": 0.08540000999346375, + "p99_ms": 0.10970202041789893, + "mean_ms": 0.059107033167189606 + }, + { + "knob": 48, + "nq": 2000, + "recall": 0.29555, + "qps_1t": 8272.51313010353, + "qps_1t_runs": [ + 8272.51313010353, + 7559.8903719100435, + 8736.27203433525 + ], + "median_ms": 0.12099999003112316, + "p95_ms": 0.17010501469485464, + "p99_ms": 0.21370299742557114, + "mean_ms": 0.12254153349203989 + }, + { + "knob": 96, + "nq": 2000, + "recall": 0.42335, + "qps_1t": 4567.178743169348, + "qps_1t_runs": [ + 4610.583226204444, + 4567.178743169348, + 4124.817809605718 + ], + "median_ms": 0.2243000199086964, + "p95_ms": 0.29360502085182816, + "p99_ms": 0.35770098795183003, + "mean_ms": 0.22609360007724413 + }, + { + "knob": 192, + "nq": 2000, + "recall": 0.583, + "qps_1t": 2097.7287790037176, + "qps_1t_runs": [ + 2082.618519497406, + 2097.7287790037176, + 2167.555508147719 + ], + "median_ms": 0.4835000145249069, + "p95_ms": 0.5848099710419774, + "p99_ms": 0.6378999684238806, + "mean_ms": 0.47273999990041676 + }, + { + "knob": 640, + "nq": 1501, + "recall": 0.835176548967355, + "qps_1t": 541.4554097121783, + "qps_1t_runs": [ + 541.4554097121783, + 537.8322333921916, + 547.5736231871928 + ], + "median_ms": 1.9076999742537737, + "p95_ms": 2.2440199798438694, + "p99_ms": 2.4566780216991897, + "mean_ms": 1.8441428163266933 + } + ] + } + } + }, + "faiss_hnsw|sift|1000000": { + "mem_baseline": { + "rss_mb": 35.41015625, + "peak_mb": 35.41015625 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 99.66364149999572, + "index_s": 4.800036549568176e-06, + "total_s": 99.66364630003227, + "first_query_ms": 4.775700042955577, + "ready_s": 99.66842200007522 + }, + "mem_after_build": { + "rss_mb": 698.6015625, + "peak_mb": 849.3046875 + }, + "mem_after_first_query": { + "rss_mb": 698.69921875, + "peak_mb": 849.3046875 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6877, + "qps_1t_runs": [ + 15221.60368264573, + 17411.713858712134, + 17554.26899415655 + ], + "qps_1t": 17411.713858712134, + "qps_1t_min": 15221.60368264573, + "qps_1t_max": 17554.26899415655, + "median_ms": 0.05490001058205962, + "p95_ms": 0.09160499903373422, + "p99_ms": 0.16472397372126632, + "mean_ms": 0.06003163346516279, + "knob": 10, + "batch": { + "recall": 0.6877, + "qps_batch_runs": [ + 135452.71655586155, + 139613.13202342496, + 133927.9732603827 + ], + "qps_batch": 135452.71655586155, + "qps_batch_min": 133927.9732603827, + "qps_batch_max": 139613.13202342496, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.78015, + "qps_1t_runs": [ + 12996.523927629638, + 13849.993583143503, + 13831.392592896178 + ], + "qps_1t": 13831.392592896178, + "qps_1t_min": 12996.523927629638, + "qps_1t_max": 13849.993583143503, + "median_ms": 0.06969997775740921, + "p95_ms": 0.10400003520771861, + "p99_ms": 0.2012190181994815, + "mean_ms": 0.073815050078944, + "knob": 16, + "batch": { + "recall": 0.78015, + "qps_batch_runs": [ + 85890.36092934318, + 93999.53931667774, + 92819.915264173 + ], + "qps_batch": 92819.915264173, + "qps_batch_min": 85890.36092934318, + "qps_batch_max": 93999.53931667774, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8484, + "qps_1t_runs": [ + 11082.579713142077, + 12039.917093270498, + 11228.347557247738 + ], + "qps_1t": 11228.347557247738, + "qps_1t_min": 11082.579713142077, + "qps_1t_max": 12039.917093270498, + "median_ms": 0.08460000390186906, + "p95_ms": 0.1236050098668784, + "p99_ms": 0.1687040040269495, + "mean_ms": 0.08744968314810346, + "knob": 24, + "batch": { + "recall": 0.8484, + "qps_batch_runs": [ + 75110.88255893446, + 72881.78213417878, + 68304.84447250169 + ], + "qps_batch": 72881.78213417878, + "qps_batch_min": 68304.84447250169, + "qps_batch_max": 75110.88255893446, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.88915, + "qps_1t_runs": [ + 8957.322889695202, + 9247.622692760386, + 9049.540365327706 + ], + "qps_1t": 9049.540365327706, + "qps_1t_min": 8957.322889695202, + "qps_1t_max": 9247.622692760386, + "median_ms": 0.10679999832063913, + "p95_ms": 0.15270499861799183, + "p99_ms": 0.2480120252585041, + "mean_ms": 0.1100930829513042, + "knob": 32, + "batch": { + "recall": 0.88915, + "qps_batch_runs": [ + 61178.35637065269, + 57189.58922444197, + 60222.09905381038 + ], + "qps_batch": 60222.09905381038, + "qps_batch_min": 57189.58922444197, + "qps_batch_max": 61178.35637065269, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.93395, + "qps_1t_runs": [ + 6490.191817015184, + 6449.413851717736, + 7112.552572912351 + ], + "qps_1t": 6490.191817015184, + "qps_1t_min": 6449.413851717736, + "qps_1t_max": 7112.552572912351, + "median_ms": 0.1482999650761485, + "p95_ms": 0.2014049649005756, + "p99_ms": 0.2750010561430827, + "mean_ms": 0.14990933384979144, + "knob": 48, + "batch": { + "recall": 0.93395, + "qps_batch_runs": [ + 44632.79484573443, + 42181.10030571881, + 45456.92163567897 + ], + "qps_batch": 44632.79484573443, + "qps_batch_min": 42181.10030571881, + "qps_batch_max": 45456.92163567897, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.95725, + "qps_1t_runs": [ + 5242.815771186856, + 5346.925925907517, + 5201.269644431854 + ], + "qps_1t": 5242.815771186856, + "qps_1t_min": 5201.269644431854, + "qps_1t_max": 5346.925925907517, + "median_ms": 0.18960004672408104, + "p95_ms": 0.25480001932010055, + "p99_ms": 0.3242010233225301, + "mean_ms": 0.19000709987206696, + "knob": 64, + "batch": { + "recall": 0.95725, + "qps_batch_runs": [ + 34094.899722281334, + 34115.487748806205, + 32618.818094757655 + ], + "qps_batch": 34094.899722281334, + "qps_batch_min": 32618.818094757655, + "qps_batch_max": 34115.487748806205, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9778, + "qps_1t_runs": [ + 3677.5474920151423, + 3715.6895513562404, + 4040.752599524377 + ], + "qps_1t": 3715.6895513562404, + "qps_1t_min": 3677.5474920151423, + "qps_1t_max": 4040.752599524377, + "median_ms": 0.2654999843798578, + "p95_ms": 0.3440100001171232, + "p99_ms": 0.41020498028956365, + "mean_ms": 0.26284268345140543, + "knob": 96, + "batch": { + "recall": 0.9778, + "qps_batch_runs": [ + 26176.94830262621, + 24544.213953662176, + 22960.030030076927 + ], + "qps_batch": 24544.213953662176, + "qps_batch_min": 22960.030030076927, + "qps_batch_max": 26176.94830262621, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.98655, + "qps_1t_runs": [ + 2680.3439188855923, + 2621.8817786382783, + 2367.6991835381095 + ], + "qps_1t": 2621.8817786382783, + "qps_1t_min": 2367.6991835381095, + "qps_1t_max": 2680.3439188855923, + "median_ms": 0.38979999953880906, + "p95_ms": 0.5123100010678172, + "p99_ms": 0.6094059837050737, + "mean_ms": 0.39228095043411787, + "knob": 128, + "batch": { + "recall": 0.98655, + "qps_batch_runs": [ + 15782.6165510127, + 16773.317006425434, + 16650.155266836067 + ], + "qps_batch": 16650.155266836067, + "qps_batch_min": 15782.6165510127, + "qps_batch_max": 16773.317006425434, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9936, + "qps_1t_runs": [ + 1707.4400499424014, + 1789.5660063661223, + 1799.9969418573462 + ], + "qps_1t": 1789.5660063661223, + "qps_1t_min": 1707.4400499424014, + "qps_1t_max": 1799.9969418573462, + "median_ms": 0.5837000207975507, + "p95_ms": 0.7234250340843574, + "p99_ms": 0.8352020144229755, + "mean_ms": 0.5666744332702365, + "knob": 192, + "batch": { + "recall": 0.9936, + "qps_batch_runs": [ + 12020.162620381148, + 11858.471514675508, + 11390.248125827971 + ], + "qps_batch": 11858.471514675508, + "qps_batch_min": 11390.248125827971, + "qps_batch_max": 12020.162620381148, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99625, + "qps_1t_runs": [ + 1364.3275830496264, + 1384.428817197928, + 1414.5979295631594 + ], + "qps_1t": 1384.428817197928, + "qps_1t_min": 1364.3275830496264, + "qps_1t_max": 1414.5979295631594, + "median_ms": 0.742000003810972, + "p95_ms": 0.9175000013783574, + "p99_ms": 1.0287189751397823, + "mean_ms": 0.7207319996183893, + "knob": 256, + "batch": { + "recall": 0.99625, + "qps_batch_runs": [ + 8889.177294124036, + 8639.61114871948, + 8866.152134119544 + ], + "qps_batch": 8866.152134119544, + "qps_batch_min": 8639.61114871948, + "qps_batch_max": 8889.177294124036, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9984, + "qps_1t_runs": [ + 809.4912201268561, + 836.4593911104172, + 802.845927782934 + ], + "qps_1t": 809.4912201268561, + "qps_1t_min": 802.845927782934, + "qps_1t_max": 836.4593911104172, + "median_ms": 1.2682999949902296, + "p95_ms": 1.4987050031777471, + "p99_ms": 1.6465010220417753, + "mean_ms": 1.2254760501091368, + "knob": 400, + "batch": { + "recall": 0.9984, + "qps_batch_runs": [ + 5540.9536037766475, + 5685.58164945011, + 6897.1605779151405 + ], + "qps_batch": 5685.58164945011, + "qps_batch_min": 5540.9536037766475, + "qps_batch_max": 6897.1605779151405, + "threads": 32 + } + }, + { + "nq": 1572, + "recall_queries": 1572, + "recall": 0.9989821882951654, + "qps_1t_runs": [ + 482.46921135496234, + 482.44073784558555, + 488.31076602188097 + ], + "qps_1t": 482.46921135496234, + "qps_1t_min": 482.44073784558555, + "qps_1t_max": 488.31076602188097, + "median_ms": 2.1400999976322055, + "p95_ms": 2.510875026928261, + "p99_ms": 2.6561249687802047, + "mean_ms": 2.0644469246648622, + "knob": 640, + "batch": { + "recall": 0.99905, + "qps_batch_runs": [ + 3593.852284451932, + 3978.1543630467136, + 4054.1874583677513 + ], + "qps_batch": 3978.1543630467136, + "qps_batch_min": 3593.852284451932, + "qps_batch_max": 4054.1874583677513, + "threads": 32 + } + } + ], + "saveload": { + "save_s": 1.122972299985122, + "disk_mb": 625.8567447662354, + "load_s": 1.0980615000007674, + "first_query_after_load_ms": 0.566599948797375, + "time_to_ready_s": 1.0986280999495648 + }, + "load_before_pct": 8.231249999999996, + "load_contaminated": false + }, + "faiss_hnsw|sift|100000": { + "mem_baseline": { + "rss_mb": 35.94921875, + "peak_mb": 35.94921875 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 1.9758728001033887, + "index_s": 3.300025127828121e-06, + "total_s": 1.9758761001285166, + "first_query_ms": 1.9087999826297164, + "ready_s": 1.9777849001111463, + "total_s_runs": [ + 1.9758761001285166, + 1.9990646000951529, + 2.0857574000256136 + ] + }, + "mem_after_build": { + "rss_mb": 121.0078125, + "peak_mb": 167.9765625 + }, + "mem_after_first_query": { + "rss_mb": 121.20703125, + "peak_mb": 167.9765625 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7926, + "qps_1t_runs": [ + 35531.23656587686, + 35472.44006732308, + 34556.57872236042 + ], + "qps_1t": 35472.44006732308, + "qps_1t_min": 34556.57872236042, + "qps_1t_max": 35531.23656587686, + "median_ms": 0.026799971237778664, + "p95_ms": 0.03809999907389283, + "p99_ms": 0.052900995360687396, + "mean_ms": 0.028424399827296536, + "knob": 10, + "batch": { + "recall": 0.7926, + "qps_batch_runs": [ + 579256.806981008, + 665335.991676808, + 480746.1164904281 + ], + "qps_batch": 579256.806981008, + "qps_batch_min": 480746.1164904281, + "qps_batch_max": 665335.991676808, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.87365, + "qps_1t_runs": [ + 28188.626704953174, + 27241.980895966684, + 28108.837376632495 + ], + "qps_1t": 28108.837376632495, + "qps_1t_min": 27241.980895966684, + "qps_1t_max": 28188.626704953174, + "median_ms": 0.03430002834647894, + "p95_ms": 0.047900015488266945, + "p99_ms": 0.061102999607101145, + "mean_ms": 0.03591978352051228, + "knob": 16, + "batch": { + "recall": 0.87365, + "qps_batch_runs": [ + 407672.3955752733, + 396518.56779264717, + 484954.28964742157 + ], + "qps_batch": 407672.3955752733, + "qps_batch_min": 396518.56779264717, + "qps_batch_max": 484954.28964742157, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.92575, + "qps_1t_runs": [ + 21336.997793962426, + 21006.527781138193, + 21197.331364720452 + ], + "qps_1t": 21197.331364720452, + "qps_1t_min": 21006.527781138193, + "qps_1t_max": 21336.997793962426, + "median_ms": 0.04529999569058418, + "p95_ms": 0.06290001329034567, + "p99_ms": 0.08430500165559363, + "mean_ms": 0.04721564978050689, + "knob": 24, + "batch": { + "recall": 0.92575, + "qps_batch_runs": [ + 376208.5723213932, + 289775.27956675127, + 354905.68312752625 + ], + "qps_batch": 354905.68312752625, + "qps_batch_min": 289775.27956675127, + "qps_batch_max": 376208.5723213932, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.95325, + "qps_1t_runs": [ + 17367.88044096626, + 16936.98087835346, + 16705.911637072553 + ], + "qps_1t": 16936.98087835346, + "qps_1t_min": 16705.911637072553, + "qps_1t_max": 17367.88044096626, + "median_ms": 0.055500015150755644, + "p95_ms": 0.07790501113049687, + "p99_ms": 0.12070598779246224, + "mean_ms": 0.05882633325139371, + "knob": 32, + "batch": { + "recall": 0.95325, + "qps_batch_runs": [ + 338833.7337702731, + 328882.4557782537, + 194992.5905542895 + ], + "qps_batch": 328882.4557782537, + "qps_batch_min": 194992.5905542895, + "qps_batch_max": 338833.7337702731, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.97775, + "qps_1t_runs": [ + 12452.920234205405, + 12092.541804703415, + 12677.283845810938 + ], + "qps_1t": 12452.920234205405, + "qps_1t_min": 12092.541804703415, + "qps_1t_max": 12677.283845810938, + "median_ms": 0.07830001413822174, + "p95_ms": 0.10891001438722019, + "p99_ms": 0.14870496874209505, + "mean_ms": 0.08062643330777064, + "knob": 48, + "batch": { + "recall": 0.97775, + "qps_batch_runs": [ + 182715.1471712587, + 214167.15796670993, + 191734.33231475647 + ], + "qps_batch": 191734.33231475647, + "qps_batch_min": 182715.1471712587, + "qps_batch_max": 214167.15796670993, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.98885, + "qps_1t_runs": [ + 9172.05680832665, + 9527.325560037729, + 7962.528332573384 + ], + "qps_1t": 9172.05680832665, + "qps_1t_min": 7962.528332573384, + "qps_1t_max": 9527.325560037729, + "median_ms": 0.10090001160278916, + "p95_ms": 0.14780501951463523, + "p99_ms": 0.2626010205131024, + "mean_ms": 0.11319209993234836, + "knob": 64, + "batch": { + "recall": 0.98885, + "qps_batch_runs": [ + 140577.7751604037, + 153848.52034566354, + 126686.51459788925 + ], + "qps_batch": 140577.7751604037, + "qps_batch_min": 126686.51459788925, + "qps_batch_max": 153848.52034566354, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99575, + "qps_1t_runs": [ + 5876.443678501833, + 6013.169443623567, + 6247.579067269414 + ], + "qps_1t": 6013.169443623567, + "qps_1t_min": 5876.443678501833, + "qps_1t_max": 6247.579067269414, + "median_ms": 0.14990003546699882, + "p95_ms": 0.22800998995080593, + "p99_ms": 0.40872402663808366, + "mean_ms": 0.16551153309410438, + "knob": 96, + "batch": { + "recall": 0.99575, + "qps_batch_runs": [ + 98180.22949951982, + 108692.69854053286, + 99527.74065783831 + ], + "qps_batch": 99527.74065783831, + "qps_batch_min": 98180.22949951982, + "qps_batch_max": 108692.69854053286, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99815, + "qps_1t_runs": [ + 4910.015393694244, + 4996.54362960391, + 5057.248053808577 + ], + "qps_1t": 4996.54362960391, + "qps_1t_min": 4910.015393694244, + "qps_1t_max": 5057.248053808577, + "median_ms": 0.19130000146105886, + "p95_ms": 0.26441003137733793, + "p99_ms": 0.39741199929267196, + "mean_ms": 0.20051323315904787, + "knob": 128, + "batch": { + "recall": 0.99815, + "qps_batch_runs": [ + 90957.87754942862, + 90236.01242785207, + 94692.48616042655 + ], + "qps_batch": 90957.87754942862, + "qps_batch_min": 90236.01242785207, + "qps_batch_max": 94692.48616042655, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9992, + "qps_1t_runs": [ + 3317.5286232630474, + 3279.131472997281, + 3385.8346492080973 + ], + "qps_1t": 3317.5286232630474, + "qps_1t_min": 3279.131472997281, + "qps_1t_max": 3385.8346492080973, + "median_ms": 0.2888999879360199, + "p95_ms": 0.39281002245843416, + "p99_ms": 0.5657670000800877, + "mean_ms": 0.30057871689011034, + "knob": 192, + "batch": { + "recall": 0.9992, + "qps_batch_runs": [ + 63597.04910850862, + 70887.54756771697, + 70792.95189234504 + ], + "qps_batch": 70792.95189234504, + "qps_batch_min": 63597.04910850862, + "qps_batch_max": 70887.54756771697, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99965, + "qps_1t_runs": [ + 2662.9708237718537, + 2528.408561910027, + 2456.427580322251 + ], + "qps_1t": 2528.408561910027, + "qps_1t_min": 2456.427580322251, + "qps_1t_max": 2662.9708237718537, + "median_ms": 0.3800999838858843, + "p95_ms": 0.5081150273326784, + "p99_ms": 0.8009799866704288, + "mean_ms": 0.3927071333261362, + "knob": 256, + "batch": { + "recall": 0.99965, + "qps_batch_runs": [ + 53417.09134018126, + 44524.58869696301, + 48638.48714060023 + ], + "qps_batch": 48638.48714060023, + "qps_batch_min": 44524.58869696301, + "qps_batch_max": 53417.09134018126, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9999, + "qps_1t_runs": [ + 1537.3599611503926, + 1490.9581216083075, + 1551.6797010265384 + ], + "qps_1t": 1537.3599611503926, + "qps_1t_min": 1490.9581216083075, + "qps_1t_max": 1551.6797010265384, + "median_ms": 0.6274500046856701, + "p95_ms": 0.8356150181498379, + "p99_ms": 1.329462994472125, + "mean_ms": 0.6552127666557984, + "knob": 400, + "batch": { + "recall": 0.9999, + "qps_batch_runs": [ + 31586.24545067845, + 29965.958671634555, + 26070.45280001906 + ], + "qps_batch": 29965.958671634555, + "qps_batch_min": 26070.45280001906, + "qps_batch_max": 31586.24545067845, + "threads": 32 + } + } + ], + "mutation": { + "ref_knob": 32, + "recall_before": 0.95325, + "insert_10k": { + "seconds": 0.9947571000084281, + "per_s": 10052.705328683027, + "batch": 1000 + }, + "update_10k": { + "unsupported": true + }, + "delete_10k": { + "unsupported": true + }, + "mem_after": { + "rss_mb": 143.51953125, + "peak_mb": 235.96875 + } + }, + "saveload": { + "save_s": 0.043917100003454834, + "disk_mb": 68.8533878326416, + "load_s": 0.039114899991545826, + "first_query_after_load_ms": 0.340400030836463, + "time_to_ready_s": 0.03945530002238229 + }, + "load_before_pct": 8.315625000000004, + "load_contaminated": false + }, + "faiss_hnsw|text384|1000000": { + "mem_baseline": { + "rss_mb": 37.30078125, + "peak_mb": 37.30078125 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 220.9135291997809, + "index_s": 5.800044164061546e-06, + "total_s": 220.91353499982506, + "first_query_ms": 3.805199987255037, + "ready_s": 220.91734019981232 + }, + "mem_after_build": { + "rss_mb": 1677.4140625, + "peak_mb": 2205.703125 + }, + "mem_after_first_query": { + "rss_mb": 1677.51171875, + "peak_mb": 2205.703125 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.56475, + "qps_1t_runs": [ + 10192.261730123666, + 11088.195530166031, + 11826.237086049165 + ], + "qps_1t": 11088.195530166031, + "qps_1t_min": 10192.261730123666, + "qps_1t_max": 11826.237086049165, + "median_ms": 0.08379999781027436, + "p95_ms": 0.1366999960737303, + "p99_ms": 0.17521299538202612, + "mean_ms": 0.09095246663006644, + "knob": 10, + "batch": { + "recall": 0.56475, + "qps_batch_runs": [ + 44237.70688109123, + 44630.40445190999, + 44533.11482424754 + ], + "qps_batch": 44533.11482424754, + "qps_batch_min": 44237.70688109123, + "qps_batch_max": 44630.40445190999, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.67145, + "qps_1t_runs": [ + 8336.70622747123, + 8861.622680781658, + 8785.690236420953 + ], + "qps_1t": 8785.690236420953, + "qps_1t_min": 8336.70622747123, + "qps_1t_max": 8861.622680781658, + "median_ms": 0.10959996143355966, + "p95_ms": 0.16750499489717188, + "p99_ms": 0.2025009709177539, + "mean_ms": 0.11553968316487347, + "knob": 16, + "batch": { + "recall": 0.67145, + "qps_batch_runs": [ + 31232.57808054674, + 30551.749328506827, + 31149.97938836586 + ], + "qps_batch": 31149.97938836586, + "qps_batch_min": 30551.749328506827, + "qps_batch_max": 31232.57808054674, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7482, + "qps_1t_runs": [ + 6565.699329251801, + 5968.191327827954, + 6666.446668629778 + ], + "qps_1t": 6565.699329251801, + "qps_1t_min": 5968.191327827954, + "qps_1t_max": 6666.446668629778, + "median_ms": 0.1463499793317169, + "p95_ms": 0.224510018597357, + "p99_ms": 0.28600202349480247, + "mean_ms": 0.1566222001662633, + "knob": 24, + "batch": { + "recall": 0.7482, + "qps_batch_runs": [ + 24787.81629498942, + 23401.392395765477, + 24366.649848515193 + ], + "qps_batch": 24366.649848515193, + "qps_batch_min": 23401.392395765477, + "qps_batch_max": 24787.81629498942, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.79355, + "qps_1t_runs": [ + 5567.81554088785, + 5568.8790942069445, + 5413.383555844972 + ], + "qps_1t": 5567.81554088785, + "qps_1t_min": 5413.383555844972, + "qps_1t_max": 5568.8790942069445, + "median_ms": 0.17315000877715647, + "p95_ms": 0.24710501602385196, + "p99_ms": 0.3212120011448863, + "mean_ms": 0.18130011716857553, + "knob": 32, + "batch": { + "recall": 0.79355, + "qps_batch_runs": [ + 19412.631989268186, + 19416.702834836407, + 19348.759455371794 + ], + "qps_batch": 19412.631989268186, + "qps_batch_min": 19348.759455371794, + "qps_batch_max": 19416.702834836407, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8516, + "qps_1t_runs": [ + 3891.822891329091, + 4031.562294452521, + 4186.749983768726 + ], + "qps_1t": 4031.562294452521, + "qps_1t_min": 3891.822891329091, + "qps_1t_max": 4186.749983768726, + "median_ms": 0.23800000781193376, + "p95_ms": 0.33120496082119644, + "p99_ms": 0.40860502573195856, + "mean_ms": 0.24794684998535862, + "knob": 48, + "batch": { + "recall": 0.8516, + "qps_batch_runs": [ + 14630.941807717025, + 14764.048361847717, + 14767.907569448676 + ], + "qps_batch": 14764.048361847717, + "qps_batch_min": 14630.941807717025, + "qps_batch_max": 14767.907569448676, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8875, + "qps_1t_runs": [ + 3259.3328942954463, + 3139.5611861162934, + 3290.2773097408053 + ], + "qps_1t": 3259.3328942954463, + "qps_1t_min": 3139.5611861162934, + "qps_1t_max": 3290.2773097408053, + "median_ms": 0.29880000511184335, + "p95_ms": 0.4010000266134739, + "p99_ms": 0.5098089820239695, + "mean_ms": 0.3097509501676541, + "knob": 64, + "batch": { + "recall": 0.8875, + "qps_batch_runs": [ + 11896.42732243021, + 11597.255158434835, + 11569.588181406783 + ], + "qps_batch": 11597.255158434835, + "qps_batch_min": 11569.588181406783, + "qps_batch_max": 11896.42732243021, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9255, + "qps_1t_runs": [ + 2124.4096816103493, + 2238.0321778978687, + 2263.3523633250843 + ], + "qps_1t": 2238.0321778978687, + "qps_1t_min": 2124.4096816103493, + "qps_1t_max": 2263.3523633250843, + "median_ms": 0.42630001553334296, + "p95_ms": 0.5663250049110503, + "p99_ms": 0.7917200319934663, + "mean_ms": 0.453120866654596, + "knob": 96, + "batch": { + "recall": 0.9255, + "qps_batch_runs": [ + 8492.082195002055, + 8358.869680357435, + 7487.19876206358 + ], + "qps_batch": 8358.869680357435, + "qps_batch_min": 7487.19876206358, + "qps_batch_max": 8492.082195002055, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9445, + "qps_1t_runs": [ + 1665.2790724754075, + 1742.5921323283403, + 1720.937430948472 + ], + "qps_1t": 1720.937430948472, + "qps_1t_min": 1665.2790724754075, + "qps_1t_max": 1742.5921323283403, + "median_ms": 0.5580500001087785, + "p95_ms": 0.7169050426455215, + "p99_ms": 0.9819570259423939, + "mean_ms": 0.5851454501777577, + "knob": 128, + "batch": { + "recall": 0.9445, + "qps_batch_runs": [ + 6658.22847238178, + 6312.125118810746, + 6399.639573130844 + ], + "qps_batch": 6399.639573130844, + "qps_batch_min": 6312.125118810746, + "qps_batch_max": 6658.22847238178, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9648, + "qps_1t_runs": [ + 1232.91989873788, + 1191.4535375292921, + 1283.9168301239383 + ], + "qps_1t": 1232.91989873788, + "qps_1t_min": 1191.4535375292921, + "qps_1t_max": 1283.9168301239383, + "median_ms": 0.7961000083014369, + "p95_ms": 0.979115001973696, + "p99_ms": 1.2354869773844277, + "mean_ms": 0.8097534330154303, + "knob": 192, + "batch": { + "recall": 0.9648, + "qps_batch_runs": [ + 4678.933895546764, + 4329.592853323451, + 4243.057615628609 + ], + "qps_batch": 4329.592853323451, + "qps_batch_min": 4243.057615628609, + "qps_batch_max": 4678.933895546764, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9742, + "qps_1t_runs": [ + 852.976347304822, + 925.788772123046, + 912.5564250682239 + ], + "qps_1t": 912.5564250682239, + "qps_1t_min": 852.976347304822, + "qps_1t_max": 925.788772123046, + "median_ms": 1.0916500177700073, + "p95_ms": 1.429825002560392, + "p99_ms": 1.8939099856652322, + "mean_ms": 1.1161160335953657, + "knob": 256, + "batch": { + "recall": 0.9742, + "qps_batch_runs": [ + 3591.7946888631536, + 3634.7663220055592, + 3621.346242645393 + ], + "qps_batch": 3621.346242645393, + "qps_batch_min": 3591.7946888631536, + "qps_batch_max": 3634.7663220055592, + "threads": 32 + } + }, + { + "nq": 1626, + "recall_queries": 1626, + "recall": 0.9843173431734318, + "qps_1t_runs": [ + 602.3082854523135, + 593.8931567535175, + 578.4804627686716 + ], + "qps_1t": 593.8931567535175, + "qps_1t_min": 578.4804627686716, + "qps_1t_max": 602.3082854523135, + "median_ms": 1.654949999647215, + "p95_ms": 2.1025050024036314, + "p99_ms": 3.031699963030401, + "mean_ms": 1.6909169124860535, + "knob": 400, + "batch": { + "recall": 0.98445, + "qps_batch_runs": [ + 2356.491438736329, + 2369.8403687891728, + 2364.0044390906096 + ], + "qps_batch": 2364.0044390906096, + "qps_batch_min": 2356.491438736329, + "qps_batch_max": 2369.8403687891728, + "threads": 32 + } + }, + { + "nq": 1086, + "recall_queries": 1086, + "recall": 0.9913443830570903, + "qps_1t_runs": [ + 376.68850383676767, + 369.90787830462375, + 371.48460884239967 + ], + "qps_1t": 371.48460884239967, + "qps_1t_min": 369.90787830462375, + "qps_1t_max": 376.68850383676767, + "median_ms": 2.6522499974817038, + "p95_ms": 3.1428549991687746, + "p99_ms": 3.9103779970900914, + "mean_ms": 2.6833302326715667, + "knob": 640, + "batch": { + "recall": 0.9909, + "qps_batch_runs": [ + 1590.1997895392503, + 1596.7146000049204, + 1471.514098992545 + ], + "qps_batch": 1590.1997895392503, + "qps_batch_min": 1471.514098992545, + "qps_batch_max": 1596.7146000049204, + "threads": 32 + } + } + ], + "mutation": { + "ref_knob": 192, + "recall_before": 0.9648, + "insert_10k": { + "seconds": 4.320585099980235, + "per_s": 2314.5013391926354, + "batch": 1000 + }, + "update_10k": { + "unsupported": true + }, + "delete_10k": { + "unsupported": true + }, + "mem_after": { + "rss_mb": 1751.83203125, + "peak_mb": 2205.703125 + } + }, + "saveload": { + "save_s": 0.9943152000196278, + "disk_mb": 1618.4442081451416, + "load_s": 0.8049297000397928, + "first_query_after_load_ms": 1.2053000391460955, + "time_to_ready_s": 0.8061350000789389 + }, + "load_before_pct": 8.909374999999995, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 268.1877243000781, + "mem": { + "rss_mb": 1678.1875, + "peak_mb": 2206.24609375 + }, + "subsets": { + "f10": [ + { + "knob": 16, + "nq": 2000, + "recall": 0.44155, + "qps_1t": 5466.842380570978, + "qps_1t_runs": [ + 5466.842380570978, + 5953.92436367418, + 4664.952613533861 + ], + "median_ms": 0.16110000433400273, + "p95_ms": 0.2903049986343831, + "p99_ms": 0.4365100053837547, + "mean_ms": 0.18841394981912649 + }, + { + "knob": 48, + "nq": 2000, + "recall": 0.6739, + "qps_1t": 2877.110826475339, + "qps_1t_runs": [ + 2870.8482890137743, + 2877.110826475339, + 2918.275728731884 + ], + "median_ms": 0.3234999894630164, + "p95_ms": 0.5018999829189852, + "p99_ms": 0.7591719599440709, + "mean_ms": 0.3461893667408731 + }, + { + "knob": 96, + "nq": 2000, + "recall": 0.8077, + "qps_1t": 1695.5023579818276, + "qps_1t_runs": [ + 1699.9336612510183, + 1695.5023579818276, + 1678.5396781116997 + ], + "median_ms": 0.5709000397473574, + "p95_ms": 0.7965450116898866, + "p99_ms": 1.0679200402228166, + "mean_ms": 0.5912699664089208 + }, + { + "knob": 192, + "nq": 2000, + "recall": 0.89915, + "qps_1t": 874.4338363345231, + "qps_1t_runs": [ + 874.4338363345231, + 846.3365538837717, + 876.6589896918307 + ], + "median_ms": 1.1131999781355262, + "p95_ms": 1.5285150526324287, + "p99_ms": 2.5221679959213374, + "mean_ms": 1.1552848501693613 + }, + { + "knob": 640, + "nq": 776, + "recall": 0.9730670103092783, + "qps_1t": 266.70005458656505, + "qps_1t_runs": [ + 266.70005458656505, + 267.1959442475345, + 255.4049497183635 + ], + "median_ms": 3.818400000454858, + "p95_ms": 4.594715003622696, + "p99_ms": 5.634321011020805, + "mean_ms": 3.8024844065479795 + } + ], + "f01": [ + { + "knob": 16, + "nq": 2000, + "recall": 0.1634, + "qps_1t": 6291.692266024276, + "qps_1t_runs": [ + 6291.692266024276, + 6300.593752965803, + 6159.548938791261 + ], + "median_ms": 0.1503999810665846, + "p95_ms": 0.24009996850509197, + "p99_ms": 0.30784095753915697, + "mean_ms": 0.1600015004320691 + }, + { + "knob": 48, + "nq": 2000, + "recall": 0.314, + "qps_1t": 2850.207893216705, + "qps_1t_runs": [ + 2889.053276076182, + 2850.207893216705, + 2846.8223126044777 + ], + "median_ms": 0.33660000190138817, + "p95_ms": 0.4846100200666115, + "p99_ms": 0.6095239846035844, + "mean_ms": 0.3494182000479971 + }, + { + "knob": 96, + "nq": 2000, + "recall": 0.4525, + "qps_1t": 1677.5708767278245, + "qps_1t_runs": [ + 1677.5708767278245, + 1671.4556662592422, + 1681.7205881031446 + ], + "median_ms": 0.5917000235058367, + "p95_ms": 0.763929996173829, + "p99_ms": 0.8816130191553387, + "mean_ms": 0.5963366667274386 + }, + { + "knob": 192, + "nq": 2000, + "recall": 0.61315, + "qps_1t": 882.2221907440756, + "qps_1t_runs": [ + 882.2221907440756, + 862.7187933461809, + 906.4746583933228 + ], + "median_ms": 1.1019499797839671, + "p95_ms": 1.42611003248021, + "p99_ms": 1.838933972758246, + "mean_ms": 1.1319341167400125 + }, + { + "knob": 640, + "nq": 765, + "recall": 0.8538562091503268, + "qps_1t": 274.6660034651382, + "qps_1t_runs": [ + 266.67058946641896, + 274.6660034651382, + 280.1825561204588 + ], + "median_ms": 3.65949998376891, + "p95_ms": 4.610290005803108, + "p99_ms": 5.334977973252532, + "mean_ms": 3.6532772976848293 + } + ] + } + } + }, + "faiss_hnsw|text384|100000": { + "mem_baseline": { + "rss_mb": 37.3515625, + "peak_mb": 37.3515625 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 8.898113100090995, + "index_s": 3.400025889277458e-06, + "total_s": 8.898116500116885, + "first_query_ms": 3.1128000118769705, + "ready_s": 8.901229300128762, + "total_s_runs": [ + 8.898116500116885, + 8.407066700048745, + 8.672152799903415 + ] + }, + "mem_after_build": { + "rss_mb": 221.4609375, + "peak_mb": 365.76953125 + }, + "mem_after_first_query": { + "rss_mb": 221.65625, + "peak_mb": 365.76953125 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.63615, + "qps_1t_runs": [ + 15712.745621311362, + 15199.693601217044, + 17056.58023085163 + ], + "qps_1t": 15712.745621311362, + "qps_1t_min": 15199.693601217044, + "qps_1t_max": 17056.58023085163, + "median_ms": 0.058399979025125504, + "p95_ms": 0.08829997386783361, + "p99_ms": 0.11770196550060068, + "mean_ms": 0.06268726679263636, + "knob": 10, + "batch": { + "recall": 0.63615, + "qps_batch_runs": [ + 90515.34926644799, + 97671.98825100507, + 96808.70128416846 + ], + "qps_batch": 96808.70128416846, + "qps_batch_min": 90515.34926644799, + "qps_batch_max": 97671.98825100507, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.73145, + "qps_1t_runs": [ + 12778.55657739224, + 12881.062004773192, + 12783.195732363236 + ], + "qps_1t": 12783.195732363236, + "qps_1t_min": 12778.55657739224, + "qps_1t_max": 12881.062004773192, + "median_ms": 0.07349997758865356, + "p95_ms": 0.10460503690410407, + "p99_ms": 0.13550298404879874, + "mean_ms": 0.07803905007313006, + "knob": 16, + "batch": { + "recall": 0.73145, + "qps_batch_runs": [ + 72568.677135575, + 66867.0449477335, + 70071.68336399632 + ], + "qps_batch": 70071.68336399632, + "qps_batch_min": 66867.0449477335, + "qps_batch_max": 72568.677135575, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8014, + "qps_1t_runs": [ + 9414.094982830675, + 9092.02076580962, + 9647.909209724548 + ], + "qps_1t": 9414.094982830675, + "qps_1t_min": 9092.02076580962, + "qps_1t_max": 9647.909209724548, + "median_ms": 0.0971999834291637, + "p95_ms": 0.13510503631550821, + "p99_ms": 0.21710802335292118, + "mean_ms": 0.10661988338688388, + "knob": 24, + "batch": { + "recall": 0.8014, + "qps_batch_runs": [ + 50166.42710141613, + 52264.21652425431, + 50335.359347950674 + ], + "qps_batch": 50335.359347950674, + "qps_batch_min": 50166.42710141613, + "qps_batch_max": 52264.21652425431, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.84925, + "qps_1t_runs": [ + 7577.838667905151, + 7879.373105523631, + 8242.793130391423 + ], + "qps_1t": 7879.373105523631, + "qps_1t_min": 7577.838667905151, + "qps_1t_max": 8242.793130391423, + "median_ms": 0.12129999231547117, + "p95_ms": 0.1642999704927206, + "p99_ms": 0.2095230232225736, + "mean_ms": 0.12673183292887794, + "knob": 32, + "batch": { + "recall": 0.84925, + "qps_batch_runs": [ + 42663.9361539615, + 44864.47565823431, + 45496.21587440867 + ], + "qps_batch": 44864.47565823431, + "qps_batch_min": 42663.9361539615, + "qps_batch_max": 45496.21587440867, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.90045, + "qps_1t_runs": [ + 5918.579847600545, + 5913.146525522864, + 5762.908202399839 + ], + "qps_1t": 5913.146525522864, + "qps_1t_min": 5762.908202399839, + "qps_1t_max": 5918.579847600545, + "median_ms": 0.16529997810721397, + "p95_ms": 0.20900001982226968, + "p99_ms": 0.2608020068146289, + "mean_ms": 0.1705325501097832, + "knob": 48, + "batch": { + "recall": 0.90045, + "qps_batch_runs": [ + 30514.87753081251, + 30793.439167646644, + 30746.714733704695 + ], + "qps_batch": 30746.714733704695, + "qps_batch_min": 30514.87753081251, + "qps_batch_max": 30793.439167646644, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9279, + "qps_1t_runs": [ + 3995.9880323559482, + 4554.424680026955, + 4703.849190498367 + ], + "qps_1t": 4554.424680026955, + "qps_1t_min": 3995.9880323559482, + "qps_1t_max": 4703.849190498367, + "median_ms": 0.21744999685324728, + "p95_ms": 0.304805047926493, + "p99_ms": 0.39480297942645853, + "mean_ms": 0.22746984999200018, + "knob": 64, + "batch": { + "recall": 0.9279, + "qps_batch_runs": [ + 24861.95399342435, + 25427.014940882702, + 24381.205018600456 + ], + "qps_batch": 24861.95399342435, + "qps_batch_min": 24381.205018600456, + "qps_batch_max": 25427.014940882702, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.95695, + "qps_1t_runs": [ + 3312.498820793018, + 3103.066172948751, + 3272.6775254334375 + ], + "qps_1t": 3272.6775254334375, + "qps_1t_min": 3103.066172948751, + "qps_1t_max": 3312.498820793018, + "median_ms": 0.29639998683705926, + "p95_ms": 0.37670496094506234, + "p99_ms": 0.5525669583585128, + "mean_ms": 0.3099029999769603, + "knob": 96, + "batch": { + "recall": 0.95695, + "qps_batch_runs": [ + 18590.660606781523, + 18644.489660895728, + 18068.185716811222 + ], + "qps_batch": 18590.660606781523, + "qps_batch_min": 18068.185716811222, + "qps_batch_max": 18644.489660895728, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9714, + "qps_1t_runs": [ + 2277.247095234753, + 2431.755218577571, + 2369.900745145576 + ], + "qps_1t": 2369.900745145576, + "qps_1t_min": 2277.247095234753, + "qps_1t_max": 2431.755218577571, + "median_ms": 0.38599997060373425, + "p95_ms": 0.5194200522964821, + "p99_ms": 0.9286969696404433, + "mean_ms": 0.42410363312228583, + "knob": 128, + "batch": { + "recall": 0.9714, + "qps_batch_runs": [ + 14389.545131548419, + 14274.223535869021, + 13814.692479831621 + ], + "qps_batch": 14274.223535869021, + "qps_batch_min": 13814.692479831621, + "qps_batch_max": 14389.545131548419, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9842, + "qps_1t_runs": [ + 1700.5611198062475, + 1641.858591965703, + 1724.1346603584495 + ], + "qps_1t": 1700.5611198062475, + "qps_1t_min": 1641.858591965703, + "qps_1t_max": 1724.1346603584495, + "median_ms": 0.5584499740507454, + "p95_ms": 0.6910200201673434, + "p99_ms": 1.0444640362402438, + "mean_ms": 0.592369383336821, + "knob": 192, + "batch": { + "recall": 0.9842, + "qps_batch_runs": [ + 10424.827351795268, + 10206.466613284618, + 9914.345015554363 + ], + "qps_batch": 10206.466613284618, + "qps_batch_min": 9914.345015554363, + "qps_batch_max": 10424.827351795268, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99085, + "qps_1t_runs": [ + 1281.855964406836, + 1317.9759256987318, + 1248.615909055167 + ], + "qps_1t": 1281.855964406836, + "qps_1t_min": 1248.615909055167, + "qps_1t_max": 1317.9759256987318, + "median_ms": 0.7422000053338706, + "p95_ms": 0.967204972403124, + "p99_ms": 1.591091007576326, + "mean_ms": 0.7799149499624036, + "knob": 256, + "batch": { + "recall": 0.99085, + "qps_batch_runs": [ + 7492.407382294504, + 7844.496967176118, + 8105.389515179969 + ], + "qps_batch": 7844.496967176118, + "qps_batch_min": 7492.407382294504, + "qps_batch_max": 8105.389515179969, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9958, + "qps_1t_runs": [ + 780.9628438166951, + 838.6397214042684, + 823.4041092042909 + ], + "qps_1t": 823.4041092042909, + "qps_1t_min": 780.9628438166951, + "qps_1t_max": 838.6397214042684, + "median_ms": 1.1656999704428017, + "p95_ms": 1.496334985131398, + "p99_ms": 2.4191169685218488, + "mean_ms": 1.2291160997798822, + "knob": 400, + "batch": { + "recall": 0.9958, + "qps_batch_runs": [ + 5363.90041428864, + 5285.262810873027, + 4851.04855408874 + ], + "qps_batch": 5285.262810873027, + "qps_batch_min": 4851.04855408874, + "qps_batch_max": 5363.90041428864, + "threads": 32 + } + }, + { + "nq": 1585, + "recall_queries": 1585, + "recall": 0.9982965299684543, + "qps_1t_runs": [ + 515.1336179804398, + 508.02444649877924, + 492.43010663020107 + ], + "qps_1t": 508.02444649877924, + "qps_1t_min": 492.43010663020107, + "qps_1t_max": 515.1336179804398, + "median_ms": 1.8851000349968672, + "p95_ms": 2.443619980476797, + "p99_ms": 4.007778021041305, + "mean_ms": 1.980132723523874, + "knob": 640, + "batch": { + "recall": 0.99855, + "qps_batch_runs": [ + 3847.067525050958, + 3804.6900415597056, + 3885.931587008845 + ], + "qps_batch": 3847.067525050958, + "qps_batch_min": 3804.6900415597056, + "qps_batch_max": 3885.931587008845, + "threads": 32 + } + } + ], + "mutation": { + "ref_knob": 96, + "recall_before": 0.95695, + "insert_10k": { + "seconds": 2.837573600001633, + "per_s": 3524.1376646562558, + "batch": 1000 + }, + "update_10k": { + "unsupported": true + }, + "delete_10k": { + "unsupported": true + }, + "mem_after": { + "rss_mb": 271.62109375, + "peak_mb": 531.39453125 + } + }, + "saveload": { + "save_s": 0.13437949999934062, + "disk_mb": 176.2752628326416, + "load_s": 0.07410839997464791, + "first_query_after_load_ms": 0.4959999932907522, + "time_to_ready_s": 0.07460439996793866 + }, + "load_before_pct": 8.999999999999996, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 13.72953139996389, + "mem": { + "rss_mb": 221.21875, + "peak_mb": 365.46875 + }, + "subsets": { + "f10": [ + { + "knob": 16, + "nq": 2000, + "recall": 0.49035, + "qps_1t": 9177.78514624439, + "qps_1t_runs": [ + 8415.447028125576, + 9177.78514624439, + 9200.334540849128 + ], + "median_ms": 0.10820000898092985, + "p95_ms": 0.15601002087350938, + "p99_ms": 0.21310405049007394, + "mean_ms": 0.11215985017285372 + }, + { + "knob": 48, + "nq": 2000, + "recall": 0.7286, + "qps_1t": 4095.5240055168656, + "qps_1t_runs": [ + 4311.129940938423, + 3866.2965105164344, + 4095.5240055168656 + ], + "median_ms": 0.24279998615384102, + "p95_ms": 0.3362099989317358, + "p99_ms": 0.42862000409513756, + "mean_ms": 0.24492406644276343 + }, + { + "knob": 96, + "nq": 2000, + "recall": 0.85835, + "qps_1t": 2346.184168661592, + "qps_1t_runs": [ + 2350.0958986093956, + 2346.184168661592, + 2327.5741798904346 + ], + "median_ms": 0.43130002450197935, + "p95_ms": 0.5379999667638913, + "p99_ms": 0.6185499951243412, + "mean_ms": 0.42712346655510675 + }, + { + "knob": 192, + "nq": 2000, + "recall": 0.9435, + "qps_1t": 1279.2286969116685, + "qps_1t_runs": [ + 1292.1180423004619, + 1279.2286969116685, + 1244.0417399722471 + ], + "median_ms": 0.7985500269569457, + "p95_ms": 1.0106150089995936, + "p99_ms": 1.1330060119507837, + "mean_ms": 0.7864918997123217 + }, + { + "knob": 640, + "nq": 1009, + "recall": 0.9924677898909812, + "qps_1t": 376.1127896790685, + "qps_1t_runs": [ + 384.2244376254349, + 376.1127896790685, + 361.92478129769273 + ], + "median_ms": 2.741199976298958, + "p95_ms": 3.2588100177235897, + "p99_ms": 3.5108140157535668, + "mean_ms": 2.6748091509940455 + } + ], + "f01": [ + { + "knob": 16, + "nq": 2000, + "recall": 0.1894, + "qps_1t": 9230.882866141921, + "qps_1t_runs": [ + 9378.47446222709, + 9222.369791819785, + 9230.882866141921 + ], + "median_ms": 0.10490004206076264, + "p95_ms": 0.15209999401122332, + "p99_ms": 0.18720602791290744, + "mean_ms": 0.10779704974265769 + }, + { + "knob": 48, + "nq": 2000, + "recall": 0.3687, + "qps_1t": 4099.546843558394, + "qps_1t_runs": [ + 4202.854130580572, + 4099.546843558394, + 4070.13658853396 + ], + "median_ms": 0.2410999732092023, + "p95_ms": 0.31101503700483596, + "p99_ms": 0.3716159774921838, + "mean_ms": 0.24251831635289514 + }, + { + "knob": 96, + "nq": 2000, + "recall": 0.52705, + "qps_1t": 2491.699210748766, + "qps_1t_runs": [ + 2509.0860180991162, + 2491.699210748766, + 2277.0158222760533 + ], + "median_ms": 0.41310000233352184, + "p95_ms": 0.5335149879101664, + "p99_ms": 0.6408009899314493, + "mean_ms": 0.413018450946159 + }, + { + "knob": 192, + "nq": 2000, + "recall": 0.7021, + "qps_1t": 1285.794344449443, + "qps_1t_runs": [ + 1303.0277738859409, + 1266.3730959873144, + 1285.794344449443 + ], + "median_ms": 0.7897000177763402, + "p95_ms": 0.9795050049433485, + "p99_ms": 1.0780070006148892, + "mean_ms": 0.7782764667645097 + }, + { + "knob": 640, + "nq": 1013, + "recall": 0.9277393879565646, + "qps_1t": 377.32581029791163, + "qps_1t_runs": [ + 377.32581029791163, + 380.2200878508526, + 370.64568175764583 + ], + "median_ms": 2.72910000057891, + "p95_ms": 3.215020033530891, + "p99_ms": 3.4220020379871126, + "mean_ms": 2.659426554453698 + } + ] + } + } + }, + "faiss_ivfflat|glove|1000000": { + "mem_baseline": { + "rss_mb": 35.08984375, + "peak_mb": 35.08984375 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 3.609998384490609e-05, + "index_s": 11.909425000019837, + "total_s": 11.909461100003682, + "first_query_ms": 31.58200002508238, + "ready_s": 11.941043100028764 + }, + "mem_after_build": { + "rss_mb": 569.12890625, + "peak_mb": 1018.16796875 + }, + "mem_after_first_query": { + "rss_mb": 569.30859375, + "peak_mb": 1018.16796875 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.3046, + "qps_1t_runs": [ + 19407.451816610996, + 18048.29362201203, + 19281.977684321428 + ], + "qps_1t": 19281.977684321428, + "qps_1t_min": 18048.29362201203, + "qps_1t_max": 19407.451816610996, + "median_ms": 0.050350005039945245, + "p95_ms": 0.07059995841700584, + "p99_ms": 0.0911029952112586, + "mean_ms": 0.05293179991228195, + "knob": 1, + "batch": { + "recall": 0.3046, + "qps_batch_runs": [ + 174596.24662671867, + 170065.134591196, + 169922.09095888014 + ], + "qps_batch": 170065.134591196, + "qps_batch_min": 169922.09095888014, + "qps_batch_max": 174596.24662671867, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.42815, + "qps_1t_runs": [ + 16240.607886157788, + 15642.084130302677, + 13254.864542624031 + ], + "qps_1t": 15642.084130302677, + "qps_1t_min": 13254.864542624031, + "qps_1t_max": 16240.607886157788, + "median_ms": 0.060699996538460255, + "p95_ms": 0.09070499218069018, + "p99_ms": 0.13461197726428534, + "mean_ms": 0.06698271655477583, + "knob": 2, + "batch": { + "recall": 0.42815, + "qps_batch_runs": [ + 117091.90527067326, + 130277.0339776239, + 96350.71649683625 + ], + "qps_batch": 117091.90527067326, + "qps_batch_min": 96350.71649683625, + "qps_batch_max": 130277.0339776239, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.5371, + "qps_1t_runs": [ + 11602.590427840798, + 12379.079562858198, + 12354.578805281451 + ], + "qps_1t": 12354.578805281451, + "qps_1t_min": 11602.590427840798, + "qps_1t_max": 12379.079562858198, + "median_ms": 0.07765000918880105, + "p95_ms": 0.10619999666232616, + "p99_ms": 0.13381302880588947, + "mean_ms": 0.08263691684502798, + "knob": 4, + "batch": { + "recall": 0.5371, + "qps_batch_runs": [ + 82463.34510119348, + 83272.61374473717, + 83461.65567814918 + ], + "qps_batch": 83272.61374473717, + "qps_batch_min": 82463.34510119348, + "qps_batch_max": 83461.65567814918, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.63475, + "qps_1t_runs": [ + 8461.346462714502, + 8221.824846522439, + 8861.799353882217 + ], + "qps_1t": 8461.346462714502, + "qps_1t_min": 8221.824846522439, + "qps_1t_max": 8861.799353882217, + "median_ms": 0.11230004020035267, + "p95_ms": 0.16260495758615437, + "p99_ms": 0.21591701370198316, + "mean_ms": 0.11755196656061646, + "knob": 8, + "batch": { + "recall": 0.63475, + "qps_batch_runs": [ + 49956.038668181696, + 49251.500312317956, + 49977.13545806013 + ], + "qps_batch": 49956.038668181696, + "qps_batch_min": 49251.500312317956, + "qps_batch_max": 49977.13545806013, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7203, + "qps_1t_runs": [ + 5748.170372087822, + 5760.302320134666, + 5575.506562798144 + ], + "qps_1t": 5748.170372087822, + "qps_1t_min": 5575.506562798144, + "qps_1t_max": 5760.302320134666, + "median_ms": 0.1700000138953328, + "p95_ms": 0.23919996165204793, + "p99_ms": 0.2879049803595991, + "mean_ms": 0.1756420998038569, + "knob": 16, + "batch": { + "recall": 0.7203, + "qps_batch_runs": [ + 26552.596724555224, + 25542.784165509365, + 26399.643067961286 + ], + "qps_batch": 26399.643067961286, + "qps_batch_min": 25542.784165509365, + "qps_batch_max": 26552.596724555224, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.79335, + "qps_1t_runs": [ + 3221.387438729675, + 3127.584654353327, + 3364.702453708167 + ], + "qps_1t": 3221.387438729675, + "qps_1t_min": 3127.584654353327, + "qps_1t_max": 3364.702453708167, + "median_ms": 0.28600002406165004, + "p95_ms": 0.40630495350342244, + "p99_ms": 0.5269339960068472, + "mean_ms": 0.309121299952191, + "knob": 32, + "batch": { + "recall": 0.79335, + "qps_batch_runs": [ + 14470.17804969104, + 14622.43769506232, + 14831.349025807702 + ], + "qps_batch": 14622.43769506232, + "qps_batch_min": 14470.17804969104, + "qps_batch_max": 14831.349025807702, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.85875, + "qps_1t_runs": [ + 1792.8785955925964, + 1819.1575486770291, + 1796.05442919011 + ], + "qps_1t": 1796.05442919011, + "qps_1t_min": 1792.8785955925964, + "qps_1t_max": 1819.1575486770291, + "median_ms": 0.5222000181674957, + "p95_ms": 0.7214249664684759, + "p99_ms": 0.8821380225708716, + "mean_ms": 0.5547477498863979, + "knob": 64, + "batch": { + "recall": 0.85875, + "qps_batch_runs": [ + 7866.236225995296, + 7626.802262251877, + 7595.762628754349 + ], + "qps_batch": 7626.802262251877, + "qps_batch_min": 7595.762628754349, + "qps_batch_max": 7866.236225995296, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.91225, + "qps_1t_runs": [ + 988.6746336637101, + 997.061311137976, + 976.702376154054 + ], + "qps_1t": 988.6746336637101, + "qps_1t_min": 976.702376154054, + "qps_1t_max": 997.061311137976, + "median_ms": 0.9730500169098377, + "p95_ms": 1.2777250085491687, + "p99_ms": 1.6372900089481888, + "mean_ms": 1.0127519333230641, + "knob": 128, + "batch": { + "recall": 0.91225, + "qps_batch_runs": [ + 3992.9667886658317, + 3977.6693640971093, + 4103.244189362854 + ], + "qps_batch": 3992.9667886658317, + "qps_batch_min": 3977.6693640971093, + "qps_batch_max": 4103.244189362854, + "threads": 32 + } + }, + { + "nq": 1649, + "recall_queries": 1649, + "recall": 0.9487568223165554, + "qps_1t_runs": [ + 534.3025477109227, + 524.7253180281099, + 518.5877216780946 + ], + "qps_1t": 524.7253180281099, + "qps_1t_min": 518.5877216780946, + "qps_1t_max": 534.3025477109227, + "median_ms": 1.8497999990358949, + "p95_ms": 2.341410017106682, + "p99_ms": 3.1939419882837647, + "mean_ms": 1.9018906204056085, + "knob": 256, + "batch": { + "recall": 0.9484, + "qps_batch_runs": [ + 2015.9313003813513, + 2098.031354166346, + 2064.2278296885274 + ], + "qps_batch": 2064.2278296885274, + "qps_batch_min": 2015.9313003813513, + "qps_batch_max": 2098.031354166346, + "threads": 32 + } + }, + { + "nq": 136, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 43.766656276399935, + 43.78137552342066, + 43.020590821539116 + ], + "qps_1t": 43.766656276399935, + "qps_1t_min": 43.020590821539116, + "qps_1t_max": 43.78137552342066, + "median_ms": 22.24394999211654, + "p95_ms": 28.908635003608637, + "p99_ms": 31.374543039128188, + "mean_ms": 22.977963232894098, + "knob": 4000, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 157.31589507569646, + 159.5238862848845, + 159.04147737307707 + ], + "qps_batch": 159.04147737307707, + "qps_batch_min": 157.31589507569646, + "qps_batch_max": 159.5238862848845, + "threads": 32 + } + } + ], + "mutation": { + "ref_knob": 4000, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.04435360000934452, + "per_s": 225460.84191346762, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.08335299999453127, + "per_s": 119971.6866898143, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.004664399952162057, + "per_s": 2143898.486956456, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 587.8359375, + "peak_mb": 1018.16796875 + }, + "first_query_after_ms": 31.066800002008677, + "after": { + "nq": 130, + "recall_queries": 130, + "recall": 1.0, + "qps_1t_runs": [ + 42.239189092513755, + 41.53314956842764, + 42.63354490640715 + ], + "qps_1t": 42.239189092513755, + "qps_1t_min": 41.53314956842764, + "qps_1t_max": 42.63354490640715, + "median_ms": 22.592400026042014, + "p95_ms": 31.08705000195185, + "p99_ms": 38.17615797044714, + "mean_ms": 23.735852820727114 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 0.14943550003226846, + "disk_mb": 390.655650138855, + "load_s": 0.24632809998001903, + "first_query_after_load_ms": 23.79170001950115, + "time_to_ready_s": 0.2701197999995202 + }, + "load_before_pct": 6.406250000000003, + "load_contaminated": false + }, + "faiss_ivfflat|glove|100000": { + "mem_baseline": { + "rss_mb": 35.11328125, + "peak_mb": 35.11328125 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 5.600042641162872e-06, + "index_s": 0.9731661999830976, + "total_s": 0.9731718000257388, + "first_query_ms": 3.0707999831065536, + "ready_s": 0.9762426000088453, + "total_s_runs": [ + 0.9731718000257388, + 1.0775378000107594, + 1.0854720000061207 + ] + }, + "mem_after_build": { + "rss_mb": 125.71875, + "peak_mb": 189.9453125 + }, + "mem_after_first_query": { + "rss_mb": 125.85546875, + "peak_mb": 189.9453125 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.28265, + "qps_1t_runs": [ + 50764.12710632269, + 51839.256755303286, + 51820.72081536658 + ], + "qps_1t": 51820.72081536658, + "qps_1t_min": 50764.12710632269, + "qps_1t_max": 51839.256755303286, + "median_ms": 0.018899969290941954, + "p95_ms": 0.022299995180219412, + "p99_ms": 0.029099989333190024, + "mean_ms": 0.019428883359069005, + "knob": 1, + "batch": { + "recall": 0.28265, + "qps_batch_runs": [ + 367329.6957723643, + 310318.0757221067, + 292056.0751934181 + ], + "qps_batch": 310318.0757221067, + "qps_batch_min": 292056.0751934181, + "qps_batch_max": 367329.6957723643, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.40495, + "qps_1t_runs": [ + 45530.70516741316, + 44673.171046688614, + 41407.953405237175 + ], + "qps_1t": 44673.171046688614, + "qps_1t_min": 41407.953405237175, + "qps_1t_max": 45530.70516741316, + "median_ms": 0.021199986804276705, + "p95_ms": 0.025899967295117676, + "p99_ms": 0.041602025157772046, + "mean_ms": 0.022832650089791667, + "knob": 2, + "batch": { + "recall": 0.40495, + "qps_batch_runs": [ + 382168.03673912975, + 441783.9222497578, + 311827.6216124506 + ], + "qps_batch": 382168.03673912975, + "qps_batch_min": 311827.6216124506, + "qps_batch_max": 441783.9222497578, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.5251, + "qps_1t_runs": [ + 39053.72819832607, + 34299.02230730964, + 35841.715715522325 + ], + "qps_1t": 35841.715715522325, + "qps_1t_min": 34299.02230730964, + "qps_1t_max": 39053.72819832607, + "median_ms": 0.02629996743053198, + "p95_ms": 0.03350002225488424, + "p99_ms": 0.04200602823402746, + "mean_ms": 0.027553850018496934, + "knob": 4, + "batch": { + "recall": 0.5251, + "qps_batch_runs": [ + 266481.906291436, + 275216.73397126055, + 306306.8578009863 + ], + "qps_batch": 275216.73397126055, + "qps_batch_min": 266481.906291436, + "qps_batch_max": 306306.8578009863, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6316, + "qps_1t_runs": [ + 27987.72467294657, + 28594.91742960397, + 27218.698087547822 + ], + "qps_1t": 27987.72467294657, + "qps_1t_min": 27218.698087547822, + "qps_1t_max": 28594.91742960397, + "median_ms": 0.033800024539232254, + "p95_ms": 0.04349998198449612, + "p99_ms": 0.055702016688883346, + "mean_ms": 0.03581354992153744, + "knob": 8, + "batch": { + "recall": 0.6316, + "qps_batch_runs": [ + 219440.2076507734, + 246831.30321182124, + 260718.80074205264 + ], + "qps_batch": 246831.30321182124, + "qps_batch_min": 219440.2076507734, + "qps_batch_max": 260718.80074205264, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7287, + "qps_1t_runs": [ + 19426.07604238988, + 19775.449641076913, + 11660.128260928292 + ], + "qps_1t": 19426.07604238988, + "qps_1t_min": 11660.128260928292, + "qps_1t_max": 19775.449641076913, + "median_ms": 0.04929996794089675, + "p95_ms": 0.06619998021051288, + "p99_ms": 0.15700697316788154, + "mean_ms": 0.06260243338571551, + "knob": 16, + "batch": { + "recall": 0.7287, + "qps_batch_runs": [ + 134086.0031675445, + 187015.51193360728, + 62529.11510069719 + ], + "qps_batch": 134086.0031675445, + "qps_batch_min": 62529.11510069719, + "qps_batch_max": 187015.51193360728, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.81195, + "qps_1t_runs": [ + 12410.15821948934, + 10512.317795088215, + 10764.054056587604 + ], + "qps_1t": 10764.054056587604, + "qps_1t_min": 10512.317795088215, + "qps_1t_max": 12410.15821948934, + "median_ms": 0.083099992480129, + "p95_ms": 0.11841002560686324, + "p99_ms": 0.16481203201692576, + "mean_ms": 0.08953581677633338, + "knob": 32, + "batch": { + "recall": 0.81195, + "qps_batch_runs": [ + 137203.38349474253, + 153105.35888254055, + 156825.84540283465 + ], + "qps_batch": 153105.35888254055, + "qps_batch_min": 137203.38349474253, + "qps_batch_max": 156825.84540283465, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.87955, + "qps_1t_runs": [ + 5917.858332807343, + 6870.736706255522, + 6347.765518460254 + ], + "qps_1t": 6347.765518460254, + "qps_1t_min": 5917.858332807343, + "qps_1t_max": 6870.736706255522, + "median_ms": 0.14929997269064188, + "p95_ms": 0.20280497265048328, + "p99_ms": 0.2977019967511297, + "mean_ms": 0.15735353342218636, + "knob": 64, + "batch": { + "recall": 0.87955, + "qps_batch_runs": [ + 84246.70815372968, + 93353.6842871729, + 94936.5586810349 + ], + "qps_batch": 93353.6842871729, + "qps_batch_min": 84246.70815372968, + "qps_batch_max": 94936.5586810349, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.93315, + "qps_1t_runs": [ + 3637.090404540063, + 3718.7875250712545, + 3728.4120285061463 + ], + "qps_1t": 3718.7875250712545, + "qps_1t_min": 3637.090404540063, + "qps_1t_max": 3728.4120285061463, + "median_ms": 0.25590002769604325, + "p95_ms": 0.33351997844874864, + "p99_ms": 0.4978390067117297, + "mean_ms": 0.2706868662304866, + "knob": 128, + "batch": { + "recall": 0.93315, + "qps_batch_runs": [ + 53001.473415159344, + 53712.039015396615, + 53383.015123214165 + ], + "qps_batch": 53383.015123214165, + "qps_batch_min": 53001.473415159344, + "qps_batch_max": 53712.039015396615, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.97375, + "qps_1t_runs": [ + 2050.8910114333194, + 2030.7749735870777, + 1979.474824344494 + ], + "qps_1t": 2030.7749735870777, + "qps_1t_min": 1979.474824344494, + "qps_1t_max": 2050.8910114333194, + "median_ms": 0.457499991171062, + "p95_ms": 0.603009996120818, + "p99_ms": 0.990019025630329, + "mean_ms": 0.4950667669957814, + "knob": 256, + "batch": { + "recall": 0.97375, + "qps_batch_runs": [ + 24478.9053023146, + 26799.05399898174, + 27518.71959977751 + ], + "qps_batch": 26799.05399898174, + "qps_batch_min": 24478.9053023146, + "qps_batch_max": 27518.71959977751, + "threads": 32 + } + }, + { + "nq": 1387, + "recall_queries": 1387, + "recall": 1.0, + "qps_1t_runs": [ + 437.65144063976624, + 428.20064626510197, + 438.13187792093464 + ], + "qps_1t": 437.65144063976624, + "qps_1t_min": 428.20064626510197, + "qps_1t_max": 438.13187792093464, + "median_ms": 2.1810000180266798, + "p95_ms": 2.7548999642021954, + "p99_ms": 4.526239994447587, + "mean_ms": 2.300898317157317, + "knob": 1265, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 6205.656704038925, + 6444.906042742331, + 6399.543328686767 + ], + "qps_batch": 6399.543328686767, + "qps_batch_min": 6205.656704038925, + "qps_batch_max": 6444.906042742331, + "threads": 32 + } + } + ], + "mutation": { + "ref_knob": 256, + "recall_before": 0.97375, + "insert_10k": { + "seconds": 0.014954000012949109, + "per_s": 668717.3994476866, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.016666199953760952, + "per_s": 600016.8021351121, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.001879999996162951, + "per_s": 5319148.947026508, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 148.92578125, + "peak_mb": 249.453125 + }, + "first_query_after_ms": 1.3157000066712499, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.97375, + "qps_1t_runs": [ + 1999.9248001595286, + 2063.5051990951074, + 1877.156090388782 + ], + "qps_1t": 1999.9248001595286, + "qps_1t_min": 1877.156090388782, + "qps_1t_max": 2063.5051990951074, + "median_ms": 0.4807999939657748, + "p95_ms": 0.6394049793016165, + "p99_ms": 0.8056510222377267, + "mean_ms": 0.5057839500659611 + }, + "recall_after": 0.97375 + }, + "saveload": { + "save_s": 0.01961500005563721, + "disk_mb": 39.402255058288574, + "load_s": 0.02694720000727102, + "first_query_after_load_ms": 0.7131000165827572, + "time_to_ready_s": 0.02766030002385378 + }, + "load_before_pct": 9.865624999999998, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 1.0329756001010537, + "mem": { + "rss_mb": 125.35546875, + "peak_mb": 189.16796875 + }, + "subsets": { + "f10": [ + { + "knob": 2, + "nq": 2000, + "recall": 0.25245, + "qps_1t": 46568.26834281507, + "qps_1t_runs": [ + 46836.43299188914, + 46568.26834281507, + 46301.44129870635 + ], + "median_ms": 0.02099998528137803, + "p95_ms": 0.023800006601959467, + "p99_ms": 0.027801036485470852, + "mean_ms": 0.021474116559450824 + }, + { + "knob": 8, + "nq": 2000, + "recall": 0.4757, + "qps_1t": 37922.02518435879, + "qps_1t_runs": [ + 38069.49521159052, + 37502.8359749178, + 37922.02518435879 + ], + "median_ms": 0.02600002335384488, + "p95_ms": 0.029600050766021013, + "p99_ms": 0.03320101939607414, + "mean_ms": 0.026434100087499246 + }, + { + "knob": 32, + "nq": 2000, + "recall": 0.6965, + "qps_1t": 22912.31583810015, + "qps_1t_runs": [ + 22912.31583810015, + 23068.529704173736, + 22905.466859203912 + ], + "median_ms": 0.042700034100562334, + "p95_ms": 0.04989997250959277, + "p99_ms": 0.057500030379742384, + "mean_ms": 0.04355048323244167 + }, + { + "knob": 64, + "nq": 2000, + "recall": 0.79565, + "qps_1t": 15191.946480572195, + "qps_1t_runs": [ + 14187.154178696659, + 15191.946480572195, + 15580.449698923905 + ], + "median_ms": 0.06549997488036752, + "p95_ms": 0.07910501735750587, + "p99_ms": 0.09650303574744619, + "mean_ms": 0.06683121641981415 + }, + { + "knob": 1265, + "nq": 2000, + "recall": 1.0, + "qps_1t": 1083.0819719056162, + "qps_1t_runs": [ + 1295.036848554639, + 1083.0819719056162, + 1057.6836882883838 + ], + "median_ms": 0.9545999928377569, + "p95_ms": 1.1356250324752182, + "p99_ms": 1.2982409901451328, + "mean_ms": 0.8803107336959025 + } + ], + "f01": [ + { + "knob": 2, + "nq": 2000, + "recall": 0.07965, + "qps_1t": 31984.903219243304, + "qps_1t_runs": [ + 36245.67266438837, + 31984.903219243304, + 31314.145038388862 + ], + "median_ms": 0.032400013878941536, + "p95_ms": 0.034400029107928276, + "p99_ms": 0.044000985799357316, + "mean_ms": 0.030262900108937174 + }, + { + "knob": 8, + "nq": 2000, + "recall": 0.22195, + "qps_1t": 26685.4177381236, + "qps_1t_runs": [ + 26685.4177381236, + 29704.92586738681, + 26623.678403054608 + ], + "median_ms": 0.037299992982298136, + "p95_ms": 0.04129996814299375, + "p99_ms": 0.05170204443857078, + "mean_ms": 0.03623288332407052 + }, + { + "knob": 32, + "nq": 2000, + "recall": 0.45625, + "qps_1t": 19772.986221011946, + "qps_1t_runs": [ + 19635.564119728824, + 20179.27265375653, + 19772.986221011946 + ], + "median_ms": 0.053499999921768904, + "p95_ms": 0.06029999349266291, + "p99_ms": 0.07230202609207485, + "mean_ms": 0.05035261661396362 + }, + { + "knob": 64, + "nq": 2000, + "recall": 0.5928, + "qps_1t": 14100.305376488077, + "qps_1t_runs": [ + 14100.305376488077, + 14085.002939840428, + 15017.044303743827 + ], + "median_ms": 0.07399998139590025, + "p95_ms": 0.08390000148210675, + "p99_ms": 0.09510102507192644, + "mean_ms": 0.0695029834265976 + }, + { + "knob": 1265, + "nq": 2000, + "recall": 1.0, + "qps_1t": 1909.7279259837674, + "qps_1t_runs": [ + 1959.5918473009788, + 1907.8686436107294, + 1909.7279259837674 + ], + "median_ms": 0.5600000149570405, + "p95_ms": 0.609705041279085, + "p99_ms": 0.6907009630231187, + "mean_ms": 0.5193634161454005 + } + ] + } + } + }, + "faiss_ivfflat|sift|1000000": { + "mem_baseline": { + "rss_mb": 35.5703125, + "peak_mb": 35.5703125 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 6.360013503581285e-05, + "index_s": 26.820929800043814, + "total_s": 26.82099340017885, + "first_query_ms": 77.96510000480339, + "ready_s": 26.898958500183653 + }, + "mem_after_build": { + "rss_mb": 719.046875, + "peak_mb": 1293.51171875 + }, + "mem_after_first_query": { + "rss_mb": 719.1953125, + "peak_mb": 1293.51171875 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.2635, + "qps_1t_runs": [ + 9853.396307648662, + 10714.077144156134, + 9769.207334896133 + ], + "qps_1t": 9853.396307648662, + "qps_1t_min": 9769.207334896133, + "qps_1t_max": 10714.077144156134, + "median_ms": 0.0974999857135117, + "p95_ms": 0.13760500005446377, + "p99_ms": 0.17810201679822063, + "mean_ms": 0.09906181662033002, + "knob": 1, + "batch": { + "recall": 0.2635, + "qps_batch_runs": [ + 35564.913062274325, + 47615.193074488714, + 47014.90608411602 + ], + "qps_batch": 47014.90608411602, + "qps_batch_min": 35564.913062274325, + "qps_batch_max": 47615.193074488714, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.3934, + "qps_1t_runs": [ + 9319.021735079097, + 9352.19235245332, + 9919.242489558554 + ], + "qps_1t": 9352.19235245332, + "qps_1t_min": 9319.021735079097, + "qps_1t_max": 9919.242489558554, + "median_ms": 0.10659999679774046, + "p95_ms": 0.14620000729337335, + "p99_ms": 0.20221102633513535, + "mean_ms": 0.10501611686777323, + "knob": 2, + "batch": { + "recall": 0.3934, + "qps_batch_runs": [ + 53014.96081705539, + 58774.26280339925, + 52214.004336904516 + ], + "qps_batch": 53014.96081705539, + "qps_batch_min": 52214.004336904516, + "qps_batch_max": 58774.26280339925, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.54085, + "qps_1t_runs": [ + 6865.368706107131, + 7229.084632254392, + 7675.100713065397 + ], + "qps_1t": 7229.084632254392, + "qps_1t_min": 6865.368706107131, + "qps_1t_max": 7675.100713065397, + "median_ms": 0.13499998021870852, + "p95_ms": 0.19749999046325684, + "p99_ms": 0.25320000713691115, + "mean_ms": 0.13809338357532397, + "knob": 4, + "batch": { + "recall": 0.54085, + "qps_batch_runs": [ + 38314.24961928604, + 41490.50490655643, + 37140.54914806244 + ], + "qps_batch": 38314.24961928604, + "qps_batch_min": 37140.54914806244, + "qps_batch_max": 41490.50490655643, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.69175, + "qps_1t_runs": [ + 4758.556786293455, + 5567.311851330581, + 5382.155942135849 + ], + "qps_1t": 5382.155942135849, + "qps_1t_min": 4758.556786293455, + "qps_1t_max": 5567.311851330581, + "median_ms": 0.18660002388060093, + "p95_ms": 0.28009997913613915, + "p99_ms": 0.3491160389967266, + "mean_ms": 0.191855600035827, + "knob": 8, + "batch": { + "recall": 0.69175, + "qps_batch_runs": [ + 24214.921968571074, + 23059.140929699992, + 22450.7458757803 + ], + "qps_batch": 23059.140929699992, + "qps_batch_min": 22450.7458757803, + "qps_batch_max": 24214.921968571074, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.82295, + "qps_1t_runs": [ + 3540.9768010642715, + 3727.9762848837395, + 3576.300002495727 + ], + "qps_1t": 3576.300002495727, + "qps_1t_min": 3540.9768010642715, + "qps_1t_max": 3727.9762848837395, + "median_ms": 0.26699999580159783, + "p95_ms": 0.40949997783172876, + "p99_ms": 0.48300495254807185, + "mean_ms": 0.2767561999789905, + "knob": 16, + "batch": { + "recall": 0.82295, + "qps_batch_runs": [ + 13486.567713335664, + 14027.380040111697, + 13782.505177573274 + ], + "qps_batch": 13782.505177573274, + "qps_batch_min": 13486.567713335664, + "qps_batch_max": 14027.380040111697, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9173, + "qps_1t_runs": [ + 2230.347132253294, + 2167.904181056244, + 2208.673237858 + ], + "qps_1t": 2208.673237858, + "qps_1t_min": 2167.904181056244, + "qps_1t_max": 2230.347132253294, + "median_ms": 0.4336500132922083, + "p95_ms": 0.663514988264069, + "p99_ms": 0.7766079762950542, + "mean_ms": 0.4541320663508183, + "knob": 32, + "batch": { + "recall": 0.9173, + "qps_batch_runs": [ + 7504.127269866864, + 7912.563014781055, + 7423.519194051266 + ], + "qps_batch": 7504.127269866864, + "qps_batch_min": 7423.519194051266, + "qps_batch_max": 7912.563014781055, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.97005, + "qps_1t_runs": [ + 1269.8222424623443, + 1270.4119468765543, + 1265.9179692673463 + ], + "qps_1t": 1269.8222424623443, + "qps_1t_min": 1265.9179692673463, + "qps_1t_max": 1270.4119468765543, + "median_ms": 0.7607000297866762, + "p95_ms": 1.1436299770139158, + "p99_ms": 1.302521022735164, + "mean_ms": 0.7881995500065386, + "knob": 64, + "batch": { + "recall": 0.97005, + "qps_batch_runs": [ + 3831.8756874844803, + 3253.9298524527885, + 3108.588595018012 + ], + "qps_batch": 3253.9298524527885, + "qps_batch_min": 3108.588595018012, + "qps_batch_max": 3831.8756874844803, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9923, + "qps_1t_runs": [ + 678.4205974514858, + 662.9495085604967, + 668.0849664744334 + ], + "qps_1t": 668.0849664744334, + "qps_1t_min": 662.9495085604967, + "qps_1t_max": 678.4205974514858, + "median_ms": 1.4595000247936696, + "p95_ms": 2.0172150485450406, + "p99_ms": 2.264212010777556, + "mean_ms": 1.4930793168896344, + "knob": 128, + "batch": { + "recall": 0.9923, + "qps_batch_runs": [ + 1990.1527242880686, + 2325.5202828837096, + 2590.068666695632 + ], + "qps_batch": 2325.5202828837096, + "qps_batch_min": 1990.1527242880686, + "qps_batch_max": 2590.068666695632, + "threads": 32 + } + }, + { + "nq": 1035, + "recall_queries": 1035, + "recall": 0.9983574879227053, + "qps_1t_runs": [ + 345.9760561921172, + 362.5302466304616, + 358.0662182325903 + ], + "qps_1t": 358.0662182325903, + "qps_1t_min": 345.9760561921172, + "qps_1t_max": 362.5302466304616, + "median_ms": 2.798899949993938, + "p95_ms": 3.625900007318705, + "p99_ms": 4.040848035365343, + "mean_ms": 2.813847826219399, + "knob": 256, + "batch": { + "recall": 0.9982, + "qps_batch_runs": [ + 1224.0601757720042, + 1106.9108150222319, + 1114.6970295162796 + ], + "qps_batch": 1114.6970295162796, + "qps_batch_min": 1106.9108150222319, + "qps_batch_max": 1224.0601757720042, + "threads": 32 + } + }, + { + "nq": 86, + "recall_queries": 86, + "recall": 0.9988372093023256, + "qps_1t_runs": [ + 26.94771588663001, + 27.40591159928293, + 28.092552287989317 + ], + "qps_1t": 27.40591159928293, + "qps_1t_min": 26.94771588663001, + "qps_1t_max": 28.092552287989317, + "median_ms": 36.22710003401153, + "p95_ms": 39.043385011609644, + "p99_ms": 40.78133401402738, + "mean_ms": 36.39799922405987, + "knob": 4000, + "batch": { + "recall": 0.9994, + "qps_batch_runs": [ + 78.89019513958107, + 81.98307717623237, + 79.83002750199307 + ], + "qps_batch": 79.83002750199307, + "qps_batch_min": 78.89019513958107, + "qps_batch_max": 81.98307717623237, + "threads": 32 + } + } + ], + "saveload": { + "save_s": 0.35315360000822693, + "disk_mb": 497.894419670105, + "load_s": 0.7231700000120327, + "first_query_after_load_ms": 1.6272999928332865, + "time_to_ready_s": 0.724797300004866 + }, + "load_before_pct": 12.340624999999987, + "load_contaminated": false + }, + "faiss_ivfflat|sift|100000": { + "mem_baseline": { + "rss_mb": 35.3671875, + "peak_mb": 35.3671875 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 7.90000194683671e-06, + "index_s": 1.2938719000085257, + "total_s": 1.2938798000104725, + "first_query_ms": 6.0953000211156905, + "ready_s": 1.2999751000315882, + "total_s_runs": [ + 1.2938798000104725, + 1.2565003000781871, + 1.2489298000000417 + ] + }, + "mem_after_build": { + "rss_mb": 140.640625, + "peak_mb": 216.1875 + }, + "mem_after_first_query": { + "rss_mb": 140.765625, + "peak_mb": 216.1875 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.28365, + "qps_1t_runs": [ + 45779.16058838538, + 47166.47381183704, + 46449.19107151231 + ], + "qps_1t": 46449.19107151231, + "qps_1t_min": 45779.16058838538, + "qps_1t_max": 47166.47381183704, + "median_ms": 0.020799983758479357, + "p95_ms": 0.02540001878514886, + "p99_ms": 0.030800001695752144, + "mean_ms": 0.02152480023020568, + "knob": 1, + "batch": { + "recall": 0.28365, + "qps_batch_runs": [ + 260844.61482609718, + 260023.9218380907, + 269030.5494736345 + ], + "qps_batch": 260844.61482609718, + "qps_batch_min": 260023.9218380907, + "qps_batch_max": 269030.5494736345, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.4257, + "qps_1t_runs": [ + 40290.73925960876, + 39424.326068370654, + 39479.97042904718 + ], + "qps_1t": 39479.97042904718, + "qps_1t_min": 39424.326068370654, + "qps_1t_max": 40290.73925960876, + "median_ms": 0.023900007363408804, + "p95_ms": 0.03130000550299883, + "p99_ms": 0.040902020409703296, + "mean_ms": 0.025171316257910803, + "knob": 2, + "batch": { + "recall": 0.4257, + "qps_batch_runs": [ + 299056.47718483285, + 250400.64129287703, + 242045.77099248278 + ], + "qps_batch": 250400.64129287703, + "qps_batch_min": 242045.77099248278, + "qps_batch_max": 299056.47718483285, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.5849, + "qps_1t_runs": [ + 32811.57505688234, + 30978.31057355483, + 28531.891573655146 + ], + "qps_1t": 30978.31057355483, + "qps_1t_min": 28531.891573655146, + "qps_1t_max": 32811.57505688234, + "median_ms": 0.029799994081258774, + "p95_ms": 0.041605025762692115, + "p99_ms": 0.06180000491440296, + "mean_ms": 0.03260206649429165, + "knob": 4, + "batch": { + "recall": 0.5849, + "qps_batch_runs": [ + 228287.0495511245, + 215375.66797053098, + 221614.01404819093 + ], + "qps_batch": 221614.01404819093, + "qps_batch_min": 215375.66797053098, + "qps_batch_max": 228287.0495511245, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7447, + "qps_1t_runs": [ + 22574.71396180766, + 24141.672987691236, + 21736.295353263977 + ], + "qps_1t": 22574.71396180766, + "qps_1t_min": 21736.295353263977, + "qps_1t_max": 24141.672987691236, + "median_ms": 0.040599959902465343, + "p95_ms": 0.05829998117405921, + "p99_ms": 0.08330805168952811, + "mean_ms": 0.043908499860359974, + "knob": 8, + "batch": { + "recall": 0.7447, + "qps_batch_runs": [ + 179953.21226942915, + 207688.6335376515, + 212357.05710580616 + ], + "qps_batch": 207688.6335376515, + "qps_batch_min": 179953.21226942915, + "qps_batch_max": 212357.05710580616, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.86855, + "qps_1t_runs": [ + 15617.532835481019, + 14663.971379752347, + 14009.840427012357 + ], + "qps_1t": 14663.971379752347, + "qps_1t_min": 14009.840427012357, + "qps_1t_max": 15617.532835481019, + "median_ms": 0.06360001862049103, + "p95_ms": 0.09540497267153115, + "p99_ms": 0.13230801792815344, + "mean_ms": 0.06786778359673917, + "knob": 16, + "batch": { + "recall": 0.86855, + "qps_batch_runs": [ + 144067.3088262884, + 142585.21253684963, + 146522.29360921946 + ], + "qps_batch": 144067.3088262884, + "qps_batch_min": 142585.21253684963, + "qps_batch_max": 146522.29360921946, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.95045, + "qps_1t_runs": [ + 8823.154425692188, + 8763.631273373754, + 8546.840522530436 + ], + "qps_1t": 8763.631273373754, + "qps_1t_min": 8546.840522530436, + "qps_1t_max": 8823.154425692188, + "median_ms": 0.10529998689889908, + "p95_ms": 0.15952499234117576, + "p99_ms": 0.24826096952892973, + "mean_ms": 0.11481613341796522, + "knob": 32, + "batch": { + "recall": 0.95045, + "qps_batch_runs": [ + 112537.20766997218, + 120023.5246781843, + 108717.51366544623 + ], + "qps_batch": 112537.20766997218, + "qps_batch_min": 108717.51366544623, + "qps_batch_max": 120023.5246781843, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9876, + "qps_1t_runs": [ + 4987.015040976421, + 5212.723412740188, + 4990.789484955886 + ], + "qps_1t": 4990.789484955886, + "qps_1t_min": 4987.015040976421, + "qps_1t_max": 5212.723412740188, + "median_ms": 0.18099998123943806, + "p95_ms": 0.278900028206408, + "p99_ms": 0.6100069882813842, + "mean_ms": 0.19757605055929162, + "knob": 64, + "batch": { + "recall": 0.9876, + "qps_batch_runs": [ + 51455.54878984149, + 51942.520410659665, + 51490.52186920943 + ], + "qps_batch": 51490.52186920943, + "qps_batch_min": 51455.54878984149, + "qps_batch_max": 51942.520410659665, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9986, + "qps_1t_runs": [ + 2803.8677723626506, + 2623.92950416551, + 2812.937541746591 + ], + "qps_1t": 2803.8677723626506, + "qps_1t_min": 2623.92950416551, + "qps_1t_max": 2812.937541746591, + "median_ms": 0.34009997034445405, + "p95_ms": 0.5055150075349957, + "p99_ms": 0.8544079866260292, + "mean_ms": 0.36441941644685966, + "knob": 128, + "batch": { + "recall": 0.9986, + "qps_batch_runs": [ + 30326.142518890818, + 28045.41675089572, + 29165.105100498964 + ], + "qps_batch": 29165.105100498964, + "qps_batch_min": 28045.41675089572, + "qps_batch_max": 30326.142518890818, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99995, + "qps_1t_runs": [ + 1475.1417608256018, + 1423.1979422029708, + 1438.301331700387 + ], + "qps_1t": 1438.301331700387, + "qps_1t_min": 1423.1979422029708, + "qps_1t_max": 1475.1417608256018, + "median_ms": 0.6542000046465546, + "p95_ms": 1.0054100275738165, + "p99_ms": 1.5070700080832482, + "mean_ms": 0.6919361662876327, + "knob": 256, + "batch": { + "recall": 0.99995, + "qps_batch_runs": [ + 15345.661012639717, + 16373.727046861477, + 15784.821316471978 + ], + "qps_batch": 15784.821316471978, + "qps_batch_min": 15345.661012639717, + "qps_batch_max": 16373.727046861477, + "threads": 32 + } + }, + { + "nq": 988, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 307.81225947274436, + 299.73582896198633, + 310.6211038313147 + ], + "qps_1t": 307.81225947274436, + "qps_1t_min": 299.73582896198633, + "qps_1t_max": 310.6211038313147, + "median_ms": 3.133050020551309, + "p95_ms": 3.9111950376536693, + "p99_ms": 4.8217820015270245, + "mean_ms": 3.2681203109505064, + "knob": 1265, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 3785.6625977814692, + 3415.4416901498516, + 3461.6845182099487 + ], + "qps_batch": 3461.6845182099487, + "qps_batch_min": 3415.4416901498516, + "qps_batch_max": 3785.6625977814692, + "threads": 32 + } + } + ], + "mutation": { + "ref_knob": 32, + "recall_before": 0.95045, + "insert_10k": { + "seconds": 0.03519810002762824, + "per_s": 284106.2441481399, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.0634624999947846, + "per_s": 157573.37011340255, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.0035829999833367765, + "per_s": 2790957.3113330575, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 168.4140625, + "peak_mb": 290.11328125 + }, + "first_query_after_ms": 0.43760001426562667, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.94675, + "qps_1t_runs": [ + 8355.81043831318, + 9084.48800273748, + 8754.639393979862 + ], + "qps_1t": 8754.639393979862, + "qps_1t_min": 8355.81043831318, + "qps_1t_max": 9084.48800273748, + "median_ms": 0.10555001790635288, + "p95_ms": 0.15769998135510832, + "p99_ms": 0.22260300698690122, + "mean_ms": 0.11466003355841774 + }, + "recall_after": 0.94675 + }, + "saveload": { + "save_s": 0.02542570000514388, + "disk_mb": 50.21852397918701, + "load_s": 0.03533669997705147, + "first_query_after_load_ms": 0.2161000156775117, + "time_to_ready_s": 0.03555279999272898 + }, + "load_before_pct": 9.115625000000003, + "load_contaminated": false + }, + "faiss_ivfflat|text384|1000000": { + "mem_baseline": { + "rss_mb": 37.30859375, + "peak_mb": 37.30859375 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 6.080017192289233e-05, + "index_s": 18.49423830001615, + "total_s": 18.494299100188073, + "first_query_ms": 80.72970004286617, + "ready_s": 18.57502880023094 + }, + "mem_after_build": { + "rss_mb": 1922.3203125, + "peak_mb": 3629.421875 + }, + "mem_after_first_query": { + "rss_mb": 1922.4453125, + "peak_mb": 3629.421875 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.27105, + "qps_1t_runs": [ + 5732.901260874405, + 5665.234498165314, + 5464.012098871559 + ], + "qps_1t": 5665.234498165314, + "qps_1t_min": 5464.012098871559, + "qps_1t_max": 5732.901260874405, + "median_ms": 0.16820000018924475, + "p95_ms": 0.21871003555133942, + "p99_ms": 0.3441310004564009, + "mean_ms": 0.17798755005545294, + "knob": 1, + "batch": { + "recall": 0.27105, + "qps_batch_runs": [ + 53177.48784421624, + 43160.468418000586, + 48472.74502099718 + ], + "qps_batch": 48472.74502099718, + "qps_batch_min": 43160.468418000586, + "qps_batch_max": 53177.48784421624, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.38275, + "qps_1t_runs": [ + 4593.111188797486, + 4539.455240808732, + 4849.87925672094 + ], + "qps_1t": 4593.111188797486, + "qps_1t_min": 4539.455240808732, + "qps_1t_max": 4849.87925672094, + "median_ms": 0.1965999836102128, + "p95_ms": 0.2596049977000803, + "p99_ms": 0.4619120241841304, + "mean_ms": 0.21473293371188143, + "knob": 2, + "batch": { + "recall": 0.38275, + "qps_batch_runs": [ + 29571.49425121084, + 34184.644924359236, + 33893.99652168202 + ], + "qps_batch": 33893.99652168202, + "qps_batch_min": 29571.49425121084, + "qps_batch_max": 34184.644924359236, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.4944, + "qps_1t_runs": [ + 3766.98700524271, + 3673.866454891245, + 3513.13561316215 + ], + "qps_1t": 3673.866454891245, + "qps_1t_min": 3513.13561316215, + "qps_1t_max": 3766.98700524271, + "median_ms": 0.25450001703575253, + "p95_ms": 0.3500999911921099, + "p99_ms": 0.653701971168632, + "mean_ms": 0.274100983311655, + "knob": 4, + "batch": { + "recall": 0.4944, + "qps_batch_runs": [ + 17023.66715560851, + 17039.54878939169, + 17988.992535419085 + ], + "qps_batch": 17039.54878939169, + "qps_batch_min": 17023.66715560851, + "qps_batch_max": 17988.992535419085, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.60865, + "qps_1t_runs": [ + 2710.319803372828, + 2699.503793986944, + 2773.988587727927 + ], + "qps_1t": 2710.319803372828, + "qps_1t_min": 2699.503793986944, + "qps_1t_max": 2773.988587727927, + "median_ms": 0.3511499962769449, + "p95_ms": 0.4747050057630986, + "p99_ms": 0.6702169805066686, + "mean_ms": 0.366630116312687, + "knob": 8, + "batch": { + "recall": 0.60865, + "qps_batch_runs": [ + 11569.300401222521, + 11014.374310297293, + 11517.08847674131 + ], + "qps_batch": 11517.08847674131, + "qps_batch_min": 11014.374310297293, + "qps_batch_max": 11569.300401222521, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7099, + "qps_1t_runs": [ + 1643.6749417390806, + 1693.4195893400558, + 1826.5639197918922 + ], + "qps_1t": 1693.4195893400558, + "qps_1t_min": 1643.6749417390806, + "qps_1t_max": 1826.5639197918922, + "median_ms": 0.5374000174924731, + "p95_ms": 0.7504300097934903, + "p99_ms": 1.50892302277498, + "mean_ms": 0.5821299836000738, + "knob": 16, + "batch": { + "recall": 0.7099, + "qps_batch_runs": [ + 6113.211169174637, + 6334.993840232643, + 6263.064361043098 + ], + "qps_batch": 6263.064361043098, + "qps_batch_min": 6113.211169174637, + "qps_batch_max": 6334.993840232643, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7952, + "qps_1t_runs": [ + 1086.8689221971993, + 1069.1211977486498, + 1079.1109751208205 + ], + "qps_1t": 1079.1109751208205, + "qps_1t_min": 1069.1211977486498, + "qps_1t_max": 1086.8689221971993, + "median_ms": 0.8744499937165529, + "p95_ms": 1.1628400068730118, + "p99_ms": 1.6152060148306258, + "mean_ms": 0.927370183247452, + "knob": 32, + "batch": { + "recall": 0.7952, + "qps_batch_runs": [ + 3435.3677637225355, + 3438.737515409886, + 3173.5594264085485 + ], + "qps_batch": 3435.3677637225355, + "qps_batch_min": 3173.5594264085485, + "qps_batch_max": 3438.737515409886, + "threads": 32 + } + }, + { + "nq": 1690, + "recall_queries": 1690, + "recall": 0.8620118343195267, + "qps_1t_runs": [ + 597.9830632198161, + 577.9825118404891, + 578.6310854243291 + ], + "qps_1t": 578.6310854243291, + "qps_1t_min": 577.9825118404891, + "qps_1t_max": 597.9830632198161, + "median_ms": 1.5928999637253582, + "p95_ms": 2.0974849903723225, + "p99_ms": 3.3699640113627956, + "mean_ms": 1.7102203948291925, + "knob": 64, + "batch": { + "recall": 0.8651, + "qps_batch_runs": [ + 1747.707270206291, + 1818.0234848510586, + 1783.6145572898147 + ], + "qps_batch": 1783.6145572898147, + "qps_batch_min": 1747.707270206291, + "qps_batch_max": 1818.0234848510586, + "threads": 32 + } + }, + { + "nq": 986, + "recall_queries": 1000, + "recall": 0.9183, + "qps_1t_runs": [ + 314.76592156322806, + 324.40771256497766, + 321.21529615018915 + ], + "qps_1t": 321.21529615018915, + "qps_1t_min": 314.76592156322806, + "qps_1t_max": 324.40771256497766, + "median_ms": 2.9966000001877546, + "p95_ms": 3.7562450132099916, + "p99_ms": 5.053043008665543, + "mean_ms": 3.1242271805343615, + "knob": 128, + "batch": { + "recall": 0.91875, + "qps_batch_runs": [ + 910.3805044678606, + 907.4954907645792, + 913.484812563989 + ], + "qps_batch": 910.3805044678606, + "qps_batch_min": 907.4954907645792, + "qps_batch_max": 913.484812563989, + "threads": 32 + } + }, + { + "nq": 541, + "recall_queries": 1000, + "recall": 0.9561, + "qps_1t_runs": [ + 171.32075374782696, + 171.2566129190882, + 169.00607316506571 + ], + "qps_1t": 171.2566129190882, + "qps_1t_min": 169.00607316506571, + "qps_1t_max": 171.32075374782696, + "median_ms": 5.629299965221435, + "p95_ms": 7.034130021929739, + "p99_ms": 13.249214024981484, + "mean_ms": 5.864380715166415, + "knob": 256, + "batch": { + "recall": 0.9572, + "qps_batch_runs": [ + 466.90232160621355, + 468.0216267183672, + 456.78724174622664 + ], + "qps_batch": 466.90232160621355, + "qps_batch_min": 456.78724174622664, + "qps_batch_max": 468.0216267183672, + "threads": 32 + } + }, + { + "nq": 38, + "recall_queries": 38, + "recall": 1.0, + "qps_1t_runs": [ + 12.34720496393977, + 12.22943029109082, + 12.599246838612803 + ], + "qps_1t": 12.34720496393977, + "qps_1t_min": 12.22943029109082, + "qps_1t_max": 12.599246838612803, + "median_ms": 78.59804999316111, + "p95_ms": 89.40011499798857, + "p99_ms": 100.34008598478978, + "mean_ms": 80.70992367962997, + "knob": 4000, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 30.929301084455542, + 30.449447254460516, + 27.25492746363667 + ], + "qps_batch": 30.449447254460516, + "qps_batch_min": 27.25492746363667, + "qps_batch_max": 30.929301084455542, + "threads": 32 + } + } + ], + "mutation": { + "ref_knob": 256, + "recall_before": 0.9561, + "insert_10k": { + "seconds": 0.12511570000788197, + "per_s": 79926.0204704128, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.13949179998598993, + "per_s": 71688.80178622948, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.004724500002339482, + "per_s": 2116626.0969516756, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 1976.7578125, + "peak_mb": 3629.421875 + }, + "first_query_after_ms": 8.726300031412393, + "after": { + "nq": 502, + "recall_queries": 1000, + "recall": 0.9571000000000001, + "qps_1t_runs": [ + 159.49925111018794, + 157.23342836365086, + 157.79887793672094 + ], + "qps_1t": 157.79887793672094, + "qps_1t_min": 157.23342836365086, + "qps_1t_max": 159.49925111018794, + "median_ms": 6.059849984012544, + "p95_ms": 7.609975000377744, + "p99_ms": 13.724885028204888, + "mean_ms": 6.3222577687706005 + }, + "recall_after": 0.9571000000000001 + }, + "saveload": { + "save_s": 0.41686230001505464, + "disk_mb": 1478.363169670105, + "load_s": 0.6695059000048786, + "first_query_after_load_ms": 6.217199959792197, + "time_to_ready_s": 0.6757230999646708 + }, + "load_before_pct": 8.796874999999998, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 46.97382059984375, + "mem": { + "rss_mb": 1913.45703125, + "peak_mb": 3619.25 + }, + "subsets": { + "f10": [ + { + "knob": 2, + "nq": 2000, + "recall": 0.31405, + "qps_1t": 3902.1382789327786, + "qps_1t_runs": [ + 3902.1382789327786, + 4053.5489980521975, + 3787.378114240684 + ], + "median_ms": 0.26890001026913524, + "p95_ms": 0.31049997778609395, + "p99_ms": 0.3722069825744258, + "mean_ms": 0.2556673498669018 + }, + { + "knob": 8, + "nq": 2000, + "recall": 0.54505, + "qps_1t": 3212.525763997164, + "qps_1t_runs": [ + 3199.3596261992006, + 3315.013461573325, + 3212.525763997164 + ], + "median_ms": 0.32130000181496143, + "p95_ms": 0.37839997094124556, + "p99_ms": 0.4318020283244551, + "mean_ms": 0.30850066649145447 + }, + { + "knob": 32, + "nq": 2000, + "recall": 0.7427, + "qps_1t": 2010.0108607920495, + "qps_1t_runs": [ + 2010.0108607920495, + 2052.8251439403884, + 2006.1228889847303 + ], + "median_ms": 0.5012999754399061, + "p95_ms": 0.6028999923728406, + "p99_ms": 0.738600977929309, + "mean_ms": 0.49437241643317975 + }, + { + "knob": 64, + "nq": 2000, + "recall": 0.82635, + "qps_1t": 1419.4566932721766, + "qps_1t_runs": [ + 1461.192342100759, + 1419.4566932721766, + 1410.3788514493797 + ], + "median_ms": 0.7134999905247241, + "p95_ms": 0.8912149758543819, + "p99_ms": 1.0548009991180152, + "mean_ms": 0.6992989491360883 + }, + { + "knob": 4000, + "nq": 118, + "recall": 1.0, + "qps_1t": 38.09917777729757, + "qps_1t_runs": [ + 38.09917777729757, + 36.752284114688536, + 38.722345062547184 + ], + "median_ms": 26.602250029100105, + "p95_ms": 29.446885021752678, + "p99_ms": 31.17722800117917, + "mean_ms": 26.427120342258743 + } + ], + "f01": [ + { + "knob": 2, + "nq": 2000, + "recall": 0.1722, + "qps_1t": 4159.5322092282795, + "qps_1t_runs": [ + 4159.5322092282795, + 4096.676660594222, + 4263.111564658021 + ], + "median_ms": 0.2539000124670565, + "p95_ms": 0.2825999981723726, + "p99_ms": 0.3076009562937543, + "mean_ms": 0.23969413319719024 + }, + { + "knob": 8, + "nq": 2000, + "recall": 0.3754, + "qps_1t": 3921.391784770546, + "qps_1t_runs": [ + 3937.841930935292, + 3776.852233549111, + 3921.391784770546 + ], + "median_ms": 0.2696499868761748, + "p95_ms": 0.29510497697629035, + "p99_ms": 0.31650102289859205, + "mean_ms": 0.25790948382928036 + }, + { + "knob": 32, + "nq": 2000, + "recall": 0.61645, + "qps_1t": 3325.8003924184354, + "qps_1t_runs": [ + 3325.8003924184354, + 3180.073404658829, + 3413.454508696176 + ], + "median_ms": 0.32049999572336674, + "p95_ms": 0.3630099701695145, + "p99_ms": 0.4293040093034507, + "mean_ms": 0.3026986667246092 + }, + { + "knob": 64, + "nq": 2000, + "recall": 0.7293, + "qps_1t": 2711.3313291631353, + "qps_1t_runs": [ + 2748.871477922609, + 2697.958672405094, + 2711.3313291631353 + ], + "median_ms": 0.38700003642588854, + "p95_ms": 0.4525050084339455, + "p99_ms": 0.6172019854420796, + "mean_ms": 0.3677529170042059 + }, + { + "knob": 4000, + "nq": 414, + "recall": 1.0, + "qps_1t": 140.23270566994586, + "qps_1t_runs": [ + 143.38686778958785, + 138.5265347887052, + 140.23270566994586 + ], + "median_ms": 7.258399971760809, + "p95_ms": 8.107459978782572, + "p99_ms": 8.53707101137843, + "mean_ms": 7.1079923501185736 + } + ] + } + } + }, + "faiss_ivfflat|text384|100000": { + "mem_baseline": { + "rss_mb": 37.5546875, + "peak_mb": 37.5546875 + }, + "version": "faiss-cpu 1.16.0", + "build": { + "ingest_s": 8.200004231184721e-06, + "index_s": 1.7210487000411376, + "total_s": 1.7210569000453688, + "first_query_ms": 15.245300019159913, + "ready_s": 1.7363022000645287, + "total_s_runs": [ + 1.7210569000453688, + 1.854149300022982, + 1.829295700066723 + ] + }, + "mem_after_build": { + "rss_mb": 275.8515625, + "peak_mb": 496.47265625 + }, + "mem_after_first_query": { + "rss_mb": 275.98046875, + "peak_mb": 496.47265625 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.2834, + "qps_1t_runs": [ + 18898.46518103432, + 18005.803256934512, + 18763.310235400597 + ], + "qps_1t": 18763.310235400597, + "qps_1t_min": 18005.803256934512, + "qps_1t_max": 18898.46518103432, + "median_ms": 0.052099989261478186, + "p95_ms": 0.06479996955022216, + "p99_ms": 0.08200098411180082, + "mean_ms": 0.05391583331705382, + "knob": 1, + "batch": { + "recall": 0.2834, + "qps_batch_runs": [ + 212442.7725172613, + 181796.6966135266, + 192036.2570791064 + ], + "qps_batch": 192036.2570791064, + "qps_batch_min": 181796.6966135266, + "qps_batch_max": 212442.7725172613, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.40995, + "qps_1t_runs": [ + 16180.58840526749, + 16032.848155309437, + 16022.110509262971 + ], + "qps_1t": 16032.848155309437, + "qps_1t_min": 16022.110509262971, + "qps_1t_max": 16180.58840526749, + "median_ms": 0.060399994254112244, + "p95_ms": 0.07639999967068434, + "p99_ms": 0.09510102507192644, + "mean_ms": 0.062196049810154364, + "knob": 2, + "batch": { + "recall": 0.40995, + "qps_batch_runs": [ + 168357.25406953352, + 172376.6431441996, + 176973.9225600307 + ], + "qps_batch": 172376.6431441996, + "qps_batch_min": 168357.25406953352, + "qps_batch_max": 176973.9225600307, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.5287, + "qps_1t_runs": [ + 12948.629586596435, + 12664.199290334738, + 11615.966841498226 + ], + "qps_1t": 12664.199290334738, + "qps_1t_min": 11615.966841498226, + "qps_1t_max": 12948.629586596435, + "median_ms": 0.07720000576227903, + "p95_ms": 0.10081000800710176, + "p99_ms": 0.14101596781984008, + "mean_ms": 0.0807597998646088, + "knob": 4, + "batch": { + "recall": 0.5287, + "qps_batch_runs": [ + 124651.7542054606, + 116179.10171446901, + 99298.94919005745 + ], + "qps_batch": 116179.10171446901, + "qps_batch_min": 99298.94919005745, + "qps_batch_max": 124651.7542054606, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6503, + "qps_1t_runs": [ + 8997.951993443592, + 9414.086148625704, + 9036.0928464684 + ], + "qps_1t": 9036.0928464684, + "qps_1t_min": 8997.951993443592, + "qps_1t_max": 9414.086148625704, + "median_ms": 0.1066999975591898, + "p95_ms": 0.13449997641146183, + "p99_ms": 0.17090502078644942, + "mean_ms": 0.10934250022789153, + "knob": 8, + "batch": { + "recall": 0.6503, + "qps_batch_runs": [ + 76763.05559563964, + 86381.52109212214, + 84703.76978550976 + ], + "qps_batch": 84703.76978550976, + "qps_batch_min": 76763.05559563964, + "qps_batch_max": 86381.52109212214, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.75515, + "qps_1t_runs": [ + 6280.884320968016, + 5829.1329518186985, + 6072.218710602774 + ], + "qps_1t": 6072.218710602774, + "qps_1t_min": 5829.1329518186985, + "qps_1t_max": 6280.884320968016, + "median_ms": 0.15670002903789282, + "p95_ms": 0.19890000112354755, + "p99_ms": 0.2494169783312831, + "mean_ms": 0.16514993316377513, + "knob": 16, + "batch": { + "recall": 0.75515, + "qps_batch_runs": [ + 38405.26001015603, + 35027.8646929996, + 40084.417782680204 + ], + "qps_batch": 38405.26001015603, + "qps_batch_min": 35027.8646929996, + "qps_batch_max": 40084.417782680204, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8439, + "qps_1t_runs": [ + 3760.506864140523, + 3838.3295611262834, + 3701.0252573141947 + ], + "qps_1t": 3760.506864140523, + "qps_1t_min": 3701.0252573141947, + "qps_1t_max": 3838.3295611262834, + "median_ms": 0.25549999554641545, + "p95_ms": 0.324005022412166, + "p99_ms": 0.3853259654715663, + "mean_ms": 0.26554899978024576, + "knob": 32, + "batch": { + "recall": 0.8439, + "qps_batch_runs": [ + 23326.60749431601, + 24435.65846811203, + 22952.889184927895 + ], + "qps_batch": 23326.60749431601, + "qps_batch_min": 22952.889184927895, + "qps_batch_max": 24435.65846811203, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.90965, + "qps_1t_runs": [ + 2080.7086007328244, + 2078.0254932820503, + 2072.0381349543754 + ], + "qps_1t": 2078.0254932820503, + "qps_1t_min": 2072.0381349543754, + "qps_1t_max": 2080.7086007328244, + "median_ms": 0.458099995739758, + "p95_ms": 0.5707149801310152, + "p99_ms": 0.6885090039577337, + "mean_ms": 0.4814827171309541, + "knob": 64, + "batch": { + "recall": 0.90965, + "qps_batch_runs": [ + 10561.976356621335, + 11828.524250734354, + 11707.936167779182 + ], + "qps_batch": 11707.936167779182, + "qps_batch_min": 10561.976356621335, + "qps_batch_max": 11828.524250734354, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9549, + "qps_1t_runs": [ + 1102.7866649101884, + 1153.5756408268974, + 1115.5847295945691 + ], + "qps_1t": 1115.5847295945691, + "qps_1t_min": 1102.7866649101884, + "qps_1t_max": 1153.5756408268974, + "median_ms": 0.8658000151626766, + "p95_ms": 1.0673050332115963, + "p99_ms": 1.2555330136092386, + "mean_ms": 0.890018166557032, + "knob": 128, + "batch": { + "recall": 0.9549, + "qps_batch_runs": [ + 5728.257753342597, + 5564.328647498335, + 5727.952609826657 + ], + "qps_batch": 5727.952609826657, + "qps_batch_min": 5564.328647498335, + "qps_batch_max": 5728.257753342597, + "threads": 32 + } + }, + { + "nq": 1702, + "recall_queries": 1702, + "recall": 0.9840775558166862, + "qps_1t_runs": [ + 566.3842996186253, + 567.0691171290053, + 570.2379160019698 + ], + "qps_1t": 567.0691171290053, + "qps_1t_min": 566.3842996186253, + "qps_1t_max": 570.2379160019698, + "median_ms": 1.7172999796457589, + "p95_ms": 2.000599983148277, + "p99_ms": 2.4252599629107845, + "mean_ms": 1.7608977087543196, + "knob": 256, + "batch": { + "recall": 0.9838, + "qps_batch_runs": [ + 2529.3963278486594, + 2808.9942873922123, + 2721.057707816942 + ], + "qps_batch": 2721.057707816942, + "qps_batch_min": 2529.3963278486594, + "qps_batch_max": 2808.9942873922123, + "threads": 32 + } + }, + { + "nq": 377, + "recall_queries": 1000, + "recall": 0.9999, + "qps_1t_runs": [ + 120.67298137777473, + 119.72903636238806, + 117.61160101239489 + ], + "qps_1t": 119.72903636238806, + "qps_1t_min": 117.61160101239489, + "qps_1t_max": 120.67298137777473, + "median_ms": 8.104599954094738, + "p95_ms": 9.462000016355887, + "p99_ms": 12.528770003700634, + "mean_ms": 8.380538196023801, + "knob": 1265, + "batch": { + "recall": 0.99995, + "qps_batch_runs": [ + 502.46965090218185, + 508.4982128193498, + 505.0018922478272 + ], + "qps_batch": 505.0018922478272, + "qps_batch_min": 502.46965090218185, + "qps_batch_max": 508.4982128193498, + "threads": 32 + } + } + ], + "mutation": { + "ref_knob": 128, + "recall_before": 0.9549, + "insert_10k": { + "seconds": 0.04946050001308322, + "per_s": 202181.53875021107, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.04727209999691695, + "per_s": 211541.2685421674, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.0020620999857783318, + "per_s": 4849425.376541835, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 329.33203125, + "peak_mb": 681.0625 + }, + "first_query_after_ms": 1.2323000119067729, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.951, + "qps_1t_runs": [ + 1061.7292582537762, + 1121.1331069947676, + 1059.470853151706 + ], + "qps_1t": 1061.7292582537762, + "qps_1t_min": 1059.470853151706, + "qps_1t_max": 1121.1331069947676, + "median_ms": 0.8800000068731606, + "p95_ms": 1.1297499877400698, + "p99_ms": 1.6815099975792716, + "mean_ms": 0.9258939498880258 + }, + "recall_after": 0.951 + }, + "saveload": { + "save_s": 0.05474939994746819, + "disk_mb": 149.110125541687, + "load_s": 0.09070970001630485, + "first_query_after_load_ms": 1.2865000171586871, + "time_to_ready_s": 0.09199620003346354 + }, + "load_before_pct": 7.553124999999996, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 1.6864017000771128, + "mem": { + "rss_mb": 273.50390625, + "peak_mb": 494.09375 + }, + "subsets": { + "f10": [ + { + "knob": 2, + "nq": 2000, + "recall": 0.28885, + "qps_1t": 18864.489932928645, + "qps_1t_runs": [ + 18864.489932928645, + 18109.16743247217, + 19321.095468209747 + ], + "median_ms": 0.05059997783973813, + "p95_ms": 0.0674049893859774, + "p99_ms": 0.08729996683541685, + "mean_ms": 0.05332906647042061 + }, + { + "knob": 8, + "nq": 2000, + "recall": 0.52915, + "qps_1t": 15522.997005674655, + "qps_1t_runs": [ + 15518.347649691634, + 15659.19331034095, + 15522.997005674655 + ], + "median_ms": 0.0619000056758523, + "p95_ms": 0.07750000804662704, + "p99_ms": 0.10620199376717214, + "mean_ms": 0.06424021668499336 + }, + { + "knob": 32, + "nq": 2000, + "recall": 0.77835, + "qps_1t": 6741.713155971033, + "qps_1t_runs": [ + 7703.9821735060095, + 6741.713155971033, + 6637.891430592654 + ], + "median_ms": 0.152900000102818, + "p95_ms": 0.19241500995121902, + "p99_ms": 0.22960000205785036, + "mean_ms": 0.14292783334773654 + }, + { + "knob": 64, + "nq": 2000, + "recall": 0.8687, + "qps_1t": 4476.739638094907, + "qps_1t_runs": [ + 4476.739638094907, + 4428.985318577982, + 4478.646372956246 + ], + "median_ms": 0.2300000051036477, + "p95_ms": 0.2772050123894587, + "p99_ms": 0.322203008690849, + "mean_ms": 0.22414798342894451 + }, + { + "knob": 1265, + "nq": 1084, + "recall": 1.0, + "qps_1t": 388.2701160434854, + "qps_1t_runs": [ + 401.68952670004603, + 369.6673733555414, + 388.2701160434854 + ], + "median_ms": 2.6224000030197203, + "p95_ms": 3.2083049823995675, + "p99_ms": 3.5304290201747786, + "mean_ms": 2.5900486785783015 + } + ], + "f01": [ + { + "knob": 2, + "nq": 2000, + "recall": 0.1001, + "qps_1t": 13077.237460110497, + "qps_1t_runs": [ + 13077.237460110497, + 14393.65645248317, + 12968.150237898188 + ], + "median_ms": 0.07670000195503235, + "p95_ms": 0.08709996764082462, + "p99_ms": 0.0955999712459743, + "mean_ms": 0.07435193321240756 + }, + { + "knob": 8, + "nq": 2000, + "recall": 0.26595, + "qps_1t": 12459.40565021944, + "qps_1t_runs": [ + 12457.131851567365, + 12517.320862792621, + 12459.40565021944 + ], + "median_ms": 0.08190004155039787, + "p95_ms": 0.09240000508725643, + "p99_ms": 0.10210102074779573, + "mean_ms": 0.08014175015462872 + }, + { + "knob": 32, + "nq": 2000, + "recall": 0.55835, + "qps_1t": 9902.176417425559, + "qps_1t_runs": [ + 9793.134730328331, + 9902.176417425559, + 10347.48938370298 + ], + "median_ms": 0.10190001921728253, + "p95_ms": 0.11519994877744466, + "p99_ms": 0.12640303059015429, + "mean_ms": 0.09991401658044197 + }, + { + "knob": 64, + "nq": 2000, + "recall": 0.7189, + "qps_1t": 8412.063329895049, + "qps_1t_runs": [ + 8412.063329895049, + 8220.196039263788, + 8874.271951516337 + ], + "median_ms": 0.12469995999708772, + "p95_ms": 0.1466000103391707, + "p99_ms": 0.17181002709548943, + "mean_ms": 0.11773793297470547 + }, + { + "knob": 1265, + "nq": 2000, + "recall": 1.0, + "qps_1t": 1629.7196985702874, + "qps_1t_runs": [ + 1606.3573184628947, + 1634.6548127756025, + 1629.7196985702874 + ], + "median_ms": 0.6629499839618802, + "p95_ms": 0.7409099955111742, + "p99_ms": 0.8447539713233722, + "mean_ms": 0.6159596335298071 + } + ] + } + } + }, + "hnswlib|glove|1000000": { + "mem_baseline": { + "rss_mb": 35.22265625, + "peak_mb": 35.22265625 + }, + "version": "hnswlib 0.8.0", + "build": { + "ingest_s": 50.97159750020364, + "index_s": 3.900029696524143e-06, + "total_s": 50.97160140023334, + "first_query_ms": 2.3651000228710473, + "ready_s": 50.97396650025621 + }, + "mem_after_build": { + "rss_mb": 783.9765625, + "peak_mb": 792.1015625 + }, + "mem_after_first_query": { + "rss_mb": 783.9921875, + "peak_mb": 792.1015625 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.4829, + "qps_1t_runs": [ + 24853.73573143744, + 28305.478216910997, + 27767.287816735425 + ], + "qps_1t": 27767.287816735425, + "qps_1t_min": 24853.73573143744, + "qps_1t_max": 28305.478216910997, + "median_ms": 0.03459997242316604, + "p95_ms": 0.0565050228033215, + "p99_ms": 0.07450298522599048, + "mean_ms": 0.03719261679604339, + "knob": 10, + "batch": { + "recall": 0.4829, + "qps_batch_runs": [ + 146679.18347670152, + 141004.94194485634, + 138433.2129237236 + ], + "qps_batch": 141004.94194485634, + "qps_batch_min": 138433.2129237236, + "qps_batch_max": 146679.18347670152, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.56575, + "qps_1t_runs": [ + 21474.296079339165, + 23346.51252732373, + 22942.33110313481 + ], + "qps_1t": 22942.33110313481, + "qps_1t_min": 21474.296079339165, + "qps_1t_max": 23346.51252732373, + "median_ms": 0.04160002572461963, + "p95_ms": 0.06470502412412317, + "p99_ms": 0.07860101643018427, + "mean_ms": 0.044329266742958374, + "knob": 16, + "batch": { + "recall": 0.56575, + "qps_batch_runs": [ + 105028.77786889613, + 82217.57232420969, + 113747.52594929886 + ], + "qps_batch": 105028.77786889613, + "qps_batch_min": 82217.57232420969, + "qps_batch_max": 113747.52594929886, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.63085, + "qps_1t_runs": [ + 17787.580990259372, + 15586.42355286752, + 17334.537539932342 + ], + "qps_1t": 17334.537539932342, + "qps_1t_min": 15586.42355286752, + "qps_1t_max": 17787.580990259372, + "median_ms": 0.05400000372901559, + "p95_ms": 0.08020002860575914, + "p99_ms": 0.09729998419061303, + "mean_ms": 0.059355233068345115, + "knob": 24, + "batch": { + "recall": 0.63085, + "qps_batch_runs": [ + 82463.68491743511, + 85007.26817182369, + 78122.86383128668 + ], + "qps_batch": 82463.68491743511, + "qps_batch_min": 78122.86383128668, + "qps_batch_max": 85007.26817182369, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6724, + "qps_1t_runs": [ + 13831.669807020386, + 13632.075732251442, + 14170.357475793706 + ], + "qps_1t": 13831.669807020386, + "qps_1t_min": 13632.075732251442, + "qps_1t_max": 14170.357475793706, + "median_ms": 0.06749999010935426, + "p95_ms": 0.10120501101482661, + "p99_ms": 0.12520202202722436, + "mean_ms": 0.07207470022452374, + "knob": 32, + "batch": { + "recall": 0.6724, + "qps_batch_runs": [ + 54492.79467104369, + 67729.3400241416, + 69469.04818270568 + ], + "qps_batch": 67729.3400241416, + "qps_batch_min": 54492.79467104369, + "qps_batch_max": 69469.04818270568, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7266, + "qps_1t_runs": [ + 10123.547759460356, + 10275.34853224859, + 10617.638690352958 + ], + "qps_1t": 10275.34853224859, + "qps_1t_min": 10123.547759460356, + "qps_1t_max": 10617.638690352958, + "median_ms": 0.08999998681247234, + "p95_ms": 0.12709998118225485, + "p99_ms": 0.15799998072907329, + "mean_ms": 0.09676093323893535, + "knob": 48, + "batch": { + "recall": 0.7266, + "qps_batch_runs": [ + 51812.531929188146, + 55714.18622761616, + 54141.118876686385 + ], + "qps_batch": 54141.118876686385, + "qps_batch_min": 51812.531929188146, + "qps_batch_max": 55714.18622761616, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.76205, + "qps_1t_runs": [ + 8779.36147075755, + 8681.222448936891, + 8081.444813229237 + ], + "qps_1t": 8681.222448936891, + "qps_1t_min": 8081.444813229237, + "qps_1t_max": 8779.36147075755, + "median_ms": 0.11070002801716328, + "p95_ms": 0.15420500712934884, + "p99_ms": 0.1892010431038216, + "mean_ms": 0.11761163340027754, + "knob": 64, + "batch": { + "recall": 0.76205, + "qps_batch_runs": [ + 35241.98025144597, + 45377.09502003826, + 44181.710554651 + ], + "qps_batch": 44181.710554651, + "qps_batch_min": 35241.98025144597, + "qps_batch_max": 45377.09502003826, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8022, + "qps_1t_runs": [ + 6226.710156250394, + 5702.889708318204, + 6005.077287115498 + ], + "qps_1t": 6005.077287115498, + "qps_1t_min": 5702.889708318204, + "qps_1t_max": 6226.710156250394, + "median_ms": 0.15820004045963287, + "p95_ms": 0.2236050117062405, + "p99_ms": 0.44870404002722364, + "mean_ms": 0.1674913000994517, + "knob": 96, + "batch": { + "recall": 0.8022, + "qps_batch_runs": [ + 29464.97497352799, + 31593.23082658677, + 32031.929431727487 + ], + "qps_batch": 31593.23082658677, + "qps_batch_min": 29464.97497352799, + "qps_batch_max": 32031.929431727487, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.82825, + "qps_1t_runs": [ + 4751.732664296486, + 5057.079258155335, + 4940.955583864194 + ], + "qps_1t": 4940.955583864194, + "qps_1t_min": 4751.732664296486, + "qps_1t_max": 5057.079258155335, + "median_ms": 0.1984999980777502, + "p95_ms": 0.26020000223070383, + "p99_ms": 0.31050097779370844, + "mean_ms": 0.20352738316675337, + "knob": 128, + "batch": { + "recall": 0.82825, + "qps_batch_runs": [ + 24803.740402503612, + 26478.142292399258, + 25813.646117851957 + ], + "qps_batch": 25813.646117851957, + "qps_batch_min": 24803.740402503612, + "qps_batch_max": 26478.142292399258, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8605, + "qps_1t_runs": [ + 3291.709662069511, + 3346.307526538395, + 2942.3109896850424 + ], + "qps_1t": 3291.709662069511, + "qps_1t_min": 2942.3109896850424, + "qps_1t_max": 3346.307526538395, + "median_ms": 0.2888999879360199, + "p95_ms": 0.3837100113742054, + "p99_ms": 0.8220430277288057, + "mean_ms": 0.314166416763328, + "knob": 192, + "batch": { + "recall": 0.8605, + "qps_batch_runs": [ + 17874.985925617406, + 17880.53122846641, + 17010.52015956229 + ], + "qps_batch": 17874.985925617406, + "qps_batch_min": 17010.52015956229, + "qps_batch_max": 17880.53122846641, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8807, + "qps_1t_runs": [ + 2492.540446495755, + 2466.2606301818614, + 2481.26766831609 + ], + "qps_1t": 2481.26766831609, + "qps_1t_min": 2466.2606301818614, + "qps_1t_max": 2492.540446495755, + "median_ms": 0.3797000099439174, + "p95_ms": 0.5393049737904221, + "p99_ms": 1.1234569968655739, + "mean_ms": 0.4032296835503075, + "knob": 256, + "batch": { + "recall": 0.8807, + "qps_batch_runs": [ + 10539.545693398934, + 10936.414048208235, + 11943.321775795503 + ], + "qps_batch": 10936.414048208235, + "qps_batch_min": 10539.545693398934, + "qps_batch_max": 11943.321775795503, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.91185, + "qps_1t_runs": [ + 1583.6276972738062, + 1659.9601510429572, + 1692.3552089859636 + ], + "qps_1t": 1659.9601510429572, + "qps_1t_min": 1583.6276972738062, + "qps_1t_max": 1692.3552089859636, + "median_ms": 0.5854500050190836, + "p95_ms": 0.8317649888340386, + "p99_ms": 1.3181159668602054, + "mean_ms": 0.6082593831912769, + "knob": 400, + "batch": { + "recall": 0.91185, + "qps_batch_runs": [ + 8483.617075655224, + 8092.674067316121, + 7735.920430702874 + ], + "qps_batch": 8092.674067316121, + "qps_batch_min": 7735.920430702874, + "qps_batch_max": 8483.617075655224, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9367, + "qps_1t_runs": [ + 1100.9520757753453, + 1071.3724203800332, + 1085.8803198148994 + ], + "qps_1t": 1085.8803198148994, + "qps_1t_min": 1071.3724203800332, + "qps_1t_max": 1100.9520757753453, + "median_ms": 0.8930000476539135, + "p95_ms": 1.2401150132063776, + "p99_ms": 1.8217029579682342, + "mean_ms": 0.920866266610877, + "knob": 640, + "batch": { + "recall": 0.9367, + "qps_batch_runs": [ + 4815.955839557642, + 5044.858885123081, + 5013.180906027894 + ], + "qps_batch": 5013.180906027894, + "qps_batch_min": 4815.955839557642, + "qps_batch_max": 5044.858885123081, + "threads": 32 + } + } + ], + "mutation": { + "ref_knob": 640, + "recall_before": 0.9367, + "insert_10k": { + "seconds": 1.0653810999938287, + "per_s": 9386.312559944912, + "batch": 1000 + }, + "update_10k": { + "seconds": 2.6550877999980003, + "per_s": 3766.3537906383103, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.014886100019793957, + "per_s": 671767.6212509025, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 801.5234375, + "peak_mb": 801.60546875 + }, + "first_query_after_ms": 2.2681999835185707, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9355, + "qps_1t_runs": [ + 1032.29434678509, + 1038.462307205691, + 1001.8454492622539 + ], + "qps_1t": 1032.29434678509, + "qps_1t_min": 1001.8454492622539, + "qps_1t_max": 1038.462307205691, + "median_ms": 0.9595500014256686, + "p95_ms": 1.350234978599474, + "p99_ms": 1.802758990088479, + "mean_ms": 0.9766120504646096 + }, + "recall_after": 0.9355 + }, + "saveload": { + "save_s": 0.7770867999643087, + "disk_mb": 528.3318481445312, + "load_s": 3.656790999986697, + "first_query_after_load_ms": 3.2733999541960657, + "time_to_ready_s": 3.660064399940893 + }, + "load_before_pct": 6.612500000000003, + "load_contaminated": false + }, + "hnswlib|glove|100000": { + "mem_baseline": { + "rss_mb": 35.20703125, + "peak_mb": 35.20703125 + }, + "version": "hnswlib 0.8.0", + "build": { + "ingest_s": 1.3345952999661677, + "index_s": 2.0999577827751637e-06, + "total_s": 1.3345973999239504, + "first_query_ms": 0.8077000384218991, + "ready_s": 1.3354050999623723, + "total_s_runs": [ + 1.3345973999239504, + 1.2821055001113564, + 1.3087505000294186 + ] + }, + "mem_after_build": { + "rss_mb": 122.84765625, + "peak_mb": 130.97265625 + }, + "mem_after_first_query": { + "rss_mb": 122.8671875, + "peak_mb": 130.97265625 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.54205, + "qps_1t_runs": [ + 44723.819005460275, + 46903.54526299682, + 41330.25599087344 + ], + "qps_1t": 44723.819005460275, + "qps_1t_min": 41330.25599087344, + "qps_1t_max": 46903.54526299682, + "median_ms": 0.021599989850074053, + "p95_ms": 0.031200004741549492, + "p99_ms": 0.0396010105032474, + "mean_ms": 0.022625049954513088, + "knob": 10, + "batch": { + "recall": 0.54205, + "qps_batch_runs": [ + 271506.7273731674, + 373517.6042328868, + 387574.3656137073 + ], + "qps_batch": 373517.6042328868, + "qps_batch_min": 271506.7273731674, + "qps_batch_max": 387574.3656137073, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6244, + "qps_1t_runs": [ + 34289.31990045052, + 36876.69168737768, + 36870.777463584236 + ], + "qps_1t": 36870.777463584236, + "qps_1t_min": 34289.31990045052, + "qps_1t_max": 36876.69168737768, + "median_ms": 0.026900030206888914, + "p95_ms": 0.03789999755099416, + "p99_ms": 0.04439998883754015, + "mean_ms": 0.027800916500079136, + "knob": 16, + "batch": { + "recall": 0.6244, + "qps_batch_runs": [ + 278067.4315904601, + 345172.7590557858, + 328116.28323354566 + ], + "qps_batch": 328116.28323354566, + "qps_batch_min": 278067.4315904601, + "qps_batch_max": 345172.7590557858, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.69495, + "qps_1t_runs": [ + 29071.035227971708, + 27705.858117625008, + 27312.591215687415 + ], + "qps_1t": 27705.858117625008, + "qps_1t_min": 27312.591215687415, + "qps_1t_max": 29071.035227971708, + "median_ms": 0.03460003063082695, + "p95_ms": 0.0475000124424696, + "p99_ms": 0.057500972761772594, + "mean_ms": 0.03570169989446489, + "knob": 24, + "batch": { + "recall": 0.69495, + "qps_batch_runs": [ + 212972.133282918, + 210768.14537390825, + 294399.05868354416 + ], + "qps_batch": 212972.133282918, + "qps_batch_min": 210768.14537390825, + "qps_batch_max": 294399.05868354416, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7392, + "qps_1t_runs": [ + 23316.89868393719, + 23596.84328586468, + 23476.823851824578 + ], + "qps_1t": 23476.823851824578, + "qps_1t_min": 23316.89868393719, + "qps_1t_max": 23596.84328586468, + "median_ms": 0.04160002572461963, + "p95_ms": 0.05520001286640763, + "p99_ms": 0.06500197108834986, + "mean_ms": 0.042620366594443716, + "knob": 32, + "batch": { + "recall": 0.7392, + "qps_batch_runs": [ + 162867.77625371164, + 179768.81694323747, + 171278.3354111458 + ], + "qps_batch": 171278.3354111458, + "qps_batch_min": 162867.77625371164, + "qps_batch_max": 179768.81694323747, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.79375, + "qps_1t_runs": [ + 15501.160239324272, + 17468.88136227897, + 17283.799087645886 + ], + "qps_1t": 17283.799087645886, + "qps_1t_min": 15501.160239324272, + "qps_1t_max": 17468.88136227897, + "median_ms": 0.05879998207092285, + "p95_ms": 0.07819995807949454, + "p99_ms": 0.08970198454335336, + "mean_ms": 0.05987120007436412, + "knob": 48, + "batch": { + "recall": 0.79375, + "qps_batch_runs": [ + 132860.35036971865, + 153449.54578700237, + 94064.08575280246 + ], + "qps_batch": 132860.35036971865, + "qps_batch_min": 94064.08575280246, + "qps_batch_max": 153449.54578700237, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8287, + "qps_1t_runs": [ + 14084.21943420152, + 12806.950650724291, + 12917.760342862528 + ], + "qps_1t": 12917.760342862528, + "qps_1t_min": 12806.950650724291, + "qps_1t_max": 14084.21943420152, + "median_ms": 0.07289997301995754, + "p95_ms": 0.0947999651543796, + "p99_ms": 0.1116060349158944, + "mean_ms": 0.07549894988187589, + "knob": 64, + "batch": { + "recall": 0.8287, + "qps_batch_runs": [ + 132275.13230674472, + 155902.87238800808, + 150099.44084168825 + ], + "qps_batch": 150099.44084168825, + "qps_batch_min": 132275.13230674472, + "qps_batch_max": 155902.87238800808, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.87055, + "qps_1t_runs": [ + 9613.332494154269, + 7599.156672345472, + 9091.917473638105 + ], + "qps_1t": 9091.917473638105, + "qps_1t_min": 7599.156672345472, + "qps_1t_max": 9613.332494154269, + "median_ms": 0.10270002530887723, + "p95_ms": 0.1342049741651863, + "p99_ms": 0.2162590157240664, + "mean_ms": 0.11520118331342626, + "knob": 96, + "batch": { + "recall": 0.87055, + "qps_batch_runs": [ + 103974.42219421266, + 100314.98903498605, + 108647.82348704936 + ], + "qps_batch": 103974.42219421266, + "qps_batch_min": 100314.98903498605, + "qps_batch_max": 108647.82348704936, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.89495, + "qps_1t_runs": [ + 7640.566370589797, + 7197.195987622838, + 7769.594247664829 + ], + "qps_1t": 7640.566370589797, + "qps_1t_min": 7197.195987622838, + "qps_1t_max": 7769.594247664829, + "median_ms": 0.13080000644549727, + "p95_ms": 0.16399997111875564, + "p99_ms": 0.19450502586551022, + "mean_ms": 0.13284339980843166, + "knob": 128, + "batch": { + "recall": 0.89495, + "qps_batch_runs": [ + 59020.379746301645, + 71296.67263987656, + 88748.46908499993 + ], + "qps_batch": 71296.67263987656, + "qps_batch_min": 59020.379746301645, + "qps_batch_max": 88748.46908499993, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9278, + "qps_1t_runs": [ + 5038.43696484007, + 5168.247122880161, + 5112.2235319167785 + ], + "qps_1t": 5112.2235319167785, + "qps_1t_min": 5038.43696484007, + "qps_1t_max": 5168.247122880161, + "median_ms": 0.18969998927786946, + "p95_ms": 0.23820500937290492, + "p99_ms": 0.3203219943679874, + "mean_ms": 0.1958576833906894, + "knob": 192, + "batch": { + "recall": 0.9278, + "qps_batch_runs": [ + 56815.59929728225, + 63882.66026907072, + 65531.65828765434 + ], + "qps_batch": 63882.66026907072, + "qps_batch_min": 56815.59929728225, + "qps_batch_max": 65531.65828765434, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.94665, + "qps_1t_runs": [ + 4050.3326978571886, + 3983.4114815115527, + 3913.2579445526244 + ], + "qps_1t": 3983.4114815115527, + "qps_1t_min": 3913.2579445526244, + "qps_1t_max": 4050.3326978571886, + "median_ms": 0.24289998691529036, + "p95_ms": 0.30960002914071083, + "p99_ms": 0.41153204801958126, + "mean_ms": 0.25115864945109934, + "knob": 256, + "batch": { + "recall": 0.94665, + "qps_batch_runs": [ + 42867.21663636747, + 51909.76006280174, + 52287.03484600862 + ], + "qps_batch": 51909.76006280174, + "qps_batch_min": 42867.21663636747, + "qps_batch_max": 52287.03484600862, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9692, + "qps_1t_runs": [ + 2708.985725087262, + 2730.634882965438, + 2669.608217338934 + ], + "qps_1t": 2708.985725087262, + "qps_1t_min": 2669.608217338934, + "qps_1t_max": 2730.634882965438, + "median_ms": 0.36604999331757426, + "p95_ms": 0.4523000097833574, + "p99_ms": 0.5730289977509535, + "mean_ms": 0.3699812831667562, + "knob": 400, + "batch": { + "recall": 0.9692, + "qps_batch_runs": [ + 33451.36087631815, + 35141.60308682183, + 35275.48386166561 + ], + "qps_batch": 35141.60308682183, + "qps_batch_min": 33451.36087631815, + "qps_batch_max": 35275.48386166561, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9843, + "qps_1t_runs": [ + 1808.9008385495708, + 1750.1215026648167, + 1769.4494125945832 + ], + "qps_1t": 1769.4494125945832, + "qps_1t_min": 1750.1215026648167, + "qps_1t_max": 1808.9008385495708, + "median_ms": 0.5594499816652387, + "p95_ms": 0.6926250352989883, + "p99_ms": 0.8525169722270224, + "mean_ms": 0.5631194501571978, + "knob": 640, + "batch": { + "recall": 0.9843, + "qps_batch_runs": [ + 21716.422601259816, + 23110.726962525463, + 23068.928816427026 + ], + "qps_batch": 23068.928816427026, + "qps_batch_min": 21716.422601259816, + "qps_batch_max": 23110.726962525463, + "threads": 32 + } + } + ], + "mutation": { + "ref_knob": 400, + "recall_before": 0.9692, + "insert_10k": { + "seconds": 0.22615729999961331, + "per_s": 44217.011787888776, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.9343183999881148, + "per_s": 10702.989473531943, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.008049399999435991, + "per_s": 1242328.6208538134, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 141.1171875, + "peak_mb": 218.20703125 + }, + "first_query_after_ms": 0.6444000173360109, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9686, + "qps_1t_runs": [ + 2307.1608589658426, + 2289.887379307556, + 2229.330072772871 + ], + "qps_1t": 2289.887379307556, + "qps_1t_min": 2229.330072772871, + "qps_1t_max": 2307.1608589658426, + "median_ms": 0.43440001900307834, + "p95_ms": 0.5435300030512736, + "p99_ms": 0.6905010191258043, + "mean_ms": 0.4395670331335471 + }, + "recall_after": 0.9686 + }, + "saveload": { + "save_s": 0.03637069999240339, + "disk_mb": 57.546424865722656, + "load_s": 0.141510899993591, + "first_query_after_load_ms": 0.7082999800331891, + "time_to_ready_s": 0.1422191999736242 + }, + "load_before_pct": 3.2156249999999984, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 1.3072742999647744, + "mem": { + "rss_mb": 122.0703125, + "peak_mb": 130.19921875 + }, + "subsets": { + "f10": [ + { + "knob": 16, + "nq": 2000, + "recall": 0.8166, + "qps_1t": 3677.6387820146956, + "qps_1t_runs": [ + 3759.364577268687, + 3532.096783489829, + 3677.6387820146956 + ], + "median_ms": 0.2583499881438911, + "p95_ms": 0.394509980105795, + "p99_ms": 0.5736070021521301, + "mean_ms": 0.27367798321453546 + }, + { + "knob": 48, + "nq": 2000, + "recall": 0.93485, + "qps_1t": 1354.4558920117986, + "qps_1t_runs": [ + 1316.3519541187577, + 1383.4948914627462, + 1354.4558920117986 + ], + "median_ms": 0.6934000120963901, + "p95_ms": 1.02491500438191, + "p99_ms": 2.0306119776796594, + "mean_ms": 0.7402620835540196 + }, + { + "knob": 96, + "nq": 2000, + "recall": 0.975, + "qps_1t": 728.8365452804516, + "qps_1t_runs": [ + 728.8365452804516, + 739.2378702419824, + 721.7520067027885 + ], + "median_ms": 1.3130000152159482, + "p95_ms": 1.8889250030042608, + "p99_ms": 3.229097022558567, + "mean_ms": 1.370103916793596 + }, + { + "knob": 192, + "nq": 1268, + "recall": 0.9927444794952681, + "qps_1t": 384.4068922023777, + "qps_1t_runs": [ + 384.4068922023777, + 375.6272536580646, + 389.81977657218965 + ], + "median_ms": 2.4999500019475818, + "p95_ms": 3.4383249963866547, + "p99_ms": 5.298062009387644, + "mean_ms": 2.609637276877034 + }, + { + "knob": 640, + "nq": 430, + "recall": 1.0, + "qps_1t": 137.21710496936936, + "qps_1t_runs": [ + 136.7346556167165, + 137.78870658602034, + 137.21710496936936 + ], + "median_ms": 7.10880002588965, + "p95_ms": 8.725929964566603, + "p99_ms": 12.637714000302358, + "mean_ms": 7.286214961421265 + } + ], + "f01": [ + { + "knob": 16, + "nq": 1186, + "recall": 0.9507588532883643, + "qps_1t": 448.8249489720564, + "qps_1t_runs": [ + 464.19604221110353, + 448.8249489720564, + 441.90245786025764 + ], + "median_ms": 2.122300007613376, + "p95_ms": 2.9921900160843506, + "p99_ms": 3.8339270139112958, + "mean_ms": 2.2150817320624783 + }, + { + "knob": 48, + "nq": 455, + "recall": 0.9958241758241758, + "qps_1t": 166.91723210228446, + "qps_1t_runs": [ + 166.91723210228446, + 162.06699750473518, + 174.38353408205953 + ], + "median_ms": 5.707699980121106, + "p95_ms": 7.627739978488534, + "p99_ms": 14.033827986568172, + "mean_ms": 5.965255750317618 + }, + { + "knob": 96, + "nq": 286, + "recall": 0.9996503496503496, + "qps_1t": 95.20224302845789, + "qps_1t_runs": [ + 95.04263593748968, + 96.66358540018246, + 95.20224302845789 + ], + "median_ms": 10.192600020673126, + "p95_ms": 12.572064984124152, + "p99_ms": 19.371890987385974, + "mean_ms": 10.45690174655509 + }, + { + "knob": 192, + "nq": 172, + "recall": 1.0, + "qps_1t": 57.15175459003141, + "qps_1t_runs": [ + 57.013561371532646, + 57.15175459003141, + 57.40438466791265 + ], + "median_ms": 16.920199996093288, + "p95_ms": 21.02540002670139, + "p99_ms": 27.633194957161344, + "mean_ms": 17.48574476588929 + }, + { + "knob": 640, + "nq": 79, + "recall": 1.0, + "qps_1t": 25.329869448785068, + "qps_1t_runs": [ + 25.4083418025337, + 24.806453922089396, + 25.329869448785068 + ], + "median_ms": 38.42579998308793, + "p95_ms": 47.238399996422224, + "p99_ms": 59.03204800793901, + "mean_ms": 39.71610843883788 + } + ] + } + } + }, + "hnswlib|sift|1000000": { + "mem_baseline": { + "rss_mb": 35.75, + "peak_mb": 35.75 + }, + "version": "hnswlib 0.8.0", + "build": { + "ingest_s": 60.422680200252216, + "index_s": 3.3999676816165447e-06, + "total_s": 60.4226836002199, + "first_query_ms": 1.896000001579523, + "ready_s": 60.42457960022148 + }, + "mem_after_build": { + "rss_mb": 889.25390625, + "peak_mb": 899.5234375 + }, + "mem_after_first_query": { + "rss_mb": 889.26953125, + "peak_mb": 899.5234375 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6795, + "qps_1t_runs": [ + 22475.091832658785, + 22476.68311615447, + 21929.031050822185 + ], + "qps_1t": 22475.091832658785, + "qps_1t_min": 21929.031050822185, + "qps_1t_max": 22476.68311615447, + "median_ms": 0.04379998426884413, + "p95_ms": 0.062399954185821116, + "p99_ms": 0.07270702917594478, + "mean_ms": 0.04486196687018188, + "knob": 10, + "batch": { + "recall": 0.6795, + "qps_batch_runs": [ + 89007.96166477687, + 80772.50824594033, + 73767.80114057181 + ], + "qps_batch": 80772.50824594033, + "qps_batch_min": 73767.80114057181, + "qps_batch_max": 89007.96166477687, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7752, + "qps_1t_runs": [ + 15590.821951365464, + 17309.543255140707, + 17159.634345843588 + ], + "qps_1t": 17159.634345843588, + "qps_1t_min": 15590.821951365464, + "qps_1t_max": 17309.543255140707, + "median_ms": 0.05849997978657484, + "p95_ms": 0.08780002826824784, + "p99_ms": 0.11110297287814326, + "mean_ms": 0.06006273323631225, + "knob": 16, + "batch": { + "recall": 0.7752, + "qps_batch_runs": [ + 79260.65661241273, + 72842.76165548181, + 70652.90347222987 + ], + "qps_batch": 72842.76165548181, + "qps_batch_min": 70652.90347222987, + "qps_batch_max": 79260.65661241273, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8472, + "qps_1t_runs": [ + 14934.701728235763, + 13360.67822834269, + 13153.07489103741 + ], + "qps_1t": 13360.67822834269, + "qps_1t_min": 13153.07489103741, + "qps_1t_max": 14934.701728235763, + "median_ms": 0.07319997530430555, + "p95_ms": 0.09820499108172955, + "p99_ms": 0.11510100332088771, + "mean_ms": 0.07261083316795218, + "knob": 24, + "batch": { + "recall": 0.8472, + "qps_batch_runs": [ + 58473.71895197593, + 52926.715000554956, + 51003.103559514704 + ], + "qps_batch": 52926.715000554956, + "qps_batch_min": 51003.103559514704, + "qps_batch_max": 58473.71895197593, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.89, + "qps_1t_runs": [ + 11346.005010787292, + 11456.853503939941, + 11208.469128958717 + ], + "qps_1t": 11346.005010787292, + "qps_1t_min": 11208.469128958717, + "qps_1t_max": 11456.853503939941, + "median_ms": 0.08900003740563989, + "p95_ms": 0.11650501401163639, + "p99_ms": 0.13640099088661375, + "mean_ms": 0.08821299972866352, + "knob": 32, + "batch": { + "recall": 0.89, + "qps_batch_runs": [ + 55623.84931818337, + 58617.506080550585, + 52415.571658493 + ], + "qps_batch": 55623.84931818337, + "qps_batch_min": 52415.571658493, + "qps_batch_max": 58617.506080550585, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9339, + "qps_1t_runs": [ + 8629.08440927331, + 8445.81041408436, + 8865.507562237099 + ], + "qps_1t": 8629.08440927331, + "qps_1t_min": 8445.81041408436, + "qps_1t_max": 8865.507562237099, + "median_ms": 0.11729996185749769, + "p95_ms": 0.15490501536987725, + "p99_ms": 0.18040103430394086, + "mean_ms": 0.11569525014298658, + "knob": 48, + "batch": { + "recall": 0.9339, + "qps_batch_runs": [ + 44422.8253599564, + 35113.34587798847, + 41343.58377407923 + ], + "qps_batch": 41343.58377407923, + "qps_batch_min": 35113.34587798847, + "qps_batch_max": 44422.8253599564, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.95755, + "qps_1t_runs": [ + 6689.793275359951, + 6875.940507499319, + 6764.684698092554 + ], + "qps_1t": 6764.684698092554, + "qps_1t_min": 6689.793275359951, + "qps_1t_max": 6875.940507499319, + "median_ms": 0.15119998715817928, + "p95_ms": 0.1933050167281181, + "p99_ms": 0.22629997751209885, + "mean_ms": 0.14758088329108432, + "knob": 64, + "batch": { + "recall": 0.95755, + "qps_batch_runs": [ + 34527.823403220515, + 35560.67628041415, + 35637.66471861058 + ], + "qps_batch": 35560.67628041415, + "qps_batch_min": 34527.823403220515, + "qps_batch_max": 35637.66471861058, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9768, + "qps_1t_runs": [ + 4866.216761466154, + 4879.756700329306, + 4847.488307564959 + ], + "qps_1t": 4866.216761466154, + "qps_1t_min": 4847.488307564959, + "qps_1t_max": 4879.756700329306, + "median_ms": 0.21044997265562415, + "p95_ms": 0.2673999988473952, + "p99_ms": 0.3121029905742035, + "mean_ms": 0.20557303334741542, + "knob": 96, + "batch": { + "recall": 0.9768, + "qps_batch_runs": [ + 25048.092333879147, + 24020.15771325015, + 27240.645233337054 + ], + "qps_batch": 25048.092333879147, + "qps_batch_min": 24020.15771325015, + "qps_batch_max": 27240.645233337054, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9862, + "qps_1t_runs": [ + 3729.2608961709225, + 3956.6250975409653, + 4162.767545359252 + ], + "qps_1t": 3956.6250975409653, + "qps_1t_min": 3729.2608961709225, + "qps_1t_max": 4162.767545359252, + "median_ms": 0.25670003378763795, + "p95_ms": 0.3282999969087541, + "p99_ms": 0.37360305024776613, + "mean_ms": 0.25370503297502484, + "knob": 128, + "batch": { + "recall": 0.9862, + "qps_batch_runs": [ + 24450.146768424864, + 22507.56816715879, + 24599.429052640924 + ], + "qps_batch": 24450.146768424864, + "qps_batch_min": 22507.56816715879, + "qps_batch_max": 24599.429052640924, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99365, + "qps_1t_runs": [ + 2702.800223433291, + 2717.1328775732827, + 2840.061109111305 + ], + "qps_1t": 2717.1328775732827, + "qps_1t_min": 2702.800223433291, + "qps_1t_max": 2840.061109111305, + "median_ms": 0.374099996406585, + "p95_ms": 0.4678050085203722, + "p99_ms": 0.5398049775976689, + "mean_ms": 0.3633755833725445, + "knob": 192, + "batch": { + "recall": 0.99365, + "qps_batch_runs": [ + 14539.131344918409, + 14951.270073134194, + 15398.462157050204 + ], + "qps_batch": 14951.270073134194, + "qps_batch_min": 14539.131344918409, + "qps_batch_max": 15398.462157050204, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99655, + "qps_1t_runs": [ + 2229.0212344468423, + 2159.2796956749307, + 1950.4551245233356 + ], + "qps_1t": 2159.2796956749307, + "qps_1t_min": 1950.4551245233356, + "qps_1t_max": 2229.0212344468423, + "median_ms": 0.48809999134391546, + "p95_ms": 0.6267049990128726, + "p99_ms": 0.7245240447809925, + "mean_ms": 0.47481521671094623, + "knob": 256, + "batch": { + "recall": 0.99655, + "qps_batch_runs": [ + 11588.170330215526, + 11572.165471434459, + 11233.317120954891 + ], + "qps_batch": 11572.165471434459, + "qps_batch_min": 11233.317120954891, + "qps_batch_max": 11588.170330215526, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99865, + "qps_1t_runs": [ + 1411.8767911897073, + 1490.2587747883554, + 1475.1483973311501 + ], + "qps_1t": 1475.1483973311501, + "qps_1t_min": 1411.8767911897073, + "qps_1t_max": 1490.2587747883554, + "median_ms": 0.7050499843899161, + "p95_ms": 0.8956050092820078, + "p99_ms": 0.994307946530171, + "mean_ms": 0.685733133306106, + "knob": 400, + "batch": { + "recall": 0.99865, + "qps_batch_runs": [ + 9635.517287900113, + 8898.06773955238, + 8766.408525530082 + ], + "qps_batch": 8898.06773955238, + "qps_batch_min": 8766.408525530082, + "qps_batch_max": 9635.517287900113, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99935, + "qps_1t_runs": [ + 950.0436192333061, + 975.9859629121419, + 972.9978032743089 + ], + "qps_1t": 972.9978032743089, + "qps_1t_min": 950.0436192333061, + "qps_1t_max": 975.9859629121419, + "median_ms": 1.0636000079102814, + "p95_ms": 1.3488999713445082, + "p99_ms": 1.4980329980608083, + "mean_ms": 1.0349798997340258, + "knob": 640, + "batch": { + "recall": 0.99935, + "qps_batch_runs": [ + 5242.292192309696, + 5333.336177800954, + 5401.266867138309 + ], + "qps_batch": 5333.336177800954, + "qps_batch_min": 5242.292192309696, + "qps_batch_max": 5401.266867138309, + "threads": 32 + } + } + ], + "saveload": { + "save_s": 1.3896516999811865, + "disk_mb": 629.9174461364746, + "load_s": 4.90050660003908, + "first_query_after_load_ms": 0.935999967623502, + "time_to_ready_s": 4.901442600006703 + }, + "load_before_pct": 8.974999999999998, + "load_contaminated": false + }, + "hnswlib|sift|100000": { + "mem_baseline": { + "rss_mb": 35.44921875, + "peak_mb": 35.44921875 + }, + "version": "hnswlib 0.8.0", + "build": { + "ingest_s": 1.1284188999561593, + "index_s": 4.099973011761904e-06, + "total_s": 1.128422999929171, + "first_query_ms": 0.7225999725051224, + "ready_s": 1.1291455999016762, + "total_s_runs": [ + 1.128422999929171, + 1.088209499954246, + 1.1993346998933703 + ] + }, + "mem_after_build": { + "rss_mb": 132.87890625, + "peak_mb": 143.14453125 + }, + "mem_after_first_query": { + "rss_mb": 132.89453125, + "peak_mb": 143.14453125 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7877, + "qps_1t_runs": [ + 44466.479261374996, + 45863.24908288654, + 42336.37541892785 + ], + "qps_1t": 44466.479261374996, + "qps_1t_min": 42336.37541892785, + "qps_1t_max": 45863.24908288654, + "median_ms": 0.021199986804276705, + "p95_ms": 0.030800001695752144, + "p99_ms": 0.043000978766940555, + "mean_ms": 0.022637716834045325, + "knob": 10, + "batch": { + "recall": 0.7877, + "qps_batch_runs": [ + 359725.1687996511, + 422448.94000532664, + 392472.37695326173 + ], + "qps_batch": 392472.37695326173, + "qps_batch_min": 359725.1687996511, + "qps_batch_max": 422448.94000532664, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.86895, + "qps_1t_runs": [ + 34241.06380580334, + 35361.24168269553, + 36359.27349157056 + ], + "qps_1t": 35361.24168269553, + "qps_1t_min": 34241.06380580334, + "qps_1t_max": 36359.27349157056, + "median_ms": 0.027500034775584936, + "p95_ms": 0.03840000135824084, + "p99_ms": 0.04859996261075139, + "mean_ms": 0.02832918328931555, + "knob": 16, + "batch": { + "recall": 0.86895, + "qps_batch_runs": [ + 275546.6144899087, + 342941.40878269455, + 310298.8173090875 + ], + "qps_batch": 310298.8173090875, + "qps_batch_min": 275546.6144899087, + "qps_batch_max": 342941.40878269455, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9228, + "qps_1t_runs": [ + 26531.70817352751, + 27773.573253201233, + 26390.16789665563 + ], + "qps_1t": 26531.70817352751, + "qps_1t_min": 26390.16789665563, + "qps_1t_max": 27773.573253201233, + "median_ms": 0.035999983083456755, + "p95_ms": 0.05080498231109234, + "p99_ms": 0.06810599472373736, + "mean_ms": 0.03719636697011689, + "knob": 24, + "batch": { + "recall": 0.9228, + "qps_batch_runs": [ + 281781.4233230866, + 262598.14541322534, + 281005.43732487067 + ], + "qps_batch": 281005.43732487067, + "qps_batch_min": 262598.14541322534, + "qps_batch_max": 281781.4233230866, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.95025, + "qps_1t_runs": [ + 22404.244233921945, + 22519.707471806014, + 21712.768402343423 + ], + "qps_1t": 22404.244233921945, + "qps_1t_min": 21712.768402343423, + "qps_1t_max": 22519.707471806014, + "median_ms": 0.04369998350739479, + "p95_ms": 0.061400001868605614, + "p99_ms": 0.07870601722970617, + "mean_ms": 0.04503193333706198, + "knob": 32, + "batch": { + "recall": 0.95025, + "qps_batch_runs": [ + 183135.0889854097, + 246035.7483069214, + 192350.2311013815 + ], + "qps_batch": 192350.2311013815, + "qps_batch_min": 183135.0889854097, + "qps_batch_max": 246035.7483069214, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.97625, + "qps_1t_runs": [ + 16464.618393411292, + 15859.736493281327, + 16412.006981227994 + ], + "qps_1t": 16412.006981227994, + "qps_1t_min": 15859.736493281327, + "qps_1t_max": 16464.618393411292, + "median_ms": 0.060250022215768695, + "p95_ms": 0.08149998320732266, + "p99_ms": 0.10349997377488762, + "mean_ms": 0.061573350006559245, + "knob": 48, + "batch": { + "recall": 0.97625, + "qps_batch_runs": [ + 178550.7036492056, + 157970.39619589678, + 175231.08682717508 + ], + "qps_batch": 175231.08682717508, + "qps_batch_min": 157970.39619589678, + "qps_batch_max": 178550.7036492056, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.98785, + "qps_1t_runs": [ + 13235.373926141097, + 12557.442419530878, + 12747.898839214908 + ], + "qps_1t": 12747.898839214908, + "qps_1t_min": 12557.442419530878, + "qps_1t_max": 13235.373926141097, + "median_ms": 0.07499998901039362, + "p95_ms": 0.10520504147280009, + "p99_ms": 0.1367020508041606, + "mean_ms": 0.07787781669564235, + "knob": 64, + "batch": { + "recall": 0.98785, + "qps_batch_runs": [ + 132068.99282907954, + 130072.84107308216, + 130878.06088491467 + ], + "qps_batch": 130878.06088491467, + "qps_batch_min": 130072.84107308216, + "qps_batch_max": 132068.99282907954, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99545, + "qps_1t_runs": [ + 8413.740697985422, + 8801.104723281625, + 8850.32898238257 + ], + "qps_1t": 8801.104723281625, + "qps_1t_min": 8413.740697985422, + "qps_1t_max": 8850.32898238257, + "median_ms": 0.10740000288933516, + "p95_ms": 0.15650002751499414, + "p99_ms": 0.24602401070296817, + "mean_ms": 0.11515514975568901, + "knob": 96, + "batch": { + "recall": 0.99545, + "qps_batch_runs": [ + 90674.98473305057, + 101372.07100820688, + 99853.71426416578 + ], + "qps_batch": 99853.71426416578, + "qps_batch_min": 90674.98473305057, + "qps_batch_max": 101372.07100820688, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99775, + "qps_1t_runs": [ + 7469.088230636029, + 6788.983787934223, + 6287.8627539312665 + ], + "qps_1t": 6788.983787934223, + "qps_1t_min": 6287.8627539312665, + "qps_1t_max": 7469.088230636029, + "median_ms": 0.13689999468624592, + "p95_ms": 0.21040502761024985, + "p99_ms": 0.3733939904486781, + "mean_ms": 0.14673971678712405, + "knob": 128, + "batch": { + "recall": 0.99775, + "qps_batch_runs": [ + 77868.58121009485, + 80264.55203437756, + 83760.86262273634 + ], + "qps_batch": 80264.55203437756, + "qps_batch_min": 77868.58121009485, + "qps_batch_max": 83760.86262273634, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.999, + "qps_1t_runs": [ + 4496.088517143993, + 4893.009454854954, + 4937.852192321759 + ], + "qps_1t": 4893.009454854954, + "qps_1t_min": 4496.088517143993, + "qps_1t_max": 4937.852192321759, + "median_ms": 0.19324998720549047, + "p95_ms": 0.3084249619860203, + "p99_ms": 0.6013419787632355, + "mean_ms": 0.2097686499764677, + "knob": 192, + "batch": { + "recall": 0.999, + "qps_batch_runs": [ + 62395.68220625229, + 53647.20525730502, + 49378.936394904355 + ], + "qps_batch": 53647.20525730502, + "qps_batch_min": 49378.936394904355, + "qps_batch_max": 62395.68220625229, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9997, + "qps_1t_runs": [ + 4377.404015145242, + 4260.5614547100195, + 4274.168237013549 + ], + "qps_1t": 4274.168237013549, + "qps_1t_min": 4260.5614547100195, + "qps_1t_max": 4377.404015145242, + "median_ms": 0.22960000205785036, + "p95_ms": 0.31460000900551677, + "p99_ms": 0.41180399246513855, + "mean_ms": 0.2323734665211911, + "knob": 256, + "batch": { + "recall": 0.9997, + "qps_batch_runs": [ + 47855.13299708468, + 46291.47427771338, + 51960.468458886695 + ], + "qps_batch": 47855.13299708468, + "qps_batch_min": 46291.47427771338, + "qps_batch_max": 51960.468458886695, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99995, + "qps_1t_runs": [ + 2973.0152739995783, + 2940.223492592194, + 2765.8056811963256 + ], + "qps_1t": 2940.223492592194, + "qps_1t_min": 2765.8056811963256, + "qps_1t_max": 2973.0152739995783, + "median_ms": 0.32920000376179814, + "p95_ms": 0.4738099989481271, + "p99_ms": 0.8397219912149017, + "mean_ms": 0.34600911678474705, + "knob": 400, + "batch": { + "recall": 0.99995, + "qps_batch_runs": [ + 33657.341359052145, + 36979.923600510476, + 36822.44404639132 + ], + "qps_batch": 36822.44404639132, + "qps_batch_min": 33657.341359052145, + "qps_batch_max": 36979.923600510476, + "threads": 32 + } + } + ], + "mutation": { + "ref_knob": 32, + "recall_before": 0.95025, + "insert_10k": { + "seconds": 0.23788480000803247, + "per_s": 42037.15411687648, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.5695672999718226, + "per_s": 17557.1877116097, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.007995199994184077, + "per_s": 1250750.4511799915, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 152.32421875, + "peak_mb": 239.421875 + }, + "first_query_after_ms": 0.10080001084133983, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.94095, + "qps_1t_runs": [ + 20147.236095205542, + 20960.932950558163, + 20559.82347651121 + ], + "qps_1t": 20559.82347651121, + "qps_1t_min": 20147.236095205542, + "qps_1t_max": 20960.932950558163, + "median_ms": 0.04689994966611266, + "p95_ms": 0.06659998325631022, + "p99_ms": 0.08729996683541685, + "mean_ms": 0.048660316601550825 + }, + "recall_after": 0.94095 + }, + "saveload": { + "save_s": 0.06717080000089481, + "disk_mb": 69.29621124267578, + "load_s": 0.15757999999914318, + "first_query_after_load_ms": 0.2136999974027276, + "time_to_ready_s": 0.1577936999965459 + }, + "load_before_pct": 8.856250000000006, + "load_contaminated": false + }, + "hnswlib|text384|1000000": { + "mem_baseline": { + "rss_mb": 37.37890625, + "peak_mb": 37.37890625 + }, + "version": "hnswlib 0.8.0", + "build": { + "ingest_s": 172.9470293999184, + "index_s": 4.800036549568176e-06, + "total_s": 172.94703419995494, + "first_query_ms": 3.6525000468827784, + "ready_s": 172.95068670000182 + }, + "mem_after_build": { + "rss_mb": 1867.3984375, + "peak_mb": 1897.1953125 + }, + "mem_after_first_query": { + "rss_mb": 1867.4140625, + "peak_mb": 1897.1953125 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.5589, + "qps_1t_runs": [ + 7939.9454134163125, + 9989.71059184505, + 11884.57227022613 + ], + "qps_1t": 9989.71059184505, + "qps_1t_min": 7939.9454134163125, + "qps_1t_max": 11884.57227022613, + "median_ms": 0.09200000204145908, + "p95_ms": 0.16950501012615862, + "p99_ms": 0.25110899121500574, + "mean_ms": 0.1033970501254468, + "knob": 10, + "batch": { + "recall": 0.5589, + "qps_batch_runs": [ + 45334.923072481906, + 47940.01741762274, + 45715.2261976148 + ], + "qps_batch": 45715.2261976148, + "qps_batch_min": 45334.923072481906, + "qps_batch_max": 47940.01741762274, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6512, + "qps_1t_runs": [ + 8071.373527158176, + 8872.409890884685, + 8803.514351564298 + ], + "qps_1t": 8803.514351564298, + "qps_1t_min": 8071.373527158176, + "qps_1t_max": 8872.409890884685, + "median_ms": 0.10870001278817654, + "p95_ms": 0.16351502272300428, + "p99_ms": 0.20780301070772117, + "mean_ms": 0.11673153335383783, + "knob": 16, + "batch": { + "recall": 0.6512, + "qps_batch_runs": [ + 34565.835815985214, + 36683.112920381, + 37715.14130693933 + ], + "qps_batch": 36683.112920381, + "qps_batch_min": 34565.835815985214, + "qps_batch_max": 37715.14130693933, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.72775, + "qps_1t_runs": [ + 6565.330768994968, + 6992.124406289892, + 7079.8840931398545 + ], + "qps_1t": 6992.124406289892, + "qps_1t_min": 6565.330768994968, + "qps_1t_max": 7079.8840931398545, + "median_ms": 0.13610004680231214, + "p95_ms": 0.19749999046325684, + "p99_ms": 0.2486010303255171, + "mean_ms": 0.1455261836235877, + "knob": 24, + "batch": { + "recall": 0.72775, + "qps_batch_runs": [ + 27745.061706597382, + 27873.28247232849, + 28350.095336296166 + ], + "qps_batch": 27873.28247232849, + "qps_batch_min": 27745.061706597382, + "qps_batch_max": 28350.095336296166, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.77635, + "qps_1t_runs": [ + 5293.43493108574, + 5557.560899104856, + 5665.078842947724 + ], + "qps_1t": 5557.560899104856, + "qps_1t_min": 5293.43493108574, + "qps_1t_max": 5665.078842947724, + "median_ms": 0.1729000359773636, + "p95_ms": 0.2474049630109221, + "p99_ms": 0.31220499076880526, + "mean_ms": 0.1817894502892159, + "knob": 32, + "batch": { + "recall": 0.77635, + "qps_batch_runs": [ + 22528.205305215724, + 23128.63336314208, + 22938.568208679393 + ], + "qps_batch": 22938.568208679393, + "qps_batch_min": 22528.205305215724, + "qps_batch_max": 23128.63336314208, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8354, + "qps_1t_runs": [ + 4386.61424591747, + 4275.635692782341, + 4196.184188634086 + ], + "qps_1t": 4275.635692782341, + "qps_1t_min": 4196.184188634086, + "qps_1t_max": 4386.61424591747, + "median_ms": 0.22429999080486596, + "p95_ms": 0.30629994871560484, + "p99_ms": 0.39223196043167335, + "mean_ms": 0.2333871169539634, + "knob": 48, + "batch": { + "recall": 0.8354, + "qps_batch_runs": [ + 16080.531299932483, + 17164.77410402789, + 17655.83480798102 + ], + "qps_batch": 17164.77410402789, + "qps_batch_min": 16080.531299932483, + "qps_batch_max": 17655.83480798102, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8727, + "qps_1t_runs": [ + 3223.399826829237, + 3211.24500821457, + 3269.922989365925 + ], + "qps_1t": 3223.399826829237, + "qps_1t_min": 3211.24500821457, + "qps_1t_max": 3269.922989365925, + "median_ms": 0.29559998074546456, + "p95_ms": 0.40610500727780163, + "p99_ms": 0.5568379908800133, + "mean_ms": 0.30915158306015655, + "knob": 64, + "batch": { + "recall": 0.8727, + "qps_batch_runs": [ + 13480.168315181072, + 13764.767012079901, + 13524.698806255925 + ], + "qps_batch": 13524.698806255925, + "qps_batch_min": 13480.168315181072, + "qps_batch_max": 13764.767012079901, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.91005, + "qps_1t_runs": [ + 2481.8662463574624, + 2351.890293282098, + 2493.6673315274925 + ], + "qps_1t": 2481.8662463574624, + "qps_1t_min": 2351.890293282098, + "qps_1t_max": 2493.6673315274925, + "median_ms": 0.39840000681579113, + "p95_ms": 0.5230050242971629, + "p99_ms": 0.663223986048252, + "mean_ms": 0.4097094334332117, + "knob": 96, + "batch": { + "recall": 0.91005, + "qps_batch_runs": [ + 9534.210895498365, + 9471.589494207263, + 9318.40090669534 + ], + "qps_batch": 9471.589494207263, + "qps_batch_min": 9318.40090669534, + "qps_batch_max": 9534.210895498365, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.93155, + "qps_1t_runs": [ + 1862.1670481633507, + 1987.39672440239, + 1908.1160091121474 + ], + "qps_1t": 1908.1160091121474, + "qps_1t_min": 1862.1670481633507, + "qps_1t_max": 1987.39672440239, + "median_ms": 0.5098499823361635, + "p95_ms": 0.6543450319441037, + "p99_ms": 0.8159069844987245, + "mean_ms": 0.5214189002678419, + "knob": 128, + "batch": { + "recall": 0.93155, + "qps_batch_runs": [ + 7621.167028207698, + 7534.069061083572, + 7173.608867673934 + ], + "qps_batch": 7534.069061083572, + "qps_batch_min": 7173.608867673934, + "qps_batch_max": 7621.167028207698, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9572, + "qps_1t_runs": [ + 1385.3156401395422, + 1348.0865230657812, + 1332.3331063397723 + ], + "qps_1t": 1348.0865230657812, + "qps_1t_min": 1332.3331063397723, + "qps_1t_max": 1385.3156401395422, + "median_ms": 0.7179499953053892, + "p95_ms": 0.9054200229002164, + "p99_ms": 1.2947230215650056, + "mean_ms": 0.7380707833120445, + "knob": 192, + "batch": { + "recall": 0.9572, + "qps_batch_runs": [ + 5355.257042319188, + 5337.06358438381, + 5443.139603460202 + ], + "qps_batch": 5355.257042319188, + "qps_batch_min": 5337.06358438381, + "qps_batch_max": 5443.139603460202, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.96845, + "qps_1t_runs": [ + 1042.9634193553227, + 1036.723278516595, + 1047.174587481958 + ], + "qps_1t": 1042.9634193553227, + "qps_1t_min": 1036.723278516595, + "qps_1t_max": 1047.174587481958, + "median_ms": 0.9442000009585172, + "p95_ms": 1.1992049927357584, + "p99_ms": 1.527310988167301, + "mean_ms": 0.9594448493347348, + "knob": 256, + "batch": { + "recall": 0.96845, + "qps_batch_runs": [ + 4286.852138612777, + 4059.25373856563, + 3975.8886207427563 + ], + "qps_batch": 4059.25373856563, + "qps_batch_min": 3975.8886207427563, + "qps_batch_max": 4286.852138612777, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9817, + "qps_1t_runs": [ + 685.2266474192253, + 720.9815992185978, + 700.4394766196394 + ], + "qps_1t": 700.4394766196394, + "qps_1t_min": 685.2266474192253, + "qps_1t_max": 720.9815992185978, + "median_ms": 1.406650000717491, + "p95_ms": 1.7451050167437643, + "p99_ms": 2.3742699756985544, + "mean_ms": 1.4246813998130772, + "knob": 400, + "batch": { + "recall": 0.9817, + "qps_batch_runs": [ + 2649.38034302476, + 2766.009177006968, + 2680.8576976003537 + ], + "qps_batch": 2680.8576976003537, + "qps_batch_min": 2649.38034302476, + "qps_batch_max": 2766.009177006968, + "threads": 32 + } + }, + { + "nq": 1224, + "recall_queries": 1224, + "recall": 0.9901143790849674, + "qps_1t_runs": [ + 460.69300640039273, + 471.90366917754915, + 472.23481150639606 + ], + "qps_1t": 471.90366917754915, + "qps_1t_min": 460.69300640039273, + "qps_1t_max": 472.23481150639606, + "median_ms": 2.146399987395853, + "p95_ms": 2.574079984333366, + "p99_ms": 2.989146026084199, + "mean_ms": 2.135770016271608, + "knob": 640, + "batch": { + "recall": 0.98985, + "qps_batch_runs": [ + 1842.6957016255428, + 1863.4264910216637, + 1848.0949144102544 + ], + "qps_batch": 1848.0949144102544, + "qps_batch_min": 1842.6957016255428, + "qps_batch_max": 1863.4264910216637, + "threads": 32 + } + } + ], + "mutation": { + "ref_knob": 192, + "recall_before": 0.9572, + "insert_10k": { + "seconds": 2.094509400019888, + "per_s": 4774.3877396325115, + "batch": 1000 + }, + "update_10k": { + "seconds": 3.0900770999724045, + "per_s": 3236.1652076866635, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.008781400043517351, + "per_s": 1138770.5776349695, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 1919.6796875, + "peak_mb": 1919.8125 + }, + "first_query_after_ms": 0.7746999617666006, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.95635, + "qps_1t_runs": [ + 1319.6338612899997, + 1335.8710433239014, + 1320.6679515979642 + ], + "qps_1t": 1320.6679515979642, + "qps_1t_min": 1319.6338612899997, + "qps_1t_max": 1335.8710433239014, + "median_ms": 0.7574500050395727, + "p95_ms": 0.916799996048212, + "p99_ms": 1.09075798187405, + "mean_ms": 0.754517933470197 + }, + "recall_after": 0.95635 + }, + "saveload": { + "save_s": 1.0720006999908946, + "disk_mb": 1622.5444107055664, + "load_s": 2.1154646999784745, + "first_query_after_load_ms": 1.5783000271767378, + "time_to_ready_s": 2.1170430000056513 + }, + "load_before_pct": 5.88125, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 150.55821330007166, + "mem": { + "rss_mb": 1868.0, + "peak_mb": 1897.7734375 + }, + "subsets": { + "f10": [ + { + "knob": 16, + "nq": 2000, + "recall": 0.8597, + "qps_1t": 965.5264101051243, + "qps_1t_runs": [ + 815.9749286167372, + 965.5264101051243, + 1311.292860013389 + ], + "median_ms": 0.8855000196490437, + "p95_ms": 1.792909973300994, + "p99_ms": 3.1010729796253185, + "mean_ms": 1.007946150319185 + }, + { + "knob": 48, + "nq": 1915, + "recall": 0.9550913838120104, + "qps_1t": 406.06719944948367, + "qps_1t_runs": [ + 406.06719944948367 + ], + "median_ms": 2.4363999837078154, + "p95_ms": 3.3190599933732297, + "p99_ms": 3.7982520007062694, + "mean_ms": 2.4626465800629234 + }, + { + "knob": 96, + "nq": 743, + "recall": 0.9826379542395693, + "qps_1t": 206.41749494898892, + "qps_1t_runs": [ + 206.41749494898892, + 206.1225098979031, + 207.06033962903118 + ], + "median_ms": 4.820000031031668, + "p95_ms": 6.258079991675912, + "p99_ms": 7.112372014671559, + "mean_ms": 4.841848137342868 + }, + { + "knob": 192, + "nq": 342, + "recall": 0.9953216374269006, + "qps_1t": 111.41473254946023, + "qps_1t_runs": [ + 110.39197802661296, + 111.41473254946023, + 113.79916722395953 + ], + "median_ms": 8.90740001341328, + "p95_ms": 11.198549997061491, + "p99_ms": 12.305099968216382, + "mean_ms": 8.940504483522785 + }, + { + "knob": 640, + "nq": 120, + "recall": 0.9991666666666666, + "qps_1t": 40.45463865958441, + "qps_1t_runs": [ + 40.41673561092942, + 40.77178405057407, + 40.45463865958441 + ], + "median_ms": 24.713349994271994, + "p95_ms": 29.000344980158843, + "p99_ms": 31.51762301393323, + "mean_ms": 24.662678613417988 + } + ], + "f01": [ + { + "knob": 16, + "nq": 486, + "recall": 0.9446502057613169, + "qps_1t": 135.0047551718261, + "qps_1t_runs": [ + 135.0047551718261, + 132.9490940372456, + 138.19347852491904 + ], + "median_ms": 7.16199999442324, + "p95_ms": 10.532300028717144, + "p99_ms": 12.1265950112138, + "mean_ms": 7.388351303483749 + }, + { + "knob": 48, + "nq": 168, + "recall": 0.9910714285714286, + "qps_1t": 52.143865538030056, + "qps_1t_runs": [ + 51.028790542712166, + 52.702241560200754, + 52.143865538030056 + ], + "median_ms": 19.08254998852499, + "p95_ms": 25.072974979411807, + "p99_ms": 29.955342992907365, + "mean_ms": 19.249672221901296 + }, + { + "knob": 96, + "nq": 84, + "recall": 0.9940476190476191, + "qps_1t": 28.00126939239545, + "qps_1t_runs": [ + 27.301230465099742, + 28.00126939239545, + 28.041923611467304 + ], + "median_ms": 36.020000028656796, + "p95_ms": 44.82396502280608, + "p99_ms": 48.251943037030294, + "mean_ms": 36.00064801543744 + }, + { + "knob": 192, + "nq": 49, + "recall": 1.0, + "qps_1t": 16.3899491341929, + "qps_1t_runs": [ + 16.52350009199954, + 15.453822305968659, + 16.3899491341929 + ], + "median_ms": 62.02630000188947, + "p95_ms": 77.42504997877401, + "p99_ms": 82.25276600336656, + "mean_ms": 62.08059252049698 + }, + { + "knob": 640, + "nq": 30, + "recall": 1.0, + "qps_1t": 6.01650013147678, + "qps_1t_runs": [ + 6.01650013147678 + ], + "median_ms": 167.38199998508207, + "p95_ms": 190.8775699936086, + "p99_ms": 201.73997197591234, + "mean_ms": 166.20958666123138 + } + ] + } + } + }, + "hnswlib|text384|100000": { + "mem_baseline": { + "rss_mb": 37.33984375, + "peak_mb": 37.33984375 + }, + "version": "hnswlib 0.8.0", + "build": { + "ingest_s": 6.211588800011668, + "index_s": 2.6999623514711857e-06, + "total_s": 6.21159149997402, + "first_query_ms": 1.7135999514721334, + "ready_s": 6.213305099925492, + "total_s_runs": [ + 6.21159149997402, + 6.21282560005784, + 6.091282200010028 + ] + }, + "mem_after_build": { + "rss_mb": 231.16796875, + "peak_mb": 262.8984375 + }, + "mem_after_first_query": { + "rss_mb": 231.19921875, + "peak_mb": 262.8984375 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.61845, + "qps_1t_runs": [ + 17225.623834351933, + 16406.823625073837, + 16855.447770790626 + ], + "qps_1t": 16855.447770790626, + "qps_1t_min": 16406.823625073837, + "qps_1t_max": 17225.623834351933, + "median_ms": 0.05549995694309473, + "p95_ms": 0.08239998714998364, + "p99_ms": 0.10400803468655813, + "mean_ms": 0.059443766493738316, + "knob": 10, + "batch": { + "recall": 0.61845, + "qps_batch_runs": [ + 93453.14035921605, + 95652.13252624813, + 89512.69293528692 + ], + "qps_batch": 93453.14035921605, + "qps_batch_min": 89512.69293528692, + "qps_batch_max": 95652.13252624813, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.71605, + "qps_1t_runs": [ + 13312.096264093334, + 13172.479852289463, + 13035.277370789938 + ], + "qps_1t": 13172.479852289463, + "qps_1t_min": 13035.277370789938, + "qps_1t_max": 13312.096264093334, + "median_ms": 0.07240002742037177, + "p95_ms": 0.10459998447913677, + "p99_ms": 0.13390297186560934, + "mean_ms": 0.07591679958083357, + "knob": 16, + "batch": { + "recall": 0.71605, + "qps_batch_runs": [ + 63072.381851184095, + 75401.98687064977, + 75207.19569016236 + ], + "qps_batch": 75207.19569016236, + "qps_batch_min": 63072.381851184095, + "qps_batch_max": 75401.98687064977, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.787, + "qps_1t_runs": [ + 9859.793670188494, + 9987.146542771392, + 9649.864291725911 + ], + "qps_1t": 9859.793670188494, + "qps_1t_min": 9649.864291725911, + "qps_1t_max": 9987.146542771392, + "median_ms": 0.09560002945363522, + "p95_ms": 0.13160501257516447, + "p99_ms": 0.1758019998669625, + "mean_ms": 0.10172636699280702, + "knob": 24, + "batch": { + "recall": 0.787, + "qps_batch_runs": [ + 51953.044843418575, + 52453.925795430834, + 57681.10426298905 + ], + "qps_batch": 52453.925795430834, + "qps_batch_min": 51953.044843418575, + "qps_batch_max": 57681.10426298905, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8309, + "qps_1t_runs": [ + 8229.501019281879, + 7449.621947561647, + 7739.848228119891 + ], + "qps_1t": 7739.848228119891, + "qps_1t_min": 7449.621947561647, + "qps_1t_max": 8229.501019281879, + "median_ms": 0.11600001016631722, + "p95_ms": 0.16080500208772722, + "p99_ms": 0.24430599703919156, + "mean_ms": 0.12831684993579984, + "knob": 32, + "batch": { + "recall": 0.8309, + "qps_batch_runs": [ + 44188.348425563265, + 48013.67435386259, + 46850.147496390935 + ], + "qps_batch": 46850.147496390935, + "qps_batch_min": 44188.348425563265, + "qps_batch_max": 48013.67435386259, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8855, + "qps_1t_runs": [ + 6085.143953116595, + 5802.094611550192, + 6165.389660777488 + ], + "qps_1t": 6085.143953116595, + "qps_1t_min": 5802.094611550192, + "qps_1t_max": 6165.389660777488, + "median_ms": 0.1572000328451395, + "p95_ms": 0.20570499473251405, + "p99_ms": 0.26632199063897183, + "mean_ms": 0.16629398326040246, + "knob": 48, + "batch": { + "recall": 0.8855, + "qps_batch_runs": [ + 33392.716714494665, + 31074.52604981799, + 33902.15501498349 + ], + "qps_batch": 33392.716714494665, + "qps_batch_min": 31074.52604981799, + "qps_batch_max": 33902.15501498349, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9162, + "qps_1t_runs": [ + 4838.660895718044, + 4725.756108459523, + 4337.287420002313 + ], + "qps_1t": 4725.756108459523, + "qps_1t_min": 4337.287420002313, + "qps_1t_max": 4838.660895718044, + "median_ms": 0.20209999638609588, + "p95_ms": 0.27279998175799847, + "p99_ms": 0.3774740215158106, + "mean_ms": 0.2162779833500584, + "knob": 64, + "batch": { + "recall": 0.9162, + "qps_batch_runs": [ + 26177.153875592645, + 27882.76412342768, + 26037.767778750873 + ], + "qps_batch": 26177.153875592645, + "qps_batch_min": 26037.767778750873, + "qps_batch_max": 27882.76412342768, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9488, + "qps_1t_runs": [ + 3413.8297308579226, + 3563.313753701257, + 3575.553866166564 + ], + "qps_1t": 3563.313753701257, + "qps_1t_min": 3413.8297308579226, + "qps_1t_max": 3575.553866166564, + "median_ms": 0.2798999776132405, + "p95_ms": 0.343804998556152, + "p99_ms": 0.4275569960009319, + "mean_ms": 0.2844135835475754, + "knob": 96, + "batch": { + "recall": 0.9488, + "qps_batch_runs": [ + 20021.002020894895, + 20348.05345627372, + 20267.120647963467 + ], + "qps_batch": 20267.120647963467, + "qps_batch_min": 20021.002020894895, + "qps_batch_max": 20348.05345627372, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.96445, + "qps_1t_runs": [ + 2715.3986587391196, + 2704.1121273724907, + 2727.729249379135 + ], + "qps_1t": 2715.3986587391196, + "qps_1t_min": 2704.1121273724907, + "qps_1t_max": 2727.729249379135, + "median_ms": 0.3541000187397003, + "p95_ms": 0.439604974235408, + "p99_ms": 0.5871070461580531, + "mean_ms": 0.36822750015805167, + "knob": 128, + "batch": { + "recall": 0.96445, + "qps_batch_runs": [ + 15528.251714547954, + 15857.259293311603, + 15704.73835204133 + ], + "qps_batch": 15704.73835204133, + "qps_batch_min": 15528.251714547954, + "qps_batch_max": 15857.259293311603, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.98095, + "qps_1t_runs": [ + 1959.1762580813638, + 1962.101230154518, + 1929.5917084535288 + ], + "qps_1t": 1959.1762580813638, + "qps_1t_min": 1929.5917084535288, + "qps_1t_max": 1962.101230154518, + "median_ms": 0.5038999952375889, + "p95_ms": 0.6147000240162015, + "p99_ms": 0.7766069762874396, + "mean_ms": 0.5127735493782287, + "knob": 192, + "batch": { + "recall": 0.98095, + "qps_batch_runs": [ + 11013.14308374829, + 10961.066295289133, + 10596.655271550744 + ], + "qps_batch": 10961.066295289133, + "qps_batch_min": 10596.655271550744, + "qps_batch_max": 11013.14308374829, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.98695, + "qps_1t_runs": [ + 1553.4706954621568, + 1558.0544324838709, + 1565.162120047441 + ], + "qps_1t": 1558.0544324838709, + "qps_1t_min": 1553.4706954621568, + "qps_1t_max": 1565.162120047441, + "median_ms": 0.6339999963529408, + "p95_ms": 0.758910016156733, + "p99_ms": 0.8619120437651875, + "mean_ms": 0.6414858165953774, + "knob": 256, + "batch": { + "recall": 0.98695, + "qps_batch_runs": [ + 8700.441199111725, + 8390.595653448469, + 8427.677672575583 + ], + "qps_batch": 8427.677672575583, + "qps_batch_min": 8390.595653448469, + "qps_batch_max": 8700.441199111725, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9933, + "qps_1t_runs": [ + 1026.0785023027886, + 1037.470314984736, + 1017.1449962061066 + ], + "qps_1t": 1026.0785023027886, + "qps_1t_min": 1017.1449962061066, + "qps_1t_max": 1037.470314984736, + "median_ms": 0.9490000084042549, + "p95_ms": 1.169709998066537, + "p99_ms": 1.4647279772907502, + "mean_ms": 0.9738704337602636, + "knob": 400, + "batch": { + "recall": 0.9933, + "qps_batch_runs": [ + 6018.400658157129, + 5967.642843979703, + 6069.327500040121 + ], + "qps_batch": 6018.400658157129, + "qps_batch_min": 5967.642843979703, + "qps_batch_max": 6069.327500040121, + "threads": 32 + } + }, + { + "nq": 1874, + "recall_queries": 1874, + "recall": 0.9969583778014941, + "qps_1t_runs": [ + 689.6798930975095, + 697.2763284482164, + 695.8861615719146 + ], + "qps_1t": 695.8861615719146, + "qps_1t_min": 689.6798930975095, + "qps_1t_max": 697.2763284482164, + "median_ms": 1.4125999878160655, + "p95_ms": 1.7367000109516084, + "p99_ms": 2.2665380290709436, + "mean_ms": 1.440372110005867, + "knob": 640, + "batch": { + "recall": 0.997, + "qps_batch_runs": [ + 4124.082932811876, + 3945.9242645267195, + 3492.5963072247396 + ], + "qps_batch": 3945.9242645267195, + "qps_batch_min": 3492.5963072247396, + "qps_batch_max": 4124.082932811876, + "threads": 32 + } + } + ], + "mutation": { + "ref_knob": 128, + "recall_before": 0.96445, + "insert_10k": { + "seconds": 1.105076999985613, + "per_s": 9049.143181995636, + "batch": 1000 + }, + "update_10k": { + "seconds": 2.965886700025294, + "per_s": 3371.672963742923, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.011413100000936538, + "per_s": 876186.1369110424, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 284.1796875, + "peak_mb": 458.48046875 + }, + "first_query_after_ms": 0.6867000483907759, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9683, + "qps_1t_runs": [ + 2083.5219350971756, + 2218.385892404205, + 2238.180451599162 + ], + "qps_1t": 2218.385892404205, + "qps_1t_min": 2083.5219350971756, + "qps_1t_max": 2238.180451599162, + "median_ms": 0.4464000230655074, + "p95_ms": 0.5581100005656481, + "p99_ms": 0.6933049822691829, + "mean_ms": 0.45917541658855043 + }, + "recall_after": 0.9683 + }, + "saveload": { + "save_s": 0.14747150003677234, + "disk_mb": 176.71808624267578, + "load_s": 0.22453179996227846, + "first_query_after_load_ms": 0.6153000285848975, + "time_to_ready_s": 0.22514709999086335 + }, + "load_before_pct": 3.5593749999999975, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 6.4425217001116835, + "mem": { + "rss_mb": 230.86328125, + "peak_mb": 262.44140625 + }, + "subsets": { + "f10": [ + { + "knob": 16, + "nq": 2000, + "recall": 0.91235, + "qps_1t": 1778.6843554561513, + "qps_1t_runs": [ + 1714.9114129067339, + 1778.6843554561513, + 1781.5343607501627 + ], + "median_ms": 0.5446000141091645, + "p95_ms": 0.8071250311331825, + "p99_ms": 1.042873024125585, + "mean_ms": 0.5688825663431393 + }, + { + "knob": 48, + "nq": 2000, + "recall": 0.9833, + "qps_1t": 681.4732403313656, + "qps_1t_runs": [ + 697.3406114780528, + 647.9030068889845, + 681.4732403313656 + ], + "median_ms": 1.3814999838359654, + "p95_ms": 1.894315041135997, + "p99_ms": 3.1411739939358125, + "mean_ms": 1.4816231331399952 + }, + { + "knob": 96, + "nq": 1244, + "recall": 0.9949356913183279, + "qps_1t": 385.2228753884162, + "qps_1t_runs": [ + 372.54788090474375, + 387.9784442935935, + 385.2228753884162 + ], + "median_ms": 2.520250011002645, + "p95_ms": 3.426375013077631, + "p99_ms": 5.214121007011277, + "mean_ms": 2.6191937833898904 + }, + { + "knob": 192, + "nq": 701, + "recall": 0.998716119828816, + "qps_1t": 217.47272683683482, + "qps_1t_runs": [ + 217.47272683683482, + 216.6985222890845, + 228.98169362459495 + ], + "median_ms": 4.44779999088496, + "p95_ms": 5.740439996588975, + "p99_ms": 7.4728159874212015, + "mean_ms": 4.526715073059169 + }, + { + "knob": 640, + "nq": 265, + "recall": 1.0, + "qps_1t": 85.00550387736226, + "qps_1t_runs": [ + 85.00550387736226, + 86.72104458735168, + 83.45960930995703 + ], + "median_ms": 11.389799998141825, + "p95_ms": 13.891609973506998, + "p99_ms": 21.025283990893488, + "mean_ms": 11.759004778369277 + } + ], + "f01": [ + { + "knob": 16, + "nq": 762, + "recall": 0.9813648293963254, + "qps_1t": 258.7360249980111, + "qps_1t_runs": [ + 241.37487880135672, + 258.7360249980111, + 270.94914233767344 + ], + "median_ms": 3.725250018760562, + "p95_ms": 5.672749975929037, + "p99_ms": 8.250459967530347, + "mean_ms": 3.899535258347046 + }, + { + "knob": 48, + "nq": 358, + "recall": 0.9986033519553073, + "qps_1t": 107.85546934809844, + "qps_1t_runs": [ + 107.82210844960854, + 109.81509869096007, + 107.85546934809844 + ], + "median_ms": 8.957550016930327, + "p95_ms": 12.161819997709244, + "p99_ms": 18.01534997648559, + "mean_ms": 9.217472904410778 + }, + { + "knob": 96, + "nq": 200, + "recall": 1.0, + "qps_1t": 63.843694279518886, + "qps_1t_runs": [ + 63.18820047455781, + 63.843694279518886, + 64.12194248917788 + ], + "median_ms": 15.221199981169775, + "p95_ms": 19.741240015719086, + "p99_ms": 24.973395992419682, + "mean_ms": 15.694759667094331 + }, + { + "knob": 192, + "nq": 120, + "recall": 1.0, + "qps_1t": 39.407314004695856, + "qps_1t_runs": [ + 39.01131416105161, + 39.51068258689269, + 39.407314004695856 + ], + "median_ms": 25.014050013851374, + "p95_ms": 29.422394998255186, + "p99_ms": 39.64057695528036, + "mean_ms": 25.43973333102056 + }, + { + "knob": 640, + "nq": 60, + "recall": 1.0, + "qps_1t": 19.460341381440205, + "qps_1t_runs": [ + 19.56792010367712, + 19.460341381440205, + 19.355191474301996 + ], + "median_ms": 50.41320002055727, + "p95_ms": 58.380130023579106, + "p99_ms": 65.33958000713028, + "mean_ms": 51.38544555488301 + } + ] + } + } + }, + "isong_py|glove|1000000": { + "mem_baseline": { + "rss_mb": 35.0078125, + "peak_mb": 35.0078125 + }, + "version": "isongraph-vector (python, repo HEAD)", + "build": { + "ingest_s": 27.908122700056992, + "index_s": 4.200031980872154e-06, + "total_s": 27.908126900088973, + "first_query_ms": 4899.777499958873, + "ready_s": 32.807904400047846 + }, + "mem_after_build": { + "rss_mb": 729.9296875, + "peak_mb": 798.8125 + }, + "mem_after_first_query": { + "rss_mb": 1459.21875, + "peak_mb": 1470.53515625 + }, + "sweep": [ + { + "nq": 84, + "recall_queries": 84, + "recall": 1.0, + "qps_1t_runs": [ + 27.887184253289014, + 26.46472568856041, + 26.159219335618886 + ], + "qps_1t": 26.46472568856041, + "qps_1t_min": 26.159219335618886, + "qps_1t_max": 27.887184253289014, + "median_ms": 37.09220001474023, + "p95_ms": 42.527489989879534, + "p99_ms": 47.820610992494046, + "mean_ms": 37.29078412635274, + "knob": null + } + ], + "loop_allthreads": { + "nq": 93, + "threads": 32, + "qps": 27.714460003367048, + "median_ms": 35.600699950009584, + "p95_ms": 49.929859989788376, + "p99_ms": 52.247072028694596, + "mean_ms": 36.24233978739556 + }, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.8805213999585249, + "per_s": 11356.907396539174, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.6018026000238024, + "per_s": 16616.74442683445, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.5086630000150762, + "per_s": 19659.381554592353, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 1472.03515625, + "peak_mb": 1472.03515625 + }, + "first_query_after_ms": 100.25179997319356, + "after": { + "nq": 77, + "recall_queries": 77, + "recall": 1.0, + "qps_1t_runs": [ + 26.34213520958877, + 27.546546775197086, + 27.93582075127861 + ], + "qps_1t": 27.546546775197086, + "qps_1t_min": 26.34213520958877, + "qps_1t_max": 27.93582075127861, + "median_ms": 36.20279999449849, + "p95_ms": 42.87535001640208, + "p99_ms": 47.78328002430498, + "mean_ms": 36.68684026179029 + }, + "recall_after": 1.0 + }, + "load_before_pct": 7.521874999999993, + "load_contaminated": false + }, + "isong_py|glove|100000": { + "mem_baseline": { + "rss_mb": 35.02734375, + "peak_mb": 35.02734375 + }, + "version": "isongraph-vector (python, repo HEAD)", + "build": { + "ingest_s": 0.9344128000084311, + "index_s": 2.400018274784088e-06, + "total_s": 0.9344152000267059, + "first_query_ms": 179.07509999349713, + "ready_s": 1.113490300020203, + "total_s_runs": [ + 0.9344152000267059, + 0.9137653000070713, + 0.9312418999616057 + ] + }, + "mem_after_build": { + "rss_mb": 111.9140625, + "peak_mb": 199.09765625 + }, + "mem_after_first_query": { + "rss_mb": 194.46484375, + "peak_mb": 199.09765625 + }, + "sweep": [ + { + "nq": 1462, + "recall_queries": 1462, + "recall": 1.0, + "qps_1t_runs": [ + 470.57277024090325, + 487.232330096314, + 468.2267177290404 + ], + "qps_1t": 470.57277024090325, + "qps_1t_min": 468.2267177290404, + "qps_1t_max": 487.232330096314, + "median_ms": 1.9831500248983502, + "p95_ms": 2.7151750109624118, + "p99_ms": 4.885184983140761, + "mean_ms": 2.1043987691772332, + "knob": null + } + ], + "loop_allthreads": { + "nq": 2000, + "threads": 32, + "qps": 733.8303687431866, + "median_ms": 1.276099996175617, + "p95_ms": 1.7859099782072008, + "p99_ms": 3.2207130180904646, + "mean_ms": 1.3543596997818288 + }, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.12977739999769256, + "per_s": 77055.01882591115, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.1261600000434555, + "per_s": 79264.42609825241, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.11555340001359582, + "per_s": 86540.07583354031, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 217.94921875, + "peak_mb": 339.4375 + }, + "first_query_after_ms": 3.411400015465915, + "after": { + "nq": 1133, + "recall_queries": 1133, + "recall": 1.0, + "qps_1t_runs": [ + 368.6149218486817, + 362.15402827025093, + 359.13217803847925 + ], + "qps_1t": 362.15402827025093, + "qps_1t_min": 359.13217803847925, + "qps_1t_max": 368.6149218486817, + "median_ms": 2.634999982547015, + "p95_ms": 3.3076899824663992, + "p99_ms": 4.197101996978744, + "mean_ms": 2.7528681083551927 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 4.347207100014202, + "disk_mb": 215.12163829803467, + "load_s": 3.3704436999978498, + "first_query_after_load_ms": 173.42160001862794, + "time_to_ready_s": 3.5438653000164777 + }, + "load_before_pct": 3.5187499999999963, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 0.951974900031928, + "mem": { + "rss_mb": 119.0625, + "peak_mb": 198.6328125 + }, + "subsets": { + "f10": [ + { + "knob": null, + "nq": 806, + "recall": 1.0, + "qps_1t": 368.30133653177387, + "qps_1t_runs": [ + 364.21885098908365, + 368.30133653177387, + 374.0330828163496 + ], + "median_ms": 2.841649984475225, + "p95_ms": 3.657465026481077, + "p99_ms": 5.016813997644929, + "mean_ms": 2.711443424126059 + } + ], + "f01": [ + { + "knob": null, + "nq": 1224, + "recall": 1.0, + "qps_1t": 409.2525993129238, + "qps_1t_runs": [ + 402.3647606355641, + 409.2525993129238, + 410.3109945700142 + ], + "median_ms": 2.3779500043019652, + "p95_ms": 2.875359982135705, + "p99_ms": 3.5022469749674197, + "mean_ms": 2.4553205072302786 + } + ] + } + } + }, + "isong_py|sift|100000": { + "mem_baseline": { + "rss_mb": 35.4453125, + "peak_mb": 35.4453125 + }, + "version": "isongraph-vector (python, repo HEAD)", + "build": { + "ingest_s": 1.2639303999603726, + "index_s": 2.800021320581436e-06, + "total_s": 1.2639331999816932, + "first_query_ms": 217.70449995528907, + "ready_s": 1.4816376999369822, + "total_s_runs": [ + 1.2639331999816932, + 4.486591300053988, + 4.277331799909007 + ] + }, + "mem_after_build": { + "rss_mb": 120.2890625, + "peak_mb": 230.625 + }, + "mem_after_first_query": { + "rss_mb": 205.3359375, + "peak_mb": 230.625 + }, + "sweep": [ + { + "nq": 1127, + "recall_queries": 1127, + "recall": 1.0, + "qps_1t_runs": [ + 323.34627532053725, + 318.4160723281366, + 265.48087410898836 + ], + "qps_1t": 318.4160723281366, + "qps_1t_min": 265.48087410898836, + "qps_1t_max": 323.34627532053725, + "median_ms": 3.2240000437013805, + "p95_ms": 4.41530003445223, + "p99_ms": 5.68031995790078, + "mean_ms": 3.3333181309646243, + "knob": null + } + ], + "loop_allthreads": { + "nq": 827, + "threads": 32, + "qps": 221.62838441176189, + "median_ms": 3.9862999692559242, + "p95_ms": 6.613600009586662, + "p99_ms": 9.214140009134965, + "mean_ms": 4.304634542628621 + }, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.809980700025335, + "per_s": 12345.973181443971, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.6764652999700047, + "per_s": 14782.724258647726, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.4257113999919966, + "per_s": 23490.09211448883, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 250.2421875, + "peak_mb": 391.02734375 + }, + "first_query_after_ms": 10.10459999088198, + "after": { + "nq": 669, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 202.95658902539296, + 215.85714318386917, + 211.1228945033651 + ], + "qps_1t": 211.1228945033651, + "qps_1t_min": 202.95658902539296, + "qps_1t_max": 215.85714318386917, + "median_ms": 4.807600053027272, + "p95_ms": 5.517460004193706, + "p99_ms": 6.0025979694910365, + "mean_ms": 4.765477778511164 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 10.997030099970289, + "disk_mb": 222.15946578979492, + "load_s": 9.703692900016904, + "first_query_after_load_ms": 528.2480000169016, + "time_to_ready_s": 10.231940900033806 + }, + "load_before_pct": 15.33437500000001, + "load_contaminated": false + }, + "isong_py|text384|1000000": { + "mem_baseline": { + "rss_mb": 37.44140625, + "peak_mb": 37.44140625 + }, + "version": "isongraph-vector (python, repo HEAD)", + "build": { + "ingest_s": 19.603895099658985, + "index_s": 2.8999638743698597e-06, + "total_s": 19.60389799962286, + "first_query_ms": 3159.0028000064194, + "ready_s": 22.76290079962928 + }, + "mem_after_build": { + "rss_mb": 2458.02734375, + "peak_mb": 2719.56640625 + }, + "mem_after_first_query": { + "rss_mb": 4260.78125, + "peak_mb": 4282.1875 + }, + "sweep": [ + { + "nq": 36, + "recall_queries": 36, + "recall": 1.0, + "qps_1t_runs": [ + 12.691703331185803, + 12.691139131289624, + 12.803293120965305 + ], + "qps_1t": 12.691703331185803, + "qps_1t_min": 12.691139131289624, + "qps_1t_max": 12.803293120965305, + "median_ms": 77.24439998855814, + "p95_ms": 86.11922999843955, + "p99_ms": 92.96392500284126, + "mean_ms": 78.56388981464853, + "knob": null + } + ], + "loop_allthreads": { + "nq": 64, + "threads": 32, + "qps": 21.255942570846003, + "median_ms": 46.58279998693615, + "p95_ms": 51.560625023557805, + "p99_ms": 56.12793797801715, + "mean_ms": 47.07301093882658 + }, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.28422920004231855, + "per_s": 35182.873534848324, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.21219099999871105, + "per_s": 47127.352244255155, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.1480594999738969, + "per_s": 67540.41450743124, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 4319.8125, + "peak_mb": 4319.8125 + }, + "first_query_after_ms": 76.06789999408647, + "after": { + "nq": 37, + "recall_queries": 37, + "recall": 1.0, + "qps_1t_runs": [ + 12.633175004505267, + 12.728609263827707, + 12.417905062494837 + ], + "qps_1t": 12.633175004505267, + "qps_1t_min": 12.417905062494837, + "qps_1t_max": 12.728609263827707, + "median_ms": 78.01810000091791, + "p95_ms": 86.07399999164045, + "p99_ms": 98.09642998152417, + "mean_ms": 79.41624143702053 + }, + "recall_after": 1.0 + }, + "load_before_pct": 4.121874999999999, + "load_contaminated": false + }, + "isong_py|text384|100000": { + "mem_baseline": { + "rss_mb": 37.32421875, + "peak_mb": 37.32421875 + }, + "version": "isongraph-vector (python, repo HEAD)", + "build": { + "ingest_s": 2.0098258999641985, + "index_s": 2.6999623514711857e-06, + "total_s": 2.00982859992655, + "first_query_ms": 302.6059999829158, + "ready_s": 2.3124345999094658, + "total_s_runs": [ + 2.00982859992655, + 1.9037308999104425, + 1.9565701999817975 + ] + }, + "mem_after_build": { + "rss_mb": 301.0703125, + "peak_mb": 608.5 + }, + "mem_after_first_query": { + "rss_mb": 491.3671875, + "peak_mb": 608.5 + }, + "sweep": [ + { + "nq": 411, + "recall_queries": 1000, + "recall": 0.9999, + "qps_1t_runs": [ + 130.79180149272884, + 129.21784657734457, + 130.55264871905612 + ], + "qps_1t": 130.55264871905612, + "qps_1t_min": 129.21784657734457, + "qps_1t_max": 130.79180149272884, + "median_ms": 7.368899998255074, + "p95_ms": 8.921960019506512, + "p99_ms": 14.31764797540383, + "mean_ms": 7.681451094858364, + "knob": null + } + ], + "loop_allthreads": { + "nq": 592, + "threads": 32, + "qps": 182.78046255038845, + "median_ms": 5.295849987305701, + "p95_ms": 5.986950010992587, + "p99_ms": 11.848275025840849, + "mean_ms": 5.468754278898904 + }, + "mutation": { + "ref_knob": null, + "recall_before": 0.9999, + "insert_10k": { + "seconds": 0.20596729998942465, + "per_s": 48551.39626782236, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.19502439995994791, + "per_s": 51275.63526437562, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.1280096999835223, + "per_s": 78119.0800485215, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 607.47265625, + "peak_mb": 1030.41015625 + }, + "first_query_after_ms": 9.15479997638613, + "after": { + "nq": 335, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 117.71077355262582, + 111.40207559022471, + 113.48448766836302 + ], + "qps_1t": 113.48448766836302, + "qps_1t_min": 111.40207559022471, + "qps_1t_max": 117.71077355262582, + "median_ms": 8.436200034338981, + "p95_ms": 10.36600002553314, + "p99_ms": 13.535275987815114, + "mean_ms": 8.761223183657547 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 15.478363099973649, + "disk_mb": 807.4883146286011, + "load_s": 10.803367899963632, + "first_query_after_load_ms": 290.74789996957406, + "time_to_ready_s": 11.094115799933206 + }, + "load_before_pct": 4.221874999999997, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 1.971411700011231, + "mem": { + "rss_mb": 305.13671875, + "peak_mb": 608.5234375 + }, + "subsets": { + "f10": [ + { + "knob": null, + "nq": 373, + "recall": 1.0, + "qps_1t": 120.92616604587073, + "qps_1t_runs": [ + 120.56053215637243, + 120.92616604587073, + 121.09234516207893 + ], + "median_ms": 8.045599970500916, + "p95_ms": 9.732029965380201, + "p99_ms": 14.989941958338022, + "mean_ms": 8.274085879593445 + } + ], + "f01": [ + { + "knob": null, + "nq": 413, + "recall": 1.0, + "qps_1t": 130.75389562362255, + "qps_1t_runs": [ + 130.40696316614958, + 132.61044267594022, + 130.75389562362255 + ], + "median_ms": 7.410000020172447, + "p95_ms": 8.720459992764518, + "p99_ms": 10.668384009040883, + "mean_ms": 7.619047377057843 + } + ] + } + } + }, + "isong_py|text384|10000": { + "mem_baseline": { + "rss_mb": 37.21875, + "peak_mb": 37.21875 + }, + "version": "isongraph-vector (python, repo HEAD)", + "build": { + "ingest_s": 0.18982440000399947, + "index_s": 2.700020559132099e-06, + "total_s": 0.1898271000245586, + "first_query_ms": 32.44519996223971, + "ready_s": 0.22227229998679832 + }, + "mem_after_build": { + "rss_mb": 71.58203125, + "peak_mb": 232.5390625 + }, + "mem_after_first_query": { + "rss_mb": 89.04296875, + "peak_mb": 232.5390625 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 1.0, + "qps_1t_runs": [ + 2336.9320918560566, + 2206.460009760146, + 2201.8797907733897 + ], + "qps_1t": 2206.460009760146, + "qps_1t_min": 2201.8797907733897, + "qps_1t_max": 2336.9320918560566, + "median_ms": 0.42050000047311187, + "p95_ms": 0.5032299901358784, + "p99_ms": 0.8731300116051017, + "mean_ms": 0.4450944997079205, + "knob": null + } + ], + "loop_allthreads": { + "nq": 2000, + "threads": 32, + "qps": 3333.930658197846, + "median_ms": 0.2832000027410686, + "p95_ms": 0.400005019037053, + "p99_ms": 0.6143000209704041, + "mean_ms": 0.3009677502947549 + }, + "load_before_pct": 5.9937500000000075, + "load_contaminated": false + }, + "isong_py_ivf|glove|1000000": { + "mem_baseline": { + "rss_mb": 35.23828125, + "peak_mb": 35.23828125 + }, + "version": "isongraph-vector (python, repo HEAD)", + "build": { + "ingest_s": 10.126719099993352, + "index_s": 11.820377000025474, + "total_s": 21.947096100018825, + "first_query_ms": 28.81889999844134, + "ready_s": 21.975915000017267 + }, + "mem_after_build": { + "rss_mb": 1479.4296875, + "peak_mb": 1668.84375 + }, + "mem_after_first_query": { + "rss_mb": 1479.52734375, + "peak_mb": 1668.84375 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.3014, + "qps_1t_runs": [ + 7673.6312150786935, + 7436.414915525494, + 7446.640160591303 + ], + "qps_1t": 7446.640160591303, + "qps_1t_min": 7436.414915525494, + "qps_1t_max": 7673.6312150786935, + "median_ms": 0.12725000851787627, + "p95_ms": 0.18259999342262745, + "p99_ms": 0.22690398152917632, + "mean_ms": 0.13302618371866023, + "knob": 1 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.41955, + "qps_1t_runs": [ + 5473.482212912351, + 5429.38126627681, + 5494.671545272573 + ], + "qps_1t": 5473.482212912351, + "qps_1t_min": 5429.38126627681, + "qps_1t_max": 5494.671545272573, + "median_ms": 0.17030001617968082, + "p95_ms": 0.24620500917080793, + "p99_ms": 0.3202259936369961, + "mean_ms": 0.18295886652776971, + "knob": 2 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.53585, + "qps_1t_runs": [ + 3936.734298016734, + 3631.4954561523077, + 3761.8734120014833 + ], + "qps_1t": 3761.8734120014833, + "qps_1t_min": 3631.4954561523077, + "qps_1t_max": 3936.734298016734, + "median_ms": 0.24750002194195986, + "p95_ms": 0.3604999539675191, + "p99_ms": 0.4447320097824566, + "mean_ms": 0.26507043366048794, + "knob": 4 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.639, + "qps_1t_runs": [ + 2411.959267253749, + 2437.508973631519, + 2541.966274453487 + ], + "qps_1t": 2437.508973631519, + "qps_1t_min": 2411.959267253749, + "qps_1t_max": 2541.966274453487, + "median_ms": 0.3833499795291573, + "p95_ms": 0.5698150314856321, + "p99_ms": 0.6890139501774687, + "mean_ms": 0.4060839498900653, + "knob": 8 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7264, + "qps_1t_runs": [ + 1512.3768765919078, + 1502.6607620354866, + 1496.5012562519441 + ], + "qps_1t": 1502.6607620354866, + "qps_1t_min": 1496.5012562519441, + "qps_1t_max": 1512.3768765919078, + "median_ms": 0.6358500104397535, + "p95_ms": 0.9423200157471002, + "p99_ms": 1.0954140167450535, + "mean_ms": 0.6649741163516106, + "knob": 16 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7996, + "qps_1t_runs": [ + 881.0629739591949, + 853.6949537055935, + 890.8565734356268 + ], + "qps_1t": 881.0629739591949, + "qps_1t_min": 853.6949537055935, + "qps_1t_max": 890.8565734356268, + "median_ms": 1.0962999949697405, + "p95_ms": 1.5826100017875433, + "p99_ms": 1.8951249937526908, + "mean_ms": 1.1429621335313034, + "knob": 32 + }, + { + "nq": 1635, + "recall_queries": 1635, + "recall": 0.8634250764525994, + "qps_1t_runs": [ + 498.4330757686888, + 492.3623903698503, + 492.0661185138374 + ], + "qps_1t": 492.3623903698503, + "qps_1t_min": 492.0661185138374, + "qps_1t_max": 498.4330757686888, + "median_ms": 1.9783999887295067, + "p95_ms": 2.7038399945013225, + "p99_ms": 3.073048002552241, + "mean_ms": 2.0231863199927815, + "knob": 64 + }, + { + "nq": 774, + "recall_queries": 1000, + "recall": 0.9207000000000001, + "qps_1t_runs": [ + 251.2547479487054, + 256.49987753534083, + 254.68469296302763 + ], + "qps_1t": 254.68469296302763, + "qps_1t_min": 251.2547479487054, + "qps_1t_max": 256.49987753534083, + "median_ms": 3.775549994315952, + "p95_ms": 5.023495008936152, + "p99_ms": 11.496879992191678, + "mean_ms": 3.9350284242885816, + "knob": 128 + }, + { + "nq": 418, + "recall_queries": 1000, + "recall": 0.9574, + "qps_1t_runs": [ + 138.35026249925076, + 136.85363665981453, + 140.22167975817968 + ], + "qps_1t": 138.35026249925076, + "qps_1t_min": 136.85363665981453, + "qps_1t_max": 140.22167975817968, + "median_ms": 6.950399983907118, + "p95_ms": 9.135000038077123, + "p99_ms": 14.947907001478601, + "mean_ms": 7.222224162364946, + "knob": 256 + }, + { + "nq": 111, + "recall_queries": 111, + "recall": 1.0, + "qps_1t_runs": [ + 35.67577928921252, + 32.765002232607976, + 34.60663402601248 + ], + "qps_1t": 34.60663402601248, + "qps_1t_min": 32.765002232607976, + "qps_1t_max": 35.67577928921252, + "median_ms": 27.95209997566417, + "p95_ms": 34.97580002294853, + "p99_ms": 53.4547840058804, + "mean_ms": 29.148928528999793, + "knob": 4000 + } + ], + "mutation": { + "ref_knob": 256, + "recall_before": 0.9574, + "insert_10k": { + "seconds": 0.16740310000022873, + "per_s": 59736.05028811495, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.15563410002505407, + "per_s": 64253.27096304855, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.1476425000000745, + "per_s": 67731.17496652355, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 1496.15234375, + "peak_mb": 1668.84375 + }, + "first_query_after_ms": 293.5758999665268, + "after": { + "nq": 423, + "recall_queries": 1000, + "recall": 0.9566999999999999, + "qps_1t_runs": [ + 141.2508143568193, + 135.43345478895924, + 138.8761622412545 + ], + "qps_1t": 138.8761622412545, + "qps_1t_min": 135.43345478895924, + "qps_1t_max": 141.2508143568193, + "median_ms": 7.083099975716323, + "p95_ms": 9.031679993495345, + "p99_ms": 10.92401604168117, + "mean_ms": 7.221321670062931 + }, + "recall_after": 0.9566999999999999, + "index_stats_after": { + "enabled": true, + "mode": "ivf", + "active": true, + "nlist": 4000, + "nprobe": 256, + "trained_n": 1000000, + "pending": 0, + "churn": 30000, + "build_seconds": 9.951588800002355, + "index_bytes": 10132072, + "min_size": 50000 + } + }, + "load_before_pct": 5.75625, + "load_contaminated": false + }, + "isong_py_ivf|glove|100000": { + "mem_baseline": { + "rss_mb": 35.4765625, + "peak_mb": 35.4765625 + }, + "version": "isongraph-vector (python, repo HEAD)", + "build": { + "ingest_s": 0.9291130999918096, + "index_s": 0.8479947000159882, + "total_s": 1.7771078000077978, + "first_query_ms": 3.174100012984127, + "ready_s": 1.780281900020782, + "total_s_runs": [ + 1.7771078000077978, + 1.7793858999502845, + 1.7270096999127418 + ] + }, + "mem_after_build": { + "rss_mb": 211.15234375, + "peak_mb": 313.45703125 + }, + "mem_after_first_query": { + "rss_mb": 211.2578125, + "peak_mb": 313.45703125 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.28525, + "qps_1t_runs": [ + 14707.947768870014, + 14669.736592289886, + 16767.593652751257 + ], + "qps_1t": 14707.947768870014, + "qps_1t_min": 14669.736592289886, + "qps_1t_max": 16767.593652751257, + "median_ms": 0.05859998054802418, + "p95_ms": 0.08270498947240414, + "p99_ms": 0.11720601934939635, + "mean_ms": 0.06526561683858745, + "knob": 1 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.40225, + "qps_1t_runs": [ + 14100.911595623664, + 12857.081349663158, + 12777.454446384369 + ], + "qps_1t": 12857.081349663158, + "qps_1t_min": 12777.454446384369, + "qps_1t_max": 14100.911595623664, + "median_ms": 0.06910000229254365, + "p95_ms": 0.09180000051856041, + "p99_ms": 0.12980999890714906, + "mean_ms": 0.07565280021905589, + "knob": 2 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.51915, + "qps_1t_runs": [ + 9812.348654203588, + 10234.48222592726, + 9844.026336901727 + ], + "qps_1t": 9844.026336901727, + "qps_1t_min": 9812.348654203588, + "qps_1t_max": 10234.48222592726, + "median_ms": 0.0944999628700316, + "p95_ms": 0.12749998131766915, + "p99_ms": 0.16630298574455088, + "mean_ms": 0.10040191657996426, + "knob": 4 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6309, + "qps_1t_runs": [ + 6997.952063673539, + 6339.397393306068, + 6091.877696571202 + ], + "qps_1t": 6339.397393306068, + "qps_1t_min": 6091.877696571202, + "qps_1t_max": 6997.952063673539, + "median_ms": 0.13280002167448401, + "p95_ms": 0.1836050010751933, + "p99_ms": 0.24820502672810119, + "mean_ms": 0.15493188323065019, + "knob": 8 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7282, + "qps_1t_runs": [ + 4917.7917231284255, + 4775.358762802328, + 4946.8277885201705 + ], + "qps_1t": 4917.7917231284255, + "qps_1t_min": 4775.358762802328, + "qps_1t_max": 4946.8277885201705, + "median_ms": 0.19829999655485153, + "p95_ms": 0.26099995302502066, + "p99_ms": 0.30341198202222613, + "mean_ms": 0.2049671334971208, + "knob": 16 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.81335, + "qps_1t_runs": [ + 3151.535437876544, + 3178.276355214525, + 3160.883939295367 + ], + "qps_1t": 3160.883939295367, + "qps_1t_min": 3151.535437876544, + "qps_1t_max": 3178.276355214525, + "median_ms": 0.31274999491870403, + "p95_ms": 0.39591004315298056, + "p99_ms": 0.45373502012807954, + "mean_ms": 0.31610293308040127, + "knob": 32 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.87955, + "qps_1t_runs": [ + 1876.346863706935, + 1876.4220349860614, + 1866.647627425756 + ], + "qps_1t": 1876.346863706935, + "qps_1t_min": 1866.647627425756, + "qps_1t_max": 1876.4220349860614, + "median_ms": 0.5209000082686543, + "p95_ms": 0.6576200015842916, + "p99_ms": 0.7712239929242065, + "mean_ms": 0.5338664667312211, + "knob": 64 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.93615, + "qps_1t_runs": [ + 1071.4241894386548, + 1035.3602444476332, + 1010.8230846182003 + ], + "qps_1t": 1035.3602444476332, + "qps_1t_min": 1010.8230846182003, + "qps_1t_max": 1071.4241894386548, + "median_ms": 0.9303499828092754, + "p95_ms": 1.1795050173532218, + "p99_ms": 1.7031010455684736, + "mean_ms": 0.9628257835671926, + "knob": 128 + }, + { + "nq": 1803, + "recall_queries": 1803, + "recall": 0.9728230726566833, + "qps_1t_runs": [ + 546.5862462760346, + 558.3354306974928, + 534.8742457788961 + ], + "qps_1t": 546.5862462760346, + "qps_1t_min": 534.8742457788961, + "qps_1t_max": 558.3354306974928, + "median_ms": 1.7321999766863883, + "p95_ms": 2.310219977516681, + "p99_ms": 3.5273239715024833, + "mean_ms": 1.8300579403861799, + "knob": 256 + }, + { + "nq": 1414, + "recall_queries": 1414, + "recall": 1.0, + "qps_1t_runs": [ + 435.42610792602176, + 440.0730658606763, + 446.3582924195241 + ], + "qps_1t": 440.0730658606763, + "qps_1t_min": 435.42610792602176, + "qps_1t_max": 446.3582924195241, + "median_ms": 2.0861999946646392, + "p95_ms": 3.1089449854334807, + "p99_ms": 5.4630870069377195, + "mean_ms": 2.2697678454782775, + "knob": 1265 + } + ], + "mutation": { + "ref_knob": 256, + "recall_before": 0.9728230726566833, + "insert_10k": { + "seconds": 0.13686079997569323, + "per_s": 73066.94102165135, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.14041749999159947, + "per_s": 71216.19456690409, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.11773510003695264, + "per_s": 84936.43779010145, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 238.40625, + "peak_mb": 461.671875 + }, + "first_query_after_ms": 78.54189997306094, + "after": { + "nq": 1711, + "recall_queries": 1711, + "recall": 0.9715955581531268, + "qps_1t_runs": [ + 532.9944762334069, + 531.9560717725213, + 527.8145138522884 + ], + "qps_1t": 531.9560717725213, + "qps_1t_min": 527.8145138522884, + "qps_1t_max": 532.9944762334069, + "median_ms": 1.7930999747477472, + "p95_ms": 2.330859994981438, + "p99_ms": 3.5363199957646447, + "mean_ms": 1.8835504969410322 + }, + "recall_after": 0.9715955581531268, + "index_stats_after": { + "enabled": true, + "mode": "ivf", + "active": true, + "nlist": 1265, + "nprobe": 256, + "trained_n": 100000, + "pending": 0, + "churn": 30000, + "build_seconds": 0.6770595000125468, + "index_bytes": 1366192, + "min_size": 50000 + } + }, + "saveload": { + "save_s": 4.42302980000386, + "disk_mb": 215.12163829803467, + "load_s": 3.408074300037697, + "first_query_after_load_ms": 2017.010500014294, + "time_to_ready_s": 5.425084800051991 + }, + "load_before_pct": 5.421875000000004, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 1.7763816000078805, + "mem": { + "rss_mb": 208.26171875, + "peak_mb": 310.90625 + }, + "subsets": { + "f10": [ + { + "knob": 2, + "nq": 2000, + "recall": 0.26685, + "qps_1t": 16284.563057732048, + "qps_1t_runs": [ + 15431.634240173593, + 16284.563057732048, + 16703.372267074268 + ], + "median_ms": 0.05570001667365432, + "p95_ms": 0.1017050148220733, + "p99_ms": 0.11810496798716497, + "mean_ms": 0.06202598349773325 + }, + { + "knob": 8, + "nq": 2000, + "recall": 0.47395, + "qps_1t": 12530.668764699758, + "qps_1t_runs": [ + 12530.668764699758, + 11967.649048568923, + 12809.616446447559 + ], + "median_ms": 0.07450004341080785, + "p95_ms": 0.1108050288166851, + "p99_ms": 0.17210397170856603, + "mean_ms": 0.08047638341668062 + }, + { + "knob": 32, + "nq": 2000, + "recall": 0.699, + "qps_1t": 7019.023997824779, + "qps_1t_runs": [ + 7019.023997824779, + 7024.553625664872, + 6576.78613658603 + ], + "median_ms": 0.1403999631293118, + "p95_ms": 0.17070496105588975, + "p99_ms": 0.20820995618123583, + "mean_ms": 0.14562589992419817 + }, + { + "knob": 64, + "nq": 2000, + "recall": 0.79565, + "qps_1t": 4564.469702070527, + "qps_1t_runs": [ + 4564.469702070527, + 4598.897913853672, + 4419.855325873309 + ], + "median_ms": 0.21739999647252262, + "p95_ms": 0.2603050001198426, + "p99_ms": 0.32729998929426074, + "mean_ms": 0.22092621674528345 + }, + { + "knob": 1265, + "nq": 844, + "recall": 1.0, + "qps_1t": 366.3635385348709, + "qps_1t_runs": [ + 382.9506904201276, + 366.3635385348709, + 366.235165687388 + ], + "median_ms": 2.8377499838825315, + "p95_ms": 3.57357996981591, + "p99_ms": 4.024586987798103, + "mean_ms": 2.6904391787381985 + } + ], + "f01": [ + { + "knob": 2, + "nq": 2000, + "recall": 0.3628, + "qps_1t": 6464.186786614411, + "qps_1t_runs": [ + 6481.589195167696, + 6074.568384296385, + 6464.186786614411 + ], + "median_ms": 0.17140002455562353, + "p95_ms": 0.2121049852576107, + "p99_ms": 0.26980401657056075, + "mean_ms": 0.15786746660402665 + }, + { + "knob": 8, + "nq": 2000, + "recall": 0.36295, + "qps_1t": 7849.228862823635, + "qps_1t_runs": [ + 7511.838698306427, + 8149.676974218535, + 7849.228862823635 + ], + "median_ms": 0.14084999565966427, + "p95_ms": 0.18860503914766016, + "p99_ms": 0.3476340271299712, + "mean_ms": 0.1277428331086412 + }, + { + "knob": 32, + "nq": 2000, + "recall": 0.4572, + "qps_1t": 8799.702935747748, + "qps_1t_runs": [ + 8682.390738034337, + 8799.702935747748, + 9079.621469865477 + ], + "median_ms": 0.10870001278817654, + "p95_ms": 0.13619998935610056, + "p99_ms": 0.2081040129996837, + "mean_ms": 0.11298420006642118 + }, + { + "knob": 64, + "nq": 2000, + "recall": 0.5944, + "qps_1t": 5803.310142003998, + "qps_1t_runs": [ + 5844.977769729354, + 5725.799470926135, + 5803.310142003998 + ], + "median_ms": 0.16659998800605536, + "p95_ms": 0.20750500843860212, + "p99_ms": 0.283606005832553, + "mean_ms": 0.1726835332713866 + }, + { + "knob": 1265, + "nq": 1296, + "recall": 1.0, + "qps_1t": 399.70808973227605, + "qps_1t_runs": [ + 392.5772843516169, + 418.222874125607, + 399.70808973227605 + ], + "median_ms": 2.395149989752099, + "p95_ms": 2.927895021275617, + "p99_ms": 3.7017560144886374, + "mean_ms": 2.480054835647818 + } + ] + } + } + }, + "isong_py_ivf|sift|100000": { + "mem_baseline": { + "rss_mb": 35.49609375, + "peak_mb": 35.49609375 + }, + "version": "isongraph-vector (python, repo HEAD)", + "build": { + "ingest_s": 1.1146168000414036, + "index_s": 0.9648579999920912, + "total_s": 2.079474800033495, + "first_query_ms": 3.1757999677211046, + "ready_s": 2.082650600001216, + "total_s_runs": [ + 2.079474800033495, + 2.0232003999408334, + 2.079858400102239 + ] + }, + "mem_after_build": { + "rss_mb": 226.88671875, + "peak_mb": 338.74609375 + }, + "mem_after_first_query": { + "rss_mb": 227.01953125, + "peak_mb": 338.74609375 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.28555, + "qps_1t_runs": [ + 15392.27607357927, + 15419.58226987032, + 15348.958528000368 + ], + "qps_1t": 15392.27607357927, + "qps_1t_min": 15348.958528000368, + "qps_1t_max": 15419.58226987032, + "median_ms": 0.06079999729990959, + "p95_ms": 0.08440000237897038, + "p99_ms": 0.12070198776200418, + "mean_ms": 0.06499041660572402, + "knob": 1 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.4253, + "qps_1t_runs": [ + 13796.071975524428, + 13991.003778776161, + 13604.211294384017 + ], + "qps_1t": 13796.071975524428, + "qps_1t_min": 13604.211294384017, + "qps_1t_max": 13991.003778776161, + "median_ms": 0.06940000457689166, + "p95_ms": 0.09139999747276306, + "p99_ms": 0.11860903003253062, + "mean_ms": 0.07248851673405927, + "knob": 2 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.5855, + "qps_1t_runs": [ + 10214.968975221796, + 10221.510350151724, + 10168.264424150182 + ], + "qps_1t": 10214.968975221796, + "qps_1t_min": 10168.264424150182, + "qps_1t_max": 10221.510350151724, + "median_ms": 0.09300000965595245, + "p95_ms": 0.13000000035390258, + "p99_ms": 0.162600958137773, + "mean_ms": 0.09802454985522975, + "knob": 4 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.74265, + "qps_1t_runs": [ + 7152.30843273225, + 7097.662790220524, + 6537.391899606096 + ], + "qps_1t": 7097.662790220524, + "qps_1t_min": 6537.391899606096, + "qps_1t_max": 7152.30843273225, + "median_ms": 0.13615001807920635, + "p95_ms": 0.18560501630418005, + "p99_ms": 0.2301119477488103, + "mean_ms": 0.1445575498995216, + "knob": 8 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.86895, + "qps_1t_runs": [ + 4347.524612886644, + 4513.194431744968, + 4293.412687223287 + ], + "qps_1t": 4347.524612886644, + "qps_1t_min": 4293.412687223287, + "qps_1t_max": 4513.194431744968, + "median_ms": 0.2115000388585031, + "p95_ms": 0.2977049967739731, + "p99_ms": 0.45842899824492694, + "mean_ms": 0.22816781633688757, + "knob": 16 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.94685, + "qps_1t_runs": [ + 2760.6593944371084, + 2610.4272750694454, + 2722.420756329228 + ], + "qps_1t": 2722.420756329228, + "qps_1t_min": 2610.4272750694454, + "qps_1t_max": 2760.6593944371084, + "median_ms": 0.33924999297596514, + "p95_ms": 0.5158350249985234, + "p99_ms": 0.9291180310538043, + "mean_ms": 0.3708771662786603, + "knob": 32 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.98685, + "qps_1t_runs": [ + 1663.2730355330273, + 1711.4863581881468, + 1687.841766628338 + ], + "qps_1t": 1687.841766628338, + "qps_1t_min": 1663.2730355330273, + "qps_1t_max": 1711.4863581881468, + "median_ms": 0.5703000060748309, + "p95_ms": 0.7775450387271122, + "p99_ms": 1.0802220174809922, + "mean_ms": 0.5926614166722478, + "knob": 64 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9976, + "qps_1t_runs": [ + 893.8901448134695, + 875.8977675382142, + 880.1468017721687 + ], + "qps_1t": 880.1468017721687, + "qps_1t_min": 875.8977675382142, + "qps_1t_max": 893.8901448134695, + "median_ms": 1.065549993654713, + "p95_ms": 1.5799250395502895, + "p99_ms": 2.2450089815538377, + "mean_ms": 1.1321885165913652, + "knob": 128 + }, + { + "nq": 1542, + "recall_queries": 1542, + "recall": 0.9998054474708171, + "qps_1t_runs": [ + 509.2492979560646, + 495.6974616079729, + 499.1166249830159 + ], + "qps_1t": 499.1166249830159, + "qps_1t_min": 495.6974616079729, + "qps_1t_max": 509.2492979560646, + "median_ms": 1.900399976875633, + "p95_ms": 2.5255000218749046, + "p99_ms": 3.3655000297585502, + "mean_ms": 1.994858020490782, + "knob": 256 + }, + { + "nq": 1068, + "recall_queries": 1068, + "recall": 1.0, + "qps_1t_runs": [ + 338.395514854715, + 337.1322796044335, + 354.2290672561122 + ], + "qps_1t": 338.395514854715, + "qps_1t_min": 337.1322796044335, + "qps_1t_max": 354.2290672561122, + "median_ms": 2.7924499881919473, + "p95_ms": 3.714610004681162, + "p99_ms": 4.719091969891444, + "mean_ms": 2.914782895979247, + "knob": 1265 + } + ], + "mutation": { + "ref_knob": 64, + "recall_before": 0.98685, + "insert_10k": { + "seconds": 0.15103710000403225, + "per_s": 66208.89834175199, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.1571733999880962, + "per_s": 63623.99744967894, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.13506170001346618, + "per_s": 74040.23493709143, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 273.59375, + "peak_mb": 522.51171875 + }, + "first_query_after_ms": 96.44670004490763, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.98565, + "qps_1t_runs": [ + 1594.6595487628774, + 1523.5822339139527, + 1296.6704087350088 + ], + "qps_1t": 1523.5822339139527, + "qps_1t_min": 1296.6704087350088, + "qps_1t_max": 1594.6595487628774, + "median_ms": 0.6274499755818397, + "p95_ms": 0.9622199926525355, + "p99_ms": 1.872863000025974, + "mean_ms": 0.6848823336185887 + }, + "recall_after": 0.98565, + "index_stats_after": { + "enabled": true, + "mode": "ivf", + "active": true, + "nlist": 1265, + "nprobe": 64, + "trained_n": 100000, + "pending": 0, + "churn": 30000, + "build_seconds": 0.7610118000302464, + "index_bytes": 1507872, + "min_size": 50000 + } + }, + "saveload": { + "save_s": 4.966320100007579, + "disk_mb": 222.15946578979492, + "load_s": 3.661656900017988, + "first_query_after_load_ms": 2379.858599975705, + "time_to_ready_s": 6.041515499993693 + }, + "load_before_pct": 8.184375000000001, + "load_contaminated": false + }, + "isong_py_ivf|text384|1000000": { + "mem_baseline": { + "rss_mb": 37.265625, + "peak_mb": 37.265625 + }, + "version": "isongraph-vector (python, repo HEAD)", + "build": { + "ingest_s": 20.75935379974544, + "index_s": 21.402648299990688, + "total_s": 42.16200209973613, + "first_query_ms": 78.90149997547269, + "ready_s": 42.2409035997116 + }, + "mem_after_build": { + "rss_mb": 4287.9765625, + "peak_mb": 4832.640625 + }, + "mem_after_first_query": { + "rss_mb": 4288.07421875, + "peak_mb": 4832.640625 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.2745, + "qps_1t_runs": [ + 3293.372653744326, + 3317.069190745804, + 3159.7134000636274 + ], + "qps_1t": 3293.372653744326, + "qps_1t_min": 3159.7134000636274, + "qps_1t_max": 3317.069190745804, + "median_ms": 0.28979999478906393, + "p95_ms": 0.42301001667510724, + "p99_ms": 0.5193099961616101, + "mean_ms": 0.30719848316706094, + "knob": 1 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.37635, + "qps_1t_runs": [ + 2355.801677791145, + 2438.017073734718, + 2389.2213614623442 + ], + "qps_1t": 2389.2213614623442, + "qps_1t_min": 2355.801677791145, + "qps_1t_max": 2438.017073734718, + "median_ms": 0.39329996798187494, + "p95_ms": 0.5690999707439914, + "p99_ms": 0.7120090076932686, + "mean_ms": 0.41773324977839366, + "knob": 2 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.494, + "qps_1t_runs": [ + 1688.896628288766, + 1741.5275347085244, + 1711.4826977091589 + ], + "qps_1t": 1711.4826977091589, + "qps_1t_min": 1688.896628288766, + "qps_1t_max": 1741.5275347085244, + "median_ms": 0.5573999951593578, + "p95_ms": 0.7863999926485121, + "p99_ms": 1.025820011273027, + "mean_ms": 0.583533266413724, + "knob": 4 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6007, + "qps_1t_runs": [ + 1077.1719936220263, + 1165.851736510476, + 1175.020958467421 + ], + "qps_1t": 1165.851736510476, + "qps_1t_min": 1077.1719936220263, + "qps_1t_max": 1175.020958467421, + "median_ms": 0.85704997763969, + "p95_ms": 1.131199998781085, + "p99_ms": 1.326803975389339, + "mean_ms": 0.8790491668332834, + "knob": 8 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.70055, + "qps_1t_runs": [ + 657.8049600662753, + 667.5915257092452, + 686.6548691486038 + ], + "qps_1t": 667.5915257092452, + "qps_1t_min": 657.8049600662753, + "qps_1t_max": 686.6548691486038, + "median_ms": 1.444549998268485, + "p95_ms": 1.8761549668852242, + "p99_ms": 2.2502150211948906, + "mean_ms": 1.4914883664265897, + "knob": 16 + }, + { + "nq": 1191, + "recall_queries": 1191, + "recall": 0.7812762384550798, + "qps_1t_runs": [ + 395.66668252476427, + 399.41108508059085, + 384.71149400228575 + ], + "qps_1t": 395.66668252476427, + "qps_1t_min": 384.71149400228575, + "qps_1t_max": 399.41108508059085, + "median_ms": 2.495800028555095, + "p95_ms": 3.082580014597624, + "p99_ms": 3.6152119957841924, + "mean_ms": 2.5434721521071397, + "knob": 32 + }, + { + "nq": 689, + "recall_queries": 1000, + "recall": 0.854, + "qps_1t_runs": [ + 217.04706556251222, + 213.03474582024813, + 219.32285258958467 + ], + "qps_1t": 217.04706556251222, + "qps_1t_min": 213.03474582024813, + "qps_1t_max": 219.32285258958467, + "median_ms": 4.465100006200373, + "p95_ms": 5.410000000847503, + "p99_ms": 6.545270028291263, + "mean_ms": 4.620284663808875, + "knob": 64 + }, + { + "nq": 404, + "recall_queries": 1000, + "recall": 0.9118999999999999, + "qps_1t_runs": [ + 124.9592336207618, + 124.59377876794932, + 124.61836394926401 + ], + "qps_1t": 124.61836394926401, + "qps_1t_min": 124.59377876794932, + "qps_1t_max": 124.9592336207618, + "median_ms": 7.763800007523969, + "p95_ms": 9.83091500820592, + "p99_ms": 15.338851984124673, + "mean_ms": 8.017730776064187, + "knob": 128 + }, + { + "nq": 211, + "recall_queries": 1000, + "recall": 0.9531999999999999, + "qps_1t_runs": [ + 65.01708763627119, + 67.3822125431702, + 65.83238830865464 + ], + "qps_1t": 65.83238830865464, + "qps_1t_min": 65.01708763627119, + "qps_1t_max": 67.3822125431702, + "median_ms": 14.624100003857166, + "p95_ms": 18.05828002979979, + "p99_ms": 23.72098397929221, + "mean_ms": 15.137125117543167, + "knob": 256 + }, + { + "nq": 37, + "recall_queries": 37, + "recall": 1.0, + "qps_1t_runs": [ + 12.055673098901895, + 11.484309793851029, + 11.445846791054567 + ], + "qps_1t": 11.484309793851029, + "qps_1t_min": 11.445846791054567, + "qps_1t_max": 12.055673098901895, + "median_ms": 83.045499981381, + "p95_ms": 99.29074998944998, + "p99_ms": 139.0585399756675, + "mean_ms": 85.7972531534337, + "knob": 4000 + } + ], + "mutation": { + "ref_knob": 256, + "recall_before": 0.9531999999999999, + "insert_10k": { + "seconds": 0.268599400005769, + "per_s": 37230.16507030626, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.22824040002888069, + "per_s": 43813.4528275215, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.16251339996233582, + "per_s": 61533.38741493073, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 4352.0078125, + "peak_mb": 4832.640625 + }, + "first_query_after_ms": 664.8137000156567, + "after": { + "nq": 212, + "recall_queries": 1000, + "recall": 0.9540000000000001, + "qps_1t_runs": [ + 66.50885036239598, + 68.82862351398286, + 65.08190988094326 + ], + "qps_1t": 66.50885036239598, + "qps_1t_min": 65.08190988094326, + "qps_1t_max": 68.82862351398286, + "median_ms": 14.517750008963048, + "p95_ms": 17.528975018649362, + "p99_ms": 23.239050002302967, + "mean_ms": 14.976561635548876 + }, + "recall_after": 0.9540000000000001, + "index_stats_after": { + "enabled": true, + "mode": "ivf", + "active": true, + "nlist": 4000, + "nprobe": 256, + "trained_n": 1000000, + "pending": 0, + "churn": 30000, + "build_seconds": 17.958885200030636, + "index_bytes": 14676072, + "min_size": 50000 + } + }, + "load_before_pct": 6.046874999999994, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 116.88233719981508, + "mem": { + "rss_mb": 4288.765625, + "peak_mb": 4833.14453125 + }, + "subsets": { + "f10": [ + { + "knob": 2, + "nq": 2000, + "recall": 0.30765, + "qps_1t": 2767.7964137097997, + "qps_1t_runs": [ + 2521.6521614777334, + 2824.470190279404, + 2767.7964137097997 + ], + "median_ms": 0.3793000360019505, + "p95_ms": 0.4487999831326306, + "p99_ms": 0.5065029574325309, + "mean_ms": 0.3706374502896021 + }, + { + "knob": 8, + "nq": 2000, + "recall": 0.5351, + "qps_1t": 1946.4960742901098, + "qps_1t_runs": [ + 1946.4960742901098, + 1937.1118584742012, + 2003.7612600927503 + ], + "median_ms": 0.5242999759502709, + "p95_ms": 0.6293050159001723, + "p99_ms": 0.6993030279409141, + "mean_ms": 0.5096791835967451 + }, + { + "knob": 32, + "nq": 2000, + "recall": 0.7416, + "qps_1t": 1040.4644423938623, + "qps_1t_runs": [ + 1026.1317259853643, + 1042.7509102610582, + 1040.4644423938623 + ], + "median_ms": 0.9917500137817115, + "p95_ms": 1.1970100313192233, + "p99_ms": 1.3191239745356147, + "mean_ms": 0.9648816004240265 + }, + { + "knob": 64, + "nq": 1909, + "recall": 0.8242011524358303, + "qps_1t": 649.2092520951015, + "qps_1t_runs": [ + 659.6261847780818, + 633.7770360220844, + 649.2092520951015 + ], + "median_ms": 1.553500012960285, + "p95_ms": 1.8928999779745936, + "p99_ms": 2.099613980390131, + "mean_ms": 1.544729177592059 + }, + { + "knob": 4000, + "nq": 30, + "recall": 1.0, + "qps_1t": 8.642143777309547, + "qps_1t_runs": [ + 8.386630369495343, + 8.642143777309547, + 8.651004464746551 + ], + "median_ms": 116.28655000822619, + "p95_ms": 129.68221000337508, + "p99_ms": 135.23232901701704, + "mean_ms": 116.84764777827594 + } + ], + "f01": [ + { + "knob": 2, + "nq": 2000, + "recall": 0.33035, + "qps_1t": 1627.7126539663636, + "qps_1t_runs": [ + 1595.1886566576757, + 1627.7126539663636, + 1672.4455245077775 + ], + "median_ms": 0.680800003465265, + "p95_ms": 0.7686050317715853, + "p99_ms": 1.1919149948516863, + "mean_ms": 0.6130569831390554 + }, + { + "knob": 8, + "nq": 2000, + "recall": 0.3679, + "qps_1t": 2595.411206890395, + "qps_1t_runs": [ + 2543.8188644426073, + 2805.059317766705, + 2595.411206890395 + ], + "median_ms": 0.3963999915868044, + "p95_ms": 0.4947999870637432, + "p99_ms": 0.7482999935746193, + "mean_ms": 0.3783012837035737 + }, + { + "knob": 32, + "nq": 2000, + "recall": 0.60955, + "qps_1t": 1654.3760308882959, + "qps_1t_runs": [ + 1671.0643516605112, + 1648.7714509287766, + 1654.3760308882959 + ], + "median_ms": 0.6339500250760466, + "p95_ms": 0.8098299964331092, + "p99_ms": 1.0052180284401406, + "mean_ms": 0.6031302333188554 + }, + { + "knob": 64, + "nq": 2000, + "recall": 0.72475, + "qps_1t": 1119.139256835492, + "qps_1t_runs": [ + 1146.3782935707418, + 1068.1510258643498, + 1119.139256835492 + ], + "median_ms": 0.9490499796811491, + "p95_ms": 1.1690999963320792, + "p99_ms": 1.3412139686988669, + "mean_ms": 0.9006844832232067 + }, + { + "knob": 4000, + "nq": 30, + "recall": 1.0, + "qps_1t": 9.524133344978656, + "qps_1t_runs": [ + 9.325284287417265, + 9.529939846115665, + 9.524133344978656 + ], + "median_ms": 105.76075001154095, + "p95_ms": 115.39345996570773, + "p99_ms": 116.6629690042464, + "mean_ms": 105.72140777496517 + } + ] + } + } + }, + "isong_py_ivf|text384|100000": { + "mem_baseline": { + "rss_mb": 37.4296875, + "peak_mb": 37.4296875 + }, + "version": "isongraph-vector (python, repo HEAD)", + "build": { + "ingest_s": 1.9661553999758326, + "index_s": 1.598260400001891, + "total_s": 3.5644157999777235, + "first_query_ms": 7.300400000531226, + "ready_s": 3.5717161999782547, + "total_s_runs": [ + 3.5644157999777235, + 3.5697892999160103, + 3.4769746000529267 + ] + }, + "mem_after_build": { + "rss_mb": 518.12890625, + "peak_mb": 730.7890625 + }, + "mem_after_first_query": { + "rss_mb": 518.26171875, + "peak_mb": 730.7890625 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.2833, + "qps_1t_runs": [ + 8545.117014877103, + 8513.975024681084, + 8572.634849849783 + ], + "qps_1t": 8545.117014877103, + "qps_1t_min": 8513.975024681084, + "qps_1t_max": 8572.634849849783, + "median_ms": 0.11399999493733048, + "p95_ms": 0.1388999546179548, + "p99_ms": 0.17499999434221536, + "mean_ms": 0.11704336649078566, + "knob": 1 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.40835, + "qps_1t_runs": [ + 7155.804035074119, + 7225.869465126067, + 6793.505962437482 + ], + "qps_1t": 7155.804035074119, + "qps_1t_min": 6793.505962437482, + "qps_1t_max": 7225.869465126067, + "median_ms": 0.13539998326450586, + "p95_ms": 0.1750050490954891, + "p99_ms": 0.21620101644657555, + "mean_ms": 0.141779250028776, + "knob": 2 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.53055, + "qps_1t_runs": [ + 5451.192477884488, + 5400.288278604827, + 5018.829396046299 + ], + "qps_1t": 5400.288278604827, + "qps_1t_min": 5018.829396046299, + "qps_1t_max": 5451.192477884488, + "median_ms": 0.18009997438639402, + "p95_ms": 0.2297000028192997, + "p99_ms": 0.32366996165365136, + "mean_ms": 0.18929034980828874, + "knob": 4 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.65005, + "qps_1t_runs": [ + 3854.1423096422295, + 4005.6471591323034, + 3877.4155091854054 + ], + "qps_1t": 3877.4155091854054, + "qps_1t_min": 3854.1423096422295, + "qps_1t_max": 4005.6471591323034, + "median_ms": 0.24500000290572643, + "p95_ms": 0.3024050296517089, + "p99_ms": 0.3510289953555918, + "mean_ms": 0.25567079975735396, + "knob": 8 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.75425, + "qps_1t_runs": [ + 2640.2807580120398, + 2668.5303656896854, + 2574.986832836467 + ], + "qps_1t": 2640.2807580120398, + "qps_1t_min": 2574.986832836467, + "qps_1t_max": 2668.5303656896854, + "median_ms": 0.36914998781867325, + "p95_ms": 0.4582199966534973, + "p99_ms": 0.5930029758019374, + "mean_ms": 0.38061240019548376, + "knob": 16 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.84285, + "qps_1t_runs": [ + 1578.517514672899, + 1603.0987263170168, + 1580.8420906025253 + ], + "qps_1t": 1580.8420906025253, + "qps_1t_min": 1578.517514672899, + "qps_1t_max": 1603.0987263170168, + "median_ms": 0.5910000181756914, + "p95_ms": 0.740724994102493, + "p99_ms": 1.075628983089701, + "mean_ms": 0.6299573164239215, + "knob": 32 + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.90905, + "qps_1t_runs": [ + 947.1340107756721, + 937.1362879959365, + 927.44732622626 + ], + "qps_1t": 937.1362879959365, + "qps_1t_min": 927.44732622626, + "qps_1t_max": 947.1340107756721, + "median_ms": 1.0270499915350229, + "p95_ms": 1.264340023044497, + "p99_ms": 1.6760009556310274, + "mean_ms": 1.0670419333231014, + "knob": 64 + }, + { + "nq": 1546, + "recall_queries": 1546, + "recall": 0.9540103492884864, + "qps_1t_runs": [ + 506.33548171855824, + 500.7509319772026, + 499.6443347609951 + ], + "qps_1t": 500.7509319772026, + "qps_1t_min": 499.6443347609951, + "qps_1t_max": 506.33548171855824, + "median_ms": 1.9275000086054206, + "p95_ms": 2.336295036366209, + "p99_ms": 2.831875989795663, + "mean_ms": 1.991133204036412, + "knob": 128 + }, + { + "nq": 814, + "recall_queries": 1000, + "recall": 0.9809000000000001, + "qps_1t_runs": [ + 270.967193538073, + 267.5820998714532, + 267.91264258534295 + ], + "qps_1t": 267.91264258534295, + "qps_1t_min": 267.5820998714532, + "qps_1t_max": 270.967193538073, + "median_ms": 3.5941500100307167, + "p95_ms": 4.1410549863940105, + "p99_ms": 5.3537270141532955, + "mean_ms": 3.720071457291516, + "knob": 256 + }, + { + "nq": 422, + "recall_queries": 1000, + "recall": 0.9999, + "qps_1t_runs": [ + 132.1166454887518, + 131.20162407022406, + 129.7549899662039 + ], + "qps_1t": 131.20162407022406, + "qps_1t_min": 129.7549899662039, + "qps_1t_max": 132.1166454887518, + "median_ms": 7.360049989074469, + "p95_ms": 8.474500005831942, + "p99_ms": 14.194564969511685, + "mean_ms": 7.63258617693164, + "knob": 1265 + } + ], + "mutation": { + "ref_knob": 128, + "recall_before": 0.9540103492884864, + "insert_10k": { + "seconds": 0.21102709998376667, + "per_s": 47387.278699130366, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.2037728000432253, + "per_s": 49074.2630904554, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.125521200010553, + "per_s": 79667.8170632472, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 626.26171875, + "peak_mb": 1189.75390625 + }, + "first_query_after_ms": 202.5108000379987, + "after": { + "nq": 1348, + "recall_queries": 1348, + "recall": 0.9481454005934719, + "qps_1t_runs": [ + 480.78613822988945, + 485.09851486509643, + 491.1899543299876 + ], + "qps_1t": 485.09851486509643, + "qps_1t_min": 480.78613822988945, + "qps_1t_max": 491.1899543299876, + "median_ms": 1.9911999988835305, + "p95_ms": 2.4066649901214987, + "p99_ms": 2.9588390036951755, + "mean_ms": 2.0590786843932105 + }, + "recall_after": 0.9481454005934719, + "index_stats_after": { + "enabled": true, + "mode": "ivf", + "active": true, + "nlist": 1265, + "nprobe": 128, + "trained_n": 100000, + "pending": 0, + "churn": 30000, + "build_seconds": 1.2876625999924727, + "index_bytes": 2803232, + "min_size": 50000 + } + }, + "saveload": { + "save_s": 15.985765799996443, + "disk_mb": 807.4883146286011, + "load_s": 10.858780499955174, + "first_query_after_load_ms": 5128.686800017022, + "time_to_ready_s": 15.987467299972195 + }, + "load_before_pct": 6.862500000000005, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 3.6155435000546277, + "mem": { + "rss_mb": 516.1875, + "peak_mb": 728.88671875 + }, + "subsets": { + "f10": [ + { + "knob": 2, + "nq": 2000, + "recall": 0.3022, + "qps_1t": 8861.818984661579, + "qps_1t_runs": [ + 8900.08316833595, + 8637.521665845838, + 8861.818984661579 + ], + "median_ms": 0.10250002378597856, + "p95_ms": 0.18759997619781643, + "p99_ms": 0.21991198649629976, + "mean_ms": 0.11365870014803174 + }, + { + "knob": 8, + "nq": 2000, + "recall": 0.53195, + "qps_1t": 7773.371979587393, + "qps_1t_runs": [ + 7606.6509746269, + 7820.546622743308, + 7773.371979587393 + ], + "median_ms": 0.12510002125054598, + "p95_ms": 0.1470999588491395, + "p99_ms": 0.19650604052003487, + "mean_ms": 0.12932549992304607 + }, + { + "knob": 32, + "nq": 2000, + "recall": 0.76625, + "qps_1t": 4816.732941998437, + "qps_1t_runs": [ + 4925.221581385811, + 4816.732941998437, + 4764.162068792711 + ], + "median_ms": 0.2001000102609396, + "p95_ms": 0.2452050044666976, + "p99_ms": 0.3167250246042391, + "mean_ms": 0.20684888349690786 + }, + { + "knob": 64, + "nq": 2000, + "recall": 0.86115, + "qps_1t": 3413.984740720125, + "qps_1t_runs": [ + 3503.986221983041, + 3413.2453730892416, + 3413.984740720125 + ], + "median_ms": 0.2775000175461173, + "p95_ms": 0.3369000041857362, + "p99_ms": 0.4359090013895186, + "mean_ms": 0.29042613327813643 + }, + { + "knob": 1265, + "nq": 348, + "recall": 1.0, + "qps_1t": 121.55768630020441, + "qps_1t_runs": [ + 123.99780021377535, + 121.55768630020441, + 121.52248393576335 + ], + "median_ms": 7.936599984532222, + "p95_ms": 9.63313004467636, + "p99_ms": 13.48838001489628, + "mean_ms": 8.173378641289833 + } + ], + "f01": [ + { + "knob": 2, + "nq": 2000, + "recall": 0.4917, + "qps_1t": 3663.1022882635857, + "qps_1t_runs": [ + 3663.587424894048, + 3566.1220518706914, + 3663.1022882635857 + ], + "median_ms": 0.2905000001192093, + "p95_ms": 0.34620501392055303, + "p99_ms": 0.5144060170277954, + "mean_ms": 0.27545526656710234 + }, + { + "knob": 8, + "nq": 2000, + "recall": 0.49185, + "qps_1t": 4826.572783158953, + "qps_1t_runs": [ + 4826.572783158953, + 4862.180286632508, + 4607.146321359404 + ], + "median_ms": 0.22950000129640102, + "p95_ms": 0.27199997566640377, + "p99_ms": 0.41464995592832676, + "mean_ms": 0.20996983356114166 + }, + { + "knob": 32, + "nq": 2000, + "recall": 0.5694, + "qps_1t": 6221.664220722495, + "qps_1t_runs": [ + 5961.758304437987, + 6221.664220722495, + 6352.509660554177 + ], + "median_ms": 0.15339997480623424, + "p95_ms": 0.18230499117635193, + "p99_ms": 0.2765170094789941, + "mean_ms": 0.16196084981008121 + }, + { + "knob": 64, + "nq": 2000, + "recall": 0.7223, + "qps_1t": 4839.23692163944, + "qps_1t_runs": [ + 4810.332213747565, + 4839.23692163944, + 4856.693546769946 + ], + "median_ms": 0.20220002625137568, + "p95_ms": 0.2329100243514404, + "p99_ms": 0.28094104316551327, + "mean_ms": 0.20681046652801646 + }, + { + "knob": 1265, + "nq": 422, + "recall": 1.0, + "qps_1t": 129.51690195616382, + "qps_1t_runs": [ + 129.7527478603694, + 128.89391337834743, + 129.51690195616382 + ], + "median_ms": 7.393149979179725, + "p95_ms": 9.319625023636036, + "p99_ms": 13.083250002819108, + "mean_ms": 7.728761373892188 + } + ] + } + } + }, + "js_exact|glove|100000": { + "version": "isongraph-vector-js (repo HEAD, Node v24.18.0)", + "build": { + "ingest_s": 0.028693200030829757, + "index_s": 0, + "total_s": 0.028693200030829757, + "first_query_ms": 24.664999975357205, + "ready_s": 0.05335820000618696 + }, + "mem_baseline": { + "rss_mb": 52.39453125, + "peak_mb": 52.39453125, + "heap_mb": 10.311317443847656 + }, + "mem_after_build": { + "rss_mb": 360.25, + "peak_mb": 360.25, + "heap_mb": 204.57239532470703 + }, + "mem_after_first_query": { + "rss_mb": 360.25, + "peak_mb": 360.25, + "heap_mb": 204.5730438232422 + }, + "sweep": [ + { + "nq": 167, + "recall": 1, + "qps_1t": 48.0650022974727, + "qps_1t_min": 47.828708381835746, + "qps_1t_max": 48.86487053063795, + "qps_1t_runs": [ + 47.828708381835746, + 48.0650022974727, + 48.86487053063795 + ], + "median_ms": 20.08340001339093, + "p95_ms": 24.64339998550713, + "p99_ms": 27.087900030892342, + "mean_ms": 20.72590099765683, + "knob": null + } + ], + "load_before_pct": 3.3781249999999976, + "load_contaminated": false + }, + "js_ivf|glove|100000": { + "version": "isongraph-vector-js (repo HEAD, Node v24.18.0)", + "build": { + "ingest_s": 0.02707650000229478, + "index_s": 20.63768899999559, + "total_s": 20.664765499997884, + "first_query_ms": 0, + "ready_s": 20.664765499997884 + }, + "mem_baseline": { + "rss_mb": 52.39453125, + "peak_mb": 52.39453125, + "heap_mb": 10.311317443847656 + }, + "mem_after_build": { + "rss_mb": 694.3125, + "peak_mb": 697.76171875, + "heap_mb": 215.28913116455078 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.2795, + "qps_1t": 6774.909387662348, + "qps_1t_min": 6500.906542238136, + "qps_1t_max": 6777.76785109088, + "qps_1t_runs": [ + 6500.906542238136, + 6774.909387662348, + 6777.76785109088 + ], + "median_ms": 0.14459999511018395, + "p95_ms": 0.1841000048443675, + "p99_ms": 0.23309997050091624, + "mean_ms": 0.14965645001696734, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.3983, + "qps_1t": 5710.070046012471, + "qps_1t_min": 5550.393691614603, + "qps_1t_max": 5862.431026587508, + "qps_1t_runs": [ + 5550.393691614603, + 5862.431026587508, + 5710.070046012471 + ], + "median_ms": 0.1710999640636146, + "p95_ms": 0.2237000153400004, + "p99_ms": 0.25049998657777905, + "mean_ms": 0.17529143330951533, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.51655, + "qps_1t": 4523.073553428299, + "qps_1t_min": 4348.697717073671, + "qps_1t_max": 4603.7071359589545, + "qps_1t_runs": [ + 4603.7071359589545, + 4348.697717073671, + 4523.073553428299 + ], + "median_ms": 0.21810003090649843, + "p95_ms": 0.28979999478906393, + "p99_ms": 0.3281999961473048, + "mean_ms": 0.22275291668483987, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.62575, + "qps_1t": 3532.9784113280625, + "qps_1t_min": 3512.2102015833266, + "qps_1t_max": 3552.699505303434, + "qps_1t_runs": [ + 3512.2102015833266, + 3532.9784113280625, + 3552.699505303434 + ], + "median_ms": 0.2793999738059938, + "p95_ms": 0.3706999705173075, + "p99_ms": 0.4146000137552619, + "mean_ms": 0.2830814664484933, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.72415, + "qps_1t": 2300.654341422209, + "qps_1t_min": 2292.8893274212364, + "qps_1t_max": 2327.541402666945, + "qps_1t_runs": [ + 2300.654341422209, + 2327.541402666945, + 2292.8893274212364 + ], + "median_ms": 0.42610004311427474, + "p95_ms": 0.5797999911010265, + "p99_ms": 0.6764999707229435, + "mean_ms": 0.43347593313471106, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.8079, + "qps_1t": 1400.9757097129889, + "qps_1t_min": 1350.2549073477542, + "qps_1t_max": 1412.358661277198, + "qps_1t_runs": [ + 1412.358661277198, + 1350.2549073477542, + 1400.9757097129889 + ], + "median_ms": 0.7085999823175371, + "p95_ms": 0.9676999761722982, + "p99_ms": 1.130899996496737, + "mean_ms": 0.720808199994887, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.87635, + "qps_1t": 778.8949381767982, + "qps_1t_min": 764.2925575667001, + "qps_1t_max": 799.1614241173585, + "qps_1t_runs": [ + 799.1614241173585, + 778.8949381767982, + 764.2925575667001 + ], + "median_ms": 1.2710000155493617, + "p95_ms": 1.6903000068850815, + "p99_ms": 1.9036000012420118, + "mean_ms": 1.2811937831381024, + "knob": 64 + }, + { + "nq": 1366, + "recall": 0.9320644216691069, + "qps_1t": 432.3019019324809, + "qps_1t_min": 431.0375566428076, + "qps_1t_max": 446.930620647938, + "qps_1t_runs": [ + 446.930620647938, + 431.0375566428076, + 432.3019019324809 + ], + "median_ms": 2.28159996913746, + "p95_ms": 2.958500001113862, + "p99_ms": 3.2582999556325376, + "mean_ms": 2.290221815281118, + "knob": 128 + }, + { + "nq": 651, + "recall": 0.9745007680491552, + "qps_1t": 229.55479554178177, + "qps_1t_min": 221.3337592742448, + "qps_1t_max": 232.24724033020672, + "qps_1t_runs": [ + 229.55479554178177, + 221.3337592742448, + 232.24724033020672 + ], + "median_ms": 4.37069998588413, + "p95_ms": 5.45820000115782, + "p99_ms": 6.29390001995489, + "mean_ms": 4.393359396143645, + "knob": 256 + }, + { + "nq": 154, + "recall": 1, + "qps_1t": 50.813091960134905, + "qps_1t_min": 49.47061145795657, + "qps_1t_max": 51.25979091027489, + "qps_1t_runs": [ + 51.25979091027489, + 50.813091960134905, + 49.47061145795657 + ], + "median_ms": 19.09049996174872, + "p95_ms": 23.098700039554387, + "p99_ms": 24.940599978435785, + "mean_ms": 19.800819047107826, + "knob": 1265 + } + ], + "mutation": { + "ref_knob": 256, + "recall_before": 0.9745007680491552, + "insert_10k": { + "seconds": 0.01814970001578331, + "per_s": 550973.2938452876 + }, + "update_10k": { + "seconds": 0.008544000040274113, + "per_s": 1170411.97950172 + }, + "delete_10k": { + "seconds": 0.0057483999989926815, + "per_s": 1739614.5017313245 + }, + "first_query_after_ms": 1173.6763999797404, + "recall_after": 0.9744485294117647, + "after": { + "nq": 544, + "recall": 0.9744485294117647, + "qps_1t": 224.84353221535994, + "qps_1t_min": 222.28020758946403, + "qps_1t_max": 226.69062031863888, + "qps_1t_runs": [ + 224.84353221535994, + 226.69062031863888, + 222.28020758946403 + ], + "median_ms": 4.454899986740202, + "p95_ms": 5.3301999578252435, + "p99_ms": 6.0882000252604485, + "mean_ms": 4.452553982927758 + }, + "mem_after": { + "rss_mb": 680.4375, + "peak_mb": 710.4453125, + "heap_mb": 161.70631408691406 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 100000 + } + }, + "load_before_pct": 3.3781249999999976, + "load_contaminated": false + }, + "js_exact|sift|100000": { + "version": "isongraph-vector-js (repo HEAD, Node v24.18.0)", + "build": { + "ingest_s": 0.034534100035671145, + "index_s": 0, + "total_s": 0.034534100035671145, + "first_query_ms": 28.81559997331351, + "ready_s": 0.06334970000898466 + }, + "mem_baseline": { + "rss_mb": 52.86328125, + "peak_mb": 52.86328125, + "heap_mb": 9.740829467773438 + }, + "mem_after_build": { + "rss_mb": 369.75390625, + "peak_mb": 369.75390625, + "heap_mb": 207.31876373291016 + }, + "mem_after_first_query": { + "rss_mb": 369.75390625, + "peak_mb": 369.75390625, + "heap_mb": 207.3194122314453 + }, + "sweep": [ + { + "nq": 106, + "recall": 1, + "qps_1t": 29.015488331823676, + "qps_1t_min": 27.56688246071011, + "qps_1t_max": 32.53022648455043, + "qps_1t_runs": [ + 32.53022648455043, + 29.015488331823676, + 27.56688246071011 + ], + "median_ms": 33.672999998088926, + "p95_ms": 41.62020003423095, + "p99_ms": 48.66359999869019, + "mean_ms": 33.82680125742471, + "knob": null + } + ], + "load_before_pct": 11.05312499999999, + "load_contaminated": false + }, + "js_ivf|sift|100000": { + "version": "isongraph-vector-js (repo HEAD, Node v24.18.0)", + "build": { + "ingest_s": 0.04355299996677786, + "index_s": 35.42172629997367, + "total_s": 35.46527929994045, + "first_query_ms": 0, + "ready_s": 35.46527929994045 + }, + "mem_baseline": { + "rss_mb": 52.86328125, + "peak_mb": 52.86328125, + "heap_mb": 9.740829467773438 + }, + "mem_after_build": { + "rss_mb": 703.66796875, + "peak_mb": 703.66796875, + "heap_mb": 213.23184967041016 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.2886, + "qps_1t": 3499.20865791667, + "qps_1t_min": 3416.21702191343, + "qps_1t_max": 3524.8470916818233, + "qps_1t_runs": [ + 3524.8470916818233, + 3416.21702191343, + 3499.20865791667 + ], + "median_ms": 0.28899998869746923, + "p95_ms": 0.3502999898046255, + "p99_ms": 0.424400030169636, + "mean_ms": 0.2874001997697633, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.4338, + "qps_1t": 3126.918173166386, + "qps_1t_min": 3107.9813753619237, + "qps_1t_max": 3163.5022934552576, + "qps_1t_runs": [ + 3107.9813753619237, + 3163.5022934552576, + 3126.918173166386 + ], + "median_ms": 0.3170999698340893, + "p95_ms": 0.3800999838858843, + "p99_ms": 0.42619998566806316, + "mean_ms": 0.31922043316687143, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.5894, + "qps_1t": 2439.1082697930437, + "qps_1t_min": 2407.44255962951, + "qps_1t_max": 2486.2732852665545, + "qps_1t_runs": [ + 2486.2732852665545, + 2407.44255962951, + 2439.1082697930437 + ], + "median_ms": 0.3988000098615885, + "p95_ms": 0.5239000311121345, + "p99_ms": 0.6170999840833247, + "mean_ms": 0.4091909501585178, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.7425, + "qps_1t": 1899.5105716300106, + "qps_1t_min": 1884.0905609592085, + "qps_1t_max": 1909.3722194016998, + "qps_1t_runs": [ + 1909.3722194016998, + 1899.5105716300106, + 1884.0905609592085 + ], + "median_ms": 0.5158000276423991, + "p95_ms": 0.6834000232629478, + "p99_ms": 0.7966000121086836, + "mean_ms": 0.5269812667393126, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.8685, + "qps_1t": 1318.562966176771, + "qps_1t_min": 1308.1881259520942, + "qps_1t_max": 1363.2442757955162, + "qps_1t_runs": [ + 1308.1881259520942, + 1318.562966176771, + 1363.2442757955162 + ], + "median_ms": 0.7391000399366021, + "p95_ms": 1.0248000035062432, + "p99_ms": 1.1904999846592546, + "mean_ms": 0.7521205668454058, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.9486, + "qps_1t": 809.7634476612682, + "qps_1t_min": 763.2318549516632, + "qps_1t_max": 820.0141509468529, + "qps_1t_runs": [ + 820.0141509468529, + 763.2318549516632, + 809.7634476612682 + ], + "median_ms": 1.2365999864414334, + "p95_ms": 1.7020999803207815, + "p99_ms": 1.9302999717183411, + "mean_ms": 1.2548791832232382, + "knob": 32 + }, + { + "nq": 1328, + "recall": 0.9876506024096385, + "qps_1t": 480.1718147239224, + "qps_1t_min": 463.69302783714807, + "qps_1t_max": 481.6145634518156, + "qps_1t_runs": [ + 480.1718147239224, + 481.6145634518156, + 463.69302783714807 + ], + "median_ms": 2.1003999863751233, + "p95_ms": 2.732599969021976, + "p99_ms": 3.1218999647535384, + "mean_ms": 2.105178740222697, + "knob": 64 + }, + { + "nq": 761, + "recall": 0.9984231274638633, + "qps_1t": 248.22625874677325, + "qps_1t_min": 244.3492156576467, + "qps_1t_max": 255.56793874842808, + "qps_1t_runs": [ + 255.56793874842808, + 244.3492156576467, + 248.22625874677325 + ], + "median_ms": 3.9916999521665275, + "p95_ms": 4.9854000099003315, + "p99_ms": 5.646600038744509, + "mean_ms": 4.011313315671455, + "knob": 128 + }, + { + "nq": 424, + "recall": 1, + "qps_1t": 142.33386449958948, + "qps_1t_min": 140.8073427916259, + "qps_1t_max": 145.45854247428872, + "qps_1t_runs": [ + 140.8073427916259, + 145.45854247428872, + 142.33386449958948 + ], + "median_ms": 7.057500013615936, + "p95_ms": 8.092699979897588, + "p99_ms": 8.661899948492646, + "mean_ms": 7.000816116053113, + "knob": 256 + }, + { + "nq": 79, + "recall": 1, + "qps_1t": 27.81044429292253, + "qps_1t_min": 26.752337358303926, + "qps_1t_max": 27.959403795720092, + "qps_1t_runs": [ + 26.752337358303926, + 27.959403795720092, + 27.81044429292253 + ], + "median_ms": 35.214200033806264, + "p95_ms": 42.42409998551011, + "p99_ms": 45.87979998905212, + "mean_ms": 36.36792236285771, + "knob": 1265 + } + ], + "mutation": { + "ref_knob": 64, + "recall_before": 0.9876506024096385, + "insert_10k": { + "seconds": 0.04443340003490448, + "per_s": 225055.92622091805 + }, + "update_10k": { + "seconds": 0.011832899996079504, + "per_s": 845101.370189321 + }, + "delete_10k": { + "seconds": 0.007865299994591624, + "per_s": 1271407.3216376044 + }, + "first_query_after_ms": 2042.7271999651566, + "recall_after": 0.9873373807458803, + "after": { + "nq": 1153, + "recall": 0.9873373807458803, + "qps_1t": 447.9219278040597, + "qps_1t_min": 441.951051784361, + "qps_1t_max": 457.85640285476217, + "qps_1t_runs": [ + 441.951051784361, + 457.85640285476217, + 447.9219278040597 + ], + "median_ms": 2.2129000280983746, + "p95_ms": 2.8608000138774514, + "p99_ms": 3.22979997145012, + "mean_ms": 2.2264389418742434 + }, + "mem_after": { + "rss_mb": 745.4140625, + "peak_mb": 745.4140625, + "heap_mb": 200.06658935546875 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 100000 + } + }, + "load_before_pct": 11.05312499999999, + "load_contaminated": false + }, + "js_exact|text384|1000000": { + "version": "isongraph-vector-js (repo HEAD, Node v24.18.0)", + "build": { + "ingest_s": 0.4697339002741501, + "index_s": 0, + "total_s": 0.4697339002741501, + "first_query_ms": 557.7046000398695, + "ready_s": 1.0274385003140196 + }, + "mem_baseline": { + "rss_mb": 66.1796875, + "peak_mb": 66.1796875, + "heap_mb": 16.08751678466797 + }, + "mem_after_build": { + "rss_mb": 3686.65234375, + "peak_mb": 3686.65234375, + "heap_mb": 3380.0526733398438 + }, + "mem_after_first_query": { + "rss_mb": 3686.65234375, + "peak_mb": 3686.65234375, + "heap_mb": 3380.053321838379 + }, + "sweep": [ + { + "nq": 30, + "recall": 1, + "qps_1t": 1.668134513829699, + "qps_1t_min": 1.668134513829699, + "qps_1t_max": 1.668134513829699, + "qps_1t_runs": [ + 1.668134513829699 + ], + "median_ms": 544.5717999828048, + "p95_ms": 838.1194999674335, + "p99_ms": 871.8156000250019, + "mean_ms": 599.4720400000612, + "knob": null + } + ], + "load_before_pct": 2.968749999999999, + "load_contaminated": false + }, + "js_ivf|text384|1000000": { + "version": "isongraph-vector-js (repo HEAD, Node v24.18.0)", + "build": { + "ingest_s": 0.690798299969174, + "index_s": 521.2741673999699, + "total_s": 521.9649656999391, + "first_query_ms": 0, + "ready_s": 521.9649656999391 + }, + "mem_baseline": { + "rss_mb": 66.1796875, + "peak_mb": 66.1796875, + "heap_mb": 16.08751678466797 + }, + "mem_after_build": { + "rss_mb": 7068.578125, + "peak_mb": 7068.578125, + "heap_mb": 3351.2668838500977 + }, + "sweep": [ + { + "nq": 1907, + "recall": 0.2670162558993183, + "qps_1t": 666.1068387526135, + "qps_1t_min": 659.6800313784976, + "qps_1t_max": 668.1428406805679, + "qps_1t_runs": [ + 659.6800313784976, + 668.1428406805679, + 666.1068387526135 + ], + "median_ms": 1.4743999927304685, + "p95_ms": 1.7371000139974058, + "p99_ms": 1.9876000005751848, + "mean_ms": 1.5046110121358927, + "knob": 1 + }, + { + "nq": 1634, + "recall": 0.3726438188494492, + "qps_1t": 593.6938882283083, + "qps_1t_min": 586.5755120760934, + "qps_1t_max": 594.3714982716806, + "qps_1t_runs": [ + 594.3714982716806, + 586.5755120760934, + 593.6938882283083 + ], + "median_ms": 1.659700006712228, + "p95_ms": 2.0185000030323863, + "p99_ms": 2.2839000448584557, + "mean_ms": 1.6905431661824952, + "knob": 2 + }, + { + "nq": 1452, + "recall": 0.48326446280991736, + "qps_1t": 492.7275015615987, + "qps_1t_min": 487.9994975615684, + "qps_1t_max": 493.328278641302, + "qps_1t_runs": [ + 492.7275015615987, + 487.9994975615684, + 493.328278641302 + ], + "median_ms": 2.0038000075146556, + "p95_ms": 2.546000003349036, + "p99_ms": 2.801199967507273, + "mean_ms": 2.0352498621597954, + "knob": 4 + }, + { + "nq": 1071, + "recall": 0.592250233426704, + "qps_1t": 368.0081163474449, + "qps_1t_min": 365.7176232817451, + "qps_1t_max": 368.12085748602084, + "qps_1t_runs": [ + 368.12085748602084, + 365.7176232817451, + 368.0081163474449 + ], + "median_ms": 2.697399992030114, + "p95_ms": 3.3989999792538583, + "p99_ms": 3.6818000371567905, + "mean_ms": 2.7227268594379748, + "knob": 8 + }, + { + "nq": 647, + "recall": 0.6891808346213292, + "qps_1t": 245.12318526070712, + "qps_1t_min": 243.61615537122444, + "qps_1t_max": 248.90575720084718, + "qps_1t_runs": [ + 248.90575720084718, + 243.61615537122444, + 245.12318526070712 + ], + "median_ms": 4.0283999987877905, + "p95_ms": 5.053000000771135, + "p99_ms": 5.611600005067885, + "mean_ms": 4.067328129499184, + "knob": 16 + }, + { + "nq": 465, + "recall": 0.7795698924731183, + "qps_1t": 158.0498349946911, + "qps_1t_min": 157.87083910584533, + "qps_1t_max": 161.07181691948702, + "qps_1t_runs": [ + 158.0498349946911, + 161.07181691948702, + 157.87083910584533 + ], + "median_ms": 6.228200043551624, + "p95_ms": 7.667800004128367, + "p99_ms": 8.3147999830544, + "mean_ms": 6.289940358074483, + "knob": 32 + }, + { + "nq": 252, + "recall": 0.8444444444444444, + "qps_1t": 95.89655226979065, + "qps_1t_min": 94.75926015701279, + "qps_1t_max": 96.216059115072, + "qps_1t_runs": [ + 94.75926015701279, + 96.216059115072, + 95.89655226979065 + ], + "median_ms": 10.337699961382896, + "p95_ms": 12.434499978553504, + "p99_ms": 13.532700017094612, + "mean_ms": 10.458079100726296, + "knob": 64 + }, + { + "nq": 167, + "recall": 0.9131736526946108, + "qps_1t": 52.037531084342305, + "qps_1t_min": 50.88291299377793, + "qps_1t_max": 54.306735500045434, + "qps_1t_runs": [ + 52.037531084342305, + 50.88291299377793, + 54.306735500045434 + ], + "median_ms": 18.741499981842935, + "p95_ms": 23.26260000700131, + "p99_ms": 25.570300000254065, + "mean_ms": 19.094594810627395, + "knob": 128 + }, + { + "nq": 84, + "recall": 0.9488095238095238, + "qps_1t": 28.994268559004134, + "qps_1t_min": 28.34453443266689, + "qps_1t_max": 29.417070282163937, + "qps_1t_runs": [ + 28.994268559004134, + 29.417070282163937, + 28.34453443266689 + ], + "median_ms": 34.329500049352646, + "p95_ms": 40.43839999940246, + "p99_ms": 42.12669999105856, + "mean_ms": 34.58787103105957, + "knob": 256 + } + ], + "mutation": { + "ref_knob": 256, + "recall_before": 0, + "insert_10k": { + "seconds": 0.016971000004559755, + "per_s": 589240.4688770967 + }, + "update_10k": { + "seconds": 0.014216500043403357, + "per_s": 703408.009669731 + }, + "delete_10k": { + "seconds": 0.010688800015486777, + "per_s": 935558.7143094839 + }, + "first_query_after_ms": 5743.163299979642, + "recall_after": 0.9459770114942528, + "after": { + "nq": 87, + "recall": 0.9459770114942528, + "qps_1t": 28.75972781281632, + "qps_1t_min": 28.535344715200925, + "qps_1t_max": 28.786808735306945, + "qps_1t_runs": [ + 28.75972781281632, + 28.535344715200925, + 28.786808735306945 + ], + "median_ms": 34.423000004608184, + "p95_ms": 41.20670002885163, + "p99_ms": 43.94539998611435, + "mean_ms": 34.85107854262023 + }, + "mem_after": { + "rss_mb": 6926.3984375, + "peak_mb": 7069.29296875, + "heap_mb": 3353.5948333740234 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 1000000 + } + }, + "load_before_pct": 2.968749999999999, + "load_contaminated": false + }, + "js_exact|text384|100000": { + "version": "isongraph-vector-js (repo HEAD, Node v24.18.0)", + "build": { + "ingest_s": 0.029746399959549308, + "index_s": 0, + "total_s": 0.029746399959549308, + "first_query_ms": 65.6650000018999, + "ready_s": 0.0954113999614492 + }, + "mem_baseline": { + "rss_mb": 66.21875, + "peak_mb": 66.21875, + "heap_mb": 16.054977416992188 + }, + "mem_after_build": { + "rss_mb": 583.72265625, + "peak_mb": 583.72265625, + "heap_mb": 362.2503356933594 + }, + "mem_after_first_query": { + "rss_mb": 583.72265625, + "peak_mb": 583.72265625, + "heap_mb": 362.25098419189453 + }, + "sweep": [ + { + "nq": 56, + "recall": 1, + "qps_1t": 19.70100419273891, + "qps_1t_min": 19.095522829155946, + "qps_1t_max": 20.27124885418181, + "qps_1t_runs": [ + 19.095522829155946, + 19.70100419273891, + 20.27124885418181 + ], + "median_ms": 50.79419998219237, + "p95_ms": 55.78120000427589, + "p99_ms": 58.090100006666034, + "mean_ms": 50.819360716212984, + "knob": null + } + ], + "load_before_pct": 5.646875000000006, + "load_contaminated": false + }, + "js_ivf|text384|100000": { + "version": "isongraph-vector-js (repo HEAD, Node v24.18.0)", + "build": { + "ingest_s": 0.027143400046043098, + "index_s": 56.6768264999846, + "total_s": 56.70396990003064, + "first_query_ms": 0, + "ready_s": 56.70396990003064 + }, + "mem_baseline": { + "rss_mb": 66.21875, + "peak_mb": 66.21875, + "heap_mb": 16.054977416992188 + }, + "mem_after_build": { + "rss_mb": 1282.41796875, + "peak_mb": 1282.41796875, + "heap_mb": 410.84786224365234 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.2654, + "qps_1t": 2125.2072196805125, + "qps_1t_min": 2089.4242216666457, + "qps_1t_max": 2145.2915898579677, + "qps_1t_runs": [ + 2089.4242216666457, + 2125.2072196805125, + 2145.2915898579677 + ], + "median_ms": 0.46749995090067387, + "p95_ms": 0.5320000345818698, + "p99_ms": 0.5902000120840967, + "mean_ms": 0.47176006677909754, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.38675, + "qps_1t": 1927.4951502629824, + "qps_1t_min": 1927.2314023746917, + "qps_1t_max": 1931.0118954294092, + "qps_1t_runs": [ + 1927.4951502629824, + 1931.0118954294092, + 1927.2314023746917 + ], + "median_ms": 0.5130000063218176, + "p95_ms": 0.596800004132092, + "p99_ms": 0.6743999547325075, + "mean_ms": 0.5185167668581319, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.5155, + "qps_1t": 1636.652640164911, + "qps_1t_min": 1631.503683179281, + "qps_1t_max": 1653.452960212079, + "qps_1t_runs": [ + 1636.652640164911, + 1653.452960212079, + 1631.503683179281 + ], + "median_ms": 0.6022999878041446, + "p95_ms": 0.7175999926403165, + "p99_ms": 0.7940000505186617, + "mean_ms": 0.6095765498757828, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.6369, + "qps_1t": 1303.9286126904303, + "qps_1t_min": 1294.4784402719067, + "qps_1t_max": 1306.6609854318106, + "qps_1t_runs": [ + 1303.9286126904303, + 1306.6609854318106, + 1294.4784402719067 + ], + "median_ms": 0.762699986808002, + "p95_ms": 0.8972999639809132, + "p99_ms": 0.9675000328570604, + "mean_ms": 0.7682448335302373, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.74205, + "qps_1t": 932.5690992060768, + "qps_1t_min": 929.0743700854791, + "qps_1t_max": 936.7351994916859, + "qps_1t_runs": [ + 936.7351994916859, + 929.0743700854791, + 932.5690992060768 + ], + "median_ms": 1.0694999946281314, + "p95_ms": 1.272300025448203, + "p99_ms": 1.3978999922983348, + "mean_ms": 1.0720614165572138, + "knob": 16 + }, + { + "nq": 1779, + "recall": 0.835300730747611, + "qps_1t": 607.3269644619176, + "qps_1t_min": 604.7278088714621, + "qps_1t_max": 608.8654713724035, + "qps_1t_runs": [ + 604.7278088714621, + 607.3269644619176, + 608.8654713724035 + ], + "median_ms": 1.6472999705001712, + "p95_ms": 1.9453000277280807, + "p99_ms": 2.113300026394427, + "mean_ms": 1.647531665914481, + "knob": 32 + }, + { + "nq": 1172, + "recall": 0.9045221843003413, + "qps_1t": 358.0915880647533, + "qps_1t_min": 357.00586855232217, + "qps_1t_max": 358.6797500413248, + "qps_1t_runs": [ + 358.0915880647533, + 358.6797500413248, + 357.00586855232217 + ], + "median_ms": 2.773099986370653, + "p95_ms": 3.2890000147745013, + "p99_ms": 3.549800021573901, + "mean_ms": 2.793886149220012, + "knob": 64 + }, + { + "nq": 607, + "recall": 0.9527182866556837, + "qps_1t": 193.73100507323974, + "qps_1t_min": 193.129416216683, + "qps_1t_max": 194.75259506946915, + "qps_1t_runs": [ + 193.73100507323974, + 194.75259506946915, + 193.129416216683 + ], + "median_ms": 5.108300014398992, + "p95_ms": 6.063799955882132, + "p99_ms": 6.7069000215269625, + "mean_ms": 5.1581304233197995, + "knob": 128 + }, + { + "nq": 319, + "recall": 0.9827586206896551, + "qps_1t": 103.04111809385445, + "qps_1t_min": 100.37806026264553, + "qps_1t_max": 103.14827949154343, + "qps_1t_runs": [ + 103.04111809385445, + 100.37806026264553, + 103.14827949154343 + ], + "median_ms": 9.70460003009066, + "p95_ms": 11.326299980282784, + "p99_ms": 12.252999993506819, + "mean_ms": 9.787327064312663, + "knob": 256 + }, + { + "nq": 59, + "recall": 1, + "qps_1t": 19.541129888221302, + "qps_1t_min": 19.409294888010816, + "qps_1t_max": 19.96571717127789, + "qps_1t_runs": [ + 19.96571717127789, + 19.541129888221302, + 19.409294888010816 + ], + "median_ms": 50.44710001675412, + "p95_ms": 54.28759998176247, + "p99_ms": 59.094499971251935, + "mean_ms": 50.927224859074386, + "knob": 1265 + } + ], + "mutation": { + "ref_knob": 128, + "recall_before": 0.9527182866556837, + "insert_10k": { + "seconds": 0.0631569999968633, + "per_s": 158335.5764285297 + }, + "update_10k": { + "seconds": 0.013717400026507676, + "per_s": 729001.121252998 + }, + "delete_10k": { + "seconds": 0.008221099968068302, + "per_s": 1216382.2406783947 + }, + "first_query_after_ms": 3283.551600005012, + "recall_after": 0.9515044247787611, + "after": { + "nq": 565, + "recall": 0.9515044247787611, + "qps_1t": 193.6929463448165, + "qps_1t_min": 186.21395780264191, + "qps_1t_max": 194.44508590031936, + "qps_1t_runs": [ + 193.6929463448165, + 194.44508590031936, + 186.21395780264191 + ], + "median_ms": 5.142899986822158, + "p95_ms": 6.2559000216424465, + "p99_ms": 6.848499993793666, + "mean_ms": 5.22527250679001 + }, + "mem_after": { + "rss_mb": 1745.265625, + "peak_mb": 1745.265625, + "heap_mb": 441.2213897705078 + }, + "index_stats_after": { + "pending": 0, + "churn": 30000, + "trained_n": 100000 + } + }, + "load_before_pct": 5.646875000000006, + "load_contaminated": false + }, + "js_exact|text384|10000": { + "version": "isongraph-vector-js (repo HEAD, Node v24.18.0)", + "build": { + "ingest_s": 0.002464399964082986, + "index_s": 0, + "total_s": 0.002464399964082986, + "first_query_ms": 5.883199977688491, + "ready_s": 0.008347599941771477 + }, + "mem_baseline": { + "rss_mb": 66.22265625, + "peak_mb": 66.22265625, + "heap_mb": 16.078323364257812 + }, + "mem_after_build": { + "rss_mb": 161.3046875, + "peak_mb": 161.3046875, + "heap_mb": 58.83563995361328 + }, + "mem_after_first_query": { + "rss_mb": 161.3046875, + "peak_mb": 161.3046875, + "heap_mb": 58.83628845214844 + }, + "sweep": [ + { + "nq": 565, + "recall": 1, + "qps_1t": 197.694587196636, + "qps_1t_min": 192.32202157720317, + "qps_1t_max": 198.39146314010614, + "qps_1t_runs": [ + 197.694587196636, + 198.39146314010614, + 192.32202157720317 + ], + "median_ms": 4.995900031644851, + "p95_ms": 5.642500007525086, + "p99_ms": 6.035899976268411, + "mean_ms": 5.099486488887386, + "knob": null + } + ], + "load_before_pct": 4.290624999999999, + "load_contaminated": false + }, + "js_ivf|text384|10000": { + "version": "isongraph-vector-js (repo HEAD, Node v24.18.0)", + "build": { + "ingest_s": 0.0027119999867863953, + "index_s": 8.043748299998697, + "total_s": 8.046460299985483, + "first_query_ms": 0, + "ready_s": 8.046460299985483 + }, + "mem_baseline": { + "rss_mb": 66.22265625, + "peak_mb": 66.22265625, + "heap_mb": 16.078323364257812 + }, + "mem_after_build": { + "rss_mb": 265.80859375, + "peak_mb": 265.80859375, + "heap_mb": 84.01072692871094 + }, + "sweep": [ + { + "nq": 2000, + "recall": 0.2792, + "qps_1t": 6642.333585191237, + "qps_1t_min": 6415.460750179452, + "qps_1t_max": 6646.735318534214, + "qps_1t_runs": [ + 6415.460750179452, + 6646.735318534214, + 6642.333585191237 + ], + "median_ms": 0.14650000957772136, + "p95_ms": 0.17690000822767615, + "p99_ms": 0.30099996365606785, + "mean_ms": 0.15229091667182124, + "knob": 1 + }, + { + "nq": 2000, + "recall": 0.40335, + "qps_1t": 6277.737793039991, + "qps_1t_min": 6191.871864416425, + "qps_1t_max": 6289.440687618349, + "qps_1t_runs": [ + 6277.737793039991, + 6191.871864416425, + 6289.440687618349 + ], + "median_ms": 0.15750003512948751, + "p95_ms": 0.17559999832883477, + "p99_ms": 0.19120000069960952, + "mean_ms": 0.1599305833709271, + "knob": 2 + }, + { + "nq": 2000, + "recall": 0.54455, + "qps_1t": 5561.6455653699395, + "qps_1t_min": 5467.792918603301, + "qps_1t_max": 5578.946843235145, + "qps_1t_runs": [ + 5561.6455653699395, + 5467.792918603301, + 5578.946843235145 + ], + "median_ms": 0.17870002193376422, + "p95_ms": 0.20180002320557833, + "p99_ms": 0.21959998412057757, + "mean_ms": 0.18064578325720504, + "knob": 4 + }, + { + "nq": 2000, + "recall": 0.6832, + "qps_1t": 4575.091777079824, + "qps_1t_min": 4523.33543844947, + "qps_1t_max": 4588.684123994372, + "qps_1t_runs": [ + 4588.684123994372, + 4575.091777079824, + 4523.33543844947 + ], + "median_ms": 0.21769996965304017, + "p95_ms": 0.24750002194195986, + "p99_ms": 0.26700005400925875, + "mean_ms": 0.21919268331839703, + "knob": 8 + }, + { + "nq": 2000, + "recall": 0.8017, + "qps_1t": 3366.767581514926, + "qps_1t_min": 3339.9631596225513, + "qps_1t_max": 3380.7654704598326, + "qps_1t_runs": [ + 3339.9631596225513, + 3366.767581514926, + 3380.7654704598326 + ], + "median_ms": 0.2946000313386321, + "p95_ms": 0.3440000000409782, + "p99_ms": 0.37679995875805616, + "mean_ms": 0.2974054333753884, + "knob": 16 + }, + { + "nq": 2000, + "recall": 0.8925, + "qps_1t": 2256.327872437399, + "qps_1t_min": 2255.5736366301467, + "qps_1t_max": 2262.2937290449036, + "qps_1t_runs": [ + 2262.2937290449036, + 2256.327872437399, + 2255.5736366301467 + ], + "median_ms": 0.43839996214956045, + "p95_ms": 0.5108999903313816, + "p99_ms": 0.5484999855980277, + "mean_ms": 0.4428578164661303, + "knob": 32 + }, + { + "nq": 2000, + "recall": 0.9516, + "qps_1t": 1361.3603853694185, + "qps_1t_min": 1359.5598775289182, + "qps_1t_max": 1367.0311987711439, + "qps_1t_runs": [ + 1367.0311987711439, + 1359.5598775289182, + 1361.3603853694185 + ], + "median_ms": 0.7284999592229724, + "p95_ms": 0.830999983008951, + "p99_ms": 0.8974000229500234, + "mean_ms": 0.7338679001841228, + "knob": 64 + }, + { + "nq": 2000, + "recall": 0.9873, + "qps_1t": 765.1226492834795, + "qps_1t_min": 762.8747318258785, + "qps_1t_max": 767.6268424457456, + "qps_1t_runs": [ + 762.8747318258785, + 765.1226492834795, + 767.6268424457456 + ], + "median_ms": 1.3051999849267304, + "p95_ms": 1.4507999876514077, + "p99_ms": 1.5788000309839845, + "mean_ms": 1.3068424998685562, + "knob": 128 + }, + { + "nq": 1244, + "recall": 0.9986334405144695, + "qps_1t": 407.8588366045327, + "qps_1t_min": 407.7402604716675, + "qps_1t_max": 408.5740646426448, + "qps_1t_runs": [ + 407.8588366045327, + 408.5740646426448, + 407.7402604716675 + ], + "median_ms": 2.430399996228516, + "p95_ms": 2.711000037379563, + "p99_ms": 2.9160999692976475, + "mean_ms": 2.450635691370769, + "knob": 256 + }, + { + "nq": 606, + "recall": 1, + "qps_1t": 198.14701802026764, + "qps_1t_min": 196.64134634792745, + "qps_1t_max": 201.90120298376974, + "qps_1t_runs": [ + 198.14701802026764, + 201.90120298376974, + 196.64134634792745 + ], + "median_ms": 4.968799999915063, + "p95_ms": 5.559799959883094, + "p99_ms": 5.925300007220358, + "mean_ms": 5.0283585810110205, + "knob": 400 + } + ], + "load_before_pct": 4.290624999999999, + "load_contaminated": false + }, + "lancedb_flat|glove|1000000": { + "mem_baseline": { + "rss_mb": 35.12109375, + "peak_mb": 35.12109375 + }, + "version": "lancedb 0.41.0", + "build": { + "ingest_s": 0.8012462998158298, + "index_s": 1.800013706088066e-06, + "total_s": 0.8012480998295359, + "first_query_ms": 156.952099991031, + "ready_s": 0.9582001998205669 + }, + "mem_after_build": { + "rss_mb": 128.203125, + "peak_mb": 158.734375 + }, + "mem_after_first_query": { + "rss_mb": 141.90625, + "peak_mb": 286.6875 + }, + "sweep": [ + { + "nq": 30, + "recall_queries": 30, + "recall": 1.0, + "qps_1t_runs": [ + 2.8897626910166934 + ], + "qps_1t": 2.8897626910166934, + "qps_1t_min": 2.8897626910166934, + "qps_1t_max": 2.8897626910166934, + "median_ms": 345.8859999955166, + "p95_ms": 384.75180996756535, + "p99_ms": 392.7767049934482, + "mean_ms": 346.0491766706885, + "knob": null + } + ], + "loop_allthreads": { + "nq": 30, + "threads": 32, + "qps": 9.45213711840502, + "median_ms": 105.48774999915622, + "p95_ms": 124.36967001412992, + "p99_ms": 137.11867800157052, + "mean_ms": 106.23351222214599 + }, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.0704729999997653, + "per_s": 141898.3156674656, + "batch": 1000 + }, + "update_10k": { + "seconds": 1.495726499997545, + "per_s": 6685.7142666232185, + "batch": 1000 + }, + "delete_10k": { + "seconds": 1.3979403999983333, + "per_s": 7153.380787916225, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 176.37890625, + "peak_mb": 305.52734375 + }, + "first_query_after_ms": 272.6897999527864, + "after": { + "nq": 30, + "recall_queries": 30, + "recall": 1.0, + "qps_1t_runs": [ + 2.267650244375897 + ], + "qps_1t": 2.267650244375897, + "qps_1t_min": 2.267650244375897, + "qps_1t_max": 2.267650244375897, + "median_ms": 440.3876999858767, + "p95_ms": 467.7804049686529, + "p99_ms": 483.53027496719733, + "mean_ms": 440.98511332607205 + }, + "recall_after": 1.0 + }, + "load_before_pct": 12.109374999999995, + "load_contaminated": false + }, + "lancedb_flat|glove|100000": { + "mem_baseline": { + "rss_mb": 35.25, + "peak_mb": 35.25 + }, + "version": "lancedb 0.41.0", + "build": { + "ingest_s": 0.0828034000005573, + "index_s": 2.600019797682762e-06, + "total_s": 0.08280600002035499, + "first_query_ms": 21.85830002417788, + "ready_s": 0.10466430004453287, + "total_s_runs": [ + 0.08280600002035499, + 0.07991450000554323, + 0.07684410008369014 + ] + }, + "mem_after_build": { + "rss_mb": 126.44140625, + "peak_mb": 156.97265625 + }, + "mem_after_first_query": { + "rss_mb": 134.87890625, + "peak_mb": 179.64453125 + }, + "sweep": [ + { + "nq": 86, + "recall_queries": 86, + "recall": 1.0, + "qps_1t_runs": [ + 31.198842740640725, + 34.47822143451636, + 34.250152835093274 + ], + "qps_1t": 34.250152835093274, + "qps_1t_min": 31.198842740640725, + "qps_1t_max": 34.47822143451636, + "median_ms": 29.508699983125553, + "p95_ms": 34.16231001028791, + "p99_ms": 37.37298197462224, + "mean_ms": 30.084412403733204, + "knob": null + } + ], + "loop_allthreads": { + "nq": 255, + "threads": 32, + "qps": 86.46535722215252, + "median_ms": 11.355900031048805, + "p95_ms": 13.478979992214589, + "p99_ms": 15.230768015608191, + "mean_ms": 11.593433857052373 + }, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.02863959997193888, + "per_s": 349166.88814780983, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.134306599968113, + "per_s": 74456.50476130133, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.1077742999768816, + "per_s": 92786.4992131248, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 200.8046875, + "peak_mb": 222.77734375 + }, + "first_query_after_ms": 20.78080002684146, + "after": { + "nq": 71, + "recall_queries": 71, + "recall": 1.0, + "qps_1t_runs": [ + 24.111326952514904, + 23.934592088038578, + 22.96083974063845 + ], + "qps_1t": 23.934592088038578, + "qps_1t_min": 22.96083974063845, + "qps_1t_max": 24.111326952514904, + "median_ms": 41.49259999394417, + "p95_ms": 46.80060000391676, + "p99_ms": 51.528455985244356, + "mean_ms": 42.269076524300175 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 6.00004568696022e-07, + "disk_mb": 46.77094554901123, + "load_s": 0.0015787999727763236, + "first_query_after_load_ms": 16.915499989409, + "time_to_ready_s": 0.018494299962185323 + }, + "load_before_pct": 10.987500000000008, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 0.08135599992237985, + "mem": { + "rss_mb": 126.08984375, + "peak_mb": 156.6171875 + }, + "subsets": { + "f10": [ + { + "knob": null, + "nq": 86, + "recall": 1.0, + "qps_1t": 33.84476313064172, + "qps_1t_runs": [ + 31.98960858190996, + 35.077751262317385, + 33.84476313064172 + ], + "median_ms": 28.858350007794797, + "p95_ms": 37.26772001537029, + "p99_ms": 40.7329980126815, + "mean_ms": 29.77163953388059 + } + ], + "f01": [ + { + "knob": null, + "nq": 109, + "recall": 1.0, + "qps_1t": 37.5524200749648, + "qps_1t_runs": [ + 36.17342388999656, + 37.5524200749648, + 37.58643111291594 + ], + "median_ms": 26.721499976702034, + "p95_ms": 30.50053000333719, + "p99_ms": 33.50158200482839, + "mean_ms": 26.959797555604943 + } + ] + } + } + }, + "lancedb_flat|sift|100000": { + "mem_baseline": { + "rss_mb": 35.3671875, + "peak_mb": 35.3671875 + }, + "version": "lancedb 0.41.0", + "build": { + "ingest_s": 0.10293530009221286, + "index_s": 1.9000144675374031e-06, + "total_s": 0.1029372001066804, + "first_query_ms": 33.70490000816062, + "ready_s": 0.13664210011484101, + "total_s_runs": [ + 0.1029372001066804, + 0.22343769995495677, + 0.247466099972371 + ] + }, + "mem_after_build": { + "rss_mb": 131.12109375, + "peak_mb": 160.4296875 + }, + "mem_after_first_query": { + "rss_mb": 141.91015625, + "peak_mb": 204.19140625 + }, + "sweep": [ + { + "nq": 43, + "recall_queries": 43, + "recall": 1.0, + "qps_1t_runs": [ + 13.65146235646176, + 15.077235594120301, + 15.965725523477303 + ], + "qps_1t": 15.077235594120301, + "qps_1t_min": 13.65146235646176, + "qps_1t_max": 15.965725523477303, + "median_ms": 66.86670001363382, + "p95_ms": 78.52365997387096, + "p99_ms": 81.96539199445397, + "mean_ms": 67.40385116043667, + "knob": null + } + ], + "loop_allthreads": { + "nq": 56, + "threads": 32, + "qps": 21.055871491797273, + "median_ms": 45.960849965922534, + "p95_ms": 56.49838999379426, + "p99_ms": 64.64650301611984, + "mean_ms": 46.83740595161604 + }, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.12003140000160784, + "per_s": 83311.53348095622, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.36558129999320954, + "per_s": 27353.696702171976, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.24850059999153018, + "per_s": 40241.35153130752, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 186.97265625, + "peak_mb": 233.9453125 + }, + "first_query_after_ms": 77.36070000100881, + "after": { + "nq": 35, + "recall_queries": 35, + "recall": 1.0, + "qps_1t_runs": [ + 11.753031862202603, + 11.860933737857877, + 12.405521322502707 + ], + "qps_1t": 11.860933737857877, + "qps_1t_min": 11.753031862202603, + "qps_1t_max": 12.405521322502707, + "median_ms": 82.52759999595582, + "p95_ms": 94.37460000626743, + "p99_ms": 99.6326959808357, + "mean_ms": 83.33469714104596 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 7.00005330145359e-07, + "disk_mb": 59.58809280395508, + "load_s": 0.00497519999044016, + "first_query_after_load_ms": 74.04520001728088, + "time_to_ready_s": 0.07902040000772104 + }, + "load_before_pct": 12.684374999999996, + "load_contaminated": false + }, + "lancedb_flat|text384|1000000": { + "mem_baseline": { + "rss_mb": 37.4765625, + "peak_mb": 37.4765625 + }, + "version": "lancedb 0.41.0", + "build": { + "ingest_s": 2.0412177999387495, + "index_s": 2.400018274784088e-06, + "total_s": 2.0412201999570243, + "first_query_ms": 653.5677000065334, + "ready_s": 2.6947878999635577 + }, + "mem_after_build": { + "rss_mb": 175.48046875, + "peak_mb": 263.3125 + }, + "mem_after_first_query": { + "rss_mb": 189.359375, + "peak_mb": 601.17578125 + }, + "sweep": [ + { + "nq": 30, + "recall_queries": 30, + "recall": 1.0, + "qps_1t_runs": [ + 0.7249557861538458 + ], + "qps_1t": 0.7249557861538458, + "qps_1t_min": 0.7249557861538458, + "qps_1t_max": 0.7249557861538458, + "median_ms": 1372.2370000032242, + "p95_ms": 1437.713464992703, + "p99_ms": 1462.75608900527, + "mean_ms": 1379.3944666686002, + "knob": null + } + ], + "loop_allthreads": { + "nq": 30, + "threads": 32, + "qps": 1.7299119097310964, + "median_ms": 575.7210999727249, + "p95_ms": 602.9495449736714, + "p99_ms": 608.2479439658346, + "mean_ms": 578.0641166609712 + }, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.055588999995961785, + "per_s": 179891.7052065416, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.6347060999833047, + "per_s": 15755.323606095859, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.3578336000209674, + "per_s": 27945.950294813134, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 205.18359375, + "peak_mb": 628.53125 + }, + "first_query_after_ms": 939.8463999968953, + "after": { + "nq": 30, + "recall_queries": 30, + "recall": 1.0, + "qps_1t_runs": [ + 0.5560314855568307 + ], + "qps_1t": 0.5560314855568307, + "qps_1t_min": 0.5560314855568307, + "qps_1t_max": 0.5560314855568307, + "median_ms": 1796.5057000110392, + "p95_ms": 1852.6564899861114, + "p99_ms": 1873.9484239957528, + "mean_ms": 1798.4593066677917 + }, + "recall_after": 1.0 + }, + "load_before_pct": 3.549999999999999, + "load_contaminated": false + }, + "lancedb_flat|text384|100000": { + "mem_baseline": { + "rss_mb": 37.3515625, + "peak_mb": 37.3515625 + }, + "version": "lancedb 0.41.0", + "build": { + "ingest_s": 0.21901230001822114, + "index_s": 2.400018274784088e-06, + "total_s": 0.21901470003649592, + "first_query_ms": 64.86110005062073, + "ready_s": 0.28387580008711666, + "total_s_runs": [ + 0.21901470003649592, + 0.21189370000502095, + 0.21135139995021746 + ] + }, + "mem_after_build": { + "rss_mb": 173.78515625, + "peak_mb": 261.6875 + }, + "mem_after_first_query": { + "rss_mb": 186.32421875, + "peak_mb": 361.078125 + }, + "sweep": [ + { + "nq": 30, + "recall_queries": 30, + "recall": 1.0, + "qps_1t_runs": [ + 7.163113575546817, + 7.216794403598273, + 7.188175853432592 + ], + "qps_1t": 7.188175853432592, + "qps_1t_min": 7.163113575546817, + "qps_1t_max": 7.216794403598273, + "median_ms": 136.85725000686944, + "p95_ms": 150.51211501995567, + "p99_ms": 172.52110999659635, + "mean_ms": 139.09570889081806, + "knob": null + } + ], + "loop_allthreads": { + "nq": 42, + "threads": 32, + "qps": 13.747629719819061, + "median_ms": 72.05929997144267, + "p95_ms": 80.92444996873382, + "p99_ms": 90.0290749996202, + "mean_ms": 72.53502539473601 + }, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.052557399962097406, + "per_s": 190268.1640874864, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.16496710001956671, + "per_s": 60618.14749009895, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.11116339999716729, + "per_s": 89957.66592470925, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 252.078125, + "peak_mb": 395.10546875 + }, + "first_query_after_ms": 103.32120000384748, + "after": { + "nq": 30, + "recall_queries": 30, + "recall": 1.0, + "qps_1t_runs": [ + 4.803872766892777 + ], + "qps_1t": 4.803872766892777, + "qps_1t_min": 4.803872766892777, + "qps_1t_max": 4.803872766892777, + "median_ms": 202.50729998224415, + "p95_ms": 225.15360498800874, + "p99_ms": 309.4573300110643, + "mean_ms": 208.165380001689 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 1.100008375942707e-06, + "disk_mb": 176.77569484710693, + "load_s": 0.002555200015194714, + "first_query_after_load_ms": 110.22700002649799, + "time_to_ready_s": 0.1127822000416927 + }, + "load_before_pct": 4.387499999999998, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 0.2166323999990709, + "mem": { + "rss_mb": 173.4296875, + "peak_mb": 261.3203125 + }, + "subsets": { + "f10": [ + { + "knob": null, + "nq": 30, + "recall": 1.0, + "qps_1t": 7.091514744945411, + "qps_1t_runs": [ + 7.091514744945411, + 7.249012370459008, + 6.987596248668594 + ], + "median_ms": 140.09994998923503, + "p95_ms": 149.52533498581033, + "p99_ms": 157.88445995829534, + "mean_ms": 140.69138443964118 + } + ], + "f01": [ + { + "knob": null, + "nq": 30, + "recall": 1.0, + "qps_1t": 7.280470869889407, + "qps_1t_runs": [ + 7.294221087910168, + 7.2739716952257645, + 7.280470869889407 + ], + "median_ms": 134.84800001606345, + "p95_ms": 152.26042499416508, + "p99_ms": 163.86655697773676, + "mean_ms": 137.30835332987934 + } + ] + } + } + }, + "lancedb_flat|text384|10000": { + "mem_baseline": { + "rss_mb": 37.515625, + "peak_mb": 37.515625 + }, + "version": "lancedb 0.41.0", + "build": { + "ingest_s": 0.03669500001706183, + "index_s": 1.300009898841381e-06, + "total_s": 0.03669630002696067, + "first_query_ms": 16.3079000194557, + "ready_s": 0.05300420004641637 + }, + "mem_after_build": { + "rss_mb": 141.32421875, + "peak_mb": 183.375 + }, + "mem_after_first_query": { + "rss_mb": 149.83203125, + "peak_mb": 183.375 + }, + "sweep": [ + { + "nq": 181, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 60.40533849721971, + 60.93997452638205, + 61.141338538648114 + ], + "qps_1t": 60.93997452638205, + "qps_1t_min": 60.40533849721971, + "qps_1t_max": 61.141338538648114, + "median_ms": 16.04149997001514, + "p95_ms": 18.860060034785416, + "p99_ms": 23.160338011803127, + "mean_ms": 16.43998821524982, + "knob": null + } + ], + "loop_allthreads": { + "nq": 219, + "threads": 32, + "qps": 74.59007392256773, + "median_ms": 13.29600001918152, + "p95_ms": 14.838460006285459, + "p99_ms": 16.455072010867294, + "mean_ms": 13.422500151028798 + }, + "load_before_pct": 4.124999999999997, + "load_contaminated": false + }, + "lancedb_ivfflat|glove|1000000": { + "mem_baseline": { + "rss_mb": 35.19921875, + "peak_mb": 35.19921875 + }, + "version": "lancedb 0.41.0", + "build": { + "ingest_s": 0.7647575001465157, + "index_s": 8.916056599991862, + "total_s": 9.680814100138377, + "first_query_ms": 281.7618999979459, + "ready_s": 9.962576000136323 + }, + "mem_after_build": { + "rss_mb": 185.77734375, + "peak_mb": 901.3671875 + }, + "mem_after_first_query": { + "rss_mb": 589.79296875, + "peak_mb": 901.3671875 + }, + "sweep": [ + { + "nq": 710, + "recall_queries": 1000, + "recall": 0.2826, + "qps_1t_runs": [ + 256.52459469079184, + 232.00775005017073, + 165.08332383338188 + ], + "qps_1t": 232.00775005017073, + "qps_1t_min": 165.08332383338188, + "qps_1t_max": 256.52459469079184, + "median_ms": 4.1416500171180815, + "p95_ms": 7.376519998069853, + "p99_ms": 9.202730014221748, + "mean_ms": 4.755336525243327, + "knob": 1 + }, + { + "nq": 499, + "recall_queries": 1000, + "recall": 0.3981, + "qps_1t_runs": [ + 158.2638400017644, + 212.1748929719422, + 233.77547819396054 + ], + "qps_1t": 212.1748929719422, + "qps_1t_min": 158.2638400017644, + "qps_1t_max": 233.77547819396054, + "median_ms": 4.477099981158972, + "p95_ms": 7.542659982573241, + "p99_ms": 10.214051986113187, + "mean_ms": 5.103088109267213, + "knob": 2 + }, + { + "nq": 787, + "recall_queries": 1000, + "recall": 0.5174, + "qps_1t_runs": [ + 260.2982694323632, + 261.4322200443843, + 220.0374488931418 + ], + "qps_1t": 260.2982694323632, + "qps_1t_min": 220.0374488931418, + "qps_1t_max": 261.4322200443843, + "median_ms": 3.812399983871728, + "p95_ms": 6.056600017473102, + "p99_ms": 8.138520014472324, + "mean_ms": 4.070503599819448, + "knob": 4 + }, + { + "nq": 638, + "recall_queries": 1000, + "recall": 0.6209, + "qps_1t_runs": [ + 158.831494116633, + 165.95902570030233, + 166.3314315144138 + ], + "qps_1t": 165.95902570030233, + "qps_1t_min": 158.831494116633, + "qps_1t_max": 166.3314315144138, + "median_ms": 6.228350015589967, + "p95_ms": 7.774985005380585, + "p99_ms": 8.855420989566479, + "mean_ms": 6.111219017616792, + "knob": 8 + }, + { + "nq": 481, + "recall_queries": 1000, + "recall": 0.7151000000000001, + "qps_1t_runs": [ + 167.83595288354198, + 166.7475963982963, + 177.41365562357663 + ], + "qps_1t": 167.83595288354198, + "qps_1t_min": 166.7475963982963, + "qps_1t_max": 177.41365562357663, + "median_ms": 5.806700035464019, + "p95_ms": 7.55847999243997, + "p99_ms": 8.25189002789557, + "mean_ms": 5.863943867466386, + "knob": 16 + }, + { + "nq": 557, + "recall_queries": 1000, + "recall": 0.7928, + "qps_1t_runs": [ + 171.24781734170855, + 160.0558109168945, + 166.48933210881273 + ], + "qps_1t": 166.48933210881273, + "qps_1t_min": 160.0558109168945, + "qps_1t_max": 171.24781734170855, + "median_ms": 5.927999969571829, + "p95_ms": 7.746899995254353, + "p99_ms": 8.367049985099582, + "mean_ms": 6.031233991857424, + "knob": 32 + }, + { + "nq": 384, + "recall_queries": 1000, + "recall": 0.8552000000000001, + "qps_1t_runs": [ + 145.92519288300105, + 149.77800676363572, + 151.59595006369514 + ], + "qps_1t": 149.77800676363572, + "qps_1t_min": 145.92519288300105, + "qps_1t_max": 151.59595006369514, + "median_ms": 6.700850004563108, + "p95_ms": 8.267495036125183, + "p99_ms": 9.446597989881411, + "mean_ms": 6.708618749952519, + "knob": 64 + }, + { + "nq": 338, + "recall_queries": 1000, + "recall": 0.9125999999999999, + "qps_1t_runs": [ + 124.44170591030655, + 133.33127156472432, + 134.13723604082497 + ], + "qps_1t": 133.33127156472432, + "qps_1t_min": 124.44170591030655, + "qps_1t_max": 134.13723604082497, + "median_ms": 7.718499982729554, + "p95_ms": 9.20445000519976, + "p99_ms": 11.316130001214333, + "mean_ms": 7.663686193307197, + "knob": 128 + }, + { + "nq": 323, + "recall_queries": 1000, + "recall": 0.9512999999999999, + "qps_1t_runs": [ + 108.56475778048762, + 106.81718873717033, + 111.20980363393076 + ], + "qps_1t": 108.56475778048762, + "qps_1t_min": 106.81718873717033, + "qps_1t_max": 111.20980363393076, + "median_ms": 9.323399979621172, + "p95_ms": 10.863739997148514, + "p99_ms": 11.396468023303887, + "mean_ms": 9.188298140861521, + "knob": 256 + }, + { + "nq": 59, + "recall_queries": 59, + "recall": 1.0, + "qps_1t_runs": [ + 20.22210243508479, + 21.213294755503018, + 20.885383071862652 + ], + "qps_1t": 20.885383071862652, + "qps_1t_min": 20.22210243508479, + "qps_1t_max": 21.213294755503018, + "median_ms": 48.00289997365326, + "p95_ms": 52.13980001863092, + "p99_ms": 54.887856019195205, + "mean_ms": 48.15715593009619, + "knob": 4000 + } + ], + "mutation": { + "ref_knob": 256, + "recall_before": 0.9512999999999999, + "insert_10k": { + "seconds": 0.10521310003241524, + "per_s": 95045.19871498023, + "batch": 1000 + }, + "update_10k": { + "seconds": 1.294876700034365, + "per_s": 7722.7430223546435, + "batch": 1000 + }, + "delete_10k": { + "seconds": 1.2662446000031196, + "per_s": 7897.36832834301, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 614.109375, + "peak_mb": 901.3671875 + }, + "first_query_after_ms": 41.94409999763593, + "after": { + "nq": 117, + "recall_queries": 117, + "recall": 0.9393162393162393, + "qps_1t_runs": [ + 43.460760270633635, + 41.13110394804053, + 40.475915119885435 + ], + "qps_1t": 41.13110394804053, + "qps_1t_min": 40.475915119885435, + "qps_1t_max": 43.460760270633635, + "median_ms": 23.94420001655817, + "p95_ms": 29.041250032605603, + "p99_ms": 30.291050003143027, + "mean_ms": 24.009270942769945 + }, + "recall_after": 0.9393162393162393 + }, + "load_before_pct": 8.259375000000002, + "load_contaminated": false + }, + "lancedb_ivfflat|glove|100000": { + "mem_baseline": { + "rss_mb": 35.12109375, + "peak_mb": 35.12109375 + }, + "version": "lancedb 0.41.0", + "build": { + "ingest_s": 0.08521079999627545, + "index_s": 0.6434958000318147, + "total_s": 0.7287066000280902, + "first_query_ms": 68.63890000386164, + "ready_s": 0.7973455000319518, + "total_s_runs": [ + 0.7287066000280902, + 0.6973931000684388, + 0.6925771000678651 + ] + }, + "mem_after_build": { + "rss_mb": 166.86328125, + "peak_mb": 235.97265625 + }, + "mem_after_first_query": { + "rss_mb": 208.12109375, + "peak_mb": 235.97265625 + }, + "sweep": [ + { + "nq": 950, + "recall_queries": 1000, + "recall": 0.25559999999999994, + "qps_1t_runs": [ + 308.46440287274817, + 306.93527368700455, + 309.8845134375193 + ], + "qps_1t": 308.46440287274817, + "qps_1t_min": 306.93527368700455, + "qps_1t_max": 309.8845134375193, + "median_ms": 3.2040500082075596, + "p95_ms": 3.6748650163644925, + "p99_ms": 4.23184000479523, + "mean_ms": 3.242296561284252, + "knob": 1 + }, + { + "nq": 922, + "recall_queries": 1000, + "recall": 0.37179999999999996, + "qps_1t_runs": [ + 308.6557069689085, + 313.83323811540305, + 303.77133096482214 + ], + "qps_1t": 308.6557069689085, + "qps_1t_min": 303.77133096482214, + "qps_1t_max": 313.83323811540305, + "median_ms": 3.181600011885166, + "p95_ms": 3.741824970347807, + "p99_ms": 4.388720021233891, + "mean_ms": 3.2394037961866236, + "knob": 2 + }, + { + "nq": 987, + "recall_queries": 1000, + "recall": 0.49420000000000003, + "qps_1t_runs": [ + 308.8301099786555, + 308.8613543381432, + 306.5842270858264 + ], + "qps_1t": 308.8301099786555, + "qps_1t_min": 306.5842270858264, + "qps_1t_max": 308.8613543381432, + "median_ms": 3.202899999450892, + "p95_ms": 3.709100012201816, + "p99_ms": 4.47353997733444, + "mean_ms": 3.2458237754453094, + "knob": 4 + }, + { + "nq": 959, + "recall_queries": 1000, + "recall": 0.6105, + "qps_1t_runs": [ + 304.3762834280036, + 296.55246770715104, + 301.6791860492452 + ], + "qps_1t": 301.6791860492452, + "qps_1t_min": 296.55246770715104, + "qps_1t_max": 304.3762834280036, + "median_ms": 3.252300026360899, + "p95_ms": 3.8504400290548797, + "p99_ms": 4.720644003245979, + "mean_ms": 3.3240904069324677, + "knob": 8 + }, + { + "nq": 922, + "recall_queries": 1000, + "recall": 0.711, + "qps_1t_runs": [ + 304.32276962406553, + 310.17596160351616, + 304.7157035284071 + ], + "qps_1t": 304.7157035284071, + "qps_1t_min": 304.32276962406553, + "qps_1t_max": 310.17596160351616, + "median_ms": 3.2418500049971044, + "p95_ms": 3.6611749965231866, + "p99_ms": 4.371875012293456, + "mean_ms": 3.2639029285012233, + "knob": 16 + }, + { + "nq": 887, + "recall_queries": 1000, + "recall": 0.803, + "qps_1t_runs": [ + 297.3240102081176, + 297.86377947374024, + 299.8711096058121 + ], + "qps_1t": 297.86377947374024, + "qps_1t_min": 297.3240102081176, + "qps_1t_max": 299.8711096058121, + "median_ms": 3.3300999784842134, + "p95_ms": 3.740200016181916, + "p99_ms": 4.408160003367828, + "mean_ms": 3.351779856434885, + "knob": 32 + }, + { + "nq": 880, + "recall_queries": 1000, + "recall": 0.8755, + "qps_1t_runs": [ + 283.58769103631505, + 278.4807922801689, + 282.8018671293463 + ], + "qps_1t": 282.8018671293463, + "qps_1t_min": 278.4807922801689, + "qps_1t_max": 283.58769103631505, + "median_ms": 3.4990000422112644, + "p95_ms": 3.8997000083327293, + "p99_ms": 4.581024025683294, + "mean_ms": 3.551067538677616, + "knob": 64 + }, + { + "nq": 786, + "recall_queries": 1000, + "recall": 0.9348, + "qps_1t_runs": [ + 256.25471802676833, + 247.43556332202778, + 253.02816926525713 + ], + "qps_1t": 253.02816926525713, + "qps_1t_min": 247.43556332202778, + "qps_1t_max": 256.25471802676833, + "median_ms": 3.837800002656877, + "p95_ms": 4.676574986660853, + "p99_ms": 5.990969007834784, + "mean_ms": 3.9653175150704247, + "knob": 128 + }, + { + "nq": 670, + "recall_queries": 1000, + "recall": 0.974, + "qps_1t_runs": [ + 207.08804896050407, + 217.03634918144752, + 219.11147158043107 + ], + "qps_1t": 217.03634918144752, + "qps_1t_min": 207.08804896050407, + "qps_1t_max": 219.11147158043107, + "median_ms": 4.454350011656061, + "p95_ms": 5.512090030242689, + "p99_ms": 7.193224999355153, + "mean_ms": 4.666758010677057, + "knob": 256 + }, + { + "nq": 352, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 112.78721528209984, + 113.7208833336949, + 112.3670039183191 + ], + "qps_1t": 112.78721528209984, + "qps_1t_min": 112.3670039183191, + "qps_1t_max": 113.7208833336949, + "median_ms": 8.53935000486672, + "p95_ms": 10.213950008619577, + "p99_ms": 16.33733499620577, + "mean_ms": 8.853040814453076, + "knob": 1265 + } + ], + "mutation": { + "ref_knob": 256, + "recall_before": 0.974, + "insert_10k": { + "seconds": 0.03130010003224015, + "per_s": 319487.7968345042, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.13761199999134988, + "per_s": 72668.08127654993, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.10307250003097579, + "per_s": 97019.08847650689, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 253.96875, + "peak_mb": 323.70703125 + }, + "first_query_after_ms": 9.27480001701042, + "after": { + "nq": 181, + "recall_queries": 1000, + "recall": 0.9773999999999998, + "qps_1t_runs": [ + 64.06556688330947, + 62.784884231407126, + 66.1675875753147 + ], + "qps_1t": 64.06556688330947, + "qps_1t_min": 62.784884231407126, + "qps_1t_max": 66.1675875753147, + "median_ms": 14.918000029865652, + "p95_ms": 18.37568997289054, + "p99_ms": 27.89207602036186, + "mean_ms": 15.54984972652727 + }, + "recall_after": 0.9773999999999998 + }, + "saveload": { + "save_s": 1.200009137392044e-06, + "disk_mb": 86.09145164489746, + "load_s": 0.0016540000215172768, + "first_query_after_load_ms": 26.843000028748065, + "time_to_ready_s": 0.028497000050265342 + }, + "load_before_pct": 3.506249999999997, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 0.7403945000260137, + "mem": { + "rss_mb": 166.5546875, + "peak_mb": 237.5625 + }, + "subsets": { + "f10": [ + { + "knob": 2, + "nq": 451, + "recall": 0.21263858093126387, + "qps_1t": 157.1920084239539, + "qps_1t_runs": [ + 145.87104093734274, + 157.1920084239539, + 160.48983061336196 + ], + "median_ms": 6.462000019382685, + "p95_ms": 7.775840000249441, + "p99_ms": 13.47486401209608, + "mean_ms": 6.4826470804304375 + }, + { + "knob": 8, + "nq": 473, + "recall": 0.42959830866807613, + "qps_1t": 156.44470831367647, + "qps_1t_runs": [ + 154.21222544456882, + 159.12505303808453, + 156.44470831367647 + ], + "median_ms": 6.462799967266619, + "p95_ms": 7.3917799978517, + "p99_ms": 9.57161603029816, + "mean_ms": 6.386990134618714 + }, + { + "knob": 32, + "nq": 455, + "recall": 0.6760439560439561, + "qps_1t": 160.103913433019, + "qps_1t_runs": [ + 153.81023270379276, + 160.103913433019, + 161.71620772535354 + ], + "median_ms": 6.4830000046640635, + "p95_ms": 7.28233999107033, + "p99_ms": 8.925828018691382, + "mean_ms": 6.310377875812038 + }, + { + "knob": 64, + "nq": 428, + "recall": 0.789018691588785, + "qps_1t": 150.59075202851295, + "qps_1t_runs": [ + 150.59075202851295, + 148.46389470852142, + 152.68430962905364 + ], + "median_ms": 6.660499988356605, + "p95_ms": 8.035800003563054, + "p99_ms": 11.029024035087787, + "mean_ms": 6.641873287623081 + }, + { + "knob": 1265, + "nq": 229, + "recall": 1.0, + "qps_1t": 88.26139619078172, + "qps_1t_runs": [ + 88.26139619078172, + 89.62518434181486, + 87.52938636075316 + ], + "median_ms": 11.270400020293891, + "p95_ms": 12.537630024598911, + "p99_ms": 13.556226032087578, + "mean_ms": 11.304098107967949 + } + ], + "f01": [ + { + "knob": 2, + "nq": 568, + "recall": 0.06654929577464788, + "qps_1t": 188.47488048163868, + "qps_1t_runs": [ + 188.47488048163868, + 189.30004805282664, + 174.68660135026875 + ], + "median_ms": 5.444300011731684, + "p95_ms": 6.220150031731464, + "p99_ms": 7.358800988295116, + "mean_ms": 5.437634389828661 + }, + { + "knob": 8, + "nq": 485, + "recall": 0.18639175257731957, + "qps_1t": 169.46758145277738, + "qps_1t_runs": [ + 169.46758145277738, + 169.18342417652553, + 172.9895537865669 + ], + "median_ms": 6.002800015266985, + "p95_ms": 6.690100039122624, + "p99_ms": 7.8513219871092605, + "mean_ms": 5.864091339413896 + }, + { + "knob": 32, + "nq": 475, + "recall": 0.43115789473684213, + "qps_1t": 161.83975345934857, + "qps_1t_runs": [ + 161.83975345934857, + 163.79662226719077, + 158.62341327720793 + ], + "median_ms": 6.294700026046485, + "p95_ms": 7.222119998186827, + "p99_ms": 8.880116001237182, + "mean_ms": 6.196107718472679 + }, + { + "knob": 64, + "nq": 428, + "recall": 0.572429906542056, + "qps_1t": 154.17383698209585, + "qps_1t_runs": [ + 149.86519133864815, + 154.17383698209585, + 157.73335111207084 + ], + "median_ms": 6.517199974041432, + "p95_ms": 7.923950001713818, + "p99_ms": 9.571372971986433, + "mean_ms": 6.4995538942806705 + }, + { + "knob": 1265, + "nq": 318, + "recall": 1.0, + "qps_1t": 98.98701729962468, + "qps_1t_runs": [ + 98.98701729962468, + 97.31415978439246, + 100.58018000632342 + ], + "median_ms": 9.800450003240258, + "p95_ms": 11.970265029231085, + "p99_ms": 15.246728970087144, + "mean_ms": 10.106882809190786 + } + ] + } + } + }, + "lancedb_ivfflat|sift|100000": { + "mem_baseline": { + "rss_mb": 35.49609375, + "peak_mb": 35.49609375 + }, + "version": "lancedb 0.41.0", + "build": { + "ingest_s": 0.11340080003719777, + "index_s": 0.8575120000168681, + "total_s": 0.9709128000540659, + "first_query_ms": 91.44479996757582, + "ready_s": 1.0623576000216417, + "total_s_runs": [ + 0.9709128000540659, + 0.8295959000242874, + 0.8694884999422356 + ] + }, + "mem_after_build": { + "rss_mb": 173.86328125, + "peak_mb": 265.78515625 + }, + "mem_after_first_query": { + "rss_mb": 224.69921875, + "peak_mb": 265.78515625 + }, + "sweep": [ + { + "nq": 835, + "recall_queries": 1000, + "recall": 0.2785, + "qps_1t_runs": [ + 270.3759264629438, + 279.07354817999885, + 277.26289523624047 + ], + "qps_1t": 277.26289523624047, + "qps_1t_min": 270.3759264629438, + "qps_1t_max": 279.07354817999885, + "median_ms": 3.546499996446073, + "p95_ms": 4.483719985000789, + "p99_ms": 5.873563967179508, + "mean_ms": 3.6295080632055012, + "knob": 1 + }, + { + "nq": 780, + "recall_queries": 1000, + "recall": 0.4209, + "qps_1t_runs": [ + 286.7034863200027, + 288.6847032611311, + 268.40922516003894 + ], + "qps_1t": 286.7034863200027, + "qps_1t_min": 268.40922516003894, + "qps_1t_max": 288.6847032611311, + "median_ms": 3.4685999853536487, + "p95_ms": 4.268470022361726, + "p99_ms": 5.938413013354875, + "mean_ms": 3.559188418932116, + "knob": 2 + }, + { + "nq": 832, + "recall_queries": 1000, + "recall": 0.5785, + "qps_1t_runs": [ + 270.26455484794405, + 275.54052895406556, + 283.07718645067973 + ], + "qps_1t": 275.54052895406556, + "qps_1t_min": 270.26455484794405, + "qps_1t_max": 283.07718645067973, + "median_ms": 3.530949994456023, + "p95_ms": 4.427474981639534, + "p99_ms": 5.89647502347364, + "mean_ms": 3.620637940679071, + "knob": 4 + }, + { + "nq": 825, + "recall_queries": 1000, + "recall": 0.7387999999999999, + "qps_1t_runs": [ + 280.6730955796045, + 264.3531079845733, + 268.7196631886525 + ], + "qps_1t": 268.7196631886525, + "qps_1t_min": 264.3531079845733, + "qps_1t_max": 280.6730955796045, + "median_ms": 3.587900020647794, + "p95_ms": 4.5391900290269405, + "p99_ms": 6.010548011399791, + "mean_ms": 3.689011071145422, + "knob": 8 + }, + { + "nq": 800, + "recall_queries": 1000, + "recall": 0.8673, + "qps_1t_runs": [ + 270.4215785601032, + 263.2281178825422, + 263.3481377813385 + ], + "qps_1t": 263.3481377813385, + "qps_1t_min": 263.2281178825422, + "qps_1t_max": 270.4215785601032, + "median_ms": 3.683699993416667, + "p95_ms": 4.553804980241692, + "p99_ms": 6.2035610264865575, + "mean_ms": 3.7647236239232975, + "knob": 16 + }, + { + "nq": 766, + "recall_queries": 1000, + "recall": 0.9486, + "qps_1t_runs": [ + 251.3068283345626, + 246.7051894232069, + 247.05439241022614 + ], + "qps_1t": 247.05439241022614, + "qps_1t_min": 246.7051894232069, + "qps_1t_max": 251.3068283345626, + "median_ms": 3.8989500026218593, + "p95_ms": 4.889725011889825, + "p99_ms": 7.055723987868997, + "mean_ms": 4.02677071339441, + "knob": 32 + }, + { + "nq": 731, + "recall_queries": 1000, + "recall": 0.9868000000000001, + "qps_1t_runs": [ + 243.86317977526355, + 238.109422861669, + 234.43039803763216 + ], + "qps_1t": 238.109422861669, + "qps_1t_min": 234.43039803763216, + "qps_1t_max": 243.86317977526355, + "median_ms": 4.064999986439943, + "p95_ms": 5.078340030740948, + "p99_ms": 6.776079996488985, + "mean_ms": 4.188689421444054, + "knob": 64 + }, + { + "nq": 624, + "recall_queries": 1000, + "recall": 0.9976999999999999, + "qps_1t_runs": [ + 213.42368631321537, + 215.44596330083124, + 202.54623994690402 + ], + "qps_1t": 213.42368631321537, + "qps_1t_min": 202.54623994690402, + "qps_1t_max": 215.44596330083124, + "median_ms": 4.58559999242425, + "p95_ms": 5.782204965362324, + "p99_ms": 8.624416001257483, + "mean_ms": 4.754731624136464, + "knob": 128 + }, + { + "nq": 550, + "recall_queries": 1000, + "recall": 0.9999, + "qps_1t_runs": [ + 186.37544032037258, + 188.95889145778986, + 185.44762167997942 + ], + "qps_1t": 186.37544032037258, + "qps_1t_min": 185.44762167997942, + "qps_1t_max": 188.95889145778986, + "median_ms": 5.207700014580041, + "p95_ms": 6.414425006369129, + "p99_ms": 8.255622963770293, + "mean_ms": 5.350009453885088, + "knob": 256 + }, + { + "nq": 300, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 92.33298018754584, + 92.32238435926035, + 90.62980007498007 + ], + "qps_1t": 92.32238435926035, + "qps_1t_min": 90.62980007498007, + "qps_1t_max": 92.33298018754584, + "median_ms": 10.567049990640953, + "p95_ms": 12.906670029042282, + "p99_ms": 17.296480967779644, + "mean_ms": 10.89862466602224, + "knob": 1265 + } + ], + "mutation": { + "ref_knob": 64, + "recall_before": 0.9868000000000001, + "insert_10k": { + "seconds": 0.039654400025028735, + "per_s": 252178.8248892502, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.14905529998941347, + "per_s": 67089.1944178452, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.12040869996417314, + "per_s": 83050.47727427867, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 272.796875, + "peak_mb": 371.64453125 + }, + "first_query_after_ms": 12.059500033501536, + "after": { + "nq": 168, + "recall_queries": 1000, + "recall": 0.9886000000000001, + "qps_1t_runs": [ + 57.838612370683805, + 60.023188248452946, + 58.993988151833705 + ], + "qps_1t": 58.993988151833705, + "qps_1t_min": 57.838612370683805, + "qps_1t_max": 60.023188248452946, + "median_ms": 16.647149983327836, + "p95_ms": 19.76132999407127, + "p99_ms": 26.17706401797472, + "mean_ms": 16.966865278109687 + }, + "recall_after": 0.9886000000000001 + }, + "saveload": { + "save_s": 1.400010660290718e-06, + "disk_mb": 109.67045783996582, + "load_s": 0.00807640003040433, + "first_query_after_load_ms": 18.645100004505366, + "time_to_ready_s": 0.026721500034909695 + }, + "load_before_pct": 8.309375000000001, + "load_contaminated": false + }, + "lancedb_ivfflat|text384|1000000": { + "mem_baseline": { + "rss_mb": 37.3125, + "peak_mb": 37.3125 + }, + "version": "lancedb 0.41.0", + "build": { + "ingest_s": 2.0603686998365447, + "index_s": 10.740709300036542, + "total_s": 12.801077999873087, + "first_query_ms": 586.2553999759257, + "ready_s": 13.387333399849012 + }, + "mem_after_build": { + "rss_mb": 278.03125, + "peak_mb": 3117.9375 + }, + "mem_after_first_query": { + "rss_mb": 1773.515625, + "peak_mb": 3117.9375 + }, + "sweep": [ + { + "nq": 688, + "recall_queries": 1000, + "recall": 0.21470000000000003, + "qps_1t_runs": [ + 225.31000264479792, + 228.49204847797822, + 231.2099721997163 + ], + "qps_1t": 228.49204847797822, + "qps_1t_min": 225.31000264479792, + "qps_1t_max": 231.2099721997163, + "median_ms": 4.262500006007031, + "p95_ms": 5.053334953845479, + "p99_ms": 5.759006004664102, + "mean_ms": 4.379974079386931, + "knob": 1 + }, + { + "nq": 703, + "recall_queries": 1000, + "recall": 0.31039999999999995, + "qps_1t_runs": [ + 228.53095520185963, + 229.49241275427826, + 222.8880822844825 + ], + "qps_1t": 228.53095520185963, + "qps_1t_min": 222.8880822844825, + "qps_1t_max": 229.49241275427826, + "median_ms": 4.291099961847067, + "p95_ms": 5.032539984676987, + "p99_ms": 6.111732022836811, + "mean_ms": 4.406591323140307, + "knob": 2 + }, + { + "nq": 728, + "recall_queries": 1000, + "recall": 0.41129999999999994, + "qps_1t_runs": [ + 234.35203613647982, + 239.50827115438165, + 242.6207628877579 + ], + "qps_1t": 239.50827115438165, + "qps_1t_min": 234.35203613647982, + "qps_1t_max": 242.6207628877579, + "median_ms": 4.12694999249652, + "p95_ms": 4.658775025745854, + "p99_ms": 5.246869000839074, + "mean_ms": 4.1879882316204, + "knob": 4 + }, + { + "nq": 630, + "recall_queries": 1000, + "recall": 0.5227, + "qps_1t_runs": [ + 222.50830435735676, + 225.0725536951612, + 234.27616476960497 + ], + "qps_1t": 225.0725536951612, + "qps_1t_min": 222.50830435735676, + "qps_1t_max": 234.27616476960497, + "median_ms": 4.310999996960163, + "p95_ms": 4.950819979421794, + "p99_ms": 5.880025012302205, + "mean_ms": 4.401897566107215, + "knob": 8 + }, + { + "nq": 689, + "recall_queries": 1000, + "recall": 0.6346, + "qps_1t_runs": [ + 220.4087638911061, + 211.2752155297306, + 210.9917968788648 + ], + "qps_1t": 211.2752155297306, + "qps_1t_min": 210.9917968788648, + "qps_1t_max": 220.4087638911061, + "median_ms": 4.570600052829832, + "p95_ms": 5.352519970620051, + "p99_ms": 6.3183380104601365, + "mean_ms": 4.669902756751389, + "knob": 16 + }, + { + "nq": 593, + "recall_queries": 1000, + "recall": 0.7313, + "qps_1t_runs": [ + 192.08556304150832, + 184.5083262103106, + 184.9112253169848 + ], + "qps_1t": 184.9112253169848, + "qps_1t_min": 184.5083262103106, + "qps_1t_max": 192.08556304150832, + "median_ms": 5.15889999223873, + "p95_ms": 6.382949993712827, + "p99_ms": 8.698084019124511, + "mean_ms": 5.344607813117325, + "knob": 32 + }, + { + "nq": 525, + "recall_queries": 1000, + "recall": 0.815, + "qps_1t_runs": [ + 164.5230823711992, + 162.30969726152478, + 158.68173142225018 + ], + "qps_1t": 162.30969726152478, + "qps_1t_min": 158.68173142225018, + "qps_1t_max": 164.5230823711992, + "median_ms": 5.95159997465089, + "p95_ms": 7.128359994385391, + "p99_ms": 9.413218001136554, + "mean_ms": 6.180386158992492, + "knob": 64 + }, + { + "nq": 389, + "recall_queries": 1000, + "recall": 0.8876000000000001, + "qps_1t_runs": [ + 134.58084127274407, + 133.98940608014908, + 127.46982619661769 + ], + "qps_1t": 133.98940608014908, + "qps_1t_min": 127.46982619661769, + "qps_1t_max": 134.58084127274407, + "median_ms": 7.287700020242482, + "p95_ms": 8.524770010262728, + "p99_ms": 10.185072009917329, + "mean_ms": 7.57958286110872, + "knob": 128 + }, + { + "nq": 299, + "recall_queries": 1000, + "recall": 0.9378, + "qps_1t_runs": [ + 96.74604859295353, + 94.9675308270553, + 95.42913893748785 + ], + "qps_1t": 95.42913893748785, + "qps_1t_min": 94.9675308270553, + "qps_1t_max": 96.74604859295353, + "median_ms": 10.187000036239624, + "p95_ms": 11.666379997041075, + "p99_ms": 13.311188004445283, + "mean_ms": 10.44841125780621, + "knob": 256 + }, + { + "nq": 33, + "recall_queries": 33, + "recall": 1.0, + "qps_1t_runs": [ + 11.258384809990291, + 11.495695541636517, + 11.43679443504118 + ], + "qps_1t": 11.43679443504118, + "qps_1t_min": 11.258384809990291, + "qps_1t_max": 11.495695541636517, + "median_ms": 87.44259999366477, + "p95_ms": 90.38233000319451, + "p99_ms": 92.83751600421965, + "mean_ms": 87.74961918919357, + "knob": 4000 + } + ], + "mutation": { + "ref_knob": 4000, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.05524190003052354, + "per_s": 181022.01398711064, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.6025168999913149, + "per_s": 16597.04483001912, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.35617240000283346, + "per_s": 28076.291144177503, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 1792.80859375, + "peak_mb": 3117.9375 + }, + "first_query_after_ms": 108.42829995090142, + "after": { + "nq": 30, + "recall_queries": 30, + "recall": 1.0, + "qps_1t_runs": [ + 8.02267657690412, + 7.867899850251763, + 7.969609011448342 + ], + "qps_1t": 7.969609011448342, + "qps_1t_min": 7.867899850251763, + "qps_1t_max": 8.02267657690412, + "median_ms": 123.97595003130846, + "p95_ms": 134.71618500188924, + "p99_ms": 143.3858879870968, + "mean_ms": 125.74068999836325 + }, + "recall_after": 1.0 + }, + "load_before_pct": 8.137499999999992, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 25.306510100199375, + "mem": { + "rss_mb": 214.53515625, + "peak_mb": 3118.65625 + }, + "subsets": { + "f10": [ + { + "knob": 2, + "nq": 82, + "recall": 0.22073170731707317, + "qps_1t": 27.01546220061804, + "qps_1t_runs": [ + 25.754254973711564, + 27.385659116378317, + 27.01546220061804 + ], + "median_ms": 37.50959996250458, + "p95_ms": 43.866074993275106, + "p99_ms": 47.37958000041547, + "mean_ms": 37.45327885920462 + }, + { + "knob": 8, + "nq": 80, + "recall": 0.45875, + "qps_1t": 26.275066574724107, + "qps_1t_runs": [ + 25.07087614678705, + 26.275066574724107, + 26.49751448280967 + ], + "median_ms": 38.82885002531111, + "p95_ms": 45.00488999765365, + "p99_ms": 49.018052007886574, + "mean_ms": 38.56173417031338 + }, + { + "knob": 32, + "nq": 72, + "recall": 0.6847222222222222, + "qps_1t": 14.408384814559135, + "qps_1t_runs": [ + 14.408384814559135 + ], + "median_ms": 46.742299979086965, + "p95_ms": 223.27411498408765, + "p99_ms": 236.85879700351518, + "mean_ms": 69.40403194878147 + }, + { + "knob": 64, + "nq": 80, + "recall": 0.785, + "qps_1t": 26.082502933587808, + "qps_1t_runs": [ + 22.607663417416827, + 26.45873015012876, + 26.082502933587808 + ], + "median_ms": 39.41040000063367, + "p95_ms": 50.009849999332786, + "p99_ms": 56.919133985065834, + "mean_ms": 40.12245875031416 + }, + { + "knob": 4000, + "nq": 30, + "recall": 1.0, + "qps_1t": 8.726117706820212, + "qps_1t_runs": [ + 8.726117706820212, + 9.161802843296455, + 8.506619794499755 + ], + "median_ms": 113.79905001376756, + "p95_ms": 125.15862001455389, + "p99_ms": 132.09589600854088, + "mean_ms": 113.76761000202451 + } + ], + "f01": [ + { + "knob": 2, + "nq": 101, + "recall": 0.12673267326732673, + "qps_1t": 30.090289338055868, + "qps_1t_runs": [ + 29.935552314106573, + 30.090289338055868, + 30.23800271517329 + ], + "median_ms": 33.27429998898879, + "p95_ms": 39.68294995720498, + "p99_ms": 40.48900998313911, + "mean_ms": 33.23645874774913 + }, + { + "knob": 8, + "nq": 88, + "recall": 0.3125, + "qps_1t": 30.311280312820497, + "qps_1t_runs": [ + 30.221107580814248, + 30.311280312820497, + 30.720493673177607 + ], + "median_ms": 32.935950002865866, + "p95_ms": 39.694530025008135, + "p99_ms": 42.30855902133044, + "mean_ms": 32.87734469830122 + }, + { + "knob": 32, + "nq": 94, + "recall": 0.5457446808510639, + "qps_1t": 28.900017373680924, + "qps_1t_runs": [ + 28.606340480035, + 28.900017373680924, + 29.398921532514578 + ], + "median_ms": 34.50405001058243, + "p95_ms": 40.28495498932898, + "p99_ms": 42.53876401053276, + "mean_ms": 34.52473120777934 + }, + { + "knob": 64, + "nq": 88, + "recall": 0.6875, + "qps_1t": 29.43461293085843, + "qps_1t_runs": [ + 28.7293776505382, + 29.658033438730197, + 29.43461293085843 + ], + "median_ms": 34.2530999914743, + "p95_ms": 41.02235503378324, + "p99_ms": 42.667963021667674, + "mean_ms": 34.16628712169988 + }, + { + "knob": 4000, + "nq": 40, + "recall": 1.0, + "qps_1t": 13.069751302577982, + "qps_1t_runs": [ + 13.069751302577982, + 13.096706407126604, + 12.932709046552187 + ], + "median_ms": 77.22004997776821, + "p95_ms": 85.2545350062428, + "p99_ms": 87.36987999291159, + "mean_ms": 76.7303149982278 + } + ] + } + } + }, + "lancedb_ivfflat|text384|100000": { + "mem_baseline": { + "rss_mb": 37.25390625, + "peak_mb": 37.25390625 + }, + "version": "lancedb 0.41.0", + "build": { + "ingest_s": 0.23378770000999793, + "index_s": 1.660392599995248, + "total_s": 1.894180300005246, + "first_query_ms": 89.4075000542216, + "ready_s": 1.9835878000594676, + "total_s_runs": [ + 1.894180300005246, + 1.7727591000148095, + 1.800324699957855 + ] + }, + "mem_after_build": { + "rss_mb": 236.921875, + "peak_mb": 524.640625 + }, + "mem_after_first_query": { + "rss_mb": 375.859375, + "peak_mb": 524.640625 + }, + "sweep": [ + { + "nq": 924, + "recall_queries": 1000, + "recall": 0.2415, + "qps_1t_runs": [ + 287.87555996423646, + 198.49865843185904 + ], + "qps_1t": 243.18710919804775, + "qps_1t_min": 198.49865843185904, + "qps_1t_max": 287.87555996423646, + "median_ms": 3.284099977463484, + "p95_ms": 4.56145500356797, + "p99_ms": 12.32441597850993, + "mean_ms": 4.255770292044647, + "knob": 1 + }, + { + "nq": 912, + "recall_queries": 1000, + "recall": 0.3584, + "qps_1t_runs": [ + 298.3341766201878, + 300.05830418776014, + 301.058636997805 + ], + "qps_1t": 300.05830418776014, + "qps_1t_min": 298.3341766201878, + "qps_1t_max": 301.058636997805, + "median_ms": 3.2394500158261508, + "p95_ms": 4.0596499893581495, + "p99_ms": 5.787509997026071, + "mean_ms": 3.335414509761606, + "knob": 2 + }, + { + "nq": 933, + "recall_queries": 1000, + "recall": 0.4898, + "qps_1t_runs": [ + 282.24981717587826, + 284.96744500341686, + 300.0064825058929 + ], + "qps_1t": 284.96744500341686, + "qps_1t_min": 282.24981717587826, + "qps_1t_max": 300.0064825058929, + "median_ms": 3.2953000045381486, + "p95_ms": 4.062800022074952, + "p99_ms": 6.324091986753043, + "mean_ms": 3.461798249337777, + "knob": 4 + }, + { + "nq": 843, + "recall_queries": 1000, + "recall": 0.616, + "qps_1t_runs": [ + 285.8630448873732, + 299.1199698851492, + 294.169723509095 + ], + "qps_1t": 294.169723509095, + "qps_1t_min": 285.8630448873732, + "qps_1t_max": 299.1199698851492, + "median_ms": 3.3267999533563852, + "p95_ms": 3.944119997322559, + "p99_ms": 5.098276003263885, + "mean_ms": 3.413572321596569, + "knob": 8 + }, + { + "nq": 903, + "recall_queries": 1000, + "recall": 0.7339, + "qps_1t_runs": [ + 277.7639783679517, + 287.77295722676075, + 285.5820983997752 + ], + "qps_1t": 285.5820983997752, + "qps_1t_min": 277.7639783679517, + "qps_1t_max": 287.77295722676075, + "median_ms": 3.4129000268876553, + "p95_ms": 4.028900002595037, + "p99_ms": 5.674276025965811, + "mean_ms": 3.525586859057508, + "knob": 16 + }, + { + "nq": 786, + "recall_queries": 1000, + "recall": 0.8351000000000001, + "qps_1t_runs": [ + 264.6997165339855, + 266.3056113975814, + 277.96179883554464 + ], + "qps_1t": 266.3056113975814, + "qps_1t_min": 264.6997165339855, + "qps_1t_max": 277.96179883554464, + "median_ms": 3.5851499997079372, + "p95_ms": 4.306825014646166, + "p99_ms": 5.363074974738984, + "mean_ms": 3.710188888498377, + "knob": 32 + }, + { + "nq": 790, + "recall_queries": 1000, + "recall": 0.9089999999999999, + "qps_1t_runs": [ + 255.77491176843697, + 256.77042674030986, + 251.2046295598156 + ], + "qps_1t": 255.77491176843697, + "qps_1t_min": 251.2046295598156, + "qps_1t_max": 256.77042674030986, + "median_ms": 3.863599995383993, + "p95_ms": 4.430014971876517, + "p99_ms": 5.061538993613793, + "mean_ms": 3.928345148004631, + "knob": 64 + }, + { + "nq": 652, + "recall_queries": 1000, + "recall": 0.9556, + "qps_1t_runs": [ + 218.04052600866663, + 220.33545525956112, + 212.13579422114074 + ], + "qps_1t": 218.04052600866663, + "qps_1t_min": 212.13579422114074, + "qps_1t_max": 220.33545525956112, + "median_ms": 4.518050001934171, + "p95_ms": 5.162800007383339, + "p99_ms": 6.057295014034028, + "mean_ms": 4.612933078046092, + "knob": 128 + }, + { + "nq": 500, + "recall_queries": 1000, + "recall": 0.9828, + "qps_1t_runs": [ + 177.89543227601519, + 177.232167352002, + 168.22906608561948 + ], + "qps_1t": 177.232167352002, + "qps_1t_min": 168.22906608561948, + "qps_1t_max": 177.89543227601519, + "median_ms": 5.611349974060431, + "p95_ms": 6.478485025581904, + "p99_ms": 7.699603011715226, + "mean_ms": 5.7359574668419855, + "knob": 256 + }, + { + "nq": 210, + "recall_queries": 1000, + "recall": 0.9999, + "qps_1t_runs": [ + 68.98402397160703, + 70.27806452146946, + 71.13250053012611 + ], + "qps_1t": 70.27806452146946, + "qps_1t_min": 68.98402397160703, + "qps_1t_max": 71.13250053012611, + "median_ms": 14.031900005647913, + "p95_ms": 15.610205003758892, + "p99_ms": 17.41453402442858, + "mean_ms": 14.2611907964151, + "knob": 1265 + } + ], + "mutation": { + "ref_knob": 128, + "recall_before": 0.9556, + "insert_10k": { + "seconds": 0.055947599990759045, + "per_s": 178738.67693434065, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.2072751999949105, + "per_s": 48245.03848142731, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.13450540002668276, + "per_s": 74346.4574508996, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 412.0625, + "peak_mb": 711.171875 + }, + "first_query_after_ms": 15.473199950065464, + "after": { + "nq": 97, + "recall_queries": 97, + "recall": 0.9690721649484536, + "qps_1t_runs": [ + 31.710146511916395, + 31.651814891227882, + 31.762547646825023 + ], + "qps_1t": 31.710146511916395, + "qps_1t_min": 31.651814891227882, + "qps_1t_max": 31.762547646825023, + "median_ms": 31.298300018534064, + "p95_ms": 33.82444998715073, + "p99_ms": 39.01119997608477, + "mean_ms": 31.53767766346163 + }, + "recall_after": 0.9690721649484536 + }, + "saveload": { + "save_s": 1.100008375942707e-06, + "disk_mb": 325.56323432922363, + "load_s": 0.008934999990742654, + "first_query_after_load_ms": 23.7724999897182, + "time_to_ready_s": 0.03270749998046085 + }, + "load_before_pct": 5.153124999999999, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 1.9351456998847425, + "mem": { + "rss_mb": 229.95703125, + "peak_mb": 517.33203125 + }, + "subsets": { + "f10": [ + { + "knob": 2, + "nq": 477, + "recall": 0.23542976939203356, + "qps_1t": 156.29965769121, + "qps_1t_runs": [ + 156.29965769121, + 160.60083160604404, + 156.20551976064894 + ], + "median_ms": 6.506800011266023, + "p95_ms": 7.435499981511384, + "p99_ms": 8.532039989950142, + "mean_ms": 6.342135638891505 + }, + { + "knob": 8, + "nq": 447, + "recall": 0.48456375838926175, + "qps_1t": 158.70324330305317, + "qps_1t_runs": [ + 150.53946166259996, + 160.732543442604, + 158.70324330305317 + ], + "median_ms": 6.571299978531897, + "p95_ms": 7.721600006334484, + "p99_ms": 8.746199985034762, + "mean_ms": 6.388453467157694 + }, + { + "knob": 32, + "nq": 414, + "recall": 0.7618357487922706, + "qps_1t": 149.94613886216828, + "qps_1t_runs": [ + 147.76194720490454, + 154.06413372590055, + 149.94613886216828 + ], + "median_ms": 6.751450011506677, + "p95_ms": 7.889020003494807, + "p99_ms": 9.078151986468571, + "mean_ms": 6.6425023355724475 + }, + { + "knob": 64, + "nq": 437, + "recall": 0.862929061784897, + "qps_1t": 143.90365977640565, + "qps_1t_runs": [ + 146.35898860294085, + 140.1701781747389, + 143.90365977640565 + ], + "median_ms": 7.028100022580475, + "p95_ms": 8.533449989045039, + "p99_ms": 10.071499965852132, + "mean_ms": 6.971931272844516 + }, + { + "knob": 1265, + "nq": 215, + "recall": 1.0, + "qps_1t": 73.06069393557968, + "qps_1t_runs": [ + 73.06069393557968, + 74.25336684646925, + 71.87058989742357 + ], + "median_ms": 13.177199987694621, + "p95_ms": 16.332939988933493, + "p99_ms": 23.4779840195551, + "mean_ms": 13.689516743468193 + } + ], + "f01": [ + { + "knob": 2, + "nq": 671, + "recall": 0.08614008941877795, + "qps_1t": 172.60083425363078, + "qps_1t_runs": [ + 169.5663395174669, + 172.7977238919334, + 172.60083425363078 + ], + "median_ms": 5.83899999037385, + "p95_ms": 7.211340020876378, + "p99_ms": 9.6610160032287, + "mean_ms": 5.826074963017291 + }, + { + "knob": 8, + "nq": 420, + "recall": 0.25476190476190474, + "qps_1t": 155.09228343919395, + "qps_1t_runs": [ + 155.09228343919395, + 157.13828141319163, + 152.37187500238755 + ], + "median_ms": 6.434749986510724, + "p95_ms": 8.244884968735276, + "p99_ms": 10.475211981683973, + "mean_ms": 6.458162222406827 + }, + { + "knob": 32, + "nq": 447, + "recall": 0.5715883668903803, + "qps_1t": 152.91142323495305, + "qps_1t_runs": [ + 152.91142323495305, + 151.34390334826222, + 156.0606412691341 + ], + "median_ms": 6.626600050367415, + "p95_ms": 7.770700030960143, + "p99_ms": 10.051760007627282, + "mean_ms": 6.518322445409154 + }, + { + "knob": 64, + "nq": 451, + "recall": 0.7354767184035477, + "qps_1t": 154.99109996789653, + "qps_1t_runs": [ + 158.87435512382578, + 154.99109996789653, + 149.35043820710197 + ], + "median_ms": 6.613400008063763, + "p95_ms": 7.6994200120680025, + "p99_ms": 9.462751988321546, + "mean_ms": 6.480642350897171 + }, + { + "knob": 1265, + "nq": 319, + "recall": 1.0, + "qps_1t": 97.76160253340882, + "qps_1t_runs": [ + 95.7891049534074, + 97.76160253340882, + 98.02989111359042 + ], + "median_ms": 10.096799989696592, + "p95_ms": 12.153680028859526, + "p99_ms": 16.170268012210638, + "mean_ms": 10.28984524580906 + } + ] + } + } + }, + "numpy_bf|glove|1000000": { + "mem_baseline": { + "rss_mb": 35.03125, + "peak_mb": 35.03125 + }, + "version": "numpy 2.5.5", + "build": { + "ingest_s": 0.1222563999472186, + "index_s": 1.600012183189392e-06, + "total_s": 0.12225799995940179, + "first_query_ms": 26.943999982904643, + "ready_s": 0.14920199994230643 + }, + "mem_after_build": { + "rss_mb": 527.4140625, + "peak_mb": 535.046875 + }, + "mem_after_first_query": { + "rss_mb": 527.6328125, + "peak_mb": 542.87890625 + }, + "sweep": [ + { + "nq": 121, + "recall_queries": 121, + "recall": 1.0, + "qps_1t_runs": [ + 36.63080399859705, + 34.43434249912364, + 33.251017371399705 + ], + "qps_1t": 34.43434249912364, + "qps_1t_min": 33.251017371399705, + "qps_1t_max": 36.63080399859705, + "median_ms": 28.821000014431775, + "p95_ms": 33.2050500146579, + "p99_ms": 35.98132596584037, + "mean_ms": 28.804823692558337, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 59.1624292069753, + 69.20876433070468, + 72.77909382401293 + ], + "qps_batch": 69.20876433070468, + "qps_batch_min": 59.1624292069753, + "qps_batch_max": 72.77909382401293, + "threads": 32 + } + } + ], + "loop_allthreads": { + "nq": 114, + "threads": 32, + "qps": 39.3697456266803, + "median_ms": 24.71554998192005, + "p95_ms": 29.404569999314845, + "p99_ms": 37.80972401495081, + "mean_ms": 25.274040059687582 + }, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.0035168000031262636, + "per_s": 2843494.0830045748, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.15098780003609136, + "per_s": 66230.51662193668, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.023817700042854995, + "per_s": 419855.82075545, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 511.70703125, + "peak_mb": 23324.5078125 + }, + "first_query_after_ms": 68.85280000278726, + "after": { + "nq": 101, + "recall_queries": 101, + "recall": 1.0, + "qps_1t_runs": [ + 31.983570703500607, + 33.1606212116313, + 32.34572753361104 + ], + "qps_1t": 32.34572753361104, + "qps_1t_min": 31.983570703500607, + "qps_1t_max": 33.1606212116313, + "median_ms": 30.406599980778992, + "p95_ms": 34.835900011239566, + "p99_ms": 44.31360597838653, + "mean_ms": 30.779429044798878 + }, + "recall_after": 1.0 + }, + "load_before_pct": 11.518749999999995, + "load_contaminated": false + }, + "numpy_bf|glove|100000": { + "mem_baseline": { + "rss_mb": 35.04296875, + "peak_mb": 35.04296875 + }, + "version": "numpy 2.5.5", + "build": { + "ingest_s": 0.011630199966020882, + "index_s": 2.100015990436077e-06, + "total_s": 0.011632299982011318, + "first_query_ms": 2.3740999749861658, + "ready_s": 0.014006399956997484, + "total_s_runs": [ + 0.011632299982011318, + 0.011848699941765517, + 0.011565599997993559 + ] + }, + "mem_after_build": { + "rss_mb": 86.27734375, + "peak_mb": 96.0390625 + }, + "mem_after_first_query": { + "rss_mb": 86.12890625, + "peak_mb": 96.0390625 + }, + "sweep": [ + { + "nq": 1607, + "recall_queries": 1607, + "recall": 1.0, + "qps_1t_runs": [ + 478.8508532046889, + 528.2931208601073, + 537.2645987466597 + ], + "qps_1t": 528.2931208601073, + "qps_1t_min": 478.8508532046889, + "qps_1t_max": 537.2645987466597, + "median_ms": 1.816099975258112, + "p95_ms": 2.5314000085927546, + "p99_ms": 3.69705999037251, + "mean_ms": 1.9475005815455486, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 1687.3489242631317, + 1732.7897802056275, + 1745.4020437750667 + ], + "qps_batch": 1732.7897802056275, + "qps_batch_min": 1687.3489242631317, + "qps_batch_max": 1745.4020437750667, + "threads": 32 + } + } + ], + "loop_allthreads": { + "nq": 2000, + "threads": 32, + "qps": 1017.9281058967854, + "median_ms": 0.9073999826796353, + "p95_ms": 1.29273000638932, + "p99_ms": 1.6822410031454647, + "mean_ms": 0.95744280005844 + }, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.0011317000025883317, + "per_s": 8836264.007359562, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.006155200011562556, + "per_s": 1624642.5755808062, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.0021666999673470855, + "per_s": 4615313.680114203, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 105.03515625, + "peak_mb": 3144.34375 + }, + "first_query_after_ms": 2.9347999952733517, + "after": { + "nq": 1677, + "recall_queries": 1677, + "recall": 1.0, + "qps_1t_runs": [ + 486.3989849075593, + 484.53277203690453, + 548.716773449077 + ], + "qps_1t": 486.3989849075593, + "qps_1t_min": 484.53277203690453, + "qps_1t_max": 548.716773449077, + "median_ms": 1.908699981868267, + "p95_ms": 2.492800005711615, + "p99_ms": 3.2826900191139394, + "mean_ms": 1.9807343469810312 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 0.024996799998916686, + "disk_mb": 42.8011474609375, + "load_s": 0.01716030004899949, + "first_query_after_load_ms": 2.3825999815016985, + "time_to_ready_s": 0.019542900030501187 + }, + "load_before_pct": 4.5249999999999995, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 0.012173999974038452, + "mem": { + "rss_mb": 85.78515625, + "peak_mb": 95.54296875 + }, + "subsets": { + "f10": [ + { + "knob": null, + "nq": 1659, + "recall": 1.0, + "qps_1t": 552.0243702974032, + "qps_1t_runs": [ + 528.7185643145853, + 554.6350455179166, + 552.0243702974032 + ], + "median_ms": 1.7308000242337584, + "p95_ms": 2.2066600271500647, + "p99_ms": 3.2523079845122935, + "mean_ms": 1.8352890102372774 + } + ], + "f01": [ + { + "knob": null, + "nq": 1751, + "recall": 1.0, + "qps_1t": 623.9321931919695, + "qps_1t_runs": [ + 574.6952337204558, + 623.9321931919695, + 648.6188142113433 + ], + "median_ms": 1.518300035968423, + "p95_ms": 2.1711000008508563, + "p99_ms": 3.3314000186510375, + "mean_ms": 1.6281761847415686 + } + ] + } + } + }, + "numpy_bf|sift|1000000": { + "mem_baseline": { + "rss_mb": 35.53125, + "peak_mb": 35.53125 + }, + "version": "numpy 2.5.5", + "build": { + "ingest_s": 0.13666780025232583, + "index_s": 2.100015990436077e-06, + "total_s": 0.13666990026831627, + "first_query_ms": 29.749699984677136, + "ready_s": 0.1664196002529934 + }, + "mem_after_build": { + "rss_mb": 634.73046875, + "peak_mb": 644.5 + }, + "mem_after_first_query": { + "rss_mb": 634.94921875, + "peak_mb": 650.1953125 + }, + "sweep": [ + { + "nq": 110, + "recall_queries": 110, + "recall": 0.9981818181818182, + "qps_1t_runs": [ + 28.12480880033291, + 26.871127609015293, + 26.663160701938843 + ], + "qps_1t": 26.871127609015293, + "qps_1t_min": 26.663160701938843, + "qps_1t_max": 28.12480880033291, + "median_ms": 37.35805000178516, + "p95_ms": 41.83341998432297, + "p99_ms": 46.82670901587697, + "mean_ms": 36.75846424355934, + "knob": null, + "batch": { + "recall": 0.99925, + "qps_batch_runs": [ + 48.02491809377273, + 67.38187995185218, + 68.93683072445786 + ], + "qps_batch": 67.38187995185218, + "qps_batch_min": 48.02491809377273, + "qps_batch_max": 68.93683072445786, + "threads": 32 + } + } + ], + "loop_allthreads": { + "nq": 98, + "threads": 32, + "qps": 31.04580650055689, + "median_ms": 32.03544998541474, + "p95_ms": 39.03979499591514, + "p99_ms": 40.99516700254752, + "mean_ms": 32.78946632865443 + }, + "load_before_pct": 11.124999999999995, + "load_contaminated": false + }, + "numpy_bf|sift|100000": { + "mem_baseline": { + "rss_mb": 35.43359375, + "peak_mb": 35.43359375 + }, + "version": "numpy 2.5.5", + "build": { + "ingest_s": 0.015353899914771318, + "index_s": 2.00001522898674e-06, + "total_s": 0.015355899930000305, + "first_query_ms": 3.0196000006981194, + "ready_s": 0.018375499930698425, + "total_s_runs": [ + 0.015355899930000305, + 0.017753299965988845, + 0.015054100076667964 + ] + }, + "mem_after_build": { + "rss_mb": 97.43359375, + "peak_mb": 109.296875 + }, + "mem_after_first_query": { + "rss_mb": 97.27734375, + "peak_mb": 109.296875 + }, + "sweep": [ + { + "nq": 842, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 282.134764185857, + 297.3362380747439, + 266.56199198650955 + ], + "qps_1t": 282.134764185857, + "qps_1t_min": 266.56199198650955, + "qps_1t_max": 297.3362380747439, + "median_ms": 3.343299962580204, + "p95_ms": 4.870125005254522, + "p99_ms": 6.689900023047812, + "mean_ms": 3.553024624666621, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 1543.385293765154, + 1542.770258454746, + 1529.395751041704 + ], + "qps_batch": 1542.770258454746, + "qps_batch_min": 1529.395751041704, + "qps_batch_max": 1543.385293765154, + "threads": 32 + } + } + ], + "loop_allthreads": { + "nq": 830, + "threads": 32, + "qps": 427.9685161615656, + "median_ms": 2.1524999756366014, + "p95_ms": 4.043454964994452, + "p99_ms": 5.958811984746724, + "mean_ms": 2.3629968275267434 + }, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.0016330999787896872, + "per_s": 6123323.819654408, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.006400400015991181, + "per_s": 1562402.3459495252, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.002769000013358891, + "per_s": 3611412.044693225, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 120.86328125, + "peak_mb": 3156.97265625 + }, + "first_query_after_ms": 3.0731000006198883, + "after": { + "nq": 856, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 286.2934374403579, + 261.30254062037386, + 274.0523431570449 + ], + "qps_1t": 274.0523431570449, + "qps_1t_min": 261.30254062037386, + "qps_1t_max": 286.2934374403579, + "median_ms": 3.4618499921634793, + "p95_ms": 4.889120016014205, + "p99_ms": 6.9433079991722435, + "mean_ms": 3.656279751098921 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 0.030838600010611117, + "disk_mb": 54.5504150390625, + "load_s": 0.025650800031144172, + "first_query_after_load_ms": 4.709500004537404, + "time_to_ready_s": 0.030360300035681576 + }, + "load_before_pct": 12.63750000000001, + "load_contaminated": false + }, + "numpy_bf|text384|1000000": { + "mem_baseline": { + "rss_mb": 37.40625, + "peak_mb": 37.40625 + }, + "version": "numpy 2.5.5", + "build": { + "ingest_s": 0.29768730007344857, + "index_s": 2.100015990436077e-06, + "total_s": 0.297689400089439, + "first_query_ms": 70.26820001192391, + "ready_s": 0.3679576001013629 + }, + "mem_after_build": { + "rss_mb": 1613.10546875, + "peak_mb": 1642.40625 + }, + "mem_after_first_query": { + "rss_mb": 1613.32421875, + "peak_mb": 1642.40625 + }, + "sweep": [ + { + "nq": 42, + "recall_queries": 42, + "recall": 1.0, + "qps_1t_runs": [ + 14.425729730066768, + 14.061476305081415, + 14.181329092005516 + ], + "qps_1t": 14.181329092005516, + "qps_1t_min": 14.061476305081415, + "qps_1t_max": 14.425729730066768, + "median_ms": 69.28120003431104, + "p95_ms": 77.11289999133442, + "p99_ms": 83.95727499737404, + "mean_ms": 70.31737460610678, + "knob": null, + "batch": { + "recall": 0.99995, + "qps_batch_runs": [ + 112.22853446621616, + 142.438233031093, + 145.18666707814978 + ], + "qps_batch": 142.438233031093, + "qps_batch_min": 112.22853446621616, + "qps_batch_max": 145.18666707814978, + "threads": 32 + } + } + ], + "loop_allthreads": { + "nq": 77, + "threads": 32, + "qps": 27.952094173654178, + "median_ms": 35.07129999343306, + "p95_ms": 39.29985000286251, + "p99_ms": 41.545520001091056, + "mean_ms": 35.583145023440515 + }, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.0038690000073984265, + "per_s": 2584647.190715347, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.08686679997481406, + "per_s": 115118.7795901239, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.014911399979609996, + "per_s": 670627.8427025031, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 1631.4375, + "peak_mb": 24393.6484375 + }, + "first_query_after_ms": 73.75340000726283, + "after": { + "nq": 40, + "recall_queries": 40, + "recall": 1.0, + "qps_1t_runs": [ + 14.081808691877153, + 14.042345887366727, + 14.327790915131303 + ], + "qps_1t": 14.081808691877153, + "qps_1t_min": 14.042345887366727, + "qps_1t_max": 14.327790915131303, + "median_ms": 68.76729999203235, + "p95_ms": 81.97205997421406, + "p99_ms": 88.8461139757419, + "mean_ms": 70.67373582879857 + }, + "recall_after": 1.0 + }, + "load_before_pct": 9.706249999999995, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 0.3003055000444874, + "mem": { + "rss_mb": 1614.1484375, + "peak_mb": 1643.4453125 + }, + "subsets": { + "f10": [ + { + "knob": null, + "nq": 41, + "recall": 1.0, + "qps_1t": 12.887030124605031, + "qps_1t_runs": [ + 12.887030124605031, + 12.594177956844689, + 13.63567323591825 + ], + "median_ms": 74.57850000355393, + "p95_ms": 93.15719002042897, + "p99_ms": 100.24626799160615, + "mean_ms": 76.77873740033284 + } + ], + "f01": [ + { + "knob": null, + "nq": 43, + "recall": 1.0, + "qps_1t": 13.966856066036692, + "qps_1t_runs": [ + 13.814059608766382, + 14.032562660715573, + 13.966856066036692 + ], + "median_ms": 69.72070003394037, + "p95_ms": 83.76633997540921, + "p99_ms": 93.87779998360202, + "mean_ms": 71.75030387475971 + } + ] + } + } + }, + "numpy_bf|text384|100000": { + "mem_baseline": { + "rss_mb": 37.5234375, + "peak_mb": 37.5234375 + }, + "version": "numpy 2.5.5", + "build": { + "ingest_s": 0.02955939999083057, + "index_s": 1.9999570213258266e-06, + "total_s": 0.029561399947851896, + "first_query_ms": 7.355800014920533, + "ready_s": 0.03691719996277243, + "total_s_runs": [ + 0.029561399947851896, + 0.029899799963459373, + 0.029745299951173365 + ] + }, + "mem_after_build": { + "rss_mb": 197.0625, + "peak_mb": 228.515625 + }, + "mem_after_first_query": { + "rss_mb": 196.8828125, + "peak_mb": 228.515625 + }, + "sweep": [ + { + "nq": 448, + "recall_queries": 1000, + "recall": 0.9999, + "qps_1t_runs": [ + 143.21663674351632, + 145.64401038113556, + 144.26549799463044 + ], + "qps_1t": 144.26549799463044, + "qps_1t_min": 143.21663674351632, + "qps_1t_max": 145.64401038113556, + "median_ms": 6.752399989636615, + "p95_ms": 7.655209978111083, + "p99_ms": 10.1361830078531, + "mean_ms": 6.9267165908535215, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 1552.8583269017806, + 1563.6238546363275, + 1543.6353291333107 + ], + "qps_batch": 1552.8583269017806, + "qps_batch_min": 1543.6353291333107, + "qps_batch_max": 1563.6238546363275, + "threads": 32 + } + } + ], + "loop_allthreads": { + "nq": 582, + "threads": 32, + "qps": 216.94630995397839, + "median_ms": 4.542099981335923, + "p95_ms": 5.10002500959672, + "p99_ms": 6.665445011458358, + "mean_ms": 4.615391064897971 + }, + "mutation": { + "ref_knob": null, + "recall_before": 0.9999, + "insert_10k": { + "seconds": 0.0031792999943718314, + "per_s": 3145346.4654806214, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.0074541999492794275, + "per_s": 1341525.592021001, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.0020655999542213976, + "per_s": 4841208.472900734, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 251.890625, + "peak_mb": 3258.85546875 + }, + "first_query_after_ms": 7.095300010405481, + "after": { + "nq": 421, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 132.06745992427594, + 134.18411802835428, + 134.63707648157887 + ], + "qps_1t": 134.18411802835428, + "qps_1t_min": 132.06745992427594, + "qps_1t_max": 134.63707648157887, + "median_ms": 7.252100040204823, + "p95_ms": 8.472410024842246, + "p99_ms": 12.065381995634917, + "mean_ms": 7.483903087840747 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 0.11037170002236962, + "disk_mb": 161.9722900390625, + "load_s": 0.0681961000082083, + "first_query_after_load_ms": 9.157199994660914, + "time_to_ready_s": 0.07735330000286922 + }, + "load_before_pct": 5.184375000000003, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 0.029936499951872975, + "mem": { + "rss_mb": 196.30078125, + "peak_mb": 227.7265625 + }, + "subsets": { + "f10": [ + { + "knob": null, + "nq": 415, + "recall": 1.0, + "qps_1t": 141.22187135997544, + "qps_1t_runs": [ + 138.51827369505224, + 141.22187135997544, + 141.79592363184014 + ], + "median_ms": 6.85239996528253, + "p95_ms": 8.506780001334846, + "p99_ms": 13.748928005807068, + "mean_ms": 7.117569718807262 + } + ], + "f01": [ + { + "knob": null, + "nq": 441, + "recall": 1.0, + "qps_1t": 146.07631563739315, + "qps_1t_runs": [ + 142.94661528395815, + 146.2720236747061, + 146.07631563739315 + ], + "median_ms": 6.655600038357079, + "p95_ms": 7.938750035827979, + "p99_ms": 12.056917967274762, + "mean_ms": 6.892644217465206 + } + ] + } + } + }, + "numpy_bf|text384|10000": { + "mem_baseline": { + "rss_mb": 37.44140625, + "peak_mb": 37.44140625 + }, + "version": "numpy 2.5.5", + "build": { + "ingest_s": 0.0029279000009410083, + "index_s": 1.200009137392044e-06, + "total_s": 0.0029291000100784004, + "first_query_ms": 0.8420000085607171, + "ready_s": 0.0037711000186391175 + }, + "mem_after_build": { + "rss_mb": 54.22265625, + "peak_mb": 68.875 + }, + "mem_after_first_query": { + "rss_mb": 54.44140625, + "peak_mb": 68.875 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 1.0, + "qps_1t_runs": [ + 3364.167051224355, + 3126.0887583616272, + 3518.481704649937 + ], + "qps_1t": 3364.167051224355, + "qps_1t_min": 3126.0887583616272, + "qps_1t_max": 3518.481704649937, + "median_ms": 0.3035000408999622, + "p95_ms": 0.3500199876725674, + "p99_ms": 0.4527010128367692, + "mean_ms": 0.30045081649829325, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 14581.978423595207, + 14956.87186371766, + 14689.982611883277 + ], + "qps_batch": 14689.982611883277, + "qps_batch_min": 14581.978423595207, + "qps_batch_max": 14956.87186371766, + "threads": 32 + } + } + ], + "loop_allthreads": { + "nq": 2000, + "threads": 32, + "qps": 6581.068879255676, + "median_ms": 0.15139998868107796, + "p95_ms": 0.1896050438517705, + "p99_ms": 0.23000400455202916, + "mean_ms": 0.15553231681891097 + }, + "load_before_pct": 3.9468749999999986, + "load_contaminated": false + }, + "qdrant_local|text384|100000": { + "mem_baseline": { + "rss_mb": 37.4296875, + "peak_mb": 37.4296875 + }, + "version": "qdrant-client 1.20.0 (local mode)", + "build": { + "ingest_s": 284.33503050002037, + "index_s": 2.300017513334751e-06, + "total_s": 284.3350328000379, + "first_query_ms": 126.82280002627522, + "ready_s": 284.46185560006415 + }, + "mem_after_build": { + "rss_mb": 431.7421875, + "peak_mb": 603.9453125 + }, + "mem_after_first_query": { + "rss_mb": 436.0, + "peak_mb": 726.62109375 + }, + "sweep": [ + { + "nq": 30, + "recall_queries": 30, + "recall": 1.0, + "qps_1t_runs": [ + 7.250915113595278, + 7.141317168949704, + 7.241052014743276 + ], + "qps_1t": 7.241052014743276, + "qps_1t_min": 7.141317168949704, + "qps_1t_max": 7.250915113595278, + "median_ms": 136.17230000090785, + "p95_ms": 152.33883499749936, + "p99_ms": 157.8155379614327, + "mean_ms": 138.68176555649066, + "knob": null + } + ], + "loop_allthreads": { + "nq": 30, + "threads": 32, + "qps": 7.554045037370992, + "median_ms": 132.17250001616776, + "p95_ms": 139.86650497536175, + "p99_ms": 144.6756109787384, + "mean_ms": 132.34423555420815 + }, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 25.664128399977926, + "per_s": 389.6489233590571, + "batch": 1000 + }, + "update_10k": { + "seconds": 26.403800099971704, + "per_s": 378.73336270299654, + "batch": 1000 + }, + "delete_10k": { + "seconds": 31.76803919998929, + "per_s": 314.7817823142, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 466.4453125, + "peak_mb": 730.24609375 + }, + "first_query_after_ms": 136.2393000163138, + "after": { + "nq": 30, + "recall_queries": 30, + "recall": 1.0, + "qps_1t_runs": [ + 6.493147683826058 + ], + "qps_1t": 6.493147683826058, + "qps_1t_min": 6.493147683826058, + "qps_1t_max": 6.493147683826058, + "median_ms": 153.4129999927245, + "p95_ms": 163.65067503065802, + "p99_ms": 169.4600670260843, + "mean_ms": 154.0085100006157 + }, + "recall_after": 1.0 + }, + "load_before_pct": 5.025000000000002, + "load_contaminated": false + }, + "qdrant_local|text384|10000": { + "mem_baseline": { + "rss_mb": 37.36328125, + "peak_mb": 37.36328125 + }, + "version": "qdrant-client 1.20.0 (local mode)", + "build": { + "ingest_s": 33.30397010000888, + "index_s": 1.300009898841381e-06, + "total_s": 33.30397140001878, + "first_query_ms": 39.89770001498982, + "ready_s": 33.34386910003377 + }, + "mem_after_build": { + "rss_mb": 147.74609375, + "peak_mb": 198.59375 + }, + "mem_after_first_query": { + "rss_mb": 148.328125, + "peak_mb": 198.59375 + }, + "sweep": [ + { + "nq": 231, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 72.99020291564169, + 73.43907393726488, + 72.42674590726222 + ], + "qps_1t": 72.99020291564169, + "qps_1t_min": 72.42674590726222, + "qps_1t_max": 73.43907393726488, + "median_ms": 13.216899998951703, + "p95_ms": 15.94205998117104, + "p99_ms": 22.059979962650694, + "mean_ms": 13.7080841274928, + "knob": null + } + ], + "loop_allthreads": { + "nq": 210, + "threads": 32, + "qps": 79.2029707126397, + "median_ms": 12.475349998567253, + "p95_ms": 14.004170024418272, + "p99_ms": 15.141900998423829, + "mean_ms": 12.645105237606913 + }, + "load_before_pct": 3.565624999999999, + "load_contaminated": false + }, + "rust_default_exact|glove|1000000": { + "build": { + "first_query_ms": 663.5181, + "index_s": 0.0, + "ingest_s": 0.7856887000000002, + "ready_s": 1.4492068000000002, + "total_s": 0.7856887000000002 + }, + "loop_allthreads": { + "mean_ms": 14.153318148148127, + "median_ms": 13.2813, + "nq": 180, + "p95_ms": 19.4111, + "p99_ms": 24.898400000000002, + "qps": 70.89076929717626, + "qps_runs": [ + 67.65923262101195, + 73.67135552101732, + 70.89076929717626 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 1359.53125, + "rss_mb": 1351.91015625 + }, + "mem_after_first_query": { + "peak_mb": 1359.53125, + "rss_mb": 1351.91015625 + }, + "mem_baseline": { + "peak_mb": 6.58984375, + "rss_mb": 6.58984375 + }, + "sweep": [ + { + "knob": null, + "mean_ms": 22.32282999999998, + "median_ms": 22.503700000000002, + "nq": 170, + "p95_ms": 24.6228, + "p99_ms": 25.876199999999997, + "qps_1t": 43.495671374749, + "qps_1t_max": 48.56086450221941, + "qps_1t_min": 42.76248683481381, + "qps_1t_runs": [ + 48.56086450221941, + 43.495671374749, + 42.76248683481381 + ], + "recall": 1.0 + } + ], + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "load_before_pct": 8.693749999999996, + "load_contaminated": false + }, + "rust_default_ivf|glove|1000000": { + "build": { + "cache_s": 0.6635181, + "first_query_ms": 0.0, + "index_s": 5.6935397, + "ingest_s": 0.7856887000000002, + "ready_s": 7.142746499999999, + "total_s": 7.142746499999999 + }, + "mem_after_build": { + "peak_mb": 1428.23046875, + "rss_mb": 1362.37890625 + }, + "mem_baseline": { + "peak_mb": 6.58984375, + "rss_mb": 6.58984375 + }, + "mutation": { + "after": { + "mean_ms": 6.04390242165242, + "median_ms": 6.0344999999999995, + "nq": 468, + "p95_ms": 7.765000000000001, + "p99_ms": 8.5421, + "qps_1t": 164.65186263475061, + "qps_1t_max": 167.33848778783036, + "qps_1t_min": 164.40945598878318, + "qps_1t_runs": [ + 164.40945598878318, + 167.33848778783036, + 164.65186263475061 + ], + "recall": 0.9540598290598291 + }, + "delete_10k": { + "per_s": 303254.5275900969, + "seconds": 0.0329756 + }, + "first_query_after_ms": 745.865, + "index_stats_after": { + "builds": 90140, + "churn": 30000, + "pending": 0, + "trained_n": 1000000 + }, + "insert_10k": { + "per_s": 64013.9294310442, + "seconds": 0.156216 + }, + "mem_after": { + "peak_mb": 1748.73046875, + "rss_mb": 1378.09765625 + }, + "recall_after": 0.9540598290598291, + "recall_before": 0.9536842105263158, + "ref_knob": 256, + "update_10k": { + "per_s": 238537.66867594898, + "seconds": 0.0419221 + } + }, + "saveload": { + "disk_mb": 1215.590708732605, + "first_query_after_load_ms": 7592.225799999999, + "load_s": 10.7524962, + "note": "load = JSON parse + insert; first query includes the scan-cache build and the IVF index build (the index is never persisted)", + "save_s": 23.1342869, + "time_to_ready_s": 18.344721999999997 + }, + "sweep": [ + { + "allthreads": { + "median_ms": 0.1735, + "qps": 5473.557108768614, + "recall": 0.30875 + }, + "knob": 1, + "mean_ms": 0.09495421666666651, + "median_ms": 0.0922, + "nq": 2000, + "p95_ms": 0.134, + "p99_ms": 0.1669, + "qps_1t": 10525.25218504233, + "qps_1t_max": 10565.854328566365, + "qps_1t_min": 10503.25837332886, + "qps_1t_runs": [ + 10565.854328566365, + 10525.25218504233, + 10503.25837332886 + ], + "recall": 0.30875 + }, + { + "allthreads": { + "median_ms": 0.2337, + "qps": 4107.446698177634, + "recall": 0.42305 + }, + "knob": 2, + "mean_ms": 0.13864956666666656, + "median_ms": 0.13540000000000002, + "nq": 2000, + "p95_ms": 0.1972, + "p99_ms": 0.2407, + "qps_1t": 7216.137014236721, + "qps_1t_max": 7478.044461461132, + "qps_1t_min": 6961.577660575742, + "qps_1t_runs": [ + 6961.577660575742, + 7216.137014236721, + 7478.044461461132 + ], + "recall": 0.42305 + }, + { + "allthreads": { + "median_ms": 0.3635, + "qps": 2611.1378607678635, + "recall": 0.5377 + }, + "knob": 4, + "mean_ms": 0.19636315000000049, + "median_ms": 0.1918, + "nq": 2000, + "p95_ms": 0.2825, + "p99_ms": 0.3448, + "qps_1t": 5053.718500804303, + "qps_1t_max": 5225.215951643762, + "qps_1t_min": 5004.110877085523, + "qps_1t_runs": [ + 5004.110877085523, + 5225.215951643762, + 5053.718500804303 + ], + "recall": 0.5377 + }, + { + "allthreads": { + "median_ms": 0.589, + "qps": 1646.895412753984, + "recall": 0.6346 + }, + "knob": 8, + "mean_ms": 0.30542776666666693, + "median_ms": 0.2967, + "nq": 2000, + "p95_ms": 0.4406, + "p99_ms": 0.5218, + "qps_1t": 3277.4307433819245, + "qps_1t_max": 3328.959635199287, + "qps_1t_min": 3217.792073355365, + "qps_1t_runs": [ + 3217.792073355365, + 3277.4307433819245, + 3328.959635199287 + ], + "recall": 0.6346 + }, + { + "allthreads": { + "median_ms": 0.9058999999999999, + "qps": 1028.4823796284809, + "recall": 0.72835 + }, + "knob": 16, + "mean_ms": 0.5200717500000002, + "median_ms": 0.5075000000000001, + "nq": 2000, + "p95_ms": 0.7512000000000001, + "p99_ms": 0.8566999999999999, + "qps_1t": 1922.5306655656982, + "qps_1t_max": 1938.2230177550957, + "qps_1t_min": 1907.9199665732406, + "qps_1t_runs": [ + 1938.2230177550957, + 1907.9199665732406, + 1922.5306655656982 + ], + "recall": 0.72835 + }, + { + "allthreads": { + "median_ms": 1.747, + "qps": 563.9185987796458, + "recall": 0.8031725888324873 + }, + "knob": 32, + "mean_ms": 0.8734791833333301, + "median_ms": 0.8537, + "nq": 2000, + "p95_ms": 1.2114, + "p99_ms": 1.39, + "qps_1t": 1167.9571425134307, + "qps_1t_max": 1174.0134471500257, + "qps_1t_min": 1095.9351491334842, + "qps_1t_runs": [ + 1174.0134471500257, + 1167.9571425134307, + 1095.9351491334842 + ], + "recall": 0.8013 + }, + { + "allthreads": { + "median_ms": 2.6516, + "qps": 363.3975807228954, + "recall": 0.8654895666131621 + }, + "knob": 64, + "mean_ms": 1.6018584961142626, + "median_ms": 1.5775, + "nq": 1587, + "p95_ms": 2.1917999999999997, + "p99_ms": 2.4734000000000003, + "qps_1t": 644.0744923489667, + "qps_1t_max": 644.3474246606822, + "qps_1t_min": 587.8886927778146, + "qps_1t_runs": [ + 587.8886927778146, + 644.3474246606822, + 644.0744923489667 + ], + "recall": 0.8640201638311279 + }, + { + "allthreads": { + "median_ms": 3.3215, + "qps": 272.8295808389948, + "recall": 0.9198317307692307 + }, + "knob": 128, + "mean_ms": 3.1683282663771237, + "median_ms": 3.1863, + "nq": 921, + "p95_ms": 4.2356, + "p99_ms": 4.6311, + "qps_1t": 313.6266915452204, + "qps_1t_max": 321.3772505174752, + "qps_1t_min": 312.02486929681385, + "qps_1t_runs": [ + 313.6266915452204, + 321.3772505174752, + 312.02486929681385 + ], + "recall": 0.9204125950054289 + }, + { + "allthreads": { + "median_ms": 5.0913, + "qps": 166.43642960571202, + "recall": 0.9556492411467117 + }, + "knob": 256, + "mean_ms": 5.9860054035087655, + "median_ms": 6.014, + "nq": 475, + "p95_ms": 7.6666, + "p99_ms": 8.415799999999999, + "qps_1t": 168.77358877387718, + "qps_1t_max": 170.49829580671812, + "qps_1t_min": 162.13348828971255, + "qps_1t_runs": [ + 170.49829580671812, + 168.77358877387718, + 162.13348828971255 + ], + "recall": 0.9536842105263158 + } + ], + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "load_before_pct": 8.693749999999996, + "load_contaminated": false + }, + "rust_default_exact|glove|100000": { + "build": { + "first_query_ms": 40.8248, + "index_s": 0.0, + "ingest_s": 0.053535200000000005, + "ready_s": 0.09436, + "total_s": 0.053535200000000005 + }, + "loop_allthreads": { + "mean_ms": 0.6032887499999995, + "median_ms": 0.5783, + "nq": 2000, + "p95_ms": 0.7379, + "p99_ms": 0.9842, + "qps": 1669.8171942579288, + "qps_runs": [ + 1669.8171942579288, + 1693.179601655797, + 1611.8802018396402 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 127.77734375, + "rss_mb": 127.77734375 + }, + "mem_after_first_query": { + "peak_mb": 127.77734375, + "rss_mb": 127.77734375 + }, + "mem_baseline": { + "peak_mb": 6.5703125, + "rss_mb": 6.5703125 + }, + "sweep": [ + { + "knob": null, + "mean_ms": 1.4547141333333304, + "median_ms": 1.4215, + "nq": 2000, + "p95_ms": 1.7947, + "p99_ms": 2.1357, + "qps_1t": 699.5854990897174, + "qps_1t_max": 704.9802519169391, + "qps_1t_min": 659.5239780164837, + "qps_1t_runs": [ + 704.9802519169391, + 659.5239780164837, + 699.5854990897174 + ], + "recall": 1.0 + } + ], + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "load_before_pct": 5.503125000000001, + "load_contaminated": false + }, + "rust_default_ivf|glove|100000": { + "build": { + "cache_s": 0.0408248, + "first_query_ms": 0.0, + "index_s": 0.2123974, + "ingest_s": 0.053535200000000005, + "ready_s": 0.30675739999999996, + "total_s": 0.30675739999999996 + }, + "mem_after_build": { + "peak_mb": 150.78515625, + "rss_mb": 131.36328125 + }, + "mem_baseline": { + "peak_mb": 6.5703125, + "rss_mb": 6.5703125 + }, + "mutation": { + "after": { + "mean_ms": 0.7657480499999967, + "median_ms": 0.7532, + "nq": 2000, + "p95_ms": 0.9539000000000001, + "p99_ms": 1.1238000000000001, + "qps_1t": 1314.1168277456336, + "qps_1t_max": 1333.9626079609309, + "qps_1t_min": 1271.2447240165836, + "qps_1t_runs": [ + 1314.1168277456336, + 1333.9626079609309, + 1271.2447240165836 + ], + "recall": 0.9743 + }, + "delete_10k": { + "per_s": 1127217.8010235138, + "seconds": 0.0088714 + }, + "first_query_after_ms": 83.6679, + "index_stats_after": { + "builds": 115046, + "churn": 30000, + "pending": 0, + "trained_n": 100000 + }, + "insert_10k": { + "per_s": 689579.080929001, + "seconds": 0.0145016 + }, + "mem_after": { + "peak_mb": 172.7265625, + "rss_mb": 145.47265625 + }, + "recall_after": 0.9743, + "recall_before": 0.974, + "ref_knob": 256, + "update_10k": { + "per_s": 702020.414753661, + "seconds": 0.0142446 + } + }, + "saveload": { + "disk_mb": 121.47405910491943, + "first_query_after_load_ms": 237.37449999999998, + "load_s": 0.4137497, + "note": "load = JSON parse + insert; first query includes the scan-cache build and the IVF index build (the index is never persisted)", + "save_s": 0.7539028, + "time_to_ready_s": 0.6511241999999999 + }, + "sweep": [ + { + "allthreads": { + "median_ms": 0.018199999999999997, + "qps": 53680.17886235612, + "recall": 0.28505 + }, + "knob": 1, + "mean_ms": 0.019093250000000127, + "median_ms": 0.018199999999999997, + "nq": 2000, + "p95_ms": 0.0248, + "p99_ms": 0.0368, + "qps_1t": 52956.284587073395, + "qps_1t_max": 55463.578454618284, + "qps_1t_min": 49100.47922067721, + "qps_1t_runs": [ + 49100.47922067721, + 55463.578454618284, + 52956.284587073395 + ], + "recall": 0.28505 + }, + { + "allthreads": { + "median_ms": 0.0229, + "qps": 42249.89120652995, + "recall": 0.4016 + }, + "knob": 2, + "mean_ms": 0.024383399999999965, + "median_ms": 0.0237, + "nq": 2000, + "p95_ms": 0.0312, + "p99_ms": 0.0383, + "qps_1t": 40791.43543021715, + "qps_1t_max": 41482.75976504172, + "qps_1t_min": 40768.31975408543, + "qps_1t_runs": [ + 41482.75976504172, + 40768.31975408543, + 40791.43543021715 + ], + "recall": 0.4016 + }, + { + "allthreads": { + "median_ms": 0.030000000000000002, + "qps": 32287.59203981703, + "recall": 0.52085 + }, + "knob": 4, + "mean_ms": 0.03110996666666654, + "median_ms": 0.030199999999999998, + "nq": 2000, + "p95_ms": 0.0406, + "p99_ms": 0.050300000000000004, + "qps_1t": 32167.7354396767, + "qps_1t_max": 32233.005953436226, + "qps_1t_min": 32032.032032032013, + "qps_1t_runs": [ + 32167.7354396767, + 32233.005953436226, + 32032.032032032013 + ], + "recall": 0.52085 + }, + { + "allthreads": { + "median_ms": 0.0449, + "qps": 21856.609711984496, + "recall": 0.63505 + }, + "knob": 8, + "mean_ms": 0.04629701666666656, + "median_ms": 0.0451, + "nq": 2000, + "p95_ms": 0.061900000000000004, + "p99_ms": 0.076, + "qps_1t": 21634.90662374299, + "qps_1t_max": 22195.415093104213, + "qps_1t_min": 21001.741044332583, + "qps_1t_runs": [ + 22195.415093104213, + 21001.741044332583, + 21634.90662374299 + ], + "recall": 0.63505 + }, + { + "allthreads": { + "median_ms": 0.0687, + "qps": 14291.34916052615, + "recall": 0.73085 + }, + "knob": 16, + "mean_ms": 0.07220015000000007, + "median_ms": 0.0704, + "nq": 2000, + "p95_ms": 0.09720000000000001, + "p99_ms": 0.1202, + "qps_1t": 13906.494124158584, + "qps_1t_max": 14166.814237648312, + "qps_1t_min": 13494.529992267668, + "qps_1t_runs": [ + 14166.814237648312, + 13906.494124158584, + 13494.529992267668 + ], + "recall": 0.73085 + }, + { + "allthreads": { + "median_ms": 0.118, + "qps": 8302.59298281447, + "recall": 0.8134 + }, + "knob": 32, + "mean_ms": 0.11708581666666681, + "median_ms": 0.1147, + "nq": 2000, + "p95_ms": 0.1538, + "p99_ms": 0.17980000000000002, + "qps_1t": 8541.744788040876, + "qps_1t_max": 8572.763473169398, + "qps_1t_min": 8507.970053647017, + "qps_1t_runs": [ + 8507.970053647017, + 8572.763473169398, + 8541.744788040876 + ], + "recall": 0.8134 + }, + { + "allthreads": { + "median_ms": 0.2125, + "qps": 4573.128207191982, + "recall": 0.88165 + }, + "knob": 64, + "mean_ms": 0.20716886666666656, + "median_ms": 0.205, + "nq": 2000, + "p95_ms": 0.26699999999999996, + "p99_ms": 0.3014, + "qps_1t": 4863.969368666501, + "qps_1t_max": 4874.662673342992, + "qps_1t_min": 4744.490223503442, + "qps_1t_runs": [ + 4874.662673342992, + 4863.969368666501, + 4744.490223503442 + ], + "recall": 0.88165 + }, + { + "allthreads": { + "median_ms": 0.37579999999999997, + "qps": 2638.3887676402624, + "recall": 0.9355 + }, + "knob": 128, + "mean_ms": 0.38995593333333417, + "median_ms": 0.38430000000000003, + "nq": 2000, + "p95_ms": 0.507, + "p99_ms": 0.5695, + "qps_1t": 2594.9842069261135, + "qps_1t_max": 2618.5636523725793, + "qps_1t_min": 2483.730016063527, + "qps_1t_runs": [ + 2618.5636523725793, + 2594.9842069261135, + 2483.730016063527 + ], + "recall": 0.9355 + }, + { + "allthreads": { + "median_ms": 0.48950000000000005, + "qps": 2009.531812245019, + "recall": 0.974 + }, + "knob": 256, + "mean_ms": 0.7395931666666631, + "median_ms": 0.7328, + "nq": 2000, + "p95_ms": 0.9119, + "p99_ms": 1.0108000000000001, + "qps_1t": 1350.3063237410697, + "qps_1t_max": 1358.6289206039664, + "qps_1t_min": 1347.3990112651325, + "qps_1t_runs": [ + 1350.3063237410697, + 1347.3990112651325, + 1358.6289206039664 + ], + "recall": 0.974 + }, + { + "allthreads": { + "median_ms": 0.5703, + "qps": 1703.3327493740856, + "recall": 1.0 + }, + "knob": 1265, + "mean_ms": 1.4638045163464755, + "median_ms": 1.4307, + "nq": 1978, + "p95_ms": 1.7236, + "p99_ms": 2.1279, + "qps_1t": 683.7856683847615, + "qps_1t_max": 693.3252686074593, + "qps_1t_min": 672.6566663455278, + "qps_1t_runs": [ + 672.6566663455278, + 683.7856683847615, + 693.3252686074593 + ], + "recall": 1.0 + } + ], + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "load_before_pct": 5.503125000000001, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 2.1289925, + "mem": { + "peak_mb": 149.01953125, + "rss_mb": 130.48828125 + }, + "subsets": { + "f01": [ + { + "knob": 2, + "mean_ms": 0.032043500000000086, + "median_ms": 0.0347, + "nq": 2000, + "p95_ms": 0.039799999999999995, + "p99_ms": 0.0443, + "qps_1t": 31466.677575114914, + "qps_1t_max": 32388.768870506432, + "qps_1t_min": 29872.191827267052, + "qps_1t_runs": [ + 29872.191827267052, + 31466.677575114914, + 32388.768870506432 + ], + "recall": 0.3727 + }, + { + "knob": 8, + "mean_ms": 0.021686716666666755, + "median_ms": 0.0255, + "nq": 2000, + "p95_ms": 0.0294, + "p99_ms": 0.032799999999999996, + "qps_1t": 45984.411284574424, + "qps_1t_max": 46678.69421020812, + "qps_1t_min": 45681.70813043051, + "qps_1t_runs": [ + 45984.411284574424, + 45681.70813043051, + 46678.69421020812 + ], + "recall": 0.3727 + }, + { + "knob": 32, + "mean_ms": 0.016466833333333136, + "median_ms": 0.016399999999999998, + "nq": 2000, + "p95_ms": 0.018199999999999997, + "p99_ms": 0.0209, + "qps_1t": 60890.21494245865, + "qps_1t_max": 61049.688341340894, + "qps_1t_min": 60250.40066516447, + "qps_1t_runs": [ + 60250.40066516447, + 60890.21494245865, + 61049.688341340894 + ], + "recall": 0.4615 + }, + { + "knob": 64, + "mean_ms": 0.02137643333333331, + "median_ms": 0.0212, + "nq": 2000, + "p95_ms": 0.024, + "p99_ms": 0.026699999999999998, + "qps_1t": 46963.34980181477, + "qps_1t_max": 47052.0701734575, + "qps_1t_min": 46332.654559944094, + "qps_1t_runs": [ + 47052.0701734575, + 46332.654559944094, + 46963.34980181477 + ], + "recall": 0.5994 + }, + { + "knob": 1265, + "mean_ms": 0.05480208333333471, + "median_ms": 0.0531, + "nq": 2000, + "p95_ms": 0.0562, + "p99_ms": 0.0608, + "qps_1t": 18088.579775158898, + "qps_1t_max": 18702.96788045809, + "qps_1t_min": 17967.740718314293, + "qps_1t_runs": [ + 18088.579775158898, + 18702.96788045809, + 17967.740718314293 + ], + "recall": 1.0 + }, + { + "knob": "exact", + "mean_ms": 0.053449250000001405, + "median_ms": 0.053, + "nq": 2000, + "p95_ms": 0.055099999999999996, + "p99_ms": 0.0579, + "qps_1t": 18699.435557537647, + "qps_1t_max": 18756.570661159716, + "qps_1t_min": 18672.201107074772, + "qps_1t_runs": [ + 18699.435557537647, + 18672.201107074772, + 18756.570661159716 + ], + "recall": 1.0 + } + ], + "f10": [ + { + "knob": 2, + "mean_ms": 0.014949683333333517, + "median_ms": 0.0138, + "nq": 2000, + "p95_ms": 0.025, + "p99_ms": 0.0285, + "qps_1t": 67053.5657409922, + "qps_1t_max": 70436.42408362214, + "qps_1t_min": 63538.86036700058, + "qps_1t_runs": [ + 63538.86036700058, + 67053.5657409922, + 70436.42408362214 + ], + "recall": 0.266 + }, + { + "knob": 8, + "mean_ms": 0.01640416666666663, + "median_ms": 0.0162, + "nq": 2000, + "p95_ms": 0.018600000000000002, + "p99_ms": 0.020999999999999998, + "qps_1t": 62135.922330096895, + "qps_1t_max": 62710.66865250436, + "qps_1t_min": 58232.63939438042, + "qps_1t_runs": [ + 62135.922330096895, + 58232.63939438042, + 62710.66865250436 + ], + "recall": 0.47545 + }, + { + "knob": 32, + "mean_ms": 0.023570316666666476, + "median_ms": 0.0234, + "nq": 2000, + "p95_ms": 0.0269, + "p99_ms": 0.0294, + "qps_1t": 43076.15433324578, + "qps_1t_max": 43110.788259207904, + "qps_1t_min": 41151.92467551716, + "qps_1t_runs": [ + 43110.788259207904, + 43076.15433324578, + 41151.92467551716 + ], + "recall": 0.7007 + }, + { + "knob": 64, + "mean_ms": 0.03252056666666669, + "median_ms": 0.032, + "nq": 2000, + "p95_ms": 0.0387, + "p99_ms": 0.0458, + "qps_1t": 30830.227187944096, + "qps_1t_max": 31068.39551931605, + "qps_1t_min": 30359.19481343518, + "qps_1t_runs": [ + 30830.227187944096, + 31068.39551931605, + 30359.19481343518 + ], + "recall": 0.797 + }, + { + "knob": 1265, + "mean_ms": 0.12008315000000111, + "median_ms": 0.1185, + "nq": 2000, + "p95_ms": 0.12769999999999998, + "p99_ms": 0.1515, + "qps_1t": 8310.507140387728, + "qps_1t_max": 8379.844964488313, + "qps_1t_min": 8292.843524823385, + "qps_1t_runs": [ + 8292.843524823385, + 8379.844964488313, + 8310.507140387728 + ], + "recall": 1.0 + }, + { + "knob": "exact", + "mean_ms": 0.11908006666666739, + "median_ms": 0.1182, + "nq": 2000, + "p95_ms": 0.127, + "p99_ms": 0.1471, + "qps_1t": 8377.475648772612, + "qps_1t_max": 8441.443605036664, + "qps_1t_min": 8374.553583953015, + "qps_1t_runs": [ + 8377.475648772612, + 8441.443605036664, + 8374.553583953015 + ], + "recall": 1.0 + } + ] + } + } + }, + "rust_default_exact|sift|1000000": { + "build": { + "first_query_ms": 840.6548, + "index_s": 0.0, + "ingest_s": 0.8266650999999998, + "ready_s": 1.6673198999999999, + "total_s": 0.8266650999999998 + }, + "loop_allthreads": { + "mean_ms": 16.292859021406727, + "median_ms": 16.191, + "nq": 218, + "p95_ms": 18.8101, + "p99_ms": 23.867099999999997, + "qps": 61.30097714038762, + "qps_runs": [ + 61.30097714038762, + 60.87214991637757, + 61.96650886430912 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 1574.12890625, + "rss_mb": 1566.5078125 + }, + "mem_after_first_query": { + "peak_mb": 1574.12890625, + "rss_mb": 1566.5078125 + }, + "mem_baseline": { + "peak_mb": 6.99609375, + "rss_mb": 6.99609375 + }, + "sweep": [ + { + "knob": null, + "mean_ms": 30.004278730158727, + "median_ms": 30.2601, + "nq": 105, + "p95_ms": 31.850499999999997, + "p99_ms": 32.3378, + "qps_1t": 33.386588651854005, + "qps_1t_max": 33.850588595646414, + "qps_1t_min": 32.76635984559368, + "qps_1t_runs": [ + 33.850588595646414, + 32.76635984559368, + 33.386588651854005 + ], + "recall": 0.9990476190476191 + } + ], + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "load_before_pct": 7.599999999999997, + "load_contaminated": false + }, + "rust_default_ivf|sift|1000000": { + "build": { + "cache_s": 0.8406548, + "first_query_ms": 0.0, + "index_s": 6.3692511, + "ingest_s": 0.8266650999999998, + "ready_s": 8.036570999999999, + "total_s": 8.036570999999999 + }, + "mem_after_build": { + "peak_mb": 1661.16796875, + "rss_mb": 1577.37890625 + }, + "mem_baseline": { + "peak_mb": 6.99609375, + "rss_mb": 6.99609375 + }, + "saveload": { + "disk_mb": 1285.1193675994873, + "first_query_after_load_ms": 8389.5167, + "load_s": 11.2341163, + "note": "load = JSON parse + insert; first query includes the scan-cache build and the IVF index build (the index is never persisted)", + "save_s": 22.7026778, + "time_to_ready_s": 19.623632999999998 + }, + "sweep": [ + { + "allthreads": { + "median_ms": 0.1628, + "qps": 6063.900780029878, + "recall": 0.2633 + }, + "knob": 1, + "mean_ms": 0.09371651666666696, + "median_ms": 0.0919, + "nq": 2000, + "p95_ms": 0.1224, + "p99_ms": 0.1487, + "qps_1t": 10886.972539789167, + "qps_1t_max": 10960.039149259854, + "qps_1t_min": 10198.243658477139, + "qps_1t_runs": [ + 10198.243658477139, + 10886.972539789167, + 10960.039149259854 + ], + "recall": 0.2633 + }, + { + "allthreads": { + "median_ms": 0.21350000000000002, + "qps": 4494.123484132142, + "recall": 0.39335 + }, + "knob": 2, + "mean_ms": 0.12690718333333284, + "median_ms": 0.12329999999999999, + "nq": 2000, + "p95_ms": 0.1676, + "p99_ms": 0.2053, + "qps_1t": 7936.621316812469, + "qps_1t_max": 8142.414079211031, + "qps_1t_min": 7580.946503913854, + "qps_1t_runs": [ + 8142.414079211031, + 7936.621316812469, + 7580.946503913854 + ], + "recall": 0.39335 + }, + { + "allthreads": { + "median_ms": 0.314, + "qps": 3000.2271171927664, + "recall": 0.53655 + }, + "knob": 4, + "mean_ms": 0.17329533333333352, + "median_ms": 0.1653, + "nq": 2000, + "p95_ms": 0.2448, + "p99_ms": 0.3079, + "qps_1t": 5762.821846396589, + "qps_1t_max": 5831.696084657568, + "qps_1t_min": 5718.102139027636, + "qps_1t_runs": [ + 5831.696084657568, + 5718.102139027636, + 5762.821846396589 + ], + "recall": 0.53655 + }, + { + "allthreads": { + "median_ms": 0.49229999999999996, + "qps": 1973.826274075255, + "recall": 0.68765 + }, + "knob": 8, + "mean_ms": 0.29107343333333235, + "median_ms": 0.2767, + "nq": 2000, + "p95_ms": 0.4417, + "p99_ms": 0.545, + "qps_1t": 3348.591749739646, + "qps_1t_max": 3667.8444115072025, + "qps_1t_min": 3311.8329140552855, + "qps_1t_runs": [ + 3667.8444115072025, + 3348.591749739646, + 3311.8329140552855 + ], + "recall": 0.68765 + }, + { + "allthreads": { + "median_ms": 0.7952, + "qps": 1223.8839983164276, + "recall": 0.82165 + }, + "knob": 16, + "mean_ms": 0.47877408333333255, + "median_ms": 0.4527, + "nq": 2000, + "p95_ms": 0.7337, + "p99_ms": 0.8621, + "qps_1t": 2089.486653874136, + "qps_1t_max": 2111.0962278827087, + "qps_1t_min": 2065.909751765446, + "qps_1t_runs": [ + 2065.909751765446, + 2111.0962278827087, + 2089.486653874136 + ], + "recall": 0.82165 + }, + { + "allthreads": { + "median_ms": 1.5816999999999999, + "qps": 600.1781508805259, + "recall": 0.9163 + }, + "knob": 32, + "mean_ms": 0.9342941666666669, + "median_ms": 0.8911, + "nq": 2000, + "p95_ms": 1.407, + "p99_ms": 1.6033, + "qps_1t": 1077.7035581299028, + "qps_1t_max": 1088.24579728995, + "qps_1t_min": 1045.9446893988809, + "qps_1t_runs": [ + 1077.7035581299028, + 1088.24579728995, + 1045.9446893988809 + ], + "recall": 0.9163 + }, + { + "allthreads": { + "median_ms": 2.5244999999999997, + "qps": 381.9570133889485, + "recall": 0.9689274447949526 + }, + "knob": 64, + "mean_ms": 1.8168356609573981, + "median_ms": 1.7577, + "nq": 1518, + "p95_ms": 2.6174000000000004, + "p99_ms": 2.8911000000000002, + "qps_1t": 546.4091219943318, + "qps_1t_max": 564.8487900161236, + "qps_1t_min": 540.5431197798689, + "qps_1t_runs": [ + 546.4091219943318, + 540.5431197798689, + 564.8487900161236 + ], + "recall": 0.9700922266139658 + }, + { + "allthreads": { + "median_ms": 3.0686, + "qps": 289.5903006952332, + "recall": 0.9923736075407027 + }, + "knob": 128, + "mean_ms": 3.3970247534516775, + "median_ms": 3.3392, + "nq": 1014, + "p95_ms": 4.5659, + "p99_ms": 5.0150999999999994, + "qps_1t": 293.9971096082981, + "qps_1t_max": 295.42758510499584, + "qps_1t_min": 293.7068017107755, + "qps_1t_runs": [ + 295.42758510499584, + 293.7068017107755, + 293.9971096082981 + ], + "recall": 0.9924063116370808 + }, + { + "allthreads": { + "median_ms": 3.9513, + "qps": 239.0854404798855, + "recall": 0.9989942528735632 + }, + "knob": 256, + "mean_ms": 6.336291766268254, + "median_ms": 6.3290999999999995, + "nq": 502, + "p95_ms": 7.865, + "p99_ms": 8.5034, + "qps_1t": 157.06441969014625, + "qps_1t_max": 161.9820876851634, + "qps_1t_min": 154.59441831001496, + "qps_1t_runs": [ + 154.59441831001496, + 157.06441969014625, + 161.9820876851634 + ], + "recall": 0.998804780876494 + } + ], + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "load_before_pct": 7.599999999999997, + "load_contaminated": false + }, + "rust_default_exact|sift|100000": { + "build": { + "first_query_ms": 50.649899999999995, + "index_s": 0.0, + "ingest_s": 0.0554142, + "ready_s": 0.1060641, + "total_s": 0.0554142 + }, + "loop_allthreads": { + "mean_ms": 3.08890448999512, + "median_ms": 2.8602, + "nq": 683, + "p95_ms": 4.7961, + "p99_ms": 6.5978, + "qps": 335.9577553044066, + "qps_runs": [ + 335.9577553044066, + 299.33044643799354, + 339.05668968280753 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 149.80078125, + "rss_mb": 149.80078125 + }, + "mem_after_first_query": { + "peak_mb": 149.80078125, + "rss_mb": 149.80078125 + }, + "mem_baseline": { + "peak_mb": 7.01171875, + "rss_mb": 7.01171875 + }, + "sweep": [ + { + "knob": null, + "mean_ms": 2.807940888166804, + "median_ms": 2.8586, + "nq": 1231, + "p95_ms": 3.3598999999999997, + "p99_ms": 3.8371, + "qps_1t": 343.6908042496044, + "qps_1t_max": 397.7487999521281, + "qps_1t_min": 333.32432554354506, + "qps_1t_runs": [ + 397.7487999521281, + 343.6908042496044, + 333.32432554354506 + ], + "recall": 1.0 + } + ], + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "load_before_pct": 10.334375, + "load_contaminated": false + }, + "rust_default_ivf|sift|100000": { + "build": { + "cache_s": 0.0506499, + "first_query_ms": 0.0, + "index_s": 0.5648412, + "ingest_s": 0.0554142, + "ready_s": 0.6709053, + "total_s": 0.6709053 + }, + "mem_after_build": { + "peak_mb": 178.62890625, + "rss_mb": 152.56640625 + }, + "mem_baseline": { + "peak_mb": 7.01171875, + "rss_mb": 7.01171875 + }, + "mutation": { + "after": { + "mean_ms": 0.3573184499999988, + "median_ms": 0.3478, + "nq": 2000, + "p95_ms": 0.4643, + "p99_ms": 0.5712999999999999, + "qps_1t": 2802.712577340857, + "qps_1t_max": 2807.7371126972325, + "qps_1t_min": 2785.519587843383, + "qps_1t_runs": [ + 2802.712577340857, + 2807.7371126972325, + 2785.519587843383 + ], + "recall": 0.98545 + }, + "delete_10k": { + "per_s": 463419.9465213382, + "seconds": 0.0215787 + }, + "first_query_after_ms": 260.6058, + "index_stats_after": { + "builds": 114242, + "churn": 30000, + "pending": 0, + "trained_n": 100000 + }, + "insert_10k": { + "per_s": 253924.40162714757, + "seconds": 0.0393818 + }, + "mem_after": { + "peak_mb": 206.8515625, + "rss_mb": 170.859375 + }, + "recall_after": 0.98545, + "recall_before": 0.98605, + "ref_knob": 64, + "update_10k": { + "per_s": 304479.81146610074, + "seconds": 0.0328429 + } + }, + "saveload": { + "disk_mb": 127.68917655944824, + "first_query_after_load_ms": 852.5371, + "load_s": 0.9036185, + "note": "load = JSON parse + insert; first query includes the scan-cache build and the IVF index build (the index is never persisted)", + "save_s": 1.6734498, + "time_to_ready_s": 1.7561556 + }, + "sweep": [ + { + "allthreads": { + "median_ms": 0.0582, + "qps": 16984.315833543515, + "recall": 0.2912 + }, + "knob": 1, + "mean_ms": 0.029398816666666765, + "median_ms": 0.0286, + "nq": 2000, + "p95_ms": 0.037700000000000004, + "p99_ms": 0.0449, + "qps_1t": 33756.41371860651, + "qps_1t_max": 34626.09872939535, + "qps_1t_min": 33678.53835143551, + "qps_1t_runs": [ + 34626.09872939535, + 33756.41371860651, + 33678.53835143551 + ], + "recall": 0.2912 + }, + { + "allthreads": { + "median_ms": 0.06949999999999999, + "qps": 14002.89579885121, + "recall": 0.43565 + }, + "knob": 2, + "mean_ms": 0.03880953333333346, + "median_ms": 0.0376, + "nq": 2000, + "p95_ms": 0.050899999999999994, + "p99_ms": 0.0599, + "qps_1t": 25888.76117100045, + "qps_1t_max": 26284.282918764464, + "qps_1t_min": 25153.27778651157, + "qps_1t_runs": [ + 25153.27778651157, + 25888.76117100045, + 26284.282918764464 + ], + "recall": 0.43565 + }, + { + "allthreads": { + "median_ms": 0.09519999999999999, + "qps": 10165.375409855225, + "recall": 0.5955 + }, + "knob": 4, + "mean_ms": 0.04875518333333353, + "median_ms": 0.0478, + "nq": 2000, + "p95_ms": 0.066, + "p99_ms": 0.0786, + "qps_1t": 20026.434894060145, + "qps_1t_max": 22804.185024035596, + "qps_1t_min": 19054.896203216656, + "qps_1t_runs": [ + 22804.185024035596, + 20026.434894060145, + 19054.896203216656 + ], + "recall": 0.5955 + }, + { + "allthreads": { + "median_ms": 0.1365, + "qps": 6767.7451971004275, + "recall": 0.74745 + }, + "knob": 8, + "mean_ms": 0.07386681666666645, + "median_ms": 0.07200000000000001, + "nq": 2000, + "p95_ms": 0.1002, + "p99_ms": 0.1166, + "qps_1t": 13524.515889953715, + "qps_1t_max": 13580.70128025269, + "qps_1t_min": 13508.630326199658, + "qps_1t_runs": [ + 13580.70128025269, + 13508.630326199658, + 13524.515889953715 + ], + "recall": 0.74745 + }, + { + "allthreads": { + "median_ms": 0.22, + "qps": 4404.844888893299, + "recall": 0.87095 + }, + "knob": 16, + "mean_ms": 0.11963656666666658, + "median_ms": 0.1164, + "nq": 2000, + "p95_ms": 0.16210000000000002, + "p99_ms": 0.1908, + "qps_1t": 8412.816926587659, + "qps_1t_max": 8451.160111876434, + "qps_1t_min": 8215.81297523342, + "qps_1t_runs": [ + 8215.81297523342, + 8412.816926587659, + 8451.160111876434 + ], + "recall": 0.87095 + }, + { + "allthreads": { + "median_ms": 0.3655, + "qps": 2647.667709229952, + "recall": 0.94865 + }, + "knob": 32, + "mean_ms": 0.203042766666666, + "median_ms": 0.1993, + "nq": 2000, + "p95_ms": 0.2758, + "p99_ms": 0.332, + "qps_1t": 4980.120603580655, + "qps_1t_max": 5131.027193161364, + "qps_1t_min": 4685.218884055826, + "qps_1t_runs": [ + 5131.027193161364, + 4685.218884055826, + 4980.120603580655 + ], + "recall": 0.94865 + }, + { + "allthreads": { + "median_ms": 0.6529, + "qps": 1464.4384552165918, + "recall": 0.98605 + }, + "knob": 64, + "mean_ms": 0.3729039833333329, + "median_ms": 0.3651, + "nq": 2000, + "p95_ms": 0.5085, + "p99_ms": 0.6264000000000001, + "qps_1t": 2725.124585883258, + "qps_1t_max": 2746.5966577489226, + "qps_1t_min": 2579.5181614845446, + "qps_1t_runs": [ + 2746.5966577489226, + 2579.5181614845446, + 2725.124585883258 + ], + "recall": 0.98605 + }, + { + "allthreads": { + "median_ms": 1.3495, + "qps": 738.048748931721, + "recall": 0.9984 + }, + "knob": 128, + "mean_ms": 0.6693367500000026, + "median_ms": 0.6657, + "nq": 2000, + "p95_ms": 0.8558, + "p99_ms": 0.9892, + "qps_1t": 1490.0364984440298, + "qps_1t_max": 1517.2565937316394, + "qps_1t_min": 1475.358307362724, + "qps_1t_runs": [ + 1490.0364984440298, + 1517.2565937316394, + 1475.358307362724 + ], + "recall": 0.9984 + }, + { + "allthreads": { + "median_ms": 1.8943, + "qps": 468.1828655235003, + "recall": 0.9998267898383372 + }, + "knob": 256, + "mean_ms": 1.2183222833333338, + "median_ms": 1.2026000000000001, + "nq": 2000, + "p95_ms": 1.502, + "p99_ms": 1.6647, + "qps_1t": 826.939198261772, + "qps_1t_max": 836.6879640009979, + "qps_1t_min": 799.6805116419888, + "qps_1t_runs": [ + 799.6805116419888, + 826.939198261772, + 836.6879640009979 + ], + "recall": 0.9998 + }, + { + "allthreads": { + "median_ms": 2.5709, + "qps": 378.46356182338485, + "recall": 1.0 + }, + "knob": 1265, + "mean_ms": 2.9812179288025864, + "median_ms": 2.9638, + "nq": 1030, + "p95_ms": 3.4282000000000004, + "p99_ms": 3.8087, + "qps_1t": 334.57601796134, + "qps_1t_max": 339.145519242327, + "qps_1t_min": 332.6448016547239, + "qps_1t_runs": [ + 334.57601796134, + 339.145519242327, + 332.6448016547239 + ], + "recall": 1.0 + } + ], + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "load_before_pct": 10.334375, + "load_contaminated": false + }, + "rust_default_exact|text384|1000000": { + "build": { + "first_query_ms": 1029.9592, + "index_s": 0.0, + "ingest_s": 0.8120639999999997, + "ready_s": 1.8420231999999996, + "total_s": 0.8120639999999997 + }, + "loop_allthreads": { + "mean_ms": 29.137532362459545, + "median_ms": 28.7247, + "nq": 103, + "p95_ms": 31.866699999999998, + "p99_ms": 33.9555, + "qps": 34.190669346418446, + "qps_runs": [ + 34.72379312737248, + 34.190669346418446, + 34.05280744073018 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 3535.3359375, + "rss_mb": 3527.71484375 + }, + "mem_after_first_query": { + "peak_mb": 3535.3359375, + "rss_mb": 3527.71484375 + }, + "mem_baseline": { + "peak_mb": 10.8515625, + "rss_mb": 10.8515625 + }, + "sweep": [ + { + "knob": null, + "mean_ms": 66.36352753623184, + "median_ms": 65.3193, + "nq": 46, + "p95_ms": 70.0012, + "p99_ms": 80.529, + "qps_1t": 15.0615107186387, + "qps_1t_max": 15.089622846686218, + "qps_1t_min": 15.05446591222192, + "qps_1t_runs": [ + 15.05446591222192, + 15.0615107186387, + 15.089622846686218 + ], + "recall": 1.0 + } + ], + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "load_before_pct": 5.3343750000000005, + "load_contaminated": false + }, + "rust_default_ivf|text384|1000000": { + "build": { + "cache_s": 1.0299592, + "first_query_ms": 0.0, + "index_s": 5.7057285, + "ingest_s": 0.8120639999999997, + "ready_s": 7.5477517, + "total_s": 7.5477517 + }, + "mem_after_build": { + "peak_mb": 3790.33203125, + "rss_mb": 3542.54296875 + }, + "mem_baseline": { + "peak_mb": 10.8515625, + "rss_mb": 10.8515625 + }, + "mutation": { + "after": { + "mean_ms": 9.852475945017186, + "median_ms": 9.7885, + "nq": 291, + "p95_ms": 11.424999999999999, + "p99_ms": 11.9871, + "qps_1t": 100.85102321507371, + "qps_1t_max": 102.88985425422318, + "qps_1t_min": 100.77921728903141, + "qps_1t_runs": [ + 100.77921728903141, + 100.85102321507371, + 102.88985425422318 + ], + "recall": 0.952233676975945 + }, + "delete_10k": { + "per_s": 591194.7454611023, + "seconds": 0.0169149 + }, + "first_query_after_ms": 611.4287, + "index_stats_after": { + "builds": 93416, + "churn": 30000, + "pending": 0, + "trained_n": 1000000 + }, + "insert_10k": { + "per_s": 35306.14665890873, + "seconds": 0.2832368 + }, + "mem_after": { + "peak_mb": 5023.3984375, + "rss_mb": 3591.1953125 + }, + "recall_after": 0.952233676975945, + "recall_before": 0.9512195121951219, + "ref_knob": 256, + "update_10k": { + "per_s": 428753.95525523723, + "seconds": 0.0233234 + } + }, + "saveload": { + "disk_mb": 4534.105839729309, + "first_query_after_load_ms": 6601.3625999999995, + "load_s": 13.0069672, + "note": "load = JSON parse + insert; first query includes the scan-cache build and the IVF index build (the index is never persisted)", + "save_s": 21.4794608, + "time_to_ready_s": 19.6083298 + }, + "sweep": [ + { + "allthreads": { + "median_ms": 0.1698, + "qps": 5692.130316769877, + "recall": 0.27565 + }, + "knob": 1, + "mean_ms": 0.1834446999999998, + "median_ms": 0.1754, + "nq": 2000, + "p95_ms": 0.24830000000000002, + "p99_ms": 0.3236, + "qps_1t": 5323.396326856535, + "qps_1t_max": 5828.839055553209, + "qps_1t_min": 5237.704031486981, + "qps_1t_runs": [ + 5323.396326856535, + 5237.704031486981, + 5828.839055553209 + ], + "recall": 0.27565 + }, + { + "allthreads": { + "median_ms": 0.2248, + "qps": 4320.463464756792, + "recall": 0.3806 + }, + "knob": 2, + "mean_ms": 0.23702303333333366, + "median_ms": 0.2286, + "nq": 2000, + "p95_ms": 0.3297, + "p99_ms": 0.4081, + "qps_1t": 4242.414986246092, + "qps_1t_max": 4392.364664656133, + "qps_1t_min": 4037.3617455937197, + "qps_1t_runs": [ + 4242.414986246092, + 4037.3617455937197, + 4392.364664656133 + ], + "recall": 0.3806 + }, + { + "allthreads": { + "median_ms": 0.3305, + "qps": 2997.2870056668235, + "recall": 0.49615 + }, + "knob": 4, + "mean_ms": 0.3279156166666667, + "median_ms": 0.3226, + "nq": 2000, + "p95_ms": 0.4365, + "p99_ms": 0.5088999999999999, + "qps_1t": 3079.6042277422785, + "qps_1t_max": 3091.231197199843, + "qps_1t_min": 2980.323013368833, + "qps_1t_runs": [ + 3091.231197199843, + 3079.6042277422785, + 2980.323013368833 + ], + "recall": 0.49615 + }, + { + "allthreads": { + "median_ms": 0.5065999999999999, + "qps": 1970.3886059520344, + "recall": 0.60455 + }, + "knob": 8, + "mean_ms": 0.515896800000001, + "median_ms": 0.5069, + "nq": 2000, + "p95_ms": 0.6728, + "p99_ms": 0.791, + "qps_1t": 1935.9401283412142, + "qps_1t_max": 1992.9153850975156, + "qps_1t_min": 1889.044871521808, + "qps_1t_runs": [ + 1889.044871521808, + 1992.9153850975156, + 1935.9401283412142 + ], + "recall": 0.60455 + }, + { + "allthreads": { + "median_ms": 0.8159000000000001, + "qps": 1232.5509305451378, + "recall": 0.70215 + }, + "knob": 16, + "mean_ms": 0.8710186166666669, + "median_ms": 0.8673, + "nq": 2000, + "p95_ms": 1.0901999999999998, + "p99_ms": 1.2177, + "qps_1t": 1147.0627994514973, + "qps_1t_max": 1157.0312383392954, + "qps_1t_min": 1140.2728285185901, + "qps_1t_runs": [ + 1147.0627994514973, + 1157.0312383392954, + 1140.2728285185901 + ], + "recall": 0.70215 + }, + { + "allthreads": { + "median_ms": 0.88, + "qps": 1130.7853309777547, + "recall": 0.79095 + }, + "knob": 32, + "mean_ms": 1.5408190379058335, + "median_ms": 1.5315, + "nq": 1961, + "p95_ms": 1.8802999999999999, + "p99_ms": 2.0486, + "qps_1t": 648.3679016030798, + "qps_1t_max": 651.969589824354, + "qps_1t_min": 646.7012629307287, + "qps_1t_runs": [ + 648.3679016030798, + 646.7012629307287, + 651.969589824354 + ], + "recall": 0.7901580826109128 + }, + { + "allthreads": { + "median_ms": 1.1032, + "qps": 866.8276249989277, + "recall": 0.8621 + }, + "knob": 64, + "mean_ms": 2.8165679593721156, + "median_ms": 2.7748, + "nq": 1083, + "p95_ms": 3.3748, + "p99_ms": 3.9991999999999996, + "qps_1t": 355.70602719722376, + "qps_1t_max": 356.3128235767884, + "qps_1t_min": 353.1234288982718, + "qps_1t_runs": [ + 356.3128235767884, + 355.70602719722376, + 353.1234288982718 + ], + "recall": 0.8605724838411819 + }, + { + "allthreads": { + "median_ms": 1.6937, + "qps": 579.1982003109285, + "recall": 0.9145584725536993 + }, + "knob": 128, + "mean_ms": 5.220931326644365, + "median_ms": 5.1327, + "nq": 598, + "p95_ms": 6.2302, + "p99_ms": 7.188899999999999, + "qps_1t": 191.55693278654812, + "qps_1t_max": 193.13226821640987, + "qps_1t_min": 189.94740521521246, + "qps_1t_runs": [ + 189.94740521521246, + 191.55693278654812, + 193.13226821640987 + ], + "recall": 0.9148829431438127 + }, + { + "allthreads": { + "median_ms": 2.7998, + "qps": 350.32285542751816, + "recall": 0.9523489932885906 + }, + "knob": 256, + "mean_ms": 9.970505691056907, + "median_ms": 9.888, + "nq": 287, + "p95_ms": 11.433499999999999, + "p99_ms": 12.3086, + "qps_1t": 100.31053976122804, + "qps_1t_max": 100.39728640477311, + "qps_1t_min": 100.17985949896871, + "qps_1t_runs": [ + 100.31053976122804, + 100.17985949896871, + 100.39728640477311 + ], + "recall": 0.9512195121951219 + } + ], + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "load_before_pct": 5.3343750000000005, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 69.5115023, + "mem": { + "peak_mb": 3789.87109375, + "rss_mb": 3542.55859375 + }, + "subsets": { + "f01": [ + { + "knob": 2, + "mean_ms": 0.22849560000000035, + "median_ms": 0.2512, + "nq": 2000, + "p95_ms": 0.2949, + "p99_ms": 0.3957, + "qps_1t": 4343.88637088276, + "qps_1t_max": 4532.608720285908, + "qps_1t_min": 4261.580952778712, + "qps_1t_runs": [ + 4343.88637088276, + 4261.580952778712, + 4532.608720285908 + ], + "recall": 0.3293 + }, + { + "knob": 8, + "mean_ms": 0.14312516666666655, + "median_ms": 0.1407, + "nq": 2000, + "p95_ms": 0.1632, + "p99_ms": 0.2797, + "qps_1t": 7008.936393902229, + "qps_1t_max": 7067.087865103409, + "qps_1t_min": 6887.0760574244505, + "qps_1t_runs": [ + 7008.936393902229, + 6887.0760574244505, + 7067.087865103409 + ], + "recall": 0.37265 + }, + { + "knob": 16, + "mean_ms": 0.15801510000000052, + "median_ms": 0.1568, + "nq": 2000, + "p95_ms": 0.18159999999999998, + "p99_ms": 0.20359999999999998, + "qps_1t": 6308.2169572441635, + "qps_1t_max": 6388.04566686086, + "qps_1t_min": 6290.119166307609, + "qps_1t_runs": [ + 6388.04566686086, + 6308.2169572441635, + 6290.119166307609 + ], + "recall": 0.49165 + }, + { + "knob": 64, + "mean_ms": 0.2480489333333339, + "median_ms": 0.2409, + "nq": 2000, + "p95_ms": 0.3098, + "p99_ms": 0.375, + "qps_1t": 4115.572689141949, + "qps_1t_max": 4164.387532073587, + "qps_1t_min": 3830.8906935789278, + "qps_1t_runs": [ + 4164.387532073587, + 4115.572689141949, + 3830.8906935789278 + ], + "recall": 0.7302 + }, + { + "knob": 256, + "mean_ms": 0.5232255500000014, + "median_ms": 0.5025000000000001, + "nq": 2000, + "p95_ms": 0.6904, + "p99_ms": 0.9068999999999999, + "qps_1t": 1944.2501814471507, + "qps_1t_max": 2027.054077241906, + "qps_1t_min": 1779.3189051921954, + "qps_1t_runs": [ + 2027.054077241906, + 1944.2501814471507, + 1779.3189051921954 + ], + "recall": 0.9062 + }, + { + "knob": "exact", + "mean_ms": 0.9152129166666687, + "median_ms": 0.8466, + "nq": 2000, + "p95_ms": 1.3227, + "p99_ms": 1.6739000000000002, + "qps_1t": 1079.5255722605523, + "qps_1t_max": 1192.1108014537072, + "qps_1t_min": 1019.9318134785411, + "qps_1t_runs": [ + 1019.9318134785411, + 1079.5255722605523, + 1192.1108014537072 + ], + "recall": 1.0 + } + ], + "f10": [ + { + "knob": 2, + "mean_ms": 0.13679368333333372, + "median_ms": 0.136, + "nq": 2000, + "p95_ms": 0.1553, + "p99_ms": 0.1747, + "qps_1t": 7286.338152395938, + "qps_1t_max": 7503.403731517722, + "qps_1t_min": 7149.748990187306, + "qps_1t_runs": [ + 7149.748990187306, + 7286.338152395938, + 7503.403731517722 + ], + "recall": 0.31865 + }, + { + "knob": 8, + "mean_ms": 0.1785827333333338, + "median_ms": 0.1762, + "nq": 2000, + "p95_ms": 0.2102, + "p99_ms": 0.2475, + "qps_1t": 5638.922322999166, + "qps_1t_max": 5778.755157900156, + "qps_1t_min": 5394.857729509385, + "qps_1t_runs": [ + 5394.857729509385, + 5778.755157900156, + 5638.922322999166 + ], + "recall": 0.5444 + }, + { + "knob": 16, + "mean_ms": 0.2260592333333335, + "median_ms": 0.2223, + "nq": 2000, + "p95_ms": 0.2692, + "p99_ms": 0.3067, + "qps_1t": 4413.919737283497, + "qps_1t_max": 4551.118619445478, + "qps_1t_min": 4312.287215577705, + "qps_1t_runs": [ + 4551.118619445478, + 4413.919737283497, + 4312.287215577705 + ], + "recall": 0.6526 + }, + { + "knob": 64, + "mean_ms": 0.5247319166666653, + "median_ms": 0.5024, + "nq": 2000, + "p95_ms": 0.6893999999999999, + "p99_ms": 0.9543, + "qps_1t": 1900.8400192212903, + "qps_1t_max": 2024.7858080332992, + "qps_1t_min": 1804.2948170819445, + "qps_1t_runs": [ + 1804.2948170819445, + 1900.8400192212903, + 2024.7858080332992 + ], + "recall": 0.828 + }, + { + "knob": 256, + "mean_ms": 1.5236947868116706, + "median_ms": 1.5024000000000002, + "nq": 1931, + "p95_ms": 1.8276, + "p99_ms": 2.1075999999999997, + "qps_1t": 649.9776681675847, + "qps_1t_max": 674.8989425336649, + "qps_1t_min": 644.8007637432971, + "qps_1t_runs": [ + 674.8989425336649, + 644.8007637432971, + 649.9776681675847 + ], + "recall": 0.9400310719834283 + }, + { + "knob": "exact", + "mean_ms": 7.103582568807335, + "median_ms": 6.963699999999999, + "nq": 436, + "p95_ms": 7.9895000000000005, + "p99_ms": 8.9993, + "qps_1t": 140.65354605655205, + "qps_1t_max": 141.26098628161424, + "qps_1t_min": 140.41030467361486, + "qps_1t_runs": [ + 140.41030467361486, + 140.65354605655205, + 141.26098628161424 + ], + "recall": 1.0 + } + ] + } + } + }, + "rust_default_exact|text384|100000": { + "build": { + "first_query_ms": 79.9521, + "index_s": 0.0, + "ingest_s": 0.0558613, + "ready_s": 0.1358134, + "total_s": 0.0558613 + }, + "loop_allthreads": { + "mean_ms": 3.3514406727828745, + "median_ms": 3.21, + "nq": 981, + "p95_ms": 4.2322999999999995, + "p99_ms": 5.5107, + "qps": 298.7845954337795, + "qps_runs": [ + 296.9535651803086, + 298.7845954337795, + 299.41021681053775 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 349.43359375, + "rss_mb": 349.43359375 + }, + "mem_after_first_query": { + "peak_mb": 349.43359375, + "rss_mb": 349.43359375 + }, + "mem_baseline": { + "peak_mb": 10.84375, + "rss_mb": 10.84375 + }, + "sweep": [ + { + "knob": null, + "mean_ms": 6.48268382352941, + "median_ms": 6.4139, + "nq": 476, + "p95_ms": 6.9882, + "p99_ms": 7.859, + "qps_1t": 154.3898916915987, + "qps_1t_max": 154.44166773899806, + "qps_1t_min": 153.9407224082359, + "qps_1t_runs": [ + 154.44166773899806, + 153.9407224082359, + 154.3898916915987 + ], + "recall": 1.0 + } + ], + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "load_before_pct": 5.584375000000002, + "load_contaminated": false + }, + "rust_default_ivf|text384|100000": { + "build": { + "cache_s": 0.0799521, + "first_query_ms": 0.0, + "index_s": 0.4803373, + "ingest_s": 0.0558613, + "ready_s": 0.6161507, + "total_s": 0.6161507 + }, + "mem_after_build": { + "peak_mb": 430.984375, + "rss_mb": 353.2890625 + }, + "mem_baseline": { + "peak_mb": 10.84375, + "rss_mb": 10.84375 + }, + "mutation": { + "after": { + "mean_ms": 1.2855967499999983, + "median_ms": 1.2834, + "nq": 2000, + "p95_ms": 1.5433000000000001, + "p99_ms": 1.6485, + "qps_1t": 777.8958710142853, + "qps_1t_max": 781.6779992388792, + "qps_1t_min": 774.0105822726825, + "qps_1t_runs": [ + 777.8958710142853, + 781.6779992388792, + 774.0105822726825 + ], + "recall": 0.94895 + }, + "delete_10k": { + "per_s": 1037086.2026051604, + "seconds": 0.0096424 + }, + "first_query_after_ms": 191.6616, + "index_stats_after": { + "builds": 110999, + "churn": 30000, + "pending": 0, + "trained_n": 100000 + }, + "insert_10k": { + "per_s": 228709.9614623715, + "seconds": 0.0437235 + }, + "mem_after": { + "peak_mb": 515.1328125, + "rss_mb": 400.9921875 + }, + "recall_after": 0.94895, + "recall_before": 0.9529, + "ref_knob": 128, + "update_10k": { + "per_s": 579979.120751653, + "seconds": 0.017242 + } + }, + "saveload": { + "disk_mb": 453.31724739074707, + "first_query_after_load_ms": 546.3891, + "load_s": 1.2547969, + "note": "load = JSON parse + insert; first query includes the scan-cache build and the IVF index build (the index is never persisted)", + "save_s": 1.9371505999999998, + "time_to_ready_s": 1.801186 + }, + "sweep": [ + { + "allthreads": { + "median_ms": 0.0424, + "qps": 22169.041155716477, + "recall": 0.273 + }, + "knob": 1, + "mean_ms": 0.04181829999999983, + "median_ms": 0.0408, + "nq": 2000, + "p95_ms": 0.052500000000000005, + "p99_ms": 0.0609, + "qps_1t": 24266.219844671956, + "qps_1t_max": 24373.68762925675, + "qps_1t_min": 23138.774801874253, + "qps_1t_runs": [ + 23138.774801874253, + 24266.219844671956, + 24373.68762925675 + ], + "recall": 0.273 + }, + { + "allthreads": { + "median_ms": 0.05450000000000001, + "qps": 17989.073436794522, + "recall": 0.39185 + }, + "knob": 2, + "mean_ms": 0.05683525000000003, + "median_ms": 0.0559, + "nq": 2000, + "p95_ms": 0.0712, + "p99_ms": 0.0787, + "qps_1t": 17640.215457591614, + "qps_1t_max": 17742.89131059643, + "qps_1t_min": 17404.47120865353, + "qps_1t_runs": [ + 17404.47120865353, + 17742.89131059643, + 17640.215457591614 + ], + "recall": 0.39185 + }, + { + "allthreads": { + "median_ms": 0.0797, + "qps": 12368.285489712995, + "recall": 0.51675 + }, + "knob": 4, + "mean_ms": 0.08036593333333353, + "median_ms": 0.07880000000000001, + "nq": 2000, + "p95_ms": 0.1023, + "p99_ms": 0.11649999999999999, + "qps_1t": 12425.223452112215, + "qps_1t_max": 12885.352575459821, + "qps_1t_min": 12046.9082513491, + "qps_1t_runs": [ + 12885.352575459821, + 12046.9082513491, + 12425.223452112215 + ], + "recall": 0.51675 + }, + { + "allthreads": { + "median_ms": 0.1171, + "qps": 8335.879249787527, + "recall": 0.64025 + }, + "knob": 8, + "mean_ms": 0.12940155000000014, + "median_ms": 0.1262, + "nq": 2000, + "p95_ms": 0.1685, + "p99_ms": 0.201, + "qps_1t": 7663.9363893279515, + "qps_1t_max": 8108.156318767302, + "qps_1t_min": 7440.985543653292, + "qps_1t_runs": [ + 7440.985543653292, + 7663.9363893279515, + 8108.156318767302 + ], + "recall": 0.64025 + }, + { + "allthreads": { + "median_ms": 0.1966, + "qps": 5069.301148678294, + "recall": 0.74505 + }, + "knob": 16, + "mean_ms": 0.19838436666666703, + "median_ms": 0.19560000000000002, + "nq": 2000, + "p95_ms": 0.251, + "p99_ms": 0.2852, + "qps_1t": 5069.430925961983, + "qps_1t_max": 5090.874658084133, + "qps_1t_min": 4963.705386216, + "qps_1t_runs": [ + 5090.874658084133, + 5069.430925961983, + 4963.705386216 + ], + "recall": 0.74505 + }, + { + "allthreads": { + "median_ms": 0.3455, + "qps": 2890.5585397360746, + "recall": 0.83455 + }, + "knob": 32, + "mean_ms": 0.3545716166666663, + "median_ms": 0.34909999999999997, + "nq": 2000, + "p95_ms": 0.45189999999999997, + "p99_ms": 0.5088999999999999, + "qps_1t": 2839.9215727258534, + "qps_1t_max": 2847.863703521438, + "qps_1t_min": 2774.29411207399, + "qps_1t_runs": [ + 2839.9215727258534, + 2774.29411207399, + 2847.863703521438 + ], + "recall": 0.83455 + }, + { + "allthreads": { + "median_ms": 0.5929000000000001, + "qps": 1667.319700215919, + "recall": 0.90705 + }, + "knob": 64, + "mean_ms": 0.6426998166666651, + "median_ms": 0.6338999999999999, + "nq": 2000, + "p95_ms": 0.7996, + "p99_ms": 0.87, + "qps_1t": 1562.1560815751616, + "qps_1t_max": 1568.613917840707, + "qps_1t_min": 1537.3899776471187, + "qps_1t_runs": [ + 1562.1560815751616, + 1568.613917840707, + 1537.3899776471187 + ], + "recall": 0.90705 + }, + { + "allthreads": { + "median_ms": 0.6632, + "qps": 1508.873572313256, + "recall": 0.9529 + }, + "knob": 128, + "mean_ms": 1.2603591499999984, + "median_ms": 1.2541, + "nq": 2000, + "p95_ms": 1.5132999999999999, + "p99_ms": 1.6332, + "qps_1t": 793.386520378886, + "qps_1t_max": 796.1783756440444, + "qps_1t_min": 790.7277367838166, + "qps_1t_runs": [ + 790.7277367838166, + 796.1783756440444, + 793.386520378886 + ], + "recall": 0.9529 + }, + { + "allthreads": { + "median_ms": 0.8999, + "qps": 1083.9053824880696, + "recall": 0.9808383233532935 + }, + "knob": 256, + "mean_ms": 2.5085312427410043, + "median_ms": 2.5017, + "nq": 1148, + "p95_ms": 2.8765, + "p99_ms": 3.198, + "qps_1t": 400.75782326922064, + "qps_1t_max": 402.1308451101971, + "qps_1t_min": 393.1484633734097, + "qps_1t_runs": [ + 393.1484633734097, + 400.75782326922064, + 402.1308451101971 + ], + "recall": 0.9806620209059234 + }, + { + "allthreads": { + "median_ms": 3.2567999999999997, + "qps": 299.54243283765044, + "recall": 0.9998976458546571 + }, + "knob": 1265, + "mean_ms": 6.4665334728033494, + "median_ms": 6.3221, + "nq": 478, + "p95_ms": 7.301, + "p99_ms": 8.6458, + "qps_1t": 154.1176827148959, + "qps_1t_max": 157.30762536208323, + "qps_1t_min": 152.5766834291411, + "qps_1t_runs": [ + 157.30762536208323, + 152.5766834291411, + 154.1176827148959 + ], + "recall": 1.0 + } + ], + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "load_before_pct": 5.584375000000002, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 4.7653895, + "mem": { + "peak_mb": 429.875, + "rss_mb": 352.44140625 + }, + "subsets": { + "f01": [ + { + "knob": 2, + "mean_ms": 0.08471788333333351, + "median_ms": 0.09079999999999999, + "nq": 2000, + "p95_ms": 0.1007, + "p99_ms": 0.107, + "qps_1t": 11832.485141356823, + "qps_1t_max": 11890.981107014688, + "qps_1t_min": 11689.999906480021, + "qps_1t_runs": [ + 11689.999906480021, + 11890.981107014688, + 11832.485141356823 + ], + "recall": 0.5042 + }, + { + "knob": 8, + "mean_ms": 0.06029223333333303, + "median_ms": 0.0669, + "nq": 2000, + "p95_ms": 0.0752, + "p99_ms": 0.0849, + "qps_1t": 16608.55058009517, + "qps_1t_max": 16694.281791129502, + "qps_1t_min": 16456.57110884376, + "qps_1t_runs": [ + 16608.55058009517, + 16694.281791129502, + 16456.57110884376 + ], + "recall": 0.5042 + }, + { + "knob": 32, + "mean_ms": 0.04070648333333328, + "median_ms": 0.040299999999999996, + "nq": 2000, + "p95_ms": 0.0448, + "p99_ms": 0.0524, + "qps_1t": 24552.680845839874, + "qps_1t_max": 24690.93126785461, + "qps_1t_min": 24455.857177794118, + "qps_1t_runs": [ + 24455.857177794118, + 24552.680845839874, + 24690.93126785461 + ], + "recall": 0.57315 + }, + { + "knob": 64, + "mean_ms": 0.04964830000000025, + "median_ms": 0.0482, + "nq": 2000, + "p95_ms": 0.0575, + "p99_ms": 0.07100000000000001, + "qps_1t": 20376.641847916915, + "qps_1t_max": 20493.819064170217, + "qps_1t_min": 19579.47209827325, + "qps_1t_runs": [ + 20376.641847916915, + 20493.819064170217, + 19579.47209827325 + ], + "recall": 0.72055 + }, + { + "knob": 1265, + "mean_ms": 0.06680314999999908, + "median_ms": 0.0653, + "nq": 2000, + "p95_ms": 0.0731, + "p99_ms": 0.08009999999999999, + "qps_1t": 15041.699351025905, + "qps_1t_max": 15208.685984739608, + "qps_1t_min": 14667.982873663208, + "qps_1t_runs": [ + 14667.982873663208, + 15041.699351025905, + 15208.685984739608 + ], + "recall": 1.0 + }, + { + "knob": "exact", + "mean_ms": 0.06590931666666716, + "median_ms": 0.06499999999999999, + "nq": 2000, + "p95_ms": 0.0698, + "p99_ms": 0.0733, + "qps_1t": 15288.018626921888, + "qps_1t_max": 15408.403589233587, + "qps_1t_min": 14832.921966964139, + "qps_1t_runs": [ + 14832.921966964139, + 15408.403589233587, + 15288.018626921888 + ], + "recall": 1.0 + } + ], + "f10": [ + { + "knob": 2, + "mean_ms": 0.03395223333333317, + "median_ms": 0.0316, + "nq": 2000, + "p95_ms": 0.0597, + "p99_ms": 0.0652, + "qps_1t": 29580.547831745884, + "qps_1t_max": 29633.58077373282, + "qps_1t_min": 29150.099693340973, + "qps_1t_runs": [ + 29150.099693340973, + 29633.58077373282, + 29580.547831745884 + ], + "recall": 0.3079 + }, + { + "knob": 8, + "mean_ms": 0.041291883333333286, + "median_ms": 0.0408, + "nq": 2000, + "p95_ms": 0.0479, + "p99_ms": 0.0517, + "qps_1t": 24351.13360614719, + "qps_1t_max": 24935.10638563146, + "qps_1t_min": 23416.077947440273, + "qps_1t_runs": [ + 24351.13360614719, + 23416.077947440273, + 24935.10638563146 + ], + "recall": 0.5295 + }, + { + "knob": 32, + "mean_ms": 0.06929940000000015, + "median_ms": 0.068, + "nq": 2000, + "p95_ms": 0.0807, + "p99_ms": 0.0863, + "qps_1t": 14452.109682841197, + "qps_1t_max": 14461.912023850613, + "qps_1t_min": 14376.698247480468, + "qps_1t_runs": [ + 14461.912023850613, + 14452.109682841197, + 14376.698247480468 + ], + "recall": 0.7686 + }, + { + "knob": 64, + "mean_ms": 0.09254078333333327, + "median_ms": 0.0893, + "nq": 2000, + "p95_ms": 0.1181, + "p99_ms": 0.1393, + "qps_1t": 11118.622358215353, + "qps_1t_max": 11504.309226628562, + "qps_1t_min": 9924.652041699432, + "qps_1t_runs": [ + 9924.652041699432, + 11118.622358215353, + 11504.309226628562 + ], + "recall": 0.86425 + }, + { + "knob": 1265, + "mean_ms": 0.32901843333333386, + "median_ms": 0.32220000000000004, + "nq": 2000, + "p95_ms": 0.3634, + "p99_ms": 0.42069999999999996, + "qps_1t": 3051.619517365011, + "qps_1t_max": 3082.655393190381, + "qps_1t_min": 2985.388315428961, + "qps_1t_runs": [ + 3051.619517365011, + 3082.655393190381, + 2985.388315428961 + ], + "recall": 1.0 + }, + { + "knob": "exact", + "mean_ms": 0.3240598166666661, + "median_ms": 0.32239999999999996, + "nq": 2000, + "p95_ms": 0.3554, + "p99_ms": 0.3848, + "qps_1t": 3093.7669722122528, + "qps_1t_max": 3108.4470338498995, + "qps_1t_min": 3055.815851647473, + "qps_1t_runs": [ + 3055.815851647473, + 3108.4470338498995, + 3093.7669722122528 + ], + "recall": 1.0 + } + ] + } + } + }, + "rust_default_exact|text384|10000": { + "build": { + "first_query_ms": 7.188899999999999, + "index_s": 0.0, + "ingest_s": 0.0058946, + "ready_s": 0.0130835, + "total_s": 0.0058946 + }, + "loop_allthreads": { + "mean_ms": 0.26278995000000105, + "median_ms": 0.2576, + "nq": 2000, + "p95_ms": 0.3039, + "p99_ms": 0.4231, + "qps": 3896.271891203626, + "qps_runs": [ + 3621.769223400328, + 3912.269699353802, + 3896.271891203626 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 46.09765625, + "rss_mb": 46.09765625 + }, + "mem_after_first_query": { + "peak_mb": 46.09765625, + "rss_mb": 46.09765625 + }, + "mem_baseline": { + "peak_mb": 10.8515625, + "rss_mb": 10.8515625 + }, + "sweep": [ + { + "knob": null, + "mean_ms": 0.2693790666666671, + "median_ms": 0.27349999999999997, + "nq": 2000, + "p95_ms": 0.3096, + "p99_ms": 0.3631, + "qps_1t": 3727.6705544406655, + "qps_1t_max": 3796.0813431902357, + "qps_1t_min": 3617.374393908926, + "qps_1t_runs": [ + 3617.374393908926, + 3727.6705544406655, + 3796.0813431902357 + ], + "recall": 1.0 + } + ], + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "load_before_pct": 4.58125, + "load_contaminated": false + }, + "rust_default_ivf|text384|10000": { + "build": { + "cache_s": 0.0071889, + "first_query_ms": 0.0, + "index_s": 0.0527886, + "ingest_s": 0.0058946, + "ready_s": 0.0658721, + "total_s": 0.0658721 + }, + "mem_after_build": { + "peak_mb": 63.515625, + "rss_mb": 47.203125 + }, + "mem_baseline": { + "peak_mb": 10.8515625, + "rss_mb": 10.8515625 + }, + "sweep": [ + { + "allthreads": { + "median_ms": 0.0101, + "qps": 94795.71523367145, + "recall": 0.28205 + }, + "knob": 1, + "mean_ms": 0.0103977833333331, + "median_ms": 0.01, + "nq": 2000, + "p95_ms": 0.012700000000000001, + "p99_ms": 0.02, + "qps_1t": 97584.30063771328, + "qps_1t_max": 98118.57629945799, + "qps_1t_min": 92988.22304155189, + "qps_1t_runs": [ + 92988.22304155189, + 97584.30063771328, + 98118.57629945799 + ], + "recall": 0.28205 + }, + { + "allthreads": { + "median_ms": 0.0122, + "qps": 77811.62583501612, + "recall": 0.41255 + }, + "knob": 2, + "mean_ms": 0.012802383333333433, + "median_ms": 0.0121, + "nq": 2000, + "p95_ms": 0.015400000000000002, + "p99_ms": 0.0198, + "qps_1t": 77013.41958836328, + "qps_1t_max": 81272.40070544426, + "qps_1t_min": 76230.55168050241, + "qps_1t_runs": [ + 77013.41958836328, + 76230.55168050241, + 81272.40070544426 + ], + "recall": 0.41255 + }, + { + "allthreads": { + "median_ms": 0.0167, + "qps": 58913.1119967717, + "recall": 0.54975 + }, + "knob": 4, + "mean_ms": 0.017383666666666648, + "median_ms": 0.0166, + "nq": 2000, + "p95_ms": 0.022799999999999997, + "p99_ms": 0.028, + "qps_1t": 56952.148804574426, + "qps_1t_max": 60021.6678220838, + "qps_1t_min": 55767.005451224824, + "qps_1t_runs": [ + 55767.005451224824, + 56952.148804574426, + 60021.6678220838 + ], + "recall": 0.54975 + }, + { + "allthreads": { + "median_ms": 0.0209, + "qps": 45763.134591666996, + "recall": 0.6894 + }, + "knob": 8, + "mean_ms": 0.02615191666666659, + "median_ms": 0.0245, + "nq": 2000, + "p95_ms": 0.0389, + "p99_ms": 0.0521, + "qps_1t": 38850.41618508339, + "qps_1t_max": 42739.243600866714, + "qps_1t_min": 34108.38964059989, + "qps_1t_runs": [ + 42739.243600866714, + 38850.41618508339, + 34108.38964059989 + ], + "recall": 0.6894 + }, + { + "allthreads": { + "median_ms": 0.0313, + "qps": 31228.042782418597, + "recall": 0.80415 + }, + "knob": 16, + "mean_ms": 0.033053599999999995, + "median_ms": 0.032100000000000004, + "nq": 2000, + "p95_ms": 0.043699999999999996, + "p99_ms": 0.054599999999999996, + "qps_1t": 30262.954814382112, + "qps_1t_max": 32184.662720827047, + "qps_1t_min": 28533.601168736335, + "qps_1t_runs": [ + 30262.954814382112, + 28533.601168736335, + 32184.662720827047 + ], + "recall": 0.80415 + }, + { + "allthreads": { + "median_ms": 0.048600000000000004, + "qps": 20209.67538208919, + "recall": 0.8948 + }, + "knob": 32, + "mean_ms": 0.050496883333333346, + "median_ms": 0.0492, + "nq": 2000, + "p95_ms": 0.060000000000000005, + "p99_ms": 0.0781, + "qps_1t": 19591.57444749312, + "qps_1t_max": 20393.617205687005, + "qps_1t_min": 19450.20116370554, + "qps_1t_runs": [ + 20393.617205687005, + 19450.20116370554, + 19591.57444749312 + ], + "recall": 0.8948 + }, + { + "allthreads": { + "median_ms": 0.0821, + "qps": 11971.947333009288, + "recall": 0.9531 + }, + "knob": 64, + "mean_ms": 0.0849742833333333, + "median_ms": 0.0818, + "nq": 2000, + "p95_ms": 0.1002, + "p99_ms": 0.12760000000000002, + "qps_1t": 11870.344970030317, + "qps_1t_max": 12313.185882932386, + "qps_1t_min": 11177.486753281008, + "qps_1t_runs": [ + 11177.486753281008, + 11870.344970030317, + 12313.185882932386 + ], + "recall": 0.9531 + }, + { + "allthreads": { + "median_ms": 0.1516, + "qps": 6451.17588808501, + "recall": 0.98575 + }, + "knob": 128, + "mean_ms": 0.16351716666666705, + "median_ms": 0.158, + "nq": 2000, + "p95_ms": 0.1936, + "p99_ms": 0.2422, + "qps_1t": 6232.9198518185685, + "qps_1t_max": 6255.802256592984, + "qps_1t_min": 5873.320120320836, + "qps_1t_runs": [ + 5873.320120320836, + 6255.802256592984, + 6232.9198518185685 + ], + "recall": 0.98575 + }, + { + "allthreads": { + "median_ms": 0.24649999999999997, + "qps": 3737.7350895206196, + "recall": 0.9979 + }, + "knob": 256, + "mean_ms": 0.2847525000000006, + "median_ms": 0.2816, + "nq": 2000, + "p95_ms": 0.3328, + "p99_ms": 0.39189999999999997, + "qps_1t": 3569.374269059082, + "qps_1t_max": 3591.080760892564, + "qps_1t_min": 3382.6220834820947, + "qps_1t_runs": [ + 3591.080760892564, + 3382.6220834820947, + 3569.374269059082 + ], + "recall": 0.9979 + }, + { + "allthreads": { + "median_ms": 0.25470000000000004, + "qps": 3821.5796193630345, + "recall": 1.0 + }, + "knob": 400, + "mean_ms": 0.2501355666666668, + "median_ms": 0.24449999999999997, + "nq": 2000, + "p95_ms": 0.2837, + "p99_ms": 0.3343, + "qps_1t": 4006.9352034501276, + "qps_1t_max": 4048.6739681802546, + "qps_1t_min": 3939.412621759858, + "qps_1t_runs": [ + 4048.6739681802546, + 3939.412621759858, + 4006.9352034501276 + ], + "recall": 1.0 + } + ], + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "load_before_pct": 4.58125, + "load_contaminated": false + }, + "rust_native_exact|glove|1000000": { + "build": { + "first_query_ms": 658.0114, + "index_s": 0.0, + "ingest_s": 0.7495724000000001, + "ready_s": 1.4075838000000003, + "total_s": 0.7495724000000001 + }, + "loop_allthreads": { + "mean_ms": 14.124988545688547, + "median_ms": 12.790700000000001, + "nq": 259, + "p95_ms": 22.253700000000002, + "p99_ms": 27.1099, + "qps": 68.82732194883368, + "qps_runs": [ + 64.35732538807778, + 68.82732194883368, + 81.250576243831 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 1359.609375, + "rss_mb": 1351.9921875 + }, + "mem_after_first_query": { + "peak_mb": 1359.609375, + "rss_mb": 1351.9921875 + }, + "mem_baseline": { + "peak_mb": 6.58984375, + "rss_mb": 6.58984375 + }, + "sweep": [ + { + "knob": null, + "mean_ms": 25.172556394129973, + "median_ms": 25.696900000000003, + "nq": 159, + "p95_ms": 26.857499999999998, + "p99_ms": 28.1042, + "qps_1t": 40.13650146474251, + "qps_1t_max": 40.184567973435136, + "qps_1t_min": 38.884001759513325, + "qps_1t_runs": [ + 40.184567973435136, + 38.884001759513325, + 40.13650146474251 + ], + "recall": 1.0 + } + ], + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "load_before_pct": 5.837499999999997, + "load_contaminated": false + }, + "rust_native_ivf|glove|1000000": { + "build": { + "cache_s": 0.6580114, + "first_query_ms": 0.0, + "index_s": 5.6914566, + "ingest_s": 0.7495724000000001, + "ready_s": 7.099040400000001, + "total_s": 7.099040400000001 + }, + "mem_after_build": { + "peak_mb": 1428.37109375, + "rss_mb": 1362.56640625 + }, + "mem_baseline": { + "peak_mb": 6.58984375, + "rss_mb": 6.58984375 + }, + "mutation": { + "after": { + "mean_ms": 6.415073264604813, + "median_ms": 6.4448, + "nq": 485, + "p95_ms": 7.9916, + "p99_ms": 8.532100000000002, + "qps_1t": 155.55716921617923, + "qps_1t_max": 157.51122137546145, + "qps_1t_min": 154.60822896401254, + "qps_1t_runs": [ + 157.51122137546145, + 154.60822896401254, + 155.55716921617923 + ], + "recall": 0.9531958762886598 + }, + "delete_10k": { + "per_s": 297216.5668514363, + "seconds": 0.0336455 + }, + "first_query_after_ms": 749.9970999999999, + "index_stats_after": { + "builds": 89609, + "churn": 30000, + "pending": 0, + "trained_n": 1000000 + }, + "insert_10k": { + "per_s": 61105.53353380022, + "seconds": 0.1636513 + }, + "mem_after": { + "peak_mb": 1748.8046875, + "rss_mb": 1378.28515625 + }, + "recall_after": 0.9531958762886598, + "recall_before": 0.9543795620437956, + "ref_knob": 256, + "update_10k": { + "per_s": 253556.12464819086, + "seconds": 0.039439 + } + }, + "sweep": [ + { + "allthreads": { + "median_ms": 0.1747, + "qps": 5548.384268888027, + "recall": 0.30875 + }, + "knob": 1, + "mean_ms": 0.10140630000000009, + "median_ms": 0.0991, + "nq": 2000, + "p95_ms": 0.13970000000000002, + "p99_ms": 0.16469999999999999, + "qps_1t": 9787.46040360551, + "qps_1t_max": 10042.277990339335, + "qps_1t_min": 9759.110984025803, + "qps_1t_runs": [ + 9759.110984025803, + 9787.46040360551, + 10042.277990339335 + ], + "recall": 0.30875 + }, + { + "allthreads": { + "median_ms": 0.2436, + "qps": 3924.265598219951, + "recall": 0.42305 + }, + "knob": 2, + "mean_ms": 0.1308827333333332, + "median_ms": 0.1285, + "nq": 2000, + "p95_ms": 0.1777, + "p99_ms": 0.211, + "qps_1t": 7585.6188803019595, + "qps_1t_max": 7780.729700173469, + "qps_1t_min": 7558.74181718956, + "qps_1t_runs": [ + 7780.729700173469, + 7585.6188803019595, + 7558.74181718956 + ], + "recall": 0.42305 + }, + { + "allthreads": { + "median_ms": 0.3667, + "qps": 2675.3193629048014, + "recall": 0.5377 + }, + "knob": 4, + "mean_ms": 0.19379598333333323, + "median_ms": 0.19039999999999999, + "nq": 2000, + "p95_ms": 0.2703, + "p99_ms": 0.32239999999999996, + "qps_1t": 5135.5463258375785, + "qps_1t_max": 5308.778463347533, + "qps_1t_min": 5042.878333752326, + "qps_1t_runs": [ + 5135.5463258375785, + 5308.778463347533, + 5042.878333752326 + ], + "recall": 0.5377 + }, + { + "allthreads": { + "median_ms": 0.5979, + "qps": 1595.88478765117, + "recall": 0.6346 + }, + "knob": 8, + "mean_ms": 0.30831666666666685, + "median_ms": 0.3004, + "nq": 2000, + "p95_ms": 0.4412, + "p99_ms": 0.5203, + "qps_1t": 3220.1027148363946, + "qps_1t_max": 3376.5184414463597, + "qps_1t_min": 3142.3037831609236, + "qps_1t_runs": [ + 3220.1027148363946, + 3142.3037831609236, + 3376.5184414463597 + ], + "recall": 0.6346 + }, + { + "allthreads": { + "median_ms": 0.9238000000000001, + "qps": 1049.2226545427593, + "recall": 0.72835 + }, + "knob": 16, + "mean_ms": 0.5305095166666661, + "median_ms": 0.5186000000000001, + "nq": 2000, + "p95_ms": 0.7647, + "p99_ms": 0.8749, + "qps_1t": 1902.5639903616077, + "qps_1t_max": 1965.0307065523355, + "qps_1t_min": 1795.2542991179096, + "qps_1t_runs": [ + 1965.0307065523355, + 1795.2542991179096, + 1902.5639903616077 + ], + "recall": 0.72835 + }, + { + "allthreads": { + "median_ms": 1.5516, + "qps": 633.6315663725784, + "recall": 0.8026859504132231 + }, + "knob": 32, + "mean_ms": 0.9899157500000009, + "median_ms": 0.971, + "nq": 2000, + "p95_ms": 1.3912, + "p99_ms": 1.5382, + "qps_1t": 1009.7905250143275, + "qps_1t_max": 1016.0905560225331, + "qps_1t_min": 1004.7437973648804, + "qps_1t_runs": [ + 1016.0905560225331, + 1009.7905250143275, + 1004.7437973648804 + ], + "recall": 0.8013 + }, + { + "allthreads": { + "median_ms": 2.7607999999999997, + "qps": 335.76141020546766, + "recall": 0.8657303370786517 + }, + "knob": 64, + "mean_ms": 1.7541737734487721, + "median_ms": 1.7332999999999998, + "nq": 1848, + "p95_ms": 2.3914, + "p99_ms": 2.6355, + "qps_1t": 575.5939207314974, + "qps_1t_max": 581.3493867975117, + "qps_1t_min": 554.0014568919268, + "qps_1t_runs": [ + 581.3493867975117, + 575.5939207314974, + 554.0014568919268 + ], + "recall": 0.8625541125541125 + }, + { + "allthreads": { + "median_ms": 3.0884, + "qps": 292.19850478467004, + "recall": 0.9199261992619926 + }, + "knob": 128, + "mean_ms": 3.078618540051677, + "median_ms": 3.0542, + "nq": 1032, + "p95_ms": 4.0786, + "p99_ms": 4.5251, + "qps_1t": 324.82901387860284, + "qps_1t_max": 343.6943640585216, + "qps_1t_min": 307.9053719671433, + "qps_1t_runs": [ + 343.6943640585216, + 324.82901387860284, + 307.9053719671433 + ], + "recall": 0.9165697674418605 + }, + { + "allthreads": { + "median_ms": 5.473999999999999, + "qps": 163.77652385618495, + "recall": 0.9553939393939394 + }, + "knob": 256, + "mean_ms": 6.364565754257904, + "median_ms": 6.3609, + "nq": 548, + "p95_ms": 8.0134, + "p99_ms": 8.5396, + "qps_1t": 157.21381770604563, + "qps_1t_max": 158.38667105640525, + "qps_1t_min": 155.7809382293614, + "qps_1t_runs": [ + 158.38667105640525, + 157.21381770604563, + 155.7809382293614 + ], + "recall": 0.9543795620437956 + } + ], + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "load_before_pct": 5.837499999999997, + "load_contaminated": false + }, + "rust_native_exact|glove|100000": { + "build": { + "first_query_ms": 41.7821, + "index_s": 0.0, + "ingest_s": 0.052907600000000006, + "ready_s": 0.09468970000000002, + "total_s": 0.052907600000000006 + }, + "loop_allthreads": { + "mean_ms": 0.6440888666666664, + "median_ms": 0.6114999999999999, + "nq": 2000, + "p95_ms": 0.8467, + "p99_ms": 1.141, + "qps": 1593.736107103529, + "qps_runs": [ + 1462.1382587702526, + 1610.616410280693, + 1593.736107103529 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 127.73828125, + "rss_mb": 127.73828125 + }, + "mem_after_first_query": { + "peak_mb": 127.73828125, + "rss_mb": 127.73828125 + }, + "mem_baseline": { + "peak_mb": 6.58203125, + "rss_mb": 6.58203125 + }, + "sweep": [ + { + "knob": null, + "mean_ms": 1.6096058700209652, + "median_ms": 1.5729, + "nq": 1749, + "p95_ms": 1.9405, + "p99_ms": 2.2458, + "qps_1t": 621.1011829225351, + "qps_1t_max": 627.0951800331468, + "qps_1t_min": 615.718154433941, + "qps_1t_runs": [ + 627.0951800331468, + 621.1011829225351, + 615.718154433941 + ], + "recall": 1.0 + } + ], + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "load_before_pct": 4.156249999999997, + "load_contaminated": false + }, + "rust_native_ivf|glove|100000": { + "build": { + "cache_s": 0.0417821, + "first_query_ms": 0.0, + "index_s": 0.1755758, + "ingest_s": 0.052907600000000006, + "ready_s": 0.27026550000000005, + "total_s": 0.27026550000000005 + }, + "mem_after_build": { + "peak_mb": 150.78125, + "rss_mb": 131.375 + }, + "mem_baseline": { + "peak_mb": 6.58203125, + "rss_mb": 6.58203125 + }, + "mutation": { + "after": { + "mean_ms": 0.7658948000000008, + "median_ms": 0.7534, + "nq": 2000, + "p95_ms": 0.929, + "p99_ms": 1.0816000000000001, + "qps_1t": 1301.8391537368536, + "qps_1t_max": 1315.7182344666462, + "qps_1t_min": 1299.546510249782, + "qps_1t_runs": [ + 1315.7182344666462, + 1299.546510249782, + 1301.8391537368536 + ], + "recall": 0.9743 + }, + "delete_10k": { + "per_s": 1092382.8146335601, + "seconds": 0.0091543 + }, + "first_query_after_ms": 83.5681, + "index_stats_after": { + "builds": 115046, + "churn": 30000, + "pending": 0, + "trained_n": 100000 + }, + "insert_10k": { + "per_s": 708566.5698292354, + "seconds": 0.014113 + }, + "mem_after": { + "peak_mb": 173.14453125, + "rss_mb": 145.70703125 + }, + "recall_after": 0.9743, + "recall_before": 0.974, + "ref_knob": 256, + "update_10k": { + "per_s": 732219.8709828587, + "seconds": 0.0136571 + } + }, + "sweep": [ + { + "allthreads": { + "median_ms": 0.0188, + "qps": 52173.005686857614, + "recall": 0.28505 + }, + "knob": 1, + "mean_ms": 0.01918759999999997, + "median_ms": 0.0187, + "nq": 2000, + "p95_ms": 0.0245, + "p99_ms": 0.0285, + "qps_1t": 52724.123659159646, + "qps_1t_max": 53186.82136940121, + "qps_1t_min": 50519.08358382383, + "qps_1t_runs": [ + 50519.08358382383, + 52724.123659159646, + 53186.82136940121 + ], + "recall": 0.28505 + }, + { + "allthreads": { + "median_ms": 0.023, + "qps": 42747.55644280486, + "recall": 0.4016 + }, + "knob": 2, + "mean_ms": 0.023989816666666594, + "median_ms": 0.0236, + "nq": 2000, + "p95_ms": 0.0304, + "p99_ms": 0.0334, + "qps_1t": 40953.14346090927, + "qps_1t_max": 43266.81781208359, + "qps_1t_min": 40918.371939817254, + "qps_1t_runs": [ + 40953.14346090927, + 40918.371939817254, + 43266.81781208359 + ], + "recall": 0.4016 + }, + { + "allthreads": { + "median_ms": 0.0303, + "qps": 32611.106038272366, + "recall": 0.52085 + }, + "knob": 4, + "mean_ms": 0.03310233333333326, + "median_ms": 0.032, + "nq": 2000, + "p95_ms": 0.0455, + "p99_ms": 0.054799999999999995, + "qps_1t": 31286.37040559653, + "qps_1t_max": 32777.3744339757, + "qps_1t_min": 27147.83489229774, + "qps_1t_runs": [ + 32777.3744339757, + 31286.37040559653, + 27147.83489229774 + ], + "recall": 0.52085 + }, + { + "allthreads": { + "median_ms": 0.0468, + "qps": 20838.28242540935, + "recall": 0.63505 + }, + "knob": 8, + "mean_ms": 0.04619751666666673, + "median_ms": 0.0446, + "nq": 2000, + "p95_ms": 0.0593, + "p99_ms": 0.06820000000000001, + "qps_1t": 21678.52481974306, + "qps_1t_max": 22276.850008855068, + "qps_1t_min": 21019.75226119985, + "qps_1t_runs": [ + 22276.850008855068, + 21019.75226119985, + 21678.52481974306 + ], + "recall": 0.63505 + }, + { + "allthreads": { + "median_ms": 0.0709, + "qps": 14078.09258739152, + "recall": 0.73085 + }, + "knob": 16, + "mean_ms": 0.07260376666666657, + "median_ms": 0.07150000000000001, + "nq": 2000, + "p95_ms": 0.09519999999999999, + "p99_ms": 0.1077, + "qps_1t": 13851.202076572197, + "qps_1t_max": 13940.371455137816, + "qps_1t_min": 13535.224745639289, + "qps_1t_runs": [ + 13851.202076572197, + 13940.371455137816, + 13535.224745639289 + ], + "recall": 0.73085 + }, + { + "allthreads": { + "median_ms": 0.1227, + "qps": 7956.737626178936, + "recall": 0.8134 + }, + "knob": 32, + "mean_ms": 0.12016353333333346, + "median_ms": 0.1177, + "nq": 2000, + "p95_ms": 0.1536, + "p99_ms": 0.1707, + "qps_1t": 8218.07821062851, + "qps_1t_max": 8558.738407723744, + "qps_1t_min": 8198.871835235475, + "qps_1t_runs": [ + 8198.871835235475, + 8218.07821062851, + 8558.738407723744 + ], + "recall": 0.8134 + }, + { + "allthreads": { + "median_ms": 0.2099, + "qps": 4644.946124431138, + "recall": 0.88165 + }, + "knob": 64, + "mean_ms": 0.20941661666666714, + "median_ms": 0.2074, + "nq": 2000, + "p95_ms": 0.2671, + "p99_ms": 0.2966, + "qps_1t": 4858.765407145102, + "qps_1t_max": 4861.232473130136, + "qps_1t_min": 4614.097637074454, + "qps_1t_runs": [ + 4861.232473130136, + 4614.097637074454, + 4858.765407145102 + ], + "recall": 0.88165 + }, + { + "allthreads": { + "median_ms": 0.3767, + "qps": 2610.207870429288, + "recall": 0.9355 + }, + "knob": 128, + "mean_ms": 0.3895535166666657, + "median_ms": 0.3824, + "nq": 2000, + "p95_ms": 0.512, + "p99_ms": 0.5889, + "qps_1t": 2583.9018778765258, + "qps_1t_max": 2715.61656037291, + "qps_1t_min": 2418.915239879592, + "qps_1t_runs": [ + 2715.61656037291, + 2418.915239879592, + 2583.9018778765258 + ], + "recall": 0.9355 + }, + { + "allthreads": { + "median_ms": 0.49660000000000004, + "qps": 1940.3932536795403, + "recall": 0.974 + }, + "knob": 256, + "mean_ms": 0.7330722166666664, + "median_ms": 0.7251, + "nq": 2000, + "p95_ms": 0.9089, + "p99_ms": 1.0217, + "qps_1t": 1357.672693760426, + "qps_1t_max": 1382.6379252190575, + "qps_1t_min": 1352.4352185220912, + "qps_1t_runs": [ + 1357.672693760426, + 1382.6379252190575, + 1352.4352185220912 + ], + "recall": 0.974 + }, + { + "allthreads": { + "median_ms": 0.5972, + "qps": 1624.7220303696256, + "recall": 1.0 + }, + "knob": 1265, + "mean_ms": 1.5772921, + "median_ms": 1.5485, + "nq": 2000, + "p95_ms": 1.8404999999999998, + "p99_ms": 2.0385, + "qps_1t": 635.5073999117142, + "qps_1t_max": 639.6698893981976, + "qps_1t_min": 626.9497549613916, + "qps_1t_runs": [ + 639.6698893981976, + 626.9497549613916, + 635.5073999117142 + ], + "recall": 1.0 + } + ], + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "load_before_pct": 4.156249999999997, + "load_contaminated": false + }, + "rust_native_exact|sift|100000": { + "build": { + "first_query_ms": 58.5794, + "index_s": 0.0, + "ingest_s": 0.05671, + "ready_s": 0.1152894, + "total_s": 0.05671 + }, + "loop_allthreads": { + "mean_ms": 3.0940368769806974, + "median_ms": 2.8814, + "nq": 1157, + "p95_ms": 4.8353, + "p99_ms": 6.1524, + "qps": 328.0837290093046, + "qps_runs": [ + 337.99778060783245, + 328.0837290093046, + 305.2960714860383 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 149.76171875, + "rss_mb": 149.76171875 + }, + "mem_after_first_query": { + "peak_mb": 149.76171875, + "rss_mb": 149.76171875 + }, + "mem_baseline": { + "peak_mb": 7.0078125, + "rss_mb": 7.0078125 + }, + "sweep": [ + { + "knob": null, + "mean_ms": 2.5556045202220448, + "median_ms": 2.6197999999999997, + "nq": 1261, + "p95_ms": 3.1369000000000002, + "p99_ms": 3.4226, + "qps_1t": 363.7788119814834, + "qps_1t_max": 464.61190057880094, + "qps_1t_min": 361.59089210752825, + "qps_1t_runs": [ + 464.61190057880094, + 361.59089210752825, + 363.7788119814834 + ], + "recall": 1.0 + } + ], + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "load_before_pct": 12.721874999999995, + "load_contaminated": false + }, + "rust_native_ivf|sift|100000": { + "build": { + "cache_s": 0.0585794, + "first_query_ms": 0.0, + "index_s": 0.5745037, + "ingest_s": 0.05671, + "ready_s": 0.6897930999999999, + "total_s": 0.6897930999999999 + }, + "mem_after_build": { + "peak_mb": 178.640625, + "rss_mb": 152.59765625 + }, + "mem_baseline": { + "peak_mb": 7.0078125, + "rss_mb": 7.0078125 + }, + "mutation": { + "after": { + "mean_ms": 0.37663301666666676, + "median_ms": 0.3698, + "nq": 2000, + "p95_ms": 0.505, + "p99_ms": 0.5996, + "qps_1t": 2637.741196703611, + "qps_1t_max": 2708.0804381132552, + "qps_1t_min": 2621.0839204233707, + "qps_1t_runs": [ + 2621.0839204233707, + 2637.741196703611, + 2708.0804381132552 + ], + "recall": 0.98545 + }, + "delete_10k": { + "per_s": 506183.02565841755, + "seconds": 0.0197557 + }, + "first_query_after_ms": 276.5351, + "index_stats_after": { + "builds": 113339, + "churn": 30000, + "pending": 0, + "trained_n": 100000 + }, + "insert_10k": { + "per_s": 354948.5147179402, + "seconds": 0.0281731 + }, + "mem_after": { + "peak_mb": 206.8046875, + "rss_mb": 170.87109375 + }, + "recall_after": 0.98545, + "recall_before": 0.98605, + "ref_knob": 64, + "update_10k": { + "per_s": 342604.20306836325, + "seconds": 0.0291882 + } + }, + "sweep": [ + { + "allthreads": { + "median_ms": 0.059, + "qps": 16215.076778388535, + "recall": 0.2912 + }, + "knob": 1, + "mean_ms": 0.02688296666666659, + "median_ms": 0.027, + "nq": 2000, + "p95_ms": 0.0348, + "p99_ms": 0.0422, + "qps_1t": 36184.817574242225, + "qps_1t_max": 42398.66189823047, + "qps_1t_min": 33981.992941940065, + "qps_1t_runs": [ + 42398.66189823047, + 36184.817574242225, + 33981.992941940065 + ], + "recall": 0.2912 + }, + { + "allthreads": { + "median_ms": 0.0678, + "qps": 14669.047942115954, + "recall": 0.43565 + }, + "knob": 2, + "mean_ms": 0.037223333333333365, + "median_ms": 0.0365, + "nq": 2000, + "p95_ms": 0.048600000000000004, + "p99_ms": 0.056, + "qps_1t": 27078.07285355503, + "qps_1t_max": 27601.968572398593, + "qps_1t_min": 25966.97779433897, + "qps_1t_runs": [ + 25966.97779433897, + 27078.07285355503, + 27601.968572398593 + ], + "recall": 0.43565 + }, + { + "allthreads": { + "median_ms": 0.0958, + "qps": 10145.779634680906, + "recall": 0.5955 + }, + "knob": 4, + "mean_ms": 0.047606150000000014, + "median_ms": 0.047, + "nq": 2000, + "p95_ms": 0.0633, + "p99_ms": 0.07279999999999999, + "qps_1t": 20257.616103994493, + "qps_1t_max": 22787.40039057601, + "qps_1t_min": 20173.32924487196, + "qps_1t_runs": [ + 22787.40039057601, + 20173.32924487196, + 20257.616103994493 + ], + "recall": 0.5955 + }, + { + "allthreads": { + "median_ms": 0.13620000000000002, + "qps": 7175.915045776964, + "recall": 0.74745 + }, + "knob": 8, + "mean_ms": 0.07279359999999997, + "median_ms": 0.07050000000000001, + "nq": 2000, + "p95_ms": 0.09849999999999999, + "p99_ms": 0.1144, + "qps_1t": 13567.56375398205, + "qps_1t_max": 14108.819917138895, + "qps_1t_min": 13550.511904463468, + "qps_1t_runs": [ + 13550.511904463468, + 14108.819917138895, + 13567.56375398205 + ], + "recall": 0.74745 + }, + { + "allthreads": { + "median_ms": 0.2271, + "qps": 4303.130592052672, + "recall": 0.87095 + }, + "knob": 16, + "mean_ms": 0.11392756666666629, + "median_ms": 0.1121, + "nq": 2000, + "p95_ms": 0.1571, + "p99_ms": 0.1795, + "qps_1t": 8585.732401609643, + "qps_1t_max": 9421.647459853168, + "qps_1t_min": 8391.243401860416, + "qps_1t_runs": [ + 9421.647459853168, + 8585.732401609643, + 8391.243401860416 + ], + "recall": 0.87095 + }, + { + "allthreads": { + "median_ms": 0.384, + "qps": 2522.9496965459157, + "recall": 0.94865 + }, + "knob": 32, + "mean_ms": 0.20777173333333251, + "median_ms": 0.2044, + "nq": 2000, + "p95_ms": 0.2761, + "p99_ms": 0.317, + "qps_1t": 4740.742988204084, + "qps_1t_max": 4973.279810896004, + "qps_1t_min": 4732.534875824197, + "qps_1t_runs": [ + 4973.279810896004, + 4732.534875824197, + 4740.742988204084 + ], + "recall": 0.94865 + }, + { + "allthreads": { + "median_ms": 0.7008, + "qps": 1399.2686442577065, + "recall": 0.98605 + }, + "knob": 64, + "mean_ms": 0.3611575166666662, + "median_ms": 0.3566, + "nq": 2000, + "p95_ms": 0.47590000000000005, + "p99_ms": 0.5452, + "qps_1t": 2780.4095126150673, + "qps_1t_max": 2800.678380317286, + "qps_1t_min": 2726.6013636551397, + "qps_1t_runs": [ + 2800.678380317286, + 2780.4095126150673, + 2726.6013636551397 + ], + "recall": 0.98605 + }, + { + "allthreads": { + "median_ms": 1.3581999999999999, + "qps": 743.5913272451731, + "recall": 0.9984118581259925 + }, + "knob": 128, + "mean_ms": 0.6374388666666685, + "median_ms": 0.6277, + "nq": 2000, + "p95_ms": 0.8291999999999999, + "p99_ms": 0.9763999999999999, + "qps_1t": 1611.6373106930503, + "qps_1t_max": 1643.4874672576204, + "qps_1t_min": 1463.3416812098444, + "qps_1t_runs": [ + 1643.4874672576204, + 1611.6373106930503, + 1463.3416812098444 + ], + "recall": 0.9984 + }, + { + "allthreads": { + "median_ms": 1.9338, + "qps": 485.74485986662154, + "recall": 0.9998776009791922 + }, + "knob": 256, + "mean_ms": 1.2109834905660386, + "median_ms": 1.1862000000000001, + "nq": 1908, + "p95_ms": 1.5536, + "p99_ms": 1.9023, + "qps_1t": 839.4112928799261, + "qps_1t_max": 840.4635447175514, + "qps_1t_min": 798.8370473324763, + "qps_1t_runs": [ + 798.8370473324763, + 839.4112928799261, + 840.4635447175514 + ], + "recall": 0.9997903563941299 + }, + { + "allthreads": { + "median_ms": 2.9061999999999997, + "qps": 316.4481758992458, + "recall": 1.0 + }, + "knob": 1265, + "mean_ms": 2.8010908129344116, + "median_ms": 2.7870999999999997, + "nq": 1103, + "p95_ms": 3.2592, + "p99_ms": 3.6491000000000002, + "qps_1t": 359.53671461478245, + "qps_1t_max": 359.69023594277206, + "qps_1t_min": 351.8964170085482, + "qps_1t_runs": [ + 359.69023594277206, + 351.8964170085482, + 359.53671461478245 + ], + "recall": 1.0 + } + ], + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "load_before_pct": 12.721874999999995, + "load_contaminated": false + }, + "rust_native_exact|text384|1000000": { + "build": { + "first_query_ms": 1069.9223, + "index_s": 0.0, + "ingest_s": 0.7628488999999998, + "ready_s": 1.8327711999999998, + "total_s": 0.7628488999999998 + }, + "loop_allthreads": { + "mean_ms": 31.419985947712394, + "median_ms": 30.944699999999997, + "nq": 102, + "p95_ms": 35.4849, + "p99_ms": 39.5858, + "qps": 31.93727669150855, + "qps_runs": [ + 31.34563588079623, + 31.93727669150855, + 32.210043843238005 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 3535.34765625, + "rss_mb": 3527.73046875 + }, + "mem_after_first_query": { + "peak_mb": 3535.34765625, + "rss_mb": 3527.73046875 + }, + "mem_baseline": { + "peak_mb": 10.83984375, + "rss_mb": 10.83984375 + }, + "sweep": [ + { + "knob": null, + "mean_ms": 63.51418027210884, + "median_ms": 62.73819999999999, + "nq": 49, + "p95_ms": 68.93730000000001, + "p99_ms": 71.3337, + "qps_1t": 15.79984116322945, + "qps_1t_max": 15.816851958397413, + "qps_1t_min": 15.61839654592467, + "qps_1t_runs": [ + 15.61839654592467, + 15.816851958397413, + 15.79984116322945 + ], + "recall": 1.0 + } + ], + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "load_before_pct": 3.4124999999999983, + "load_contaminated": false + }, + "rust_native_ivf|text384|1000000": { + "build": { + "cache_s": 1.0699223, + "first_query_ms": 0.0, + "index_s": 5.5632862, + "ingest_s": 0.7628488999999998, + "ready_s": 7.3960574, + "total_s": 7.3960574 + }, + "mem_after_build": { + "peak_mb": 3790.25390625, + "rss_mb": 3542.5 + }, + "mem_baseline": { + "peak_mb": 10.83984375, + "rss_mb": 10.83984375 + }, + "mutation": { + "after": { + "mean_ms": 10.060370670995669, + "median_ms": 10.0075, + "nq": 308, + "p95_ms": 11.6226, + "p99_ms": 12.583499999999999, + "qps_1t": 99.10852844010927, + "qps_1t_max": 101.6050530436107, + "qps_1t_min": 97.56923051677892, + "qps_1t_runs": [ + 101.6050530436107, + 97.56923051677892, + 99.10852844010927 + ], + "recall": 0.9538961038961039 + }, + "delete_10k": { + "per_s": 645561.1217270051, + "seconds": 0.0154904 + }, + "first_query_after_ms": 601.5160000000001, + "index_stats_after": { + "builds": 93479, + "churn": 30000, + "pending": 0, + "trained_n": 1000000 + }, + "insert_10k": { + "per_s": 36068.959522331555, + "seconds": 0.2772467 + }, + "mem_after": { + "peak_mb": 5023.4375, + "rss_mb": 3590.8828125 + }, + "recall_after": 0.9538961038961039, + "recall_before": 0.952054794520548, + "ref_knob": 256, + "update_10k": { + "per_s": 479409.3676590441, + "seconds": 0.020859 + } + }, + "sweep": [ + { + "allthreads": { + "median_ms": 0.158, + "qps": 6133.458537666965, + "recall": 0.27565 + }, + "knob": 1, + "mean_ms": 0.17581004999999988, + "median_ms": 0.17129999999999998, + "nq": 2000, + "p95_ms": 0.2428, + "p99_ms": 0.2827, + "qps_1t": 5804.502610720169, + "qps_1t_max": 5871.959572732735, + "qps_1t_min": 5409.815135797173, + "qps_1t_runs": [ + 5409.815135797173, + 5871.959572732735, + 5804.502610720169 + ], + "recall": 0.27565 + }, + { + "allthreads": { + "median_ms": 0.20729999999999998, + "qps": 4672.436502756033, + "recall": 0.3806 + }, + "knob": 2, + "mean_ms": 0.2122564666666669, + "median_ms": 0.2063, + "nq": 2000, + "p95_ms": 0.2942, + "p99_ms": 0.3421, + "qps_1t": 4753.362290816402, + "qps_1t_max": 4780.507766412916, + "qps_1t_min": 4603.856558561967, + "qps_1t_runs": [ + 4603.856558561967, + 4753.362290816402, + 4780.507766412916 + ], + "recall": 0.3806 + }, + { + "allthreads": { + "median_ms": 0.32699999999999996, + "qps": 3006.9847745836405, + "recall": 0.49615 + }, + "knob": 4, + "mean_ms": 0.32868351666666656, + "median_ms": 0.3151, + "nq": 2000, + "p95_ms": 0.4435, + "p99_ms": 0.5546, + "qps_1t": 3049.643313718026, + "qps_1t_max": 3121.599894864514, + "qps_1t_min": 2960.377128363144, + "qps_1t_runs": [ + 3049.643313718026, + 2960.377128363144, + 3121.599894864514 + ], + "recall": 0.49615 + }, + { + "allthreads": { + "median_ms": 0.5069, + "qps": 1956.3372905471977, + "recall": 0.60455 + }, + "knob": 8, + "mean_ms": 0.5167608833333333, + "median_ms": 0.5096999999999999, + "nq": 2000, + "p95_ms": 0.6784, + "p99_ms": 0.7736999999999999, + "qps_1t": 1935.0837455868034, + "qps_1t_max": 1949.9591727298232, + "qps_1t_min": 1920.5731604894327, + "qps_1t_runs": [ + 1949.9591727298232, + 1920.5731604894327, + 1935.0837455868034 + ], + "recall": 0.60455 + }, + { + "allthreads": { + "median_ms": 0.825, + "qps": 1200.3922161527014, + "recall": 0.70215 + }, + "knob": 16, + "mean_ms": 0.8775537833333348, + "median_ms": 0.8694, + "nq": 2000, + "p95_ms": 1.1192000000000002, + "p99_ms": 1.2642, + "qps_1t": 1143.451803134876, + "qps_1t_max": 1148.3809207075149, + "qps_1t_min": 1126.9825313199697, + "qps_1t_runs": [ + 1126.9825313199697, + 1143.451803134876, + 1148.3809207075149 + ], + "recall": 0.70215 + }, + { + "allthreads": { + "median_ms": 0.8964, + "qps": 1085.6667614272806, + "recall": 0.79095 + }, + "knob": 32, + "mean_ms": 1.5492485733467605, + "median_ms": 1.5375, + "nq": 1986, + "p95_ms": 1.887, + "p99_ms": 2.0597, + "qps_1t": 647.4727404521726, + "qps_1t_max": 649.5527930410323, + "qps_1t_min": 639.48497052541, + "qps_1t_runs": [ + 647.4727404521726, + 639.48497052541, + 649.5527930410323 + ], + "recall": 0.790785498489426 + }, + { + "allthreads": { + "median_ms": 1.1093, + "qps": 872.4742742235514, + "recall": 0.8621 + }, + "knob": 64, + "mean_ms": 2.8110195242239597, + "median_ms": 2.7883, + "nq": 1149, + "p95_ms": 3.3551, + "p99_ms": 3.6066000000000003, + "qps_1t": 354.0230373790096, + "qps_1t_max": 360.8796531057676, + "qps_1t_min": 352.4382144983147, + "qps_1t_runs": [ + 352.4382144983147, + 354.0230373790096, + 360.8796531057676 + ], + "recall": 0.8592689295039164 + }, + { + "allthreads": { + "median_ms": 1.7229, + "qps": 548.307555633623, + "recall": 0.914856429463171 + }, + "knob": 128, + "mean_ms": 5.274658128654975, + "median_ms": 5.2091, + "nq": 570, + "p95_ms": 6.1895999999999995, + "p99_ms": 6.6829, + "qps_1t": 189.59317727996515, + "qps_1t_max": 191.19237122313282, + "qps_1t_min": 187.99859205545658, + "qps_1t_runs": [ + 191.19237122313282, + 187.99859205545658, + 189.59317727996515 + ], + "recall": 0.9149122807017543 + }, + { + "allthreads": { + "median_ms": 2.944, + "qps": 332.76858381862684, + "recall": 0.9527654867256637 + }, + "knob": 256, + "mean_ms": 10.030933675799082, + "median_ms": 9.9133, + "nq": 292, + "p95_ms": 11.668, + "p99_ms": 12.490599999999999, + "qps_1t": 99.81722167719664, + "qps_1t_max": 100.33586054854786, + "qps_1t_min": 98.93189802574274, + "qps_1t_runs": [ + 99.81722167719664, + 100.33586054854786, + 98.93189802574274 + ], + "recall": 0.952054794520548 + } + ], + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "load_before_pct": 3.4124999999999983, + "load_contaminated": false + }, + "rust_native_exact|text384|100000": { + "build": { + "first_query_ms": 79.10849999999999, + "index_s": 0.0, + "ingest_s": 0.058094299999999995, + "ready_s": 0.13720279999999999, + "total_s": 0.058094299999999995 + }, + "loop_allthreads": { + "mean_ms": 3.4037624275362415, + "median_ms": 3.2439999999999998, + "nq": 920, + "p95_ms": 4.3783, + "p99_ms": 5.5847, + "qps": 293.33622673385435, + "qps_runs": [ + 283.30055617749935, + 293.33622673385435, + 305.5851904913447 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 349.41796875, + "rss_mb": 349.41796875 + }, + "mem_after_first_query": { + "peak_mb": 349.41796875, + "rss_mb": 349.41796875 + }, + "mem_baseline": { + "peak_mb": 10.8515625, + "rss_mb": 10.8515625 + }, + "sweep": [ + { + "knob": null, + "mean_ms": 6.148305056553553, + "median_ms": 6.061000000000001, + "nq": 501, + "p95_ms": 6.7933, + "p99_ms": 7.3915, + "qps_1t": 163.27361712134723, + "qps_1t_max": 163.96568921406532, + "qps_1t_min": 160.735783609724, + "qps_1t_runs": [ + 163.27361712134723, + 160.735783609724, + 163.96568921406532 + ], + "recall": 1.0 + } + ], + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "load_before_pct": 4.084374999999999, + "load_contaminated": false + }, + "rust_native_ivf|text384|100000": { + "build": { + "cache_s": 0.0791085, + "first_query_ms": 0.0, + "index_s": 0.4714389, + "ingest_s": 0.058094299999999995, + "ready_s": 0.6086417, + "total_s": 0.6086417 + }, + "mem_after_build": { + "peak_mb": 431.03515625, + "rss_mb": 353.34375 + }, + "mem_baseline": { + "peak_mb": 10.8515625, + "rss_mb": 10.8515625 + }, + "mutation": { + "after": { + "mean_ms": 1.300234283333337, + "median_ms": 1.2962, + "nq": 2000, + "p95_ms": 1.5609, + "p99_ms": 1.6765, + "qps_1t": 768.3209294962226, + "qps_1t_max": 772.804066618648, + "qps_1t_min": 766.1811619267572, + "qps_1t_runs": [ + 766.1811619267572, + 768.3209294962226, + 772.804066618648 + ], + "recall": 0.94895 + }, + "delete_10k": { + "per_s": 1032983.1520447901, + "seconds": 0.0096807 + }, + "first_query_after_ms": 192.1449, + "index_stats_after": { + "builds": 112427, + "churn": 30000, + "pending": 0, + "trained_n": 100000 + }, + "insert_10k": { + "per_s": 256464.83738846987, + "seconds": 0.0389917 + }, + "mem_after": { + "peak_mb": 515.2578125, + "rss_mb": 400.87109375 + }, + "recall_after": 0.94895, + "recall_before": 0.9529, + "ref_knob": 128, + "update_10k": { + "per_s": 606159.7958453808, + "seconds": 0.0164973 + } + }, + "sweep": [ + { + "allthreads": { + "median_ms": 0.039799999999999995, + "qps": 24453.1959715805, + "recall": 0.273 + }, + "knob": 1, + "mean_ms": 0.04075911666666657, + "median_ms": 0.039700000000000006, + "nq": 2000, + "p95_ms": 0.0514, + "p99_ms": 0.0601, + "qps_1t": 24669.18621288522, + "qps_1t_max": 24957.4164082534, + "qps_1t_min": 23996.52530313614, + "qps_1t_runs": [ + 23996.52530313614, + 24957.4164082534, + 24669.18621288522 + ], + "recall": 0.273 + }, + { + "allthreads": { + "median_ms": 0.0538, + "qps": 18451.64837800783, + "recall": 0.39185 + }, + "knob": 2, + "mean_ms": 0.056549433333333364, + "median_ms": 0.0555, + "nq": 2000, + "p95_ms": 0.0708, + "p99_ms": 0.0813, + "qps_1t": 17732.178495655146, + "qps_1t_max": 17841.897592860394, + "qps_1t_min": 17480.74495942719, + "qps_1t_runs": [ + 17732.178495655146, + 17480.74495942719, + 17841.897592860394 + ], + "recall": 0.39185 + }, + { + "allthreads": { + "median_ms": 0.0772, + "qps": 12737.263373489703, + "recall": 0.51675 + }, + "knob": 4, + "mean_ms": 0.07671965000000035, + "median_ms": 0.0751, + "nq": 2000, + "p95_ms": 0.0992, + "p99_ms": 0.1108, + "qps_1t": 13134.201328918492, + "qps_1t_max": 13139.25773705193, + "qps_1t_min": 12834.655623728966, + "qps_1t_runs": [ + 12834.655623728966, + 13134.201328918492, + 13139.25773705193 + ], + "recall": 0.51675 + }, + { + "allthreads": { + "median_ms": 0.1171, + "qps": 8381.091586892628, + "recall": 0.64025 + }, + "knob": 8, + "mean_ms": 0.11834945000000005, + "median_ms": 0.11620000000000001, + "nq": 2000, + "p95_ms": 0.15330000000000002, + "p99_ms": 0.177, + "qps_1t": 8489.24900283158, + "qps_1t_max": 8498.266990903881, + "qps_1t_min": 8362.515026394198, + "qps_1t_runs": [ + 8362.515026394198, + 8498.266990903881, + 8489.24900283158 + ], + "recall": 0.64025 + }, + { + "allthreads": { + "median_ms": 0.196, + "qps": 5054.506534845138, + "recall": 0.74505 + }, + "knob": 16, + "mean_ms": 0.20743258333333306, + "median_ms": 0.2029, + "nq": 2000, + "p95_ms": 0.2688, + "p99_ms": 0.3103, + "qps_1t": 4821.191644296347, + "qps_1t_max": 4992.871427818932, + "qps_1t_min": 4659.949504787166, + "qps_1t_runs": [ + 4659.949504787166, + 4821.191644296347, + 4992.871427818932 + ], + "recall": 0.74505 + }, + { + "allthreads": { + "median_ms": 0.35100000000000003, + "qps": 2837.647811300961, + "recall": 0.83455 + }, + "knob": 32, + "mean_ms": 0.36021103333333326, + "median_ms": 0.3549, + "nq": 2000, + "p95_ms": 0.45790000000000003, + "p99_ms": 0.5084, + "qps_1t": 2788.6764676560247, + "qps_1t_max": 2792.409616332693, + "qps_1t_min": 2747.808348061581, + "qps_1t_runs": [ + 2788.6764676560247, + 2747.808348061581, + 2792.409616332693 + ], + "recall": 0.83455 + }, + { + "allthreads": { + "median_ms": 0.5975, + "qps": 1676.1664798718773, + "recall": 0.90705 + }, + "knob": 64, + "mean_ms": 0.6645801333333337, + "median_ms": 0.656, + "nq": 2000, + "p95_ms": 0.8297, + "p99_ms": 0.9142, + "qps_1t": 1506.1073405713832, + "qps_1t_max": 1528.7722972395031, + "qps_1t_min": 1480.0399225968717, + "qps_1t_runs": [ + 1506.1073405713832, + 1528.7722972395031, + 1480.0399225968717 + ], + "recall": 0.90705 + }, + { + "allthreads": { + "median_ms": 0.6681, + "qps": 1471.7698717080295, + "recall": 0.9529 + }, + "knob": 128, + "mean_ms": 1.3025767833333324, + "median_ms": 1.297, + "nq": 2000, + "p95_ms": 1.5675000000000001, + "p99_ms": 1.7002, + "qps_1t": 768.5030854822505, + "qps_1t_max": 774.0316409428224, + "qps_1t_min": 760.7093325456456, + "qps_1t_runs": [ + 768.5030854822505, + 760.7093325456456, + 774.0316409428224 + ], + "recall": 0.9529 + }, + { + "allthreads": { + "median_ms": 0.8917, + "qps": 1094.7782362465746, + "recall": 0.9814 + }, + "knob": 256, + "mean_ms": 2.5845094350783757, + "median_ms": 2.5845, + "nq": 1127, + "p95_ms": 2.9424, + "p99_ms": 3.1992000000000003, + "qps_1t": 386.975323953629, + "qps_1t_max": 388.54662462962256, + "qps_1t_min": 385.2539421699303, + "qps_1t_runs": [ + 388.54662462962256, + 385.2539421699303, + 386.975323953629 + ], + "recall": 0.9807453416149068 + }, + { + "allthreads": { + "median_ms": 3.1554, + "qps": 310.45320705826, + "recall": 0.9998734177215189 + }, + "knob": 1265, + "mean_ms": 6.188903120567375, + "median_ms": 6.0545, + "nq": 470, + "p95_ms": 6.985, + "p99_ms": 8.324, + "qps_1t": 161.0875093156563, + "qps_1t_max": 162.91040336823824, + "qps_1t_min": 160.75723086024405, + "qps_1t_runs": [ + 162.91040336823824, + 160.75723086024405, + 161.0875093156563 + ], + "recall": 1.0 + } + ], + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "load_before_pct": 4.084374999999999, + "load_contaminated": false + }, + "rust_native_exact|text384|10000": { + "build": { + "first_query_ms": 7.3756, + "index_s": 0.0, + "ingest_s": 0.0043495, + "ready_s": 0.0117251, + "total_s": 0.0043495 + }, + "loop_allthreads": { + "mean_ms": 0.2615493333333339, + "median_ms": 0.2612, + "nq": 2000, + "p95_ms": 0.2938, + "p99_ms": 0.3441, + "qps": 3836.2003816252136, + "qps_runs": [ + 3724.243596814881, + 3836.2003816252136, + 3914.468855507168 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 46.09765625, + "rss_mb": 46.09765625 + }, + "mem_after_first_query": { + "peak_mb": 46.09765625, + "rss_mb": 46.09765625 + }, + "mem_baseline": { + "peak_mb": 10.84765625, + "rss_mb": 10.84765625 + }, + "sweep": [ + { + "knob": null, + "mean_ms": 0.28007021666666715, + "median_ms": 0.2744, + "nq": 2000, + "p95_ms": 0.3085, + "p99_ms": 0.3507, + "qps_1t": 3617.4830059692117, + "qps_1t_max": 3629.814131737575, + "qps_1t_min": 3468.859957440568, + "qps_1t_runs": [ + 3468.859957440568, + 3617.4830059692117, + 3629.814131737575 + ], + "recall": 1.0 + } + ], + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "load_before_pct": 6.084375, + "load_contaminated": false + }, + "rust_native_ivf|text384|10000": { + "build": { + "cache_s": 0.0073756, + "first_query_ms": 0.0, + "index_s": 0.0535153, + "ingest_s": 0.0043495, + "ready_s": 0.0652404, + "total_s": 0.0652404 + }, + "mem_after_build": { + "peak_mb": 63.44921875, + "rss_mb": 47.1484375 + }, + "mem_baseline": { + "peak_mb": 10.84765625, + "rss_mb": 10.84765625 + }, + "sweep": [ + { + "allthreads": { + "median_ms": 0.013, + "qps": 74975.16447676686, + "recall": 0.28205 + }, + "knob": 1, + "mean_ms": 0.013493199999999768, + "median_ms": 0.0129, + "nq": 2000, + "p95_ms": 0.0165, + "p99_ms": 0.0264, + "qps_1t": 73753.11145938959, + "qps_1t_max": 74847.49822237175, + "qps_1t_min": 73744.40925197362, + "qps_1t_runs": [ + 73744.40925197362, + 74847.49822237175, + 73753.11145938959 + ], + "recall": 0.28205 + }, + { + "allthreads": { + "median_ms": 0.0149, + "qps": 64299.17118368329, + "recall": 0.41255 + }, + "knob": 2, + "mean_ms": 0.014960416666666485, + "median_ms": 0.0146, + "nq": 2000, + "p95_ms": 0.018000000000000002, + "p99_ms": 0.0219, + "qps_1t": 68011.48033788093, + "qps_1t_max": 68450.48634070536, + "qps_1t_min": 64231.23243677243, + "qps_1t_runs": [ + 64231.23243677243, + 68450.48634070536, + 68011.48033788093 + ], + "recall": 0.41255 + }, + { + "allthreads": { + "median_ms": 0.0157, + "qps": 62532.24318789372, + "recall": 0.54975 + }, + "knob": 4, + "mean_ms": 0.01879223333333348, + "median_ms": 0.018099999999999998, + "nq": 2000, + "p95_ms": 0.0247, + "p99_ms": 0.0304, + "qps_1t": 53059.827608620035, + "qps_1t_max": 57683.26694950694, + "qps_1t_min": 49519.65930474398, + "qps_1t_runs": [ + 53059.827608620035, + 49519.65930474398, + 57683.26694950694 + ], + "recall": 0.54975 + }, + { + "allthreads": { + "median_ms": 0.0218, + "qps": 44628.11392664926, + "recall": 0.6894 + }, + "knob": 8, + "mean_ms": 0.02193473333333346, + "median_ms": 0.0214, + "nq": 2000, + "p95_ms": 0.026699999999999998, + "p99_ms": 0.032, + "qps_1t": 45586.01967948471, + "qps_1t_max": 45604.21018068391, + "qps_1t_min": 45579.16302982923, + "qps_1t_runs": [ + 45586.01967948471, + 45579.16302982923, + 45604.21018068391 + ], + "recall": 0.6894 + }, + { + "allthreads": { + "median_ms": 0.0307, + "qps": 32141.52556536941, + "recall": 0.80415 + }, + "knob": 16, + "mean_ms": 0.0311168166666667, + "median_ms": 0.030800000000000004, + "nq": 2000, + "p95_ms": 0.0371, + "p99_ms": 0.0435, + "qps_1t": 32066.955803718098, + "qps_1t_max": 32367.95895742802, + "qps_1t_min": 31978.561572321916, + "qps_1t_runs": [ + 32367.95895742802, + 32066.955803718098, + 31978.561572321916 + ], + "recall": 0.80415 + }, + { + "allthreads": { + "median_ms": 0.0506, + "qps": 19418.90857966218, + "recall": 0.8948 + }, + "knob": 32, + "mean_ms": 0.054750150000000164, + "median_ms": 0.0535, + "nq": 2000, + "p95_ms": 0.0688, + "p99_ms": 0.0818, + "qps_1t": 18089.675950589823, + "qps_1t_max": 19270.86745882779, + "qps_1t_min": 17519.731597711925, + "qps_1t_runs": [ + 18089.675950589823, + 17519.731597711925, + 19270.86745882779 + ], + "recall": 0.8948 + }, + { + "allthreads": { + "median_ms": 0.0885, + "qps": 11124.924559105326, + "recall": 0.9531 + }, + "knob": 64, + "mean_ms": 0.09016785000000026, + "median_ms": 0.0891, + "nq": 2000, + "p95_ms": 0.1038, + "p99_ms": 0.1254, + "qps_1t": 11118.338030831155, + "qps_1t_max": 11142.110939770182, + "qps_1t_min": 11011.70599405702, + "qps_1t_runs": [ + 11118.338030831155, + 11142.110939770182, + 11011.70599405702 + ], + "recall": 0.9531 + }, + { + "allthreads": { + "median_ms": 0.1587, + "qps": 6285.841080108644, + "recall": 0.98575 + }, + "knob": 128, + "mean_ms": 0.1609034833333329, + "median_ms": 0.1587, + "nq": 2000, + "p95_ms": 0.1858, + "p99_ms": 0.2132, + "qps_1t": 6288.175118131226, + "qps_1t_max": 6391.391562532255, + "qps_1t_min": 5980.097637054107, + "qps_1t_runs": [ + 6391.391562532255, + 5980.097637054107, + 6288.175118131226 + ], + "recall": 0.98575 + }, + { + "allthreads": { + "median_ms": 0.2221, + "qps": 4352.253085257807, + "recall": 0.9979 + }, + "knob": 256, + "mean_ms": 0.29612661666666823, + "median_ms": 0.2924, + "nq": 2000, + "p95_ms": 0.3412, + "p99_ms": 0.3877, + "qps_1t": 3438.407632164651, + "qps_1t_max": 3449.474257004806, + "qps_1t_min": 3250.464897742001, + "qps_1t_runs": [ + 3449.474257004806, + 3250.464897742001, + 3438.407632164651 + ], + "recall": 0.9979 + }, + { + "allthreads": { + "median_ms": 0.2497, + "qps": 3937.1241276809405, + "recall": 1.0 + }, + "knob": 400, + "mean_ms": 0.2279436166666678, + "median_ms": 0.2276, + "nq": 2000, + "p95_ms": 0.24649999999999997, + "p99_ms": 0.28350000000000003, + "qps_1t": 4401.856967380271, + "qps_1t_max": 4409.43937505897, + "qps_1t_min": 4350.326579016282, + "qps_1t_runs": [ + 4401.856967380271, + 4409.43937505897, + 4350.326579016282 + ], + "recall": 1.0 + } + ], + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "load_before_pct": 6.084375, + "load_contaminated": false + }, + "sqlite_vec|glove|100000": { + "mem_baseline": { + "rss_mb": 35.01953125, + "peak_mb": 35.01953125 + }, + "version": "sqlite-vec 0.1.9 / sqlite 3.45.3", + "build": { + "ingest_s": 0.5290911999763921, + "index_s": 1.6999547369778156e-06, + "total_s": 0.5290928999311291, + "first_query_ms": 25.250599952414632, + "ready_s": 0.5543434998835437, + "total_s_runs": [ + 0.5290928999311291, + 0.5210702000767924, + 0.4929345999844372 + ] + }, + "mem_after_build": { + "rss_mb": 38.76171875, + "peak_mb": 54.02734375 + }, + "mem_after_first_query": { + "rss_mb": 39.03125, + "peak_mb": 54.02734375 + }, + "sweep": [ + { + "nq": 125, + "recall_queries": 125, + "recall": 1.0, + "qps_1t_runs": [ + 39.98742922997561, + 40.43658834708476, + 40.07162112720647 + ], + "qps_1t": 40.07162112720647, + "qps_1t_min": 39.98742922997561, + "qps_1t_max": 40.43658834708476, + "median_ms": 24.331900000106543, + "p95_ms": 27.37560999230482, + "p99_ms": 34.3428380077239, + "mean_ms": 24.89775146730244, + "knob": null + } + ], + "loop_allthreads": { + "nq": 125, + "threads": 32, + "qps": 40.51256103382289, + "median_ms": 24.473500030580908, + "p95_ms": 25.93897999613546, + "p99_ms": 27.342113993363455, + "mean_ms": 24.618752799462527 + }, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.08024839998688549, + "per_s": 124613.07641814963, + "batch": 1000 + }, + "update_10k": { + "seconds": 2.47424809995573, + "per_s": 4041.6318800765866, + "batch": 1000 + }, + "delete_10k": { + "seconds": 2.067099599982612, + "per_s": 4837.69625812134, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 48.06640625, + "peak_mb": 57.9609375 + }, + "first_query_after_ms": 28.84109999286011, + "after": { + "nq": 99, + "recall_queries": 99, + "recall": 1.0, + "qps_1t_runs": [ + 33.202292713679284, + 33.191395969538874, + 33.58924378676408 + ], + "qps_1t": 33.202292713679284, + "qps_1t_min": 33.191395969538874, + "qps_1t_max": 33.58924378676408, + "median_ms": 29.00550002232194, + "p95_ms": 36.38529997551814, + "p99_ms": 43.65457602776593, + "mean_ms": 30.006042425873492 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 0.0003665999975055456, + "disk_mb": 48.875, + "load_s": 0.00038629997288808227, + "first_query_after_load_ms": 30.68840003106743, + "time_to_ready_s": 0.031074700003955513 + }, + "load_before_pct": 4.171875, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 0.5791347998892888, + "mem": { + "rss_mb": 38.34375, + "peak_mb": 53.60546875 + }, + "subsets": { + "f10": [ + { + "knob": null, + "nq": 142, + "recall": 1.0, + "qps_1t": 46.12656550824415, + "qps_1t_runs": [ + 45.99653750272907, + 46.89542075728957, + 46.12656550824415 + ], + "median_ms": 21.14765002625063, + "p95_ms": 24.147375021129847, + "p99_ms": 28.752274985890836, + "mean_ms": 21.581430518026835 + } + ], + "f01": [ + { + "knob": null, + "nq": 135, + "recall": 1.0, + "qps_1t": 47.97134170944395, + "qps_1t_runs": [ + 47.366501457312545, + 48.01380236947476, + 47.97134170944395 + ], + "median_ms": 20.34819999244064, + "p95_ms": 24.34583998983725, + "p99_ms": 29.691936005838212, + "mean_ms": 20.928363457040426 + } + ] + } + } + }, + "sqlite_vec|sift|100000": { + "mem_baseline": { + "rss_mb": 35.421875, + "peak_mb": 35.421875 + }, + "version": "sqlite-vec 0.1.9 / sqlite 3.45.3", + "build": { + "ingest_s": 0.5556321000331081, + "index_s": 2.400018274784088e-06, + "total_s": 0.5556345000513829, + "first_query_ms": 32.17329998733476, + "ready_s": 0.5878078000387177, + "total_s_runs": [ + 0.5556345000513829, + 1.9912856000592, + 2.163471699866932 + ] + }, + "mem_after_build": { + "rss_mb": 39.12890625, + "peak_mb": 58.66796875 + }, + "mem_after_first_query": { + "rss_mb": 39.51953125, + "peak_mb": 58.66796875 + }, + "sweep": [ + { + "nq": 93, + "recall_queries": 93, + "recall": 1.0, + "qps_1t_runs": [ + 22.39173649564215, + 21.024955742055827, + 21.553401877047236 + ], + "qps_1t": 21.553401877047236, + "qps_1t_min": 21.024955742055827, + "qps_1t_max": 22.39173649564215, + "median_ms": 46.99310002615675, + "p95_ms": 52.64740001875907, + "p99_ms": 57.69439799711084, + "mean_ms": 46.2060820792825, + "knob": null + } + ], + "loop_allthreads": { + "nq": 35, + "threads": 32, + "qps": 11.035035922920613, + "median_ms": 89.86820001155138, + "p95_ms": 108.1220000050962, + "p99_ms": 113.00987200811505, + "mean_ms": 90.91911619263036 + }, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.307022099965252, + "per_s": 32570.945222287828, + "batch": 1000 + }, + "update_10k": { + "seconds": 10.963700100022834, + "per_s": 912.1008335478981, + "batch": 1000 + }, + "delete_10k": { + "seconds": 8.295781599998008, + "per_s": 1205.43192699316, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 50.640625, + "peak_mb": 62.6171875 + }, + "first_query_after_ms": 109.39749999670312, + "after": { + "nq": 56, + "recall_queries": 56, + "recall": 1.0, + "qps_1t_runs": [ + 18.043468713921857, + 18.005915586202843, + 17.695198498405215 + ], + "qps_1t": 18.005915586202843, + "qps_1t_min": 17.695198498405215, + "qps_1t_max": 18.043468713921857, + "median_ms": 55.82765000872314, + "p95_ms": 66.20149501541165, + "p99_ms": 83.73340703430587, + "mean_ms": 55.82384166787268 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 0.0011999999987892807, + "disk_mb": 61.7890625, + "load_s": 0.0007896000170148909, + "first_query_after_load_ms": 104.93339999811724, + "time_to_ready_s": 0.10572300001513213 + }, + "load_before_pct": 12.049999999999994, + "load_contaminated": false + }, + "sqlite_vec|text384|1000000": { + "mem_baseline": { + "rss_mb": 37.296875, + "peak_mb": 37.296875 + }, + "version": "sqlite-vec 0.1.9 / sqlite 3.45.3", + "build": { + "ingest_s": 9.600988200050779, + "index_s": 1.9999570213258266e-06, + "total_s": 9.6009902000078, + "first_query_ms": 942.8271999931894, + "ready_s": 10.54381740000099 + }, + "mem_after_build": { + "rss_mb": 41.10546875, + "peak_mb": 99.70703125 + }, + "mem_after_first_query": { + "rss_mb": 41.140625, + "peak_mb": 99.70703125 + }, + "sweep": [ + { + "nq": 30, + "recall_queries": 30, + "recall": 1.0, + "qps_1t_runs": [ + 1.0509534128538331 + ], + "qps_1t": 1.0509534128538331, + "qps_1t_min": 1.0509534128538331, + "qps_1t_max": 1.0509534128538331, + "median_ms": 946.5698999993037, + "p95_ms": 988.4569849993568, + "p99_ms": 990.9356639964972, + "mean_ms": 951.5169633300198, + "knob": null + } + ], + "loop_allthreads": { + "nq": 30, + "threads": 32, + "qps": 1.0638727873009715, + "median_ms": 937.6317000424024, + "p95_ms": 963.8271150091896, + "p99_ms": 971.6743419924751, + "mean_ms": 939.9620066765541 + }, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.1939520000014454, + "per_s": 51559.14865495316, + "batch": 1000 + }, + "update_10k": { + "seconds": 10.536050499998964, + "per_s": 949.122254112296, + "batch": 1000 + }, + "delete_10k": { + "seconds": 8.067454600008205, + "per_s": 1239.5483452723527, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 72.34765625, + "peak_mb": 99.70703125 + }, + "first_query_after_ms": 957.5068000121973, + "after": { + "nq": 30, + "recall_queries": 30, + "recall": 1.0, + "qps_1t_runs": [ + 1.0352260402141704 + ], + "qps_1t": 1.0352260402141704, + "qps_1t_min": 1.0352260402141704, + "qps_1t_max": 1.0352260402141704, + "median_ms": 963.0893999710679, + "p95_ms": 991.10578498221, + "p99_ms": 993.1055910210125, + "mean_ms": 965.972609994545 + }, + "recall_after": 1.0 + }, + "load_before_pct": 4.396875, + "load_contaminated": false + }, + "sqlite_vec|text384|100000": { + "mem_baseline": { + "rss_mb": 37.41796875, + "peak_mb": 37.41796875 + }, + "version": "sqlite-vec 0.1.9 / sqlite 3.45.3", + "build": { + "ingest_s": 0.966977899952326, + "index_s": 2.100015990436077e-06, + "total_s": 0.9669799999683164, + "first_query_ms": 98.54739997535944, + "ready_s": 1.0655273999436758, + "total_s_runs": [ + 0.9669799999683164, + 0.9519995000446215, + 0.9635596000007354 + ] + }, + "mem_after_build": { + "rss_mb": 41.12890625, + "peak_mb": 99.73046875 + }, + "mem_after_first_query": { + "rss_mb": 41.1640625, + "peak_mb": 99.73046875 + }, + "sweep": [ + { + "nq": 30, + "recall_queries": 30, + "recall": 1.0, + "qps_1t_runs": [ + 10.174947023143666, + 10.198895459094988, + 10.27547099182717 + ], + "qps_1t": 10.198895459094988, + "qps_1t_min": 10.174947023143666, + "qps_1t_max": 10.27547099182717, + "median_ms": 95.47854997799732, + "p95_ms": 115.11505498201586, + "p99_ms": 121.78105900005903, + "mean_ms": 97.88319444051012, + "knob": null + } + ], + "loop_allthreads": { + "nq": 31, + "threads": 32, + "qps": 10.361603934464103, + "median_ms": 95.295500010252, + "p95_ms": 99.65811998117715, + "p99_ms": 111.8514999793842, + "mean_ms": 96.14113225634159 + }, + "mutation": { + "ref_knob": null, + "recall_before": 1.0, + "insert_10k": { + "seconds": 0.16956050001317635, + "per_s": 58975.99971233225, + "batch": 1000 + }, + "update_10k": { + "seconds": 9.847085700021125, + "per_s": 1015.5288889156867, + "batch": 1000 + }, + "delete_10k": { + "seconds": 7.372649600030854, + "per_s": 1356.3644744432381, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 72.2734375, + "peak_mb": 103.69140625 + }, + "first_query_after_ms": 110.41389999445528, + "after": { + "nq": 30, + "recall_queries": 30, + "recall": 1.0, + "qps_1t_runs": [ + 9.00273548151586, + 8.826417987026353, + 9.040549395469546 + ], + "qps_1t": 9.00273548151586, + "qps_1t_min": 8.826417987026353, + "qps_1t_max": 9.040549395469546, + "median_ms": 109.6875499933958, + "p95_ms": 122.77052999706939, + "p99_ms": 127.13630900194403, + "mean_ms": 111.66211333087024 + }, + "recall_after": 1.0 + }, + "saveload": { + "save_s": 0.0004209000035189092, + "disk_mb": 179.953125, + "load_s": 0.00042400002712383866, + "first_query_after_load_ms": 109.11669995402917, + "time_to_ready_s": 0.10954069998115301 + }, + "load_before_pct": 3.9343749999999957, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 1.007789200055413, + "mem": { + "rss_mb": 40.53515625, + "peak_mb": 99.1328125 + }, + "subsets": { + "f10": [ + { + "knob": null, + "nq": 37, + "recall": 1.0, + "qps_1t": 12.954744350426363, + "qps_1t_runs": [ + 12.510534631107083, + 13.056111391789301, + 12.954744350426363 + ], + "median_ms": 75.81910002045333, + "p95_ms": 85.41519998107105, + "p99_ms": 88.92741004237907, + "mean_ms": 77.90563873860667 + } + ], + "f01": [ + { + "knob": null, + "nq": 40, + "recall": 1.0, + "qps_1t": 13.286640975914697, + "qps_1t_runs": [ + 13.404244466664046, + 13.286640975914697, + 12.986777707918678 + ], + "median_ms": 73.68184998631477, + "p95_ms": 84.5282300200779, + "p99_ms": 94.23895399377216, + "mean_ms": 75.62273250126357 + } + ] + } + } + }, + "sqlite_vec|text384|10000": { + "mem_baseline": { + "rss_mb": 37.29296875, + "peak_mb": 37.29296875 + }, + "version": "sqlite-vec 0.1.9 / sqlite 3.45.3", + "build": { + "ingest_s": 0.12505999999120831, + "index_s": 2.400018274784088e-06, + "total_s": 0.1250624000094831, + "first_query_ms": 9.759800042957067, + "ready_s": 0.13482220005244017 + }, + "mem_after_build": { + "rss_mb": 40.8359375, + "peak_mb": 55.48828125 + }, + "mem_after_first_query": { + "rss_mb": 40.87890625, + "peak_mb": 55.48828125 + }, + "sweep": [ + { + "nq": 292, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 103.71207908579683, + 102.51495514706376, + 104.71020944458607 + ], + "qps_1t": 103.71207908579683, + "qps_1t_min": 102.51495514706376, + "qps_1t_max": 104.71020944458607, + "median_ms": 9.35500001651235, + "p95_ms": 11.23795002058614, + "p99_ms": 13.979025025037117, + "mean_ms": 9.648973288556208, + "knob": null + } + ], + "loop_allthreads": { + "nq": 293, + "threads": 32, + "qps": 102.81119130601117, + "median_ms": 9.56530001712963, + "p95_ms": 10.552820021985099, + "p99_ms": 11.41818599542603, + "mean_ms": 9.673479749713584 + }, + "load_before_pct": 4.631250000000001, + "load_contaminated": false + }, + "usearch|glove|1000000": { + "mem_baseline": { + "rss_mb": 35.15625, + "peak_mb": 35.15625 + }, + "version": "usearch 2.26.6", + "build": { + "ingest_s": 104.53917879989604, + "index_s": 4.199973773211241e-06, + "total_s": 104.53918299986981, + "first_query_ms": 5.705899966415018, + "ready_s": 104.54488889983622 + }, + "mem_after_build": { + "rss_mb": 665.5546875, + "peak_mb": 674.57421875 + }, + "mem_after_first_query": { + "rss_mb": 665.6875, + "peak_mb": 674.57421875 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.47335, + "qps_1t_runs": [ + 10941.613329159567, + 11107.192721715397, + 10420.118326591039 + ], + "qps_1t": 10941.613329159567, + "qps_1t_min": 10420.118326591039, + "qps_1t_max": 11107.192721715397, + "median_ms": 0.0874000252224505, + "p95_ms": 0.13460503250826153, + "p99_ms": 0.16270201595034456, + "mean_ms": 0.09246471682369399, + "knob": 10, + "batch": { + "recall": 0.47335, + "qps_batch_runs": [ + 69913.93591626002, + 60766.81650976942, + 58869.41289680278 + ], + "qps_batch": 60766.81650976942, + "qps_batch_min": 58869.41289680278, + "qps_batch_max": 69913.93591626002, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.567, + "qps_1t_runs": [ + 6804.033276685068, + 7684.77658632071, + 8556.592660722596 + ], + "qps_1t": 7684.77658632071, + "qps_1t_min": 6804.033276685068, + "qps_1t_max": 8556.592660722596, + "median_ms": 0.12359998072497547, + "p95_ms": 0.1940149639267475, + "p99_ms": 0.24041302560362993, + "mean_ms": 0.13132266694447026, + "knob": 16, + "batch": { + "recall": 0.567, + "qps_batch_runs": [ + 56015.6619157394, + 54146.83539688662, + 40165.48179190745 + ], + "qps_batch": 54146.83539688662, + "qps_batch_min": 40165.48179190745, + "qps_batch_max": 56015.6619157394, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6345, + "qps_1t_runs": [ + 6416.786274578042, + 6102.139437938941, + 6459.837737367857 + ], + "qps_1t": 6416.786274578042, + "qps_1t_min": 6102.139437938941, + "qps_1t_max": 6459.837737367857, + "median_ms": 0.14970003394410014, + "p95_ms": 0.22990500146988782, + "p99_ms": 0.2809019858250395, + "mean_ms": 0.15817361698524715, + "knob": 24, + "batch": { + "recall": 0.6345, + "qps_batch_runs": [ + 41150.56993937414, + 45841.17406224451, + 47479.21001112127 + ], + "qps_batch": 45841.17406224451, + "qps_batch_min": 41150.56993937414, + "qps_batch_max": 47479.21001112127, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.6755, + "qps_1t_runs": [ + 5292.066487605985, + 5218.5924821847475, + 5339.289132981858 + ], + "qps_1t": 5292.066487605985, + "qps_1t_min": 5218.5924821847475, + "qps_1t_max": 5339.289132981858, + "median_ms": 0.18039997667074203, + "p95_ms": 0.27090496732853353, + "p99_ms": 0.33311797538772264, + "mean_ms": 0.18929183359917565, + "knob": 32, + "batch": { + "recall": 0.6755, + "qps_batch_runs": [ + 33522.00636620724, + 32467.216217857058, + 28126.94252091266 + ], + "qps_batch": 32467.216217857058, + "qps_batch_min": 28126.94252091266, + "qps_batch_max": 33522.00636620724, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.73065, + "qps_1t_runs": [ + 3636.9660498550597, + 3787.6033175220136, + 3776.656109626694 + ], + "qps_1t": 3776.656109626694, + "qps_1t_min": 3636.9660498550597, + "qps_1t_max": 3787.6033175220136, + "median_ms": 0.24810002651065588, + "p95_ms": 0.3694050159538165, + "p99_ms": 0.4698229691712191, + "mean_ms": 0.2679193835647311, + "knob": 48, + "batch": { + "recall": 0.73065, + "qps_batch_runs": [ + 20223.551134285553, + 22369.988428338755, + 27052.06845977377 + ], + "qps_batch": 22369.988428338755, + "qps_batch_min": 20223.551134285553, + "qps_batch_max": 27052.06845977377, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.76465, + "qps_1t_runs": [ + 2889.8426668563043, + 2822.5178219471654, + 2878.642614136965 + ], + "qps_1t": 2878.642614136965, + "qps_1t_min": 2822.5178219471654, + "qps_1t_max": 2889.8426668563043, + "median_ms": 0.33980002626776695, + "p95_ms": 0.5118049972224981, + "p99_ms": 0.6458090274827557, + "mean_ms": 0.3492397166749773, + "knob": 64, + "batch": { + "recall": 0.76465, + "qps_batch_runs": [ + 17938.764236955303, + 19235.744637154483, + 23485.011288594756 + ], + "qps_batch": 19235.744637154483, + "qps_batch_min": 17938.764236955303, + "qps_batch_max": 23485.011288594756, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8055, + "qps_1t_runs": [ + 1988.0881682927268, + 2059.058323372624, + 2036.819380828179 + ], + "qps_1t": 2036.819380828179, + "qps_1t_min": 1988.0881682927268, + "qps_1t_max": 2059.058323372624, + "median_ms": 0.4835499858018011, + "p95_ms": 0.7098999922163785, + "p99_ms": 0.853150977054612, + "mean_ms": 0.49320541678268154, + "knob": 96, + "batch": { + "recall": 0.8055, + "qps_batch_runs": [ + 14093.369984391786, + 13278.343284263068, + 13595.09326114477 + ], + "qps_batch": 13595.09326114477, + "qps_batch_min": 13278.343284263068, + "qps_batch_max": 14093.369984391786, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.83145, + "qps_1t_runs": [ + 1635.506718806674, + 2462.2912408801676, + 2194.1447283036414 + ], + "qps_1t": 2194.1447283036414, + "qps_1t_min": 1635.506718806674, + "qps_1t_max": 2462.2912408801676, + "median_ms": 0.44319999869912863, + "p95_ms": 0.7982150214957073, + "p99_ms": 1.1405300110345713, + "mean_ms": 0.4911051834157358, + "knob": 128, + "batch": { + "recall": 0.83145, + "qps_batch_runs": [ + 9926.50909024146, + 11891.023523822738, + 11994.492130560624 + ], + "qps_batch": 11891.023523822738, + "qps_batch_min": 9926.50909024146, + "qps_batch_max": 11994.492130560624, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8649, + "qps_1t_runs": [ + 1137.1260912095722, + 1399.6818943829937, + 1461.412392621064 + ], + "qps_1t": 1399.6818943829937, + "qps_1t_min": 1137.1260912095722, + "qps_1t_max": 1461.412392621064, + "median_ms": 0.7115000043995678, + "p95_ms": 1.146019995212555, + "p99_ms": 1.611593045527122, + "mean_ms": 0.7593758502140796, + "knob": 192, + "batch": { + "recall": 0.8649, + "qps_batch_runs": [ + 15057.531058562437, + 16330.276575524489, + 16744.739006367203 + ], + "qps_batch": 16330.276575524489, + "qps_batch_min": 15057.531058562437, + "qps_batch_max": 16744.739006367203, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8839, + "qps_1t_runs": [ + 1018.8648417119123, + 1070.3990194754808, + 1041.0692448105042 + ], + "qps_1t": 1041.0692448105042, + "qps_1t_min": 1018.8648417119123, + "qps_1t_max": 1070.3990194754808, + "median_ms": 0.9078500152099878, + "p95_ms": 1.3639299664646387, + "p99_ms": 2.44202402594965, + "mean_ms": 0.9587554670773292, + "knob": 256, + "batch": { + "recall": 0.8839, + "qps_batch_runs": [ + 12800.688165467176, + 12406.539982445247, + 12443.049715220604 + ], + "qps_batch": 12443.049715220604, + "qps_batch_min": 12406.539982445247, + "qps_batch_max": 12800.688165467176, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.91205, + "qps_1t_runs": [ + 712.4709743073782, + 715.8785440638719, + 726.031266484282 + ], + "qps_1t": 715.8785440638719, + "qps_1t_min": 712.4709743073782, + "qps_1t_max": 726.031266484282, + "median_ms": 1.3280999846756458, + "p95_ms": 1.9126050086924806, + "p99_ms": 3.533559009083561, + "mean_ms": 1.3926007001136895, + "knob": 400, + "batch": { + "recall": 0.91205, + "qps_batch_runs": [ + 9105.859257960406, + 9088.137210748393, + 8353.887377893667 + ], + "qps_batch": 9088.137210748393, + "qps_batch_min": 8353.887377893667, + "qps_batch_max": 9105.859257960406, + "threads": 32 + } + }, + { + "nq": 1353, + "recall_queries": 1353, + "recall": 0.9393939393939394, + "qps_1t_runs": [ + 436.7728524609268, + 451.0766077970741, + 403.8289429781995 + ], + "qps_1t": 436.7728524609268, + "qps_1t_min": 403.8289429781995, + "qps_1t_max": 451.0766077970741, + "median_ms": 2.2125000250525773, + "p95_ms": 3.6095300049055368, + "p99_ms": 5.660058009671049, + "mean_ms": 2.3275779748469017, + "knob": 640, + "batch": { + "recall": 0.93665, + "qps_batch_runs": [ + 2154.7479655729303, + 2503.090377849747, + 2683.6671613000744 + ], + "qps_batch": 2503.090377849747, + "qps_batch_min": 2154.7479655729303, + "qps_batch_max": 2683.6671613000744, + "threads": 32 + } + } + ], + "mutation": { + "ref_knob": 640, + "recall_before": 0.9393939393939394, + "insert_10k": { + "seconds": 1.367447299999185, + "per_s": 7312.89608016774, + "batch": 1000 + }, + "update_10k": { + "seconds": 1.4873334000003524, + "per_s": 6723.442101143987, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.004167300008703023, + "per_s": 2399635.250429755, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 682.38671875, + "peak_mb": 716.30859375 + }, + "first_query_after_ms": 25.090099952649325, + "after": { + "nq": 1240, + "recall_queries": 1240, + "recall": 0.9389516129032258, + "qps_1t_runs": [ + 453.52336445809095, + 470.42636226205815, + 459.54450033204455 + ], + "qps_1t": 459.54450033204455, + "qps_1t_min": 453.52336445809095, + "qps_1t_max": 470.42636226205815, + "median_ms": 2.134649985237047, + "p95_ms": 2.9372099554166193, + "p99_ms": 4.37878997530788, + "mean_ms": 2.1689190320865883 + }, + "recall_after": 0.9389516129032258 + }, + "saveload": { + "save_s": 0.5854958000127226, + "disk_mb": 528.3713569641113, + "load_s": 0.4993411999894306, + "first_query_after_load_ms": 3.3715000026859343, + "time_to_ready_s": 0.5027126999921165 + }, + "load_before_pct": 9.062499999999982, + "load_contaminated": false + }, + "usearch|glove|100000": { + "mem_baseline": { + "rss_mb": 35.14453125, + "peak_mb": 35.14453125 + }, + "version": "usearch 2.26.6", + "build": { + "ingest_s": 1.7242632998968475, + "index_s": 2.800021320581436e-06, + "total_s": 1.724266099918168, + "first_query_ms": 2.073699957691133, + "ready_s": 1.7263397998758592, + "total_s_runs": [ + 1.724266099918168, + 1.6228081999579445, + 1.7237644000560977 + ] + }, + "mem_after_build": { + "rss_mb": 110.3203125, + "peak_mb": 119.50390625 + }, + "mem_after_first_query": { + "rss_mb": 110.55859375, + "peak_mb": 119.50390625 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.53765, + "qps_1t_runs": [ + 26481.40271492198, + 27475.05602791111, + 27781.095971433166 + ], + "qps_1t": 27475.05602791111, + "qps_1t_min": 26481.40271492198, + "qps_1t_max": 27781.095971433166, + "median_ms": 0.03539997851476073, + "p95_ms": 0.05009997694287449, + "p99_ms": 0.05940098664723339, + "mean_ms": 0.03671823343029246, + "knob": 10, + "batch": { + "recall": 0.53765, + "qps_batch_runs": [ + 293590.91006227705, + 287220.1400897643, + 349919.51960857597 + ], + "qps_batch": 293590.91006227705, + "qps_batch_min": 287220.1400897643, + "qps_batch_max": 349919.51960857597, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.62375, + "qps_1t_runs": [ + 21251.004326648366, + 21039.23276226155, + 19802.03909154337 + ], + "qps_1t": 21039.23276226155, + "qps_1t_min": 19802.03909154337, + "qps_1t_max": 21251.004326648366, + "median_ms": 0.045900000259280205, + "p95_ms": 0.06699998630210757, + "p99_ms": 0.09479996573645622, + "mean_ms": 0.04836223322005632, + "knob": 16, + "batch": { + "recall": 0.62375, + "qps_batch_runs": [ + 264872.5947182476, + 282601.62875678693, + 282829.4254517652 + ], + "qps_batch": 282601.62875678693, + "qps_batch_min": 264872.5947182476, + "qps_batch_max": 282829.4254517652, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.693, + "qps_1t_runs": [ + 15664.736939501921, + 15794.03309058259, + 16626.55510416158 + ], + "qps_1t": 15794.03309058259, + "qps_1t_min": 15664.736939501921, + "qps_1t_max": 16626.55510416158, + "median_ms": 0.05989999044686556, + "p95_ms": 0.08620001608505845, + "p99_ms": 0.1082030090037734, + "mean_ms": 0.06243248356622644, + "knob": 24, + "batch": { + "recall": 0.693, + "qps_batch_runs": [ + 227663.37812938154, + 178469.44605269586, + 250884.36769002103 + ], + "qps_batch": 227663.37812938154, + "qps_batch_min": 178469.44605269586, + "qps_batch_max": 250884.36769002103, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.73455, + "qps_1t_runs": [ + 13645.124448001072, + 12832.67092027009, + 11837.709675190305 + ], + "qps_1t": 12832.67092027009, + "qps_1t_min": 11837.709675190305, + "qps_1t_max": 13645.124448001072, + "median_ms": 0.07349997758865356, + "p95_ms": 0.10190501634497197, + "p99_ms": 0.13021300139371333, + "mean_ms": 0.07856271683704108, + "knob": 32, + "batch": { + "recall": 0.73455, + "qps_batch_runs": [ + 146126.19430047876, + 193687.7175194879, + 190170.1063411548 + ], + "qps_batch": 190170.1063411548, + "qps_batch_min": 146126.19430047876, + "qps_batch_max": 193687.7175194879, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.78995, + "qps_1t_runs": [ + 9549.251407179689, + 9153.506599548024, + 10111.095688205 + ], + "qps_1t": 9549.251407179689, + "qps_1t_min": 9153.506599548024, + "qps_1t_max": 10111.095688205, + "median_ms": 0.10119998478330672, + "p95_ms": 0.14110997144598517, + "p99_ms": 0.17130702384747581, + "mean_ms": 0.10428975008350487, + "knob": 48, + "batch": { + "recall": 0.78995, + "qps_batch_runs": [ + 119156.84586972021, + 121533.26332737689, + 150440.4142423235 + ], + "qps_batch": 121533.26332737689, + "qps_batch_min": 119156.84586972021, + "qps_batch_max": 150440.4142423235, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8295, + "qps_1t_runs": [ + 7919.067096959967, + 7238.341203368562, + 7210.338193967392 + ], + "qps_1t": 7238.341203368562, + "qps_1t_min": 7210.338193967392, + "qps_1t_max": 7919.067096959967, + "median_ms": 0.1297999988310039, + "p95_ms": 0.1813049864722416, + "p99_ms": 0.22730398457497367, + "mean_ms": 0.13437348346148306, + "knob": 64, + "batch": { + "recall": 0.8295, + "qps_batch_runs": [ + 80585.04732029412, + 108435.76001904267, + 126035.85726901238 + ], + "qps_batch": 108435.76001904267, + "qps_batch_min": 80585.04732029412, + "qps_batch_max": 126035.85726901238, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.86895, + "qps_1t_runs": [ + 5185.8330213365425, + 5093.542919099803, + 5408.940193440393 + ], + "qps_1t": 5185.8330213365425, + "qps_1t_min": 5093.542919099803, + "qps_1t_max": 5408.940193440393, + "median_ms": 0.1812000118661672, + "p95_ms": 0.2411050314549357, + "p99_ms": 0.32941500539891455, + "mean_ms": 0.1913463837117888, + "knob": 96, + "batch": { + "recall": 0.86895, + "qps_batch_runs": [ + 59542.06199390638, + 86920.61974165458, + 80682.57463165042 + ], + "qps_batch": 80682.57463165042, + "qps_batch_min": 59542.06199390638, + "qps_batch_max": 86920.61974165458, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.895, + "qps_1t_runs": [ + 4152.0893783467445, + 3966.425789650838, + 3947.660343681471 + ], + "qps_1t": 3966.425789650838, + "qps_1t_min": 3947.660343681471, + "qps_1t_max": 4152.0893783467445, + "median_ms": 0.23969996254891157, + "p95_ms": 0.31560501374769956, + "p99_ms": 0.44911798555403987, + "mean_ms": 0.24875778320711106, + "knob": 128, + "batch": { + "recall": 0.895, + "qps_batch_runs": [ + 60906.90384019251, + 67475.47260757248, + 48475.79965746959 + ], + "qps_batch": 60906.90384019251, + "qps_batch_min": 48475.79965746959, + "qps_batch_max": 67475.47260757248, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9264, + "qps_1t_runs": [ + 2900.5407477842627, + 2795.5239185552487, + 2848.318359748414 + ], + "qps_1t": 2848.318359748414, + "qps_1t_min": 2795.5239185552487, + "qps_1t_max": 2900.5407477842627, + "median_ms": 0.3449000068940222, + "p95_ms": 0.44660502462647855, + "p99_ms": 0.5491049902047963, + "mean_ms": 0.3511874500254635, + "knob": 192, + "batch": { + "recall": 0.9264, + "qps_batch_runs": [ + 38682.54974045804, + 48220.54148622443, + 51362.12352558602 + ], + "qps_batch": 48220.54148622443, + "qps_batch_min": 38682.54974045804, + "qps_batch_max": 51362.12352558602, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9456, + "qps_1t_runs": [ + 2134.5277566378336, + 2189.6346202453196, + 1925.888534938385 + ], + "qps_1t": 2134.5277566378336, + "qps_1t_min": 1925.888534938385, + "qps_1t_max": 2189.6346202453196, + "median_ms": 0.47119997907429934, + "p95_ms": 0.6373050186084583, + "p99_ms": 0.7360079581849279, + "mean_ms": 0.4814752501163942, + "knob": 256, + "batch": { + "recall": 0.9456, + "qps_batch_runs": [ + 27037.25734654794, + 29189.963904610235, + 33315.17655442926 + ], + "qps_batch": 29189.963904610235, + "qps_batch_min": 27037.25734654794, + "qps_batch_max": 33315.17655442926, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.96665, + "qps_1t_runs": [ + 1375.5997352042907, + 1372.137190512153, + 1374.707445329708 + ], + "qps_1t": 1374.707445329708, + "qps_1t_min": 1372.137190512153, + "qps_1t_max": 1375.5997352042907, + "median_ms": 0.7197500090114772, + "p95_ms": 0.9486000053584576, + "p99_ms": 1.1774120014160874, + "mean_ms": 0.7277244166374052, + "knob": 400, + "batch": { + "recall": 0.96665, + "qps_batch_runs": [ + 20147.56072548454, + 23417.44880109996, + 25174.902625504725 + ], + "qps_batch": 23417.44880109996, + "qps_batch_min": 20147.56072548454, + "qps_batch_max": 25174.902625504725, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.98195, + "qps_1t_runs": [ + 852.5408659493986, + 895.8063451111749, + 875.7549389770719 + ], + "qps_1t": 875.7549389770719, + "qps_1t_min": 852.5408659493986, + "qps_1t_max": 895.8063451111749, + "median_ms": 1.1377999908290803, + "p95_ms": 1.473804988199845, + "p99_ms": 1.8019379832549052, + "mean_ms": 1.1437163167671922, + "knob": 640, + "batch": { + "recall": 0.98195, + "qps_batch_runs": [ + 15709.401209119207, + 15940.255918684243, + 15402.565295445569 + ], + "qps_batch": 15709.401209119207, + "qps_batch_min": 15402.565295445569, + "qps_batch_max": 15940.255918684243, + "threads": 32 + } + } + ], + "mutation": { + "ref_knob": 400, + "recall_before": 0.96665, + "insert_10k": { + "seconds": 0.26082840003073215, + "per_s": 38339.383283498835, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.2808492000331171, + "per_s": 35606.296898195986, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.0026926000136882067, + "per_s": 3713882.473877891, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 130.9609375, + "peak_mb": 191.05078125 + }, + "first_query_after_ms": 3.6702000070363283, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9636, + "qps_1t_runs": [ + 1237.7031883711013, + 1299.8645938257835, + 1271.7753373789055 + ], + "qps_1t": 1271.7753373789055, + "qps_1t_min": 1237.7031883711013, + "qps_1t_max": 1299.8645938257835, + "median_ms": 0.7844500069040805, + "p95_ms": 1.0358100262237713, + "p99_ms": 1.2599719897843913, + "mean_ms": 0.7878538161652008 + }, + "recall_after": 0.9636 + }, + "saveload": { + "save_s": 0.09747639996930957, + "disk_mb": 57.543521881103516, + "load_s": 0.051671900029759854, + "first_query_after_load_ms": 1.5568999806419015, + "time_to_ready_s": 0.053228800010401756 + }, + "load_before_pct": 3.081249999999999, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 1.5688168000197038, + "mem": { + "rss_mb": 108.9609375, + "peak_mb": 117.99609375 + }, + "subsets": { + "f10": { + "unsupported": true + }, + "f01": { + "unsupported": true + } + } + } + }, + "usearch|sift|100000": { + "mem_baseline": { + "rss_mb": 35.41015625, + "peak_mb": 35.41015625 + }, + "version": "usearch 2.26.6", + "build": { + "ingest_s": 1.5662795999087393, + "index_s": 3.8000289350748062e-06, + "total_s": 1.5662833999376744, + "first_query_ms": 1.3413000269792974, + "ready_s": 1.5676246999646537, + "total_s_runs": [ + 1.5662833999376744, + 1.5631513000116684, + 1.3388760999660008 + ] + }, + "mem_after_build": { + "rss_mb": 120.14453125, + "peak_mb": 130.94921875 + }, + "mem_after_first_query": { + "rss_mb": 120.25390625, + "peak_mb": 130.94921875 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7861, + "qps_1t_runs": [ + 27069.130961110088, + 27812.311509041658, + 26424.23333457998 + ], + "qps_1t": 27069.130961110088, + "qps_1t_min": 26424.23333457998, + "qps_1t_max": 27812.311509041658, + "median_ms": 0.03459997242316604, + "p95_ms": 0.0515049876412377, + "p99_ms": 0.07180096523370597, + "mean_ms": 0.036913932980193444, + "knob": 10, + "batch": { + "recall": 0.7861, + "qps_batch_runs": [ + 357110.9698777513, + 392803.83348365396, + 360457.78201546194 + ], + "qps_batch": 360457.78201546194, + "qps_batch_min": 357110.9698777513, + "qps_batch_max": 392803.83348365396, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.87025, + "qps_1t_runs": [ + 21361.975675150676, + 19975.210776979882, + 19615.45851640123 + ], + "qps_1t": 19975.210776979882, + "qps_1t_min": 19615.45851640123, + "qps_1t_max": 21361.975675150676, + "median_ms": 0.044899992644786835, + "p95_ms": 0.06560003384947777, + "p99_ms": 0.0894059822894634, + "mean_ms": 0.04928479986847378, + "knob": 16, + "batch": { + "recall": 0.87025, + "qps_batch_runs": [ + 235977.06348661665, + 117907.84324861944, + 273119.57188006246 + ], + "qps_batch": 235977.06348661665, + "qps_batch_min": 117907.84324861944, + "qps_batch_max": 273119.57188006246, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9224, + "qps_1t_runs": [ + 16878.91906143038, + 16246.372866444264, + 15886.027276176243 + ], + "qps_1t": 16246.372866444264, + "qps_1t_min": 15886.027276176243, + "qps_1t_max": 16878.91906143038, + "median_ms": 0.057800032664090395, + "p95_ms": 0.0799999717855826, + "p99_ms": 0.1031009707367048, + "mean_ms": 0.06124870019266382, + "knob": 24, + "batch": { + "recall": 0.9224, + "qps_batch_runs": [ + 160194.79671496156, + 126416.65684440518, + 240589.92740892884 + ], + "qps_batch": 160194.79671496156, + "qps_batch_min": 126416.65684440518, + "qps_batch_max": 240589.92740892884, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9517, + "qps_1t_runs": [ + 13737.197886268134, + 12861.678421829287, + 13047.565576531997 + ], + "qps_1t": 13047.565576531997, + "qps_1t_min": 12861.678421829287, + "qps_1t_max": 13737.197886268134, + "median_ms": 0.07189996540546417, + "p95_ms": 0.10080001084133983, + "p99_ms": 0.1329100219300019, + "mean_ms": 0.07572934981241512, + "knob": 32, + "batch": { + "recall": 0.9517, + "qps_batch_runs": [ + 110714.38452159616, + 103358.10472761495, + 74285.37453718168 + ], + "qps_batch": 103358.10472761495, + "qps_batch_min": 74285.37453718168, + "qps_batch_max": 110714.38452159616, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9768, + "qps_1t_runs": [ + 8228.539437715186, + 9040.23857087095, + 9296.406150467108 + ], + "qps_1t": 9040.23857087095, + "qps_1t_min": 8228.539437715186, + "qps_1t_max": 9296.406150467108, + "median_ms": 0.10459998156875372, + "p95_ms": 0.16680498956702652, + "p99_ms": 0.2566010324517265, + "mean_ms": 0.11323775000831422, + "knob": 48, + "batch": { + "recall": 0.9768, + "qps_batch_runs": [ + 106356.38964060324, + 142566.9172567634, + 133913.62584870064 + ], + "qps_batch": 133913.62584870064, + "qps_batch_min": 106356.38964060324, + "qps_batch_max": 142566.9172567634, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9868, + "qps_1t_runs": [ + 6439.467281229915, + 7549.511578564022, + 6417.66344014736 + ], + "qps_1t": 6439.467281229915, + "qps_1t_min": 6417.66344014736, + "qps_1t_max": 7549.511578564022, + "median_ms": 0.13369997031986713, + "p95_ms": 0.22220500395633283, + "p99_ms": 0.31881104048807196, + "mean_ms": 0.14785706653492525, + "knob": 64, + "batch": { + "recall": 0.9868, + "qps_batch_runs": [ + 90733.8098028361, + 91077.58428874474, + 101010.10096353963 + ], + "qps_batch": 91077.58428874474, + "qps_batch_min": 90733.8098028361, + "qps_batch_max": 101010.10096353963, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9939, + "qps_1t_runs": [ + 5047.802693810271, + 5249.557399203695, + 5360.190025936672 + ], + "qps_1t": 5249.557399203695, + "qps_1t_min": 5047.802693810271, + "qps_1t_max": 5360.190025936672, + "median_ms": 0.18119998276233673, + "p95_ms": 0.2735149872023613, + "p99_ms": 0.3934120270423594, + "mean_ms": 0.19171959998008484, + "knob": 96, + "batch": { + "recall": 0.9939, + "qps_batch_runs": [ + 62887.93994649447, + 85289.66511592029, + 77948.09445414573 + ], + "qps_batch": 77948.09445414573, + "qps_batch_min": 62887.93994649447, + "qps_batch_max": 85289.66511592029, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99575, + "qps_1t_runs": [ + 4190.171792421206, + 4301.8681119205285, + 4062.1913385550233 + ], + "qps_1t": 4190.171792421206, + "qps_1t_min": 4062.1913385550233, + "qps_1t_max": 4301.8681119205285, + "median_ms": 0.22819999139755964, + "p95_ms": 0.33569999504834414, + "p99_ms": 0.6329029879998417, + "mean_ms": 0.2390944664075505, + "knob": 128, + "batch": { + "recall": 0.99575, + "qps_batch_runs": [ + 32942.19961171052, + 16449.830485719525, + 32827.13885251534 + ], + "qps_batch": 32827.13885251534, + "qps_batch_min": 16449.830485719525, + "qps_batch_max": 32942.19961171052, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99695, + "qps_1t_runs": [ + 2674.003067122305, + 2769.1319326123185, + 3016.1545197719274 + ], + "qps_1t": 2769.1319326123185, + "qps_1t_min": 2674.003067122305, + "qps_1t_max": 3016.1545197719274, + "median_ms": 0.3363499999977648, + "p95_ms": 0.5188199924305082, + "p99_ms": 0.8846420416375631, + "mean_ms": 0.35554771654036205, + "knob": 192, + "batch": { + "recall": 0.99695, + "qps_batch_runs": [ + 41251.91304261989, + 53736.57250767222, + 47475.37808868633 + ], + "qps_batch": 47475.37808868633, + "qps_batch_min": 41251.91304261989, + "qps_batch_max": 53736.57250767222, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9975, + "qps_1t_runs": [ + 2330.3325577074784, + 2436.9066586890453, + 1993.29932409281 + ], + "qps_1t": 2330.3325577074784, + "qps_1t_min": 1993.29932409281, + "qps_1t_max": 2436.9066586890453, + "median_ms": 0.4217000096105039, + "p95_ms": 0.6297750194789844, + "p99_ms": 0.9572960122022799, + "mean_ms": 0.4470534671466642, + "knob": 256, + "batch": { + "recall": 0.9975, + "qps_batch_runs": [ + 35042.9626709708, + 38197.6806189497, + 37724.31821204922 + ], + "qps_batch": 37724.31821204922, + "qps_batch_min": 35042.9626709708, + "qps_batch_max": 38197.6806189497, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9976, + "qps_1t_runs": [ + 1656.308939527239, + 1591.9763119441568, + 1525.4890147096621 + ], + "qps_1t": 1591.9763119441568, + "qps_1t_min": 1525.4890147096621, + "qps_1t_max": 1656.308939527239, + "median_ms": 0.6066000205464661, + "p95_ms": 0.8747950225370008, + "p99_ms": 1.322138996911236, + "mean_ms": 0.6291432165598962, + "knob": 400, + "batch": { + "recall": 0.9976, + "qps_batch_runs": [ + 25792.2741858166, + 23588.187984603996, + 27727.32944475074 + ], + "qps_batch": 25792.2741858166, + "qps_batch_min": 23588.187984603996, + "qps_batch_max": 27727.32944475074, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.99765, + "qps_1t_runs": [ + 1108.3640302381805, + 1063.633015502327, + 1029.843209069429 + ], + "qps_1t": 1063.633015502327, + "qps_1t_min": 1029.843209069429, + "qps_1t_max": 1108.3640302381805, + "median_ms": 0.9094499982893467, + "p95_ms": 1.2913049955386668, + "p99_ms": 1.822889968752863, + "mean_ms": 0.9378087165241595, + "knob": 640, + "batch": { + "recall": 0.99765, + "qps_batch_runs": [ + 13320.456894933395, + 13864.885311145908, + 16800.087358381366 + ], + "qps_batch": 13864.885311145908, + "qps_batch_min": 13320.456894933395, + "qps_batch_max": 16800.087358381366, + "threads": 32 + } + } + ], + "mutation": { + "ref_knob": 32, + "recall_before": 0.9517, + "insert_10k": { + "seconds": 0.2155102000106126, + "per_s": 46401.51602804674, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.25849350000498816, + "per_s": 38685.69228938843, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.002912099997047335, + "per_s": 3433948.013508911, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 141.046875, + "peak_mb": 210.8359375 + }, + "first_query_after_ms": 2.8415999840945005, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.945, + "qps_1t_runs": [ + 12260.807078531554, + 10577.39346901822, + 10986.155814392707 + ], + "qps_1t": 10986.155814392707, + "qps_1t_min": 10577.39346901822, + "qps_1t_max": 12260.807078531554, + "median_ms": 0.08269998943433166, + "p95_ms": 0.1221999991685152, + "p99_ms": 0.18990499083884071, + "mean_ms": 0.08904186639119871 + }, + "recall_after": 0.945 + }, + "saveload": { + "save_s": 0.07970520004164428, + "disk_mb": 69.29278945922852, + "load_s": 0.06134930002735928, + "first_query_after_load_ms": 0.20030001178383827, + "time_to_ready_s": 0.061549600039143115 + }, + "load_before_pct": 12.12187500000001, + "load_contaminated": false + }, + "usearch|text384|1000000": { + "mem_baseline": { + "rss_mb": 37.3671875, + "peak_mb": 37.3671875 + }, + "version": "usearch 2.26.6", + "build": { + "ingest_s": 173.75032389973057, + "index_s": 3.100023604929447e-06, + "total_s": 173.75032699975418, + "first_query_ms": 5.395699990913272, + "ready_s": 173.7557226997451 + }, + "mem_after_build": { + "rss_mb": 1751.875, + "peak_mb": 1782.6640625 + }, + "mem_after_first_query": { + "rss_mb": 1751.9375, + "peak_mb": 1782.6640625 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.5583, + "qps_1t_runs": [ + 5335.188558767594, + 6033.5301370247835, + 5894.473018927222 + ], + "qps_1t": 5894.473018927222, + "qps_1t_min": 5335.188558767594, + "qps_1t_max": 6033.5301370247835, + "median_ms": 0.16639998648315668, + "p95_ms": 0.2515100495656953, + "p99_ms": 0.30662000644952103, + "mean_ms": 0.1742752332938835, + "knob": 10, + "batch": { + "recall": 0.5583, + "qps_batch_runs": [ + 47578.37943490698, + 47684.89814743232, + 47737.482062584415 + ], + "qps_batch": 47684.89814743232, + "qps_batch_min": 47578.37943490698, + "qps_batch_max": 47737.482062584415, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.64885, + "qps_1t_runs": [ + 4309.308358560452, + 4676.670220743963, + 4212.3921812005165 + ], + "qps_1t": 4309.308358560452, + "qps_1t_min": 4212.3921812005165, + "qps_1t_max": 4676.670220743963, + "median_ms": 0.21650001872330904, + "p95_ms": 0.3289099986432121, + "p99_ms": 0.3963029908481986, + "mean_ms": 0.22775931681583947, + "knob": 16, + "batch": { + "recall": 0.64885, + "qps_batch_runs": [ + 33250.37366577372, + 34043.48032530006, + 32971.960645154984 + ], + "qps_batch": 33250.37366577372, + "qps_batch_min": 32971.960645154984, + "qps_batch_max": 34043.48032530006, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7299, + "qps_1t_runs": [ + 3584.134954498596, + 3614.3058536737617, + 3019.6938427585874 + ], + "qps_1t": 3584.134954498596, + "qps_1t_min": 3019.6938427585874, + "qps_1t_max": 3614.3058536737617, + "median_ms": 0.2711999695748091, + "p95_ms": 0.4070000140927732, + "p99_ms": 0.5640459869755442, + "mean_ms": 0.2956150170066394, + "knob": 24, + "batch": { + "recall": 0.7299, + "qps_batch_runs": [ + 27184.661342805575, + 27095.387959379364, + 27939.516552147372 + ], + "qps_batch": 27184.661342805575, + "qps_batch_min": 27095.387959379364, + "qps_batch_max": 27939.516552147372, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7786, + "qps_1t_runs": [ + 2935.3084432497985, + 2957.1052653075467, + 2832.7524548543925 + ], + "qps_1t": 2935.3084432497985, + "qps_1t_min": 2832.7524548543925, + "qps_1t_max": 2957.1052653075467, + "median_ms": 0.3276999923400581, + "p95_ms": 0.4734049958642572, + "p99_ms": 0.5659370019566276, + "mean_ms": 0.34395393333397806, + "knob": 32, + "batch": { + "recall": 0.7786, + "qps_batch_runs": [ + 22419.865799794956, + 21227.366317750708, + 22714.882789895833 + ], + "qps_batch": 22419.865799794956, + "qps_batch_min": 21227.366317750708, + "qps_batch_max": 22714.882789895833, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.83595, + "qps_1t_runs": [ + 2154.607525434639, + 2082.1634167633856, + 2038.7987484718174 + ], + "qps_1t": 2082.1634167633856, + "qps_1t_min": 2038.7987484718174, + "qps_1t_max": 2154.607525434639, + "median_ms": 0.45035002403892577, + "p95_ms": 0.6310049764579162, + "p99_ms": 0.8108210033969958, + "mean_ms": 0.4782920833968092, + "knob": 48, + "batch": { + "recall": 0.83595, + "qps_batch_runs": [ + 15680.531129883453, + 15943.737740466944, + 15618.167053521367 + ], + "qps_batch": 15680.531129883453, + "qps_batch_min": 15618.167053521367, + "qps_batch_max": 15943.737740466944, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8685, + "qps_1t_runs": [ + 1655.0363123926268, + 1637.7821947676293, + 1650.9278216153132 + ], + "qps_1t": 1650.9278216153132, + "qps_1t_min": 1637.7821947676293, + "qps_1t_max": 1655.0363123926268, + "median_ms": 0.5744999798480421, + "p95_ms": 0.7804250053595753, + "p99_ms": 1.1520349816419213, + "mean_ms": 0.6068393832538277, + "knob": 64, + "batch": { + "recall": 0.8685, + "qps_batch_runs": [ + 13217.634967754377, + 13011.650629423228, + 13608.052699849144 + ], + "qps_batch": 13217.634967754377, + "qps_batch_min": 13011.650629423228, + "qps_batch_max": 13608.052699849144, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.90895, + "qps_1t_runs": [ + 1140.7990103440372, + 1228.939585185393, + 1183.1633253678037 + ], + "qps_1t": 1183.1633253678037, + "qps_1t_min": 1140.7990103440372, + "qps_1t_max": 1228.939585185393, + "median_ms": 0.8043000125326216, + "p95_ms": 1.0800250165630132, + "p99_ms": 1.640918980701827, + "mean_ms": 0.8451600336411502, + "knob": 96, + "batch": { + "recall": 0.90895, + "qps_batch_runs": [ + 9784.099174065563, + 9540.546367565983, + 9227.25435599558 + ], + "qps_batch": 9540.546367565983, + "qps_batch_min": 9227.25435599558, + "qps_batch_max": 9784.099174065563, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.93075, + "qps_1t_runs": [ + 920.4036317979316, + 926.2288373562499, + 907.8523096341713 + ], + "qps_1t": 920.4036317979316, + "qps_1t_min": 907.8523096341713, + "qps_1t_max": 926.2288373562499, + "median_ms": 1.0709500056691468, + "p95_ms": 1.3511150405975059, + "p99_ms": 1.702712042606436, + "mean_ms": 1.089209133303181, + "knob": 128, + "batch": { + "recall": 0.93075, + "qps_batch_runs": [ + 7677.392879812757, + 7010.113139607297, + 7203.585944043157 + ], + "qps_batch": 7203.585944043157, + "qps_batch_min": 7010.113139607297, + "qps_batch_max": 7677.392879812757, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.95695, + "qps_1t_runs": [ + 641.868525528768, + 626.6844889436964, + 669.4238224782664 + ], + "qps_1t": 641.868525528768, + "qps_1t_min": 626.6844889436964, + "qps_1t_max": 669.4238224782664, + "median_ms": 1.5242999943438917, + "p95_ms": 1.9273800076916816, + "p99_ms": 2.4350569728994755, + "mean_ms": 1.5491575665267494, + "knob": 192, + "batch": { + "recall": 0.95695, + "qps_batch_runs": [ + 5397.919803609916, + 5506.814132166697, + 5107.8222979705815 + ], + "qps_batch": 5397.919803609916, + "qps_batch_min": 5107.8222979705815, + "qps_batch_max": 5506.814132166697, + "threads": 32 + } + }, + { + "nq": 1639, + "recall_queries": 1639, + "recall": 0.9672971323978036, + "qps_1t_runs": [ + 521.8354546542845, + 540.8268862508924, + 510.652637746035 + ], + "qps_1t": 521.8354546542845, + "qps_1t_min": 510.652637746035, + "qps_1t_max": 540.8268862508924, + "median_ms": 1.901899988297373, + "p95_ms": 2.3345600231550634, + "p99_ms": 2.785200020298362, + "mean_ms": 1.9078705718044555, + "knob": 256, + "batch": { + "recall": 0.96785, + "qps_batch_runs": [ + 4125.263630185285, + 3963.3375423748707, + 4146.053009424596 + ], + "qps_batch": 4125.263630185285, + "qps_batch_min": 3963.3375423748707, + "qps_batch_max": 4146.053009424596, + "threads": 32 + } + }, + { + "nq": 999, + "recall_queries": 1000, + "recall": 0.9822000000000001, + "qps_1t_runs": [ + 334.5629441679244, + 332.55070197854235, + 338.70559736581873 + ], + "qps_1t": 334.5629441679244, + "qps_1t_min": 332.55070197854235, + "qps_1t_max": 338.70559736581873, + "median_ms": 2.9717000434175134, + "p95_ms": 3.692380001302808, + "p99_ms": 4.919920025859027, + "mean_ms": 2.9828169835524116, + "knob": 400, + "batch": { + "recall": 0.9813, + "qps_batch_runs": [ + 2615.9428373802743, + 2836.315388149292, + 2549.444565422957 + ], + "qps_batch": 2615.9428373802743, + "qps_batch_min": 2549.444565422957, + "qps_batch_max": 2836.315388149292, + "threads": 32 + } + }, + { + "nq": 634, + "recall_queries": 1000, + "recall": 0.99, + "qps_1t_runs": [ + 213.65551236378653, + 208.08420252920422, + 219.82433400068635 + ], + "qps_1t": 213.65551236378653, + "qps_1t_min": 208.08420252920422, + "qps_1t_max": 219.82433400068635, + "median_ms": 4.6139500045683235, + "p95_ms": 5.832479998935014, + "p99_ms": 11.148029983160088, + "mean_ms": 4.678421766794904, + "knob": 640, + "batch": { + "recall": 0.9898, + "qps_batch_runs": [ + 1850.4987602462766, + 1807.3735599749343, + 1810.2565151027852 + ], + "qps_batch": 1810.2565151027852, + "qps_batch_min": 1807.3735599749343, + "qps_batch_max": 1850.4987602462766, + "threads": 32 + } + } + ], + "mutation": { + "ref_knob": 192, + "recall_before": 0.95695, + "insert_10k": { + "seconds": 2.2522861000034027, + "per_s": 4439.933274900064, + "batch": 1000 + }, + "update_10k": { + "seconds": 2.3669153000228107, + "per_s": 4224.908259245114, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.00460629997542128, + "per_s": 2170939.8114232514, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 1800.89453125, + "peak_mb": 1834.61328125 + }, + "first_query_after_ms": 23.924000037368387, + "after": { + "nq": 1961, + "recall_queries": 1961, + "recall": 0.955583885772565, + "qps_1t_runs": [ + 623.3032590172232, + 624.2202221970825, + 641.6424581996398 + ], + "qps_1t": 624.2202221970825, + "qps_1t_min": 623.3032590172232, + "qps_1t_max": 641.6424581996398, + "median_ms": 1.5801999834366143, + "p95_ms": 1.9869599957019086, + "p99_ms": 2.5361139862798163, + "mean_ms": 1.588284854240337 + }, + "recall_after": 0.955583885772565 + }, + "saveload": { + "save_s": 1.5433163999696262, + "disk_mb": 1622.5791206359863, + "load_s": 1.296950799995102, + "first_query_after_load_ms": 1.7228000215254724, + "time_to_ready_s": 1.2986736000166275 + }, + "load_before_pct": 6.6625000000000005, + "load_contaminated": false + }, + "usearch|text384|100000": { + "mem_baseline": { + "rss_mb": 37.51171875, + "peak_mb": 37.51171875 + }, + "version": "usearch 2.26.6", + "build": { + "ingest_s": 5.863001300080214, + "index_s": 2.5999615900218487e-06, + "total_s": 5.863003900041804, + "first_query_ms": 2.5531999999657273, + "ready_s": 5.8655571000417694, + "total_s_runs": [ + 5.863003900041804, + 5.733341399987694, + 6.0116672000149265 + ] + }, + "mem_after_build": { + "rss_mb": 219.58203125, + "peak_mb": 250.08203125 + }, + "mem_after_first_query": { + "rss_mb": 219.703125, + "peak_mb": 250.08203125 + }, + "sweep": [ + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.61185, + "qps_1t_runs": [ + 10699.678717746468, + 12257.680972262828, + 11865.471602766822 + ], + "qps_1t": 11865.471602766822, + "qps_1t_min": 10699.678717746468, + "qps_1t_max": 12257.680972262828, + "median_ms": 0.07960002403706312, + "p95_ms": 0.12690503499470654, + "p99_ms": 0.172304030857049, + "mean_ms": 0.08644013352265272, + "knob": 10, + "batch": { + "recall": 0.61185, + "qps_batch_runs": [ + 100641.08380517158, + 97938.87652022688, + 96350.71649683625 + ], + "qps_batch": 97938.87652022688, + "qps_batch_min": 96350.71649683625, + "qps_batch_max": 100641.08380517158, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7104, + "qps_1t_runs": [ + 8530.32344209426, + 7906.328968693185, + 8812.443860092553 + ], + "qps_1t": 8530.32344209426, + "qps_1t_min": 7906.328968693185, + "qps_1t_max": 8812.443860092553, + "median_ms": 0.10880001354962587, + "p95_ms": 0.16560498042963448, + "p99_ms": 0.21780097100418064, + "mean_ms": 0.11906189982740518, + "knob": 16, + "batch": { + "recall": 0.7104, + "qps_batch_runs": [ + 71225.57848736992, + 79789.67441438432, + 81063.22535712857 + ], + "qps_batch": 79789.67441438432, + "qps_batch_min": 71225.57848736992, + "qps_batch_max": 81063.22535712857, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.7848, + "qps_1t_runs": [ + 6731.395754631413, + 6755.9944317212585, + 6269.462777002373 + ], + "qps_1t": 6731.395754631413, + "qps_1t_min": 6269.462777002373, + "qps_1t_max": 6755.9944317212585, + "median_ms": 0.14044999261386693, + "p95_ms": 0.19750499050132933, + "p99_ms": 0.26801900297869036, + "mean_ms": 0.1520258666035564, + "knob": 24, + "batch": { + "recall": 0.7848, + "qps_batch_runs": [ + 54533.65547796089, + 58747.675846622864, + 60987.323791175004 + ], + "qps_batch": 58747.675846622864, + "qps_batch_min": 54533.65547796089, + "qps_batch_max": 60987.323791175004, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8294, + "qps_1t_runs": [ + 5330.226176255889, + 5371.659770193802, + 5488.245271587127 + ], + "qps_1t": 5371.659770193802, + "qps_1t_min": 5330.226176255889, + "qps_1t_max": 5488.245271587127, + "median_ms": 0.17180002760142088, + "p95_ms": 0.22860999451950195, + "p99_ms": 0.2799119782866913, + "mean_ms": 0.1853263665786168, + "knob": 32, + "batch": { + "recall": 0.8294, + "qps_batch_runs": [ + 45147.74599771415, + 48488.84516066182, + 49929.84856335997 + ], + "qps_batch": 48488.84516066182, + "qps_batch_min": 45147.74599771415, + "qps_batch_max": 49929.84856335997, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.8844, + "qps_1t_runs": [ + 4106.810754089227, + 4055.057951099779, + 4092.170409055547 + ], + "qps_1t": 4092.170409055547, + "qps_1t_min": 4055.057951099779, + "qps_1t_max": 4106.810754089227, + "median_ms": 0.23519998649135232, + "p95_ms": 0.3022099728696049, + "p99_ms": 0.3680390078807258, + "mean_ms": 0.24482421677870056, + "knob": 48, + "batch": { + "recall": 0.8844, + "qps_batch_runs": [ + 33536.733643833795, + 33181.30764467409, + 36037.26974287856 + ], + "qps_batch": 33536.733643833795, + "qps_batch_min": 33181.30764467409, + "qps_batch_max": 36037.26974287856, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.91485, + "qps_1t_runs": [ + 3227.9514774261716, + 3248.1331783857645, + 3164.0993669390855 + ], + "qps_1t": 3227.9514774261716, + "qps_1t_min": 3164.0993669390855, + "qps_1t_max": 3248.1331783857645, + "median_ms": 0.2987499756272882, + "p95_ms": 0.3779999678954482, + "p99_ms": 0.45251301082316814, + "mean_ms": 0.31123628333443776, + "knob": 64, + "batch": { + "recall": 0.91485, + "qps_batch_runs": [ + 26210.94569523811, + 25761.11205407629, + 28458.28656619501 + ], + "qps_batch": 26210.94569523811, + "qps_batch_min": 25761.11205407629, + "qps_batch_max": 28458.28656619501, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9468, + "qps_1t_runs": [ + 2290.17397480596, + 2333.795269558051, + 2352.842082347129 + ], + "qps_1t": 2333.795269558051, + "qps_1t_min": 2290.17397480596, + "qps_1t_max": 2352.842082347129, + "median_ms": 0.4180999821983278, + "p95_ms": 0.5196149984840304, + "p99_ms": 0.6176080455770718, + "mean_ms": 0.430050849817538, + "knob": 96, + "batch": { + "recall": 0.9468, + "qps_batch_runs": [ + 19483.720374828627, + 20839.80234978227, + 20803.363482419667 + ], + "qps_batch": 20803.363482419667, + "qps_batch_min": 19483.720374828627, + "qps_batch_max": 20839.80234978227, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.96425, + "qps_1t_runs": [ + 1833.2582024325486, + 1775.4250651327163, + 1820.0640949159829 + ], + "qps_1t": 1820.0640949159829, + "qps_1t_min": 1775.4250651327163, + "qps_1t_max": 1833.2582024325486, + "median_ms": 0.539999979082495, + "p95_ms": 0.6812099512899294, + "p99_ms": 0.8476079930551352, + "mean_ms": 0.5527178329842476, + "knob": 128, + "batch": { + "recall": 0.96425, + "qps_batch_runs": [ + 14550.862177947769, + 14806.336814294576, + 16095.905848644685 + ], + "qps_batch": 14806.336814294576, + "qps_batch_min": 14550.862177947769, + "qps_batch_max": 16095.905848644685, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.979, + "qps_1t_runs": [ + 1266.1291410883366, + 1296.8990303849139, + 1306.9749596369795 + ], + "qps_1t": 1296.8990303849139, + "qps_1t_min": 1266.1291410883366, + "qps_1t_max": 1306.9749596369795, + "median_ms": 0.7648000027984381, + "p95_ms": 0.9273150149965659, + "p99_ms": 1.0611080465605485, + "mean_ms": 0.7753348330055209, + "knob": 192, + "batch": { + "recall": 0.979, + "qps_batch_runs": [ + 10812.751259208284, + 11567.453581478621, + 11369.592481371807 + ], + "qps_batch": 11369.592481371807, + "qps_batch_min": 10812.751259208284, + "qps_batch_max": 11567.453581478621, + "threads": 32 + } + }, + { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.9854, + "qps_1t_runs": [ + 1000.5676723145737, + 1028.7305937045494, + 1019.9870019656166 + ], + "qps_1t": 1019.9870019656166, + "qps_1t_min": 1000.5676723145737, + "qps_1t_max": 1028.7305937045494, + "median_ms": 0.9744999988470227, + "p95_ms": 1.1896099778823555, + "p99_ms": 1.3780320150544874, + "mean_ms": 0.9839697000473583, + "knob": 256, + "batch": { + "recall": 0.9854, + "qps_batch_runs": [ + 8865.224648840747, + 8830.080550985269, + 9006.915509959501 + ], + "qps_batch": 8865.224648840747, + "qps_batch_min": 8830.080550985269, + "qps_batch_max": 9006.915509959501, + "threads": 32 + } + }, + { + "nq": 1759, + "recall_queries": 1759, + "recall": 0.9913018760659466, + "qps_1t_runs": [ + 656.1471496655383, + 679.2696743304351, + 691.571198001111 + ], + "qps_1t": 679.2696743304351, + "qps_1t_min": 656.1471496655383, + "qps_1t_max": 691.571198001111, + "median_ms": 1.455599965993315, + "p95_ms": 1.805500010959804, + "p99_ms": 2.2556999907828836, + "mean_ms": 1.4807334846264728, + "knob": 400, + "batch": { + "recall": 0.99115, + "qps_batch_runs": [ + 5737.157301596366, + 6206.819929218143, + 5836.416906433218 + ], + "qps_batch": 5836.416906433218, + "qps_batch_min": 5737.157301596366, + "qps_batch_max": 6206.819929218143, + "threads": 32 + } + }, + { + "nq": 1355, + "recall_queries": 1355, + "recall": 0.9950553505535056, + "qps_1t_runs": [ + 451.45550569803294, + 459.5257985769428, + 443.3981628480064 + ], + "qps_1t": 451.45550569803294, + "qps_1t_min": 443.3981628480064, + "qps_1t_max": 459.5257985769428, + "median_ms": 2.1974999690428376, + "p95_ms": 2.7249800274148583, + "p99_ms": 3.2800079789012715, + "mean_ms": 2.2155077986792646, + "knob": 640, + "batch": { + "recall": 0.9947, + "qps_batch_runs": [ + 4035.423756900103, + 3628.4345173594993, + 3704.1193883276096 + ], + "qps_batch": 3704.1193883276096, + "qps_batch_min": 3628.4345173594993, + "qps_batch_max": 4035.423756900103, + "threads": 32 + } + } + ], + "mutation": { + "ref_knob": 128, + "recall_before": 0.96425, + "insert_10k": { + "seconds": 1.1832042000023648, + "per_s": 8451.626523959274, + "batch": 1000 + }, + "update_10k": { + "seconds": 1.1856134999543428, + "per_s": 8434.451868492635, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.0029568999889306724, + "per_s": 3381920.266980819, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 274.125, + "peak_mb": 430.4609375 + }, + "first_query_after_ms": 3.576699993573129, + "after": { + "nq": 2000, + "recall_queries": 2000, + "recall": 0.95805, + "qps_1t_runs": [ + 1604.565823716765, + 1642.7042660657337, + 1636.1476065261588 + ], + "qps_1t": 1636.1476065261588, + "qps_1t_min": 1604.565823716765, + "qps_1t_max": 1642.7042660657337, + "median_ms": 0.6064499903004616, + "p95_ms": 0.755919993389398, + "p99_ms": 0.9202139644185084, + "mean_ms": 0.6143885498750024 + }, + "recall_after": 0.95805 + }, + "saveload": { + "save_s": 0.1780498999869451, + "disk_mb": 176.71466445922852, + "load_s": 0.1356891999603249, + "first_query_after_load_ms": 0.9485000045970082, + "time_to_ready_s": 0.13663769996492192 + }, + "load_before_pct": 5.878125000000004, + "load_contaminated": false, + "filter": { + "build_with_metadata_s": 5.679011999978684, + "mem": { + "rss_mb": 220.6875, + "peak_mb": 251.28515625 + }, + "subsets": { + "f10": { + "unsupported": true + }, + "f01": { + "unsupported": true + } + } + } + } +} \ No newline at end of file diff --git a/benchmarks/competitive/results/tables_cells.md b/benchmarks/competitive/results/tables_cells.md new file mode 100644 index 0000000..9c7f821 --- /dev/null +++ b/benchmarks/competitive/results/tables_cells.md @@ -0,0 +1,1715 @@ +### text384, N = 100,000 + +Queries/second at recall@10 = 0.90 / 0.95 / 0.99, log-linearly interpolated between the two measured sweep points that bracket the target (no extrapolation; `*` = the two bracketing points are more than 0.05 recall apart, so the measured point above the target is shown instead, a conservative value; `-` = never reached; exact scans appear at their single recall-1.0 point). 1T = one pinned P-core, one query at a time. All-core = batched call on 32 logical CPUs (`†` = the ISONGraph ports have no batch API, so this is the same one-query loop with automatic scan threads). Latency columns are for the first measured sweep point with recall >= 0.95. + +| system | QPS 1T @0.90 | @0.95 | @0.99 | all-core QPS @0.90 | @0.95 | 1T latency ms median / p95 / p99 | +|---|---|---|---|---|---|---| +| **ISONGraph C++, /arch:AVX2 (exact)** | 23.0 | 23.0 | 23.0 | - | - | 43.5 / 45.4 / 47.6 (recall 1.000) | +| **ISONGraph C++, /arch:AVX2 (IVF)** | 1697* | 957 | 114 | - | - | 1.10 / 1.30 / 1.48 (recall 0.954) | +| **ISONGraph C++ (exact)** | 22.6 | 22.6 | 22.6 | - | - | 44.2 / 47.2 / 57.7 (recall 1.000) | +| **ISONGraph C++ (IVF)** | 1232* | 681 | 93.3 | - | - | 1.55 / 1.81 / 2.00 (recall 0.954) | +| **ISONGraph C# (exact)** | 39.1 | 39.1 | 39.1 | - | - | 23.6 / 26.7 / 78.4 (recall 1.000) | +| **ISONGraph C# (IVF)** | 1134* | 615 | 112 | - | - | 1.66 / 2.00 / 2.22 (recall 0.952) | +| **ISONGraph Python (exact)** | 131 | 131 | 131 | - | - | 7.37 / 8.92 / 14.3 (recall 1.000) | +| **ISONGraph Python (IVF)** | 937* | 530 | 190 | - | - | 1.93 / 2.34 / 2.83 (recall 0.954) | +| **ISONGraph JS (exact)** | 19.7 | 19.7 | 19.7 | - | - | 50.8 / 55.8 / 58.1 (recall 1.000) | +| **ISONGraph JS (IVF)** | 358* | 201 | 51.3 | - | - | 5.11 / 6.06 / 6.71 (recall 0.953) | +| **ISONGraph Rust (exact)** | 154 | 154 | 154 | - | - | 6.41 / 6.99 / 7.86 (recall 1.000) | +| **ISONGraph Rust (IVF)** | 1562* | 828 | 253 | 1667* † | 1518 † | 1.25 / 1.51 / 1.63 (recall 0.953) | +| **ISONGraph Rust, target-cpu=native (exact)** | 163 | 163 | 163 | - | - | 6.06 / 6.79 / 7.39 (recall 1.000) | +| **ISONGraph Rust, target-cpu=native (IVF)** | 1506* | 802 | 254 | 1676* † | 1484 † | 1.30 / 1.57 / 1.70 (recall 0.953) | +| chromadb (HNSW) | 1029* | 906 | 602 | 4130* | 2890 | 1.14 / 1.34 / 1.54 (recall 0.965) | +| faiss IndexFlatIP | 131 | 131 | 131 | 5437 | 5437 | 7.29 / 8.84 / 15.0 (recall 1.000) | +| faiss HNSWFlat | 5913* | 3542 | 1329 | 30747* | 19930 | 0.296 / 0.377 / 0.553 (recall 0.957) | +| faiss IVFFlat | 2078* | 1193 | 317 | 11708* | 6189 | 0.866 / 1.07 / 1.26 (recall 0.955) | +| hnswlib | 5400 | 3490 | 1275 | 29766 | 19875 | 0.354 / 0.440 / 0.587 (recall 0.964) | +| lancedb (no index) | 7.19 | 7.19 | 7.19 | - | - | 137 / 151 / 173 (recall 1.000) | +| lancedb IVF_FLAT | 256* | 222 | 120 | - | - | 4.52 / 5.16 / 6.06 (recall 0.956) | +| numpy brute force | 144 | 144 | 144 | 1553 | 1553 | 6.75 / 7.66 / 10.1 (recall 1.000) | +| qdrant-client local mode (exact numpy) | 7.24 | 7.24 | 7.24 | - | - | 136 / 152 / 158 (recall 1.000) | +| sqlite-vec (vec0, exact) | 10.2 | 10.2 | 10.2 | - | - | 95.5 / 115 / 122 (recall 1.000) | +| usearch (HNSW) | 3624 | 2230 | 743 | 29559 | 19546 | 0.540 / 0.681 / 0.848 (recall 0.964) | + +Build and memory (RSS growth over the process baseline after imports; the raw vectors are streamed in from disk and are not held by the harness). + +| system | ingest s | index s | total build s | 3 builds: median (min-max) s | ready s (incl. first query) | steady RSS +MB | peak RSS +MB (build) | +|---|---|---|---|---|---|---|---| +| **ISONGraph C++, /arch:AVX2 (exact)** | 0.081 | 0.000 | 0.081 | - | 0.125 | 192 | 206 | +| **ISONGraph C++, /arch:AVX2 (IVF)** | 0.081 | 0.525 | 0.607 | - | 0.607 | 342 | 417 | +| **ISONGraph C++ (exact)** | 0.077 | 0.000 | 0.077 | - | 0.122 | 192 | 206 | +| **ISONGraph C++ (IVF)** | 0.077 | 0.891 | 0.968 | - | 0.968 | 342 | 417 | +| **ISONGraph C# (exact)** | 0.018 | 0.000 | 0.018 | - | 0.065 | 291 | 291 | +| **ISONGraph C# (IVF)** | 0.016 | 1.27 | 1.28 | - | 1.28 | 788 | 788 | +| **ISONGraph Python (exact)** | 2.01 | 0.000 | 2.01 | 1.96 (1.90-2.01) | 2.31 | 454 | 571 | +| **ISONGraph Python (IVF)** | 1.97 | 1.60 | 3.56 | 3.56 (3.48-3.57) | 3.57 | 481 | 693 | +| **ISONGraph JS (exact)** | 0.030 | 0.000 | 0.030 | - | 0.095 | 518 | 518 | +| **ISONGraph JS (IVF)** | 0.027 | 56.7 | 56.7 | - | 56.7 | 1,216 | 1,216 | +| **ISONGraph Rust (exact)** | 0.056 | 0.000 | 0.056 | - | 0.136 | 339 | 339 | +| **ISONGraph Rust (IVF)** | 0.056 | 0.480 | 0.616 | - | 0.616 | 342 | 420 | +| **ISONGraph Rust, target-cpu=native (exact)** | 0.058 | 0.000 | 0.058 | - | 0.137 | 339 | 339 | +| **ISONGraph Rust, target-cpu=native (IVF)** | 0.058 | 0.471 | 0.609 | - | 0.609 | 342 | 420 | +| chromadb (HNSW) | 14.6 | 0.000 | 14.6 | 14.8 (14.6-15.0) | 14.7 | 486 | 385 | +| faiss IndexFlatIP | 0.108 | 0.000 | 0.108 | 0.108 (0.108-0.111) | 0.117 | 172 | 317 | +| faiss HNSWFlat | 8.90 | 0.000 | 8.90 | 8.67 (8.41-8.90) | 8.90 | 184 | 328 | +| faiss IVFFlat | 0.000 | 1.72 | 1.72 | 1.83 (1.72-1.85) | 1.74 | 238 | 459 | +| hnswlib | 6.21 | 0.000 | 6.21 | 6.21 (6.09-6.21) | 6.21 | 194 | 226 | +| lancedb (no index) | 0.219 | 0.000 | 0.219 | 0.212 (0.211-0.219) | 0.284 | 149 | 224 | +| lancedb IVF_FLAT | 0.234 | 1.66 | 1.89 | 1.80 (1.77-1.89) | 1.98 | 339 | 487 | +| numpy brute force | 0.030 | 0.000 | 0.030 | 0.030 (0.030-0.030) | 0.037 | 159 | 191 | +| qdrant-client local mode (exact numpy) | 284 | 0.000 | 284 | - | 284 | 399 | 567 | +| sqlite-vec (vec0, exact) | 0.967 | 0.000 | 0.967 | 0.964 (0.952-0.967) | 1.07 | 4 | 62 | +| usearch (HNSW) | 5.86 | 0.000 | 5.86 | 5.86 (5.73-6.01) | 5.87 | 182 | 213 | + +
Full accuracy sweeps (recall@10, single-thread QPS, latency, all-core QPS) + +**ISONGraph C++, /arch:AVX2 (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 23.0 (23.0-23.1) | 43.5 | 45.4 | 47.6 | - | 68 | + +**ISONGraph C++, /arch:AVX2 (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2808 | 17264 (17032-17852) | 0.056 | 0.069 | 0.080 | - | 2000 | +| 2 | 0.4022 | 14310 (14254-14358) | 0.068 | 0.086 | 0.102 | - | 2000 | +| 4 | 0.5291 | 11488 (11327-11627) | 0.085 | 0.110 | 0.132 | - | 2000 | +| 8 | 0.6479 | 8130 (7995-8407) | 0.118 | 0.158 | 0.212 | - | 2000 | +| 16 | 0.7548 | 5301 (5296-5315) | 0.183 | 0.243 | 0.305 | - | 2000 | +| 32 | 0.8427 | 3053 (3021-3054) | 0.319 | 0.414 | 0.486 | - | 2000 | +| 64 | 0.9091 | 1697 (1694-1723) | 0.576 | 0.717 | 0.869 | - | 2000 | +| 128 | 0.9543 | 902 (900-906) | 1.10 | 1.30 | 1.48 | - | 2000 | +| 256 | 0.9813 | 455 (452-462) | 2.15 | 2.59 | 2.98 | - | 985 | +| 1265 | 1.0000 | 23.0 (23.0-23.2) | 43.4 | 45.5 | 46.5 | - | 65 | + +**ISONGraph C++ (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 22.6 (22.5-23.3) | 44.2 | 47.2 | 57.7 | - | 66 | + +**ISONGraph C++ (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2808 | 16276 (16014-17282) | 0.059 | 0.077 | 0.090 | - | 2000 | +| 2 | 0.4022 | 14590 (14424-14658) | 0.067 | 0.089 | 0.110 | - | 2000 | +| 4 | 0.5291 | 10815 (10091-10998) | 0.091 | 0.127 | 0.151 | - | 2000 | +| 8 | 0.6479 | 6399 (6091-6902) | 0.152 | 0.206 | 0.247 | - | 2000 | +| 16 | 0.7548 | 3908 (3831-3928) | 0.250 | 0.336 | 0.409 | - | 2000 | +| 32 | 0.8427 | 2255 (2200-2287) | 0.436 | 0.570 | 0.649 | - | 2000 | +| 64 | 0.9091 | 1232 (1225-1234) | 0.806 | 0.993 | 1.10 | - | 2000 | +| 128 | 0.9542 | 641 (639-645) | 1.55 | 1.81 | 2.00 | - | 1869 | +| 256 | 0.9811 | 331 (330-332) | 3.01 | 3.38 | 3.66 | - | 1016 | +| 1265 | 1.0000 | 22.5 (22.4-22.6) | 44.5 | 46.8 | 48.1 | - | 67 | + +**ISONGraph C# (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 39.1 (38.6-39.9) | 23.6 | 26.7 | 78.4 | - | 128 | + +**ISONGraph C# (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2776 | 9706 (5430-9894) | 0.104 | 0.212 | 0.231 | - | 2000 | +| 2 | 0.3992 | 8524 (8519-8604) | 0.113 | 0.136 | 0.157 | - | 2000 | +| 4 | 0.5201 | 6881 (6817-6932) | 0.143 | 0.176 | 0.200 | - | 2000 | +| 8 | 0.6376 | 4982 (4893-5001) | 0.197 | 0.248 | 0.277 | - | 2000 | +| 16 | 0.7452 | 3315 (3315-3349) | 0.296 | 0.372 | 0.411 | - | 2000 | +| 32 | 0.8359 | 1975 (1971-2019) | 0.494 | 0.620 | 0.690 | - | 2000 | +| 64 | 0.9060 | 1134 (1127-1135) | 0.878 | 1.07 | 1.16 | - | 2000 | +| 128 | 0.9521 | 598 (591-600) | 1.66 | 2.00 | 2.22 | - | 1848 | +| 256 | 0.9805 | 306 (305-306) | 3.27 | 3.69 | 3.90 | - | 897 | +| 1265 | 1.0000 | 38.6 (37.4-39.4) | 23.7 | 25.1 | 95.1 | - | 122 | + +**ISONGraph Python (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 0.9999 | 131 (129-131) | 7.37 | 8.92 | 14.3 | - | 411 | + +**ISONGraph Python (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2833 | 8545 (8514-8573) | 0.114 | 0.139 | 0.175 | - | 2000 | +| 2 | 0.4083 | 7156 (6794-7226) | 0.135 | 0.175 | 0.216 | - | 2000 | +| 4 | 0.5305 | 5400 (5019-5451) | 0.180 | 0.230 | 0.324 | - | 2000 | +| 8 | 0.6501 | 3877 (3854-4006) | 0.245 | 0.302 | 0.351 | - | 2000 | +| 16 | 0.7542 | 2640 (2575-2669) | 0.369 | 0.458 | 0.593 | - | 2000 | +| 32 | 0.8428 | 1581 (1579-1603) | 0.591 | 0.741 | 1.08 | - | 2000 | +| 64 | 0.9091 | 937 (927-947) | 1.03 | 1.26 | 1.68 | - | 2000 | +| 128 | 0.9540 | 501 (500-506) | 1.93 | 2.34 | 2.83 | - | 1546 | +| 256 | 0.9809 | 268 (268-271) | 3.59 | 4.14 | 5.35 | - | 814 | +| 1265 | 0.9999 | 131 (130-132) | 7.36 | 8.47 | 14.2 | - | 422 | + +**ISONGraph JS (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 19.7 (19.1-20.3) | 50.8 | 55.8 | 58.1 | - | 56 | + +**ISONGraph JS (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2654 | 2125 (2089-2145) | 0.467 | 0.532 | 0.590 | - | 2000 | +| 2 | 0.3867 | 1927 (1927-1931) | 0.513 | 0.597 | 0.674 | - | 2000 | +| 4 | 0.5155 | 1637 (1632-1653) | 0.602 | 0.718 | 0.794 | - | 2000 | +| 8 | 0.6369 | 1304 (1294-1307) | 0.763 | 0.897 | 0.968 | - | 2000 | +| 16 | 0.7420 | 933 (929-937) | 1.07 | 1.27 | 1.40 | - | 2000 | +| 32 | 0.8353 | 607 (605-609) | 1.65 | 1.95 | 2.11 | - | 1779 | +| 64 | 0.9045 | 358 (357-359) | 2.77 | 3.29 | 3.55 | - | 1172 | +| 128 | 0.9527 | 194 (193-195) | 5.11 | 6.06 | 6.71 | - | 607 | +| 256 | 0.9828 | 103 (100-103) | 9.70 | 11.3 | 12.3 | - | 319 | +| 1265 | 1.0000 | 19.5 (19.4-20.0) | 50.4 | 54.3 | 59.1 | - | 59 | + +**ISONGraph Rust (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 154 (154-154) | 6.41 | 6.99 | 7.86 | - | 476 | + +**ISONGraph Rust (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2730 | 24266 (23139-24374) | 0.041 | 0.053 | 0.061 | 22169 † | 2000 | +| 2 | 0.3918 | 17640 (17404-17743) | 0.056 | 0.071 | 0.079 | 17989 † | 2000 | +| 4 | 0.5168 | 12425 (12047-12885) | 0.079 | 0.102 | 0.116 | 12368 † | 2000 | +| 8 | 0.6402 | 7664 (7441-8108) | 0.126 | 0.169 | 0.201 | 8336 † | 2000 | +| 16 | 0.7450 | 5069 (4964-5091) | 0.196 | 0.251 | 0.285 | 5069 † | 2000 | +| 32 | 0.8346 | 2840 (2774-2848) | 0.349 | 0.452 | 0.509 | 2891 † | 2000 | +| 64 | 0.9071 | 1562 (1537-1569) | 0.634 | 0.800 | 0.870 | 1667 † | 2000 | +| 128 | 0.9529 | 793 (791-796) | 1.25 | 1.51 | 1.63 | 1509 † | 2000 | +| 256 | 0.9807 | 401 (393-402) | 2.50 | 2.88 | 3.20 | 1084 † | 1148 | +| 1265 | 1.0000 | 154 (153-157) | 6.32 | 7.30 | 8.65 | 300 † | 478 | + +**ISONGraph Rust, target-cpu=native (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 163 (161-164) | 6.06 | 6.79 | 7.39 | - | 501 | + +**ISONGraph Rust, target-cpu=native (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2730 | 24669 (23997-24957) | 0.040 | 0.051 | 0.060 | 24453 † | 2000 | +| 2 | 0.3918 | 17732 (17481-17842) | 0.056 | 0.071 | 0.081 | 18452 † | 2000 | +| 4 | 0.5168 | 13134 (12835-13139) | 0.075 | 0.099 | 0.111 | 12737 † | 2000 | +| 8 | 0.6402 | 8489 (8363-8498) | 0.116 | 0.153 | 0.177 | 8381 † | 2000 | +| 16 | 0.7450 | 4821 (4660-4993) | 0.203 | 0.269 | 0.310 | 5055 † | 2000 | +| 32 | 0.8346 | 2789 (2748-2792) | 0.355 | 0.458 | 0.508 | 2838 † | 2000 | +| 64 | 0.9071 | 1506 (1480-1529) | 0.656 | 0.830 | 0.914 | 1676 † | 2000 | +| 128 | 0.9529 | 769 (761-774) | 1.30 | 1.57 | 1.70 | 1472 † | 2000 | +| 256 | 0.9807 | 387 (385-389) | 2.58 | 2.94 | 3.20 | 1095 † | 1127 | +| 1265 | 1.0000 | 161 (161-163) | 6.05 | 6.99 | 8.32 | 310 † | 470 | + +**chromadb (HNSW)** (ef_search) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.6149 | 1361 (1337-1385) | 0.702 | 0.850 | 0.974 | 10714 | 2000 | +| 16 | 0.7140 | 1254 (1229-1255) | 0.787 | 0.910 | 1.07 | 8882 | 2000 | +| 32 | 0.8313 | 1170 (1131-1187) | 0.848 | 0.988 | 1.28 | 6354 | 2000 | +| 64 | 0.9152 | 1029 (1013-1034) | 0.964 | 1.12 | 1.39 | 4130 | 2000 | +| 128 | 0.9646 | 858 (851-898) | 1.14 | 1.34 | 1.54 | 2489 | 2000 | +| 256 | 0.9870 | 680 (676-703) | 1.44 | 1.66 | 1.96 | 1445 | 2000 | +| 512 | 0.9955 | 483 (466-490) | 2.06 | 2.41 | 2.86 | 821 | 1479 | + +**faiss IndexFlatIP** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 0.9999 | 131 (130-135) | 7.29 | 8.84 | 15.0 | 5437 | 410 | + +**faiss HNSWFlat** (efSearch) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.6361 | 15713 (15200-17057) | 0.058 | 0.088 | 0.118 | 96809 | 2000 | +| 16 | 0.7315 | 12783 (12779-12881) | 0.073 | 0.105 | 0.136 | 70072 | 2000 | +| 24 | 0.8014 | 9414 (9092-9648) | 0.097 | 0.135 | 0.217 | 50335 | 2000 | +| 32 | 0.8492 | 7879 (7578-8243) | 0.121 | 0.164 | 0.210 | 44864 | 2000 | +| 48 | 0.9004 | 5913 (5763-5919) | 0.165 | 0.209 | 0.261 | 30747 | 2000 | +| 64 | 0.9279 | 4554 (3996-4704) | 0.217 | 0.305 | 0.395 | 24862 | 2000 | +| 96 | 0.9569 | 3273 (3103-3312) | 0.296 | 0.377 | 0.553 | 18591 | 2000 | +| 128 | 0.9714 | 2370 (2277-2432) | 0.386 | 0.519 | 0.929 | 14274 | 2000 | +| 192 | 0.9842 | 1701 (1642-1724) | 0.558 | 0.691 | 1.04 | 10206 | 2000 | +| 256 | 0.9909 | 1282 (1249-1318) | 0.742 | 0.967 | 1.59 | 7844 | 2000 | +| 400 | 0.9958 | 823 (781-839) | 1.17 | 1.50 | 2.42 | 5285 | 2000 | +| 640 | 0.9983 | 508 (492-515) | 1.89 | 2.44 | 4.01 | 3847 | 1585 | + +**faiss IVFFlat** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2834 | 18763 (18006-18898) | 0.052 | 0.065 | 0.082 | 192,036 | 2000 | +| 2 | 0.4099 | 16033 (16022-16181) | 0.060 | 0.076 | 0.095 | 172,377 | 2000 | +| 4 | 0.5287 | 12664 (11616-12949) | 0.077 | 0.101 | 0.141 | 116,179 | 2000 | +| 8 | 0.6503 | 9036 (8998-9414) | 0.107 | 0.134 | 0.171 | 84704 | 2000 | +| 16 | 0.7551 | 6072 (5829-6281) | 0.157 | 0.199 | 0.249 | 38405 | 2000 | +| 32 | 0.8439 | 3761 (3701-3838) | 0.255 | 0.324 | 0.385 | 23327 | 2000 | +| 64 | 0.9096 | 2078 (2072-2081) | 0.458 | 0.571 | 0.689 | 11708 | 2000 | +| 128 | 0.9549 | 1116 (1103-1154) | 0.866 | 1.07 | 1.26 | 5728 | 2000 | +| 256 | 0.9841 | 567 (566-570) | 1.72 | 2.00 | 2.43 | 2721 | 1702 | +| 1265 | 0.9999 | 120 (118-121) | 8.10 | 9.46 | 12.5 | 505 | 377 | + +**hnswlib** (ef) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.6185 | 16855 (16407-17226) | 0.055 | 0.082 | 0.104 | 93453 | 2000 | +| 16 | 0.7160 | 13172 (13035-13312) | 0.072 | 0.105 | 0.134 | 75207 | 2000 | +| 24 | 0.7870 | 9860 (9650-9987) | 0.096 | 0.132 | 0.176 | 52454 | 2000 | +| 32 | 0.8309 | 7740 (7450-8230) | 0.116 | 0.161 | 0.244 | 46850 | 2000 | +| 48 | 0.8855 | 6085 (5802-6165) | 0.157 | 0.206 | 0.266 | 33393 | 2000 | +| 64 | 0.9162 | 4726 (4337-4839) | 0.202 | 0.273 | 0.377 | 26177 | 2000 | +| 96 | 0.9488 | 3563 (3414-3576) | 0.280 | 0.344 | 0.428 | 20267 | 2000 | +| 128 | 0.9645 | 2715 (2704-2728) | 0.354 | 0.440 | 0.587 | 15705 | 2000 | +| 192 | 0.9809 | 1959 (1930-1962) | 0.504 | 0.615 | 0.777 | 10961 | 2000 | +| 256 | 0.9869 | 1558 (1553-1565) | 0.634 | 0.759 | 0.862 | 8428 | 2000 | +| 400 | 0.9933 | 1026 (1017-1037) | 0.949 | 1.17 | 1.46 | 6018 | 2000 | +| 640 | 0.9970 | 696 (690-697) | 1.41 | 1.74 | 2.27 | 3946 | 1874 | + +**lancedb (no index)** (nprobes) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 7.19 (7.16-7.22) | 137 | 151 | 173 | - | 30 | + +**lancedb IVF_FLAT** (nprobes) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2415 | 243 (198-288) | 3.28 | 4.56 | 12.3 | - | 924 | +| 2 | 0.3584 | 300 (298-301) | 3.24 | 4.06 | 5.79 | - | 912 | +| 4 | 0.4898 | 285 (282-300) | 3.30 | 4.06 | 6.32 | - | 933 | +| 8 | 0.6160 | 294 (286-299) | 3.33 | 3.94 | 5.10 | - | 843 | +| 16 | 0.7339 | 286 (278-288) | 3.41 | 4.03 | 5.67 | - | 903 | +| 32 | 0.8351 | 266 (265-278) | 3.59 | 4.31 | 5.36 | - | 786 | +| 64 | 0.9090 | 256 (251-257) | 3.86 | 4.43 | 5.06 | - | 790 | +| 128 | 0.9556 | 218 (212-220) | 4.52 | 5.16 | 6.06 | - | 652 | +| 256 | 0.9828 | 177 (168-178) | 5.61 | 6.48 | 7.70 | - | 500 | +| 1265 | 0.9999 | 70.3 (69.0-71.1) | 14.0 | 15.6 | 17.4 | - | 210 | + +**numpy brute force** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 0.9999 | 144 (143-146) | 6.75 | 7.66 | 10.1 | 1553 | 448 | + +**qdrant-client local mode (exact numpy)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 7.24 (7.14-7.25) | 136 | 152 | 158 | - | 30 | + +**sqlite-vec (vec0, exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 10.2 (10.2-10.3) | 95.5 | 115 | 122 | - | 30 | + +**usearch (HNSW)** (expansion_search) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.6119 | 11865 (10700-12258) | 0.080 | 0.127 | 0.172 | 97939 | 2000 | +| 16 | 0.7104 | 8530 (7906-8812) | 0.109 | 0.166 | 0.218 | 79790 | 2000 | +| 24 | 0.7848 | 6731 (6269-6756) | 0.140 | 0.198 | 0.268 | 58748 | 2000 | +| 32 | 0.8294 | 5372 (5330-5488) | 0.172 | 0.229 | 0.280 | 48489 | 2000 | +| 48 | 0.8844 | 4092 (4055-4107) | 0.235 | 0.302 | 0.368 | 33537 | 2000 | +| 64 | 0.9149 | 3228 (3164-3248) | 0.299 | 0.378 | 0.453 | 26211 | 2000 | +| 96 | 0.9468 | 2334 (2290-2353) | 0.418 | 0.520 | 0.618 | 20803 | 2000 | +| 128 | 0.9643 | 1820 (1775-1833) | 0.540 | 0.681 | 0.848 | 14806 | 2000 | +| 192 | 0.9790 | 1297 (1266-1307) | 0.765 | 0.927 | 1.06 | 11370 | 2000 | +| 256 | 0.9854 | 1020 (1001-1029) | 0.974 | 1.19 | 1.38 | 8865 | 2000 | +| 400 | 0.9913 | 679 (656-692) | 1.46 | 1.81 | 2.26 | 5836 | 1759 | +| 640 | 0.9951 | 451 (443-460) | 2.20 | 2.72 | 3.28 | 3704 | 1355 | + +
+ +### glove, N = 100,000 + +Queries/second at recall@10 = 0.90 / 0.95 / 0.99, log-linearly interpolated between the two measured sweep points that bracket the target (no extrapolation; `*` = the two bracketing points are more than 0.05 recall apart, so the measured point above the target is shown instead, a conservative value; `-` = never reached; exact scans appear at their single recall-1.0 point). 1T = one pinned P-core, one query at a time. All-core = batched call on 32 logical CPUs (`†` = the ISONGraph ports have no batch API, so this is the same one-query loop with automatic scan threads). Latency columns are for the first measured sweep point with recall >= 0.95. + +| system | QPS 1T @0.90 | @0.95 | @0.99 | all-core QPS @0.90 | @0.95 | 1T latency ms median / p95 / p99 | +|---|---|---|---|---|---|---| +| **ISONGraph C++, /arch:AVX2 (exact)** | 40.2 | 40.2 | 40.2 | - | - | 24.4 / 28.5 / 31.7 (recall 1.000) | +| **ISONGraph C++, /arch:AVX2 (IVF)** | 2655* | 2167 | 162 | - | - | 0.708 / 0.882 / 1.03 (recall 0.974) | +| **ISONGraph C++ (exact)** | 40.1 | 40.1 | 40.1 | - | - | 24.5 / 28.4 / 29.7 (recall 1.000) | +| **ISONGraph C++ (IVF)** | 2361* | 1919 | 150 | - | - | 0.781 / 1.00 / 1.20 (recall 0.974) | +| **ISONGraph C# (exact)** | 108 | 108 | 108 | - | - | 7.79 / 26.4 / 35.6 (recall 1.000) | +| **ISONGraph C# (IVF)** | 1677* | 1268 | 220 | - | - | 1.14 / 1.43 / 1.62 (recall 0.973) | +| **ISONGraph Python (exact)** | 471 | 471 | 471 | - | - | 1.98 / 2.72 / 4.89 (recall 1.000) | +| **ISONGraph Python (IVF)** | 1035* | 813 | 477 | - | - | 1.73 / 2.31 / 3.53 (recall 0.973) | +| **ISONGraph JS (exact)** | 48.1 | 48.1 | 48.1 | - | - | 20.1 / 24.6 / 27.1 (recall 1.000) | +| **ISONGraph JS (IVF)** | 432* | 331 | 91.8 | - | - | 4.37 / 5.46 / 6.29 (recall 0.975) | +| **ISONGraph Rust (exact)** | 700 | 700 | 700 | - | - | 1.42 / 1.79 / 2.14 (recall 1.000) | +| **ISONGraph Rust (IVF)** | 2595* | 2029 | 888 | 2638* † | 2381 † | 0.733 / 0.912 / 1.01 (recall 0.974) | +| **ISONGraph Rust, target-cpu=native (exact)** | 621 | 621 | 621 | - | - | 1.57 / 1.94 / 2.25 (recall 1.000) | +| **ISONGraph Rust, target-cpu=native (IVF)** | 2584* | 2028 | 851 | 2610* † | 2334 † | 0.725 / 0.909 / 1.02 (recall 0.974) | +| chromadb (HNSW) | 956* | 933 | - | 3684* | 3468 | 1.29 / 1.50 / 1.92 (recall 0.977) | +| faiss IndexFlatIP | 548 | 548 | 548 | 16430 | 16430 | 1.72 / 2.23 / 3.01 (recall 1.000) | +| faiss HNSWFlat | 4756 | 2152 | - | 94888 | 51124 | 0.679 / 0.908 / 1.71 (recall 0.970) | +| faiss IVFFlat | 3719* | 2893 | 785 | 53383* | 40104 | 0.457 / 0.603 / 0.990 (recall 0.974) | +| hnswlib | 7183 | 3762 | - | 70103 | 48987 | 0.366 / 0.452 / 0.573 (recall 0.969) | +| lancedb (no index) | 34.3 | 34.3 | 34.3 | - | - | 29.5 / 34.2 / 37.4 (recall 1.000) | +| lancedb IVF_FLAT | 253* | 238 | 145 | - | - | 4.45 / 5.51 / 7.19 (recall 0.974) | +| numpy brute force | 528 | 528 | 528 | 1733 | 1733 | 1.82 / 2.53 / 3.70 (recall 1.000) | +| sqlite-vec (vec0, exact) | 40.1 | 40.1 | 40.1 | - | - | 24.3 / 27.4 / 34.3 (recall 1.000) | +| usearch (HNSW) | 3763 | 1947 | - | 58683 | 27876 | 0.720 / 0.949 / 1.18 (recall 0.967) | + +Build and memory (RSS growth over the process baseline after imports; the raw vectors are streamed in from disk and are not held by the harness). + +| system | ingest s | index s | total build s | 3 builds: median (min-max) s | ready s (incl. first query) | steady RSS +MB | peak RSS +MB (build) | +|---|---|---|---|---|---|---|---| +| **ISONGraph C++, /arch:AVX2 (exact)** | 0.042 | 0.000 | 0.042 | - | 0.069 | 72 | 84 | +| **ISONGraph C++, /arch:AVX2 (IVF)** | 0.042 | 0.253 | 0.295 | - | 0.295 | 111 | 131 | +| **ISONGraph C++ (exact)** | 0.038 | 0.000 | 0.038 | - | 0.064 | 72 | 84 | +| **ISONGraph C++ (IVF)** | 0.038 | 0.290 | 0.328 | - | 0.328 | 112 | 132 | +| **ISONGraph C# (exact)** | 0.017 | 0.000 | 0.017 | - | 0.035 | 87 | 87 | +| **ISONGraph C# (IVF)** | 0.011 | 0.602 | 0.613 | - | 0.613 | 249 | 249 | +| **ISONGraph Python (exact)** | 0.934 | 0.000 | 0.934 | 0.931 (0.914-0.934) | 1.11 | 159 | 164 | +| **ISONGraph Python (IVF)** | 0.929 | 0.848 | 1.78 | 1.78 (1.73-1.78) | 1.78 | 176 | 278 | +| **ISONGraph JS (exact)** | 0.029 | 0.000 | 0.029 | - | 0.053 | 308 | 308 | +| **ISONGraph JS (IVF)** | 0.027 | 20.6 | 20.7 | - | 20.7 | 642 | 645 | +| **ISONGraph Rust (exact)** | 0.054 | 0.000 | 0.054 | - | 0.094 | 121 | 121 | +| **ISONGraph Rust (IVF)** | 0.054 | 0.212 | 0.307 | - | 0.307 | 125 | 144 | +| **ISONGraph Rust, target-cpu=native (exact)** | 0.053 | 0.000 | 0.053 | - | 0.095 | 121 | 121 | +| **ISONGraph Rust, target-cpu=native (IVF)** | 0.053 | 0.176 | 0.270 | - | 0.270 | 125 | 144 | +| chromadb (HNSW) | 9.46 | 0.000 | 9.46 | 9.40 (9.34-9.46) | 9.55 | 264 | 202 | +| faiss IndexFlatIP | 0.031 | 0.000 | 0.031 | 0.035 (0.031-0.047) | 0.033 | 63 | 101 | +| faiss HNSWFlat | 2.25 | 0.000 | 2.25 | 2.28 (2.25-2.41) | 2.26 | 76 | 112 | +| faiss IVFFlat | 0.000 | 0.973 | 0.973 | 1.08 (0.973-1.09) | 0.976 | 91 | 155 | +| hnswlib | 1.33 | 0.000 | 1.33 | 1.31 (1.28-1.33) | 1.34 | 88 | 96 | +| lancedb (no index) | 0.083 | 0.000 | 0.083 | 0.080 (0.077-0.083) | 0.105 | 100 | 122 | +| lancedb IVF_FLAT | 0.085 | 0.643 | 0.729 | 0.697 (0.693-0.729) | 0.797 | 173 | 201 | +| numpy brute force | 0.012 | 0.000 | 0.012 | 0.012 (0.012-0.012) | 0.014 | 51 | 61 | +| sqlite-vec (vec0, exact) | 0.529 | 0.000 | 0.529 | 0.521 (0.493-0.529) | 0.554 | 4 | 19 | +| usearch (HNSW) | 1.72 | 0.000 | 1.72 | 1.72 (1.62-1.72) | 1.73 | 75 | 84 | + +
Full accuracy sweeps (recall@10, single-thread QPS, latency, all-core QPS) + +**ISONGraph C++, /arch:AVX2 (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 40.2 (39.7-41.2) | 24.4 | 28.5 | 31.7 | - | 126 | + +**ISONGraph C++, /arch:AVX2 (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2848 | 35648 (33711-35885) | 0.028 | 0.034 | 0.039 | - | 2000 | +| 2 | 0.4074 | 30432 (29216-30692) | 0.033 | 0.041 | 0.046 | - | 2000 | +| 4 | 0.5255 | 25325 (24998-25645) | 0.039 | 0.049 | 0.055 | - | 2000 | +| 8 | 0.6348 | 19291 (19117-19436) | 0.051 | 0.066 | 0.075 | - | 2000 | +| 16 | 0.7357 | 13080 (13064-13094) | 0.075 | 0.098 | 0.120 | - | 2000 | +| 32 | 0.8185 | 8058 (7722-8187) | 0.123 | 0.161 | 0.185 | - | 2000 | +| 64 | 0.8845 | 4787 (4719-4842) | 0.205 | 0.265 | 0.309 | - | 2000 | +| 128 | 0.9386 | 2655 (2628-2693) | 0.371 | 0.464 | 0.524 | - | 2000 | +| 256 | 0.9741 | 1408 (1345-1414) | 0.708 | 0.882 | 1.03 | - | 2000 | +| 1265 | 1.0000 | 41.6 (41.5-41.7) | 23.5 | 26.8 | 27.9 | - | 125 | + +**ISONGraph C++ (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 40.1 (39.7-40.1) | 24.5 | 28.4 | 29.7 | - | 123 | + +**ISONGraph C++ (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2848 | 31630 (24601-32671) | 0.032 | 0.047 | 0.058 | - | 2000 | +| 2 | 0.4074 | 27775 (27261-27919) | 0.035 | 0.044 | 0.060 | - | 2000 | +| 4 | 0.5255 | 22908 (22737-23047) | 0.042 | 0.055 | 0.074 | - | 2000 | +| 8 | 0.6348 | 16744 (16490-17301) | 0.057 | 0.077 | 0.099 | - | 2000 | +| 16 | 0.7357 | 11815 (11676-12044) | 0.082 | 0.110 | 0.136 | - | 2000 | +| 32 | 0.8185 | 7116 (6877-7441) | 0.136 | 0.187 | 0.224 | - | 2000 | +| 64 | 0.8845 | 4281 (4218-4369) | 0.227 | 0.304 | 0.353 | - | 2000 | +| 128 | 0.9386 | 2361 (2314-2397) | 0.411 | 0.547 | 0.648 | - | 2000 | +| 256 | 0.9741 | 1235 (1234-1247) | 0.781 | 1.00 | 1.20 | - | 2000 | +| 1265 | 1.0000 | 39.7 (39.7-40.3) | 24.6 | 27.8 | 29.0 | - | 116 | + +**ISONGraph C# (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 108 (107-108) | 7.79 | 26.4 | 35.6 | - | 387 | + +**ISONGraph C# (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2868 | 14066 (6337-14394) | 0.069 | 0.212 | 0.232 | - | 2000 | +| 2 | 0.4092 | 13476 (13083-13665) | 0.073 | 0.084 | 0.091 | - | 2000 | +| 4 | 0.5273 | 12150 (11932-12195) | 0.081 | 0.096 | 0.106 | - | 2000 | +| 8 | 0.6336 | 9682 (9672-9773) | 0.100 | 0.123 | 0.139 | - | 2000 | +| 16 | 0.7249 | 7268 (7001-7362) | 0.136 | 0.175 | 0.195 | - | 2000 | +| 32 | 0.8096 | 4612 (4436-4695) | 0.215 | 0.276 | 0.309 | - | 2000 | +| 64 | 0.8780 | 2983 (2918-2984) | 0.334 | 0.423 | 0.482 | - | 2000 | +| 128 | 0.9323 | 1677 (1632-1700) | 0.594 | 0.748 | 0.829 | - | 2000 | +| 256 | 0.9727 | 886 (826-886) | 1.14 | 1.43 | 1.62 | - | 2000 | +| 1265 | 1.0000 | 98.7 (95.5-101) | 7.90 | 38.7 | 49.3 | - | 329 | + +**ISONGraph Python (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 471 (468-487) | 1.98 | 2.72 | 4.89 | - | 1462 | + +**ISONGraph Python (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2853 | 14708 (14670-16768) | 0.059 | 0.083 | 0.117 | - | 2000 | +| 2 | 0.4022 | 12857 (12777-14101) | 0.069 | 0.092 | 0.130 | - | 2000 | +| 4 | 0.5192 | 9844 (9812-10234) | 0.094 | 0.127 | 0.166 | - | 2000 | +| 8 | 0.6309 | 6339 (6092-6998) | 0.133 | 0.184 | 0.248 | - | 2000 | +| 16 | 0.7282 | 4918 (4775-4947) | 0.198 | 0.261 | 0.303 | - | 2000 | +| 32 | 0.8134 | 3161 (3152-3178) | 0.313 | 0.396 | 0.454 | - | 2000 | +| 64 | 0.8796 | 1876 (1867-1876) | 0.521 | 0.658 | 0.771 | - | 2000 | +| 128 | 0.9362 | 1035 (1011-1071) | 0.930 | 1.18 | 1.70 | - | 2000 | +| 256 | 0.9728 | 547 (535-558) | 1.73 | 2.31 | 3.53 | - | 1803 | +| 1265 | 1.0000 | 440 (435-446) | 2.09 | 3.11 | 5.46 | - | 1414 | + +**ISONGraph JS (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 48.1 (47.8-48.9) | 20.1 | 24.6 | 27.1 | - | 167 | + +**ISONGraph JS (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2795 | 6775 (6501-6778) | 0.145 | 0.184 | 0.233 | - | 2000 | +| 2 | 0.3983 | 5710 (5550-5862) | 0.171 | 0.224 | 0.250 | - | 2000 | +| 4 | 0.5165 | 4523 (4349-4604) | 0.218 | 0.290 | 0.328 | - | 2000 | +| 8 | 0.6258 | 3533 (3512-3553) | 0.279 | 0.371 | 0.415 | - | 2000 | +| 16 | 0.7241 | 2301 (2293-2328) | 0.426 | 0.580 | 0.676 | - | 2000 | +| 32 | 0.8079 | 1401 (1350-1412) | 0.709 | 0.968 | 1.13 | - | 2000 | +| 64 | 0.8763 | 779 (764-799) | 1.27 | 1.69 | 1.90 | - | 2000 | +| 128 | 0.9321 | 432 (431-447) | 2.28 | 2.96 | 3.26 | - | 1366 | +| 256 | 0.9745 | 230 (221-232) | 4.37 | 5.46 | 6.29 | - | 651 | +| 1265 | 1.0000 | 50.8 (49.5-51.3) | 19.1 | 23.1 | 24.9 | - | 154 | + +**ISONGraph Rust (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 700 (660-705) | 1.42 | 1.79 | 2.14 | - | 2000 | + +**ISONGraph Rust (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2851 | 52956 (49100-55464) | 0.018 | 0.025 | 0.037 | 53680 † | 2000 | +| 2 | 0.4016 | 40791 (40768-41483) | 0.024 | 0.031 | 0.038 | 42250 † | 2000 | +| 4 | 0.5209 | 32168 (32032-32233) | 0.030 | 0.041 | 0.050 | 32288 † | 2000 | +| 8 | 0.6351 | 21635 (21002-22195) | 0.045 | 0.062 | 0.076 | 21857 † | 2000 | +| 16 | 0.7308 | 13906 (13495-14167) | 0.070 | 0.097 | 0.120 | 14291 † | 2000 | +| 32 | 0.8134 | 8542 (8508-8573) | 0.115 | 0.154 | 0.180 | 8303 † | 2000 | +| 64 | 0.8817 | 4864 (4744-4875) | 0.205 | 0.267 | 0.301 | 4573 † | 2000 | +| 128 | 0.9355 | 2595 (2484-2619) | 0.384 | 0.507 | 0.570 | 2638 † | 2000 | +| 256 | 0.9740 | 1350 (1347-1359) | 0.733 | 0.912 | 1.01 | 2010 † | 2000 | +| 1265 | 1.0000 | 684 (673-693) | 1.43 | 1.72 | 2.13 | 1703 † | 1978 | + +**ISONGraph Rust, target-cpu=native (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 621 (616-627) | 1.57 | 1.94 | 2.25 | - | 1749 | + +**ISONGraph Rust, target-cpu=native (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2851 | 52724 (50519-53187) | 0.019 | 0.025 | 0.029 | 52173 † | 2000 | +| 2 | 0.4016 | 40953 (40918-43267) | 0.024 | 0.030 | 0.033 | 42748 † | 2000 | +| 4 | 0.5209 | 31286 (27148-32777) | 0.032 | 0.045 | 0.055 | 32611 † | 2000 | +| 8 | 0.6351 | 21679 (21020-22277) | 0.045 | 0.059 | 0.068 | 20838 † | 2000 | +| 16 | 0.7308 | 13851 (13535-13940) | 0.072 | 0.095 | 0.108 | 14078 † | 2000 | +| 32 | 0.8134 | 8218 (8199-8559) | 0.118 | 0.154 | 0.171 | 7957 † | 2000 | +| 64 | 0.8817 | 4859 (4614-4861) | 0.207 | 0.267 | 0.297 | 4645 † | 2000 | +| 128 | 0.9355 | 2584 (2419-2716) | 0.382 | 0.512 | 0.589 | 2610 † | 2000 | +| 256 | 0.9740 | 1358 (1352-1383) | 0.725 | 0.909 | 1.02 | 1940 † | 2000 | +| 1265 | 1.0000 | 636 (627-640) | 1.55 | 1.84 | 2.04 | 1625 † | 2000 | + +**chromadb (HNSW)** (ef_search) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.5396 | 1412 (1390-1486) | 0.717 | 0.801 | 0.950 | 23630 | 2000 | +| 16 | 0.6245 | 1404 (1357-1450) | 0.714 | 0.815 | 0.942 | 20762 | 2000 | +| 32 | 0.7422 | 1367 (1291-1381) | 0.753 | 0.860 | 0.953 | 15763 | 2000 | +| 64 | 0.8288 | 1153 (1132-1160) | 0.861 | 0.966 | 1.12 | 9913 | 2000 | +| 128 | 0.8957 | 1042 (1034-1051) | 0.940 | 1.09 | 1.35 | 6356 | 2000 | +| 256 | 0.9470 | 956 (933-981) | 1.03 | 1.21 | 1.43 | 3684 | 2000 | +| 512 | 0.9767 | 753 (749-776) | 1.29 | 1.50 | 1.92 | 2029 | 2000 | + +**faiss IndexFlatIP** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 548 (543-570) | 1.72 | 2.23 | 3.01 | 16430 | 1809 | + +**faiss HNSWFlat** (efSearch) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.5377 | 32429 (29418-34917) | 0.028 | 0.043 | 0.063 | 823,215 | 2000 | +| 16 | 0.6264 | 25691 (25351-27102) | 0.036 | 0.053 | 0.068 | 502,740 | 2000 | +| 24 | 0.6956 | 20186 (19218-20931) | 0.047 | 0.068 | 0.088 | 499,650 | 2000 | +| 32 | 0.7423 | 17280 (15500-17482) | 0.057 | 0.079 | 0.108 | 381,767 | 2000 | +| 48 | 0.7966 | 11435 (11370-12845) | 0.078 | 0.108 | 0.177 | 257,689 | 2000 | +| 64 | 0.8334 | 9621 (8971-9751) | 0.100 | 0.137 | 0.240 | 211,013 | 2000 | +| 96 | 0.8744 | 6390 (4882-6712) | 0.149 | 0.208 | 0.355 | 120,766 | 2000 | +| 128 | 0.9023 | 4629 (4219-4853) | 0.200 | 0.274 | 0.430 | 92811 | 2000 | +| 192 | 0.9305 | 3206 (3076-3243) | 0.296 | 0.401 | 0.833 | 66248 | 2000 | +| 256 | 0.9483 | 2237 (2230-2250) | 0.408 | 0.539 | 1.09 | 54004 | 2000 | +| 400 | 0.9696 | 1381 (1379-1435) | 0.679 | 0.908 | 1.71 | 27177 | 2000 | +| 640 | 0.9842 | 785 (780-789) | 1.23 | 1.58 | 2.59 | 17016 | 2000 | + +**faiss IVFFlat** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2827 | 51821 (50764-51839) | 0.019 | 0.022 | 0.029 | 310,318 | 2000 | +| 2 | 0.4049 | 44673 (41408-45531) | 0.021 | 0.026 | 0.042 | 382,168 | 2000 | +| 4 | 0.5251 | 35842 (34299-39054) | 0.026 | 0.034 | 0.042 | 275,217 | 2000 | +| 8 | 0.6316 | 27988 (27219-28595) | 0.034 | 0.043 | 0.056 | 246,831 | 2000 | +| 16 | 0.7287 | 19426 (11660-19775) | 0.049 | 0.066 | 0.157 | 134,086 | 2000 | +| 32 | 0.8119 | 10764 (10512-12410) | 0.083 | 0.118 | 0.165 | 153,105 | 2000 | +| 64 | 0.8796 | 6348 (5918-6871) | 0.149 | 0.203 | 0.298 | 93354 | 2000 | +| 128 | 0.9332 | 3719 (3637-3728) | 0.256 | 0.334 | 0.498 | 53383 | 2000 | +| 256 | 0.9738 | 2031 (1979-2051) | 0.457 | 0.603 | 0.990 | 26799 | 2000 | +| 1265 | 1.0000 | 438 (428-438) | 2.18 | 2.75 | 4.53 | 6400 | 1387 | + +**hnswlib** (ef) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.5421 | 44724 (41330-46904) | 0.022 | 0.031 | 0.040 | 373,518 | 2000 | +| 16 | 0.6244 | 36871 (34289-36877) | 0.027 | 0.038 | 0.044 | 328,116 | 2000 | +| 24 | 0.6949 | 27706 (27313-29071) | 0.035 | 0.048 | 0.058 | 212,972 | 2000 | +| 32 | 0.7392 | 23477 (23317-23597) | 0.042 | 0.055 | 0.065 | 171,278 | 2000 | +| 48 | 0.7937 | 17284 (15501-17469) | 0.059 | 0.078 | 0.090 | 132,860 | 2000 | +| 64 | 0.8287 | 12918 (12807-14084) | 0.073 | 0.095 | 0.112 | 150,099 | 2000 | +| 96 | 0.8706 | 9092 (7599-9613) | 0.103 | 0.134 | 0.216 | 103,974 | 2000 | +| 128 | 0.8950 | 7641 (7197-7770) | 0.131 | 0.164 | 0.195 | 71297 | 2000 | +| 192 | 0.9278 | 5112 (5038-5168) | 0.190 | 0.238 | 0.320 | 63883 | 2000 | +| 256 | 0.9466 | 3983 (3913-4050) | 0.243 | 0.310 | 0.412 | 51910 | 2000 | +| 400 | 0.9692 | 2709 (2670-2731) | 0.366 | 0.452 | 0.573 | 35142 | 2000 | +| 640 | 0.9843 | 1769 (1750-1809) | 0.559 | 0.693 | 0.853 | 23069 | 2000 | + +**lancedb (no index)** (nprobes) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 34.3 (31.2-34.5) | 29.5 | 34.2 | 37.4 | - | 86 | + +**lancedb IVF_FLAT** (nprobes) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2556 | 308 (307-310) | 3.20 | 3.67 | 4.23 | - | 950 | +| 2 | 0.3718 | 309 (304-314) | 3.18 | 3.74 | 4.39 | - | 922 | +| 4 | 0.4942 | 309 (307-309) | 3.20 | 3.71 | 4.47 | - | 987 | +| 8 | 0.6105 | 302 (297-304) | 3.25 | 3.85 | 4.72 | - | 959 | +| 16 | 0.7110 | 305 (304-310) | 3.24 | 3.66 | 4.37 | - | 922 | +| 32 | 0.8030 | 298 (297-300) | 3.33 | 3.74 | 4.41 | - | 887 | +| 64 | 0.8755 | 283 (278-284) | 3.50 | 3.90 | 4.58 | - | 880 | +| 128 | 0.9348 | 253 (247-256) | 3.84 | 4.68 | 5.99 | - | 786 | +| 256 | 0.9740 | 217 (207-219) | 4.45 | 5.51 | 7.19 | - | 670 | +| 1265 | 1.0000 | 113 (112-114) | 8.54 | 10.2 | 16.3 | - | 352 | + +**numpy brute force** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 528 (479-537) | 1.82 | 2.53 | 3.70 | 1733 | 1607 | + +**sqlite-vec (vec0, exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 40.1 (40.0-40.4) | 24.3 | 27.4 | 34.3 | - | 125 | + +**usearch (HNSW)** (expansion_search) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.5376 | 27475 (26481-27781) | 0.035 | 0.050 | 0.059 | 293,591 | 2000 | +| 16 | 0.6238 | 21039 (19802-21251) | 0.046 | 0.067 | 0.095 | 282,602 | 2000 | +| 24 | 0.6930 | 15794 (15665-16627) | 0.060 | 0.086 | 0.108 | 227,663 | 2000 | +| 32 | 0.7346 | 12833 (11838-13645) | 0.073 | 0.102 | 0.130 | 190,170 | 2000 | +| 48 | 0.7900 | 9549 (9154-10111) | 0.101 | 0.141 | 0.171 | 121,533 | 2000 | +| 64 | 0.8295 | 7238 (7210-7919) | 0.130 | 0.181 | 0.227 | 108,436 | 2000 | +| 96 | 0.8690 | 5186 (5094-5409) | 0.181 | 0.241 | 0.329 | 80683 | 2000 | +| 128 | 0.8950 | 3966 (3948-4152) | 0.240 | 0.316 | 0.449 | 60907 | 2000 | +| 192 | 0.9264 | 2848 (2796-2901) | 0.345 | 0.447 | 0.549 | 48221 | 2000 | +| 256 | 0.9456 | 2135 (1926-2190) | 0.471 | 0.637 | 0.736 | 29190 | 2000 | +| 400 | 0.9667 | 1375 (1372-1376) | 0.720 | 0.949 | 1.18 | 23417 | 2000 | +| 640 | 0.9819 | 876 (853-896) | 1.14 | 1.47 | 1.80 | 15709 | 2000 | + +
+ +### sift, N = 100,000 + +Queries/second at recall@10 = 0.90 / 0.95 / 0.99, log-linearly interpolated between the two measured sweep points that bracket the target (no extrapolation; `*` = the two bracketing points are more than 0.05 recall apart, so the measured point above the target is shown instead, a conservative value; `-` = never reached; exact scans appear at their single recall-1.0 point). 1T = one pinned P-core, one query at a time. All-core = batched call on 32 logical CPUs (`†` = the ISONGraph ports have no batch API, so this is the same one-query loop with automatic scan threads). Latency columns are for the first measured sweep point with recall >= 0.95. + +| system | QPS 1T @0.90 | @0.95 | @0.99 | all-core QPS @0.90 | @0.95 | 1T latency ms median / p95 / p99 | +|---|---|---|---|---|---|---| +| **ISONGraph C++, /arch:AVX2 (exact)** | 23.4 | 23.4 | 23.4 | - | - | 42.3 / 45.4 / 47.4 (recall 1.000) | +| **ISONGraph C++, /arch:AVX2 (IVF)** | 4693* | 4693* | 2451 | - | - | 0.211 / 0.293 / 0.336 (recall 0.952) | +| **ISONGraph C++ (exact)** | 22.8 | 22.8 | 22.8 | - | - | 44.2 / 47.5 / 49.5 (recall 1.000) | +| **ISONGraph C++ (IVF)** | 4054* | 4054* | 2083 | - | - | 0.241 / 0.336 / 0.393 (recall 0.952) | +| **ISONGraph C# (exact)** | 55.0 | 55.0 | 55.0 | - | - | 14.6 / 42.8 / 80.8 (recall 1.000) | +| **ISONGraph C# (IVF)** | 2805* | 2805* | 1635 | - | - | 0.362 / 0.482 / 0.555 (recall 0.953) | +| **ISONGraph Python (exact)** | 318 | 318 | 318 | - | - | 3.22 / 4.42 / 5.68 (recall 1.000) | +| **ISONGraph Python (IVF)** | 2722* | 2622 | 1395 | - | - | 0.570 / 0.778 / 1.08 (recall 0.987) | +| **ISONGraph JS (exact)** | 29.0 | 29.0 | 29.0 | - | - | 33.7 / 41.6 / 48.7 (recall 1.000) | +| **ISONGraph JS (IVF)** | 810* | 795 | 416 | - | - | 2.10 / 2.73 / 3.12 (recall 0.988) | +| **ISONGraph Rust (exact)** | 344 | 344 | 344 | - | - | 2.86 / 3.36 / 3.84 (recall 1.000) | +| **ISONGraph Rust (IVF)** | 4980* | 4873 | 2247 | 2648* † | 2592 † | 0.365 / 0.508 / 0.626 (recall 0.986) | +| **ISONGraph Rust, target-cpu=native (exact)** | 364 | 364 | 364 | - | - | 2.62 / 3.14 / 3.42 (recall 1.000) | +| **ISONGraph Rust, target-cpu=native (IVF)** | 4741* | 4650 | 2335 | 2523* † | 2470 † | 0.357 / 0.476 / 0.545 (recall 0.986) | +| chromadb (HNSW) | 1163* | 1163* | 1107 | 12847* | 12847* | 0.818 / 1.11 / 1.83 (recall 0.952) | +| faiss IndexFlatIP | 283 | 283 | 283 | 4537 | 4537 | 3.30 / 4.26 / 6.04 (recall 1.000) | +| faiss HNSWFlat | 21197* | 17392 | 8549 | 354,906* | 331,856 | 0.056 / 0.078 / 0.121 (recall 0.953) | +| faiss IVFFlat | 8764* | 8764* | 4401 | 112,537* | 112,537* | 0.105 / 0.160 / 0.248 (recall 0.950) | +| hnswlib | 26532* | 22439 | 11479 | 281,005* | 193,015 | 0.044 / 0.061 / 0.079 (recall 0.950) | +| lancedb (no index) | 15.1 | 15.1 | 15.1 | - | - | 66.9 / 78.5 / 82.0 (recall 1.000) | +| lancedb IVF_FLAT | 247* | 247 | 231 | - | - | 4.06 / 5.08 / 6.78 (recall 0.987) | +| numpy brute force | 282 | 282 | 282 | 1543 | 1543 | 3.34 / 4.87 / 6.69 (recall 1.000) | +| sqlite-vec (vec0, exact) | 21.6 | 21.6 | 21.6 | - | - | 47.0 / 52.6 / 57.7 (recall 1.000) | +| usearch (HNSW) | 16246* | 13215 | 5873 | 160,195* | 106,020 | 0.072 / 0.101 / 0.133 (recall 0.952) | + +Build and memory (RSS growth over the process baseline after imports; the raw vectors are streamed in from disk and are not held by the harness). + +| system | ingest s | index s | total build s | 3 builds: median (min-max) s | ready s (incl. first query) | steady RSS +MB | peak RSS +MB (build) | +|---|---|---|---|---|---|---|---| +| **ISONGraph C++, /arch:AVX2 (exact)** | 0.049 | 0.000 | 0.049 | - | 0.079 | 84 | 102 | +| **ISONGraph C++, /arch:AVX2 (IVF)** | 0.049 | 0.638 | 0.687 | - | 0.687 | 135 | 160 | +| **ISONGraph C++ (exact)** | 0.050 | 0.000 | 0.050 | - | 0.081 | 84 | 102 | +| **ISONGraph C++ (IVF)** | 0.050 | 0.970 | 1.02 | - | 1.02 | 134 | 160 | +| **ISONGraph C# (exact)** | 0.018 | 0.000 | 0.018 | - | 0.041 | 108 | 108 | +| **ISONGraph C# (IVF)** | 0.028 | 1.48 | 1.51 | - | 1.51 | 311 | 311 | +| **ISONGraph Python (exact)** | 1.26 | 0.000 | 1.26 | 4.28 (1.26-4.49) | 1.48 | 170 | 195 | +| **ISONGraph Python (IVF)** | 1.11 | 0.965 | 2.08 | 2.08 (2.02-2.08) | 2.08 | 192 | 303 | +| **ISONGraph JS (exact)** | 0.035 | 0.000 | 0.035 | - | 0.063 | 317 | 317 | +| **ISONGraph JS (IVF)** | 0.044 | 35.4 | 35.5 | - | 35.5 | 651 | 651 | +| **ISONGraph Rust (exact)** | 0.055 | 0.000 | 0.055 | - | 0.106 | 143 | 143 | +| **ISONGraph Rust (IVF)** | 0.055 | 0.565 | 0.671 | - | 0.671 | 146 | 172 | +| **ISONGraph Rust, target-cpu=native (exact)** | 0.057 | 0.000 | 0.057 | - | 0.115 | 143 | 143 | +| **ISONGraph Rust, target-cpu=native (IVF)** | 0.057 | 0.575 | 0.690 | - | 0.690 | 146 | 172 | +| chromadb (HNSW) | 9.06 | 0.000 | 9.06 | 9.15 (9.06-9.23) | 9.16 | 289 | 219 | +| faiss IndexFlatIP | 0.051 | 0.000 | 0.051 | 0.082 (0.051-0.088) | 0.054 | 74 | 122 | +| faiss HNSWFlat | 1.98 | 0.000 | 1.98 | 2.00 (1.98-2.09) | 1.98 | 85 | 132 | +| faiss IVFFlat | 0.000 | 1.29 | 1.29 | 1.26 (1.25-1.29) | 1.30 | 105 | 181 | +| hnswlib | 1.13 | 0.000 | 1.13 | 1.13 (1.09-1.20) | 1.13 | 97 | 108 | +| lancedb (no index) | 0.103 | 0.000 | 0.103 | 0.223 (0.103-0.247) | 0.137 | 107 | 125 | +| lancedb IVF_FLAT | 0.113 | 0.858 | 0.971 | 0.869 (0.830-0.971) | 1.06 | 189 | 230 | +| numpy brute force | 0.015 | 0.000 | 0.015 | 0.015 (0.015-0.018) | 0.018 | 62 | 74 | +| sqlite-vec (vec0, exact) | 0.556 | 0.000 | 0.556 | 1.99 (0.556-2.16) | 0.588 | 4 | 23 | +| usearch (HNSW) | 1.57 | 0.000 | 1.57 | 1.56 (1.34-1.57) | 1.57 | 85 | 96 | + +
Full accuracy sweeps (recall@10, single-thread QPS, latency, all-core QPS) + +**ISONGraph C++, /arch:AVX2 (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 23.4 (23.3-27.3) | 42.3 | 45.4 | 47.4 | - | 97 | + +**ISONGraph C++, /arch:AVX2 (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2796 | 23360 (21573-23763) | 0.045 | 0.056 | 0.066 | - | 2000 | +| 2 | 0.4219 | 19563 (18794-20324) | 0.051 | 0.066 | 0.077 | - | 2000 | +| 4 | 0.5793 | 16494 (16250-17277) | 0.059 | 0.078 | 0.091 | - | 2000 | +| 8 | 0.7401 | 11995 (11642-12084) | 0.082 | 0.112 | 0.127 | - | 2000 | +| 16 | 0.8696 | 7631 (7570-7635) | 0.128 | 0.177 | 0.203 | - | 2000 | +| 32 | 0.9522 | 4693 (4524-4768) | 0.211 | 0.293 | 0.336 | - | 2000 | +| 64 | 0.9882 | 2719 (2711-2761) | 0.361 | 0.484 | 0.553 | - | 2000 | +| 128 | 0.9984 | 1488 (1403-1494) | 0.675 | 0.885 | 1.02 | - | 2000 | +| 256 | 0.9999 | 787 (783-833) | 1.24 | 1.52 | 1.67 | - | 2000 | +| 1265 | 1.0000 | 23.0 (22.9-23.3) | 43.3 | 46.3 | 48.4 | - | 69 | + +**ISONGraph C++ (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 22.8 (22.4-25.2) | 44.2 | 47.5 | 49.5 | - | 92 | + +**ISONGraph C++ (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2796 | 19362 (18833-19708) | 0.052 | 0.065 | 0.072 | - | 2000 | +| 2 | 0.4219 | 16480 (16363-16797) | 0.059 | 0.075 | 0.086 | - | 2000 | +| 4 | 0.5793 | 13533 (13443-13864) | 0.071 | 0.097 | 0.118 | - | 2000 | +| 8 | 0.7401 | 10563 (10319-10609) | 0.092 | 0.129 | 0.154 | - | 2000 | +| 16 | 0.8696 | 6672 (6639-6802) | 0.144 | 0.204 | 0.242 | - | 2000 | +| 32 | 0.9522 | 4054 (3966-4060) | 0.241 | 0.336 | 0.393 | - | 2000 | +| 64 | 0.9882 | 2290 (2202-2307) | 0.430 | 0.605 | 0.739 | - | 2000 | +| 128 | 0.9984 | 1321 (1305-1339) | 0.737 | 1.02 | 1.22 | - | 2000 | +| 256 | 0.9999 | 679 (677-688) | 1.45 | 1.84 | 2.06 | - | 2000 | +| 1265 | 1.0000 | 22.3 (22.3-23.5) | 44.8 | 47.5 | 49.3 | - | 66 | + +**ISONGraph C# (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 55.0 (55.0-72.9) | 14.6 | 42.8 | 80.8 | - | 314 | + +**ISONGraph C# (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2850 | 8528 (5455-9395) | 0.116 | 0.295 | 0.324 | - | 2000 | +| 2 | 0.4301 | 7905 (7740-8001) | 0.124 | 0.142 | 0.156 | - | 2000 | +| 4 | 0.5908 | 6901 (6828-7039) | 0.143 | 0.170 | 0.187 | - | 2000 | +| 8 | 0.7513 | 5823 (5639-5949) | 0.172 | 0.212 | 0.236 | - | 2000 | +| 16 | 0.8749 | 4284 (4243-4379) | 0.231 | 0.294 | 0.331 | - | 2000 | +| 32 | 0.9526 | 2805 (2528-2838) | 0.362 | 0.482 | 0.555 | - | 2000 | +| 64 | 0.9889 | 1727 (1719-1855) | 0.556 | 0.748 | 0.870 | - | 2000 | +| 128 | 0.9984 | 1056 (1029-1103) | 0.915 | 1.28 | 1.76 | - | 2000 | +| 256 | 0.9999 | 622 (597-652) | 1.58 | 2.12 | 2.44 | - | 1547 | +| 1265 | 1.0000 | 53.6 (53.0-53.7) | 15.4 | 40.9 | 87.7 | - | 198 | + +**ISONGraph Python (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 318 (265-323) | 3.22 | 4.42 | 5.68 | - | 1127 | + +**ISONGraph Python (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2856 | 15392 (15349-15420) | 0.061 | 0.084 | 0.121 | - | 2000 | +| 2 | 0.4253 | 13796 (13604-13991) | 0.069 | 0.091 | 0.119 | - | 2000 | +| 4 | 0.5855 | 10215 (10168-10222) | 0.093 | 0.130 | 0.163 | - | 2000 | +| 8 | 0.7427 | 7098 (6537-7152) | 0.136 | 0.186 | 0.230 | - | 2000 | +| 16 | 0.8690 | 4348 (4293-4513) | 0.212 | 0.298 | 0.458 | - | 2000 | +| 32 | 0.9468 | 2722 (2610-2761) | 0.339 | 0.516 | 0.929 | - | 2000 | +| 64 | 0.9869 | 1688 (1663-1711) | 0.570 | 0.778 | 1.08 | - | 2000 | +| 128 | 0.9976 | 880 (876-894) | 1.07 | 1.58 | 2.25 | - | 2000 | +| 256 | 0.9998 | 499 (496-509) | 1.90 | 2.53 | 3.37 | - | 1542 | +| 1265 | 1.0000 | 338 (337-354) | 2.79 | 3.71 | 4.72 | - | 1068 | + +**ISONGraph JS (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 29.0 (27.6-32.5) | 33.7 | 41.6 | 48.7 | - | 106 | + +**ISONGraph JS (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2886 | 3499 (3416-3525) | 0.289 | 0.350 | 0.424 | - | 2000 | +| 2 | 0.4338 | 3127 (3108-3164) | 0.317 | 0.380 | 0.426 | - | 2000 | +| 4 | 0.5894 | 2439 (2407-2486) | 0.399 | 0.524 | 0.617 | - | 2000 | +| 8 | 0.7425 | 1900 (1884-1909) | 0.516 | 0.683 | 0.797 | - | 2000 | +| 16 | 0.8685 | 1319 (1308-1363) | 0.739 | 1.02 | 1.19 | - | 2000 | +| 32 | 0.9486 | 810 (763-820) | 1.24 | 1.70 | 1.93 | - | 2000 | +| 64 | 0.9877 | 480 (464-482) | 2.10 | 2.73 | 3.12 | - | 1328 | +| 128 | 0.9984 | 248 (244-256) | 3.99 | 4.99 | 5.65 | - | 761 | +| 256 | 1.0000 | 142 (141-145) | 7.06 | 8.09 | 8.66 | - | 424 | +| 1265 | 1.0000 | 27.8 (26.8-28.0) | 35.2 | 42.4 | 45.9 | - | 79 | + +**ISONGraph Rust (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 344 (333-398) | 2.86 | 3.36 | 3.84 | - | 1231 | + +**ISONGraph Rust (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2912 | 33756 (33679-34626) | 0.029 | 0.038 | 0.045 | 16984 † | 2000 | +| 2 | 0.4356 | 25889 (25153-26284) | 0.038 | 0.051 | 0.060 | 14003 † | 2000 | +| 4 | 0.5955 | 20026 (19055-22804) | 0.048 | 0.066 | 0.079 | 10165 † | 2000 | +| 8 | 0.7474 | 13525 (13509-13581) | 0.072 | 0.100 | 0.117 | 6768 † | 2000 | +| 16 | 0.8710 | 8413 (8216-8451) | 0.116 | 0.162 | 0.191 | 4405 † | 2000 | +| 32 | 0.9486 | 4980 (4685-5131) | 0.199 | 0.276 | 0.332 | 2648 † | 2000 | +| 64 | 0.9860 | 2725 (2580-2747) | 0.365 | 0.508 | 0.626 | 1464 † | 2000 | +| 128 | 0.9984 | 1490 (1475-1517) | 0.666 | 0.856 | 0.989 | 738 † | 2000 | +| 256 | 0.9998 | 827 (800-837) | 1.20 | 1.50 | 1.66 | 468 † | 2000 | +| 1265 | 1.0000 | 335 (333-339) | 2.96 | 3.43 | 3.81 | 378 † | 1030 | + +**ISONGraph Rust, target-cpu=native (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 364 (362-465) | 2.62 | 3.14 | 3.42 | - | 1261 | + +**ISONGraph Rust, target-cpu=native (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2912 | 36185 (33982-42399) | 0.027 | 0.035 | 0.042 | 16215 † | 2000 | +| 2 | 0.4356 | 27078 (25967-27602) | 0.036 | 0.049 | 0.056 | 14669 † | 2000 | +| 4 | 0.5955 | 20258 (20173-22787) | 0.047 | 0.063 | 0.073 | 10146 † | 2000 | +| 8 | 0.7474 | 13568 (13551-14109) | 0.071 | 0.098 | 0.114 | 7176 † | 2000 | +| 16 | 0.8710 | 8586 (8391-9422) | 0.112 | 0.157 | 0.179 | 4303 † | 2000 | +| 32 | 0.9486 | 4741 (4733-4973) | 0.204 | 0.276 | 0.317 | 2523 † | 2000 | +| 64 | 0.9860 | 2780 (2727-2801) | 0.357 | 0.476 | 0.545 | 1399 † | 2000 | +| 128 | 0.9984 | 1612 (1463-1643) | 0.628 | 0.829 | 0.976 | 744 † | 2000 | +| 256 | 0.9998 | 839 (799-840) | 1.19 | 1.55 | 1.90 | 486 † | 1908 | +| 1265 | 1.0000 | 360 (352-360) | 2.79 | 3.26 | 3.65 | 316 † | 1103 | + +**chromadb (HNSW)** (ef_search) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.7916 | 1279 (1260-1281) | 0.751 | 0.990 | 1.55 | 21363 | 2000 | +| 16 | 0.8714 | 1258 (1240-1307) | 0.766 | 0.934 | 1.69 | 18336 | 2000 | +| 32 | 0.9524 | 1163 (1116-1185) | 0.818 | 1.11 | 1.83 | 12847 | 2000 | +| 64 | 0.9888 | 1129 (1112-1154) | 0.841 | 1.09 | 1.71 | 8413 | 2000 | +| 128 | 0.9978 | 982 (965-1008) | 0.952 | 1.35 | 2.21 | 5786 | 2000 | +| 256 | 0.9998 | 812 (779-818) | 1.19 | 1.66 | 2.62 | 3652 | 2000 | +| 512 | 0.9999 | 764 (757-783) | 1.24 | 1.72 | 2.46 | 2105 | 2000 | + +**faiss IndexFlatIP** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 283 (280-332) | 3.30 | 4.26 | 6.04 | 4537 | 920 | + +**faiss HNSWFlat** (efSearch) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.7926 | 35472 (34557-35531) | 0.027 | 0.038 | 0.053 | 579,257 | 2000 | +| 16 | 0.8737 | 28109 (27242-28189) | 0.034 | 0.048 | 0.061 | 407,672 | 2000 | +| 24 | 0.9257 | 21197 (21007-21337) | 0.045 | 0.063 | 0.084 | 354,906 | 2000 | +| 32 | 0.9533 | 16937 (16706-17368) | 0.056 | 0.078 | 0.121 | 328,882 | 2000 | +| 48 | 0.9778 | 12453 (12093-12677) | 0.078 | 0.109 | 0.149 | 191,734 | 2000 | +| 64 | 0.9889 | 9172 (7963-9527) | 0.101 | 0.148 | 0.263 | 140,578 | 2000 | +| 96 | 0.9958 | 6013 (5876-6248) | 0.150 | 0.228 | 0.409 | 99528 | 2000 | +| 128 | 0.9981 | 4997 (4910-5057) | 0.191 | 0.264 | 0.397 | 90958 | 2000 | +| 192 | 0.9992 | 3318 (3279-3386) | 0.289 | 0.393 | 0.566 | 70793 | 2000 | +| 256 | 0.9997 | 2528 (2456-2663) | 0.380 | 0.508 | 0.801 | 48638 | 2000 | +| 400 | 0.9999 | 1537 (1491-1552) | 0.627 | 0.836 | 1.33 | 29966 | 2000 | + +**faiss IVFFlat** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2837 | 46449 (45779-47166) | 0.021 | 0.025 | 0.031 | 260,845 | 2000 | +| 2 | 0.4257 | 39480 (39424-40291) | 0.024 | 0.031 | 0.041 | 250,401 | 2000 | +| 4 | 0.5849 | 30978 (28532-32812) | 0.030 | 0.042 | 0.062 | 221,614 | 2000 | +| 8 | 0.7447 | 22575 (21736-24142) | 0.041 | 0.058 | 0.083 | 207,689 | 2000 | +| 16 | 0.8686 | 14664 (14010-15618) | 0.064 | 0.095 | 0.132 | 144,067 | 2000 | +| 32 | 0.9505 | 8764 (8547-8823) | 0.105 | 0.160 | 0.248 | 112,537 | 2000 | +| 64 | 0.9876 | 4991 (4987-5213) | 0.181 | 0.279 | 0.610 | 51491 | 2000 | +| 128 | 0.9986 | 2804 (2624-2813) | 0.340 | 0.506 | 0.854 | 29165 | 2000 | +| 256 | 1.0000 | 1438 (1423-1475) | 0.654 | 1.01 | 1.51 | 15785 | 2000 | +| 1265 | 1.0000 | 308 (300-311) | 3.13 | 3.91 | 4.82 | 3462 | 988 | + +**hnswlib** (ef) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.7877 | 44466 (42336-45863) | 0.021 | 0.031 | 0.043 | 392,472 | 2000 | +| 16 | 0.8690 | 35361 (34241-36359) | 0.028 | 0.038 | 0.049 | 310,299 | 2000 | +| 24 | 0.9228 | 26532 (26390-27774) | 0.036 | 0.051 | 0.068 | 281,005 | 2000 | +| 32 | 0.9503 | 22404 (21713-22520) | 0.044 | 0.061 | 0.079 | 192,350 | 2000 | +| 48 | 0.9762 | 16412 (15860-16465) | 0.060 | 0.081 | 0.103 | 175,231 | 2000 | +| 64 | 0.9879 | 12748 (12557-13235) | 0.075 | 0.105 | 0.137 | 130,878 | 2000 | +| 96 | 0.9954 | 8801 (8414-8850) | 0.107 | 0.157 | 0.246 | 99854 | 2000 | +| 128 | 0.9978 | 6789 (6288-7469) | 0.137 | 0.210 | 0.373 | 80265 | 2000 | +| 192 | 0.9990 | 4893 (4496-4938) | 0.193 | 0.308 | 0.601 | 53647 | 2000 | +| 256 | 0.9997 | 4274 (4261-4377) | 0.230 | 0.315 | 0.412 | 47855 | 2000 | +| 400 | 1.0000 | 2940 (2766-2973) | 0.329 | 0.474 | 0.840 | 36822 | 2000 | + +**lancedb (no index)** (nprobes) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 15.1 (13.7-16.0) | 66.9 | 78.5 | 82.0 | - | 43 | + +**lancedb IVF_FLAT** (nprobes) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2785 | 277 (270-279) | 3.55 | 4.48 | 5.87 | - | 835 | +| 2 | 0.4209 | 287 (268-289) | 3.47 | 4.27 | 5.94 | - | 780 | +| 4 | 0.5785 | 276 (270-283) | 3.53 | 4.43 | 5.90 | - | 832 | +| 8 | 0.7388 | 269 (264-281) | 3.59 | 4.54 | 6.01 | - | 825 | +| 16 | 0.8673 | 263 (263-270) | 3.68 | 4.55 | 6.20 | - | 800 | +| 32 | 0.9486 | 247 (247-251) | 3.90 | 4.89 | 7.06 | - | 766 | +| 64 | 0.9868 | 238 (234-244) | 4.06 | 5.08 | 6.78 | - | 731 | +| 128 | 0.9977 | 213 (203-215) | 4.59 | 5.78 | 8.62 | - | 624 | +| 256 | 0.9999 | 186 (185-189) | 5.21 | 6.41 | 8.26 | - | 550 | +| 1265 | 1.0000 | 92.3 (90.6-92.3) | 10.6 | 12.9 | 17.3 | - | 300 | + +**numpy brute force** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 282 (267-297) | 3.34 | 4.87 | 6.69 | 1543 | 842 | + +**sqlite-vec (vec0, exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 21.6 (21.0-22.4) | 47.0 | 52.6 | 57.7 | - | 93 | + +**usearch (HNSW)** (expansion_search) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.7861 | 27069 (26424-27812) | 0.035 | 0.052 | 0.072 | 360,458 | 2000 | +| 16 | 0.8702 | 19975 (19615-21362) | 0.045 | 0.066 | 0.089 | 235,977 | 2000 | +| 24 | 0.9224 | 16246 (15886-16879) | 0.058 | 0.080 | 0.103 | 160,195 | 2000 | +| 32 | 0.9517 | 13048 (12862-13737) | 0.072 | 0.101 | 0.133 | 103,358 | 2000 | +| 48 | 0.9768 | 9040 (8229-9296) | 0.105 | 0.167 | 0.257 | 133,914 | 2000 | +| 64 | 0.9868 | 6439 (6418-7550) | 0.134 | 0.222 | 0.319 | 91078 | 2000 | +| 96 | 0.9939 | 5250 (5048-5360) | 0.181 | 0.274 | 0.393 | 77948 | 2000 | +| 128 | 0.9958 | 4190 (4062-4302) | 0.228 | 0.336 | 0.633 | 32827 | 2000 | +| 192 | 0.9970 | 2769 (2674-3016) | 0.336 | 0.519 | 0.885 | 47475 | 2000 | +| 256 | 0.9975 | 2330 (1993-2437) | 0.422 | 0.630 | 0.957 | 37724 | 2000 | +| 400 | 0.9976 | 1592 (1525-1656) | 0.607 | 0.875 | 1.32 | 25792 | 2000 | +| 640 | 0.9977 | 1064 (1030-1108) | 0.909 | 1.29 | 1.82 | 13865 | 2000 | + +
+ +### text384, N = 1,000,000 + +Queries/second at recall@10 = 0.90 / 0.95 / 0.99, log-linearly interpolated between the two measured sweep points that bracket the target (no extrapolation; `*` = the two bracketing points are more than 0.05 recall apart, so the measured point above the target is shown instead, a conservative value; `-` = never reached; exact scans appear at their single recall-1.0 point). 1T = one pinned P-core, one query at a time. All-core = batched call on 32 logical CPUs (`†` = the ISONGraph ports have no batch API, so this is the same one-query loop with automatic scan threads). Latency columns are for the first measured sweep point with recall >= 0.95. + +| system | QPS 1T @0.90 | @0.95 | @0.99 | all-core QPS @0.90 | @0.95 | 1T latency ms median / p95 / p99 | +|---|---|---|---|---|---|---| +| **ISONGraph C++, /arch:AVX2 (exact)** | 2.13 | 2.13 | 2.13 | - | - | 466 / 491 / 493 (recall 1.000) | +| **ISONGraph C++, /arch:AVX2 (IVF)** | 221* | 128 | - | - | - | 8.38 / 9.75 / 11.4 (recall 0.955) | +| **ISONGraph C++ (exact)** | 2.11 | 2.11 | 2.11 | - | - | 469 / 500 / 516 (recall 1.000) | +| **ISONGraph C++ (IVF)** | 173* | 94.0 | - | - | - | 11.1 / 12.9 / 14.1 (recall 0.953) | +| **ISONGraph C# (exact)** | 3.72 | 3.72 | 3.72 | - | - | 245 / 394 / 398 (recall 1.000) | +| **ISONGraph C# (IVF)** | 141* | - | - | - | - | - | +| **ISONGraph Python (exact)** | 12.7 | 12.7 | 12.7 | - | - | 77.2 / 86.1 / 93.0 (recall 1.000) | +| **ISONGraph Python (IVF)** | 125* | 69.2 | 16.7 | - | - | 14.6 / 18.1 / 23.7 (recall 0.953) | +| **ISONGraph JS (exact)** | 1.67 | 1.67 | 1.67 | - | - | 545 / 838 / 872 (recall 1.000) | +| **ISONGraph JS (IVF)** | 52.0* | - | - | - | - | - | +| **ISONGraph Rust (exact)** | 15.1 | 15.1 | 15.1 | - | - | 65.3 / 70.0 / 80.5 (recall 1.000) | +| **ISONGraph Rust (IVF)** | 192* | 103 | - | 579* † | 356 † | 9.89 / 11.4 / 12.3 (recall 0.951) | +| **ISONGraph Rust, target-cpu=native (exact)** | 15.8 | 15.8 | 15.8 | - | - | 62.7 / 68.9 / 71.3 (recall 1.000) | +| **ISONGraph Rust, target-cpu=native (IVF)** | 190* | 103 | - | 548* † | 342 † | 9.91 / 11.7 / 12.5 (recall 0.952) | +| chromadb (HNSW) | 687* | 605 | - | 1939* | 1464 | 1.83 / 2.15 / 2.93 (recall 0.967) | +| faiss IndexFlatIP | 13.4 | 13.4 | 13.4 | 540 | 540 | 73.4 / 83.8 / 87.6 (recall 1.000) | +| faiss HNSWFlat | 2880 | 1572 | 406 | 10413 | 5757 | 0.796 / 0.979 / 1.24 (recall 0.965) | +| faiss IVFFlat | 321* | 190 | 22.5 | 910* | 520 | 5.63 / 7.03 / 13.2 (recall 0.956) | +| hnswlib | 2663 | 1486 | 474 | 10424 | 5894 | 0.718 / 0.905 / 1.29 (recall 0.957) | +| lancedb (no index) | 0.725 | 0.725 | 0.725 | - | - | 1372 / 1438 / 1463 (recall 1.000) | +| lancedb IVF_FLAT | 95.4* | 11.4* | 11.4* | - | - | 87.4 / 90.4 / 92.8 (recall 1.000) | +| numpy brute force | 14.2 | 14.2 | 14.2 | 142 | 142 | 69.3 / 77.1 / 84.0 (recall 1.000) | +| sqlite-vec (vec0, exact) | 1.05 | 1.05 | 1.05 | - | - | 947 / 988 / 991 (recall 1.000) | +| usearch (HNSW) | 1274 | 706 | 214 | 10254 | 5827 | 1.52 / 1.93 / 2.44 (recall 0.957) | + +Build and memory (RSS growth over the process baseline after imports; the raw vectors are streamed in from disk and are not held by the harness). + +| system | ingest s | index s | total build s | 3 builds: median (min-max) s | ready s (incl. first query) | steady RSS +MB | peak RSS +MB (build) | +|---|---|---|---|---|---|---|---| +| **ISONGraph C++, /arch:AVX2 (exact)** | 0.922 | 0.000 | 0.922 | - | 1.43 | 1,783 | 1,922 | +| **ISONGraph C++, /arch:AVX2 (IVF)** | 0.922 | 6.20 | 7.12 | - | 7.12 | 3,279 | 3,514 | +| **ISONGraph C++ (exact)** | 0.917 | 0.000 | 0.917 | - | 1.43 | 1,783 | 1,922 | +| **ISONGraph C++ (IVF)** | 0.917 | 11.4 | 12.4 | - | 12.4 | 3,279 | 3,514 | +| **ISONGraph C# (exact)** | 0.217 | 0.000 | 0.217 | - | 0.515 | 2,447 | 2,447 | +| **ISONGraph C# (IVF)** | 0.177 | 10.7 | 10.9 | - | 10.9 | 6,656 | 6,664 | +| **ISONGraph Python (exact)** | 19.6 | 0.000 | 19.6 | - | 22.8 | 4,223 | 2,682 | +| **ISONGraph Python (IVF)** | 20.8 | 21.4 | 42.2 | - | 42.2 | 4,251 | 4,795 | +| **ISONGraph JS (exact)** | 0.470 | 0.000 | 0.470 | - | 1.03 | 3,620 | 3,620 | +| **ISONGraph JS (IVF)** | 0.691 | 521 | 522 | - | 522 | 7,002 | 7,002 | +| **ISONGraph Rust (exact)** | 0.812 | 0.000 | 0.812 | - | 1.84 | 3,517 | 3,524 | +| **ISONGraph Rust (IVF)** | 0.812 | 5.71 | 7.55 | - | 7.55 | 3,532 | 3,779 | +| **ISONGraph Rust, target-cpu=native (exact)** | 0.763 | 0.000 | 0.763 | - | 1.83 | 3,517 | 3,525 | +| **ISONGraph Rust, target-cpu=native (IVF)** | 0.763 | 5.56 | 7.40 | - | 7.40 | 3,532 | 3,779 | +| chromadb (HNSW) | 343 | 0.000 | 343 | - | 345 | 4,023 | 2,147 | +| faiss IndexFlatIP | 1.10 | 0.000 | 1.10 | - | 1.18 | 1,553 | 2,122 | +| faiss HNSWFlat | 221 | 0.000 | 221 | - | 221 | 1,640 | 2,168 | +| faiss IVFFlat | 0.000 | 18.5 | 18.5 | - | 18.6 | 1,885 | 3,592 | +| hnswlib | 173 | 0.000 | 173 | - | 173 | 1,830 | 1,860 | +| lancedb (no index) | 2.04 | 0.000 | 2.04 | - | 2.69 | 152 | 226 | +| lancedb IVF_FLAT | 2.06 | 10.7 | 12.8 | - | 13.4 | 1,736 | 3,081 | +| numpy brute force | 0.298 | 0.000 | 0.298 | - | 0.368 | 1,576 | 1,605 | +| sqlite-vec (vec0, exact) | 9.60 | 0.000 | 9.60 | - | 10.5 | 4 | 62 | +| usearch (HNSW) | 174 | 0.000 | 174 | - | 174 | 1,715 | 1,745 | + +
Full accuracy sweeps (recall@10, single-thread QPS, latency, all-core QPS) + +**ISONGraph C++, /arch:AVX2 (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 2.13 (2.13-2.13) | 466 | 491 | 493 | - | 30 | + +**ISONGraph C++, /arch:AVX2 (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2666 | 4780 (4475-4797) | 0.206 | 0.281 | 0.373 | - | 2000 | +| 2 | 0.3725 | 3920 (3909-3942) | 0.246 | 0.330 | 0.381 | - | 2000 | +| 4 | 0.4895 | 3016 (2956-3039) | 0.323 | 0.432 | 0.499 | - | 2000 | +| 8 | 0.6020 | 1989 (1984-2021) | 0.489 | 0.658 | 0.772 | - | 2000 | +| 16 | 0.7007 | 1243 (1230-1245) | 0.795 | 1.02 | 1.15 | - | 2000 | +| 32 | 0.7890 | 713 (696-718) | 1.39 | 1.74 | 2.00 | - | 2000 | +| 64 | 0.8603 | 404 (397-406) | 2.45 | 3.04 | 3.41 | - | 1301 | +| 128 | 0.9141 | 221 (216-223) | 4.50 | 5.33 | 5.77 | - | 714 | +| 256 | 0.9547 | 119 (115-120) | 8.38 | 9.75 | 11.4 | - | 360 | + +**ISONGraph C++ (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 2.11 (2.11-2.11) | 469 | 500 | 516 | - | 30 | + +**ISONGraph C++ (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2669 | 4177 (3635-4194) | 0.239 | 0.334 | 0.550 | - | 2000 | +| 2 | 0.3726 | 3308 (3222-3337) | 0.297 | 0.399 | 0.457 | - | 2000 | +| 4 | 0.4897 | 2387 (2353-2421) | 0.409 | 0.556 | 0.668 | - | 2000 | +| 8 | 0.6026 | 1589 (1576-1602) | 0.613 | 0.819 | 0.954 | - | 2000 | +| 16 | 0.7010 | 961 (954-985) | 1.02 | 1.32 | 1.49 | - | 2000 | +| 32 | 0.7849 | 559 (554-559) | 1.78 | 2.21 | 2.50 | - | 1627 | +| 64 | 0.8613 | 311 (303-313) | 3.16 | 3.87 | 4.28 | - | 969 | +| 128 | 0.9152 | 173 (171-173) | 5.78 | 6.77 | 7.48 | - | 499 | +| 256 | 0.9528 | 89.5 (89.5-90.5) | 11.1 | 12.9 | 14.1 | - | 286 | + +**ISONGraph C# (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 3.72 (3.72-3.72) | 245 | 394 | 398 | - | 30 | + +**ISONGraph C# (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2741 | 2529 (2179-2620) | 0.384 | 0.630 | 0.733 | - | 2000 | +| 2 | 0.3823 | 2129 (2066-2179) | 0.455 | 0.612 | 0.754 | - | 2000 | +| 4 | 0.5001 | 1685 (1601-1685) | 0.591 | 0.783 | 0.916 | - | 2000 | +| 8 | 0.6082 | 1171 (1136-1175) | 0.840 | 1.11 | 1.26 | - | 2000 | +| 16 | 0.7054 | 747 (734-756) | 1.32 | 1.74 | 2.02 | - | 2000 | +| 32 | 0.7822 | 445 (438-446) | 2.23 | 2.83 | 3.18 | - | 1183 | +| 64 | 0.8550 | 253 (250-258) | 3.88 | 4.87 | 5.62 | - | 857 | +| 128 | 0.9096 | 141 (141-145) | 6.93 | 8.40 | 9.48 | - | 426 | +| 256 | 0.9462 | 74.5 (72.8-77.0) | 13.1 | 16.2 | 19.2 | - | 234 | + +**ISONGraph Python (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 12.7 (12.7-12.8) | 77.2 | 86.1 | 93.0 | - | 36 | + +**ISONGraph Python (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2745 | 3293 (3160-3317) | 0.290 | 0.423 | 0.519 | - | 2000 | +| 2 | 0.3764 | 2389 (2356-2438) | 0.393 | 0.569 | 0.712 | - | 2000 | +| 4 | 0.4940 | 1711 (1689-1742) | 0.557 | 0.786 | 1.03 | - | 2000 | +| 8 | 0.6007 | 1166 (1077-1175) | 0.857 | 1.13 | 1.33 | - | 2000 | +| 16 | 0.7006 | 668 (658-687) | 1.44 | 1.88 | 2.25 | - | 2000 | +| 32 | 0.7813 | 396 (385-399) | 2.50 | 3.08 | 3.62 | - | 1191 | +| 64 | 0.8540 | 217 (213-219) | 4.47 | 5.41 | 6.55 | - | 689 | +| 128 | 0.9119 | 125 (125-125) | 7.76 | 9.83 | 15.3 | - | 404 | +| 256 | 0.9532 | 65.8 (65.0-67.4) | 14.6 | 18.1 | 23.7 | - | 211 | +| 4000 | 1.0000 | 11.5 (11.4-12.1) | 83.0 | 99.3 | 139 | - | 37 | + +**ISONGraph JS (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 1.67 (1.67-1.67) | 545 | 838 | 872 | - | 30 | + +**ISONGraph JS (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2670 | 666 (660-668) | 1.47 | 1.74 | 1.99 | - | 1907 | +| 2 | 0.3726 | 594 (587-594) | 1.66 | 2.02 | 2.28 | - | 1634 | +| 4 | 0.4833 | 493 (488-493) | 2.00 | 2.55 | 2.80 | - | 1452 | +| 8 | 0.5923 | 368 (366-368) | 2.70 | 3.40 | 3.68 | - | 1071 | +| 16 | 0.6892 | 245 (244-249) | 4.03 | 5.05 | 5.61 | - | 647 | +| 32 | 0.7796 | 158 (158-161) | 6.23 | 7.67 | 8.31 | - | 465 | +| 64 | 0.8444 | 95.9 (94.8-96.2) | 10.3 | 12.4 | 13.5 | - | 252 | +| 128 | 0.9132 | 52.0 (50.9-54.3) | 18.7 | 23.3 | 25.6 | - | 167 | +| 256 | 0.9488 | 29.0 (28.3-29.4) | 34.3 | 40.4 | 42.1 | - | 84 | + +**ISONGraph Rust (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 15.1 (15.1-15.1) | 65.3 | 70.0 | 80.5 | - | 46 | + +**ISONGraph Rust (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2757 | 5323 (5238-5829) | 0.175 | 0.248 | 0.324 | 5692 † | 2000 | +| 2 | 0.3806 | 4242 (4037-4392) | 0.229 | 0.330 | 0.408 | 4320 † | 2000 | +| 4 | 0.4961 | 3080 (2980-3091) | 0.323 | 0.436 | 0.509 | 2997 † | 2000 | +| 8 | 0.6046 | 1936 (1889-1993) | 0.507 | 0.673 | 0.791 | 1970 † | 2000 | +| 16 | 0.7022 | 1147 (1140-1157) | 0.867 | 1.09 | 1.22 | 1233 † | 2000 | +| 32 | 0.7902 | 648 (647-652) | 1.53 | 1.88 | 2.05 | 1131 † | 1961 | +| 64 | 0.8606 | 356 (353-356) | 2.77 | 3.37 | 4.00 | 867 † | 1083 | +| 128 | 0.9149 | 192 (190-193) | 5.13 | 6.23 | 7.19 | 579 † | 598 | +| 256 | 0.9512 | 100 (100-100) | 9.89 | 11.4 | 12.3 | 350 † | 287 | + +**ISONGraph Rust, target-cpu=native (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 15.8 (15.6-15.8) | 62.7 | 68.9 | 71.3 | - | 49 | + +**ISONGraph Rust, target-cpu=native (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2757 | 5805 (5410-5872) | 0.171 | 0.243 | 0.283 | 6133 † | 2000 | +| 2 | 0.3806 | 4753 (4604-4781) | 0.206 | 0.294 | 0.342 | 4672 † | 2000 | +| 4 | 0.4961 | 3050 (2960-3122) | 0.315 | 0.444 | 0.555 | 3007 † | 2000 | +| 8 | 0.6046 | 1935 (1921-1950) | 0.510 | 0.678 | 0.774 | 1956 † | 2000 | +| 16 | 0.7022 | 1143 (1127-1148) | 0.869 | 1.12 | 1.26 | 1200 † | 2000 | +| 32 | 0.7908 | 647 (639-650) | 1.54 | 1.89 | 2.06 | 1086 † | 1986 | +| 64 | 0.8593 | 354 (352-361) | 2.79 | 3.36 | 3.61 | 872 † | 1149 | +| 128 | 0.9149 | 190 (188-191) | 5.21 | 6.19 | 6.68 | 548 † | 570 | +| 256 | 0.9521 | 99.8 (98.9-100) | 9.91 | 11.7 | 12.5 | 333 † | 292 | + +**chromadb (HNSW)** (ef_search) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.5578 | 1205 (1198-1215) | 0.821 | 0.960 | 1.14 | 8619 | 2000 | +| 16 | 0.6533 | 1110 (1108-1170) | 0.866 | 1.02 | 1.36 | 6744 | 2000 | +| 32 | 0.7740 | 992 (991-1007) | 0.968 | 1.15 | 1.61 | 5012 | 2000 | +| 64 | 0.8717 | 878 (851-887) | 1.09 | 1.29 | 1.89 | 3256 | 2000 | +| 128 | 0.9323 | 687 (646-713) | 1.42 | 1.70 | 2.43 | 1939 | 2000 | +| 256 | 0.9674 | 535 (535-540) | 1.83 | 2.15 | 2.93 | 1112 | 1640 | +| 512 | 0.9859 | 349 (348-365) | 2.77 | 3.50 | 4.82 | 614 | 1103 | + +**faiss IndexFlatIP** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 13.4 (13.3-13.5) | 73.4 | 83.8 | 87.6 | 540 | 40 | + +**faiss HNSWFlat** (efSearch) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.5647 | 11088 (10192-11826) | 0.084 | 0.137 | 0.175 | 44533 | 2000 | +| 16 | 0.6714 | 8786 (8337-8862) | 0.110 | 0.168 | 0.203 | 31150 | 2000 | +| 24 | 0.7482 | 6566 (5968-6666) | 0.146 | 0.225 | 0.286 | 24367 | 2000 | +| 32 | 0.7935 | 5568 (5413-5569) | 0.173 | 0.247 | 0.321 | 19413 | 2000 | +| 48 | 0.8516 | 4032 (3892-4187) | 0.238 | 0.331 | 0.409 | 14764 | 2000 | +| 64 | 0.8875 | 3259 (3140-3290) | 0.299 | 0.401 | 0.510 | 11597 | 2000 | +| 96 | 0.9255 | 2238 (2124-2263) | 0.426 | 0.566 | 0.792 | 8359 | 2000 | +| 128 | 0.9445 | 1721 (1665-1743) | 0.558 | 0.717 | 0.982 | 6400 | 2000 | +| 192 | 0.9648 | 1233 (1191-1284) | 0.796 | 0.979 | 1.24 | 4330 | 2000 | +| 256 | 0.9742 | 913 (853-926) | 1.09 | 1.43 | 1.89 | 3621 | 2000 | +| 400 | 0.9843 | 594 (578-602) | 1.65 | 2.10 | 3.03 | 2364 | 1626 | +| 640 | 0.9913 | 371 (370-377) | 2.65 | 3.14 | 3.91 | 1590 | 1086 | + +**faiss IVFFlat** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2711 | 5665 (5464-5733) | 0.168 | 0.219 | 0.344 | 48473 | 2000 | +| 2 | 0.3827 | 4593 (4539-4850) | 0.197 | 0.260 | 0.462 | 33894 | 2000 | +| 4 | 0.4944 | 3674 (3513-3767) | 0.255 | 0.350 | 0.654 | 17040 | 2000 | +| 8 | 0.6087 | 2710 (2700-2774) | 0.351 | 0.475 | 0.670 | 11517 | 2000 | +| 16 | 0.7099 | 1693 (1644-1827) | 0.537 | 0.750 | 1.51 | 6263 | 2000 | +| 32 | 0.7952 | 1079 (1069-1087) | 0.874 | 1.16 | 1.62 | 3435 | 2000 | +| 64 | 0.8620 | 579 (578-598) | 1.59 | 2.10 | 3.37 | 1784 | 1690 | +| 128 | 0.9183 | 321 (315-324) | 3.00 | 3.76 | 5.05 | 910 | 986 | +| 256 | 0.9561 | 171 (169-171) | 5.63 | 7.03 | 13.2 | 467 | 541 | +| 4000 | 1.0000 | 12.3 (12.2-12.6) | 78.6 | 89.4 | 100 | 30.4 | 38 | + +**hnswlib** (ef) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.5589 | 9990 (7940-11885) | 0.092 | 0.170 | 0.251 | 45715 | 2000 | +| 16 | 0.6512 | 8804 (8071-8872) | 0.109 | 0.164 | 0.208 | 36683 | 2000 | +| 24 | 0.7278 | 6992 (6565-7080) | 0.136 | 0.197 | 0.249 | 27873 | 2000 | +| 32 | 0.7763 | 5558 (5293-5665) | 0.173 | 0.247 | 0.312 | 22939 | 2000 | +| 48 | 0.8354 | 4276 (4196-4387) | 0.224 | 0.306 | 0.392 | 17165 | 2000 | +| 64 | 0.8727 | 3223 (3211-3270) | 0.296 | 0.406 | 0.557 | 13525 | 2000 | +| 96 | 0.9101 | 2482 (2352-2494) | 0.398 | 0.523 | 0.663 | 9472 | 2000 | +| 128 | 0.9315 | 1908 (1862-1987) | 0.510 | 0.654 | 0.816 | 7534 | 2000 | +| 192 | 0.9572 | 1348 (1332-1385) | 0.718 | 0.905 | 1.29 | 5355 | 2000 | +| 256 | 0.9685 | 1043 (1037-1047) | 0.944 | 1.20 | 1.53 | 4059 | 2000 | +| 400 | 0.9817 | 700 (685-721) | 1.41 | 1.75 | 2.37 | 2681 | 2000 | +| 640 | 0.9901 | 472 (461-472) | 2.15 | 2.57 | 2.99 | 1848 | 1224 | + +**lancedb (no index)** (nprobes) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 0.725 (0.725-0.725) | 1372 | 1438 | 1463 | - | 30 | + +**lancedb IVF_FLAT** (nprobes) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2147 | 228 (225-231) | 4.26 | 5.05 | 5.76 | - | 688 | +| 2 | 0.3104 | 229 (223-229) | 4.29 | 5.03 | 6.11 | - | 703 | +| 4 | 0.4113 | 240 (234-243) | 4.13 | 4.66 | 5.25 | - | 728 | +| 8 | 0.5227 | 225 (223-234) | 4.31 | 4.95 | 5.88 | - | 630 | +| 16 | 0.6346 | 211 (211-220) | 4.57 | 5.35 | 6.32 | - | 689 | +| 32 | 0.7313 | 185 (185-192) | 5.16 | 6.38 | 8.70 | - | 593 | +| 64 | 0.8150 | 162 (159-165) | 5.95 | 7.13 | 9.41 | - | 525 | +| 128 | 0.8876 | 134 (127-135) | 7.29 | 8.52 | 10.2 | - | 389 | +| 256 | 0.9378 | 95.4 (95.0-96.7) | 10.2 | 11.7 | 13.3 | - | 299 | +| 4000 | 1.0000 | 11.4 (11.3-11.5) | 87.4 | 90.4 | 92.8 | - | 33 | + +**numpy brute force** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 14.2 (14.1-14.4) | 69.3 | 77.1 | 84.0 | 142 | 42 | + +**sqlite-vec (vec0, exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 1.05 (1.05-1.05) | 947 | 988 | 991 | - | 30 | + +**usearch (HNSW)** (expansion_search) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.5583 | 5894 (5335-6034) | 0.166 | 0.252 | 0.307 | 47685 | 2000 | +| 16 | 0.6489 | 4309 (4212-4677) | 0.217 | 0.329 | 0.396 | 33250 | 2000 | +| 24 | 0.7299 | 3584 (3020-3614) | 0.271 | 0.407 | 0.564 | 27185 | 2000 | +| 32 | 0.7786 | 2935 (2833-2957) | 0.328 | 0.473 | 0.566 | 22420 | 2000 | +| 48 | 0.8359 | 2082 (2039-2155) | 0.450 | 0.631 | 0.811 | 15681 | 2000 | +| 64 | 0.8685 | 1651 (1638-1655) | 0.574 | 0.780 | 1.15 | 13218 | 2000 | +| 96 | 0.9090 | 1183 (1141-1229) | 0.804 | 1.08 | 1.64 | 9541 | 2000 | +| 128 | 0.9307 | 920 (908-926) | 1.07 | 1.35 | 1.70 | 7204 | 2000 | +| 192 | 0.9569 | 642 (627-669) | 1.52 | 1.93 | 2.44 | 5398 | 2000 | +| 256 | 0.9673 | 522 (511-541) | 1.90 | 2.33 | 2.79 | 4125 | 1639 | +| 400 | 0.9822 | 335 (333-339) | 2.97 | 3.69 | 4.92 | 2616 | 999 | +| 640 | 0.9900 | 214 (208-220) | 4.61 | 5.83 | 11.1 | 1810 | 634 | + +
+ +### glove, N = 1,000,000 + +Queries/second at recall@10 = 0.90 / 0.95 / 0.99, log-linearly interpolated between the two measured sweep points that bracket the target (no extrapolation; `*` = the two bracketing points are more than 0.05 recall apart, so the measured point above the target is shown instead, a conservative value; `-` = never reached; exact scans appear at their single recall-1.0 point). 1T = one pinned P-core, one query at a time. All-core = batched call on 32 logical CPUs (`†` = the ISONGraph ports have no batch API, so this is the same one-query loop with automatic scan threads). Latency columns are for the first measured sweep point with recall >= 0.95. + +| system | QPS 1T @0.90 | @0.95 | @0.99 | all-core QPS @0.90 | @0.95 | 1T latency ms median / p95 / p99 | +|---|---|---|---|---|---|---| +| **ISONGraph C++, /arch:AVX2 (exact)** | 2.58 | 2.58 | 2.58 | - | - | 387 / 405 / 407 (recall 1.000) | +| **ISONGraph C++, /arch:AVX2 (IVF)** | 325* | 185 | - | - | - | 5.63 / 7.38 / 8.03 (recall 0.953) | +| **ISONGraph C++ (exact)** | 2.48 | 2.48 | 2.48 | - | - | 406 / 415 / 420 (recall 1.000) | +| **ISONGraph C++ (IVF)** | 322* | 175 | - | - | - | 5.80 / 7.43 / 8.02 (recall 0.951) | +| **ISONGraph C# (exact)** | 5.81 | 5.81 | 5.81 | - | - | 126 / 370 / 399 (recall 1.000) | +| **ISONGraph C# (IVF)** | 248* | - | - | - | - | - | +| **ISONGraph Python (exact)** | 26.5 | 26.5 | 26.5 | - | - | 37.1 / 42.5 / 47.8 (recall 1.000) | +| **ISONGraph Python (IVF)** | 255* | 156 | 47.9 | - | - | 6.95 / 9.14 / 14.9 (recall 0.957) | +| **ISONGraph Rust (exact)** | 43.5 | 43.5 | 43.5 | - | - | 22.5 / 24.6 / 25.9 (recall 1.000) | +| **ISONGraph Rust (IVF)** | 314* | 181 | - | 273* † | 176 † | 6.01 / 7.67 / 8.42 (recall 0.954) | +| **ISONGraph Rust, target-cpu=native (exact)** | 40.1 | 40.1 | 40.1 | - | - | 25.7 / 26.9 / 28.1 (recall 1.000) | +| **ISONGraph Rust, target-cpu=native (IVF)** | 325* | 171 | - | 292* † | 175 † | 6.36 / 8.01 / 8.54 (recall 0.954) | +| faiss IndexFlatIP | 40.7 | 40.7 | 40.7 | 639 | 639 | 24.2 / 27.1 / 33.2 (recall 1.000) | +| faiss HNSWFlat | 1405 | - | - | 11921 | - | - | +| faiss IVFFlat | 989* | 43.8* | 43.8* | 3993* | 159* | 22.2 / 28.9 / 31.4 (recall 1.000) | +| hnswlib | 1934 | - | - | 9075 | - | - | +| lancedb (no index) | 2.89 | 2.89 | 2.89 | - | - | 346 / 385 / 393 (recall 1.000) | +| lancedb IVF_FLAT | 133* | 109 | 29.3 | - | - | 9.32 / 10.9 / 11.4 (recall 0.951) | +| numpy brute force | 34.4 | 34.4 | 34.4 | 69.2 | 69.2 | 28.8 / 33.2 / 36.0 (recall 1.000) | +| usearch (HNSW) | 840 | - | - | 10396 | - | - | + +Build and memory (RSS growth over the process baseline after imports; the raw vectors are streamed in from disk and are not held by the harness). + +| system | ingest s | index s | total build s | 3 builds: median (min-max) s | ready s (incl. first query) | steady RSS +MB | peak RSS +MB (build) | +|---|---|---|---|---|---|---|---| +| **ISONGraph C++, /arch:AVX2 (exact)** | 0.556 | 0.000 | 0.556 | - | 0.871 | 671 | 810 | +| **ISONGraph C++, /arch:AVX2 (IVF)** | 0.556 | 6.73 | 7.29 | - | 7.29 | 1,079 | 1,141 | +| **ISONGraph C++ (exact)** | 0.533 | 0.000 | 0.533 | - | 0.815 | 672 | 810 | +| **ISONGraph C++ (IVF)** | 0.533 | 8.72 | 9.25 | - | 9.25 | 1,080 | 1,141 | +| **ISONGraph C# (exact)** | 0.198 | 0.000 | 0.198 | - | 0.320 | 810 | 810 | +| **ISONGraph C# (IVF)** | 0.339 | 11.3 | 11.6 | - | 11.6 | 2,073 | 2,073 | +| **ISONGraph Python (exact)** | 27.9 | 0.000 | 27.9 | - | 32.8 | 1,424 | 764 | +| **ISONGraph Python (IVF)** | 10.1 | 11.8 | 21.9 | - | 22.0 | 1,444 | 1,634 | +| **ISONGraph Rust (exact)** | 0.786 | 0.000 | 0.786 | - | 1.45 | 1,345 | 1,353 | +| **ISONGraph Rust (IVF)** | 0.786 | 5.69 | 7.14 | - | 7.14 | 1,356 | 1,422 | +| **ISONGraph Rust, target-cpu=native (exact)** | 0.750 | 0.000 | 0.750 | - | 1.41 | 1,345 | 1,353 | +| **ISONGraph Rust, target-cpu=native (IVF)** | 0.750 | 5.69 | 7.10 | - | 7.10 | 1,356 | 1,422 | +| faiss IndexFlatIP | 0.375 | 0.000 | 0.375 | - | 0.404 | 470 | 605 | +| faiss HNSWFlat | 60.1 | 0.000 | 60.1 | - | 60.2 | 546 | 666 | +| faiss IVFFlat | 0.000 | 11.9 | 11.9 | - | 11.9 | 534 | 983 | +| hnswlib | 51.0 | 0.000 | 51.0 | - | 51.0 | 749 | 757 | +| lancedb (no index) | 0.801 | 0.000 | 0.801 | - | 0.958 | 107 | 124 | +| lancedb IVF_FLAT | 0.765 | 8.92 | 9.68 | - | 9.96 | 555 | 866 | +| numpy brute force | 0.122 | 0.000 | 0.122 | - | 0.149 | 493 | 500 | +| usearch (HNSW) | 105 | 0.000 | 105 | - | 105 | 631 | 639 | + +
Full accuracy sweeps (recall@10, single-thread QPS, latency, all-core QPS) + +**ISONGraph C++, /arch:AVX2 (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 2.58 (2.58-2.58) | 387 | 405 | 407 | - | 30 | + +**ISONGraph C++, /arch:AVX2 (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2955 | 7425 (7276-7598) | 0.137 | 0.190 | 0.229 | - | 2000 | +| 2 | 0.4190 | 6023 (5907-6306) | 0.165 | 0.232 | 0.292 | - | 2000 | +| 4 | 0.5330 | 4520 (4502-4652) | 0.218 | 0.318 | 0.386 | - | 2000 | +| 8 | 0.6322 | 2864 (2804-3030) | 0.339 | 0.494 | 0.556 | - | 2000 | +| 16 | 0.7206 | 1825 (1797-1840) | 0.538 | 0.785 | 0.872 | - | 2000 | +| 32 | 0.7965 | 1111 (1061-1130) | 0.890 | 1.32 | 1.50 | - | 2000 | +| 64 | 0.8616 | 610 (601-633) | 1.60 | 2.33 | 2.64 | - | 1991 | +| 128 | 0.9203 | 325 (323-343) | 3.02 | 4.14 | 4.55 | - | 943 | +| 256 | 0.9527 | 176 (170-183) | 5.63 | 7.38 | 8.03 | - | 552 | + +**ISONGraph C++ (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 2.48 (2.48-2.48) | 406 | 415 | 420 | - | 30 | + +**ISONGraph C++ (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2955 | 6471 (6388-6694) | 0.153 | 0.198 | 0.223 | - | 2000 | +| 2 | 0.4190 | 5429 (5367-5468) | 0.183 | 0.237 | 0.260 | - | 2000 | +| 4 | 0.5330 | 4324 (4301-4344) | 0.228 | 0.309 | 0.346 | - | 2000 | +| 8 | 0.6322 | 2985 (2982-3000) | 0.327 | 0.459 | 0.511 | - | 2000 | +| 16 | 0.7206 | 1908 (1818-1972) | 0.514 | 0.745 | 0.848 | - | 2000 | +| 32 | 0.7965 | 1086 (1065-1086) | 0.905 | 1.29 | 1.43 | - | 2000 | +| 64 | 0.8636 | 589 (588-594) | 1.67 | 2.30 | 2.51 | - | 1797 | +| 128 | 0.9202 | 322 (313-326) | 3.10 | 4.07 | 4.39 | - | 924 | +| 256 | 0.9508 | 172 (167-175) | 5.80 | 7.43 | 8.02 | - | 494 | + +**ISONGraph C# (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 5.81 (5.81-5.81) | 126 | 370 | 399 | - | 30 | + +**ISONGraph C# (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2969 | 2483 (2139-2665) | 0.411 | 0.500 | 0.945 | - | 2000 | +| 2 | 0.4143 | 2367 (2353-2403) | 0.431 | 0.501 | 0.540 | - | 2000 | +| 4 | 0.5305 | 2036 (2021-2060) | 0.499 | 0.601 | 0.652 | - | 2000 | +| 8 | 0.6330 | 1558 (1547-1575) | 0.637 | 0.814 | 0.892 | - | 2000 | +| 16 | 0.7188 | 1108 (1092-1115) | 0.893 | 1.19 | 1.32 | - | 2000 | +| 32 | 0.7955 | 713 (707-719) | 1.38 | 1.88 | 2.04 | - | 2000 | +| 64 | 0.8603 | 423 (417-425) | 2.35 | 3.13 | 3.45 | - | 1483 | +| 128 | 0.9160 | 248 (246-273) | 3.86 | 5.20 | 5.63 | - | 674 | +| 256 | 0.9485 | 131 (129-145) | 7.37 | 9.61 | 10.5 | - | 336 | + +**ISONGraph Python (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 26.5 (26.2-27.9) | 37.1 | 42.5 | 47.8 | - | 84 | + +**ISONGraph Python (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.3014 | 7447 (7436-7674) | 0.127 | 0.183 | 0.227 | - | 2000 | +| 2 | 0.4195 | 5473 (5429-5495) | 0.170 | 0.246 | 0.320 | - | 2000 | +| 4 | 0.5359 | 3762 (3631-3937) | 0.248 | 0.360 | 0.445 | - | 2000 | +| 8 | 0.6390 | 2438 (2412-2542) | 0.383 | 0.570 | 0.689 | - | 2000 | +| 16 | 0.7264 | 1503 (1497-1512) | 0.636 | 0.942 | 1.10 | - | 2000 | +| 32 | 0.7996 | 881 (854-891) | 1.10 | 1.58 | 1.90 | - | 2000 | +| 64 | 0.8634 | 492 (492-498) | 1.98 | 2.70 | 3.07 | - | 1635 | +| 128 | 0.9207 | 255 (251-256) | 3.78 | 5.02 | 11.5 | - | 774 | +| 256 | 0.9574 | 138 (137-140) | 6.95 | 9.14 | 14.9 | - | 418 | +| 4000 | 1.0000 | 34.6 (32.8-35.7) | 28.0 | 35.0 | 53.5 | - | 111 | + +**ISONGraph Rust (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 43.5 (42.8-48.6) | 22.5 | 24.6 | 25.9 | - | 170 | + +**ISONGraph Rust (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.3088 | 10525 (10503-10566) | 0.092 | 0.134 | 0.167 | 5474 † | 2000 | +| 2 | 0.4230 | 7216 (6962-7478) | 0.135 | 0.197 | 0.241 | 4107 † | 2000 | +| 4 | 0.5377 | 5054 (5004-5225) | 0.192 | 0.282 | 0.345 | 2611 † | 2000 | +| 8 | 0.6346 | 3277 (3218-3329) | 0.297 | 0.441 | 0.522 | 1647 † | 2000 | +| 16 | 0.7284 | 1923 (1908-1938) | 0.508 | 0.751 | 0.857 | 1028 † | 2000 | +| 32 | 0.8013 | 1168 (1096-1174) | 0.854 | 1.21 | 1.39 | 564 † | 2000 | +| 64 | 0.8640 | 644 (588-644) | 1.58 | 2.19 | 2.47 | 363 † | 1587 | +| 128 | 0.9204 | 314 (312-321) | 3.19 | 4.24 | 4.63 | 273 † | 921 | +| 256 | 0.9537 | 169 (162-170) | 6.01 | 7.67 | 8.42 | 166 † | 475 | + +**ISONGraph Rust, target-cpu=native (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 40.1 (38.9-40.2) | 25.7 | 26.9 | 28.1 | - | 159 | + +**ISONGraph Rust, target-cpu=native (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.3088 | 9787 (9759-10042) | 0.099 | 0.140 | 0.165 | 5548 † | 2000 | +| 2 | 0.4230 | 7586 (7559-7781) | 0.129 | 0.178 | 0.211 | 3924 † | 2000 | +| 4 | 0.5377 | 5136 (5043-5309) | 0.190 | 0.270 | 0.322 | 2675 † | 2000 | +| 8 | 0.6346 | 3220 (3142-3377) | 0.300 | 0.441 | 0.520 | 1596 † | 2000 | +| 16 | 0.7284 | 1903 (1795-1965) | 0.519 | 0.765 | 0.875 | 1049 † | 2000 | +| 32 | 0.8013 | 1010 (1005-1016) | 0.971 | 1.39 | 1.54 | 634 † | 2000 | +| 64 | 0.8626 | 576 (554-581) | 1.73 | 2.39 | 2.64 | 336 † | 1848 | +| 128 | 0.9166 | 325 (308-344) | 3.05 | 4.08 | 4.53 | 292 † | 1032 | +| 256 | 0.9544 | 157 (156-158) | 6.36 | 8.01 | 8.54 | 164 † | 548 | + +**faiss IndexFlatIP** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 40.7 (39.8-42.6) | 24.2 | 27.1 | 33.2 | 639 | 149 | + +**faiss HNSWFlat** (efSearch) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.4765 | 22888 (18723-24051) | 0.041 | 0.066 | 0.086 | 166,477 | 2000 | +| 16 | 0.5605 | 17882 (14705-18039) | 0.052 | 0.083 | 0.117 | 134,983 | 2000 | +| 24 | 0.6299 | 13430 (13067-13770) | 0.071 | 0.108 | 0.134 | 103,151 | 2000 | +| 32 | 0.6724 | 10956 (10906-11267) | 0.086 | 0.129 | 0.157 | 73210 | 2000 | +| 48 | 0.7281 | 7966 (7703-8175) | 0.118 | 0.179 | 0.224 | 57418 | 2000 | +| 64 | 0.7658 | 6097 (6026-6304) | 0.151 | 0.224 | 0.295 | 48203 | 2000 | +| 96 | 0.8093 | 4504 (4414-4594) | 0.213 | 0.301 | 0.391 | 33683 | 2000 | +| 128 | 0.8340 | 3283 (3280-3489) | 0.282 | 0.392 | 0.503 | 27685 | 2000 | +| 192 | 0.8646 | 2297 (2103-2329) | 0.419 | 0.551 | 0.717 | 19009 | 2000 | +| 256 | 0.8850 | 1782 (1671-1796) | 0.553 | 0.701 | 0.980 | 14590 | 2000 | +| 400 | 0.9119 | 1164 (1152-1232) | 0.803 | 1.16 | 1.87 | 10161 | 2000 | +| 640 | 0.9347 | 653 (650-654) | 1.52 | 1.79 | 2.02 | 6535 | 1892 | + +**faiss IVFFlat** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.3046 | 19282 (18048-19407) | 0.050 | 0.071 | 0.091 | 170,065 | 2000 | +| 2 | 0.4281 | 15642 (13255-16241) | 0.061 | 0.091 | 0.135 | 117,092 | 2000 | +| 4 | 0.5371 | 12355 (11603-12379) | 0.078 | 0.106 | 0.134 | 83273 | 2000 | +| 8 | 0.6348 | 8461 (8222-8862) | 0.112 | 0.163 | 0.216 | 49956 | 2000 | +| 16 | 0.7203 | 5748 (5576-5760) | 0.170 | 0.239 | 0.288 | 26400 | 2000 | +| 32 | 0.7933 | 3221 (3128-3365) | 0.286 | 0.406 | 0.527 | 14622 | 2000 | +| 64 | 0.8588 | 1796 (1793-1819) | 0.522 | 0.721 | 0.882 | 7627 | 2000 | +| 128 | 0.9123 | 989 (977-997) | 0.973 | 1.28 | 1.64 | 3993 | 2000 | +| 256 | 0.9488 | 525 (519-534) | 1.85 | 2.34 | 3.19 | 2064 | 1649 | +| 4000 | 1.0000 | 43.8 (43.0-43.8) | 22.2 | 28.9 | 31.4 | 159 | 136 | + +**hnswlib** (ef) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.4829 | 27767 (24854-28305) | 0.035 | 0.057 | 0.075 | 141,005 | 2000 | +| 16 | 0.5657 | 22942 (21474-23347) | 0.042 | 0.065 | 0.079 | 105,029 | 2000 | +| 24 | 0.6309 | 17335 (15586-17788) | 0.054 | 0.080 | 0.097 | 82464 | 2000 | +| 32 | 0.6724 | 13832 (13632-14170) | 0.067 | 0.101 | 0.125 | 67729 | 2000 | +| 48 | 0.7266 | 10275 (10124-10618) | 0.090 | 0.127 | 0.158 | 54141 | 2000 | +| 64 | 0.7621 | 8681 (8081-8779) | 0.111 | 0.154 | 0.189 | 44182 | 2000 | +| 96 | 0.8022 | 6005 (5703-6227) | 0.158 | 0.224 | 0.449 | 31593 | 2000 | +| 128 | 0.8283 | 4941 (4752-5057) | 0.198 | 0.260 | 0.311 | 25814 | 2000 | +| 192 | 0.8605 | 3292 (2942-3346) | 0.289 | 0.384 | 0.822 | 17875 | 2000 | +| 256 | 0.8807 | 2481 (2466-2493) | 0.380 | 0.539 | 1.12 | 10936 | 2000 | +| 400 | 0.9119 | 1660 (1584-1692) | 0.585 | 0.832 | 1.32 | 8093 | 2000 | +| 640 | 0.9367 | 1086 (1071-1101) | 0.893 | 1.24 | 1.82 | 5013 | 2000 | + +**lancedb (no index)** (nprobes) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 2.89 (2.89-2.89) | 346 | 385 | 393 | - | 30 | + +**lancedb IVF_FLAT** (nprobes) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2826 | 232 (165-257) | 4.14 | 7.38 | 9.20 | - | 710 | +| 2 | 0.3981 | 212 (158-234) | 4.48 | 7.54 | 10.2 | - | 499 | +| 4 | 0.5174 | 260 (220-261) | 3.81 | 6.06 | 8.14 | - | 787 | +| 8 | 0.6209 | 166 (159-166) | 6.23 | 7.77 | 8.86 | - | 638 | +| 16 | 0.7151 | 168 (167-177) | 5.81 | 7.56 | 8.25 | - | 481 | +| 32 | 0.7928 | 166 (160-171) | 5.93 | 7.75 | 8.37 | - | 557 | +| 64 | 0.8552 | 150 (146-152) | 6.70 | 8.27 | 9.45 | - | 384 | +| 128 | 0.9126 | 133 (124-134) | 7.72 | 9.20 | 11.3 | - | 338 | +| 256 | 0.9513 | 109 (107-111) | 9.32 | 10.9 | 11.4 | - | 323 | +| 4000 | 1.0000 | 20.9 (20.2-21.2) | 48.0 | 52.1 | 54.9 | - | 59 | + +**numpy brute force** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 34.4 (33.3-36.6) | 28.8 | 33.2 | 36.0 | 69.2 | 121 | + +**usearch (HNSW)** (expansion_search) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.4733 | 10942 (10420-11107) | 0.087 | 0.135 | 0.163 | 60767 | 2000 | +| 16 | 0.5670 | 7685 (6804-8557) | 0.124 | 0.194 | 0.240 | 54147 | 2000 | +| 24 | 0.6345 | 6417 (6102-6460) | 0.150 | 0.230 | 0.281 | 45841 | 2000 | +| 32 | 0.6755 | 5292 (5219-5339) | 0.180 | 0.271 | 0.333 | 32467 | 2000 | +| 48 | 0.7307 | 3777 (3637-3788) | 0.248 | 0.369 | 0.470 | 22370 | 2000 | +| 64 | 0.7647 | 2879 (2823-2890) | 0.340 | 0.512 | 0.646 | 19236 | 2000 | +| 96 | 0.8055 | 2037 (1988-2059) | 0.484 | 0.710 | 0.853 | 13595 | 2000 | +| 128 | 0.8315 | 2194 (1636-2462) | 0.443 | 0.798 | 1.14 | 11891 | 2000 | +| 192 | 0.8649 | 1400 (1137-1461) | 0.712 | 1.15 | 1.61 | 16330 | 2000 | +| 256 | 0.8839 | 1041 (1019-1070) | 0.908 | 1.36 | 2.44 | 12443 | 2000 | +| 400 | 0.9121 | 716 (712-726) | 1.33 | 1.91 | 3.53 | 9088 | 2000 | +| 640 | 0.9394 | 437 (404-451) | 2.21 | 3.61 | 5.66 | 2503 | 1353 | + +
+ +### sift, N = 1,000,000 + +Queries/second at recall@10 = 0.90 / 0.95 / 0.99, log-linearly interpolated between the two measured sweep points that bracket the target (no extrapolation; `*` = the two bracketing points are more than 0.05 recall apart, so the measured point above the target is shown instead, a conservative value; `-` = never reached; exact scans appear at their single recall-1.0 point). 1T = one pinned P-core, one query at a time. All-core = batched call on 32 logical CPUs (`†` = the ISONGraph ports have no batch API, so this is the same one-query loop with automatic scan threads). Latency columns are for the first measured sweep point with recall >= 0.95. + +| system | QPS 1T @0.90 | @0.95 | @0.99 | all-core QPS @0.90 | @0.95 | 1T latency ms median / p95 / p99 | +|---|---|---|---|---|---|---| +| **ISONGraph C++ (exact)** | 2.19 | 2.19 | 2.19 | - | - | 459 / 472 / 474 (recall 1.000) | +| **ISONGraph C++ (IVF)** | 979* | 562* | 314 | - | - | 1.75 / 2.54 / 2.83 (recall 0.970) | +| **ISONGraph Rust (exact)** | 33.4 | 33.4 | 33.4 | - | - | 30.3 / 31.9 / 32.3 (recall 0.999) | +| **ISONGraph Rust (IVF)** | 1078* | 546* | 314 | 600* † | 382* † | 1.76 / 2.62 / 2.89 (recall 0.970) | +| faiss HNSWFlat | 8350 | 5603 | 2175 | 56008 | 37075 | 0.190 / 0.255 / 0.324 (recall 0.957) | +| faiss IVFFlat | 2209* | 1270* | 714 | 7504* | 3254* | 0.761 / 1.14 / 1.30 (recall 0.970) | +| hnswlib | 10660 | 7311 | 3266 | 51989 | 37313 | 0.151 / 0.193 / 0.226 (recall 0.958) | +| numpy brute force | 26.9 | 26.9 | 26.9 | 67.4 | 67.4 | 37.4 / 41.8 / 46.8 (recall 0.998) | + +Build and memory (RSS growth over the process baseline after imports; the raw vectors are streamed in from disk and are not held by the harness). + +| system | ingest s | index s | total build s | 3 builds: median (min-max) s | ready s (incl. first query) | steady RSS +MB | peak RSS +MB (build) | +|---|---|---|---|---|---|---|---| +| **ISONGraph C++ (exact)** | 0.615 | 0.000 | 0.615 | - | 0.944 | 783 | 922 | +| **ISONGraph C++ (IVF)** | 0.615 | 10.9 | 11.5 | - | 11.5 | 1,299 | 1,377 | +| **ISONGraph Rust (exact)** | 0.827 | 0.000 | 0.827 | - | 1.67 | 1,560 | 1,567 | +| **ISONGraph Rust (IVF)** | 0.827 | 6.37 | 8.04 | - | 8.04 | 1,570 | 1,654 | +| faiss HNSWFlat | 99.7 | 0.000 | 99.7 | - | 99.7 | 663 | 814 | +| faiss IVFFlat | 0.000 | 26.8 | 26.8 | - | 26.9 | 684 | 1,258 | +| hnswlib | 60.4 | 0.000 | 60.4 | - | 60.4 | 854 | 864 | +| numpy brute force | 0.137 | 0.000 | 0.137 | - | 0.166 | 599 | 609 | + +
Full accuracy sweeps (recall@10, single-thread QPS, latency, all-core QPS) + +**ISONGraph C++ (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 1.0000 | 2.19 (2.19-2.19) | 459 | 472 | 474 | - | 30 | + +**ISONGraph C++ (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2549 | 5716 (5606-5905) | 0.174 | 0.222 | 0.277 | - | 2000 | +| 2 | 0.3890 | 4864 (4720-4877) | 0.204 | 0.259 | 0.301 | - | 2000 | +| 4 | 0.5391 | 3921 (3791-3942) | 0.251 | 0.344 | 0.417 | - | 2000 | +| 8 | 0.6881 | 2732 (2601-2735) | 0.356 | 0.526 | 0.640 | - | 2000 | +| 16 | 0.8212 | 1686 (1661-1687) | 0.566 | 0.879 | 1.05 | - | 2000 | +| 32 | 0.9147 | 979 (944-989) | 0.984 | 1.51 | 1.74 | - | 2000 | +| 64 | 0.9700 | 562 (525-579) | 1.75 | 2.54 | 2.83 | - | 1522 | +| 128 | 0.9918 | 297 (292-302) | 3.30 | 4.47 | 5.00 | - | 870 | +| 256 | 0.9978 | 155 (154-174) | 6.18 | 7.79 | 8.36 | - | 411 | + +**ISONGraph Rust (exact)** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 0.9990 | 33.4 (32.8-33.9) | 30.3 | 31.9 | 32.3 | - | 105 | + +**ISONGraph Rust (IVF)** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2633 | 10887 (10198-10960) | 0.092 | 0.122 | 0.149 | 6064 † | 2000 | +| 2 | 0.3933 | 7937 (7581-8142) | 0.123 | 0.168 | 0.205 | 4494 † | 2000 | +| 4 | 0.5365 | 5763 (5718-5832) | 0.165 | 0.245 | 0.308 | 3000 † | 2000 | +| 8 | 0.6876 | 3349 (3312-3668) | 0.277 | 0.442 | 0.545 | 1974 † | 2000 | +| 16 | 0.8216 | 2089 (2066-2111) | 0.453 | 0.734 | 0.862 | 1224 † | 2000 | +| 32 | 0.9163 | 1078 (1046-1088) | 0.891 | 1.41 | 1.60 | 600 † | 2000 | +| 64 | 0.9701 | 546 (541-565) | 1.76 | 2.62 | 2.89 | 382 † | 1518 | +| 128 | 0.9924 | 294 (294-295) | 3.34 | 4.57 | 5.02 | 290 † | 1014 | +| 256 | 0.9988 | 157 (155-162) | 6.33 | 7.87 | 8.50 | 239 † | 502 | + +**faiss HNSWFlat** (efSearch) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.6877 | 17412 (15222-17554) | 0.055 | 0.092 | 0.165 | 135,453 | 2000 | +| 16 | 0.7802 | 13831 (12997-13850) | 0.070 | 0.104 | 0.201 | 92820 | 2000 | +| 24 | 0.8484 | 11228 (11083-12040) | 0.085 | 0.124 | 0.169 | 72882 | 2000 | +| 32 | 0.8891 | 9050 (8957-9248) | 0.107 | 0.153 | 0.248 | 60222 | 2000 | +| 48 | 0.9339 | 6490 (6449-7113) | 0.148 | 0.201 | 0.275 | 44633 | 2000 | +| 64 | 0.9573 | 5243 (5201-5347) | 0.190 | 0.255 | 0.324 | 34095 | 2000 | +| 96 | 0.9778 | 3716 (3678-4041) | 0.265 | 0.344 | 0.410 | 24544 | 2000 | +| 128 | 0.9866 | 2622 (2368-2680) | 0.390 | 0.512 | 0.609 | 16650 | 2000 | +| 192 | 0.9936 | 1790 (1707-1800) | 0.584 | 0.723 | 0.835 | 11858 | 2000 | +| 256 | 0.9962 | 1384 (1364-1415) | 0.742 | 0.918 | 1.03 | 8866 | 2000 | +| 400 | 0.9984 | 809 (803-836) | 1.27 | 1.50 | 1.65 | 5686 | 2000 | +| 640 | 0.9990 | 482 (482-488) | 2.14 | 2.51 | 2.66 | 3978 | 1572 | + +**faiss IVFFlat** (nprobe) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 1 | 0.2635 | 9853 (9769-10714) | 0.097 | 0.138 | 0.178 | 47015 | 2000 | +| 2 | 0.3934 | 9352 (9319-9919) | 0.107 | 0.146 | 0.202 | 53015 | 2000 | +| 4 | 0.5409 | 7229 (6865-7675) | 0.135 | 0.197 | 0.253 | 38314 | 2000 | +| 8 | 0.6917 | 5382 (4759-5567) | 0.187 | 0.280 | 0.349 | 23059 | 2000 | +| 16 | 0.8229 | 3576 (3541-3728) | 0.267 | 0.409 | 0.483 | 13783 | 2000 | +| 32 | 0.9173 | 2209 (2168-2230) | 0.434 | 0.664 | 0.777 | 7504 | 2000 | +| 64 | 0.9700 | 1270 (1266-1270) | 0.761 | 1.14 | 1.30 | 3254 | 2000 | +| 128 | 0.9923 | 668 (663-678) | 1.46 | 2.02 | 2.26 | 2326 | 2000 | +| 256 | 0.9984 | 358 (346-363) | 2.80 | 3.63 | 4.04 | 1115 | 1035 | +| 4000 | 0.9988 | 27.4 (26.9-28.1) | 36.2 | 39.0 | 40.8 | 79.8 | 86 | + +**hnswlib** (ef) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| 10 | 0.6795 | 22475 (21929-22477) | 0.044 | 0.062 | 0.073 | 80773 | 2000 | +| 16 | 0.7752 | 17160 (15591-17310) | 0.058 | 0.088 | 0.111 | 72843 | 2000 | +| 24 | 0.8472 | 13361 (13153-14935) | 0.073 | 0.098 | 0.115 | 52927 | 2000 | +| 32 | 0.8900 | 11346 (11208-11457) | 0.089 | 0.117 | 0.136 | 55624 | 2000 | +| 48 | 0.9339 | 8629 (8446-8866) | 0.117 | 0.155 | 0.180 | 41344 | 2000 | +| 64 | 0.9576 | 6765 (6690-6876) | 0.151 | 0.193 | 0.226 | 35561 | 2000 | +| 96 | 0.9768 | 4866 (4847-4880) | 0.210 | 0.267 | 0.312 | 25048 | 2000 | +| 128 | 0.9862 | 3957 (3729-4163) | 0.257 | 0.328 | 0.374 | 24450 | 2000 | +| 192 | 0.9937 | 2717 (2703-2840) | 0.374 | 0.468 | 0.540 | 14951 | 2000 | +| 256 | 0.9966 | 2159 (1950-2229) | 0.488 | 0.627 | 0.725 | 11572 | 2000 | +| 400 | 0.9987 | 1475 (1412-1490) | 0.705 | 0.896 | 0.994 | 8898 | 2000 | +| 640 | 0.9993 | 973 (950-976) | 1.06 | 1.35 | 1.50 | 5333 | 2000 | + +**numpy brute force** (exact) + +| knob | recall@10 | QPS 1T (min-max) | median ms | p95 ms | p99 ms | all-core QPS | queries/run | +|---|---|---|---|---|---|---|---| +| - | 0.9982 | 26.9 (26.7-28.1) | 37.4 | 41.8 | 46.8 | 67.4 | 110 | + +
+ diff --git a/benchmarks/competitive/results/tables_exact.md b/benchmarks/competitive/results/tables_exact.md new file mode 100644 index 0000000..f5e52e3 --- /dev/null +++ b/benchmarks/competitive/results/tables_exact.md @@ -0,0 +1,84 @@ +#### text384 + +| N | system | median ms (1T) | p95 | p99 | QPS 1T (min-max) | all-threads single query | batch QPS | recall | first query ms | +|---|---|---|---|---|---|---|---|---|---| +| 10,000 | **ISONGraph C++, /arch:AVX2 (exact)** | 3.08 | 3.48 | 3.84 | 328 (308-328) | - | - | 1.000 | 3.58 | +| 10,000 | **ISONGraph C++ (exact)** | 3.23 | 3.63 | 4.30 | 311 (289-311) | - | - | 1.000 | 4.08 | +| 10,000 | **ISONGraph C# (exact)** | 2.33 | 2.76 | 4.29 | 422 (382-423) | - | - | 1.000 | 24.2 | +| 10,000 | **ISONGraph Python (exact)** | 0.421 | 0.503 | 0.873 | 2206 (2202-2337) | 0.283 ms (3334 QPS) | - | 1.000 | 32.4 | +| 10,000 | **ISONGraph JS (exact)** | 5.00 | 5.64 | 6.04 | 198 (192-198) | - | - | 1.000 | 5.88 | +| 10,000 | **ISONGraph Rust (exact)** | 0.273 | 0.310 | 0.363 | 3728 (3617-3796) | 0.258 ms (3896 QPS) | - | 1.000 | 7.19 | +| 10,000 | **ISONGraph Rust, target-cpu=native (exact)** | 0.274 | 0.308 | 0.351 | 3617 (3469-3630) | 0.261 ms (3836 QPS) | - | 1.000 | 7.38 | +| 10,000 | faiss IndexFlatIP | 0.336 | 0.441 | 0.590 | 2845 (2753-2860) | 3.48 ms (293 QPS) | 55693 | 1.000 | 0.873 | +| 10,000 | lancedb (no index) | 16.0 | 18.9 | 23.2 | 60.9 (60.4-61.1) | 13.3 ms (74.6 QPS) | - | 1.000 | 16.3 | +| 10,000 | numpy brute force | 0.304 | 0.350 | 0.453 | 3364 (3126-3518) | 0.151 ms (6581 QPS) | 14690 | 1.000 | 0.842 | +| 10,000 | qdrant-client local mode (exact numpy) | 13.2 | 15.9 | 22.1 | 73.0 (72.4-73.4) | 12.5 ms (79.2 QPS) | - | 1.000 | 39.9 | +| 10,000 | sqlite-vec (vec0, exact) | 9.36 | 11.2 | 14.0 | 104 (103-105) | 9.57 ms (103 QPS) | - | 1.000 | 9.76 | +| 100,000 | **ISONGraph C++, /arch:AVX2 (exact)** | 43.5 | 45.4 | 47.6 | 23.0 (23.0-23.1) | - | - | 1.000 | 43.8 | +| 100,000 | **ISONGraph C++ (exact)** | 44.2 | 47.2 | 57.7 | 22.6 (22.5-23.3) | - | - | 1.000 | 45.0 | +| 100,000 | **ISONGraph C# (exact)** | 23.6 | 26.7 | 78.4 | 39.1 (38.6-39.9) | - | - | 1.000 | 46.9 | +| 100,000 | **ISONGraph Python (exact)** | 7.37 | 8.92 | 14.3 | 131 (129-131) | 5.30 ms (183 QPS) | - | 1.000 | 303 | +| 100,000 | **ISONGraph JS (exact)** | 50.8 | 55.8 | 58.1 | 19.7 (19.1-20.3) | - | - | 1.000 | 65.7 | +| 100,000 | **ISONGraph Rust (exact)** | 6.41 | 6.99 | 7.86 | 154 (154-154) | 3.21 ms (299 QPS) | - | 1.000 | 80.0 | +| 100,000 | **ISONGraph Rust, target-cpu=native (exact)** | 6.06 | 6.79 | 7.39 | 163 (161-164) | 3.24 ms (293 QPS) | - | 1.000 | 79.1 | +| 100,000 | faiss IndexFlatIP | 7.29 | 8.84 | 15.0 | 131 (130-135) | 4.97 ms (198 QPS) | 5437 | 1.000 | 8.38 | +| 100,000 | lancedb (no index) | 137 | 151 | 173 | 7.19 (7.16-7.22) | 72.1 ms (13.7 QPS) | - | 1.000 | 64.9 | +| 100,000 | numpy brute force | 6.75 | 7.66 | 10.1 | 144 (143-146) | 4.54 ms (217 QPS) | 1553 | 1.000 | 7.36 | +| 100,000 | qdrant-client local mode (exact numpy) | 136 | 152 | 158 | 7.24 (7.14-7.25) | 132 ms (7.55 QPS) | - | 1.000 | 127 | +| 100,000 | sqlite-vec (vec0, exact) | 95.5 | 115 | 122 | 10.2 (10.2-10.3) | 95.3 ms (10.4 QPS) | - | 1.000 | 98.5 | +| 1,000,000 | **ISONGraph C++, /arch:AVX2 (exact)** | 466 | 491 | 493 | 2.13 (2.13-2.13) | - | - | 1.000 | 510 | +| 1,000,000 | **ISONGraph C++ (exact)** | 469 | 500 | 516 | 2.11 (2.11-2.11) | - | - | 1.000 | 509 | +| 1,000,000 | **ISONGraph C# (exact)** | 245 | 394 | 398 | 3.72 (3.72-3.72) | - | - | 1.000 | 299 | +| 1,000,000 | **ISONGraph Python (exact)** | 77.2 | 86.1 | 93.0 | 12.7 (12.7-12.8) | 46.6 ms (21.3 QPS) | - | 1.000 | 3159 | +| 1,000,000 | **ISONGraph JS (exact)** | 545 | 838 | 872 | 1.67 (1.67-1.67) | - | - | 1.000 | 558 | +| 1,000,000 | **ISONGraph Rust (exact)** | 65.3 | 70.0 | 80.5 | 15.1 (15.1-15.1) | 28.7 ms (34.2 QPS) | - | 1.000 | 1030 | +| 1,000,000 | **ISONGraph Rust, target-cpu=native (exact)** | 62.7 | 68.9 | 71.3 | 15.8 (15.6-15.8) | 30.9 ms (31.9 QPS) | - | 1.000 | 1070 | +| 1,000,000 | faiss IndexFlatIP | 73.4 | 83.8 | 87.6 | 13.4 (13.3-13.5) | 43.1 ms (22.8 QPS) | 540 | 1.000 | 77.6 | +| 1,000,000 | lancedb (no index) | 1372 | 1438 | 1463 | 0.725 (0.725-0.725) | 576 ms (1.73 QPS) | - | 1.000 | 654 | +| 1,000,000 | numpy brute force | 69.3 | 77.1 | 84.0 | 14.2 (14.1-14.4) | 35.1 ms (28.0 QPS) | 142 | 1.000 | 70.3 | +| 1,000,000 | sqlite-vec (vec0, exact) | 947 | 988 | 991 | 1.05 (1.05-1.05) | 938 ms (1.06 QPS) | - | 1.000 | 943 | + +#### glove + +| N | system | median ms (1T) | p95 | p99 | QPS 1T (min-max) | all-threads single query | batch QPS | recall | first query ms | +|---|---|---|---|---|---|---|---|---|---| +| 100,000 | **ISONGraph C++, /arch:AVX2 (exact)** | 24.4 | 28.5 | 31.7 | 40.2 (39.7-41.2) | - | - | 1.000 | 27.1 | +| 100,000 | **ISONGraph C++ (exact)** | 24.5 | 28.4 | 29.7 | 40.1 (39.7-40.1) | - | - | 1.000 | 25.5 | +| 100,000 | **ISONGraph C# (exact)** | 7.79 | 26.4 | 35.6 | 108 (107-108) | - | - | 1.000 | 18.3 | +| 100,000 | **ISONGraph Python (exact)** | 1.98 | 2.72 | 4.89 | 471 (468-487) | 1.28 ms (734 QPS) | - | 1.000 | 179 | +| 100,000 | **ISONGraph JS (exact)** | 20.1 | 24.6 | 27.1 | 48.1 (47.8-48.9) | - | - | 1.000 | 24.7 | +| 100,000 | **ISONGraph Rust (exact)** | 1.42 | 1.79 | 2.14 | 700 (660-705) | 0.578 ms (1670 QPS) | - | 1.000 | 40.8 | +| 100,000 | **ISONGraph Rust, target-cpu=native (exact)** | 1.57 | 1.94 | 2.25 | 621 (616-627) | 0.611 ms (1594 QPS) | - | 1.000 | 41.8 | +| 100,000 | faiss IndexFlatIP | 1.72 | 2.23 | 3.01 | 548 (543-570) | 0.413 ms (2335 QPS) | 16430 | 1.000 | 2.19 | +| 100,000 | lancedb (no index) | 29.5 | 34.2 | 37.4 | 34.3 (31.2-34.5) | 11.4 ms (86.5 QPS) | - | 1.000 | 21.9 | +| 100,000 | numpy brute force | 1.82 | 2.53 | 3.70 | 528 (479-537) | 0.907 ms (1018 QPS) | 1733 | 1.000 | 2.37 | +| 100,000 | sqlite-vec (vec0, exact) | 24.3 | 27.4 | 34.3 | 40.1 (40.0-40.4) | 24.5 ms (40.5 QPS) | - | 1.000 | 25.3 | +| 1,000,000 | **ISONGraph C++, /arch:AVX2 (exact)** | 387 | 405 | 407 | 2.58 (2.58-2.58) | - | - | 1.000 | 316 | +| 1,000,000 | **ISONGraph C++ (exact)** | 406 | 415 | 420 | 2.48 (2.48-2.48) | - | - | 1.000 | 281 | +| 1,000,000 | **ISONGraph C# (exact)** | 126 | 370 | 399 | 5.81 (5.81-5.81) | - | - | 1.000 | 123 | +| 1,000,000 | **ISONGraph Python (exact)** | 37.1 | 42.5 | 47.8 | 26.5 (26.2-27.9) | 35.6 ms (27.7 QPS) | - | 1.000 | 4900 | +| 1,000,000 | **ISONGraph Rust (exact)** | 22.5 | 24.6 | 25.9 | 43.5 (42.8-48.6) | 13.3 ms (70.9 QPS) | - | 1.000 | 664 | +| 1,000,000 | **ISONGraph Rust, target-cpu=native (exact)** | 25.7 | 26.9 | 28.1 | 40.1 (38.9-40.2) | 12.8 ms (68.8 QPS) | - | 1.000 | 658 | +| 1,000,000 | faiss IndexFlatIP | 24.2 | 27.1 | 33.2 | 40.7 (39.8-42.6) | 20.4 ms (49.7 QPS) | 639 | 1.000 | 28.8 | +| 1,000,000 | lancedb (no index) | 346 | 385 | 393 | 2.89 (2.89-2.89) | 105 ms (9.45 QPS) | - | 1.000 | 157 | +| 1,000,000 | numpy brute force | 28.8 | 33.2 | 36.0 | 34.4 (33.3-36.6) | 24.7 ms (39.4 QPS) | 69.2 | 1.000 | 26.9 | + +#### sift + +| N | system | median ms (1T) | p95 | p99 | QPS 1T (min-max) | all-threads single query | batch QPS | recall | first query ms | +|---|---|---|---|---|---|---|---|---|---| +| 100,000 | **ISONGraph C++, /arch:AVX2 (exact)** | 42.3 | 45.4 | 47.4 | 23.4 (23.3-27.3) | - | - | 1.000 | 29.6 | +| 100,000 | **ISONGraph C++ (exact)** | 44.2 | 47.5 | 49.5 | 22.8 (22.4-25.2) | - | - | 1.000 | 31.6 | +| 100,000 | **ISONGraph C# (exact)** | 14.6 | 42.8 | 80.8 | 55.0 (55.0-72.9) | - | - | 1.000 | 22.9 | +| 100,000 | **ISONGraph Python (exact)** | 3.22 | 4.42 | 5.68 | 318 (265-323) | 3.99 ms (222 QPS) | - | 1.000 | 218 | +| 100,000 | **ISONGraph JS (exact)** | 33.7 | 41.6 | 48.7 | 29.0 (27.6-32.5) | - | - | 1.000 | 28.8 | +| 100,000 | **ISONGraph Rust (exact)** | 2.86 | 3.36 | 3.84 | 344 (333-398) | 2.86 ms (336 QPS) | - | 1.000 | 50.6 | +| 100,000 | **ISONGraph Rust, target-cpu=native (exact)** | 2.62 | 3.14 | 3.42 | 364 (362-465) | 2.88 ms (328 QPS) | - | 1.000 | 58.6 | +| 100,000 | faiss IndexFlatIP | 3.30 | 4.26 | 6.04 | 283 (280-332) | 3.44 ms (285 QPS) | 4537 | 1.000 | 3.51 | +| 100,000 | lancedb (no index) | 66.9 | 78.5 | 82.0 | 15.1 (13.7-16.0) | 46.0 ms (21.1 QPS) | - | 1.000 | 33.7 | +| 100,000 | numpy brute force | 3.34 | 4.87 | 6.69 | 282 (267-297) | 2.15 ms (428 QPS) | 1543 | 1.000 | 3.02 | +| 100,000 | sqlite-vec (vec0, exact) | 47.0 | 52.6 | 57.7 | 21.6 (21.0-22.4) | 89.9 ms (11.0 QPS) | - | 1.000 | 32.2 | +| 1,000,000 | **ISONGraph C++ (exact)** | 459 | 472 | 474 | 2.19 (2.19-2.19) | - | - | 1.000 | 330 | +| 1,000,000 | **ISONGraph Rust (exact)** | 30.3 | 31.9 | 32.3 | 33.4 (32.8-33.9) | 16.2 ms (61.3 QPS) | - | 0.999 | 841 | +| 1,000,000 | numpy brute force | 37.4 | 41.8 | 46.8 | 26.9 (26.7-28.1) | 32.0 ms (31.0 QPS) | 67.4 | 0.998 | 29.7 | + diff --git a/benchmarks/competitive/results/tables_failures.md b/benchmarks/competitive/results/tables_failures.md new file mode 100644 index 0000000..214112b --- /dev/null +++ b/benchmarks/competitive/results/tables_failures.md @@ -0,0 +1,2 @@ +| system | dataset | N | status | +|---|---|---|---| diff --git a/benchmarks/competitive/results/tables_filter.md b/benchmarks/competitive/results/tables_filter.md new file mode 100644 index 0000000..3b26100 --- /dev/null +++ b/benchmarks/competitive/results/tables_filter.md @@ -0,0 +1,52 @@ +#### glove, N = 100,000 + +| system | build with metadata s | 10% subset: QPS 1T @ recall>=0.90 | max recall | 1% subset: QPS 1T @ recall>=0.90 | max recall | +|---|---|---|---|---|---| +| **ISONGraph C++ (IVF)** | 0.354 | 152 @ 1.000 | 1.000 | 316 @ 1.000 | 1.000 | +| **ISONGraph Python (exact)** | 0.952 | 368 @ 1.000 | 1.000 | 409 @ 1.000 | 1.000 | +| **ISONGraph Python (IVF)** | 1.78 | 366 @ 1.000 | 1.000 | 400 @ 1.000 | 1.000 | +| **ISONGraph Rust (IVF)** | 2.13 | 8377 @ 1.000 | 1.000 | 18699 @ 1.000 | 1.000 | +| chromadb (HNSW) | 11.1 | 4.23 @ 0.912 | 1.000 | 28.0 @ 0.936 | 1.000 | +| faiss IndexFlatIP | 0.036 | 1813 @ 1.000 | 1.000 | 2913 @ 1.000 | 1.000 | +| faiss HNSWFlat | 3.85 | 534 @ 0.957 | 0.957 | - (max recall 0.835 at 541 QPS) | 0.835 | +| faiss IVFFlat | 1.03 | 1083 @ 1.000 | 1.000 | 1910 @ 1.000 | 1.000 | +| hnswlib | 1.31 | 1354 @ 0.935 | 1.000 | 449 @ 0.951 | 1.000 | +| lancedb (no index) | 0.081 | 33.8 @ 1.000 | 1.000 | 37.6 @ 1.000 | 1.000 | +| lancedb IVF_FLAT | 0.740 | 88.3 @ 1.000 | 1.000 | 99.0 @ 1.000 | 1.000 | +| numpy brute force | 0.012 | 552 @ 1.000 | 1.000 | 624 @ 1.000 | 1.000 | +| sqlite-vec (vec0, exact) | 0.579 | 46.1 @ 1.000 | 1.000 | 48.0 @ 1.000 | 1.000 | +| usearch (HNSW) | 1.57 | unsupported | - | unsupported | - | + +#### text384, N = 100,000 + +| system | build with metadata s | 10% subset: QPS 1T @ recall>=0.90 | max recall | 1% subset: QPS 1T @ recall>=0.90 | max recall | +|---|---|---|---|---|---| +| **ISONGraph C++ (IVF)** | 0.986 | 116 @ 1.000 | 1.000 | 345 @ 1.000 | 1.000 | +| **ISONGraph Python (exact)** | 1.97 | 121 @ 1.000 | 1.000 | 131 @ 1.000 | 1.000 | +| **ISONGraph Python (IVF)** | 3.62 | 122 @ 1.000 | 1.000 | 130 @ 1.000 | 1.000 | +| **ISONGraph Rust (IVF)** | 4.77 | 3094 @ 1.000 | 1.000 | 15288 @ 1.000 | 1.000 | +| chromadb (HNSW) | 16.5 | 2.24 @ 0.953 | 1.000 | 20.9 @ 0.971 | 1.000 | +| faiss IndexFlatIP | 0.109 | 464 @ 1.000 | 1.000 | 2491 @ 1.000 | 1.000 | +| faiss HNSWFlat | 13.7 | 1279 @ 0.944 | 0.992 | 377 @ 0.928 | 0.928 | +| faiss IVFFlat | 1.69 | 388 @ 1.000 | 1.000 | 1630 @ 1.000 | 1.000 | +| hnswlib | 6.44 | 1779 @ 0.912 | 1.000 | 259 @ 0.981 | 1.000 | +| lancedb (no index) | 0.217 | 7.09 @ 1.000 | 1.000 | 7.28 @ 1.000 | 1.000 | +| lancedb IVF_FLAT | 1.94 | 73.1 @ 1.000 | 1.000 | 97.8 @ 1.000 | 1.000 | +| numpy brute force | 0.030 | 141 @ 1.000 | 1.000 | 146 @ 1.000 | 1.000 | +| sqlite-vec (vec0, exact) | 1.01 | 13.0 @ 1.000 | 1.000 | 13.3 @ 1.000 | 1.000 | +| usearch (HNSW) | 5.68 | unsupported | - | unsupported | - | + +#### text384, N = 1,000,000 + +| system | build with metadata s | 10% subset: QPS 1T @ recall>=0.90 | max recall | 1% subset: QPS 1T @ recall>=0.90 | max recall | +|---|---|---|---|---|---| +| **ISONGraph C++ (IVF)** | 12.6 | 308 @ 0.943 | 1.000 | 506 @ 0.905 | 1.000 | +| **ISONGraph Python (IVF)** | 117 | 8.64 @ 1.000 | 1.000 | 9.52 @ 1.000 | 1.000 | +| **ISONGraph Rust (IVF)** | 69.5 | 650 @ 0.940 | 1.000 | 1944 @ 0.906 | 1.000 | +| faiss IndexFlatIP | 1.13 | 39.7 @ 1.000 | 1.000 | 172 @ 1.000 | 1.000 | +| faiss HNSWFlat | 268 | 267 @ 0.973 | 0.973 | - (max recall 0.854 at 275 QPS) | 0.854 | +| faiss IVFFlat | 47.0 | 38.1 @ 1.000 | 1.000 | 140 @ 1.000 | 1.000 | +| hnswlib | 151 | 406 @ 0.955 | 0.999 | 135 @ 0.945 | 1.000 | +| lancedb IVF_FLAT | 25.3 | 8.73 @ 1.000 | 1.000 | 13.1 @ 1.000 | 1.000 | +| numpy brute force | 0.300 | 12.9 @ 1.000 | 1.000 | 14.0 @ 1.000 | 1.000 | + diff --git a/benchmarks/competitive/results/tables_mutation.md b/benchmarks/competitive/results/tables_mutation.md new file mode 100644 index 0000000..3362423 --- /dev/null +++ b/benchmarks/competitive/results/tables_mutation.md @@ -0,0 +1,90 @@ +| dataset | N | system | insert 10k | update 10k | delete 10k | recall@10 before | after | first query after ms | +|---|---|---|---|---|---|---|---|---| +| glove | 100,000 | **ISONGraph C++, /arch:AVX2 (IVF)** | 0.012 s (851,332/s) | 0.006 s (1,540,002/s) | 0.004 s (2,662,903/s) | 0.974 | 0.973 | 19.9 | +| glove | 100,000 | **ISONGraph C++ (IVF)** | 0.013 s (797,925/s) | 0.007 s (1,488,427/s) | 0.004 s (2,524,870/s) | 0.974 | 0.973 | 22.0 | +| glove | 100,000 | **ISONGraph C# (IVF)** | 0.049 s (202,219/s) | 0.008 s (1,294,080/s) | 0.003 s (2,900,316/s) | 0.973 | 0.972 | 33.1 | +| glove | 100,000 | **ISONGraph Python (exact)** | 0.130 s (77055/s) | 0.126 s (79264/s) | 0.116 s (86540/s) | 1.00 | 1.000 | 3.41 | +| glove | 100,000 | **ISONGraph Python (IVF)** | 0.137 s (73067/s) | 0.140 s (71216/s) | 0.118 s (84936/s) | 0.973 | 0.972 | 78.5 | +| glove | 100,000 | **ISONGraph JS (IVF)** | 0.018 s (550,973/s) | 0.009 s (1,170,412/s) | 0.006 s (1,739,615/s) | 0.975 | 0.974 | 1174 | +| glove | 100,000 | **ISONGraph Rust (IVF)** | 0.015 s (689,579/s) | 0.014 s (702,020/s) | 0.009 s (1,127,218/s) | 0.974 | 0.974 | 83.7 | +| glove | 100,000 | **ISONGraph Rust, target-cpu=native (IVF)** | 0.014 s (708,567/s) | 0.014 s (732,220/s) | 0.009 s (1,092,383/s) | 0.974 | 0.974 | 83.6 | +| glove | 100,000 | chromadb (HNSW) | 2.42 s (4125/s) | 3.46 s (2892/s) | 1.95 s (5121/s) | 0.977 | 0.976 | 89.0 | +| glove | 100,000 | faiss IndexFlatIP | 0.002 s (4,972,403/s) | 0.102 s (97716/s) | 0.099 s (101,166/s) | 1.00 | 1.000 | 2.72 | +| glove | 100,000 | faiss HNSWFlat | 0.989 s (10112/s) | unsupported | unsupported | 0.970 | n/a | - | +| glove | 100,000 | faiss IVFFlat | 0.015 s (668,717/s) | 0.017 s (600,017/s) | 0.002 s (5,319,149/s) | 0.974 | 0.974 | 1.32 | +| glove | 100,000 | hnswlib | 0.226 s (44217/s) | 0.934 s (10703/s) | 0.008 s (1,242,329/s) | 0.969 | 0.969 | 0.644 | +| glove | 100,000 | lancedb (no index) | 0.029 s (349,167/s) | 0.134 s (74457/s) | 0.108 s (92786/s) | 1.00 | 1.000 | 20.8 | +| glove | 100,000 | lancedb IVF_FLAT | 0.031 s (319,488/s) | 0.138 s (72668/s) | 0.103 s (97019/s) | 0.974 | 0.977 | 9.27 | +| glove | 100,000 | numpy brute force | 0.001 s (8,836,264/s) | 0.006 s (1,624,643/s) | 0.002 s (4,615,314/s) | 1.00 | 1.000 | 2.93 | +| glove | 100,000 | sqlite-vec (vec0, exact) | 0.080 s (124,613/s) | 2.47 s (4042/s) | 2.07 s (4838/s) | 1.00 | 1.000 | 28.8 | +| glove | 100,000 | usearch (HNSW) | 0.261 s (38339/s) | 0.281 s (35606/s) | 0.003 s (3,713,882/s) | 0.967 | 0.964 | 3.67 | +| glove | 1,000,000 | **ISONGraph C++, /arch:AVX2 (IVF)** | 0.159 s (63000/s) | 0.018 s (571,167/s) | 0.013 s (783,687/s) | 0.953 | 0.951 | 86.5 | +| glove | 1,000,000 | **ISONGraph C++ (IVF)** | 0.155 s (64395/s) | 0.016 s (620,409/s) | 0.012 s (803,006/s) | 0.951 | 0.951 | 92.5 | +| glove | 1,000,000 | **ISONGraph C# (IVF)** | 0.029 s (350,733/s) | 0.016 s (616,922/s) | 0.039 s (258,405/s) | 0.000 | 0.949 | 136 | +| glove | 1,000,000 | **ISONGraph Python (exact)** | 0.881 s (11357/s) | 0.602 s (16617/s) | 0.509 s (19659/s) | 1.00 | 1.000 | 100 | +| glove | 1,000,000 | **ISONGraph Python (IVF)** | 0.167 s (59736/s) | 0.156 s (64253/s) | 0.148 s (67731/s) | 0.957 | 0.957 | 294 | +| glove | 1,000,000 | **ISONGraph Rust (IVF)** | 0.156 s (64014/s) | 0.042 s (238,538/s) | 0.033 s (303,255/s) | 0.954 | 0.954 | 746 | +| glove | 1,000,000 | **ISONGraph Rust, target-cpu=native (IVF)** | 0.164 s (61106/s) | 0.039 s (253,556/s) | 0.034 s (297,217/s) | 0.954 | 0.953 | 750 | +| glove | 1,000,000 | faiss IndexFlatIP | 0.004 s (2,325,527/s) | 3.94 s (2536/s) | 4.19 s (2387/s) | 1.00 | 1.000 | 65.7 | +| glove | 1,000,000 | faiss HNSWFlat | 1.79 s (5594/s) | unsupported | unsupported | 0.935 | n/a | - | +| glove | 1,000,000 | faiss IVFFlat | 0.044 s (225,461/s) | 0.083 s (119,972/s) | 0.005 s (2,143,898/s) | 1.00 | 1.000 | 31.1 | +| glove | 1,000,000 | hnswlib | 1.07 s (9386/s) | 2.66 s (3766/s) | 0.015 s (671,768/s) | 0.937 | 0.935 | 2.27 | +| glove | 1,000,000 | lancedb (no index) | 0.070 s (141,898/s) | 1.50 s (6686/s) | 1.40 s (7153/s) | 1.00 | 1.000 | 273 | +| glove | 1,000,000 | lancedb IVF_FLAT | 0.105 s (95045/s) | 1.29 s (7723/s) | 1.27 s (7897/s) | 0.951 | 0.939 | 41.9 | +| glove | 1,000,000 | numpy brute force | 0.004 s (2,843,494/s) | 0.151 s (66231/s) | 0.024 s (419,856/s) | 1.00 | 1.000 | 68.9 | +| glove | 1,000,000 | usearch (HNSW) | 1.37 s (7313/s) | 1.49 s (6723/s) | 0.004 s (2,399,635/s) | 0.939 | 0.939 | 25.1 | +| sift | 100,000 | **ISONGraph C++, /arch:AVX2 (IVF)** | 0.030 s (333,745/s) | 0.011 s (876,263/s) | 0.006 s (1,582,804/s) | 0.952 | 0.949 | 30.4 | +| sift | 100,000 | **ISONGraph C++ (IVF)** | 0.025 s (400,699/s) | 0.012 s (835,939/s) | 0.007 s (1,374,004/s) | 0.952 | 0.949 | 37.1 | +| sift | 100,000 | **ISONGraph C# (IVF)** | 0.030 s (332,383/s) | 0.015 s (655,476/s) | 0.009 s (1,169,946/s) | 0.953 | 0.949 | 51.8 | +| sift | 100,000 | **ISONGraph Python (exact)** | 0.810 s (12346/s) | 0.676 s (14783/s) | 0.426 s (23490/s) | 1.00 | 1.000 | 10.1 | +| sift | 100,000 | **ISONGraph Python (IVF)** | 0.151 s (66209/s) | 0.157 s (63624/s) | 0.135 s (74040/s) | 0.987 | 0.986 | 96.4 | +| sift | 100,000 | **ISONGraph JS (IVF)** | 0.044 s (225,056/s) | 0.012 s (845,101/s) | 0.008 s (1,271,407/s) | 0.988 | 0.987 | 2043 | +| sift | 100,000 | **ISONGraph Rust (IVF)** | 0.039 s (253,924/s) | 0.033 s (304,480/s) | 0.022 s (463,420/s) | 0.986 | 0.985 | 261 | +| sift | 100,000 | **ISONGraph Rust, target-cpu=native (IVF)** | 0.028 s (354,949/s) | 0.029 s (342,604/s) | 0.020 s (506,183/s) | 0.986 | 0.985 | 277 | +| sift | 100,000 | chromadb (HNSW) | 2.33 s (4284/s) | 3.34 s (2991/s) | 2.33 s (4296/s) | 0.952 | 0.942 | 111 | +| sift | 100,000 | faiss IndexFlatIP | 0.005 s (2,211,215/s) | 0.520 s (19235/s) | 0.361 s (27688/s) | 1.00 | 1.000 | 7.96 | +| sift | 100,000 | faiss HNSWFlat | 0.995 s (10053/s) | unsupported | unsupported | 0.953 | n/a | - | +| sift | 100,000 | faiss IVFFlat | 0.035 s (284,106/s) | 0.063 s (157,573/s) | 0.004 s (2,790,957/s) | 0.950 | 0.947 | 0.438 | +| sift | 100,000 | hnswlib | 0.238 s (42037/s) | 0.570 s (17557/s) | 0.008 s (1,250,750/s) | 0.950 | 0.941 | 0.101 | +| sift | 100,000 | lancedb (no index) | 0.120 s (83312/s) | 0.366 s (27354/s) | 0.249 s (40241/s) | 1.00 | 1.000 | 77.4 | +| sift | 100,000 | lancedb IVF_FLAT | 0.040 s (252,179/s) | 0.149 s (67089/s) | 0.120 s (83050/s) | 0.987 | 0.989 | 12.1 | +| sift | 100,000 | numpy brute force | 0.002 s (6,123,324/s) | 0.006 s (1,562,402/s) | 0.003 s (3,611,412/s) | 1.00 | 1.000 | 3.07 | +| sift | 100,000 | sqlite-vec (vec0, exact) | 0.307 s (32571/s) | 11.0 s (912/s) | 8.30 s (1205/s) | 1.00 | 1.000 | 109 | +| sift | 100,000 | usearch (HNSW) | 0.216 s (46402/s) | 0.258 s (38686/s) | 0.003 s (3,433,948/s) | 0.952 | 0.945 | 2.84 | +| text384 | 100,000 | **ISONGraph C++, /arch:AVX2 (IVF)** | 0.038 s (262,907/s) | 0.014 s (708,602/s) | 0.005 s (2,130,243/s) | 0.954 | 0.951 | 43.8 | +| text384 | 100,000 | **ISONGraph C++ (IVF)** | 0.038 s (265,111/s) | 0.013 s (764,427/s) | 0.004 s (2,329,482/s) | 0.954 | 0.950 | 64.3 | +| text384 | 100,000 | **ISONGraph C# (IVF)** | 0.026 s (386,698/s) | 0.009 s (1,148,805/s) | 0.004 s (2,422,891/s) | 0.952 | 0.949 | 44.8 | +| text384 | 100,000 | **ISONGraph Python (exact)** | 0.206 s (48551/s) | 0.195 s (51276/s) | 0.128 s (78119/s) | 1.000 | 1.000 | 9.15 | +| text384 | 100,000 | **ISONGraph Python (IVF)** | 0.211 s (47387/s) | 0.204 s (49074/s) | 0.126 s (79668/s) | 0.954 | 0.948 | 203 | +| text384 | 100,000 | **ISONGraph JS (IVF)** | 0.063 s (158,336/s) | 0.014 s (729,001/s) | 0.008 s (1,216,382/s) | 0.953 | 0.952 | 3284 | +| text384 | 100,000 | **ISONGraph Rust (IVF)** | 0.044 s (228,710/s) | 0.017 s (579,979/s) | 0.010 s (1,037,086/s) | 0.953 | 0.949 | 192 | +| text384 | 100,000 | **ISONGraph Rust, target-cpu=native (IVF)** | 0.039 s (256,465/s) | 0.016 s (606,160/s) | 0.010 s (1,032,983/s) | 0.953 | 0.949 | 192 | +| text384 | 100,000 | chromadb (HNSW) | 3.49 s (2866/s) | 5.23 s (1912/s) | 1.98 s (5063/s) | 0.965 | 0.965 | 171 | +| text384 | 100,000 | faiss IndexFlatIP | 0.005 s (1,992,310/s) | 0.225 s (44475/s) | 0.196 s (51042/s) | 1.000 | 1.000 | 9.09 | +| text384 | 100,000 | faiss HNSWFlat | 2.84 s (3524/s) | unsupported | unsupported | 0.957 | n/a | - | +| text384 | 100,000 | faiss IVFFlat | 0.049 s (202,182/s) | 0.047 s (211,541/s) | 0.002 s (4,849,425/s) | 0.955 | 0.951 | 1.23 | +| text384 | 100,000 | hnswlib | 1.11 s (9049/s) | 2.97 s (3372/s) | 0.011 s (876,186/s) | 0.964 | 0.968 | 0.687 | +| text384 | 100,000 | lancedb (no index) | 0.053 s (190,268/s) | 0.165 s (60618/s) | 0.111 s (89958/s) | 1.00 | 1.000 | 103 | +| text384 | 100,000 | lancedb IVF_FLAT | 0.056 s (178,739/s) | 0.207 s (48245/s) | 0.135 s (74346/s) | 0.956 | 0.969 | 15.5 | +| text384 | 100,000 | numpy brute force | 0.003 s (3,145,346/s) | 0.007 s (1,341,526/s) | 0.002 s (4,841,208/s) | 1.000 | 1.000 | 7.10 | +| text384 | 100,000 | qdrant-client local mode (exact numpy) | 25.7 s (390/s) | 26.4 s (379/s) | 31.8 s (315/s) | 1.00 | 1.000 | 136 | +| text384 | 100,000 | sqlite-vec (vec0, exact) | 0.170 s (58976/s) | 9.85 s (1016/s) | 7.37 s (1356/s) | 1.00 | 1.000 | 110 | +| text384 | 100,000 | usearch (HNSW) | 1.18 s (8452/s) | 1.19 s (8434/s) | 0.003 s (3,381,920/s) | 0.964 | 0.958 | 3.58 | +| text384 | 1,000,000 | **ISONGraph C++, /arch:AVX2 (IVF)** | 0.288 s (34735/s) | 0.016 s (615,483/s) | 0.007 s (1,382,361/s) | 0.955 | 0.956 | 81.2 | +| text384 | 1,000,000 | **ISONGraph C++ (IVF)** | 0.292 s (34211/s) | 0.018 s (569,256/s) | 0.007 s (1,378,607/s) | 0.953 | 0.953 | 149 | +| text384 | 1,000,000 | **ISONGraph C# (IVF)** | 0.015 s (677,952/s) | 0.009 s (1,124,227/s) | 0.006 s (1,552,650/s) | 0.000 | 0.946 | 111 | +| text384 | 1,000,000 | **ISONGraph Python (exact)** | 0.284 s (35183/s) | 0.212 s (47127/s) | 0.148 s (67540/s) | 1.00 | 1.000 | 76.1 | +| text384 | 1,000,000 | **ISONGraph Python (IVF)** | 0.269 s (37230/s) | 0.228 s (43813/s) | 0.163 s (61533/s) | 0.953 | 0.954 | 665 | +| text384 | 1,000,000 | **ISONGraph JS (IVF)** | 0.017 s (589,240/s) | 0.014 s (703,408/s) | 0.011 s (935,559/s) | 0.000 | 0.946 | 5743 | +| text384 | 1,000,000 | **ISONGraph Rust (IVF)** | 0.283 s (35306/s) | 0.023 s (428,754/s) | 0.017 s (591,195/s) | 0.951 | 0.952 | 611 | +| text384 | 1,000,000 | **ISONGraph Rust, target-cpu=native (IVF)** | 0.277 s (36069/s) | 0.021 s (479,409/s) | 0.015 s (645,561/s) | 0.952 | 0.954 | 602 | +| text384 | 1,000,000 | chromadb (HNSW) | 12.7 s (789/s) | 14.4 s (694/s) | 10.7 s (937/s) | 0.967 | 0.968 | 2631 | +| text384 | 1,000,000 | faiss IndexFlatIP | 0.007 s (1,357,958/s) | 2.31 s (4331/s) | 2.25 s (4445/s) | 1.00 | 1.000 | 72.1 | +| text384 | 1,000,000 | faiss HNSWFlat | 4.32 s (2315/s) | unsupported | unsupported | 0.965 | n/a | - | +| text384 | 1,000,000 | faiss IVFFlat | 0.125 s (79926/s) | 0.139 s (71689/s) | 0.005 s (2,116,626/s) | 0.956 | 0.957 | 8.73 | +| text384 | 1,000,000 | hnswlib | 2.09 s (4774/s) | 3.09 s (3236/s) | 0.009 s (1,138,771/s) | 0.957 | 0.956 | 0.775 | +| text384 | 1,000,000 | lancedb (no index) | 0.056 s (179,892/s) | 0.635 s (15755/s) | 0.358 s (27946/s) | 1.00 | 1.000 | 940 | +| text384 | 1,000,000 | lancedb IVF_FLAT | 0.055 s (181,022/s) | 0.603 s (16597/s) | 0.356 s (28076/s) | 1.00 | 1.000 | 108 | +| text384 | 1,000,000 | numpy brute force | 0.004 s (2,584,647/s) | 0.087 s (115,119/s) | 0.015 s (670,628/s) | 1.00 | 1.000 | 73.8 | +| text384 | 1,000,000 | sqlite-vec (vec0, exact) | 0.194 s (51559/s) | 10.5 s (949/s) | 8.07 s (1240/s) | 1.00 | 1.000 | 958 | +| text384 | 1,000,000 | usearch (HNSW) | 2.25 s (4440/s) | 2.37 s (4225/s) | 0.005 s (2,170,940/s) | 0.957 | 0.956 | 23.9 | diff --git a/benchmarks/competitive/results/tables_saveload.md b/benchmarks/competitive/results/tables_saveload.md new file mode 100644 index 0000000..9165a63 --- /dev/null +++ b/benchmarks/competitive/results/tables_saveload.md @@ -0,0 +1,66 @@ +| dataset | N | system | save s | on-disk MB | load s | first query after load ms | time to ready s | +|---|---|---|---|---|---|---|---| +| glove | 100,000 | **ISONGraph C++, /arch:AVX2 (IVF)** | 4.26 | 133 | 2.02 | 236 | 2.25 | +| glove | 100,000 | **ISONGraph C++ (IVF)** | 4.40 | 133 | 2.03 | 302 | 2.33 | +| glove | 100,000 | **ISONGraph C# (IVF)** | 1.36 | 121 | 1.27 | 333 | 1.61 | +| glove | 100,000 | **ISONGraph Python (exact)** | 4.35 | 215 | 3.37 | 173 | 3.54 | +| glove | 100,000 | **ISONGraph Python (IVF)** | 4.42 | 215 | 3.41 | 2017 | 5.43 | +| glove | 100,000 | **ISONGraph Rust (IVF)** | 0.754 | 121 | 0.414 | 237 | 0.651 | +| glove | 100,000 | chromadb (HNSW) | 0.000 | 76.3 | 0.012 | 87.3 | 0.099 | +| glove | 100,000 | faiss IndexFlatIP | 0.045 | 38.9 | 0.025 | 2.13 | 0.027 | +| glove | 100,000 | faiss HNSWFlat | 0.052 | 57.1 | 0.029 | 1.18 | 0.031 | +| glove | 100,000 | faiss IVFFlat | 0.020 | 39.4 | 0.027 | 0.713 | 0.028 | +| glove | 100,000 | hnswlib | 0.036 | 57.5 | 0.142 | 0.708 | 0.142 | +| glove | 100,000 | lancedb (no index) | 0.000 | 46.8 | 0.002 | 16.9 | 0.018 | +| glove | 100,000 | lancedb IVF_FLAT | 0.000 | 86.1 | 0.002 | 26.8 | 0.028 | +| glove | 100,000 | numpy brute force | 0.025 | 42.8 | 0.017 | 2.38 | 0.020 | +| glove | 100,000 | sqlite-vec (vec0, exact) | 0.000 | 48.9 | 0.000 | 30.7 | 0.031 | +| glove | 100,000 | usearch (HNSW) | 0.097 | 57.5 | 0.052 | 1.56 | 0.053 | +| glove | 1,000,000 | **ISONGraph Rust (IVF)** | 23.1 | 1216 | 10.8 | 7592 | 18.3 | +| glove | 1,000,000 | faiss IndexFlatIP | 0.655 | 389 | 0.973 | 66.8 | 1.04 | +| glove | 1,000,000 | faiss HNSWFlat | 0.307 | 524 | 0.228 | 2.15 | 0.230 | +| glove | 1,000,000 | faiss IVFFlat | 0.149 | 391 | 0.246 | 23.8 | 0.270 | +| glove | 1,000,000 | hnswlib | 0.777 | 528 | 3.66 | 3.27 | 3.66 | +| glove | 1,000,000 | usearch (HNSW) | 0.585 | 528 | 0.499 | 3.37 | 0.503 | +| sift | 100,000 | **ISONGraph C++, /arch:AVX2 (IVF)** | 12.2 | 134 | 5.14 | 636 | 5.78 | +| sift | 100,000 | **ISONGraph C++ (IVF)** | 12.4 | 134 | 4.97 | 826 | 5.80 | +| sift | 100,000 | **ISONGraph C# (IVF)** | 3.51 | 122 | 3.06 | 1133 | 4.20 | +| sift | 100,000 | **ISONGraph Python (exact)** | 11.0 | 222 | 9.70 | 528 | 10.2 | +| sift | 100,000 | **ISONGraph Python (IVF)** | 4.97 | 222 | 3.66 | 2380 | 6.04 | +| sift | 100,000 | **ISONGraph Rust (IVF)** | 1.67 | 128 | 0.904 | 853 | 1.76 | +| sift | 100,000 | chromadb (HNSW) | 0.000 | 88.5 | 0.015 | 108 | 0.123 | +| sift | 100,000 | faiss IndexFlatIP | 0.060 | 49.6 | 0.087 | 12.9 | 0.100 | +| sift | 100,000 | faiss HNSWFlat | 0.044 | 68.9 | 0.039 | 0.340 | 0.039 | +| sift | 100,000 | faiss IVFFlat | 0.025 | 50.2 | 0.035 | 0.216 | 0.036 | +| sift | 100,000 | hnswlib | 0.067 | 69.3 | 0.158 | 0.214 | 0.158 | +| sift | 100,000 | lancedb (no index) | 0.000 | 59.6 | 0.005 | 74.0 | 0.079 | +| sift | 100,000 | lancedb IVF_FLAT | 0.000 | 110 | 0.008 | 18.6 | 0.027 | +| sift | 100,000 | numpy brute force | 0.031 | 54.6 | 0.026 | 4.71 | 0.030 | +| sift | 100,000 | sqlite-vec (vec0, exact) | 0.001 | 61.8 | 0.001 | 105 | 0.106 | +| sift | 100,000 | usearch (HNSW) | 0.080 | 69.3 | 0.061 | 0.200 | 0.062 | +| sift | 1,000,000 | **ISONGraph Rust (IVF)** | 22.7 | 1285 | 11.2 | 8390 | 19.6 | +| sift | 1,000,000 | faiss HNSWFlat | 1.12 | 626 | 1.10 | 0.567 | 1.10 | +| sift | 1,000,000 | faiss IVFFlat | 0.353 | 498 | 0.723 | 1.63 | 0.725 | +| sift | 1,000,000 | hnswlib | 1.39 | 630 | 4.90 | 0.936 | 4.90 | +| text384 | 100,000 | **ISONGraph C++, /arch:AVX2 (IVF)** | 16.4 | 500 | 6.95 | 531 | 7.48 | +| text384 | 100,000 | **ISONGraph C++ (IVF)** | 16.5 | 500 | 7.34 | 903 | 8.24 | +| text384 | 100,000 | **ISONGraph C# (IVF)** | 4.41 | 453 | 3.95 | 911 | 4.86 | +| text384 | 100,000 | **ISONGraph Python (exact)** | 15.5 | 807 | 10.8 | 291 | 11.1 | +| text384 | 100,000 | **ISONGraph Python (IVF)** | 16.0 | 807 | 10.9 | 5129 | 16.0 | +| text384 | 100,000 | **ISONGraph Rust (IVF)** | 1.94 | 453 | 1.25 | 546 | 1.80 | +| text384 | 100,000 | chromadb (HNSW) | 0.000 | 202 | 0.012 | 175 | 0.187 | +| text384 | 100,000 | faiss IndexFlatIP | 0.128 | 147 | 0.069 | 7.27 | 0.076 | +| text384 | 100,000 | faiss HNSWFlat | 0.134 | 176 | 0.074 | 0.496 | 0.075 | +| text384 | 100,000 | faiss IVFFlat | 0.055 | 149 | 0.091 | 1.29 | 0.092 | +| text384 | 100,000 | hnswlib | 0.147 | 177 | 0.225 | 0.615 | 0.225 | +| text384 | 100,000 | lancedb (no index) | 0.000 | 177 | 0.003 | 110 | 0.113 | +| text384 | 100,000 | lancedb IVF_FLAT | 0.000 | 326 | 0.009 | 23.8 | 0.033 | +| text384 | 100,000 | numpy brute force | 0.110 | 162 | 0.068 | 9.16 | 0.077 | +| text384 | 100,000 | sqlite-vec (vec0, exact) | 0.000 | 180 | 0.000 | 109 | 0.110 | +| text384 | 100,000 | usearch (HNSW) | 0.178 | 177 | 0.136 | 0.949 | 0.137 | +| text384 | 1,000,000 | **ISONGraph Rust (IVF)** | 21.5 | 4534 | 13.0 | 6601 | 19.6 | +| text384 | 1,000,000 | faiss IndexFlatIP | 0.968 | 1472 | 0.736 | 74.1 | 0.811 | +| text384 | 1,000,000 | faiss HNSWFlat | 0.994 | 1618 | 0.805 | 1.21 | 0.806 | +| text384 | 1,000,000 | faiss IVFFlat | 0.417 | 1478 | 0.670 | 6.22 | 0.676 | +| text384 | 1,000,000 | hnswlib | 1.07 | 1623 | 2.12 | 1.58 | 2.12 | +| text384 | 1,000,000 | usearch (HNSW) | 1.54 | 1623 | 1.30 | 1.72 | 1.30 | diff --git a/benchmarks/competitive/run_all.py b/benchmarks/competitive/run_all.py new file mode 100644 index 0000000..8f3c83e --- /dev/null +++ b/benchmarks/competitive/run_all.py @@ -0,0 +1,236 @@ +"""Run the whole competitive matrix, one process per (system, dataset, n, experiment-group), sequentially. + + python run_all.py --list # show the job list + python run_all.py --out results/raw # run everything not yet done (resumable) + python run_all.py --only text384:100000 --systems faiss_hnsw,hnswlib + python run_all.py --wait-flag C:/.../B_DONE --wait-minutes 120 + +Timed runs must not overlap anything else. Before every job the runner measures machine CPU load (all other +processes) and waits until it is low; the load it saw is stored beside the result (`load_before_pct`). +Never run two copies of this script at once. +""" +import argparse +import json +import os +import subprocess +import sys +import time + +import psutil + +HERE = os.path.dirname(os.path.abspath(__file__)) +PY = sys.executable +DATA = r"D:\Goldmine\ISON\bench_data" +PREP = os.path.join(DATA, "prep") +RS_DEFAULT = os.path.join(DATA, "rs_target_default", "release", "examples", "competitive_bench.exe") +RS_NATIVE = os.path.join(DATA, "rs_target_native", "release", "examples", "competitive_bench.exe") +CPP_DEFAULT = os.path.join(DATA, "cpp_build", "cb_default.exe") +CPP_AVX2 = os.path.join(DATA, "cpp_build", "cb_avx2.exe") +CS_DLL = os.path.join(DATA, "cs_build", "IsonGraph.Vector.CompetitiveBench.dll") +JS_ENTRY = os.path.join(os.path.dirname(HERE), "..", "isongraph-vector-js", "bench", "competitive.mjs") + +PY_ANN = ["isong_py_ivf", "faiss_ivfflat", "faiss_hnsw", "hnswlib", "usearch", "chromadb", "lancedb_ivfflat"] +PY_EXACT = ["numpy_bf", "isong_py", "faiss_flat", "lancedb_flat", "sqlite_vec"] +NATIVE = ["rust_default", "rust_native", "cpp_default", "cpp_avx2", "csharp", "js"] + +CELLS = [("text384", 100_000), ("glove", 100_000), ("sift", 100_000), ("text384", 1_000_000), ("glove", 1_000_000), ("sift", 1_000_000)] + + +def jobs(): + out = [] + + def add(system, ds, n, exp, timeout_min, prio, extra=None): + out.append(dict(system=system, ds=ds, n=n, exp=exp, timeout=timeout_min * 60, prio=prio, extra=extra or [])) + + for ds, n in CELLS: + big = n >= 1_000_000 + mut = not (ds == "sift" and big) + base_prio = (0 if ds == "text384" else 1 if ds == "glove" else 22) + (10 if big else 0) + for s in PY_ANN + PY_EXACT + NATIVE: + if s == "js" and big and ds != "text384": + continue # JS k-means over 1M plain arrays takes ~30 min per cell; one 1M row (text) is enough + if s in ("sqlite_vec", "chromadb") and big and ds != "text384": + continue # one 1M row (text) is enough: sqlite-vec is exact at ~0.5 s/query; chroma 1M ingest is very slow + exp = ["core"] + if mut: + exp.append("mutation") + if not (big and s in ("isong_py", "isong_py_ivf", "cpp_default", "cpp_avx2", "csharp", "js", "chromadb", "lancedb_flat", "lancedb_ivfflat", "sqlite_vec", "numpy_bf")): + exp.append("saveload") + if s in ("rust_native",): + exp = [e for e in exp if e != "saveload"] # save/load is not SIMD-dependent; measured on rust_default + to = (90 if s == "chromadb" else 150) if big else 60 + extra = ["--build-repeats", "3"] if (not big and s in PY_ANN + PY_EXACT) else [] + add(s, ds, n, ",".join(exp), to, base_prio, extra) + # filtered search + for ds, n in [("text384", 100_000), ("glove", 100_000), ("text384", 1_000_000)]: + big = n >= 1_000_000 + for s in PY_ANN + PY_EXACT + ["rust_default", "cpp_default"]: + if s in ("usearch",): + pass # still run: records "unsupported" + if big and s not in ("isong_py_ivf", "faiss_ivfflat", "faiss_hnsw", "hnswlib", "lancedb_ivfflat", "numpy_bf", "faiss_flat", "rust_default", "cpp_default"): + continue + add(s, ds, n, "filter", 150 if big else 60, (3 if ds == "text384" else 4) + (10 if big else 0)) + # tiny exact-latency cells + for s in PY_EXACT + ["qdrant_local", "rust_default", "rust_native", "cpp_default", "cpp_avx2", "csharp", "js"]: + add(s, "text384", 10_000, "core", 30, 5) + add("qdrant_local", "text384", 100_000, "core,mutation", 90, 6) + return sorted(out, key=lambda j: j["prio"]) + + +def job_name(j): + return f"{j['system']}__{j['ds']}__{j['n']}__{j['exp'].replace(',', '+')}" + + +def command(j, out_path, queries, budget): + s, ds, n = j["system"], j["ds"], j["n"] + ddir = os.path.join(PREP, ds) + common = ["--ds-dir", ddir, "--ds", ds, "--n", str(n), "--exp", j["exp"], "--queries", str(queries), "--budget", str(budget), "--out", out_path] + if s.startswith("rust"): + return [RS_NATIVE if s == "rust_native" else RS_DEFAULT] + common + ["--label", s] + if s.startswith("cpp"): + return [CPP_AVX2 if s == "cpp_avx2" else CPP_DEFAULT] + common + ["--label", s] + if s == "csharp": + return ["dotnet", CS_DLL] + common + ["--label", "csharp"] + if s == "js": + return ["node", "--max-old-space-size=24000", JS_ENTRY] + common + ["--label", "js"] + return [PY, os.path.join(HERE, "bench_py.py"), "--system", s, "--ds", ds, "--n", str(n), "--exp", j["exp"], "--queries", str(queries), + "--budget", str(budget), "--out", out_path] + j["extra"] + + +def cpu_load(seconds=3.0): + me = os.getpid() + mine = {p.pid for p in psutil.Process(me).children(recursive=True)} | {me} + for p in psutil.process_iter(): + try: + p.cpu_percent(None) + except Exception: # noqa: BLE001 + pass + time.sleep(seconds) + tot = 0.0 + top = [] + for p in psutil.process_iter(["pid", "name"]): + try: + c = p.cpu_percent(None) + if p.pid in mine or p.pid == 0: + continue + tot += c + if c > 20: + top.append((round(c), p.info["name"], p.pid)) + except Exception: # noqa: BLE001 + pass + return tot / (psutil.cpu_count() or 1), top + + +def wait_quiet(max_wait_min, thresh=25.0): + t0 = time.time() + while True: + load, top = cpu_load() + if load <= thresh: + return load, False + if time.time() - t0 > max_wait_min * 60: + return load, True + time.sleep(20) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--out", default=os.path.join(HERE, "results", "raw")) + ap.add_argument("--list", action="store_true") + ap.add_argument("--only", default="", help="ds:n[,ds:n]") + ap.add_argument("--systems", default="") + ap.add_argument("--exp", default="", help="only jobs whose exp contains this") + ap.add_argument("--queries", type=int, default=2000) + ap.add_argument("--budget", type=float, default=3.0) + ap.add_argument("--wait-flag", default="") + ap.add_argument("--wait-minutes", type=float, default=120) + ap.add_argument("--quiet-wait-min", type=float, default=20, help="per job: max minutes to wait for low CPU load") + ap.add_argument("--redo", action="store_true") + ap.add_argument("--dry", action="store_true", help="tiny queries/budget; output to --out") + a = ap.parse_args() + js = jobs() + if a.only: + keep = {tuple(x.split(":")) for x in a.only.split(",")} + js = [j for j in js if (j["ds"], str(j["n"])) in keep] + if a.systems: + keep = set(a.systems.split(",")) + js = [j for j in js if j["system"] in keep] + if a.exp: + js = [j for j in js if a.exp in j["exp"].split(",")] + if a.list: + for j in js: + print(job_name(j), "prio", j["prio"], "timeout_min", j["timeout"] // 60) + print(len(js), "jobs") + return + os.makedirs(a.out, exist_ok=True) + flag_waited = None + if a.wait_flag: + t0 = time.time() + while not os.path.exists(a.wait_flag): + if time.time() - t0 > a.wait_minutes * 60: + flag_waited = f"flag file did not appear within {a.wait_minutes} min; ran anyway (timings may be contaminated)" + break + time.sleep(60) + if flag_waited is None: + flag_waited = "flag file present" + print(flag_waited, flush=True) + log = open(os.path.join(a.out, "..", "run_log.txt"), "a") + for j in js: + name = job_name(j) + out_path = os.path.join(a.out, name + ".json") + if os.path.exists(out_path) and not a.redo: + continue + if not all(os.path.exists(p) for p in [os.path.join(PREP, j["ds"], "meta.json")]): + continue + exe = command(j, out_path, a.queries, a.budget) + if not a.dry: + load, contaminated = wait_quiet(a.quiet_wait_min) + else: + load, contaminated = cpu_load(1.0)[0], False + t0 = time.time() + msg = f"[{time.strftime('%H:%M:%S')}] START {name} load_before={load:.1f}%{' CONTAMINATED' if contaminated else ''}" + print(msg, flush=True) + log.write(msg + "\n") + log.flush() + env = dict(os.environ, RUST_LOG="error", PYTHONIOENCODING="utf-8", PYTHONUNBUFFERED="1") + stdout_path = out_path[:-5] + ".stdout.txt" + status = "ok" + with open(stdout_path, "w", encoding="utf-8", errors="replace") as so: + p = subprocess.Popen(exe, stdout=so, stderr=subprocess.STDOUT, env=env, creationflags=0x00000080) # HIGH_PRIORITY_CLASS + if j["system"] == "js": + try: # node is single-threaded for these loops: keep it on four P-cores (8,10,12,14), one per physical core + psutil.Process(p.pid).cpu_affinity([8, 10, 12, 14]) + except Exception: # noqa: BLE001 + pass + try: + p.wait(timeout=j["timeout"]) + except subprocess.TimeoutExpired: + status = "timeout" + try: + for ch in psutil.Process(p.pid).children(recursive=True): + ch.kill() + except Exception: # noqa: BLE001 + pass + p.kill() + dt = time.time() - t0 + if status == "timeout" or not os.path.exists(out_path): + if not os.path.exists(out_path) or status == "timeout": + json.dump({"system": j["system"], "ds": j["ds"], "n": j["n"], "exps": j["exp"].split(","), + "status": "timeout" if status == "timeout" else "crashed", "limit_s": j["timeout"], "exit": p.returncode}, open(out_path, "w")) + # annotate with run context + try: + d = json.load(open(out_path)) + d["load_before_pct"] = load + d["load_contaminated"] = contaminated + d["wall_s"] = dt + d["flag"] = flag_waited + json.dump(d, open(out_path, "w"), indent=1) + except Exception as e: # noqa: BLE001 + print("annotate failed", e) + msg = f"[{time.strftime('%H:%M:%S')}] END {name} {status} {dt:.0f}s" + print(msg, flush=True) + log.write(msg + "\n") + log.flush() + + +if __name__ == "__main__": + main() diff --git a/benchmarks/multihop/README.md b/benchmarks/multihop/README.md new file mode 100644 index 0000000..1ac84dc --- /dev/null +++ b/benchmarks/multihop/README.md @@ -0,0 +1,481 @@ +# Multi-hop retrieval benchmark: does graph-aware retrieval beat plain vector search on real data? + +Real public multi-hop QA data (HotpotQA, 2WikiMultihopQA, MuSiQue), real embeddings (all-MiniLM-L6-v2 and bge-small-en-v1.5), a graph that is built **without** the question, the answer or any supporting-fact annotation, and the library under test (`isongraph_vector-py`, imported unmodified from this checkout). Plain vector search over the same vectors is what every vector store returns, so it is the baseline that matters. The earlier `benchmarks/retrieval_quality.py` used synthetic data; **nothing from it is reused here**. + +Run date 2026-11-15/16, one Windows 11 machine (32 logical cores, RTX 4060 used only for embedding/re-ranking), Python 3.12, numpy 2.4, scipy 1.16, torch 2.5.1+cu121, sentence-transformers 5.1.2, `ison-graph` 1.4.0. + +## 1. Result in one page + +**Headline.** With a graph built from nothing but passage titles and text (title mentions, a Wikipedia-hyperlink proxy), `semantic_multi_hop` (parameters tuned on a separate dev split) retrieves **all** supporting passages significantly more often than plain vector top-k, at every k (5/10/20), on all three datasets, with both encoders (paired p < 0.001 in all 18 cells; all-supporting@10 vs plain vector shown below, test split, 1,000 questions each). With the *untuned library defaults* the gain at k = 10 is still significant everywhere, but at k = 20 it is not significant on MuSiQue and HotpotQA/bge: + +| Dataset / encoder | plain vector | ISONGraph multi_hop (library defaults) | ISONGraph multi_hop (tuned on dev) | cross-encoder rerank | hybrid RRF | independent PPR re-rank (tuned) | +|---|---|---|---|---|---|---| +| HotpotQA / MiniLM | 58.2 | 77.3 (+19.1) | **81.5 (+23.3)** | 68.4 | 72.5 | 81.4 | +| HotpotQA / bge-small | 78.5 | 87.5 (+9.0) | **91.5 (+13.0)** | 84.1 | 83.8 | 91.9 | +| 2Wiki / MiniLM | 32.5 | 79.5 (+47.0) | **81.0 (+48.5)** | 37.5 | 41.1 | 78.7 | +| 2Wiki / bge-small | 43.9 | 94.6 (+50.7) | **94.6 (+50.7)** | 46.0 | 45.6 | 92.8 | +| MuSiQue / MiniLM | 18.2 | 24.5 (+6.3) | **27.4 (+9.2)** | 23.7 | 20.4 | 28.9 | +| MuSiQue / bge-small | 20.9 | 26.3 (+5.4) | **29.6 (+8.7)** | 24.6 | 23.3 | 29.1 | + +(All-supporting@10, %. 95% CIs and paired p-values are in the tables below and in `results/`.) Negative controls (same graphs with node labels randomly permuted) give **exactly the plain-vector numbers** after tuning (|delta| <= 0.2 pp), and the library defaults on a permuted graph are *worse* than plain vector (-5 to -11 pp): the gain comes from the graph's content, not from the expansion machinery. + +**What this does NOT show - read before quoting the table:** + +1. **The library does not beat a simple independent baseline.** A HippoRAG-style personalised-PageRank re-ranker that we implemented separately (no ISONGraph code) on the *same* graph is statistically indistinguishable from `semantic_multi_hop` at k = 10 and 20 on HotpotQA and MuSiQue (differences within about 1.5 pp, CIs include 0). `semantic_multi_hop` is better at small budgets (k = 5: +7.5 pp HotpotQA/MiniLM, +12.0 / +18.9 pp on 2Wiki MiniLM / bge, paired p < 0.001; not significant on HotpotQA/bge and MuSiQue) and slightly better at k = 10 on 2Wiki (+0.9 to +2.3 pp), but that is a statement about *our* PPR re-ranker's score fusion at small k, not a structural advantage. **The defensible differentiator is "graph-aware retrieval", not "the ISONGraph implementation of it".** +2. **The gain depends on graph quality.** Title-mention edges (+9 to +51 pp) >> shared-entity edges (+14 to +30 pp, only run on HotpotQA/MiniLM and 2Wiki) >> kNN edges (+4 to +20 pp). The kNN graph - the only one that needs no text structure at all - gives +4.0 / +4.5 pp on HotpotQA (MiniLM / bge) and +7.3 / +20.0 pp on 2Wiki. +3. **The mention graph is aligned with how these datasets were built** (hyperlink-based question construction; see Threats 1 and 2), so the large numbers are an upper-ish bound. MuSiQue, where only 15% of gold pairs are directly linked by title mentions, shows the realistic scale of the gain when the graph is a weak proxy: **+9 pp** (18 -> 27%). +4. **Graph expansion hurts one question type.** On HotpotQA *comparison* questions (both entities named in the question) tuned `multi_hop` is -5.6 pp (MiniLM) / -0.9 pp (bge, n.s.) vs plain vector, and on 2Wiki comparison -8.0 pp (MiniLM) / -0.8 pp (bge, n.s.). The gain is entirely in *bridge / compositional / inference / bridge-comparison* questions. +5. **Deep chains stay hard.** MuSiQue 3-hop questions: all methods are at 7-17% all-supporting@10 and `multi_hop` adds only +3.4 / +3.8 pp (CIs barely exclude 0); 4-hop questions: 0-3% for every method, no gain. +6. **Not every ISONGraph feature was run to completion.** The union graph `all` (mention+entity+kNN) never finished within the time budget and has **no results**; `mention+entity` exists only for 2Wiki/MiniLM; `entity` is missing for HotpotQA/bge and all MuSiQue runs; `knn` is missing for MuSiQue. Details in Section 4. + +## 1b. Claims the evidence supports, and claims it does not + +Supported (test split, CIs in `results/tables.md`): + +* "Graph-aware retrieval over a title-mention graph retrieves all supporting passages more often than plain vector top-k on HotpotQA, 2WikiMultihopQA and MuSiQue, with two different encoders, at budgets of 5, 10 and 20 passages." (+8.7 to +50.7 pp at k = 10, every paired p < 0.001; every CI excludes 0.) +* "The gain exceeds what a hybrid (vector+BM25) or a cross-encoder reranker over the same candidates gives": `multi_hop` minus RRF = +6.3 to +49.0 pp, minus cross-encoder = +3.7 to +48.6 pp at k = 10 (all p <= 0.005; `results/pairs_vs_nongraph.md`). Exception: MuSiQue/MiniLM at k = 5, where it ties the cross-encoder (+0.6, p = 0.67). +* "The gain is a property of the graph, not of the traversal mechanics": label-permuted graphs give no gain (Section 5). +* "Library defaults already capture most of the gain" (`max_hops=2, top_k_seeds=5, decay=0.8, blend=0`, mention graph): +5.4 to +50.7 pp over vector at k = 10 (all p < 0.001); tuning on dev adds 0 to +4.2 pp on top (at k = 20 the defaults are not significantly better than vector on MuSiQue and HotpotQA/bge). +* "semantic_multi_hop is faster than the PPR re-ranker at equal quality in some configurations and about the same in others" (latency table; single-thread, ms-scale). + +Not supported: + +* "ISONGraph's traversal is better than other graph-aware re-ranking." It ties an independent PPR re-ranker at k >= 10 (see 1). +* "Graph-aware retrieval helps when the graph is built from embedding similarity alone." Gains are small on HotpotQA (+4 pp), larger on 2Wiki; kNN edges on MuSiQue were not run. +* "Graph-aware retrieval helps on every question type" (comparison questions get worse), or on deep chains (3-4 hops on MuSiQue). +* "These gains will hold in a 5M-passage index" (pooled corpora of 9k-15k passages; Threat 2) or with stronger encoders (only two small ones). +* "`semantic_subgraph(connect=True)` adds value over `semantic_multi_hop`": in these runs the connector step added almost no nodes (mean set size 8-12 vs k = 10), so the set quality equals `multi_hop`'s; the size-matched comparison against vector top-|S| is in Section 3. +* Anything about answer quality / generation: only retrieval of supporting paragraphs is measured. +* Anything about the `all`, and most `mention+entity` / `entity` / `knn` variants outside the cells listed in Section 4. + + +## 2. What was run + +### 2.1 Data (dev sets only, pinned revisions) + +| Dataset | Source (HuggingFace dataset repo @ revision, file) | Questions available | Sampled | dev / test | Pooled corpus (unique passages) | +|---|---|---|---|---|---| +| HotpotQA (distractor) | `hotpotqa/hotpot_qa` @ `1908d6afbbea...`, `distractor/validation-00000-of-00001.parquet` | 7,405 | 1,500 | 500 / 1,000 | 14,570 | +| 2WikiMultihopQA | `framolfese/2WikiMultihopQA` @ `fe713bfbd1af...`, `data/validation-00000-of-00001.parquet` (mirror of the official dev set, 12,576 q) | 12,576 | 1,500 | 500 / 1,000 | 9,239 | +| MuSiQue (answerable) | `dgslibisey/MuSiQue` @ `c8f4f8c9465f...`, `musique_ans_v1.0_dev.jsonl` (official v1.0 answerable dev) | 2,417 | 1,500 | 500 / 1,000 | 15,405 | + +* Sampling: questions sorted by id, `random.Random(13).sample(..., 1500)`, then a seeded shuffle assigns the first 500 to **dev** (used only to choose hyper-parameters and the graph variant) and the other 1,000 to **test** (all reported numbers). The exact ids and split are in `manifests/_sample.json`. +* Corpus: **one corpus per dataset** = every context paragraph (gold + the dataset's own distractors) of all 1,500 sampled questions, de-duplicated by (title, text). A question is searched against the whole pool, so its competitors are its own distractors *and* every other question's gold and distractor paragraphs. Passage text embedded: `". <text>"`. +* Gold = the dataset's supporting paragraphs (HotpotQA/2Wiki: paragraphs whose title is in `supporting_facts`; MuSiQue: `is_supporting`). 2 gold passages for every HotpotQA question; 2 or 4 for 2Wiki (2: 1,183 q, 4: 317 q); 2/3/4 for MuSiQue (784/459/257 q). +* Hop count = number of gold passages (HotpotQA: always 2). + +### 2.2 Embeddings + +`sentence-transformers/all-MiniLM-L6-v2` (384-d) and `BAAI/bge-small-en-v1.5` (384-d; documented query instruction prefix on questions only). `max_seq_length=256`, L2-normalised, fp32, computed once on GPU and cached as `.npy` in `D:\Goldmine\ISON\bench_data\multihop\emb_<dataset>__<model>__{passages,queries}.npy`. Identical vectors feed every method. + +### 2.3 Graph construction (no gold annotation, no question, no answer) + +Built once per dataset from passage text/titles only (kNN: from the passage embeddings). Constants were fixed a priori and not tuned. + +| Variant | Edges | +|---|---| +| `mention` | directed A -> B if B's title (parenthetical stripped, >= 4 chars, exact case-sensitive token match) occurs in A's text; titles mentioned by > 30 passages dropped as hubs. (A proxy for Wikipedia hyperlinks: the HF copies carry no link markup.) | +| `entity` | undirected, A -- B if they share an "entity": maximal run of capitalised tokens (with of/the/de connectors; sentence-initial single tokens and a stop list removed) or A/B's own title; entities with document frequency outside [2, 15] dropped. Edge weight `w = n/(n+1)` for `n` shared entities. | +| `knn` | undirected, mutual-or-one-way 5-nearest-neighbours under the *same* encoder; `w` = cosine. | +| `mention+entity`, `all` | unions (separate relationship types). | +| `ctrl_mention`, `ctrl_entity` | **negative controls**: the `mention` / `entity` edge lists with node labels randomly permuted (an isomorphic graph whose edges no longer correspond to the text). | + +`results/tables.md` ("Graph diagnostics") reports edge counts, degrees and - for *description only* - how often two gold passages of a question are directly linked. + +### 2.4 Methods (all return passage ids; identical budget k, k in {5, 10, 20}) + +1. **Vector top-k**: exact cosine brute force (numpy). Checked equal to `SemanticGraph.similarity_search` (see Section 6). +2. **BM25**: own Okapi implementation (k1=1.2, b=0.75), lower-cased `[a-z0-9]+` tokens, small stop list, no stemming. +3. **Hybrid**: reciprocal rank fusion (k0=60) of vector and BM25 full rankings. +4. **ISONGraph Vector `semantic_multi_hop`** from this checkout, unmodified, queried with the cached query vector (`threshold=-1.0` so seeds are never dropped). Searched on **dev**: random sample of 100 configurations (seeded; **40 in the runs marked "lean" in Section 4**, which were re-launched with a smaller budget when the first pass ran out of time) from `hops` {1,2,(3 for mention graphs)} x `top_k_seeds` {5,10,15,20} x `decay` {.3,.5,.7,.9} x `blend` {0,.3,.6} x `direction` {OUT, BOTH} (graphs with directed mention edges) x `weight_field` {none, `w`} (weighted graphs), plus the library defaults (`max_hops=2, top_k_seeds=5, decay=0.8, blend=0`) as a separate untuned row. One configuration is selected **per (dataset, encoder, graph, k)** by dev all-supporting@k (ties: dev recall@k) and then evaluated once on test. `top_k_results=20`, truncated to k (identical to asking for k, the list is sorted). +5. **`semantic_subgraph(connect=True)`** with the dev-selected parameters, `connect_max_hops` in {2, 3}. The returned set can be larger than k (connectors are extra), so it is compared with plain vector search at *the same per-query set size* (vector top-|S|), not only at k. +6. **Personalised PageRank re-ranking** (HippoRAG-style, implemented independently in `methods.py`, scipy sparse, 25 power iterations): restart distribution = top-`seeds` cosine hits weighted by cosine, `p <- alpha P^T p + (1-alpha) s` (dangling mass restarts), final score `(1-beta) p/max(p) + beta cos/max(cos)`. **Full** grid on dev (72 configurations; 144 for union graphs): `alpha` {.5,.7,.85} x `seeds` {5,10,20} x `beta` {0,.3,.6,.85} x directed/symmetrised mention edges or unweighted/weighted edges. (PPR gets a comparable or larger search than the library, which is conservative for the library.) +7. **Cross-encoder rerank**: vector top-30 re-scored by `cross-encoder/ms-marco-MiniLM-L-6-v2`, top-k returned. + +"Graph chosen on dev" rows: among the five real graph variants (controls excluded), the one with the best dev all-supporting@k for that method. + +### 2.5 Metrics and statistics + +Per question: recall@k (|gold in top-k| / |gold|), **all-supporting@k** (every gold passage in top-k; the multi-hop success metric), MRR@k of the first gold passage, number of passages actually returned. Means over the 1,000 test questions; 95% CIs by question-level percentile bootstrap (2,000 resamples). Paired comparison with plain vector search: difference of per-question values, bootstrap CI of the mean difference, and a paired sign-flip permutation test (10,000 permutations), Holm-corrected within each table. + +## 3. Results + +All numbers: test split (1,000 questions per dataset), percent unless stated; "(+x, p)" is the paired difference to plain vector search in percentage points with the paired sign-flip p-value; 95% CIs (bootstrap) are in the full tables `results/tables.md` and `results/summary.json`. "[graph]" = the graph variant chosen on dev for that method and k. BM25 does not depend on the encoder (the two columns of the same dataset show the same system). + + +### Headline, k = 10: all-supporting@10 % [95% CI] (Delta vs plain vector in pp, paired p) + +| Method | hotpotqa/minilm | hotpotqa/bge | 2wiki/minilm | 2wiki/bge | musique/minilm | musique/bge | +|---|---|---|---|---|---|---| +| Vector top-k (plain) | 58.2 [55.2, 61.3] | 78.5 [76.0, 80.9] | 32.5 [29.6, 35.2] | 43.9 [40.9, 46.9] | 18.2 [15.8, 20.6] | 20.9 [18.4, 23.4] | +| BM25 | 69.1 [66.3, 71.9] (+10.9, p<0.001) | 69.1 [66.3, 71.9] (-9.4, p<0.001) | 41.5 [38.5, 44.5] (+9.0, p<0.001) | 41.5 [38.5, 44.5] (-2.4, p=0.019) | 15.2 [13.0, 17.3] (-3.0, p=0.013) | 15.2 [13.0, 17.3] (-5.7, p<0.001) | +| Hybrid RRF (vector+BM25) | 72.5 [69.7, 75.1] (+14.3, p<0.001) | 83.8 [81.4, 86.0] (+5.3, p<0.001) | 41.1 [38.1, 44.1] (+8.6, p<0.001) | 45.6 [42.4, 48.5] (+1.7, p=0.046) | 20.4 [17.8, 22.9] (+2.2, p=0.033) | 23.3 [20.8, 26.0] (+2.4, p=0.040) | +| Vector top-30 + cross-encoder | 68.4 [65.7, 71.1] (+10.2, p<0.001) | 84.1 [81.9, 86.3] (+5.6, p<0.001) | 37.5 [34.5, 40.2] (+5.0, p<0.001) | 46.0 [43.0, 49.0] (+2.1, p=0.011) | 23.7 [21.2, 26.3] (+5.5, p<0.001) | 24.6 [22.1, 27.1] (+3.7, p<0.001) | +| ISONGraph multi_hop, library defaults | 77.3 [74.7, 79.8] (+19.1, p<0.001) [mention] | 87.5 [85.4, 89.5] (+9.0, p<0.001) [mention] | 79.5 [77.0, 81.9] (+47.0, p<0.001) [mention] | 94.6 [93.2, 96.0] (+50.7, p<0.001) [mention] | 24.5 [21.8, 27.3] (+6.3, p<0.001) [mention] | 26.3 [23.6, 29.0] (+5.4, p<0.001) [mention] | +| ISONGraph multi_hop, tuned | 81.5 [79.1, 83.8] (+23.3, p<0.001) [mention] | 91.5 [89.7, 93.2] (+13.0, p<0.001) [mention] | 81.0 [78.6, 83.4] (+48.5, p<0.001) [mention+entity] | 94.6 [93.2, 96.0] (+50.7, p<0.001) [mention] | 27.4 [24.7, 30.2] (+9.2, p<0.001) [mention] | 29.6 [26.8, 32.4] (+8.7, p<0.001) [mention] | +| PPR re-rank, tuned (independent impl.) | 81.4 [79.0, 83.7] (+23.2, p<0.001) [mention] | 91.9 [90.1, 93.5] (+13.4, p<0.001) [mention] | 78.7 [76.2, 81.1] (+46.2, p<0.001) [mention] | 92.8 [91.2, 94.4] (+48.9, p<0.001) [mention] | 28.9 [26.1, 31.6] (+10.7, p<0.001) [mention] | 29.1 [26.4, 31.9] (+8.2, p<0.001) [mention] | +| CONTROL: multi_hop on label-shuffled mention graph | 58.2 [55.2, 61.3] (+0.0, p=1.000) | 78.4 [75.9, 80.8] (-0.1, p=1.000) | 32.5 [29.6, 35.2] (+0.0, p=1.000) | 43.7 [40.7, 46.7] (-0.2, p=0.497) | 18.2 [15.8, 20.6] (+0.0, p=1.000) | 20.9 [18.4, 23.4] (+0.0, p=1.000) | +| CONTROL: multi_hop on label-shuffled entity graph | n/a | n/a | 32.5 [29.6, 35.2] (+0.0, p=1.000) | 43.8 [40.8, 46.8] (-0.1, p=1.000) | n/a | n/a | + +### Headline, k = 5: all-supporting@5 % [95% CI] (Delta vs plain vector in pp, paired p) + +| Method | hotpotqa/minilm | hotpotqa/bge | 2wiki/minilm | 2wiki/bge | musique/minilm | musique/bge | +|---|---|---|---|---|---|---| +| Vector top-k (plain) | 47.2 [44.1, 50.3] | 68.4 [65.5, 71.2] | 26.2 [23.5, 28.7] | 38.3 [35.3, 41.2] | 11.9 [10.0, 13.9] | 14.3 [12.2, 16.4] | +| BM25 | 48.7 [45.7, 51.7] (+1.5, p=0.437) | 48.7 [45.7, 51.7] (-19.7, p<0.001) | 33.8 [31.1, 36.7] (+7.6, p<0.001) | 33.8 [31.1, 36.7] (-4.5, p<0.001) | 11.1 [9.3, 12.9] (-0.8, p=0.503) | 11.1 [9.3, 12.9] (-3.2, p=0.003) | +| Hybrid RRF (vector+BM25) | 54.3 [51.1, 57.4] (+7.1, p<0.001) | 69.1 [66.3, 71.9] (+0.7, p=0.621) | 33.2 [30.3, 36.0] (+7.0, p<0.001) | 38.7 [35.8, 41.8] (+0.4, p=0.695) | 12.6 [10.6, 14.6] (+0.7, p=0.451) | 14.9 [12.8, 17.2] (+0.6, p=0.585) | +| Vector top-30 + cross-encoder | 59.7 [56.6, 62.6] (+12.5, p<0.001) | 71.6 [68.8, 74.4] (+3.2, p=0.028) | 33.8 [30.9, 36.5] (+7.6, p<0.001) | 39.4 [36.4, 42.4] (+1.1, p=0.289) | 17.8 [15.5, 20.2] (+5.9, p<0.001) | 18.9 [16.6, 21.2] (+4.6, p<0.001) | +| ISONGraph multi_hop, library defaults | 67.6 [64.8, 70.4] (+20.4, p<0.001) [mention] | 73.4 [70.8, 76.1] (+5.0, p<0.001) [mention] | 53.5 [50.5, 56.6] (+27.3, p<0.001) [mention] | 47.7 [44.7, 50.8] (+9.4, p<0.001) [mention] | 18.2 [15.9, 20.6] (+6.3, p<0.001) [mention] | 17.3 [15.1, 19.7] (+3.0, p<0.001) [mention] | +| ISONGraph multi_hop, tuned | 71.1 [68.3, 73.8] (+23.9, p<0.001) [mention] | 81.3 [78.8, 83.6] (+12.9, p<0.001) [mention] | 63.8 [60.8, 66.8] (+37.6, p<0.001) [mention+entity] | 77.8 [75.3, 80.4] (+39.5, p<0.001) [mention] | 18.4 [16.1, 20.8] (+6.5, p<0.001) [mention] | 21.6 [19.1, 24.2] (+7.3, p<0.001) [mention] | +| PPR re-rank, tuned (independent impl.) | 63.6 [60.8, 66.4] (+16.4, p<0.001) [mention] | 79.6 [77.1, 82.0] (+11.2, p<0.001) [mention] | 51.8 [48.7, 54.9] (+25.6, p<0.001) [mention] | 58.9 [55.9, 61.9] (+20.6, p<0.001) [mention] | 18.3 [16.0, 20.6] (+6.4, p<0.001) [mention] | 20.2 [17.7, 22.6] (+5.9, p<0.001) [mention] | +| CONTROL: multi_hop on label-shuffled mention graph | 47.2 [44.1, 50.3] (+0.0, p=1.000) | 67.5 [64.5, 70.4] (-0.9, p=0.003) | 26.2 [23.5, 28.8] (+0.0, p=1.000) | 38.4 [35.4, 41.3] (+0.1, p=1.000) | 12.0 [10.1, 14.0] (+0.1, p=1.000) | 14.3 [12.2, 16.4] (+0.0, p=1.000) | +| CONTROL: multi_hop on label-shuffled entity graph | n/a | n/a | 26.2 [23.5, 28.8] (+0.0, p=1.000) | 38.4 [35.4, 41.3] (+0.1, p=1.000) | n/a | n/a | + +### Headline, k = 20: all-supporting@20 % [95% CI] (Delta vs plain vector in pp, paired p) + +| Method | hotpotqa/minilm | hotpotqa/bge | 2wiki/minilm | 2wiki/bge | musique/minilm | musique/bge | +|---|---|---|---|---|---|---| +| Vector top-k (plain) | 66.2 [63.4, 69.1] | 85.4 [83.2, 87.5] | 38.4 [35.3, 41.3] | 48.2 [45.2, 51.2] | 26.3 [23.6, 29.0] | 28.6 [25.9, 31.4] | +| BM25 | 82.2 [79.8, 84.4] (+16.0, p<0.001) | 82.2 [79.8, 84.4] (-3.2, p=0.015) | 46.7 [43.8, 49.8] (+8.3, p<0.001) | 46.7 [43.8, 49.8] (-1.5, p=0.114) | 23.5 [21.0, 26.1] (-2.8, p=0.047) | 23.5 [21.0, 26.1] (-5.1, p<0.001) | +| Hybrid RRF (vector+BM25) | 83.8 [81.6, 86.0] (+17.6, p<0.001) | 89.6 [87.8, 91.5] (+4.2, p<0.001) | 46.7 [43.7, 49.7] (+8.3, p<0.001) | 49.4 [46.4, 52.4] (+1.2, p=0.173) | 31.9 [29.0, 34.8] (+5.6, p<0.001) | 32.9 [30.1, 35.8] (+4.3, p=0.001) | +| Vector top-30 + cross-encoder | 70.3 [67.7, 73.0] (+4.1, p<0.001) | 87.5 [85.4, 89.5] (+2.1, p<0.001) | 39.0 [35.9, 41.9] (+0.6, p=0.216) | 50.1 [47.0, 53.1] (+1.9, p=0.004) | 29.2 [26.4, 32.0] (+2.9, p<0.001) | 31.6 [28.8, 34.4] (+3.0, p<0.001) | +| ISONGraph multi_hop, library defaults | 77.5 [74.9, 80.0] (+11.3, p<0.001) [mention] | 86.5 [84.4, 88.5] (+1.1, p=0.374) [knn] | 79.9 [77.4, 82.3] (+41.5, p<0.001) [mention] | 94.8 [93.4, 96.1] (+46.6, p<0.001) [mention] | 26.4 [23.8, 29.1] (+0.1, p=1.000) [mention] | 28.8 [26.0, 31.6] (+0.2, p=0.942) [mention] | +| ISONGraph multi_hop, tuned | 87.3 [85.2, 89.3] (+21.1, p<0.001) [mention] | 95.2 [93.9, 96.5] (+9.8, p<0.001) [mention] | 87.3 [85.1, 89.3] (+48.9, p<0.001) [mention+entity] | 96.4 [95.2, 97.5] (+48.2, p<0.001) [mention] | 37.7 [34.7, 40.7] (+11.4, p<0.001) [mention] | 38.7 [35.7, 41.5] (+10.1, p<0.001) [mention] | +| PPR re-rank, tuned (independent impl.) | 87.3 [85.3, 89.3] (+21.1, p<0.001) [mention] | 95.2 [93.8, 96.5] (+9.8, p<0.001) [mention] | 85.7 [83.5, 87.9] (+47.3, p<0.001) [mention] | 95.5 [94.2, 96.7] (+47.3, p<0.001) [mention] | 38.1 [35.1, 41.2] (+11.8, p<0.001) [mention] | 39.2 [36.2, 42.2] (+10.6, p<0.001) [mention] | +| CONTROL: multi_hop on label-shuffled mention graph | 66.2 [63.4, 69.1] (+0.0, p=1.000) | 85.4 [83.2, 87.5] (+0.0, p=1.000) | 38.4 [35.3, 41.3] (+0.0, p=1.000) | 48.2 [45.2, 51.2] (+0.0, p=1.000) | 26.3 [23.6, 29.0] (+0.0, p=1.000) | 28.6 [25.9, 31.4] (+0.0, p=1.000) | +| CONTROL: multi_hop on label-shuffled entity graph | n/a | n/a | 38.4 [35.3, 41.3] (+0.0, p=1.000) | 48.2 [45.2, 51.2] (+0.0, p=1.000) | n/a | n/a | + +### Every graph variant, k = 10: Delta all-supporting@10 vs plain vector, pp [95% CI] (paired p) + +| Method: graph | hotpotqa/minilm | hotpotqa/bge | 2wiki/minilm | 2wiki/bge | musique/minilm | musique/bge | +|---|---|---|---|---|---|---| +| ISONGraph multi_hop (tuned): mention | +23.3 [+20.4, +26.4] (p<0.001) | +13.0 [+10.6, +15.4] (p<0.001) | +47.1 [+43.7, +50.5] (p<0.001) | +50.7 [+47.5, +53.9] (p<0.001) | +9.2 [+7.2, +11.3] (p<0.001) | +8.7 [+6.6, +10.8] (p<0.001) | +| ISONGraph multi_hop (tuned): entity | +13.6 [+11.2, +16.0] (p<0.001) | n/a | +28.1 [+24.9, +31.1] (p<0.001) | +30.3 [+27.3, +33.0] (p<0.001) | n/a | n/a | +| ISONGraph multi_hop (tuned): knn | +4.0 [+1.9, +6.2] (p<0.001) | +4.5 [+2.5, +6.4] (p<0.001) | +7.3 [+4.8, +9.8] (p<0.001) | +20.0 [+17.5, +22.5] (p<0.001) | n/a | n/a | +| ISONGraph multi_hop (tuned): mention+entity | n/a | n/a | +48.5 [+45.2, +51.8] (p<0.001) | n/a | n/a | n/a | +| ISONGraph multi_hop (tuned): ctrl_mention | +0.0 [+0.0, +0.0] (p=1.000) | -0.1 [-0.3, +0.0] (p=1.000) | +0.0 [+0.0, +0.0] (p=1.000) | -0.2 [-0.5, +0.0] (p=0.497) | +0.0 [+0.0, +0.0] (p=1.000) | +0.0 [+0.0, +0.0] (p=1.000) | +| ISONGraph multi_hop (tuned): ctrl_entity | n/a | n/a | +0.0 [+0.0, +0.0] (p=1.000) | -0.1 [-0.3, +0.0] (p=1.000) | n/a | n/a | +| PPR re-rank (tuned): mention | +23.2 [+20.5, +26.1] (p<0.001) | +13.4 [+11.0, +15.8] (p<0.001) | +46.2 [+42.8, +49.7] (p<0.001) | +48.9 [+45.8, +52.1] (p<0.001) | +10.7 [+8.4, +13.3] (p<0.001) | +8.2 [+5.9, +10.5] (p<0.001) | +| PPR re-rank (tuned): entity | +9.2 [+7.1, +11.5] (p<0.001) | n/a | +22.4 [+19.7, +25.1] (p<0.001) | +25.8 [+23.0, +28.6] (p<0.001) | n/a | n/a | +| PPR re-rank (tuned): knn | +4.0 [+2.1, +5.9] (p<0.001) | +3.2 [+1.2, +5.2] (p=0.002) | +7.4 [+5.3, +9.6] (p<0.001) | +14.6 [+12.3, +16.8] (p<0.001) | n/a | n/a | +| PPR re-rank (tuned): mention+entity | n/a | n/a | +39.5 [+36.3, +42.7] (p<0.001) | n/a | n/a | n/a | +| PPR re-rank (tuned): ctrl_mention | +0.1 [-0.5, +0.7] (p=1.000) | -0.3 [-0.9, +0.3] (p=0.544) | +0.2 [-0.3, +0.7] (p=0.688) | +0.0 [+0.0, +0.0] (p=1.000) | +0.0 [-0.3, +0.3] (p=1.000) | -0.1 [-0.3, +0.0] (p=1.000) | +| PPR re-rank (tuned): ctrl_entity | n/a | n/a | -0.9 [-1.7, -0.1] (p=0.052) | +0.0 [-0.3, +0.3] (p=1.000) | n/a | n/a | +| ISONGraph multi_hop (defaults): mention | +19.1 [+16.2, +22.3] (p<0.001) | +9.0 [+6.6, +11.5] (p<0.001) | +47.0 [+43.5, +50.4] (p<0.001) | +50.7 [+47.5, +53.9] (p<0.001) | +6.3 [+3.8, +9.1] (p<0.001) | +5.4 [+3.0, +8.0] (p<0.001) | +| ISONGraph multi_hop (defaults): entity | +4.6 [+1.6, +7.7] (p=0.004) | n/a | +24.0 [+20.6, +27.2] (p<0.001) | +25.4 [+22.4, +28.3] (p<0.001) | n/a | n/a | +| ISONGraph multi_hop (defaults): knn | -0.5 [-3.2, +2.3] (p=0.772) | +2.4 [+0.3, +4.4] (p=0.040) | +7.3 [+4.8, +9.8] (p<0.001) | +17.5 [+15.0, +19.9] (p<0.001) | n/a | n/a | +| ISONGraph multi_hop (defaults): mention+entity | n/a | n/a | +36.9 [+33.5, +40.3] (p<0.001) | n/a | n/a | n/a | +| ISONGraph multi_hop (defaults): ctrl_mention | -11.1 [-13.1, -9.2] (p<0.001) | -10.2 [-12.2, -8.4] (p<0.001) | -6.3 [-7.9, -4.9] (p<0.001) | -5.5 [-6.9, -4.1] (p<0.001) | -6.6 [-8.2, -5.0] (p<0.001) | -6.7 [-8.3, -5.1] (p<0.001) | +| ISONGraph multi_hop (defaults): ctrl_entity | n/a | n/a | -9.6 [-11.5, -7.8] (p<0.001) | -5.8 [-7.3, -4.3] (p<0.001) | n/a | n/a | + +### 3.1 ISONGraph multi_hop vs the non-graph alternatives and vs PPR, dev-selected graph (paired, all-supporting@k, pp) + +| Dataset/encoder | k | graph (dev-selected) | multi_hop - RRF (pp) [95% CI] (p) | multi_hop - cross-encoder rerank (pp) [95% CI] (p) | multi_hop - PPR (pp) [95% CI] (p) | +|---|---|---|---|---|---| +| hotpotqa/minilm | 5 | mention | +16.8 [+13.6, +20.2] (p=<0.001) | +11.4 [+8.1, +14.9] (p=<0.001) | +7.5 [+4.9, +10.0] (p=<0.001; PPR graph=mention) | +| hotpotqa/minilm | 10 | mention | +9.0 [+6.0, +12.1] (p=<0.001) | +13.1 [+10.1, +16.2] (p=<0.001) | +0.1 [-1.4, +1.5] (p=1.000; PPR graph=mention) | +| hotpotqa/minilm | 20 | mention | +3.5 [+0.9, +6.0] (p=0.013) | +17.0 [+14.2, +19.7] (p=<0.001) | +0.0 [-1.1, +1.1] (p=1.000; PPR graph=mention) | +| hotpotqa/bge | 5 | mention | +12.2 [+9.4, +15.3] (p=<0.001) | +9.7 [+6.8, +12.8] (p=<0.001) | +1.7 [-0.6, +3.9] (p=0.167; PPR graph=mention) | +| hotpotqa/bge | 10 | mention | +7.7 [+5.3, +10.2] (p=<0.001) | +7.4 [+5.1, +9.8] (p=<0.001) | -0.4 [-1.4, +0.6] (p=0.549; PPR graph=mention) | +| hotpotqa/bge | 20 | mention | +5.6 [+3.8, +7.4] (p=<0.001) | +7.7 [+6.1, +9.5] (p=<0.001) | +0.0 [-0.9, +0.9] (p=1.000; PPR graph=mention) | +| 2wiki/minilm | 5 | mention+entity | +30.6 [+26.9, +34.2] (p=<0.001) | +30.0 [+26.3, +33.6] (p=<0.001) | +12.0 [+9.6, +14.4] (p=<0.001; PPR graph=mention) | +| 2wiki/minilm | 10 | mention+entity | +39.9 [+36.0, +43.6] (p=<0.001) | +43.5 [+40.0, +47.2] (p=<0.001) | +2.3 [+1.3, +3.4] (p=<0.001; PPR graph=mention) | +| 2wiki/minilm | 20 | mention+entity | +40.6 [+36.9, +44.2] (p=<0.001) | +48.3 [+45.2, +51.6] (p=<0.001) | +1.6 [+0.3, +2.9] (p=0.019; PPR graph=mention) | +| 2wiki/bge | 5 | mention | +39.1 [+35.7, +42.2] (p=<0.001) | +38.4 [+35.1, +41.7] (p=<0.001) | +18.9 [+15.9, +21.9] (p=<0.001; PPR graph=mention) | +| 2wiki/bge | 10 | mention | +49.0 [+45.7, +52.1] (p=<0.001) | +48.6 [+45.4, +51.8] (p=<0.001) | +1.8 [+0.7, +2.9] (p=0.002; PPR graph=mention) | +| 2wiki/bge | 20 | mention | +47.0 [+43.8, +50.1] (p=<0.001) | +46.3 [+43.3, +49.5] (p=<0.001) | +0.9 [+0.4, +1.5] (p=0.004; PPR graph=mention) | +| musique/minilm | 5 | mention | +5.8 [+3.7, +8.0] (p=<0.001) | +0.6 [-1.6, +2.9] (p=0.672) | +0.1 [-1.6, +1.7] (p=1.000; PPR graph=mention) | +| musique/minilm | 10 | mention | +7.0 [+4.4, +9.8] (p=<0.001) | +3.7 [+1.2, +6.4] (p=0.005) | -1.5 [-3.3, +0.2] (p=0.120; PPR graph=mention) | +| musique/minilm | 20 | mention | +5.8 [+2.6, +9.0] (p=<0.001) | +8.5 [+5.6, +11.5] (p=<0.001) | -0.4 [-1.9, +1.1] (p=0.679; PPR graph=mention) | +| musique/bge | 5 | mention | +6.7 [+4.5, +9.2] (p=<0.001) | +2.7 [+0.2, +5.5] (p=0.046) | +1.4 [-0.1, +2.9] (p=0.091; PPR graph=mention) | +| musique/bge | 10 | mention | +6.3 [+3.5, +9.2] (p=<0.001) | +5.0 [+2.4, +7.7] (p=<0.001) | +0.5 [-1.0, +2.0] (p=0.624; PPR graph=mention) | +| musique/bge | 20 | mention | +5.8 [+2.7, +8.9] (p=<0.001) | +7.1 [+4.2, +9.7] (p=<0.001) | -0.5 [-2.0, +1.0] (p=0.611; PPR graph=mention) | + +### 3.2 Library minus independent PPR on the same graph (all-supporting, pp [95% CI], paired p) + +#### hotpotqa/minilm: ISONGraph multi_hop (tuned) minus PPR re-rank (tuned), same graph; all-supporting (pp [95% CI], paired p) + +| Graph | k=5 | k=10 | k=20 | +|---|---|---|---| +| mention | +7.5 [+4.9, +10.0] (p=<0.001) | +0.1 [-1.4, +1.5] (p=1.000) | +0.0 [-1.1, +1.1] (p=1.000) | +| entity | +8.9 [+6.5, +11.3] (p=<0.001) | +4.4 [+2.5, +6.4] (p=<0.001) | +1.2 [-0.3, +2.7] (p=0.145) | +| knn | +0.6 [-0.6, +1.8] (p=0.393) | +0.0 [-1.5, +1.6] (p=1.000) | +1.3 [+0.0, +2.7] (p=0.069) | + +#### hotpotqa/bge: ISONGraph multi_hop (tuned) minus PPR re-rank (tuned), same graph; all-supporting (pp [95% CI], paired p) + +| Graph | k=5 | k=10 | k=20 | +|---|---|---|---| +| mention | +1.7 [-0.6, +3.9] (p=0.167) | -0.4 [-1.4, +0.6] (p=0.549) | +0.0 [-0.9, +0.9] (p=1.000) | +| knn | +1.4 [-0.7, +3.4] (p=0.200) | +1.3 [+0.1, +2.5] (p=0.052) | +1.9 [+0.6, +3.3] (p=0.011) | + +#### 2wiki/minilm: ISONGraph multi_hop (tuned) minus PPR re-rank (tuned), same graph; all-supporting (pp [95% CI], paired p) + +| Graph | k=5 | k=10 | k=20 | +|---|---|---|---| +| mention | +12.0 [+9.6, +14.4] (p=<0.001) | +0.9 [+0.2, +1.7] (p=0.036) | +0.9 [-0.3, +2.0] (p=0.169) | +| entity | +11.7 [+9.2, +14.1] (p=<0.001) | +5.7 [+3.3, +8.1] (p=<0.001) | +6.0 [+3.7, +8.4] (p=<0.001) | +| knn | +0.7 [-0.8, +2.2] (p=0.409) | -0.1 [-1.9, +1.8] (p=1.000) | +1.9 [+0.3, +3.5] (p=0.026) | +| mention+entity | +25.9 [+22.9, +28.9] (p=<0.001) | +9.0 [+7.0, +10.9] (p=<0.001) | +5.8 [+4.0, +7.6] (p=<0.001) | + +#### 2wiki/bge: ISONGraph multi_hop (tuned) minus PPR re-rank (tuned), same graph; all-supporting (pp [95% CI], paired p) + +| Graph | k=5 | k=10 | k=20 | +|---|---|---|---| +| mention | +18.9 [+15.9, +21.9] (p=<0.001) | +1.8 [+0.7, +2.9] (p=0.002) | +0.9 [+0.4, +1.5] (p=0.004) | +| entity | +10.2 [+7.6, +12.7] (p=<0.001) | +4.5 [+2.2, +6.7] (p=<0.001) | +1.3 [-0.6, +3.3] (p=0.235) | +| knn | +6.8 [+4.7, +8.8] (p=<0.001) | +5.4 [+3.6, +7.2] (p=<0.001) | +1.6 [+0.3, +2.9] (p=0.034) | + +#### musique/minilm: ISONGraph multi_hop (tuned) minus PPR re-rank (tuned), same graph; all-supporting (pp [95% CI], paired p) + +| Graph | k=5 | k=10 | k=20 | +|---|---|---|---| +| mention | +0.1 [-1.6, +1.7] (p=1.000) | -1.5 [-3.3, +0.2] (p=0.120) | -0.4 [-1.9, +1.1] (p=0.679) | + +#### musique/bge: ISONGraph multi_hop (tuned) minus PPR re-rank (tuned), same graph; all-supporting (pp [95% CI], paired p) + +| Graph | k=5 | k=10 | k=20 | +|---|---|---|---| +| mention | +1.4 [-0.1, +2.9] (p=0.091) | +0.5 [-1.0, +2.0] (p=0.624) | -0.5 [-2.0, +1.0] (p=0.611) | + +### 3.3 By question type and hop count (k = 10, all-supporting %, dev-selected graph, delta vs plain vector in pp [95% CI]) + +#### hotpotqa/minilm: breakdown by question type, k=10 -- All-supporting@10 % (Δ vs vector, pp [95% CI]) + +| Group | n | Vector top-k (plain) | Hybrid vector+BM25 (RRF) | Vector top-30 + cross-encoder | ISONGraph multi_hop (tuned) | PPR re-rank (tuned, independent impl.) | +|---|---|---|---|---|---|---| +| bridge | 784 | 51.3 | 66.1 (+14.8 [+12.0, +17.7]) | 62.9 (+11.6 [+9.4, +14.0]) | 82.5 (+31.2 [+27.7, +34.7]) [mention] | 81.6 (+30.4 [+27.0, +33.5]) [mention] | +| comparison | 216 | 83.3 | 95.8 (+12.5 [+8.3, +17.1]) | 88.4 (+5.1 [+2.3, +8.3]) | 77.8 (-5.6 [-9.3, -2.3]) [mention] | 80.6 (-2.8 [-6.0, +0.0]) [mention] | + +#### hotpotqa/bge: breakdown by question type, k=10 -- All-supporting@10 % (Δ vs vector, pp [95% CI]) + +| Group | n | Vector top-k (plain) | Hybrid vector+BM25 (RRF) | Vector top-30 + cross-encoder | ISONGraph multi_hop (tuned) | PPR re-rank (tuned, independent impl.) | +|---|---|---|---|---|---|---| +| bridge | 784 | 73.1 | 79.8 (+6.8 [+4.3, +9.2]) | 80.0 (+6.9 [+4.5, +9.3]) | 89.9 (+16.8 [+13.8, +19.8]) [mention] | 90.4 (+17.3 [+14.4, +20.2]) [mention] | +| comparison | 216 | 98.1 | 98.1 (+0.0 [-1.4, +1.4]) | 99.1 (+0.9 [-0.9, +2.8]) | 97.2 (-0.9 [-2.8, +0.9]) [mention] | 97.2 (-0.9 [-2.8, +0.9]) [mention] | + +#### 2wiki/minilm: breakdown by question type, k=10 -- All-supporting@10 % (Δ vs vector, pp [95% CI]) + +| Group | n | Vector top-k (plain) | Hybrid vector+BM25 (RRF) | Vector top-30 + cross-encoder | ISONGraph multi_hop (tuned) | PPR re-rank (tuned, independent impl.) | +|---|---|---|---|---|---|---| +| bridge_comparison | 221 | 0.0 | 1.4 (+1.4 [+0.0, +3.2]) | 0.0 (+0.0 [+0.0, +0.0]) | 66.5 (+66.5 [+60.2, +72.4]) [mention+entity] | 64.7 (+64.7 [+57.9, +71.0]) [mention] | +| comparison | 238 | 71.0 | 92.0 (+21.0 [+16.0, +26.5]) | 81.5 (+10.5 [+6.7, +14.7]) | 63.0 (-8.0 [-12.6, -3.8]) [mention+entity] | 62.6 (-8.4 [-12.6, -4.6]) [mention] | +| compositional | 422 | 25.1 | 30.1 (+5.0 [+2.1, +7.8]) | 28.2 (+3.1 [+0.9, +5.2]) | 95.0 (+69.9 [+65.2, +74.4]) [mention+entity] | 92.2 (+67.1 [+61.8, +71.8]) [mention] | +| inference | 119 | 42.0 | 52.1 (+10.1 [+4.2, +16.0]) | 52.1 (+10.1 [+4.2, +16.8]) | 94.1 (+52.1 [+42.9, +61.3]) [mention+entity] | 89.1 (+47.1 [+37.8, +56.3]) [mention] | + +#### 2wiki/minilm: breakdown by hop count (number of supporting passages), k=10 -- All-supporting@10 % (Δ vs vector, pp [95% CI]) + +| Group | n | Vector top-k (plain) | Hybrid vector+BM25 (RRF) | Vector top-30 + cross-encoder | ISONGraph multi_hop (tuned) | PPR re-rank (tuned, independent impl.) | +|---|---|---|---|---|---|---| +| 2 | 779 | 41.7 | 52.4 (+10.7 [+8.3, +13.1]) | 48.1 (+6.4 [+4.6, +8.5]) | 85.1 (+43.4 [+39.5, +47.2]) [mention+entity] | 82.7 (+40.9 [+37.0, +44.7]) [mention] | +| 4 | 221 | 0.0 | 1.4 (+1.4 [+0.0, +3.2]) | 0.0 (+0.0 [+0.0, +0.0]) | 66.5 (+66.5 [+60.2, +72.4]) [mention+entity] | 64.7 (+64.7 [+57.9, +71.0]) [mention] | + +#### 2wiki/bge: breakdown by question type, k=10 -- All-supporting@10 % (Δ vs vector, pp [95% CI]) + +| Group | n | Vector top-k (plain) | Hybrid vector+BM25 (RRF) | Vector top-30 + cross-encoder | ISONGraph multi_hop (tuned) | PPR re-rank (tuned, independent impl.) | +|---|---|---|---|---|---|---| +| bridge_comparison | 221 | 0.9 | 1.8 (+0.9 [+0.0, +2.3]) | 1.4 (+0.5 [+0.0, +1.4]) | 90.5 (+89.6 [+85.5, +93.7]) [mention] | 81.9 (+81.0 [+75.6, +86.0]) [mention] | +| comparison | 238 | 97.9 | 99.2 (+1.3 [+0.0, +2.9]) | 97.9 (+0.0 [-1.3, +1.3]) | 97.1 (-0.8 [-2.5, +0.8]) [mention] | 97.9 (+0.0 [-1.3, +1.3]) [mention] | +| compositional | 422 | 32.9 | 34.4 (+1.4 [-1.7, +4.5]) | 36.7 (+3.8 [+0.7, +6.9]) | 95.3 (+62.3 [+57.6, +67.3]) [mention] | 95.0 (+62.1 [+57.6, +66.8]) [mention] | +| inference | 119 | 54.6 | 59.7 (+5.0 [-0.8, +11.8]) | 58.0 (+3.4 [-3.4, +10.1]) | 95.0 (+40.3 [+31.1, +49.6]) [mention] | 95.0 (+40.3 [+31.9, +48.7]) [mention] | + +#### 2wiki/bge: breakdown by hop count (number of supporting passages), k=10 -- All-supporting@10 % (Δ vs vector, pp [95% CI]) + +| Group | n | Vector top-k (plain) | Hybrid vector+BM25 (RRF) | Vector top-30 + cross-encoder | ISONGraph multi_hop (tuned) | PPR re-rank (tuned, independent impl.) | +|---|---|---|---|---|---|---| +| 2 | 779 | 56.1 | 58.0 (+1.9 [-0.0, +3.9]) | 58.7 (+2.6 [+0.5, +4.7]) | 95.8 (+39.7 [+36.2, +43.1]) [mention] | 95.9 (+39.8 [+36.3, +43.3]) [mention] | +| 4 | 221 | 0.9 | 1.8 (+0.9 [+0.0, +2.3]) | 1.4 (+0.5 [+0.0, +1.4]) | 90.5 (+89.6 [+85.5, +93.7]) [mention] | 81.9 (+81.0 [+75.6, +86.0]) [mention] | + +#### musique/minilm: breakdown by question type, k=10 -- All-supporting@10 % (Δ vs vector, pp [95% CI]) + +| Group | n | Vector top-k (plain) | Hybrid vector+BM25 (RRF) | Vector top-30 + cross-encoder | ISONGraph multi_hop (tuned) | PPR re-rank (tuned, independent impl.) | +|---|---|---|---|---|---|---| +| 2hop | 519 | 30.8 | 34.1 (+3.3 [+0.2, +6.4]) | 37.8 (+6.9 [+4.4, +9.6]) | 46.2 (+15.4 [+11.9, +19.1]) [mention] | 48.0 (+17.1 [+12.9, +21.2]) [mention] | +| 3hop1 | 213 | 4.7 | 5.6 (+0.9 [-2.8, +4.7]) | 11.7 (+7.0 [+3.3, +10.8]) | 9.4 (+4.7 [+1.4, +8.0]) [mention] | 11.3 (+6.6 [+2.3, +10.8]) [mention] | +| 3hop2 | 78 | 12.8 | 16.7 (+3.8 [-5.1, +11.5]) | 16.7 (+3.8 [-3.8, +11.5]) | 14.1 (+1.3 [-2.6, +6.4]) [mention] | 14.1 (+1.3 [-3.8, +6.4]) [mention] | +| 4hop1 | 113 | 0.9 | 0.0 (-0.9 [-2.7, +0.0]) | 1.8 (+0.9 [-1.8, +3.5]) | 1.8 (+0.9 [+0.0, +2.7]) [mention] | 3.5 (+2.7 [+0.0, +6.2]) [mention] | +| 4hop2 | 30 | 0.0 | 0.0 (+0.0 [+0.0, +0.0]) | 0.0 (+0.0 [+0.0, +0.0]) | 0.0 (+0.0 [+0.0, +0.0]) [mention] | 0.0 (+0.0 [+0.0, +0.0]) [mention] | +| 4hop3 | 47 | 2.1 | 4.3 (+2.1 [-4.3, +10.6]) | 2.1 (+0.0 [+0.0, +0.0]) | 2.1 (+0.0 [+0.0, +0.0]) [mention] | 2.1 (+0.0 [+0.0, +0.0]) [mention] | + +#### musique/minilm: breakdown by hop count (number of supporting passages), k=10 -- All-supporting@10 % (Δ vs vector, pp [95% CI]) + +| Group | n | Vector top-k (plain) | Hybrid vector+BM25 (RRF) | Vector top-30 + cross-encoder | ISONGraph multi_hop (tuned) | PPR re-rank (tuned, independent impl.) | +|---|---|---|---|---|---|---| +| 2 | 519 | 30.8 | 34.1 (+3.3 [+0.2, +6.4]) | 37.8 (+6.9 [+4.4, +9.6]) | 46.2 (+15.4 [+11.9, +19.1]) [mention] | 48.0 (+17.1 [+12.9, +21.2]) [mention] | +| 3 | 291 | 6.9 | 8.6 (+1.7 [-1.7, +5.2]) | 13.1 (+6.2 [+2.7, +9.6]) | 10.7 (+3.8 [+1.4, +6.5]) [mention] | 12.0 (+5.2 [+1.7, +8.6]) [mention] | +| 4 | 190 | 1.1 | 1.1 (+0.0 [-2.1, +2.1]) | 1.6 (+0.5 [-1.1, +2.1]) | 1.6 (+0.5 [+0.0, +1.6]) [mention] | 2.6 (+1.6 [+0.0, +3.7]) [mention] | + +#### musique/bge: breakdown by question type, k=10 -- All-supporting@10 % (Δ vs vector, pp [95% CI]) + +| Group | n | Vector top-k (plain) | Hybrid vector+BM25 (RRF) | Vector top-30 + cross-encoder | ISONGraph multi_hop (tuned) | PPR re-rank (tuned, independent impl.) | +|---|---|---|---|---|---|---| +| 2hop | 519 | 33.7 | 37.2 (+3.5 [+0.0, +6.9]) | 39.7 (+6.0 [+2.7, +9.1]) | 48.6 (+14.8 [+11.4, +18.5]) [mention] | 49.1 (+15.4 [+11.6, +19.3]) [mention] | +| 3hop1 | 213 | 10.8 | 13.6 (+2.8 [-2.3, +8.0]) | 12.2 (+1.4 [-2.3, +5.2]) | 15.0 (+4.2 [+0.0, +8.5]) [mention] | 13.6 (+2.8 [-0.9, +6.6]) [mention] | +| 3hop2 | 78 | 14.1 | 12.8 (-1.3 [-6.4, +2.6]) | 16.7 (+2.6 [-2.6, +7.7]) | 15.4 (+1.3 [-2.6, +6.4]) [mention] | 9.0 (-5.1 [-11.5, +0.0]) [mention] | +| 4hop1 | 113 | 0.0 | 0.9 (+0.9 [+0.0, +2.7]) | 0.9 (+0.9 [+0.0, +2.7]) | 0.0 (+0.0 [+0.0, +0.0]) [mention] | 0.0 (+0.0 [+0.0, +0.0]) [mention] | +| 4hop2 | 30 | 0.0 | 0.0 (+0.0 [+0.0, +0.0]) | 0.0 (+0.0 [+0.0, +0.0]) | 0.0 (+0.0 [+0.0, +0.0]) [mention] | 0.0 (+0.0 [+0.0, +0.0]) [mention] | +| 4hop3 | 47 | 0.0 | 0.0 (+0.0 [+0.0, +0.0]) | 0.0 (+0.0 [+0.0, +0.0]) | 0.0 (+0.0 [+0.0, +0.0]) [mention] | 0.0 (+0.0 [+0.0, +0.0]) [mention] | + +#### musique/bge: breakdown by hop count (number of supporting passages), k=10 -- All-supporting@10 % (Δ vs vector, pp [95% CI]) + +| Group | n | Vector top-k (plain) | Hybrid vector+BM25 (RRF) | Vector top-30 + cross-encoder | ISONGraph multi_hop (tuned) | PPR re-rank (tuned, independent impl.) | +|---|---|---|---|---|---|---| +| 2 | 519 | 33.7 | 37.2 (+3.5 [+0.0, +6.9]) | 39.7 (+6.0 [+2.7, +9.1]) | 48.6 (+14.8 [+11.4, +18.5]) [mention] | 49.1 (+15.4 [+11.6, +19.3]) [mention] | +| 3 | 291 | 11.7 | 13.4 (+1.7 [-2.4, +5.9]) | 13.4 (+1.7 [-1.0, +4.8]) | 15.1 (+3.4 [+0.3, +6.5]) [mention] | 12.4 (+0.7 [-2.7, +4.1]) [mention] | +| 4 | 190 | 0.0 | 0.5 (+0.5 [+0.0, +1.6]) | 0.5 (+0.5 [+0.0, +1.6]) | 0.0 (+0.0 [+0.0, +0.0]) [mention] | 0.0 (+0.0 [+0.0, +0.0]) [mention] | + +### 3.4 `semantic_subgraph(connect=True)`, set quality vs plain vector of the same size (per query) + +`semantic_subgraph` returns an unordered set (no MRR). Parameters = the dev-selected `multi_hop` parameters for that k; `connect_max_hops` 2 and 3. Recall and all-supporting of the returned set are compared with plain vector top-|S_q| (same number of passages for that question). Full table for every k/graph in `results/tables.md` (`semantic_subgraph(connect=True)` sections). Example, k = 10: + +**hotpotqa / minilm** + +| Graph | connect_max_hops | mean |S| | Recall % | All-supp. % | Vector@|S| recall % | Vector@|S| all % | Δ all (pp) [CI] | p | Δ recall (pp) [CI] | p | +|---|---|---|---|---|---|---|---|---|---|---| +| mention | 2 | 9.5 | 88.8 [87.2, 90.3] | 81.7 [79.3, 84.0] | 76.5 [74.8, 78.3] | 57.1 [54.1, 60.1] | +24.6 [+21.8, +27.7] | <0.001 | +12.3 [+10.8, +13.9] | <0.001 | +| mention | 3 | 9.5 | 88.8 [87.2, 90.3] | 81.8 [79.4, 84.1] | 76.6 [74.8, 78.3] | 57.1 [54.1, 60.1] | +24.7 [+21.8, +27.8] | <0.001 | +12.3 [+10.8, +13.9] | <0.001 | +| entity | 2 | 10.3 | 84.7 [83.0, 86.3] | 72.1 [69.4, 74.8] | 77.5 [75.8, 79.2] | 58.6 [55.6, 61.6] | +13.5 [+11.1, +16.0] | <0.001 | +7.2 [+6.0, +8.4] | <0.001 | +| entity | 3 | 11.5 | 84.9 [83.2, 86.4] | 72.3 [69.6, 75.0] | 78.5 [76.7, 80.1] | 60.0 [57.0, 63.0] | +12.3 [+9.7, +14.9] | <0.001 | +6.4 [+5.1, +7.8] | <0.001 | +| knn | 2 | 10.4 | 78.9 [77.1, 80.5] | 62.2 [59.2, 65.3] | 77.5 [75.8, 79.2] | 58.7 [55.7, 61.8] | +3.5 [+1.4, +5.7] | 0.002 | +1.3 [+0.2, +2.4] | 0.034 | +| knn | 3 | 11.2 | 78.9 [77.1, 80.6] | 62.3 [59.3, 65.3] | 78.2 [76.5, 80.0] | 59.7 [56.7, 62.8] | +2.6 [+0.5, +4.9] | 0.028 | +0.7 [-0.4, +1.9] | 0.285 | +| ctrl_mention | 2 | 10.0 | 77.2 [75.5, 79.0] | 58.2 [55.2, 61.3] | 77.2 [75.5, 79.0] | 58.2 [55.2, 61.3] | +0.0 [+0.0, +0.0] | 1.000 | +0.0 [+0.0, +0.0] | 1.000 | +| ctrl_mention | 3 | 10.0 | 77.2 [75.5, 79.0] | 58.2 [55.2, 61.3] | 77.2 [75.5, 79.0] | 58.2 [55.2, 61.3] | +0.0 [+0.0, +0.0] | 1.000 | +0.0 [+0.0, +0.0] | 1.000 | + +**2wiki / bge** + +| Graph | connect_max_hops | mean |S| | Recall % | All-supp. % | Vector@|S| recall % | Vector@|S| all % | Δ all (pp) [CI] | p | Δ recall (pp) [CI] | p | +|---|---|---|---|---|---|---|---|---|---|---| +| mention | 2 | 8.2 | 97.4 [96.7, 98.1] | 94.6 [93.2, 96.0] | 71.5 [69.9, 73.0] | 42.4 [39.4, 45.4] | +52.2 [+49.0, +55.3] | <0.001 | +25.9 [+24.3, +27.5] | <0.001 | +| mention | 3 | 8.2 | 97.4 [96.7, 98.1] | 94.6 [93.2, 96.0] | 71.5 [69.9, 73.0] | 42.4 [39.4, 45.4] | +52.2 [+49.0, +55.3] | <0.001 | +25.9 [+24.3, +27.5] | <0.001 | +| entity | 2 | 10.5 | 89.7 [88.6, 90.9] | 74.4 [71.7, 77.1] | 72.6 [71.0, 74.1] | 44.5 [41.4, 47.6] | +29.9 [+26.9, +32.7] | <0.001 | +17.1 [+15.6, +18.5] | <0.001 | +| entity | 3 | 12.3 | 90.0 [88.8, 91.1] | 75.1 [72.4, 77.7] | 73.0 [71.5, 74.6] | 45.0 [42.0, 48.0] | +30.1 [+27.1, +32.9] | <0.001 | +17.0 [+15.4, +18.4] | <0.001 | +| knn | 2 | 10.7 | 84.1 [82.7, 85.5] | 64.3 [61.3, 67.1] | 72.5 [71.0, 74.1] | 44.3 [41.3, 47.3] | +20.0 [+17.4, +22.4] | <0.001 | +11.6 [+10.3, +12.9] | <0.001 | +| knn | 3 | 12.1 | 84.5 [83.2, 85.9] | 65.0 [62.1, 67.8] | 72.8 [71.3, 74.4] | 44.7 [41.7, 47.7] | +20.3 [+17.7, +22.7] | <0.001 | +11.7 [+10.4, +13.0] | <0.001 | +| ctrl_mention | 2 | 10.0 | 72.2 [70.7, 73.7] | 43.7 [40.7, 46.7] | 72.3 [70.8, 73.8] | 43.9 [40.9, 46.9] | -0.2 [-0.5, +0.0] | 0.497 | -0.1 [-0.2, +0.0] | 0.497 | +| ctrl_mention | 3 | 10.0 | 72.2 [70.7, 73.7] | 43.7 [40.7, 46.7] | 72.3 [70.8, 73.8] | 43.9 [40.9, 46.9] | -0.2 [-0.5, +0.0] | 0.497 | -0.1 [-0.2, +0.0] | 0.497 | +| ctrl_entity | 2 | 10.3 | 72.3 [70.7, 73.8] | 43.8 [40.8, 46.8] | 72.4 [70.8, 73.9] | 44.0 [40.9, 47.0] | -0.2 [-0.5, +0.0] | 0.497 | -0.1 [-0.2, +0.0] | 0.497 | +| ctrl_entity | 3 | 15.9 | 72.3 [70.7, 73.8] | 43.8 [40.8, 46.8] | 74.0 [72.5, 75.6] | 46.7 [43.6, 49.9] | -2.9 [-4.0, -1.9] | <0.001 | -1.8 [-2.3, -1.2] | <0.001 | + +**musique / minilm** + +| Graph | connect_max_hops | mean |S| | Recall % | All-supp. % | Vector@|S| recall % | Vector@|S| all % | Δ all (pp) [CI] | p | Δ recall (pp) [CI] | p | +|---|---|---|---|---|---|---|---|---|---|---| +| mention | 2 | 10.3 | 57.7 [55.7, 59.6] | 27.4 [24.7, 30.2] | 53.6 [51.8, 55.3] | 18.5 [16.1, 20.9] | +8.9 [+6.9, +11.0] | <0.001 | +4.1 [+3.0, +5.2] | <0.001 | +| mention | 3 | 10.8 | 57.9 [56.0, 59.8] | 27.7 [24.9, 30.4] | 54.1 [52.4, 55.9] | 19.0 [16.6, 21.5] | +8.7 [+6.6, +10.9] | <0.001 | +3.8 [+2.6, +4.9] | <0.001 | +| ctrl_mention | 2 | 10.0 | 53.2 [51.4, 54.9] | 18.2 [15.8, 20.6] | 53.2 [51.5, 55.0] | 18.2 [15.8, 20.6] | +0.0 [+0.0, +0.0] | 1.000 | -0.0 [-0.1, +0.0] | 1.000 | +| ctrl_mention | 3 | 10.0 | 53.2 [51.4, 54.9] | 18.2 [15.8, 20.6] | 53.2 [51.5, 55.0] | 18.2 [15.8, 20.6] | +0.0 [+0.0, +0.0] | 1.000 | -0.0 [-0.1, +0.0] | 1.000 | + +In every run the connector step added at most about one node on average (mean |S| 8 to 12), so `connect=True` is equivalent to `multi_hop` here; whether connectors matter for LLM context assembly is not something this benchmark tests. + +### 3.5 Metrics not in the headline + +Recall@k, MRR@k and the mean number of returned passages for every method, graph and k, with Holm-adjusted p-values, are in `results/tables.md`. Observations: (i) MRR of the first supporting passage is *not* improved by the graph methods (tuned `multi_hop` MRR@10 equals plain vector's within 0.002 in all six dataset/encoder cells, because the vector hits stay on top; PPR is lower by 0.015 to 0.09) - graph expansion adds the missing hop *after* the first hit; the cross-encoder is the only method that raises MRR (0.94 vs 0.86); (ii) with library defaults `semantic_multi_hop` returns fewer than k passages (7.8-9.5 of 10 on average), because `top_k_seeds=5` seeds only expand to a few neighbours; this is *not* padded and costs it a little recall. + +## 4. What ran and what did not + +Graph jobs = (dataset, encoder, graph variant): tune on dev, evaluate on test. "100" = 100 random library configurations + library defaults; "lean" = 40 + defaults, subgraph only for k = 10 (re-launched when the first pass was too slow). PPR always evaluates its whole grid (72 configurations; 144 for union graphs). + +| Graph variant | hotpotqa/minilm | hotpotqa/bge | 2wiki/minilm | 2wiki/bge | musique/minilm | musique/bge | +|---|---|---|---|---|---|---| +| mention | 100 | 100 | 100 | 100 | lean | lean | +| entity | 100 | not run | 100 | lean | not run | not run | +| knn | 100 | 100 | 100 | lean | not run | not run | +| mention+entity | not run | not run | 100 | not run | not run | not run | +| all | not run | not run | not run | not run | not run | not run | +| ctrl_mention | 100 | 100 | 100 | 100 | lean | lean | +| ctrl_entity | not run | not run | 100 | 100 | not run | not run | + +* Baselines (vector, BM25, RRF, cross-encoder) ran on all 6 dataset/encoder combinations. +* The first pass (22 parallel jobs on 32 logical cores) was far slower than estimated; at the 110-minute mark the unfinished jobs were killed. **Not run / killed unfinished:** `all` (mention+entity+kNN) everywhere; `mention+entity` except 2Wiki/MiniLM; `entity` for HotpotQA/bge and MuSiQue; `knn` for MuSiQue; the entity negative control for HotpotQA and MuSiQue. `ctrl_mention` ran for every dataset/encoder. +* Latency was measured last on a (mostly) idle machine: single thread, warm, 200 queries (100 for subgraph, 8 for the CPU cross-encoder). Other users' background processes (editor / unrelated services) were present, so treat the absolute numbers as +/- 30%: the same cell measured twice (an aborted first attempt) gave 1.8 ms vs 0.7 ms for the library's plain `similarity_search`. + +## 5. Negative controls + +`ctrl_mention` / `ctrl_entity` use the same edge lists with node labels permuted. After dev tuning the library picks a configuration that is effectively "plain vector" (8 cells = 6 ctrl_mention + 2 ctrl_entity: delta 0.0 to -0.2 pp, every p >= 0.5); the PPR re-ranker likewise (|delta| <= 0.9). With the **library defaults** on a permuted graph the result is *worse* than plain vector by 5.5 to 11.1 pp (all p < 0.001): a random graph injects wrong neighbours into the top-k. In other words: the library defaults are only safe when the graph is informative, and a bad graph costs up to 11 pp of all-supporting@10. + +## 6. Latency (ms per query, single thread, warm; query embedding excluded for every method) + +| Method (mean ms/query; median in brackets) | hotpotqa/minilm | 2wiki/minilm | musique/minilm | hotpotqa/bge | +|---|---|---|---|---| +| vector (numpy brute force top-20) | 0.49 (0.47) | 0.76 (0.73) | 2.06 (1.92) | 1.34 (1.31) | +| ISONGraph similarity_search top-20 (plain vector via the library) | 0.69 (0.68) | 0.94 (0.85) | 2.22 (2.17) | 2.10 (2.01) | +| BM25 (scipy sparse) | 0.15 (0.15) | 0.46 (0.44) | 0.41 (0.38) | 0.53 (0.51) | +| Hybrid RRF (dense + BM25 + fusion) | 5.39 (5.38) | 3.87 (3.79) | 6.22 (6.09) | 6.04 (5.83) | +| PPR re-rank, graph=mention | 11.18 (11.42) | 6.83 (6.73) | 13.70 (13.62) | 13.12 (12.99) | +| ISONGraph semantic_multi_hop (tuned) | 12.34 (12.09) | 3.42 (2.92) | 16.80 (16.36) | 2.15 (2.11) | +| ISONGraph semantic_subgraph(connect=True) | 20.60 (20.79) | 39.91 (35.44) | 31.09 (30.49) | 10.56 (10.06) | +| Vector top-30 + cross-encoder (CPU, 1 thread) | 6843.02 (7238.13) | - | - | - | + +Timed multi_hop/subgraph configurations (dev-selected at k=10): + +- hotpotqa/minilm: graph `mention`, config `{'hops': 2, 'seeds': 5, 'decay': 0.5, 'blend': 0.3, 'direction': 'both', 'weight_field': None}` +- 2wiki/minilm: graph `mention+entity`, config `{'hops': 2, 'seeds': 5, 'decay': 0.7, 'blend': 0.0, 'direction': 'out', 'weight_field': 'w'}` +- musique/minilm: graph `mention`, config `{'hops': 2, 'seeds': 10, 'decay': 0.9, 'blend': 0.3, 'direction': 'both', 'weight_field': None}` +- hotpotqa/bge: graph `mention`, config `{'hops': 2, 'seeds': 5, 'decay': 0.7, 'blend': 0.0, 'direction': 'both', 'weight_field': None}` + + +* Plain vector search over 9k-15k vectors is sub-millisecond to 2 ms; everything graph-aware costs 2-17 ms per query in Python (`semantic_multi_hop` with `blend=0`: 2-3 ms; with `blend>0` it scores every node and builds a dict of all N scores per call: 11-17 ms here, growing linearly with corpus size). +* The cross-encoder is by far the most expensive (about 7 s per query for 30 candidates on one CPU thread; on the GPU it is a few tens of ms, not measured). +* The independent PPR re-ranker costs 7-14 ms per query (25 sparse power iterations over the whole corpus) - the same order as `multi_hop` with blend. + +## 7. Library notes (no bug found; behaviours worth knowing) + +The library was used unmodified. No incorrect result was found: `SemanticGraph.similarity_search` returns the same top-20 *set* as an independent numpy brute force for 200/200 queries on 4 of 6 setups and 198/200 on both 2Wiki setups (boundary ties). Observations: + +1. `semantic_multi_hop` does not guarantee `top_k_results` results: with `top_k_seeds=5` and `top_k_results=10` it returned 7.8-9.5 passages on average. Callers who need exactly k must raise `top_k_seeds` or pad themselves. A documented "fill from remaining seeds / vector ranks" option would remove a recall handicap. +2. With `blend > 0`, `EmbeddingStore.score_map` materialises `dict(zip(cache.refs, scores.tolist()))` for the whole corpus on every query (see `isongraph_vector-py/__init__.py`, `score_map`, `semantic_multi_hop`): measured 11-17 ms per query at 15k nodes vs 2-3 ms with `blend=0`. Linear in N, in Python objects, so it will dominate at 1M nodes; a numpy-array path restricted to the visited nodes would avoid it. +3. `semantic_subgraph` builds a new `SemanticGraph` (with its own in-memory SQLite) per call: 10-40 ms per query vs 2-17 ms for `multi_hop`. For per-query use this dominates. +4. The default `threshold=0.3` of `semantic_multi_hop` was irrelevant here (top-1 cosine >= 0.3 for all 6,000 test queries with both encoders; `results/threshold_check.json`), but it silently depends on the encoder's cosine scale. +5. Library defaults (`decay=0.8`, `top_k_seeds=5`, `max_hops=2`) were close to the tuned optimum on mention graphs; the dev-selected settings at k = 10 on the mention graph varied between cells (hops 1-3, seeds 5-20, decay 0.5-0.9, blend 0-0.3; listed at the end of `results/tables.md`), which suggests the defaults are a reasonable ship-with choice. + +## 8. Reproduction + +Everything is driven by `run.py` (documented in its docstring). Seeds: sampling `13`, library random search `13`, bootstrap/permutation `20240607`, label permutation for controls `7`. Embeddings and corpora are cached outside the repo in `D:\Goldmine\ISON\bench_data\multihop\` (`corpus_<ds>.json`, `questions_<ds>.json`, `emb_<ds>__<model>__{passages,queries}.npy`, edge pickles); raw per-question results are in `results/raw/*.npz|json`. + +```powershell +# one-time: a venv that re-uses the already-installed torch / sentence-transformers / datasets +# (nothing is installed into the global Python or the repo; no extra packages were needed) +python -m venv --system-site-packages D:\Goldmine\ISON\bench_env_mh +$py = "D:\Goldmine\ISON\bench_env_mh\Scripts\python.exe" +cd isongraph-vector\benchmarks\multihop + +& $py run.py prepare # download pinned dev sets, sample 1,500 q per dataset, build pooled corpora +& $py run.py embed # MiniLM + bge-small embeddings (GPU if available) ~ 5 min +& $py run.py edges # mention / entity / kNN edge lists ~ 20 s +& $py run.py baselines # vector, BM25, RRF, cross-encoder ~ 8 min +& $py run.py graphs --workers 12 # tune on dev, evaluate on test, all variants + controls + # (the full 100-config budget takes hours of CPU; --variants / --jobs ds:model:variant + # select subsets; env MH_N_CONFIGS=40 MH_SUB_KS=10 gives the "lean" budget) +$env:OMP_NUM_THREADS=1; $env:MKL_NUM_THREADS=1; $env:OPENBLAS_NUM_THREADS=1 +& $py run.py latency # on an idle machine +& $py run.py report # tables.md, headline.md, summary.json +& $py extra_pairs.py # paired tests vs RRF / cross-encoder / PPR +& $py threshold_check.py # default seed-threshold check +``` + +Files: `common.py` (constants), `prepare_data.py`, `embed.py`, `graphs.py` (edge construction, negative controls, diagnostics), `methods.py` (metrics, BM25, RRF, cross-encoder, PPR, library wrappers), `jobs.py`, `report.py` + `headline.py`, `latency.py`, `extra_pairs.py`, `threshold_check.py`; `manifests/` (sampled ids and split per dataset, corpus hash); `results/`. + +## 9. Suggested text for the root README / CHANGELOG (not applied) + +README, "Retrieval quality" section - replace any synthetic-benchmark claim with: + +> **Does graph-aware retrieval beat plain vector search on real data?** Yes, when the graph carries relevant structure. On HotpotQA, 2WikiMultihopQA and MuSiQue (1,000 held-out dev questions each, MiniLM and bge-small embeddings, a graph built only from title mentions in the passage text, no gold annotations), `semantic_multi_hop` retrieves *all* supporting passages for +9 to +51 percentage points more questions at k=10 than plain vector top-k (paired p < 0.001), and also beats hybrid BM25 fusion and a cross-encoder re-ranker over the same candidates. The gain disappears on a randomly re-labelled graph, is smaller for similarity-only (kNN) graphs (+4 to +20 pp), absent for questions that name both entities (comparison questions, where it costs up to 8 pp), and an independently implemented personalised-PageRank re-ranker over the same graph performs the same at k >= 10. The advantage is therefore *graph-aware retrieval*, not a unique property of this implementation. Numbers, method and threats to validity: `benchmarks/multihop/README.md`. + +CHANGELOG: + +> - Added `benchmarks/multihop/`: multi-hop retrieval benchmark on real data (HotpotQA, 2WikiMultihopQA, MuSiQue), two encoders, graph variants, negative controls, PPR / BM25 / hybrid / cross-encoder baselines, bootstrap CIs and paired tests. Supersedes the synthetic-data claim in `benchmarks/retrieval_quality`. + +Also worth considering for the docs (from Section 7): state that `semantic_multi_hop` may return fewer than `top_k_results`, and that `blend > 0` scores all nodes per query. + +## 10. Threats to validity (everything we could identify) + +1. **The graph is aligned with how the datasets were made.** HotpotQA and 2WikiMultihopQA questions were *constructed* from pairs of Wikipedia paragraphs linked by a hyperlink / shared entity, and MuSiQue was composed from single-hop questions about chained entities. A title-mention graph recovers exactly that relation, which is why it works so well there (see "gold pairs directly linked" in the diagnostics). That is *legitimate* (the graph never sees the question or the annotation, and the shuffled-label controls collapse to or below plain vector search), but it means the size of the gain is an upper-ish estimate for corpora that have comparably informative links, and says nothing about corpora whose graph is noisier (the `knn`/`entity` rows are the closer analogues). +2. **Corpus pooling effect.** Corpora are 9k-15k passages pooled from 1,500 questions, built from each question's own distractors. In this pool a gold passage's hyperlink neighbours are overwhelmingly *other passages of the same question's context* (the "neighbours of gold inside the same context" column), which makes graph expansion more precise than it would be in a full-Wikipedia (5M+) index, where hub pages and many more same-named entities dilute edges. Absolute numbers are inflated for every method relative to open-domain retrieval; the gaps between methods are the only transferable quantity, and even those may shrink or grow at scale. Not tested. +3. **Encoder strength.** Two small encoders (MiniLM-L6, bge-small). The graph gain is larger for the weaker encoder (it leaves more headroom). Larger / multi-vector encoders and late-interaction models were not tested. `bge-small-en-v1.5` and the MS-MARCO cross-encoder may have seen HotpotQA or MuSiQue *training* data during (fine-)tuning; dev questions were used here, but exposure of the encoder to the training distribution cannot be ruled out and would favour the vector/rerank baselines, not the graph. +4. **Hyper-parameter selection.** The library got a random search of 100 configurations (40 in the lean runs, see Section 4) per graph and k, PPR a full 72/144-point grid, both selected on a 500-question dev split and evaluated on a disjoint 1,000-question test split. Selection optimism (dev vs test) is tabulated in `results/tables.md`. The vector / BM25 / RRF / rerank baselines have no tuned parameters (BM25 k1/b and RRF k0 are the textbook defaults; rerank depth 30 was fixed a priori) - they could be tuned too, and the reranker depth in particular is a lever we did not pull. Random search over 100 points can miss the library's optimum; the *library-defaults* row shows the gain does not depend on tuning. +5. **Edge-construction constants** (hub cap 30 for mentions, entity df window [2,15], kNN k=5, min title length 4) were fixed a priori and not swept; different values would change the `mention`/`entity` results. Entity extraction is a crude capitalisation heuristic. +6. **Multiple comparisons.** Many rows per table; Holm correction is applied within each table but the choice of which graph/k/model to look at is the reader's. We report all variants including those that did not help. +7. **Small effects.** Where differences are within a couple of points (e.g. library vs PPR on the same graph, `knn` graphs, MuSiQue's 3-4 hop strata with n in the tens) the CIs overlap zero and no ranking should be inferred. Strata with few questions are reported with their n. +8. **Metric definition.** "All-supporting@k" counts gold *paragraphs*, not supporting sentences, and treats a paragraph as retrieved when its id is in the top-k. A returned list shorter than k (the library returns fewer than k when seeds have too few reachable neighbours) is *not* padded - this penalises the library slightly. MRR is MRR@k. +9. **Single-machine, single-run latency**, brute-force vector search at 9k-15k vectors (no ANN). Latency ranks of the methods at larger scale could differ. +10. **Dataset copies.** 2Wiki and MuSiQue were read from HuggingFace mirrors rather than the original hosting sites. 2Wiki's mirror carries the official dev split; MuSiQue's file is the official `musique_ans_v1.0_dev.jsonl`. Questions whose supporting title is missing from the context were dropped (none for the samples drawn). +11. **The libraries' two similarity paths differ on ties**: in 2 of 200 checked queries on one setup `SemanticGraph.similarity_search` and our numpy scan return different top-20 *sets* (boundary ties); negligible, but "plain vector" here is the numpy scan. diff --git a/benchmarks/multihop/common.py b/benchmarks/multihop/common.py new file mode 100644 index 0000000..c664ab5 --- /dev/null +++ b/benchmarks/multihop/common.py @@ -0,0 +1,51 @@ +"""Shared constants and helpers for the multi-hop retrieval benchmark.""" +from __future__ import annotations + +import json +import os +import sys +from pathlib import Path + +os.environ.setdefault("USE_TF", "0") +os.environ.setdefault("TRANSFORMERS_NO_TF", "1") +os.environ.setdefault("TOKENIZERS_PARALLELISM", "false") + +HERE = Path(__file__).resolve().parent +REPO_ROOT = HERE.parent.parent +DATA_DIR = Path(os.environ.get("MH_DATA_DIR", r"D:\Goldmine\ISON\bench_data\multihop")) +RAW_DIR = DATA_DIR / "raw" +RESULTS_DIR = HERE / "results" +MANIFEST_DIR = HERE / "manifests" +for _d in (DATA_DIR, RAW_DIR, RESULTS_DIR, MANIFEST_DIR): + _d.mkdir(parents=True, exist_ok=True) + +DATASETS = ["hotpotqa", "2wiki", "musique"] +MODELS = { + # short name -> (HF id, query prefix, passage prefix) + "minilm": ("sentence-transformers/all-MiniLM-L6-v2", "", ""), + "bge": ("BAAI/bge-small-en-v1.5", + "Represent this sentence for searching relevant passages: ", ""), +} +RERANKER = "cross-encoder/ms-marco-MiniLM-L-6-v2" + +SEED = 13 +N_QUESTIONS = 1500 # sampled per dataset +N_DEV = 500 # hyper-parameter selection split; the rest (1000) is the test split +KS = (5, 10, 20) + + +def import_lib(): + """Import isongraph_vector from the repo checkout, unmodified.""" + sys.path.insert(0, str(REPO_ROOT / "benchmarks")) + from _common import import_package + return import_package() + + +def load_json(path): + with open(path, "r", encoding="utf-8") as fh: + return json.load(fh) + + +def dump_json(obj, path, indent=None): + with open(path, "w", encoding="utf-8") as fh: + json.dump(obj, fh, ensure_ascii=False, indent=indent) diff --git a/benchmarks/multihop/embed.py b/benchmarks/multihop/embed.py new file mode 100644 index 0000000..051242e --- /dev/null +++ b/benchmarks/multihop/embed.py @@ -0,0 +1,54 @@ +"""Embed corpus passages and questions once, cache as .npy outside the repo. + +Files (MH_DATA_DIR): emb_<ds>__<model>__passages.npy (N x d, float32, L2-normalised) + emb_<ds>__<model>__queries.npy (Q x d, float32, L2-normalised) +Passage text embedded = "<title>. <text>" ; max_seq_length 256 for both models. +bge-small-en-v1.5 gets its documented query instruction prefix on questions only. +""" +from __future__ import annotations + +import sys +import time + +import numpy as np + +from common import DATA_DIR, MODELS, DATASETS, load_json + + +def emb_paths(ds, model): + return (DATA_DIR / f"emb_{ds}__{model}__passages.npy", DATA_DIR / f"emb_{ds}__{model}__queries.npy") + + +def load_embeddings(ds, model): + p, q = emb_paths(ds, model) + return np.load(p), np.load(q) + + +def embed(ds, model, force=False): + pp, qp = emb_paths(ds, model) + if pp.exists() and qp.exists() and not force: + print(f"[embed] cached {ds}/{model}") + return + import torch + from sentence_transformers import SentenceTransformer + hf_id, qprefix, pprefix = MODELS[model] + corpus = load_json(DATA_DIR / f"corpus_{ds}.json") + questions = load_json(DATA_DIR / f"questions_{ds}.json") + dev = "cuda" if torch.cuda.is_available() else "cpu" + m = SentenceTransformer(hf_id, device=dev) + m.max_seq_length = 256 + t0 = time.time() + ptexts = [pprefix + f"{c['title']}. {c['text']}" for c in corpus] + P = m.encode(ptexts, batch_size=128, normalize_embeddings=True, convert_to_numpy=True, + show_progress_bar=False).astype(np.float32) + Q = m.encode([qprefix + q["question"] for q in questions], batch_size=128, normalize_embeddings=True, + convert_to_numpy=True, show_progress_bar=False).astype(np.float32) + np.save(pp, P) + np.save(qp, Q) + print(f"[embed] {ds}/{model}: P{P.shape} Q{Q.shape} on {dev} in {time.time()-t0:.0f}s") + + +if __name__ == "__main__": + for ds in DATASETS: + for model in MODELS: + embed(ds, model) diff --git a/benchmarks/multihop/extra_pairs.py b/benchmarks/multihop/extra_pairs.py new file mode 100644 index 0000000..702c0fd --- /dev/null +++ b/benchmarks/multihop/extra_pairs.py @@ -0,0 +1,29 @@ +"""Paired comparisons of the dev-selected ISONGraph multi_hop against the non-graph baselines +(hybrid RRF, cross-encoder rerank) and against plain vector, per k. -> results/pairs_vs_nongraph.md/json""" +import json +import numpy as np +import report +from common import KS, RESULTS_DIR + +data = report.load_all() +rows, js = [], {} +rows.append("| Dataset/encoder | k | graph (dev-selected) | multi_hop - RRF (pp) [95% CI] (p) | multi_hop - cross-encoder rerank (pp) [95% CI] (p) | multi_hop - PPR (pp) [95% CI] (p) |") +rows.append("|---|---|---|---|---|---|") +for (ds, m), e in data.items(): + st = report.Stats(len(e["test"])) + for k in KS: + v = report.best_variant(e, "lib_mh", k) + a = e["arr"][(f"lib_mh:{v}", k, "all")] + cells = [] + for other in ("rrf", "rerank"): + d = st.diff(a, e["arr"][(other, k, "all")]) + cells.append(f"{report.fmt_d(d)} (p={report.fp(d[3])})") + js[f"{ds}/{m}/k{k}/lib_mh-{other}"] = d + vp = report.best_variant(e, "ppr", k) + d = st.diff(a, e["arr"][(f"ppr:{vp}", k, "all")]) + cells.append(f"{report.fmt_d(d)} (p={report.fp(d[3])}; PPR graph={vp})") + js[f"{ds}/{m}/k{k}/lib_mh-ppr_best"] = d + rows.append(f"| {ds}/{m} | {k} | {v} | " + " | ".join(cells) + " |") +(RESULTS_DIR / "pairs_vs_nongraph.md").write_text("\n".join(rows), encoding="utf-8") +json.dump(js, open(RESULTS_DIR / "pairs_vs_nongraph.json", "w"), indent=1, default=float) +print("\n".join(rows)) diff --git a/benchmarks/multihop/graphs.py b/benchmarks/multihop/graphs.py new file mode 100644 index 0000000..b8eb8f4 --- /dev/null +++ b/benchmarks/multihop/graphs.py @@ -0,0 +1,264 @@ +"""Graph construction that uses ONLY passage text/titles (and, for kNN, the passage +embeddings). No question, no answer, no supporting-fact annotation is touched. + +Edge families + mention : directed A -> B when B's title (parenthetical stripped, >=4 chars, + case-sensitive exact token match) occurs in A's text. Titles that + occur in more than MENTION_CAP passages are dropped as hubs. + (A "hyperlink" proxy: the HF copies carry no Wikipedia link markup.) + entity : undirected A -- B when they share >=1 'entity', an entity being a + maximal run of capitalised tokens (with of/the/de/... connectors), + sentence-initial single tokens and a small stop list removed, plus + the passage's own title. Entities with document frequency outside + [2, ENTITY_CAP] are dropped (singletons carry no edge, hubs no + information). Edge weight w = n/(n+1) for n shared entities. + knn : undirected A -- B when B is among A's KNN_K nearest passages (cosine, + same encoder as retrieval) or vice-versa. Weight w = cosine. +Union graphs combine families under separate relationship types. +Constants are fixed a priori (not tuned on any question). +""" +from __future__ import annotations + +import math +import re +from collections import defaultdict + +import numpy as np + +MENTION_CAP = 30 +ENTITY_CAP = 15 +KNN_K = 5 + +_TOK = re.compile(r"\w[\w'’\-]*|[.!?]") +_WORD = re.compile(r"\w+") +_CONNECT = {"of", "the", "de", "del", "la", "le", "von", "van", "der", "den", "du", "da", "di", "bin", "al", "and", "for", "in"} +_CAP_STOP = { + "the", "a", "an", "in", "on", "at", "of", "he", "she", "it", "his", "her", "its", "they", "their", "this", "that", + "these", "those", "after", "before", "during", "as", "by", "for", "from", "with", "when", "while", "since", "although", + "however", "also", "but", "and", "or", "if", "then", "there", "who", "which", "what", "where", "one", "two", "three", + "first", "second", "following", "following", "both", "some", "many", "most", "other", "such", "since", "until", "upon", + "into", "over", "under", "between", "among", "through", "about", "against", "within", "without", "he", "we", "i", "you", + "mr", "mrs", "dr", "st", "jr", "sr", "january", "february", "march", "april", "may", "june", "july", "august", + "september", "october", "november", "december", "monday", "tuesday", "wednesday", "thursday", "friday", + "saturday", "sunday", "born", "died", "known", "called", "named", "currently", "originally", "today", "now", +} + + +def strip_title(t: str) -> str: + return re.sub(r"\s*\([^)]*\)\s*$", "", t).strip() + + +def _words(text): + return _WORD.findall(text) + + +def mention_edges(corpus): + """-> (src, dst) int arrays, directed mentioner -> mentioned.""" + title_map = defaultdict(list) + maxn = 1 + for c in corpus: + base = strip_title(c["title"]) + if len(base) < 4: + continue + toks = tuple(_words(base)) + if not toks: + continue + title_map[toks].append(c["id"]) + maxn = max(maxn, len(toks)) + maxn = min(maxn, 12) + pairs = defaultdict(set) # title_key -> set of mentioning passages + for c in corpus: + toks = _words(c["text"]) + n = len(toks) + for i in range(n): + for L in range(1, min(maxn, n - i) + 1): + key = tuple(toks[i:i + L]) + if key in title_map: + pairs[key].add(c["id"]) + src, dst = [], [] + for key, mentioners in pairs.items(): + if len(mentioners) > MENTION_CAP: + continue + for a in mentioners: + for b in title_map[key]: + if a != b: + src.append(a) + dst.append(b) + pr = sorted(set(zip(src, dst))) + return (np.array([p[0] for p in pr], dtype=np.int64), np.array([p[1] for p in pr], dtype=np.int64)) + + +def extract_entities(text: str): + toks = _TOK.findall(text) + ents = set() + i, n = 0, len(toks) + sent_start = True + while i < n: + t = toks[i] + if t in ".!?": + sent_start = True + i += 1 + continue + if t[0].isupper(): + j = i + 1 + last_cap = i + while j < n: + u = toks[j] + if u not in ".!?" and u[0].isupper(): + last_cap = j + j += 1 + elif (u.lower() in _CONNECT and j + 1 < n + and toks[j + 1] not in ".!?" and toks[j + 1][0].isupper()): + j += 1 + else: + break + span = toks[i:last_cap + 1] + initial = sent_start + while span and span[0].lower() in _CAP_STOP: + span = span[1:] + initial = False + # a lone sentence-initial capitalised word is just a capitalised word + if span and not (len(span) == 1 and initial): + ents.add(" ".join(span).lower()) + i = last_cap + 1 + sent_start = False + continue + sent_start = False + i += 1 + return ents + + +def entity_edges(corpus): + """-> (a, b, w) arrays, a<b, undirected.""" + inv = defaultdict(list) + for c in corpus: + ents = extract_entities(c["text"]) + ents.add(strip_title(c["title"]).lower()) + for e in ents: + inv[e].append(c["id"]) + shared = defaultdict(int) + for e, docs in inv.items(): + if not (2 <= len(docs) <= ENTITY_CAP): + continue + docs = sorted(set(docs)) + for x in range(len(docs)): + for y in range(x + 1, len(docs)): + shared[(docs[x], docs[y])] += 1 + items = sorted(shared.items()) + a = np.array([k[0] for k, _ in items], dtype=np.int64) + b = np.array([k[1] for k, _ in items], dtype=np.int64) + w = np.array([n / (n + 1.0) for _, n in items], dtype=np.float32) + return a, b, w + + +def knn_edges(P: np.ndarray, k: int = KNN_K, block: int = 2048): + """Symmetrised kNN graph from L2-normalised passage embeddings. -> (a, b, w), a<b.""" + n = P.shape[0] + try: + import torch + dev = "cuda" if torch.cuda.is_available() else "cpu" + Pt = torch.from_numpy(P).to(dev) + nbrs = [] + for s in range(0, n, block): + sim = Pt[s:s + block] @ Pt.T + idx = torch.arange(s, min(s + block, n), device=dev) + sim[torch.arange(len(idx), device=dev), idx] = -2.0 + top = torch.topk(sim, k, dim=1).indices.cpu().numpy() + nbrs.append(top) + nbrs = np.concatenate(nbrs, 0) + except Exception: + nbrs = np.zeros((n, k), dtype=np.int64) + for s in range(0, n, block): + sim = P[s:s + block] @ P.T + for r in range(sim.shape[0]): + sim[r, s + r] = -2.0 + nbrs[s:s + block] = np.argpartition(-sim, k, axis=1)[:, :k] + pairs = set() + for i in range(n): + for j in nbrs[i]: + j = int(j) + pairs.add((i, j) if i < j else (j, i)) + pr = sorted(pairs) + a = np.array([p[0] for p in pr], dtype=np.int64) + b = np.array([p[1] for p in pr], dtype=np.int64) + w = np.clip((P[a] * P[b]).sum(1), 0.0, 1.0).astype(np.float32) + return a, b, w + + +VARIANTS = { + # name -> families + "mention": ("mention",), + "entity": ("entity",), + "knn": ("knn",), + "mention+entity": ("mention", "entity"), + "all": ("mention", "entity", "knn"), +} + + +CONTROLS = { + # negative controls: the SAME edge lists with node labels randomly permuted (seeded), i.e. an isomorphic + # graph whose edges no longer line up with the passages' text. Any gain here is an artefact of + # expansion/re-ranking mechanics, not of graph content. + "ctrl_mention": ("mention_rand",), + "ctrl_entity": ("entity_rand",), +} +ALL_VARIANTS = {**VARIANTS, **CONTROLS} + + +def permute_edges(edges, n, seed=7): + rng = np.random.default_rng(seed) + pi = rng.permutation(n) + out = {} + s, d = edges["mention"] + out["mention_rand"] = (pi[s], pi[d]) + a, b, w = edges["entity"] + out["entity_rand"] = (pi[a], pi[b], w) + return out + + +def graph_diagnostics(fams, edges, questions, n_nodes): + """Descriptive statistics. Gold annotations are used HERE ONLY (reporting how + useful an already-built graph is); they never influence construction.""" + und = defaultdict(set) # undirected adjacency union over families in `fams` + n_edges = 0 + for f in fams: + if f.startswith("mention"): + s, d = edges[f] + n_edges += len(s) + for a, b in zip(s.tolist(), d.tolist()): + und[a].add(b) + und[b].add(a) + else: + a_, b_, _ = edges[f] + n_edges += len(a_) + for a, b in zip(a_.tolist(), b_.tolist()): + und[a].add(b) + und[b].add(a) + deg = np.array([len(und[i]) for i in range(n_nodes)]) + linked1 = linked2 = tot = 0 + frac_ctx = [] + for q in questions: + g = q["gold"] + ctx = set(q["context"]) + for x in range(len(g)): + for y in range(x + 1, len(g)): + tot += 1 + if g[y] in und[g[x]]: + linked1 += 1 + elif und[g[x]] & und[g[y]]: + linked2 += 1 + nb = set() + for x in g: + nb |= und[x] + nb -= set(g) + if nb: + frac_ctx.append(len(nb & ctx) / len(nb)) + return { + "n_edges_undirected_union": int(sum(len(v) for v in und.values()) // 2), + "n_edges_by_family_directed_or_undirected": int(n_edges), + "mean_degree": float(deg.mean()), "median_degree": float(np.median(deg)), "max_degree": int(deg.max()), + "isolated_frac": float((deg == 0).mean()), + "gold_pairs_directly_linked": linked1 / max(tot, 1), + "gold_pairs_linked_within_2": (linked1 + linked2) / max(tot, 1), + "neighbours_of_gold_that_are_in_same_question_context": float(np.mean(frac_ctx)) if frac_ctx else None, + } diff --git a/benchmarks/multihop/headline.py b/benchmarks/multihop/headline.py new file mode 100644 index 0000000..0dfc31f --- /dev/null +++ b/benchmarks/multihop/headline.py @@ -0,0 +1,58 @@ +"""Compact README tables built from the computed summary (called by report.py).""" + + +def _p(p): + return "p<0.001" if p < 0.001 else "p=" + format(p, ".3f") + + +def headline_md(out, DATASETS, MODELS, ALLV, fmt_ci, fmt_d): + L = [] + order = [("vector", "Vector top-k (plain)"), ("bm25", "BM25"), ("rrf", "Hybrid RRF (vector+BM25)"), + ("rerank", "Vector top-30 + cross-encoder"), + ("lib_default@best", "ISONGraph multi_hop, library defaults"), ("lib_mh@best", "ISONGraph multi_hop, tuned"), + ("ppr@best", "PPR re-rank, tuned (independent impl.)"), + ("lib_mh:ctrl_mention", "CONTROL: multi_hop on label-shuffled mention graph"), + ("lib_mh:ctrl_entity", "CONTROL: multi_hop on label-shuffled entity graph")] + cols = [(ds, m) for ds in DATASETS for m in MODELS if m in out["results"].get(ds, {})] + for k in (10, 5, 20): + L.append("") + L.append(f"### Headline, k = {k}: all-supporting@{k} % [95% CI] (Delta vs plain vector in pp, paired p)") + L.append("") + L.append("| Method | " + " | ".join(f"{ds}/{m}" for ds, m in cols) + " |") + L.append("|---|" + "---|" * len(cols)) + for key, label in order: + cells = [] + for ds, m in cols: + rows = {r["key"]: r for r in out["results"][ds][m].get("tables", {}).get(str(k), [])} + r = rows.get(key) + if r is None: + cells.append("n/a") + continue + c = fmt_ci(r["all"]) + if "d_all" in r: + d = r["d_all"] + c += f" ({100 * d[0]:+.1f}, {_p(d[3])})" + if "@best" in key: + c += f" [{r['variant']}]" + cells.append(c) + L.append(f"| {label} | " + " | ".join(cells) + " |") + L.append("") + L.append("### Every graph variant, k = 10: Delta all-supporting@10 vs plain vector, pp [95% CI] (paired p)") + L.append("") + L.append("| Method: graph | " + " | ".join(f"{ds}/{m}" for ds, m in cols) + " |") + L.append("|---|" + "---|" * len(cols)) + for meth, mlabel in (("lib_mh", "ISONGraph multi_hop (tuned)"), ("ppr", "PPR re-rank (tuned)"), + ("lib_default", "ISONGraph multi_hop (defaults)")): + for v in ALLV: + cells, seen = [], False + for ds, m in cols: + rows = {r["key"]: r for r in out["results"][ds][m].get("tables", {}).get("10", [])} + r = rows.get(f"{meth}:{v}") + if r is None: + cells.append("n/a") + else: + seen = True + cells.append(f"{fmt_d(r['d_all'])} ({_p(r['d_all'][3])})") + if seen: + L.append(f"| {mlabel}: {v} | " + " | ".join(cells) + " |") + return "\n".join(L) diff --git a/benchmarks/multihop/jobs.py b/benchmarks/multihop/jobs.py new file mode 100644 index 0000000..ca08381 --- /dev/null +++ b/benchmarks/multihop/jobs.py @@ -0,0 +1,241 @@ +"""Experiment jobs. One job = (dataset, model[, graph variant]). Each job writes +results/raw/<name>.npz (per-question metric arrays, TEST and DEV questions both) +and results/raw/<name>.json (chosen hyper-parameters, full dev grid log).""" +from __future__ import annotations + +import itertools +import pickle +import random +import time + +import numpy as np + +import graphs +import methods +from common import (DATA_DIR, KS, N_DEV, RERANKER, RESULTS_DIR, SEED, load_json) +from embed import load_embeddings + +RAW = RESULTS_DIR / "raw" +RAW.mkdir(exist_ok=True) +import os +N_LIB_CONFIGS = int(os.environ.get('MH_N_CONFIGS', '100')) # random-search budget for the library per graph variant +RERANK_DEPTH = 30 + + +def load_ds(ds, model): + corpus = load_json(DATA_DIR / f"corpus_{ds}.json") + qs = load_json(DATA_DIR / f"questions_{ds}.json") + P, Q = load_embeddings(ds, model) + dev = np.array([i for i, q in enumerate(qs) if q["split"] == "dev"]) + test = np.array([i for i, q in enumerate(qs) if q["split"] == "test"]) + return corpus, qs, P, Q, dev, test + + +def get_edges(ds, model, corpus, P, need_knn=True): + f = DATA_DIR / f"edges_{ds}.pkl" + if f.exists(): + edges = pickle.load(open(f, "rb")) + else: + edges = {"mention": graphs.mention_edges(corpus), "entity": graphs.entity_edges(corpus)} + pickle.dump(edges, open(f, "wb")) + edges.update(graphs.permute_edges(edges, len(corpus))) + if need_knn: + fk = DATA_DIR / f"edges_knn_{ds}_{model}.pkl" + if fk.exists(): + edges["knn"] = pickle.load(open(fk, "rb")) + else: + edges["knn"] = graphs.knn_edges(P) + pickle.dump(edges["knn"], open(fk, "wb")) + return edges + + +def _save(name, arrays, meta): + np.savez_compressed(RAW / f"{name}.npz", **arrays) + import json + with open(RAW / f"{name}.json", "w", encoding="utf-8") as fh: + json.dump(meta, fh, ensure_ascii=False, indent=1, default=float) + + +def _store(arrays, key, rankings, qs, ks=KS): + for k in ks: + m = methods.metrics_at(rankings, qs, k) + for mk, v in m.items(): + arrays[f"{key}|{k}|{mk}"] = v.astype(np.float32) + + +# --------------------------------------------------------------------------- +# baselines: vector, BM25, RRF, cross-encoder rerank +# --------------------------------------------------------------------------- +def baselines_job(ds, model): + t0 = time.time() + corpus, qs, P, Q, dev, test = load_ds(ds, model) + arrays, meta = {}, {"dataset": ds, "model": model, "corpus_size": len(corpus), "n_questions": len(qs)} + vr, S = methods.vector_rankings(Q, P, n=50) + np.save(RAW / f"{ds}__{model}__vector_top50.npy", np.array(vr, dtype=np.int32)) + _store(arrays, "vector", vr, qs) + + docs = [f"{c['title']}. {c['text']}" for c in corpus] + bm = methods.BM25(docs) + B = bm.scores([q["question"] for q in qs]) + br = list(methods.topn_from_scores(B, 50)) + np.save(RAW / f"{ds}__{model}__bm25_top50.npy", np.array(br, dtype=np.int32)) + _store(arrays, "bm25", br, qs) + rr = methods.rrf([S, B], n=50) + _store(arrays, "rrf", rr, qs) + + cand = np.array(vr)[:, :RERANK_DEPTH] + rs = methods.rerank_scores([q["question"] for q in qs], cand, corpus, RERANKER) + np.save(RAW / f"{ds}__{model}__rerank_scores.npy", rs) + rer = methods.rerank_rankings(cand, rs, n=20) + _store(arrays, "rerank", rer, qs) + + # sanity: the library's own similarity_search returns the same top-20 as the numpy scan + lib = methods.import_lib() + g = lib.SemanticGraph(name="chk", embedding_db=":memory:", auto_embed=False) + for c in corpus[:0]: + pass + for c in corpus: + g.add_node("passage", c["id"], title=c["title"]) + g.embedding_store.add_batch([(("passage", c["id"]), c["title"], P[c["id"]].tolist()) for c in corpus]) + same = 0 + n_chk = 200 + for i in test[:n_chk]: + res = g.similarity_search(Q[i].tolist(), top_k=20, threshold=-1.0) + same += int(set(r.node_ref[1] for r in res) == set(int(x) for x in vr[i][:20])) + meta["lib_similarity_search_same_top20_set"] = f"{same}/{n_chk}" + meta["seconds"] = time.time() - t0 + _save(f"{ds}__{model}__baselines", arrays, meta) + return f"baselines {ds}/{model} done in {time.time()-t0:.0f}s" + + +# --------------------------------------------------------------------------- +# graph jobs: ISONGraph Vector (multi_hop / subgraph) and PPR on one graph variant +# --------------------------------------------------------------------------- +def lib_grid(variant): + fams = graphs.ALL_VARIANTS[variant] + hops = [1, 2, 3] if variant in ("mention", "ctrl_mention") else [1, 2] + dirs = ["out", "both"] if any(f.startswith("mention") for f in fams) else ["out"] + wfs = [None] if fams[0].startswith("mention") and len(fams) == 1 else [None, "w"] + grid = [dict(hops=h, seeds=s, decay=d, blend=b, direction=di, weight_field=w) + for h, s, d, b, di, w in itertools.product(hops, [5, 10, 15, 20], [0.3, 0.5, 0.7, 0.9], + [0.0, 0.3, 0.6], dirs, wfs)] + rng = random.Random(SEED) + grid = rng.sample(grid, min(N_LIB_CONFIGS, len(grid))) + default = dict(hops=2, seeds=5, decay=0.8, blend=0.0, direction="out", weight_field=None) # library defaults + return [default] + grid + + +def ppr_grid(variant): + fams = graphs.ALL_VARIANTS[variant] + dirs = [True, False] if any(f.startswith("mention") for f in fams) else [True] + wfs = [False] if fams[0].startswith("mention") and len(fams) == 1 else [False, True] + g = [dict(alpha=a, seeds=s, beta=b, directed_mention=d, weighted=w) + for a, s, b, d, w in itertools.product([0.5, 0.7, 0.85], [5, 10, 20], [0.0, 0.3, 0.6, 0.85], dirs, wfs)] + return g[:int(os.environ.get('MH_N_PPR', '100000'))] + + +def _score(m): + return (float(m["all"].mean()), float(m["recall"].mean())) + + +def graph_job(ds, model, variant, do_subgraph=True): + t0 = time.time() + out_name = f"{os.environ.get('MH_PREFIX', '')}{ds}__{model}__{variant.replace('+', '_')}" + if (RAW / f"{out_name}.npz").exists() and not os.environ.get("MH_FORCE"): + return f"graph {ds}/{model}/{variant} cached" + corpus, qs, P, Q, dev, test = load_ds(ds, model) + edges = get_edges(ds, model, corpus, P, need_knn=("knn" in graphs.ALL_VARIANTS[variant])) + fams = graphs.ALL_VARIANTS[variant] + n = len(corpus) + arrays = {} + meta = {"dataset": ds, "model": model, "variant": variant, "families": list(fams), + "diagnostics": graphs.graph_diagnostics(fams, edges, qs, n)} + dev_qs = [qs[i] for i in dev] + test_qs = [qs[i] for i in test] + + # -------------------- PPR (independent baseline) -------------------- + S = methods.dense_scores(Q, P) + log, best = [], {} + adj_cache = {} + for cfg in ppr_grid(variant): + akey = (cfg["directed_mention"], cfg["weighted"]) + if akey not in adj_cache: + adj_cache[akey] = methods.build_adjacency(n, edges, fams, cfg["directed_mention"], cfg["weighted"]) + r = methods.ppr_rankings(S[dev], adj_cache[akey], cfg["seeds"], cfg["alpha"], cfg["beta"]) + row = {"cfg": cfg} + for k in KS: + m = methods.metrics_at(r, dev_qs, k) + row[f"all@{k}"], row[f"recall@{k}"] = _score(m) + if k not in best or (row[f"all@{k}"], row[f"recall@{k}"]) > best[k][0]: + best[k] = ((row[f"all@{k}"], row[f"recall@{k}"]), cfg) + log.append(row) + meta["ppr_dev_grid"] = log + log_ppr = log + meta["ppr_selected"] = {k: best[k][1] for k in KS} + for k in KS: + cfg = best[k][1] + adj = adj_cache[(cfg["directed_mention"], cfg["weighted"])] + r_dev = methods.ppr_rankings(S[dev], adj, cfg["seeds"], cfg["alpha"], cfg["beta"]) + r_test = methods.ppr_rankings(S[test], adj, cfg["seeds"], cfg["alpha"], cfg["beta"]) + for split, r, sq in (("dev", r_dev, dev_qs), ("test", r_test, test_qs)): + m = methods.metrics_at(r, sq, k) + for mk, v in m.items(): + arrays[f"ppr|{split}|{k}|{mk}"] = v.astype(np.float32) + + # -------------------- ISONGraph Vector -------------------- + g = methods.build_lib_graph(corpus, P, edges, fams) + grid = lib_grid(variant) + log, best, times = [], {}, [] + for ci, cfg in enumerate(grid): + tm = [] + r = methods.lib_multihop_rankings(g, Q, cfg, idx=dev, timing=tm) + row = {"cfg": cfg, "ms_per_query": 1000 * float(np.mean(tm)), "default": ci == 0} + for k in KS: + m = methods.metrics_at(r, dev_qs, k) + row[f"all@{k}"], row[f"recall@{k}"] = _score(m) + row[f"n_ret@{k}"] = float(m["n_ret"].mean()) + if k not in best or (row[f"all@{k}"], row[f"recall@{k}"]) > best[k][0]: + best[k] = ((row[f"all@{k}"], row[f"recall@{k}"]), cfg) + log.append(row) + meta["lib_dev_grid"] = log + meta["lib_selected"] = {k: best[k][1] for k in KS} + uniq = {} + for k in KS: + uniq.setdefault(tuple(sorted(best[k][1].items(), key=lambda x: x[0])), best[k][1]) + cache_test = {} + for key, cfg in list(uniq.items()) + [("default", grid[0])]: + ck = tuple(sorted(cfg.items(), key=lambda x: x[0])) + if ck not in cache_test: + cache_test[ck] = (methods.lib_multihop_rankings(g, Q, cfg, idx=dev), + methods.lib_multihop_rankings(g, Q, cfg, idx=test)) + for k in KS: + ck = tuple(sorted(best[k][1].items(), key=lambda x: x[0])) + r_dev, r_test = cache_test[ck] + for split, r, sq in (("dev", r_dev, dev_qs), ("test", r_test, test_qs)): + m = methods.metrics_at(r, sq, k) + for mk, v in m.items(): + arrays[f"lib_mh|{split}|{k}|{mk}"] = v.astype(np.float32) + ck = tuple(sorted(grid[0].items(), key=lambda x: x[0])) + r_dev, r_test = cache_test[ck] + for k in KS: + for split, r, sq in (("dev", r_dev, dev_qs), ("test", r_test, test_qs)): + m = methods.metrics_at(r, sq, k) + for mk, v in m.items(): + arrays[f"lib_default|{split}|{k}|{mk}"] = v.astype(np.float32) + + # -------------------- semantic_subgraph(connect=True) with the tuned params -------------------- + if do_subgraph: + for k in [int(x) for x in os.environ.get('MH_SUB_KS', '5,10,20').split(',')]: + cfg = best[k][1] + for ch in (2, 3): + sets = methods.lib_subgraph_sets(g, Q, cfg, k, True, ch, idx=test) + m = methods.set_metrics(sets, test_qs) + for mk, v in m.items(): + arrays[f"lib_sub_c{ch}|test|{k}|{mk}"] = v.astype(np.float32) + # per-query sets sizes also needed for size-matched vector baseline: stored as 'size' + meta["n_lib_configs"] = len(grid) + meta["n_ppr_configs"] = len(log_ppr) + meta["sub_ks"] = os.environ.get("MH_SUB_KS", "5,10,20") + meta["seconds"] = time.time() - t0 + _save(f"{os.environ.get('MH_PREFIX', '')}{ds}__{model}__{variant.replace('+', '_')}", arrays, meta) + return f"graph {ds}/{model}/{variant} done in {time.time()-t0:.0f}s" diff --git a/benchmarks/multihop/latency.py b/benchmarks/multihop/latency.py new file mode 100644 index 0000000..f0fb409 --- /dev/null +++ b/benchmarks/multihop/latency.py @@ -0,0 +1,122 @@ +"""Per-query latency, single thread, warm, query embeddings precomputed (encoding +cost is identical for every method and excluded). Run on an otherwise idle +machine with OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1.""" +from __future__ import annotations + +import json +import platform +import time + +import numpy as np + +import graphs +import methods +import report +from common import DATA_DIR, RERANKER, RESULTS_DIR, load_json +from jobs import get_edges, load_ds + +N_Q = 200 +N_Q_RERANK = 8 + + +def _stat(ts): + a = np.array(ts) * 1000.0 + return {"mean_ms": float(a.mean()), "median_ms": float(np.median(a)), "p95_ms": float(np.percentile(a, 95)), "n": len(a)} + + +def timeit(fn, idx, warm=10): + for i in idx[:warm]: + fn(i) + ts = [] + for i in idx: + t = time.perf_counter() + fn(i) + ts.append(time.perf_counter() - t) + return _stat(ts) + + +def run(combos=(("hotpotqa", "minilm"), ("2wiki", "minilm"), ("musique", "minilm"), ("hotpotqa", "bge"))): + import torch + torch.set_num_threads(1) + data = report.load_all() + out = {"machine": {"platform": platform.platform(), "processor": platform.processor(), "threads": 1, + "note": "single-threaded, warm, query vectors precomputed; timed on a machine that may have " + "had other light background processes (see README)"}, "results": {}} + k = 10 + for ds, model in combos: + corpus, qs, P, Q, dev, test = load_ds(ds, model) + edges = get_edges(ds, model, corpus, P) + e = data[(ds, model)] + idx = list(test[:N_Q]) + res = {} + # vector (numpy brute force) + def vec(i): + s = P @ Q[i] + t = np.argpartition(-s, 20)[:20] + return t[np.argsort(-s[t])] + res["vector (numpy brute force top-20)"] = timeit(vec, idx) + # library similarity_search + lib = methods.import_lib() + g0 = lib.SemanticGraph(name="x", embedding_db=":memory:", auto_embed=False) + for c in corpus: + g0.add_node("passage", c["id"], title=c["title"]) + g0.embedding_store.add_batch([(("passage", c["id"]), c["title"], P[c["id"]].tolist()) for c in corpus]) + res["ISONGraph similarity_search top-20 (plain vector via the library)"] = timeit( + lambda i: g0.similarity_search(Q[i].tolist(), top_k=20, threshold=-1.0), idx) + # BM25 + docs = [f"{c['title']}. {c['text']}" for c in corpus] + bm = methods.BM25(docs) + qtxt = [q["question"] for q in qs] + def bm25(i): + s = bm.scores([qtxt[i]]) + return methods.topn_from_scores(s, 20) + res["BM25 (scipy sparse)"] = timeit(bm25, idx) + def hyb(i): + s1 = P @ Q[i] + s2 = bm.scores([qtxt[i]]) + return methods.rrf([s1[None, :], s2], n=20) + res["Hybrid RRF (dense + BM25 + fusion)"] = timeit(hyb, idx) + # PPR + ISONGraph with the dev-selected graph at k=10 + for meth in ("ppr", "lib_mh"): + v = report.best_variant(e, meth, k) + fams = graphs.VARIANTS[v] + sel = e["meta"][v]["ppr_selected" if meth == "ppr" else "lib_selected"][str(k)] + if meth == "ppr": + A = methods.build_adjacency(len(corpus), edges, fams, sel["directed_mention"], sel["weighted"]) + def ppr(i): + s = (P @ Q[i])[None, :] + return methods.ppr_rankings(s, A, sel["seeds"], sel["alpha"], sel["beta"], n=20) + res[f"PPR re-rank, graph={v}"] = timeit(ppr, idx) + else: + t0 = time.perf_counter() + g = methods.build_lib_graph(corpus, P, edges, fams) + res["_graph_build_seconds"] = {v: time.perf_counter() - t0} + tm = [] + methods.lib_multihop_rankings(g, Q, sel, idx=idx[:10]) + methods.lib_multihop_rankings(g, Q, sel, idx=idx, timing=tm) + res[f"ISONGraph semantic_multi_hop (tuned), graph={v}"] = _stat(tm) + tm = [] + methods.lib_subgraph_sets(g, Q, sel, k, True, 2, idx=idx[:5]) + methods.lib_subgraph_sets(g, Q, sel, k, True, 2, idx=idx[:100], timing=tm) + res[f"ISONGraph semantic_subgraph(connect=True), graph={v}"] = _stat(tm) + res["_selected_cfg"] = {"variant": v, "cfg": sel} + # cross-encoder rerank on CPU (1 thread), 30 candidates (cost depends only on the model and + # passage length, so measured once, on the first combination) + if (ds, model) != combos[0]: + out["results"][f"{ds}/{model}"] = res + print(ds, model, json.dumps({k_: (v_["mean_ms"] if isinstance(v_, dict) and "mean_ms" in v_ else None) + for k_, v_ in res.items()}, indent=1), flush=True) + continue + from sentence_transformers import CrossEncoder + ce = CrossEncoder(RERANKER, device="cpu", max_length=256) + vr = np.load(RESULTS_DIR / "raw" / f"{ds}__{model}__vector_top50.npy") + def rr(i): + cand = vr[i][:30] + pairs = [(qtxt[i], f"{corpus[int(p)]['title']}. {corpus[int(p)]['text']}") for p in cand] + return ce.predict(pairs, batch_size=32, show_progress_bar=False) + res["Vector top-30 + cross-encoder (CPU, 1 thread)"] = timeit(rr, idx[:N_Q_RERANK], warm=2) + out["results"][f"{ds}/{model}"] = res + print(ds, model, json.dumps({k_: (v_["mean_ms"] if isinstance(v_, dict) and "mean_ms" in v_ else None) + for k_, v_ in res.items()}, indent=1), flush=True) + with open(RESULTS_DIR / "latency.json", "w", encoding="utf-8") as fh: + json.dump(out, fh, indent=1) diff --git a/benchmarks/multihop/manifests/2wiki_sample.json b/benchmarks/multihop/manifests/2wiki_sample.json new file mode 100644 index 0000000..582d626 --- /dev/null +++ b/benchmarks/multihop/manifests/2wiki_sample.json @@ -0,0 +1 @@ +{"dataset": "2wiki", "source": {"repo": "framolfese/2WikiMultihopQA", "revision": "fe713bfbd1afbca1a65246741a75890405d56a3a", "file": "data/validation-00000-of-00001.parquet"}, "seed": 13, "n_questions": 1500, "n_dev": 500, "n_test": 1000, "corpus_size": 9239, "n_available": 12576, "ids": [["53e1d0ec0bb011ebab90acde48001122", "test"], ["5d45350a0bda11eba7f7acde48001122", "test"], ["e332f484092211ebbdaeac1f6bf848b6", "test"], ["e2c3977808ae11ebbd83ac1f6bf848b6", "test"], ["3be2048e0bde11eba7f7acde48001122", "test"], ["d8390b36093e11ebbdaeac1f6bf848b6", "test"], ["4a3fda120bdc11eba7f7acde48001122", "test"], ["dcfe2a500bd911eba7f7acde48001122", "test"], ["2f96d5c40bde11eba7f7acde48001122", "test"], ["48c623840bdd11eba7f7acde48001122", "test"], ["d3c6d69e085511ebbd59ac1f6bf848b6", "test"], ["f4a52e17085e11ebbd5cac1f6bf848b6", "dev"], ["3c7fbc13085311ebbd59ac1f6bf848b6", "test"], ["2a98df6408d111ebbd95ac1f6bf848b6", "dev"], ["17384fa3087411ebbd67ac1f6bf848b6", "test"], ["aca606aa0bdb11eba7f7acde48001122", "test"], ["445f3df708b511ebbd88ac1f6bf848b6", "dev"], ["f7ec568e0bd911eba7f7acde48001122", "dev"], ["5ea1f59a0bdb11eba7f7acde48001122", "dev"], ["096ef44808b011ebbd84ac1f6bf848b6", "test"], ["899815760bdc11eba7f7acde48001122", "dev"], ["2935f1640bda11eba7f7acde48001122", "dev"], ["e2cadf420bdd11eba7f7acde48001122", "dev"], ["c70a8cae08af11ebbd83ac1f6bf848b6", "test"], ["04bacc520bde11eba7f7acde48001122", "dev"], ["59603ef00bdd11eba7f7acde48001122", "dev"], ["2f74627b090c11ebbdadac1f6bf848b6", "test"], ["1bf402f708fc11ebbdadac1f6bf848b6", "test"], ["551e024408a611ebbd7fac1f6bf848b6", "test"], ["900c696108f911ebbdaaac1f6bf848b6", "test"], ["ff9aaaf708c411ebbd8eac1f6bf848b6", "dev"], ["8b47b5aa0bd911eba7f7acde48001122", "test"], ["2dba81d80bde11eba7f7acde48001122", "test"], ["5335a4e00bdb11eba7f7acde48001122", "test"], ["6f8bb027090811ebbdadac1f6bf848b6", "test"], ["4b26a6fc0bde11eba7f7acde48001122", "dev"], ["9cb70ed60bdd11eba7f7acde48001122", "test"], ["fb590ce20bdc11eba7f7acde48001122", "test"], ["b358b348084e11ebbd56ac1f6bf848b6", "test"], ["b8310aea0bdd11eba7f7acde48001122", "dev"], ["89476fee0bdd11eba7f7acde48001122", "test"], ["de6a45640bdd11eba7f7acde48001122", "test"], ["726edd860bdb11eba7f7acde48001122", "test"], ["895740ae0bb011ebab90acde48001122", "test"], ["d2d43732088611ebbd6dac1f6bf848b6", "dev"], ["6402ff7a089811ebbd77ac1f6bf848b6", "dev"], ["d99f2d79084d11ebbd56ac1f6bf848b6", "dev"], ["26497355084c11ebbd56ac1f6bf848b6", "test"], ["6ce16be80bdd11eba7f7acde48001122", "test"], ["c2fe1cb60baf11ebab90acde48001122", "test"], ["d0df520a0bdc11eba7f7acde48001122", "dev"], ["556087660baf11ebab90acde48001122", "test"], ["e5354de60bdb11eba7f7acde48001122", "test"], ["8eeb6b8f08cd11ebbd93ac1f6bf848b6", "dev"], ["b51a82dc08db11ebbd9cac1f6bf848b6", "test"], ["c7f219200bdc11eba7f7acde48001122", "dev"], ["f9302d0808e211ebbda4ac1f6bf848b6", "test"], ["2c43f3d40bde11eba7f7acde48001122", "dev"], ["8c68615008cd11ebbd93ac1f6bf848b6", "test"], ["e053179a0bdb11eba7f7acde48001122", "dev"], ["f33eda240bae11ebab90acde48001122", "test"], ["8d4ce1380baf11ebab90acde48001122", "test"], ["adf59bec0bdb11eba7f7acde48001122", "test"], ["38ad21b40bdd11eba7f7acde48001122", "test"], ["5e02b99208b811ebbd88ac1f6bf848b6", "test"], ["416fdbde084a11ebbd56ac1f6bf848b6", "test"], ["39c1e0de0bde11eba7f7acde48001122", "dev"], ["a9e43ad00baf11ebab90acde48001122", "test"], ["6fd4b19e086211ebbd5dac1f6bf848b6", "test"], ["51fec64e08b111ebbd85ac1f6bf848b6", "test"], ["74cb18460baf11ebab90acde48001122", "test"], ["9291f6300bda11eba7f7acde48001122", "test"], ["54ec0e7e0bda11eba7f7acde48001122", "test"], ["c6243b900bdd11eba7f7acde48001122", "dev"], ["5a9ca2660bda11eba7f7acde48001122", "dev"], ["7da6577c0bb011ebab90acde48001122", "dev"], ["2ad0448f088811ebbd6eac1f6bf848b6", "dev"], ["bd058dbd08fd11ebbdadac1f6bf848b6", "dev"], ["9e1457800baf11ebab90acde48001122", "test"], ["b393c8180bda11eba7f7acde48001122", "dev"], ["4b8a03ce086811ebbd5eac1f6bf848b6", "test"], ["b8a8f33a08a311ebbd7cac1f6bf848b6", "test"], ["4baf9bba0bde11eba7f7acde48001122", "dev"], ["3ed396b0087f11ebbd6bac1f6bf848b6", "dev"], ["e902aff708a511ebbd7dac1f6bf848b6", "dev"], ["7368acd60bda11eba7f7acde48001122", "dev"], ["2afaf71c0bdd11eba7f7acde48001122", "test"], ["17f38e8208cb11ebbd92ac1f6bf848b6", "test"], ["897058c40baf11ebab90acde48001122", "dev"], ["d7ddc436087311ebbd67ac1f6bf848b6", "test"], ["cf06000e094611ebbdaeac1f6bf848b6", "test"], ["940c5a420bd911eba7f7acde48001122", "test"], ["7b1ce2b6086111ebbd5dac1f6bf848b6", "test"], ["00c75ffc0bde11eba7f7acde48001122", "dev"], ["f0216f4a0baf11ebab90acde48001122", "dev"], ["87a707740bda11eba7f7acde48001122", "test"], ["0dc3d0280bde11eba7f7acde48001122", "test"], ["48be5d560bde11eba7f7acde48001122", "test"], ["2e4ac7e80bde11eba7f7acde48001122", "test"], ["a16158840bdc11eba7f7acde48001122", "test"], ["e7b1d7340bda11eba7f7acde48001122", "test"], ["8dcdb1040bdd11eba7f7acde48001122", "test"], ["d2f969d40bd911eba7f7acde48001122", "test"], ["523dd3220bdc11eba7f7acde48001122", "test"], ["33e0c5680bde11eba7f7acde48001122", "test"], ["f480bf400bdd11eba7f7acde48001122", "test"], ["83cf4607089a11ebbd77ac1f6bf848b6", "test"], ["f57e63780bda11eba7f7acde48001122", "test"], ["5136ceac08f011ebbda8ac1f6bf848b6", "dev"], ["fcff07fc084a11ebbd56ac1f6bf848b6", "test"], ["6e31773a0bd911eba7f7acde48001122", "test"], ["41b295500bb011ebab90acde48001122", "test"], ["ea897033084e11ebbd56ac1f6bf848b6", "test"], ["cfad61500bdd11eba7f7acde48001122", "dev"], ["75fd9bea0bdb11eba7f7acde48001122", "test"], ["a9411340084a11ebbd56ac1f6bf848b6", "dev"], ["ef22b9d208b211ebbd85ac1f6bf848b6", "dev"], ["da208aa4084911ebbd55ac1f6bf848b6", "dev"], ["2d02a860088c11ebbd6fac1f6bf848b6", "dev"], ["e9e79ad008d211ebbd96ac1f6bf848b6", "dev"], ["c581e00c08ae11ebbd83ac1f6bf848b6", "dev"], ["4c28e77a0bdb11eba7f7acde48001122", "dev"], ["f1416dc00bdd11eba7f7acde48001122", "dev"], ["a0a37ebf085211ebbd58ac1f6bf848b6", "test"], ["43c31e26089911ebbd77ac1f6bf848b6", "test"], ["ca7930c4088611ebbd6dac1f6bf848b6", "dev"], ["0d836ca60baf11ebab90acde48001122", "test"], ["be3da63008ce11ebbd94ac1f6bf848b6", "dev"], ["e932be4c09a611ebbdb0ac1f6bf848b6", "dev"], ["c1f7e2a60bdd11eba7f7acde48001122", "test"], ["0c5cf4a80bde11eba7f7acde48001122", "test"], ["70ef2e500bdd11eba7f7acde48001122", "dev"], ["bf122d2c0bdc11eba7f7acde48001122", "test"], ["f2c9d9260baf11ebab90acde48001122", "test"], ["ba3933c8085c11ebbd5cac1f6bf848b6", "test"], ["6b850e0e0baf11ebab90acde48001122", "dev"], ["3ac46dd20baf11ebab90acde48001122", "dev"], ["5eb40fa40baf11ebab90acde48001122", "test"], ["3e1f8f82084b11ebbd56ac1f6bf848b6", "test"], ["3580f2820bdc11eba7f7acde48001122", "dev"], ["fff47f36086f11ebbd62ac1f6bf848b6", "test"], ["b5790c240bda11eba7f7acde48001122", "test"], ["24ba4db60bde11eba7f7acde48001122", "dev"], ["48da05b6084f11ebbd56ac1f6bf848b6", "dev"], ["3628f5ff094d11ebbdaeac1f6bf848b6", "dev"], ["16e40ff008b511ebbd88ac1f6bf848b6", "test"], ["8919fb50085311ebbd59ac1f6bf848b6", "test"], ["d4c2c0880bdb11eba7f7acde48001122", "test"], ["c31b2d920bdc11eba7f7acde48001122", "dev"], ["b5bb630a0bdd11eba7f7acde48001122", "dev"], ["b0e112940bdd11eba7f7acde48001122", "dev"], ["e69a4b4408d611ebbd99ac1f6bf848b6", "test"], ["0c7cb3660bdd11eba7f7acde48001122", "test"], ["8c1162a5086111ebbd5dac1f6bf848b6", "dev"], ["6c0e5916089111ebbd73ac1f6bf848b6", "test"], ["d5732dd508c911ebbd92ac1f6bf848b6", "test"], ["1e71fb080bdd11eba7f7acde48001122", "test"], ["ab1d47a108d211ebbd96ac1f6bf848b6", "test"], ["00f34c7e088311ebbd6dac1f6bf848b6", "test"], ["29427eea0bdd11eba7f7acde48001122", "test"], ["63033a79087911ebbd67ac1f6bf848b6", "test"], ["27cf588e085011ebbd56ac1f6bf848b6", "test"], ["81f6dbba098111ebbdb0ac1f6bf848b6", "dev"], ["60b2aa720baf11ebab90acde48001122", "test"], ["5760d86408a711ebbd7fac1f6bf848b6", "test"], ["b8960ed408b011ebbd85ac1f6bf848b6", "test"], ["c3b8a8a4086b11ebbd60ac1f6bf848b6", "test"], ["e7efe4dc089411ebbd75ac1f6bf848b6", "test"], ["18eaf3aa08b411ebbd87ac1f6bf848b6", "test"], ["962889100baf11ebab90acde48001122", "dev"], ["a9ab265b087b11ebbd68ac1f6bf848b6", "test"], ["73755d740bd911eba7f7acde48001122", "test"], ["0ba80d960bdd11eba7f7acde48001122", "test"], ["01a2e4dc0bde11eba7f7acde48001122", "test"], ["65ab3b8708ab11ebbd82ac1f6bf848b6", "test"], ["6a4840d608dc11ebbd9cac1f6bf848b6", "test"], ["326202060bde11eba7f7acde48001122", "dev"], ["a0a792ae08d111ebbd95ac1f6bf848b6", "dev"], ["b6be6890084e11ebbd56ac1f6bf848b6", "test"], ["acf41d660bdd11eba7f7acde48001122", "test"], ["a83d0a42087e11ebbd6bac1f6bf848b6", "test"], ["97fdcafc08ad11ebbd83ac1f6bf848b6", "test"], ["4813c0960baf11ebab90acde48001122", "test"], ["ddf04e3f08af11ebbd83ac1f6bf848b6", "test"], ["42d8bc420bde11eba7f7acde48001122", "dev"], ["5f2bb1060bd911eba7f7acde48001122", "test"], ["09e7fffe08b311ebbd86ac1f6bf848b6", "test"], ["767efc000bde11eba7f7acde48001122", "test"], ["a283d9e208be11ebbd8aac1f6bf848b6", "test"], ["ee6ab6980bdc11eba7f7acde48001122", "test"], ["32efd6180bb011ebab90acde48001122", "dev"], ["f642ff480bae11ebab90acde48001122", "test"], ["2f6c336a0bb011ebab90acde48001122", "dev"], ["53b662800bde11eba7f7acde48001122", "test"], ["87d0e9a00bd911eba7f7acde48001122", "test"], ["fc9d859008a011ebbd78ac1f6bf848b6", "test"], ["450019aa08b111ebbd85ac1f6bf848b6", "test"], ["53d6ad2a0bb011ebab90acde48001122", "dev"], ["87b30d140bb011ebab90acde48001122", "dev"], ["c091b7620baf11ebab90acde48001122", "test"], ["4bb0b62e0bdb11eba7f7acde48001122", "dev"], ["6f73a8a20bde11eba7f7acde48001122", "test"], ["4ed186f00bde11eba7f7acde48001122", "dev"], ["c21f7548085c11ebbd5cac1f6bf848b6", "test"], ["5cadf57a0bb011ebab90acde48001122", "test"], ["88cc034008c211ebbd8cac1f6bf848b6", "dev"], ["421514d5088411ebbd6dac1f6bf848b6", "dev"], ["d3afbe600bdd11eba7f7acde48001122", "test"], ["6e4854a00bde11eba7f7acde48001122", "test"], ["87d1e62a08c511ebbd8eac1f6bf848b6", "test"], ["0c4974760bdb11eba7f7acde48001122", "test"], ["653ca2e20bdb11eba7f7acde48001122", "dev"], ["7e671272097711ebbdb0ac1f6bf848b6", "dev"], ["92f031d708c511ebbd8eac1f6bf848b6", "test"], ["1c7876600bb011ebab90acde48001122", "test"], ["16cbb9f40bdb11eba7f7acde48001122", "test"], ["0e967bac08a711ebbd7fac1f6bf848b6", "test"], ["c4881a4c08de11ebbd9fac1f6bf848b6", "test"], ["01547cde0bde11eba7f7acde48001122", "test"], ["d8c252d5088611ebbd6dac1f6bf848b6", "dev"], ["61d018d8086011ebbd5dac1f6bf848b6", "test"], ["3d2481920bb011ebab90acde48001122", "test"], ["8f4ad680089d11ebbd78ac1f6bf848b6", "dev"], ["9ac2603a0bdd11eba7f7acde48001122", "test"], ["b53c4ade08cc11ebbd93ac1f6bf848b6", "dev"], ["fd848f860bdd11eba7f7acde48001122", "test"], ["69abbde408d511ebbd98ac1f6bf848b6", "dev"], ["145752ee085e11ebbd5cac1f6bf848b6", "dev"], ["5cfd6cd00bde11eba7f7acde48001122", "dev"], ["ab9e91140bdb11eba7f7acde48001122", "test"], ["8a85285308c611ebbd90ac1f6bf848b6", "test"], ["fd19f801086411ebbd5eac1f6bf848b6", "test"], ["3011c56a0bdc11eba7f7acde48001122", "test"], ["11335e4808a711ebbd7fac1f6bf848b6", "test"], ["249f509c08e611ebbda4ac1f6bf848b6", "dev"], ["740e162f08cb11ebbd92ac1f6bf848b6", "test"], ["32d6c99a086b11ebbd5fac1f6bf848b6", "dev"], ["4648da32088811ebbd6eac1f6bf848b6", "dev"], ["6246a517084c11ebbd56ac1f6bf848b6", "test"], ["611ddbf80bdc11eba7f7acde48001122", "dev"], ["c573992a089f11ebbd78ac1f6bf848b6", "dev"], ["7abe5a5408cd11ebbd93ac1f6bf848b6", "dev"], ["5d4ed25e0bda11eba7f7acde48001122", "test"], ["0a795a4008a111ebbd78ac1f6bf848b6", "test"], ["6d51cfcc0bde11eba7f7acde48001122", "test"], ["29b6c2280bdd11eba7f7acde48001122", "dev"], ["20a713da0bde11eba7f7acde48001122", "test"], ["7161d1980bde11eba7f7acde48001122", "dev"], ["fdc0221c0bdd11eba7f7acde48001122", "test"], ["8cec5646086f11ebbd62ac1f6bf848b6", "test"], ["bfea318408e311ebbda4ac1f6bf848b6", "dev"], ["bf38e404087511ebbd67ac1f6bf848b6", "test"], ["8a6ca5ce0bb011ebab90acde48001122", "test"], ["0dd13aed084b11ebbd56ac1f6bf848b6", "test"], ["6e5de2c908b611ebbd88ac1f6bf848b6", "test"], ["ad9f226a0bdd11eba7f7acde48001122", "dev"], ["5ed727980bda11eba7f7acde48001122", "dev"], ["2a1d94ce08b511ebbd88ac1f6bf848b6", "test"], ["0fd5f2ec0bde11eba7f7acde48001122", "test"], ["66c2c79c0bde11eba7f7acde48001122", "dev"], ["36157ef00bde11eba7f7acde48001122", "dev"], ["b151e4ec0bdd11eba7f7acde48001122", "test"], ["f68691d208bf11ebbd8aac1f6bf848b6", "dev"], ["28d052040bdb11eba7f7acde48001122", "test"], ["01758bfb087c11ebbd69ac1f6bf848b6", "test"], ["505170320bdc11eba7f7acde48001122", "test"], ["e1cfab340bda11eba7f7acde48001122", "test"], ["3084cf1e0bb011ebab90acde48001122", "test"], ["1d5abddf088811ebbd6eac1f6bf848b6", "dev"], ["99edb50e093411ebbdaeac1f6bf848b6", "dev"], ["74eb875e086a11ebbd5fac1f6bf848b6", "dev"], ["1729b4a20bde11eba7f7acde48001122", "dev"], ["6c8013de0bb011ebab90acde48001122", "test"], ["55c03a3a096111ebbdafac1f6bf848b6", "dev"], ["443c63f108a911ebbd80ac1f6bf848b6", "dev"], ["0f06923008d311ebbd96ac1f6bf848b6", "test"], ["efda356c093411ebbdaeac1f6bf848b6", "test"], ["4c405d3608c411ebbd8eac1f6bf848b6", "test"], ["dd17999f08d811ebbd9cac1f6bf848b6", "dev"], ["4c87d6be087111ebbd63ac1f6bf848b6", "test"], ["a91562e00bdd11eba7f7acde48001122", "test"], ["8c76a4920bdc11eba7f7acde48001122", "dev"], ["3effaaac08f711ebbdaaac1f6bf848b6", "test"], ["bb93fea608e811ebbda6ac1f6bf848b6", "test"], ["7c643c21088211ebbd6dac1f6bf848b6", "test"], ["cc3587140bdd11eba7f7acde48001122", "dev"], ["11ce77860bde11eba7f7acde48001122", "dev"], ["b22b87b40bd911eba7f7acde48001122", "test"], ["10f096f00bde11eba7f7acde48001122", "test"], ["259e9c10087711ebbd67ac1f6bf848b6", "test"], ["c8c9d8d80baf11ebab90acde48001122", "dev"], ["2d1c1cd008c511ebbd8eac1f6bf848b6", "test"], ["ad55c0b2085d11ebbd5cac1f6bf848b6", "test"], ["74c24c8c0bde11eba7f7acde48001122", "test"], ["ad4373200bdd11eba7f7acde48001122", "test"], ["7df493300bdb11eba7f7acde48001122", "test"], ["605287bc0bde11eba7f7acde48001122", "test"], ["3039f2ae0bde11eba7f7acde48001122", "test"], ["b164cca208e911ebbda7ac1f6bf848b6", "test"], ["5fd2272b08f711ebbdaaac1f6bf848b6", "dev"], ["5495b6560bde11eba7f7acde48001122", "test"], ["1c7eb2640bb011ebab90acde48001122", "test"], ["7529522e0bde11eba7f7acde48001122", "test"], ["10f93d880bdd11eba7f7acde48001122", "test"], ["0988901808b211ebbd85ac1f6bf848b6", "test"], ["361f73d80bde11eba7f7acde48001122", "test"], ["f29525d60bdd11eba7f7acde48001122", "test"], ["0518615a0bde11eba7f7acde48001122", "test"], ["22dec2c40bde11eba7f7acde48001122", "test"], ["aa5a8a01092f11ebbdaeac1f6bf848b6", "test"], ["ef0b2f6e0bdd11eba7f7acde48001122", "dev"], ["57f2630e08ae11ebbd83ac1f6bf848b6", "dev"], ["1db3dc120bde11eba7f7acde48001122", "test"], ["69d3571508ff11ebbdadac1f6bf848b6", "dev"], ["ce81b9a708af11ebbd83ac1f6bf848b6", "test"], ["87a715180bb011ebab90acde48001122", "dev"], ["3413a86c0bb011ebab90acde48001122", "test"], ["ccd086f80bdb11eba7f7acde48001122", "test"], ["df99f4120bdc11eba7f7acde48001122", "test"], ["eeef32180bd911eba7f7acde48001122", "test"], ["e8d37e800bdd11eba7f7acde48001122", "test"], ["5d3c575a0bda11eba7f7acde48001122", "dev"], ["0ce627320bde11eba7f7acde48001122", "test"], ["23a101880bdc11eba7f7acde48001122", "dev"], ["1f288ed00bde11eba7f7acde48001122", "test"], ["626c9fc7088011ebbd6cac1f6bf848b6", "test"], ["812e2ca9090d11ebbdadac1f6bf848b6", "dev"], ["7a8eae7d088111ebbd6dac1f6bf848b6", "test"], ["96a6f01a0bdd11eba7f7acde48001122", "test"], ["7b3926180bdd11eba7f7acde48001122", "test"], ["570bf6f40bdc11eba7f7acde48001122", "test"], ["30ee74680bde11eba7f7acde48001122", "dev"], ["3079ddde0bb011ebab90acde48001122", "test"], ["8fa2b4b908b611ebbd88ac1f6bf848b6", "test"], ["608c01f00bdd11eba7f7acde48001122", "test"], ["01245b60088411ebbd6dac1f6bf848b6", "dev"], ["ab13f1ec09af11ebbdb0ac1f6bf848b6", "test"], ["094dfa56086311ebbd5eac1f6bf848b6", "dev"], ["acd6da260bdd11eba7f7acde48001122", "dev"], ["fffc9979086711ebbd5eac1f6bf848b6", "test"], ["597a6ac20bd911eba7f7acde48001122", "dev"], ["6956b8a60bde11eba7f7acde48001122", "dev"], ["492c06120bde11eba7f7acde48001122", "test"], ["4ff4ea2a087511ebbd67ac1f6bf848b6", "test"], ["7603449c08e711ebbda6ac1f6bf848b6", "test"], ["84c9e4880bb011ebab90acde48001122", "test"], ["32e50df00bb011ebab90acde48001122", "dev"], ["9bd0c7e60bdd11eba7f7acde48001122", "dev"], ["8272c3eb093111ebbdaeac1f6bf848b6", "test"], ["54b951ed096f11ebbdb0ac1f6bf848b6", "test"], ["fbe7c55e085811ebbd5bac1f6bf848b6", "test"], ["931f84c608f311ebbdaaac1f6bf848b6", "test"], ["bd54609908d711ebbd9aac1f6bf848b6", "test"], ["ae331827088b11ebbd6fac1f6bf848b6", "test"], ["93d2a92a0baf11ebab90acde48001122", "test"], ["3e87136c0bdb11eba7f7acde48001122", "test"], ["76526054085b11ebbd5cac1f6bf848b6", "test"], ["ee5ce46a087a11ebbd67ac1f6bf848b6", "test"], ["98da1bb608d411ebbd96ac1f6bf848b6", "test"], ["3d328861086711ebbd5eac1f6bf848b6", "dev"], ["0941e98b08b511ebbd88ac1f6bf848b6", "dev"], ["bb74f632086b11ebbd60ac1f6bf848b6", "test"], ["0ad93d5a086411ebbd5eac1f6bf848b6", "test"], ["9b1bbca80bdc11eba7f7acde48001122", "test"], ["a26599ea097011ebbdb0ac1f6bf848b6", "test"], ["3995f87a08c811ebbd91ac1f6bf848b6", "test"], ["28aacbc5087011ebbd62ac1f6bf848b6", "test"], ["2185d4500bdc11eba7f7acde48001122", "test"], ["019bed300bde11eba7f7acde48001122", "dev"], ["c107508209b911ebbdb0ac1f6bf848b6", "dev"], ["2b94cd280bde11eba7f7acde48001122", "dev"], ["62153ccf097211ebbdb0ac1f6bf848b6", "test"], ["1a5823a1088e11ebbd70ac1f6bf848b6", "dev"], ["a0c0b0720bdd11eba7f7acde48001122", "dev"], ["baa08dbf085411ebbd59ac1f6bf848b6", "test"], ["a98eb1e4087311ebbd66ac1f6bf848b6", "test"], ["f1d813c2087e11ebbd6bac1f6bf848b6", "test"], ["12372ebd08e711ebbda5ac1f6bf848b6", "dev"], ["44953da6087a11ebbd67ac1f6bf848b6", "dev"], ["fd316388090f11ebbdadac1f6bf848b6", "dev"], ["d6f86e420bdc11eba7f7acde48001122", "test"], ["e56744640bda11eba7f7acde48001122", "dev"], ["88ee60c00bdd11eba7f7acde48001122", "test"], ["089b1e3a0bde11eba7f7acde48001122", "dev"], ["acb68e0e089211ebbd73ac1f6bf848b6", "dev"], ["074763cc0bde11eba7f7acde48001122", "dev"], ["e1abbba9098a11ebbdb0ac1f6bf848b6", "dev"], ["2eaadcce0bb011ebab90acde48001122", "dev"], ["5ff45ef00baf11ebab90acde48001122", "test"], ["cbc9d19f094911ebbdaeac1f6bf848b6", "test"], ["31df8b0a0bde11eba7f7acde48001122", "dev"], ["d76a8d8d089511ebbd75ac1f6bf848b6", "dev"], ["119454120bb011ebab90acde48001122", "test"], ["f41926f9084c11ebbd56ac1f6bf848b6", "test"], ["a2503fd5089f11ebbd78ac1f6bf848b6", "test"], ["88d01fb008b911ebbd88ac1f6bf848b6", "test"], ["535ebae60bdc11eba7f7acde48001122", "test"], ["48ecfa4e0bde11eba7f7acde48001122", "test"], ["9c996f860bdb11eba7f7acde48001122", "dev"], ["071e95ae08f111ebbda8ac1f6bf848b6", "dev"], ["c6ed0ee60bdb11eba7f7acde48001122", "test"], ["1ed79b0a087411ebbd67ac1f6bf848b6", "test"], ["4eb8cb5208e411ebbda4ac1f6bf848b6", "test"], ["64f4337108a911ebbd80ac1f6bf848b6", "dev"], ["ac6e394e086311ebbd5eac1f6bf848b6", "test"], ["a37ece260baf11ebab90acde48001122", "test"], ["02e9e3dc0bb011ebab90acde48001122", "dev"], ["bab3b6d00bda11eba7f7acde48001122", "dev"], ["8f494dcc0bdd11eba7f7acde48001122", "test"], ["6b62ba5a0bde11eba7f7acde48001122", "test"], ["d89acde60bda11eba7f7acde48001122", "test"], ["47133d560bdd11eba7f7acde48001122", "dev"], ["c6f06d000bdd11eba7f7acde48001122", "dev"], ["104c34700bde11eba7f7acde48001122", "dev"], ["aa772e500bda11eba7f7acde48001122", "dev"], ["e8aad7f8089711ebbd77ac1f6bf848b6", "dev"], ["887f71380bb011ebab90acde48001122", "test"], ["fdbb770408da11ebbd9cac1f6bf848b6", "test"], ["a5ae76280bdd11eba7f7acde48001122", "dev"], ["f6b058440bdc11eba7f7acde48001122", "test"], ["042bf0e00bde11eba7f7acde48001122", "test"], ["20d3483b08c111ebbd8bac1f6bf848b6", "dev"], ["742d99bd08b811ebbd88ac1f6bf848b6", "test"], ["fada589e08cf11ebbd95ac1f6bf848b6", "test"], ["9531d8b4087a11ebbd67ac1f6bf848b6", "test"], ["6ea43168084c11ebbd56ac1f6bf848b6", "test"], ["2bb3448b086211ebbd5dac1f6bf848b6", "test"], ["e1b201da0bdd11eba7f7acde48001122", "test"], ["6c7906f2084f11ebbd56ac1f6bf848b6", "test"], ["f0951102087811ebbd67ac1f6bf848b6", "test"], ["494077aa0bde11eba7f7acde48001122", "test"], ["dd41d993086b11ebbd60ac1f6bf848b6", "test"], ["5b95df0c0bda11eba7f7acde48001122", "test"], ["c1bdec7a089c11ebbd78ac1f6bf848b6", "test"], ["1e924cea0bde11eba7f7acde48001122", "test"], ["5352c0240baf11ebab90acde48001122", "dev"], ["ede465800bdb11eba7f7acde48001122", "test"], ["2bb67943092811ebbdaeac1f6bf848b6", "dev"], ["28ad4ad0092a11ebbdaeac1f6bf848b6", "test"], ["f26ca8aa087611ebbd67ac1f6bf848b6", "test"], ["3972d2b60baf11ebab90acde48001122", "dev"], ["c387443908e811ebbda6ac1f6bf848b6", "dev"], ["21eecd840baf11ebab90acde48001122", "test"], ["12c66ab4093211ebbdaeac1f6bf848b6", "test"], ["b2094022092811ebbdaeac1f6bf848b6", "test"], ["d4052464087611ebbd67ac1f6bf848b6", "dev"], ["2e4804e8086611ebbd5eac1f6bf848b6", "dev"], ["c789d1a20bdd11eba7f7acde48001122", "test"], ["f509868308d011ebbd95ac1f6bf848b6", "test"], ["a635f66e088d11ebbd70ac1f6bf848b6", "dev"], ["085e01bc0bde11eba7f7acde48001122", "test"], ["937ac6e2089b11ebbd78ac1f6bf848b6", "test"], ["6fd8bb2f08dd11ebbd9dac1f6bf848b6", "test"], ["78045f580bdd11eba7f7acde48001122", "test"], ["bd4901b0085e11ebbd5cac1f6bf848b6", "test"], ["ba9e5b1e08e711ebbda6ac1f6bf848b6", "test"], ["a1d29c500bdd11eba7f7acde48001122", "test"], ["bacc6ba00bdd11eba7f7acde48001122", "dev"], ["a6f0c6ec089c11ebbd78ac1f6bf848b6", "test"], ["83f01c4208b511ebbd88ac1f6bf848b6", "test"], ["168d081a0bb011ebab90acde48001122", "test"], ["8da6e4200bb011ebab90acde48001122", "test"], ["ab69635e089811ebbd77ac1f6bf848b6", "test"], ["5ac0a1d6086f11ebbd62ac1f6bf848b6", "dev"], ["6ae33560087211ebbd65ac1f6bf848b6", "test"], ["77da5fc80bb011ebab90acde48001122", "dev"], ["deea8eb80bdd11eba7f7acde48001122", "dev"], ["286b03160bb011ebab90acde48001122", "test"], ["428c6db9090311ebbdadac1f6bf848b6", "dev"], ["e2a3174c088511ebbd6dac1f6bf848b6", "test"], ["67ecf3100bdc11eba7f7acde48001122", "dev"], ["763c2361087511ebbd67ac1f6bf848b6", "test"], ["bdfee2540baf11ebab90acde48001122", "test"], ["1900f7600baf11ebab90acde48001122", "dev"], ["9997225c0bda11eba7f7acde48001122", "test"], ["843ef89008fa11ebbdacac1f6bf848b6", "test"], ["6071d8cc08de11ebbd9eac1f6bf848b6", "dev"], ["0864c8b40baf11ebab90acde48001122", "test"], ["ee0650aa0bdb11eba7f7acde48001122", "dev"], ["8ec4cd9d085c11ebbd5cac1f6bf848b6", "dev"], ["7b0efff80bda11eba7f7acde48001122", "test"], ["82e4ec120bb011ebab90acde48001122", "test"], ["9b02e81c08cc11ebbd93ac1f6bf848b6", "dev"], ["b838d62f086711ebbd5eac1f6bf848b6", "test"], ["1442e9a20bde11eba7f7acde48001122", "test"], ["1a19804009c511ebbdb0ac1f6bf848b6", "test"], ["50f1350208af11ebbd83ac1f6bf848b6", "test"], ["453be8e608c411ebbd8eac1f6bf848b6", "dev"], ["0105676c0bdd11eba7f7acde48001122", "test"], ["2f394208090e11ebbdadac1f6bf848b6", "test"], ["ace717d0088411ebbd6dac1f6bf848b6", "test"], ["6743b14c0bdc11eba7f7acde48001122", "test"], ["6070241c0bb011ebab90acde48001122", "dev"], ["ac94af4a0bdb11eba7f7acde48001122", "test"], ["42eb321a0bdd11eba7f7acde48001122", "test"], ["8988d5b60bdc11eba7f7acde48001122", "dev"], ["b74ad1d208f511ebbdaaac1f6bf848b6", "dev"], ["5ea18ece0bdc11eba7f7acde48001122", "test"], ["03c63bd8085011ebbd56ac1f6bf848b6", "test"], ["fec888ea0bae11ebab90acde48001122", "dev"], ["cef85d800bda11eba7f7acde48001122", "test"], ["1d2df1ce0bde11eba7f7acde48001122", "test"], ["24e4900e087211ebbd64ac1f6bf848b6", "test"], ["d9d9e0c40bdb11eba7f7acde48001122", "dev"], ["29bfaeec0bb011ebab90acde48001122", "test"], ["164f0f7f08c711ebbd91ac1f6bf848b6", "dev"], ["b536dc7a0bdd11eba7f7acde48001122", "test"], ["7a298108087611ebbd67ac1f6bf848b6", "test"], ["ce9c3713087e11ebbd6bac1f6bf848b6", "test"], ["43d7f2180baf11ebab90acde48001122", "dev"], ["63e7a68c0bde11eba7f7acde48001122", "test"], ["33bc09c008a611ebbd7eac1f6bf848b6", "test"], ["0518d9a00bde11eba7f7acde48001122", "test"], ["33ecec4e0bde11eba7f7acde48001122", "dev"], ["e7b41722089e11ebbd78ac1f6bf848b6", "test"], ["3bbc5b180bb011ebab90acde48001122", "dev"], ["0fdab496088f11ebbd70ac1f6bf848b6", "dev"], ["125411ec087411ebbd67ac1f6bf848b6", "test"], ["4a9ba16c0baf11ebab90acde48001122", "test"], ["8b8c4bda0bdd11eba7f7acde48001122", "dev"], ["b8be5e8e08ec11ebbda7ac1f6bf848b6", "dev"], ["3956e2ea0bdc11eba7f7acde48001122", "test"], ["db0b2ec2086f11ebbd62ac1f6bf848b6", "test"], ["eb9bd1cd089e11ebbd78ac1f6bf848b6", "test"], ["828b1390093511ebbdaeac1f6bf848b6", "test"], ["e89d3cc508fb11ebbdacac1f6bf848b6", "test"], ["f7f0408d087311ebbd67ac1f6bf848b6", "dev"], ["9cbb70ea097111ebbdb0ac1f6bf848b6", "test"], ["f6805e220bdd11eba7f7acde48001122", "test"], ["f86dd5180bdb11eba7f7acde48001122", "dev"], ["247c4c6c08d911ebbd9cac1f6bf848b6", "test"], ["11b996700bdb11eba7f7acde48001122", "test"], ["6da7cd5a0bd911eba7f7acde48001122", "dev"], ["cc66e2500bdd11eba7f7acde48001122", "dev"], ["10cedc9c0baf11ebab90acde48001122", "test"], ["a8bff83c0bdd11eba7f7acde48001122", "test"], ["a96fec4c0bdd11eba7f7acde48001122", "test"], ["37f7ad7e0bde11eba7f7acde48001122", "test"], ["1dda782c0bde11eba7f7acde48001122", "dev"], ["eae8b12c0bdd11eba7f7acde48001122", "test"], ["882f660e0bdb11eba7f7acde48001122", "test"], ["617ad9b80bda11eba7f7acde48001122", "test"], ["86e985300bdc11eba7f7acde48001122", "test"], ["5f96ce98085311ebbd59ac1f6bf848b6", "dev"], ["4f0d28fb088b11ebbd6fac1f6bf848b6", "test"], ["5510d0620bdd11eba7f7acde48001122", "test"], ["5b2491a1092b11ebbdaeac1f6bf848b6", "test"], ["bda4c9080bdd11eba7f7acde48001122", "test"], ["eec16bb008c011ebbd8aac1f6bf848b6", "test"], ["b50759c3087811ebbd67ac1f6bf848b6", "dev"], ["8563457a085d11ebbd5cac1f6bf848b6", "test"], ["5b6fc7880bdd11eba7f7acde48001122", "test"], ["5aa25a540bde11eba7f7acde48001122", "test"], ["eee1f0af085911ebbd5bac1f6bf848b6", "dev"], ["43c4199e0bde11eba7f7acde48001122", "dev"], ["1fcb35680bde11eba7f7acde48001122", "dev"], ["ffeee7fa0bae11ebab90acde48001122", "test"], ["1bd49ae00bdd11eba7f7acde48001122", "test"], ["3416f6240bde11eba7f7acde48001122", "test"], ["0f0a793b086111ebbd5dac1f6bf848b6", "test"], ["270ef250097211ebbdb0ac1f6bf848b6", "test"], ["525d90d8086f11ebbd62ac1f6bf848b6", "test"], ["307a3d240bb011ebab90acde48001122", "test"], ["46c87e080bdb11eba7f7acde48001122", "dev"], ["cd67568c0bdb11eba7f7acde48001122", "test"], ["618c39ce0bda11eba7f7acde48001122", "test"], ["20c85b260bde11eba7f7acde48001122", "test"], ["21a93ff60bde11eba7f7acde48001122", "dev"], ["9f500386089111ebbd73ac1f6bf848b6", "test"], ["8cd406cc0bdd11eba7f7acde48001122", "test"], ["c451c9500bdc11eba7f7acde48001122", "dev"], ["21eb73c708af11ebbd83ac1f6bf848b6", "test"], ["0ea215140bdd11eba7f7acde48001122", "test"], ["ea294116094111ebbdaeac1f6bf848b6", "dev"], ["bc76478c0bdd11eba7f7acde48001122", "test"], ["b137755a084d11ebbd56ac1f6bf848b6", "test"], ["86806877099411ebbdb0ac1f6bf848b6", "dev"], ["2cf819720bde11eba7f7acde48001122", "dev"], ["ec3d39280bae11ebab90acde48001122", "test"], ["994bfa400bdd11eba7f7acde48001122", "test"], ["4b00867a0bde11eba7f7acde48001122", "test"], ["9980b1ed089d11ebbd78ac1f6bf848b6", "test"], ["694a87fc0bde11eba7f7acde48001122", "test"], ["7246ee5a0bb011ebab90acde48001122", "test"], ["288f010c0bde11eba7f7acde48001122", "test"], ["00c2089e089911ebbd77ac1f6bf848b6", "dev"], ["edb5bc2e0bdd11eba7f7acde48001122", "test"], ["9da14ff1086311ebbd5eac1f6bf848b6", "dev"], ["1cf6b11808a111ebbd78ac1f6bf848b6", "dev"], ["e6e862f208b511ebbd88ac1f6bf848b6", "test"], ["2102541508ac11ebbd82ac1f6bf848b6", "test"], ["575babe4087711ebbd67ac1f6bf848b6", "dev"], ["8ebd30080bb011ebab90acde48001122", "dev"], ["050b59380bde11eba7f7acde48001122", "dev"], ["86e8e3720bb011ebab90acde48001122", "test"], ["2dabcfee0bde11eba7f7acde48001122", "test"], ["eafa4baa0bae11ebab90acde48001122", "test"], ["007b76e80bdc11eba7f7acde48001122", "test"], ["d03d56d808a511ebbd7dac1f6bf848b6", "dev"], ["a944870c0bdb11eba7f7acde48001122", "test"], ["2dee8d7a0bde11eba7f7acde48001122", "test"], ["4ebea1ab08ac11ebbd82ac1f6bf848b6", "dev"], ["6d6c12880bd911eba7f7acde48001122", "test"], ["30feb4ae0bde11eba7f7acde48001122", "dev"], ["88fd0c0c0bdc11eba7f7acde48001122", "dev"], ["8b14eee20bdc11eba7f7acde48001122", "test"], ["065b409008d311ebbd96ac1f6bf848b6", "test"], ["700e3e0808aa11ebbd80ac1f6bf848b6", "test"], ["71d1a425087c11ebbd6aac1f6bf848b6", "test"], ["683cfb340bb011ebab90acde48001122", "test"], ["6dc4964e088711ebbd6dac1f6bf848b6", "test"], ["fcb1a7f4086c11ebbd61ac1f6bf848b6", "dev"], ["22d296de0bde11eba7f7acde48001122", "test"], ["30001f5c0bde11eba7f7acde48001122", "test"], ["33e3d8420bda11eba7f7acde48001122", "test"], ["b7f5f7200bdd11eba7f7acde48001122", "test"], ["367b700c0bde11eba7f7acde48001122", "dev"], ["cf9733f20bd911eba7f7acde48001122", "dev"], ["8890a29708d611ebbd98ac1f6bf848b6", "test"], ["12d10eb408be11ebbd8aac1f6bf848b6", "test"], ["c471dbc40bdb11eba7f7acde48001122", "test"], ["89a9c0ea0bb011ebab90acde48001122", "dev"], ["283419940bda11eba7f7acde48001122", "dev"], ["9eb76ea408ba11ebbd88ac1f6bf848b6", "dev"], ["82e0e3020bdc11eba7f7acde48001122", "test"], ["b411d0f6084d11ebbd56ac1f6bf848b6", "dev"], ["430b891e0bda11eba7f7acde48001122", "test"], ["3203b06209c111ebbdb0ac1f6bf848b6", "dev"], ["f7bae9100bdd11eba7f7acde48001122", "test"], ["769106c40bda11eba7f7acde48001122", "dev"], ["06d0699e0bb011ebab90acde48001122", "test"], ["7e83cbd40bb011ebab90acde48001122", "dev"], ["7cb81afc0bd911eba7f7acde48001122", "dev"], ["6b68236008b311ebbd86ac1f6bf848b6", "dev"], ["8913b9be0bd911eba7f7acde48001122", "test"], ["7f51c24a0bd911eba7f7acde48001122", "dev"], ["5cafe1780bb011ebab90acde48001122", "test"], ["1a301a200bb011ebab90acde48001122", "test"], ["b57e6576097611ebbdb0ac1f6bf848b6", "test"], ["4e1dc515091f11ebbdaeac1f6bf848b6", "dev"], ["7f800bee08c511ebbd8eac1f6bf848b6", "dev"], ["3647118c0bdd11eba7f7acde48001122", "test"], ["c2ccbd23087811ebbd67ac1f6bf848b6", "test"], ["b66d42be0bdd11eba7f7acde48001122", "dev"], ["0fd8f56e08be11ebbd8aac1f6bf848b6", "test"], ["933d94640bd911eba7f7acde48001122", "test"], ["8c6ba65608e511ebbda4ac1f6bf848b6", "dev"], ["218609020bdc11eba7f7acde48001122", "dev"], ["ae27375f08b811ebbd88ac1f6bf848b6", "test"], ["a685a14408b711ebbd88ac1f6bf848b6", "test"], ["f137fe640bda11eba7f7acde48001122", "test"], ["907764820baf11ebab90acde48001122", "dev"], ["2e9f306a0bdb11eba7f7acde48001122", "test"], ["b65557620bdd11eba7f7acde48001122", "test"], ["0dc9691b087911ebbd67ac1f6bf848b6", "dev"], ["b56bf714088311ebbd6dac1f6bf848b6", "test"], ["fd01563408e711ebbda6ac1f6bf848b6", "dev"], ["3cd79ef40bdd11eba7f7acde48001122", "test"], ["0899a4880bde11eba7f7acde48001122", "test"], ["4db672260bd911eba7f7acde48001122", "dev"], ["8a7214a20bdb11eba7f7acde48001122", "dev"], ["0e2fe3b40bdc11eba7f7acde48001122", "test"], ["a19ca5c80bdd11eba7f7acde48001122", "dev"], ["588039620bd911eba7f7acde48001122", "test"], ["87abb0ec0bdc11eba7f7acde48001122", "dev"], ["0d8b44260baf11ebab90acde48001122", "dev"], ["6de3bc0c0bd911eba7f7acde48001122", "dev"], ["f850838108d611ebbd99ac1f6bf848b6", "test"], ["925e59120bdd11eba7f7acde48001122", "test"], ["7f5dbf6e08cd11ebbd93ac1f6bf848b6", "test"], ["c9eb430908c311ebbd8dac1f6bf848b6", "test"], ["33e124b80bde11eba7f7acde48001122", "dev"], ["8c26eb3b087f11ebbd6cac1f6bf848b6", "test"], ["4294eaae0bb011ebab90acde48001122", "test"], ["8985546c0bdd11eba7f7acde48001122", "test"], ["b94fd2500bdb11eba7f7acde48001122", "test"], ["7c7bf2d308c011ebbd8aac1f6bf848b6", "test"], ["130cd6580bdc11eba7f7acde48001122", "test"], ["7b8cd8bc0bb011ebab90acde48001122", "test"], ["804026ed086911ebbd5eac1f6bf848b6", "test"], ["47bafeb40bde11eba7f7acde48001122", "test"], ["dbf6cc6d090d11ebbdadac1f6bf848b6", "test"], ["3d69910508ba11ebbd88ac1f6bf848b6", "test"], ["b81fbe620bdc11eba7f7acde48001122", "test"], ["8738be5c0bdc11eba7f7acde48001122", "test"], ["42359f760bde11eba7f7acde48001122", "test"], ["0c8a27700bde11eba7f7acde48001122", "test"], ["0c0830a00baf11ebab90acde48001122", "dev"], ["e81159820bdc11eba7f7acde48001122", "dev"], ["058f0bd308ac11ebbd82ac1f6bf848b6", "test"], ["8e634869088111ebbd6dac1f6bf848b6", "dev"], ["aa7f8f0308bd11ebbd8aac1f6bf848b6", "test"], ["f1bb0b420bda11eba7f7acde48001122", "dev"], ["e3280f82092b11ebbdaeac1f6bf848b6", "dev"], ["c81d185a0baf11ebab90acde48001122", "test"], ["f9b5c851087411ebbd67ac1f6bf848b6", "dev"], ["7ff5ce48091b11ebbdaeac1f6bf848b6", "test"], ["a3939d3b08a111ebbd79ac1f6bf848b6", "dev"], ["2b6af57408b511ebbd88ac1f6bf848b6", "dev"], ["90b5ca2c0bd911eba7f7acde48001122", "test"], ["1c536c5e0bdc11eba7f7acde48001122", "test"], ["9efb478a0bdc11eba7f7acde48001122", "dev"], ["9d6c6c5c087e11ebbd6bac1f6bf848b6", "test"], ["c04d7e5008d311ebbd96ac1f6bf848b6", "dev"], ["cee7790e08cc11ebbd93ac1f6bf848b6", "test"], ["f63c53fe08d011ebbd95ac1f6bf848b6", "test"], ["1aabf3040bdb11eba7f7acde48001122", "test"], ["ad95061a0bdb11eba7f7acde48001122", "test"], ["a6689af208b911ebbd88ac1f6bf848b6", "test"], ["e4b754ce085011ebbd58ac1f6bf848b6", "dev"], ["a5249d200bda11eba7f7acde48001122", "test"], ["046b3d900bde11eba7f7acde48001122", "dev"], ["13f3480e0bdc11eba7f7acde48001122", "test"], ["ecb808ba0bdb11eba7f7acde48001122", "dev"], ["9d5555ba0baf11ebab90acde48001122", "test"], ["d3061581096511ebbdafac1f6bf848b6", "test"], ["38fdf5420bda11eba7f7acde48001122", "test"], ["4ce7cb8e0bdd11eba7f7acde48001122", "test"], ["f18d54ec087911ebbd67ac1f6bf848b6", "test"], ["251db78c0bdb11eba7f7acde48001122", "dev"], ["08bc79d60bde11eba7f7acde48001122", "test"], ["f43518160bdc11eba7f7acde48001122", "dev"], ["fcdafe320bdb11eba7f7acde48001122", "test"], ["b6f2bce20baf11ebab90acde48001122", "dev"], ["41e2469e0bdb11eba7f7acde48001122", "dev"], ["94a64ce80bdd11eba7f7acde48001122", "test"], ["9aa497b808d511ebbd98ac1f6bf848b6", "dev"], ["fbf191f3086611ebbd5eac1f6bf848b6", "test"], ["844db8ce0bdb11eba7f7acde48001122", "dev"], ["0d3fcc480bdd11eba7f7acde48001122", "test"], ["415e3b580bde11eba7f7acde48001122", "test"], ["59a6e7020baf11ebab90acde48001122", "dev"], ["a1e68bee08c111ebbd8bac1f6bf848b6", "dev"], ["aac8784d087d11ebbd6bac1f6bf848b6", "test"], ["3013b498085711ebbd5aac1f6bf848b6", "test"], ["2fe34c7e0bde11eba7f7acde48001122", "test"], ["326b739408ba11ebbd88ac1f6bf848b6", "test"], ["f0696baa0bdd11eba7f7acde48001122", "test"], ["05c81d7a0bde11eba7f7acde48001122", "dev"], ["391edf020bb011ebab90acde48001122", "test"], ["b8a95fa60bdb11eba7f7acde48001122", "test"], ["164376040bde11eba7f7acde48001122", "test"], ["5ead80ee0baf11ebab90acde48001122", "test"], ["365f3eec08d111ebbd95ac1f6bf848b6", "test"], ["e6a47550097611ebbdb0ac1f6bf848b6", "dev"], ["98c9943608ee11ebbda8ac1f6bf848b6", "dev"], ["3eb2c1080bd911eba7f7acde48001122", "dev"], ["dcfa685e089411ebbd75ac1f6bf848b6", "dev"], ["9fdebb500baf11ebab90acde48001122", "test"], ["00b92f9d089e11ebbd78ac1f6bf848b6", "test"], ["c598b04f095211ebbdaeac1f6bf848b6", "dev"], ["ef77191208af11ebbd84ac1f6bf848b6", "test"], ["5068b0620bdc11eba7f7acde48001122", "dev"], ["16a4a1960bb011ebab90acde48001122", "test"], ["9588e1c2088911ebbd6eac1f6bf848b6", "dev"], ["25f075ca088411ebbd6dac1f6bf848b6", "test"], ["159f1f9e08fc11ebbdadac1f6bf848b6", "dev"], ["f08298de08ef11ebbda8ac1f6bf848b6", "test"], ["c32ac4ca0bda11eba7f7acde48001122", "test"], ["1600683e088c11ebbd6fac1f6bf848b6", "test"], ["f8eeca420bdb11eba7f7acde48001122", "test"], ["ab4cf04e089711ebbd77ac1f6bf848b6", "dev"], ["e73a5394089211ebbd74ac1f6bf848b6", "test"], ["39ca32a80bdd11eba7f7acde48001122", "test"], ["3ddc20060bdb11eba7f7acde48001122", "test"], ["b7e51fae089011ebbd72ac1f6bf848b6", "dev"], ["12cf000e0bdb11eba7f7acde48001122", "test"], ["08cae5720baf11ebab90acde48001122", "test"], ["783ed2cc08b111ebbd85ac1f6bf848b6", "dev"], ["cd80e22a08d711ebbd9aac1f6bf848b6", "test"], ["4269c2260bdc11eba7f7acde48001122", "dev"], ["f0465b7c086511ebbd5eac1f6bf848b6", "test"], ["8ff387aa096a11ebbdafac1f6bf848b6", "test"], ["e06e9794084d11ebbd56ac1f6bf848b6", "test"], ["f3a0ade8087611ebbd67ac1f6bf848b6", "dev"], ["2e90b7440bde11eba7f7acde48001122", "dev"], ["8ede14580bdd11eba7f7acde48001122", "dev"], ["ceef393e0bdb11eba7f7acde48001122", "test"], ["c4a3fde40bda11eba7f7acde48001122", "dev"], ["d1c8b2a00bdd11eba7f7acde48001122", "test"], ["384093b208b011ebbd84ac1f6bf848b6", "test"], ["2de08f6d08a611ebbd7eac1f6bf848b6", "test"], ["4d7754b20bdc11eba7f7acde48001122", "dev"], ["bdd526c4091811ebbdaeac1f6bf848b6", "test"], ["a9f8a706086511ebbd5eac1f6bf848b6", "test"], ["e9606c2408f711ebbdaaac1f6bf848b6", "test"], ["eccdbf80085b11ebbd5cac1f6bf848b6", "test"], ["bacbc6580bda11eba7f7acde48001122", "test"], ["e7745802092611ebbdaeac1f6bf848b6", "test"], ["18ebaf020bde11eba7f7acde48001122", "test"], ["a368fe1608b211ebbd85ac1f6bf848b6", "test"], ["8948e5220bdd11eba7f7acde48001122", "dev"], ["f2dbc5b4089e11ebbd78ac1f6bf848b6", "dev"], ["3685f31a0bde11eba7f7acde48001122", "test"], ["7f2f0f540baf11ebab90acde48001122", "test"], ["cafd9a02089b11ebbd78ac1f6bf848b6", "dev"], ["83dcd4ae08c211ebbd8cac1f6bf848b6", "test"], ["4f84536c0bdd11eba7f7acde48001122", "test"], ["629b294c088311ebbd6dac1f6bf848b6", "test"], ["5c8407dc0bde11eba7f7acde48001122", "dev"], ["9021269708e711ebbda6ac1f6bf848b6", "test"], ["01e265d20bdc11eba7f7acde48001122", "dev"], ["22d0830d087911ebbd67ac1f6bf848b6", "test"], ["8d0d8a680bd911eba7f7acde48001122", "test"], ["922879780bb011ebab90acde48001122", "dev"], ["4b6e9f5c0bde11eba7f7acde48001122", "test"], ["edfc5c27087011ebbd63ac1f6bf848b6", "test"], ["de7b90800bdd11eba7f7acde48001122", "dev"], ["3bc681f00bd911eba7f7acde48001122", "test"], ["55c66eca093111ebbdaeac1f6bf848b6", "test"], ["06ceba5e0bb011ebab90acde48001122", "test"], ["9436525108de11ebbd9eac1f6bf848b6", "test"], ["4601087408c411ebbd8eac1f6bf848b6", "test"], ["b6e4474808a311ebbd7bac1f6bf848b6", "dev"], ["5953e3b0091111ebbdadac1f6bf848b6", "test"], ["c624a31e0bdd11eba7f7acde48001122", "dev"], ["aed628ae085911ebbd5bac1f6bf848b6", "test"], ["ea862fd00baf11ebab90acde48001122", "test"], ["c4fd6fe80bdd11eba7f7acde48001122", "dev"], ["00310dc8094611ebbdaeac1f6bf848b6", "test"], ["dbf41c00086911ebbd5fac1f6bf848b6", "test"], ["be0a595e0baf11ebab90acde48001122", "test"], ["299660ba0baf11ebab90acde48001122", "test"], ["8715f72a0bda11eba7f7acde48001122", "dev"], ["68cb7bc00bdd11eba7f7acde48001122", "test"], ["4dad8e800bda11eba7f7acde48001122", "test"], ["8cb889a1096711ebbdafac1f6bf848b6", "test"], ["05ff422508cc11ebbd93ac1f6bf848b6", "test"], ["e5bc717c0bdb11eba7f7acde48001122", "test"], ["7f33c8cd087711ebbd67ac1f6bf848b6", "test"], ["57eb21ba0bde11eba7f7acde48001122", "test"], ["f6f9fbdc0bd911eba7f7acde48001122", "dev"], ["2d7f8dac088311ebbd6dac1f6bf848b6", "test"], ["0b91d3200bdb11eba7f7acde48001122", "dev"], ["683740ae0bb011ebab90acde48001122", "dev"], ["8cf72457085211ebbd58ac1f6bf848b6", "dev"], ["08ef74120bde11eba7f7acde48001122", "test"], ["1732aeb80bde11eba7f7acde48001122", "dev"], ["186e19be08d911ebbd9cac1f6bf848b6", "test"], ["23a5200e08c311ebbd8dac1f6bf848b6", "test"], ["72a61b200bdb11eba7f7acde48001122", "test"], ["ff2926840bd911eba7f7acde48001122", "test"], ["cb9d5ad408a911ebbd80ac1f6bf848b6", "test"], ["dc6abf5c08a211ebbd7bac1f6bf848b6", "test"], ["38779cfe0bda11eba7f7acde48001122", "dev"], ["ba08756a0bdd11eba7f7acde48001122", "test"], ["9c7a105b08e311ebbda4ac1f6bf848b6", "test"], ["bac2cfd20bdd11eba7f7acde48001122", "test"], ["956a70660bdb11eba7f7acde48001122", "test"], ["bc7073a80baf11ebab90acde48001122", "test"], ["ee97ee8a08d811ebbd9cac1f6bf848b6", "dev"], ["004430780bde11eba7f7acde48001122", "test"], ["9522a858088911ebbd6eac1f6bf848b6", "dev"], ["f69997160bdd11eba7f7acde48001122", "test"], ["0603066a0bde11eba7f7acde48001122", "dev"], ["cb2449d80baf11ebab90acde48001122", "dev"], ["6b58975d08df11ebbda0ac1f6bf848b6", "test"], ["b4a5545c096311ebbdafac1f6bf848b6", "test"], ["6c7f03240bdb11eba7f7acde48001122", "test"], ["19e6c50508d711ebbd99ac1f6bf848b6", "dev"], ["81676e5c0bdb11eba7f7acde48001122", "test"], ["f87501ce0bdd11eba7f7acde48001122", "test"], ["1b1b8ed6085111ebbd58ac1f6bf848b6", "test"], ["c3317246088511ebbd6dac1f6bf848b6", "test"], ["433de6f40bdd11eba7f7acde48001122", "test"], ["6b80af760baf11ebab90acde48001122", "dev"], ["327d52320bdd11eba7f7acde48001122", "test"], ["26d641160bdb11eba7f7acde48001122", "test"], ["bbee76c00bda11eba7f7acde48001122", "dev"], ["320bdd100baf11ebab90acde48001122", "test"], ["1831cf5f086811ebbd5eac1f6bf848b6", "test"], ["7ab366f60bdb11eba7f7acde48001122", "test"], ["bf29f36c0bdc11eba7f7acde48001122", "dev"], ["5db7d3f00bdd11eba7f7acde48001122", "dev"], ["e3d274ec0bae11ebab90acde48001122", "dev"], ["a075ffa20bdb11eba7f7acde48001122", "test"], ["5c2541900bdc11eba7f7acde48001122", "dev"], ["dda5755c08c911ebbd92ac1f6bf848b6", "test"], ["8540ae66088011ebbd6cac1f6bf848b6", "dev"], ["4002084208f111ebbda8ac1f6bf848b6", "test"], ["a0b9480a0bdd11eba7f7acde48001122", "test"], ["3e4f77020bb011ebab90acde48001122", "dev"], ["21c90ee5085f11ebbd5cac1f6bf848b6", "test"], ["bd4bd0fc0bdb11eba7f7acde48001122", "dev"], ["ae09896c0bdc11eba7f7acde48001122", "dev"], ["79675c6e0bd911eba7f7acde48001122", "dev"], ["60bd48de0bdb11eba7f7acde48001122", "dev"], ["0ee18e960bde11eba7f7acde48001122", "dev"], ["f64be57a0bdd11eba7f7acde48001122", "test"], ["61b28c2609a411ebbdb0ac1f6bf848b6", "dev"], ["902000320bd911eba7f7acde48001122", "dev"], ["ee51acb0091111ebbdadac1f6bf848b6", "dev"], ["971e4fb20bdc11eba7f7acde48001122", "test"], ["025318340bde11eba7f7acde48001122", "test"], ["e28c02bf08e411ebbda4ac1f6bf848b6", "dev"], ["0f15e3e40bde11eba7f7acde48001122", "dev"], ["f4cc9d4908b311ebbd86ac1f6bf848b6", "dev"], ["a9406286088d11ebbd70ac1f6bf848b6", "test"], ["174d30440bde11eba7f7acde48001122", "test"], ["97c19ef20bdb11eba7f7acde48001122", "test"], ["c822da15089a11ebbd77ac1f6bf848b6", "test"], ["61ad4c640bde11eba7f7acde48001122", "test"], ["43765d20087411ebbd67ac1f6bf848b6", "dev"], ["d3dd2d04085f11ebbd5dac1f6bf848b6", "test"], ["27ee96820bb011ebab90acde48001122", "test"], ["c507cba60baf11ebab90acde48001122", "test"], ["ca7775140bdc11eba7f7acde48001122", "dev"], ["e4cd03ae0bda11eba7f7acde48001122", "test"], ["dff7c3b60bdd11eba7f7acde48001122", "test"], ["66d8f7560bde11eba7f7acde48001122", "dev"], ["038d34e608d211ebbd95ac1f6bf848b6", "test"], ["60acf35c0bdc11eba7f7acde48001122", "dev"], ["9686afee0bdd11eba7f7acde48001122", "test"], ["06d9ed240bde11eba7f7acde48001122", "test"], ["d6efd86c0bdd11eba7f7acde48001122", "test"], ["35f99ea60bde11eba7f7acde48001122", "test"], ["ab5beae00bda11eba7f7acde48001122", "test"], ["6856ba560bdd11eba7f7acde48001122", "test"], ["0c4973520bb011ebab90acde48001122", "test"], ["5d85a6c808ae11ebbd83ac1f6bf848b6", "test"], ["8956ddd00bb011ebab90acde48001122", "test"], ["0056b9ff08e711ebbda5ac1f6bf848b6", "test"], ["875e6f8a0baf11ebab90acde48001122", "test"], ["2bb730d9087411ebbd67ac1f6bf848b6", "test"], ["e29b4d360bdd11eba7f7acde48001122", "test"], ["86fcf5a00bd911eba7f7acde48001122", "test"], ["1029efd6089211ebbd73ac1f6bf848b6", "test"], ["7470af90089511ebbd75ac1f6bf848b6", "test"], ["31d4d9de091e11ebbdaeac1f6bf848b6", "test"], ["360e44320bde11eba7f7acde48001122", "test"], ["72905066086f11ebbd62ac1f6bf848b6", "dev"], ["df4013ca08e411ebbda4ac1f6bf848b6", "dev"], ["79053ffa087f11ebbd6cac1f6bf848b6", "test"], ["f0d6f50c08eb11ebbda7ac1f6bf848b6", "test"], ["60465ce2085411ebbd59ac1f6bf848b6", "test"], ["a7b9672009c311ebbdb0ac1f6bf848b6", "test"], ["9a89b5460bdd11eba7f7acde48001122", "dev"], ["429cb2db087611ebbd67ac1f6bf848b6", "test"], ["079947de0baf11ebab90acde48001122", "test"], ["1f9ccd180bde11eba7f7acde48001122", "test"], ["6450154e08e311ebbda4ac1f6bf848b6", "dev"], ["f11bc9580bdd11eba7f7acde48001122", "dev"], ["12851c8c08b611ebbd88ac1f6bf848b6", "test"], ["4fa75a2e0bdd11eba7f7acde48001122", "test"], ["46d9df32097211ebbdb0ac1f6bf848b6", "dev"], ["5b1d862408e711ebbda6ac1f6bf848b6", "test"], ["6fe30f58097211ebbdb0ac1f6bf848b6", "test"], ["c2b3754008b611ebbd88ac1f6bf848b6", "test"], ["7efb04e708a011ebbd78ac1f6bf848b6", "test"], ["a8f04a7f090211ebbdadac1f6bf848b6", "dev"], ["cfd9bbba087711ebbd67ac1f6bf848b6", "test"], ["5c83d5960bde11eba7f7acde48001122", "dev"], ["03e971d208d911ebbd9cac1f6bf848b6", "test"], ["04bacaac08d611ebbd98ac1f6bf848b6", "test"], ["3d24d6740bb011ebab90acde48001122", "test"], ["53ca74b408e611ebbda5ac1f6bf848b6", "test"], ["1604c03a0bde11eba7f7acde48001122", "test"], ["b3b6e11708cb11ebbd92ac1f6bf848b6", "test"], ["4de007f40bdd11eba7f7acde48001122", "test"], ["2d76a8f408dd11ebbd9dac1f6bf848b6", "dev"], ["f718f7080bd911eba7f7acde48001122", "dev"], ["850908d4087211ebbd66ac1f6bf848b6", "dev"], ["fe1ac99c0bdd11eba7f7acde48001122", "dev"], ["efeac4af085d11ebbd5cac1f6bf848b6", "dev"], ["f5b8d1ca095511ebbdaeac1f6bf848b6", "test"], ["645e86920bdb11eba7f7acde48001122", "test"], ["f178e982098311ebbdb0ac1f6bf848b6", "test"], ["52d0c351086511ebbd5eac1f6bf848b6", "test"], ["1597b4be086311ebbd5eac1f6bf848b6", "test"], ["e5fa9df9085c11ebbd5cac1f6bf848b6", "dev"], ["8792c6b40bdb11eba7f7acde48001122", "test"], ["f6e4582a0bae11ebab90acde48001122", "test"], ["d7245d400baf11ebab90acde48001122", "test"], ["496c1afc087f11ebbd6bac1f6bf848b6", "test"], ["8a6f75ce0bdd11eba7f7acde48001122", "test"], ["28016c540bdc11eba7f7acde48001122", "test"], ["9b231f480bdc11eba7f7acde48001122", "test"], ["4f81c5260bdc11eba7f7acde48001122", "test"], ["22457b2c0bda11eba7f7acde48001122", "test"], ["f12995e108ab11ebbd82ac1f6bf848b6", "test"], ["42a1596708e911ebbda7ac1f6bf848b6", "test"], ["262fa5ee0bdc11eba7f7acde48001122", "test"], ["69e0e0a008e611ebbda5ac1f6bf848b6", "test"], ["fac4b0f80bda11eba7f7acde48001122", "dev"], ["ec5882f008bb11ebbd88ac1f6bf848b6", "test"], ["e08b814c0bdc11eba7f7acde48001122", "dev"], ["1dba599508b811ebbd88ac1f6bf848b6", "test"], ["d9b4231a0bdc11eba7f7acde48001122", "test"], ["6ce9fd9e088611ebbd6dac1f6bf848b6", "test"], ["7ebeb03c08e511ebbda4ac1f6bf848b6", "dev"], ["50406110087c11ebbd6aac1f6bf848b6", "test"], ["3d1200fc0bde11eba7f7acde48001122", "test"], ["c226acb1087e11ebbd6bac1f6bf848b6", "test"], ["a79d3abe0bdd11eba7f7acde48001122", "test"], ["6577dfbe0baf11ebab90acde48001122", "test"], ["2858ddc609b211ebbdb0ac1f6bf848b6", "dev"], ["19ab7c2a0bdd11eba7f7acde48001122", "dev"], ["e1af9bde08a411ebbd7dac1f6bf848b6", "test"], ["6800d81a08a511ebbd7dac1f6bf848b6", "test"], ["835ec3be0bda11eba7f7acde48001122", "test"], ["955ddc6a0bdc11eba7f7acde48001122", "test"], ["e410241608e511ebbda4ac1f6bf848b6", "test"], ["86ea0e1e0bb011ebab90acde48001122", "test"], ["7a81f2720bdd11eba7f7acde48001122", "dev"], ["7eb46304099211ebbdb0ac1f6bf848b6", "dev"], ["1867be900bde11eba7f7acde48001122", "test"], ["736afe9e0baf11ebab90acde48001122", "dev"], ["f26d17b20bdd11eba7f7acde48001122", "test"], ["83d15c08085b11ebbd5cac1f6bf848b6", "dev"], ["a4f44cee0bdd11eba7f7acde48001122", "test"], ["396b04e7089911ebbd77ac1f6bf848b6", "dev"], ["54c9ff0c0bdd11eba7f7acde48001122", "test"], ["5f177a0c089a11ebbd77ac1f6bf848b6", "test"], ["e50c35fa0bdb11eba7f7acde48001122", "test"], ["762412ee0bda11eba7f7acde48001122", "test"], ["9a55ba400bdb11eba7f7acde48001122", "dev"], ["b1a94a9a0bdb11eba7f7acde48001122", "dev"], ["16a5155d08e411ebbda4ac1f6bf848b6", "test"], ["aca6514c0bda11eba7f7acde48001122", "dev"], ["2f09d428089e11ebbd78ac1f6bf848b6", "dev"], ["160a7c3c0bde11eba7f7acde48001122", "dev"], ["6ec70dc60bda11eba7f7acde48001122", "test"], ["f5a7d78908b711ebbd88ac1f6bf848b6", "test"], ["ca787aa80bdd11eba7f7acde48001122", "dev"], ["044a6058086d11ebbd61ac1f6bf848b6", "test"], ["cda7579c08b011ebbd85ac1f6bf848b6", "test"], ["6970df82088811ebbd6eac1f6bf848b6", "test"], ["6f8b791e085011ebbd57ac1f6bf848b6", "dev"], ["856267f20bd911eba7f7acde48001122", "test"], ["bde91bf60bda11eba7f7acde48001122", "dev"], ["8473b61c0bdd11eba7f7acde48001122", "test"], ["a82f0cd20bdd11eba7f7acde48001122", "test"], ["7a5fd5940baf11ebab90acde48001122", "dev"], ["32f471ae0bde11eba7f7acde48001122", "test"], ["22ea14cd08ef11ebbda8ac1f6bf848b6", "test"], ["d8ffd42a0bda11eba7f7acde48001122", "test"], ["44311522097311ebbdb0ac1f6bf848b6", "dev"], ["f341b024086511ebbd5eac1f6bf848b6", "test"], ["d02fa7e508ab11ebbd82ac1f6bf848b6", "test"], ["a365c2b20bd911eba7f7acde48001122", "dev"], ["21d77262096b11ebbdafac1f6bf848b6", "test"], ["c69fb1bc0bdd11eba7f7acde48001122", "test"], ["429d41f80bde11eba7f7acde48001122", "test"], ["286461f808ba11ebbd88ac1f6bf848b6", "test"], ["dd97cc92088111ebbd6dac1f6bf848b6", "test"], ["d05ec6000bda11eba7f7acde48001122", "test"], ["7db34f5808c811ebbd91ac1f6bf848b6", "dev"], ["82cf1051087111ebbd63ac1f6bf848b6", "test"], ["f1baf1f7089d11ebbd78ac1f6bf848b6", "test"], ["df25946e08b211ebbd85ac1f6bf848b6", "dev"], ["a917110f089911ebbd77ac1f6bf848b6", "test"], ["429238be087b11ebbd68ac1f6bf848b6", "test"], ["8055aca40bdb11eba7f7acde48001122", "test"], ["b8fd57ac0bd911eba7f7acde48001122", "test"], ["179836cc0baf11ebab90acde48001122", "test"], ["57c5d42508d611ebbd98ac1f6bf848b6", "dev"], ["8f153a80084911ebbd55ac1f6bf848b6", "test"], ["e6a74e020bdd11eba7f7acde48001122", "dev"], ["8bebbfd9092211ebbdaeac1f6bf848b6", "test"], ["022a3dd40bdd11eba7f7acde48001122", "test"], ["eaeffe520bae11ebab90acde48001122", "test"], ["3d28c6da0bb011ebab90acde48001122", "test"], ["5bf96996085111ebbd58ac1f6bf848b6", "dev"], ["a14e5c3e0bdc11eba7f7acde48001122", "test"], ["59364e060bb011ebab90acde48001122", "test"], ["3a98adc608fa11ebbdacac1f6bf848b6", "dev"], ["75e5c06d087811ebbd67ac1f6bf848b6", "dev"], ["e1db74e80bdd11eba7f7acde48001122", "dev"], ["5cad7ed608a111ebbd79ac1f6bf848b6", "test"], ["6a925b440bde11eba7f7acde48001122", "test"], ["ff243e0708e811ebbda7ac1f6bf848b6", "test"], ["3adf4e5a091511ebbdadac1f6bf848b6", "test"], ["0dedb0de0bdc11eba7f7acde48001122", "test"], ["64b8e7b80bdb11eba7f7acde48001122", "test"], ["1c6b605a0bde11eba7f7acde48001122", "dev"], ["2f8d8a780bde11eba7f7acde48001122", "test"], ["4aaa6e5c086911ebbd5eac1f6bf848b6", "test"], ["d7e4a2cc088e11ebbd70ac1f6bf848b6", "dev"], ["53e04f7e0bb011ebab90acde48001122", "test"], ["5dfccdee0baf11ebab90acde48001122", "test"], ["c0521bca087611ebbd67ac1f6bf848b6", "test"], ["72f602e60bb011ebab90acde48001122", "test"], ["09646113087011ebbd62ac1f6bf848b6", "test"], ["996220180bdd11eba7f7acde48001122", "test"], ["f5acb304087e11ebbd6bac1f6bf848b6", "test"], ["f1a9a49e0bdd11eba7f7acde48001122", "dev"], ["bbc587ba0bda11eba7f7acde48001122", "dev"], ["77e657d80bb011ebab90acde48001122", "dev"], ["946f76c2088c11ebbd6fac1f6bf848b6", "dev"], ["875f18220baf11ebab90acde48001122", "test"], ["6e6f7ec20bde11eba7f7acde48001122", "test"], ["8498b1b20bdb11eba7f7acde48001122", "test"], ["2020864008c211ebbd8bac1f6bf848b6", "test"], ["06992de80bde11eba7f7acde48001122", "dev"], ["541c0d400bdb11eba7f7acde48001122", "test"], ["390829840bdc11eba7f7acde48001122", "dev"], ["f9d3a86e0baf11ebab90acde48001122", "test"], ["170b21e60bb011ebab90acde48001122", "test"], ["02a0888a096811ebbdafac1f6bf848b6", "dev"], ["056492e3088611ebbd6dac1f6bf848b6", "test"], ["f38f2c1c086211ebbd5eac1f6bf848b6", "dev"], ["844783260bb011ebab90acde48001122", "test"], ["4c5074f9089711ebbd76ac1f6bf848b6", "test"], ["371d9a0008cb11ebbd92ac1f6bf848b6", "dev"], ["41d2f75d096c11ebbdafac1f6bf848b6", "test"], ["90c88e000bd911eba7f7acde48001122", "dev"], ["3c742be008da11ebbd9cac1f6bf848b6", "test"], ["6914c1780bdc11eba7f7acde48001122", "test"], ["22d0e6f208b611ebbd88ac1f6bf848b6", "test"], ["b051c536089e11ebbd78ac1f6bf848b6", "dev"], ["899f2e000bb011ebab90acde48001122", "dev"], ["aaa28a1b08e611ebbda5ac1f6bf848b6", "test"], ["1a2188840bb011ebab90acde48001122", "test"], ["27487988089e11ebbd78ac1f6bf848b6", "test"], ["f240f3720baf11ebab90acde48001122", "dev"], ["befd7f2008bf11ebbd8aac1f6bf848b6", "test"], ["c7f7a0de0baf11ebab90acde48001122", "test"], ["eb305dfc0baf11ebab90acde48001122", "test"], ["11c51df808ed11ebbda7ac1f6bf848b6", "test"], ["8e624f8b08cc11ebbd93ac1f6bf848b6", "dev"], ["5e5e5ee908d911ebbd9cac1f6bf848b6", "dev"], ["151b99840bdc11eba7f7acde48001122", "test"], ["ced4db140bdd11eba7f7acde48001122", "test"], ["ba6055720bd911eba7f7acde48001122", "test"], ["6350d1ba0bdb11eba7f7acde48001122", "test"], ["52a91e8c086411ebbd5eac1f6bf848b6", "test"], ["422bf7bc087011ebbd62ac1f6bf848b6", "test"], ["24ce50da091111ebbdadac1f6bf848b6", "dev"], ["9a6007700bdb11eba7f7acde48001122", "test"], ["c55939520bda11eba7f7acde48001122", "dev"], ["17e09010084f11ebbd56ac1f6bf848b6", "dev"], ["95c505f4097f11ebbdb0ac1f6bf848b6", "dev"], ["0bf76b020bdd11eba7f7acde48001122", "test"], ["ff918a88086611ebbd5eac1f6bf848b6", "test"], ["d3867189095111ebbdaeac1f6bf848b6", "test"], ["8d0836540bdd11eba7f7acde48001122", "dev"], ["367ea308088211ebbd6dac1f6bf848b6", "test"], ["72ef81bf08db11ebbd9cac1f6bf848b6", "test"], ["a0ed670b08bf11ebbd8aac1f6bf848b6", "test"], ["401541c40bde11eba7f7acde48001122", "dev"], ["c51148bc0bdb11eba7f7acde48001122", "test"], ["5418bd9b08c811ebbd91ac1f6bf848b6", "test"], ["95c63b740bdd11eba7f7acde48001122", "test"], ["ab8ae3a4088311ebbd6dac1f6bf848b6", "test"], ["84798504087911ebbd67ac1f6bf848b6", "test"], ["652d8d6608c511ebbd8eac1f6bf848b6", "test"], ["5773fdb20bdc11eba7f7acde48001122", "test"], ["76204d4e0bda11eba7f7acde48001122", "test"], ["31b35063089611ebbd76ac1f6bf848b6", "test"], ["66a3188c08f211ebbdaaac1f6bf848b6", "dev"], ["d0389d340baf11ebab90acde48001122", "dev"], ["b43d2c2e093a11ebbdaeac1f6bf848b6", "dev"], ["0a6484280bb011ebab90acde48001122", "test"], ["1b8b2c9b085711ebbd59ac1f6bf848b6", "test"], ["9dac19cc08df11ebbda1ac1f6bf848b6", "dev"], ["431aca62090711ebbdadac1f6bf848b6", "test"], ["87a079740bb011ebab90acde48001122", "test"], ["aeade7d408c911ebbd92ac1f6bf848b6", "dev"], ["5dea524a088511ebbd6dac1f6bf848b6", "dev"], ["6c8073820bd911eba7f7acde48001122", "test"], ["887aebd60bb011ebab90acde48001122", "dev"], ["c969c20c087711ebbd67ac1f6bf848b6", "test"], ["f16f85580baf11ebab90acde48001122", "test"], ["7a8cb2f20bdd11eba7f7acde48001122", "test"], ["b9687149089811ebbd77ac1f6bf848b6", "test"], ["d7261b96086511ebbd5eac1f6bf848b6", "dev"], ["102dfda108ca11ebbd92ac1f6bf848b6", "dev"], ["9a7f003a0bdb11eba7f7acde48001122", "test"], ["819007ee0bdc11eba7f7acde48001122", "test"], ["a469ea08084b11ebbd56ac1f6bf848b6", "test"], ["b930a43b095211ebbdaeac1f6bf848b6", "test"], ["0323cdea090d11ebbdadac1f6bf848b6", "dev"], ["f88dcc88088611ebbd6dac1f6bf848b6", "dev"], ["5a3102b8084c11ebbd56ac1f6bf848b6", "test"], ["6184a78608b011ebbd84ac1f6bf848b6", "test"], ["18a3dbaa08d411ebbd96ac1f6bf848b6", "dev"], ["cad698fe086311ebbd5eac1f6bf848b6", "test"], ["87b020b5091111ebbdadac1f6bf848b6", "test"], ["1a3bb74508dc11ebbd9cac1f6bf848b6", "dev"], ["b3d611b3088a11ebbd6fac1f6bf848b6", "test"], ["11684e340bde11eba7f7acde48001122", "test"], ["61811fc80bb011ebab90acde48001122", "test"], ["4d8f2e6a0bb011ebab90acde48001122", "test"], ["dd98f482089511ebbd75ac1f6bf848b6", "dev"], ["e19960120bdd11eba7f7acde48001122", "test"], ["4348efde0bda11eba7f7acde48001122", "test"], ["181db6e40baf11ebab90acde48001122", "test"], ["b4ecd693086911ebbd5eac1f6bf848b6", "test"], ["8c232c6208e211ebbda4ac1f6bf848b6", "test"], ["06b9dfa20bde11eba7f7acde48001122", "test"], ["29a735160baf11ebab90acde48001122", "test"], ["936c70440bda11eba7f7acde48001122", "dev"], ["9cdcbb3e0bda11eba7f7acde48001122", "test"], ["4352bcd8088011ebbd6cac1f6bf848b6", "test"], ["01293a720bdb11eba7f7acde48001122", "dev"], ["03f23e500bb011ebab90acde48001122", "test"], ["22f9ba0c0bde11eba7f7acde48001122", "test"], ["3ae1f320086211ebbd5dac1f6bf848b6", "test"], ["0c3675d40bda11eba7f7acde48001122", "test"], ["a42210100bda11eba7f7acde48001122", "test"], ["28c8c38f08cc11ebbd93ac1f6bf848b6", "test"], ["5eb2723808ba11ebbd88ac1f6bf848b6", "test"], ["f7bec14908d611ebbd99ac1f6bf848b6", "test"], ["e668ce940baf11ebab90acde48001122", "test"], ["be0fbe8e0bdd11eba7f7acde48001122", "test"], ["e22b34b2089d11ebbd78ac1f6bf848b6", "test"], ["8e64d2c208b411ebbd88ac1f6bf848b6", "test"], ["e54e143a08ce11ebbd94ac1f6bf848b6", "test"], ["8da9b95c0bb011ebab90acde48001122", "test"], ["207cabb80bde11eba7f7acde48001122", "test"], ["5f1a91840bdc11eba7f7acde48001122", "test"], ["7aec38ed089211ebbd73ac1f6bf848b6", "dev"], ["4acc2a100bde11eba7f7acde48001122", "test"], ["0bb7a3b60bdb11eba7f7acde48001122", "dev"], ["04059478085911ebbd5bac1f6bf848b6", "dev"], ["9c754a2608e211ebbda4ac1f6bf848b6", "dev"], ["591d13b6085711ebbd5aac1f6bf848b6", "dev"], ["d0c7b05e084a11ebbd56ac1f6bf848b6", "test"], ["9926fbf6089e11ebbd78ac1f6bf848b6", "dev"], ["a995507c0bdd11eba7f7acde48001122", "test"], ["3b744aa00bdb11eba7f7acde48001122", "test"], ["2e99eb8e0bde11eba7f7acde48001122", "test"], ["3b5a4070089711ebbd76ac1f6bf848b6", "dev"], ["21e851f60bdd11eba7f7acde48001122", "test"], ["09945977084d11ebbd56ac1f6bf848b6", "test"], ["de7fdc4e0bdd11eba7f7acde48001122", "test"], ["fcc1bcc008d311ebbd96ac1f6bf848b6", "test"], ["b15a17f708b911ebbd88ac1f6bf848b6", "dev"], ["ab18549a0bdc11eba7f7acde48001122", "test"], ["08b1888a089811ebbd77ac1f6bf848b6", "test"], ["0d5bc1440bda11eba7f7acde48001122", "dev"], ["57a045a108d211ebbd95ac1f6bf848b6", "dev"], ["3549a13c0bdd11eba7f7acde48001122", "dev"], ["59f6f19809ae11ebbdb0ac1f6bf848b6", "dev"], ["6605282808b811ebbd88ac1f6bf848b6", "test"], ["d133120e0bdd11eba7f7acde48001122", "test"], ["d1c2e8d40bdd11eba7f7acde48001122", "test"], ["cce18dc7086211ebbd5dac1f6bf848b6", "test"], ["790a75c40bdb11eba7f7acde48001122", "dev"], ["6faf10f40bde11eba7f7acde48001122", "test"], ["4957b7520bda11eba7f7acde48001122", "test"], ["433bcf3c08b711ebbd88ac1f6bf848b6", "test"], ["5b797cbd088911ebbd6eac1f6bf848b6", "dev"], ["b77d6f620bdd11eba7f7acde48001122", "test"], ["ffe22f5b08c111ebbd8bac1f6bf848b6", "test"], ["67aaa1940bdd11eba7f7acde48001122", "dev"], ["6726d43e0bda11eba7f7acde48001122", "test"], ["18c0ccb1086911ebbd5eac1f6bf848b6", "test"], ["78505ad8088c11ebbd6fac1f6bf848b6", "dev"], ["387c056e0bda11eba7f7acde48001122", "dev"], ["0c97f4720bde11eba7f7acde48001122", "test"], ["2b871cac0bdc11eba7f7acde48001122", "test"], ["acfd28ca094b11ebbdaeac1f6bf848b6", "test"], ["4175aadc0bdd11eba7f7acde48001122", "test"], ["dd9911900bda11eba7f7acde48001122", "test"], ["cb61a4da08c211ebbd8cac1f6bf848b6", "dev"], ["95a081f508a411ebbd7dac1f6bf848b6", "test"], ["960f40120bdd11eba7f7acde48001122", "test"], ["df6b3eda086f11ebbd62ac1f6bf848b6", "test"], ["529e60d20bd911eba7f7acde48001122", "test"], ["0e6f0d400bb011ebab90acde48001122", "dev"], ["1bca00b5089d11ebbd78ac1f6bf848b6", "test"], ["131fe1ba087011ebbd62ac1f6bf848b6", "test"], ["cc93bb5c090711ebbdadac1f6bf848b6", "dev"], ["e08d59e00bdc11eba7f7acde48001122", "test"], ["1793f8ec0bdb11eba7f7acde48001122", "test"], ["e1228e9f08ca11ebbd92ac1f6bf848b6", "test"], ["56c857fe0bb011ebab90acde48001122", "test"], ["97c6181e0baf11ebab90acde48001122", "test"], ["2033984c09a611ebbdb0ac1f6bf848b6", "dev"], ["9704513d099f11ebbdb0ac1f6bf848b6", "test"], ["d594f50208c111ebbd8bac1f6bf848b6", "test"], ["c9c9b1b208b311ebbd86ac1f6bf848b6", "test"], ["6cc9a2ca0baf11ebab90acde48001122", "dev"], ["5108efaa088511ebbd6dac1f6bf848b6", "test"], ["9771c3900bdc11eba7f7acde48001122", "test"], ["c1b4a0e60baf11ebab90acde48001122", "test"], ["084c53d20bdd11eba7f7acde48001122", "test"], ["c1139a99088a11ebbd6fac1f6bf848b6", "dev"], ["297aac2208a711ebbd7fac1f6bf848b6", "test"], ["6825f602087a11ebbd67ac1f6bf848b6", "test"], ["a8b954da0bda11eba7f7acde48001122", "test"], ["5d348f1608e211ebbda3ac1f6bf848b6", "test"], ["f4d74be40bdd11eba7f7acde48001122", "test"], ["5d0eadd80bde11eba7f7acde48001122", "dev"], ["7991e0360bdb11eba7f7acde48001122", "test"], ["523dfd220bda11eba7f7acde48001122", "test"], ["6db45b3208bf11ebbd8aac1f6bf848b6", "test"], ["85b6ab18087e11ebbd6bac1f6bf848b6", "test"], ["9407eccd093211ebbdaeac1f6bf848b6", "dev"], ["ad06f84c0bdc11eba7f7acde48001122", "test"], ["8129f9bc0bb011ebab90acde48001122", "test"], ["3675c54c08a911ebbd80ac1f6bf848b6", "dev"], ["eb3ff1900baf11ebab90acde48001122", "dev"], ["1514ccda08d011ebbd95ac1f6bf848b6", "test"], ["3654974a08df11ebbda0ac1f6bf848b6", "test"], ["9584cc16095311ebbdaeac1f6bf848b6", "dev"], ["4d7818b008bd11ebbd8aac1f6bf848b6", "test"], ["a8776d3a0bdb11eba7f7acde48001122", "test"], ["8813c44a0bda11eba7f7acde48001122", "test"], ["3085824c0bb011ebab90acde48001122", "test"], ["d10e511a0bda11eba7f7acde48001122", "dev"], ["c81f50b00bdd11eba7f7acde48001122", "test"], ["2f6b6a520bb011ebab90acde48001122", "dev"], ["0e91112508d111ebbd95ac1f6bf848b6", "test"], ["676bf7000bb011ebab90acde48001122", "dev"], ["2e4716e6089d11ebbd78ac1f6bf848b6", "test"], ["d4f125200bdd11eba7f7acde48001122", "dev"], ["6b2794bf088911ebbd6eac1f6bf848b6", "test"], ["21692f6a0bde11eba7f7acde48001122", "test"], ["b742119c0bdd11eba7f7acde48001122", "test"], ["c058eb020bdd11eba7f7acde48001122", "dev"], ["a5e0af240bda11eba7f7acde48001122", "test"], ["8f6592fc0bdd11eba7f7acde48001122", "dev"], ["6b0aeede0bdd11eba7f7acde48001122", "test"], ["a6bb56bc0bdd11eba7f7acde48001122", "test"], ["b8974ae60bdb11eba7f7acde48001122", "dev"], ["726d5adc08b211ebbd85ac1f6bf848b6", "test"], ["f3cdfe9108d211ebbd96ac1f6bf848b6", "test"], ["430e8a1608a511ebbd7dac1f6bf848b6", "test"], ["02acaafe084e11ebbd56ac1f6bf848b6", "dev"], ["be9369c80bdd11eba7f7acde48001122", "dev"], ["a8a18e800bdb11eba7f7acde48001122", "test"], ["85f17aea08c911ebbd92ac1f6bf848b6", "test"], ["a34b238a0bd911eba7f7acde48001122", "test"], ["5f6af38408c311ebbd8dac1f6bf848b6", "test"], ["7b61849208f311ebbdaaac1f6bf848b6", "test"], ["31eeaa6608cf11ebbd94ac1f6bf848b6", "dev"], ["fd4096540bd911eba7f7acde48001122", "dev"], ["a7b2d375089f11ebbd78ac1f6bf848b6", "dev"], ["0bc8afb8087e11ebbd6bac1f6bf848b6", "dev"], ["f96bdeb40baf11ebab90acde48001122", "test"], ["bfde72d6089b11ebbd78ac1f6bf848b6", "test"], ["f0231ce60baf11ebab90acde48001122", "test"], ["493552080bde11eba7f7acde48001122", "test"], ["674a7b8a0bdc11eba7f7acde48001122", "test"], ["33323be20bde11eba7f7acde48001122", "test"], ["243f2cd6085611ebbd59ac1f6bf848b6", "test"], ["12d8772a0bda11eba7f7acde48001122", "test"], ["2541dda808b911ebbd88ac1f6bf848b6", "test"], ["ea696896089911ebbd77ac1f6bf848b6", "test"], ["f8569b9c0bda11eba7f7acde48001122", "dev"], ["383c7761087311ebbd66ac1f6bf848b6", "dev"], ["df343d700bdc11eba7f7acde48001122", "dev"], ["28fb498a0bdd11eba7f7acde48001122", "test"], ["8374ba040bdd11eba7f7acde48001122", "test"], ["e35b10de08cd11ebbd93ac1f6bf848b6", "dev"], ["89022ba60bdc11eba7f7acde48001122", "test"], ["bd41e7840bdd11eba7f7acde48001122", "dev"], ["5485e4ac08ae11ebbd83ac1f6bf848b6", "test"], ["440317820bdb11eba7f7acde48001122", "dev"], ["aa201d800bdb11eba7f7acde48001122", "test"], ["a6961cb20bdd11eba7f7acde48001122", "dev"], ["d9d7844e0bdd11eba7f7acde48001122", "test"], ["01a3f8600bdc11eba7f7acde48001122", "test"], ["0ea12031093c11ebbdaeac1f6bf848b6", "test"], ["82fbc7500bda11eba7f7acde48001122", "test"], ["46cb0ba408ad11ebbd83ac1f6bf848b6", "test"], ["f0cba8e408cc11ebbd93ac1f6bf848b6", "test"], ["4dea6f320bdd11eba7f7acde48001122", "test"], ["8d1e4c000bdd11eba7f7acde48001122", "test"], ["5a64415f086111ebbd5dac1f6bf848b6", "test"], ["dc4c7573087211ebbd66ac1f6bf848b6", "test"], ["ab7c8844087011ebbd63ac1f6bf848b6", "dev"], ["5e735d77092011ebbdaeac1f6bf848b6", "test"], ["4228d7300bdb11eba7f7acde48001122", "dev"], ["dddda68c0bdc11eba7f7acde48001122", "test"], ["9e0a6c700baf11ebab90acde48001122", "dev"], ["140ef7b108b811ebbd88ac1f6bf848b6", "dev"], ["83938bc8084c11ebbd56ac1f6bf848b6", "test"], ["d2d5243f089111ebbd73ac1f6bf848b6", "test"], ["584ec9960bdd11eba7f7acde48001122", "test"], ["a601a62608e611ebbda5ac1f6bf848b6", "dev"], ["f5aba0260bdb11eba7f7acde48001122", "dev"], ["dc8d75280baf11ebab90acde48001122", "dev"], ["6fc6126e08ad11ebbd83ac1f6bf848b6", "dev"], ["dc5b9266091f11ebbdaeac1f6bf848b6", "test"], ["664ed504085e11ebbd5cac1f6bf848b6", "dev"], ["d66b7cac08bd11ebbd8aac1f6bf848b6", "test"], ["8fd6a5000bb011ebab90acde48001122", "test"], ["9028ba520bdd11eba7f7acde48001122", "dev"], ["4b9580400bde11eba7f7acde48001122", "test"], ["653886e40bdb11eba7f7acde48001122", "dev"], ["ef7b692008d911ebbd9cac1f6bf848b6", "test"], ["691636fa0bde11eba7f7acde48001122", "test"], ["4999c8ae08e211ebbda3ac1f6bf848b6", "test"], ["d7e238dd095f11ebbdafac1f6bf848b6", "test"], ["5591be31086211ebbd5dac1f6bf848b6", "test"], ["3ede24dd08ba11ebbd88ac1f6bf848b6", "dev"], ["645c9a5a08c311ebbd8dac1f6bf848b6", "test"], ["78b10fb00bdc11eba7f7acde48001122", "test"], ["38be3008086a11ebbd5fac1f6bf848b6", "test"], ["6bda7b1c0bde11eba7f7acde48001122", "test"], ["acdf3b1e08d411ebbd96ac1f6bf848b6", "dev"], ["6b8402c40bdd11eba7f7acde48001122", "test"], ["e43f26380bda11eba7f7acde48001122", "test"], ["d6e9771908dd11ebbd9eac1f6bf848b6", "test"], ["0c0b964c0bdb11eba7f7acde48001122", "test"], ["93d53e0a0bdd11eba7f7acde48001122", "test"], ["7c236af308d611ebbd98ac1f6bf848b6", "test"], ["75668b40094111ebbdaeac1f6bf848b6", "dev"], ["7bd922e00baf11ebab90acde48001122", "test"], ["b8ef144a0bdd11eba7f7acde48001122", "test"], ["2751a4720bdc11eba7f7acde48001122", "test"], ["f31d38460bdc11eba7f7acde48001122", "test"], ["9d58483c0bdd11eba7f7acde48001122", "dev"], ["0e5e54c908c211ebbd8bac1f6bf848b6", "test"], ["f4885f140bda11eba7f7acde48001122", "dev"], ["1f3da3b60bb011ebab90acde48001122", "dev"], ["aa0f65420bdd11eba7f7acde48001122", "test"], ["b40e36b60bda11eba7f7acde48001122", "dev"], ["a206979a0baf11ebab90acde48001122", "dev"], ["1fd0e1ac08af11ebbd83ac1f6bf848b6", "test"], ["4b07f85a0bda11eba7f7acde48001122", "dev"], ["0e0c10a60bdc11eba7f7acde48001122", "test"], ["1da795920bde11eba7f7acde48001122", "test"], ["fd28aace085e11ebbd5cac1f6bf848b6", "test"], ["91f1a2ae0bdd11eba7f7acde48001122", "test"], ["1c7395fa0bb011ebab90acde48001122", "test"], ["9237272a085911ebbd5bac1f6bf848b6", "dev"], ["8854559c08b911ebbd88ac1f6bf848b6", "dev"], ["b0a2e8ba0bd911eba7f7acde48001122", "dev"], ["59acfe98096c11ebbdafac1f6bf848b6", "test"], ["27c5abf80bda11eba7f7acde48001122", "test"], ["09558be00bdd11eba7f7acde48001122", "dev"], ["b5dfb818089011ebbd72ac1f6bf848b6", "dev"], ["45709660095311ebbdaeac1f6bf848b6", "test"], ["6bab217b091311ebbdadac1f6bf848b6", "dev"], ["f15b6f8e084e11ebbd56ac1f6bf848b6", "test"], ["ce82a5d8093a11ebbdaeac1f6bf848b6", "dev"], ["0cde5033087011ebbd62ac1f6bf848b6", "test"], ["b1d1edc6086911ebbd5eac1f6bf848b6", "test"], ["625f61c9090511ebbdadac1f6bf848b6", "test"], ["6a37394e0bde11eba7f7acde48001122", "test"], ["27e926ca0bb011ebab90acde48001122", "test"], ["c6c2d01c087111ebbd63ac1f6bf848b6", "test"], ["57df2e7e0bb011ebab90acde48001122", "test"], ["39bdb2980bde11eba7f7acde48001122", "dev"], ["c1327e9908be11ebbd8aac1f6bf848b6", "test"], ["ff8987ca0bae11ebab90acde48001122", "test"], ["b130234608a611ebbd7fac1f6bf848b6", "test"], ["b678eea20bdd11eba7f7acde48001122", "test"], ["7f98e4a2086311ebbd5eac1f6bf848b6", "test"], ["44d17fa408d711ebbd9aac1f6bf848b6", "dev"], ["29c40a78086811ebbd5eac1f6bf848b6", "test"], ["38b6a361092111ebbdaeac1f6bf848b6", "dev"], ["fff608920bda11eba7f7acde48001122", "test"], ["395a691b08cb11ebbd92ac1f6bf848b6", "dev"], ["51e8895608d711ebbd9aac1f6bf848b6", "test"], ["a2c1645c0bdd11eba7f7acde48001122", "test"], ["bf3e001408c111ebbd8bac1f6bf848b6", "dev"], ["e598d0600bda11eba7f7acde48001122", "dev"], ["a8396b140bdd11eba7f7acde48001122", "test"], ["a1f15751088e11ebbd70ac1f6bf848b6", "dev"], ["40a0ab780bda11eba7f7acde48001122", "test"], ["643b96320bdb11eba7f7acde48001122", "test"], ["54a7772a0bb011ebab90acde48001122", "dev"], ["8e71c6c80baf11ebab90acde48001122", "test"], ["cac4e3ac096711ebbdafac1f6bf848b6", "test"], ["240f079c0bdb11eba7f7acde48001122", "test"], ["709cfbb60bde11eba7f7acde48001122", "dev"], ["2023639c0bb011ebab90acde48001122", "test"], ["71cf03c20bdd11eba7f7acde48001122", "test"], ["13c15658087311ebbd66ac1f6bf848b6", "test"], ["4e0814b20bdb11eba7f7acde48001122", "test"], ["68f2479208c611ebbd90ac1f6bf848b6", "dev"], ["50b6626d08cb11ebbd92ac1f6bf848b6", "test"], ["08ff4b6c0bde11eba7f7acde48001122", "test"], ["df9e3cf60bdd11eba7f7acde48001122", "test"], ["bd69a8af088c11ebbd6fac1f6bf848b6", "dev"], ["257155d40bdd11eba7f7acde48001122", "test"], ["6b6fd02a0baf11ebab90acde48001122", "test"], ["438a3de40bdb11eba7f7acde48001122", "test"], ["08fe2b440bdc11eba7f7acde48001122", "test"], ["ab42b5c0085111ebbd58ac1f6bf848b6", "test"], ["acaa580c0bdd11eba7f7acde48001122", "test"], ["d671b93e08b111ebbd85ac1f6bf848b6", "test"], ["0404d2cc0bb011ebab90acde48001122", "dev"], ["5c0983a1094611ebbdaeac1f6bf848b6", "test"], ["10499c7508af11ebbd83ac1f6bf848b6", "test"], ["eb46e4360bdd11eba7f7acde48001122", "dev"], ["50a7273c088c11ebbd6fac1f6bf848b6", "test"], ["37a485c308c311ebbd8dac1f6bf848b6", "dev"], ["0dfb89600bdd11eba7f7acde48001122", "dev"], ["8dc5682e086711ebbd5eac1f6bf848b6", "test"], ["2ecc6bf40bde11eba7f7acde48001122", "dev"], ["6a419ba2087611ebbd67ac1f6bf848b6", "dev"], ["e01e25380bdd11eba7f7acde48001122", "dev"], ["161093c40bde11eba7f7acde48001122", "dev"], ["ad43c9300bdb11eba7f7acde48001122", "dev"], ["1da3cd28089a11ebbd77ac1f6bf848b6", "test"], ["d6aa5bb20bdc11eba7f7acde48001122", "dev"], ["7adb71b80bdc11eba7f7acde48001122", "test"], ["6b0c5a6008c911ebbd92ac1f6bf848b6", "dev"], ["bc29cc49085611ebbd59ac1f6bf848b6", "test"], ["45a190120bdd11eba7f7acde48001122", "test"], ["fe3a70f40bdc11eba7f7acde48001122", "test"], ["6b6a8d6008c411ebbd8eac1f6bf848b6", "test"], ["8471ae5808d111ebbd95ac1f6bf848b6", "test"], ["347300820bb011ebab90acde48001122", "dev"]], "corpus_sha1": "523dcd1a1c2ef4ebe2a3856f2c3ec8613fb52c37"} \ No newline at end of file diff --git a/benchmarks/multihop/manifests/hotpotqa_sample.json b/benchmarks/multihop/manifests/hotpotqa_sample.json new file mode 100644 index 0000000..5f29cce --- /dev/null +++ b/benchmarks/multihop/manifests/hotpotqa_sample.json @@ -0,0 +1 @@ +{"dataset": "hotpotqa", "source": {"repo": "hotpotqa/hotpot_qa", "revision": "1908d6afbbead072334abe2965f91bd2709910ab", "file": "distractor/validation-00000-of-00001.parquet"}, "seed": 13, "n_questions": 1500, "n_dev": 500, "n_test": 1000, "corpus_size": 14570, "n_available": 7405, "ids": [["5a8466a75542993344746108", "test"], ["5a87081755429960ec39b6fe", "test"], ["5adc8b5a5542994d58a2f648", "test"], ["5adc7b855542994650320d1e", "test"], ["5ae249925542994d89d5b3c1", "test"], ["5ae528e655429908b632652c", "test"], ["5ae7690c5542995703ce8c22", "dev"], ["5a7e513a5542994959419996", "dev"], ["5ac3b40a554299741d48a2e5", "dev"], ["5a81d54a55429903bc27b9d4", "test"], ["5ac4f12d55429924173fb4f6", "dev"], ["5a7bacf35542997c3ec9724b", "test"], ["5ae628f55542995703ce8b32", "test"], ["5a816db055429903bc27b961", "dev"], ["5ac2eaab5542990b17b154a4", "test"], ["5adf0e7c5542992d7e9f9297", "test"], ["5a7e7c725542991319bc94be", "dev"], ["5a7a88e455429941d65f268c", "dev"], ["5a7633ad5542992db9473741", "dev"], ["5aba549f554299232ef4a290", "dev"], ["5ae4ea2a5542993aec5ec0ee", "test"], ["5a808cbd5542996402f6a54b", "dev"], ["5adf88995542993344016cd4", "dev"], ["5a8752955542994775f607d1", "dev"], ["5a7327265542994cef4bc480", "test"], ["5ab2b863554299295394685a", "test"], ["5a7a4ca955429941d65f25a3", "test"], ["5ae4903855429913cc2044a0", "test"], ["5adc7dbf5542994d58a2f618", "test"], ["5ac046c65542996f0d89cb71", "test"], ["5a72339855429971e9dc934d", "test"], ["5a85b0c85542994c784ddb40", "test"], ["5ae4627d5542991a06ce9a4d", "test"], ["5a7ba83f5542995eb53be981", "dev"], ["5a775bd65542993569682d6b", "dev"], ["5ae644c55542992663a4f27e", "dev"], ["5ae1f528554299234fd04370", "test"], ["5ae2dfd155429928c423951b", "dev"], ["5ae75c395542991bbc9761f7", "test"], ["5ae28f43554299492dc91c2f", "dev"], ["5a849c1b5542997175ce1ef9", "test"], ["5ae3d5ec5542992f92d8238c", "test"], ["5ab43b8a5542991779162c22", "dev"], ["5ae80d25554299540e5a5709", "test"], ["5ab3010455429976abd1bc15", "dev"], ["5a7b43735542992d025e67cb", "dev"], ["5ae1f3df554299234fd04364", "dev"], ["5ae68a065542991bbc97611e", "test"], ["5a8434445542996488c2e517", "dev"], ["5a8baf4b5542996e8ac889c5", "test"], ["5ae52cb955429908b6326540", "dev"], ["5a82028a554299676cceb1d6", "test"], ["5ab6ecd85542991d322236cf", "test"], ["5adfef32554299603e4183ca", "test"], ["5abbb42555429931dba144af", "test"], ["5abcce3355429965836004a5", "test"], ["5ab2a308554299340b52553b", "dev"], ["5ac5391e5542994611c8b439", "dev"], ["5a8c7703554299585d9e36b9", "test"], ["5ae75e335542991e8301cc70", "dev"], ["5ab2a54b554299295394678e", "test"], ["5ac2c338554299657fa29039", "test"], ["5a88c92d5542993b751ca84e", "test"], ["5ac3d667554299076e296ca5", "test"], ["5a79e3925542994bb9457130", "test"], ["5a8b3d795542997f31a41cc1", "test"], ["5ae361d15542990afbd1e126", "dev"], ["5abece075542997ec76fd36d", "test"], ["5ae212c9554299495565d9de", "test"], ["5ac285335542996773102552", "test"], ["5a84a5385542990548d0b2d9", "test"], ["5adcceda5542990d50227d31", "test"], ["5ab42b905542992339550040", "test"], ["5abbe188554299642a094ba0", "dev"], ["5ac083b0554299294b21900d", "dev"], ["5adfa6db55429942ec259ad6", "dev"], ["5a7b01645542992d025e6734", "test"], ["5ab3abe9554299753aec597e", "dev"], ["5adbf81a554299438c868cfd", "test"], ["5adea95055429939a52fe931", "dev"], ["5ab3d2595542992ade7c6ebf", "test"], ["5aba901055429955dce3ee87", "test"], ["5a7dc2185542990b8f503a7b", "test"], ["5a873b0d5542994775f607ad", "test"], ["5a7fac0055429969796c1b1b", "dev"], ["5a7df6e65542995f4f402353", "dev"], ["5ab972485542996be20204f6", "test"], ["5a8bb7f95542996e8ac889ff", "dev"], ["5a83b4d35542996488c2e495", "dev"], ["5a8d0be85542994ba4e3dbb2", "test"], ["5ab514dc5542990594ba9cfa", "test"], ["5a8494b15542991dd0999d35", "test"], ["5ac00de75542996f0d89cb13", "test"], ["5a85fc995542996432c5715f", "test"], ["5a8f1adc554299458435d577", "dev"], ["5ae762835542997b22f6a711", "test"], ["5a7a93115542995eb53be86f", "test"], ["5abd74c95542993062266ca2", "test"], ["5ab736a65542992aa3b8c7db", "test"], ["5abbbaca55429931dba144ca", "test"], ["5a82119155429926c1cdae28", "test"], ["5abce73055429959677d6b34", "test"], ["5a821c95554299676cceb219", "test"], ["5ae74351554299572ea5479f", "test"], ["5a7f40055542992e7d278cc0", "test"], ["5add43335542995b365faae1", "test"], ["5a8cacce554299585d9e3718", "test"], ["5a7a954855429941d65f26cd", "test"], ["5a7662745542992db9473786", "dev"], ["5ab2af1655429929539467eb", "test"], ["5ac3ae475542995ef918c1e5", "test"], ["5ac2546a554299636651997b", "dev"], ["5ab55b0c554299488d4d995e", "test"], ["5a8ec3205542995a26add506", "dev"], ["5a710a915542994082a3e504", "test"], ["5ade286e5542997c77aded81", "test"], ["5ab27f7b554299722f9b4d4f", "test"], ["5ae6152d5542996de7b71b38", "test"], ["5a73f4f75542993a88ae2ebc", "dev"], ["5a815f445542995ce29dcbec", "dev"], ["5a7b63eb55429931da12ca7e", "test"], ["5ab7f78f5542991d322237e6", "dev"], ["5ae7f0c75542994a481bbe44", "dev"], ["5add21555542992ae4cec496", "test"], ["5ae4932855429970de88d9b8", "test"], ["5ab3ede755429976abd1bcf4", "dev"], ["5ac2cb09554299657fa29066", "test"], ["5a83c3275542996488c2e4be", "dev"], ["5a7c9f325542990527d554e6", "dev"], ["5adf04915542995534e8c72f", "test"], ["5a9084e75542990a98493667", "test"], ["5adf33b35542993a75d26420", "test"], ["5ae6196d5542995703ce8afd", "dev"], ["5a8393665542990548d0b1ea", "test"], ["5ae73d415542991e8301cc2b", "test"], ["5ae038e955429942ec259c4d", "test"], ["5ae25e835542992decbdccd1", "test"], ["5a8b768655429949d91db5a9", "test"], ["5a7fb1b25542992097ad2fb2", "test"], ["5add67705542992ae4cec54f", "dev"], ["5ae54e005542993aec5ec19a", "test"], ["5ac264195542992f1f2b38a5", "test"], ["5a8d7e1055429941ae14dfd8", "dev"], ["5ab9547755429970cfb8ea6c", "dev"], ["5ade0fbb5542995b365fabfb", "dev"], ["5ac3e0f7554299194317388b", "test"], ["5a7b215f55429931da12c9e3", "test"], ["5add5ac95542992200553ace", "test"], ["5abff65c5542994516f45551", "dev"], ["5a8257fa55429940e5e1a866", "test"], ["5ade653f55429975fa854ebb", "dev"], ["5ab7ee895542995dae37e9f6", "dev"], ["5a80799b5542992bc0c4a72e", "test"], ["5ae5e1345542996de7b71a50", "dev"], ["5ac13d5c5542996f0d89cc78", "test"], ["5a73d33e5542992d56e7e3a9", "test"], ["5abdb5b455429965af743dfa", "test"], ["5add44c25542997545bbbd0a", "test"], ["5abe953b5542993f32c2a170", "test"], ["5a7384755542992d56e7e354", "test"], ["5a8c06bb5542995e66a47530", "test"], ["5abdd8d95542991f6610606a", "test"], ["5ade9c2155429939a52fe8fe", "test"], ["5abd077055429965836004f4", "test"], ["5a8af3b155429950cd6afc2c", "test"], ["5a7e1a2955429965cec5ea5d", "test"], ["5a8756f25542994775f607dc", "test"], ["5a7f2d3e5542993067513675", "test"], ["5a7cd8da55429909bec76845", "test"], ["5ae1af14554299234fd042c4", "dev"], ["5ae830f05542997ec2727758", "test"], ["5abbf519554299114383a0ad", "test"], ["5a7949d45542994bb9456fdd", "test"], ["5a81737155429903bc27b96f", "test"], ["5a7cfb2755429907fabef084", "test"], ["5a7621bb554299109176e650", "test"], ["5ab2a01d5542992953946778", "test"], ["5ac3126f5542995ef918c0e8", "test"], ["5abed6905542990832d3a0ed", "test"], ["5abbfb5e554299114383a0d0", "dev"], ["5ae2a9c6554299492dc91c42", "test"], ["5abb14a55542992ccd8e7f06", "test"], ["5adcfb5f5542990d50227d81", "test"], ["5a738a5955429908901be2f6", "test"], ["5ab3849a55429969a97a8170", "dev"], ["5ae3c0cc5542992f92d82369", "test"], ["5a8b057255429971feec4611", "dev"], ["5ac325495542995ef918c12e", "test"], ["5a779fd95542997042120ab5", "dev"], ["5ab9bc6155429970cfb8eb90", "test"], ["5a7119f75542994082a3e58e", "test"], ["5a7a4f405542994f819ef1c0", "test"], ["5a88982155429938390d3f81", "dev"], ["5a7a11c85542996a35c170b4", "test"], ["5a90206155429933b8a20481", "test"], ["5a87b93c5542996e4f3088ec", "test"], ["5a84e0a45542991dd0999e11", "test"], ["5abce2fc5542993a06baf994", "dev"], ["5abf11d45542997719eab660", "test"], ["5add292b5542990d50227e0a", "test"], ["5a77174f55429966f1a36c4c", "dev"], ["5ab5ceb65542997d4ad1f1e2", "test"], ["5ab96ab755429970cfb8eacd", "dev"], ["5a8cb288554299585d9e3726", "test"], ["5ae537a855429960a22e0266", "test"], ["5ae46b615542995dadf24327", "dev"], ["5a7360235542991f29ee2dc7", "dev"], ["5a7133fa5542994082a3e660", "dev"], ["5a8986fd55429938390d4046", "test"], ["5a8a5c0d554299515336137b", "test"], ["5a7c66d555429935c91b518f", "test"], ["5ab7ef0f5542991d322237ca", "dev"], ["5abc3c545542993a06baf89f", "test"], ["5aba6f3e55429955dce3ee21", "test"], ["5ab92f1f554299753720f77b", "test"], ["5ae3a1aa5542994393b9e72d", "test"], ["5ab602405542992aa134a401", "test"], ["5a812cd555429926c1cdacfc", "test"], ["5ac5275755429924173fb617", "dev"], ["5a801e845542992097ad2fd9", "dev"], ["5a8763a25542993e715abf1f", "test"], ["5ae213de5542997283cd23b9", "test"], ["5a733b835542991f9a20c6b2", "dev"], ["5a8d9573554299068b959d51", "test"], ["5ab8261655429934fafe6cf1", "dev"], ["5addffff5542992200553bdd", "test"], ["5a7c76275542990527d554b4", "test"], ["5adf3ab95542992d7e9f92f3", "dev"], ["5a7ba11d554299294a54aa39", "dev"], ["5a845e9055429933447460ec", "test"], ["5ab288a05542993be8fa993b", "test"], ["5ae72ab25542991e8301cb77", "test"], ["5ae39fb85542990afbd1e197", "test"], ["5ae527085542993aec5ec165", "test"], ["5a80a1b25542992bc0c4a78c", "test"], ["5a8463545542990548d0b2a3", "test"], ["5ab281f45542993be8fa9938", "dev"], ["5abe535055429976d4830aea", "dev"], ["5a8220db5542995ce29dccc1", "test"], ["5a8badef554299240d9c2080", "test"], ["5ae0ead25542990adbacf6c8", "dev"], ["5a83172455429954d2e2ec14", "dev"], ["5abea7c25542990832d3a068", "test"], ["5a866fed554299211dda2b2e", "test"], ["5ab29956554299194fa9342d", "test"], ["5a7fb90655429969796c1b34", "test"], ["5ac2e9c45542990b17b154a0", "dev"], ["5a8b770f5542995d1e6f13a7", "test"], ["5ab28a87554299449642c8ec", "test"], ["5a7380e055429908901be2d6", "test"], ["5a89007355429938390d4026", "test"], ["5a8f44135542992414482a24", "test"], ["5ab527205542996a3a96a03d", "test"], ["5a776a805542993735360269", "test"], ["5a7615fb5542992db9473727", "test"], ["5ae0156455429942ec259c15", "dev"], ["5a745a6055429974ef308be4", "dev"], ["5abfa18f5542990832d3a176", "test"], ["5a7125b95542994082a3e5d5", "test"], ["5ac3bff15542995ef918c213", "dev"], ["5ae0f73d55429920d5234252", "dev"], ["5a8841195542994846c1ce65", "test"], ["5a7efa7a55429930675135ea", "dev"], ["5ab4313a554299233955004c", "test"], ["5ab68a71554299710c8d1ec2", "test"], ["5abbe4c0554299642a094bc0", "test"], ["5ae76c235542991e8301cc97", "test"], ["5a8a484f5542996c9b8d5e94", "test"], ["5a7587705542992db9473674", "dev"], ["5a86ed6155429960ec39b6d9", "dev"], ["5ab9d44255429939ce03dc36", "test"], ["5ab2e3c45542991669774126", "dev"], ["5ae73d235542991e8301cc29", "dev"], ["5a7bbcd65542997c3ec97268", "dev"], ["5ae5faf555429929b0807a4c", "dev"], ["5a74ee175542996c70cfae41", "test"], ["5a794119554299029c4b5f3c", "dev"], ["5a8ce15d554299653c1aa12f", "test"], ["5ae395d85542992f92d82313", "dev"], ["5a7c6ed3554299683c1c6306", "test"], ["5a80be9655429938b61421b5", "test"], ["5a88651b55429938390d3f45", "test"], ["5a87ee7b55429938390d3eb6", "test"], ["5abff0645542994516f45539", "test"], ["5a8e8f6f5542995085b37451", "test"], ["5a870a8e55429960ec39b707", "test"], ["5a734ca85542994cef4bc51b", "dev"], ["5a8b4a5f55429949d91db531", "test"], ["5a7a633a55429941d65f2609", "test"], ["5a77d65055429949eeb29f7b", "test"], ["5a8c52685542995e66a475bb", "dev"], ["5ae786c8554299540e5a55d7", "test"], ["5ab3bcc55542992ade7c6e66", "test"], ["5abe19895542991f661060c2", "test"], ["5abde4595542991f66106095", "test"], ["5ab2d9f1554299545a2cfac9", "test"], ["5a7412b655429979e28828a1", "test"], ["5a8b78775542997f31a41d3d", "test"], ["5aba7fd055429955dce3ee5e", "test"], ["5a87572d5542994846c1cd6b", "dev"], ["5a7a73e45542996c55b2ddb6", "test"], ["5ae267a35542992decbdccee", "test"], ["5a74af7e55429916b01641d2", "test"], ["5a89d36e554299515336132a", "test"], ["5a7cefcf55429909bec768af", "test"], ["5abb342055429966062416e8", "test"], ["5adf49935542993a75d26491", "test"], ["5a7a46605542994f819ef1ad", "test"], ["5a712c4b5542994082a3e622", "test"], ["5a835478554299123d8c20ed", "test"], ["5adc459255429944faac24ad", "dev"], ["5a7bd791554299683c1c62c2", "dev"], ["5a77857155429949eeb29ebf", "test"], ["5ab678f455429954757d32d3", "dev"], ["5a8d0e305542994ba4e3dbbe", "test"], ["5a7630105542992d0ec0605a", "dev"], ["5a8b2dc055429950cd6afc62", "test"], ["5a84af3e5542991dd0999d7c", "dev"], ["5ae23d945542996483e64939", "test"], ["5ae377c45542990afbd1e16b", "test"], ["5a808c215542995d8a8ddf90", "test"], ["5a74702555429929fddd8428", "dev"], ["5ade25ed5542997c77aded70", "test"], ["5a825b0e55429954d2e2eb0c", "test"], ["5ac4f33355429924173fb507", "test"], ["5ae3e59a5542994393b9e79e", "test"], ["5a8268aa55429940e5e1a8aa", "test"], ["5ab950bd55429970cfb8ea4c", "test"], ["5ae18d615542997283cd2229", "dev"], ["5ae07e5b55429945ae959381", "test"], ["5ab3acc5554299233954ff7e", "test"], ["5a7f451a55429969796c19ec", "test"], ["5abd36165542993062266bfb", "test"], ["5a8f03855542997ba9cb3186", "test"], ["5ac17c865542994ab5c67d6a", "dev"], ["5a7507455542993748c8979d", "test"], ["5ae4a0ca5542995ad6573dcb", "test"], ["5abb8aaf5542993f40c73b1c", "dev"], ["5a74e5aa5542996c70cfae2e", "dev"], ["5a79c9395542994bb94570a2", "test"], ["5ac0d5a1554299294b219025", "test"], ["5a7b220855429931da12c9e5", "dev"], ["5aba768c55429901930fa7e0", "test"], ["5ae74eeb5542991bbc9761e1", "test"], ["5ae31bbd55429928c42395fc", "dev"], ["5a8d08ee55429941ae14df70", "test"], ["5aba74ad55429955dce3ee35", "dev"], ["5a8f25ba55429924144829e9", "dev"], ["5a8796345542994846c1cda3", "test"], ["5a7bc5f255429927d897c03e", "test"], ["5abb4b4a5542993f40c73aea", "test"], ["5a877df15542993e715abf7a", "test"], ["5a8485fa5542992a431d1a41", "test"], ["5a776ac75542993569682d9b", "dev"], ["5a8d40585542994ba4e3dc39", "test"], ["5a74e6e855429974ef308c8e", "test"], ["5a732ab25542991f29ee2d25", "dev"], ["5a7cf4e355429907fabef062", "test"], ["5ade9545554299728e26c740", "test"], ["5a7252db5542990c210a4104", "test"], ["5a78f59755429970f5fffdf8", "dev"], ["5ab984a2554299131ca4235a", "test"], ["5ade0cc85542990dbb2f7f37", "test"], ["5a85018d5542997b5ce3ff98", "test"], ["5a778b345542992a6e59decf", "test"], ["5a8a521d5542992d82986f15", "test"], ["5ae2cbc9554299495565db36", "test"], ["5ac23ff0554299636651994d", "test"], ["5ab28019554299722f9b4d51", "test"], ["5a7ca98f55429935c91b5288", "dev"], ["5ac19f405542991316484b5b", "test"], ["5ae454355542996836b02c77", "test"], ["5adbd60855429944faac2377", "dev"], ["5ade0bc55542997545bbbe30", "dev"], ["5add3acb5542990dbb2f7ddc", "test"], ["5a8701b3554299211dda2b9a", "test"], ["5ae063085542993d6555eb37", "test"], ["5a73ae3b55429978a71e907c", "dev"], ["5a79178d554299029c4b5ef9", "dev"], ["5a77ace855429967ab105223", "test"], ["5a886b7555429938390d3f4d", "test"], ["5a8fb3af5542997ba9cb32ee", "dev"], ["5a8e7c095542990e94052b22", "dev"], ["5ab5de535542992aa134a3b6", "test"], ["5a8ec6fd55429917b4a5bdc1", "dev"], ["5a84cb255542991dd0999dfa", "dev"], ["5a7c00a55542996dd594b87e", "dev"], ["5ae17e4255429901ffe4aeae", "dev"], ["5a7bd25a554299294a54ab13", "test"], ["5ab4341c55429942dd415ede", "test"], ["5a87ab9b5542996e4f3088c2", "dev"], ["5a71224f5542994082a3e5c1", "dev"], ["5ab9ba6e554299743d22ebcd", "test"], ["5a7324205542994cef4bc478", "test"], ["5aba637a55429939ce03dcbb", "test"], ["5ae849cc5542997ec2727798", "test"], ["5a85bc2d5542994c784ddb70", "dev"], ["5a8a410655429970aeb70279", "dev"], ["5a8192045542995ce29dcc04", "test"], ["5a8341c255429954d2e2ec6e", "dev"], ["5ae5305b5542990ba0bbb1f5", "test"], ["5ae54c085542992663a4f1c4", "test"], ["5a8d7edc55429941ae14dfdc", "test"], ["5a90b43455429916514e7523", "dev"], ["5a7c703055429935c91b51aa", "test"], ["5ae47fe255429913cc204482", "test"], ["5ae0616255429924de1b70ca", "dev"], ["5ae234e85542994d89d5b395", "dev"], ["5ab6bf6d55429953192ad372", "test"], ["5a9042825542990a984935d6", "test"], ["5a848b5c5542997175ce1ef2", "test"], ["5ae69cfe55429908198fa65f", "test"], ["5ab529965542990594ba9d1e", "test"], ["5abd85375542993062266cba", "dev"], ["5aba94e355429901930fa845", "test"], ["5ab553e2554299494045ef81", "dev"], ["5a7f39d055429934daa2fd2a", "test"], ["5ae0a2c75542993d6555ebcf", "test"], ["5a8d8e4f5542994ba4e3dca7", "test"], ["5addffa75542992200553bdc", "test"], ["5ab632aa55429953192ad296", "test"], ["5ae406f055429970de88d874", "test"], ["5a7f00b455429930675135f1", "test"], ["5a7323ef5542994cef4bc477", "test"], ["5abd2eab5542992ac4f38210", "test"], ["5a7351785542994cef4bc539", "test"], ["5ab6a261554299710c8d1f07", "test"], ["5ae377415542992e3233c40a", "test"], ["5ab8179f5542990e739ec817", "test"], ["5a7dedef5542995f4f40231a", "test"], ["5a7a43f25542996a35c1716c", "test"], ["5a78c56455429974737f7876", "test"], ["5a7133565542994082a3e65c", "test"], ["5abe731255429965af743f0e", "test"], ["5a7ab51055429927d897bef6", "test"], ["5a884adb5542996e4f308938", "test"], ["5a77372b554299373536021c", "test"], ["5ab7f0b9554299366779405d", "dev"], ["5abd0f4b55429933744ab6e3", "test"], ["5ab9631a55429970cfb8eaad", "test"], ["5ade7fa3554299728e26c727", "test"], ["5ae6a5d15542991bbc97615d", "test"], ["5a7518cb55429916b0164259", "test"], ["5a8091515542996402f6a563", "test"], ["5ae74818554299572ea547ad", "test"], ["5adf8c965542993344016cf5", "dev"], ["5ac3a0db554299391541383a", "test"], ["5adcd2435542992c1e3a241b", "test"], ["5ab29bdc554299545a2cf9d1", "test"], ["5a73073d55429901807daf69", "test"], ["5aba5d2e55429901930fa799", "test"], ["5a72a47d55429901807daf1f", "test"], ["5ae12bcc5542997b2ef7d11a", "test"], ["5adc3a7055429947ff173913", "test"], ["5a7b76225542997c3ec9719c", "test"], ["5a877fa45542993e715abf84", "test"], ["5ae4a1875542995ad6573dd2", "test"], ["5ac173295542994ab5c67d45", "dev"], ["5a7c4a6455429907fabeef53", "test"], ["5ac3a928554299391541389f", "test"], ["5a74fd6d5542993748c8977d", "test"], ["5adef5fb5542993a75d263be", "test"], ["5ab816b05542990e739ec815", "test"], ["5ae666d35542991bbc9760cc", "dev"], ["5adfc50355429942ec259b43", "test"], ["5a84524a554299123d8c2203", "dev"], ["5a81778355429926c1cdad2e", "test"], ["5ab6e865554299710c8d1fad", "test"], ["5a72a0be5542992359bc3143", "dev"], ["5ac036b95542992a796deccd", "test"], ["5a77a7695542997042120ac4", "dev"], ["5a8314bd55429966c78a6b2a", "test"], ["5a88f9555542995153361214", "dev"], ["5aba18ee5542994dbf0198b4", "dev"], ["5ab8420555429919ba4e2290", "test"], ["5a7166395542994082a3e814", "dev"], ["5abd12bb55429933744ab703", "dev"], ["5ab430fa5542991751b4d6dd", "test"], ["5ae0380955429925eb1afc2e", "test"], ["5a8ae61955429970aeb70326", "dev"], ["5ac3ba0455429939154138f2", "test"], ["5a80e5f5554299260e20a1ae", "dev"], ["5ae4f3c455429960a22e0221", "test"], ["5a74c85055429916b0164218", "test"], ["5ab98d70554299753720f82f", "test"], ["5add3a605542990dbb2f7ddb", "test"], ["5ab295eb554299194fa93412", "test"], ["5ae7821b5542997ec2727678", "dev"], ["5ab8829d55429934fafe6e08", "test"], ["5adf51e05542995534e8c78a", "test"], ["5a721d5a55429971e9dc9283", "test"], ["5a77e70f5542992a6e59dfeb", "test"], ["5a8ce851554299441c6b9f6c", "dev"], ["5ae63b2255429908198fa526", "test"], ["5ab587f85542992aa134a34d", "dev"], ["5ae33cf45542990afbd1e0ea", "test"], ["5a8b84905542997f31a41d5d", "dev"], ["5ae406245542995dadf242e2", "test"], ["5a7ae2f2554299042af8f6aa", "dev"], ["5adc3d7455429944faac248a", "dev"], ["5a8b2ccf55429971feec4665", "test"], ["5ade4e415542992fa25da73f", "test"], ["5a81937e5542995ce29dcc08", "test"], ["5ac4f9dc55429924173fb533", "dev"], ["5a8637415542994775f60724", "test"], ["5abe8aad55429976d4830b60", "dev"], ["5a77a14655429949eeb29f13", "test"], ["5a843f305542996488c2e51d", "dev"], ["5addeb685542992200553bb1", "test"], ["5a7ae38c5542992d025e6721", "dev"], ["5ae5366e5542990ba0bbb22b", "test"], ["5ade8bd955429975fa854f04", "dev"], ["5a7de63c5542995f4f4022f5", "test"], ["5abefa5b5542997ec76fd3a3", "dev"], ["5a78d38455429970f5fffd94", "test"], ["5a75357c5542996c70cfaeab", "dev"], ["5abb7ff15542996cc5e4a015", "dev"], ["5ac2f34455429921a00ab0b6", "test"], ["5a7b60605542995eb53be931", "test"], ["5ac0714f554299294b218fe1", "test"], ["5adf2ba05542993a75d26411", "dev"], ["5ab8c3c65542991b5579effd", "test"], ["5a72f45a5542992359bc31e6", "test"], ["5ae1091055429920d523429b", "test"], ["5ab53436554299637185c504", "test"], ["5ae7922f5542994a481bbdac", "test"], ["5a8e024f5542995085b37375", "dev"], ["5abd848455429924427fd03a", "test"], ["5ae328f45542991a06ce993c", "test"], ["5abd0ee65542996e802b46d7", "dev"], ["5ab804305542995dae37ea14", "dev"], ["5ae274dd5542996483e649ad", "test"], ["5abd1b6e55429933744ab729", "dev"], ["5ab8f7535542991b5579f0a7", "test"], ["5a9092cc55429916514e74fd", "test"], ["5a7602d8554299109176e5fb", "dev"], ["5ab3e45655429976abd1bcd4", "dev"], ["5ab9cfb455429970cfb8ebae", "test"], ["5a8601f855429960ec39b5fd", "test"], ["5a8ac87d55429970aeb7031d", "test"], ["5a8dd5f955429917b4a5bcc7", "test"], ["5ac55c135542993e66e82345", "dev"], ["5a7a3bed5542996c55b2dd45", "test"], ["5a7fd95d5542992e7d278dc5", "dev"], ["5adc6f6255429947ff173961", "test"], ["5a89fea655429970aeb701eb", "dev"], ["5a8d841955429941ae14dfe6", "test"], ["5abda2b755429933744ab80f", "test"], ["5a771d6155429966f1a36c6d", "test"], ["5ae09ce855429924de1b7113", "dev"], ["5ab672165542995eadeeffe0", "test"], ["5a9094ba55429933b8a2056d", "dev"], ["5a879ba55542993e715abfc3", "dev"], ["5ae326d85542991a06ce9938", "dev"], ["5ae17a9b5542990adbacf7a2", "test"], ["5a72f74a55429901807daf59", "dev"], ["5addf0fd5542997dc79070d2", "test"], ["5ab4234f5542991751b4d685", "test"], ["5a8ec02e5542995a26add501", "test"], ["5a90580655429933b8a204ef", "test"], ["5ab69e4f554299710c8d1eee", "test"], ["5ae7db435542993210983f9a", "test"], ["5adfb78f5542995ec70e9081", "test"], ["5a7574e25542996c70cfaf01", "dev"], ["5a7732dc55429972597f149b", "test"], ["5a83870f554299123d8c2149", "dev"], ["5a80a654554299485f598679", "test"], ["5a711c535542994082a3e59b", "test"], ["5a7b8f66554299294a54a9f7", "test"], ["5aba67c255429955dce3ee10", "dev"], ["5a89e28f5542992e4fca843e", "dev"], ["5a8799c55542996e4f308877", "test"], ["5aba480c55429939ce03dc77", "test"], ["5a80297f5542992e7d278e08", "dev"], ["5ab2b7c1554299166977406a", "test"], ["5abc6c2655429959677d6aac", "test"], ["5ae8096f5542994a481bbe99", "dev"], ["5a7184675542994082a3e864", "dev"], ["5ae665795542991bbc9760c9", "test"], ["5ac250aa5542992f1f2b384d", "dev"], ["5a777fd655429949eeb29e93", "test"], ["5adfb698554299603e418385", "test"], ["5a7982435542994bb945701f", "test"], ["5ac3def75542997ea680c95f", "test"], ["5a7a68d25542996c55b2dd98", "dev"], ["5a75fa14554299109176e5dc", "test"], ["5ae6453e55429908198fa56c", "test"], ["5a8e5a635542990e94052ae4", "dev"], ["5ac249bd55429951e9e68528", "dev"], ["5a807ea1554299485f598624", "test"], ["5a88bedb554299206df2b35a", "test"], ["5a7c99705542990527d554de", "dev"], ["5a7261635542997f8278398a", "test"], ["5a7ca33f5542990527d554ee", "dev"], ["5ae75eb45542991bbc976201", "test"], ["5a7e43c9554299495941995a", "dev"], ["5a74b4dc55429979e2882a00", "test"], ["5a7521c55542993748c897c6", "test"], ["5a81db0b5542990a1d231ee2", "test"], ["5ab310bb554299233954fef9", "test"], ["5abce7d755429959677d6b3a", "test"], ["5a7ddfc55542995f4f4022e9", "test"], ["5ac42c6d5542997ea680ca1a", "test"], ["5ae0bb2c5542993d6555ec27", "dev"], ["5adda7585542997dc790700c", "test"], ["5a90485755429933b8a204d2", "test"], ["5add39e75542997545bbbcc4", "test"], ["5ae532365542992663a4f150", "dev"], ["5ae2098e5542997f29b3c21b", "test"], ["5ab6f0c25542995eadef00ea", "test"], ["5ae497f15542995ad6573db8", "test"], ["5ae7473f5542991bbc9761d2", "dev"], ["5a793bc95542994bb9456fc3", "test"], ["5a7503945542996c70cfae76", "test"], ["5a8b595855429949d91db563", "dev"], ["5ac18baa5542991316484b33", "test"], ["5a74d9c555429929fddd84fb", "test"], ["5ab94acd554299743d22eac6", "dev"], ["5ae363ff5542992f92d822bc", "test"], ["5ab95dd8554299743d22eb13", "test"], ["5ae1389655429901ffe4ae05", "test"], ["5ae150ab55429901ffe4ae53", "dev"], ["5adfb992554299603e418391", "dev"], ["5a7da4865542990b8f5039ed", "dev"], ["5a778e8755429949eeb29ef4", "dev"], ["5add74b75542992200553b24", "test"], ["5ae113de55429901ffe4ad51", "dev"], ["5ab29346554299545a2cf997", "dev"], ["5a8832d355429938390d3f03", "test"], ["5ab2342a5542993be8fa98c3", "dev"], ["5a8771595542993e715abf46", "test"], ["5a8319b255429940e5e1a960", "test"], ["5a849c135542992a431d1a7a", "dev"], ["5a86141f5542996432c571a5", "test"], ["5ae28c0a554299495565daaa", "test"], ["5abd8ee05542993062266cc8", "dev"], ["5ade04d75542997dc7907125", "dev"], ["5abbdf8055429931dba145bb", "test"], ["5ab1c693554299722f9b4c66", "test"], ["5a862c38554299211dda2a9a", "dev"], ["5a85fe2f5542996432c57165", "test"], ["5adf73a45542992d7e9f9378", "test"], ["5ade09945542997545bbbe2c", "dev"], ["5ae7376f5542992ae0d163d2", "test"], ["5a77c1505542997042120b1b", "dev"], ["5ae826675542997ec2727733", "test"], ["5a7759dd5542994aec3b729f", "dev"], ["5a7caae75542996dd594b971", "test"], ["5a7471f655429979e2882955", "test"], ["5a7a01b15542996c55b2dcd1", "test"], ["5adfd56655429925eb1afaac", "dev"], ["5a83d0ed5542996488c2e4e7", "dev"], ["5a7bd286554299683c1c62bd", "dev"], ["5ae2c416554299492dc91c67", "test"], ["5a80ddc555429938b6142224", "test"], ["5ac1bce45542994ab5c67def", "test"], ["5a8835dc5542994846c1ce2b", "test"], ["5a77e2095542995d8318131d", "test"], ["5a78c7db55429974737f7882", "test"], ["5ab7a1f65542991d3222378f", "dev"], ["5ab3b7c555429969a97a8190", "test"], ["5abf877c5542990832d3a147", "test"], ["5a78d35e55429974737f78c3", "dev"], ["5a74639a55429929fddd8404", "test"], ["5add61d65542995b365fab21", "test"], ["5abd648155429933744ab7ac", "test"], ["5abb30755542996cc5e49fd8", "test"], ["5ab1f13e554299449642c81c", "dev"], ["5a7b1e22554299042af8f6e4", "dev"], ["5addcb215542997dc7907033", "test"], ["5ab6644655429954757d327d", "test"], ["5a81f49c5542995ce29dcc9a", "dev"], ["5ab66e1355429954757d32b2", "test"], ["5a8a3e745542996c9b8d5e70", "test"], ["5a8c72545542995e66a475f3", "test"], ["5a7a3dbb5542996a35c17157", "test"], ["5a710a1e5542994082a3e4fd", "test"], ["5adde73f5542992200553b94", "test"], ["5ab7268b5542992aa3b8c7b1", "test"], ["5a77731955429972597f1541", "test"], ["5add016e5542992c1e3a250b", "test"], ["5a78a7025542990784727710", "test"], ["5ae0ba1155429924de1b7156", "test"], ["5ae2120c5542997283cd23ac", "test"], ["5ae357745542992f92d8229b", "test"], ["5a84de5f5542991dd0999e07", "test"], ["5ab41f6b5542991751b4d671", "test"], ["5a724ba25542990c210a40c7", "dev"], ["5ab225565542993be8fa98bf", "test"], ["5a7b413a5542992d025e67c4", "test"], ["5add74b95542992ae4cec56f", "dev"], ["5a70f9335542994082a3e46a", "test"], ["5ac274fc55429963665199c1", "dev"], ["5ab9560855429970cfb8ea73", "test"], ["5a7b4d765542995eb53be8fe", "dev"], ["5a83155c55429954d2e2ec0e", "test"], ["5a8b4ec155429949d91db53f", "test"], ["5a7c04c85542996dd594b881", "test"], ["5ab29c24554299449642c932", "test"], ["5ab2f85b55429929539468dc", "test"], ["5a728f015542991f9a20c4e4", "test"], ["5a8bbfb15542995e66a474f6", "test"], ["5a8c620c554299240d9c2141", "test"], ["5a8a09765542992d82986e67", "dev"], ["5adfcb5455429906c02daa48", "test"], ["5a8b5b1c5542995d1e6f1387", "test"], ["5adf9b415542995ec70e904c", "test"], ["5ae72c1e554299572ea54754", "test"], ["5a78f59755429970f5fffdf7", "test"], ["5a7bb983554299042af8f7c1", "test"], ["5adf6a5c5542995ec70e8ff9", "test"], ["5ae4075b55429970de88d87a", "test"], ["5abca1a55542993a06baf937", "test"], ["5a7d1c525542995ed0d165f2", "dev"], ["5ac25fa955429951e9e6858c", "test"], ["5ab29624554299545a2cf9b5", "test"], ["5a75383a5542996c70cfaeb3", "test"], ["5ae2138c5542997283cd23b7", "test"], ["5abf92c45542993fe9a41e07", "dev"], ["5ab2be12554299295394686b", "dev"], ["5ae3fd595542995dadf24292", "test"], ["5a7a2be25542996a35c17118", "dev"], ["5ab7484c5542992aa3b8c80d", "dev"], ["5a9052765542995651fb513d", "test"], ["5abbd3b4554299642a094b4f", "dev"], ["5a802ff25542992bc0c4a6d2", "test"], ["5a7c572655429935c91b5166", "dev"], ["5ae515bb55429913cc204584", "dev"], ["5a8d9f35554299441c6ba018", "dev"], ["5ae226755542996483e64906", "test"], ["5a7294b65542991f9a20c509", "dev"], ["5a8f47a55542997ba9cb3219", "test"], ["5ae1ce895542997f29b3c11e", "test"], ["5a8f08c2554299458435d530", "test"], ["5a8aecff55429950cd6afbef", "test"], ["5ab29ec6554299194fa93473", "dev"], ["5a8f5b6f554299458435d5e7", "test"], ["5a867089554299211dda2b30", "test"], ["5a7736d85542994aec3b722b", "test"], ["5abbf64055429965836003bc", "test"], ["5a82f35f5542995ce29dcd14", "dev"], ["5a8f64e0554299458435d60b", "test"], ["5a7d119d5542995ed0d165d5", "test"], ["5abec66b5542997ec76fd360", "test"], ["5adfc8cf554299603e4183ad", "dev"], ["5abc17c25542993a06baf85b", "test"], ["5ae402085542995dadf242ca", "dev"], ["5ae163b3554299422ee99678", "dev"], ["5ab52a255542996a3a96a048", "test"], ["5ae638865542992ae0d1628c", "test"], ["5a78c66e55429974737f787b", "dev"], ["5aba92da55429901930fa840", "dev"], ["5adfbd4655429942ec259b31", "test"], ["5ab8ee805542991b5579f051", "test"], ["5ade64b655429939a52fe89a", "dev"], ["5ab44cd555429942dd415f10", "test"], ["5adf744d5542992d7e9f937e", "dev"], ["5a7b735c5542997c3ec9718a", "dev"], ["5ae13c5155429920d5234336", "test"], ["5abc0dfe55429959677d6a05", "test"], ["5a74122f55429929fddd83da", "test"], ["5abbf43d5542993f40c73c2f", "test"], ["5ae738f75542991bbc9761c4", "test"], ["5a7eb22655429930675135cb", "test"], ["5a7306a655429901807daf67", "test"], ["5a82bc7455429940e5e1a926", "test"], ["5ab2da99554299194fa93544", "test"], ["5a74551555429974ef308bd9", "test"], ["5ab801da5542990e739ec7c9", "dev"], ["5a8545ba5542997b5ce3ffdd", "test"], ["5ab28156554299449642c8ce", "test"], ["5a73d3ea5542992d56e7e3af", "test"], ["5ae1054055429920d5234284", "test"], ["5a8b6a5155429949d91db5a3", "dev"], ["5ae53fcb55429960a22e029e", "dev"], ["5adf4da15542995534e8c780", "dev"], ["5ab506a05542990594ba9cd6", "dev"], ["5a8f5bb45542992414482a77", "dev"], ["5ac116595542992a796dedd1", "dev"], ["5a7ca0ef55429907fabeefd3", "test"], ["5ab38cdd5542992ade7c6de2", "dev"], ["5ae0cf8355429945ae959464", "test"], ["5a7ff71a5542992e7d278dd0", "test"], ["5ab2b6fb5542992953946848", "test"], ["5adfec5f55429942ec259b8d", "test"], ["5abcf70b55429959677d6b7e", "test"], ["5a8f05b1554299458435d517", "test"], ["5a753ff455429916b01642bc", "test"], ["5a8ed87955429917b4a5bdd2", "test"], ["5a8f812755429918e830d245", "test"], ["5a810221554299260e20a1f9", "dev"], ["5ae256c35542994d89d5b3e9", "dev"], ["5ac46cb85542991943173987", "test"], ["5adf29f05542993344016c09", "dev"], ["5a7f113d55429934daa2fcbe", "test"], ["5abccda855429965836004a4", "test"], ["5ab262e2554299722f9b4cfd", "test"], ["5a7fc6e45542992e7d278d73", "dev"], ["5a738b855542992d56e7e363", "test"], ["5a736e0155429901807db04f", "test"], ["5add2b435542990d50227e11", "dev"], ["5a7268235542992359bc3081", "test"], ["5ab4071955429976abd1bd38", "test"], ["5ab98e2b5542996be202054e", "test"], ["5ade74285542992fa25da78c", "dev"], ["5ae834e95542997ec272776a", "test"], ["5ac0aab35542992a796ded5c", "test"], ["5ae5f82a554299546bf82feb", "test"], ["5a8f75e6554299458435d635", "test"], ["5ab84bab55429934fafe6d49", "dev"], ["5a7aa0a55542990198eaf165", "test"], ["5ab46d195542991751b4d76a", "test"], ["5a77671255429966f1a36d21", "test"], ["5ab78a445542992aa3b8c826", "test"], ["5ab721325542992aa3b8c7a2", "test"], ["5abe3f3455429976d4830aa8", "test"], ["5ac2509755429951e9e68543", "dev"], ["5ae47df15542996836b02cba", "dev"], ["5a773c7755429972597f14b5", "test"], ["5aba7d7d554299232ef4a303", "dev"], ["5ab8ee035542991b5579f04b", "dev"], ["5adcc5715542990d50227d0a", "test"], ["5ab870ed55429934fafe6db9", "dev"], ["5a7222b155429971e9dc92c4", "dev"], ["5a7568175542992d0ec05f88", "test"], ["5addda1b5542992200553b55", "dev"], ["5ae20e265542994d89d5b325", "test"], ["5ab71f7d554299110f219ab9", "dev"], ["5ac2cb125542996773102626", "test"], ["5a7dca645542990b8f503aab", "test"], ["5a828a7b55429954d2e2eb69", "dev"], ["5ade6acf554299728e26c71a", "test"], ["5a79b495554299148911fa3b", "test"], ["5a730a6b5542992359bc3210", "test"], ["5ae37e6e5542992f92d822e0", "dev"], ["5ae7316e554299572ea5477e", "dev"], ["5abc3f505542993a06baf8b2", "test"], ["5a7fb6625542994857a767cc", "dev"], ["5ab56f7a554299637185c59a", "dev"], ["5ab680d15542995eadef0011", "dev"], ["5ae69fa755429908198fa66c", "test"], ["5a90984f55429933b8a20578", "test"], ["5a73b8d655429908901be31f", "test"], ["5a7fa55555429969796c1b0e", "dev"], ["5a85d25b5542996432c570ee", "test"], ["5adec62e5542992fa25da841", "test"], ["5ab808ea5542991d3222380e", "dev"], ["5ab9b7d555429970cfb8eb7a", "test"], ["5a7bbee85542997c3ec97270", "test"], ["5a7bb2595542995eb53be99b", "test"], ["5a7c6753554299683c1c62f9", "dev"], ["5ae1f2f3554299234fd0435a", "test"], ["5ade450b5542997c77adedc5", "dev"], ["5a7270395542992359bc30a8", "dev"], ["5a7dd6cd5542995f4f4022cd", "dev"], ["5ae780bf554299540e5a55d0", "test"], ["5a75f79555429976ec32bcca", "test"], ["5a8753d95542994846c1cd63", "dev"], ["5a7d1765554299452d57bade", "test"], ["5adcfbf45542990d50227d86", "dev"], ["5ab627d555429953192ad27e", "test"], ["5a7f3c8f55429930675136c9", "test"], ["5ac4f04e5542995c82c4ad81", "test"], ["5ab7bae95542993667793ff4", "test"], ["5a7109e95542994082a3e4fa", "test"], ["5abffa095542990832d3a1de", "dev"], ["5ade15f35542997545bbbe47", "test"], ["5a835a7b554299123d8c20fc", "dev"], ["5a760ab65542994ccc918697", "test"], ["5ab5a3fb5542997d4ad1f19b", "dev"], ["5ae1443855429920d5234362", "dev"], ["5a79cd665542994bb94570bf", "dev"], ["5a75e33855429976ec32bc74", "dev"], ["5ade48795542992fa25da729", "test"], ["5ae720e9554299572ea54715", "test"], ["5a75eb125542994ccc918659", "test"], ["5ae5a8435542992663a4f208", "dev"], ["5ab9c7c355429939ce03dc17", "test"], ["5add16b65542990d50227dd2", "dev"], ["5a7df7115542990b8f503b0f", "test"], ["5a7f1d3c5542993067513639", "test"], ["5abc8d75554299700f9d7900", "dev"], ["5a75377c55429929fddd8514", "dev"], ["5ae5086d55429908b63264eb", "dev"], ["5ae24abc5542994d89d5b3c3", "dev"], ["5a8e07185542995085b37389", "dev"], ["5ac1c4e85542994ab5c67e0c", "dev"], ["5a7fd7305542992e7d278dc2", "dev"], ["5ade3af855429939a52fe867", "test"], ["5ab439f055429942dd415eef", "dev"], ["5adbfb3f55429947ff17388a", "dev"], ["5ae351405542994393b9e67f", "test"], ["5a7b69b155429927d897bfbc", "test"], ["5ab42b24554299753aec5a43", "dev"], ["5ae26b5f5542992decbdccf8", "test"], ["5abfaa445542990832d3a18c", "dev"], ["5ac2a277554299657fa28ff3", "test"], ["5ae0aaf35542993d6555ec02", "test"], ["5a7dacff5542991319bc93e0", "test"], ["5a7b47765542992d025e67d8", "test"], ["5a829fab55429954d2e2eb83", "test"], ["5abd957a5542996e802b47eb", "test"], ["5ab97b27554299131ca4233c", "dev"], ["5ae1ab2f554299234fd042ba", "test"], ["5add48465542992200553a80", "test"], ["5addda9b5542992200553b5b", "dev"], ["5ae1e22a5542997f29b3c160", "test"], ["5abd1b5e55429933744ab728", "test"], ["5add1d435542990d50227de8", "test"], ["5ae6d3345542996d980e7c85", "test"], ["5ab841d15542990e739ec89d", "test"], ["5ade5e335542992fa25da766", "test"], ["5ae7da705542993210983f96", "test"], ["5ae306115542992decbdcdc4", "test"], ["5a7d1e685542995ed0d165f7", "test"], ["5a8f541e5542992414482a53", "dev"], ["5ac15f6c5542994d76dccdf3", "test"], ["5a9087ac5542990a98493668", "dev"], ["5a832ad55542990548d0b1b5", "test"], ["5ae136375542997b2ef7d12f", "test"], ["5a887771554299206df2b294", "dev"], ["5a8662da5542991e7718162f", "test"], ["5ab43b9d5542990594ba9bc3", "test"], ["5ae0d52e554299603e418469", "test"], ["5adfc97255429906c02daa41", "test"], ["5a713ea95542994082a3e6e4", "test"], ["5a78f3d255429970f5fffdf1", "test"], ["5ae0847455429945ae959394", "dev"], ["5ab3c270554299233954ff9e", "test"], ["5ab4ec2755429942dd415fa6", "test"], ["5a8210f355429926c1cdae24", "dev"], ["5ae55b8255429908b63265ef", "test"], ["5ac5461d5542993e66e82296", "dev"], ["5a7e4414554299495941995c", "test"], ["5ae60426554299546bf83019", "test"], ["5a72944c5542994cef4bc351", "test"], ["5ade3bf75542997c77adedb8", "test"], ["5ae0c7b055429945ae95944b", "test"], ["5ab5627b5542992aa134a2ff", "test"], ["5a80b4d1554299260e20a134", "dev"], ["5ae603ba5542996de7b71af4", "test"], ["5a85ac1e5542997b5ce4001f", "dev"], ["5ac00f795542996f0d89cb16", "test"], ["5abaa954554299232ef4a362", "dev"], ["5add67555542992ae4cec54e", "dev"], ["5abfd3d65542994516f4550b", "dev"], ["5a70f1685542994082a3e40f", "dev"], ["5ac46c83554299204fd21f45", "dev"], ["5ae14cf05542990adbacf742", "test"], ["5abda6b255429965af743d9a", "test"], ["5a7a5ea15542994f819ef1c7", "test"], ["5ab250b9554299722f9b4cd9", "dev"], ["5a8a21a455429930ff3c0caf", "dev"], ["5ae2eb7f55429928c423955b", "test"], ["5ab3b0bf5542992ade7c6e39", "dev"], ["5a7280925542991f9a20c4a1", "test"], ["5adcd9fe5542990d50227d46", "test"], ["5a8f54215542992414482a54", "test"], ["5a84fd085542994c784ddaba", "dev"], ["5ae4ce835542990ba0bbb14d", "dev"], ["5a7b3b4f5542992d025e67ac", "dev"], ["5a735c3a55429901807daff7", "dev"], ["5a8a09565542992d82986e66", "dev"], ["5ab3bfd3554299233954ff99", "test"], ["5a731c205542994cef4bc45d", "test"], ["5a7631cd5542994ccc91870c", "dev"], ["5a76e88555429972597f13f4", "test"], ["5ae5b25d5542990ba0bbb2bb", "test"], ["5a8c8502554299585d9e36d0", "test"], ["5ae7f7d85542994a481bbe52", "test"], ["5ac16c2b5542994d76dcce19", "dev"], ["5ae0945d55429906c02daadd", "test"], ["5ac4d314554299076e296e02", "test"], ["5ae0c9dd5542993d6555ec69", "test"], ["5a7db62d5542997cc2c4747c", "dev"], ["5adf95745542995534e8c7f8", "test"], ["5abd049d5542992ac4f381a8", "test"], ["5ab6ddb655429953192ad39e", "dev"], ["5abd14c455429924427fceec", "test"], ["5ab59ffe5542997d4ad1f193", "test"], ["5abd605155429924427fcfc9", "test"], ["5ade03c65542990dbb2f7f2e", "dev"], ["5a70f39c5542994082a3e429", "test"], ["5ab587ad5542992aa134a34b", "test"], ["5ae4a2e25542996836b02cee", "test"], ["5ae73d505542991e8301cc2c", "test"], ["5ae1ab715542997f29b3c0f7", "test"], ["5ac160bb5542994d76dccdfb", "test"], ["5a8ae5d355429970aeb70325", "test"], ["5abbdaac5542992ccd8e7fc0", "test"], ["5adf37955542993a75d2642e", "test"], ["5a8b2d3255429950cd6afc60", "test"], ["5a77309d55429972597f1487", "test"], ["5a8fd9ac5542990a98493570", "test"], ["5ae083b75542993d6555eb85", "test"], ["5ae552bb55429908b63265d9", "dev"], ["5a7741775542994aec3b724f", "test"], ["5abed7d75542990832d3a0f7", "dev"], ["5a804e845542992bc0c4a6e9", "dev"], ["5a8af11a55429950cd6afc21", "test"], ["5a7c68575542996dd594b91c", "test"], ["5a79f6ca554299029c4b5f90", "dev"], ["5abd455c55429924427fcf5a", "test"], ["5a7c5de155429935c91b5179", "dev"], ["5a76a0005542993569682c64", "dev"], ["5a8e8bed5542990e94052b47", "test"], ["5abde0655542991f6610608b", "test"], ["5a8720885542991e771816d8", "test"], ["5adc9b7a5542994d58a2f679", "dev"], ["5ab7de455542992aa3b8c864", "test"], ["5a864ff7554299211dda2af2", "dev"], ["5adf86355542993344016cbd", "dev"], ["5ac514d05542994611c8b372", "test"], ["5a90d25655429916514e7585", "dev"], ["5a7f66f155429969796c1a33", "test"], ["5ae531ee5542990ba0bbb1ff", "test"], ["5a7f35015542992e7d278c8a", "test"], ["5a78cc7c55429974737f789f", "test"], ["5ae49f2c5542995ad6573dc1", "dev"], ["5aba916c55429901930fa83b", "test"], ["5a8e32d35542995a26add480", "dev"], ["5a87bd4e5542996432c57279", "test"], ["5a74701655429979e2882952", "test"], ["5ae489ab5542995ad6573d62", "test"], ["5a8838885542994846c1ce39", "test"], ["5ae1dac95542997283cd229b", "test"], ["5ae141475542997b2ef7d185", "test"], ["5ab5ec2a5542992aa134a3e3", "test"], ["5a7148005542994082a3e759", "test"], ["5adc6ca85542994650320ce1", "dev"], ["5a747a3755429974ef308c09", "test"], ["5ae3df6e5542992e3233c48e", "test"], ["5ae73ddb554299572ea5478d", "dev"], ["5a7634f155429976ec32bd6b", "test"], ["5ab5f854554299637185c64e", "test"], ["5ac0b1835542992a796ded65", "test"], ["5a8838635542997e5c09a5a9", "test"], ["5a806aca5542996402f6a503", "test"], ["5ac2eec55542990b17b154c1", "test"], ["5a7a1b845542990783324e5f", "dev"], ["5abfd83f5542997ec76fd45c", "dev"], ["5ac132a755429964131be17c", "dev"], ["5adcc6765542994d58a2f6c9", "test"], ["5adbe447554299438c868ccd", "test"], ["5a89d8d055429946c8d6e9df", "test"], ["5a7172105542994082a3e837", "test"], ["5a87b30c5542994846c1cdcf", "test"], ["5ab5dc44554299494045f089", "test"], ["5a7296815542991f9a20c515", "test"], ["5ac3a0b25542993915413838", "test"], ["5a7ce50f55429909bec76869", "test"], ["5ae2075f5542994d89d5b315", "test"], ["5a8a0cb75542992d82986e7d", "test"], ["5a7380b955429908901be2d4", "dev"], ["5a87191155429960ec39b73d", "test"], ["5adf715b5542992d7e9f9354", "dev"], ["5ae3c5535542991a06ce9a20", "test"], ["5ab2a32f554299706120967b", "test"], ["5a70f4c45542994082a3e437", "test"], ["5ab266b5554299340b5254b4", "test"], ["5a7af08655429931da12c989", "dev"], ["5adc6eb65542994650320cf3", "dev"], ["5ab2383155429970612095c1", "test"], ["5a74bfa55542996c70cfadcf", "dev"], ["5a8ced46554299441c6b9f82", "test"], ["5a7c49dc55429935c91b514f", "test"], ["5a7cef5d55429909bec768ac", "test"], ["5adfc77b554299603e4183ab", "dev"], ["5a8c7ced5542995e66a4761c", "dev"], ["5ae0bae555429906c02dab08", "dev"], ["5ac562985542993e66e823ca", "dev"], ["5a8e276b5542995a26add468", "test"], ["5ae2120e5542997283cd23ad", "test"], ["5a8790785542993e715abfac", "test"], ["5ade66dd55429975fa854ebe", "dev"], ["5ab9188c5542991b5579f0f5", "dev"], ["5ab67f0055429953192ad2d8", "test"], ["5a8008b95542992097ad2fcf", "dev"], ["5a72ac8a5542992359bc3164", "test"], ["5a77beae5542992a6e59dfae", "test"], ["5a88d745554299206df2b378", "test"], ["5ae21ef35542994d89d5b35d", "test"], ["5a7525405542993748c897ce", "dev"], ["5a83316b5542993344745fe5", "test"], ["5a80de9b55429938b6142226", "test"], ["5a8612d45542996432c5719b", "test"], ["5a8bb90f5542996e8ac88a02", "test"], ["5abec12b5542997ec76fd358", "test"], ["5a8f536c5542992414482a4e", "test"], ["5ab94cca554299753720f7dd", "test"], ["5ac2680055429963665199a8", "test"], ["5ae1dadf5542997f29b3c14f", "test"], ["5a7199725542994082a3e88f", "test"], ["5a72334755429971e9dc934c", "test"], ["5adf3a0d5542992d7e9f92e8", "dev"], ["5a7efd2f5542994959419a9c", "dev"], ["5adffb4655429925eb1afbbf", "test"], ["5ae680cc5542996d980e7ba5", "dev"], ["5a75eca555429976ec32bca8", "dev"], ["5abbc3a955429931dba144f9", "test"], ["5adecc7755429975fa854f9d", "dev"], ["5adee8ff55429975fa854fb8", "test"], ["5a82e25555429954d2e2ebbf", "dev"], ["5ae199305542997b2ef7d20e", "dev"], ["5ae4ad3155429913cc2044cb", "test"], ["5a90ba1b55429933b8a205c6", "test"], ["5ae3281a5542991a06ce9939", "dev"], ["5ae1d4e8554299234fd04312", "dev"], ["5ae457a95542996836b02c85", "test"], ["5a88d7875542993b751ca876", "test"], ["5ae1c2d8554299234fd042ee", "test"], ["5a83ef745542996488c2e4f8", "test"], ["5a75e29f5542992d0ec0600c", "test"], ["5adcdf455542994ed6169c11", "test"], ["5ab27389554299722f9b4d2f", "test"], ["5ae4be5c5542995ad6573e3a", "test"], ["5ac3a364554299391541385d", "test"], ["5a81a2e25542990a1d231e32", "test"], ["5ab2da13554299194fa93542", "test"], ["5a7a1e635542990783324e69", "test"], ["5ae5390355429960a22e0271", "test"], ["5a832a625542990548d0b1b4", "dev"], ["5a78e86f55429974737f78fb", "dev"], ["5ae224da554299234fd043ee", "test"], ["5a800ddd5542992097ad2fd0", "dev"], ["5a79e1275542990198eaefe7", "test"], ["5ae5c7fd554299546bf82f5b", "test"], ["5ae6316d5542996de7b71b87", "dev"], ["5ae0d85d5542990adbacf67e", "dev"], ["5adbfc945542996e68525296", "test"], ["5a778c215542995d831811ba", "dev"], ["5ac3d7f8554299076e296cb0", "dev"], ["5a8b3e985542995d1e6f131c", "dev"], ["5a8f51185542992414482a3d", "test"], ["5a834ed95542990548d0b1ca", "test"], ["5a7ee0d05542994959419a88", "test"], ["5abea98d5542990832d3a07d", "test"], ["5ab916e155429919ba4e23a4", "test"], ["5a895f6155429946c8d6e914", "test"], ["5a7a34bf5542994f819ef151", "test"], ["5a772c925542993735360208", "dev"], ["5ae4c49a5542995dadf24400", "dev"], ["5a8a032f5542993b751ca9e5", "test"], ["5a90692f5542990a98493625", "dev"], ["5ab58f9a554299488d4d99da", "test"], ["5adca8215542994ed6169bbc", "test"], ["5ab234c65542993be8fa98c6", "test"], ["5a8e69565542990e94052afb", "dev"], ["5a8f503c5542992414482a34", "dev"], ["5a76dc2d55429972597f13eb", "test"], ["5ae6c0f35542991bbc97616f", "test"], ["5ae2c2c05542996483e64a4a", "dev"], ["5ae759b25542991e8301cc6b", "test"], ["5ab865235542992aa3b8c8db", "test"], ["5ae4b55455429913cc2044db", "test"], ["5a848eff5542990548d0b2cd", "test"], ["5a8760e75542996e4f3087af", "test"], ["5adccd795542990d50227d2c", "dev"], ["5a8d838955429941ae14dfe5", "dev"], ["5ab67c3f55429954757d32f9", "test"], ["5abb729b5542993f40c73af4", "test"], ["5ae365205542990afbd1e12d", "test"], ["5aba55f25542994dbf0198e0", "test"], ["5a7b895c5542997c3ec97206", "dev"], ["5ae331fb55429928c4239670", "test"], ["5a8b8f5b55429949d91db5e1", "test"], ["5ae444b55542996836b02c63", "test"], ["5ae172c155429920d52343de", "dev"], ["5ac13a3d5542996f0d89cc75", "dev"], ["5a721e0e55429971e9dc928d", "test"], ["5ac1c9a15542994ab5c67e1c", "dev"], ["5a8a42ec55429970aeb7027c", "test"], ["5a8bae135542996e8ac889b6", "dev"], ["5ae3d0bb5542990afbd1e1e5", "test"], ["5abd9b5255429924427fd06a", "dev"], ["5a909a0355429916514e7505", "dev"], ["5adf8f675542995534e8c7eb", "test"], ["5ab92e3c554299753720f777", "test"], ["5a8e612b5542995085b3741b", "dev"], ["5a7c762d55429935c91b51c7", "test"], ["5a78f8e555429970f5fffe01", "test"], ["5ac3c1805542995ef918c21f", "test"], ["5a808afb5542992bc0c4a76b", "test"], ["5ade45c75542992fa25da720", "test"], ["5ae328df55429928c423963c", "test"], ["5ac270e05542992f1f2b38c9", "dev"], ["5ae1e9635542997f29b3c19e", "test"], ["5a78d01d55429974737f78b3", "dev"], ["5ac02880554299012d1db5c0", "dev"], ["5ade26db5542997c77aded77", "test"], ["5a800ca555429969796c1bac", "dev"], ["5a7a3a2a5542996c55b2dd41", "dev"], ["5ac50dd05542994611c8b343", "dev"], ["5ac275e755429921a00aaf81", "dev"], ["5a8f1cfa554299458435d581", "dev"], ["5a9051615542995651fb5136", "test"], ["5ae25c815542992decbdcccc", "dev"], ["5ac561315542993e66e823a9", "dev"], ["5ab9530a55429970cfb8ea62", "test"], ["5a7fe0b55542994857a76818", "test"], ["5a8f876e5542997ba9cb32a1", "test"], ["5abcef9e55429959677d6b52", "test"], ["5a7647505542992d0ec06089", "test"], ["5a84f5195542994c784dda9d", "dev"], ["5ab42e55554299753aec5a53", "test"], ["5adcfc135542990d50227d87", "dev"], ["5ab341f755429969a97a8114", "test"], ["5a71419f5542994082a3e70d", "test"], ["5ae5fb2d554299546bf82ff5", "dev"], ["5a7effeb5542994959419aa0", "test"], ["5a8645825542991e771815eb", "dev"], ["5ab7f7465542993667794071", "dev"], ["5a859a9c5542997175ce1fd6", "test"], ["5a7e0bb65542997cc2c4750a", "test"], ["5a8d7b1755429941ae14dfc6", "test"], ["5adc51375542996e68525300", "test"], ["5a866fa855429960ec39b6a5", "test"], ["5ae2f5b955429928c423957e", "dev"], ["5a8abb115542992d82986f66", "dev"], ["5ae7ef625542994a481bbe40", "dev"], ["5a7e1b7155429965cec5ea6c", "test"], ["5a74167455429929fddd83dc", "dev"], ["5a88e605554299206df2b39c", "test"], ["5a7769a35542993569682d8f", "test"], ["5a7baae2554299042af8f794", "dev"], ["5a81e097554299676cceb129", "test"], ["5ac3ae885542995ef918c1e7", "dev"], ["5a8463d05542990548d0b2a5", "test"], ["5a8739a05542994775f607ab", "test"], ["5abe407155429976d4830ab4", "test"], ["5a8c92dd554299653c1aa0c4", "dev"], ["5a7326cb5542994cef4bc47f", "test"], ["5ab66a3555429954757d3290", "test"], ["5ae446fc5542995ad6573d24", "dev"], ["5ae256435542992decbdccc3", "test"], ["5abd420f55429933744ab75c", "test"], ["5a8de02b554299068b959e13", "test"], ["5ab56db7554299494045efc6", "dev"], ["5ab26a4b55429970612095fc", "dev"], ["5a8b7cb95542995d1e6f13bb", "test"], ["5a90ab995542990a98493683", "dev"], ["5a77cb335542997042120b3a", "dev"], ["5a728f745542991f9a20c4e6", "test"], ["5a847edd554299123d8c2272", "test"], ["5adfde09554299025d62a369", "dev"], ["5ae5fd395542996de7b71ab3", "test"], ["5a762a645542992d0ec0603d", "test"], ["5a714f6f5542994082a3e7b3", "test"], ["5a7265815542990c210a411b", "test"], ["5ae340a85542994393b9e668", "test"], ["5ae236b2554299495565da3e", "test"], ["5a825da055429954d2e2eb17", "test"], ["5a7d47da5542990b8f503975", "test"], ["5a7fb5ce5542994857a767ca", "test"], ["5ab474b35542990594ba9c25", "test"], ["5adf67e95542993344016c72", "dev"], ["5a7e782655429934daa2fc26", "dev"], ["5add564a5542990dbb2f7e38", "dev"], ["5a8a38135542996c9b8d5e59", "test"], ["5ae1d5a3554299234fd04319", "dev"], ["5abaf0ec5542996cc5e49f22", "dev"], ["5ab2b8fe5542991669774070", "test"], ["5ab9aab555429970cfb8eb65", "test"], ["5a7734ce5542993735360217", "test"], ["5a7a05485542990198eaf04c", "test"], ["5ae29d8a5542996483e64a0f", "test"], ["5abdd0f15542991f6610604d", "test"], ["5ac09d265542992a796ded39", "dev"], ["5ae09b7055429906c02daae4", "dev"], ["5a7506f455429916b0164240", "test"], ["5ae3d67f5542992f92d8238f", "dev"], ["5a874ce15542996432c57256", "test"], ["5a75d8a3554299109176e5a6", "dev"], ["5ac24d725542996366519966", "test"], ["5abd09585542996e802b469b", "test"], ["5a88a23f5542997e5c09a652", "test"], ["5a83e98e5542992ef85e2387", "test"], ["5a7fc6645542992e7d278d6e", "test"], ["5a796bfd55429970f5fffeac", "test"], ["5ab67cf455429954757d3302", "dev"], ["5abff00c5542994516f45537", "test"], ["5a7653d25542992db9473765", "test"], ["5ab5b6c1554299637185c5e5", "test"], ["5a7369da55429901807db037", "test"], ["5ae0b60655429945ae959437", "test"], ["5adeda67554299728e26c7db", "test"], ["5ac2dfac554299657fa290b3", "dev"], ["5ab3c18b55429969a97a81b0", "dev"], ["5ae33c4d5542992f92d82262", "dev"], ["5a8c90db554299653c1aa0bf", "test"], ["5ab7f1315542991d322237d1", "test"], ["5a7f697c5542992097ad2f59", "test"], ["5abfde655542997719eab6f2", "test"], ["5a847e805542993344746140", "dev"], ["5ab5b701554299637185c5e7", "dev"], ["5ab9d13455429939ce03dc31", "dev"], ["5a909a1a55429916514e7506", "test"], ["5adf9dc3554299025d62a2e6", "dev"], ["5a88fc4255429938390d4019", "dev"], ["5a84e21d5542997b5ce3ff7c", "test"], ["5a8d7f2b554299068b959d0c", "test"], ["5a7c38c955429935c91b5132", "test"], ["5a89cde555429946c8d6e9c4", "dev"], ["5ac273455542992f1f2b38d4", "test"], ["5abbd7985542992ccd8e7fb8", "dev"], ["5a7349125542994cef4bc505", "test"], ["5a774e9c55429972597f14f3", "test"], ["5ab72a025542992aa3b8c7b8", "test"], ["5a8038b55542996402f6a485", "test"], ["5ab278745542997061209617", "dev"], ["5abaa32655429955dce3eeaa", "dev"], ["5a872c3f5542994775f607a8", "test"], ["5ae60484554299546bf8301a", "dev"], ["5ab2958c554299449642c912", "dev"], ["5ac0e6fa5542996f0d89cc35", "dev"], ["5ae238375542994d89d5b39a", "test"], ["5ae207a65542994d89d5b317", "dev"], ["5abcf84d55429959677d6b86", "test"], ["5ac3a47f554299391541386d", "dev"], ["5a74c1c05542996c70cfadd8", "dev"], ["5ab67d2055429954757d3304", "test"], ["5a8ff6635542995651fb5086", "test"], ["5ab85c8d55429934fafe6d84", "dev"], ["5abcf299554299700f9d7936", "test"], ["5a716c415542994082a3e827", "test"], ["5add8ec55542997545bbbd7e", "test"], ["5ae564a155429960a22e02e4", "test"], ["5a85ea095542994775f606a8", "test"], ["5a8835245542994846c1ce27", "test"], ["5a76fda05542994aec3b71b0", "test"], ["5ac15de55542991316484afb", "test"], ["5ab281ec554299449642c8d1", "test"], ["5ae3f82a5542996836b02c01", "test"], ["5abbd12b55429931dba14577", "test"], ["5a74f7f15542993748c89765", "dev"], ["5a88341c5542994846c1ce20", "test"], ["5adf32ee5542995534e8c767", "test"], ["5a82a55955429966c78a6a70", "test"], ["5ae0198455429942ec259c1e", "test"], ["5adc396155429944faac2479", "test"], ["5ae152a65542990adbacf74f", "test"], ["5a769b0d5542993569682c50", "dev"], ["5abbdb9a55429931dba145ad", "test"], ["5ae2c696554299495565db32", "test"], ["5a7291c35542991f9a20c4f4", "dev"], ["5a7a616355429941d65f25f7", "dev"], ["5ab531cc5542990594ba9d23", "test"], ["5ab6faec5542991d32223708", "test"], ["5a804f01554299485f5985bf", "test"], ["5ae56fb85542993aec5ec1a9", "test"], ["5a72120e55429971e9dc923f", "dev"], ["5a78fe8155429970f5fffe11", "test"], ["5a7e1bae55429965cec5ea6f", "dev"], ["5a737e4655429908901be2cc", "test"], ["5ab852c05542990e739ec8c5", "test"], ["5a7a455c5542994f819ef1a7", "dev"], ["5a87556d5542994846c1cd65", "test"], ["5ae518ae5542993aec5ec13c", "dev"], ["5adf3e835542993a75d26442", "test"], ["5adf099a5542992d7e9f927f", "test"], ["5abda8d55542992ac4f382e7", "dev"], ["5adc65e85542996e68525350", "test"], ["5ab40e1c5542996a3a969f00", "test"], ["5adccf645542990d50227d32", "test"], ["5ab3e4ee554299233954ffe1", "test"], ["5a77d3f555429967ab1052c5", "test"], ["5a8762205542996e4f3087b8", "test"], ["5a8eaa74554299068b959e88", "test"], ["5a81ebee554299676cceb16d", "dev"], ["5a73630655429901807db020", "test"], ["5a721a7655429971e9dc9271", "test"], ["5ab6d9b25542995eadef00c8", "test"], ["5a8595a85542992a431d1b48", "test"], ["5ac283f7554299677310254a", "test"], ["5ab659e355429954757d324e", "test"], ["5ab965bb554299743d22eb56", "test"], ["5a7e31c055429934daa2fbee", "test"], ["5a7b72e15542997c3ec97187", "test"], ["5a7e3117554299495941991b", "test"], ["5a78d2b955429974737f78c0", "dev"], ["5a732ca45542992359bc325d", "test"], ["5ac547525542993e66e822a3", "dev"], ["5ae72e5d5542991e8301cba8", "test"], ["5ae5db98554299546bf82fa6", "test"], ["5ab9bb8355429970cfb8eb8b", "test"], ["5a73090455429901807daf6e", "test"], ["5a73cad5554299623ed4ac5e", "test"], ["5a84eecd5542994c784dda8e", "test"], ["5a7cd280554299452d57ba7f", "test"], ["5a85dfcc5542994775f60673", "test"], ["5a899f4955429946c8d6e971", "test"], ["5ac290e85542992f1f2b3916", "dev"], ["5ae1f70c5542997f29b3c1c5", "test"], ["5ac1a04e5542991316484b63", "test"], ["5ae0ee2c5542993d6555eca3", "test"], ["5a8baf635542996e8ac889c6", "dev"], ["5a8198565542995ce29dcc13", "test"], ["5a804510554299485f5985af", "dev"], ["5a862f985542991e771815af", "test"], ["5abc7bc45542993a06baf8ef", "test"], ["5ae81b2755429952e35eaa1e", "test"], ["5a89f90a5542992e4fca84be", "dev"], ["5a89e1bf5542992e4fca8433", "dev"], ["5a77101d5542994aec3b71d7", "test"], ["5a8e08ab5542995085b37393", "test"], ["5a7dbb295542990b8f503a4b", "test"], ["5a738bd0554299623ed4abee", "test"], ["5a7ab1b055429927d897bef4", "test"], ["5aba71b155429955dce3ee2a", "test"], ["5a7faed65542994857a767b6", "dev"], ["5ac5101455429924173fb592", "dev"], ["5ac1620855429964131be1bc", "test"], ["5ab5a7f45542997d4ad1f1a5", "test"], ["5ab5cd0d554299488d4d9a30", "test"], ["5adbcc085542996e6852523c", "test"], ["5a83de04554299334474609f", "test"], ["5a74581555429979e2882912", "test"], ["5a7758895542994aec3b729c", "test"], ["5a7542425542996c70cfaecc", "dev"], ["5ac195455542991316484b40", "test"], ["5adbfffb5542994650320c39", "test"], ["5ae71c4c554299572ea546ee", "dev"], ["5ae005b555429942ec259bec", "dev"], ["5adc0e0355429944faac2404", "test"], ["5a84c4bb5542991dd0999dec", "test"], ["5ab5f9a0554299488d4d9a6d", "test"], ["5a77cb5e55429967ab1052a7", "dev"], ["5ab5e9865542992aa134a3db", "dev"], ["5ac3295d5542995ef918c14d", "dev"], ["5add56ad5542992200553abf", "dev"], ["5ac113e2554299294b21909a", "test"], ["5a8b3af455429971feec467b", "dev"], ["5a83880e554299123d8c214e", "test"], ["5ab5eed65542997d4ad1f25c", "test"], ["5abe86365542993f32c2a14d", "dev"], ["5a72f0e25542991f9a20c5cf", "dev"], ["5abe765a55429976d4830b1f", "dev"], ["5a7a5aea5542996c55b2dd6f", "dev"], ["5a8a07725542992d82986e5b", "dev"], ["5ab946d7554299743d22eaaf", "dev"], ["5a86fde6554299211dda2b7f", "test"], ["5ae3e2455542994393b9e78c", "dev"], ["5a86eee85542994775f60787", "test"], ["5a8e3dc05542995085b37401", "dev"], ["5ae7f1e65542994a481bbe4a", "dev"], ["5a8b59a65542995d1e6f137c", "test"], ["5ab1dcab5542993be8fa985e", "dev"], ["5ab55a9b5542992aa134a2ba", "test"], ["5aba725455429901930fa7cc", "test"], ["5a8fae205542997ba9cb32ec", "test"], ["5a7d19d85542995ed0d165e8", "dev"], ["5ae547f85542990ba0bbb254", "test"], ["5a75d45f5542994ccc918630", "test"], ["5a7d147955429909bec76942", "test"], ["5ae21154554299495565d9d4", "test"], ["5a82a1ae55429954d2e2eb89", "dev"], ["5ab9339f554299131ca422bb", "test"], ["5ab28e2a5542993be8fa9947", "dev"], ["5a7be1ab5542997c3ec972a9", "test"], ["5ac289ff5542996366519a02", "test"], ["5ae27f22554299495565da8d", "dev"], ["5a7b9fac554299294a54aa2f", "test"], ["5ae6529a55429929b0807b32", "test"], ["5a89e6615542992e4fca8462", "dev"], ["5ae4a75a55429913cc2044bc", "test"], ["5ac313a0554299741d48a1b0", "test"], ["5a8ad8b65542992d82986fa2", "dev"], ["5a78bd515542990784727749", "dev"], ["5adcf83855429947343537e1", "test"], ["5ae4581e55429970de88d92f", "test"], ["5abe436455429976d4830ac6", "test"], ["5addd6045542997dc7907049", "dev"], ["5ab88c075542991b5579efbe", "test"], ["5ab43179554299233955004d", "test"], ["5a8ad6805542992d82986f93", "test"], ["5ab8f33155429919ba4e237f", "test"], ["5abce3015542993a06baf995", "dev"], ["5ae7d03b55429952e35ea9c2", "dev"], ["5ae361805542992e3233c3c9", "dev"], ["5a8031c1554299485f598585", "test"], ["5a7150c75542994082a3e7be", "test"], ["5abdba405542993f32c2a023", "test"], ["5ab938db5542996be2020454", "dev"], ["5ab1e7685542993be8fa9877", "test"], ["5ab83e1a55429919ba4e2280", "dev"], ["5a876b265542993e715abf33", "test"], ["5a8ed30655429917b4a5bdcd", "test"], ["5a7c53a855429935c91b515d", "dev"], ["5ae74fc45542997b22f6a6b4", "test"], ["5ab9180b5542991b5579f0f3", "test"], ["5a7365b155429901807db02a", "dev"], ["5ae5d63b5542996de7b71a2d", "test"], ["5abe18e055429976d4830a74", "test"], ["5ae1e2765542997f29b3c162", "test"], ["5addd82c5542990dbb2f7eb9", "test"], ["5a8193275542995ce29dcc07", "test"], ["5ae4606c5542996836b02c91", "dev"], ["5a7c827e55429907fabeefa7", "test"], ["5a79305755429907847277dd", "test"], ["5ae33a3b5542992f92d8224a", "dev"]], "corpus_sha1": "08d46385cfd99160e576b627f6ff6b14e84b47e4"} \ No newline at end of file diff --git a/benchmarks/multihop/manifests/musique_sample.json b/benchmarks/multihop/manifests/musique_sample.json new file mode 100644 index 0000000..27e1aae --- /dev/null +++ b/benchmarks/multihop/manifests/musique_sample.json @@ -0,0 +1 @@ +{"dataset": "musique", "source": {"repo": "dgslibisey/MuSiQue", "revision": "c8f4f8c9465fb69d31a8eae894c3fd509c4ca321", "file": "musique_ans_v1.0_dev.jsonl"}, "seed": 13, "n_questions": 1500, "n_dev": 500, "n_test": 1000, "corpus_size": 15405, "n_available": 2417, "ids": [["2hop__79875_84469", "test"], ["2hop__88834_425488", "test"], ["2hop__58939_189318", "test"], ["2hop__709357_136043", "test"], ["2hop__472083_7298", "test"], ["2hop__697729_576069", "test"], ["2hop__593397_224832", "test"], ["2hop__415518_616002", "test"], ["2hop__235471_565925", "test"], ["4hop1__707078_378185_282674_759393", "test"], ["2hop__6736_6733", "test"], ["2hop__91667_67223", "dev"], ["2hop__145282_21711", "test"], ["3hop1__838590_831637_91775", "dev"], ["2hop__40270_11402", "test"], ["2hop__131275_72870", "test"], ["2hop__837900_71701", "dev"], ["2hop__47071_162399", "test"], ["2hop__274110_413723", "dev"], ["2hop__80834_148696", "test"], ["3hop2__132957_483843_40768", "dev"], ["3hop1__857_846_7770", "dev"], ["2hop__44652_165532", "dev"], ["2hop__788306_131879", "test"], ["3hop1__305282_282081_10972", "dev"], ["2hop__72118_83769", "dev"], ["3hop2__90098_60649_10557", "test"], ["4hop1__93601_46196_497223_15840", "test"], ["4hop2__71753_729371_70784_79935", "test"], ["3hop1__829944_474097_84283", "test"], ["3hop1__358853_339990_54675", "dev"], ["3hop1__830700_538202_84283", "test"], ["3hop1__108062_720914_41132", "dev"], ["2hop__373544_66723", "test"], ["3hop1__222497_309482_89834", "test"], ["2hop__808562_72813", "test"], ["3hop2__127483_91678_10557", "dev"], ["4hop2__161602_426860_88460_18966", "test"], ["2hop__433694_20273", "test"], ["3hop1__857_846_7890", "dev"], ["3hop1__9285_5188_86687", "test"], ["4hop1__749065_698949_157828_162309", "test"], ["2hop__55966_162341", "dev"], ["2hop__90773_715759", "test"], ["2hop__632870_22402", "dev"], ["2hop__56806_7298", "dev"], ["4hop1__525129_315334_131926_90707", "dev"], ["3hop1__305566_568433_47686", "test"], ["2hop__77832_159673", "test"], ["3hop1__404541_629431_124169", "test"], ["3hop2__30152_105895_20999", "test"], ["2hop__806869_56873", "test"], ["2hop__84937_21969", "test"], ["3hop1__603558_87694_124169", "dev"], ["2hop__418360_67370", "test"], ["3hop2__9951_20335_9331", "dev"], ["4hop1__93661_59747_211319_557671", "test"], ["2hop__726717_610238", "dev"], ["4hop2__9988_158985_70784_61381", "test"], ["2hop__72686_83769", "dev"], ["2hop__61318_362941", "test"], ["3hop1__379231_40769_64047", "test"], ["2hop__418941_429437", "test"], ["2hop__245835_762148", "test"], ["3hop1__831499_228453_10972", "test"], ["3hop2__304722_689786_63959", "test"], ["3hop1__540585_568433_47686", "dev"], ["2hop__10620_79092", "test"], ["3hop1__791757_15840_36014", "test"], ["2hop__160092_80144", "test"], ["2hop__69765_43945", "test"], ["2hop__45296_83837", "test"], ["4hop1__151650_5274_458768_33633", "test"], ["3hop2__103889_30152_20999", "test"], ["2hop__779424_75487", "dev"], ["2hop__507528_139339", "dev"], ["3hop1__698501_503371_21711", "dev"], ["2hop__77233_33207", "dev"], ["3hop1__268450_503371_21711", "test"], ["2hop__635185_141338", "dev"], ["3hop1__446612_160545_34754", "dev"], ["4hop1__47210_53706_795904_580996", "test"], ["2hop__442175_56873", "dev"], ["2hop__731584_700117", "dev"], ["4hop1__131820_59747_211319_557671", "test"], ["2hop__66843_494136", "test"], ["2hop__159903_154896", "test"], ["2hop__157285_556157", "test"], ["3hop1__320697_547811_80702", "dev"], ["3hop1__174833_650651_7262", "test"], ["2hop__57638_72467", "test"], ["2hop__9174_9237", "test"], ["2hop__608902_653666", "test"], ["2hop__518354_67465", "dev"], ["4hop2__161602_426860_88460_21034", "dev"], ["2hop__35686_58556", "test"], ["2hop__697790_864352", "test"], ["2hop__530579_13529", "test"], ["2hop__231805_514393", "test"], ["3hop1__823336_228453_10972", "test"], ["4hop2__161602_426860_88460_63559", "test"], ["4hop1__88342_49853_128008_48717", "test"], ["2hop__157339_156796", "test"], ["3hop1__857_846_7877", "test"], ["3hop1__19768_19788_19761", "test"], ["2hop__29893_29898", "test"], ["4hop1__58323_375563_161848_53331", "test"], ["2hop__107548_124896", "test"], ["2hop__403060_92763", "dev"], ["3hop1__106042_64399_59680", "test"], ["2hop__388523_75487", "test"], ["3hop1__664835_650651_7262", "test"], ["2hop__96062_159673", "test"], ["2hop__821197_368148", "dev"], ["2hop__25396_593388", "test"], ["3hop2__36497_89854_76291", "test"], ["4hop1__525129_315334_131926_87157", "dev"], ["3hop1__379315_160713_77246", "dev"], ["2hop__153573_45901", "dev"], ["2hop__12039_11957", "dev"], ["3hop1__129499_33897_81096", "test"], ["3hop1__145924_131905_41948", "dev"], ["2hop__497191_129669", "dev"], ["4hop3__152056_698586_89595_54362", "test"], ["4hop2__5206_14670_8987_8529", "test"], ["4hop1__71715_59747_211319_557671", "dev"], ["4hop1__40316_497223_15840_36002", "test"], ["3hop2__57233_11282_56883", "dev"], ["2hop__691455_64006", "dev"], ["2hop__647996_551681", "test"], ["2hop__92590_43786", "test"], ["2hop__149983_108549", "test"], ["3hop1__458916_651917_47686", "test"], ["4hop1__152562_5274_458768_33677", "test"], ["2hop__50087_807969", "test"], ["2hop__468778_88165", "test"], ["2hop__797443_120537", "dev"], ["3hop1__798544_514761_594810", "test"], ["2hop__67660_158861", "test"], ["2hop__79875_72187", "dev"], ["3hop1__796649_720914_41132", "test"], ["4hop3__316459_41402_157761_13584", "dev"], ["3hop1__30390_36635_87083", "test"], ["2hop__75207_365216", "dev"], ["2hop__86458_20273", "dev"], ["3hop1__818422_160545_34754", "test"], ["2hop__636597_480615", "test"], ["4hop3__175072_88460_30152_20999", "test"], ["4hop2__161602_426860_88460_86452", "test"], ["2hop__156702_51769", "dev"], ["3hop1__119402_102039_158266", "dev"], ["3hop1__617062_127905_46894", "test"], ["3hop2__304722_297619_63959", "test"], ["2hop__280451_84616", "dev"], ["2hop__2299_1783", "test"], ["2hop__161915_78851", "test"], ["2hop__108158_83769", "test"], ["3hop1__103180_214799_259594", "test"], ["2hop__60001_512508", "test"], ["3hop2__132957_133264_40768", "test"], ["3hop2__809962_89048_66294", "test"], ["4hop3__312119_132409_223216_35031", "test"], ["3hop1__145194_160545_34754", "dev"], ["2hop__21075_5028", "test"], ["2hop__88149_807969", "test"], ["4hop1__606685_508773_85832_745702", "test"], ["3hop1__850160_15840_36014", "test"], ["4hop3__15567_29339_508306_70744", "test"], ["2hop__475808_71302", "test"], ["2hop__178922_5385", "dev"], ["2hop__35566_17873", "dev"], ["3hop1__389955_132457_47686", "test"], ["2hop__499577_617190", "dev"], ["2hop__6827_79845", "test"], ["3hop1__103890_10659_23296", "test"], ["2hop__96410_47902", "test"], ["3hop1__536452_15840_36002", "dev"], ["2hop__88906_55840", "dev"], ["2hop__152027_141308", "test"], ["3hop1__232315_831637_91775", "test"], ["2hop__40970_90367", "test"], ["2hop__315267_277284", "test"], ["3hop1__346416_831637_91775", "dev"], ["4hop3__324979_836463_161616_77103", "test"], ["3hop1__90327_78276_68042", "dev"], ["3hop1__84887_538202_84283", "test"], ["2hop__160223_58067", "test"], ["3hop1__273571_834494_33939", "test"], ["4hop1__726391_153080_33952_34053", "test"], ["2hop__92051_827343", "test"], ["2hop__41158_35123", "dev"], ["2hop__169443_83769", "dev"], ["3hop1__131829_39743_24526", "dev"], ["2hop__525581_161450", "test"], ["2hop__39954_80940", "test"], ["2hop__109242_113442", "test"], ["2hop__761291_87295", "dev"], ["2hop__478978_124498", "dev"], ["2hop__286_574195", "dev"], ["3hop2__781093_467331_162182", "test"], ["3hop1__406043_831637_91775", "dev"], ["2hop__23459_35123", "test"], ["3hop1__222380_443779_52195", "dev"], ["2hop__810411_159673", "test"], ["2hop__673655_316668", "test"], ["2hop__166484_798067", "test"], ["2hop__731956_126089", "dev"], ["2hop__734582_74481", "test"], ["4hop1__46196_497223_15840_36014", "test"], ["3hop1__862117_792411_51423", "test"], ["2hop__613959_135844", "test"], ["3hop1__579562_629431_124169", "dev"], ["2hop__194469_83289", "dev"], ["2hop__176_51164", "test"], ["2hop__367318_7292", "dev"], ["4hop1__9007_698949_157828_239539", "test"], ["4hop1__726152_153080_159767_81096", "test"], ["3hop1__401538_132457_47686", "test"], ["4hop1__72247_497223_15840_36014", "test"], ["3hop1__604991_339990_15538", "dev"], ["2hop__95673_62765", "test"], ["2hop__47712_25719", "dev"], ["2hop__94507_654855", "test"], ["2hop__80546_75184", "test"], ["4hop2__105527_39078_8987_8974", "dev"], ["3hop1__415141_42197_18397", "test"], ["2hop__17827_54024", "dev"], ["2hop__146919_29244", "dev"], ["2hop__528263_865831", "test"], ["2hop__132469_533882", "test"], ["2hop__345_59409", "test"], ["4hop1__567737_141375_458768_33633", "test"], ["2hop__82766_67668", "test"], ["2hop__29339_40482", "test"], ["3hop1__145212_443779_52195", "test"], ["3hop1__79462_91850_685675", "test"], ["2hop__50936_90973", "dev"], ["2hop__88834_356793", "dev"], ["2hop__159146_120065", "test"], ["2hop__35137_22901", "dev"], ["2hop__30390_92972", "dev"], ["3hop1__103890_10659_83548", "test"], ["3hop1__641921_42197_18397", "dev"], ["3hop1__529164_567566_84283", "dev"], ["3hop2__43283_12039_93690", "test"], ["3hop1__540585_568433_51423", "dev"], ["2hop__818302_25719", "test"], ["2hop__48481_319887", "dev"], ["2hop__47808_198548", "test"], ["3hop2__132957_222979_40768", "test"], ["2hop__95970_152907", "dev"], ["2hop__107905_110222", "test"], ["4hop1__58323_375563_161848_44231", "test"], ["2hop__145082_35034", "test"], ["4hop3__862_846_613770_7713", "test"], ["2hop__839126_472799", "test"], ["3hop1__142044_538202_84283", "dev"], ["2hop__700093_455653", "dev"], ["2hop__75714_148696", "test"], ["4hop3__160230_470287_345497_72134", "dev"], ["3hop1__742464_15840_36002", "test"], ["2hop__500379_126537", "test"], ["3hop2__88342_93066_90766", "dev"], ["3hop1__672543_503371_21711", "test"], ["2hop__80637_19033", "test"], ["4hop2__567956_14670_8987_8529", "test"], ["3hop2__304722_372858_63959", "test"], ["3hop2__304722_602987_63959", "dev"], ["2hop__611538_152907", "dev"], ["3hop1__454441_55349_651302", "dev"], ["3hop2__655849_467331_162182", "dev"], ["2hop__60229_38663", "dev"], ["4hop1__608132_508773_85832_745702", "test"], ["2hop__152700_141308", "dev"], ["3hop2__851134_613770_7713", "dev"], ["4hop1__152562_5274_458768_33633", "test"], ["2hop__567471_123283", "test"], ["2hop__39954_80303", "test"], ["2hop__32887_235620", "test"], ["2hop__119423_779396", "test"], ["2hop__42464_718451", "test"], ["3hop1__103662_443779_52195", "dev"], ["2hop__259267_22402", "test"], ["4hop1__145422_698949_157828_162309", "test"], ["2hop__197470_271394", "test"], ["2hop__6744_6731", "dev"], ["3hop1__818422_160545_62931", "dev"], ["2hop__142881_127399", "test"], ["2hop__135993_160249", "test"], ["2hop__460946_294723", "dev"], ["2hop__94629_1110", "dev"], ["2hop__49409_67668", "test"], ["2hop__189094_612080", "test"], ["4hop1__152562_5274_458768_33632", "test"], ["4hop2__161602_426860_88460_20985", "test"], ["2hop__789565_416401", "test"], ["2hop__698577_851686", "test"], ["3hop2__90098_19639_10557", "test"], ["2hop__13592_85544", "test"], ["2hop__301587_233146", "dev"], ["2hop__749997_96912", "test"], ["3hop1__109196_720914_41132", "test"], ["4hop1__17192_17130_70784_61381", "test"], ["2hop__124807_135019", "test"], ["3hop2__132957_132968_40768", "test"], ["3hop1__163024_443779_52195", "test"], ["2hop__684839_139312", "dev"], ["2hop__17508_17335", "dev"], ["3hop1__8148_24918_24939", "test"], ["4hop3__322354_466199_695123_72134", "test"], ["4hop1__341777_508773_85832_745702", "test"], ["2hop__130404_59180", "dev"], ["2hop__31995_8583", "test"], ["3hop1__390489_502967_84283", "dev"], ["4hop1__9007_698949_157828_162309", "test"], ["3hop2__326964_464925_7713", "dev"], ["3hop1__275261_443779_52195", "test"], ["2hop__42578_55840", "dev"], ["3hop1__220023_503371_21711", "test"], ["2hop__701525_342810", "test"], ["2hop__85596_807969", "test"], ["2hop__29893_29905", "dev"], ["2hop__789398_74735", "test"], ["4hop1__344155_378185_282674_759393", "dev"], ["4hop1__525129_315334_131926_89261", "test"], ["2hop__5658_161105", "test"], ["2hop__333020_69048", "dev"], ["2hop__199513_801817", "test"], ["4hop3__275416_24325_125104_10557", "test"], ["2hop__142443_768138", "test"], ["3hop2__304722_466316_63959", "test"], ["4hop3__387712_132409_223216_35031", "dev"], ["3hop1__487301_443779_52195", "test"], ["4hop1__151903_5274_458768_33677", "test"], ["3hop1__699507_792411_51423", "test"], ["2hop__22885_160040", "dev"], ["3hop2__101809_30152_20999", "test"], ["2hop__849534_122868", "dev"], ["3hop1__159737_160713_77246", "dev"], ["3hop1__46571_87694_64412", "test"], ["2hop__398344_54166", "test"], ["2hop__644898_64046", "test"], ["3hop1__136852_30587_83479", "test"], ["3hop1__446612_160545_62931", "test"], ["2hop__256336_714772", "test"], ["3hop2__76873_69814_64554", "test"], ["3hop1__702699_792411_51423", "dev"], ["2hop__95773_51329", "dev"], ["2hop__142621_225454", "test"], ["3hop2__127483_76623_10557", "test"], ["3hop1__539312_744503_34754", "test"], ["3hop1__68802_89752_49923", "test"], ["3hop2__84553_90098_10557", "test"], ["2hop__207597_73244", "test"], ["2hop__25797_990", "test"], ["2hop__770487_494646", "test"], ["2hop__67660_67223", "test"], ["2hop__107601_110222", "test"], ["2hop__466700_214359", "test"], ["3hop1__135392_87694_64412", "dev"], ["2hop__95687_577502", "dev"], ["2hop__650138_22402", "test"], ["4hop3__746868_88460_30152_20999", "test"], ["4hop3__771862_466199_695123_72134", "test"], ["2hop__128342_375952", "test"], ["4hop3__524186_219173_548463_72134", "test"], ["2hop__286268_97805", "test"], ["2hop__359312_18803", "test"], ["2hop__409884_83837", "dev"], ["2hop__227031_144857", "dev"], ["2hop__84172_198548", "dev"], ["2hop__609313_20273", "test"], ["3hop1__108833_720914_27537", "dev"], ["2hop__34495_49388", "dev"], ["4hop1__61681_53706_795904_580996", "dev"], ["2hop__25797_25839", "test"], ["4hop1__131611_32392_823060_610794", "test"], ["2hop__123148_5385", "dev"], ["4hop1__163763_49925_13759_736921", "dev"], ["3hop1__241085_42197_18397", "dev"], ["4hop2__71753_73205_70784_79935", "test"], ["2hop__14095_15463", "dev"], ["2hop__144150_82341", "test"], ["2hop__367336_97805", "test"], ["2hop__6827_86319", "dev"], ["2hop__85463_50252", "dev"], ["4hop1__105773_457883_650651_7262", "dev"], ["4hop1__88342_49853_128008_89859", "dev"], ["4hop1__44149_32392_823060_610794", "test"], ["3hop1__140554_2053_52946", "test"], ["3hop1__512574_42197_18397", "dev"], ["3hop1__28841_547811_41132", "dev"], ["2hop__64175_90973", "test"], ["3hop1__254856_400692_51423", "test"], ["2hop__158247_71142", "test"], ["3hop1__617062_127905_80286", "test"], ["2hop__748615_123956", "test"], ["3hop1__813285_774554_124169", "test"], ["4hop3__668721_132409_146285_35031", "test"], ["4hop2__71753_261673_70784_79935", "test"], ["2hop__198650_84856", "test"], ["4hop1__814349_624859_355213_203322", "test"], ["2hop__55552_158105", "test"], ["3hop1__106042_64399_53006", "dev"], ["4hop1__33012_53706_795904_580996", "dev"], ["2hop__800166_156034", "test"], ["3hop1__680374_503371_21711", "dev"], ["2hop__803903_78606", "dev"], ["3hop1__53690_161697_85460", "test"], ["3hop1__558449_503371_21711", "test"], ["4hop1__58323_375563_161848_61344", "test"], ["2hop__280583_229757", "dev"], ["2hop__16267_25719", "dev"], ["3hop1__280834_547811_80702", "dev"], ["3hop1__532517_214799_259594", "dev"], ["2hop__671664_646310", "dev"], ["2hop__48784_70131", "test"], ["2hop__663184_57816", "test"], ["2hop__476927_31270", "dev"], ["2hop__39063_593388", "dev"], ["2hop__42027_42004", "dev"], ["2hop__450608_373421", "dev"], ["3hop1__635099_131926_90707", "test"], ["4hop3__323121_88460_30152_20999", "test"], ["3hop1__312146_720914_27537", "test"], ["2hop__84103_345851", "test"], ["2hop__657913_88628", "test"], ["3hop2__145209_20335_9331", "test"], ["2hop__452580_70131", "test"], ["2hop__448615_127908", "dev"], ["4hop1__842143_153080_33897_81096", "test"], ["3hop1__312146_720914_41132", "test"], ["2hop__342746_679190", "test"], ["2hop__16844_20510", "test"], ["3hop1__765388_42197_18397", "test"], ["3hop1__136016_87694_64412", "dev"], ["2hop__159902_362941", "test"], ["2hop__259228_793698", "dev"], ["2hop__13986_161674", "test"], ["3hop1__820301_720914_41132", "test"], ["2hop__204493_69048", "dev"], ["3hop1__458916_665330_47686", "dev"], ["2hop__40485_40501", "test"], ["2hop__247353_55227", "test"], ["3hop1__639955_834494_34053", "test"], ["2hop__35137_22932", "dev"], ["3hop1__106847_160713_77246", "dev"], ["2hop__488811_30078", "test"], ["2hop__171254_383727", "test"], ["2hop__176712_8311", "dev"], ["2hop__754753_57816", "test"], ["2hop__685280_510757", "test"], ["4hop1__749065_698949_157828_239539", "test"], ["4hop1__726391_153080_33952_33939", "test"], ["2hop__137061_45752", "test"], ["3hop1__274148_792411_51423", "test"], ["4hop1__199362_378185_282674_759393", "test"], ["2hop__865828_377862", "dev"], ["2hop__82910_75184", "test"], ["2hop__66167_88526", "test"], ["4hop2__105527_14670_8987_8974", "test"], ["3hop1__295815_40769_64047", "test"], ["2hop__735014_83837", "test"], ["2hop__37097_53663", "test"], ["2hop__73615_16848", "test"], ["2hop__52283_75184", "dev"], ["3hop1__765847_831637_91775", "dev"], ["2hop__576118_25719", "test"], ["2hop__21104_16254", "test"], ["2hop__104095_40501", "test"], ["4hop1__324759_378185_282674_759393", "dev"], ["4hop3__507819_88460_30152_20999", "test"], ["2hop__75207_7363", "test"], ["2hop__161151_42352", "dev"], ["3hop1__239036_15840_36014", "dev"], ["2hop__17455_21567", "test"], ["3hop1__511157_709625_84283", "dev"], ["3hop1__617062_127905_86588", "test"], ["4hop1__436202_765799_282674_759393", "dev"], ["4hop1__58323_375563_161848_20546", "dev"], ["2hop__431934_30351", "test"], ["2hop__693444_220151", "test"], ["2hop__123145_80420", "dev"], ["3hop2__90327_89854_76291", "dev"], ["3hop2__304722_230978_63959", "test"], ["2hop__23297_23366", "test"], ["3hop2__92991_38242_76291", "test"], ["2hop__10122_18974", "test"], ["3hop1__109196_720914_27537", "test"], ["2hop__80375_90181", "test"], ["2hop__23459_22946", "test"], ["2hop__390055_82341", "test"], ["2hop__554167_451128", "dev"], ["2hop__24976_25002", "test"], ["4hop3__160230_470287_128875_72134", "test"], ["3hop2__140194_16797_51068", "dev"], ["2hop__6827_75783", "test"], ["4hop1__39871_314549_131976_56103", "test"], ["4hop1__342858_131850_33897_81096", "test"], ["2hop__575282_114648", "dev"], ["2hop__51760_602275", "test"], ["3hop1__45612_502967_84283", "dev"], ["4hop1__105401_17130_70784_79935", "test"], ["3hop1__41865_55331_34700", "dev"], ["2hop__310456_846599", "test"], ["4hop1__443274_17130_70784_61381", "dev"], ["2hop__24661_81961", "test"], ["3hop1__401538_132457_51423", "test"], ["4hop1__151650_5274_458768_33632", "test"], ["2hop__852657_155922", "test"], ["2hop__273020_57594", "test"], ["3hop1__109113_720914_27537", "test"], ["4hop1__83058_53706_795904_580996", "test"], ["2hop__144658_599630", "dev"], ["3hop1__617062_127905_87812", "test"], ["2hop__95687_2684", "dev"], ["2hop__677929_696450", "dev"], ["4hop1__130508_32392_823060_610794", "test"], ["2hop__21103_16254", "test"], ["2hop__754021_20057", "dev"], ["2hop__64323_55840", "dev"], ["4hop3__710473_354093_254885_76356", "test"], ["2hop__33288_157788", "test"], ["3hop1__671542_665330_47686", "test"], ["2hop__253612_110882", "test"], ["3hop1__105767_443779_52195", "dev"], ["2hop__593866_80026", "dev"], ["3hop1__617062_127905_92225", "test"], ["2hop__124933_653666", "test"], ["2hop__623501_297043", "dev"], ["3hop1__593288_720914_41132", "dev"], ["2hop__61752_54024", "test"], ["2hop__53937_92673", "dev"], ["3hop2__222979_132536_40768", "dev"], ["2hop__6745_6731", "test"], ["2hop__62823_654855", "test"], ["2hop__461639_83837", "test"], ["2hop__158010_58067", "test"], ["2hop__846388_19320", "dev"], ["2hop__388467_110882", "dev"], ["2hop__629030_56873", "test"], ["2hop__272714_113442", "test"], ["3hop1__747772_514761_594810", "test"], ["2hop__9285_42311", "dev"], ["4hop1__567737_141375_458768_33637", "test"], ["2hop__848156_130869", "test"], ["4hop1__482815_624859_355213_203322", "test"], ["3hop1__564628_42197_18397", "test"], ["4hop3__857_846_613770_7713", "test"], ["4hop1__391525_49925_13759_736921", "dev"], ["2hop__175175_347233", "test"], ["2hop__819974_129669", "test"], ["3hop1__801799_547811_80702", "dev"], ["3hop1__446612_160545_60577", "dev"], ["2hop__78766_461854", "test"], ["3hop1__346812_15840_36002", "dev"], ["2hop__69_59409", "test"], ["3hop2__89854_36497_76291", "test"], ["2hop__2416_53663", "test"], ["3hop2__104979_23308_28343", "test"], ["4hop1__178366_229349_66759_75165", "test"], ["2hop__847760_80026", "test"], ["4hop1__429751_32392_823060_610794", "test"], ["2hop__84439_482364", "dev"], ["3hop1__346812_15840_36014", "test"], ["2hop__30384_30351", "test"], ["2hop__130867_54221", "test"], ["2hop__385392_756039", "test"], ["2hop__68158_189357", "test"], ["2hop__136067_160249", "test"], ["2hop__765180_123956", "test"], ["2hop__43579_289940", "test"], ["4hop1__94201_642284_131926_90707", "dev"], ["2hop__669456_108549", "test"], ["2hop__13592_49388", "test"], ["3hop1__527372_339990_54675", "test"], ["2hop__556426_9237", "dev"], ["2hop__30741_85990", "dev"], ["3hop1__398232_326948_10972", "test"], ["2hop__70584_198548", "test"], ["4hop3__373866_5189_31809_86687", "test"], ["4hop3__310746_88460_30152_20999", "test"], ["4hop1__88342_75218_128008_87812", "test"], ["3hop2__326964_265410_7713", "test"], ["2hop__683835_709167", "test"], ["3hop1__516535_834494_34088", "dev"], ["4hop2__71753_158985_70784_79935", "test"], ["2hop__67660_81007", "dev"], ["3hop1__462125_538202_84283", "dev"], ["2hop__85713_72187", "test"], ["2hop__61714_89309", "test"], ["3hop2__304722_781203_63959", "dev"], ["2hop__61714_42553", "dev"], ["2hop__631139_75487", "dev"], ["2hop__353836_80026", "test"], ["3hop1__626214_503371_21711", "dev"], ["3hop1__140994_2053_52946", "test"], ["3hop1__38738_78276_68042", "test"], ["2hop__307719_121900", "dev"], ["2hop__85325_88165", "test"], ["4hop2__9988_158279_70784_61381", "test"], ["2hop__668407_683671", "dev"], ["2hop__326404_120065", "test"], ["2hop__13539_13493", "test"], ["2hop__13353_17738", "dev"], ["3hop1__248929_160713_77246", "dev"], ["2hop__123214_22402", "test"], ["2hop__691639_154896", "dev"], ["4hop3__673447_132409_371500_35031", "test"], ["3hop1__351696_42197_18397", "test"], ["3hop1__19768_19788_10117", "test"], ["3hop1__102461_24918_24991", "dev"], ["3hop2__11282_57233_56883", "test"], ["2hop__63002_461854", "test"], ["3hop1__857_846_7795", "test"], ["2hop__776377_857193", "test"], ["2hop__223823_29898", "dev"], ["3hop1__844109_720914_27537", "dev"], ["4hop1__343844_17130_70784_61381", "dev"], ["4hop3__279362_88460_30152_20999", "dev"], ["2hop__754682_730690", "test"], ["2hop__75207_53316", "dev"], ["4hop3__426860_88460_30152_20999", "test"], ["3hop1__108833_720914_41132", "test"], ["3hop2__565959_223623_162182", "dev"], ["2hop__158275_35739", "dev"], ["2hop__81825_49084", "test"], ["3hop2__19639_90098_10557", "dev"], ["4hop1__343844_17130_70784_79935", "test"], ["2hop__780652_93362", "dev"], ["2hop__122565_37168", "test"], ["4hop3__754156_88460_30152_20999", "dev"], ["3hop1__30240_740597_41132", "test"], ["3hop1__617062_127905_80487", "dev"], ["2hop__75207_46837", "test"], ["4hop2__567956_39078_8987_8974", "dev"], ["2hop__284789_52835", "test"], ["2hop__38030_23241", "test"], ["3hop1__347858_503371_21711", "test"], ["2hop__200391_44187", "test"], ["2hop__130984_55721", "test"], ["3hop2__72083_92991_76291", "test"], ["4hop1__88342_49853_128008_80487", "dev"], ["4hop3__373573_132409_223216_35031", "dev"], ["2hop__32441_522949", "test"], ["2hop__71715_50429", "test"], ["2hop__73717_653666", "test"], ["2hop__133116_533882", "test"], ["4hop2__161602_474028_88460_21062", "test"], ["3hop1__707569_228453_10972", "test"], ["2hop__13574_13498", "test"], ["2hop__323534_810083", "dev"], ["2hop__445963_6063", "dev"], ["3hop1__373866_5189_63702", "test"], ["3hop1__622497_160088_821792", "dev"], ["2hop__765678_310556", "test"], ["2hop__80860_510545", "test"], ["2hop__24697_578772", "test"], ["2hop__24661_66233", "dev"], ["4hop1__151650_5274_458768_33637", "dev"], ["3hop1__604991_339990_54675", "dev"], ["3hop1__691197_15840_36014", "test"], ["3hop1__337705_132457_47686", "test"], ["2hop__123509_5385", "test"], ["2hop__286093_361551", "dev"], ["2hop__85463_33311", "dev"], ["4hop1__88342_75218_128008_67954", "dev"], ["2hop__472486_97805", "dev"], ["3hop2__25177_19506_23299", "dev"], ["3hop1__23444_160713_77246", "test"], ["4hop1__814776_765799_282674_759393", "test"], ["2hop__31091_31122", "dev"], ["3hop1__145194_160545_60577", "test"], ["2hop__756620_127399", "test"], ["2hop__103889_86452", "test"], ["2hop__96137_121319", "test"], ["3hop1__315504_629431_64412", "test"], ["2hop__398344_439882", "test"], ["2hop__15258_54866", "dev"], ["2hop__719184_55227", "test"], ["3hop1__545624_2053_5289", "test"], ["2hop__20268_42014", "test"], ["2hop__150866_11402", "test"], ["4hop1__463635_624859_355213_203322", "dev"], ["2hop__93145_54024", "test"], ["2hop__151705_141308", "test"], ["2hop__61143_165532", "test"], ["2hop__809785_606637", "dev"], ["3hop1__241001_568433_47686", "dev"], ["4hop1__846600_153080_33897_81096", "dev"], ["2hop__30905_213732", "dev"], ["3hop2__326964_263934_7713", "test"], ["2hop__144317_120065", "dev"], ["3hop1__377943_650651_7262", "dev"], ["3hop1__848113_15840_36002", "test"], ["4hop1__567737_141375_458768_33632", "dev"], ["3hop2__831456_223623_162182", "test"], ["2hop__327688_56873", "dev"], ["3hop1__324448_831637_91775", "test"], ["4hop2__9988_729371_70784_61381", "test"], ["3hop1__159803_89752_75165", "test"], ["3hop1__635099_131926_13165", "dev"], ["3hop2__326964_7721_7713", "test"], ["3hop2__14777_8533_9331", "test"], ["2hop__695364_513121", "test"], ["3hop1__857_846_7810", "test"], ["4hop1__199881_378185_282674_759393", "test"], ["3hop1__127821_157828_162309", "test"], ["3hop1__498954_160713_77246", "dev"], ["2hop__283298_22402", "dev"], ["2hop__233551_746883", "test"], ["2hop__384238_74735", "dev"], ["2hop__88372_83837", "dev"], ["2hop__80673_510545", "dev"], ["3hop1__582007_160713_77246", "dev"], ["3hop1__273571_834494_34053", "test"], ["4hop3__129318_29339_508306_70744", "test"], ["4hop3__373573_132409_146285_35031", "test"], ["2hop__860241_766393", "test"], ["3hop1__241001_568433_51423", "dev"], ["4hop3__131820_29339_508306_70744", "test"], ["2hop__77518_548781", "dev"], ["4hop1__88342_75218_128008_46748", "dev"], ["2hop__6750_6733", "test"], ["3hop1__857_846_7791", "dev"], ["3hop1__64957_87694_124169", "test"], ["4hop3__678814_466199_695123_72134", "test"], ["2hop__6265_718451", "dev"], ["3hop1__104996_160713_77246", "test"], ["2hop__646573_83769", "test"], ["2hop__391258_161450", "test"], ["3hop1__53690_161697_73916", "test"], ["2hop__496732_658428", "test"], ["4hop1__455862_508773_85832_745702", "test"], ["4hop1__43565_624859_355213_203322", "test"], ["3hop2__127483_19639_10557", "test"], ["3hop2__222979_380281_40768", "test"], ["4hop1__161605_32392_823060_610794", "test"], ["2hop__1845_38366", "test"], ["3hop1__491648_339990_54675", "dev"], ["2hop__68590_27628", "test"], ["2hop__7046_44085", "test"], ["2hop__373544_80420", "test"], ["3hop1__305566_568433_51423", "test"], ["3hop1__147162_131905_41948", "dev"], ["3hop2__90098_76623_10557", "dev"], ["3hop1__662618_339990_54675", "test"], ["4hop1__145494_698949_157828_162309", "test"], ["2hop__423045_570226", "test"], ["4hop1__571599_49925_13759_736921", "test"], ["3hop1__326941_366209_84283", "test"], ["3hop1__516535_834494_33939", "test"], ["2hop__71294_494136", "dev"], ["2hop__35136_35117", "test"], ["3hop1__844109_720914_41132", "test"], ["2hop__444265_82341", "test"], ["2hop__96194_78303", "test"], ["4hop2__71753_623626_70784_61381", "dev"], ["3hop1__637555_339990_15538", "test"], ["3hop1__278222_42197_18397", "test"], ["4hop1__629520_49925_13759_736921", "test"], ["2hop__10114_599630", "dev"], ["4hop1__310029_153080_33897_81096", "test"], ["3hop1__5544_34179_23375", "test"], ["2hop__88632_67668", "test"], ["2hop__215896_460425", "test"], ["2hop__66708_573255", "test"], ["2hop__519160_78303", "dev"], ["3hop2__132957_52222_40768", "dev"], ["3hop2__304722_203717_63959", "test"], ["2hop__123545_47134", "test"], ["4hop3__531933_836463_161616_77103", "test"], ["3hop1__694534_160088_73916", "test"], ["2hop__435087_55984", "dev"], ["3hop2__802066_223623_162182", "test"], ["4hop2__9988_261673_70784_79935", "test"], ["2hop__133102_417697", "dev"], ["3hop1__759384_42197_18397", "test"], ["2hop__68590_27619", "dev"], ["2hop__131061_54221", "test"], ["2hop__153573_109006", "test"], ["2hop__156700_63853", "test"], ["2hop__195917_575657", "dev"], ["2hop__57638_88526", "test"], ["3hop2__87184_92991_76291", "test"], ["3hop1__105126_443779_52195", "dev"], ["3hop1__49236_89752_49923", "test"], ["3hop1__144439_443779_52195", "dev"], ["3hop1__516535_834494_34053", "test"], ["2hop__283_574195", "test"], ["3hop1__418091_15840_36014", "test"], ["2hop__857585_14251", "dev"], ["2hop__747894_135844", "dev"], ["2hop__85993_512508", "test"], ["3hop2__127483_84553_10557", "test"], ["4hop1__21282_692112_214799_259594", "dev"], ["3hop1__309596_160713_77246", "test"], ["4hop1__39871_314549_131976_90181", "dev"], ["2hop__71871_494136", "test"], ["2hop__787538_31270", "test"], ["3hop1__315504_629431_124169", "test"], ["3hop1__632623_160713_77246", "test"], ["3hop1__694534_160088_49779", "dev"], ["2hop__539184_119915", "test"], ["2hop__83938_5666", "test"], ["4hop1__89530_624859_355213_203322", "test"], ["2hop__55032_786853", "test"], ["2hop__157602_21567", "dev"], ["2hop__635187_861533", "test"], ["3hop1__491648_339990_15538", "dev"], ["4hop1__130404_59747_211319_557671", "test"], ["2hop__159215_779396", "test"], ["2hop__93230_319887", "test"], ["2hop__335418_279684", "test"], ["2hop__542569_87295", "test"], ["4hop1__228808_49925_13759_736921", "dev"], ["2hop__204146_850295", "dev"], ["2hop__864101_135373", "test"], ["2hop__25396_212301", "dev"], ["2hop__15650_77173", "test"], ["4hop1__567737_141375_458768_33677", "test"], ["2hop__89403_602275", "test"], ["2hop__46862_68042", "test"], ["3hop1__208108_547811_41132", "test"], ["2hop__75781_512508", "test"], ["2hop__458672_20057", "test"], ["3hop1__409517_547811_41132", "test"], ["3hop1__145427_106426_157788", "dev"], ["2hop__656411_345851", "test"], ["2hop__3131_3300", "test"], ["2hop__76314_177869", "test"], ["2hop__31085_23241", "dev"], ["3hop2__79978_90098_10557", "dev"], ["2hop__87589_362941", "test"], ["3hop2__209741_89048_66294", "test"], ["2hop__606439_127399", "dev"], ["2hop__485221_544917", "dev"], ["2hop__13731_801817", "dev"], ["4hop2__9988_158279_70784_79935", "test"], ["2hop__488811_30110", "test"], ["3hop1__68802_89752_75165", "test"], ["4hop1__668178_765799_282674_759393", "test"], ["2hop__728887_159109", "test"], ["4hop2__103790_39078_8987_8529", "test"], ["3hop1__178435_547811_41132", "dev"], ["3hop2__87184_38738_76291", "dev"], ["3hop2__92991_27069_76291", "test"], ["3hop2__368986_89048_66294", "test"], ["2hop__153573_44085", "test"], ["2hop__729651_631003", "test"], ["3hop1__865037_214799_259594", "test"], ["3hop1__649930_15840_36002", "test"], ["2hop__337223_56873", "test"], ["3hop1__131783_131926_13165", "test"], ["2hop__208602_11402", "test"], ["2hop__95970_456836", "test"], ["4hop1__696484_49925_13759_736921", "test"], ["4hop3__673447_132409_145082_35031", "test"], ["3hop1__158848_66089_86107", "test"], ["2hop__510860_22402", "test"], ["2hop__115515_779396", "dev"], ["2hop__481809_184805", "test"], ["4hop1__370129_621192_10659_59955", "test"], ["2hop__125412_544665", "dev"], ["3hop1__135392_87694_124169", "dev"], ["2hop__2682_577502", "test"], ["3hop1__786067_228453_10972", "dev"], ["4hop3__129456_29339_508306_70744", "test"], ["2hop__135029_712629", "dev"], ["4hop1__159467_725495_49925_349426", "test"], ["4hop3__373866_5189_38229_86687", "test"], ["3hop2__156850_90098_10557", "dev"], ["2hop__676045_7292", "dev"], ["3hop1__595371_40769_64047", "dev"], ["3hop1__9285_5188_6375", "dev"], ["2hop__226204_69048", "test"], ["3hop1__504362_443779_52195", "test"], ["2hop__665660_135481", "test"], ["2hop__141365_417697", "test"], ["2hop__58691_192417", "test"], ["2hop__25344_88628", "test"], ["2hop__2683_577502", "dev"], ["4hop3__130276_29339_508306_70744", "test"], ["3hop1__152880_131926_89261", "test"], ["2hop__605207_135019", "dev"], ["3hop1__796649_720914_27537", "test"], ["4hop1__638706_378185_282674_759393", "test"], ["3hop2__132957_76377_40768", "test"], ["4hop1__436202_378185_282674_759393", "dev"], ["3hop1__841091_339990_54675", "test"], ["2hop__32887_578772", "dev"], ["2hop__128420_375952", "dev"], ["2hop__167577_31122", "test"], ["4hop2__161602_474028_88460_20985", "dev"], ["3hop1__813285_774554_64412", "test"], ["2hop__144303_483189", "dev"], ["2hop__391857_162018", "test"], ["2hop__35137_22940", "test"], ["3hop2__90098_79978_10557", "test"], ["2hop__564984_139312", "test"], ["3hop2__222979_840908_40768", "test"], ["2hop__623931_656446", "dev"], ["2hop__80070_89752", "test"], ["3hop1__109185_720914_27537", "test"], ["3hop1__689538_547811_80702", "test"], ["4hop2__71753_158279_70784_61381", "test"], ["2hop__123104_77615", "dev"], ["3hop2__132957_132967_40768", "test"], ["4hop1__88342_49853_128008_54974", "test"], ["2hop__151750_141308", "test"], ["2hop__835710_7298", "dev"], ["3hop1__374018_160713_77246", "dev"], ["3hop1__376309_160713_77246", "test"], ["2hop__386236_350892", "dev"], ["4hop3__31333_29339_508306_70744", "dev"], ["3hop2__782397_467331_162182", "dev"], ["2hop__65576_80026", "test"], ["3hop2__304722_238553_63959", "dev"], ["3hop2__569322_223623_162182", "test"], ["4hop1__373866_5186_10659_23296", "test"], ["2hop__501624_181960", "test"], ["3hop1__64694_34179_23375", "test"], ["2hop__40697_46621", "test"], ["3hop1__79039_131926_89261", "test"], ["3hop1__222497_737465_93723", "test"], ["4hop3__152056_698586_57596_54362", "dev"], ["4hop1__341156_49925_13759_736921", "test"], ["3hop1__135659_87694_124169", "dev"], ["2hop__36269_63853", "test"], ["3hop2__132957_51933_40768", "test"], ["3hop1__488744_443779_52195", "test"], ["2hop__739909_807845", "test"], ["2hop__390947_232243", "dev"], ["4hop1__9002_698949_157828_239539", "test"], ["3hop1__816887_127905_80286", "dev"], ["4hop1__88342_49853_128008_80286", "test"], ["2hop__203437_152093", "test"], ["2hop__527400_159673", "test"], ["2hop__35137_22899", "test"], ["3hop1__462960_160545_34751", "dev"], ["3hop1__160324_131905_41948", "dev"], ["4hop2__105527_14670_8987_8529", "dev"], ["3hop1__139773_88110_77129", "test"], ["2hop__552546_70131", "test"], ["2hop__62369_84616", "dev"], ["2hop__39420_39345", "test"], ["2hop__304852_758385", "dev"], ["2hop__9238_9237", "dev"], ["3hop1__609479_503371_21711", "dev"], ["2hop__506099_20057", "test"], ["2hop__21103_16334", "dev"], ["2hop__157498_127908", "dev"], ["3hop1__17192_78396_157843", "test"], ["2hop__517145_367341", "test"], ["2hop__645367_68633", "test"], ["2hop__71900_712629", "test"], ["3hop1__208108_547811_80702", "test"], ["3hop2__49541_140712_51068", "test"], ["3hop1__786067_228453_86925", "test"], ["3hop1__321720_650651_7262", "test"], ["3hop2__304722_747004_63959", "dev"], ["2hop__580458_139390", "test"], ["3hop2__125104_127483_10557", "dev"], ["2hop__646493_5385", "dev"], ["2hop__780238_110949", "dev"], ["2hop__286621_67370", "test"], ["4hop3__372437_88460_30152_20999", "test"], ["2hop__475070_532404", "dev"], ["3hop1__222497_309482_27537", "dev"], ["2hop__593633_82341", "test"], ["2hop__736167_74735", "dev"], ["2hop__829474_123956", "dev"], ["4hop1__711773_508773_85832_745702", "test"], ["2hop__814671_82341", "dev"], ["2hop__243084_523417", "test"], ["3hop1__389069_132457_51423", "test"], ["4hop3__439878_88460_30152_20999", "test"], ["3hop1__139787_88110_77129", "test"], ["3hop1__483843_40769_64047", "test"], ["2hop__122140_37168", "test"], ["3hop1__106864_160713_77246", "test"], ["2hop__524623_82341", "test"], ["2hop__563476_61845", "test"], ["3hop2__90098_125104_10557", "test"], ["2hop__88406_83289", "test"], ["2hop__648648_173389", "test"], ["2hop__796221_18803", "dev"], ["2hop__611178_131968", "test"], ["2hop__257831_123283", "test"], ["2hop__19007_60935", "dev"], ["3hop1__136129_87694_64412", "dev"], ["2hop__34595_160249", "test"], ["3hop2__304722_330033_63959", "test"], ["3hop1__109185_720914_41132", "test"], ["4hop1__31050_725495_49925_349426", "test"], ["4hop1__788226_32392_823060_610794", "dev"], ["4hop1__88342_75218_128008_86588", "dev"], ["4hop1__88342_49853_128008_87812", "test"], ["2hop__21103_16335", "dev"], ["3hop1__145411_443779_52195", "test"], ["4hop1__399219_765799_282674_759393", "test"], ["2hop__62020_158262", "dev"], ["2hop__642686_7292", "test"], ["3hop1__857241_503371_21711", "dev"], ["3hop1__152880_131926_87157", "dev"], ["2hop__802991_443552", "test"], ["3hop2__127483_156850_10557", "test"], ["4hop3__257340_88460_30152_20999", "test"], ["2hop__85463_54688", "test"], ["2hop__15368_17873", "test"], ["2hop__434218_29898", "test"], ["2hop__6956_192417", "dev"], ["2hop__683203_72813", "test"], ["2hop__107690_124896", "dev"], ["3hop1__857_846_7702", "test"], ["3hop1__579562_629431_64412", "dev"], ["2hop__455392_40315", "test"], ["2hop__763840_144857", "dev"], ["2hop__442516_595885", "dev"], ["2hop__293131_110222", "test"], ["4hop3__827497_354093_254885_76356", "test"], ["3hop2__30152_105395_20999", "dev"], ["2hop__536283_67465", "dev"], ["3hop2__90098_84553_10557", "test"], ["2hop__29349_92763", "test"], ["2hop__135963_486257", "test"], ["2hop__50199_59409", "test"], ["2hop__160151_23241", "test"], ["4hop3__316459_41402_145282_13584", "test"], ["2hop__370564_71701", "dev"], ["3hop1__135794_87694_64412", "dev"], ["2hop__41158_35124", "test"], ["4hop3__700962_88460_30152_20999", "test"], ["2hop__91021_19033", "test"], ["2hop__57638_90991", "test"], ["2hop__131455_11960", "test"], ["3hop2__67618_30587_44003", "test"], ["2hop__161151_50883", "test"], ["3hop1__850160_15840_36002", "test"], ["2hop__862994_69048", "test"], ["2hop__599396_19320", "dev"], ["2hop__713299_55227", "dev"], ["3hop1__458916_665330_51423", "test"], ["2hop__557496_57594", "test"], ["4hop1__15607_59747_211319_557671", "test"], ["4hop3__316459_41402_599212_13584", "test"], ["2hop__124932_278133", "test"], ["2hop__414193_176917", "test"], ["3hop1__438364_443779_52195", "test"], ["2hop__37656_36240", "test"], ["2hop__153532_72380", "dev"], ["2hop__10900_21567", "test"], ["2hop__123166_47134", "test"], ["4hop1__3496_5274_458768_33633", "test"], ["3hop1__83970_685536_47686", "test"], ["2hop__37656_160040", "test"], ["2hop__276637_126089", "test"], ["3hop1__857_846_7874", "dev"], ["2hop__701895_752697", "test"], ["2hop__301011_820181", "dev"], ["2hop__520033_123956", "dev"], ["3hop1__67704_237521_291682", "test"], ["2hop__826864_17335", "test"], ["2hop__677160_110882", "dev"], ["2hop__80834_21969", "test"], ["2hop__157737_46621", "dev"], ["2hop__20536_96912", "test"], ["3hop1__66872_709625_84283", "dev"], ["2hop__89403_122023", "dev"], ["3hop1__102087_214799_259594", "test"], ["4hop3__661103_698586_91248_54362", "test"], ["4hop1__105688_17130_70784_61381", "test"], ["4hop1__131113_59747_211319_557671", "test"], ["2hop__471509_136043", "dev"], ["2hop__149862_108549", "dev"], ["3hop1__751782_2053_52946", "test"], ["2hop__85931_108632", "test"], ["2hop__498251_126089", "test"], ["2hop__40485_40502", "test"], ["2hop__326140_9237", "dev"], ["2hop__199513_13732", "dev"], ["4hop1__668178_378185_282674_759393", "test"], ["4hop2__71753_623626_70784_79935", "test"], ["2hop__155781_155922", "test"], ["2hop__723101_150107", "test"], ["3hop1__92788_648104_13493", "dev"], ["3hop1__93963_170667_18967", "dev"], ["3hop1__131102_39743_24526", "test"], ["3hop1__622145_42197_18397", "test"], ["3hop2__31398_2130_2232", "test"], ["3hop1__158139_19788_15107", "dev"], ["3hop2__132957_295815_40768", "test"], ["2hop__31326_31270", "dev"], ["4hop1__320457_508773_85832_745702", "dev"], ["2hop__412633_15781", "test"], ["3hop1__135793_87694_64412", "test"], ["2hop__80_59409", "test"], ["3hop1__92251_34179_23375", "test"], ["2hop__503085_110222", "test"], ["2hop__43283_86694", "dev"], ["3hop1__28404_160713_77246", "dev"], ["2hop__252521_80650", "dev"], ["2hop__51074_75184", "dev"], ["3hop2__38242_92991_76291", "dev"], ["4hop1__60000_59747_211319_557671", "test"], ["2hop__132909_33564", "test"], ["2hop__159741_161674", "test"], ["3hop2__573858_326964_7713", "test"], ["4hop1__342858_131850_159767_81096", "test"], ["2hop__669160_139339", "dev"], ["3hop1__640171_228453_10972", "test"], ["4hop2__161602_474028_88460_18966", "test"], ["4hop1__802394_153080_33897_81096", "test"], ["2hop__6735_6733", "test"], ["4hop1__512794_508773_85832_745702", "dev"], ["4hop1__58323_375563_161848_84681", "test"], ["3hop1__305282_282081_73772", "test"], ["3hop1__302383_503371_21711", "dev"], ["2hop__141456_6455", "dev"], ["3hop2__88342_93066_47738", "dev"], ["2hop__635544_110949", "dev"], ["4hop3__387712_132409_146285_35031", "dev"], ["4hop1__342858_131850_33952_34109", "test"], ["4hop1__860115_798482_131926_89261", "test"], ["3hop2__65854_88342_58657", "dev"], ["2hop__446324_620302", "test"], ["4hop1__94201_642284_131926_89261", "dev"], ["4hop2__161324_14670_8987_8529", "test"], ["3hop1__104762_24918_24991", "dev"], ["4hop1__636860_49925_13759_736921", "test"], ["4hop1__129721_32392_823060_610794", "dev"], ["2hop__83785_461854", "test"], ["2hop__142699_67465", "test"], ["3hop2__91678_90098_10557", "test"], ["3hop1__742464_15840_36014", "dev"], ["3hop2__222979_132957_40768", "test"], ["2hop__156085_436709", "test"], ["2hop__840174_78851", "test"], ["3hop2__127483_60649_10557", "test"], ["3hop1__145427_106426_77199", "test"], ["2hop__215898_67465", "dev"], ["2hop__71016_75184", "test"], ["4hop2__161602_474028_88460_63559", "test"], ["2hop__335899_124498", "test"], ["2hop__10620_49084", "test"], ["2hop__121145_561444", "test"], ["3hop1__783711_503371_21711", "dev"], ["2hop__29367_29376", "test"], ["2hop__13353_85402", "test"], ["2hop__778212_22458", "test"], ["2hop__21104_16335", "dev"], ["2hop__47257_43786", "test"], ["4hop3__474028_88460_30152_20999", "dev"], ["3hop2__304722_423353_63959", "dev"], ["3hop1__639955_834494_34109", "test"], ["3hop1__139862_2053_5289", "test"], ["2hop__689472_52835", "test"], ["2hop__170823_120171", "test"], ["3hop1__285585_538202_84283", "dev"], ["4hop3__316459_41402_146281_13584", "test"], ["2hop__370911_589237", "dev"], ["3hop1__840353_503371_21711", "test"], ["2hop__121386_37168", "test"], ["2hop__747758_58067", "dev"], ["4hop3__130404_29339_508306_70744", "test"], ["4hop1__814776_378185_282674_759393", "dev"], ["4hop1__726675_508773_85832_745702", "test"], ["2hop__29752_814677", "test"], ["3hop1__756602_831637_91775", "test"], ["2hop__29606_600504", "dev"], ["4hop2__71753_158985_70784_61381", "test"], ["2hop__131951_643670", "test"], ["2hop__543050_245420", "test"], ["2hop__829358_754719", "test"], ["2hop__809096_491515", "dev"], ["3hop2__132957_593597_40768", "dev"], ["2hop__135822_46621", "test"], ["3hop1__198276_709625_84283", "test"], ["2hop__26642_66233", "dev"], ["2hop__445963_6095", "test"], ["2hop__507722_124896", "dev"], ["3hop1__433054_831637_91775", "test"], ["4hop1__130276_59747_211319_557671", "test"], ["2hop__362107_652514", "dev"], ["3hop1__106900_443779_52195", "test"], ["3hop2__89854_90327_76291", "test"], ["2hop__84971_67668", "test"], ["2hop__358582_189042", "test"], ["2hop__516918_126537", "test"], ["2hop__511296_2684", "dev"], ["2hop__157251_556157", "test"], ["2hop__603365_120259", "test"], ["3hop1__617062_127905_46748", "test"], ["2hop__13548_13529", "dev"], ["2hop__223823_29905", "dev"], ["2hop__81659_52835", "test"], ["2hop__32441_274635", "test"], ["4hop3__661103_698586_89595_54362", "test"], ["2hop__72487_510545", "dev"], ["3hop1__136129_87694_124169", "dev"], ["2hop__136179_13529", "test"], ["2hop__159816_37168", "test"], ["2hop__144763_599630", "test"], ["3hop1__68981_91191_16530", "test"], ["4hop1__342858_131850_33952_34053", "dev"], ["3hop1__49236_89752_54221", "test"], ["4hop2__567956_14670_8987_8974", "test"], ["2hop__726892_131968", "test"], ["2hop__551235_310309", "test"], ["4hop1__342858_131850_33952_33939", "test"], ["2hop__728969_131890", "test"], ["4hop1__364476_439408_131976_90181", "dev"], ["2hop__215124_83837", "test"], ["2hop__51760_122023", "test"], ["3hop2__9988_73205_70784", "test"], ["2hop__463884_807572", "test"], ["3hop2__92991_72083_76291", "test"], ["2hop__46376_158105", "test"], ["2hop__60686_229757", "test"], ["2hop__403580_37168", "test"], ["3hop2__326964_7707_7713", "test"], ["2hop__66149_85990", "test"], ["2hop__711946_269414", "dev"], ["4hop1__444892_32392_823060_610794", "test"], ["2hop__63483_786853", "test"], ["3hop1__133292_40769_64047", "dev"], ["3hop1__699454_831637_91775", "test"], ["3hop1__652852_2053_5289", "dev"], ["3hop1__33845_42197_18397", "test"], ["3hop1__603558_87694_64412", "test"], ["3hop1__107195_160713_77246", "test"], ["4hop2__161602_426860_88460_21074", "dev"], ["2hop__67057_52871", "dev"], ["3hop2__851134_326964_7713", "test"], ["2hop__24441_36630", "test"], ["2hop__136214_483189", "test"], ["2hop__511454_120259", "test"], ["4hop3__800262_466199_695123_72134", "dev"], ["2hop__538735_3299", "test"], ["2hop__175168_110222", "test"], ["2hop__8496_21567", "test"], ["2hop__785711_63853", "dev"], ["4hop3__275416_24325_156850_10557", "test"], ["2hop__538735_3296", "test"], ["2hop__788087_150107", "dev"], ["3hop1__818422_160545_34751", "test"], ["2hop__57638_615257", "dev"], ["3hop2__38326_92991_76291", "test"], ["3hop2__304722_582839_63959", "test"], ["3hop1__145194_160545_62931", "test"], ["2hop__799936_20273", "test"], ["2hop__66167_615257", "test"], ["3hop1__23058_20608_36327", "test"], ["2hop__475808_74481", "test"], ["2hop__61664_786853", "test"], ["4hop1__324759_765799_282674_759393", "test"], ["4hop1__860115_798482_131926_90707", "test"], ["4hop2__161602_426860_88460_21062", "test"], ["2hop__133241_33564", "test"], ["4hop1__105002_17130_70784_61381", "test"], ["3hop1__820301_720914_27537", "dev"], ["2hop__35989_623159", "dev"], ["3hop2__304722_740784_63959", "test"], ["2hop__25788_25855", "test"], ["4hop1__473563_309800_355213_203322", "test"], ["3hop2__106842_30645_84681", "test"], ["2hop__201704_135481", "test"], ["2hop__734057_88628", "test"], ["2hop__28287_89399", "test"], ["3hop1__586087_339990_54675", "dev"], ["2hop__64537_482364", "test"], ["3hop1__773338_42197_18397", "test"], ["3hop2__326964_846_7713", "test"], ["4hop3__813358_24325_125104_10557", "dev"], ["2hop__34594_160249", "test"], ["2hop__807257_590611", "test"], ["3hop1__695430_831637_91775", "dev"], ["2hop__107238_64918", "test"], ["2hop__87053_319330", "test"], ["2hop__64372_83289", "test"], ["3hop2__831456_467331_162182", "dev"], ["3hop1__24980_42197_18397", "dev"], ["2hop__38828_89399", "test"], ["3hop1__38738_73181_68042", "dev"], ["4hop3__668721_132409_371500_35031", "dev"], ["3hop1__131820_59747_90139", "test"], ["4hop3__857_846_326964_7713", "test"], ["3hop1__136083_87694_64412", "test"], ["2hop__753260_71701", "test"], ["2hop__145288_502838", "test"], ["2hop__84937_148696", "test"], ["3hop2__132957_379231_40768", "test"], ["2hop__785711_73244", "test"], ["4hop1__370129_621192_10659_42311", "test"], ["3hop1__398232_326948_78782", "dev"], ["2hop__510647_507085", "test"], ["4hop3__839787_88460_30152_20999", "test"], ["4hop3__312119_132409_371500_35031", "dev"], ["2hop__373759_136043", "test"], ["3hop1__131820_59747_48534", "dev"], ["3hop1__6362_14336_21111", "test"], ["3hop1__639955_834494_33939", "test"], ["2hop__362912_97805", "test"], ["2hop__422295_8311", "test"], ["2hop__31094_31122", "dev"], ["3hop1__408237_400692_51423", "dev"], ["2hop__603598_25719", "dev"], ["2hop__284613_87295", "test"], ["2hop__544760_35739", "dev"], ["2hop__815205_590407", "dev"], ["3hop1__559908_42197_18397", "dev"], ["4hop1__152146_5274_458768_33632", "test"], ["3hop1__726561_132457_51423", "dev"], ["2hop__71165_807969", "test"], ["2hop__61714_64633", "test"], ["2hop__591435_51329", "dev"], ["2hop__61714_52026", "test"], ["4hop1__130623_32392_823060_610794", "dev"], ["3hop1__857_846_7846", "dev"], ["4hop3__673447_132409_223216_35031", "dev"], ["3hop2__855821_467331_162182", "dev"], ["2hop__1667_40502", "test"], ["4hop3__373866_5189_276146_86687", "dev"], ["2hop__766973_770570", "test"], ["3hop2__89048_228_66294", "dev"], ["2hop__37097_86208", "dev"], ["4hop1__46196_497223_15840_36002", "test"], ["2hop__161872_21567", "test"], ["2hop__704217_82341", "test"], ["3hop1__161433_33952_33939", "dev"], ["2hop__705035_152093", "dev"], ["3hop1__749581_443779_52195", "dev"], ["3hop1__836699_15840_36002", "test"], ["2hop__207597_63853", "test"], ["3hop1__157807_19788_15107", "test"], ["2hop__347_59409", "dev"], ["2hop__96411_47902", "dev"], ["3hop1__831162_831637_91775", "test"], ["2hop__135653_669373", "test"], ["2hop__495316_660579", "test"], ["4hop1__726391_153080_159767_81096", "test"], ["3hop1__354480_834494_34053", "test"], ["2hop__434218_29905", "test"], ["3hop1__77966_91191_156667", "test"], ["2hop__2148_78851", "test"], ["3hop2__27069_92991_76291", "test"], ["2hop__34734_92852", "test"], ["2hop__216256_141338", "dev"], ["3hop1__140855_2053_52946", "test"], ["4hop1__130480_53706_795904_580996", "test"], ["4hop2__71753_685934_70784_79935", "test"], ["2hop__771727_93362", "test"], ["4hop2__161602_474028_88460_126088", "dev"], ["3hop1__465684_160545_34754", "test"], ["2hop__31585_61048", "dev"], ["4hop1__72247_497223_15840_36002", "test"], ["2hop__417937_158105", "dev"], ["3hop2__127483_79978_10557", "test"], ["2hop__196348_150107", "test"], ["4hop3__39836_29339_508306_70744", "dev"], ["4hop1__88342_49853_128008_67954", "test"], ["3hop2__326964_7845_7713", "test"], ["2hop__145203_34088", "test"], ["3hop2__90098_156850_10557", "test"], ["2hop__51965_165532", "test"], ["4hop1__271272_508773_85832_745702", "dev"], ["2hop__13106_158105", "test"], ["2hop__20256_71302", "test"], ["2hop__540035_113442", "test"], ["2hop__341176_711757", "test"], ["2hop__861047_110949", "test"], ["4hop2__103790_14670_8987_8974", "dev"], ["2hop__458131_174974", "test"], ["4hop2__160585_39078_8987_8974", "dev"], ["3hop2__326964_734214_7713", "dev"], ["4hop1__107309_457883_650651_7262", "test"], ["2hop__279729_20057", "test"], ["2hop__825727_584042", "dev"], ["3hop1__103881_443779_52195", "test"], ["2hop__242753_658778", "test"], ["4hop2__9988_685934_70784_61381", "test"], ["3hop1__273571_834494_34109", "dev"], ["2hop__814856_110882", "dev"], ["2hop__612719_6455", "test"], ["2hop__861157_135481", "dev"], ["2hop__351045_47134", "test"], ["3hop2__49541_121067_51068", "test"], ["3hop1__131820_59747_60748", "test"], ["2hop__2613_59409", "dev"], ["2hop__14092_8311", "test"], ["3hop1__104237_214799_259594", "dev"], ["2hop__557743_92763", "test"], ["2hop__35137_22943", "test"], ["3hop1__132968_40769_64047", "test"], ["4hop1__860115_798482_131926_87157", "test"], ["4hop3__152056_698586_91248_54362", "test"], ["3hop1__459153_544076_84283", "test"], ["2hop__73069_88165", "test"], ["2hop__89991_58067", "dev"], ["2hop__128945_47686", "dev"], ["2hop__151455_120171", "dev"], ["3hop1__765565_326948_78782", "dev"], ["4hop1__89632_32392_823060_610794", "test"], ["3hop1__139862_2053_52946", "dev"], ["2hop__86689_728109", "dev"], ["4hop1__17192_17130_70784_79935", "test"], ["3hop1__462960_160545_62931", "test"], ["2hop__2150_78851", "test"], ["2hop__17854_17873", "test"], ["3hop1__662413_42197_18397", "test"], ["2hop__77601_814677", "test"], ["4hop1__342858_131850_33952_34088", "test"], ["2hop__3739_13529", "test"], ["2hop__734582_71302", "dev"], ["3hop2__326964_851134_7713", "test"], ["4hop1__638988_17130_70784_79935", "dev"], ["2hop__274529_5385", "dev"], ["2hop__51113_84616", "test"], ["2hop__6827_88924", "test"], ["2hop__401710_456439", "test"], ["2hop__628752_538661", "test"], ["4hop1__708184_153080_33897_81096", "test"], ["2hop__95673_121319", "test"], ["3hop1__255970_831637_91775", "dev"], ["2hop__70490_21567", "test"], ["2hop__50910_177869", "dev"], ["3hop1__28841_547811_80702", "test"], ["2hop__42747_83837", "test"], ["2hop__6827_49664", "dev"], ["2hop__771477_149987", "dev"], ["3hop1__751782_2053_5289", "test"], ["2hop__53433_59035", "dev"], ["2hop__143650_152023", "test"], ["3hop1__64957_87694_64412", "dev"], ["3hop1__132798_40769_64047", "test"], ["4hop3__220945_88460_30152_20999", "test"], ["2hop__850984_7292", "test"], ["2hop__29191_59955", "test"], ["2hop__742831_156034", "dev"], ["2hop__130508_32379", "test"], ["2hop__703767_37168", "dev"], ["3hop2__668732_467331_162182", "dev"], ["2hop__463353_80420", "dev"], ["3hop2__28476_14777_9331", "test"], ["2hop__22930_23241", "dev"], ["3hop1__128516_30587_83479", "test"], ["2hop__462179_643013", "test"], ["3hop1__635099_131926_89261", "test"], ["3hop1__506062_792411_51423", "test"], ["3hop1__140786_2053_52946", "test"], ["2hop__817096_15463", "test"], ["2hop__91626_96912", "test"], ["3hop1__635099_131926_87157", "dev"], ["3hop1__604797_443779_52195", "test"], ["4hop1__277409_49925_13759_736921", "test"], ["2hop__36747_18974", "test"], ["2hop__559273_152023", "test"], ["4hop3__638064_88460_30152_20999", "dev"], ["2hop__20268_42004", "test"], ["2hop__821368_14251", "test"], ["3hop2__36852_613770_7713", "test"], ["3hop1__365934_87694_124169", "test"], ["3hop1__194006_42197_18397", "dev"], ["2hop__126299_396277", "test"], ["2hop__865961_159109", "test"], ["2hop__363216_787353", "dev"], ["2hop__156658_155922", "test"], ["2hop__25788_990", "dev"], ["4hop2__161602_474028_88460_21074", "test"], ["3hop1__847684_339990_54675", "test"], ["4hop1__310029_153080_159767_81096", "test"], ["3hop1__354480_834494_34109", "dev"], ["2hop__70338_160040", "test"], ["3hop1__848113_15840_36014", "dev"], ["3hop1__465684_160545_62931", "test"], ["3hop1__222497_309482_41132", "test"], ["4hop1__161765_53706_795904_580996", "test"], ["2hop__622308_61845", "test"], ["4hop1__236903_153080_33897_81096", "test"], ["2hop__289517_139339", "test"], ["4hop1__88342_75218_128008_48717", "test"], ["2hop__170341_43945", "dev"], ["2hop__149710_108549", "test"], ["3hop1__715607_792411_51423", "test"], ["3hop1__857_846_7849", "test"], ["2hop__557411_652060", "dev"], ["4hop3__719125_132409_371500_35031", "test"], ["4hop2__161602_474028_88460_21057", "test"], ["2hop__147212_60904", "test"], ["3hop1__742780_567566_84283", "test"], ["4hop1__638706_765799_282674_759393", "test"], ["2hop__835171_141338", "test"], ["2hop__61714_80069", "dev"], ["2hop__96414_47902", "dev"], ["2hop__58939_213732", "dev"], ["3hop1__837871_831637_91775", "test"], ["2hop__435184_67370", "test"], ["2hop__39063_212301", "dev"], ["3hop2__60649_90098_10557", "test"], ["2hop__2299_38663", "test"]], "corpus_sha1": "9e1068af919025baa7b62fab47d4ce5ddf1267ed"} \ No newline at end of file diff --git a/benchmarks/multihop/methods.py b/benchmarks/multihop/methods.py new file mode 100644 index 0000000..bb68a58 --- /dev/null +++ b/benchmarks/multihop/methods.py @@ -0,0 +1,279 @@ +"""Retrieval methods and metrics. Every function returns rankings as a list of +int arrays (passage ids, best first, length <= MAXK) so all methods are scored by +the same code.""" +from __future__ import annotations + +import re +import time + +import numpy as np +import scipy.sparse as sp + +from common import KS, import_lib + +MAXK = max(KS) + + +# ---------------------------------------------------------------------------- +# metrics +# ---------------------------------------------------------------------------- +def metrics_at(rankings, questions, k): + """Per-question arrays for a returned list truncated to k. + recall = |gold ∩ top-k| / |gold|; all = 1 if every gold passage is within top-k; + mrr = 1/rank of the first gold passage within top-k (0 if none: MRR@k); + n_ret = number of passages actually returned (<= k).""" + n = len(questions) + rec, alls, mrr, nret = np.zeros(n), np.zeros(n), np.zeros(n), np.zeros(n) + for i, (r, q) in enumerate(zip(rankings, questions)): + gold = set(q["gold"]) + r = [int(x) for x in r[:k]] + nret[i] = len(r) + hit = len(gold.intersection(r)) + rec[i] = hit / len(gold) + alls[i] = float(hit == len(gold)) + for pos, p in enumerate(r): + if p in gold: + mrr[i] = 1.0 / (pos + 1) + break + return {"recall": rec, "all": alls, "mrr": mrr, "n_ret": nret} + + +def set_metrics(sets, questions): + """Metrics for unordered returned sets (no MRR, no k): recall, all, size.""" + n = len(questions) + rec, alls, size = np.zeros(n), np.zeros(n), np.zeros(n) + for i, (s, q) in enumerate(zip(sets, questions)): + gold = set(q["gold"]) + hit = len(gold.intersection(s)) + rec[i] = hit / len(gold) + alls[i] = float(hit == len(gold)) + size[i] = len(s) + return {"recall": rec, "all": alls, "size": size} + + +# ---------------------------------------------------------------------------- +# dense vector search (what any vector store does) +# ---------------------------------------------------------------------------- +def dense_scores(Q, P): + return Q @ P.T # cosine: both L2-normalised + + +def topn_from_scores(S, n): + idx = np.argpartition(-S, n, axis=1)[:, :n] + rows = np.arange(S.shape[0])[:, None] + order = np.argsort(-S[rows, idx], axis=1, kind="stable") + return idx[rows, order] + + +def vector_rankings(Q, P, n=MAXK): + S = dense_scores(Q, P) + return list(topn_from_scores(S, n)), S + + +# ---------------------------------------------------------------------------- +# BM25 (Okapi, k1=1.2 b=0.75), own implementation +# ---------------------------------------------------------------------------- +_STOP = set("""a an and are as at be but by for from has have he her his i in is it its of on or she that the their there +they this to was were which who whom will with you your we our us not no do does did so than then these those what when +where while about after also been being both can could during each had how into more most other over such up very would""".split()) +_TOK = re.compile(r"[a-z0-9]+") + + +def _tok(s): + return [t for t in _TOK.findall(s.lower()) if t not in _STOP] + + +class BM25: + def __init__(self, docs, k1=1.2, b=0.75): + vocab, rows, cols, vals = {}, [], [], [] + lens = [] + for i, d in enumerate(docs): + toks = _tok(d) + lens.append(len(toks)) + tf = {} + for t in toks: + j = vocab.setdefault(t, len(vocab)) + tf[j] = tf.get(j, 0) + 1 + for j, c in tf.items(): + rows.append(i) + cols.append(j) + vals.append(c) + n = len(docs) + self.vocab = vocab + lens = np.array(lens, dtype=np.float64) + avg = lens.mean() + tfm = sp.csr_matrix((vals, (rows, cols)), shape=(n, len(vocab)), dtype=np.float64) + df = np.asarray((tfm > 0).sum(0)).ravel() + idf = np.log(1.0 + (n - df + 0.5) / (df + 0.5)) + tfm = tfm.tocoo() + denom = tfm.data + k1 * (1 - b + b * lens[tfm.row] / avg) + w = tfm.data * (k1 + 1) / denom * idf[tfm.col] + self.W = sp.csr_matrix((w, (tfm.row, tfm.col)), shape=tfm.shape).T.tocsr() # vocab x docs + + def query_matrix(self, queries): + rows, cols = [], [] + for i, q in enumerate(queries): + for t in set(_tok(q)): + j = self.vocab.get(t) + if j is not None: + rows.append(i) + cols.append(j) + return sp.csr_matrix((np.ones(len(rows)), (rows, cols)), shape=(len(queries), len(self.vocab))) + + def scores(self, queries): + return np.asarray((self.query_matrix(queries) @ self.W).todense(), dtype=np.float32) + + +def rrf(rank_lists_full, n=MAXK, k0=60): + """Reciprocal rank fusion over full-score matrices' rank positions. + rank_lists_full: list of score matrices (Q x N).""" + fused = None + for S in rank_lists_full: + order = np.argsort(-S, axis=1, kind="stable") + ranks = np.empty_like(order) + rows = np.arange(S.shape[0])[:, None] + ranks[rows, order] = np.arange(S.shape[1])[None, :] + contrib = 1.0 / (k0 + ranks + 1) + fused = contrib if fused is None else fused + contrib + return list(topn_from_scores(fused.astype(np.float32), n)) + + +# ---------------------------------------------------------------------------- +# cross-encoder rerank of vector top-N +# ---------------------------------------------------------------------------- +def rerank_scores(queries, cand, corpus, model_id, batch=128): + import torch + from sentence_transformers import CrossEncoder + ce = CrossEncoder(model_id, device="cuda" if torch.cuda.is_available() else "cpu", max_length=256) + pairs = [] + for q, c in zip(queries, cand): + for p in c: + pairs.append((q, f"{corpus[int(p)]['title']}. {corpus[int(p)]['text']}")) + s = ce.predict(pairs, batch_size=batch, show_progress_bar=False) + return np.asarray(s, dtype=np.float32).reshape(len(queries), -1) + + +def rerank_rankings(cand, scores, n=MAXK): + out = [] + for c, s in zip(cand, scores): + o = np.argsort(-s, kind="stable")[:n] + out.append(np.asarray(c)[o]) + return out + + +# ---------------------------------------------------------------------------- +# Personalised PageRank re-ranking (independent of the library; HippoRAG-style) +# ---------------------------------------------------------------------------- +def build_adjacency(n, edges, fams, directed_mention, weighted): + """Sparse adjacency A (row = source). mention edges are directed unless + directed_mention is False (then symmetrised); entity/knn are symmetric.""" + rows, cols, vals = [], [], [] + for f in fams: + if f.startswith("mention"): + s, d = edges[f] + rows += [s]; cols += [d]; vals += [np.ones(len(s))] + if not directed_mention: + rows += [d]; cols += [s]; vals += [np.ones(len(s))] + else: + a, b, w = edges[f] + w = w.astype(np.float64) if weighted else np.ones(len(a)) + rows += [a, b]; cols += [b, a]; vals += [w, w] + if not rows: + return sp.csr_matrix((n, n)) + A = sp.coo_matrix((np.concatenate(vals), (np.concatenate(rows), np.concatenate(cols))), shape=(n, n)).tocsr() + A.sum_duplicates() + return A + + +def ppr_rankings(S, A, seeds_k, alpha, beta, n=MAXK, iters=25, block=256): + """S: Q x N dense cosine scores. Seeds = top `seeds_k` by cosine, weight ∝ max(sim,0). + p <- alpha * P^T p + (1-alpha) * s (dangling mass restarts at s), 25 power iterations. + final = (1-beta) * p/max(p) + beta * max(sim,0)/max(sim) -> top n.""" + nq, N = S.shape + deg = np.asarray(A.sum(1)).ravel() + inv = np.where(deg > 0, 1.0 / np.maximum(deg, 1e-12), 0.0) + PT = (sp.diags(inv) @ A).T.tocsr().astype(np.float32) # column-stochastic for non-dangling nodes + dangling = (deg == 0).astype(np.float32) + out = [] + for s0 in range(0, nq, block): + Sb = S[s0:s0 + block] + nb = Sb.shape[0] + top = np.argpartition(-Sb, seeds_k, axis=1)[:, :seeds_k] + rows = np.arange(nb)[:, None] + R = np.zeros((N, nb), dtype=np.float32) + R[top, rows] = np.maximum(Sb[rows, top], 0).astype(np.float32) + R += 0.0 + sums = R.sum(0, keepdims=True) + sums[sums == 0] = 1.0 + R /= sums + p = R.copy() + for _ in range(iters): + dm = (dangling[:, None] * p).sum(0, keepdims=True) + p = alpha * (PT @ p) + (alpha * dm + (1 - alpha)) * R + p = p.T + pn = p / np.maximum(p.max(1, keepdims=True), 1e-12) + sim = np.maximum(Sb, 0) + sn = sim / np.maximum(sim.max(1, keepdims=True), 1e-12) + fin = (1 - beta) * pn + beta * sn + out += list(topn_from_scores(fin.astype(np.float32), n)) + return out + + +# ---------------------------------------------------------------------------- +# ISONGraph Vector (library under test, unmodified) +# ---------------------------------------------------------------------------- +def build_lib_graph(corpus, P, edges, fams): + lib = import_lib() + from ison_graph import Direction # noqa: F401 + g = lib.SemanticGraph(name="mh", directed=True, embedding_db=":memory:", auto_embed=False) + for c in corpus: + g.add_node("passage", c["id"], title=c["title"]) + g.embedding_store.add_batch([(("passage", c["id"]), c["title"], P[c["id"]].tolist()) for c in corpus]) + ref = lambda i: ("passage", int(i)) # noqa: E731 + for f in fams: + if f.startswith("mention"): + s, d = edges[f] + for a, b in zip(s.tolist(), d.tolist()): + g.add_edge("MENTIONS", ref(a), ref(b), w=1.0) + else: + a_, b_, w_ = edges[f] + rel = {"entity": "SHARES_ENTITY", "entity_rand": "SHARES_ENTITY", "knn": "NEAR"}[f] + for a, b, w in zip(a_.tolist(), b_.tolist(), w_.tolist()): + g.add_edge(rel, ref(a), ref(b), w=float(w)) + g.add_edge(rel, ref(b), ref(a), w=float(w)) + return g + + +def lib_multihop_rankings(g, Q, params, n=MAXK, idx=None, timing=None): + """Run semantic_multi_hop with a precomputed query vector for each query.""" + from ison_graph import Direction + direction = Direction.BOTH if params["direction"] == "both" else Direction.OUT + out = [] + qs = range(len(Q)) if idx is None else idx + for i in qs: + t0 = time.perf_counter() + res = g.semantic_multi_hop( + Q[i].tolist(), max_hops=params["hops"], top_k_seeds=params["seeds"], top_k_results=n, + direction=direction, decay=params["decay"], threshold=-1.0, blend=params["blend"], + weight_field=params["weight_field"]) + if timing is not None: + timing.append(time.perf_counter() - t0) + out.append(np.array([r.node_ref[1] for r in res], dtype=np.int64)) + return out + + +def lib_subgraph_sets(g, Q, params, k, connect, connect_hops, idx=None, timing=None): + from ison_graph import Direction + direction = Direction.BOTH if params["direction"] == "both" else Direction.OUT + out = [] + qs = range(len(Q)) if idx is None else idx + for i in qs: + t0 = time.perf_counter() + sub = g.semantic_subgraph( + Q[i].tolist(), max_hops=params["hops"], top_k_seeds=params["seeds"], top_k_results=k, + direction=direction, decay=params["decay"], threshold=-1.0, blend=params["blend"], + weight_field=params["weight_field"], connect=connect, connect_max_hops=connect_hops) + if timing is not None: + timing.append(time.perf_counter() - t0) + out.append(set(int(n.id) for n in sub.nodes())) + return out diff --git a/benchmarks/multihop/prepare_data.py b/benchmarks/multihop/prepare_data.py new file mode 100644 index 0000000..ef80714 --- /dev/null +++ b/benchmarks/multihop/prepare_data.py @@ -0,0 +1,118 @@ +"""Sample questions and build one pooled retrieval corpus per dataset. + +Sources (dev sets only, pinned HuggingFace revisions): + hotpotqa : hotpotqa/hotpot_qa distractor/validation rev 1908d6af... + 2wiki : framolfese/2WikiMultihopQA data/validation rev fe713bfb... + musique : dgslibisey/MuSiQue musique_ans_v1.0_dev rev c8f4f8c9... + (the answerable dev split of the official MuSiQue v1.0 release) + +For each dataset: N_QUESTIONS questions are sampled with a fixed seed; their +context paragraphs (gold + the dataset's own distractors) are pooled and +de-duplicated by (title, text) into ONE corpus that every question is searched +against. Questions are split into a `dev` split (hyper-parameter selection) +and a `test` split (reported numbers). + +Outputs (outside the repo, in MH_DATA_DIR): corpus_<ds>.json, questions_<ds>.json +Outputs (in the repo): manifests/<ds>_sample.json (ids+split) +""" +from __future__ import annotations + +import hashlib +import json +import random +from collections import Counter + +from common import (DATA_DIR, MANIFEST_DIR, N_DEV, N_QUESTIONS, RAW_DIR, SEED, + dump_json) + +SOURCES = { + "hotpotqa": {"repo": "hotpotqa/hotpot_qa", "revision": "1908d6afbbead072334abe2965f91bd2709910ab", + "file": "distractor/validation-00000-of-00001.parquet"}, + "2wiki": {"repo": "framolfese/2WikiMultihopQA", "revision": "fe713bfbd1afbca1a65246741a75890405d56a3a", + "file": "data/validation-00000-of-00001.parquet"}, + "musique": {"repo": "dgslibisey/MuSiQue", "revision": "c8f4f8c9465fb69d31a8eae894c3fd509c4ca321", + "file": "musique_ans_v1.0_dev.jsonl"}, +} + + +def download(ds): + from huggingface_hub import hf_hub_download + s = SOURCES[ds] + return hf_hub_download(s["repo"], s["file"], repo_type="dataset", revision=s["revision"], + local_dir=str(RAW_DIR / s["repo"].replace("/", "__"))) + + +def load_raw(ds): + """-> list of dict(id, question, type, hops, paragraphs=[(title,text)], gold=indices)""" + path = download(ds) + out = [] + if ds in ("hotpotqa", "2wiki"): + import pandas as pd + df = pd.read_parquet(path) + for _, r in df.iterrows(): + titles = list(r["context"]["title"]) + sents = r["context"]["sentences"] + paras = [(t, " ".join(x.strip() for x in s).strip()) for t, s in zip(titles, sents)] + gold_titles = set(r["supporting_facts"]["title"]) + gold = [i for i, (t, _) in enumerate(paras) if t in gold_titles] + missing = gold_titles - {t for t, _ in paras} + if missing: + continue # supporting title absent from the context: drop + out.append({"id": r["id"], "question": r["question"], "type": r["type"], + "hops": len(gold_titles), "paragraphs": paras, "gold": gold}) + else: + with open(path, encoding="utf-8") as fh: + for line in fh: + r = json.loads(line) + paras = [(p["title"], p["paragraph_text"].strip()) for p in r["paragraphs"]] + gold = [i for i, p in enumerate(r["paragraphs"]) if p["is_supporting"]] + prefix = r["id"].split("_")[0] + out.append({"id": r["id"], "question": r["question"], "type": prefix, + "hops": int(prefix[0]), "paragraphs": paras, "gold": gold}) + return out + + +def build(ds): + rows = load_raw(ds) + rows.sort(key=lambda r: r["id"]) + rng = random.Random(SEED) + sample = rng.sample(rows, min(N_QUESTIONS, len(rows))) + order = list(range(len(sample))) + rng.shuffle(order) + split = {} + for rank, i in enumerate(order): + split[sample[i]["id"]] = "dev" if rank < N_DEV else "test" + + key2pid, corpus = {}, [] + questions = [] + for r in sample: + pids = [] + for title, text in r["paragraphs"]: + k = (title, text) + if k not in key2pid: + key2pid[k] = len(corpus) + corpus.append({"id": len(corpus), "title": title, "text": text}) + pids.append(key2pid[k]) + gold = sorted({pids[i] for i in r["gold"]}) + questions.append({"id": r["id"], "question": r["question"], "type": r["type"], "hops": r["hops"], + "gold": gold, "n_gold": len(gold), "split": split[r["id"]], + "context": pids}) + dump_json(corpus, DATA_DIR / f"corpus_{ds}.json") + dump_json(questions, DATA_DIR / f"questions_{ds}.json") + dump_json({"dataset": ds, "source": SOURCES[ds], "seed": SEED, "n_questions": len(questions), + "n_dev": sum(q["split"] == "dev" for q in questions), + "n_test": sum(q["split"] == "test" for q in questions), + "corpus_size": len(corpus), "n_available": len(rows), + "ids": [[q["id"], q["split"]] for q in questions], + "corpus_sha1": hashlib.sha1(json.dumps([(c["title"], c["text"]) for c in corpus]).encode()).hexdigest()}, + MANIFEST_DIR / f"{ds}_sample.json") + c = Counter(q["type"] for q in questions) + h = Counter(q["n_gold"] for q in questions) + print(f"{ds}: available={len(rows)} sampled={len(questions)} corpus={len(corpus)} " + f"passages (raw pooled {sum(len(r['paragraphs']) for r in sample)}) types={dict(c)} gold-count={dict(h)}") + + +if __name__ == "__main__": + import sys + for ds in (sys.argv[1:] or SOURCES): + build(ds) diff --git a/benchmarks/multihop/report.py b/benchmarks/multihop/report.py new file mode 100644 index 0000000..23bf6e0 --- /dev/null +++ b/benchmarks/multihop/report.py @@ -0,0 +1,312 @@ +"""Aggregate raw per-question results: bootstrap CIs, paired sign-flip tests, +Holm correction, breakdowns by question type / hop count. Writes +results/summary.json and results/tables.md.""" +from __future__ import annotations + +import json +from collections import defaultdict + +import numpy as np + +import graphs +from headline import headline_md +from common import DATASETS, KS, MODELS, DATA_DIR, RESULTS_DIR, load_json +from jobs import RAW + +B_BOOT = 2000 +B_PERM = 10000 +RNG_SEED = 20240607 +BASELINES = ["vector", "bm25", "rrf", "rerank"] +VARIANTS = list(graphs.VARIANTS) # real graphs (eligible for dev-based selection) +ALLV = list(graphs.ALL_VARIANTS) # + negative controls +NICE = {"vector": "Vector top-k (plain)", "bm25": "BM25", "rrf": "Hybrid vector+BM25 (RRF)", + "rerank": "Vector top-30 + cross-encoder", "lib_mh": "ISONGraph multi_hop (tuned)", + "lib_default": "ISONGraph multi_hop (library defaults)", "ppr": "PPR re-rank (tuned, independent impl.)"} + + +def vname(v): + return v + + +def holm(pvals): + order = np.argsort(pvals) + m = len(pvals) + adj = np.empty(m) + run = 0.0 + for rank, i in enumerate(order): + run = max(run, (m - rank) * pvals[i]) + adj[i] = min(1.0, run) + return adj + + +class Stats: + def __init__(self, n, seed=RNG_SEED): + rng = np.random.default_rng(seed) + self.n = n + self.boot = rng.integers(0, n, size=(B_BOOT, n)) + self.signs = rng.choice(np.array([-1.0, 1.0], dtype=np.float32), size=(B_PERM, n)) + + def ci(self, x): + means = x[self.boot].mean(1) + return float(x.mean()), float(np.percentile(means, 2.5)), float(np.percentile(means, 97.5)) + + def diff(self, x, y): + d = (x - y).astype(np.float32) + obs = float(d.mean()) + means = d[self.boot].mean(1) + lo, hi = float(np.percentile(means, 2.5)), float(np.percentile(means, 97.5)) + perm = np.abs((self.signs @ d) / self.n) + p = (1 + int((perm >= abs(obs) - 1e-12).sum())) / (1 + B_PERM) + return obs, lo, hi, p + + +def fmt_ci(t, pct=True, nd=1): + m, lo, hi = t + if pct: + return f"{100*m:.{nd}f} [{100*lo:.{nd}f}, {100*hi:.{nd}f}]" + return f"{m:.3f} [{lo:.3f}, {hi:.3f}]" + + +def fmt_d(t, nd=1): + d, lo, hi, p = t + return f"{100*d:+.{nd}f} [{100*lo:+.{nd}f}, {100*hi:+.{nd}f}]" + + +def fp(p): + return "<0.001" if p < 0.001 else f"{p:.3f}" + + +def load_all(): + data = {} + for ds in DATASETS: + qs = load_json(DATA_DIR / f"questions_{ds}.json") + test = np.array([i for i, q in enumerate(qs) if q["split"] == "test"]) + dev = np.array([i for i, q in enumerate(qs) if q["split"] == "dev"]) + for m in MODELS: + e = {"qs": qs, "test": test, "dev": dev, "meta": {}, "arr": {}, "devarr": {}, "sets": {}} + bf = RAW / f"{ds}__{m}__baselines.npz" + if not bf.exists(): + continue + z = np.load(bf) + for key in z.files: + meth, k, mk = key.split("|") + e["arr"][(meth, int(k), mk)] = z[key][test] + e["devarr"][(meth, int(k), mk)] = z[key][dev] + e["meta"]["baselines"] = json.load(open(RAW / f"{ds}__{m}__baselines.json", encoding="utf-8")) + for v in ALLV: + f = RAW / f"{ds}__{m}__{v.replace('+', '_')}.npz" + if not f.exists(): + continue + z = np.load(f) + for key in z.files: + meth, split, k, mk = key.split("|") + if meth.startswith("lib_sub"): + e["sets"][(meth, v, int(k), mk)] = z[key] + else: + (e["arr"] if split == "test" else e["devarr"])[(f"{meth}:{v}", int(k), mk)] = z[key] + e["meta"][v] = json.load(open(f.with_suffix(".json"), encoding="utf-8")) + data[(ds, m)] = e + return data + + +def best_variant(e, meth, k): + """Graph variant chosen on DEV (all@k, then recall@k) for a method.""" + best, bv = None, None + for v in VARIANTS: + if (f"{meth}:{v}", k, "all") not in e["devarr"]: + continue + s = (e["devarr"][(f"{meth}:{v}", k, "all")].mean(), e["devarr"][(f"{meth}:{v}", k, "recall")].mean()) + if best is None or s > best: + best, bv = s, v + return bv + + +def get(e, name, k, mk): + """name: 'vector' | 'lib_mh@dev-best' | 'lib_mh:variant' ...""" + if name.endswith("@best"): + meth = name[:-5] + v = best_variant(e, meth, k) + return e["arr"].get((f"{meth}:{v}", k, mk)), v + return e["arr"].get((name, k, mk)), None + + +def run(): + data = load_all() + out = {"B_boot": B_BOOT, "B_perm": B_PERM, "results": {}} + md = [] + for (ds, model), e in data.items(): + n_test = len(e["test"]) + st = Stats(n_test) + res = out["results"].setdefault(ds, {}).setdefault(model, {}) + res["n_test"] = n_test + res["corpus_size"] = e["meta"]["baselines"]["corpus_size"] + res["lib_similarity_search_same_top20_set"] = e["meta"]["baselines"]["lib_similarity_search_same_top20_set"] + res["graphs"] = {v: e["meta"][v]["diagnostics"] for v in ALLV if v in e["meta"]} + res["selected"] = {v: {"lib": e["meta"][v]["lib_selected"], "ppr": e["meta"][v]["ppr_selected"]} + for v in ALLV if v in e["meta"]} + md.append(f"\n## {ds} / {model} (corpus {res['corpus_size']} passages, {n_test} test questions)\n") + for k in KS: + rows = [] # (label, variant, per-question arrays dict) + names = BASELINES + [f"lib_mh@best", f"ppr@best", f"lib_default@best"] + \ + [f"lib_mh:{v}" for v in ALLV] + [f"ppr:{v}" for v in ALLV] + [f"lib_default:{v}" for v in ALLV] + vec = {mk: e["arr"][("vector", k, mk)] for mk in ("recall", "all", "mrr", "n_ret")} + table = [] + for nm in names: + arrs = {} + var = None + for mk in ("recall", "all", "mrr", "n_ret"): + a, var = get(e, nm, k, mk) + if a is None: + break + arrs[mk] = a + if len(arrs) < 4: + continue + label = nm + base = nm.split(":")[0].split("@")[0] + if "@best" in nm: + label = f"{NICE[base]} [graph chosen on dev: {var}]" + elif ":" in nm: + label = f"{NICE[base]} [graph: {nm.split(':')[1]}]" + else: + label = NICE[nm] + rec = {"label": label, "key": nm, "variant": var or (nm.split(":")[1] if ":" in nm else None)} + for mk in ("recall", "all", "mrr"): + rec[mk] = st.ci(arrs[mk]) + rec["n_ret"] = float(arrs["n_ret"].mean()) + if nm != "vector": + rec["d_all"] = st.diff(arrs["all"], vec["all"]) + rec["d_recall"] = st.diff(arrs["recall"], vec["recall"]) + rec["d_mrr"] = st.diff(arrs["mrr"], vec["mrr"]) + table.append(rec) + # Holm over the main comparisons (non-@best rows are exploratory; Holm family = all rows) + for dk in ("d_all", "d_recall"): + idx = [i for i, r in enumerate(table) if dk in r] + adj = holm(np.array([table[i][dk][3] for i in idx])) + for i, a in zip(idx, adj): + table[i][dk + "_holm"] = float(a) + res.setdefault("tables", {})[str(k)] = table + md.append(f"\n### k = {k} (returned-passage budget = {k})\n") + md.append("| Method | Recall@k % [95% CI] | All-supporting@k % [95% CI] | MRR@k [95% CI] | mean returned | " + "Δ all vs vector (pp) [95% CI] | p (Holm) | Δ recall vs vector (pp) [95% CI] | p (Holm) |") + md.append("|---|---|---|---|---|---|---|---|---|") + for r in table: + dall = fmt_d(r["d_all"]) if "d_all" in r else "-" + pa = f"{fp(r['d_all'][3])} ({fp(r['d_all_holm'])})" if "d_all" in r else "-" + drec = fmt_d(r["d_recall"]) if "d_recall" in r else "-" + pr = f"{fp(r['d_recall'][3])} ({fp(r['d_recall_holm'])})" if "d_recall" in r else "-" + md.append(f"| {r['label']} | {fmt_ci(r['recall'])} | {fmt_ci(r['all'])} | {fmt_ci(r['mrr'], pct=False)} | " + f"{r['n_ret']:.1f} | {dall} | {pa} | {drec} | {pr} |") + + # ---- semantic_subgraph(connect=True): size-matched comparison ---- + vtop = np.load(RAW / f"{ds}__{model}__vector_top50.npy")[e["test"]] + sub_rows = [] + for v in ALLV: + for ch in (2, 3): + key = (f"lib_sub_c{ch}", v, k, "recall") + if key not in e["sets"]: + continue + srec, sall, ssize = (e["sets"][(f"lib_sub_c{ch}", v, k, x)] for x in ("recall", "all", "size")) + # vector top-|S_q| per query + vrec, vall = np.zeros(n_test), np.zeros(n_test) + for qi, q in enumerate(e["test"]): + gold = set(e["qs"][q]["gold"]) + s = int(min(ssize[qi], 50)) + hit = len(gold.intersection(int(x) for x in vtop[qi][:s])) + vrec[qi] = hit / len(gold) + vall[qi] = float(hit == len(gold)) + sub_rows.append({ + "variant": v, "connect_max_hops": ch, "size": float(ssize.mean()), + "recall": st.ci(srec), "all": st.ci(sall), + "vec_matched_recall": st.ci(vrec), "vec_matched_all": st.ci(vall), + "d_all": st.diff(sall, vall), "d_recall": st.diff(srec, vrec)}) + if sub_rows: + res.setdefault("subgraph", {})[str(k)] = sub_rows + md.append(f"\n#### semantic_subgraph(connect=True), top_k_results={k}: set quality vs plain vector top-|S| (size-matched per query)\n") + md.append("| Graph | connect_max_hops | mean |S| | Recall % | All-supp. % | Vector@|S| recall % | Vector@|S| all % | Δ all (pp) [CI] | p | Δ recall (pp) [CI] | p |") + md.append("|---|---|---|---|---|---|---|---|---|---|---|") + for r in sub_rows: + md.append(f"| {r['variant']} | {r['connect_max_hops']} | {r['size']:.1f} | {fmt_ci(r['recall'])} | {fmt_ci(r['all'])} | " + f"{fmt_ci(r['vec_matched_recall'])} | {fmt_ci(r['vec_matched_all'])} | {fmt_d(r['d_all'])} | {fp(r['d_all'][3])} | " + f"{fmt_d(r['d_recall'])} | {fp(r['d_recall'][3])} |") + + # ---- library vs independent PPR on the same graph (paired) ---- + md.append(f"\n#### {ds}/{model}: ISONGraph multi_hop (tuned) minus PPR re-rank (tuned), same graph; all-supporting (pp [95% CI], paired p)\n") + md.append("| Graph | k=5 | k=10 | k=20 |\n|---|---|---|---|") + lp = res.setdefault("lib_minus_ppr", {}) + for v in VARIANTS: + cells = [] + for k in KS: + a = e["arr"].get((f"lib_mh:{v}", k, "all")) + b = e["arr"].get((f"ppr:{v}", k, "all")) + if a is None or b is None: + cells.append("n/a") + continue + d = st.diff(a, b) + lp.setdefault(v, {})[str(k)] = d + cells.append(f"{fmt_d(d)} (p={fp(d[3])})") + if any(c != "n/a" for c in cells): + md.append(f"| {v} | " + " | ".join(cells) + " |") + + # ---- breakdowns at k=10 ---- + k = 10 + qs_test = [e["qs"][i] for i in e["test"]] + for attr, title in (("type", "question type"), ("hops", "hop count (number of supporting passages)")): + groups = defaultdict(list) + for i, q in enumerate(qs_test): + groups[q[attr]].append(i) + md.append(f"\n#### {ds}/{model}: breakdown by {title}, k=10 -- All-supporting@10 % (Δ vs vector, pp [95% CI])\n") + cols = ["vector", "rrf", "rerank", "lib_mh@best", "ppr@best"] + md.append("| Group | n | " + " | ".join(NICE[c.split('@')[0]] if c != "vector" else NICE[c] for c in cols) + " |") + md.append("|---|---|" + "---|" * len(cols)) + bd = res.setdefault("breakdown", {}).setdefault(attr, {}) + for gname in sorted(groups, key=str): + ix = np.array(groups[gname]) + sts = Stats(len(ix), seed=RNG_SEED + 1) + vecall = e["arr"][("vector", k, "all")][ix] + cells = [] + bd[str(gname)] = {"n": int(len(ix))} + for c in cols: + a, var = get(e, c, k, "all") + if a is None: + cells.append("n/a") + continue + a = a[ix] + cell = f"{100*a.mean():.1f}" + if c != "vector": + d = sts.diff(a, vecall) + cell += f" ({fmt_d(d)})" + if var: + cell += f" [{var}]" + cells.append(cell) + bd[str(gname)][c] = float(a.mean()) + md.append(f"| {gname} | {len(ix)} | " + " | ".join(cells) + " |") + + # ---- hyper-parameters / dev-vs-test optimism ---- + md.append(f"\n#### {ds}/{model}: dev vs test All-supporting@10 for tuned configs (selection optimism)\n") + md.append("| Method | Graph | dev % | test % |\n|---|---|---|---|") + for meth in ("lib_mh", "ppr"): + for v in ALLV: + kd = (f"{meth}:{v}", 10, "all") + if kd in e["devarr"]: + md.append(f"| {meth} | {v} | {100*e['devarr'][kd].mean():.1f} | {100*e['arr'][kd].mean():.1f} |") + md.append(f"| vector | - | {100*e['devarr'][('vector',10,'all')].mean():.1f} | {100*e['arr'][('vector',10,'all')].mean():.1f} |") + + md.append("\n## Graph diagnostics (gold used for REPORTING only)\n") + md.append("| Dataset | model | Graph | undirected edges | mean degree | isolated % | gold pairs directly linked % | gold pairs linked within 2 hops % | nbrs of gold inside same question context % |\n|---|---|---|---|---|---|---|---|---|") + for (ds, model), e in data.items(): + for v in ALLV: + if v in e["meta"]: + d = e["meta"][v]["diagnostics"] + md.append(f"| {ds} | {model} | {v} | {d['n_edges_undirected_union']} | {d['mean_degree']:.1f} | {100*d['isolated_frac']:.0f} | " + f"{100*d['gold_pairs_directly_linked']:.0f} | {100*d['gold_pairs_linked_within_2']:.0f} | " + f"{100*(d['neighbours_of_gold_that_are_in_same_question_context'] or 0):.0f} |") + md.append("\n## Selected hyper-parameters (dev), ISONGraph multi_hop, k=10\n") + for (ds, model), e in data.items(): + for v in ALLV: + if v in e["meta"]: + md.append(f"- {ds}/{model}/{v}: lib `{e['meta'][v]['lib_selected']['10']}`; ppr `{e['meta'][v]['ppr_selected']['10']}`") + (RESULTS_DIR / "tables.md").write_text("\n".join(md), encoding="utf-8") + (RESULTS_DIR / "headline.md").write_text(headline_md(out, DATASETS, MODELS, ALLV, fmt_ci, fmt_d), encoding="utf-8") + with open(RESULTS_DIR / "summary.json", "w", encoding="utf-8") as fh: + json.dump(out, fh, ensure_ascii=False, indent=1, default=float) + print("wrote tables.md, summary.json") diff --git a/benchmarks/multihop/results/headline.md b/benchmarks/multihop/results/headline.md new file mode 100644 index 0000000..93bd5ab --- /dev/null +++ b/benchmarks/multihop/results/headline.md @@ -0,0 +1,65 @@ + +### Headline, k = 10: all-supporting@10 % [95% CI] (Delta vs plain vector in pp, paired p) + +| Method | hotpotqa/minilm | hotpotqa/bge | 2wiki/minilm | 2wiki/bge | musique/minilm | musique/bge | +|---|---|---|---|---|---|---| +| Vector top-k (plain) | 58.2 [55.2, 61.3] | 78.5 [76.0, 80.9] | 32.5 [29.6, 35.2] | 43.9 [40.9, 46.9] | 18.2 [15.8, 20.6] | 20.9 [18.4, 23.4] | +| BM25 | 69.1 [66.3, 71.9] (+10.9, p<0.001) | 69.1 [66.3, 71.9] (-9.4, p<0.001) | 41.5 [38.5, 44.5] (+9.0, p<0.001) | 41.5 [38.5, 44.5] (-2.4, p=0.019) | 15.2 [13.0, 17.3] (-3.0, p=0.013) | 15.2 [13.0, 17.3] (-5.7, p<0.001) | +| Hybrid RRF (vector+BM25) | 72.5 [69.7, 75.1] (+14.3, p<0.001) | 83.8 [81.4, 86.0] (+5.3, p<0.001) | 41.1 [38.1, 44.1] (+8.6, p<0.001) | 45.6 [42.4, 48.5] (+1.7, p=0.046) | 20.4 [17.8, 22.9] (+2.2, p=0.033) | 23.3 [20.8, 26.0] (+2.4, p=0.040) | +| Vector top-30 + cross-encoder | 68.4 [65.7, 71.1] (+10.2, p<0.001) | 84.1 [81.9, 86.3] (+5.6, p<0.001) | 37.5 [34.5, 40.2] (+5.0, p<0.001) | 46.0 [43.0, 49.0] (+2.1, p=0.011) | 23.7 [21.2, 26.3] (+5.5, p<0.001) | 24.6 [22.1, 27.1] (+3.7, p<0.001) | +| ISONGraph multi_hop, library defaults | 77.3 [74.7, 79.8] (+19.1, p<0.001) [mention] | 87.5 [85.4, 89.5] (+9.0, p<0.001) [mention] | 79.5 [77.0, 81.9] (+47.0, p<0.001) [mention] | 94.6 [93.2, 96.0] (+50.7, p<0.001) [mention] | 24.5 [21.8, 27.3] (+6.3, p<0.001) [mention] | 26.3 [23.6, 29.0] (+5.4, p<0.001) [mention] | +| ISONGraph multi_hop, tuned | 81.5 [79.1, 83.8] (+23.3, p<0.001) [mention] | 91.5 [89.7, 93.2] (+13.0, p<0.001) [mention] | 81.0 [78.6, 83.4] (+48.5, p<0.001) [mention+entity] | 94.6 [93.2, 96.0] (+50.7, p<0.001) [mention] | 27.4 [24.7, 30.2] (+9.2, p<0.001) [mention] | 29.6 [26.8, 32.4] (+8.7, p<0.001) [mention] | +| PPR re-rank, tuned (independent impl.) | 81.4 [79.0, 83.7] (+23.2, p<0.001) [mention] | 91.9 [90.1, 93.5] (+13.4, p<0.001) [mention] | 78.7 [76.2, 81.1] (+46.2, p<0.001) [mention] | 92.8 [91.2, 94.4] (+48.9, p<0.001) [mention] | 28.9 [26.1, 31.6] (+10.7, p<0.001) [mention] | 29.1 [26.4, 31.9] (+8.2, p<0.001) [mention] | +| CONTROL: multi_hop on label-shuffled mention graph | 58.2 [55.2, 61.3] (+0.0, p=1.000) | 78.4 [75.9, 80.8] (-0.1, p=1.000) | 32.5 [29.6, 35.2] (+0.0, p=1.000) | 43.7 [40.7, 46.7] (-0.2, p=0.497) | 18.2 [15.8, 20.6] (+0.0, p=1.000) | 20.9 [18.4, 23.4] (+0.0, p=1.000) | +| CONTROL: multi_hop on label-shuffled entity graph | n/a | n/a | 32.5 [29.6, 35.2] (+0.0, p=1.000) | 43.8 [40.8, 46.8] (-0.1, p=1.000) | n/a | n/a | + +### Headline, k = 5: all-supporting@5 % [95% CI] (Delta vs plain vector in pp, paired p) + +| Method | hotpotqa/minilm | hotpotqa/bge | 2wiki/minilm | 2wiki/bge | musique/minilm | musique/bge | +|---|---|---|---|---|---|---| +| Vector top-k (plain) | 47.2 [44.1, 50.3] | 68.4 [65.5, 71.2] | 26.2 [23.5, 28.7] | 38.3 [35.3, 41.2] | 11.9 [10.0, 13.9] | 14.3 [12.2, 16.4] | +| BM25 | 48.7 [45.7, 51.7] (+1.5, p=0.437) | 48.7 [45.7, 51.7] (-19.7, p<0.001) | 33.8 [31.1, 36.7] (+7.6, p<0.001) | 33.8 [31.1, 36.7] (-4.5, p<0.001) | 11.1 [9.3, 12.9] (-0.8, p=0.503) | 11.1 [9.3, 12.9] (-3.2, p=0.003) | +| Hybrid RRF (vector+BM25) | 54.3 [51.1, 57.4] (+7.1, p<0.001) | 69.1 [66.3, 71.9] (+0.7, p=0.621) | 33.2 [30.3, 36.0] (+7.0, p<0.001) | 38.7 [35.8, 41.8] (+0.4, p=0.695) | 12.6 [10.6, 14.6] (+0.7, p=0.451) | 14.9 [12.8, 17.2] (+0.6, p=0.585) | +| Vector top-30 + cross-encoder | 59.7 [56.6, 62.6] (+12.5, p<0.001) | 71.6 [68.8, 74.4] (+3.2, p=0.028) | 33.8 [30.9, 36.5] (+7.6, p<0.001) | 39.4 [36.4, 42.4] (+1.1, p=0.289) | 17.8 [15.5, 20.2] (+5.9, p<0.001) | 18.9 [16.6, 21.2] (+4.6, p<0.001) | +| ISONGraph multi_hop, library defaults | 67.6 [64.8, 70.4] (+20.4, p<0.001) [mention] | 73.4 [70.8, 76.1] (+5.0, p<0.001) [mention] | 53.5 [50.5, 56.6] (+27.3, p<0.001) [mention] | 47.7 [44.7, 50.8] (+9.4, p<0.001) [mention] | 18.2 [15.9, 20.6] (+6.3, p<0.001) [mention] | 17.3 [15.1, 19.7] (+3.0, p<0.001) [mention] | +| ISONGraph multi_hop, tuned | 71.1 [68.3, 73.8] (+23.9, p<0.001) [mention] | 81.3 [78.8, 83.6] (+12.9, p<0.001) [mention] | 63.8 [60.8, 66.8] (+37.6, p<0.001) [mention+entity] | 77.8 [75.3, 80.4] (+39.5, p<0.001) [mention] | 18.4 [16.1, 20.8] (+6.5, p<0.001) [mention] | 21.6 [19.1, 24.2] (+7.3, p<0.001) [mention] | +| PPR re-rank, tuned (independent impl.) | 63.6 [60.8, 66.4] (+16.4, p<0.001) [mention] | 79.6 [77.1, 82.0] (+11.2, p<0.001) [mention] | 51.8 [48.7, 54.9] (+25.6, p<0.001) [mention] | 58.9 [55.9, 61.9] (+20.6, p<0.001) [mention] | 18.3 [16.0, 20.6] (+6.4, p<0.001) [mention] | 20.2 [17.7, 22.6] (+5.9, p<0.001) [mention] | +| CONTROL: multi_hop on label-shuffled mention graph | 47.2 [44.1, 50.3] (+0.0, p=1.000) | 67.5 [64.5, 70.4] (-0.9, p=0.003) | 26.2 [23.5, 28.8] (+0.0, p=1.000) | 38.4 [35.4, 41.3] (+0.1, p=1.000) | 12.0 [10.1, 14.0] (+0.1, p=1.000) | 14.3 [12.2, 16.4] (+0.0, p=1.000) | +| CONTROL: multi_hop on label-shuffled entity graph | n/a | n/a | 26.2 [23.5, 28.8] (+0.0, p=1.000) | 38.4 [35.4, 41.3] (+0.1, p=1.000) | n/a | n/a | + +### Headline, k = 20: all-supporting@20 % [95% CI] (Delta vs plain vector in pp, paired p) + +| Method | hotpotqa/minilm | hotpotqa/bge | 2wiki/minilm | 2wiki/bge | musique/minilm | musique/bge | +|---|---|---|---|---|---|---| +| Vector top-k (plain) | 66.2 [63.4, 69.1] | 85.4 [83.2, 87.5] | 38.4 [35.3, 41.3] | 48.2 [45.2, 51.2] | 26.3 [23.6, 29.0] | 28.6 [25.9, 31.4] | +| BM25 | 82.2 [79.8, 84.4] (+16.0, p<0.001) | 82.2 [79.8, 84.4] (-3.2, p=0.015) | 46.7 [43.8, 49.8] (+8.3, p<0.001) | 46.7 [43.8, 49.8] (-1.5, p=0.114) | 23.5 [21.0, 26.1] (-2.8, p=0.047) | 23.5 [21.0, 26.1] (-5.1, p<0.001) | +| Hybrid RRF (vector+BM25) | 83.8 [81.6, 86.0] (+17.6, p<0.001) | 89.6 [87.8, 91.5] (+4.2, p<0.001) | 46.7 [43.7, 49.7] (+8.3, p<0.001) | 49.4 [46.4, 52.4] (+1.2, p=0.173) | 31.9 [29.0, 34.8] (+5.6, p<0.001) | 32.9 [30.1, 35.8] (+4.3, p=0.001) | +| Vector top-30 + cross-encoder | 70.3 [67.7, 73.0] (+4.1, p<0.001) | 87.5 [85.4, 89.5] (+2.1, p<0.001) | 39.0 [35.9, 41.9] (+0.6, p=0.216) | 50.1 [47.0, 53.1] (+1.9, p=0.004) | 29.2 [26.4, 32.0] (+2.9, p<0.001) | 31.6 [28.8, 34.4] (+3.0, p<0.001) | +| ISONGraph multi_hop, library defaults | 77.5 [74.9, 80.0] (+11.3, p<0.001) [mention] | 86.5 [84.4, 88.5] (+1.1, p=0.374) [knn] | 79.9 [77.4, 82.3] (+41.5, p<0.001) [mention] | 94.8 [93.4, 96.1] (+46.6, p<0.001) [mention] | 26.4 [23.8, 29.1] (+0.1, p=1.000) [mention] | 28.8 [26.0, 31.6] (+0.2, p=0.942) [mention] | +| ISONGraph multi_hop, tuned | 87.3 [85.2, 89.3] (+21.1, p<0.001) [mention] | 95.2 [93.9, 96.5] (+9.8, p<0.001) [mention] | 87.3 [85.1, 89.3] (+48.9, p<0.001) [mention+entity] | 96.4 [95.2, 97.5] (+48.2, p<0.001) [mention] | 37.7 [34.7, 40.7] (+11.4, p<0.001) [mention] | 38.7 [35.7, 41.5] (+10.1, p<0.001) [mention] | +| PPR re-rank, tuned (independent impl.) | 87.3 [85.3, 89.3] (+21.1, p<0.001) [mention] | 95.2 [93.8, 96.5] (+9.8, p<0.001) [mention] | 85.7 [83.5, 87.9] (+47.3, p<0.001) [mention] | 95.5 [94.2, 96.7] (+47.3, p<0.001) [mention] | 38.1 [35.1, 41.2] (+11.8, p<0.001) [mention] | 39.2 [36.2, 42.2] (+10.6, p<0.001) [mention] | +| CONTROL: multi_hop on label-shuffled mention graph | 66.2 [63.4, 69.1] (+0.0, p=1.000) | 85.4 [83.2, 87.5] (+0.0, p=1.000) | 38.4 [35.3, 41.3] (+0.0, p=1.000) | 48.2 [45.2, 51.2] (+0.0, p=1.000) | 26.3 [23.6, 29.0] (+0.0, p=1.000) | 28.6 [25.9, 31.4] (+0.0, p=1.000) | +| CONTROL: multi_hop on label-shuffled entity graph | n/a | n/a | 38.4 [35.3, 41.3] (+0.0, p=1.000) | 48.2 [45.2, 51.2] (+0.0, p=1.000) | n/a | n/a | + +### Every graph variant, k = 10: Delta all-supporting@10 vs plain vector, pp [95% CI] (paired p) + +| Method: graph | hotpotqa/minilm | hotpotqa/bge | 2wiki/minilm | 2wiki/bge | musique/minilm | musique/bge | +|---|---|---|---|---|---|---| +| ISONGraph multi_hop (tuned): mention | +23.3 [+20.4, +26.4] (p<0.001) | +13.0 [+10.6, +15.4] (p<0.001) | +47.1 [+43.7, +50.5] (p<0.001) | +50.7 [+47.5, +53.9] (p<0.001) | +9.2 [+7.2, +11.3] (p<0.001) | +8.7 [+6.6, +10.8] (p<0.001) | +| ISONGraph multi_hop (tuned): entity | +13.6 [+11.2, +16.0] (p<0.001) | n/a | +28.1 [+24.9, +31.1] (p<0.001) | +30.3 [+27.3, +33.0] (p<0.001) | n/a | n/a | +| ISONGraph multi_hop (tuned): knn | +4.0 [+1.9, +6.2] (p<0.001) | +4.5 [+2.5, +6.4] (p<0.001) | +7.3 [+4.8, +9.8] (p<0.001) | +20.0 [+17.5, +22.5] (p<0.001) | n/a | n/a | +| ISONGraph multi_hop (tuned): mention+entity | n/a | n/a | +48.5 [+45.2, +51.8] (p<0.001) | n/a | n/a | n/a | +| ISONGraph multi_hop (tuned): ctrl_mention | +0.0 [+0.0, +0.0] (p=1.000) | -0.1 [-0.3, +0.0] (p=1.000) | +0.0 [+0.0, +0.0] (p=1.000) | -0.2 [-0.5, +0.0] (p=0.497) | +0.0 [+0.0, +0.0] (p=1.000) | +0.0 [+0.0, +0.0] (p=1.000) | +| ISONGraph multi_hop (tuned): ctrl_entity | n/a | n/a | +0.0 [+0.0, +0.0] (p=1.000) | -0.1 [-0.3, +0.0] (p=1.000) | n/a | n/a | +| PPR re-rank (tuned): mention | +23.2 [+20.5, +26.1] (p<0.001) | +13.4 [+11.0, +15.8] (p<0.001) | +46.2 [+42.8, +49.7] (p<0.001) | +48.9 [+45.8, +52.1] (p<0.001) | +10.7 [+8.4, +13.3] (p<0.001) | +8.2 [+5.9, +10.5] (p<0.001) | +| PPR re-rank (tuned): entity | +9.2 [+7.1, +11.5] (p<0.001) | n/a | +22.4 [+19.7, +25.1] (p<0.001) | +25.8 [+23.0, +28.6] (p<0.001) | n/a | n/a | +| PPR re-rank (tuned): knn | +4.0 [+2.1, +5.9] (p<0.001) | +3.2 [+1.2, +5.2] (p=0.002) | +7.4 [+5.3, +9.6] (p<0.001) | +14.6 [+12.3, +16.8] (p<0.001) | n/a | n/a | +| PPR re-rank (tuned): mention+entity | n/a | n/a | +39.5 [+36.3, +42.7] (p<0.001) | n/a | n/a | n/a | +| PPR re-rank (tuned): ctrl_mention | +0.1 [-0.5, +0.7] (p=1.000) | -0.3 [-0.9, +0.3] (p=0.544) | +0.2 [-0.3, +0.7] (p=0.688) | +0.0 [+0.0, +0.0] (p=1.000) | +0.0 [-0.3, +0.3] (p=1.000) | -0.1 [-0.3, +0.0] (p=1.000) | +| PPR re-rank (tuned): ctrl_entity | n/a | n/a | -0.9 [-1.7, -0.1] (p=0.052) | +0.0 [-0.3, +0.3] (p=1.000) | n/a | n/a | +| ISONGraph multi_hop (defaults): mention | +19.1 [+16.2, +22.3] (p<0.001) | +9.0 [+6.6, +11.5] (p<0.001) | +47.0 [+43.5, +50.4] (p<0.001) | +50.7 [+47.5, +53.9] (p<0.001) | +6.3 [+3.8, +9.1] (p<0.001) | +5.4 [+3.0, +8.0] (p<0.001) | +| ISONGraph multi_hop (defaults): entity | +4.6 [+1.6, +7.7] (p=0.004) | n/a | +24.0 [+20.6, +27.2] (p<0.001) | +25.4 [+22.4, +28.3] (p<0.001) | n/a | n/a | +| ISONGraph multi_hop (defaults): knn | -0.5 [-3.2, +2.3] (p=0.772) | +2.4 [+0.3, +4.4] (p=0.040) | +7.3 [+4.8, +9.8] (p<0.001) | +17.5 [+15.0, +19.9] (p<0.001) | n/a | n/a | +| ISONGraph multi_hop (defaults): mention+entity | n/a | n/a | +36.9 [+33.5, +40.3] (p<0.001) | n/a | n/a | n/a | +| ISONGraph multi_hop (defaults): ctrl_mention | -11.1 [-13.1, -9.2] (p<0.001) | -10.2 [-12.2, -8.4] (p<0.001) | -6.3 [-7.9, -4.9] (p<0.001) | -5.5 [-6.9, -4.1] (p<0.001) | -6.6 [-8.2, -5.0] (p<0.001) | -6.7 [-8.3, -5.1] (p<0.001) | +| ISONGraph multi_hop (defaults): ctrl_entity | n/a | n/a | -9.6 [-11.5, -7.8] (p<0.001) | -5.8 [-7.3, -4.3] (p<0.001) | n/a | n/a | \ No newline at end of file diff --git a/benchmarks/multihop/results/latency.json b/benchmarks/multihop/results/latency.json new file mode 100644 index 0000000..b32892d --- /dev/null +++ b/benchmarks/multihop/results/latency.json @@ -0,0 +1,248 @@ +{ + "machine": { + "platform": "Windows-11-10.0.26300-SP0", + "processor": "Intel64 Family 6 Model 183 Stepping 1, GenuineIntel", + "threads": 1, + "note": "single-threaded, warm, query vectors precomputed; timed on a machine that may have had other light background processes (see README)" + }, + "results": { + "hotpotqa/minilm": { + "vector (numpy brute force top-20)": { + "mean_ms": 0.48564050026470795, + "median_ms": 0.47075003385543823, + "p95_ms": 0.5728049960453063, + "n": 200 + }, + "ISONGraph similarity_search top-20 (plain vector via the library)": { + "mean_ms": 0.6935720011824742, + "median_ms": 0.6781999836675823, + "p95_ms": 0.8835000306135042, + "n": 200 + }, + "BM25 (scipy sparse)": { + "mean_ms": 0.1536945000407286, + "median_ms": 0.14625000767409801, + "p95_ms": 0.20997496903873974, + "n": 200 + }, + "Hybrid RRF (dense + BM25 + fusion)": { + "mean_ms": 5.391248998057563, + "median_ms": 5.384700023569167, + "p95_ms": 7.199424967984668, + "n": 200 + }, + "PPR re-rank, graph=mention": { + "mean_ms": 11.179658999026287, + "median_ms": 11.421350005548447, + "p95_ms": 13.548449991503729, + "n": 200 + }, + "_graph_build_seconds": { + "mention": 0.9577896000118926 + }, + "ISONGraph semantic_multi_hop (tuned), graph=mention": { + "mean_ms": 12.342348497768398, + "median_ms": 12.086300004739314, + "p95_ms": 13.929454979370346, + "n": 200 + }, + "ISONGraph semantic_subgraph(connect=True), graph=mention": { + "mean_ms": 20.597783994162455, + "median_ms": 20.786299981409684, + "p95_ms": 23.595270034275014, + "n": 100 + }, + "_selected_cfg": { + "variant": "mention", + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + } + }, + "Vector top-30 + cross-encoder (CPU, 1 thread)": { + "mean_ms": 6843.021799999406, + "median_ms": 7238.126849988475, + "p95_ms": 7484.873605016037, + "n": 8 + } + }, + "2wiki/minilm": { + "vector (numpy brute force top-20)": { + "mean_ms": 0.7558300020173192, + "median_ms": 0.7286499603651464, + "p95_ms": 0.9162000438664106, + "n": 200 + }, + "ISONGraph similarity_search top-20 (plain vector via the library)": { + "mean_ms": 0.9439495002152398, + "median_ms": 0.8543499861843884, + "p95_ms": 1.5255200152751052, + "n": 200 + }, + "BM25 (scipy sparse)": { + "mean_ms": 0.45629350119270384, + "median_ms": 0.4443999787326902, + "p95_ms": 0.7269749941769982, + "n": 200 + }, + "Hybrid RRF (dense + BM25 + fusion)": { + "mean_ms": 3.873069500841666, + "median_ms": 3.7907000223640352, + "p95_ms": 5.11852501949761, + "n": 200 + }, + "PPR re-rank, graph=mention": { + "mean_ms": 6.827249999041669, + "median_ms": 6.732499983627349, + "p95_ms": 8.859889965970066, + "n": 200 + }, + "_graph_build_seconds": { + "mention+entity": 1.1235644000116736 + }, + "ISONGraph semantic_multi_hop (tuned), graph=mention+entity": { + "mean_ms": 3.4164134989259765, + "median_ms": 2.922200015746057, + "p95_ms": 7.105119986226778, + "n": 200 + }, + "ISONGraph semantic_subgraph(connect=True), graph=mention+entity": { + "mean_ms": 39.90853499970399, + "median_ms": 35.43980003450997, + "p95_ms": 61.64026499318424, + "n": 100 + }, + "_selected_cfg": { + "variant": "mention+entity", + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + } + } + }, + "musique/minilm": { + "vector (numpy brute force top-20)": { + "mean_ms": 2.0554560009622946, + "median_ms": 1.9247499876655638, + "p95_ms": 3.100280024227686, + "n": 200 + }, + "ISONGraph similarity_search top-20 (plain vector via the library)": { + "mean_ms": 2.2241929985466413, + "median_ms": 2.1745999983977526, + "p95_ms": 3.0762100213905796, + "n": 200 + }, + "BM25 (scipy sparse)": { + "mean_ms": 0.4085455002496019, + "median_ms": 0.3771499905269593, + "p95_ms": 0.5929799750447272, + "n": 200 + }, + "Hybrid RRF (dense + BM25 + fusion)": { + "mean_ms": 6.215742999047507, + "median_ms": 6.086799985496327, + "p95_ms": 8.073384992894717, + "n": 200 + }, + "PPR re-rank, graph=mention": { + "mean_ms": 13.699477003247011, + "median_ms": 13.619150005979463, + "p95_ms": 16.363100049784407, + "n": 200 + }, + "_graph_build_seconds": { + "mention": 1.0767072000307962 + }, + "ISONGraph semantic_multi_hop (tuned), graph=mention": { + "mean_ms": 16.80029300332535, + "median_ms": 16.35764999082312, + "p95_ms": 21.890034989337433, + "n": 200 + }, + "ISONGraph semantic_subgraph(connect=True), graph=mention": { + "mean_ms": 31.09345300239511, + "median_ms": 30.48549999948591, + "p95_ms": 38.64339004503563, + "n": 100 + }, + "_selected_cfg": { + "variant": "mention", + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + } + } + }, + "hotpotqa/bge": { + "vector (numpy brute force top-20)": { + "mean_ms": 1.3350584977888502, + "median_ms": 1.3147999998182058, + "p95_ms": 1.8358299974352117, + "n": 200 + }, + "ISONGraph similarity_search top-20 (plain vector via the library)": { + "mean_ms": 2.095423499995377, + "median_ms": 2.0103500282857567, + "p95_ms": 3.095170049346052, + "n": 200 + }, + "BM25 (scipy sparse)": { + "mean_ms": 0.5253385010291822, + "median_ms": 0.5105999880470335, + "p95_ms": 0.7368400198174639, + "n": 200 + }, + "Hybrid RRF (dense + BM25 + fusion)": { + "mean_ms": 6.03914749895921, + "median_ms": 5.834200012031943, + "p95_ms": 8.131710000452584, + "n": 200 + }, + "PPR re-rank, graph=mention": { + "mean_ms": 13.122622499649879, + "median_ms": 12.988400005269796, + "p95_ms": 15.208380005788056, + "n": 200 + }, + "_graph_build_seconds": { + "mention": 0.950012199988123 + }, + "ISONGraph semantic_multi_hop (tuned), graph=mention": { + "mean_ms": 2.1502415000577457, + "median_ms": 2.1122499892953783, + "p95_ms": 2.7931800083024423, + "n": 200 + }, + "ISONGraph semantic_subgraph(connect=True), graph=mention": { + "mean_ms": 10.563406005385332, + "median_ms": 10.059049993287772, + "p95_ms": 14.326804989832453, + "n": 100 + }, + "_selected_cfg": { + "variant": "mention", + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + } + } + } + } +} \ No newline at end of file diff --git a/benchmarks/multihop/results/latency_table.md b/benchmarks/multihop/results/latency_table.md new file mode 100644 index 0000000..344167a --- /dev/null +++ b/benchmarks/multihop/results/latency_table.md @@ -0,0 +1,17 @@ +| Method (mean ms/query; median in brackets) | hotpotqa/minilm | 2wiki/minilm | musique/minilm | hotpotqa/bge | +|---|---|---|---|---| +| vector (numpy brute force top-20) | 0.49 (0.47) | 0.76 (0.73) | 2.06 (1.92) | 1.34 (1.31) | +| ISONGraph similarity_search top-20 (plain vector via the library) | 0.69 (0.68) | 0.94 (0.85) | 2.22 (2.17) | 2.10 (2.01) | +| BM25 (scipy sparse) | 0.15 (0.15) | 0.46 (0.44) | 0.41 (0.38) | 0.53 (0.51) | +| Hybrid RRF (dense + BM25 + fusion) | 5.39 (5.38) | 3.87 (3.79) | 6.22 (6.09) | 6.04 (5.83) | +| PPR re-rank, graph=mention | 11.18 (11.42) | 6.83 (6.73) | 13.70 (13.62) | 13.12 (12.99) | +| ISONGraph semantic_multi_hop (tuned) | 12.34 (12.09) | 3.42 (2.92) | 16.80 (16.36) | 2.15 (2.11) | +| ISONGraph semantic_subgraph(connect=True) | 20.60 (20.79) | 39.91 (35.44) | 31.09 (30.49) | 10.56 (10.06) | +| Vector top-30 + cross-encoder (CPU, 1 thread) | 6843.02 (7238.13) | - | - | - | + +Timed multi_hop/subgraph configurations (dev-selected at k=10): + +- hotpotqa/minilm: graph `mention`, config `{'hops': 2, 'seeds': 5, 'decay': 0.5, 'blend': 0.3, 'direction': 'both', 'weight_field': None}` +- 2wiki/minilm: graph `mention+entity`, config `{'hops': 2, 'seeds': 5, 'decay': 0.7, 'blend': 0.0, 'direction': 'out', 'weight_field': 'w'}` +- musique/minilm: graph `mention`, config `{'hops': 2, 'seeds': 10, 'decay': 0.9, 'blend': 0.3, 'direction': 'both', 'weight_field': None}` +- hotpotqa/bge: graph `mention`, config `{'hops': 2, 'seeds': 5, 'decay': 0.7, 'blend': 0.0, 'direction': 'both', 'weight_field': None}` diff --git a/benchmarks/multihop/results/pairs_vs_nongraph.json b/benchmarks/multihop/results/pairs_vs_nongraph.json new file mode 100644 index 0000000..d789fd2 --- /dev/null +++ b/benchmarks/multihop/results/pairs_vs_nongraph.json @@ -0,0 +1,326 @@ +{ + "hotpotqa/minilm/k5/lib_mh-rrf": [ + 0.1679999977350235, + 0.13597500324249268, + 0.20200000703334808, + 9.999000099990002e-05 + ], + "hotpotqa/minilm/k5/lib_mh-rerank": [ + 0.11400000005960464, + 0.08100000023841858, + 0.14900000393390656, + 9.999000099990002e-05 + ], + "hotpotqa/minilm/k5/lib_mh-ppr_best": [ + 0.07500000298023224, + 0.04899999871850014, + 0.10000000149011612, + 9.999000099990002e-05 + ], + "hotpotqa/minilm/k10/lib_mh-rrf": [ + 0.09000000357627869, + 0.05999999865889549, + 0.12099999934434891, + 9.999000099990002e-05 + ], + "hotpotqa/minilm/k10/lib_mh-rerank": [ + 0.13099999725818634, + 0.10100000351667404, + 0.16200000047683716, + 9.999000099990002e-05 + ], + "hotpotqa/minilm/k10/lib_mh-ppr_best": [ + 0.0010000000474974513, + -0.014000000432133675, + 0.014999999664723873, + 1.0 + ], + "hotpotqa/minilm/k20/lib_mh-rrf": [ + 0.03500000014901161, + 0.008999999612569809, + 0.06002499908208847, + 0.013498650134986502 + ], + "hotpotqa/minilm/k20/lib_mh-rerank": [ + 0.17000000178813934, + 0.1420000046491623, + 0.19699999690055847, + 9.999000099990002e-05 + ], + "hotpotqa/minilm/k20/lib_mh-ppr_best": [ + 0.0, + -0.010999999940395355, + 0.010999999940395355, + 1.0 + ], + "hotpotqa/bge/k5/lib_mh-rrf": [ + 0.12200000137090683, + 0.09399999678134918, + 0.15299999713897705, + 9.999000099990002e-05 + ], + "hotpotqa/bge/k5/lib_mh-rerank": [ + 0.09700000286102295, + 0.0679750069975853, + 0.12800000607967377, + 9.999000099990002e-05 + ], + "hotpotqa/bge/k5/lib_mh-ppr_best": [ + 0.017000000923871994, + -0.006000000052154064, + 0.039000000804662704, + 0.1665833416658334 + ], + "hotpotqa/bge/k10/lib_mh-rrf": [ + 0.07699999958276749, + 0.05299999937415123, + 0.10199999809265137, + 9.999000099990002e-05 + ], + "hotpotqa/bge/k10/lib_mh-rerank": [ + 0.07400000095367432, + 0.050999999046325684, + 0.09799999743700027, + 9.999000099990002e-05 + ], + "hotpotqa/bge/k10/lib_mh-ppr_best": [ + -0.004000000189989805, + -0.014000000432133675, + 0.006000000052154064, + 0.5494450554944506 + ], + "hotpotqa/bge/k20/lib_mh-rrf": [ + 0.0560000017285347, + 0.03799999877810478, + 0.07400000095367432, + 9.999000099990002e-05 + ], + "hotpotqa/bge/k20/lib_mh-rerank": [ + 0.07699999958276749, + 0.061000000685453415, + 0.0949999988079071, + 9.999000099990002e-05 + ], + "hotpotqa/bge/k20/lib_mh-ppr_best": [ + 0.0, + -0.008999999612569809, + 0.008999999612569809, + 1.0 + ], + "2wiki/minilm/k5/lib_mh-rrf": [ + 0.3059999942779541, + 0.26897498965263367, + 0.34200000762939453, + 9.999000099990002e-05 + ], + "2wiki/minilm/k5/lib_mh-rerank": [ + 0.30000001192092896, + 0.2630000114440918, + 0.335999995470047, + 9.999000099990002e-05 + ], + "2wiki/minilm/k5/lib_mh-ppr_best": [ + 0.11999999731779099, + 0.09600000083446503, + 0.14399999380111694, + 9.999000099990002e-05 + ], + "2wiki/minilm/k10/lib_mh-rrf": [ + 0.39899998903274536, + 0.36000001430511475, + 0.4360249936580658, + 9.999000099990002e-05 + ], + "2wiki/minilm/k10/lib_mh-rerank": [ + 0.4350000023841858, + 0.4000000059604645, + 0.47200000286102295, + 9.999000099990002e-05 + ], + "2wiki/minilm/k10/lib_mh-ppr_best": [ + 0.023000000044703484, + 0.013000000268220901, + 0.03400000184774399, + 9.999000099990002e-05 + ], + "2wiki/minilm/k20/lib_mh-rrf": [ + 0.4059999883174896, + 0.36899998784065247, + 0.44200000166893005, + 9.999000099990002e-05 + ], + "2wiki/minilm/k20/lib_mh-rerank": [ + 0.4830000102519989, + 0.4519749879837036, + 0.515999972820282, + 9.999000099990002e-05 + ], + "2wiki/minilm/k20/lib_mh-ppr_best": [ + 0.01600000075995922, + 0.003000000026077032, + 0.028999999165534973, + 0.018698130186981302 + ], + "2wiki/bge/k5/lib_mh-rrf": [ + 0.39100000262260437, + 0.3569999933242798, + 0.421999990940094, + 9.999000099990002e-05 + ], + "2wiki/bge/k5/lib_mh-rerank": [ + 0.3840000033378601, + 0.35100001096725464, + 0.4169999957084656, + 9.999000099990002e-05 + ], + "2wiki/bge/k5/lib_mh-ppr_best": [ + 0.1889999955892563, + 0.1589999943971634, + 0.21899999678134918, + 9.999000099990002e-05 + ], + "2wiki/bge/k10/lib_mh-rrf": [ + 0.49000000953674316, + 0.4569999873638153, + 0.5210000276565552, + 9.999000099990002e-05 + ], + "2wiki/bge/k10/lib_mh-rerank": [ + 0.4860000014305115, + 0.45399999618530273, + 0.5180000066757202, + 9.999000099990002e-05 + ], + "2wiki/bge/k10/lib_mh-ppr_best": [ + 0.017999999225139618, + 0.007000000216066837, + 0.028999999165534973, + 0.001999800019998 + ], + "2wiki/bge/k20/lib_mh-rrf": [ + 0.4699999988079071, + 0.43799999356269836, + 0.5010249614715576, + 9.999000099990002e-05 + ], + "2wiki/bge/k20/lib_mh-rerank": [ + 0.46299999952316284, + 0.43299999833106995, + 0.4950000047683716, + 9.999000099990002e-05 + ], + "2wiki/bge/k20/lib_mh-ppr_best": [ + 0.008999999612569809, + 0.004000000189989805, + 0.014999999664723873, + 0.003999600039996 + ], + "musique/minilm/k5/lib_mh-rrf": [ + 0.057999998331069946, + 0.03700000047683716, + 0.07999999821186066, + 9.999000099990002e-05 + ], + "musique/minilm/k5/lib_mh-rerank": [ + 0.006000000052154064, + -0.01600000075995922, + 0.02902499958872795, + 0.6723327667233276 + ], + "musique/minilm/k5/lib_mh-ppr_best": [ + 0.0010000000474974513, + -0.01600000075995922, + 0.017000000923871994, + 1.0 + ], + "musique/minilm/k10/lib_mh-rrf": [ + 0.07000000029802322, + 0.04397499933838844, + 0.09799999743700027, + 9.999000099990002e-05 + ], + "musique/minilm/k10/lib_mh-rerank": [ + 0.03700000047683716, + 0.012000000104308128, + 0.06400000303983688, + 0.0047995200479952005 + ], + "musique/minilm/k10/lib_mh-ppr_best": [ + -0.014999999664723873, + -0.032999999821186066, + 0.0020000000949949026, + 0.1197880211978802 + ], + "musique/minilm/k20/lib_mh-rrf": [ + 0.057999998331069946, + 0.026000000536441803, + 0.09000000357627869, + 0.00039996000399960006 + ], + "musique/minilm/k20/lib_mh-rerank": [ + 0.08500000089406967, + 0.0560000017285347, + 0.11500000208616257, + 9.999000099990002e-05 + ], + "musique/minilm/k20/lib_mh-ppr_best": [ + -0.004000000189989805, + -0.01899999938905239, + 0.010999999940395355, + 0.6785321467853215 + ], + "musique/bge/k5/lib_mh-rrf": [ + 0.06700000166893005, + 0.044975001364946365, + 0.09200000017881393, + 9.999000099990002e-05 + ], + "musique/bge/k5/lib_mh-rerank": [ + 0.027000000700354576, + 0.0020000000949949026, + 0.054999999701976776, + 0.0463953604639536 + ], + "musique/bge/k5/lib_mh-ppr_best": [ + 0.014000000432133675, + -0.0010000000474974513, + 0.028999999165534973, + 0.09109089091090891 + ], + "musique/bge/k10/lib_mh-rrf": [ + 0.06300000101327896, + 0.03500000014901161, + 0.09200000017881393, + 9.999000099990002e-05 + ], + "musique/bge/k10/lib_mh-rerank": [ + 0.05000000074505806, + 0.024000000208616257, + 0.07699999958276749, + 0.00039996000399960006 + ], + "musique/bge/k10/lib_mh-ppr_best": [ + 0.004999999888241291, + -0.009999999776482582, + 0.019999999552965164, + 0.6238376162383762 + ], + "musique/bge/k20/lib_mh-rrf": [ + 0.057999998331069946, + 0.026975000277161598, + 0.08902499824762344, + 0.00029997000299970003 + ], + "musique/bge/k20/lib_mh-rerank": [ + 0.07100000232458115, + 0.041999999433755875, + 0.09702499955892563, + 9.999000099990002e-05 + ], + "musique/bge/k20/lib_mh-ppr_best": [ + -0.004999999888241291, + -0.019999999552965164, + 0.009999999776482582, + 0.6113388661133886 + ] +} \ No newline at end of file diff --git a/benchmarks/multihop/results/pairs_vs_nongraph.md b/benchmarks/multihop/results/pairs_vs_nongraph.md new file mode 100644 index 0000000..224cd02 --- /dev/null +++ b/benchmarks/multihop/results/pairs_vs_nongraph.md @@ -0,0 +1,20 @@ +| Dataset/encoder | k | graph (dev-selected) | multi_hop - RRF (pp) [95% CI] (p) | multi_hop - cross-encoder rerank (pp) [95% CI] (p) | multi_hop - PPR (pp) [95% CI] (p) | +|---|---|---|---|---|---| +| hotpotqa/minilm | 5 | mention | +16.8 [+13.6, +20.2] (p=<0.001) | +11.4 [+8.1, +14.9] (p=<0.001) | +7.5 [+4.9, +10.0] (p=<0.001; PPR graph=mention) | +| hotpotqa/minilm | 10 | mention | +9.0 [+6.0, +12.1] (p=<0.001) | +13.1 [+10.1, +16.2] (p=<0.001) | +0.1 [-1.4, +1.5] (p=1.000; PPR graph=mention) | +| hotpotqa/minilm | 20 | mention | +3.5 [+0.9, +6.0] (p=0.013) | +17.0 [+14.2, +19.7] (p=<0.001) | +0.0 [-1.1, +1.1] (p=1.000; PPR graph=mention) | +| hotpotqa/bge | 5 | mention | +12.2 [+9.4, +15.3] (p=<0.001) | +9.7 [+6.8, +12.8] (p=<0.001) | +1.7 [-0.6, +3.9] (p=0.167; PPR graph=mention) | +| hotpotqa/bge | 10 | mention | +7.7 [+5.3, +10.2] (p=<0.001) | +7.4 [+5.1, +9.8] (p=<0.001) | -0.4 [-1.4, +0.6] (p=0.549; PPR graph=mention) | +| hotpotqa/bge | 20 | mention | +5.6 [+3.8, +7.4] (p=<0.001) | +7.7 [+6.1, +9.5] (p=<0.001) | +0.0 [-0.9, +0.9] (p=1.000; PPR graph=mention) | +| 2wiki/minilm | 5 | mention+entity | +30.6 [+26.9, +34.2] (p=<0.001) | +30.0 [+26.3, +33.6] (p=<0.001) | +12.0 [+9.6, +14.4] (p=<0.001; PPR graph=mention) | +| 2wiki/minilm | 10 | mention+entity | +39.9 [+36.0, +43.6] (p=<0.001) | +43.5 [+40.0, +47.2] (p=<0.001) | +2.3 [+1.3, +3.4] (p=<0.001; PPR graph=mention) | +| 2wiki/minilm | 20 | mention+entity | +40.6 [+36.9, +44.2] (p=<0.001) | +48.3 [+45.2, +51.6] (p=<0.001) | +1.6 [+0.3, +2.9] (p=0.019; PPR graph=mention) | +| 2wiki/bge | 5 | mention | +39.1 [+35.7, +42.2] (p=<0.001) | +38.4 [+35.1, +41.7] (p=<0.001) | +18.9 [+15.9, +21.9] (p=<0.001; PPR graph=mention) | +| 2wiki/bge | 10 | mention | +49.0 [+45.7, +52.1] (p=<0.001) | +48.6 [+45.4, +51.8] (p=<0.001) | +1.8 [+0.7, +2.9] (p=0.002; PPR graph=mention) | +| 2wiki/bge | 20 | mention | +47.0 [+43.8, +50.1] (p=<0.001) | +46.3 [+43.3, +49.5] (p=<0.001) | +0.9 [+0.4, +1.5] (p=0.004; PPR graph=mention) | +| musique/minilm | 5 | mention | +5.8 [+3.7, +8.0] (p=<0.001) | +0.6 [-1.6, +2.9] (p=0.672) | +0.1 [-1.6, +1.7] (p=1.000; PPR graph=mention) | +| musique/minilm | 10 | mention | +7.0 [+4.4, +9.8] (p=<0.001) | +3.7 [+1.2, +6.4] (p=0.005) | -1.5 [-3.3, +0.2] (p=0.120; PPR graph=mention) | +| musique/minilm | 20 | mention | +5.8 [+2.6, +9.0] (p=<0.001) | +8.5 [+5.6, +11.5] (p=<0.001) | -0.4 [-1.9, +1.1] (p=0.679; PPR graph=mention) | +| musique/bge | 5 | mention | +6.7 [+4.5, +9.2] (p=<0.001) | +2.7 [+0.2, +5.5] (p=0.046) | +1.4 [-0.1, +2.9] (p=0.091; PPR graph=mention) | +| musique/bge | 10 | mention | +6.3 [+3.5, +9.2] (p=<0.001) | +5.0 [+2.4, +7.7] (p=<0.001) | +0.5 [-1.0, +2.0] (p=0.624; PPR graph=mention) | +| musique/bge | 20 | mention | +5.8 [+2.7, +8.9] (p=<0.001) | +7.1 [+4.2, +9.7] (p=<0.001) | -0.5 [-2.0, +1.0] (p=0.611; PPR graph=mention) | \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/2wiki__bge__baselines.json b/benchmarks/multihop/results/raw/2wiki__bge__baselines.json new file mode 100644 index 0000000..1b2b225 --- /dev/null +++ b/benchmarks/multihop/results/raw/2wiki__bge__baselines.json @@ -0,0 +1,8 @@ +{ + "dataset": "2wiki", + "model": "bge", + "corpus_size": 9239, + "n_questions": 1500, + "lib_similarity_search_same_top20_set": "198/200", + "seconds": 63.759765625 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/2wiki__bge__baselines.npz b/benchmarks/multihop/results/raw/2wiki__bge__baselines.npz new file mode 100644 index 0000000..71a2bfc Binary files /dev/null and b/benchmarks/multihop/results/raw/2wiki__bge__baselines.npz differ diff --git a/benchmarks/multihop/results/raw/2wiki__bge__bm25_top50.npy b/benchmarks/multihop/results/raw/2wiki__bge__bm25_top50.npy new file mode 100644 index 0000000..a9340b8 Binary files /dev/null and b/benchmarks/multihop/results/raw/2wiki__bge__bm25_top50.npy differ diff --git a/benchmarks/multihop/results/raw/2wiki__bge__ctrl_entity.json b/benchmarks/multihop/results/raw/2wiki__bge__ctrl_entity.json new file mode 100644 index 0000000..6fda278 --- /dev/null +++ b/benchmarks/multihop/results/raw/2wiki__bge__ctrl_entity.json @@ -0,0 +1,3274 @@ +{ + "dataset": "2wiki", + "model": "bge", + "variant": "ctrl_entity", + "families": [ + "entity_rand" + ], + "diagnostics": { + "n_edges_undirected_union": 30240, + "n_edges_by_family_directed_or_undirected": 30240, + "mean_degree": 6.546163004654184, + "median_degree": 4.0, + "max_degree": 86, + "isolated_frac": 0.21798895984413896, + "gold_pairs_directly_linked": 0.0012965964343598054, + "gold_pairs_linked_within_2": 0.0074554294975688815, + "neighbours_of_gold_that_are_in_same_question_context": 0.0006118890958802483 + }, + "ppr_dev_grid": [ + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.422, + "recall@10": 0.712, + "all@20": 0.422, + "recall@20": 0.712 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.422, + "recall@10": 0.712, + "all@20": 0.422, + "recall@20": 0.712 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.438, + "recall@10": 0.7225, + "all@20": 0.5, + "recall@20": 0.757 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.434, + "recall@10": 0.7205, + "all@20": 0.498, + "recall@20": 0.756 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.474, + "recall@10": 0.742, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.474, + "recall@10": 0.742, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.402, + "recall@5": 0.6965, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.476, + "recall@20": 0.744 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.4, + "recall@5": 0.6955, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.476, + "recall@20": 0.744 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.41, + "recall@5": 0.7045, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.484, + "recall@20": 0.748 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.41, + "recall@5": 0.7045, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.48, + "recall@20": 0.7465 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.418, + "recall@5": 0.709, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.52, + "recall@20": 0.769 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.416, + "recall@5": 0.708, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.52, + "recall@20": 0.769 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.418, + "recall@5": 0.709, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.418, + "recall@5": 0.709, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.374, + "recall@5": 0.6765, + "all@10": 0.46, + "recall@10": 0.7335, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.37, + "recall@5": 0.6745, + "all@10": 0.46, + "recall@10": 0.7335, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.392, + "recall@5": 0.6915, + "all@10": 0.464, + "recall@10": 0.7355, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.396, + "recall@5": 0.6925, + "all@10": 0.46, + "recall@10": 0.7335, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.416, + "recall@5": 0.7065, + "all@10": 0.472, + "recall@10": 0.741, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.414, + "recall@5": 0.7055, + "all@10": 0.472, + "recall@10": 0.741, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.418, + "recall@5": 0.709, + "all@10": 0.47, + "recall@10": 0.741, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.418, + "recall@5": 0.709, + "all@10": 0.47, + "recall@10": 0.7405, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.404, + "recall@5": 0.7005, + "all@10": 0.422, + "recall@10": 0.712, + "all@20": 0.422, + "recall@20": 0.712 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.404, + "recall@5": 0.7005, + "all@10": 0.422, + "recall@10": 0.712, + "all@20": 0.422, + "recall@20": 0.712 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.416, + "recall@5": 0.7085, + "all@10": 0.428, + "recall@10": 0.7155, + "all@20": 0.464, + "recall@20": 0.738 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.416, + "recall@5": 0.7085, + "all@10": 0.428, + "recall@10": 0.7155, + "all@20": 0.462, + "recall@20": 0.737 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.462, + "recall@10": 0.736, + "all@20": 0.518, + "recall@20": 0.7675 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.462, + "recall@10": 0.736, + "all@20": 0.52, + "recall@20": 0.7685 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.326, + "recall@5": 0.6375, + "all@10": 0.468, + "recall@10": 0.739, + "all@20": 0.476, + "recall@20": 0.744 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.318, + "recall@5": 0.633, + "all@10": 0.468, + "recall@10": 0.739, + "all@20": 0.476, + "recall@20": 0.744 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.38, + "recall@5": 0.685, + "all@10": 0.468, + "recall@10": 0.7385, + "all@20": 0.476, + "recall@20": 0.744 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.378, + "recall@5": 0.683, + "all@10": 0.468, + "recall@10": 0.7395, + "all@20": 0.476, + "recall@20": 0.744 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.404, + "recall@5": 0.7015, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.508, + "recall@20": 0.762 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.406, + "recall@5": 0.702, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.508, + "recall@20": 0.762 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.711, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.422, + "recall@5": 0.7115, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.216, + "recall@5": 0.5405, + "all@10": 0.43, + "recall@10": 0.711, + "all@20": 0.514, + "recall@20": 0.766 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.212, + "recall@5": 0.5365, + "all@10": 0.426, + "recall@10": 0.709, + "all@20": 0.512, + "recall@20": 0.765 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.336, + "recall@5": 0.651, + "all@10": 0.444, + "recall@10": 0.724, + "all@20": 0.518, + "recall@20": 0.768 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.33, + "recall@5": 0.6465, + "all@10": 0.446, + "recall@10": 0.725, + "all@20": 0.516, + "recall@20": 0.767 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.384, + "recall@5": 0.6855, + "all@10": 0.458, + "recall@10": 0.7325, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.384, + "recall@5": 0.6855, + "all@10": 0.458, + "recall@10": 0.7325, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.7095, + "all@10": 0.47, + "recall@10": 0.7405, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.42, + "recall@5": 0.7095, + "all@10": 0.47, + "recall@10": 0.7405, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.356, + "recall@5": 0.6645, + "all@10": 0.422, + "recall@10": 0.712, + "all@20": 0.422, + "recall@20": 0.712 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.358, + "recall@5": 0.665, + "all@10": 0.422, + "recall@10": 0.712, + "all@20": 0.422, + "recall@20": 0.712 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.382, + "recall@5": 0.689, + "all@10": 0.426, + "recall@10": 0.715, + "all@20": 0.45, + "recall@20": 0.7285 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.382, + "recall@5": 0.689, + "all@10": 0.426, + "recall@10": 0.715, + "all@20": 0.45, + "recall@20": 0.7285 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.408, + "recall@5": 0.7045, + "all@10": 0.456, + "recall@10": 0.7315, + "all@20": 0.51, + "recall@20": 0.763 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.408, + "recall@5": 0.7045, + "all@10": 0.454, + "recall@10": 0.731, + "all@20": 0.508, + "recall@20": 0.7615 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.23, + "recall@5": 0.527, + "all@10": 0.434, + "recall@10": 0.7155, + "all@20": 0.476, + "recall@20": 0.744 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.232, + "recall@5": 0.5265, + "all@10": 0.438, + "recall@10": 0.7175, + "all@20": 0.476, + "recall@20": 0.744 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.298, + "recall@5": 0.6205, + "all@10": 0.46, + "recall@10": 0.7345, + "all@20": 0.476, + "recall@20": 0.7445 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.294, + "recall@5": 0.617, + "all@10": 0.462, + "recall@10": 0.735, + "all@20": 0.476, + "recall@20": 0.7445 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.378, + "recall@5": 0.682, + "all@10": 0.472, + "recall@10": 0.7405, + "all@20": 0.502, + "recall@20": 0.7585 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.378, + "recall@5": 0.682, + "all@10": 0.472, + "recall@10": 0.7405, + "all@20": 0.504, + "recall@20": 0.7595 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.416, + "recall@5": 0.708, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.416, + "recall@5": 0.708, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.102, + "recall@5": 0.336, + "all@10": 0.304, + "recall@10": 0.6115, + "all@20": 0.484, + "recall@20": 0.749 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.104, + "recall@5": 0.333, + "all@10": 0.3, + "recall@10": 0.607, + "all@20": 0.482, + "recall@20": 0.748 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.178, + "recall@5": 0.4855, + "all@10": 0.406, + "recall@10": 0.696, + "all@20": 0.506, + "recall@20": 0.761 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.174, + "recall@5": 0.4845, + "all@10": 0.406, + "recall@10": 0.6965, + "all@20": 0.506, + "recall@20": 0.7605 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.332, + "recall@5": 0.6495, + "all@10": 0.452, + "recall@10": 0.728, + "all@20": 0.516, + "recall@20": 0.767 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.33, + "recall@5": 0.6475, + "all@10": 0.452, + "recall@10": 0.7285, + "all@20": 0.512, + "recall@20": 0.765 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.41, + "recall@5": 0.7035, + "all@10": 0.47, + "recall@10": 0.7395, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.41, + "recall@5": 0.7035, + "all@10": 0.47, + "recall@10": 0.7395, + "all@20": 0.526, + "recall@20": 0.772 + } + ], + "ppr_selected": { + "5": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "20": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + } + }, + "lib_dev_grid": [ + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 10.577222801395692, + "default": true, + "all@5": 0.414, + "recall@5": 0.708, + "n_ret@5": 5.0, + "all@10": 0.416, + "recall@10": 0.709, + "n_ret@10": 9.994, + "all@20": 0.422, + "recall@20": 0.7115, + "n_ret@20": 19.964 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.829011199995875, + "default": false, + "all@5": 0.308, + "recall@5": 0.6455, + "n_ret@5": 5.0, + "all@10": 0.386, + "recall@10": 0.69, + "n_ret@10": 10.0, + "all@20": 0.444, + "recall@20": 0.726, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 25.496251801378094, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 30.927422799402848, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 58.87801399931777, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 30.220528800506145, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.478, + "recall@20": 0.7445, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 43.14372220088262, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.472, + "recall@10": 0.742, + "n_ret@10": 10.0, + "all@20": 0.504, + "recall@20": 0.758, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 26.642941998667084, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.502, + "recall@20": 0.7575, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 49.83281799929682, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 6.933214199380018, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.744, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 23.87231539771892, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.502, + "recall@20": 0.757, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 40.41735139931552, + "default": false, + "all@5": 0.404, + "recall@5": 0.7025, + "n_ret@5": 5.0, + "all@10": 0.456, + "recall@10": 0.733, + "n_ret@10": 10.0, + "all@20": 0.496, + "recall@20": 0.753, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 28.11415379983373, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 23.22867519920692, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.478, + "recall@20": 0.7445, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 4.871948202024214, + "default": false, + "all@5": 0.308, + "recall@5": 0.6455, + "n_ret@5": 5.0, + "all@10": 0.36, + "recall@10": 0.675, + "n_ret@10": 9.98, + "all@20": 0.4, + "recall@20": 0.699, + "n_ret@20": 19.4 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 31.52848780050408, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.48, + "recall@20": 0.7455, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.291283598868176, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.7565, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 21.951486197533086, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 4.482011197833344, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.712, + "n_ret@10": 9.98, + "all@20": 0.424, + "recall@20": 0.7125, + "n_ret@20": 19.4 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 23.286489798221737, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.424, + "recall@10": 0.7125, + "n_ret@10": 9.994, + "all@20": 0.424, + "recall@20": 0.7125, + "n_ret@20": 19.964 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 20.116168599692173, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.478, + "recall@20": 0.7445, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 35.30166980053764, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.502, + "recall@20": 0.7575, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 17.456893999129534, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.424, + "recall@10": 0.7125, + "n_ret@10": 9.98, + "all@20": 0.424, + "recall@20": 0.7125, + "n_ret@20": 19.4 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 20.754883599001914, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.472, + "recall@10": 0.742, + "n_ret@10": 10.0, + "all@20": 0.502, + "recall@20": 0.7575, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 42.53731780021917, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 18.910336001659743, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.424, + "recall@10": 0.7125, + "n_ret@10": 9.98, + "all@20": 0.424, + "recall@20": 0.7125, + "n_ret@20": 19.4 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 5.371904801460914, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.7565, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 9.343970000511035, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.712, + "n_ret@10": 9.994, + "all@20": 0.424, + "recall@20": 0.7125, + "n_ret@20": 19.964 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 24.106033198535442, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.424, + "recall@10": 0.7125, + "n_ret@10": 9.994, + "all@20": 0.424, + "recall@20": 0.7125, + "n_ret@20": 19.964 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 19.082635000464506, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.478, + "recall@20": 0.7445, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 21.122584199765697, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.502, + "recall@20": 0.7575, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 5.899421598878689, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 53.614048599265516, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 38.29484479967505, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.48, + "recall@20": 0.7455, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 28.324261198169552, + "default": false, + "all@5": 0.404, + "recall@5": 0.7025, + "n_ret@5": 5.0, + "all@10": 0.456, + "recall@10": 0.733, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.743, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 16.62184499949217, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.7565, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 8.519428399275057, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.712, + "n_ret@10": 9.994, + "all@20": 0.424, + "recall@20": 0.7125, + "n_ret@20": 19.964 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 19.32404540060088, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 44.5806927990634, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 20.259770400589332, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 4.482110598823056, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.744, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 20.977156199747697, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 28.174259000457823, + "default": false, + "all@5": 0.304, + "recall@5": 0.6435, + "n_ret@5": 5.0, + "all@10": 0.378, + "recall@10": 0.686, + "n_ret@10": 10.0, + "all@20": 0.424, + "recall@20": 0.716, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 14.195062398444861, + "default": false, + "all@5": 0.304, + "recall@5": 0.6435, + "n_ret@5": 5.0, + "all@10": 0.352, + "recall@10": 0.671, + "n_ret@10": 9.994, + "all@20": 0.386, + "recall@20": 0.692, + "n_ret@20": 19.964 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 39.76521019812208, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.478, + "recall@20": 0.7445, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 17.52786700008437, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.478, + "recall@20": 0.7445, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 28.234992601675913, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.424, + "recall@10": 0.7125, + "n_ret@10": 9.994, + "all@20": 0.426, + "recall@20": 0.7135, + "n_ret@20": 19.964 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 40.008396399440244, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.424, + "recall@10": 0.7125, + "n_ret@10": 9.994, + "all@20": 0.426, + "recall@20": 0.7135, + "n_ret@20": 19.964 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 45.020729399402626, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.48, + "recall@20": 0.7455, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 33.773096801596694, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.478, + "recall@20": 0.7445, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 55.774766799411736, + "default": false, + "all@5": 0.404, + "recall@5": 0.7025, + "n_ret@5": 5.0, + "all@10": 0.456, + "recall@10": 0.733, + "n_ret@10": 10.0, + "all@20": 0.514, + "recall@20": 0.763, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 32.624886000179686, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.502, + "recall@20": 0.757, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 46.73503539932426, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.48, + "recall@20": 0.7455, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 22.52130699902773, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.744, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 48.48705279943533, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.502, + "recall@20": 0.7575, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 31.295089801191356, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.502, + "recall@20": 0.7575, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 30.775039801024832, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 33.47062940057367, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.424, + "recall@10": 0.7125, + "n_ret@10": 9.994, + "all@20": 0.424, + "recall@20": 0.7125, + "n_ret@20": 19.964 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 21.626721799373627, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.7565, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 29.5892150006257, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.502, + "recall@20": 0.757, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 35.48175859905314, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.424, + "recall@10": 0.7125, + "n_ret@10": 9.994, + "all@20": 0.426, + "recall@20": 0.7135, + "n_ret@20": 19.964 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 52.1708749999525, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 39.65824139909819, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.48, + "recall@20": 0.7455, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.55823640001472, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.7565, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 42.18353479914367, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.478, + "recall@20": 0.7445, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 8.373558600666001, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.7565, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 30.419629999552853, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 28.608547998592257, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.472, + "recall@10": 0.742, + "n_ret@10": 10.0, + "all@20": 0.478, + "recall@20": 0.7445, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 30.47101260058116, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.424, + "recall@10": 0.7125, + "n_ret@10": 9.98, + "all@20": 0.424, + "recall@20": 0.7125, + "n_ret@20": 19.4 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 28.484094401123002, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.424, + "recall@10": 0.7125, + "n_ret@10": 9.98, + "all@20": 0.424, + "recall@20": 0.7125, + "n_ret@20": 19.4 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 8.237838399247266, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.7565, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 30.841262201196514, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 50.27559399919119, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.502, + "recall@20": 0.757, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 45.662932200706564, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.502, + "recall@20": 0.7575, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 22.40358920302242, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.744, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.975325600011274, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.712, + "n_ret@10": 9.98, + "all@20": 0.424, + "recall@20": 0.7125, + "n_ret@20": 19.4 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 32.0263021994615, + "default": false, + "all@5": 0.404, + "recall@5": 0.7025, + "n_ret@5": 5.0, + "all@10": 0.456, + "recall@10": 0.733, + "n_ret@10": 10.0, + "all@20": 0.518, + "recall@20": 0.766, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 51.645846199360676, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 32.332785999868065, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 29.596756402868778, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.424, + "recall@10": 0.7125, + "n_ret@10": 9.98, + "all@20": 0.424, + "recall@20": 0.7125, + "n_ret@20": 19.4 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 28.89024980040267, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.478, + "recall@20": 0.7445, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 47.224693799973466, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 28.784583399421535, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.502, + "recall@20": 0.757, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 13.648798400303349, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.712, + "n_ret@10": 9.994, + "all@20": 0.424, + "recall@20": 0.7125, + "n_ret@20": 19.964 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 18.433417599415407, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.744, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 35.63115179829765, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.424, + "recall@10": 0.7125, + "n_ret@10": 9.994, + "all@20": 0.426, + "recall@20": 0.7135, + "n_ret@20": 19.964 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 31.299394799629226, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.502, + "recall@20": 0.757, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 30.663248598575592, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 47.4756233979715, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 27.027133999275975, + "default": false, + "all@5": 0.404, + "recall@5": 0.7025, + "n_ret@5": 5.0, + "all@10": 0.42, + "recall@10": 0.7105, + "n_ret@10": 9.98, + "all@20": 0.424, + "recall@20": 0.7125, + "n_ret@20": 19.4 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 49.98390180035494, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 27.885638201260008, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.502, + "recall@20": 0.7575, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 27.108054198673926, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.472, + "recall@10": 0.742, + "n_ret@10": 10.0, + "all@20": 0.528, + "recall@20": 0.7725, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 31.21313740068581, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.424, + "recall@10": 0.7125, + "n_ret@10": 9.994, + "all@20": 0.424, + "recall@20": 0.7125, + "n_ret@20": 19.964 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 11.599137002369389, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.712, + "n_ret@10": 9.994, + "all@20": 0.424, + "recall@20": 0.7125, + "n_ret@20": 19.964 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 26.72277139907237, + "default": false, + "all@5": 0.404, + "recall@5": 0.7025, + "n_ret@5": 5.0, + "all@10": 0.456, + "recall@10": 0.733, + "n_ret@10": 10.0, + "all@20": 0.478, + "recall@20": 0.7445, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 6.914080401067622, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.744, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.724858998903073, + "default": false, + "all@5": 0.308, + "recall@5": 0.6455, + "n_ret@5": 5.0, + "all@10": 0.386, + "recall@10": 0.69, + "n_ret@10": 10.0, + "all@20": 0.452, + "recall@20": 0.7305, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 15.961013798951171, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.744, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 25.0876039970899, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 48.19127539964393, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + } + ], + "lib_selected": { + "5": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + } + }, + "seconds": 4256.362103223801 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/2wiki__bge__ctrl_entity.npz b/benchmarks/multihop/results/raw/2wiki__bge__ctrl_entity.npz new file mode 100644 index 0000000..db49cc5 Binary files /dev/null and b/benchmarks/multihop/results/raw/2wiki__bge__ctrl_entity.npz differ diff --git a/benchmarks/multihop/results/raw/2wiki__bge__ctrl_mention.json b/benchmarks/multihop/results/raw/2wiki__bge__ctrl_mention.json new file mode 100644 index 0000000..a3a7a29 --- /dev/null +++ b/benchmarks/multihop/results/raw/2wiki__bge__ctrl_mention.json @@ -0,0 +1,3274 @@ +{ + "dataset": "2wiki", + "model": "bge", + "variant": "ctrl_mention", + "families": [ + "mention_rand" + ], + "diagnostics": { + "n_edges_undirected_union": 5383, + "n_edges_by_family_directed_or_undirected": 6115, + "mean_degree": 1.165277627448858, + "median_degree": 1.0, + "max_degree": 30, + "isolated_frac": 0.4217988959844139, + "gold_pairs_directly_linked": 0.00032414910858995135, + "gold_pairs_linked_within_2": 0.00032414910858995135, + "neighbours_of_gold_that_are_in_same_question_context": 0.0003952569169960474 + }, + "ppr_dev_grid": [ + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.422, + "recall@10": 0.712, + "all@20": 0.422, + "recall@20": 0.712 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.422, + "recall@10": 0.712, + "all@20": 0.422, + "recall@20": 0.712 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.446, + "recall@10": 0.7255, + "all@20": 0.504, + "recall@20": 0.7595 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.436, + "recall@10": 0.7205, + "all@20": 0.496, + "recall@20": 0.7555 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.47, + "recall@10": 0.741, + "all@20": 0.522, + "recall@20": 0.7695 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.464, + "recall@10": 0.737, + "all@20": 0.522, + "recall@20": 0.7695 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.424, + "recall@5": 0.713, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.476, + "recall@20": 0.744 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.362, + "recall@5": 0.6545, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.476, + "recall@20": 0.7445 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.424, + "recall@5": 0.713, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.488, + "recall@20": 0.751 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.394, + "recall@5": 0.6865, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.48, + "recall@20": 0.746 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.426, + "recall@5": 0.714, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.516, + "recall@20": 0.766 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.418, + "recall@5": 0.705, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.512, + "recall@20": 0.763 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.426, + "recall@5": 0.714, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.418, + "recall@5": 0.7075, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.406, + "recall@5": 0.7015, + "all@10": 0.462, + "recall@10": 0.735, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.26, + "recall@5": 0.5395, + "all@10": 0.426, + "recall@10": 0.703, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.41, + "recall@5": 0.7045, + "all@10": 0.462, + "recall@10": 0.735, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.326, + "recall@5": 0.63, + "all@10": 0.44, + "recall@10": 0.7195, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.412, + "recall@5": 0.7055, + "all@10": 0.462, + "recall@10": 0.735, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.398, + "recall@5": 0.693, + "all@10": 0.45, + "recall@10": 0.7255, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.426, + "recall@5": 0.714, + "all@10": 0.466, + "recall@10": 0.7375, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.418, + "recall@5": 0.7075, + "all@10": 0.472, + "recall@10": 0.74, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.41, + "recall@5": 0.7045, + "all@10": 0.422, + "recall@10": 0.712, + "all@20": 0.422, + "recall@20": 0.712 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.332, + "recall@5": 0.631, + "all@10": 0.422, + "recall@10": 0.712, + "all@20": 0.422, + "recall@20": 0.712 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.414, + "recall@5": 0.7075, + "all@10": 0.446, + "recall@10": 0.7245, + "all@20": 0.5, + "recall@20": 0.7575 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.382, + "recall@5": 0.688, + "all@10": 0.432, + "recall@10": 0.718, + "all@20": 0.488, + "recall@20": 0.75 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.7115, + "all@10": 0.46, + "recall@10": 0.7345, + "all@20": 0.516, + "recall@20": 0.766 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.71, + "all@10": 0.454, + "recall@10": 0.7315, + "all@20": 0.518, + "recall@20": 0.7665 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.37, + "recall@5": 0.676, + "all@10": 0.472, + "recall@10": 0.7405, + "all@20": 0.476, + "recall@20": 0.744 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.138, + "recall@5": 0.423, + "all@10": 0.41, + "recall@10": 0.6875, + "all@20": 0.476, + "recall@20": 0.7445 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.408, + "recall@5": 0.7045, + "all@10": 0.474, + "recall@10": 0.742, + "all@20": 0.486, + "recall@20": 0.75 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.591, + "all@10": 0.46, + "recall@10": 0.7325, + "all@20": 0.478, + "recall@20": 0.745 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.424, + "recall@5": 0.713, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.504, + "recall@20": 0.7595 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.38, + "recall@5": 0.674, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.502, + "recall@20": 0.758 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.426, + "recall@5": 0.7145, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.7085, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.276, + "recall@5": 0.5785, + "all@10": 0.456, + "recall@10": 0.729, + "all@20": 0.514, + "recall@20": 0.765 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.058, + "recall@5": 0.287, + "all@10": 0.196, + "recall@10": 0.4635, + "all@20": 0.46, + "recall@20": 0.724 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.376, + "recall@5": 0.6805, + "all@10": 0.458, + "recall@10": 0.7325, + "all@20": 0.518, + "recall@20": 0.768 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.146, + "recall@5": 0.393, + "all@10": 0.356, + "recall@10": 0.6435, + "all@20": 0.494, + "recall@20": 0.753 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.408, + "recall@5": 0.7035, + "all@10": 0.462, + "recall@10": 0.735, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.591, + "all@10": 0.432, + "recall@10": 0.7125, + "all@20": 0.524, + "recall@20": 0.771 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.414, + "recall@5": 0.707, + "all@10": 0.462, + "recall@10": 0.7355, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.408, + "recall@5": 0.7005, + "all@10": 0.454, + "recall@10": 0.728, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.36, + "recall@5": 0.6625, + "all@10": 0.422, + "recall@10": 0.711, + "all@20": 0.422, + "recall@20": 0.712 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.178, + "recall@5": 0.4555, + "all@10": 0.402, + "recall@10": 0.6975, + "all@20": 0.422, + "recall@20": 0.712 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.398, + "recall@5": 0.6955, + "all@10": 0.444, + "recall@10": 0.7235, + "all@20": 0.5, + "recall@20": 0.7575 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.264, + "recall@5": 0.571, + "all@10": 0.428, + "recall@10": 0.716, + "all@20": 0.47, + "recall@20": 0.741 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.414, + "recall@5": 0.707, + "all@10": 0.458, + "recall@10": 0.7335, + "all@20": 0.516, + "recall@20": 0.766 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.378, + "recall@5": 0.686, + "all@10": 0.454, + "recall@10": 0.7305, + "all@20": 0.51, + "recall@20": 0.7625 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.268, + "recall@5": 0.5675, + "all@10": 0.452, + "recall@10": 0.727, + "all@20": 0.476, + "recall@20": 0.744 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.042, + "recall@5": 0.226, + "all@10": 0.216, + "recall@10": 0.47, + "all@20": 0.472, + "recall@20": 0.742 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.324, + "recall@5": 0.629, + "all@10": 0.464, + "recall@10": 0.7365, + "all@20": 0.486, + "recall@20": 0.7495 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.084, + "recall@5": 0.3275, + "all@10": 0.362, + "recall@10": 0.6375, + "all@20": 0.476, + "recall@20": 0.744 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.402, + "recall@5": 0.7, + "all@10": 0.474, + "recall@10": 0.742, + "all@20": 0.506, + "recall@20": 0.7605 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.312, + "recall@5": 0.61, + "all@10": 0.452, + "recall@10": 0.7285, + "all@20": 0.5, + "recall@20": 0.7565 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.426, + "recall@5": 0.7145, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.414, + "recall@5": 0.7015, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.138, + "recall@5": 0.357, + "all@10": 0.376, + "recall@10": 0.66, + "all@20": 0.504, + "recall@20": 0.759 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.01, + "recall@5": 0.147, + "all@10": 0.052, + "recall@10": 0.246, + "all@20": 0.222, + "recall@20": 0.481 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.196, + "recall@5": 0.472, + "all@10": 0.426, + "recall@10": 0.706, + "all@20": 0.508, + "recall@20": 0.762 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.028, + "recall@5": 0.2075, + "all@10": 0.13, + "recall@10": 0.36, + "all@20": 0.412, + "recall@20": 0.689 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.366, + "recall@5": 0.6745, + "all@10": 0.456, + "recall@10": 0.732, + "all@20": 0.518, + "recall@20": 0.7675 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.168, + "recall@5": 0.4345, + "all@10": 0.38, + "recall@10": 0.662, + "all@20": 0.49, + "recall@20": 0.751 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.41, + "recall@5": 0.705, + "all@10": 0.462, + "recall@10": 0.7355, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.386, + "recall@5": 0.683, + "all@10": 0.45, + "recall@10": 0.7245, + "all@20": 0.526, + "recall@20": 0.772 + } + ], + "ppr_selected": { + "5": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "20": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + } + }, + "lib_dev_grid": [ + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.26395259948913, + "default": true, + "all@5": 0.418, + "recall@5": 0.71, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.712, + "n_ret@10": 8.752, + "all@20": 0.422, + "recall@20": 0.712, + "n_ret@20": 10.198 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.226740200305358, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.744, + "n_ret@20": 18.036 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 25.672917798627168, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.502, + "recall@20": 0.7575, + "n_ret@20": 19.964 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 27.8267631996423, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.474, + "recall@10": 0.743, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 27.75959319761023, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.712, + "n_ret@10": 8.752, + "all@20": 0.422, + "recall@20": 0.712, + "n_ret@20": 10.198 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 27.919745198916644, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 5.780258400714956, + "default": false, + "all@5": 0.386, + "recall@5": 0.6905, + "n_ret@5": 5.0, + "all@10": 0.416, + "recall@10": 0.709, + "n_ret@10": 9.756, + "all@20": 0.42, + "recall@20": 0.711, + "n_ret@20": 16.34 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 31.329060999327343, + "default": false, + "all@5": 0.416, + "recall@5": 0.709, + "n_ret@5": 5.0, + "all@10": 0.474, + "recall@10": 0.743, + "n_ret@10": 10.0, + "all@20": 0.524, + "recall@20": 0.7705, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.014242199365981, + "default": false, + "all@5": 0.404, + "recall@5": 0.701, + "n_ret@5": 5.0, + "all@10": 0.47, + "recall@10": 0.7385, + "n_ret@10": 10.0, + "all@20": 0.498, + "recall@20": 0.7555, + "n_ret@20": 19.95 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.525584400398657, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.744, + "n_ret@20": 16.614 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 27.056374998181127, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 5.587241198052652, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.712, + "n_ret@10": 9.756, + "all@20": 0.422, + "recall@20": 0.712, + "n_ret@20": 16.34 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.820790800033137, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.7565, + "n_ret@20": 19.964 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 25.112744799582288, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.712, + "n_ret@10": 9.216, + "all@20": 0.422, + "recall@20": 0.712, + "n_ret@20": 11.128 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 26.313480600714684, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.478, + "recall@20": 0.745, + "n_ret@20": 18.314 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 25.65244339837227, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.7565, + "n_ret@20": 19.95 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 5.43875240185298, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.712, + "n_ret@10": 9.216, + "all@20": 0.422, + "recall@20": 0.712, + "n_ret@20": 11.128 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 26.737925999681465, + "default": false, + "all@5": 0.416, + "recall@5": 0.709, + "n_ret@5": 5.0, + "all@10": 0.474, + "recall@10": 0.743, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.7445, + "n_ret@20": 19.9 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 7.585457200068049, + "default": false, + "all@5": 0.386, + "recall@5": 0.6905, + "n_ret@5": 5.0, + "all@10": 0.458, + "recall@10": 0.732, + "n_ret@10": 10.0, + "all@20": 0.502, + "recall@20": 0.7595, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 5.82088859914802, + "default": false, + "all@5": 0.39, + "recall@5": 0.6925, + "n_ret@5": 5.0, + "all@10": 0.464, + "recall@10": 0.7355, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.7445, + "n_ret@20": 19.108 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 25.07820519944653, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.478, + "recall@20": 0.745, + "n_ret@20": 18.036 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 24.63808880012948, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.478, + "recall@20": 0.745, + "n_ret@20": 18.314 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 7.067970000905916, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.7445, + "n_ret@20": 19.936 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 29.004728599684313, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.474, + "recall@10": 0.743, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.757, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 29.058380399364978, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.7445, + "n_ret@20": 19.9 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 27.628368803067133, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 25.72132200235501, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.712, + "n_ret@10": 9.756, + "all@20": 0.422, + "recall@20": 0.712, + "n_ret@20": 16.34 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 28.4583301988896, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.757, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 25.90112639858853, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.7445, + "n_ret@20": 19.936 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.683869798434898, + "default": false, + "all@5": 0.402, + "recall@5": 0.7, + "n_ret@5": 5.0, + "all@10": 0.47, + "recall@10": 0.7385, + "n_ret@10": 10.0, + "all@20": 0.518, + "recall@20": 0.767, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 26.32500560034532, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 23.280336801079102, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.7565, + "n_ret@20": 19.964 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.866055399528705, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.7565, + "n_ret@20": 19.95 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 25.596031199442223, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 22.758683399646543, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.424, + "recall@10": 0.713, + "n_ret@10": 9.802, + "all@20": 0.424, + "recall@20": 0.713, + "n_ret@20": 17.588 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 23.21592200046871, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.7565, + "n_ret@20": 19.964 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 4.320312199182808, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.712, + "n_ret@10": 8.82, + "all@20": 0.422, + "recall@20": 0.712, + "n_ret@20": 10.904 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.355412200675346, + "default": false, + "all@5": 0.402, + "recall@5": 0.7, + "n_ret@5": 5.0, + "all@10": 0.47, + "recall@10": 0.7385, + "n_ret@10": 10.0, + "all@20": 0.498, + "recall@20": 0.7555, + "n_ret@20": 19.964 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 5.180904401349835, + "default": false, + "all@5": 0.39, + "recall@5": 0.6925, + "n_ret@5": 5.0, + "all@10": 0.464, + "recall@10": 0.7355, + "n_ret@10": 10.0, + "all@20": 0.492, + "recall@20": 0.7535, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 22.15498519851826, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 4.880883199279197, + "default": false, + "all@5": 0.402, + "recall@5": 0.7, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.712, + "n_ret@10": 8.752, + "all@20": 0.422, + "recall@20": 0.712, + "n_ret@20": 10.198 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 6.701561800087802, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 4.478446200257167, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.7565, + "n_ret@20": 19.964 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 21.256149601889774, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.474, + "recall@10": 0.743, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 19.536554398946464, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.744, + "n_ret@20": 16.614 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 18.689961399999447, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.712, + "n_ret@10": 8.752, + "all@20": 0.422, + "recall@20": 0.712, + "n_ret@20": 10.198 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 20.136225198977627, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 5.215832202578895, + "default": false, + "all@5": 0.386, + "recall@5": 0.6905, + "n_ret@5": 5.0, + "all@10": 0.458, + "recall@10": 0.732, + "n_ret@10": 10.0, + "all@20": 0.486, + "recall@20": 0.7505, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 19.26985419820994, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.712, + "n_ret@10": 9.756, + "all@20": 0.422, + "recall@20": 0.712, + "n_ret@20": 16.34 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 17.80014419928193, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.7445, + "n_ret@20": 19.9 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 19.82372180209495, + "default": false, + "all@5": 0.416, + "recall@5": 0.709, + "n_ret@5": 5.0, + "all@10": 0.474, + "recall@10": 0.743, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.7445, + "n_ret@20": 19.108 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 4.848706798395142, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 19.890778997913003, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.757, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 18.394215000676923, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.478, + "recall@20": 0.745, + "n_ret@20": 18.314 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 4.757466199458577, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.7445, + "n_ret@20": 19.9 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 4.650957201258279, + "default": false, + "all@5": 0.39, + "recall@5": 0.6925, + "n_ret@5": 5.0, + "all@10": 0.464, + "recall@10": 0.7355, + "n_ret@10": 10.0, + "all@20": 0.51, + "recall@20": 0.7635, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 21.35695280053187, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 21.541093400213867, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 19.46436659921892, + "default": false, + "all@5": 0.416, + "recall@5": 0.709, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.712, + "n_ret@10": 9.756, + "all@20": 0.422, + "recall@20": 0.712, + "n_ret@20": 16.34 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 20.217881599906832, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.7565, + "n_ret@20": 19.966 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 21.399649598519318, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.757, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.3, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 5.7432631989941, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 25.80817320139613, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.712, + "n_ret@10": 9.756, + "all@20": 0.422, + "recall@20": 0.712, + "n_ret@20": 16.34 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 25.201705798739567, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.7445, + "n_ret@20": 19.108 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 26.03043220145628, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.757, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 19.191810798482038, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.478, + "recall@20": 0.7455, + "n_ret@20": 19.9 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 22.13984240067657, + "default": false, + "all@5": 0.418, + "recall@5": 0.71, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.478, + "recall@20": 0.745, + "n_ret@20": 18.036 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 18.415210800943896, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.7565, + "n_ret@20": 19.966 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 4.3021748007740825, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.7565, + "n_ret@20": 19.964 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 21.583758999826387, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.7445, + "n_ret@20": 19.936 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 5.1019180000294, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.757, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 19.389006200246513, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 22.210441999370232, + "default": false, + "all@5": 0.416, + "recall@5": 0.709, + "n_ret@5": 5.0, + "all@10": 0.474, + "recall@10": 0.743, + "n_ret@10": 10.0, + "all@20": 0.524, + "recall@20": 0.7705, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 20.31162819848396, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.7445, + "n_ret@20": 19.936 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 4.063373200246133, + "default": false, + "all@5": 0.404, + "recall@5": 0.701, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.712, + "n_ret@10": 8.192, + "all@20": 0.422, + "recall@20": 0.712, + "n_ret@20": 8.584 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 24.414907599100843, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.712, + "n_ret@10": 8.752, + "all@20": 0.422, + "recall@20": 0.712, + "n_ret@20": 10.198 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 35.51391339825932, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.502, + "recall@20": 0.7575, + "n_ret@20": 19.964 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 25.35989359987434, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.757, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 17.905126001685858, + "default": false, + "all@5": 0.418, + "recall@5": 0.71, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.524, + "recall@20": 0.7705, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 19.763397599221207, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.712, + "n_ret@10": 9.756, + "all@20": 0.422, + "recall@20": 0.712, + "n_ret@20": 16.34 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.232187999528833, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.712, + "n_ret@10": 8.192, + "all@20": 0.422, + "recall@20": 0.712, + "n_ret@20": 8.584 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 31.72468419827055, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.065344200702384, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.712, + "n_ret@10": 8.752, + "all@20": 0.422, + "recall@20": 0.712, + "n_ret@20": 10.198 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 32.79332379880361, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.712, + "n_ret@10": 9.216, + "all@20": 0.422, + "recall@20": 0.712, + "n_ret@20": 11.128 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 31.587799799861386, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.757, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 35.602204400580376, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 8.684948799433187, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.7445, + "n_ret@20": 19.9 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 33.07543280068785, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.299939198768698, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.7565, + "n_ret@20": 19.966 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 8.275785401812755, + "default": false, + "all@5": 0.386, + "recall@5": 0.6905, + "n_ret@5": 5.0, + "all@10": 0.458, + "recall@10": 0.732, + "n_ret@10": 10.0, + "all@20": 0.486, + "recall@20": 0.7505, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 33.09408879803959, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.478, + "recall@20": 0.745, + "n_ret@20": 18.314 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 34.01197279873304, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.744, + "n_ret@20": 16.614 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 8.532018001191318, + "default": false, + "all@5": 0.386, + "recall@5": 0.6905, + "n_ret@5": 5.0, + "all@10": 0.458, + "recall@10": 0.732, + "n_ret@10": 10.0, + "all@20": 0.504, + "recall@20": 0.7605, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 31.850185599992983, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.712, + "n_ret@10": 9.756, + "all@20": 0.422, + "recall@20": 0.712, + "n_ret@20": 16.34 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 30.15574579918757, + "default": false, + "all@5": 0.418, + "recall@5": 0.71, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.524, + "recall@20": 0.7705, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.839685399318114, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.712, + "n_ret@10": 8.82, + "all@20": 0.422, + "recall@20": 0.712, + "n_ret@20": 10.904 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 30.460872599622235, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 28.580208200146444, + "default": false, + "all@5": 0.418, + "recall@5": 0.71, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.478, + "recall@20": 0.745, + "n_ret@20": 18.314 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.995786000043154, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 6.755439400556497, + "default": false, + "all@5": 0.386, + "recall@5": 0.6905, + "n_ret@5": 5.0, + "all@10": 0.416, + "recall@10": 0.709, + "n_ret@10": 9.802, + "all@20": 0.42, + "recall@20": 0.711, + "n_ret@20": 17.588 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 28.284267801209353, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.7445, + "n_ret@20": 19.936 + } + ], + "lib_selected": { + "5": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + } + }, + "seconds": 1541.9231624603271 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/2wiki__bge__ctrl_mention.npz b/benchmarks/multihop/results/raw/2wiki__bge__ctrl_mention.npz new file mode 100644 index 0000000..fd4f35c Binary files /dev/null and b/benchmarks/multihop/results/raw/2wiki__bge__ctrl_mention.npz differ diff --git a/benchmarks/multihop/results/raw/2wiki__bge__entity.json b/benchmarks/multihop/results/raw/2wiki__bge__entity.json new file mode 100644 index 0000000..3c87636 --- /dev/null +++ b/benchmarks/multihop/results/raw/2wiki__bge__entity.json @@ -0,0 +1,2017 @@ +{ + "dataset": "2wiki", + "model": "bge", + "variant": "entity", + "families": [ + "entity" + ], + "diagnostics": { + "n_edges_undirected_union": 30240, + "n_edges_by_family_directed_or_undirected": 30240, + "mean_degree": 6.546163004654184, + "median_degree": 4.0, + "max_degree": 86, + "isolated_frac": 0.21798895984413896, + "gold_pairs_directly_linked": 0.4051863857374392, + "gold_pairs_linked_within_2": 0.426580226904376, + "neighbours_of_gold_that_are_in_same_question_context": 0.11197227702934706 + }, + "ppr_dev_grid": [ + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.424, + "recall@5": 0.713, + "all@10": 0.666, + "recall@10": 0.853, + "all@20": 0.802, + "recall@20": 0.914 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.424, + "recall@5": 0.713, + "all@10": 0.678, + "recall@10": 0.8595, + "all@20": 0.824, + "recall@20": 0.924 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.424, + "recall@5": 0.713, + "all@10": 0.702, + "recall@10": 0.8695, + "all@20": 0.822, + "recall@20": 0.9255 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.424, + "recall@5": 0.713, + "all@10": 0.714, + "recall@10": 0.8745, + "all@20": 0.824, + "recall@20": 0.9265 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.636, + "recall@10": 0.8385, + "all@20": 0.698, + "recall@20": 0.8715 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.638, + "recall@10": 0.8395, + "all@20": 0.702, + "recall@20": 0.8735 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.52, + "recall@10": 0.7695, + "all@20": 0.582, + "recall@20": 0.803 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.522, + "recall@10": 0.7705, + "all@20": 0.584, + "recall@20": 0.804 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.406, + "recall@5": 0.6995, + "all@10": 0.478, + "recall@10": 0.745, + "all@20": 0.73, + "recall@20": 0.888 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.406, + "recall@5": 0.6985, + "all@10": 0.478, + "recall@10": 0.745, + "all@20": 0.742, + "recall@20": 0.894 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.418, + "recall@5": 0.711, + "all@10": 0.478, + "recall@10": 0.745, + "all@20": 0.76, + "recall@20": 0.899 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.41, + "recall@5": 0.7065, + "all@10": 0.478, + "recall@10": 0.745, + "all@20": 0.768, + "recall@20": 0.903 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.713, + "all@10": 0.478, + "recall@10": 0.745, + "all@20": 0.696, + "recall@20": 0.8705 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.424, + "recall@5": 0.714, + "all@10": 0.478, + "recall@10": 0.745, + "all@20": 0.698, + "recall@20": 0.872 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.43, + "recall@5": 0.717, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.576, + "recall@20": 0.8 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.432, + "recall@5": 0.719, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.58, + "recall@20": 0.802 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.376, + "recall@5": 0.6745, + "all@10": 0.496, + "recall@10": 0.755, + "all@20": 0.528, + "recall@20": 0.773 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.384, + "recall@5": 0.6775, + "all@10": 0.494, + "recall@10": 0.754, + "all@20": 0.528, + "recall@20": 0.773 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.408, + "recall@5": 0.7025, + "all@10": 0.502, + "recall@10": 0.758, + "all@20": 0.528, + "recall@20": 0.773 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.404, + "recall@5": 0.697, + "all@10": 0.502, + "recall@10": 0.758, + "all@20": 0.528, + "recall@20": 0.773 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.436, + "recall@5": 0.7205, + "all@10": 0.494, + "recall@10": 0.753, + "all@20": 0.528, + "recall@20": 0.773 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.438, + "recall@5": 0.7205, + "all@10": 0.498, + "recall@10": 0.755, + "all@20": 0.528, + "recall@20": 0.773 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.432, + "recall@5": 0.7185, + "all@10": 0.496, + "recall@10": 0.755, + "all@20": 0.528, + "recall@20": 0.773 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.44, + "recall@5": 0.7225, + "all@10": 0.498, + "recall@10": 0.756, + "all@20": 0.528, + "recall@20": 0.773 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.45, + "recall@5": 0.7345, + "all@10": 0.662, + "recall@10": 0.852, + "all@20": 0.788, + "recall@20": 0.9085 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.448, + "recall@5": 0.7335, + "all@10": 0.676, + "recall@10": 0.8565, + "all@20": 0.816, + "recall@20": 0.9195 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.45, + "recall@5": 0.7345, + "all@10": 0.698, + "recall@10": 0.8675, + "all@20": 0.828, + "recall@20": 0.927 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.45, + "recall@5": 0.7345, + "all@10": 0.704, + "recall@10": 0.87, + "all@20": 0.832, + "recall@20": 0.9285 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.44, + "recall@5": 0.723, + "all@10": 0.666, + "recall@10": 0.8565, + "all@20": 0.74, + "recall@20": 0.892 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.442, + "recall@5": 0.724, + "all@10": 0.674, + "recall@10": 0.86, + "all@20": 0.746, + "recall@20": 0.8945 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.424, + "recall@5": 0.7135, + "all@10": 0.552, + "recall@10": 0.7865, + "all@20": 0.6, + "recall@20": 0.8145 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.426, + "recall@5": 0.7145, + "all@10": 0.56, + "recall@10": 0.7905, + "all@20": 0.604, + "recall@20": 0.8165 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.378, + "recall@5": 0.68, + "all@10": 0.512, + "recall@10": 0.771, + "all@20": 0.726, + "recall@20": 0.886 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.368, + "recall@5": 0.6735, + "all@10": 0.51, + "recall@10": 0.7705, + "all@20": 0.736, + "recall@20": 0.891 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.406, + "recall@5": 0.706, + "all@10": 0.51, + "recall@10": 0.77, + "all@20": 0.754, + "recall@20": 0.897 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.41, + "recall@5": 0.7075, + "all@10": 0.508, + "recall@10": 0.7685, + "all@20": 0.76, + "recall@20": 0.8995 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.715, + "all@10": 0.502, + "recall@10": 0.76, + "all@20": 0.726, + "recall@20": 0.885 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.422, + "recall@5": 0.715, + "all@10": 0.5, + "recall@10": 0.759, + "all@20": 0.728, + "recall@20": 0.886 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.432, + "recall@5": 0.718, + "all@10": 0.484, + "recall@10": 0.7485, + "all@20": 0.598, + "recall@20": 0.8135 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.436, + "recall@5": 0.7215, + "all@10": 0.484, + "recall@10": 0.7485, + "all@20": 0.6, + "recall@20": 0.8145 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.588, + "all@10": 0.482, + "recall@10": 0.747, + "all@20": 0.574, + "recall@20": 0.804 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.298, + "recall@5": 0.5885, + "all@10": 0.478, + "recall@10": 0.7445, + "all@20": 0.576, + "recall@20": 0.805 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.376, + "recall@5": 0.6715, + "all@10": 0.51, + "recall@10": 0.766, + "all@20": 0.564, + "recall@20": 0.798 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.378, + "recall@5": 0.6725, + "all@10": 0.512, + "recall@10": 0.767, + "all@20": 0.564, + "recall@20": 0.798 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.406, + "recall@5": 0.7035, + "all@10": 0.516, + "recall@10": 0.765, + "all@20": 0.556, + "recall@20": 0.79 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.406, + "recall@5": 0.7035, + "all@10": 0.51, + "recall@10": 0.763, + "all@20": 0.554, + "recall@20": 0.7895 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.432, + "recall@5": 0.7195, + "all@10": 0.5, + "recall@10": 0.7565, + "all@20": 0.538, + "recall@20": 0.778 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.438, + "recall@5": 0.7225, + "all@10": 0.5, + "recall@10": 0.7565, + "all@20": 0.536, + "recall@20": 0.777 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.436, + "recall@5": 0.7315, + "all@10": 0.66, + "recall@10": 0.851, + "all@20": 0.776, + "recall@20": 0.904 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.438, + "recall@5": 0.7325, + "all@10": 0.658, + "recall@10": 0.849, + "all@20": 0.794, + "recall@20": 0.911 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.448, + "recall@5": 0.739, + "all@10": 0.678, + "recall@10": 0.859, + "all@20": 0.808, + "recall@20": 0.918 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.45, + "recall@5": 0.7395, + "all@10": 0.686, + "recall@10": 0.862, + "all@20": 0.818, + "recall@20": 0.922 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.448, + "recall@5": 0.736, + "all@10": 0.672, + "recall@10": 0.859, + "all@20": 0.762, + "recall@20": 0.9015 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.448, + "recall@5": 0.7355, + "all@10": 0.678, + "recall@10": 0.862, + "all@20": 0.764, + "recall@20": 0.903 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.432, + "recall@5": 0.718, + "all@10": 0.568, + "recall@10": 0.7975, + "all@20": 0.616, + "recall@20": 0.8245 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.432, + "recall@5": 0.718, + "all@10": 0.57, + "recall@10": 0.798, + "all@20": 0.616, + "recall@20": 0.825 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.324, + "recall@5": 0.6265, + "all@10": 0.528, + "recall@10": 0.783, + "all@20": 0.718, + "recall@20": 0.882 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.328, + "recall@5": 0.63, + "all@10": 0.526, + "recall@10": 0.7825, + "all@20": 0.728, + "recall@20": 0.8865 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.372, + "recall@5": 0.675, + "all@10": 0.534, + "recall@10": 0.7865, + "all@20": 0.742, + "recall@20": 0.8925 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.362, + "recall@5": 0.672, + "all@10": 0.534, + "recall@10": 0.787, + "all@20": 0.75, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.418, + "recall@5": 0.717, + "all@10": 0.522, + "recall@10": 0.777, + "all@20": 0.72, + "recall@20": 0.883 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.418, + "recall@5": 0.716, + "all@10": 0.526, + "recall@10": 0.778, + "all@20": 0.728, + "recall@20": 0.887 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.44, + "recall@5": 0.7225, + "all@10": 0.498, + "recall@10": 0.7555, + "all@20": 0.614, + "recall@20": 0.8235 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.436, + "recall@5": 0.7205, + "all@10": 0.496, + "recall@10": 0.7545, + "all@20": 0.612, + "recall@20": 0.823 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.2, + "recall@5": 0.47, + "all@10": 0.422, + "recall@10": 0.691, + "all@20": 0.594, + "recall@20": 0.817 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.19, + "recall@5": 0.4665, + "all@10": 0.418, + "recall@10": 0.6855, + "all@20": 0.59, + "recall@20": 0.815 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.286, + "recall@5": 0.575, + "all@10": 0.48, + "recall@10": 0.749, + "all@20": 0.594, + "recall@20": 0.8175 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.272, + "recall@5": 0.566, + "all@10": 0.494, + "recall@10": 0.756, + "all@20": 0.596, + "recall@20": 0.818 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.358, + "recall@5": 0.6755, + "all@10": 0.51, + "recall@10": 0.7675, + "all@20": 0.574, + "recall@20": 0.8035 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.372, + "recall@5": 0.68, + "all@10": 0.512, + "recall@10": 0.768, + "all@20": 0.574, + "recall@20": 0.8045 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.436, + "recall@5": 0.721, + "all@10": 0.504, + "recall@10": 0.7585, + "all@20": 0.544, + "recall@20": 0.783 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.436, + "recall@5": 0.721, + "all@10": 0.508, + "recall@10": 0.7605, + "all@20": 0.544, + "recall@20": 0.783 + } + ], + "ppr_selected": { + "5": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "20": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + } + }, + "lib_dev_grid": [ + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 25.964235399267636, + "default": true, + "all@5": 0.472, + "recall@5": 0.741, + "n_ret@5": 5.0, + "all@10": 0.716, + "recall@10": 0.879, + "n_ret@10": 9.994, + "all@20": 0.848, + "recall@20": 0.935, + "n_ret@20": 19.912 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 14.802927198936231, + "default": false, + "all@5": 0.562, + "recall@5": 0.801, + "n_ret@5": 5.0, + "all@10": 0.726, + "recall@10": 0.8835, + "n_ret@10": 10.0, + "all@20": 0.846, + "recall@20": 0.939, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 53.3103260002099, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 74.64119659934659, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 117.58022859937046, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 52.45265440118965, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.678, + "recall@20": 0.8565, + "n_ret@20": 19.958 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 89.51382299698889, + "default": false, + "all@5": 0.486, + "recall@5": 0.747, + "n_ret@5": 5.0, + "all@10": 0.578, + "recall@10": 0.7995, + "n_ret@10": 10.0, + "all@20": 0.686, + "recall@20": 0.8625, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 51.44594699982554, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.608, + "recall@20": 0.819, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 99.97704560053535, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 14.139295800006948, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.686, + "recall@20": 0.8595, + "n_ret@20": 19.958 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 53.11420520034153, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.608, + "recall@20": 0.819, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 84.04861159890424, + "default": false, + "all@5": 0.594, + "recall@5": 0.8115, + "n_ret@5": 5.0, + "all@10": 0.706, + "recall@10": 0.875, + "n_ret@10": 10.0, + "all@20": 0.798, + "recall@20": 0.9215, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 71.00688800017815, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.528, + "recall@20": 0.773, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 50.61851159995422, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.684, + "recall@20": 0.8615, + "n_ret@20": 19.958 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 11.731029199319892, + "default": false, + "all@5": 0.562, + "recall@5": 0.801, + "n_ret@5": 5.0, + "all@10": 0.73, + "recall@10": 0.883, + "n_ret@10": 9.962, + "all@20": 0.848, + "recall@20": 0.935, + "n_ret@20": 19.204 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 72.62786039873026, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.702, + "recall@20": 0.8755, + "n_ret@20": 19.98 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 15.148122999584302, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.758, + "recall@20": 0.9015, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 53.087144399876706, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 12.067207998945378, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.626, + "recall@10": 0.8335, + "n_ret@10": 9.962, + "all@20": 0.84, + "recall@20": 0.9315, + "n_ret@20": 19.204 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 55.72173739923164, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.744, + "recall@10": 0.893, + "n_ret@10": 9.994, + "all@20": 0.846, + "recall@20": 0.936, + "n_ret@20": 19.912 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 47.02616579947062, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.919, + "n_ret@20": 19.958 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 87.27009940100834, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.608, + "recall@20": 0.819, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 43.63208380131982, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.644, + "recall@10": 0.838, + "n_ret@10": 9.962, + "all@20": 0.772, + "recall@20": 0.9045, + "n_ret@20": 19.204 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 46.09904059825931, + "default": false, + "all@5": 0.486, + "recall@5": 0.747, + "n_ret@5": 5.0, + "all@10": 0.58, + "recall@10": 0.8, + "n_ret@10": 10.0, + "all@20": 0.706, + "recall@20": 0.874, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 102.61121879972052, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 43.3626200016588, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.658, + "recall@10": 0.8445, + "n_ret@10": 9.962, + "all@20": 0.806, + "recall@20": 0.9215, + "n_ret@20": 19.204 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 14.717656199121848, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.582, + "recall@20": 0.8035, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 26.566999198985286, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.626, + "recall@10": 0.8335, + "n_ret@10": 9.994, + "all@20": 0.84, + "recall@20": 0.9315, + "n_ret@20": 19.912 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 56.45319259981625, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.652, + "recall@10": 0.8415, + "n_ret@10": 9.994, + "all@20": 0.792, + "recall@20": 0.9165, + "n_ret@20": 19.912 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 47.675671799108386, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.648, + "recall@20": 0.844, + "n_ret@20": 19.958 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 49.441061998251826, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.608, + "recall@20": 0.82, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 16.015221999026835, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.528, + "recall@20": 0.773, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 108.1627736001974, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 69.36555560003035, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.662, + "recall@20": 0.849, + "n_ret@20": 19.98 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 65.40906580036972, + "default": false, + "all@5": 0.594, + "recall@5": 0.8115, + "n_ret@5": 5.0, + "all@10": 0.706, + "recall@10": 0.875, + "n_ret@10": 10.0, + "all@20": 0.82, + "recall@20": 0.9295, + "n_ret@20": 19.98 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 47.44155480037443, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.76, + "recall@20": 0.902, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 25.886367600644007, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.626, + "recall@10": 0.8335, + "n_ret@10": 9.994, + "all@20": 0.84, + "recall@20": 0.9315, + "n_ret@20": 19.912 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 46.79397600120865, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 96.08374719985295, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 43.75565839780029, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 11.266052601393312, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.82, + "recall@20": 0.926, + "n_ret@20": 19.958 + } + ], + "lib_selected": { + "5": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + } + }, + "n_lib_configs": 41, + "n_ppr_configs": 72, + "sub_ks": "10", + "seconds": 2353.4207859039307 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/2wiki__bge__entity.npz b/benchmarks/multihop/results/raw/2wiki__bge__entity.npz new file mode 100644 index 0000000..a98b800 Binary files /dev/null and b/benchmarks/multihop/results/raw/2wiki__bge__entity.npz differ diff --git a/benchmarks/multihop/results/raw/2wiki__bge__knn.json b/benchmarks/multihop/results/raw/2wiki__bge__knn.json new file mode 100644 index 0000000..51d4dfc --- /dev/null +++ b/benchmarks/multihop/results/raw/2wiki__bge__knn.json @@ -0,0 +1,2017 @@ +{ + "dataset": "2wiki", + "model": "bge", + "variant": "knn", + "families": [ + "knn" + ], + "diagnostics": { + "n_edges_undirected_union": 34499, + "n_edges_by_family_directed_or_undirected": 34499, + "mean_degree": 7.468124255871848, + "median_degree": 6.0, + "max_degree": 154, + "isolated_frac": 0.0, + "gold_pairs_directly_linked": 0.32220421393841164, + "gold_pairs_linked_within_2": 0.37212317666126415, + "neighbours_of_gold_that_are_in_same_question_context": 0.1272243175430044 + }, + "ppr_dev_grid": [ + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.58, + "recall@10": 0.803, + "all@20": 0.702, + "recall@20": 0.8725 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.604, + "recall@10": 0.8155, + "all@20": 0.71, + "recall@20": 0.8775 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.562, + "recall@10": 0.7905, + "all@20": 0.676, + "recall@20": 0.858 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.574, + "recall@10": 0.7965, + "all@20": 0.678, + "recall@20": 0.8595 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.52, + "recall@10": 0.7675, + "all@20": 0.594, + "recall@20": 0.81 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.524, + "recall@10": 0.77, + "all@20": 0.596, + "recall@20": 0.811 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.492, + "recall@10": 0.753, + "all@20": 0.544, + "recall@20": 0.7815 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.492, + "recall@10": 0.753, + "all@20": 0.544, + "recall@20": 0.7815 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.406, + "recall@5": 0.697, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.65, + "recall@20": 0.845 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.404, + "recall@5": 0.6965, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.66, + "recall@20": 0.8505 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.418, + "recall@5": 0.707, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.628, + "recall@20": 0.8295 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.414, + "recall@5": 0.7055, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.626, + "recall@20": 0.8315 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.436, + "recall@5": 0.7185, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.582, + "recall@20": 0.803 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.436, + "recall@5": 0.7185, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.586, + "recall@20": 0.805 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.426, + "recall@5": 0.7155, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.546, + "recall@20": 0.7825 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.424, + "recall@5": 0.7145, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.548, + "recall@20": 0.7835 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.364, + "recall@5": 0.658, + "all@10": 0.486, + "recall@10": 0.747, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.366, + "recall@5": 0.656, + "all@10": 0.486, + "recall@10": 0.747, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.396, + "recall@5": 0.691, + "all@10": 0.49, + "recall@10": 0.7505, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.396, + "recall@5": 0.691, + "all@10": 0.492, + "recall@10": 0.7515, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.418, + "recall@5": 0.7095, + "all@10": 0.492, + "recall@10": 0.752, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.422, + "recall@5": 0.711, + "all@10": 0.494, + "recall@10": 0.7525, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.426, + "recall@5": 0.7145, + "all@10": 0.484, + "recall@10": 0.7475, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.428, + "recall@5": 0.7155, + "all@10": 0.488, + "recall@10": 0.7495, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.424, + "recall@5": 0.713, + "all@10": 0.58, + "recall@10": 0.8035, + "all@20": 0.7, + "recall@20": 0.8715 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.424, + "recall@5": 0.713, + "all@10": 0.594, + "recall@10": 0.8115, + "all@20": 0.71, + "recall@20": 0.8775 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.424, + "recall@5": 0.713, + "all@10": 0.564, + "recall@10": 0.7925, + "all@20": 0.69, + "recall@20": 0.8665 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.424, + "recall@5": 0.713, + "all@10": 0.578, + "recall@10": 0.8, + "all@20": 0.694, + "recall@20": 0.869 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.424, + "recall@5": 0.713, + "all@10": 0.538, + "recall@10": 0.7775, + "all@20": 0.614, + "recall@20": 0.822 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.424, + "recall@5": 0.713, + "all@10": 0.546, + "recall@10": 0.781, + "all@20": 0.618, + "recall@20": 0.8245 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.424, + "recall@5": 0.713, + "all@10": 0.496, + "recall@10": 0.756, + "all@20": 0.552, + "recall@20": 0.7855 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.424, + "recall@5": 0.713, + "all@10": 0.496, + "recall@10": 0.756, + "all@20": 0.554, + "recall@20": 0.787 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.38, + "recall@5": 0.676, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.646, + "recall@20": 0.844 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.376, + "recall@5": 0.6735, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.664, + "recall@20": 0.853 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.404, + "recall@5": 0.6955, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.62, + "recall@20": 0.826 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.4, + "recall@5": 0.6935, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.626, + "recall@20": 0.8305 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.7095, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.594, + "recall@20": 0.8095 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.418, + "recall@5": 0.7075, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.6, + "recall@20": 0.813 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.43, + "recall@5": 0.717, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.556, + "recall@20": 0.7875 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.428, + "recall@5": 0.7165, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.556, + "recall@20": 0.788 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.314, + "recall@5": 0.5965, + "all@10": 0.468, + "recall@10": 0.7335, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.312, + "recall@5": 0.592, + "all@10": 0.468, + "recall@10": 0.7325, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.356, + "recall@5": 0.6505, + "all@10": 0.486, + "recall@10": 0.748, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.358, + "recall@5": 0.647, + "all@10": 0.482, + "recall@10": 0.745, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.402, + "recall@5": 0.694, + "all@10": 0.49, + "recall@10": 0.751, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.402, + "recall@5": 0.694, + "all@10": 0.494, + "recall@10": 0.7525, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.424, + "recall@5": 0.7125, + "all@10": 0.49, + "recall@10": 0.749, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.49, + "recall@10": 0.7495, + "all@20": 0.526, + "recall@20": 0.772 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.7105, + "all@10": 0.58, + "recall@10": 0.803, + "all@20": 0.698, + "recall@20": 0.87 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.598, + "recall@10": 0.811, + "all@20": 0.706, + "recall@20": 0.875 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.426, + "recall@5": 0.714, + "all@10": 0.562, + "recall@10": 0.792, + "all@20": 0.69, + "recall@20": 0.8675 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.422, + "recall@5": 0.712, + "all@10": 0.572, + "recall@10": 0.7985, + "all@20": 0.692, + "recall@20": 0.8685 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.428, + "recall@5": 0.715, + "all@10": 0.542, + "recall@10": 0.7795, + "all@20": 0.622, + "recall@20": 0.8285 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.428, + "recall@5": 0.715, + "all@10": 0.542, + "recall@10": 0.7805, + "all@20": 0.626, + "recall@20": 0.831 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.426, + "recall@5": 0.714, + "all@10": 0.504, + "recall@10": 0.759, + "all@20": 0.566, + "recall@20": 0.7935 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.428, + "recall@5": 0.715, + "all@10": 0.504, + "recall@10": 0.759, + "all@20": 0.566, + "recall@20": 0.794 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.348, + "recall@5": 0.6425, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.63, + "recall@20": 0.833 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.346, + "recall@5": 0.641, + "all@10": 0.472, + "recall@10": 0.742, + "all@20": 0.638, + "recall@20": 0.839 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.392, + "recall@5": 0.6855, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.62, + "recall@20": 0.825 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.388, + "recall@5": 0.6815, + "all@10": 0.478, + "recall@10": 0.745, + "all@20": 0.624, + "recall@20": 0.828 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.41, + "recall@5": 0.7015, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.6, + "recall@20": 0.8135 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.41, + "recall@5": 0.7015, + "all@10": 0.478, + "recall@10": 0.745, + "all@20": 0.6, + "recall@20": 0.8135 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.428, + "recall@5": 0.7155, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.558, + "recall@20": 0.789 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.428, + "recall@5": 0.715, + "all@10": 0.476, + "recall@10": 0.744, + "all@20": 0.564, + "recall@20": 0.7925 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.266, + "recall@5": 0.529, + "all@10": 0.442, + "recall@10": 0.7045, + "all@20": 0.534, + "recall@20": 0.7765 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.252, + "recall@5": 0.5155, + "all@10": 0.44, + "recall@10": 0.6975, + "all@20": 0.536, + "recall@20": 0.7775 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.326, + "recall@5": 0.6035, + "all@10": 0.468, + "recall@10": 0.736, + "all@20": 0.532, + "recall@20": 0.7755 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.322, + "recall@5": 0.5955, + "all@10": 0.468, + "recall@10": 0.7355, + "all@20": 0.534, + "recall@20": 0.7765 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.386, + "recall@5": 0.6775, + "all@10": 0.484, + "recall@10": 0.7475, + "all@20": 0.532, + "recall@20": 0.775 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.384, + "recall@5": 0.6765, + "all@10": 0.484, + "recall@10": 0.7475, + "all@20": 0.534, + "recall@20": 0.7765 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.7105, + "all@10": 0.494, + "recall@10": 0.752, + "all@20": 0.532, + "recall@20": 0.775 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.426, + "recall@5": 0.7125, + "all@10": 0.494, + "recall@10": 0.752, + "all@20": 0.534, + "recall@20": 0.776 + } + ], + "ppr_selected": { + "5": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "20": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + } + }, + "lib_dev_grid": [ + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 24.842296000337228, + "default": true, + "all@5": 0.438, + "recall@5": 0.721, + "n_ret@5": 5.0, + "all@10": 0.642, + "recall@10": 0.8405, + "n_ret@10": 10.0, + "all@20": 0.73, + "recall@20": 0.8865, + "n_ret@20": 19.978 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 15.367021197569557, + "default": false, + "all@5": 0.438, + "recall@5": 0.72, + "n_ret@5": 5.0, + "all@10": 0.656, + "recall@10": 0.846, + "n_ret@10": 10.0, + "all@20": 0.728, + "recall@20": 0.886, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 52.871395400259644, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 79.98466900049243, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 119.69893800129648, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 52.02228739857674, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.618, + "recall@20": 0.822, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 86.9482675998006, + "default": false, + "all@5": 0.466, + "recall@5": 0.7345, + "n_ret@5": 5.0, + "all@10": 0.546, + "recall@10": 0.782, + "n_ret@10": 10.0, + "all@20": 0.622, + "recall@20": 0.826, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 53.372957198298536, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.592, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 101.85373279987834, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 14.32496160082519, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.706, + "recall@20": 0.8765, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 52.04616440006066, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.624, + "recall@20": 0.83, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 81.91277320019435, + "default": false, + "all@5": 0.484, + "recall@5": 0.744, + "n_ret@5": 5.0, + "all@10": 0.632, + "recall@10": 0.8325, + "n_ret@10": 10.0, + "all@20": 0.708, + "recall@20": 0.8755, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 75.31337620096747, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 50.38138040190097, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.656, + "recall@20": 0.848, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 12.143615799373947, + "default": false, + "all@5": 0.438, + "recall@5": 0.72, + "n_ret@5": 5.0, + "all@10": 0.658, + "recall@10": 0.846, + "n_ret@10": 9.996, + "all@20": 0.728, + "recall@20": 0.885, + "n_ret@20": 19.788 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 75.09753559855744, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.478, + "recall@10": 0.745, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.8325, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 14.549843598157167, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.688, + "recall@20": 0.8655, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 51.63233640010003, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 12.254491599975154, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.666, + "recall@10": 0.855, + "n_ret@10": 9.996, + "all@20": 0.718, + "recall@20": 0.88, + "n_ret@20": 19.788 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 58.229798799497075, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.636, + "recall@10": 0.8305, + "n_ret@10": 10.0, + "all@20": 0.72, + "recall@20": 0.88, + "n_ret@20": 19.978 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 45.283698000363074, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.696, + "recall@20": 0.869, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 85.50054399902001, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.592, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 45.329123998992145, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.578, + "recall@10": 0.7985, + "n_ret@10": 9.996, + "all@20": 0.654, + "recall@20": 0.8455, + "n_ret@20": 19.788 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 49.75840440078173, + "default": false, + "all@5": 0.47, + "recall@5": 0.7365, + "n_ret@5": 5.0, + "all@10": 0.554, + "recall@10": 0.786, + "n_ret@10": 10.0, + "all@20": 0.636, + "recall@20": 0.833, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 103.66720479982905, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 45.885010999510996, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.62, + "recall@10": 0.828, + "n_ret@10": 9.996, + "all@20": 0.69, + "recall@20": 0.866, + "n_ret@20": 19.788 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 15.681103799608538, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.68, + "recall@20": 0.864, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 27.0679563988233, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.666, + "recall@10": 0.855, + "n_ret@10": 10.0, + "all@20": 0.72, + "recall@20": 0.881, + "n_ret@20": 19.978 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 59.178716000169516, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.61, + "recall@10": 0.821, + "n_ret@10": 10.0, + "all@20": 0.688, + "recall@20": 0.865, + "n_ret@20": 19.978 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 47.88553140091244, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.604, + "recall@20": 0.815, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 52.049629800603725, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.604, + "recall@20": 0.8135, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 18.250650599831715, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 98.96375840029214, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 68.69485039974097, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.598, + "recall@20": 0.811, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 63.990619600750506, + "default": false, + "all@5": 0.484, + "recall@5": 0.744, + "n_ret@5": 5.0, + "all@10": 0.632, + "recall@10": 0.8325, + "n_ret@10": 10.0, + "all@20": 0.718, + "recall@20": 0.882, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 44.45963719999418, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.688, + "recall@20": 0.866, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 25.02902580006048, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.666, + "recall@10": 0.855, + "n_ret@10": 10.0, + "all@20": 0.72, + "recall@20": 0.881, + "n_ret@20": 19.978 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 45.984056201647036, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 94.78861040063202, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 46.004347800742835, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 12.435969198588282, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.718, + "recall@20": 0.8825, + "n_ret@20": 20.0 + } + ], + "lib_selected": { + "5": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "20": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + } + }, + "n_lib_configs": 41, + "n_ppr_configs": 72, + "sub_ks": "10", + "seconds": 2296.73876619339 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/2wiki__bge__knn.npz b/benchmarks/multihop/results/raw/2wiki__bge__knn.npz new file mode 100644 index 0000000..a0e4ec0 Binary files /dev/null and b/benchmarks/multihop/results/raw/2wiki__bge__knn.npz differ diff --git a/benchmarks/multihop/results/raw/2wiki__bge__mention.json b/benchmarks/multihop/results/raw/2wiki__bge__mention.json new file mode 100644 index 0000000..ad7358b --- /dev/null +++ b/benchmarks/multihop/results/raw/2wiki__bge__mention.json @@ -0,0 +1,3274 @@ +{ + "dataset": "2wiki", + "model": "bge", + "variant": "mention", + "families": [ + "mention" + ], + "diagnostics": { + "n_edges_undirected_union": 5383, + "n_edges_by_family_directed_or_undirected": 6115, + "mean_degree": 1.165277627448858, + "median_degree": 1.0, + "max_degree": 30, + "isolated_frac": 0.4217988959844139, + "gold_pairs_directly_linked": 0.44278768233387356, + "gold_pairs_linked_within_2": 0.4447325769854133, + "neighbours_of_gold_that_are_in_same_question_context": 0.42326321090525637 + }, + "ppr_dev_grid": [ + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.448, + "recall@5": 0.7245, + "all@10": 0.92, + "recall@10": 0.9645, + "all@20": 0.946, + "recall@20": 0.972 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.43, + "recall@5": 0.716, + "all@10": 0.9, + "recall@10": 0.9545, + "all@20": 0.934, + "recall@20": 0.967 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.446, + "recall@5": 0.724, + "all@10": 0.928, + "recall@10": 0.97, + "all@20": 0.952, + "recall@20": 0.977 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.432, + "recall@5": 0.717, + "all@10": 0.898, + "recall@10": 0.955, + "all@20": 0.94, + "recall@20": 0.9725 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.432, + "recall@5": 0.717, + "all@10": 0.868, + "recall@10": 0.948, + "all@20": 0.886, + "recall@20": 0.955 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.43, + "recall@5": 0.716, + "all@10": 0.826, + "recall@10": 0.9295, + "all@20": 0.854, + "recall@20": 0.942 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.428, + "recall@5": 0.715, + "all@10": 0.588, + "recall@10": 0.8085, + "all@20": 0.648, + "recall@20": 0.844 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.424, + "recall@5": 0.713, + "all@10": 0.582, + "recall@10": 0.8055, + "all@20": 0.638, + "recall@20": 0.837 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.448, + "recall@5": 0.7155, + "all@10": 0.496, + "recall@10": 0.7555, + "all@20": 0.952, + "recall@20": 0.979 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.342, + "recall@5": 0.653, + "all@10": 0.482, + "recall@10": 0.747, + "all@20": 0.93, + "recall@20": 0.9695 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.46, + "recall@5": 0.7275, + "all@10": 0.494, + "recall@10": 0.7545, + "all@20": 0.94, + "recall@20": 0.9755 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.394, + "recall@5": 0.691, + "all@10": 0.48, + "recall@10": 0.746, + "all@20": 0.924, + "recall@20": 0.9685 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.464, + "recall@5": 0.7335, + "all@10": 0.492, + "recall@10": 0.752, + "all@20": 0.878, + "recall@20": 0.952 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.446, + "recall@5": 0.727, + "all@10": 0.478, + "recall@10": 0.745, + "all@20": 0.852, + "recall@20": 0.9425 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.458, + "recall@5": 0.7315, + "all@10": 0.478, + "recall@10": 0.745, + "all@20": 0.638, + "recall@20": 0.8395 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.456, + "recall@5": 0.7335, + "all@10": 0.478, + "recall@10": 0.745, + "all@20": 0.628, + "recall@20": 0.8315 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.396, + "recall@5": 0.6405, + "all@10": 0.534, + "recall@10": 0.775, + "all@20": 0.548, + "recall@20": 0.7845 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.264, + "recall@5": 0.57, + "all@10": 0.436, + "recall@10": 0.7165, + "all@20": 0.536, + "recall@20": 0.778 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.444, + "recall@5": 0.6975, + "all@10": 0.536, + "recall@10": 0.7765, + "all@20": 0.544, + "recall@20": 0.7825 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.358, + "recall@5": 0.6635, + "all@10": 0.49, + "recall@10": 0.7525, + "all@20": 0.534, + "recall@20": 0.776 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.476, + "recall@5": 0.7395, + "all@10": 0.532, + "recall@10": 0.7735, + "all@20": 0.542, + "recall@20": 0.78 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.456, + "recall@5": 0.733, + "all@10": 0.512, + "recall@10": 0.765, + "all@20": 0.534, + "recall@20": 0.776 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.474, + "recall@5": 0.741, + "all@10": 0.514, + "recall@10": 0.766, + "all@20": 0.534, + "recall@20": 0.776 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.464, + "recall@5": 0.739, + "all@10": 0.502, + "recall@10": 0.76, + "all@20": 0.532, + "recall@20": 0.775 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.478, + "recall@5": 0.7395, + "all@10": 0.902, + "recall@10": 0.958, + "all@20": 0.946, + "recall@20": 0.972 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.542, + "recall@5": 0.7825, + "all@10": 0.89, + "recall@10": 0.9515, + "all@20": 0.936, + "recall@20": 0.968 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.476, + "recall@5": 0.744, + "all@10": 0.918, + "recall@10": 0.966, + "all@20": 0.952, + "recall@20": 0.977 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.552, + "recall@5": 0.792, + "all@10": 0.9, + "recall@10": 0.956, + "all@20": 0.94, + "recall@20": 0.972 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.476, + "recall@5": 0.7435, + "all@10": 0.874, + "recall@10": 0.952, + "all@20": 0.906, + "recall@20": 0.962 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.512, + "recall@5": 0.766, + "all@10": 0.866, + "recall@10": 0.945, + "all@20": 0.904, + "recall@20": 0.9605 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.446, + "recall@5": 0.726, + "all@10": 0.618, + "recall@10": 0.8265, + "all@20": 0.68, + "recall@20": 0.8615 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.442, + "recall@5": 0.7235, + "all@10": 0.63, + "recall@10": 0.8325, + "all@20": 0.688, + "recall@20": 0.8655 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.398, + "recall@5": 0.6185, + "all@10": 0.554, + "recall@10": 0.7895, + "all@20": 0.932, + "recall@20": 0.973 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.312, + "recall@5": 0.576, + "all@10": 0.616, + "recall@10": 0.814, + "all@20": 0.92, + "recall@20": 0.967 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.448, + "recall@5": 0.7005, + "all@10": 0.546, + "recall@10": 0.7885, + "all@20": 0.934, + "recall@20": 0.973 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.376, + "recall@5": 0.6535, + "all@10": 0.644, + "recall@10": 0.841, + "all@20": 0.92, + "recall@20": 0.967 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.482, + "recall@5": 0.7405, + "all@10": 0.53, + "recall@10": 0.777, + "all@20": 0.872, + "recall@20": 0.952 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.418, + "recall@5": 0.7035, + "all@10": 0.582, + "recall@10": 0.8075, + "all@20": 0.892, + "recall@20": 0.9575 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.468, + "recall@5": 0.735, + "all@10": 0.506, + "recall@10": 0.7605, + "all@20": 0.656, + "recall@20": 0.847 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.454, + "recall@5": 0.7325, + "all@10": 0.504, + "recall@10": 0.7595, + "all@20": 0.674, + "recall@20": 0.8565 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.26, + "recall@5": 0.4505, + "all@10": 0.51, + "recall@10": 0.7175, + "all@20": 0.602, + "recall@20": 0.82 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.226, + "recall@5": 0.466, + "all@10": 0.406, + "recall@10": 0.643, + "all@20": 0.694, + "recall@20": 0.8545 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.362, + "recall@5": 0.5855, + "all@10": 0.558, + "recall@10": 0.7855, + "all@20": 0.59, + "recall@20": 0.8135 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.238, + "recall@5": 0.5165, + "all@10": 0.464, + "recall@10": 0.7215, + "all@20": 0.726, + "recall@20": 0.8815 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.454, + "recall@5": 0.703, + "all@10": 0.548, + "recall@10": 0.785, + "all@20": 0.566, + "recall@20": 0.7985 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.368, + "recall@5": 0.6685, + "all@10": 0.526, + "recall@10": 0.7715, + "all@20": 0.64, + "recall@20": 0.8405 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.48, + "recall@5": 0.743, + "all@10": 0.524, + "recall@10": 0.77, + "all@20": 0.544, + "recall@20": 0.782 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.468, + "recall@5": 0.741, + "all@10": 0.51, + "recall@10": 0.764, + "all@20": 0.566, + "recall@20": 0.794 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.532, + "recall@5": 0.742, + "all@10": 0.878, + "recall@10": 0.9495, + "all@20": 0.946, + "recall@20": 0.972 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.486, + "recall@5": 0.72, + "all@10": 0.874, + "recall@10": 0.9425, + "all@20": 0.932, + "recall@20": 0.9665 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.504, + "recall@5": 0.743, + "all@10": 0.884, + "recall@10": 0.9545, + "all@20": 0.946, + "recall@20": 0.975 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.546, + "recall@5": 0.7615, + "all@10": 0.882, + "recall@10": 0.9485, + "all@20": 0.936, + "recall@20": 0.9705 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.494, + "recall@5": 0.755, + "all@10": 0.82, + "recall@10": 0.9335, + "all@20": 0.866, + "recall@20": 0.9505 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.566, + "recall@5": 0.8, + "all@10": 0.862, + "recall@10": 0.942, + "all@20": 0.912, + "recall@20": 0.9635 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.466, + "recall@5": 0.737, + "all@10": 0.62, + "recall@10": 0.828, + "all@20": 0.666, + "recall@20": 0.8555 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.492, + "recall@5": 0.7525, + "all@10": 0.654, + "recall@10": 0.8455, + "all@20": 0.722, + "recall@20": 0.882 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.34, + "recall@5": 0.5335, + "all@10": 0.666, + "recall@10": 0.8425, + "all@20": 0.918, + "recall@20": 0.968 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.316, + "recall@5": 0.5255, + "all@10": 0.546, + "recall@10": 0.727, + "all@20": 0.908, + "recall@20": 0.96 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.372, + "recall@5": 0.597, + "all@10": 0.6, + "recall@10": 0.815, + "all@20": 0.914, + "recall@20": 0.9675 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.332, + "recall@5": 0.571, + "all@10": 0.64, + "recall@10": 0.8145, + "all@20": 0.91, + "recall@20": 0.9635 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.448, + "recall@5": 0.6985, + "all@10": 0.566, + "recall@10": 0.7995, + "all@20": 0.836, + "recall@20": 0.937 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.428, + "recall@5": 0.685, + "all@10": 0.672, + "recall@10": 0.857, + "all@20": 0.902, + "recall@20": 0.96 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.486, + "recall@5": 0.745, + "all@10": 0.52, + "recall@10": 0.7695, + "all@20": 0.64, + "recall@20": 0.8395 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.464, + "recall@5": 0.7385, + "all@10": 0.548, + "recall@10": 0.786, + "all@20": 0.694, + "recall@20": 0.8705 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.156, + "recall@5": 0.3165, + "all@10": 0.432, + "recall@10": 0.6045, + "all@20": 0.778, + "recall@20": 0.9055 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.262, + "recall@5": 0.4745, + "all@10": 0.374, + "recall@10": 0.5695, + "all@20": 0.566, + "recall@20": 0.732 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.224, + "recall@5": 0.404, + "all@10": 0.5, + "recall@10": 0.697, + "all@20": 0.648, + "recall@20": 0.845 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.276, + "recall@5": 0.4865, + "all@10": 0.416, + "recall@10": 0.6205, + "all@20": 0.712, + "recall@20": 0.8555 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.38, + "recall@5": 0.6045, + "all@10": 0.56, + "recall@10": 0.786, + "all@20": 0.596, + "recall@20": 0.817 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.328, + "recall@5": 0.5865, + "all@10": 0.534, + "recall@10": 0.7695, + "all@20": 0.722, + "recall@20": 0.8825 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.478, + "recall@5": 0.738, + "all@10": 0.53, + "recall@10": 0.7745, + "all@20": 0.548, + "recall@20": 0.786 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.456, + "recall@5": 0.7345, + "all@10": 0.53, + "recall@10": 0.7745, + "all@20": 0.606, + "recall@20": 0.818 + } + ], + "ppr_selected": { + "5": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "20": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + } + }, + "lib_dev_grid": [ + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.804044398595579, + "default": true, + "all@5": 0.512, + "recall@5": 0.764, + "n_ret@5": 5.0, + "all@10": 0.938, + "recall@10": 0.97, + "n_ret@10": 8.366, + "all@20": 0.946, + "recall@20": 0.972, + "n_ret@20": 9.86 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.721563198720105, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.482, + "recall@10": 0.747, + "n_ret@10": 10.0, + "all@20": 0.96, + "recall@20": 0.981, + "n_ret@20": 17.578 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 29.39426799863577, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.778, + "recall@20": 0.9145, + "n_ret@20": 19.868 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 29.694834199151956, + "default": false, + "all@5": 0.508, + "recall@5": 0.758, + "n_ret@5": 5.0, + "all@10": 0.588, + "recall@10": 0.804, + "n_ret@10": 10.0, + "all@20": 0.644, + "recall@20": 0.841, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 27.502350000548176, + "default": false, + "all@5": 0.51, + "recall@5": 0.76, + "n_ret@5": 5.0, + "all@10": 0.9, + "recall@10": 0.9585, + "n_ret@10": 8.366, + "all@20": 0.946, + "recall@20": 0.972, + "n_ret@20": 9.86 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 29.061040399363264, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 7.296280400827527, + "default": false, + "all@5": 0.78, + "recall@5": 0.9115, + "n_ret@5": 5.0, + "all@10": 0.938, + "recall@10": 0.97, + "n_ret@10": 9.468, + "all@20": 0.942, + "recall@20": 0.971, + "n_ret@20": 15.534 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 31.901256398530677, + "default": false, + "all@5": 0.644, + "recall@5": 0.838, + "n_ret@5": 5.0, + "all@10": 0.76, + "recall@10": 0.903, + "n_ret@10": 10.0, + "all@20": 0.824, + "recall@20": 0.9335, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.238364599179476, + "default": false, + "all@5": 0.78, + "recall@5": 0.9115, + "n_ret@5": 5.0, + "all@10": 0.908, + "recall@10": 0.9635, + "n_ret@10": 10.0, + "all@20": 0.956, + "recall@20": 0.9795, + "n_ret@20": 19.748 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.12475059949793, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.482, + "recall@10": 0.747, + "n_ret@10": 10.0, + "all@20": 0.96, + "recall@20": 0.981, + "n_ret@20": 16.474 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 29.826584001420997, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 7.193369799177162, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.934, + "recall@10": 0.969, + "n_ret@10": 9.468, + "all@20": 0.946, + "recall@20": 0.972, + "n_ret@20": 15.534 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.198859800235368, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.954, + "recall@20": 0.979, + "n_ret@20": 19.868 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 29.75518059823662, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.908, + "recall@10": 0.959, + "n_ret@10": 9.002, + "all@20": 0.946, + "recall@20": 0.972, + "n_ret@20": 11.706 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 30.160872000386007, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.916, + "recall@20": 0.9685, + "n_ret@20": 17.79 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 29.555952400784008, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.478, + "recall@10": 0.745, + "n_ret@10": 10.0, + "all@20": 0.842, + "recall@20": 0.9415, + "n_ret@20": 19.748 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 7.471093802596442, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.934, + "recall@10": 0.969, + "n_ret@10": 9.002, + "all@20": 0.946, + "recall@20": 0.972, + "n_ret@20": 11.706 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 28.75617039820645, + "default": false, + "all@5": 0.644, + "recall@5": 0.838, + "n_ret@5": 5.0, + "all@10": 0.76, + "recall@10": 0.903, + "n_ret@10": 10.0, + "all@20": 0.944, + "recall@20": 0.9765, + "n_ret@20": 19.61 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 11.275420801714063, + "default": false, + "all@5": 0.78, + "recall@5": 0.9115, + "n_ret@5": 5.0, + "all@10": 0.904, + "recall@10": 0.9625, + "n_ret@10": 10.0, + "all@20": 0.938, + "recall@20": 0.9735, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 6.2304738006787375, + "default": false, + "all@5": 0.78, + "recall@5": 0.9115, + "n_ret@5": 5.0, + "all@10": 0.904, + "recall@10": 0.9625, + "n_ret@10": 10.0, + "all@20": 0.958, + "recall@20": 0.9805, + "n_ret@20": 18.828 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 26.622458399506286, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.478, + "recall@10": 0.745, + "n_ret@10": 10.0, + "all@20": 0.956, + "recall@20": 0.98, + "n_ret@20": 17.578 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 26.10796979884617, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.478, + "recall@10": 0.745, + "n_ret@10": 10.0, + "all@20": 0.956, + "recall@20": 0.98, + "n_ret@20": 17.79 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 8.137769997003488, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.958, + "recall@20": 0.9805, + "n_ret@20": 19.706 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 26.868533200933598, + "default": false, + "all@5": 0.508, + "recall@5": 0.758, + "n_ret@5": 5.0, + "all@10": 0.588, + "recall@10": 0.804, + "n_ret@10": 10.0, + "all@20": 0.788, + "recall@20": 0.9195, + "n_ret@20": 19.976 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 28.105736599420197, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.832, + "recall@20": 0.9355, + "n_ret@20": 19.61 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 27.47347820107825, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.478, + "recall@10": 0.745, + "n_ret@10": 10.0, + "all@20": 0.528, + "recall@20": 0.773, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 26.77929680107627, + "default": false, + "all@5": 0.506, + "recall@5": 0.757, + "n_ret@5": 5.0, + "all@10": 0.796, + "recall@10": 0.919, + "n_ret@10": 9.468, + "all@20": 0.916, + "recall@20": 0.9625, + "n_ret@20": 15.534 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 28.703389199799858, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.854, + "recall@20": 0.9455, + "n_ret@20": 19.988 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 29.251413398887962, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.884, + "recall@20": 0.957, + "n_ret@20": 19.706 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.521938000689261, + "default": false, + "all@5": 0.78, + "recall@5": 0.9115, + "n_ret@5": 5.0, + "all@10": 0.908, + "recall@10": 0.9635, + "n_ret@10": 10.0, + "all@20": 0.94, + "recall@20": 0.974, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 27.95988379814662, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 27.528164399787784, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.478, + "recall@10": 0.745, + "n_ret@10": 10.0, + "all@20": 0.842, + "recall@20": 0.941, + "n_ret@20": 19.868 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.988831198308617, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.482, + "recall@10": 0.747, + "n_ret@10": 10.0, + "all@20": 0.954, + "recall@20": 0.979, + "n_ret@20": 19.748 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 27.980914601357654, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 28.204360399977304, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.804, + "recall@10": 0.92, + "n_ret@10": 9.53, + "all@20": 0.93, + "recall@20": 0.9665, + "n_ret@20": 16.586 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 25.9935732006561, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.904, + "recall@20": 0.9655, + "n_ret@20": 19.868 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.685947798890993, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.938, + "recall@10": 0.97, + "n_ret@10": 8.438, + "all@20": 0.946, + "recall@20": 0.972, + "n_ret@20": 10.452 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.871445999597199, + "default": false, + "all@5": 0.78, + "recall@5": 0.9115, + "n_ret@5": 5.0, + "all@10": 0.908, + "recall@10": 0.9635, + "n_ret@10": 10.0, + "all@20": 0.956, + "recall@20": 0.9795, + "n_ret@20": 19.868 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 6.093692199559882, + "default": false, + "all@5": 0.78, + "recall@5": 0.9115, + "n_ret@5": 5.0, + "all@10": 0.904, + "recall@10": 0.9625, + "n_ret@10": 10.0, + "all@20": 0.95, + "recall@20": 0.9775, + "n_ret@20": 19.976 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 27.910317602101713, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.296203201869503, + "default": false, + "all@5": 0.78, + "recall@5": 0.9115, + "n_ret@5": 5.0, + "all@10": 0.94, + "recall@10": 0.9705, + "n_ret@10": 8.366, + "all@20": 0.946, + "recall@20": 0.972, + "n_ret@20": 9.86 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 10.78531760070473, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.420444599818438, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.482, + "recall@10": 0.747, + "n_ret@10": 10.0, + "all@20": 0.954, + "recall@20": 0.979, + "n_ret@20": 19.868 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 27.642092400812544, + "default": false, + "all@5": 0.506, + "recall@5": 0.757, + "n_ret@5": 5.0, + "all@10": 0.588, + "recall@10": 0.804, + "n_ret@10": 10.0, + "all@20": 0.644, + "recall@20": 0.841, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 26.781346400734037, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.94, + "recall@20": 0.976, + "n_ret@20": 16.474 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 26.42311760049779, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.902, + "recall@10": 0.9585, + "n_ret@10": 8.366, + "all@20": 0.946, + "recall@20": 0.972, + "n_ret@20": 9.86 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 27.106535999220796, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 9.232422801549546, + "default": false, + "all@5": 0.78, + "recall@5": 0.9115, + "n_ret@5": 5.0, + "all@10": 0.904, + "recall@10": 0.9625, + "n_ret@10": 10.0, + "all@20": 0.948, + "recall@20": 0.977, + "n_ret@20": 19.988 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 26.22369539900683, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.804, + "recall@10": 0.92, + "n_ret@10": 9.468, + "all@20": 0.93, + "recall@20": 0.9665, + "n_ret@20": 15.534 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 29.38549320178572, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.478, + "recall@10": 0.745, + "n_ret@10": 10.0, + "all@20": 0.844, + "recall@20": 0.941, + "n_ret@20": 19.61 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 28.430660399026237, + "default": false, + "all@5": 0.644, + "recall@5": 0.838, + "n_ret@5": 5.0, + "all@10": 0.762, + "recall@10": 0.9035, + "n_ret@10": 10.0, + "all@20": 0.946, + "recall@20": 0.977, + "n_ret@20": 18.828 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.722401999984868, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 28.102778599830344, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.814, + "recall@20": 0.9305, + "n_ret@20": 19.976 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 30.11347940028645, + "default": false, + "all@5": 0.51, + "recall@5": 0.76, + "n_ret@5": 5.0, + "all@10": 0.588, + "recall@10": 0.804, + "n_ret@10": 10.0, + "all@20": 0.932, + "recall@20": 0.973, + "n_ret@20": 17.79 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 7.886546398396604, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.482, + "recall@10": 0.747, + "n_ret@10": 10.0, + "all@20": 0.958, + "recall@20": 0.9805, + "n_ret@20": 19.61 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 6.962971001048572, + "default": false, + "all@5": 0.78, + "recall@5": 0.9115, + "n_ret@5": 5.0, + "all@10": 0.904, + "recall@10": 0.9625, + "n_ret@10": 10.0, + "all@20": 0.94, + "recall@20": 0.974, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 30.13092579948716, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 29.935367400874384, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.478, + "recall@10": 0.745, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 26.709233402623795, + "default": false, + "all@5": 0.644, + "recall@5": 0.838, + "n_ret@5": 5.0, + "all@10": 0.906, + "recall@10": 0.9585, + "n_ret@10": 9.468, + "all@20": 0.94, + "recall@20": 0.9705, + "n_ret@20": 15.534 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 26.01809200027492, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.808, + "recall@20": 0.927, + "n_ret@20": 19.876 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 27.501755201257765, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.718, + "recall@20": 0.884, + "n_ret@20": 19.988 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.3, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 9.598686599871144, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 25.380085199140012, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.77, + "recall@10": 0.906, + "n_ret@10": 9.468, + "all@20": 0.922, + "recall@20": 0.964, + "n_ret@20": 15.534 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 25.482977401581593, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.922, + "recall@20": 0.9685, + "n_ret@20": 18.828 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 26.361152397003025, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.814, + "recall@20": 0.9305, + "n_ret@20": 19.988 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 26.184054601122625, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.8, + "recall@20": 0.921, + "n_ret@20": 19.61 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 25.876473798649386, + "default": false, + "all@5": 0.654, + "recall@5": 0.844, + "n_ret@5": 5.0, + "all@10": 0.764, + "recall@10": 0.905, + "n_ret@10": 10.0, + "all@20": 0.954, + "recall@20": 0.9795, + "n_ret@20": 17.578 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 25.114452598965727, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.478, + "recall@10": 0.745, + "n_ret@10": 10.0, + "all@20": 0.842, + "recall@20": 0.941, + "n_ret@20": 19.876 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.16260160098318, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.954, + "recall@20": 0.979, + "n_ret@20": 19.868 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 25.892845599213615, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.798, + "recall@20": 0.9205, + "n_ret@20": 19.706 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 6.120450198999606, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.948, + "recall@20": 0.977, + "n_ret@20": 19.976 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 26.814557400764897, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 25.29205260029994, + "default": false, + "all@5": 0.644, + "recall@5": 0.838, + "n_ret@5": 5.0, + "all@10": 0.762, + "recall@10": 0.9035, + "n_ret@10": 10.0, + "all@20": 0.824, + "recall@20": 0.9335, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 27.356733600841835, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.922, + "recall@20": 0.9685, + "n_ret@20": 19.706 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.474992799456231, + "default": false, + "all@5": 0.78, + "recall@5": 0.9115, + "n_ret@5": 5.0, + "all@10": 0.942, + "recall@10": 0.971, + "n_ret@10": 8.018, + "all@20": 0.946, + "recall@20": 0.972, + "n_ret@20": 8.546 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 24.91571559989825, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.928, + "recall@10": 0.9675, + "n_ret@10": 8.366, + "all@20": 0.946, + "recall@20": 0.972, + "n_ret@20": 9.86 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 26.453116198652424, + "default": false, + "all@5": 0.51, + "recall@5": 0.76, + "n_ret@5": 5.0, + "all@10": 0.588, + "recall@10": 0.804, + "n_ret@10": 10.0, + "all@20": 0.854, + "recall@20": 0.9455, + "n_ret@20": 19.868 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 52.06280939967837, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.478, + "recall@10": 0.745, + "n_ret@10": 10.0, + "all@20": 0.766, + "recall@20": 0.9065, + "n_ret@20": 19.988 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 51.615499400068074, + "default": false, + "all@5": 0.654, + "recall@5": 0.844, + "n_ret@5": 5.0, + "all@10": 0.764, + "recall@10": 0.905, + "n_ret@10": 10.0, + "all@20": 0.826, + "recall@20": 0.9345, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 50.634756402112544, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.848, + "recall@10": 0.939, + "n_ret@10": 9.468, + "all@20": 0.94, + "recall@20": 0.97, + "n_ret@20": 15.534 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 12.156288599479012, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.938, + "recall@10": 0.97, + "n_ret@10": 8.018, + "all@20": 0.946, + "recall@20": 0.972, + "n_ret@20": 8.546 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 52.045509200077504, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.478, + "recall@10": 0.745, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 11.26719799998682, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.938, + "recall@10": 0.97, + "n_ret@10": 8.366, + "all@20": 0.946, + "recall@20": 0.972, + "n_ret@20": 9.86 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 50.72812140232418, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.794, + "recall@10": 0.9165, + "n_ret@10": 9.002, + "all@20": 0.934, + "recall@20": 0.9675, + "n_ret@20": 11.706 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 50.802479597972706, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.854, + "recall@20": 0.9455, + "n_ret@20": 19.976 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 60.20642020087689, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.478, + "recall@10": 0.745, + "n_ret@10": 10.0, + "all@20": 0.528, + "recall@20": 0.773, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 14.341141001088545, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.958, + "recall@20": 0.9805, + "n_ret@20": 19.61 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 54.708737000939436, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 12.913764001103118, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.476, + "recall@10": 0.744, + "n_ret@10": 10.0, + "all@20": 0.954, + "recall@20": 0.979, + "n_ret@20": 19.876 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 15.638072800938971, + "default": false, + "all@5": 0.78, + "recall@5": 0.9115, + "n_ret@5": 5.0, + "all@10": 0.904, + "recall@10": 0.9625, + "n_ret@10": 10.0, + "all@20": 0.948, + "recall@20": 0.977, + "n_ret@20": 19.988 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 53.86375959881116, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.478, + "recall@10": 0.745, + "n_ret@10": 10.0, + "all@20": 0.934, + "recall@20": 0.974, + "n_ret@20": 17.79 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 55.02060040005017, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.478, + "recall@10": 0.745, + "n_ret@10": 10.0, + "all@20": 0.942, + "recall@20": 0.9765, + "n_ret@20": 16.474 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 16.13294979883358, + "default": false, + "all@5": 0.78, + "recall@5": 0.9115, + "n_ret@5": 5.0, + "all@10": 0.904, + "recall@10": 0.9625, + "n_ret@10": 10.0, + "all@20": 0.938, + "recall@20": 0.9735, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 54.47277520003263, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.874, + "recall@10": 0.9485, + "n_ret@10": 9.468, + "all@20": 0.946, + "recall@20": 0.972, + "n_ret@20": 15.534 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 52.121233400539495, + "default": false, + "all@5": 0.654, + "recall@5": 0.844, + "n_ret@5": 5.0, + "all@10": 0.764, + "recall@10": 0.905, + "n_ret@10": 10.0, + "all@20": 0.826, + "recall@20": 0.9345, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 12.217201801599003, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.938, + "recall@10": 0.97, + "n_ret@10": 8.438, + "all@20": 0.946, + "recall@20": 0.972, + "n_ret@20": 10.452 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 53.479334000032395, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.478, + "recall@10": 0.745, + "n_ret@10": 10.0, + "all@20": 0.526, + "recall@20": 0.772, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 53.10991299850866, + "default": false, + "all@5": 0.654, + "recall@5": 0.844, + "n_ret@5": 5.0, + "all@10": 0.764, + "recall@10": 0.905, + "n_ret@10": 10.0, + "all@20": 0.954, + "recall@20": 0.9795, + "n_ret@20": 17.79 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 13.946781399892643, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.482, + "recall@10": 0.747, + "n_ret@10": 10.0, + "all@20": 0.534, + "recall@20": 0.776, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 14.204302401281893, + "default": false, + "all@5": 0.78, + "recall@5": 0.9115, + "n_ret@5": 5.0, + "all@10": 0.938, + "recall@10": 0.97, + "n_ret@10": 9.53, + "all@20": 0.942, + "recall@20": 0.971, + "n_ret@20": 16.586 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 56.94439539895393, + "default": false, + "all@5": 0.422, + "recall@5": 0.712, + "n_ret@5": 5.0, + "all@10": 0.478, + "recall@10": 0.745, + "n_ret@10": 10.0, + "all@20": 0.938, + "recall@20": 0.9745, + "n_ret@20": 19.706 + } + ], + "lib_selected": { + "5": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "10": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + } + }, + "seconds": 1856.4762604236603 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/2wiki__bge__mention.npz b/benchmarks/multihop/results/raw/2wiki__bge__mention.npz new file mode 100644 index 0000000..ce66c88 Binary files /dev/null and b/benchmarks/multihop/results/raw/2wiki__bge__mention.npz differ diff --git a/benchmarks/multihop/results/raw/2wiki__bge__rerank_scores.npy b/benchmarks/multihop/results/raw/2wiki__bge__rerank_scores.npy new file mode 100644 index 0000000..c3a82cf Binary files /dev/null and b/benchmarks/multihop/results/raw/2wiki__bge__rerank_scores.npy differ diff --git a/benchmarks/multihop/results/raw/2wiki__bge__vector_top50.npy b/benchmarks/multihop/results/raw/2wiki__bge__vector_top50.npy new file mode 100644 index 0000000..a3d9d8d Binary files /dev/null and b/benchmarks/multihop/results/raw/2wiki__bge__vector_top50.npy differ diff --git a/benchmarks/multihop/results/raw/2wiki__minilm__baselines.json b/benchmarks/multihop/results/raw/2wiki__minilm__baselines.json new file mode 100644 index 0000000..5a36221 --- /dev/null +++ b/benchmarks/multihop/results/raw/2wiki__minilm__baselines.json @@ -0,0 +1,8 @@ +{ + "dataset": "2wiki", + "model": "minilm", + "corpus_size": 9239, + "n_questions": 1500, + "lib_similarity_search_same_top20_set": "198/200", + "seconds": 67.21856546401978 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/2wiki__minilm__baselines.npz b/benchmarks/multihop/results/raw/2wiki__minilm__baselines.npz new file mode 100644 index 0000000..e31aba1 Binary files /dev/null and b/benchmarks/multihop/results/raw/2wiki__minilm__baselines.npz differ diff --git a/benchmarks/multihop/results/raw/2wiki__minilm__bm25_top50.npy b/benchmarks/multihop/results/raw/2wiki__minilm__bm25_top50.npy new file mode 100644 index 0000000..a9340b8 Binary files /dev/null and b/benchmarks/multihop/results/raw/2wiki__minilm__bm25_top50.npy differ diff --git a/benchmarks/multihop/results/raw/2wiki__minilm__ctrl_entity.json b/benchmarks/multihop/results/raw/2wiki__minilm__ctrl_entity.json new file mode 100644 index 0000000..62b618e --- /dev/null +++ b/benchmarks/multihop/results/raw/2wiki__minilm__ctrl_entity.json @@ -0,0 +1,3274 @@ +{ + "dataset": "2wiki", + "model": "minilm", + "variant": "ctrl_entity", + "families": [ + "entity_rand" + ], + "diagnostics": { + "n_edges_undirected_union": 30240, + "n_edges_by_family_directed_or_undirected": 30240, + "mean_degree": 6.546163004654184, + "median_degree": 4.0, + "max_degree": 86, + "isolated_frac": 0.21798895984413896, + "gold_pairs_directly_linked": 0.0012965964343598054, + "gold_pairs_linked_within_2": 0.0074554294975688815, + "neighbours_of_gold_that_are_in_same_question_context": 0.0006118890958802483 + }, + "ppr_dev_grid": [ + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.296, + "recall@10": 0.6265, + "all@20": 0.296, + "recall@20": 0.627 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.296, + "recall@10": 0.6265, + "all@20": 0.296, + "recall@20": 0.627 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.364, + "recall@10": 0.6725, + "all@20": 0.414, + "recall@20": 0.705 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.362, + "recall@10": 0.6715, + "all@20": 0.414, + "recall@20": 0.705 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.294, + "recall@5": 0.6185, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.372, + "recall@20": 0.679 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.292, + "recall@5": 0.6175, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.372, + "recall@20": 0.679 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.298, + "recall@5": 0.624, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.41, + "recall@20": 0.7025 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.622, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.41, + "recall@20": 0.7025 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.298, + "recall@5": 0.627, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.626, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.627, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.627, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.27, + "recall@5": 0.6015, + "all@10": 0.38, + "recall@10": 0.6805, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.268, + "recall@5": 0.601, + "all@10": 0.38, + "recall@10": 0.6805, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.284, + "recall@5": 0.6145, + "all@10": 0.386, + "recall@10": 0.6845, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.286, + "recall@5": 0.6145, + "all@10": 0.386, + "recall@10": 0.6845, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6235, + "all@10": 0.382, + "recall@10": 0.6825, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.294, + "recall@5": 0.6225, + "all@10": 0.382, + "recall@10": 0.6825, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.627, + "all@10": 0.378, + "recall@10": 0.681, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.627, + "all@10": 0.378, + "recall@10": 0.681, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.284, + "recall@5": 0.6175, + "all@10": 0.296, + "recall@10": 0.6265, + "all@20": 0.296, + "recall@20": 0.627 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.284, + "recall@5": 0.6175, + "all@10": 0.296, + "recall@10": 0.6265, + "all@20": 0.296, + "recall@20": 0.627 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.294, + "recall@5": 0.6245, + "all@10": 0.342, + "recall@10": 0.6575, + "all@20": 0.412, + "recall@20": 0.704 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.294, + "recall@5": 0.6245, + "all@10": 0.34, + "recall@10": 0.657, + "all@20": 0.412, + "recall@20": 0.7045 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.37, + "recall@10": 0.677, + "all@20": 0.416, + "recall@20": 0.706 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.37, + "recall@10": 0.677, + "all@20": 0.416, + "recall@20": 0.706 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.236, + "recall@5": 0.5775, + "all@10": 0.37, + "recall@10": 0.677, + "all@20": 0.372, + "recall@20": 0.679 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.232, + "recall@5": 0.5735, + "all@10": 0.37, + "recall@10": 0.677, + "all@20": 0.372, + "recall@20": 0.679 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.28, + "recall@5": 0.606, + "all@10": 0.372, + "recall@10": 0.6785, + "all@20": 0.404, + "recall@20": 0.698 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.278, + "recall@5": 0.6065, + "all@10": 0.372, + "recall@10": 0.6785, + "all@20": 0.402, + "recall@20": 0.6965 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.621, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.707 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.6215, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.707 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.292, + "recall@5": 0.6245, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.292, + "recall@5": 0.6245, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.206, + "recall@5": 0.5335, + "all@10": 0.34, + "recall@10": 0.6535, + "all@20": 0.414, + "recall@20": 0.7055 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.194, + "recall@5": 0.526, + "all@10": 0.338, + "recall@10": 0.6515, + "all@20": 0.414, + "recall@20": 0.7055 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.252, + "recall@5": 0.585, + "all@10": 0.374, + "recall@10": 0.6765, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.246, + "recall@5": 0.5825, + "all@10": 0.376, + "recall@10": 0.6775, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.276, + "recall@5": 0.6045, + "all@10": 0.384, + "recall@10": 0.6825, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.276, + "recall@5": 0.605, + "all@10": 0.384, + "recall@10": 0.6825, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.292, + "recall@5": 0.622, + "all@10": 0.382, + "recall@10": 0.683, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.292, + "recall@5": 0.622, + "all@10": 0.38, + "recall@10": 0.682, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.25, + "recall@5": 0.587, + "all@10": 0.296, + "recall@10": 0.6265, + "all@20": 0.296, + "recall@20": 0.627 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.25, + "recall@5": 0.586, + "all@10": 0.296, + "recall@10": 0.6265, + "all@20": 0.296, + "recall@20": 0.627 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.284, + "recall@5": 0.6155, + "all@10": 0.322, + "recall@10": 0.645, + "all@20": 0.4, + "recall@20": 0.695 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.284, + "recall@5": 0.616, + "all@10": 0.32, + "recall@10": 0.644, + "all@20": 0.4, + "recall@20": 0.695 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.372, + "recall@10": 0.678, + "all@20": 0.416, + "recall@20": 0.706 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.372, + "recall@10": 0.678, + "all@20": 0.416, + "recall@20": 0.706 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.707 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.707 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.164, + "recall@5": 0.499, + "all@10": 0.34, + "recall@10": 0.654, + "all@20": 0.372, + "recall@20": 0.679 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.166, + "recall@5": 0.4975, + "all@10": 0.34, + "recall@10": 0.653, + "all@20": 0.372, + "recall@20": 0.679 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.222, + "recall@5": 0.563, + "all@10": 0.364, + "recall@10": 0.6735, + "all@20": 0.396, + "recall@20": 0.692 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.216, + "recall@5": 0.56, + "all@10": 0.366, + "recall@10": 0.6745, + "all@20": 0.396, + "recall@20": 0.692 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.29, + "recall@5": 0.6155, + "all@10": 0.372, + "recall@10": 0.6785, + "all@20": 0.418, + "recall@20": 0.707 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.286, + "recall@5": 0.6135, + "all@10": 0.372, + "recall@10": 0.678, + "all@20": 0.418, + "recall@20": 0.707 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.294, + "recall@5": 0.6245, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.707 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.292, + "recall@5": 0.6235, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.707 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.076, + "recall@5": 0.3715, + "all@10": 0.272, + "recall@10": 0.5975, + "all@20": 0.404, + "recall@20": 0.698 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.076, + "recall@5": 0.372, + "all@10": 0.274, + "recall@10": 0.5965, + "all@20": 0.402, + "recall@20": 0.6965 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.174, + "recall@5": 0.5065, + "all@10": 0.342, + "recall@10": 0.6555, + "all@20": 0.416, + "recall@20": 0.7065 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.172, + "recall@5": 0.5045, + "all@10": 0.342, + "recall@10": 0.655, + "all@20": 0.416, + "recall@20": 0.7065 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.268, + "recall@5": 0.6005, + "all@10": 0.384, + "recall@10": 0.6825, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.262, + "recall@5": 0.595, + "all@10": 0.382, + "recall@10": 0.681, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.282, + "recall@5": 0.6155, + "all@10": 0.38, + "recall@10": 0.6815, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.28, + "recall@5": 0.6145, + "all@10": 0.38, + "recall@10": 0.6815, + "all@20": 0.42, + "recall@20": 0.7085 + } + ], + "ppr_selected": { + "5": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "20": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + } + }, + "lib_dev_grid": [ + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 9.70037860085722, + "default": true, + "all@5": 0.24, + "recall@5": 0.589, + "n_ret@5": 5.0, + "all@10": 0.256, + "recall@10": 0.6015, + "n_ret@10": 9.99, + "all@20": 0.282, + "recall@20": 0.618, + "n_ret@20": 19.884 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.479574400815181, + "default": false, + "all@5": 0.168, + "recall@5": 0.54, + "n_ret@5": 5.0, + "all@10": 0.236, + "recall@10": 0.5855, + "n_ret@10": 10.0, + "all@20": 0.32, + "recall@20": 0.6455, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 26.559706200729124, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 31.00983419874683, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 57.11673119862098, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 29.754085200605914, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.68, + "n_ret@20": 19.996 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 43.44560460024513, + "default": false, + "all@5": 0.294, + "recall@5": 0.626, + "n_ret@5": 5.0, + "all@10": 0.37, + "recall@10": 0.6775, + "n_ret@10": 10.0, + "all@20": 0.414, + "recall@20": 0.7065, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 27.61531440075487, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.7045, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 46.17716519976966, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 7.164511799812317, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.679, + "n_ret@20": 19.996 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 25.37962519878056, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.704, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 40.76534119900316, + "default": false, + "all@5": 0.272, + "recall@5": 0.6115, + "n_ret@5": 5.0, + "all@10": 0.342, + "recall@10": 0.6605, + "n_ret@10": 10.0, + "all@20": 0.394, + "recall@20": 0.6945, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 27.33320020127576, + "default": false, + "all@5": 0.292, + "recall@5": 0.624, + "n_ret@5": 5.0, + "all@10": 0.368, + "recall@10": 0.677, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.708, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 23.02837659965735, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.68, + "n_ret@20": 19.996 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.124299202114344, + "default": false, + "all@5": 0.168, + "recall@5": 0.54, + "n_ret@5": 5.0, + "all@10": 0.224, + "recall@10": 0.577, + "n_ret@10": 9.97, + "all@20": 0.272, + "recall@20": 0.611, + "n_ret@20": 19.192 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 31.413454800611362, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.38, + "recall@20": 0.6865, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.963516399613582, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.703, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 21.727521598455496, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 4.502768401172943, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6265, + "n_ret@10": 9.97, + "all@20": 0.296, + "recall@20": 0.627, + "n_ret@20": 19.192 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 23.26533480035141, + "default": false, + "all@5": 0.292, + "recall@5": 0.6245, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6275, + "n_ret@10": 9.99, + "all@20": 0.296, + "recall@20": 0.628, + "n_ret@20": 19.884 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 20.023524000891484, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.6805, + "n_ret@20": 19.996 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 35.50505080027506, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.416, + "recall@20": 0.707, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 17.891970800585113, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6275, + "n_ret@10": 9.97, + "all@20": 0.296, + "recall@20": 0.6275, + "n_ret@20": 19.192 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 19.237613399862312, + "default": false, + "all@5": 0.294, + "recall@5": 0.626, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.704, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 40.98543140059337, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 18.610738999326713, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6275, + "n_ret@10": 9.97, + "all@20": 0.296, + "recall@20": 0.6275, + "n_ret@20": 19.192 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 5.577082399162464, + "default": false, + "all@5": 0.292, + "recall@5": 0.624, + "n_ret@5": 5.0, + "all@10": 0.368, + "recall@10": 0.677, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.703, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 8.326953800278716, + "default": false, + "all@5": 0.292, + "recall@5": 0.624, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6265, + "n_ret@10": 9.99, + "all@20": 0.296, + "recall@20": 0.627, + "n_ret@20": 19.884 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 23.752253597253002, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6275, + "n_ret@10": 9.99, + "all@20": 0.296, + "recall@20": 0.628, + "n_ret@20": 19.884 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 19.149972800863907, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.68, + "n_ret@20": 19.996 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 19.423407399561256, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.7045, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 5.868508600164205, + "default": false, + "all@5": 0.292, + "recall@5": 0.624, + "n_ret@5": 5.0, + "all@10": 0.368, + "recall@10": 0.677, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.708, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 47.610690399073064, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 38.82264720322564, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.384, + "recall@20": 0.6875, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 32.87231500004418, + "default": false, + "all@5": 0.272, + "recall@5": 0.6115, + "n_ret@5": 5.0, + "all@10": 0.342, + "recall@10": 0.6605, + "n_ret@10": 10.0, + "all@20": 0.362, + "recall@20": 0.6755, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 18.098257000325248, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.703, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 8.465132197830826, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6265, + "n_ret@10": 9.99, + "all@20": 0.296, + "recall@20": 0.627, + "n_ret@20": 19.884 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 18.637053599581122, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 41.03778120095376, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 22.39806059934199, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.303091401234269, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.6795, + "n_ret@20": 19.996 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 19.69146920007188, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 21.596769000287168, + "default": false, + "all@5": 0.16, + "recall@5": 0.5335, + "n_ret@5": 5.0, + "all@10": 0.212, + "recall@10": 0.5685, + "n_ret@10": 10.0, + "all@20": 0.264, + "recall@20": 0.6065, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 11.943352200090885, + "default": false, + "all@5": 0.16, + "recall@5": 0.5335, + "n_ret@5": 5.0, + "all@10": 0.202, + "recall@10": 0.5615, + "n_ret@10": 9.99, + "all@20": 0.236, + "recall@20": 0.585, + "n_ret@20": 19.884 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 47.87004400033038, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.68, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 18.589885401306674, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.68, + "n_ret@20": 19.996 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 20.79188200237695, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6295, + "n_ret@10": 9.99, + "all@20": 0.296, + "recall@20": 0.6305, + "n_ret@20": 19.884 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 33.840438200277276, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6295, + "n_ret@10": 9.99, + "all@20": 0.298, + "recall@20": 0.6315, + "n_ret@20": 19.884 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 44.79827460111119, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.384, + "recall@20": 0.6875, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 32.21831140154973, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.68, + "n_ret@20": 19.996 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 54.7792096020421, + "default": false, + "all@5": 0.272, + "recall@5": 0.6115, + "n_ret@5": 5.0, + "all@10": 0.342, + "recall@10": 0.6605, + "n_ret@10": 10.0, + "all@20": 0.394, + "recall@20": 0.6945, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 33.23783159896266, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.7045, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 43.95230879809242, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.378, + "recall@20": 0.685, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 23.153030199930072, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.679, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 49.874222999671474, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.414, + "recall@20": 0.7065, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 29.36266620073002, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.7045, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 31.353462401311845, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 34.57325460144784, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6275, + "n_ret@10": 9.99, + "all@20": 0.296, + "recall@20": 0.6275, + "n_ret@20": 19.884 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 21.723160200286657, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.703, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 28.613109200145118, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.7045, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 33.66422779893037, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.298, + "recall@10": 0.6305, + "n_ret@10": 9.99, + "all@20": 0.298, + "recall@20": 0.632, + "n_ret@20": 19.884 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 51.44571039965376, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 38.78572180075571, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.384, + "recall@20": 0.6875, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.784952001296915, + "default": false, + "all@5": 0.282, + "recall@5": 0.6185, + "n_ret@5": 5.0, + "all@10": 0.354, + "recall@10": 0.667, + "n_ret@10": 10.0, + "all@20": 0.402, + "recall@20": 0.6985, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 39.26118819869589, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.68, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 9.483567598857917, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.703, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 30.520762398606166, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 29.04175299941562, + "default": false, + "all@5": 0.294, + "recall@5": 0.626, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.68, + "n_ret@20": 19.996 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 27.580910001997836, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6275, + "n_ret@10": 9.97, + "all@20": 0.296, + "recall@20": 0.6275, + "n_ret@20": 19.192 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 28.94592759932857, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6275, + "n_ret@10": 9.97, + "all@20": 0.296, + "recall@20": 0.6275, + "n_ret@20": 19.192 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 8.40568999981042, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.703, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 30.170002598897554, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 45.181269800290465, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.704, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 46.23324760165997, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.414, + "recall@20": 0.7055, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 20.109952398692258, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.679, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.887552000116557, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6265, + "n_ret@10": 9.97, + "all@20": 0.296, + "recall@20": 0.6275, + "n_ret@20": 19.192 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 32.34675359877292, + "default": false, + "all@5": 0.278, + "recall@5": 0.615, + "n_ret@5": 5.0, + "all@10": 0.36, + "recall@10": 0.6705, + "n_ret@10": 10.0, + "all@20": 0.404, + "recall@20": 0.7005, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 49.9435270014219, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 32.800697601167485, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 30.832269600010477, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6275, + "n_ret@10": 9.97, + "all@20": 0.296, + "recall@20": 0.6275, + "n_ret@20": 19.192 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 27.360855600913055, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.68, + "n_ret@20": 19.996 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 48.37920380150899, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 27.83831959834788, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.704, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 12.546833400148898, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6265, + "n_ret@10": 9.99, + "all@20": 0.296, + "recall@20": 0.627, + "n_ret@20": 19.884 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 16.223251199815422, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.6795, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 32.275051600066945, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.629, + "n_ret@10": 9.99, + "all@20": 0.298, + "recall@20": 0.632, + "n_ret@20": 19.884 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 31.473540598643012, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.7035, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 30.400454599177465, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 47.93325899995398, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 27.749127999180928, + "default": false, + "all@5": 0.278, + "recall@5": 0.615, + "n_ret@5": 5.0, + "all@10": 0.29, + "recall@10": 0.624, + "n_ret@10": 9.97, + "all@20": 0.294, + "recall@20": 0.627, + "n_ret@20": 19.192 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 50.94687499967404, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 28.24520939856302, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.7045, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 27.64285400044173, + "default": false, + "all@5": 0.294, + "recall@5": 0.626, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.418, + "recall@20": 0.7075, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 30.377794599626213, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6275, + "n_ret@10": 9.99, + "all@20": 0.296, + "recall@20": 0.628, + "n_ret@20": 19.884 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 11.966699600685388, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6265, + "n_ret@10": 9.99, + "all@20": 0.296, + "recall@20": 0.627, + "n_ret@20": 19.884 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 25.28726040176116, + "default": false, + "all@5": 0.278, + "recall@5": 0.615, + "n_ret@5": 5.0, + "all@10": 0.36, + "recall@10": 0.6705, + "n_ret@10": 10.0, + "all@20": 0.368, + "recall@20": 0.678, + "n_ret@20": 19.996 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 7.480565001489595, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.679, + "n_ret@20": 19.996 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.045814999379218, + "default": false, + "all@5": 0.168, + "recall@5": 0.54, + "n_ret@5": 5.0, + "all@10": 0.236, + "recall@10": 0.5855, + "n_ret@10": 10.0, + "all@20": 0.32, + "recall@20": 0.6455, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 14.98842639895156, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.6795, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 24.160753198782913, + "default": false, + "all@5": 0.282, + "recall@5": 0.6185, + "n_ret@5": 5.0, + "all@10": 0.354, + "recall@10": 0.667, + "n_ret@10": 10.0, + "all@20": 0.406, + "recall@20": 0.7005, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 48.311981601291336, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + } + ], + "lib_selected": { + "5": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + } + }, + "seconds": 4126.133343219757 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/2wiki__minilm__ctrl_entity.npz b/benchmarks/multihop/results/raw/2wiki__minilm__ctrl_entity.npz new file mode 100644 index 0000000..170fc2b Binary files /dev/null and b/benchmarks/multihop/results/raw/2wiki__minilm__ctrl_entity.npz differ diff --git a/benchmarks/multihop/results/raw/2wiki__minilm__ctrl_mention.json b/benchmarks/multihop/results/raw/2wiki__minilm__ctrl_mention.json new file mode 100644 index 0000000..5e4f01d --- /dev/null +++ b/benchmarks/multihop/results/raw/2wiki__minilm__ctrl_mention.json @@ -0,0 +1,3274 @@ +{ + "dataset": "2wiki", + "model": "minilm", + "variant": "ctrl_mention", + "families": [ + "mention_rand" + ], + "diagnostics": { + "n_edges_undirected_union": 5383, + "n_edges_by_family_directed_or_undirected": 6115, + "mean_degree": 1.165277627448858, + "median_degree": 1.0, + "max_degree": 30, + "isolated_frac": 0.4217988959844139, + "gold_pairs_directly_linked": 0.00032414910858995135, + "gold_pairs_linked_within_2": 0.00032414910858995135, + "neighbours_of_gold_that_are_in_same_question_context": 0.0003952569169960474 + }, + "ppr_dev_grid": [ + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.294, + "recall@5": 0.6255, + "all@10": 0.296, + "recall@10": 0.6265, + "all@20": 0.296, + "recall@20": 0.6265 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.292, + "recall@5": 0.6245, + "all@10": 0.296, + "recall@10": 0.627, + "all@20": 0.296, + "recall@20": 0.627 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.35, + "recall@10": 0.6645, + "all@20": 0.416, + "recall@20": 0.705 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.342, + "recall@10": 0.6585, + "all@20": 0.412, + "recall@20": 0.704 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.298, + "recall@5": 0.626, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.372, + "recall@20": 0.679 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.258, + "recall@5": 0.591, + "all@10": 0.37, + "recall@10": 0.678, + "all@20": 0.372, + "recall@20": 0.679 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.294, + "recall@5": 0.624, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.406, + "recall@20": 0.701 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.266, + "recall@5": 0.6005, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.406, + "recall@20": 0.7 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.292, + "recall@5": 0.623, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.278, + "recall@5": 0.613, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.707 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.294, + "recall@5": 0.626, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.292, + "recall@5": 0.626, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.286, + "recall@5": 0.6135, + "all@10": 0.372, + "recall@10": 0.6785, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.226, + "recall@5": 0.5535, + "all@10": 0.318, + "recall@10": 0.6445, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.288, + "recall@5": 0.6175, + "all@10": 0.372, + "recall@10": 0.678, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.252, + "recall@5": 0.5895, + "all@10": 0.336, + "recall@10": 0.655, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.29, + "recall@5": 0.6215, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.276, + "recall@5": 0.614, + "all@10": 0.354, + "recall@10": 0.668, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.294, + "recall@5": 0.626, + "all@10": 0.376, + "recall@10": 0.681, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.292, + "recall@5": 0.626, + "all@10": 0.37, + "recall@10": 0.6785, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.276, + "recall@5": 0.6145, + "all@10": 0.296, + "recall@10": 0.6265, + "all@20": 0.296, + "recall@20": 0.6265 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.248, + "recall@5": 0.5695, + "all@10": 0.296, + "recall@10": 0.627, + "all@20": 0.296, + "recall@20": 0.627 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.284, + "recall@5": 0.6205, + "all@10": 0.344, + "recall@10": 0.6595, + "all@20": 0.41, + "recall@20": 0.7025 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.286, + "recall@5": 0.619, + "all@10": 0.324, + "recall@10": 0.645, + "all@20": 0.404, + "recall@20": 0.699 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.368, + "recall@10": 0.676, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.294, + "recall@5": 0.6255, + "all@10": 0.368, + "recall@10": 0.6755, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.262, + "recall@5": 0.5975, + "all@10": 0.37, + "recall@10": 0.676, + "all@20": 0.372, + "recall@20": 0.679 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.148, + "recall@5": 0.4495, + "all@10": 0.312, + "recall@10": 0.6335, + "all@20": 0.372, + "recall@20": 0.679 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.282, + "recall@5": 0.6175, + "all@10": 0.37, + "recall@10": 0.678, + "all@20": 0.4, + "recall@20": 0.6955 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.228, + "recall@5": 0.5595, + "all@10": 0.358, + "recall@10": 0.6685, + "all@20": 0.386, + "recall@20": 0.6875 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.292, + "recall@5": 0.623, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.254, + "recall@5": 0.5945, + "all@10": 0.372, + "recall@10": 0.6785, + "all@20": 0.416, + "recall@20": 0.7055 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.29, + "recall@5": 0.624, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.282, + "recall@5": 0.619, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.21, + "recall@5": 0.5455, + "all@10": 0.348, + "recall@10": 0.662, + "all@20": 0.414, + "recall@20": 0.7055 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.082, + "recall@5": 0.338, + "all@10": 0.21, + "recall@10": 0.524, + "all@20": 0.36, + "recall@20": 0.6695 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.268, + "recall@5": 0.6025, + "all@10": 0.364, + "recall@10": 0.673, + "all@20": 0.418, + "recall@20": 0.707 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.162, + "recall@5": 0.4645, + "all@10": 0.3, + "recall@10": 0.6245, + "all@20": 0.412, + "recall@20": 0.704 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.282, + "recall@5": 0.613, + "all@10": 0.372, + "recall@10": 0.6785, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.236, + "recall@5": 0.57, + "all@10": 0.33, + "recall@10": 0.6515, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.29, + "recall@5": 0.6235, + "all@10": 0.374, + "recall@10": 0.68, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.284, + "recall@5": 0.6205, + "all@10": 0.364, + "recall@10": 0.6735, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.236, + "recall@5": 0.5835, + "all@10": 0.296, + "recall@10": 0.6245, + "all@20": 0.296, + "recall@20": 0.6265 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.132, + "recall@5": 0.4275, + "all@10": 0.294, + "recall@10": 0.6235, + "all@20": 0.296, + "recall@20": 0.627 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.274, + "recall@5": 0.6135, + "all@10": 0.336, + "recall@10": 0.655, + "all@20": 0.408, + "recall@20": 0.7015 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.222, + "recall@5": 0.5495, + "all@10": 0.316, + "recall@10": 0.639, + "all@20": 0.404, + "recall@20": 0.698 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.292, + "recall@5": 0.624, + "all@10": 0.364, + "recall@10": 0.6735, + "all@20": 0.418, + "recall@20": 0.707 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.29, + "recall@5": 0.622, + "all@10": 0.362, + "recall@10": 0.6725, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.186, + "recall@5": 0.5205, + "all@10": 0.358, + "recall@10": 0.6645, + "all@20": 0.37, + "recall@20": 0.678 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.042, + "recall@5": 0.2425, + "all@10": 0.176, + "recall@10": 0.478, + "all@20": 0.37, + "recall@20": 0.6775 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.238, + "recall@5": 0.581, + "all@10": 0.362, + "recall@10": 0.6725, + "all@20": 0.4, + "recall@20": 0.6955 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.118, + "recall@5": 0.401, + "all@10": 0.294, + "recall@10": 0.619, + "all@20": 0.382, + "recall@20": 0.685 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.29, + "recall@5": 0.6215, + "all@10": 0.37, + "recall@10": 0.678, + "all@20": 0.418, + "recall@20": 0.707 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.242, + "recall@5": 0.571, + "all@10": 0.366, + "recall@10": 0.6735, + "all@20": 0.414, + "recall@20": 0.7045 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.292, + "recall@5": 0.624, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.276, + "recall@5": 0.6125, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.418, + "recall@20": 0.7075 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.102, + "recall@5": 0.3815, + "all@10": 0.294, + "recall@10": 0.6115, + "all@20": 0.4, + "recall@20": 0.6985 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.02, + "recall@5": 0.1655, + "all@10": 0.048, + "recall@10": 0.258, + "all@20": 0.214, + "recall@20": 0.526 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.176, + "recall@5": 0.504, + "all@10": 0.344, + "recall@10": 0.6575, + "all@20": 0.414, + "recall@20": 0.7055 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.056, + "recall@5": 0.279, + "all@10": 0.174, + "recall@10": 0.469, + "all@20": 0.36, + "recall@20": 0.6705 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.274, + "recall@5": 0.606, + "all@10": 0.368, + "recall@10": 0.675, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.182, + "recall@5": 0.4955, + "all@10": 0.31, + "recall@10": 0.6345, + "all@20": 0.42, + "recall@20": 0.708 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.29, + "recall@5": 0.6225, + "all@10": 0.374, + "recall@10": 0.6795, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.27, + "recall@5": 0.6095, + "all@10": 0.354, + "recall@10": 0.667, + "all@20": 0.42, + "recall@20": 0.7085 + } + ], + "ppr_selected": { + "5": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "20": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + } + }, + "lib_dev_grid": [ + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.335922198952176, + "default": true, + "all@5": 0.278, + "recall@5": 0.615, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6265, + "n_ret@10": 8.45, + "all@20": 0.296, + "recall@20": 0.6265, + "n_ret@20": 9.626 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.758550799335353, + "default": false, + "all@5": 0.292, + "recall@5": 0.625, + "n_ret@5": 5.0, + "all@10": 0.368, + "recall@10": 0.6765, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.679, + "n_ret@20": 17.798 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 23.643307000631467, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.7035, + "n_ret@20": 19.936 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 24.8698132004356, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 24.764849799801596, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6265, + "n_ret@10": 8.45, + "all@20": 0.296, + "recall@20": 0.6265, + "n_ret@20": 9.626 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 24.478339000954293, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 5.994853000971489, + "default": false, + "all@5": 0.222, + "recall@5": 0.578, + "n_ret@5": 5.0, + "all@10": 0.276, + "recall@10": 0.6135, + "n_ret@10": 9.692, + "all@20": 0.292, + "recall@20": 0.6235, + "n_ret@20": 15.674 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 27.873272403143346, + "default": false, + "all@5": 0.288, + "recall@5": 0.623, + "n_ret@5": 5.0, + "all@10": 0.36, + "recall@10": 0.672, + "n_ret@10": 10.0, + "all@20": 0.412, + "recall@20": 0.7045, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.778978200163692, + "default": false, + "all@5": 0.268, + "recall@5": 0.6065, + "n_ret@5": 5.0, + "all@10": 0.35, + "recall@10": 0.666, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.703, + "n_ret@20": 19.882 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.970341798500158, + "default": false, + "all@5": 0.292, + "recall@5": 0.625, + "n_ret@5": 5.0, + "all@10": 0.368, + "recall@10": 0.6765, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.679, + "n_ret@20": 16.35 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 24.062203598907217, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 5.762982998741791, + "default": false, + "all@5": 0.29, + "recall@5": 0.624, + "n_ret@5": 5.0, + "all@10": 0.294, + "recall@10": 0.6265, + "n_ret@10": 9.692, + "all@20": 0.296, + "recall@20": 0.627, + "n_ret@20": 15.674 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.467351796571165, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.703, + "n_ret@20": 19.936 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 23.23925079999026, + "default": false, + "all@5": 0.294, + "recall@5": 0.6255, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.627, + "n_ret@10": 9.05, + "all@20": 0.296, + "recall@20": 0.627, + "n_ret@20": 10.792 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 23.808312199776992, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.679, + "n_ret@20": 18.11 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 22.781761999009177, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.703, + "n_ret@20": 19.882 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 5.382498598890379, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.627, + "n_ret@10": 9.05, + "all@20": 0.296, + "recall@20": 0.627, + "n_ret@20": 10.792 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 23.83904699992854, + "default": false, + "all@5": 0.288, + "recall@5": 0.623, + "n_ret@5": 5.0, + "all@10": 0.362, + "recall@10": 0.673, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.6795, + "n_ret@20": 19.882 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 7.104690998443402, + "default": false, + "all@5": 0.222, + "recall@5": 0.578, + "n_ret@5": 5.0, + "all@10": 0.308, + "recall@10": 0.636, + "n_ret@10": 10.0, + "all@20": 0.378, + "recall@20": 0.6835, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 5.699404001003131, + "default": false, + "all@5": 0.24, + "recall@5": 0.589, + "n_ret@5": 5.0, + "all@10": 0.326, + "recall@10": 0.651, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.6795, + "n_ret@20": 18.902 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 23.769105198676698, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.679, + "n_ret@20": 17.798 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 22.614524198812433, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.679, + "n_ret@20": 18.11 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 6.146002399385907, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.6795, + "n_ret@20": 19.924 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 23.64479379868135, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.7035, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 25.624509200337343, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.6795, + "n_ret@20": 19.882 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 26.088509200373664, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 26.2458028011024, + "default": false, + "all@5": 0.294, + "recall@5": 0.6255, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.627, + "n_ret@10": 9.692, + "all@20": 0.296, + "recall@20": 0.627, + "n_ret@20": 15.674 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 25.11741019843612, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.7035, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 24.986486600944772, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.6795, + "n_ret@20": 19.924 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.764379998552613, + "default": false, + "all@5": 0.264, + "recall@5": 0.6035, + "n_ret@5": 5.0, + "all@10": 0.346, + "recall@10": 0.6625, + "n_ret@10": 10.0, + "all@20": 0.408, + "recall@20": 0.701, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 22.123096399591304, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 22.43221999972593, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.7035, + "n_ret@20": 19.936 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.7834043994080275, + "default": false, + "all@5": 0.292, + "recall@5": 0.625, + "n_ret@5": 5.0, + "all@10": 0.368, + "recall@10": 0.6765, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.703, + "n_ret@20": 19.882 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 24.2078130019363, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 23.163488199934363, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.298, + "recall@10": 0.628, + "n_ret@10": 9.73, + "all@20": 0.298, + "recall@20": 0.6285, + "n_ret@20": 17.072 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 22.7661717988085, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.703, + "n_ret@20": 19.936 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.211222200188786, + "default": false, + "all@5": 0.292, + "recall@5": 0.625, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6265, + "n_ret@10": 8.522, + "all@20": 0.296, + "recall@20": 0.6265, + "n_ret@20": 10.218 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.01394579873886, + "default": false, + "all@5": 0.264, + "recall@5": 0.6035, + "n_ret@5": 5.0, + "all@10": 0.346, + "recall@10": 0.6625, + "n_ret@10": 10.0, + "all@20": 0.406, + "recall@20": 0.7, + "n_ret@20": 19.936 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 4.7927453990560025, + "default": false, + "all@5": 0.24, + "recall@5": 0.589, + "n_ret@5": 5.0, + "all@10": 0.326, + "recall@10": 0.651, + "n_ret@10": 10.0, + "all@20": 0.402, + "recall@20": 0.6995, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 20.820234998245724, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.215205200249329, + "default": false, + "all@5": 0.264, + "recall@5": 0.6035, + "n_ret@5": 5.0, + "all@10": 0.294, + "recall@10": 0.625, + "n_ret@10": 8.45, + "all@20": 0.296, + "recall@20": 0.6265, + "n_ret@20": 9.626 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 6.040445597143844, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.131131000933237, + "default": false, + "all@5": 0.292, + "recall@5": 0.625, + "n_ret@5": 5.0, + "all@10": 0.368, + "recall@10": 0.6765, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.703, + "n_ret@20": 19.936 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 20.21546980109997, + "default": false, + "all@5": 0.294, + "recall@5": 0.6255, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 21.658493399852887, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.679, + "n_ret@20": 16.35 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 18.543820799910463, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6265, + "n_ret@10": 8.45, + "all@20": 0.296, + "recall@20": 0.6265, + "n_ret@20": 9.626 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 18.926354600000195, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 5.497896199580282, + "default": false, + "all@5": 0.222, + "recall@5": 0.578, + "n_ret@5": 5.0, + "all@10": 0.308, + "recall@10": 0.636, + "n_ret@10": 10.0, + "all@20": 0.378, + "recall@20": 0.683, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 17.434661800274625, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.627, + "n_ret@10": 9.692, + "all@20": 0.296, + "recall@20": 0.627, + "n_ret@20": 15.674 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 18.830576800275594, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.6795, + "n_ret@20": 19.882 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 17.380418600980192, + "default": false, + "all@5": 0.29, + "recall@5": 0.624, + "n_ret@5": 5.0, + "all@10": 0.366, + "recall@10": 0.6755, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.6795, + "n_ret@20": 18.902 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 4.832833800232038, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 16.812359800678678, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.7035, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 16.706839199294336, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.679, + "n_ret@20": 18.11 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 4.827093001222238, + "default": false, + "all@5": 0.29, + "recall@5": 0.624, + "n_ret@5": 5.0, + "all@10": 0.362, + "recall@10": 0.674, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.6795, + "n_ret@20": 19.882 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 4.718253199593164, + "default": false, + "all@5": 0.24, + "recall@5": 0.589, + "n_ret@5": 5.0, + "all@10": 0.326, + "recall@10": 0.651, + "n_ret@10": 10.0, + "all@20": 0.402, + "recall@20": 0.7, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 20.413024599547498, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 18.621380400727503, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 18.143923199968413, + "default": false, + "all@5": 0.288, + "recall@5": 0.623, + "n_ret@5": 5.0, + "all@10": 0.294, + "recall@10": 0.626, + "n_ret@10": 9.692, + "all@20": 0.296, + "recall@20": 0.627, + "n_ret@20": 15.674 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 18.251084399526007, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.7035, + "n_ret@20": 19.938 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 19.101046200376004, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.704, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.3, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 5.733968198765069, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 17.960164197953418, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.627, + "n_ret@10": 9.692, + "all@20": 0.296, + "recall@20": 0.627, + "n_ret@20": 15.674 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 17.715890200342983, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.6795, + "n_ret@20": 18.902 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 19.86672620009631, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.7035, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 21.76279180054553, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.6795, + "n_ret@20": 19.882 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 25.84097239980474, + "default": false, + "all@5": 0.292, + "recall@5": 0.625, + "n_ret@5": 5.0, + "all@10": 0.368, + "recall@10": 0.676, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.679, + "n_ret@20": 17.798 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 23.897996600717306, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.7035, + "n_ret@20": 19.938 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.9548574016662315, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.703, + "n_ret@20": 19.936 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 20.80993519886397, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.374, + "recall@20": 0.681, + "n_ret@20": 19.924 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 4.338810599176213, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.7035, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 21.204689602251165, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 17.51509439991787, + "default": false, + "all@5": 0.29, + "recall@5": 0.624, + "n_ret@5": 5.0, + "all@10": 0.366, + "recall@10": 0.6755, + "n_ret@10": 10.0, + "all@20": 0.416, + "recall@20": 0.7065, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 18.088098399690352, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.6795, + "n_ret@20": 19.924 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.495527799590491, + "default": false, + "all@5": 0.268, + "recall@5": 0.6065, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6265, + "n_ret@10": 7.942, + "all@20": 0.296, + "recall@20": 0.6265, + "n_ret@20": 8.274 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 17.891399799031205, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6265, + "n_ret@10": 8.45, + "all@20": 0.296, + "recall@20": 0.6265, + "n_ret@20": 9.626 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 18.961574599612504, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.7035, + "n_ret@20": 19.936 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 19.64618099841755, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.704, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 18.79869640036486, + "default": false, + "all@5": 0.292, + "recall@5": 0.625, + "n_ret@5": 5.0, + "all@10": 0.368, + "recall@10": 0.676, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.708, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 23.69698039884679, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.627, + "n_ret@10": 9.692, + "all@20": 0.296, + "recall@20": 0.627, + "n_ret@20": 15.674 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.982974799466319, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6265, + "n_ret@10": 7.942, + "all@20": 0.296, + "recall@20": 0.6265, + "n_ret@20": 8.274 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 36.57430900086183, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.861648200196214, + "default": false, + "all@5": 0.292, + "recall@5": 0.625, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6265, + "n_ret@10": 8.45, + "all@20": 0.296, + "recall@20": 0.6265, + "n_ret@20": 9.626 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 17.323072400758974, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.627, + "n_ret@10": 9.05, + "all@20": 0.296, + "recall@20": 0.627, + "n_ret@20": 10.792 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 16.79155580000952, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.7035, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 24.564231600263156, + "default": false, + "all@5": 0.294, + "recall@5": 0.6255, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 8.97447779995855, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.6795, + "n_ret@20": 19.882 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 32.49368559930008, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.03927080030553, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.41, + "recall@20": 0.703, + "n_ret@20": 19.938 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 8.367777800187469, + "default": false, + "all@5": 0.222, + "recall@5": 0.578, + "n_ret@5": 5.0, + "all@10": 0.31, + "recall@10": 0.638, + "n_ret@10": 10.0, + "all@20": 0.388, + "recall@20": 0.689, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 30.95316160155926, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.679, + "n_ret@20": 18.11 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 32.5455913990736, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.679, + "n_ret@20": 16.35 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 9.254265800467692, + "default": false, + "all@5": 0.222, + "recall@5": 0.578, + "n_ret@5": 5.0, + "all@10": 0.31, + "recall@10": 0.638, + "n_ret@10": 10.0, + "all@20": 0.388, + "recall@20": 0.6895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 32.72846039722208, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.627, + "n_ret@10": 9.692, + "all@20": 0.296, + "recall@20": 0.627, + "n_ret@20": 15.674 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 31.844935002387505, + "default": false, + "all@5": 0.292, + "recall@5": 0.625, + "n_ret@5": 5.0, + "all@10": 0.368, + "recall@10": 0.676, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.708, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.930666398140602, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6265, + "n_ret@10": 8.522, + "all@20": 0.296, + "recall@20": 0.6265, + "n_ret@20": 10.218 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 33.037449400522746, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 32.33682639990002, + "default": false, + "all@5": 0.292, + "recall@5": 0.625, + "n_ret@5": 5.0, + "all@10": 0.368, + "recall@10": 0.676, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.679, + "n_ret@20": 18.11 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.259654600988142, + "default": false, + "all@5": 0.292, + "recall@5": 0.625, + "n_ret@5": 5.0, + "all@10": 0.368, + "recall@10": 0.6765, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.708, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 7.878328999970109, + "default": false, + "all@5": 0.222, + "recall@5": 0.578, + "n_ret@5": 5.0, + "all@10": 0.274, + "recall@10": 0.612, + "n_ret@10": 9.73, + "all@20": 0.288, + "recall@20": 0.621, + "n_ret@20": 17.072 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 30.508648598915897, + "default": false, + "all@5": 0.294, + "recall@5": 0.6255, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.6795, + "n_ret@20": 19.924 + } + ], + "lib_selected": { + "5": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + } + }, + "seconds": 1597.260213136673 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/2wiki__minilm__ctrl_mention.npz b/benchmarks/multihop/results/raw/2wiki__minilm__ctrl_mention.npz new file mode 100644 index 0000000..ed8958d Binary files /dev/null and b/benchmarks/multihop/results/raw/2wiki__minilm__ctrl_mention.npz differ diff --git a/benchmarks/multihop/results/raw/2wiki__minilm__entity.json b/benchmarks/multihop/results/raw/2wiki__minilm__entity.json new file mode 100644 index 0000000..f03c150 --- /dev/null +++ b/benchmarks/multihop/results/raw/2wiki__minilm__entity.json @@ -0,0 +1,3274 @@ +{ + "dataset": "2wiki", + "model": "minilm", + "variant": "entity", + "families": [ + "entity" + ], + "diagnostics": { + "n_edges_undirected_union": 30240, + "n_edges_by_family_directed_or_undirected": 30240, + "mean_degree": 6.546163004654184, + "median_degree": 4.0, + "max_degree": 86, + "isolated_frac": 0.21798895984413896, + "gold_pairs_directly_linked": 0.4051863857374392, + "gold_pairs_linked_within_2": 0.426580226904376, + "neighbours_of_gold_that_are_in_same_question_context": 0.11197227702934706 + }, + "ppr_dev_grid": [ + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.304, + "recall@5": 0.6305, + "all@10": 0.558, + "recall@10": 0.781, + "all@20": 0.668, + "recall@20": 0.833 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.304, + "recall@5": 0.6305, + "all@10": 0.576, + "recall@10": 0.7885, + "all@20": 0.686, + "recall@20": 0.841 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.568, + "recall@10": 0.791, + "all@20": 0.664, + "recall@20": 0.847 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.58, + "recall@10": 0.796, + "all@20": 0.664, + "recall@20": 0.8465 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.458, + "recall@10": 0.737, + "all@20": 0.534, + "recall@20": 0.781 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.462, + "recall@10": 0.7385, + "all@20": 0.54, + "recall@20": 0.784 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.386, + "recall@10": 0.6875, + "all@20": 0.438, + "recall@20": 0.721 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.39, + "recall@10": 0.6895, + "all@20": 0.44, + "recall@20": 0.722 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6195, + "all@10": 0.38, + "recall@10": 0.6835, + "all@20": 0.628, + "recall@20": 0.8245 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.298, + "recall@5": 0.6215, + "all@10": 0.38, + "recall@10": 0.6835, + "all@20": 0.634, + "recall@20": 0.8265 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.312, + "recall@5": 0.632, + "all@10": 0.376, + "recall@10": 0.681, + "all@20": 0.644, + "recall@20": 0.8335 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.31, + "recall@5": 0.63, + "all@10": 0.376, + "recall@10": 0.681, + "all@20": 0.648, + "recall@20": 0.8345 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.316, + "recall@5": 0.6365, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.53, + "recall@20": 0.78 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.316, + "recall@5": 0.6355, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.534, + "recall@20": 0.782 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.304, + "recall@5": 0.63, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.438, + "recall@20": 0.7205 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.306, + "recall@5": 0.6305, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.44, + "recall@20": 0.7215 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.266, + "recall@5": 0.593, + "all@10": 0.398, + "recall@10": 0.6895, + "all@20": 0.436, + "recall@20": 0.7175 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.27, + "recall@5": 0.597, + "all@10": 0.396, + "recall@10": 0.6875, + "all@20": 0.432, + "recall@20": 0.7155 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.294, + "recall@5": 0.616, + "all@10": 0.396, + "recall@10": 0.6875, + "all@20": 0.43, + "recall@20": 0.7135 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.294, + "recall@5": 0.615, + "all@10": 0.396, + "recall@10": 0.6875, + "all@20": 0.43, + "recall@20": 0.7135 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.31, + "recall@5": 0.6305, + "all@10": 0.392, + "recall@10": 0.688, + "all@20": 0.424, + "recall@20": 0.7105 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.314, + "recall@5": 0.633, + "all@10": 0.39, + "recall@10": 0.687, + "all@20": 0.424, + "recall@20": 0.7105 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.308, + "recall@5": 0.632, + "all@10": 0.386, + "recall@10": 0.686, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.308, + "recall@5": 0.6315, + "all@10": 0.386, + "recall@10": 0.686, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.338, + "recall@5": 0.6575, + "all@10": 0.552, + "recall@10": 0.778, + "all@20": 0.662, + "recall@20": 0.8305 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.336, + "recall@5": 0.656, + "all@10": 0.56, + "recall@10": 0.7815, + "all@20": 0.674, + "recall@20": 0.836 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.326, + "recall@5": 0.649, + "all@10": 0.562, + "recall@10": 0.7865, + "all@20": 0.692, + "recall@20": 0.8535 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.322, + "recall@5": 0.6465, + "all@10": 0.578, + "recall@10": 0.7935, + "all@20": 0.69, + "recall@20": 0.8535 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.312, + "recall@5": 0.6355, + "all@10": 0.494, + "recall@10": 0.7565, + "all@20": 0.578, + "recall@20": 0.8045 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.31, + "recall@5": 0.6345, + "all@10": 0.498, + "recall@10": 0.759, + "all@20": 0.574, + "recall@20": 0.803 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.3, + "recall@5": 0.629, + "all@10": 0.396, + "recall@10": 0.695, + "all@20": 0.468, + "recall@20": 0.739 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.3, + "recall@5": 0.629, + "all@10": 0.394, + "recall@10": 0.694, + "all@20": 0.466, + "recall@20": 0.7375 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.292, + "recall@5": 0.6125, + "all@10": 0.424, + "recall@10": 0.716, + "all@20": 0.624, + "recall@20": 0.8215 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.302, + "recall@5": 0.6175, + "all@10": 0.428, + "recall@10": 0.718, + "all@20": 0.634, + "recall@20": 0.827 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.304, + "recall@5": 0.631, + "all@10": 0.412, + "recall@10": 0.707, + "all@20": 0.644, + "recall@20": 0.833 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.3, + "recall@5": 0.628, + "all@10": 0.412, + "recall@10": 0.707, + "all@20": 0.652, + "recall@20": 0.8355 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.318, + "recall@5": 0.636, + "all@10": 0.392, + "recall@10": 0.693, + "all@20": 0.57, + "recall@20": 0.798 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.318, + "recall@5": 0.635, + "all@10": 0.392, + "recall@10": 0.693, + "all@20": 0.574, + "recall@20": 0.7995 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.308, + "recall@5": 0.63, + "all@10": 0.378, + "recall@10": 0.6825, + "all@20": 0.466, + "recall@20": 0.7375 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.308, + "recall@5": 0.63, + "all@10": 0.378, + "recall@10": 0.6825, + "all@20": 0.464, + "recall@20": 0.736 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.23, + "recall@5": 0.5425, + "all@10": 0.386, + "recall@10": 0.6795, + "all@20": 0.474, + "recall@20": 0.748 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.238, + "recall@5": 0.546, + "all@10": 0.392, + "recall@10": 0.683, + "all@20": 0.476, + "recall@20": 0.7495 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.254, + "recall@5": 0.5835, + "all@10": 0.408, + "recall@10": 0.698, + "all@20": 0.46, + "recall@20": 0.738 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.256, + "recall@5": 0.585, + "all@10": 0.408, + "recall@10": 0.6965, + "all@20": 0.456, + "recall@20": 0.7365 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.286, + "recall@5": 0.6145, + "all@10": 0.404, + "recall@10": 0.695, + "all@20": 0.446, + "recall@20": 0.7265 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.288, + "recall@5": 0.616, + "all@10": 0.402, + "recall@10": 0.6935, + "all@20": 0.444, + "recall@20": 0.725 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.302, + "recall@5": 0.625, + "all@10": 0.384, + "recall@10": 0.6845, + "all@20": 0.424, + "recall@20": 0.7115 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.302, + "recall@5": 0.625, + "all@10": 0.388, + "recall@10": 0.686, + "all@20": 0.424, + "recall@20": 0.7115 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.334, + "recall@5": 0.645, + "all@10": 0.528, + "recall@10": 0.767, + "all@20": 0.65, + "recall@20": 0.825 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.336, + "recall@5": 0.6475, + "all@10": 0.542, + "recall@10": 0.773, + "all@20": 0.662, + "recall@20": 0.8305 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.342, + "recall@5": 0.658, + "all@10": 0.554, + "recall@10": 0.7805, + "all@20": 0.674, + "recall@20": 0.8465 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.344, + "recall@5": 0.6595, + "all@10": 0.562, + "recall@10": 0.7845, + "all@20": 0.674, + "recall@20": 0.8465 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.318, + "recall@5": 0.645, + "all@10": 0.508, + "recall@10": 0.7645, + "all@20": 0.594, + "recall@20": 0.8145 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.32, + "recall@5": 0.644, + "all@10": 0.508, + "recall@10": 0.7645, + "all@20": 0.59, + "recall@20": 0.8135 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.302, + "recall@5": 0.6305, + "all@10": 0.4, + "recall@10": 0.698, + "all@20": 0.472, + "recall@20": 0.743 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.3, + "recall@5": 0.629, + "all@10": 0.4, + "recall@10": 0.698, + "all@20": 0.47, + "recall@20": 0.7415 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.264, + "recall@5": 0.567, + "all@10": 0.432, + "recall@10": 0.7185, + "all@20": 0.612, + "recall@20": 0.8165 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.274, + "recall@5": 0.572, + "all@10": 0.432, + "recall@10": 0.72, + "all@20": 0.622, + "recall@20": 0.8215 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.28, + "recall@5": 0.5985, + "all@10": 0.43, + "recall@10": 0.7205, + "all@20": 0.628, + "recall@20": 0.826 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.292, + "recall@5": 0.606, + "all@10": 0.428, + "recall@10": 0.7205, + "all@20": 0.634, + "recall@20": 0.828 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.306, + "recall@5": 0.635, + "all@10": 0.41, + "recall@10": 0.707, + "all@20": 0.576, + "recall@20": 0.8015 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.308, + "recall@5": 0.634, + "all@10": 0.41, + "recall@10": 0.707, + "all@20": 0.578, + "recall@20": 0.802 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.308, + "recall@5": 0.6285, + "all@10": 0.384, + "recall@10": 0.6865, + "all@20": 0.466, + "recall@20": 0.7395 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.306, + "recall@5": 0.627, + "all@10": 0.382, + "recall@10": 0.6855, + "all@20": 0.462, + "recall@20": 0.737 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.158, + "recall@5": 0.416, + "all@10": 0.356, + "recall@10": 0.6345, + "all@20": 0.5, + "recall@20": 0.7585 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.164, + "recall@5": 0.414, + "all@10": 0.36, + "recall@10": 0.636, + "all@20": 0.498, + "recall@20": 0.7595 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.194, + "recall@5": 0.491, + "all@10": 0.384, + "recall@10": 0.678, + "all@20": 0.49, + "recall@20": 0.7575 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.204, + "recall@5": 0.494, + "all@10": 0.386, + "recall@10": 0.6805, + "all@20": 0.492, + "recall@20": 0.7585 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.26, + "recall@5": 0.5945, + "all@10": 0.402, + "recall@10": 0.6985, + "all@20": 0.464, + "recall@20": 0.7395 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.256, + "recall@5": 0.592, + "all@10": 0.402, + "recall@10": 0.6975, + "all@20": 0.466, + "recall@20": 0.7405 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.29, + "recall@5": 0.6185, + "all@10": 0.388, + "recall@10": 0.6855, + "all@20": 0.432, + "recall@20": 0.7165 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.284, + "recall@5": 0.6145, + "all@10": 0.388, + "recall@10": 0.6855, + "all@20": 0.434, + "recall@20": 0.7175 + } + ], + "ppr_selected": { + "5": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "20": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + } + }, + "lib_dev_grid": [ + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 10.077736201346852, + "default": true, + "all@5": 0.408, + "recall@5": 0.6935, + "n_ret@5": 5.0, + "all@10": 0.582, + "recall@10": 0.794, + "n_ret@10": 9.966, + "all@20": 0.696, + "recall@20": 0.8505, + "n_ret@20": 19.734 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.0778761997353286, + "default": false, + "all@5": 0.388, + "recall@5": 0.6855, + "n_ret@5": 5.0, + "all@10": 0.568, + "recall@10": 0.786, + "n_ret@10": 10.0, + "all@20": 0.706, + "recall@20": 0.865, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 28.82229260064196, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 26.99003179895226, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 51.519576600170694, + "default": false, + "all@5": 0.3, + "recall@5": 0.6285, + "n_ret@5": 5.0, + "all@10": 0.388, + "recall@10": 0.687, + "n_ret@10": 10.0, + "all@20": 0.442, + "recall@20": 0.7195, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 27.76413020165637, + "default": false, + "all@5": 0.298, + "recall@5": 0.6275, + "n_ret@5": 5.0, + "all@10": 0.374, + "recall@10": 0.68, + "n_ret@10": 10.0, + "all@20": 0.636, + "recall@20": 0.828, + "n_ret@20": 19.9 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 40.48759239935316, + "default": false, + "all@5": 0.376, + "recall@5": 0.671, + "n_ret@5": 5.0, + "all@10": 0.46, + "recall@10": 0.729, + "n_ret@10": 10.0, + "all@20": 0.592, + "recall@20": 0.8115, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 28.924394599511288, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.53, + "recall@20": 0.775, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 43.139962399494834, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 6.360710801440291, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.708, + "recall@20": 0.8625, + "n_ret@20": 19.9 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 25.895725398673676, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.62, + "recall@20": 0.8255, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 37.5268968003802, + "default": false, + "all@5": 0.444, + "recall@5": 0.713, + "n_ret@5": 5.0, + "all@10": 0.574, + "recall@10": 0.7985, + "n_ret@10": 10.0, + "all@20": 0.688, + "recall@20": 0.862, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 25.08842320088297, + "default": false, + "all@5": 0.314, + "recall@5": 0.6355, + "n_ret@5": 5.0, + "all@10": 0.394, + "recall@10": 0.69, + "n_ret@10": 10.0, + "all@20": 0.448, + "recall@20": 0.7225, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 24.80217480007559, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.672, + "recall@20": 0.848, + "n_ret@20": 19.9 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.823397400090471, + "default": false, + "all@5": 0.388, + "recall@5": 0.6855, + "n_ret@5": 5.0, + "all@10": 0.574, + "recall@10": 0.785, + "n_ret@10": 9.91, + "all@20": 0.7, + "recall@20": 0.849, + "n_ret@20": 18.082 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 32.07463439949788, + "default": false, + "all@5": 0.31, + "recall@5": 0.6335, + "n_ret@5": 5.0, + "all@10": 0.394, + "recall@10": 0.69, + "n_ret@10": 10.0, + "all@20": 0.628, + "recall@20": 0.8285, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.6413401999743655, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.66, + "recall@20": 0.846, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 28.52643400116358, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 6.371532000717707, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.63, + "recall@10": 0.8165, + "n_ret@10": 9.91, + "all@20": 0.72, + "recall@20": 0.859, + "n_ret@20": 18.082 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 32.99732759967446, + "default": false, + "all@5": 0.34, + "recall@5": 0.6485, + "n_ret@5": 5.0, + "all@10": 0.626, + "recall@10": 0.816, + "n_ret@10": 9.966, + "all@20": 0.722, + "recall@20": 0.8615, + "n_ret@20": 19.734 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 25.667195400455967, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.7, + "recall@20": 0.861, + "n_ret@20": 19.9 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 41.55993320024572, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.52, + "recall@20": 0.7695, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 23.721735999104567, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.588, + "recall@10": 0.796, + "n_ret@10": 9.91, + "all@20": 0.716, + "recall@20": 0.856, + "n_ret@20": 18.082 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 24.673956600250676, + "default": false, + "all@5": 0.382, + "recall@5": 0.6745, + "n_ret@5": 5.0, + "all@10": 0.47, + "recall@10": 0.735, + "n_ret@10": 10.0, + "all@20": 0.618, + "recall@20": 0.828, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 45.88920819957275, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 23.260251399478875, + "default": false, + "all@5": 0.3, + "recall@5": 0.6285, + "n_ret@5": 5.0, + "all@10": 0.636, + "recall@10": 0.82, + "n_ret@10": 9.91, + "all@20": 0.722, + "recall@20": 0.86, + "n_ret@20": 18.082 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 6.577535399817862, + "default": false, + "all@5": 0.314, + "recall@5": 0.6355, + "n_ret@5": 5.0, + "all@10": 0.394, + "recall@10": 0.69, + "n_ret@10": 10.0, + "all@20": 0.662, + "recall@20": 0.846, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 9.160200400045142, + "default": false, + "all@5": 0.314, + "recall@5": 0.6355, + "n_ret@5": 5.0, + "all@10": 0.63, + "recall@10": 0.8165, + "n_ret@10": 9.966, + "all@20": 0.72, + "recall@20": 0.858, + "n_ret@20": 19.734 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 26.912837000912987, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.63, + "recall@10": 0.817, + "n_ret@10": 9.966, + "all@20": 0.728, + "recall@20": 0.863, + "n_ret@20": 19.734 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 23.10748999984935, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.608, + "recall@20": 0.8125, + "n_ret@20": 19.9 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 21.205239199916832, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.55, + "recall@20": 0.7875, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 6.489484601072036, + "default": false, + "all@5": 0.314, + "recall@5": 0.6355, + "n_ret@5": 5.0, + "all@10": 0.394, + "recall@10": 0.69, + "n_ret@10": 10.0, + "all@20": 0.448, + "recall@20": 0.7225, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 38.93990000104532, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 25.713946800213307, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.574, + "recall@20": 0.798, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 24.573018598603085, + "default": false, + "all@5": 0.444, + "recall@5": 0.713, + "n_ret@5": 5.0, + "all@10": 0.574, + "recall@10": 0.7985, + "n_ret@10": 10.0, + "all@20": 0.692, + "recall@20": 0.8605, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 12.378472800715826, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.66, + "recall@20": 0.846, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 7.515526600182056, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.63, + "recall@10": 0.8165, + "n_ret@10": 9.966, + "all@20": 0.72, + "recall@20": 0.859, + "n_ret@20": 19.734 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 19.29064479866065, + "default": false, + "all@5": 0.3, + "recall@5": 0.6285, + "n_ret@5": 5.0, + "all@10": 0.388, + "recall@10": 0.687, + "n_ret@10": 10.0, + "all@20": 0.442, + "recall@20": 0.7195, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 35.48078040033579, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 18.984256200841628, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 4.769926201202907, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.698, + "recall@20": 0.8595, + "n_ret@20": 19.9 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 18.40023019979708, + "default": false, + "all@5": 0.31, + "recall@5": 0.6335, + "n_ret@5": 5.0, + "all@10": 0.394, + "recall@10": 0.69, + "n_ret@10": 10.0, + "all@20": 0.456, + "recall@20": 0.7265, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 15.33328900008928, + "default": false, + "all@5": 0.392, + "recall@5": 0.6865, + "n_ret@5": 5.0, + "all@10": 0.552, + "recall@10": 0.773, + "n_ret@10": 10.0, + "all@20": 0.654, + "recall@20": 0.8335, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.287669800338335, + "default": false, + "all@5": 0.392, + "recall@5": 0.6865, + "n_ret@5": 5.0, + "all@10": 0.552, + "recall@10": 0.7715, + "n_ret@10": 9.966, + "all@20": 0.658, + "recall@20": 0.828, + "n_ret@20": 19.734 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 26.946429600939155, + "default": false, + "all@5": 0.3, + "recall@5": 0.6285, + "n_ret@5": 5.0, + "all@10": 0.388, + "recall@10": 0.687, + "n_ret@10": 10.0, + "all@20": 0.698, + "recall@20": 0.8585, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 18.976001802715473, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.636, + "recall@20": 0.828, + "n_ret@20": 19.9 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 22.19753720110748, + "default": false, + "all@5": 0.31, + "recall@5": 0.6335, + "n_ret@5": 5.0, + "all@10": 0.558, + "recall@10": 0.7815, + "n_ret@10": 9.966, + "all@20": 0.674, + "recall@20": 0.84, + "n_ret@20": 19.734 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 22.77557079924736, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.554, + "recall@10": 0.7775, + "n_ret@10": 9.966, + "all@20": 0.664, + "recall@20": 0.8355, + "n_ret@20": 19.734 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 38.966716999188066, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.6, + "recall@20": 0.8105, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 26.040046400972642, + "default": false, + "all@5": 0.3, + "recall@5": 0.6285, + "n_ret@5": 5.0, + "all@10": 0.388, + "recall@10": 0.687, + "n_ret@10": 10.0, + "all@20": 0.698, + "recall@20": 0.8585, + "n_ret@20": 19.9 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 33.521275000297464, + "default": false, + "all@5": 0.444, + "recall@5": 0.713, + "n_ret@5": 5.0, + "all@10": 0.574, + "recall@10": 0.7985, + "n_ret@10": 10.0, + "all@20": 0.674, + "recall@20": 0.8555, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 21.95013319910504, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.584, + "recall@20": 0.8055, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 25.178564200992696, + "default": false, + "all@5": 0.298, + "recall@5": 0.6275, + "n_ret@5": 5.0, + "all@10": 0.374, + "recall@10": 0.68, + "n_ret@10": 10.0, + "all@20": 0.618, + "recall@20": 0.819, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 14.1273157998221, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.708, + "recall@20": 0.8625, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 30.427568800165318, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.54, + "recall@20": 0.783, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 21.388833598815836, + "default": false, + "all@5": 0.31, + "recall@5": 0.6335, + "n_ret@5": 5.0, + "all@10": 0.394, + "recall@10": 0.69, + "n_ret@10": 10.0, + "all@20": 0.594, + "recall@20": 0.8125, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 19.861578199081123, + "default": false, + "all@5": 0.298, + "recall@5": 0.6275, + "n_ret@5": 5.0, + "all@10": 0.374, + "recall@10": 0.68, + "n_ret@10": 10.0, + "all@20": 0.432, + "recall@20": 0.7145, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 27.603397398488596, + "default": false, + "all@5": 0.3, + "recall@5": 0.6285, + "n_ret@5": 5.0, + "all@10": 0.636, + "recall@10": 0.82, + "n_ret@10": 9.966, + "all@20": 0.726, + "recall@20": 0.862, + "n_ret@20": 19.734 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 19.253192001837306, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.66, + "recall@20": 0.846, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 34.973033600719646, + "default": false, + "all@5": 0.298, + "recall@5": 0.6275, + "n_ret@5": 5.0, + "all@10": 0.374, + "recall@10": 0.68, + "n_ret@10": 10.0, + "all@20": 0.572, + "recall@20": 0.799, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 20.579228399903513, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.496, + "recall@10": 0.745, + "n_ret@10": 9.966, + "all@20": 0.618, + "recall@20": 0.8125, + "n_ret@20": 19.734 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 42.53371020022314, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 42.109819598379545, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.528, + "recall@20": 0.773, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.45905180205591, + "default": false, + "all@5": 0.368, + "recall@5": 0.6675, + "n_ret@5": 5.0, + "all@10": 0.492, + "recall@10": 0.7485, + "n_ret@10": 10.0, + "all@20": 0.678, + "recall@20": 0.8565, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 41.80156519880984, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.672, + "recall@20": 0.848, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 9.201936799683608, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.662, + "recall@20": 0.846, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 34.64652119996026, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 32.344066800316796, + "default": false, + "all@5": 0.382, + "recall@5": 0.6745, + "n_ret@5": 5.0, + "all@10": 0.47, + "recall@10": 0.735, + "n_ret@10": 10.0, + "all@20": 0.67, + "recall@20": 0.845, + "n_ret@20": 19.9 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 32.35669340076856, + "default": false, + "all@5": 0.31, + "recall@5": 0.6335, + "n_ret@5": 5.0, + "all@10": 0.586, + "recall@10": 0.7945, + "n_ret@10": 9.91, + "all@20": 0.708, + "recall@20": 0.853, + "n_ret@20": 18.082 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 32.59721600054763, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.604, + "recall@10": 0.8045, + "n_ret@10": 9.91, + "all@20": 0.718, + "recall@20": 0.8585, + "n_ret@20": 18.082 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 9.270661200629547, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.662, + "recall@20": 0.846, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 32.89101279946044, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 43.23168959922623, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.62, + "recall@20": 0.8255, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 41.82199159846641, + "default": false, + "all@5": 0.31, + "recall@5": 0.6335, + "n_ret@5": 5.0, + "all@10": 0.394, + "recall@10": 0.69, + "n_ret@10": 10.0, + "all@20": 0.584, + "recall@20": 0.807, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 17.65710580139421, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.708, + "recall@20": 0.8625, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.590577800641768, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.594, + "recall@10": 0.799, + "n_ret@10": 9.91, + "all@20": 0.708, + "recall@20": 0.8545, + "n_ret@20": 18.082 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 29.829709000652656, + "default": false, + "all@5": 0.45, + "recall@5": 0.7165, + "n_ret@5": 5.0, + "all@10": 0.578, + "recall@10": 0.8035, + "n_ret@10": 10.0, + "all@20": 0.688, + "recall@20": 0.8625, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 42.714165401412174, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 27.535386799718253, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 28.214693198213354, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.552, + "recall@10": 0.7765, + "n_ret@10": 9.91, + "all@20": 0.696, + "recall@20": 0.847, + "n_ret@20": 18.082 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 27.75866279983893, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.604, + "recall@20": 0.8135, + "n_ret@20": 19.9 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 41.45068800041918, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 31.936921599437486, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.65, + "recall@20": 0.84, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 13.1609679996036, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.63, + "recall@10": 0.8165, + "n_ret@10": 9.966, + "all@20": 0.72, + "recall@20": 0.859, + "n_ret@20": 19.734 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 14.375748201273382, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.698, + "recall@20": 0.8595, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 32.12127380119637, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.53, + "recall@10": 0.764, + "n_ret@10": 9.966, + "all@20": 0.648, + "recall@20": 0.8285, + "n_ret@20": 19.734 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 29.87190640112385, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.684, + "recall@20": 0.857, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 31.186174201080576, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 42.886621800251305, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 27.579287598957308, + "default": false, + "all@5": 0.45, + "recall@5": 0.7165, + "n_ret@5": 5.0, + "all@10": 0.63, + "recall@10": 0.8205, + "n_ret@10": 9.91, + "all@20": 0.718, + "recall@20": 0.8595, + "n_ret@20": 18.082 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 49.19100719864946, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 30.186425998690538, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.534, + "recall@20": 0.7765, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 32.993400599458255, + "default": false, + "all@5": 0.382, + "recall@5": 0.6745, + "n_ret@5": 5.0, + "all@10": 0.47, + "recall@10": 0.735, + "n_ret@10": 10.0, + "all@20": 0.552, + "recall@20": 0.79, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 33.2452411983395, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.622, + "recall@10": 0.813, + "n_ret@10": 9.966, + "all@20": 0.72, + "recall@20": 0.8595, + "n_ret@20": 19.734 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 12.798152200179175, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.63, + "recall@10": 0.8165, + "n_ret@10": 9.966, + "all@20": 0.72, + "recall@20": 0.859, + "n_ret@20": 19.734 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 30.591044202563353, + "default": false, + "all@5": 0.45, + "recall@5": 0.7165, + "n_ret@5": 5.0, + "all@10": 0.578, + "recall@10": 0.8035, + "n_ret@10": 10.0, + "all@20": 0.718, + "recall@20": 0.871, + "n_ret@20": 19.9 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 7.5778082004981115, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.708, + "recall@20": 0.8625, + "n_ret@20": 19.9 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.543998600682244, + "default": false, + "all@5": 0.388, + "recall@5": 0.6855, + "n_ret@5": 5.0, + "all@10": 0.568, + "recall@10": 0.786, + "n_ret@10": 10.0, + "all@20": 0.704, + "recall@20": 0.864, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 15.872731201001443, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.698, + "recall@20": 0.8595, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 22.027241601375863, + "default": false, + "all@5": 0.368, + "recall@5": 0.6675, + "n_ret@5": 5.0, + "all@10": 0.492, + "recall@10": 0.7485, + "n_ret@10": 10.0, + "all@20": 0.6, + "recall@20": 0.818, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 48.09618520177901, + "default": false, + "all@5": 0.298, + "recall@5": 0.6275, + "n_ret@5": 5.0, + "all@10": 0.374, + "recall@10": 0.68, + "n_ret@10": 10.0, + "all@20": 0.432, + "recall@20": 0.7145, + "n_ret@20": 20.0 + } + ], + "lib_selected": { + "5": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "20": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + } + }, + "seconds": 2647.36417222023 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/2wiki__minilm__entity.npz b/benchmarks/multihop/results/raw/2wiki__minilm__entity.npz new file mode 100644 index 0000000..30399cb Binary files /dev/null and b/benchmarks/multihop/results/raw/2wiki__minilm__entity.npz differ diff --git a/benchmarks/multihop/results/raw/2wiki__minilm__knn.json b/benchmarks/multihop/results/raw/2wiki__minilm__knn.json new file mode 100644 index 0000000..dfe208f --- /dev/null +++ b/benchmarks/multihop/results/raw/2wiki__minilm__knn.json @@ -0,0 +1,3274 @@ +{ + "dataset": "2wiki", + "model": "minilm", + "variant": "knn", + "families": [ + "knn" + ], + "diagnostics": { + "n_edges_undirected_union": 34360, + "n_edges_by_family_directed_or_undirected": 34360, + "mean_degree": 7.438034419309449, + "median_degree": 6.0, + "max_degree": 52, + "isolated_frac": 0.0, + "gold_pairs_directly_linked": 0.18217179902755268, + "gold_pairs_linked_within_2": 0.233387358184765, + "neighbours_of_gold_that_are_in_same_question_context": 0.13361729246028692 + }, + "ppr_dev_grid": [ + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.452, + "recall@10": 0.714, + "all@20": 0.504, + "recall@20": 0.7465 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.458, + "recall@10": 0.719, + "all@20": 0.51, + "recall@20": 0.75 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.432, + "recall@10": 0.7045, + "all@20": 0.496, + "recall@20": 0.7475 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.438, + "recall@10": 0.7085, + "all@20": 0.498, + "recall@20": 0.75 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.404, + "recall@10": 0.695, + "all@20": 0.462, + "recall@20": 0.7295 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.4, + "recall@10": 0.6935, + "all@20": 0.462, + "recall@20": 0.7295 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.382, + "recall@10": 0.684, + "all@20": 0.424, + "recall@20": 0.711 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.384, + "recall@10": 0.685, + "all@20": 0.428, + "recall@20": 0.713 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.28, + "recall@5": 0.6065, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.512, + "recall@20": 0.7545 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.28, + "recall@5": 0.6045, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.52, + "recall@20": 0.7585 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.286, + "recall@5": 0.612, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.5, + "recall@20": 0.7455 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.29, + "recall@5": 0.6145, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.502, + "recall@20": 0.7475 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.3, + "recall@5": 0.6245, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.472, + "recall@20": 0.7335 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.306, + "recall@5": 0.6275, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.47, + "recall@20": 0.7335 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.29, + "recall@5": 0.6215, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.43, + "recall@20": 0.713 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.6245, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.436, + "recall@20": 0.716 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.234, + "recall@5": 0.579, + "all@10": 0.372, + "recall@10": 0.6715, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.234, + "recall@5": 0.5725, + "all@10": 0.368, + "recall@10": 0.6675, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.264, + "recall@5": 0.6, + "all@10": 0.376, + "recall@10": 0.6755, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.264, + "recall@5": 0.599, + "all@10": 0.38, + "recall@10": 0.6775, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.286, + "recall@5": 0.615, + "all@10": 0.388, + "recall@10": 0.685, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.29, + "recall@5": 0.614, + "all@10": 0.392, + "recall@10": 0.685, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.29, + "recall@5": 0.622, + "all@10": 0.378, + "recall@10": 0.682, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.29, + "recall@5": 0.622, + "all@10": 0.38, + "recall@10": 0.683, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.626, + "all@10": 0.442, + "recall@10": 0.7085, + "all@20": 0.508, + "recall@20": 0.7485 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.6255, + "all@10": 0.452, + "recall@10": 0.7165, + "all@20": 0.51, + "recall@20": 0.75 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.442, + "recall@10": 0.7095, + "all@20": 0.486, + "recall@20": 0.74 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.6255, + "all@10": 0.442, + "recall@10": 0.7095, + "all@20": 0.488, + "recall@20": 0.742 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.408, + "recall@10": 0.695, + "all@20": 0.478, + "recall@20": 0.737 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.626, + "all@10": 0.414, + "recall@10": 0.698, + "all@20": 0.474, + "recall@20": 0.7355 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.384, + "recall@10": 0.685, + "all@20": 0.436, + "recall@20": 0.717 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.388, + "recall@10": 0.687, + "all@20": 0.44, + "recall@20": 0.7195 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.252, + "recall@5": 0.584, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.512, + "recall@20": 0.754 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.25, + "recall@5": 0.5805, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.518, + "recall@20": 0.7575 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.272, + "recall@5": 0.6015, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.51, + "recall@20": 0.751 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.262, + "recall@5": 0.595, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.508, + "recall@20": 0.7495 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.29, + "recall@5": 0.6145, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.478, + "recall@20": 0.7345 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.292, + "recall@5": 0.6155, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.474, + "recall@20": 0.7315 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.298, + "recall@5": 0.626, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.446, + "recall@20": 0.721 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.3, + "recall@5": 0.6265, + "all@10": 0.374, + "recall@10": 0.68, + "all@20": 0.452, + "recall@20": 0.7245 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.212, + "recall@5": 0.5415, + "all@10": 0.358, + "recall@10": 0.6585, + "all@20": 0.424, + "recall@20": 0.7105 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.202, + "recall@5": 0.532, + "all@10": 0.356, + "recall@10": 0.6575, + "all@20": 0.424, + "recall@20": 0.7105 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.234, + "recall@5": 0.5725, + "all@10": 0.37, + "recall@10": 0.6695, + "all@20": 0.422, + "recall@20": 0.7095 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.234, + "recall@5": 0.5685, + "all@10": 0.37, + "recall@10": 0.6675, + "all@20": 0.422, + "recall@20": 0.7095 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.266, + "recall@5": 0.601, + "all@10": 0.378, + "recall@10": 0.6765, + "all@20": 0.422, + "recall@20": 0.7095 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.27, + "recall@5": 0.6025, + "all@10": 0.38, + "recall@10": 0.678, + "all@20": 0.422, + "recall@20": 0.7095 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.288, + "recall@5": 0.619, + "all@10": 0.38, + "recall@10": 0.683, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.292, + "recall@5": 0.619, + "all@10": 0.386, + "recall@10": 0.6865, + "all@20": 0.422, + "recall@20": 0.7095 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.286, + "recall@5": 0.62, + "all@10": 0.43, + "recall@10": 0.702, + "all@20": 0.502, + "recall@20": 0.7455 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.282, + "recall@5": 0.6165, + "all@10": 0.444, + "recall@10": 0.7095, + "all@20": 0.502, + "recall@20": 0.746 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.292, + "recall@5": 0.6235, + "all@10": 0.436, + "recall@10": 0.7065, + "all@20": 0.492, + "recall@20": 0.742 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.282, + "recall@5": 0.6175, + "all@10": 0.432, + "recall@10": 0.704, + "all@20": 0.494, + "recall@20": 0.744 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6255, + "all@10": 0.416, + "recall@10": 0.6975, + "all@20": 0.48, + "recall@20": 0.737 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.294, + "recall@5": 0.6245, + "all@10": 0.416, + "recall@10": 0.698, + "all@20": 0.486, + "recall@20": 0.739 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.388, + "recall@10": 0.687, + "all@20": 0.448, + "recall@20": 0.7235 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.626, + "all@10": 0.388, + "recall@10": 0.6875, + "all@20": 0.448, + "recall@20": 0.7235 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.228, + "recall@5": 0.562, + "all@10": 0.372, + "recall@10": 0.6785, + "all@20": 0.502, + "recall@20": 0.749 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.218, + "recall@5": 0.554, + "all@10": 0.37, + "recall@10": 0.6775, + "all@20": 0.508, + "recall@20": 0.7515 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.248, + "recall@5": 0.583, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.504, + "recall@20": 0.748 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.244, + "recall@5": 0.5795, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.512, + "recall@20": 0.7515 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.28, + "recall@5": 0.6075, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.482, + "recall@20": 0.736 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.272, + "recall@5": 0.6035, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.482, + "recall@20": 0.7365 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.294, + "recall@5": 0.6225, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.448, + "recall@20": 0.7225 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.298, + "recall@5": 0.6235, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.452, + "recall@20": 0.7245 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.174, + "recall@5": 0.4995, + "all@10": 0.33, + "recall@10": 0.634, + "all@20": 0.428, + "recall@20": 0.712 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.166, + "recall@5": 0.479, + "all@10": 0.322, + "recall@10": 0.627, + "all@20": 0.43, + "recall@20": 0.713 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.21, + "recall@5": 0.543, + "all@10": 0.352, + "recall@10": 0.6545, + "all@20": 0.426, + "recall@20": 0.711 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.2, + "recall@5": 0.5285, + "all@10": 0.35, + "recall@10": 0.6555, + "all@20": 0.426, + "recall@20": 0.711 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.252, + "recall@5": 0.5885, + "all@10": 0.372, + "recall@10": 0.6725, + "all@20": 0.422, + "recall@20": 0.7095 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.256, + "recall@5": 0.5895, + "all@10": 0.37, + "recall@10": 0.6705, + "all@20": 0.424, + "recall@20": 0.7105 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.288, + "recall@5": 0.617, + "all@10": 0.374, + "recall@10": 0.68, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.292, + "recall@5": 0.6185, + "all@10": 0.376, + "recall@10": 0.681, + "all@20": 0.42, + "recall@20": 0.7085 + } + ], + "ppr_selected": { + "5": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "20": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + } + }, + "lib_dev_grid": [ + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 9.275670999544673, + "default": true, + "all@5": 0.284, + "recall@5": 0.6105, + "n_ret@5": 5.0, + "all@10": 0.45, + "recall@10": 0.708, + "n_ret@10": 10.0, + "all@20": 0.492, + "recall@20": 0.738, + "n_ret@20": 19.98 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.807041398831643, + "default": false, + "all@5": 0.218, + "recall@5": 0.5665, + "n_ret@5": 5.0, + "all@10": 0.438, + "recall@10": 0.6985, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.7485, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 26.6261607989436, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 25.815652200602926, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 47.93606639991049, + "default": false, + "all@5": 0.3, + "recall@5": 0.6285, + "n_ret@5": 5.0, + "all@10": 0.376, + "recall@10": 0.681, + "n_ret@10": 10.0, + "all@20": 0.426, + "recall@20": 0.7115, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 26.509740801062435, + "default": false, + "all@5": 0.298, + "recall@5": 0.6275, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.488, + "recall@20": 0.739, + "n_ret@20": 19.998 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 36.8409654017305, + "default": false, + "all@5": 0.31, + "recall@5": 0.6305, + "n_ret@5": 5.0, + "all@10": 0.41, + "recall@10": 0.6945, + "n_ret@10": 10.0, + "all@20": 0.486, + "recall@20": 0.7415, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 25.357862601056695, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.462, + "recall@20": 0.73, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 41.76671579980757, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 6.778771798941307, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.506, + "recall@20": 0.751, + "n_ret@20": 19.998 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 26.219807600486092, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.482, + "recall@20": 0.7405, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 35.204891398898326, + "default": false, + "all@5": 0.3, + "recall@5": 0.616, + "n_ret@5": 5.0, + "all@10": 0.414, + "recall@10": 0.6905, + "n_ret@10": 10.0, + "all@20": 0.494, + "recall@20": 0.741, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 27.36414159811102, + "default": false, + "all@5": 0.302, + "recall@5": 0.6295, + "n_ret@5": 5.0, + "all@10": 0.378, + "recall@10": 0.682, + "n_ret@10": 10.0, + "all@20": 0.432, + "recall@20": 0.7145, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 27.765198401291855, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.496, + "recall@20": 0.7445, + "n_ret@20": 19.998 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.0425742018269375, + "default": false, + "all@5": 0.218, + "recall@5": 0.5665, + "n_ret@5": 5.0, + "all@10": 0.438, + "recall@10": 0.698, + "n_ret@10": 9.994, + "all@20": 0.486, + "recall@20": 0.735, + "n_ret@20": 19.738 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 34.15251579938922, + "default": false, + "all@5": 0.306, + "recall@5": 0.6315, + "n_ret@5": 5.0, + "all@10": 0.386, + "recall@10": 0.686, + "n_ret@10": 10.0, + "all@20": 0.494, + "recall@20": 0.7445, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.227595601580106, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.52, + "recall@20": 0.7605, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 27.179280199692585, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 6.899278000113554, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.448, + "recall@10": 0.712, + "n_ret@10": 9.994, + "all@20": 0.494, + "recall@20": 0.739, + "n_ret@20": 19.738 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 27.12383479962591, + "default": false, + "all@5": 0.31, + "recall@5": 0.6335, + "n_ret@5": 5.0, + "all@10": 0.444, + "recall@10": 0.71, + "n_ret@10": 10.0, + "all@20": 0.494, + "recall@20": 0.7415, + "n_ret@20": 19.98 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 23.82703760138247, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.508, + "recall@20": 0.7535, + "n_ret@20": 19.998 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 38.583100400283, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.462, + "recall@20": 0.73, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 24.336357799824327, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.426, + "recall@10": 0.702, + "n_ret@10": 9.994, + "all@20": 0.482, + "recall@20": 0.734, + "n_ret@20": 19.738 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 23.142585398163646, + "default": false, + "all@5": 0.32, + "recall@5": 0.636, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.703, + "n_ret@10": 10.0, + "all@20": 0.496, + "recall@20": 0.749, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 40.60610359988641, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 20.31779460143298, + "default": false, + "all@5": 0.3, + "recall@5": 0.6285, + "n_ret@5": 5.0, + "all@10": 0.438, + "recall@10": 0.7055, + "n_ret@10": 9.994, + "all@20": 0.49, + "recall@20": 0.74, + "n_ret@20": 19.738 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 6.482169798924588, + "default": false, + "all@5": 0.302, + "recall@5": 0.6295, + "n_ret@5": 5.0, + "all@10": 0.378, + "recall@10": 0.682, + "n_ret@10": 10.0, + "all@20": 0.512, + "recall@20": 0.757, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 8.883856998872943, + "default": false, + "all@5": 0.302, + "recall@5": 0.6295, + "n_ret@5": 5.0, + "all@10": 0.448, + "recall@10": 0.712, + "n_ret@10": 10.0, + "all@20": 0.496, + "recall@20": 0.74, + "n_ret@20": 19.98 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 23.274469401221722, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.438, + "recall@10": 0.7065, + "n_ret@10": 10.0, + "all@20": 0.486, + "recall@20": 0.7375, + "n_ret@20": 19.98 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 19.6310000004014, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.464, + "recall@20": 0.728, + "n_ret@20": 19.998 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 19.25905440095812, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.464, + "recall@20": 0.731, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 5.925865400233306, + "default": false, + "all@5": 0.302, + "recall@5": 0.6295, + "n_ret@5": 5.0, + "all@10": 0.378, + "recall@10": 0.682, + "n_ret@10": 10.0, + "all@20": 0.432, + "recall@20": 0.7145, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 34.61177259904798, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 23.008987799403258, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.474, + "recall@20": 0.7335, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 23.961671002209187, + "default": false, + "all@5": 0.3, + "recall@5": 0.616, + "n_ret@5": 5.0, + "all@10": 0.414, + "recall@10": 0.6905, + "n_ret@10": 10.0, + "all@20": 0.492, + "recall@20": 0.7385, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 13.026575599680655, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.52, + "recall@20": 0.7605, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 7.680948396795429, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.448, + "recall@10": 0.712, + "n_ret@10": 10.0, + "all@20": 0.496, + "recall@20": 0.74, + "n_ret@20": 19.98 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 19.331532200449146, + "default": false, + "all@5": 0.3, + "recall@5": 0.6285, + "n_ret@5": 5.0, + "all@10": 0.376, + "recall@10": 0.681, + "n_ret@10": 10.0, + "all@20": 0.426, + "recall@20": 0.7115, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 35.1889284014469, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 19.983334397664294, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 4.915908400318585, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.514, + "recall@20": 0.755, + "n_ret@20": 19.998 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 19.922124599455856, + "default": false, + "all@5": 0.306, + "recall@5": 0.6315, + "n_ret@5": 5.0, + "all@10": 0.386, + "recall@10": 0.686, + "n_ret@10": 10.0, + "all@20": 0.452, + "recall@20": 0.724, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 15.284995797788724, + "default": false, + "all@5": 0.218, + "recall@5": 0.5665, + "n_ret@5": 5.0, + "all@10": 0.394, + "recall@10": 0.6705, + "n_ret@10": 10.0, + "all@20": 0.438, + "recall@20": 0.7035, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.837978600291536, + "default": false, + "all@5": 0.218, + "recall@5": 0.5665, + "n_ret@5": 5.0, + "all@10": 0.394, + "recall@10": 0.67, + "n_ret@10": 10.0, + "all@20": 0.434, + "recall@20": 0.698, + "n_ret@20": 19.98 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 29.44530580053106, + "default": false, + "all@5": 0.3, + "recall@5": 0.6285, + "n_ret@5": 5.0, + "all@10": 0.376, + "recall@10": 0.681, + "n_ret@10": 10.0, + "all@20": 0.498, + "recall@20": 0.745, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 27.4285072007915, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.492, + "recall@20": 0.7425, + "n_ret@20": 19.998 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 30.220299398875795, + "default": false, + "all@5": 0.306, + "recall@5": 0.6315, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.699, + "n_ret@10": 10.0, + "all@20": 0.484, + "recall@20": 0.7365, + "n_ret@20": 19.98 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 23.92033519956749, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.414, + "recall@10": 0.6965, + "n_ret@10": 10.0, + "all@20": 0.478, + "recall@20": 0.732, + "n_ret@20": 19.98 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 28.216232400154695, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.48, + "recall@20": 0.7365, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 18.166003400692716, + "default": false, + "all@5": 0.3, + "recall@5": 0.6285, + "n_ret@5": 5.0, + "all@10": 0.376, + "recall@10": 0.681, + "n_ret@10": 10.0, + "all@20": 0.498, + "recall@20": 0.745, + "n_ret@20": 19.998 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 33.73006639862433, + "default": false, + "all@5": 0.3, + "recall@5": 0.616, + "n_ret@5": 5.0, + "all@10": 0.414, + "recall@10": 0.6905, + "n_ret@10": 10.0, + "all@20": 0.494, + "recall@20": 0.741, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 20.428019800805487, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.478, + "recall@20": 0.738, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 27.241327798576094, + "default": false, + "all@5": 0.298, + "recall@5": 0.6275, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.488, + "recall@20": 0.739, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 11.974084599176422, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.506, + "recall@20": 0.751, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 39.899418798973784, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.464, + "recall@20": 0.731, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 36.66336999903433, + "default": false, + "all@5": 0.306, + "recall@5": 0.6315, + "n_ret@5": 5.0, + "all@10": 0.386, + "recall@10": 0.686, + "n_ret@10": 10.0, + "all@20": 0.49, + "recall@20": 0.746, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 19.223744199611247, + "default": false, + "all@5": 0.298, + "recall@5": 0.6275, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.422, + "recall@20": 0.7095, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 20.674894799361937, + "default": false, + "all@5": 0.3, + "recall@5": 0.6285, + "n_ret@5": 5.0, + "all@10": 0.438, + "recall@10": 0.7055, + "n_ret@10": 10.0, + "all@20": 0.492, + "recall@20": 0.741, + "n_ret@20": 19.98 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 17.051202799892053, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.52, + "recall@20": 0.7605, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 33.93435499991756, + "default": false, + "all@5": 0.298, + "recall@5": 0.6275, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.472, + "recall@20": 0.735, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 37.498356600059196, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.408, + "recall@10": 0.6925, + "n_ret@10": 10.0, + "all@20": 0.472, + "recall@20": 0.7275, + "n_ret@20": 19.98 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 53.01767439942341, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 42.30803359905258, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.462, + "recall@20": 0.727, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 9.480163000989705, + "default": false, + "all@5": 0.31, + "recall@5": 0.633, + "n_ret@5": 5.0, + "all@10": 0.428, + "recall@10": 0.706, + "n_ret@10": 10.0, + "all@20": 0.522, + "recall@20": 0.7615, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 41.70174619858153, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.496, + "recall@20": 0.7445, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 9.373043800937012, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.512, + "recall@20": 0.757, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 35.551733999163844, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 32.18419280170929, + "default": false, + "all@5": 0.32, + "recall@5": 0.636, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.703, + "n_ret@10": 10.0, + "all@20": 0.498, + "recall@20": 0.748, + "n_ret@20": 19.998 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 29.22142459906172, + "default": false, + "all@5": 0.306, + "recall@5": 0.6315, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.699, + "n_ret@10": 9.994, + "all@20": 0.48, + "recall@20": 0.7345, + "n_ret@20": 19.738 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 29.654934799065813, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.428, + "recall@10": 0.7025, + "n_ret@10": 9.994, + "all@20": 0.48, + "recall@20": 0.7335, + "n_ret@20": 19.738 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 9.246965199825354, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.512, + "recall@20": 0.757, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 30.562896200455725, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 41.5029724012129, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.482, + "recall@20": 0.7405, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 38.96746079984587, + "default": false, + "all@5": 0.306, + "recall@5": 0.6315, + "n_ret@5": 5.0, + "all@10": 0.386, + "recall@10": 0.686, + "n_ret@10": 10.0, + "all@20": 0.49, + "recall@20": 0.746, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 17.629447799758054, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.506, + "recall@20": 0.751, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.559785401215777, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.448, + "recall@10": 0.7105, + "n_ret@10": 9.994, + "all@20": 0.49, + "recall@20": 0.7385, + "n_ret@20": 19.738 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 30.503145598224364, + "default": false, + "all@5": 0.302, + "recall@5": 0.6175, + "n_ret@5": 5.0, + "all@10": 0.44, + "recall@10": 0.7065, + "n_ret@10": 10.0, + "all@20": 0.514, + "recall@20": 0.7585, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 41.261122798896395, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 26.718426601728424, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 31.54559560213238, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.406, + "recall@10": 0.691, + "n_ret@10": 9.994, + "all@20": 0.47, + "recall@20": 0.7275, + "n_ret@20": 19.738 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 29.863152400008403, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.474, + "recall@20": 0.7335, + "n_ret@20": 19.998 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 44.33402960142121, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 30.687199001782574, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.494, + "recall@20": 0.7465, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 11.9787397995824, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.448, + "recall@10": 0.712, + "n_ret@10": 10.0, + "all@20": 0.496, + "recall@20": 0.74, + "n_ret@20": 19.98 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 15.890479199821131, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.512, + "recall@20": 0.754, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 32.18080880038906, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.41, + "recall@10": 0.694, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.7305, + "n_ret@20": 19.98 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 29.8200850012945, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.502, + "recall@20": 0.7515, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 34.560347201651894, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 45.51996379927732, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 32.07435079792049, + "default": false, + "all@5": 0.302, + "recall@5": 0.6175, + "n_ret@5": 5.0, + "all@10": 0.444, + "recall@10": 0.707, + "n_ret@10": 9.994, + "all@20": 0.5, + "recall@20": 0.744, + "n_ret@20": 19.738 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 48.943360201315954, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 30.72998420044314, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.462, + "recall@20": 0.73, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 32.10710279934574, + "default": false, + "all@5": 0.32, + "recall@5": 0.636, + "n_ret@5": 5.0, + "all@10": 0.422, + "recall@10": 0.703, + "n_ret@10": 10.0, + "all@20": 0.488, + "recall@20": 0.7455, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 33.2378687991295, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.438, + "recall@10": 0.707, + "n_ret@10": 10.0, + "all@20": 0.494, + "recall@20": 0.7425, + "n_ret@20": 19.98 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 12.244516599457711, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.448, + "recall@10": 0.712, + "n_ret@10": 10.0, + "all@20": 0.496, + "recall@20": 0.74, + "n_ret@20": 19.98 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 28.837783598108217, + "default": false, + "all@5": 0.302, + "recall@5": 0.6175, + "n_ret@5": 5.0, + "all@10": 0.44, + "recall@10": 0.7065, + "n_ret@10": 10.0, + "all@20": 0.514, + "recall@20": 0.7565, + "n_ret@20": 19.998 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 7.556238398887217, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.506, + "recall@20": 0.751, + "n_ret@20": 19.998 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.099460800294764, + "default": false, + "all@5": 0.218, + "recall@5": 0.5665, + "n_ret@5": 5.0, + "all@10": 0.438, + "recall@10": 0.6985, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.7485, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 14.215506000909954, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.512, + "recall@20": 0.754, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 22.44568059965968, + "default": false, + "all@5": 0.31, + "recall@5": 0.633, + "n_ret@5": 5.0, + "all@10": 0.428, + "recall@10": 0.706, + "n_ret@10": 10.0, + "all@20": 0.496, + "recall@20": 0.7475, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 45.053210798767395, + "default": false, + "all@5": 0.298, + "recall@5": 0.6275, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.422, + "recall@20": 0.7095, + "n_ret@20": 20.0 + } + ], + "lib_selected": { + "5": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + } + }, + "seconds": 2934.6193776130676 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/2wiki__minilm__knn.npz b/benchmarks/multihop/results/raw/2wiki__minilm__knn.npz new file mode 100644 index 0000000..35a90b0 Binary files /dev/null and b/benchmarks/multihop/results/raw/2wiki__minilm__knn.npz differ diff --git a/benchmarks/multihop/results/raw/2wiki__minilm__mention.json b/benchmarks/multihop/results/raw/2wiki__minilm__mention.json new file mode 100644 index 0000000..b34a9a4 --- /dev/null +++ b/benchmarks/multihop/results/raw/2wiki__minilm__mention.json @@ -0,0 +1,3274 @@ +{ + "dataset": "2wiki", + "model": "minilm", + "variant": "mention", + "families": [ + "mention" + ], + "diagnostics": { + "n_edges_undirected_union": 5383, + "n_edges_by_family_directed_or_undirected": 6115, + "mean_degree": 1.165277627448858, + "median_degree": 1.0, + "max_degree": 30, + "isolated_frac": 0.4217988959844139, + "gold_pairs_directly_linked": 0.44278768233387356, + "gold_pairs_linked_within_2": 0.4447325769854133, + "neighbours_of_gold_that_are_in_same_question_context": 0.42326321090525637 + }, + "ppr_dev_grid": [ + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.326, + "recall@5": 0.6425, + "all@10": 0.798, + "recall@10": 0.8915, + "all@20": 0.81, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.32, + "recall@5": 0.6385, + "all@10": 0.77, + "recall@10": 0.8785, + "all@20": 0.812, + "recall@20": 0.896 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.302, + "recall@5": 0.6305, + "all@10": 0.79, + "recall@10": 0.898, + "all@20": 0.838, + "recall@20": 0.9255 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.308, + "recall@5": 0.6325, + "all@10": 0.748, + "recall@10": 0.8765, + "all@20": 0.81, + "recall@20": 0.913 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.3, + "recall@5": 0.629, + "all@10": 0.624, + "recall@10": 0.8245, + "all@20": 0.712, + "recall@20": 0.8715 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.302, + "recall@5": 0.6295, + "all@10": 0.606, + "recall@10": 0.813, + "all@20": 0.678, + "recall@20": 0.856 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.3, + "recall@5": 0.629, + "all@10": 0.43, + "recall@10": 0.7115, + "all@20": 0.494, + "recall@20": 0.7515 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.42, + "recall@10": 0.707, + "all@20": 0.476, + "recall@20": 0.7425 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.376, + "recall@5": 0.6645, + "all@10": 0.414, + "recall@10": 0.7015, + "all@20": 0.858, + "recall@20": 0.9255 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.324, + "recall@5": 0.6335, + "all@10": 0.434, + "recall@10": 0.7115, + "all@20": 0.838, + "recall@20": 0.918 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.354, + "recall@5": 0.6545, + "all@10": 0.396, + "recall@10": 0.6915, + "all@20": 0.862, + "recall@20": 0.932 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.328, + "recall@5": 0.64, + "all@10": 0.408, + "recall@10": 0.697, + "all@20": 0.828, + "recall@20": 0.918 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.346, + "recall@5": 0.6515, + "all@10": 0.382, + "recall@10": 0.6845, + "all@20": 0.694, + "recall@20": 0.86 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.334, + "recall@5": 0.646, + "all@10": 0.382, + "recall@10": 0.684, + "all@20": 0.688, + "recall@20": 0.857 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.332, + "recall@5": 0.6455, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.492, + "recall@20": 0.751 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.326, + "recall@5": 0.6445, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.482, + "recall@20": 0.7445 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.358, + "recall@5": 0.64, + "all@10": 0.44, + "recall@10": 0.7125, + "all@20": 0.48, + "recall@20": 0.7405 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6045, + "all@10": 0.41, + "recall@10": 0.689, + "all@20": 0.5, + "recall@20": 0.7565 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.364, + "recall@5": 0.652, + "all@10": 0.418, + "recall@10": 0.7005, + "all@20": 0.45, + "recall@20": 0.7245 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.308, + "recall@5": 0.6275, + "all@10": 0.4, + "recall@10": 0.6905, + "all@20": 0.46, + "recall@20": 0.729 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.356, + "recall@5": 0.653, + "all@10": 0.404, + "recall@10": 0.693, + "all@20": 0.436, + "recall@20": 0.717 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.338, + "recall@5": 0.648, + "all@10": 0.404, + "recall@10": 0.695, + "all@20": 0.444, + "recall@20": 0.7205 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.336, + "recall@5": 0.648, + "all@10": 0.398, + "recall@10": 0.692, + "all@20": 0.424, + "recall@20": 0.7105 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.336, + "recall@5": 0.6485, + "all@10": 0.4, + "recall@10": 0.6935, + "all@20": 0.422, + "recall@20": 0.7095 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.468, + "recall@5": 0.722, + "all@10": 0.796, + "recall@10": 0.8905, + "all@20": 0.81, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.486, + "recall@5": 0.7385, + "all@10": 0.764, + "recall@10": 0.8765, + "all@20": 0.81, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.404, + "recall@5": 0.6865, + "all@10": 0.792, + "recall@10": 0.898, + "all@20": 0.836, + "recall@20": 0.9245 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.464, + "recall@5": 0.727, + "all@10": 0.764, + "recall@10": 0.8835, + "all@20": 0.832, + "recall@20": 0.9205 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.342, + "recall@5": 0.6525, + "all@10": 0.68, + "recall@10": 0.8485, + "all@20": 0.746, + "recall@20": 0.8865 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.362, + "recall@5": 0.6655, + "all@10": 0.646, + "recall@10": 0.832, + "all@20": 0.72, + "recall@20": 0.874 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.308, + "recall@5": 0.633, + "all@10": 0.446, + "recall@10": 0.7235, + "all@20": 0.536, + "recall@20": 0.775 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.324, + "recall@5": 0.6415, + "all@10": 0.442, + "recall@10": 0.721, + "all@20": 0.532, + "recall@20": 0.773 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.476, + "recall@5": 0.709, + "all@10": 0.562, + "recall@10": 0.7915, + "all@20": 0.856, + "recall@20": 0.9245 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.372, + "recall@5": 0.645, + "all@10": 0.612, + "recall@10": 0.8165, + "all@20": 0.834, + "recall@20": 0.9165 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.6875, + "all@10": 0.496, + "recall@10": 0.7485, + "all@20": 0.86, + "recall@20": 0.931 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.376, + "recall@5": 0.6615, + "all@10": 0.566, + "recall@10": 0.798, + "all@20": 0.848, + "recall@20": 0.9235 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.374, + "recall@5": 0.6665, + "all@10": 0.428, + "recall@10": 0.71, + "all@20": 0.724, + "recall@20": 0.874 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.35, + "recall@5": 0.6525, + "all@10": 0.47, + "recall@10": 0.7375, + "all@20": 0.734, + "recall@20": 0.877 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.342, + "recall@5": 0.651, + "all@10": 0.386, + "recall@10": 0.6865, + "all@20": 0.522, + "recall@20": 0.7665 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.328, + "recall@5": 0.644, + "all@10": 0.408, + "recall@10": 0.6975, + "all@20": 0.524, + "recall@20": 0.769 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.386, + "recall@5": 0.629, + "all@10": 0.566, + "recall@10": 0.789, + "all@20": 0.644, + "recall@20": 0.844 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.262, + "recall@5": 0.5305, + "all@10": 0.492, + "recall@10": 0.735, + "all@20": 0.678, + "recall@20": 0.8565 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.398, + "recall@5": 0.6565, + "all@10": 0.496, + "recall@10": 0.746, + "all@20": 0.574, + "recall@20": 0.7975 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.3, + "recall@5": 0.595, + "all@10": 0.484, + "recall@10": 0.7345, + "all@20": 0.644, + "recall@20": 0.842 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.368, + "recall@5": 0.656, + "all@10": 0.436, + "recall@10": 0.711, + "all@20": 0.49, + "recall@20": 0.7485 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.336, + "recall@5": 0.6415, + "all@10": 0.436, + "recall@10": 0.7095, + "all@20": 0.536, + "recall@20": 0.778 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.35, + "recall@5": 0.656, + "all@10": 0.4, + "recall@10": 0.6925, + "all@20": 0.436, + "recall@20": 0.7175 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.328, + "recall@5": 0.6445, + "all@10": 0.402, + "recall@10": 0.6945, + "all@20": 0.45, + "recall@20": 0.726 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.532, + "recall@5": 0.76, + "all@10": 0.788, + "recall@10": 0.887, + "all@20": 0.81, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.48, + "recall@5": 0.7065, + "all@10": 0.756, + "recall@10": 0.873, + "all@20": 0.81, + "recall@20": 0.8945 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.472, + "recall@5": 0.7275, + "all@10": 0.78, + "recall@10": 0.892, + "all@20": 0.828, + "recall@20": 0.9205 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.496, + "recall@5": 0.7385, + "all@10": 0.758, + "recall@10": 0.8785, + "all@20": 0.834, + "recall@20": 0.9205 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.382, + "recall@5": 0.677, + "all@10": 0.68, + "recall@10": 0.8495, + "all@20": 0.738, + "recall@20": 0.8825 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.456, + "recall@5": 0.718, + "all@10": 0.67, + "recall@10": 0.8425, + "all@20": 0.746, + "recall@20": 0.8865 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.32, + "recall@5": 0.6415, + "all@10": 0.462, + "recall@10": 0.731, + "all@20": 0.538, + "recall@20": 0.7795 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.344, + "recall@5": 0.653, + "all@10": 0.466, + "recall@10": 0.734, + "all@20": 0.544, + "recall@20": 0.783 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.466, + "recall@5": 0.687, + "all@10": 0.67, + "recall@10": 0.8495, + "all@20": 0.852, + "recall@20": 0.9235 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.318, + "recall@5": 0.556, + "all@10": 0.606, + "recall@10": 0.785, + "all@20": 0.83, + "recall@20": 0.914 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.462, + "recall@5": 0.6965, + "all@10": 0.58, + "recall@10": 0.797, + "all@20": 0.846, + "recall@20": 0.9265 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.374, + "recall@5": 0.63, + "all@10": 0.628, + "recall@10": 0.8155, + "all@20": 0.84, + "recall@20": 0.921 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.398, + "recall@5": 0.676, + "all@10": 0.462, + "recall@10": 0.732, + "all@20": 0.708, + "recall@20": 0.8675 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.382, + "recall@5": 0.6705, + "all@10": 0.546, + "recall@10": 0.782, + "all@20": 0.76, + "recall@20": 0.8895 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.342, + "recall@5": 0.6535, + "all@10": 0.398, + "recall@10": 0.6935, + "all@20": 0.52, + "recall@20": 0.766 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.342, + "recall@5": 0.6485, + "all@10": 0.418, + "recall@10": 0.7045, + "all@20": 0.536, + "recall@20": 0.7785 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.306, + "recall@5": 0.5245, + "all@10": 0.586, + "recall@10": 0.786, + "all@20": 0.742, + "recall@20": 0.8885 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.248, + "recall@5": 0.465, + "all@10": 0.394, + "recall@10": 0.6025, + "all@20": 0.682, + "recall@20": 0.8335 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.36, + "recall@5": 0.601, + "all@10": 0.546, + "recall@10": 0.7695, + "all@20": 0.614, + "recall@20": 0.825 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.272, + "recall@5": 0.5125, + "all@10": 0.464, + "recall@10": 0.701, + "all@20": 0.706, + "recall@20": 0.864 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.376, + "recall@5": 0.654, + "all@10": 0.454, + "recall@10": 0.7245, + "all@20": 0.53, + "recall@20": 0.771 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.32, + "recall@5": 0.62, + "all@10": 0.48, + "recall@10": 0.7345, + "all@20": 0.616, + "recall@20": 0.8245 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.34, + "recall@5": 0.6485, + "all@10": 0.404, + "recall@10": 0.696, + "all@20": 0.456, + "recall@20": 0.728 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.332, + "recall@5": 0.6435, + "all@10": 0.422, + "recall@10": 0.7025, + "all@20": 0.486, + "recall@20": 0.747 + } + ], + "ppr_selected": { + "5": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "20": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + } + }, + "lib_dev_grid": [ + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.6198549991240725, + "default": true, + "all@5": 0.556, + "recall@5": 0.7795, + "n_ret@5": 5.0, + "all@10": 0.802, + "recall@10": 0.8925, + "n_ret@10": 7.804, + "all@20": 0.808, + "recall@20": 0.894, + "n_ret@20": 8.38 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.228924200637266, + "default": false, + "all@5": 0.442, + "recall@5": 0.7075, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.784, + "n_ret@10": 10.0, + "all@20": 0.86, + "recall@20": 0.926, + "n_ret@20": 15.672 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 26.139204400591552, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.756, + "recall@20": 0.892, + "n_ret@20": 19.52 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 30.12361459992826, + "default": false, + "all@5": 0.406, + "recall@5": 0.687, + "n_ret@5": 5.0, + "all@10": 0.496, + "recall@10": 0.749, + "n_ret@10": 10.0, + "all@20": 0.578, + "recall@20": 0.804, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 30.11535700096283, + "default": false, + "all@5": 0.408, + "recall@5": 0.6875, + "n_ret@5": 5.0, + "all@10": 0.796, + "recall@10": 0.89, + "n_ret@10": 7.804, + "all@20": 0.808, + "recall@20": 0.894, + "n_ret@20": 8.38 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 30.238502600695938, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 6.3727490020683035, + "default": false, + "all@5": 0.642, + "recall@5": 0.8265, + "n_ret@5": 5.0, + "all@10": 0.77, + "recall@10": 0.8815, + "n_ret@10": 9.144, + "all@20": 0.806, + "recall@20": 0.894, + "n_ret@20": 13.876 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 33.06163360073697, + "default": false, + "all@5": 0.52, + "recall@5": 0.7605, + "n_ret@5": 5.0, + "all@10": 0.636, + "recall@10": 0.836, + "n_ret@10": 10.0, + "all@20": 0.746, + "recall@20": 0.8925, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.509596999268979, + "default": false, + "all@5": 0.656, + "recall@5": 0.8335, + "n_ret@5": 5.0, + "all@10": 0.764, + "recall@10": 0.8915, + "n_ret@10": 10.0, + "all@20": 0.872, + "recall@20": 0.936, + "n_ret@20": 19.262 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.401913798996247, + "default": false, + "all@5": 0.442, + "recall@5": 0.7075, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.784, + "n_ret@10": 10.0, + "all@20": 0.86, + "recall@20": 0.926, + "n_ret@20": 14.484 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 30.548253200249746, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 6.732100799679756, + "default": false, + "all@5": 0.442, + "recall@5": 0.7075, + "n_ret@5": 5.0, + "all@10": 0.798, + "recall@10": 0.8925, + "n_ret@10": 9.144, + "all@20": 0.814, + "recall@20": 0.8975, + "n_ret@20": 13.876 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.564859202713706, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.874, + "recall@20": 0.9375, + "n_ret@20": 19.52 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 28.20999740017578, + "default": false, + "all@5": 0.352, + "recall@5": 0.6545, + "n_ret@5": 5.0, + "all@10": 0.772, + "recall@10": 0.881, + "n_ret@10": 8.55, + "all@20": 0.808, + "recall@20": 0.8945, + "n_ret@20": 9.8 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 29.534538399777375, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.854, + "recall@20": 0.923, + "n_ret@20": 16.072 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 28.599669199902564, + "default": false, + "all@5": 0.308, + "recall@5": 0.6325, + "n_ret@5": 5.0, + "all@10": 0.396, + "recall@10": 0.691, + "n_ret@10": 10.0, + "all@20": 0.796, + "recall@20": 0.91, + "n_ret@20": 19.262 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 5.733079400495626, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.798, + "recall@10": 0.892, + "n_ret@10": 8.55, + "all@20": 0.81, + "recall@20": 0.8955, + "n_ret@20": 9.8 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 27.83940039924346, + "default": false, + "all@5": 0.52, + "recall@5": 0.7605, + "n_ret@5": 5.0, + "all@10": 0.638, + "recall@10": 0.8375, + "n_ret@10": 10.0, + "all@20": 0.844, + "recall@20": 0.9215, + "n_ret@20": 18.858 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 7.4503921997966245, + "default": false, + "all@5": 0.642, + "recall@5": 0.8255, + "n_ret@5": 5.0, + "all@10": 0.732, + "recall@10": 0.8765, + "n_ret@10": 10.0, + "all@20": 0.818, + "recall@20": 0.917, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 5.921646399889141, + "default": false, + "all@5": 0.652, + "recall@5": 0.832, + "n_ret@5": 5.0, + "all@10": 0.76, + "recall@10": 0.8895, + "n_ret@10": 10.0, + "all@20": 0.856, + "recall@20": 0.926, + "n_ret@20": 17.33 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 27.576754999696277, + "default": false, + "all@5": 0.352, + "recall@5": 0.6545, + "n_ret@5": 5.0, + "all@10": 0.464, + "recall@10": 0.727, + "n_ret@10": 10.0, + "all@20": 0.86, + "recall@20": 0.926, + "n_ret@20": 15.672 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 26.36280759877991, + "default": false, + "all@5": 0.352, + "recall@5": 0.6545, + "n_ret@5": 5.0, + "all@10": 0.464, + "recall@10": 0.727, + "n_ret@10": 10.0, + "all@20": 0.86, + "recall@20": 0.926, + "n_ret@20": 16.072 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 6.737295797793195, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.858, + "recall@20": 0.927, + "n_ret@20": 19.214 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 27.436741000739858, + "default": false, + "all@5": 0.406, + "recall@5": 0.687, + "n_ret@5": 5.0, + "all@10": 0.496, + "recall@10": 0.749, + "n_ret@10": 10.0, + "all@20": 0.736, + "recall@20": 0.8855, + "n_ret@20": 19.824 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 26.596900599775836, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.79, + "recall@20": 0.9, + "n_ret@20": 18.858 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 26.30688279995229, + "default": false, + "all@5": 0.352, + "recall@5": 0.6545, + "n_ret@5": 5.0, + "all@10": 0.464, + "recall@10": 0.727, + "n_ret@10": 10.0, + "all@20": 0.568, + "recall@20": 0.79, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 24.81525879900437, + "default": false, + "all@5": 0.406, + "recall@5": 0.687, + "n_ret@5": 5.0, + "all@10": 0.706, + "recall@10": 0.855, + "n_ret@10": 9.144, + "all@20": 0.804, + "recall@20": 0.8935, + "n_ret@20": 13.876 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 26.279867599485442, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.784, + "recall@20": 0.908, + "n_ret@20": 19.94 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 25.48036419961136, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.838, + "recall@20": 0.9185, + "n_ret@20": 19.214 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.706620399141684, + "default": false, + "all@5": 0.652, + "recall@5": 0.832, + "n_ret@5": 5.0, + "all@10": 0.76, + "recall@10": 0.8895, + "n_ret@10": 10.0, + "all@20": 0.844, + "recall@20": 0.9285, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 26.080517799360678, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 24.746894802316092, + "default": false, + "all@5": 0.308, + "recall@5": 0.6325, + "n_ret@5": 5.0, + "all@10": 0.396, + "recall@10": 0.691, + "n_ret@10": 10.0, + "all@20": 0.788, + "recall@20": 0.9065, + "n_ret@20": 19.52 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.322889401577413, + "default": false, + "all@5": 0.442, + "recall@5": 0.7075, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.784, + "n_ret@10": 10.0, + "all@20": 0.874, + "recall@20": 0.9375, + "n_ret@20": 19.262 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 24.52363700058777, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 25.046798400348052, + "default": false, + "all@5": 0.308, + "recall@5": 0.6325, + "n_ret@5": 5.0, + "all@10": 0.71, + "recall@10": 0.8575, + "n_ret@10": 9.226, + "all@20": 0.802, + "recall@20": 0.8925, + "n_ret@20": 15.262 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 26.9276329978602, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.828, + "recall@20": 0.9235, + "n_ret@20": 19.52 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.6031754004070535, + "default": false, + "all@5": 0.442, + "recall@5": 0.7075, + "n_ret@5": 5.0, + "all@10": 0.806, + "recall@10": 0.894, + "n_ret@10": 7.932, + "all@20": 0.81, + "recall@20": 0.895, + "n_ret@20": 8.874 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.312946200603619, + "default": false, + "all@5": 0.652, + "recall@5": 0.832, + "n_ret@5": 5.0, + "all@10": 0.76, + "recall@10": 0.8895, + "n_ret@10": 10.0, + "all@20": 0.87, + "recall@20": 0.9355, + "n_ret@20": 19.52 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 7.008032999699935, + "default": false, + "all@5": 0.652, + "recall@5": 0.832, + "n_ret@5": 5.0, + "all@10": 0.76, + "recall@10": 0.8895, + "n_ret@10": 10.0, + "all@20": 0.862, + "recall@20": 0.9345, + "n_ret@20": 19.824 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 25.87375739926938, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.187916200840846, + "default": false, + "all@5": 0.652, + "recall@5": 0.832, + "n_ret@5": 5.0, + "all@10": 0.796, + "recall@10": 0.89, + "n_ret@10": 7.804, + "all@20": 0.808, + "recall@20": 0.894, + "n_ret@20": 8.38 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 7.703221801086329, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.43, + "recall@20": 0.7135, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.101366000366397, + "default": false, + "all@5": 0.442, + "recall@5": 0.7075, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.784, + "n_ret@10": 10.0, + "all@20": 0.874, + "recall@20": 0.9375, + "n_ret@20": 19.52 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 27.127562600420788, + "default": false, + "all@5": 0.406, + "recall@5": 0.687, + "n_ret@5": 5.0, + "all@10": 0.494, + "recall@10": 0.748, + "n_ret@10": 10.0, + "all@20": 0.576, + "recall@20": 0.8035, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 24.264357998850755, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.86, + "recall@20": 0.9255, + "n_ret@20": 14.484 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 23.585485599236563, + "default": false, + "all@5": 0.308, + "recall@5": 0.6325, + "n_ret@5": 5.0, + "all@10": 0.796, + "recall@10": 0.8895, + "n_ret@10": 7.804, + "all@20": 0.808, + "recall@20": 0.894, + "n_ret@20": 8.38 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 26.897643001168035, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 6.199232599232346, + "default": false, + "all@5": 0.642, + "recall@5": 0.8255, + "n_ret@5": 5.0, + "all@10": 0.732, + "recall@10": 0.8765, + "n_ret@10": 10.0, + "all@20": 0.828, + "recall@20": 0.9205, + "n_ret@20": 19.94 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 23.890334799536504, + "default": false, + "all@5": 0.308, + "recall@5": 0.6325, + "n_ret@5": 5.0, + "all@10": 0.714, + "recall@10": 0.8595, + "n_ret@10": 9.144, + "all@20": 0.808, + "recall@20": 0.895, + "n_ret@20": 13.876 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 24.044635600643232, + "default": false, + "all@5": 0.308, + "recall@5": 0.6325, + "n_ret@5": 5.0, + "all@10": 0.396, + "recall@10": 0.691, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.9035, + "n_ret@20": 18.858 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 21.83143380016554, + "default": false, + "all@5": 0.524, + "recall@5": 0.763, + "n_ret@5": 5.0, + "all@10": 0.644, + "recall@10": 0.8395, + "n_ret@10": 10.0, + "all@20": 0.854, + "recall@20": 0.9245, + "n_ret@20": 17.33 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.144775798078626, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.43, + "recall@20": 0.7135, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 22.595436600618996, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.75, + "recall@20": 0.8925, + "n_ret@20": 19.824 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 20.261350400163792, + "default": false, + "all@5": 0.408, + "recall@5": 0.6875, + "n_ret@5": 5.0, + "all@10": 0.496, + "recall@10": 0.749, + "n_ret@10": 10.0, + "all@20": 0.856, + "recall@20": 0.924, + "n_ret@20": 16.072 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 5.217967800446786, + "default": false, + "all@5": 0.442, + "recall@5": 0.7075, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.784, + "n_ret@10": 10.0, + "all@20": 0.856, + "recall@20": 0.9265, + "n_ret@20": 18.858 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 5.325431000092067, + "default": false, + "all@5": 0.652, + "recall@5": 0.832, + "n_ret@5": 5.0, + "all@10": 0.76, + "recall@10": 0.8895, + "n_ret@10": 10.0, + "all@20": 0.844, + "recall@20": 0.9285, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 22.633173799025826, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 20.66802700026892, + "default": false, + "all@5": 0.308, + "recall@5": 0.6325, + "n_ret@5": 5.0, + "all@10": 0.396, + "recall@10": 0.691, + "n_ret@10": 10.0, + "all@20": 0.47, + "recall@20": 0.7335, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 19.363637599279173, + "default": false, + "all@5": 0.52, + "recall@5": 0.7605, + "n_ret@5": 5.0, + "all@10": 0.742, + "recall@10": 0.8735, + "n_ret@10": 9.144, + "all@20": 0.812, + "recall@20": 0.896, + "n_ret@20": 13.876 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 19.331212200573646, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.778, + "recall@20": 0.901, + "n_ret@20": 19.55 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 18.328255600761622, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.678, + "recall@20": 0.8575, + "n_ret@20": 19.928 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.3, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 5.365943400422111, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 18.60048479947727, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.694, + "recall@10": 0.8495, + "n_ret@10": 9.144, + "all@20": 0.806, + "recall@20": 0.8945, + "n_ret@20": 13.876 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 17.48487380077131, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.848, + "recall@20": 0.9215, + "n_ret@20": 17.33 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 20.60179799841717, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.748, + "recall@20": 0.8915, + "n_ret@20": 19.928 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 19.376735400757752, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.768, + "recall@20": 0.8905, + "n_ret@20": 18.858 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 18.15993940003682, + "default": false, + "all@5": 0.538, + "recall@5": 0.7705, + "n_ret@5": 5.0, + "all@10": 0.658, + "recall@10": 0.8465, + "n_ret@10": 10.0, + "all@20": 0.858, + "recall@20": 0.9255, + "n_ret@20": 15.672 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 19.20221100037452, + "default": false, + "all@5": 0.308, + "recall@5": 0.6325, + "n_ret@5": 5.0, + "all@10": 0.396, + "recall@10": 0.691, + "n_ret@10": 10.0, + "all@20": 0.786, + "recall@20": 0.906, + "n_ret@20": 19.55 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 4.46271779993549, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.874, + "recall@20": 0.9375, + "n_ret@20": 19.52 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 19.827189400210045, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.744, + "recall@20": 0.8805, + "n_ret@20": 19.214 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 4.577445201342925, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.86, + "recall@20": 0.9345, + "n_ret@20": 19.824 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 19.69713860214688, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 19.193523600464687, + "default": false, + "all@5": 0.524, + "recall@5": 0.763, + "n_ret@5": 5.0, + "all@10": 0.644, + "recall@10": 0.8395, + "n_ret@10": 10.0, + "all@20": 0.744, + "recall@20": 0.892, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 21.16207940049935, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.85, + "recall@20": 0.9225, + "n_ret@20": 19.214 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 4.0852350008208305, + "default": false, + "all@5": 0.656, + "recall@5": 0.8335, + "n_ret@5": 5.0, + "all@10": 0.804, + "recall@10": 0.892, + "n_ret@10": 7.374, + "all@20": 0.808, + "recall@20": 0.893, + "n_ret@20": 7.534 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 22.1025869990699, + "default": false, + "all@5": 0.352, + "recall@5": 0.6545, + "n_ret@5": 5.0, + "all@10": 0.802, + "recall@10": 0.892, + "n_ret@10": 7.804, + "all@20": 0.808, + "recall@20": 0.894, + "n_ret@20": 8.38 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 25.267256799619645, + "default": false, + "all@5": 0.408, + "recall@5": 0.6875, + "n_ret@5": 5.0, + "all@10": 0.496, + "recall@10": 0.749, + "n_ret@10": 10.0, + "all@20": 0.8, + "recall@20": 0.912, + "n_ret@20": 19.52 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 26.17599739995785, + "default": false, + "all@5": 0.308, + "recall@5": 0.6325, + "n_ret@5": 5.0, + "all@10": 0.396, + "recall@10": 0.691, + "n_ret@10": 10.0, + "all@20": 0.696, + "recall@20": 0.867, + "n_ret@20": 19.928 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 21.590061198105104, + "default": false, + "all@5": 0.538, + "recall@5": 0.7705, + "n_ret@5": 5.0, + "all@10": 0.658, + "recall@10": 0.846, + "n_ret@10": 10.0, + "all@20": 0.75, + "recall@20": 0.8945, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 21.762394599732943, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.758, + "recall@10": 0.876, + "n_ret@10": 9.144, + "all@20": 0.814, + "recall@20": 0.897, + "n_ret@20": 13.876 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 4.014102199231274, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.804, + "recall@10": 0.892, + "n_ret@10": 7.374, + "all@20": 0.808, + "recall@20": 0.893, + "n_ret@20": 7.534 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 18.031143398373388, + "default": false, + "all@5": 0.308, + "recall@5": 0.6325, + "n_ret@5": 5.0, + "all@10": 0.396, + "recall@10": 0.691, + "n_ret@10": 10.0, + "all@20": 0.47, + "recall@20": 0.7335, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 4.007800999679603, + "default": false, + "all@5": 0.442, + "recall@5": 0.7075, + "n_ret@5": 5.0, + "all@10": 0.804, + "recall@10": 0.893, + "n_ret@10": 7.804, + "all@20": 0.808, + "recall@20": 0.894, + "n_ret@20": 8.38 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 20.007408199016936, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.746, + "recall@10": 0.87, + "n_ret@10": 8.55, + "all@20": 0.808, + "recall@20": 0.8945, + "n_ret@20": 9.8 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 19.58132479933556, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.784, + "recall@20": 0.908, + "n_ret@20": 19.824 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 22.558195198653266, + "default": false, + "all@5": 0.352, + "recall@5": 0.6545, + "n_ret@5": 5.0, + "all@10": 0.462, + "recall@10": 0.726, + "n_ret@10": 10.0, + "all@20": 0.568, + "recall@20": 0.79, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 4.7456554003292695, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.858, + "recall@20": 0.927, + "n_ret@20": 18.858 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 18.8855358005967, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 4.829060400952585, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.874, + "recall@20": 0.9375, + "n_ret@20": 19.55 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 4.597480601049028, + "default": false, + "all@5": 0.642, + "recall@5": 0.8265, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.8785, + "n_ret@10": 10.0, + "all@20": 0.838, + "recall@20": 0.9265, + "n_ret@20": 19.928 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 24.604974799556658, + "default": false, + "all@5": 0.308, + "recall@5": 0.6325, + "n_ret@5": 5.0, + "all@10": 0.396, + "recall@10": 0.691, + "n_ret@10": 10.0, + "all@20": 0.856, + "recall@20": 0.924, + "n_ret@20": 16.072 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 35.787782199447975, + "default": false, + "all@5": 0.308, + "recall@5": 0.6325, + "n_ret@5": 5.0, + "all@10": 0.396, + "recall@10": 0.691, + "n_ret@10": 10.0, + "all@20": 0.86, + "recall@20": 0.9255, + "n_ret@20": 14.484 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 8.08873200067319, + "default": false, + "all@5": 0.642, + "recall@5": 0.8265, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.8785, + "n_ret@10": 10.0, + "all@20": 0.828, + "recall@20": 0.9225, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 20.957857200410217, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.768, + "recall@10": 0.8795, + "n_ret@10": 9.144, + "all@20": 0.814, + "recall@20": 0.897, + "n_ret@20": 13.876 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 17.94206299923826, + "default": false, + "all@5": 0.538, + "recall@5": 0.7705, + "n_ret@5": 5.0, + "all@10": 0.658, + "recall@10": 0.8465, + "n_ret@10": 10.0, + "all@20": 0.75, + "recall@20": 0.8945, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 3.800366398296319, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.806, + "recall@10": 0.894, + "n_ret@10": 7.932, + "all@20": 0.81, + "recall@20": 0.895, + "n_ret@20": 8.874 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 25.146082199411467, + "default": false, + "all@5": 0.308, + "recall@5": 0.6325, + "n_ret@5": 5.0, + "all@10": 0.396, + "recall@10": 0.691, + "n_ret@10": 10.0, + "all@20": 0.47, + "recall@20": 0.7335, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 33.959542001714, + "default": false, + "all@5": 0.538, + "recall@5": 0.7705, + "n_ret@5": 5.0, + "all@10": 0.658, + "recall@10": 0.846, + "n_ret@10": 10.0, + "all@20": 0.858, + "recall@20": 0.9255, + "n_ret@20": 16.072 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.828612600453198, + "default": false, + "all@5": 0.442, + "recall@5": 0.7075, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.784, + "n_ret@10": 10.0, + "all@20": 0.646, + "recall@20": 0.8455, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 7.687146000331268, + "default": false, + "all@5": 0.642, + "recall@5": 0.8255, + "n_ret@5": 5.0, + "all@10": 0.764, + "recall@10": 0.8785, + "n_ret@10": 9.226, + "all@20": 0.794, + "recall@20": 0.8885, + "n_ret@20": 15.262 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 32.258046199567616, + "default": false, + "all@5": 0.352, + "recall@5": 0.6545, + "n_ret@5": 5.0, + "all@10": 0.462, + "recall@10": 0.726, + "n_ret@10": 10.0, + "all@20": 0.852, + "recall@20": 0.924, + "n_ret@20": 19.214 + } + ], + "lib_selected": { + "5": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 3, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + } + }, + "seconds": 1397.5198228359222 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/2wiki__minilm__mention.npz b/benchmarks/multihop/results/raw/2wiki__minilm__mention.npz new file mode 100644 index 0000000..172dd1a Binary files /dev/null and b/benchmarks/multihop/results/raw/2wiki__minilm__mention.npz differ diff --git a/benchmarks/multihop/results/raw/2wiki__minilm__mention_entity.json b/benchmarks/multihop/results/raw/2wiki__minilm__mention_entity.json new file mode 100644 index 0000000..f08129a --- /dev/null +++ b/benchmarks/multihop/results/raw/2wiki__minilm__mention_entity.json @@ -0,0 +1,4355 @@ +{ + "dataset": "2wiki", + "model": "minilm", + "variant": "mention+entity", + "families": [ + "mention", + "entity" + ], + "diagnostics": { + "n_edges_undirected_union": 31907, + "n_edges_by_family_directed_or_undirected": 36355, + "mean_degree": 6.907024569758632, + "median_degree": 4.0, + "max_degree": 86, + "isolated_frac": 0.17761662517588483, + "gold_pairs_directly_linked": 0.45899513776337114, + "gold_pairs_linked_within_2": 0.46612641815235006, + "neighbours_of_gold_that_are_in_same_question_context": 0.1263623088924525 + }, + "ppr_dev_grid": [ + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.304, + "recall@5": 0.6305, + "all@10": 0.7, + "recall@10": 0.8485, + "all@20": 0.788, + "recall@20": 0.888 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.304, + "recall@5": 0.6305, + "all@10": 0.732, + "recall@10": 0.868, + "all@20": 0.82, + "recall@20": 0.902 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.302, + "recall@5": 0.6295, + "all@10": 0.698, + "recall@10": 0.848, + "all@20": 0.79, + "recall@20": 0.888 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.302, + "recall@5": 0.6295, + "all@10": 0.724, + "recall@10": 0.8605, + "all@20": 0.816, + "recall@20": 0.9 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.298, + "recall@5": 0.6275, + "all@10": 0.674, + "recall@10": 0.8425, + "all@20": 0.764, + "recall@20": 0.893 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.298, + "recall@5": 0.6275, + "all@10": 0.716, + "recall@10": 0.862, + "all@20": 0.788, + "recall@20": 0.9035 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.298, + "recall@5": 0.6275, + "all@10": 0.644, + "recall@10": 0.827, + "all@20": 0.748, + "recall@20": 0.8855 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.3, + "recall@5": 0.6285, + "all@10": 0.688, + "recall@10": 0.848, + "all@20": 0.768, + "recall@20": 0.894 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.492, + "recall@10": 0.755, + "all@20": 0.584, + "recall@20": 0.807 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.514, + "recall@10": 0.7665, + "all@20": 0.596, + "recall@20": 0.8145 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.49, + "recall@10": 0.754, + "all@20": 0.578, + "recall@20": 0.804 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.504, + "recall@10": 0.7595, + "all@20": 0.592, + "recall@20": 0.811 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.402, + "recall@10": 0.695, + "all@20": 0.454, + "recall@20": 0.731 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.406, + "recall@10": 0.6975, + "all@20": 0.462, + "recall@20": 0.7355 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.4, + "recall@10": 0.694, + "all@20": 0.452, + "recall@20": 0.729 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.6265, + "all@10": 0.404, + "recall@10": 0.6965, + "all@20": 0.456, + "recall@20": 0.7315 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.306, + "recall@5": 0.626, + "all@10": 0.384, + "recall@10": 0.686, + "all@20": 0.756, + "recall@20": 0.885 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.318, + "recall@5": 0.6295, + "all@10": 0.388, + "recall@10": 0.688, + "all@20": 0.79, + "recall@20": 0.8995 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.306, + "recall@5": 0.626, + "all@10": 0.382, + "recall@10": 0.6845, + "all@20": 0.75, + "recall@20": 0.882 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.31, + "recall@5": 0.626, + "all@10": 0.382, + "recall@10": 0.6845, + "all@20": 0.792, + "recall@20": 0.898 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.312, + "recall@5": 0.6325, + "all@10": 0.376, + "recall@10": 0.681, + "all@20": 0.75, + "recall@20": 0.883 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.318, + "recall@5": 0.6335, + "all@10": 0.378, + "recall@10": 0.682, + "all@20": 0.778, + "recall@20": 0.896 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.314, + "recall@5": 0.6315, + "all@10": 0.378, + "recall@10": 0.682, + "all@20": 0.74, + "recall@20": 0.8775 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.316, + "recall@5": 0.634, + "all@10": 0.378, + "recall@10": 0.682, + "all@20": 0.758, + "recall@20": 0.8865 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.318, + "recall@5": 0.634, + "all@10": 0.374, + "recall@10": 0.68, + "all@20": 0.58, + "recall@20": 0.803 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.324, + "recall@5": 0.6385, + "all@10": 0.374, + "recall@10": 0.68, + "all@20": 0.602, + "recall@20": 0.815 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.312, + "recall@5": 0.6325, + "all@10": 0.374, + "recall@10": 0.68, + "all@20": 0.576, + "recall@20": 0.801 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.316, + "recall@5": 0.6355, + "all@10": 0.374, + "recall@10": 0.68, + "all@20": 0.598, + "recall@20": 0.8125 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.316, + "recall@5": 0.6355, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.456, + "recall@20": 0.7315 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.314, + "recall@5": 0.6355, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.462, + "recall@20": 0.735 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.316, + "recall@5": 0.6355, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.456, + "recall@20": 0.731 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.318, + "recall@5": 0.636, + "all@10": 0.372, + "recall@10": 0.679, + "all@20": 0.456, + "recall@20": 0.731 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.29, + "recall@5": 0.608, + "all@10": 0.398, + "recall@10": 0.69, + "all@20": 0.436, + "recall@20": 0.718 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.29, + "recall@5": 0.606, + "all@10": 0.402, + "recall@10": 0.691, + "all@20": 0.44, + "recall@20": 0.72 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.28, + "recall@5": 0.6005, + "all@10": 0.392, + "recall@10": 0.685, + "all@20": 0.436, + "recall@20": 0.717 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.286, + "recall@5": 0.604, + "all@10": 0.396, + "recall@10": 0.688, + "all@20": 0.436, + "recall@20": 0.717 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.31, + "recall@5": 0.6245, + "all@10": 0.4, + "recall@10": 0.6915, + "all@20": 0.43, + "recall@20": 0.714 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.322, + "recall@5": 0.628, + "all@10": 0.398, + "recall@10": 0.691, + "all@20": 0.432, + "recall@20": 0.715 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.312, + "recall@5": 0.6265, + "all@10": 0.396, + "recall@10": 0.6885, + "all@20": 0.426, + "recall@20": 0.7115 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.312, + "recall@5": 0.6285, + "all@10": 0.394, + "recall@10": 0.688, + "all@20": 0.43, + "recall@20": 0.7135 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.32, + "recall@5": 0.633, + "all@10": 0.396, + "recall@10": 0.69, + "all@20": 0.424, + "recall@20": 0.7105 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.328, + "recall@5": 0.638, + "all@10": 0.396, + "recall@10": 0.6895, + "all@20": 0.426, + "recall@20": 0.7115 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.312, + "recall@5": 0.6315, + "all@10": 0.396, + "recall@10": 0.6905, + "all@20": 0.424, + "recall@20": 0.7105 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.318, + "recall@5": 0.6345, + "all@10": 0.4, + "recall@10": 0.6925, + "all@20": 0.424, + "recall@20": 0.7105 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.318, + "recall@5": 0.6365, + "all@10": 0.392, + "recall@10": 0.6885, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.318, + "recall@5": 0.637, + "all@10": 0.392, + "recall@10": 0.689, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.314, + "recall@5": 0.634, + "all@10": 0.394, + "recall@10": 0.6895, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.32, + "recall@5": 0.6375, + "all@10": 0.396, + "recall@10": 0.6905, + "all@20": 0.42, + "recall@20": 0.7085 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.364, + "recall@5": 0.6715, + "all@10": 0.688, + "recall@10": 0.8435, + "all@20": 0.776, + "recall@20": 0.8835 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.376, + "recall@5": 0.678, + "all@10": 0.714, + "recall@10": 0.8575, + "all@20": 0.81, + "recall@20": 0.8985 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.36, + "recall@5": 0.668, + "all@10": 0.686, + "recall@10": 0.8415, + "all@20": 0.782, + "recall@20": 0.8865 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.364, + "recall@5": 0.67, + "all@10": 0.722, + "recall@10": 0.8595, + "all@20": 0.808, + "recall@20": 0.897 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.332, + "recall@5": 0.6525, + "all@10": 0.68, + "recall@10": 0.8455, + "all@20": 0.782, + "recall@20": 0.897 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.336, + "recall@5": 0.654, + "all@10": 0.71, + "recall@10": 0.862, + "all@20": 0.804, + "recall@20": 0.908 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.334, + "recall@5": 0.652, + "all@10": 0.666, + "recall@10": 0.838, + "all@20": 0.774, + "recall@20": 0.8925 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.338, + "recall@5": 0.656, + "all@10": 0.7, + "recall@10": 0.8535, + "all@20": 0.786, + "recall@20": 0.899 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.308, + "recall@5": 0.634, + "all@10": 0.546, + "recall@10": 0.7825, + "all@20": 0.62, + "recall@20": 0.827 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.312, + "recall@5": 0.637, + "all@10": 0.564, + "recall@10": 0.7925, + "all@20": 0.646, + "recall@20": 0.8395 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.312, + "recall@5": 0.6365, + "all@10": 0.544, + "recall@10": 0.782, + "all@20": 0.62, + "recall@20": 0.827 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.316, + "recall@5": 0.639, + "all@10": 0.562, + "recall@10": 0.79, + "all@20": 0.644, + "recall@20": 0.8385 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.298, + "recall@5": 0.628, + "all@10": 0.406, + "recall@10": 0.701, + "all@20": 0.482, + "recall@20": 0.746 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.298, + "recall@5": 0.6275, + "all@10": 0.412, + "recall@10": 0.704, + "all@20": 0.496, + "recall@20": 0.7525 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.3, + "recall@5": 0.629, + "all@10": 0.41, + "recall@10": 0.7025, + "all@20": 0.476, + "recall@20": 0.743 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.298, + "recall@5": 0.6275, + "all@10": 0.412, + "recall@10": 0.704, + "all@20": 0.486, + "recall@20": 0.7475 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.33, + "recall@5": 0.6325, + "all@10": 0.456, + "recall@10": 0.734, + "all@20": 0.748, + "recall@20": 0.88 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.342, + "recall@5": 0.636, + "all@10": 0.474, + "recall@10": 0.744, + "all@20": 0.776, + "recall@20": 0.8935 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.32, + "recall@5": 0.6275, + "all@10": 0.454, + "recall@10": 0.7325, + "all@20": 0.748, + "recall@20": 0.879 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.322, + "recall@5": 0.6275, + "all@10": 0.476, + "recall@10": 0.7445, + "all@20": 0.772, + "recall@20": 0.8905 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.32, + "recall@5": 0.6355, + "all@10": 0.432, + "recall@10": 0.7175, + "all@20": 0.758, + "recall@20": 0.8845 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.336, + "recall@5": 0.6445, + "all@10": 0.434, + "recall@10": 0.7225, + "all@20": 0.794, + "recall@20": 0.901 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.328, + "recall@5": 0.64, + "all@10": 0.426, + "recall@10": 0.7165, + "all@20": 0.746, + "recall@20": 0.878 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.334, + "recall@5": 0.64, + "all@10": 0.438, + "recall@10": 0.7245, + "all@20": 0.77, + "recall@20": 0.8895 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.324, + "recall@5": 0.639, + "all@10": 0.406, + "recall@10": 0.7, + "all@20": 0.616, + "recall@20": 0.821 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.326, + "recall@5": 0.6375, + "all@10": 0.412, + "recall@10": 0.703, + "all@20": 0.644, + "recall@20": 0.8355 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.32, + "recall@5": 0.634, + "all@10": 0.402, + "recall@10": 0.698, + "all@20": 0.622, + "recall@20": 0.8225 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.322, + "recall@5": 0.638, + "all@10": 0.412, + "recall@10": 0.7035, + "all@20": 0.64, + "recall@20": 0.832 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.312, + "recall@5": 0.633, + "all@10": 0.38, + "recall@10": 0.683, + "all@20": 0.478, + "recall@20": 0.744 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.312, + "recall@5": 0.632, + "all@10": 0.38, + "recall@10": 0.683, + "all@20": 0.492, + "recall@20": 0.751 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.31, + "recall@5": 0.6315, + "all@10": 0.378, + "recall@10": 0.682, + "all@20": 0.47, + "recall@20": 0.74 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.312, + "recall@5": 0.6325, + "all@10": 0.38, + "recall@10": 0.683, + "all@20": 0.482, + "recall@20": 0.746 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.256, + "recall@5": 0.562, + "all@10": 0.42, + "recall@10": 0.697, + "all@20": 0.524, + "recall@20": 0.774 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.27, + "recall@5": 0.562, + "all@10": 0.426, + "recall@10": 0.7035, + "all@20": 0.538, + "recall@20": 0.782 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.246, + "recall@5": 0.5615, + "all@10": 0.414, + "recall@10": 0.692, + "all@20": 0.514, + "recall@20": 0.7695 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.246, + "recall@5": 0.5585, + "all@10": 0.426, + "recall@10": 0.698, + "all@20": 0.528, + "recall@20": 0.7775 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.288, + "recall@5": 0.6005, + "all@10": 0.424, + "recall@10": 0.704, + "all@20": 0.484, + "recall@20": 0.751 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.294, + "recall@5": 0.6035, + "all@10": 0.43, + "recall@10": 0.7085, + "all@20": 0.498, + "recall@20": 0.759 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.284, + "recall@5": 0.599, + "all@10": 0.416, + "recall@10": 0.703, + "all@20": 0.49, + "recall@20": 0.754 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.292, + "recall@5": 0.601, + "all@10": 0.42, + "recall@10": 0.7015, + "all@20": 0.502, + "recall@20": 0.761 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.304, + "recall@5": 0.6245, + "all@10": 0.414, + "recall@10": 0.701, + "all@20": 0.466, + "recall@20": 0.737 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.312, + "recall@5": 0.627, + "all@10": 0.414, + "recall@10": 0.6995, + "all@20": 0.47, + "recall@20": 0.7385 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.304, + "recall@5": 0.6235, + "all@10": 0.41, + "recall@10": 0.6995, + "all@20": 0.462, + "recall@20": 0.736 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.304, + "recall@5": 0.6255, + "all@10": 0.41, + "recall@10": 0.698, + "all@20": 0.46, + "recall@20": 0.735 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.314, + "recall@5": 0.6315, + "all@10": 0.388, + "recall@10": 0.687, + "all@20": 0.424, + "recall@20": 0.711 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.314, + "recall@5": 0.6325, + "all@10": 0.388, + "recall@10": 0.686, + "all@20": 0.43, + "recall@20": 0.714 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.312, + "recall@5": 0.631, + "all@10": 0.394, + "recall@10": 0.689, + "all@20": 0.428, + "recall@20": 0.713 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.308, + "recall@5": 0.628, + "all@10": 0.394, + "recall@10": 0.689, + "all@20": 0.432, + "recall@20": 0.715 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.394, + "recall@5": 0.685, + "all@10": 0.658, + "recall@10": 0.8305, + "all@20": 0.76, + "recall@20": 0.8765 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.414, + "recall@5": 0.692, + "all@10": 0.702, + "recall@10": 0.8515, + "all@20": 0.794, + "recall@20": 0.892 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.378, + "recall@5": 0.6745, + "all@10": 0.65, + "recall@10": 0.8245, + "all@20": 0.764, + "recall@20": 0.878 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.394, + "recall@5": 0.683, + "all@10": 0.688, + "recall@10": 0.8435, + "all@20": 0.8, + "recall@20": 0.8935 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.386, + "recall@5": 0.6855, + "all@10": 0.67, + "recall@10": 0.8395, + "all@20": 0.772, + "recall@20": 0.892 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.4, + "recall@5": 0.6925, + "all@10": 0.698, + "recall@10": 0.854, + "all@20": 0.8, + "recall@20": 0.9055 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.374, + "recall@5": 0.6775, + "all@10": 0.646, + "recall@10": 0.828, + "all@20": 0.778, + "recall@20": 0.893 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.388, + "recall@5": 0.684, + "all@10": 0.682, + "recall@10": 0.8435, + "all@20": 0.798, + "recall@20": 0.9025 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.328, + "recall@5": 0.649, + "all@10": 0.558, + "recall@10": 0.791, + "all@20": 0.644, + "recall@20": 0.837 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.342, + "recall@5": 0.657, + "all@10": 0.58, + "recall@10": 0.803, + "all@20": 0.68, + "recall@20": 0.8555 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.326, + "recall@5": 0.649, + "all@10": 0.56, + "recall@10": 0.79, + "all@20": 0.646, + "recall@20": 0.84 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.336, + "recall@5": 0.655, + "all@10": 0.58, + "recall@10": 0.8, + "all@20": 0.666, + "recall@20": 0.847 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.304, + "recall@5": 0.6315, + "all@10": 0.416, + "recall@10": 0.707, + "all@20": 0.494, + "recall@20": 0.754 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.308, + "recall@5": 0.634, + "all@10": 0.426, + "recall@10": 0.713, + "all@20": 0.496, + "recall@20": 0.756 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.306, + "recall@5": 0.633, + "all@10": 0.412, + "recall@10": 0.7045, + "all@20": 0.492, + "recall@20": 0.753 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.308, + "recall@5": 0.6335, + "all@10": 0.422, + "recall@10": 0.71, + "all@20": 0.498, + "recall@20": 0.756 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.298, + "recall@5": 0.5875, + "all@10": 0.49, + "recall@10": 0.748, + "all@20": 0.736, + "recall@20": 0.8735 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.314, + "recall@5": 0.593, + "all@10": 0.516, + "recall@10": 0.7585, + "all@20": 0.756, + "recall@20": 0.885 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.284, + "recall@5": 0.5865, + "all@10": 0.484, + "recall@10": 0.7465, + "all@20": 0.73, + "recall@20": 0.8705 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.302, + "recall@5": 0.585, + "all@10": 0.514, + "recall@10": 0.757, + "all@20": 0.758, + "recall@20": 0.8855 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.312, + "recall@5": 0.618, + "all@10": 0.464, + "recall@10": 0.74, + "all@20": 0.748, + "recall@20": 0.88 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.33, + "recall@5": 0.625, + "all@10": 0.482, + "recall@10": 0.7495, + "all@20": 0.768, + "recall@20": 0.89 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.306, + "recall@5": 0.62, + "all@10": 0.466, + "recall@10": 0.7395, + "all@20": 0.746, + "recall@20": 0.876 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.318, + "recall@5": 0.6235, + "all@10": 0.48, + "recall@10": 0.7475, + "all@20": 0.754, + "recall@20": 0.884 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.324, + "recall@5": 0.642, + "all@10": 0.424, + "recall@10": 0.716, + "all@20": 0.626, + "recall@20": 0.8255 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.334, + "recall@5": 0.643, + "all@10": 0.434, + "recall@10": 0.723, + "all@20": 0.66, + "recall@20": 0.8425 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.322, + "recall@5": 0.6395, + "all@10": 0.424, + "recall@10": 0.7165, + "all@20": 0.63, + "recall@20": 0.828 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.33, + "recall@5": 0.641, + "all@10": 0.434, + "recall@10": 0.722, + "all@20": 0.652, + "recall@20": 0.8385 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.318, + "recall@5": 0.6355, + "all@10": 0.388, + "recall@10": 0.6885, + "all@20": 0.486, + "recall@20": 0.7495 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.322, + "recall@5": 0.638, + "all@10": 0.392, + "recall@10": 0.691, + "all@20": 0.488, + "recall@20": 0.752 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.318, + "recall@5": 0.635, + "all@10": 0.39, + "recall@10": 0.6895, + "all@20": 0.484, + "recall@20": 0.749 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.32, + "recall@5": 0.636, + "all@10": 0.392, + "recall@10": 0.6905, + "all@20": 0.488, + "recall@20": 0.751 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.196, + "recall@5": 0.4525, + "all@10": 0.4, + "recall@10": 0.663, + "all@20": 0.554, + "recall@20": 0.7905 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.192, + "recall@5": 0.4455, + "all@10": 0.408, + "recall@10": 0.664, + "all@20": 0.588, + "recall@20": 0.8065 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.192, + "recall@5": 0.457, + "all@10": 0.404, + "recall@10": 0.671, + "all@20": 0.548, + "recall@20": 0.785 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.19, + "recall@5": 0.457, + "all@10": 0.39, + "recall@10": 0.659, + "all@20": 0.582, + "recall@20": 0.8035 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.232, + "recall@5": 0.52, + "all@10": 0.44, + "recall@10": 0.704, + "all@20": 0.536, + "recall@20": 0.7805 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.24, + "recall@5": 0.524, + "all@10": 0.446, + "recall@10": 0.7075, + "all@20": 0.544, + "recall@20": 0.7855 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.234, + "recall@5": 0.532, + "all@10": 0.42, + "recall@10": 0.697, + "all@20": 0.53, + "recall@20": 0.7755 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.23, + "recall@5": 0.5255, + "all@10": 0.428, + "recall@10": 0.7005, + "all@20": 0.548, + "recall@20": 0.7875 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.286, + "recall@5": 0.6075, + "all@10": 0.424, + "recall@10": 0.7075, + "all@20": 0.478, + "recall@20": 0.746 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.292, + "recall@5": 0.6055, + "all@10": 0.418, + "recall@10": 0.706, + "all@20": 0.492, + "recall@20": 0.754 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.278, + "recall@5": 0.6065, + "all@10": 0.424, + "recall@10": 0.709, + "all@20": 0.478, + "recall@20": 0.7485 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.29, + "recall@5": 0.6095, + "all@10": 0.418, + "recall@10": 0.7055, + "all@20": 0.478, + "recall@20": 0.75 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.31, + "recall@5": 0.6295, + "all@10": 0.392, + "recall@10": 0.689, + "all@20": 0.442, + "recall@20": 0.7225 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.31, + "recall@5": 0.629, + "all@10": 0.396, + "recall@10": 0.6905, + "all@20": 0.446, + "recall@20": 0.7245 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.31, + "recall@5": 0.6305, + "all@10": 0.4, + "recall@10": 0.693, + "all@20": 0.442, + "recall@20": 0.7225 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": true + }, + "all@5": 0.31, + "recall@5": 0.63, + "all@10": 0.404, + "recall@10": 0.6935, + "all@20": 0.44, + "recall@20": 0.722 + } + ], + "ppr_selected": { + "5": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "20": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + } + }, + "lib_dev_grid": [ + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.966423799400218, + "default": true, + "all@5": 0.52, + "recall@5": 0.7605, + "n_ret@5": 5.0, + "all@10": 0.702, + "recall@10": 0.8585, + "n_ret@10": 9.974, + "all@20": 0.77, + "recall@20": 0.887, + "n_ret@20": 19.76 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.012842399184592, + "default": false, + "all@5": 0.562, + "recall@5": 0.7855, + "n_ret@5": 5.0, + "all@10": 0.68, + "recall@10": 0.847, + "n_ret@10": 10.0, + "all@20": 0.786, + "recall@20": 0.9035, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 19.14798499934841, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 20.470075798919424, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.43, + "recall@20": 0.7135, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 36.63007439801004, + "default": false, + "all@5": 0.408, + "recall@5": 0.6875, + "n_ret@5": 5.0, + "all@10": 0.496, + "recall@10": 0.749, + "n_ret@10": 10.0, + "all@20": 0.58, + "recall@20": 0.805, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 19.8123777982546, + "default": false, + "all@5": 0.406, + "recall@5": 0.687, + "n_ret@5": 5.0, + "all@10": 0.496, + "recall@10": 0.749, + "n_ret@10": 10.0, + "all@20": 0.822, + "recall@20": 0.913, + "n_ret@20": 19.928 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 29.959460198762827, + "default": false, + "all@5": 0.408, + "recall@5": 0.6875, + "n_ret@5": 5.0, + "all@10": 0.496, + "recall@10": 0.749, + "n_ret@10": 10.0, + "all@20": 0.776, + "recall@20": 0.9015, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 20.106643800390884, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.602, + "recall@20": 0.816, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 35.180556799750775, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 4.847729599103332, + "default": false, + "all@5": 0.442, + "recall@5": 0.7075, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.784, + "n_ret@10": 10.0, + "all@20": 0.864, + "recall@20": 0.93, + "n_ret@20": 19.928 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 20.15974840137642, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.788, + "recall@20": 0.91, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 29.001779000856914, + "default": false, + "all@5": 0.482, + "recall@5": 0.7335, + "n_ret@5": 5.0, + "all@10": 0.61, + "recall@10": 0.818, + "n_ret@10": 10.0, + "all@20": 0.74, + "recall@20": 0.8905, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 43.72425979981199, + "default": false, + "all@5": 0.52, + "recall@5": 0.7605, + "n_ret@5": 5.0, + "all@10": 0.64, + "recall@10": 0.8385, + "n_ret@10": 10.0, + "all@20": 0.75, + "recall@20": 0.8945, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 19.517645999672823, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.756, + "recall@20": 0.891, + "n_ret@20": 19.928 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 4.0027255999157205, + "default": false, + "all@5": 0.562, + "recall@5": 0.7855, + "n_ret@5": 5.0, + "all@10": 0.694, + "recall@10": 0.848, + "n_ret@10": 9.924, + "all@20": 0.784, + "recall@20": 0.8895, + "n_ret@20": 18.24 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 37.21771519910544, + "default": false, + "all@5": 0.31, + "recall@5": 0.6335, + "n_ret@5": 5.0, + "all@10": 0.398, + "recall@10": 0.692, + "n_ret@10": 10.0, + "all@20": 0.666, + "recall@20": 0.8495, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 6.51433599984739, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.878, + "recall@20": 0.9395, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 27.569557201233692, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 6.473259598133154, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.802, + "recall@10": 0.8965, + "n_ret@10": 9.936, + "all@20": 0.828, + "recall@20": 0.9065, + "n_ret@20": 18.44 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 20.704877198208123, + "default": false, + "all@5": 0.35, + "recall@5": 0.6535, + "n_ret@5": 5.0, + "all@10": 0.68, + "recall@10": 0.8445, + "n_ret@10": 9.974, + "all@20": 0.786, + "recall@20": 0.8925, + "n_ret@20": 19.76 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 21.296663998509757, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.758, + "recall@20": 0.892, + "n_ret@20": 19.92 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 30.227872998919338, + "default": false, + "all@5": 0.392, + "recall@5": 0.6785, + "n_ret@5": 5.0, + "all@10": 0.482, + "recall@10": 0.74, + "n_ret@10": 10.0, + "all@20": 0.576, + "recall@20": 0.802, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 39.77204239892308, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.658, + "recall@20": 0.8475, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 20.581112000043504, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.748, + "recall@10": 0.8765, + "n_ret@10": 9.936, + "all@20": 0.824, + "recall@20": 0.9055, + "n_ret@20": 18.44 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 18.96603259758558, + "default": false, + "all@5": 0.408, + "recall@5": 0.6875, + "n_ret@5": 5.0, + "all@10": 0.498, + "recall@10": 0.75, + "n_ret@10": 10.0, + "all@20": 0.78, + "recall@20": 0.9035, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 33.19710560084786, + "default": false, + "all@5": 0.644, + "recall@5": 0.8275, + "n_ret@5": 5.0, + "all@10": 0.738, + "recall@10": 0.8795, + "n_ret@10": 10.0, + "all@20": 0.83, + "recall@20": 0.9235, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 35.0065874006832, + "default": false, + "all@5": 0.524, + "recall@5": 0.763, + "n_ret@5": 5.0, + "all@10": 0.78, + "recall@10": 0.887, + "n_ret@10": 9.936, + "all@20": 0.828, + "recall@20": 0.9065, + "n_ret@20": 18.44 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 8.235507398843765, + "default": false, + "all@5": 0.564, + "recall@5": 0.786, + "n_ret@5": 5.0, + "all@10": 0.68, + "recall@10": 0.847, + "n_ret@10": 10.0, + "all@20": 0.786, + "recall@20": 0.9035, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 9.816510400618427, + "default": false, + "all@5": 0.564, + "recall@5": 0.7845, + "n_ret@5": 5.0, + "all@10": 0.662, + "recall@10": 0.8315, + "n_ret@10": 9.982, + "all@20": 0.73, + "recall@20": 0.865, + "n_ret@20": 19.812 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 22.185584398568608, + "default": false, + "all@5": 0.352, + "recall@5": 0.6545, + "n_ret@5": 5.0, + "all@10": 0.778, + "recall@10": 0.8865, + "n_ret@10": 9.982, + "all@20": 0.832, + "recall@20": 0.9085, + "n_ret@20": 19.812 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 19.504241601214744, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.756, + "recall@20": 0.8875, + "n_ret@20": 19.928 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 31.805959998397157, + "default": false, + "all@5": 0.308, + "recall@5": 0.6325, + "n_ret@5": 5.0, + "all@10": 0.396, + "recall@10": 0.691, + "n_ret@10": 10.0, + "all@20": 0.708, + "recall@20": 0.8725, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 9.594706201227382, + "default": false, + "all@5": 0.564, + "recall@5": 0.786, + "n_ret@5": 5.0, + "all@10": 0.68, + "recall@10": 0.847, + "n_ret@10": 10.0, + "all@20": 0.784, + "recall@20": 0.9025, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 34.21518659917638, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.66, + "recall@20": 0.849, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 42.463042800081894, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.672, + "recall@20": 0.8475, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 44.441912400652654, + "default": false, + "all@5": 0.538, + "recall@5": 0.7705, + "n_ret@5": 5.0, + "all@10": 0.66, + "recall@10": 0.8475, + "n_ret@10": 10.0, + "all@20": 0.866, + "recall@20": 0.9295, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 21.459048800868914, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.776, + "recall@20": 0.9095, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 17.090781399863772, + "default": false, + "all@5": 0.442, + "recall@5": 0.7075, + "n_ret@5": 5.0, + "all@10": 0.798, + "recall@10": 0.895, + "n_ret@10": 9.982, + "all@20": 0.826, + "recall@20": 0.9055, + "n_ret@20": 19.812 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 65.66136440110859, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 32.898643601336516, + "default": false, + "all@5": 0.486, + "recall@5": 0.7355, + "n_ret@5": 5.0, + "all@10": 0.62, + "recall@10": 0.826, + "n_ret@10": 10.0, + "all@20": 0.738, + "recall@20": 0.889, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 48.76754999964032, + "default": false, + "all@5": 0.352, + "recall@5": 0.6545, + "n_ret@5": 5.0, + "all@10": 0.464, + "recall@10": 0.727, + "n_ret@10": 10.0, + "all@20": 0.568, + "recall@20": 0.79, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 23.73608340078499, + "default": false, + "all@5": 0.564, + "recall@5": 0.7845, + "n_ret@5": 5.0, + "all@10": 0.654, + "recall@10": 0.831, + "n_ret@10": 10.0, + "all@20": 0.73, + "recall@20": 0.8725, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 29.91011100052856, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 48.714223998831585, + "default": false, + "all@5": 0.406, + "recall@5": 0.687, + "n_ret@5": 5.0, + "all@10": 0.494, + "recall@10": 0.748, + "n_ret@10": 10.0, + "all@20": 0.694, + "recall@20": 0.8675, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.89089799800422, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.906, + "n_ret@20": 19.92 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 29.633629400283098, + "default": false, + "all@5": 0.308, + "recall@5": 0.6325, + "n_ret@5": 5.0, + "all@10": 0.396, + "recall@10": 0.691, + "n_ret@10": 10.0, + "all@20": 0.47, + "recall@20": 0.7335, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 40.1897517995676, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.73, + "recall@20": 0.882, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 18.223574801231734, + "default": false, + "all@5": 0.56, + "recall@5": 0.7835, + "n_ret@5": 5.0, + "all@10": 0.654, + "recall@10": 0.8305, + "n_ret@10": 10.0, + "all@20": 0.732, + "recall@20": 0.873, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 46.399829998612404, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 12.395715800463222, + "default": false, + "all@5": 0.654, + "recall@5": 0.833, + "n_ret@5": 5.0, + "all@10": 0.806, + "recall@10": 0.897, + "n_ret@10": 9.974, + "all@20": 0.828, + "recall@20": 0.9055, + "n_ret@20": 19.76 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 44.34080200083554, + "default": false, + "all@5": 0.52, + "recall@5": 0.7605, + "n_ret@5": 5.0, + "all@10": 0.64, + "recall@10": 0.8385, + "n_ret@10": 10.0, + "all@20": 0.848, + "recall@20": 0.9235, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 18.830230798223056, + "default": false, + "all@5": 0.436, + "recall@5": 0.704, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.783, + "n_ret@10": 10.0, + "all@20": 0.774, + "recall@20": 0.908, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 28.683650199207477, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.79, + "recall@20": 0.9005, + "n_ret@20": 19.92 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 32.63095640181564, + "default": false, + "all@5": 0.308, + "recall@5": 0.6325, + "n_ret@5": 5.0, + "all@10": 0.734, + "recall@10": 0.869, + "n_ret@10": 9.974, + "all@20": 0.804, + "recall@20": 0.8995, + "n_ret@20": 19.76 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 36.307534001767635, + "default": false, + "all@5": 0.308, + "recall@5": 0.6325, + "n_ret@5": 5.0, + "all@10": 0.686, + "recall@10": 0.847, + "n_ret@10": 9.982, + "all@20": 0.8, + "recall@20": 0.8965, + "n_ret@20": 19.812 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 38.023308399599046, + "default": false, + "all@5": 0.31, + "recall@5": 0.6335, + "n_ret@5": 5.0, + "all@10": 0.398, + "recall@10": 0.692, + "n_ret@10": 10.0, + "all@20": 0.66, + "recall@20": 0.845, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 28.979541798937134, + "default": false, + "all@5": 0.486, + "recall@5": 0.7355, + "n_ret@5": 5.0, + "all@10": 0.62, + "recall@10": 0.826, + "n_ret@10": 10.0, + "all@20": 0.776, + "recall@20": 0.9005, + "n_ret@20": 19.928 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 61.21226439985912, + "default": false, + "all@5": 0.352, + "recall@5": 0.6545, + "n_ret@5": 5.0, + "all@10": 0.462, + "recall@10": 0.726, + "n_ret@10": 10.0, + "all@20": 0.568, + "recall@20": 0.79, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 31.589760000933893, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.746, + "recall@20": 0.8905, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 31.130216998863034, + "default": false, + "all@5": 0.524, + "recall@5": 0.763, + "n_ret@5": 5.0, + "all@10": 0.646, + "recall@10": 0.8405, + "n_ret@10": 10.0, + "all@20": 0.86, + "recall@20": 0.9275, + "n_ret@20": 19.928 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 25.252191202598624, + "default": false, + "all@5": 0.442, + "recall@5": 0.7075, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.784, + "n_ret@10": 10.0, + "all@20": 0.862, + "recall@20": 0.9295, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 44.96502840030007, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 31.295503596775237, + "default": false, + "all@5": 0.308, + "recall@5": 0.6325, + "n_ret@5": 5.0, + "all@10": 0.396, + "recall@10": 0.691, + "n_ret@10": 10.0, + "all@20": 0.756, + "recall@20": 0.893, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 30.168131599202752, + "default": false, + "all@5": 0.396, + "recall@5": 0.6815, + "n_ret@5": 5.0, + "all@10": 0.49, + "recall@10": 0.7455, + "n_ret@10": 10.0, + "all@20": 0.586, + "recall@20": 0.807, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 36.80246040108614, + "default": false, + "all@5": 0.52, + "recall@5": 0.7605, + "n_ret@5": 5.0, + "all@10": 0.748, + "recall@10": 0.8785, + "n_ret@10": 9.982, + "all@20": 0.828, + "recall@20": 0.906, + "n_ret@20": 19.812 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 21.742863400373608, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 42.187493199133314, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.6, + "recall@20": 0.815, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 29.40930939966347, + "default": false, + "all@5": 0.406, + "recall@5": 0.687, + "n_ret@5": 5.0, + "all@10": 0.496, + "recall@10": 0.749, + "n_ret@10": 10.0, + "all@20": 0.734, + "recall@20": 0.885, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 35.25819939991925, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.55, + "recall@10": 0.774, + "n_ret@10": 9.982, + "all@20": 0.672, + "recall@20": 0.842, + "n_ret@20": 19.812 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 56.90940500004217, + "default": false, + "all@5": 0.308, + "recall@5": 0.6325, + "n_ret@5": 5.0, + "all@10": 0.396, + "recall@10": 0.691, + "n_ret@10": 10.0, + "all@20": 0.47, + "recall@20": 0.7335, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 49.08066220174078, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.746, + "recall@20": 0.8905, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 43.1585750002414, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.654, + "recall@20": 0.8385, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 25.848119399510324, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.43, + "recall@20": 0.7135, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.457869599224068, + "default": false, + "all@5": 0.436, + "recall@5": 0.704, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.783, + "n_ret@10": 10.0, + "all@20": 0.774, + "recall@20": 0.908, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 21.57506760081742, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.878, + "recall@20": 0.9395, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 26.52566220064182, + "default": false, + "all@5": 0.436, + "recall@5": 0.704, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.783, + "n_ret@10": 10.0, + "all@20": 0.646, + "recall@20": 0.8455, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 7.726412601186894, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.864, + "recall@20": 0.9365, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 27.848582598613575, + "default": false, + "all@5": 0.524, + "recall@5": 0.763, + "n_ret@5": 5.0, + "all@10": 0.646, + "recall@10": 0.8405, + "n_ret@10": 10.0, + "all@20": 0.746, + "recall@20": 0.893, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 27.826852201134898, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.854, + "recall@20": 0.9245, + "n_ret@20": 19.928 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 5.80705659952946, + "default": false, + "all@5": 0.658, + "recall@5": 0.8345, + "n_ret@5": 5.0, + "all@10": 0.818, + "recall@10": 0.901, + "n_ret@10": 9.924, + "all@20": 0.828, + "recall@20": 0.9055, + "n_ret@20": 18.24 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 40.517328599933535, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 31.230128000490367, + "default": false, + "all@5": 0.408, + "recall@5": 0.6875, + "n_ret@5": 5.0, + "all@10": 0.754, + "recall@10": 0.8785, + "n_ret@10": 9.974, + "all@20": 0.82, + "recall@20": 0.905, + "n_ret@20": 19.76 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 33.27708679949865, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.75, + "recall@10": 0.8775, + "n_ret@10": 9.982, + "all@20": 0.83, + "recall@20": 0.9075, + "n_ret@20": 19.812 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 7.102312802802771, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.818, + "recall@10": 0.901, + "n_ret@10": 9.924, + "all@20": 0.828, + "recall@20": 0.9055, + "n_ret@20": 18.24 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 10.334866199060343, + "default": false, + "all@5": 0.442, + "recall@5": 0.7075, + "n_ret@5": 5.0, + "all@10": 0.818, + "recall@10": 0.902, + "n_ret@10": 9.974, + "all@20": 0.826, + "recall@20": 0.9045, + "n_ret@20": 19.76 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 26.352435400127433, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.608, + "recall@10": 0.8065, + "n_ret@10": 9.936, + "all@20": 0.766, + "recall@20": 0.88, + "n_ret@20": 18.44 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 27.5895792003721, + "default": false, + "all@5": 0.564, + "recall@5": 0.7845, + "n_ret@5": 5.0, + "all@10": 0.654, + "recall@10": 0.831, + "n_ret@10": 10.0, + "all@20": 0.728, + "recall@20": 0.8715, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.894680000958033, + "default": false, + "all@5": 0.436, + "recall@5": 0.704, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.783, + "n_ret@10": 10.0, + "all@20": 0.792, + "recall@20": 0.9045, + "n_ret@20": 19.92 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 17.302491800510325, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.796, + "recall@20": 0.907, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 40.47926979896147, + "default": false, + "all@5": 0.478, + "recall@5": 0.731, + "n_ret@5": 5.0, + "all@10": 0.602, + "recall@10": 0.814, + "n_ret@10": 10.0, + "all@20": 0.736, + "recall@20": 0.8895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 41.460269799106754, + "default": false, + "all@5": 0.31, + "recall@5": 0.6335, + "n_ret@5": 5.0, + "all@10": 0.398, + "recall@10": 0.692, + "n_ret@10": 10.0, + "all@20": 0.62, + "recall@20": 0.8265, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 27.984650400234386, + "default": false, + "all@5": 0.308, + "recall@5": 0.6325, + "n_ret@5": 5.0, + "all@10": 0.396, + "recall@10": 0.691, + "n_ret@10": 10.0, + "all@20": 0.822, + "recall@20": 0.912, + "n_ret@20": 19.92 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 33.31021980068181, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.778, + "recall@10": 0.8865, + "n_ret@10": 9.982, + "all@20": 0.832, + "recall@20": 0.9085, + "n_ret@20": 19.812 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 26.80167599883862, + "default": false, + "all@5": 0.31, + "recall@5": 0.6335, + "n_ret@5": 5.0, + "all@10": 0.398, + "recall@10": 0.692, + "n_ret@10": 10.0, + "all@20": 0.628, + "recall@20": 0.8315, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 28.618761998368427, + "default": false, + "all@5": 0.308, + "recall@5": 0.6325, + "n_ret@5": 5.0, + "all@10": 0.396, + "recall@10": 0.691, + "n_ret@10": 10.0, + "all@20": 0.47, + "recall@20": 0.7335, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 46.150611399440095, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.42, + "recall@20": 0.7085, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": "w" + }, + "ms_per_query": 27.539099600864574, + "default": false, + "all@5": 0.406, + "recall@5": 0.687, + "n_ret@5": 5.0, + "all@10": 0.496, + "recall@10": 0.749, + "n_ret@10": 10.0, + "all@20": 0.58, + "recall@20": 0.805, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 17.4855618000729, + "default": false, + "all@5": 0.436, + "recall@5": 0.704, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.783, + "n_ret@10": 10.0, + "all@20": 0.792, + "recall@20": 0.905, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 27.346062599448487, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.67, + "recall@20": 0.8475, + "n_ret@20": 19.928 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 7.939108799910173, + "default": false, + "all@5": 0.442, + "recall@5": 0.7075, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.784, + "n_ret@10": 10.0, + "all@20": 0.878, + "recall@20": 0.9395, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 38.889687999733724, + "default": false, + "all@5": 0.296, + "recall@5": 0.6265, + "n_ret@5": 5.0, + "all@10": 0.372, + "recall@10": 0.679, + "n_ret@10": 10.0, + "all@20": 0.554, + "recall@20": 0.7875, + "n_ret@20": 20.0 + } + ], + "lib_selected": { + "5": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "10": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "20": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + } + }, + "seconds": 3791.3357696533203 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/2wiki__minilm__mention_entity.npz b/benchmarks/multihop/results/raw/2wiki__minilm__mention_entity.npz new file mode 100644 index 0000000..a4b68d2 Binary files /dev/null and b/benchmarks/multihop/results/raw/2wiki__minilm__mention_entity.npz differ diff --git a/benchmarks/multihop/results/raw/2wiki__minilm__rerank_scores.npy b/benchmarks/multihop/results/raw/2wiki__minilm__rerank_scores.npy new file mode 100644 index 0000000..4f92cd3 Binary files /dev/null and b/benchmarks/multihop/results/raw/2wiki__minilm__rerank_scores.npy differ diff --git a/benchmarks/multihop/results/raw/2wiki__minilm__vector_top50.npy b/benchmarks/multihop/results/raw/2wiki__minilm__vector_top50.npy new file mode 100644 index 0000000..db9482c Binary files /dev/null and b/benchmarks/multihop/results/raw/2wiki__minilm__vector_top50.npy differ diff --git a/benchmarks/multihop/results/raw/hotpotqa__bge__baselines.json b/benchmarks/multihop/results/raw/hotpotqa__bge__baselines.json new file mode 100644 index 0000000..ecf52bc --- /dev/null +++ b/benchmarks/multihop/results/raw/hotpotqa__bge__baselines.json @@ -0,0 +1,8 @@ +{ + "dataset": "hotpotqa", + "model": "bge", + "corpus_size": 14570, + "n_questions": 1500, + "lib_similarity_search_same_top20_set": "200/200", + "seconds": 68.6482903957367 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/hotpotqa__bge__baselines.npz b/benchmarks/multihop/results/raw/hotpotqa__bge__baselines.npz new file mode 100644 index 0000000..8670032 Binary files /dev/null and b/benchmarks/multihop/results/raw/hotpotqa__bge__baselines.npz differ diff --git a/benchmarks/multihop/results/raw/hotpotqa__bge__bm25_top50.npy b/benchmarks/multihop/results/raw/hotpotqa__bge__bm25_top50.npy new file mode 100644 index 0000000..895a397 Binary files /dev/null and b/benchmarks/multihop/results/raw/hotpotqa__bge__bm25_top50.npy differ diff --git a/benchmarks/multihop/results/raw/hotpotqa__bge__ctrl_mention.json b/benchmarks/multihop/results/raw/hotpotqa__bge__ctrl_mention.json new file mode 100644 index 0000000..5b7ee5d --- /dev/null +++ b/benchmarks/multihop/results/raw/hotpotqa__bge__ctrl_mention.json @@ -0,0 +1,3274 @@ +{ + "dataset": "hotpotqa", + "model": "bge", + "variant": "ctrl_mention", + "families": [ + "mention_rand" + ], + "diagnostics": { + "n_edges_undirected_union": 11348, + "n_edges_by_family_directed_or_undirected": 12354, + "mean_degree": 1.5577213452299246, + "median_degree": 1.0, + "max_degree": 29, + "isolated_frac": 0.39958819492107067, + "gold_pairs_directly_linked": 0.0, + "gold_pairs_linked_within_2": 0.0, + "neighbours_of_gold_that_are_in_same_question_context": 0.00026518164942985947 + }, + "ppr_dev_grid": [ + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.606, + "recall@5": 0.788, + "all@10": 0.608, + "recall@10": 0.789, + "all@20": 0.608, + "recall@20": 0.789 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.606, + "recall@5": 0.788, + "all@10": 0.608, + "recall@10": 0.789, + "all@20": 0.608, + "recall@20": 0.789 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.608, + "recall@5": 0.789, + "all@10": 0.646, + "recall@10": 0.811, + "all@20": 0.78, + "recall@20": 0.888 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.606, + "recall@5": 0.788, + "all@10": 0.646, + "recall@10": 0.812, + "all@20": 0.772, + "recall@20": 0.884 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.608, + "recall@5": 0.789, + "all@10": 0.722, + "recall@10": 0.853, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.608, + "recall@5": 0.789, + "all@10": 0.724, + "recall@10": 0.854, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.608, + "recall@5": 0.789, + "all@10": 0.736, + "recall@10": 0.862, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.608, + "recall@5": 0.789, + "all@10": 0.736, + "recall@10": 0.862, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.602, + "recall@5": 0.787, + "all@10": 0.736, + "recall@10": 0.862, + "all@20": 0.736, + "recall@20": 0.862 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.53, + "recall@5": 0.749, + "all@10": 0.736, + "recall@10": 0.862, + "all@20": 0.736, + "recall@20": 0.862 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.608, + "recall@5": 0.79, + "all@10": 0.736, + "recall@10": 0.862, + "all@20": 0.754, + "recall@20": 0.873 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.57, + "recall@5": 0.769, + "all@10": 0.736, + "recall@10": 0.862, + "all@20": 0.752, + "recall@20": 0.87 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.61, + "recall@5": 0.79, + "all@10": 0.736, + "recall@10": 0.862, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.594, + "recall@5": 0.779, + "all@10": 0.736, + "recall@10": 0.862, + "all@20": 0.79, + "recall@20": 0.893 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.606, + "recall@5": 0.788, + "all@10": 0.736, + "recall@10": 0.862, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.612, + "recall@5": 0.789, + "all@10": 0.736, + "recall@10": 0.862, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.58, + "recall@5": 0.776, + "all@10": 0.736, + "recall@10": 0.862, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.434, + "recall@5": 0.677, + "all@10": 0.682, + "recall@10": 0.83, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.588, + "recall@5": 0.78, + "all@10": 0.736, + "recall@10": 0.862, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.504, + "recall@5": 0.725, + "all@10": 0.714, + "recall@10": 0.849, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.6, + "recall@5": 0.785, + "all@10": 0.74, + "recall@10": 0.863, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.57, + "recall@5": 0.765, + "all@10": 0.722, + "recall@10": 0.853, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.608, + "recall@5": 0.789, + "all@10": 0.734, + "recall@10": 0.86, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.612, + "recall@5": 0.789, + "all@10": 0.734, + "recall@10": 0.859, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.574, + "recall@5": 0.771, + "all@10": 0.608, + "recall@10": 0.789, + "all@20": 0.608, + "recall@20": 0.789 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.492, + "recall@5": 0.72, + "all@10": 0.608, + "recall@10": 0.789, + "all@20": 0.608, + "recall@20": 0.789 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.592, + "recall@5": 0.78, + "all@10": 0.638, + "recall@10": 0.807, + "all@20": 0.76, + "recall@20": 0.877 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.58, + "recall@5": 0.775, + "all@10": 0.628, + "recall@10": 0.802, + "all@20": 0.758, + "recall@20": 0.874 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.606, + "recall@5": 0.788, + "all@10": 0.704, + "recall@10": 0.843, + "all@20": 0.788, + "recall@20": 0.892 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.606, + "recall@5": 0.788, + "all@10": 0.698, + "recall@10": 0.84, + "all@20": 0.79, + "recall@20": 0.893 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.608, + "recall@5": 0.789, + "all@10": 0.736, + "recall@10": 0.862, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.608, + "recall@5": 0.789, + "all@10": 0.734, + "recall@10": 0.861, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.51, + "recall@5": 0.734, + "all@10": 0.722, + "recall@10": 0.853, + "all@20": 0.736, + "recall@20": 0.862 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.288, + "recall@5": 0.56, + "all@10": 0.656, + "recall@10": 0.817, + "all@20": 0.736, + "recall@20": 0.862 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.564, + "recall@5": 0.766, + "all@10": 0.726, + "recall@10": 0.855, + "all@20": 0.748, + "recall@20": 0.869 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.49, + "recall@5": 0.72, + "all@10": 0.71, + "recall@10": 0.848, + "all@20": 0.742, + "recall@20": 0.865 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.6, + "recall@5": 0.785, + "all@10": 0.736, + "recall@10": 0.862, + "all@20": 0.778, + "recall@20": 0.887 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.558, + "recall@5": 0.763, + "all@10": 0.734, + "recall@10": 0.861, + "all@20": 0.782, + "recall@20": 0.889 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.61, + "recall@5": 0.79, + "all@10": 0.736, + "recall@10": 0.862, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.602, + "recall@5": 0.784, + "all@10": 0.736, + "recall@10": 0.862, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.388, + "recall@5": 0.642, + "all@10": 0.686, + "recall@10": 0.835, + "all@20": 0.79, + "recall@20": 0.893 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.132, + "recall@5": 0.375, + "all@10": 0.412, + "recall@10": 0.657, + "all@20": 0.754, + "recall@20": 0.871 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.52, + "recall@5": 0.743, + "all@10": 0.718, + "recall@10": 0.852, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.324, + "recall@5": 0.59, + "all@10": 0.618, + "recall@10": 0.796, + "all@20": 0.782, + "recall@20": 0.888 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.58, + "recall@5": 0.775, + "all@10": 0.738, + "recall@10": 0.863, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.476, + "recall@5": 0.706, + "all@10": 0.702, + "recall@10": 0.843, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.6, + "recall@5": 0.785, + "all@10": 0.738, + "recall@10": 0.862, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.58, + "recall@5": 0.773, + "all@10": 0.726, + "recall@10": 0.855, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.472, + "recall@5": 0.697, + "all@10": 0.606, + "recall@10": 0.787, + "all@20": 0.608, + "recall@20": 0.789 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.28, + "recall@5": 0.536, + "all@10": 0.602, + "recall@10": 0.786, + "all@20": 0.608, + "recall@20": 0.789 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.532, + "recall@5": 0.745, + "all@10": 0.642, + "recall@10": 0.809, + "all@20": 0.754, + "recall@20": 0.873 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.444, + "recall@5": 0.682, + "all@10": 0.618, + "recall@10": 0.796, + "all@20": 0.73, + "recall@20": 0.859 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.58, + "recall@5": 0.774, + "all@10": 0.686, + "recall@10": 0.834, + "all@20": 0.786, + "recall@20": 0.891 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.582, + "recall@5": 0.776, + "all@10": 0.684, + "recall@10": 0.833, + "all@20": 0.786, + "recall@20": 0.891 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.608, + "recall@5": 0.789, + "all@10": 0.732, + "recall@10": 0.86, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.608, + "recall@5": 0.789, + "all@10": 0.73, + "recall@10": 0.859, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.294, + "recall@5": 0.543, + "all@10": 0.654, + "recall@10": 0.81, + "all@20": 0.736, + "recall@20": 0.861 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.048, + "recall@5": 0.274, + "all@10": 0.406, + "recall@10": 0.649, + "all@20": 0.734, + "recall@20": 0.861 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.426, + "recall@5": 0.667, + "all@10": 0.696, + "recall@10": 0.838, + "all@20": 0.748, + "recall@20": 0.869 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.198, + "recall@5": 0.458, + "all@10": 0.61, + "recall@10": 0.791, + "all@20": 0.738, + "recall@20": 0.863 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.56, + "recall@5": 0.764, + "all@10": 0.72, + "recall@10": 0.852, + "all@20": 0.776, + "recall@20": 0.885 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.5, + "recall@5": 0.731, + "all@10": 0.716, + "recall@10": 0.852, + "all@20": 0.774, + "recall@20": 0.885 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.606, + "recall@5": 0.788, + "all@10": 0.736, + "recall@10": 0.862, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.592, + "recall@5": 0.777, + "all@10": 0.734, + "recall@10": 0.861, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.289, + "all@10": 0.458, + "recall@10": 0.668, + "all@20": 0.768, + "recall@20": 0.881 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.016, + "recall@5": 0.14, + "all@10": 0.084, + "recall@10": 0.311, + "all@20": 0.542, + "recall@20": 0.743 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.252, + "recall@5": 0.48, + "all@10": 0.596, + "recall@10": 0.782, + "all@20": 0.786, + "recall@20": 0.891 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.066, + "recall@5": 0.258, + "all@10": 0.314, + "recall@10": 0.573, + "all@20": 0.724, + "recall@20": 0.854 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.516, + "recall@5": 0.741, + "all@10": 0.694, + "recall@10": 0.84, + "all@20": 0.792, + "recall@20": 0.894 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.376, + "recall@5": 0.637, + "all@10": 0.638, + "recall@10": 0.807, + "all@20": 0.784, + "recall@20": 0.89 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.598, + "recall@5": 0.784, + "all@10": 0.738, + "recall@10": 0.862, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.564, + "recall@5": 0.762, + "all@10": 0.722, + "recall@10": 0.852, + "all@20": 0.794, + "recall@20": 0.895 + } + ], + "ppr_selected": { + "5": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "20": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + } + }, + "lib_dev_grid": [ + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.31531220045872, + "default": true, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.608, + "recall@10": 0.789, + "n_ret@10": 9.072, + "all@20": 0.608, + "recall@20": 0.789, + "n_ret@20": 11.578 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 10.039590999367647, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.736, + "recall@20": 0.862, + "n_ret@20": 18.938 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 40.73271780007053, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.776, + "recall@20": 0.886, + "n_ret@20": 19.994 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 43.284435999114066, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 42.64406720187981, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.608, + "recall@10": 0.789, + "n_ret@10": 9.072, + "all@20": 0.608, + "recall@20": 0.789, + "n_ret@20": 11.578 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 43.22096939932089, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 8.003842401318252, + "default": false, + "all@5": 0.558, + "recall@5": 0.763, + "n_ret@5": 5.0, + "all@10": 0.594, + "recall@10": 0.782, + "n_ret@10": 9.874, + "all@20": 0.604, + "recall@20": 0.787, + "n_ret@20": 18.624 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 45.66481660027057, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.732, + "recall@10": 0.86, + "n_ret@10": 10.0, + "all@20": 0.792, + "recall@20": 0.894, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.11001839931123, + "default": false, + "all@5": 0.59, + "recall@5": 0.78, + "n_ret@5": 5.0, + "all@10": 0.712, + "recall@10": 0.849, + "n_ret@10": 10.0, + "all@20": 0.772, + "recall@20": 0.884, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.375466000055894, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.736, + "recall@20": 0.862, + "n_ret@20": 17.556 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 39.83734060020652, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 7.59590080101043, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.608, + "recall@10": 0.789, + "n_ret@10": 9.874, + "all@20": 0.608, + "recall@20": 0.789, + "n_ret@20": 18.624 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.718994401395321, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.776, + "recall@20": 0.886, + "n_ret@20": 19.994 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 35.01529299991671, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.608, + "recall@10": 0.789, + "n_ret@10": 9.298, + "all@20": 0.608, + "recall@20": 0.789, + "n_ret@20": 12.2 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 34.83533740066923, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.736, + "recall@20": 0.862, + "n_ret@20": 19.138 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 33.90688440098893, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.776, + "recall@20": 0.886, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 5.869834600365721, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.608, + "recall@10": 0.789, + "n_ret@10": 9.298, + "all@20": 0.608, + "recall@20": 0.789, + "n_ret@20": 12.2 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 34.3147689999314, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.732, + "recall@10": 0.86, + "n_ret@10": 10.0, + "all@20": 0.736, + "recall@20": 0.862, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 10.03717120049987, + "default": false, + "all@5": 0.558, + "recall@5": 0.763, + "n_ret@5": 5.0, + "all@10": 0.684, + "recall@10": 0.835, + "n_ret@10": 10.0, + "all@20": 0.758, + "recall@20": 0.877, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 6.7159284016815946, + "default": false, + "all@5": 0.57, + "recall@5": 0.77, + "n_ret@5": 5.0, + "all@10": 0.692, + "recall@10": 0.839, + "n_ret@10": 10.0, + "all@20": 0.728, + "recall@20": 0.858, + "n_ret@20": 19.266 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 33.193373802118, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.736, + "recall@20": 0.862, + "n_ret@20": 18.938 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 35.42327599879354, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.736, + "recall@20": 0.862, + "n_ret@20": 19.138 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 7.8944728015922, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.736, + "recall@20": 0.862, + "n_ret@20": 19.982 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 34.65090259979479, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.776, + "recall@20": 0.886, + "n_ret@20": 19.998 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 37.50085660000332, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.736, + "recall@20": 0.862, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 43.38657379837241, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 39.92434560076799, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.608, + "recall@10": 0.789, + "n_ret@10": 9.874, + "all@20": 0.608, + "recall@20": 0.789, + "n_ret@20": 18.624 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 39.485435600043274, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.776, + "recall@20": 0.886, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 37.908321599825285, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.736, + "recall@20": 0.862, + "n_ret@20": 19.982 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.877205999218859, + "default": false, + "all@5": 0.586, + "recall@5": 0.778, + "n_ret@5": 5.0, + "all@10": 0.712, + "recall@10": 0.849, + "n_ret@10": 10.0, + "all@20": 0.786, + "recall@20": 0.891, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 35.748755400883965, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 34.880461800144985, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.776, + "recall@20": 0.886, + "n_ret@20": 19.994 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.94953279895708, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.776, + "recall@20": 0.886, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 56.97445679991506, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 35.33631879824679, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.608, + "recall@10": 0.789, + "n_ret@10": 9.894, + "all@20": 0.608, + "recall@20": 0.789, + "n_ret@20": 19.262 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 43.40903720003553, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.776, + "recall@20": 0.886, + "n_ret@20": 19.994 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 9.85258739895653, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.608, + "recall@10": 0.789, + "n_ret@10": 9.15, + "all@20": 0.608, + "recall@20": 0.789, + "n_ret@20": 12.58 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 10.401101197465323, + "default": false, + "all@5": 0.586, + "recall@5": 0.778, + "n_ret@5": 5.0, + "all@10": 0.712, + "recall@10": 0.849, + "n_ret@10": 10.0, + "all@20": 0.772, + "recall@20": 0.884, + "n_ret@20": 19.994 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 10.550564401899464, + "default": false, + "all@5": 0.57, + "recall@5": 0.77, + "n_ret@5": 5.0, + "all@10": 0.692, + "recall@10": 0.839, + "n_ret@10": 10.0, + "all@20": 0.762, + "recall@20": 0.879, + "n_ret@20": 19.998 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 52.536360599449836, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 9.64534879999701, + "default": false, + "all@5": 0.586, + "recall@5": 0.778, + "n_ret@5": 5.0, + "all@10": 0.608, + "recall@10": 0.789, + "n_ret@10": 9.072, + "all@20": 0.608, + "recall@20": 0.789, + "n_ret@20": 11.578 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 16.402442397666164, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 10.122084200265817, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.776, + "recall@20": 0.886, + "n_ret@20": 19.994 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 55.42428879986983, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 52.200478601502255, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.736, + "recall@20": 0.862, + "n_ret@20": 17.556 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 51.83053760020994, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.608, + "recall@10": 0.789, + "n_ret@10": 9.072, + "all@20": 0.608, + "recall@20": 0.789, + "n_ret@20": 11.578 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 48.716949399444275, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 12.524338802089915, + "default": false, + "all@5": 0.558, + "recall@5": 0.763, + "n_ret@5": 5.0, + "all@10": 0.684, + "recall@10": 0.835, + "n_ret@10": 10.0, + "all@20": 0.748, + "recall@20": 0.872, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 47.65806899906602, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.608, + "recall@10": 0.789, + "n_ret@10": 9.874, + "all@20": 0.608, + "recall@20": 0.789, + "n_ret@20": 18.624 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 44.84827999887057, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.736, + "recall@20": 0.862, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 46.228193000890315, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.732, + "recall@10": 0.86, + "n_ret@10": 10.0, + "all@20": 0.736, + "recall@20": 0.862, + "n_ret@20": 19.266 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 9.417956798453815, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 46.015431799925864, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.776, + "recall@20": 0.886, + "n_ret@20": 19.998 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 44.98944779892918, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.736, + "recall@20": 0.862, + "n_ret@20": 19.138 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 10.379355199984275, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.736, + "recall@20": 0.862, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 10.875360999256372, + "default": false, + "all@5": 0.57, + "recall@5": 0.77, + "n_ret@5": 5.0, + "all@10": 0.692, + "recall@10": 0.839, + "n_ret@10": 10.0, + "all@20": 0.776, + "recall@20": 0.886, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 51.17604639963247, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 49.05599820113275, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 47.34934219776187, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.608, + "recall@10": 0.789, + "n_ret@10": 9.874, + "all@20": 0.608, + "recall@20": 0.789, + "n_ret@20": 18.624 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 46.00555160024669, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.776, + "recall@20": 0.886, + "n_ret@20": 19.994 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 50.357948799966834, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.776, + "recall@20": 0.886, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.3, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 15.381075400975533, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 49.51358559995424, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.608, + "recall@10": 0.789, + "n_ret@10": 9.874, + "all@20": 0.608, + "recall@20": 0.789, + "n_ret@20": 18.624 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 50.42279619758483, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.736, + "recall@20": 0.862, + "n_ret@20": 19.266 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 51.789792999159545, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.776, + "recall@20": 0.886, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 48.16277979989536, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.736, + "recall@20": 0.862, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 47.02332860138267, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.732, + "recall@10": 0.86, + "n_ret@10": 10.0, + "all@20": 0.736, + "recall@20": 0.862, + "n_ret@20": 18.938 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 43.81570719985757, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.776, + "recall@20": 0.886, + "n_ret@20": 19.994 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 9.23296640033368, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.776, + "recall@20": 0.886, + "n_ret@20": 19.994 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 46.97995540010743, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.736, + "recall@20": 0.862, + "n_ret@20": 19.982 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 9.773532600258477, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.776, + "recall@20": 0.886, + "n_ret@20": 19.998 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 48.34592180128675, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 48.351126800407656, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.732, + "recall@10": 0.86, + "n_ret@10": 10.0, + "all@20": 0.792, + "recall@20": 0.894, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 47.52969079907052, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.736, + "recall@20": 0.862, + "n_ret@20": 19.982 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.479493599617854, + "default": false, + "all@5": 0.59, + "recall@5": 0.78, + "n_ret@5": 5.0, + "all@10": 0.608, + "recall@10": 0.789, + "n_ret@10": 8.514, + "all@20": 0.608, + "recall@20": 0.789, + "n_ret@20": 9.194 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 41.93298800021876, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.608, + "recall@10": 0.789, + "n_ret@10": 9.072, + "all@20": 0.608, + "recall@20": 0.789, + "n_ret@20": 11.578 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 41.85968760063406, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.776, + "recall@20": 0.886, + "n_ret@20": 19.994 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 42.839294400881045, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.776, + "recall@20": 0.886, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 43.48712100018747, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.732, + "recall@10": 0.86, + "n_ret@10": 10.0, + "all@20": 0.792, + "recall@20": 0.894, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 42.58338779991027, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.608, + "recall@10": 0.789, + "n_ret@10": 9.874, + "all@20": 0.608, + "recall@20": 0.789, + "n_ret@20": 18.624 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.280888000270352, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.608, + "recall@10": 0.789, + "n_ret@10": 8.514, + "all@20": 0.608, + "recall@20": 0.789, + "n_ret@20": 9.194 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 42.69403860042803, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.568369399756193, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.608, + "recall@10": 0.789, + "n_ret@10": 9.072, + "all@20": 0.608, + "recall@20": 0.789, + "n_ret@20": 11.578 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 41.11193259968422, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.608, + "recall@10": 0.789, + "n_ret@10": 9.298, + "all@20": 0.608, + "recall@20": 0.789, + "n_ret@20": 12.2 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 40.85534339933656, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.776, + "recall@20": 0.886, + "n_ret@20": 19.998 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 46.432914000353776, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 8.376725201262161, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.736, + "recall@20": 0.862, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 42.815662199864164, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.843004199792631, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.776, + "recall@20": 0.886, + "n_ret@20": 19.994 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 9.381546200136654, + "default": false, + "all@5": 0.558, + "recall@5": 0.763, + "n_ret@5": 5.0, + "all@10": 0.684, + "recall@10": 0.835, + "n_ret@10": 10.0, + "all@20": 0.748, + "recall@20": 0.872, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 41.68823140009772, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.736, + "recall@20": 0.862, + "n_ret@20": 19.138 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 41.0754895999562, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.736, + "recall@20": 0.862, + "n_ret@20": 17.556 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 10.618407599977218, + "default": false, + "all@5": 0.558, + "recall@5": 0.763, + "n_ret@5": 5.0, + "all@10": 0.684, + "recall@10": 0.835, + "n_ret@10": 10.0, + "all@20": 0.76, + "recall@20": 0.878, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 41.133327400893904, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.608, + "recall@10": 0.789, + "n_ret@10": 9.874, + "all@20": 0.608, + "recall@20": 0.789, + "n_ret@20": 18.624 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 42.16855980025139, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.732, + "recall@10": 0.86, + "n_ret@10": 10.0, + "all@20": 0.792, + "recall@20": 0.894, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.464434399269521, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.608, + "recall@10": 0.789, + "n_ret@10": 9.15, + "all@20": 0.608, + "recall@20": 0.789, + "n_ret@20": 12.58 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 44.779588400386274, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 43.65045060124248, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.732, + "recall@10": 0.86, + "n_ret@10": 10.0, + "all@20": 0.736, + "recall@20": 0.862, + "n_ret@20": 19.138 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 9.435047000064515, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 10.13337100064382, + "default": false, + "all@5": 0.558, + "recall@5": 0.763, + "n_ret@5": 5.0, + "all@10": 0.594, + "recall@10": 0.782, + "n_ret@10": 9.894, + "all@20": 0.604, + "recall@20": 0.787, + "n_ret@20": 19.262 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 42.79533520003315, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.736, + "recall@20": 0.862, + "n_ret@20": 19.982 + } + ], + "lib_selected": { + "5": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + } + }, + "seconds": 2567.042277574539 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/hotpotqa__bge__ctrl_mention.npz b/benchmarks/multihop/results/raw/hotpotqa__bge__ctrl_mention.npz new file mode 100644 index 0000000..34dd145 Binary files /dev/null and b/benchmarks/multihop/results/raw/hotpotqa__bge__ctrl_mention.npz differ diff --git a/benchmarks/multihop/results/raw/hotpotqa__bge__knn.json b/benchmarks/multihop/results/raw/hotpotqa__bge__knn.json new file mode 100644 index 0000000..1e926e2 --- /dev/null +++ b/benchmarks/multihop/results/raw/hotpotqa__bge__knn.json @@ -0,0 +1,3274 @@ +{ + "dataset": "hotpotqa", + "model": "bge", + "variant": "knn", + "families": [ + "knn" + ], + "diagnostics": { + "n_edges_undirected_union": 54857, + "n_edges_by_family_directed_or_undirected": 54857, + "mean_degree": 7.530130404941661, + "median_degree": 6.0, + "max_degree": 118, + "isolated_frac": 0.0, + "gold_pairs_directly_linked": 0.5393333333333333, + "gold_pairs_linked_within_2": 0.6733333333333333, + "neighbours_of_gold_that_are_in_same_question_context": 0.24001444287618007 + }, + "ppr_dev_grid": [ + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.608, + "recall@5": 0.789, + "all@10": 0.788, + "recall@10": 0.886, + "all@20": 0.866, + "recall@20": 0.927 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.608, + "recall@5": 0.789, + "all@10": 0.798, + "recall@10": 0.891, + "all@20": 0.864, + "recall@20": 0.927 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.608, + "recall@5": 0.789, + "all@10": 0.796, + "recall@10": 0.892, + "all@20": 0.896, + "recall@20": 0.946 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.608, + "recall@5": 0.789, + "all@10": 0.804, + "recall@10": 0.896, + "all@20": 0.898, + "recall@20": 0.947 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.608, + "recall@5": 0.789, + "all@10": 0.776, + "recall@10": 0.884, + "all@20": 0.878, + "recall@20": 0.938 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.608, + "recall@5": 0.789, + "all@10": 0.776, + "recall@10": 0.884, + "all@20": 0.882, + "recall@20": 0.94 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.608, + "recall@5": 0.789, + "all@10": 0.742, + "recall@10": 0.866, + "all@20": 0.816, + "recall@20": 0.907 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.608, + "recall@5": 0.789, + "all@10": 0.742, + "recall@10": 0.866, + "all@20": 0.816, + "recall@20": 0.907 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.516, + "recall@5": 0.741, + "all@10": 0.736, + "recall@10": 0.862, + "all@20": 0.874, + "recall@20": 0.934 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.516, + "recall@5": 0.738, + "all@10": 0.736, + "recall@10": 0.862, + "all@20": 0.882, + "recall@20": 0.938 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.554, + "recall@5": 0.76, + "all@10": 0.736, + "recall@10": 0.862, + "all@20": 0.886, + "recall@20": 0.94 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.548, + "recall@5": 0.757, + "all@10": 0.736, + "recall@10": 0.862, + "all@20": 0.89, + "recall@20": 0.943 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.584, + "recall@5": 0.775, + "all@10": 0.736, + "recall@10": 0.862, + "all@20": 0.872, + "recall@20": 0.935 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.576, + "recall@5": 0.771, + "all@10": 0.736, + "recall@10": 0.862, + "all@20": 0.874, + "recall@20": 0.935 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.618, + "recall@5": 0.796, + "all@10": 0.736, + "recall@10": 0.862, + "all@20": 0.814, + "recall@20": 0.906 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.618, + "recall@5": 0.796, + "all@10": 0.736, + "recall@10": 0.862, + "all@20": 0.814, + "recall@20": 0.906 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.44, + "recall@5": 0.687, + "all@10": 0.702, + "recall@10": 0.845, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.43, + "recall@5": 0.681, + "all@10": 0.698, + "recall@10": 0.842, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.506, + "recall@5": 0.731, + "all@10": 0.712, + "recall@10": 0.85, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.496, + "recall@5": 0.727, + "all@10": 0.712, + "recall@10": 0.85, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.576, + "recall@5": 0.771, + "all@10": 0.734, + "recall@10": 0.862, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.572, + "recall@5": 0.77, + "all@10": 0.736, + "recall@10": 0.863, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.618, + "recall@5": 0.795, + "all@10": 0.74, + "recall@10": 0.865, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.616, + "recall@5": 0.794, + "all@10": 0.738, + "recall@10": 0.864, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.608, + "recall@5": 0.789, + "all@10": 0.78, + "recall@10": 0.882, + "all@20": 0.862, + "recall@20": 0.925 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.608, + "recall@5": 0.789, + "all@10": 0.796, + "recall@10": 0.89, + "all@20": 0.862, + "recall@20": 0.925 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.608, + "recall@5": 0.789, + "all@10": 0.796, + "recall@10": 0.89, + "all@20": 0.892, + "recall@20": 0.944 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.608, + "recall@5": 0.789, + "all@10": 0.8, + "recall@10": 0.892, + "all@20": 0.892, + "recall@20": 0.944 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.608, + "recall@5": 0.789, + "all@10": 0.786, + "recall@10": 0.887, + "all@20": 0.892, + "recall@20": 0.945 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.608, + "recall@5": 0.789, + "all@10": 0.794, + "recall@10": 0.891, + "all@20": 0.89, + "recall@20": 0.944 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.608, + "recall@5": 0.789, + "all@10": 0.752, + "recall@10": 0.872, + "all@20": 0.822, + "recall@20": 0.91 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.608, + "recall@5": 0.789, + "all@10": 0.752, + "recall@10": 0.872, + "all@20": 0.824, + "recall@20": 0.911 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.472, + "recall@5": 0.711, + "all@10": 0.734, + "recall@10": 0.861, + "all@20": 0.864, + "recall@20": 0.929 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.464, + "recall@5": 0.704, + "all@10": 0.734, + "recall@10": 0.861, + "all@20": 0.87, + "recall@20": 0.932 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.514, + "recall@5": 0.738, + "all@10": 0.734, + "recall@10": 0.861, + "all@20": 0.884, + "recall@20": 0.939 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.508, + "recall@5": 0.733, + "all@10": 0.732, + "recall@10": 0.86, + "all@20": 0.89, + "recall@20": 0.942 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.558, + "recall@5": 0.763, + "all@10": 0.734, + "recall@10": 0.861, + "all@20": 0.864, + "recall@20": 0.931 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.558, + "recall@5": 0.763, + "all@10": 0.734, + "recall@10": 0.861, + "all@20": 0.868, + "recall@20": 0.933 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.608, + "recall@5": 0.791, + "all@10": 0.734, + "recall@10": 0.861, + "all@20": 0.82, + "recall@20": 0.909 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.61, + "recall@5": 0.792, + "all@10": 0.734, + "recall@10": 0.861, + "all@20": 0.82, + "recall@20": 0.909 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.36, + "recall@5": 0.622, + "all@10": 0.66, + "recall@10": 0.815, + "all@20": 0.798, + "recall@20": 0.897 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.35, + "recall@5": 0.616, + "all@10": 0.658, + "recall@10": 0.814, + "all@20": 0.798, + "recall@20": 0.897 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.434, + "recall@5": 0.682, + "all@10": 0.696, + "recall@10": 0.837, + "all@20": 0.8, + "recall@20": 0.898 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.424, + "recall@5": 0.676, + "all@10": 0.694, + "recall@10": 0.835, + "all@20": 0.798, + "recall@20": 0.897 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.526, + "recall@5": 0.746, + "all@10": 0.714, + "recall@10": 0.851, + "all@20": 0.796, + "recall@20": 0.896 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.528, + "recall@5": 0.745, + "all@10": 0.72, + "recall@10": 0.855, + "all@20": 0.796, + "recall@20": 0.896 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.592, + "recall@5": 0.782, + "all@10": 0.734, + "recall@10": 0.862, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.594, + "recall@5": 0.783, + "all@10": 0.734, + "recall@10": 0.862, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.588, + "recall@5": 0.776, + "all@10": 0.764, + "recall@10": 0.874, + "all@20": 0.864, + "recall@20": 0.926 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.582, + "recall@5": 0.772, + "all@10": 0.77, + "recall@10": 0.876, + "all@20": 0.862, + "recall@20": 0.925 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.59, + "recall@5": 0.779, + "all@10": 0.782, + "recall@10": 0.883, + "all@20": 0.882, + "recall@20": 0.938 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.59, + "recall@5": 0.779, + "all@10": 0.784, + "recall@10": 0.884, + "all@20": 0.882, + "recall@20": 0.938 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.6, + "recall@5": 0.784, + "all@10": 0.782, + "recall@10": 0.885, + "all@20": 0.894, + "recall@20": 0.946 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.6, + "recall@5": 0.784, + "all@10": 0.786, + "recall@10": 0.887, + "all@20": 0.894, + "recall@20": 0.946 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.606, + "recall@5": 0.788, + "all@10": 0.758, + "recall@10": 0.875, + "all@20": 0.832, + "recall@20": 0.915 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.604, + "recall@5": 0.787, + "all@10": 0.756, + "recall@10": 0.874, + "all@20": 0.832, + "recall@20": 0.915 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.398, + "recall@5": 0.659, + "all@10": 0.728, + "recall@10": 0.859, + "all@20": 0.852, + "recall@20": 0.923 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.388, + "recall@5": 0.646, + "all@10": 0.726, + "recall@10": 0.858, + "all@20": 0.848, + "recall@20": 0.921 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.472, + "recall@5": 0.709, + "all@10": 0.732, + "recall@10": 0.861, + "all@20": 0.858, + "recall@20": 0.926 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.464, + "recall@5": 0.704, + "all@10": 0.73, + "recall@10": 0.86, + "all@20": 0.864, + "recall@20": 0.929 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.538, + "recall@5": 0.753, + "all@10": 0.736, + "recall@10": 0.863, + "all@20": 0.87, + "recall@20": 0.933 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.538, + "recall@5": 0.752, + "all@10": 0.738, + "recall@10": 0.864, + "all@20": 0.868, + "recall@20": 0.932 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.59, + "recall@5": 0.781, + "all@10": 0.736, + "recall@10": 0.862, + "all@20": 0.822, + "recall@20": 0.91 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.588, + "recall@5": 0.78, + "all@10": 0.734, + "recall@10": 0.861, + "all@20": 0.824, + "recall@20": 0.911 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.264, + "recall@5": 0.533, + "all@10": 0.608, + "recall@10": 0.78, + "all@20": 0.802, + "recall@20": 0.899 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.27, + "recall@5": 0.536, + "all@10": 0.598, + "recall@10": 0.773, + "all@20": 0.798, + "recall@20": 0.896 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.366, + "recall@5": 0.625, + "all@10": 0.668, + "recall@10": 0.817, + "all@20": 0.808, + "recall@20": 0.902 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.362, + "recall@5": 0.622, + "all@10": 0.664, + "recall@10": 0.816, + "all@20": 0.806, + "recall@20": 0.901 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.484, + "recall@5": 0.719, + "all@10": 0.704, + "recall@10": 0.844, + "all@20": 0.8, + "recall@20": 0.898 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.468, + "recall@5": 0.712, + "all@10": 0.706, + "recall@10": 0.844, + "all@20": 0.802, + "recall@20": 0.899 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.584, + "recall@5": 0.778, + "all@10": 0.73, + "recall@10": 0.86, + "all@20": 0.8, + "recall@20": 0.898 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.576, + "recall@5": 0.773, + "all@10": 0.73, + "recall@10": 0.86, + "all@20": 0.8, + "recall@20": 0.898 + } + ], + "ppr_selected": { + "5": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "20": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + } + }, + "lib_dev_grid": [ + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 12.107529400265776, + "default": true, + "all@5": 0.628, + "recall@5": 0.798, + "n_ret@5": 5.0, + "all@10": 0.796, + "recall@10": 0.891, + "n_ret@10": 9.996, + "all@20": 0.854, + "recall@20": 0.923, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.674756601103581, + "default": false, + "all@5": 0.56, + "recall@5": 0.764, + "n_ret@5": 5.0, + "all@10": 0.784, + "recall@10": 0.888, + "n_ret@10": 10.0, + "all@20": 0.878, + "recall@20": 0.937, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 40.09200239821803, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 27.309269800549373, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 61.654503599507734, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 36.62650980043691, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.866, + "recall@20": 0.931, + "n_ret@20": 19.992 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 46.67496819922235, + "default": false, + "all@5": 0.654, + "recall@5": 0.812, + "n_ret@5": 5.0, + "all@10": 0.776, + "recall@10": 0.884, + "n_ret@10": 10.0, + "all@20": 0.888, + "recall@20": 0.943, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 39.627406600513496, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.846, + "recall@20": 0.922, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 52.18845220049843, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 7.375948601518758, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.872, + "recall@20": 0.933, + "n_ret@20": 19.992 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 39.040560398600064, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.864, + "recall@20": 0.93, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 50.37481719930656, + "default": false, + "all@5": 0.66, + "recall@5": 0.815, + "n_ret@5": 5.0, + "all@10": 0.816, + "recall@10": 0.904, + "n_ret@10": 10.0, + "all@20": 0.906, + "recall@20": 0.951, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 40.13471420109272, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 40.116407600813545, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.878, + "recall@20": 0.936, + "n_ret@20": 19.992 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.227778796805069, + "default": false, + "all@5": 0.56, + "recall@5": 0.764, + "n_ret@5": 5.0, + "all@10": 0.77, + "recall@10": 0.878, + "n_ret@10": 9.992, + "all@20": 0.854, + "recall@20": 0.923, + "n_ret@20": 19.704 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 44.75225040107034, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.88, + "recall@20": 0.939, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.142972198897041, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.888, + "recall@20": 0.942, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 39.210223399451934, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 7.012974799028598, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.79, + "recall@10": 0.886, + "n_ret@10": 9.992, + "all@20": 0.85, + "recall@20": 0.92, + "n_ret@20": 19.704 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 41.822184600285254, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.822, + "recall@10": 0.908, + "n_ret@10": 9.996, + "all@20": 0.88, + "recall@20": 0.938, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 46.235334399621934, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.898, + "recall@20": 0.948, + "n_ret@20": 19.992 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 61.028010600595735, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.846, + "recall@20": 0.922, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 44.42451680067461, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.788, + "recall@10": 0.889, + "n_ret@10": 9.992, + "all@20": 0.86, + "recall@20": 0.928, + "n_ret@20": 19.704 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 54.45027399994433, + "default": false, + "all@5": 0.654, + "recall@5": 0.812, + "n_ret@5": 5.0, + "all@10": 0.78, + "recall@10": 0.886, + "n_ret@10": 10.0, + "all@20": 0.884, + "recall@20": 0.941, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 83.20742660143878, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 51.699535000137985, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.8, + "recall@10": 0.895, + "n_ret@10": 9.992, + "all@20": 0.866, + "recall@20": 0.93, + "n_ret@20": 19.704 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 13.178274398553185, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.864, + "recall@20": 0.93, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 17.451066998648457, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.79, + "recall@10": 0.886, + "n_ret@10": 9.996, + "all@20": 0.852, + "recall@20": 0.921, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 59.26767979969736, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.8, + "recall@10": 0.895, + "n_ret@10": 9.996, + "all@20": 0.872, + "recall@20": 0.934, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 49.33142359973863, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.856, + "recall@20": 0.927, + "n_ret@20": 19.992 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 51.429114399827085, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.852, + "recall@20": 0.925, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 11.892233598977327, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 68.46975620044395, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 55.29112120135687, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.854, + "recall@20": 0.926, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 54.89873500005342, + "default": false, + "all@5": 0.66, + "recall@5": 0.815, + "n_ret@5": 5.0, + "all@10": 0.816, + "recall@10": 0.904, + "n_ret@10": 10.0, + "all@20": 0.904, + "recall@20": 0.95, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 26.410688201198354, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.888, + "recall@20": 0.942, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 16.845014399383217, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.79, + "recall@10": 0.886, + "n_ret@10": 9.996, + "all@20": 0.852, + "recall@20": 0.921, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 48.73680780129507, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 73.23387860169169, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 48.567653200239874, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 10.418275800184347, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.886, + "recall@20": 0.94, + "n_ret@20": 19.992 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 52.88559160043951, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 29.862568599404767, + "default": false, + "all@5": 0.56, + "recall@5": 0.764, + "n_ret@5": 5.0, + "all@10": 0.772, + "recall@10": 0.882, + "n_ret@10": 10.0, + "all@20": 0.866, + "recall@20": 0.931, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 14.5496247994015, + "default": false, + "all@5": 0.56, + "recall@5": 0.764, + "n_ret@5": 5.0, + "all@10": 0.76, + "recall@10": 0.873, + "n_ret@10": 9.996, + "all@20": 0.836, + "recall@20": 0.914, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 62.80954340205062, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.884, + "recall@20": 0.939, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 50.29610799928196, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.872, + "recall@20": 0.935, + "n_ret@20": 19.992 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 51.15714399830904, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.798, + "recall@10": 0.896, + "n_ret@10": 9.996, + "all@20": 0.872, + "recall@20": 0.934, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 52.15276159753557, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.782, + "recall@10": 0.886, + "n_ret@10": 9.996, + "all@20": 0.868, + "recall@20": 0.932, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 54.81002720037941, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.862, + "recall@20": 0.93, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 47.167928200098686, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.884, + "recall@20": 0.939, + "n_ret@20": 19.992 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 67.32840219943319, + "default": false, + "all@5": 0.66, + "recall@5": 0.815, + "n_ret@5": 5.0, + "all@10": 0.816, + "recall@10": 0.904, + "n_ret@10": 10.0, + "all@20": 0.894, + "recall@20": 0.945, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 48.739229201572016, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.862, + "recall@20": 0.929, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 56.10495300148614, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.866, + "recall@20": 0.931, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 21.349660800304264, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.872, + "recall@20": 0.933, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 60.51868819841184, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.852, + "recall@20": 0.925, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 43.01797460077796, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.87, + "recall@20": 0.934, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 44.66077199962456, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 47.05496280093212, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.802, + "recall@10": 0.896, + "n_ret@10": 9.996, + "all@20": 0.868, + "recall@20": 0.931, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 22.75868860038463, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.888, + "recall@20": 0.942, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 45.14310719957575, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.86, + "recall@20": 0.928, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 47.52752840111498, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.746, + "recall@10": 0.869, + "n_ret@10": 9.996, + "all@20": 0.862, + "recall@20": 0.929, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 66.0639502000995, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 54.509840800077654, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.838, + "recall@20": 0.918, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 9.682101200800389, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.738, + "recall@10": 0.863, + "n_ret@10": 10.0, + "all@20": 0.888, + "recall@20": 0.942, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 52.34508339955937, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.878, + "recall@20": 0.936, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 10.767994000692852, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.864, + "recall@20": 0.93, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 45.43377799890004, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 40.88360580080189, + "default": false, + "all@5": 0.654, + "recall@5": 0.812, + "n_ret@5": 5.0, + "all@10": 0.78, + "recall@10": 0.886, + "n_ret@10": 10.0, + "all@20": 0.892, + "recall@20": 0.945, + "n_ret@20": 19.992 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 43.89348960330244, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.796, + "recall@10": 0.895, + "n_ret@10": 9.992, + "all@20": 0.866, + "recall@20": 0.931, + "n_ret@20": 19.704 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 43.47900179843418, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.796, + "recall@10": 0.893, + "n_ret@10": 9.992, + "all@20": 0.868, + "recall@20": 0.932, + "n_ret@20": 19.704 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 11.367452599457465, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.864, + "recall@20": 0.93, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 46.051590400282294, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 57.714035398792475, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.864, + "recall@20": 0.93, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 54.174885001033545, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.87, + "recall@20": 0.934, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 20.571032999199815, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.872, + "recall@20": 0.933, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.704824800253846, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.796, + "recall@10": 0.891, + "n_ret@10": 9.992, + "all@20": 0.854, + "recall@20": 0.923, + "n_ret@20": 19.704 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 41.69615339965094, + "default": false, + "all@5": 0.66, + "recall@5": 0.815, + "n_ret@5": 5.0, + "all@10": 0.818, + "recall@10": 0.905, + "n_ret@10": 10.0, + "all@20": 0.886, + "recall@20": 0.941, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 55.7483021998778, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 31.539213000447493, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 41.80117759865243, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.744, + "recall@10": 0.868, + "n_ret@10": 9.992, + "all@20": 0.854, + "recall@20": 0.925, + "n_ret@20": 19.704 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 38.871580401435494, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.854, + "recall@20": 0.926, + "n_ret@20": 19.992 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 57.74281620129477, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 42.7125760007184, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.864, + "recall@20": 0.93, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 14.10069399769418, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.79, + "recall@10": 0.886, + "n_ret@10": 9.996, + "all@20": 0.852, + "recall@20": 0.921, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 20.49296679859981, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.886, + "recall@20": 0.94, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 90.37304059928283, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.766, + "recall@10": 0.879, + "n_ret@10": 9.996, + "all@20": 0.864, + "recall@20": 0.93, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 83.88123980071396, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.898, + "recall@20": 0.947, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 88.17351899889763, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 114.05274799908511, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 84.983908201335, + "default": false, + "all@5": 0.66, + "recall@5": 0.815, + "n_ret@5": 5.0, + "all@10": 0.82, + "recall@10": 0.907, + "n_ret@10": 9.992, + "all@20": 0.874, + "recall@20": 0.934, + "n_ret@20": 19.704 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 128.49457399884705, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 87.77097100077663, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.842, + "recall@20": 0.92, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 85.45722519862466, + "default": false, + "all@5": 0.654, + "recall@5": 0.812, + "n_ret@5": 5.0, + "all@10": 0.78, + "recall@10": 0.886, + "n_ret@10": 10.0, + "all@20": 0.866, + "recall@20": 0.931, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 93.66910859837662, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.822, + "recall@10": 0.908, + "n_ret@10": 9.996, + "all@20": 0.882, + "recall@20": 0.939, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 28.755833799717948, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.79, + "recall@10": 0.886, + "n_ret@10": 9.996, + "all@20": 0.852, + "recall@20": 0.921, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 86.6611048001796, + "default": false, + "all@5": 0.66, + "recall@5": 0.815, + "n_ret@5": 5.0, + "all@10": 0.818, + "recall@10": 0.905, + "n_ret@10": 10.0, + "all@20": 0.898, + "recall@20": 0.947, + "n_ret@20": 19.992 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 19.358300799387507, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.872, + "recall@20": 0.933, + "n_ret@20": 19.992 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 20.156545999692753, + "default": false, + "all@5": 0.56, + "recall@5": 0.764, + "n_ret@5": 5.0, + "all@10": 0.784, + "recall@10": 0.888, + "n_ret@10": 10.0, + "all@20": 0.878, + "recall@20": 0.937, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 32.36959100118838, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.886, + "recall@20": 0.94, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 51.987502999720164, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.738, + "recall@10": 0.863, + "n_ret@10": 10.0, + "all@20": 0.798, + "recall@20": 0.897, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 128.83182839956135, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + } + ], + "lib_selected": { + "5": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + } + }, + "seconds": 5251.630596160889 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/hotpotqa__bge__knn.npz b/benchmarks/multihop/results/raw/hotpotqa__bge__knn.npz new file mode 100644 index 0000000..2008d45 Binary files /dev/null and b/benchmarks/multihop/results/raw/hotpotqa__bge__knn.npz differ diff --git a/benchmarks/multihop/results/raw/hotpotqa__bge__mention.json b/benchmarks/multihop/results/raw/hotpotqa__bge__mention.json new file mode 100644 index 0000000..4685747 --- /dev/null +++ b/benchmarks/multihop/results/raw/hotpotqa__bge__mention.json @@ -0,0 +1,3274 @@ +{ + "dataset": "hotpotqa", + "model": "bge", + "variant": "mention", + "families": [ + "mention" + ], + "diagnostics": { + "n_edges_undirected_union": 11348, + "n_edges_by_family_directed_or_undirected": 12354, + "mean_degree": 1.5577213452299246, + "median_degree": 1.0, + "max_degree": 29, + "isolated_frac": 0.39958819492107067, + "gold_pairs_directly_linked": 0.6513333333333333, + "gold_pairs_linked_within_2": 0.6673333333333333, + "neighbours_of_gold_that_are_in_same_question_context": 0.5890391209006267 + }, + "ppr_dev_grid": [ + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.616, + "recall@5": 0.791, + "all@10": 0.838, + "recall@10": 0.908, + "all@20": 0.854, + "recall@20": 0.916 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.614, + "recall@5": 0.792, + "all@10": 0.872, + "recall@10": 0.927, + "all@20": 0.908, + "recall@20": 0.945 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.618, + "recall@5": 0.792, + "all@10": 0.88, + "recall@10": 0.932, + "all@20": 0.922, + "recall@20": 0.958 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.616, + "recall@5": 0.793, + "all@10": 0.896, + "recall@10": 0.94, + "all@20": 0.946, + "recall@20": 0.97 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.62, + "recall@5": 0.795, + "all@10": 0.88, + "recall@10": 0.935, + "all@20": 0.922, + "recall@20": 0.959 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.616, + "recall@5": 0.793, + "all@10": 0.886, + "recall@10": 0.938, + "all@20": 0.93, + "recall@20": 0.963 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.618, + "recall@5": 0.794, + "all@10": 0.824, + "recall@10": 0.908, + "all@20": 0.882, + "recall@20": 0.939 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.614, + "recall@5": 0.792, + "all@10": 0.83, + "recall@10": 0.911, + "all@20": 0.882, + "recall@20": 0.939 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.644, + "recall@5": 0.804, + "all@10": 0.742, + "recall@10": 0.865, + "all@20": 0.892, + "recall@20": 0.942 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.614, + "recall@5": 0.792, + "all@10": 0.742, + "recall@10": 0.865, + "all@20": 0.924, + "recall@20": 0.958 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.656, + "recall@5": 0.811, + "all@10": 0.742, + "recall@10": 0.865, + "all@20": 0.914, + "recall@20": 0.954 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.646, + "recall@5": 0.809, + "all@10": 0.742, + "recall@10": 0.865, + "all@20": 0.936, + "recall@20": 0.965 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.672, + "recall@5": 0.819, + "all@10": 0.742, + "recall@10": 0.865, + "all@20": 0.918, + "recall@20": 0.957 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.676, + "recall@5": 0.827, + "all@10": 0.742, + "recall@10": 0.865, + "all@20": 0.93, + "recall@20": 0.963 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.672, + "recall@5": 0.824, + "all@10": 0.74, + "recall@10": 0.864, + "all@20": 0.882, + "recall@20": 0.939 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.674, + "recall@5": 0.826, + "all@10": 0.74, + "recall@10": 0.864, + "all@20": 0.884, + "recall@20": 0.94 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.564, + "recall@5": 0.756, + "all@10": 0.754, + "recall@10": 0.874, + "all@20": 0.802, + "recall@20": 0.899 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.56, + "recall@5": 0.753, + "all@10": 0.716, + "recall@10": 0.851, + "all@20": 0.796, + "recall@20": 0.896 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.624, + "recall@5": 0.79, + "all@10": 0.754, + "recall@10": 0.874, + "all@20": 0.8, + "recall@20": 0.898 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.63, + "recall@5": 0.799, + "all@10": 0.744, + "recall@10": 0.867, + "all@20": 0.796, + "recall@20": 0.896 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.666, + "recall@5": 0.82, + "all@10": 0.762, + "recall@10": 0.878, + "all@20": 0.798, + "recall@20": 0.897 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.684, + "recall@5": 0.831, + "all@10": 0.754, + "recall@10": 0.873, + "all@20": 0.796, + "recall@20": 0.896 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.684, + "recall@5": 0.83, + "all@10": 0.762, + "recall@10": 0.878, + "all@20": 0.798, + "recall@20": 0.897 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.69, + "recall@5": 0.835, + "all@10": 0.766, + "recall@10": 0.88, + "all@20": 0.794, + "recall@20": 0.895 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.628, + "recall@5": 0.793, + "all@10": 0.83, + "recall@10": 0.903, + "all@20": 0.854, + "recall@20": 0.916 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.704, + "recall@5": 0.833, + "all@10": 0.87, + "recall@10": 0.926, + "all@20": 0.906, + "recall@20": 0.944 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.636, + "recall@5": 0.8, + "all@10": 0.866, + "recall@10": 0.925, + "all@20": 0.92, + "recall@20": 0.957 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.72, + "recall@5": 0.843, + "all@10": 0.89, + "recall@10": 0.937, + "all@20": 0.946, + "recall@20": 0.969 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.642, + "recall@5": 0.804, + "all@10": 0.88, + "recall@10": 0.933, + "all@20": 0.92, + "recall@20": 0.958 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.706, + "recall@5": 0.838, + "all@10": 0.892, + "recall@10": 0.939, + "all@20": 0.942, + "recall@20": 0.968 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.63, + "recall@5": 0.8, + "all@10": 0.846, + "recall@10": 0.919, + "all@20": 0.89, + "recall@20": 0.943 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.652, + "recall@5": 0.811, + "all@10": 0.848, + "recall@10": 0.92, + "all@20": 0.894, + "recall@20": 0.945 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.59, + "recall@5": 0.765, + "all@10": 0.772, + "recall@10": 0.88, + "all@20": 0.89, + "recall@20": 0.941 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.56, + "recall@5": 0.742, + "all@10": 0.808, + "recall@10": 0.898, + "all@20": 0.922, + "recall@20": 0.957 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.648, + "recall@5": 0.802, + "all@10": 0.768, + "recall@10": 0.878, + "all@20": 0.91, + "recall@20": 0.952 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.642, + "recall@5": 0.799, + "all@10": 0.82, + "recall@10": 0.904, + "all@20": 0.932, + "recall@20": 0.963 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.68, + "recall@5": 0.821, + "all@10": 0.762, + "recall@10": 0.875, + "all@20": 0.916, + "recall@20": 0.955 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.672, + "recall@5": 0.823, + "all@10": 0.81, + "recall@10": 0.899, + "all@20": 0.942, + "recall@20": 0.968 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.682, + "recall@5": 0.827, + "all@10": 0.752, + "recall@10": 0.871, + "all@20": 0.882, + "recall@20": 0.939 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.674, + "recall@5": 0.826, + "all@10": 0.764, + "recall@10": 0.876, + "all@20": 0.898, + "recall@20": 0.947 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.414, + "recall@5": 0.659, + "all@10": 0.732, + "recall@10": 0.854, + "all@20": 0.832, + "recall@20": 0.914 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.476, + "recall@5": 0.665, + "all@10": 0.654, + "recall@10": 0.804, + "all@20": 0.866, + "recall@20": 0.931 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.546, + "recall@5": 0.742, + "all@10": 0.766, + "recall@10": 0.879, + "all@20": 0.828, + "recall@20": 0.912 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.536, + "recall@5": 0.723, + "all@10": 0.76, + "recall@10": 0.872, + "all@20": 0.874, + "recall@20": 0.935 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.648, + "recall@5": 0.801, + "all@10": 0.77, + "recall@10": 0.882, + "all@20": 0.822, + "recall@20": 0.909 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.66, + "recall@5": 0.817, + "all@10": 0.772, + "recall@10": 0.881, + "all@20": 0.87, + "recall@20": 0.933 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.682, + "recall@5": 0.829, + "all@10": 0.768, + "recall@10": 0.88, + "all@20": 0.808, + "recall@20": 0.902 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.69, + "recall@5": 0.836, + "all@10": 0.758, + "recall@10": 0.876, + "all@20": 0.828, + "recall@20": 0.912 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.63, + "recall@5": 0.781, + "all@10": 0.824, + "recall@10": 0.899, + "all@20": 0.854, + "recall@20": 0.916 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.634, + "recall@5": 0.781, + "all@10": 0.852, + "recall@10": 0.917, + "all@20": 0.908, + "recall@20": 0.945 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.654, + "recall@5": 0.8, + "all@10": 0.862, + "recall@10": 0.923, + "all@20": 0.916, + "recall@20": 0.955 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.696, + "recall@5": 0.826, + "all@10": 0.874, + "recall@10": 0.929, + "all@20": 0.942, + "recall@20": 0.965 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.64, + "recall@5": 0.801, + "all@10": 0.868, + "recall@10": 0.927, + "all@20": 0.918, + "recall@20": 0.957 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.74, + "recall@5": 0.856, + "all@10": 0.884, + "recall@10": 0.935, + "all@20": 0.942, + "recall@20": 0.968 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.642, + "recall@5": 0.806, + "all@10": 0.834, + "recall@10": 0.913, + "all@20": 0.886, + "recall@20": 0.941 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.7, + "recall@5": 0.837, + "all@10": 0.846, + "recall@10": 0.919, + "all@20": 0.9, + "recall@20": 0.948 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.496, + "recall@5": 0.679, + "all@10": 0.782, + "recall@10": 0.877, + "all@20": 0.88, + "recall@20": 0.936 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.496, + "recall@5": 0.685, + "all@10": 0.726, + "recall@10": 0.845, + "all@20": 0.914, + "recall@20": 0.953 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.574, + "recall@5": 0.738, + "all@10": 0.796, + "recall@10": 0.892, + "all@20": 0.906, + "recall@20": 0.95 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.572, + "recall@5": 0.739, + "all@10": 0.828, + "recall@10": 0.91, + "all@20": 0.924, + "recall@20": 0.959 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.65, + "recall@5": 0.8, + "all@10": 0.774, + "recall@10": 0.882, + "all@20": 0.91, + "recall@20": 0.953 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.684, + "recall@5": 0.822, + "all@10": 0.834, + "recall@10": 0.913, + "all@20": 0.936, + "recall@20": 0.965 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.682, + "recall@5": 0.826, + "all@10": 0.758, + "recall@10": 0.874, + "all@20": 0.876, + "recall@20": 0.936 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.688, + "recall@5": 0.832, + "all@10": 0.8, + "recall@10": 0.895, + "all@20": 0.896, + "recall@20": 0.946 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.264, + "recall@5": 0.494, + "all@10": 0.616, + "recall@10": 0.769, + "all@20": 0.854, + "recall@20": 0.923 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.412, + "recall@5": 0.611, + "all@10": 0.572, + "recall@10": 0.729, + "all@20": 0.8, + "recall@20": 0.887 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.396, + "recall@5": 0.61, + "all@10": 0.728, + "recall@10": 0.846, + "all@20": 0.854, + "recall@20": 0.925 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.492, + "recall@5": 0.668, + "all@10": 0.696, + "recall@10": 0.824, + "all@20": 0.874, + "recall@20": 0.933 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.584, + "recall@5": 0.759, + "all@10": 0.764, + "recall@10": 0.876, + "all@20": 0.836, + "recall@20": 0.916 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.628, + "recall@5": 0.782, + "all@10": 0.794, + "recall@10": 0.891, + "all@20": 0.874, + "recall@20": 0.935 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.67, + "recall@5": 0.822, + "all@10": 0.772, + "recall@10": 0.882, + "all@20": 0.818, + "recall@20": 0.907 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.69, + "recall@5": 0.835, + "all@10": 0.776, + "recall@10": 0.885, + "all@20": 0.86, + "recall@20": 0.928 + } + ], + "ppr_selected": { + "5": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "20": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + } + }, + "lib_dev_grid": [ + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.588862999342382, + "default": true, + "all@5": 0.664, + "recall@5": 0.817, + "n_ret@5": 5.0, + "all@10": 0.852, + "recall@10": 0.915, + "n_ret@10": 7.948, + "all@20": 0.854, + "recall@20": 0.916, + "n_ret@20": 9.532 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.253230998991057, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.74, + "recall@10": 0.864, + "n_ret@10": 10.0, + "all@20": 0.9, + "recall@20": 0.946, + "n_ret@20": 16.394 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 42.34714399999939, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.916, + "recall@20": 0.956, + "n_ret@20": 19.564 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 46.06393560033757, + "default": false, + "all@5": 0.7, + "recall@5": 0.835, + "n_ret@5": 5.0, + "all@10": 0.824, + "recall@10": 0.907, + "n_ret@10": 10.0, + "all@20": 0.898, + "recall@20": 0.947, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 45.93310220167041, + "default": false, + "all@5": 0.698, + "recall@5": 0.834, + "n_ret@5": 5.0, + "all@10": 0.854, + "recall@10": 0.916, + "n_ret@10": 7.948, + "all@20": 0.854, + "recall@20": 0.916, + "n_ret@20": 9.532 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 48.665244801552035, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 7.854789400706069, + "default": false, + "all@5": 0.788, + "recall@5": 0.878, + "n_ret@5": 5.0, + "all@10": 0.89, + "recall@10": 0.936, + "n_ret@10": 9.536, + "all@20": 0.906, + "recall@20": 0.944, + "n_ret@20": 16.794 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 52.76311580114998, + "default": false, + "all@5": 0.782, + "recall@5": 0.876, + "n_ret@5": 5.0, + "all@10": 0.88, + "recall@10": 0.935, + "n_ret@10": 10.0, + "all@20": 0.938, + "recall@20": 0.967, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.254376999451779, + "default": false, + "all@5": 0.794, + "recall@5": 0.882, + "n_ret@5": 5.0, + "all@10": 0.884, + "recall@10": 0.937, + "n_ret@10": 10.0, + "all@20": 0.918, + "recall@20": 0.957, + "n_ret@20": 19.336 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.783841399941593, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.74, + "recall@10": 0.864, + "n_ret@10": 10.0, + "all@20": 0.9, + "recall@20": 0.946, + "n_ret@20": 15.178 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 49.62202600063756, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 8.599834598600864, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.898, + "recall@10": 0.94, + "n_ret@10": 9.536, + "all@20": 0.908, + "recall@20": 0.945, + "n_ret@20": 16.794 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.14800839824602, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.918, + "recall@20": 0.957, + "n_ret@20": 19.564 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 45.812308998429216, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.896, + "recall@10": 0.939, + "n_ret@10": 9.086, + "all@20": 0.908, + "recall@20": 0.945, + "n_ret@20": 13.14 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 46.51193619868718, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.902, + "recall@20": 0.947, + "n_ret@20": 16.7 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 43.790022198925726, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.92, + "recall@20": 0.958, + "n_ret@20": 19.336 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 7.8713607985991985, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.898, + "recall@10": 0.939, + "n_ret@10": 9.086, + "all@20": 0.906, + "recall@20": 0.944, + "n_ret@20": 13.14 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 42.636104602366686, + "default": false, + "all@5": 0.782, + "recall@5": 0.876, + "n_ret@5": 5.0, + "all@10": 0.88, + "recall@10": 0.935, + "n_ret@10": 10.0, + "all@20": 0.946, + "recall@20": 0.969, + "n_ret@20": 19.658 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 11.61183839966543, + "default": false, + "all@5": 0.788, + "recall@5": 0.878, + "n_ret@5": 5.0, + "all@10": 0.888, + "recall@10": 0.939, + "n_ret@10": 10.0, + "all@20": 0.944, + "recall@20": 0.97, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 7.573410801589489, + "default": false, + "all@5": 0.788, + "recall@5": 0.878, + "n_ret@5": 5.0, + "all@10": 0.892, + "recall@10": 0.941, + "n_ret@10": 10.0, + "all@20": 0.938, + "recall@20": 0.965, + "n_ret@20": 18.562 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 38.46694839920383, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.9, + "recall@20": 0.946, + "n_ret@20": 16.394 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 35.77961100114044, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.9, + "recall@20": 0.946, + "n_ret@20": 16.7 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 8.195730399922468, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.938, + "recall@20": 0.965, + "n_ret@20": 19.712 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 36.172022400889546, + "default": false, + "all@5": 0.7, + "recall@5": 0.835, + "n_ret@5": 5.0, + "all@10": 0.824, + "recall@10": 0.907, + "n_ret@10": 10.0, + "all@20": 0.95, + "recall@20": 0.973, + "n_ret@20": 19.926 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 35.906195199582726, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.944, + "recall@20": 0.968, + "n_ret@20": 19.658 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 37.09599460102618, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 36.201059998595156, + "default": false, + "all@5": 0.7, + "recall@5": 0.835, + "n_ret@5": 5.0, + "all@10": 0.892, + "recall@10": 0.937, + "n_ret@10": 9.536, + "all@20": 0.914, + "recall@20": 0.948, + "n_ret@20": 16.794 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 40.752687000553124, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.948, + "recall@20": 0.972, + "n_ret@20": 19.978 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 39.59476100071333, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.944, + "recall@20": 0.968, + "n_ret@20": 19.712 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.768668798264116, + "default": false, + "all@5": 0.794, + "recall@5": 0.882, + "n_ret@5": 5.0, + "all@10": 0.884, + "recall@10": 0.937, + "n_ret@10": 10.0, + "all@20": 0.92, + "recall@20": 0.958, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 38.783743000007235, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 45.137863000738434, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.92, + "recall@20": 0.958, + "n_ret@20": 19.564 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.56195260048844, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.74, + "recall@10": 0.864, + "n_ret@10": 10.0, + "all@20": 0.918, + "recall@20": 0.957, + "n_ret@20": 19.336 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 39.74335019965656, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 39.47671380173415, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.89, + "recall@10": 0.936, + "n_ret@10": 9.58, + "all@20": 0.914, + "recall@20": 0.948, + "n_ret@20": 17.666 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 37.84058080019895, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.92, + "recall@20": 0.958, + "n_ret@20": 19.564 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.600062199286185, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.852, + "recall@10": 0.915, + "n_ret@10": 8.048, + "all@20": 0.854, + "recall@20": 0.916, + "n_ret@20": 10.266 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.086719999206252, + "default": false, + "all@5": 0.794, + "recall@5": 0.882, + "n_ret@5": 5.0, + "all@10": 0.884, + "recall@10": 0.937, + "n_ret@10": 10.0, + "all@20": 0.918, + "recall@20": 0.957, + "n_ret@20": 19.564 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 8.48339620186016, + "default": false, + "all@5": 0.788, + "recall@5": 0.878, + "n_ret@5": 5.0, + "all@10": 0.892, + "recall@10": 0.941, + "n_ret@10": 10.0, + "all@20": 0.946, + "recall@20": 0.971, + "n_ret@20": 19.926 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 37.51093799911905, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.130707998527214, + "default": false, + "all@5": 0.794, + "recall@5": 0.882, + "n_ret@5": 5.0, + "all@10": 0.852, + "recall@10": 0.915, + "n_ret@10": 7.948, + "all@20": 0.854, + "recall@20": 0.916, + "n_ret@20": 9.532 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 17.111535600502975, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 10.016591799794696, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.74, + "recall@10": 0.864, + "n_ret@10": 10.0, + "all@20": 0.918, + "recall@20": 0.957, + "n_ret@20": 19.564 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 52.014953199541196, + "default": false, + "all@5": 0.7, + "recall@5": 0.835, + "n_ret@5": 5.0, + "all@10": 0.822, + "recall@10": 0.906, + "n_ret@10": 10.0, + "all@20": 0.898, + "recall@20": 0.947, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 35.75559500057716, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.9, + "recall@20": 0.946, + "n_ret@20": 15.178 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 52.76872360042762, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.852, + "recall@10": 0.915, + "n_ret@10": 7.948, + "all@20": 0.854, + "recall@20": 0.916, + "n_ret@20": 9.532 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 54.412689200253226, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 15.834160198341126, + "default": false, + "all@5": 0.788, + "recall@5": 0.878, + "n_ret@5": 5.0, + "all@10": 0.888, + "recall@10": 0.939, + "n_ret@10": 10.0, + "all@20": 0.944, + "recall@20": 0.97, + "n_ret@20": 19.978 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 54.563673399039544, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.89, + "recall@10": 0.936, + "n_ret@10": 9.536, + "all@20": 0.914, + "recall@20": 0.948, + "n_ret@20": 16.794 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 53.83277519850526, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.946, + "recall@20": 0.969, + "n_ret@20": 19.658 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 53.83806099917274, + "default": false, + "all@5": 0.782, + "recall@5": 0.876, + "n_ret@5": 5.0, + "all@10": 0.878, + "recall@10": 0.934, + "n_ret@10": 10.0, + "all@20": 0.946, + "recall@20": 0.969, + "n_ret@20": 18.562 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 10.734754000906833, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 50.554550801054575, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.948, + "recall@20": 0.972, + "n_ret@20": 19.926 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 46.65320919919759, + "default": false, + "all@5": 0.698, + "recall@5": 0.834, + "n_ret@5": 5.0, + "all@10": 0.82, + "recall@10": 0.905, + "n_ret@10": 10.0, + "all@20": 0.902, + "recall@20": 0.947, + "n_ret@20": 16.7 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 10.646568201249465, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.74, + "recall@10": 0.864, + "n_ret@10": 10.0, + "all@20": 0.938, + "recall@20": 0.965, + "n_ret@20": 19.658 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 9.561955200042576, + "default": false, + "all@5": 0.788, + "recall@5": 0.878, + "n_ret@5": 5.0, + "all@10": 0.892, + "recall@10": 0.941, + "n_ret@10": 10.0, + "all@20": 0.946, + "recall@20": 0.971, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 52.75560820021201, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 53.05751140078064, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 46.695570401265286, + "default": false, + "all@5": 0.782, + "recall@5": 0.876, + "n_ret@5": 5.0, + "all@10": 0.898, + "recall@10": 0.94, + "n_ret@10": 9.536, + "all@20": 0.914, + "recall@20": 0.948, + "n_ret@20": 16.794 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 45.57725440070499, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.918, + "recall@20": 0.957, + "n_ret@20": 19.588 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 50.98286500014365, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.934, + "recall@20": 0.965, + "n_ret@20": 19.974 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.3, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 16.355866000405513, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 45.90812180109788, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.876, + "recall@10": 0.929, + "n_ret@10": 9.536, + "all@20": 0.914, + "recall@20": 0.948, + "n_ret@20": 16.794 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 48.660836600582115, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.944, + "recall@20": 0.968, + "n_ret@20": 18.562 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 46.30176319973543, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.948, + "recall@20": 0.972, + "n_ret@20": 19.974 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 51.5261552024167, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.94, + "recall@20": 0.966, + "n_ret@20": 19.658 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 47.87030560139101, + "default": false, + "all@5": 0.778, + "recall@5": 0.874, + "n_ret@5": 5.0, + "all@10": 0.872, + "recall@10": 0.931, + "n_ret@10": 10.0, + "all@20": 0.902, + "recall@20": 0.947, + "n_ret@20": 16.394 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 51.31862419832032, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.92, + "recall@20": 0.958, + "n_ret@20": 19.588 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 10.046980998595245, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.918, + "recall@20": 0.957, + "n_ret@20": 19.564 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 52.49310959910508, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.94, + "recall@20": 0.966, + "n_ret@20": 19.712 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 9.853598599904217, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.942, + "recall@20": 0.969, + "n_ret@20": 19.926 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 49.42355859873351, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 46.81777859968133, + "default": false, + "all@5": 0.782, + "recall@5": 0.876, + "n_ret@5": 5.0, + "all@10": 0.878, + "recall@10": 0.934, + "n_ret@10": 10.0, + "all@20": 0.938, + "recall@20": 0.967, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 48.033757399301976, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.944, + "recall@20": 0.968, + "n_ret@20": 19.712 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.571059999638237, + "default": false, + "all@5": 0.794, + "recall@5": 0.882, + "n_ret@5": 5.0, + "all@10": 0.852, + "recall@10": 0.914, + "n_ret@10": 7.462, + "all@20": 0.852, + "recall@20": 0.914, + "n_ret@20": 7.91 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 46.74531659774948, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.852, + "recall@10": 0.915, + "n_ret@10": 7.948, + "all@20": 0.854, + "recall@20": 0.916, + "n_ret@20": 9.532 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 46.37206019973382, + "default": false, + "all@5": 0.698, + "recall@5": 0.834, + "n_ret@5": 5.0, + "all@10": 0.82, + "recall@10": 0.905, + "n_ret@10": 10.0, + "all@20": 0.92, + "recall@20": 0.958, + "n_ret@20": 19.564 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 49.35107280139346, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.944, + "recall@20": 0.97, + "n_ret@20": 19.974 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 45.939895599382, + "default": false, + "all@5": 0.778, + "recall@5": 0.874, + "n_ret@5": 5.0, + "all@10": 0.872, + "recall@10": 0.931, + "n_ret@10": 10.0, + "all@20": 0.916, + "recall@20": 0.956, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 44.539168199757114, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.892, + "recall@10": 0.937, + "n_ret@10": 9.536, + "all@20": 0.914, + "recall@20": 0.948, + "n_ret@20": 16.794 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.233273598947562, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.852, + "recall@10": 0.914, + "n_ret@10": 7.462, + "all@20": 0.852, + "recall@20": 0.914, + "n_ret@20": 7.91 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 42.429277798277326, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.658107800409198, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.852, + "recall@10": 0.915, + "n_ret@10": 7.948, + "all@20": 0.854, + "recall@20": 0.916, + "n_ret@20": 9.532 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 41.374080201028846, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.886, + "recall@10": 0.934, + "n_ret@10": 9.086, + "all@20": 0.908, + "recall@20": 0.945, + "n_ret@20": 13.14 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 44.43524779868312, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.948, + "recall@20": 0.972, + "n_ret@20": 19.926 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 47.89098820067011, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 8.637068200157955, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.938, + "recall@20": 0.965, + "n_ret@20": 19.658 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 43.483818798093125, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.636351399472915, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.918, + "recall@20": 0.957, + "n_ret@20": 19.588 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 10.088635401800275, + "default": false, + "all@5": 0.788, + "recall@5": 0.878, + "n_ret@5": 5.0, + "all@10": 0.888, + "recall@10": 0.939, + "n_ret@10": 10.0, + "all@20": 0.944, + "recall@20": 0.97, + "n_ret@20": 19.974 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 43.014091400196776, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.902, + "recall@20": 0.947, + "n_ret@20": 16.7 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 42.15678039856721, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.9, + "recall@20": 0.946, + "n_ret@20": 15.178 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 10.762002998962998, + "default": false, + "all@5": 0.788, + "recall@5": 0.878, + "n_ret@5": 5.0, + "all@10": 0.888, + "recall@10": 0.939, + "n_ret@10": 10.0, + "all@20": 0.944, + "recall@20": 0.97, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 41.75044420082122, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.896, + "recall@10": 0.939, + "n_ret@10": 9.536, + "all@20": 0.914, + "recall@20": 0.948, + "n_ret@20": 16.794 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 44.54331199883018, + "default": false, + "all@5": 0.778, + "recall@5": 0.874, + "n_ret@5": 5.0, + "all@10": 0.872, + "recall@10": 0.931, + "n_ret@10": 10.0, + "all@20": 0.916, + "recall@20": 0.956, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.6489980013575405, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.852, + "recall@10": 0.915, + "n_ret@10": 8.048, + "all@20": 0.854, + "recall@20": 0.916, + "n_ret@20": 10.266 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 42.16088919970207, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.895, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 43.36007239879109, + "default": false, + "all@5": 0.778, + "recall@5": 0.874, + "n_ret@5": 5.0, + "all@10": 0.872, + "recall@10": 0.931, + "n_ret@10": 10.0, + "all@20": 0.902, + "recall@20": 0.947, + "n_ret@20": 16.7 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.566000998718664, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.74, + "recall@10": 0.864, + "n_ret@10": 10.0, + "all@20": 0.798, + "recall@20": 0.897, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 9.294969800743274, + "default": false, + "all@5": 0.788, + "recall@5": 0.878, + "n_ret@5": 5.0, + "all@10": 0.89, + "recall@10": 0.936, + "n_ret@10": 9.58, + "all@20": 0.908, + "recall@20": 0.945, + "n_ret@20": 17.666 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 43.734463598695584, + "default": false, + "all@5": 0.608, + "recall@5": 0.789, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.862, + "n_ret@10": 10.0, + "all@20": 0.946, + "recall@20": 0.969, + "n_ret@20": 19.712 + } + ], + "lib_selected": { + "5": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "20": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + } + }, + "seconds": 2544.3448207378387 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/hotpotqa__bge__mention.npz b/benchmarks/multihop/results/raw/hotpotqa__bge__mention.npz new file mode 100644 index 0000000..b378adb Binary files /dev/null and b/benchmarks/multihop/results/raw/hotpotqa__bge__mention.npz differ diff --git a/benchmarks/multihop/results/raw/hotpotqa__bge__rerank_scores.npy b/benchmarks/multihop/results/raw/hotpotqa__bge__rerank_scores.npy new file mode 100644 index 0000000..d7f16b2 Binary files /dev/null and b/benchmarks/multihop/results/raw/hotpotqa__bge__rerank_scores.npy differ diff --git a/benchmarks/multihop/results/raw/hotpotqa__bge__vector_top50.npy b/benchmarks/multihop/results/raw/hotpotqa__bge__vector_top50.npy new file mode 100644 index 0000000..c2dd2a3 Binary files /dev/null and b/benchmarks/multihop/results/raw/hotpotqa__bge__vector_top50.npy differ diff --git a/benchmarks/multihop/results/raw/hotpotqa__minilm__baselines.json b/benchmarks/multihop/results/raw/hotpotqa__minilm__baselines.json new file mode 100644 index 0000000..e5e96d1 --- /dev/null +++ b/benchmarks/multihop/results/raw/hotpotqa__minilm__baselines.json @@ -0,0 +1,8 @@ +{ + "dataset": "hotpotqa", + "model": "minilm", + "corpus_size": 14570, + "n_questions": 1500, + "lib_similarity_search_same_top20_set": "200/200", + "seconds": 119.7573504447937 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/hotpotqa__minilm__baselines.npz b/benchmarks/multihop/results/raw/hotpotqa__minilm__baselines.npz new file mode 100644 index 0000000..68d66d8 Binary files /dev/null and b/benchmarks/multihop/results/raw/hotpotqa__minilm__baselines.npz differ diff --git a/benchmarks/multihop/results/raw/hotpotqa__minilm__bm25_top50.npy b/benchmarks/multihop/results/raw/hotpotqa__minilm__bm25_top50.npy new file mode 100644 index 0000000..895a397 Binary files /dev/null and b/benchmarks/multihop/results/raw/hotpotqa__minilm__bm25_top50.npy differ diff --git a/benchmarks/multihop/results/raw/hotpotqa__minilm__ctrl_mention.json b/benchmarks/multihop/results/raw/hotpotqa__minilm__ctrl_mention.json new file mode 100644 index 0000000..10d7aa5 --- /dev/null +++ b/benchmarks/multihop/results/raw/hotpotqa__minilm__ctrl_mention.json @@ -0,0 +1,3274 @@ +{ + "dataset": "hotpotqa", + "model": "minilm", + "variant": "ctrl_mention", + "families": [ + "mention_rand" + ], + "diagnostics": { + "n_edges_undirected_union": 11348, + "n_edges_by_family_directed_or_undirected": 12354, + "mean_degree": 1.5577213452299246, + "median_degree": 1.0, + "max_degree": 29, + "isolated_frac": 0.39958819492107067, + "gold_pairs_directly_linked": 0.0, + "gold_pairs_linked_within_2": 0.0, + "neighbours_of_gold_that_are_in_same_question_context": 0.00026518164942985947 + }, + "ppr_dev_grid": [ + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.42, + "recall@10": 0.685, + "all@20": 0.42, + "recall@20": 0.685 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.418, + "recall@5": 0.683, + "all@10": 0.42, + "recall@10": 0.684, + "all@20": 0.42, + "recall@20": 0.684 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.508, + "recall@10": 0.738, + "all@20": 0.618, + "recall@20": 0.802 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.522, + "recall@10": 0.744, + "all@20": 0.618, + "recall@20": 0.801 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.554, + "recall@10": 0.765, + "all@20": 0.634, + "recall@20": 0.81 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.554, + "recall@10": 0.764, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.41, + "recall@5": 0.679, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.556, + "recall@20": 0.766 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.4, + "recall@5": 0.664, + "all@10": 0.554, + "recall@10": 0.764, + "all@20": 0.556, + "recall@20": 0.765 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.412, + "recall@5": 0.679, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.602, + "recall@20": 0.794 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.41, + "recall@5": 0.673, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.604, + "recall@20": 0.794 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.41, + "recall@5": 0.678, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.634, + "recall@20": 0.81 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.41, + "recall@5": 0.675, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.414, + "recall@5": 0.68, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.412, + "recall@5": 0.678, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.392, + "recall@5": 0.669, + "all@10": 0.55, + "recall@10": 0.764, + "all@20": 0.632, + "recall@20": 0.808 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.348, + "recall@5": 0.631, + "all@10": 0.534, + "recall@10": 0.749, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.402, + "recall@5": 0.673, + "all@10": 0.55, + "recall@10": 0.764, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.376, + "recall@5": 0.652, + "all@10": 0.538, + "recall@10": 0.753, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.406, + "recall@5": 0.676, + "all@10": 0.554, + "recall@10": 0.766, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.402, + "recall@5": 0.671, + "all@10": 0.55, + "recall@10": 0.762, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.414, + "recall@5": 0.68, + "all@10": 0.556, + "recall@10": 0.767, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.412, + "recall@5": 0.678, + "all@10": 0.55, + "recall@10": 0.761, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.372, + "recall@5": 0.653, + "all@10": 0.42, + "recall@10": 0.685, + "all@20": 0.42, + "recall@20": 0.685 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.33, + "recall@5": 0.618, + "all@10": 0.42, + "recall@10": 0.684, + "all@20": 0.42, + "recall@20": 0.684 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.406, + "recall@5": 0.677, + "all@10": 0.472, + "recall@10": 0.719, + "all@20": 0.606, + "recall@20": 0.796 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.41, + "recall@5": 0.679, + "all@10": 0.486, + "recall@10": 0.724, + "all@20": 0.604, + "recall@20": 0.795 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.546, + "recall@10": 0.761, + "all@20": 0.632, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.546, + "recall@10": 0.76, + "all@20": 0.632, + "recall@20": 0.808 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.346, + "recall@5": 0.63, + "all@10": 0.538, + "recall@10": 0.755, + "all@20": 0.556, + "recall@20": 0.766 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.254, + "recall@5": 0.548, + "all@10": 0.496, + "recall@10": 0.728, + "all@20": 0.556, + "recall@20": 0.765 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.394, + "recall@5": 0.671, + "all@10": 0.548, + "recall@10": 0.761, + "all@20": 0.584, + "recall@20": 0.784 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.372, + "recall@5": 0.644, + "all@10": 0.55, + "recall@10": 0.761, + "all@20": 0.59, + "recall@20": 0.786 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.412, + "recall@5": 0.679, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.63, + "recall@20": 0.808 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.404, + "recall@5": 0.67, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.414, + "recall@5": 0.68, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.412, + "recall@5": 0.678, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.302, + "recall@5": 0.599, + "all@10": 0.506, + "recall@10": 0.739, + "all@20": 0.616, + "recall@20": 0.8 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.184, + "recall@5": 0.487, + "all@10": 0.372, + "recall@10": 0.642, + "all@20": 0.59, + "recall@20": 0.786 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.378, + "recall@5": 0.659, + "all@10": 0.544, + "recall@10": 0.761, + "all@20": 0.628, + "recall@20": 0.806 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.308, + "recall@5": 0.592, + "all@10": 0.506, + "recall@10": 0.733, + "all@20": 0.628, + "recall@20": 0.806 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.396, + "recall@5": 0.67, + "all@10": 0.55, + "recall@10": 0.764, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.362, + "recall@5": 0.641, + "all@10": 0.532, + "recall@10": 0.749, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.412, + "recall@5": 0.679, + "all@10": 0.556, + "recall@10": 0.767, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.404, + "recall@5": 0.675, + "all@10": 0.552, + "recall@10": 0.763, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.282, + "recall@5": 0.581, + "all@10": 0.414, + "recall@10": 0.682, + "all@20": 0.42, + "recall@20": 0.685 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.204, + "recall@5": 0.501, + "all@10": 0.412, + "recall@10": 0.677, + "all@20": 0.42, + "recall@20": 0.684 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.364, + "recall@5": 0.645, + "all@10": 0.462, + "recall@10": 0.713, + "all@20": 0.594, + "recall@20": 0.789 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.334, + "recall@5": 0.624, + "all@10": 0.46, + "recall@10": 0.71, + "all@20": 0.596, + "recall@20": 0.79 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.414, + "recall@5": 0.681, + "all@10": 0.538, + "recall@10": 0.757, + "all@20": 0.628, + "recall@20": 0.807 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.416, + "recall@5": 0.682, + "all@10": 0.542, + "recall@10": 0.757, + "all@20": 0.632, + "recall@20": 0.808 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.634, + "recall@20": 0.81 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.214, + "recall@5": 0.501, + "all@10": 0.468, + "recall@10": 0.709, + "all@20": 0.556, + "recall@20": 0.766 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.068, + "recall@5": 0.317, + "all@10": 0.33, + "recall@10": 0.605, + "all@20": 0.554, + "recall@20": 0.764 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.314, + "recall@5": 0.608, + "all@10": 0.526, + "recall@10": 0.748, + "all@20": 0.574, + "recall@20": 0.779 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.24, + "recall@5": 0.538, + "all@10": 0.49, + "recall@10": 0.725, + "all@20": 0.58, + "recall@20": 0.781 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.402, + "recall@5": 0.674, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.626, + "recall@20": 0.806 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.38, + "recall@5": 0.655, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.622, + "recall@20": 0.803 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.412, + "recall@5": 0.679, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.418, + "recall@5": 0.679, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.108, + "recall@5": 0.348, + "all@10": 0.366, + "recall@10": 0.628, + "all@20": 0.568, + "recall@20": 0.775 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.022, + "recall@5": 0.171, + "all@10": 0.106, + "recall@10": 0.377, + "all@20": 0.464, + "recall@20": 0.699 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.246, + "recall@5": 0.55, + "all@10": 0.486, + "recall@10": 0.725, + "all@20": 0.614, + "recall@20": 0.798 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.132, + "recall@5": 0.436, + "all@10": 0.348, + "recall@10": 0.628, + "all@20": 0.596, + "recall@20": 0.789 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.39, + "recall@5": 0.667, + "all@10": 0.546, + "recall@10": 0.762, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.336, + "recall@5": 0.62, + "all@10": 0.514, + "recall@10": 0.738, + "all@20": 0.63, + "recall@20": 0.807 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.402, + "recall@5": 0.673, + "all@10": 0.554, + "recall@10": 0.766, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.396, + "recall@5": 0.667, + "all@10": 0.546, + "recall@10": 0.76, + "all@20": 0.634, + "recall@20": 0.809 + } + ], + "ppr_selected": { + "5": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "20": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + } + }, + "lib_dev_grid": [ + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.987389998161234, + "default": true, + "all@5": 0.382, + "recall@5": 0.664, + "n_ret@5": 5.0, + "all@10": 0.42, + "recall@10": 0.685, + "n_ret@10": 9.068, + "all@20": 0.42, + "recall@20": 0.685, + "n_ret@20": 11.434 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 9.029374001314864, + "default": false, + "all@5": 0.4, + "recall@5": 0.674, + "n_ret@5": 5.0, + "all@10": 0.546, + "recall@10": 0.76, + "n_ret@10": 10.0, + "all@20": 0.556, + "recall@20": 0.766, + "n_ret@20": 18.86 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 39.89868619909976, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.6, + "recall@20": 0.793, + "n_ret@20": 19.992 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 43.362493398133665, + "default": false, + "all@5": 0.418, + "recall@5": 0.683, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 41.3141888033133, + "default": false, + "all@5": 0.418, + "recall@5": 0.683, + "n_ret@5": 5.0, + "all@10": 0.42, + "recall@10": 0.685, + "n_ret@10": 9.068, + "all@20": 0.42, + "recall@20": 0.685, + "n_ret@20": 11.434 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 43.3599624002818, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 8.403384599252604, + "default": false, + "all@5": 0.294, + "recall@5": 0.613, + "n_ret@5": 5.0, + "all@10": 0.344, + "recall@10": 0.644, + "n_ret@10": 9.906, + "all@20": 0.392, + "recall@20": 0.67, + "n_ret@20": 18.688 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 45.00274899892975, + "default": false, + "all@5": 0.398, + "recall@5": 0.673, + "n_ret@5": 5.0, + "all@10": 0.528, + "recall@10": 0.751, + "n_ret@10": 10.0, + "all@20": 0.604, + "recall@20": 0.794, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.238218000973575, + "default": false, + "all@5": 0.36, + "recall@5": 0.648, + "n_ret@5": 5.0, + "all@10": 0.504, + "recall@10": 0.738, + "n_ret@10": 10.0, + "all@20": 0.588, + "recall@20": 0.787, + "n_ret@20": 19.976 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.798350200522691, + "default": false, + "all@5": 0.4, + "recall@5": 0.674, + "n_ret@5": 5.0, + "all@10": 0.546, + "recall@10": 0.76, + "n_ret@10": 10.0, + "all@20": 0.556, + "recall@20": 0.766, + "n_ret@20": 17.442 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 39.1442842008546, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 7.15493699896615, + "default": false, + "all@5": 0.392, + "recall@5": 0.67, + "n_ret@5": 5.0, + "all@10": 0.416, + "recall@10": 0.683, + "n_ret@10": 9.906, + "all@20": 0.42, + "recall@20": 0.685, + "n_ret@20": 18.688 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.567230200045742, + "default": false, + "all@5": 0.418, + "recall@5": 0.683, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.6, + "recall@20": 0.793, + "n_ret@20": 19.992 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 34.98334680055268, + "default": false, + "all@5": 0.418, + "recall@5": 0.683, + "n_ret@5": 5.0, + "all@10": 0.42, + "recall@10": 0.685, + "n_ret@10": 9.3, + "all@20": 0.42, + "recall@20": 0.685, + "n_ret@20": 12.316 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 33.87356400187127, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.556, + "recall@20": 0.766, + "n_ret@20": 19.066 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 33.54672219825443, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.6, + "recall@20": 0.793, + "n_ret@20": 19.976 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 6.2200327987084165, + "default": false, + "all@5": 0.418, + "recall@5": 0.683, + "n_ret@5": 5.0, + "all@10": 0.42, + "recall@10": 0.685, + "n_ret@10": 9.3, + "all@20": 0.42, + "recall@20": 0.685, + "n_ret@20": 12.316 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 33.30078340170439, + "default": false, + "all@5": 0.398, + "recall@5": 0.673, + "n_ret@5": 5.0, + "all@10": 0.53, + "recall@10": 0.752, + "n_ret@10": 10.0, + "all@20": 0.554, + "recall@20": 0.765, + "n_ret@20": 19.978 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 10.346550401183777, + "default": false, + "all@5": 0.294, + "recall@5": 0.613, + "n_ret@5": 5.0, + "all@10": 0.382, + "recall@10": 0.673, + "n_ret@10": 10.0, + "all@20": 0.478, + "recall@20": 0.729, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 6.569187002023682, + "default": false, + "all@5": 0.332, + "recall@5": 0.633, + "n_ret@5": 5.0, + "all@10": 0.46, + "recall@10": 0.714, + "n_ret@10": 10.0, + "all@20": 0.538, + "recall@20": 0.756, + "n_ret@20": 19.248 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 33.257701399503276, + "default": false, + "all@5": 0.418, + "recall@5": 0.683, + "n_ret@5": 5.0, + "all@10": 0.554, + "recall@10": 0.764, + "n_ret@10": 10.0, + "all@20": 0.556, + "recall@20": 0.766, + "n_ret@20": 18.86 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 34.63236860057805, + "default": false, + "all@5": 0.418, + "recall@5": 0.683, + "n_ret@5": 5.0, + "all@10": 0.554, + "recall@10": 0.764, + "n_ret@10": 10.0, + "all@20": 0.556, + "recall@20": 0.766, + "n_ret@20": 19.066 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 8.189999599591829, + "default": false, + "all@5": 0.418, + "recall@5": 0.683, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.556, + "recall@20": 0.766, + "n_ret@20": 19.984 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 35.73399759957101, + "default": false, + "all@5": 0.418, + "recall@5": 0.683, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.6, + "recall@20": 0.793, + "n_ret@20": 19.998 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 36.61581400001887, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.556, + "recall@20": 0.766, + "n_ret@20": 19.978 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 39.464206598117016, + "default": false, + "all@5": 0.418, + "recall@5": 0.683, + "n_ret@5": 5.0, + "all@10": 0.554, + "recall@10": 0.764, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 43.011161001515575, + "default": false, + "all@5": 0.418, + "recall@5": 0.683, + "n_ret@5": 5.0, + "all@10": 0.42, + "recall@10": 0.685, + "n_ret@10": 9.906, + "all@20": 0.42, + "recall@20": 0.685, + "n_ret@20": 18.688 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 39.55982419941574, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.6, + "recall@20": 0.793, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 35.28023160225712, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.558, + "recall@20": 0.767, + "n_ret@20": 19.984 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.656727200141177, + "default": false, + "all@5": 0.34, + "recall@5": 0.638, + "n_ret@5": 5.0, + "all@10": 0.486, + "recall@10": 0.729, + "n_ret@10": 10.0, + "all@20": 0.59, + "recall@20": 0.788, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 35.34424299921375, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 36.20578300044872, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.6, + "recall@20": 0.793, + "n_ret@20": 19.992 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.340789801673964, + "default": false, + "all@5": 0.4, + "recall@5": 0.674, + "n_ret@5": 5.0, + "all@10": 0.546, + "recall@10": 0.76, + "n_ret@10": 10.0, + "all@20": 0.598, + "recall@20": 0.792, + "n_ret@20": 19.976 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 52.31733760109637, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 41.177195798954926, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.424, + "recall@10": 0.687, + "n_ret@10": 9.922, + "all@20": 0.424, + "recall@20": 0.687, + "n_ret@20": 19.25 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 38.02411640086211, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.6, + "recall@20": 0.793, + "n_ret@20": 19.992 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 10.688293402316049, + "default": false, + "all@5": 0.4, + "recall@5": 0.674, + "n_ret@5": 5.0, + "all@10": 0.42, + "recall@10": 0.685, + "n_ret@10": 9.152, + "all@20": 0.42, + "recall@20": 0.685, + "n_ret@20": 12.49 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 9.978550399653614, + "default": false, + "all@5": 0.34, + "recall@5": 0.638, + "n_ret@5": 5.0, + "all@10": 0.486, + "recall@10": 0.729, + "n_ret@10": 10.0, + "all@20": 0.58, + "recall@20": 0.783, + "n_ret@20": 19.992 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 10.431842001038603, + "default": false, + "all@5": 0.332, + "recall@5": 0.633, + "n_ret@5": 5.0, + "all@10": 0.46, + "recall@10": 0.714, + "n_ret@10": 10.0, + "all@20": 0.56, + "recall@20": 0.772, + "n_ret@20": 19.998 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 51.68011659907643, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 9.818737600697204, + "default": false, + "all@5": 0.34, + "recall@5": 0.638, + "n_ret@5": 5.0, + "all@10": 0.41, + "recall@10": 0.679, + "n_ret@10": 9.068, + "all@20": 0.42, + "recall@20": 0.685, + "n_ret@20": 11.434 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 16.493829600862227, + "default": false, + "all@5": 0.418, + "recall@5": 0.683, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 10.264900399488397, + "default": false, + "all@5": 0.4, + "recall@5": 0.674, + "n_ret@5": 5.0, + "all@10": 0.546, + "recall@10": 0.76, + "n_ret@10": 10.0, + "all@20": 0.598, + "recall@20": 0.792, + "n_ret@20": 19.992 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 54.94846939924173, + "default": false, + "all@5": 0.418, + "recall@5": 0.683, + "n_ret@5": 5.0, + "all@10": 0.554, + "recall@10": 0.764, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 51.090475799981505, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.556, + "recall@20": 0.766, + "n_ret@20": 17.442 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 50.640900401282124, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.42, + "recall@10": 0.685, + "n_ret@10": 9.068, + "all@20": 0.42, + "recall@20": 0.685, + "n_ret@20": 11.434 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 50.175432999501936, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 13.758954799850471, + "default": false, + "all@5": 0.294, + "recall@5": 0.613, + "n_ret@5": 5.0, + "all@10": 0.382, + "recall@10": 0.673, + "n_ret@10": 10.0, + "all@20": 0.476, + "recall@20": 0.728, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 47.57435440237168, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.42, + "recall@10": 0.685, + "n_ret@10": 9.906, + "all@20": 0.42, + "recall@20": 0.685, + "n_ret@20": 18.688 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 46.177022799733095, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.558, + "recall@20": 0.767, + "n_ret@20": 19.978 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 46.177075399202295, + "default": false, + "all@5": 0.402, + "recall@5": 0.675, + "n_ret@5": 5.0, + "all@10": 0.546, + "recall@10": 0.76, + "n_ret@10": 10.0, + "all@20": 0.556, + "recall@20": 0.766, + "n_ret@20": 19.248 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 10.237895600148477, + "default": false, + "all@5": 0.418, + "recall@5": 0.683, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 45.3210542013403, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.6, + "recall@20": 0.793, + "n_ret@20": 19.998 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 44.71177560032811, + "default": false, + "all@5": 0.418, + "recall@5": 0.683, + "n_ret@5": 5.0, + "all@10": 0.554, + "recall@10": 0.764, + "n_ret@10": 10.0, + "all@20": 0.556, + "recall@20": 0.766, + "n_ret@20": 19.066 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 9.60816139925737, + "default": false, + "all@5": 0.392, + "recall@5": 0.67, + "n_ret@5": 5.0, + "all@10": 0.538, + "recall@10": 0.756, + "n_ret@10": 10.0, + "all@20": 0.556, + "recall@20": 0.766, + "n_ret@20": 19.978 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 9.847889598459005, + "default": false, + "all@5": 0.332, + "recall@5": 0.633, + "n_ret@5": 5.0, + "all@10": 0.46, + "recall@10": 0.714, + "n_ret@10": 10.0, + "all@20": 0.564, + "recall@20": 0.774, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 53.217779000871815, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 50.06842240050901, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 47.08269439963624, + "default": false, + "all@5": 0.398, + "recall@5": 0.673, + "n_ret@5": 5.0, + "all@10": 0.416, + "recall@10": 0.683, + "n_ret@10": 9.906, + "all@20": 0.42, + "recall@20": 0.685, + "n_ret@20": 18.688 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 45.60812120139599, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.6, + "recall@20": 0.793, + "n_ret@20": 19.992 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 49.22705719911028, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.6, + "recall@20": 0.793, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.3, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 15.765672401757909, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 47.518695799284615, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.42, + "recall@10": 0.685, + "n_ret@10": 9.906, + "all@20": 0.42, + "recall@20": 0.685, + "n_ret@20": 18.688 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 48.944558198913, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.556, + "recall@20": 0.766, + "n_ret@20": 19.248 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 50.478545999270864, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.6, + "recall@20": 0.793, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 48.88791339995805, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.558, + "recall@20": 0.767, + "n_ret@20": 19.978 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 45.39428799995221, + "default": false, + "all@5": 0.408, + "recall@5": 0.678, + "n_ret@5": 5.0, + "all@10": 0.55, + "recall@10": 0.762, + "n_ret@10": 10.0, + "all@20": 0.556, + "recall@20": 0.766, + "n_ret@20": 18.86 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 45.036040400736965, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.6, + "recall@20": 0.793, + "n_ret@20": 19.992 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.82876539870631, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.6, + "recall@20": 0.793, + "n_ret@20": 19.992 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 46.08725360000972, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.56, + "recall@20": 0.768, + "n_ret@20": 19.984 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 9.041779399616644, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.6, + "recall@20": 0.793, + "n_ret@20": 19.998 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 50.1556453993544, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 46.49660419940483, + "default": false, + "all@5": 0.402, + "recall@5": 0.675, + "n_ret@5": 5.0, + "all@10": 0.546, + "recall@10": 0.76, + "n_ret@10": 10.0, + "all@20": 0.622, + "recall@20": 0.804, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 47.76059060159605, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.558, + "recall@20": 0.767, + "n_ret@20": 19.984 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.15887199959252, + "default": false, + "all@5": 0.36, + "recall@5": 0.648, + "n_ret@5": 5.0, + "all@10": 0.416, + "recall@10": 0.682, + "n_ret@10": 8.502, + "all@20": 0.42, + "recall@20": 0.685, + "n_ret@20": 9.158 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 43.915332601754926, + "default": false, + "all@5": 0.418, + "recall@5": 0.683, + "n_ret@5": 5.0, + "all@10": 0.42, + "recall@10": 0.685, + "n_ret@10": 9.068, + "all@20": 0.42, + "recall@20": 0.685, + "n_ret@20": 11.434 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 41.721165200229734, + "default": false, + "all@5": 0.418, + "recall@5": 0.683, + "n_ret@5": 5.0, + "all@10": 0.554, + "recall@10": 0.764, + "n_ret@10": 10.0, + "all@20": 0.6, + "recall@20": 0.793, + "n_ret@20": 19.992 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 43.230711199110374, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.6, + "recall@20": 0.793, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 42.530025800690055, + "default": false, + "all@5": 0.408, + "recall@5": 0.678, + "n_ret@5": 5.0, + "all@10": 0.55, + "recall@10": 0.762, + "n_ret@10": 10.0, + "all@20": 0.628, + "recall@20": 0.807, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 41.40522660093848, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.42, + "recall@10": 0.685, + "n_ret@10": 9.906, + "all@20": 0.42, + "recall@20": 0.685, + "n_ret@20": 18.688 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.656284198747016, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.42, + "recall@10": 0.685, + "n_ret@10": 8.502, + "all@20": 0.42, + "recall@20": 0.685, + "n_ret@20": 9.158 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 42.08521940000355, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.4475822016829625, + "default": false, + "all@5": 0.4, + "recall@5": 0.674, + "n_ret@5": 5.0, + "all@10": 0.42, + "recall@10": 0.685, + "n_ret@10": 9.068, + "all@20": 0.42, + "recall@20": 0.685, + "n_ret@20": 11.434 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 41.744623399106786, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.42, + "recall@10": 0.685, + "n_ret@10": 9.3, + "all@20": 0.42, + "recall@20": 0.685, + "n_ret@20": 12.316 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 42.395740200416185, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.6, + "recall@20": 0.793, + "n_ret@20": 19.998 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 45.46189459855668, + "default": false, + "all@5": 0.418, + "recall@5": 0.683, + "n_ret@5": 5.0, + "all@10": 0.554, + "recall@10": 0.764, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 9.45820819993969, + "default": false, + "all@5": 0.418, + "recall@5": 0.683, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.556, + "recall@20": 0.766, + "n_ret@20": 19.978 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 41.72288359934464, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.51931760145817, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.6, + "recall@20": 0.793, + "n_ret@20": 19.992 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 10.161786201409996, + "default": false, + "all@5": 0.294, + "recall@5": 0.613, + "n_ret@5": 5.0, + "all@10": 0.392, + "recall@10": 0.678, + "n_ret@10": 10.0, + "all@20": 0.5, + "recall@20": 0.74, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 41.21735179948155, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.556, + "recall@20": 0.766, + "n_ret@20": 19.066 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 41.20093039947096, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.556, + "recall@20": 0.766, + "n_ret@20": 17.442 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 10.631520000519231, + "default": false, + "all@5": 0.294, + "recall@5": 0.613, + "n_ret@5": 5.0, + "all@10": 0.392, + "recall@10": 0.678, + "n_ret@10": 10.0, + "all@20": 0.502, + "recall@20": 0.741, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 41.426421200390905, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.42, + "recall@10": 0.685, + "n_ret@10": 9.906, + "all@20": 0.42, + "recall@20": 0.685, + "n_ret@20": 18.688 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 40.15078940044623, + "default": false, + "all@5": 0.408, + "recall@5": 0.678, + "n_ret@5": 5.0, + "all@10": 0.55, + "recall@10": 0.762, + "n_ret@10": 10.0, + "all@20": 0.628, + "recall@20": 0.807, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.827237399527802, + "default": false, + "all@5": 0.418, + "recall@5": 0.683, + "n_ret@5": 5.0, + "all@10": 0.42, + "recall@10": 0.685, + "n_ret@10": 9.152, + "all@20": 0.42, + "recall@20": 0.685, + "n_ret@20": 12.49 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 43.907018199330196, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 43.58352700306568, + "default": false, + "all@5": 0.408, + "recall@5": 0.678, + "n_ret@5": 5.0, + "all@10": 0.55, + "recall@10": 0.762, + "n_ret@10": 10.0, + "all@20": 0.556, + "recall@20": 0.766, + "n_ret@20": 19.066 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.249862600117922, + "default": false, + "all@5": 0.4, + "recall@5": 0.674, + "n_ret@5": 5.0, + "all@10": 0.546, + "recall@10": 0.76, + "n_ret@10": 10.0, + "all@20": 0.628, + "recall@20": 0.806, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 10.916192201897502, + "default": false, + "all@5": 0.294, + "recall@5": 0.613, + "n_ret@5": 5.0, + "all@10": 0.334, + "recall@10": 0.639, + "n_ret@10": 9.922, + "all@20": 0.378, + "recall@20": 0.663, + "n_ret@20": 19.25 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 44.69700400053989, + "default": false, + "all@5": 0.418, + "recall@5": 0.683, + "n_ret@5": 5.0, + "all@10": 0.554, + "recall@10": 0.764, + "n_ret@10": 10.0, + "all@20": 0.556, + "recall@20": 0.766, + "n_ret@20": 19.984 + } + ], + "lib_selected": { + "5": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + } + }, + "seconds": 2882.970358610153 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/hotpotqa__minilm__ctrl_mention.npz b/benchmarks/multihop/results/raw/hotpotqa__minilm__ctrl_mention.npz new file mode 100644 index 0000000..4e7751c Binary files /dev/null and b/benchmarks/multihop/results/raw/hotpotqa__minilm__ctrl_mention.npz differ diff --git a/benchmarks/multihop/results/raw/hotpotqa__minilm__entity.json b/benchmarks/multihop/results/raw/hotpotqa__minilm__entity.json new file mode 100644 index 0000000..65eefbb --- /dev/null +++ b/benchmarks/multihop/results/raw/hotpotqa__minilm__entity.json @@ -0,0 +1,3274 @@ +{ + "dataset": "hotpotqa", + "model": "minilm", + "variant": "entity", + "families": [ + "entity" + ], + "diagnostics": { + "n_edges_undirected_union": 59717, + "n_edges_by_family_directed_or_undirected": 59717, + "mean_degree": 8.197254632807137, + "median_degree": 6.0, + "max_degree": 75, + "isolated_frac": 0.14406314344543583, + "gold_pairs_directly_linked": 0.498, + "gold_pairs_linked_within_2": 0.5433333333333333, + "neighbours_of_gold_that_are_in_same_question_context": 0.29354594497084746 + }, + "ppr_dev_grid": [ + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.436, + "recall@5": 0.692, + "all@10": 0.592, + "recall@10": 0.781, + "all@20": 0.666, + "recall@20": 0.819 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.436, + "recall@5": 0.692, + "all@10": 0.604, + "recall@10": 0.787, + "all@20": 0.67, + "recall@20": 0.821 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.428, + "recall@5": 0.688, + "all@10": 0.658, + "recall@10": 0.819, + "all@20": 0.754, + "recall@20": 0.871 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.426, + "recall@5": 0.687, + "all@10": 0.664, + "recall@10": 0.822, + "all@20": 0.752, + "recall@20": 0.87 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.62, + "recall@10": 0.8, + "all@20": 0.704, + "recall@20": 0.846 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.624, + "recall@10": 0.802, + "all@20": 0.706, + "recall@20": 0.847 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.578, + "recall@10": 0.778, + "all@20": 0.658, + "recall@20": 0.821 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.58, + "recall@10": 0.779, + "all@20": 0.662, + "recall@20": 0.823 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.68, + "all@10": 0.576, + "recall@10": 0.775, + "all@20": 0.702, + "recall@20": 0.842 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.416, + "recall@5": 0.679, + "all@10": 0.576, + "recall@10": 0.775, + "all@20": 0.706, + "recall@20": 0.844 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.436, + "recall@5": 0.695, + "all@10": 0.564, + "recall@10": 0.769, + "all@20": 0.746, + "recall@20": 0.866 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.432, + "recall@5": 0.695, + "all@10": 0.564, + "recall@10": 0.769, + "all@20": 0.742, + "recall@20": 0.864 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.442, + "recall@5": 0.7, + "all@10": 0.56, + "recall@10": 0.767, + "all@20": 0.702, + "recall@20": 0.844 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.436, + "recall@5": 0.697, + "all@10": 0.56, + "recall@10": 0.767, + "all@20": 0.702, + "recall@20": 0.844 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.434, + "recall@5": 0.695, + "all@10": 0.558, + "recall@10": 0.766, + "all@20": 0.66, + "recall@20": 0.822 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.434, + "recall@5": 0.695, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.662, + "recall@20": 0.823 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.38, + "recall@5": 0.655, + "all@10": 0.57, + "recall@10": 0.773, + "all@20": 0.65, + "recall@20": 0.817 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.368, + "recall@5": 0.65, + "all@10": 0.574, + "recall@10": 0.774, + "all@20": 0.65, + "recall@20": 0.817 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.404, + "recall@5": 0.673, + "all@10": 0.568, + "recall@10": 0.774, + "all@20": 0.65, + "recall@20": 0.817 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.392, + "recall@5": 0.668, + "all@10": 0.568, + "recall@10": 0.774, + "all@20": 0.65, + "recall@20": 0.817 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.44, + "recall@5": 0.695, + "all@10": 0.564, + "recall@10": 0.774, + "all@20": 0.642, + "recall@20": 0.813 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.43, + "recall@5": 0.69, + "all@10": 0.56, + "recall@10": 0.772, + "all@20": 0.642, + "recall@20": 0.813 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.43, + "recall@5": 0.693, + "all@10": 0.57, + "recall@10": 0.775, + "all@20": 0.636, + "recall@20": 0.81 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.432, + "recall@5": 0.694, + "all@10": 0.568, + "recall@10": 0.774, + "all@20": 0.636, + "recall@20": 0.81 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.466, + "recall@5": 0.707, + "all@10": 0.594, + "recall@10": 0.782, + "all@20": 0.662, + "recall@20": 0.818 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.462, + "recall@5": 0.706, + "all@10": 0.598, + "recall@10": 0.784, + "all@20": 0.66, + "recall@20": 0.817 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.456, + "recall@5": 0.702, + "all@10": 0.65, + "recall@10": 0.813, + "all@20": 0.766, + "recall@20": 0.876 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.452, + "recall@5": 0.7, + "all@10": 0.648, + "recall@10": 0.813, + "all@20": 0.772, + "recall@20": 0.879 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.442, + "recall@5": 0.695, + "all@10": 0.63, + "recall@10": 0.805, + "all@20": 0.718, + "recall@20": 0.853 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.44, + "recall@5": 0.694, + "all@10": 0.634, + "recall@10": 0.808, + "all@20": 0.72, + "recall@20": 0.854 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.584, + "recall@10": 0.782, + "all@20": 0.672, + "recall@20": 0.829 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.586, + "recall@10": 0.783, + "all@20": 0.672, + "recall@20": 0.829 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.406, + "recall@5": 0.668, + "all@10": 0.6, + "recall@10": 0.786, + "all@20": 0.702, + "recall@20": 0.842 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.406, + "recall@5": 0.667, + "all@10": 0.6, + "recall@10": 0.786, + "all@20": 0.704, + "recall@20": 0.843 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.68, + "all@10": 0.596, + "recall@10": 0.785, + "all@20": 0.736, + "recall@20": 0.86 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.422, + "recall@5": 0.68, + "all@10": 0.596, + "recall@10": 0.785, + "all@20": 0.736, + "recall@20": 0.86 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.45, + "recall@5": 0.703, + "all@10": 0.58, + "recall@10": 0.777, + "all@20": 0.716, + "recall@20": 0.851 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.44, + "recall@5": 0.697, + "all@10": 0.582, + "recall@10": 0.778, + "all@20": 0.718, + "recall@20": 0.852 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.446, + "recall@5": 0.7, + "all@10": 0.566, + "recall@10": 0.77, + "all@20": 0.676, + "recall@20": 0.83 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.448, + "recall@5": 0.701, + "all@10": 0.566, + "recall@10": 0.77, + "all@20": 0.676, + "recall@20": 0.83 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.336, + "recall@5": 0.613, + "all@10": 0.558, + "recall@10": 0.762, + "all@20": 0.668, + "recall@20": 0.826 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.328, + "recall@5": 0.607, + "all@10": 0.562, + "recall@10": 0.763, + "all@20": 0.668, + "recall@20": 0.826 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.372, + "recall@5": 0.645, + "all@10": 0.584, + "recall@10": 0.779, + "all@20": 0.67, + "recall@20": 0.827 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.372, + "recall@5": 0.646, + "all@10": 0.584, + "recall@10": 0.779, + "all@20": 0.67, + "recall@20": 0.827 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.416, + "recall@5": 0.681, + "all@10": 0.578, + "recall@10": 0.778, + "all@20": 0.66, + "recall@20": 0.822 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.408, + "recall@5": 0.677, + "all@10": 0.582, + "recall@10": 0.78, + "all@20": 0.66, + "recall@20": 0.822 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.434, + "recall@5": 0.695, + "all@10": 0.562, + "recall@10": 0.772, + "all@20": 0.642, + "recall@20": 0.813 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.442, + "recall@5": 0.699, + "all@10": 0.564, + "recall@10": 0.773, + "all@20": 0.642, + "recall@20": 0.813 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.45, + "recall@5": 0.701, + "all@10": 0.588, + "recall@10": 0.779, + "all@20": 0.65, + "recall@20": 0.812 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.45, + "recall@5": 0.701, + "all@10": 0.594, + "recall@10": 0.782, + "all@20": 0.654, + "recall@20": 0.813 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.468, + "recall@5": 0.71, + "all@10": 0.632, + "recall@10": 0.804, + "all@20": 0.768, + "recall@20": 0.876 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.47, + "recall@5": 0.711, + "all@10": 0.638, + "recall@10": 0.808, + "all@20": 0.764, + "recall@20": 0.874 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.462, + "recall@5": 0.708, + "all@10": 0.632, + "recall@10": 0.807, + "all@20": 0.72, + "recall@20": 0.854 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.462, + "recall@5": 0.708, + "all@10": 0.638, + "recall@10": 0.81, + "all@20": 0.722, + "recall@20": 0.855 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.428, + "recall@5": 0.688, + "all@10": 0.586, + "recall@10": 0.783, + "all@20": 0.68, + "recall@20": 0.833 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.428, + "recall@5": 0.689, + "all@10": 0.59, + "recall@10": 0.785, + "all@20": 0.682, + "recall@20": 0.834 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.346, + "recall@5": 0.62, + "all@10": 0.576, + "recall@10": 0.771, + "all@20": 0.692, + "recall@20": 0.837 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.338, + "recall@5": 0.614, + "all@10": 0.58, + "recall@10": 0.775, + "all@20": 0.698, + "recall@20": 0.84 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.408, + "recall@5": 0.667, + "all@10": 0.598, + "recall@10": 0.785, + "all@20": 0.734, + "recall@20": 0.859 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.406, + "recall@5": 0.662, + "all@10": 0.596, + "recall@10": 0.784, + "all@20": 0.732, + "recall@20": 0.858 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.448, + "recall@5": 0.698, + "all@10": 0.598, + "recall@10": 0.786, + "all@20": 0.72, + "recall@20": 0.853 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.444, + "recall@5": 0.696, + "all@10": 0.598, + "recall@10": 0.786, + "all@20": 0.72, + "recall@20": 0.854 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.444, + "recall@5": 0.7, + "all@10": 0.574, + "recall@10": 0.774, + "all@20": 0.676, + "recall@20": 0.83 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.442, + "recall@5": 0.699, + "all@10": 0.574, + "recall@10": 0.774, + "all@20": 0.68, + "recall@20": 0.832 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.26, + "recall@5": 0.53, + "all@10": 0.466, + "recall@10": 0.705, + "all@20": 0.65, + "recall@20": 0.817 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.254, + "recall@5": 0.53, + "all@10": 0.466, + "recall@10": 0.705, + "all@20": 0.652, + "recall@20": 0.818 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.33, + "recall@5": 0.604, + "all@10": 0.55, + "recall@10": 0.756, + "all@20": 0.668, + "recall@20": 0.826 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.328, + "recall@5": 0.6, + "all@10": 0.55, + "recall@10": 0.756, + "all@20": 0.67, + "recall@20": 0.827 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.39, + "recall@5": 0.659, + "all@10": 0.582, + "recall@10": 0.779, + "all@20": 0.666, + "recall@20": 0.825 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.388, + "recall@5": 0.656, + "all@10": 0.58, + "recall@10": 0.778, + "all@20": 0.67, + "recall@20": 0.827 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.438, + "recall@5": 0.697, + "all@10": 0.566, + "recall@10": 0.772, + "all@20": 0.652, + "recall@20": 0.818 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.434, + "recall@5": 0.695, + "all@10": 0.568, + "recall@10": 0.774, + "all@20": 0.652, + "recall@20": 0.818 + } + ], + "ppr_selected": { + "5": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "20": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + } + }, + "lib_dev_grid": [ + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 12.096273999428377, + "default": true, + "all@5": 0.452, + "recall@5": 0.7, + "n_ret@5": 5.0, + "all@10": 0.582, + "recall@10": 0.772, + "n_ret@10": 9.994, + "all@20": 0.706, + "recall@20": 0.84, + "n_ret@20": 19.918 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.503918600385077, + "default": false, + "all@5": 0.408, + "recall@5": 0.667, + "n_ret@5": 5.0, + "all@10": 0.574, + "recall@10": 0.764, + "n_ret@10": 10.0, + "all@20": 0.728, + "recall@20": 0.854, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 41.24755279824603, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 29.105205400264822, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 60.460107600316405, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.56, + "recall@10": 0.767, + "n_ret@10": 10.0, + "all@20": 0.656, + "recall@20": 0.82, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 36.26034399913624, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.772, + "recall@20": 0.879, + "n_ret@20": 19.986 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 47.60419240046758, + "default": false, + "all@5": 0.478, + "recall@5": 0.718, + "n_ret@5": 5.0, + "all@10": 0.65, + "recall@10": 0.816, + "n_ret@10": 10.0, + "all@20": 0.76, + "recall@20": 0.875, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 35.23642179952003, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.714, + "recall@20": 0.85, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 53.23323400109075, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 7.113785200519487, + "default": false, + "all@5": 0.422, + "recall@5": 0.685, + "n_ret@5": 5.0, + "all@10": 0.558, + "recall@10": 0.766, + "n_ret@10": 10.0, + "all@20": 0.758, + "recall@20": 0.87, + "n_ret@20": 19.986 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 37.418342400342226, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.748, + "recall@20": 0.868, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 48.705274202278815, + "default": false, + "all@5": 0.524, + "recall@5": 0.738, + "n_ret@5": 5.0, + "all@10": 0.694, + "recall@10": 0.837, + "n_ret@10": 10.0, + "all@20": 0.792, + "recall@20": 0.89, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 28.924456198699772, + "default": false, + "all@5": 0.424, + "recall@5": 0.686, + "n_ret@5": 5.0, + "all@10": 0.584, + "recall@10": 0.78, + "n_ret@10": 10.0, + "all@20": 0.666, + "recall@20": 0.826, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 42.121001401217654, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.786, + "recall@20": 0.886, + "n_ret@20": 19.986 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.875902399537154, + "default": false, + "all@5": 0.408, + "recall@5": 0.667, + "n_ret@5": 5.0, + "all@10": 0.564, + "recall@10": 0.757, + "n_ret@10": 9.958, + "all@20": 0.698, + "recall@20": 0.835, + "n_ret@20": 19.038 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 48.10611440089997, + "default": false, + "all@5": 0.442, + "recall@5": 0.696, + "n_ret@5": 5.0, + "all@10": 0.584, + "recall@10": 0.78, + "n_ret@10": 10.0, + "all@20": 0.78, + "recall@20": 0.883, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.80572639987804, + "default": false, + "all@5": 0.426, + "recall@5": 0.687, + "n_ret@5": 5.0, + "all@10": 0.568, + "recall@10": 0.771, + "n_ret@10": 10.0, + "all@20": 0.728, + "recall@20": 0.857, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 36.330995200783946, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 8.190830399049446, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.63, + "recall@10": 0.797, + "n_ret@10": 9.958, + "all@20": 0.712, + "recall@20": 0.843, + "n_ret@20": 19.038 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 40.51005599962082, + "default": false, + "all@5": 0.47, + "recall@5": 0.711, + "n_ret@5": 5.0, + "all@10": 0.684, + "recall@10": 0.829, + "n_ret@10": 9.994, + "all@20": 0.748, + "recall@20": 0.862, + "n_ret@20": 19.918 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 36.76197099906858, + "default": false, + "all@5": 0.424, + "recall@5": 0.686, + "n_ret@5": 5.0, + "all@10": 0.56, + "recall@10": 0.767, + "n_ret@10": 10.0, + "all@20": 0.796, + "recall@20": 0.891, + "n_ret@20": 19.986 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 71.2857833992457, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.712, + "recall@20": 0.849, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 33.18248699745163, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.67, + "recall@10": 0.822, + "n_ret@10": 9.958, + "all@20": 0.734, + "recall@20": 0.855, + "n_ret@20": 19.038 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 49.333327801083215, + "default": false, + "all@5": 0.486, + "recall@5": 0.722, + "n_ret@5": 5.0, + "all@10": 0.658, + "recall@10": 0.82, + "n_ret@10": 10.0, + "all@20": 0.774, + "recall@20": 0.882, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 79.92245539976284, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.636, + "recall@20": 0.81, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 53.196307400474325, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.67, + "recall@10": 0.821, + "n_ret@10": 9.958, + "all@20": 0.734, + "recall@20": 0.855, + "n_ret@20": 19.038 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 11.643046600627713, + "default": false, + "all@5": 0.424, + "recall@5": 0.686, + "n_ret@5": 5.0, + "all@10": 0.584, + "recall@10": 0.78, + "n_ret@10": 10.0, + "all@20": 0.728, + "recall@20": 0.858, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 17.632180601591244, + "default": false, + "all@5": 0.424, + "recall@5": 0.686, + "n_ret@5": 5.0, + "all@10": 0.634, + "recall@10": 0.799, + "n_ret@10": 9.994, + "all@20": 0.716, + "recall@20": 0.845, + "n_ret@20": 19.918 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 58.99574920197483, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.67, + "recall@10": 0.822, + "n_ret@10": 9.994, + "all@20": 0.746, + "recall@20": 0.861, + "n_ret@20": 19.918 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 51.54488340159878, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.754, + "recall@20": 0.87, + "n_ret@20": 19.986 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 49.84020480082836, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.728, + "recall@20": 0.858, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 12.09801640023943, + "default": false, + "all@5": 0.424, + "recall@5": 0.686, + "n_ret@5": 5.0, + "all@10": 0.584, + "recall@10": 0.78, + "n_ret@10": 10.0, + "all@20": 0.666, + "recall@20": 0.826, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 71.95074059942272, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 56.280846800189465, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.748, + "recall@20": 0.867, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 53.784068400389515, + "default": false, + "all@5": 0.524, + "recall@5": 0.738, + "n_ret@5": 5.0, + "all@10": 0.694, + "recall@10": 0.837, + "n_ret@10": 10.0, + "all@20": 0.79, + "recall@20": 0.888, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 23.99466380022932, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.73, + "recall@20": 0.858, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 14.574964398285374, + "default": false, + "all@5": 0.422, + "recall@5": 0.685, + "n_ret@5": 5.0, + "all@10": 0.634, + "recall@10": 0.799, + "n_ret@10": 9.994, + "all@20": 0.716, + "recall@20": 0.845, + "n_ret@20": 19.918 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 50.387703801039606, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.56, + "recall@10": 0.767, + "n_ret@10": 10.0, + "all@20": 0.656, + "recall@20": 0.82, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 70.19729499879759, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 49.223396398709156, + "default": false, + "all@5": 0.424, + "recall@5": 0.686, + "n_ret@5": 5.0, + "all@10": 0.56, + "recall@10": 0.767, + "n_ret@10": 10.0, + "all@20": 0.64, + "recall@20": 0.812, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 10.013047799933702, + "default": false, + "all@5": 0.426, + "recall@5": 0.687, + "n_ret@5": 5.0, + "all@10": 0.568, + "recall@10": 0.771, + "n_ret@10": 10.0, + "all@20": 0.762, + "recall@20": 0.873, + "n_ret@20": 19.986 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 47.324484399636276, + "default": false, + "all@5": 0.442, + "recall@5": 0.696, + "n_ret@5": 5.0, + "all@10": 0.584, + "recall@10": 0.78, + "n_ret@10": 10.0, + "all@20": 0.686, + "recall@20": 0.835, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 30.332870800164528, + "default": false, + "all@5": 0.394, + "recall@5": 0.659, + "n_ret@5": 5.0, + "all@10": 0.552, + "recall@10": 0.753, + "n_ret@10": 10.0, + "all@20": 0.678, + "recall@20": 0.828, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 15.951037200051358, + "default": false, + "all@5": 0.394, + "recall@5": 0.659, + "n_ret@5": 5.0, + "all@10": 0.546, + "recall@10": 0.748, + "n_ret@10": 9.994, + "all@20": 0.67, + "recall@20": 0.82, + "n_ret@20": 19.918 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 59.73629819822963, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.56, + "recall@10": 0.767, + "n_ret@10": 10.0, + "all@20": 0.784, + "recall@20": 0.885, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 51.795406799879856, + "default": false, + "all@5": 0.422, + "recall@5": 0.685, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.776, + "recall@20": 0.881, + "n_ret@20": 19.986 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 53.4796443998348, + "default": false, + "all@5": 0.442, + "recall@5": 0.696, + "n_ret@5": 5.0, + "all@10": 0.66, + "recall@10": 0.817, + "n_ret@10": 9.994, + "all@20": 0.734, + "recall@20": 0.855, + "n_ret@20": 19.918 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 51.88920939946547, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.65, + "recall@10": 0.812, + "n_ret@10": 9.994, + "all@20": 0.724, + "recall@20": 0.85, + "n_ret@20": 19.918 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 58.93901459942572, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.758, + "recall@20": 0.872, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 43.93431599973701, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.56, + "recall@10": 0.767, + "n_ret@10": 10.0, + "all@20": 0.782, + "recall@20": 0.884, + "n_ret@20": 19.986 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 65.70842680102214, + "default": false, + "all@5": 0.524, + "recall@5": 0.738, + "n_ret@5": 5.0, + "all@10": 0.694, + "recall@10": 0.837, + "n_ret@10": 10.0, + "all@20": 0.794, + "recall@20": 0.891, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 48.97305039817002, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.746, + "recall@20": 0.867, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 58.41989400004968, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.766, + "recall@20": 0.876, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 22.421261400100775, + "default": false, + "all@5": 0.422, + "recall@5": 0.685, + "n_ret@5": 5.0, + "all@10": 0.558, + "recall@10": 0.766, + "n_ret@10": 10.0, + "all@20": 0.758, + "recall@20": 0.87, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 60.55015080142766, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.728, + "recall@20": 0.858, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 43.13469359802548, + "default": false, + "all@5": 0.442, + "recall@5": 0.696, + "n_ret@5": 5.0, + "all@10": 0.584, + "recall@10": 0.78, + "n_ret@10": 10.0, + "all@20": 0.756, + "recall@20": 0.873, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 43.13456459902227, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.642, + "recall@20": 0.813, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 48.534640999278054, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.672, + "recall@10": 0.822, + "n_ret@10": 9.994, + "all@20": 0.744, + "recall@20": 0.86, + "n_ret@20": 19.918 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 22.224474198999815, + "default": false, + "all@5": 0.426, + "recall@5": 0.687, + "n_ret@5": 5.0, + "all@10": 0.568, + "recall@10": 0.771, + "n_ret@10": 10.0, + "all@20": 0.728, + "recall@20": 0.857, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 43.457998200901784, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.738, + "recall@20": 0.863, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 48.18315199855715, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.626, + "recall@10": 0.799, + "n_ret@10": 9.994, + "all@20": 0.696, + "recall@20": 0.836, + "n_ret@20": 19.918 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 65.77528280031402, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 51.979147400241345, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.72, + "recall@20": 0.852, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 9.697423198842444, + "default": false, + "all@5": 0.462, + "recall@5": 0.706, + "n_ret@5": 5.0, + "all@10": 0.626, + "recall@10": 0.8, + "n_ret@10": 10.0, + "all@20": 0.758, + "recall@20": 0.872, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 54.134895000373945, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.786, + "recall@20": 0.886, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 9.816906599909998, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.728, + "recall@20": 0.858, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 43.233635999960825, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 43.6056675988948, + "default": false, + "all@5": 0.486, + "recall@5": 0.722, + "n_ret@5": 5.0, + "all@10": 0.658, + "recall@10": 0.82, + "n_ret@10": 10.0, + "all@20": 0.79, + "recall@20": 0.888, + "n_ret@20": 19.986 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 39.88668619841337, + "default": false, + "all@5": 0.442, + "recall@5": 0.696, + "n_ret@5": 5.0, + "all@10": 0.666, + "recall@10": 0.82, + "n_ret@10": 9.958, + "all@20": 0.736, + "recall@20": 0.856, + "n_ret@20": 19.038 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 44.088561601238325, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.672, + "recall@10": 0.823, + "n_ret@10": 9.958, + "all@20": 0.734, + "recall@20": 0.855, + "n_ret@20": 19.038 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 9.662041601142846, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.728, + "recall@20": 0.858, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 44.43233900028281, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 61.31153540138621, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.748, + "recall@20": 0.868, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 54.04040039982647, + "default": false, + "all@5": 0.442, + "recall@5": 0.696, + "n_ret@5": 5.0, + "all@10": 0.584, + "recall@10": 0.78, + "n_ret@10": 10.0, + "all@20": 0.756, + "recall@20": 0.873, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 19.89827020058874, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.758, + "recall@20": 0.87, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.80967799841892, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.612, + "recall@10": 0.789, + "n_ret@10": 9.958, + "all@20": 0.714, + "recall@20": 0.844, + "n_ret@20": 19.038 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 42.11591259960551, + "default": false, + "all@5": 0.53, + "recall@5": 0.741, + "n_ret@5": 5.0, + "all@10": 0.698, + "recall@10": 0.839, + "n_ret@10": 10.0, + "all@20": 0.792, + "recall@20": 0.89, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 55.154467600630596, + "default": false, + "all@5": 0.424, + "recall@5": 0.686, + "n_ret@5": 5.0, + "all@10": 0.56, + "recall@10": 0.767, + "n_ret@10": 10.0, + "all@20": 0.64, + "recall@20": 0.812, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 29.830163000151515, + "default": false, + "all@5": 0.422, + "recall@5": 0.685, + "n_ret@5": 5.0, + "all@10": 0.558, + "recall@10": 0.766, + "n_ret@10": 10.0, + "all@20": 0.638, + "recall@20": 0.811, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 40.322534000035375, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.646, + "recall@10": 0.809, + "n_ret@10": 9.958, + "all@20": 0.732, + "recall@20": 0.854, + "n_ret@20": 19.038 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 41.604245399241336, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.752, + "recall@20": 0.869, + "n_ret@20": 19.986 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 56.23811099934392, + "default": false, + "all@5": 0.422, + "recall@5": 0.685, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.636, + "recall@20": 0.81, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 41.162443601177074, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.758, + "recall@20": 0.874, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 13.465492402552627, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.634, + "recall@10": 0.799, + "n_ret@10": 9.994, + "all@20": 0.716, + "recall@20": 0.845, + "n_ret@20": 19.918 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 16.596781400148757, + "default": false, + "all@5": 0.426, + "recall@5": 0.687, + "n_ret@5": 5.0, + "all@10": 0.568, + "recall@10": 0.771, + "n_ret@10": 10.0, + "all@20": 0.762, + "recall@20": 0.873, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 44.958047798019834, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.64, + "recall@10": 0.806, + "n_ret@10": 9.994, + "all@20": 0.71, + "recall@20": 0.843, + "n_ret@20": 19.918 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 69.61430540005676, + "default": false, + "all@5": 0.424, + "recall@5": 0.686, + "n_ret@5": 5.0, + "all@10": 0.56, + "recall@10": 0.767, + "n_ret@10": 10.0, + "all@20": 0.79, + "recall@20": 0.89, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 84.79976699885447, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 112.77270860085264, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 80.60474380070809, + "default": false, + "all@5": 0.53, + "recall@5": 0.741, + "n_ret@5": 5.0, + "all@10": 0.68, + "recall@10": 0.827, + "n_ret@10": 9.958, + "all@20": 0.734, + "recall@20": 0.855, + "n_ret@20": 19.038 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 124.74612099980004, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 87.59093299985398, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.728, + "recall@20": 0.857, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 87.09544080018532, + "default": false, + "all@5": 0.486, + "recall@5": 0.722, + "n_ret@5": 5.0, + "all@10": 0.658, + "recall@10": 0.82, + "n_ret@10": 10.0, + "all@20": 0.764, + "recall@20": 0.875, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 92.10805980081204, + "default": false, + "all@5": 0.424, + "recall@5": 0.686, + "n_ret@5": 5.0, + "all@10": 0.686, + "recall@10": 0.83, + "n_ret@10": 9.994, + "all@20": 0.744, + "recall@20": 0.86, + "n_ret@20": 19.918 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 25.633311597979628, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.634, + "recall@10": 0.799, + "n_ret@10": 9.994, + "all@20": 0.716, + "recall@20": 0.845, + "n_ret@20": 19.918 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 84.67613700137008, + "default": false, + "all@5": 0.53, + "recall@5": 0.741, + "n_ret@5": 5.0, + "all@10": 0.698, + "recall@10": 0.839, + "n_ret@10": 10.0, + "all@20": 0.786, + "recall@20": 0.886, + "n_ret@20": 19.986 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 18.282444999204017, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.758, + "recall@20": 0.87, + "n_ret@20": 19.986 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 19.38434799935203, + "default": false, + "all@5": 0.408, + "recall@5": 0.667, + "n_ret@5": 5.0, + "all@10": 0.574, + "recall@10": 0.764, + "n_ret@10": 10.0, + "all@20": 0.728, + "recall@20": 0.854, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 34.748928600456566, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.76, + "recall@20": 0.872, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 50.0694196006516, + "default": false, + "all@5": 0.462, + "recall@5": 0.706, + "n_ret@5": 5.0, + "all@10": 0.626, + "recall@10": 0.8, + "n_ret@10": 10.0, + "all@20": 0.75, + "recall@20": 0.868, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 127.69086439930835, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.642, + "recall@20": 0.813, + "n_ret@20": 20.0 + } + ], + "lib_selected": { + "5": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + } + }, + "seconds": 5283.290151119232 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/hotpotqa__minilm__entity.npz b/benchmarks/multihop/results/raw/hotpotqa__minilm__entity.npz new file mode 100644 index 0000000..6b34e82 Binary files /dev/null and b/benchmarks/multihop/results/raw/hotpotqa__minilm__entity.npz differ diff --git a/benchmarks/multihop/results/raw/hotpotqa__minilm__knn.json b/benchmarks/multihop/results/raw/hotpotqa__minilm__knn.json new file mode 100644 index 0000000..77d4eaf --- /dev/null +++ b/benchmarks/multihop/results/raw/hotpotqa__minilm__knn.json @@ -0,0 +1,3274 @@ +{ + "dataset": "hotpotqa", + "model": "minilm", + "variant": "knn", + "families": [ + "knn" + ], + "diagnostics": { + "n_edges_undirected_union": 52527, + "n_edges_by_family_directed_or_undirected": 52527, + "mean_degree": 7.210295126973232, + "median_degree": 6.0, + "max_degree": 41, + "isolated_frac": 0.0, + "gold_pairs_directly_linked": 0.4126666666666667, + "gold_pairs_linked_within_2": 0.5146666666666667, + "neighbours_of_gold_that_are_in_same_question_context": 0.22458679835250345 + }, + "ppr_dev_grid": [ + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.564, + "recall@10": 0.762, + "all@20": 0.662, + "recall@20": 0.818 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.58, + "recall@10": 0.771, + "all@20": 0.672, + "recall@20": 0.823 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.58, + "recall@10": 0.775, + "all@20": 0.696, + "recall@20": 0.84 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.592, + "recall@10": 0.78, + "all@20": 0.7, + "recall@20": 0.842 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.57, + "recall@10": 0.774, + "all@20": 0.666, + "recall@20": 0.825 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.572, + "recall@10": 0.775, + "all@20": 0.664, + "recall@20": 0.824 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.562, + "recall@10": 0.769, + "all@20": 0.648, + "recall@20": 0.816 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.558, + "recall@10": 0.767, + "all@20": 0.65, + "recall@20": 0.817 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.372, + "recall@5": 0.647, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.682, + "recall@20": 0.831 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.364, + "recall@5": 0.639, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.688, + "recall@20": 0.834 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.392, + "recall@5": 0.66, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.686, + "recall@20": 0.834 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.388, + "recall@5": 0.658, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.688, + "recall@20": 0.835 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.404, + "recall@5": 0.67, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.664, + "recall@20": 0.824 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.392, + "recall@5": 0.664, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.664, + "recall@20": 0.824 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.408, + "recall@5": 0.678, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.648, + "recall@20": 0.817 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.412, + "recall@5": 0.68, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.65, + "recall@20": 0.818 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.318, + "recall@5": 0.605, + "all@10": 0.516, + "recall@10": 0.743, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.304, + "recall@5": 0.593, + "all@10": 0.512, + "recall@10": 0.74, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.366, + "recall@5": 0.643, + "all@10": 0.524, + "recall@10": 0.75, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.362, + "recall@5": 0.638, + "all@10": 0.52, + "recall@10": 0.747, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.394, + "recall@5": 0.664, + "all@10": 0.55, + "recall@10": 0.763, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.39, + "recall@5": 0.662, + "all@10": 0.544, + "recall@10": 0.76, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.406, + "recall@5": 0.678, + "all@10": 0.554, + "recall@10": 0.765, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.402, + "recall@5": 0.675, + "all@10": 0.554, + "recall@10": 0.764, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.558, + "recall@10": 0.76, + "all@20": 0.656, + "recall@20": 0.815 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.572, + "recall@10": 0.767, + "all@20": 0.666, + "recall@20": 0.82 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.572, + "recall@10": 0.77, + "all@20": 0.702, + "recall@20": 0.842 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.574, + "recall@10": 0.771, + "all@20": 0.708, + "recall@20": 0.845 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.578, + "recall@10": 0.778, + "all@20": 0.676, + "recall@20": 0.83 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.58, + "recall@10": 0.779, + "all@20": 0.68, + "recall@20": 0.832 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.558, + "recall@10": 0.768, + "all@20": 0.65, + "recall@20": 0.817 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.558, + "recall@10": 0.768, + "all@20": 0.652, + "recall@20": 0.818 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.334, + "recall@5": 0.608, + "all@10": 0.552, + "recall@10": 0.763, + "all@20": 0.68, + "recall@20": 0.83 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.31, + "recall@5": 0.593, + "all@10": 0.552, + "recall@10": 0.763, + "all@20": 0.686, + "recall@20": 0.833 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.362, + "recall@5": 0.637, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.68, + "recall@20": 0.831 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.348, + "recall@5": 0.628, + "all@10": 0.554, + "recall@10": 0.764, + "all@20": 0.684, + "recall@20": 0.832 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.388, + "recall@5": 0.661, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.672, + "recall@20": 0.828 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.386, + "recall@5": 0.66, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.676, + "recall@20": 0.83 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.404, + "recall@5": 0.674, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.65, + "recall@20": 0.817 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.406, + "recall@5": 0.675, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.65, + "recall@20": 0.817 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.25, + "recall@5": 0.549, + "all@10": 0.494, + "recall@10": 0.726, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.246, + "recall@5": 0.542, + "all@10": 0.478, + "recall@10": 0.714, + "all@20": 0.632, + "recall@20": 0.808 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.298, + "recall@5": 0.59, + "all@10": 0.51, + "recall@10": 0.738, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.296, + "recall@5": 0.586, + "all@10": 0.504, + "recall@10": 0.735, + "all@20": 0.632, + "recall@20": 0.808 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.376, + "recall@5": 0.652, + "all@10": 0.526, + "recall@10": 0.752, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.372, + "recall@5": 0.647, + "all@10": 0.52, + "recall@10": 0.748, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.396, + "recall@5": 0.67, + "all@10": 0.55, + "recall@10": 0.764, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.402, + "recall@5": 0.671, + "all@10": 0.55, + "recall@10": 0.762, + "all@20": 0.634, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.4, + "recall@5": 0.671, + "all@10": 0.552, + "recall@10": 0.757, + "all@20": 0.652, + "recall@20": 0.813 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.396, + "recall@5": 0.667, + "all@10": 0.56, + "recall@10": 0.762, + "all@20": 0.66, + "recall@20": 0.817 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.41, + "recall@5": 0.677, + "all@10": 0.564, + "recall@10": 0.765, + "all@20": 0.692, + "recall@20": 0.834 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.406, + "recall@5": 0.674, + "all@10": 0.566, + "recall@10": 0.765, + "all@20": 0.692, + "recall@20": 0.833 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.414, + "recall@5": 0.68, + "all@10": 0.572, + "recall@10": 0.774, + "all@20": 0.682, + "recall@20": 0.833 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.412, + "recall@5": 0.677, + "all@10": 0.574, + "recall@10": 0.775, + "all@20": 0.68, + "recall@20": 0.832 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.416, + "recall@5": 0.682, + "all@10": 0.552, + "recall@10": 0.765, + "all@20": 0.654, + "recall@20": 0.819 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.416, + "recall@5": 0.681, + "all@10": 0.554, + "recall@10": 0.766, + "all@20": 0.656, + "recall@20": 0.82 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.288, + "recall@5": 0.579, + "all@10": 0.558, + "recall@10": 0.766, + "all@20": 0.666, + "recall@20": 0.823 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.274, + "recall@5": 0.568, + "all@10": 0.554, + "recall@10": 0.764, + "all@20": 0.676, + "recall@20": 0.826 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.342, + "recall@5": 0.615, + "all@10": 0.554, + "recall@10": 0.764, + "all@20": 0.68, + "recall@20": 0.83 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.314, + "recall@5": 0.602, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.682, + "recall@20": 0.831 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.376, + "recall@5": 0.651, + "all@10": 0.552, + "recall@10": 0.764, + "all@20": 0.676, + "recall@20": 0.829 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.374, + "recall@5": 0.646, + "all@10": 0.55, + "recall@10": 0.763, + "all@20": 0.67, + "recall@20": 0.826 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.402, + "recall@5": 0.672, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.65, + "recall@20": 0.817 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.4, + "recall@5": 0.671, + "all@10": 0.556, + "recall@10": 0.765, + "all@20": 0.652, + "recall@20": 0.818 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.224, + "recall@5": 0.519, + "all@10": 0.454, + "recall@10": 0.693, + "all@20": 0.636, + "recall@20": 0.809 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.216, + "recall@5": 0.506, + "all@10": 0.45, + "recall@10": 0.687, + "all@20": 0.64, + "recall@20": 0.811 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.26, + "recall@5": 0.559, + "all@10": 0.5, + "recall@10": 0.728, + "all@20": 0.638, + "recall@20": 0.811 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.256, + "recall@5": 0.549, + "all@10": 0.488, + "recall@10": 0.721, + "all@20": 0.636, + "recall@20": 0.81 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.36, + "recall@5": 0.638, + "all@10": 0.51, + "recall@10": 0.743, + "all@20": 0.638, + "recall@20": 0.811 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.354, + "recall@5": 0.628, + "all@10": 0.51, + "recall@10": 0.741, + "all@20": 0.636, + "recall@20": 0.81 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.392, + "recall@5": 0.666, + "all@10": 0.546, + "recall@10": 0.762, + "all@20": 0.63, + "recall@20": 0.807 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": true + }, + "all@5": 0.396, + "recall@5": 0.667, + "all@10": 0.55, + "recall@10": 0.763, + "all@20": 0.632, + "recall@20": 0.808 + } + ], + "ppr_selected": { + "5": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "20": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + } + }, + "lib_dev_grid": [ + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 10.007694001193158, + "default": true, + "all@5": 0.374, + "recall@5": 0.658, + "n_ret@5": 5.0, + "all@10": 0.554, + "recall@10": 0.76, + "n_ret@10": 10.0, + "all@20": 0.65, + "recall@20": 0.814, + "n_ret@20": 19.986 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.883520800387487, + "default": false, + "all@5": 0.294, + "recall@5": 0.605, + "n_ret@5": 5.0, + "all@10": 0.514, + "recall@10": 0.735, + "n_ret@10": 10.0, + "all@20": 0.688, + "recall@20": 0.833, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 34.73906319972593, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 19.270320998854004, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 48.14305059995968, + "default": false, + "all@5": 0.424, + "recall@5": 0.686, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.636, + "recall@20": 0.81, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 32.91202459880151, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.68, + "recall@20": 0.831, + "n_ret@20": 19.99 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 38.37710099981632, + "default": false, + "all@5": 0.424, + "recall@5": 0.688, + "n_ret@5": 5.0, + "all@10": 0.574, + "recall@10": 0.776, + "n_ret@10": 10.0, + "all@20": 0.67, + "recall@20": 0.828, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 32.03303339902777, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.66, + "recall@20": 0.823, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 41.76664640009403, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 7.4156300004106015, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.704, + "recall@20": 0.842, + "n_ret@20": 19.99 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 31.818526599090546, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.682, + "recall@20": 0.834, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 40.36382720107213, + "default": false, + "all@5": 0.404, + "recall@5": 0.677, + "n_ret@5": 5.0, + "all@10": 0.57, + "recall@10": 0.774, + "n_ret@10": 10.0, + "all@20": 0.666, + "recall@20": 0.826, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 15.956608599401077, + "default": false, + "all@5": 0.424, + "recall@5": 0.686, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.638, + "recall@20": 0.811, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 33.66823660081718, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.696, + "recall@20": 0.839, + "n_ret@20": 19.99 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.36150639958214, + "default": false, + "all@5": 0.294, + "recall@5": 0.605, + "n_ret@5": 5.0, + "all@10": 0.512, + "recall@10": 0.733, + "n_ret@10": 10.0, + "all@20": 0.668, + "recall@20": 0.822, + "n_ret@20": 19.68 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 43.166914601344615, + "default": false, + "all@5": 0.422, + "recall@5": 0.686, + "n_ret@5": 5.0, + "all@10": 0.558, + "recall@10": 0.767, + "n_ret@10": 10.0, + "all@20": 0.692, + "recall@20": 0.838, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 9.668889200664125, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.56, + "recall@10": 0.767, + "n_ret@10": 10.0, + "all@20": 0.706, + "recall@20": 0.845, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 43.20189699833281, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 6.022533199749887, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.596, + "recall@10": 0.782, + "n_ret@10": 10.0, + "all@20": 0.672, + "recall@20": 0.823, + "n_ret@20": 19.68 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 35.91024900134653, + "default": false, + "all@5": 0.426, + "recall@5": 0.688, + "n_ret@5": 5.0, + "all@10": 0.61, + "recall@10": 0.792, + "n_ret@10": 10.0, + "all@20": 0.68, + "recall@20": 0.829, + "n_ret@20": 19.986 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 31.074730800581165, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.714, + "recall@20": 0.848, + "n_ret@20": 19.99 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 43.45662660046946, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.66, + "recall@20": 0.823, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 33.75970740080811, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.584, + "recall@10": 0.779, + "n_ret@10": 10.0, + "all@20": 0.678, + "recall@20": 0.828, + "n_ret@20": 19.68 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 40.563729198649526, + "default": false, + "all@5": 0.422, + "recall@5": 0.687, + "n_ret@5": 5.0, + "all@10": 0.592, + "recall@10": 0.785, + "n_ret@10": 10.0, + "all@20": 0.7, + "recall@20": 0.843, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 59.28304100094829, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 34.52858059795108, + "default": false, + "all@5": 0.424, + "recall@5": 0.686, + "n_ret@5": 5.0, + "all@10": 0.588, + "recall@10": 0.778, + "n_ret@10": 10.0, + "all@20": 0.68, + "recall@20": 0.828, + "n_ret@20": 19.68 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 12.425727197784, + "default": false, + "all@5": 0.424, + "recall@5": 0.686, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.69, + "recall@20": 0.838, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 13.958025798550807, + "default": false, + "all@5": 0.424, + "recall@5": 0.686, + "n_ret@5": 5.0, + "all@10": 0.596, + "recall@10": 0.782, + "n_ret@10": 10.0, + "all@20": 0.678, + "recall@20": 0.826, + "n_ret@20": 19.986 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 53.17408180027269, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.592, + "recall@10": 0.78, + "n_ret@10": 10.0, + "all@20": 0.68, + "recall@20": 0.829, + "n_ret@20": 19.986 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 52.6195988021791, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.674, + "recall@20": 0.828, + "n_ret@20": 19.99 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 52.260424800915644, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.668, + "recall@20": 0.827, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 12.870334000210278, + "default": false, + "all@5": 0.424, + "recall@5": 0.686, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.638, + "recall@20": 0.811, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 68.29155560093932, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 53.801806002040394, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.672, + "recall@20": 0.827, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 52.130022198776715, + "default": false, + "all@5": 0.404, + "recall@5": 0.677, + "n_ret@5": 5.0, + "all@10": 0.57, + "recall@10": 0.774, + "n_ret@10": 10.0, + "all@20": 0.666, + "recall@20": 0.824, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 18.004079399746843, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.704, + "recall@20": 0.844, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 13.360770200262778, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.596, + "recall@10": 0.782, + "n_ret@10": 10.0, + "all@20": 0.674, + "recall@20": 0.824, + "n_ret@20": 19.986 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 45.7738343992969, + "default": false, + "all@5": 0.424, + "recall@5": 0.686, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.636, + "recall@20": 0.81, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 60.499449999653734, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 45.20547560136765, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.636, + "recall@20": 0.81, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 9.66725720197428, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.56, + "recall@10": 0.767, + "n_ret@10": 10.0, + "all@20": 0.71, + "recall@20": 0.846, + "n_ret@20": 19.99 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 46.54201679874677, + "default": false, + "all@5": 0.422, + "recall@5": 0.686, + "n_ret@5": 5.0, + "all@10": 0.558, + "recall@10": 0.767, + "n_ret@10": 10.0, + "all@20": 0.66, + "recall@20": 0.822, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 22.82728579849936, + "default": false, + "all@5": 0.294, + "recall@5": 0.605, + "n_ret@5": 5.0, + "all@10": 0.462, + "recall@10": 0.709, + "n_ret@10": 10.0, + "all@20": 0.564, + "recall@20": 0.771, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 12.48558599955868, + "default": false, + "all@5": 0.294, + "recall@5": 0.605, + "n_ret@5": 5.0, + "all@10": 0.464, + "recall@10": 0.709, + "n_ret@10": 10.0, + "all@20": 0.56, + "recall@20": 0.768, + "n_ret@20": 19.986 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 50.930302000720985, + "default": false, + "all@5": 0.424, + "recall@5": 0.686, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.7, + "recall@20": 0.84, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 45.973593002185225, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.692, + "recall@20": 0.838, + "n_ret@20": 19.99 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 48.662607999751344, + "default": false, + "all@5": 0.422, + "recall@5": 0.686, + "n_ret@5": 5.0, + "all@10": 0.582, + "recall@10": 0.778, + "n_ret@10": 10.0, + "all@20": 0.674, + "recall@20": 0.827, + "n_ret@20": 19.986 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 48.85192700242624, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.576, + "recall@10": 0.775, + "n_ret@10": 10.0, + "all@20": 0.67, + "recall@20": 0.825, + "n_ret@20": 19.986 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 55.125571601092815, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.672, + "recall@20": 0.827, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 50.01284260035027, + "default": false, + "all@5": 0.424, + "recall@5": 0.686, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.7, + "recall@20": 0.84, + "n_ret@20": 19.99 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 59.60047319915611, + "default": false, + "all@5": 0.404, + "recall@5": 0.677, + "n_ret@5": 5.0, + "all@10": 0.57, + "recall@10": 0.774, + "n_ret@10": 10.0, + "all@20": 0.666, + "recall@20": 0.826, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 49.06433399976231, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.672, + "recall@20": 0.829, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 51.19966359925456, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.678, + "recall@20": 0.83, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 17.565809999010526, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.704, + "recall@20": 0.842, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 52.352408400853164, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.668, + "recall@20": 0.827, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 43.229230602039024, + "default": false, + "all@5": 0.422, + "recall@5": 0.686, + "n_ret@5": 5.0, + "all@10": 0.558, + "recall@10": 0.767, + "n_ret@10": 10.0, + "all@20": 0.696, + "recall@20": 0.841, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 49.550135401659645, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.636, + "recall@20": 0.81, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 48.77765280031599, + "default": false, + "all@5": 0.424, + "recall@5": 0.686, + "n_ret@5": 5.0, + "all@10": 0.588, + "recall@10": 0.778, + "n_ret@10": 10.0, + "all@20": 0.684, + "recall@20": 0.83, + "n_ret@20": 19.986 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 19.032801400404423, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.56, + "recall@10": 0.767, + "n_ret@10": 10.0, + "all@20": 0.704, + "recall@20": 0.844, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 45.356396399089135, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.672, + "recall@20": 0.829, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 46.11327580106445, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.564, + "recall@10": 0.77, + "n_ret@10": 10.0, + "all@20": 0.662, + "recall@20": 0.821, + "n_ret@20": 19.986 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 54.0647393980762, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 47.58643119910266, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.66, + "recall@20": 0.821, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 9.817642401438206, + "default": false, + "all@5": 0.422, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.602, + "recall@10": 0.789, + "n_ret@10": 10.0, + "all@20": 0.708, + "recall@20": 0.846, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 47.29018900054507, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.696, + "recall@20": 0.839, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 9.930021399399266, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.69, + "recall@20": 0.838, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 42.195513801067136, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 42.520741401356645, + "default": false, + "all@5": 0.422, + "recall@5": 0.687, + "n_ret@5": 5.0, + "all@10": 0.592, + "recall@10": 0.785, + "n_ret@10": 10.0, + "all@20": 0.694, + "recall@20": 0.839, + "n_ret@20": 19.99 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 42.36472880013753, + "default": false, + "all@5": 0.422, + "recall@5": 0.686, + "n_ret@5": 5.0, + "all@10": 0.584, + "recall@10": 0.779, + "n_ret@10": 10.0, + "all@20": 0.68, + "recall@20": 0.829, + "n_ret@20": 19.68 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 41.12671839958057, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.578, + "recall@10": 0.774, + "n_ret@10": 10.0, + "all@20": 0.676, + "recall@20": 0.827, + "n_ret@20": 19.68 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 9.310583201935515, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.69, + "recall@20": 0.838, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 42.55374780099373, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 53.081053800066, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.682, + "recall@20": 0.834, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 48.254396199132316, + "default": false, + "all@5": 0.422, + "recall@5": 0.686, + "n_ret@5": 5.0, + "all@10": 0.558, + "recall@10": 0.767, + "n_ret@10": 10.0, + "all@20": 0.692, + "recall@20": 0.839, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 16.60390400059987, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.704, + "recall@20": 0.842, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.349254400236532, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.576, + "recall@10": 0.771, + "n_ret@10": 10.0, + "all@20": 0.67, + "recall@20": 0.824, + "n_ret@20": 19.68 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 41.37222100177314, + "default": false, + "all@5": 0.4, + "recall@5": 0.675, + "n_ret@5": 5.0, + "all@10": 0.604, + "recall@10": 0.791, + "n_ret@10": 10.0, + "all@20": 0.716, + "recall@20": 0.851, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 52.00125119835138, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.636, + "recall@20": 0.81, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 24.156625600298867, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 43.251429001335055, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.564, + "recall@10": 0.769, + "n_ret@10": 10.0, + "all@20": 0.658, + "recall@20": 0.818, + "n_ret@20": 19.68 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 45.19443519890774, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.67, + "recall@20": 0.826, + "n_ret@20": 19.99 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 50.48719879903365, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 41.96019920089748, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.69, + "recall@20": 0.838, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 11.323992002988234, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.596, + "recall@10": 0.782, + "n_ret@10": 10.0, + "all@20": 0.674, + "recall@20": 0.824, + "n_ret@20": 19.986 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 13.573320201830938, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.56, + "recall@10": 0.767, + "n_ret@10": 10.0, + "all@20": 0.71, + "recall@20": 0.845, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 42.79207419953309, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.574, + "recall@10": 0.775, + "n_ret@10": 10.0, + "all@20": 0.666, + "recall@20": 0.823, + "n_ret@20": 19.986 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 39.74776019947603, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.708, + "recall@20": 0.847, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 39.443983601173386, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 49.9893424018519, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 39.18934299785178, + "default": false, + "all@5": 0.4, + "recall@5": 0.675, + "n_ret@5": 5.0, + "all@10": 0.596, + "recall@10": 0.785, + "n_ret@10": 10.0, + "all@20": 0.678, + "recall@20": 0.828, + "n_ret@20": 19.68 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 50.524574201670475, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 40.25869820313528, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.67, + "recall@20": 0.828, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 40.439221800188534, + "default": false, + "all@5": 0.422, + "recall@5": 0.687, + "n_ret@5": 5.0, + "all@10": 0.592, + "recall@10": 0.785, + "n_ret@10": 10.0, + "all@20": 0.696, + "recall@20": 0.84, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 51.44901659828611, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.606, + "recall@10": 0.79, + "n_ret@10": 10.0, + "all@20": 0.686, + "recall@20": 0.832, + "n_ret@20": 19.986 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 21.944314800784923, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.596, + "recall@10": 0.782, + "n_ret@10": 10.0, + "all@20": 0.674, + "recall@20": 0.824, + "n_ret@20": 19.986 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 78.41790220118128, + "default": false, + "all@5": 0.4, + "recall@5": 0.675, + "n_ret@5": 5.0, + "all@10": 0.604, + "recall@10": 0.791, + "n_ret@10": 10.0, + "all@20": 0.716, + "recall@20": 0.849, + "n_ret@20": 19.99 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 17.1031742002815, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.704, + "recall@20": 0.842, + "n_ret@20": 19.99 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 18.62983520119451, + "default": false, + "all@5": 0.294, + "recall@5": 0.605, + "n_ret@5": 5.0, + "all@10": 0.514, + "recall@10": 0.735, + "n_ret@10": 10.0, + "all@20": 0.688, + "recall@20": 0.833, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 26.51167600089684, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.71, + "recall@20": 0.845, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 36.34505740040913, + "default": false, + "all@5": 0.422, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.6, + "recall@10": 0.788, + "n_ret@10": 10.0, + "all@20": 0.694, + "recall@20": 0.839, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": "w" + }, + "ms_per_query": 100.30847600009292, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.636, + "recall@20": 0.81, + "n_ret@20": 20.0 + } + ], + "lib_selected": { + "5": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + } + }, + "seconds": 4856.67511677742 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/hotpotqa__minilm__knn.npz b/benchmarks/multihop/results/raw/hotpotqa__minilm__knn.npz new file mode 100644 index 0000000..1dbbf69 Binary files /dev/null and b/benchmarks/multihop/results/raw/hotpotqa__minilm__knn.npz differ diff --git a/benchmarks/multihop/results/raw/hotpotqa__minilm__mention.json b/benchmarks/multihop/results/raw/hotpotqa__minilm__mention.json new file mode 100644 index 0000000..ce6e701 --- /dev/null +++ b/benchmarks/multihop/results/raw/hotpotqa__minilm__mention.json @@ -0,0 +1,3274 @@ +{ + "dataset": "hotpotqa", + "model": "minilm", + "variant": "mention", + "families": [ + "mention" + ], + "diagnostics": { + "n_edges_undirected_union": 11348, + "n_edges_by_family_directed_or_undirected": 12354, + "mean_degree": 1.5577213452299246, + "median_degree": 1.0, + "max_degree": 29, + "isolated_frac": 0.39958819492107067, + "gold_pairs_directly_linked": 0.6513333333333333, + "gold_pairs_linked_within_2": 0.6673333333333333, + "neighbours_of_gold_that_are_in_same_question_context": 0.5890391209006267 + }, + "ppr_dev_grid": [ + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.448, + "recall@5": 0.7, + "all@10": 0.738, + "recall@10": 0.853, + "all@20": 0.758, + "recall@20": 0.863 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.48, + "recall@5": 0.715, + "all@10": 0.778, + "recall@10": 0.875, + "all@20": 0.84, + "recall@20": 0.907 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.436, + "recall@5": 0.694, + "all@10": 0.804, + "recall@10": 0.891, + "all@20": 0.862, + "recall@20": 0.923 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.458, + "recall@5": 0.704, + "all@10": 0.816, + "recall@10": 0.897, + "all@20": 0.89, + "recall@20": 0.938 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.424, + "recall@5": 0.688, + "all@10": 0.744, + "recall@10": 0.861, + "all@20": 0.808, + "recall@20": 0.896 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.43, + "recall@5": 0.69, + "all@10": 0.742, + "recall@10": 0.861, + "all@20": 0.808, + "recall@20": 0.897 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.42, + "recall@5": 0.684, + "all@10": 0.616, + "recall@10": 0.796, + "all@20": 0.702, + "recall@20": 0.843 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.422, + "recall@5": 0.685, + "all@10": 0.618, + "recall@10": 0.799, + "all@20": 0.708, + "recall@20": 0.846 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.486, + "recall@5": 0.722, + "all@10": 0.598, + "recall@10": 0.786, + "all@20": 0.828, + "recall@20": 0.905 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.476, + "recall@5": 0.718, + "all@10": 0.634, + "recall@10": 0.804, + "all@20": 0.866, + "recall@20": 0.925 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.476, + "recall@5": 0.718, + "all@10": 0.58, + "recall@10": 0.777, + "all@20": 0.852, + "recall@20": 0.919 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.476, + "recall@5": 0.717, + "all@10": 0.596, + "recall@10": 0.786, + "all@20": 0.88, + "recall@20": 0.934 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.468, + "recall@5": 0.714, + "all@10": 0.57, + "recall@10": 0.772, + "all@20": 0.806, + "recall@20": 0.896 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.468, + "recall@5": 0.714, + "all@10": 0.578, + "recall@10": 0.776, + "all@20": 0.806, + "recall@20": 0.897 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.448, + "recall@5": 0.704, + "all@10": 0.56, + "recall@10": 0.767, + "all@20": 0.7, + "recall@20": 0.843 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.446, + "recall@5": 0.7, + "all@10": 0.562, + "recall@10": 0.768, + "all@20": 0.708, + "recall@20": 0.847 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.464, + "recall@5": 0.704, + "all@10": 0.624, + "recall@10": 0.799, + "all@20": 0.69, + "recall@20": 0.837 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.472, + "recall@5": 0.715, + "all@10": 0.614, + "recall@10": 0.796, + "all@20": 0.712, + "recall@20": 0.848 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.478, + "recall@5": 0.717, + "all@10": 0.604, + "recall@10": 0.79, + "all@20": 0.664, + "recall@20": 0.824 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.49, + "recall@5": 0.725, + "all@10": 0.612, + "recall@10": 0.797, + "all@20": 0.68, + "recall@20": 0.832 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.48, + "recall@5": 0.721, + "all@10": 0.592, + "recall@10": 0.785, + "all@20": 0.652, + "recall@20": 0.818 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.494, + "recall@5": 0.729, + "all@10": 0.598, + "recall@10": 0.79, + "all@20": 0.662, + "recall@20": 0.823 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.452, + "recall@5": 0.707, + "all@10": 0.582, + "recall@10": 0.779, + "all@20": 0.64, + "recall@20": 0.812 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.454, + "recall@5": 0.705, + "all@10": 0.586, + "recall@10": 0.784, + "all@20": 0.642, + "recall@20": 0.813 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.556, + "recall@5": 0.753, + "all@10": 0.732, + "recall@10": 0.85, + "all@20": 0.758, + "recall@20": 0.863 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.574, + "recall@5": 0.764, + "all@10": 0.776, + "recall@10": 0.873, + "all@20": 0.836, + "recall@20": 0.905 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.512, + "recall@5": 0.734, + "all@10": 0.792, + "recall@10": 0.884, + "all@20": 0.862, + "recall@20": 0.923 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.57, + "recall@5": 0.762, + "all@10": 0.814, + "recall@10": 0.896, + "all@20": 0.902, + "recall@20": 0.944 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.464, + "recall@5": 0.711, + "all@10": 0.762, + "recall@10": 0.87, + "all@20": 0.822, + "recall@20": 0.903 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.506, + "recall@5": 0.73, + "all@10": 0.78, + "recall@10": 0.88, + "all@20": 0.834, + "recall@20": 0.91 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.432, + "recall@5": 0.692, + "all@10": 0.636, + "recall@10": 0.807, + "all@20": 0.722, + "recall@20": 0.853 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.45, + "recall@5": 0.7, + "all@10": 0.642, + "recall@10": 0.811, + "all@20": 0.728, + "recall@20": 0.856 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.546, + "recall@5": 0.745, + "all@10": 0.698, + "recall@10": 0.837, + "all@20": 0.822, + "recall@20": 0.902 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.542, + "recall@5": 0.745, + "all@10": 0.712, + "recall@10": 0.844, + "all@20": 0.864, + "recall@20": 0.924 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.522, + "recall@5": 0.74, + "all@10": 0.668, + "recall@10": 0.823, + "all@20": 0.85, + "recall@20": 0.918 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.55, + "recall@5": 0.756, + "all@10": 0.71, + "recall@10": 0.843, + "all@20": 0.888, + "recall@20": 0.937 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.488, + "recall@5": 0.724, + "all@10": 0.612, + "recall@10": 0.794, + "all@20": 0.82, + "recall@20": 0.903 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.52, + "recall@5": 0.742, + "all@10": 0.656, + "recall@10": 0.816, + "all@20": 0.83, + "recall@20": 0.909 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.456, + "recall@5": 0.708, + "all@10": 0.568, + "recall@10": 0.771, + "all@20": 0.718, + "recall@20": 0.852 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.458, + "recall@5": 0.708, + "all@10": 0.588, + "recall@10": 0.782, + "all@20": 0.726, + "recall@20": 0.856 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.46, + "recall@5": 0.681, + "all@10": 0.684, + "recall@10": 0.829, + "all@20": 0.774, + "recall@20": 0.879 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.452, + "recall@5": 0.681, + "all@10": 0.634, + "recall@10": 0.804, + "all@20": 0.798, + "recall@20": 0.892 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.498, + "recall@5": 0.72, + "all@10": 0.664, + "recall@10": 0.819, + "all@20": 0.754, + "recall@20": 0.869 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.494, + "recall@5": 0.726, + "all@10": 0.672, + "recall@10": 0.824, + "all@20": 0.782, + "recall@20": 0.883 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.482, + "recall@5": 0.72, + "all@10": 0.626, + "recall@10": 0.801, + "all@20": 0.7, + "recall@20": 0.842 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.514, + "recall@5": 0.74, + "all@10": 0.636, + "recall@10": 0.809, + "all@20": 0.734, + "recall@20": 0.859 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.46, + "recall@5": 0.711, + "all@10": 0.578, + "recall@10": 0.777, + "all@20": 0.654, + "recall@20": 0.819 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.46, + "recall@5": 0.709, + "all@10": 0.592, + "recall@10": 0.787, + "all@20": 0.666, + "recall@20": 0.825 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.576, + "recall@5": 0.751, + "all@10": 0.72, + "recall@10": 0.842, + "all@20": 0.758, + "recall@20": 0.863 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.576, + "recall@5": 0.765, + "all@10": 0.766, + "recall@10": 0.868, + "all@20": 0.838, + "recall@20": 0.906 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.56, + "recall@5": 0.753, + "all@10": 0.778, + "recall@10": 0.877, + "all@20": 0.85, + "recall@20": 0.917 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.604, + "recall@5": 0.781, + "all@10": 0.798, + "recall@10": 0.887, + "all@20": 0.896, + "recall@20": 0.94 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.508, + "recall@5": 0.733, + "all@10": 0.752, + "recall@10": 0.865, + "all@20": 0.816, + "recall@20": 0.9 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.562, + "recall@5": 0.762, + "all@10": 0.786, + "recall@10": 0.883, + "all@20": 0.844, + "recall@20": 0.915 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.452, + "recall@5": 0.705, + "all@10": 0.656, + "recall@10": 0.817, + "all@20": 0.722, + "recall@20": 0.853 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.47, + "recall@5": 0.713, + "all@10": 0.658, + "recall@10": 0.819, + "all@20": 0.736, + "recall@20": 0.86 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.474, + "recall@5": 0.677, + "all@10": 0.698, + "recall@10": 0.833, + "all@20": 0.816, + "recall@20": 0.899 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.482, + "recall@5": 0.68, + "all@10": 0.698, + "recall@10": 0.837, + "all@20": 0.858, + "recall@20": 0.921 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.526, + "recall@5": 0.731, + "all@10": 0.696, + "recall@10": 0.835, + "all@20": 0.84, + "recall@20": 0.913 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.552, + "recall@5": 0.749, + "all@10": 0.744, + "recall@10": 0.862, + "all@20": 0.88, + "recall@20": 0.933 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.508, + "recall@5": 0.733, + "all@10": 0.644, + "recall@10": 0.81, + "all@20": 0.806, + "recall@20": 0.896 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.544, + "recall@5": 0.754, + "all@10": 0.698, + "recall@10": 0.839, + "all@20": 0.838, + "recall@20": 0.913 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.46, + "recall@5": 0.71, + "all@10": 0.578, + "recall@10": 0.777, + "all@20": 0.714, + "recall@20": 0.85 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.466, + "recall@5": 0.713, + "all@10": 0.61, + "recall@10": 0.793, + "all@20": 0.73, + "recall@20": 0.858 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.318, + "recall@5": 0.532, + "all@10": 0.598, + "recall@10": 0.766, + "all@20": 0.776, + "recall@20": 0.879 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.412, + "recall@5": 0.612, + "all@10": 0.564, + "recall@10": 0.744, + "all@20": 0.776, + "recall@20": 0.88 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.438, + "recall@5": 0.667, + "all@10": 0.662, + "recall@10": 0.818, + "all@20": 0.758, + "recall@20": 0.871 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.456, + "recall@5": 0.685, + "all@10": 0.662, + "recall@10": 0.818, + "all@20": 0.812, + "recall@20": 0.9 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.484, + "recall@5": 0.719, + "all@10": 0.63, + "recall@10": 0.803, + "all@20": 0.722, + "recall@20": 0.853 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.504, + "recall@5": 0.732, + "all@10": 0.668, + "recall@10": 0.823, + "all@20": 0.78, + "recall@20": 0.883 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.454, + "recall@5": 0.707, + "all@10": 0.584, + "recall@10": 0.78, + "all@20": 0.664, + "recall@20": 0.824 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.466, + "recall@5": 0.712, + "all@10": 0.606, + "recall@10": 0.794, + "all@20": 0.696, + "recall@20": 0.84 + } + ], + "ppr_selected": { + "5": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "20": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + } + }, + "lib_dev_grid": [ + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 0.7045161988353357, + "default": true, + "all@5": 0.63, + "recall@5": 0.792, + "n_ret@5": 5.0, + "all@10": 0.758, + "recall@10": 0.863, + "n_ret@10": 8.014, + "all@20": 0.758, + "recall@20": 0.863, + "n_ret@20": 9.312 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 0.5669986017746851, + "default": false, + "all@5": 0.558, + "recall@5": 0.755, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.856, + "n_ret@10": 10.0, + "all@20": 0.834, + "recall@20": 0.908, + "n_ret@20": 16.562 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.162526800762862, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.832, + "recall@20": 0.909, + "n_ret@20": 19.72 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 6.568269399926066, + "default": false, + "all@5": 0.53, + "recall@5": 0.743, + "n_ret@5": 5.0, + "all@10": 0.706, + "recall@10": 0.842, + "n_ret@10": 10.0, + "all@20": 0.82, + "recall@20": 0.903, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.076521799433976, + "default": false, + "all@5": 0.528, + "recall@5": 0.742, + "n_ret@5": 5.0, + "all@10": 0.756, + "recall@10": 0.862, + "n_ret@10": 8.014, + "all@20": 0.758, + "recall@20": 0.863, + "n_ret@20": 9.312 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 7.606956399511546, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 0.7832314023980871, + "default": false, + "all@5": 0.664, + "recall@5": 0.806, + "n_ret@5": 5.0, + "all@10": 0.784, + "recall@10": 0.878, + "n_ret@10": 9.496, + "all@20": 0.834, + "recall@20": 0.903, + "n_ret@20": 16.226 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 6.827820400823839, + "default": false, + "all@5": 0.634, + "recall@5": 0.795, + "n_ret@5": 5.0, + "all@10": 0.804, + "recall@10": 0.891, + "n_ret@10": 10.0, + "all@20": 0.888, + "recall@20": 0.938, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 0.6937971988227218, + "default": false, + "all@5": 0.684, + "recall@5": 0.819, + "n_ret@5": 5.0, + "all@10": 0.816, + "recall@10": 0.896, + "n_ret@10": 10.0, + "all@20": 0.854, + "recall@20": 0.919, + "n_ret@20": 19.52 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 0.7171797996852547, + "default": false, + "all@5": 0.558, + "recall@5": 0.755, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.856, + "n_ret@10": 10.0, + "all@20": 0.834, + "recall@20": 0.908, + "n_ret@20": 15.246 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.940069200121798, + "default": false, + "all@5": 0.422, + "recall@5": 0.685, + "n_ret@5": 5.0, + "all@10": 0.564, + "recall@10": 0.769, + "n_ret@10": 10.0, + "all@20": 0.65, + "recall@20": 0.817, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 0.7250361985061318, + "default": false, + "all@5": 0.568, + "recall@5": 0.761, + "n_ret@5": 5.0, + "all@10": 0.816, + "recall@10": 0.894, + "n_ret@10": 9.496, + "all@20": 0.84, + "recall@20": 0.906, + "n_ret@20": 16.226 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 0.7522220007376745, + "default": false, + "all@5": 0.428, + "recall@5": 0.688, + "n_ret@5": 5.0, + "all@10": 0.572, + "recall@10": 0.773, + "n_ret@10": 10.0, + "all@20": 0.854, + "recall@20": 0.919, + "n_ret@20": 19.72 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 6.716227199067362, + "default": false, + "all@5": 0.502, + "recall@5": 0.727, + "n_ret@5": 5.0, + "all@10": 0.818, + "recall@10": 0.895, + "n_ret@10": 8.834, + "all@20": 0.832, + "recall@20": 0.902, + "n_ret@20": 11.262 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.05975500040222, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.838, + "recall@20": 0.91, + "n_ret@20": 16.888 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.646558202453889, + "default": false, + "all@5": 0.45, + "recall@5": 0.7, + "n_ret@5": 5.0, + "all@10": 0.594, + "recall@10": 0.785, + "n_ret@10": 10.0, + "all@20": 0.848, + "recall@20": 0.917, + "n_ret@20": 19.52 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 0.51097140123602, + "default": false, + "all@5": 0.428, + "recall@5": 0.688, + "n_ret@5": 5.0, + "all@10": 0.812, + "recall@10": 0.892, + "n_ret@10": 8.834, + "all@20": 0.83, + "recall@20": 0.901, + "n_ret@20": 11.262 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 6.567553399247117, + "default": false, + "all@5": 0.634, + "recall@5": 0.795, + "n_ret@5": 5.0, + "all@10": 0.804, + "recall@10": 0.891, + "n_ret@10": 10.0, + "all@20": 0.89, + "recall@20": 0.938, + "n_ret@20": 19.42 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 1.5628433992387727, + "default": false, + "all@5": 0.664, + "recall@5": 0.806, + "n_ret@5": 5.0, + "all@10": 0.802, + "recall@10": 0.888, + "n_ret@10": 10.0, + "all@20": 0.872, + "recall@20": 0.928, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 0.6851301995338872, + "default": false, + "all@5": 0.674, + "recall@5": 0.811, + "n_ret@5": 5.0, + "all@10": 0.818, + "recall@10": 0.897, + "n_ret@10": 10.0, + "all@20": 0.886, + "recall@20": 0.933, + "n_ret@20": 17.68 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.521379799232818, + "default": false, + "all@5": 0.502, + "recall@5": 0.727, + "n_ret@5": 5.0, + "all@10": 0.676, + "recall@10": 0.826, + "n_ret@10": 10.0, + "all@20": 0.836, + "recall@20": 0.909, + "n_ret@20": 16.562 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.358034598175436, + "default": false, + "all@5": 0.502, + "recall@5": 0.727, + "n_ret@5": 5.0, + "all@10": 0.676, + "recall@10": 0.826, + "n_ret@10": 10.0, + "all@20": 0.836, + "recall@20": 0.909, + "n_ret@20": 16.888 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 0.9310452009085566, + "default": false, + "all@5": 0.428, + "recall@5": 0.688, + "n_ret@5": 5.0, + "all@10": 0.574, + "recall@10": 0.774, + "n_ret@10": 10.0, + "all@20": 0.89, + "recall@20": 0.935, + "n_ret@20": 19.516 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 4.714459600159898, + "default": false, + "all@5": 0.53, + "recall@5": 0.743, + "n_ret@5": 5.0, + "all@10": 0.706, + "recall@10": 0.842, + "n_ret@10": 10.0, + "all@20": 0.884, + "recall@20": 0.936, + "n_ret@20": 19.86 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 4.862142400466837, + "default": false, + "all@5": 0.422, + "recall@5": 0.685, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.88, + "recall@20": 0.933, + "n_ret@20": 19.42 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.235039399121888, + "default": false, + "all@5": 0.502, + "recall@5": 0.727, + "n_ret@5": 5.0, + "all@10": 0.676, + "recall@10": 0.826, + "n_ret@10": 10.0, + "all@20": 0.782, + "recall@20": 0.883, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 5.949927201261744, + "default": false, + "all@5": 0.528, + "recall@5": 0.742, + "n_ret@5": 5.0, + "all@10": 0.806, + "recall@10": 0.89, + "n_ret@10": 9.496, + "all@20": 0.838, + "recall@20": 0.906, + "n_ret@20": 16.226 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 7.4949249987257645, + "default": false, + "all@5": 0.422, + "recall@5": 0.685, + "n_ret@5": 5.0, + "all@10": 0.564, + "recall@10": 0.769, + "n_ret@10": 10.0, + "all@20": 0.896, + "recall@20": 0.942, + "n_ret@20": 19.978 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 8.653615199495107, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.896, + "recall@20": 0.941, + "n_ret@20": 19.516 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 1.051062598126009, + "default": false, + "all@5": 0.68, + "recall@5": 0.817, + "n_ret@5": 5.0, + "all@10": 0.816, + "recall@10": 0.896, + "n_ret@10": 10.0, + "all@20": 0.858, + "recall@20": 0.921, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 9.678072199807502, + "default": false, + "all@5": 0.422, + "recall@5": 0.685, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.642, + "recall@20": 0.813, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 9.781774000148289, + "default": false, + "all@5": 0.45, + "recall@5": 0.7, + "n_ret@5": 5.0, + "all@10": 0.594, + "recall@10": 0.785, + "n_ret@10": 10.0, + "all@20": 0.848, + "recall@20": 0.917, + "n_ret@20": 19.72 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 1.0785455998266116, + "default": false, + "all@5": 0.558, + "recall@5": 0.755, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.856, + "n_ret@10": 10.0, + "all@20": 0.854, + "recall@20": 0.919, + "n_ret@20": 19.52 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 10.231653400463983, + "default": false, + "all@5": 0.422, + "recall@5": 0.685, + "n_ret@5": 5.0, + "all@10": 0.564, + "recall@10": 0.769, + "n_ret@10": 10.0, + "all@20": 0.65, + "recall@20": 0.817, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 7.1576721991878, + "default": false, + "all@5": 0.45, + "recall@5": 0.7, + "n_ret@5": 5.0, + "all@10": 0.804, + "recall@10": 0.889, + "n_ret@10": 9.55, + "all@20": 0.842, + "recall@20": 0.908, + "n_ret@20": 17.278 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 9.68466379889287, + "default": false, + "all@5": 0.422, + "recall@5": 0.685, + "n_ret@5": 5.0, + "all@10": 0.564, + "recall@10": 0.769, + "n_ret@10": 10.0, + "all@20": 0.856, + "recall@20": 0.921, + "n_ret@20": 19.72 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 0.9520795999560505, + "default": false, + "all@5": 0.558, + "recall@5": 0.755, + "n_ret@5": 5.0, + "all@10": 0.758, + "recall@10": 0.863, + "n_ret@10": 8.142, + "all@20": 0.758, + "recall@20": 0.863, + "n_ret@20": 10.026 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 1.145584799349308, + "default": false, + "all@5": 0.68, + "recall@5": 0.817, + "n_ret@5": 5.0, + "all@10": 0.816, + "recall@10": 0.896, + "n_ret@10": 10.0, + "all@20": 0.852, + "recall@20": 0.918, + "n_ret@20": 19.72 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 1.1911342000821605, + "default": false, + "all@5": 0.674, + "recall@5": 0.811, + "n_ret@5": 5.0, + "all@10": 0.818, + "recall@10": 0.897, + "n_ret@10": 10.0, + "all@20": 0.898, + "recall@20": 0.942, + "n_ret@20": 19.86 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 9.515886799315922, + "default": false, + "all@5": 0.422, + "recall@5": 0.685, + "n_ret@5": 5.0, + "all@10": 0.564, + "recall@10": 0.769, + "n_ret@10": 10.0, + "all@20": 0.65, + "recall@20": 0.817, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 0.9294956013327464, + "default": false, + "all@5": 0.68, + "recall@5": 0.817, + "n_ret@5": 5.0, + "all@10": 0.758, + "recall@10": 0.863, + "n_ret@10": 8.014, + "all@20": 0.758, + "recall@20": 0.863, + "n_ret@20": 9.312 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 2.3928662004182115, + "default": false, + "all@5": 0.428, + "recall@5": 0.688, + "n_ret@5": 5.0, + "all@10": 0.574, + "recall@10": 0.774, + "n_ret@10": 10.0, + "all@20": 0.684, + "recall@20": 0.834, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 0.9155509987613186, + "default": false, + "all@5": 0.558, + "recall@5": 0.755, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.856, + "n_ret@10": 10.0, + "all@20": 0.854, + "recall@20": 0.919, + "n_ret@20": 19.72 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 9.467846798943356, + "default": false, + "all@5": 0.528, + "recall@5": 0.742, + "n_ret@5": 5.0, + "all@10": 0.7, + "recall@10": 0.839, + "n_ret@10": 10.0, + "all@20": 0.822, + "recall@20": 0.904, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 10.164252001908608, + "default": false, + "all@5": 0.422, + "recall@5": 0.685, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.836, + "recall@20": 0.909, + "n_ret@20": 15.246 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.508445998420939, + "default": false, + "all@5": 0.45, + "recall@5": 0.7, + "n_ret@5": 5.0, + "all@10": 0.756, + "recall@10": 0.862, + "n_ret@10": 8.014, + "all@20": 0.758, + "recall@20": 0.863, + "n_ret@20": 9.312 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 10.98228360072244, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 1.5317041986854747, + "default": false, + "all@5": 0.664, + "recall@5": 0.806, + "n_ret@5": 5.0, + "all@10": 0.802, + "recall@10": 0.888, + "n_ret@10": 10.0, + "all@20": 0.874, + "recall@20": 0.929, + "n_ret@20": 19.978 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 9.782331000664271, + "default": false, + "all@5": 0.45, + "recall@5": 0.7, + "n_ret@5": 5.0, + "all@10": 0.806, + "recall@10": 0.89, + "n_ret@10": 9.496, + "all@20": 0.838, + "recall@20": 0.906, + "n_ret@20": 16.226 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 11.623088400345296, + "default": false, + "all@5": 0.45, + "recall@5": 0.7, + "n_ret@5": 5.0, + "all@10": 0.598, + "recall@10": 0.787, + "n_ret@10": 10.0, + "all@20": 0.878, + "recall@20": 0.932, + "n_ret@20": 19.42 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 8.933985398500226, + "default": false, + "all@5": 0.634, + "recall@5": 0.795, + "n_ret@5": 5.0, + "all@10": 0.81, + "recall@10": 0.894, + "n_ret@10": 10.0, + "all@20": 0.89, + "recall@20": 0.937, + "n_ret@20": 17.68 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 1.1976331993937492, + "default": false, + "all@5": 0.428, + "recall@5": 0.688, + "n_ret@5": 5.0, + "all@10": 0.572, + "recall@10": 0.773, + "n_ret@10": 10.0, + "all@20": 0.68, + "recall@20": 0.832, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 10.006813399377279, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.894, + "recall@20": 0.941, + "n_ret@20": 19.86 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.42452040174976, + "default": false, + "all@5": 0.528, + "recall@5": 0.742, + "n_ret@5": 5.0, + "all@10": 0.698, + "recall@10": 0.837, + "n_ret@10": 10.0, + "all@20": 0.838, + "recall@20": 0.91, + "n_ret@20": 16.888 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 0.8280999998096377, + "default": false, + "all@5": 0.568, + "recall@5": 0.761, + "n_ret@5": 5.0, + "all@10": 0.744, + "recall@10": 0.861, + "n_ret@10": 10.0, + "all@20": 0.89, + "recall@20": 0.935, + "n_ret@20": 19.42 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 0.7600302003556862, + "default": false, + "all@5": 0.674, + "recall@5": 0.811, + "n_ret@5": 5.0, + "all@10": 0.818, + "recall@10": 0.897, + "n_ret@10": 10.0, + "all@20": 0.896, + "recall@20": 0.941, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 6.6052754000993446, + "default": false, + "all@5": 0.422, + "recall@5": 0.685, + "n_ret@5": 5.0, + "all@10": 0.564, + "recall@10": 0.769, + "n_ret@10": 10.0, + "all@20": 0.65, + "recall@20": 0.817, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 8.240677198278718, + "default": false, + "all@5": 0.45, + "recall@5": 0.7, + "n_ret@5": 5.0, + "all@10": 0.598, + "recall@10": 0.787, + "n_ret@10": 10.0, + "all@20": 0.712, + "recall@20": 0.848, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 7.98493480158504, + "default": false, + "all@5": 0.634, + "recall@5": 0.795, + "n_ret@5": 5.0, + "all@10": 0.812, + "recall@10": 0.893, + "n_ret@10": 9.496, + "all@20": 0.84, + "recall@20": 0.907, + "n_ret@20": 16.226 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.688490997883491, + "default": false, + "all@5": 0.422, + "recall@5": 0.685, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.842, + "recall@20": 0.914, + "n_ret@20": 19.74 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 8.930097399861552, + "default": false, + "all@5": 0.422, + "recall@5": 0.685, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.854, + "recall@20": 0.921, + "n_ret@20": 19.976 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.3, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 2.111152600380592, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.634, + "recall@20": 0.809, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 7.537661398993805, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.798, + "recall@10": 0.886, + "n_ret@10": 9.496, + "all@20": 0.838, + "recall@20": 0.906, + "n_ret@20": 16.226 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 8.453675200347789, + "default": false, + "all@5": 0.422, + "recall@5": 0.685, + "n_ret@5": 5.0, + "all@10": 0.564, + "recall@10": 0.769, + "n_ret@10": 10.0, + "all@20": 0.892, + "recall@20": 0.938, + "n_ret@20": 17.68 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 7.818959400174208, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.894, + "recall@20": 0.941, + "n_ret@20": 19.976 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 5.298991399118677, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.87, + "recall@20": 0.928, + "n_ret@20": 19.42 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.445435201283544, + "default": false, + "all@5": 0.638, + "recall@5": 0.797, + "n_ret@5": 5.0, + "all@10": 0.81, + "recall@10": 0.893, + "n_ret@10": 10.0, + "all@20": 0.838, + "recall@20": 0.91, + "n_ret@20": 16.562 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.162870999774896, + "default": false, + "all@5": 0.45, + "recall@5": 0.7, + "n_ret@5": 5.0, + "all@10": 0.594, + "recall@10": 0.785, + "n_ret@10": 10.0, + "all@20": 0.848, + "recall@20": 0.917, + "n_ret@20": 19.74 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 0.9560778003651649, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.854, + "recall@20": 0.919, + "n_ret@20": 19.72 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 9.328439400414936, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.874, + "recall@20": 0.93, + "n_ret@20": 19.516 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 1.4181053998181596, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.896, + "recall@20": 0.941, + "n_ret@20": 19.86 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 10.01976099901367, + "default": false, + "all@5": 0.422, + "recall@5": 0.685, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.642, + "recall@20": 0.813, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 8.956458201399073, + "default": false, + "all@5": 0.634, + "recall@5": 0.795, + "n_ret@5": 5.0, + "all@10": 0.81, + "recall@10": 0.894, + "n_ret@10": 10.0, + "all@20": 0.884, + "recall@20": 0.936, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 7.111721800640225, + "default": false, + "all@5": 0.422, + "recall@5": 0.685, + "n_ret@5": 5.0, + "all@10": 0.564, + "recall@10": 0.769, + "n_ret@10": 10.0, + "all@20": 0.896, + "recall@20": 0.941, + "n_ret@20": 19.516 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 0.6890758002409711, + "default": false, + "all@5": 0.684, + "recall@5": 0.819, + "n_ret@5": 5.0, + "all@10": 0.754, + "recall@10": 0.861, + "n_ret@10": 7.504, + "all@20": 0.754, + "recall@20": 0.861, + "n_ret@20": 7.884 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.91118119924795, + "default": false, + "all@5": 0.502, + "recall@5": 0.727, + "n_ret@5": 5.0, + "all@10": 0.758, + "recall@10": 0.863, + "n_ret@10": 8.014, + "all@20": 0.758, + "recall@20": 0.863, + "n_ret@20": 9.312 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.972533801104873, + "default": false, + "all@5": 0.528, + "recall@5": 0.742, + "n_ret@5": 5.0, + "all@10": 0.698, + "recall@10": 0.837, + "n_ret@10": 10.0, + "all@20": 0.848, + "recall@20": 0.917, + "n_ret@20": 19.72 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 8.430698000942357, + "default": false, + "all@5": 0.45, + "recall@5": 0.7, + "n_ret@5": 5.0, + "all@10": 0.598, + "recall@10": 0.787, + "n_ret@10": 10.0, + "all@20": 0.864, + "recall@20": 0.926, + "n_ret@20": 19.976 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 8.514911000151187, + "default": false, + "all@5": 0.638, + "recall@5": 0.797, + "n_ret@5": 5.0, + "all@10": 0.81, + "recall@10": 0.893, + "n_ret@10": 10.0, + "all@20": 0.856, + "recall@20": 0.921, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 8.236777800368145, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.82, + "recall@10": 0.896, + "n_ret@10": 9.496, + "all@20": 0.842, + "recall@20": 0.908, + "n_ret@20": 16.226 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 0.6856006013695151, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.754, + "recall@10": 0.861, + "n_ret@10": 7.504, + "all@20": 0.754, + "recall@20": 0.861, + "n_ret@20": 7.884 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 7.946169399307109, + "default": false, + "all@5": 0.45, + "recall@5": 0.7, + "n_ret@5": 5.0, + "all@10": 0.594, + "recall@10": 0.785, + "n_ret@10": 10.0, + "all@20": 0.704, + "recall@20": 0.844, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 0.7935324013233185, + "default": false, + "all@5": 0.558, + "recall@5": 0.755, + "n_ret@5": 5.0, + "all@10": 0.758, + "recall@10": 0.863, + "n_ret@10": 8.014, + "all@20": 0.758, + "recall@20": 0.863, + "n_ret@20": 9.312 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 8.833568002097309, + "default": false, + "all@5": 0.422, + "recall@5": 0.685, + "n_ret@5": 5.0, + "all@10": 0.808, + "recall@10": 0.89, + "n_ret@10": 8.834, + "all@20": 0.832, + "recall@20": 0.902, + "n_ret@20": 11.262 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 8.741570800892077, + "default": false, + "all@5": 0.422, + "recall@5": 0.685, + "n_ret@5": 5.0, + "all@10": 0.564, + "recall@10": 0.769, + "n_ret@10": 10.0, + "all@20": 0.894, + "recall@20": 0.941, + "n_ret@20": 19.86 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 7.704637999995612, + "default": false, + "all@5": 0.502, + "recall@5": 0.727, + "n_ret@5": 5.0, + "all@10": 0.684, + "recall@10": 0.831, + "n_ret@10": 10.0, + "all@20": 0.8, + "recall@20": 0.893, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 0.798737604636699, + "default": false, + "all@5": 0.428, + "recall@5": 0.688, + "n_ret@5": 5.0, + "all@10": 0.574, + "recall@10": 0.774, + "n_ret@10": 10.0, + "all@20": 0.89, + "recall@20": 0.935, + "n_ret@20": 19.42 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.70329519896768, + "default": false, + "all@5": 0.422, + "recall@5": 0.685, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.642, + "recall@20": 0.813, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.3, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 0.6271901988657191, + "default": false, + "all@5": 0.42, + "recall@5": 0.684, + "n_ret@5": 5.0, + "all@10": 0.556, + "recall@10": 0.765, + "n_ret@10": 10.0, + "all@20": 0.854, + "recall@20": 0.919, + "n_ret@20": 19.74 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 0.815403601503931, + "default": false, + "all@5": 0.664, + "recall@5": 0.806, + "n_ret@5": 5.0, + "all@10": 0.802, + "recall@10": 0.888, + "n_ret@10": 10.0, + "all@20": 0.88, + "recall@20": 0.932, + "n_ret@20": 19.976 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.7686539981514215, + "default": false, + "all@5": 0.45, + "recall@5": 0.7, + "n_ret@5": 5.0, + "all@10": 0.594, + "recall@10": 0.785, + "n_ret@10": 10.0, + "all@20": 0.836, + "recall@20": 0.909, + "n_ret@20": 16.888 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 5.937825400731526, + "default": false, + "all@5": 0.45, + "recall@5": 0.7, + "n_ret@5": 5.0, + "all@10": 0.594, + "recall@10": 0.785, + "n_ret@10": 10.0, + "all@20": 0.836, + "recall@20": 0.909, + "n_ret@20": 15.246 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 0.9947382006794214, + "default": false, + "all@5": 0.664, + "recall@5": 0.806, + "n_ret@5": 5.0, + "all@10": 0.802, + "recall@10": 0.888, + "n_ret@10": 10.0, + "all@20": 0.878, + "recall@20": 0.931, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 5.942922599264421, + "default": false, + "all@5": 0.422, + "recall@5": 0.685, + "n_ret@5": 5.0, + "all@10": 0.822, + "recall@10": 0.897, + "n_ret@10": 9.496, + "all@20": 0.842, + "recall@20": 0.908, + "n_ret@20": 16.226 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.307049399591051, + "default": false, + "all@5": 0.638, + "recall@5": 0.797, + "n_ret@5": 5.0, + "all@10": 0.81, + "recall@10": 0.893, + "n_ret@10": 10.0, + "all@20": 0.856, + "recall@20": 0.921, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 0.6160143993329257, + "default": false, + "all@5": 0.428, + "recall@5": 0.688, + "n_ret@5": 5.0, + "all@10": 0.758, + "recall@10": 0.863, + "n_ret@10": 8.142, + "all@20": 0.758, + "recall@20": 0.863, + "n_ret@20": 10.026 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 5.859152799239382, + "default": false, + "all@5": 0.45, + "recall@5": 0.7, + "n_ret@5": 5.0, + "all@10": 0.598, + "recall@10": 0.787, + "n_ret@10": 10.0, + "all@20": 0.712, + "recall@20": 0.848, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 6.636166800395586, + "default": false, + "all@5": 0.638, + "recall@5": 0.797, + "n_ret@5": 5.0, + "all@10": 0.81, + "recall@10": 0.893, + "n_ret@10": 10.0, + "all@20": 0.836, + "recall@20": 0.909, + "n_ret@20": 16.888 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 0.8375354023883119, + "default": false, + "all@5": 0.558, + "recall@5": 0.755, + "n_ret@5": 5.0, + "all@10": 0.736, + "recall@10": 0.856, + "n_ret@10": 10.0, + "all@20": 0.822, + "recall@20": 0.903, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 1.1795628010295331, + "default": false, + "all@5": 0.664, + "recall@5": 0.806, + "n_ret@5": 5.0, + "all@10": 0.784, + "recall@10": 0.878, + "n_ret@10": 9.55, + "all@20": 0.83, + "recall@20": 0.901, + "n_ret@20": 17.278 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 6.26379219989758, + "default": false, + "all@5": 0.502, + "recall@5": 0.727, + "n_ret@5": 5.0, + "all@10": 0.684, + "recall@10": 0.831, + "n_ret@10": 10.0, + "all@20": 0.894, + "recall@20": 0.94, + "n_ret@20": 19.516 + } + ], + "lib_selected": { + "5": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "20": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + } + }, + "seconds": 516.1581194400787 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/hotpotqa__minilm__mention.npz b/benchmarks/multihop/results/raw/hotpotqa__minilm__mention.npz new file mode 100644 index 0000000..8f980be Binary files /dev/null and b/benchmarks/multihop/results/raw/hotpotqa__minilm__mention.npz differ diff --git a/benchmarks/multihop/results/raw/hotpotqa__minilm__rerank_scores.npy b/benchmarks/multihop/results/raw/hotpotqa__minilm__rerank_scores.npy new file mode 100644 index 0000000..7745e0e Binary files /dev/null and b/benchmarks/multihop/results/raw/hotpotqa__minilm__rerank_scores.npy differ diff --git a/benchmarks/multihop/results/raw/hotpotqa__minilm__vector_top50.npy b/benchmarks/multihop/results/raw/hotpotqa__minilm__vector_top50.npy new file mode 100644 index 0000000..d0a4bc9 Binary files /dev/null and b/benchmarks/multihop/results/raw/hotpotqa__minilm__vector_top50.npy differ diff --git a/benchmarks/multihop/results/raw/musique__bge__baselines.json b/benchmarks/multihop/results/raw/musique__bge__baselines.json new file mode 100644 index 0000000..887f2ef --- /dev/null +++ b/benchmarks/multihop/results/raw/musique__bge__baselines.json @@ -0,0 +1,8 @@ +{ + "dataset": "musique", + "model": "bge", + "corpus_size": 15405, + "n_questions": 1500, + "lib_similarity_search_same_top20_set": "200/200", + "seconds": 65.22805333137512 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/musique__bge__baselines.npz b/benchmarks/multihop/results/raw/musique__bge__baselines.npz new file mode 100644 index 0000000..fbf46ed Binary files /dev/null and b/benchmarks/multihop/results/raw/musique__bge__baselines.npz differ diff --git a/benchmarks/multihop/results/raw/musique__bge__bm25_top50.npy b/benchmarks/multihop/results/raw/musique__bge__bm25_top50.npy new file mode 100644 index 0000000..3ff4907 Binary files /dev/null and b/benchmarks/multihop/results/raw/musique__bge__bm25_top50.npy differ diff --git a/benchmarks/multihop/results/raw/musique__bge__ctrl_mention.json b/benchmarks/multihop/results/raw/musique__bge__ctrl_mention.json new file mode 100644 index 0000000..3856911 --- /dev/null +++ b/benchmarks/multihop/results/raw/musique__bge__ctrl_mention.json @@ -0,0 +1,2017 @@ +{ + "dataset": "musique", + "model": "bge", + "variant": "ctrl_mention", + "families": [ + "mention_rand" + ], + "diagnostics": { + "n_edges_undirected_union": 20272, + "n_edges_by_family_directed_or_undirected": 22465, + "mean_degree": 2.6318727685816294, + "median_degree": 0.0, + "max_degree": 54, + "isolated_frac": 0.5272314183706589, + "gold_pairs_directly_linked": 0.0, + "gold_pairs_linked_within_2": 0.0, + "neighbours_of_gold_that_are_in_same_question_context": 0.001101741312327797 + }, + "ppr_dev_grid": [ + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "all@10": 0.13, + "recall@10": 0.4943333333333333, + "all@20": 0.13, + "recall@20": 0.4948333333333333 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "all@10": 0.13, + "recall@10": 0.4943333333333333, + "all@20": 0.13, + "recall@20": 0.4948333333333333 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "all@10": 0.178, + "recall@10": 0.5353333333333332, + "all@20": 0.264, + "recall@20": 0.6113333333333334 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "all@10": 0.174, + "recall@10": 0.5331666666666667, + "all@20": 0.264, + "recall@20": 0.6106666666666667 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "all@10": 0.202, + "recall@10": 0.5603333333333333, + "all@20": 0.292, + "recall@20": 0.6276666666666667 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "all@10": 0.204, + "recall@10": 0.5586666666666668, + "all@20": 0.296, + "recall@20": 0.6296666666666667 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "all@10": 0.21, + "recall@10": 0.5665, + "all@20": 0.3, + "recall@20": 0.6333333333333333 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "all@10": 0.21, + "recall@10": 0.5665, + "all@20": 0.3, + "recall@20": 0.6333333333333333 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.132, + "recall@5": 0.4925, + "all@10": 0.21, + "recall@10": 0.5665, + "all@20": 0.21, + "recall@20": 0.5665 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.122, + "recall@5": 0.4723333333333333, + "all@10": 0.21, + "recall@10": 0.5665, + "all@20": 0.21, + "recall@20": 0.5665 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.132, + "recall@5": 0.4938333333333333, + "all@10": 0.21, + "recall@10": 0.5665, + "all@20": 0.246, + "recall@20": 0.6003333333333333 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.128, + "recall@5": 0.47933333333333333, + "all@10": 0.21, + "recall@10": 0.5665, + "all@20": 0.25, + "recall@20": 0.6005 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.128, + "recall@5": 0.493, + "all@10": 0.21, + "recall@10": 0.5665, + "all@20": 0.286, + "recall@20": 0.6251666666666668 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.134, + "recall@5": 0.4853333333333334, + "all@10": 0.21, + "recall@10": 0.5665, + "all@20": 0.29, + "recall@20": 0.626 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.132, + "recall@5": 0.4945, + "all@10": 0.21, + "recall@10": 0.5665, + "all@20": 0.3, + "recall@20": 0.6333333333333333 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.132, + "recall@5": 0.4895, + "all@10": 0.21, + "recall@10": 0.5665, + "all@20": 0.3, + "recall@20": 0.6333333333333333 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.128, + "recall@5": 0.48266666666666663, + "all@10": 0.198, + "recall@10": 0.5608333333333334, + "all@20": 0.3, + "recall@20": 0.6333333333333333 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.09, + "recall@5": 0.4275, + "all@10": 0.19, + "recall@10": 0.5416666666666667, + "all@20": 0.3, + "recall@20": 0.6333333333333333 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.128, + "recall@5": 0.4868333333333334, + "all@10": 0.198, + "recall@10": 0.5618333333333334, + "all@20": 0.3, + "recall@20": 0.6333333333333333 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.108, + "recall@5": 0.45083333333333336, + "all@10": 0.198, + "recall@10": 0.5496666666666667, + "all@20": 0.3, + "recall@20": 0.6333333333333333 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.126, + "recall@5": 0.4895, + "all@10": 0.2, + "recall@10": 0.5628333333333334, + "all@20": 0.3, + "recall@20": 0.6333333333333333 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.124, + "recall@5": 0.4678333333333333, + "all@10": 0.2, + "recall@10": 0.5565, + "all@20": 0.3, + "recall@20": 0.6333333333333333 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.132, + "recall@5": 0.4931666666666667, + "all@10": 0.206, + "recall@10": 0.5648333333333334, + "all@20": 0.3, + "recall@20": 0.6333333333333333 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.4873333333333334, + "all@10": 0.2, + "recall@10": 0.5585, + "all@20": 0.3, + "recall@20": 0.6333333333333333 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.49116666666666664, + "all@10": 0.13, + "recall@10": 0.4943333333333333, + "all@20": 0.13, + "recall@20": 0.4948333333333333 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.118, + "recall@5": 0.47433333333333333, + "all@10": 0.13, + "recall@10": 0.4943333333333333, + "all@20": 0.13, + "recall@20": 0.4948333333333333 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.4918333333333333, + "all@10": 0.164, + "recall@10": 0.5236666666666667, + "all@20": 0.244, + "recall@20": 0.5991666666666667 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.124, + "recall@5": 0.487, + "all@10": 0.15, + "recall@10": 0.5145, + "all@20": 0.244, + "recall@20": 0.5926666666666667 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "all@10": 0.192, + "recall@10": 0.5511666666666667, + "all@20": 0.28, + "recall@20": 0.6196666666666666 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "all@10": 0.19, + "recall@10": 0.5503333333333333, + "all@20": 0.282, + "recall@20": 0.6203333333333333 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "all@10": 0.21, + "recall@10": 0.5665, + "all@20": 0.3, + "recall@20": 0.6333333333333333 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "all@10": 0.21, + "recall@10": 0.5665, + "all@20": 0.3, + "recall@20": 0.6333333333333333 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.124, + "recall@5": 0.47966666666666663, + "all@10": 0.206, + "recall@10": 0.5628333333333334, + "all@20": 0.21, + "recall@20": 0.5665 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.094, + "recall@5": 0.4106666666666666, + "all@10": 0.196, + "recall@10": 0.5486666666666667, + "all@20": 0.21, + "recall@20": 0.5665 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.128, + "recall@5": 0.485, + "all@10": 0.208, + "recall@10": 0.5641666666666667, + "all@20": 0.236, + "recall@20": 0.588 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.118, + "recall@5": 0.4513333333333333, + "all@10": 0.202, + "recall@10": 0.5588333333333334, + "all@20": 0.232, + "recall@20": 0.5805 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.132, + "recall@5": 0.4925, + "all@10": 0.21, + "recall@10": 0.5658333333333334, + "all@20": 0.27, + "recall@20": 0.614 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.122, + "recall@5": 0.4735, + "all@10": 0.21, + "recall@10": 0.5665, + "all@20": 0.278, + "recall@20": 0.6185 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.128, + "recall@5": 0.493, + "all@10": 0.21, + "recall@10": 0.5665, + "all@20": 0.3, + "recall@20": 0.6333333333333333 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.132, + "recall@5": 0.4843333333333334, + "all@10": 0.21, + "recall@10": 0.5665, + "all@20": 0.3, + "recall@20": 0.6328333333333332 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.104, + "recall@5": 0.4526666666666666, + "all@10": 0.194, + "recall@10": 0.5556666666666666, + "all@20": 0.298, + "recall@20": 0.6316666666666667 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.032, + "recall@5": 0.286, + "all@10": 0.144, + "recall@10": 0.47866666666666663, + "all@20": 0.282, + "recall@20": 0.622 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.118, + "recall@5": 0.4731666666666666, + "all@10": 0.194, + "recall@10": 0.557, + "all@20": 0.298, + "recall@20": 0.6316666666666667 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.072, + "recall@5": 0.382, + "all@10": 0.18, + "recall@10": 0.5241666666666667, + "all@20": 0.294, + "recall@20": 0.6293333333333334 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.126, + "recall@5": 0.485, + "all@10": 0.198, + "recall@10": 0.5611666666666667, + "all@20": 0.3, + "recall@20": 0.6333333333333333 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.102, + "recall@5": 0.4405, + "all@10": 0.192, + "recall@10": 0.5458333333333334, + "all@20": 0.3, + "recall@20": 0.6326666666666666 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.128, + "recall@5": 0.4901666666666667, + "all@10": 0.202, + "recall@10": 0.5638333333333334, + "all@20": 0.3, + "recall@20": 0.6333333333333333 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.126, + "recall@5": 0.4768333333333334, + "all@10": 0.196, + "recall@10": 0.556, + "all@20": 0.3, + "recall@20": 0.6333333333333333 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.12, + "recall@5": 0.4693333333333334, + "all@10": 0.13, + "recall@10": 0.4943333333333333, + "all@20": 0.13, + "recall@20": 0.4948333333333333 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.094, + "recall@5": 0.4278333333333333, + "all@10": 0.128, + "recall@10": 0.4933333333333333, + "all@20": 0.13, + "recall@20": 0.4948333333333333 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.122, + "recall@5": 0.477, + "all@10": 0.154, + "recall@10": 0.5196666666666666, + "all@20": 0.218, + "recall@20": 0.5838333333333334 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.112, + "recall@5": 0.4611666666666666, + "all@10": 0.142, + "recall@10": 0.5081666666666667, + "all@20": 0.236, + "recall@20": 0.58 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.49116666666666664, + "all@10": 0.184, + "recall@10": 0.5428333333333333, + "all@20": 0.274, + "recall@20": 0.6175 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.122, + "recall@5": 0.48633333333333334, + "all@10": 0.182, + "recall@10": 0.5431666666666668, + "all@20": 0.272, + "recall@20": 0.6153333333333333 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "all@10": 0.21, + "recall@10": 0.5648333333333334, + "all@20": 0.3, + "recall@20": 0.6326666666666667 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "all@10": 0.21, + "recall@10": 0.5665, + "all@20": 0.3, + "recall@20": 0.6326666666666667 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.094, + "recall@5": 0.42116666666666663, + "all@10": 0.19, + "recall@10": 0.5425, + "all@20": 0.21, + "recall@20": 0.5665 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.028, + "recall@5": 0.2706666666666666, + "all@10": 0.17, + "recall@10": 0.5098333333333334, + "all@20": 0.208, + "recall@20": 0.5655 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.108, + "recall@5": 0.4545, + "all@10": 0.196, + "recall@10": 0.5533333333333332, + "all@20": 0.234, + "recall@20": 0.5861666666666666 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.076, + "recall@5": 0.381, + "all@10": 0.182, + "recall@10": 0.5386666666666666, + "all@20": 0.224, + "recall@20": 0.5761666666666667 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.126, + "recall@5": 0.4848333333333334, + "all@10": 0.202, + "recall@10": 0.5598333333333333, + "all@20": 0.262, + "recall@20": 0.61 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.114, + "recall@5": 0.45133333333333336, + "all@10": 0.202, + "recall@10": 0.5585, + "all@20": 0.262, + "recall@20": 0.6108333333333332 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.494, + "all@10": 0.21, + "recall@10": 0.5658333333333334, + "all@20": 0.298, + "recall@20": 0.6318333333333332 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.126, + "recall@5": 0.48083333333333333, + "all@10": 0.208, + "recall@10": 0.5651666666666667, + "all@20": 0.3, + "recall@20": 0.6321666666666668 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.064, + "recall@5": 0.32783333333333337, + "all@10": 0.16, + "recall@10": 0.5138333333333333, + "all@20": 0.266, + "recall@20": 0.6118333333333332 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.008, + "recall@5": 0.10899999999999999, + "all@10": 0.054, + "recall@10": 0.3203333333333333, + "all@20": 0.244, + "recall@20": 0.5833333333333333 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.084, + "recall@5": 0.4083333333333333, + "all@10": 0.18, + "recall@10": 0.5403333333333332, + "all@20": 0.282, + "recall@20": 0.6205 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.018, + "recall@5": 0.22066666666666668, + "all@10": 0.136, + "recall@10": 0.4555, + "all@20": 0.268, + "recall@20": 0.6126666666666666 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.108, + "recall@5": 0.4666666666666666, + "all@10": 0.192, + "recall@10": 0.5541666666666666, + "all@20": 0.292, + "recall@20": 0.6278333333333332 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.08, + "recall@5": 0.39899999999999997, + "all@10": 0.182, + "recall@10": 0.5328333333333334, + "all@20": 0.29, + "recall@20": 0.6278333333333332 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.128, + "recall@5": 0.4898333333333334, + "all@10": 0.198, + "recall@10": 0.5611666666666667, + "all@20": 0.3, + "recall@20": 0.6333333333333333 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.118, + "recall@5": 0.4656666666666667, + "all@10": 0.2, + "recall@10": 0.5553333333333332, + "all@20": 0.3, + "recall@20": 0.6326666666666666 + } + ], + "ppr_selected": { + "5": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "20": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + } + }, + "lib_dev_grid": [ + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 16.696043799514882, + "default": true, + "all@5": 0.128, + "recall@5": 0.4933333333333333, + "n_ret@5": 5.0, + "all@10": 0.13, + "recall@10": 0.4943333333333333, + "n_ret@10": 8.936, + "all@20": 0.13, + "recall@20": 0.4948333333333333, + "n_ret@20": 13.586 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 16.960055398521945, + "default": false, + "all@5": 0.128, + "recall@5": 0.4933333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.21, + "recall@20": 0.5665, + "n_ret@20": 19.054 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 77.00467179866973, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.248, + "recall@20": 0.603, + "n_ret@20": 19.948 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 80.37308420008048, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.298, + "recall@20": 0.6323333333333333, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 82.5734239993617, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.13, + "recall@10": 0.4948333333333333, + "n_ret@10": 8.936, + "all@20": 0.13, + "recall@20": 0.4948333333333333, + "n_ret@20": 13.586 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 86.25658740045037, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.3, + "recall@20": 0.6333333333333333, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 17.201544200186618, + "default": false, + "all@5": 0.118, + "recall@5": 0.4803333333333334, + "n_ret@5": 5.0, + "all@10": 0.122, + "recall@10": 0.4851666666666667, + "n_ret@10": 9.768, + "all@20": 0.126, + "recall@20": 0.4903333333333334, + "n_ret@20": 18.734 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 16.721821197657846, + "default": false, + "all@5": 0.122, + "recall@5": 0.48416666666666663, + "n_ret@5": 5.0, + "all@10": 0.184, + "recall@10": 0.55, + "n_ret@10": 10.0, + "all@20": 0.24, + "recall@20": 0.5978333333333332, + "n_ret@20": 19.904 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 15.650050599011593, + "default": false, + "all@5": 0.128, + "recall@5": 0.4933333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.21, + "recall@20": 0.5665, + "n_ret@20": 18.154 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 81.61394079809543, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.3, + "recall@20": 0.6333333333333333, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 17.857304401230067, + "default": false, + "all@5": 0.128, + "recall@5": 0.4933333333333333, + "n_ret@5": 5.0, + "all@10": 0.13, + "recall@10": 0.4943333333333333, + "n_ret@10": 9.768, + "all@20": 0.13, + "recall@20": 0.4948333333333333, + "n_ret@20": 18.734 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 17.24265179957729, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.248, + "recall@20": 0.6015, + "n_ret@20": 19.948 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 73.42480360192712, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.13, + "recall@10": 0.4943333333333333, + "n_ret@10": 9.224, + "all@20": 0.13, + "recall@20": 0.4943333333333333, + "n_ret@20": 14.006 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 76.66849920083769, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.21, + "recall@20": 0.5685, + "n_ret@20": 19.19 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 74.30276040080935, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.248, + "recall@20": 0.6025, + "n_ret@20": 19.904 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 15.673322399379686, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.13, + "recall@10": 0.4943333333333333, + "n_ret@10": 9.224, + "all@20": 0.13, + "recall@20": 0.4943333333333333, + "n_ret@20": 14.006 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 82.70799239911139, + "default": false, + "all@5": 0.126, + "recall@5": 0.49266666666666664, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5621666666666667, + "n_ret@10": 10.0, + "all@20": 0.212, + "recall@20": 0.568, + "n_ret@20": 19.946 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 16.562658801325597, + "default": false, + "all@5": 0.12, + "recall@5": 0.4813333333333334, + "n_ret@5": 5.0, + "all@10": 0.18, + "recall@10": 0.5435, + "n_ret@10": 10.0, + "all@20": 0.2, + "recall@20": 0.5573333333333332, + "n_ret@20": 19.368 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 79.21772139973473, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.21, + "recall@20": 0.5665, + "n_ret@20": 19.054 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 81.49611020099837, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.21, + "recall@20": 0.5665, + "n_ret@20": 19.19 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 28.31719859968871, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.21, + "recall@20": 0.5665, + "n_ret@20": 19.978 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 81.13066059770063, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.248, + "recall@20": 0.6025, + "n_ret@20": 19.98 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 81.40837079880293, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.212, + "recall@20": 0.568, + "n_ret@20": 19.946 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 78.60095359978732, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.3, + "recall@20": 0.6333333333333333, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 74.63762059877627, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.132, + "recall@10": 0.4958333333333333, + "n_ret@10": 9.768, + "all@20": 0.132, + "recall@20": 0.4958333333333333, + "n_ret@20": 18.734 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 89.0320389980916, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.248, + "recall@20": 0.6025, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 91.63665740040597, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.3, + "recall@20": 0.6333333333333333, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 81.2276695992332, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.21, + "recall@20": 0.5671666666666667, + "n_ret@20": 19.978 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 15.731986199272797, + "default": false, + "all@5": 0.122, + "recall@5": 0.48416666666666663, + "n_ret@5": 5.0, + "all@10": 0.184, + "recall@10": 0.55, + "n_ret@10": 10.0, + "all@20": 0.272, + "recall@20": 0.6203333333333333, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 69.58201360271778, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.248, + "recall@20": 0.6035, + "n_ret@20": 19.948 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 13.48943560058251, + "default": false, + "all@5": 0.128, + "recall@5": 0.4933333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.248, + "recall@20": 0.6015, + "n_ret@20": 19.904 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 74.38857719872613, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.3, + "recall@20": 0.6333333333333333, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 24.775220401817933, + "default": false, + "all@5": 0.128, + "recall@5": 0.4933333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.21, + "recall@20": 0.5665, + "n_ret@20": 19.978 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 36.149081999319606, + "default": false, + "all@5": 0.118, + "recall@5": 0.4803333333333334, + "n_ret@5": 5.0, + "all@10": 0.176, + "recall@10": 0.5415, + "n_ret@10": 10.0, + "all@20": 0.256, + "recall@20": 0.6068333333333333, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 68.57957200030796, + "default": false, + "all@5": 0.126, + "recall@5": 0.49266666666666664, + "n_ret@5": 5.0, + "all@10": 0.208, + "recall@10": 0.5631666666666667, + "n_ret@10": 10.0, + "all@20": 0.248, + "recall@20": 0.602, + "n_ret@20": 19.98 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 78.84680179948919, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.21, + "recall@20": 0.5665, + "n_ret@20": 19.978 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 79.15926519944333, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.22, + "recall@20": 0.5726666666666668, + "n_ret@20": 19.978 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 69.14882020023651, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.3, + "recall@20": 0.6333333333333333, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 13.416089400299825, + "default": false, + "all@5": 0.122, + "recall@5": 0.48416666666666663, + "n_ret@5": 5.0, + "all@10": 0.184, + "recall@10": 0.55, + "n_ret@10": 10.0, + "all@20": 0.272, + "recall@20": 0.6203333333333333, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 15.207757201162167, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.13, + "recall@10": 0.4943333333333333, + "n_ret@10": 9.768, + "all@20": 0.13, + "recall@20": 0.4948333333333333, + "n_ret@20": 18.734 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 13.831199400592595, + "default": false, + "all@5": 0.12, + "recall@5": 0.4813333333333334, + "n_ret@5": 5.0, + "all@10": 0.18, + "recall@10": 0.5435, + "n_ret@10": 10.0, + "all@20": 0.262, + "recall@20": 0.6101666666666666, + "n_ret@20": 20.0 + } + ], + "lib_selected": { + "5": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + } + }, + "n_lib_configs": 41, + "n_ppr_configs": 72, + "sub_ks": "10", + "seconds": 2425.5253031253815 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/musique__bge__ctrl_mention.npz b/benchmarks/multihop/results/raw/musique__bge__ctrl_mention.npz new file mode 100644 index 0000000..1d96787 Binary files /dev/null and b/benchmarks/multihop/results/raw/musique__bge__ctrl_mention.npz differ diff --git a/benchmarks/multihop/results/raw/musique__bge__mention.json b/benchmarks/multihop/results/raw/musique__bge__mention.json new file mode 100644 index 0000000..19f68d7 --- /dev/null +++ b/benchmarks/multihop/results/raw/musique__bge__mention.json @@ -0,0 +1,2017 @@ +{ + "dataset": "musique", + "model": "bge", + "variant": "mention", + "families": [ + "mention" + ], + "diagnostics": { + "n_edges_undirected_union": 20272, + "n_edges_by_family_directed_or_undirected": 22465, + "mean_degree": 2.6318727685816294, + "median_degree": 0.0, + "max_degree": 54, + "isolated_frac": 0.5272314183706589, + "gold_pairs_directly_linked": 0.14690791250337565, + "gold_pairs_linked_within_2": 0.2292735619767756, + "neighbours_of_gold_that_are_in_same_question_context": 0.1532126205792047 + }, + "ppr_dev_grid": [ + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.132, + "recall@5": 0.4936666666666667, + "all@10": 0.254, + "recall@10": 0.5763333333333333, + "all@20": 0.284, + "recall@20": 0.5991666666666666 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.132, + "recall@5": 0.4953333333333333, + "all@10": 0.266, + "recall@10": 0.5866666666666668, + "all@20": 0.33, + "recall@20": 0.6323333333333333 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.4926666666666667, + "all@10": 0.278, + "recall@10": 0.5973333333333333, + "all@20": 0.366, + "recall@20": 0.6735 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "all@10": 0.288, + "recall@10": 0.6005, + "all@20": 0.388, + "recall@20": 0.683 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.4936666666666667, + "all@10": 0.282, + "recall@10": 0.608, + "all@20": 0.378, + "recall@20": 0.6823333333333333 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "all@10": 0.29, + "recall@10": 0.6156666666666667, + "all@20": 0.39, + "recall@20": 0.6861666666666667 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "all@10": 0.244, + "recall@10": 0.5895, + "all@20": 0.342, + "recall@20": 0.6601666666666668 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "all@10": 0.252, + "recall@10": 0.595, + "all@20": 0.34, + "recall@20": 0.6591666666666668 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.4848333333333334, + "all@10": 0.204, + "recall@10": 0.5643333333333332, + "all@20": 0.324, + "recall@20": 0.648 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.114, + "recall@5": 0.4676666666666666, + "all@10": 0.212, + "recall@10": 0.5691666666666667, + "all@20": 0.352, + "recall@20": 0.6628333333333333 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.4861666666666667, + "all@10": 0.206, + "recall@10": 0.5676666666666667, + "all@20": 0.36, + "recall@20": 0.6705 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.122, + "recall@5": 0.479, + "all@10": 0.212, + "recall@10": 0.5691666666666667, + "all@20": 0.386, + "recall@20": 0.6821666666666666 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.132, + "recall@5": 0.4911666666666667, + "all@10": 0.202, + "recall@10": 0.5656666666666667, + "all@20": 0.372, + "recall@20": 0.6786666666666666 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.48783333333333334, + "all@10": 0.21, + "recall@10": 0.5681666666666667, + "all@20": 0.386, + "recall@20": 0.6868333333333333 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.4961666666666667, + "all@10": 0.21, + "recall@10": 0.5665, + "all@20": 0.338, + "recall@20": 0.6596666666666666 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.14, + "recall@5": 0.49783333333333335, + "all@10": 0.21, + "recall@10": 0.567, + "all@20": 0.336, + "recall@20": 0.6573333333333332 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.122, + "recall@5": 0.4508333333333333, + "all@10": 0.212, + "recall@10": 0.5613333333333332, + "all@20": 0.304, + "recall@20": 0.6376666666666666 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.098, + "recall@5": 0.4033333333333333, + "all@10": 0.202, + "recall@10": 0.5493333333333333, + "all@20": 0.298, + "recall@20": 0.636 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.122, + "recall@5": 0.4615, + "all@10": 0.214, + "recall@10": 0.5655, + "all@20": 0.304, + "recall@20": 0.6376666666666666 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.112, + "recall@5": 0.43916666666666665, + "all@10": 0.21, + "recall@10": 0.5595, + "all@20": 0.298, + "recall@20": 0.636 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.132, + "recall@5": 0.4818333333333334, + "all@10": 0.214, + "recall@10": 0.5668333333333333, + "all@20": 0.302, + "recall@20": 0.6373333333333333 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.124, + "recall@5": 0.468, + "all@10": 0.224, + "recall@10": 0.5686666666666668, + "all@20": 0.302, + "recall@20": 0.6373333333333333 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.142, + "recall@5": 0.49733333333333335, + "all@10": 0.218, + "recall@10": 0.5721666666666666, + "all@20": 0.3, + "recall@20": 0.635 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.142, + "recall@5": 0.4955, + "all@10": 0.218, + "recall@10": 0.5711666666666666, + "all@20": 0.298, + "recall@20": 0.6348333333333332 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.15, + "recall@5": 0.4968333333333333, + "all@10": 0.25, + "recall@10": 0.5746666666666668, + "all@20": 0.28, + "recall@20": 0.5986666666666667 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.19, + "recall@5": 0.5226666666666667, + "all@10": 0.27, + "recall@10": 0.589, + "all@20": 0.33, + "recall@20": 0.632 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.148, + "recall@5": 0.4966666666666667, + "all@10": 0.274, + "recall@10": 0.5936666666666668, + "all@20": 0.36, + "recall@20": 0.6621666666666667 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.192, + "recall@5": 0.5251666666666667, + "all@10": 0.288, + "recall@10": 0.5996666666666668, + "all@20": 0.384, + "recall@20": 0.6768333333333333 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.144, + "recall@5": 0.49633333333333335, + "all@10": 0.28, + "recall@10": 0.6028333333333333, + "all@20": 0.374, + "recall@20": 0.6823333333333332 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.176, + "recall@5": 0.5206666666666667, + "all@10": 0.296, + "recall@10": 0.6146666666666667, + "all@20": 0.4, + "recall@20": 0.6933333333333334 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.4953333333333333, + "all@10": 0.266, + "recall@10": 0.6003333333333333, + "all@20": 0.366, + "recall@20": 0.674 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.134, + "recall@5": 0.4961666666666667, + "all@10": 0.266, + "recall@10": 0.6043333333333334, + "all@20": 0.358, + "recall@20": 0.6723333333333332 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.138, + "recall@5": 0.4675, + "all@10": 0.218, + "recall@10": 0.5698333333333334, + "all@20": 0.32, + "recall@20": 0.6473333333333332 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.146, + "recall@5": 0.4465, + "all@10": 0.26, + "recall@10": 0.5915, + "all@20": 0.358, + "recall@20": 0.667 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.138, + "recall@5": 0.4766666666666667, + "all@10": 0.218, + "recall@10": 0.5706666666666668, + "all@20": 0.344, + "recall@20": 0.6616666666666667 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.152, + "recall@5": 0.478, + "all@10": 0.262, + "recall@10": 0.5956666666666668, + "all@20": 0.38, + "recall@20": 0.6785 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.48616666666666675, + "all@10": 0.216, + "recall@10": 0.5715, + "all@20": 0.37, + "recall@20": 0.6781666666666667 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.154, + "recall@5": 0.49483333333333335, + "all@10": 0.254, + "recall@10": 0.5926666666666667, + "all@20": 0.38, + "recall@20": 0.6825 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.134, + "recall@5": 0.4936666666666667, + "all@10": 0.212, + "recall@10": 0.57, + "all@20": 0.364, + "recall@20": 0.672 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.132, + "recall@5": 0.494, + "all@10": 0.224, + "recall@10": 0.5761666666666666, + "all@20": 0.35, + "recall@20": 0.668 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.108, + "recall@5": 0.3993333333333334, + "all@10": 0.206, + "recall@10": 0.5448333333333334, + "all@20": 0.304, + "recall@20": 0.6403333333333333 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.108, + "recall@5": 0.3436666666666666, + "all@10": 0.214, + "recall@10": 0.5255, + "all@20": 0.354, + "recall@20": 0.6695 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.116, + "recall@5": 0.43116666666666664, + "all@10": 0.224, + "recall@10": 0.5645, + "all@20": 0.31, + "recall@20": 0.6425 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.124, + "recall@5": 0.3971666666666666, + "all@10": 0.242, + "recall@10": 0.5601666666666666, + "all@20": 0.352, + "recall@20": 0.669 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.134, + "recall@5": 0.4703333333333334, + "all@10": 0.22, + "recall@10": 0.5693333333333334, + "all@20": 0.314, + "recall@20": 0.6438333333333333 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.128, + "recall@5": 0.44583333333333336, + "all@10": 0.244, + "recall@10": 0.5755, + "all@20": 0.342, + "recall@20": 0.6615 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.14, + "recall@5": 0.49216666666666675, + "all@10": 0.222, + "recall@10": 0.5741666666666667, + "all@20": 0.304, + "recall@20": 0.6388333333333333 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.138, + "recall@5": 0.48499999999999993, + "all@10": 0.228, + "recall@10": 0.577, + "all@20": 0.318, + "recall@20": 0.6483333333333332 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.184, + "recall@5": 0.493, + "all@10": 0.248, + "recall@10": 0.5706666666666667, + "all@20": 0.28, + "recall@20": 0.5986666666666667 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.176, + "recall@5": 0.4996666666666667, + "all@10": 0.274, + "recall@10": 0.5898333333333333, + "all@20": 0.33, + "recall@20": 0.6316666666666667 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.164, + "recall@5": 0.4883333333333334, + "all@10": 0.268, + "recall@10": 0.5871666666666666, + "all@20": 0.352, + "recall@20": 0.6566666666666666 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.194, + "recall@5": 0.5153333333333334, + "all@10": 0.288, + "recall@10": 0.5985, + "all@20": 0.368, + "recall@20": 0.6656666666666667 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.15, + "recall@5": 0.4971666666666667, + "all@10": 0.272, + "recall@10": 0.5971666666666667, + "all@20": 0.364, + "recall@20": 0.6746666666666666 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.204, + "recall@5": 0.534, + "all@10": 0.296, + "recall@10": 0.6086666666666667, + "all@20": 0.388, + "recall@20": 0.6843333333333333 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.146, + "recall@5": 0.499, + "all@10": 0.274, + "recall@10": 0.6033333333333333, + "all@20": 0.358, + "recall@20": 0.6706666666666666 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.166, + "recall@5": 0.5126666666666667, + "all@10": 0.276, + "recall@10": 0.6073333333333333, + "all@20": 0.362, + "recall@20": 0.6758333333333334 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.14, + "recall@5": 0.4225, + "all@10": 0.256, + "recall@10": 0.5796666666666667, + "all@20": 0.32, + "recall@20": 0.6455 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.128, + "recall@5": 0.3705, + "all@10": 0.256, + "recall@10": 0.5705, + "all@20": 0.354, + "recall@20": 0.6643333333333332 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.44833333333333336, + "all@10": 0.238, + "recall@10": 0.5736666666666667, + "all@20": 0.342, + "recall@20": 0.6613333333333332 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.144, + "recall@5": 0.42416666666666664, + "all@10": 0.27, + "recall@10": 0.5901666666666667, + "all@20": 0.37, + "recall@20": 0.6753333333333332 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.4708333333333334, + "all@10": 0.222, + "recall@10": 0.5735, + "all@20": 0.356, + "recall@20": 0.6676666666666666 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.168, + "recall@5": 0.488, + "all@10": 0.272, + "recall@10": 0.6003333333333333, + "all@20": 0.384, + "recall@20": 0.6845 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.14, + "recall@5": 0.4965, + "all@10": 0.216, + "recall@10": 0.5728333333333333, + "all@20": 0.352, + "recall@20": 0.6673333333333333 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.154, + "recall@5": 0.5028333333333334, + "all@10": 0.236, + "recall@10": 0.583, + "all@20": 0.354, + "recall@20": 0.6716666666666666 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.088, + "recall@5": 0.30033333333333334, + "all@10": 0.222, + "recall@10": 0.5136666666666667, + "all@20": 0.344, + "recall@20": 0.6513333333333333 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.094, + "recall@5": 0.267, + "all@10": 0.176, + "recall@10": 0.4351666666666667, + "all@20": 0.344, + "recall@20": 0.6411666666666668 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.094, + "recall@5": 0.35916666666666663, + "all@10": 0.218, + "recall@10": 0.5423333333333333, + "all@20": 0.332, + "recall@20": 0.6515 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.106, + "recall@5": 0.3258333333333333, + "all@10": 0.212, + "recall@10": 0.5138333333333333, + "all@20": 0.356, + "recall@20": 0.6573333333333333 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.12, + "recall@5": 0.44233333333333336, + "all@10": 0.216, + "recall@10": 0.5621666666666667, + "all@20": 0.316, + "recall@20": 0.6483333333333332 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.132, + "recall@5": 0.42516666666666664, + "all@10": 0.256, + "recall@10": 0.5721666666666666, + "all@20": 0.362, + "recall@20": 0.6728333333333333 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.134, + "recall@5": 0.4888333333333334, + "all@10": 0.222, + "recall@10": 0.5741666666666666, + "all@20": 0.31, + "recall@20": 0.6406666666666667 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.148, + "recall@5": 0.488, + "all@10": 0.246, + "recall@10": 0.5858333333333332, + "all@20": 0.334, + "recall@20": 0.6578333333333334 + } + ], + "ppr_selected": { + "5": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "10": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "20": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + } + }, + "lib_dev_grid": [ + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 17.40815560007468, + "default": true, + "all@5": 0.166, + "recall@5": 0.5146666666666667, + "n_ret@5": 5.0, + "all@10": 0.272, + "recall@10": 0.586, + "n_ret@10": 9.058, + "all@20": 0.292, + "recall@20": 0.6011666666666666, + "n_ret@20": 14.284 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 17.54534860001877, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.332, + "recall@20": 0.648, + "n_ret@20": 19.018 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 80.29818600101862, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.36, + "recall@20": 0.6725, + "n_ret@20": 19.952 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 82.57285419781692, + "default": false, + "all@5": 0.154, + "recall@5": 0.5085, + "n_ret@5": 5.0, + "all@10": 0.242, + "recall@10": 0.585, + "n_ret@10": 10.0, + "all@20": 0.342, + "recall@20": 0.6576666666666666, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 83.66237799951341, + "default": false, + "all@5": 0.152, + "recall@5": 0.5075, + "n_ret@5": 5.0, + "all@10": 0.28, + "recall@10": 0.5906666666666667, + "n_ret@10": 9.058, + "all@20": 0.296, + "recall@20": 0.6063333333333333, + "n_ret@20": 14.284 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 83.11357479798608, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.3, + "recall@20": 0.6333333333333333, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 18.237747801002115, + "default": false, + "all@5": 0.21, + "recall@5": 0.5413333333333332, + "n_ret@5": 5.0, + "all@10": 0.294, + "recall@10": 0.599, + "n_ret@10": 9.684, + "all@20": 0.352, + "recall@20": 0.6365, + "n_ret@20": 18.506 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 17.315247199498117, + "default": false, + "all@5": 0.212, + "recall@5": 0.5415, + "n_ret@5": 5.0, + "all@10": 0.304, + "recall@10": 0.6191666666666666, + "n_ret@10": 10.0, + "all@20": 0.356, + "recall@20": 0.6691666666666666, + "n_ret@20": 19.89 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 16.20437399914954, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.334, + "recall@20": 0.6481666666666667, + "n_ret@20": 18.116 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 80.6959956001956, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.3, + "recall@20": 0.6333333333333333, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 17.739281199406832, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.29, + "recall@10": 0.5986666666666668, + "n_ret@10": 9.684, + "all@20": 0.348, + "recall@20": 0.6346666666666666, + "n_ret@20": 18.506 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 18.300373398931697, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.362, + "recall@20": 0.674, + "n_ret@20": 19.952 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 72.78561819973402, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.6043333333333334, + "n_ret@10": 9.268, + "all@20": 0.324, + "recall@20": 0.6235, + "n_ret@20": 14.79 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 76.74991360027343, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.346, + "recall@20": 0.6576666666666666, + "n_ret@20": 19.13 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 75.64348980062641, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.6811666666666666, + "n_ret@20": 19.89 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 15.177146799280308, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.288, + "recall@10": 0.5965, + "n_ret@10": 9.268, + "all@20": 0.312, + "recall@20": 0.6158333333333332, + "n_ret@20": 14.79 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 82.97796300100163, + "default": false, + "all@5": 0.202, + "recall@5": 0.5373333333333333, + "n_ret@5": 5.0, + "all@10": 0.298, + "recall@10": 0.6148333333333333, + "n_ret@10": 10.0, + "all@20": 0.402, + "recall@20": 0.6883333333333332, + "n_ret@20": 19.942 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 16.069689200259745, + "default": false, + "all@5": 0.21, + "recall@5": 0.5401666666666667, + "n_ret@5": 5.0, + "all@10": 0.306, + "recall@10": 0.6173333333333333, + "n_ret@10": 10.0, + "all@20": 0.354, + "recall@20": 0.658, + "n_ret@20": 19.124 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 76.18558619986288, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.344, + "recall@20": 0.6565, + "n_ret@20": 19.018 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 80.09164339897688, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.344, + "recall@20": 0.6565, + "n_ret@20": 19.13 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 33.9119835983729, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.362, + "recall@20": 0.6631666666666667, + "n_ret@20": 19.956 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 80.57351780065801, + "default": false, + "all@5": 0.154, + "recall@5": 0.5085, + "n_ret@5": 5.0, + "all@10": 0.242, + "recall@10": 0.585, + "n_ret@10": 10.0, + "all@20": 0.384, + "recall@20": 0.6876666666666668, + "n_ret@20": 19.96 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 84.23143780010287, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.378, + "recall@20": 0.6791666666666666, + "n_ret@20": 19.942 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 76.4540695984615, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.3, + "recall@20": 0.6333333333333333, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 74.50407280295622, + "default": false, + "all@5": 0.154, + "recall@5": 0.5085, + "n_ret@5": 5.0, + "all@10": 0.294, + "recall@10": 0.607, + "n_ret@10": 9.684, + "all@20": 0.378, + "recall@20": 0.6673333333333333, + "n_ret@20": 18.506 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 92.46652659820393, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.384, + "recall@20": 0.6876666666666668, + "n_ret@20": 19.994 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 100.22463000111748, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.3, + "recall@20": 0.6333333333333333, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 86.58316760044545, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.376, + "recall@20": 0.6758333333333333, + "n_ret@20": 19.956 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 16.043498400831595, + "default": false, + "all@5": 0.212, + "recall@5": 0.542, + "n_ret@5": 5.0, + "all@10": 0.306, + "recall@10": 0.6198333333333332, + "n_ret@10": 10.0, + "all@20": 0.38, + "recall@20": 0.6795, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 71.46779899823014, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.372, + "recall@20": 0.6811666666666666, + "n_ret@20": 19.952 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 14.1149324021535, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.362, + "recall@20": 0.674, + "n_ret@20": 19.89 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 76.88650420086924, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.3, + "recall@20": 0.6333333333333333, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 28.870331398560666, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.362, + "recall@20": 0.6631666666666667, + "n_ret@20": 19.956 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 42.14411380048841, + "default": false, + "all@5": 0.21, + "recall@5": 0.5413333333333332, + "n_ret@5": 5.0, + "all@10": 0.308, + "recall@10": 0.6198333333333332, + "n_ret@10": 10.0, + "all@20": 0.398, + "recall@20": 0.6898333333333333, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 71.27257139910944, + "default": false, + "all@5": 0.202, + "recall@5": 0.5373333333333333, + "n_ret@5": 5.0, + "all@10": 0.294, + "recall@10": 0.6125, + "n_ret@10": 10.0, + "all@20": 0.382, + "recall@20": 0.6855, + "n_ret@20": 19.96 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 81.22463360207621, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.376, + "recall@20": 0.6748333333333333, + "n_ret@20": 19.956 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 81.19792039995082, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.378, + "recall@20": 0.6791666666666666, + "n_ret@20": 19.956 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 71.63435560080688, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5665, + "n_ret@10": 10.0, + "all@20": 0.3, + "recall@20": 0.6333333333333333, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 13.81371419946663, + "default": false, + "all@5": 0.212, + "recall@5": 0.5415, + "n_ret@5": 5.0, + "all@10": 0.304, + "recall@10": 0.6191666666666666, + "n_ret@10": 10.0, + "all@20": 0.378, + "recall@20": 0.6785, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 15.682332400232552, + "default": false, + "all@5": 0.13, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.29, + "recall@10": 0.5986666666666668, + "n_ret@10": 9.684, + "all@20": 0.348, + "recall@20": 0.6346666666666666, + "n_ret@20": 18.506 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 13.807287599192932, + "default": false, + "all@5": 0.21, + "recall@5": 0.5401666666666667, + "n_ret@5": 5.0, + "all@10": 0.306, + "recall@10": 0.6173333333333333, + "n_ret@10": 10.0, + "all@20": 0.388, + "recall@20": 0.683, + "n_ret@20": 20.0 + } + ], + "lib_selected": { + "5": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "20": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + } + }, + "n_lib_configs": 41, + "n_ppr_configs": 72, + "sub_ks": "10", + "seconds": 2487.9375369548798 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/musique__bge__mention.npz b/benchmarks/multihop/results/raw/musique__bge__mention.npz new file mode 100644 index 0000000..3c065cf Binary files /dev/null and b/benchmarks/multihop/results/raw/musique__bge__mention.npz differ diff --git a/benchmarks/multihop/results/raw/musique__bge__rerank_scores.npy b/benchmarks/multihop/results/raw/musique__bge__rerank_scores.npy new file mode 100644 index 0000000..bea8a95 Binary files /dev/null and b/benchmarks/multihop/results/raw/musique__bge__rerank_scores.npy differ diff --git a/benchmarks/multihop/results/raw/musique__bge__vector_top50.npy b/benchmarks/multihop/results/raw/musique__bge__vector_top50.npy new file mode 100644 index 0000000..0f8ba14 Binary files /dev/null and b/benchmarks/multihop/results/raw/musique__bge__vector_top50.npy differ diff --git a/benchmarks/multihop/results/raw/musique__minilm__baselines.json b/benchmarks/multihop/results/raw/musique__minilm__baselines.json new file mode 100644 index 0000000..17b0888 --- /dev/null +++ b/benchmarks/multihop/results/raw/musique__minilm__baselines.json @@ -0,0 +1,8 @@ +{ + "dataset": "musique", + "model": "minilm", + "corpus_size": 15405, + "n_questions": 1500, + "lib_similarity_search_same_top20_set": "200/200", + "seconds": 67.22751545906067 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/musique__minilm__baselines.npz b/benchmarks/multihop/results/raw/musique__minilm__baselines.npz new file mode 100644 index 0000000..7264e20 Binary files /dev/null and b/benchmarks/multihop/results/raw/musique__minilm__baselines.npz differ diff --git a/benchmarks/multihop/results/raw/musique__minilm__bm25_top50.npy b/benchmarks/multihop/results/raw/musique__minilm__bm25_top50.npy new file mode 100644 index 0000000..3ff4907 Binary files /dev/null and b/benchmarks/multihop/results/raw/musique__minilm__bm25_top50.npy differ diff --git a/benchmarks/multihop/results/raw/musique__minilm__ctrl_mention.json b/benchmarks/multihop/results/raw/musique__minilm__ctrl_mention.json new file mode 100644 index 0000000..c73f9b0 --- /dev/null +++ b/benchmarks/multihop/results/raw/musique__minilm__ctrl_mention.json @@ -0,0 +1,2017 @@ +{ + "dataset": "musique", + "model": "minilm", + "variant": "ctrl_mention", + "families": [ + "mention_rand" + ], + "diagnostics": { + "n_edges_undirected_union": 20272, + "n_edges_by_family_directed_or_undirected": 22465, + "mean_degree": 2.6318727685816294, + "median_degree": 0.0, + "max_degree": 54, + "isolated_frac": 0.5272314183706589, + "gold_pairs_directly_linked": 0.0, + "gold_pairs_linked_within_2": 0.0, + "neighbours_of_gold_that_are_in_same_question_context": 0.001101741312327797 + }, + "ppr_dev_grid": [ + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "all@10": 0.136, + "recall@10": 0.4963333333333333, + "all@20": 0.136, + "recall@20": 0.4963333333333333 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "all@10": 0.136, + "recall@10": 0.4963333333333333, + "all@20": 0.14, + "recall@20": 0.499 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "all@10": 0.192, + "recall@10": 0.5506666666666666, + "all@20": 0.272, + "recall@20": 0.6278333333333332 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "all@10": 0.192, + "recall@10": 0.5495, + "all@20": 0.27, + "recall@20": 0.626 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.14, + "recall@5": 0.49566666666666664, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "all@20": 0.206, + "recall@20": 0.5648333333333333 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.124, + "recall@5": 0.4835, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "all@20": 0.206, + "recall@20": 0.5648333333333333 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.142, + "recall@5": 0.497, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "all@20": 0.264, + "recall@20": 0.62 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.48799999999999993, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "all@20": 0.262, + "recall@20": 0.6203333333333334 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.138, + "recall@5": 0.496, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.4871666666666667, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.138, + "recall@5": 0.4963333333333333, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.4938333333333333, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.14, + "recall@5": 0.49516666666666664, + "all@10": 0.21, + "recall@10": 0.5638333333333333, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.118, + "recall@5": 0.4678333333333333, + "all@10": 0.186, + "recall@10": 0.5501666666666666, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.142, + "recall@5": 0.49583333333333324, + "all@10": 0.21, + "recall@10": 0.5645, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.126, + "recall@5": 0.47816666666666663, + "all@10": 0.198, + "recall@10": 0.5553333333333333, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.138, + "recall@5": 0.4948333333333333, + "all@10": 0.21, + "recall@10": 0.566, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.126, + "recall@5": 0.485, + "all@10": 0.206, + "recall@10": 0.5646666666666668, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.138, + "recall@5": 0.4963333333333333, + "all@10": 0.206, + "recall@10": 0.5641666666666666, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.49316666666666664, + "all@10": 0.208, + "recall@10": 0.5653333333333332, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.49366666666666664, + "all@10": 0.136, + "recall@10": 0.4963333333333333, + "all@20": 0.136, + "recall@20": 0.4963333333333333 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.12, + "recall@5": 0.4765, + "all@10": 0.136, + "recall@10": 0.4963333333333333, + "all@20": 0.138, + "recall@20": 0.498 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "all@10": 0.178, + "recall@10": 0.5355, + "all@20": 0.26, + "recall@20": 0.6168333333333333 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "all@10": 0.178, + "recall@10": 0.5358333333333334, + "all@20": 0.26, + "recall@20": 0.6198333333333333 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "all@10": 0.204, + "recall@10": 0.5631666666666666, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "all@10": 0.2, + "recall@10": 0.5605, + "all@20": 0.29, + "recall@20": 0.6363333333333334 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.4903333333333333, + "all@10": 0.198, + "recall@10": 0.5561666666666666, + "all@20": 0.206, + "recall@20": 0.5648333333333333 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.096, + "recall@5": 0.4375, + "all@10": 0.186, + "recall@10": 0.5481666666666667, + "all@20": 0.206, + "recall@20": 0.5648333333333333 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.14, + "recall@5": 0.496, + "all@10": 0.206, + "recall@10": 0.5641666666666666, + "all@20": 0.244, + "recall@20": 0.6068333333333333 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.12, + "recall@5": 0.47483333333333333, + "all@10": 0.198, + "recall@10": 0.5601666666666666, + "all@20": 0.238, + "recall@20": 0.6076666666666667 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.14, + "recall@5": 0.49766666666666665, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.126, + "recall@5": 0.48566666666666664, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "all@20": 0.29, + "recall@20": 0.6363333333333334 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.14, + "recall@5": 0.4963333333333333, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.134, + "recall@5": 0.49133333333333334, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.126, + "recall@5": 0.4805, + "all@10": 0.2, + "recall@10": 0.5538333333333333, + "all@20": 0.276, + "recall@20": 0.629 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.074, + "recall@5": 0.384, + "all@10": 0.168, + "recall@10": 0.522, + "all@20": 0.258, + "recall@20": 0.6126666666666667 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.142, + "recall@5": 0.495, + "all@10": 0.21, + "recall@10": 0.5618333333333333, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.118, + "recall@5": 0.4525, + "all@10": 0.19, + "recall@10": 0.5515, + "all@20": 0.288, + "recall@20": 0.635 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.138, + "recall@5": 0.49516666666666664, + "all@10": 0.21, + "recall@10": 0.5638333333333333, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.124, + "recall@5": 0.47433333333333333, + "all@10": 0.19, + "recall@10": 0.5523333333333332, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.14, + "recall@5": 0.49516666666666664, + "all@10": 0.206, + "recall@10": 0.5643333333333332, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.134, + "recall@5": 0.49183333333333334, + "all@10": 0.206, + "recall@10": 0.5621666666666667, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.132, + "recall@5": 0.47733333333333333, + "all@10": 0.136, + "recall@10": 0.4958333333333333, + "all@20": 0.136, + "recall@20": 0.4963333333333333 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.092, + "recall@5": 0.43616666666666665, + "all@10": 0.134, + "recall@10": 0.49366666666666664, + "all@20": 0.138, + "recall@20": 0.498 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.132, + "recall@5": 0.488, + "all@10": 0.168, + "recall@10": 0.5295, + "all@20": 0.252, + "recall@20": 0.6115 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.12, + "recall@5": 0.4746666666666666, + "all@10": 0.166, + "recall@10": 0.5263333333333333, + "all@20": 0.252, + "recall@20": 0.6128333333333333 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.134, + "recall@5": 0.49533333333333324, + "all@10": 0.202, + "recall@10": 0.5603333333333332, + "all@20": 0.286, + "recall@20": 0.6343333333333334 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "all@10": 0.2, + "recall@10": 0.5593333333333332, + "all@20": 0.284, + "recall@20": 0.6331666666666668 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.114, + "recall@5": 0.4453333333333333, + "all@10": 0.182, + "recall@10": 0.5416666666666666, + "all@20": 0.206, + "recall@20": 0.5648333333333333 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.046, + "recall@5": 0.32333333333333336, + "all@10": 0.164, + "recall@10": 0.5173333333333333, + "all@20": 0.206, + "recall@20": 0.5648333333333333 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.128, + "recall@5": 0.47983333333333333, + "all@10": 0.196, + "recall@10": 0.5533333333333332, + "all@20": 0.238, + "recall@20": 0.6015 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.092, + "recall@5": 0.43316666666666664, + "all@10": 0.186, + "recall@10": 0.549, + "all@20": 0.24, + "recall@20": 0.6028333333333333 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.14, + "recall@5": 0.496, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "all@20": 0.288, + "recall@20": 0.6341666666666668 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.12, + "recall@5": 0.48066666666666663, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "all@20": 0.28, + "recall@20": 0.6305 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.142, + "recall@5": 0.498, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.13, + "recall@5": 0.489, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.092, + "recall@5": 0.393, + "all@10": 0.18, + "recall@10": 0.5298333333333334, + "all@20": 0.25, + "recall@20": 0.6115 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.014, + "recall@5": 0.19816666666666666, + "all@10": 0.098, + "recall@10": 0.4048333333333333, + "all@20": 0.23, + "recall@20": 0.5813333333333333 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.118, + "recall@5": 0.4631666666666666, + "all@10": 0.196, + "recall@10": 0.549, + "all@20": 0.276, + "recall@20": 0.627 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.068, + "recall@5": 0.3673333333333333, + "all@10": 0.164, + "recall@10": 0.5193333333333332, + "all@20": 0.262, + "recall@20": 0.6183333333333333 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.492, + "all@10": 0.21, + "recall@10": 0.5625, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.12, + "recall@5": 0.4675, + "all@10": 0.186, + "recall@10": 0.5506666666666667, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.142, + "recall@5": 0.49683333333333324, + "all@10": 0.21, + "recall@10": 0.5656666666666668, + "all@20": 0.292, + "recall@20": 0.637 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.128, + "recall@5": 0.48483333333333334, + "all@10": 0.21, + "recall@10": 0.5621666666666667, + "all@20": 0.292, + "recall@20": 0.637 + } + ], + "ppr_selected": { + "5": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "20": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + } + }, + "lib_dev_grid": [ + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 17.535096999141388, + "default": true, + "all@5": 0.126, + "recall@5": 0.48866666666666664, + "n_ret@5": 5.0, + "all@10": 0.132, + "recall@10": 0.49366666666666664, + "n_ret@10": 9.108, + "all@20": 0.136, + "recall@20": 0.4963333333333333, + "n_ret@20": 14.224 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 16.201021600863896, + "default": false, + "all@5": 0.132, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.202, + "recall@10": 0.5596666666666666, + "n_ret@10": 10.0, + "all@20": 0.204, + "recall@20": 0.5638333333333333, + "n_ret@20": 19.126 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 77.18184940121137, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.242, + "recall@20": 0.6043333333333333, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 82.44513519876637, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.292, + "recall@20": 0.637, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 82.27653679950163, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.136, + "recall@10": 0.4963333333333333, + "n_ret@10": 9.108, + "all@20": 0.136, + "recall@20": 0.4963333333333333, + "n_ret@20": 14.224 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 83.72574719786644, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.292, + "recall@20": 0.637, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 18.665394600480795, + "default": false, + "all@5": 0.114, + "recall@5": 0.4596666666666666, + "n_ret@5": 5.0, + "all@10": 0.124, + "recall@10": 0.47883333333333333, + "n_ret@10": 9.804, + "all@20": 0.132, + "recall@20": 0.49066666666666664, + "n_ret@20": 18.898 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 16.18706979986746, + "default": false, + "all@5": 0.116, + "recall@5": 0.4686666666666666, + "n_ret@5": 5.0, + "all@10": 0.17, + "recall@10": 0.5276666666666667, + "n_ret@10": 10.0, + "all@20": 0.228, + "recall@20": 0.5901666666666667, + "n_ret@20": 19.94 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 16.184752200031653, + "default": false, + "all@5": 0.132, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.202, + "recall@10": 0.5596666666666666, + "n_ret@10": 10.0, + "all@20": 0.204, + "recall@20": 0.5638333333333333, + "n_ret@20": 18.202 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 82.00871140148956, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.292, + "recall@20": 0.637, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 17.888841799809597, + "default": false, + "all@5": 0.132, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.136, + "recall@10": 0.4963333333333333, + "n_ret@10": 9.804, + "all@20": 0.14, + "recall@20": 0.499, + "n_ret@20": 18.898 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 16.82623459934257, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.242, + "recall@20": 0.6043333333333333, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 73.14955199952237, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.138, + "recall@10": 0.4973333333333333, + "n_ret@10": 9.278, + "all@20": 0.138, + "recall@20": 0.4973333333333333, + "n_ret@20": 14.362 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 76.69009819976054, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.206, + "recall@20": 0.5653333333333332, + "n_ret@20": 19.308 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 76.91127519938163, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.242, + "recall@20": 0.6043333333333333, + "n_ret@20": 19.94 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 15.309481397503987, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.136, + "recall@10": 0.4963333333333333, + "n_ret@10": 9.278, + "all@20": 0.138, + "recall@20": 0.4973333333333333, + "n_ret@20": 14.362 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 79.84269959956873, + "default": false, + "all@5": 0.134, + "recall@5": 0.4953333333333333, + "n_ret@5": 5.0, + "all@10": 0.198, + "recall@10": 0.5583333333333333, + "n_ret@10": 10.0, + "all@20": 0.208, + "recall@20": 0.5658333333333333, + "n_ret@20": 19.948 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 15.828685600310564, + "default": false, + "all@5": 0.114, + "recall@5": 0.4626666666666666, + "n_ret@5": 5.0, + "all@10": 0.16, + "recall@10": 0.514, + "n_ret@10": 10.0, + "all@20": 0.19, + "recall@20": 0.5485, + "n_ret@20": 19.348 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 74.57226740033366, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.206, + "recall@20": 0.5648333333333333, + "n_ret@20": 19.126 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 77.92274939978961, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.206, + "recall@20": 0.5653333333333332, + "n_ret@20": 19.308 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 28.2100367997773, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.206, + "recall@20": 0.5648333333333333, + "n_ret@20": 19.988 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 81.84277840191498, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.242, + "recall@20": 0.6043333333333333, + "n_ret@20": 19.99 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 81.62459000083618, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.208, + "recall@20": 0.5665, + "n_ret@20": 19.948 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 77.72839259868488, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.292, + "recall@20": 0.637, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 73.60971079906449, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.14, + "recall@10": 0.499, + "n_ret@10": 9.804, + "all@20": 0.14, + "recall@20": 0.499, + "n_ret@20": 18.898 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 89.84426980104763, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.244, + "recall@20": 0.606, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 90.01915119949263, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.292, + "recall@20": 0.637, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 83.6438863981748, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.212, + "recall@20": 0.5685, + "n_ret@20": 19.988 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 15.864981799735688, + "default": false, + "all@5": 0.116, + "recall@5": 0.4679999999999999, + "n_ret@5": 5.0, + "all@10": 0.166, + "recall@10": 0.5236666666666667, + "n_ret@10": 10.0, + "all@20": 0.22, + "recall@20": 0.5848333333333332, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 69.85755539964885, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.242, + "recall@20": 0.6043333333333333, + "n_ret@20": 19.972 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 13.378630400402471, + "default": false, + "all@5": 0.132, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.202, + "recall@10": 0.5596666666666666, + "n_ret@10": 10.0, + "all@20": 0.24, + "recall@20": 0.6033333333333333, + "n_ret@20": 19.94 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 74.93218559864908, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.292, + "recall@20": 0.637, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 25.966374799609184, + "default": false, + "all@5": 0.132, + "recall@5": 0.4943333333333333, + "n_ret@5": 5.0, + "all@10": 0.198, + "recall@10": 0.5571666666666666, + "n_ret@10": 10.0, + "all@20": 0.204, + "recall@20": 0.5626666666666668, + "n_ret@20": 19.988 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 36.00333920016419, + "default": false, + "all@5": 0.114, + "recall@5": 0.4596666666666666, + "n_ret@5": 5.0, + "all@10": 0.154, + "recall@10": 0.5065, + "n_ret@10": 10.0, + "all@20": 0.184, + "recall@20": 0.5511666666666666, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 68.13168340025004, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.196, + "recall@10": 0.5573333333333333, + "n_ret@10": 10.0, + "all@20": 0.24, + "recall@20": 0.6036666666666666, + "n_ret@20": 19.99 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 76.62388240126893, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.208, + "recall@20": 0.5665, + "n_ret@20": 19.988 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 74.97056219971273, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.214, + "recall@20": 0.5701666666666667, + "n_ret@20": 19.988 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 72.3728930017678, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.292, + "recall@20": 0.637, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 13.554040601127781, + "default": false, + "all@5": 0.116, + "recall@5": 0.4686666666666666, + "n_ret@5": 5.0, + "all@10": 0.17, + "recall@10": 0.5276666666666667, + "n_ret@10": 10.0, + "all@20": 0.24, + "recall@20": 0.597, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 14.95122299855575, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.136, + "recall@10": 0.4963333333333333, + "n_ret@10": 9.804, + "all@20": 0.14, + "recall@20": 0.499, + "n_ret@20": 18.898 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 13.607935400097631, + "default": false, + "all@5": 0.114, + "recall@5": 0.4626666666666666, + "n_ret@5": 5.0, + "all@10": 0.16, + "recall@10": 0.514, + "n_ret@10": 10.0, + "all@20": 0.214, + "recall@20": 0.5733333333333333, + "n_ret@20": 20.0 + } + ], + "lib_selected": { + "5": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + } + }, + "n_lib_configs": 41, + "n_ppr_configs": 72, + "sub_ks": "10", + "seconds": 2504.375075817108 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/musique__minilm__ctrl_mention.npz b/benchmarks/multihop/results/raw/musique__minilm__ctrl_mention.npz new file mode 100644 index 0000000..a9ec6e9 Binary files /dev/null and b/benchmarks/multihop/results/raw/musique__minilm__ctrl_mention.npz differ diff --git a/benchmarks/multihop/results/raw/musique__minilm__mention.json b/benchmarks/multihop/results/raw/musique__minilm__mention.json new file mode 100644 index 0000000..5d342ed --- /dev/null +++ b/benchmarks/multihop/results/raw/musique__minilm__mention.json @@ -0,0 +1,2017 @@ +{ + "dataset": "musique", + "model": "minilm", + "variant": "mention", + "families": [ + "mention" + ], + "diagnostics": { + "n_edges_undirected_union": 20272, + "n_edges_by_family_directed_or_undirected": 22465, + "mean_degree": 2.6318727685816294, + "median_degree": 0.0, + "max_degree": 54, + "isolated_frac": 0.5272314183706589, + "gold_pairs_directly_linked": 0.14690791250337565, + "gold_pairs_linked_within_2": 0.2292735619767756, + "neighbours_of_gold_that_are_in_same_question_context": 0.1532126205792047 + }, + "ppr_dev_grid": [ + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.144, + "recall@5": 0.49816666666666665, + "all@10": 0.27, + "recall@10": 0.5776666666666668, + "all@20": 0.284, + "recall@20": 0.5915 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.15, + "recall@5": 0.5035, + "all@10": 0.294, + "recall@10": 0.5908333333333332, + "all@20": 0.352, + "recall@20": 0.6285 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.138, + "recall@5": 0.4958333333333333, + "all@10": 0.302, + "recall@10": 0.6068333333333332, + "all@20": 0.372, + "recall@20": 0.6766666666666666 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.14, + "recall@5": 0.49783333333333324, + "all@10": 0.31, + "recall@10": 0.6086666666666666, + "all@20": 0.386, + "recall@20": 0.6806666666666666 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.138, + "recall@5": 0.4958333333333333, + "all@10": 0.274, + "recall@10": 0.596, + "all@20": 0.354, + "recall@20": 0.6683333333333332 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.14, + "recall@5": 0.4978333333333333, + "all@10": 0.274, + "recall@10": 0.598, + "all@20": 0.354, + "recall@20": 0.6686666666666667 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.4958333333333333, + "all@10": 0.212, + "recall@10": 0.5696666666666667, + "all@20": 0.306, + "recall@20": 0.645 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.4968333333333333, + "all@10": 0.214, + "recall@10": 0.5701666666666667, + "all@20": 0.31, + "recall@20": 0.6466666666666666 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.14, + "recall@5": 0.49166666666666664, + "all@10": 0.216, + "recall@10": 0.5715, + "all@20": 0.34, + "recall@20": 0.6465 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.142, + "recall@5": 0.48683333333333334, + "all@10": 0.226, + "recall@10": 0.576, + "all@20": 0.366, + "recall@20": 0.66 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.138, + "recall@5": 0.49366666666666664, + "all@10": 0.206, + "recall@10": 0.5658333333333333, + "all@20": 0.358, + "recall@20": 0.6685 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.146, + "recall@5": 0.4946666666666667, + "all@10": 0.214, + "recall@10": 0.57, + "all@20": 0.376, + "recall@20": 0.677 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.146, + "recall@5": 0.4988333333333333, + "all@10": 0.206, + "recall@10": 0.5656666666666667, + "all@20": 0.35, + "recall@20": 0.666 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.146, + "recall@5": 0.49866666666666665, + "all@10": 0.21, + "recall@10": 0.5666666666666668, + "all@20": 0.348, + "recall@20": 0.6668333333333334 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.148, + "recall@5": 0.49983333333333335, + "all@10": 0.206, + "recall@10": 0.5641666666666666, + "all@20": 0.308, + "recall@20": 0.6463333333333332 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.15, + "recall@5": 0.501, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "all@20": 0.31, + "recall@20": 0.6453333333333334 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.14, + "recall@5": 0.4748333333333333, + "all@10": 0.228, + "recall@10": 0.5736666666666668, + "all@20": 0.31, + "recall@20": 0.6496666666666667 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.124, + "recall@5": 0.462, + "all@10": 0.228, + "recall@10": 0.5723333333333332, + "all@20": 0.324, + "recall@20": 0.6541666666666667 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.142, + "recall@5": 0.4898333333333333, + "all@10": 0.218, + "recall@10": 0.5696666666666667, + "all@20": 0.302, + "recall@20": 0.6453333333333332 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.144, + "recall@5": 0.47783333333333333, + "all@10": 0.226, + "recall@10": 0.573, + "all@20": 0.31, + "recall@20": 0.6475 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.14, + "recall@5": 0.495, + "all@10": 0.21, + "recall@10": 0.5685, + "all@20": 0.294, + "recall@20": 0.6408333333333333 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.138, + "recall@5": 0.49016666666666675, + "all@10": 0.226, + "recall@10": 0.5731666666666666, + "all@20": 0.302, + "recall@20": 0.6431666666666667 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.146, + "recall@5": 0.4973333333333333, + "all@10": 0.204, + "recall@10": 0.565, + "all@20": 0.292, + "recall@20": 0.6386666666666667 + }, + { + "cfg": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.148, + "recall@5": 0.5016666666666667, + "all@10": 0.21, + "recall@10": 0.5665, + "all@20": 0.292, + "recall@20": 0.6376666666666667 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.186, + "recall@5": 0.519, + "all@10": 0.27, + "recall@10": 0.5781666666666667, + "all@20": 0.282, + "recall@20": 0.59 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.18, + "recall@5": 0.5135, + "all@10": 0.292, + "recall@10": 0.59, + "all@20": 0.35, + "recall@20": 0.6281666666666668 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.162, + "recall@5": 0.509, + "all@10": 0.294, + "recall@10": 0.6003333333333334, + "all@20": 0.362, + "recall@20": 0.6703333333333332 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.186, + "recall@5": 0.5183333333333333, + "all@10": 0.316, + "recall@10": 0.6118333333333332, + "all@20": 0.392, + "recall@20": 0.6758333333333333 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.15, + "recall@5": 0.5015, + "all@10": 0.28, + "recall@10": 0.5995, + "all@20": 0.366, + "recall@20": 0.6728333333333333 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.162, + "recall@5": 0.5101666666666667, + "all@10": 0.296, + "recall@10": 0.6055, + "all@20": 0.362, + "recall@20": 0.67 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.49516666666666664, + "all@10": 0.22, + "recall@10": 0.573, + "all@20": 0.316, + "recall@20": 0.6511666666666668 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.142, + "recall@5": 0.49916666666666665, + "all@10": 0.236, + "recall@10": 0.5811666666666667, + "all@20": 0.322, + "recall@20": 0.6513333333333332 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.162, + "recall@5": 0.48883333333333334, + "all@10": 0.254, + "recall@10": 0.5881666666666667, + "all@20": 0.34, + "recall@20": 0.6461666666666667 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.162, + "recall@5": 0.4651666666666666, + "all@10": 0.26, + "recall@10": 0.5878333333333332, + "all@20": 0.364, + "recall@20": 0.6595 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.154, + "recall@5": 0.4948333333333333, + "all@10": 0.232, + "recall@10": 0.58, + "all@20": 0.35, + "recall@20": 0.6613333333333332 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.17, + "recall@5": 0.4905, + "all@10": 0.258, + "recall@10": 0.5925, + "all@20": 0.382, + "recall@20": 0.6756666666666666 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.146, + "recall@5": 0.5005, + "all@10": 0.212, + "recall@10": 0.5723333333333334, + "all@20": 0.36, + "recall@20": 0.6686666666666666 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.156, + "recall@5": 0.4983333333333333, + "all@10": 0.246, + "recall@10": 0.5853333333333334, + "all@20": 0.358, + "recall@20": 0.6691666666666666 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.148, + "recall@5": 0.49916666666666665, + "all@10": 0.204, + "recall@10": 0.5638333333333333, + "all@20": 0.316, + "recall@20": 0.6498333333333334 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.146, + "recall@5": 0.49916666666666665, + "all@10": 0.208, + "recall@10": 0.5673333333333332, + "all@20": 0.322, + "recall@20": 0.651 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.146, + "recall@5": 0.45, + "all@10": 0.258, + "recall@10": 0.5775, + "all@20": 0.338, + "recall@20": 0.6615 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.41716666666666663, + "all@10": 0.226, + "recall@10": 0.554, + "all@20": 0.35, + "recall@20": 0.6665 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.144, + "recall@5": 0.4751666666666666, + "all@10": 0.242, + "recall@10": 0.5806666666666667, + "all@20": 0.326, + "recall@20": 0.6556666666666667 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.146, + "recall@5": 0.464, + "all@10": 0.254, + "recall@10": 0.58, + "all@20": 0.342, + "recall@20": 0.6646666666666667 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.14, + "recall@5": 0.49699999999999994, + "all@10": 0.216, + "recall@10": 0.5726666666666667, + "all@20": 0.31, + "recall@20": 0.6476666666666666 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.142, + "recall@5": 0.48116666666666663, + "all@10": 0.25, + "recall@10": 0.5835, + "all@20": 0.336, + "recall@20": 0.6586666666666667 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.148, + "recall@5": 0.49933333333333324, + "all@10": 0.202, + "recall@10": 0.564, + "all@20": 0.296, + "recall@20": 0.6411666666666667 + }, + { + "cfg": { + "alpha": 0.7, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.146, + "recall@5": 0.49533333333333335, + "all@10": 0.218, + "recall@10": 0.5685, + "all@20": 0.298, + "recall@20": 0.6411666666666667 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.182, + "recall@5": 0.48833333333333334, + "all@10": 0.264, + "recall@10": 0.5731666666666667, + "all@20": 0.282, + "recall@20": 0.59 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.18, + "recall@5": 0.497, + "all@10": 0.288, + "recall@10": 0.588, + "all@20": 0.346, + "recall@20": 0.6275 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.18, + "recall@5": 0.5098333333333332, + "all@10": 0.288, + "recall@10": 0.5928333333333332, + "all@20": 0.354, + "recall@20": 0.6593333333333332 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.19, + "recall@5": 0.5193333333333332, + "all@10": 0.308, + "recall@10": 0.6103333333333334, + "all@20": 0.394, + "recall@20": 0.6728333333333333 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.16, + "recall@5": 0.5089999999999999, + "all@10": 0.282, + "recall@10": 0.6013333333333334, + "all@20": 0.366, + "recall@20": 0.6743333333333333 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.186, + "recall@5": 0.5213333333333332, + "all@10": 0.306, + "recall@10": 0.6056666666666666, + "all@20": 0.38, + "recall@20": 0.6785 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.144, + "recall@5": 0.4985, + "all@10": 0.226, + "recall@10": 0.5776666666666668, + "all@20": 0.316, + "recall@20": 0.6511666666666668 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.146, + "recall@5": 0.5005, + "all@10": 0.248, + "recall@10": 0.5841666666666666, + "all@20": 0.322, + "recall@20": 0.6518333333333333 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.156, + "recall@5": 0.43833333333333335, + "all@10": 0.246, + "recall@10": 0.5685, + "all@20": 0.332, + "recall@20": 0.6411666666666668 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.156, + "recall@5": 0.41233333333333333, + "all@10": 0.246, + "recall@10": 0.5643333333333332, + "all@20": 0.364, + "recall@20": 0.6601666666666667 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.162, + "recall@5": 0.4746666666666666, + "all@10": 0.254, + "recall@10": 0.5831666666666667, + "all@20": 0.344, + "recall@20": 0.6535 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.162, + "recall@5": 0.4665, + "all@10": 0.28, + "recall@10": 0.601, + "all@20": 0.384, + "recall@20": 0.675 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.152, + "recall@5": 0.5016666666666666, + "all@10": 0.23, + "recall@10": 0.5791666666666666, + "all@20": 0.352, + "recall@20": 0.665 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.172, + "recall@5": 0.49599999999999994, + "all@10": 0.266, + "recall@10": 0.5956666666666668, + "all@20": 0.368, + "recall@20": 0.673 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.146, + "recall@5": 0.499, + "all@10": 0.206, + "recall@10": 0.5656666666666667, + "all@20": 0.322, + "recall@20": 0.6525 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.148, + "recall@5": 0.5003333333333333, + "all@10": 0.216, + "recall@10": 0.5716666666666667, + "all@20": 0.324, + "recall@20": 0.6523333333333332 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.126, + "recall@5": 0.38166666666666665, + "all@10": 0.234, + "recall@10": 0.529, + "all@20": 0.326, + "recall@20": 0.6448333333333333 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.0, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.122, + "recall@5": 0.3213333333333333, + "all@10": 0.206, + "recall@10": 0.48833333333333334, + "all@20": 0.35, + "recall@20": 0.6433333333333333 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.435, + "all@10": 0.252, + "recall@10": 0.5696666666666667, + "all@20": 0.324, + "recall@20": 0.6516666666666666 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.136, + "recall@5": 0.4086666666666666, + "all@10": 0.248, + "recall@10": 0.5668333333333334, + "all@20": 0.36, + "recall@20": 0.6673333333333332 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.144, + "recall@5": 0.489, + "all@10": 0.23, + "recall@10": 0.5763333333333333, + "all@20": 0.324, + "recall@20": 0.6531666666666667 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.154, + "recall@5": 0.47866666666666663, + "all@10": 0.262, + "recall@10": 0.5846666666666668, + "all@20": 0.34, + "recall@20": 0.6613333333333332 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "all@5": 0.144, + "recall@5": 0.4978333333333333, + "all@10": 0.208, + "recall@10": 0.5678333333333333, + "all@20": 0.296, + "recall@20": 0.6426666666666666 + }, + { + "cfg": { + "alpha": 0.85, + "seeds": 20, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "all@5": 0.144, + "recall@5": 0.4975, + "all@10": 0.222, + "recall@10": 0.5726666666666668, + "all@20": 0.31, + "recall@20": 0.6476666666666666 + } + ], + "ppr_selected": { + "5": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "10": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "20": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + } + }, + "lib_dev_grid": [ + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 17.17872359859757, + "default": true, + "all@5": 0.208, + "recall@5": 0.5348333333333333, + "n_ret@5": 5.0, + "all@10": 0.276, + "recall@10": 0.5796666666666667, + "n_ret@10": 8.78, + "all@20": 0.296, + "recall@20": 0.596, + "n_ret@20": 13.244 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 16.771423799800687, + "default": false, + "all@5": 0.19, + "recall@5": 0.5256666666666666, + "n_ret@5": 5.0, + "all@10": 0.268, + "recall@10": 0.5956666666666668, + "n_ret@10": 10.0, + "all@20": 0.34, + "recall@20": 0.6463333333333332, + "n_ret@20": 18.42 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 77.72793480055407, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.348, + "recall@20": 0.6641666666666668, + "n_ret@20": 19.902 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 79.86942400026601, + "default": false, + "all@5": 0.166, + "recall@5": 0.5123333333333333, + "n_ret@5": 5.0, + "all@10": 0.242, + "recall@10": 0.5845, + "n_ret@10": 10.0, + "all@20": 0.336, + "recall@20": 0.6618333333333333, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 77.71402279788163, + "default": false, + "all@5": 0.164, + "recall@5": 0.5098333333333334, + "n_ret@5": 5.0, + "all@10": 0.274, + "recall@10": 0.5825, + "n_ret@10": 8.78, + "all@20": 0.296, + "recall@20": 0.6001666666666666, + "n_ret@20": 13.244 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 86.0032239984721, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.292, + "recall@20": 0.637, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 17.97306899970863, + "default": false, + "all@5": 0.196, + "recall@5": 0.5071666666666667, + "n_ret@5": 5.0, + "all@10": 0.286, + "recall@10": 0.575, + "n_ret@10": 9.606, + "all@20": 0.348, + "recall@20": 0.6208333333333333, + "n_ret@20": 18.026 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 16.594663199735805, + "default": false, + "all@5": 0.208, + "recall@5": 0.5216666666666666, + "n_ret@5": 5.0, + "all@10": 0.286, + "recall@10": 0.595, + "n_ret@10": 10.0, + "all@20": 0.36, + "recall@20": 0.6636666666666666, + "n_ret@20": 19.792 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 16.536048600566573, + "default": false, + "all@5": 0.19, + "recall@5": 0.5256666666666666, + "n_ret@5": 5.0, + "all@10": 0.268, + "recall@10": 0.5956666666666668, + "n_ret@10": 10.0, + "all@20": 0.34, + "recall@20": 0.6458333333333333, + "n_ret@20": 17.482 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 81.40909900015686, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.208, + "recall@10": 0.5658333333333333, + "n_ret@10": 10.0, + "all@20": 0.296, + "recall@20": 0.6386666666666667, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 18.280491001554765, + "default": false, + "all@5": 0.19, + "recall@5": 0.5243333333333332, + "n_ret@5": 5.0, + "all@10": 0.292, + "recall@10": 0.5871666666666667, + "n_ret@10": 9.606, + "all@20": 0.348, + "recall@20": 0.6245, + "n_ret@20": 18.026 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 17.29915940097999, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.218, + "recall@10": 0.5705, + "n_ret@10": 10.0, + "all@20": 0.368, + "recall@20": 0.6753333333333333, + "n_ret@20": 19.902 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 74.75133060035296, + "default": false, + "all@5": 0.158, + "recall@5": 0.5076666666666666, + "n_ret@5": 5.0, + "all@10": 0.298, + "recall@10": 0.5973333333333334, + "n_ret@10": 9.052, + "all@20": 0.336, + "recall@20": 0.622, + "n_ret@20": 13.914 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 78.24859460047446, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.336, + "recall@20": 0.6453333333333332, + "n_ret@20": 18.642 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 75.15391299990006, + "default": false, + "all@5": 0.138, + "recall@5": 0.4983333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5678333333333333, + "n_ret@10": 10.0, + "all@20": 0.362, + "recall@20": 0.6718333333333334, + "n_ret@20": 19.792 + }, + { + "cfg": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 16.194531799294055, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.29, + "recall@10": 0.5868333333333333, + "n_ret@10": 9.052, + "all@20": 0.32, + "recall@20": 0.6083333333333333, + "n_ret@20": 13.914 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 80.2047837987775, + "default": false, + "all@5": 0.212, + "recall@5": 0.5335, + "n_ret@5": 5.0, + "all@10": 0.316, + "recall@10": 0.6168333333333332, + "n_ret@10": 10.0, + "all@20": 0.396, + "recall@20": 0.6815, + "n_ret@20": 19.81 + }, + { + "cfg": { + "hops": 1, + "seeds": 10, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 15.839632401359268, + "default": false, + "all@5": 0.196, + "recall@5": 0.509, + "n_ret@5": 5.0, + "all@10": 0.282, + "recall@10": 0.5836666666666668, + "n_ret@10": 10.0, + "all@20": 0.348, + "recall@20": 0.6375, + "n_ret@20": 18.82 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 76.56100100069307, + "default": false, + "all@5": 0.156, + "recall@5": 0.5073333333333333, + "n_ret@5": 5.0, + "all@10": 0.234, + "recall@10": 0.5791666666666666, + "n_ret@10": 10.0, + "all@20": 0.338, + "recall@20": 0.6475, + "n_ret@20": 18.42 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 80.66739179950673, + "default": false, + "all@5": 0.156, + "recall@5": 0.5073333333333333, + "n_ret@5": 5.0, + "all@10": 0.234, + "recall@10": 0.5791666666666666, + "n_ret@10": 10.0, + "all@20": 0.338, + "recall@20": 0.6475, + "n_ret@20": 18.642 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 30.45321339997463, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.218, + "recall@10": 0.5705, + "n_ret@10": 10.0, + "all@20": 0.366, + "recall@20": 0.6541666666666667, + "n_ret@20": 19.864 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 85.1823989986442, + "default": false, + "all@5": 0.166, + "recall@5": 0.5123333333333333, + "n_ret@5": 5.0, + "all@10": 0.242, + "recall@10": 0.5845, + "n_ret@10": 10.0, + "all@20": 0.374, + "recall@20": 0.6756666666666666, + "n_ret@20": 19.892 + }, + { + "cfg": { + "hops": 2, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 83.39649600035045, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.386, + "recall@20": 0.6735, + "n_ret@20": 19.81 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 76.2925341995433, + "default": false, + "all@5": 0.156, + "recall@5": 0.5073333333333333, + "n_ret@5": 5.0, + "all@10": 0.234, + "recall@10": 0.5791666666666666, + "n_ret@10": 10.0, + "all@20": 0.338, + "recall@20": 0.659, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 74.43941820063628, + "default": false, + "all@5": 0.17, + "recall@5": 0.5143333333333333, + "n_ret@5": 5.0, + "all@10": 0.312, + "recall@10": 0.6118333333333333, + "n_ret@10": 9.606, + "all@20": 0.356, + "recall@20": 0.6473333333333332, + "n_ret@20": 18.026 + }, + { + "cfg": { + "hops": 3, + "seeds": 15, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 89.72099620033987, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.208, + "recall@10": 0.5658333333333333, + "n_ret@10": 10.0, + "all@20": 0.386, + "recall@20": 0.684, + "n_ret@20": 19.99 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.7, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 97.73494020209182, + "default": false, + "all@5": 0.138, + "recall@5": 0.4983333333333333, + "n_ret@5": 5.0, + "all@10": 0.212, + "recall@10": 0.5688333333333333, + "n_ret@10": 10.0, + "all@20": 0.308, + "recall@20": 0.644, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 82.57190200185869, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.382, + "recall@20": 0.6703333333333332, + "n_ret@20": 19.864 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 15.720678200596014, + "default": false, + "all@5": 0.206, + "recall@5": 0.5206666666666666, + "n_ret@5": 5.0, + "all@10": 0.284, + "recall@10": 0.5916666666666667, + "n_ret@10": 10.0, + "all@20": 0.356, + "recall@20": 0.6576666666666666, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 15, + "decay": 0.7, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 67.9759269998176, + "default": false, + "all@5": 0.138, + "recall@5": 0.4983333333333333, + "n_ret@5": 5.0, + "all@10": 0.21, + "recall@10": 0.5678333333333333, + "n_ret@10": 10.0, + "all@20": 0.362, + "recall@20": 0.6711666666666667, + "n_ret@20": 19.902 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 13.701403600629419, + "default": false, + "all@5": 0.19, + "recall@5": 0.5256666666666666, + "n_ret@5": 5.0, + "all@10": 0.268, + "recall@10": 0.5956666666666668, + "n_ret@10": 10.0, + "all@20": 0.368, + "recall@20": 0.6753333333333333, + "n_ret@20": 19.792 + }, + { + "cfg": { + "hops": 2, + "seeds": 20, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 75.39702739822678, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.208, + "recall@10": 0.5658333333333333, + "n_ret@10": 10.0, + "all@20": 0.296, + "recall@20": 0.6386666666666667, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 27.779880998888984, + "default": false, + "all@5": 0.19, + "recall@5": 0.5243333333333332, + "n_ret@5": 5.0, + "all@10": 0.272, + "recall@10": 0.597, + "n_ret@10": 10.0, + "all@20": 0.374, + "recall@20": 0.6588333333333333, + "n_ret@20": 19.864 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 39.358650201698765, + "default": false, + "all@5": 0.196, + "recall@5": 0.5065, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.5883333333333334, + "n_ret@10": 10.0, + "all@20": 0.374, + "recall@20": 0.6508333333333333, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 65.9130734000355, + "default": false, + "all@5": 0.21, + "recall@5": 0.5325, + "n_ret@5": 5.0, + "all@10": 0.298, + "recall@10": 0.6071666666666667, + "n_ret@10": 10.0, + "all@20": 0.382, + "recall@20": 0.6823333333333332, + "n_ret@20": 19.892 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 78.43063599953894, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.208, + "recall@10": 0.5658333333333333, + "n_ret@10": 10.0, + "all@20": 0.384, + "recall@20": 0.6713333333333332, + "n_ret@20": 19.864 + }, + { + "cfg": { + "hops": 3, + "seeds": 10, + "decay": 0.5, + "blend": 0.6, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 76.84294040000532, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.382, + "recall@20": 0.6708333333333333, + "n_ret@20": 19.864 + }, + { + "cfg": { + "hops": 3, + "seeds": 20, + "decay": 0.5, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 71.9971677995054, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.206, + "recall@10": 0.5648333333333333, + "n_ret@10": 10.0, + "all@20": 0.292, + "recall@20": 0.637, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "ms_per_query": 14.141759798512794, + "default": false, + "all@5": 0.208, + "recall@5": 0.5216666666666666, + "n_ret@5": 5.0, + "all@10": 0.286, + "recall@10": 0.595, + "n_ret@10": 10.0, + "all@20": 0.366, + "recall@20": 0.6678333333333334, + "n_ret@20": 20.0 + }, + { + "cfg": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 15.13236160040833, + "default": false, + "all@5": 0.136, + "recall@5": 0.4963333333333333, + "n_ret@5": 5.0, + "all@10": 0.296, + "recall@10": 0.5895, + "n_ret@10": 9.606, + "all@20": 0.348, + "recall@20": 0.6248333333333332, + "n_ret@20": 18.026 + }, + { + "cfg": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "ms_per_query": 14.666240599472076, + "default": false, + "all@5": 0.196, + "recall@5": 0.509, + "n_ret@5": 5.0, + "all@10": 0.282, + "recall@10": 0.5836666666666668, + "n_ret@10": 10.0, + "all@20": 0.36, + "recall@20": 0.6518333333333334, + "n_ret@20": 20.0 + } + ], + "lib_selected": { + "5": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "10": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "20": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + } + }, + "n_lib_configs": 41, + "n_ppr_configs": 72, + "sub_ks": "10", + "seconds": 2433.0824415683746 +} \ No newline at end of file diff --git a/benchmarks/multihop/results/raw/musique__minilm__mention.npz b/benchmarks/multihop/results/raw/musique__minilm__mention.npz new file mode 100644 index 0000000..35f1a39 Binary files /dev/null and b/benchmarks/multihop/results/raw/musique__minilm__mention.npz differ diff --git a/benchmarks/multihop/results/raw/musique__minilm__rerank_scores.npy b/benchmarks/multihop/results/raw/musique__minilm__rerank_scores.npy new file mode 100644 index 0000000..dcb8f59 Binary files /dev/null and b/benchmarks/multihop/results/raw/musique__minilm__rerank_scores.npy differ diff --git a/benchmarks/multihop/results/raw/musique__minilm__vector_top50.npy b/benchmarks/multihop/results/raw/musique__minilm__vector_top50.npy new file mode 100644 index 0000000..b64dea9 Binary files /dev/null and b/benchmarks/multihop/results/raw/musique__minilm__vector_top50.npy differ diff --git a/benchmarks/multihop/results/summary.json b/benchmarks/multihop/results/summary.json new file mode 100644 index 0000000..2de0f5d --- /dev/null +++ b/benchmarks/multihop/results/summary.json @@ -0,0 +1,19054 @@ +{ + "B_boot": 2000, + "B_perm": 10000, + "results": { + "hotpotqa": { + "minilm": { + "n_test": 1000, + "corpus_size": 14570, + "lib_similarity_search_same_top20_set": "200/200", + "graphs": { + "mention": { + "n_edges_undirected_union": 11348, + "n_edges_by_family_directed_or_undirected": 12354, + "mean_degree": 1.5577213452299246, + "median_degree": 1.0, + "max_degree": 29, + "isolated_frac": 0.39958819492107067, + "gold_pairs_directly_linked": 0.6513333333333333, + "gold_pairs_linked_within_2": 0.6673333333333333, + "neighbours_of_gold_that_are_in_same_question_context": 0.5890391209006267 + }, + "entity": { + "n_edges_undirected_union": 59717, + "n_edges_by_family_directed_or_undirected": 59717, + "mean_degree": 8.197254632807137, + "median_degree": 6.0, + "max_degree": 75, + "isolated_frac": 0.14406314344543583, + "gold_pairs_directly_linked": 0.498, + "gold_pairs_linked_within_2": 0.5433333333333333, + "neighbours_of_gold_that_are_in_same_question_context": 0.29354594497084746 + }, + "knn": { + "n_edges_undirected_union": 52527, + "n_edges_by_family_directed_or_undirected": 52527, + "mean_degree": 7.210295126973232, + "median_degree": 6.0, + "max_degree": 41, + "isolated_frac": 0.0, + "gold_pairs_directly_linked": 0.4126666666666667, + "gold_pairs_linked_within_2": 0.5146666666666667, + "neighbours_of_gold_that_are_in_same_question_context": 0.22458679835250345 + }, + "ctrl_mention": { + "n_edges_undirected_union": 11348, + "n_edges_by_family_directed_or_undirected": 12354, + "mean_degree": 1.5577213452299246, + "median_degree": 1.0, + "max_degree": 29, + "isolated_frac": 0.39958819492107067, + "gold_pairs_directly_linked": 0.0, + "gold_pairs_linked_within_2": 0.0, + "neighbours_of_gold_that_are_in_same_question_context": 0.00026518164942985947 + } + }, + "selected": { + "mention": { + "lib": { + "5": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 2, + "seeds": 5, + "decay": 0.5, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "20": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + } + }, + "ppr": { + "5": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "20": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + } + } + }, + "entity": { + "lib": { + "5": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 1, + "seeds": 10, + "decay": 0.5, + "blend": 0.3, + "direction": "out", + "weight_field": null + } + }, + "ppr": { + "5": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "20": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + } + } + }, + "knn": { + "lib": { + "5": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + } + }, + "ppr": { + "5": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "20": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + } + } + }, + "ctrl_mention": { + "lib": { + "5": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + } + }, + "ppr": { + "5": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "20": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + } + } + } + }, + "tables": { + "5": [ + { + "label": "Vector top-k (plain)", + "key": "vector", + "variant": null, + "recall": [ + 0.703000009059906, + 0.684499979019165, + 0.722000002861023 + ], + "all": [ + 0.47200000286102295, + 0.4410000145435333, + 0.503000020980835 + ], + "mrr": [ + 0.8558666706085205, + 0.8361496329307556, + 0.8745683431625366 + ], + "n_ret": 5.0 + }, + { + "label": "BM25", + "key": "bm25", + "variant": null, + "recall": [ + 0.722000002861023, + 0.7039999961853027, + 0.7390000224113464 + ], + "all": [ + 0.4869999885559082, + 0.4569999873638153, + 0.5170000195503235 + ], + "mrr": [ + 0.8460832834243774, + 0.8281295895576477, + 0.8622362613677979 + ], + "n_ret": 5.0, + "d_all": [ + 0.014999999664723873, + -0.020999999716877937, + 0.04899999871850014, + 0.4373562643735626 + ], + "d_recall": [ + 0.01899999938905239, + -0.003000000026077032, + 0.039500001817941666, + 0.08739126087391261 + ], + "d_mrr": [ + -0.009783332236111164, + -0.03131749853491783, + 0.011884999461472034, + 0.38346165383461656 + ], + "d_all_holm": 1.0, + "d_recall_holm": 0.6117388261173883 + }, + { + "label": "Hybrid vector+BM25 (RRF)", + "key": "rrf", + "variant": null, + "recall": [ + 0.7559999823570251, + 0.7384999990463257, + 0.7730000019073486 + ], + "all": [ + 0.5429999828338623, + 0.5109999775886536, + 0.5740000009536743 + ], + "mrr": [ + 0.8672500252723694, + 0.8505958318710327, + 0.8841674327850342 + ], + "n_ret": 5.0, + "d_all": [ + 0.07100000232458115, + 0.04600000008940697, + 0.09402499347925186, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.05299999937415123, + 0.03849999979138374, + 0.06650000065565109, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.01138333324342966, + -0.0029512490145862103, + 0.02545125037431717, + 0.10738926107389261 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "Vector top-30 + cross-encoder", + "key": "rerank", + "variant": null, + "recall": [ + 0.7875000238418579, + 0.7704874873161316, + 0.8034999966621399 + ], + "all": [ + 0.597000002861023, + 0.56597501039505, + 0.6259999871253967 + ], + "mrr": [ + 0.9378166794776917, + 0.9243828654289246, + 0.9503666758537292 + ], + "n_ret": 5.0, + "d_all": [ + 0.125, + 0.0989999994635582, + 0.15199999511241913, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.08449999988079071, + 0.06948749721050262, + 0.09950000047683716, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.08194999396800995, + 0.06418208032846451, + 0.0998566746711731, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph chosen on dev: mention]", + "key": "lib_mh@best", + "variant": "mention", + "recall": [ + 0.8230000138282776, + 0.8040000200271606, + 0.8410000205039978 + ], + "all": [ + 0.7110000252723694, + 0.6830000281333923, + 0.7379999756813049 + ], + "mrr": [ + 0.8549500107765198, + 0.8350499868392944, + 0.8740166425704956 + ], + "n_ret": 5.0, + "d_all": [ + 0.23899999260902405, + 0.20999999344348907, + 0.26899999380111694, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.11999999731779099, + 0.1054999977350235, + 0.1354999989271164, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0009166666422970593, + -0.0025833332911133766, + 0.0006504167686216533, + 0.2746725327467253 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention]", + "key": "ppr@best", + "variant": "mention", + "recall": [ + 0.7870000004768372, + 0.7684999704360962, + 0.8050000071525574 + ], + "all": [ + 0.6359999775886536, + 0.6079750061035156, + 0.6639999747276306 + ], + "mrr": [ + 0.7632166743278503, + 0.742199182510376, + 0.7836516499519348 + ], + "n_ret": 5.0, + "d_all": [ + 0.164000004529953, + 0.1340000033378601, + 0.19499999284744263, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.08399999886751175, + 0.06849999725818634, + 0.10050000250339508, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.09264999628067017, + -0.11145374178886414, + -0.07264832407236099, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph chosen on dev: mention]", + "key": "lib_default@best", + "variant": "mention", + "recall": [ + 0.8044999837875366, + 0.7864999771118164, + 0.8220125436782837 + ], + "all": [ + 0.6759999990463257, + 0.6480000019073486, + 0.7039999961853027 + ], + "mrr": [ + 0.8556166887283325, + 0.8359425067901611, + 0.8743837475776672 + ], + "n_ret": 5.0, + "d_all": [ + 0.20399999618530273, + 0.17800000309944153, + 0.23100000619888306, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.1014999970793724, + 0.0885000005364418, + 0.11500000208616257, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0002500000409781933, + -0.0012333333725109696, + 0.0006666666595265269, + 0.5052494750524947 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: mention]", + "key": "lib_mh:mention", + "variant": "mention", + "recall": [ + 0.8230000138282776, + 0.8040000200271606, + 0.8410000205039978 + ], + "all": [ + 0.7110000252723694, + 0.6830000281333923, + 0.7379999756813049 + ], + "mrr": [ + 0.8549500107765198, + 0.8350499868392944, + 0.8740166425704956 + ], + "n_ret": 5.0, + "d_all": [ + 0.23899999260902405, + 0.20999999344348907, + 0.26899999380111694, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.11999999731779099, + 0.1054999977350235, + 0.1354999989271164, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0009166666422970593, + -0.0025833332911133766, + 0.0006504167686216533, + 0.2746725327467253 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: entity]", + "key": "lib_mh:entity", + "variant": "entity", + "recall": [ + 0.7490000128746033, + 0.7304999828338623, + 0.7680000066757202 + ], + "all": [ + 0.5619999766349792, + 0.531000018119812, + 0.5920000076293945 + ], + "mrr": [ + 0.8562166690826416, + 0.8364816904067993, + 0.8748870491981506 + ], + "n_ret": 5.0, + "d_all": [ + 0.09000000357627869, + 0.06499999761581421, + 0.11500000208616257, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.04600000008940697, + 0.032999999821186066, + 0.05900000035762787, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.00035000007483176887, + -0.0016504165250808, + 0.002617083489894867, + 0.7470252974702529 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: knn]", + "key": "lib_mh:knn", + "variant": "knn", + "recall": [ + 0.7059999704360962, + 0.6875, + 0.7250000238418579 + ], + "all": [ + 0.4779999852180481, + 0.4480000138282776, + 0.5099999904632568 + ], + "mrr": [ + 0.8558666706085205, + 0.8361496329307556, + 0.8745683431625366 + ], + "n_ret": 5.0, + "d_all": [ + 0.006000000052154064, + -0.006000000052154064, + 0.017999999225139618, + 0.39336066393360664 + ], + "d_recall": [ + 0.003000000026077032, + -0.003000000026077032, + 0.008999999612569809, + 0.39336066393360664 + ], + "d_mrr": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: ctrl_mention]", + "key": "lib_mh:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.703000009059906, + 0.684499979019165, + 0.722000002861023 + ], + "all": [ + 0.47200000286102295, + 0.4410000145435333, + 0.503000020980835 + ], + "mrr": [ + 0.8558666706085205, + 0.8361496329307556, + 0.8745683431625366 + ], + "n_ret": 5.0, + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_mrr": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: mention]", + "key": "ppr:mention", + "variant": "mention", + "recall": [ + 0.7870000004768372, + 0.7684999704360962, + 0.8050000071525574 + ], + "all": [ + 0.6359999775886536, + 0.6079750061035156, + 0.6639999747276306 + ], + "mrr": [ + 0.7632166743278503, + 0.742199182510376, + 0.7836516499519348 + ], + "n_ret": 5.0, + "d_all": [ + 0.164000004529953, + 0.1340000033378601, + 0.19499999284744263, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.08399999886751175, + 0.06849999725818634, + 0.10050000250339508, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.09264999628067017, + -0.11145374178886414, + -0.07264832407236099, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: entity]", + "key": "ppr:entity", + "variant": "entity", + "recall": [ + 0.7009999752044678, + 0.6819999814033508, + 0.7200000286102295 + ], + "all": [ + 0.4729999899864197, + 0.44200000166893005, + 0.5040000081062317 + ], + "mrr": [ + 0.7645666599273682, + 0.7428662776947021, + 0.7858337759971619 + ], + "n_ret": 5.0, + "d_all": [ + 0.0010000000474974513, + -0.014000000432133675, + 0.014999999664723873, + 1.0 + ], + "d_recall": [ + -0.0020000000949949026, + -0.009999999776482582, + 0.006000000052154064, + 0.7191280871912809 + ], + "d_mrr": [ + -0.09130000323057175, + -0.10929999500513077, + -0.07476499676704407, + 9.999000099990002e-05 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: knn]", + "key": "ppr:knn", + "variant": "knn", + "recall": [ + 0.703000009059906, + 0.684499979019165, + 0.722000002861023 + ], + "all": [ + 0.47200000286102295, + 0.4410000145435333, + 0.503000020980835 + ], + "mrr": [ + 0.8143499493598938, + 0.7939162850379944, + 0.8337845802307129 + ], + "n_ret": 5.0, + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_mrr": [ + -0.041516661643981934, + -0.052170827984809875, + -0.03058166429400444, + 9.999000099990002e-05 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: ctrl_mention]", + "key": "ppr:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.7024999856948853, + 0.6840000152587891, + 0.7214999794960022 + ], + "all": [ + 0.47099998593330383, + 0.4399999976158142, + 0.5020249485969543 + ], + "mrr": [ + 0.8358666896820068, + 0.8155491948127747, + 0.8540350198745728 + ], + "n_ret": 5.0, + "d_all": [ + -0.0010000000474974513, + -0.003000000026077032, + 0.0, + 1.0 + ], + "d_recall": [ + -0.0005000000237487257, + -0.001500000013038516, + 0.0, + 1.0 + ], + "d_mrr": [ + -0.019999999552965164, + -0.028299998492002487, + -0.011982916854321957, + 9.999000099990002e-05 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: mention]", + "key": "lib_default:mention", + "variant": "mention", + "recall": [ + 0.8044999837875366, + 0.7864999771118164, + 0.8220125436782837 + ], + "all": [ + 0.6759999990463257, + 0.6480000019073486, + 0.7039999961853027 + ], + "mrr": [ + 0.8556166887283325, + 0.8359425067901611, + 0.8743837475776672 + ], + "n_ret": 5.0, + "d_all": [ + 0.20399999618530273, + 0.17800000309944153, + 0.23100000619888306, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.1014999970793724, + 0.0885000005364418, + 0.11500000208616257, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0002500000409781933, + -0.0012333333725109696, + 0.0006666666595265269, + 0.5052494750524947 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: entity]", + "key": "lib_default:entity", + "variant": "entity", + "recall": [ + 0.7039999961853027, + 0.6850000023841858, + 0.7235000133514404 + ], + "all": [ + 0.4749999940395355, + 0.4429999887943268, + 0.5059999823570251 + ], + "mrr": [ + 0.8554499745368958, + 0.8355995416641235, + 0.8741008043289185 + ], + "n_ret": 5.0, + "d_all": [ + 0.003000000026077032, + -0.02500000037252903, + 0.028999999165534973, + 0.8833116688331167 + ], + "d_recall": [ + 0.0010000000474974513, + -0.013000000268220901, + 0.014000000432133675, + 0.9448055194480552 + ], + "d_mrr": [ + -0.0004166666476521641, + -0.0014666665811091661, + 0.0006170833366923034, + 0.532046795320468 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: knn]", + "key": "lib_default:knn", + "variant": "knn", + "recall": [ + 0.6669999957084656, + 0.6480000019073486, + 0.6850000023841858 + ], + "all": [ + 0.40400001406669617, + 0.37299999594688416, + 0.4350000023841858 + ], + "mrr": [ + 0.8545833826065063, + 0.8344995975494385, + 0.8735508918762207 + ], + "n_ret": 5.0, + "d_all": [ + -0.06800000369548798, + -0.09399999678134918, + -0.0430000014603138, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.035999998450279236, + -0.04899999871850014, + -0.023499999195337296, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0012833332875743508, + -0.002500416710972786, + -0.0003000000142492354, + 0.03189681031896811 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: ctrl_mention]", + "key": "lib_default:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.6825000047683716, + 0.6639999747276306, + 0.7014999985694885 + ], + "all": [ + 0.43299999833106995, + 0.40299999713897705, + 0.46502500772476196 + ], + "mrr": [ + 0.8550333380699158, + 0.8350154161453247, + 0.8738675117492676 + ], + "n_ret": 5.0, + "d_all": [ + -0.039000000804662704, + -0.052000001072883606, + -0.027000000700354576, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.020500000566244125, + -0.027000000700354576, + -0.014499999582767487, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0008333332953043282, + -0.001750416704453528, + -0.00013333333481568843, + 0.1255874412558744 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + } + ], + "10": [ + { + "label": "Vector top-k (plain)", + "key": "vector", + "variant": null, + "recall": [ + 0.7724999785423279, + 0.7549999952316284, + 0.7895125150680542 + ], + "all": [ + 0.5820000171661377, + 0.5519750118255615, + 0.6129999756813049 + ], + "mrr": [ + 0.8597674369812012, + 0.8405397534370422, + 0.8778821229934692 + ], + "n_ret": 10.0 + }, + { + "label": "BM25", + "key": "bm25", + "variant": null, + "recall": [ + 0.8414999842643738, + 0.8264999985694885, + 0.8560000061988831 + ], + "all": [ + 0.6909999847412109, + 0.6629999876022339, + 0.718999981880188 + ], + "mrr": [ + 0.8509678244590759, + 0.8340070843696594, + 0.8668574690818787 + ], + "n_ret": 10.0, + "d_all": [ + 0.10899999737739563, + 0.07697500288486481, + 0.1420000046491623, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.0689999982714653, + 0.051500000059604645, + 0.08701249957084656, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.008799602277576923, + -0.029796507209539413, + 0.01244074385613203, + 0.41475852414758524 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "Hybrid vector+BM25 (RRF)", + "key": "rrf", + "variant": null, + "recall": [ + 0.859499990940094, + 0.8445000052452087, + 0.8734999895095825 + ], + "all": [ + 0.7250000238418579, + 0.6970000267028809, + 0.7510249614715576 + ], + "mrr": [ + 0.8708540201187134, + 0.8549973368644714, + 0.8870794177055359 + ], + "n_ret": 10.0, + "d_all": [ + 0.14300000667572021, + 0.11800000071525574, + 0.16699999570846558, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.08699999749660492, + 0.07398749887943268, + 0.10000000149011612, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.011086507700383663, + -0.0026989784091711044, + 0.02502189204096794, + 0.1060893910608939 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "Vector top-30 + cross-encoder", + "key": "rerank", + "variant": null, + "recall": [ + 0.8349999785423279, + 0.8199999928474426, + 0.8495000004768372 + ], + "all": [ + 0.6840000152587891, + 0.6570000052452087, + 0.7110000252723694 + ], + "mrr": [ + 0.9390119314193726, + 0.9260640144348145, + 0.9511301517486572 + ], + "n_ret": 10.0, + "d_all": [ + 0.10199999809265137, + 0.08297500014305115, + 0.12200000137090683, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.0625, + 0.05249999836087227, + 0.07349999994039536, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.07924444228410721, + 0.062108419835567474, + 0.09647171199321747, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph chosen on dev: mention]", + "key": "lib_mh@best", + "variant": "mention", + "recall": [ + 0.8865000009536743, + 0.8700000047683716, + 0.9014999866485596 + ], + "all": [ + 0.8149999976158142, + 0.790975034236908, + 0.8379999995231628 + ], + "mrr": [ + 0.8594388365745544, + 0.8402285575866699, + 0.8775531053543091 + ], + "n_ret": 9.404000282287598, + "d_all": [ + 0.2329999953508377, + 0.20399999618530273, + 0.2639999985694885, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.11400000005960464, + 0.0984874963760376, + 0.12999999523162842, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.00032857139012776315, + -0.001620277762413025, + 0.0009190674172714353, + 0.6085391460853915 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention]", + "key": "ppr@best", + "variant": "mention", + "recall": [ + 0.887499988079071, + 0.8715000152587891, + 0.9024999737739563 + ], + "all": [ + 0.8140000104904175, + 0.7900000214576721, + 0.8370000123977661 + ], + "mrr": [ + 0.8453448414802551, + 0.8275462985038757, + 0.8634601831436157 + ], + "n_ret": 10.0, + "d_all": [ + 0.23199999332427979, + 0.20499999821186066, + 0.26100000739097595, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.11500000208616257, + 0.10100000351667404, + 0.12999999523162842, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.01442261878401041, + -0.027753878384828568, + -0.0005916164955124259, + 0.035896410358964105 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph chosen on dev: mention]", + "key": "lib_default@best", + "variant": "mention", + "recall": [ + 0.8579999804496765, + 0.8399999737739563, + 0.8744999766349792 + ], + "all": [ + 0.7730000019073486, + 0.746999979019165, + 0.7979999780654907 + ], + "mrr": [ + 0.8571146726608276, + 0.837664008140564, + 0.875609278678894 + ], + "n_ret": 7.916999816894531, + "d_all": [ + 0.19099999964237213, + 0.16200000047683716, + 0.22300000488758087, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.08550000190734863, + 0.0689999982714653, + 0.10151249915361404, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0026527775917202234, + -0.0041321925818920135, + -0.0011384623358026147, + 0.0004999500049995 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: mention]", + "key": "lib_mh:mention", + "variant": "mention", + "recall": [ + 0.8865000009536743, + 0.8700000047683716, + 0.9014999866485596 + ], + "all": [ + 0.8149999976158142, + 0.790975034236908, + 0.8379999995231628 + ], + "mrr": [ + 0.8594388365745544, + 0.8402285575866699, + 0.8775531053543091 + ], + "n_ret": 9.404000282287598, + "d_all": [ + 0.2329999953508377, + 0.20399999618530273, + 0.2639999985694885, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.11400000005960464, + 0.0984874963760376, + 0.12999999523162842, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.00032857139012776315, + -0.001620277762413025, + 0.0009190674172714353, + 0.6085391460853915 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: entity]", + "key": "lib_mh:entity", + "variant": "entity", + "recall": [ + 0.8450000286102295, + 0.828499972820282, + 0.8610000014305115 + ], + "all": [ + 0.7179999947547913, + 0.6899999976158142, + 0.7450000047683716 + ], + "mrr": [ + 0.8609659075737, + 0.8419222831726074, + 0.8788793087005615 + ], + "n_ret": 10.0, + "d_all": [ + 0.13600000739097595, + 0.1120000034570694, + 0.1599999964237213, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.07249999791383743, + 0.05999999865889549, + 0.08500000089406967, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.0011984127340838313, + -0.0006258926587179303, + 0.003170635085552931, + 0.23557644235576441 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: knn]", + "key": "lib_mh:knn", + "variant": "knn", + "recall": [ + 0.7885000109672546, + 0.7710000276565552, + 0.8054999709129333 + ], + "all": [ + 0.621999979019165, + 0.5920000076293945, + 0.652999997138977 + ], + "mrr": [ + 0.8588646054267883, + 0.8394932150840759, + 0.8770387768745422 + ], + "n_ret": 10.0, + "d_all": [ + 0.03999999910593033, + 0.01899999938905239, + 0.06199999898672104, + 0.00039996000399960006 + ], + "d_recall": [ + 0.01600000075995922, + 0.004999999888241291, + 0.027499999850988388, + 0.009299070092990702 + ], + "d_mrr": [ + -0.0009027777705341578, + -0.0019234424689784646, + 7.820440805517137e-05, + 0.0771922807719228 + ], + "d_all_holm": 0.001999800019998, + "d_recall_holm": 0.04649535046495351 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: ctrl_mention]", + "key": "lib_mh:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.7724999785423279, + 0.7549999952316284, + 0.7895125150680542 + ], + "all": [ + 0.5820000171661377, + 0.5519750118255615, + 0.6129999756813049 + ], + "mrr": [ + 0.8597674369812012, + 0.8405397534370422, + 0.8778821229934692 + ], + "n_ret": 10.0, + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_mrr": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: mention]", + "key": "ppr:mention", + "variant": "mention", + "recall": [ + 0.887499988079071, + 0.8715000152587891, + 0.9024999737739563 + ], + "all": [ + 0.8140000104904175, + 0.7900000214576721, + 0.8370000123977661 + ], + "mrr": [ + 0.8453448414802551, + 0.8275462985038757, + 0.8634601831436157 + ], + "n_ret": 10.0, + "d_all": [ + 0.23199999332427979, + 0.20499999821186066, + 0.26100000739097595, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.11500000208616257, + 0.10100000351667404, + 0.12999999523162842, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.01442261878401041, + -0.027753878384828568, + -0.0005916164955124259, + 0.035896410358964105 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: entity]", + "key": "ppr:entity", + "variant": "entity", + "recall": [ + 0.8190000057220459, + 0.8019999861717224, + 0.8349999785423279 + ], + "all": [ + 0.6740000247955322, + 0.6459750533103943, + 0.7020000219345093 + ], + "mrr": [ + 0.837376594543457, + 0.8186529874801636, + 0.8564754724502563 + ], + "n_ret": 10.0, + "d_all": [ + 0.09200000017881393, + 0.07100000232458115, + 0.11502499878406525, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.04650000110268593, + 0.03500000014901161, + 0.05849999934434891, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.02239087224006653, + -0.03412008658051491, + -0.012033670209348202, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: knn]", + "key": "ppr:knn", + "variant": "knn", + "recall": [ + 0.7904999852180481, + 0.7735000252723694, + 0.8080000281333923 + ], + "all": [ + 0.621999979019165, + 0.5929999947547913, + 0.6520000100135803 + ], + "mrr": [ + 0.8254500031471252, + 0.8055899739265442, + 0.8436196446418762 + ], + "n_ret": 10.0, + "d_all": [ + 0.03999999910593033, + 0.020999999716877937, + 0.05900000035762787, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.017999999225139618, + 0.00800000037997961, + 0.028012501075863838, + 0.0012998700129987 + ], + "d_mrr": [ + -0.03431745991110802, + -0.04446464404463768, + -0.02446752041578293, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0077992200779922 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: ctrl_mention]", + "key": "ppr:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.7730000019073486, + 0.7555000185966492, + 0.790012538433075 + ], + "all": [ + 0.5830000042915344, + 0.5539750456809998, + 0.6129999756813049 + ], + "mrr": [ + 0.8595544099807739, + 0.8403394818305969, + 0.8779441714286804 + ], + "n_ret": 10.0, + "d_all": [ + 0.0010000000474974513, + -0.004999999888241291, + 0.007000000216066837, + 1.0 + ], + "d_recall": [ + 0.0005000000237487257, + -0.0024999999441206455, + 0.0035000001080334187, + 1.0 + ], + "d_mrr": [ + -0.0002130951324943453, + -0.0026073909830302, + 0.002335763769224286, + 0.8783121687831217 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: mention]", + "key": "lib_default:mention", + "variant": "mention", + "recall": [ + 0.8579999804496765, + 0.8399999737739563, + 0.8744999766349792 + ], + "all": [ + 0.7730000019073486, + 0.746999979019165, + 0.7979999780654907 + ], + "mrr": [ + 0.8571146726608276, + 0.837664008140564, + 0.875609278678894 + ], + "n_ret": 7.916999816894531, + "d_all": [ + 0.19099999964237213, + 0.16200000047683716, + 0.22300000488758087, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.08550000190734863, + 0.0689999982714653, + 0.10151249915361404, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0026527775917202234, + -0.0041321925818920135, + -0.0011384623358026147, + 0.0004999500049995 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: entity]", + "key": "lib_default:entity", + "variant": "entity", + "recall": [ + 0.7894999980926514, + 0.7714999914169312, + 0.8075000047683716 + ], + "all": [ + 0.628000020980835, + 0.5979999899864197, + 0.6580249667167664 + ], + "mrr": [ + 0.8580031991004944, + 0.8387415409088135, + 0.8763478398323059 + ], + "n_ret": 9.972999572753906, + "d_all": [ + 0.04600000008940697, + 0.01600000075995922, + 0.07699999958276749, + 0.004299570042995701 + ], + "d_recall": [ + 0.017000000923871994, + 0.0010000000474974513, + 0.033012498170137405, + 0.0458954104589541 + ], + "d_mrr": [ + -0.0017642858438193798, + -0.0030303376261144876, + -0.00043134920997545123, + 0.0084991500849915 + ], + "d_all_holm": 0.017198280171982803, + "d_recall_holm": 0.1835816418358164 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: knn]", + "key": "lib_default:knn", + "variant": "knn", + "recall": [ + 0.7630000114440918, + 0.7444999814033508, + 0.7805125117301941 + ], + "all": [ + 0.5770000219345093, + 0.5460000038146973, + 0.6069999933242798 + ], + "mrr": [ + 0.8572210073471069, + 0.8374678492546082, + 0.8757367730140686 + ], + "n_ret": 10.0, + "d_all": [ + -0.004999999888241291, + -0.03200000151991844, + 0.023000000044703484, + 0.7721227877212279 + ], + "d_recall": [ + -0.009499999694526196, + -0.023499999195337296, + 0.004999999888241291, + 0.2243775622437756 + ], + "d_mrr": [ + -0.0025464287027716637, + -0.003963491879403591, + -0.001207896857522428, + 9.999000099990002e-05 + ], + "d_all_holm": 1.0, + "d_recall_holm": 0.6731326867313269 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: ctrl_mention]", + "key": "lib_default:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.7024999856948853, + 0.6840000152587891, + 0.721012532711029 + ], + "all": [ + 0.47099998593330383, + 0.4410000145435333, + 0.5019999742507935 + ], + "mrr": [ + 0.8553667068481445, + 0.8354166746139526, + 0.8740337491035461 + ], + "n_ret": 9.097000122070312, + "d_all": [ + -0.11100000143051147, + -0.13099999725818634, + -0.09200000017881393, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.07000000029802322, + -0.08250000327825546, + -0.05900000035762787, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0044007934629917145, + -0.0060623218305408955, + -0.0028224701527506113, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + } + ], + "20": [ + { + "label": "Vector top-k (plain)", + "key": "vector", + "variant": null, + "recall": [ + 0.8224999904632568, + 0.8075000047683716, + 0.8385000228881836 + ], + "all": [ + 0.6620000004768372, + 0.6340000033378601, + 0.6909999847412109 + ], + "mrr": [ + 0.8612282872200012, + 0.8423323035240173, + 0.8791601657867432 + ], + "n_ret": 20.0 + }, + { + "label": "BM25", + "key": "bm25", + "variant": null, + "recall": [ + 0.9100000262260437, + 0.8980000019073486, + 0.921500027179718 + ], + "all": [ + 0.8220000267028809, + 0.7979999780654907, + 0.8440249562263489 + ], + "mrr": [ + 0.8514444828033447, + 0.8346499800682068, + 0.8671927452087402 + ], + "n_ret": 20.0, + "d_all": [ + 0.1599999964237213, + 0.13099999725818634, + 0.19200000166893005, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.08749999850988388, + 0.07150000333786011, + 0.10400000214576721, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.009783754125237465, + -0.030627422034740448, + 0.011115241795778275, + 0.36276372362763726 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "Hybrid vector+BM25 (RRF)", + "key": "rrf", + "variant": null, + "recall": [ + 0.9175000190734863, + 0.906000018119812, + 0.9290000200271606 + ], + "all": [ + 0.8379999995231628, + 0.8159999847412109, + 0.8600000143051147 + ], + "mrr": [ + 0.8710806369781494, + 0.855238139629364, + 0.8873094916343689 + ], + "n_ret": 20.0, + "d_all": [ + 0.17599999904632568, + 0.1509999930858612, + 0.20100000500679016, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.0949999988079071, + 0.0820000022649765, + 0.1080000028014183, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.00985235720872879, + -0.0038166851736605167, + 0.023490553721785545, + 0.146985301469853 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "Vector top-30 + cross-encoder", + "key": "rerank", + "variant": null, + "recall": [ + 0.8450000286102295, + 0.8305000066757202, + 0.8595125079154968 + ], + "all": [ + 0.703000009059906, + 0.6769750118255615, + 0.7300000190734863 + ], + "mrr": [ + 0.9391027688980103, + 0.9261455535888672, + 0.9512282013893127 + ], + "n_ret": 20.0, + "d_all": [ + 0.04100000113248825, + 0.028999999165534973, + 0.05400000140070915, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.02250000089406967, + 0.016499999910593033, + 0.028999999165534973, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.07787454128265381, + 0.06086345016956329, + 0.0947604700922966, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph chosen on dev: mention]", + "key": "lib_mh@best", + "variant": "mention", + "recall": [ + 0.9254999756813049, + 0.9125000238418579, + 0.9380000233650208 + ], + "all": [ + 0.8730000257492065, + 0.8519999980926514, + 0.8930249810218811 + ], + "mrr": [ + 0.858988344669342, + 0.8396490216255188, + 0.8772426843643188 + ], + "n_ret": 19.836000442504883, + "d_all": [ + 0.210999995470047, + 0.18400000035762787, + 0.2370000034570694, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.10300000011920929, + 0.08900000154972076, + 0.11649999767541885, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0022399218287318945, + -0.0042543914169073105, + -0.0003451411612331867, + 0.0197980201979802 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention]", + "key": "ppr@best", + "variant": "mention", + "recall": [ + 0.9269999861717224, + 0.9144999980926514, + 0.9390000104904175 + ], + "all": [ + 0.8730000257492065, + 0.8529999852180481, + 0.8930000066757202 + ], + "mrr": [ + 0.8165765404701233, + 0.7979017496109009, + 0.8356704711914062 + ], + "n_ret": 20.0, + "d_all": [ + 0.210999995470047, + 0.1860000044107437, + 0.23600000143051147, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.10450000315904617, + 0.09149999916553497, + 0.11749999970197678, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.04465177282691002, + -0.060815636068582535, + -0.027720607817173004, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph chosen on dev: mention]", + "key": "lib_default@best", + "variant": "mention", + "recall": [ + 0.859000027179718, + 0.8410000205039978, + 0.8755125403404236 + ], + "all": [ + 0.7749999761581421, + 0.7490000128746033, + 0.800000011920929 + ], + "mrr": [ + 0.8571146726608276, + 0.837664008140564, + 0.875609278678894 + ], + "n_ret": 9.375, + "d_all": [ + 0.11299999803304672, + 0.08299999684095383, + 0.14499999582767487, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.0364999994635582, + 0.01848750002682209, + 0.05400000140070915, + 0.00019998000199980003 + ], + "d_mrr": [ + -0.004113584291189909, + -0.00575342308729887, + -0.0026255652774125338, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: mention]", + "key": "lib_mh:mention", + "variant": "mention", + "recall": [ + 0.9254999756813049, + 0.9125000238418579, + 0.9380000233650208 + ], + "all": [ + 0.8730000257492065, + 0.8519999980926514, + 0.8930249810218811 + ], + "mrr": [ + 0.858988344669342, + 0.8396490216255188, + 0.8772426843643188 + ], + "n_ret": 19.836000442504883, + "d_all": [ + 0.210999995470047, + 0.18400000035762787, + 0.2370000034570694, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.10300000011920929, + 0.08900000154972076, + 0.11649999767541885, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0022399218287318945, + -0.0042543914169073105, + -0.0003451411612331867, + 0.0197980201979802 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: entity]", + "key": "lib_mh:entity", + "variant": "entity", + "recall": [ + 0.8870000243186951, + 0.8730000257492065, + 0.9014999866485596 + ], + "all": [ + 0.7919999957084656, + 0.7670000195503235, + 0.8180000185966492 + ], + "mrr": [ + 0.861248254776001, + 0.8422926068305969, + 0.8791594505310059 + ], + "n_ret": 19.961000442504883, + "d_all": [ + 0.12999999523162842, + 0.10499999672174454, + 0.1550000011920929, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.06449999660253525, + 0.051500000059604645, + 0.07750000059604645, + 9.999000099990002e-05 + ], + "d_mrr": [ + 1.9967734260717407e-05, + -0.0005363375530578196, + 0.0005883976700715721, + 0.9472052794720528 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: knn]", + "key": "lib_mh:knn", + "variant": "knn", + "recall": [ + 0.8519999980926514, + 0.8364999890327454, + 0.8669999837875366 + ], + "all": [ + 0.7250000238418579, + 0.6980000138282776, + 0.7509999871253967 + ], + "mrr": [ + 0.8604547381401062, + 0.8414711356163025, + 0.878466010093689 + ], + "n_ret": 20.0, + "d_all": [ + 0.06300000101327896, + 0.041999999433755875, + 0.08399999886751175, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.029500000178813934, + 0.01850000023841858, + 0.04050000011920929, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0007735782419331372, + -0.0019331652438268065, + 0.00031544023659080267, + 0.20377962203779623 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: ctrl_mention]", + "key": "lib_mh:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.8224999904632568, + 0.8075000047683716, + 0.8385000228881836 + ], + "all": [ + 0.6620000004768372, + 0.6340000033378601, + 0.6909999847412109 + ], + "mrr": [ + 0.8612282872200012, + 0.8423323035240173, + 0.8791601657867432 + ], + "n_ret": 20.0, + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_mrr": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: mention]", + "key": "ppr:mention", + "variant": "mention", + "recall": [ + 0.9269999861717224, + 0.9144999980926514, + 0.9390000104904175 + ], + "all": [ + 0.8730000257492065, + 0.8529999852180481, + 0.8930000066757202 + ], + "mrr": [ + 0.8165765404701233, + 0.7979017496109009, + 0.8356704711914062 + ], + "n_ret": 20.0, + "d_all": [ + 0.210999995470047, + 0.1860000044107437, + 0.23600000143051147, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.10450000315904617, + 0.09149999916553497, + 0.11749999970197678, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.04465177282691002, + -0.060815636068582535, + -0.027720607817173004, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: entity]", + "key": "ppr:entity", + "variant": "entity", + "recall": [ + 0.8794999718666077, + 0.8650000095367432, + 0.8939999938011169 + ], + "all": [ + 0.7799999713897705, + 0.7540000081062317, + 0.8059999942779541 + ], + "mrr": [ + 0.8142668604850769, + 0.7937544584274292, + 0.833128035068512 + ], + "n_ret": 20.0, + "d_all": [ + 0.11800000071525574, + 0.09700000286102295, + 0.14100000262260437, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.05700000002980232, + 0.04600000008940697, + 0.0689999982714653, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.04696144908666611, + -0.06086188554763794, + -0.033907100558280945, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: knn]", + "key": "ppr:knn", + "variant": "knn", + "recall": [ + 0.8420000076293945, + 0.8259999752044678, + 0.8575000166893005 + ], + "all": [ + 0.7120000123977661, + 0.6850000023841858, + 0.7390000224113464 + ], + "mrr": [ + 0.8019974231719971, + 0.7821558713912964, + 0.8206881880760193 + ], + "n_ret": 20.0, + "d_all": [ + 0.05000000074505806, + 0.028999999165534973, + 0.07000000029802322, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.019500000402331352, + 0.008500000461935997, + 0.030500000342726707, + 0.0013998600139986002 + ], + "d_mrr": [ + -0.059230830520391464, + -0.07200013846158981, + -0.04689715430140495, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.006999300069993001 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: ctrl_mention]", + "key": "ppr:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.8220000267028809, + 0.8065000176429749, + 0.8375124931335449 + ], + "all": [ + 0.6610000133514404, + 0.6330000162124634, + 0.6899999976158142 + ], + "mrr": [ + 0.8569718599319458, + 0.8385047912597656, + 0.8748267889022827 + ], + "n_ret": 20.0, + "d_all": [ + -0.0010000000474974513, + -0.003000000026077032, + 0.0, + 1.0 + ], + "d_recall": [ + -0.0005000000237487257, + -0.001500000013038516, + 0.0, + 1.0 + ], + "d_mrr": [ + -0.0042564100585877895, + -0.008851250633597374, + 0.0004840063047595322, + 0.0655934406559344 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: mention]", + "key": "lib_default:mention", + "variant": "mention", + "recall": [ + 0.859000027179718, + 0.8410000205039978, + 0.8755125403404236 + ], + "all": [ + 0.7749999761581421, + 0.7490000128746033, + 0.800000011920929 + ], + "mrr": [ + 0.8571146726608276, + 0.837664008140564, + 0.875609278678894 + ], + "n_ret": 9.375, + "d_all": [ + 0.11299999803304672, + 0.08299999684095383, + 0.14499999582767487, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.0364999994635582, + 0.01848750002682209, + 0.05400000140070915, + 0.00019998000199980003 + ], + "d_mrr": [ + -0.004113584291189909, + -0.00575342308729887, + -0.0026255652774125338, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: entity]", + "key": "lib_default:entity", + "variant": "entity", + "recall": [ + 0.8389999866485596, + 0.8215000033378601, + 0.8554999828338623 + ], + "all": [ + 0.7170000076293945, + 0.6890000104904175, + 0.7440000176429749 + ], + "mrr": [ + 0.8587110638618469, + 0.8395909667015076, + 0.8769223690032959 + ], + "n_ret": 19.829999923706055, + "d_all": [ + 0.054999999701976776, + 0.024000000208616257, + 0.0860000029206276, + 0.0006999300069993001 + ], + "d_recall": [ + 0.016499999910593033, + 0.0, + 0.032999999821186066, + 0.05649435056494351 + ], + "d_mrr": [ + -0.00251725479029119, + -0.00380460312590003, + -0.0012195897288620472, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0027997200279972004, + "d_recall_holm": 0.22597740225977403 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: knn]", + "key": "lib_default:knn", + "variant": "knn", + "recall": [ + 0.8109999895095825, + 0.793987512588501, + 0.828000009059906 + ], + "all": [ + 0.6610000133514404, + 0.6309999823570251, + 0.6899999976158142 + ], + "mrr": [ + 0.858066737651825, + 0.8384552597999573, + 0.8763984441757202 + ], + "n_ret": 19.975000381469727, + "d_all": [ + -0.0010000000474974513, + -0.028999999165534973, + 0.027000000700354576, + 1.0 + ], + "d_recall": [ + -0.011500000022351742, + -0.026499999687075615, + 0.0035000001080334187, + 0.14878512148785122 + ], + "d_mrr": [ + -0.0031615407206118107, + -0.0045406450517475605, + -0.0018780347891151905, + 9.999000099990002e-05 + ], + "d_all_holm": 1.0, + "d_recall_holm": 0.44635536446355367 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: ctrl_mention]", + "key": "lib_default:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.703000009059906, + 0.684499979019165, + 0.722000002861023 + ], + "all": [ + 0.47200000286102295, + 0.4410000145435333, + 0.503000020980835 + ], + "mrr": [ + 0.8553667068481445, + 0.8354166746139526, + 0.8740337491035461 + ], + "n_ret": 11.569999694824219, + "d_all": [ + -0.1899999976158142, + -0.21400000154972076, + -0.16599999368190765, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.11949999630451202, + -0.1340000033378601, + -0.1054999977350235, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.005861600395292044, + -0.007653497625142336, + -0.004160683136433363, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0017998200179982003, + "d_recall_holm": 0.0017998200179982003 + } + ] + }, + "subgraph": { + "5": [ + { + "variant": "mention", + "connect_max_hops": 2, + "size": 5.007999897003174, + "recall": [ + 0.8234999775886536, + 0.8050000071525574, + 0.8414999842643738 + ], + "all": [ + 0.7120000123977661, + 0.6830000281333923, + 0.7380249500274658 + ], + "vec_matched_recall": [ + 0.7035, + 0.685, + 0.7225 + ], + "vec_matched_all": [ + 0.473, + 0.443, + 0.504 + ], + "d_all": [ + 0.23899999260902405, + 0.20999999344348907, + 0.26899999380111694, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.11999999731779099, + 0.1054999977350235, + 0.1354999989271164, + 9.999000099990002e-05 + ] + }, + { + "variant": "mention", + "connect_max_hops": 3, + "size": 5.01800012588501, + "recall": [ + 0.8234999775886536, + 0.8050000071525574, + 0.8414999842643738 + ], + "all": [ + 0.7120000123977661, + 0.6830000281333923, + 0.7380249500274658 + ], + "vec_matched_recall": [ + 0.7035, + 0.685, + 0.7225 + ], + "vec_matched_all": [ + 0.473, + 0.443, + 0.504 + ], + "d_all": [ + 0.23899999260902405, + 0.20999999344348907, + 0.26899999380111694, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.11999999731779099, + 0.1054999977350235, + 0.1354999989271164, + 9.999000099990002e-05 + ] + }, + { + "variant": "entity", + "connect_max_hops": 2, + "size": 5.118000030517578, + "recall": [ + 0.7515000104904175, + 0.7325000166893005, + 0.7705125212669373 + ], + "all": [ + 0.5669999718666077, + 0.5370000004768372, + 0.5979999899864197 + ], + "vec_matched_recall": [ + 0.705, + 0.6869875000000001, + 0.724 + ], + "vec_matched_all": [ + 0.476, + 0.444975, + 0.507 + ], + "d_all": [ + 0.09099999815225601, + 0.06599999964237213, + 0.11599999666213989, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.04650000110268593, + 0.03348750248551369, + 0.05951249971985817, + 9.999000099990002e-05 + ] + }, + { + "variant": "entity", + "connect_max_hops": 3, + "size": 5.406000137329102, + "recall": [ + 0.7524999976158142, + 0.7339875102043152, + 0.7720000147819519 + ], + "all": [ + 0.5690000057220459, + 0.5389999747276306, + 0.5989999771118164 + ], + "vec_matched_recall": [ + 0.7085, + 0.6905, + 0.7270125 + ], + "vec_matched_all": [ + 0.48, + 0.449975, + 0.511 + ], + "d_all": [ + 0.08900000154972076, + 0.06400000303983688, + 0.11500000208616257, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.04399999976158142, + 0.03099999949336052, + 0.057500001043081284, + 9.999000099990002e-05 + ] + }, + { + "variant": "knn", + "connect_max_hops": 2, + "size": 5.361999988555908, + "recall": [ + 0.7114999890327454, + 0.6934874653816223, + 0.7304999828338623 + ], + "all": [ + 0.4880000054836273, + 0.4580000042915344, + 0.5199999809265137 + ], + "vec_matched_recall": [ + 0.708, + 0.6895, + 0.7265 + ], + "vec_matched_all": [ + 0.481, + 0.450975, + 0.512 + ], + "d_all": [ + 0.007000000216066837, + -0.006000000052154064, + 0.019999999552965164, + 0.38296170382961703 + ], + "d_recall": [ + 0.0035000001080334187, + -0.0035000001080334187, + 0.010499999858438969, + 0.39386061393860616 + ] + }, + { + "variant": "knn", + "connect_max_hops": 3, + "size": 5.9770002365112305, + "recall": [ + 0.7149999737739563, + 0.6970000267028809, + 0.7340125441551208 + ], + "all": [ + 0.49300000071525574, + 0.46299999952316284, + 0.5239999890327454 + ], + "vec_matched_recall": [ + 0.718, + 0.7, + 0.7365 + ], + "vec_matched_all": [ + 0.494, + 0.463, + 0.525 + ], + "d_all": [ + -0.0010000000474974513, + -0.01600000075995922, + 0.014000000432133675, + 1.0 + ], + "d_recall": [ + -0.003000000026077032, + -0.010999999940395355, + 0.004999999888241291, + 0.5625437456254374 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 2, + "size": 5.0, + "recall": [ + 0.703000009059906, + 0.684499979019165, + 0.722000002861023 + ], + "all": [ + 0.47200000286102295, + 0.4410000145435333, + 0.503000020980835 + ], + "vec_matched_recall": [ + 0.703, + 0.6845, + 0.722 + ], + "vec_matched_all": [ + 0.472, + 0.441, + 0.503 + ], + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 3, + "size": 5.0, + "recall": [ + 0.703000009059906, + 0.684499979019165, + 0.722000002861023 + ], + "all": [ + 0.47200000286102295, + 0.4410000145435333, + 0.503000020980835 + ], + "vec_matched_recall": [ + 0.703, + 0.6845, + 0.722 + ], + "vec_matched_all": [ + 0.472, + 0.441, + 0.503 + ], + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ] + } + ], + "10": [ + { + "variant": "mention", + "connect_max_hops": 2, + "size": 9.454999923706055, + "recall": [ + 0.8880000114440918, + 0.871999979019165, + 0.902999997138977 + ], + "all": [ + 0.8169999718666077, + 0.7929999828338623, + 0.8399999737739563 + ], + "vec_matched_recall": [ + 0.7655, + 0.7475, + 0.783 + ], + "vec_matched_all": [ + 0.571, + 0.541, + 0.601 + ], + "d_all": [ + 0.2460000067949295, + 0.21797499060630798, + 0.2770000100135803, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.12250000238418579, + 0.10750000178813934, + 0.13850000500679016, + 9.999000099990002e-05 + ] + }, + { + "variant": "mention", + "connect_max_hops": 3, + "size": 9.512999534606934, + "recall": [ + 0.8884999752044678, + 0.871999979019165, + 0.902999997138977 + ], + "all": [ + 0.8180000185966492, + 0.7940000295639038, + 0.8410000205039978 + ], + "vec_matched_recall": [ + 0.766, + 0.7484875000000001, + 0.783 + ], + "vec_matched_all": [ + 0.571, + 0.541, + 0.601 + ], + "d_all": [ + 0.24699999392032623, + 0.21799999475479126, + 0.27799999713897705, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.12250000238418579, + 0.10750000178813934, + 0.13850000500679016, + 9.999000099990002e-05 + ] + }, + { + "variant": "entity", + "connect_max_hops": 2, + "size": 10.300999641418457, + "recall": [ + 0.847000002861023, + 0.8304874897003174, + 0.8629999756813049 + ], + "all": [ + 0.7210000157356262, + 0.6940000057220459, + 0.7480000257492065 + ], + "vec_matched_recall": [ + 0.775, + 0.758, + 0.792 + ], + "vec_matched_all": [ + 0.586, + 0.556, + 0.6160249999999998 + ], + "d_all": [ + 0.13500000536441803, + 0.11100000143051147, + 0.1599999964237213, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.07199999690055847, + 0.05950000137090683, + 0.08449999988079071, + 9.999000099990002e-05 + ] + }, + { + "variant": "entity", + "connect_max_hops": 3, + "size": 11.51200008392334, + "recall": [ + 0.8485000133514404, + 0.8324999809265137, + 0.8644999861717224 + ], + "all": [ + 0.7229999899864197, + 0.6959999799728394, + 0.7500249743461609 + ], + "vec_matched_recall": [ + 0.7845, + 0.7674875, + 0.801 + ], + "vec_matched_all": [ + 0.6, + 0.57, + 0.6300249999999998 + ], + "d_all": [ + 0.12300000339746475, + 0.09700000286102295, + 0.14900000393390656, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.06400000303983688, + 0.050987500697374344, + 0.07750000059604645, + 9.999000099990002e-05 + ] + }, + { + "variant": "knn", + "connect_max_hops": 2, + "size": 10.397000312805176, + "recall": [ + 0.7885000109672546, + 0.7710000276565552, + 0.8054999709129333 + ], + "all": [ + 0.621999979019165, + 0.5920000076293945, + 0.652999997138977 + ], + "vec_matched_recall": [ + 0.7755, + 0.7585, + 0.7925 + ], + "vec_matched_all": [ + 0.587, + 0.556975, + 0.618 + ], + "d_all": [ + 0.03500000014901161, + 0.014000000432133675, + 0.05700000002980232, + 0.001999800019998 + ], + "d_recall": [ + 0.013000000268220901, + 0.0020000000949949026, + 0.02449999935925007, + 0.0341965803419658 + ] + }, + { + "variant": "knn", + "connect_max_hops": 3, + "size": 11.154000282287598, + "recall": [ + 0.7889999747276306, + 0.7714999914169312, + 0.8059999942779541 + ], + "all": [ + 0.6230000257492065, + 0.5929999947547913, + 0.6530249714851379 + ], + "vec_matched_recall": [ + 0.782, + 0.765, + 0.7995 + ], + "vec_matched_all": [ + 0.597, + 0.567, + 0.628 + ], + "d_all": [ + 0.026000000536441803, + 0.004999999888241291, + 0.04899999871850014, + 0.027597240275972403 + ], + "d_recall": [ + 0.007000000216066837, + -0.004000000189989805, + 0.01899999938905239, + 0.28477152284771523 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 2, + "size": 10.003000259399414, + "recall": [ + 0.7724999785423279, + 0.7549999952316284, + 0.7895125150680542 + ], + "all": [ + 0.5820000171661377, + 0.5519750118255615, + 0.6129999756813049 + ], + "vec_matched_recall": [ + 0.7725, + 0.755, + 0.7895125 + ], + "vec_matched_all": [ + 0.582, + 0.551975, + 0.613 + ], + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 3, + "size": 10.005000114440918, + "recall": [ + 0.7724999785423279, + 0.7549999952316284, + 0.7895125150680542 + ], + "all": [ + 0.5820000171661377, + 0.5519750118255615, + 0.6129999756813049 + ], + "vec_matched_recall": [ + 0.7725, + 0.755, + 0.7895125 + ], + "vec_matched_all": [ + 0.582, + 0.551975, + 0.613 + ], + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ] + } + ], + "20": [ + { + "variant": "mention", + "connect_max_hops": 2, + "size": 20.000999450683594, + "recall": [ + 0.9254999756813049, + 0.9125000238418579, + 0.9380000233650208 + ], + "all": [ + 0.8730000257492065, + 0.8519999980926514, + 0.8930249810218811 + ], + "vec_matched_recall": [ + 0.8235, + 0.808, + 0.8390124999999999 + ], + "vec_matched_all": [ + 0.663, + 0.635, + 0.692 + ], + "d_all": [ + 0.20999999344348907, + 0.18299999833106995, + 0.23600000143051147, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.10199999809265137, + 0.08799999952316284, + 0.11550000309944153, + 9.999000099990002e-05 + ] + }, + { + "variant": "mention", + "connect_max_hops": 3, + "size": 20.37299919128418, + "recall": [ + 0.9265000224113464, + 0.9139999747276306, + 0.9385125041007996 + ], + "all": [ + 0.8740000128746033, + 0.8529999852180481, + 0.8939999938011169 + ], + "vec_matched_recall": [ + 0.8245, + 0.809, + 0.84 + ], + "vec_matched_all": [ + 0.665, + 0.636, + 0.694 + ], + "d_all": [ + 0.20900000631809235, + 0.18297499418258667, + 0.23499999940395355, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.10199999809265137, + 0.08799999952316284, + 0.11551250517368317, + 9.999000099990002e-05 + ] + }, + { + "variant": "entity", + "connect_max_hops": 2, + "size": 20.731000900268555, + "recall": [ + 0.8880000114440918, + 0.8740000128746033, + 0.9020125269889832 + ], + "all": [ + 0.7940000295639038, + 0.7689999938011169, + 0.8199999928474426 + ], + "vec_matched_recall": [ + 0.825, + 0.8099875000000001, + 0.8405 + ], + "vec_matched_all": [ + 0.666, + 0.638, + 0.695 + ], + "d_all": [ + 0.12800000607967377, + 0.10300000011920929, + 0.15302500128746033, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.06300000101327896, + 0.05000000074505806, + 0.07649999856948853, + 9.999000099990002e-05 + ] + }, + { + "variant": "entity", + "connect_max_hops": 3, + "size": 23.677000045776367, + "recall": [ + 0.8880000114440918, + 0.8740000128746033, + 0.9020125269889832 + ], + "all": [ + 0.7940000295639038, + 0.7689999938011169, + 0.8199999928474426 + ], + "vec_matched_recall": [ + 0.832, + 0.817, + 0.8470125 + ], + "vec_matched_all": [ + 0.68, + 0.652, + 0.709 + ], + "d_all": [ + 0.11400000005960464, + 0.08799999952316284, + 0.14000000059604645, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.0560000017285347, + 0.042500000447034836, + 0.0695125013589859, + 9.999000099990002e-05 + ] + }, + { + "variant": "knn", + "connect_max_hops": 2, + "size": 21.329999923706055, + "recall": [ + 0.8539999723434448, + 0.8389999866485596, + 0.8690000176429749 + ], + "all": [ + 0.7289999723434448, + 0.7029750347137451, + 0.7549999952316284 + ], + "vec_matched_recall": [ + 0.8265, + 0.8115, + 0.842 + ], + "vec_matched_all": [ + 0.669, + 0.64, + 0.698 + ], + "d_all": [ + 0.05999999865889549, + 0.039000000804662704, + 0.08100000023841858, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.027499999850988388, + 0.016499999910593033, + 0.03849999979138374, + 9.999000099990002e-05 + ] + }, + { + "variant": "knn", + "connect_max_hops": 3, + "size": 23.3799991607666, + "recall": [ + 0.8550000190734863, + 0.8399999737739563, + 0.8695124983787537 + ], + "all": [ + 0.7300000190734863, + 0.7039999961853027, + 0.7550249695777893 + ], + "vec_matched_recall": [ + 0.8325, + 0.817, + 0.8485 + ], + "vec_matched_all": [ + 0.681, + 0.653, + 0.71 + ], + "d_all": [ + 0.04899999871850014, + 0.02797500044107437, + 0.07000000029802322, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.02250000089406967, + 0.011500000022351742, + 0.03350000083446503, + 0.00019998000199980003 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 2, + "size": 20.054000854492188, + "recall": [ + 0.8224999904632568, + 0.8075000047683716, + 0.8385000228881836 + ], + "all": [ + 0.6620000004768372, + 0.6340000033378601, + 0.6909999847412109 + ], + "vec_matched_recall": [ + 0.8225, + 0.8075, + 0.8385 + ], + "vec_matched_all": [ + 0.662, + 0.634, + 0.691 + ], + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 3, + "size": 20.268999099731445, + "recall": [ + 0.8224999904632568, + 0.8075000047683716, + 0.8385000228881836 + ], + "all": [ + 0.6620000004768372, + 0.6340000033378601, + 0.6909999847412109 + ], + "vec_matched_recall": [ + 0.8235, + 0.8085, + 0.8395 + ], + "vec_matched_all": [ + 0.664, + 0.636, + 0.693 + ], + "d_all": [ + -0.0020000000949949026, + -0.004999999888241291, + 0.0, + 0.49425057494250574 + ], + "d_recall": [ + -0.0010000000474974513, + -0.0024999999441206455, + 0.0, + 0.49425057494250574 + ] + } + ] + }, + "lib_minus_ppr": { + "mention": { + "5": [ + 0.07500000298023224, + 0.04899999871850014, + 0.10000000149011612, + 9.999000099990002e-05 + ], + "10": [ + 0.0010000000474974513, + -0.014000000432133675, + 0.014999999664723873, + 1.0 + ], + "20": [ + 0.0, + -0.010999999940395355, + 0.010999999940395355, + 1.0 + ] + }, + "entity": { + "5": [ + 0.08900000154972076, + 0.06499999761581421, + 0.11299999803304672, + 9.999000099990002e-05 + ], + "10": [ + 0.04399999976158142, + 0.02500000037252903, + 0.06400000303983688, + 9.999000099990002e-05 + ], + "20": [ + 0.012000000104308128, + -0.003000000026077032, + 0.027000000700354576, + 0.1454854514548545 + ] + }, + "knn": { + "5": [ + 0.006000000052154064, + -0.006000000052154064, + 0.017999999225139618, + 0.39336066393360664 + ], + "10": [ + 0.0, + -0.014999999664723873, + 0.01600000075995922, + 1.0 + ], + "20": [ + 0.013000000268220901, + 0.0, + 0.027000000700354576, + 0.0688931106889311 + ] + } + }, + "breakdown": { + "type": { + "bridge": { + "n": 784, + "vector": 0.5127550959587097, + "rrf": 0.6607142686843872, + "rerank": 0.6288265585899353, + "lib_mh@best": 0.8252550959587097, + "ppr@best": 0.8163265585899353 + }, + "comparison": { + "n": 216, + "vector": 0.8333333134651184, + "rrf": 0.9583333134651184, + "rerank": 0.8842592835426331, + "lib_mh@best": 0.7777777910232544, + "ppr@best": 0.8055555820465088 + } + }, + "hops": { + "2": { + "n": 1000, + "vector": 0.5820000171661377, + "rrf": 0.7250000238418579, + "rerank": 0.6840000152587891, + "lib_mh@best": 0.8149999976158142, + "ppr@best": 0.8140000104904175 + } + } + } + }, + "bge": { + "n_test": 1000, + "corpus_size": 14570, + "lib_similarity_search_same_top20_set": "200/200", + "graphs": { + "mention": { + "n_edges_undirected_union": 11348, + "n_edges_by_family_directed_or_undirected": 12354, + "mean_degree": 1.5577213452299246, + "median_degree": 1.0, + "max_degree": 29, + "isolated_frac": 0.39958819492107067, + "gold_pairs_directly_linked": 0.6513333333333333, + "gold_pairs_linked_within_2": 0.6673333333333333, + "neighbours_of_gold_that_are_in_same_question_context": 0.5890391209006267 + }, + "knn": { + "n_edges_undirected_union": 54857, + "n_edges_by_family_directed_or_undirected": 54857, + "mean_degree": 7.530130404941661, + "median_degree": 6.0, + "max_degree": 118, + "isolated_frac": 0.0, + "gold_pairs_directly_linked": 0.5393333333333333, + "gold_pairs_linked_within_2": 0.6733333333333333, + "neighbours_of_gold_that_are_in_same_question_context": 0.24001444287618007 + }, + "ctrl_mention": { + "n_edges_undirected_union": 11348, + "n_edges_by_family_directed_or_undirected": 12354, + "mean_degree": 1.5577213452299246, + "median_degree": 1.0, + "max_degree": 29, + "isolated_frac": 0.39958819492107067, + "gold_pairs_directly_linked": 0.0, + "gold_pairs_linked_within_2": 0.0, + "neighbours_of_gold_that_are_in_same_question_context": 0.00026518164942985947 + } + }, + "selected": { + "mention": { + "lib": { + "5": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "20": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + } + }, + "ppr": { + "5": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "20": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + } + } + }, + "knn": { + "lib": { + "5": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + } + }, + "ppr": { + "5": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "20": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + } + } + }, + "ctrl_mention": { + "lib": { + "5": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + } + }, + "ppr": { + "5": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.85, + "directed_mention": false, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "20": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + } + } + } + }, + "tables": { + "5": [ + { + "label": "Vector top-k (plain)", + "key": "vector", + "variant": null, + "recall": [ + 0.8305000066757202, + 0.8144999742507935, + 0.8460000157356262 + ], + "all": [ + 0.6840000152587891, + 0.6549999713897705, + 0.712024986743927 + ], + "mrr": [ + 0.9240333437919617, + 0.9100645780563354, + 0.9378671050071716 + ], + "n_ret": 5.0 + }, + { + "label": "BM25", + "key": "bm25", + "variant": null, + "recall": [ + 0.722000002861023, + 0.7039999961853027, + 0.7390000224113464 + ], + "all": [ + 0.4869999885559082, + 0.4569999873638153, + 0.5170000195503235 + ], + "mrr": [ + 0.8460832834243774, + 0.8281295895576477, + 0.8622362613677979 + ], + "n_ret": 5.0, + "d_all": [ + -0.19699999690055847, + -0.23000000417232513, + -0.16300000250339508, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.10849999636411667, + -0.1274999976158142, + -0.08900000154972076, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.07794999331235886, + -0.09783750027418137, + -0.05921666696667671, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0014998500149985003, + "d_recall_holm": 0.0014998500149985003 + }, + { + "label": "Hybrid vector+BM25 (RRF)", + "key": "rrf", + "variant": null, + "recall": [ + 0.840499997138977, + 0.8255000114440918, + 0.8554999828338623 + ], + "all": [ + 0.6909999847412109, + 0.662975013256073, + 0.718999981880188 + ], + "mrr": [ + 0.9136333465576172, + 0.8998662829399109, + 0.9269499778747559 + ], + "n_ret": 5.0, + "d_all": [ + 0.007000000216066837, + -0.017000000923871994, + 0.03202500194311142, + 0.6208379162083791 + ], + "d_recall": [ + 0.009999999776482582, + -0.003000000026077032, + 0.023499999195337296, + 0.15808419158084192 + ], + "d_mrr": [ + -0.010400000028312206, + -0.022633330896496773, + 0.001850833767093718, + 0.1024897510248975 + ], + "d_all_holm": 1.0, + "d_recall_holm": 0.7904209579042096 + }, + { + "label": "Vector top-30 + cross-encoder", + "key": "rerank", + "variant": null, + "recall": [ + 0.8529999852180481, + 0.8379999995231628, + 0.8675000071525574 + ], + "all": [ + 0.7160000205039978, + 0.6880000233650208, + 0.7440000176429749 + ], + "mrr": [ + 0.9503333568572998, + 0.9387833476066589, + 0.9607695937156677 + ], + "n_ret": 5.0, + "d_all": [ + 0.03200000151991844, + 0.003000000026077032, + 0.05999999865889549, + 0.027697230276972302 + ], + "d_recall": [ + 0.02250000089406967, + 0.006500000134110451, + 0.03750000149011612, + 0.0038996100389961006 + ], + "d_mrr": [ + 0.02630000002682209, + 0.01243249885737896, + 0.03958375006914139, + 9.999000099990002e-05 + ], + "d_all_holm": 0.16618338166183383, + "d_recall_holm": 0.0263973602639736 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph chosen on dev: mention]", + "key": "lib_mh@best", + "variant": "mention", + "recall": [ + 0.8949999809265137, + 0.8805000185966492, + 0.9089999794960022 + ], + "all": [ + 0.8130000233650208, + 0.7879999876022339, + 0.8360000252723694 + ], + "mrr": [ + 0.9240666627883911, + 0.9099658131599426, + 0.9379696249961853 + ], + "n_ret": 5.0, + "d_all": [ + 0.1289999932050705, + 0.1080000028014183, + 0.15199999511241913, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.06449999660253525, + 0.05400000140070915, + 0.07599999755620956, + 9.999000099990002e-05 + ], + "d_mrr": [ + 3.333331551402807e-05, + -0.000400000048102811, + 0.0004999999655410647, + 1.0 + ], + "d_all_holm": 0.0014998500149985003, + "d_recall_holm": 0.0014998500149985003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention]", + "key": "ppr@best", + "variant": "mention", + "recall": [ + 0.8849999904632568, + 0.8700000047683716, + 0.8995000123977661 + ], + "all": [ + 0.7960000038146973, + 0.7710000276565552, + 0.8199999928474426 + ], + "mrr": [ + 0.8370000123977661, + 0.8191992044448853, + 0.8541174530982971 + ], + "n_ret": 5.0, + "d_all": [ + 0.1120000034570694, + 0.08699999749660492, + 0.13702499866485596, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.054499998688697815, + 0.041999999433755875, + 0.06750000268220901, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.08703332394361496, + -0.10248416662216187, + -0.07121624052524567, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0014998500149985003, + "d_recall_holm": 0.0014998500149985003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph chosen on dev: mention]", + "key": "lib_default@best", + "variant": "mention", + "recall": [ + 0.8554999828338623, + 0.840499997138977, + 0.8705000281333923 + ], + "all": [ + 0.734000027179718, + 0.7080000042915344, + 0.7609999775886536 + ], + "mrr": [ + 0.9240333437919617, + 0.9100645780563354, + 0.9378671050071716 + ], + "n_ret": 5.0, + "d_all": [ + 0.05000000074505806, + 0.035999998450279236, + 0.06499999761581421, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.02500000037252903, + 0.017999999225139618, + 0.032499998807907104, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_all_holm": 0.0014998500149985003, + "d_recall_holm": 0.0014998500149985003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: mention]", + "key": "lib_mh:mention", + "variant": "mention", + "recall": [ + 0.8949999809265137, + 0.8805000185966492, + 0.9089999794960022 + ], + "all": [ + 0.8130000233650208, + 0.7879999876022339, + 0.8360000252723694 + ], + "mrr": [ + 0.9240666627883911, + 0.9099658131599426, + 0.9379696249961853 + ], + "n_ret": 5.0, + "d_all": [ + 0.1289999932050705, + 0.1080000028014183, + 0.15199999511241913, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.06449999660253525, + 0.05400000140070915, + 0.07599999755620956, + 9.999000099990002e-05 + ], + "d_mrr": [ + 3.333331551402807e-05, + -0.000400000048102811, + 0.0004999999655410647, + 1.0 + ], + "d_all_holm": 0.0014998500149985003, + "d_recall_holm": 0.0014998500149985003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: knn]", + "key": "lib_mh:knn", + "variant": "knn", + "recall": [ + 0.8349999785423279, + 0.8190000057220459, + 0.8505125045776367 + ], + "all": [ + 0.6930000185966492, + 0.6649750471115112, + 0.722000002861023 + ], + "mrr": [ + 0.9240333437919617, + 0.9100645780563354, + 0.9378671050071716 + ], + "n_ret": 5.0, + "d_all": [ + 0.008999999612569809, + -0.010999999940395355, + 0.028025001287460327, + 0.42525747425257476 + ], + "d_recall": [ + 0.0044999998062849045, + -0.005499999970197678, + 0.014012500643730164, + 0.42525747425257476 + ], + "d_mrr": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: ctrl_mention]", + "key": "lib_mh:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.8259999752044678, + 0.809499979019165, + 0.8414999842643738 + ], + "all": [ + 0.675000011920929, + 0.6449999809265137, + 0.7039999961853027 + ], + "mrr": [ + 0.9240333437919617, + 0.9100645780563354, + 0.9378671050071716 + ], + "n_ret": 5.0, + "d_all": [ + -0.008999999612569809, + -0.014999999664723873, + -0.004000000189989805, + 0.0032996700329967 + ], + "d_recall": [ + -0.0044999998062849045, + -0.007499999832361937, + -0.0020000000949949026, + 0.0032996700329967 + ], + "d_mrr": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_all_holm": 0.0263973602639736, + "d_recall_holm": 0.0263973602639736 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: mention]", + "key": "ppr:mention", + "variant": "mention", + "recall": [ + 0.8849999904632568, + 0.8700000047683716, + 0.8995000123977661 + ], + "all": [ + 0.7960000038146973, + 0.7710000276565552, + 0.8199999928474426 + ], + "mrr": [ + 0.8370000123977661, + 0.8191992044448853, + 0.8541174530982971 + ], + "n_ret": 5.0, + "d_all": [ + 0.1120000034570694, + 0.08699999749660492, + 0.13702499866485596, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.054499998688697815, + 0.041999999433755875, + 0.06750000268220901, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.08703332394361496, + -0.10248416662216187, + -0.07121624052524567, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0014998500149985003, + "d_recall_holm": 0.0014998500149985003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: knn]", + "key": "ppr:knn", + "variant": "knn", + "recall": [ + 0.8299999833106995, + 0.8140000104904175, + 0.8460000157356262 + ], + "all": [ + 0.6790000200271606, + 0.6489999890327454, + 0.7089999914169312 + ], + "mrr": [ + 0.9167166352272034, + 0.9019950032234192, + 0.9308208227157593 + ], + "n_ret": 5.0, + "d_all": [ + -0.004999999888241291, + -0.017000000923871994, + 0.007000000216066837, + 0.5048495150484952 + ], + "d_recall": [ + -0.0005000000237487257, + -0.007000000216066837, + 0.006000000052154064, + 1.0 + ], + "d_mrr": [ + -0.007316666655242443, + -0.01345041487365961, + -0.0017829167190939188, + 0.013998600139986002 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: ctrl_mention]", + "key": "ppr:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.8309999704360962, + 0.815500020980835, + 0.847000002861023 + ], + "all": [ + 0.6840000152587891, + 0.6559750437736511, + 0.7129999995231628 + ], + "mrr": [ + 0.9239500761032104, + 0.9096153974533081, + 0.937885046005249 + ], + "n_ret": 5.0, + "d_all": [ + 0.0, + -0.010999999940395355, + 0.010999999940395355, + 1.0 + ], + "d_recall": [ + 0.0005000000237487257, + -0.004999999888241291, + 0.006500000134110451, + 1.0 + ], + "d_mrr": [ + -8.333337609656155e-05, + -0.005517083220183849, + 0.0052845836617052555, + 0.9767023297670233 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: mention]", + "key": "lib_default:mention", + "variant": "mention", + "recall": [ + 0.8554999828338623, + 0.840499997138977, + 0.8705000281333923 + ], + "all": [ + 0.734000027179718, + 0.7080000042915344, + 0.7609999775886536 + ], + "mrr": [ + 0.9240333437919617, + 0.9100645780563354, + 0.9378671050071716 + ], + "n_ret": 5.0, + "d_all": [ + 0.05000000074505806, + 0.035999998450279236, + 0.06499999761581421, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.02500000037252903, + 0.017999999225139618, + 0.032499998807907104, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_all_holm": 0.0014998500149985003, + "d_recall_holm": 0.0014998500149985003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: knn]", + "key": "lib_default:knn", + "variant": "knn", + "recall": [ + 0.8309999704360962, + 0.8154875040054321, + 0.8464999794960022 + ], + "all": [ + 0.6850000023841858, + 0.6569750308990479, + 0.7129999995231628 + ], + "mrr": [ + 0.9240333437919617, + 0.9100645780563354, + 0.9378671050071716 + ], + "n_ret": 5.0, + "d_all": [ + 0.0010000000474974513, + -0.010999999940395355, + 0.01302500069141388, + 1.0 + ], + "d_recall": [ + 0.0005000000237487257, + -0.005499999970197678, + 0.00651250034570694, + 1.0 + ], + "d_mrr": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: ctrl_mention]", + "key": "lib_default:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.8259999752044678, + 0.809499979019165, + 0.8414999842643738 + ], + "all": [ + 0.675000011920929, + 0.6449999809265137, + 0.7039999961853027 + ], + "mrr": [ + 0.9240333437919617, + 0.9100645780563354, + 0.9378671050071716 + ], + "n_ret": 5.0, + "d_all": [ + -0.008999999612569809, + -0.014999999664723873, + -0.004000000189989805, + 0.0032996700329967 + ], + "d_recall": [ + -0.0044999998062849045, + -0.007499999832361937, + -0.0020000000949949026, + 0.0032996700329967 + ], + "d_mrr": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_all_holm": 0.0263973602639736, + "d_recall_holm": 0.0263973602639736 + } + ], + "10": [ + { + "label": "Vector top-k (plain)", + "key": "vector", + "variant": null, + "recall": [ + 0.8855000138282776, + 0.871999979019165, + 0.8989999890327454 + ], + "all": [ + 0.7850000262260437, + 0.7599999904632568, + 0.8090000152587891 + ], + "mrr": [ + 0.9253368973731995, + 0.9117423295974731, + 0.9387794733047485 + ], + "n_ret": 10.0 + }, + { + "label": "BM25", + "key": "bm25", + "variant": null, + "recall": [ + 0.8414999842643738, + 0.8264999985694885, + 0.8560000061988831 + ], + "all": [ + 0.6909999847412109, + 0.6629999876022339, + 0.718999981880188 + ], + "mrr": [ + 0.8509678244590759, + 0.8340070843696594, + 0.8668574690818787 + ], + "n_ret": 10.0, + "d_all": [ + -0.09399999678134918, + -0.12200000137090683, + -0.06400000303983688, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.04399999976158142, + -0.05900000035762787, + -0.02800000086426735, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.07436905056238174, + -0.09378770738840103, + -0.05595141649246216, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0014998500149985003, + "d_recall_holm": 0.0014998500149985003 + }, + { + "label": "Hybrid vector+BM25 (RRF)", + "key": "rrf", + "variant": null, + "recall": [ + 0.9185000061988831, + 0.906499981880188, + 0.9294999837875366 + ], + "all": [ + 0.8379999995231628, + 0.8140000104904175, + 0.8600000143051147 + ], + "mrr": [ + 0.9148861169815063, + 0.9015665650367737, + 0.9278804063796997 + ], + "n_ret": 10.0, + "d_all": [ + 0.05299999937415123, + 0.032999999821186066, + 0.0729999989271164, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.032999999821186066, + 0.0215000007301569, + 0.04399999976158142, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.01045079343020916, + -0.022503212094306946, + 0.0014644446782767773, + 0.0917908209179082 + ], + "d_all_holm": 0.0014998500149985003, + "d_recall_holm": 0.0014998500149985003 + }, + { + "label": "Vector top-30 + cross-encoder", + "key": "rerank", + "variant": null, + "recall": [ + 0.9185000061988831, + 0.906499981880188, + 0.9300000071525574 + ], + "all": [ + 0.8410000205039978, + 0.8190000057220459, + 0.8629999756813049 + ], + "mrr": [ + 0.9512857794761658, + 0.9399682879447937, + 0.9615660309791565 + ], + "n_ret": 10.0, + "d_all": [ + 0.0560000017285347, + 0.035999998450279236, + 0.07500000298023224, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.032999999821186066, + 0.02250000089406967, + 0.0435124970972538, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.02594880759716034, + 0.01272848155349493, + 0.038887616246938705, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0014998500149985003, + "d_recall_holm": 0.0014998500149985003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph chosen on dev: mention]", + "key": "lib_mh@best", + "variant": "mention", + "recall": [ + 0.9474999904632568, + 0.9355000257492065, + 0.9585000276565552 + ], + "all": [ + 0.9150000214576721, + 0.8970000147819519, + 0.9319999814033508 + ], + "mrr": [ + 0.924477756023407, + 0.9105108380317688, + 0.9381338357925415 + ], + "n_ret": 9.54800033569336, + "d_all": [ + 0.12999999523162842, + 0.10599999874830246, + 0.15402500331401825, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.06199999898672104, + 0.05000000074505806, + 0.07450000196695328, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0008591270307078958, + -0.0015476191183552146, + -0.0002678571327123791, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0014998500149985003, + "d_recall_holm": 0.0014998500149985003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention]", + "key": "ppr@best", + "variant": "mention", + "recall": [ + 0.9514999985694885, + 0.9394999742507935, + 0.9614999890327454 + ], + "all": [ + 0.9190000295639038, + 0.9010000228881836, + 0.9350000023841858 + ], + "mrr": [ + 0.8811274170875549, + 0.8652105331420898, + 0.8960633277893066 + ], + "n_ret": 10.0, + "d_all": [ + 0.1340000033378601, + 0.10999999940395355, + 0.15800000727176666, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.06599999964237213, + 0.05400000140070915, + 0.07800000160932541, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.044209521263837814, + -0.05744348093867302, + -0.030766574665904045, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0014998500149985003, + "d_recall_holm": 0.0014998500149985003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph chosen on dev: mention]", + "key": "lib_default@best", + "variant": "mention", + "recall": [ + 0.9265000224113464, + 0.9125000238418579, + 0.9390000104904175 + ], + "all": [ + 0.875, + 0.8539999723434448, + 0.8949999809265137 + ], + "mrr": [ + 0.9241999983787537, + 0.9100645780563354, + 0.9378845691680908 + ], + "n_ret": 7.843999862670898, + "d_all": [ + 0.09000000357627869, + 0.06599999964237213, + 0.11500000208616257, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.04100000113248825, + 0.028999999165534973, + 0.05351249873638153, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0011369046987965703, + -0.0019705358427017927, + -0.0004285714530851692, + 0.007399260073992601 + ], + "d_all_holm": 0.0014998500149985003, + "d_recall_holm": 0.0014998500149985003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: mention]", + "key": "lib_mh:mention", + "variant": "mention", + "recall": [ + 0.9474999904632568, + 0.9355000257492065, + 0.9585000276565552 + ], + "all": [ + 0.9150000214576721, + 0.8970000147819519, + 0.9319999814033508 + ], + "mrr": [ + 0.924477756023407, + 0.9105108380317688, + 0.9381338357925415 + ], + "n_ret": 9.54800033569336, + "d_all": [ + 0.12999999523162842, + 0.10599999874830246, + 0.15402500331401825, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.06199999898672104, + 0.05000000074505806, + 0.07450000196695328, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0008591270307078958, + -0.0015476191183552146, + -0.0002678571327123791, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0014998500149985003, + "d_recall_holm": 0.0014998500149985003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: knn]", + "key": "lib_mh:knn", + "variant": "knn", + "recall": [ + 0.9079999923706055, + 0.8949999809265137, + 0.9200000166893005 + ], + "all": [ + 0.8299999833106995, + 0.8079750537872314, + 0.8519999980926514 + ], + "mrr": [ + 0.9254123568534851, + 0.9118461012840271, + 0.9388067126274109 + ], + "n_ret": 10.0, + "d_all": [ + 0.04500000178813934, + 0.02500000037252903, + 0.06400000303983688, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.02250000089406967, + 0.012987500056624413, + 0.032499998807907104, + 9.999000099990002e-05 + ], + "d_mrr": [ + 7.539683429058641e-05, + -0.0005178571445867419, + 0.0006626984686590731, + 0.7785221477852214 + ], + "d_all_holm": 0.0014998500149985003, + "d_recall_holm": 0.0014998500149985003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: ctrl_mention]", + "key": "lib_mh:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.8849999904632568, + 0.8715000152587891, + 0.8985000252723694 + ], + "all": [ + 0.7839999794960022, + 0.7590000033378601, + 0.8080000281333923 + ], + "mrr": [ + 0.9253368973731995, + 0.9117423295974731, + 0.9387794733047485 + ], + "n_ret": 10.0, + "d_all": [ + -0.0010000000474974513, + -0.003000000026077032, + 0.0, + 1.0 + ], + "d_recall": [ + -0.0005000000237487257, + -0.001500000013038516, + 0.0, + 1.0 + ], + "d_mrr": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: mention]", + "key": "ppr:mention", + "variant": "mention", + "recall": [ + 0.9514999985694885, + 0.9394999742507935, + 0.9614999890327454 + ], + "all": [ + 0.9190000295639038, + 0.9010000228881836, + 0.9350000023841858 + ], + "mrr": [ + 0.8811274170875549, + 0.8652105331420898, + 0.8960633277893066 + ], + "n_ret": 10.0, + "d_all": [ + 0.1340000033378601, + 0.10999999940395355, + 0.15800000727176666, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.06599999964237213, + 0.05400000140070915, + 0.07800000160932541, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.044209521263837814, + -0.05744348093867302, + -0.030766574665904045, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0014998500149985003, + "d_recall_holm": 0.0014998500149985003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: knn]", + "key": "ppr:knn", + "variant": "knn", + "recall": [ + 0.9010000228881836, + 0.887499988079071, + 0.9139999747276306 + ], + "all": [ + 0.8169999718666077, + 0.7940000295639038, + 0.8410000205039978 + ], + "mrr": [ + 0.8828147053718567, + 0.866718053817749, + 0.8981935977935791 + ], + "n_ret": 10.0, + "d_all": [ + 0.03200000151991844, + 0.012000000104308128, + 0.052000001072883606, + 0.001999800019998 + ], + "d_recall": [ + 0.01549999974668026, + 0.005499999970197678, + 0.025499999523162842, + 0.0035996400359964 + ], + "d_mrr": [ + -0.04252222552895546, + -0.0541926845908165, + -0.031085476279258728, + 9.999000099990002e-05 + ], + "d_all_holm": 0.007999200079992, + "d_recall_holm": 0.0143985601439856 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: ctrl_mention]", + "key": "ppr:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.8845000267028809, + 0.8705000281333923, + 0.8980000019073486 + ], + "all": [ + 0.7820000052452087, + 0.7559999823570251, + 0.8069999814033508 + ], + "mrr": [ + 0.9120824933052063, + 0.8977774977684021, + 0.9266164302825928 + ], + "n_ret": 10.0, + "d_all": [ + -0.003000000026077032, + -0.009025000035762787, + 0.003000000026077032, + 0.5436456354364564 + ], + "d_recall": [ + -0.0010000000474974513, + -0.0044999998062849045, + 0.0024999999441206455, + 0.7676232376762324 + ], + "d_mrr": [ + -0.013254365883767605, + -0.020502887666225433, + -0.005330685060471296, + 0.00039996000399960006 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: mention]", + "key": "lib_default:mention", + "variant": "mention", + "recall": [ + 0.9265000224113464, + 0.9125000238418579, + 0.9390000104904175 + ], + "all": [ + 0.875, + 0.8539999723434448, + 0.8949999809265137 + ], + "mrr": [ + 0.9241999983787537, + 0.9100645780563354, + 0.9378845691680908 + ], + "n_ret": 7.843999862670898, + "d_all": [ + 0.09000000357627869, + 0.06599999964237213, + 0.11500000208616257, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.04100000113248825, + 0.028999999165534973, + 0.05351249873638153, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0011369046987965703, + -0.0019705358427017927, + -0.0004285714530851692, + 0.007399260073992601 + ], + "d_all_holm": 0.0014998500149985003, + "d_recall_holm": 0.0014998500149985003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: knn]", + "key": "lib_default:knn", + "variant": "knn", + "recall": [ + 0.8945000171661377, + 0.8804875016212463, + 0.9075000286102295 + ], + "all": [ + 0.8090000152587891, + 0.7850000262260437, + 0.8320000171661377 + ], + "mrr": [ + 0.9244250655174255, + 0.9107639789581299, + 0.9380185008049011 + ], + "n_ret": 10.0, + "d_all": [ + 0.024000000208616257, + 0.003000000026077032, + 0.0440250001847744, + 0.039896010398960104 + ], + "d_recall": [ + 0.008999999612569809, + -0.0020000000949949026, + 0.019999999552965164, + 0.1376862313768623 + ], + "d_mrr": [ + -0.0009119047899730504, + -0.0016812203684821725, + -0.0002857143117580563, + 0.0163983601639836 + ], + "d_all_holm": 0.11968803119688032, + "d_recall_holm": 0.41305869413058693 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: ctrl_mention]", + "key": "lib_default:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.8299999833106995, + 0.8140000104904175, + 0.8460000157356262 + ], + "all": [ + 0.6830000281333923, + 0.6539999842643738, + 0.7120000123977661 + ], + "mrr": [ + 0.9240333437919617, + 0.9100645780563354, + 0.9378671050071716 + ], + "n_ret": 9.081000328063965, + "d_all": [ + -0.10199999809265137, + -0.12200000137090683, + -0.08399999886751175, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.05550000071525574, + -0.06599999964237213, + -0.04600000008940697, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0013035715091973543, + -0.0021967263892292976, + -0.0005416666972450912, + 0.0036996300369963003 + ], + "d_all_holm": 0.0014998500149985003, + "d_recall_holm": 0.0014998500149985003 + } + ], + "20": [ + { + "label": "Vector top-k (plain)", + "key": "vector", + "variant": null, + "recall": [ + 0.9240000247955322, + 0.9124875068664551, + 0.9350000023841858 + ], + "all": [ + 0.8539999723434448, + 0.8320000171661377, + 0.875 + ], + "mrr": [ + 0.92588210105896, + 0.9123752117156982, + 0.9392346143722534 + ], + "n_ret": 20.0 + }, + { + "label": "BM25", + "key": "bm25", + "variant": null, + "recall": [ + 0.9100000262260437, + 0.8980000019073486, + 0.921500027179718 + ], + "all": [ + 0.8220000267028809, + 0.7979999780654907, + 0.8440249562263489 + ], + "mrr": [ + 0.8514444828033447, + 0.8346499800682068, + 0.8671927452087402 + ], + "n_ret": 20.0, + "d_all": [ + -0.03200000151991844, + -0.0560000017285347, + -0.007000000216066837, + 0.014898510148985102 + ], + "d_recall": [ + -0.014000000432133675, + -0.026000000536441803, + -0.001500000013038516, + 0.03899610038996101 + ], + "d_mrr": [ + -0.07443755865097046, + -0.09374308586120605, + -0.05617493763566017, + 9.999000099990002e-05 + ], + "d_all_holm": 0.10428957104289571, + "d_recall_holm": 0.27297270272972707 + }, + { + "label": "Hybrid vector+BM25 (RRF)", + "key": "rrf", + "variant": null, + "recall": [ + 0.9480000138282776, + 0.9390000104904175, + 0.9574999809265137 + ], + "all": [ + 0.8960000276565552, + 0.878000020980835, + 0.9150000214576721 + ], + "mrr": [ + 0.914941668510437, + 0.9016207456588745, + 0.9279752373695374 + ], + "n_ret": 20.0, + "d_all": [ + 0.041999999433755875, + 0.026000000536441803, + 0.05900000035762787, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.024000000208616257, + 0.014999999664723873, + 0.03350000083446503, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.01094040647149086, + -0.022886604070663452, + 0.0008312311838380992, + 0.0761923807619238 + ], + "d_all_holm": 0.0014998500149985003, + "d_recall_holm": 0.0014998500149985003 + }, + { + "label": "Vector top-30 + cross-encoder", + "key": "rerank", + "variant": null, + "recall": [ + 0.9359999895095825, + 0.9259874820709229, + 0.9465000033378601 + ], + "all": [ + 0.875, + 0.8539999723434448, + 0.8949999809265137 + ], + "mrr": [ + 0.9513765573501587, + 0.9401631355285645, + 0.9617266058921814 + ], + "n_ret": 20.0, + "d_all": [ + 0.020999999716877937, + 0.010999999940395355, + 0.03200000151991844, + 0.00019998000199980003 + ], + "d_recall": [ + 0.012000000104308128, + 0.006000000052154064, + 0.017999999225139618, + 0.00029997000299970003 + ], + "d_mrr": [ + 0.02549455128610134, + 0.012252472341060638, + 0.038382381200790405, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0015998400159984002, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph chosen on dev: mention]", + "key": "lib_mh@best", + "variant": "mention", + "recall": [ + 0.972000002861023, + 0.9635000228881836, + 0.9800000190734863 + ], + "all": [ + 0.9520000219345093, + 0.9390000104904175, + 0.9649999737739563 + ], + "mrr": [ + 0.9259817004203796, + 0.9125874042510986, + 0.9393846988677979 + ], + "n_ret": 19.94300079345703, + "d_all": [ + 0.09799999743700027, + 0.07900000363588333, + 0.11800000071525574, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.04800000041723251, + 0.03849999979138374, + 0.057999998331069946, + 9.999000099990002e-05 + ], + "d_mrr": [ + 9.95870286715217e-05, + -0.0002302631619386375, + 0.0005714286235161126, + 0.8729127087291271 + ], + "d_all_holm": 0.0014998500149985003, + "d_recall_holm": 0.0014998500149985003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention]", + "key": "ppr@best", + "variant": "mention", + "recall": [ + 0.9704999923706055, + 0.9610000252723694, + 0.9785000085830688 + ], + "all": [ + 0.9520000219345093, + 0.9380000233650208, + 0.9649999737739563 + ], + "mrr": [ + 0.881475567817688, + 0.8655428886413574, + 0.8963151574134827 + ], + "n_ret": 20.0, + "d_all": [ + 0.09799999743700027, + 0.07900000363588333, + 0.11900000274181366, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.04650000110268593, + 0.0364999994635582, + 0.057500001043081284, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.04440649598836899, + -0.057639915496110916, + -0.031009133905172348, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0014998500149985003, + "d_recall_holm": 0.0014998500149985003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph chosen on dev: knn]", + "key": "lib_default@best", + "variant": "knn", + "recall": [ + 0.925000011920929, + 0.9125000238418579, + 0.9365000128746033 + ], + "all": [ + 0.8650000095367432, + 0.843999981880188, + 0.8849999904632568 + ], + "mrr": [ + 0.9247738718986511, + 0.9111548066139221, + 0.9384029507637024 + ], + "n_ret": 20.0, + "d_all": [ + 0.010999999940395355, + -0.010999999940395355, + 0.03200000151991844, + 0.37396260373962603 + ], + "d_recall": [ + 0.0010000000474974513, + -0.010499999858438969, + 0.012012500315904617, + 0.9319068093190681 + ], + "d_mrr": [ + -0.0011081950506195426, + -0.0019020981853827834, + -0.0004430222907103598, + 0.0013998600139986002 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: mention]", + "key": "lib_mh:mention", + "variant": "mention", + "recall": [ + 0.972000002861023, + 0.9635000228881836, + 0.9800000190734863 + ], + "all": [ + 0.9520000219345093, + 0.9390000104904175, + 0.9649999737739563 + ], + "mrr": [ + 0.9259817004203796, + 0.9125874042510986, + 0.9393846988677979 + ], + "n_ret": 19.94300079345703, + "d_all": [ + 0.09799999743700027, + 0.07900000363588333, + 0.11800000071525574, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.04800000041723251, + 0.03849999979138374, + 0.057999998331069946, + 9.999000099990002e-05 + ], + "d_mrr": [ + 9.95870286715217e-05, + -0.0002302631619386375, + 0.0005714286235161126, + 0.8729127087291271 + ], + "d_all_holm": 0.0014998500149985003, + "d_recall_holm": 0.0014998500149985003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: knn]", + "key": "lib_mh:knn", + "variant": "knn", + "recall": [ + 0.9449999928474426, + 0.934499979019165, + 0.9555000066757202 + ], + "all": [ + 0.8960000276565552, + 0.8769999742507935, + 0.9160000085830688 + ], + "mrr": [ + 0.9258019924163818, + 0.9122942090034485, + 0.9391942024230957 + ], + "n_ret": 20.0, + "d_all": [ + 0.041999999433755875, + 0.024000000208616257, + 0.05999999865889549, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.020999999716877937, + 0.011500000022351742, + 0.029999999329447746, + 9.999000099990002e-05 + ], + "d_mrr": [ + -8.008525765035301e-05, + -0.0003342888958286494, + 0.00015882353181950748, + 0.43515648435156484 + ], + "d_all_holm": 0.0014998500149985003, + "d_recall_holm": 0.0014998500149985003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: ctrl_mention]", + "key": "lib_mh:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.9240000247955322, + 0.9124875068664551, + 0.9350000023841858 + ], + "all": [ + 0.8539999723434448, + 0.8320000171661377, + 0.875 + ], + "mrr": [ + 0.92588210105896, + 0.9123752117156982, + 0.9392346143722534 + ], + "n_ret": 20.0, + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_mrr": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: mention]", + "key": "ppr:mention", + "variant": "mention", + "recall": [ + 0.9704999923706055, + 0.9610000252723694, + 0.9785000085830688 + ], + "all": [ + 0.9520000219345093, + 0.9380000233650208, + 0.9649999737739563 + ], + "mrr": [ + 0.881475567817688, + 0.8655428886413574, + 0.8963151574134827 + ], + "n_ret": 20.0, + "d_all": [ + 0.09799999743700027, + 0.07900000363588333, + 0.11900000274181366, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.04650000110268593, + 0.0364999994635582, + 0.057500001043081284, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.04440649598836899, + -0.057639915496110916, + -0.031009133905172348, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0014998500149985003, + "d_recall_holm": 0.0014998500149985003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: knn]", + "key": "ppr:knn", + "variant": "knn", + "recall": [ + 0.934499979019165, + 0.9235000014305115, + 0.9449999928474426 + ], + "all": [ + 0.8769999742507935, + 0.8569999933242798, + 0.8970000147819519 + ], + "mrr": [ + 0.8832784295082092, + 0.8672871589660645, + 0.8986768126487732 + ], + "n_ret": 20.0, + "d_all": [ + 0.023000000044703484, + 0.004999999888241291, + 0.03999999910593033, + 0.0198980101989801 + ], + "d_recall": [ + 0.010499999858438969, + 0.0005000000237487257, + 0.019999999552965164, + 0.0484951504849515 + ], + "d_mrr": [ + -0.042603664100170135, + -0.054047707468271255, + -0.03112073987722397, + 9.999000099990002e-05 + ], + "d_all_holm": 0.1193880611938806, + "d_recall_holm": 0.290970902909709 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: ctrl_mention]", + "key": "ppr:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.9225000143051147, + 0.9104999899864197, + 0.9334999918937683 + ], + "all": [ + 0.8510000109672546, + 0.8289999961853027, + 0.871999979019165 + ], + "mrr": [ + 0.9156147241592407, + 0.9013051986694336, + 0.9295632243156433 + ], + "n_ret": 20.0, + "d_all": [ + -0.003000000026077032, + -0.007000000216066837, + 0.0, + 0.2532746725327467 + ], + "d_recall": [ + -0.001500000013038516, + -0.0035000001080334187, + 0.0, + 0.2532746725327467 + ], + "d_mrr": [ + -0.010267305187880993, + -0.01647806353867054, + -0.003846929408609867, + 0.0014998500149985001 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: mention]", + "key": "lib_default:mention", + "variant": "mention", + "recall": [ + 0.9265000224113464, + 0.9125000238418579, + 0.9390000104904175 + ], + "all": [ + 0.875, + 0.8539999723434448, + 0.8949999809265137 + ], + "mrr": [ + 0.9241999983787537, + 0.9100645780563354, + 0.9378845691680908 + ], + "n_ret": 9.368000030517578, + "d_all": [ + 0.020999999716877937, + -0.004000000189989805, + 0.04500000178813934, + 0.11228877112288771 + ], + "d_recall": [ + 0.0024999999441206455, + -0.010499999858438969, + 0.01549999974668026, + 0.7736226377362264 + ], + "d_mrr": [ + -0.001682072994299233, + -0.0026214146055281162, + -0.0008762746583670378, + 9.999000099990002e-05 + ], + "d_all_holm": 0.5614438556144385, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: knn]", + "key": "lib_default:knn", + "variant": "knn", + "recall": [ + 0.925000011920929, + 0.9125000238418579, + 0.9365000128746033 + ], + "all": [ + 0.8650000095367432, + 0.843999981880188, + 0.8849999904632568 + ], + "mrr": [ + 0.9247738718986511, + 0.9111548066139221, + 0.9384029507637024 + ], + "n_ret": 20.0, + "d_all": [ + 0.010999999940395355, + -0.010999999940395355, + 0.03200000151991844, + 0.37396260373962603 + ], + "d_recall": [ + 0.0010000000474974513, + -0.010499999858438969, + 0.012012500315904617, + 0.9319068093190681 + ], + "d_mrr": [ + -0.0011081950506195426, + -0.0019020981853827834, + -0.0004430222907103598, + 0.0013998600139986002 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: ctrl_mention]", + "key": "lib_default:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.8299999833106995, + 0.8140000104904175, + 0.8460000157356262 + ], + "all": [ + 0.6830000281333923, + 0.6539999842643738, + 0.7120000123977661 + ], + "mrr": [ + 0.9240333437919617, + 0.9100645780563354, + 0.9378671050071716 + ], + "n_ret": 11.4399995803833, + "d_all": [ + -0.17100000381469727, + -0.19499999284744263, + -0.14800000190734863, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.09399999678134918, + -0.10700000077486038, + -0.0820000022649765, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0018487396882846951, + -0.0028377061244100332, + -0.0009928615763783455, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0014998500149985003, + "d_recall_holm": 0.0014998500149985003 + } + ] + }, + "subgraph": { + "5": [ + { + "variant": "mention", + "connect_max_hops": 2, + "size": 5.013000011444092, + "recall": [ + 0.8980000019073486, + 0.8834999799728394, + 0.9114999771118164 + ], + "all": [ + 0.8180000185966492, + 0.7939750552177429, + 0.8420000076293945 + ], + "vec_matched_recall": [ + 0.8325, + 0.8165, + 0.8485 + ], + "vec_matched_all": [ + 0.687, + 0.658, + 0.716 + ], + "d_all": [ + 0.13099999725818634, + 0.10999999940395355, + 0.1550000011920929, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.06549999862909317, + 0.054999999701976776, + 0.07750000059604645, + 9.999000099990002e-05 + ] + }, + { + "variant": "mention", + "connect_max_hops": 3, + "size": 5.014999866485596, + "recall": [ + 0.8980000019073486, + 0.8834999799728394, + 0.9114999771118164 + ], + "all": [ + 0.8180000185966492, + 0.7939750552177429, + 0.8420000076293945 + ], + "vec_matched_recall": [ + 0.8325, + 0.8165, + 0.8485 + ], + "vec_matched_all": [ + 0.687, + 0.658, + 0.716 + ], + "d_all": [ + 0.13099999725818634, + 0.10999999940395355, + 0.1550000011920929, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.06549999862909317, + 0.054999999701976776, + 0.07750000059604645, + 9.999000099990002e-05 + ] + }, + { + "variant": "knn", + "connect_max_hops": 2, + "size": 5.23799991607666, + "recall": [ + 0.8420000076293945, + 0.8259999752044678, + 0.8575000166893005 + ], + "all": [ + 0.7070000171661377, + 0.6790000200271606, + 0.7360000014305115 + ], + "vec_matched_recall": [ + 0.836, + 0.8205, + 0.852 + ], + "vec_matched_all": [ + 0.691, + 0.661, + 0.719 + ], + "d_all": [ + 0.01600000075995922, + -0.004999999888241291, + 0.03700000047683716, + 0.1531846815318468 + ], + "d_recall": [ + 0.006000000052154064, + -0.004999999888241291, + 0.017000000923871994, + 0.30646935306469353 + ] + }, + { + "variant": "knn", + "connect_max_hops": 3, + "size": 5.635000228881836, + "recall": [ + 0.8454999923706055, + 0.8295000195503235, + 0.8615125417709351 + ], + "all": [ + 0.7139999866485596, + 0.6850000023841858, + 0.7429999709129333 + ], + "vec_matched_recall": [ + 0.84, + 0.8249875, + 0.8550125 + ], + "vec_matched_all": [ + 0.697, + 0.668, + 0.724 + ], + "d_all": [ + 0.017000000923871994, + -0.004999999888241291, + 0.03799999877810478, + 0.135986401359864 + ], + "d_recall": [ + 0.005499999970197678, + -0.005499999970197678, + 0.016499999910593033, + 0.36726327367263273 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 2, + "size": 5.000999927520752, + "recall": [ + 0.8259999752044678, + 0.809499979019165, + 0.8414999842643738 + ], + "all": [ + 0.675000011920929, + 0.6449999809265137, + 0.7039999961853027 + ], + "vec_matched_recall": [ + 0.8305, + 0.8145, + 0.846 + ], + "vec_matched_all": [ + 0.684, + 0.655, + 0.7120249999999998 + ], + "d_all": [ + -0.008999999612569809, + -0.014999999664723873, + -0.004000000189989805, + 0.0032996700329967 + ], + "d_recall": [ + -0.0044999998062849045, + -0.007499999832361937, + -0.0020000000949949026, + 0.0032996700329967 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 3, + "size": 5.000999927520752, + "recall": [ + 0.8259999752044678, + 0.809499979019165, + 0.8414999842643738 + ], + "all": [ + 0.675000011920929, + 0.6449999809265137, + 0.7039999961853027 + ], + "vec_matched_recall": [ + 0.8305, + 0.8145, + 0.846 + ], + "vec_matched_all": [ + 0.684, + 0.655, + 0.7120249999999998 + ], + "d_all": [ + -0.008999999612569809, + -0.014999999664723873, + -0.004000000189989805, + 0.0032996700329967 + ], + "d_recall": [ + -0.0044999998062849045, + -0.007499999832361937, + -0.0020000000949949026, + 0.0032996700329967 + ] + } + ], + "10": [ + { + "variant": "mention", + "connect_max_hops": 2, + "size": 9.579000473022461, + "recall": [ + 0.9480000138282776, + 0.9359999895095825, + 0.9589999914169312 + ], + "all": [ + 0.9160000085830688, + 0.8980000019073486, + 0.9330000281333923 + ], + "vec_matched_recall": [ + 0.88, + 0.866, + 0.8935 + ], + "vec_matched_all": [ + 0.774, + 0.748, + 0.799 + ], + "d_all": [ + 0.1420000046491623, + 0.11699999868869781, + 0.16899999976158142, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.06800000369548798, + 0.054999999701976776, + 0.08150000125169754, + 9.999000099990002e-05 + ] + }, + { + "variant": "mention", + "connect_max_hops": 3, + "size": 9.640999794006348, + "recall": [ + 0.9480000138282776, + 0.9359999895095825, + 0.9589999914169312 + ], + "all": [ + 0.9160000085830688, + 0.8980000019073486, + 0.9330000281333923 + ], + "vec_matched_recall": [ + 0.881, + 0.8669875, + 0.8945 + ], + "vec_matched_all": [ + 0.776, + 0.75, + 0.801 + ], + "d_all": [ + 0.14000000059604645, + 0.11500000208616257, + 0.16699999570846558, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.06700000166893005, + 0.05400000140070915, + 0.08051250129938126, + 9.999000099990002e-05 + ] + }, + { + "variant": "knn", + "connect_max_hops": 2, + "size": 10.473999977111816, + "recall": [ + 0.9114999771118164, + 0.8985000252723694, + 0.9240000247955322 + ], + "all": [ + 0.8370000123977661, + 0.8140000104904175, + 0.859000027179718 + ], + "vec_matched_recall": [ + 0.8885, + 0.875, + 0.902 + ], + "vec_matched_all": [ + 0.791, + 0.765, + 0.8150249999999998 + ], + "d_all": [ + 0.04600000008940697, + 0.026000000536441803, + 0.06599999964237213, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.023000000044703484, + 0.012500000186264515, + 0.03350000083446503, + 9.999000099990002e-05 + ] + }, + { + "variant": "knn", + "connect_max_hops": 3, + "size": 11.097000122070312, + "recall": [ + 0.9120000004768372, + 0.8985000252723694, + 0.9244999885559082 + ], + "all": [ + 0.8379999995231628, + 0.8159999847412109, + 0.8600000143051147 + ], + "vec_matched_recall": [ + 0.8915, + 0.878, + 0.9045124999999999 + ], + "vec_matched_all": [ + 0.796, + 0.771, + 0.821 + ], + "d_all": [ + 0.041999999433755875, + 0.02199999988079071, + 0.06199999898672104, + 0.00019998000199980003 + ], + "d_recall": [ + 0.020500000566244125, + 0.010499999858438969, + 0.03099999949336052, + 0.00029997000299970003 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 2, + "size": 10.00100040435791, + "recall": [ + 0.8849999904632568, + 0.8715000152587891, + 0.8985000252723694 + ], + "all": [ + 0.7839999794960022, + 0.7590000033378601, + 0.8080000281333923 + ], + "vec_matched_recall": [ + 0.8855, + 0.872, + 0.899 + ], + "vec_matched_all": [ + 0.785, + 0.76, + 0.809 + ], + "d_all": [ + -0.0010000000474974513, + -0.003000000026077032, + 0.0, + 1.0 + ], + "d_recall": [ + -0.0005000000237487257, + -0.001500000013038516, + 0.0, + 1.0 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 3, + "size": 10.003000259399414, + "recall": [ + 0.8849999904632568, + 0.8715000152587891, + 0.8985000252723694 + ], + "all": [ + 0.7839999794960022, + 0.7590000033378601, + 0.8080000281333923 + ], + "vec_matched_recall": [ + 0.8855, + 0.872, + 0.899 + ], + "vec_matched_all": [ + 0.785, + 0.76, + 0.809 + ], + "d_all": [ + -0.0010000000474974513, + -0.003000000026077032, + 0.0, + 1.0 + ], + "d_recall": [ + -0.0005000000237487257, + -0.001500000013038516, + 0.0, + 1.0 + ] + } + ], + "20": [ + { + "variant": "mention", + "connect_max_hops": 2, + "size": 20.329999923706055, + "recall": [ + 0.9725000262260437, + 0.9639999866485596, + 0.9804999828338623 + ], + "all": [ + 0.953000009059906, + 0.9390000104904175, + 0.9660000205039978 + ], + "vec_matched_recall": [ + 0.9255, + 0.914, + 0.9365 + ], + "vec_matched_all": [ + 0.856, + 0.834, + 0.876 + ], + "d_all": [ + 0.09700000286102295, + 0.07900000363588333, + 0.11599999666213989, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.04699999839067459, + 0.03750000149011612, + 0.05649999901652336, + 9.999000099990002e-05 + ] + }, + { + "variant": "mention", + "connect_max_hops": 3, + "size": 20.95400047302246, + "recall": [ + 0.9725000262260437, + 0.9639999866485596, + 0.9804999828338623 + ], + "all": [ + 0.953000009059906, + 0.9390000104904175, + 0.9660000205039978 + ], + "vec_matched_recall": [ + 0.926, + 0.9145, + 0.937 + ], + "vec_matched_all": [ + 0.857, + 0.836, + 0.877 + ], + "d_all": [ + 0.09600000083446503, + 0.07800000160932541, + 0.11500000208616257, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.04650000110268593, + 0.03750000149011612, + 0.0560000017285347, + 9.999000099990002e-05 + ] + }, + { + "variant": "knn", + "connect_max_hops": 2, + "size": 21.111000061035156, + "recall": [ + 0.9465000033378601, + 0.9365000128746033, + 0.9564999938011169 + ], + "all": [ + 0.8980000019073486, + 0.8799999952316284, + 0.9169999957084656 + ], + "vec_matched_recall": [ + 0.928, + 0.917, + 0.9385 + ], + "vec_matched_all": [ + 0.861, + 0.84, + 0.881 + ], + "d_all": [ + 0.03700000047683716, + 0.01899999938905239, + 0.054999999701976776, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.01850000023841858, + 0.008999999612569809, + 0.027512500062584877, + 0.00019998000199980003 + ] + }, + { + "variant": "knn", + "connect_max_hops": 3, + "size": 22.09000015258789, + "recall": [ + 0.9470000267028809, + 0.9369999766349792, + 0.9570000171661377 + ], + "all": [ + 0.8980000019073486, + 0.8799999952316284, + 0.9169999957084656 + ], + "vec_matched_recall": [ + 0.931, + 0.92, + 0.9415 + ], + "vec_matched_all": [ + 0.866, + 0.845, + 0.886 + ], + "d_all": [ + 0.03200000151991844, + 0.014999999664723873, + 0.04802500084042549, + 0.0004999500049995 + ], + "d_recall": [ + 0.01600000075995922, + 0.007000000216066837, + 0.02449999935925007, + 0.00039996000399960006 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 2, + "size": 20.05699920654297, + "recall": [ + 0.9240000247955322, + 0.9124875068664551, + 0.9350000023841858 + ], + "all": [ + 0.8539999723434448, + 0.8320000171661377, + 0.875 + ], + "vec_matched_recall": [ + 0.924, + 0.9124875, + 0.935 + ], + "vec_matched_all": [ + 0.854, + 0.832, + 0.875 + ], + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 3, + "size": 20.23699951171875, + "recall": [ + 0.9240000247955322, + 0.9124875068664551, + 0.9350000023841858 + ], + "all": [ + 0.8539999723434448, + 0.8320000171661377, + 0.875 + ], + "vec_matched_recall": [ + 0.924, + 0.9124875, + 0.935 + ], + "vec_matched_all": [ + 0.854, + 0.832, + 0.875 + ], + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ] + } + ] + }, + "lib_minus_ppr": { + "mention": { + "5": [ + 0.017000000923871994, + -0.006000000052154064, + 0.039000000804662704, + 0.1665833416658334 + ], + "10": [ + -0.004000000189989805, + -0.014000000432133675, + 0.006000000052154064, + 0.5494450554944506 + ], + "20": [ + 0.0, + -0.008999999612569809, + 0.008999999612569809, + 1.0 + ] + }, + "knn": { + "5": [ + 0.014000000432133675, + -0.007000000216066837, + 0.03400000184774399, + 0.19988001199880012 + ], + "10": [ + 0.013000000268220901, + 0.0010000000474974513, + 0.025025000795722008, + 0.05219478052194781 + ], + "20": [ + 0.01899999938905239, + 0.006000000052154064, + 0.032999999821186066, + 0.011498850114988501 + ] + } + }, + "breakdown": { + "type": { + "bridge": { + "n": 784, + "vector": 0.730867326259613, + "rrf": 0.7984693646430969, + "rerank": 0.7997449040412903, + "lib_mh@best": 0.8992347121238708, + "ppr@best": 0.9043367505073547 + }, + "comparison": { + "n": 216, + "vector": 0.9814814925193787, + "rrf": 0.9814814925193787, + "rerank": 0.9907407164573669, + "lib_mh@best": 0.9722222089767456, + "ppr@best": 0.9722222089767456 + } + }, + "hops": { + "2": { + "n": 1000, + "vector": 0.7850000262260437, + "rrf": 0.8379999995231628, + "rerank": 0.8410000205039978, + "lib_mh@best": 0.9150000214576721, + "ppr@best": 0.9190000295639038 + } + } + } + } + }, + "2wiki": { + "minilm": { + "n_test": 1000, + "corpus_size": 9239, + "lib_similarity_search_same_top20_set": "198/200", + "graphs": { + "mention": { + "n_edges_undirected_union": 5383, + "n_edges_by_family_directed_or_undirected": 6115, + "mean_degree": 1.165277627448858, + "median_degree": 1.0, + "max_degree": 30, + "isolated_frac": 0.4217988959844139, + "gold_pairs_directly_linked": 0.44278768233387356, + "gold_pairs_linked_within_2": 0.4447325769854133, + "neighbours_of_gold_that_are_in_same_question_context": 0.42326321090525637 + }, + "entity": { + "n_edges_undirected_union": 30240, + "n_edges_by_family_directed_or_undirected": 30240, + "mean_degree": 6.546163004654184, + "median_degree": 4.0, + "max_degree": 86, + "isolated_frac": 0.21798895984413896, + "gold_pairs_directly_linked": 0.4051863857374392, + "gold_pairs_linked_within_2": 0.426580226904376, + "neighbours_of_gold_that_are_in_same_question_context": 0.11197227702934706 + }, + "knn": { + "n_edges_undirected_union": 34360, + "n_edges_by_family_directed_or_undirected": 34360, + "mean_degree": 7.438034419309449, + "median_degree": 6.0, + "max_degree": 52, + "isolated_frac": 0.0, + "gold_pairs_directly_linked": 0.18217179902755268, + "gold_pairs_linked_within_2": 0.233387358184765, + "neighbours_of_gold_that_are_in_same_question_context": 0.13361729246028692 + }, + "mention+entity": { + "n_edges_undirected_union": 31907, + "n_edges_by_family_directed_or_undirected": 36355, + "mean_degree": 6.907024569758632, + "median_degree": 4.0, + "max_degree": 86, + "isolated_frac": 0.17761662517588483, + "gold_pairs_directly_linked": 0.45899513776337114, + "gold_pairs_linked_within_2": 0.46612641815235006, + "neighbours_of_gold_that_are_in_same_question_context": 0.1263623088924525 + }, + "ctrl_mention": { + "n_edges_undirected_union": 5383, + "n_edges_by_family_directed_or_undirected": 6115, + "mean_degree": 1.165277627448858, + "median_degree": 1.0, + "max_degree": 30, + "isolated_frac": 0.4217988959844139, + "gold_pairs_directly_linked": 0.00032414910858995135, + "gold_pairs_linked_within_2": 0.00032414910858995135, + "neighbours_of_gold_that_are_in_same_question_context": 0.0003952569169960474 + }, + "ctrl_entity": { + "n_edges_undirected_union": 30240, + "n_edges_by_family_directed_or_undirected": 30240, + "mean_degree": 6.546163004654184, + "median_degree": 4.0, + "max_degree": 86, + "isolated_frac": 0.21798895984413896, + "gold_pairs_directly_linked": 0.0012965964343598054, + "gold_pairs_linked_within_2": 0.0074554294975688815, + "neighbours_of_gold_that_are_in_same_question_context": 0.0006118890958802483 + } + }, + "selected": { + "mention": { + "lib": { + "5": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 3, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 2, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": null + } + }, + "ppr": { + "5": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "20": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.3, + "directed_mention": true, + "weighted": false + } + } + }, + "entity": { + "lib": { + "5": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + }, + "20": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": "w" + } + }, + "ppr": { + "5": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "20": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + } + } + }, + "knn": { + "lib": { + "5": { + "hops": 1, + "seeds": 15, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 1, + "seeds": 15, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + } + }, + "ppr": { + "5": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": true + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "20": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": true + } + } + }, + "mention+entity": { + "lib": { + "5": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "10": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "20": { + "hops": 1, + "seeds": 15, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + } + }, + "ppr": { + "5": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "20": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + } + } + }, + "ctrl_mention": { + "lib": { + "5": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + } + }, + "ppr": { + "5": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "20": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + } + } + }, + "ctrl_entity": { + "lib": { + "5": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + } + }, + "ppr": { + "5": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "20": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.0, + "directed_mention": true, + "weighted": false + } + } + } + }, + "tables": { + "5": [ + { + "label": "Vector top-k (plain)", + "key": "vector", + "variant": null, + "recall": [ + 0.6079999804496765, + 0.5924999713897705, + 0.623006284236908 + ], + "all": [ + 0.2619999945163727, + 0.23499999940395355, + 0.28702500462532043 + ], + "mrr": [ + 0.9123666882514954, + 0.8983820676803589, + 0.9259337782859802 + ], + "n_ret": 5.0 + }, + { + "label": "BM25", + "key": "bm25", + "variant": null, + "recall": [ + 0.6635000109672546, + 0.6489999890327454, + 0.6787499785423279 + ], + "all": [ + 0.33799999952316284, + 0.3109999895095825, + 0.367000013589859 + ], + "mrr": [ + 0.9372166991233826, + 0.9249820113182068, + 0.9491349458694458 + ], + "n_ret": 5.0, + "d_all": [ + 0.07599999755620956, + 0.052000001072883606, + 0.10100000351667404, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.05550000071525574, + 0.041743747889995575, + 0.07074999809265137, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.02484999969601631, + 0.008281665854156017, + 0.04218500107526779, + 0.0029997000299970002 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "Hybrid vector+BM25 (RRF)", + "key": "rrf", + "variant": null, + "recall": [ + 0.6650000214576721, + 0.6499999761581421, + 0.6797500252723694 + ], + "all": [ + 0.3319999873638153, + 0.30300000309944153, + 0.360025018453598 + ], + "mrr": [ + 0.9478999972343445, + 0.9375995397567749, + 0.9578170776367188 + ], + "n_ret": 5.0, + "d_all": [ + 0.07000000029802322, + 0.050999999046325684, + 0.09000000357627869, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.05700000002980232, + 0.04625000059604645, + 0.06800000369548798, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.03553333133459091, + 0.02473166584968567, + 0.04705125093460083, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "Vector top-30 + cross-encoder", + "key": "rerank", + "variant": null, + "recall": [ + 0.6647499799728394, + 0.6495000123977661, + 0.6790000200271606 + ], + "all": [ + 0.33799999952316284, + 0.3089999854564667, + 0.36500000953674316 + ], + "mrr": [ + 0.9700333476066589, + 0.9618308544158936, + 0.978000819683075 + ], + "n_ret": 5.0, + "d_all": [ + 0.07599999755620956, + 0.057999998331069946, + 0.0949999988079071, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.05674999952316284, + 0.04600000008940697, + 0.06775625050067902, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.05766666680574417, + 0.044799163937568665, + 0.07126916944980621, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph chosen on dev: mention+entity]", + "key": "lib_mh@best", + "variant": "mention+entity", + "recall": [ + 0.8252500295639038, + 0.8097437620162964, + 0.840499997138977 + ], + "all": [ + 0.6380000114440918, + 0.6079999804496765, + 0.6679999828338623 + ], + "mrr": [ + 0.9111167192459106, + 0.8970287442207336, + 0.924584150314331 + ], + "n_ret": 5.0, + "d_all": [ + 0.37599998712539673, + 0.3439750075340271, + 0.40700000524520874, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.2172500044107437, + 0.20200000703334808, + 0.23250000178813934, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0012499999720603228, + -0.0042500002309679985, + 0.0017500000540167093, + 0.42395760423957607 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention]", + "key": "ppr@best", + "variant": "mention", + "recall": [ + 0.7482500076293945, + 0.7302500009536743, + 0.765999972820282 + ], + "all": [ + 0.5180000066757202, + 0.4869999885559082, + 0.5490000247955322 + ], + "mrr": [ + 0.7580333352088928, + 0.738064169883728, + 0.7791008353233337 + ], + "n_ret": 5.0, + "d_all": [ + 0.25600001215934753, + 0.22499999403953552, + 0.2849999964237213, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.14024999737739563, + 0.12399999797344208, + 0.15575625002384186, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.15433332324028015, + -0.17395250499248505, + -0.13348250091075897, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph chosen on dev: mention]", + "key": "lib_default@best", + "variant": "mention", + "recall": [ + 0.768750011920929, + 0.7524999976158142, + 0.7850000262260437 + ], + "all": [ + 0.5350000262260437, + 0.5049999952316284, + 0.5659999847412109 + ], + "mrr": [ + 0.9114500284194946, + 0.8973987698554993, + 0.9246516823768616 + ], + "n_ret": 5.0, + "d_all": [ + 0.27300000190734863, + 0.24500000476837158, + 0.30300000309944153, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.16075000166893005, + 0.14749999344348907, + 0.17524999380111694, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0009166665258817375, + -0.0038333332631736994, + 0.0020000000949949026, + 0.7224277572242775 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: mention]", + "key": "lib_mh:mention", + "variant": "mention", + "recall": [ + 0.8252500295639038, + 0.8097437620162964, + 0.840499997138977 + ], + "all": [ + 0.6380000114440918, + 0.6079999804496765, + 0.6679999828338623 + ], + "mrr": [ + 0.9111167192459106, + 0.8970287442207336, + 0.924584150314331 + ], + "n_ret": 5.0, + "d_all": [ + 0.37599998712539673, + 0.3439750075340271, + 0.40700000524520874, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.2172500044107437, + 0.20200000703334808, + 0.23250000178813934, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0012499999720603228, + -0.0042500002309679985, + 0.0017500000540167093, + 0.42395760423957607 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: entity]", + "key": "lib_mh:entity", + "variant": "entity", + "recall": [ + 0.7039999961853027, + 0.6865000128746033, + 0.7207562327384949 + ], + "all": [ + 0.4339999854564667, + 0.40400001406669617, + 0.46402499079704285 + ], + "mrr": [ + 0.9115833640098572, + 0.8974999785423279, + 0.9247854351997375 + ], + "n_ret": 5.0, + "d_all": [ + 0.1720000058412552, + 0.1459749937057495, + 0.1979999989271164, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.09600000083446503, + 0.08250000327825546, + 0.10850624740123749, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0007833333220332861, + -0.0036012500058859587, + 0.0021504168398678303, + 0.6000399960003999 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: knn]", + "key": "lib_mh:knn", + "variant": "knn", + "recall": [ + 0.6044999957084656, + 0.5889999866485596, + 0.6194999814033508 + ], + "all": [ + 0.2630000114440918, + 0.23600000143051147, + 0.289000004529953 + ], + "mrr": [ + 0.9121999740600586, + 0.8984824419021606, + 0.9254516959190369 + ], + "n_ret": 5.0, + "d_all": [ + 0.0010000000474974513, + -0.013000000268220901, + 0.014000000432133675, + 1.0 + ], + "d_recall": [ + -0.0035000001080334187, + -0.010999999940395355, + 0.0037499999161809683, + 0.40205979402059794 + ], + "d_mrr": [ + -0.00016666650481056422, + -0.0031666664872318506, + 0.002884166780859232, + 0.939006099390061 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: mention+entity]", + "key": "lib_mh:mention+entity", + "variant": "mention+entity", + "recall": [ + 0.8252500295639038, + 0.8097437620162964, + 0.840499997138977 + ], + "all": [ + 0.6380000114440918, + 0.6079999804496765, + 0.6679999828338623 + ], + "mrr": [ + 0.9111167192459106, + 0.8970287442207336, + 0.924584150314331 + ], + "n_ret": 5.0, + "d_all": [ + 0.37599998712539673, + 0.3439750075340271, + 0.40700000524520874, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.2172500044107437, + 0.20200000703334808, + 0.23250000178813934, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0012499999720603228, + -0.0042500002309679985, + 0.0017500000540167093, + 0.42395760423957607 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: ctrl_mention]", + "key": "lib_mh:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.6079999804496765, + 0.5924999713897705, + 0.623006284236908 + ], + "all": [ + 0.2619999945163727, + 0.23499999940395355, + 0.2879999876022339 + ], + "mrr": [ + 0.9113667011260986, + 0.8972324132919312, + 0.9245682954788208 + ], + "n_ret": 5.0, + "d_all": [ + 0.0, + -0.003000000026077032, + 0.003000000026077032, + 1.0 + ], + "d_recall": [ + 0.0, + -0.001500000013038516, + 0.001500000013038516, + 1.0 + ], + "d_mrr": [ + -0.0010000000474974513, + -0.004000000189989805, + 0.0020000000949949026, + 0.7224277572242775 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: ctrl_entity]", + "key": "lib_mh:ctrl_entity", + "variant": "ctrl_entity", + "recall": [ + 0.6079999804496765, + 0.5924999713897705, + 0.623006284236908 + ], + "all": [ + 0.2619999945163727, + 0.23499999940395355, + 0.2879999876022339 + ], + "mrr": [ + 0.9113667011260986, + 0.8972324132919312, + 0.9245682954788208 + ], + "n_ret": 5.0, + "d_all": [ + 0.0, + -0.003000000026077032, + 0.003000000026077032, + 1.0 + ], + "d_recall": [ + 0.0, + -0.001500000013038516, + 0.001500000013038516, + 1.0 + ], + "d_mrr": [ + -0.0010000000474974513, + -0.004000000189989805, + 0.0020000000949949026, + 0.7224277572242775 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: mention]", + "key": "ppr:mention", + "variant": "mention", + "recall": [ + 0.7482500076293945, + 0.7302500009536743, + 0.765999972820282 + ], + "all": [ + 0.5180000066757202, + 0.4869999885559082, + 0.5490000247955322 + ], + "mrr": [ + 0.7580333352088928, + 0.738064169883728, + 0.7791008353233337 + ], + "n_ret": 5.0, + "d_all": [ + 0.25600001215934753, + 0.22499999403953552, + 0.2849999964237213, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.14024999737739563, + 0.12399999797344208, + 0.15575625002384186, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.15433332324028015, + -0.17395250499248505, + -0.13348250091075897, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: entity]", + "key": "ppr:entity", + "variant": "entity", + "recall": [ + 0.6384999752044678, + 0.6217437386512756, + 0.6542500257492065 + ], + "all": [ + 0.31700000166893005, + 0.289000004529953, + 0.34599998593330383 + ], + "mrr": [ + 0.7509333491325378, + 0.7308658361434937, + 0.7707350254058838 + ], + "n_ret": 5.0, + "d_all": [ + 0.054999999701976776, + 0.035999998450279236, + 0.0729999989271164, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.030500000342726707, + 0.020250000059604645, + 0.04025625064969063, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.16143332421779633, + -0.18117457628250122, + -0.14208123087882996, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: knn]", + "key": "ppr:knn", + "variant": "knn", + "recall": [ + 0.6002500057220459, + 0.5845000147819519, + 0.6162562370300293 + ], + "all": [ + 0.25600001215934753, + 0.23000000417232513, + 0.28302499651908875 + ], + "mrr": [ + 0.8948166370391846, + 0.8795995712280273, + 0.9096003770828247 + ], + "n_ret": 5.0, + "d_all": [ + -0.006000000052154064, + -0.017999999225139618, + 0.006025000009685755, + 0.43845615438456154 + ], + "d_recall": [ + -0.00774999987334013, + -0.014750000089406967, + -0.000750000006519258, + 0.045495450454954504 + ], + "d_mrr": [ + -0.017549999058246613, + -0.02546791546046734, + -0.009733332321047783, + 9.999000099990002e-05 + ], + "d_all_holm": 1.0, + "d_recall_holm": 0.18198180181981802 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: mention+entity]", + "key": "ppr:mention+entity", + "variant": "mention+entity", + "recall": [ + 0.6692500114440918, + 0.6520000100135803, + 0.6860062479972839 + ], + "all": [ + 0.3790000081062317, + 0.35100001096725464, + 0.40802499651908875 + ], + "mrr": [ + 0.7012332677841187, + 0.6797654032707214, + 0.7225174903869629 + ], + "n_ret": 5.0, + "d_all": [ + 0.11699999868869781, + 0.09300000220537186, + 0.1420000046491623, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.061250001192092896, + 0.04699374735355377, + 0.07575000077486038, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.21113333106040955, + -0.23398497700691223, + -0.1889328956604004, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: ctrl_mention]", + "key": "ppr:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.6004999876022339, + 0.5854937434196472, + 0.6159999966621399 + ], + "all": [ + 0.2549999952316284, + 0.227974995970726, + 0.2810249924659729 + ], + "mrr": [ + 0.8665833473205566, + 0.8510825037956238, + 0.8826012015342712 + ], + "n_ret": 5.0, + "d_all": [ + -0.007000000216066837, + -0.017000000923871994, + 0.003000000026077032, + 0.23497650234976503 + ], + "d_recall": [ + -0.007499999832361937, + -0.013000000268220901, + -0.0020000000949949026, + 0.011398860113988602 + ], + "d_mrr": [ + -0.045783333480358124, + -0.05708541348576546, + -0.03484958037734032, + 9.999000099990002e-05 + ], + "d_all_holm": 1.0, + "d_recall_holm": 0.05699430056994301 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: ctrl_entity]", + "key": "ppr:ctrl_entity", + "variant": "ctrl_entity", + "recall": [ + 0.6000000238418579, + 0.5842499732971191, + 0.6152562499046326 + ], + "all": [ + 0.2529999911785126, + 0.22599999606609344, + 0.27900001406669617 + ], + "mrr": [ + 0.8996166586875916, + 0.8845317363739014, + 0.9143508076667786 + ], + "n_ret": 5.0, + "d_all": [ + -0.008999999612569809, + -0.01899999938905239, + 0.0010000000474974513, + 0.11248875112488751 + ], + "d_recall": [ + -0.00800000037997961, + -0.013749999925494194, + -0.0024999999441206455, + 0.007699230076992301 + ], + "d_mrr": [ + -0.012749999761581421, + -0.019634583964943886, + -0.00531541695818305, + 0.00039996000399960006 + ], + "d_all_holm": 0.674932506749325, + "d_recall_holm": 0.0461953804619538 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: mention]", + "key": "lib_default:mention", + "variant": "mention", + "recall": [ + 0.768750011920929, + 0.7524999976158142, + 0.7850000262260437 + ], + "all": [ + 0.5350000262260437, + 0.5049999952316284, + 0.5659999847412109 + ], + "mrr": [ + 0.9114500284194946, + 0.8973987698554993, + 0.9246516823768616 + ], + "n_ret": 5.0, + "d_all": [ + 0.27300000190734863, + 0.24500000476837158, + 0.30300000309944153, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.16075000166893005, + 0.14749999344348907, + 0.17524999380111694, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0009166665258817375, + -0.0038333332631736994, + 0.0020000000949949026, + 0.7224277572242775 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: entity]", + "key": "lib_default:entity", + "variant": "entity", + "recall": [ + 0.6767500042915344, + 0.6602500081062317, + 0.6932562589645386 + ], + "all": [ + 0.3779999911785126, + 0.3490000069141388, + 0.40700000524520874 + ], + "mrr": [ + 0.9111500382423401, + 0.897216260433197, + 0.9244354367256165 + ], + "n_ret": 5.0, + "d_all": [ + 0.11599999666213989, + 0.08900000154972076, + 0.1420000046491623, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.06875000149011612, + 0.054499998688697815, + 0.0820000022649765, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.001216666423715651, + -0.004100000020116568, + 0.0016666667070239782, + 0.49865013498650135 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: knn]", + "key": "lib_default:knn", + "variant": "knn", + "recall": [ + 0.5855000019073486, + 0.5702499747276306, + 0.6004999876022339 + ], + "all": [ + 0.23600000143051147, + 0.20999999344348907, + 0.2619999945163727 + ], + "mrr": [ + 0.9117332696914673, + 0.8980817198753357, + 0.9250507950782776 + ], + "n_ret": 5.0, + "d_all": [ + -0.026000000536441803, + -0.04699999839067459, + -0.004000000189989805, + 0.027297270272972702 + ], + "d_recall": [ + -0.02250000089406967, + -0.03350000083446503, + -0.01075000036507845, + 0.00019998000199980003 + ], + "d_mrr": [ + -0.000633333227597177, + -0.003583333222195506, + 0.002317083301022649, + 0.7154284571542846 + ], + "d_all_holm": 0.1910808919108089, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: mention+entity]", + "key": "lib_default:mention+entity", + "variant": "mention+entity", + "recall": [ + 0.7540000081062317, + 0.737500011920929, + 0.7707499861717224 + ], + "all": [ + 0.5120000243186951, + 0.4819999933242798, + 0.5429999828338623 + ], + "mrr": [ + 0.9111500382423401, + 0.897216260433197, + 0.9244354367256165 + ], + "n_ret": 5.0, + "d_all": [ + 0.25, + 0.2199999988079071, + 0.27902501821517944, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.1459999978542328, + 0.13075000047683716, + 0.16124999523162842, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.001216666423715651, + -0.004100000020116568, + 0.0016666667070239782, + 0.49865013498650135 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: ctrl_mention]", + "key": "lib_default:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.597000002861023, + 0.5820000171661377, + 0.6115062832832336 + ], + "all": [ + 0.24199999868869781, + 0.2160000056028366, + 0.2669999897480011 + ], + "mrr": [ + 0.9111167192459106, + 0.8971995115280151, + 0.9244008660316467 + ], + "n_ret": 5.0, + "d_all": [ + -0.019999999552965164, + -0.028999999165534973, + -0.01197499968111515, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.010999999940395355, + -0.01575000025331974, + -0.006500000134110451, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0012499999720603228, + -0.004000000189989805, + 0.001500000013038516, + 0.49865013498650135 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: ctrl_entity]", + "key": "lib_default:ctrl_entity", + "variant": "ctrl_entity", + "recall": [ + 0.5695000290870667, + 0.555243730545044, + 0.5837500095367432 + ], + "all": [ + 0.20000000298023224, + 0.17499999701976776, + 0.22300000488758087 + ], + "mrr": [ + 0.9111167192459106, + 0.8971995115280151, + 0.9244008660316467 + ], + "n_ret": 5.0, + "d_all": [ + -0.06199999898672104, + -0.07699999958276749, + -0.04699999839067459, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.03849999979138374, + -0.04625000059604645, + -0.030500000342726707, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0012499999720603228, + -0.004000000189989805, + 0.001500000013038516, + 0.49865013498650135 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + } + ], + "10": [ + { + "label": "Vector top-k (plain)", + "key": "vector", + "variant": null, + "recall": [ + 0.6520000100135803, + 0.6362437605857849, + 0.6669999957084656 + ], + "all": [ + 0.32499998807907104, + 0.29600000381469727, + 0.35199999809265137 + ], + "mrr": [ + 0.9136483669281006, + 0.8997189998626709, + 0.9267587065696716 + ], + "n_ret": 10.0 + }, + { + "label": "BM25", + "key": "bm25", + "variant": null, + "recall": [ + 0.7122499942779541, + 0.6972500085830688, + 0.7275000214576721 + ], + "all": [ + 0.41499999165534973, + 0.38499999046325684, + 0.4449999928474426 + ], + "mrr": [ + 0.9382262229919434, + 0.9262092709541321, + 0.9497938752174377 + ], + "n_ret": 10.0, + "d_all": [ + 0.09000000357627869, + 0.06599999964237213, + 0.11400000005960464, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.06024999916553497, + 0.04675000160932541, + 0.07374999672174454, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.024577777832746506, + 0.008423987776041031, + 0.04121582955121994, + 0.0026997300269973002 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "Hybrid vector+BM25 (RRF)", + "key": "rrf", + "variant": null, + "recall": [ + 0.7135000228881836, + 0.6987437605857849, + 0.7287562489509583 + ], + "all": [ + 0.41100001335144043, + 0.38100001215934753, + 0.4410000145435333 + ], + "mrr": [ + 0.9481916427612305, + 0.9380325078964233, + 0.9580018520355225 + ], + "n_ret": 10.0, + "d_all": [ + 0.0860000029206276, + 0.06700000166893005, + 0.10499999672174454, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.061500001698732376, + 0.05050000175833702, + 0.07275000214576721, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.034543249756097794, + 0.023813560605049133, + 0.04556208476424217, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "Vector top-30 + cross-encoder", + "key": "rerank", + "variant": null, + "recall": [ + 0.6862499713897705, + 0.6704999804496765, + 0.7005000114440918 + ], + "all": [ + 0.375, + 0.3449999988079071, + 0.4020000100135803 + ], + "mrr": [ + 0.9700333476066589, + 0.9618308544158936, + 0.978000819683075 + ], + "n_ret": 10.0, + "d_all": [ + 0.05000000074505806, + 0.035999998450279236, + 0.06499999761581421, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.03424999862909317, + 0.026499999687075615, + 0.042500000447034836, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.056384917348623276, + 0.04403044655919075, + 0.06959021836519241, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph chosen on dev: mention+entity]", + "key": "lib_mh@best", + "variant": "mention+entity", + "recall": [ + 0.8980000019073486, + 0.8849999904632568, + 0.9110000133514404 + ], + "all": [ + 0.8100000023841858, + 0.7860000133514404, + 0.8339999914169312 + ], + "mrr": [ + 0.9118012189865112, + 0.8979467153549194, + 0.9250941276550293 + ], + "n_ret": 9.979999542236328, + "d_all": [ + 0.48500001430511475, + 0.4519999921321869, + 0.5180000066757202, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.2460000067949295, + 0.2292499989271164, + 0.26249998807907104, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0018472223309800029, + -0.004847569856792688, + 0.0010416667209938169, + 0.2166783321667833 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention]", + "key": "ppr@best", + "variant": "mention", + "recall": [ + 0.8865000009536743, + 0.8727437257766724, + 0.9002500176429749 + ], + "all": [ + 0.7870000004768372, + 0.7620000243186951, + 0.8109999895095825 + ], + "mrr": [ + 0.8508524298667908, + 0.8340439200401306, + 0.8676282167434692 + ], + "n_ret": 10.0, + "d_all": [ + 0.4620000123977661, + 0.42800000309944153, + 0.4970000088214874, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.2345000058412552, + 0.2175000011920929, + 0.25200000405311584, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.06279602646827698, + -0.07707859575748444, + -0.04798600450158119, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph chosen on dev: mention]", + "key": "lib_default@best", + "variant": "mention", + "recall": [ + 0.890250027179718, + 0.8767499923706055, + 0.9035000205039978 + ], + "all": [ + 0.7950000166893005, + 0.7699999809265137, + 0.8190000057220459 + ], + "mrr": [ + 0.9117594957351685, + 0.8978590965270996, + 0.9251335859298706 + ], + "n_ret": 7.867000102996826, + "d_all": [ + 0.4699999988079071, + 0.4350000023841858, + 0.5040000081062317, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.2382500022649765, + 0.2212499976158142, + 0.2554999887943268, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0018888888880610466, + -0.004888888914138079, + 0.0010416667209938169, + 0.2131786821317868 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: mention]", + "key": "lib_mh:mention", + "variant": "mention", + "recall": [ + 0.890500009059906, + 0.8769999742507935, + 0.9037500023841858 + ], + "all": [ + 0.7960000038146973, + 0.7710000276565552, + 0.8199999928474426 + ], + "mrr": [ + 0.9116762280464172, + 0.8978411555290222, + 0.9249223470687866 + ], + "n_ret": 7.959000110626221, + "d_all": [ + 0.47099998593330383, + 0.43700000643730164, + 0.5049999952316284, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.23849999904632568, + 0.2214999943971634, + 0.25600001215934753, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0019722222350537777, + -0.00497222226113081, + 0.0009305555722676218, + 0.1877812218778122 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: entity]", + "key": "lib_mh:entity", + "variant": "entity", + "recall": [ + 0.8090000152587891, + 0.7929937243461609, + 0.8240062594413757 + ], + "all": [ + 0.6060000061988831, + 0.5759999752044678, + 0.6340000033378601 + ], + "mrr": [ + 0.9118012189865112, + 0.8979603052139282, + 0.9250691533088684 + ], + "n_ret": 9.887999534606934, + "d_all": [ + 0.2809999883174896, + 0.24899999797344208, + 0.3109999895095825, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.15700000524520874, + 0.1420000046491623, + 0.17175623774528503, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0018472223309800029, + -0.0047918651252985, + 0.0010798114817589521, + 0.21547845215478453 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: knn]", + "key": "lib_mh:knn", + "variant": "knn", + "recall": [ + 0.6782500147819519, + 0.661493718624115, + 0.6955062747001648 + ], + "all": [ + 0.39800000190734863, + 0.36800000071525574, + 0.42899999022483826 + ], + "mrr": [ + 0.9121777415275574, + 0.8983324766159058, + 0.9253398180007935 + ], + "n_ret": 10.0, + "d_all": [ + 0.0729999989271164, + 0.04800000041723251, + 0.09799999743700027, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.026249999180436134, + 0.012500000186264515, + 0.039506252855062485, + 0.00029997000299970003 + ], + "d_mrr": [ + -0.0014706349465996027, + -0.0045388988219201565, + 0.0015555955469608307, + 0.3408659134086591 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: mention+entity]", + "key": "lib_mh:mention+entity", + "variant": "mention+entity", + "recall": [ + 0.8980000019073486, + 0.8849999904632568, + 0.9110000133514404 + ], + "all": [ + 0.8100000023841858, + 0.7860000133514404, + 0.8339999914169312 + ], + "mrr": [ + 0.9118012189865112, + 0.8979467153549194, + 0.9250941276550293 + ], + "n_ret": 9.979999542236328, + "d_all": [ + 0.48500001430511475, + 0.4519999921321869, + 0.5180000066757202, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.2460000067949295, + 0.2292499989271164, + 0.26249998807907104, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0018472223309800029, + -0.004847569856792688, + 0.0010416667209938169, + 0.2166783321667833 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: ctrl_mention]", + "key": "lib_mh:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.6520000100135803, + 0.6362437605857849, + 0.6669999957084656 + ], + "all": [ + 0.32499998807907104, + 0.29600000381469727, + 0.35199999809265137 + ], + "mrr": [ + 0.9126483798027039, + 0.8987202644348145, + 0.925782322883606 + ], + "n_ret": 10.0, + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_mrr": [ + -0.0010000000474974513, + -0.004000000189989805, + 0.0020000000949949026, + 0.7224277572242775 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: ctrl_entity]", + "key": "lib_mh:ctrl_entity", + "variant": "ctrl_entity", + "recall": [ + 0.6520000100135803, + 0.6362437605857849, + 0.6669999957084656 + ], + "all": [ + 0.32499998807907104, + 0.29600000381469727, + 0.35199999809265137 + ], + "mrr": [ + 0.9126483798027039, + 0.8987202644348145, + 0.925782322883606 + ], + "n_ret": 10.0, + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_mrr": [ + -0.0010000000474974513, + -0.004000000189989805, + 0.0020000000949949026, + 0.7224277572242775 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: mention]", + "key": "ppr:mention", + "variant": "mention", + "recall": [ + 0.8865000009536743, + 0.8727437257766724, + 0.9002500176429749 + ], + "all": [ + 0.7870000004768372, + 0.7620000243186951, + 0.8109999895095825 + ], + "mrr": [ + 0.8508524298667908, + 0.8340439200401306, + 0.8676282167434692 + ], + "n_ret": 10.0, + "d_all": [ + 0.4620000123977661, + 0.42800000309944153, + 0.4970000088214874, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.2345000058412552, + 0.2175000011920929, + 0.25200000405311584, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.06279602646827698, + -0.07707859575748444, + -0.04798600450158119, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: entity]", + "key": "ppr:entity", + "variant": "entity", + "recall": [ + 0.781499981880188, + 0.7645000219345093, + 0.7970062494277954 + ], + "all": [ + 0.5490000247955322, + 0.5170000195503235, + 0.5789999961853027 + ], + "mrr": [ + 0.8734285831451416, + 0.8571339845657349, + 0.8886814713478088 + ], + "n_ret": 10.0, + "d_all": [ + 0.2240000069141388, + 0.1969749927520752, + 0.25099998712539673, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.12950000166893005, + 0.11524999886751175, + 0.1432500034570694, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.04021983966231346, + -0.05168936029076576, + -0.02844073250889778, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: knn]", + "key": "ppr:knn", + "variant": "knn", + "recall": [ + 0.6855000257492065, + 0.6679999828338623, + 0.7020000219345093 + ], + "all": [ + 0.39899998903274536, + 0.367000013589859, + 0.42899999022483826 + ], + "mrr": [ + 0.8733416795730591, + 0.8570196032524109, + 0.8888764977455139 + ], + "n_ret": 10.0, + "d_all": [ + 0.07400000095367432, + 0.05299999937415123, + 0.09600000083446503, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.03350000083446503, + 0.02225000038743019, + 0.045249998569488525, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.04030674695968628, + -0.051518477499485016, + -0.02944096177816391, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: mention+entity]", + "key": "ppr:mention+entity", + "variant": "mention+entity", + "recall": [ + 0.859749972820282, + 0.845493733882904, + 0.8742499947547913 + ], + "all": [ + 0.7200000286102295, + 0.6919999718666077, + 0.746999979019165 + ], + "mrr": [ + 0.8552928566932678, + 0.839108407497406, + 0.8717008829116821 + ], + "n_ret": 10.0, + "d_all": [ + 0.39500001072883606, + 0.3630000054836273, + 0.4269999861717224, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.20774999260902405, + 0.19199375808238983, + 0.2240000069141388, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.05835555121302605, + -0.07258359342813492, + -0.04412252828478813, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: ctrl_mention]", + "key": "ppr:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.653249979019165, + 0.6377500295639038, + 0.6682500243186951 + ], + "all": [ + 0.3269999921321869, + 0.2980000078678131, + 0.35499998927116394 + ], + "mrr": [ + 0.9114397168159485, + 0.8972062468528748, + 0.9247171878814697 + ], + "n_ret": 10.0, + "d_all": [ + 0.0020000000949949026, + -0.003000000026077032, + 0.007000000216066837, + 0.6879312068793121 + ], + "d_recall": [ + 0.0012499999720603228, + -0.0010000000474974513, + 0.0037499999161809683, + 0.45645435456454353 + ], + "d_mrr": [ + -0.002208729973062873, + -0.0062682535499334335, + 0.0016031349077820778, + 0.27587241275872415 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: ctrl_entity]", + "key": "ppr:ctrl_entity", + "variant": "ctrl_entity", + "recall": [ + 0.6470000147819519, + 0.6309999823570251, + 0.6622499823570251 + ], + "all": [ + 0.3160000145435333, + 0.2879999876022339, + 0.3440000116825104 + ], + "mrr": [ + 0.8771424293518066, + 0.8615588545799255, + 0.8928989768028259 + ], + "n_ret": 10.0, + "d_all": [ + -0.008999999612569809, + -0.017000000923871994, + -0.0010000000474974513, + 0.05169483051694831 + ], + "d_recall": [ + -0.004999999888241291, + -0.009999999776482582, + 0.0, + 0.0673932606739326 + ], + "d_mrr": [ + -0.03650595247745514, + -0.04604654386639595, + -0.026297569274902344, + 9.999000099990002e-05 + ], + "d_all_holm": 0.20677932206779323, + "d_recall_holm": 0.2695730426957304 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: mention]", + "key": "lib_default:mention", + "variant": "mention", + "recall": [ + 0.890250027179718, + 0.8767499923706055, + 0.9035000205039978 + ], + "all": [ + 0.7950000166893005, + 0.7699999809265137, + 0.8190000057220459 + ], + "mrr": [ + 0.9117594957351685, + 0.8978590965270996, + 0.9251335859298706 + ], + "n_ret": 7.867000102996826, + "d_all": [ + 0.4699999988079071, + 0.4350000023841858, + 0.5040000081062317, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.2382500022649765, + 0.2212499976158142, + 0.2554999887943268, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0018888888880610466, + -0.004888888914138079, + 0.0010416667209938169, + 0.2131786821317868 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: entity]", + "key": "lib_default:entity", + "variant": "entity", + "recall": [ + 0.7892500162124634, + 0.773493766784668, + 0.8047500252723694 + ], + "all": [ + 0.5649999976158142, + 0.5350000262260437, + 0.5950000286102295 + ], + "mrr": [ + 0.9113166332244873, + 0.8973478674888611, + 0.9246670603752136 + ], + "n_ret": 9.97700023651123, + "d_all": [ + 0.23999999463558197, + 0.20600000023841858, + 0.2720000147819519, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.13725000619888306, + 0.12025000154972076, + 0.15350624918937683, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.002331746043637395, + -0.005351497791707516, + 0.0007199404644779861, + 0.12108789121087891 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: knn]", + "key": "lib_default:knn", + "variant": "knn", + "recall": [ + 0.6782500147819519, + 0.661493718624115, + 0.6955062747001648 + ], + "all": [ + 0.39800000190734863, + 0.36800000071525574, + 0.42899999022483826 + ], + "mrr": [ + 0.9121777415275574, + 0.8983324766159058, + 0.9253398180007935 + ], + "n_ret": 10.0, + "d_all": [ + 0.0729999989271164, + 0.04800000041723251, + 0.09799999743700027, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.026249999180436134, + 0.012500000186264515, + 0.039506252855062485, + 0.00029997000299970003 + ], + "d_mrr": [ + -0.0014706349465996027, + -0.0045388988219201565, + 0.0015555955469608307, + 0.3408659134086591 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: mention+entity]", + "key": "lib_default:mention+entity", + "variant": "mention+entity", + "recall": [ + 0.8585000038146973, + 0.843500018119812, + 0.8730000257492065 + ], + "all": [ + 0.6940000057220459, + 0.6639999747276306, + 0.7229999899864197 + ], + "mrr": [ + 0.9114833474159241, + 0.8975145816802979, + 0.924851655960083 + ], + "n_ret": 9.979999542236328, + "d_all": [ + 0.36899998784065247, + 0.33500000834465027, + 0.40299999713897705, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.20649999380111694, + 0.18949376046657562, + 0.22375623881816864, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0021650793496519327, + -0.005237093195319176, + 0.0008667758665978909, + 0.1581841815818418 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: ctrl_mention]", + "key": "lib_default:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.6079999804496765, + 0.5924999713897705, + 0.623006284236908 + ], + "all": [ + 0.2619999945163727, + 0.23499999940395355, + 0.2879999876022339 + ], + "mrr": [ + 0.9111167192459106, + 0.8971995115280151, + 0.9244008660316467 + ], + "n_ret": 8.487000465393066, + "d_all": [ + -0.06300000101327896, + -0.07900000363588333, + -0.04899999871850014, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.04399999976158142, + -0.05249999836087227, + -0.035743750631809235, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.002531746169552207, + -0.005559821613132954, + 0.000500247988384217, + 0.0908909109089091 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: ctrl_entity]", + "key": "lib_default:ctrl_entity", + "variant": "ctrl_entity", + "recall": [ + 0.5879999995231628, + 0.5730000138282776, + 0.6025062799453735 + ], + "all": [ + 0.2290000021457672, + 0.2029999941587448, + 0.2549999952316284 + ], + "mrr": [ + 0.9111167192459106, + 0.8971995115280151, + 0.9244008660316467 + ], + "n_ret": 9.987000465393066, + "d_all": [ + -0.09600000083446503, + -0.11500000208616257, + -0.07800000160932541, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.06400000303983688, + -0.07425624877214432, + -0.05400000140070915, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.002531746169552207, + -0.005559821613132954, + 0.000500247988384217, + 0.0908909109089091 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + } + ], + "20": [ + { + "label": "Vector top-k (plain)", + "key": "vector", + "variant": null, + "recall": [ + 0.6897500157356262, + 0.6737499833106995, + 0.7047500014305115 + ], + "all": [ + 0.3840000033378601, + 0.3529999852180481, + 0.4129999876022339 + ], + "mrr": [ + 0.9141561985015869, + 0.9002171158790588, + 0.9272523522377014 + ], + "n_ret": 20.0 + }, + { + "label": "BM25", + "key": "bm25", + "variant": null, + "recall": [ + 0.7427499890327454, + 0.7279937267303467, + 0.7584999799728394 + ], + "all": [ + 0.46700000762939453, + 0.4379749894142151, + 0.49799999594688416 + ], + "mrr": [ + 0.9385273456573486, + 0.9265788793563843, + 0.9500040411949158 + ], + "n_ret": 20.0, + "d_all": [ + 0.08299999684095383, + 0.05999999865889549, + 0.10599999874830246, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.05299999937415123, + 0.04125000163912773, + 0.06549999862909317, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.024371124804019928, + 0.008442817255854607, + 0.0408574603497982, + 0.0028997100289971 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "Hybrid vector+BM25 (RRF)", + "key": "rrf", + "variant": null, + "recall": [ + 0.7442499995231628, + 0.7294999957084656, + 0.7587562799453735 + ], + "all": [ + 0.46700000762939453, + 0.43700000643730164, + 0.4970000088214874 + ], + "mrr": [ + 0.9481916427612305, + 0.9380325078964233, + 0.9580018520355225 + ], + "n_ret": 20.0, + "d_all": [ + 0.08299999684095383, + 0.06499999761581421, + 0.10199999809265137, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.054499998688697815, + 0.04475000128149986, + 0.06525000184774399, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.03403548523783684, + 0.02344565838575363, + 0.04493105039000511, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "Vector top-30 + cross-encoder", + "key": "rerank", + "variant": null, + "recall": [ + 0.6952499747276306, + 0.6792437434196472, + 0.7099999785423279 + ], + "all": [ + 0.38999998569488525, + 0.35899999737739563, + 0.4189999997615814 + ], + "mrr": [ + 0.9700333476066589, + 0.9618308544158936, + 0.978000819683075 + ], + "n_ret": 20.0, + "d_all": [ + 0.006000000052154064, + -0.0020000000949949026, + 0.014000000432133675, + 0.21557844215578442 + ], + "d_recall": [ + 0.005499999970197678, + 0.0010000000474974513, + 0.009999999776482582, + 0.018998100189981004 + ], + "d_mrr": [ + 0.05587714910507202, + 0.043528418987989426, + 0.06900747120380402, + 9.999000099990002e-05 + ], + "d_all_holm": 1.0, + "d_recall_holm": 0.09499050094990502 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph chosen on dev: mention+entity]", + "key": "lib_mh@best", + "variant": "mention+entity", + "recall": [ + 0.9390000104904175, + 0.9282500147819519, + 0.9490000009536743 + ], + "all": [ + 0.8730000257492065, + 0.8510000109672546, + 0.8930000066757202 + ], + "mrr": [ + 0.9131500124931335, + 0.8993034362792969, + 0.926099419593811 + ], + "n_ret": 20.0, + "d_all": [ + 0.48899999260902405, + 0.4580000042915344, + 0.5220000147819519, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.24924999475479126, + 0.2339937537908554, + 0.26499998569488525, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0010061919456347823, + -0.003895713947713375, + 0.0018628627294674516, + 0.506049395060494 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention]", + "key": "ppr@best", + "variant": "mention", + "recall": [ + 0.9300000071525574, + 0.9185000061988831, + 0.940750002861023 + ], + "all": [ + 0.8569999933242798, + 0.8349999785423279, + 0.8790000081062317 + ], + "mrr": [ + 0.8122603893280029, + 0.7943459749221802, + 0.8300310969352722 + ], + "n_ret": 20.0, + "d_all": [ + 0.4729999899864197, + 0.4419749975204468, + 0.5049999952316284, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.24025000631809235, + 0.22474999725818634, + 0.2562499940395355, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.10189582407474518, + -0.11837921291589737, + -0.08511381596326828, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph chosen on dev: mention]", + "key": "lib_default@best", + "variant": "mention", + "recall": [ + 0.8912500143051147, + 0.878000020980835, + 0.9047499895095825 + ], + "all": [ + 0.7990000247955322, + 0.7739999890327454, + 0.8230000138282776 + ], + "mrr": [ + 0.9117594957351685, + 0.8978590965270996, + 0.9251335859298706 + ], + "n_ret": 8.440999984741211, + "d_all": [ + 0.41499999165534973, + 0.3779999911785126, + 0.45100000500679016, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.20149999856948853, + 0.18324999511241913, + 0.2202499955892563, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.002396659692749381, + -0.005368917714804411, + 0.0005437714862637222, + 0.11038896110388961 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: mention]", + "key": "lib_mh:mention", + "variant": "mention", + "recall": [ + 0.9355000257492065, + 0.9244999885559082, + 0.9457499980926514 + ], + "all": [ + 0.8659999966621399, + 0.8429999947547913, + 0.8870000243186951 + ], + "mrr": [ + 0.9131500124931335, + 0.8993034362792969, + 0.926099419593811 + ], + "n_ret": 19.444000244140625, + "d_all": [ + 0.4819999933242798, + 0.45100000500679016, + 0.5139999985694885, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.24574999511241913, + 0.23025000095367432, + 0.2617500126361847, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0010061919456347823, + -0.003895713947713375, + 0.0018628627294674516, + 0.506049395060494 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: entity]", + "key": "lib_mh:entity", + "variant": "entity", + "recall": [ + 0.8659999966621399, + 0.8510000109672546, + 0.8809999823570251 + ], + "all": [ + 0.7329999804496765, + 0.7059999704360962, + 0.7609999775886536 + ], + "mrr": [ + 0.9118916392326355, + 0.8980745673179626, + 0.9250958561897278 + ], + "n_ret": 19.819000244140625, + "d_all": [ + 0.3490000069141388, + 0.31299999356269836, + 0.38499999046325684, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.17624999582767487, + 0.1587499976158142, + 0.19350624084472656, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0022645166609436274, + -0.005204010289162397, + 0.000665517698507756, + 0.12348765123487651 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: knn]", + "key": "lib_mh:knn", + "variant": "knn", + "recall": [ + 0.7422500252723694, + 0.7267500162124634, + 0.7580062747001648 + ], + "all": [ + 0.48500001430511475, + 0.45500001311302185, + 0.5149999856948853 + ], + "mrr": [ + 0.9134118556976318, + 0.8996719121932983, + 0.9264995455741882 + ], + "n_ret": 20.0, + "d_all": [ + 0.10100000351667404, + 0.07900000363588333, + 0.12200000137090683, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.05249999836087227, + 0.04149999842047691, + 0.06324999779462814, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0007442872738465667, + -0.0035590704064816236, + 0.002038202714174986, + 0.5941405859414058 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: mention+entity]", + "key": "lib_mh:mention+entity", + "variant": "mention+entity", + "recall": [ + 0.9390000104904175, + 0.9282500147819519, + 0.9490000009536743 + ], + "all": [ + 0.8730000257492065, + 0.8510000109672546, + 0.8930000066757202 + ], + "mrr": [ + 0.9131500124931335, + 0.8993034362792969, + 0.926099419593811 + ], + "n_ret": 20.0, + "d_all": [ + 0.48899999260902405, + 0.4580000042915344, + 0.5220000147819519, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.24924999475479126, + 0.2339937537908554, + 0.26499998569488525, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0010061919456347823, + -0.003895713947713375, + 0.0018628627294674516, + 0.506049395060494 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: ctrl_mention]", + "key": "lib_mh:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.6897500157356262, + 0.6737499833106995, + 0.7047500014305115 + ], + "all": [ + 0.3840000033378601, + 0.3529999852180481, + 0.4129999876022339 + ], + "mrr": [ + 0.9131562113761902, + 0.8993553519248962, + 0.9261235594749451 + ], + "n_ret": 20.0, + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_mrr": [ + -0.0010000000474974513, + -0.004000000189989805, + 0.0020000000949949026, + 0.7224277572242775 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: ctrl_entity]", + "key": "lib_mh:ctrl_entity", + "variant": "ctrl_entity", + "recall": [ + 0.6897500157356262, + 0.6737499833106995, + 0.7047500014305115 + ], + "all": [ + 0.3840000033378601, + 0.3529999852180481, + 0.4129999876022339 + ], + "mrr": [ + 0.9131562113761902, + 0.8993553519248962, + 0.9261235594749451 + ], + "n_ret": 20.0, + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_mrr": [ + -0.0010000000474974513, + -0.004000000189989805, + 0.0020000000949949026, + 0.7224277572242775 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: mention]", + "key": "ppr:mention", + "variant": "mention", + "recall": [ + 0.9300000071525574, + 0.9185000061988831, + 0.940750002861023 + ], + "all": [ + 0.8569999933242798, + 0.8349999785423279, + 0.8790000081062317 + ], + "mrr": [ + 0.8122603893280029, + 0.7943459749221802, + 0.8300310969352722 + ], + "n_ret": 20.0, + "d_all": [ + 0.4729999899864197, + 0.4419749975204468, + 0.5049999952316284, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.24025000631809235, + 0.22474999725818634, + 0.2562499940395355, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.10189582407474518, + -0.11837921291589737, + -0.08511381596326828, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: entity]", + "key": "ppr:entity", + "variant": "entity", + "recall": [ + 0.8475000262260437, + 0.8322499990463257, + 0.8615062832832336 + ], + "all": [ + 0.6729999780654907, + 0.6430000066757202, + 0.7020000219345093 + ], + "mrr": [ + 0.8205918073654175, + 0.8038225769996643, + 0.8376319408416748 + ], + "n_ret": 20.0, + "d_all": [ + 0.289000004529953, + 0.2590000033378601, + 0.3190000057220459, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.1577499955892563, + 0.14324375987052917, + 0.1720062494277954, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.09356437623500824, + -0.10915587842464447, + -0.07787053287029266, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: knn]", + "key": "ppr:knn", + "variant": "knn", + "recall": [ + 0.7297499775886536, + 0.7135000228881836, + 0.7462562322616577 + ], + "all": [ + 0.4659999907016754, + 0.4350000023841858, + 0.4970000088214874 + ], + "mrr": [ + 0.8446986079216003, + 0.8270958065986633, + 0.8616726994514465 + ], + "n_ret": 20.0, + "d_all": [ + 0.0820000022649765, + 0.05999999865889549, + 0.10400000214576721, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.03999999910593033, + 0.02800000086426735, + 0.0510062500834465, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0694575160741806, + -0.08318424969911575, + -0.05617678910493851, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: mention+entity]", + "key": "ppr:mention+entity", + "variant": "mention+entity", + "recall": [ + 0.9004999995231628, + 0.8870000243186951, + 0.9135000109672546 + ], + "all": [ + 0.8149999976158142, + 0.7900000214576721, + 0.8379999995231628 + ], + "mrr": [ + 0.8552928566932678, + 0.839108407497406, + 0.8717008829116821 + ], + "n_ret": 20.0, + "d_all": [ + 0.4309999942779541, + 0.3959999978542328, + 0.46700000762939453, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.21074999868869781, + 0.19374999403953552, + 0.22925624251365662, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0588633231818676, + -0.07309281826019287, + -0.04472741484642029, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: ctrl_mention]", + "key": "ppr:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.6897500157356262, + 0.6737499833106995, + 0.7047500014305115 + ], + "all": [ + 0.3840000033378601, + 0.3529999852180481, + 0.4129999876022339 + ], + "mrr": [ + 0.912039577960968, + 0.8982824683189392, + 0.9249224066734314 + ], + "n_ret": 20.0, + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_mrr": [ + -0.00211666664108634, + -0.00580083392560482, + 0.0014337500324472785, + 0.2736726327367263 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: ctrl_entity]", + "key": "ppr:ctrl_entity", + "variant": "ctrl_entity", + "recall": [ + 0.6897500157356262, + 0.6737499833106995, + 0.7047500014305115 + ], + "all": [ + 0.3840000033378601, + 0.3529999852180481, + 0.4129999876022339 + ], + "mrr": [ + 0.8370300531387329, + 0.8200353384017944, + 0.8548807501792908 + ], + "n_ret": 20.0, + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_mrr": [ + -0.07712611556053162, + -0.09025240689516068, + -0.06347676366567612, + 9.999000099990002e-05 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: mention]", + "key": "lib_default:mention", + "variant": "mention", + "recall": [ + 0.8912500143051147, + 0.878000020980835, + 0.9047499895095825 + ], + "all": [ + 0.7990000247955322, + 0.7739999890327454, + 0.8230000138282776 + ], + "mrr": [ + 0.9117594957351685, + 0.8978590965270996, + 0.9251335859298706 + ], + "n_ret": 8.440999984741211, + "d_all": [ + 0.41499999165534973, + 0.3779999911785126, + 0.45100000500679016, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.20149999856948853, + 0.18324999511241913, + 0.2202499955892563, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.002396659692749381, + -0.005368917714804411, + 0.0005437714862637222, + 0.11038896110388961 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: entity]", + "key": "lib_default:entity", + "variant": "entity", + "recall": [ + 0.8477500081062317, + 0.8327500224113464, + 0.8619999885559082 + ], + "all": [ + 0.6850000023841858, + 0.656000018119812, + 0.7120000123977661 + ], + "mrr": [ + 0.9114075899124146, + 0.8975058794021606, + 0.924759030342102 + ], + "n_ret": 19.819000244140625, + "d_all": [ + 0.3009999990463257, + 0.26499998569488525, + 0.33500000834465027, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.15800000727176666, + 0.1404999941587448, + 0.1757500022649765, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0027486076578497887, + -0.0057261427864432335, + 0.00036120275035500526, + 0.06819318068193181 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: knn]", + "key": "lib_default:knn", + "variant": "knn", + "recall": [ + 0.7214999794960022, + 0.7055000066757202, + 0.7384999990463257 + ], + "all": [ + 0.4620000123977661, + 0.4320000112056732, + 0.493025004863739 + ], + "mrr": [ + 0.9124353528022766, + 0.8986316323280334, + 0.925529420375824 + ], + "n_ret": 19.992000579833984, + "d_all": [ + 0.07800000160932541, + 0.052000001072883606, + 0.10199999809265137, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.03175000101327896, + 0.017993750050663948, + 0.045249998569488525, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0017208296339958906, + -0.004698171280324459, + 0.001260249875485897, + 0.2546745325467453 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: mention+entity]", + "key": "lib_default:mention+entity", + "variant": "mention+entity", + "recall": [ + 0.8882499933242798, + 0.8744999766349792, + 0.9017500281333923 + ], + "all": [ + 0.7710000276565552, + 0.7450000047683716, + 0.7960000038146973 + ], + "mrr": [ + 0.9115742444992065, + 0.8976725935935974, + 0.924983561038971 + ], + "n_ret": 19.85099983215332, + "d_all": [ + 0.3869999945163727, + 0.35100001096725464, + 0.42100000381469727, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.19850000739097595, + 0.18074999749660492, + 0.2165062427520752, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0025819409638643265, + -0.0055593037977814674, + 0.0004560228262562305, + 0.0903909609039096 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: ctrl_mention]", + "key": "lib_default:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.6079999804496765, + 0.5924999713897705, + 0.623006284236908 + ], + "all": [ + 0.2619999945163727, + 0.23499999940395355, + 0.2879999876022339 + ], + "mrr": [ + 0.9111167192459106, + 0.8971995115280151, + 0.9244008660316467 + ], + "n_ret": 9.717000007629395, + "d_all": [ + -0.12200000137090683, + -0.14300000667572021, + -0.10300000011920929, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.08174999803304672, + -0.09350000321865082, + -0.07050000131130219, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.003039516741409898, + -0.005966195836663246, + 4.250828715157695e-05, + 0.041995800419958006 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: ctrl_entity]", + "key": "lib_default:ctrl_entity", + "variant": "ctrl_entity", + "recall": [ + 0.5987499952316284, + 0.5835000276565552, + 0.6134999990463257 + ], + "all": [ + 0.24500000476837158, + 0.21799999475479126, + 0.2709999978542328 + ], + "mrr": [ + 0.9111167192459106, + 0.8971995115280151, + 0.9244008660316467 + ], + "n_ret": 19.937000274658203, + "d_all": [ + -0.13899999856948853, + -0.16099999845027924, + -0.11900000274181366, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.09099999815225601, + -0.10275625437498093, + -0.0794999971985817, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.003039516741409898, + -0.005966195836663246, + 4.250828715157695e-05, + 0.041995800419958006 + ], + "d_all_holm": 0.0023997600239976003, + "d_recall_holm": 0.0023997600239976003 + } + ] + }, + "subgraph": { + "5": [ + { + "variant": "mention", + "connect_max_hops": 2, + "size": 5.011000156402588, + "recall": [ + 0.8262500166893005, + 0.8105000257492065, + 0.8417500257492065 + ], + "all": [ + 0.6399999856948853, + 0.6100000143051147, + 0.6690250039100647 + ], + "vec_matched_recall": [ + 0.608, + 0.5925, + 0.62300625 + ], + "vec_matched_all": [ + 0.262, + 0.235, + 0.28702499999999986 + ], + "d_all": [ + 0.3779999911785126, + 0.34597498178482056, + 0.4090000092983246, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.21825000643730164, + 0.2029937505722046, + 0.23350000381469727, + 9.999000099990002e-05 + ] + }, + { + "variant": "mention", + "connect_max_hops": 3, + "size": 5.013000011444092, + "recall": [ + 0.8267499804496765, + 0.8109937310218811, + 0.8420000076293945 + ], + "all": [ + 0.640999972820282, + 0.6110000014305115, + 0.6700249910354614 + ], + "vec_matched_recall": [ + 0.608, + 0.5925, + 0.62300625 + ], + "vec_matched_all": [ + 0.262, + 0.235, + 0.28702499999999986 + ], + "d_all": [ + 0.3790000081062317, + 0.34599998593330383, + 0.4100250005722046, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.21875, + 0.20324376225471497, + 0.23399999737739563, + 9.999000099990002e-05 + ] + }, + { + "variant": "entity", + "connect_max_hops": 2, + "size": 5.0980000495910645, + "recall": [ + 0.7070000171661377, + 0.6894999742507935, + 0.7239999771118164 + ], + "all": [ + 0.4399999976158142, + 0.4090000092983246, + 0.4699999988079071 + ], + "vec_matched_recall": [ + 0.6105, + 0.59549375, + 0.625 + ], + "vec_matched_all": [ + 0.265, + 0.238, + 0.291 + ], + "d_all": [ + 0.17499999701976776, + 0.14800000190734863, + 0.20002500712871552, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.09650000184774399, + 0.08250000327825546, + 0.10975000262260437, + 9.999000099990002e-05 + ] + }, + { + "variant": "entity", + "connect_max_hops": 3, + "size": 5.572999954223633, + "recall": [ + 0.7129999995231628, + 0.69574373960495, + 0.7300000190734863 + ], + "all": [ + 0.453000009059906, + 0.4219749867916107, + 0.4830250144004822 + ], + "vec_matched_recall": [ + 0.61575, + 0.6005, + 0.63025625 + ], + "vec_matched_all": [ + 0.271, + 0.243, + 0.297 + ], + "d_all": [ + 0.18199999630451202, + 0.1539749950170517, + 0.20800000429153442, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.09724999964237213, + 0.08275000005960464, + 0.11100000143051147, + 9.999000099990002e-05 + ] + }, + { + "variant": "knn", + "connect_max_hops": 2, + "size": 5.255000114440918, + "recall": [ + 0.6060000061988831, + 0.590499997138977, + 0.6209999918937683 + ], + "all": [ + 0.26600000262260437, + 0.23899999260902405, + 0.2919999957084656 + ], + "vec_matched_recall": [ + 0.6125, + 0.59725, + 0.62775 + ], + "vec_matched_all": [ + 0.268, + 0.241, + 0.29402499999999987 + ], + "d_all": [ + -0.0020000000949949026, + -0.017000000923871994, + 0.012000000104308128, + 0.8872112788721128 + ], + "d_recall": [ + -0.006500000134110451, + -0.014256250113248825, + 0.0010000000474974513, + 0.12288771122887711 + ] + }, + { + "variant": "knn", + "connect_max_hops": 3, + "size": 5.881999969482422, + "recall": [ + 0.6100000143051147, + 0.5945000052452087, + 0.6255000233650208 + ], + "all": [ + 0.2709999978542328, + 0.24400000274181366, + 0.29702499508857727 + ], + "vec_matched_recall": [ + 0.616, + 0.60075, + 0.631 + ], + "vec_matched_all": [ + 0.271, + 0.244, + 0.298 + ], + "d_all": [ + 0.0, + -0.014999999664723873, + 0.014000000432133675, + 1.0 + ], + "d_recall": [ + -0.006000000052154064, + -0.01425000000745058, + 0.0022562500089406967, + 0.18768123187681232 + ] + }, + { + "variant": "mention+entity", + "connect_max_hops": 2, + "size": 5.14900016784668, + "recall": [ + 0.8292499780654907, + 0.8139937520027161, + 0.8442500233650208 + ], + "all": [ + 0.6460000276565552, + 0.6159999966621399, + 0.675000011920929 + ], + "vec_matched_recall": [ + 0.612, + 0.5965, + 0.627 + ], + "vec_matched_all": [ + 0.269, + 0.241975, + 0.295 + ], + "d_all": [ + 0.37700000405311584, + 0.3449999988079071, + 0.40799999237060547, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.2172500044107437, + 0.20200000703334808, + 0.23225000500679016, + 9.999000099990002e-05 + ] + }, + { + "variant": "mention+entity", + "connect_max_hops": 3, + "size": 5.835999965667725, + "recall": [ + 0.8314999938011169, + 0.8159999847412109, + 0.846750020980835 + ], + "all": [ + 0.6520000100135803, + 0.621999979019165, + 0.6809999942779541 + ], + "vec_matched_recall": [ + 0.617, + 0.60125, + 0.63175 + ], + "vec_matched_all": [ + 0.274, + 0.246, + 0.301 + ], + "d_all": [ + 0.3779999911785126, + 0.34599998593330383, + 0.4090000092983246, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.21449999511241913, + 0.1987437605857849, + 0.22949999570846558, + 9.999000099990002e-05 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 2, + "size": 5.001999855041504, + "recall": [ + 0.6079999804496765, + 0.5924999713897705, + 0.623006284236908 + ], + "all": [ + 0.2619999945163727, + 0.23499999940395355, + 0.2879999876022339 + ], + "vec_matched_recall": [ + 0.608, + 0.5925, + 0.62300625 + ], + "vec_matched_all": [ + 0.262, + 0.235, + 0.28702499999999986 + ], + "d_all": [ + 0.0, + -0.003000000026077032, + 0.003000000026077032, + 1.0 + ], + "d_recall": [ + 0.0, + -0.001500000013038516, + 0.001500000013038516, + 1.0 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 3, + "size": 5.001999855041504, + "recall": [ + 0.6079999804496765, + 0.5924999713897705, + 0.623006284236908 + ], + "all": [ + 0.2619999945163727, + 0.23499999940395355, + 0.2879999876022339 + ], + "vec_matched_recall": [ + 0.608, + 0.5925, + 0.62300625 + ], + "vec_matched_all": [ + 0.262, + 0.235, + 0.28702499999999986 + ], + "d_all": [ + 0.0, + -0.003000000026077032, + 0.003000000026077032, + 1.0 + ], + "d_recall": [ + 0.0, + -0.001500000013038516, + 0.001500000013038516, + 1.0 + ] + }, + { + "variant": "ctrl_entity", + "connect_max_hops": 2, + "size": 5.059000015258789, + "recall": [ + 0.6079999804496765, + 0.5924999713897705, + 0.623006284236908 + ], + "all": [ + 0.2619999945163727, + 0.23499999940395355, + 0.2879999876022339 + ], + "vec_matched_recall": [ + 0.60875, + 0.59349375, + 0.624 + ], + "vec_matched_all": [ + 0.263, + 0.236, + 0.289 + ], + "d_all": [ + -0.0010000000474974513, + -0.004000000189989805, + 0.0020000000949949026, + 1.0 + ], + "d_recall": [ + -0.000750000006519258, + -0.002749999985098839, + 0.0010000000474974513, + 0.628937106289371 + ] + }, + { + "variant": "ctrl_entity", + "connect_max_hops": 3, + "size": 6.184000015258789, + "recall": [ + 0.6079999804496765, + 0.5924999713897705, + 0.623006284236908 + ], + "all": [ + 0.2619999945163727, + 0.23499999940395355, + 0.2879999876022339 + ], + "vec_matched_recall": [ + 0.61825, + 0.603, + 0.63300625 + ], + "vec_matched_all": [ + 0.277, + 0.249, + 0.303 + ], + "d_all": [ + -0.014999999664723873, + -0.024000000208616257, + -0.00800000037997961, + 0.00039996000399960006 + ], + "d_recall": [ + -0.010250000283122063, + -0.014999999664723873, + -0.0062500000931322575, + 9.999000099990002e-05 + ] + } + ], + "10": [ + { + "variant": "mention", + "connect_max_hops": 2, + "size": 7.965000152587891, + "recall": [ + 0.890500009059906, + 0.8769999742507935, + 0.9037500023841858 + ], + "all": [ + 0.7960000038146973, + 0.7710000276565552, + 0.8199999928474426 + ], + "vec_matched_recall": [ + 0.63875, + 0.6234937500000001, + 0.6535 + ], + "vec_matched_all": [ + 0.306, + 0.277, + 0.333 + ], + "d_all": [ + 0.49000000953674316, + 0.4569999873638153, + 0.5230000019073486, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.25174999237060547, + 0.23600000143051147, + 0.26800626516342163, + 9.999000099990002e-05 + ] + }, + { + "variant": "mention", + "connect_max_hops": 3, + "size": 7.9710001945495605, + "recall": [ + 0.890500009059906, + 0.8769999742507935, + 0.9037500023841858 + ], + "all": [ + 0.7960000038146973, + 0.7710000276565552, + 0.8199999928474426 + ], + "vec_matched_recall": [ + 0.63875, + 0.6234937500000001, + 0.6535 + ], + "vec_matched_all": [ + 0.306, + 0.277, + 0.333 + ], + "d_all": [ + 0.49000000953674316, + 0.4569999873638153, + 0.5230000019073486, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.25174999237060547, + 0.23600000143051147, + 0.26800626516342163, + 9.999000099990002e-05 + ] + }, + { + "variant": "entity", + "connect_max_hops": 2, + "size": 10.14799976348877, + "recall": [ + 0.8119999766349792, + 0.7957500219345093, + 0.8270000219345093 + ], + "all": [ + 0.6119999885559082, + 0.5819750428199768, + 0.640999972820282 + ], + "vec_matched_recall": [ + 0.652, + 0.636, + 0.667 + ], + "vec_matched_all": [ + 0.325, + 0.297, + 0.352 + ], + "d_all": [ + 0.28700000047683716, + 0.2549999952316284, + 0.31700000166893005, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.1599999964237213, + 0.14474375545978546, + 0.17499999701976776, + 9.999000099990002e-05 + ] + }, + { + "variant": "entity", + "connect_max_hops": 3, + "size": 11.331000328063965, + "recall": [ + 0.815750002861023, + 0.7994999885559082, + 0.8307562470436096 + ], + "all": [ + 0.6200000047683716, + 0.5889999866485596, + 0.6489999890327454 + ], + "vec_matched_recall": [ + 0.658, + 0.64224375, + 0.67275 + ], + "vec_matched_all": [ + 0.334, + 0.305, + 0.361 + ], + "d_all": [ + 0.28600001335144043, + 0.2540000081062317, + 0.3160000145435333, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.1577499955892563, + 0.1420000046491623, + 0.17300623655319214, + 9.999000099990002e-05 + ] + }, + { + "variant": "knn", + "connect_max_hops": 2, + "size": 10.361000061035156, + "recall": [ + 0.6794999837875366, + 0.6627500057220459, + 0.6970000267028809 + ], + "all": [ + 0.4000000059604645, + 0.37097498774528503, + 0.4309999942779541 + ], + "vec_matched_recall": [ + 0.65475, + 0.63925, + 0.6695 + ], + "vec_matched_all": [ + 0.328, + 0.299, + 0.355 + ], + "d_all": [ + 0.07199999690055847, + 0.04699999839067459, + 0.09700000286102295, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.02474999986588955, + 0.01075000036507845, + 0.038750000298023224, + 0.00039996000399960006 + ] + }, + { + "variant": "knn", + "connect_max_hops": 3, + "size": 11.178000450134277, + "recall": [ + 0.6815000176429749, + 0.6647437214851379, + 0.6987500190734863 + ], + "all": [ + 0.4009999930858612, + 0.3720000088214874, + 0.4320000112056732 + ], + "vec_matched_recall": [ + 0.6575, + 0.64224375, + 0.672 + ], + "vec_matched_all": [ + 0.332, + 0.303, + 0.359 + ], + "d_all": [ + 0.0689999982714653, + 0.0430000014603138, + 0.09402499347925186, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.024000000208616257, + 0.009993749670684338, + 0.03799999877810478, + 0.0005999400059994001 + ] + }, + { + "variant": "mention+entity", + "connect_max_hops": 2, + "size": 10.305000305175781, + "recall": [ + 0.8980000019073486, + 0.8849999904632568, + 0.9110000133514404 + ], + "all": [ + 0.8100000023841858, + 0.7860000133514404, + 0.8339999914169312 + ], + "vec_matched_recall": [ + 0.65275, + 0.63675, + 0.66775 + ], + "vec_matched_all": [ + 0.326, + 0.297, + 0.353 + ], + "d_all": [ + 0.48399999737739563, + 0.45100000500679016, + 0.5160249471664429, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.24525000154972076, + 0.22849375009536743, + 0.2617500126361847, + 9.999000099990002e-05 + ] + }, + { + "variant": "mention+entity", + "connect_max_hops": 3, + "size": 11.710000038146973, + "recall": [ + 0.8980000019073486, + 0.8849999904632568, + 0.9110000133514404 + ], + "all": [ + 0.8100000023841858, + 0.7860000133514404, + 0.8339999914169312 + ], + "vec_matched_recall": [ + 0.65875, + 0.64275, + 0.67375 + ], + "vec_matched_all": [ + 0.335, + 0.306, + 0.362 + ], + "d_all": [ + 0.4749999940395355, + 0.44200000166893005, + 0.5080249905586243, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.23925000429153442, + 0.2227499932050705, + 0.2562499940395355, + 9.999000099990002e-05 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 2, + "size": 10.003999710083008, + "recall": [ + 0.6520000100135803, + 0.6362437605857849, + 0.6669999957084656 + ], + "all": [ + 0.32499998807907104, + 0.29600000381469727, + 0.35199999809265137 + ], + "vec_matched_recall": [ + 0.652, + 0.63624375, + 0.667 + ], + "vec_matched_all": [ + 0.325, + 0.296, + 0.352 + ], + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 3, + "size": 10.005999565124512, + "recall": [ + 0.6520000100135803, + 0.6362437605857849, + 0.6669999957084656 + ], + "all": [ + 0.32499998807907104, + 0.29600000381469727, + 0.35199999809265137 + ], + "vec_matched_recall": [ + 0.652, + 0.63624375, + 0.667 + ], + "vec_matched_all": [ + 0.325, + 0.296, + 0.352 + ], + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ] + }, + { + "variant": "ctrl_entity", + "connect_max_hops": 2, + "size": 10.284000396728516, + "recall": [ + 0.6520000100135803, + 0.6362437605857849, + 0.6669999957084656 + ], + "all": [ + 0.32499998807907104, + 0.29600000381469727, + 0.35199999809265137 + ], + "vec_matched_recall": [ + 0.6535, + 0.6375, + 0.66850625 + ], + "vec_matched_all": [ + 0.328, + 0.299, + 0.356 + ], + "d_all": [ + -0.003000000026077032, + -0.007000000216066837, + 0.0, + 0.25137486251374863 + ], + "d_recall": [ + -0.001500000013038516, + -0.0035000001080334187, + 0.0, + 0.25137486251374863 + ] + }, + { + "variant": "ctrl_entity", + "connect_max_hops": 3, + "size": 15.565999984741211, + "recall": [ + 0.6522499918937683, + 0.6364937424659729, + 0.667006254196167 + ], + "all": [ + 0.32499998807907104, + 0.29600000381469727, + 0.35199999809265137 + ], + "vec_matched_recall": [ + 0.67375, + 0.65825, + 0.68875625 + ], + "vec_matched_all": [ + 0.358, + 0.328975, + 0.386 + ], + "d_all": [ + -0.032999999821186066, + -0.04500000178813934, + -0.023000000044703484, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.0215000007301569, + -0.02775000035762787, + -0.01599375158548355, + 9.999000099990002e-05 + ] + } + ], + "20": [ + { + "variant": "mention", + "connect_max_hops": 2, + "size": 19.47100067138672, + "recall": [ + 0.9355000257492065, + 0.9244999885559082, + 0.9457499980926514 + ], + "all": [ + 0.8659999966621399, + 0.8429999947547913, + 0.8870000243186951 + ], + "vec_matched_recall": [ + 0.68925, + 0.673, + 0.70425 + ], + "vec_matched_all": [ + 0.383, + 0.352, + 0.412 + ], + "d_all": [ + 0.4830000102519989, + 0.45100000500679016, + 0.5149999856948853, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.2462500035762787, + 0.23074999451637268, + 0.26249998807907104, + 9.999000099990002e-05 + ] + }, + { + "variant": "mention", + "connect_max_hops": 3, + "size": 19.490999221801758, + "recall": [ + 0.9355000257492065, + 0.9244999885559082, + 0.9457499980926514 + ], + "all": [ + 0.8659999966621399, + 0.8429999947547913, + 0.8870000243186951 + ], + "vec_matched_recall": [ + 0.68925, + 0.673, + 0.70425 + ], + "vec_matched_all": [ + 0.383, + 0.352, + 0.412 + ], + "d_all": [ + 0.4830000102519989, + 0.45100000500679016, + 0.5149999856948853, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.2462500035762787, + 0.23074999451637268, + 0.26249998807907104, + 9.999000099990002e-05 + ] + }, + { + "variant": "entity", + "connect_max_hops": 2, + "size": 20.375999450683594, + "recall": [ + 0.8665000200271606, + 0.8514999747276306, + 0.8815000057220459 + ], + "all": [ + 0.734000027179718, + 0.7070000171661377, + 0.7620000243186951 + ], + "vec_matched_recall": [ + 0.68825, + 0.67225, + 0.7035 + ], + "vec_matched_all": [ + 0.382, + 0.351975, + 0.412 + ], + "d_all": [ + 0.35199999809265137, + 0.31700000166893005, + 0.3880000114440918, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.1782499998807907, + 0.16124999523162842, + 0.19574999809265137, + 9.999000099990002e-05 + ] + }, + { + "variant": "entity", + "connect_max_hops": 3, + "size": 21.58099937438965, + "recall": [ + 0.8672500252723694, + 0.8522499799728394, + 0.8820000290870667 + ], + "all": [ + 0.7360000014305115, + 0.7089999914169312, + 0.7630000114440918 + ], + "vec_matched_recall": [ + 0.69025, + 0.67449375, + 0.7055 + ], + "vec_matched_all": [ + 0.385, + 0.355, + 0.414 + ], + "d_all": [ + 0.35100001096725464, + 0.3160000145435333, + 0.3880000114440918, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.1770000010728836, + 0.15949375927448273, + 0.19449999928474426, + 9.999000099990002e-05 + ] + }, + { + "variant": "knn", + "connect_max_hops": 2, + "size": 22.788999557495117, + "recall": [ + 0.7442499995231628, + 0.7289999723434448, + 0.7602499723434448 + ], + "all": [ + 0.4860000014305115, + 0.4560000002384186, + 0.5170000195503235 + ], + "vec_matched_recall": [ + 0.6925, + 0.67674375, + 0.70725 + ], + "vec_matched_all": [ + 0.387, + 0.357, + 0.416 + ], + "d_all": [ + 0.0989999994635582, + 0.07800000160932541, + 0.11999999731779099, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.051750000566244125, + 0.04074374958872795, + 0.0625, + 9.999000099990002e-05 + ] + }, + { + "variant": "knn", + "connect_max_hops": 3, + "size": 26.92300033569336, + "recall": [ + 0.7482500076293945, + 0.7329999804496765, + 0.7642499804496765 + ], + "all": [ + 0.49000000953674316, + 0.46000000834465027, + 0.5210000276565552 + ], + "vec_matched_recall": [ + 0.696, + 0.68, + 0.71075625 + ], + "vec_matched_all": [ + 0.392, + 0.361975, + 0.421 + ], + "d_all": [ + 0.09799999743700027, + 0.07800000160932541, + 0.11900000274181366, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.05225000157952309, + 0.04125000163912773, + 0.06325624883174896, + 9.999000099990002e-05 + ] + }, + { + "variant": "mention+entity", + "connect_max_hops": 2, + "size": 21.93400001525879, + "recall": [ + 0.940500020980835, + 0.9297500252723694, + 0.9505000114440918 + ], + "all": [ + 0.8759999871253967, + 0.8539999723434448, + 0.8960000276565552 + ], + "vec_matched_recall": [ + 0.69175, + 0.67575, + 0.70675 + ], + "vec_matched_all": [ + 0.386, + 0.355, + 0.415 + ], + "d_all": [ + 0.49000000953674316, + 0.45899999141693115, + 0.5230000019073486, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.2487500011920929, + 0.23374375700950623, + 0.2642500102519989, + 9.999000099990002e-05 + ] + }, + { + "variant": "mention+entity", + "connect_max_hops": 3, + "size": 28.594999313354492, + "recall": [ + 0.9415000081062317, + 0.9309937357902527, + 0.9514999985694885 + ], + "all": [ + 0.8799999952316284, + 0.859000027179718, + 0.8999999761581421 + ], + "vec_matched_recall": [ + 0.69975, + 0.684, + 0.7145 + ], + "vec_matched_all": [ + 0.395, + 0.365, + 0.424 + ], + "d_all": [ + 0.48500001430511475, + 0.45399999618530273, + 0.5170000195503235, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.24175000190734863, + 0.226500004529953, + 0.2574999928474426, + 9.999000099990002e-05 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 2, + "size": 20.03700065612793, + "recall": [ + 0.6897500157356262, + 0.6737499833106995, + 0.7047500014305115 + ], + "all": [ + 0.3840000033378601, + 0.3529999852180481, + 0.4129999876022339 + ], + "vec_matched_recall": [ + 0.68975, + 0.67375, + 0.70475 + ], + "vec_matched_all": [ + 0.384, + 0.353, + 0.413 + ], + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 3, + "size": 20.13599967956543, + "recall": [ + 0.6897500157356262, + 0.6737499833106995, + 0.7047500014305115 + ], + "all": [ + 0.3840000033378601, + 0.3529999852180481, + 0.4129999876022339 + ], + "vec_matched_recall": [ + 0.69, + 0.674, + 0.70475625 + ], + "vec_matched_all": [ + 0.384, + 0.353, + 0.413 + ], + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + -0.0002500000118743628, + -0.000750000006519258, + 0.0, + 1.0 + ] + }, + { + "variant": "ctrl_entity", + "connect_max_hops": 2, + "size": 21.475000381469727, + "recall": [ + 0.6897500157356262, + 0.6737499833106995, + 0.7047500014305115 + ], + "all": [ + 0.3840000033378601, + 0.3529999852180481, + 0.4129999876022339 + ], + "vec_matched_recall": [ + 0.69125, + 0.6755, + 0.70625 + ], + "vec_matched_all": [ + 0.386, + 0.355975, + 0.415 + ], + "d_all": [ + -0.0020000000949949026, + -0.004999999888241291, + 0.0, + 0.49625037496250374 + ], + "d_recall": [ + -0.001500000013038516, + -0.0032500000670552254, + -0.0002500000118743628, + 0.126987301269873 + ] + }, + { + "variant": "ctrl_entity", + "connect_max_hops": 3, + "size": 37.540000915527344, + "recall": [ + 0.6897500157356262, + 0.6737499833106995, + 0.7047500014305115 + ], + "all": [ + 0.3840000033378601, + 0.3529999852180481, + 0.4129999876022339 + ], + "vec_matched_recall": [ + 0.71, + 0.6945, + 0.72500625 + ], + "vec_matched_all": [ + 0.412, + 0.381, + 0.442 + ], + "d_all": [ + -0.02800000086426735, + -0.03902500122785568, + -0.017999999225139618, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.020250000059604645, + -0.026499999687075615, + -0.014750000089406967, + 9.999000099990002e-05 + ] + } + ] + }, + "lib_minus_ppr": { + "mention": { + "5": [ + 0.11999999731779099, + 0.09600000083446503, + 0.14399999380111694, + 9.999000099990002e-05 + ], + "10": [ + 0.008999999612569809, + 0.0020000000949949026, + 0.017000000923871994, + 0.036396360363963605 + ], + "20": [ + 0.008999999612569809, + -0.003000000026077032, + 0.019999999552965164, + 0.16928307169283072 + ] + }, + "entity": { + "5": [ + 0.11699999868869781, + 0.09200000017881393, + 0.14100000262260437, + 9.999000099990002e-05 + ], + "10": [ + 0.05700000002980232, + 0.032999999821186066, + 0.08100000023841858, + 9.999000099990002e-05 + ], + "20": [ + 0.05999999865889549, + 0.03700000047683716, + 0.08399999886751175, + 9.999000099990002e-05 + ] + }, + "knn": { + "5": [ + 0.007000000216066837, + -0.00800000037997961, + 0.02199999988079071, + 0.40855914408559146 + ], + "10": [ + -0.0010000000474974513, + -0.01899999938905239, + 0.017999999225139618, + 1.0 + ], + "20": [ + 0.01899999938905239, + 0.003000000026077032, + 0.03502500057220459, + 0.0263973602639736 + ] + }, + "mention+entity": { + "5": [ + 0.2590000033378601, + 0.22897499799728394, + 0.289000004529953, + 9.999000099990002e-05 + ], + "10": [ + 0.09000000357627869, + 0.07000000029802322, + 0.10899999737739563, + 9.999000099990002e-05 + ], + "20": [ + 0.057999998331069946, + 0.03999999910593033, + 0.07599999755620956, + 9.999000099990002e-05 + ] + } + }, + "breakdown": { + "type": { + "bridge_comparison": { + "n": 221, + "vector": 0.0, + "rrf": 0.013574660755693913, + "rerank": 0.0, + "lib_mh@best": 0.6651583909988403, + "ppr@best": 0.6470588445663452 + }, + "comparison": { + "n": 238, + "vector": 0.7100840210914612, + "rrf": 0.9201680421829224, + "rerank": 0.8151260614395142, + "lib_mh@best": 0.6302521228790283, + "ppr@best": 0.6260504126548767 + }, + "compositional": { + "n": 422, + "vector": 0.2511848211288452, + "rrf": 0.3009478747844696, + "rerank": 0.28199052810668945, + "lib_mh@best": 0.950236976146698, + "ppr@best": 0.921800971031189 + }, + "inference": { + "n": 119, + "vector": 0.42016807198524475, + "rrf": 0.5210084319114685, + "rerank": 0.5210084319114685, + "lib_mh@best": 0.9411764740943909, + "ppr@best": 0.8907563090324402 + } + }, + "hops": { + "2": { + "n": 779, + "vector": 0.41720154881477356, + "rrf": 0.5237483978271484, + "rerank": 0.4813863933086395, + "lib_mh@best": 0.8510911464691162, + "ppr@best": 0.8267009258270264 + }, + "4": { + "n": 221, + "vector": 0.0, + "rrf": 0.013574660755693913, + "rerank": 0.0, + "lib_mh@best": 0.6651583909988403, + "ppr@best": 0.6470588445663452 + } + } + } + }, + "bge": { + "n_test": 1000, + "corpus_size": 9239, + "lib_similarity_search_same_top20_set": "198/200", + "graphs": { + "mention": { + "n_edges_undirected_union": 5383, + "n_edges_by_family_directed_or_undirected": 6115, + "mean_degree": 1.165277627448858, + "median_degree": 1.0, + "max_degree": 30, + "isolated_frac": 0.4217988959844139, + "gold_pairs_directly_linked": 0.44278768233387356, + "gold_pairs_linked_within_2": 0.4447325769854133, + "neighbours_of_gold_that_are_in_same_question_context": 0.42326321090525637 + }, + "entity": { + "n_edges_undirected_union": 30240, + "n_edges_by_family_directed_or_undirected": 30240, + "mean_degree": 6.546163004654184, + "median_degree": 4.0, + "max_degree": 86, + "isolated_frac": 0.21798895984413896, + "gold_pairs_directly_linked": 0.4051863857374392, + "gold_pairs_linked_within_2": 0.426580226904376, + "neighbours_of_gold_that_are_in_same_question_context": 0.11197227702934706 + }, + "knn": { + "n_edges_undirected_union": 34499, + "n_edges_by_family_directed_or_undirected": 34499, + "mean_degree": 7.468124255871848, + "median_degree": 6.0, + "max_degree": 154, + "isolated_frac": 0.0, + "gold_pairs_directly_linked": 0.32220421393841164, + "gold_pairs_linked_within_2": 0.37212317666126415, + "neighbours_of_gold_that_are_in_same_question_context": 0.1272243175430044 + }, + "ctrl_mention": { + "n_edges_undirected_union": 5383, + "n_edges_by_family_directed_or_undirected": 6115, + "mean_degree": 1.165277627448858, + "median_degree": 1.0, + "max_degree": 30, + "isolated_frac": 0.4217988959844139, + "gold_pairs_directly_linked": 0.00032414910858995135, + "gold_pairs_linked_within_2": 0.00032414910858995135, + "neighbours_of_gold_that_are_in_same_question_context": 0.0003952569169960474 + }, + "ctrl_entity": { + "n_edges_undirected_union": 30240, + "n_edges_by_family_directed_or_undirected": 30240, + "mean_degree": 6.546163004654184, + "median_degree": 4.0, + "max_degree": 86, + "isolated_frac": 0.21798895984413896, + "gold_pairs_directly_linked": 0.0012965964343598054, + "gold_pairs_linked_within_2": 0.0074554294975688815, + "neighbours_of_gold_that_are_in_same_question_context": 0.0006118890958802483 + } + }, + "selected": { + "mention": { + "lib": { + "5": { + "hops": 2, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "10": { + "hops": 1, + "seeds": 5, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + } + }, + "ppr": { + "5": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": false + }, + "20": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.0, + "directed_mention": true, + "weighted": false + } + } + }, + "entity": { + "lib": { + "5": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 2, + "seeds": 5, + "decay": 0.7, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + } + }, + "ppr": { + "5": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + }, + "20": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": true, + "weighted": true + } + } + }, + "knn": { + "lib": { + "5": { + "hops": 2, + "seeds": 15, + "decay": 0.9, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 1, + "seeds": 5, + "decay": 0.5, + "blend": 0.0, + "direction": "out", + "weight_field": "w" + }, + "20": { + "hops": 2, + "seeds": 5, + "decay": 0.8, + "blend": 0.0, + "direction": "out", + "weight_field": null + } + }, + "ppr": { + "5": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + }, + "20": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": true + } + } + }, + "ctrl_mention": { + "lib": { + "5": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + } + }, + "ppr": { + "5": { + "alpha": 0.7, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "20": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + } + } + }, + "ctrl_entity": { + "lib": { + "5": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "out", + "weight_field": null + } + }, + "ppr": { + "5": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.0, + "directed_mention": true, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "20": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + } + } + } + }, + "tables": { + "5": [ + { + "label": "Vector top-k (plain)", + "key": "vector", + "variant": null, + "recall": [ + 0.690750002861023, + 0.6757437586784363, + 0.7062562704086304 + ], + "all": [ + 0.382999986410141, + 0.3529749810695648, + 0.41200000047683716 + ], + "mrr": [ + 0.9747833013534546, + 0.9673312306404114, + 0.9816687107086182 + ], + "n_ret": 5.0 + }, + { + "label": "BM25", + "key": "bm25", + "variant": null, + "recall": [ + 0.6635000109672546, + 0.6489999890327454, + 0.6787499785423279 + ], + "all": [ + 0.33799999952316284, + 0.3109999895095825, + 0.367000013589859 + ], + "mrr": [ + 0.9372166991233826, + 0.9249820113182068, + 0.9491349458694458 + ], + "n_ret": 5.0, + "d_all": [ + -0.04500000178813934, + -0.06700000166893005, + -0.024000000208616257, + 0.00019998000199980003 + ], + "d_recall": [ + -0.027249999344348907, + -0.038750000298023224, + -0.016249999403953552, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.037566665560007095, + -0.05000458285212517, + -0.02448291704058647, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0021997800219978004, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "Hybrid vector+BM25 (RRF)", + "key": "rrf", + "variant": null, + "recall": [ + 0.6984999775886536, + 0.6832500100135803, + 0.714006245136261 + ], + "all": [ + 0.3869999945163727, + 0.3580000102519989, + 0.4180000126361847 + ], + "mrr": [ + 0.9679166078567505, + 0.9596666097640991, + 0.9753354787826538 + ], + "n_ret": 5.0, + "d_all": [ + 0.004000000189989805, + -0.010999999940395355, + 0.01899999938905239, + 0.6948305169483052 + ], + "d_recall": [ + 0.00774999987334013, + -0.0002562500012572855, + 0.01575000025331974, + 0.07489251074892511 + ], + "d_mrr": [ + -0.006866666488349438, + -0.014702083542943, + 0.001084166462533176, + 0.08899110088991101 + ], + "d_all_holm": 1.0, + "d_recall_holm": 0.5242475752424758 + }, + { + "label": "Vector top-30 + cross-encoder", + "key": "rerank", + "variant": null, + "recall": [ + 0.7017499804496765, + 0.6869999766349792, + 0.7170000076293945 + ], + "all": [ + 0.39399999380111694, + 0.36399999260902405, + 0.42399999499320984 + ], + "mrr": [ + 0.9735333323478699, + 0.9664316773414612, + 0.9803333878517151 + ], + "n_ret": 5.0, + "d_all": [ + 0.010999999940395355, + -0.007000000216066837, + 0.029999999329447746, + 0.2886711328867113 + ], + "d_recall": [ + 0.010999999940395355, + 0.0010000000474974513, + 0.020999999716877937, + 0.0292970702929707 + ], + "d_mrr": [ + -0.001249999739229679, + -0.010100833140313625, + 0.007967500016093254, + 0.7869213078692131 + ], + "d_all_holm": 1.0, + "d_recall_holm": 0.2343765623437656 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph chosen on dev: mention]", + "key": "lib_mh@best", + "variant": "mention", + "recall": [ + 0.9135000109672546, + 0.9027437567710876, + 0.9242500066757202 + ], + "all": [ + 0.777999997138977, + 0.753000020980835, + 0.8040000200271606 + ], + "mrr": [ + 0.9747833013534546, + 0.9668149948120117, + 0.9819999933242798 + ], + "n_ret": 5.0, + "d_all": [ + 0.39500001072883606, + 0.3630000054836273, + 0.42500001192092896, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.2227499932050705, + 0.20724999904632568, + 0.23800000548362732, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.0, + -0.0035000001080334187, + 0.0035000001080334187, + 1.0 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention]", + "key": "ppr@best", + "variant": "mention", + "recall": [ + 0.8144999742507935, + 0.7994999885559082, + 0.8297500014305115 + ], + "all": [ + 0.5889999866485596, + 0.5590000152587891, + 0.6190249919891357 + ], + "mrr": [ + 0.8317166566848755, + 0.8140999674797058, + 0.8483670949935913 + ], + "n_ret": 5.0, + "d_all": [ + 0.20600000023841858, + 0.17499999701976776, + 0.23800000548362732, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.1237500011920929, + 0.10774999856948853, + 0.14025624096393585, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.1430666446685791, + -0.16093415021896362, + -0.1268279105424881, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph chosen on dev: mention]", + "key": "lib_default@best", + "variant": "mention", + "recall": [ + 0.7459999918937683, + 0.7315000295639038, + 0.761006236076355 + ], + "all": [ + 0.47699999809265137, + 0.44699999690055847, + 0.5080000162124634 + ], + "mrr": [ + 0.9747833013534546, + 0.9668149948120117, + 0.9819999933242798 + ], + "n_ret": 5.0, + "d_all": [ + 0.09399999678134918, + 0.07699999958276749, + 0.1130249947309494, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.05525000020861626, + 0.04650000110268593, + 0.06499999761581421, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.0, + -0.0035000001080334187, + 0.0035000001080334187, + 1.0 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: mention]", + "key": "lib_mh:mention", + "variant": "mention", + "recall": [ + 0.9135000109672546, + 0.9027437567710876, + 0.9242500066757202 + ], + "all": [ + 0.777999997138977, + 0.753000020980835, + 0.8040000200271606 + ], + "mrr": [ + 0.9747833013534546, + 0.9668149948120117, + 0.9819999933242798 + ], + "n_ret": 5.0, + "d_all": [ + 0.39500001072883606, + 0.3630000054836273, + 0.42500001192092896, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.2227499932050705, + 0.20724999904632568, + 0.23800000548362732, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.0, + -0.0035000001080334187, + 0.0035000001080334187, + 1.0 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: entity]", + "key": "lib_mh:entity", + "variant": "entity", + "recall": [ + 0.78125, + 0.7664999961853027, + 0.796999990940094 + ], + "all": [ + 0.5370000004768372, + 0.5059999823570251, + 0.5680000185966492 + ], + "mrr": [ + 0.9747833013534546, + 0.9668149948120117, + 0.9819999933242798 + ], + "n_ret": 5.0, + "d_all": [ + 0.15399999916553497, + 0.12999999523162842, + 0.1780250072479248, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.09049999713897705, + 0.07850000262260437, + 0.10275000333786011, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.0, + -0.0035000001080334187, + 0.0035000001080334187, + 1.0 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: knn]", + "key": "lib_mh:knn", + "variant": "knn", + "recall": [ + 0.7289999723434448, + 0.7129999995231628, + 0.7442499995231628 + ], + "all": [ + 0.45500001311302185, + 0.4249750077724457, + 0.48500001430511475 + ], + "mrr": [ + 0.9747833013534546, + 0.9668149948120117, + 0.9819999933242798 + ], + "n_ret": 5.0, + "d_all": [ + 0.07199999690055847, + 0.052000001072883606, + 0.09200000017881393, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.03824999928474426, + 0.02800000086426735, + 0.048250000923871994, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.0, + -0.0035000001080334187, + 0.0035000001080334187, + 1.0 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: ctrl_mention]", + "key": "lib_mh:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.6912500262260437, + 0.6759937405586243, + 0.7070000171661377 + ], + "all": [ + 0.3840000033378601, + 0.3540000021457672, + 0.4129999876022339 + ], + "mrr": [ + 0.9747833013534546, + 0.9668149948120117, + 0.9819999933242798 + ], + "n_ret": 5.0, + "d_all": [ + 0.0010000000474974513, + 0.0, + 0.003000000026077032, + 1.0 + ], + "d_recall": [ + 0.0005000000237487257, + 0.0, + 0.001500000013038516, + 1.0 + ], + "d_mrr": [ + 0.0, + -0.0035000001080334187, + 0.0035000001080334187, + 1.0 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: ctrl_entity]", + "key": "lib_mh:ctrl_entity", + "variant": "ctrl_entity", + "recall": [ + 0.6912500262260437, + 0.6759937405586243, + 0.7070000171661377 + ], + "all": [ + 0.3840000033378601, + 0.3540000021457672, + 0.4129999876022339 + ], + "mrr": [ + 0.9747833013534546, + 0.9668149948120117, + 0.9819999933242798 + ], + "n_ret": 5.0, + "d_all": [ + 0.0010000000474974513, + 0.0, + 0.003000000026077032, + 1.0 + ], + "d_recall": [ + 0.0005000000237487257, + 0.0, + 0.001500000013038516, + 1.0 + ], + "d_mrr": [ + 0.0, + -0.0035000001080334187, + 0.0035000001080334187, + 1.0 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: mention]", + "key": "ppr:mention", + "variant": "mention", + "recall": [ + 0.8144999742507935, + 0.7994999885559082, + 0.8297500014305115 + ], + "all": [ + 0.5889999866485596, + 0.5590000152587891, + 0.6190249919891357 + ], + "mrr": [ + 0.8317166566848755, + 0.8140999674797058, + 0.8483670949935913 + ], + "n_ret": 5.0, + "d_all": [ + 0.20600000023841858, + 0.17499999701976776, + 0.23800000548362732, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.1237500011920929, + 0.10774999856948853, + 0.14025624096393585, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.1430666446685791, + -0.16093415021896362, + -0.1268279105424881, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: entity]", + "key": "ppr:entity", + "variant": "entity", + "recall": [ + 0.7287499904632568, + 0.7139937281608582, + 0.7442499995231628 + ], + "all": [ + 0.4350000023841858, + 0.4050000011920929, + 0.4659999907016754 + ], + "mrr": [ + 0.8079000115394592, + 0.7905983328819275, + 0.8256517052650452 + ], + "n_ret": 5.0, + "d_all": [ + 0.052000001072883606, + 0.032999999821186066, + 0.07100000232458115, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.03799999877810478, + 0.027499999850988388, + 0.04800000041723251, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.16688333451747894, + -0.1846054047346115, + -0.14861458539962769, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: knn]", + "key": "ppr:knn", + "variant": "knn", + "recall": [ + 0.6909999847412109, + 0.6757500171661377, + 0.7067562341690063 + ], + "all": [ + 0.3869999945163727, + 0.3580000102519989, + 0.41702499985694885 + ], + "mrr": [ + 0.9536499977111816, + 0.9435654282569885, + 0.9635008573532104 + ], + "n_ret": 5.0, + "d_all": [ + 0.004000000189989805, + -0.010999999940395355, + 0.01899999938905239, + 0.6868313168683132 + ], + "d_recall": [ + 0.0002500000118743628, + -0.007249999791383743, + 0.00774999987334013, + 1.0 + ], + "d_mrr": [ + -0.021133333444595337, + -0.029700415208935738, + -0.013179999776184559, + 9.999000099990002e-05 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: ctrl_mention]", + "key": "ppr:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.6875, + 0.6725000143051147, + 0.7024999856948853 + ], + "all": [ + 0.3779999911785126, + 0.34997498989105225, + 0.40700000524520874 + ], + "mrr": [ + 0.9655999541282654, + 0.9570167064666748, + 0.9737833142280579 + ], + "n_ret": 5.0, + "d_all": [ + -0.004999999888241291, + -0.013000000268220901, + 0.003000000026077032, + 0.35666433356664334 + ], + "d_recall": [ + -0.0032500000670552254, + -0.007499999832361937, + 0.0010000000474974513, + 0.1795820417958204 + ], + "d_mrr": [ + -0.00918333325535059, + -0.014716665260493755, + -0.0035487499553710222, + 0.0007999200079992001 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: ctrl_entity]", + "key": "ppr:ctrl_entity", + "variant": "ctrl_entity", + "recall": [ + 0.690750002861023, + 0.6757437586784363, + 0.7062562704086304 + ], + "all": [ + 0.382999986410141, + 0.3529749810695648, + 0.41200000047683716 + ], + "mrr": [ + 0.9018666744232178, + 0.8886328935623169, + 0.9150021076202393 + ], + "n_ret": 5.0, + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_mrr": [ + -0.0729166641831398, + -0.08568374812602997, + -0.060032498091459274, + 9.999000099990002e-05 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: mention]", + "key": "lib_default:mention", + "variant": "mention", + "recall": [ + 0.7459999918937683, + 0.7315000295639038, + 0.761006236076355 + ], + "all": [ + 0.47699999809265137, + 0.44699999690055847, + 0.5080000162124634 + ], + "mrr": [ + 0.9747833013534546, + 0.9668149948120117, + 0.9819999933242798 + ], + "n_ret": 5.0, + "d_all": [ + 0.09399999678134918, + 0.07699999958276749, + 0.1130249947309494, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.05525000020861626, + 0.04650000110268593, + 0.06499999761581421, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.0, + -0.0035000001080334187, + 0.0035000001080334187, + 1.0 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: entity]", + "key": "lib_default:entity", + "variant": "entity", + "recall": [ + 0.7192500233650208, + 0.7042499780654907, + 0.7345062494277954 + ], + "all": [ + 0.42899999022483826, + 0.39899998903274536, + 0.46000000834465027 + ], + "mrr": [ + 0.9747833013534546, + 0.9668149948120117, + 0.9819999933242798 + ], + "n_ret": 5.0, + "d_all": [ + 0.04600000008940697, + 0.03099999949336052, + 0.06199999898672104, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.02850000001490116, + 0.020749999210238457, + 0.03674999997019768, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.0, + -0.0035000001080334187, + 0.0035000001080334187, + 1.0 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: knn]", + "key": "lib_default:knn", + "variant": "knn", + "recall": [ + 0.703499972820282, + 0.6875, + 0.71875 + ], + "all": [ + 0.4059999883174896, + 0.37599998712539673, + 0.4359999895095825 + ], + "mrr": [ + 0.9747833013534546, + 0.9668149948120117, + 0.9819999933242798 + ], + "n_ret": 5.0, + "d_all": [ + 0.023000000044703484, + 0.009999999776482582, + 0.03700000047683716, + 0.0011998800119988001 + ], + "d_recall": [ + 0.012749999761581421, + 0.006000000052154064, + 0.019999999552965164, + 0.0006999300069993001 + ], + "d_mrr": [ + 0.0, + -0.0035000001080334187, + 0.0035000001080334187, + 1.0 + ], + "d_all_holm": 0.010798920107989201, + "d_recall_holm": 0.006999300069993001 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: ctrl_mention]", + "key": "lib_default:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.6902499794960022, + 0.675000011920929, + 0.7057499885559082 + ], + "all": [ + 0.38199999928474426, + 0.35199999809265137, + 0.41100001335144043 + ], + "mrr": [ + 0.9747833013534546, + 0.9668149948120117, + 0.9819999933242798 + ], + "n_ret": 5.0, + "d_all": [ + -0.0010000000474974513, + -0.004000000189989805, + 0.0020000000949949026, + 1.0 + ], + "d_recall": [ + -0.0005000000237487257, + -0.0020000000949949026, + 0.0010000000474974513, + 1.0 + ], + "d_mrr": [ + 0.0, + -0.0035000001080334187, + 0.0035000001080334187, + 1.0 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: ctrl_entity]", + "key": "lib_default:ctrl_entity", + "variant": "ctrl_entity", + "recall": [ + 0.6847500205039978, + 0.6697499752044678, + 0.699999988079071 + ], + "all": [ + 0.3709999918937683, + 0.3409999907016754, + 0.39902499318122864 + ], + "mrr": [ + 0.9747833013534546, + 0.9668149948120117, + 0.9819999933242798 + ], + "n_ret": 5.0, + "d_all": [ + -0.012000000104308128, + -0.01899999938905239, + -0.006000000052154064, + 0.0007999200079992001 + ], + "d_recall": [ + -0.006000000052154064, + -0.009499999694526196, + -0.003000000026077032, + 0.0007999200079992001 + ], + "d_mrr": [ + 0.0, + -0.0035000001080334187, + 0.0035000001080334187, + 1.0 + ], + "d_all_holm": 0.007999200079992, + "d_recall_holm": 0.007199280071992801 + } + ], + "10": [ + { + "label": "Vector top-k (plain)", + "key": "vector", + "variant": null, + "recall": [ + 0.7229999899864197, + 0.7074999809265137, + 0.7382562756538391 + ], + "all": [ + 0.4390000104904175, + 0.4090000092983246, + 0.4690000116825104 + ], + "mrr": [ + 0.9749499559402466, + 0.9675329327583313, + 0.9817500114440918 + ], + "n_ret": 10.0 + }, + { + "label": "BM25", + "key": "bm25", + "variant": null, + "recall": [ + 0.7122499942779541, + 0.6972500085830688, + 0.7275000214576721 + ], + "all": [ + 0.41499999165534973, + 0.38499999046325684, + 0.4449999928474426 + ], + "mrr": [ + 0.9382262229919434, + 0.9262092709541321, + 0.9497938752174377 + ], + "n_ret": 10.0, + "d_all": [ + -0.024000000208616257, + -0.043025001883506775, + -0.004000000189989805, + 0.019098090190980903 + ], + "d_recall": [ + -0.01075000036507845, + -0.020749999210238457, + 0.0, + 0.047495250474952504 + ], + "d_mrr": [ + -0.03672380745410919, + -0.04887839034199715, + -0.02356666512787342, + 9.999000099990002e-05 + ], + "d_all_holm": 0.11458854114588542, + "d_recall_holm": 0.23747625237476253 + }, + { + "label": "Hybrid vector+BM25 (RRF)", + "key": "rrf", + "variant": null, + "recall": [ + 0.7377499938011169, + 0.7222499847412109, + 0.7524999976158142 + ], + "all": [ + 0.4560000002384186, + 0.42399999499320984, + 0.48500001430511475 + ], + "mrr": [ + 0.9681944847106934, + 0.9599999785423279, + 0.9755284786224365 + ], + "n_ret": 10.0, + "d_all": [ + 0.017000000923871994, + 0.0010000000474974513, + 0.03200000151991844, + 0.045995400459954004 + ], + "d_recall": [ + 0.014750000089406967, + 0.006500000134110451, + 0.023250000551342964, + 0.000999900009999 + ], + "d_mrr": [ + -0.006755555048584938, + -0.014452638104557991, + 0.001061527873389423, + 0.08959104089591041 + ], + "d_all_holm": 0.22997700229977003, + "d_recall_holm": 0.006999300069993001 + }, + { + "label": "Vector top-30 + cross-encoder", + "key": "rerank", + "variant": null, + "recall": [ + 0.7365000247955322, + 0.7210000157356262, + 0.7522500157356262 + ], + "all": [ + 0.46000000834465027, + 0.4299750030040741, + 0.49000000953674316 + ], + "mrr": [ + 0.9735333323478699, + 0.9664316773414612, + 0.9803333878517151 + ], + "n_ret": 10.0, + "d_all": [ + 0.020999999716877937, + 0.006000000052154064, + 0.035999998450279236, + 0.010698930106989302 + ], + "d_recall": [ + 0.013500000350177288, + 0.005499999970197678, + 0.0215000007301569, + 0.0012998700129987 + ], + "d_mrr": [ + -0.0014166665496304631, + -0.010184166952967644, + 0.007633750326931477, + 0.7594240575942406 + ], + "d_all_holm": 0.07489251074892511, + "d_recall_holm": 0.0077992200779922 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph chosen on dev: mention]", + "key": "lib_mh@best", + "variant": "mention", + "recall": [ + 0.9739999771118164, + 0.9669937491416931, + 0.9807562828063965 + ], + "all": [ + 0.9459999799728394, + 0.9319999814033508, + 0.9599999785423279 + ], + "mrr": [ + 0.9749261736869812, + 0.9669857025146484, + 0.9820101261138916 + ], + "n_ret": 8.145999908447266, + "d_all": [ + 0.5070000290870667, + 0.4749999940395355, + 0.5389999747276306, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.25099998712539673, + 0.23524999618530273, + 0.2667500078678131, + 9.999000099990002e-05 + ], + "d_mrr": [ + -2.3809670892660506e-05, + -0.003357142908498645, + 0.0034523806534707546, + 0.891010898910109 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention]", + "key": "ppr@best", + "variant": "mention", + "recall": [ + 0.9702500104904175, + 0.9629999995231628, + 0.9772562384605408 + ], + "all": [ + 0.9279999732971191, + 0.9120000004768372, + 0.9440000057220459 + ], + "mrr": [ + 0.8695666790008545, + 0.8550808429718018, + 0.8848166465759277 + ], + "n_ret": 10.0, + "d_all": [ + 0.48899999260902405, + 0.4580000042915344, + 0.5210000276565552, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.2472500056028366, + 0.23250000178813934, + 0.2627499997615814, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.10538332909345627, + -0.12013541907072067, + -0.09062833338975906, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph chosen on dev: mention]", + "key": "lib_default@best", + "variant": "mention", + "recall": [ + 0.9739999771118164, + 0.9669937491416931, + 0.9807562828063965 + ], + "all": [ + 0.9459999799728394, + 0.9319999814033508, + 0.9599999785423279 + ], + "mrr": [ + 0.9749261736869812, + 0.9669857025146484, + 0.9820101261138916 + ], + "n_ret": 8.57699966430664, + "d_all": [ + 0.5070000290870667, + 0.4749999940395355, + 0.5389999747276306, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.25099998712539673, + 0.23524999618530273, + 0.2667500078678131, + 9.999000099990002e-05 + ], + "d_mrr": [ + -2.3809670892660506e-05, + -0.003357142908498645, + 0.0034523806534707546, + 0.891010898910109 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: mention]", + "key": "lib_mh:mention", + "variant": "mention", + "recall": [ + 0.9739999771118164, + 0.9669937491416931, + 0.9807562828063965 + ], + "all": [ + 0.9459999799728394, + 0.9319999814033508, + 0.9599999785423279 + ], + "mrr": [ + 0.9749261736869812, + 0.9669857025146484, + 0.9820101261138916 + ], + "n_ret": 8.145999908447266, + "d_all": [ + 0.5070000290870667, + 0.4749999940395355, + 0.5389999747276306, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.25099998712539673, + 0.23524999618530273, + 0.2667500078678131, + 9.999000099990002e-05 + ], + "d_mrr": [ + -2.3809670892660506e-05, + -0.003357142908498645, + 0.0034523806534707546, + 0.891010898910109 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: entity]", + "key": "lib_mh:entity", + "variant": "entity", + "recall": [ + 0.8960000276565552, + 0.8847500085830688, + 0.9075000286102295 + ], + "all": [ + 0.7419999837875366, + 0.7160000205039978, + 0.7680000066757202 + ], + "mrr": [ + 0.9751166701316833, + 0.967366635799408, + 0.9820833802223206 + ], + "n_ret": 9.98900032043457, + "d_all": [ + 0.30300000309944153, + 0.27300000190734863, + 0.33000001311302185, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.17299999296665192, + 0.15800000727176666, + 0.18725000321865082, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.0001666667521931231, + -0.0031666664872318506, + 0.003666666569188237, + 0.8811118888111189 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: knn]", + "key": "lib_mh:knn", + "variant": "knn", + "recall": [ + 0.8385000228881836, + 0.8242499828338623, + 0.8520062565803528 + ], + "all": [ + 0.6389999985694885, + 0.609000027179718, + 0.6669999957084656 + ], + "mrr": [ + 0.9749499559402466, + 0.9670658111572266, + 0.9820333123207092 + ], + "n_ret": 9.996000289916992, + "d_all": [ + 0.20000000298023224, + 0.17497499287128448, + 0.22499999403953552, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.11550000309944153, + 0.10225000232458115, + 0.12825624644756317, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.0, + -0.003333333181217313, + 0.0035000001080334187, + 1.0 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: ctrl_mention]", + "key": "lib_mh:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.722000002861023, + 0.7067499756813049, + 0.7367562651634216 + ], + "all": [ + 0.43700000643730164, + 0.40700000524520874, + 0.46700000762939453 + ], + "mrr": [ + 0.9749499559402466, + 0.9668992161750793, + 0.9820833206176758 + ], + "n_ret": 10.0, + "d_all": [ + -0.0020000000949949026, + -0.004999999888241291, + 0.0, + 0.49725027497250274 + ], + "d_recall": [ + -0.0010000000474974513, + -0.0024999999441206455, + 0.0, + 0.49725027497250274 + ], + "d_mrr": [ + 0.0, + -0.0035000001080334187, + 0.0035000001080334187, + 1.0 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: ctrl_entity]", + "key": "lib_mh:ctrl_entity", + "variant": "ctrl_entity", + "recall": [ + 0.7225000262260437, + 0.7070000171661377, + 0.737500011920929 + ], + "all": [ + 0.43799999356269836, + 0.40799999237060547, + 0.46799999475479126 + ], + "mrr": [ + 0.9749499559402466, + 0.9668992161750793, + 0.9820833206176758 + ], + "n_ret": 10.0, + "d_all": [ + -0.0010000000474974513, + -0.003000000026077032, + 0.0, + 1.0 + ], + "d_recall": [ + -0.0005000000237487257, + -0.001500000013038516, + 0.0, + 1.0 + ], + "d_mrr": [ + 0.0, + -0.0035000001080334187, + 0.0035000001080334187, + 1.0 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: mention]", + "key": "ppr:mention", + "variant": "mention", + "recall": [ + 0.9702500104904175, + 0.9629999995231628, + 0.9772562384605408 + ], + "all": [ + 0.9279999732971191, + 0.9120000004768372, + 0.9440000057220459 + ], + "mrr": [ + 0.8695666790008545, + 0.8550808429718018, + 0.8848166465759277 + ], + "n_ret": 10.0, + "d_all": [ + 0.48899999260902405, + 0.4580000042915344, + 0.5210000276565552, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.2472500056028366, + 0.23250000178813934, + 0.2627499997615814, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.10538332909345627, + -0.12013541907072067, + -0.09062833338975906, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: entity]", + "key": "ppr:entity", + "variant": "entity", + "recall": [ + 0.8740000128746033, + 0.8614937663078308, + 0.8865000009536743 + ], + "all": [ + 0.6970000267028809, + 0.6700000166893005, + 0.7260000109672546 + ], + "mrr": [ + 0.9202666878700256, + 0.9078662395477295, + 0.9322345852851868 + ], + "n_ret": 10.0, + "d_all": [ + 0.257999986410141, + 0.23000000417232513, + 0.28600001335144043, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.1509999930858612, + 0.1367499977350235, + 0.1652500033378601, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.05468333512544632, + -0.06760041415691376, + -0.04238249734044075, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: knn]", + "key": "ppr:knn", + "variant": "knn", + "recall": [ + 0.8125, + 0.7982500195503235, + 0.8259999752044678 + ], + "all": [ + 0.5849999785423279, + 0.5550000071525574, + 0.6140000224113464 + ], + "mrr": [ + 0.9193166494369507, + 0.9074954390525818, + 0.931868314743042 + ], + "n_ret": 10.0, + "d_all": [ + 0.1459999978542328, + 0.12300000339746475, + 0.1679999977350235, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.08950000256299973, + 0.07699999958276749, + 0.10175000131130219, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.05563333258032799, + -0.06746874749660492, + -0.04396583139896393, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: ctrl_mention]", + "key": "ppr:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.7229999899864197, + 0.7074999809265137, + 0.7382562756538391 + ], + "all": [ + 0.4390000104904175, + 0.4090000092983246, + 0.4690000116825104 + ], + "mrr": [ + 0.9734500050544739, + 0.9654000401496887, + 0.9806166291236877 + ], + "n_ret": 10.0, + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_mrr": [ + -0.0014999998966231942, + -0.00549999950453639, + 0.002333333482965827, + 0.4694530546945305 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: ctrl_entity]", + "key": "ppr:ctrl_entity", + "variant": "ctrl_entity", + "recall": [ + 0.7229999899864197, + 0.7074999809265137, + 0.7379999756813049 + ], + "all": [ + 0.4390000104904175, + 0.4089750051498413, + 0.4699999988079071 + ], + "mrr": [ + 0.9742500185966492, + 0.9666666984558105, + 0.981333315372467 + ], + "n_ret": 10.0, + "d_all": [ + 0.0, + -0.003000000026077032, + 0.003000000026077032, + 1.0 + ], + "d_recall": [ + 0.0, + -0.001500000013038516, + 0.001500000013038516, + 1.0 + ], + "d_mrr": [ + -0.0007000000332482159, + -0.005534583702683449, + 0.0041841669008135796, + 0.777922207779222 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: mention]", + "key": "lib_default:mention", + "variant": "mention", + "recall": [ + 0.9739999771118164, + 0.9669937491416931, + 0.9807562828063965 + ], + "all": [ + 0.9459999799728394, + 0.9319999814033508, + 0.9599999785423279 + ], + "mrr": [ + 0.9749261736869812, + 0.9669857025146484, + 0.9820101261138916 + ], + "n_ret": 8.57699966430664, + "d_all": [ + 0.5070000290870667, + 0.4749999940395355, + 0.5389999747276306, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.25099998712539673, + 0.23524999618530273, + 0.2667500078678131, + 9.999000099990002e-05 + ], + "d_mrr": [ + -2.3809670892660506e-05, + -0.003357142908498645, + 0.0034523806534707546, + 0.891010898910109 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: entity]", + "key": "lib_default:entity", + "variant": "entity", + "recall": [ + 0.871999979019165, + 0.8587437272071838, + 0.8842499852180481 + ], + "all": [ + 0.6930000185966492, + 0.6639999747276306, + 0.7200250029563904 + ], + "mrr": [ + 0.9749499559402466, + 0.967033326625824, + 0.9820670485496521 + ], + "n_ret": 9.98900032043457, + "d_all": [ + 0.2540000081062317, + 0.2240000069141388, + 0.28299999237060547, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.14900000393390656, + 0.13349999487400055, + 0.16349999606609344, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.0, + -0.003333333181217313, + 0.0035000001080334187, + 1.0 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: knn]", + "key": "lib_default:knn", + "variant": "knn", + "recall": [ + 0.8267499804496765, + 0.8125, + 0.8402562737464905 + ], + "all": [ + 0.6140000224113464, + 0.5839750170707703, + 0.6430000066757202 + ], + "mrr": [ + 0.9747833013534546, + 0.9668149948120117, + 0.9819999933242798 + ], + "n_ret": 10.0, + "d_all": [ + 0.17499999701976776, + 0.15000000596046448, + 0.19900000095367432, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.10374999791383743, + 0.09074375033378601, + 0.11675625294446945, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0001666667521931231, + -0.0035000001080334187, + 0.003333333181217313, + 0.8875112488751125 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: ctrl_mention]", + "key": "lib_default:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.6912500262260437, + 0.6759937405586243, + 0.7070000171661377 + ], + "all": [ + 0.3840000033378601, + 0.3540000021457672, + 0.4129999876022339 + ], + "mrr": [ + 0.9747833013534546, + 0.9668149948120117, + 0.9819999933242798 + ], + "n_ret": 8.651000022888184, + "d_all": [ + -0.054999999701976776, + -0.0689999982714653, + -0.04100000113248825, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.03175000101327896, + -0.039000000804662704, + -0.02449999935925007, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0001666667521931231, + -0.0035000001080334187, + 0.003333333181217313, + 0.8875112488751125 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: ctrl_entity]", + "key": "lib_default:ctrl_entity", + "variant": "ctrl_entity", + "recall": [ + 0.6897500157356262, + 0.6747499704360962, + 0.7052500247955322 + ], + "all": [ + 0.38100001215934753, + 0.3519749939441681, + 0.4100250005722046 + ], + "mrr": [ + 0.9747833013534546, + 0.9668149948120117, + 0.9819999933242798 + ], + "n_ret": 9.994000434875488, + "d_all": [ + -0.057999998331069946, + -0.0729999989271164, + -0.0430000014603138, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.033250000327825546, + -0.04075000062584877, + -0.025750000029802322, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0001666667521931231, + -0.0035000001080334187, + 0.003333333181217313, + 0.8875112488751125 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + } + ], + "20": [ + { + "label": "Vector top-k (plain)", + "key": "vector", + "variant": null, + "recall": [ + 0.7490000128746033, + 0.734250009059906, + 0.7642499804496765 + ], + "all": [ + 0.4819999933242798, + 0.4519999921321869, + 0.512024998664856 + ], + "mrr": [ + 0.9750983119010925, + 0.9676762223243713, + 0.9818168878555298 + ], + "n_ret": 20.0 + }, + { + "label": "BM25", + "key": "bm25", + "variant": null, + "recall": [ + 0.7427499890327454, + 0.7279937267303467, + 0.7584999799728394 + ], + "all": [ + 0.46700000762939453, + 0.4379749894142151, + 0.49799999594688416 + ], + "mrr": [ + 0.9385273456573486, + 0.9265788793563843, + 0.9500040411949158 + ], + "n_ret": 20.0, + "d_all": [ + -0.014999999664723873, + -0.032999999821186066, + 0.0020000000949949026, + 0.1141885811418858 + ], + "d_recall": [ + -0.0062500000931322575, + -0.015250000171363354, + 0.0027562500908970833, + 0.19388061193880612 + ], + "d_mrr": [ + -0.036571044474840164, + -0.048798948526382446, + -0.023603303357958794, + 9.999000099990002e-05 + ], + "d_all_holm": 0.6851314868513149, + "d_recall_holm": 0.9694030596940306 + }, + { + "label": "Hybrid vector+BM25 (RRF)", + "key": "rrf", + "variant": null, + "recall": [ + 0.7582499980926514, + 0.7429999709129333, + 0.7732499837875366 + ], + "all": [ + 0.49399998784065247, + 0.46399998664855957, + 0.5239999890327454 + ], + "mrr": [ + 0.9681944847106934, + 0.9599999785423279, + 0.9755284786224365 + ], + "n_ret": 20.0, + "d_all": [ + 0.012000000104308128, + -0.004000000189989805, + 0.027000000700354576, + 0.1728827117288271 + ], + "d_recall": [ + 0.00925000011920929, + 0.000750000006519258, + 0.017000000923871994, + 0.030996900309969003 + ], + "d_mrr": [ + -0.0069039068184792995, + -0.014627084136009216, + 0.0008636901620775461, + 0.0831916808319168 + ], + "d_all_holm": 0.8644135586441355, + "d_recall_holm": 0.18598140185981402 + }, + { + "label": "Vector top-30 + cross-encoder", + "key": "rerank", + "variant": null, + "recall": [ + 0.7595000267028809, + 0.7442437410354614, + 0.7747499942779541 + ], + "all": [ + 0.5009999871253967, + 0.4699999988079071, + 0.531000018119812 + ], + "mrr": [ + 0.9735333323478699, + 0.9664316773414612, + 0.9803333878517151 + ], + "n_ret": 20.0, + "d_all": [ + 0.01899999938905239, + 0.00800000037997961, + 0.03099999949336052, + 0.0036996300369963003 + ], + "d_recall": [ + 0.010499999858438969, + 0.004749999847263098, + 0.016750000417232513, + 0.0011998800119988001 + ], + "d_mrr": [ + -0.0015650182031095028, + -0.010337738320231438, + 0.0074683744460344315, + 0.7318268173182682 + ], + "d_all_holm": 0.025897410258974102, + "d_recall_holm": 0.008399160083991601 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph chosen on dev: mention]", + "key": "lib_mh@best", + "variant": "mention", + "recall": [ + 0.9829999804496765, + 0.9769999980926514, + 0.9884999990463257 + ], + "all": [ + 0.9639999866485596, + 0.9520000219345093, + 0.9750000238418579 + ], + "mrr": [ + 0.9750333428382874, + 0.9670658111572266, + 0.9820842146873474 + ], + "n_ret": 17.611000061035156, + "d_all": [ + 0.4819999933242798, + 0.45100000500679016, + 0.5130000114440918, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.23399999737739563, + 0.21825000643730164, + 0.2487500011920929, + 9.999000099990002e-05 + ], + "d_mrr": [ + -6.501829921035096e-05, + -0.0034935898147523403, + 0.0033573030959814787, + 0.996000399960004 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention]", + "key": "ppr@best", + "variant": "mention", + "recall": [ + 0.9800000190734863, + 0.9735000133514404, + 0.9857500195503235 + ], + "all": [ + 0.9549999833106995, + 0.9419749975204468, + 0.9670000076293945 + ], + "mrr": [ + 0.711575448513031, + 0.6915011405944824, + 0.7322697639465332 + ], + "n_ret": 20.0, + "d_all": [ + 0.4729999899864197, + 0.44200000166893005, + 0.503000020980835, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.23100000619888306, + 0.21525000035762787, + 0.2460000067949295, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.2635229527950287, + -0.28323760628700256, + -0.2422732561826706, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph chosen on dev: mention]", + "key": "lib_default@best", + "variant": "mention", + "recall": [ + 0.9745000004768372, + 0.9674999713897705, + 0.9812562465667725 + ], + "all": [ + 0.9480000138282776, + 0.9340000152587891, + 0.9610249996185303 + ], + "mrr": [ + 0.9749261736869812, + 0.9669857025146484, + 0.9820101261138916 + ], + "n_ret": 10.10200023651123, + "d_all": [ + 0.4659999907016754, + 0.4339999854564667, + 0.4970250129699707, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.22550000250339508, + 0.2094937562942505, + 0.24124999344348907, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.00017216122068930417, + -0.0034767859615385532, + 0.003214606549590826, + 0.8557144285571443 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: mention]", + "key": "lib_mh:mention", + "variant": "mention", + "recall": [ + 0.9829999804496765, + 0.9769999980926514, + 0.9884999990463257 + ], + "all": [ + 0.9639999866485596, + 0.9520000219345093, + 0.9750000238418579 + ], + "mrr": [ + 0.9750333428382874, + 0.9670658111572266, + 0.9820842146873474 + ], + "n_ret": 17.611000061035156, + "d_all": [ + 0.4819999933242798, + 0.45100000500679016, + 0.5130000114440918, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.23399999737739563, + 0.21825000643730164, + 0.2487500011920929, + 9.999000099990002e-05 + ], + "d_mrr": [ + -6.501829921035096e-05, + -0.0034935898147523403, + 0.0033573030959814787, + 0.996000399960004 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: entity]", + "key": "lib_mh:entity", + "variant": "entity", + "recall": [ + 0.9340000152587891, + 0.9242500066757202, + 0.9440000057220459 + ], + "all": [ + 0.8429999947547913, + 0.8209999799728394, + 0.8650000095367432 + ], + "mrr": [ + 0.9750213623046875, + 0.9671761989593506, + 0.9820833802223206 + ], + "n_ret": 19.895999908447266, + "d_all": [ + 0.3610000014305115, + 0.32897499203681946, + 0.39500001072883606, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.1850000023841858, + 0.16899999976158142, + 0.20125000178813934, + 9.999000099990002e-05 + ], + "d_mrr": [ + -7.692313374718651e-05, + -0.0034102564677596092, + 0.003322481643408537, + 0.9027097290270973 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: knn]", + "key": "lib_mh:knn", + "variant": "knn", + "recall": [ + 0.8827499747276306, + 0.8699937462806702, + 0.8952500224113464 + ], + "all": [ + 0.7179999947547913, + 0.6890000104904175, + 0.7450249791145325 + ], + "mrr": [ + 0.9748602509498596, + 0.9668998718261719, + 0.9820008277893066 + ], + "n_ret": 19.98900032043457, + "d_all": [ + 0.23600000143051147, + 0.20800000429153442, + 0.2619999945163727, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.13375000655651093, + 0.11900000274181366, + 0.14749999344348907, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0002380953956162557, + -0.0035714285913854837, + 0.003166987095028162, + 0.8295170482951705 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: ctrl_mention]", + "key": "lib_mh:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.7490000128746033, + 0.734250009059906, + 0.7642499804496765 + ], + "all": [ + 0.4819999933242798, + 0.4519999921321869, + 0.512024998664856 + ], + "mrr": [ + 0.9750983119010925, + 0.967162549495697, + 0.9821966886520386 + ], + "n_ret": 20.0, + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_mrr": [ + 0.0, + -0.0035000001080334187, + 0.0035000001080334187, + 1.0 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: ctrl_entity]", + "key": "lib_mh:ctrl_entity", + "variant": "ctrl_entity", + "recall": [ + 0.7490000128746033, + 0.734250009059906, + 0.7642499804496765 + ], + "all": [ + 0.4819999933242798, + 0.4519999921321869, + 0.512024998664856 + ], + "mrr": [ + 0.9750983119010925, + 0.967162549495697, + 0.9821966886520386 + ], + "n_ret": 20.0, + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_mrr": [ + 0.0, + -0.0035000001080334187, + 0.0035000001080334187, + 1.0 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: mention]", + "key": "ppr:mention", + "variant": "mention", + "recall": [ + 0.9800000190734863, + 0.9735000133514404, + 0.9857500195503235 + ], + "all": [ + 0.9549999833106995, + 0.9419749975204468, + 0.9670000076293945 + ], + "mrr": [ + 0.711575448513031, + 0.6915011405944824, + 0.7322697639465332 + ], + "n_ret": 20.0, + "d_all": [ + 0.4729999899864197, + 0.44200000166893005, + 0.503000020980835, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.23100000619888306, + 0.21525000035762787, + 0.2460000067949295, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.2635229527950287, + -0.28323760628700256, + -0.2422732561826706, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: entity]", + "key": "ppr:entity", + "variant": "entity", + "recall": [ + 0.9325000047683716, + 0.9219937324523926, + 0.9424999952316284 + ], + "all": [ + 0.8299999833106995, + 0.8050000071525574, + 0.853024959564209 + ], + "mrr": [ + 0.8725833296775818, + 0.8576666712760925, + 0.8879004120826721 + ], + "n_ret": 20.0, + "d_all": [ + 0.3479999899864197, + 0.3179999887943268, + 0.3779999911785126, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.1835000067949295, + 0.1689937561750412, + 0.1979999989271164, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.10251501202583313, + -0.11773949861526489, + -0.08733005821704865, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: knn]", + "key": "ppr:knn", + "variant": "knn", + "recall": [ + 0.875249981880188, + 0.8622499704360962, + 0.8877500295639038 + ], + "all": [ + 0.7020000219345093, + 0.6749750375747681, + 0.7300000190734863 + ], + "mrr": [ + 0.9193999767303467, + 0.9075807929039001, + 0.9320171475410461 + ], + "n_ret": 20.0, + "d_all": [ + 0.2199999988079071, + 0.19099999964237213, + 0.2460000067949295, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.1262499988079071, + 0.11174999922513962, + 0.1397562474012375, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.05569835007190704, + -0.06752645969390869, + -0.0440867617726326, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: ctrl_mention]", + "key": "ppr:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.7490000128746033, + 0.734250009059906, + 0.7642499804496765 + ], + "all": [ + 0.4819999933242798, + 0.4519999921321869, + 0.512024998664856 + ], + "mrr": [ + 0.973598301410675, + 0.9656568765640259, + 0.9806906580924988 + ], + "n_ret": 20.0, + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_mrr": [ + -0.0014999998966231942, + -0.00549999950453639, + 0.002333333482965827, + 0.4694530546945305 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: ctrl_entity]", + "key": "ppr:ctrl_entity", + "variant": "ctrl_entity", + "recall": [ + 0.7484999895095825, + 0.7332500219345093, + 0.7639999985694885 + ], + "all": [ + 0.48100000619888306, + 0.45100000500679016, + 0.5120000243186951 + ], + "mrr": [ + 0.9643464088439941, + 0.9553632736206055, + 0.9727344512939453 + ], + "n_ret": 20.0, + "d_all": [ + -0.0010000000474974513, + -0.004999999888241291, + 0.0020000000949949026, + 1.0 + ], + "d_recall": [ + -0.0005000000237487257, + -0.0024999999441206455, + 0.0010000000474974513, + 1.0 + ], + "d_mrr": [ + -0.010751922614872456, + -0.01804332435131073, + -0.003754553385078907, + 0.0025997400259974 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: mention]", + "key": "lib_default:mention", + "variant": "mention", + "recall": [ + 0.9745000004768372, + 0.9674999713897705, + 0.9812562465667725 + ], + "all": [ + 0.9480000138282776, + 0.9340000152587891, + 0.9610249996185303 + ], + "mrr": [ + 0.9749261736869812, + 0.9669857025146484, + 0.9820101261138916 + ], + "n_ret": 10.10200023651123, + "d_all": [ + 0.4659999907016754, + 0.4339999854564667, + 0.4970250129699707, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.22550000250339508, + 0.2094937562942505, + 0.24124999344348907, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.00017216122068930417, + -0.0034767859615385532, + 0.003214606549590826, + 0.8557144285571443 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: entity]", + "key": "lib_default:entity", + "variant": "entity", + "recall": [ + 0.9340000152587891, + 0.9242500066757202, + 0.9440000057220459 + ], + "all": [ + 0.8429999947547913, + 0.8209999799728394, + 0.8650000095367432 + ], + "mrr": [ + 0.9750213623046875, + 0.9671761989593506, + 0.9820833802223206 + ], + "n_ret": 19.895999908447266, + "d_all": [ + 0.3610000014305115, + 0.32897499203681946, + 0.39500001072883606, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.1850000023841858, + 0.16899999976158142, + 0.20125000178813934, + 9.999000099990002e-05 + ], + "d_mrr": [ + -7.692313374718651e-05, + -0.0034102564677596092, + 0.003322481643408537, + 0.9027097290270973 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: knn]", + "key": "lib_default:knn", + "variant": "knn", + "recall": [ + 0.8827499747276306, + 0.8699937462806702, + 0.8952500224113464 + ], + "all": [ + 0.7179999947547913, + 0.6890000104904175, + 0.7450249791145325 + ], + "mrr": [ + 0.9748602509498596, + 0.9668998718261719, + 0.9820008277893066 + ], + "n_ret": 19.98900032043457, + "d_all": [ + 0.23600000143051147, + 0.20800000429153442, + 0.2619999945163727, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.13375000655651093, + 0.11900000274181366, + 0.14749999344348907, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0002380953956162557, + -0.0035714285913854837, + 0.003166987095028162, + 0.8295170482951705 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: ctrl_mention]", + "key": "lib_default:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.6912500262260437, + 0.6759937405586243, + 0.7070000171661377 + ], + "all": [ + 0.3840000033378601, + 0.3540000021457672, + 0.4129999876022339 + ], + "mrr": [ + 0.9747833013534546, + 0.9668149948120117, + 0.9819999933242798 + ], + "n_ret": 9.920999526977539, + "d_all": [ + -0.09799999743700027, + -0.11599999666213989, + -0.07999999821186066, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.057750001549720764, + -0.06725624948740005, + -0.048493750393390656, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.00031501840567216277, + -0.003642857074737549, + 0.0030384156852960587, + 0.7809219078092191 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: ctrl_entity]", + "key": "lib_default:ctrl_entity", + "variant": "ctrl_entity", + "recall": [ + 0.6917499899864197, + 0.6765000224113464, + 0.7074999809265137 + ], + "all": [ + 0.38499999046325684, + 0.35497498512268066, + 0.4140250086784363 + ], + "mrr": [ + 0.9747833013534546, + 0.9668149948120117, + 0.9819999933242798 + ], + "n_ret": 19.93400001525879, + "d_all": [ + -0.09700000286102295, + -0.11500000208616257, + -0.07900000363588333, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.0572500005364418, + -0.06674999743700027, + -0.04800000041723251, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.00031501840567216277, + -0.003642857074737549, + 0.0030384156852960587, + 0.7809219078092191 + ], + "d_all_holm": 0.0020997900209979003, + "d_recall_holm": 0.0020997900209979003 + } + ] + }, + "subgraph": { + "5": [ + { + "variant": "mention", + "connect_max_hops": 2, + "size": 5.035999774932861, + "recall": [ + 0.9139999747276306, + 0.903249979019165, + 0.9247499704360962 + ], + "all": [ + 0.7789999842643738, + 0.7539750337600708, + 0.8050000071525574 + ], + "vec_matched_recall": [ + 0.69175, + 0.6765, + 0.7075 + ], + "vec_matched_all": [ + 0.385, + 0.355, + 0.414 + ], + "d_all": [ + 0.39399999380111694, + 0.3610000014305115, + 0.42500001192092896, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.22224999964237213, + 0.2070000022649765, + 0.23749999701976776, + 9.999000099990002e-05 + ] + }, + { + "variant": "mention", + "connect_max_hops": 3, + "size": 5.079999923706055, + "recall": [ + 0.9150000214576721, + 0.9045000076293945, + 0.9257500171661377 + ], + "all": [ + 0.7799999713897705, + 0.7549999952316284, + 0.8059999942779541 + ], + "vec_matched_recall": [ + 0.69225, + 0.67725, + 0.708 + ], + "vec_matched_all": [ + 0.385, + 0.355, + 0.414 + ], + "d_all": [ + 0.39500001072883606, + 0.3619999885559082, + 0.4259999990463257, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.2227499932050705, + 0.20724999904632568, + 0.23800000548362732, + 9.999000099990002e-05 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 2, + "size": 5.0, + "recall": [ + 0.6912500262260437, + 0.6759937405586243, + 0.7070000171661377 + ], + "all": [ + 0.3840000033378601, + 0.3540000021457672, + 0.4129999876022339 + ], + "vec_matched_recall": [ + 0.69075, + 0.67574375, + 0.70625625 + ], + "vec_matched_all": [ + 0.383, + 0.352975, + 0.412 + ], + "d_all": [ + 0.0010000000474974513, + 0.0, + 0.003000000026077032, + 1.0 + ], + "d_recall": [ + 0.0005000000237487257, + 0.0, + 0.001500000013038516, + 1.0 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 3, + "size": 5.0, + "recall": [ + 0.6912500262260437, + 0.6759937405586243, + 0.7070000171661377 + ], + "all": [ + 0.3840000033378601, + 0.3540000021457672, + 0.4129999876022339 + ], + "vec_matched_recall": [ + 0.69075, + 0.67574375, + 0.70625625 + ], + "vec_matched_all": [ + 0.383, + 0.352975, + 0.412 + ], + "d_all": [ + 0.0010000000474974513, + 0.0, + 0.003000000026077032, + 1.0 + ], + "d_recall": [ + 0.0005000000237487257, + 0.0, + 0.001500000013038516, + 1.0 + ] + }, + { + "variant": "ctrl_entity", + "connect_max_hops": 2, + "size": 5.074999809265137, + "recall": [ + 0.6912500262260437, + 0.6759937405586243, + 0.7070000171661377 + ], + "all": [ + 0.3840000033378601, + 0.3540000021457672, + 0.4129999876022339 + ], + "vec_matched_recall": [ + 0.69175, + 0.67674375, + 0.70725 + ], + "vec_matched_all": [ + 0.384, + 0.354, + 0.413 + ], + "d_all": [ + 0.0, + -0.003000000026077032, + 0.003000000026077032, + 1.0 + ], + "d_recall": [ + -0.0005000000237487257, + -0.0020000000949949026, + 0.0010000000474974513, + 1.0 + ] + }, + { + "variant": "ctrl_entity", + "connect_max_hops": 3, + "size": 6.283999919891357, + "recall": [ + 0.6912500262260437, + 0.6759937405586243, + 0.7070000171661377 + ], + "all": [ + 0.3840000033378601, + 0.3540000021457672, + 0.4129999876022339 + ], + "vec_matched_recall": [ + 0.70375, + 0.6885, + 0.7195 + ], + "vec_matched_all": [ + 0.406, + 0.376, + 0.436 + ], + "d_all": [ + -0.02199999988079071, + -0.03200000151991844, + -0.013000000268220901, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.012500000186264515, + -0.017750000581145287, + -0.00774999987334013, + 9.999000099990002e-05 + ] + } + ], + "10": [ + { + "variant": "mention", + "connect_max_hops": 2, + "size": 8.159000396728516, + "recall": [ + 0.9739999771118164, + 0.9669937491416931, + 0.9807562828063965 + ], + "all": [ + 0.9459999799728394, + 0.9319999814033508, + 0.9599999785423279 + ], + "vec_matched_recall": [ + 0.7145, + 0.699, + 0.73 + ], + "vec_matched_all": [ + 0.424, + 0.394, + 0.454 + ], + "d_all": [ + 0.5220000147819519, + 0.49000000953674316, + 0.5529999732971191, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.25949999690055847, + 0.24349375069141388, + 0.27524998784065247, + 9.999000099990002e-05 + ] + }, + { + "variant": "mention", + "connect_max_hops": 3, + "size": 8.164999961853027, + "recall": [ + 0.9739999771118164, + 0.9669937491416931, + 0.9807562828063965 + ], + "all": [ + 0.9459999799728394, + 0.9319999814033508, + 0.9599999785423279 + ], + "vec_matched_recall": [ + 0.7145, + 0.699, + 0.73 + ], + "vec_matched_all": [ + 0.424, + 0.394, + 0.454 + ], + "d_all": [ + 0.5220000147819519, + 0.49000000953674316, + 0.5529999732971191, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.25949999690055847, + 0.24349375069141388, + 0.27524998784065247, + 9.999000099990002e-05 + ] + }, + { + "variant": "entity", + "connect_max_hops": 2, + "size": 10.543000221252441, + "recall": [ + 0.8970000147819519, + 0.8855000138282776, + 0.9087499976158142 + ], + "all": [ + 0.7440000176429749, + 0.7170000076293945, + 0.7710000276565552 + ], + "vec_matched_recall": [ + 0.72575, + 0.7102437500000001, + 0.74125625 + ], + "vec_matched_all": [ + 0.445, + 0.414, + 0.476 + ], + "d_all": [ + 0.29899999499320984, + 0.26899999380111694, + 0.3269999921321869, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.17125000059604645, + 0.15600000321865082, + 0.18524999916553497, + 9.999000099990002e-05 + ] + }, + { + "variant": "entity", + "connect_max_hops": 3, + "size": 12.279000282287598, + "recall": [ + 0.8997499942779541, + 0.8884999752044678, + 0.9112562537193298 + ], + "all": [ + 0.7509999871253967, + 0.7239999771118164, + 0.7770000100135803 + ], + "vec_matched_recall": [ + 0.73, + 0.71475, + 0.746 + ], + "vec_matched_all": [ + 0.45, + 0.419975, + 0.48 + ], + "d_all": [ + 0.3009999990463257, + 0.2709999978542328, + 0.32899999618530273, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.16975000500679016, + 0.15399999916553497, + 0.18400000035762787, + 9.999000099990002e-05 + ] + }, + { + "variant": "knn", + "connect_max_hops": 2, + "size": 10.694999694824219, + "recall": [ + 0.8412500023841858, + 0.8274999856948853, + 0.8547499775886536 + ], + "all": [ + 0.6430000066757202, + 0.6129999756813049, + 0.6710000038146973 + ], + "vec_matched_recall": [ + 0.72525, + 0.70975, + 0.74050625 + ], + "vec_matched_all": [ + 0.443, + 0.413, + 0.473 + ], + "d_all": [ + 0.20000000298023224, + 0.17399999499320984, + 0.22402501106262207, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.11599999666213989, + 0.10274375230073929, + 0.1287499964237213, + 9.999000099990002e-05 + ] + }, + { + "variant": "knn", + "connect_max_hops": 3, + "size": 12.062000274658203, + "recall": [ + 0.8454999923706055, + 0.8317437171936035, + 0.859250009059906 + ], + "all": [ + 0.6499999761581421, + 0.6209999918937683, + 0.6779999732971191 + ], + "vec_matched_recall": [ + 0.72825, + 0.71275, + 0.74375 + ], + "vec_matched_all": [ + 0.447, + 0.417, + 0.477 + ], + "d_all": [ + 0.2029999941587448, + 0.1770000010728836, + 0.22699999809265137, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.1172500029206276, + 0.10374999791383743, + 0.12974999845027924, + 9.999000099990002e-05 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 2, + "size": 10.003000259399414, + "recall": [ + 0.722000002861023, + 0.7067499756813049, + 0.7367562651634216 + ], + "all": [ + 0.43700000643730164, + 0.40700000524520874, + 0.46700000762939453 + ], + "vec_matched_recall": [ + 0.723, + 0.7075, + 0.73825625 + ], + "vec_matched_all": [ + 0.439, + 0.409, + 0.469 + ], + "d_all": [ + -0.0020000000949949026, + -0.004999999888241291, + 0.0, + 0.49725027497250274 + ], + "d_recall": [ + -0.0010000000474974513, + -0.0024999999441206455, + 0.0, + 0.49725027497250274 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 3, + "size": 10.003000259399414, + "recall": [ + 0.722000002861023, + 0.7067499756813049, + 0.7367562651634216 + ], + "all": [ + 0.43700000643730164, + 0.40700000524520874, + 0.46700000762939453 + ], + "vec_matched_recall": [ + 0.723, + 0.7075, + 0.73825625 + ], + "vec_matched_all": [ + 0.439, + 0.409, + 0.469 + ], + "d_all": [ + -0.0020000000949949026, + -0.004999999888241291, + 0.0, + 0.49725027497250274 + ], + "d_recall": [ + -0.0010000000474974513, + -0.0024999999441206455, + 0.0, + 0.49725027497250274 + ] + }, + { + "variant": "ctrl_entity", + "connect_max_hops": 2, + "size": 10.336000442504883, + "recall": [ + 0.7225000262260437, + 0.7070000171661377, + 0.737500011920929 + ], + "all": [ + 0.43799999356269836, + 0.40799999237060547, + 0.46799999475479126 + ], + "vec_matched_recall": [ + 0.7235, + 0.708, + 0.73875625 + ], + "vec_matched_all": [ + 0.44, + 0.409, + 0.47 + ], + "d_all": [ + -0.0020000000949949026, + -0.004999999888241291, + 0.0, + 0.49725027497250274 + ], + "d_recall": [ + -0.0010000000474974513, + -0.0024999999441206455, + 0.0, + 0.49725027497250274 + ] + }, + { + "variant": "ctrl_entity", + "connect_max_hops": 3, + "size": 15.878000259399414, + "recall": [ + 0.7227500081062317, + 0.7074937224388123, + 0.7377499938011169 + ], + "all": [ + 0.43799999356269836, + 0.40799999237060547, + 0.46799999475479126 + ], + "vec_matched_recall": [ + 0.74025, + 0.72474375, + 0.75575625 + ], + "vec_matched_all": [ + 0.467, + 0.436, + 0.499 + ], + "d_all": [ + -0.028999999165534973, + -0.03999999910593033, + -0.01899999938905239, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.017500000074505806, + -0.023006249219179153, + -0.012249999679625034, + 9.999000099990002e-05 + ] + } + ], + "20": [ + { + "variant": "mention", + "connect_max_hops": 2, + "size": 17.639999389648438, + "recall": [ + 0.9829999804496765, + 0.9769999980926514, + 0.9884999990463257 + ], + "all": [ + 0.9639999866485596, + 0.9520000219345093, + 0.9750000238418579 + ], + "vec_matched_recall": [ + 0.74325, + 0.728, + 0.759 + ], + "vec_matched_all": [ + 0.471, + 0.441, + 0.502 + ], + "d_all": [ + 0.49300000071525574, + 0.460999995470047, + 0.5239999890327454, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.2397499978542328, + 0.2240000069141388, + 0.25475001335144043, + 9.999000099990002e-05 + ] + }, + { + "variant": "mention", + "connect_max_hops": 3, + "size": 17.697999954223633, + "recall": [ + 0.9829999804496765, + 0.9769999980926514, + 0.9884999990463257 + ], + "all": [ + 0.9639999866485596, + 0.9520000219345093, + 0.9750000238418579 + ], + "vec_matched_recall": [ + 0.74325, + 0.728, + 0.759 + ], + "vec_matched_all": [ + 0.471, + 0.441, + 0.502 + ], + "d_all": [ + 0.49300000071525574, + 0.460999995470047, + 0.5239999890327454, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.2397499978542328, + 0.2240000069141388, + 0.25475001335144043, + 9.999000099990002e-05 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 2, + "size": 20.041000366210938, + "recall": [ + 0.7490000128746033, + 0.734250009059906, + 0.7642499804496765 + ], + "all": [ + 0.4819999933242798, + 0.4519999921321869, + 0.512024998664856 + ], + "vec_matched_recall": [ + 0.749, + 0.73425, + 0.76425 + ], + "vec_matched_all": [ + 0.482, + 0.452, + 0.5120249999999998 + ], + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 3, + "size": 20.158000946044922, + "recall": [ + 0.7490000128746033, + 0.734250009059906, + 0.7642499804496765 + ], + "all": [ + 0.4819999933242798, + 0.4519999921321869, + 0.512024998664856 + ], + "vec_matched_recall": [ + 0.7495, + 0.73474375, + 0.76475625 + ], + "vec_matched_all": [ + 0.483, + 0.453, + 0.514 + ], + "d_all": [ + -0.0010000000474974513, + -0.004000000189989805, + 0.0, + 1.0 + ], + "d_recall": [ + -0.0005000000237487257, + -0.0020000000949949026, + 0.0, + 1.0 + ] + }, + { + "variant": "ctrl_entity", + "connect_max_hops": 2, + "size": 21.51300048828125, + "recall": [ + 0.7490000128746033, + 0.734250009059906, + 0.7642499804496765 + ], + "all": [ + 0.4819999933242798, + 0.4519999921321869, + 0.512024998664856 + ], + "vec_matched_recall": [ + 0.751, + 0.736, + 0.7665 + ], + "vec_matched_all": [ + 0.485, + 0.455, + 0.515 + ], + "d_all": [ + -0.003000000026077032, + -0.007000000216066837, + 0.0, + 0.24017598240175983 + ], + "d_recall": [ + -0.0020000000949949026, + -0.004000000189989805, + -0.0005000000237487257, + 0.05879412058794121 + ] + }, + { + "variant": "ctrl_entity", + "connect_max_hops": 3, + "size": 37.33000183105469, + "recall": [ + 0.749750018119812, + 0.7350000143051147, + 0.765250027179718 + ], + "all": [ + 0.4830000102519989, + 0.453000009059906, + 0.5130000114440918 + ], + "vec_matched_recall": [ + 0.77575, + 0.76099375, + 0.791 + ], + "vec_matched_all": [ + 0.53, + 0.5, + 0.5600249999999999 + ], + "d_all": [ + -0.04699999839067459, + -0.061000000685453415, + -0.03400000184774399, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.026000000536441803, + -0.032999999821186066, + -0.01924999989569187, + 9.999000099990002e-05 + ] + } + ] + }, + "lib_minus_ppr": { + "mention": { + "5": [ + 0.1889999955892563, + 0.1589999943971634, + 0.21899999678134918, + 9.999000099990002e-05 + ], + "10": [ + 0.017999999225139618, + 0.007000000216066837, + 0.028999999165534973, + 0.001999800019998 + ], + "20": [ + 0.008999999612569809, + 0.004000000189989805, + 0.014999999664723873, + 0.003999600039996 + ] + }, + "entity": { + "5": [ + 0.10199999809265137, + 0.07599999755620956, + 0.12700000405311584, + 9.999000099990002e-05 + ], + "10": [ + 0.04500000178813934, + 0.02199999988079071, + 0.06700000166893005, + 0.00019998000199980003 + ], + "20": [ + 0.013000000268220901, + -0.006000000052154064, + 0.032999999821186066, + 0.23467653234676533 + ] + }, + "knn": { + "5": [ + 0.06800000369548798, + 0.04699999839067459, + 0.08799999952316284, + 9.999000099990002e-05 + ], + "10": [ + 0.05400000140070915, + 0.035999998450279236, + 0.07199999690055847, + 9.999000099990002e-05 + ], + "20": [ + 0.01600000075995922, + 0.003000000026077032, + 0.02902499958872795, + 0.0336966303369663 + ] + } + }, + "breakdown": { + "type": { + "bridge_comparison": { + "n": 221, + "vector": 0.009049774147570133, + "rrf": 0.018099548295140266, + "rerank": 0.013574660755693913, + "lib_mh@best": 0.9049773812294006, + "ppr@best": 0.8190045356750488 + }, + "comparison": { + "n": 238, + "vector": 0.9789915680885315, + "rrf": 0.9915966391563416, + "rerank": 0.9789915680885315, + "lib_mh@best": 0.970588207244873, + "ppr@best": 0.9789915680885315 + }, + "compositional": { + "n": 422, + "vector": 0.32938387989997864, + "rrf": 0.34360188245773315, + "rerank": 0.36729857325553894, + "lib_mh@best": 0.9526066184043884, + "ppr@best": 0.950236976146698 + }, + "inference": { + "n": 119, + "vector": 0.5462185144424438, + "rrf": 0.5966386795043945, + "rerank": 0.5798319578170776, + "lib_mh@best": 0.9495798349380493, + "ppr@best": 0.9495798349380493 + } + }, + "hops": { + "2": { + "n": 779, + "vector": 0.5609756112098694, + "rrf": 0.5802310705184937, + "rerank": 0.5866495370864868, + "lib_mh@best": 0.9576380252838135, + "ppr@best": 0.9589216709136963 + }, + "4": { + "n": 221, + "vector": 0.009049774147570133, + "rrf": 0.018099548295140266, + "rerank": 0.013574660755693913, + "lib_mh@best": 0.9049773812294006, + "ppr@best": 0.8190045356750488 + } + } + } + } + }, + "musique": { + "minilm": { + "n_test": 1000, + "corpus_size": 15405, + "lib_similarity_search_same_top20_set": "200/200", + "graphs": { + "mention": { + "n_edges_undirected_union": 20272, + "n_edges_by_family_directed_or_undirected": 22465, + "mean_degree": 2.6318727685816294, + "median_degree": 0.0, + "max_degree": 54, + "isolated_frac": 0.5272314183706589, + "gold_pairs_directly_linked": 0.14690791250337565, + "gold_pairs_linked_within_2": 0.2292735619767756, + "neighbours_of_gold_that_are_in_same_question_context": 0.1532126205792047 + }, + "ctrl_mention": { + "n_edges_undirected_union": 20272, + "n_edges_by_family_directed_or_undirected": 22465, + "mean_degree": 2.6318727685816294, + "median_degree": 0.0, + "max_degree": 54, + "isolated_frac": 0.5272314183706589, + "gold_pairs_directly_linked": 0.0, + "gold_pairs_linked_within_2": 0.0, + "neighbours_of_gold_that_are_in_same_question_context": 0.001101741312327797 + } + }, + "selected": { + "mention": { + "lib": { + "5": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "10": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + }, + "20": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + } + }, + "ppr": { + "5": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "10": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + }, + "20": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.3, + "directed_mention": false, + "weighted": false + } + } + }, + "ctrl_mention": { + "lib": { + "5": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 1, + "seeds": 20, + "decay": 0.9, + "blend": 0.6, + "direction": "both", + "weight_field": null + } + }, + "ppr": { + "5": { + "alpha": 0.85, + "seeds": 10, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 20, + "beta": 0.6, + "directed_mention": true, + "weighted": false + }, + "20": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.6, + "directed_mention": true, + "weighted": false + } + } + } + }, + "tables": { + "5": [ + { + "label": "Vector top-k (plain)", + "key": "vector", + "variant": null, + "recall": [ + 0.45483338832855225, + 0.4379124939441681, + 0.4726666808128357 + ], + "all": [ + 0.11900000274181366, + 0.10000000149011612, + 0.13899999856948853 + ], + "mrr": [ + 0.7016333341598511, + 0.6775162220001221, + 0.7258337736129761 + ], + "n_ret": 5.0 + }, + { + "label": "BM25", + "key": "bm25", + "variant": null, + "recall": [ + 0.4493333399295807, + 0.4325000047683716, + 0.4660833477973938 + ], + "all": [ + 0.11100000143051147, + 0.09300000220537186, + 0.12902499735355377 + ], + "mrr": [ + 0.7223166227340698, + 0.6981974244117737, + 0.7454354166984558 + ], + "n_ret": 5.0, + "d_all": [ + -0.00800000037997961, + -0.027025001123547554, + 0.012000000104308128, + 0.5032496750324967 + ], + "d_recall": [ + -0.005499999038875103, + -0.02341875061392784, + 0.01216875109821558, + 0.5484451554844516 + ], + "d_mrr": [ + 0.02068333327770233, + -0.00790791492909193, + 0.04956916347146034, + 0.14858514148585142 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "Hybrid vector+BM25 (RRF)", + "key": "rrf", + "variant": null, + "recall": [ + 0.4867500066757202, + 0.46974998712539673, + 0.5026687979698181 + ], + "all": [ + 0.12600000202655792, + 0.10599999874830246, + 0.1459999978542328 + ], + "mrr": [ + 0.7326999306678772, + 0.7103966474533081, + 0.7538179159164429 + ], + "n_ret": 5.0, + "d_all": [ + 0.007000000216066837, + -0.008999999612569809, + 0.023000000044703484, + 0.45085491450854914 + ], + "d_recall": [ + 0.03191666677594185, + 0.01783333346247673, + 0.045249998569488525, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.031066667288541794, + 0.011599168181419373, + 0.05070125311613083, + 0.0026997300269973002 + ], + "d_all_holm": 1.0, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "Vector top-30 + cross-encoder", + "key": "rerank", + "variant": null, + "recall": [ + 0.5238333344459534, + 0.5062479376792908, + 0.5410854816436768 + ], + "all": [ + 0.17800000309944153, + 0.15497499704360962, + 0.20200000703334808 + ], + "mrr": [ + 0.7995499968528748, + 0.7788604497909546, + 0.8205670714378357 + ], + "n_ret": 5.0, + "d_all": [ + 0.05900000035762787, + 0.041999999433755875, + 0.07699999958276749, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.0689999982714653, + 0.0554145909845829, + 0.08224999904632568, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.09791667014360428, + 0.07679291814565659, + 0.11993333697319031, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph chosen on dev: mention]", + "key": "lib_mh@best", + "variant": "mention", + "recall": [ + 0.48500001430511475, + 0.46649792790412903, + 0.5039166808128357 + ], + "all": [ + 0.18400000035762787, + 0.16099999845027924, + 0.20800000429153442 + ], + "mrr": [ + 0.7017333507537842, + 0.6778157949447632, + 0.7259862422943115 + ], + "n_ret": 5.0, + "d_all": [ + 0.06499999761581421, + 0.04899999871850014, + 0.08202499896287918, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.030166668817400932, + 0.021333331242203712, + 0.039250001311302185, + 9.999000099990002e-05 + ], + "d_mrr": [ + 9.999996109399945e-05, + -0.0014837499475106597, + 0.001850416767410934, + 0.9173082691730827 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention]", + "key": "ppr@best", + "variant": "mention", + "recall": [ + 0.4833333492279053, + 0.4643312692642212, + 0.5029187798500061 + ], + "all": [ + 0.18299999833106995, + 0.1599999964237213, + 0.20600000023841858 + ], + "mrr": [ + 0.5881500244140625, + 0.5643445253372192, + 0.6131362915039062 + ], + "n_ret": 5.0, + "d_all": [ + 0.06400000303983688, + 0.044975001364946365, + 0.08402499556541443, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.02850000001490116, + 0.017495833337306976, + 0.03916666656732559, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.11348332464694977, + -0.13361792266368866, + -0.09276541322469711, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph chosen on dev: mention]", + "key": "lib_default@best", + "variant": "mention", + "recall": [ + 0.48666664958000183, + 0.46791043877601624, + 0.5058333277702332 + ], + "all": [ + 0.18199999630451202, + 0.1589999943971634, + 0.20600000023841858 + ], + "mrr": [ + 0.7020166516304016, + 0.6774487495422363, + 0.7261512875556946 + ], + "n_ret": 5.0, + "d_all": [ + 0.06300000101327896, + 0.04699999839067459, + 0.08100000023841858, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.031833335757255554, + 0.02316666580736637, + 0.04100208356976509, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.0003833333612419665, + -0.0005666665965691209, + 0.0016333332750946283, + 0.5072492750724927 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: mention]", + "key": "lib_mh:mention", + "variant": "mention", + "recall": [ + 0.48500001430511475, + 0.46649792790412903, + 0.5039166808128357 + ], + "all": [ + 0.18400000035762787, + 0.16099999845027924, + 0.20800000429153442 + ], + "mrr": [ + 0.7017333507537842, + 0.6778157949447632, + 0.7259862422943115 + ], + "n_ret": 5.0, + "d_all": [ + 0.06499999761581421, + 0.04899999871850014, + 0.08202499896287918, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.030166668817400932, + 0.021333331242203712, + 0.039250001311302185, + 9.999000099990002e-05 + ], + "d_mrr": [ + 9.999996109399945e-05, + -0.0014837499475106597, + 0.001850416767410934, + 0.9173082691730827 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: ctrl_mention]", + "key": "lib_mh:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.4553333818912506, + 0.438581258058548, + 0.4730854332447052 + ], + "all": [ + 0.11999999731779099, + 0.10100000351667404, + 0.14002500474452972 + ], + "mrr": [ + 0.7016833424568176, + 0.6775492429733276, + 0.7259004712104797 + ], + "n_ret": 5.0, + "d_all": [ + 0.0010000000474974513, + 0.0, + 0.003000000026077032, + 1.0 + ], + "d_recall": [ + 0.0005000000237487257, + 0.0, + 0.001500000013038516, + 1.0 + ], + "d_mrr": [ + 4.999999873689376e-05, + 0.0, + 0.00019999999494757503, + 1.0 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: mention]", + "key": "ppr:mention", + "variant": "mention", + "recall": [ + 0.4833333492279053, + 0.4643312692642212, + 0.5029187798500061 + ], + "all": [ + 0.18299999833106995, + 0.1599999964237213, + 0.20600000023841858 + ], + "mrr": [ + 0.5881500244140625, + 0.5643445253372192, + 0.6131362915039062 + ], + "n_ret": 5.0, + "d_all": [ + 0.06400000303983688, + 0.044975001364946365, + 0.08402499556541443, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.02850000001490116, + 0.017495833337306976, + 0.03916666656732559, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.11348332464694977, + -0.13361792266368866, + -0.09276541322469711, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: ctrl_mention]", + "key": "ppr:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.4543333649635315, + 0.43708333373069763, + 0.47241875529289246 + ], + "all": [ + 0.11999999731779099, + 0.10100000351667404, + 0.14000000059604645 + ], + "mrr": [ + 0.6952999830245972, + 0.670647919178009, + 0.7195345759391785 + ], + "n_ret": 5.0, + "d_all": [ + 0.0010000000474974513, + -0.0020000000949949026, + 0.004000000189989805, + 1.0 + ], + "d_recall": [ + -0.0005000000237487257, + -0.0029166664462536573, + 0.0018333334010094404, + 0.7715228477152285 + ], + "d_mrr": [ + -0.006333332974463701, + -0.011201249435544014, + -0.0011491667246446013, + 0.0124987501249875 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: mention]", + "key": "lib_default:mention", + "variant": "mention", + "recall": [ + 0.48666664958000183, + 0.46791043877601624, + 0.5058333277702332 + ], + "all": [ + 0.18199999630451202, + 0.1589999943971634, + 0.20600000023841858 + ], + "mrr": [ + 0.7020166516304016, + 0.6774487495422363, + 0.7261512875556946 + ], + "n_ret": 5.0, + "d_all": [ + 0.06300000101327896, + 0.04699999839067459, + 0.08100000023841858, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.031833335757255554, + 0.02316666580736637, + 0.04100208356976509, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.0003833333612419665, + -0.0005666665965691209, + 0.0016333332750946283, + 0.5072492750724927 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: ctrl_mention]", + "key": "lib_default:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.44749999046325684, + 0.4304979145526886, + 0.46491873264312744 + ], + "all": [ + 0.11299999803304672, + 0.09399999678134918, + 0.13199999928474426 + ], + "mrr": [ + 0.7014333605766296, + 0.6772996187210083, + 0.7257345914840698 + ], + "n_ret": 5.0, + "d_all": [ + -0.006000000052154064, + -0.012000000104308128, + -0.0010000000474974513, + 0.07039296070392961 + ], + "d_recall": [ + -0.007333333604037762, + -0.01100208330899477, + -0.003833333496004343, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.00020000000949949026, + -0.000699999975040555, + 9.999999747378752e-05, + 0.7477252274772522 + ], + "d_all_holm": 0.3519648035196481, + "d_recall_holm": 0.0011998800119988001 + } + ], + "10": [ + { + "label": "Vector top-k (plain)", + "key": "vector", + "variant": null, + "recall": [ + 0.5315833687782288, + 0.5140812993049622, + 0.5493375062942505 + ], + "all": [ + 0.18199999630451202, + 0.15800000727176666, + 0.20602500438690186 + ], + "mrr": [ + 0.7099040150642395, + 0.6861887574195862, + 0.7334710359573364 + ], + "n_ret": 10.0 + }, + { + "label": "BM25", + "key": "bm25", + "variant": null, + "recall": [ + 0.5149166584014893, + 0.49758121371269226, + 0.5310021042823792 + ], + "all": [ + 0.15199999511241913, + 0.12997499108314514, + 0.17299999296665192 + ], + "mrr": [ + 0.7303885221481323, + 0.7070243954658508, + 0.7524423599243164 + ], + "n_ret": 10.0, + "d_all": [ + -0.029999999329447746, + -0.05299999937415123, + -0.00800000037997961, + 0.012798720127987202 + ], + "d_recall": [ + -0.01666666753590107, + -0.035352081060409546, + 0.001085416879504919, + 0.0705929407059294 + ], + "d_mrr": [ + 0.020484523847699165, + -0.006716178730130196, + 0.048126816749572754, + 0.1374862513748625 + ], + "d_all_holm": 0.05119488051194881, + "d_recall_holm": 0.35296470352964704 + }, + { + "label": "Hybrid vector+BM25 (RRF)", + "key": "rrf", + "variant": null, + "recall": [ + 0.5673334002494812, + 0.5507500171661377, + 0.584502100944519 + ], + "all": [ + 0.20399999618530273, + 0.17800000309944153, + 0.2290000021457672 + ], + "mrr": [ + 0.7406325340270996, + 0.7189154028892517, + 0.7607734799385071 + ], + "n_ret": 10.0, + "d_all": [ + 0.02199999988079071, + 0.0020000000949949026, + 0.04100000113248825, + 0.0326967303269673 + ], + "d_recall": [ + 0.035750001668930054, + 0.021412501111626625, + 0.05008541792631149, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.030728571116924286, + 0.01133356336504221, + 0.04974677413702011, + 0.0018998100189981002 + ], + "d_all_holm": 0.09809019098090191, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "Vector top-30 + cross-encoder", + "key": "rerank", + "variant": null, + "recall": [ + 0.5820000171661377, + 0.5645833015441895, + 0.5989999771118164 + ], + "all": [ + 0.2370000034570694, + 0.21199999749660492, + 0.2630000114440918 + ], + "mrr": [ + 0.8039527535438538, + 0.7838360667228699, + 0.8244226574897766 + ], + "n_ret": 10.0, + "d_all": [ + 0.054999999701976776, + 0.03799999877810478, + 0.07199999690055847, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.05041666701436043, + 0.03941666707396507, + 0.06191875413060188, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.09404881298542023, + 0.07362593710422516, + 0.11531351506710052, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph chosen on dev: mention]", + "key": "lib_mh@best", + "variant": "mention", + "recall": [ + 0.5752500295639038, + 0.5560791492462158, + 0.5945854783058167 + ], + "all": [ + 0.27399998903274536, + 0.24699999392032623, + 0.3019999861717224 + ], + "mrr": [ + 0.7100067138671875, + 0.6865110993385315, + 0.7334798574447632 + ], + "n_ret": 10.0, + "d_all": [ + 0.09200000017881393, + 0.07199999690055847, + 0.11299999803304672, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.04366666451096535, + 0.03316666558384895, + 0.055416665971279144, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.00010277784167556092, + -0.0011806149268522859, + 0.0014033630723133683, + 0.8734126587341265 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention]", + "key": "ppr@best", + "variant": "mention", + "recall": [ + 0.5710833668708801, + 0.5509999990463257, + 0.5910021066665649 + ], + "all": [ + 0.289000004529953, + 0.26100000739097595, + 0.3160250186920166 + ], + "mrr": [ + 0.6477341055870056, + 0.6238757371902466, + 0.6717676520347595 + ], + "n_ret": 10.0, + "d_all": [ + 0.10700000077486038, + 0.08399999886751175, + 0.13300000131130219, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.039500001817941666, + 0.026664583012461662, + 0.05366874486207962, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.062169838696718216, + -0.07928238064050674, + -0.04527798667550087, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph chosen on dev: mention]", + "key": "lib_default@best", + "variant": "mention", + "recall": [ + 0.5356667041778564, + 0.5157458186149597, + 0.5562541484832764 + ], + "all": [ + 0.24500000476837158, + 0.21799999475479126, + 0.27300000190734863 + ], + "mrr": [ + 0.7046713829040527, + 0.6809232234954834, + 0.7284615635871887 + ], + "n_ret": 8.722000122070312, + "d_all": [ + 0.06300000101327896, + 0.037974998354911804, + 0.09099999815225601, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.004083332139998674, + -0.011418749578297138, + 0.019333332777023315, + 0.5923407659234077 + ], + "d_mrr": [ + -0.005232539493590593, + -0.007319534197449684, + -0.0031730057671666145, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: mention]", + "key": "lib_mh:mention", + "variant": "mention", + "recall": [ + 0.5752500295639038, + 0.5560791492462158, + 0.5945854783058167 + ], + "all": [ + 0.27399998903274536, + 0.24699999392032623, + 0.3019999861717224 + ], + "mrr": [ + 0.7100067138671875, + 0.6865110993385315, + 0.7334798574447632 + ], + "n_ret": 10.0, + "d_all": [ + 0.09200000017881393, + 0.07199999690055847, + 0.11299999803304672, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.04366666451096535, + 0.03316666558384895, + 0.055416665971279144, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.00010277784167556092, + -0.0011806149268522859, + 0.0014033630723133683, + 0.8734126587341265 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: ctrl_mention]", + "key": "lib_mh:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.5315833687782288, + 0.5140812993049622, + 0.5493375062942505 + ], + "all": [ + 0.18199999630451202, + 0.15800000727176666, + 0.20602500438690186 + ], + "mrr": [ + 0.7099539637565613, + 0.6861893534660339, + 0.7335197925567627 + ], + "n_ret": 10.0, + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_mrr": [ + 4.999999873689376e-05, + 0.0, + 0.00019999999494757503, + 1.0 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: mention]", + "key": "ppr:mention", + "variant": "mention", + "recall": [ + 0.5710833668708801, + 0.5509999990463257, + 0.5910021066665649 + ], + "all": [ + 0.289000004529953, + 0.26100000739097595, + 0.3160250186920166 + ], + "mrr": [ + 0.6477341055870056, + 0.6238757371902466, + 0.6717676520347595 + ], + "n_ret": 10.0, + "d_all": [ + 0.10700000077486038, + 0.08399999886751175, + 0.13300000131130219, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.039500001817941666, + 0.026664583012461662, + 0.05366874486207962, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.062169838696718216, + -0.07928238064050674, + -0.04527798667550087, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: ctrl_mention]", + "key": "ppr:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.53125, + 0.5134978890419006, + 0.5491728782653809 + ], + "all": [ + 0.18199999630451202, + 0.15800000727176666, + 0.20600000023841858 + ], + "mrr": [ + 0.705642819404602, + 0.6819239258766174, + 0.7295958995819092 + ], + "n_ret": 10.0, + "d_all": [ + 0.0, + -0.003000000026077032, + 0.003000000026077032, + 1.0 + ], + "d_recall": [ + -0.00033333332976326346, + -0.002333333482965827, + 0.001668750075623393, + 0.815018498150185 + ], + "d_mrr": [ + -0.0042611113749444485, + -0.00840915646404028, + 5.312476059771143e-05, + 0.043695630436956304 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: mention]", + "key": "lib_default:mention", + "variant": "mention", + "recall": [ + 0.5356667041778564, + 0.5157458186149597, + 0.5562541484832764 + ], + "all": [ + 0.24500000476837158, + 0.21799999475479126, + 0.27300000190734863 + ], + "mrr": [ + 0.7046713829040527, + 0.6809232234954834, + 0.7284615635871887 + ], + "n_ret": 8.722000122070312, + "d_all": [ + 0.06300000101327896, + 0.037974998354911804, + 0.09099999815225601, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.004083332139998674, + -0.011418749578297138, + 0.019333332777023315, + 0.5923407659234077 + ], + "d_mrr": [ + -0.005232539493590593, + -0.007319534197449684, + -0.0031730057671666145, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: ctrl_mention]", + "key": "lib_default:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.45216667652130127, + 0.4349958300590515, + 0.4701666831970215 + ], + "all": [ + 0.11599999666213989, + 0.09700000286102295, + 0.13600000739097595 + ], + "mrr": [ + 0.7015444040298462, + 0.6774381399154663, + 0.7258176207542419 + ], + "n_ret": 8.940999984741211, + "d_all": [ + -0.06599999964237213, + -0.0820000022649765, + -0.05000000074505806, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.0794166699051857, + -0.09041666239500046, + -0.06899791210889816, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.008359523490071297, + -0.01055846270173788, + -0.006347956135869026, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + } + ], + "20": [ + { + "label": "Vector top-k (plain)", + "key": "vector", + "variant": null, + "recall": [ + 0.606916606426239, + 0.5888313055038452, + 0.6246687173843384 + ], + "all": [ + 0.2630000114440918, + 0.23600000143051147, + 0.28999999165534973 + ], + "mrr": [ + 0.7130058407783508, + 0.6900275349617004, + 0.7361331582069397 + ], + "n_ret": 20.0 + }, + { + "label": "BM25", + "key": "bm25", + "variant": null, + "recall": [ + 0.5867499709129333, + 0.5701646208763123, + 0.6030853986740112 + ], + "all": [ + 0.23499999940395355, + 0.20999999344348907, + 0.26100000739097595 + ], + "mrr": [ + 0.73301762342453, + 0.7099869847297668, + 0.7549823522567749 + ], + "n_ret": 20.0, + "d_all": [ + -0.02800000086426735, + -0.05400000140070915, + -0.0010000000474974513, + 0.046995300469953004 + ], + "d_recall": [ + -0.0201666671782732, + -0.036670833826065063, + -0.003412501187995076, + 0.0235976402359764 + ], + "d_mrr": [ + 0.020011797547340393, + -0.007200962398201227, + 0.047040294855833054, + 0.14038596140385962 + ], + "d_all_holm": 0.23497650234976503, + "d_recall_holm": 0.0707929207079292 + }, + { + "label": "Hybrid vector+BM25 (RRF)", + "key": "rrf", + "variant": null, + "recall": [ + 0.6613333821296692, + 0.6451645493507385, + 0.6779187917709351 + ], + "all": [ + 0.3190000057220459, + 0.28999999165534973, + 0.3479999899864197 + ], + "mrr": [ + 0.7433933615684509, + 0.7220412492752075, + 0.7630098462104797 + ], + "n_ret": 20.0, + "d_all": [ + 0.0560000017285347, + 0.03200000151991844, + 0.07900000363588333, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.05441666767001152, + 0.04124791920185089, + 0.06816667318344116, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.03038751147687435, + 0.011561351828277111, + 0.048800501972436905, + 0.0014998500149985001 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "Vector top-30 + cross-encoder", + "key": "rerank", + "variant": null, + "recall": [ + 0.627333402633667, + 0.6099979281425476, + 0.6452521085739136 + ], + "all": [ + 0.2919999957084656, + 0.26397499442100525, + 0.3199999928474426 + ], + "mrr": [ + 0.804988443851471, + 0.7850074768066406, + 0.8252594470977783 + ], + "n_ret": 20.0, + "d_all": [ + 0.028999999165534973, + 0.01600000075995922, + 0.0430000014603138, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.020416665822267532, + 0.012749999761581421, + 0.02850000001490116, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.09198261797428131, + 0.07161832600831985, + 0.11309535801410675, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph chosen on dev: mention]", + "key": "lib_mh@best", + "variant": "mention", + "recall": [ + 0.6467499732971191, + 0.6263333559036255, + 0.6660833358764648 + ], + "all": [ + 0.37700000405311584, + 0.3469749987125397, + 0.40700000524520874 + ], + "mrr": [ + 0.7117802500724792, + 0.6887549757957458, + 0.735133171081543 + ], + "n_ret": 19.788999557495117, + "d_all": [ + 0.11400000005960464, + 0.08799999952316284, + 0.1420000046491623, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.03983333334326744, + 0.02449999935925007, + 0.05508958548307419, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0012255297042429447, + -0.0026845489628612995, + 0.00018042759620584548, + 0.0837916208379162 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention]", + "key": "ppr@best", + "variant": "mention", + "recall": [ + 0.6527500152587891, + 0.6337437629699707, + 0.671999990940094 + ], + "all": [ + 0.38100001215934753, + 0.35100001096725464, + 0.41200000047683716 + ], + "mrr": [ + 0.5982714295387268, + 0.5753381252288818, + 0.6221896409988403 + ], + "n_ret": 20.0, + "d_all": [ + 0.11800000071525574, + 0.09399999678134918, + 0.14300000667572021, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.04583333060145378, + 0.0325833335518837, + 0.05992082878947258, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.11473437398672104, + -0.1348227560520172, + -0.094353087246418, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph chosen on dev: mention]", + "key": "lib_default@best", + "variant": "mention", + "recall": [ + 0.5535833239555359, + 0.5342479348182678, + 0.5738354325294495 + ], + "all": [ + 0.2639999985694885, + 0.23800000548362732, + 0.29100000858306885 + ], + "mrr": [ + 0.7055317163467407, + 0.681695818901062, + 0.7292566299438477 + ], + "n_ret": 13.380000114440918, + "d_all": [ + 0.0010000000474974513, + -0.028999999165534973, + 0.032999999821186066, + 1.0 + ], + "d_recall": [ + -0.0533333346247673, + -0.0716666728258133, + -0.03541458398103714, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0074740503914654255, + -0.009646699763834476, + -0.00527218310162425, + 9.999000099990002e-05 + ], + "d_all_holm": 1.0, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: mention]", + "key": "lib_mh:mention", + "variant": "mention", + "recall": [ + 0.6467499732971191, + 0.6263333559036255, + 0.6660833358764648 + ], + "all": [ + 0.37700000405311584, + 0.3469749987125397, + 0.40700000524520874 + ], + "mrr": [ + 0.7117802500724792, + 0.6887549757957458, + 0.735133171081543 + ], + "n_ret": 19.788999557495117, + "d_all": [ + 0.11400000005960464, + 0.08799999952316284, + 0.1420000046491623, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.03983333334326744, + 0.02449999935925007, + 0.05508958548307419, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0012255297042429447, + -0.0026845489628612995, + 0.00018042759620584548, + 0.0837916208379162 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: ctrl_mention]", + "key": "lib_mh:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.6072499752044678, + 0.5892479419708252, + 0.6249187588691711 + ], + "all": [ + 0.2630000114440918, + 0.23600000143051147, + 0.28999999165534973 + ], + "mrr": [ + 0.7130558490753174, + 0.6900776028633118, + 0.7362024784088135 + ], + "n_ret": 20.0, + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.00033333332976326346, + 0.0, + 0.0010000000474974513, + 1.0 + ], + "d_mrr": [ + 4.999999873689376e-05, + 0.0, + 0.00019999999494757503, + 1.0 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: mention]", + "key": "ppr:mention", + "variant": "mention", + "recall": [ + 0.6527500152587891, + 0.6337437629699707, + 0.671999990940094 + ], + "all": [ + 0.38100001215934753, + 0.35100001096725464, + 0.41200000047683716 + ], + "mrr": [ + 0.5982714295387268, + 0.5753381252288818, + 0.6221896409988403 + ], + "n_ret": 20.0, + "d_all": [ + 0.11800000071525574, + 0.09399999678134918, + 0.14300000667572021, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.04583333060145378, + 0.0325833335518837, + 0.05992082878947258, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.11473437398672104, + -0.1348227560520172, + -0.094353087246418, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: ctrl_mention]", + "key": "ppr:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.606666624546051, + 0.5884146094322205, + 0.6244999766349792 + ], + "all": [ + 0.2630000114440918, + 0.23600000143051147, + 0.28999999165534973 + ], + "mrr": [ + 0.7104296088218689, + 0.6872822046279907, + 0.7335828542709351 + ], + "n_ret": 20.0, + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + -0.0002500000118743628, + -0.000750000006519258, + 0.0, + 1.0 + ], + "d_mrr": [ + -0.002576190512627363, + -0.006505535449832678, + 0.0013818155275657773, + 0.1977802219778022 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: mention]", + "key": "lib_default:mention", + "variant": "mention", + "recall": [ + 0.5535833239555359, + 0.5342479348182678, + 0.5738354325294495 + ], + "all": [ + 0.2639999985694885, + 0.23800000548362732, + 0.29100000858306885 + ], + "mrr": [ + 0.7055317163467407, + 0.681695818901062, + 0.7292566299438477 + ], + "n_ret": 13.380000114440918, + "d_all": [ + 0.0010000000474974513, + -0.028999999165534973, + 0.032999999821186066, + 1.0 + ], + "d_recall": [ + -0.0533333346247673, + -0.0716666728258133, + -0.03541458398103714, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0074740503914654255, + -0.009646699763834476, + -0.00527218310162425, + 9.999000099990002e-05 + ], + "d_all_holm": 1.0, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: ctrl_mention]", + "key": "lib_default:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.45399999618530273, + 0.43691667914390564, + 0.4720853865146637 + ], + "all": [ + 0.11800000071525574, + 0.09897500276565552, + 0.13899999856948853 + ], + "mrr": [ + 0.7015444040298462, + 0.6774381399154663, + 0.7258176207542419 + ], + "n_ret": 13.737000465393066, + "d_all": [ + -0.14499999582767487, + -0.16699999570846558, + -0.12300000339746475, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.15291666984558105, + -0.1670854240655899, + -0.13866248726844788, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.011461387388408184, + -0.013721570372581482, + -0.00939079187810421, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + } + ] + }, + "subgraph": { + "10": [ + { + "variant": "mention", + "connect_max_hops": 2, + "size": 10.333000183105469, + "recall": [ + 0.5765833854675293, + 0.5572500228881836, + 0.5958333015441895 + ], + "all": [ + 0.27399998903274536, + 0.24699999392032623, + 0.3019999861717224 + ], + "vec_matched_recall": [ + 0.53575, + 0.5179145833333333, + 0.5534166666666667 + ], + "vec_matched_all": [ + 0.185, + 0.161, + 0.209 + ], + "d_all": [ + 0.08900000154972076, + 0.0689999982714653, + 0.10999999940395355, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.04083333536982536, + 0.029833335429430008, + 0.0523395836353302, + 9.999000099990002e-05 + ] + }, + { + "variant": "mention", + "connect_max_hops": 3, + "size": 10.758999824523926, + "recall": [ + 0.578916609287262, + 0.5596666932106018, + 0.5982499718666077 + ], + "all": [ + 0.2770000100135803, + 0.24899999797344208, + 0.30399999022483826 + ], + "vec_matched_recall": [ + 0.5414166666666668, + 0.5235833333333334, + 0.5588333333333333 + ], + "vec_matched_all": [ + 0.19, + 0.166, + 0.215 + ], + "d_all": [ + 0.08699999749660492, + 0.06599999964237213, + 0.10899999737739563, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.03750000149011612, + 0.026164585724473, + 0.04924999922513962, + 9.999000099990002e-05 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 2, + "size": 10.005999565124512, + "recall": [ + 0.5315833687782288, + 0.5140812993049622, + 0.5493375062942505 + ], + "all": [ + 0.18199999630451202, + 0.15800000727176666, + 0.20602500438690186 + ], + "vec_matched_recall": [ + 0.5320833333333332, + 0.51458125, + 0.54950625 + ], + "vec_matched_all": [ + 0.182, + 0.158, + 0.20602499999999985 + ], + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + -0.0004999961820431054, + -0.0014999965205788612, + 4.152705290749736e-09, + 1.0 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 3, + "size": 10.02299976348877, + "recall": [ + 0.5315833687782288, + 0.5140812993049622, + 0.5493375062942505 + ], + "all": [ + 0.18199999630451202, + 0.15800000727176666, + 0.20602500438690186 + ], + "vec_matched_recall": [ + 0.5320833333333332, + 0.51458125, + 0.54950625 + ], + "vec_matched_all": [ + 0.182, + 0.158, + 0.20602499999999985 + ], + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + -0.0004999961820431054, + -0.0014999965205788612, + 4.152705290749736e-09, + 1.0 + ] + } + ] + }, + "lib_minus_ppr": { + "mention": { + "5": [ + 0.0010000000474974513, + -0.01600000075995922, + 0.017000000923871994, + 1.0 + ], + "10": [ + -0.014999999664723873, + -0.032999999821186066, + 0.0020000000949949026, + 0.1197880211978802 + ], + "20": [ + -0.004000000189989805, + -0.01899999938905239, + 0.010999999940395355, + 0.6785321467853215 + ] + } + }, + "breakdown": { + "type": { + "2hop": { + "n": 519, + "vector": 0.3082851767539978, + "rrf": 0.3410404622554779, + "rerank": 0.37764933705329895, + "lib_mh@best": 0.4624277353286743, + "ppr@best": 0.4797687828540802 + }, + "3hop1": { + "n": 213, + "vector": 0.04694835841655731, + "rrf": 0.056338027119636536, + "rerank": 0.11737088859081268, + "lib_mh@best": 0.09389671683311462, + "ppr@best": 0.11267605423927307 + }, + "3hop2": { + "n": 78, + "vector": 0.12820513546466827, + "rrf": 0.1666666716337204, + "rerank": 0.1666666716337204, + "lib_mh@best": 0.14102564752101898, + "ppr@best": 0.14102564752101898 + }, + "4hop1": { + "n": 113, + "vector": 0.008849557489156723, + "rrf": 0.0, + "rerank": 0.017699114978313446, + "lib_mh@best": 0.017699114978313446, + "ppr@best": 0.03539822995662689 + }, + "4hop2": { + "n": 30, + "vector": 0.0, + "rrf": 0.0, + "rerank": 0.0, + "lib_mh@best": 0.0, + "ppr@best": 0.0 + }, + "4hop3": { + "n": 47, + "vector": 0.021276595070958138, + "rrf": 0.042553190141916275, + "rerank": 0.021276595070958138, + "lib_mh@best": 0.021276595070958138, + "ppr@best": 0.021276595070958138 + } + }, + "hops": { + "2": { + "n": 519, + "vector": 0.3082851767539978, + "rrf": 0.3410404622554779, + "rerank": 0.37764933705329895, + "lib_mh@best": 0.4624277353286743, + "ppr@best": 0.4797687828540802 + }, + "3": { + "n": 291, + "vector": 0.06872852146625519, + "rrf": 0.08591065555810928, + "rerank": 0.13058419525623322, + "lib_mh@best": 0.10652920603752136, + "ppr@best": 0.12027491629123688 + }, + "4": { + "n": 190, + "vector": 0.010526316240429878, + "rrf": 0.010526316240429878, + "rerank": 0.015789473429322243, + "lib_mh@best": 0.015789473429322243, + "ppr@best": 0.02631578966975212 + } + } + } + }, + "bge": { + "n_test": 1000, + "corpus_size": 15405, + "lib_similarity_search_same_top20_set": "200/200", + "graphs": { + "mention": { + "n_edges_undirected_union": 20272, + "n_edges_by_family_directed_or_undirected": 22465, + "mean_degree": 2.6318727685816294, + "median_degree": 0.0, + "max_degree": 54, + "isolated_frac": 0.5272314183706589, + "gold_pairs_directly_linked": 0.14690791250337565, + "gold_pairs_linked_within_2": 0.2292735619767756, + "neighbours_of_gold_that_are_in_same_question_context": 0.1532126205792047 + }, + "ctrl_mention": { + "n_edges_undirected_union": 20272, + "n_edges_by_family_directed_or_undirected": 22465, + "mean_degree": 2.6318727685816294, + "median_degree": 0.0, + "max_degree": 54, + "isolated_frac": 0.5272314183706589, + "gold_pairs_directly_linked": 0.0, + "gold_pairs_linked_within_2": 0.0, + "neighbours_of_gold_that_are_in_same_question_context": 0.001101741312327797 + } + }, + "selected": { + "mention": { + "lib": { + "5": { + "hops": 2, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 3, + "seeds": 20, + "decay": 0.9, + "blend": 0.0, + "direction": "both", + "weight_field": null + }, + "20": { + "hops": 2, + "seeds": 10, + "decay": 0.9, + "blend": 0.3, + "direction": "both", + "weight_field": null + } + }, + "ppr": { + "5": { + "alpha": 0.85, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "10": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "20": { + "alpha": 0.7, + "seeds": 5, + "beta": 0.6, + "directed_mention": false, + "weighted": false + } + } + }, + "ctrl_mention": { + "lib": { + "5": { + "hops": 2, + "seeds": 15, + "decay": 0.3, + "blend": 0.6, + "direction": "out", + "weight_field": null + }, + "10": { + "hops": 2, + "seeds": 10, + "decay": 0.7, + "blend": 0.0, + "direction": "out", + "weight_field": null + }, + "20": { + "hops": 1, + "seeds": 20, + "decay": 0.3, + "blend": 0.3, + "direction": "both", + "weight_field": null + } + }, + "ppr": { + "5": { + "alpha": 0.5, + "seeds": 10, + "beta": 0.6, + "directed_mention": false, + "weighted": false + }, + "10": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + }, + "20": { + "alpha": 0.5, + "seeds": 5, + "beta": 0.85, + "directed_mention": true, + "weighted": false + } + } + } + }, + "tables": { + "5": [ + { + "label": "Vector top-k (plain)", + "key": "vector", + "variant": null, + "recall": [ + 0.4830000102519989, + 0.4660812318325043, + 0.4999166429042816 + ], + "all": [ + 0.14300000667572021, + 0.12200000137090683, + 0.164000004529953 + ], + "mrr": [ + 0.7489333748817444, + 0.726815402507782, + 0.7703171372413635 + ], + "n_ret": 5.0 + }, + { + "label": "BM25", + "key": "bm25", + "variant": null, + "recall": [ + 0.4493333399295807, + 0.4325000047683716, + 0.4660833477973938 + ], + "all": [ + 0.11100000143051147, + 0.09300000220537186, + 0.12902499735355377 + ], + "mrr": [ + 0.7223166227340698, + 0.6981974244117737, + 0.7454354166984558 + ], + "n_ret": 5.0, + "d_all": [ + -0.03200000151991844, + -0.052000001072883606, + -0.012000000104308128, + 0.0029997000299970002 + ], + "d_recall": [ + -0.033666662871837616, + -0.04975416511297226, + -0.016583334654569626, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.026616666465997696, + -0.05285083130002022, + -0.00203083292581141, + 0.042695730426957304 + ], + "d_all_holm": 0.014998500149985002, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "Hybrid vector+BM25 (RRF)", + "key": "rrf", + "variant": null, + "recall": [ + 0.5019999742507935, + 0.4847458302974701, + 0.5182499885559082 + ], + "all": [ + 0.14900000393390656, + 0.12800000607967377, + 0.1720000058412552 + ], + "mrr": [ + 0.7803333401679993, + 0.7589325308799744, + 0.8000516295433044 + ], + "n_ret": 5.0, + "d_all": [ + 0.006000000052154064, + -0.012000000104308128, + 0.023000000044703484, + 0.5845415458454155 + ], + "d_recall": [ + 0.01900000125169754, + 0.00458333408460021, + 0.03166874870657921, + 0.005099490050994901 + ], + "d_mrr": [ + 0.03139999881386757, + 0.01219874992966652, + 0.04933374747633934, + 0.0014998500149985001 + ], + "d_all_holm": 1.0, + "d_recall_holm": 0.020397960203979604 + }, + { + "label": "Vector top-30 + cross-encoder", + "key": "rerank", + "variant": null, + "recall": [ + 0.534500002861023, + 0.5181645750999451, + 0.5516687631607056 + ], + "all": [ + 0.1889999955892563, + 0.16599999368190765, + 0.21199999749660492 + ], + "mrr": [ + 0.8166332840919495, + 0.7965996265411377, + 0.8370846509933472 + ], + "n_ret": 5.0, + "d_all": [ + 0.04600000008940697, + 0.02797500044107437, + 0.06499999761581421, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.051500000059604645, + 0.038083337247371674, + 0.06458333134651184, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.06769999861717224, + 0.047082915902137756, + 0.08796791732311249, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph chosen on dev: mention]", + "key": "lib_mh@best", + "variant": "mention", + "recall": [ + 0.5262500047683716, + 0.5071645975112915, + 0.5445854663848877 + ], + "all": [ + 0.2160000056028366, + 0.19099999964237213, + 0.24199999868869781 + ], + "mrr": [ + 0.7488833069801331, + 0.7267150282859802, + 0.7704046368598938 + ], + "n_ret": 5.0, + "d_all": [ + 0.0729999989271164, + 0.0560000017285347, + 0.09099999815225601, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.04325000196695328, + 0.03449999913573265, + 0.05283333361148834, + 9.999000099990002e-05 + ], + "d_mrr": [ + -5.0000013288808987e-05, + -0.0010999999940395355, + 0.0008999999845400453, + 1.0 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention]", + "key": "ppr@best", + "variant": "mention", + "recall": [ + 0.5169166922569275, + 0.4986666738986969, + 0.535504162311554 + ], + "all": [ + 0.20200000703334808, + 0.1770000010728836, + 0.22599999606609344 + ], + "mrr": [ + 0.6247666478157043, + 0.6030666828155518, + 0.6464687585830688 + ], + "n_ret": 5.0, + "d_all": [ + 0.05900000035762787, + 0.04100000113248825, + 0.07800000160932541, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.033916667103767395, + 0.02316666767001152, + 0.044337499886751175, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.12416666746139526, + -0.1438683271408081, + -0.10587874799966812, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph chosen on dev: mention]", + "key": "lib_default@best", + "variant": "mention", + "recall": [ + 0.4984999895095825, + 0.4811646044254303, + 0.5165020823478699 + ], + "all": [ + 0.17299999296665192, + 0.1509999930858612, + 0.19699999690055847 + ], + "mrr": [ + 0.7489333748817444, + 0.726815402507782, + 0.7703171372413635 + ], + "n_ret": 5.0, + "d_all": [ + 0.029999999329447746, + 0.019999999552965164, + 0.04100000113248825, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.01549999974668026, + 0.01075000036507845, + 0.021250000223517418, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: mention]", + "key": "lib_mh:mention", + "variant": "mention", + "recall": [ + 0.5262500047683716, + 0.5071645975112915, + 0.5445854663848877 + ], + "all": [ + 0.2160000056028366, + 0.19099999964237213, + 0.24199999868869781 + ], + "mrr": [ + 0.7488833069801331, + 0.7267150282859802, + 0.7704046368598938 + ], + "n_ret": 5.0, + "d_all": [ + 0.0729999989271164, + 0.0560000017285347, + 0.09099999815225601, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.04325000196695328, + 0.03449999913573265, + 0.05283333361148834, + 9.999000099990002e-05 + ], + "d_mrr": [ + -5.0000013288808987e-05, + -0.0010999999940395355, + 0.0008999999845400453, + 1.0 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: ctrl_mention]", + "key": "lib_mh:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.4830000102519989, + 0.4660812318325043, + 0.4999166429042816 + ], + "all": [ + 0.14300000667572021, + 0.12200000137090683, + 0.164000004529953 + ], + "mrr": [ + 0.7489333748817444, + 0.726815402507782, + 0.7703171372413635 + ], + "n_ret": 5.0, + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_mrr": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: mention]", + "key": "ppr:mention", + "variant": "mention", + "recall": [ + 0.5169166922569275, + 0.4986666738986969, + 0.535504162311554 + ], + "all": [ + 0.20200000703334808, + 0.1770000010728836, + 0.22599999606609344 + ], + "mrr": [ + 0.6247666478157043, + 0.6030666828155518, + 0.6464687585830688 + ], + "n_ret": 5.0, + "d_all": [ + 0.05900000035762787, + 0.04100000113248825, + 0.07800000160932541, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.033916667103767395, + 0.02316666767001152, + 0.044337499886751175, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.12416666746139526, + -0.1438683271408081, + -0.10587874799966812, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: ctrl_mention]", + "key": "ppr:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.47566670179367065, + 0.4583291709423065, + 0.4926687777042389 + ], + "all": [ + 0.14000000059604645, + 0.11900000274181366, + 0.16200000047683716 + ], + "mrr": [ + 0.7089166045188904, + 0.6852995753288269, + 0.7314366698265076 + ], + "n_ret": 5.0, + "d_all": [ + -0.003000000026077032, + -0.013000000268220901, + 0.007000000216066837, + 0.6904309569043096 + ], + "d_recall": [ + -0.007333333604037762, + -0.014418751001358032, + -0.0006645835237577558, + 0.0306969303069693 + ], + "d_mrr": [ + -0.04001666605472565, + -0.05178333446383476, + -0.028815416619181633, + 9.999000099990002e-05 + ], + "d_all_holm": 1.0, + "d_recall_holm": 0.09209079092090791 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: mention]", + "key": "lib_default:mention", + "variant": "mention", + "recall": [ + 0.4984999895095825, + 0.4811646044254303, + 0.5165020823478699 + ], + "all": [ + 0.17299999296665192, + 0.1509999930858612, + 0.19699999690055847 + ], + "mrr": [ + 0.7489333748817444, + 0.726815402507782, + 0.7703171372413635 + ], + "n_ret": 5.0, + "d_all": [ + 0.029999999329447746, + 0.019999999552965164, + 0.04100000113248825, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.01549999974668026, + 0.01075000036507845, + 0.021250000223517418, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: ctrl_mention]", + "key": "lib_default:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.48249998688697815, + 0.4655812680721283, + 0.49916666746139526 + ], + "all": [ + 0.1420000046491623, + 0.12099999934434891, + 0.16300000250339508 + ], + "mrr": [ + 0.7489333748817444, + 0.726815402507782, + 0.7703171372413635 + ], + "n_ret": 5.0, + "d_all": [ + -0.0010000000474974513, + -0.003000000026077032, + 0.0, + 1.0 + ], + "d_recall": [ + -0.0005000000237487257, + -0.001500000013038516, + 0.0, + 1.0 + ], + "d_mrr": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + } + ], + "10": [ + { + "label": "Vector top-k (plain)", + "key": "vector", + "variant": null, + "recall": [ + 0.5571666359901428, + 0.540164589881897, + 0.5741688013076782 + ], + "all": [ + 0.20900000631809235, + 0.1839749962091446, + 0.23399999737739563 + ], + "mrr": [ + 0.7558729648590088, + 0.7347243428230286, + 0.7766838669776917 + ], + "n_ret": 10.0 + }, + { + "label": "BM25", + "key": "bm25", + "variant": null, + "recall": [ + 0.5149166584014893, + 0.49758121371269226, + 0.5310021042823792 + ], + "all": [ + 0.15199999511241913, + 0.12997499108314514, + 0.17299999296665192 + ], + "mrr": [ + 0.7303885221481323, + 0.7070243954658508, + 0.7524423599243164 + ], + "n_ret": 10.0, + "d_all": [ + -0.05700000002980232, + -0.08102499693632126, + -0.03400000184774399, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.042249999940395355, + -0.0599166639149189, + -0.025581251829862595, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.02548452466726303, + -0.05083653703331947, + -0.001360081136226654, + 0.0443955604439556 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "Hybrid vector+BM25 (RRF)", + "key": "rrf", + "variant": null, + "recall": [ + 0.5849166512489319, + 0.5683333873748779, + 0.6018374562263489 + ], + "all": [ + 0.2329999953508377, + 0.20800000429153442, + 0.25999999046325684 + ], + "mrr": [ + 0.7857797741889954, + 0.7652465105056763, + 0.8050572872161865 + ], + "n_ret": 10.0, + "d_all": [ + 0.024000000208616257, + 0.0020000000949949026, + 0.04699999839067459, + 0.039996000399960006 + ], + "d_recall": [ + 0.02775000035762787, + 0.012497915886342525, + 0.041754163801670074, + 0.00019998000199980003 + ], + "d_mrr": [ + 0.02990674413740635, + 0.011499602347612381, + 0.04714306443929672, + 0.0017998200179982 + ], + "d_all_holm": 0.11998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "Vector top-30 + cross-encoder", + "key": "rerank", + "variant": null, + "recall": [ + 0.5956667065620422, + 0.5793333649635315, + 0.6122541427612305 + ], + "all": [ + 0.2460000067949295, + 0.22100000083446503, + 0.27102500200271606 + ], + "mrr": [ + 0.8213452100753784, + 0.8019863367080688, + 0.8408030867576599 + ], + "n_ret": 10.0, + "d_all": [ + 0.03700000047683716, + 0.01899999938905239, + 0.0560000017285347, + 0.00019998000199980003 + ], + "d_recall": [ + 0.03849999979138374, + 0.027249999344348907, + 0.05008333548903465, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.06547222286462784, + 0.04545881971716881, + 0.08504962176084518, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph chosen on dev: mention]", + "key": "lib_mh@best", + "variant": "mention", + "recall": [ + 0.6079999804496765, + 0.5900833606719971, + 0.6262521147727966 + ], + "all": [ + 0.29600000381469727, + 0.2680000066757202, + 0.3240000009536743 + ], + "mrr": [ + 0.7555786371231079, + 0.7343944907188416, + 0.7764427661895752 + ], + "n_ret": 10.0, + "d_all": [ + 0.08699999749660492, + 0.06599999964237213, + 0.1080000028014183, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.050833337008953094, + 0.03958333656191826, + 0.062168750911951065, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.00029444447136484087, + -0.0016242560232058167, + 0.000951567490119487, + 0.654934506549345 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention]", + "key": "ppr@best", + "variant": "mention", + "recall": [ + 0.6023333072662354, + 0.5831583142280579, + 0.6206666231155396 + ], + "all": [ + 0.29100000858306885, + 0.2639999985694885, + 0.3190000057220459 + ], + "mrr": [ + 0.6903226375579834, + 0.6684051752090454, + 0.7125791907310486 + ], + "n_ret": 10.0, + "d_all": [ + 0.0820000022649765, + 0.05900000035762787, + 0.10502499341964722, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.04516666755080223, + 0.031416669487953186, + 0.05908333510160446, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.06555039435625076, + -0.08192408084869385, + -0.04997457191348076, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph chosen on dev: mention]", + "key": "lib_default@best", + "variant": "mention", + "recall": [ + 0.5716666579246521, + 0.5518312454223633, + 0.5910000205039978 + ], + "all": [ + 0.2630000114440918, + 0.23600000143051147, + 0.28999999165534973 + ], + "mrr": [ + 0.7516947984695435, + 0.7297875285148621, + 0.7728981971740723 + ], + "n_ret": 9.006999969482422, + "d_all": [ + 0.05400000140070915, + 0.02997499890625477, + 0.07999999821186066, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.014500000514090061, + -0.0005854163900949061, + 0.030166666954755783, + 0.04559544045595441 + ], + "d_mrr": [ + -0.004178174771368504, + -0.0060469345189630985, + -0.002401924692094326, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.18238176182381763 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: mention]", + "key": "lib_mh:mention", + "variant": "mention", + "recall": [ + 0.6079999804496765, + 0.5900833606719971, + 0.6262521147727966 + ], + "all": [ + 0.29600000381469727, + 0.2680000066757202, + 0.3240000009536743 + ], + "mrr": [ + 0.7555786371231079, + 0.7343944907188416, + 0.7764427661895752 + ], + "n_ret": 10.0, + "d_all": [ + 0.08699999749660492, + 0.06599999964237213, + 0.1080000028014183, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.050833337008953094, + 0.03958333656191826, + 0.062168750911951065, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.00029444447136484087, + -0.0016242560232058167, + 0.000951567490119487, + 0.654934506549345 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: ctrl_mention]", + "key": "lib_mh:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.5571666359901428, + 0.540164589881897, + 0.5741688013076782 + ], + "all": [ + 0.20900000631809235, + 0.1839749962091446, + 0.23399999737739563 + ], + "mrr": [ + 0.7558729648590088, + 0.7347243428230286, + 0.7766838669776917 + ], + "n_ret": 10.0, + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_mrr": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: mention]", + "key": "ppr:mention", + "variant": "mention", + "recall": [ + 0.6023333072662354, + 0.5831583142280579, + 0.6206666231155396 + ], + "all": [ + 0.29100000858306885, + 0.2639999985694885, + 0.3190000057220459 + ], + "mrr": [ + 0.6903226375579834, + 0.6684051752090454, + 0.7125791907310486 + ], + "n_ret": 10.0, + "d_all": [ + 0.0820000022649765, + 0.05900000035762787, + 0.10502499341964722, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.04516666755080223, + 0.031416669487953186, + 0.05908333510160446, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.06555039435625076, + -0.08192408084869385, + -0.04997457191348076, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: ctrl_mention]", + "key": "ppr:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.5568333268165588, + 0.5400000214576721, + 0.5738333463668823 + ], + "all": [ + 0.20800000429153442, + 0.18299999833106995, + 0.2329999953508377 + ], + "mrr": [ + 0.7587218284606934, + 0.7372958064079285, + 0.7791936993598938 + ], + "n_ret": 10.0, + "d_all": [ + -0.0010000000474974513, + -0.003000000026077032, + 0.0, + 1.0 + ], + "d_recall": [ + -0.000333333300659433, + -0.0009999999310821295, + 0.0, + 1.0 + ], + "d_mrr": [ + 0.002848809352144599, + -0.0005865177954547107, + 0.0069005051627755165, + 0.14578542145785423 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: mention]", + "key": "lib_default:mention", + "variant": "mention", + "recall": [ + 0.5716666579246521, + 0.5518312454223633, + 0.5910000205039978 + ], + "all": [ + 0.2630000114440918, + 0.23600000143051147, + 0.28999999165534973 + ], + "mrr": [ + 0.7516947984695435, + 0.7297875285148621, + 0.7728981971740723 + ], + "n_ret": 9.006999969482422, + "d_all": [ + 0.05400000140070915, + 0.02997499890625477, + 0.07999999821186066, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.014500000514090061, + -0.0005854163900949061, + 0.030166666954755783, + 0.04559544045595441 + ], + "d_mrr": [ + -0.004178174771368504, + -0.0060469345189630985, + -0.002401924692094326, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.18238176182381763 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: ctrl_mention]", + "key": "lib_default:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.48249998688697815, + 0.4655812680721283, + 0.49916666746139526 + ], + "all": [ + 0.1420000046491623, + 0.12099999934434891, + 0.16300000250339508 + ], + "mrr": [ + 0.7489333748817444, + 0.726815402507782, + 0.7703171372413635 + ], + "n_ret": 8.956999778747559, + "d_all": [ + -0.06700000166893005, + -0.08299999684095383, + -0.050999999046325684, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.07466667145490646, + -0.08475624769926071, + -0.0645812451839447, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.006939683109521866, + -0.008962837979197502, + -0.005149513948708773, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + } + ], + "20": [ + { + "label": "Vector top-k (plain)", + "key": "vector", + "variant": null, + "recall": [ + 0.6230000257492065, + 0.6059166789054871, + 0.6405041813850403 + ], + "all": [ + 0.28600001335144043, + 0.2590000033378601, + 0.3140000104904175 + ], + "mrr": [ + 0.7578624486923218, + 0.7367129325866699, + 0.7784180045127869 + ], + "n_ret": 20.0 + }, + { + "label": "BM25", + "key": "bm25", + "variant": null, + "recall": [ + 0.5867499709129333, + 0.5701646208763123, + 0.6030853986740112 + ], + "all": [ + 0.23499999940395355, + 0.20999999344348907, + 0.26100000739097595 + ], + "mrr": [ + 0.73301762342453, + 0.7099869847297668, + 0.7549823522567749 + ], + "n_ret": 20.0, + "d_all": [ + -0.050999999046325684, + -0.07800000160932541, + -0.023000000044703484, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.036249998956918716, + -0.05400000140070915, + -0.01849791780114174, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.024844780564308167, + -0.049501556903123856, + -0.0009675421752035618, + 0.0466953304669533 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "Hybrid vector+BM25 (RRF)", + "key": "rrf", + "variant": null, + "recall": [ + 0.6673333048820496, + 0.6514958143234253, + 0.6835020780563354 + ], + "all": [ + 0.32899999618530273, + 0.3009999990463257, + 0.3580000102519989 + ], + "mrr": [ + 0.7887436151504517, + 0.7686433792114258, + 0.8075794577598572 + ], + "n_ret": 20.0, + "d_all": [ + 0.0430000014603138, + 0.017999999225139618, + 0.06800000369548798, + 0.0011998800119988001 + ], + "d_recall": [ + 0.044333335012197495, + 0.029914584010839462, + 0.058916665613651276, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.030881190672516823, + 0.012654127553105354, + 0.04804946482181549, + 0.0012998700129987 + ], + "d_all_holm": 0.0059994000599940004, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "Vector top-30 + cross-encoder", + "key": "rerank", + "variant": null, + "recall": [ + 0.6431666016578674, + 0.6265000104904175, + 0.6596665978431702 + ], + "all": [ + 0.3160000145435333, + 0.2879749834537506, + 0.3440000116825104 + ], + "mrr": [ + 0.8226098418235779, + 0.8032659292221069, + 0.842025101184845 + ], + "n_ret": 20.0, + "d_all": [ + 0.029999999329447746, + 0.017000000923871994, + 0.04399999976158142, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.0201666671782732, + 0.012500000186264515, + 0.028416665270924568, + 9.999000099990002e-05 + ], + "d_mrr": [ + 0.06474745273590088, + 0.04514456167817116, + 0.08422671258449554, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph chosen on dev: mention]", + "key": "lib_mh@best", + "variant": "mention", + "recall": [ + 0.6739166975021362, + 0.6558291912078857, + 0.691254198551178 + ], + "all": [ + 0.3869999945163727, + 0.3569999933242798, + 0.41499999165534973 + ], + "mrr": [ + 0.7572759985923767, + 0.7359405159950256, + 0.7778505086898804 + ], + "n_ret": 19.917999267578125, + "d_all": [ + 0.10100000351667404, + 0.07500000298023224, + 0.12700000405311584, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.05091666430234909, + 0.03758333623409271, + 0.06475000083446503, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0005863671540282667, + -0.0014113662764430046, + 0.0001893729786388576, + 0.1598840115988401 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention]", + "key": "ppr@best", + "variant": "mention", + "recall": [ + 0.6819999814033508, + 0.6639167070388794, + 0.6995000243186951 + ], + "all": [ + 0.3919999897480011, + 0.3619999885559082, + 0.421999990940094 + ], + "mrr": [ + 0.6927845478057861, + 0.6711047887802124, + 0.7145397067070007 + ], + "n_ret": 20.0, + "d_all": [ + 0.10599999874830246, + 0.08500000089406967, + 0.1289999932050705, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.05900000035762787, + 0.04741458222270012, + 0.07108332961797714, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.06507788598537445, + -0.08151596784591675, + -0.0494878925383091, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph chosen on dev: mention]", + "key": "lib_default@best", + "variant": "mention", + "recall": [ + 0.590416669845581, + 0.5709999799728394, + 0.6098437309265137 + ], + "all": [ + 0.2879999876022339, + 0.25999999046325684, + 0.3160000145435333 + ], + "mrr": [ + 0.7520973086357117, + 0.7301554679870605, + 0.7732515335083008 + ], + "n_ret": 14.204000473022461, + "d_all": [ + 0.0020000000949949026, + -0.02602500095963478, + 0.03200000151991844, + 0.942005799420058 + ], + "d_recall": [ + -0.032583337277173996, + -0.0500020831823349, + -0.014747917652130127, + 0.00039996000399960006 + ], + "d_mrr": [ + -0.00576509814709425, + -0.007635636255145073, + -0.004038706421852112, + 9.999000099990002e-05 + ], + "d_all_holm": 1.0, + "d_recall_holm": 0.0015998400159984002 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: mention]", + "key": "lib_mh:mention", + "variant": "mention", + "recall": [ + 0.6739166975021362, + 0.6558291912078857, + 0.691254198551178 + ], + "all": [ + 0.3869999945163727, + 0.3569999933242798, + 0.41499999165534973 + ], + "mrr": [ + 0.7572759985923767, + 0.7359405159950256, + 0.7778505086898804 + ], + "n_ret": 19.917999267578125, + "d_all": [ + 0.10100000351667404, + 0.07500000298023224, + 0.12700000405311584, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.05091666430234909, + 0.03758333623409271, + 0.06475000083446503, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.0005863671540282667, + -0.0014113662764430046, + 0.0001893729786388576, + 0.1598840115988401 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "ISONGraph multi_hop (tuned) [graph: ctrl_mention]", + "key": "lib_mh:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.6230000257492065, + 0.6059166789054871, + 0.6405041813850403 + ], + "all": [ + 0.28600001335144043, + 0.2590000033378601, + 0.3140000104904175 + ], + "mrr": [ + 0.7578672170639038, + 0.7367221713066101, + 0.7784227728843689 + ], + "n_ret": 20.0, + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_mrr": [ + 4.761904619954294e-06, + 0.0, + 1.9047618479817174e-05, + 1.0 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: mention]", + "key": "ppr:mention", + "variant": "mention", + "recall": [ + 0.6819999814033508, + 0.6639167070388794, + 0.6995000243186951 + ], + "all": [ + 0.3919999897480011, + 0.3619999885559082, + 0.421999990940094 + ], + "mrr": [ + 0.6927845478057861, + 0.6711047887802124, + 0.7145397067070007 + ], + "n_ret": 20.0, + "d_all": [ + 0.10599999874830246, + 0.08500000089406967, + 0.1289999932050705, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.05900000035762787, + 0.04741458222270012, + 0.07108332961797714, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.06507788598537445, + -0.08151596784591675, + -0.0494878925383091, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + }, + { + "label": "PPR re-rank (tuned, independent impl.) [graph: ctrl_mention]", + "key": "ppr:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.6230000257492065, + 0.6059166789054871, + 0.6405041813850403 + ], + "all": [ + 0.28600001335144043, + 0.2590000033378601, + 0.3140000104904175 + ], + "mrr": [ + 0.7607111930847168, + 0.7394207715988159, + 0.7810519933700562 + ], + "n_ret": 20.0, + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_mrr": [ + 0.002848809352144599, + -0.0005865177954547107, + 0.0069005051627755165, + 0.14578542145785423 + ], + "d_all_holm": 1.0, + "d_recall_holm": 1.0 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: mention]", + "key": "lib_default:mention", + "variant": "mention", + "recall": [ + 0.590416669845581, + 0.5709999799728394, + 0.6098437309265137 + ], + "all": [ + 0.2879999876022339, + 0.25999999046325684, + 0.3160000145435333 + ], + "mrr": [ + 0.7520973086357117, + 0.7301554679870605, + 0.7732515335083008 + ], + "n_ret": 14.204000473022461, + "d_all": [ + 0.0020000000949949026, + -0.02602500095963478, + 0.03200000151991844, + 0.942005799420058 + ], + "d_recall": [ + -0.032583337277173996, + -0.0500020831823349, + -0.014747917652130127, + 0.00039996000399960006 + ], + "d_mrr": [ + -0.00576509814709425, + -0.007635636255145073, + -0.004038706421852112, + 9.999000099990002e-05 + ], + "d_all_holm": 1.0, + "d_recall_holm": 0.0015998400159984002 + }, + { + "label": "ISONGraph multi_hop (library defaults) [graph: ctrl_mention]", + "key": "lib_default:ctrl_mention", + "variant": "ctrl_mention", + "recall": [ + 0.4830000102519989, + 0.4660812318325043, + 0.4999166429042816 + ], + "all": [ + 0.14300000667572021, + 0.12200000137090683, + 0.164000004529953 + ], + "mrr": [ + 0.7489333748817444, + 0.726815402507782, + 0.7703171372413635 + ], + "n_ret": 13.753999710083008, + "d_all": [ + -0.14300000667572021, + -0.16500000655651093, + -0.12200000137090683, + 9.999000099990002e-05 + ], + "d_recall": [ + -0.14000000059604645, + -0.1538333296775818, + -0.12666456401348114, + 9.999000099990002e-05 + ], + "d_mrr": [ + -0.00892908126115799, + -0.011042586527764797, + -0.007071079686284065, + 9.999000099990002e-05 + ], + "d_all_holm": 0.0011998800119988001, + "d_recall_holm": 0.0011998800119988001 + } + ] + }, + "subgraph": { + "10": [ + { + "variant": "mention", + "connect_max_hops": 2, + "size": 10.357999801635742, + "recall": [ + 0.612083375453949, + 0.5943312048912048, + 0.6296687722206116 + ], + "all": [ + 0.30000001192092896, + 0.27197501063346863, + 0.32702499628067017 + ], + "vec_matched_recall": [ + 0.5603333333333332, + 0.5431666666666666, + 0.5773354166666665 + ], + "vec_matched_all": [ + 0.212, + 0.187, + 0.237 + ], + "d_all": [ + 0.08799999952316284, + 0.06700000166893005, + 0.10999999940395355, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.051750000566244125, + 0.04008125513792038, + 0.06333541870117188, + 9.999000099990002e-05 + ] + }, + { + "variant": "mention", + "connect_max_hops": 3, + "size": 10.890999794006348, + "recall": [ + 0.6140000224113464, + 0.5962479114532471, + 0.6315833330154419 + ], + "all": [ + 0.30300000309944153, + 0.2750000059604645, + 0.3310000002384186 + ], + "vec_matched_recall": [ + 0.5656666666666667, + 0.5487479166666667, + 0.5826666666666667 + ], + "vec_matched_all": [ + 0.218, + 0.192, + 0.244 + ], + "d_all": [ + 0.08500000089406967, + 0.06300000101327896, + 0.1080000028014183, + 9.999000099990002e-05 + ], + "d_recall": [ + 0.04833333566784859, + 0.03641041740775108, + 0.060085419565439224, + 9.999000099990002e-05 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 2, + "size": 10.005000114440918, + "recall": [ + 0.5571666359901428, + 0.540164589881897, + 0.5741688013076782 + ], + "all": [ + 0.20900000631809235, + 0.1839749962091446, + 0.23399999737739563 + ], + "vec_matched_recall": [ + 0.5571666666666667, + 0.5401645833333334, + 0.5741687499999999 + ], + "vec_matched_all": [ + 0.209, + 0.183975, + 0.234 + ], + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 3.804763171189052e-09, + 3.3775966468141405e-09, + 4.241864193232914e-09, + 9.999000099990002e-05 + ] + }, + { + "variant": "ctrl_mention", + "connect_max_hops": 3, + "size": 10.017000198364258, + "recall": [ + 0.5571666359901428, + 0.540164589881897, + 0.5741688013076782 + ], + "all": [ + 0.20900000631809235, + 0.1839749962091446, + 0.23399999737739563 + ], + "vec_matched_recall": [ + 0.5571666666666667, + 0.5401645833333334, + 0.5741687499999999 + ], + "vec_matched_all": [ + 0.209, + 0.183975, + 0.234 + ], + "d_all": [ + 0.0, + 0.0, + 0.0, + 1.0 + ], + "d_recall": [ + 3.804763171189052e-09, + 3.3775966468141405e-09, + 4.241864193232914e-09, + 9.999000099990002e-05 + ] + } + ] + }, + "lib_minus_ppr": { + "mention": { + "5": [ + 0.014000000432133675, + -0.0010000000474974513, + 0.028999999165534973, + 0.09109089091090891 + ], + "10": [ + 0.004999999888241291, + -0.009999999776482582, + 0.019999999552965164, + 0.6238376162383762 + ], + "20": [ + -0.004999999888241291, + -0.019999999552965164, + 0.009999999776482582, + 0.6113388661133886 + ] + } + }, + "breakdown": { + "type": { + "2hop": { + "n": 519, + "vector": 0.33718690276145935, + "rrf": 0.37186896800994873, + "rerank": 0.3969171345233917, + "lib_mh@best": 0.48554912209510803, + "ppr@best": 0.49132949113845825 + }, + "3hop1": { + "n": 213, + "vector": 0.10798121988773346, + "rrf": 0.13615024089813232, + "rerank": 0.12206573039293289, + "lib_mh@best": 0.15023474395275116, + "ppr@best": 0.13615024089813232 + }, + "3hop2": { + "n": 78, + "vector": 0.14102564752101898, + "rrf": 0.12820513546466827, + "rerank": 0.1666666716337204, + "lib_mh@best": 0.1538461595773697, + "ppr@best": 0.08974359184503555 + }, + "4hop1": { + "n": 113, + "vector": 0.0, + "rrf": 0.008849557489156723, + "rerank": 0.008849557489156723, + "lib_mh@best": 0.0, + "ppr@best": 0.0 + }, + "4hop2": { + "n": 30, + "vector": 0.0, + "rrf": 0.0, + "rerank": 0.0, + "lib_mh@best": 0.0, + "ppr@best": 0.0 + }, + "4hop3": { + "n": 47, + "vector": 0.0, + "rrf": 0.0, + "rerank": 0.0, + "lib_mh@best": 0.0, + "ppr@best": 0.0 + } + }, + "hops": { + "2": { + "n": 519, + "vector": 0.33718690276145935, + "rrf": 0.37186896800994873, + "rerank": 0.3969171345233917, + "lib_mh@best": 0.48554912209510803, + "ppr@best": 0.49132949113845825 + }, + "3": { + "n": 291, + "vector": 0.1168384850025177, + "rrf": 0.1340206116437912, + "rerank": 0.1340206116437912, + "lib_mh@best": 0.1512027531862259, + "ppr@best": 0.12371134012937546 + }, + "4": { + "n": 190, + "vector": 0.0, + "rrf": 0.005263158120214939, + "rerank": 0.005263158120214939, + "lib_mh@best": 0.0, + "ppr@best": 0.0 + } + } + } + } + } + } +} \ No newline at end of file diff --git a/benchmarks/multihop/results/tables.md b/benchmarks/multihop/results/tables.md new file mode 100644 index 0000000..27c4bc2 --- /dev/null +++ b/benchmarks/multihop/results/tables.md @@ -0,0 +1,882 @@ + +## hotpotqa / minilm (corpus 14570 passages, 1000 test questions) + + +### k = 5 (returned-passage budget = 5) + +| Method | Recall@k % [95% CI] | All-supporting@k % [95% CI] | MRR@k [95% CI] | mean returned | Δ all vs vector (pp) [95% CI] | p (Holm) | Δ recall vs vector (pp) [95% CI] | p (Holm) | +|---|---|---|---|---|---|---|---|---| +| Vector top-k (plain) | 70.3 [68.4, 72.2] | 47.2 [44.1, 50.3] | 0.856 [0.836, 0.875] | 5.0 | - | - | - | - | +| BM25 | 72.2 [70.4, 73.9] | 48.7 [45.7, 51.7] | 0.846 [0.828, 0.862] | 5.0 | +1.5 [-2.1, +4.9] | 0.437 (1.000) | +1.9 [-0.3, +4.0] | 0.087 (0.612) | +| Hybrid vector+BM25 (RRF) | 75.6 [73.8, 77.3] | 54.3 [51.1, 57.4] | 0.867 [0.851, 0.884] | 5.0 | +7.1 [+4.6, +9.4] | <0.001 (0.002) | +5.3 [+3.8, +6.7] | <0.001 (0.002) | +| Vector top-30 + cross-encoder | 78.8 [77.0, 80.3] | 59.7 [56.6, 62.6] | 0.938 [0.924, 0.950] | 5.0 | +12.5 [+9.9, +15.2] | <0.001 (0.002) | +8.4 [+6.9, +10.0] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph chosen on dev: mention] | 82.3 [80.4, 84.1] | 71.1 [68.3, 73.8] | 0.855 [0.835, 0.874] | 5.0 | +23.9 [+21.0, +26.9] | <0.001 (0.002) | +12.0 [+10.5, +13.5] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention] | 78.7 [76.8, 80.5] | 63.6 [60.8, 66.4] | 0.763 [0.742, 0.784] | 5.0 | +16.4 [+13.4, +19.5] | <0.001 (0.002) | +8.4 [+6.8, +10.1] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph chosen on dev: mention] | 80.4 [78.6, 82.2] | 67.6 [64.8, 70.4] | 0.856 [0.836, 0.874] | 5.0 | +20.4 [+17.8, +23.1] | <0.001 (0.002) | +10.1 [+8.9, +11.5] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: mention] | 82.3 [80.4, 84.1] | 71.1 [68.3, 73.8] | 0.855 [0.835, 0.874] | 5.0 | +23.9 [+21.0, +26.9] | <0.001 (0.002) | +12.0 [+10.5, +13.5] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: entity] | 74.9 [73.0, 76.8] | 56.2 [53.1, 59.2] | 0.856 [0.836, 0.875] | 5.0 | +9.0 [+6.5, +11.5] | <0.001 (0.002) | +4.6 [+3.3, +5.9] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: knn] | 70.6 [68.8, 72.5] | 47.8 [44.8, 51.0] | 0.856 [0.836, 0.875] | 5.0 | +0.6 [-0.6, +1.8] | 0.393 (1.000) | +0.3 [-0.3, +0.9] | 0.393 (1.000) | +| ISONGraph multi_hop (tuned) [graph: ctrl_mention] | 70.3 [68.4, 72.2] | 47.2 [44.1, 50.3] | 0.856 [0.836, 0.875] | 5.0 | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +| PPR re-rank (tuned, independent impl.) [graph: mention] | 78.7 [76.8, 80.5] | 63.6 [60.8, 66.4] | 0.763 [0.742, 0.784] | 5.0 | +16.4 [+13.4, +19.5] | <0.001 (0.002) | +8.4 [+6.8, +10.1] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph: entity] | 70.1 [68.2, 72.0] | 47.3 [44.2, 50.4] | 0.765 [0.743, 0.786] | 5.0 | +0.1 [-1.4, +1.5] | 1.000 (1.000) | -0.2 [-1.0, +0.6] | 0.719 (1.000) | +| PPR re-rank (tuned, independent impl.) [graph: knn] | 70.3 [68.4, 72.2] | 47.2 [44.1, 50.3] | 0.814 [0.794, 0.834] | 5.0 | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +| PPR re-rank (tuned, independent impl.) [graph: ctrl_mention] | 70.2 [68.4, 72.1] | 47.1 [44.0, 50.2] | 0.836 [0.816, 0.854] | 5.0 | -0.1 [-0.3, +0.0] | 1.000 (1.000) | -0.1 [-0.2, +0.0] | 1.000 (1.000) | +| ISONGraph multi_hop (library defaults) [graph: mention] | 80.4 [78.6, 82.2] | 67.6 [64.8, 70.4] | 0.856 [0.836, 0.874] | 5.0 | +20.4 [+17.8, +23.1] | <0.001 (0.002) | +10.1 [+8.9, +11.5] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: entity] | 70.4 [68.5, 72.4] | 47.5 [44.3, 50.6] | 0.855 [0.836, 0.874] | 5.0 | +0.3 [-2.5, +2.9] | 0.883 (1.000) | +0.1 [-1.3, +1.4] | 0.945 (1.000) | +| ISONGraph multi_hop (library defaults) [graph: knn] | 66.7 [64.8, 68.5] | 40.4 [37.3, 43.5] | 0.855 [0.834, 0.874] | 5.0 | -6.8 [-9.4, -4.3] | <0.001 (0.002) | -3.6 [-4.9, -2.3] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: ctrl_mention] | 68.3 [66.4, 70.1] | 43.3 [40.3, 46.5] | 0.855 [0.835, 0.874] | 5.0 | -3.9 [-5.2, -2.7] | <0.001 (0.002) | -2.1 [-2.7, -1.4] | <0.001 (0.002) | + +#### semantic_subgraph(connect=True), top_k_results=5: set quality vs plain vector top-|S| (size-matched per query) + +| Graph | connect_max_hops | mean |S| | Recall % | All-supp. % | Vector@|S| recall % | Vector@|S| all % | Δ all (pp) [CI] | p | Δ recall (pp) [CI] | p | +|---|---|---|---|---|---|---|---|---|---|---| +| mention | 2 | 5.0 | 82.3 [80.5, 84.1] | 71.2 [68.3, 73.8] | 70.3 [68.5, 72.2] | 47.3 [44.3, 50.4] | +23.9 [+21.0, +26.9] | <0.001 | +12.0 [+10.5, +13.5] | <0.001 | +| mention | 3 | 5.0 | 82.3 [80.5, 84.1] | 71.2 [68.3, 73.8] | 70.3 [68.5, 72.2] | 47.3 [44.3, 50.4] | +23.9 [+21.0, +26.9] | <0.001 | +12.0 [+10.5, +13.5] | <0.001 | +| entity | 2 | 5.1 | 75.2 [73.3, 77.1] | 56.7 [53.7, 59.8] | 70.5 [68.7, 72.4] | 47.6 [44.5, 50.7] | +9.1 [+6.6, +11.6] | <0.001 | +4.7 [+3.3, +6.0] | <0.001 | +| entity | 3 | 5.4 | 75.2 [73.4, 77.2] | 56.9 [53.9, 59.9] | 70.9 [69.0, 72.7] | 48.0 [45.0, 51.1] | +8.9 [+6.4, +11.5] | <0.001 | +4.4 [+3.1, +5.8] | <0.001 | +| knn | 2 | 5.4 | 71.1 [69.3, 73.0] | 48.8 [45.8, 52.0] | 70.8 [69.0, 72.7] | 48.1 [45.1, 51.2] | +0.7 [-0.6, +2.0] | 0.383 | +0.4 [-0.4, +1.0] | 0.394 | +| knn | 3 | 6.0 | 71.5 [69.7, 73.4] | 49.3 [46.3, 52.4] | 71.8 [70.0, 73.7] | 49.4 [46.3, 52.5] | -0.1 [-1.6, +1.4] | 1.000 | -0.3 [-1.1, +0.5] | 0.563 | +| ctrl_mention | 2 | 5.0 | 70.3 [68.4, 72.2] | 47.2 [44.1, 50.3] | 70.3 [68.5, 72.2] | 47.2 [44.1, 50.3] | +0.0 [+0.0, +0.0] | 1.000 | +0.0 [+0.0, +0.0] | 1.000 | +| ctrl_mention | 3 | 5.0 | 70.3 [68.4, 72.2] | 47.2 [44.1, 50.3] | 70.3 [68.5, 72.2] | 47.2 [44.1, 50.3] | +0.0 [+0.0, +0.0] | 1.000 | +0.0 [+0.0, +0.0] | 1.000 | + +### k = 10 (returned-passage budget = 10) + +| Method | Recall@k % [95% CI] | All-supporting@k % [95% CI] | MRR@k [95% CI] | mean returned | Δ all vs vector (pp) [95% CI] | p (Holm) | Δ recall vs vector (pp) [95% CI] | p (Holm) | +|---|---|---|---|---|---|---|---|---| +| Vector top-k (plain) | 77.2 [75.5, 79.0] | 58.2 [55.2, 61.3] | 0.860 [0.841, 0.878] | 10.0 | - | - | - | - | +| BM25 | 84.1 [82.6, 85.6] | 69.1 [66.3, 71.9] | 0.851 [0.834, 0.867] | 10.0 | +10.9 [+7.7, +14.2] | <0.001 (0.002) | +6.9 [+5.2, +8.7] | <0.001 (0.002) | +| Hybrid vector+BM25 (RRF) | 85.9 [84.5, 87.3] | 72.5 [69.7, 75.1] | 0.871 [0.855, 0.887] | 10.0 | +14.3 [+11.8, +16.7] | <0.001 (0.002) | +8.7 [+7.4, +10.0] | <0.001 (0.002) | +| Vector top-30 + cross-encoder | 83.5 [82.0, 85.0] | 68.4 [65.7, 71.1] | 0.939 [0.926, 0.951] | 10.0 | +10.2 [+8.3, +12.2] | <0.001 (0.002) | +6.2 [+5.2, +7.3] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph chosen on dev: mention] | 88.7 [87.0, 90.1] | 81.5 [79.1, 83.8] | 0.859 [0.840, 0.878] | 9.4 | +23.3 [+20.4, +26.4] | <0.001 (0.002) | +11.4 [+9.8, +13.0] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention] | 88.7 [87.2, 90.2] | 81.4 [79.0, 83.7] | 0.845 [0.828, 0.863] | 10.0 | +23.2 [+20.5, +26.1] | <0.001 (0.002) | +11.5 [+10.1, +13.0] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph chosen on dev: mention] | 85.8 [84.0, 87.4] | 77.3 [74.7, 79.8] | 0.857 [0.838, 0.876] | 7.9 | +19.1 [+16.2, +22.3] | <0.001 (0.002) | +8.6 [+6.9, +10.2] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: mention] | 88.7 [87.0, 90.1] | 81.5 [79.1, 83.8] | 0.859 [0.840, 0.878] | 9.4 | +23.3 [+20.4, +26.4] | <0.001 (0.002) | +11.4 [+9.8, +13.0] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: entity] | 84.5 [82.8, 86.1] | 71.8 [69.0, 74.5] | 0.861 [0.842, 0.879] | 10.0 | +13.6 [+11.2, +16.0] | <0.001 (0.002) | +7.2 [+6.0, +8.5] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: knn] | 78.9 [77.1, 80.5] | 62.2 [59.2, 65.3] | 0.859 [0.839, 0.877] | 10.0 | +4.0 [+1.9, +6.2] | <0.001 (0.002) | +1.6 [+0.5, +2.7] | 0.009 (0.046) | +| ISONGraph multi_hop (tuned) [graph: ctrl_mention] | 77.2 [75.5, 79.0] | 58.2 [55.2, 61.3] | 0.860 [0.841, 0.878] | 10.0 | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +| PPR re-rank (tuned, independent impl.) [graph: mention] | 88.7 [87.2, 90.2] | 81.4 [79.0, 83.7] | 0.845 [0.828, 0.863] | 10.0 | +23.2 [+20.5, +26.1] | <0.001 (0.002) | +11.5 [+10.1, +13.0] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph: entity] | 81.9 [80.2, 83.5] | 67.4 [64.6, 70.2] | 0.837 [0.819, 0.856] | 10.0 | +9.2 [+7.1, +11.5] | <0.001 (0.002) | +4.7 [+3.5, +5.8] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph: knn] | 79.0 [77.4, 80.8] | 62.2 [59.3, 65.2] | 0.825 [0.806, 0.844] | 10.0 | +4.0 [+2.1, +5.9] | <0.001 (0.002) | +1.8 [+0.8, +2.8] | 0.001 (0.008) | +| PPR re-rank (tuned, independent impl.) [graph: ctrl_mention] | 77.3 [75.6, 79.0] | 58.3 [55.4, 61.3] | 0.860 [0.840, 0.878] | 10.0 | +0.1 [-0.5, +0.7] | 1.000 (1.000) | +0.1 [-0.2, +0.4] | 1.000 (1.000) | +| ISONGraph multi_hop (library defaults) [graph: mention] | 85.8 [84.0, 87.4] | 77.3 [74.7, 79.8] | 0.857 [0.838, 0.876] | 7.9 | +19.1 [+16.2, +22.3] | <0.001 (0.002) | +8.6 [+6.9, +10.2] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: entity] | 78.9 [77.1, 80.8] | 62.8 [59.8, 65.8] | 0.858 [0.839, 0.876] | 10.0 | +4.6 [+1.6, +7.7] | 0.004 (0.017) | +1.7 [+0.1, +3.3] | 0.046 (0.184) | +| ISONGraph multi_hop (library defaults) [graph: knn] | 76.3 [74.4, 78.1] | 57.7 [54.6, 60.7] | 0.857 [0.837, 0.876] | 10.0 | -0.5 [-3.2, +2.3] | 0.772 (1.000) | -0.9 [-2.3, +0.5] | 0.224 (0.673) | +| ISONGraph multi_hop (library defaults) [graph: ctrl_mention] | 70.2 [68.4, 72.1] | 47.1 [44.1, 50.2] | 0.855 [0.835, 0.874] | 9.1 | -11.1 [-13.1, -9.2] | <0.001 (0.002) | -7.0 [-8.3, -5.9] | <0.001 (0.002) | + +#### semantic_subgraph(connect=True), top_k_results=10: set quality vs plain vector top-|S| (size-matched per query) + +| Graph | connect_max_hops | mean |S| | Recall % | All-supp. % | Vector@|S| recall % | Vector@|S| all % | Δ all (pp) [CI] | p | Δ recall (pp) [CI] | p | +|---|---|---|---|---|---|---|---|---|---|---| +| mention | 2 | 9.5 | 88.8 [87.2, 90.3] | 81.7 [79.3, 84.0] | 76.5 [74.8, 78.3] | 57.1 [54.1, 60.1] | +24.6 [+21.8, +27.7] | <0.001 | +12.3 [+10.8, +13.9] | <0.001 | +| mention | 3 | 9.5 | 88.8 [87.2, 90.3] | 81.8 [79.4, 84.1] | 76.6 [74.8, 78.3] | 57.1 [54.1, 60.1] | +24.7 [+21.8, +27.8] | <0.001 | +12.3 [+10.8, +13.9] | <0.001 | +| entity | 2 | 10.3 | 84.7 [83.0, 86.3] | 72.1 [69.4, 74.8] | 77.5 [75.8, 79.2] | 58.6 [55.6, 61.6] | +13.5 [+11.1, +16.0] | <0.001 | +7.2 [+6.0, +8.4] | <0.001 | +| entity | 3 | 11.5 | 84.9 [83.2, 86.4] | 72.3 [69.6, 75.0] | 78.5 [76.7, 80.1] | 60.0 [57.0, 63.0] | +12.3 [+9.7, +14.9] | <0.001 | +6.4 [+5.1, +7.8] | <0.001 | +| knn | 2 | 10.4 | 78.9 [77.1, 80.5] | 62.2 [59.2, 65.3] | 77.5 [75.8, 79.2] | 58.7 [55.7, 61.8] | +3.5 [+1.4, +5.7] | 0.002 | +1.3 [+0.2, +2.4] | 0.034 | +| knn | 3 | 11.2 | 78.9 [77.1, 80.6] | 62.3 [59.3, 65.3] | 78.2 [76.5, 80.0] | 59.7 [56.7, 62.8] | +2.6 [+0.5, +4.9] | 0.028 | +0.7 [-0.4, +1.9] | 0.285 | +| ctrl_mention | 2 | 10.0 | 77.2 [75.5, 79.0] | 58.2 [55.2, 61.3] | 77.2 [75.5, 79.0] | 58.2 [55.2, 61.3] | +0.0 [+0.0, +0.0] | 1.000 | +0.0 [+0.0, +0.0] | 1.000 | +| ctrl_mention | 3 | 10.0 | 77.2 [75.5, 79.0] | 58.2 [55.2, 61.3] | 77.2 [75.5, 79.0] | 58.2 [55.2, 61.3] | +0.0 [+0.0, +0.0] | 1.000 | +0.0 [+0.0, +0.0] | 1.000 | + +### k = 20 (returned-passage budget = 20) + +| Method | Recall@k % [95% CI] | All-supporting@k % [95% CI] | MRR@k [95% CI] | mean returned | Δ all vs vector (pp) [95% CI] | p (Holm) | Δ recall vs vector (pp) [95% CI] | p (Holm) | +|---|---|---|---|---|---|---|---|---| +| Vector top-k (plain) | 82.2 [80.8, 83.9] | 66.2 [63.4, 69.1] | 0.861 [0.842, 0.879] | 20.0 | - | - | - | - | +| BM25 | 91.0 [89.8, 92.2] | 82.2 [79.8, 84.4] | 0.851 [0.835, 0.867] | 20.0 | +16.0 [+13.1, +19.2] | <0.001 (0.002) | +8.7 [+7.2, +10.4] | <0.001 (0.002) | +| Hybrid vector+BM25 (RRF) | 91.8 [90.6, 92.9] | 83.8 [81.6, 86.0] | 0.871 [0.855, 0.887] | 20.0 | +17.6 [+15.1, +20.1] | <0.001 (0.002) | +9.5 [+8.2, +10.8] | <0.001 (0.002) | +| Vector top-30 + cross-encoder | 84.5 [83.1, 86.0] | 70.3 [67.7, 73.0] | 0.939 [0.926, 0.951] | 20.0 | +4.1 [+2.9, +5.4] | <0.001 (0.002) | +2.3 [+1.6, +2.9] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph chosen on dev: mention] | 92.5 [91.3, 93.8] | 87.3 [85.2, 89.3] | 0.859 [0.840, 0.877] | 19.8 | +21.1 [+18.4, +23.7] | <0.001 (0.002) | +10.3 [+8.9, +11.6] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention] | 92.7 [91.4, 93.9] | 87.3 [85.3, 89.3] | 0.817 [0.798, 0.836] | 20.0 | +21.1 [+18.6, +23.6] | <0.001 (0.002) | +10.5 [+9.1, +11.7] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph chosen on dev: mention] | 85.9 [84.1, 87.6] | 77.5 [74.9, 80.0] | 0.857 [0.838, 0.876] | 9.4 | +11.3 [+8.3, +14.5] | <0.001 (0.002) | +3.6 [+1.8, +5.4] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: mention] | 92.5 [91.3, 93.8] | 87.3 [85.2, 89.3] | 0.859 [0.840, 0.877] | 19.8 | +21.1 [+18.4, +23.7] | <0.001 (0.002) | +10.3 [+8.9, +11.6] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: entity] | 88.7 [87.3, 90.1] | 79.2 [76.7, 81.8] | 0.861 [0.842, 0.879] | 20.0 | +13.0 [+10.5, +15.5] | <0.001 (0.002) | +6.4 [+5.2, +7.8] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: knn] | 85.2 [83.6, 86.7] | 72.5 [69.8, 75.1] | 0.860 [0.841, 0.878] | 20.0 | +6.3 [+4.2, +8.4] | <0.001 (0.002) | +3.0 [+1.9, +4.1] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: ctrl_mention] | 82.2 [80.8, 83.9] | 66.2 [63.4, 69.1] | 0.861 [0.842, 0.879] | 20.0 | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +| PPR re-rank (tuned, independent impl.) [graph: mention] | 92.7 [91.4, 93.9] | 87.3 [85.3, 89.3] | 0.817 [0.798, 0.836] | 20.0 | +21.1 [+18.6, +23.6] | <0.001 (0.002) | +10.5 [+9.1, +11.7] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph: entity] | 87.9 [86.5, 89.4] | 78.0 [75.4, 80.6] | 0.814 [0.794, 0.833] | 20.0 | +11.8 [+9.7, +14.1] | <0.001 (0.002) | +5.7 [+4.6, +6.9] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph: knn] | 84.2 [82.6, 85.8] | 71.2 [68.5, 73.9] | 0.802 [0.782, 0.821] | 20.0 | +5.0 [+2.9, +7.0] | <0.001 (0.002) | +2.0 [+0.9, +3.1] | 0.001 (0.007) | +| PPR re-rank (tuned, independent impl.) [graph: ctrl_mention] | 82.2 [80.7, 83.8] | 66.1 [63.3, 69.0] | 0.857 [0.839, 0.875] | 20.0 | -0.1 [-0.3, +0.0] | 1.000 (1.000) | -0.1 [-0.2, +0.0] | 1.000 (1.000) | +| ISONGraph multi_hop (library defaults) [graph: mention] | 85.9 [84.1, 87.6] | 77.5 [74.9, 80.0] | 0.857 [0.838, 0.876] | 9.4 | +11.3 [+8.3, +14.5] | <0.001 (0.002) | +3.6 [+1.8, +5.4] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: entity] | 83.9 [82.2, 85.5] | 71.7 [68.9, 74.4] | 0.859 [0.840, 0.877] | 19.8 | +5.5 [+2.4, +8.6] | <0.001 (0.003) | +1.6 [+0.0, +3.3] | 0.056 (0.226) | +| ISONGraph multi_hop (library defaults) [graph: knn] | 81.1 [79.4, 82.8] | 66.1 [63.1, 69.0] | 0.858 [0.838, 0.876] | 20.0 | -0.1 [-2.9, +2.7] | 1.000 (1.000) | -1.2 [-2.6, +0.4] | 0.149 (0.446) | +| ISONGraph multi_hop (library defaults) [graph: ctrl_mention] | 70.3 [68.4, 72.2] | 47.2 [44.1, 50.3] | 0.855 [0.835, 0.874] | 11.6 | -19.0 [-21.4, -16.6] | <0.001 (0.002) | -11.9 [-13.4, -10.5] | <0.001 (0.002) | + +#### semantic_subgraph(connect=True), top_k_results=20: set quality vs plain vector top-|S| (size-matched per query) + +| Graph | connect_max_hops | mean |S| | Recall % | All-supp. % | Vector@|S| recall % | Vector@|S| all % | Δ all (pp) [CI] | p | Δ recall (pp) [CI] | p | +|---|---|---|---|---|---|---|---|---|---|---| +| mention | 2 | 20.0 | 92.5 [91.3, 93.8] | 87.3 [85.2, 89.3] | 82.3 [80.8, 83.9] | 66.3 [63.5, 69.2] | +21.0 [+18.3, +23.6] | <0.001 | +10.2 [+8.8, +11.6] | <0.001 | +| mention | 3 | 20.4 | 92.7 [91.4, 93.9] | 87.4 [85.3, 89.4] | 82.5 [80.9, 84.0] | 66.5 [63.6, 69.4] | +20.9 [+18.3, +23.5] | <0.001 | +10.2 [+8.8, +11.6] | <0.001 | +| entity | 2 | 20.7 | 88.8 [87.4, 90.2] | 79.4 [76.9, 82.0] | 82.5 [81.0, 84.0] | 66.6 [63.8, 69.5] | +12.8 [+10.3, +15.3] | <0.001 | +6.3 [+5.0, +7.6] | <0.001 | +| entity | 3 | 23.7 | 88.8 [87.4, 90.2] | 79.4 [76.9, 82.0] | 83.2 [81.7, 84.7] | 68.0 [65.2, 70.9] | +11.4 [+8.8, +14.0] | <0.001 | +5.6 [+4.3, +7.0] | <0.001 | +| knn | 2 | 21.3 | 85.4 [83.9, 86.9] | 72.9 [70.3, 75.5] | 82.7 [81.2, 84.2] | 66.9 [64.0, 69.8] | +6.0 [+3.9, +8.1] | <0.001 | +2.7 [+1.6, +3.8] | <0.001 | +| knn | 3 | 23.4 | 85.5 [84.0, 87.0] | 73.0 [70.4, 75.5] | 83.2 [81.7, 84.9] | 68.1 [65.3, 71.0] | +4.9 [+2.8, +7.0] | <0.001 | +2.3 [+1.2, +3.4] | <0.001 | +| ctrl_mention | 2 | 20.1 | 82.2 [80.8, 83.9] | 66.2 [63.4, 69.1] | 82.2 [80.8, 83.9] | 66.2 [63.4, 69.1] | +0.0 [+0.0, +0.0] | 1.000 | +0.0 [+0.0, +0.0] | 1.000 | +| ctrl_mention | 3 | 20.3 | 82.2 [80.8, 83.9] | 66.2 [63.4, 69.1] | 82.3 [80.8, 84.0] | 66.4 [63.6, 69.3] | -0.2 [-0.5, +0.0] | 0.494 | -0.1 [-0.2, +0.0] | 0.494 | + +#### hotpotqa/minilm: ISONGraph multi_hop (tuned) minus PPR re-rank (tuned), same graph; all-supporting (pp [95% CI], paired p) + +| Graph | k=5 | k=10 | k=20 | +|---|---|---|---| +| mention | +7.5 [+4.9, +10.0] (p=<0.001) | +0.1 [-1.4, +1.5] (p=1.000) | +0.0 [-1.1, +1.1] (p=1.000) | +| entity | +8.9 [+6.5, +11.3] (p=<0.001) | +4.4 [+2.5, +6.4] (p=<0.001) | +1.2 [-0.3, +2.7] (p=0.145) | +| knn | +0.6 [-0.6, +1.8] (p=0.393) | +0.0 [-1.5, +1.6] (p=1.000) | +1.3 [+0.0, +2.7] (p=0.069) | + +#### hotpotqa/minilm: breakdown by question type, k=10 -- All-supporting@10 % (Δ vs vector, pp [95% CI]) + +| Group | n | Vector top-k (plain) | Hybrid vector+BM25 (RRF) | Vector top-30 + cross-encoder | ISONGraph multi_hop (tuned) | PPR re-rank (tuned, independent impl.) | +|---|---|---|---|---|---|---| +| bridge | 784 | 51.3 | 66.1 (+14.8 [+12.0, +17.7]) | 62.9 (+11.6 [+9.4, +14.0]) | 82.5 (+31.2 [+27.7, +34.7]) [mention] | 81.6 (+30.4 [+27.0, +33.5]) [mention] | +| comparison | 216 | 83.3 | 95.8 (+12.5 [+8.3, +17.1]) | 88.4 (+5.1 [+2.3, +8.3]) | 77.8 (-5.6 [-9.3, -2.3]) [mention] | 80.6 (-2.8 [-6.0, +0.0]) [mention] | + +#### hotpotqa/minilm: breakdown by hop count (number of supporting passages), k=10 -- All-supporting@10 % (Δ vs vector, pp [95% CI]) + +| Group | n | Vector top-k (plain) | Hybrid vector+BM25 (RRF) | Vector top-30 + cross-encoder | ISONGraph multi_hop (tuned) | PPR re-rank (tuned, independent impl.) | +|---|---|---|---|---|---|---| +| 2 | 1000 | 58.2 | 72.5 (+14.3 [+11.9, +16.8]) | 68.4 (+10.2 [+8.3, +12.3]) | 81.5 (+23.3 [+20.4, +26.2]) [mention] | 81.4 (+23.2 [+20.6, +26.1]) [mention] | + +#### hotpotqa/minilm: dev vs test All-supporting@10 for tuned configs (selection optimism) + +| Method | Graph | dev % | test % | +|---|---|---|---| +| lib_mh | mention | 82.2 | 81.5 | +| lib_mh | entity | 69.8 | 71.8 | +| lib_mh | knn | 61.0 | 62.2 | +| lib_mh | ctrl_mention | 55.6 | 58.2 | +| ppr | mention | 81.6 | 81.4 | +| ppr | entity | 66.4 | 67.4 | +| ppr | knn | 59.2 | 62.2 | +| ppr | ctrl_mention | 55.6 | 58.3 | +| vector | - | 55.6 | 58.2 | + +## hotpotqa / bge (corpus 14570 passages, 1000 test questions) + + +### k = 5 (returned-passage budget = 5) + +| Method | Recall@k % [95% CI] | All-supporting@k % [95% CI] | MRR@k [95% CI] | mean returned | Δ all vs vector (pp) [95% CI] | p (Holm) | Δ recall vs vector (pp) [95% CI] | p (Holm) | +|---|---|---|---|---|---|---|---|---| +| Vector top-k (plain) | 83.1 [81.4, 84.6] | 68.4 [65.5, 71.2] | 0.924 [0.910, 0.938] | 5.0 | - | - | - | - | +| BM25 | 72.2 [70.4, 73.9] | 48.7 [45.7, 51.7] | 0.846 [0.828, 0.862] | 5.0 | -19.7 [-23.0, -16.3] | <0.001 (0.001) | -10.8 [-12.7, -8.9] | <0.001 (0.001) | +| Hybrid vector+BM25 (RRF) | 84.0 [82.6, 85.5] | 69.1 [66.3, 71.9] | 0.914 [0.900, 0.927] | 5.0 | +0.7 [-1.7, +3.2] | 0.621 (1.000) | +1.0 [-0.3, +2.3] | 0.158 (0.790) | +| Vector top-30 + cross-encoder | 85.3 [83.8, 86.8] | 71.6 [68.8, 74.4] | 0.950 [0.939, 0.961] | 5.0 | +3.2 [+0.3, +6.0] | 0.028 (0.166) | +2.3 [+0.7, +3.8] | 0.004 (0.026) | +| ISONGraph multi_hop (tuned) [graph chosen on dev: mention] | 89.5 [88.1, 90.9] | 81.3 [78.8, 83.6] | 0.924 [0.910, 0.938] | 5.0 | +12.9 [+10.8, +15.2] | <0.001 (0.001) | +6.4 [+5.4, +7.6] | <0.001 (0.001) | +| PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention] | 88.5 [87.0, 90.0] | 79.6 [77.1, 82.0] | 0.837 [0.819, 0.854] | 5.0 | +11.2 [+8.7, +13.7] | <0.001 (0.001) | +5.4 [+4.2, +6.8] | <0.001 (0.001) | +| ISONGraph multi_hop (library defaults) [graph chosen on dev: mention] | 85.5 [84.0, 87.1] | 73.4 [70.8, 76.1] | 0.924 [0.910, 0.938] | 5.0 | +5.0 [+3.6, +6.5] | <0.001 (0.001) | +2.5 [+1.8, +3.2] | <0.001 (0.001) | +| ISONGraph multi_hop (tuned) [graph: mention] | 89.5 [88.1, 90.9] | 81.3 [78.8, 83.6] | 0.924 [0.910, 0.938] | 5.0 | +12.9 [+10.8, +15.2] | <0.001 (0.001) | +6.4 [+5.4, +7.6] | <0.001 (0.001) | +| ISONGraph multi_hop (tuned) [graph: knn] | 83.5 [81.9, 85.1] | 69.3 [66.5, 72.2] | 0.924 [0.910, 0.938] | 5.0 | +0.9 [-1.1, +2.8] | 0.425 (1.000) | +0.4 [-0.5, +1.4] | 0.425 (1.000) | +| ISONGraph multi_hop (tuned) [graph: ctrl_mention] | 82.6 [80.9, 84.1] | 67.5 [64.5, 70.4] | 0.924 [0.910, 0.938] | 5.0 | -0.9 [-1.5, -0.4] | 0.003 (0.026) | -0.4 [-0.7, -0.2] | 0.003 (0.026) | +| PPR re-rank (tuned, independent impl.) [graph: mention] | 88.5 [87.0, 90.0] | 79.6 [77.1, 82.0] | 0.837 [0.819, 0.854] | 5.0 | +11.2 [+8.7, +13.7] | <0.001 (0.001) | +5.4 [+4.2, +6.8] | <0.001 (0.001) | +| PPR re-rank (tuned, independent impl.) [graph: knn] | 83.0 [81.4, 84.6] | 67.9 [64.9, 70.9] | 0.917 [0.902, 0.931] | 5.0 | -0.5 [-1.7, +0.7] | 0.505 (1.000) | -0.1 [-0.7, +0.6] | 1.000 (1.000) | +| PPR re-rank (tuned, independent impl.) [graph: ctrl_mention] | 83.1 [81.6, 84.7] | 68.4 [65.6, 71.3] | 0.924 [0.910, 0.938] | 5.0 | +0.0 [-1.1, +1.1] | 1.000 (1.000) | +0.1 [-0.5, +0.7] | 1.000 (1.000) | +| ISONGraph multi_hop (library defaults) [graph: mention] | 85.5 [84.0, 87.1] | 73.4 [70.8, 76.1] | 0.924 [0.910, 0.938] | 5.0 | +5.0 [+3.6, +6.5] | <0.001 (0.001) | +2.5 [+1.8, +3.2] | <0.001 (0.001) | +| ISONGraph multi_hop (library defaults) [graph: knn] | 83.1 [81.5, 84.6] | 68.5 [65.7, 71.3] | 0.924 [0.910, 0.938] | 5.0 | +0.1 [-1.1, +1.3] | 1.000 (1.000) | +0.1 [-0.5, +0.7] | 1.000 (1.000) | +| ISONGraph multi_hop (library defaults) [graph: ctrl_mention] | 82.6 [80.9, 84.1] | 67.5 [64.5, 70.4] | 0.924 [0.910, 0.938] | 5.0 | -0.9 [-1.5, -0.4] | 0.003 (0.026) | -0.4 [-0.7, -0.2] | 0.003 (0.026) | + +#### semantic_subgraph(connect=True), top_k_results=5: set quality vs plain vector top-|S| (size-matched per query) + +| Graph | connect_max_hops | mean |S| | Recall % | All-supp. % | Vector@|S| recall % | Vector@|S| all % | Δ all (pp) [CI] | p | Δ recall (pp) [CI] | p | +|---|---|---|---|---|---|---|---|---|---|---| +| mention | 2 | 5.0 | 89.8 [88.3, 91.1] | 81.8 [79.4, 84.2] | 83.2 [81.7, 84.9] | 68.7 [65.8, 71.6] | +13.1 [+11.0, +15.5] | <0.001 | +6.5 [+5.5, +7.8] | <0.001 | +| mention | 3 | 5.0 | 89.8 [88.3, 91.1] | 81.8 [79.4, 84.2] | 83.2 [81.7, 84.9] | 68.7 [65.8, 71.6] | +13.1 [+11.0, +15.5] | <0.001 | +6.5 [+5.5, +7.8] | <0.001 | +| knn | 2 | 5.2 | 84.2 [82.6, 85.8] | 70.7 [67.9, 73.6] | 83.6 [82.0, 85.2] | 69.1 [66.1, 71.9] | +1.6 [-0.5, +3.7] | 0.153 | +0.6 [-0.5, +1.7] | 0.306 | +| knn | 3 | 5.6 | 84.5 [83.0, 86.2] | 71.4 [68.5, 74.3] | 84.0 [82.5, 85.5] | 69.7 [66.8, 72.4] | +1.7 [-0.5, +3.8] | 0.136 | +0.5 [-0.5, +1.6] | 0.367 | +| ctrl_mention | 2 | 5.0 | 82.6 [80.9, 84.1] | 67.5 [64.5, 70.4] | 83.0 [81.5, 84.6] | 68.4 [65.5, 71.2] | -0.9 [-1.5, -0.4] | 0.003 | -0.4 [-0.7, -0.2] | 0.003 | +| ctrl_mention | 3 | 5.0 | 82.6 [80.9, 84.1] | 67.5 [64.5, 70.4] | 83.0 [81.5, 84.6] | 68.4 [65.5, 71.2] | -0.9 [-1.5, -0.4] | 0.003 | -0.4 [-0.7, -0.2] | 0.003 | + +### k = 10 (returned-passage budget = 10) + +| Method | Recall@k % [95% CI] | All-supporting@k % [95% CI] | MRR@k [95% CI] | mean returned | Δ all vs vector (pp) [95% CI] | p (Holm) | Δ recall vs vector (pp) [95% CI] | p (Holm) | +|---|---|---|---|---|---|---|---|---| +| Vector top-k (plain) | 88.6 [87.2, 89.9] | 78.5 [76.0, 80.9] | 0.925 [0.912, 0.939] | 10.0 | - | - | - | - | +| BM25 | 84.1 [82.6, 85.6] | 69.1 [66.3, 71.9] | 0.851 [0.834, 0.867] | 10.0 | -9.4 [-12.2, -6.4] | <0.001 (0.001) | -4.4 [-5.9, -2.8] | <0.001 (0.001) | +| Hybrid vector+BM25 (RRF) | 91.9 [90.6, 92.9] | 83.8 [81.4, 86.0] | 0.915 [0.902, 0.928] | 10.0 | +5.3 [+3.3, +7.3] | <0.001 (0.001) | +3.3 [+2.2, +4.4] | <0.001 (0.001) | +| Vector top-30 + cross-encoder | 91.9 [90.6, 93.0] | 84.1 [81.9, 86.3] | 0.951 [0.940, 0.962] | 10.0 | +5.6 [+3.6, +7.5] | <0.001 (0.001) | +3.3 [+2.3, +4.4] | <0.001 (0.001) | +| ISONGraph multi_hop (tuned) [graph chosen on dev: mention] | 94.7 [93.6, 95.9] | 91.5 [89.7, 93.2] | 0.924 [0.911, 0.938] | 9.5 | +13.0 [+10.6, +15.4] | <0.001 (0.001) | +6.2 [+5.0, +7.5] | <0.001 (0.001) | +| PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention] | 95.1 [93.9, 96.1] | 91.9 [90.1, 93.5] | 0.881 [0.865, 0.896] | 10.0 | +13.4 [+11.0, +15.8] | <0.001 (0.001) | +6.6 [+5.4, +7.8] | <0.001 (0.001) | +| ISONGraph multi_hop (library defaults) [graph chosen on dev: mention] | 92.7 [91.3, 93.9] | 87.5 [85.4, 89.5] | 0.924 [0.910, 0.938] | 7.8 | +9.0 [+6.6, +11.5] | <0.001 (0.001) | +4.1 [+2.9, +5.4] | <0.001 (0.001) | +| ISONGraph multi_hop (tuned) [graph: mention] | 94.7 [93.6, 95.9] | 91.5 [89.7, 93.2] | 0.924 [0.911, 0.938] | 9.5 | +13.0 [+10.6, +15.4] | <0.001 (0.001) | +6.2 [+5.0, +7.5] | <0.001 (0.001) | +| ISONGraph multi_hop (tuned) [graph: knn] | 90.8 [89.5, 92.0] | 83.0 [80.8, 85.2] | 0.925 [0.912, 0.939] | 10.0 | +4.5 [+2.5, +6.4] | <0.001 (0.001) | +2.3 [+1.3, +3.2] | <0.001 (0.001) | +| ISONGraph multi_hop (tuned) [graph: ctrl_mention] | 88.5 [87.2, 89.9] | 78.4 [75.9, 80.8] | 0.925 [0.912, 0.939] | 10.0 | -0.1 [-0.3, +0.0] | 1.000 (1.000) | -0.1 [-0.2, +0.0] | 1.000 (1.000) | +| PPR re-rank (tuned, independent impl.) [graph: mention] | 95.1 [93.9, 96.1] | 91.9 [90.1, 93.5] | 0.881 [0.865, 0.896] | 10.0 | +13.4 [+11.0, +15.8] | <0.001 (0.001) | +6.6 [+5.4, +7.8] | <0.001 (0.001) | +| PPR re-rank (tuned, independent impl.) [graph: knn] | 90.1 [88.7, 91.4] | 81.7 [79.4, 84.1] | 0.883 [0.867, 0.898] | 10.0 | +3.2 [+1.2, +5.2] | 0.002 (0.008) | +1.5 [+0.5, +2.5] | 0.004 (0.014) | +| PPR re-rank (tuned, independent impl.) [graph: ctrl_mention] | 88.5 [87.1, 89.8] | 78.2 [75.6, 80.7] | 0.912 [0.898, 0.927] | 10.0 | -0.3 [-0.9, +0.3] | 0.544 (1.000) | -0.1 [-0.4, +0.2] | 0.768 (1.000) | +| ISONGraph multi_hop (library defaults) [graph: mention] | 92.7 [91.3, 93.9] | 87.5 [85.4, 89.5] | 0.924 [0.910, 0.938] | 7.8 | +9.0 [+6.6, +11.5] | <0.001 (0.001) | +4.1 [+2.9, +5.4] | <0.001 (0.001) | +| ISONGraph multi_hop (library defaults) [graph: knn] | 89.5 [88.0, 90.8] | 80.9 [78.5, 83.2] | 0.924 [0.911, 0.938] | 10.0 | +2.4 [+0.3, +4.4] | 0.040 (0.120) | +0.9 [-0.2, +2.0] | 0.138 (0.413) | +| ISONGraph multi_hop (library defaults) [graph: ctrl_mention] | 83.0 [81.4, 84.6] | 68.3 [65.4, 71.2] | 0.924 [0.910, 0.938] | 9.1 | -10.2 [-12.2, -8.4] | <0.001 (0.001) | -5.6 [-6.6, -4.6] | <0.001 (0.001) | + +#### semantic_subgraph(connect=True), top_k_results=10: set quality vs plain vector top-|S| (size-matched per query) + +| Graph | connect_max_hops | mean |S| | Recall % | All-supp. % | Vector@|S| recall % | Vector@|S| all % | Δ all (pp) [CI] | p | Δ recall (pp) [CI] | p | +|---|---|---|---|---|---|---|---|---|---|---| +| mention | 2 | 9.6 | 94.8 [93.6, 95.9] | 91.6 [89.8, 93.3] | 88.0 [86.6, 89.3] | 77.4 [74.8, 79.9] | +14.2 [+11.7, +16.9] | <0.001 | +6.8 [+5.5, +8.2] | <0.001 | +| mention | 3 | 9.6 | 94.8 [93.6, 95.9] | 91.6 [89.8, 93.3] | 88.1 [86.7, 89.5] | 77.6 [75.0, 80.1] | +14.0 [+11.5, +16.7] | <0.001 | +6.7 [+5.4, +8.1] | <0.001 | +| knn | 2 | 10.5 | 91.1 [89.9, 92.4] | 83.7 [81.4, 85.9] | 88.8 [87.5, 90.2] | 79.1 [76.5, 81.5] | +4.6 [+2.6, +6.6] | <0.001 | +2.3 [+1.3, +3.4] | <0.001 | +| knn | 3 | 11.1 | 91.2 [89.9, 92.4] | 83.8 [81.6, 86.0] | 89.1 [87.8, 90.5] | 79.6 [77.1, 82.1] | +4.2 [+2.2, +6.2] | <0.001 | +2.1 [+1.0, +3.1] | <0.001 | +| ctrl_mention | 2 | 10.0 | 88.5 [87.2, 89.9] | 78.4 [75.9, 80.8] | 88.5 [87.2, 89.9] | 78.5 [76.0, 80.9] | -0.1 [-0.3, +0.0] | 1.000 | -0.1 [-0.2, +0.0] | 1.000 | +| ctrl_mention | 3 | 10.0 | 88.5 [87.2, 89.9] | 78.4 [75.9, 80.8] | 88.5 [87.2, 89.9] | 78.5 [76.0, 80.9] | -0.1 [-0.3, +0.0] | 1.000 | -0.1 [-0.2, +0.0] | 1.000 | + +### k = 20 (returned-passage budget = 20) + +| Method | Recall@k % [95% CI] | All-supporting@k % [95% CI] | MRR@k [95% CI] | mean returned | Δ all vs vector (pp) [95% CI] | p (Holm) | Δ recall vs vector (pp) [95% CI] | p (Holm) | +|---|---|---|---|---|---|---|---|---| +| Vector top-k (plain) | 92.4 [91.2, 93.5] | 85.4 [83.2, 87.5] | 0.926 [0.912, 0.939] | 20.0 | - | - | - | - | +| BM25 | 91.0 [89.8, 92.2] | 82.2 [79.8, 84.4] | 0.851 [0.835, 0.867] | 20.0 | -3.2 [-5.6, -0.7] | 0.015 (0.104) | -1.4 [-2.6, -0.2] | 0.039 (0.273) | +| Hybrid vector+BM25 (RRF) | 94.8 [93.9, 95.7] | 89.6 [87.8, 91.5] | 0.915 [0.902, 0.928] | 20.0 | +4.2 [+2.6, +5.9] | <0.001 (0.001) | +2.4 [+1.5, +3.4] | <0.001 (0.001) | +| Vector top-30 + cross-encoder | 93.6 [92.6, 94.7] | 87.5 [85.4, 89.5] | 0.951 [0.940, 0.962] | 20.0 | +2.1 [+1.1, +3.2] | <0.001 (0.002) | +1.2 [+0.6, +1.8] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph chosen on dev: mention] | 97.2 [96.4, 98.0] | 95.2 [93.9, 96.5] | 0.926 [0.913, 0.939] | 19.9 | +9.8 [+7.9, +11.8] | <0.001 (0.001) | +4.8 [+3.8, +5.8] | <0.001 (0.001) | +| PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention] | 97.0 [96.1, 97.9] | 95.2 [93.8, 96.5] | 0.881 [0.866, 0.896] | 20.0 | +9.8 [+7.9, +11.9] | <0.001 (0.001) | +4.7 [+3.6, +5.8] | <0.001 (0.001) | +| ISONGraph multi_hop (library defaults) [graph chosen on dev: knn] | 92.5 [91.3, 93.7] | 86.5 [84.4, 88.5] | 0.925 [0.911, 0.938] | 20.0 | +1.1 [-1.1, +3.2] | 0.374 (1.000) | +0.1 [-1.0, +1.2] | 0.932 (1.000) | +| ISONGraph multi_hop (tuned) [graph: mention] | 97.2 [96.4, 98.0] | 95.2 [93.9, 96.5] | 0.926 [0.913, 0.939] | 19.9 | +9.8 [+7.9, +11.8] | <0.001 (0.001) | +4.8 [+3.8, +5.8] | <0.001 (0.001) | +| ISONGraph multi_hop (tuned) [graph: knn] | 94.5 [93.4, 95.6] | 89.6 [87.7, 91.6] | 0.926 [0.912, 0.939] | 20.0 | +4.2 [+2.4, +6.0] | <0.001 (0.001) | +2.1 [+1.2, +3.0] | <0.001 (0.001) | +| ISONGraph multi_hop (tuned) [graph: ctrl_mention] | 92.4 [91.2, 93.5] | 85.4 [83.2, 87.5] | 0.926 [0.912, 0.939] | 20.0 | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +| PPR re-rank (tuned, independent impl.) [graph: mention] | 97.0 [96.1, 97.9] | 95.2 [93.8, 96.5] | 0.881 [0.866, 0.896] | 20.0 | +9.8 [+7.9, +11.9] | <0.001 (0.001) | +4.7 [+3.6, +5.8] | <0.001 (0.001) | +| PPR re-rank (tuned, independent impl.) [graph: knn] | 93.4 [92.4, 94.5] | 87.7 [85.7, 89.7] | 0.883 [0.867, 0.899] | 20.0 | +2.3 [+0.5, +4.0] | 0.020 (0.119) | +1.0 [+0.1, +2.0] | 0.048 (0.291) | +| PPR re-rank (tuned, independent impl.) [graph: ctrl_mention] | 92.3 [91.0, 93.3] | 85.1 [82.9, 87.2] | 0.916 [0.901, 0.930] | 20.0 | -0.3 [-0.7, +0.0] | 0.253 (1.000) | -0.2 [-0.4, +0.0] | 0.253 (1.000) | +| ISONGraph multi_hop (library defaults) [graph: mention] | 92.7 [91.3, 93.9] | 87.5 [85.4, 89.5] | 0.924 [0.910, 0.938] | 9.4 | +2.1 [-0.4, +4.5] | 0.112 (0.561) | +0.2 [-1.0, +1.5] | 0.774 (1.000) | +| ISONGraph multi_hop (library defaults) [graph: knn] | 92.5 [91.3, 93.7] | 86.5 [84.4, 88.5] | 0.925 [0.911, 0.938] | 20.0 | +1.1 [-1.1, +3.2] | 0.374 (1.000) | +0.1 [-1.0, +1.2] | 0.932 (1.000) | +| ISONGraph multi_hop (library defaults) [graph: ctrl_mention] | 83.0 [81.4, 84.6] | 68.3 [65.4, 71.2] | 0.924 [0.910, 0.938] | 11.4 | -17.1 [-19.5, -14.8] | <0.001 (0.001) | -9.4 [-10.7, -8.2] | <0.001 (0.001) | + +#### semantic_subgraph(connect=True), top_k_results=20: set quality vs plain vector top-|S| (size-matched per query) + +| Graph | connect_max_hops | mean |S| | Recall % | All-supp. % | Vector@|S| recall % | Vector@|S| all % | Δ all (pp) [CI] | p | Δ recall (pp) [CI] | p | +|---|---|---|---|---|---|---|---|---|---|---| +| mention | 2 | 20.3 | 97.3 [96.4, 98.0] | 95.3 [93.9, 96.6] | 92.5 [91.4, 93.7] | 85.6 [83.4, 87.6] | +9.7 [+7.9, +11.6] | <0.001 | +4.7 [+3.8, +5.6] | <0.001 | +| mention | 3 | 21.0 | 97.3 [96.4, 98.0] | 95.3 [93.9, 96.6] | 92.6 [91.5, 93.7] | 85.7 [83.6, 87.7] | +9.6 [+7.8, +11.5] | <0.001 | +4.7 [+3.8, +5.6] | <0.001 | +| knn | 2 | 21.1 | 94.7 [93.7, 95.6] | 89.8 [88.0, 91.7] | 92.8 [91.7, 93.8] | 86.1 [84.0, 88.1] | +3.7 [+1.9, +5.5] | <0.001 | +1.9 [+0.9, +2.8] | <0.001 | +| knn | 3 | 22.1 | 94.7 [93.7, 95.7] | 89.8 [88.0, 91.7] | 93.1 [92.0, 94.2] | 86.6 [84.5, 88.6] | +3.2 [+1.5, +4.8] | <0.001 | +1.6 [+0.7, +2.4] | <0.001 | +| ctrl_mention | 2 | 20.1 | 92.4 [91.2, 93.5] | 85.4 [83.2, 87.5] | 92.4 [91.2, 93.5] | 85.4 [83.2, 87.5] | +0.0 [+0.0, +0.0] | 1.000 | +0.0 [+0.0, +0.0] | 1.000 | +| ctrl_mention | 3 | 20.2 | 92.4 [91.2, 93.5] | 85.4 [83.2, 87.5] | 92.4 [91.2, 93.5] | 85.4 [83.2, 87.5] | +0.0 [+0.0, +0.0] | 1.000 | +0.0 [+0.0, +0.0] | 1.000 | + +#### hotpotqa/bge: ISONGraph multi_hop (tuned) minus PPR re-rank (tuned), same graph; all-supporting (pp [95% CI], paired p) + +| Graph | k=5 | k=10 | k=20 | +|---|---|---|---| +| mention | +1.7 [-0.6, +3.9] (p=0.167) | -0.4 [-1.4, +0.6] (p=0.549) | +0.0 [-0.9, +0.9] (p=1.000) | +| knn | +1.4 [-0.7, +3.4] (p=0.200) | +1.3 [+0.1, +2.5] (p=0.052) | +1.9 [+0.6, +3.3] (p=0.011) | + +#### hotpotqa/bge: breakdown by question type, k=10 -- All-supporting@10 % (Δ vs vector, pp [95% CI]) + +| Group | n | Vector top-k (plain) | Hybrid vector+BM25 (RRF) | Vector top-30 + cross-encoder | ISONGraph multi_hop (tuned) | PPR re-rank (tuned, independent impl.) | +|---|---|---|---|---|---|---| +| bridge | 784 | 73.1 | 79.8 (+6.8 [+4.3, +9.2]) | 80.0 (+6.9 [+4.5, +9.3]) | 89.9 (+16.8 [+13.8, +19.8]) [mention] | 90.4 (+17.3 [+14.4, +20.2]) [mention] | +| comparison | 216 | 98.1 | 98.1 (+0.0 [-1.4, +1.4]) | 99.1 (+0.9 [-0.9, +2.8]) | 97.2 (-0.9 [-2.8, +0.9]) [mention] | 97.2 (-0.9 [-2.8, +0.9]) [mention] | + +#### hotpotqa/bge: breakdown by hop count (number of supporting passages), k=10 -- All-supporting@10 % (Δ vs vector, pp [95% CI]) + +| Group | n | Vector top-k (plain) | Hybrid vector+BM25 (RRF) | Vector top-30 + cross-encoder | ISONGraph multi_hop (tuned) | PPR re-rank (tuned, independent impl.) | +|---|---|---|---|---|---|---| +| 2 | 1000 | 78.5 | 83.8 (+5.3 [+3.3, +7.3]) | 84.1 (+5.6 [+3.6, +7.6]) | 91.5 (+13.0 [+10.6, +15.5]) [mention] | 91.9 (+13.4 [+11.1, +15.7]) [mention] | + +#### hotpotqa/bge: dev vs test All-supporting@10 for tuned configs (selection optimism) + +| Method | Graph | dev % | test % | +|---|---|---|---| +| lib_mh | mention | 89.8 | 91.5 | +| lib_mh | knn | 82.2 | 83.0 | +| lib_mh | ctrl_mention | 73.6 | 78.4 | +| ppr | mention | 89.6 | 91.9 | +| ppr | knn | 80.4 | 81.7 | +| ppr | ctrl_mention | 74.0 | 78.2 | +| vector | - | 73.6 | 78.5 | + +## 2wiki / minilm (corpus 9239 passages, 1000 test questions) + + +### k = 5 (returned-passage budget = 5) + +| Method | Recall@k % [95% CI] | All-supporting@k % [95% CI] | MRR@k [95% CI] | mean returned | Δ all vs vector (pp) [95% CI] | p (Holm) | Δ recall vs vector (pp) [95% CI] | p (Holm) | +|---|---|---|---|---|---|---|---|---| +| Vector top-k (plain) | 60.8 [59.2, 62.3] | 26.2 [23.5, 28.7] | 0.912 [0.898, 0.926] | 5.0 | - | - | - | - | +| BM25 | 66.4 [64.9, 67.9] | 33.8 [31.1, 36.7] | 0.937 [0.925, 0.949] | 5.0 | +7.6 [+5.2, +10.1] | <0.001 (0.002) | +5.6 [+4.2, +7.1] | <0.001 (0.002) | +| Hybrid vector+BM25 (RRF) | 66.5 [65.0, 68.0] | 33.2 [30.3, 36.0] | 0.948 [0.938, 0.958] | 5.0 | +7.0 [+5.1, +9.0] | <0.001 (0.002) | +5.7 [+4.6, +6.8] | <0.001 (0.002) | +| Vector top-30 + cross-encoder | 66.5 [65.0, 67.9] | 33.8 [30.9, 36.5] | 0.970 [0.962, 0.978] | 5.0 | +7.6 [+5.8, +9.5] | <0.001 (0.002) | +5.7 [+4.6, +6.8] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph chosen on dev: mention+entity] | 82.5 [81.0, 84.0] | 63.8 [60.8, 66.8] | 0.911 [0.897, 0.925] | 5.0 | +37.6 [+34.4, +40.7] | <0.001 (0.002) | +21.7 [+20.2, +23.3] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention] | 74.8 [73.0, 76.6] | 51.8 [48.7, 54.9] | 0.758 [0.738, 0.779] | 5.0 | +25.6 [+22.5, +28.5] | <0.001 (0.002) | +14.0 [+12.4, +15.6] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph chosen on dev: mention] | 76.9 [75.2, 78.5] | 53.5 [50.5, 56.6] | 0.911 [0.897, 0.925] | 5.0 | +27.3 [+24.5, +30.3] | <0.001 (0.002) | +16.1 [+14.7, +17.5] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: mention] | 82.5 [81.0, 84.0] | 63.8 [60.8, 66.8] | 0.911 [0.897, 0.925] | 5.0 | +37.6 [+34.4, +40.7] | <0.001 (0.002) | +21.7 [+20.2, +23.3] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: entity] | 70.4 [68.7, 72.1] | 43.4 [40.4, 46.4] | 0.912 [0.897, 0.925] | 5.0 | +17.2 [+14.6, +19.8] | <0.001 (0.002) | +9.6 [+8.3, +10.9] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: knn] | 60.4 [58.9, 61.9] | 26.3 [23.6, 28.9] | 0.912 [0.898, 0.925] | 5.0 | +0.1 [-1.3, +1.4] | 1.000 (1.000) | -0.4 [-1.1, +0.4] | 0.402 (1.000) | +| ISONGraph multi_hop (tuned) [graph: mention+entity] | 82.5 [81.0, 84.0] | 63.8 [60.8, 66.8] | 0.911 [0.897, 0.925] | 5.0 | +37.6 [+34.4, +40.7] | <0.001 (0.002) | +21.7 [+20.2, +23.3] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: ctrl_mention] | 60.8 [59.2, 62.3] | 26.2 [23.5, 28.8] | 0.911 [0.897, 0.925] | 5.0 | +0.0 [-0.3, +0.3] | 1.000 (1.000) | +0.0 [-0.2, +0.2] | 1.000 (1.000) | +| ISONGraph multi_hop (tuned) [graph: ctrl_entity] | 60.8 [59.2, 62.3] | 26.2 [23.5, 28.8] | 0.911 [0.897, 0.925] | 5.0 | +0.0 [-0.3, +0.3] | 1.000 (1.000) | +0.0 [-0.2, +0.2] | 1.000 (1.000) | +| PPR re-rank (tuned, independent impl.) [graph: mention] | 74.8 [73.0, 76.6] | 51.8 [48.7, 54.9] | 0.758 [0.738, 0.779] | 5.0 | +25.6 [+22.5, +28.5] | <0.001 (0.002) | +14.0 [+12.4, +15.6] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph: entity] | 63.8 [62.2, 65.4] | 31.7 [28.9, 34.6] | 0.751 [0.731, 0.771] | 5.0 | +5.5 [+3.6, +7.3] | <0.001 (0.002) | +3.1 [+2.0, +4.0] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph: knn] | 60.0 [58.5, 61.6] | 25.6 [23.0, 28.3] | 0.895 [0.880, 0.910] | 5.0 | -0.6 [-1.8, +0.6] | 0.438 (1.000) | -0.8 [-1.5, -0.1] | 0.045 (0.182) | +| PPR re-rank (tuned, independent impl.) [graph: mention+entity] | 66.9 [65.2, 68.6] | 37.9 [35.1, 40.8] | 0.701 [0.680, 0.723] | 5.0 | +11.7 [+9.3, +14.2] | <0.001 (0.002) | +6.1 [+4.7, +7.6] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph: ctrl_mention] | 60.0 [58.5, 61.6] | 25.5 [22.8, 28.1] | 0.867 [0.851, 0.883] | 5.0 | -0.7 [-1.7, +0.3] | 0.235 (1.000) | -0.7 [-1.3, -0.2] | 0.011 (0.057) | +| PPR re-rank (tuned, independent impl.) [graph: ctrl_entity] | 60.0 [58.4, 61.5] | 25.3 [22.6, 27.9] | 0.900 [0.885, 0.914] | 5.0 | -0.9 [-1.9, +0.1] | 0.112 (0.675) | -0.8 [-1.4, -0.2] | 0.008 (0.046) | +| ISONGraph multi_hop (library defaults) [graph: mention] | 76.9 [75.2, 78.5] | 53.5 [50.5, 56.6] | 0.911 [0.897, 0.925] | 5.0 | +27.3 [+24.5, +30.3] | <0.001 (0.002) | +16.1 [+14.7, +17.5] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: entity] | 67.7 [66.0, 69.3] | 37.8 [34.9, 40.7] | 0.911 [0.897, 0.924] | 5.0 | +11.6 [+8.9, +14.2] | <0.001 (0.002) | +6.9 [+5.4, +8.2] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: knn] | 58.6 [57.0, 60.0] | 23.6 [21.0, 26.2] | 0.912 [0.898, 0.925] | 5.0 | -2.6 [-4.7, -0.4] | 0.027 (0.191) | -2.3 [-3.4, -1.1] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: mention+entity] | 75.4 [73.8, 77.1] | 51.2 [48.2, 54.3] | 0.911 [0.897, 0.924] | 5.0 | +25.0 [+22.0, +27.9] | <0.001 (0.002) | +14.6 [+13.1, +16.1] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: ctrl_mention] | 59.7 [58.2, 61.2] | 24.2 [21.6, 26.7] | 0.911 [0.897, 0.924] | 5.0 | -2.0 [-2.9, -1.2] | <0.001 (0.002) | -1.1 [-1.6, -0.7] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: ctrl_entity] | 57.0 [55.5, 58.4] | 20.0 [17.5, 22.3] | 0.911 [0.897, 0.924] | 5.0 | -6.2 [-7.7, -4.7] | <0.001 (0.002) | -3.8 [-4.6, -3.1] | <0.001 (0.002) | + +#### semantic_subgraph(connect=True), top_k_results=5: set quality vs plain vector top-|S| (size-matched per query) + +| Graph | connect_max_hops | mean |S| | Recall % | All-supp. % | Vector@|S| recall % | Vector@|S| all % | Δ all (pp) [CI] | p | Δ recall (pp) [CI] | p | +|---|---|---|---|---|---|---|---|---|---|---| +| mention | 2 | 5.0 | 82.6 [81.1, 84.2] | 64.0 [61.0, 66.9] | 60.8 [59.2, 62.3] | 26.2 [23.5, 28.7] | +37.8 [+34.6, +40.9] | <0.001 | +21.8 [+20.3, +23.4] | <0.001 | +| mention | 3 | 5.0 | 82.7 [81.1, 84.2] | 64.1 [61.1, 67.0] | 60.8 [59.2, 62.3] | 26.2 [23.5, 28.7] | +37.9 [+34.6, +41.0] | <0.001 | +21.9 [+20.3, +23.4] | <0.001 | +| entity | 2 | 5.1 | 70.7 [68.9, 72.4] | 44.0 [40.9, 47.0] | 61.1 [59.5, 62.5] | 26.5 [23.8, 29.1] | +17.5 [+14.8, +20.0] | <0.001 | +9.7 [+8.3, +11.0] | <0.001 | +| entity | 3 | 5.6 | 71.3 [69.6, 73.0] | 45.3 [42.2, 48.3] | 61.6 [60.1, 63.0] | 27.1 [24.3, 29.7] | +18.2 [+15.4, +20.8] | <0.001 | +9.7 [+8.3, +11.1] | <0.001 | +| knn | 2 | 5.3 | 60.6 [59.0, 62.1] | 26.6 [23.9, 29.2] | 61.3 [59.7, 62.8] | 26.8 [24.1, 29.4] | -0.2 [-1.7, +1.2] | 0.887 | -0.7 [-1.4, +0.1] | 0.123 | +| knn | 3 | 5.9 | 61.0 [59.5, 62.6] | 27.1 [24.4, 29.7] | 61.6 [60.1, 63.1] | 27.1 [24.4, 29.8] | +0.0 [-1.5, +1.4] | 1.000 | -0.6 [-1.4, +0.2] | 0.188 | +| mention+entity | 2 | 5.1 | 82.9 [81.4, 84.4] | 64.6 [61.6, 67.5] | 61.2 [59.7, 62.7] | 26.9 [24.2, 29.5] | +37.7 [+34.5, +40.8] | <0.001 | +21.7 [+20.2, +23.2] | <0.001 | +| mention+entity | 3 | 5.8 | 83.1 [81.6, 84.7] | 65.2 [62.2, 68.1] | 61.7 [60.1, 63.2] | 27.4 [24.6, 30.1] | +37.8 [+34.6, +40.9] | <0.001 | +21.4 [+19.9, +22.9] | <0.001 | +| ctrl_mention | 2 | 5.0 | 60.8 [59.2, 62.3] | 26.2 [23.5, 28.8] | 60.8 [59.2, 62.3] | 26.2 [23.5, 28.7] | +0.0 [-0.3, +0.3] | 1.000 | +0.0 [-0.2, +0.2] | 1.000 | +| ctrl_mention | 3 | 5.0 | 60.8 [59.2, 62.3] | 26.2 [23.5, 28.8] | 60.8 [59.2, 62.3] | 26.2 [23.5, 28.7] | +0.0 [-0.3, +0.3] | 1.000 | +0.0 [-0.2, +0.2] | 1.000 | +| ctrl_entity | 2 | 5.1 | 60.8 [59.2, 62.3] | 26.2 [23.5, 28.8] | 60.9 [59.3, 62.4] | 26.3 [23.6, 28.9] | -0.1 [-0.4, +0.2] | 1.000 | -0.1 [-0.3, +0.1] | 0.629 | +| ctrl_entity | 3 | 6.2 | 60.8 [59.2, 62.3] | 26.2 [23.5, 28.8] | 61.8 [60.3, 63.3] | 27.7 [24.9, 30.3] | -1.5 [-2.4, -0.8] | <0.001 | -1.0 [-1.5, -0.6] | <0.001 | + +### k = 10 (returned-passage budget = 10) + +| Method | Recall@k % [95% CI] | All-supporting@k % [95% CI] | MRR@k [95% CI] | mean returned | Δ all vs vector (pp) [95% CI] | p (Holm) | Δ recall vs vector (pp) [95% CI] | p (Holm) | +|---|---|---|---|---|---|---|---|---| +| Vector top-k (plain) | 65.2 [63.6, 66.7] | 32.5 [29.6, 35.2] | 0.914 [0.900, 0.927] | 10.0 | - | - | - | - | +| BM25 | 71.2 [69.7, 72.8] | 41.5 [38.5, 44.5] | 0.938 [0.926, 0.950] | 10.0 | +9.0 [+6.6, +11.4] | <0.001 (0.002) | +6.0 [+4.7, +7.4] | <0.001 (0.002) | +| Hybrid vector+BM25 (RRF) | 71.4 [69.9, 72.9] | 41.1 [38.1, 44.1] | 0.948 [0.938, 0.958] | 10.0 | +8.6 [+6.7, +10.5] | <0.001 (0.002) | +6.2 [+5.1, +7.3] | <0.001 (0.002) | +| Vector top-30 + cross-encoder | 68.6 [67.0, 70.1] | 37.5 [34.5, 40.2] | 0.970 [0.962, 0.978] | 10.0 | +5.0 [+3.6, +6.5] | <0.001 (0.002) | +3.4 [+2.6, +4.3] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph chosen on dev: mention+entity] | 89.8 [88.5, 91.1] | 81.0 [78.6, 83.4] | 0.912 [0.898, 0.925] | 10.0 | +48.5 [+45.2, +51.8] | <0.001 (0.002) | +24.6 [+22.9, +26.2] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention] | 88.7 [87.3, 90.0] | 78.7 [76.2, 81.1] | 0.851 [0.834, 0.868] | 10.0 | +46.2 [+42.8, +49.7] | <0.001 (0.002) | +23.5 [+21.8, +25.2] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph chosen on dev: mention] | 89.0 [87.7, 90.4] | 79.5 [77.0, 81.9] | 0.912 [0.898, 0.925] | 7.9 | +47.0 [+43.5, +50.4] | <0.001 (0.002) | +23.8 [+22.1, +25.5] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: mention] | 89.1 [87.7, 90.4] | 79.6 [77.1, 82.0] | 0.912 [0.898, 0.925] | 8.0 | +47.1 [+43.7, +50.5] | <0.001 (0.002) | +23.8 [+22.1, +25.6] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: entity] | 80.9 [79.3, 82.4] | 60.6 [57.6, 63.4] | 0.912 [0.898, 0.925] | 9.9 | +28.1 [+24.9, +31.1] | <0.001 (0.002) | +15.7 [+14.2, +17.2] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: knn] | 67.8 [66.1, 69.6] | 39.8 [36.8, 42.9] | 0.912 [0.898, 0.925] | 10.0 | +7.3 [+4.8, +9.8] | <0.001 (0.002) | +2.6 [+1.3, +4.0] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: mention+entity] | 89.8 [88.5, 91.1] | 81.0 [78.6, 83.4] | 0.912 [0.898, 0.925] | 10.0 | +48.5 [+45.2, +51.8] | <0.001 (0.002) | +24.6 [+22.9, +26.2] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: ctrl_mention] | 65.2 [63.6, 66.7] | 32.5 [29.6, 35.2] | 0.913 [0.899, 0.926] | 10.0 | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +| ISONGraph multi_hop (tuned) [graph: ctrl_entity] | 65.2 [63.6, 66.7] | 32.5 [29.6, 35.2] | 0.913 [0.899, 0.926] | 10.0 | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +| PPR re-rank (tuned, independent impl.) [graph: mention] | 88.7 [87.3, 90.0] | 78.7 [76.2, 81.1] | 0.851 [0.834, 0.868] | 10.0 | +46.2 [+42.8, +49.7] | <0.001 (0.002) | +23.5 [+21.8, +25.2] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph: entity] | 78.1 [76.5, 79.7] | 54.9 [51.7, 57.9] | 0.873 [0.857, 0.889] | 10.0 | +22.4 [+19.7, +25.1] | <0.001 (0.002) | +13.0 [+11.5, +14.3] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph: knn] | 68.6 [66.8, 70.2] | 39.9 [36.7, 42.9] | 0.873 [0.857, 0.889] | 10.0 | +7.4 [+5.3, +9.6] | <0.001 (0.002) | +3.4 [+2.2, +4.5] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph: mention+entity] | 86.0 [84.5, 87.4] | 72.0 [69.2, 74.7] | 0.855 [0.839, 0.872] | 10.0 | +39.5 [+36.3, +42.7] | <0.001 (0.002) | +20.8 [+19.2, +22.4] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph: ctrl_mention] | 65.3 [63.8, 66.8] | 32.7 [29.8, 35.5] | 0.911 [0.897, 0.925] | 10.0 | +0.2 [-0.3, +0.7] | 0.688 (1.000) | +0.1 [-0.1, +0.4] | 0.456 (1.000) | +| PPR re-rank (tuned, independent impl.) [graph: ctrl_entity] | 64.7 [63.1, 66.2] | 31.6 [28.8, 34.4] | 0.877 [0.862, 0.893] | 10.0 | -0.9 [-1.7, -0.1] | 0.052 (0.207) | -0.5 [-1.0, +0.0] | 0.067 (0.270) | +| ISONGraph multi_hop (library defaults) [graph: mention] | 89.0 [87.7, 90.4] | 79.5 [77.0, 81.9] | 0.912 [0.898, 0.925] | 7.9 | +47.0 [+43.5, +50.4] | <0.001 (0.002) | +23.8 [+22.1, +25.5] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: entity] | 78.9 [77.3, 80.5] | 56.5 [53.5, 59.5] | 0.911 [0.897, 0.925] | 10.0 | +24.0 [+20.6, +27.2] | <0.001 (0.002) | +13.7 [+12.0, +15.4] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: knn] | 67.8 [66.1, 69.6] | 39.8 [36.8, 42.9] | 0.912 [0.898, 0.925] | 10.0 | +7.3 [+4.8, +9.8] | <0.001 (0.002) | +2.6 [+1.3, +4.0] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: mention+entity] | 85.9 [84.4, 87.3] | 69.4 [66.4, 72.3] | 0.911 [0.898, 0.925] | 10.0 | +36.9 [+33.5, +40.3] | <0.001 (0.002) | +20.6 [+18.9, +22.4] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: ctrl_mention] | 60.8 [59.2, 62.3] | 26.2 [23.5, 28.8] | 0.911 [0.897, 0.924] | 8.5 | -6.3 [-7.9, -4.9] | <0.001 (0.002) | -4.4 [-5.2, -3.6] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: ctrl_entity] | 58.8 [57.3, 60.3] | 22.9 [20.3, 25.5] | 0.911 [0.897, 0.924] | 10.0 | -9.6 [-11.5, -7.8] | <0.001 (0.002) | -6.4 [-7.4, -5.4] | <0.001 (0.002) | + +#### semantic_subgraph(connect=True), top_k_results=10: set quality vs plain vector top-|S| (size-matched per query) + +| Graph | connect_max_hops | mean |S| | Recall % | All-supp. % | Vector@|S| recall % | Vector@|S| all % | Δ all (pp) [CI] | p | Δ recall (pp) [CI] | p | +|---|---|---|---|---|---|---|---|---|---|---| +| mention | 2 | 8.0 | 89.1 [87.7, 90.4] | 79.6 [77.1, 82.0] | 63.9 [62.3, 65.3] | 30.6 [27.7, 33.3] | +49.0 [+45.7, +52.3] | <0.001 | +25.2 [+23.6, +26.8] | <0.001 | +| mention | 3 | 8.0 | 89.1 [87.7, 90.4] | 79.6 [77.1, 82.0] | 63.9 [62.3, 65.3] | 30.6 [27.7, 33.3] | +49.0 [+45.7, +52.3] | <0.001 | +25.2 [+23.6, +26.8] | <0.001 | +| entity | 2 | 10.1 | 81.2 [79.6, 82.7] | 61.2 [58.2, 64.1] | 65.2 [63.6, 66.7] | 32.5 [29.7, 35.2] | +28.7 [+25.5, +31.7] | <0.001 | +16.0 [+14.5, +17.5] | <0.001 | +| entity | 3 | 11.3 | 81.6 [79.9, 83.1] | 62.0 [58.9, 64.9] | 65.8 [64.2, 67.3] | 33.4 [30.5, 36.1] | +28.6 [+25.4, +31.6] | <0.001 | +15.8 [+14.2, +17.3] | <0.001 | +| knn | 2 | 10.4 | 67.9 [66.3, 69.7] | 40.0 [37.1, 43.1] | 65.5 [63.9, 67.0] | 32.8 [29.9, 35.5] | +7.2 [+4.7, +9.7] | <0.001 | +2.5 [+1.1, +3.9] | <0.001 | +| knn | 3 | 11.2 | 68.2 [66.5, 69.9] | 40.1 [37.2, 43.2] | 65.8 [64.2, 67.2] | 33.2 [30.3, 35.9] | +6.9 [+4.3, +9.4] | <0.001 | +2.4 [+1.0, +3.8] | <0.001 | +| mention+entity | 2 | 10.3 | 89.8 [88.5, 91.1] | 81.0 [78.6, 83.4] | 65.3 [63.7, 66.8] | 32.6 [29.7, 35.3] | +48.4 [+45.1, +51.6] | <0.001 | +24.5 [+22.8, +26.2] | <0.001 | +| mention+entity | 3 | 11.7 | 89.8 [88.5, 91.1] | 81.0 [78.6, 83.4] | 65.9 [64.3, 67.4] | 33.5 [30.6, 36.2] | +47.5 [+44.2, +50.8] | <0.001 | +23.9 [+22.3, +25.6] | <0.001 | +| ctrl_mention | 2 | 10.0 | 65.2 [63.6, 66.7] | 32.5 [29.6, 35.2] | 65.2 [63.6, 66.7] | 32.5 [29.6, 35.2] | +0.0 [+0.0, +0.0] | 1.000 | +0.0 [+0.0, +0.0] | 1.000 | +| ctrl_mention | 3 | 10.0 | 65.2 [63.6, 66.7] | 32.5 [29.6, 35.2] | 65.2 [63.6, 66.7] | 32.5 [29.6, 35.2] | +0.0 [+0.0, +0.0] | 1.000 | +0.0 [+0.0, +0.0] | 1.000 | +| ctrl_entity | 2 | 10.3 | 65.2 [63.6, 66.7] | 32.5 [29.6, 35.2] | 65.3 [63.7, 66.9] | 32.8 [29.9, 35.6] | -0.3 [-0.7, +0.0] | 0.251 | -0.2 [-0.4, +0.0] | 0.251 | +| ctrl_entity | 3 | 15.6 | 65.2 [63.6, 66.7] | 32.5 [29.6, 35.2] | 67.4 [65.8, 68.9] | 35.8 [32.9, 38.6] | -3.3 [-4.5, -2.3] | <0.001 | -2.2 [-2.8, -1.6] | <0.001 | + +### k = 20 (returned-passage budget = 20) + +| Method | Recall@k % [95% CI] | All-supporting@k % [95% CI] | MRR@k [95% CI] | mean returned | Δ all vs vector (pp) [95% CI] | p (Holm) | Δ recall vs vector (pp) [95% CI] | p (Holm) | +|---|---|---|---|---|---|---|---|---| +| Vector top-k (plain) | 69.0 [67.4, 70.5] | 38.4 [35.3, 41.3] | 0.914 [0.900, 0.927] | 20.0 | - | - | - | - | +| BM25 | 74.3 [72.8, 75.8] | 46.7 [43.8, 49.8] | 0.939 [0.927, 0.950] | 20.0 | +8.3 [+6.0, +10.6] | <0.001 (0.002) | +5.3 [+4.1, +6.5] | <0.001 (0.002) | +| Hybrid vector+BM25 (RRF) | 74.4 [72.9, 75.9] | 46.7 [43.7, 49.7] | 0.948 [0.938, 0.958] | 20.0 | +8.3 [+6.5, +10.2] | <0.001 (0.002) | +5.4 [+4.5, +6.5] | <0.001 (0.002) | +| Vector top-30 + cross-encoder | 69.5 [67.9, 71.0] | 39.0 [35.9, 41.9] | 0.970 [0.962, 0.978] | 20.0 | +0.6 [-0.2, +1.4] | 0.216 (1.000) | +0.5 [+0.1, +1.0] | 0.019 (0.095) | +| ISONGraph multi_hop (tuned) [graph chosen on dev: mention+entity] | 93.9 [92.8, 94.9] | 87.3 [85.1, 89.3] | 0.913 [0.899, 0.926] | 20.0 | +48.9 [+45.8, +52.2] | <0.001 (0.002) | +24.9 [+23.4, +26.5] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention] | 93.0 [91.9, 94.1] | 85.7 [83.5, 87.9] | 0.812 [0.794, 0.830] | 20.0 | +47.3 [+44.2, +50.5] | <0.001 (0.002) | +24.0 [+22.5, +25.6] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph chosen on dev: mention] | 89.1 [87.8, 90.5] | 79.9 [77.4, 82.3] | 0.912 [0.898, 0.925] | 8.4 | +41.5 [+37.8, +45.1] | <0.001 (0.002) | +20.1 [+18.3, +22.0] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: mention] | 93.6 [92.4, 94.6] | 86.6 [84.3, 88.7] | 0.913 [0.899, 0.926] | 19.4 | +48.2 [+45.1, +51.4] | <0.001 (0.002) | +24.6 [+23.0, +26.2] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: entity] | 86.6 [85.1, 88.1] | 73.3 [70.6, 76.1] | 0.912 [0.898, 0.925] | 19.8 | +34.9 [+31.3, +38.5] | <0.001 (0.002) | +17.6 [+15.9, +19.4] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: knn] | 74.2 [72.7, 75.8] | 48.5 [45.5, 51.5] | 0.913 [0.900, 0.926] | 20.0 | +10.1 [+7.9, +12.2] | <0.001 (0.002) | +5.2 [+4.1, +6.3] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: mention+entity] | 93.9 [92.8, 94.9] | 87.3 [85.1, 89.3] | 0.913 [0.899, 0.926] | 20.0 | +48.9 [+45.8, +52.2] | <0.001 (0.002) | +24.9 [+23.4, +26.5] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: ctrl_mention] | 69.0 [67.4, 70.5] | 38.4 [35.3, 41.3] | 0.913 [0.899, 0.926] | 20.0 | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +| ISONGraph multi_hop (tuned) [graph: ctrl_entity] | 69.0 [67.4, 70.5] | 38.4 [35.3, 41.3] | 0.913 [0.899, 0.926] | 20.0 | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +| PPR re-rank (tuned, independent impl.) [graph: mention] | 93.0 [91.9, 94.1] | 85.7 [83.5, 87.9] | 0.812 [0.794, 0.830] | 20.0 | +47.3 [+44.2, +50.5] | <0.001 (0.002) | +24.0 [+22.5, +25.6] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph: entity] | 84.8 [83.2, 86.2] | 67.3 [64.3, 70.2] | 0.821 [0.804, 0.838] | 20.0 | +28.9 [+25.9, +31.9] | <0.001 (0.002) | +15.8 [+14.3, +17.2] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph: knn] | 73.0 [71.4, 74.6] | 46.6 [43.5, 49.7] | 0.845 [0.827, 0.862] | 20.0 | +8.2 [+6.0, +10.4] | <0.001 (0.002) | +4.0 [+2.8, +5.1] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph: mention+entity] | 90.0 [88.7, 91.4] | 81.5 [79.0, 83.8] | 0.855 [0.839, 0.872] | 20.0 | +43.1 [+39.6, +46.7] | <0.001 (0.002) | +21.1 [+19.4, +22.9] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph: ctrl_mention] | 69.0 [67.4, 70.5] | 38.4 [35.3, 41.3] | 0.912 [0.898, 0.925] | 20.0 | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +| PPR re-rank (tuned, independent impl.) [graph: ctrl_entity] | 69.0 [67.4, 70.5] | 38.4 [35.3, 41.3] | 0.837 [0.820, 0.855] | 20.0 | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +| ISONGraph multi_hop (library defaults) [graph: mention] | 89.1 [87.8, 90.5] | 79.9 [77.4, 82.3] | 0.912 [0.898, 0.925] | 8.4 | +41.5 [+37.8, +45.1] | <0.001 (0.002) | +20.1 [+18.3, +22.0] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: entity] | 84.8 [83.3, 86.2] | 68.5 [65.6, 71.2] | 0.911 [0.898, 0.925] | 19.8 | +30.1 [+26.5, +33.5] | <0.001 (0.002) | +15.8 [+14.0, +17.6] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: knn] | 72.1 [70.6, 73.8] | 46.2 [43.2, 49.3] | 0.912 [0.899, 0.926] | 20.0 | +7.8 [+5.2, +10.2] | <0.001 (0.002) | +3.2 [+1.8, +4.5] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: mention+entity] | 88.8 [87.4, 90.2] | 77.1 [74.5, 79.6] | 0.912 [0.898, 0.925] | 19.9 | +38.7 [+35.1, +42.1] | <0.001 (0.002) | +19.9 [+18.1, +21.7] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: ctrl_mention] | 60.8 [59.2, 62.3] | 26.2 [23.5, 28.8] | 0.911 [0.897, 0.924] | 9.7 | -12.2 [-14.3, -10.3] | <0.001 (0.002) | -8.2 [-9.4, -7.1] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: ctrl_entity] | 59.9 [58.4, 61.3] | 24.5 [21.8, 27.1] | 0.911 [0.897, 0.924] | 19.9 | -13.9 [-16.1, -11.9] | <0.001 (0.002) | -9.1 [-10.3, -7.9] | <0.001 (0.002) | + +#### semantic_subgraph(connect=True), top_k_results=20: set quality vs plain vector top-|S| (size-matched per query) + +| Graph | connect_max_hops | mean |S| | Recall % | All-supp. % | Vector@|S| recall % | Vector@|S| all % | Δ all (pp) [CI] | p | Δ recall (pp) [CI] | p | +|---|---|---|---|---|---|---|---|---|---|---| +| mention | 2 | 19.5 | 93.6 [92.4, 94.6] | 86.6 [84.3, 88.7] | 68.9 [67.3, 70.4] | 38.3 [35.2, 41.2] | +48.3 [+45.1, +51.5] | <0.001 | +24.6 [+23.1, +26.2] | <0.001 | +| mention | 3 | 19.5 | 93.6 [92.4, 94.6] | 86.6 [84.3, 88.7] | 68.9 [67.3, 70.4] | 38.3 [35.2, 41.2] | +48.3 [+45.1, +51.5] | <0.001 | +24.6 [+23.1, +26.2] | <0.001 | +| entity | 2 | 20.4 | 86.7 [85.1, 88.2] | 73.4 [70.7, 76.2] | 68.8 [67.2, 70.3] | 38.2 [35.2, 41.2] | +35.2 [+31.7, +38.8] | <0.001 | +17.8 [+16.1, +19.6] | <0.001 | +| entity | 3 | 21.6 | 86.7 [85.2, 88.2] | 73.6 [70.9, 76.3] | 69.0 [67.4, 70.5] | 38.5 [35.5, 41.4] | +35.1 [+31.6, +38.8] | <0.001 | +17.7 [+15.9, +19.4] | <0.001 | +| knn | 2 | 22.8 | 74.4 [72.9, 76.0] | 48.6 [45.6, 51.7] | 69.2 [67.7, 70.7] | 38.7 [35.7, 41.6] | +9.9 [+7.8, +12.0] | <0.001 | +5.2 [+4.1, +6.2] | <0.001 | +| knn | 3 | 26.9 | 74.8 [73.3, 76.4] | 49.0 [46.0, 52.1] | 69.6 [68.0, 71.1] | 39.2 [36.2, 42.1] | +9.8 [+7.8, +11.9] | <0.001 | +5.2 [+4.1, +6.3] | <0.001 | +| mention+entity | 2 | 21.9 | 94.1 [93.0, 95.1] | 87.6 [85.4, 89.6] | 69.2 [67.6, 70.7] | 38.6 [35.5, 41.5] | +49.0 [+45.9, +52.3] | <0.001 | +24.9 [+23.4, +26.4] | <0.001 | +| mention+entity | 3 | 28.6 | 94.2 [93.1, 95.1] | 88.0 [85.9, 90.0] | 70.0 [68.4, 71.5] | 39.5 [36.5, 42.4] | +48.5 [+45.4, +51.7] | <0.001 | +24.2 [+22.7, +25.7] | <0.001 | +| ctrl_mention | 2 | 20.0 | 69.0 [67.4, 70.5] | 38.4 [35.3, 41.3] | 69.0 [67.4, 70.5] | 38.4 [35.3, 41.3] | +0.0 [+0.0, +0.0] | 1.000 | +0.0 [+0.0, +0.0] | 1.000 | +| ctrl_mention | 3 | 20.1 | 69.0 [67.4, 70.5] | 38.4 [35.3, 41.3] | 69.0 [67.4, 70.5] | 38.4 [35.3, 41.3] | +0.0 [+0.0, +0.0] | 1.000 | -0.0 [-0.1, +0.0] | 1.000 | +| ctrl_entity | 2 | 21.5 | 69.0 [67.4, 70.5] | 38.4 [35.3, 41.3] | 69.1 [67.5, 70.6] | 38.6 [35.6, 41.5] | -0.2 [-0.5, +0.0] | 0.496 | -0.2 [-0.3, -0.0] | 0.127 | +| ctrl_entity | 3 | 37.5 | 69.0 [67.4, 70.5] | 38.4 [35.3, 41.3] | 71.0 [69.5, 72.5] | 41.2 [38.1, 44.2] | -2.8 [-3.9, -1.8] | <0.001 | -2.0 [-2.6, -1.5] | <0.001 | + +#### 2wiki/minilm: ISONGraph multi_hop (tuned) minus PPR re-rank (tuned), same graph; all-supporting (pp [95% CI], paired p) + +| Graph | k=5 | k=10 | k=20 | +|---|---|---|---| +| mention | +12.0 [+9.6, +14.4] (p=<0.001) | +0.9 [+0.2, +1.7] (p=0.036) | +0.9 [-0.3, +2.0] (p=0.169) | +| entity | +11.7 [+9.2, +14.1] (p=<0.001) | +5.7 [+3.3, +8.1] (p=<0.001) | +6.0 [+3.7, +8.4] (p=<0.001) | +| knn | +0.7 [-0.8, +2.2] (p=0.409) | -0.1 [-1.9, +1.8] (p=1.000) | +1.9 [+0.3, +3.5] (p=0.026) | +| mention+entity | +25.9 [+22.9, +28.9] (p=<0.001) | +9.0 [+7.0, +10.9] (p=<0.001) | +5.8 [+4.0, +7.6] (p=<0.001) | + +#### 2wiki/minilm: breakdown by question type, k=10 -- All-supporting@10 % (Δ vs vector, pp [95% CI]) + +| Group | n | Vector top-k (plain) | Hybrid vector+BM25 (RRF) | Vector top-30 + cross-encoder | ISONGraph multi_hop (tuned) | PPR re-rank (tuned, independent impl.) | +|---|---|---|---|---|---|---| +| bridge_comparison | 221 | 0.0 | 1.4 (+1.4 [+0.0, +3.2]) | 0.0 (+0.0 [+0.0, +0.0]) | 66.5 (+66.5 [+60.2, +72.4]) [mention+entity] | 64.7 (+64.7 [+57.9, +71.0]) [mention] | +| comparison | 238 | 71.0 | 92.0 (+21.0 [+16.0, +26.5]) | 81.5 (+10.5 [+6.7, +14.7]) | 63.0 (-8.0 [-12.6, -3.8]) [mention+entity] | 62.6 (-8.4 [-12.6, -4.6]) [mention] | +| compositional | 422 | 25.1 | 30.1 (+5.0 [+2.1, +7.8]) | 28.2 (+3.1 [+0.9, +5.2]) | 95.0 (+69.9 [+65.2, +74.4]) [mention+entity] | 92.2 (+67.1 [+61.8, +71.8]) [mention] | +| inference | 119 | 42.0 | 52.1 (+10.1 [+4.2, +16.0]) | 52.1 (+10.1 [+4.2, +16.8]) | 94.1 (+52.1 [+42.9, +61.3]) [mention+entity] | 89.1 (+47.1 [+37.8, +56.3]) [mention] | + +#### 2wiki/minilm: breakdown by hop count (number of supporting passages), k=10 -- All-supporting@10 % (Δ vs vector, pp [95% CI]) + +| Group | n | Vector top-k (plain) | Hybrid vector+BM25 (RRF) | Vector top-30 + cross-encoder | ISONGraph multi_hop (tuned) | PPR re-rank (tuned, independent impl.) | +|---|---|---|---|---|---|---| +| 2 | 779 | 41.7 | 52.4 (+10.7 [+8.3, +13.1]) | 48.1 (+6.4 [+4.6, +8.5]) | 85.1 (+43.4 [+39.5, +47.2]) [mention+entity] | 82.7 (+40.9 [+37.0, +44.7]) [mention] | +| 4 | 221 | 0.0 | 1.4 (+1.4 [+0.0, +3.2]) | 0.0 (+0.0 [+0.0, +0.0]) | 66.5 (+66.5 [+60.2, +72.4]) [mention+entity] | 64.7 (+64.7 [+57.9, +71.0]) [mention] | + +#### 2wiki/minilm: dev vs test All-supporting@10 for tuned configs (selection optimism) + +| Method | Graph | dev % | test % | +|---|---|---|---| +| lib_mh | mention | 80.6 | 79.6 | +| lib_mh | entity | 63.6 | 60.6 | +| lib_mh | knn | 45.0 | 39.8 | +| lib_mh | mention+entity | 81.8 | 81.0 | +| lib_mh | ctrl_mention | 37.2 | 32.5 | +| lib_mh | ctrl_entity | 37.2 | 32.5 | +| ppr | mention | 79.8 | 78.7 | +| ppr | entity | 58.0 | 54.9 | +| ppr | knn | 45.8 | 39.9 | +| ppr | mention+entity | 73.2 | 72.0 | +| ppr | ctrl_mention | 37.6 | 32.7 | +| ppr | ctrl_entity | 38.6 | 31.6 | +| vector | - | 37.2 | 32.5 | + +## 2wiki / bge (corpus 9239 passages, 1000 test questions) + + +### k = 5 (returned-passage budget = 5) + +| Method | Recall@k % [95% CI] | All-supporting@k % [95% CI] | MRR@k [95% CI] | mean returned | Δ all vs vector (pp) [95% CI] | p (Holm) | Δ recall vs vector (pp) [95% CI] | p (Holm) | +|---|---|---|---|---|---|---|---|---| +| Vector top-k (plain) | 69.1 [67.6, 70.6] | 38.3 [35.3, 41.2] | 0.975 [0.967, 0.982] | 5.0 | - | - | - | - | +| BM25 | 66.4 [64.9, 67.9] | 33.8 [31.1, 36.7] | 0.937 [0.925, 0.949] | 5.0 | -4.5 [-6.7, -2.4] | <0.001 (0.002) | -2.7 [-3.9, -1.6] | <0.001 (0.002) | +| Hybrid vector+BM25 (RRF) | 69.8 [68.3, 71.4] | 38.7 [35.8, 41.8] | 0.968 [0.960, 0.975] | 5.0 | +0.4 [-1.1, +1.9] | 0.695 (1.000) | +0.8 [-0.0, +1.6] | 0.075 (0.524) | +| Vector top-30 + cross-encoder | 70.2 [68.7, 71.7] | 39.4 [36.4, 42.4] | 0.974 [0.966, 0.980] | 5.0 | +1.1 [-0.7, +3.0] | 0.289 (1.000) | +1.1 [+0.1, +2.1] | 0.029 (0.234) | +| ISONGraph multi_hop (tuned) [graph chosen on dev: mention] | 91.4 [90.3, 92.4] | 77.8 [75.3, 80.4] | 0.975 [0.967, 0.982] | 5.0 | +39.5 [+36.3, +42.5] | <0.001 (0.002) | +22.3 [+20.7, +23.8] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention] | 81.4 [79.9, 83.0] | 58.9 [55.9, 61.9] | 0.832 [0.814, 0.848] | 5.0 | +20.6 [+17.5, +23.8] | <0.001 (0.002) | +12.4 [+10.8, +14.0] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph chosen on dev: mention] | 74.6 [73.2, 76.1] | 47.7 [44.7, 50.8] | 0.975 [0.967, 0.982] | 5.0 | +9.4 [+7.7, +11.3] | <0.001 (0.002) | +5.5 [+4.7, +6.5] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: mention] | 91.4 [90.3, 92.4] | 77.8 [75.3, 80.4] | 0.975 [0.967, 0.982] | 5.0 | +39.5 [+36.3, +42.5] | <0.001 (0.002) | +22.3 [+20.7, +23.8] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: entity] | 78.1 [76.6, 79.7] | 53.7 [50.6, 56.8] | 0.975 [0.967, 0.982] | 5.0 | +15.4 [+13.0, +17.8] | <0.001 (0.002) | +9.0 [+7.9, +10.3] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: knn] | 72.9 [71.3, 74.4] | 45.5 [42.5, 48.5] | 0.975 [0.967, 0.982] | 5.0 | +7.2 [+5.2, +9.2] | <0.001 (0.002) | +3.8 [+2.8, +4.8] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: ctrl_mention] | 69.1 [67.6, 70.7] | 38.4 [35.4, 41.3] | 0.975 [0.967, 0.982] | 5.0 | +0.1 [+0.0, +0.3] | 1.000 (1.000) | +0.1 [+0.0, +0.2] | 1.000 (1.000) | +| ISONGraph multi_hop (tuned) [graph: ctrl_entity] | 69.1 [67.6, 70.7] | 38.4 [35.4, 41.3] | 0.975 [0.967, 0.982] | 5.0 | +0.1 [+0.0, +0.3] | 1.000 (1.000) | +0.1 [+0.0, +0.2] | 1.000 (1.000) | +| PPR re-rank (tuned, independent impl.) [graph: mention] | 81.4 [79.9, 83.0] | 58.9 [55.9, 61.9] | 0.832 [0.814, 0.848] | 5.0 | +20.6 [+17.5, +23.8] | <0.001 (0.002) | +12.4 [+10.8, +14.0] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph: entity] | 72.9 [71.4, 74.4] | 43.5 [40.5, 46.6] | 0.808 [0.791, 0.826] | 5.0 | +5.2 [+3.3, +7.1] | <0.001 (0.002) | +3.8 [+2.7, +4.8] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph: knn] | 69.1 [67.6, 70.7] | 38.7 [35.8, 41.7] | 0.954 [0.944, 0.964] | 5.0 | +0.4 [-1.1, +1.9] | 0.687 (1.000) | +0.0 [-0.7, +0.8] | 1.000 (1.000) | +| PPR re-rank (tuned, independent impl.) [graph: ctrl_mention] | 68.8 [67.3, 70.2] | 37.8 [35.0, 40.7] | 0.966 [0.957, 0.974] | 5.0 | -0.5 [-1.3, +0.3] | 0.357 (1.000) | -0.3 [-0.7, +0.1] | 0.180 (1.000) | +| PPR re-rank (tuned, independent impl.) [graph: ctrl_entity] | 69.1 [67.6, 70.6] | 38.3 [35.3, 41.2] | 0.902 [0.889, 0.915] | 5.0 | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +| ISONGraph multi_hop (library defaults) [graph: mention] | 74.6 [73.2, 76.1] | 47.7 [44.7, 50.8] | 0.975 [0.967, 0.982] | 5.0 | +9.4 [+7.7, +11.3] | <0.001 (0.002) | +5.5 [+4.7, +6.5] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: entity] | 71.9 [70.4, 73.5] | 42.9 [39.9, 46.0] | 0.975 [0.967, 0.982] | 5.0 | +4.6 [+3.1, +6.2] | <0.001 (0.002) | +2.9 [+2.1, +3.7] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: knn] | 70.3 [68.8, 71.9] | 40.6 [37.6, 43.6] | 0.975 [0.967, 0.982] | 5.0 | +2.3 [+1.0, +3.7] | 0.001 (0.011) | +1.3 [+0.6, +2.0] | <0.001 (0.007) | +| ISONGraph multi_hop (library defaults) [graph: ctrl_mention] | 69.0 [67.5, 70.6] | 38.2 [35.2, 41.1] | 0.975 [0.967, 0.982] | 5.0 | -0.1 [-0.4, +0.2] | 1.000 (1.000) | -0.1 [-0.2, +0.1] | 1.000 (1.000) | +| ISONGraph multi_hop (library defaults) [graph: ctrl_entity] | 68.5 [67.0, 70.0] | 37.1 [34.1, 39.9] | 0.975 [0.967, 0.982] | 5.0 | -1.2 [-1.9, -0.6] | <0.001 (0.008) | -0.6 [-0.9, -0.3] | <0.001 (0.007) | + +#### semantic_subgraph(connect=True), top_k_results=5: set quality vs plain vector top-|S| (size-matched per query) + +| Graph | connect_max_hops | mean |S| | Recall % | All-supp. % | Vector@|S| recall % | Vector@|S| all % | Δ all (pp) [CI] | p | Δ recall (pp) [CI] | p | +|---|---|---|---|---|---|---|---|---|---|---| +| mention | 2 | 5.0 | 91.4 [90.3, 92.5] | 77.9 [75.4, 80.5] | 69.2 [67.7, 70.8] | 38.5 [35.5, 41.4] | +39.4 [+36.1, +42.5] | <0.001 | +22.2 [+20.7, +23.7] | <0.001 | +| mention | 3 | 5.1 | 91.5 [90.5, 92.6] | 78.0 [75.5, 80.6] | 69.2 [67.7, 70.8] | 38.5 [35.5, 41.4] | +39.5 [+36.2, +42.6] | <0.001 | +22.3 [+20.7, +23.8] | <0.001 | +| ctrl_mention | 2 | 5.0 | 69.1 [67.6, 70.7] | 38.4 [35.4, 41.3] | 69.1 [67.6, 70.6] | 38.3 [35.3, 41.2] | +0.1 [+0.0, +0.3] | 1.000 | +0.1 [+0.0, +0.2] | 1.000 | +| ctrl_mention | 3 | 5.0 | 69.1 [67.6, 70.7] | 38.4 [35.4, 41.3] | 69.1 [67.6, 70.6] | 38.3 [35.3, 41.2] | +0.1 [+0.0, +0.3] | 1.000 | +0.1 [+0.0, +0.2] | 1.000 | +| ctrl_entity | 2 | 5.1 | 69.1 [67.6, 70.7] | 38.4 [35.4, 41.3] | 69.2 [67.7, 70.7] | 38.4 [35.4, 41.3] | +0.0 [-0.3, +0.3] | 1.000 | -0.1 [-0.2, +0.1] | 1.000 | +| ctrl_entity | 3 | 6.3 | 69.1 [67.6, 70.7] | 38.4 [35.4, 41.3] | 70.4 [68.8, 72.0] | 40.6 [37.6, 43.6] | -2.2 [-3.2, -1.3] | <0.001 | -1.3 [-1.8, -0.8] | <0.001 | + +### k = 10 (returned-passage budget = 10) + +| Method | Recall@k % [95% CI] | All-supporting@k % [95% CI] | MRR@k [95% CI] | mean returned | Δ all vs vector (pp) [95% CI] | p (Holm) | Δ recall vs vector (pp) [95% CI] | p (Holm) | +|---|---|---|---|---|---|---|---|---| +| Vector top-k (plain) | 72.3 [70.7, 73.8] | 43.9 [40.9, 46.9] | 0.975 [0.968, 0.982] | 10.0 | - | - | - | - | +| BM25 | 71.2 [69.7, 72.8] | 41.5 [38.5, 44.5] | 0.938 [0.926, 0.950] | 10.0 | -2.4 [-4.3, -0.4] | 0.019 (0.115) | -1.1 [-2.1, +0.0] | 0.047 (0.237) | +| Hybrid vector+BM25 (RRF) | 73.8 [72.2, 75.2] | 45.6 [42.4, 48.5] | 0.968 [0.960, 0.976] | 10.0 | +1.7 [+0.1, +3.2] | 0.046 (0.230) | +1.5 [+0.7, +2.3] | <0.001 (0.007) | +| Vector top-30 + cross-encoder | 73.7 [72.1, 75.2] | 46.0 [43.0, 49.0] | 0.974 [0.966, 0.980] | 10.0 | +2.1 [+0.6, +3.6] | 0.011 (0.075) | +1.4 [+0.5, +2.2] | 0.001 (0.008) | +| ISONGraph multi_hop (tuned) [graph chosen on dev: mention] | 97.4 [96.7, 98.1] | 94.6 [93.2, 96.0] | 0.975 [0.967, 0.982] | 8.1 | +50.7 [+47.5, +53.9] | <0.001 (0.002) | +25.1 [+23.5, +26.7] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention] | 97.0 [96.3, 97.7] | 92.8 [91.2, 94.4] | 0.870 [0.855, 0.885] | 10.0 | +48.9 [+45.8, +52.1] | <0.001 (0.002) | +24.7 [+23.3, +26.3] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph chosen on dev: mention] | 97.4 [96.7, 98.1] | 94.6 [93.2, 96.0] | 0.975 [0.967, 0.982] | 8.6 | +50.7 [+47.5, +53.9] | <0.001 (0.002) | +25.1 [+23.5, +26.7] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: mention] | 97.4 [96.7, 98.1] | 94.6 [93.2, 96.0] | 0.975 [0.967, 0.982] | 8.1 | +50.7 [+47.5, +53.9] | <0.001 (0.002) | +25.1 [+23.5, +26.7] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: entity] | 89.6 [88.5, 90.8] | 74.2 [71.6, 76.8] | 0.975 [0.967, 0.982] | 10.0 | +30.3 [+27.3, +33.0] | <0.001 (0.002) | +17.3 [+15.8, +18.7] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: knn] | 83.9 [82.4, 85.2] | 63.9 [60.9, 66.7] | 0.975 [0.967, 0.982] | 10.0 | +20.0 [+17.5, +22.5] | <0.001 (0.002) | +11.6 [+10.2, +12.8] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: ctrl_mention] | 72.2 [70.7, 73.7] | 43.7 [40.7, 46.7] | 0.975 [0.967, 0.982] | 10.0 | -0.2 [-0.5, +0.0] | 0.497 (1.000) | -0.1 [-0.2, +0.0] | 0.497 (1.000) | +| ISONGraph multi_hop (tuned) [graph: ctrl_entity] | 72.3 [70.7, 73.8] | 43.8 [40.8, 46.8] | 0.975 [0.967, 0.982] | 10.0 | -0.1 [-0.3, +0.0] | 1.000 (1.000) | -0.1 [-0.2, +0.0] | 1.000 (1.000) | +| PPR re-rank (tuned, independent impl.) [graph: mention] | 97.0 [96.3, 97.7] | 92.8 [91.2, 94.4] | 0.870 [0.855, 0.885] | 10.0 | +48.9 [+45.8, +52.1] | <0.001 (0.002) | +24.7 [+23.3, +26.3] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph: entity] | 87.4 [86.1, 88.7] | 69.7 [67.0, 72.6] | 0.920 [0.908, 0.932] | 10.0 | +25.8 [+23.0, +28.6] | <0.001 (0.002) | +15.1 [+13.7, +16.5] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph: knn] | 81.2 [79.8, 82.6] | 58.5 [55.5, 61.4] | 0.919 [0.907, 0.932] | 10.0 | +14.6 [+12.3, +16.8] | <0.001 (0.002) | +9.0 [+7.7, +10.2] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph: ctrl_mention] | 72.3 [70.7, 73.8] | 43.9 [40.9, 46.9] | 0.973 [0.965, 0.981] | 10.0 | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +| PPR re-rank (tuned, independent impl.) [graph: ctrl_entity] | 72.3 [70.7, 73.8] | 43.9 [40.9, 47.0] | 0.974 [0.967, 0.981] | 10.0 | +0.0 [-0.3, +0.3] | 1.000 (1.000) | +0.0 [-0.2, +0.2] | 1.000 (1.000) | +| ISONGraph multi_hop (library defaults) [graph: mention] | 97.4 [96.7, 98.1] | 94.6 [93.2, 96.0] | 0.975 [0.967, 0.982] | 8.6 | +50.7 [+47.5, +53.9] | <0.001 (0.002) | +25.1 [+23.5, +26.7] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: entity] | 87.2 [85.9, 88.4] | 69.3 [66.4, 72.0] | 0.975 [0.967, 0.982] | 10.0 | +25.4 [+22.4, +28.3] | <0.001 (0.002) | +14.9 [+13.3, +16.3] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: knn] | 82.7 [81.2, 84.0] | 61.4 [58.4, 64.3] | 0.975 [0.967, 0.982] | 10.0 | +17.5 [+15.0, +19.9] | <0.001 (0.002) | +10.4 [+9.1, +11.7] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: ctrl_mention] | 69.1 [67.6, 70.7] | 38.4 [35.4, 41.3] | 0.975 [0.967, 0.982] | 8.7 | -5.5 [-6.9, -4.1] | <0.001 (0.002) | -3.2 [-3.9, -2.4] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: ctrl_entity] | 69.0 [67.5, 70.5] | 38.1 [35.2, 41.0] | 0.975 [0.967, 0.982] | 10.0 | -5.8 [-7.3, -4.3] | <0.001 (0.002) | -3.3 [-4.1, -2.6] | <0.001 (0.002) | + +#### semantic_subgraph(connect=True), top_k_results=10: set quality vs plain vector top-|S| (size-matched per query) + +| Graph | connect_max_hops | mean |S| | Recall % | All-supp. % | Vector@|S| recall % | Vector@|S| all % | Δ all (pp) [CI] | p | Δ recall (pp) [CI] | p | +|---|---|---|---|---|---|---|---|---|---|---| +| mention | 2 | 8.2 | 97.4 [96.7, 98.1] | 94.6 [93.2, 96.0] | 71.5 [69.9, 73.0] | 42.4 [39.4, 45.4] | +52.2 [+49.0, +55.3] | <0.001 | +25.9 [+24.3, +27.5] | <0.001 | +| mention | 3 | 8.2 | 97.4 [96.7, 98.1] | 94.6 [93.2, 96.0] | 71.5 [69.9, 73.0] | 42.4 [39.4, 45.4] | +52.2 [+49.0, +55.3] | <0.001 | +25.9 [+24.3, +27.5] | <0.001 | +| entity | 2 | 10.5 | 89.7 [88.6, 90.9] | 74.4 [71.7, 77.1] | 72.6 [71.0, 74.1] | 44.5 [41.4, 47.6] | +29.9 [+26.9, +32.7] | <0.001 | +17.1 [+15.6, +18.5] | <0.001 | +| entity | 3 | 12.3 | 90.0 [88.8, 91.1] | 75.1 [72.4, 77.7] | 73.0 [71.5, 74.6] | 45.0 [42.0, 48.0] | +30.1 [+27.1, +32.9] | <0.001 | +17.0 [+15.4, +18.4] | <0.001 | +| knn | 2 | 10.7 | 84.1 [82.7, 85.5] | 64.3 [61.3, 67.1] | 72.5 [71.0, 74.1] | 44.3 [41.3, 47.3] | +20.0 [+17.4, +22.4] | <0.001 | +11.6 [+10.3, +12.9] | <0.001 | +| knn | 3 | 12.1 | 84.5 [83.2, 85.9] | 65.0 [62.1, 67.8] | 72.8 [71.3, 74.4] | 44.7 [41.7, 47.7] | +20.3 [+17.7, +22.7] | <0.001 | +11.7 [+10.4, +13.0] | <0.001 | +| ctrl_mention | 2 | 10.0 | 72.2 [70.7, 73.7] | 43.7 [40.7, 46.7] | 72.3 [70.8, 73.8] | 43.9 [40.9, 46.9] | -0.2 [-0.5, +0.0] | 0.497 | -0.1 [-0.2, +0.0] | 0.497 | +| ctrl_mention | 3 | 10.0 | 72.2 [70.7, 73.7] | 43.7 [40.7, 46.7] | 72.3 [70.8, 73.8] | 43.9 [40.9, 46.9] | -0.2 [-0.5, +0.0] | 0.497 | -0.1 [-0.2, +0.0] | 0.497 | +| ctrl_entity | 2 | 10.3 | 72.3 [70.7, 73.8] | 43.8 [40.8, 46.8] | 72.4 [70.8, 73.9] | 44.0 [40.9, 47.0] | -0.2 [-0.5, +0.0] | 0.497 | -0.1 [-0.2, +0.0] | 0.497 | +| ctrl_entity | 3 | 15.9 | 72.3 [70.7, 73.8] | 43.8 [40.8, 46.8] | 74.0 [72.5, 75.6] | 46.7 [43.6, 49.9] | -2.9 [-4.0, -1.9] | <0.001 | -1.8 [-2.3, -1.2] | <0.001 | + +### k = 20 (returned-passage budget = 20) + +| Method | Recall@k % [95% CI] | All-supporting@k % [95% CI] | MRR@k [95% CI] | mean returned | Δ all vs vector (pp) [95% CI] | p (Holm) | Δ recall vs vector (pp) [95% CI] | p (Holm) | +|---|---|---|---|---|---|---|---|---| +| Vector top-k (plain) | 74.9 [73.4, 76.4] | 48.2 [45.2, 51.2] | 0.975 [0.968, 0.982] | 20.0 | - | - | - | - | +| BM25 | 74.3 [72.8, 75.8] | 46.7 [43.8, 49.8] | 0.939 [0.927, 0.950] | 20.0 | -1.5 [-3.3, +0.2] | 0.114 (0.685) | -0.6 [-1.5, +0.3] | 0.194 (0.969) | +| Hybrid vector+BM25 (RRF) | 75.8 [74.3, 77.3] | 49.4 [46.4, 52.4] | 0.968 [0.960, 0.976] | 20.0 | +1.2 [-0.4, +2.7] | 0.173 (0.864) | +0.9 [+0.1, +1.7] | 0.031 (0.186) | +| Vector top-30 + cross-encoder | 76.0 [74.4, 77.5] | 50.1 [47.0, 53.1] | 0.974 [0.966, 0.980] | 20.0 | +1.9 [+0.8, +3.1] | 0.004 (0.026) | +1.0 [+0.5, +1.7] | 0.001 (0.008) | +| ISONGraph multi_hop (tuned) [graph chosen on dev: mention] | 98.3 [97.7, 98.8] | 96.4 [95.2, 97.5] | 0.975 [0.967, 0.982] | 17.6 | +48.2 [+45.1, +51.3] | <0.001 (0.002) | +23.4 [+21.8, +24.9] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention] | 98.0 [97.4, 98.6] | 95.5 [94.2, 96.7] | 0.712 [0.692, 0.732] | 20.0 | +47.3 [+44.2, +50.3] | <0.001 (0.002) | +23.1 [+21.5, +24.6] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph chosen on dev: mention] | 97.5 [96.7, 98.1] | 94.8 [93.4, 96.1] | 0.975 [0.967, 0.982] | 10.1 | +46.6 [+43.4, +49.7] | <0.001 (0.002) | +22.6 [+20.9, +24.1] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: mention] | 98.3 [97.7, 98.8] | 96.4 [95.2, 97.5] | 0.975 [0.967, 0.982] | 17.6 | +48.2 [+45.1, +51.3] | <0.001 (0.002) | +23.4 [+21.8, +24.9] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: entity] | 93.4 [92.4, 94.4] | 84.3 [82.1, 86.5] | 0.975 [0.967, 0.982] | 19.9 | +36.1 [+32.9, +39.5] | <0.001 (0.002) | +18.5 [+16.9, +20.1] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: knn] | 88.3 [87.0, 89.5] | 71.8 [68.9, 74.5] | 0.975 [0.967, 0.982] | 20.0 | +23.6 [+20.8, +26.2] | <0.001 (0.002) | +13.4 [+11.9, +14.7] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: ctrl_mention] | 74.9 [73.4, 76.4] | 48.2 [45.2, 51.2] | 0.975 [0.967, 0.982] | 20.0 | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +| ISONGraph multi_hop (tuned) [graph: ctrl_entity] | 74.9 [73.4, 76.4] | 48.2 [45.2, 51.2] | 0.975 [0.967, 0.982] | 20.0 | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +| PPR re-rank (tuned, independent impl.) [graph: mention] | 98.0 [97.4, 98.6] | 95.5 [94.2, 96.7] | 0.712 [0.692, 0.732] | 20.0 | +47.3 [+44.2, +50.3] | <0.001 (0.002) | +23.1 [+21.5, +24.6] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph: entity] | 93.3 [92.2, 94.2] | 83.0 [80.5, 85.3] | 0.873 [0.858, 0.888] | 20.0 | +34.8 [+31.8, +37.8] | <0.001 (0.002) | +18.4 [+16.9, +19.8] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph: knn] | 87.5 [86.2, 88.8] | 70.2 [67.5, 73.0] | 0.919 [0.908, 0.932] | 20.0 | +22.0 [+19.1, +24.6] | <0.001 (0.002) | +12.6 [+11.2, +14.0] | <0.001 (0.002) | +| PPR re-rank (tuned, independent impl.) [graph: ctrl_mention] | 74.9 [73.4, 76.4] | 48.2 [45.2, 51.2] | 0.974 [0.966, 0.981] | 20.0 | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +| PPR re-rank (tuned, independent impl.) [graph: ctrl_entity] | 74.8 [73.3, 76.4] | 48.1 [45.1, 51.2] | 0.964 [0.955, 0.973] | 20.0 | -0.1 [-0.5, +0.2] | 1.000 (1.000) | -0.1 [-0.2, +0.1] | 1.000 (1.000) | +| ISONGraph multi_hop (library defaults) [graph: mention] | 97.5 [96.7, 98.1] | 94.8 [93.4, 96.1] | 0.975 [0.967, 0.982] | 10.1 | +46.6 [+43.4, +49.7] | <0.001 (0.002) | +22.6 [+20.9, +24.1] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: entity] | 93.4 [92.4, 94.4] | 84.3 [82.1, 86.5] | 0.975 [0.967, 0.982] | 19.9 | +36.1 [+32.9, +39.5] | <0.001 (0.002) | +18.5 [+16.9, +20.1] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: knn] | 88.3 [87.0, 89.5] | 71.8 [68.9, 74.5] | 0.975 [0.967, 0.982] | 20.0 | +23.6 [+20.8, +26.2] | <0.001 (0.002) | +13.4 [+11.9, +14.7] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: ctrl_mention] | 69.1 [67.6, 70.7] | 38.4 [35.4, 41.3] | 0.975 [0.967, 0.982] | 9.9 | -9.8 [-11.6, -8.0] | <0.001 (0.002) | -5.8 [-6.7, -4.8] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: ctrl_entity] | 69.2 [67.7, 70.7] | 38.5 [35.5, 41.4] | 0.975 [0.967, 0.982] | 19.9 | -9.7 [-11.5, -7.9] | <0.001 (0.002) | -5.7 [-6.7, -4.8] | <0.001 (0.002) | + +#### semantic_subgraph(connect=True), top_k_results=20: set quality vs plain vector top-|S| (size-matched per query) + +| Graph | connect_max_hops | mean |S| | Recall % | All-supp. % | Vector@|S| recall % | Vector@|S| all % | Δ all (pp) [CI] | p | Δ recall (pp) [CI] | p | +|---|---|---|---|---|---|---|---|---|---|---| +| mention | 2 | 17.6 | 98.3 [97.7, 98.8] | 96.4 [95.2, 97.5] | 74.3 [72.8, 75.9] | 47.1 [44.1, 50.2] | +49.3 [+46.1, +52.4] | <0.001 | +24.0 [+22.4, +25.5] | <0.001 | +| mention | 3 | 17.7 | 98.3 [97.7, 98.8] | 96.4 [95.2, 97.5] | 74.3 [72.8, 75.9] | 47.1 [44.1, 50.2] | +49.3 [+46.1, +52.4] | <0.001 | +24.0 [+22.4, +25.5] | <0.001 | +| ctrl_mention | 2 | 20.0 | 74.9 [73.4, 76.4] | 48.2 [45.2, 51.2] | 74.9 [73.4, 76.4] | 48.2 [45.2, 51.2] | +0.0 [+0.0, +0.0] | 1.000 | +0.0 [+0.0, +0.0] | 1.000 | +| ctrl_mention | 3 | 20.2 | 74.9 [73.4, 76.4] | 48.2 [45.2, 51.2] | 75.0 [73.5, 76.5] | 48.3 [45.3, 51.4] | -0.1 [-0.4, +0.0] | 1.000 | -0.1 [-0.2, +0.0] | 1.000 | +| ctrl_entity | 2 | 21.5 | 74.9 [73.4, 76.4] | 48.2 [45.2, 51.2] | 75.1 [73.6, 76.6] | 48.5 [45.5, 51.5] | -0.3 [-0.7, +0.0] | 0.240 | -0.2 [-0.4, -0.1] | 0.059 | +| ctrl_entity | 3 | 37.3 | 75.0 [73.5, 76.5] | 48.3 [45.3, 51.3] | 77.6 [76.1, 79.1] | 53.0 [50.0, 56.0] | -4.7 [-6.1, -3.4] | <0.001 | -2.6 [-3.3, -1.9] | <0.001 | + +#### 2wiki/bge: ISONGraph multi_hop (tuned) minus PPR re-rank (tuned), same graph; all-supporting (pp [95% CI], paired p) + +| Graph | k=5 | k=10 | k=20 | +|---|---|---|---| +| mention | +18.9 [+15.9, +21.9] (p=<0.001) | +1.8 [+0.7, +2.9] (p=0.002) | +0.9 [+0.4, +1.5] (p=0.004) | +| entity | +10.2 [+7.6, +12.7] (p=<0.001) | +4.5 [+2.2, +6.7] (p=<0.001) | +1.3 [-0.6, +3.3] (p=0.235) | +| knn | +6.8 [+4.7, +8.8] (p=<0.001) | +5.4 [+3.6, +7.2] (p=<0.001) | +1.6 [+0.3, +2.9] (p=0.034) | + +#### 2wiki/bge: breakdown by question type, k=10 -- All-supporting@10 % (Δ vs vector, pp [95% CI]) + +| Group | n | Vector top-k (plain) | Hybrid vector+BM25 (RRF) | Vector top-30 + cross-encoder | ISONGraph multi_hop (tuned) | PPR re-rank (tuned, independent impl.) | +|---|---|---|---|---|---|---| +| bridge_comparison | 221 | 0.9 | 1.8 (+0.9 [+0.0, +2.3]) | 1.4 (+0.5 [+0.0, +1.4]) | 90.5 (+89.6 [+85.5, +93.7]) [mention] | 81.9 (+81.0 [+75.6, +86.0]) [mention] | +| comparison | 238 | 97.9 | 99.2 (+1.3 [+0.0, +2.9]) | 97.9 (+0.0 [-1.3, +1.3]) | 97.1 (-0.8 [-2.5, +0.8]) [mention] | 97.9 (+0.0 [-1.3, +1.3]) [mention] | +| compositional | 422 | 32.9 | 34.4 (+1.4 [-1.7, +4.5]) | 36.7 (+3.8 [+0.7, +6.9]) | 95.3 (+62.3 [+57.6, +67.3]) [mention] | 95.0 (+62.1 [+57.6, +66.8]) [mention] | +| inference | 119 | 54.6 | 59.7 (+5.0 [-0.8, +11.8]) | 58.0 (+3.4 [-3.4, +10.1]) | 95.0 (+40.3 [+31.1, +49.6]) [mention] | 95.0 (+40.3 [+31.9, +48.7]) [mention] | + +#### 2wiki/bge: breakdown by hop count (number of supporting passages), k=10 -- All-supporting@10 % (Δ vs vector, pp [95% CI]) + +| Group | n | Vector top-k (plain) | Hybrid vector+BM25 (RRF) | Vector top-30 + cross-encoder | ISONGraph multi_hop (tuned) | PPR re-rank (tuned, independent impl.) | +|---|---|---|---|---|---|---| +| 2 | 779 | 56.1 | 58.0 (+1.9 [-0.0, +3.9]) | 58.7 (+2.6 [+0.5, +4.7]) | 95.8 (+39.7 [+36.2, +43.1]) [mention] | 95.9 (+39.8 [+36.3, +43.3]) [mention] | +| 4 | 221 | 0.9 | 1.8 (+0.9 [+0.0, +2.3]) | 1.4 (+0.5 [+0.0, +1.4]) | 90.5 (+89.6 [+85.5, +93.7]) [mention] | 81.9 (+81.0 [+75.6, +86.0]) [mention] | + +#### 2wiki/bge: dev vs test All-supporting@10 for tuned configs (selection optimism) + +| Method | Graph | dev % | test % | +|---|---|---|---| +| lib_mh | mention | 94.2 | 94.6 | +| lib_mh | entity | 74.4 | 74.2 | +| lib_mh | knn | 66.6 | 63.9 | +| lib_mh | ctrl_mention | 47.6 | 43.7 | +| lib_mh | ctrl_entity | 47.6 | 43.8 | +| ppr | mention | 92.8 | 92.8 | +| ppr | entity | 71.4 | 69.7 | +| ppr | knn | 60.4 | 58.5 | +| ppr | ctrl_mention | 47.6 | 43.9 | +| ppr | ctrl_entity | 47.6 | 43.9 | +| vector | - | 47.6 | 43.9 | + +## musique / minilm (corpus 15405 passages, 1000 test questions) + + +### k = 5 (returned-passage budget = 5) + +| Method | Recall@k % [95% CI] | All-supporting@k % [95% CI] | MRR@k [95% CI] | mean returned | Δ all vs vector (pp) [95% CI] | p (Holm) | Δ recall vs vector (pp) [95% CI] | p (Holm) | +|---|---|---|---|---|---|---|---|---| +| Vector top-k (plain) | 45.5 [43.8, 47.3] | 11.9 [10.0, 13.9] | 0.702 [0.678, 0.726] | 5.0 | - | - | - | - | +| BM25 | 44.9 [43.3, 46.6] | 11.1 [9.3, 12.9] | 0.722 [0.698, 0.745] | 5.0 | -0.8 [-2.7, +1.2] | 0.503 (1.000) | -0.5 [-2.3, +1.2] | 0.548 (1.000) | +| Hybrid vector+BM25 (RRF) | 48.7 [47.0, 50.3] | 12.6 [10.6, 14.6] | 0.733 [0.710, 0.754] | 5.0 | +0.7 [-0.9, +2.3] | 0.451 (1.000) | +3.2 [+1.8, +4.5] | <0.001 (0.001) | +| Vector top-30 + cross-encoder | 52.4 [50.6, 54.1] | 17.8 [15.5, 20.2] | 0.800 [0.779, 0.821] | 5.0 | +5.9 [+4.2, +7.7] | <0.001 (0.001) | +6.9 [+5.5, +8.2] | <0.001 (0.001) | +| ISONGraph multi_hop (tuned) [graph chosen on dev: mention] | 48.5 [46.6, 50.4] | 18.4 [16.1, 20.8] | 0.702 [0.678, 0.726] | 5.0 | +6.5 [+4.9, +8.2] | <0.001 (0.001) | +3.0 [+2.1, +3.9] | <0.001 (0.001) | +| PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention] | 48.3 [46.4, 50.3] | 18.3 [16.0, 20.6] | 0.588 [0.564, 0.613] | 5.0 | +6.4 [+4.5, +8.4] | <0.001 (0.001) | +2.9 [+1.7, +3.9] | <0.001 (0.001) | +| ISONGraph multi_hop (library defaults) [graph chosen on dev: mention] | 48.7 [46.8, 50.6] | 18.2 [15.9, 20.6] | 0.702 [0.677, 0.726] | 5.0 | +6.3 [+4.7, +8.1] | <0.001 (0.001) | +3.2 [+2.3, +4.1] | <0.001 (0.001) | +| ISONGraph multi_hop (tuned) [graph: mention] | 48.5 [46.6, 50.4] | 18.4 [16.1, 20.8] | 0.702 [0.678, 0.726] | 5.0 | +6.5 [+4.9, +8.2] | <0.001 (0.001) | +3.0 [+2.1, +3.9] | <0.001 (0.001) | +| ISONGraph multi_hop (tuned) [graph: ctrl_mention] | 45.5 [43.9, 47.3] | 12.0 [10.1, 14.0] | 0.702 [0.678, 0.726] | 5.0 | +0.1 [+0.0, +0.3] | 1.000 (1.000) | +0.1 [+0.0, +0.2] | 1.000 (1.000) | +| PPR re-rank (tuned, independent impl.) [graph: mention] | 48.3 [46.4, 50.3] | 18.3 [16.0, 20.6] | 0.588 [0.564, 0.613] | 5.0 | +6.4 [+4.5, +8.4] | <0.001 (0.001) | +2.9 [+1.7, +3.9] | <0.001 (0.001) | +| PPR re-rank (tuned, independent impl.) [graph: ctrl_mention] | 45.4 [43.7, 47.2] | 12.0 [10.1, 14.0] | 0.695 [0.671, 0.720] | 5.0 | +0.1 [-0.2, +0.4] | 1.000 (1.000) | -0.1 [-0.3, +0.2] | 0.772 (1.000) | +| ISONGraph multi_hop (library defaults) [graph: mention] | 48.7 [46.8, 50.6] | 18.2 [15.9, 20.6] | 0.702 [0.677, 0.726] | 5.0 | +6.3 [+4.7, +8.1] | <0.001 (0.001) | +3.2 [+2.3, +4.1] | <0.001 (0.001) | +| ISONGraph multi_hop (library defaults) [graph: ctrl_mention] | 44.7 [43.0, 46.5] | 11.3 [9.4, 13.2] | 0.701 [0.677, 0.726] | 5.0 | -0.6 [-1.2, -0.1] | 0.070 (0.352) | -0.7 [-1.1, -0.4] | <0.001 (0.001) | + +### k = 10 (returned-passage budget = 10) + +| Method | Recall@k % [95% CI] | All-supporting@k % [95% CI] | MRR@k [95% CI] | mean returned | Δ all vs vector (pp) [95% CI] | p (Holm) | Δ recall vs vector (pp) [95% CI] | p (Holm) | +|---|---|---|---|---|---|---|---|---| +| Vector top-k (plain) | 53.2 [51.4, 54.9] | 18.2 [15.8, 20.6] | 0.710 [0.686, 0.733] | 10.0 | - | - | - | - | +| BM25 | 51.5 [49.8, 53.1] | 15.2 [13.0, 17.3] | 0.730 [0.707, 0.752] | 10.0 | -3.0 [-5.3, -0.8] | 0.013 (0.051) | -1.7 [-3.5, +0.1] | 0.071 (0.353) | +| Hybrid vector+BM25 (RRF) | 56.7 [55.1, 58.5] | 20.4 [17.8, 22.9] | 0.741 [0.719, 0.761] | 10.0 | +2.2 [+0.2, +4.1] | 0.033 (0.098) | +3.6 [+2.1, +5.0] | <0.001 (0.001) | +| Vector top-30 + cross-encoder | 58.2 [56.5, 59.9] | 23.7 [21.2, 26.3] | 0.804 [0.784, 0.824] | 10.0 | +5.5 [+3.8, +7.2] | <0.001 (0.001) | +5.0 [+3.9, +6.2] | <0.001 (0.001) | +| ISONGraph multi_hop (tuned) [graph chosen on dev: mention] | 57.5 [55.6, 59.5] | 27.4 [24.7, 30.2] | 0.710 [0.687, 0.733] | 10.0 | +9.2 [+7.2, +11.3] | <0.001 (0.001) | +4.4 [+3.3, +5.5] | <0.001 (0.001) | +| PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention] | 57.1 [55.1, 59.1] | 28.9 [26.1, 31.6] | 0.648 [0.624, 0.672] | 10.0 | +10.7 [+8.4, +13.3] | <0.001 (0.001) | +4.0 [+2.7, +5.4] | <0.001 (0.001) | +| ISONGraph multi_hop (library defaults) [graph chosen on dev: mention] | 53.6 [51.6, 55.6] | 24.5 [21.8, 27.3] | 0.705 [0.681, 0.728] | 8.7 | +6.3 [+3.8, +9.1] | <0.001 (0.001) | +0.4 [-1.1, +1.9] | 0.592 (1.000) | +| ISONGraph multi_hop (tuned) [graph: mention] | 57.5 [55.6, 59.5] | 27.4 [24.7, 30.2] | 0.710 [0.687, 0.733] | 10.0 | +9.2 [+7.2, +11.3] | <0.001 (0.001) | +4.4 [+3.3, +5.5] | <0.001 (0.001) | +| ISONGraph multi_hop (tuned) [graph: ctrl_mention] | 53.2 [51.4, 54.9] | 18.2 [15.8, 20.6] | 0.710 [0.686, 0.734] | 10.0 | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +| PPR re-rank (tuned, independent impl.) [graph: mention] | 57.1 [55.1, 59.1] | 28.9 [26.1, 31.6] | 0.648 [0.624, 0.672] | 10.0 | +10.7 [+8.4, +13.3] | <0.001 (0.001) | +4.0 [+2.7, +5.4] | <0.001 (0.001) | +| PPR re-rank (tuned, independent impl.) [graph: ctrl_mention] | 53.1 [51.3, 54.9] | 18.2 [15.8, 20.6] | 0.706 [0.682, 0.730] | 10.0 | +0.0 [-0.3, +0.3] | 1.000 (1.000) | -0.0 [-0.2, +0.2] | 0.815 (1.000) | +| ISONGraph multi_hop (library defaults) [graph: mention] | 53.6 [51.6, 55.6] | 24.5 [21.8, 27.3] | 0.705 [0.681, 0.728] | 8.7 | +6.3 [+3.8, +9.1] | <0.001 (0.001) | +0.4 [-1.1, +1.9] | 0.592 (1.000) | +| ISONGraph multi_hop (library defaults) [graph: ctrl_mention] | 45.2 [43.5, 47.0] | 11.6 [9.7, 13.6] | 0.702 [0.677, 0.726] | 8.9 | -6.6 [-8.2, -5.0] | <0.001 (0.001) | -7.9 [-9.0, -6.9] | <0.001 (0.001) | + +#### semantic_subgraph(connect=True), top_k_results=10: set quality vs plain vector top-|S| (size-matched per query) + +| Graph | connect_max_hops | mean |S| | Recall % | All-supp. % | Vector@|S| recall % | Vector@|S| all % | Δ all (pp) [CI] | p | Δ recall (pp) [CI] | p | +|---|---|---|---|---|---|---|---|---|---|---| +| mention | 2 | 10.3 | 57.7 [55.7, 59.6] | 27.4 [24.7, 30.2] | 53.6 [51.8, 55.3] | 18.5 [16.1, 20.9] | +8.9 [+6.9, +11.0] | <0.001 | +4.1 [+3.0, +5.2] | <0.001 | +| mention | 3 | 10.8 | 57.9 [56.0, 59.8] | 27.7 [24.9, 30.4] | 54.1 [52.4, 55.9] | 19.0 [16.6, 21.5] | +8.7 [+6.6, +10.9] | <0.001 | +3.8 [+2.6, +4.9] | <0.001 | +| ctrl_mention | 2 | 10.0 | 53.2 [51.4, 54.9] | 18.2 [15.8, 20.6] | 53.2 [51.5, 55.0] | 18.2 [15.8, 20.6] | +0.0 [+0.0, +0.0] | 1.000 | -0.0 [-0.1, +0.0] | 1.000 | +| ctrl_mention | 3 | 10.0 | 53.2 [51.4, 54.9] | 18.2 [15.8, 20.6] | 53.2 [51.5, 55.0] | 18.2 [15.8, 20.6] | +0.0 [+0.0, +0.0] | 1.000 | -0.0 [-0.1, +0.0] | 1.000 | + +### k = 20 (returned-passage budget = 20) + +| Method | Recall@k % [95% CI] | All-supporting@k % [95% CI] | MRR@k [95% CI] | mean returned | Δ all vs vector (pp) [95% CI] | p (Holm) | Δ recall vs vector (pp) [95% CI] | p (Holm) | +|---|---|---|---|---|---|---|---|---| +| Vector top-k (plain) | 60.7 [58.9, 62.5] | 26.3 [23.6, 29.0] | 0.713 [0.690, 0.736] | 20.0 | - | - | - | - | +| BM25 | 58.7 [57.0, 60.3] | 23.5 [21.0, 26.1] | 0.733 [0.710, 0.755] | 20.0 | -2.8 [-5.4, -0.1] | 0.047 (0.235) | -2.0 [-3.7, -0.3] | 0.024 (0.071) | +| Hybrid vector+BM25 (RRF) | 66.1 [64.5, 67.8] | 31.9 [29.0, 34.8] | 0.743 [0.722, 0.763] | 20.0 | +5.6 [+3.2, +7.9] | <0.001 (0.001) | +5.4 [+4.1, +6.8] | <0.001 (0.001) | +| Vector top-30 + cross-encoder | 62.7 [61.0, 64.5] | 29.2 [26.4, 32.0] | 0.805 [0.785, 0.825] | 20.0 | +2.9 [+1.6, +4.3] | <0.001 (0.001) | +2.0 [+1.3, +2.9] | <0.001 (0.001) | +| ISONGraph multi_hop (tuned) [graph chosen on dev: mention] | 64.7 [62.6, 66.6] | 37.7 [34.7, 40.7] | 0.712 [0.689, 0.735] | 19.8 | +11.4 [+8.8, +14.2] | <0.001 (0.001) | +4.0 [+2.4, +5.5] | <0.001 (0.001) | +| PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention] | 65.3 [63.4, 67.2] | 38.1 [35.1, 41.2] | 0.598 [0.575, 0.622] | 20.0 | +11.8 [+9.4, +14.3] | <0.001 (0.001) | +4.6 [+3.3, +6.0] | <0.001 (0.001) | +| ISONGraph multi_hop (library defaults) [graph chosen on dev: mention] | 55.4 [53.4, 57.4] | 26.4 [23.8, 29.1] | 0.706 [0.682, 0.729] | 13.4 | +0.1 [-2.9, +3.3] | 1.000 (1.000) | -5.3 [-7.2, -3.5] | <0.001 (0.001) | +| ISONGraph multi_hop (tuned) [graph: mention] | 64.7 [62.6, 66.6] | 37.7 [34.7, 40.7] | 0.712 [0.689, 0.735] | 19.8 | +11.4 [+8.8, +14.2] | <0.001 (0.001) | +4.0 [+2.4, +5.5] | <0.001 (0.001) | +| ISONGraph multi_hop (tuned) [graph: ctrl_mention] | 60.7 [58.9, 62.5] | 26.3 [23.6, 29.0] | 0.713 [0.690, 0.736] | 20.0 | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +0.0 [+0.0, +0.1] | 1.000 (1.000) | +| PPR re-rank (tuned, independent impl.) [graph: mention] | 65.3 [63.4, 67.2] | 38.1 [35.1, 41.2] | 0.598 [0.575, 0.622] | 20.0 | +11.8 [+9.4, +14.3] | <0.001 (0.001) | +4.6 [+3.3, +6.0] | <0.001 (0.001) | +| PPR re-rank (tuned, independent impl.) [graph: ctrl_mention] | 60.7 [58.8, 62.4] | 26.3 [23.6, 29.0] | 0.710 [0.687, 0.734] | 20.0 | +0.0 [+0.0, +0.0] | 1.000 (1.000) | -0.0 [-0.1, +0.0] | 1.000 (1.000) | +| ISONGraph multi_hop (library defaults) [graph: mention] | 55.4 [53.4, 57.4] | 26.4 [23.8, 29.1] | 0.706 [0.682, 0.729] | 13.4 | +0.1 [-2.9, +3.3] | 1.000 (1.000) | -5.3 [-7.2, -3.5] | <0.001 (0.001) | +| ISONGraph multi_hop (library defaults) [graph: ctrl_mention] | 45.4 [43.7, 47.2] | 11.8 [9.9, 13.9] | 0.702 [0.677, 0.726] | 13.7 | -14.5 [-16.7, -12.3] | <0.001 (0.001) | -15.3 [-16.7, -13.9] | <0.001 (0.001) | + +#### musique/minilm: ISONGraph multi_hop (tuned) minus PPR re-rank (tuned), same graph; all-supporting (pp [95% CI], paired p) + +| Graph | k=5 | k=10 | k=20 | +|---|---|---|---| +| mention | +0.1 [-1.6, +1.7] (p=1.000) | -1.5 [-3.3, +0.2] (p=0.120) | -0.4 [-1.9, +1.1] (p=0.679) | + +#### musique/minilm: breakdown by question type, k=10 -- All-supporting@10 % (Δ vs vector, pp [95% CI]) + +| Group | n | Vector top-k (plain) | Hybrid vector+BM25 (RRF) | Vector top-30 + cross-encoder | ISONGraph multi_hop (tuned) | PPR re-rank (tuned, independent impl.) | +|---|---|---|---|---|---|---| +| 2hop | 519 | 30.8 | 34.1 (+3.3 [+0.2, +6.4]) | 37.8 (+6.9 [+4.4, +9.6]) | 46.2 (+15.4 [+11.9, +19.1]) [mention] | 48.0 (+17.1 [+12.9, +21.2]) [mention] | +| 3hop1 | 213 | 4.7 | 5.6 (+0.9 [-2.8, +4.7]) | 11.7 (+7.0 [+3.3, +10.8]) | 9.4 (+4.7 [+1.4, +8.0]) [mention] | 11.3 (+6.6 [+2.3, +10.8]) [mention] | +| 3hop2 | 78 | 12.8 | 16.7 (+3.8 [-5.1, +11.5]) | 16.7 (+3.8 [-3.8, +11.5]) | 14.1 (+1.3 [-2.6, +6.4]) [mention] | 14.1 (+1.3 [-3.8, +6.4]) [mention] | +| 4hop1 | 113 | 0.9 | 0.0 (-0.9 [-2.7, +0.0]) | 1.8 (+0.9 [-1.8, +3.5]) | 1.8 (+0.9 [+0.0, +2.7]) [mention] | 3.5 (+2.7 [+0.0, +6.2]) [mention] | +| 4hop2 | 30 | 0.0 | 0.0 (+0.0 [+0.0, +0.0]) | 0.0 (+0.0 [+0.0, +0.0]) | 0.0 (+0.0 [+0.0, +0.0]) [mention] | 0.0 (+0.0 [+0.0, +0.0]) [mention] | +| 4hop3 | 47 | 2.1 | 4.3 (+2.1 [-4.3, +10.6]) | 2.1 (+0.0 [+0.0, +0.0]) | 2.1 (+0.0 [+0.0, +0.0]) [mention] | 2.1 (+0.0 [+0.0, +0.0]) [mention] | + +#### musique/minilm: breakdown by hop count (number of supporting passages), k=10 -- All-supporting@10 % (Δ vs vector, pp [95% CI]) + +| Group | n | Vector top-k (plain) | Hybrid vector+BM25 (RRF) | Vector top-30 + cross-encoder | ISONGraph multi_hop (tuned) | PPR re-rank (tuned, independent impl.) | +|---|---|---|---|---|---|---| +| 2 | 519 | 30.8 | 34.1 (+3.3 [+0.2, +6.4]) | 37.8 (+6.9 [+4.4, +9.6]) | 46.2 (+15.4 [+11.9, +19.1]) [mention] | 48.0 (+17.1 [+12.9, +21.2]) [mention] | +| 3 | 291 | 6.9 | 8.6 (+1.7 [-1.7, +5.2]) | 13.1 (+6.2 [+2.7, +9.6]) | 10.7 (+3.8 [+1.4, +6.5]) [mention] | 12.0 (+5.2 [+1.7, +8.6]) [mention] | +| 4 | 190 | 1.1 | 1.1 (+0.0 [-2.1, +2.1]) | 1.6 (+0.5 [-1.1, +2.1]) | 1.6 (+0.5 [+0.0, +1.6]) [mention] | 2.6 (+1.6 [+0.0, +3.7]) [mention] | + +#### musique/minilm: dev vs test All-supporting@10 for tuned configs (selection optimism) + +| Method | Graph | dev % | test % | +|---|---|---|---| +| lib_mh | mention | 31.6 | 27.4 | +| lib_mh | ctrl_mention | 20.6 | 18.2 | +| ppr | mention | 31.6 | 28.9 | +| ppr | ctrl_mention | 21.0 | 18.2 | +| vector | - | 20.6 | 18.2 | + +## musique / bge (corpus 15405 passages, 1000 test questions) + + +### k = 5 (returned-passage budget = 5) + +| Method | Recall@k % [95% CI] | All-supporting@k % [95% CI] | MRR@k [95% CI] | mean returned | Δ all vs vector (pp) [95% CI] | p (Holm) | Δ recall vs vector (pp) [95% CI] | p (Holm) | +|---|---|---|---|---|---|---|---|---| +| Vector top-k (plain) | 48.3 [46.6, 50.0] | 14.3 [12.2, 16.4] | 0.749 [0.727, 0.770] | 5.0 | - | - | - | - | +| BM25 | 44.9 [43.3, 46.6] | 11.1 [9.3, 12.9] | 0.722 [0.698, 0.745] | 5.0 | -3.2 [-5.2, -1.2] | 0.003 (0.015) | -3.4 [-5.0, -1.7] | <0.001 (0.001) | +| Hybrid vector+BM25 (RRF) | 50.2 [48.5, 51.8] | 14.9 [12.8, 17.2] | 0.780 [0.759, 0.800] | 5.0 | +0.6 [-1.2, +2.3] | 0.585 (1.000) | +1.9 [+0.5, +3.2] | 0.005 (0.020) | +| Vector top-30 + cross-encoder | 53.5 [51.8, 55.2] | 18.9 [16.6, 21.2] | 0.817 [0.797, 0.837] | 5.0 | +4.6 [+2.8, +6.5] | <0.001 (0.001) | +5.2 [+3.8, +6.5] | <0.001 (0.001) | +| ISONGraph multi_hop (tuned) [graph chosen on dev: mention] | 52.6 [50.7, 54.5] | 21.6 [19.1, 24.2] | 0.749 [0.727, 0.770] | 5.0 | +7.3 [+5.6, +9.1] | <0.001 (0.001) | +4.3 [+3.4, +5.3] | <0.001 (0.001) | +| PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention] | 51.7 [49.9, 53.6] | 20.2 [17.7, 22.6] | 0.625 [0.603, 0.646] | 5.0 | +5.9 [+4.1, +7.8] | <0.001 (0.001) | +3.4 [+2.3, +4.4] | <0.001 (0.001) | +| ISONGraph multi_hop (library defaults) [graph chosen on dev: mention] | 49.8 [48.1, 51.7] | 17.3 [15.1, 19.7] | 0.749 [0.727, 0.770] | 5.0 | +3.0 [+2.0, +4.1] | <0.001 (0.001) | +1.5 [+1.1, +2.1] | <0.001 (0.001) | +| ISONGraph multi_hop (tuned) [graph: mention] | 52.6 [50.7, 54.5] | 21.6 [19.1, 24.2] | 0.749 [0.727, 0.770] | 5.0 | +7.3 [+5.6, +9.1] | <0.001 (0.001) | +4.3 [+3.4, +5.3] | <0.001 (0.001) | +| ISONGraph multi_hop (tuned) [graph: ctrl_mention] | 48.3 [46.6, 50.0] | 14.3 [12.2, 16.4] | 0.749 [0.727, 0.770] | 5.0 | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +| PPR re-rank (tuned, independent impl.) [graph: mention] | 51.7 [49.9, 53.6] | 20.2 [17.7, 22.6] | 0.625 [0.603, 0.646] | 5.0 | +5.9 [+4.1, +7.8] | <0.001 (0.001) | +3.4 [+2.3, +4.4] | <0.001 (0.001) | +| PPR re-rank (tuned, independent impl.) [graph: ctrl_mention] | 47.6 [45.8, 49.3] | 14.0 [11.9, 16.2] | 0.709 [0.685, 0.731] | 5.0 | -0.3 [-1.3, +0.7] | 0.690 (1.000) | -0.7 [-1.4, -0.1] | 0.031 (0.092) | +| ISONGraph multi_hop (library defaults) [graph: mention] | 49.8 [48.1, 51.7] | 17.3 [15.1, 19.7] | 0.749 [0.727, 0.770] | 5.0 | +3.0 [+2.0, +4.1] | <0.001 (0.001) | +1.5 [+1.1, +2.1] | <0.001 (0.001) | +| ISONGraph multi_hop (library defaults) [graph: ctrl_mention] | 48.2 [46.6, 49.9] | 14.2 [12.1, 16.3] | 0.749 [0.727, 0.770] | 5.0 | -0.1 [-0.3, +0.0] | 1.000 (1.000) | -0.1 [-0.2, +0.0] | 1.000 (1.000) | + +### k = 10 (returned-passage budget = 10) + +| Method | Recall@k % [95% CI] | All-supporting@k % [95% CI] | MRR@k [95% CI] | mean returned | Δ all vs vector (pp) [95% CI] | p (Holm) | Δ recall vs vector (pp) [95% CI] | p (Holm) | +|---|---|---|---|---|---|---|---|---| +| Vector top-k (plain) | 55.7 [54.0, 57.4] | 20.9 [18.4, 23.4] | 0.756 [0.735, 0.777] | 10.0 | - | - | - | - | +| BM25 | 51.5 [49.8, 53.1] | 15.2 [13.0, 17.3] | 0.730 [0.707, 0.752] | 10.0 | -5.7 [-8.1, -3.4] | <0.001 (0.001) | -4.2 [-6.0, -2.6] | <0.001 (0.001) | +| Hybrid vector+BM25 (RRF) | 58.5 [56.8, 60.2] | 23.3 [20.8, 26.0] | 0.786 [0.765, 0.805] | 10.0 | +2.4 [+0.2, +4.7] | 0.040 (0.120) | +2.8 [+1.2, +4.2] | <0.001 (0.001) | +| Vector top-30 + cross-encoder | 59.6 [57.9, 61.2] | 24.6 [22.1, 27.1] | 0.821 [0.802, 0.841] | 10.0 | +3.7 [+1.9, +5.6] | <0.001 (0.001) | +3.8 [+2.7, +5.0] | <0.001 (0.001) | +| ISONGraph multi_hop (tuned) [graph chosen on dev: mention] | 60.8 [59.0, 62.6] | 29.6 [26.8, 32.4] | 0.756 [0.734, 0.776] | 10.0 | +8.7 [+6.6, +10.8] | <0.001 (0.001) | +5.1 [+4.0, +6.2] | <0.001 (0.001) | +| PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention] | 60.2 [58.3, 62.1] | 29.1 [26.4, 31.9] | 0.690 [0.668, 0.713] | 10.0 | +8.2 [+5.9, +10.5] | <0.001 (0.001) | +4.5 [+3.1, +5.9] | <0.001 (0.001) | +| ISONGraph multi_hop (library defaults) [graph chosen on dev: mention] | 57.2 [55.2, 59.1] | 26.3 [23.6, 29.0] | 0.752 [0.730, 0.773] | 9.0 | +5.4 [+3.0, +8.0] | <0.001 (0.001) | +1.5 [-0.1, +3.0] | 0.046 (0.182) | +| ISONGraph multi_hop (tuned) [graph: mention] | 60.8 [59.0, 62.6] | 29.6 [26.8, 32.4] | 0.756 [0.734, 0.776] | 10.0 | +8.7 [+6.6, +10.8] | <0.001 (0.001) | +5.1 [+4.0, +6.2] | <0.001 (0.001) | +| ISONGraph multi_hop (tuned) [graph: ctrl_mention] | 55.7 [54.0, 57.4] | 20.9 [18.4, 23.4] | 0.756 [0.735, 0.777] | 10.0 | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +| PPR re-rank (tuned, independent impl.) [graph: mention] | 60.2 [58.3, 62.1] | 29.1 [26.4, 31.9] | 0.690 [0.668, 0.713] | 10.0 | +8.2 [+5.9, +10.5] | <0.001 (0.001) | +4.5 [+3.1, +5.9] | <0.001 (0.001) | +| PPR re-rank (tuned, independent impl.) [graph: ctrl_mention] | 55.7 [54.0, 57.4] | 20.8 [18.3, 23.3] | 0.759 [0.737, 0.779] | 10.0 | -0.1 [-0.3, +0.0] | 1.000 (1.000) | -0.0 [-0.1, +0.0] | 1.000 (1.000) | +| ISONGraph multi_hop (library defaults) [graph: mention] | 57.2 [55.2, 59.1] | 26.3 [23.6, 29.0] | 0.752 [0.730, 0.773] | 9.0 | +5.4 [+3.0, +8.0] | <0.001 (0.001) | +1.5 [-0.1, +3.0] | 0.046 (0.182) | +| ISONGraph multi_hop (library defaults) [graph: ctrl_mention] | 48.2 [46.6, 49.9] | 14.2 [12.1, 16.3] | 0.749 [0.727, 0.770] | 9.0 | -6.7 [-8.3, -5.1] | <0.001 (0.001) | -7.5 [-8.5, -6.5] | <0.001 (0.001) | + +#### semantic_subgraph(connect=True), top_k_results=10: set quality vs plain vector top-|S| (size-matched per query) + +| Graph | connect_max_hops | mean |S| | Recall % | All-supp. % | Vector@|S| recall % | Vector@|S| all % | Δ all (pp) [CI] | p | Δ recall (pp) [CI] | p | +|---|---|---|---|---|---|---|---|---|---|---| +| mention | 2 | 10.4 | 61.2 [59.4, 63.0] | 30.0 [27.2, 32.7] | 56.0 [54.3, 57.7] | 21.2 [18.7, 23.7] | +8.8 [+6.7, +11.0] | <0.001 | +5.2 [+4.0, +6.3] | <0.001 | +| mention | 3 | 10.9 | 61.4 [59.6, 63.2] | 30.3 [27.5, 33.1] | 56.6 [54.9, 58.3] | 21.8 [19.2, 24.4] | +8.5 [+6.3, +10.8] | <0.001 | +4.8 [+3.6, +6.0] | <0.001 | +| ctrl_mention | 2 | 10.0 | 55.7 [54.0, 57.4] | 20.9 [18.4, 23.4] | 55.7 [54.0, 57.4] | 20.9 [18.4, 23.4] | +0.0 [+0.0, +0.0] | 1.000 | +0.0 [+0.0, +0.0] | <0.001 | +| ctrl_mention | 3 | 10.0 | 55.7 [54.0, 57.4] | 20.9 [18.4, 23.4] | 55.7 [54.0, 57.4] | 20.9 [18.4, 23.4] | +0.0 [+0.0, +0.0] | 1.000 | +0.0 [+0.0, +0.0] | <0.001 | + +### k = 20 (returned-passage budget = 20) + +| Method | Recall@k % [95% CI] | All-supporting@k % [95% CI] | MRR@k [95% CI] | mean returned | Δ all vs vector (pp) [95% CI] | p (Holm) | Δ recall vs vector (pp) [95% CI] | p (Holm) | +|---|---|---|---|---|---|---|---|---| +| Vector top-k (plain) | 62.3 [60.6, 64.1] | 28.6 [25.9, 31.4] | 0.758 [0.737, 0.778] | 20.0 | - | - | - | - | +| BM25 | 58.7 [57.0, 60.3] | 23.5 [21.0, 26.1] | 0.733 [0.710, 0.755] | 20.0 | -5.1 [-7.8, -2.3] | <0.001 (0.001) | -3.6 [-5.4, -1.8] | <0.001 (0.001) | +| Hybrid vector+BM25 (RRF) | 66.7 [65.1, 68.4] | 32.9 [30.1, 35.8] | 0.789 [0.769, 0.808] | 20.0 | +4.3 [+1.8, +6.8] | 0.001 (0.006) | +4.4 [+3.0, +5.9] | <0.001 (0.001) | +| Vector top-30 + cross-encoder | 64.3 [62.7, 66.0] | 31.6 [28.8, 34.4] | 0.823 [0.803, 0.842] | 20.0 | +3.0 [+1.7, +4.4] | <0.001 (0.001) | +2.0 [+1.3, +2.8] | <0.001 (0.001) | +| ISONGraph multi_hop (tuned) [graph chosen on dev: mention] | 67.4 [65.6, 69.1] | 38.7 [35.7, 41.5] | 0.757 [0.736, 0.778] | 19.9 | +10.1 [+7.5, +12.7] | <0.001 (0.001) | +5.1 [+3.8, +6.5] | <0.001 (0.001) | +| PPR re-rank (tuned, independent impl.) [graph chosen on dev: mention] | 68.2 [66.4, 70.0] | 39.2 [36.2, 42.2] | 0.693 [0.671, 0.715] | 20.0 | +10.6 [+8.5, +12.9] | <0.001 (0.001) | +5.9 [+4.7, +7.1] | <0.001 (0.001) | +| ISONGraph multi_hop (library defaults) [graph chosen on dev: mention] | 59.0 [57.1, 61.0] | 28.8 [26.0, 31.6] | 0.752 [0.730, 0.773] | 14.2 | +0.2 [-2.6, +3.2] | 0.942 (1.000) | -3.3 [-5.0, -1.5] | <0.001 (0.002) | +| ISONGraph multi_hop (tuned) [graph: mention] | 67.4 [65.6, 69.1] | 38.7 [35.7, 41.5] | 0.757 [0.736, 0.778] | 19.9 | +10.1 [+7.5, +12.7] | <0.001 (0.001) | +5.1 [+3.8, +6.5] | <0.001 (0.001) | +| ISONGraph multi_hop (tuned) [graph: ctrl_mention] | 62.3 [60.6, 64.1] | 28.6 [25.9, 31.4] | 0.758 [0.737, 0.778] | 20.0 | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +| PPR re-rank (tuned, independent impl.) [graph: mention] | 68.2 [66.4, 70.0] | 39.2 [36.2, 42.2] | 0.693 [0.671, 0.715] | 20.0 | +10.6 [+8.5, +12.9] | <0.001 (0.001) | +5.9 [+4.7, +7.1] | <0.001 (0.001) | +| PPR re-rank (tuned, independent impl.) [graph: ctrl_mention] | 62.3 [60.6, 64.1] | 28.6 [25.9, 31.4] | 0.761 [0.739, 0.781] | 20.0 | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +0.0 [+0.0, +0.0] | 1.000 (1.000) | +| ISONGraph multi_hop (library defaults) [graph: mention] | 59.0 [57.1, 61.0] | 28.8 [26.0, 31.6] | 0.752 [0.730, 0.773] | 14.2 | +0.2 [-2.6, +3.2] | 0.942 (1.000) | -3.3 [-5.0, -1.5] | <0.001 (0.002) | +| ISONGraph multi_hop (library defaults) [graph: ctrl_mention] | 48.3 [46.6, 50.0] | 14.3 [12.2, 16.4] | 0.749 [0.727, 0.770] | 13.8 | -14.3 [-16.5, -12.2] | <0.001 (0.001) | -14.0 [-15.4, -12.7] | <0.001 (0.001) | + +#### musique/bge: ISONGraph multi_hop (tuned) minus PPR re-rank (tuned), same graph; all-supporting (pp [95% CI], paired p) + +| Graph | k=5 | k=10 | k=20 | +|---|---|---|---| +| mention | +1.4 [-0.1, +2.9] (p=0.091) | +0.5 [-1.0, +2.0] (p=0.624) | -0.5 [-2.0, +1.0] (p=0.611) | + +#### musique/bge: breakdown by question type, k=10 -- All-supporting@10 % (Δ vs vector, pp [95% CI]) + +| Group | n | Vector top-k (plain) | Hybrid vector+BM25 (RRF) | Vector top-30 + cross-encoder | ISONGraph multi_hop (tuned) | PPR re-rank (tuned, independent impl.) | +|---|---|---|---|---|---|---| +| 2hop | 519 | 33.7 | 37.2 (+3.5 [+0.0, +6.9]) | 39.7 (+6.0 [+2.7, +9.1]) | 48.6 (+14.8 [+11.4, +18.5]) [mention] | 49.1 (+15.4 [+11.6, +19.3]) [mention] | +| 3hop1 | 213 | 10.8 | 13.6 (+2.8 [-2.3, +8.0]) | 12.2 (+1.4 [-2.3, +5.2]) | 15.0 (+4.2 [+0.0, +8.5]) [mention] | 13.6 (+2.8 [-0.9, +6.6]) [mention] | +| 3hop2 | 78 | 14.1 | 12.8 (-1.3 [-6.4, +2.6]) | 16.7 (+2.6 [-2.6, +7.7]) | 15.4 (+1.3 [-2.6, +6.4]) [mention] | 9.0 (-5.1 [-11.5, +0.0]) [mention] | +| 4hop1 | 113 | 0.0 | 0.9 (+0.9 [+0.0, +2.7]) | 0.9 (+0.9 [+0.0, +2.7]) | 0.0 (+0.0 [+0.0, +0.0]) [mention] | 0.0 (+0.0 [+0.0, +0.0]) [mention] | +| 4hop2 | 30 | 0.0 | 0.0 (+0.0 [+0.0, +0.0]) | 0.0 (+0.0 [+0.0, +0.0]) | 0.0 (+0.0 [+0.0, +0.0]) [mention] | 0.0 (+0.0 [+0.0, +0.0]) [mention] | +| 4hop3 | 47 | 0.0 | 0.0 (+0.0 [+0.0, +0.0]) | 0.0 (+0.0 [+0.0, +0.0]) | 0.0 (+0.0 [+0.0, +0.0]) [mention] | 0.0 (+0.0 [+0.0, +0.0]) [mention] | + +#### musique/bge: breakdown by hop count (number of supporting passages), k=10 -- All-supporting@10 % (Δ vs vector, pp [95% CI]) + +| Group | n | Vector top-k (plain) | Hybrid vector+BM25 (RRF) | Vector top-30 + cross-encoder | ISONGraph multi_hop (tuned) | PPR re-rank (tuned, independent impl.) | +|---|---|---|---|---|---|---| +| 2 | 519 | 33.7 | 37.2 (+3.5 [+0.0, +6.9]) | 39.7 (+6.0 [+2.7, +9.1]) | 48.6 (+14.8 [+11.4, +18.5]) [mention] | 49.1 (+15.4 [+11.6, +19.3]) [mention] | +| 3 | 291 | 11.7 | 13.4 (+1.7 [-2.4, +5.9]) | 13.4 (+1.7 [-1.0, +4.8]) | 15.1 (+3.4 [+0.3, +6.5]) [mention] | 12.4 (+0.7 [-2.7, +4.1]) [mention] | +| 4 | 190 | 0.0 | 0.5 (+0.5 [+0.0, +1.6]) | 0.5 (+0.5 [+0.0, +1.6]) | 0.0 (+0.0 [+0.0, +0.0]) [mention] | 0.0 (+0.0 [+0.0, +0.0]) [mention] | + +#### musique/bge: dev vs test All-supporting@10 for tuned configs (selection optimism) + +| Method | Graph | dev % | test % | +|---|---|---|---| +| lib_mh | mention | 30.8 | 29.6 | +| lib_mh | ctrl_mention | 21.0 | 20.9 | +| ppr | mention | 29.6 | 29.1 | +| ppr | ctrl_mention | 21.0 | 20.8 | +| vector | - | 21.0 | 20.9 | + +## Graph diagnostics (gold used for REPORTING only) + +| Dataset | model | Graph | undirected edges | mean degree | isolated % | gold pairs directly linked % | gold pairs linked within 2 hops % | nbrs of gold inside same question context % | +|---|---|---|---|---|---|---|---|---| +| hotpotqa | minilm | mention | 11348 | 1.6 | 40 | 65 | 67 | 59 | +| hotpotqa | minilm | entity | 59717 | 8.2 | 14 | 50 | 54 | 29 | +| hotpotqa | minilm | knn | 52527 | 7.2 | 0 | 41 | 51 | 22 | +| hotpotqa | minilm | ctrl_mention | 11348 | 1.6 | 40 | 0 | 0 | 0 | +| hotpotqa | bge | mention | 11348 | 1.6 | 40 | 65 | 67 | 59 | +| hotpotqa | bge | knn | 54857 | 7.5 | 0 | 54 | 67 | 24 | +| hotpotqa | bge | ctrl_mention | 11348 | 1.6 | 40 | 0 | 0 | 0 | +| 2wiki | minilm | mention | 5383 | 1.2 | 42 | 44 | 44 | 42 | +| 2wiki | minilm | entity | 30240 | 6.5 | 22 | 41 | 43 | 11 | +| 2wiki | minilm | knn | 34360 | 7.4 | 0 | 18 | 23 | 13 | +| 2wiki | minilm | mention+entity | 31907 | 6.9 | 18 | 46 | 47 | 13 | +| 2wiki | minilm | ctrl_mention | 5383 | 1.2 | 42 | 0 | 0 | 0 | +| 2wiki | minilm | ctrl_entity | 30240 | 6.5 | 22 | 0 | 1 | 0 | +| 2wiki | bge | mention | 5383 | 1.2 | 42 | 44 | 44 | 42 | +| 2wiki | bge | entity | 30240 | 6.5 | 22 | 41 | 43 | 11 | +| 2wiki | bge | knn | 34499 | 7.5 | 0 | 32 | 37 | 13 | +| 2wiki | bge | ctrl_mention | 5383 | 1.2 | 42 | 0 | 0 | 0 | +| 2wiki | bge | ctrl_entity | 30240 | 6.5 | 22 | 0 | 1 | 0 | +| musique | minilm | mention | 20272 | 2.6 | 53 | 15 | 23 | 15 | +| musique | minilm | ctrl_mention | 20272 | 2.6 | 53 | 0 | 0 | 0 | +| musique | bge | mention | 20272 | 2.6 | 53 | 15 | 23 | 15 | +| musique | bge | ctrl_mention | 20272 | 2.6 | 53 | 0 | 0 | 0 | + +## Selected hyper-parameters (dev), ISONGraph multi_hop, k=10 + +- hotpotqa/minilm/mention: lib `{'hops': 2, 'seeds': 5, 'decay': 0.5, 'blend': 0.3, 'direction': 'both', 'weight_field': None}`; ppr `{'alpha': 0.5, 'seeds': 5, 'beta': 0.3, 'directed_mention': False, 'weighted': False}` +- hotpotqa/minilm/entity: lib `{'hops': 1, 'seeds': 20, 'decay': 0.9, 'blend': 0.3, 'direction': 'out', 'weight_field': None}`; ppr `{'alpha': 0.5, 'seeds': 5, 'beta': 0.3, 'directed_mention': True, 'weighted': True}` +- hotpotqa/minilm/knn: lib `{'hops': 2, 'seeds': 5, 'decay': 0.7, 'blend': 0.3, 'direction': 'out', 'weight_field': None}`; ppr `{'alpha': 0.5, 'seeds': 5, 'beta': 0.3, 'directed_mention': True, 'weighted': True}` +- hotpotqa/minilm/ctrl_mention: lib `{'hops': 2, 'seeds': 15, 'decay': 0.3, 'blend': 0.6, 'direction': 'out', 'weight_field': None}`; ppr `{'alpha': 0.5, 'seeds': 20, 'beta': 0.85, 'directed_mention': True, 'weighted': False}` +- hotpotqa/bge/mention: lib `{'hops': 2, 'seeds': 5, 'decay': 0.7, 'blend': 0.0, 'direction': 'both', 'weight_field': None}`; ppr `{'alpha': 0.5, 'seeds': 5, 'beta': 0.3, 'directed_mention': False, 'weighted': False}` +- hotpotqa/bge/knn: lib `{'hops': 2, 'seeds': 5, 'decay': 0.7, 'blend': 0.3, 'direction': 'out', 'weight_field': None}`; ppr `{'alpha': 0.5, 'seeds': 5, 'beta': 0.3, 'directed_mention': True, 'weighted': True}` +- hotpotqa/bge/ctrl_mention: lib `{'hops': 2, 'seeds': 10, 'decay': 0.7, 'blend': 0.0, 'direction': 'out', 'weight_field': None}`; ppr `{'alpha': 0.5, 'seeds': 20, 'beta': 0.6, 'directed_mention': True, 'weighted': False}` +- 2wiki/minilm/mention: lib `{'hops': 3, 'seeds': 5, 'decay': 0.7, 'blend': 0.0, 'direction': 'out', 'weight_field': None}`; ppr `{'alpha': 0.5, 'seeds': 5, 'beta': 0.0, 'directed_mention': True, 'weighted': False}` +- 2wiki/minilm/entity: lib `{'hops': 1, 'seeds': 5, 'decay': 0.9, 'blend': 0.3, 'direction': 'out', 'weight_field': 'w'}`; ppr `{'alpha': 0.5, 'seeds': 5, 'beta': 0.3, 'directed_mention': True, 'weighted': True}` +- 2wiki/minilm/knn: lib `{'hops': 2, 'seeds': 5, 'decay': 0.8, 'blend': 0.0, 'direction': 'out', 'weight_field': None}`; ppr `{'alpha': 0.5, 'seeds': 5, 'beta': 0.0, 'directed_mention': True, 'weighted': True}` +- 2wiki/minilm/mention+entity: lib `{'hops': 2, 'seeds': 5, 'decay': 0.7, 'blend': 0.0, 'direction': 'out', 'weight_field': 'w'}`; ppr `{'alpha': 0.5, 'seeds': 5, 'beta': 0.0, 'directed_mention': True, 'weighted': True}` +- 2wiki/minilm/ctrl_mention: lib `{'hops': 2, 'seeds': 15, 'decay': 0.3, 'blend': 0.6, 'direction': 'out', 'weight_field': None}`; ppr `{'alpha': 0.5, 'seeds': 20, 'beta': 0.85, 'directed_mention': True, 'weighted': False}` +- 2wiki/minilm/ctrl_entity: lib `{'hops': 1, 'seeds': 20, 'decay': 0.3, 'blend': 0.3, 'direction': 'out', 'weight_field': None}`; ppr `{'alpha': 0.5, 'seeds': 20, 'beta': 0.3, 'directed_mention': True, 'weighted': False}` +- 2wiki/bge/mention: lib `{'hops': 1, 'seeds': 5, 'decay': 0.9, 'blend': 0.0, 'direction': 'out', 'weight_field': None}`; ppr `{'alpha': 0.5, 'seeds': 5, 'beta': 0.3, 'directed_mention': True, 'weighted': False}` +- 2wiki/bge/entity: lib `{'hops': 2, 'seeds': 5, 'decay': 0.7, 'blend': 0.3, 'direction': 'out', 'weight_field': None}`; ppr `{'alpha': 0.5, 'seeds': 5, 'beta': 0.3, 'directed_mention': True, 'weighted': True}` +- 2wiki/bge/knn: lib `{'hops': 1, 'seeds': 5, 'decay': 0.5, 'blend': 0.0, 'direction': 'out', 'weight_field': 'w'}`; ppr `{'alpha': 0.5, 'seeds': 5, 'beta': 0.0, 'directed_mention': True, 'weighted': True}` +- 2wiki/bge/ctrl_mention: lib `{'hops': 2, 'seeds': 10, 'decay': 0.7, 'blend': 0.0, 'direction': 'out', 'weight_field': None}`; ppr `{'alpha': 0.5, 'seeds': 5, 'beta': 0.85, 'directed_mention': True, 'weighted': False}` +- 2wiki/bge/ctrl_entity: lib `{'hops': 1, 'seeds': 20, 'decay': 0.3, 'blend': 0.3, 'direction': 'out', 'weight_field': None}`; ppr `{'alpha': 0.5, 'seeds': 5, 'beta': 0.85, 'directed_mention': True, 'weighted': False}` +- musique/minilm/mention: lib `{'hops': 2, 'seeds': 10, 'decay': 0.9, 'blend': 0.3, 'direction': 'both', 'weight_field': None}`; ppr `{'alpha': 0.7, 'seeds': 5, 'beta': 0.3, 'directed_mention': False, 'weighted': False}` +- musique/minilm/ctrl_mention: lib `{'hops': 2, 'seeds': 15, 'decay': 0.3, 'blend': 0.6, 'direction': 'out', 'weight_field': None}`; ppr `{'alpha': 0.5, 'seeds': 20, 'beta': 0.6, 'directed_mention': True, 'weighted': False}` +- musique/bge/mention: lib `{'hops': 3, 'seeds': 20, 'decay': 0.9, 'blend': 0.0, 'direction': 'both', 'weight_field': None}`; ppr `{'alpha': 0.7, 'seeds': 5, 'beta': 0.6, 'directed_mention': False, 'weighted': False}` +- musique/bge/ctrl_mention: lib `{'hops': 2, 'seeds': 10, 'decay': 0.7, 'blend': 0.0, 'direction': 'out', 'weight_field': None}`; ppr `{'alpha': 0.5, 'seeds': 5, 'beta': 0.85, 'directed_mention': True, 'weighted': False}` \ No newline at end of file diff --git a/benchmarks/multihop/results/threshold_check.json b/benchmarks/multihop/results/threshold_check.json new file mode 100644 index 0000000..da1ffbf --- /dev/null +++ b/benchmarks/multihop/results/threshold_check.json @@ -0,0 +1,98 @@ +{ + "hotpotqa/minilm": { + "frac_test_queries_top1_cos_below_0.3": 0.0, + "mean_top1_cos": 0.6355993747711182, + "threshold=0.3": { + "all@10": 0.772, + "recall@10": 0.8575, + "mean_returned": 7.899, + "frac_empty": 0.0 + }, + "threshold=-1.0": { + "all@10": 0.773, + "recall@10": 0.858, + "mean_returned": 7.917, + "frac_empty": 0.0 + } + }, + "hotpotqa/bge": { + "frac_test_queries_top1_cos_below_0.3": 0.0, + "mean_top1_cos": 0.744337260723114, + "threshold=0.3": { + "all@10": 0.875, + "recall@10": 0.9265, + "mean_returned": 7.844, + "frac_empty": 0.0 + }, + "threshold=-1.0": { + "all@10": 0.875, + "recall@10": 0.9265, + "mean_returned": 7.844, + "frac_empty": 0.0 + } + }, + "2wiki/minilm": { + "frac_test_queries_top1_cos_below_0.3": 0.0, + "mean_top1_cos": 0.6492449045181274, + "threshold=0.3": { + "all@10": 0.795, + "recall@10": 0.89025, + "mean_returned": 7.865, + "frac_empty": 0.0 + }, + "threshold=-1.0": { + "all@10": 0.795, + "recall@10": 0.89025, + "mean_returned": 7.867, + "frac_empty": 0.0 + } + }, + "2wiki/bge": { + "frac_test_queries_top1_cos_below_0.3": 0.0, + "mean_top1_cos": 0.7257882356643677, + "threshold=0.3": { + "all@10": 0.946, + "recall@10": 0.974, + "mean_returned": 8.577, + "frac_empty": 0.0 + }, + "threshold=-1.0": { + "all@10": 0.946, + "recall@10": 0.974, + "mean_returned": 8.577, + "frac_empty": 0.0 + } + }, + "musique/minilm": { + "frac_test_queries_top1_cos_below_0.3": 0.0, + "mean_top1_cos": 0.5775226950645447, + "threshold=0.3": { + "all@10": 0.245, + "recall@10": 0.5356666666666666, + "mean_returned": 8.722, + "frac_empty": 0.0 + }, + "threshold=-1.0": { + "all@10": 0.245, + "recall@10": 0.5356666666666666, + "mean_returned": 8.722, + "frac_empty": 0.0 + } + }, + "musique/bge": { + "frac_test_queries_top1_cos_below_0.3": 0.0, + "mean_top1_cos": 0.6993334889411926, + "threshold=0.3": { + "all@10": 0.263, + "recall@10": 0.5716666666666667, + "mean_returned": 9.007, + "frac_empty": 0.0 + }, + "threshold=-1.0": { + "all@10": 0.263, + "recall@10": 0.5716666666666667, + "mean_returned": 9.007, + "frac_empty": 0.0 + } + } +} \ No newline at end of file diff --git a/benchmarks/multihop/run.py b/benchmarks/multihop/run.py new file mode 100644 index 0000000..5ff4d16 --- /dev/null +++ b/benchmarks/multihop/run.py @@ -0,0 +1,102 @@ +#!/usr/bin/env python3 +"""Entry point for the multi-hop retrieval benchmark. + + python run.py prepare # download pinned dev sets, sample questions, build pooled corpora + python run.py embed # embed passages + questions (MiniLM, bge-small), cache .npy + python run.py edges # build and cache graph edge lists (mention / entity / kNN) + python run.py baselines # vector, BM25, RRF hybrid, cross-encoder rerank (GPU for rerank) + python run.py graphs [--workers N] [--datasets ..] [--models ..] [--variants ..] + # tune on dev, evaluate on test: ISONGraph multi_hop/subgraph + PPR + python run.py latency # single-thread per-query latency (run on an otherwise idle machine) + python run.py report # bootstrap CIs, paired tests, tables -> results/*.json, results/tables.md + python run.py all # everything above in order + +Run with the benchmark venv: D:\\Goldmine\\ISON\\bench_env_mh\\Scripts\\python.exe run.py <command> +(a venv created with --system-site-packages: torch, sentence-transformers, datasets, scipy, numpy, +ison-graph; the library under test is imported from this checkout, unmodified). +""" +from __future__ import annotations + +import os +import sys + +os.environ.setdefault("OMP_NUM_THREADS", "2") +os.environ.setdefault("MKL_NUM_THREADS", "2") +os.environ.setdefault("OPENBLAS_NUM_THREADS", "2") + +import argparse +import time +from concurrent.futures import ProcessPoolExecutor, as_completed + +from common import DATASETS, MODELS # noqa: E402 + + +def _graph_task(args): + import jobs + return jobs.graph_job(*args) + + +def _baseline_task(args): + import jobs + return jobs.baselines_job(*args) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("cmd", choices=["prepare", "embed", "edges", "baselines", "graphs", "latency", "report", "all"]) + ap.add_argument("--workers", type=int, default=12) + ap.add_argument("--datasets", nargs="*", default=DATASETS) + ap.add_argument("--models", nargs="*", default=list(MODELS)) + ap.add_argument("--variants", nargs="*", default=None) + ap.add_argument("--jobs", nargs="*", default=None, help="explicit ds:model:variant triples") + a = ap.parse_args() + cmds = ["prepare", "embed", "edges", "baselines", "graphs", "latency", "report"] if a.cmd == "all" else [a.cmd] + for cmd in cmds: + t0 = time.time() + if cmd == "prepare": + import prepare_data + for ds in a.datasets: + prepare_data.build(ds) + elif cmd == "embed": + import embed + for ds in a.datasets: + for m in a.models: + embed.embed(ds, m) + elif cmd == "edges": + import jobs + from jobs import load_ds, get_edges + for ds in a.datasets: + for m in a.models: + corpus, qs, P, Q, dev, test = load_ds(ds, m) + e = get_edges(ds, m, corpus, P) + print(ds, m, {k: len(v[0]) for k, v in e.items()}) + elif cmd == "baselines": + tasks = [(ds, m) for ds in a.datasets for m in a.models] + # reranking uses the GPU: run sequentially to avoid GPU contention + import jobs + for t in tasks: + print(jobs.baselines_job(*t), flush=True) + elif cmd == "graphs": + import graphs + variants = a.variants or list(graphs.VARIANTS) + tasks = [(ds, m, v) for ds in a.datasets for m in a.models for v in variants] + if a.jobs: + tasks = [tuple(x.split(":")) for x in a.jobs] + with ProcessPoolExecutor(max_workers=a.workers) as ex: + futs = {ex.submit(_graph_task, t): t for t in tasks} + for f in as_completed(futs): + try: + print(f.result(), flush=True) + except Exception as e: # keep going; report at the end + print("FAILED", futs[f], repr(e), flush=True) + elif cmd == "latency": + import latency + latency.run() + elif cmd == "report": + import report + report.run() + print(f"[{cmd}] {time.time()-t0:.0f}s", flush=True) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/multihop/threshold_check.py b/benchmarks/multihop/threshold_check.py new file mode 100644 index 0000000..b5b8ea4 --- /dev/null +++ b/benchmarks/multihop/threshold_check.py @@ -0,0 +1,34 @@ +"""How much does the library's default seed threshold (0.3) matter on real embeddings? +semantic_multi_hop(threshold=0.3) drops any seed whose cosine to the query is < 0.3. +Runs the library-default configuration (max_hops=2, top_k_seeds=5, decay=0.8, top_k_results=10) +on the `mention` graph with threshold=0.3 (the true default) vs -1.0 (used everywhere else).""" +import json +import numpy as np +import graphs, methods +from common import RESULTS_DIR +from jobs import get_edges, load_ds +from ison_graph import Direction + +out = {} +for ds in ("hotpotqa", "2wiki", "musique"): + for model in ("minilm", "bge"): + corpus, qs, P, Q, dev, test = load_ds(ds, model) + edges = get_edges(ds, model, corpus, P, need_knn=False) + g = methods.build_lib_graph(corpus, P, edges, ("mention",)) + S = Q @ P.T + top1 = S.max(1)[test] + res = {"frac_test_queries_top1_cos_below_0.3": float((top1 < 0.3).mean()), + "mean_top1_cos": float(top1.mean())} + for thr in (0.3, -1.0): + rk = [] + for i in test: + r = g.semantic_multi_hop(Q[i].tolist(), max_hops=2, top_k_seeds=5, top_k_results=10, + direction=Direction.OUT, decay=0.8, threshold=thr) + rk.append(np.array([x.node_ref[1] for x in r])) + m = methods.metrics_at(rk, [qs[i] for i in test], 10) + res[f"threshold={thr}"] = {"all@10": float(m["all"].mean()), "recall@10": float(m["recall"].mean()), + "mean_returned": float(m["n_ret"].mean()), + "frac_empty": float((m["n_ret"] == 0).mean())} + out[f"{ds}/{model}"] = res + print(ds, model, json.dumps(res), flush=True) +json.dump(out, open(RESULTS_DIR / "threshold_check.json", "w"), indent=1) diff --git a/benchmarks/rust-stage-a/RESULTS.md b/benchmarks/rust-stage-a/RESULTS.md new file mode 100644 index 0000000..fb2e38b --- /dev/null +++ b/benchmarks/rust-stage-a/RESULTS.md @@ -0,0 +1,253 @@ +# Rust stage A: concurrency, one copy, no write stall, deletes + +Same-session A/B on the i9-14900HX laptop (8 P + 16 E cores, 32 threads), `text384` / `glove` / `sift` +data from `D:\Goldmine\ISON\bench_data`, method as in `benchmarks/competitive/README_method.md` +(single-thread loops pinned to one P-core, three timed runs per point, ground truth float64 exact). +"old" = the commit before the change (`git archive HEAD`, built into its own target dir); "new" = the working tree. +Old and new ran back to back, one timed job at a time (`run_ab.sh`; raw JSON in `raw/`, tables by `summarize.py`). +The machine drifted by up to ~25% between hours and by ~10% between repeats, so read ratios, not absolute QPS, +and treat differences under ~10% as noise. text384 100k ran 3x per side, 1M 1x (competitive_bench) and 2x +(writes); glove/sift 100k 1x. + +Not in the tables: sidecar and graph file bytes are identical old vs new (SemanticGraph built from a shuffled +history with updates and deletes, saved, compared with `cmp`); the `get()` API change is described in the report. + +Reading notes: +* "IVF ... after the 30k writes" is `competitive_bench`'s mutation test (10k inserts + 10k updates + 10k deletes + on the built index at the nprobe that first reached recall 0.95). sift 100k picked nprobe 64 for old + (0.9860 pre-mutation) but 32 for new (0.9513), so its `recall after` rows compare different operating + points, not a recall loss; at equal nprobe the sweep tables are within 0.003. +* `exact batch` / `IVF batch` are new (old has no batch call); the old one-query loop with automatic threads is the comparison. +* `update` throughput is lower than before by design: the fold work that used to be deferred to the first search + after the writes is now paid by the writes (every 256th write folds 256 rows). + +### Rust stage A: before / after (same session) + +#### text384 n=100,000 (median over 3 old / 3 new run(s)) + +| metric | old | new | new/old | +|---|---|---|---| +| exact, 1 thread (pinned), QPS | 137.5 | 130.9 | 0.95x | +| exact, one-query loop with auto threads, QPS | 243.4 | 248.6 | 1.02x | +| 8 concurrent clients (1 thread each), exact QPS | 137.8 | 756.0 | 5.49x | +| exact batch, all threads, QPS | - | 10,376 | - | +| IVF build, s | 0.49 | 0.60 | 1.22x | +| IVF 1-thread QPS at recall 0.95 (interpolated) | 749 | 843 | 1.12x | +| IVF one-query loop, auto threads, QPS at ref nprobe | 1,396 | 1,598 | 1.15x | +| IVF batch, all threads, QPS at ref nprobe | - | 41,704 | - | +| ingest, s | 0.06 | 0.12 | 2.08x | +| working set after ingest, MB | 181 | 162 | 0.89x | +| working set after IVF build, MB | 342 | 168 | 0.49x | +| peak working set after IVF build, MB | 420 | 247 | 0.59x | +| insert 10k, per s | 224,393 | 242,247 | 1.08x | +| update 10k, per s | 469,598 | 265,214 | 0.56x | +| delete 10k, per s | 840,845 | 1,629,009 | 1.94x | +| first query after the 30k writes, ms | 224.92 | 1.48 | 0.01x | +| recall@10 after the writes | 0.9490 | 0.9518 | - | + +recall@10 / 1-thread QPS by nprobe (median of runs): + +| nprobe | old recall | new recall | old QPS | new QPS | +|---|---|---|---|---| +| 1 | 0.2730 | 0.2879 | 21,715 | 26,723 | +| 2 | 0.3918 | 0.4120 | 16,409 | 18,338 | +| 4 | 0.5168 | 0.5379 | 12,143 | 12,750 | +| 8 | 0.6402 | 0.6493 | 7,178 | 8,305 | +| 16 | 0.7450 | 0.7529 | 4,716 | 5,203 | +| 32 | 0.8346 | 0.8434 | 2,697 | 2,450 | +| 64 | 0.9071 | 0.9075 | 1,459 | 1,484 | +| 128 | 0.9529 | 0.9548 | 716 | 790 | +| 256 | 0.9807 | 0.9834 | 385 | 446 | + +#### text384 n=1,000,000 (median over 1 old / 1 new run(s)) + +| metric | old | new | new/old | +|---|---|---|---| +| exact, 1 thread (pinned), QPS | 15.2 | 14.5 | 0.95x | +| exact, one-query loop with auto threads, QPS | 31.8 | 29.3 | 0.92x | +| 8 concurrent clients (1 thread each), exact QPS | 14.8 | 60.6 | 4.08x | +| exact batch, all threads, QPS | - | 1,178 | - | +| IVF build, s | 6.16 | 7.49 | 1.22x | +| IVF 1-thread QPS at recall 0.95 (interpolated) | 103 | 133 | 1.28x | +| IVF one-query loop, auto threads, QPS at ref nprobe | 250 | 316 | 1.27x | +| IVF batch, all threads, QPS at ref nprobe | - | 11,911 | - | +| ingest, s | 0.82 | 1.15 | 1.40x | +| working set after ingest, MB | 1,932 | 1,622 | 0.84x | +| working set after IVF build, MB | 3,532 | 1,643 | 0.47x | +| peak working set after IVF build, MB | 3,779 | 1,888 | 0.50x | +| insert 10k, per s | 26,383 | 125,089 | 4.74x | +| update 10k, per s | 365,429 | 143,493 | 0.39x | +| delete 10k, per s | 564,927 | 969,039 | 1.72x | +| first query after the 30k writes, ms | 616.13 | 7.63 | 0.01x | +| recall@10 after the writes | 0.9521 | 0.9564 | - | + +recall@10 / 1-thread QPS by nprobe (median of runs): + +| nprobe | old recall | new recall | old QPS | new QPS | +|---|---|---|---|---| +| 1 | 0.2757 | 0.2709 | 5,244 | 4,942 | +| 2 | 0.3806 | 0.3805 | 4,302 | 4,742 | +| 4 | 0.4961 | 0.4949 | 2,797 | 3,312 | +| 8 | 0.6046 | 0.6065 | 1,827 | 2,320 | +| 16 | 0.7022 | 0.7056 | 1,058 | 1,420 | +| 32 | 0.7905 | 0.7865 | 612 | 809 | +| 64 | 0.8600 | 0.8582 | 352 | 436 | +| 128 | 0.9150 | 0.9158 | 187 | 240 | +| 256 | 0.9521 | 0.9526 | 100 | 127 | + +#### glove n=100,000 (median over 1 old / 1 new run(s)) + +| metric | old | new | new/old | +|---|---|---|---| +| exact, 1 thread (pinned), QPS | 664.1 | 579.7 | 0.87x | +| exact, one-query loop with auto threads, QPS | 1,612.0 | 1,913.9 | 1.19x | +| 8 concurrent clients (1 thread each), exact QPS | 671.3 | 3,051.9 | 4.55x | +| exact batch, all threads, QPS | - | 25,961 | - | +| IVF build, s | 0.22 | 0.18 | 0.84x | +| IVF 1-thread QPS at recall 0.95 (interpolated) | 1,857 | 2,251 | 1.21x | +| IVF one-query loop, auto threads, QPS at ref nprobe | 1,853 | 2,634 | 1.42x | +| IVF batch, all threads, QPS at ref nprobe | - | 52,421 | - | +| ingest, s | 0.06 | 0.05 | 0.91x | +| working set after ingest, MB | 72 | 53 | 0.74x | +| working set after IVF build, MB | 125 | 61 | 0.49x | +| peak working set after IVF build, MB | 144 | 80 | 0.56x | +| insert 10k, per s | 675,680 | 649,836 | 0.96x | +| update 10k, per s | 705,696 | 531,547 | 0.75x | +| delete 10k, per s | 1,030,121 | 2,199,107 | 2.13x | +| first query after the 30k writes, ms | 82.47 | 1.02 | 0.01x | +| recall@10 after the writes | 0.9743 | 0.9732 | - | + +recall@10 / 1-thread QPS by nprobe (median of runs): + +| nprobe | old recall | new recall | old QPS | new QPS | +|---|---|---|---|---| +| 1 | 0.2851 | 0.2832 | 52,731 | 65,848 | +| 2 | 0.4016 | 0.4071 | 40,514 | 51,856 | +| 4 | 0.5209 | 0.5229 | 30,934 | 39,881 | +| 8 | 0.6351 | 0.6361 | 20,568 | 27,274 | +| 16 | 0.7308 | 0.7323 | 13,275 | 16,357 | +| 32 | 0.8134 | 0.8163 | 7,466 | 9,201 | +| 64 | 0.8817 | 0.8825 | 4,323 | 5,297 | +| 128 | 0.9355 | 0.9361 | 2,277 | 2,851 | +| 256 | 0.9740 | 0.9743 | 1,325 | 1,490 | +| 1265 | 1.0000 | 1.0000 | 638 | 578 | + +#### sift n=100,000 (median over 1 old / 1 new run(s)) + +| metric | old | new | new/old | +|---|---|---|---| +| exact, 1 thread (pinned), QPS | 437.7 | 463.8 | 1.06x | +| exact, one-query loop with auto threads, QPS | 1,030.1 | 1,724.3 | 1.67x | +| 8 concurrent clients (1 thread each), exact QPS | 487.6 | 2,214.6 | 4.54x | +| exact batch, all threads, QPS | - | 29,394 | - | +| IVF build, s | 0.20 | 0.18 | 0.90x | +| IVF 1-thread QPS at recall 0.95 (interpolated) | 7,552 | 9,047 | 1.20x | +| IVF one-query loop, auto threads, QPS at ref nprobe | 4,604 | 9,019 | 1.96x | +| IVF batch, all threads, QPS at ref nprobe | - | 117,271 | - | +| ingest, s | 0.06 | 0.05 | 0.88x | +| working set after ingest, MB | 83 | 64 | 0.78x | +| working set after IVF build, MB | 146 | 71 | 0.49x | +| peak working set after IVF build, MB | 172 | 97 | 0.56x | +| insert 10k, per s | 558,403 | 623,756 | 1.12x | +| update 10k, per s | 647,040 | 532,984 | 0.82x | +| delete 10k, per s | 1,041,743 | 2,025,316 | 1.94x | +| first query after the 30k writes, ms | 75.33 | 0.19 | 0.00x | +| recall@10 after the writes | 0.9855 | 0.9480 | - | + +recall@10 / 1-thread QPS by nprobe (median of runs): + +| nprobe | old recall | new recall | old QPS | new QPS | +|---|---|---|---|---| +| 1 | 0.2912 | 0.2806 | 48,238 | 58,763 | +| 2 | 0.4356 | 0.4286 | 37,905 | 45,946 | +| 4 | 0.5955 | 0.5891 | 28,205 | 35,948 | +| 8 | 0.7474 | 0.7420 | 20,786 | 25,516 | +| 16 | 0.8710 | 0.8711 | 13,859 | 16,166 | +| 32 | 0.9486 | 0.9513 | 7,691 | 8,964 | +| 64 | 0.9860 | 0.9876 | 4,639 | 4,759 | +| 128 | 0.9984 | 0.9984 | 2,558 | 2,712 | +| 256 | 0.9998 | 0.9999 | 1,360 | 1,473 | +| 1265 | 1.0000 | 1.0000 | 544 | 471 | + +#### Concurrent single-query callers, 100k x 384 (QPS; efficiency = QPS / (clients x 1-client QPS)) + +exact: + +| clients | old QPS | old eff. | new QPS | new eff. | +|---|---|---|---|---| +| 1 | 145 | 1.00 | 156 | 1.00 | +| 2 | 152 | 0.52 | 238 | 0.76 | +| 4 | 148 | 0.26 | 450 | 0.72 | +| 8 | 149 | 0.13 | 942 | 0.75 | +| 16 | 151 | 0.07 | 1,089 | 0.44 | +| 24 | 149 | 0.04 | 1,410 | 0.38 | +| 32 | 152 | 0.03 | 1,179 | 0.24 | + +ivf: + +| clients | old QPS | old eff. | new QPS | new eff. | +|---|---|---|---|---| +| 1 | 798 | 1.00 | 1,017 | 1.00 | +| 2 | 773 | 0.48 | 1,678 | 0.82 | +| 4 | 764 | 0.24 | 2,482 | 0.61 | +| 8 | 720 | 0.11 | 3,170 | 0.39 | +| 16 | 798 | 0.06 | 3,333 | 0.20 | +| 24 | 798 | 0.04 | 3,734 | 0.15 | +| 32 | 780 | 0.03 | 3,930 | 0.12 | + +#### Batch search, 100k x 384, 2000 queries (QPS by thread cap) + +| threads | exact batch | IVF batch (nprobe 128, recall 0.955) | +|---|---|---| +| 1 | 1,217 | 8,006 | +| 2 | 2,432 | 15,497 | +| 4 | 4,648 | 28,464 | +| 8 | 8,269 | 45,112 | +| 16 | 11,514 | 56,782 | +| 24 | 13,136 | 59,332 | +| 32 | 13,553 | 57,944 | + +#### One-query exact loop by thread cap, 100k x 384 (QPS, unpinned, median of 5) + +| threads | old | new | +|---|---|---| +| 1 | 151.9 | 144.0 | +| 2 | 218.3 | 234.1 | +| 4 | 273.0 | 271.7 | +| 8 | 297.3 | 243.8 | +| auto | 296.8 | 282.3 | + +#### Memory, 1M x 384 (working set minus baseline, MB) + +| stage | old | new | +|---|---|---| +| raw vectors | 1465 | 1465 | +| after ingest | 1932 (1.32x) | 1621 (1.11x) | +| peak during ingest | 1973 (1.35x) | 1649 (1.13x) | +| after first exact query | 3516 (2.40x) | 1622 (1.11x) | +| after IVF build | 3531 (2.41x) | 1641 (1.12x) | +| peak during IVF build | 3778 (2.58x) | 1884 (1.29x) | + +#### Writes on a built IVF index, 1M x 384 (median of 2 / 2 run(s)) + +| measure | old | new | +|---|---|---| +| insert 30k, per s | 75,848 | 126,149 | +| slowest single insert, ms | 361.44 | 3.28 | +| insert p99, ms | 0.0041 | 0.0070 | +| update 10k, per s | 339,493 | 125,666 | +| slowest single update, ms | 0.14 | 1.87 | +| delete 10k, per s | 447,213 | 1,038,675 | +| first query after 30k inserts (1 thread), ms | 1,015.85 | 5.06 | +| steady query median after them (1 thread), ms | 5.53 | 4.46 | +| first query after all 50k writes (1 thread), ms | 8.38 | 4.87 | +| query median before any write (1 thread), ms | 5.59 | 4.23 | + +#### Deleting 10,000 random ids from 1M x 384 + +| store state | old per s (ns each) | new per s (ns each) | +|---|---|---| +| after_exact_search | 535,240 (1868) | 1,770,883 (565) | +| ivf_built | 567,585 (1762) | 281,683 (3550) | +| never_searched | 978,119 (1022) | 2,062,408 (485) | + diff --git a/benchmarks/rust-stage-a/raw/cb_new_glove_100000.json b/benchmarks/rust-stage-a/raw/cb_new_glove_100000.json new file mode 100644 index 0000000..526bf37 --- /dev/null +++ b/benchmarks/rust-stage-a/raw/cb_new_glove_100000.json @@ -0,0 +1,511 @@ +{ + "label": "new", + "runs": [ + { + "all_threads": 32, + "batch_allthreads": { + "nq": 2000, + "qps": 25961.483543015584, + "qps_runs": [ + 25924.164042925233, + 26353.51661325687, + 25961.483543015584 + ], + "recall": 1.0, + "threads": 32 + }, + "build": { + "first_query_ms": 2.5244999999999997, + "index_s": 0.0, + "ingest_s": 0.051049300000000006, + "ready_s": 0.053573800000000005, + "total_s": 0.051049300000000006 + }, + "concurrent_clients": { + "clients": 8, + "qps": 3051.89984580276, + "scan_threads": 1 + }, + "d": 100, + "ds": "glove", + "k": 10, + "loop_allthreads": { + "mean_ms": 0.5364724500000004, + "median_ms": 0.49660000000000004, + "nq": 2000, + "p95_ms": 0.755, + "p99_ms": 1.3289, + "qps": 1913.9292613658913, + "qps_runs": [ + 1913.9292613658913, + 1690.7627123587395, + 2018.2329180045444 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 65.703125, + "rss_mb": 61.55078125 + }, + "mem_after_first_query": { + "peak_mb": 65.703125, + "rss_mb": 61.55078125 + }, + "mem_after_ingest": { + "peak_mb": 65.703125, + "rss_mb": 59.9140625 + }, + "mem_after_sweep": { + "peak_mb": 74.8984375, + "rss_mb": 65.37890625 + }, + "mem_baseline": { + "peak_mb": 6.578125, + "rss_mb": 6.578125 + }, + "n": 100000, + "queries": 2000, + "sweep": [ + { + "knob": null, + "mean_ms": 1.7512217926524307, + "median_ms": 1.7085, + "nq": 1733, + "p95_ms": 2.0888, + "p99_ms": 2.3765, + "qps_1t": 579.7091559357759, + "qps_1t_max": 584.3323010466527, + "qps_1t_min": 550.2646255288873, + "qps_1t_runs": [ + 550.2646255288873, + 579.7091559357759, + 584.3323010466527 + ], + "recall": 1.0 + } + ], + "system": "new_exact", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + }, + { + "all_threads": 32, + "build": { + "cache_s": 0.0025245, + "first_query_ms": 0.0, + "index_s": 0.1837203, + "ingest_s": 0.051049300000000006, + "ready_s": 0.2372941, + "total_s": 0.2372941 + }, + "d": 100, + "ds": "glove", + "index_stats": { + "build_seconds": 0.1835584, + "index_bytes": 1736360, + "nlist": 1265, + "trained_n": 100000 + }, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 86.875, + "rss_mb": 67.328125 + }, + "mem_after_ingest": { + "peak_mb": 65.703125, + "rss_mb": 59.9140625 + }, + "mem_after_sweep": { + "peak_mb": 92.05078125, + "rss_mb": 72.95703125 + }, + "mem_baseline": { + "peak_mb": 6.578125, + "rss_mb": 6.578125 + }, + "mutation": { + "after": { + "mean_ms": 0.697520399999998, + "median_ms": 0.6887, + "nq": 2000, + "p95_ms": 0.8679, + "p99_ms": 0.9856999999999999, + "qps_1t": 1431.6763254709942, + "qps_1t_max": 1477.7001016509926, + "qps_1t_min": 1394.01576910638, + "qps_1t_runs": [ + 1477.7001016509926, + 1394.01576910638, + 1431.6763254709942 + ], + "recall": 0.9732 + }, + "delete_10k": { + "per_s": 2199107.1624920284, + "seconds": 0.0045473 + }, + "first_query_after_ms": 1.0215, + "index_stats_after": { + "builds": 187046, + "churn": 30000, + "pending": 32, + "trained_n": 100000 + }, + "insert_10k": { + "per_s": 649835.9164311012, + "seconds": 0.0153885 + }, + "mem_after": { + "peak_mb": 92.05078125, + "rss_mb": 83.12109375 + }, + "recall_after": 0.9732, + "recall_before": 0.9743, + "ref_knob": 256, + "update_10k": { + "per_s": 531547.3342901185, + "seconds": 0.018813 + } + }, + "n": 100000, + "queries": 2000, + "ref_knob": 256, + "ref_recall_pre": 0.9743, + "sweep": [ + { + "allthreads": { + "median_ms": 0.0147, + "qps": 67241.582194429, + "recall": 0.2832 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 184160.3668474508, + "qps_runs": [ + 175726.85018407388, + 184160.3668474508, + 186959.569992989 + ], + "recall": 0.2832, + "threads": 32 + }, + "knob": 1, + "mean_ms": 0.01576313333333332, + "median_ms": 0.0153, + "nq": 2000, + "p95_ms": 0.020999999999999998, + "p99_ms": 0.0252, + "qps_1t": 65847.95706713197, + "qps_1t_max": 66113.2983593986, + "qps_1t_min": 58902.007674931614, + "qps_1t_runs": [ + 58902.007674931614, + 66113.2983593986, + 65847.95706713197 + ], + "recall": 0.2832 + }, + { + "allthreads": { + "median_ms": 0.018600000000000002, + "qps": 52697.31191011954, + "recall": 0.40705 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 171271.00210663333, + "qps_runs": [ + 183341.58370460005, + 171271.00210663333, + 165995.76710793874 + ], + "recall": 0.40705, + "threads": 32 + }, + "knob": 2, + "mean_ms": 0.019150716666666665, + "median_ms": 0.0187, + "nq": 2000, + "p95_ms": 0.025, + "p99_ms": 0.0293, + "qps_1t": 51855.78905065024, + "qps_1t_max": 53585.6861915046, + "qps_1t_min": 51265.75140211821, + "qps_1t_runs": [ + 51265.75140211821, + 51855.78905065024, + 53585.6861915046 + ], + "recall": 0.40705 + }, + { + "allthreads": { + "median_ms": 0.0253, + "qps": 38925.348965753496, + "recall": 0.52285 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 165267.40265749983, + "qps_runs": [ + 168064.4022789533, + 165267.40265749983, + 164786.7247814516 + ], + "recall": 0.52285, + "threads": 32 + }, + "knob": 4, + "mean_ms": 0.025555166666666684, + "median_ms": 0.025, + "nq": 2000, + "p95_ms": 0.033800000000000004, + "p99_ms": 0.0388, + "qps_1t": 39881.23368608292, + "qps_1t_max": 39883.619598012956, + "qps_1t_min": 37710.092352016174, + "qps_1t_runs": [ + 39883.619598012956, + 39881.23368608292, + 37710.092352016174 + ], + "recall": 0.52285 + }, + { + "allthreads": { + "median_ms": 0.038, + "qps": 25573.74701426499, + "recall": 0.6361 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 142524.24693751024, + "qps_runs": [ + 150586.53455208035, + 142524.24693751024, + 139652.12655275708 + ], + "recall": 0.6361, + "threads": 32 + }, + "knob": 8, + "mean_ms": 0.037538783333333256, + "median_ms": 0.036399999999999995, + "nq": 2000, + "p95_ms": 0.0518, + "p99_ms": 0.0671, + "qps_1t": 27274.227355231815, + "qps_1t_max": 27545.743149718, + "qps_1t_min": 25221.666925188754, + "qps_1t_runs": [ + 25221.666925188754, + 27274.227355231815, + 27545.743149718 + ], + "recall": 0.6361 + }, + { + "allthreads": { + "median_ms": 0.0608, + "qps": 16301.898111506633, + "recall": 0.73235 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 138454.29623681222, + "qps_runs": [ + 140066.81186926164, + 124766.06363069246, + 138454.29623681222 + ], + "recall": 0.73235, + "threads": 32 + }, + "knob": 16, + "mean_ms": 0.06470711666666663, + "median_ms": 0.0631, + "nq": 2000, + "p95_ms": 0.0896, + "p99_ms": 0.1065, + "qps_1t": 16357.161901552781, + "qps_1t_max": 16432.462987931147, + "qps_1t_min": 13863.683941279, + "qps_1t_runs": [ + 13863.683941279, + 16357.161901552781, + 16432.462987931147 + ], + "recall": 0.73235 + }, + { + "allthreads": { + "median_ms": 0.101, + "qps": 9856.028069967944, + "recall": 0.81625 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 114752.59340861102, + "qps_runs": [ + 110399.64672113048, + 114872.54890698771, + 114752.59340861102 + ], + "recall": 0.81625, + "threads": 32 + }, + "knob": 32, + "mean_ms": 0.10773759999999985, + "median_ms": 0.1062, + "nq": 2000, + "p95_ms": 0.1414, + "p99_ms": 0.1666, + "qps_1t": 9200.677353866788, + "qps_1t_max": 9463.150727645303, + "qps_1t_min": 9186.777287714242, + "qps_1t_runs": [ + 9186.777287714242, + 9200.677353866788, + 9463.150727645303 + ], + "recall": 0.81625 + }, + { + "allthreads": { + "median_ms": 0.1832, + "qps": 5457.279055235853, + "recall": 0.8825 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 96282.53146031714, + "qps_runs": [ + 97756.48858692995, + 96282.53146031714, + 91983.20386697388 + ], + "recall": 0.8825, + "threads": 32 + }, + "knob": 64, + "mean_ms": 0.1875770666666666, + "median_ms": 0.1841, + "nq": 2000, + "p95_ms": 0.24420000000000003, + "p99_ms": 0.2757, + "qps_1t": 5297.499633148142, + "qps_1t_max": 5655.393676195344, + "qps_1t_min": 5072.52313133954, + "qps_1t_runs": [ + 5072.52313133954, + 5655.393676195344, + 5297.499633148142 + ], + "recall": 0.8825 + }, + { + "allthreads": { + "median_ms": 0.3292, + "qps": 3006.604156359655, + "recall": 0.9361 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 70933.80811695567, + "qps_runs": [ + 71460.47342563645, + 70246.42445699514, + 70933.80811695567 + ], + "recall": 0.9361, + "threads": 32 + }, + "knob": 128, + "mean_ms": 0.3541269666666678, + "median_ms": 0.3472, + "nq": 2000, + "p95_ms": 0.45189999999999997, + "p99_ms": 0.5279, + "qps_1t": 2851.1529920142034, + "qps_1t_max": 2896.9649512143865, + "qps_1t_min": 2728.834985301815, + "qps_1t_runs": [ + 2851.1529920142034, + 2728.834985301815, + 2896.9649512143865 + ], + "recall": 0.9361 + }, + { + "allthreads": { + "median_ms": 0.3608, + "qps": 2633.7507521004495, + "recall": 0.9743 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 52421.204377170565, + "qps_runs": [ + 54878.567449875285, + 50770.4414489884, + 52421.204377170565 + ], + "recall": 0.9743, + "threads": 32 + }, + "knob": 256, + "mean_ms": 0.6923626999999997, + "median_ms": 0.6673, + "nq": 2000, + "p95_ms": 0.8875000000000001, + "p99_ms": 1.3855, + "qps_1t": 1489.8242767163856, + "qps_1t_max": 1514.9416039247267, + "qps_1t_min": 1340.8842098883383, + "qps_1t_runs": [ + 1514.9416039247267, + 1340.8842098883383, + 1489.8242767163856 + ], + "recall": 0.9743 + }, + { + "allthreads": { + "median_ms": 0.4992, + "qps": 1912.452148056688, + "recall": 1.0 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 27194.604590449257, + "qps_runs": [ + 27484.004309491873, + 27194.604590449257, + 26594.294725952444 + ], + "recall": 1.0, + "threads": 32 + }, + "knob": 1265, + "mean_ms": 1.7281033547008489, + "median_ms": 1.6927, + "nq": 1560, + "p95_ms": 2.0214, + "p99_ms": 2.2926, + "qps_1t": 577.7273657157928, + "qps_1t_max": 587.4850544367041, + "qps_1t_min": 571.0308578486766, + "qps_1t_runs": [ + 577.7273657157928, + 571.0308578486766, + 587.4850544367041 + ], + "recall": 1.0 + } + ], + "system": "new_ivf", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + } + ], + "status": "ok" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-a/raw/cb_new_glove_100000.txt b/benchmarks/rust-stage-a/raw/cb_new_glove_100000.txt new file mode 100644 index 0000000..f6235d2 --- /dev/null +++ b/benchmarks/rust-stage-a/raw/cb_new_glove_100000.txt @@ -0,0 +1,19 @@ +new: ds=glove n=100000 d=100 queries=2000 cpus=32 release=true + ingest 0.05s + exact 1T recall=1.0 qps=579.7 median=1.708ms + exact auto-threads qps=1913.9 median=0.497ms + exact batch qps=25961.5 + exact 8 concurrent clients (scan threads 1): 3051.9 qps + ivf build 0.18s nlist=Some(1265) + ivf nprobe=1 recall=0.2832 qps1t=65848 median=0.015ms allthreads_qps=67242 batch_qps=184160 + ivf nprobe=2 recall=0.4071 qps1t=51856 median=0.019ms allthreads_qps=52697 batch_qps=171271 + ivf nprobe=4 recall=0.5229 qps1t=39881 median=0.025ms allthreads_qps=38925 batch_qps=165267 + ivf nprobe=8 recall=0.6361 qps1t=27274 median=0.036ms allthreads_qps=25574 batch_qps=142524 + ivf nprobe=16 recall=0.7323 qps1t=16357 median=0.063ms allthreads_qps=16302 batch_qps=138454 + ivf nprobe=32 recall=0.8163 qps1t=9201 median=0.106ms allthreads_qps=9856 batch_qps=114753 + ivf nprobe=64 recall=0.8825 qps1t=5297 median=0.184ms allthreads_qps=5457 batch_qps=96283 + ivf nprobe=128 recall=0.9361 qps1t=2851 median=0.347ms allthreads_qps=3007 batch_qps=70934 + ivf nprobe=256 recall=0.9743 qps1t=1490 median=0.667ms allthreads_qps=2634 batch_qps=52421 + ivf nprobe=1265 recall=1.0000 qps1t=578 median=1.693ms allthreads_qps=1912 batch_qps=27195 + mutation: ins 0.02s upd 0.02s del 0.00s recall_after=0.9732 +wrote D:/Goldmine/ISON/ison_dev/isongraphossworking/isongraph-vector/benchmarks/rust-stage-a/raw/cb_new_glove_100000.json diff --git a/benchmarks/rust-stage-a/raw/cb_new_sift_100000.json b/benchmarks/rust-stage-a/raw/cb_new_sift_100000.json new file mode 100644 index 0000000..eab14a1 --- /dev/null +++ b/benchmarks/rust-stage-a/raw/cb_new_sift_100000.json @@ -0,0 +1,511 @@ +{ + "label": "new", + "runs": [ + { + "all_threads": 32, + "batch_allthreads": { + "nq": 2000, + "qps": 29394.041827721518, + "qps_runs": [ + 29756.67963066009, + 28757.406829596544, + 29394.041827721518 + ], + "recall": 1.0, + "threads": 32 + }, + "build": { + "first_query_ms": 2.3725, + "index_s": 0.0, + "ingest_s": 0.04956590000000001, + "ready_s": 0.05193840000000001, + "total_s": 0.04956590000000001 + }, + "concurrent_clients": { + "clients": 8, + "qps": 2214.5586190898275, + "scan_threads": 1 + }, + "d": 128, + "ds": "sift", + "k": 10, + "loop_allthreads": { + "mean_ms": 0.5799744500000009, + "median_ms": 0.5398000000000001, + "nq": 2000, + "p95_ms": 0.8495, + "p99_ms": 1.3958, + "qps": 1724.2965365865991, + "qps_runs": [ + 1574.9339807361987, + 1724.2965365865991, + 1904.6550626826734 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 78.91796875, + "rss_mb": 72.1953125 + }, + "mem_after_first_query": { + "peak_mb": 78.91796875, + "rss_mb": 72.1953125 + }, + "mem_after_ingest": { + "peak_mb": 78.91796875, + "rss_mb": 71.12890625 + }, + "mem_after_sweep": { + "peak_mb": 85.84375, + "rss_mb": 75.7421875 + }, + "mem_baseline": { + "peak_mb": 6.9921875, + "rss_mb": 6.9921875 + }, + "n": 100000, + "queries": 2000, + "sweep": [ + { + "knob": null, + "mean_ms": 2.1864944792599195, + "median_ms": 2.1368, + "nq": 1117, + "p95_ms": 2.6132999999999997, + "p99_ms": 3.0438, + "qps_1t": 463.83200395149953, + "qps_1t_max": 472.6494638623304, + "qps_1t_min": 437.1016070273425, + "qps_1t_runs": [ + 437.1016070273425, + 463.83200395149953, + 472.6494638623304 + ], + "recall": 1.0 + } + ], + "system": "new_exact", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + }, + { + "all_threads": 32, + "build": { + "cache_s": 0.0023725, + "first_query_ms": 0.0, + "index_s": 0.1819441, + "ingest_s": 0.04956590000000001, + "ready_s": 0.2338825, + "total_s": 0.2338825 + }, + "d": 128, + "ds": "sift", + "index_stats": { + "build_seconds": 0.1818547, + "index_bytes": 1878040, + "nlist": 1265, + "trained_n": 100000 + }, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 103.83203125, + "rss_mb": 78.31640625 + }, + "mem_after_ingest": { + "peak_mb": 78.91796875, + "rss_mb": 71.12890625 + }, + "mem_after_sweep": { + "peak_mb": 103.83203125, + "rss_mb": 86.375 + }, + "mem_baseline": { + "peak_mb": 6.9921875, + "rss_mb": 6.9921875 + }, + "mutation": { + "after": { + "mean_ms": 0.12329048333333356, + "median_ms": 0.1193, + "nq": 2000, + "p95_ms": 0.1682, + "p99_ms": 0.1912, + "qps_1t": 7990.702019130536, + "qps_1t_max": 8686.927477185958, + "qps_1t_min": 7715.424290470302, + "qps_1t_runs": [ + 7715.424290470302, + 7990.702019130536, + 8686.927477185958 + ], + "recall": 0.948 + }, + "delete_10k": { + "per_s": 2025316.4556962026, + "seconds": 0.0049375 + }, + "first_query_after_ms": 0.1906, + "index_stats_after": { + "builds": 187046, + "churn": 30000, + "pending": 32, + "trained_n": 100000 + }, + "insert_10k": { + "per_s": 623756.3857059986, + "seconds": 0.0160319 + }, + "mem_after": { + "peak_mb": 105.671875, + "rss_mb": 96.24609375 + }, + "recall_after": 0.948, + "recall_before": 0.95125, + "ref_knob": 32, + "update_10k": { + "per_s": 532983.6960287385, + "seconds": 0.0187623 + } + }, + "n": 100000, + "queries": 2000, + "ref_knob": 32, + "ref_recall_pre": 0.95125, + "sweep": [ + { + "allthreads": { + "median_ms": 0.0163, + "qps": 60165.93765605552, + "recall": 0.2806 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 159907.89305360112, + "qps_runs": [ + 171408.98183064794, + 159907.89305360112, + 156226.81008287834 + ], + "recall": 0.2806, + "threads": 32 + }, + "knob": 1, + "mean_ms": 0.017032683333333107, + "median_ms": 0.0166, + "nq": 2000, + "p95_ms": 0.0216, + "p99_ms": 0.0249, + "qps_1t": 58763.21070430653, + "qps_1t_max": 60443.77821968892, + "qps_1t_min": 57024.571888026505, + "qps_1t_runs": [ + 57024.571888026505, + 58763.21070430653, + 60443.77821968892 + ], + "recall": 0.2806 + }, + { + "allthreads": { + "median_ms": 0.0209, + "qps": 46962.35732248807, + "recall": 0.4286 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 137330.56841122266, + "qps_runs": [ + 154724.5130045953, + 123529.99308232038, + 137330.56841122266 + ], + "recall": 0.4286, + "threads": 32 + }, + "knob": 2, + "mean_ms": 0.02188320000000003, + "median_ms": 0.0211, + "nq": 2000, + "p95_ms": 0.0285, + "p99_ms": 0.0356, + "qps_1t": 45945.641711291304, + "qps_1t_max": 46328.25421239663, + "qps_1t_min": 44843.753153076344, + "qps_1t_runs": [ + 46328.25421239663, + 45945.641711291304, + 44843.753153076344 + ], + "recall": 0.4286 + }, + { + "allthreads": { + "median_ms": 0.0268, + "qps": 36377.19513636907, + "recall": 0.5891 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 159305.42833247044, + "qps_runs": [ + 162748.49661076255, + 150618.28807253778, + 159305.42833247044 + ], + "recall": 0.5891, + "threads": 32 + }, + "knob": 4, + "mean_ms": 0.029415683333333248, + "median_ms": 0.0277, + "nq": 2000, + "p95_ms": 0.043000000000000003, + "p99_ms": 0.0563, + "qps_1t": 35948.076598161606, + "qps_1t_max": 36433.920708858335, + "qps_1t_min": 30319.384395219236, + "qps_1t_runs": [ + 30319.384395219236, + 36433.920708858335, + 35948.076598161606 + ], + "recall": 0.5891 + }, + { + "allthreads": { + "median_ms": 0.0375, + "qps": 26007.159771085004, + "recall": 0.74205 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 148816.16738842506, + "qps_runs": [ + 148059.31256061178, + 148816.16738842506, + 153602.7525613259 + ], + "recall": 0.74205, + "threads": 32 + }, + "knob": 8, + "mean_ms": 0.039136966666666474, + "median_ms": 0.0379, + "nq": 2000, + "p95_ms": 0.052500000000000005, + "p99_ms": 0.0606, + "qps_1t": 25516.257683583088, + "qps_1t_max": 25644.280492011174, + "qps_1t_min": 25493.847697204485, + "qps_1t_runs": [ + 25493.847697204485, + 25516.257683583088, + 25644.280492011174 + ], + "recall": 0.74205 + }, + { + "allthreads": { + "median_ms": 0.0627, + "qps": 15565.752853202519, + "recall": 0.8711 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 138405.4310291136, + "qps_runs": [ + 140952.27356017253, + 138405.4310291136, + 135714.67347049565 + ], + "recall": 0.8711, + "threads": 32 + }, + "knob": 16, + "mean_ms": 0.06210798333333354, + "median_ms": 0.0602, + "nq": 2000, + "p95_ms": 0.0839, + "p99_ms": 0.0979, + "qps_1t": 16165.678808977465, + "qps_1t_max": 16234.939049980085, + "qps_1t_min": 15906.10309222597, + "qps_1t_runs": [ + 16165.678808977465, + 16234.939049980085, + 15906.10309222597 + ], + "recall": 0.8711 + }, + { + "allthreads": { + "median_ms": 0.1076, + "qps": 9019.283679470886, + "recall": 0.95125 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 117271.10146882055, + "qps_runs": [ + 111528.10229357543, + 118393.6351581739, + 117271.10146882055 + ], + "recall": 0.95125, + "threads": 32 + }, + "knob": 32, + "mean_ms": 0.11064838333333324, + "median_ms": 0.1079, + "nq": 2000, + "p95_ms": 0.1492, + "p99_ms": 0.17670000000000002, + "qps_1t": 8963.838530791018, + "qps_1t_max": 9261.909194389882, + "qps_1t_min": 8895.475494299326, + "qps_1t_runs": [ + 9261.909194389882, + 8895.475494299326, + 8963.838530791018 + ], + "recall": 0.95125 + }, + { + "allthreads": { + "median_ms": 0.1932, + "qps": 5165.06781992299, + "recall": 0.9876 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 101547.58519842399, + "qps_runs": [ + 101374.64012002757, + 101678.71559446461, + 101547.58519842399 + ], + "recall": 0.9876, + "threads": 32 + }, + "knob": 64, + "mean_ms": 0.20629421666666667, + "median_ms": 0.1994, + "nq": 2000, + "p95_ms": 0.2814, + "p99_ms": 0.3338, + "qps_1t": 4759.4207975266345, + "qps_1t_max": 5184.820711492202, + "qps_1t_min": 4631.7235546532975, + "qps_1t_runs": [ + 5184.820711492202, + 4631.7235546532975, + 4759.4207975266345 + ], + "recall": 0.9876 + }, + { + "allthreads": { + "median_ms": 0.36269999999999997, + "qps": 2645.829524441241, + "recall": 0.9984 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 76888.77270141014, + "qps_runs": [ + 75002.71884855826, + 76888.77270141014, + 77596.07363867389 + ], + "recall": 0.9984, + "threads": 32 + }, + "knob": 128, + "mean_ms": 0.3662878833333332, + "median_ms": 0.3548, + "nq": 2000, + "p95_ms": 0.48899999999999993, + "p99_ms": 0.6087, + "qps_1t": 2712.174422106826, + "qps_1t_max": 2812.51744639729, + "qps_1t_min": 2669.496689089993, + "qps_1t_runs": [ + 2712.174422106826, + 2812.51744639729, + 2669.496689089993 + ], + "recall": 0.9984 + }, + { + "allthreads": { + "median_ms": 0.3448, + "qps": 2829.4402122306483, + "recall": 0.99985 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 59676.19695532043, + "qps_runs": [ + 62288.0260613101, + 59676.19695532043, + 58952.3576521634 + ], + "recall": 0.99985, + "threads": 32 + }, + "knob": 256, + "mean_ms": 0.690862283333336, + "median_ms": 0.6688, + "nq": 2000, + "p95_ms": 0.9222, + "p99_ms": 1.3037, + "qps_1t": 1472.6368247122612, + "qps_1t_max": 1521.6954974704083, + "qps_1t_min": 1358.0109268275207, + "qps_1t_runs": [ + 1358.0109268275207, + 1521.6954974704083, + 1472.6368247122612 + ], + "recall": 0.99985 + }, + { + "allthreads": { + "median_ms": 0.5376, + "qps": 1810.6224693721913, + "recall": 1.0 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 28498.14762040467, + "qps_runs": [ + 28498.14762040467, + 29130.229604469743, + 28361.673735813845 + ], + "recall": 1.0, + "threads": 32 + }, + "knob": 1265, + "mean_ms": 2.0968548542625487, + "median_ms": 2.0289, + "nq": 1521, + "p95_ms": 2.5597, + "p99_ms": 3.305, + "qps_1t": 470.96210529288334, + "qps_1t_max": 496.04395973531985, + "qps_1t_min": 464.8349744157402, + "qps_1t_runs": [ + 464.8349744157402, + 496.04395973531985, + 470.96210529288334 + ], + "recall": 1.0 + } + ], + "system": "new_ivf", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + } + ], + "status": "ok" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-a/raw/cb_new_sift_100000.txt b/benchmarks/rust-stage-a/raw/cb_new_sift_100000.txt new file mode 100644 index 0000000..ce4056b --- /dev/null +++ b/benchmarks/rust-stage-a/raw/cb_new_sift_100000.txt @@ -0,0 +1,19 @@ +new: ds=sift n=100000 d=128 queries=2000 cpus=32 release=true + ingest 0.05s + exact 1T recall=1.0 qps=463.8 median=2.137ms + exact auto-threads qps=1724.3 median=0.540ms + exact batch qps=29394.0 + exact 8 concurrent clients (scan threads 1): 2214.6 qps + ivf build 0.18s nlist=Some(1265) + ivf nprobe=1 recall=0.2806 qps1t=58763 median=0.017ms allthreads_qps=60166 batch_qps=159908 + ivf nprobe=2 recall=0.4286 qps1t=45946 median=0.021ms allthreads_qps=46962 batch_qps=137331 + ivf nprobe=4 recall=0.5891 qps1t=35948 median=0.028ms allthreads_qps=36377 batch_qps=159305 + ivf nprobe=8 recall=0.7420 qps1t=25516 median=0.038ms allthreads_qps=26007 batch_qps=148816 + ivf nprobe=16 recall=0.8711 qps1t=16166 median=0.060ms allthreads_qps=15566 batch_qps=138405 + ivf nprobe=32 recall=0.9513 qps1t=8964 median=0.108ms allthreads_qps=9019 batch_qps=117271 + ivf nprobe=64 recall=0.9876 qps1t=4759 median=0.199ms allthreads_qps=5165 batch_qps=101548 + ivf nprobe=128 recall=0.9984 qps1t=2712 median=0.355ms allthreads_qps=2646 batch_qps=76889 + ivf nprobe=256 recall=0.9999 qps1t=1473 median=0.669ms allthreads_qps=2829 batch_qps=59676 + ivf nprobe=1265 recall=1.0000 qps1t=471 median=2.029ms allthreads_qps=1811 batch_qps=28498 + mutation: ins 0.02s upd 0.02s del 0.00s recall_after=0.948 +wrote D:/Goldmine/ISON/ison_dev/isongraphossworking/isongraph-vector/benchmarks/rust-stage-a/raw/cb_new_sift_100000.json diff --git a/benchmarks/rust-stage-a/raw/cb_new_text384_1000000.json b/benchmarks/rust-stage-a/raw/cb_new_text384_1000000.json new file mode 100644 index 0000000..e2cff6d --- /dev/null +++ b/benchmarks/rust-stage-a/raw/cb_new_text384_1000000.json @@ -0,0 +1,478 @@ +{ + "label": "new", + "runs": [ + { + "all_threads": 32, + "batch_allthreads": { + "nq": 2000, + "qps": 1177.8978539937893, + "qps_runs": [ + 1177.8978539937893, + 1172.9136929639196, + 1193.593245026133 + ], + "recall": 1.0, + "threads": 32 + }, + "build": { + "first_query_ms": 83.2087, + "index_s": 0.0, + "ingest_s": 1.1464454000000002, + "ready_s": 1.2296541, + "total_s": 1.1464454000000002 + }, + "concurrent_clients": { + "clients": 8, + "qps": 60.55253675088621, + "scan_threads": 1 + }, + "d": 384, + "ds": "text384", + "k": 10, + "loop_allthreads": { + "mean_ms": 34.14873902439024, + "median_ms": 33.898900000000005, + "nq": 82, + "p95_ms": 37.709199999999996, + "p99_ms": 39.5006, + "qps": 29.305670400626024, + "qps_runs": [ + 28.908786046391754, + 29.305670400626024, + 29.645819943003385 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 1659.85546875, + "rss_mb": 1634.95703125 + }, + "mem_after_first_query": { + "peak_mb": 1659.85546875, + "rss_mb": 1634.95703125 + }, + "mem_after_ingest": { + "peak_mb": 1659.85546875, + "rss_mb": 1632.76171875 + }, + "mem_after_sweep": { + "peak_mb": 1659.85546875, + "rss_mb": 1635.90625 + }, + "mem_baseline": { + "peak_mb": 10.8359375, + "rss_mb": 10.8359375 + }, + "n": 1000000, + "queries": 2000, + "sweep": [ + { + "knob": null, + "mean_ms": 69.27620814814819, + "median_ms": 67.5136, + "nq": 45, + "p95_ms": 76.5097, + "p99_ms": 101.4826, + "qps_1t": 14.493739944364004, + "qps_1t_max": 14.88931137034174, + "qps_1t_min": 13.952635638919562, + "qps_1t_runs": [ + 13.952635638919562, + 14.493739944364004, + 14.88931137034174 + ], + "recall": 1.0 + } + ], + "system": "new_exact", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + }, + { + "all_threads": 32, + "build": { + "cache_s": 0.0832087, + "first_query_ms": 0.0, + "index_s": 7.4889216, + "ingest_s": 1.1464454000000002, + "ready_s": 8.7185757, + "total_s": 8.7185757 + }, + "d": 384, + "ds": "text384", + "index_stats": { + "build_seconds": 7.4874841, + "index_bytes": 18240000, + "nlist": 4000, + "trained_n": 1000000 + }, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 1898.40234375, + "rss_mb": 1654.32421875 + }, + "mem_after_ingest": { + "peak_mb": 1659.85546875, + "rss_mb": 1632.76171875 + }, + "mem_after_sweep": { + "peak_mb": 1898.40234375, + "rss_mb": 1670.12109375 + }, + "mem_baseline": { + "peak_mb": 10.8359375, + "rss_mb": 10.8359375 + }, + "mutation": { + "after": { + "mean_ms": 8.168543664717347, + "median_ms": 8.057699999999999, + "nq": 342, + "p95_ms": 9.585700000000001, + "p99_ms": 11.7216, + "qps_1t": 121.62173407414949, + "qps_1t_max": 124.4584873082548, + "qps_1t_min": 121.23256510879988, + "qps_1t_runs": [ + 124.4584873082548, + 121.62173407414949, + 121.23256510879988 + ], + "recall": 0.9564327485380117 + }, + "delete_10k": { + "per_s": 969039.1976355443, + "seconds": 0.0103195 + }, + "first_query_after_ms": 7.6308, + "index_stats_after": { + "builds": 168347, + "churn": 30000, + "pending": 32, + "trained_n": 1000000 + }, + "insert_10k": { + "per_s": 125089.43894884843, + "seconds": 0.0799428 + }, + "mem_after": { + "peak_mb": 1898.40234375, + "rss_mb": 1688.4765625 + }, + "recall_after": 0.9564327485380117, + "recall_before": 0.9525925925925925, + "ref_knob": 256, + "update_10k": { + "per_s": 143493.02193434333, + "seconds": 0.0696898 + } + }, + "n": 1000000, + "queries": 2000, + "ref_knob": 256, + "ref_recall_pre": 0.9525925925925925, + "sweep": [ + { + "allthreads": { + "median_ms": 0.14679999999999999, + "qps": 6778.116984876662, + "recall": 0.2709 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 49016.964771507424, + "qps_runs": [ + 45744.08471804489, + 49016.964771507424, + 50712.63936467205 + ], + "recall": 0.2709, + "threads": 32 + }, + "knob": 1, + "mean_ms": 0.20196731666666615, + "median_ms": 0.1907, + "nq": 2000, + "p95_ms": 0.3075, + "p99_ms": 0.3904, + "qps_1t": 4941.642903541345, + "qps_1t_max": 6044.111131862553, + "qps_1t_min": 4200.095930191047, + "qps_1t_runs": [ + 4200.095930191047, + 4941.642903541345, + 6044.111131862553 + ], + "recall": 0.2709 + }, + { + "allthreads": { + "median_ms": 0.1912, + "qps": 5107.104927768941, + "recall": 0.3805 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 41142.1894638967, + "qps_runs": [ + 31949.698394847153, + 41142.1894638967, + 42629.9253550007 + ], + "recall": 0.3805, + "threads": 32 + }, + "knob": 2, + "mean_ms": 0.21030843333333352, + "median_ms": 0.20579999999999998, + "nq": 2000, + "p95_ms": 0.27080000000000004, + "p99_ms": 0.314, + "qps_1t": 4742.283001428843, + "qps_1t_max": 4848.362610979016, + "qps_1t_min": 4677.24222314936, + "qps_1t_runs": [ + 4677.24222314936, + 4742.283001428843, + 4848.362610979016 + ], + "recall": 0.3805 + }, + { + "allthreads": { + "median_ms": 0.2751, + "qps": 3611.7329311307476, + "recall": 0.4949 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 36381.95964149217, + "qps_runs": [ + 36944.19763667967, + 36381.95964149217, + 28058.951857853353 + ], + "recall": 0.4949, + "threads": 32 + }, + "knob": 4, + "mean_ms": 0.2965477, + "median_ms": 0.2883, + "nq": 2000, + "p95_ms": 0.3938, + "p99_ms": 0.4792, + "qps_1t": 3311.7479123983076, + "qps_1t_max": 3500.070876435248, + "qps_1t_min": 3311.4858057300307, + "qps_1t_runs": [ + 3311.4858057300307, + 3500.070876435248, + 3311.7479123983076 + ], + "recall": 0.4949 + }, + { + "allthreads": { + "median_ms": 0.4156, + "qps": 2368.380229235525, + "recall": 0.6065 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 31236.431674991178, + "qps_runs": [ + 29735.487966791403, + 31538.42878701626, + 31236.431674991178 + ], + "recall": 0.6065, + "threads": 32 + }, + "knob": 8, + "mean_ms": 0.4302633166666668, + "median_ms": 0.421, + "nq": 2000, + "p95_ms": 0.5677, + "p99_ms": 0.6745, + "qps_1t": 2319.8675077268963, + "qps_1t_max": 2340.824791956275, + "qps_1t_min": 2311.973014650973, + "qps_1t_runs": [ + 2311.973014650973, + 2319.8675077268963, + 2340.824791956275 + ], + "recall": 0.6065 + }, + { + "allthreads": { + "median_ms": 0.6162, + "qps": 1616.8431073654963, + "recall": 0.7056 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 27400.187965289446, + "qps_runs": [ + 27400.187965289446, + 27492.693816618237, + 26823.89042977237 + ], + "recall": 0.7056, + "threads": 32 + }, + "knob": 16, + "mean_ms": 0.7042135833333327, + "median_ms": 0.6949000000000001, + "nq": 2000, + "p95_ms": 0.8919, + "p99_ms": 1.0105, + "qps_1t": 1419.6224329588626, + "qps_1t_max": 1420.8957497313781, + "qps_1t_min": 1419.5538143623928, + "qps_1t_runs": [ + 1420.8957497313781, + 1419.6224329588626, + 1419.5538143623928 + ], + "recall": 0.7056 + }, + { + "allthreads": { + "median_ms": 0.7150000000000001, + "qps": 1367.5545628628913, + "recall": 0.78655 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 25206.345445402425, + "qps_runs": [ + 24967.6980406599, + 25290.908677057858, + 25206.345445402425 + ], + "recall": 0.78655, + "threads": 32 + }, + "knob": 32, + "mean_ms": 1.2481377166666643, + "median_ms": 1.2319, + "nq": 2000, + "p95_ms": 1.5495999999999999, + "p99_ms": 1.7941, + "qps_1t": 808.7393669980795, + "qps_1t_max": 810.666260252649, + "qps_1t_min": 784.7029381475173, + "qps_1t_runs": [ + 810.666260252649, + 784.7029381475173, + 808.7393669980795 + ], + "recall": 0.78655 + }, + { + "allthreads": { + "median_ms": 0.9322, + "qps": 1035.542350549623, + "recall": 0.85785 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 20536.347795371723, + "qps_runs": [ + 19578.801248735945, + 20758.386907270207, + 20536.347795371723 + ], + "recall": 0.85785, + "threads": 32 + }, + "knob": 64, + "mean_ms": 2.2787580146460535, + "median_ms": 2.2425, + "nq": 1229, + "p95_ms": 2.7716, + "p99_ms": 3.5010000000000003, + "qps_1t": 436.0936047777688, + "qps_1t_max": 445.66531195048873, + "qps_1t_min": 434.90512433243947, + "qps_1t_runs": [ + 445.66531195048873, + 436.0936047777688, + 434.90512433243947 + ], + "recall": 0.8581773799837266 + }, + { + "allthreads": { + "median_ms": 1.5423, + "qps": 618.0029284415251, + "recall": 0.9152073732718894 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 17929.935192249246, + "qps_runs": [ + 17929.935192249246, + 16889.638038167202, + 18595.431288486732 + ], + "recall": 0.91575, + "threads": 32 + }, + "knob": 128, + "mean_ms": 4.205399667616332, + "median_ms": 4.159999999999999, + "nq": 702, + "p95_ms": 4.9195, + "p99_ms": 6.3551, + "qps_1t": 239.71305732389465, + "qps_1t_max": 240.46179351545777, + "qps_1t_min": 233.32431611296872, + "qps_1t_runs": [ + 240.46179351545777, + 239.71305732389465, + 233.32431611296872 + ], + "recall": 0.9158119658119658 + }, + { + "allthreads": { + "median_ms": 3.0018000000000002, + "qps": 316.39441382267023, + "recall": 0.9544014084507042 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 11910.674703989956, + "qps_runs": [ + 11910.674703989956, + 12040.228812508352, + 11579.87741541768 + ], + "recall": 0.9556, + "threads": 32 + }, + "knob": 256, + "mean_ms": 7.861692921810702, + "median_ms": 7.765499999999999, + "nq": 405, + "p95_ms": 9.024799999999999, + "p99_ms": 10.9761, + "qps_1t": 126.82391182500187, + "qps_1t_max": 129.83974857638734, + "qps_1t_min": 125.02612042620875, + "qps_1t_runs": [ + 125.02612042620875, + 129.83974857638734, + 126.82391182500187 + ], + "recall": 0.9525925925925925 + } + ], + "system": "new_ivf", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + } + ], + "status": "ok" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-a/raw/cb_new_text384_1000000.txt b/benchmarks/rust-stage-a/raw/cb_new_text384_1000000.txt new file mode 100644 index 0000000..1c3e885 --- /dev/null +++ b/benchmarks/rust-stage-a/raw/cb_new_text384_1000000.txt @@ -0,0 +1,18 @@ +new: ds=text384 n=1000000 d=384 queries=2000 cpus=32 release=true + ingest 1.15s + exact 1T recall=1.0 qps=14.5 median=67.514ms + exact auto-threads qps=29.3 median=33.899ms + exact batch qps=1177.9 + exact 8 concurrent clients (scan threads 1): 60.6 qps + ivf build 7.49s nlist=Some(4000) + ivf nprobe=1 recall=0.2709 qps1t=4942 median=0.191ms allthreads_qps=6778 batch_qps=49017 + ivf nprobe=2 recall=0.3805 qps1t=4742 median=0.206ms allthreads_qps=5107 batch_qps=41142 + ivf nprobe=4 recall=0.4949 qps1t=3312 median=0.288ms allthreads_qps=3612 batch_qps=36382 + ivf nprobe=8 recall=0.6065 qps1t=2320 median=0.421ms allthreads_qps=2368 batch_qps=31236 + ivf nprobe=16 recall=0.7056 qps1t=1420 median=0.695ms allthreads_qps=1617 batch_qps=27400 + ivf nprobe=32 recall=0.7865 qps1t=809 median=1.232ms allthreads_qps=1368 batch_qps=25206 + ivf nprobe=64 recall=0.8582 qps1t=436 median=2.243ms allthreads_qps=1036 batch_qps=20536 + ivf nprobe=128 recall=0.9158 qps1t=240 median=4.160ms allthreads_qps=618 batch_qps=17930 + ivf nprobe=256 recall=0.9526 qps1t=127 median=7.765ms allthreads_qps=316 batch_qps=11911 + mutation: ins 0.08s upd 0.07s del 0.01s recall_after=0.9564327485380117 +wrote D:/Goldmine/ISON/ison_dev/isongraphossworking/isongraph-vector/benchmarks/rust-stage-a/raw/cb_new_text384_1000000.json diff --git a/benchmarks/rust-stage-a/raw/cb_new_text384_100000_r1.json b/benchmarks/rust-stage-a/raw/cb_new_text384_100000_r1.json new file mode 100644 index 0000000..34c0b04 --- /dev/null +++ b/benchmarks/rust-stage-a/raw/cb_new_text384_100000_r1.json @@ -0,0 +1,511 @@ +{ + "label": "new", + "runs": [ + { + "all_threads": 32, + "batch_allthreads": { + "nq": 2000, + "qps": 10688.471147808195, + "qps_runs": [ + 10688.471147808195, + 10313.690908997149, + 10817.336295841167 + ], + "recall": 0.99995, + "threads": 32 + }, + "build": { + "first_query_ms": 9.9504, + "index_s": 0.0, + "ingest_s": 0.1178776, + "ready_s": 0.127828, + "total_s": 0.1178776 + }, + "concurrent_clients": { + "clients": 8, + "qps": 783.4123404678695, + "scan_threads": 1 + }, + "d": 384, + "ds": "text384", + "k": 10, + "loop_allthreads": { + "mean_ms": 4.042343433083955, + "median_ms": 3.9147000000000003, + "nq": 802, + "p95_ms": 5.2619, + "p99_ms": 5.973199999999999, + "qps": 248.61322180912046, + "qps_runs": [ + 248.61322180912046, + 250.26519685479704, + 243.37077874977416 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 199.1484375, + "rss_mb": 174.609375 + }, + "mem_after_first_query": { + "peak_mb": 199.1484375, + "rss_mb": 174.609375 + }, + "mem_after_ingest": { + "peak_mb": 199.1484375, + "rss_mb": 172.359375 + }, + "mem_after_sweep": { + "peak_mb": 199.1484375, + "rss_mb": 175.703125 + }, + "mem_baseline": { + "peak_mb": 10.8515625, + "rss_mb": 10.8515625 + }, + "n": 100000, + "queries": 2000, + "sweep": [ + { + "knob": null, + "mean_ms": 7.83940918727915, + "median_ms": 7.49, + "nq": 283, + "p95_ms": 10.074100000000001, + "p99_ms": 12.9789, + "qps_1t": 130.90529150179955, + "qps_1t_max": 137.46686647775246, + "qps_1t_min": 116.21641089063577, + "qps_1t_runs": [ + 116.21641089063577, + 130.90529150179955, + 137.46686647775246 + ], + "recall": 1.0 + } + ], + "system": "new_exact", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + }, + { + "all_threads": 32, + "build": { + "cache_s": 0.0099504, + "first_query_ms": 0.0, + "index_s": 0.5902345, + "ingest_s": 0.1178776, + "ready_s": 0.7180625, + "total_s": 0.7180625 + }, + "d": 384, + "ds": "text384", + "index_stats": { + "build_seconds": 0.5897921, + "index_bytes": 3173400, + "nlist": 1265, + "trained_n": 100000 + }, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 257.66015625, + "rss_mb": 178.98828125 + }, + "mem_after_ingest": { + "peak_mb": 199.1484375, + "rss_mb": 172.359375 + }, + "mem_after_sweep": { + "peak_mb": 257.66015625, + "rss_mb": 187.5625 + }, + "mem_baseline": { + "peak_mb": 10.8515625, + "rss_mb": 10.8515625 + }, + "mutation": { + "after": { + "mean_ms": 1.1187959333333297, + "median_ms": 1.0928000000000002, + "nq": 2000, + "p95_ms": 1.3813, + "p99_ms": 1.8929, + "qps_1t": 895.581991046239, + "qps_1t_max": 925.6653254238935, + "qps_1t_min": 862.4471039629965, + "qps_1t_runs": [ + 895.581991046239, + 925.6653254238935, + 862.4471039629965 + ], + "recall": 0.9518 + }, + "delete_10k": { + "per_s": 1629009.3993842343, + "seconds": 0.0061387 + }, + "first_query_after_ms": 1.4813999999999998, + "index_stats_after": { + "builds": 184994, + "churn": 30000, + "pending": 32, + "trained_n": 100000 + }, + "insert_10k": { + "per_s": 242246.88833871926, + "seconds": 0.0412802 + }, + "mem_after": { + "peak_mb": 257.66015625, + "rss_mb": 204.9609375 + }, + "recall_after": 0.9518, + "recall_before": 0.9548, + "ref_knob": 128, + "update_10k": { + "per_s": 265214.0011775502, + "seconds": 0.0377054 + } + }, + "n": 100000, + "queries": 2000, + "ref_knob": 128, + "ref_recall_pre": 0.9548, + "sweep": [ + { + "allthreads": { + "median_ms": 0.0351, + "qps": 27888.051782534578, + "recall": 0.2879 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 125300.72173215718, + "qps_runs": [ + 133868.80856760376, + 123163.32688778591, + 125300.72173215718 + ], + "recall": 0.2879, + "threads": 32 + }, + "knob": 1, + "mean_ms": 0.04051036666666665, + "median_ms": 0.0385, + "nq": 2000, + "p95_ms": 0.057, + "p99_ms": 0.0691, + "qps_1t": 26723.355812129517, + "qps_1t_max": 27328.601841401185, + "qps_1t_min": 21044.236036360267, + "qps_1t_runs": [ + 21044.236036360267, + 27328.601841401185, + 26723.355812129517 + ], + "recall": 0.2879 + }, + { + "allthreads": { + "median_ms": 0.052, + "qps": 18871.645447074006, + "recall": 0.412 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 122579.81478189986, + "qps_runs": [ + 120631.6272000193, + 122579.81478189986, + 124513.6186770428 + ], + "recall": 0.412, + "threads": 32 + }, + "knob": 2, + "mean_ms": 0.05394906666666689, + "median_ms": 0.0528, + "nq": 2000, + "p95_ms": 0.0691, + "p99_ms": 0.0787, + "qps_1t": 18518.227027907902, + "qps_1t_max": 18676.246335953907, + "qps_1t_min": 18415.392321517993, + "qps_1t_runs": [ + 18676.246335953907, + 18415.392321517993, + 18518.227027907902 + ], + "recall": 0.412 + }, + { + "allthreads": { + "median_ms": 0.0773, + "qps": 12789.506465735014, + "recall": 0.5379 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 113347.19947406898, + "qps_runs": [ + 113347.19947406898, + 110316.22144878295, + 114920.73342412071 + ], + "recall": 0.5379, + "threads": 32 + }, + "knob": 4, + "mean_ms": 0.07707878333333344, + "median_ms": 0.076, + "nq": 2000, + "p95_ms": 0.09670000000000001, + "p99_ms": 0.1069, + "qps_1t": 12921.173733579606, + "qps_1t_max": 13176.315572757934, + "qps_1t_min": 12828.695225480387, + "qps_1t_runs": [ + 12921.173733579606, + 13176.315572757934, + 12828.695225480387 + ], + "recall": 0.5379 + }, + { + "allthreads": { + "median_ms": 0.1168, + "qps": 8534.338337015899, + "recall": 0.64935 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 100856.26973000777, + "qps_runs": [ + 101171.0549611756, + 88189.6430083251, + 100856.26973000777 + ], + "recall": 0.64935, + "threads": 32 + }, + "knob": 8, + "mean_ms": 0.1214126833333329, + "median_ms": 0.1193, + "nq": 2000, + "p95_ms": 0.1537, + "p99_ms": 0.17200000000000001, + "qps_1t": 8304.51666052132, + "qps_1t_max": 8318.851599174433, + "qps_1t_min": 8089.7803826870595, + "qps_1t_runs": [ + 8089.7803826870595, + 8318.851599174433, + 8304.51666052132 + ], + "recall": 0.64935 + }, + { + "allthreads": { + "median_ms": 0.1885, + "qps": 5217.98116308801, + "recall": 0.75285 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 96194.08117818511, + "qps_runs": [ + 97735.4691791198, + 86638.97107557951, + 96194.08117818511 + ], + "recall": 0.75285, + "threads": 32 + }, + "knob": 16, + "mean_ms": 0.1932207499999998, + "median_ms": 0.1898, + "nq": 2000, + "p95_ms": 0.245, + "p99_ms": 0.27940000000000004, + "qps_1t": 5203.381990158321, + "qps_1t_max": 5216.6446437318555, + "qps_1t_min": 5107.631849047002, + "qps_1t_runs": [ + 5216.6446437318555, + 5107.631849047002, + 5203.381990158321 + ], + "recall": 0.75285 + }, + { + "allthreads": { + "median_ms": 0.37389999999999995, + "qps": 2713.931778811304, + "recall": 0.8434 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 63294.14370935329, + "qps_runs": [ + 66280.69210298694, + 55222.22805123518, + 63294.14370935329 + ], + "recall": 0.8434, + "threads": 32 + }, + "knob": 32, + "mean_ms": 0.3885988333333322, + "median_ms": 0.3495, + "nq": 2000, + "p95_ms": 0.6267, + "p99_ms": 1.085, + "qps_1t": 2449.946670785845, + "qps_1t_max": 2905.5881433408035, + "qps_1t_min": 2418.6139431400793, + "qps_1t_runs": [ + 2418.6139431400793, + 2449.946670785845, + 2905.5881433408035 + ], + "recall": 0.8434 + }, + { + "allthreads": { + "median_ms": 0.48990000000000006, + "qps": 2025.2565696285233, + "recall": 0.90755 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 73253.63338021566, + "qps_runs": [ + 76271.25107733141, + 73253.63338021566, + 69347.64668760965 + ], + "recall": 0.90755, + "threads": 32 + }, + "knob": 64, + "mean_ms": 0.684927933333334, + "median_ms": 0.656, + "nq": 2000, + "p95_ms": 0.9626, + "p99_ms": 1.2635, + "qps_1t": 1484.3943029243533, + "qps_1t_max": 1592.7300156979456, + "qps_1t_min": 1327.570702930766, + "qps_1t_runs": [ + 1327.570702930766, + 1484.3943029243533, + 1592.7300156979456 + ], + "recall": 0.90755 + }, + { + "allthreads": { + "median_ms": 0.6183, + "qps": 1598.2315887117218, + "recall": 0.9548 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 41704.20044706903, + "qps_runs": [ + 47509.32370477706, + 41704.20044706903, + 41566.906092045756 + ], + "recall": 0.9548, + "threads": 32 + }, + "knob": 128, + "mean_ms": 1.2583521666666668, + "median_ms": 1.2106000000000001, + "nq": 2000, + "p95_ms": 1.6656, + "p99_ms": 2.4541, + "qps_1t": 790.3024202853564, + "qps_1t_max": 819.289818283977, + "qps_1t_min": 775.7056439101505, + "qps_1t_runs": [ + 819.289818283977, + 790.3024202853564, + 775.7056439101505 + ], + "recall": 0.9548 + }, + { + "allthreads": { + "median_ms": 1.0264, + "qps": 926.343509571885, + "recall": 0.9825 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 30279.72409115408, + "qps_runs": [ + 29338.762960398537, + 30447.47126139306, + 30279.72409115408 + ], + "recall": 0.9825, + "threads": 32 + }, + "knob": 256, + "mean_ms": 2.3451813069908787, + "median_ms": 2.236, + "nq": 1316, + "p95_ms": 3.0219, + "p99_ms": 4.8873, + "qps_1t": 446.1110405634933, + "qps_1t_max": 456.7869798216787, + "qps_1t_min": 383.91469765492224, + "qps_1t_runs": [ + 456.7869798216787, + 446.1110405634933, + 383.91469765492224 + ], + "recall": 0.9833586626139817 + }, + { + "allthreads": { + "median_ms": 4.0726, + "qps": 243.9302292440877, + "recall": 1.0 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 10937.299198022536, + "qps_runs": [ + 11168.93517047425, + 10937.299198022536, + 10566.183668800859 + ], + "recall": 0.99995, + "threads": 32 + }, + "knob": 1265, + "mean_ms": 8.006735629251704, + "median_ms": 7.541499999999999, + "nq": 392, + "p95_ms": 10.390500000000001, + "p99_ms": 16.1296, + "qps_1t": 124.73372372867544, + "qps_1t_max": 133.030855556726, + "qps_1t_min": 117.84009802131752, + "qps_1t_runs": [ + 133.030855556726, + 124.73372372867544, + 117.84009802131752 + ], + "recall": 1.0 + } + ], + "system": "new_ivf", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + } + ], + "status": "ok" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-a/raw/cb_new_text384_100000_r1.txt b/benchmarks/rust-stage-a/raw/cb_new_text384_100000_r1.txt new file mode 100644 index 0000000..223a7c8 --- /dev/null +++ b/benchmarks/rust-stage-a/raw/cb_new_text384_100000_r1.txt @@ -0,0 +1,19 @@ +new: ds=text384 n=100000 d=384 queries=2000 cpus=32 release=true + ingest 0.12s + exact 1T recall=1.0 qps=130.9 median=7.490ms + exact auto-threads qps=248.6 median=3.915ms + exact batch qps=10688.5 + exact 8 concurrent clients (scan threads 1): 783.4 qps + ivf build 0.59s nlist=Some(1265) + ivf nprobe=1 recall=0.2879 qps1t=26723 median=0.038ms allthreads_qps=27888 batch_qps=125301 + ivf nprobe=2 recall=0.4120 qps1t=18518 median=0.053ms allthreads_qps=18872 batch_qps=122580 + ivf nprobe=4 recall=0.5379 qps1t=12921 median=0.076ms allthreads_qps=12790 batch_qps=113347 + ivf nprobe=8 recall=0.6493 qps1t=8305 median=0.119ms allthreads_qps=8534 batch_qps=100856 + ivf nprobe=16 recall=0.7529 qps1t=5203 median=0.190ms allthreads_qps=5218 batch_qps=96194 + ivf nprobe=32 recall=0.8434 qps1t=2450 median=0.349ms allthreads_qps=2714 batch_qps=63294 + ivf nprobe=64 recall=0.9075 qps1t=1484 median=0.656ms allthreads_qps=2025 batch_qps=73254 + ivf nprobe=128 recall=0.9548 qps1t=790 median=1.211ms allthreads_qps=1598 batch_qps=41704 + ivf nprobe=256 recall=0.9834 qps1t=446 median=2.236ms allthreads_qps=926 batch_qps=30280 + ivf nprobe=1265 recall=1.0000 qps1t=125 median=7.541ms allthreads_qps=244 batch_qps=10937 + mutation: ins 0.04s upd 0.04s del 0.01s recall_after=0.9518 +wrote D:/Goldmine/ISON/ison_dev/isongraphossworking/isongraph-vector/benchmarks/rust-stage-a/raw/cb_new_text384_100000_r1.json diff --git a/benchmarks/rust-stage-a/raw/cb_new_text384_100000_r2.json b/benchmarks/rust-stage-a/raw/cb_new_text384_100000_r2.json new file mode 100644 index 0000000..e233f4b --- /dev/null +++ b/benchmarks/rust-stage-a/raw/cb_new_text384_100000_r2.json @@ -0,0 +1,511 @@ +{ + "label": "new", + "runs": [ + { + "all_threads": 32, + "batch_allthreads": { + "nq": 2000, + "qps": 10376.452508792747, + "qps_runs": [ + 10376.452508792747, + 10351.848995508333, + 10435.314356018489 + ], + "recall": 0.99995, + "threads": 32 + }, + "build": { + "first_query_ms": 7.1846000000000005, + "index_s": 0.0, + "ingest_s": 0.0991351, + "ready_s": 0.1063197, + "total_s": 0.0991351 + }, + "concurrent_clients": { + "clients": 8, + "qps": 755.9558929974671, + "scan_threads": 1 + }, + "d": 384, + "ds": "text384", + "k": 10, + "loop_allthreads": { + "mean_ms": 3.6259954945054975, + "median_ms": 3.5127, + "nq": 910, + "p95_ms": 4.6993, + "p99_ms": 5.8246, + "qps": 271.15949947412923, + "qps_runs": [ + 264.91968726784097, + 292.792227556832, + 271.15949947412923 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 199.1796875, + "rss_mb": 174.6484375 + }, + "mem_after_first_query": { + "peak_mb": 199.1796875, + "rss_mb": 174.6484375 + }, + "mem_after_ingest": { + "peak_mb": 199.1796875, + "rss_mb": 172.3984375 + }, + "mem_after_sweep": { + "peak_mb": 199.1796875, + "rss_mb": 175.78125 + }, + "mem_baseline": { + "peak_mb": 10.84765625, + "rss_mb": 10.84765625 + }, + "n": 100000, + "queries": 2000, + "sweep": [ + { + "knob": null, + "mean_ms": 6.769789596167015, + "median_ms": 6.5457, + "nq": 487, + "p95_ms": 7.894699999999999, + "p99_ms": 10.435099999999998, + "qps_1t": 146.238460036347, + "qps_1t_max": 153.04316433177897, + "qps_1t_min": 144.15208014855654, + "qps_1t_runs": [ + 144.15208014855654, + 146.238460036347, + 153.04316433177897 + ], + "recall": 1.0 + } + ], + "system": "new_exact", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + }, + { + "all_threads": 32, + "build": { + "cache_s": 0.0071846, + "first_query_ms": 0.0, + "index_s": 0.5995115, + "ingest_s": 0.0991351, + "ready_s": 0.7058312, + "total_s": 0.7058312 + }, + "d": 384, + "ds": "text384", + "index_stats": { + "build_seconds": 0.5990612, + "index_bytes": 3173400, + "nlist": 1265, + "trained_n": 100000 + }, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 257.67578125, + "rss_mb": 179.04296875 + }, + "mem_after_ingest": { + "peak_mb": 199.1796875, + "rss_mb": 172.3984375 + }, + "mem_after_sweep": { + "peak_mb": 257.67578125, + "rss_mb": 187.58203125 + }, + "mem_baseline": { + "peak_mb": 10.84765625, + "rss_mb": 10.84765625 + }, + "mutation": { + "after": { + "mean_ms": 1.0914846999999992, + "median_ms": 1.0384, + "nq": 2000, + "p95_ms": 1.4838, + "p99_ms": 2.1894, + "qps_1t": 899.3997136491204, + "qps_1t_max": 978.964883648554, + "qps_1t_min": 876.3364020589346, + "qps_1t_runs": [ + 899.3997136491204, + 876.3364020589346, + 978.964883648554 + ], + "recall": 0.9518 + }, + "delete_10k": { + "per_s": 2201430.930104568, + "seconds": 0.0045425 + }, + "first_query_after_ms": 1.0471, + "index_stats_after": { + "builds": 185462, + "churn": 30000, + "pending": 32, + "trained_n": 100000 + }, + "insert_10k": { + "per_s": 298345.37654169975, + "seconds": 0.0335182 + }, + "mem_after": { + "peak_mb": 257.67578125, + "rss_mb": 211.59375 + }, + "recall_after": 0.9518, + "recall_before": 0.9548, + "ref_knob": 128, + "update_10k": { + "per_s": 281349.35148974485, + "seconds": 0.035543 + } + }, + "n": 100000, + "queries": 2000, + "ref_knob": 128, + "ref_recall_pre": 0.9548, + "sweep": [ + { + "allthreads": { + "median_ms": 0.0359, + "qps": 27064.222045703398, + "recall": 0.2879 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 123295.44053460902, + "qps_runs": [ + 130944.04106405129, + 123295.44053460902, + 117962.78274204487 + ], + "recall": 0.2879, + "threads": 32 + }, + "knob": 1, + "mean_ms": 0.03874041666666659, + "median_ms": 0.037, + "nq": 2000, + "p95_ms": 0.0541, + "p99_ms": 0.06559999999999999, + "qps_1t": 27644.240737105963, + "qps_1t_max": 28125.914092208048, + "qps_1t_min": 22475.47083301961, + "qps_1t_runs": [ + 22475.47083301961, + 27644.240737105963, + 28125.914092208048 + ], + "recall": 0.2879 + }, + { + "allthreads": { + "median_ms": 0.0524, + "qps": 18559.34908650885, + "recall": 0.412 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 113138.81000599636, + "qps_runs": [ + 100726.2361627333, + 113138.81000599636, + 113475.17730496456 + ], + "recall": 0.412, + "threads": 32 + }, + "knob": 2, + "mean_ms": 0.054330599999999986, + "median_ms": 0.0536, + "nq": 2000, + "p95_ms": 0.06849999999999999, + "p99_ms": 0.0756, + "qps_1t": 18337.940740944545, + "qps_1t_max": 18591.904898688106, + "qps_1t_min": 18290.49698938422, + "qps_1t_runs": [ + 18290.49698938422, + 18591.904898688106, + 18337.940740944545 + ], + "recall": 0.412 + }, + { + "allthreads": { + "median_ms": 0.0732, + "qps": 12985.663827134866, + "recall": 0.5379 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 106713.33596559563, + "qps_runs": [ + 108388.15967743685, + 106713.33596559563, + 102355.19299071639 + ], + "recall": 0.5379, + "threads": 32 + }, + "knob": 4, + "mean_ms": 0.07783946666666697, + "median_ms": 0.07629999999999999, + "nq": 2000, + "p95_ms": 0.0987, + "p99_ms": 0.1133, + "qps_1t": 12750.491372061242, + "qps_1t_max": 13357.332006505028, + "qps_1t_min": 12464.973424676657, + "qps_1t_runs": [ + 12464.973424676657, + 12750.491372061242, + 13357.332006505028 + ], + "recall": 0.5379 + }, + { + "allthreads": { + "median_ms": 0.1125, + "qps": 8900.33669973733, + "recall": 0.64935 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 100920.39399321814, + "qps_runs": [ + 94147.77434661445, + 100920.39399321814, + 104815.21078338889 + ], + "recall": 0.64935, + "threads": 32 + }, + "knob": 8, + "mean_ms": 0.11404573333333319, + "median_ms": 0.1121, + "nq": 2000, + "p95_ms": 0.1419, + "p99_ms": 0.16, + "qps_1t": 8625.418764080996, + "qps_1t_max": 9161.92231789306, + "qps_1t_min": 8543.109384263926, + "qps_1t_runs": [ + 8543.109384263926, + 8625.418764080996, + 9161.92231789306 + ], + "recall": 0.64935 + }, + { + "allthreads": { + "median_ms": 0.1797, + "qps": 5704.301556760948, + "recall": 0.75285 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 100468.68642215937, + "qps_runs": [ + 97086.4360540189, + 101497.59704438997, + 100468.68642215937 + ], + "recall": 0.75285, + "threads": 32 + }, + "knob": 16, + "mean_ms": 0.17648231666666647, + "median_ms": 0.1736, + "nq": 2000, + "p95_ms": 0.22039999999999998, + "p99_ms": 0.2491, + "qps_1t": 5629.922135361903, + "qps_1t_max": 5819.348227179214, + "qps_1t_min": 5556.047883131876, + "qps_1t_runs": [ + 5556.047883131876, + 5629.922135361903, + 5819.348227179214 + ], + "recall": 0.75285 + }, + { + "allthreads": { + "median_ms": 0.292, + "qps": 3354.215561346423, + "recall": 0.8434 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 95432.14059062951, + "qps_runs": [ + 97006.38302000272, + 95432.14059062951, + 95327.97590108769 + ], + "recall": 0.8434, + "threads": 32 + }, + "knob": 32, + "mean_ms": 0.2910502000000004, + "median_ms": 0.2875, + "nq": 2000, + "p95_ms": 0.3559, + "p99_ms": 0.3928, + "qps_1t": 3472.03234545333, + "qps_1t_max": 3481.69351661664, + "qps_1t_min": 3356.624953888368, + "qps_1t_runs": [ + 3356.624953888368, + 3472.03234545333, + 3481.69351661664 + ], + "recall": 0.8434 + }, + { + "allthreads": { + "median_ms": 0.46549999999999997, + "qps": 2126.887559536898, + "recall": 0.90755 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 76977.55334544447, + "qps_runs": [ + 76977.55334544447, + 79766.44385240017, + 75567.79753875683 + ], + "recall": 0.90755, + "threads": 32 + }, + "knob": 64, + "mean_ms": 0.5238249500000017, + "median_ms": 0.5201, + "nq": 2000, + "p95_ms": 0.6351, + "p99_ms": 0.6946, + "qps_1t": 1924.3418077497888, + "qps_1t_max": 1939.8698541314898, + "qps_1t_min": 1864.5649042457303, + "qps_1t_runs": [ + 1864.5649042457303, + 1939.8698541314898, + 1924.3418077497888 + ], + "recall": 0.90755 + }, + { + "allthreads": { + "median_ms": 0.5310999999999999, + "qps": 1867.2671797916894, + "recall": 0.9548 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 51698.68969670963, + "qps_runs": [ + 60406.53598719381, + 51001.672854869634, + 51698.68969670963 + ], + "recall": 0.9548, + "threads": 32 + }, + "knob": 128, + "mean_ms": 1.0607772666666675, + "median_ms": 1.0428, + "nq": 2000, + "p95_ms": 1.2871, + "p99_ms": 1.616, + "qps_1t": 951.941835211537, + "qps_1t_max": 959.0259518656227, + "qps_1t_min": 918.1701090088275, + "qps_1t_runs": [ + 959.0259518656227, + 918.1701090088275, + 951.941835211537 + ], + "recall": 0.9548 + }, + { + "allthreads": { + "median_ms": 0.7902, + "qps": 1225.5640903838794, + "recall": 0.9825 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 38667.29307681451, + "qps_runs": [ + 40590.01647954669, + 38667.29307681451, + 38463.23972021839 + ], + "recall": 0.9825, + "threads": 32 + }, + "knob": 256, + "mean_ms": 2.0016179574275355, + "median_ms": 1.9925, + "nq": 1472, + "p95_ms": 2.2713, + "p99_ms": 2.4863, + "qps_1t": 502.8425120139906, + "qps_1t_max": 502.9957119273846, + "qps_1t_min": 493.07935325005417, + "qps_1t_runs": [ + 502.8425120139906, + 493.07935325005417, + 502.9957119273846 + ], + "recall": 0.983491847826087 + }, + { + "allthreads": { + "median_ms": 3.2626999999999997, + "qps": 302.6828202750725, + "recall": 0.9999019607843137 + }, + "batch_allthreads": { + "nq": 2000, + "qps": 13798.965629536411, + "qps_runs": [ + 13471.016098537788, + 13798.965629536411, + 14042.7940104675 + ], + "recall": 0.99995, + "threads": 32 + }, + "knob": 1265, + "mean_ms": 6.429342714996442, + "median_ms": 6.3814, + "nq": 469, + "p95_ms": 6.8987, + "p99_ms": 7.308, + "qps_1t": 155.63790685001598, + "qps_1t_max": 156.73159877459267, + "qps_1t_min": 154.26090492209704, + "qps_1t_runs": [ + 154.26090492209704, + 155.63790685001598, + 156.73159877459267 + ], + "recall": 1.0 + } + ], + "system": "new_ivf", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + } + ], + "status": "ok" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-a/raw/cb_new_text384_100000_r2.txt b/benchmarks/rust-stage-a/raw/cb_new_text384_100000_r2.txt new file mode 100644 index 0000000..07ab26e --- /dev/null +++ b/benchmarks/rust-stage-a/raw/cb_new_text384_100000_r2.txt @@ -0,0 +1,19 @@ +new: ds=text384 n=100000 d=384 queries=2000 cpus=32 release=true + ingest 0.10s + exact 1T recall=1.0 qps=146.2 median=6.546ms + exact auto-threads qps=271.2 median=3.513ms + exact batch qps=10376.5 + exact 8 concurrent clients (scan threads 1): 756.0 qps + ivf build 0.60s nlist=Some(1265) + ivf nprobe=1 recall=0.2879 qps1t=27644 median=0.037ms allthreads_qps=27064 batch_qps=123295 + ivf nprobe=2 recall=0.4120 qps1t=18338 median=0.054ms allthreads_qps=18559 batch_qps=113139 + ivf nprobe=4 recall=0.5379 qps1t=12750 median=0.076ms allthreads_qps=12986 batch_qps=106713 + ivf nprobe=8 recall=0.6493 qps1t=8625 median=0.112ms allthreads_qps=8900 batch_qps=100920 + ivf nprobe=16 recall=0.7529 qps1t=5630 median=0.174ms allthreads_qps=5704 batch_qps=100469 + ivf nprobe=32 recall=0.8434 qps1t=3472 median=0.287ms allthreads_qps=3354 batch_qps=95432 + ivf nprobe=64 recall=0.9075 qps1t=1924 median=0.520ms allthreads_qps=2127 batch_qps=76978 + ivf nprobe=128 recall=0.9548 qps1t=952 median=1.043ms allthreads_qps=1867 batch_qps=51699 + ivf nprobe=256 recall=0.9835 qps1t=503 median=1.992ms allthreads_qps=1226 batch_qps=38667 + ivf nprobe=1265 recall=1.0000 qps1t=156 median=6.381ms allthreads_qps=303 batch_qps=13799 + mutation: ins 0.03s upd 0.04s del 0.00s recall_after=0.9518 +wrote D:/Goldmine/ISON/ison_dev/isongraphossworking/isongraph-vector/benchmarks/rust-stage-a/raw/cb_new_text384_100000_r2.json diff --git a/benchmarks/rust-stage-a/raw/cb_old_glove_100000.json b/benchmarks/rust-stage-a/raw/cb_old_glove_100000.json new file mode 100644 index 0000000..bb2db7d --- /dev/null +++ b/benchmarks/rust-stage-a/raw/cb_old_glove_100000.json @@ -0,0 +1,390 @@ +{ + "label": "old", + "runs": [ + { + "all_threads": 32, + "build": { + "first_query_ms": 48.898700000000005, + "index_s": 0.0, + "ingest_s": 0.0559634, + "ready_s": 0.10486210000000001, + "total_s": 0.0559634 + }, + "concurrent_clients": { + "clients": 8, + "qps": 671.3442577827682, + "scan_threads": 1 + }, + "d": 100, + "ds": "glove", + "k": 10, + "loop_allthreads": { + "mean_ms": 0.6168386166666662, + "median_ms": 0.5848, + "nq": 2000, + "p95_ms": 0.8011999999999999, + "p99_ms": 1.111, + "qps": 1612.0471507671143, + "qps_runs": [ + 1612.0471507671143, + 1670.131214693949, + 1583.7036890397826 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 127.71875, + "rss_mb": 127.71875 + }, + "mem_after_first_query": { + "peak_mb": 127.71875, + "rss_mb": 127.71875 + }, + "mem_after_ingest": { + "peak_mb": 83.90234375, + "rss_mb": 78.4453125 + }, + "mem_after_sweep": { + "peak_mb": 127.9921875, + "rss_mb": 127.8671875 + }, + "mem_baseline": { + "peak_mb": 6.56640625, + "rss_mb": 6.56640625 + }, + "n": 100000, + "queries": 2000, + "sweep": [ + { + "knob": null, + "mean_ms": 1.5006669833333328, + "median_ms": 1.4621, + "nq": 2000, + "p95_ms": 1.8216, + "p99_ms": 2.3749000000000002, + "qps_1t": 664.1304410629717, + "qps_1t_max": 680.0059813326119, + "qps_1t_min": 655.437987094884, + "qps_1t_runs": [ + 664.1304410629717, + 680.0059813326119, + 655.437987094884 + ], + "recall": 1.0 + } + ], + "system": "old_exact", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + }, + { + "all_threads": 32, + "build": { + "cache_s": 0.0488987, + "first_query_ms": 0.0, + "index_s": 0.2190106, + "ingest_s": 0.0559634, + "ready_s": 0.3238727, + "total_s": 0.3238727 + }, + "d": 100, + "ds": "glove", + "index_stats": { + "build_seconds": 0.2189998, + "index_bytes": 1316128, + "nlist": 1265, + "trained_n": 100000 + }, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 150.69140625, + "rss_mb": 131.2890625 + }, + "mem_after_ingest": { + "peak_mb": 83.90234375, + "rss_mb": 78.4453125 + }, + "mem_after_sweep": { + "peak_mb": 150.69140625, + "rss_mb": 131.296875 + }, + "mem_baseline": { + "peak_mb": 6.56640625, + "rss_mb": 6.56640625 + }, + "mutation": { + "after": { + "mean_ms": 0.8263747999999989, + "median_ms": 0.8136, + "nq": 2000, + "p95_ms": 1.0383, + "p99_ms": 1.1816000000000002, + "qps_1t": 1227.0015431998427, + "qps_1t_max": 1234.752809278724, + "qps_1t_min": 1170.6164044563968, + "qps_1t_runs": [ + 1170.6164044563968, + 1227.0015431998427, + 1234.752809278724 + ], + "recall": 0.9743 + }, + "delete_10k": { + "per_s": 1030120.7301495735, + "seconds": 0.0097076 + }, + "first_query_after_ms": 82.4661, + "index_stats_after": { + "builds": 115046, + "churn": 30000, + "pending": 0, + "trained_n": 100000 + }, + "insert_10k": { + "per_s": 675680.2410827101, + "seconds": 0.0147999 + }, + "mem_after": { + "peak_mb": 172.6875, + "rss_mb": 145.3125 + }, + "recall_after": 0.9743, + "recall_before": 0.974, + "ref_knob": 256, + "update_10k": { + "per_s": 705696.3811889573, + "seconds": 0.0141704 + } + }, + "n": 100000, + "queries": 2000, + "ref_knob": 256, + "ref_recall_pre": 0.974, + "sweep": [ + { + "allthreads": { + "median_ms": 0.019, + "qps": 51525.409755821085, + "recall": 0.28505 + }, + "knob": 1, + "mean_ms": 0.018996650000000087, + "median_ms": 0.0184, + "nq": 2000, + "p95_ms": 0.024300000000000002, + "p99_ms": 0.0307, + "qps_1t": 52731.21318702172, + "qps_1t_max": 53166.03753522241, + "qps_1t_min": 52037.66486182696, + "qps_1t_runs": [ + 53166.03753522241, + 52731.21318702172, + 52037.66486182696 + ], + "recall": 0.28505 + }, + { + "allthreads": { + "median_ms": 0.0241, + "qps": 40142.82818267393, + "recall": 0.4016 + }, + "knob": 2, + "mean_ms": 0.024484899999999907, + "median_ms": 0.0237, + "nq": 2000, + "p95_ms": 0.0313, + "p99_ms": 0.0406, + "qps_1t": 40514.20630644133, + "qps_1t_max": 41691.160223297826, + "qps_1t_min": 40345.193475375294, + "qps_1t_runs": [ + 41691.160223297826, + 40345.193475375294, + 40514.20630644133 + ], + "recall": 0.4016 + }, + { + "allthreads": { + "median_ms": 0.0304, + "qps": 31997.389013056545, + "recall": 0.52085 + }, + "knob": 4, + "mean_ms": 0.03286151666666674, + "median_ms": 0.031900000000000005, + "nq": 2000, + "p95_ms": 0.0435, + "p99_ms": 0.0527, + "qps_1t": 30934.324881560147, + "qps_1t_max": 31195.84596115179, + "qps_1t_min": 29237.67157031151, + "qps_1t_runs": [ + 29237.67157031151, + 30934.324881560147, + 31195.84596115179 + ], + "recall": 0.52085 + }, + { + "allthreads": { + "median_ms": 0.0463, + "qps": 20861.14819759678, + "recall": 0.63505 + }, + "knob": 8, + "mean_ms": 0.04882274999999991, + "median_ms": 0.0472, + "nq": 2000, + "p95_ms": 0.06499999999999999, + "p99_ms": 0.0785, + "qps_1t": 20568.069511847738, + "qps_1t_max": 20804.661908640555, + "qps_1t_min": 20087.158179340186, + "qps_1t_runs": [ + 20804.661908640555, + 20568.069511847738, + 20087.158179340186 + ], + "recall": 0.63505 + }, + { + "allthreads": { + "median_ms": 0.0804, + "qps": 11555.717335598252, + "recall": 0.73085 + }, + "knob": 16, + "mean_ms": 0.07509308333333325, + "median_ms": 0.0735, + "nq": 2000, + "p95_ms": 0.0991, + "p99_ms": 0.116, + "qps_1t": 13275.267314101447, + "qps_1t_max": 13656.062506529279, + "qps_1t_min": 13033.790754219857, + "qps_1t_runs": [ + 13275.267314101447, + 13656.062506529279, + 13033.790754219857 + ], + "recall": 0.73085 + }, + { + "allthreads": { + "median_ms": 0.1318, + "qps": 7145.148694116895, + "recall": 0.8134 + }, + "knob": 32, + "mean_ms": 0.13647804999999966, + "median_ms": 0.1328, + "nq": 2000, + "p95_ms": 0.1874, + "p99_ms": 0.2273, + "qps_1t": 7465.77037604711, + "qps_1t_max": 7644.599510898529, + "qps_1t_min": 6911.891356126906, + "qps_1t_runs": [ + 6911.891356126906, + 7465.77037604711, + 7644.599510898529 + ], + "recall": 0.8134 + }, + { + "allthreads": { + "median_ms": 0.221, + "qps": 4461.099214846539, + "recall": 0.88165 + }, + "knob": 64, + "mean_ms": 0.22983740000000022, + "median_ms": 0.22540000000000002, + "nq": 2000, + "p95_ms": 0.30519999999999997, + "p99_ms": 0.3569, + "qps_1t": 4322.698055218146, + "qps_1t_max": 4562.164509827354, + "qps_1t_min": 4184.433072085219, + "qps_1t_runs": [ + 4562.164509827354, + 4322.698055218146, + 4184.433072085219 + ], + "recall": 0.88165 + }, + { + "allthreads": { + "median_ms": 0.3781, + "qps": 2602.880138931326, + "recall": 0.9355 + }, + "knob": 128, + "mean_ms": 0.42356358333333266, + "median_ms": 0.40440000000000004, + "nq": 2000, + "p95_ms": 0.6018, + "p99_ms": 0.759, + "qps_1t": 2276.832038669714, + "qps_1t_max": 2571.7389803234933, + "qps_1t_min": 2259.161776428608, + "qps_1t_runs": [ + 2259.161776428608, + 2571.7389803234933, + 2276.832038669714 + ], + "recall": 0.9355 + }, + { + "allthreads": { + "median_ms": 0.514, + "qps": 1852.8249280548034, + "recall": 0.974 + }, + "knob": 256, + "mean_ms": 0.7563961666666665, + "median_ms": 0.7435, + "nq": 2000, + "p95_ms": 0.9576, + "p99_ms": 1.0976, + "qps_1t": 1324.9131569109893, + "qps_1t_max": 1328.7160450766926, + "qps_1t_min": 1312.653268677284, + "qps_1t_runs": [ + 1312.653268677284, + 1324.9131569109893, + 1328.7160450766926 + ], + "recall": 0.974 + }, + { + "allthreads": { + "median_ms": 0.6959, + "qps": 1282.619745700115, + "recall": 1.0 + }, + "knob": 1265, + "mean_ms": 1.5376417833333307, + "median_ms": 1.4878, + "nq": 2000, + "p95_ms": 1.918, + "p99_ms": 2.5446, + "qps_1t": 637.945977267369, + "qps_1t_max": 687.0460085856016, + "qps_1t_min": 628.9751623369186, + "qps_1t_runs": [ + 687.0460085856016, + 637.945977267369, + 628.9751623369186 + ], + "recall": 1.0 + } + ], + "system": "old_ivf", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + } + ], + "status": "ok" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-a/raw/cb_old_glove_100000.txt b/benchmarks/rust-stage-a/raw/cb_old_glove_100000.txt new file mode 100644 index 0000000..f7c9bfe --- /dev/null +++ b/benchmarks/rust-stage-a/raw/cb_old_glove_100000.txt @@ -0,0 +1,18 @@ +old: ds=glove n=100000 d=100 queries=2000 cpus=32 release=true + ingest 0.06s + exact 1T recall=1.0 qps=664.1 median=1.462ms + exact auto-threads qps=1612.0 median=0.585ms + exact 8 concurrent clients (scan threads 1): 671.3 qps + ivf build 0.22s nlist=Some(1265) + ivf nprobe=1 recall=0.2851 qps1t=52731 median=0.018ms allthreads_qps=51525 + ivf nprobe=2 recall=0.4016 qps1t=40514 median=0.024ms allthreads_qps=40143 + ivf nprobe=4 recall=0.5209 qps1t=30934 median=0.032ms allthreads_qps=31997 + ivf nprobe=8 recall=0.6351 qps1t=20568 median=0.047ms allthreads_qps=20861 + ivf nprobe=16 recall=0.7308 qps1t=13275 median=0.073ms allthreads_qps=11556 + ivf nprobe=32 recall=0.8134 qps1t=7466 median=0.133ms allthreads_qps=7145 + ivf nprobe=64 recall=0.8817 qps1t=4323 median=0.225ms allthreads_qps=4461 + ivf nprobe=128 recall=0.9355 qps1t=2277 median=0.404ms allthreads_qps=2603 + ivf nprobe=256 recall=0.9740 qps1t=1325 median=0.744ms allthreads_qps=1853 + ivf nprobe=1265 recall=1.0000 qps1t=638 median=1.488ms allthreads_qps=1283 + mutation: ins 0.01s upd 0.01s del 0.01s recall_after=0.9743 +wrote D:/Goldmine/ISON/ison_dev/isongraphossworking/isongraph-vector/benchmarks/rust-stage-a/raw/cb_old_glove_100000.json diff --git a/benchmarks/rust-stage-a/raw/cb_old_sift_100000.json b/benchmarks/rust-stage-a/raw/cb_old_sift_100000.json new file mode 100644 index 0000000..452a64a --- /dev/null +++ b/benchmarks/rust-stage-a/raw/cb_old_sift_100000.json @@ -0,0 +1,390 @@ +{ + "label": "old", + "runs": [ + { + "all_threads": 32, + "build": { + "first_query_ms": 51.0198, + "index_s": 0.0, + "ingest_s": 0.0563772, + "ready_s": 0.10739699999999999, + "total_s": 0.0563772 + }, + "concurrent_clients": { + "clients": 8, + "qps": 487.5955092941189, + "scan_threads": 1 + }, + "d": 128, + "ds": "sift", + "k": 10, + "loop_allthreads": { + "mean_ms": 0.9803539166666683, + "median_ms": 0.8455999999999999, + "nq": 2000, + "p95_ms": 1.893, + "p99_ms": 2.4463000000000004, + "qps": 1030.1208362644572, + "qps_runs": [ + 1186.2721971546434, + 1030.1208362644572, + 887.0556405650545 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 149.765625, + "rss_mb": 149.765625 + }, + "mem_after_first_query": { + "peak_mb": 149.765625, + "rss_mb": 149.765625 + }, + "mem_after_ingest": { + "peak_mb": 97.25, + "rss_mb": 89.6640625 + }, + "mem_after_sweep": { + "peak_mb": 150.07421875, + "rss_mb": 149.94921875 + }, + "mem_baseline": { + "peak_mb": 7.01171875, + "rss_mb": 7.01171875 + }, + "n": 100000, + "queries": 2000, + "sweep": [ + { + "knob": null, + "mean_ms": 2.3268668231611884, + "median_ms": 2.1822999999999997, + "nq": 1278, + "p95_ms": 3.2617000000000003, + "p99_ms": 4.3731, + "qps_1t": 437.72777771155853, + "qps_1t_max": 438.37184716999127, + "qps_1t_min": 414.09458736900024, + "qps_1t_runs": [ + 414.09458736900024, + 438.37184716999127, + 437.72777771155853 + ], + "recall": 1.0 + } + ], + "system": "old_exact", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + }, + { + "all_threads": 32, + "build": { + "cache_s": 0.0510198, + "first_query_ms": 0.0, + "index_s": 0.2024824, + "ingest_s": 0.0563772, + "ready_s": 0.3098794, + "total_s": 0.3098794 + }, + "d": 128, + "ds": "sift", + "index_stats": { + "build_seconds": 0.2024088, + "index_bytes": 1457808, + "nlist": 1265, + "trained_n": 100000 + }, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 178.88671875, + "rss_mb": 152.8203125 + }, + "mem_after_ingest": { + "peak_mb": 97.25, + "rss_mb": 89.6640625 + }, + "mem_after_sweep": { + "peak_mb": 178.88671875, + "rss_mb": 152.8125 + }, + "mem_baseline": { + "peak_mb": 7.01171875, + "rss_mb": 7.01171875 + }, + "mutation": { + "after": { + "mean_ms": 0.22069976666666716, + "median_ms": 0.2152, + "nq": 2000, + "p95_ms": 0.289, + "p99_ms": 0.3305, + "qps_1t": 4497.469611160026, + "qps_1t_max": 4606.14256748229, + "qps_1t_min": 4491.341031342137, + "qps_1t_runs": [ + 4497.469611160026, + 4491.341031342137, + 4606.14256748229 + ], + "recall": 0.98545 + }, + "delete_10k": { + "per_s": 1041742.627066557, + "seconds": 0.0095993 + }, + "first_query_after_ms": 75.3283, + "index_stats_after": { + "builds": 115046, + "churn": 30000, + "pending": 0, + "trained_n": 100000 + }, + "insert_10k": { + "per_s": 558403.4129616601, + "seconds": 0.0179082 + }, + "mem_after": { + "peak_mb": 206.8203125, + "rss_mb": 170.8046875 + }, + "recall_after": 0.98545, + "recall_before": 0.98605, + "ref_knob": 64, + "update_10k": { + "per_s": 647039.7929472663, + "seconds": 0.015455 + } + }, + "n": 100000, + "queries": 2000, + "ref_knob": 64, + "ref_recall_pre": 0.98605, + "sweep": [ + { + "allthreads": { + "median_ms": 0.0203, + "qps": 48288.06729425059, + "recall": 0.2912 + }, + "knob": 1, + "mean_ms": 0.021887566666666473, + "median_ms": 0.0207, + "nq": 2000, + "p95_ms": 0.0301, + "p99_ms": 0.0353, + "qps_1t": 48238.219623790166, + "qps_1t_max": 48779.773954527474, + "qps_1t_min": 40930.01172644834, + "qps_1t_runs": [ + 40930.01172644834, + 48779.773954527474, + 48238.219623790166 + ], + "recall": 0.2912 + }, + { + "allthreads": { + "median_ms": 0.0255, + "qps": 38581.512125204754, + "recall": 0.43565 + }, + "knob": 2, + "mean_ms": 0.02654659999999985, + "median_ms": 0.026, + "nq": 2000, + "p95_ms": 0.0331, + "p99_ms": 0.0379, + "qps_1t": 37904.77562268077, + "qps_1t_max": 38259.57206668649, + "qps_1t_min": 36872.27260408586, + "qps_1t_runs": [ + 37904.77562268077, + 38259.57206668649, + 36872.27260408586 + ], + "recall": 0.43565 + }, + { + "allthreads": { + "median_ms": 0.0329, + "qps": 30561.179661534974, + "recall": 0.5955 + }, + "knob": 4, + "mean_ms": 0.035091883333333525, + "median_ms": 0.0343, + "nq": 2000, + "p95_ms": 0.0458, + "p99_ms": 0.0531, + "qps_1t": 28205.363249822003, + "qps_1t_max": 29721.480010818672, + "qps_1t_min": 27642.86523826771, + "qps_1t_runs": [ + 29721.480010818672, + 28205.363249822003, + 27642.86523826771 + ], + "recall": 0.5955 + }, + { + "allthreads": { + "median_ms": 0.046, + "qps": 21162.47596736322, + "recall": 0.74745 + }, + "knob": 8, + "mean_ms": 0.04864388333333326, + "median_ms": 0.0474, + "nq": 2000, + "p95_ms": 0.0649, + "p99_ms": 0.0742, + "qps_1t": 20785.95866919976, + "qps_1t_max": 21251.162172931337, + "qps_1t_min": 19698.22322026554, + "qps_1t_runs": [ + 21251.162172931337, + 19698.22322026554, + 20785.95866919976 + ], + "recall": 0.74745 + }, + { + "allthreads": { + "median_ms": 0.0737, + "qps": 13221.567020123197, + "recall": 0.87095 + }, + "knob": 16, + "mean_ms": 0.07253490000000024, + "median_ms": 0.0709, + "nq": 2000, + "p95_ms": 0.0969, + "p99_ms": 0.1098, + "qps_1t": 13858.99580488195, + "qps_1t_max": 13967.71920414728, + "qps_1t_min": 13539.907183936275, + "qps_1t_runs": [ + 13539.907183936275, + 13858.99580488195, + 13967.71920414728 + ], + "recall": 0.87095 + }, + { + "allthreads": { + "median_ms": 0.13799999999999998, + "qps": 7238.810156774532, + "recall": 0.94865 + }, + "knob": 32, + "mean_ms": 0.12946330000000003, + "median_ms": 0.1268, + "nq": 2000, + "p95_ms": 0.1719, + "p99_ms": 0.1967, + "qps_1t": 7691.3196535522, + "qps_1t_max": 7793.29682953097, + "qps_1t_min": 7688.88909392592, + "qps_1t_runs": [ + 7691.3196535522, + 7688.88909392592, + 7793.29682953097 + ], + "recall": 0.94865 + }, + { + "allthreads": { + "median_ms": 0.2111, + "qps": 4603.7230308150265, + "recall": 0.98605 + }, + "knob": 64, + "mean_ms": 0.21435153333333323, + "median_ms": 0.2099, + "nq": 2000, + "p95_ms": 0.2808, + "p99_ms": 0.31770000000000004, + "qps_1t": 4639.1581598036955, + "qps_1t_max": 4724.3071508294015, + "qps_1t_min": 4633.340592233598, + "qps_1t_runs": [ + 4724.3071508294015, + 4639.1581598036955, + 4633.340592233598 + ], + "recall": 0.98605 + }, + { + "allthreads": { + "median_ms": 0.3883, + "qps": 2568.91103861073, + "recall": 0.9984 + }, + "knob": 128, + "mean_ms": 0.39038914999999946, + "median_ms": 0.3815, + "nq": 2000, + "p95_ms": 0.5117999999999999, + "p99_ms": 0.5795, + "qps_1t": 2558.1257329030254, + "qps_1t_max": 2624.3419790530297, + "qps_1t_min": 2504.957624258111, + "qps_1t_runs": [ + 2558.1257329030254, + 2504.957624258111, + 2624.3419790530297 + ], + "recall": 0.9984 + }, + { + "allthreads": { + "median_ms": 0.5193, + "qps": 1893.6487116892388, + "recall": 0.9998 + }, + "knob": 256, + "mean_ms": 0.7479780999999991, + "median_ms": 0.7308, + "nq": 2000, + "p95_ms": 0.9751, + "p99_ms": 1.1279000000000001, + "qps_1t": 1360.1006583295214, + "qps_1t_max": 1393.3581820186978, + "qps_1t_min": 1264.2160302466243, + "qps_1t_runs": [ + 1264.2160302466243, + 1360.1006583295214, + 1393.3581820186978 + ], + "recall": 0.9998 + }, + { + "allthreads": { + "median_ms": 0.7676999999999999, + "qps": 1206.453294543449, + "recall": 1.0 + }, + "knob": 1265, + "mean_ms": 1.848655512321659, + "median_ms": 1.8041, + "nq": 1799, + "p95_ms": 2.2217000000000002, + "p99_ms": 2.5569, + "qps_1t": 543.8026045514911, + "qps_1t_max": 545.0260930863368, + "qps_1t_min": 534.1054536738919, + "qps_1t_runs": [ + 545.0260930863368, + 543.8026045514911, + 534.1054536738919 + ], + "recall": 1.0 + } + ], + "system": "old_ivf", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + } + ], + "status": "ok" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-a/raw/cb_old_sift_100000.txt b/benchmarks/rust-stage-a/raw/cb_old_sift_100000.txt new file mode 100644 index 0000000..7c4504b --- /dev/null +++ b/benchmarks/rust-stage-a/raw/cb_old_sift_100000.txt @@ -0,0 +1,18 @@ +old: ds=sift n=100000 d=128 queries=2000 cpus=32 release=true + ingest 0.06s + exact 1T recall=1.0 qps=437.7 median=2.182ms + exact auto-threads qps=1030.1 median=0.846ms + exact 8 concurrent clients (scan threads 1): 487.6 qps + ivf build 0.20s nlist=Some(1265) + ivf nprobe=1 recall=0.2912 qps1t=48238 median=0.021ms allthreads_qps=48288 + ivf nprobe=2 recall=0.4356 qps1t=37905 median=0.026ms allthreads_qps=38582 + ivf nprobe=4 recall=0.5955 qps1t=28205 median=0.034ms allthreads_qps=30561 + ivf nprobe=8 recall=0.7474 qps1t=20786 median=0.047ms allthreads_qps=21162 + ivf nprobe=16 recall=0.8710 qps1t=13859 median=0.071ms allthreads_qps=13222 + ivf nprobe=32 recall=0.9486 qps1t=7691 median=0.127ms allthreads_qps=7239 + ivf nprobe=64 recall=0.9860 qps1t=4639 median=0.210ms allthreads_qps=4604 + ivf nprobe=128 recall=0.9984 qps1t=2558 median=0.382ms allthreads_qps=2569 + ivf nprobe=256 recall=0.9998 qps1t=1360 median=0.731ms allthreads_qps=1894 + ivf nprobe=1265 recall=1.0000 qps1t=544 median=1.804ms allthreads_qps=1206 + mutation: ins 0.02s upd 0.02s del 0.01s recall_after=0.98545 +wrote D:/Goldmine/ISON/ison_dev/isongraphossworking/isongraph-vector/benchmarks/rust-stage-a/raw/cb_old_sift_100000.json diff --git a/benchmarks/rust-stage-a/raw/cb_old_text384_1000000.json b/benchmarks/rust-stage-a/raw/cb_old_text384_1000000.json new file mode 100644 index 0000000..4780184 --- /dev/null +++ b/benchmarks/rust-stage-a/raw/cb_old_text384_1000000.json @@ -0,0 +1,368 @@ +{ + "label": "old", + "runs": [ + { + "all_threads": 32, + "build": { + "first_query_ms": 1109.7949, + "index_s": 0.0, + "ingest_s": 0.8167169999999998, + "ready_s": 1.9265119, + "total_s": 0.8167169999999998 + }, + "concurrent_clients": { + "clients": 8, + "qps": 14.84180320473194, + "scan_threads": 1 + }, + "d": 384, + "ds": "text384", + "k": 10, + "loop_allthreads": { + "mean_ms": 31.752071134020643, + "median_ms": 31.0402, + "nq": 97, + "p95_ms": 36.7504, + "p99_ms": 41.62, + "qps": 31.75655490487357, + "qps_runs": [ + 30.83912136481871, + 31.907792925792297, + 31.75655490487357 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 3535.3515625, + "rss_mb": 3527.73046875 + }, + "mem_after_first_query": { + "peak_mb": 3535.3515625, + "rss_mb": 3527.73046875 + }, + "mem_after_ingest": { + "peak_mb": 1983.8203125, + "rss_mb": 1942.61328125 + }, + "mem_after_sweep": { + "peak_mb": 3535.3515625, + "rss_mb": 3527.95703125 + }, + "mem_baseline": { + "peak_mb": 10.84765625, + "rss_mb": 10.84765625 + }, + "n": 1000000, + "queries": 2000, + "sweep": [ + { + "knob": null, + "mean_ms": 65.78699275362317, + "median_ms": 65.3573, + "nq": 46, + "p95_ms": 69.8196, + "p99_ms": 71.589, + "qps_1t": 15.208418481190785, + "qps_1t_max": 15.29761900209042, + "qps_1t_min": 15.097012252045936, + "qps_1t_runs": [ + 15.208418481190785, + 15.097012252045936, + 15.29761900209042 + ], + "recall": 1.0 + } + ], + "system": "old_exact", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + }, + { + "all_threads": 32, + "build": { + "cache_s": 1.1097949, + "first_query_ms": 0.0, + "index_s": 6.1597909, + "ingest_s": 0.8167169999999998, + "ready_s": 8.0863028, + "total_s": 8.0863028 + }, + "d": 384, + "ds": "text384", + "index_stats": { + "build_seconds": 6.1584683, + "index_bytes": 14176008, + "nlist": 4000, + "trained_n": 1000000 + }, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 3790.29296875, + "rss_mb": 3542.52734375 + }, + "mem_after_ingest": { + "peak_mb": 1983.8203125, + "rss_mb": 1942.61328125 + }, + "mem_after_sweep": { + "peak_mb": 3790.29296875, + "rss_mb": 3542.9296875 + }, + "mem_baseline": { + "peak_mb": 10.84765625, + "rss_mb": 10.84765625 + }, + "mutation": { + "after": { + "mean_ms": 10.589013675213677, + "median_ms": 10.2808, + "nq": 234, + "p95_ms": 13.633799999999999, + "p99_ms": 16.7765, + "qps_1t": 92.92091167644968, + "qps_1t_max": 99.06944577630755, + "qps_1t_min": 91.6483484203878, + "qps_1t_runs": [ + 91.6483484203878, + 92.92091167644968, + 99.06944577630755 + ], + "recall": 0.9521367521367521 + }, + "delete_10k": { + "per_s": 564927.0679155322, + "seconds": 0.0177014 + }, + "first_query_after_ms": 616.1346, + "index_stats_after": { + "builds": 91616, + "churn": 30000, + "pending": 0, + "trained_n": 1000000 + }, + "insert_10k": { + "per_s": 26382.655620072874, + "seconds": 0.3790369 + }, + "mem_after": { + "peak_mb": 5023.33203125, + "rss_mb": 3591.13671875 + }, + "recall_after": 0.9521367521367521, + "recall_before": 0.952054794520548, + "ref_knob": 256, + "update_10k": { + "per_s": 365428.95878326776, + "seconds": 0.0273651 + } + }, + "n": 1000000, + "queries": 2000, + "ref_knob": 256, + "ref_recall_pre": 0.952054794520548, + "sweep": [ + { + "allthreads": { + "median_ms": 0.1634, + "qps": 5981.655459038213, + "recall": 0.27565 + }, + "knob": 1, + "mean_ms": 0.19949629999999988, + "median_ms": 0.1893, + "nq": 2000, + "p95_ms": 0.2847, + "p99_ms": 0.359, + "qps_1t": 5244.008981938591, + "qps_1t_max": 5358.709322707367, + "qps_1t_min": 4521.143125827937, + "qps_1t_runs": [ + 4521.143125827937, + 5244.008981938591, + 5358.709322707367 + ], + "recall": 0.27565 + }, + { + "allthreads": { + "median_ms": 0.22469999999999998, + "qps": 4251.485575347164, + "recall": 0.3806 + }, + "knob": 2, + "mean_ms": 0.23732268333333298, + "median_ms": 0.2275, + "nq": 2000, + "p95_ms": 0.33780000000000004, + "p99_ms": 0.41, + "qps_1t": 4302.133686223014, + "qps_1t_max": 4305.688374081651, + "qps_1t_min": 4044.091107710937, + "qps_1t_runs": [ + 4305.688374081651, + 4302.133686223014, + 4044.091107710937 + ], + "recall": 0.3806 + }, + { + "allthreads": { + "median_ms": 0.3369, + "qps": 2924.5816203205763, + "recall": 0.49615 + }, + "knob": 4, + "mean_ms": 0.35731308333333434, + "median_ms": 0.3439, + "nq": 2000, + "p95_ms": 0.5071, + "p99_ms": 0.6568, + "qps_1t": 2796.7177720227564, + "qps_1t_max": 2950.2567534696095, + "qps_1t_min": 2663.6570893100256, + "qps_1t_runs": [ + 2796.7177720227564, + 2950.2567534696095, + 2663.6570893100256 + ], + "recall": 0.49615 + }, + { + "allthreads": { + "median_ms": 0.5411, + "qps": 1803.8813031671452, + "recall": 0.60455 + }, + "knob": 8, + "mean_ms": 0.5422527000000004, + "median_ms": 0.5327, + "nq": 2000, + "p95_ms": 0.722, + "p99_ms": 0.8317, + "qps_1t": 1827.2093953645353, + "qps_1t_max": 1884.311561118208, + "qps_1t_min": 1822.2316451845334, + "qps_1t_runs": [ + 1884.311561118208, + 1822.2316451845334, + 1827.2093953645353 + ], + "recall": 0.60455 + }, + { + "allthreads": { + "median_ms": 0.8752, + "qps": 1142.6856747096153, + "recall": 0.70215 + }, + "knob": 16, + "mean_ms": 0.9339536166666674, + "median_ms": 0.9103, + "nq": 2000, + "p95_ms": 1.2344000000000002, + "p99_ms": 1.5279, + "qps_1t": 1057.5694262601262, + "qps_1t_max": 1106.1307573760257, + "qps_1t_min": 1050.150791152102, + "qps_1t_runs": [ + 1050.150791152102, + 1057.5694262601262, + 1106.1307573760257 + ], + "recall": 0.70215 + }, + { + "allthreads": { + "median_ms": 0.9502999999999999, + "qps": 992.8306211600111, + "recall": 0.79095 + }, + "knob": 32, + "mean_ms": 1.6393046755555545, + "median_ms": 1.6023999999999998, + "nq": 1875, + "p95_ms": 2.076, + "p99_ms": 2.5322999999999998, + "qps_1t": 611.73810914067, + "qps_1t_max": 636.2827783194188, + "qps_1t_min": 584.2488928950888, + "qps_1t_runs": [ + 611.73810914067, + 584.2488928950888, + 636.2827783194188 + ], + "recall": 0.7905066666666667 + }, + { + "allthreads": { + "median_ms": 1.1544999999999999, + "qps": 797.5341520074589, + "recall": 0.8621 + }, + "knob": 64, + "mean_ms": 2.860732973362132, + "median_ms": 2.8205, + "nq": 926, + "p95_ms": 3.4762999999999997, + "p99_ms": 4.048699999999999, + "qps_1t": 352.48818593591943, + "qps_1t_max": 355.0968185280777, + "qps_1t_min": 341.4028404937542, + "qps_1t_runs": [ + 352.48818593591943, + 355.0968185280777, + 341.4028404937542 + ], + "recall": 0.8600431965442764 + }, + { + "allthreads": { + "median_ms": 1.8103, + "qps": 499.64214868509094, + "recall": 0.9138360175695461 + }, + "knob": 128, + "mean_ms": 5.329950824499422, + "median_ms": 5.2277000000000005, + "nq": 566, + "p95_ms": 6.3114, + "p99_ms": 7.453, + "qps_1t": 187.34865637912884, + "qps_1t_max": 190.49287102306587, + "qps_1t_min": 185.09364970176543, + "qps_1t_runs": [ + 190.49287102306587, + 185.09364970176543, + 187.34865637912884 + ], + "recall": 0.915017667844523 + }, + { + "allthreads": { + "median_ms": 3.4764, + "qps": 250.06329475339268, + "recall": 0.9531686358754028 + }, + "knob": 256, + "mean_ms": 10.023837671232886, + "median_ms": 9.8964, + "nq": 292, + "p95_ms": 11.641399999999999, + "p99_ms": 13.424999999999999, + "qps_1t": 99.90336744143785, + "qps_1t_max": 101.86168627207617, + "qps_1t_min": 97.61233868098515, + "qps_1t_runs": [ + 97.61233868098515, + 99.90336744143785, + 101.86168627207617 + ], + "recall": 0.952054794520548 + } + ], + "system": "old_ivf", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + } + ], + "status": "ok" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-a/raw/cb_old_text384_1000000.txt b/benchmarks/rust-stage-a/raw/cb_old_text384_1000000.txt new file mode 100644 index 0000000..cfbd2ea --- /dev/null +++ b/benchmarks/rust-stage-a/raw/cb_old_text384_1000000.txt @@ -0,0 +1,17 @@ +old: ds=text384 n=1000000 d=384 queries=2000 cpus=32 release=true + ingest 0.82s + exact 1T recall=1.0 qps=15.2 median=65.357ms + exact auto-threads qps=31.8 median=31.040ms + exact 8 concurrent clients (scan threads 1): 14.8 qps + ivf build 6.16s nlist=Some(4000) + ivf nprobe=1 recall=0.2757 qps1t=5244 median=0.189ms allthreads_qps=5982 + ivf nprobe=2 recall=0.3806 qps1t=4302 median=0.228ms allthreads_qps=4251 + ivf nprobe=4 recall=0.4961 qps1t=2797 median=0.344ms allthreads_qps=2925 + ivf nprobe=8 recall=0.6046 qps1t=1827 median=0.533ms allthreads_qps=1804 + ivf nprobe=16 recall=0.7022 qps1t=1058 median=0.910ms allthreads_qps=1143 + ivf nprobe=32 recall=0.7905 qps1t=612 median=1.602ms allthreads_qps=993 + ivf nprobe=64 recall=0.8600 qps1t=352 median=2.821ms allthreads_qps=798 + ivf nprobe=128 recall=0.9150 qps1t=187 median=5.228ms allthreads_qps=500 + ivf nprobe=256 recall=0.9521 qps1t=100 median=9.896ms allthreads_qps=250 + mutation: ins 0.38s upd 0.03s del 0.02s recall_after=0.9521367521367521 +wrote D:/Goldmine/ISON/ison_dev/isongraphossworking/isongraph-vector/benchmarks/rust-stage-a/raw/cb_old_text384_1000000.json diff --git a/benchmarks/rust-stage-a/raw/cb_old_text384_100000_r1.json b/benchmarks/rust-stage-a/raw/cb_old_text384_100000_r1.json new file mode 100644 index 0000000..221d3d1 --- /dev/null +++ b/benchmarks/rust-stage-a/raw/cb_old_text384_100000_r1.json @@ -0,0 +1,390 @@ +{ + "label": "old", + "runs": [ + { + "all_threads": 32, + "build": { + "first_query_ms": 81.0275, + "index_s": 0.0, + "ingest_s": 0.0566341, + "ready_s": 0.1376616, + "total_s": 0.0566341 + }, + "concurrent_clients": { + "clients": 8, + "qps": 137.82164211867973, + "scan_threads": 1 + }, + "d": 384, + "ds": "text384", + "k": 10, + "loop_allthreads": { + "mean_ms": 3.654132224168121, + "median_ms": 3.4972, + "nq": 571, + "p95_ms": 4.6046000000000005, + "p99_ms": 6.8165, + "qps": 278.0569032982855, + "qps_runs": [ + 261.05323872203144, + 278.0569032982855, + 282.8554850706548 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 349.43359375, + "rss_mb": 349.43359375 + }, + "mem_after_first_query": { + "peak_mb": 349.43359375, + "rss_mb": 349.43359375 + }, + "mem_after_ingest": { + "peak_mb": 218.8203125, + "rss_mb": 191.69140625 + }, + "mem_after_sweep": { + "peak_mb": 349.73046875, + "rss_mb": 349.609375 + }, + "mem_baseline": { + "peak_mb": 10.86328125, + "rss_mb": 10.86328125 + }, + "n": 100000, + "queries": 2000, + "sweep": [ + { + "knob": null, + "mean_ms": 7.26497219387754, + "median_ms": 7.1639, + "nq": 392, + "p95_ms": 8.105899999999998, + "p99_ms": 8.5646, + "qps_1t": 137.5727450984795, + "qps_1t_max": 138.0058791208618, + "qps_1t_min": 137.3632631063651, + "qps_1t_runs": [ + 137.5727450984795, + 138.0058791208618, + 137.3632631063651 + ], + "recall": 1.0 + } + ], + "system": "old_exact", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + }, + { + "all_threads": 32, + "build": { + "cache_s": 0.0810275, + "first_query_ms": 0.0, + "index_s": 0.4912815, + "ingest_s": 0.0566341, + "ready_s": 0.6289431, + "total_s": 0.6289431 + }, + "d": 384, + "ds": "text384", + "index_stats": { + "build_seconds": 0.4911176, + "index_bytes": 2753168, + "nlist": 1265, + "trained_n": 100000 + }, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 431.00390625, + "rss_mb": 353.24609375 + }, + "mem_after_ingest": { + "peak_mb": 218.8203125, + "rss_mb": 191.69140625 + }, + "mem_after_sweep": { + "peak_mb": 431.00390625, + "rss_mb": 353.4296875 + }, + "mem_baseline": { + "peak_mb": 10.86328125, + "rss_mb": 10.86328125 + }, + "mutation": { + "after": { + "mean_ms": 1.4987331255411247, + "median_ms": 1.4674, + "nq": 1925, + "p95_ms": 1.8986999999999998, + "p99_ms": 2.4007, + "qps_1t": 652.2508499167421, + "qps_1t_max": 713.7900605865, + "qps_1t_min": 640.1742284877944, + "qps_1t_runs": [ + 652.2508499167421, + 640.1742284877944, + 713.7900605865 + ], + "recall": 0.949038961038961 + }, + "delete_10k": { + "per_s": 840844.8809363649, + "seconds": 0.0118928 + }, + "first_query_after_ms": 224.9186, + "index_stats_after": { + "builds": 111986, + "churn": 30000, + "pending": 0, + "trained_n": 100000 + }, + "insert_10k": { + "per_s": 224393.35257132343, + "seconds": 0.0445646 + }, + "mem_after": { + "peak_mb": 515.25390625, + "rss_mb": 401.03125 + }, + "recall_after": 0.949038961038961, + "recall_before": 0.9529, + "ref_knob": 128, + "update_10k": { + "per_s": 469598.2117700096, + "seconds": 0.0212948 + } + }, + "n": 100000, + "queries": 2000, + "ref_knob": 128, + "ref_recall_pre": 0.9529, + "sweep": [ + { + "allthreads": { + "median_ms": 0.0438, + "qps": 22385.361764235167, + "recall": 0.273 + }, + "knob": 1, + "mean_ms": 0.04593181666666694, + "median_ms": 0.0449, + "nq": 2000, + "p95_ms": 0.058499999999999996, + "p99_ms": 0.0673, + "qps_1t": 21714.53636207685, + "qps_1t_max": 22634.83002374395, + "qps_1t_min": 21024.458804149777, + "qps_1t_runs": [ + 21024.458804149777, + 21714.53636207685, + 22634.83002374395 + ], + "recall": 0.273 + }, + { + "allthreads": { + "median_ms": 0.0577, + "qps": 17156.955258092086, + "recall": 0.39185 + }, + "knob": 2, + "mean_ms": 0.06017973333333311, + "median_ms": 0.0592, + "nq": 2000, + "p95_ms": 0.07669999999999999, + "p99_ms": 0.0868, + "qps_1t": 16409.300463316606, + "qps_1t_max": 17101.793294044815, + "qps_1t_min": 16359.985112413535, + "qps_1t_runs": [ + 17101.793294044815, + 16359.985112413535, + 16409.300463316606 + ], + "recall": 0.39185 + }, + { + "allthreads": { + "median_ms": 0.0826, + "qps": 11819.86991051177, + "recall": 0.51675 + }, + "knob": 4, + "mean_ms": 0.08220305000000014, + "median_ms": 0.0807, + "nq": 2000, + "p95_ms": 0.1046, + "p99_ms": 0.1173, + "qps_1t": 12143.453039445569, + "qps_1t_max": 12223.907763281588, + "qps_1t_min": 12128.069993517533, + "qps_1t_runs": [ + 12223.907763281588, + 12128.069993517533, + 12143.453039445569 + ], + "recall": 0.51675 + }, + { + "allthreads": { + "median_ms": 0.1271, + "qps": 7724.656175553638, + "recall": 0.64025 + }, + "knob": 8, + "mean_ms": 0.13137423333333373, + "median_ms": 0.1285, + "nq": 2000, + "p95_ms": 0.1698, + "p99_ms": 0.19249999999999998, + "qps_1t": 7681.969534077018, + "qps_1t_max": 7796.161559856003, + "qps_1t_min": 7370.310179503899, + "qps_1t_runs": [ + 7681.969534077018, + 7796.161559856003, + 7370.310179503899 + ], + "recall": 0.64025 + }, + { + "allthreads": { + "median_ms": 0.21230000000000002, + "qps": 4629.286719502255, + "recall": 0.74505 + }, + "knob": 16, + "mean_ms": 0.22852681666666633, + "median_ms": 0.2141, + "nq": 2000, + "p95_ms": 0.3456, + "p99_ms": 0.4809, + "qps_1t": 4715.660969440867, + "qps_1t_max": 4758.588478933846, + "qps_1t_min": 3796.87118829728, + "qps_1t_runs": [ + 4758.588478933846, + 4715.660969440867, + 3796.87118829728 + ], + "recall": 0.74505 + }, + { + "allthreads": { + "median_ms": 0.3684, + "qps": 2703.905575291058, + "recall": 0.83455 + }, + "knob": 32, + "mean_ms": 0.36945785000000025, + "median_ms": 0.364, + "nq": 2000, + "p95_ms": 0.46959999999999996, + "p99_ms": 0.5231, + "qps_1t": 2697.2946269756176, + "qps_1t_max": 2736.5591162446485, + "qps_1t_min": 2686.6604354055607, + "qps_1t_runs": [ + 2736.5591162446485, + 2686.6604354055607, + 2697.2946269756176 + ], + "recall": 0.83455 + }, + { + "allthreads": { + "median_ms": 0.6352000000000001, + "qps": 1538.530417284835, + "recall": 0.90705 + }, + "knob": 64, + "mean_ms": 0.6857617166666652, + "median_ms": 0.6777000000000001, + "nq": 2000, + "p95_ms": 0.8566999999999999, + "p99_ms": 0.9430999999999999, + "qps_1t": 1458.892423461032, + "qps_1t_max": 1460.4757295222314, + "qps_1t_min": 1455.3387464658008, + "qps_1t_runs": [ + 1460.4757295222314, + 1458.892423461032, + 1455.3387464658008 + ], + "recall": 0.90705 + }, + { + "allthreads": { + "median_ms": 0.6944, + "qps": 1395.5409675006426, + "recall": 0.9529 + }, + "knob": 128, + "mean_ms": 1.392172549999999, + "median_ms": 1.3641, + "nq": 2000, + "p95_ms": 1.7413999999999998, + "p99_ms": 2.143, + "qps_1t": 716.2167112768218, + "qps_1t_max": 731.9671118393063, + "qps_1t_min": 707.1582661214754, + "qps_1t_runs": [ + 707.1582661214754, + 731.9671118393063, + 716.2167112768218 + ], + "recall": 0.9529 + }, + { + "allthreads": { + "median_ms": 0.9401999999999999, + "qps": 1034.9483426233746, + "recall": 0.9814 + }, + "knob": 256, + "mean_ms": 2.626881895734594, + "median_ms": 2.5878, + "nq": 1055, + "p95_ms": 3.0547999999999997, + "p99_ms": 4.0882000000000005, + "qps_1t": 384.50901733782115, + "qps_1t_max": 388.65891087679086, + "qps_1t_min": 369.4158677246084, + "qps_1t_runs": [ + 388.65891087679086, + 369.4158677246084, + 384.50901733782115 + ], + "recall": 0.9808530805687203 + }, + { + "allthreads": { + "median_ms": 4.0718000000000005, + "qps": 237.84215640513614, + "recall": 0.9998589562764457 + }, + "knob": 1265, + "mean_ms": 8.08909857035365, + "median_ms": 7.5706999999999995, + "nq": 443, + "p95_ms": 11.2471, + "p99_ms": 16.4949, + "qps_1t": 117.05068009087361, + "qps_1t_max": 140.57357254280166, + "qps_1t_min": 116.14030682381454, + "qps_1t_runs": [ + 140.57357254280166, + 117.05068009087361, + 116.14030682381454 + ], + "recall": 1.0 + } + ], + "system": "old_ivf", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + } + ], + "status": "ok" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-a/raw/cb_old_text384_100000_r1.txt b/benchmarks/rust-stage-a/raw/cb_old_text384_100000_r1.txt new file mode 100644 index 0000000..90410b4 --- /dev/null +++ b/benchmarks/rust-stage-a/raw/cb_old_text384_100000_r1.txt @@ -0,0 +1,18 @@ +old: ds=text384 n=100000 d=384 queries=2000 cpus=32 release=true + ingest 0.06s + exact 1T recall=1.0 qps=137.6 median=7.164ms + exact auto-threads qps=278.1 median=3.497ms + exact 8 concurrent clients (scan threads 1): 137.8 qps + ivf build 0.49s nlist=Some(1265) + ivf nprobe=1 recall=0.2730 qps1t=21715 median=0.045ms allthreads_qps=22385 + ivf nprobe=2 recall=0.3918 qps1t=16409 median=0.059ms allthreads_qps=17157 + ivf nprobe=4 recall=0.5168 qps1t=12143 median=0.081ms allthreads_qps=11820 + ivf nprobe=8 recall=0.6402 qps1t=7682 median=0.129ms allthreads_qps=7725 + ivf nprobe=16 recall=0.7450 qps1t=4716 median=0.214ms allthreads_qps=4629 + ivf nprobe=32 recall=0.8346 qps1t=2697 median=0.364ms allthreads_qps=2704 + ivf nprobe=64 recall=0.9071 qps1t=1459 median=0.678ms allthreads_qps=1539 + ivf nprobe=128 recall=0.9529 qps1t=716 median=1.364ms allthreads_qps=1396 + ivf nprobe=256 recall=0.9809 qps1t=385 median=2.588ms allthreads_qps=1035 + ivf nprobe=1265 recall=1.0000 qps1t=117 median=7.571ms allthreads_qps=238 + mutation: ins 0.04s upd 0.02s del 0.01s recall_after=0.949038961038961 +wrote D:/Goldmine/ISON/ison_dev/isongraphossworking/isongraph-vector/benchmarks/rust-stage-a/raw/cb_old_text384_100000_r1.json diff --git a/benchmarks/rust-stage-a/raw/cb_old_text384_100000_r2.json b/benchmarks/rust-stage-a/raw/cb_old_text384_100000_r2.json new file mode 100644 index 0000000..17360f1 --- /dev/null +++ b/benchmarks/rust-stage-a/raw/cb_old_text384_100000_r2.json @@ -0,0 +1,390 @@ +{ + "label": "old", + "runs": [ + { + "all_threads": 32, + "build": { + "first_query_ms": 81.12049999999999, + "index_s": 0.0, + "ingest_s": 0.056418300000000005, + "ready_s": 0.13753880000000002, + "total_s": 0.056418300000000005 + }, + "concurrent_clients": { + "clients": 8, + "qps": 150.46587998089083, + "scan_threads": 1 + }, + "d": 384, + "ds": "text384", + "k": 10, + "loop_allthreads": { + "mean_ms": 3.9873781595092055, + "median_ms": 3.7563, + "nq": 815, + "p95_ms": 5.7321, + "p99_ms": 6.655399999999999, + "qps": 243.37166083629788, + "qps_runs": [ + 240.89563044588525, + 243.37166083629788, + 270.12299478834944 + ], + "threads": "auto" + }, + "mem_after_build": { + "peak_mb": 349.44140625, + "rss_mb": 349.44140625 + }, + "mem_after_first_query": { + "peak_mb": 349.44140625, + "rss_mb": 349.44140625 + }, + "mem_after_ingest": { + "peak_mb": 218.828125, + "rss_mb": 191.703125 + }, + "mem_after_sweep": { + "peak_mb": 349.74609375, + "rss_mb": 349.62109375 + }, + "mem_baseline": { + "peak_mb": 10.87109375, + "rss_mb": 10.87109375 + }, + "n": 100000, + "queries": 2000, + "sweep": [ + { + "knob": null, + "mean_ms": 7.242163092269331, + "median_ms": 7.0958000000000006, + "nq": 401, + "p95_ms": 8.675699999999999, + "p99_ms": 10.7294, + "qps_1t": 137.54933596257212, + "qps_1t_max": 139.29603742721267, + "qps_1t_min": 137.41142103421393, + "qps_1t_runs": [ + 137.54933596257212, + 139.29603742721267, + 137.41142103421393 + ], + "recall": 1.0 + } + ], + "system": "old_exact", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + }, + { + "all_threads": 32, + "build": { + "cache_s": 0.0811205, + "first_query_ms": 0.0, + "index_s": 0.4834523, + "ingest_s": 0.056418300000000005, + "ready_s": 0.6209911, + "total_s": 0.6209911 + }, + "d": 384, + "ds": "text384", + "index_stats": { + "build_seconds": 0.4832773, + "index_bytes": 2753168, + "nlist": 1265, + "trained_n": 100000 + }, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 431.015625, + "rss_mb": 353.2578125 + }, + "mem_after_ingest": { + "peak_mb": 218.828125, + "rss_mb": 191.703125 + }, + "mem_after_sweep": { + "peak_mb": 431.015625, + "rss_mb": 353.35546875 + }, + "mem_baseline": { + "peak_mb": 10.87109375, + "rss_mb": 10.87109375 + }, + "mutation": { + "after": { + "mean_ms": 1.4837090999999967, + "median_ms": 1.4388, + "nq": 2000, + "p95_ms": 1.9863, + "p99_ms": 2.5242, + "qps_1t": 669.6601816105027, + "qps_1t_max": 695.380842355145, + "qps_1t_min": 657.9935657441184, + "qps_1t_runs": [ + 669.6601816105027, + 657.9935657441184, + 695.380842355145 + ], + "recall": 0.94895 + }, + "delete_10k": { + "per_s": 1012514.6814628812, + "seconds": 0.0098764 + }, + "first_query_after_ms": 193.76940000000002, + "index_stats_after": { + "builds": 112442, + "churn": 30000, + "pending": 0, + "trained_n": 100000 + }, + "insert_10k": { + "per_s": 252573.72627069842, + "seconds": 0.0395924 + }, + "mem_after": { + "peak_mb": 514.65234375, + "rss_mb": 400.84765625 + }, + "recall_after": 0.94895, + "recall_before": 0.9529, + "ref_knob": 128, + "update_10k": { + "per_s": 562085.1109274967, + "seconds": 0.0177909 + } + }, + "n": 100000, + "queries": 2000, + "ref_knob": 128, + "ref_recall_pre": 0.9529, + "sweep": [ + { + "allthreads": { + "median_ms": 0.0414, + "qps": 23615.762104258924, + "recall": 0.273 + }, + "knob": 1, + "mean_ms": 0.04401364999999977, + "median_ms": 0.0424, + "nq": 2000, + "p95_ms": 0.0583, + "p99_ms": 0.0768, + "qps_1t": 23589.217368740698, + "qps_1t_max": 23871.24803658986, + "qps_1t_min": 20939.20719973701, + "qps_1t_runs": [ + 20939.20719973701, + 23871.24803658986, + 23589.217368740698 + ], + "recall": 0.273 + }, + { + "allthreads": { + "median_ms": 0.0566, + "qps": 17522.003255588195, + "recall": 0.39185 + }, + "knob": 2, + "mean_ms": 0.05769310000000008, + "median_ms": 0.0566, + "nq": 2000, + "p95_ms": 0.073, + "p99_ms": 0.0828, + "qps_1t": 17248.60588142963, + "qps_1t_max": 17608.532390455097, + "qps_1t_min": 17148.84944081886, + "qps_1t_runs": [ + 17148.84944081886, + 17608.532390455097, + 17248.60588142963 + ], + "recall": 0.39185 + }, + { + "allthreads": { + "median_ms": 0.0791, + "qps": 12374.637191105843, + "recall": 0.51675 + }, + "knob": 4, + "mean_ms": 0.08151221666666703, + "median_ms": 0.0798, + "nq": 2000, + "p95_ms": 0.1038, + "p99_ms": 0.11739999999999999, + "qps_1t": 12291.39203087599, + "qps_1t_max": 12366.855343656349, + "qps_1t_min": 12148.070400497618, + "qps_1t_runs": [ + 12366.855343656349, + 12291.39203087599, + 12148.070400497618 + ], + "recall": 0.51675 + }, + { + "allthreads": { + "median_ms": 0.122, + "qps": 7999.049712894101, + "recall": 0.64025 + }, + "knob": 8, + "mean_ms": 0.13717311666666707, + "median_ms": 0.1292, + "nq": 2000, + "p95_ms": 0.1991, + "p99_ms": 0.2763, + "qps_1t": 7178.495810809307, + "qps_1t_max": 8022.221553703758, + "qps_1t_min": 6776.8744580618095, + "qps_1t_runs": [ + 7178.495810809307, + 6776.8744580618095, + 8022.221553703758 + ], + "recall": 0.64025 + }, + { + "allthreads": { + "median_ms": 0.2016, + "qps": 4868.129669449124, + "recall": 0.74505 + }, + "knob": 16, + "mean_ms": 0.20576109999999936, + "median_ms": 0.2031, + "nq": 2000, + "p95_ms": 0.2606, + "p99_ms": 0.2977, + "qps_1t": 4813.685114232353, + "qps_1t_max": 5019.276531519304, + "qps_1t_min": 4754.877731885286, + "qps_1t_runs": [ + 5019.276531519304, + 4813.685114232353, + 4754.877731885286 + ], + "recall": 0.74505 + }, + { + "allthreads": { + "median_ms": 0.35409999999999997, + "qps": 2809.442536364726, + "recall": 0.83455 + }, + "knob": 32, + "mean_ms": 0.3553311000000002, + "median_ms": 0.34959999999999997, + "nq": 2000, + "p95_ms": 0.4543, + "p99_ms": 0.5216, + "qps_1t": 2849.5114583842374, + "qps_1t_max": 2851.708722478696, + "qps_1t_min": 2744.319943796319, + "qps_1t_runs": [ + 2744.319943796319, + 2849.5114583842374, + 2851.708722478696 + ], + "recall": 0.83455 + }, + { + "allthreads": { + "median_ms": 0.5820000000000001, + "qps": 1735.2799240780332, + "recall": 0.90705 + }, + "knob": 64, + "mean_ms": 0.6529522499999993, + "median_ms": 0.6449999999999999, + "nq": 2000, + "p95_ms": 0.8205, + "p99_ms": 0.9077, + "qps_1t": 1555.7606393802853, + "qps_1t_max": 1573.0386214015361, + "qps_1t_min": 1469.7840688781992, + "qps_1t_runs": [ + 1469.7840688781992, + 1555.7606393802853, + 1573.0386214015361 + ], + "recall": 0.90705 + }, + { + "allthreads": { + "median_ms": 0.669, + "qps": 1476.2920415679591, + "recall": 0.9529 + }, + "knob": 128, + "mean_ms": 1.2698241833333341, + "median_ms": 1.2642, + "nq": 2000, + "p95_ms": 1.5264, + "p99_ms": 1.6646, + "qps_1t": 783.6909217047687, + "qps_1t_max": 797.9451953685034, + "qps_1t_min": 781.10329903665, + "qps_1t_runs": [ + 797.9451953685034, + 781.10329903665, + 783.6909217047687 + ], + "recall": 0.9529 + }, + { + "allthreads": { + "median_ms": 0.9357, + "qps": 1030.1644513323527, + "recall": 0.9814 + }, + "knob": 256, + "mean_ms": 2.6000160777385175, + "median_ms": 2.5725000000000002, + "nq": 1132, + "p95_ms": 3.0794, + "p99_ms": 3.7182, + "qps_1t": 385.55640318927396, + "qps_1t_max": 390.45938028991407, + "qps_1t_min": 378.0277955956952, + "qps_1t_runs": [ + 385.55640318927396, + 390.45938028991407, + 378.0277955956952 + ], + "recall": 0.9807420494699647 + }, + { + "allthreads": { + "median_ms": 3.9173000000000004, + "qps": 231.7361083914122, + "recall": 0.9998851894374282 + }, + "knob": 1265, + "mean_ms": 6.651653205128204, + "median_ms": 6.4808, + "nq": 468, + "p95_ms": 7.6102, + "p99_ms": 9.4851, + "qps_1t": 150.49222534011233, + "qps_1t_max": 152.4153286440869, + "qps_1t_min": 148.16838234889062, + "qps_1t_runs": [ + 148.16838234889062, + 150.49222534011233, + 152.4153286440869 + ], + "recall": 1.0 + } + ], + "system": "old_ivf", + "version": "isongraph-vector-rs (repo HEAD, cargo release)" + } + ], + "status": "ok" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-a/raw/cb_old_text384_100000_r2.txt b/benchmarks/rust-stage-a/raw/cb_old_text384_100000_r2.txt new file mode 100644 index 0000000..c478aee --- /dev/null +++ b/benchmarks/rust-stage-a/raw/cb_old_text384_100000_r2.txt @@ -0,0 +1,18 @@ +old: ds=text384 n=100000 d=384 queries=2000 cpus=32 release=true + ingest 0.06s + exact 1T recall=1.0 qps=137.5 median=7.096ms + exact auto-threads qps=243.4 median=3.756ms + exact 8 concurrent clients (scan threads 1): 150.5 qps + ivf build 0.48s nlist=Some(1265) + ivf nprobe=1 recall=0.2730 qps1t=23589 median=0.042ms allthreads_qps=23616 + ivf nprobe=2 recall=0.3918 qps1t=17249 median=0.057ms allthreads_qps=17522 + ivf nprobe=4 recall=0.5168 qps1t=12291 median=0.080ms allthreads_qps=12375 + ivf nprobe=8 recall=0.6402 qps1t=7178 median=0.129ms allthreads_qps=7999 + ivf nprobe=16 recall=0.7450 qps1t=4814 median=0.203ms allthreads_qps=4868 + ivf nprobe=32 recall=0.8346 qps1t=2850 median=0.350ms allthreads_qps=2809 + ivf nprobe=64 recall=0.9071 qps1t=1556 median=0.645ms allthreads_qps=1735 + ivf nprobe=128 recall=0.9529 qps1t=784 median=1.264ms allthreads_qps=1476 + ivf nprobe=256 recall=0.9807 qps1t=386 median=2.573ms allthreads_qps=1030 + ivf nprobe=1265 recall=1.0000 qps1t=150 median=6.481ms allthreads_qps=232 + mutation: ins 0.04s upd 0.02s del 0.01s recall_after=0.94895 +wrote D:/Goldmine/ISON/ison_dev/isongraphossworking/isongraph-vector/benchmarks/rust-stage-a/raw/cb_old_text384_100000_r2.json diff --git a/benchmarks/rust-stage-a/raw/eb_new_1000000_deletes.json b/benchmarks/rust-stage-a/raw/eb_new_1000000_deletes.json new file mode 100644 index 0000000..d7b5a53 --- /dev/null +++ b/benchmarks/rust-stage-a/raw/eb_new_1000000_deletes.json @@ -0,0 +1,20 @@ +{ + "d": 384, + "deletes": { + "after_exact_search": { + "ns_each": 564.69, + "per_s": 1770883.1394216297 + }, + "ivf_built": { + "ns_each": 3550.09, + "per_s": 281682.99958592595 + }, + "never_searched": { + "ns_each": 484.87, + "per_s": 2062408.4806236723 + } + }, + "label": "new", + "n": 1000000, + "nprobe": 128 +} \ No newline at end of file diff --git a/benchmarks/rust-stage-a/raw/eb_new_1000000_deletes.txt b/benchmarks/rust-stage-a/raw/eb_new_1000000_deletes.txt new file mode 100644 index 0000000..b97dc29 --- /dev/null +++ b/benchmarks/rust-stage-a/raw/eb_new_1000000_deletes.txt @@ -0,0 +1,5 @@ +new: n=1000000 d=384 queries=2000 nprobe=128 has_batch=true + delete 10000 (never_searched): 2062408/s = 485 ns each + delete 10000 (after_exact_search): 1770883/s = 565 ns each + delete 10000 (ivf_built): 281683/s = 3550 ns each +wrote D:/Goldmine/ISON/ison_dev/isongraphossworking/isongraph-vector/benchmarks/rust-stage-a/raw/eb_new_1000000_deletes.json diff --git a/benchmarks/rust-stage-a/raw/eb_new_1000000_mem.json b/benchmarks/rust-stage-a/raw/eb_new_1000000_mem.json new file mode 100644 index 0000000..9eedc1f --- /dev/null +++ b/benchmarks/rust-stage-a/raw/eb_new_1000000_mem.json @@ -0,0 +1,15 @@ +{ + "d": 384, + "label": "new", + "mem": { + "after_first_query_mb": 1621.90625, + "after_ingest_mb": 1621.296875, + "after_ivf_build_mb": 1640.859375, + "ingest_s": 1.1243485000000002, + "peak_after_build_mb": 1884.359375, + "peak_after_ingest_mb": 1648.5078125, + "raw_mb": 1464.84375 + }, + "n": 1000000, + "nprobe": 128 +} \ No newline at end of file diff --git a/benchmarks/rust-stage-a/raw/eb_new_1000000_mem.txt b/benchmarks/rust-stage-a/raw/eb_new_1000000_mem.txt new file mode 100644 index 0000000..de36c45 --- /dev/null +++ b/benchmarks/rust-stage-a/raw/eb_new_1000000_mem.txt @@ -0,0 +1,3 @@ +new: n=1000000 d=384 queries=2000 nprobe=128 has_batch=true + mem raw=1465 MB: after ingest 1621 (1.11x raw, peak 1649), after first query 1622, after IVF build 1641 (1.12x raw, peak 1884); ingest 1.1s +wrote D:/Goldmine/ISON/ison_dev/isongraphossworking/isongraph-vector/benchmarks/rust-stage-a/raw/eb_new_1000000_mem.json diff --git a/benchmarks/rust-stage-a/raw/eb_new_1000000_writes_r1.json b/benchmarks/rust-stage-a/raw/eb_new_1000000_writes_r1.json new file mode 100644 index 0000000..db3da31 --- /dev/null +++ b/benchmarks/rust-stage-a/raw/eb_new_1000000_writes_r1.json @@ -0,0 +1,45 @@ +{ + "d": 384, + "label": "new", + "n": 1000000, + "nprobe": 128, + "writes": { + "after_30k_inserts": { + "first_ms": 4.9426000000000005, + "median_ms": 4.5267, + "p99_ms": 9.5967 + }, + "after_all": { + "first_ms": 4.6946, + "median_ms": 4.3962, + "p99_ms": 8.350200000000001 + }, + "before": { + "first_ms": 2.0964, + "median_ms": 1.5231999999999999, + "p99_ms": 2.0195 + }, + "before_1t": { + "first_ms": 4.3345, + "median_ms": 4.1154, + "p99_ms": 6.0178 + }, + "delete_10k": { + "per_s": 1083106.783497785 + }, + "insert_30k": { + "max_ms": 3.6654999999999998, + "median_ms": 0.001, + "p99_ms": 0.007500000000000001, + "per_s": 119198.22505896339, + "seconds": 0.2516816 + }, + "peak_mb": 1883.875, + "rss_after_mb": 1691.69140625, + "rss_built_mb": 1640.37109375, + "update_10k": { + "max_ms": 1.6418, + "per_s": 142181.77940496925 + } + } +} \ No newline at end of file diff --git a/benchmarks/rust-stage-a/raw/eb_new_1000000_writes_r1.txt b/benchmarks/rust-stage-a/raw/eb_new_1000000_writes_r1.txt new file mode 100644 index 0000000..a5fb076 --- /dev/null +++ b/benchmarks/rust-stage-a/raw/eb_new_1000000_writes_r1.txt @@ -0,0 +1,9 @@ +new: n=1000000 d=384 queries=2000 nprobe=128 has_batch=true + built: first query 2.10 ms, then median 1.523 ms p99 2.019 ms + built (1 thread): first query 4.33 ms, then median 4.115 ms p99 6.018 ms + insert 30k: 0.25s = 119198/s, per-write median 0.0010 ms p99 0.0075 ms max 3.665 ms + after 30k inserts (1 thread): first query 4.94 ms, then median 4.527 ms p99 9.597 ms + update 10k: 142182/s max 1.642 ms + delete 10k: 1083107/s + after 30k writes total (1 thread): first query 4.69 ms, then median 4.396 ms p99 8.350 ms +wrote D:/Goldmine/ISON/ison_dev/isongraphossworking/isongraph-vector/benchmarks/rust-stage-a/raw/eb_new_1000000_writes_r1.json diff --git a/benchmarks/rust-stage-a/raw/eb_new_1000000_writes_r2.json b/benchmarks/rust-stage-a/raw/eb_new_1000000_writes_r2.json new file mode 100644 index 0000000..166a094 --- /dev/null +++ b/benchmarks/rust-stage-a/raw/eb_new_1000000_writes_r2.json @@ -0,0 +1,45 @@ +{ + "d": 384, + "label": "new", + "n": 1000000, + "nprobe": 128, + "writes": { + "after_30k_inserts": { + "first_ms": 5.1708, + "median_ms": 4.3844, + "p99_ms": 8.286999999999999 + }, + "after_all": { + "first_ms": 5.0542, + "median_ms": 4.129, + "p99_ms": 5.5055 + }, + "before": { + "first_ms": 2.0494, + "median_ms": 1.6974, + "p99_ms": 2.1645000000000003 + }, + "before_1t": { + "first_ms": 4.2712, + "median_ms": 4.3532, + "p99_ms": 10.0106 + }, + "delete_10k": { + "per_s": 994243.3311128566 + }, + "insert_30k": { + "max_ms": 2.9027, + "median_ms": 0.0009, + "p99_ms": 0.0065, + "per_s": 133099.84662461007, + "seconds": 0.2253947 + }, + "peak_mb": 1884.51171875, + "rss_after_mb": 1691.078125, + "rss_built_mb": 1641.1328125, + "update_10k": { + "max_ms": 2.0961, + "per_s": 109151.00167874241 + } + } +} \ No newline at end of file diff --git a/benchmarks/rust-stage-a/raw/eb_new_1000000_writes_r2.txt b/benchmarks/rust-stage-a/raw/eb_new_1000000_writes_r2.txt new file mode 100644 index 0000000..e1b5703 --- /dev/null +++ b/benchmarks/rust-stage-a/raw/eb_new_1000000_writes_r2.txt @@ -0,0 +1,9 @@ +new: n=1000000 d=384 queries=2000 nprobe=128 has_batch=true + built: first query 2.05 ms, then median 1.697 ms p99 2.165 ms + built (1 thread): first query 4.27 ms, then median 4.353 ms p99 10.011 ms + insert 30k: 0.23s = 133100/s, per-write median 0.0009 ms p99 0.0065 ms max 2.903 ms + after 30k inserts (1 thread): first query 5.17 ms, then median 4.384 ms p99 8.287 ms + update 10k: 109151/s max 2.096 ms + delete 10k: 994243/s + after 30k writes total (1 thread): first query 5.05 ms, then median 4.129 ms p99 5.505 ms +wrote D:/Goldmine/ISON/ison_dev/isongraphossworking/isongraph-vector/benchmarks/rust-stage-a/raw/eb_new_1000000_writes_r2.json diff --git a/benchmarks/rust-stage-a/raw/eb_new_100000_scaling,loop,batch.json b/benchmarks/rust-stage-a/raw/eb_new_100000_scaling,loop,batch.json new file mode 100644 index 0000000..26578a4 --- /dev/null +++ b/benchmarks/rust-stage-a/raw/eb_new_100000_scaling,loop,batch.json @@ -0,0 +1,369 @@ +{ + "batch": { + "exact": [ + { + "equals_single": true, + "qps": 1216.6461773311685, + "recall": 0.99995, + "runs": [ + 1210.2681815109388, + 1216.6461773311685, + 1228.3915937969418 + ], + "threads": 1 + }, + { + "equals_single": true, + "qps": 2432.3446379388115, + "recall": 0.99995, + "runs": [ + 2432.3446379388115, + 2442.0680408997555, + 2403.1402314031793 + ], + "threads": 2 + }, + { + "equals_single": true, + "qps": 4647.926536732331, + "recall": 0.99995, + "runs": [ + 4680.917094639016, + 4547.399592280153, + 4647.926536732331 + ], + "threads": 4 + }, + { + "equals_single": true, + "qps": 8269.475752449936, + "recall": 0.99995, + "runs": [ + 7833.469831545148, + 8353.193049809255, + 8269.475752449936 + ], + "threads": 8 + }, + { + "equals_single": true, + "qps": 11514.356675620998, + "recall": 0.99995, + "runs": [ + 11362.235451651699, + 11514.356675620998, + 11574.697017837187 + ], + "threads": 16 + }, + { + "equals_single": true, + "qps": 13135.521141949666, + "recall": 0.99995, + "runs": [ + 13223.944861439504, + 13135.521141949666, + 12707.214966049498 + ], + "threads": 24 + }, + { + "equals_single": true, + "qps": 13553.386110218848, + "recall": 0.99995, + "runs": [ + 13620.637990683485, + 13450.360436033785, + 13553.386110218848 + ], + "threads": 32 + } + ], + "ivf": [ + { + "equals_single": true, + "qps": 8005.994888972863, + "recall": 0.9548, + "runs": [ + 7947.464083422941, + 8006.440380642189, + 8005.994888972863 + ], + "threads": 1 + }, + { + "equals_single": true, + "qps": 15497.292622978766, + "recall": 0.9548, + "runs": [ + 15438.66257953806, + 15538.50636107604, + 15497.292622978766 + ], + "threads": 2 + }, + { + "equals_single": true, + "qps": 28463.956803099154, + "recall": 0.9548, + "runs": [ + 28463.956803099154, + 28352.86845970207, + 28633.522410026315 + ], + "threads": 4 + }, + { + "equals_single": true, + "qps": 45111.90006811897, + "recall": 0.9548, + "runs": [ + 45111.90006811897, + 44687.644536600295, + 45512.263279340616 + ], + "threads": 8 + }, + { + "equals_single": true, + "qps": 56781.564161747956, + "recall": 0.9548, + "runs": [ + 57163.76274751909, + 56703.96842723038, + 56781.564161747956 + ], + "threads": 16 + }, + { + "equals_single": true, + "qps": 59332.03989486363, + "recall": 0.9548, + "runs": [ + 57075.18228386341, + 59332.03989486363, + 60362.233764822704 + ], + "threads": 24 + }, + { + "equals_single": true, + "qps": 57944.47760156218, + "recall": 0.9548, + "runs": [ + 57944.47760156218, + 57102.722086761874, + 60104.46155418117 + ], + "threads": 32 + } + ] + }, + "d": 384, + "label": "new", + "loop": [ + { + "qps": 143.9961800693351, + "runs": [ + 143.9961800693351, + 139.2615812484095, + 147.5936479236181, + 140.23063639280434, + 146.3436335836395 + ], + "threads": 1 + }, + { + "qps": 234.05763247896056, + "runs": [ + 215.02845328169974, + 234.79888536273143, + 217.1907334270439, + 234.05763247896056, + 237.23970652499344 + ], + "threads": 2 + }, + { + "qps": 271.6661447658174, + "runs": [ + 271.6661447658174, + 286.7384060670788, + 273.98095978452676, + 262.7527024115443, + 256.92353253414126 + ], + "threads": 4 + }, + { + "qps": 243.77008970251762, + "runs": [ + 288.3744715537809, + 243.77008970251762, + 240.34240139872867, + 255.14569329378463, + 236.61646033448417 + ], + "threads": 8 + }, + { + "qps": 282.25948340116656, + "runs": [ + 264.62629210402775, + 295.6452635263621, + 282.25948340116656, + 285.87295881942856, + 224.881318883959 + ], + "threads": 0 + } + ], + "n": 100000, + "nprobe": 128, + "scaling": { + "exact": [ + { + "clients": 1, + "efficiency": 1.0, + "qps": 156.4020240152313, + "runs": [ + 157.78247056724462, + 156.4020240152313, + 155.7264975523488 + ] + }, + { + "clients": 2, + "efficiency": 0.761953705462014, + "qps": 238.34220348032878, + "runs": [ + 238.34220348032878, + 232.35451142211897, + 244.0445077334679 + ] + }, + { + "clients": 4, + "efficiency": 0.7193072277088778, + "qps": 450.00442520981335, + "runs": [ + 440.322733949222, + 450.00442520981335, + 495.3427311263366 + ] + }, + { + "clients": 8, + "efficiency": 0.752865097918202, + "qps": 941.9970009986566, + "runs": [ + 936.1170940769736, + 941.9970009986566, + 953.0327785627338 + ] + }, + { + "clients": 16, + "efficiency": 0.43532302754991625, + "qps": 1089.366441747923, + "runs": [ + 1093.5145254570482, + 1089.366441747923, + 1036.5216850223837 + ] + }, + { + "clients": 24, + "efficiency": 0.3756244714017275, + "qps": 1409.962262325157, + "runs": [ + 1426.8691174341093, + 1392.7848313708537, + 1409.962262325157 + ] + }, + { + "clients": 32, + "efficiency": 0.23549284484781297, + "qps": 1178.609842409689, + "runs": [ + 1090.1976011284041, + 1178.609842409689, + 1198.8967386356746 + ] + } + ], + "ivf": [ + { + "clients": 1, + "efficiency": 1.0, + "qps": 1017.1954339286419, + "runs": [ + 1017.1954339286419, + 1016.9908786048111, + 1027.9743989390372 + ] + }, + { + "clients": 2, + "efficiency": 0.8248141655992336, + "qps": 1677.9944061744063, + "runs": [ + 1622.5554802361082, + 1700.251068630458, + 1677.9944061744063 + ] + }, + { + "clients": 4, + "efficiency": 0.6100342226638981, + "qps": 2482.096103335702, + "runs": [ + 2482.096103335702, + 2588.9748982042456, + 2444.517237327745 + ] + }, + { + "clients": 8, + "efficiency": 0.3895956076309213, + "qps": 3170.3589852866235, + "runs": [ + 3232.769566340298, + 3170.3589852866235, + 2979.4422151088665 + ] + }, + { + "clients": 16, + "efficiency": 0.2048013635501372, + "qps": 3333.1681898489496, + "runs": [ + 3528.339344701413, + 3333.1681898489496, + 3243.3868670301285 + ] + }, + { + "clients": 24, + "efficiency": 0.15295789777937088, + "qps": 3734.113804908, + "runs": [ + 3779.8318964769965, + 3554.859224199541, + 3734.113804908 + ] + }, + { + "clients": 32, + "efficiency": 0.1207225850729783, + "qps": 3929.5507938654578, + "runs": [ + 3705.4007793761607, + 3947.4681155797234, + 3929.5507938654578 + ] + } + ] + } +} \ No newline at end of file diff --git a/benchmarks/rust-stage-a/raw/eb_new_100000_scaling,loop,batch.txt b/benchmarks/rust-stage-a/raw/eb_new_100000_scaling,loop,batch.txt new file mode 100644 index 0000000..1041ce0 --- /dev/null +++ b/benchmarks/rust-stage-a/raw/eb_new_100000_scaling,loop,batch.txt @@ -0,0 +1,35 @@ +new: n=100000 d=384 queries=2000 nprobe=128 has_batch=true + scaling exact clients= 1 qps= 156.4 efficiency=1.00 + scaling exact clients= 2 qps= 238.3 efficiency=0.76 + scaling exact clients= 4 qps= 450.0 efficiency=0.72 + scaling exact clients= 8 qps= 942.0 efficiency=0.75 + scaling exact clients=16 qps= 1089.4 efficiency=0.44 + scaling exact clients=24 qps= 1410.0 efficiency=0.38 + scaling exact clients=32 qps= 1178.6 efficiency=0.24 + scaling ivf clients= 1 qps= 1017.2 efficiency=1.00 + scaling ivf clients= 2 qps= 1678.0 efficiency=0.82 + scaling ivf clients= 4 qps= 2482.1 efficiency=0.61 + scaling ivf clients= 8 qps= 3170.4 efficiency=0.39 + scaling ivf clients=16 qps= 3333.2 efficiency=0.20 + scaling ivf clients=24 qps= 3734.1 efficiency=0.15 + scaling ivf clients=32 qps= 3929.6 efficiency=0.12 + exact loop threads=1 qps=144.0 (min 139.3 max 147.6) + exact loop threads=2 qps=234.1 (min 215.0 max 237.2) + exact loop threads=4 qps=271.7 (min 256.9 max 286.7) + exact loop threads=8 qps=243.8 (min 236.6 max 288.4) + exact loop threads=0 qps=282.3 (min 224.9 max 295.6) + batch exact threads= 1 qps= 1216.6 recall=1.0000 equals_single(first200)=true + batch exact threads= 2 qps= 2432.3 recall=1.0000 equals_single(first200)=true + batch exact threads= 4 qps= 4647.9 recall=1.0000 equals_single(first200)=true + batch exact threads= 8 qps= 8269.5 recall=1.0000 equals_single(first200)=true + batch exact threads=16 qps= 11514.4 recall=1.0000 equals_single(first200)=true + batch exact threads=24 qps= 13135.5 recall=1.0000 equals_single(first200)=true + batch exact threads=32 qps= 13553.4 recall=1.0000 equals_single(first200)=true + batch ivf threads= 1 qps= 8006.0 recall=0.9548 equals_single(first200)=true + batch ivf threads= 2 qps= 15497.3 recall=0.9548 equals_single(first200)=true + batch ivf threads= 4 qps= 28464.0 recall=0.9548 equals_single(first200)=true + batch ivf threads= 8 qps= 45111.9 recall=0.9548 equals_single(first200)=true + batch ivf threads=16 qps= 56781.6 recall=0.9548 equals_single(first200)=true + batch ivf threads=24 qps= 59332.0 recall=0.9548 equals_single(first200)=true + batch ivf threads=32 qps= 57944.5 recall=0.9548 equals_single(first200)=true +wrote D:/Goldmine/ISON/ison_dev/isongraphossworking/isongraph-vector/benchmarks/rust-stage-a/raw/eb_new_100000_scaling,loop,batch.json diff --git a/benchmarks/rust-stage-a/raw/eb_old_1000000_deletes.json b/benchmarks/rust-stage-a/raw/eb_old_1000000_deletes.json new file mode 100644 index 0000000..e6478bf --- /dev/null +++ b/benchmarks/rust-stage-a/raw/eb_old_1000000_deletes.json @@ -0,0 +1,20 @@ +{ + "d": 384, + "deletes": { + "after_exact_search": { + "ns_each": 1868.3200000000002, + "per_s": 535240.215808855 + }, + "ivf_built": { + "ns_each": 1761.85, + "per_s": 567585.2087294606 + }, + "never_searched": { + "ns_each": 1022.37, + "per_s": 978119.4675117618 + } + }, + "label": "old", + "n": 1000000, + "nprobe": 128 +} \ No newline at end of file diff --git a/benchmarks/rust-stage-a/raw/eb_old_1000000_deletes.txt b/benchmarks/rust-stage-a/raw/eb_old_1000000_deletes.txt new file mode 100644 index 0000000..84fcf08 --- /dev/null +++ b/benchmarks/rust-stage-a/raw/eb_old_1000000_deletes.txt @@ -0,0 +1,5 @@ +old: n=1000000 d=384 queries=2000 nprobe=128 has_batch=false + delete 10000 (never_searched): 978119/s = 1022 ns each + delete 10000 (after_exact_search): 535240/s = 1868 ns each + delete 10000 (ivf_built): 567585/s = 1762 ns each +wrote D:/Goldmine/ISON/ison_dev/isongraphossworking/isongraph-vector/benchmarks/rust-stage-a/raw/eb_old_1000000_deletes.json diff --git a/benchmarks/rust-stage-a/raw/eb_old_1000000_mem.json b/benchmarks/rust-stage-a/raw/eb_old_1000000_mem.json new file mode 100644 index 0000000..feb3e6e --- /dev/null +++ b/benchmarks/rust-stage-a/raw/eb_old_1000000_mem.json @@ -0,0 +1,15 @@ +{ + "d": 384, + "label": "old", + "mem": { + "after_first_query_mb": 3516.01171875, + "after_ingest_mb": 1931.70703125, + "after_ivf_build_mb": 3530.66015625, + "ingest_s": 0.7871528000000002, + "peak_after_build_mb": 3778.43359375, + "peak_after_ingest_mb": 1972.95703125, + "raw_mb": 1464.84375 + }, + "n": 1000000, + "nprobe": 128 +} \ No newline at end of file diff --git a/benchmarks/rust-stage-a/raw/eb_old_1000000_mem.txt b/benchmarks/rust-stage-a/raw/eb_old_1000000_mem.txt new file mode 100644 index 0000000..4c4b9d7 --- /dev/null +++ b/benchmarks/rust-stage-a/raw/eb_old_1000000_mem.txt @@ -0,0 +1,3 @@ +old: n=1000000 d=384 queries=2000 nprobe=128 has_batch=false + mem raw=1465 MB: after ingest 1932 (1.32x raw, peak 1973), after first query 3516, after IVF build 3531 (2.41x raw, peak 3778); ingest 0.8s +wrote D:/Goldmine/ISON/ison_dev/isongraphossworking/isongraph-vector/benchmarks/rust-stage-a/raw/eb_old_1000000_mem.json diff --git a/benchmarks/rust-stage-a/raw/eb_old_1000000_writes_r1.json b/benchmarks/rust-stage-a/raw/eb_old_1000000_writes_r1.json new file mode 100644 index 0000000..d6b876c --- /dev/null +++ b/benchmarks/rust-stage-a/raw/eb_old_1000000_writes_r1.json @@ -0,0 +1,45 @@ +{ + "d": 384, + "label": "old", + "n": 1000000, + "nprobe": 128, + "writes": { + "after_30k_inserts": { + "first_ms": 959.1622, + "median_ms": 5.3058000000000005, + "p99_ms": 6.7501999999999995 + }, + "after_all": { + "first_ms": 7.3589, + "median_ms": 6.5307, + "p99_ms": 7.9989 + }, + "before": { + "first_ms": 3.091, + "median_ms": 1.9763, + "p99_ms": 4.7564 + }, + "before_1t": { + "first_ms": 5.5753, + "median_ms": 5.459300000000001, + "p99_ms": 11.073400000000001 + }, + "delete_10k": { + "per_s": 619547.9777954004 + }, + "insert_30k": { + "max_ms": 396.6751, + "median_ms": 0.001, + "p99_ms": 0.0042, + "per_s": 68948.25640201797, + "seconds": 0.4351089 + }, + "peak_mb": 3778.39453125, + "rss_after_mb": 3641.09375, + "rss_built_mb": 3530.53515625, + "update_10k": { + "max_ms": 0.17340000000000003, + "per_s": 439431.0247092065 + } + } +} \ No newline at end of file diff --git a/benchmarks/rust-stage-a/raw/eb_old_1000000_writes_r1.txt b/benchmarks/rust-stage-a/raw/eb_old_1000000_writes_r1.txt new file mode 100644 index 0000000..08ee66e --- /dev/null +++ b/benchmarks/rust-stage-a/raw/eb_old_1000000_writes_r1.txt @@ -0,0 +1,9 @@ +old: n=1000000 d=384 queries=2000 nprobe=128 has_batch=false + built: first query 3.09 ms, then median 1.976 ms p99 4.756 ms + built (1 thread): first query 5.58 ms, then median 5.459 ms p99 11.073 ms + insert 30k: 0.44s = 68948/s, per-write median 0.0010 ms p99 0.0042 ms max 396.675 ms + after 30k inserts (1 thread): first query 959.16 ms, then median 5.306 ms p99 6.750 ms + update 10k: 439431/s max 0.173 ms + delete 10k: 619548/s + after 30k writes total (1 thread): first query 7.36 ms, then median 6.531 ms p99 7.999 ms +wrote D:/Goldmine/ISON/ison_dev/isongraphossworking/isongraph-vector/benchmarks/rust-stage-a/raw/eb_old_1000000_writes_r1.json diff --git a/benchmarks/rust-stage-a/raw/eb_old_1000000_writes_r2.json b/benchmarks/rust-stage-a/raw/eb_old_1000000_writes_r2.json new file mode 100644 index 0000000..55e97be --- /dev/null +++ b/benchmarks/rust-stage-a/raw/eb_old_1000000_writes_r2.json @@ -0,0 +1,45 @@ +{ + "d": 384, + "label": "old", + "n": 1000000, + "nprobe": 128, + "writes": { + "after_30k_inserts": { + "first_ms": 1072.5458999999998, + "median_ms": 5.754, + "p99_ms": 10.38 + }, + "after_all": { + "first_ms": 9.4106, + "median_ms": 6.9894, + "p99_ms": 13.9317 + }, + "before": { + "first_ms": 3.0078, + "median_ms": 2.0219, + "p99_ms": 3.3252 + }, + "before_1t": { + "first_ms": 5.7204999999999995, + "median_ms": 5.7134, + "p99_ms": 13.029900000000001 + }, + "delete_10k": { + "per_s": 274878.57239064644 + }, + "insert_30k": { + "max_ms": 326.1998, + "median_ms": 0.001, + "p99_ms": 0.004, + "per_s": 82747.23010542824, + "seconds": 0.3625499 + }, + "peak_mb": 3778.3515625, + "rss_after_mb": 3641.0703125, + "rss_built_mb": 3530.515625, + "update_10k": { + "max_ms": 0.11230000000000001, + "per_s": 239554.2374749067 + } + } +} \ No newline at end of file diff --git a/benchmarks/rust-stage-a/raw/eb_old_1000000_writes_r2.txt b/benchmarks/rust-stage-a/raw/eb_old_1000000_writes_r2.txt new file mode 100644 index 0000000..6f07c00 --- /dev/null +++ b/benchmarks/rust-stage-a/raw/eb_old_1000000_writes_r2.txt @@ -0,0 +1,9 @@ +old: n=1000000 d=384 queries=2000 nprobe=128 has_batch=false + built: first query 3.01 ms, then median 2.022 ms p99 3.325 ms + built (1 thread): first query 5.72 ms, then median 5.713 ms p99 13.030 ms + insert 30k: 0.36s = 82747/s, per-write median 0.0010 ms p99 0.0040 ms max 326.200 ms + after 30k inserts (1 thread): first query 1072.55 ms, then median 5.754 ms p99 10.380 ms + update 10k: 239554/s max 0.112 ms + delete 10k: 274879/s + after 30k writes total (1 thread): first query 9.41 ms, then median 6.989 ms p99 13.932 ms +wrote D:/Goldmine/ISON/ison_dev/isongraphossworking/isongraph-vector/benchmarks/rust-stage-a/raw/eb_old_1000000_writes_r2.json diff --git a/benchmarks/rust-stage-a/raw/eb_old_100000_scaling,loop.json b/benchmarks/rust-stage-a/raw/eb_old_100000_scaling,loop.json new file mode 100644 index 0000000..311ef3d --- /dev/null +++ b/benchmarks/rust-stage-a/raw/eb_old_100000_scaling,loop.json @@ -0,0 +1,209 @@ +{ + "d": 384, + "label": "old", + "loop": [ + { + "qps": 151.94083140402242, + "runs": [ + 151.94083140402242, + 150.2947430205376, + 152.26033512702776, + 151.823813929592, + 154.18198593181887 + ], + "threads": 1 + }, + { + "qps": 218.27347716779397, + "runs": [ + 218.27347716779397, + 235.0433310215516, + 230.8162817805168, + 207.82517192337346, + 180.77958665832188 + ], + "threads": 2 + }, + { + "qps": 272.9510429550335, + "runs": [ + 272.26155695856977, + 274.8948939373031, + 260.9091780369437, + 275.35895334226103, + 272.9510429550335 + ], + "threads": 4 + }, + { + "qps": 297.32985893482174, + "runs": [ + 299.4294670934002, + 300.4056678138158, + 297.32985893482174, + 289.91445590787674, + 284.4478156778154 + ], + "threads": 8 + }, + { + "qps": 296.7973194849023, + "runs": [ + 296.7973194849023, + 284.01597836025456, + 281.55266911930323, + 298.2157157296464, + 311.114890580893 + ], + "threads": 0 + } + ], + "n": 100000, + "nprobe": 128, + "scaling": { + "exact": [ + { + "clients": 1, + "efficiency": 1.0, + "qps": 144.84875858287165, + "runs": [ + 144.84875858287165, + 140.48531417459867, + 151.10329870815664 + ] + }, + { + "clients": 2, + "efficiency": 0.5230120678806545, + "qps": 151.5152975127468, + "runs": [ + 151.5587619244506, + 149.47943785766068, + 151.5152975127468 + ] + }, + { + "clients": 4, + "efficiency": 0.25605386684945014, + "qps": 148.35633897394706, + "runs": [ + 144.7027973032949, + 148.35633897394706, + 154.8404873333037 + ] + }, + { + "clients": 8, + "efficiency": 0.1282177829378841, + "qps": 148.5774934944049, + "runs": [ + 148.5774934944049, + 149.6646527143326, + 140.2778517495811 + ] + }, + { + "clients": 16, + "efficiency": 0.06522340837214902, + "qps": 151.1604757319914, + "runs": [ + 152.29181926097232, + 150.92700233711588, + 151.1604757319914 + ] + }, + { + "clients": 24, + "efficiency": 0.04284704243751747, + "qps": 148.9521817445286, + "runs": [ + 153.3198563912459, + 146.75864007345047, + 148.9521817445286 + ] + }, + { + "clients": 32, + "efficiency": 0.03272427320171644, + "qps": 151.68225116145155, + "runs": [ + 151.68225116145155, + 153.89262712834122, + 145.84455599904723 + ] + } + ], + "ivf": [ + { + "clients": 1, + "efficiency": 1.0, + "qps": 797.941995980438, + "runs": [ + 797.0742310699535, + 797.941995980438, + 802.8133994055315 + ] + }, + { + "clients": 2, + "efficiency": 0.48439040944119294, + "qps": 773.0309002865743, + "runs": [ + 773.0309002865743, + 765.2603644305117, + 779.2065452017251 + ] + }, + { + "clients": 4, + "efficiency": 0.23934116579040302, + "qps": 763.9214702043165, + "runs": [ + 772.7509341644788, + 693.8355127311329, + 763.9214702043165 + ] + }, + { + "clients": 8, + "efficiency": 0.11273575123872338, + "qps": 719.6527228942487, + "runs": [ + 719.6527228942487, + 708.8938170407511, + 778.9076718386578 + ] + }, + { + "clients": 16, + "efficiency": 0.06248157093327546, + "qps": 797.7067107598582, + "runs": [ + 797.7067107598582, + 797.81022610637, + 777.1451640279603 + ] + }, + { + "clients": 24, + "efficiency": 0.04167026764294849, + "qps": 798.0109568652811, + "runs": [ + 792.0595845213677, + 798.0109568652811, + 803.1956760221474 + ] + }, + { + "clients": 32, + "efficiency": 0.0305640713054344, + "qps": 780.4273940078963, + "runs": [ + 780.4273940078963, + 766.492032971324, + 797.3140224732648 + ] + } + ] + } +} \ No newline at end of file diff --git a/benchmarks/rust-stage-a/raw/eb_old_100000_scaling,loop.txt b/benchmarks/rust-stage-a/raw/eb_old_100000_scaling,loop.txt new file mode 100644 index 0000000..bdf5733 --- /dev/null +++ b/benchmarks/rust-stage-a/raw/eb_old_100000_scaling,loop.txt @@ -0,0 +1,21 @@ +old: n=100000 d=384 queries=2000 nprobe=128 has_batch=false + scaling exact clients= 1 qps= 144.8 efficiency=1.00 + scaling exact clients= 2 qps= 151.5 efficiency=0.52 + scaling exact clients= 4 qps= 148.4 efficiency=0.26 + scaling exact clients= 8 qps= 148.6 efficiency=0.13 + scaling exact clients=16 qps= 151.2 efficiency=0.07 + scaling exact clients=24 qps= 149.0 efficiency=0.04 + scaling exact clients=32 qps= 151.7 efficiency=0.03 + scaling ivf clients= 1 qps= 797.9 efficiency=1.00 + scaling ivf clients= 2 qps= 773.0 efficiency=0.48 + scaling ivf clients= 4 qps= 763.9 efficiency=0.24 + scaling ivf clients= 8 qps= 719.7 efficiency=0.11 + scaling ivf clients=16 qps= 797.7 efficiency=0.06 + scaling ivf clients=24 qps= 798.0 efficiency=0.04 + scaling ivf clients=32 qps= 780.4 efficiency=0.03 + exact loop threads=1 qps=151.9 (min 150.3 max 154.2) + exact loop threads=2 qps=218.3 (min 180.8 max 235.0) + exact loop threads=4 qps=273.0 (min 260.9 max 275.4) + exact loop threads=8 qps=297.3 (min 284.4 max 300.4) + exact loop threads=0 qps=296.8 (min 281.6 max 311.1) +wrote D:/Goldmine/ISON/ison_dev/isongraphossworking/isongraph-vector/benchmarks/rust-stage-a/raw/eb_old_100000_scaling,loop.json diff --git a/benchmarks/rust-stage-a/run_ab.sh b/benchmarks/rust-stage-a/run_ab.sh new file mode 100644 index 0000000..7e616f2 --- /dev/null +++ b/benchmarks/rust-stage-a/run_ab.sh @@ -0,0 +1,45 @@ +#!/usr/bin/env bash +# A/B session for the Rust engine work: the pre-change commit ("old") and the working tree ("new") run +# back to back, one timed job at a time, on the same machine. Usage: +# OLD=<dir holding old build's target/release/examples> NEW=<same for new> NEWEB=<new engine_ab dir> bash run_ab.sh +# OLD = a build of the commit before the change (git archive HEAD~ of isongraph-vector-rs, CARGO_TARGET_DIR=...) +# NEW = cargo build --release --examples in isongraph-vector-rs +# NEWEB= engine_ab built with RUSTFLAGS="--cfg has_batch" (adds the batch experiments) +set -u +DATA=/d/Goldmine/ISON/bench_data/prep +OUT="$(cd "$(dirname "$0")" && pwd)/raw" +wait_quiet() { + for i in $(seq 1 30); do + load=$(powershell -NoProfile -Command "[int](Get-Counter '\Processor(_Total)\% Processor Time' -SampleInterval 2 -MaxSamples 2 | % { \$_.CounterSamples.CookedValue } | Measure-Object -Average).Average") + if [ "${load:-100}" -lt 30 ]; then echo " [cpu load before: ${load}%]"; return; fi + sleep 10 + done + echo " [cpu load stayed high: ${load}%]" +} +cb() { # which exe, label, dataset, n, out-suffix + wait_quiet; echo "== competitive_bench $2 $3 $4" + "$1/competitive_bench.exe" --ds-dir $DATA/$3 --ds $3 --n $4 --exp core,mutation --queries 2000 --budget 3 --label $2 --out "$OUT/cb_$2_$3_$4$5.json" > "$OUT/cb_$2_$3_$4$5.txt" 2>&1; tail -3 "$OUT/cb_$2_$3_$4$5.txt" +} +eb() { # exe, label, n, exps, suffix, extra args + wait_quiet; echo "== engine_ab $2 n=$3 $4" + "$1/engine_ab.exe" --ds-dir $DATA/text384 --n $3 --exp $4 --queries 2000 --label $2 --out "$OUT/eb_$2_$3_$4$5.json" ${6:-} > "$OUT/eb_$2_$3_$4$5.txt" 2>&1; tail -20 "$OUT/eb_$2_$3_$4$5.txt" +} +step=${1:-all} +if [ "$step" = all ] || [ "$step" = a ]; then + for r in 1 2; do cb $OLD old text384 100000 _r$r; cb $NEW new text384 100000 _r$r; done +fi +if [ "$step" = all ] || [ "$step" = b ]; then + for ds in glove sift; do cb $OLD old $ds 100000 ""; cb $NEW new $ds 100000 ""; done +fi +if [ "$step" = all ] || [ "$step" = c ]; then + eb $OLD old 100000 scaling,loop ""; eb $NEWEB new 100000 scaling,loop,batch "" +fi +if [ "$step" = all ] || [ "$step" = d ]; then + eb $OLD old 1000000 mem ""; eb $NEWEB new 1000000 mem "" + for r in 1 2; do eb $OLD old 1000000 writes _r$r; eb $NEWEB new 1000000 writes _r$r; done + eb $OLD old 1000000 deletes ""; eb $NEWEB new 1000000 deletes "" +fi +if [ "$step" = all ] || [ "$step" = e ]; then + cb $OLD old text384 1000000 ""; cb $NEW new text384 1000000 "" +fi +echo DONE diff --git a/benchmarks/rust-stage-a/summarize.py b/benchmarks/rust-stage-a/summarize.py new file mode 100644 index 0000000..94cac67 --- /dev/null +++ b/benchmarks/rust-stage-a/summarize.py @@ -0,0 +1,170 @@ +"""Turn raw/*.json from run_ab.sh into the markdown tables in RESULTS.md. + + python summarize.py > RESULTS_tables.md +""" +import glob +import json +import math +import os +import statistics + +HERE = os.path.dirname(os.path.abspath(__file__)) +RAW = os.path.join(HERE, "raw") + + +def load(pattern): + out = [] + for f in sorted(glob.glob(os.path.join(RAW, pattern))): + try: + out.append(json.load(open(f, encoding="utf-8"))) + except Exception: + pass + return out + + +def med(xs): + xs = [x for x in xs if x is not None] + return statistics.median(xs) if xs else float("nan") + + +def fmt(x, digits=0): + if x is None or (isinstance(x, float) and math.isnan(x)): + return "-" + return f"{x:,.{digits}f}" + + +def cb_runs(label, ds, n): + """competitive_bench result files (one per repeat) -> list of {exact, ivf} dicts.""" + runs = [] + for d in load(f"cb_{label}_{ds}_{n}*.json"): + by = {r["system"].split("_")[-1]: r for r in d["runs"]} + runs.append(by) + return runs + + +def at_recall(sweep, target): + """QPS interpolated (log-linear in qps) at the recall target; None if never reached.""" + pts = sorted(((p["recall"], p["qps_1t"], p["knob"]) for p in sweep), key=lambda t: t[0]) + for (r0, q0, k0), (r1, q1, k1) in zip(pts, pts[1:]): + if r0 <= target <= r1 and r1 > r0: + w = (target - r0) / (r1 - r0) + return math.exp(math.log(q0) * (1 - w) + math.log(q1) * w) + return None + + +def table_cb(ds, n): + old, new = cb_runs("old", ds, n), cb_runs("new", ds, n) + if not old or not new: + return "" + lines = [f"#### {ds} n={n:,} (median over {len(old)} old / {len(new)} new run(s))", ""] + lines += ["| metric | old | new | new/old |", "|---|---|---|---|"] + + def row(name, f, digits=0, ratio=True): + a = med([f(r) for r in old]) + b = med([f(r) for r in new]) + lines.append(f"| {name} | {fmt(a, digits)} | {fmt(b, digits)} | {fmt(b / a, 2) + 'x' if ratio and a else '-'} |") + + row("exact, 1 thread (pinned), QPS", lambda r: r["exact"]["sweep"][0]["qps_1t"], 1) + row("exact, one-query loop with auto threads, QPS", lambda r: r["exact"]["loop_allthreads"]["qps"], 1) + row("8 concurrent clients (1 thread each), exact QPS", lambda r: r["exact"]["concurrent_clients"]["qps"], 1) + if "batch_allthreads" in new[0]["exact"]: + lines.append(f"| exact batch, all threads, QPS | - | {fmt(med([r['exact']['batch_allthreads']['qps'] for r in new]), 0)} | - |") + row("IVF build, s", lambda r: r["ivf"]["build"]["index_s"], 2) + row("IVF 1-thread QPS at recall 0.95 (interpolated)", lambda r: at_recall(r["ivf"]["sweep"], 0.95), 0) + row("IVF one-query loop, auto threads, QPS at ref nprobe", lambda r: next((p["allthreads"]["qps"] for p in r["ivf"]["sweep"] if p["knob"] == r["ivf"]["ref_knob"]), None), 0) + if "batch_allthreads" in new[0]["ivf"]["sweep"][0]: + lines.append("| IVF batch, all threads, QPS at ref nprobe | - | " + fmt(med([next(p["batch_allthreads"]["qps"] for p in r["ivf"]["sweep"] if p["knob"] == r["ivf"]["ref_knob"]) for r in new]), 0) + " | - |") + row("ingest, s", lambda r: r["ivf"]["build"]["ingest_s"], 2) + row("working set after ingest, MB", lambda r: r["ivf"]["mem_after_ingest"]["rss_mb"] - r["ivf"]["mem_baseline"]["rss_mb"], 0) + row("working set after IVF build, MB", lambda r: r["ivf"]["mem_after_build"]["rss_mb"] - r["ivf"]["mem_baseline"]["rss_mb"], 0) + row("peak working set after IVF build, MB", lambda r: r["ivf"]["mem_after_build"]["peak_mb"] - r["ivf"]["mem_baseline"]["rss_mb"], 0) + if all("mutation" in r["ivf"] and "insert_10k" in r["ivf"]["mutation"] for r in old + new): + row("insert 10k, per s", lambda r: r["ivf"]["mutation"]["insert_10k"]["per_s"], 0) + row("update 10k, per s", lambda r: r["ivf"]["mutation"]["update_10k"]["per_s"], 0) + row("delete 10k, per s", lambda r: r["ivf"]["mutation"]["delete_10k"]["per_s"], 0) + row("first query after the 30k writes, ms", lambda r: r["ivf"]["mutation"]["first_query_after_ms"], 2) + row("recall@10 after the writes", lambda r: r["ivf"]["mutation"]["recall_after"], 4, ratio=False) + lines += ["", "recall@10 / 1-thread QPS by nprobe (median of runs):", "", "| nprobe | old recall | new recall | old QPS | new QPS |", "|---|---|---|---|---|"] + knobs = [p["knob"] for p in old[0]["ivf"]["sweep"]] + for k in knobs: + def pick(runs, key): + return med([next((p[key] for p in r["ivf"]["sweep"] if p["knob"] == k), None) for r in runs]) + lines.append(f"| {k} | {pick(old, 'recall'):.4f} | {pick(new, 'recall'):.4f} | {fmt(pick(old, 'qps_1t'))} | {fmt(pick(new, 'qps_1t'))} |") + lines.append("") + return "\n".join(lines) + + +def table_eb(): + out = [] + for kind in ["scaling"]: + old = load(f"eb_old_100000_scaling*.json") + new = load(f"eb_new_100000_scaling*.json") + if old and new and "scaling" in old[0] and "scaling" in new[0]: + out += ["#### Concurrent single-query callers, 100k x 384 (QPS; efficiency = QPS / (clients x 1-client QPS))", ""] + for mode in ["exact", "ivf"]: + out += [f"{mode}:", "", "| clients | old QPS | old eff. | new QPS | new eff. |", "|---|---|---|---|---|"] + for a, b in zip(old[0]["scaling"][mode], new[0]["scaling"][mode]): + out.append(f"| {a['clients']} | {fmt(a['qps'])} | {a['efficiency']:.2f} | {fmt(b['qps'])} | {b['efficiency']:.2f} |") + out.append("") + new = load("eb_new_100000_*batch*.json") + if new and "batch" in new[0]: + out += ["#### Batch search, 100k x 384, 2000 queries (QPS by thread cap)", "", "| threads | exact batch | IVF batch (nprobe 128, recall 0.955) |", "|---|---|---|"] + for a, b in zip(new[0]["batch"]["exact"], new[0]["batch"]["ivf"]): + out.append(f"| {a['threads']} | {fmt(a['qps'])} | {fmt(b['qps'])} |") + out.append("") + old, new = load("eb_old_100000_*loop*.json"), load("eb_new_100000_*loop*.json") + if old and new and "loop" in old[0]: + out += ["#### One-query exact loop by thread cap, 100k x 384 (QPS, unpinned, median of 5)", "", "| threads | old | new |", "|---|---|---|"] + for a, b in zip(old[0]["loop"], new[0]["loop"]): + out.append(f"| {a['threads'] or 'auto'} | {fmt(a['qps'], 1)} | {fmt(b['qps'], 1)} |") + out.append("") + old, new = load("eb_old_1000000_mem*.json"), load("eb_new_1000000_mem*.json") + if old and new: + o, n = old[0]["mem"], new[0]["mem"] + out += ["#### Memory, 1M x 384 (working set minus baseline, MB)", "", "| stage | old | new |", "|---|---|---|"] + out.append(f"| raw vectors | {o['raw_mb']:.0f} | {n['raw_mb']:.0f} |") + for key, name in [("after_ingest_mb", "after ingest"), ("peak_after_ingest_mb", "peak during ingest"), ("after_first_query_mb", "after first exact query"), ("after_ivf_build_mb", "after IVF build"), ("peak_after_build_mb", "peak during IVF build")]: + out.append(f"| {name} | {o[key]:.0f} ({o[key] / o['raw_mb']:.2f}x) | {n[key]:.0f} ({n[key] / n['raw_mb']:.2f}x) |") + out.append("") + old, new = load("eb_old_1000000_writes*.json"), load("eb_new_1000000_writes*.json") + if old and new: + out += [f"#### Writes on a built IVF index, 1M x 384 (median of {len(old)} / {len(new)} run(s))", "", "| measure | old | new |", "|---|---|---|"] + + def w(runs, path): + vals = [] + for r in runs: + x = r["writes"] + for p in path: + x = x[p] + vals.append(x) + return med(vals) + + for name, path, d in [ + ("insert 30k, per s", ["insert_30k", "per_s"], 0), + ("slowest single insert, ms", ["insert_30k", "max_ms"], 2), + ("insert p99, ms", ["insert_30k", "p99_ms"], 4), + ("update 10k, per s", ["update_10k", "per_s"], 0), + ("slowest single update, ms", ["update_10k", "max_ms"], 2), + ("delete 10k, per s", ["delete_10k", "per_s"], 0), + ("first query after 30k inserts (1 thread), ms", ["after_30k_inserts", "first_ms"], 2), + ("steady query median after them (1 thread), ms", ["after_30k_inserts", "median_ms"], 2), + ("first query after all 50k writes (1 thread), ms", ["after_all", "first_ms"], 2), + ("query median before any write (1 thread), ms", ["before_1t", "median_ms"], 2), + ]: + out.append(f"| {name} | {fmt(w(old, path), d)} | {fmt(w(new, path), d)} |") + out.append("") + old, new = load("eb_old_1000000_deletes*.json"), load("eb_new_1000000_deletes*.json") + if old and new: + out += ["#### Deleting 10,000 random ids from 1M x 384", "", "| store state | old per s (ns each) | new per s (ns each) |", "|---|---|---|"] + for k in old[0]["deletes"]: + a, b = old[0]["deletes"][k], new[0]["deletes"][k] + out.append(f"| {k} | {fmt(a['per_s'])} ({a['ns_each']:.0f}) | {fmt(b['per_s'])} ({b['ns_each']:.0f}) |") + out.append("") + return "\n".join(out) + + +if __name__ == "__main__": + print("### Rust stage A: before / after (same session)\n") + for ds, n in [("text384", 100000), ("text384", 1000000), ("glove", 100000), ("sift", 100000)]: + print(table_cb(ds, n)) + print(table_eb()) diff --git a/benchmarks/rust-stage-c/README.md b/benchmarks/rust-stage-c/README.md new file mode 100644 index 0000000..e293ea7 --- /dev/null +++ b/benchmarks/rust-stage-c/README.md @@ -0,0 +1,377 @@ +# Stage C: honest re-benchmark of the reworked Rust crate + +Crate `isongraph-vector-rs` at HEAD 5767252 (branch `feature/ann-ports-and-server`), measured against numpy, faiss +(IndexFlatIP / IVFFlat / HNSWFlat), hnswlib and usearch on the data and ground truth of `benchmarks/competitive/`. +Everything below is regenerated from `raw/` by `summarize.py`; nothing was carried over from earlier READMEs. + +## Verdict (one paragraph) + +The reworked Rust crate is now a credible ANN engine, but the old "fastest exact scan" claim does not survive a same-session +re-measurement. **Exact search is at parity, not ahead**: single-thread Rust vs faiss IndexFlatIP is 0.91-1.2x depending on the +dataset and run (it is slower than faiss on GloVe and SIFT 100k, equal at text384 1M: 11.1 vs 11.4 QPS), and it beats numpy +only on GloVe (1.45x) and, within run noise, text384 100k (1.27x); at 1M and on SIFT it ties numpy. **HNSW is the big change**: +at recall@10 0.95 it matches faiss HNSWFlat and hnswlib single-thread (text384 100k: 3,893 vs 3,965 / 2,723 QPS; 1M: 1,813 vs +1,773 / 1,517; one run each, and the discarded first attempt of the Rust cell gave 1,531), beats usearch (2,070 / 612), builds in about the same time +(145-173 s vs 157-216 s at 1M) and uses about the same memory. IVF is still slower than +faiss IVFFlat at equal recall (0.71x at 100k, 0.76x at 1M), but IVF+int8 beats it (1.9x / 1.3x). The Rust batch API is a clear +win for IVF (8x faiss IVFFlat on 32 threads at 100k, 10-21x at 1M) and level with the HNSW libraries; int8 gives a 3.0-3.6x single-thread speedup with +recall 1.000 against the f32 scan from factor 2 on real embeddings, and does nothing useful for batches. The binary snapshot +loads 6.6x faster than the JSON sidecar and is 3x smaller, but an ANN index is still never persisted: time-to-ready is 21.6 s +(IVF) / about 2.5-3 minutes (HNSW) at 1M, against about 1 s for faiss and usearch. Single-call HNSW inserts are slow (1,393/s at 1M, +versus 3,789-6,005/s for hnswlib/usearch, which were driven in parallel batches of 1,000); HNSW updates are on par with hnswlib (1,440 vs 1,696/s). The machine was a laptop with the +user's desktop apps running, not a quiet bench: run-to-run spread is visible (20-25% on one-query QPS, up to 2x on batch, build and write rates between the first and the repeated attempt of the same 1M cell), so differences under about 15% are +reported as parity. + +## Answers + +| # | question | answer | evidence (details in the tables) | +|---|---|---|---| +| 1 | Rust 1T exact still faster than numpy and faiss Flat (text384 100k/1M, glove 100k, sift 100k)? | **Refuted** as stated. Parity with faiss; slight edge over numpy on some cells | faiss Flat/Rust: text 100k 117.8 / 142.6 (Rust runs 114-146, faiss 117-136: overlap); text 1M 11.4 / 11.1; glove 605 / 548 (ranges 604-622 vs 537-548: faiss wins by ~10%); sift 432 / 397 (ranges overlap). numpy: 113 / 11.1 / 377 / 386 vs Rust 143 / 11.1 / 548 / 397 | +| 2 | HNSW vs hnswlib / faiss HNSW / usearch at recall 0.95, 1T | **Confirmed: parity with the best (at 1M ahead of hnswlib by 19% in one run), well ahead of usearch.** IVF **refuted** (0.71-0.76x faiss IVFFlat); IVF+int8 **confirmed faster** than faiss IVFFlat (1.3-1.9x); HNSW+int8 is accepted but a **no-op that costs the int8 copy's memory (+370 MB at 1M)** | Q2 table; 100k 3 runs, 1M 1 run | +| 3 | Batch / all-core vs faiss Flat batch and HNSW libs at 100k | **Confirmed for IVF (8x faiss IVFFlat at 100k, 10-21x at 1M); inconclusive for exact (5,320 vs 5,085, run ranges overlap; 560 vs 298 at 1M, 1 run); parity for HNSW (14.0k vs 16.7k faiss, 12.7k hnswlib, 13.7k usearch)** | Q3 table; batch timings varied up to 2.5x between runs | +| 4 | int8 recall on real embeddings, factors 1/2/4/8, speedup | **Confirmed**: factor >= 2 gives recall 1.0000 vs f32 scan on text384 (100k, 1M), GloVe and SIFT; factor 1 gives 0.976-0.992 (worst query 0.80). 1T speedup 3.0-3.6x. **Batch: no benefit (0.4-1.0x (slower than f32 in every measured batch))** | Q4 table | +| 5 | Persistence | Snapshot vs JSON: **confirmed big win** (1M load 4.6 s vs 30 s, save 7.7 s vs 42.9 s, 1,489 vs 4,534 MB). Competitive with faiss/hnswlib for a ready-to-search ANN index: **refuted** (index not persisted) | Q5 table | +| 6 | Writes at 1M, IVF and HNSW vs hnswlib/usearch | IVF: **faster than faiss IVFFlat on inserts/updates** (138k / 133k per s vs 87k / 69k; first query after writes 8.0 vs 8.1 ms). HNSW single-call inserts **slower** than hnswlib/usearch batch inserts (1,393 vs 6,005 / 3,789 per s), updates on par (1,440 vs 1,696 / 3,414); first query after writes 0.6 ms and recall holds (0.9549 -> 0.9550) | Q6 table | +| 7 | Memory at 1M x 384 | exact 1.12x raw, IVF 1.12x, HNSW 1.22x (faiss HNSW 1.24x, hnswlib 1.28x, usearch 1.17x, faiss IVF 1.28x); int8 adds 371 MB. **Confirmed no worse than the libraries** | Q7 table | + +## What we win and lose now (each line is exactly what the data supports) + +**Win** +* Single-thread HNSW at recall@10 ~0.95 on text384 is level with faiss HNSWFlat and 19% above hnswlib at 1M (1,813 vs 1,773 / 1,517 QPS, one run each) and 43% above hnswlib, level with faiss, at 100k (3 runs; faiss 3,965, hnswlib 2,723). The one-query loop for the libraries runs through Python, Rust's does not: part of the 100k margin over hnswlib may be that overhead. +* IVF batch search on 32 threads: 24,360 QPS vs faiss IVFFlat 3,014 at 100k (recall ~0.955, 3 runs, ranges 21-25k vs 2.9-3.6k) and 10,068 vs 471 at 1M (1 run; 4,524 in the discarded noisy first attempt). +* IVF + int8 (rerank 4) single-thread at recall 0.955: 2,059 QPS vs faiss IVFFlat f32 1,065 at 100k (1 run), 235 vs 180 at 1M (1 run). +* int8 exact scan: 3.0-3.6x single-thread QPS with exact returned scores and recall 1.0000 vs the f32 scan at factor >= 2 on all four real datasets; the copy costs 25% of the vector bytes. +* Binary snapshot load: 6.6x faster than the JSON sidecar at 100k and 1M (0.19 vs 1.27 s; 4.6 vs 30 s) and 3x smaller. +* Index build: IVF 7.6 s vs faiss 35.3 s at 1M (1 run; 18.8 s in the discarded first attempt); HNSW is level: 173 s vs hnswlib 157 s, faiss 190 s, usearch 216 s at 1M (1 run; 152 s in the discarded attempt); 7.1 s vs 8.4 / 9.6 / 6.8 s at 100k (3 runs). +* IVF memory: 1,643 MB working set at 1M vs faiss IVFFlat 1,881 MB, and a 1,886 MB build peak vs faiss 3,588 MB. +* IVF writes at 1M: 138k inserts/s and 133k updates/s one call at a time vs faiss IVFFlat 87k / 69k (1 run each; 81k / 69k in the discarded first attempt); first query after 30k writes 8.0 ms, recall 0.9526 -> 0.9530. +* 8 concurrent clients scale (HNSW 1M: 700 -> 4,990 QPS from 1 to 8 clients); numpy/faiss searches are not compared here. + +**Lose / parity** +* Exact single-thread scan: no longer ahead. Equal to faiss Flat and numpy at text384 1M (11.1 / 11.4 / 11.1); 9-10% slower than faiss Flat on GloVe 100k (548 vs 605); not distinguishable on text384 100k and SIFT 100k. +* IVF (f32) at equal recall: 0.71x faiss IVFFlat at 100k (759 vs 1,065), 0.76x at 1M (137 vs 180; 108 in the discarded first attempt). +* ANN indexes are not persisted: after a snapshot load, IVF is ready in 0.7 s (100k) / 21.6 s (1M) and HNSW in 6.1 s (100k) / about 2.5-3 minutes (1M: load 4.6 s + a 145-173 s build, not run as one measurement); faiss/usearch are ready in 0.1-1.3 s. +* HNSW insert rate: 944/s at 100k and 1,393/s at 1M one call at a time (650/s in the discarded 1M first attempt) against hnswlib 6,005/s and usearch 3,789/s in batches of 1,000 on 32 threads. Not like for like, but it is what a caller gets. HNSW updates (1,440/s at 1M) are on par with hnswlib (1,696/s) and below usearch (3,414/s). +* int8 does nothing for batches (0.4-1.0x (slower than f32 in every measured batch)) and does nothing for HNSW (setting accepted, search path unchanged, +370 MB at 1M). +* JSON save/load is still 43 s / 30 s at 1M (4.5 GB). +* Exact batch (5,320 vs faiss Flat 5,085 at 100k): not distinguishable; at 1M 560 vs 298 (1 run). + +**Anything surprising / possible bug**: `set_scan_precision(Int8)` together with `AnnConfig::hnsw()` is silently accepted. Searches never use the int8 copy (identical recall at identical ef: 0.9626 / 0.9549; QPS equal within noise), yet 37 MB (100k) / 370 MB (1M) are allocated. Not a correctness bug, but neither the README nor DESIGN.md says it, and it wastes memory. Repro: `stage_c_bench --mode hnsw --int8 4`, compare `mem_after_build` with `--mode hnsw`. No wrong answers, crashes or recall regressions were found anywhere (every int8 score checked bit-exact; every snapshot/JSON load returned the same top-10 as the original store; recall after the 30k writes within 0.005 of before at every operating point). + + + +# Tables + +Generated by `summarize.py` from `raw/`. Q1-Q7 match the questions of the stage. + +## Q1. Exact scan: ISONGraph Rust vs numpy vs faiss IndexFlatIP (single thread, one pinned P-core) + +QPS, median over runs (min-max). `ratio` = Rust / other, > 1 means Rust is faster. Rounds were interleaved (Rust, numpy, faiss, Rust, ...) in one session. + +| cell | runs | Rust exact | numpy | faiss Flat | Rust/numpy | Rust/faiss | Rust auto-thread loop | faiss Flat batch (32 thr) | numpy batch | Rust batch | +|---|---|---|---|---|---|---|---|---|---|---| +| text384 100,000 | 3/3/3 | 142.6 (114.0-145.7) | 112.7 (112.2-121.3) | 117.8 (116.5-135.5) | 1.27x | 1.21x | 269 (216-284) | 5,085 (3,070-6,552) | 673 (543-1,052) | 5,320 (5,206-12,743) | +| text384 1,000,000 | 1/1/1 | 11.1 (1 run) | 11.1 (1 run) | 11.4 (1 run) | 1.00x | 0.97x | 25 (1 run) | 298 (1 run) | 61 (1 run) | 560 (1 run) | +| glove 100,000 | 3/3/3 | 548.2 (537.1-548.4) | 377.0 (323.4-456.6) | 605.0 (603.5-621.6) | 1.45x | 0.91x | 1,568 (708-1,669) | 14,934 (5,444-15,184) | 857 (783-1,434) | 19,239 (10,552-25,285) | +| sift 100,000 | 3/3/3 | 396.9 (350.0-470.5) | 385.5 (313.9-389.3) | 431.7 (421.5-521.6) | 1.03x | 0.92x | 722 (645-1,546) | 5,898 (5,075-14,071) | 1,364 (786-1,512) | 11,457 (10,910-28,197) | + +## Q2. Approximate search at recall@10 = 0.95, single thread (pinned), text384 + +QPS interpolated (log-linear) between the two sweep points that bracket recall 0.95; `-` = not bracketed by the sweep (recall 0.95 not reached, or the lowest point already above it: then the lowest measured point is shown in the `first point` column). Build = ingest + index build on the library's own path with 32 threads. Memory = working set after the build minus baseline. + +### text384 100,000 + +| system | runs | QPS @0.95 (1T) | knob at first recall>=0.95 | recall at that knob | p99 ms there | build s | of which index s | RSS after build MB | peak MB | first point (knob: recall, QPS) | best recall reached | +|---|---|---|---|---|---|---|---|---|---|---|---| +| ISONGraph Rust HNSW | 3 | 3,893 (3,314-4,486) | 128 | 0.9626 | 0.47 | 7.1 (6.9-7.4) | 7.0 (6.8-7.3) | 199 (198-200) | 201 (201-201) | 24: 0.782, 8,823 | 0.9936 | +| ISONGraph Rust IVF | 3 | 759 (752-968) | 128 | 0.9559 | 1.81 | 0.6 (0.6-0.6) | 0.5 (0.5-0.5) | 166 (166-166) | 246 (246-246) | 16: 0.750, 5,848 | 0.9952 | +| ISONGraph Rust IVF + int8 x4 | 1 | 2,059 (1 run) | 128 | 0.9559 | 0.81 | 0.6 (1 run) | 0.5 (1 run) | 205 (1 run) | 282 (1 run) | 16: 0.750, 7,552 | 0.9953 | +| ISONGraph Rust HNSW + int8 x4 (setting on) | 1 | 4,571 (1 run) | 128 | 0.9626 | 0.47 | 5.7 (1 run) | 5.6 (1 run) | 238 (1 run) | 239 (1 run) | 64: 0.914, 7,034 | 0.9796 | +| hnswlib | 3 | 2,723 (2,480-2,751) | 128 | 0.9631 | 0.66 | 8.4 (8.0-8.7) | 0.0 (0.0-0.0) | 201 (201-201) | 231 (231-231) | 24: 0.785, 7,766 | 0.9930 | +| faiss HNSWFlat | 3 | 3,965 (3,400-4,117) | 96 | 0.9639 | 0.48 | 9.6 (7.6-12.2) | 0.0 (0.0-0.0) | 207 (206-207) | 348 (347-349) | 24: 0.810, 8,382 | 0.9971 | +| usearch | 3 | 2,070 (2,012-2,174) | 128 | 0.9634 | 0.89 | 6.8 (6.1-8.2) | 0.0 (0.0-0.0) | 182 (180-183) | 211 (210-213) | 24: 0.781, 6,608 | 0.9927 | +| faiss IVFFlat | 3 | 1,065 (1,028-1,096) | 128 | 0.9553 | 1.76 | 1.5 (1.4-1.6) | 1.5 (1.4-1.6) | 240 (240-242) | 461 (461-463) | 16: 0.751, 7,235 | 0.9958 | +| ISONGraph Rust exact (exact, recall 1.0) | 3 | 142.6 (114.0-145.7) | - | 1.000 | 10.58 | 0.10 (0.10-0.11) | 0 | - | - | - | 1.000 | +| faiss IndexFlatIP (exact, recall 1.0) | 3 | 117.8 (116.5-135.5) | - | 1.000 | 11.04 | 0.11 (0.10-0.13) | 0 | 174 (174-174) | - | - | 1.000 | + +### text384 1,000,000 + +| system | runs | QPS @0.95 (1T) | knob at first recall>=0.95 | recall at that knob | p99 ms there | build s | of which index s | RSS after build MB | peak MB | first point (knob: recall, QPS) | best recall reached | +|---|---|---|---|---|---|---|---|---|---|---|---| +| ISONGraph Rust HNSW | 1 | 1,813 (1 run) | 192 | 0.9549 | 0.90 | 172.7 (1 run) | 171.6 (1 run) | 1,786 (1 run) | 1,794 (1 run) | 48: 0.834, 4,599 | 0.9901 | +| ISONGraph Rust IVF | 1 | 137 (1 run) | 256 | 0.9526 | 9.32 | 7.6 (1 run) | 6.5 (1 run) | 1,643 (1 run) | 1,886 (1 run) | 16: 0.699, 1,412 | 0.9772 | +| ISONGraph Rust IVF + int8 x4 | 1 | 235 (1 run) | 256 | 0.9532 | 6.22 | 8.1 (1 run) | 6.9 (1 run) | 2,011 (1 run) | 2,255 (1 run) | 64: 0.858, 771 | 0.9780 | +| ISONGraph Rust HNSW + int8 x4 (setting on) | 1 | - | 192 | 0.9549 | 0.83 | 145.9 (1 run) | 144.7 (1 run) | 2,156 (1 run) | 2,164 (1 run) | 192: 0.955, 1,784 | 0.9685 | +| hnswlib | 1 | 1,517 (1 run) | 192 | 0.9552 | 0.99 | 157.2 (1 run) | 0.0 (1 run) | 1,876 (1 run) | 1,906 (1 run) | 24: 0.728, 6,005 | 0.9817 | +| faiss HNSWFlat | 1 | 1,773 (1 run) | 192 | 0.9668 | 1.03 | 189.5 (1 run) | 0.0 (1 run) | 1,815 (1 run) | 2,298 (1 run) | 24: 0.760, 6,168 | 0.9885 | +| usearch | 1 | 612 (1 run) | 192 | 0.9560 | 3.17 | 215.8 (1 run) | 0.0 (1 run) | 1,714 (1 run) | 1,744 (1 run) | 24: 0.726, 2,924 | 0.9818 | +| faiss IVFFlat | 1 | 180 (1 run) | 256 | 0.9561 | 15.10 | 35.3 (1 run) | 35.3 (1 run) | 1,881 (1 run) | 3,588 (1 run) | 16: 0.703, 1,561 | 0.9801 | +| ISONGraph Rust exact (exact, recall 1.0) | 1 | 11.1 (1 run) | - | 1.000 | 99.27 | 1.14 (1 run) | 0 | - | - | - | 1.000 | +| faiss IndexFlatIP (exact, recall 1.0) | 1 | 11.4 (1 run) | - | 1.000 | 95.71 | 1.05 (1 run) | 0 | 1,555 (1 run) | - | - | 1.000 | + +### Sweeps (median over runs; recall / QPS 1T) + +**text384 100,000** + +* ISONGraph Rust HNSW (ef_search): 24: 0.782 / 10,824; 48: 0.885 / 7,752; 64: 0.914 / 4,799; 96: 0.948 / 3,967; 128: 0.963 / 3,526; 192: 0.980 / 2,502; 256: 0.986 / 1,521; 400: 0.994 / 1,051 +* ISONGraph Rust IVF (nprobe): 16: 0.750 / 5,848; 32: 0.840 / 3,335; 64: 0.908 / 1,385; 128: 0.956 / 697; 256: 0.984 / 360; 512: 0.995 / 182 +* ISONGraph Rust IVF + int8 x4 (nprobe): 16: 0.750 / 7,552; 32: 0.840 / 5,316; 64: 0.908 / 3,251; 128: 0.956 / 1,930; 256: 0.984 / 847; 512: 0.995 / 503 +* ISONGraph Rust HNSW + int8 x4 (setting on) (ef_search): 64: 0.914 / 7,034; 96: 0.948 / 4,773; 128: 0.963 / 3,642; 192: 0.980 / 2,728 +* hnswlib (ef): 24: 0.786 / 7,766; 32: 0.831 / 6,152; 48: 0.885 / 4,375; 64: 0.914 / 3,843; 96: 0.949 / 2,789; 128: 0.963 / 2,175; 192: 0.980 / 1,572; 256: 0.986 / 1,243; 400: 0.993 / 813 +* faiss HNSWFlat (efSearch): 24: 0.811 / 8,382; 32: 0.857 / 8,922; 48: 0.907 / 6,351; 64: 0.936 / 4,365; 96: 0.964 / 3,579; 128: 0.975 / 2,808; 192: 0.987 / 1,945; 256: 0.992 / 1,518; 400: 0.997 / 965 +* usearch (expansion_search): 24: 0.781 / 6,608; 32: 0.828 / 4,933; 48: 0.882 / 4,000; 64: 0.915 / 3,141; 96: 0.948 / 2,145; 128: 0.963 / 1,780; 192: 0.978 / 1,183; 256: 0.986 / 893; 400: 0.992 / 617 +* faiss IVFFlat (nprobe): 16: 0.751 / 6,404; 32: 0.840 / 3,363; 64: 0.907 / 1,885; 128: 0.955 / 985; 256: 0.986 / 483; 512: 0.996 / 266 + +**text384 1,000,000** + +* ISONGraph Rust HNSW (ef_search): 48: 0.834 / 4,599; 64: 0.868 / 4,306; 96: 0.906 / 2,898; 128: 0.929 / 2,312; 192: 0.955 / 1,714; 256: 0.969 / 1,357; 400: 0.983 / 961; 640: 0.990 / 638 +* ISONGraph Rust IVF (nprobe): 16: 0.699 / 1,412; 32: 0.784 / 829; 64: 0.858 / 444; 128: 0.916 / 244; 256: 0.953 / 131; 512: 0.977 / 67 +* ISONGraph Rust IVF + int8 x4 (nprobe): 64: 0.858 / 771; 128: 0.915 / 424; 256: 0.953 / 223; 512: 0.978 / 121 +* ISONGraph Rust HNSW + int8 x4 (setting on) (ef_search): 192: 0.955 / 1,784; 256: 0.969 / 1,371 +* hnswlib (ef): 24: 0.728 / 6,005; 32: 0.774 / 6,181; 48: 0.832 / 4,509; 64: 0.868 / 3,471; 96: 0.905 / 2,659; 128: 0.928 / 1,772; 192: 0.955 / 1,461; 256: 0.967 / 1,136; 400: 0.982 / 760 +* faiss HNSWFlat (efSearch): 24: 0.760 / 6,168; 32: 0.802 / 5,394; 48: 0.859 / 4,129; 64: 0.892 / 3,336; 96: 0.929 / 2,386; 128: 0.947 / 1,872; 192: 0.967 / 1,357; 256: 0.977 / 1,117; 400: 0.989 / 709 +* usearch (expansion_search): 24: 0.726 / 2,924; 32: 0.773 / 2,408; 48: 0.830 / 1,762; 64: 0.866 / 1,409; 96: 0.910 / 1,051; 128: 0.931 / 803; 192: 0.956 / 561; 256: 0.968 / 432; 400: 0.982 / 288 +* faiss IVFFlat (nprobe): 16: 0.703 / 1,561; 32: 0.792 / 905; 64: 0.863 / 567; 128: 0.918 / 283; 256: 0.956 / 165; 512: 0.980 / 100 + +## Q3. Batch / all-core throughput, text384 100k (32 logical CPUs, 1,000 queries per batch, k=10) + +Batch QPS is the median of 3 timed batches after a warm-up batch, at the sweep point nearest recall 0.95 (first knob with recall >= 0.95; its recall is shown). Exact rows are recall 1.0. + +| system | runs | knob | recall there | batch QPS (32 thr) | 1T QPS there | batch / 1T | +|---|---|---|---|---|---|---| +| **text384 100,000** | | | | | | | +| ISONGraph Rust exact | 3 | - | 1.000 | 5,320 (5,206-12,743) | 143 (114-146) | 37x | +| faiss IndexFlatIP | 3 | - | 1.000 | 5,085 (3,070-6,552) | 118 (117-136) | 43x | +| numpy brute force | 3 | - | 1.000 | 673 (543-1,052) | 113 (112-121) | 6x | +| ISONGraph Rust IVF | 3 | 128 | 0.956 | 24,360 (21,141-24,903) | 697 (692-889) | 35x | +| ISONGraph Rust IVF + int8 x4 | 1 | 128 | 0.956 | 11,850 (1 run) | 1,930 (1 run) | 6x | +| faiss IVFFlat | 3 | 128 | 0.955 | 3,014 (2,922-3,561) | 985 (955-1,025) | 3x | +| ISONGraph Rust HNSW | 3 | 128 | 0.963 | 13,989 (13,919-14,608) | 3,526 (2,742-3,644) | 4x | +| faiss HNSWFlat | 3 | 96 | 0.964 | 16,694 (14,638-19,818) | 3,579 (2,966-3,622) | 5x | +| hnswlib | 3 | 128 | 0.963 | 12,743 (12,607-12,757) | 2,175 (2,074-2,226) | 6x | +| usearch | 3 | 128 | 0.963 | 13,656 (13,469-15,408) | 1,780 (1,730-1,816) | 8x | +| **text384 1,000,000** | | | | | | | +| ISONGraph Rust exact | 1 | - | 1.000 | 560 (1 run) | 11 (1 run) | 51x | +| faiss IndexFlatIP | 1 | - | 1.000 | 298 (1 run) | 11 (1 run) | 26x | +| numpy brute force | 1 | - | 1.000 | 61 (1 run) | 11 (1 run) | 5x | +| ISONGraph Rust IVF | 1 | 256 | 0.953 | 10,068 (1 run) | 131 (1 run) | 77x | +| ISONGraph Rust IVF + int8 x4 | 1 | 256 | 0.953 | 6,818 (1 run) | 223 (1 run) | 31x | +| faiss IVFFlat | 1 | 256 | 0.956 | 471 (1 run) | 165 (1 run) | 3x | +| ISONGraph Rust HNSW | 1 | 192 | 0.955 | 5,583 (1 run) | 1,714 (1 run) | 3x | +| faiss HNSWFlat | 1 | 192 | 0.967 | 4,967 (1 run) | 1,357 (1 run) | 4x | +| hnswlib | 1 | 192 | 0.955 | 6,680 (1 run) | 1,461 (1 run) | 5x | +| usearch | 1 | 192 | 0.956 | 3,973 (1 run) | 561 (1 run) | 7x | + +Rust concurrent clients (each caller runs single-scan-thread searches against one shared store; 3 s per client count, not pinned), QPS: + +| cell | mode | 1 | 2 | 4 | 8 | 16 | +|---|---|---|---|---|---|---| +| text384 100,000 | ISONGraph Rust exact | 57 | 83 | 183 | 341 | 604 | +| text384 100,000 | ISONGraph Rust IVF (knob 128) | 364 | 682 | 1,292 | 2,285 | 4,371 | +| text384 100,000 | ISONGraph Rust HNSW (knob 128) | 1,601 | 2,994 | 5,728 | 10,039 | 14,916 | +| text384 1,000,000 | ISONGraph Rust exact | 5 | 8 | 17 | 37 | 60 | +| text384 1,000,000 | ISONGraph Rust IVF (knob 256) | 129 | 216 | 335 | 419 | 385 | +| text384 1,000,000 | ISONGraph Rust HNSW (knob 192) | 1,336 | 3,037 | 3,765 | 5,535 | 5,343 | + +## Q4. int8 scan with exact rerank on real embeddings (exact search, Rust) + +`recall vs f32 scan` is `int8_recall_report` (int8 top-10 vs the library's own f32 scan, first 1,000 queries; 300 at 1M); `recall vs truth` is against the float64 ground truth in the timed loop. Speedup = int8 QPS / f32 QPS from the same process (1T = pinned single thread; auto = one query at a time on the automatic thread count; batch = 1,000-query batch, 32 threads). Medians over runs where several exist. + +| dataset | factor | recall vs f32 scan (mean / worst query) | recall vs truth | f32 1T QPS | int8 1T QPS | 1T speedup | auto-thread speedup | batch speedup | exact scores | +|---|---|---|---|---|---|---|---|---|---| +| text384 100,000 | x1 | 0.9904 / 0.80 | 0.9905 | 117 | 354 | 3.03x | 4.75x | 0.92x | True | +| text384 100,000 | x2 | 1.0000 / 1.00 | 0.9999 | 117 | 359 | 3.07x | 5.23x | 0.86x | True | +| text384 100,000 | x4 | 1.0000 / 1.00 | 0.9999 | 117 | 364 | 3.12x | 5.27x | 0.78x | True | +| text384 100,000 | x8 | 1.0000 / 1.00 | 0.9999 | 117 | 369 | 3.15x | 4.32x | 0.66x | True | +| text384 100,000 | memory | int8 copy 37 MB; working-set delta 38 MB | | | | | | | | +| text384 1,000,000 | x1 | 0.9917 / 0.90 | 0.9907 | 11 | 33 | 2.97x | 3.13x | 0.98x | True | +| text384 1,000,000 | x2 | 1.0000 / 1.00 | 1.0000 | 11 | 36 | 3.25x | 3.17x | 0.98x | True | +| text384 1,000,000 | x4 | 1.0000 / 1.00 | 1.0000 | 11 | 33 | 2.96x | 3.15x | 0.96x | True | +| text384 1,000,000 | x8 | 1.0000 / 1.00 | 1.0000 | 11 | 33 | 2.95x | 3.20x | 0.93x | True | +| text384 1,000,000 | memory | int8 copy 370 MB; working-set delta 371 MB | | | | | | | | +| glove 100,000 | x1 | 0.9832 / 0.80 | 0.9832 | 546 | 1,971 | 3.61x | 3.65x | 0.61x | True | +| glove 100,000 | x2 | 1.0000 / 1.00 | 1.0000 | 546 | 1,947 | 3.57x | 3.33x | 0.51x | True | +| glove 100,000 | x4 | 1.0000 / 1.00 | 1.0000 | 546 | 1,922 | 3.52x | 3.58x | 0.41x | True | +| glove 100,000 | x8 | 1.0000 / 1.00 | 1.0000 | 546 | 1,851 | 3.39x | 2.97x | 0.39x | True | +| glove 100,000 | memory | int8 copy 10 MB; working-set delta 8 MB | | | | | | | | +| sift 100,000 | x1 | 0.9763 / 0.80 | 0.9763 | 389 | 1,206 | 3.10x | 4.07x | 0.82x | True | +| sift 100,000 | x2 | 1.0000 / 1.00 | 1.0000 | 389 | 1,191 | 3.06x | 4.50x | 0.76x | True | +| sift 100,000 | x4 | 1.0000 / 1.00 | 1.0000 | 389 | 1,170 | 3.01x | 4.80x | 0.64x | True | +| sift 100,000 | x8 | 1.0000 / 1.00 | 1.0000 | 389 | 1,177 | 3.02x | 4.31x | 0.52x | True | +| sift 100,000 | memory | int8 copy 13 MB; working-set delta 12 MB | | | | | | | | + +IVF and HNSW combined with int8 (setting `ScanPrecision::Int8 { rerank_factor: 4 }` before the build) are in the Q2 table (`+ int8 x4` rows) with the sweeps above. + +## Q5. Persistence: ISONGraph JSON sidecar vs binary snapshot vs faiss / hnswlib / usearch index files + +ISONGraph = store only (vectors + keys), index not persisted. `ready` for the ANN rows adds the index rebuild (and a first query) that every ISONGraph load needs. Library rows are native `write_index` / `save_index` / `load`; their load time is to a usable index (the graph is inside the file). + +| N | system | save s | file MB | load s | first query after load ms | time to ready s | note | +|---|---|---|---|---|---|---|---| +| 100,000 | ISONGraph Rust JSON sidecar | 1.67 | 453 | 1.27 (runs 1.27, 1.27, 1.28) | 3.8 | 1.28 | exact search ready; answers equal original: True | +| 100,000 | ISONGraph Rust binary snapshot | 0.19 (+0.00 sidecar fingerprint) | 149 | 0.19 (runs 0.19, 0.19, 0.20) | 4.0 | 0.20 | exact search ready; answers equal original: True | +| 100,000 | ISONGraph Rust snapshot + HNSW rebuild | - | - | 0.19 | 0.2 | 6.12 | load + build 5.9 s + first query | +| 100,000 | ISONGraph Rust snapshot + IVF rebuild | - | - | 0.20 | 0.6 | 0.71 | load + build 0.5 s + first query | +| 100,000 | faiss IVFFlat | 0.061 (0.046-0.098) | 149 | 0.094 (0.087-0.170) | 1.24 | 0.095 | native index file | +| 100,000 | faiss HNSWFlat | 0.134 (0.128-0.253) | 176 | 0.091 (0.070-0.158) | 0.46 | 0.091 | native index file | +| 100,000 | hnswlib | 0.312 (0.290-0.367) | 177 | 0.595 (0.572-0.605) | 1.02 | 0.596 | native index file | +| 100,000 | usearch | 0.182 (0.160-0.331) | 177 | 0.139 (0.135-0.231) | 0.98 | 0.140 | native index file | +| 1,000,000 | ISONGraph Rust JSON sidecar | 42.86 | 4,534 | 30.12 (runs 30.12, 30.42, 26.38) | 50.3 | 30.17 | exact search ready; answers equal original: True | +| 1,000,000 | ISONGraph Rust binary snapshot | 7.68 (+0.00 sidecar fingerprint) | 1,489 | 4.56 (runs 4.40, 4.69, 4.56) | 43.6 | 4.60 | exact search ready; answers equal original: True | +| 1,000,000 | ISONGraph Rust snapshot + IVF rebuild | - | - | 4.56 | 31.1 | 21.62 | load + build 17.0 s + first query | +| 1,000,000 | faiss IVFFlat | 0.475 (1 run) | 1,478 | 0.848 (1 run) | 5.81 | 0.854 | native index file | +| 1,000,000 | faiss HNSWFlat | 1.154 (1 run) | 1,618 | 0.888 (1 run) | 1.08 | 0.889 | native index file | +| 1,000,000 | hnswlib | 2.529 (1 run) | 1,626 | 6.048 (1 run) | 3.08 | 6.051 | native index file | +| 1,000,000 | usearch | 3.247 (1 run) | 1,623 | 1.283 (1 run) | 1.80 | 1.285 | native index file | + +## Q6. Writes and the first query after writes (text384; index built, recall ~0.95 operating point) + +10k inserts, then 10k updates (new vector under an existing id), then 10k deletes, one call at a time on the built index (Rust: `add` / `remove`; libraries as in the competitive harness: calls of 1,000). Then the first search and recall@10 against the exact post-mutation float64 truth at the same knob. `p99/max` are per-write latencies (Rust only). + +| N | system | knob | insert/s | update/s | delete/s | insert p99 / max ms | update p99 / max ms | first query after ms | recall before | recall after | QPS 1T after | +|---|---|---|---|---|---|---|---|---|---|---|---| +| 100,000 | ISONGraph Rust IVF (1) | 128 | 156,140 | 128,696 | 911,295 | 0.01 / 2.1 | 0.01 / 2.2 | 3.8 | 0.9559 | 0.9529 | 857 | +| 100,000 | ISONGraph Rust HNSW (1) | 128 | 944 | 936 | 1,069,039 | 1.89 / 2.8 | 1.95 / 3.4 | 0.7 | 0.9626 | 0.9582 | 3,164 | +| 100,000 | faiss IVFFlat (3) | 128 | 235,791 | 263,826 | 2,687,522 | - | - | 1.0 | 0.9553 | 0.9521 | 1,185 | +| 100,000 | hnswlib (3) | 128 | 6,438 | 1,567 | 375,038 | - | - | 1.0 | 0.9631 | 0.9675 | 1,780 | +| 100,000 | usearch (3) | 128 | 7,693 | 7,682 | 3,068,897 | - | - | 3.7 | 0.9634 | 0.9573 | 1,494 | +| 100,000 | faiss HNSWFlat (3) | 96 | 4,352 | - (unsupported) | - (unsupported) | - | - | - | 0.9639 | - | - | +| 1,000,000 | ISONGraph Rust IVF (1) | 256 | 137,936 | 133,013 | 1,097,249 | 0.01 / 2.6 | 0.01 / 3.0 | 8.0 | 0.9526 | 0.9530 | 129 | +| 1,000,000 | ISONGraph Rust HNSW (1) | 192 | 1,393 | 1,440 | 1,382,170 | 1.49 / 4.7 | 1.35 / 6.6 | 0.6 | 0.9549 | 0.9550 | 1,603 | +| 1,000,000 | faiss IVFFlat (1) | 256 | 86,802 | 68,771 | 1,236,033 | - | - | 8.1 | 0.9561 | 0.9571 | 154 | +| 1,000,000 | hnswlib (1) | 192 | 6,005 | 1,696 | 355,494 | - | - | 1.3 | 0.9552 | 0.9557 | 1,377 | +| 1,000,000 | usearch (1) | 192 | 3,789 | 3,414 | 1,913,729 | - | - | 27.8 | 0.9560 | 0.9562 | 555 | +| 1,000,000 | faiss HNSWFlat (1) | 192 | 2,347 | - (unsupported) | - (unsupported) | - | - | - | 0.9668 | - | - | + +## Q7. Memory at 1M x 384 (working set, MB; baseline subtracted) + +Raw vectors = 1,000,000 x 384 x 4 = 1,465 MB. Each Rust row is its own process (ingest, then that mode only). Python rows: working set after the index is built and again after the sweep, with the harness' preallocation (hnswlib `max_elements` = N + 20k). + +| system | after ingest | after build / first query | peak during build | after the sweep | x raw vectors (after build) | +|---|---|---|---|---|---| +| **text384 100,000** (raw 146 MB) | | | | | | +| ISONGraph Rust exact | 161 | 164 | 188 | 166 | 1.12x | +| ISONGraph Rust IVF | 161 | 166 | 246 | 174 | 1.14x | +| ISONGraph Rust HNSW | 161 | 199 | 201 | 204 | 1.36x | +| ISONGraph Rust IVF + int8 x4 | 161 | 205 | 282 | 213 | 1.40x | +| ISONGraph Rust HNSW + int8 x4 (setting on) | 161 | 238 | 239 | 240 | 1.62x | +| faiss IndexFlatIP | - | 174 | 319 | 185 | 1.19x | +| numpy brute force | - | 159 | 191 | 168 | 1.09x | +| faiss IVFFlat | - | 240 | 461 | 232 | 1.64x | +| faiss HNSWFlat | - | 207 | 348 | 209 | 1.41x | +| hnswlib | - | 201 | 231 | 204 | 1.37x | +| usearch | - | 182 | 211 | 184 | 1.24x | +| ISONGraph Rust exact + int8 copy (any factor) | - | 203 | - | - | 1.39x | +| **text384 1,000,000** (raw 1,465 MB) | | | | | | +| ISONGraph Rust exact | 1,621 | 1,637 | 1,649 | 1,642 | 1.12x | +| ISONGraph Rust IVF | 1,621 | 1,643 | 1,886 | 1,653 | 1.12x | +| ISONGraph Rust HNSW | 1,621 | 1,786 | 1,794 | 1,795 | 1.22x | +| ISONGraph Rust IVF + int8 x4 | 1,621 | 2,011 | 2,255 | 2,058 | 1.37x | +| ISONGraph Rust HNSW + int8 x4 (setting on) | 1,622 | 2,156 | 2,164 | 2,156 | 1.47x | +| faiss IndexFlatIP | - | 1,555 | 2,124 | 1,565 | 1.06x | +| numpy brute force | - | 1,576 | 1,605 | 1,584 | 1.08x | +| faiss IVFFlat | - | 1,882 | 3,588 | 1,884 | 1.28x | +| faiss HNSWFlat | - | 1,816 | 2,298 | 1,827 | 1.24x | +| hnswlib | - | 1,876 | 1,906 | 1,878 | 1.28x | +| usearch | - | 1,714 | 1,744 | 1,716 | 1.17x | +| ISONGraph Rust exact + int8 copy (any factor) | - | 1,998 | - | - | 1.36x | + +Rust library-reported breakdown (`memory_stats`, 1M): + +* ISONGraph Rust IVF: matrix 1,466 MB, metadata 124 MB, index 17 MB, int8 copy 0 MB +* ISONGraph Rust HNSW: matrix 1,466 MB, metadata 124 MB, index 139 MB, int8 copy 0 MB +* ISONGraph Rust IVF + int8 x4: matrix 1,466 MB, metadata 124 MB, index 17 MB, int8 copy 370 MB +* ISONGraph Rust HNSW + int8 x4 (setting on): matrix 1,466 MB, metadata 124 MB, index 139 MB, int8 copy 370 MB + +## Appendix: noisy runs and machine load at job start + +| job file | total CPU % | pinned-CPU % | flagged noisy | +|---|---|---|---| +| faiss_flat_glove_100000_r1.json | 19 | 77 | True | +| faiss_flat_glove_100000_r3.json | 18 | 95 | True | +| faiss_flat_sift_100000_r2.json | 17 | 96 | True | +| faiss_hnsw_text384_100000_r2.json | 13 | 69 | True | +| faiss_ivfflat_text384_100000_r2.json | 19 | 69 | True | +| faiss_ivfflat_text384_100000_r3.json | 21 | 72 | True | +| numpy_bf_glove_100000_r3.json | 25 | 7 | False | +| rust_exact_text384_100000_r3.json | 18 | 83 | True | +| rust_hnsw_i8x4_text384_100000_r1.json | 12 | 75 | True | +| usearch_text384_100000_r3.json | 11 | 62 | True | + + +## How this was measured + +**Data and ground truth** are the ones of `benchmarks/competitive/` (`D:\Goldmine\ISON\bench_data\prep`): `text384` (real +HotpotQA passages embedded with MiniLM-L6-v2, 384-d, 2,000 real questions as queries), `glove` (100-d) and `sift` (128-d, +L2-normalised, cosine). Ground truth is the exact float64 cosine top-100; recall@10 is against it. The first 1,000 queries are +used everywhere. The vectors fed to every system are byte-identical. + +**What ran** (`run_stage_c.py` -> one process per cell, strictly one timed job at a time, never two at once): + +* Rust: `isongraph-vector-rs/examples/stage_c_bench.rs` (new in this stage; derived from `competitive_bench.rs`), built with + `cargo build --release` at HEAD 5767252 (default flags: AVX2/FMA kernels are chosen at run time; no `target-cpu=native`). + One process per (dataset, n, mode, precision) with `--mode exact|ivf|hnsw` and `--int8 4` for the int8 variants, so every + memory number is that mode alone. +* Libraries: `benchmarks/competitive/adapters.py` + `bench_py.py` through the wrapper `bench_c.py` (same code, same pinning, + same loops; only the accuracy-knob list is shortened: HNSW libs ef 24-400, faiss IVF nprobe 16-512). Equal parameters as in + the competitive benchmark: HNSW M=16, efConstruction=200; IVF nlist = 4*sqrt(N) (Rust and faiss). +* Single-thread numbers: process pinned to logical CPU 8 (a P-core) at high priority (`SetProcessAffinityMask` in Rust, + `psutil.cpu_affinity` in Python), `set_scan_threads(1)` / `omp_set_num_threads(1)` / `num_threads=1` / `threadpoolctl`; + each query timed individually; per sweep point: warm-up, then **three timed loops** (query count sized to a 3 s budget, at most + 1,000 queries), QPS = median of the three loops. +* Batch / all-core: one call with all 1,000 queries on 32 logical CPUs (24 cores, hybrid), warm-up call then 3 timed calls + (Rust `similarity_search_vector[_exact]_batch`, faiss `search` with 1,000 queries, hnswlib/usearch `knn_query`/`search` with + 32 threads, numpy BLAS matmul). +* **Process-level repeats**: text384 100k (all systems) and glove / sift 100k (exact) were run **3 times each, interleaved** (Rust, + numpy, faiss, ... then again), medians and min-max reported. The int8 study, the concurrent-client probe and the 10k-write test at + 100k ran in round 1 (the exact int8 study also in round 2 for text384). 1M cells ran **once**. +* Build time = ingest + index build + first query on each library's own path, all with 32 threads (Rust HNSW builds on the + pool; hnswlib/usearch `add_items`/`add` with 32 threads; faiss `add` with OpenMP 32). +* Memory = process working set (`GetProcessMemoryInfo` / psutil) after the step minus the baseline (before data for Rust, after + imports for Python). hnswlib is preallocated for N + 20k elements by the harness; numpy's matrix likewise. +* Interpolation to recall 0.95: log-linear in QPS between the two sweep points that bracket 0.95; nothing is extrapolated. + +**Python libraries are driven from a Python loop, Rust from a Rust loop.** Every one-query-at-a-time number for hnswlib, faiss, +usearch and numpy therefore includes the Python call (about 10-50 microseconds per query, 3-15% of a 0.3 ms HNSW query), +while Rust has none. That is the same bias the competitive benchmark had; it favours Rust slightly in the HNSW rows. A C++ driver +for hnswlib would remove it; none was built. + +## Machine state + +See `environment.json` (collected before the runs by `collect_env_c.py`). i9-14900HX laptop (8 P-cores with SMT + 16 E-cores = +32 logical CPUs), 47.7 GB, Windows 11, **Balanced** power plan, plugged-in state not recorded. **The machine was not idle in the +strict sense**: the user's desktop session was open (Chrome, VS Code, ChatGPT, Task Manager, a security-suite host process) and +the agent host itself. Mean CPU load over 15 s before the runs: 11.5% of 32 logical CPUs; the pinned CPU 8 was at 0%. Before every +job `run_stage_c.py` sampled total load and the load on CPU 8 for 4 s and waited up to 45 s for total < 30% and CPU 8 < 15% +(stored in `raw/*.load.json`); a job that still saw CPU 8 above 15% is flagged `noisy` (appendix of `tables.md`). Brief spikes on CPU 8 from desktop processes (not from this harness) are frequent on this laptop, so 22 of the first 48 jobs at 100k were flagged. Every 100k job flagged that way was moved to `raw/discarded_noisy/` and **re-run once** (`rerun_noisy.py`); the re-runs ran later in the session, so the original round-robin interleaving of Rust/numpy/faiss is only approximate for the re-run cells. 9 of the 60 jobs that count are still flagged (appendix of the tables), all at 100k; the four flagged 1M jobs were re-run once as well. The min-max shown in the tables is the honest spread. +Python library versions here differ from `benchmarks/competitive/results/environment.json` (numpy 2.4.2 vs 2.5.5, faiss-cpu +1.9.0 vs 1.16.0, usearch 2.26.0 vs 2.26.6); numbers in this stage are comparable only with each other. + +## Limitations, caveats, and what was dropped + +* **Not a quiet machine.** See "Machine state". Single-thread QPS of the same cell moved by up to 25% between runs (Rust exact text384 100k: 114 / 146 / 143 QPS; batch QPS of the same cell moved up to 2.5x). Treat differences below ~15% as parity. Where the table shows one run, there is one run. +* **Noisy-start jobs were re-run once.** 22 of 48 jobs in the first pass of the 100k cells started while CPU 8 showed > 15% load from desktop processes; all 100k jobs flagged that way were moved to `raw/discarded_noisy/` (kept, not used) and re-run once. Some re-runs were flagged again (9 of 60 jobs overall are still flagged; list in the appendix of the tables). The four flagged 1M jobs (Rust HNSW, Rust IVF, Rust IVF+int8, faiss HNSW) were re-run once as well; the first attempts are in `raw/discarded_noisy/` and differ a lot from the kept ones (Rust HNSW 1M QPS@0.95 1,531 vs 1,813; HNSW insert rate 650 vs 1,393/s; Rust IVF batch 4,524 vs 10,068 QPS; IVF build 18.8 vs 7.6 s), which is itself a measure of the noise on this machine: single-run 1M cells can move by 20-100% for batch, build and write rates. +* **1M cells are single runs**, 100k text384 cells are 3 process-level runs (int8 study, concurrent clients, 10k-write test: 1 run each, their re-runs were not repeated; the text384 100k int8 numbers come from one run). GloVe and SIFT at 1M were not run (not asked, dropped); chromadb, lancedb, sqlite-vec and qdrant-local were not re-run (outside the questions of this stage). +* **Libraries are driven from Python** (per-call overhead in every library one-query number, none in Rust). hnswlib/usearch/faiss writes were done in batches of 1,000 with parallel threads (the competitive harness), Rust writes are one `add` call at a time on one thread: the write table is not like for like and no per-call hnswlib/usearch measurement was made. +* **Recall at equal knobs differs between graph libraries** (faiss reaches 0.963 at ef 96, hnswlib and Rust at ef 128); that is why every comparison is at interpolated recall 0.95, not at equal ef. Interpolation brackets are as wide as the sweep (ef 24-400/640); at 1M the HNSW + int8 row has no bracket (both measured points are above 0.95) and is read from the ef-192 point (identical recall to the f32 run). +* **HNSW time-to-ready at 1M** after a snapshot load was not run as one measurement (it would repeat the 151 s build); the figure in the verdict is load + build measured separately. IVF 1M time-to-ready (21.6 s) was measured; its 17 s build there was slower than the 7-8 s IVF build in Stage A, and the 18.8 s build in `core` (one run each): IVF build time at 1M varies by more than 2x on this machine. +* **Build-time comparisons use 32 threads for everyone**; `faiss HNSWFlat` has no update/delete (shown as unsupported), memory of the libraries includes the harness' preallocation (N + 20k). +* The Python package versions differ from `benchmarks/competitive/results/environment.json`, so these numbers are not directly comparable with the earlier README's tables. +* `HNSW + int8` is reported as measured: accepted, no effect on search (identical recall at identical ef), memory of the int8 copy wasted. + +## Reproduce + +```text +cd isongraph-vector-rs +set CARGO_TARGET_DIR=D:/Goldmine/ISON/bench_data/rs_target_stagec +cargo build --release --example stage_c_bench +cd ../benchmarks/rust-stage-c +python collect_env_c.py # machine state -> environment.json +python run_stage_c.py --phase 1 --phase 2 --phase 4 # 100k cells (3 interleaved rounds), glove/sift exact, 100k persistence +python run_stage_c.py --phase 3 # text384 1M, one run per cell (about 70 minutes) +python rerun_noisy.py && python run_stage_c.py --phase 1 --phase 2 --phase 4 # repeat jobs that started under load +python summarize.py && python make_readme.py # raw/*.json -> tables.md, summary.json, README.md +``` +Files: `isongraph-vector-rs/examples/stage_c_bench.rs` (Rust harness), `bench_c.py` (library harness wrapper), `run_stage_c.py` (driver, load gating), +`rerun_noisy.py`, `summarize.py`, `make_readme.py`, `collect_env_c.py`, `environment.json`, `raw/` (one JSON + stdout + load record per run; +`raw/discarded_noisy/` holds the first attempts that were replaced), `tables.md`, `summary.json`. diff --git a/benchmarks/rust-stage-c/README_method.md b/benchmarks/rust-stage-c/README_method.md new file mode 100644 index 0000000..419e733 --- /dev/null +++ b/benchmarks/rust-stage-c/README_method.md @@ -0,0 +1,47 @@ +## How this was measured + +**Data and ground truth** are the ones of `benchmarks/competitive/` (`D:\Goldmine\ISON\bench_data\prep`): `text384` (real +HotpotQA passages embedded with MiniLM-L6-v2, 384-d, 2,000 real questions as queries), `glove` (100-d) and `sift` (128-d, +L2-normalised, cosine). Ground truth is the exact float64 cosine top-100; recall@10 is against it. The first 1,000 queries are +used everywhere. The vectors fed to every system are byte-identical. + +**What ran** (`run_stage_c.py` -> one process per cell, strictly one timed job at a time, never two at once): + +* Rust: `isongraph-vector-rs/examples/stage_c_bench.rs` (new in this stage; derived from `competitive_bench.rs`), built with + `cargo build --release` at HEAD 5767252 (default flags: AVX2/FMA kernels are chosen at run time; no `target-cpu=native`). + One process per (dataset, n, mode, precision) with `--mode exact|ivf|hnsw` and `--int8 4` for the int8 variants, so every + memory number is that mode alone. +* Libraries: `benchmarks/competitive/adapters.py` + `bench_py.py` through the wrapper `bench_c.py` (same code, same pinning, + same loops; only the accuracy-knob list is shortened: HNSW libs ef 24-400, faiss IVF nprobe 16-512). Equal parameters as in + the competitive benchmark: HNSW M=16, efConstruction=200; IVF nlist = 4*sqrt(N) (Rust and faiss). +* Single-thread numbers: process pinned to logical CPU 8 (a P-core) at high priority (`SetProcessAffinityMask` in Rust, + `psutil.cpu_affinity` in Python), `set_scan_threads(1)` / `omp_set_num_threads(1)` / `num_threads=1` / `threadpoolctl`; + each query timed individually; per sweep point: warm-up, then **three timed loops** (query count sized to a 3 s budget, at most + 1,000 queries), QPS = median of the three loops. +* Batch / all-core: one call with all 1,000 queries on 32 logical CPUs (24 cores, hybrid), warm-up call then 3 timed calls + (Rust `similarity_search_vector[_exact]_batch`, faiss `search` with 1,000 queries, hnswlib/usearch `knn_query`/`search` with + 32 threads, numpy BLAS matmul). +* **Process-level repeats**: text384 100k (all systems) and glove / sift 100k (exact) were run **3 times each, interleaved** (Rust, + numpy, faiss, ... then again), medians and min-max reported. The int8 study, the concurrent-client probe and the 10k-write test at + 100k ran in round 1 (the exact int8 study also in round 2 for text384). 1M cells ran **once**. +* Build time = ingest + index build + first query on each library's own path, all with 32 threads (Rust HNSW builds on the + pool; hnswlib/usearch `add_items`/`add` with 32 threads; faiss `add` with OpenMP 32). +* Memory = process working set (`GetProcessMemoryInfo` / psutil) after the step minus the baseline (before data for Rust, after + imports for Python). hnswlib is preallocated for N + 20k elements by the harness; numpy's matrix likewise. +* Interpolation to recall 0.95: log-linear in QPS between the two sweep points that bracket 0.95; nothing is extrapolated. + +**Python libraries are driven from a Python loop, Rust from a Rust loop.** Every one-query-at-a-time number for hnswlib, faiss, +usearch and numpy therefore includes the Python call (about 10-50 microseconds per query, 3-15% of a 0.3 ms HNSW query), +while Rust has none. That is the same bias the competitive benchmark had; it favours Rust slightly in the HNSW rows. A C++ driver +for hnswlib would remove it; none was built. + +## Machine state + +See `environment.json` (collected before the runs by `collect_env_c.py`). i9-14900HX laptop (8 P-cores with SMT + 16 E-cores = +32 logical CPUs), 47.7 GB, Windows 11, **Balanced** power plan, plugged-in state not recorded. **The machine was not idle in the +strict sense**: the user's desktop session was open (Chrome, VS Code, ChatGPT, Task Manager, a security-suite host process) and +the agent host itself. Mean CPU load over 15 s before the runs: 11.5% of 32 logical CPUs; the pinned CPU 8 was at 0%. Before every +job `run_stage_c.py` sampled total load and the load on CPU 8 for 4 s and waited up to 45 s for total < 30% and CPU 8 < 15% +(stored in `raw/*.load.json`); a job that still saw CPU 8 above 15% is flagged `noisy` (appendix of `tables.md`). Brief spikes on CPU 8 from desktop processes (not from this harness) are frequent on this laptop, so 22 of the first 48 jobs at 100k were flagged. Every 100k job flagged that way was moved to `raw/discarded_noisy/` and **re-run once** (`rerun_noisy.py`); the re-runs ran later in the session, so the original round-robin interleaving of Rust/numpy/faiss is only approximate for the re-run cells. 9 of the 60 jobs that count are still flagged (appendix of the tables), all at 100k; the four flagged 1M jobs were re-run once as well. The min-max shown in the tables is the honest spread. +Python library versions here differ from `benchmarks/competitive/results/environment.json` (numpy 2.4.2 vs 2.5.5, faiss-cpu +1.9.0 vs 1.16.0, usearch 2.26.0 vs 2.26.6); numbers in this stage are comparable only with each other. diff --git a/benchmarks/rust-stage-c/README_tail.md b/benchmarks/rust-stage-c/README_tail.md new file mode 100644 index 0000000..4df5c76 --- /dev/null +++ b/benchmarks/rust-stage-c/README_tail.md @@ -0,0 +1,29 @@ + +## Limitations, caveats, and what was dropped + +* **Not a quiet machine.** See "Machine state". Single-thread QPS of the same cell moved by up to 25% between runs (Rust exact text384 100k: 114 / 146 / 143 QPS; batch QPS of the same cell moved up to 2.5x). Treat differences below ~15% as parity. Where the table shows one run, there is one run. +* **Noisy-start jobs were re-run once.** 22 of 48 jobs in the first pass of the 100k cells started while CPU 8 showed > 15% load from desktop processes; all 100k jobs flagged that way were moved to `raw/discarded_noisy/` (kept, not used) and re-run once. Some re-runs were flagged again (9 of 60 jobs overall are still flagged; list in the appendix of the tables). The four flagged 1M jobs (Rust HNSW, Rust IVF, Rust IVF+int8, faiss HNSW) were re-run once as well; the first attempts are in `raw/discarded_noisy/` and differ a lot from the kept ones (Rust HNSW 1M QPS@0.95 1,531 vs 1,813; HNSW insert rate 650 vs 1,393/s; Rust IVF batch 4,524 vs 10,068 QPS; IVF build 18.8 vs 7.6 s), which is itself a measure of the noise on this machine: single-run 1M cells can move by 20-100% for batch, build and write rates. +* **1M cells are single runs**, 100k text384 cells are 3 process-level runs (int8 study, concurrent clients, 10k-write test: 1 run each, their re-runs were not repeated; the text384 100k int8 numbers come from one run). GloVe and SIFT at 1M were not run (not asked, dropped); chromadb, lancedb, sqlite-vec and qdrant-local were not re-run (outside the questions of this stage). +* **Libraries are driven from Python** (per-call overhead in every library one-query number, none in Rust). hnswlib/usearch/faiss writes were done in batches of 1,000 with parallel threads (the competitive harness), Rust writes are one `add` call at a time on one thread: the write table is not like for like and no per-call hnswlib/usearch measurement was made. +* **Recall at equal knobs differs between graph libraries** (faiss reaches 0.963 at ef 96, hnswlib and Rust at ef 128); that is why every comparison is at interpolated recall 0.95, not at equal ef. Interpolation brackets are as wide as the sweep (ef 24-400/640); at 1M the HNSW + int8 row has no bracket (both measured points are above 0.95) and is read from the ef-192 point (identical recall to the f32 run). +* **HNSW time-to-ready at 1M** after a snapshot load was not run as one measurement (it would repeat the 151 s build); the figure in the verdict is load + build measured separately. IVF 1M time-to-ready (21.6 s) was measured; its 17 s build there was slower than the 7-8 s IVF build in Stage A, and the 18.8 s build in `core` (one run each): IVF build time at 1M varies by more than 2x on this machine. +* **Build-time comparisons use 32 threads for everyone**; `faiss HNSWFlat` has no update/delete (shown as unsupported), memory of the libraries includes the harness' preallocation (N + 20k). +* The Python package versions differ from `benchmarks/competitive/results/environment.json`, so these numbers are not directly comparable with the earlier README's tables. +* `HNSW + int8` is reported as measured: accepted, no effect on search (identical recall at identical ef), memory of the int8 copy wasted. + +## Reproduce + +```text +cd isongraph-vector-rs +set CARGO_TARGET_DIR=D:/Goldmine/ISON/bench_data/rs_target_stagec +cargo build --release --example stage_c_bench +cd ../benchmarks/rust-stage-c +python collect_env_c.py # machine state -> environment.json +python run_stage_c.py --phase 1 --phase 2 --phase 4 # 100k cells (3 interleaved rounds), glove/sift exact, 100k persistence +python run_stage_c.py --phase 3 # text384 1M, one run per cell (about 70 minutes) +python rerun_noisy.py && python run_stage_c.py --phase 1 --phase 2 --phase 4 # repeat jobs that started under load +python summarize.py && python make_readme.py # raw/*.json -> tables.md, summary.json, README.md +``` +Files: `isongraph-vector-rs/examples/stage_c_bench.rs` (Rust harness), `bench_c.py` (library harness wrapper), `run_stage_c.py` (driver, load gating), +`rerun_noisy.py`, `summarize.py`, `make_readme.py`, `collect_env_c.py`, `environment.json`, `raw/` (one JSON + stdout + load record per run; +`raw/discarded_noisy/` holds the first attempts that were replaced), `tables.md`, `summary.json`. diff --git a/benchmarks/rust-stage-c/README_top.md b/benchmarks/rust-stage-c/README_top.md new file mode 100644 index 0000000..65765af --- /dev/null +++ b/benchmarks/rust-stage-c/README_top.md @@ -0,0 +1,60 @@ +# Stage C: honest re-benchmark of the reworked Rust crate + +Crate `isongraph-vector-rs` at HEAD 5767252 (branch `feature/ann-ports-and-server`), measured against numpy, faiss +(IndexFlatIP / IVFFlat / HNSWFlat), hnswlib and usearch on the data and ground truth of `benchmarks/competitive/`. +Everything below is regenerated from `raw/` by `summarize.py`; nothing was carried over from earlier READMEs. + +## Verdict (one paragraph) + +The reworked Rust crate is now a credible ANN engine, but the old "fastest exact scan" claim does not survive a same-session +re-measurement. **Exact search is at parity, not ahead**: single-thread Rust vs faiss IndexFlatIP is 0.91-1.2x depending on the +dataset and run (it is slower than faiss on GloVe and SIFT 100k, equal at text384 1M: 11.1 vs 11.4 QPS), and it beats numpy +only on GloVe (1.45x) and, within run noise, text384 100k (1.27x); at 1M and on SIFT it ties numpy. **HNSW is the big change**: +at recall@10 0.95 it matches faiss HNSWFlat and hnswlib single-thread (text384 100k: 3,893 vs 3,965 / 2,723 QPS; 1M: 1,813 vs +1,773 / 1,517; one run each, and the discarded first attempt of the Rust cell gave 1,531), beats usearch (2,070 / 612), builds in about the same time +(145-173 s vs 157-216 s at 1M) and uses about the same memory. IVF is still slower than +faiss IVFFlat at equal recall (0.71x at 100k, 0.76x at 1M), but IVF+int8 beats it (1.9x / 1.3x). The Rust batch API is a clear +win for IVF (8x faiss IVFFlat on 32 threads at 100k, 10-21x at 1M) and level with the HNSW libraries; int8 gives a 3.0-3.6x single-thread speedup with +recall 1.000 against the f32 scan from factor 2 on real embeddings, and does nothing useful for batches. The binary snapshot +loads 6.6x faster than the JSON sidecar and is 3x smaller, but an ANN index is still never persisted: time-to-ready is 21.6 s +(IVF) / about 2.5-3 minutes (HNSW) at 1M, against about 1 s for faiss and usearch. Single-call HNSW inserts are slow (1,393/s at 1M, +versus 3,789-6,005/s for hnswlib/usearch, which were driven in parallel batches of 1,000); HNSW updates are on par with hnswlib (1,440 vs 1,696/s). The machine was a laptop with the +user's desktop apps running, not a quiet bench: run-to-run spread is visible (20-25% on one-query QPS, up to 2x on batch, build and write rates between the first and the repeated attempt of the same 1M cell), so differences under about 15% are +reported as parity. + +## Answers + +| # | question | answer | evidence (details in the tables) | +|---|---|---|---| +| 1 | Rust 1T exact still faster than numpy and faiss Flat (text384 100k/1M, glove 100k, sift 100k)? | **Refuted** as stated. Parity with faiss; slight edge over numpy on some cells | faiss Flat/Rust: text 100k 117.8 / 142.6 (Rust runs 114-146, faiss 117-136: overlap); text 1M 11.4 / 11.1; glove 605 / 548 (ranges 604-622 vs 537-548: faiss wins by ~10%); sift 432 / 397 (ranges overlap). numpy: 113 / 11.1 / 377 / 386 vs Rust 143 / 11.1 / 548 / 397 | +| 2 | HNSW vs hnswlib / faiss HNSW / usearch at recall 0.95, 1T | **Confirmed: parity with the best (at 1M ahead of hnswlib by 19% in one run), well ahead of usearch.** IVF **refuted** (0.71-0.76x faiss IVFFlat); IVF+int8 **confirmed faster** than faiss IVFFlat (1.3-1.9x); HNSW+int8 is accepted but a **no-op that costs the int8 copy's memory (+370 MB at 1M)** | Q2 table; 100k 3 runs, 1M 1 run | +| 3 | Batch / all-core vs faiss Flat batch and HNSW libs at 100k | **Confirmed for IVF (8x faiss IVFFlat at 100k, 10-21x at 1M); inconclusive for exact (5,320 vs 5,085, run ranges overlap; 560 vs 298 at 1M, 1 run); parity for HNSW (14.0k vs 16.7k faiss, 12.7k hnswlib, 13.7k usearch)** | Q3 table; batch timings varied up to 2.5x between runs | +| 4 | int8 recall on real embeddings, factors 1/2/4/8, speedup | **Confirmed**: factor >= 2 gives recall 1.0000 vs f32 scan on text384 (100k, 1M), GloVe and SIFT; factor 1 gives 0.976-0.992 (worst query 0.80). 1T speedup 3.0-3.6x. **Batch: no benefit (0.4-1.0x (slower than f32 in every measured batch))** | Q4 table | +| 5 | Persistence | Snapshot vs JSON: **confirmed big win** (1M load 4.6 s vs 30 s, save 7.7 s vs 42.9 s, 1,489 vs 4,534 MB). Competitive with faiss/hnswlib for a ready-to-search ANN index: **refuted** (index not persisted) | Q5 table | +| 6 | Writes at 1M, IVF and HNSW vs hnswlib/usearch | IVF: **faster than faiss IVFFlat on inserts/updates** (138k / 133k per s vs 87k / 69k; first query after writes 8.0 vs 8.1 ms). HNSW single-call inserts **slower** than hnswlib/usearch batch inserts (1,393 vs 6,005 / 3,789 per s), updates on par (1,440 vs 1,696 / 3,414); first query after writes 0.6 ms and recall holds (0.9549 -> 0.9550) | Q6 table | +| 7 | Memory at 1M x 384 | exact 1.12x raw, IVF 1.12x, HNSW 1.22x (faiss HNSW 1.24x, hnswlib 1.28x, usearch 1.17x, faiss IVF 1.28x); int8 adds 371 MB. **Confirmed no worse than the libraries** | Q7 table | + +## What we win and lose now (each line is exactly what the data supports) + +**Win** +* Single-thread HNSW at recall@10 ~0.95 on text384 is level with faiss HNSWFlat and 19% above hnswlib at 1M (1,813 vs 1,773 / 1,517 QPS, one run each) and 43% above hnswlib, level with faiss, at 100k (3 runs; faiss 3,965, hnswlib 2,723). The one-query loop for the libraries runs through Python, Rust's does not: part of the 100k margin over hnswlib may be that overhead. +* IVF batch search on 32 threads: 24,360 QPS vs faiss IVFFlat 3,014 at 100k (recall ~0.955, 3 runs, ranges 21-25k vs 2.9-3.6k) and 10,068 vs 471 at 1M (1 run; 4,524 in the discarded noisy first attempt). +* IVF + int8 (rerank 4) single-thread at recall 0.955: 2,059 QPS vs faiss IVFFlat f32 1,065 at 100k (1 run), 235 vs 180 at 1M (1 run). +* int8 exact scan: 3.0-3.6x single-thread QPS with exact returned scores and recall 1.0000 vs the f32 scan at factor >= 2 on all four real datasets; the copy costs 25% of the vector bytes. +* Binary snapshot load: 6.6x faster than the JSON sidecar at 100k and 1M (0.19 vs 1.27 s; 4.6 vs 30 s) and 3x smaller. +* Index build: IVF 7.6 s vs faiss 35.3 s at 1M (1 run; 18.8 s in the discarded first attempt); HNSW is level: 173 s vs hnswlib 157 s, faiss 190 s, usearch 216 s at 1M (1 run; 152 s in the discarded attempt); 7.1 s vs 8.4 / 9.6 / 6.8 s at 100k (3 runs). +* IVF memory: 1,643 MB working set at 1M vs faiss IVFFlat 1,881 MB, and a 1,886 MB build peak vs faiss 3,588 MB. +* IVF writes at 1M: 138k inserts/s and 133k updates/s one call at a time vs faiss IVFFlat 87k / 69k (1 run each; 81k / 69k in the discarded first attempt); first query after 30k writes 8.0 ms, recall 0.9526 -> 0.9530. +* 8 concurrent clients scale (HNSW 1M: 700 -> 4,990 QPS from 1 to 8 clients); numpy/faiss searches are not compared here. + +**Lose / parity** +* Exact single-thread scan: no longer ahead. Equal to faiss Flat and numpy at text384 1M (11.1 / 11.4 / 11.1); 9-10% slower than faiss Flat on GloVe 100k (548 vs 605); not distinguishable on text384 100k and SIFT 100k. +* IVF (f32) at equal recall: 0.71x faiss IVFFlat at 100k (759 vs 1,065), 0.76x at 1M (137 vs 180; 108 in the discarded first attempt). +* ANN indexes are not persisted: after a snapshot load, IVF is ready in 0.7 s (100k) / 21.6 s (1M) and HNSW in 6.1 s (100k) / about 2.5-3 minutes (1M: load 4.6 s + a 145-173 s build, not run as one measurement); faiss/usearch are ready in 0.1-1.3 s. +* HNSW insert rate: 944/s at 100k and 1,393/s at 1M one call at a time (650/s in the discarded 1M first attempt) against hnswlib 6,005/s and usearch 3,789/s in batches of 1,000 on 32 threads. Not like for like, but it is what a caller gets. HNSW updates (1,440/s at 1M) are on par with hnswlib (1,696/s) and below usearch (3,414/s). +* int8 does nothing for batches (0.4-1.0x (slower than f32 in every measured batch)) and does nothing for HNSW (setting accepted, search path unchanged, +370 MB at 1M). +* JSON save/load is still 43 s / 30 s at 1M (4.5 GB). +* Exact batch (5,320 vs faiss Flat 5,085 at 100k): not distinguishable; at 1M 560 vs 298 (1 run). + +**Anything surprising / possible bug**: `set_scan_precision(Int8)` together with `AnnConfig::hnsw()` is silently accepted. Searches never use the int8 copy (identical recall at identical ef: 0.9626 / 0.9549; QPS equal within noise), yet 37 MB (100k) / 370 MB (1M) are allocated. Not a correctness bug, but neither the README nor DESIGN.md says it, and it wastes memory. Repro: `stage_c_bench --mode hnsw --int8 4`, compare `mem_after_build` with `--mode hnsw`. No wrong answers, crashes or recall regressions were found anywhere (every int8 score checked bit-exact; every snapshot/JSON load returned the same top-10 as the original store; recall after the 30k writes within 0.005 of before at every operating point). + diff --git a/benchmarks/rust-stage-c/bench_c.py b/benchmarks/rust-stage-c/bench_c.py new file mode 100644 index 0000000..960045e --- /dev/null +++ b/benchmarks/rust-stage-c/bench_c.py @@ -0,0 +1,25 @@ +"""Thin wrapper around benchmarks/competitive/bench_py.py for Stage C. + +Same adapters, same pinning (logical CPU 8), same timed loops, same ground truth; the only change is a shorter +accuracy-knob list for the ANN libraries (all the points that matter around recall 0.95), so a cell takes +minutes, not tens of minutes. Nothing under benchmarks/competitive/ is modified. + + python bench_c.py --system hnswlib --ds text384 --n 100000 --exp core,mutation,saveload --out raw/x.json +""" +import os +import sys + +HERE = os.path.dirname(os.path.abspath(__file__)) +COMP = os.path.join(os.path.dirname(HERE), "competitive") +sys.path.insert(0, COMP) +import adapters # noqa: E402 + +HN = [24, 32, 48, 64, 96, 128, 192, 256, 400] +IVF = [16, 32, 64, 128, 256, 512] +for name, kn in (("faiss_hnsw", HN), ("hnswlib", HN), ("usearch", HN), ("faiss_ivfflat", IVF)): + adapters.ADAPTERS[name].knobs = (lambda self, n, kn=kn: list(kn)) + +import bench_py # noqa: E402 + +if __name__ == "__main__": + bench_py.main() diff --git a/benchmarks/rust-stage-c/collect_env_c.py b/benchmarks/rust-stage-c/collect_env_c.py new file mode 100644 index 0000000..b3fbf32 --- /dev/null +++ b/benchmarks/rust-stage-c/collect_env_c.py @@ -0,0 +1,53 @@ +"""Machine state for the Stage C README: CPU, library versions, background load sample, top processes.""" +import json +import platform +import subprocess +import sys +import time +from importlib import metadata + +import psutil + +def ver(p): + try: + return metadata.version(p) + except Exception: # noqa: BLE001 + return None + +per = psutil.cpu_percent(interval=10, percpu=True) +procs = [] +for p in psutil.process_iter(["name"]): + try: + p.cpu_percent(None) + procs.append(p) + except Exception: # noqa: BLE001 + pass +time.sleep(5) +rows = [] +for p in procs: + try: + rows.append((p.cpu_percent(None), p.info["name"])) + except Exception: # noqa: BLE001 + pass +top = sorted(rows, reverse=True)[:8] +def cmd(c): + try: + return subprocess.run(c, capture_output=True, text=True, timeout=20).stdout.strip() + except Exception as e: # noqa: BLE001 + return str(e) +env = { + "when": time.strftime("%Y-%m-%d %H:%M:%S"), + "python": sys.version.split()[0], "platform": platform.platform(), + "cpu": "Intel Core i9-14900HX (8 P-cores with SMT + 16 E-cores = 24 cores / 32 logical)", + "logical_cpus": psutil.cpu_count(), "ram_gb": round(psutil.virtual_memory().total / 2**30, 1), + "ram_available_gb": round(psutil.virtual_memory().available / 2**30, 1), + "packages": {p: ver(p) for p in ("numpy", "faiss-cpu", "hnswlib", "usearch", "chromadb", "lancedb", "sqlite-vec", "psutil", "threadpoolctl")}, + "rustc": cmd(["rustc", "--version"]), "cargo": cmd(["cargo", "--version"]), + "rust_build": "cargo build --release --example stage_c_bench (default flags, CARGO_TARGET_DIR=bench_data/rs_target_stagec); AVX2/FMA kernels chosen at run time", + "load_sample_per_cpu_pct_15s": [round(x, 1) for x in per], + "load_mean_pct": round(sum(per) / len(per), 1), "pin_cpu8_pct": per[8], + "top_processes_cpu_pct_5s": [[round(c, 1), n] for c, n in top], + "power": cmd(["powercfg", "/getactivescheme"]), +} +json.dump(env, open("environment.json", "w"), indent=1) +print(json.dumps(env, indent=1)) diff --git a/benchmarks/rust-stage-c/environment.json b/benchmarks/rust-stage-c/environment.json new file mode 100644 index 0000000..4f39abd --- /dev/null +++ b/benchmarks/rust-stage-c/environment.json @@ -0,0 +1,94 @@ +{ + "when": "2026-11-16 19:00:49", + "python": "3.12.7", + "platform": "Windows-11-10.0.26300-SP0", + "cpu": "Intel Core i9-14900HX (8 P-cores with SMT + 16 E-cores = 24 cores / 32 logical)", + "logical_cpus": 32, + "ram_gb": 47.7, + "ram_available_gb": 14.3, + "packages": { + "numpy": "2.4.2", + "faiss-cpu": "1.9.0", + "hnswlib": null, + "usearch": "2.26.0", + "chromadb": "1.3.5", + "lancedb": "0.24.3", + "sqlite-vec": "0.1.9", + "psutil": "7.0.0", + "threadpoolctl": "3.5.0" + }, + "rustc": "rustc 1.97.1 (8bab26f4f 2026-07-14)", + "cargo": "cargo 1.97.1 (c980f4866 2026-06-30)", + "rust_build": "cargo build --release --example stage_c_bench (default flags, CARGO_TARGET_DIR=bench_data/rs_target_stagec); AVX2/FMA kernels chosen at run time", + "load_sample_per_cpu_pct_15s": [ + 21.8, + 0.0, + 31.1, + 0.0, + 11.9, + 0.0, + 7.8, + 0.0, + 0.0, + 71.8, + 78.8, + 0.0, + 0.0, + 25.2, + 23.7, + 0.0, + 0.0, + 30.0, + 7.2, + 1.3, + 26.9, + 4.9, + 0.5, + 1.3, + 13.6, + 2.4, + 0.5, + 0.5, + 7.3, + 0.0, + 0.2, + 0.9 + ], + "load_mean_pct": 11.5, + "pin_cpu8_pct": 0.0, + "top_processes_cpu_pct_5s": [ + [ + 2586.1, + "System Idle Process" + ], + [ + 103.6, + "claude.exe" + ], + [ + 57.7, + "System" + ], + [ + 38.9, + "mc-fw-host.exe" + ], + [ + 31.0, + "python.exe" + ], + [ + 28.2, + "Taskmgr.exe" + ], + [ + 14.4, + "skeleton.test.exe" + ], + [ + 10.8, + "audiodg.exe" + ] + ], + "power": "Power Scheme GUID: 381b4222-f694-41f0-9685-ff5bb260df2e (Balanced)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/make_readme.py b/benchmarks/rust-stage-c/make_readme.py new file mode 100644 index 0000000..6a59a2e --- /dev/null +++ b/benchmarks/rust-stage-c/make_readme.py @@ -0,0 +1,13 @@ +"""Assemble README.md from README_top.md (hand-written verdict), tables.md (summarize.py), README_method.md, README_tail.md.""" +import os + +H = os.path.dirname(os.path.abspath(__file__)) + + +def r(n): + return open(os.path.join(H, n), encoding="utf-8").read() + + +mid = "\n\n# Tables\n\nGenerated by `summarize.py` from `raw/`. Q1-Q7 match the questions of the stage.\n\n" +open(os.path.join(H, "README.md"), "w", encoding="utf-8").write( + r("README_top.md") + mid + r("tables.md") + "\n\n" + r("README_method.md") + r("README_tail.md")) diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_glove_100000_r1.json b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_glove_100000_r1.json new file mode 100644 index 0000000..b92d407 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_glove_100000_r1.json @@ -0,0 +1,87 @@ +{ + "system": "faiss_flat", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 19:55:27", + "mem_baseline": { + "rss_mb": 34.890625, + "peak_mb": 34.890625 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 0.03245159989455715, + "index_s": 3.100023604929447e-06, + "total_s": 0.03245469991816208, + "first_query_ms": 5.960699985735118, + "ready_s": 0.038415399903897196 + }, + "mem_after_build": { + "rss_mb": 100.34765625, + "peak_mb": 137.70703125 + }, + "mem_after_first_query": { + "rss_mb": 100.4609375, + "peak_mb": 137.70703125 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 660.6492098577073, + 575.9732304013843, + 483.8658794970514 + ], + "qps_1t": 575.9732304013843, + "qps_1t_min": 483.8658794970514, + "qps_1t_max": 660.6492098577073, + "median_ms": 1.6955499886535108, + "p95_ms": 2.218310011085123, + "p99_ms": 2.3957280226750295, + "mean_ms": 1.7721809329814278, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 5388.340386440237, + 7135.10026911901, + 6315.671138133753 + ], + "qps_batch": 6315.671138133753, + "qps_batch_min": 5388.340386440237, + "qps_batch_max": 7135.10026911901, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 477, + "threads": 32, + "qps": 179.6262350906939, + "median_ms": 5.307999963406473, + "p95_ms": 7.3647500248625875, + "p99_ms": 8.57099000713788, + "mean_ms": 5.434750664270152 + }, + "mem_after_sweep": { + "rss_mb": 110.89453125, + "peak_mb": 137.70703125 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 109.37109375, + "peak_mb": 137.70703125 + }, + "status": "ok", + "finished": "2026-11-16 19:55:41" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_glove_100000_r1.load.json b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_glove_100000_r1.load.json new file mode 100644 index 0000000..ef1406c --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_glove_100000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 14.475, "pin_cpu_pct": 26.4, "max_cpu_pct": 68.2, "noisy": true, "seconds": 14.576598167419434, "after": 14.2} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_glove_100000_r1.txt b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_glove_100000_r1.txt new file mode 100644 index 0000000..ff7eed1 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_glove_100000_r1.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_flat --ds glove --n 100000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_flat_glove_100000_r1.json + faiss_flat glove 100000 knob=None recall=1.0000 qps1t=576 med=1.696ms batch=6316 diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_glove_100000_r3.json b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_glove_100000_r3.json new file mode 100644 index 0000000..6f8f635 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_glove_100000_r3.json @@ -0,0 +1,87 @@ +{ + "system": "faiss_flat", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 20:01:28", + "mem_baseline": { + "rss_mb": 34.88671875, + "peak_mb": 34.88671875 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 0.034381299919914454, + "index_s": 2.900022082030773e-06, + "total_s": 0.034384199941996485, + "first_query_ms": 20.42259997688234, + "ready_s": 0.054806799918878824 + }, + "mem_after_build": { + "rss_mb": 100.41015625, + "peak_mb": 137.6875 + }, + "mem_after_first_query": { + "rss_mb": 100.52734375, + "peak_mb": 137.6875 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 618.3059492255824, + 576.631035315811, + 487.8039500272135 + ], + "qps_1t": 576.631035315811, + "qps_1t_min": 487.8039500272135, + "qps_1t_max": 618.3059492255824, + "median_ms": 1.753599994117394, + "p95_ms": 2.2261800098931412, + "p99_ms": 2.4307049985509366, + "mean_ms": 1.8005124668901165, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 5450.791345107572, + 6680.937149749665, + 6099.301508637006 + ], + "qps_batch": 6099.301508637006, + "qps_batch_min": 5450.791345107572, + "qps_batch_max": 6680.937149749665, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 772, + "threads": 32, + "qps": 210.6874943506204, + "median_ms": 4.643849999411032, + "p95_ms": 6.22339999245014, + "p99_ms": 7.2876400110544495, + "mean_ms": 4.785676035479745 + }, + "mem_after_sweep": { + "rss_mb": 110.6484375, + "peak_mb": 137.6875 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 109.125, + "peak_mb": 137.6875 + }, + "status": "ok", + "finished": "2026-11-16 20:01:46" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_glove_100000_r3.load.json b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_glove_100000_r3.load.json new file mode 100644 index 0000000..5bb4800 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_glove_100000_r3.load.json @@ -0,0 +1 @@ +{"total_pct": 26.509375, "pin_cpu_pct": 73.4, "max_cpu_pct": 77.6, "noisy": true, "seconds": 17.991718530654907, "after": 21.4} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_glove_100000_r3.txt b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_glove_100000_r3.txt new file mode 100644 index 0000000..0c30876 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_glove_100000_r3.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_flat --ds glove --n 100000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_flat_glove_100000_r3.json + faiss_flat glove 100000 knob=None recall=1.0000 qps1t=577 med=1.754ms batch=6099 diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_sift_100000_r2.json b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_sift_100000_r2.json new file mode 100644 index 0000000..5a7c628 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_sift_100000_r2.json @@ -0,0 +1,87 @@ +{ + "system": "faiss_flat", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 20:07:28", + "mem_baseline": { + "rss_mb": 35.1875, + "peak_mb": 35.1875 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 0.038341600040439516, + "index_s": 2.100015990436077e-06, + "total_s": 0.03834370005642995, + "first_query_ms": 8.860200003255159, + "ready_s": 0.04720390005968511 + }, + "mem_after_build": { + "rss_mb": 111.0546875, + "peak_mb": 158.95703125 + }, + "mem_after_first_query": { + "rss_mb": 111.171875, + "peak_mb": 158.95703125 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 544.5770338977912, + 446.9707030572122, + 406.1510602263066 + ], + "qps_1t": 446.9707030572122, + "qps_1t_min": 406.1510602263066, + "qps_1t_max": 544.5770338977912, + "median_ms": 2.2770999930799007, + "p95_ms": 2.6858300203457475, + "p99_ms": 2.86931496229954, + "mean_ms": 2.1785695666912943, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 4700.370201157883, + 6020.031051066025, + 6168.209541260229 + ], + "qps_batch": 6020.031051066025, + "qps_batch_min": 4700.370201157883, + "qps_batch_max": 6168.209541260229, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 471, + "threads": 32, + "qps": 154.18099583373214, + "median_ms": 6.294099963270128, + "p95_ms": 9.053700009826567, + "p99_ms": 10.937744025140997, + "mean_ms": 6.556235243718259 + }, + "mem_after_sweep": { + "rss_mb": 121.28125, + "peak_mb": 158.95703125 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 119.54296875, + "peak_mb": 158.95703125 + }, + "status": "ok", + "finished": "2026-11-16 20:07:46" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_sift_100000_r2.load.json b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_sift_100000_r2.load.json new file mode 100644 index 0000000..e391845 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_sift_100000_r2.load.json @@ -0,0 +1 @@ +{"total_pct": 12.9875, "pin_cpu_pct": 37.5, "max_cpu_pct": 51.8, "noisy": true, "seconds": 17.559574127197266, "after": 13.2} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_sift_100000_r2.txt b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_sift_100000_r2.txt new file mode 100644 index 0000000..c8e96a4 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_sift_100000_r2.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_flat --ds sift --n 100000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_flat_sift_100000_r2.json + faiss_flat sift 100000 knob=None recall=1.0000 qps1t=447 med=2.277ms batch=6020 diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_text384_100000_r3.json b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_text384_100000_r3.json new file mode 100644 index 0000000..6723643 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_text384_100000_r3.json @@ -0,0 +1,87 @@ +{ + "system": "faiss_flat", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 19:45:37", + "mem_baseline": { + "rss_mb": 37.05078125, + "peak_mb": 37.05078125 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 0.10585339996032417, + "index_s": 2.7999631129205227e-06, + "total_s": 0.10585619992343709, + "first_query_ms": 17.790500016417354, + "ready_s": 0.12364669993985444 + }, + "mem_after_build": { + "rss_mb": 210.91796875, + "peak_mb": 356.50390625 + }, + "mem_after_first_query": { + "rss_mb": 211.03515625, + "peak_mb": 356.50390625 + }, + "sweep": [ + { + "nq": 453, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 131.41923109252477, + 120.37255542458972, + 112.85982384831617 + ], + "qps_1t": 120.37255542458972, + "qps_1t_min": 112.85982384831617, + "qps_1t_max": 131.41923109252477, + "median_ms": 8.40079999761656, + "p95_ms": 9.440829994855449, + "p99_ms": 9.95619397610426, + "mean_ms": 8.259108977474895, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 2088.0113487591625, + 1791.4084052681533, + 2942.581120235563 + ], + "qps_batch": 2088.0113487591625, + "qps_batch_min": 1791.4084052681533, + "qps_batch_max": 2942.581120235563, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 152, + "threads": 32, + "qps": 52.18215114010544, + "median_ms": 19.19580003595911, + "p95_ms": 24.747024988755584, + "p99_ms": 27.075095009058714, + "mean_ms": 19.730473682437534 + }, + "mem_after_sweep": { + "rss_mb": 221.3828125, + "peak_mb": 356.50390625 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 218.44921875, + "peak_mb": 356.50390625 + }, + "status": "ok", + "finished": "2026-11-16 19:46:05" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_text384_100000_r3.load.json b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_text384_100000_r3.load.json new file mode 100644 index 0000000..854d95d --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_text384_100000_r3.load.json @@ -0,0 +1 @@ +{"total_pct": 14.909374999999999, "pin_cpu_pct": 31.9, "max_cpu_pct": 76.8, "noisy": true, "seconds": 28.55177092552185, "after": 18.0} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_text384_100000_r3.txt b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_text384_100000_r3.txt new file mode 100644 index 0000000..fd9e18b --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_flat_text384_100000_r3.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_flat --ds text384 --n 100000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_flat_text384_100000_r3.json + faiss_flat text384 100000 knob=None recall=1.0000 qps1t=120 med=8.401ms batch=2088 diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_1000000_r1.json b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_1000000_r1.json new file mode 100644 index 0000000..30faf69 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_1000000_r1.json @@ -0,0 +1,346 @@ +{ + "system": "faiss_hnsw", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 20:37:19", + "mem_baseline": { + "rss_mb": 37.0546875, + "peak_mb": 37.0546875 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 221.03498700010823, + "index_s": 3.600027412176132e-06, + "total_s": 221.03499060013564, + "first_query_ms": 1.7762999632395804, + "ready_s": 221.03676690009888 + }, + "mem_after_build": { + "rss_mb": 1851.703125, + "peak_mb": 2337.8984375 + }, + "mem_after_first_query": { + "rss_mb": 1852.7265625, + "peak_mb": 2337.8984375 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.7597, + "qps_1t_runs": [ + 5485.484566285345, + 5782.168368939731, + 5574.791688258738 + ], + "qps_1t": 5574.791688258738, + "qps_1t_min": 5485.484566285345, + "qps_1t_max": 5782.168368939731, + "median_ms": 0.16309999045915902, + "p95_ms": 0.2694199589313938, + "p99_ms": 0.3837040090002048, + "mean_ms": 0.17820790018110225, + "knob": 24, + "batch": { + "recall": 0.7597, + "qps_batch_runs": [ + 22278.562206638104, + 23928.139011713007, + 22389.897659638118 + ], + "qps_batch": 22389.897659638118, + "qps_batch_min": 22278.562206638104, + "qps_batch_max": 23928.139011713007, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8054, + "qps_1t_runs": [ + 4625.798362871505, + 4780.960303863542, + 4412.782251992047 + ], + "qps_1t": 4625.798362871505, + "qps_1t_min": 4412.782251992047, + "qps_1t_max": 4780.960303863542, + "median_ms": 0.19609997980296612, + "p95_ms": 0.3326049743918702, + "p99_ms": 0.6160330335842438, + "mean_ms": 0.21731876636234423, + "knob": 32, + "batch": { + "recall": 0.8054, + "qps_batch_runs": [ + 18963.9608858595, + 18636.932059131555, + 18643.777088099156 + ], + "qps_batch": 18643.777088099156, + "qps_batch_min": 18636.932059131555, + "qps_batch_max": 18963.9608858595, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8571, + "qps_1t_runs": [ + 3149.521978303515, + 3602.2574611918494, + 3639.3041915832337 + ], + "qps_1t": 3602.2574611918494, + "qps_1t_min": 3149.521978303515, + "qps_1t_max": 3639.3041915832337, + "median_ms": 0.2635499695315957, + "p95_ms": 0.46014495601411887, + "p99_ms": 0.7388080371310932, + "mean_ms": 0.28996333353764686, + "knob": 48, + "batch": { + "recall": 0.8571, + "qps_batch_runs": [ + 13544.75797491857, + 15442.727555723253, + 15120.929630495997 + ], + "qps_batch": 15120.929630495997, + "qps_batch_min": 13544.75797491857, + "qps_batch_max": 15442.727555723253, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8906, + "qps_1t_runs": [ + 2893.475814940856, + 2946.106866572403, + 2845.26671637232 + ], + "qps_1t": 2893.475814940856, + "qps_1t_min": 2845.26671637232, + "qps_1t_max": 2946.106866572403, + "median_ms": 0.3199499915353954, + "p95_ms": 0.5234249721979721, + "p99_ms": 0.8081239834427828, + "mean_ms": 0.3454990001046099, + "knob": 64, + "batch": { + "recall": 0.8906, + "qps_batch_runs": [ + 11119.388880723856, + 11015.87276946587, + 11098.397280971189 + ], + "qps_batch": 11098.397280971189, + "qps_batch_min": 11015.87276946587, + "qps_batch_max": 11119.388880723856, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9284, + "qps_1t_runs": [ + 2036.6233670721767, + 2083.656296282427, + 2133.900094624608 + ], + "qps_1t": 2083.656296282427, + "qps_1t_min": 2036.6233670721767, + "qps_1t_max": 2133.900094624608, + "median_ms": 0.4465500242076814, + "p95_ms": 0.7438099593855441, + "p99_ms": 1.0458089888561515, + "mean_ms": 0.47985330121203634, + "knob": 96, + "batch": { + "recall": 0.9284, + "qps_batch_runs": [ + 8271.791203917292, + 7843.463295664255, + 5960.697545859191 + ], + "qps_batch": 7843.463295664255, + "qps_batch_min": 5960.697545859191, + "qps_batch_max": 8271.791203917292, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9465, + "qps_1t_runs": [ + 1650.28865332656, + 1551.8015410805897, + 1730.8374628134359 + ], + "qps_1t": 1650.28865332656, + "qps_1t_min": 1551.8015410805897, + "qps_1t_max": 1730.8374628134359, + "median_ms": 0.5586500046774745, + "p95_ms": 0.9221000334946433, + "p99_ms": 1.2620860052993501, + "mean_ms": 0.6093739667752137, + "knob": 128, + "batch": { + "recall": 0.9465, + "qps_batch_runs": [ + 6238.198108594723, + 6474.822007634418, + 6224.289792937152 + ], + "qps_batch": 6238.198108594723, + "qps_batch_min": 6224.289792937152, + "qps_batch_max": 6474.822007634418, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9673, + "qps_1t_runs": [ + 1123.4717988789741, + 1197.2684075097384, + 1184.716211877975 + ], + "qps_1t": 1184.716211877975, + "qps_1t_min": 1123.4717988789741, + "qps_1t_max": 1197.2684075097384, + "median_ms": 0.7953000022098422, + "p95_ms": 1.2904100440209731, + "p99_ms": 1.7684349609771712, + "mean_ms": 0.8564722000737675, + "knob": 192, + "batch": { + "recall": 0.9673, + "qps_batch_runs": [ + 4544.250318943798, + 4544.196628620832, + 4611.261438469056 + ], + "qps_batch": 4544.250318943798, + "qps_batch_min": 4544.196628620832, + "qps_batch_max": 4611.261438469056, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9774, + "qps_1t_runs": [ + 952.4333821919209, + 859.5326031311689, + 955.6026983715939 + ], + "qps_1t": 952.4333821919209, + "qps_1t_min": 859.5326031311689, + "qps_1t_max": 955.6026983715939, + "median_ms": 1.0257000103592873, + "p95_ms": 1.5414349792990831, + "p99_ms": 2.056715000653632, + "mean_ms": 1.086608399966887, + "knob": 256, + "batch": { + "recall": 0.9774, + "qps_batch_runs": [ + 3465.603881506548, + 3367.772171901233, + 3337.3158636180547 + ], + "qps_batch": 3367.772171901233, + "qps_batch_min": 3337.3158636180547, + "qps_batch_max": 3465.603881506548, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9884, + "qps_1t_runs": [ + 614.0286138510663, + 602.4090215916652, + 617.3884490393476 + ], + "qps_1t": 614.0286138510663, + "qps_1t_min": 602.4090215916652, + "qps_1t_max": 617.3884490393476, + "median_ms": 1.5854999655857682, + "p95_ms": 2.228509975248016, + "p99_ms": 2.741560979047789, + "mean_ms": 1.6361053666332737, + "knob": 400, + "batch": { + "recall": 0.9884, + "qps_batch_runs": [ + 2195.870796541477, + 2287.986059705659, + 2193.3340631131364 + ], + "qps_batch": 2195.870796541477, + "qps_batch_min": 2193.3340631131364, + "qps_batch_max": 2287.986059705659, + "threads": 32 + } + } + ], + "knob_name": "efSearch", + "mem_after_sweep": { + "rss_mb": 1855.53125, + "peak_mb": 2337.8984375 + }, + "ref_knob": 192, + "ref_recall_pre": 0.9673, + "mutation": { + "ref_knob": 192, + "recall_before": 0.9673, + "insert_10k": { + "seconds": 4.7194864000193775, + "per_s": 2118.8746300781672, + "batch": 1000 + }, + "update_10k": { + "unsupported": true + }, + "delete_10k": { + "unsupported": true + }, + "mem_after": { + "rss_mb": 1900.6328125, + "peak_mb": 2337.8984375 + } + }, + "saveload": { + "save_s": 1.0356472000130452, + "disk_mb": 1618.4442081451416, + "load_s": 0.835485499992501, + "first_query_after_load_ms": 1.0848999954760075, + "time_to_ready_s": 0.836570399987977 + }, + "mem_end": { + "rss_mb": 1869.43359375, + "peak_mb": 3489.93359375 + }, + "status": "ok", + "finished": "2026-11-16 20:41:30" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_1000000_r1.load.json b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_1000000_r1.load.json new file mode 100644 index 0000000..eb90c5f --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_1000000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 13.128125, "pin_cpu_pct": 71.1, "max_cpu_pct": 72.6, "noisy": true, "seconds": 251.10076642036438, "after": 77.2} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_1000000_r1.txt b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_1000000_r1.txt new file mode 100644 index 0000000..8688cbf --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_1000000_r1.txt @@ -0,0 +1,10 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_hnsw --ds text384 --n 1000000 --exp core,mutation,saveload --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_hnsw_text384_1000000_r1.json + faiss_hnsw text384 1000000 knob=24 recall=0.7597 qps1t=5575 med=0.163ms batch=22390 + faiss_hnsw text384 1000000 knob=32 recall=0.8054 qps1t=4626 med=0.196ms batch=18644 + faiss_hnsw text384 1000000 knob=48 recall=0.8571 qps1t=3602 med=0.264ms batch=15121 + faiss_hnsw text384 1000000 knob=64 recall=0.8906 qps1t=2893 med=0.320ms batch=11098 + faiss_hnsw text384 1000000 knob=96 recall=0.9284 qps1t=2084 med=0.447ms batch=7843 + faiss_hnsw text384 1000000 knob=128 recall=0.9465 qps1t=1650 med=0.559ms batch=6238 + faiss_hnsw text384 1000000 knob=192 recall=0.9673 qps1t=1185 med=0.795ms batch=4544 + faiss_hnsw text384 1000000 knob=256 recall=0.9774 qps1t=952 med=1.026ms batch=3368 + faiss_hnsw text384 1000000 knob=400 recall=0.9884 qps1t=614 med=1.585ms batch=2196 diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_100000_r2.json b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_100000_r2.json new file mode 100644 index 0000000..51a2b70 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_100000_r2.json @@ -0,0 +1,346 @@ +{ + "system": "faiss_hnsw", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 19:39:33", + "mem_baseline": { + "rss_mb": 37.21875, + "peak_mb": 37.21875 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 11.744921299919952, + "index_s": 4.199973773211241e-06, + "total_s": 11.744925499893725, + "first_query_ms": 2.18250002944842, + "ready_s": 11.747107999923173 + }, + "mem_after_build": { + "rss_mb": 243.50390625, + "peak_mb": 385.2578125 + }, + "mem_after_first_query": { + "rss_mb": 243.640625, + "peak_mb": 385.2578125 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8114, + "qps_1t_runs": [ + 8102.939737886807, + 7611.386966695932, + 8209.91773738373 + ], + "qps_1t": 8102.939737886807, + "qps_1t_min": 7611.386966695932, + "qps_1t_max": 8209.91773738373, + "median_ms": 0.1244999875780195, + "p95_ms": 0.1638049696339294, + "p99_ms": 0.1963029819307848, + "mean_ms": 0.12553266658990955, + "knob": 24, + "batch": { + "recall": 0.8114, + "qps_batch_runs": [ + 42273.82441947607, + 39194.78233806966, + 33687.955197376614 + ], + "qps_batch": 39194.78233806966, + "qps_batch_min": 33687.955197376614, + "qps_batch_max": 42273.82441947607, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8572, + "qps_1t_runs": [ + 6754.164564269509, + 6429.235010120505, + 6138.222975898049 + ], + "qps_1t": 6429.235010120505, + "qps_1t_min": 6138.222975898049, + "qps_1t_max": 6754.164564269509, + "median_ms": 0.1545000122860074, + "p95_ms": 0.19689998589456081, + "p99_ms": 0.22671398008242216, + "mean_ms": 0.15550330030964687, + "knob": 32, + "batch": { + "recall": 0.8572, + "qps_batch_runs": [ + 28351.501917232934, + 27213.253983315735, + 30816.64098728229 + ], + "qps_batch": 28351.501917232934, + "qps_batch_min": 27213.253983315735, + "qps_batch_max": 30816.64098728229, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9093, + "qps_1t_runs": [ + 4972.838345230412, + 4441.161923870219, + 4520.066153108398 + ], + "qps_1t": 4520.066153108398, + "qps_1t_min": 4441.161923870219, + "qps_1t_max": 4972.838345230412, + "median_ms": 0.2196999848820269, + "p95_ms": 0.26470497832633555, + "p99_ms": 0.29399996856227517, + "mean_ms": 0.21583146700868383, + "knob": 48, + "batch": { + "recall": 0.9093, + "qps_batch_runs": [ + 25482.186691562943, + 26929.933713579576, + 26670.151568168807 + ], + "qps_batch": 26670.151568168807, + "qps_batch_min": 25482.186691562943, + "qps_batch_max": 26929.933713579576, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9368, + "qps_1t_runs": [ + 3753.215575325669, + 3688.7832631373326, + 3836.8377186999746 + ], + "qps_1t": 3753.215575325669, + "qps_1t_min": 3688.7832631373326, + "qps_1t_max": 3836.8377186999746, + "median_ms": 0.26684999465942383, + "p95_ms": 0.3245050262194127, + "p99_ms": 0.36320997169241287, + "mean_ms": 0.2660538661487711, + "knob": 64, + "batch": { + "recall": 0.9368, + "qps_batch_runs": [ + 18273.818558612144, + 19001.580942466368, + 18910.45519884937 + ], + "qps_batch": 18910.45519884937, + "qps_batch_min": 18273.818558612144, + "qps_batch_max": 19001.580942466368, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9636, + "qps_1t_runs": [ + 2854.4034422769955, + 2840.3805505235164, + 2644.065892676241 + ], + "qps_1t": 2840.3805505235164, + "qps_1t_min": 2644.065892676241, + "qps_1t_max": 2854.4034422769955, + "median_ms": 0.358450022758916, + "p95_ms": 0.4400299774715675, + "p99_ms": 0.5066039582015945, + "mean_ms": 0.36020226712571457, + "knob": 96, + "batch": { + "recall": 0.9636, + "qps_batch_runs": [ + 13528.358136227374, + 12271.444347283263, + 11666.555835126828 + ], + "qps_batch": 12271.444347283263, + "qps_batch_min": 11666.555835126828, + "qps_batch_max": 13528.358136227374, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9743, + "qps_1t_runs": [ + 2224.678266104106, + 2191.437873506276, + 2276.014043338266 + ], + "qps_1t": 2224.678266104106, + "qps_1t_min": 2191.437873506276, + "qps_1t_max": 2276.014043338266, + "median_ms": 0.453799992101267, + "p95_ms": 0.5518050107639283, + "p99_ms": 0.6258139922283587, + "mean_ms": 0.4483964003350896, + "knob": 128, + "batch": { + "recall": 0.9743, + "qps_batch_runs": [ + 10160.619067363856, + 11412.802202682815, + 11060.295195487774 + ], + "qps_batch": 11060.295195487774, + "qps_batch_min": 10160.619067363856, + "qps_batch_max": 11412.802202682815, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9866, + "qps_1t_runs": [ + 1573.9351284195088, + 1579.3489395309414, + 1576.255790317091 + ], + "qps_1t": 1576.255790317091, + "qps_1t_min": 1573.9351284195088, + "qps_1t_max": 1579.3489395309414, + "median_ms": 0.6413999944925308, + "p95_ms": 0.7589050161186606, + "p99_ms": 0.8242139895446595, + "mean_ms": 0.6343124330936311, + "knob": 192, + "batch": { + "recall": 0.9866, + "qps_batch_runs": [ + 6503.367118033876, + 8466.547401543075, + 7558.698967556354 + ], + "qps_batch": 7558.698967556354, + "qps_batch_min": 6503.367118033876, + "qps_batch_max": 8466.547401543075, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9918, + "qps_1t_runs": [ + 1187.831709750829, + 1168.4676845566973, + 1181.2195082419357 + ], + "qps_1t": 1181.2195082419357, + "qps_1t_min": 1168.4676845566973, + "qps_1t_max": 1187.831709750829, + "median_ms": 0.8527500031050295, + "p95_ms": 1.0291099810274318, + "p99_ms": 1.140040007303468, + "mean_ms": 0.8480914997053333, + "knob": 256, + "batch": { + "recall": 0.9918, + "qps_batch_runs": [ + 5929.8475305302945, + 6160.336304628274, + 6255.473541064877 + ], + "qps_batch": 6160.336304628274, + "qps_batch_min": 5929.8475305302945, + "qps_batch_max": 6255.473541064877, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9967, + "qps_1t_runs": [ + 791.3221090372699, + 774.2809272874983, + 772.3862638431186 + ], + "qps_1t": 774.2809272874983, + "qps_1t_min": 772.3862638431186, + "qps_1t_max": 791.3221090372699, + "median_ms": 1.2976000143680722, + "p95_ms": 1.5382900106487791, + "p99_ms": 1.628716003033332, + "mean_ms": 1.2833059332333505, + "knob": 400, + "batch": { + "recall": 0.9967, + "qps_batch_runs": [ + 3928.058396116781, + 3986.517597850768, + 3885.612238640346 + ], + "qps_batch": 3928.058396116781, + "qps_batch_min": 3885.612238640346, + "qps_batch_max": 3986.517597850768, + "threads": 32 + } + } + ], + "knob_name": "efSearch", + "mem_after_sweep": { + "rss_mb": 245.75390625, + "peak_mb": 385.2578125 + }, + "ref_knob": 96, + "ref_recall_pre": 0.9636, + "mutation": { + "ref_knob": 96, + "recall_before": 0.9636, + "insert_10k": { + "seconds": 3.564838199992664, + "per_s": 2805.1764032433725, + "batch": 1000 + }, + "update_10k": { + "unsupported": true + }, + "delete_10k": { + "unsupported": true + }, + "mem_after": { + "rss_mb": 293.8515625, + "peak_mb": 385.2578125 + } + }, + "saveload": { + "save_s": 0.2281207999913022, + "disk_mb": 176.2752628326416, + "load_s": 0.1442172999959439, + "first_query_after_load_ms": 0.6923999753780663, + "time_to_ready_s": 0.14490969997132197 + }, + "mem_end": { + "rss_mb": 262.05859375, + "peak_mb": 440.87109375 + }, + "status": "ok", + "finished": "2026-11-16 19:40:05" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_100000_r2.load.json b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_100000_r2.load.json new file mode 100644 index 0000000..1fc4357 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_100000_r2.load.json @@ -0,0 +1 @@ +{"total_pct": 14.221875, "pin_cpu_pct": 28.0, "max_cpu_pct": 70.7, "noisy": true, "seconds": 32.55344104766846, "after": 27.2} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_100000_r2.txt b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_100000_r2.txt new file mode 100644 index 0000000..693e1b1 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_100000_r2.txt @@ -0,0 +1,10 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_hnsw --ds text384 --n 100000 --exp core,mutation,saveload --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_hnsw_text384_100000_r2.json + faiss_hnsw text384 100000 knob=24 recall=0.8114 qps1t=8103 med=0.124ms batch=39195 + faiss_hnsw text384 100000 knob=32 recall=0.8572 qps1t=6429 med=0.155ms batch=28352 + faiss_hnsw text384 100000 knob=48 recall=0.9093 qps1t=4520 med=0.220ms batch=26670 + faiss_hnsw text384 100000 knob=64 recall=0.9368 qps1t=3753 med=0.267ms batch=18910 + faiss_hnsw text384 100000 knob=96 recall=0.9636 qps1t=2840 med=0.358ms batch=12271 + faiss_hnsw text384 100000 knob=128 recall=0.9743 qps1t=2225 med=0.454ms batch=11060 + faiss_hnsw text384 100000 knob=192 recall=0.9866 qps1t=1576 med=0.641ms batch=7559 + faiss_hnsw text384 100000 knob=256 recall=0.9918 qps1t=1181 med=0.853ms batch=6160 + faiss_hnsw text384 100000 knob=400 recall=0.9967 qps1t=774 med=1.298ms batch=3928 diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_100000_r3.json b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_100000_r3.json new file mode 100644 index 0000000..ee30ffb --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_100000_r3.json @@ -0,0 +1,346 @@ +{ + "system": "faiss_hnsw", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 19:49:30", + "mem_baseline": { + "rss_mb": 36.8984375, + "peak_mb": 36.8984375 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 11.495829500025138, + "index_s": 3.700028173625469e-06, + "total_s": 11.495833200053312, + "first_query_ms": 2.275099977850914, + "ready_s": 11.498108300031163 + }, + "mem_after_build": { + "rss_mb": 243.48046875, + "peak_mb": 385.88671875 + }, + "mem_after_first_query": { + "rss_mb": 243.64453125, + "peak_mb": 385.88671875 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8099, + "qps_1t_runs": [ + 7812.030013322118, + 7264.369282916077, + 6893.718134739047 + ], + "qps_1t": 7264.369282916077, + "qps_1t_min": 6893.718134739047, + "qps_1t_max": 7812.030013322118, + "median_ms": 0.13050000416114926, + "p95_ms": 0.18950499070342627, + "p99_ms": 0.23732200206723011, + "mean_ms": 0.13690850049412498, + "knob": 24, + "batch": { + "recall": 0.8099, + "qps_batch_runs": [ + 34097.45731657517, + 32114.06914492501, + 33302.47303411543 + ], + "qps_batch": 33302.47303411543, + "qps_batch_min": 32114.06914492501, + "qps_batch_max": 34097.45731657517, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8558, + "qps_1t_runs": [ + 6108.910869912414, + 5662.469262676472, + 6212.766138539095 + ], + "qps_1t": 6108.910869912414, + "qps_1t_min": 5662.469262676472, + "qps_1t_max": 6212.766138539095, + "median_ms": 0.16279995907098055, + "p95_ms": 0.22321000869851554, + "p99_ms": 0.2791179716587062, + "mean_ms": 0.16708519998549795, + "knob": 32, + "batch": { + "recall": 0.8558, + "qps_batch_runs": [ + 30719.88988076583, + 29454.733951938775, + 32567.77349157171 + ], + "qps_batch": 30719.88988076583, + "qps_batch_min": 29454.733951938775, + "qps_batch_max": 32567.77349157171, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9094, + "qps_1t_runs": [ + 4097.102984435339, + 4234.410168355165, + 4546.068278342226 + ], + "qps_1t": 4234.410168355165, + "qps_1t_min": 4097.102984435339, + "qps_1t_max": 4546.068278342226, + "median_ms": 0.22655000793747604, + "p95_ms": 0.3023050318006426, + "p99_ms": 0.35921999951824496, + "mean_ms": 0.23340186641629165, + "knob": 48, + "batch": { + "recall": 0.9094, + "qps_batch_runs": [ + 18777.544318126438, + 20765.02505742389, + 18222.736494342997 + ], + "qps_batch": 18777.544318126438, + "qps_batch_min": 18222.736494342997, + "qps_batch_max": 20765.02505742389, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9361, + "qps_1t_runs": [ + 3124.535235985163, + 3920.2111567947886, + 3885.408437318452 + ], + "qps_1t": 3885.408437318452, + "qps_1t_min": 3124.535235985163, + "qps_1t_max": 3920.2111567947886, + "median_ms": 0.26699999580159783, + "p95_ms": 0.3913200111128389, + "p99_ms": 0.46520304924342776, + "mean_ms": 0.2775030327611603, + "knob": 64, + "batch": { + "recall": 0.9361, + "qps_batch_runs": [ + 17646.58132585533, + 19451.80911258141, + 20198.02140956488 + ], + "qps_batch": 19451.80911258141, + "qps_batch_min": 17646.58132585533, + "qps_batch_max": 20198.02140956488, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9627, + "qps_1t_runs": [ + 2349.3475043425433, + 2729.41556653054, + 2588.8694120424257 + ], + "qps_1t": 2588.8694120424257, + "qps_1t_min": 2349.3475043425433, + "qps_1t_max": 2729.41556653054, + "median_ms": 0.38300000596791506, + "p95_ms": 0.5067099613370374, + "p99_ms": 0.6254069891292601, + "mean_ms": 0.3927659667873134, + "knob": 96, + "batch": { + "recall": 0.9627, + "qps_batch_runs": [ + 11981.385716893603, + 12354.555985325425, + 12483.615254835606 + ], + "qps_batch": 12354.555985325425, + "qps_batch_min": 11981.385716893603, + "qps_batch_max": 12483.615254835606, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9749, + "qps_1t_runs": [ + 2037.704460300424, + 1973.7760184370466, + 2151.853271996292 + ], + "qps_1t": 2037.704460300424, + "qps_1t_min": 1973.7760184370466, + "qps_1t_max": 2151.853271996292, + "median_ms": 0.48439996317029, + "p95_ms": 0.6152049725642428, + "p99_ms": 0.713232017005793, + "mean_ms": 0.48736903331397724, + "knob": 128, + "batch": { + "recall": 0.9749, + "qps_batch_runs": [ + 9610.79176926913, + 10130.511377667206, + 10818.471458770235 + ], + "qps_batch": 10130.511377667206, + "qps_batch_min": 9610.79176926913, + "qps_batch_max": 10818.471458770235, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9866, + "qps_1t_runs": [ + 1375.2146713340198, + 1522.0887788038135, + 1467.3271004889477 + ], + "qps_1t": 1467.3271004889477, + "qps_1t_min": 1375.2146713340198, + "qps_1t_max": 1522.0887788038135, + "median_ms": 0.6832500221207738, + "p95_ms": 0.8620149863418191, + "p99_ms": 0.9936319984262801, + "mean_ms": 0.6885541335990032, + "knob": 192, + "batch": { + "recall": 0.9866, + "qps_batch_runs": [ + 8345.155638444621, + 7871.970275624457, + 7812.67700571957 + ], + "qps_batch": 7871.970275624457, + "qps_batch_min": 7812.67700571957, + "qps_batch_max": 8345.155638444621, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9922, + "qps_1t_runs": [ + 1094.9124866619563, + 1196.247801190846, + 1156.9798342873046 + ], + "qps_1t": 1156.9798342873046, + "qps_1t_min": 1094.9124866619563, + "qps_1t_max": 1196.247801190846, + "median_ms": 0.8840500086080283, + "p95_ms": 1.079630013555288, + "p99_ms": 1.199717996641993, + "mean_ms": 0.8711938339013917, + "knob": 256, + "batch": { + "recall": 0.9922, + "qps_batch_runs": [ + 5692.076856470044, + 5728.6925572483315, + 5691.526113778687 + ], + "qps_batch": 5692.076856470044, + "qps_batch_min": 5691.526113778687, + "qps_batch_max": 5728.6925572483315, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9966, + "qps_1t_runs": [ + 754.9316662765617, + 765.9368869996546, + 760.7670599310098 + ], + "qps_1t": 760.7670599310098, + "qps_1t_min": 754.9316662765617, + "qps_1t_max": 765.9368869996546, + "median_ms": 1.3254500227048993, + "p95_ms": 1.5919000143185258, + "p99_ms": 1.7565089871641246, + "mean_ms": 1.3148922003650416, + "knob": 400, + "batch": { + "recall": 0.9966, + "qps_batch_runs": [ + 3752.6296553772045, + 3874.246603854387, + 3304.265707877729 + ], + "qps_batch": 3752.6296553772045, + "qps_batch_min": 3304.265707877729, + "qps_batch_max": 3874.246603854387, + "threads": 32 + } + } + ], + "knob_name": "efSearch", + "mem_after_sweep": { + "rss_mb": 245.67578125, + "peak_mb": 385.88671875 + }, + "ref_knob": 96, + "ref_recall_pre": 0.9627, + "mutation": { + "ref_knob": 96, + "recall_before": 0.9627, + "insert_10k": { + "seconds": 3.3892296999692917, + "per_s": 2950.5229462879443, + "batch": 1000 + }, + "update_10k": { + "unsupported": true + }, + "delete_10k": { + "unsupported": true + }, + "mem_after": { + "rss_mb": 293.88671875, + "peak_mb": 385.88671875 + } + }, + "saveload": { + "save_s": 0.2900604999740608, + "disk_mb": 176.2752628326416, + "load_s": 0.15648950001923367, + "first_query_after_load_ms": 0.8470999891869724, + "time_to_ready_s": 0.15733660000842065 + }, + "mem_end": { + "rss_mb": 260.96484375, + "peak_mb": 439.7578125 + }, + "status": "ok", + "finished": "2026-11-16 19:50:03" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_100000_r3.load.json b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_100000_r3.load.json new file mode 100644 index 0000000..14767f8 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_100000_r3.load.json @@ -0,0 +1 @@ +{"total_pct": 12.628125, "pin_cpu_pct": 41.7, "max_cpu_pct": 60.4, "noisy": true, "seconds": 33.22414565086365, "after": 23.0} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_100000_r3.txt b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_100000_r3.txt new file mode 100644 index 0000000..c108c7a --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_hnsw_text384_100000_r3.txt @@ -0,0 +1,10 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_hnsw --ds text384 --n 100000 --exp core,mutation,saveload --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_hnsw_text384_100000_r3.json + faiss_hnsw text384 100000 knob=24 recall=0.8099 qps1t=7264 med=0.131ms batch=33302 + faiss_hnsw text384 100000 knob=32 recall=0.8558 qps1t=6109 med=0.163ms batch=30720 + faiss_hnsw text384 100000 knob=48 recall=0.9094 qps1t=4234 med=0.227ms batch=18778 + faiss_hnsw text384 100000 knob=64 recall=0.9361 qps1t=3885 med=0.267ms batch=19452 + faiss_hnsw text384 100000 knob=96 recall=0.9627 qps1t=2589 med=0.383ms batch=12355 + faiss_hnsw text384 100000 knob=128 recall=0.9749 qps1t=2038 med=0.484ms batch=10131 + faiss_hnsw text384 100000 knob=192 recall=0.9866 qps1t=1467 med=0.683ms batch=7872 + faiss_hnsw text384 100000 knob=256 recall=0.9922 qps1t=1157 med=0.884ms batch=5692 + faiss_hnsw text384 100000 knob=400 recall=0.9966 qps1t=761 med=1.325ms batch=3753 diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_ivfflat_text384_100000_r2.json b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_ivfflat_text384_100000_r2.json new file mode 100644 index 0000000..530d482 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_ivfflat_text384_100000_r2.json @@ -0,0 +1,279 @@ +{ + "system": "faiss_ivfflat", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 19:37:09", + "mem_baseline": { + "rss_mb": 37.109375, + "peak_mb": 37.109375 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 9.70007386058569e-06, + "index_s": 1.3919868000084534, + "total_s": 1.391996500082314, + "first_query_ms": 3.180600004270673, + "ready_s": 1.3951771000865847 + }, + "mem_after_build": { + "rss_mb": 277.421875, + "peak_mb": 498.046875 + }, + "mem_after_first_query": { + "rss_mb": 277.5, + "peak_mb": 498.046875 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.7507, + "qps_1t_runs": [ + 7238.047822682852, + 6862.221693080631, + 7010.661113551807 + ], + "qps_1t": 7010.661113551807, + "qps_1t_min": 6862.221693080631, + "qps_1t_max": 7238.047822682852, + "median_ms": 0.13964998652227223, + "p95_ms": 0.17700500902719793, + "p99_ms": 0.20690900390036382, + "mean_ms": 0.1421746998676099, + "knob": 16, + "batch": { + "recall": 0.7507, + "qps_batch_runs": [ + 28684.774386170124, + 28716.570302477867, + 28510.172410722436 + ], + "qps_batch": 28684.774386170124, + "qps_batch_min": 28510.172410722436, + "qps_batch_max": 28716.570302477867, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8399, + "qps_1t_runs": [ + 4233.592704640369, + 4126.614430209521, + 3202.8297654180624 + ], + "qps_1t": 4126.614430209521, + "qps_1t_min": 3202.8297654180624, + "qps_1t_max": 4233.592704640369, + "median_ms": 0.24309998843818903, + "p95_ms": 0.33192496630363155, + "p99_ms": 0.40881597029510847, + "mean_ms": 0.26358643350734684, + "knob": 32, + "batch": { + "recall": 0.8399, + "qps_batch_runs": [ + 10807.037112287773, + 9729.29217132655, + 9903.166837476843 + ], + "qps_batch": 9903.166837476843, + "qps_batch_min": 9729.29217132655, + "qps_batch_max": 10807.037112287773, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9073, + "qps_1t_runs": [ + 1803.4076433128541, + 1761.5924224420373, + 1785.6189475543836 + ], + "qps_1t": 1785.6189475543836, + "qps_1t_min": 1761.5924224420373, + "qps_1t_max": 1803.4076433128541, + "median_ms": 0.5593000096268952, + "p95_ms": 0.6968150060856714, + "p99_ms": 0.7587460143258784, + "mean_ms": 0.5607346336667737, + "knob": 64, + "batch": { + "recall": 0.9073, + "qps_batch_runs": [ + 6175.275309582477, + 6895.952489733524, + 6273.285652481695 + ], + "qps_batch": 6273.285652481695, + "qps_batch_min": 6175.275309582477, + "qps_batch_max": 6895.952489733524, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9553, + "qps_1t_runs": [ + 960.8815515115626, + 963.2566715988314, + 999.162900879772 + ], + "qps_1t": 963.2566715988314, + "qps_1t_min": 960.8815515115626, + "qps_1t_max": 999.162900879772, + "median_ms": 1.0295500105712563, + "p95_ms": 1.2509999796748161, + "p99_ms": 1.3364339899271718, + "mean_ms": 1.0265645663409184, + "knob": 128, + "batch": { + "recall": 0.9553, + "qps_batch_runs": [ + 3013.3143295423215, + 3303.835587804668, + 3489.1153553123727 + ], + "qps_batch": 3303.835587804668, + "qps_batch_min": 3013.3143295423215, + "qps_batch_max": 3489.1153553123727, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9855, + "qps_1t_runs": [ + 493.86881639242927, + 497.8823069200946, + 478.66134908656926 + ], + "qps_1t": 493.86881639242927, + "qps_1t_min": 478.66134908656926, + "qps_1t_max": 497.8823069200946, + "median_ms": 2.0405499963089824, + "p95_ms": 2.3929150018375367, + "p99_ms": 2.641849034116602, + "mean_ms": 2.0408319000465176, + "knob": 256, + "batch": { + "recall": 0.9855, + "qps_batch_runs": [ + 1782.3898103234765, + 1932.1468663480082, + 1910.7040123071968 + ], + "qps_batch": 1910.7040123071968, + "qps_batch_min": 1782.3898103234765, + "qps_batch_max": 1932.1468663480082, + "threads": 32 + } + }, + { + "nq": 734, + "recall_queries": 1000, + "recall": 0.9957999999999999, + "qps_1t_runs": [ + 250.5470391339473, + 254.77163138886618, + 254.50603981847786 + ], + "qps_1t": 254.50603981847786, + "qps_1t_min": 250.5470391339473, + "qps_1t_max": 254.77163138886618, + "median_ms": 3.99364999611862, + "p95_ms": 4.475300025660545, + "p99_ms": 4.623186045791954, + "mean_ms": 3.948510036311898, + "knob": 512, + "batch": { + "recall": 0.9958, + "qps_batch_runs": [ + 952.3533794420983, + 925.1448638305183, + 785.8997019150582 + ], + "qps_batch": 925.1448638305183, + "qps_batch_min": 785.8997019150582, + "qps_batch_max": 952.3533794420983, + "threads": 32 + } + } + ], + "knob_name": "nprobe", + "mem_after_sweep": { + "rss_mb": 269.35546875, + "peak_mb": 498.046875 + }, + "ref_knob": 128, + "ref_recall_pre": 0.9553, + "mutation": { + "ref_knob": 128, + "recall_before": 0.9553, + "insert_10k": { + "seconds": 0.1518786999513395, + "per_s": 65842.01736783305, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.0693576000048779, + "per_s": 144180.3061134858, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.004910499963443726, + "per_s": 2036452.5149058376, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 325.1484375, + "peak_mb": 498.046875 + }, + "first_query_after_ms": 2.8734999941661954, + "after": { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9521, + "qps_1t_runs": [ + 985.5107249064876, + 927.3914615432604, + 925.562728418903 + ], + "qps_1t": 927.3914615432604, + "qps_1t_min": 925.562728418903, + "qps_1t_max": 985.5107249064876, + "median_ms": 1.0598500375635922, + "p95_ms": 1.2834449938964096, + "p99_ms": 1.3807280355831602, + "mean_ms": 1.057806467055343 + }, + "recall_after": 0.9521 + }, + "saveload": { + "save_s": 0.14387589995749295, + "disk_mb": 149.110125541687, + "load_s": 0.18801729998085648, + "first_query_after_load_ms": 2.2226000437512994, + "time_to_ready_s": 0.19023990002460778 + }, + "mem_end": { + "rss_mb": 245.203125, + "peak_mb": 498.046875 + }, + "status": "ok", + "finished": "2026-11-16 19:37:44" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_ivfflat_text384_100000_r2.load.json b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_ivfflat_text384_100000_r2.load.json new file mode 100644 index 0000000..c51a654 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_ivfflat_text384_100000_r2.load.json @@ -0,0 +1 @@ +{"total_pct": 13.034375, "pin_cpu_pct": 32.8, "max_cpu_pct": 68.2, "noisy": true, "seconds": 35.26289463043213, "after": 21.0} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_ivfflat_text384_100000_r2.txt b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_ivfflat_text384_100000_r2.txt new file mode 100644 index 0000000..d5a9302 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_ivfflat_text384_100000_r2.txt @@ -0,0 +1,7 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_ivfflat --ds text384 --n 100000 --exp core,mutation,saveload --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_ivfflat_text384_100000_r2.json + faiss_ivfflat text384 100000 knob=16 recall=0.7507 qps1t=7011 med=0.140ms batch=28685 + faiss_ivfflat text384 100000 knob=32 recall=0.8399 qps1t=4127 med=0.243ms batch=9903 + faiss_ivfflat text384 100000 knob=64 recall=0.9073 qps1t=1786 med=0.559ms batch=6273 + faiss_ivfflat text384 100000 knob=128 recall=0.9553 qps1t=963 med=1.030ms batch=3304 + faiss_ivfflat text384 100000 knob=256 recall=0.9855 qps1t=494 med=2.041ms batch=1911 + faiss_ivfflat text384 100000 knob=512 recall=0.9958 qps1t=255 med=3.994ms batch=925 diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_ivfflat_text384_100000_r3.json b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_ivfflat_text384_100000_r3.json new file mode 100644 index 0000000..574753c --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_ivfflat_text384_100000_r3.json @@ -0,0 +1,279 @@ +{ + "system": "faiss_ivfflat", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 19:47:35", + "mem_baseline": { + "rss_mb": 37.234375, + "peak_mb": 37.234375 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 1.0200019460171461e-05, + "index_s": 1.3304623999865726, + "total_s": 1.3304726000060327, + "first_query_ms": 4.159200005233288, + "ready_s": 1.334631800011266 + }, + "mem_after_build": { + "rss_mb": 277.8125, + "peak_mb": 498.4375 + }, + "mem_after_first_query": { + "rss_mb": 277.89453125, + "peak_mb": 498.4375 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.7507, + "qps_1t_runs": [ + 7285.502064317353, + 7507.479356603735, + 7577.364855314709 + ], + "qps_1t": 7507.479356603735, + "qps_1t_min": 7285.502064317353, + "qps_1t_max": 7577.364855314709, + "median_ms": 0.1320999872405082, + "p95_ms": 0.16470502887386826, + "p99_ms": 0.17940002726390958, + "mean_ms": 0.13414380008665225, + "knob": 16, + "batch": { + "recall": 0.7507, + "qps_batch_runs": [ + 30472.3828828369, + 28825.504796822, + 30216.592520569906 + ], + "qps_batch": 30216.592520569906, + "qps_batch_min": 28825.504796822, + "qps_batch_max": 30472.3828828369, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8399, + "qps_1t_runs": [ + 4375.929048331583, + 4577.310549641881, + 4394.99778959987 + ], + "qps_1t": 4394.99778959987, + "qps_1t_min": 4375.929048331583, + "qps_1t_max": 4577.310549641881, + "median_ms": 0.22255000658333302, + "p95_ms": 0.2799999783746898, + "p99_ms": 0.31051297788508203, + "mean_ms": 0.22484106688837832, + "knob": 32, + "batch": { + "recall": 0.8399, + "qps_batch_runs": [ + 16490.928330762854, + 9868.521686228321, + 9042.588781546929 + ], + "qps_batch": 9868.521686228321, + "qps_batch_min": 9042.588781546929, + "qps_batch_max": 16490.928330762854, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9073, + "qps_1t_runs": [ + 1803.221633746764, + 1750.7312358168404, + 1638.9584341787581 + ], + "qps_1t": 1750.7312358168404, + "qps_1t_min": 1638.9584341787581, + "qps_1t_max": 1803.221633746764, + "median_ms": 0.5783000087831169, + "p95_ms": 0.7249250455060973, + "p99_ms": 0.8152999798767269, + "mean_ms": 0.5786321671136344, + "knob": 64, + "batch": { + "recall": 0.9073, + "qps_batch_runs": [ + 4918.885123941348, + 6470.096506607937, + 6079.862643150253 + ], + "qps_batch": 6079.862643150253, + "qps_batch_min": 4918.885123941348, + "qps_batch_max": 6470.096506607937, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9553, + "qps_1t_runs": [ + 974.8544099092795, + 969.4960663926317, + 990.1512616853405 + ], + "qps_1t": 974.8544099092795, + "qps_1t_min": 969.4960663926317, + "qps_1t_max": 990.1512616853405, + "median_ms": 1.0295500105712563, + "p95_ms": 1.2401099578710273, + "p99_ms": 1.320299984072335, + "mean_ms": 1.0224015338268753, + "knob": 128, + "batch": { + "recall": 0.9553, + "qps_batch_runs": [ + 3380.908348273267, + 2739.555649635965, + 2731.6232095222103 + ], + "qps_batch": 2739.555649635965, + "qps_batch_min": 2731.6232095222103, + "qps_batch_max": 3380.908348273267, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9855, + "qps_1t_runs": [ + 506.99043482036956, + 491.12389299006503, + 513.6935810756269 + ], + "qps_1t": 506.99043482036956, + "qps_1t_min": 491.12389299006503, + "qps_1t_max": 513.6935810756269, + "median_ms": 2.0009499858133495, + "p95_ms": 2.3466749989893283, + "p99_ms": 2.4855360086075953, + "mean_ms": 1.9850852338907619, + "knob": 256, + "batch": { + "recall": 0.9855, + "qps_batch_runs": [ + 1905.0003017724198, + 1933.582597971031, + 1681.3462202389444 + ], + "qps_batch": 1905.0003017724198, + "qps_batch_min": 1681.3462202389444, + "qps_batch_max": 1933.582597971031, + "threads": 32 + } + }, + { + "nq": 747, + "recall_queries": 1000, + "recall": 0.9957999999999999, + "qps_1t_runs": [ + 246.71073808717145, + 261.4811946990141, + 266.87451674832573 + ], + "qps_1t": 261.4811946990141, + "qps_1t_min": 246.71073808717145, + "qps_1t_max": 266.87451674832573, + "median_ms": 3.873299981933087, + "p95_ms": 4.447299987077713, + "p99_ms": 4.7837199876084915, + "mean_ms": 3.874925346448702, + "knob": 512, + "batch": { + "recall": 0.9958, + "qps_batch_runs": [ + 880.3520563043093, + 920.2080258992818, + 940.3634655166154 + ], + "qps_batch": 920.2080258992818, + "qps_batch_min": 880.3520563043093, + "qps_batch_max": 940.3634655166154, + "threads": 32 + } + } + ], + "knob_name": "nprobe", + "mem_after_sweep": { + "rss_mb": 272.265625, + "peak_mb": 498.4375 + }, + "ref_knob": 128, + "ref_recall_pre": 0.9553, + "mutation": { + "ref_knob": 128, + "recall_before": 0.9553, + "insert_10k": { + "seconds": 0.07001020002644509, + "per_s": 142836.32950945263, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.06997050001518801, + "per_s": 142917.37229017043, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.003690499986987561, + "per_s": 2709659.947231889, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 328.67578125, + "peak_mb": 498.4375 + }, + "first_query_after_ms": 2.257100015413016, + "after": { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9521, + "qps_1t_runs": [ + 936.2370161475986, + 1064.5397435871023, + 1072.2346233955593 + ], + "qps_1t": 1064.5397435871023, + "qps_1t_min": 936.2370161475986, + "qps_1t_max": 1072.2346233955593, + "median_ms": 0.9387999889440835, + "p95_ms": 1.1437049775850028, + "p99_ms": 1.506989949266424, + "mean_ms": 0.980036800475015 + }, + "recall_after": 0.9521 + }, + "saveload": { + "save_s": 0.13201040000421926, + "disk_mb": 149.110125541687, + "load_s": 0.17764680000254884, + "first_query_after_load_ms": 2.119200013112277, + "time_to_ready_s": 0.17976600001566112 + }, + "mem_end": { + "rss_mb": 246.32421875, + "peak_mb": 498.4375 + }, + "status": "ok", + "finished": "2026-11-16 19:48:09" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_ivfflat_text384_100000_r3.load.json b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_ivfflat_text384_100000_r3.load.json new file mode 100644 index 0000000..2128b76 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_ivfflat_text384_100000_r3.load.json @@ -0,0 +1 @@ +{"total_pct": 14.3625, "pin_cpu_pct": 26.7, "max_cpu_pct": 73.1, "noisy": true, "seconds": 34.53615975379944, "after": 21.1} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_ivfflat_text384_100000_r3.txt b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_ivfflat_text384_100000_r3.txt new file mode 100644 index 0000000..73b1b88 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/faiss_ivfflat_text384_100000_r3.txt @@ -0,0 +1,7 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_ivfflat --ds text384 --n 100000 --exp core,mutation,saveload --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_ivfflat_text384_100000_r3.json + faiss_ivfflat text384 100000 knob=16 recall=0.7507 qps1t=7507 med=0.132ms batch=30217 + faiss_ivfflat text384 100000 knob=32 recall=0.8399 qps1t=4395 med=0.223ms batch=9869 + faiss_ivfflat text384 100000 knob=64 recall=0.9073 qps1t=1751 med=0.578ms batch=6080 + faiss_ivfflat text384 100000 knob=128 recall=0.9553 qps1t=975 med=1.030ms batch=2740 + faiss_ivfflat text384 100000 knob=256 recall=0.9855 qps1t=507 med=2.001ms batch=1905 + faiss_ivfflat text384 100000 knob=512 recall=0.9958 qps1t=261 med=3.873ms batch=920 diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_glove_100000_r2.json b/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_glove_100000_r2.json new file mode 100644 index 0000000..a106390 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_glove_100000_r2.json @@ -0,0 +1,87 @@ +{ + "system": "numpy_bf", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 19:57:21", + "mem_baseline": { + "rss_mb": 34.91796875, + "peak_mb": 34.91796875 + }, + "version": "numpy 2.4.2", + "build": { + "ingest_s": 0.011569199967198074, + "index_s": 1.700012944638729e-06, + "total_s": 0.011570899980142713, + "first_query_ms": 2.2356000263243914, + "ready_s": 0.013806500006467104 + }, + "mem_after_build": { + "rss_mb": 86.12109375, + "peak_mb": 95.8828125 + }, + "mem_after_first_query": { + "rss_mb": 85.88671875, + "peak_mb": 95.8828125 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 529.0706604317588, + 420.05137053488613, + 380.85617297359505 + ], + "qps_1t": 420.05137053488613, + "qps_1t_min": 380.85617297359505, + "qps_1t_max": 529.0706604317588, + "median_ms": 2.118349977536127, + "p95_ms": 3.0774049781030044, + "p99_ms": 3.3959160133963446, + "mean_ms": 2.298810333615014, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 753.9822325558588, + 786.9592665271316, + 842.0729984681399 + ], + "qps_batch": 786.9592665271316, + "qps_batch_min": 753.9822325558588, + "qps_batch_max": 842.0729984681399, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 881, + "threads": 32, + "qps": 308.74627739531485, + "median_ms": 3.0184999923221767, + "p95_ms": 4.503600008320063, + "p99_ms": 5.758448023116213, + "mean_ms": 3.087255882860045 + }, + "mem_after_sweep": { + "rss_mb": 91.1796875, + "peak_mb": 1617.26953125 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 89.6171875, + "peak_mb": 1617.26953125 + }, + "status": "ok", + "finished": "2026-11-16 19:57:40" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_glove_100000_r2.load.json b/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_glove_100000_r2.load.json new file mode 100644 index 0000000..51910af --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_glove_100000_r2.load.json @@ -0,0 +1 @@ +{"total_pct": 12.415625, "pin_cpu_pct": 34.7, "max_cpu_pct": 68.6, "noisy": true, "seconds": 19.34105086326599, "after": 16.8} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_glove_100000_r2.txt b/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_glove_100000_r2.txt new file mode 100644 index 0000000..38297a4 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_glove_100000_r2.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system numpy_bf --ds glove --n 100000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\numpy_bf_glove_100000_r2.json + numpy_bf glove 100000 knob=None recall=1.0000 qps1t=420 med=2.118ms batch=787 diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_sift_100000_r1.json b/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_sift_100000_r1.json new file mode 100644 index 0000000..0d18152 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_sift_100000_r1.json @@ -0,0 +1,87 @@ +{ + "system": "numpy_bf", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 20:03:52", + "mem_baseline": { + "rss_mb": 35.26953125, + "peak_mb": 35.26953125 + }, + "version": "numpy 2.4.2", + "build": { + "ingest_s": 0.014228800078853965, + "index_s": 2.200016751885414e-06, + "total_s": 0.01423100009560585, + "first_query_ms": 3.2529999734833837, + "ready_s": 0.017484000069089234 + }, + "mem_after_build": { + "rss_mb": 96.92578125, + "peak_mb": 108.82421875 + }, + "mem_after_first_query": { + "rss_mb": 96.7109375, + "peak_mb": 108.82421875 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 399.81694815538305, + 313.20186579405873, + 305.0272795404543 + ], + "qps_1t": 313.20186579405873, + "qps_1t_min": 305.0272795404543, + "qps_1t_max": 399.81694815538305, + "median_ms": 3.097050008364022, + "p95_ms": 3.686200009542517, + "p99_ms": 4.008269019541329, + "mean_ms": 2.990789633069653, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 831.5500050172542, + 820.7834854256291, + 830.8164441603105 + ], + "qps_batch": 830.8164441603105, + "qps_batch_min": 820.7834854256291, + "qps_batch_max": 831.5500050172542, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 776, + "threads": 32, + "qps": 291.31391622988434, + "median_ms": 3.4472999977879226, + "p95_ms": 5.117804990732111, + "p99_ms": 6.085433959960939, + "mean_ms": 3.5879129295217758 + }, + "mem_after_sweep": { + "rss_mb": 103.55859375, + "peak_mb": 1629.65234375 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 101.79296875, + "peak_mb": 1629.65234375 + }, + "status": "ok", + "finished": "2026-11-16 20:04:13" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_sift_100000_r1.load.json b/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_sift_100000_r1.load.json new file mode 100644 index 0000000..4481d5a --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_sift_100000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 10.0875, "pin_cpu_pct": 33.7, "max_cpu_pct": 55.1, "noisy": true, "seconds": 21.508023977279663, "after": 16.4} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_sift_100000_r1.txt b/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_sift_100000_r1.txt new file mode 100644 index 0000000..86a1bc0 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_sift_100000_r1.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system numpy_bf --ds sift --n 100000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\numpy_bf_sift_100000_r1.json + numpy_bf sift 100000 knob=None recall=1.0000 qps1t=313 med=3.097ms batch=831 diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_sift_100000_r3.json b/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_sift_100000_r3.json new file mode 100644 index 0000000..dc827d7 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_sift_100000_r3.json @@ -0,0 +1,87 @@ +{ + "system": "numpy_bf", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 20:09:19", + "mem_baseline": { + "rss_mb": 35.18359375, + "peak_mb": 35.18359375 + }, + "version": "numpy 2.4.2", + "build": { + "ingest_s": 0.013652800000272691, + "index_s": 1.700012944638729e-06, + "total_s": 0.01365450001321733, + "first_query_ms": 2.809799974784255, + "ready_s": 0.016464299988001585 + }, + "mem_after_build": { + "rss_mb": 96.875, + "peak_mb": 108.7734375 + }, + "mem_after_first_query": { + "rss_mb": 96.67578125, + "peak_mb": 108.7734375 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 413.91828347591104, + 309.43518942139724, + 293.520247103634 + ], + "qps_1t": 309.43518942139724, + "qps_1t_min": 293.520247103634, + "qps_1t_max": 413.91828347591104, + "median_ms": 3.2204999879468232, + "p95_ms": 3.798749993438831, + "p99_ms": 4.213713955250569, + "mean_ms": 3.0181834338388094, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 715.2407650659659, + 782.6293843126605, + 751.3023262352797 + ], + "qps_batch": 751.3023262352797, + "qps_batch_min": 715.2407650659659, + "qps_batch_max": 782.6293843126605, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 1000, + "threads": 32, + "qps": 261.8867721557979, + "median_ms": 3.5610499908216298, + "p95_ms": 6.239320011809468, + "p99_ms": 8.493966002133657, + "mean_ms": 3.8540874336225293 + }, + "mem_after_sweep": { + "rss_mb": 103.453125, + "peak_mb": 1629.52734375 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 101.671875, + "peak_mb": 1629.52734375 + }, + "status": "ok", + "finished": "2026-11-16 20:09:44" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_sift_100000_r3.load.json b/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_sift_100000_r3.load.json new file mode 100644 index 0000000..bbad2fb --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_sift_100000_r3.load.json @@ -0,0 +1 @@ +{"total_pct": 8.765625, "pin_cpu_pct": 25.7, "max_cpu_pct": 61.9, "noisy": true, "seconds": 25.21201467514038, "after": 14.9} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_sift_100000_r3.txt b/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_sift_100000_r3.txt new file mode 100644 index 0000000..e26fafc --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/numpy_bf_sift_100000_r3.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system numpy_bf --ds sift --n 100000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\numpy_bf_sift_100000_r3.json + numpy_bf sift 100000 knob=None recall=1.0000 qps1t=309 med=3.220ms batch=751 diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_glove_100000_r1.json b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_glove_100000_r1.json new file mode 100644 index 0000000..1ae99b9 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_glove_100000_r1.json @@ -0,0 +1,382 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_exact_query_ms": 4.5451, + "ingest_s": 0.0478865 + }, + "d": 100, + "ds": "glove", + "exps": [ + "core", + "int8" + ], + "first_exact_query_ms": 4.5451, + "int8_rerank": 0, + "int8_study": { + "by_factor": [ + { + "applies": true, + "auto_threads": { + "max_ms": 1.727, + "mean_ms": 0.45291340000000013, + "median_ms": 0.4498, + "nq": 1000, + "p95_ms": 0.6024999999999999, + "p99_ms": 0.8182, + "qps_1t": 2278.945167439797, + "qps_1t_max": 2289.579619444391, + "qps_1t_min": 2069.6250853461675, + "qps_1t_runs": [ + 2289.579619444391, + 2278.945167439797, + 2069.6250853461675 + ], + "recall": 0.9832 + }, + "batch": { + "nq": 1000, + "qps_batch": 8544.313801886414, + "qps_batch_max": 8608.837660565583, + "qps_batch_min": 8527.053783539035, + "qps_batch_runs": [ + 8544.313801886414, + 8608.837660565583, + 8527.053783539035 + ], + "recall": 0.9832, + "threads": 32 + }, + "min_recall_vs_f32_scan": 0.8, + "one_thread": { + "max_ms": 2.6056999999999997, + "mean_ms": 0.8487854999999993, + "median_ms": 0.8675, + "nq": 1000, + "p95_ms": 0.887, + "p99_ms": 0.9311, + "qps_1t": 1173.3131335744247, + "qps_1t_max": 1193.2620311539238, + "qps_1t_min": 1168.1830851294717, + "qps_1t_runs": [ + 1193.2620311539238, + 1173.3131335744247, + 1168.1830851294717 + ], + "recall": 0.9832 + }, + "recall_vs_f32_scan": 0.9831999999999973, + "report_f32_ms": 1.1483, + "report_int8_ms": 0.31129999999999997, + "report_queries": 1000, + "rerank_factor": 1, + "scores_exact": true + }, + { + "applies": true, + "auto_threads": { + "max_ms": 1.8254000000000001, + "mean_ms": 0.41373863333333344, + "median_ms": 0.4123, + "nq": 1000, + "p95_ms": 0.5479999999999999, + "p99_ms": 0.6847, + "qps_1t": 2383.697606028659, + "qps_1t_max": 2578.0670941961284, + "qps_1t_min": 2305.1454765784506, + "qps_1t_runs": [ + 2578.0670941961284, + 2305.1454765784506, + 2383.697606028659 + ], + "recall": 1.0 + }, + "batch": { + "nq": 1000, + "qps_batch": 8102.368565402725, + "qps_batch_max": 8112.918849717914, + "qps_batch_min": 7830.85356303837, + "qps_batch_runs": [ + 8112.918849717914, + 8102.368565402725, + 7830.85356303837 + ], + "recall": 1.0, + "threads": 32 + }, + "min_recall_vs_f32_scan": 1.0, + "one_thread": { + "max_ms": 1.1607, + "mean_ms": 0.8743498333333365, + "median_ms": 0.8759, + "nq": 1000, + "p95_ms": 0.8997, + "p99_ms": 0.9206000000000001, + "qps_1t": 1140.047414571973, + "qps_1t_max": 1151.1771937983774, + "qps_1t_min": 1139.9687876545945, + "qps_1t_runs": [ + 1140.047414571973, + 1139.9687876545945, + 1151.1771937983774 + ], + "recall": 1.0 + }, + "recall_vs_f32_scan": 1.0, + "report_f32_ms": 1.1483, + "report_int8_ms": 0.40509999999999996, + "report_queries": 1000, + "rerank_factor": 2, + "scores_exact": true + }, + { + "applies": true, + "auto_threads": { + "max_ms": 2.0837, + "mean_ms": 0.5178238999999991, + "median_ms": 0.5023, + "nq": 1000, + "p95_ms": 0.7371, + "p99_ms": 0.9817999999999999, + "qps_1t": 1952.580413119156, + "qps_1t_max": 1979.030587698861, + "qps_1t_min": 1865.5637453803965, + "qps_1t_runs": [ + 1952.580413119156, + 1979.030587698861, + 1865.5637453803965 + ], + "recall": 1.0 + }, + "batch": { + "nq": 1000, + "qps_batch": 6668.951894182407, + "qps_batch_max": 6680.700591709651, + "qps_batch_min": 6612.444620776301, + "qps_batch_runs": [ + 6680.700591709651, + 6612.444620776301, + 6668.951894182407 + ], + "recall": 1.0, + "threads": 32 + }, + "min_recall_vs_f32_scan": 1.0, + "one_thread": { + "max_ms": 1.3409, + "mean_ms": 0.879685399999999, + "median_ms": 0.8987999999999999, + "nq": 1000, + "p95_ms": 0.9374, + "p99_ms": 1.1155, + "qps_1t": 1142.8771268800463, + "qps_1t_max": 1152.828516642866, + "qps_1t_min": 1115.2750524792673, + "qps_1t_runs": [ + 1142.8771268800463, + 1152.828516642866, + 1115.2750524792673 + ], + "recall": 1.0 + }, + "recall_vs_f32_scan": 1.0, + "report_f32_ms": 1.1483, + "report_int8_ms": 0.50675, + "report_queries": 1000, + "rerank_factor": 4, + "scores_exact": true + }, + { + "applies": true, + "auto_threads": { + "max_ms": 1.2570999999999999, + "mean_ms": 0.5269278666666656, + "median_ms": 0.5178, + "nq": 1000, + "p95_ms": 0.7171000000000001, + "p99_ms": 0.9384, + "qps_1t": 1918.0148392972076, + "qps_1t_max": 1952.1165335485896, + "qps_1t_min": 1827.6634036173507, + "qps_1t_runs": [ + 1952.1165335485896, + 1827.6634036173507, + 1918.0148392972076 + ], + "recall": 1.0 + }, + "batch": { + "nq": 1000, + "qps_batch": 5541.679805065872, + "qps_batch_max": 5549.309499418988, + "qps_batch_min": 5333.290667007997, + "qps_batch_runs": [ + 5549.309499418988, + 5541.679805065872, + 5333.290667007997 + ], + "recall": 1.0, + "threads": 32 + }, + "min_recall_vs_f32_scan": 1.0, + "one_thread": { + "max_ms": 3.1641999999999997, + "mean_ms": 0.9189932000000032, + "median_ms": 0.9322999999999999, + "nq": 1000, + "p95_ms": 0.976, + "p99_ms": 1.0865, + "qps_1t": 1093.0937679882231, + "qps_1t_max": 1099.93685262529, + "qps_1t_min": 1071.8091785236436, + "qps_1t_runs": [ + 1071.8091785236436, + 1099.93685262529, + 1093.0937679882231 + ], + "recall": 1.0 + }, + "recall_vs_f32_scan": 1.0, + "report_f32_ms": 1.1483, + "report_int8_ms": 0.5238499999999999, + "report_queries": 1000, + "rerank_factor": 8, + "scores_exact": true + } + ], + "f32": { + "auto_threads": { + "max_ms": 3.3796999999999997, + "mean_ms": 1.227306299999999, + "median_ms": 1.1865999999999999, + "nq": 1000, + "p95_ms": 1.5836, + "p99_ms": 1.9675, + "qps_1t": 828.4732331896979, + "qps_1t_max": 834.830741406502, + "qps_1t_min": 783.0659176275735, + "qps_1t_runs": [ + 783.0659176275735, + 828.4732331896979, + 834.830741406502 + ], + "recall": 1.0 + }, + "batch": { + "nq": 1000, + "qps_batch": 10608.94270216136, + "qps_batch_max": 10850.706218214213, + "qps_batch_min": 10573.898331967537, + "qps_batch_runs": [ + 10608.94270216136, + 10850.706218214213, + 10573.898331967537 + ], + "recall": 1.0, + "threads": 32 + }, + "mem": { + "peak_mb": 67.01171875, + "rss_mb": 63.6484375 + }, + "one_thread": { + "max_ms": 3.2461, + "mean_ms": 1.8106311666666626, + "median_ms": 1.7505000000000002, + "nq": 1000, + "p95_ms": 2.1877, + "p99_ms": 2.3481, + "qps_1t": 572.2254362217556, + "qps_1t_max": 615.4009755828596, + "qps_1t_min": 485.58452504273566, + "qps_1t_runs": [ + 615.4009755828596, + 572.2254362217556, + 485.58452504273566 + ], + "recall": 1.0 + } + }, + "mem_after_int8": { + "peak_mb": 71.9921875, + "rss_mb": 71.9921875 + }, + "quant_build_s": 0.0568956, + "quant_bytes_mb": 10.0703125, + "rss_delta_mb": 8.34375 + }, + "k": 10, + "knob_name": "", + "mem_after_first_exact_query": { + "peak_mb": 98.5546875, + "rss_mb": 70.23046875 + }, + "mem_after_ingest": { + "peak_mb": 64.44140625, + "rss_mb": 58.98046875 + }, + "mem_after_sweep": { + "peak_mb": 98.5546875, + "rss_mb": 70.24609375 + }, + "mem_baseline": { + "peak_mb": 5.7265625, + "rss_mb": 5.7265625 + }, + "mem_end": { + "peak_mb": 98.5546875, + "rss_mb": 70.2578125 + }, + "mode": "exact", + "n": 100000, + "queries": 1000, + "raw_vectors_mb": 38.14697265625, + "ref_knob": null, + "started": "SystemTime { intervals: 134393539920950625 }", + "status": "ok", + "sweep": [ + { + "auto_threads": { + "median_ms": 1.2537, + "p99_ms": 2.2649, + "qps": 739.1417307964513, + "runs": [ + 729.4243966201393, + 739.1417307964513, + 838.5730873887921 + ] + }, + "batch": { + "nq": 1000, + "qps_batch": 10086.155944074282, + "qps_batch_max": 10223.224098183844, + "qps_batch_min": 9831.710609497235, + "qps_batch_runs": [ + 9831.710609497235, + 10086.155944074282, + 10223.224098183844 + ], + "recall": 1.0, + "threads": 32 + }, + "knob": null, + "max_ms": 4.617900000000001, + "mean_ms": 2.193720400000003, + "median_ms": 2.2005000000000003, + "nq": 1000, + "p95_ms": 2.4535, + "p99_ms": 2.6914000000000002, + "qps_1t": 451.68470943822797, + "qps_1t_max": 473.6119298677346, + "qps_1t_min": 443.302850942697, + "qps_1t_runs": [ + 473.6119298677346, + 451.68470943822797, + 443.302850942697 + ], + "recall": 1.0 + } + ], + "system": "rust_exact", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_glove_100000_r1.load.json b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_glove_100000_r1.load.json new file mode 100644 index 0000000..c0ecd7a --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_glove_100000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 21.240625, "pin_cpu_pct": 64.6, "max_cpu_pct": 78.6, "noisy": true, "seconds": 42.928778648376465, "after": 19.4} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_glove_100000_r1.txt b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_glove_100000_r1.txt new file mode 100644 index 0000000..d3e29aa --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_glove_100000_r1.txt @@ -0,0 +1,13 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/glove --ds glove --n 100000 --mode exact --exp core,int8 --queries 1000 --budget 3 --label rust_exact --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_exact_glove_100000_r1.json +rust_exact: ds=glove n=100000 d=100 mode=exact int8=0 queries=1000 cpus=32 exps=["core", "int8"] + ingest 0.05s rss 59 MB + int8 study: f32 1T 572.2 qps, auto 828.5 qps, batch 10609 qps + rerank x1: recall vs f32 0.9832 (worst 0.80) exact_scores=true applies=true int8 0.311 ms f32 1.148 ms + 1T 1173.3 qps (recall vs float64 truth 0.9832), auto 2278.9 qps, batch 8544 qps + rerank x2: recall vs f32 1.0000 (worst 1.00) exact_scores=true applies=true int8 0.405 ms f32 1.148 ms + 1T 1140.0 qps (recall vs float64 truth 1.0000), auto 2383.7 qps, batch 8102 qps + rerank x4: recall vs f32 1.0000 (worst 1.00) exact_scores=true applies=true int8 0.507 ms f32 1.148 ms + 1T 1142.9 qps (recall vs float64 truth 1.0000), auto 1952.6 qps, batch 6669 qps + rerank x8: recall vs f32 1.0000 (worst 1.00) exact_scores=true applies=true int8 0.524 ms f32 1.148 ms + 1T 1093.1 qps (recall vs float64 truth 1.0000), auto 1918.0 qps, batch 5542 qps + exact 1T recall=1.0000 qps=451.7 median=2.201ms | auto 739.1 qps | batch 10086 qps diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_glove_100000_r3.json b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_glove_100000_r3.json new file mode 100644 index 0000000..a829559 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_glove_100000_r3.json @@ -0,0 +1,89 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_exact_query_ms": 2.1896, + "ingest_s": 0.0482919 + }, + "d": 100, + "ds": "glove", + "exps": [ + "core" + ], + "first_exact_query_ms": 2.1896, + "int8_rerank": 0, + "k": 10, + "knob_name": "", + "mem_after_first_exact_query": { + "peak_mb": 64.42578125, + "rss_mb": 60.61328125 + }, + "mem_after_ingest": { + "peak_mb": 64.42578125, + "rss_mb": 58.96875 + }, + "mem_after_sweep": { + "peak_mb": 66.9453125, + "rss_mb": 63.48046875 + }, + "mem_baseline": { + "peak_mb": 5.7265625, + "rss_mb": 5.7265625 + }, + "mem_end": { + "peak_mb": 66.9453125, + "rss_mb": 63.50390625 + }, + "mode": "exact", + "n": 100000, + "queries": 1000, + "raw_vectors_mb": 38.14697265625, + "ref_knob": null, + "started": "SystemTime { intervals: 134393543669655366 }", + "status": "ok", + "sweep": [ + { + "auto_threads": { + "median_ms": 1.3327, + "p99_ms": 3.0884, + "qps": 648.0928022230088, + "runs": [ + 621.9519688884707, + 648.0928022230088, + 747.8636154030607 + ] + }, + "batch": { + "nq": 1000, + "qps_batch": 10014.892144619049, + "qps_batch_max": 10686.283834110403, + "qps_batch_min": 9957.41214824197, + "qps_batch_runs": [ + 10686.283834110403, + 9957.41214824197, + 10014.892144619049 + ], + "recall": 1.0, + "threads": 32 + }, + "knob": null, + "max_ms": 8.0916, + "mean_ms": 2.3622259333333284, + "median_ms": 2.2108000000000003, + "nq": 1000, + "p95_ms": 3.9519, + "p99_ms": 5.4615, + "qps_1t": 411.6460428568818, + "qps_1t_max": 495.9394215947801, + "qps_1t_min": 378.64000839066233, + "qps_1t_runs": [ + 495.9394215947801, + 378.64000839066233, + 411.6460428568818 + ], + "recall": 1.0 + } + ], + "system": "rust_exact", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_glove_100000_r3.load.json b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_glove_100000_r3.load.json new file mode 100644 index 0000000..0053b0e --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_glove_100000_r3.load.json @@ -0,0 +1 @@ +{"total_pct": 17.24375, "pin_cpu_pct": 36.6, "max_cpu_pct": 69.1, "noisy": true, "seconds": 12.352574110031128, "after": 21.8} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_glove_100000_r3.txt b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_glove_100000_r3.txt new file mode 100644 index 0000000..12d0b6c --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_glove_100000_r3.txt @@ -0,0 +1,4 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/glove --ds glove --n 100000 --mode exact --exp core --queries 1000 --budget 3 --label rust_exact --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_exact_glove_100000_r3.json +rust_exact: ds=glove n=100000 d=100 mode=exact int8=0 queries=1000 cpus=32 exps=["core"] + ingest 0.05s rss 59 MB + exact 1T recall=1.0000 qps=411.6 median=2.211ms | auto 648.1 qps | batch 10015 qps diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_sift_100000_r2.json b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_sift_100000_r2.json new file mode 100644 index 0000000..8644498 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_sift_100000_r2.json @@ -0,0 +1,89 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_exact_query_ms": 2.868, + "ingest_s": 0.0495233 + }, + "d": 128, + "ds": "sift", + "exps": [ + "core" + ], + "first_exact_query_ms": 2.868, + "int8_rerank": 0, + "k": 10, + "knob_name": "", + "mem_after_first_exact_query": { + "peak_mb": 77.51171875, + "rss_mb": 70.96875 + }, + "mem_after_ingest": { + "peak_mb": 77.51171875, + "rss_mb": 69.9765625 + }, + "mem_after_sweep": { + "peak_mb": 78.0234375, + "rss_mb": 73.9296875 + }, + "mem_baseline": { + "peak_mb": 5.859375, + "rss_mb": 5.859375 + }, + "mem_end": { + "peak_mb": 78.0234375, + "rss_mb": 73.953125 + }, + "mode": "exact", + "n": 100000, + "queries": 1000, + "raw_vectors_mb": 48.828125, + "ref_knob": null, + "started": "SystemTime { intervals: 134393547247713928 }", + "status": "ok", + "sweep": [ + { + "auto_threads": { + "median_ms": 1.4118, + "p99_ms": 1.981, + "qps": 699.8085253893678, + "runs": [ + 702.800794699026, + 673.6521248301644, + 699.8085253893678 + ] + }, + "batch": { + "nq": 1000, + "qps_batch": 11358.679848793254, + "qps_batch_max": 13009.99167360533, + "qps_batch_min": 11163.916435852694, + "qps_batch_runs": [ + 11358.679848793254, + 11163.916435852694, + 13009.99167360533 + ], + "recall": 1.0, + "threads": 32 + }, + "knob": null, + "max_ms": 29.884, + "mean_ms": 2.5058305000000023, + "median_ms": 2.5758, + "nq": 1000, + "p95_ms": 2.9754, + "p99_ms": 3.2007, + "qps_1t": 395.1776315550284, + "qps_1t_max": 459.4925603100073, + "qps_1t_min": 355.78704167379914, + "qps_1t_runs": [ + 459.4925603100073, + 395.1776315550284, + 355.78704167379914 + ], + "recall": 1.0 + } + ], + "system": "rust_exact", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_sift_100000_r2.load.json b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_sift_100000_r2.load.json new file mode 100644 index 0000000..0839f5f --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_sift_100000_r2.load.json @@ -0,0 +1 @@ +{"total_pct": 10.340625, "pin_cpu_pct": 26.4, "max_cpu_pct": 56.0, "noisy": true, "seconds": 12.572738647460938, "after": 15.3} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_sift_100000_r2.txt b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_sift_100000_r2.txt new file mode 100644 index 0000000..478681b --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_sift_100000_r2.txt @@ -0,0 +1,4 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/sift --ds sift --n 100000 --mode exact --exp core --queries 1000 --budget 3 --label rust_exact --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_exact_sift_100000_r2.json +rust_exact: ds=sift n=100000 d=128 mode=exact int8=0 queries=1000 cpus=32 exps=["core"] + ingest 0.05s rss 70 MB + exact 1T recall=1.0000 qps=395.2 median=2.576ms | auto 699.8 qps | batch 11359 qps diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_persist_r1.json b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_persist_r1.json new file mode 100644 index 0000000..90d6d28 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_persist_r1.json @@ -0,0 +1,70 @@ +{ + "all_threads": 32, + "budget": 3.0, + "d": 384, + "ds": "text384", + "exps": [ + "persist" + ], + "ingest_s": 0.10159570000000001, + "int8_rerank": 0, + "k": 10, + "mem_after_ingest": { + "peak_mb": 196.39453125, + "rss_mb": 169.06640625 + }, + "mem_baseline": { + "peak_mb": 7.87890625, + "rss_mb": 7.87890625 + }, + "mem_end": { + "peak_mb": 849.9296875, + "rss_mb": 31.2421875 + }, + "mode": "exact", + "n": 100000, + "persist": { + "json_answers_equal_original": true, + "json_first_query_ms": 5.4064, + "json_load_peak_mb": 849.9296875, + "json_load_s": 3.0140192, + "json_load_s_runs": [ + 1.8661548, + 3.0887956, + 3.0140192 + ], + "json_mb": 453.31162452697754, + "json_save_s": 1.6535682, + "sidecar_fingerprint_s": 0.0005338, + "snapshot_answers_equal_original": true, + "snapshot_first_query_ms": 4.9331000000000005, + "snapshot_load_s": 0.4115582, + "snapshot_load_s_runs": [ + 0.4115582, + 0.4445507, + 0.3977727 + ], + "snapshot_mb": 148.76275157928467, + "snapshot_save_s": 0.1824849, + "time_to_ready_after_snapshot_load": { + "hnsw": { + "build_s": 9.3586762, + "first_query_ms": 0.5157999999999999, + "load_s": 0.4025245, + "ready_s": 9.7617165 + }, + "ivf": { + "build_s": 1.3107408, + "first_query_ms": 1.1686, + "load_s": 0.4416809, + "ready_s": 1.7535903 + } + } + }, + "queries": 1000, + "raw_vectors_mb": 146.484375, + "started": "SystemTime { intervals: 134393550900098785 }", + "status": "ok", + "system": "rust_exact", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_persist_r1.load.json b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_persist_r1.load.json new file mode 100644 index 0000000..07ea27f --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_persist_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 10.21875, "pin_cpu_pct": 23.4, "max_cpu_pct": 57.4, "noisy": true, "seconds": 23.79219102859497, "after": 19.2} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_persist_r1.txt b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_persist_r1.txt new file mode 100644 index 0000000..063345d --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_persist_r1.txt @@ -0,0 +1,8 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/text384 --ds text384 --n 100000 --mode exact --exp persist --queries 1000 --budget 3 --label rust_exact --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_exact_text384_100000_persist_r1.json --ready ivf,hnsw +rust_exact: ds=text384 n=100000 d=384 mode=exact int8=0 queries=1000 cpus=32 exps=["persist"] + save: JSON 1.65s 453 MB | snapshot 0.18s 149 MB (+ sidecar fingerprint 0.00s) + load rep 0: JSON 1.87s (first query 4.9 ms) | snapshot 0.41s (first query 4.9 ms) | answers equal ref: json=true snap=true + load rep 1: JSON 3.09s (first query 4.9 ms) | snapshot 0.44s (first query 5.6 ms) | answers equal ref: json=true snap=true + load rep 2: JSON 3.01s (first query 5.4 ms) | snapshot 0.40s (first query 4.9 ms) | answers equal ref: json=true snap=true + ready ivf: load 0.44s + build 1.31s + first query 1.2 ms = 1.75s + ready hnsw: load 0.40s + build 9.36s + first query 0.5 ms = 9.76s diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_r2.json b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_r2.json new file mode 100644 index 0000000..d4e5f0f --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_r2.json @@ -0,0 +1,405 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_exact_query_ms": 7.24, + "ingest_s": 0.10720729999999999 + }, + "concurrent_clients": [ + { + "clients": 1, + "qps": 146.4381304390325 + }, + { + "clients": 2, + "qps": 244.74466530618338 + }, + { + "clients": 4, + "qps": 441.06166434270045 + }, + { + "clients": 8, + "qps": 662.1025993257715 + }, + { + "clients": 16, + "qps": 1033.9934762837363 + } + ], + "d": 384, + "ds": "text384", + "exps": [ + "core", + "int8", + "conc" + ], + "first_exact_query_ms": 7.24, + "int8_rerank": 0, + "int8_study": { + "by_factor": [ + { + "applies": true, + "auto_threads": { + "max_ms": 2.0642, + "mean_ms": 0.7818631666666674, + "median_ms": 0.7482, + "nq": 1000, + "p95_ms": 1.1072, + "p99_ms": 1.4021000000000001, + "qps_1t": 1272.0793217781875, + "qps_1t_max": 1331.976670695011, + "qps_1t_min": 1236.5355192973127, + "qps_1t_runs": [ + 1236.5355192973127, + 1331.976670695011, + 1272.0793217781875 + ], + "recall": 0.9905 + }, + "batch": { + "nq": 1000, + "qps_batch": 4697.974280470004, + "qps_batch_max": 4718.501290746028, + "qps_batch_min": 4463.893130827334, + "qps_batch_runs": [ + 4463.893130827334, + 4718.501290746028, + 4697.974280470004 + ], + "recall": 0.9905, + "threads": 32 + }, + "min_recall_vs_f32_scan": 0.8, + "one_thread": { + "max_ms": 17.8557, + "mean_ms": 2.8422506666666663, + "median_ms": 2.7439, + "nq": 1000, + "p95_ms": 3.556, + "p99_ms": 4.650600000000001, + "qps_1t": 359.52478589040624, + "qps_1t_max": 360.5579981156516, + "qps_1t_min": 336.4937159630299, + "qps_1t_runs": [ + 336.4937159630299, + 359.52478589040624, + 360.5579981156516 + ], + "recall": 0.9905 + }, + "recall_vs_f32_scan": 0.9903999999999983, + "report_f32_ms": 4.4929, + "report_int8_ms": 1.1707, + "report_queries": 1000, + "rerank_factor": 1, + "scores_exact": true + }, + { + "applies": true, + "auto_threads": { + "max_ms": 4.1244000000000005, + "mean_ms": 0.6981585666666644, + "median_ms": 0.6899000000000001, + "nq": 1000, + "p95_ms": 1.1565, + "p99_ms": 1.5410000000000001, + "qps_1t": 1264.4645258270652, + "qps_1t_max": 1992.7301219690312, + "qps_1t_min": 1247.189147458915, + "qps_1t_runs": [ + 1247.189147458915, + 1264.4645258270652, + 1992.7301219690312 + ], + "recall": 0.9999 + }, + "batch": { + "nq": 1000, + "qps_batch": 12409.272704936036, + "qps_batch_max": 12598.377580935128, + "qps_batch_min": 12289.75304970222, + "qps_batch_runs": [ + 12409.272704936036, + 12598.377580935128, + 12289.75304970222 + ], + "recall": 0.9999, + "threads": 32 + }, + "min_recall_vs_f32_scan": 1.0, + "one_thread": { + "max_ms": 5.482500000000001, + "mean_ms": 2.6941757002801103, + "median_ms": 2.627, + "nq": 952, + "p95_ms": 3.3311, + "p99_ms": 3.8906, + "qps_1t": 369.75652620268727, + "qps_1t_max": 387.49154499147244, + "qps_1t_min": 357.48204666582666, + "qps_1t_runs": [ + 369.75652620268727, + 357.48204666582666, + 387.49154499147244 + ], + "recall": 0.9998949579831933 + }, + "recall_vs_f32_scan": 1.0, + "report_f32_ms": 4.4929, + "report_int8_ms": 0.844, + "report_queries": 1000, + "rerank_factor": 2, + "scores_exact": true + }, + { + "applies": true, + "auto_threads": { + "max_ms": 5.6644, + "mean_ms": 0.3734949333333331, + "median_ms": 0.3341, + "nq": 1000, + "p95_ms": 0.5325, + "p99_ms": 0.8235, + "qps_1t": 2741.0041613925146, + "qps_1t_max": 2769.1426678917346, + "qps_1t_min": 2534.6460772296537, + "qps_1t_runs": [ + 2741.0041613925146, + 2769.1426678917346, + 2534.6460772296537 + ], + "recall": 0.9999 + }, + "batch": { + "nq": 1000, + "qps_batch": 10648.776721774086, + "qps_batch_max": 10778.254655667099, + "qps_batch_min": 10425.90880040536, + "qps_batch_runs": [ + 10425.90880040536, + 10778.254655667099, + 10648.776721774086 + ], + "recall": 0.9999, + "threads": 32 + }, + "min_recall_vs_f32_scan": 1.0, + "one_thread": { + "max_ms": 8.4794, + "mean_ms": 2.198296299999999, + "median_ms": 2.0639, + "nq": 1000, + "p95_ms": 2.9556999999999998, + "p99_ms": 3.7062000000000004, + "qps_1t": 478.0732653015953, + "qps_1t_max": 482.6410426745423, + "qps_1t_min": 411.3150973494401, + "qps_1t_runs": [ + 482.6410426745423, + 411.3150973494401, + 478.0732653015953 + ], + "recall": 0.9999 + }, + "recall_vs_f32_scan": 1.0, + "report_f32_ms": 4.4929, + "report_int8_ms": 0.89605, + "report_queries": 1000, + "rerank_factor": 4, + "scores_exact": true + }, + { + "applies": true, + "auto_threads": { + "max_ms": 4.865600000000001, + "mean_ms": 0.37606406666666703, + "median_ms": 0.328, + "nq": 1000, + "p95_ms": 0.5201, + "p99_ms": 1.6524, + "qps_1t": 2673.382202760001, + "qps_1t_max": 2825.375944523176, + "qps_1t_min": 2498.756868457942, + "qps_1t_runs": [ + 2498.756868457942, + 2825.375944523176, + 2673.382202760001 + ], + "recall": 0.9999 + }, + "batch": { + "nq": 1000, + "qps_batch": 9299.877334617957, + "qps_batch_max": 9490.907710413425, + "qps_batch_min": 9217.575335243215, + "qps_batch_runs": [ + 9490.907710413425, + 9299.877334617957, + 9217.575335243215 + ], + "recall": 0.9999, + "threads": 32 + }, + "min_recall_vs_f32_scan": 1.0, + "one_thread": { + "max_ms": 10.0705, + "mean_ms": 2.2129385333333316, + "median_ms": 2.146, + "nq": 1000, + "p95_ms": 2.7171000000000003, + "p99_ms": 3.0612, + "qps_1t": 453.6207280721552, + "qps_1t_max": 457.55657195138866, + "qps_1t_min": 444.6798876222775, + "qps_1t_runs": [ + 457.55657195138866, + 444.6798876222775, + 453.6207280721552 + ], + "recall": 0.9999 + }, + "recall_vs_f32_scan": 1.0, + "report_f32_ms": 4.4929, + "report_int8_ms": 0.8942, + "report_queries": 1000, + "rerank_factor": 8, + "scores_exact": true + } + ], + "f32": { + "auto_threads": { + "max_ms": 7.3831, + "mean_ms": 4.444786735277305, + "median_ms": 4.3, + "nq": 583, + "p95_ms": 5.4828, + "p99_ms": 6.3041, + "qps_1t": 222.58095637806488, + "qps_1t_max": 232.16957570233475, + "qps_1t_min": 220.53555186298237, + "qps_1t_runs": [ + 220.53555186298237, + 222.58095637806488, + 232.16957570233475 + ], + "recall": 1.0 + }, + "batch": { + "nq": 1000, + "qps_batch": 5328.201209927931, + "qps_batch_max": 5366.589012123661, + "qps_batch_min": 5313.112336196747, + "qps_batch_runs": [ + 5313.112336196747, + 5328.201209927931, + 5366.589012123661 + ], + "recall": 0.9999, + "threads": 32 + }, + "mem": { + "peak_mb": 196.28125, + "rss_mb": 172.12109375 + }, + "one_thread": { + "max_ms": 17.0923, + "mean_ms": 8.479580358598213, + "median_ms": 8.4966, + "nq": 409, + "p95_ms": 10.5745, + "p99_ms": 12.572099999999999, + "qps_1t": 115.56270518560937, + "qps_1t_max": 130.3905784422547, + "qps_1t_min": 109.69525558364275, + "qps_1t_runs": [ + 130.3905784422547, + 115.56270518560937, + 109.69525558364275 + ], + "recall": 1.0 + } + }, + "mem_after_int8": { + "peak_mb": 210.0625, + "rss_mb": 210.0625 + }, + "quant_build_s": 0.1934571, + "quant_bytes_mb": 37.25, + "rss_delta_mb": 37.94140625 + }, + "k": 10, + "knob_name": "", + "mem_after_first_exact_query": { + "peak_mb": 239.15625, + "rss_mb": 183.12890625 + }, + "mem_after_ingest": { + "peak_mb": 196.28125, + "rss_mb": 168.98828125 + }, + "mem_after_sweep": { + "peak_mb": 239.15625, + "rss_mb": 183.2578125 + }, + "mem_baseline": { + "peak_mb": 7.87109375, + "rss_mb": 7.87109375 + }, + "mem_end": { + "peak_mb": 239.15625, + "rss_mb": 180.484375 + }, + "mode": "exact", + "n": 100000, + "queries": 1000, + "raw_vectors_mb": 146.484375, + "ref_knob": null, + "started": "SystemTime { intervals: 134393524789088801 }", + "status": "ok", + "sweep": [ + { + "auto_threads": { + "median_ms": 3.5723000000000003, + "p99_ms": 5.6873000000000005, + "qps": 271.6170974643314, + "runs": [ + 288.8401001550736, + 271.6170974643314, + 256.2106963016493 + ] + }, + "batch": { + "nq": 1000, + "qps_batch": 12525.50505967777, + "qps_batch_max": 13315.366465515863, + "qps_batch_min": 12087.484376926443, + "qps_batch_runs": [ + 12525.50505967777, + 12087.484376926443, + 13315.366465515863 + ], + "recall": 0.9999, + "threads": 32 + }, + "knob": null, + "max_ms": 11.2932, + "mean_ms": 6.581106541405708, + "median_ms": 6.4121999999999995, + "nq": 479, + "p95_ms": 7.585500000000001, + "p99_ms": 8.7525, + "qps_1t": 151.51467704330253, + "qps_1t_max": 155.89518284861393, + "qps_1t_min": 148.61639839545174, + "qps_1t_runs": [ + 148.61639839545174, + 151.51467704330253, + 155.89518284861393 + ], + "recall": 1.0 + } + ], + "system": "rust_exact", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_r2.load.json b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_r2.load.json new file mode 100644 index 0000000..1fc2ee8 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_r2.load.json @@ -0,0 +1 @@ +{"total_pct": 12.775, "pin_cpu_pct": 68.9, "max_cpu_pct": 85.5, "noisy": true, "seconds": 104.25835061073303, "after": 25.6} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_r2.txt b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_r2.txt new file mode 100644 index 0000000..56935ed --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_r2.txt @@ -0,0 +1,18 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/text384 --ds text384 --n 100000 --mode exact --exp core,int8,conc --queries 1000 --budget 3 --label rust_exact --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_exact_text384_100000_r2.json +rust_exact: ds=text384 n=100000 d=384 mode=exact int8=0 queries=1000 cpus=32 exps=["core", "int8", "conc"] + ingest 0.11s rss 169 MB + int8 study: f32 1T 115.6 qps, auto 222.6 qps, batch 5328 qps + rerank x1: recall vs f32 0.9904 (worst 0.80) exact_scores=true applies=true int8 1.171 ms f32 4.493 ms + 1T 359.5 qps (recall vs float64 truth 0.9905), auto 1272.1 qps, batch 4698 qps + rerank x2: recall vs f32 1.0000 (worst 1.00) exact_scores=true applies=true int8 0.844 ms f32 4.493 ms + 1T 369.8 qps (recall vs float64 truth 0.9999), auto 1264.5 qps, batch 12409 qps + rerank x4: recall vs f32 1.0000 (worst 1.00) exact_scores=true applies=true int8 0.896 ms f32 4.493 ms + 1T 478.1 qps (recall vs float64 truth 0.9999), auto 2741.0 qps, batch 10649 qps + rerank x8: recall vs f32 1.0000 (worst 1.00) exact_scores=true applies=true int8 0.894 ms f32 4.493 ms + 1T 453.6 qps (recall vs float64 truth 0.9999), auto 2673.4 qps, batch 9300 qps + exact 1T recall=1.0000 qps=151.5 median=6.412ms | auto 271.6 qps | batch 12526 qps + concurrent clients=1 (1 scan thread each): 146.4 qps + concurrent clients=2 (1 scan thread each): 244.7 qps + concurrent clients=4 (1 scan thread each): 441.1 qps + concurrent clients=8 (1 scan thread each): 662.1 qps + concurrent clients=16 (1 scan thread each): 1034.0 qps diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_r3.json b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_r3.json new file mode 100644 index 0000000..e37fbe7 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_r3.json @@ -0,0 +1,89 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_exact_query_ms": 6.7026, + "ingest_s": 0.09985569999999999 + }, + "d": 384, + "ds": "text384", + "exps": [ + "core" + ], + "first_exact_query_ms": 6.7026, + "int8_rerank": 0, + "k": 10, + "knob_name": "", + "mem_after_first_exact_query": { + "peak_mb": 196.27734375, + "rss_mb": 171.50390625 + }, + "mem_after_ingest": { + "peak_mb": 196.27734375, + "rss_mb": 169.01953125 + }, + "mem_after_sweep": { + "peak_mb": 196.27734375, + "rss_mb": 172.5859375 + }, + "mem_baseline": { + "peak_mb": 7.87109375, + "rss_mb": 7.87109375 + }, + "mem_end": { + "peak_mb": 196.27734375, + "rss_mb": 172.609375 + }, + "mode": "exact", + "n": 100000, + "queries": 1000, + "raw_vectors_mb": 146.484375, + "ref_knob": null, + "started": "SystemTime { intervals: 134393533931543401 }", + "status": "ok", + "sweep": [ + { + "auto_threads": { + "median_ms": 4.3225, + "p99_ms": 5.0444, + "qps": 230.39366398590795, + "runs": [ + 231.30352931685584, + 230.25917363998002, + 230.39366398590795 + ] + }, + "batch": { + "nq": 1000, + "qps_batch": 5218.645594471575, + "qps_batch_max": 5241.002247341764, + "qps_batch_min": 5203.891053418462, + "qps_batch_runs": [ + 5203.891053418462, + 5241.002247341764, + 5218.645594471575 + ], + "recall": 0.9999, + "threads": 32 + }, + "knob": null, + "max_ms": 14.5712, + "mean_ms": 8.14311911976912, + "median_ms": 8.5013, + "nq": 462, + "p95_ms": 9.3288, + "p99_ms": 9.8575, + "qps_1t": 116.81034043564638, + "qps_1t_max": 143.30833368048891, + "qps_1t_min": 112.47948436288065, + "qps_1t_runs": [ + 143.30833368048891, + 116.81034043564638, + 112.47948436288065 + ], + "recall": 1.0 + } + ], + "system": "rust_exact", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_r3.load.json b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_r3.load.json new file mode 100644 index 0000000..7dc2d3f --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_r3.load.json @@ -0,0 +1 @@ +{"total_pct": 16.584375, "pin_cpu_pct": 29.7, "max_cpu_pct": 62.0, "noisy": true, "seconds": 22.490696907043457, "after": 19.9} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_r3.txt b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_r3.txt new file mode 100644 index 0000000..a4f0336 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_exact_text384_100000_r3.txt @@ -0,0 +1,4 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/text384 --ds text384 --n 100000 --mode exact --exp core --queries 1000 --budget 3 --label rust_exact --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_exact_text384_100000_r3.json +rust_exact: ds=text384 n=100000 d=384 mode=exact int8=0 queries=1000 cpus=32 exps=["core"] + ingest 0.10s rss 169 MB + exact 1T recall=1.0000 qps=116.8 median=8.501ms | auto 230.4 qps | batch 5219 qps diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_hnsw_i8x4_text384_100000_r1.json b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_hnsw_i8x4_text384_100000_r1.json new file mode 100644 index 0000000..a8f50a7 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_hnsw_i8x4_text384_100000_r1.json @@ -0,0 +1,198 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_query_ms": 0.4515, + "index_s": 6.8565933999999995, + "ingest_s": 0.0986465, + "stats_build_seconds": 6.8564926, + "total_s": 6.9552399 + }, + "d": 384, + "ds": "text384", + "exps": [ + "core" + ], + "first_exact_query_ms": 2.5442, + "index_stats": { + "ef_search_default": 96, + "graph_nodes": 100000, + "index_bytes": 14458880, + "max_level": 4, + "nlist": null, + "trained_n": 100000 + }, + "int8_rerank": 4, + "k": 10, + "knob_name": "ef_search", + "mem_after_build": { + "peak_mb": 246.66015625, + "rss_mb": 246.66015625 + }, + "mem_after_first_exact_query": { + "peak_mb": 207.7890625, + "rss_mb": 207.7890625 + }, + "mem_after_ingest": { + "peak_mb": 196.2578125, + "rss_mb": 168.9921875 + }, + "mem_after_sweep": { + "peak_mb": 249.19140625, + "rss_mb": 249.19140625 + }, + "mem_baseline": { + "peak_mb": 7.87109375, + "rss_mb": 7.87109375 + }, + "mem_end": { + "peak_mb": 249.21484375, + "rss_mb": 249.21484375 + }, + "memory_stats": { + "index_bytes": 14458880, + "matrix_bytes": 154140672, + "metadata_bytes": 12696122, + "quant_bytes": 39059456, + "total_bytes": 181295674 + }, + "mode": "hnsw", + "n": 100000, + "quant_build_s": 0.1071674, + "quant_bytes_mb": 37.25, + "queries": 1000, + "raw_vectors_mb": 146.484375, + "ref_knob": 128, + "ref_recall_pre": 0.9626, + "rss_delta_int8_mb": 37.02734375, + "started": "SystemTime { intervals: 134393523724343436 }", + "status": "ok", + "sweep": [ + { + "batch": { + "nq": 1000, + "qps_batch": 24497.255082567997, + "qps_batch_max": 25322.16119581374, + "qps_batch_min": 24246.421228226714, + "qps_batch_runs": [ + 24497.255082567997, + 24246.421228226714, + 25322.16119581374 + ], + "recall": 0.9145, + "threads": 32 + }, + "knob": 64, + "max_ms": 1.9908000000000001, + "mean_ms": 0.20320946666666675, + "median_ms": 0.20049999999999998, + "nq": 1000, + "p95_ms": 0.259, + "p99_ms": 0.3026, + "qps_1t": 5016.509332212305, + "qps_1t_max": 5102.295931123086, + "qps_1t_min": 4666.433967159504, + "qps_1t_runs": [ + 5102.295931123086, + 4666.433967159504, + 5016.509332212305 + ], + "recall": 0.9145 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 17919.57337079719, + "qps_batch_max": 18744.212724321365, + "qps_batch_min": 17671.9659001746, + "qps_batch_runs": [ + 18744.212724321365, + 17919.57337079719, + 17671.9659001746 + ], + "recall": 0.9476, + "threads": 32 + }, + "knob": 96, + "max_ms": 2.4105, + "mean_ms": 0.2758743000000001, + "median_ms": 0.2748, + "nq": 1000, + "p95_ms": 0.33149999999999996, + "p99_ms": 0.3653, + "qps_1t": 3584.1780046164204, + "qps_1t_max": 3712.1150815069145, + "qps_1t_min": 3581.2681055485646, + "qps_1t_runs": [ + 3712.1150815069145, + 3584.1780046164204, + 3581.2681055485646 + ], + "recall": 0.9476 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 14797.299196950573, + "qps_batch_max": 15109.140879140472, + "qps_batch_min": 14529.118532907729, + "qps_batch_runs": [ + 14797.299196950573, + 15109.140879140472, + 14529.118532907729 + ], + "recall": 0.9626, + "threads": 32 + }, + "knob": 128, + "max_ms": 2.0664, + "mean_ms": 0.35451056666666586, + "median_ms": 0.35579999999999995, + "nq": 1000, + "p95_ms": 0.4243, + "p99_ms": 0.4652, + "qps_1t": 2794.835591001526, + "qps_1t_max": 2903.817997980108, + "qps_1t_min": 2767.364033178487, + "qps_1t_runs": [ + 2903.817997980108, + 2794.835591001526, + 2767.364033178487 + ], + "recall": 0.9626 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 10341.208163349726, + "qps_batch_max": 10642.736119743553, + "qps_batch_min": 10234.021366589808, + "qps_batch_runs": [ + 10234.021366589808, + 10341.208163349726, + 10642.736119743553 + ], + "recall": 0.9796, + "threads": 32 + }, + "knob": 192, + "max_ms": 1.1415, + "mean_ms": 0.5200692666666658, + "median_ms": 0.5216, + "nq": 1000, + "p95_ms": 0.6486999999999999, + "p99_ms": 0.749, + "qps_1t": 1969.2796316029205, + "qps_1t_max": 1991.3487843413095, + "qps_1t_min": 1817.4029783963483, + "qps_1t_runs": [ + 1991.3487843413095, + 1969.2796316029205, + 1817.4029783963483 + ], + "recall": 0.9796 + } + ], + "system": "rust_hnsw_i8x4", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_hnsw_i8x4_text384_100000_r1.load.json b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_hnsw_i8x4_text384_100000_r1.load.json new file mode 100644 index 0000000..320337a --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_hnsw_i8x4_text384_100000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 27.46875, "pin_cpu_pct": 70.9, "max_cpu_pct": 78.1, "noisy": true, "seconds": 12.441469192504883, "after": 21.1} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_hnsw_i8x4_text384_100000_r1.txt b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_hnsw_i8x4_text384_100000_r1.txt new file mode 100644 index 0000000..f8f84cb --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_hnsw_i8x4_text384_100000_r1.txt @@ -0,0 +1,8 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/text384 --ds text384 --n 100000 --mode hnsw --exp core --queries 1000 --budget 3 --label rust_hnsw_i8x4 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_hnsw_i8x4_text384_100000_r1.json --int8 4 --knobs 64,96,128,192 +rust_hnsw_i8x4: ds=text384 n=100000 d=384 mode=hnsw int8=4 queries=1000 cpus=32 exps=["core"] + ingest 0.10s rss 169 MB + hnsw build 6.86s (stats Some(6.8564926)) rss 247 MB (peak 247) + hnsw+int8x4 knob=64 recall=0.9145 qps1t=5017 median=0.200ms batch=24497 + hnsw+int8x4 knob=96 recall=0.9476 qps1t=3584 median=0.275ms batch=17920 + hnsw+int8x4 knob=128 recall=0.9626 qps1t=2795 median=0.356ms batch=14797 + hnsw+int8x4 knob=192 recall=0.9796 qps1t=1969 median=0.522ms batch=10341 diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_hnsw_text384_1000000_r1.json b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_hnsw_text384_1000000_r1.json new file mode 100644 index 0000000..eb73cf6 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_hnsw_text384_1000000_r1.json @@ -0,0 +1,413 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_query_ms": 1.3404999999999998, + "index_s": 150.983496, + "ingest_s": 1.0892827999999999, + "stats_build_seconds": 150.9818527, + "total_s": 152.0727788 + }, + "concurrent_clients": [ + { + "clients": 1, + "qps": 699.7076854526074 + }, + { + "clients": 2, + "qps": 1350.2509694015534 + }, + { + "clients": 4, + "qps": 2635.449703841756 + }, + { + "clients": 8, + "qps": 4989.753333830788 + }, + { + "clients": 16, + "qps": 8191.153880216507 + } + ], + "d": 384, + "ds": "text384", + "exps": [ + "core", + "conc", + "writes" + ], + "first_exact_query_ms": 63.65410000000001, + "index_stats": { + "ef_search_default": 192, + "graph_nodes": 1000000, + "index_bytes": 146214912, + "max_level": 5, + "nlist": null, + "trained_n": 1000000 + }, + "int8_rerank": 0, + "k": 10, + "knob_name": "ef_search", + "mem_after_build": { + "peak_mb": 1800.05859375, + "rss_mb": 1792.01953125 + }, + "mem_after_first_exact_query": { + "peak_mb": 1656.0859375, + "rss_mb": 1631.1640625 + }, + "mem_after_ingest": { + "peak_mb": 1656.0859375, + "rss_mb": 1628.69921875 + }, + "mem_after_sweep": { + "peak_mb": 1802.4375, + "rss_mb": 1802.42578125 + }, + "mem_baseline": { + "peak_mb": 7.87109375, + "rss_mb": 7.87109375 + }, + "mem_end": { + "peak_mb": 1852.22265625, + "rss_mb": 1810.8984375 + }, + "memory_stats": { + "index_bytes": 146214912, + "matrix_bytes": 1536688128, + "metadata_bytes": 130484346, + "quant_bytes": 0, + "total_bytes": 1813387386 + }, + "mode": "hnsw", + "mutation": { + "after": { + "max_ms": 1.2069999999999999, + "mean_ms": 0.701261466666665, + "median_ms": 0.7142000000000001, + "nq": 1000, + "p95_ms": 0.8331999999999999, + "p99_ms": 0.8661, + "qps_1t": 1430.9164962922111, + "qps_1t_max": 1458.751113574132, + "qps_1t_min": 1390.020790540964, + "qps_1t_runs": [ + 1430.9164962922111, + 1458.751113574132, + 1390.020790540964 + ], + "recall": 0.955 + }, + "delete_10k": { + "lat": { + "max_ms": 0.0263, + "mean_ms": 0.0009471400000000335, + "median_ms": 0.0009, + "p95_ms": 0.0013, + "p99_ms": 0.0015 + }, + "per_s": 997655.5095525515, + "seconds": 0.0100235 + }, + "first_query_after_ms": 1.6148, + "index_stats_after": { + "churn": 20000, + "graph_dead": 10000, + "graph_nodes": 1010000, + "inserts": 20000, + "pending": 0, + "trained_n": 1010000 + }, + "insert_10k": { + "lat": { + "max_ms": 12.4034, + "mean_ms": 1.5381555999999976, + "median_ms": 1.4823, + "p95_ms": 2.1995999999999998, + "p99_ms": 2.7378 + }, + "per_s": 650.0126407958255, + "seconds": 15.3843162 + }, + "mem_after": { + "peak_mb": 1852.22265625, + "rss_mb": 1811.1640625 + }, + "mode": "hnsw", + "qps_after": 1430.9164962922111, + "recall_after": 0.955, + "recall_before": 0.9549, + "ref_knob": 192, + "update_10k": { + "lat": { + "max_ms": 4.9136, + "mean_ms": 1.6130593600000036, + "median_ms": 1.572, + "p95_ms": 2.3217, + "p99_ms": 2.888 + }, + "per_s": 619.8432758508861, + "seconds": 16.1331104 + } + }, + "n": 1000000, + "queries": 1000, + "raw_vectors_mb": 1464.84375, + "ref_knob": 192, + "ref_recall_pre": 0.9549, + "started": "SystemTime { intervals: 134393555852512078 }", + "status": "ok", + "sweep": [ + { + "batch": { + "nq": 1000, + "qps_batch": 19010.46716322007, + "qps_batch_max": 19310.79763249621, + "qps_batch_min": 18228.21728034998, + "qps_batch_runs": [ + 19010.46716322007, + 18228.21728034998, + 19310.79763249621 + ], + "recall": 0.8342, + "threads": 32 + }, + "knob": 48, + "max_ms": 0.5831999999999999, + "mean_ms": 0.21878056666666668, + "median_ms": 0.217, + "nq": 1000, + "p95_ms": 0.2869, + "p99_ms": 0.3341, + "qps_1t": 4632.969228281673, + "qps_1t_max": 4737.573935763237, + "qps_1t_min": 4358.838787911543, + "qps_1t_runs": [ + 4737.573935763237, + 4632.969228281673, + 4358.838787911543 + ], + "recall": 0.8342 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 15302.101743674491, + "qps_batch_max": 16735.561394406974, + "qps_batch_min": 14884.319072171085, + "qps_batch_runs": [ + 15302.101743674491, + 14884.319072171085, + 16735.561394406974 + ], + "recall": 0.8678, + "threads": 32 + }, + "knob": 64, + "max_ms": 0.5077, + "mean_ms": 0.283854533333333, + "median_ms": 0.2825, + "nq": 1000, + "p95_ms": 0.3614, + "p99_ms": 0.4084, + "qps_1t": 3480.4447729984254, + "qps_1t_max": 3630.595610101625, + "qps_1t_min": 3462.5187538671994, + "qps_1t_runs": [ + 3462.5187538671994, + 3480.4447729984254, + 3630.595610101625 + ], + "recall": 0.8678 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 12288.2277549285, + "qps_batch_max": 12929.752653831732, + "qps_batch_min": 12126.591918354081, + "qps_batch_runs": [ + 12929.752653831732, + 12126.591918354081, + 12288.2277549285 + ], + "recall": 0.9062, + "threads": 32 + }, + "knob": 96, + "max_ms": 0.6176, + "mean_ms": 0.38538939999999955, + "median_ms": 0.38880000000000003, + "nq": 1000, + "p95_ms": 0.4729, + "p99_ms": 0.5153, + "qps_1t": 2616.049252882496, + "qps_1t_max": 2652.8604733339757, + "qps_1t_min": 2519.140428979384, + "qps_1t_runs": [ + 2652.8604733339757, + 2616.049252882496, + 2519.140428979384 + ], + "recall": 0.9062 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 8966.9773126507, + "qps_batch_max": 9052.133044630637, + "qps_batch_min": 8761.400772755547, + "qps_batch_runs": [ + 9052.133044630637, + 8761.400772755547, + 8966.9773126507 + ], + "recall": 0.9287, + "threads": 32 + }, + "knob": 128, + "max_ms": 0.7454999999999999, + "mean_ms": 0.491362933333333, + "median_ms": 0.49529999999999996, + "nq": 1000, + "p95_ms": 0.594, + "p99_ms": 0.6524, + "qps_1t": 2034.9916819714977, + "qps_1t_max": 2079.6238043722856, + "qps_1t_min": 1992.706296413867, + "qps_1t_runs": [ + 1992.706296413867, + 2034.9916819714977, + 2079.6238043722856 + ], + "recall": 0.9287 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 6683.973702573864, + "qps_batch_max": 6725.496829264519, + "qps_batch_min": 6481.410666068648, + "qps_batch_runs": [ + 6481.410666068648, + 6683.973702573864, + 6725.496829264519 + ], + "recall": 0.9549, + "threads": 32 + }, + "knob": 192, + "max_ms": 2.7387, + "mean_ms": 0.6960020333333323, + "median_ms": 0.7058, + "nq": 1000, + "p95_ms": 0.8347, + "p99_ms": 0.8956, + "qps_1t": 1434.282536735202, + "qps_1t_max": 1449.689679427224, + "qps_1t_min": 1426.5526527745171, + "qps_1t_runs": [ + 1434.282536735202, + 1449.689679427224, + 1426.5526527745171 + ], + "recall": 0.9549 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 4956.971013120607, + "qps_batch_max": 5049.653240311988, + "qps_batch_min": 4932.737195600787, + "qps_batch_runs": [ + 5049.653240311988, + 4956.971013120607, + 4932.737195600787 + ], + "recall": 0.9685, + "threads": 32 + }, + "knob": 256, + "max_ms": 2.0985, + "mean_ms": 0.9015659000000011, + "median_ms": 0.9112, + "nq": 1000, + "p95_ms": 1.0843, + "p99_ms": 1.1596, + "qps_1t": 1112.3865240601995, + "qps_1t_max": 1125.1112453743858, + "qps_1t_min": 1090.5974576428316, + "qps_1t_runs": [ + 1090.5974576428316, + 1112.3865240601995, + 1125.1112453743858 + ], + "recall": 0.9685 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 3740.087832222652, + "qps_batch_max": 3865.7651702289695, + "qps_batch_min": 3406.9135815928544, + "qps_batch_runs": [ + 3865.7651702289695, + 3740.087832222652, + 3406.9135815928544 + ], + "recall": 0.9825, + "threads": 32 + }, + "knob": 400, + "max_ms": 10.0305, + "mean_ms": 1.3486220999999996, + "median_ms": 1.3767, + "nq": 1000, + "p95_ms": 1.5897000000000001, + "p99_ms": 1.6605999999999999, + "qps_1t": 742.224144417816, + "qps_1t_max": 745.9139022910971, + "qps_1t_min": 736.4165572658511, + "qps_1t_runs": [ + 742.224144417816, + 736.4165572658511, + 745.9139022910971 + ], + "recall": 0.9825 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 2276.3337096112955, + "qps_batch_max": 2321.6329995057245, + "qps_batch_min": 2268.860927446819, + "qps_batch_runs": [ + 2321.6329995057245, + 2276.3337096112955, + 2268.860927446819 + ], + "recall": 0.9901, + "threads": 32 + }, + "knob": 640, + "max_ms": 3.8981, + "mean_ms": 2.0449293999999996, + "median_ms": 2.0860000000000003, + "nq": 1000, + "p95_ms": 2.4349, + "p99_ms": 2.5506, + "qps_1t": 488.84760190675996, + "qps_1t_max": 491.1771325216529, + "qps_1t_min": 487.03619239000994, + "qps_1t_runs": [ + 491.1771325216529, + 488.84760190675996, + 487.03619239000994 + ], + "recall": 0.9901 + } + ], + "system": "rust_hnsw", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_hnsw_text384_1000000_r1.load.json b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_hnsw_text384_1000000_r1.load.json new file mode 100644 index 0000000..c747231 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_hnsw_text384_1000000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 17.1375, "pin_cpu_pct": 62.2, "max_cpu_pct": 62.2, "noisy": true, "seconds": 227.72032141685486, "after": 37.9} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_hnsw_text384_1000000_r1.txt b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_hnsw_text384_1000000_r1.txt new file mode 100644 index 0000000..bd47c0c --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_hnsw_text384_1000000_r1.txt @@ -0,0 +1,18 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/text384 --ds text384 --n 1000000 --mode hnsw --exp core,conc,writes --queries 1000 --budget 3 --label rust_hnsw --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_hnsw_text384_1000000_r1.json --knobs 48,64,96,128,192,256,400,640 +rust_hnsw: ds=text384 n=1000000 d=384 mode=hnsw int8=0 queries=1000 cpus=32 exps=["core", "conc", "writes"] + ingest 1.09s rss 1629 MB + hnsw build 150.98s (stats Some(150.9818527)) rss 1792 MB (peak 1800) + hnsw knob=48 recall=0.8342 qps1t=4633 median=0.217ms batch=19010 + hnsw knob=64 recall=0.8678 qps1t=3480 median=0.282ms batch=15302 + hnsw knob=96 recall=0.9062 qps1t=2616 median=0.389ms batch=12288 + hnsw knob=128 recall=0.9287 qps1t=2035 median=0.495ms batch=8967 + hnsw knob=192 recall=0.9549 qps1t=1434 median=0.706ms batch=6684 + hnsw knob=256 recall=0.9685 qps1t=1112 median=0.911ms batch=4957 + hnsw knob=400 recall=0.9825 qps1t=742 median=1.377ms batch=3740 + hnsw knob=640 recall=0.9901 qps1t=489 median=2.086ms batch=2276 + concurrent clients=1 (1 scan thread each): 699.7 qps + concurrent clients=2 (1 scan thread each): 1350.3 qps + concurrent clients=4 (1 scan thread each): 2635.4 qps + concurrent clients=8 (1 scan thread each): 4989.8 qps + concurrent clients=16 (1 scan thread each): 8191.2 qps + writes: ins 650/s upd 620/s del 997656/s first-query 1.6 ms recall_after=0.955 diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_i8x4_text384_1000000_r1.json b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_i8x4_text384_1000000_r1.json new file mode 100644 index 0000000..3f448c7 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_i8x4_text384_1000000_r1.json @@ -0,0 +1,198 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_query_ms": 3.7075, + "index_s": 5.9247736, + "ingest_s": 1.0833932999999998, + "stats_build_seconds": 5.924005, + "total_s": 7.0081669 + }, + "d": 384, + "ds": "text384", + "exps": [ + "core" + ], + "first_exact_query_ms": 25.9257, + "index_stats": { + "ef_search_default": null, + "graph_nodes": 0, + "index_bytes": 18240000, + "max_level": null, + "nlist": 4000, + "trained_n": 1000000 + }, + "int8_rerank": 4, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 2262.50390625, + "rss_mb": 2018.90625 + }, + "mem_after_first_exact_query": { + "peak_mb": 2000.328125, + "rss_mb": 2000.328125 + }, + "mem_after_ingest": { + "peak_mb": 1657.01171875, + "rss_mb": 1630.015625 + }, + "mem_after_sweep": { + "peak_mb": 2262.50390625, + "rss_mb": 2062.19140625 + }, + "mem_baseline": { + "peak_mb": 7.875, + "rss_mb": 7.875 + }, + "mem_end": { + "peak_mb": 2262.50390625, + "rss_mb": 2062.21484375 + }, + "memory_stats": { + "index_bytes": 18240000, + "matrix_bytes": 1536688128, + "metadata_bytes": 130484346, + "quant_bytes": 388366336, + "total_bytes": 1685412474 + }, + "mode": "ivf", + "n": 1000000, + "quant_build_s": 1.0857538, + "quant_bytes_mb": 370.375, + "queries": 1000, + "raw_vectors_mb": 1464.84375, + "ref_knob": 256, + "ref_recall_pre": 0.955281207133059, + "rss_delta_int8_mb": 367.8359375, + "started": "SystemTime { intervals: 134393572612149393 }", + "status": "ok", + "sweep": [ + { + "batch": { + "nq": 1000, + "qps_batch": 17637.15089464448, + "qps_batch_max": 18322.837401308618, + "qps_batch_min": 17208.682813000127, + "qps_batch_runs": [ + 17637.15089464448, + 17208.682813000127, + 18322.837401308618 + ], + "recall": 0.8579, + "threads": 32 + }, + "knob": 64, + "max_ms": 7.759, + "mean_ms": 1.2672843666666682, + "median_ms": 1.252, + "nq": 1000, + "p95_ms": 1.5007, + "p99_ms": 1.6334, + "qps_1t": 791.3713928104859, + "qps_1t_max": 805.7547000678858, + "qps_1t_min": 770.9200329275367, + "qps_1t_runs": [ + 805.7547000678858, + 791.3713928104859, + 770.9200329275367 + ], + "recall": 0.8579 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 13740.567100685379, + "qps_batch_max": 14288.93950348793, + "qps_batch_min": 13680.18999047859, + "qps_batch_runs": [ + 13740.567100685379, + 14288.93950348793, + 13680.18999047859 + ], + "recall": 0.9151, + "threads": 32 + }, + "knob": 128, + "max_ms": 4.0306999999999995, + "mean_ms": 2.295666066666669, + "median_ms": 2.2701000000000002, + "nq": 1000, + "p95_ms": 2.7479, + "p99_ms": 3.0989, + "qps_1t": 432.65406356916986, + "qps_1t_max": 443.09259846603106, + "qps_1t_min": 431.25416214173197, + "qps_1t_runs": [ + 431.25416214173197, + 443.09259846603106, + 432.65406356916986 + ], + "recall": 0.9151 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 9210.579640197917, + "qps_batch_max": 9235.56319849719, + "qps_batch_min": 8896.084833064968, + "qps_batch_runs": [ + 9210.579640197917, + 9235.56319849719, + 8896.084833064968 + ], + "recall": 0.9541, + "threads": 32 + }, + "knob": 256, + "max_ms": 6.0836, + "mean_ms": 4.19785272062185, + "median_ms": 4.1711, + "nq": 729, + "p95_ms": 4.8072, + "p99_ms": 5.2318, + "qps_1t": 238.12265276474616, + "qps_1t_max": 238.45585799361606, + "qps_1t_min": 238.07293771032224, + "qps_1t_runs": [ + 238.07293771032224, + 238.45585799361606, + 238.12265276474616 + ], + "recall": 0.955281207133059 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 6032.603810554523, + "qps_batch_max": 6041.05500984692, + "qps_batch_min": 5849.516508213013, + "qps_batch_runs": [ + 5849.516508213013, + 6032.603810554523, + 6041.05500984692 + ], + "recall": 0.9799, + "threads": 32 + }, + "knob": 512, + "max_ms": 13.9797, + "mean_ms": 7.967489282470487, + "median_ms": 7.9363, + "nq": 367, + "p95_ms": 8.922899999999998, + "p99_ms": 9.712200000000001, + "qps_1t": 125.6000267490411, + "qps_1t_max": 126.0548766681836, + "qps_1t_min": 124.88084427889557, + "qps_1t_runs": [ + 124.88084427889557, + 126.0548766681836, + 125.6000267490411 + ], + "recall": 0.9771117166212534 + } + ], + "system": "rust_ivf_i8x4", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_i8x4_text384_1000000_r1.load.json b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_i8x4_text384_1000000_r1.load.json new file mode 100644 index 0000000..287f21c --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_i8x4_text384_1000000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 10.865625, "pin_cpu_pct": 73.4, "max_cpu_pct": 73.4, "noisy": true, "seconds": 40.73753762245178, "after": 22.5} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_i8x4_text384_1000000_r1.txt b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_i8x4_text384_1000000_r1.txt new file mode 100644 index 0000000..62897d6 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_i8x4_text384_1000000_r1.txt @@ -0,0 +1,8 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/text384 --ds text384 --n 1000000 --mode ivf --exp core --queries 1000 --budget 3 --label rust_ivf_i8x4 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_ivf_i8x4_text384_1000000_r1.json --int8 4 --knobs 64,128,256,512 +rust_ivf_i8x4: ds=text384 n=1000000 d=384 mode=ivf int8=4 queries=1000 cpus=32 exps=["core"] + ingest 1.08s rss 1630 MB + ivf build 5.92s (stats Some(5.924005)) rss 2019 MB (peak 2263) + ivf+int8x4 knob=64 recall=0.8579 qps1t=791 median=1.252ms batch=17637 + ivf+int8x4 knob=128 recall=0.9151 qps1t=433 median=2.270ms batch=13741 + ivf+int8x4 knob=256 recall=0.9553 qps1t=238 median=4.171ms batch=9211 + ivf+int8x4 knob=512 recall=0.9771 qps1t=126 median=7.936ms batch=6033 diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_text384_1000000_r1.json b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_text384_1000000_r1.json new file mode 100644 index 0000000..948c3f1 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_text384_1000000_r1.json @@ -0,0 +1,351 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_query_ms": 15.2, + "index_s": 17.7083188, + "ingest_s": 1.0907772000000004, + "stats_build_seconds": 17.7069127, + "total_s": 18.799096000000002 + }, + "concurrent_clients": [ + { + "clients": 1, + "qps": 47.328694040149095 + }, + { + "clients": 2, + "qps": 92.25104802002207 + }, + { + "clients": 4, + "qps": 173.48220534419877 + }, + { + "clients": 8, + "qps": 308.809926396877 + }, + { + "clients": 16, + "qps": 375.49365547590077 + } + ], + "d": 384, + "ds": "text384", + "exps": [ + "core", + "conc", + "writes" + ], + "first_exact_query_ms": 64.29129999999999, + "index_stats": { + "ef_search_default": null, + "graph_nodes": 0, + "index_bytes": 18240000, + "max_level": null, + "nlist": 4000, + "trained_n": 1000000 + }, + "int8_rerank": 0, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 1894.3046875, + "rss_mb": 1650.96484375 + }, + "mem_after_first_exact_query": { + "peak_mb": 1656.61328125, + "rss_mb": 1631.39453125 + }, + "mem_after_ingest": { + "peak_mb": 1656.61328125, + "rss_mb": 1628.94140625 + }, + "mem_after_sweep": { + "peak_mb": 1894.3046875, + "rss_mb": 1655.640625 + }, + "mem_baseline": { + "peak_mb": 7.87109375, + "rss_mb": 7.87109375 + }, + "mem_end": { + "peak_mb": 1894.3046875, + "rss_mb": 1671.11328125 + }, + "memory_stats": { + "index_bytes": 18240000, + "matrix_bytes": 1536688128, + "metadata_bytes": 130484346, + "quant_bytes": 0, + "total_bytes": 1685412474 + }, + "mode": "ivf", + "mutation": { + "after": { + "max_ms": 13.3013, + "mean_ms": 10.499958153477216, + "median_ms": 10.5051, + "nq": 278, + "p95_ms": 11.8306, + "p99_ms": 12.5253, + "qps_1t": 96.12598783715804, + "qps_1t_max": 96.6268236270936, + "qps_1t_min": 93.04258040142383, + "qps_1t_runs": [ + 96.12598783715804, + 93.04258040142383, + 96.6268236270936 + ], + "recall": 0.9550359712230215 + }, + "delete_10k": { + "lat": { + "max_ms": 0.0222, + "mean_ms": 0.001170110000000077, + "median_ms": 0.0011, + "p95_ms": 0.0015, + "p99_ms": 0.0018 + }, + "per_s": 820876.5319608278, + "seconds": 0.0121821 + }, + "first_query_after_ms": 17.2975, + "index_stats_after": { + "churn": 30000, + "graph_dead": 0, + "graph_nodes": 0, + "inserts": 0, + "pending": 32, + "trained_n": 1000000 + }, + "insert_10k": { + "lat": { + "max_ms": 3.6424, + "mean_ms": 0.01233848999999994, + "median_ms": 0.0012, + "p95_ms": 0.0027, + "p99_ms": 0.0081 + }, + "per_s": 80587.83993965582, + "seconds": 0.1240882 + }, + "mem_after": { + "peak_mb": 1894.3046875, + "rss_mb": 1675.859375 + }, + "mode": "ivf", + "qps_after": 96.12598783715804, + "recall_after": 0.9550359712230215, + "recall_before": 0.9545454545454546, + "ref_knob": 256, + "update_10k": { + "lat": { + "max_ms": 4.2151000000000005, + "mean_ms": 0.014434570000000028, + "median_ms": 0.0021, + "p95_ms": 0.0033, + "p99_ms": 0.0078 + }, + "per_s": 68979.97925082225, + "seconds": 0.1449696 + } + }, + "n": 1000000, + "queries": 1000, + "raw_vectors_mb": 1464.84375, + "ref_knob": 256, + "ref_recall_pre": 0.9545454545454546, + "started": "SystemTime { intervals: 134393563309873172 }", + "status": "ok", + "sweep": [ + { + "batch": { + "nq": 1000, + "qps_batch": 11292.39163820984, + "qps_batch_max": 11963.210734349728, + "qps_batch_min": 10884.093115593421, + "qps_batch_runs": [ + 10884.093115593421, + 11963.210734349728, + 11292.39163820984 + ], + "recall": 0.6991, + "threads": 32 + }, + "knob": 16, + "max_ms": 3.9240000000000004, + "mean_ms": 1.0073266666666645, + "median_ms": 0.9978, + "nq": 1000, + "p95_ms": 1.2643000000000002, + "p99_ms": 1.4178000000000002, + "qps_1t": 987.6285694871758, + "qps_1t_max": 1008.2504122987993, + "qps_1t_min": 982.669155440082, + "qps_1t_runs": [ + 987.6285694871758, + 1008.2504122987993, + 982.669155440082 + ], + "recall": 0.6991 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 9499.23816109948, + "qps_batch_max": 9776.623701664374, + "qps_batch_min": 9447.465008951473, + "qps_batch_runs": [ + 9447.465008951473, + 9499.23816109948, + 9776.623701664374 + ], + "recall": 0.7841, + "threads": 32 + }, + "knob": 32, + "max_ms": 2.7169999999999996, + "mean_ms": 1.7304271000000018, + "median_ms": 1.7244000000000002, + "nq": 1000, + "p95_ms": 2.1395, + "p99_ms": 2.2964, + "qps_1t": 578.6619784244729, + "qps_1t_max": 580.0071422079494, + "qps_1t_min": 575.029901554882, + "qps_1t_runs": [ + 578.6619784244729, + 575.029901554882, + 580.0071422079494 + ], + "recall": 0.7841 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 8327.018677502894, + "qps_batch_max": 8534.309203143015, + "qps_batch_min": 7951.944807141484, + "qps_batch_runs": [ + 8534.309203143015, + 7951.944807141484, + 8327.018677502894 + ], + "recall": 0.8579, + "threads": 32 + }, + "knob": 64, + "max_ms": 5.6635, + "mean_ms": 3.123425456389458, + "median_ms": 3.1297, + "nq": 986, + "p95_ms": 3.735, + "p99_ms": 3.9515000000000002, + "qps_1t": 318.08221586523746, + "qps_1t_max": 324.4541059667109, + "qps_1t_min": 318.03226150225294, + "qps_1t_runs": [ + 318.03226150225294, + 318.08221586523746, + 324.4541059667109 + ], + "recall": 0.8580121703853956 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 5281.978417836185, + "qps_batch_max": 5681.5018254665365, + "qps_batch_min": 5025.638293755594, + "qps_batch_runs": [ + 5025.638293755594, + 5681.5018254665365, + 5281.978417836185 + ], + "recall": 0.9151, + "threads": 32 + }, + "knob": 128, + "max_ms": 7.3998, + "mean_ms": 5.524865714285716, + "median_ms": 5.5608, + "nq": 525, + "p95_ms": 6.3819, + "p99_ms": 6.6880999999999995, + "qps_1t": 180.47457594490479, + "qps_1t_max": 184.54126311007485, + "qps_1t_min": 178.10047397455088, + "qps_1t_runs": [ + 180.47457594490479, + 178.10047397455088, + 184.54126311007485 + ], + "recall": 0.9158095238095239 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 4523.746729896583, + "qps_batch_max": 4826.427198654778, + "qps_batch_min": 4445.902700530219, + "qps_batch_runs": [ + 4826.427198654778, + 4445.902700530219, + 4523.746729896583 + ], + "recall": 0.9541, + "threads": 32 + }, + "knob": 256, + "max_ms": 19.9664, + "mean_ms": 9.988832323232323, + "median_ms": 9.9881, + "nq": 297, + "p95_ms": 11.458, + "p99_ms": 12.0992, + "qps_1t": 100.77523299013315, + "qps_1t_max": 102.2893739525241, + "qps_1t_min": 97.39718325378406, + "qps_1t_runs": [ + 102.2893739525241, + 100.77523299013315, + 97.39718325378406 + ], + "recall": 0.9545454545454546 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 2847.14693100338, + "qps_batch_max": 2878.1914826255093, + "qps_batch_min": 2797.404679834237, + "qps_batch_runs": [ + 2847.14693100338, + 2878.1914826255093, + 2797.404679834237 + ], + "recall": 0.9799, + "threads": 32 + }, + "knob": 512, + "max_ms": 24.125400000000003, + "mean_ms": 20.05512414529914, + "median_ms": 20.128, + "nq": 156, + "p95_ms": 22.419, + "p99_ms": 22.9412, + "qps_1t": 49.85415104197092, + "qps_1t_max": 49.901465398115974, + "qps_1t_min": 49.83213916148738, + "qps_1t_runs": [ + 49.85415104197092, + 49.83213916148738, + 49.901465398115974 + ], + "recall": 0.9782051282051282 + } + ], + "system": "rust_ivf", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_text384_1000000_r1.load.json b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_text384_1000000_r1.load.json new file mode 100644 index 0000000..dc6499e --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_text384_1000000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 16.9375, "pin_cpu_pct": 61.2, "max_cpu_pct": 69.5, "noisy": true, "seconds": 98.63471007347107, "after": 24.2} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_text384_1000000_r1.txt b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_text384_1000000_r1.txt new file mode 100644 index 0000000..3689a84 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_text384_1000000_r1.txt @@ -0,0 +1,16 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/text384 --ds text384 --n 1000000 --mode ivf --exp core,conc,writes --queries 1000 --budget 3 --label rust_ivf --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_ivf_text384_1000000_r1.json +rust_ivf: ds=text384 n=1000000 d=384 mode=ivf int8=0 queries=1000 cpus=32 exps=["core", "conc", "writes"] + ingest 1.09s rss 1629 MB + ivf build 17.71s (stats Some(17.7069127)) rss 1651 MB (peak 1894) + ivf knob=16 recall=0.6991 qps1t=988 median=0.998ms batch=11292 + ivf knob=32 recall=0.7841 qps1t=579 median=1.724ms batch=9499 + ivf knob=64 recall=0.8580 qps1t=318 median=3.130ms batch=8327 + ivf knob=128 recall=0.9158 qps1t=180 median=5.561ms batch=5282 + ivf knob=256 recall=0.9545 qps1t=101 median=9.988ms batch=4524 + ivf knob=512 recall=0.9782 qps1t=50 median=20.128ms batch=2847 + concurrent clients=1 (1 scan thread each): 47.3 qps + concurrent clients=2 (1 scan thread each): 92.3 qps + concurrent clients=4 (1 scan thread each): 173.5 qps + concurrent clients=8 (1 scan thread each): 308.8 qps + concurrent clients=16 (1 scan thread each): 375.5 qps + writes: ins 80588/s upd 68980/s del 820877/s first-query 17.3 ms recall_after=0.9550359712230215 diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_text384_100000_r2.json b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_text384_100000_r2.json new file mode 100644 index 0000000..7d3acb4 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_text384_100000_r2.json @@ -0,0 +1,257 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_query_ms": 0.6953, + "index_s": 0.4904334, + "ingest_s": 0.1029835, + "stats_build_seconds": 0.4898997, + "total_s": 0.5934169 + }, + "d": 384, + "ds": "text384", + "exps": [ + "core" + ], + "first_exact_query_ms": 6.3758, + "index_stats": { + "ef_search_default": null, + "graph_nodes": 0, + "index_bytes": 3173400, + "max_level": null, + "nlist": 1265, + "trained_n": 100000 + }, + "int8_rerank": 0, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 253.5390625, + "rss_mb": 174.16796875 + }, + "mem_after_first_exact_query": { + "peak_mb": 196.43359375, + "rss_mb": 171.57421875 + }, + "mem_after_ingest": { + "peak_mb": 196.43359375, + "rss_mb": 169.08203125 + }, + "mem_after_sweep": { + "peak_mb": 253.5390625, + "rss_mb": 182.4140625 + }, + "mem_baseline": { + "peak_mb": 7.85546875, + "rss_mb": 7.85546875 + }, + "mem_end": { + "peak_mb": 253.5390625, + "rss_mb": 181.25 + }, + "memory_stats": { + "index_bytes": 3173400, + "matrix_bytes": 154140672, + "metadata_bytes": 12696122, + "quant_bytes": 0, + "total_bytes": 170010194 + }, + "mode": "ivf", + "n": 100000, + "queries": 1000, + "raw_vectors_mb": 146.484375, + "ref_knob": 128, + "ref_recall_pre": 0.9559, + "started": "SystemTime { intervals: 134393529522710927 }", + "status": "ok", + "sweep": [ + { + "batch": { + "nq": 1000, + "qps_batch": 73322.38385734397, + "qps_batch_max": 74021.98452940524, + "qps_batch_min": 69267.4276848055, + "qps_batch_runs": [ + 69267.4276848055, + 73322.38385734397, + 74021.98452940524 + ], + "recall": 0.7503, + "threads": 32 + }, + "knob": 16, + "max_ms": 0.6482, + "mean_ms": 0.16750586666666645, + "median_ms": 0.16369999999999998, + "nq": 1000, + "p95_ms": 0.21139999999999998, + "p99_ms": 0.2439, + "qps_1t": 5995.286505749187, + "qps_1t_max": 6018.226801691606, + "qps_1t_min": 5897.686927187143, + "qps_1t_runs": [ + 5897.686927187143, + 5995.286505749187, + 6018.226801691606 + ], + "recall": 0.7503 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 65185.223813465964, + "qps_batch_max": 74821.17738604735, + "qps_batch_min": 61801.64145159696, + "qps_batch_runs": [ + 61801.64145159696, + 65185.223813465964, + 74821.17738604735 + ], + "recall": 0.8401, + "threads": 32 + }, + "knob": 32, + "max_ms": 1.5475, + "mean_ms": 0.335122166666666, + "median_ms": 0.31129999999999997, + "nq": 1000, + "p95_ms": 0.49549999999999994, + "p99_ms": 0.7615999999999999, + "qps_1t": 2870.742796086256, + "qps_1t_max": 3377.772433391174, + "qps_1t_min": 2770.301810530746, + "qps_1t_runs": [ + 3377.772433391174, + 2870.742796086256, + 2770.301810530746 + ], + "recall": 0.8401 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 32849.46077610136, + "qps_batch_max": 35177.328915060825, + "qps_batch_min": 31906.57754096007, + "qps_batch_runs": [ + 32849.46077610136, + 31906.57754096007, + 35177.328915060825 + ], + "recall": 0.9083, + "threads": 32 + }, + "knob": 64, + "max_ms": 2.2275, + "mean_ms": 0.6941408000000002, + "median_ms": 0.7075, + "nq": 1000, + "p95_ms": 0.8979, + "p99_ms": 0.9986, + "qps_1t": 1310.5222090267198, + "qps_1t_max": 1827.9594068710435, + "qps_1t_min": 1294.8173511222244, + "qps_1t_runs": [ + 1827.9594068710435, + 1310.5222090267198, + 1294.8173511222244 + ], + "recall": 0.9083 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 18062.8659988223, + "qps_batch_max": 20150.524417397963, + "qps_batch_min": 16245.264505396677, + "qps_batch_runs": [ + 16245.264505396677, + 20150.524417397963, + 18062.8659988223 + ], + "recall": 0.9559, + "threads": 32 + }, + "knob": 128, + "max_ms": 3.9554000000000005, + "mean_ms": 1.4424943999999993, + "median_ms": 1.4463, + "nq": 1000, + "p95_ms": 1.7003000000000001, + "p99_ms": 1.8860999999999999, + "qps_1t": 689.5719006622039, + "qps_1t_max": 708.4174874273605, + "qps_1t_min": 682.2626914676428, + "qps_1t_runs": [ + 689.5719006622039, + 682.2626914676428, + 708.4174874273605 + ], + "recall": 0.9559 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 15158.750009474219, + "qps_batch_max": 15736.631731344221, + "qps_batch_min": 14159.893517600747, + "qps_batch_runs": [ + 14159.893517600747, + 15158.750009474219, + 15736.631731344221 + ], + "recall": 0.984, + "threads": 32 + }, + "knob": 256, + "max_ms": 4.7972, + "mean_ms": 2.8031301333333323, + "median_ms": 2.8016, + "nq": 1000, + "p95_ms": 3.1988, + "p99_ms": 3.3925, + "qps_1t": 357.63878718535835, + "qps_1t_max": 360.660924127256, + "qps_1t_min": 352.0400739889664, + "qps_1t_runs": [ + 360.660924127256, + 357.63878718535835, + 352.0400739889664 + ], + "recall": 0.984 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 9490.277210997332, + "qps_batch_max": 9740.920731309365, + "qps_batch_min": 8875.878822951958, + "qps_batch_runs": [ + 9740.920731309365, + 8875.878822951958, + 9490.277210997332 + ], + "recall": 0.9953, + "threads": 32 + }, + "knob": 512, + "max_ms": 7.6815, + "mean_ms": 5.452450030581043, + "median_ms": 5.4752, + "nq": 545, + "p95_ms": 5.983300000000001, + "p99_ms": 6.3283000000000005, + "qps_1t": 183.14093286211357, + "qps_1t_max": 184.56133901181713, + "qps_1t_min": 182.52100490238033, + "qps_1t_runs": [ + 184.56133901181713, + 182.52100490238033, + 183.14093286211357 + ], + "recall": 0.995045871559633 + } + ], + "system": "rust_ivf", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_text384_100000_r2.load.json b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_text384_100000_r2.load.json new file mode 100644 index 0000000..7ba81ef --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_text384_100000_r2.load.json @@ -0,0 +1 @@ +{"total_pct": 13.05, "pin_cpu_pct": 52.7, "max_cpu_pct": 52.7, "noisy": true, "seconds": 27.8251953125, "after": 17.9} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_text384_100000_r2.txt b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_text384_100000_r2.txt new file mode 100644 index 0000000..5d2b877 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/rust_ivf_text384_100000_r2.txt @@ -0,0 +1,10 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/text384 --ds text384 --n 100000 --mode ivf --exp core --queries 1000 --budget 3 --label rust_ivf --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_ivf_text384_100000_r2.json +rust_ivf: ds=text384 n=100000 d=384 mode=ivf int8=0 queries=1000 cpus=32 exps=["core"] + ingest 0.10s rss 169 MB + ivf build 0.49s (stats Some(0.4898997)) rss 174 MB (peak 254) + ivf knob=16 recall=0.7503 qps1t=5995 median=0.164ms batch=73322 + ivf knob=32 recall=0.8401 qps1t=2871 median=0.311ms batch=65185 + ivf knob=64 recall=0.9083 qps1t=1311 median=0.708ms batch=32849 + ivf knob=128 recall=0.9559 qps1t=690 median=1.446ms batch=18063 + ivf knob=256 recall=0.9840 qps1t=358 median=2.802ms batch=15159 + ivf knob=512 recall=0.9950 qps1t=183 median=5.475ms batch=9490 diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r1.json b/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r1.json new file mode 100644 index 0000000..52612ed --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r1.json @@ -0,0 +1,369 @@ +{ + "system": "usearch", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 19:22:08", + "mem_baseline": { + "rss_mb": 37.19140625, + "peak_mb": 37.19140625 + }, + "version": "usearch 2.26.0", + "build": { + "ingest_s": 8.92703490005806, + "index_s": 3.700028173625469e-06, + "total_s": 8.927038600086235, + "first_query_ms": 2.99290003022179, + "ready_s": 8.930031500116456 + }, + "mem_after_build": { + "rss_mb": 218.6953125, + "peak_mb": 248.93359375 + }, + "mem_after_first_query": { + "rss_mb": 218.7421875, + "peak_mb": 248.93359375 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.7797, + "qps_1t_runs": [ + 5312.522375924003, + 4951.96101608126, + 5449.130998513941 + ], + "qps_1t": 5312.522375924003, + "qps_1t_min": 4951.96101608126, + "qps_1t_max": 5449.130998513941, + "median_ms": 0.18640002235770226, + "p95_ms": 0.25590002769604325, + "p99_ms": 0.3054909966886022, + "mean_ms": 0.19123006724597266, + "knob": 24, + "batch": { + "recall": 0.7797, + "qps_batch_runs": [ + 34446.05570561102, + 23586.463249491146, + 23388.80377633546 + ], + "qps_batch": 23586.463249491146, + "qps_batch_min": 23388.80377633546, + "qps_batch_max": 34446.05570561102, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8272, + "qps_1t_runs": [ + 4606.403551101928, + 4576.918777181107, + 4136.610739501988 + ], + "qps_1t": 4576.918777181107, + "qps_1t_min": 4136.610739501988, + "qps_1t_max": 4606.403551101928, + "median_ms": 0.22065002121962607, + "p95_ms": 0.30301503720693285, + "p99_ms": 0.3678440069779743, + "mean_ms": 0.22577350004576147, + "knob": 32, + "batch": { + "recall": 0.8272, + "qps_batch_runs": [ + 21355.246671534474, + 23437.5915703307, + 24396.96794559368 + ], + "qps_batch": 23437.5915703307, + "qps_batch_min": 21355.246671534474, + "qps_batch_max": 24396.96794559368, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8829, + "qps_1t_runs": [ + 3202.8369455746897, + 3283.554646327721, + 3192.975452852312 + ], + "qps_1t": 3202.8369455746897, + "qps_1t_min": 3192.975452852312, + "qps_1t_max": 3283.554646327721, + "median_ms": 0.3064999764319509, + "p95_ms": 0.3839500132016835, + "p99_ms": 0.43693500861991114, + "mean_ms": 0.30998623330378905, + "knob": 48, + "batch": { + "recall": 0.8829, + "qps_batch_runs": [ + 21150.01067683567, + 18331.50323570675, + 19270.90458981232 + ], + "qps_batch": 19270.90458981232, + "qps_batch_min": 18331.50323570675, + "qps_batch_max": 21150.01067683567, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9138, + "qps_1t_runs": [ + 2676.8744818005607, + 2539.919921409986, + 2592.9696306550295 + ], + "qps_1t": 2592.9696306550295, + "qps_1t_min": 2539.919921409986, + "qps_1t_max": 2676.8744818005607, + "median_ms": 0.3774499928113073, + "p95_ms": 0.47571501345373685, + "p99_ms": 0.5485270434292027, + "mean_ms": 0.38431379992592457, + "knob": 64, + "batch": { + "recall": 0.9138, + "qps_batch_runs": [ + 16843.23995446471, + 14863.54522140239, + 13960.026861863955 + ], + "qps_batch": 14863.54522140239, + "qps_batch_min": 13960.026861863955, + "qps_batch_max": 16843.23995446471, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9471, + "qps_1t_runs": [ + 1777.0315778948516, + 1745.5038013643652, + 1831.1493545020512 + ], + "qps_1t": 1777.0315778948516, + "qps_1t_min": 1745.5038013643652, + "qps_1t_max": 1831.1493545020512, + "median_ms": 0.5611499946098775, + "p95_ms": 0.6917999737197533, + "p99_ms": 0.779809000669047, + "mean_ms": 0.5605806001306822, + "knob": 96, + "batch": { + "recall": 0.9471, + "qps_batch_runs": [ + 11524.574425905108, + 10927.158471437035, + 10069.438848014226 + ], + "qps_batch": 10927.158471437035, + "qps_batch_min": 10069.438848014226, + "qps_batch_max": 11524.574425905108, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9615, + "qps_1t_runs": [ + 1306.8388338323825, + 1400.0437365584621, + 1427.8864510272933 + ], + "qps_1t": 1400.0437365584621, + "qps_1t_min": 1306.8388338323825, + "qps_1t_max": 1427.8864510272933, + "median_ms": 0.7240999839268625, + "p95_ms": 0.9158100438071415, + "p99_ms": 1.075113979168236, + "mean_ms": 0.726601499656681, + "knob": 128, + "batch": { + "recall": 0.9615, + "qps_batch_runs": [ + 9273.977406352598, + 9914.880749312326, + 9670.983473232389 + ], + "qps_batch": 9670.983473232389, + "qps_batch_min": 9273.977406352598, + "qps_batch_max": 9914.880749312326, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9783, + "qps_1t_runs": [ + 1010.6175478014846, + 961.7412594816236, + 1009.4364141859559 + ], + "qps_1t": 1009.4364141859559, + "qps_1t_min": 961.7412594816236, + "qps_1t_max": 1010.6175478014846, + "median_ms": 0.9997999877668917, + "p95_ms": 1.2454449955839662, + "p99_ms": 1.375410995678976, + "mean_ms": 1.0066421664863203, + "knob": 192, + "batch": { + "recall": 0.9783, + "qps_batch_runs": [ + 6803.128348383307, + 7338.130024686379, + 6138.498020574611 + ], + "qps_batch": 6803.128348383307, + "qps_batch_min": 6138.498020574611, + "qps_batch_max": 7338.130024686379, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9851, + "qps_1t_runs": [ + 786.8071948864834, + 785.6769825703077, + 786.9717146307602 + ], + "qps_1t": 786.8071948864834, + "qps_1t_min": 785.6769825703077, + "qps_1t_max": 786.9717146307602, + "median_ms": 1.2758499942719936, + "p95_ms": 1.5642449783626942, + "p99_ms": 1.6941449779551465, + "mean_ms": 1.2714802667615004, + "knob": 256, + "batch": { + "recall": 0.9851, + "qps_batch_runs": [ + 4862.2240193934, + 4717.8022965068885, + 5061.60991628174 + ], + "qps_batch": 4862.2240193934, + "qps_batch_min": 4717.8022965068885, + "qps_batch_max": 5061.60991628174, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9922, + "qps_1t_runs": [ + 532.414740266374, + 526.0751240918476, + 497.91302242423006 + ], + "qps_1t": 526.0751240918476, + "qps_1t_min": 497.91302242423006, + "qps_1t_max": 532.414740266374, + "median_ms": 1.9222999981138855, + "p95_ms": 2.4358499795198436, + "p99_ms": 2.8765540156746248, + "mean_ms": 1.9291623667425786, + "knob": 400, + "batch": { + "recall": 0.9922, + "qps_batch_runs": [ + 3998.279140587326, + 3840.751681193535, + 3475.538465835501 + ], + "qps_batch": 3840.751681193535, + "qps_batch_min": 3475.538465835501, + "qps_batch_max": 3998.279140587326, + "threads": 32 + } + } + ], + "knob_name": "expansion_search", + "mem_after_sweep": { + "rss_mb": 221.07421875, + "peak_mb": 248.93359375 + }, + "ref_knob": 128, + "ref_recall_pre": 0.9615, + "mutation": { + "ref_knob": 128, + "recall_before": 0.9615, + "insert_10k": { + "seconds": 1.6121046000043862, + "per_s": 6203.071438399712, + "batch": 1000 + }, + "update_10k": { + "seconds": 1.7863703000475653, + "per_s": 5597.943494545186, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.005058200040366501, + "per_s": 1976987.8455173618, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 266.96484375, + "peak_mb": 270.953125 + }, + "first_query_after_ms": 4.967999993823469, + "after": { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9594, + "qps_1t_runs": [ + 1291.5965119928521, + 1300.2147827682775, + 1301.008307980244 + ], + "qps_1t": 1300.2147827682775, + "qps_1t_min": 1291.5965119928521, + "qps_1t_max": 1301.008307980244, + "median_ms": 0.7758499996270984, + "p95_ms": 0.9489200077950953, + "p99_ms": 1.061904995003715, + "mean_ms": 0.7706579663984788 + }, + "recall_after": 0.9594 + }, + "saveload": { + "save_s": 0.5954139000386931, + "disk_mb": 176.69028091430664, + "load_s": 0.2835444000083953, + "first_query_after_load_ms": 1.3526999973692, + "time_to_ready_s": 0.2848971000057645 + }, + "mem_end": { + "rss_mb": 231.10546875, + "peak_mb": 271.03125 + }, + "status": "ok", + "finished": "2026-11-16 19:22:47" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r1.load.json b/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r1.load.json new file mode 100644 index 0000000..29583e8 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 10.99375, "pin_cpu_pct": 27.0, "max_cpu_pct": 63.0, "noisy": true, "seconds": 39.47076725959778, "after": 21.9} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r1.txt b/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r1.txt new file mode 100644 index 0000000..2365425 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r1.txt @@ -0,0 +1,10 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system usearch --ds text384 --n 100000 --exp core,mutation,saveload --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\usearch_text384_100000_r1.json + usearch text384 100000 knob=24 recall=0.7797 qps1t=5313 med=0.186ms batch=23586 + usearch text384 100000 knob=32 recall=0.8272 qps1t=4577 med=0.221ms batch=23438 + usearch text384 100000 knob=48 recall=0.8829 qps1t=3203 med=0.306ms batch=19271 + usearch text384 100000 knob=64 recall=0.9138 qps1t=2593 med=0.377ms batch=14864 + usearch text384 100000 knob=96 recall=0.9471 qps1t=1777 med=0.561ms batch=10927 + usearch text384 100000 knob=128 recall=0.9615 qps1t=1400 med=0.724ms batch=9671 + usearch text384 100000 knob=192 recall=0.9783 qps1t=1009 med=1.000ms batch=6803 + usearch text384 100000 knob=256 recall=0.9851 qps1t=787 med=1.276ms batch=4862 + usearch text384 100000 knob=400 recall=0.9922 qps1t=526 med=1.922ms batch=3841 diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r2.json b/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r2.json new file mode 100644 index 0000000..2503fcf --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r2.json @@ -0,0 +1,369 @@ +{ + "system": "usearch", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 19:41:43", + "mem_baseline": { + "rss_mb": 37.0859375, + "peak_mb": 37.0859375 + }, + "version": "usearch 2.26.0", + "build": { + "ingest_s": 9.261937000090256, + "index_s": 7.099995855242014e-06, + "total_s": 9.261944100086112, + "first_query_ms": 3.8731999811716378, + "ready_s": 9.265817300067283 + }, + "mem_after_build": { + "rss_mb": 219.95703125, + "peak_mb": 250.40234375 + }, + "mem_after_first_query": { + "rss_mb": 220.0, + "peak_mb": 250.40234375 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.7817, + "qps_1t_runs": [ + 5198.458969797607, + 5205.22916116191, + 5086.1233230036305 + ], + "qps_1t": 5198.458969797607, + "qps_1t_min": 5086.1233230036305, + "qps_1t_max": 5205.22916116191, + "median_ms": 0.19029999384656549, + "p95_ms": 0.25070499104913324, + "p99_ms": 0.29663898865692223, + "mean_ms": 0.19369753351202235, + "knob": 24, + "batch": { + "recall": 0.7817, + "qps_batch_runs": [ + 29770.14470230384, + 31065.74132510641, + 30928.15386098158 + ], + "qps_batch": 30928.15386098158, + "qps_batch_min": 29770.14470230384, + "qps_batch_max": 31065.74132510641, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8283, + "qps_1t_runs": [ + 4318.926814596362, + 4335.9944757293515, + 4021.741532663495 + ], + "qps_1t": 4318.926814596362, + "qps_1t_min": 4021.741532663495, + "qps_1t_max": 4335.9944757293515, + "median_ms": 0.23314999998547137, + "p95_ms": 0.3087049670284614, + "p99_ms": 0.3585039940662681, + "mean_ms": 0.23693836707388982, + "knob": 32, + "batch": { + "recall": 0.8283, + "qps_batch_runs": [ + 18079.324852001395, + 19140.734174163666, + 22555.237783065924 + ], + "qps_batch": 19140.734174163666, + "qps_batch_min": 18079.324852001395, + "qps_batch_max": 22555.237783065924, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8832, + "qps_1t_runs": [ + 3046.6580362285754, + 3099.517131101008, + 3247.6579483347946 + ], + "qps_1t": 3099.517131101008, + "qps_1t_min": 3046.6580362285754, + "qps_1t_max": 3247.6579483347946, + "median_ms": 0.3184000088367611, + "p95_ms": 0.40581000503152603, + "p99_ms": 0.457911994308233, + "mean_ms": 0.31959120023141924, + "knob": 48, + "batch": { + "recall": 0.8832, + "qps_batch_runs": [ + 17655.678943918778, + 21423.8275784059, + 20170.60294851634 + ], + "qps_batch": 20170.60294851634, + "qps_batch_min": 17655.678943918778, + "qps_batch_max": 21423.8275784059, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9149, + "qps_1t_runs": [ + 2669.564749579523, + 2538.388687060744, + 2570.55729972696 + ], + "qps_1t": 2570.55729972696, + "qps_1t_min": 2538.388687060744, + "qps_1t_max": 2669.564749579523, + "median_ms": 0.3840999852400273, + "p95_ms": 0.4776999703608453, + "p99_ms": 0.5428090004716066, + "mean_ms": 0.38585476647131145, + "knob": 64, + "batch": { + "recall": 0.9149, + "qps_batch_runs": [ + 15409.032767190844, + 14482.301169734088, + 15581.979138836228 + ], + "qps_batch": 15409.032767190844, + "qps_batch_min": 14482.301169734088, + "qps_batch_max": 15581.979138836228, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9477, + "qps_1t_runs": [ + 1814.8859864867052, + 1836.0125027906106, + 1850.4554433564535 + ], + "qps_1t": 1836.0125027906106, + "qps_1t_min": 1814.8859864867052, + "qps_1t_max": 1850.4554433564535, + "median_ms": 0.5401999806053936, + "p95_ms": 0.6641999934799969, + "p99_ms": 0.7522279658587643, + "mean_ms": 0.5453549666563049, + "knob": 96, + "batch": { + "recall": 0.9477, + "qps_batch_runs": [ + 10126.777123806463, + 10818.178866144059, + 10917.197420512739 + ], + "qps_batch": 10818.178866144059, + "qps_batch_min": 10126.777123806463, + "qps_batch_max": 10917.197420512739, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9624, + "qps_1t_runs": [ + 1379.6793371076844, + 1369.5392021818998, + 1341.3951174324604 + ], + "qps_1t": 1369.5392021818998, + "qps_1t_min": 1341.3951174324604, + "qps_1t_max": 1379.6793371076844, + "median_ms": 0.7454999722540379, + "p95_ms": 0.9003049868624657, + "p99_ms": 0.9761020401492715, + "mean_ms": 0.7334904001909308, + "knob": 128, + "batch": { + "recall": 0.9624, + "qps_batch_runs": [ + 8281.75867843652, + 8442.99375037079, + 8192.77937492279 + ], + "qps_batch": 8281.75867843652, + "qps_batch_min": 8192.77937492279, + "qps_batch_max": 8442.99375037079, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9783, + "qps_1t_runs": [ + 984.9750913738021, + 919.3860269386845, + 956.9323969868656 + ], + "qps_1t": 956.9323969868656, + "qps_1t_min": 919.3860269386845, + "qps_1t_max": 984.9750913738021, + "median_ms": 1.057099987519905, + "p95_ms": 1.2988049944397062, + "p99_ms": 1.4006220130249853, + "mean_ms": 1.0493141329886082, + "knob": 192, + "batch": { + "recall": 0.9783, + "qps_batch_runs": [ + 6097.430846902441, + 5842.111105456666, + 5503.565208979396 + ], + "qps_batch": 5842.111105456666, + "qps_batch_min": 5503.565208979396, + "qps_batch_max": 6097.430846902441, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9852, + "qps_1t_runs": [ + 751.623526244535, + 783.8077239541193, + 752.5311199150176 + ], + "qps_1t": 752.5311199150176, + "qps_1t_min": 751.623526244535, + "qps_1t_max": 783.8077239541193, + "median_ms": 1.3120999792590737, + "p95_ms": 1.647105027223006, + "p99_ms": 1.8516830127919073, + "mean_ms": 1.3117083660326898, + "knob": 256, + "batch": { + "recall": 0.9852, + "qps_batch_runs": [ + 4890.284033722133, + 3844.397251729395, + 3371.472554795659 + ], + "qps_batch": 3844.397251729395, + "qps_batch_min": 3371.472554795659, + "qps_batch_max": 4890.284033722133, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9927, + "qps_1t_runs": [ + 491.1847805432432, + 505.8800718546854, + 512.4078537318383 + ], + "qps_1t": 505.8800718546854, + "qps_1t_min": 491.1847805432432, + "qps_1t_max": 512.4078537318383, + "median_ms": 2.007199975196272, + "p95_ms": 2.4439300381345674, + "p99_ms": 2.639901019865647, + "mean_ms": 1.9880723996126715, + "knob": 400, + "batch": { + "recall": 0.9927, + "qps_batch_runs": [ + 3475.0469041756924, + 3673.3428446468474, + 3622.0702881711127 + ], + "qps_batch": 3622.0702881711127, + "qps_batch_min": 3475.0469041756924, + "qps_batch_max": 3673.3428446468474, + "threads": 32 + } + } + ], + "knob_name": "expansion_search", + "mem_after_sweep": { + "rss_mb": 222.23828125, + "peak_mb": 250.40234375 + }, + "ref_knob": 128, + "ref_recall_pre": 0.9624, + "mutation": { + "ref_knob": 128, + "recall_before": 0.9624, + "insert_10k": { + "seconds": 1.8145926999859512, + "per_s": 5510.8785569772335, + "batch": 1000 + }, + "update_10k": { + "seconds": 1.7646502000279725, + "per_s": 5666.845474441045, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.005598100018687546, + "per_s": 1786320.3527300435, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 268.125, + "peak_mb": 272.1875 + }, + "first_query_after_ms": 6.036199978552759, + "after": { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9584, + "qps_1t_runs": [ + 1205.2753932002645, + 1275.3994201524467, + 1347.2876384278284 + ], + "qps_1t": 1275.3994201524467, + "qps_1t_min": 1205.2753932002645, + "qps_1t_max": 1347.2876384278284, + "median_ms": 0.7895999879110605, + "p95_ms": 0.9713250037748365, + "p99_ms": 1.0692049923818556, + "mean_ms": 0.7853286672034301 + }, + "recall_after": 0.9584 + }, + "saveload": { + "save_s": 0.48650389997055754, + "disk_mb": 176.69028091430664, + "load_s": 0.29026550002163276, + "first_query_after_load_ms": 1.2838999973610044, + "time_to_ready_s": 0.29154940001899377 + }, + "mem_end": { + "rss_mb": 232.609375, + "peak_mb": 272.19140625 + }, + "status": "ok", + "finished": "2026-11-16 19:42:24" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r2.load.json b/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r2.load.json new file mode 100644 index 0000000..c6752c9 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r2.load.json @@ -0,0 +1 @@ +{"total_pct": 19.63125, "pin_cpu_pct": 24.5, "max_cpu_pct": 73.0, "noisy": true, "seconds": 40.759572982788086, "after": 27.8} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r2.txt b/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r2.txt new file mode 100644 index 0000000..0780388 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r2.txt @@ -0,0 +1,10 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system usearch --ds text384 --n 100000 --exp core,mutation,saveload --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\usearch_text384_100000_r2.json + usearch text384 100000 knob=24 recall=0.7817 qps1t=5198 med=0.190ms batch=30928 + usearch text384 100000 knob=32 recall=0.8283 qps1t=4319 med=0.233ms batch=19141 + usearch text384 100000 knob=48 recall=0.8832 qps1t=3100 med=0.318ms batch=20171 + usearch text384 100000 knob=64 recall=0.9149 qps1t=2571 med=0.384ms batch=15409 + usearch text384 100000 knob=96 recall=0.9477 qps1t=1836 med=0.540ms batch=10818 + usearch text384 100000 knob=128 recall=0.9624 qps1t=1370 med=0.745ms batch=8282 + usearch text384 100000 knob=192 recall=0.9783 qps1t=957 med=1.057ms batch=5842 + usearch text384 100000 knob=256 recall=0.9852 qps1t=753 med=1.312ms batch=3844 + usearch text384 100000 knob=400 recall=0.9927 qps1t=506 med=2.007ms batch=3622 diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r3.json b/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r3.json new file mode 100644 index 0000000..dcea04c --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r3.json @@ -0,0 +1,369 @@ +{ + "system": "usearch", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 19:51:41", + "mem_baseline": { + "rss_mb": 37.13671875, + "peak_mb": 37.13671875 + }, + "version": "usearch 2.26.0", + "build": { + "ingest_s": 8.778196099970955, + "index_s": 3.3999676816165447e-06, + "total_s": 8.778199499938637, + "first_query_ms": 3.7631000159308314, + "ready_s": 8.781962599954568 + }, + "mem_after_build": { + "rss_mb": 218.9609375, + "peak_mb": 249.09375 + }, + "mem_after_first_query": { + "rss_mb": 219.01171875, + "peak_mb": 249.09375 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.7818, + "qps_1t_runs": [ + 4857.693836920501, + 5333.316266591327, + 4973.271160208158 + ], + "qps_1t": 4973.271160208158, + "qps_1t_min": 4857.693836920501, + "qps_1t_max": 5333.316266591327, + "median_ms": 0.1954999752342701, + "p95_ms": 0.26571998896542925, + "p99_ms": 0.3039049863582476, + "mean_ms": 0.19814483355730772, + "knob": 24, + "batch": { + "recall": 0.7818, + "qps_batch_runs": [ + 34362.371831452416, + 33317.118976892445, + 32231.655376411538 + ], + "qps_batch": 33317.118976892445, + "qps_batch_min": 32231.655376411538, + "qps_batch_max": 34362.371831452416, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8255, + "qps_1t_runs": [ + 4298.370501830824, + 4524.143781290379, + 4459.453964501257 + ], + "qps_1t": 4459.453964501257, + "qps_1t_min": 4298.370501830824, + "qps_1t_max": 4524.143781290379, + "median_ms": 0.22215000353753567, + "p95_ms": 0.28839998703915626, + "p99_ms": 0.33420498366467644, + "mean_ms": 0.225975100552508, + "knob": 32, + "batch": { + "recall": 0.8255, + "qps_batch_runs": [ + 17771.332724903645, + 27315.239713269904, + 24507.641490874237 + ], + "qps_batch": 24507.641490874237, + "qps_batch_min": 17771.332724903645, + "qps_batch_max": 27315.239713269904, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8825, + "qps_1t_runs": [ + 2964.9211254570846, + 2880.8771750882615, + 2979.287688090109 + ], + "qps_1t": 2964.9211254570846, + "qps_1t_min": 2880.8771750882615, + "qps_1t_max": 2979.287688090109, + "median_ms": 0.33870001789182425, + "p95_ms": 0.42150000808760524, + "p99_ms": 0.4791839810786749, + "mean_ms": 0.34001476678531617, + "knob": 48, + "batch": { + "recall": 0.8825, + "qps_batch_runs": [ + 17897.700322705598, + 17545.798927228614, + 16591.6998294424 + ], + "qps_batch": 17545.798927228614, + "qps_batch_min": 16591.6998294424, + "qps_batch_max": 17897.700322705598, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9135, + "qps_1t_runs": [ + 2435.428272581385, + 2233.347766477007, + 2257.053572254598 + ], + "qps_1t": 2257.053572254598, + "qps_1t_min": 2233.347766477007, + "qps_1t_max": 2435.428272581385, + "median_ms": 0.43704998097382486, + "p95_ms": 0.530319966492243, + "p99_ms": 0.5760140198981386, + "mean_ms": 0.4338064002610433, + "knob": 64, + "batch": { + "recall": 0.9135, + "qps_batch_runs": [ + 12622.565110324278, + 13093.786870261025, + 13425.143412998486 + ], + "qps_batch": 13093.786870261025, + "qps_batch_min": 12622.565110324278, + "qps_batch_max": 13425.143412998486, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9477, + "qps_1t_runs": [ + 1541.3022804825898, + 1645.8182624118108, + 1589.5567395108005 + ], + "qps_1t": 1589.5567395108005, + "qps_1t_min": 1541.3022804825898, + "qps_1t_max": 1645.8182624118108, + "median_ms": 0.6289500161074102, + "p95_ms": 0.7913050299976021, + "p99_ms": 0.8886190142948176, + "mean_ms": 0.6285028996838568, + "knob": 96, + "batch": { + "recall": 0.9477, + "qps_batch_runs": [ + 10961.871324873851, + 10494.027319492592, + 10540.284439790847 + ], + "qps_batch": 10540.284439790847, + "qps_batch_min": 10494.027319492592, + "qps_batch_max": 10961.871324873851, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9623, + "qps_1t_runs": [ + 1325.363477664774, + 1350.6838579185726, + 1284.716576283186 + ], + "qps_1t": 1325.363477664774, + "qps_1t_min": 1284.716576283186, + "qps_1t_max": 1350.6838579185726, + "median_ms": 0.7680999988224357, + "p95_ms": 0.9170049976091832, + "p99_ms": 0.985701996833086, + "mean_ms": 0.7577525003386351, + "knob": 128, + "batch": { + "recall": 0.9623, + "qps_batch_runs": [ + 8035.006915293899, + 6896.266361101562, + 6781.610443094545 + ], + "qps_batch": 6896.266361101562, + "qps_batch_min": 6781.610443094545, + "qps_batch_max": 8035.006915293899, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9775, + "qps_1t_runs": [ + 943.8889185394786, + 899.0372211450161, + 909.4931530620435 + ], + "qps_1t": 909.4931530620435, + "qps_1t_min": 899.0372211450161, + "qps_1t_max": 943.8889185394786, + "median_ms": 1.097600004868582, + "p95_ms": 1.34753001620993, + "p99_ms": 1.4581099851056931, + "mean_ms": 1.0904204000059206, + "knob": 192, + "batch": { + "recall": 0.9775, + "qps_batch_runs": [ + 5367.669239760785, + 5734.0567421890655, + 4939.920684069806 + ], + "qps_batch": 5367.669239760785, + "qps_batch_min": 4939.920684069806, + "qps_batch_max": 5734.0567421890655, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9851, + "qps_1t_runs": [ + 721.5426297207644, + 716.1003122292746, + 713.5354320170455 + ], + "qps_1t": 716.1003122292746, + "qps_1t_min": 713.5354320170455, + "qps_1t_max": 721.5426297207644, + "median_ms": 1.4021999668329954, + "p95_ms": 1.726524991681799, + "p99_ms": 1.830331024830229, + "mean_ms": 1.3946146663899224, + "knob": 256, + "batch": { + "recall": 0.9851, + "qps_batch_runs": [ + 4727.85072951339, + 4889.5284387658485, + 4901.203979940309 + ], + "qps_batch": 4889.5284387658485, + "qps_batch_min": 4727.85072951339, + "qps_batch_max": 4901.203979940309, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.992, + "qps_1t_runs": [ + 459.37015655473556, + 479.53100309443096, + 480.44975469738705 + ], + "qps_1t": 479.53100309443096, + "qps_1t_min": 459.37015655473556, + "qps_1t_max": 480.44975469738705, + "median_ms": 2.1160999895073473, + "p95_ms": 2.6536400080658495, + "p99_ms": 2.920917948358692, + "mean_ms": 2.1145492339661964, + "knob": 400, + "batch": { + "recall": 0.992, + "qps_batch_runs": [ + 3239.8055470348354, + 3508.991087698534, + 3454.1598446892394 + ], + "qps_batch": 3454.1598446892394, + "qps_batch_min": 3239.8055470348354, + "qps_batch_max": 3508.991087698534, + "threads": 32 + } + } + ], + "knob_name": "expansion_search", + "mem_after_sweep": { + "rss_mb": 221.44921875, + "peak_mb": 249.09375 + }, + "ref_knob": 128, + "ref_recall_pre": 0.9623, + "mutation": { + "ref_knob": 128, + "recall_before": 0.9623, + "insert_10k": { + "seconds": 1.835152699961327, + "per_s": 5449.137829353783, + "batch": 1000 + }, + "update_10k": { + "seconds": 1.84252559999004, + "per_s": 5427.333004249198, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.006663300038781017, + "per_s": 1500757.8739962305, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 267.30078125, + "peak_mb": 271.30859375 + }, + "first_query_after_ms": 6.867000018246472, + "after": { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9574, + "qps_1t_runs": [ + 1175.3636725887038, + 1199.2806234501902, + 1189.186489199336 + ], + "qps_1t": 1189.186489199336, + "qps_1t_min": 1175.3636725887038, + "qps_1t_max": 1199.2806234501902, + "median_ms": 0.8487500017508864, + "p95_ms": 1.030599992373027, + "p99_ms": 1.1329100118018685, + "mean_ms": 0.8418482333072461 + }, + "recall_after": 0.9574 + }, + "saveload": { + "save_s": 0.6613980000256561, + "disk_mb": 176.69028091430664, + "load_s": 0.3175666999886744, + "first_query_after_load_ms": 1.8114999984391034, + "time_to_ready_s": 0.3193781999871135 + }, + "mem_end": { + "rss_mb": 231.52734375, + "peak_mb": 271.3671875 + }, + "status": "ok", + "finished": "2026-11-16 19:52:23" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r3.load.json b/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r3.load.json new file mode 100644 index 0000000..c2e47f1 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r3.load.json @@ -0,0 +1 @@ +{"total_pct": 16.625, "pin_cpu_pct": 27.4, "max_cpu_pct": 65.8, "noisy": true, "seconds": 42.07283020019531, "after": 24.4} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r3.txt b/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r3.txt new file mode 100644 index 0000000..830b743 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/discarded_noisy/usearch_text384_100000_r3.txt @@ -0,0 +1,10 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system usearch --ds text384 --n 100000 --exp core,mutation,saveload --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\usearch_text384_100000_r3.json + usearch text384 100000 knob=24 recall=0.7818 qps1t=4973 med=0.195ms batch=33317 + usearch text384 100000 knob=32 recall=0.8255 qps1t=4459 med=0.222ms batch=24508 + usearch text384 100000 knob=48 recall=0.8825 qps1t=2965 med=0.339ms batch=17546 + usearch text384 100000 knob=64 recall=0.9135 qps1t=2257 med=0.437ms batch=13094 + usearch text384 100000 knob=96 recall=0.9477 qps1t=1590 med=0.629ms batch=10540 + usearch text384 100000 knob=128 recall=0.9623 qps1t=1325 med=0.768ms batch=6896 + usearch text384 100000 knob=192 recall=0.9775 qps1t=909 med=1.098ms batch=5368 + usearch text384 100000 knob=256 recall=0.9851 qps1t=716 med=1.402ms batch=4890 + usearch text384 100000 knob=400 recall=0.9920 qps1t=480 med=2.116ms batch=3454 diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r1.json b/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r1.json new file mode 100644 index 0000000..4e22380 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r1.json @@ -0,0 +1,87 @@ +{ + "system": "faiss_flat", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 21:05:38", + "mem_baseline": { + "rss_mb": 34.9765625, + "peak_mb": 34.9765625 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 0.03607400000328198, + "index_s": 3.00002284348011e-06, + "total_s": 0.03607700002612546, + "first_query_ms": 31.176600023172796, + "ready_s": 0.06725360004929826 + }, + "mem_after_build": { + "rss_mb": 100.28515625, + "peak_mb": 137.60546875 + }, + "mem_after_first_query": { + "rss_mb": 100.40234375, + "peak_mb": 137.60546875 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 598.6134556667304, + 603.5059472450935, + 608.2948543139636 + ], + "qps_1t": 603.5059472450935, + "qps_1t_min": 598.6134556667304, + "qps_1t_max": 608.2948543139636, + "median_ms": 1.5737999929115176, + "p95_ms": 2.133150002919137, + "p99_ms": 2.7411400323035138, + "mean_ms": 1.6571504000845987, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 14934.445261031204, + 14527.91539608276, + 16914.549389054624 + ], + "qps_batch": 14934.445261031204, + "qps_batch_min": 14527.91539608276, + "qps_batch_max": 16914.549389054624, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 1000, + "threads": 32, + "qps": 522.2684017654915, + "median_ms": 1.8122000037692487, + "p95_ms": 2.600135008106008, + "p99_ms": 3.92416396178305, + "mean_ms": 1.9219770000005763 + }, + "mem_after_sweep": { + "rss_mb": 110.5625, + "peak_mb": 137.60546875 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 109.0390625, + "peak_mb": 137.60546875 + }, + "status": "ok", + "finished": "2026-11-16 21:05:50" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r1.load.json b/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r1.load.json new file mode 100644 index 0000000..10310fd --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 19.478125, "pin_cpu_pct": 77.3, "max_cpu_pct": 82.9, "noisy": true, "seconds": 11.709978342056274, "after": 18.8} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r1.txt b/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r1.txt new file mode 100644 index 0000000..6d05d4c --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r1.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_flat --ds glove --n 100000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_flat_glove_100000_r1.json + faiss_flat glove 100000 knob=None recall=1.0000 qps1t=604 med=1.574ms batch=14934 diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r2.json b/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r2.json new file mode 100644 index 0000000..4fd2c24 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r2.json @@ -0,0 +1,87 @@ +{ + "system": "faiss_flat", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 19:58:20", + "mem_baseline": { + "rss_mb": 35.0078125, + "peak_mb": 35.0078125 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 0.0322325000888668, + "index_s": 1.7999554984271526e-06, + "total_s": 0.03223430004436523, + "first_query_ms": 5.6643999996595085, + "ready_s": 0.037898700044024736 + }, + "mem_after_build": { + "rss_mb": 100.40234375, + "peak_mb": 137.71484375 + }, + "mem_after_first_query": { + "rss_mb": 100.51953125, + "peak_mb": 137.71484375 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 679.3021392160266, + 605.0412763708653, + 462.4362781845557 + ], + "qps_1t": 605.0412763708653, + "qps_1t_min": 462.4362781845557, + "qps_1t_max": 679.3021392160266, + "median_ms": 1.588999992236495, + "p95_ms": 2.405404980527237, + "p99_ms": 2.717174967983736, + "mean_ms": 1.7624462999519892, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 4865.183338244908, + 5704.617203829466, + 5444.470214393566 + ], + "qps_batch": 5444.470214393566, + "qps_batch_min": 4865.183338244908, + "qps_batch_max": 5704.617203829466, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 677, + "threads": 32, + "qps": 191.33887750222092, + "median_ms": 4.889899981208146, + "p95_ms": 6.722599995555356, + "p99_ms": 7.573070027865472, + "mean_ms": 5.059992664193039 + }, + "mem_after_sweep": { + "rss_mb": 110.8359375, + "peak_mb": 137.71484375 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 109.3125, + "peak_mb": 137.71484375 + }, + "status": "ok", + "finished": "2026-11-16 19:58:37" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r2.load.json b/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r2.load.json new file mode 100644 index 0000000..aba0f16 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r2.load.json @@ -0,0 +1 @@ +{"total_pct": 10.8875, "pin_cpu_pct": 0.0, "max_cpu_pct": 53.1, "noisy": false, "seconds": 17.163042783737183, "after": 16.0} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r2.txt b/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r2.txt new file mode 100644 index 0000000..0b78e92 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r2.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_flat --ds glove --n 100000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_flat_glove_100000_r2.json + faiss_flat glove 100000 knob=None recall=1.0000 qps1t=605 med=1.589ms batch=5444 diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r3.json b/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r3.json new file mode 100644 index 0000000..f4254dc --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r3.json @@ -0,0 +1,87 @@ +{ + "system": "faiss_flat", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 21:07:35", + "mem_baseline": { + "rss_mb": 34.9296875, + "peak_mb": 34.9296875 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 0.03349719999823719, + "index_s": 2.700020559132099e-06, + "total_s": 0.033499900018796325, + "first_query_ms": 38.96620002342388, + "ready_s": 0.0724661000422202 + }, + "mem_after_build": { + "rss_mb": 100.78515625, + "peak_mb": 138.03515625 + }, + "mem_after_first_query": { + "rss_mb": 100.90234375, + "peak_mb": 138.03515625 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 621.6223379371471, + 629.7176870573993, + 595.3832198089974 + ], + "qps_1t": 621.6223379371471, + "qps_1t_min": 595.3832198089974, + "qps_1t_max": 629.7176870573993, + "median_ms": 1.5610999835189432, + "p95_ms": 2.0339350041467696, + "p99_ms": 2.455907015246339, + "mean_ms": 1.625432499917224, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 13388.027888708828, + 15183.90749504239, + 16481.062445448275 + ], + "qps_batch": 15183.90749504239, + "qps_batch_min": 13388.027888708828, + "qps_batch_max": 16481.062445448275, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 1000, + "threads": 32, + "qps": 522.1097283062809, + "median_ms": 1.8112999969162047, + "p95_ms": 2.4754349869908747, + "p99_ms": 3.105981015833088, + "mean_ms": 1.8921753673930652 + }, + "mem_after_sweep": { + "rss_mb": 110.91015625, + "peak_mb": 138.03515625 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 109.38671875, + "peak_mb": 138.03515625 + }, + "status": "ok", + "finished": "2026-11-16 21:07:47" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r3.load.json b/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r3.load.json new file mode 100644 index 0000000..70dd7c4 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r3.load.json @@ -0,0 +1 @@ +{"total_pct": 17.9125, "pin_cpu_pct": 95.3, "max_cpu_pct": 98.8, "noisy": true, "seconds": 11.469490051269531, "after": 20.0} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r3.txt b/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r3.txt new file mode 100644 index 0000000..32c2b14 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_glove_100000_r3.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_flat --ds glove --n 100000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_flat_glove_100000_r3.json + faiss_flat glove 100000 knob=None recall=1.0000 qps1t=622 med=1.561ms batch=15184 diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r1.json b/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r1.json new file mode 100644 index 0000000..c849bec --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r1.json @@ -0,0 +1,87 @@ +{ + "system": "faiss_flat", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 20:04:18", + "mem_baseline": { + "rss_mb": 35.12890625, + "peak_mb": 35.12890625 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 0.04089339997153729, + "index_s": 2.8999638743698597e-06, + "total_s": 0.04089629993541166, + "first_query_ms": 8.9548000250943, + "ready_s": 0.04985109996050596 + }, + "mem_after_build": { + "rss_mb": 111.15625, + "peak_mb": 159.03515625 + }, + "mem_after_first_query": { + "rss_mb": 111.2734375, + "peak_mb": 159.03515625 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 541.3075004476334, + 431.6622346944021, + 391.6502976133937 + ], + "qps_1t": 431.6622346944021, + "qps_1t_min": 391.6502976133937, + "qps_1t_max": 541.3075004476334, + "median_ms": 2.2421999892685562, + "p95_ms": 2.853424957720563, + "p99_ms": 3.109046040917746, + "mean_ms": 2.239101033580179, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 5075.415599745178, + 3815.710347456524, + 7228.060489264821 + ], + "qps_batch": 5075.415599745178, + "qps_batch_min": 3815.710347456524, + "qps_batch_max": 7228.060489264821, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 453, + "threads": 32, + "qps": 146.8300986301158, + "median_ms": 6.747899984475225, + "p95_ms": 9.169730002759021, + "p99_ms": 10.268844022648413, + "mean_ms": 6.898814275969749 + }, + "mem_after_sweep": { + "rss_mb": 121.625, + "peak_mb": 159.03515625 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 119.88671875, + "peak_mb": 159.03515625 + }, + "status": "ok", + "finished": "2026-11-16 20:04:35" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r1.load.json b/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r1.load.json new file mode 100644 index 0000000..4147fec --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 16.2625, "pin_cpu_pct": 0.0, "max_cpu_pct": 71.8, "noisy": false, "seconds": 17.742955207824707, "after": 19.2} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r1.txt b/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r1.txt new file mode 100644 index 0000000..58cd896 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r1.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_flat --ds sift --n 100000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_flat_sift_100000_r1.json + faiss_flat sift 100000 knob=None recall=1.0000 qps1t=432 med=2.242ms batch=5075 diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r2.json b/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r2.json new file mode 100644 index 0000000..da70314 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r2.json @@ -0,0 +1,87 @@ +{ + "system": "faiss_flat", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 21:09:35", + "mem_baseline": { + "rss_mb": 35.359375, + "peak_mb": 35.359375 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 0.04151040001306683, + "index_s": 2.3999600671231747e-06, + "total_s": 0.04151279997313395, + "first_query_ms": 13.864199980162084, + "ready_s": 0.055376999953296036 + }, + "mem_after_build": { + "rss_mb": 111.375, + "peak_mb": 159.37109375 + }, + "mem_after_first_query": { + "rss_mb": 111.5546875, + "peak_mb": 159.37109375 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 533.0766324035652, + 521.5784570587275, + 516.0439348431948 + ], + "qps_1t": 521.5784570587275, + "qps_1t_min": 516.0439348431948, + "qps_1t_max": 533.0766324035652, + "median_ms": 1.8482999876141548, + "p95_ms": 2.2726049617631356, + "p99_ms": 2.6807920390274353, + "mean_ms": 1.9103264997247607, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 13632.335900673517, + 14214.742110920848, + 14071.149362468675 + ], + "qps_batch": 14071.149362468675, + "qps_batch_min": 13632.335900673517, + "qps_batch_max": 14214.742110920848, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 1000, + "threads": 32, + "qps": 484.85532601475074, + "median_ms": 2.0098499953746796, + "p95_ms": 2.5419049721676856, + "p99_ms": 3.205100012710306, + "mean_ms": 2.070837166684214 + }, + "mem_after_sweep": { + "rss_mb": 121.484375, + "peak_mb": 159.37109375 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 119.74609375, + "peak_mb": 159.37109375 + }, + "status": "ok", + "finished": "2026-11-16 21:09:48" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r2.load.json b/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r2.load.json new file mode 100644 index 0000000..15407f5 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r2.load.json @@ -0,0 +1 @@ +{"total_pct": 16.996875, "pin_cpu_pct": 96.5, "max_cpu_pct": 96.5, "noisy": true, "seconds": 12.97842001914978, "after": 15.9} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r2.txt b/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r2.txt new file mode 100644 index 0000000..cfdae8a --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r2.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_flat --ds sift --n 100000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_flat_sift_100000_r2.json + faiss_flat sift 100000 knob=None recall=1.0000 qps1t=522 med=1.848ms batch=14071 diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r3.json b/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r3.json new file mode 100644 index 0000000..7f5cdce --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r3.json @@ -0,0 +1,87 @@ +{ + "system": "faiss_flat", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 20:10:24", + "mem_baseline": { + "rss_mb": 35.2265625, + "peak_mb": 35.2265625 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 0.04231789999175817, + "index_s": 2.600019797682762e-06, + "total_s": 0.04232050001155585, + "first_query_ms": 16.87829999718815, + "ready_s": 0.059198800008744 + }, + "mem_after_build": { + "rss_mb": 111.16015625, + "peak_mb": 159.10546875 + }, + "mem_after_first_query": { + "rss_mb": 111.27734375, + "peak_mb": 159.10546875 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 494.7421525044408, + 421.52335367303846, + 406.03395716584043 + ], + "qps_1t": 421.52335367303846, + "qps_1t_min": 406.03395716584043, + "qps_1t_max": 494.7421525044408, + "median_ms": 2.372599992668256, + "p95_ms": 2.790019998792559, + "p99_ms": 3.1132469576550634, + "mean_ms": 2.2854836323143295, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 5273.132441008991, + 6202.454311157889, + 5898.3130833805435 + ], + "qps_batch": 5898.3130833805435, + "qps_batch_min": 5273.132441008991, + "qps_batch_max": 6202.454311157889, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 382, + "threads": 32, + "qps": 146.06440361245853, + "median_ms": 6.742950004991144, + "p95_ms": 10.237375012366101, + "p99_ms": 11.081314986222422, + "mean_ms": 7.209902792654193 + }, + "mem_after_sweep": { + "rss_mb": 121.48828125, + "peak_mb": 159.10546875 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 119.75390625, + "peak_mb": 159.10546875 + }, + "status": "ok", + "finished": "2026-11-16 20:10:40" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r3.load.json b/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r3.load.json new file mode 100644 index 0000000..e23c488 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r3.load.json @@ -0,0 +1 @@ +{"total_pct": 10.88125, "pin_cpu_pct": 0.4, "max_cpu_pct": 66.0, "noisy": false, "seconds": 16.71510887145996, "after": 15.3} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r3.txt b/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r3.txt new file mode 100644 index 0000000..d8454f5 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_sift_100000_r3.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_flat --ds sift --n 100000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_flat_sift_100000_r3.json + faiss_flat sift 100000 knob=None recall=1.0000 qps1t=422 med=2.373ms batch=5898 diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_text384_1000000_r1.json b/benchmarks/rust-stage-c/raw/faiss_flat_text384_1000000_r1.json new file mode 100644 index 0000000..0979f58 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_text384_1000000_r1.json @@ -0,0 +1,87 @@ +{ + "system": "faiss_flat", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 20:18:18", + "mem_baseline": { + "rss_mb": 37.0625, + "peak_mb": 37.0625 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 1.0541833002353087, + "index_s": 2.400018274784088e-06, + "total_s": 1.0541857002535835, + "first_query_ms": 73.76410003053024, + "ready_s": 1.1279498002841137 + }, + "mem_after_build": { + "rss_mb": 1592.09375, + "peak_mb": 2160.81640625 + }, + "mem_after_first_query": { + "rss_mb": 1592.2109375, + "peak_mb": 2160.81640625 + }, + "sweep": [ + { + "nq": 35, + "recall_queries": 35, + "recall": 1.0, + "qps_1t_runs": [ + 11.40455204510648, + 11.148302556121424, + 11.368230900068069 + ], + "qps_1t": 11.368230900068069, + "qps_1t_min": 11.148302556121424, + "qps_1t_max": 11.40455204510648, + "median_ms": 89.31650000158697, + "p95_ms": 92.67392001347616, + "p99_ms": 95.70731200277805, + "mean_ms": 88.44949143129357, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 297.6627048142383, + 297.7520257982094, + 258.48313874406296 + ], + "qps_batch": 297.6627048142383, + "qps_batch_min": 258.48313874406296, + "qps_batch_max": 297.7520257982094, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 30, + "threads": 32, + "qps": 4.749054573879548, + "median_ms": 210.18525000545196, + "p95_ms": 235.18398000160232, + "p99_ms": 252.2617090289714, + "mean_ms": 210.56822667402835 + }, + "mem_after_sweep": { + "rss_mb": 1602.03515625, + "peak_mb": 2160.81640625 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 1599.1015625, + "peak_mb": 2160.81640625 + }, + "status": "ok", + "finished": "2026-11-16 20:18:55" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_text384_1000000_r1.load.json b/benchmarks/rust-stage-c/raw/faiss_flat_text384_1000000_r1.load.json new file mode 100644 index 0000000..31e98c5 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_text384_1000000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 9.665625, "pin_cpu_pct": 3.8, "max_cpu_pct": 60.9, "noisy": false, "seconds": 38.14351987838745, "after": 17.7} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_text384_1000000_r1.txt b/benchmarks/rust-stage-c/raw/faiss_flat_text384_1000000_r1.txt new file mode 100644 index 0000000..a50cdac --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_text384_1000000_r1.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_flat --ds text384 --n 1000000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_flat_text384_1000000_r1.json + faiss_flat text384 1000000 knob=None recall=1.0000 qps1t=11 med=89.317ms batch=298 diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r1.json b/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r1.json new file mode 100644 index 0000000..df3053e --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r1.json @@ -0,0 +1,87 @@ +{ + "system": "faiss_flat", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 19:10:25", + "mem_baseline": { + "rss_mb": 37.078125, + "peak_mb": 37.078125 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 0.10431710002012551, + "index_s": 2.100015990436077e-06, + "total_s": 0.10431920003611594, + "first_query_ms": 10.981299972627312, + "ready_s": 0.11530050000874326 + }, + "mem_after_build": { + "rss_mb": 210.71484375, + "peak_mb": 356.30078125 + }, + "mem_after_first_query": { + "rss_mb": 210.83203125, + "peak_mb": 356.30078125 + }, + "sweep": [ + { + "nq": 421, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 134.62781115810012, + 117.76265279742822, + 116.92255067644493 + ], + "qps_1t": 117.76265279742822, + "qps_1t_min": 116.92255067644493, + "qps_1t_max": 134.62781115810012, + "median_ms": 8.367600035853684, + "p95_ms": 9.27123999572359, + "p99_ms": 9.763318039476868, + "mean_ms": 8.15740411853018, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 3536.962138289172, + 2995.2576084331295, + 3070.2620251315043 + ], + "qps_batch": 3070.2620251315043, + "qps_batch_min": 2995.2576084331295, + "qps_batch_max": 3536.962138289172, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 161, + "threads": 32, + "qps": 55.850449008647196, + "median_ms": 18.1260000099428, + "p95_ms": 22.16910996939987, + "p99_ms": 25.339064022991806, + "mean_ms": 18.384274120725053 + }, + "mem_after_sweep": { + "rss_mb": 221.234375, + "peak_mb": 356.30078125 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 218.30078125, + "peak_mb": 356.30078125 + }, + "status": "ok", + "finished": "2026-11-16 19:10:51" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r1.load.json b/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r1.load.json new file mode 100644 index 0000000..60e0cda --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 17.38125, "pin_cpu_pct": 0.0, "max_cpu_pct": 58.3, "noisy": false, "seconds": 26.8358314037323, "after": 19.3} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r1.txt b/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r1.txt new file mode 100644 index 0000000..63c9480 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r1.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_flat --ds text384 --n 100000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_flat_text384_100000_r1.json + faiss_flat text384 100000 knob=None recall=1.0000 qps1t=118 med=8.368ms batch=3070 diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r2.json b/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r2.json new file mode 100644 index 0000000..d55e7bc --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r2.json @@ -0,0 +1,87 @@ +{ + "system": "faiss_flat", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 19:32:24", + "mem_baseline": { + "rss_mb": 37.11328125, + "peak_mb": 37.11328125 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 0.12835179991088808, + "index_s": 9.199953638017178e-06, + "total_s": 0.1283609998645261, + "first_query_ms": 57.90720001095906, + "ready_s": 0.18626819987548515 + }, + "mem_after_build": { + "rss_mb": 210.921875, + "peak_mb": 356.5859375 + }, + "mem_after_first_query": { + "rss_mb": 211.0390625, + "peak_mb": 356.5859375 + }, + "sweep": [ + { + "nq": 398, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 129.15341813985205, + 115.12677614310674, + 116.54350928026874 + ], + "qps_1t": 116.54350928026874, + "qps_1t_min": 115.12677614310674, + "qps_1t_max": 129.15341813985205, + "median_ms": 8.397850004257634, + "p95_ms": 9.747960008098742, + "p99_ms": 11.037375961313916, + "mean_ms": 8.336430904328733, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 4774.140201280389, + 5445.558302255219, + 5085.293077584747 + ], + "qps_batch": 5085.293077584747, + "qps_batch_min": 4774.140201280389, + "qps_batch_max": 5445.558302255219, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 433, + "threads": 32, + "qps": 131.524955332032, + "median_ms": 6.829699967056513, + "p95_ms": 13.166159967659027, + "p99_ms": 16.612987988628444, + "mean_ms": 7.547642571019587 + }, + "mem_after_sweep": { + "rss_mb": 221.7265625, + "peak_mb": 356.5859375 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 218.79296875, + "peak_mb": 356.5859375 + }, + "status": "ok", + "finished": "2026-11-16 19:32:50" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r2.load.json b/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r2.load.json new file mode 100644 index 0000000..e3e76a2 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r2.load.json @@ -0,0 +1 @@ +{"total_pct": 20.29375, "pin_cpu_pct": 0.0, "max_cpu_pct": 77.0, "noisy": false, "seconds": 26.559379816055298, "after": 17.5} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r2.txt b/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r2.txt new file mode 100644 index 0000000..d1bb648 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r2.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_flat --ds text384 --n 100000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_flat_text384_100000_r2.json + faiss_flat text384 100000 knob=None recall=1.0000 qps1t=117 med=8.398ms batch=5085 diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r3.json b/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r3.json new file mode 100644 index 0000000..0ac19ad --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r3.json @@ -0,0 +1,87 @@ +{ + "system": "faiss_flat", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 21:00:23", + "mem_baseline": { + "rss_mb": 37.09375, + "peak_mb": 37.09375 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 0.10657429991988465, + "index_s": 2.2999593056738377e-06, + "total_s": 0.10657659987919033, + "first_query_ms": 9.923300007358193, + "ready_s": 0.11649989988654852 + }, + "mem_after_build": { + "rss_mb": 211.3125, + "peak_mb": 356.8671875 + }, + "mem_after_first_query": { + "rss_mb": 211.4296875, + "peak_mb": 356.8671875 + }, + "sweep": [ + { + "nq": 469, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 133.6554705759962, + 135.50167209408608, + 141.84084862744749 + ], + "qps_1t": 135.50167209408608, + "qps_1t_min": 133.6554705759962, + "qps_1t_max": 141.84084862744749, + "median_ms": 6.903200002852827, + "p95_ms": 9.44167000707239, + "p99_ms": 12.594503972213722, + "mean_ms": 7.304020469505856, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 6837.331007873816, + 6552.053774895063, + 6446.850553644093 + ], + "qps_batch": 6552.053774895063, + "qps_batch_min": 6446.850553644093, + "qps_batch_max": 6837.331007873816, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 419, + "threads": 32, + "qps": 137.47756106120318, + "median_ms": 7.111600018106401, + "p95_ms": 9.397599997464567, + "p99_ms": 11.870199972763663, + "mean_ms": 7.4186954653692965 + }, + "mem_after_sweep": { + "rss_mb": 221.60546875, + "peak_mb": 356.8671875 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 218.671875, + "peak_mb": 356.8671875 + }, + "status": "ok", + "finished": "2026-11-16 21:00:48" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r3.load.json b/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r3.load.json new file mode 100644 index 0000000..73ec00d --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r3.load.json @@ -0,0 +1 @@ +{"total_pct": 13.3, "pin_cpu_pct": 3.2, "max_cpu_pct": 79.5, "noisy": false, "seconds": 25.203767776489258, "after": 20.2} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r3.txt b/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r3.txt new file mode 100644 index 0000000..29ddd92 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_flat_text384_100000_r3.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_flat --ds text384 --n 100000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_flat_text384_100000_r3.json + faiss_flat text384 100000 knob=None recall=1.0000 qps1t=136 med=6.903ms batch=6552 diff --git a/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_1000000_r1.json b/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_1000000_r1.json new file mode 100644 index 0000000..88c031d --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_1000000_r1.json @@ -0,0 +1,346 @@ +{ + "system": "faiss_hnsw", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 21:20:24", + "mem_baseline": { + "rss_mb": 37.05859375, + "peak_mb": 37.05859375 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 189.53744040016318, + "index_s": 7.599999662488699e-06, + "total_s": 189.53744800016284, + "first_query_ms": 4.938300000503659, + "ready_s": 189.54238630016334 + }, + "mem_after_build": { + "rss_mb": 1851.609375, + "peak_mb": 2334.94921875 + }, + "mem_after_first_query": { + "rss_mb": 1852.6328125, + "peak_mb": 2334.94921875 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.7605, + "qps_1t_runs": [ + 6168.114455984018, + 6018.531049668905, + 6386.544319729063 + ], + "qps_1t": 6168.114455984018, + "qps_1t_min": 6018.531049668905, + "qps_1t_max": 6386.544319729063, + "median_ms": 0.15440001152455807, + "p95_ms": 0.2197249850723891, + "p99_ms": 0.2890039887279271, + "mean_ms": 0.161618933120432, + "knob": 24, + "batch": { + "recall": 0.7605, + "qps_batch_runs": [ + 25441.990998395104, + 25046.461187929082, + 23724.060703014726 + ], + "qps_batch": 25046.461187929082, + "qps_batch_min": 23724.060703014726, + "qps_batch_max": 25441.990998395104, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8023, + "qps_1t_runs": [ + 5394.394981800675, + 5555.027838478982, + 5281.774761749516 + ], + "qps_1t": 5394.394981800675, + "qps_1t_min": 5281.774761749516, + "qps_1t_max": 5555.027838478982, + "median_ms": 0.17660000594332814, + "p95_ms": 0.2408000291325152, + "p99_ms": 0.2984240022487933, + "mean_ms": 0.1849083335061247, + "knob": 32, + "batch": { + "recall": 0.8023, + "qps_batch_runs": [ + 20290.684341598284, + 20328.50869818014, + 20311.703414193387 + ], + "qps_batch": 20311.703414193387, + "qps_batch_min": 20290.684341598284, + "qps_batch_max": 20328.50869818014, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8589, + "qps_1t_runs": [ + 4192.754312557277, + 4129.14819775005, + 3532.7475128327364 + ], + "qps_1t": 4129.14819775005, + "qps_1t_min": 3532.7475128327364, + "qps_1t_max": 4192.754312557277, + "median_ms": 0.24140000459738076, + "p95_ms": 0.339320019702427, + "p99_ms": 0.46423004125244843, + "mean_ms": 0.25458440023552004, + "knob": 48, + "batch": { + "recall": 0.8589, + "qps_batch_runs": [ + 16414.37768799544, + 15066.571652996901, + 13894.794179234232 + ], + "qps_batch": 15066.571652996901, + "qps_batch_min": 13894.794179234232, + "qps_batch_max": 16414.37768799544, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8921, + "qps_1t_runs": [ + 3336.2436391447904, + 3405.455202962166, + 3154.761232673635 + ], + "qps_1t": 3336.2436391447904, + "qps_1t_min": 3154.761232673635, + "qps_1t_max": 3405.455202962166, + "median_ms": 0.2967999898828566, + "p95_ms": 0.36800500529352576, + "p99_ms": 0.44070303731132293, + "mean_ms": 0.30345533340005204, + "knob": 64, + "batch": { + "recall": 0.8921, + "qps_batch_runs": [ + 12217.441127516431, + 12409.365102200944, + 12346.761257836584 + ], + "qps_batch": 12346.761257836584, + "qps_batch_min": 12217.441127516431, + "qps_batch_max": 12409.365102200944, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9287, + "qps_1t_runs": [ + 2493.9129800374385, + 2317.3413650119046, + 2386.408733701279 + ], + "qps_1t": 2386.408733701279, + "qps_1t_min": 2317.3413650119046, + "qps_1t_max": 2493.9129800374385, + "median_ms": 0.4097500059287995, + "p95_ms": 0.5199100007303059, + "p99_ms": 0.6048340065171934, + "mean_ms": 0.4171816665135945, + "knob": 96, + "batch": { + "recall": 0.9287, + "qps_batch_runs": [ + 8872.60273450348, + 8996.956329540148, + 9008.221804773082 + ], + "qps_batch": 8996.956329540148, + "qps_batch_min": 8872.60273450348, + "qps_batch_max": 9008.221804773082, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9466, + "qps_1t_runs": [ + 1871.961569766797, + 1932.5669401202254, + 1829.5412741974478 + ], + "qps_1t": 1871.961569766797, + "qps_1t_min": 1829.5412741974478, + "qps_1t_max": 1932.5669401202254, + "median_ms": 0.5240499740466475, + "p95_ms": 0.6653200020082294, + "p99_ms": 0.8410870004445293, + "mean_ms": 0.53274353366578, + "knob": 128, + "batch": { + "recall": 0.9466, + "qps_batch_runs": [ + 7083.211444649732, + 6964.646756988021, + 6842.467913280881 + ], + "qps_batch": 6964.646756988021, + "qps_batch_min": 6842.467913280881, + "qps_batch_max": 7083.211444649732, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9668, + "qps_1t_runs": [ + 1350.9095654468174, + 1395.3000987416892, + 1356.6682054117778 + ], + "qps_1t": 1356.6682054117778, + "qps_1t_min": 1350.9095654468174, + "qps_1t_max": 1395.3000987416892, + "median_ms": 0.7292499940376729, + "p95_ms": 0.8789249986875801, + "p99_ms": 1.0260179551551114, + "mean_ms": 0.7313445341036034, + "knob": 192, + "batch": { + "recall": 0.9668, + "qps_batch_runs": [ + 4925.509063846155, + 4967.3989593157485, + 4978.676330353712 + ], + "qps_batch": 4967.3989593157485, + "qps_batch_min": 4925.509063846155, + "qps_batch_max": 4978.676330353712, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9772, + "qps_1t_runs": [ + 1011.9089559196921, + 1117.2719973180424, + 1154.5336343476774 + ], + "qps_1t": 1117.2719973180424, + "qps_1t_min": 1011.9089559196921, + "qps_1t_max": 1154.5336343476774, + "median_ms": 0.905249995412305, + "p95_ms": 1.1718500172719357, + "p99_ms": 1.7185999895446002, + "mean_ms": 0.9164729993402337, + "knob": 256, + "batch": { + "recall": 0.9772, + "qps_batch_runs": [ + 3880.5594838921215, + 3854.5419222586024, + 3676.889645592357 + ], + "qps_batch": 3854.5419222586024, + "qps_batch_min": 3676.889645592357, + "qps_batch_max": 3880.5594838921215, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9885, + "qps_1t_runs": [ + 709.0033354047687, + 699.3332697280132, + 714.3137764283598 + ], + "qps_1t": 709.0033354047687, + "qps_1t_min": 699.3332697280132, + "qps_1t_max": 714.3137764283598, + "median_ms": 1.3901499914936721, + "p95_ms": 1.7939549812581388, + "p99_ms": 2.4693239992484375, + "mean_ms": 1.4134363335130427, + "knob": 400, + "batch": { + "recall": 0.9885, + "qps_batch_runs": [ + 2449.0049815281736, + 2600.632109791398, + 2586.359848446017 + ], + "qps_batch": 2586.359848446017, + "qps_batch_min": 2449.0049815281736, + "qps_batch_max": 2600.632109791398, + "threads": 32 + } + } + ], + "knob_name": "efSearch", + "mem_after_sweep": { + "rss_mb": 1864.03125, + "peak_mb": 2334.94921875 + }, + "ref_knob": 192, + "ref_recall_pre": 0.9668, + "mutation": { + "ref_knob": 192, + "recall_before": 0.9668, + "insert_10k": { + "seconds": 4.261078600015026, + "per_s": 2346.823642249814, + "batch": 1000 + }, + "update_10k": { + "unsupported": true + }, + "delete_10k": { + "unsupported": true + }, + "mem_after": { + "rss_mb": 1901.59375, + "peak_mb": 2334.94921875 + } + }, + "saveload": { + "save_s": 1.1536292000091635, + "disk_mb": 1618.4442081451416, + "load_s": 0.8883937000064179, + "first_query_after_load_ms": 1.081200025510043, + "time_to_ready_s": 0.889474900031928 + }, + "mem_end": { + "rss_mb": 1870.3046875, + "peak_mb": 3490.890625 + }, + "status": "ok", + "finished": "2026-11-16 21:24:00" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_1000000_r1.load.json b/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_1000000_r1.load.json new file mode 100644 index 0000000..d8121b8 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_1000000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 12.85, "pin_cpu_pct": 0.0, "max_cpu_pct": 75.5, "noisy": false, "seconds": 216.3895709514618, "after": 73.8} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_1000000_r1.txt b/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_1000000_r1.txt new file mode 100644 index 0000000..60809d4 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_1000000_r1.txt @@ -0,0 +1,10 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_hnsw --ds text384 --n 1000000 --exp core,mutation,saveload --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_hnsw_text384_1000000_r1.json + faiss_hnsw text384 1000000 knob=24 recall=0.7605 qps1t=6168 med=0.154ms batch=25046 + faiss_hnsw text384 1000000 knob=32 recall=0.8023 qps1t=5394 med=0.177ms batch=20312 + faiss_hnsw text384 1000000 knob=48 recall=0.8589 qps1t=4129 med=0.241ms batch=15067 + faiss_hnsw text384 1000000 knob=64 recall=0.8921 qps1t=3336 med=0.297ms batch=12347 + faiss_hnsw text384 1000000 knob=96 recall=0.9287 qps1t=2386 med=0.410ms batch=8997 + faiss_hnsw text384 1000000 knob=128 recall=0.9466 qps1t=1872 med=0.524ms batch=6965 + faiss_hnsw text384 1000000 knob=192 recall=0.9668 qps1t=1357 med=0.729ms batch=4967 + faiss_hnsw text384 1000000 knob=256 recall=0.9772 qps1t=1117 med=0.905ms batch=3855 + faiss_hnsw text384 1000000 knob=400 recall=0.9885 qps1t=709 med=1.390ms batch=2586 diff --git a/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r1.json b/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r1.json new file mode 100644 index 0000000..dd2e957 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r1.json @@ -0,0 +1,346 @@ +{ + "system": "faiss_hnsw", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 19:18:20", + "mem_baseline": { + "rss_mb": 37.125, + "peak_mb": 37.125 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 12.243231099913828, + "index_s": 4.300032742321491e-06, + "total_s": 12.24323539994657, + "first_query_ms": 1.200400001835078, + "ready_s": 12.244435799948405 + }, + "mem_after_build": { + "rss_mb": 243.80859375, + "peak_mb": 386.07421875 + }, + "mem_after_first_query": { + "rss_mb": 243.97265625, + "peak_mb": 386.07421875 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8103, + "qps_1t_runs": [ + 8716.252820245489, + 8382.391261733796, + 7895.856800974314 + ], + "qps_1t": 8382.391261733796, + "qps_1t_min": 7895.856800974314, + "qps_1t_max": 8716.252820245489, + "median_ms": 0.11985001037828624, + "p95_ms": 0.15949999215081334, + "p99_ms": 0.1920040062395855, + "mean_ms": 0.12022486689966172, + "knob": 24, + "batch": { + "recall": 0.8103, + "qps_batch_runs": [ + 42539.9769295381, + 35835.74325466053, + 37367.392400459416 + ], + "qps_batch": 37367.392400459416, + "qps_batch_min": 35835.74325466053, + "qps_batch_max": 42539.9769295381, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8563, + "qps_1t_runs": [ + 6415.145944212154, + 6446.796479693598, + 6522.042568192641 + ], + "qps_1t": 6446.796479693598, + "qps_1t_min": 6415.145944212154, + "qps_1t_max": 6522.042568192641, + "median_ms": 0.1532500027678907, + "p95_ms": 0.20010501029901204, + "p99_ms": 0.2366379974409929, + "mean_ms": 0.15477436650932455, + "knob": 32, + "batch": { + "recall": 0.8563, + "qps_batch_runs": [ + 29918.173770736557, + 37537.396635386525, + 37718.34211606933 + ], + "qps_batch": 37537.396635386525, + "qps_batch_min": 29918.173770736557, + "qps_batch_max": 37718.34211606933, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9082, + "qps_1t_runs": [ + 4733.064624408997, + 4745.253447173217, + 4684.656974066124 + ], + "qps_1t": 4733.064624408997, + "qps_1t_min": 4684.656974066124, + "qps_1t_max": 4745.253447173217, + "median_ms": 0.21239998750388622, + "p95_ms": 0.26490503805689514, + "p99_ms": 0.3206089965533463, + "mean_ms": 0.21182643358285227, + "knob": 48, + "batch": { + "recall": 0.9082, + "qps_batch_runs": [ + 24081.762394279416, + 25879.58231150823, + 26331.86577176797 + ], + "qps_batch": 25879.58231150823, + "qps_batch_min": 24081.762394279416, + "qps_batch_max": 26331.86577176797, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9363, + "qps_1t_runs": [ + 3985.0544493315842, + 3850.080921395115, + 3925.458679115733 + ], + "qps_1t": 3925.458679115733, + "qps_1t_min": 3850.080921395115, + "qps_1t_max": 3985.0544493315842, + "median_ms": 0.2586999908089638, + "p95_ms": 0.3139050037134439, + "p99_ms": 0.35870199557393784, + "mean_ms": 0.25513990023561445, + "knob": 64, + "batch": { + "recall": 0.9363, + "qps_batch_runs": [ + 18883.423175383832, + 15633.378517538544, + 22793.685233407887 + ], + "qps_batch": 18883.423175383832, + "qps_batch_min": 15633.378517538544, + "qps_batch_max": 22793.685233407887, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.963, + "qps_1t_runs": [ + 3127.1909740742485, + 2966.045014048757, + 2763.3294002932157 + ], + "qps_1t": 2966.045014048757, + "qps_1t_min": 2763.3294002932157, + "qps_1t_max": 3127.1909740742485, + "median_ms": 0.34095000592060387, + "p95_ms": 0.42050500051118433, + "p99_ms": 0.48161500017158654, + "mean_ms": 0.339602467235333, + "knob": 96, + "batch": { + "recall": 0.963, + "qps_batch_runs": [ + 15064.596996250815, + 14638.459325700054, + 14599.222147521072 + ], + "qps_batch": 14638.459325700054, + "qps_batch_min": 14599.222147521072, + "qps_batch_max": 15064.596996250815, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9749, + "qps_1t_runs": [ + 2338.2908257411173, + 2308.9345168083873, + 2237.9996224590946 + ], + "qps_1t": 2308.9345168083873, + "qps_1t_min": 2237.9996224590946, + "qps_1t_max": 2338.2908257411173, + "median_ms": 0.4419499891810119, + "p95_ms": 0.526399991940707, + "p99_ms": 0.5991090211318804, + "mean_ms": 0.4358635329214546, + "knob": 128, + "batch": { + "recall": 0.9749, + "qps_batch_runs": [ + 10881.712515997751, + 11909.752658758143, + 11232.206784037937 + ], + "qps_batch": 11232.206784037937, + "qps_batch_min": 10881.712515997751, + "qps_batch_max": 11909.752658758143, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9865, + "qps_1t_runs": [ + 1474.4539085742244, + 1566.5429025311116, + 1602.375616085308 + ], + "qps_1t": 1566.5429025311116, + "qps_1t_min": 1474.4539085742244, + "qps_1t_max": 1602.375616085308, + "median_ms": 0.653150025755167, + "p95_ms": 0.8102099993266165, + "p99_ms": 0.9768179873935874, + "mean_ms": 0.6468796335393563, + "knob": 192, + "batch": { + "recall": 0.9865, + "qps_batch_runs": [ + 7512.5835777624725, + 8156.467143096187, + 7878.997507412463 + ], + "qps_batch": 7878.997507412463, + "qps_batch_min": 7512.5835777624725, + "qps_batch_max": 8156.467143096187, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9923, + "qps_1t_runs": [ + 1204.6980791115575, + 1249.9400036234538, + 1191.1950185323815 + ], + "qps_1t": 1204.6980791115575, + "qps_1t_min": 1191.1950185323815, + "qps_1t_max": 1249.9400036234538, + "median_ms": 0.8328999974764884, + "p95_ms": 1.015319989528507, + "p99_ms": 1.1078999959863722, + "mean_ms": 0.8232050007403208, + "knob": 256, + "batch": { + "recall": 0.9923, + "qps_batch_runs": [ + 5369.323550150893, + 5647.251229547029, + 6070.720249151719 + ], + "qps_batch": 5647.251229547029, + "qps_batch_min": 5369.323550150893, + "qps_batch_max": 6070.720249151719, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9968, + "qps_1t_runs": [ + 819.9763288328263, + 804.6395511218196, + 850.1614289905878 + ], + "qps_1t": 819.9763288328263, + "qps_1t_min": 804.6395511218196, + "qps_1t_max": 850.1614289905878, + "median_ms": 1.233500021044165, + "p95_ms": 1.48421500925906, + "p99_ms": 1.619923994294367, + "mean_ms": 1.2128623668298435, + "knob": 400, + "batch": { + "recall": 0.9968, + "qps_batch_runs": [ + 3610.0822374506242, + 4243.289873653172, + 4053.719896438919 + ], + "qps_batch": 4053.719896438919, + "qps_batch_min": 3610.0822374506242, + "qps_batch_max": 4243.289873653172, + "threads": 32 + } + } + ], + "knob_name": "efSearch", + "mem_after_sweep": { + "rss_mb": 246.0, + "peak_mb": 386.07421875 + }, + "ref_knob": 96, + "ref_recall_pre": 0.963, + "mutation": { + "ref_knob": 96, + "recall_before": 0.963, + "insert_10k": { + "seconds": 3.2988935000030324, + "per_s": 3031.3194408945933, + "batch": 1000 + }, + "update_10k": { + "unsupported": true + }, + "delete_10k": { + "unsupported": true + }, + "mem_after": { + "rss_mb": 293.734375, + "peak_mb": 386.07421875 + } + }, + "saveload": { + "save_s": 0.2533193999552168, + "disk_mb": 176.2752628326416, + "load_s": 0.1581857000128366, + "first_query_after_load_ms": 1.0424000211060047, + "time_to_ready_s": 0.1592281000339426 + }, + "mem_end": { + "rss_mb": 260.85546875, + "peak_mb": 439.60546875 + }, + "status": "ok", + "finished": "2026-11-16 19:18:52" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r1.load.json b/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r1.load.json new file mode 100644 index 0000000..ea61666 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 13.490625, "pin_cpu_pct": 0.0, "max_cpu_pct": 60.7, "noisy": false, "seconds": 32.44398522377014, "after": 26.1} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r1.txt b/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r1.txt new file mode 100644 index 0000000..6107905 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r1.txt @@ -0,0 +1,10 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_hnsw --ds text384 --n 100000 --exp core,mutation,saveload --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_hnsw_text384_100000_r1.json + faiss_hnsw text384 100000 knob=24 recall=0.8103 qps1t=8382 med=0.120ms batch=37367 + faiss_hnsw text384 100000 knob=32 recall=0.8563 qps1t=6447 med=0.153ms batch=37537 + faiss_hnsw text384 100000 knob=48 recall=0.9082 qps1t=4733 med=0.212ms batch=25880 + faiss_hnsw text384 100000 knob=64 recall=0.9363 qps1t=3925 med=0.259ms batch=18883 + faiss_hnsw text384 100000 knob=96 recall=0.9630 qps1t=2966 med=0.341ms batch=14638 + faiss_hnsw text384 100000 knob=128 recall=0.9749 qps1t=2309 med=0.442ms batch=11232 + faiss_hnsw text384 100000 knob=192 recall=0.9865 qps1t=1567 med=0.653ms batch=7879 + faiss_hnsw text384 100000 knob=256 recall=0.9923 qps1t=1205 med=0.833ms batch=5647 + faiss_hnsw text384 100000 knob=400 recall=0.9968 qps1t=820 med=1.234ms batch=4054 diff --git a/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r2.json b/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r2.json new file mode 100644 index 0000000..f7422d3 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r2.json @@ -0,0 +1,346 @@ +{ + "system": "faiss_hnsw", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 20:57:23", + "mem_baseline": { + "rss_mb": 37.10546875, + "peak_mb": 37.10546875 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 7.602315600030124, + "index_s": 4.900037311017513e-06, + "total_s": 7.602320500067435, + "first_query_ms": 2.1071999799460173, + "ready_s": 7.604427700047381 + }, + "mem_after_build": { + "rss_mb": 243.47265625, + "peak_mb": 383.84375 + }, + "mem_after_first_query": { + "rss_mb": 243.515625, + "peak_mb": 383.84375 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8123, + "qps_1t_runs": [ + 7839.079365376666, + 8210.524415497146, + 9919.444171734638 + ], + "qps_1t": 8210.524415497146, + "qps_1t_min": 7839.079365376666, + "qps_1t_max": 9919.444171734638, + "median_ms": 0.10040000779554248, + "p95_ms": 0.1724000321701169, + "p99_ms": 0.3031459794146929, + "mean_ms": 0.11672433347363646, + "knob": 24, + "batch": { + "recall": 0.8123, + "qps_batch_runs": [ + 56814.63099666172, + 55285.57761813733, + 55580.87580145208 + ], + "qps_batch": 55580.87580145208, + "qps_batch_min": 55285.57761813733, + "qps_batch_max": 56814.63099666172, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8568, + "qps_1t_runs": [ + 8921.688859105661, + 8935.18507713038, + 8775.87895972832 + ], + "qps_1t": 8921.688859105661, + "qps_1t_min": 8775.87895972832, + "qps_1t_max": 8935.18507713038, + "median_ms": 0.10920001659542322, + "p95_ms": 0.14179997669998556, + "p99_ms": 0.1739009853918105, + "mean_ms": 0.1126507338291655, + "knob": 32, + "batch": { + "recall": 0.8568, + "qps_batch_runs": [ + 45063.720110660026, + 43412.38736370706, + 42865.65466938596 + ], + "qps_batch": 43412.38736370706, + "qps_batch_min": 42865.65466938596, + "qps_batch_max": 45063.720110660026, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9075, + "qps_1t_runs": [ + 6362.998885957821, + 6562.661584544603, + 6565.298467575711 + ], + "qps_1t": 6562.661584544603, + "qps_1t_min": 6362.998885957821, + "qps_1t_max": 6565.298467575711, + "median_ms": 0.15075001283548772, + "p95_ms": 0.18970998935401434, + "p99_ms": 0.2395000192336738, + "mean_ms": 0.1539506002639731, + "knob": 48, + "batch": { + "recall": 0.9075, + "qps_batch_runs": [ + 35048.85808033176, + 33920.61223040759, + 34007.01226369304 + ], + "qps_batch": 34007.01226369304, + "qps_batch_min": 33920.61223040759, + "qps_batch_max": 35048.85808033176, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9367, + "qps_1t_runs": [ + 5107.34094703546, + 4702.808365044961, + 4509.7191169359885 + ], + "qps_1t": 4702.808365044961, + "qps_1t_min": 4509.7191169359885, + "qps_1t_max": 5107.34094703546, + "median_ms": 0.19640004029497504, + "p95_ms": 0.26690499507822096, + "p99_ms": 0.3555130289169025, + "mean_ms": 0.21005960072701177, + "knob": 64, + "batch": { + "recall": 0.9367, + "qps_batch_runs": [ + 27048.00753247718, + 27223.181069504786, + 27671.32008287504 + ], + "qps_batch": 27223.181069504786, + "qps_batch_min": 27048.00753247718, + "qps_batch_max": 27671.32008287504, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.964, + "qps_1t_runs": [ + 3345.188679601686, + 3579.0596450414946, + 3788.541790727608 + ], + "qps_1t": 3579.0596450414946, + "qps_1t_min": 3345.188679601686, + "qps_1t_max": 3788.541790727608, + "median_ms": 0.26790000265464187, + "p95_ms": 0.3601999516831711, + "p99_ms": 0.5538109678309409, + "mean_ms": 0.2807645330321975, + "knob": 96, + "batch": { + "recall": 0.964, + "qps_batch_runs": [ + 19631.40080455144, + 19817.755917752424, + 20121.69603539579 + ], + "qps_batch": 19817.755917752424, + "qps_batch_min": 19631.40080455144, + "qps_batch_max": 20121.69603539579, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9752, + "qps_1t_runs": [ + 2818.2438546169697, + 2883.3940961433, + 2750.97336684022 + ], + "qps_1t": 2818.2438546169697, + "qps_1t_min": 2750.97336684022, + "qps_1t_max": 2883.3940961433, + "median_ms": 0.340400030836463, + "p95_ms": 0.45780499349348247, + "p99_ms": 0.6301240238826693, + "mean_ms": 0.3550506996301313, + "knob": 128, + "batch": { + "recall": 0.9752, + "qps_batch_runs": [ + 13252.282372233227, + 13886.285215821783, + 12714.315675514345 + ], + "qps_batch": 13252.282372233227, + "qps_batch_min": 12714.315675514345, + "qps_batch_max": 13886.285215821783, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9861, + "qps_1t_runs": [ + 1934.039954230481, + 1945.0878385786275, + 2036.8672968602411 + ], + "qps_1t": 1945.0878385786275, + "qps_1t_min": 1934.039954230481, + "qps_1t_max": 2036.8672968602411, + "median_ms": 0.4881000204477459, + "p95_ms": 0.6395399803295729, + "p99_ms": 0.9922559862024998, + "mean_ms": 0.5073726665577851, + "knob": 192, + "batch": { + "recall": 0.9861, + "qps_batch_runs": [ + 10604.037809227893, + 10930.873154177023, + 10225.753971637047 + ], + "qps_batch": 10604.037809227893, + "qps_batch_min": 10225.753971637047, + "qps_batch_max": 10930.873154177023, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9922, + "qps_1t_runs": [ + 1441.7720883499906, + 1556.832157495992, + 1518.3796831845787 + ], + "qps_1t": 1518.3796831845787, + "qps_1t_min": 1441.7720883499906, + "qps_1t_max": 1556.832157495992, + "median_ms": 0.638800032902509, + "p95_ms": 0.8241599891334769, + "p99_ms": 1.456407969235433, + "mean_ms": 0.6648392331941674, + "knob": 256, + "batch": { + "recall": 0.9922, + "qps_batch_runs": [ + 8146.447049148625, + 7885.894265775462, + 7919.807198118545 + ], + "qps_batch": 7919.807198118545, + "qps_batch_min": 7885.894265775462, + "qps_batch_max": 8146.447049148625, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9967, + "qps_1t_runs": [ + 1040.1739080265345, + 1005.1744365793191, + 986.063764020801 + ], + "qps_1t": 1005.1744365793191, + "qps_1t_min": 986.063764020801, + "qps_1t_max": 1040.1739080265345, + "median_ms": 0.957000011112541, + "p95_ms": 1.2564100208692246, + "p99_ms": 2.056928004021756, + "mean_ms": 0.9901210335471357, + "knob": 400, + "batch": { + "recall": 0.9967, + "qps_batch_runs": [ + 5809.7380511637275, + 5861.5685096886045, + 5629.0268640960185 + ], + "qps_batch": 5809.7380511637275, + "qps_batch_min": 5629.0268640960185, + "qps_batch_max": 5861.5685096886045, + "threads": 32 + } + } + ], + "knob_name": "efSearch", + "mem_after_sweep": { + "rss_mb": 245.6640625, + "peak_mb": 383.84375 + }, + "ref_knob": 96, + "ref_recall_pre": 0.964, + "mutation": { + "ref_knob": 96, + "recall_before": 0.964, + "insert_10k": { + "seconds": 2.297826600028202, + "per_s": 4351.938479551619, + "batch": 1000 + }, + "update_10k": { + "unsupported": true + }, + "delete_10k": { + "unsupported": true + }, + "mem_after": { + "rss_mb": 292.7421875, + "peak_mb": 383.84375 + } + }, + "saveload": { + "save_s": 0.13419620000058785, + "disk_mb": 176.2752628326416, + "load_s": 0.09073220001300797, + "first_query_after_load_ms": 0.43070001993328333, + "time_to_ready_s": 0.09116290003294125 + }, + "mem_end": { + "rss_mb": 261.0625, + "peak_mb": 439.87109375 + }, + "status": "ok", + "finished": "2026-11-16 20:57:46" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r2.load.json b/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r2.load.json new file mode 100644 index 0000000..98cf12b --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r2.load.json @@ -0,0 +1 @@ +{"total_pct": 13.021875, "pin_cpu_pct": 69.4, "max_cpu_pct": 76.7, "noisy": true, "seconds": 23.252350091934204, "after": 30.3} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r2.txt b/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r2.txt new file mode 100644 index 0000000..976f126 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r2.txt @@ -0,0 +1,10 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_hnsw --ds text384 --n 100000 --exp core,mutation,saveload --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_hnsw_text384_100000_r2.json + faiss_hnsw text384 100000 knob=24 recall=0.8123 qps1t=8211 med=0.100ms batch=55581 + faiss_hnsw text384 100000 knob=32 recall=0.8568 qps1t=8922 med=0.109ms batch=43412 + faiss_hnsw text384 100000 knob=48 recall=0.9075 qps1t=6563 med=0.151ms batch=34007 + faiss_hnsw text384 100000 knob=64 recall=0.9367 qps1t=4703 med=0.196ms batch=27223 + faiss_hnsw text384 100000 knob=96 recall=0.9640 qps1t=3579 med=0.268ms batch=19818 + faiss_hnsw text384 100000 knob=128 recall=0.9752 qps1t=2818 med=0.340ms batch=13252 + faiss_hnsw text384 100000 knob=192 recall=0.9861 qps1t=1945 med=0.488ms batch=10604 + faiss_hnsw text384 100000 knob=256 recall=0.9922 qps1t=1518 med=0.639ms batch=7920 + faiss_hnsw text384 100000 knob=400 recall=0.9967 qps1t=1005 med=0.957ms batch=5810 diff --git a/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r3.json b/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r3.json new file mode 100644 index 0000000..c04b797 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r3.json @@ -0,0 +1,346 @@ +{ + "system": "faiss_hnsw", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 21:02:22", + "mem_baseline": { + "rss_mb": 37.15625, + "peak_mb": 37.15625 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 9.616853700135835, + "index_s": 5.699985194951296e-06, + "total_s": 9.61685940012103, + "first_query_ms": 1.2756000505760312, + "ready_s": 9.618135000171605 + }, + "mem_after_build": { + "rss_mb": 243.88671875, + "peak_mb": 385.40625 + }, + "mem_after_first_query": { + "rss_mb": 244.0546875, + "peak_mb": 385.40625 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8109, + "qps_1t_runs": [ + 10837.876235338203, + 10715.678799937328, + 10967.2332051164 + ], + "qps_1t": 10837.876235338203, + "qps_1t_min": 10715.678799937328, + "qps_1t_max": 10967.2332051164, + "median_ms": 0.0877000275067985, + "p95_ms": 0.12070998782292003, + "p99_ms": 0.16090700286440535, + "mean_ms": 0.09225696643504004, + "knob": 24, + "batch": { + "recall": 0.8109, + "qps_batch_runs": [ + 44924.2352860255, + 49058.325367434234, + 51063.13448248216 + ], + "qps_batch": 49058.325367434234, + "qps_batch_min": 44924.2352860255, + "qps_batch_max": 51063.13448248216, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8575, + "qps_1t_runs": [ + 8944.319733663911, + 9140.784495159416, + 7039.730108952311 + ], + "qps_1t": 8944.319733663911, + "qps_1t_min": 7039.730108952311, + "qps_1t_max": 9140.784495159416, + "median_ms": 0.11030002497136593, + "p95_ms": 0.15071998350322235, + "p99_ms": 0.24571800837293226, + "mean_ms": 0.12108450062805787, + "knob": 32, + "batch": { + "recall": 0.8575, + "qps_batch_runs": [ + 36783.23269945178, + 32536.408265247712, + 35242.911781039795 + ], + "qps_batch": 35242.911781039795, + "qps_batch_min": 32536.408265247712, + "qps_batch_max": 36783.23269945178, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9073, + "qps_1t_runs": [ + 6351.004757989038, + 6362.792448310946, + 5517.000909855754 + ], + "qps_1t": 6351.004757989038, + "qps_1t_min": 5517.000909855754, + "qps_1t_max": 6362.792448310946, + "median_ms": 0.15089998487383127, + "p95_ms": 0.19480997289065266, + "p99_ms": 0.26249101851135276, + "mean_ms": 0.16529233348167813, + "knob": 48, + "batch": { + "recall": 0.9073, + "qps_batch_runs": [ + 27885.679859877775, + 29853.419712156, + 26110.478653492242 + ], + "qps_batch": 27885.679859877775, + "qps_batch_min": 26110.478653492242, + "qps_batch_max": 29853.419712156, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9352, + "qps_1t_runs": [ + 3201.5972160359433, + 4365.456848310397, + 4895.284961027103 + ], + "qps_1t": 4365.456848310397, + "qps_1t_min": 3201.5972160359433, + "qps_1t_max": 4895.284961027103, + "median_ms": 0.20879998919554055, + "p95_ms": 0.4324300331063567, + "p99_ms": 0.8441450243117277, + "mean_ms": 0.248564466368407, + "knob": 64, + "batch": { + "recall": 0.9352, + "qps_batch_runs": [ + 23870.735221867177, + 22432.7913435226, + 24003.898240401937 + ], + "qps_batch": 23870.735221867177, + "qps_batch_min": 22432.7913435226, + "qps_batch_max": 24003.898240401937, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9639, + "qps_1t_runs": [ + 3621.964027212969, + 3683.405408310002, + 3562.917018659141 + ], + "qps_1t": 3621.964027212969, + "qps_1t_min": 3562.917018659141, + "qps_1t_max": 3683.405408310002, + "median_ms": 0.2686000079847872, + "p95_ms": 0.35350501420907676, + "p99_ms": 0.4372130113188174, + "mean_ms": 0.2760833666349451, + "knob": 96, + "batch": { + "recall": 0.9639, + "qps_batch_runs": [ + 16693.515414707883, + 17413.62406627878, + 16421.547696162525 + ], + "qps_batch": 16693.515414707883, + "qps_batch_min": 16421.547696162525, + "qps_batch_max": 17413.62406627878, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.975, + "qps_1t_runs": [ + 2933.4261890795024, + 2807.5526533986367, + 2727.284380794491 + ], + "qps_1t": 2807.5526533986367, + "qps_1t_min": 2727.284380794491, + "qps_1t_max": 2933.4261890795024, + "median_ms": 0.34110003616660833, + "p95_ms": 0.45233495184220357, + "p99_ms": 0.6412759935483319, + "mean_ms": 0.354581833196183, + "knob": 128, + "batch": { + "recall": 0.975, + "qps_batch_runs": [ + 12312.20804763421, + 12429.818136007067, + 12792.369606461913 + ], + "qps_batch": 12429.818136007067, + "qps_batch_min": 12312.20804763421, + "qps_batch_max": 12792.369606461913, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9869, + "qps_1t_runs": [ + 1999.3266209684052, + 1907.9742065266655, + 2067.742986230507 + ], + "qps_1t": 1999.3266209684052, + "qps_1t_min": 1907.9742065266655, + "qps_1t_max": 2067.742986230507, + "median_ms": 0.4880499909631908, + "p95_ms": 0.660145017900504, + "p99_ms": 0.9313319897046305, + "mean_ms": 0.5026345334869499, + "knob": 192, + "batch": { + "recall": 0.9869, + "qps_batch_runs": [ + 9128.960258134528, + 9282.741159396755, + 6882.089845191726 + ], + "qps_batch": 9128.960258134528, + "qps_batch_min": 6882.089845191726, + "qps_batch_max": 9282.741159396755, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9936, + "qps_1t_runs": [ + 1515.4998485286947, + 1599.4098833432129, + 1569.9770144586125 + ], + "qps_1t": 1569.9770144586125, + "qps_1t_min": 1515.4998485286947, + "qps_1t_max": 1599.4098833432129, + "median_ms": 0.6257499917410314, + "p95_ms": 0.8151499816449357, + "p99_ms": 1.0291350359329945, + "mean_ms": 0.6406769668877436, + "knob": 256, + "batch": { + "recall": 0.9936, + "qps_batch_runs": [ + 7366.911222809325, + 7263.825783657629, + 7093.733041350801 + ], + "qps_batch": 7263.825783657629, + "qps_batch_min": 7093.733041350801, + "qps_batch_max": 7366.911222809325, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9971, + "qps_1t_runs": [ + 965.2321455203603, + 959.6717149484357, + 1018.2405573049627 + ], + "qps_1t": 965.2321455203603, + "qps_1t_min": 959.6717149484357, + "qps_1t_max": 1018.2405573049627, + "median_ms": 0.970749999396503, + "p95_ms": 1.3887349807191638, + "p99_ms": 1.91264001186937, + "mean_ms": 1.0200431335057754, + "knob": 400, + "batch": { + "recall": 0.9971, + "qps_batch_runs": [ + 4483.528860006445, + 4200.087109785418, + 3892.9531317103215 + ], + "qps_batch": 4200.087109785418, + "qps_batch_min": 3892.9531317103215, + "qps_batch_max": 4483.528860006445, + "threads": 32 + } + } + ], + "knob_name": "efSearch", + "mem_after_sweep": { + "rss_mb": 246.171875, + "peak_mb": 385.40625 + }, + "ref_knob": 96, + "ref_recall_pre": 0.9639, + "mutation": { + "ref_knob": 96, + "recall_before": 0.9639, + "insert_10k": { + "seconds": 2.245311999984551, + "per_s": 4453.7240259121245, + "batch": 1000 + }, + "update_10k": { + "unsupported": true + }, + "delete_10k": { + "unsupported": true + }, + "mem_after": { + "rss_mb": 295.015625, + "peak_mb": 385.40625 + } + }, + "saveload": { + "save_s": 0.1279149999609217, + "disk_mb": 176.2752628326416, + "load_s": 0.07024149998323992, + "first_query_after_load_ms": 0.46010001096874475, + "time_to_ready_s": 0.07070159999420866 + }, + "mem_end": { + "rss_mb": 260.53125, + "peak_mb": 439.34765625 + }, + "status": "ok", + "finished": "2026-11-16 21:02:48" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r3.load.json b/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r3.load.json new file mode 100644 index 0000000..35a5fc1 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r3.load.json @@ -0,0 +1 @@ +{"total_pct": 13.0875, "pin_cpu_pct": 0.0, "max_cpu_pct": 84.5, "noisy": false, "seconds": 25.59502387046814, "after": 44.3} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r3.txt b/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r3.txt new file mode 100644 index 0000000..bb7d1ee --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_hnsw_text384_100000_r3.txt @@ -0,0 +1,10 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_hnsw --ds text384 --n 100000 --exp core,mutation,saveload --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_hnsw_text384_100000_r3.json + faiss_hnsw text384 100000 knob=24 recall=0.8109 qps1t=10838 med=0.088ms batch=49058 + faiss_hnsw text384 100000 knob=32 recall=0.8575 qps1t=8944 med=0.110ms batch=35243 + faiss_hnsw text384 100000 knob=48 recall=0.9073 qps1t=6351 med=0.151ms batch=27886 + faiss_hnsw text384 100000 knob=64 recall=0.9352 qps1t=4365 med=0.209ms batch=23871 + faiss_hnsw text384 100000 knob=96 recall=0.9639 qps1t=3622 med=0.269ms batch=16694 + faiss_hnsw text384 100000 knob=128 recall=0.9750 qps1t=2808 med=0.341ms batch=12430 + faiss_hnsw text384 100000 knob=192 recall=0.9869 qps1t=1999 med=0.488ms batch=9129 + faiss_hnsw text384 100000 knob=256 recall=0.9936 qps1t=1570 med=0.626ms batch=7264 + faiss_hnsw text384 100000 knob=400 recall=0.9971 qps1t=965 med=0.971ms batch=4200 diff --git a/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_1000000_r1.json b/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_1000000_r1.json new file mode 100644 index 0000000..3a94d08 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_1000000_r1.json @@ -0,0 +1,279 @@ +{ + "system": "faiss_ivfflat", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 20:34:20", + "mem_baseline": { + "rss_mb": 37.0625, + "peak_mb": 37.0625 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 6.429990753531456e-05, + "index_s": 35.34194610000122, + "total_s": 35.34201039990876, + "first_query_ms": 27.773800014983863, + "ready_s": 35.36978419992374 + }, + "mem_after_build": { + "rss_mb": 1918.51171875, + "peak_mb": 3625.3984375 + }, + "mem_after_first_query": { + "rss_mb": 1918.58984375, + "peak_mb": 3625.3984375 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.7032, + "qps_1t_runs": [ + 1597.6380507342117, + 1561.0053350627525, + 1514.4757385097164 + ], + "qps_1t": 1561.0053350627525, + "qps_1t_min": 1514.4757385097164, + "qps_1t_max": 1597.6380507342117, + "median_ms": 0.6442000158131123, + "p95_ms": 0.8068099763477221, + "p99_ms": 0.884603983722627, + "mean_ms": 0.6422771003562957, + "knob": 16, + "batch": { + "recall": 0.7032, + "qps_batch_runs": [ + 4178.875949118008, + 3537.666597530361, + 3668.0744741210897 + ], + "qps_batch": 3668.0744741210897, + "qps_batch_min": 3537.666597530361, + "qps_batch_max": 4178.875949118008, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.7918, + "qps_1t_runs": [ + 905.1543922362266, + 893.3606146623423, + 940.8640593851909 + ], + "qps_1t": 905.1543922362266, + "qps_1t_min": 893.3606146623423, + "qps_1t_max": 940.8640593851909, + "median_ms": 1.0890000266954303, + "p95_ms": 1.38161504000891, + "p99_ms": 1.5582059900043508, + "mean_ms": 1.0956685004833464, + "knob": 32, + "batch": { + "recall": 0.7918, + "qps_batch_runs": [ + 2555.215002295411, + 2713.590066207935, + 2685.0841963886387 + ], + "qps_batch": 2685.0841963886387, + "qps_batch_min": 2555.215002295411, + "qps_batch_max": 2713.590066207935, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8632, + "qps_1t_runs": [ + 534.362901048349, + 567.5138930588728, + 566.8545091690086 + ], + "qps_1t": 566.8545091690086, + "qps_1t_min": 534.362901048349, + "qps_1t_max": 567.5138930588728, + "median_ms": 1.7817000043578446, + "p95_ms": 2.200714993523434, + "p99_ms": 2.4788450152846044, + "mean_ms": 1.799193299782928, + "knob": 64, + "batch": { + "recall": 0.8632, + "qps_batch_runs": [ + 1498.8443161017885, + 1445.7356148534118, + 1491.3782676869575 + ], + "qps_batch": 1491.3782676869575, + "qps_batch_min": 1445.7356148534118, + "qps_batch_max": 1498.8443161017885, + "threads": 32 + } + }, + { + "nq": 918, + "recall_queries": 1000, + "recall": 0.9183000000000001, + "qps_1t_runs": [ + 293.65057399904435, + 282.9343697074633, + 280.85032170407897 + ], + "qps_1t": 282.9343697074633, + "qps_1t_min": 280.85032170407897, + "qps_1t_max": 293.65057399904435, + "median_ms": 3.486350004095584, + "p95_ms": 4.1724099894054225, + "p99_ms": 4.521091970382256, + "mean_ms": 3.50013732785963, + "knob": 128, + "batch": { + "recall": 0.9183, + "qps_batch_runs": [ + 843.0735020311884, + 836.2700349010411, + 814.5620230435063 + ], + "qps_batch": 836.2700349010411, + "qps_batch_min": 814.5620230435063, + "qps_batch_max": 843.0735020311884, + "threads": 32 + } + }, + { + "nq": 499, + "recall_queries": 1000, + "recall": 0.9561, + "qps_1t_runs": [ + 164.5437934132083, + 177.45451424824347, + 144.11471605083742 + ], + "qps_1t": 164.5437934132083, + "qps_1t_min": 144.11471605083742, + "qps_1t_max": 177.45451424824347, + "median_ms": 5.686799995601177, + "p95_ms": 9.414020006079227, + "p99_ms": 15.100476013030843, + "mean_ms": 6.217190981075807, + "knob": 256, + "batch": { + "recall": 0.9561, + "qps_batch_runs": [ + 470.62373693374275, + 490.9173900972781, + 426.2463945489653 + ], + "qps_batch": 470.62373693374275, + "qps_batch_min": 426.2463945489653, + "qps_batch_max": 490.9173900972781, + "threads": 32 + } + }, + { + "nq": 311, + "recall_queries": 1000, + "recall": 0.9801, + "qps_1t_runs": [ + 101.03181420393932, + 100.12943418184089, + 97.89030033115513 + ], + "qps_1t": 100.12943418184089, + "qps_1t_min": 97.89030033115513, + "qps_1t_max": 101.03181420393932, + "median_ms": 9.875899995677173, + "p95_ms": 11.672339995857328, + "p99_ms": 13.559972026851016, + "mean_ms": 10.033487461021695, + "knob": 512, + "batch": { + "recall": 0.9801, + "qps_batch_runs": [ + 256.4958273636069, + 243.65316400821987, + 255.91612830219086 + ], + "qps_batch": 255.91612830219086, + "qps_batch_min": 243.65316400821987, + "qps_batch_max": 256.4958273636069, + "threads": 32 + } + } + ], + "knob_name": "nprobe", + "mem_after_sweep": { + "rss_mb": 1921.21875, + "peak_mb": 3625.3984375 + }, + "ref_knob": 256, + "ref_recall_pre": 0.9561, + "mutation": { + "ref_knob": 256, + "recall_before": 0.9561, + "insert_10k": { + "seconds": 0.11520519998157397, + "per_s": 86801.63743997154, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.1454105000011623, + "per_s": 68770.82466479427, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.008090400020591915, + "per_s": 1236032.8258859534, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 1976.765625, + "peak_mb": 3625.3984375 + }, + "first_query_after_ms": 8.086899993941188, + "after": { + "nq": 564, + "recall_queries": 1000, + "recall": 0.9571000000000001, + "qps_1t_runs": [ + 154.29565676022506, + 158.95662675749293, + 139.9669424741124 + ], + "qps_1t": 154.29565676022506, + "qps_1t_min": 139.9669424741124, + "qps_1t_max": 158.95662675749293, + "median_ms": 6.2579500081483275, + "p95_ms": 9.427580001647579, + "p99_ms": 12.42629600979849, + "mean_ms": 6.638877422912938 + }, + "recall_after": 0.9571000000000001 + }, + "saveload": { + "save_s": 0.4747188999899663, + "disk_mb": 1478.363169670105, + "load_s": 0.8484833000111394, + "first_query_after_load_ms": 5.810700007714331, + "time_to_ready_s": 0.8542940000188537 + }, + "mem_end": { + "rss_mb": 1597.625, + "peak_mb": 3625.3984375 + }, + "status": "ok", + "finished": "2026-11-16 20:36:29" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_1000000_r1.load.json b/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_1000000_r1.load.json new file mode 100644 index 0000000..8543bd2 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_1000000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 8.99375, "pin_cpu_pct": 0.0, "max_cpu_pct": 62.4, "noisy": false, "seconds": 129.4853892326355, "after": 36.5} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_1000000_r1.txt b/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_1000000_r1.txt new file mode 100644 index 0000000..5d50ca1 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_1000000_r1.txt @@ -0,0 +1,7 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_ivfflat --ds text384 --n 1000000 --exp core,mutation,saveload --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_ivfflat_text384_1000000_r1.json + faiss_ivfflat text384 1000000 knob=16 recall=0.7032 qps1t=1561 med=0.644ms batch=3668 + faiss_ivfflat text384 1000000 knob=32 recall=0.7918 qps1t=905 med=1.089ms batch=2685 + faiss_ivfflat text384 1000000 knob=64 recall=0.8632 qps1t=567 med=1.782ms batch=1491 + faiss_ivfflat text384 1000000 knob=128 recall=0.9183 qps1t=283 med=3.486ms batch=836 + faiss_ivfflat text384 1000000 knob=256 recall=0.9561 qps1t=165 med=5.687ms batch=471 + faiss_ivfflat text384 1000000 knob=512 recall=0.9801 qps1t=100 med=9.876ms batch=256 diff --git a/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r1.json b/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r1.json new file mode 100644 index 0000000..426b103 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r1.json @@ -0,0 +1,279 @@ +{ + "system": "faiss_ivfflat", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 19:14:17", + "mem_baseline": { + "rss_mb": 37.078125, + "peak_mb": 37.078125 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 8.90000956133008e-06, + "index_s": 1.3975608999608085, + "total_s": 1.3975697999703698, + "first_query_ms": 5.659500020556152, + "ready_s": 1.403229299990926 + }, + "mem_after_build": { + "rss_mb": 277.21484375, + "peak_mb": 497.83984375 + }, + "mem_after_first_query": { + "rss_mb": 277.29296875, + "peak_mb": 497.83984375 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.7507, + "qps_1t_runs": [ + 7212.1037031922015, + 7235.303656597772, + 7342.127736510907 + ], + "qps_1t": 7235.303656597772, + "qps_1t_min": 7212.1037031922015, + "qps_1t_max": 7342.127736510907, + "median_ms": 0.13529998250305653, + "p95_ms": 0.17060501559171823, + "p99_ms": 0.1911149994703006, + "mean_ms": 0.13768909955009198, + "knob": 16, + "batch": { + "recall": 0.7507, + "qps_batch_runs": [ + 27315.612782366148, + 25817.57815877089, + 28162.667567201323 + ], + "qps_batch": 27315.612782366148, + "qps_batch_min": 25817.57815877089, + "qps_batch_max": 28162.667567201323, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8399, + "qps_1t_runs": [ + 4376.409726276835, + 4349.46943293142, + 4475.65668497461 + ], + "qps_1t": 4376.409726276835, + "qps_1t_min": 4349.46943293142, + "qps_1t_max": 4475.65668497461, + "median_ms": 0.22314998204819858, + "p95_ms": 0.2826999989338219, + "p99_ms": 0.3134059999138115, + "mean_ms": 0.22728060028748587, + "knob": 32, + "batch": { + "recall": 0.8399, + "qps_batch_runs": [ + 10987.537935515056, + 11191.090099523844, + 11647.42317976259 + ], + "qps_batch": 11191.090099523844, + "qps_batch_min": 10987.537935515056, + "qps_batch_max": 11647.42317976259, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9073, + "qps_1t_runs": [ + 1975.3699029326538, + 1825.2325241656986, + 1885.4758209074682 + ], + "qps_1t": 1885.4758209074682, + "qps_1t_min": 1825.2325241656986, + "qps_1t_max": 1975.3699029326538, + "median_ms": 0.5107999895699322, + "p95_ms": 0.6685300264507531, + "p99_ms": 0.8988529758062201, + "mean_ms": 0.5281599335139617, + "knob": 64, + "batch": { + "recall": 0.9073, + "qps_batch_runs": [ + 5986.887520168416, + 6171.814680795845, + 6657.124786448719 + ], + "qps_batch": 6171.814680795845, + "qps_batch_min": 5986.887520168416, + "qps_batch_max": 6657.124786448719, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9553, + "qps_1t_runs": [ + 940.6844808321518, + 1024.7764300653291, + 1033.2214801681378 + ], + "qps_1t": 1024.7764300653291, + "qps_1t_min": 940.6844808321518, + "qps_1t_max": 1033.2214801681378, + "median_ms": 0.969450018601492, + "p95_ms": 1.2805199716240168, + "p99_ms": 1.861223967280231, + "mean_ms": 1.002241666816796, + "knob": 128, + "batch": { + "recall": 0.9553, + "qps_batch_runs": [ + 3562.2608389005477, + 3560.7907092530872, + 3257.84578847919 + ], + "qps_batch": 3560.7907092530872, + "qps_batch_min": 3257.84578847919, + "qps_batch_max": 3562.2608389005477, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9855, + "qps_1t_runs": [ + 482.73535625528024, + 476.0679228329434, + 511.87079540022927 + ], + "qps_1t": 482.73535625528024, + "qps_1t_min": 476.0679228329434, + "qps_1t_max": 511.87079540022927, + "median_ms": 1.944149989867583, + "p95_ms": 2.7398150210501613, + "p99_ms": 4.101471026078782, + "mean_ms": 2.0418956666447534, + "knob": 256, + "batch": { + "recall": 0.9855, + "qps_batch_runs": [ + 1760.8991291246841, + 1734.229953211009, + 1710.5037416319662 + ], + "qps_batch": 1734.229953211009, + "qps_batch_min": 1710.5037416319662, + "qps_batch_max": 1760.8991291246841, + "threads": 32 + } + }, + { + "nq": 792, + "recall_queries": 1000, + "recall": 0.9958, + "qps_1t_runs": [ + 265.8285637380983, + 266.2410395790387, + 267.9749529280147 + ], + "qps_1t": 266.2410395790387, + "qps_1t_min": 265.8285637380983, + "qps_1t_max": 267.9749529280147, + "median_ms": 3.7357000110205263, + "p95_ms": 4.28222501068376, + "p99_ms": 5.220050006755628, + "mean_ms": 3.7498366582685194, + "knob": 512, + "batch": { + "recall": 0.9958, + "qps_batch_runs": [ + 894.5477582488854, + 845.780687717045, + 850.314395233006 + ], + "qps_batch": 850.314395233006, + "qps_batch_min": 845.780687717045, + "qps_batch_max": 894.5477582488854, + "threads": 32 + } + } + ], + "knob_name": "nprobe", + "mem_after_sweep": { + "rss_mb": 270.5078125, + "peak_mb": 497.83984375 + }, + "ref_knob": 128, + "ref_recall_pre": 0.9553, + "mutation": { + "ref_knob": 128, + "recall_before": 0.9553, + "insert_10k": { + "seconds": 0.06276430003345013, + "per_s": 159326.24110633778, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.06528300000354648, + "per_s": 153179.23501457888, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.0037659999798052013, + "per_s": 2655337.2420669147, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 326.40234375, + "peak_mb": 497.83984375 + }, + "first_query_after_ms": 2.1223000367172062, + "after": { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9521, + "qps_1t_runs": [ + 1034.5674874940823, + 1016.7999764273432, + 1032.7495201107515 + ], + "qps_1t": 1032.7495201107515, + "qps_1t_min": 1016.7999764273432, + "qps_1t_max": 1034.5674874940823, + "median_ms": 0.9751000325195491, + "p95_ms": 1.1807549686636774, + "p99_ms": 1.3414379692403566, + "mean_ms": 0.9727846993870722 + }, + "recall_after": 0.9521 + }, + "saveload": { + "save_s": 0.0980316000059247, + "disk_mb": 149.110125541687, + "load_s": 0.16951289999997243, + "first_query_after_load_ms": 1.880299998447299, + "time_to_ready_s": 0.17139319999841973 + }, + "mem_end": { + "rss_mb": 245.8046875, + "peak_mb": 497.83984375 + }, + "status": "ok", + "finished": "2026-11-16 19:14:52" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r1.load.json b/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r1.load.json new file mode 100644 index 0000000..215bda2 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 16.478125, "pin_cpu_pct": 0.0, "max_cpu_pct": 73.9, "noisy": false, "seconds": 34.621156215667725, "after": 22.3} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r1.txt b/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r1.txt new file mode 100644 index 0000000..b71bfa9 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r1.txt @@ -0,0 +1,7 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_ivfflat --ds text384 --n 100000 --exp core,mutation,saveload --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_ivfflat_text384_100000_r1.json + faiss_ivfflat text384 100000 knob=16 recall=0.7507 qps1t=7235 med=0.135ms batch=27316 + faiss_ivfflat text384 100000 knob=32 recall=0.8399 qps1t=4376 med=0.223ms batch=11191 + faiss_ivfflat text384 100000 knob=64 recall=0.9073 qps1t=1885 med=0.511ms batch=6172 + faiss_ivfflat text384 100000 knob=128 recall=0.9553 qps1t=1025 med=0.969ms batch=3561 + faiss_ivfflat text384 100000 knob=256 recall=0.9855 qps1t=483 med=1.944ms batch=1734 + faiss_ivfflat text384 100000 knob=512 recall=0.9958 qps1t=266 med=3.736ms batch=850 diff --git a/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r2.json b/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r2.json new file mode 100644 index 0000000..1df2a49 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r2.json @@ -0,0 +1,279 @@ +{ + "system": "faiss_ivfflat", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 20:56:00", + "mem_baseline": { + "rss_mb": 37.0625, + "peak_mb": 37.0625 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 2.4700071662664413e-05, + "index_s": 1.6120654999976978, + "total_s": 1.6120902000693604, + "first_query_ms": 4.122300015296787, + "ready_s": 1.6162125000846572 + }, + "mem_after_build": { + "rss_mb": 277.08203125, + "peak_mb": 497.70703125 + }, + "mem_after_first_query": { + "rss_mb": 277.16015625, + "peak_mb": 497.70703125 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.7507, + "qps_1t_runs": [ + 5835.296354824729, + 5852.049651374602, + 6308.3801039693735 + ], + "qps_1t": 5852.049651374602, + "qps_1t_min": 5835.296354824729, + "qps_1t_max": 6308.3801039693735, + "median_ms": 0.15380000695586205, + "p95_ms": 0.2568399766460059, + "p99_ms": 0.36852501216344474, + "mean_ms": 0.16692350077209994, + "knob": 16, + "batch": { + "recall": 0.7507, + "qps_batch_runs": [ + 23381.803787881057, + 21765.15397905834, + 23665.446327558846 + ], + "qps_batch": 23381.803787881057, + "qps_batch_min": 21765.15397905834, + "qps_batch_max": 23665.446327558846, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8399, + "qps_1t_runs": [ + 3363.4361004243406, + 3693.052666900648, + 3307.6166059253583 + ], + "qps_1t": 3363.4361004243406, + "qps_1t_min": 3307.6166059253583, + "qps_1t_max": 3693.052666900648, + "median_ms": 0.25209999876096845, + "p95_ms": 0.4609249619534238, + "p99_ms": 0.7462489773752157, + "mean_ms": 0.29014206648571417, + "knob": 32, + "batch": { + "recall": 0.8399, + "qps_batch_runs": [ + 11593.194795742029, + 14118.132060329857, + 13262.82813529818 + ], + "qps_batch": 13262.82813529818, + "qps_batch_min": 11593.194795742029, + "qps_batch_max": 14118.132060329857, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9073, + "qps_1t_runs": [ + 2020.1367287888686, + 2000.6606185083529, + 1743.898404922012 + ], + "qps_1t": 2000.6606185083529, + "qps_1t_min": 1743.898404922012, + "qps_1t_max": 2020.1367287888686, + "median_ms": 0.45984997996129096, + "p95_ms": 0.8517399692209435, + "p99_ms": 1.3322929572313977, + "mean_ms": 0.5227595994559427, + "knob": 64, + "batch": { + "recall": 0.9073, + "qps_batch_runs": [ + 5582.522239127194, + 6897.179606442144, + 6613.708499366687 + ], + "qps_batch": 6613.708499366687, + "qps_batch_min": 5582.522239127194, + "qps_batch_max": 6897.179606442144, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9553, + "qps_1t_runs": [ + 963.4627008509955, + 985.3221482345261, + 986.9211233945241 + ], + "qps_1t": 985.3221482345261, + "qps_1t_min": 963.4627008509955, + "qps_1t_max": 986.9211233945241, + "median_ms": 1.006099977530539, + "p95_ms": 1.3074350019451229, + "p99_ms": 1.754519029636867, + "mean_ms": 1.0220238667873975, + "knob": 128, + "batch": { + "recall": 0.9553, + "qps_batch_runs": [ + 2922.154960727606, + 3280.945385081079, + 2858.021494358823 + ], + "qps_batch": 2922.154960727606, + "qps_batch_min": 2858.021494358823, + "qps_batch_max": 3280.945385081079, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9855, + "qps_1t_runs": [ + 521.2750596013093, + 482.80474041177933, + 443.03651387179195 + ], + "qps_1t": 482.80474041177933, + "qps_1t_min": 443.03651387179195, + "qps_1t_max": 521.2750596013093, + "median_ms": 1.999500032979995, + "p95_ms": 3.0419899907428745, + "p99_ms": 4.134435985470171, + "mean_ms": 2.082251332933083, + "knob": 256, + "batch": { + "recall": 0.9855, + "qps_batch_runs": [ + 1451.9891088784063, + 1336.8260140302195, + 1401.459788582554 + ], + "qps_batch": 1401.459788582554, + "qps_batch_min": 1336.8260140302195, + "qps_batch_max": 1451.9891088784063, + "threads": 32 + } + }, + { + "nq": 752, + "recall_queries": 1000, + "recall": 0.9957999999999999, + "qps_1t_runs": [ + 254.17031989052597, + 252.81994253719353, + 289.8263959574249 + ], + "qps_1t": 254.17031989052597, + "qps_1t_min": 252.81994253719353, + "qps_1t_max": 289.8263959574249, + "median_ms": 3.797799989115447, + "p95_ms": 4.604200003086589, + "p99_ms": 5.719100046553645, + "mean_ms": 3.780031737004339, + "knob": 512, + "batch": { + "recall": 0.9958, + "qps_batch_runs": [ + 1182.1872284369936, + 1080.2466635728674, + 1009.2216620333512 + ], + "qps_batch": 1080.2466635728674, + "qps_batch_min": 1009.2216620333512, + "qps_batch_max": 1182.1872284369936, + "threads": 32 + } + } + ], + "knob_name": "nprobe", + "mem_after_sweep": { + "rss_mb": 268.94921875, + "peak_mb": 497.70703125 + }, + "ref_knob": 128, + "ref_recall_pre": 0.9553, + "mutation": { + "ref_knob": 128, + "recall_before": 0.9553, + "insert_10k": { + "seconds": 0.042410399997606874, + "per_s": 235791.2210345641, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.035157600010279566, + "per_s": 284433.5221140279, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.003720899985637516, + "per_s": 2687521.8464886155, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 324.203125, + "peak_mb": 497.70703125 + }, + "first_query_after_ms": 0.9330000029876828, + "after": { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9521, + "qps_1t_runs": [ + 1156.9039397798206, + 1208.6702735388274, + 1185.2673636122327 + ], + "qps_1t": 1185.2673636122327, + "qps_1t_min": 1156.9039397798206, + "qps_1t_max": 1208.6702735388274, + "median_ms": 0.7995000050868839, + "p95_ms": 1.1663599754683671, + "p99_ms": 1.6491440421668802, + "mean_ms": 0.8451410004830299 + }, + "recall_after": 0.9521 + }, + "saveload": { + "save_s": 0.046081000007689, + "disk_mb": 149.110125541687, + "load_s": 0.09413380001205951, + "first_query_after_load_ms": 0.8945000008679926, + "time_to_ready_s": 0.0950283000129275 + }, + "mem_end": { + "rss_mb": 242.6015625, + "peak_mb": 497.70703125 + }, + "status": "ok", + "finished": "2026-11-16 20:56:34" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r2.load.json b/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r2.load.json new file mode 100644 index 0000000..c1f8ce1 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r2.load.json @@ -0,0 +1 @@ +{"total_pct": 19.15, "pin_cpu_pct": 68.8, "max_cpu_pct": 83.8, "noisy": true, "seconds": 34.137202978134155, "after": 25.5} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r2.txt b/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r2.txt new file mode 100644 index 0000000..663e637 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r2.txt @@ -0,0 +1,7 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_ivfflat --ds text384 --n 100000 --exp core,mutation,saveload --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_ivfflat_text384_100000_r2.json + faiss_ivfflat text384 100000 knob=16 recall=0.7507 qps1t=5852 med=0.154ms batch=23382 + faiss_ivfflat text384 100000 knob=32 recall=0.8399 qps1t=3363 med=0.252ms batch=13263 + faiss_ivfflat text384 100000 knob=64 recall=0.9073 qps1t=2001 med=0.460ms batch=6614 + faiss_ivfflat text384 100000 knob=128 recall=0.9553 qps1t=985 med=1.006ms batch=2922 + faiss_ivfflat text384 100000 knob=256 recall=0.9855 qps1t=483 med=2.000ms batch=1401 + faiss_ivfflat text384 100000 knob=512 recall=0.9958 qps1t=254 med=3.798ms batch=1080 diff --git a/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r3.json b/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r3.json new file mode 100644 index 0000000..b4ef262 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r3.json @@ -0,0 +1,279 @@ +{ + "system": "faiss_ivfflat", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 21:01:37", + "mem_baseline": { + "rss_mb": 37.15234375, + "peak_mb": 37.15234375 + }, + "version": "faiss-cpu 1.9.0", + "build": { + "ingest_s": 9.900075383484364e-06, + "index_s": 1.477275000012014, + "total_s": 1.4772849000873975, + "first_query_ms": 4.45659999968484, + "ready_s": 1.4817415000870824 + }, + "mem_after_build": { + "rss_mb": 279.09765625, + "peak_mb": 499.72265625 + }, + "mem_after_first_query": { + "rss_mb": 279.17578125, + "peak_mb": 499.72265625 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.7507, + "qps_1t_runs": [ + 6404.045333387775, + 6180.771498493684, + 6829.504814436052 + ], + "qps_1t": 6404.045333387775, + "qps_1t_min": 6180.771498493684, + "qps_1t_max": 6829.504814436052, + "median_ms": 0.14920003013685346, + "p95_ms": 0.200405009672977, + "p99_ms": 0.24514300341252135, + "mean_ms": 0.1547889665234834, + "knob": 16, + "batch": { + "recall": 0.7507, + "qps_batch_runs": [ + 24185.55155663921, + 23577.065192106806, + 25837.590075875127 + ], + "qps_batch": 24185.55155663921, + "qps_batch_min": 23577.065192106806, + "qps_batch_max": 25837.590075875127, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8399, + "qps_1t_runs": [ + 3360.217297636323, + 3421.163721002459, + 2704.3677156467606 + ], + "qps_1t": 3360.217297636323, + "qps_1t_min": 2704.3677156467606, + "qps_1t_max": 3421.163721002459, + "median_ms": 0.2741999924182892, + "p95_ms": 0.45695498702116283, + "p99_ms": 0.8960480103269161, + "mean_ms": 0.3198900673499641, + "knob": 32, + "batch": { + "recall": 0.8399, + "qps_batch_runs": [ + 10615.992996461158, + 9320.65475702419, + 9887.685776152743 + ], + "qps_batch": 9887.685776152743, + "qps_batch_min": 9320.65475702419, + "qps_batch_max": 10615.992996461158, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9073, + "qps_1t_runs": [ + 1811.605034865265, + 1860.5333051701461, + 1863.4101717648348 + ], + "qps_1t": 1860.5333051701461, + "qps_1t_min": 1811.605034865265, + "qps_1t_max": 1863.4101717648348, + "median_ms": 0.5305000231601298, + "p95_ms": 0.6628050381550565, + "p99_ms": 0.7416100002592427, + "mean_ms": 0.5420424999319948, + "knob": 64, + "batch": { + "recall": 0.9073, + "qps_batch_runs": [ + 6430.17346015299, + 6003.802808999361, + 5589.8461554666055 + ], + "qps_batch": 6003.802808999361, + "qps_batch_min": 5589.8461554666055, + "qps_batch_max": 6430.17346015299, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9553, + "qps_1t_runs": [ + 995.6039118187756, + 947.3776251973161, + 954.7956762493068 + ], + "qps_1t": 954.7956762493068, + "qps_1t_min": 947.3776251973161, + "qps_1t_max": 995.6039118187756, + "median_ms": 1.0159499943256378, + "p95_ms": 1.333509967662393, + "p99_ms": 1.7624290293315348, + "mean_ms": 1.0357684331247583, + "knob": 128, + "batch": { + "recall": 0.9553, + "qps_batch_runs": [ + 2994.38313656827, + 3014.384340501241, + 3178.7840325515826 + ], + "qps_batch": 3014.384340501241, + "qps_batch_min": 2994.38313656827, + "qps_batch_max": 3178.7840325515826, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9855, + "qps_1t_runs": [ + 485.87200819861215, + 646.7978139112699, + 648.7648650702354 + ], + "qps_1t": 646.7978139112699, + "qps_1t_min": 485.87200819861215, + "qps_1t_max": 648.7648650702354, + "median_ms": 1.5930499939713627, + "p95_ms": 2.2916099871508777, + "p99_ms": 3.391374977072692, + "mean_ms": 1.715207966704232, + "knob": 256, + "batch": { + "recall": 0.9855, + "qps_batch_runs": [ + 2229.3337972043923, + 2205.8795957821667, + 2057.010456145258 + ], + "qps_batch": 2205.8795957821667, + "qps_batch_min": 2057.010456145258, + "qps_batch_max": 2229.3337972043923, + "threads": 32 + } + }, + { + "nq": 973, + "recall_queries": 1000, + "recall": 0.9957999999999999, + "qps_1t_runs": [ + 319.53560211285816, + 318.3537166093594, + 321.8475438529136 + ], + "qps_1t": 319.53560211285816, + "qps_1t_min": 318.3537166093594, + "qps_1t_max": 321.8475438529136, + "median_ms": 3.0120000010356307, + "p95_ms": 3.810090018669143, + "p99_ms": 4.746737970272082, + "mean_ms": 3.125921000599537, + "knob": 512, + "batch": { + "recall": 0.9958, + "qps_batch_runs": [ + 1064.9632933672158, + 1117.7196909306876, + 1125.9596413437375 + ], + "qps_batch": 1117.7196909306876, + "qps_batch_min": 1064.9632933672158, + "qps_batch_max": 1125.9596413437375, + "threads": 32 + } + } + ], + "knob_name": "nprobe", + "mem_after_sweep": { + "rss_mb": 268.8984375, + "peak_mb": 499.72265625 + }, + "ref_knob": 128, + "ref_recall_pre": 0.9553, + "mutation": { + "ref_knob": 128, + "recall_before": 0.9553, + "insert_10k": { + "seconds": 0.04169659997569397, + "per_s": 239827.7079145364, + "batch": 1000 + }, + "update_10k": { + "seconds": 0.03790370002388954, + "per_s": 263826.4864300136, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.0023552999482490122, + "per_s": 4245743.735286984, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 324.53515625, + "peak_mb": 499.72265625 + }, + "first_query_after_ms": 0.9569000103510916, + "after": { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9521, + "qps_1t_runs": [ + 1254.2569466525288, + 1215.1817020110932, + 1278.601353151972 + ], + "qps_1t": 1254.2569466525288, + "qps_1t_min": 1215.1817020110932, + "qps_1t_max": 1278.601353151972, + "median_ms": 0.7813500124029815, + "p95_ms": 1.040960007230751, + "p99_ms": 1.2694150028983127, + "mean_ms": 0.8007705337756003 + }, + "recall_after": 0.9521 + }, + "saveload": { + "save_s": 0.06119580002268776, + "disk_mb": 149.110125541687, + "load_s": 0.08747869997750968, + "first_query_after_load_ms": 1.2355000362731516, + "time_to_ready_s": 0.08871420001378283 + }, + "mem_end": { + "rss_mb": 243.35546875, + "peak_mb": 499.72265625 + }, + "status": "ok", + "finished": "2026-11-16 21:02:09" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r3.load.json b/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r3.load.json new file mode 100644 index 0000000..9428e23 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r3.load.json @@ -0,0 +1 @@ +{"total_pct": 20.61875, "pin_cpu_pct": 72.5, "max_cpu_pct": 72.5, "noisy": true, "seconds": 31.853380918502808, "after": 27.4} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r3.txt b/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r3.txt new file mode 100644 index 0000000..c421642 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/faiss_ivfflat_text384_100000_r3.txt @@ -0,0 +1,7 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system faiss_ivfflat --ds text384 --n 100000 --exp core,mutation,saveload --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\faiss_ivfflat_text384_100000_r3.json + faiss_ivfflat text384 100000 knob=16 recall=0.7507 qps1t=6404 med=0.149ms batch=24186 + faiss_ivfflat text384 100000 knob=32 recall=0.8399 qps1t=3360 med=0.274ms batch=9888 + faiss_ivfflat text384 100000 knob=64 recall=0.9073 qps1t=1861 med=0.531ms batch=6004 + faiss_ivfflat text384 100000 knob=128 recall=0.9553 qps1t=955 med=1.016ms batch=3014 + faiss_ivfflat text384 100000 knob=256 recall=0.9855 qps1t=647 med=1.593ms batch=2206 + faiss_ivfflat text384 100000 knob=512 recall=0.9958 qps1t=320 med=3.012ms batch=1118 diff --git a/benchmarks/rust-stage-c/raw/hnswlib_text384_1000000_r1.json b/benchmarks/rust-stage-c/raw/hnswlib_text384_1000000_r1.json new file mode 100644 index 0000000..fc83dfa --- /dev/null +++ b/benchmarks/rust-stage-c/raw/hnswlib_text384_1000000_r1.json @@ -0,0 +1,369 @@ +{ + "system": "hnswlib", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 20:24:22", + "mem_baseline": { + "rss_mb": 36.97265625, + "peak_mb": 36.97265625 + }, + "version": "hnswlib 0.8.0", + "build": { + "ingest_s": 157.21686190005857, + "index_s": 6.00004568696022e-06, + "total_s": 157.21686790010426, + "first_query_ms": 3.2906000269576907, + "ready_s": 157.22015850013122 + }, + "mem_after_build": { + "rss_mb": 1913.28125, + "peak_mb": 1943.078125 + }, + "mem_after_first_query": { + "rss_mb": 1913.296875, + "peak_mb": 1943.078125 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.7277, + "qps_1t_runs": [ + 5854.6364136352095, + 6005.248592279838, + 6462.82484664481 + ], + "qps_1t": 6005.248592279838, + "qps_1t_min": 5854.6364136352095, + "qps_1t_max": 6462.82484664481, + "median_ms": 0.15669999993406236, + "p95_ms": 0.2259149739984422, + "p99_ms": 0.28410300961695606, + "mean_ms": 0.16401896664562324, + "knob": 24, + "batch": { + "recall": 0.7277, + "qps_batch_runs": [ + 33797.82811535486, + 32233.629323108595, + 34216.694314052496 + ], + "qps_batch": 33797.82811535486, + "qps_batch_min": 32233.629323108595, + "qps_batch_max": 34216.694314052496, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.7742, + "qps_1t_runs": [ + 5495.242209935225, + 6180.989265532001, + 6181.474506096939 + ], + "qps_1t": 6180.989265532001, + "qps_1t_min": 5495.242209935225, + "qps_1t_max": 6181.474506096939, + "median_ms": 0.1634499931242317, + "p95_ms": 0.22432001715060312, + "p99_ms": 0.26880495192017395, + "mean_ms": 0.16851189988665283, + "knob": 32, + "batch": { + "recall": 0.7742, + "qps_batch_runs": [ + 28360.9096123483, + 26605.226333599487, + 26180.272088245074 + ], + "qps_batch": 26605.226333599487, + "qps_batch_min": 26180.272088245074, + "qps_batch_max": 28360.9096123483, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8318, + "qps_1t_runs": [ + 4472.878029045749, + 4540.387880974803, + 4509.170071230823 + ], + "qps_1t": 4509.170071230823, + "qps_1t_min": 4472.878029045749, + "qps_1t_max": 4540.387880974803, + "median_ms": 0.21630001720041037, + "p95_ms": 0.29161500860936934, + "p99_ms": 0.3320030245231464, + "mean_ms": 0.2218618335124726, + "knob": 48, + "batch": { + "recall": 0.8318, + "qps_batch_runs": [ + 20028.56073990702, + 20792.830647115166, + 21376.336805001705 + ], + "qps_batch": 20792.830647115166, + "qps_batch_min": 20028.56073990702, + "qps_batch_max": 21376.336805001705, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8675, + "qps_1t_runs": [ + 3608.9722061464186, + 3412.6198662307584, + 3470.508664454073 + ], + "qps_1t": 3470.508664454073, + "qps_1t_min": 3412.6198662307584, + "qps_1t_max": 3608.9722061464186, + "median_ms": 0.28104998636990786, + "p95_ms": 0.3596050024498254, + "p99_ms": 0.4203069990035145, + "mean_ms": 0.28608646624100703, + "knob": 64, + "batch": { + "recall": 0.8675, + "qps_batch_runs": [ + 17933.038041727814, + 17010.76611069647, + 16875.016874163113 + ], + "qps_batch": 17010.76611069647, + "qps_batch_min": 16875.016874163113, + "qps_batch_max": 17933.038041727814, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9047, + "qps_1t_runs": [ + 2594.6273660253532, + 2659.278834962582, + 2663.707466699627 + ], + "qps_1t": 2659.278834962582, + "qps_1t_min": 2594.6273660253532, + "qps_1t_max": 2663.707466699627, + "median_ms": 0.37560000782832503, + "p95_ms": 0.4811999970115721, + "p99_ms": 0.539512975956313, + "mean_ms": 0.37895673284462345, + "knob": 96, + "batch": { + "recall": 0.9047, + "qps_batch_runs": [ + 12553.33598796274, + 13423.089059723767, + 12680.201508565466 + ], + "qps_batch": 12680.201508565466, + "qps_batch_min": 12553.33598796274, + "qps_batch_max": 13423.089059723767, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9284, + "qps_1t_runs": [ + 1777.5031739555109, + 1772.4211259090282, + 1751.9108484155654 + ], + "qps_1t": 1772.4211259090282, + "qps_1t_min": 1751.9108484155654, + "qps_1t_max": 1777.5031739555109, + "median_ms": 0.5743500078096986, + "p95_ms": 0.669010027195327, + "p99_ms": 0.718500999500975, + "mean_ms": 0.5658640672336332, + "knob": 128, + "batch": { + "recall": 0.9284, + "qps_batch_runs": [ + 10783.112785698459, + 10466.491523848343, + 10696.30827606831 + ], + "qps_batch": 10696.30827606831, + "qps_batch_min": 10466.491523848343, + "qps_batch_max": 10783.112785698459, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9552, + "qps_1t_runs": [ + 1395.9498748922238, + 1468.9464709248714, + 1460.7658745710928 + ], + "qps_1t": 1460.7658745710928, + "qps_1t_min": 1395.9498748922238, + "qps_1t_max": 1468.9464709248714, + "median_ms": 0.6882500019855797, + "p95_ms": 0.8941049978602678, + "p99_ms": 0.985810997080989, + "mean_ms": 0.693896833283361, + "knob": 192, + "batch": { + "recall": 0.9552, + "qps_batch_runs": [ + 5978.315454706129, + 6679.834845899164, + 6946.962719341702 + ], + "qps_batch": 6679.834845899164, + "qps_batch_min": 5978.315454706129, + "qps_batch_max": 6946.962719341702, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9671, + "qps_1t_runs": [ + 1141.8305397171505, + 1136.4502909011094, + 1104.803298442604 + ], + "qps_1t": 1136.4502909011094, + "qps_1t_min": 1104.803298442604, + "qps_1t_max": 1141.8305397171505, + "median_ms": 0.8868500008247793, + "p95_ms": 1.1122999741928652, + "p99_ms": 1.2026209605392064, + "mean_ms": 0.8869526997441426, + "knob": 256, + "batch": { + "recall": 0.9671, + "qps_batch_runs": [ + 5340.314192149759, + 5134.674818509389, + 5185.77252202955 + ], + "qps_batch": 5185.77252202955, + "qps_batch_min": 5134.674818509389, + "qps_batch_max": 5340.314192149759, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9817, + "qps_1t_runs": [ + 685.3328961499535, + 763.5149205478426, + 760.0760047920522 + ], + "qps_1t": 760.0760047920522, + "qps_1t_min": 685.3328961499535, + "qps_1t_max": 763.5149205478426, + "median_ms": 1.3470000121742487, + "p95_ms": 1.7182999872602522, + "p99_ms": 1.8782999838003889, + "mean_ms": 1.3615116329747252, + "knob": 400, + "batch": { + "recall": 0.9817, + "qps_batch_runs": [ + 3628.3390697662985, + 3785.4927801482013, + 3514.343088837003 + ], + "qps_batch": 3628.3390697662985, + "qps_batch_min": 3514.343088837003, + "qps_batch_max": 3785.4927801482013, + "threads": 32 + } + } + ], + "knob_name": "ef", + "mem_after_sweep": { + "rss_mb": 1915.05078125, + "peak_mb": 1943.078125 + }, + "ref_knob": 192, + "ref_recall_pre": 0.9552, + "mutation": { + "ref_knob": 192, + "recall_before": 0.9552, + "insert_10k": { + "seconds": 1.6653583000297658, + "per_s": 6004.713820335999, + "batch": 1000 + }, + "update_10k": { + "seconds": 5.8976930000353605, + "per_s": 1695.5782540630792, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.02812989999074489, + "per_s": 355493.62078393926, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 1963.1640625, + "peak_mb": 1963.29296875 + }, + "first_query_after_ms": 1.2923000031150877, + "after": { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9557, + "qps_1t_runs": [ + 1329.4280035445167, + 1409.8597698776011, + 1376.7325311634274 + ], + "qps_1t": 1376.7325311634274, + "qps_1t_min": 1329.4280035445167, + "qps_1t_max": 1409.8597698776011, + "median_ms": 0.7242999854497612, + "p95_ms": 0.9323149977717547, + "p99_ms": 1.0271070210728792, + "mean_ms": 0.729283700483696 + }, + "recall_after": 0.9557 + }, + "saveload": { + "save_s": 2.5292776999995112, + "disk_mb": 1626.394660949707, + "load_s": 6.047899500001222, + "first_query_after_load_ms": 3.0805999995209277, + "time_to_ready_s": 6.050980100000743 + }, + "mem_end": { + "rss_mb": 1824.36328125, + "peak_mb": 1963.29296875 + }, + "status": "ok", + "finished": "2026-11-16 20:27:36" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/hnswlib_text384_1000000_r1.load.json b/benchmarks/rust-stage-c/raw/hnswlib_text384_1000000_r1.load.json new file mode 100644 index 0000000..f089d66 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/hnswlib_text384_1000000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 8.496875, "pin_cpu_pct": 0.0, "max_cpu_pct": 56.3, "noisy": false, "seconds": 195.25289630889893, "after": 42.8} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/hnswlib_text384_1000000_r1.txt b/benchmarks/rust-stage-c/raw/hnswlib_text384_1000000_r1.txt new file mode 100644 index 0000000..2fa79e6 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/hnswlib_text384_1000000_r1.txt @@ -0,0 +1,10 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system hnswlib --ds text384 --n 1000000 --exp core,mutation,saveload --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\hnswlib_text384_1000000_r1.json + hnswlib text384 1000000 knob=24 recall=0.7277 qps1t=6005 med=0.157ms batch=33798 + hnswlib text384 1000000 knob=32 recall=0.7742 qps1t=6181 med=0.163ms batch=26605 + hnswlib text384 1000000 knob=48 recall=0.8318 qps1t=4509 med=0.216ms batch=20793 + hnswlib text384 1000000 knob=64 recall=0.8675 qps1t=3471 med=0.281ms batch=17011 + hnswlib text384 1000000 knob=96 recall=0.9047 qps1t=2659 med=0.376ms batch=12680 + hnswlib text384 1000000 knob=128 recall=0.9284 qps1t=1772 med=0.574ms batch=10696 + hnswlib text384 1000000 knob=192 recall=0.9552 qps1t=1461 med=0.688ms batch=6680 + hnswlib text384 1000000 knob=256 recall=0.9671 qps1t=1136 med=0.887ms batch=5186 + hnswlib text384 1000000 knob=400 recall=0.9817 qps1t=760 med=1.347ms batch=3628 diff --git a/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r1.json b/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r1.json new file mode 100644 index 0000000..17a85d6 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r1.json @@ -0,0 +1,369 @@ +{ + "system": "hnswlib", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 19:19:59", + "mem_baseline": { + "rss_mb": 37.08984375, + "peak_mb": 37.08984375 + }, + "version": "hnswlib 0.8.0", + "build": { + "ingest_s": 8.380472799995914, + "index_s": 4.00003045797348e-06, + "total_s": 8.380476800026372, + "first_query_ms": 1.8638999899849296, + "ready_s": 8.382340700016357 + }, + "mem_after_build": { + "rss_mb": 237.8671875, + "peak_mb": 267.65234375 + }, + "mem_after_first_query": { + "rss_mb": 237.8828125, + "peak_mb": 267.65234375 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.7846, + "qps_1t_runs": [ + 7952.627800031688, + 7765.7718972285475, + 7518.638689246043 + ], + "qps_1t": 7765.7718972285475, + "qps_1t_min": 7518.638689246043, + "qps_1t_max": 7952.627800031688, + "median_ms": 0.12330000754445791, + "p95_ms": 0.1653999788686633, + "p99_ms": 0.19913500233087605, + "mean_ms": 0.12917253334308043, + "knob": 24, + "batch": { + "recall": 0.7846, + "qps_batch_runs": [ + 39184.79948288626, + 38887.6574511105, + 37758.78925920335 + ], + "qps_batch": 38887.6574511105, + "qps_batch_min": 37758.78925920335, + "qps_batch_max": 39184.79948288626, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8282, + "qps_1t_runs": [ + 6152.40370120925, + 5669.898520010631, + 6223.910121178681 + ], + "qps_1t": 6152.40370120925, + "qps_1t_min": 5669.898520010631, + "qps_1t_max": 6223.910121178681, + "median_ms": 0.16080000204965472, + "p95_ms": 0.22661497932858757, + "p99_ms": 0.2843260113149875, + "mean_ms": 0.16652626633488884, + "knob": 32, + "batch": { + "recall": 0.8282, + "qps_batch_runs": [ + 32235.70748662482, + 31683.871020987044, + 29502.006129368354 + ], + "qps_batch": 31683.871020987044, + "qps_batch_min": 29502.006129368354, + "qps_batch_max": 32235.70748662482, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8841, + "qps_1t_runs": [ + 4772.953029252453, + 4887.270221263815, + 4743.860012993001 + ], + "qps_1t": 4772.953029252453, + "qps_1t_min": 4743.860012993001, + "qps_1t_max": 4887.270221263815, + "median_ms": 0.20539999241009355, + "p95_ms": 0.2600050007458776, + "p99_ms": 0.3036409837659439, + "mean_ms": 0.20830863326167068, + "knob": 48, + "batch": { + "recall": 0.8841, + "qps_batch_runs": [ + 22085.683614956375, + 25544.350095614434, + 24315.399916587638 + ], + "qps_batch": 24315.399916587638, + "qps_batch_min": 22085.683614956375, + "qps_batch_max": 25544.350095614434, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9153, + "qps_1t_runs": [ + 3936.186558004301, + 3660.419908271403, + 3842.8356945547516 + ], + "qps_1t": 3842.8356945547516, + "qps_1t_min": 3660.419908271403, + "qps_1t_max": 3936.186558004301, + "median_ms": 0.2649000089149922, + "p95_ms": 0.3237049619201571, + "p99_ms": 0.3604070085566489, + "mean_ms": 0.26249006679669645, + "knob": 64, + "batch": { + "recall": 0.9153, + "qps_batch_runs": [ + 19840.994266910293, + 21170.473139327612, + 20936.050823525344 + ], + "qps_batch": 20936.050823525344, + "qps_batch_min": 19840.994266910293, + "qps_batch_max": 21170.473139327612, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9486, + "qps_1t_runs": [ + 2738.948620423618, + 2789.2851310392243, + 2817.508566014093 + ], + "qps_1t": 2789.2851310392243, + "qps_1t_min": 2738.948620423618, + "qps_1t_max": 2817.508566014093, + "median_ms": 0.36279999767430127, + "p95_ms": 0.44160498946439475, + "p99_ms": 0.4882249922957265, + "mean_ms": 0.3595139661920257, + "knob": 96, + "batch": { + "recall": 0.9486, + "qps_batch_runs": [ + 15474.53432378325, + 18017.303826602958, + 17613.07594380563 + ], + "qps_batch": 17613.07594380563, + "qps_batch_min": 15474.53432378325, + "qps_batch_max": 18017.303826602958, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9631, + "qps_1t_runs": [ + 2229.042601252933, + 2150.1926800579904, + 2175.4524153378425 + ], + "qps_1t": 2175.4524153378425, + "qps_1t_min": 2150.1926800579904, + "qps_1t_max": 2229.042601252933, + "median_ms": 0.4549499717541039, + "p95_ms": 0.5534350202651692, + "p99_ms": 0.6623129796935242, + "mean_ms": 0.4577907327717791, + "knob": 128, + "batch": { + "recall": 0.9631, + "qps_batch_runs": [ + 12607.033712274984, + 12650.84551465253, + 12522.54057812023 + ], + "qps_batch": 12607.033712274984, + "qps_batch_min": 12522.54057812023, + "qps_batch_max": 12650.84551465253, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9798, + "qps_1t_runs": [ + 1570.4163716064475, + 1574.6848610428378, + 1571.8764880241936 + ], + "qps_1t": 1571.8764880241936, + "qps_1t_min": 1570.4163716064475, + "qps_1t_max": 1574.6848610428378, + "median_ms": 0.6351500051096082, + "p95_ms": 0.7644049968803301, + "p99_ms": 0.8649090083781628, + "mean_ms": 0.6360012668301351, + "knob": 192, + "batch": { + "recall": 0.9798, + "qps_batch_runs": [ + 9178.699175765238, + 8910.019495592653, + 8653.131435140265 + ], + "qps_batch": 8910.019495592653, + "qps_batch_min": 8653.131435140265, + "qps_batch_max": 9178.699175765238, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9863, + "qps_1t_runs": [ + 1273.4848114409926, + 1243.7374717322555, + 1252.1802032050693 + ], + "qps_1t": 1252.1802032050693, + "qps_1t_min": 1243.7374717322555, + "qps_1t_max": 1273.4848114409926, + "median_ms": 0.7979999936651438, + "p95_ms": 0.9664100245572627, + "p99_ms": 1.1038080224534494, + "mean_ms": 0.7959607328521088, + "knob": 256, + "batch": { + "recall": 0.9863, + "qps_batch_runs": [ + 7220.1801142867225, + 7096.285235332971, + 7260.118425705319 + ], + "qps_batch": 7220.1801142867225, + "qps_batch_min": 7096.285235332971, + "qps_batch_max": 7260.118425705319, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9928, + "qps_1t_runs": [ + 846.3899566234097, + 863.1680911392119, + 864.5384150673176 + ], + "qps_1t": 863.1680911392119, + "qps_1t_min": 846.3899566234097, + "qps_1t_max": 864.5384150673176, + "median_ms": 1.1739000328816473, + "p95_ms": 1.4091250195633618, + "p99_ms": 1.544650945579632, + "mean_ms": 1.1655659998844687, + "knob": 400, + "batch": { + "recall": 0.9928, + "qps_batch_runs": [ + 4938.103343436707, + 4747.032985152161, + 4480.039854738019 + ], + "qps_batch": 4747.032985152161, + "qps_batch_min": 4480.039854738019, + "qps_batch_max": 4938.103343436707, + "threads": 32 + } + } + ], + "knob_name": "ef", + "mem_after_sweep": { + "rss_mb": 241.01953125, + "peak_mb": 267.65234375 + }, + "ref_knob": 128, + "ref_recall_pre": 0.9631, + "mutation": { + "ref_knob": 128, + "recall_before": 0.9631, + "insert_10k": { + "seconds": 1.7906052999896929, + "per_s": 5584.703675375898, + "batch": 1000 + }, + "update_10k": { + "seconds": 6.081552499963436, + "per_s": 1644.3169733485195, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.026664000004529953, + "per_s": 375037.50368665985, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 286.9765625, + "peak_mb": 289.2578125 + }, + "first_query_after_ms": 0.9597000316716731, + "after": { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9667, + "qps_1t_runs": [ + 1833.2417343604654, + 1731.5144796828454, + 1806.8828858088923 + ], + "qps_1t": 1806.8828858088923, + "qps_1t_min": 1731.5144796828454, + "qps_1t_max": 1833.2417343604654, + "median_ms": 0.5528500187210739, + "p95_ms": 0.7066350255627183, + "p99_ms": 0.9801389544736585, + "mean_ms": 0.5588167334014239 + }, + "recall_after": 0.9667 + }, + "saveload": { + "save_s": 0.28984460001811385, + "disk_mb": 177.13692474365234, + "load_s": 0.5723628000123426, + "first_query_after_load_ms": 0.9837999823503196, + "time_to_ready_s": 0.573346599994693 + }, + "mem_end": { + "rss_mb": 241.9453125, + "peak_mb": 289.2578125 + }, + "status": "ok", + "finished": "2026-11-16 19:20:34" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r1.load.json b/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r1.load.json new file mode 100644 index 0000000..d8127a0 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 14.6625, "pin_cpu_pct": 0.0, "max_cpu_pct": 65.6, "noisy": false, "seconds": 34.844477891922, "after": 23.9} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r1.txt b/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r1.txt new file mode 100644 index 0000000..1c0cd8d --- /dev/null +++ b/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r1.txt @@ -0,0 +1,10 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system hnswlib --ds text384 --n 100000 --exp core,mutation,saveload --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\hnswlib_text384_100000_r1.json + hnswlib text384 100000 knob=24 recall=0.7846 qps1t=7766 med=0.123ms batch=38888 + hnswlib text384 100000 knob=32 recall=0.8282 qps1t=6152 med=0.161ms batch=31684 + hnswlib text384 100000 knob=48 recall=0.8841 qps1t=4773 med=0.205ms batch=24315 + hnswlib text384 100000 knob=64 recall=0.9153 qps1t=3843 med=0.265ms batch=20936 + hnswlib text384 100000 knob=96 recall=0.9486 qps1t=2789 med=0.363ms batch=17613 + hnswlib text384 100000 knob=128 recall=0.9631 qps1t=2175 med=0.455ms batch=12607 + hnswlib text384 100000 knob=192 recall=0.9798 qps1t=1572 med=0.635ms batch=8910 + hnswlib text384 100000 knob=256 recall=0.9863 qps1t=1252 med=0.798ms batch=7220 + hnswlib text384 100000 knob=400 recall=0.9928 qps1t=863 med=1.174ms batch=4747 diff --git a/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r2.json b/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r2.json new file mode 100644 index 0000000..781bf0d --- /dev/null +++ b/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r2.json @@ -0,0 +1,369 @@ +{ + "system": "hnswlib", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 19:40:18", + "mem_baseline": { + "rss_mb": 37.12890625, + "peak_mb": 37.12890625 + }, + "version": "hnswlib 0.8.0", + "build": { + "ingest_s": 8.737835600040853, + "index_s": 5.299982149153948e-06, + "total_s": 8.737840900023002, + "first_query_ms": 2.091500035021454, + "ready_s": 8.739932400058024 + }, + "mem_after_build": { + "rss_mb": 238.00390625, + "peak_mb": 267.80078125 + }, + "mem_after_first_query": { + "rss_mb": 238.01953125, + "peak_mb": 267.80078125 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.7875, + "qps_1t_runs": [ + 7788.252986583955, + 7666.6010062064715, + 8291.172705070932 + ], + "qps_1t": 7788.252986583955, + "qps_1t_min": 7666.6010062064715, + "qps_1t_max": 8291.172705070932, + "median_ms": 0.1238000113517046, + "p95_ms": 0.1676099956966936, + "p99_ms": 0.2011119597591457, + "mean_ms": 0.1264815332639652, + "knob": 24, + "batch": { + "recall": 0.7875, + "qps_batch_runs": [ + 39322.087236037296, + 38926.40956989728, + 39775.66526227015 + ], + "qps_batch": 39322.087236037296, + "qps_batch_min": 38926.40956989728, + "qps_batch_max": 39775.66526227015, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8317, + "qps_1t_runs": [ + 6513.066911555399, + 7048.895352079459, + 6857.003372339052 + ], + "qps_1t": 6857.003372339052, + "qps_1t_min": 6513.066911555399, + "qps_1t_max": 7048.895352079459, + "median_ms": 0.14220000593923032, + "p95_ms": 0.1866999693447724, + "p99_ms": 0.22501002473290985, + "mean_ms": 0.14707999950041994, + "knob": 32, + "batch": { + "recall": 0.8317, + "qps_batch_runs": [ + 36557.5910512413, + 35621.16178851807, + 34672.500899569546 + ], + "qps_batch": 35621.16178851807, + "qps_batch_min": 34672.500899569546, + "qps_batch_max": 36557.5910512413, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8846, + "qps_1t_runs": [ + 4310.513886342169, + 4134.9823698629825, + 4620.899125754948 + ], + "qps_1t": 4310.513886342169, + "qps_1t_min": 4134.9823698629825, + "qps_1t_max": 4620.899125754948, + "median_ms": 0.20494998898357153, + "p95_ms": 0.268414948368445, + "p99_ms": 0.343010992510244, + "mean_ms": 0.23007933353073895, + "knob": 48, + "batch": { + "recall": 0.8846, + "qps_batch_runs": [ + 25326.00907829059, + 24589.901914142065, + 24499.895894614736 + ], + "qps_batch": 24589.901914142065, + "qps_batch_min": 24499.895894614736, + "qps_batch_max": 25326.00907829059, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9139, + "qps_1t_runs": [ + 3886.263087943695, + 3888.8867262670906, + 3784.631751660809 + ], + "qps_1t": 3886.263087943695, + "qps_1t_min": 3784.631751660809, + "qps_1t_max": 3888.8867262670906, + "median_ms": 0.2592999953776598, + "p95_ms": 0.32240001019090414, + "p99_ms": 0.3617339604534201, + "mean_ms": 0.2595620328599277, + "knob": 64, + "batch": { + "recall": 0.9139, + "qps_batch_runs": [ + 20712.295832911746, + 21332.696218444966, + 20767.26746010941 + ], + "qps_batch": 20767.26746010941, + "qps_batch_min": 20712.295832911746, + "qps_batch_max": 21332.696218444966, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9483, + "qps_1t_runs": [ + 2828.1478663535586, + 2837.789131088435, + 2787.805356347573 + ], + "qps_1t": 2828.1478663535586, + "qps_1t_min": 2787.805356347573, + "qps_1t_max": 2837.789131088435, + "median_ms": 0.35654997918754816, + "p95_ms": 0.43391504441387946, + "p99_ms": 0.49220796441659315, + "mean_ms": 0.35489346738904715, + "knob": 96, + "batch": { + "recall": 0.9483, + "qps_batch_runs": [ + 16298.935515815998, + 17045.212415006547, + 15600.453669732286 + ], + "qps_batch": 16298.935515815998, + "qps_batch_min": 15600.453669732286, + "qps_batch_max": 17045.212415006547, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9631, + "qps_1t_runs": [ + 2295.4248580929925, + 2222.912807734886, + 2225.883059799709 + ], + "qps_1t": 2225.883059799709, + "qps_1t_min": 2222.912807734886, + "qps_1t_max": 2295.4248580929925, + "median_ms": 0.44144998537376523, + "p95_ms": 0.535910006146878, + "p99_ms": 0.6450669636251389, + "mean_ms": 0.44492309938262525, + "knob": 128, + "batch": { + "recall": 0.9631, + "qps_batch_runs": [ + 12771.701989206096, + 12756.61526405343, + 12536.889793785815 + ], + "qps_batch": 12756.61526405343, + "qps_batch_min": 12536.889793785815, + "qps_batch_max": 12771.701989206096, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.98, + "qps_1t_runs": [ + 1612.1769643527869, + 1620.6914327775664, + 1595.781264963553 + ], + "qps_1t": 1612.1769643527869, + "qps_1t_min": 1595.781264963553, + "qps_1t_max": 1620.6914327775664, + "median_ms": 0.6231500010471791, + "p95_ms": 0.7531000301241875, + "p99_ms": 0.8220099727623162, + "mean_ms": 0.6213174001701797, + "knob": 192, + "batch": { + "recall": 0.98, + "qps_batch_runs": [ + 9211.088676929317, + 9005.852900663962, + 9089.19867153652 + ], + "qps_batch": 9089.19867153652, + "qps_batch_min": 9005.852900663962, + "qps_batch_max": 9211.088676929317, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9861, + "qps_1t_runs": [ + 1244.3077581197833, + 1232.5377139383431, + 1243.151169837408 + ], + "qps_1t": 1243.151169837408, + "qps_1t_min": 1232.5377139383431, + "qps_1t_max": 1244.3077581197833, + "median_ms": 0.8081999840214849, + "p95_ms": 0.9893349662888793, + "p99_ms": 1.0731150215724479, + "mean_ms": 0.8064670999883674, + "knob": 256, + "batch": { + "recall": 0.9861, + "qps_batch_runs": [ + 6759.401143580954, + 6626.294860965274, + 6136.482738272839 + ], + "qps_batch": 6626.294860965274, + "qps_batch_min": 6136.482738272839, + "qps_batch_max": 6759.401143580954, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9929, + "qps_1t_runs": [ + 774.8041130624827, + 812.9795107762754, + 825.5970085186705 + ], + "qps_1t": 812.9795107762754, + "qps_1t_min": 774.8041130624827, + "qps_1t_max": 825.5970085186705, + "median_ms": 1.2487000203691423, + "p95_ms": 1.5149950078921388, + "p99_ms": 1.6898110025795174, + "mean_ms": 1.243978933392403, + "knob": 400, + "batch": { + "recall": 0.9929, + "qps_batch_runs": [ + 4518.1178778277135, + 4858.123364870439, + 4686.635169912454 + ], + "qps_batch": 4686.635169912454, + "qps_batch_min": 4518.1178778277135, + "qps_batch_max": 4858.123364870439, + "threads": 32 + } + } + ], + "knob_name": "ef", + "mem_after_sweep": { + "rss_mb": 240.75, + "peak_mb": 267.80078125 + }, + "ref_knob": 128, + "ref_recall_pre": 0.9631, + "mutation": { + "ref_knob": 128, + "recall_before": 0.9631, + "insert_10k": { + "seconds": 1.5532053000060841, + "per_s": 6438.298916415511, + "batch": 1000 + }, + "update_10k": { + "seconds": 6.382411600032356, + "per_s": 1566.8058763162978, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.04076640005223453, + "per_s": 245300.0507081044, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 287.0625, + "peak_mb": 289.203125 + }, + "first_query_after_ms": 1.2799000251106918, + "after": { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9675, + "qps_1t_runs": [ + 1779.633343935053, + 1834.422796483146, + 1736.1840551568182 + ], + "qps_1t": 1779.633343935053, + "qps_1t_min": 1736.1840551568182, + "qps_1t_max": 1834.422796483146, + "median_ms": 0.5646500212606043, + "p95_ms": 0.6845049734693021, + "p99_ms": 0.7503030088264494, + "mean_ms": 0.5610066331767788 + }, + "recall_after": 0.9675 + }, + "saveload": { + "save_s": 0.3665863000205718, + "disk_mb": 177.13251495361328, + "load_s": 0.5946352999890223, + "first_query_after_load_ms": 1.0158999939449131, + "time_to_ready_s": 0.5956511999829672 + }, + "mem_end": { + "rss_mb": 241.9921875, + "peak_mb": 289.203125 + }, + "status": "ok", + "finished": "2026-11-16 19:40:54" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r2.load.json b/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r2.load.json new file mode 100644 index 0000000..683a888 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r2.load.json @@ -0,0 +1 @@ +{"total_pct": 20.628125, "pin_cpu_pct": 0.0, "max_cpu_pct": 75.6, "noisy": false, "seconds": 35.61854791641235, "after": 37.6} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r2.txt b/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r2.txt new file mode 100644 index 0000000..6ec2581 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r2.txt @@ -0,0 +1,10 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system hnswlib --ds text384 --n 100000 --exp core,mutation,saveload --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\hnswlib_text384_100000_r2.json + hnswlib text384 100000 knob=24 recall=0.7875 qps1t=7788 med=0.124ms batch=39322 + hnswlib text384 100000 knob=32 recall=0.8317 qps1t=6857 med=0.142ms batch=35621 + hnswlib text384 100000 knob=48 recall=0.8846 qps1t=4311 med=0.205ms batch=24590 + hnswlib text384 100000 knob=64 recall=0.9139 qps1t=3886 med=0.259ms batch=20767 + hnswlib text384 100000 knob=96 recall=0.9483 qps1t=2828 med=0.357ms batch=16299 + hnswlib text384 100000 knob=128 recall=0.9631 qps1t=2226 med=0.441ms batch=12757 + hnswlib text384 100000 knob=192 recall=0.9800 qps1t=1612 med=0.623ms batch=9089 + hnswlib text384 100000 knob=256 recall=0.9861 qps1t=1243 med=0.808ms batch=6626 + hnswlib text384 100000 knob=400 recall=0.9929 qps1t=813 med=1.249ms batch=4687 diff --git a/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r3.json b/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r3.json new file mode 100644 index 0000000..b5a5848 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r3.json @@ -0,0 +1,369 @@ +{ + "system": "hnswlib", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 19:50:16", + "mem_baseline": { + "rss_mb": 36.92578125, + "peak_mb": 36.92578125 + }, + "version": "hnswlib 0.8.0", + "build": { + "ingest_s": 7.962411799933761, + "index_s": 6.699992809444666e-06, + "total_s": 7.962418499926571, + "first_query_ms": 2.738800016231835, + "ready_s": 7.965157299942803 + }, + "mem_after_build": { + "rss_mb": 238.21484375, + "peak_mb": 268.01171875 + }, + "mem_after_first_query": { + "rss_mb": 238.23046875, + "peak_mb": 268.01171875 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.7862, + "qps_1t_runs": [ + 5846.195950184001, + 6468.995745129866, + 6687.102442821054 + ], + "qps_1t": 6468.995745129866, + "qps_1t_min": 5846.195950184001, + "qps_1t_max": 6687.102442821054, + "median_ms": 0.15410000924021006, + "p95_ms": 0.2157099545001983, + "p99_ms": 0.25511402171105113, + "mean_ms": 0.1583921664278023, + "knob": 24, + "batch": { + "recall": 0.7862, + "qps_batch_runs": [ + 38348.55769049941, + 37929.21650029828, + 36982.93236084673 + ], + "qps_batch": 37929.21650029828, + "qps_batch_min": 36982.93236084673, + "qps_batch_max": 38348.55769049941, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8313, + "qps_1t_runs": [ + 6243.541856357715, + 5754.756465872688, + 5816.652160351635 + ], + "qps_1t": 5816.652160351635, + "qps_1t_min": 5754.756465872688, + "qps_1t_max": 6243.541856357715, + "median_ms": 0.16694999067112803, + "p95_ms": 0.21829997713211924, + "p99_ms": 0.25080498948227603, + "mean_ms": 0.1686183328468663, + "knob": 32, + "batch": { + "recall": 0.8313, + "qps_batch_runs": [ + 33553.331293452335, + 34255.84320453977, + 33791.546755591065 + ], + "qps_batch": 33791.546755591065, + "qps_batch_min": 33553.331293452335, + "qps_batch_max": 34255.84320453977, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8849, + "qps_1t_runs": [ + 4461.724259147984, + 4346.824419041297, + 4375.153679780181 + ], + "qps_1t": 4375.153679780181, + "qps_1t_min": 4346.824419041297, + "qps_1t_max": 4461.724259147984, + "median_ms": 0.22620003437623382, + "p95_ms": 0.27640495100058615, + "p99_ms": 0.3154030151199549, + "mean_ms": 0.22758166588998088, + "knob": 48, + "batch": { + "recall": 0.8849, + "qps_batch_runs": [ + 25505.51939708258, + 25787.95084726905, + 26237.422441405433 + ], + "qps_batch": 25787.95084726905, + "qps_batch_min": 25505.51939708258, + "qps_batch_max": 26237.422441405433, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9136, + "qps_1t_runs": [ + 3516.4575228316753, + 3567.6252321586203, + 3760.349893652435 + ], + "qps_1t": 3567.6252321586203, + "qps_1t_min": 3516.4575228316753, + "qps_1t_max": 3760.349893652435, + "median_ms": 0.27785002021119, + "p95_ms": 0.33470498747192323, + "p99_ms": 0.36760400515049685, + "mean_ms": 0.2768694339126038, + "knob": 64, + "batch": { + "recall": 0.9136, + "qps_batch_runs": [ + 19680.932722383193, + 18565.103168771264, + 21420.15636327003 + ], + "qps_batch": 19680.932722383193, + "qps_batch_min": 18565.103168771264, + "qps_batch_max": 21420.15636327003, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9492, + "qps_1t_runs": [ + 2507.0253143253217, + 2569.019274598179, + 2450.3822095836435 + ], + "qps_1t": 2507.0253143253217, + "qps_1t_min": 2450.3822095836435, + "qps_1t_max": 2569.019274598179, + "median_ms": 0.40234997868537903, + "p95_ms": 0.4723049874883145, + "p99_ms": 0.524057031725532, + "mean_ms": 0.3987440997734666, + "knob": 96, + "batch": { + "recall": 0.9492, + "qps_batch_runs": [ + 14563.439075005703, + 14294.760823345387, + 14058.963294218252 + ], + "qps_batch": 14294.760823345387, + "qps_batch_min": 14058.963294218252, + "qps_batch_max": 14563.439075005703, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9631, + "qps_1t_runs": [ + 2125.518680629813, + 1968.2931585614315, + 2073.864415720998 + ], + "qps_1t": 2073.864415720998, + "qps_1t_min": 1968.2931585614315, + "qps_1t_max": 2125.518680629813, + "median_ms": 0.4847999953199178, + "p95_ms": 0.6100249913288279, + "p99_ms": 0.7089389848988494, + "mean_ms": 0.48690646652054664, + "knob": 128, + "batch": { + "recall": 0.9631, + "qps_batch_runs": [ + 12316.378202705333, + 12949.778172304292, + 12743.269000605947 + ], + "qps_batch": 12743.269000605947, + "qps_batch_min": 12316.378202705333, + "qps_batch_max": 12949.778172304292, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9799, + "qps_1t_runs": [ + 1542.4024962952078, + 1475.1544890485577, + 1436.0994646899821 + ], + "qps_1t": 1475.1544890485577, + "qps_1t_min": 1436.0994646899821, + "qps_1t_max": 1542.4024962952078, + "median_ms": 0.679749995470047, + "p95_ms": 0.8137150231050326, + "p99_ms": 0.874941025977022, + "mean_ms": 0.6741882994732199, + "knob": 192, + "batch": { + "recall": 0.9799, + "qps_batch_runs": [ + 8901.437942654153, + 9152.337918089152, + 8719.110370113041 + ], + "qps_batch": 8901.437942654153, + "qps_batch_min": 8719.110370113041, + "qps_batch_max": 9152.337918089152, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9864, + "qps_1t_runs": [ + 1174.6559522924251, + 1225.393036646625, + 1132.3262496512023 + ], + "qps_1t": 1174.6559522924251, + "qps_1t_min": 1132.3262496512023, + "qps_1t_max": 1225.393036646625, + "median_ms": 0.8572000078856945, + "p95_ms": 1.027305019670166, + "p99_ms": 1.119604968698695, + "mean_ms": 0.8501718331244774, + "knob": 256, + "batch": { + "recall": 0.9864, + "qps_batch_runs": [ + 7058.039670887464, + 7108.300646193514, + 7248.619681529368 + ], + "qps_batch": 7108.300646193514, + "qps_batch_min": 7058.039670887464, + "qps_batch_max": 7248.619681529368, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.993, + "qps_1t_runs": [ + 795.0870616942234, + 778.6470070255878, + 804.134021198963 + ], + "qps_1t": 795.0870616942234, + "qps_1t_min": 778.6470070255878, + "qps_1t_max": 804.134021198963, + "median_ms": 1.2711500166915357, + "p95_ms": 1.536109996959567, + "p99_ms": 1.67552201077342, + "mean_ms": 1.2618588999806282, + "knob": 400, + "batch": { + "recall": 0.993, + "qps_batch_runs": [ + 4836.822536695186, + 4805.409738533648, + 4871.380929341081 + ], + "qps_batch": 4836.822536695186, + "qps_batch_min": 4805.409738533648, + "qps_batch_max": 4871.380929341081, + "threads": 32 + } + } + ], + "knob_name": "ef", + "mem_after_sweep": { + "rss_mb": 241.09765625, + "peak_mb": 268.01171875 + }, + "ref_knob": 128, + "ref_recall_pre": 0.9631, + "mutation": { + "ref_knob": 128, + "recall_before": 0.9631, + "insert_10k": { + "seconds": 1.4006904999841936, + "per_s": 7139.335920471259, + "batch": 1000 + }, + "update_10k": { + "seconds": 6.385941600019578, + "per_s": 1565.9397824698776, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.022199900005944073, + "per_s": 450452.4793950638, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 287.01171875, + "peak_mb": 289.25 + }, + "first_query_after_ms": 0.8639000006951392, + "after": { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9685, + "qps_1t_runs": [ + 1868.1043548932466, + 1774.9381911656392, + 1768.3775513264147 + ], + "qps_1t": 1774.9381911656392, + "qps_1t_min": 1768.3775513264147, + "qps_1t_max": 1868.1043548932466, + "median_ms": 0.5601000157184899, + "p95_ms": 0.6797100504627451, + "p99_ms": 0.7403010490816087, + "mean_ms": 0.554730667558033 + }, + "recall_after": 0.9685 + }, + "saveload": { + "save_s": 0.31234579998999834, + "disk_mb": 177.13692474365234, + "load_s": 0.6051081999903545, + "first_query_after_load_ms": 1.213999988976866, + "time_to_ready_s": 0.6063221999793313 + }, + "mem_end": { + "rss_mb": 241.7109375, + "peak_mb": 289.25 + }, + "status": "ok", + "finished": "2026-11-16 19:50:51" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r3.load.json b/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r3.load.json new file mode 100644 index 0000000..a58271d --- /dev/null +++ b/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r3.load.json @@ -0,0 +1 @@ +{"total_pct": 10.7875, "pin_cpu_pct": 3.9, "max_cpu_pct": 51.1, "noisy": false, "seconds": 35.428606271743774, "after": 33.5} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r3.txt b/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r3.txt new file mode 100644 index 0000000..f17749f --- /dev/null +++ b/benchmarks/rust-stage-c/raw/hnswlib_text384_100000_r3.txt @@ -0,0 +1,10 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system hnswlib --ds text384 --n 100000 --exp core,mutation,saveload --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\hnswlib_text384_100000_r3.json + hnswlib text384 100000 knob=24 recall=0.7862 qps1t=6469 med=0.154ms batch=37929 + hnswlib text384 100000 knob=32 recall=0.8313 qps1t=5817 med=0.167ms batch=33792 + hnswlib text384 100000 knob=48 recall=0.8849 qps1t=4375 med=0.226ms batch=25788 + hnswlib text384 100000 knob=64 recall=0.9136 qps1t=3568 med=0.278ms batch=19681 + hnswlib text384 100000 knob=96 recall=0.9492 qps1t=2507 med=0.402ms batch=14295 + hnswlib text384 100000 knob=128 recall=0.9631 qps1t=2074 med=0.485ms batch=12743 + hnswlib text384 100000 knob=192 recall=0.9799 qps1t=1475 med=0.680ms batch=8901 + hnswlib text384 100000 knob=256 recall=0.9864 qps1t=1175 med=0.857ms batch=7108 + hnswlib text384 100000 knob=400 recall=0.9930 qps1t=795 med=1.271ms batch=4837 diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r1.json b/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r1.json new file mode 100644 index 0000000..48cbb89 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r1.json @@ -0,0 +1,87 @@ +{ + "system": "numpy_bf", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 19:54:17", + "mem_baseline": { + "rss_mb": 35.0234375, + "peak_mb": 35.0234375 + }, + "version": "numpy 2.4.2", + "build": { + "ingest_s": 0.012114199984353036, + "index_s": 2.0999577827751637e-06, + "total_s": 0.01211629994213581, + "first_query_ms": 2.498099987860769, + "ready_s": 0.01461439992999658 + }, + "mem_after_build": { + "rss_mb": 85.94921875, + "peak_mb": 95.7109375 + }, + "mem_after_first_query": { + "rss_mb": 85.77734375, + "peak_mb": 95.7109375 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 466.2920723302142, + 351.36949237533145, + 376.9976257074899 + ], + "qps_1t": 376.9976257074899, + "qps_1t_min": 351.36949237533145, + "qps_1t_max": 466.2920723302142, + "median_ms": 2.590749994851649, + "p95_ms": 3.634129968122579, + "p99_ms": 4.481138006085512, + "mean_ms": 2.54770736687351, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 785.0583918871457, + 856.503978610503, + 872.4395643769382 + ], + "qps_batch": 856.503978610503, + "qps_batch_min": 785.0583918871457, + "qps_batch_max": 872.4395643769382, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 1000, + "threads": 32, + "qps": 331.98753340178604, + "median_ms": 2.913000003900379, + "p95_ms": 4.542615014361218, + "p99_ms": 5.911661961581556, + "mean_ms": 3.0356596668134443 + }, + "mem_after_sweep": { + "rss_mb": 90.9453125, + "peak_mb": 1617.03125 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 89.37890625, + "peak_mb": 1617.03125 + }, + "status": "ok", + "finished": "2026-11-16 19:54:37" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r1.load.json b/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r1.load.json new file mode 100644 index 0000000..4d9a07a --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 14.26875, "pin_cpu_pct": 1.2, "max_cpu_pct": 58.4, "noisy": false, "seconds": 20.908947706222534, "after": 19.1} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r1.txt b/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r1.txt new file mode 100644 index 0000000..2fe069b --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r1.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system numpy_bf --ds glove --n 100000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\numpy_bf_glove_100000_r1.json + numpy_bf glove 100000 knob=None recall=1.0000 qps1t=377 med=2.591ms batch=857 diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r2.json b/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r2.json new file mode 100644 index 0000000..2416b6a --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r2.json @@ -0,0 +1,87 @@ +{ + "system": "numpy_bf", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 21:06:21", + "mem_baseline": { + "rss_mb": 34.9453125, + "peak_mb": 34.9453125 + }, + "version": "numpy 2.4.2", + "build": { + "ingest_s": 0.012488799984566867, + "index_s": 2.300017513334751e-06, + "total_s": 0.012491100002080202, + "first_query_ms": 4.202899988740683, + "ready_s": 0.016693999990820885 + }, + "mem_after_build": { + "rss_mb": 85.96875, + "peak_mb": 95.73046875 + }, + "mem_after_first_query": { + "rss_mb": 85.7421875, + "peak_mb": 95.73046875 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 448.63173404877256, + 456.60346991576137, + 459.64097163316245 + ], + "qps_1t": 456.60346991576137, + "qps_1t_min": 448.63173404877256, + "qps_1t_max": 459.64097163316245, + "median_ms": 2.0965500152669847, + "p95_ms": 2.832465028041042, + "p99_ms": 3.6239170038606967, + "mean_ms": 2.198231633345131, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 1206.263014127061, + 1433.719505635049, + 1444.1302819700834 + ], + "qps_batch": 1433.719505635049, + "qps_batch_min": 1206.263014127061, + "qps_batch_max": 1444.1302819700834, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 1000, + "threads": 32, + "qps": 713.4562711119264, + "median_ms": 1.3008000096306205, + "p95_ms": 1.9907850190065781, + "p99_ms": 3.642297024489371, + "mean_ms": 1.4134777663081575 + }, + "mem_after_sweep": { + "rss_mb": 91.06640625, + "peak_mb": 1617.140625 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 89.5, + "peak_mb": 1617.140625 + }, + "status": "ok", + "finished": "2026-11-16 21:06:34" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r2.load.json b/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r2.load.json new file mode 100644 index 0000000..54f259a --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r2.load.json @@ -0,0 +1 @@ +{"total_pct": 22.31875, "pin_cpu_pct": 0.0, "max_cpu_pct": 94.9, "noisy": false, "seconds": 13.497948408126831, "after": 22.1} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r2.txt b/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r2.txt new file mode 100644 index 0000000..ebd5104 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r2.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system numpy_bf --ds glove --n 100000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\numpy_bf_glove_100000_r2.json + numpy_bf glove 100000 knob=None recall=1.0000 qps1t=457 med=2.097ms batch=1434 diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r3.json b/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r3.json new file mode 100644 index 0000000..298880b --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r3.json @@ -0,0 +1,87 @@ +{ + "system": "numpy_bf", + "ds": "glove", + "n": 100000, + "d": 100, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 20:00:19", + "mem_baseline": { + "rss_mb": 34.984375, + "peak_mb": 34.984375 + }, + "version": "numpy 2.4.2", + "build": { + "ingest_s": 0.01222780003445223, + "index_s": 1.500011421740055e-06, + "total_s": 0.01222930004587397, + "first_query_ms": 2.433900022879243, + "ready_s": 0.014663200068753213 + }, + "mem_after_build": { + "rss_mb": 85.97265625, + "peak_mb": 95.734375 + }, + "mem_after_first_query": { + "rss_mb": 85.80078125, + "peak_mb": 95.734375 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 425.4477305719906, + 323.3826549125653, + 318.14716170467625 + ], + "qps_1t": 323.3826549125653, + "qps_1t_min": 318.14716170467625, + "qps_1t_max": 425.4477305719906, + "median_ms": 2.6758500025607646, + "p95_ms": 4.57676498335786, + "p99_ms": 7.232719004387022, + "mean_ms": 2.86199210069996, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 782.831012841627, + 766.0898597346049, + 836.9168018474388 + ], + "qps_batch": 782.831012841627, + "qps_batch_min": 766.0898597346049, + "qps_batch_max": 836.9168018474388, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 959, + "threads": 32, + "qps": 384.9609912067793, + "median_ms": 2.5424000341445208, + "p95_ms": 3.6120600183494385, + "p99_ms": 4.393063997849817, + "mean_ms": 2.6206259643963286 + }, + "mem_after_sweep": { + "rss_mb": 90.953125, + "peak_mb": 1617.0390625 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 89.38671875, + "peak_mb": 1617.0390625 + }, + "status": "ok", + "finished": "2026-11-16 20:00:39" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r3.load.json b/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r3.load.json new file mode 100644 index 0000000..b4bad03 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r3.load.json @@ -0,0 +1 @@ +{"total_pct": 25.33125, "pin_cpu_pct": 7.4, "max_cpu_pct": 83.2, "noisy": false, "seconds": 20.424367666244507, "after": 30.5} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r3.txt b/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r3.txt new file mode 100644 index 0000000..3d85d49 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_glove_100000_r3.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system numpy_bf --ds glove --n 100000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\numpy_bf_glove_100000_r3.json + numpy_bf glove 100000 knob=None recall=1.0000 qps1t=323 med=2.676ms batch=783 diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r1.json b/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r1.json new file mode 100644 index 0000000..2530027 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r1.json @@ -0,0 +1,87 @@ +{ + "system": "numpy_bf", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 21:08:18", + "mem_baseline": { + "rss_mb": 35.234375, + "peak_mb": 35.234375 + }, + "version": "numpy 2.4.2", + "build": { + "ingest_s": 0.015269699913915247, + "index_s": 2.600019797682762e-06, + "total_s": 0.01527229993371293, + "first_query_ms": 3.146900038700551, + "ready_s": 0.01841919997241348 + }, + "mem_after_build": { + "rss_mb": 96.86328125, + "peak_mb": 108.76171875 + }, + "mem_after_first_query": { + "rss_mb": 96.68359375, + "peak_mb": 108.76171875 + }, + "sweep": [ + { + "nq": 953, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 393.7610561750364, + 371.2164165285424, + 389.30091610454076 + ], + "qps_1t": 389.30091610454076, + "qps_1t_min": 371.2164165285424, + "qps_1t_max": 393.7610561750364, + "median_ms": 2.5117999757640064, + "p95_ms": 3.22921002516523, + "p99_ms": 3.8900519872549952, + "mean_ms": 2.6007215466367257, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 1255.3381686653422, + 1431.277360646532, + 1363.6447934253554 + ], + "qps_batch": 1363.6447934253554, + "qps_batch_min": 1255.3381686653422, + "qps_batch_max": 1431.277360646532, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 1000, + "threads": 32, + "qps": 604.5765720116042, + "median_ms": 1.5417999820783734, + "p95_ms": 2.0405199960805476, + "p99_ms": 3.492516988771955, + "mean_ms": 1.6213445335548993 + }, + "mem_after_sweep": { + "rss_mb": 103.4375, + "peak_mb": 1629.50390625 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 101.65625, + "peak_mb": 1629.50390625 + }, + "status": "ok", + "finished": "2026-11-16 21:08:33" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r1.load.json b/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r1.load.json new file mode 100644 index 0000000..d248006 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 16.446875, "pin_cpu_pct": 0.0, "max_cpu_pct": 98.5, "noisy": false, "seconds": 15.187906980514526, "after": 19.8} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r1.txt b/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r1.txt new file mode 100644 index 0000000..41ddd17 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r1.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system numpy_bf --ds sift --n 100000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\numpy_bf_sift_100000_r1.json + numpy_bf sift 100000 knob=None recall=1.0000 qps1t=389 med=2.512ms batch=1364 diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r2.json b/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r2.json new file mode 100644 index 0000000..938f7ea --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r2.json @@ -0,0 +1,87 @@ +{ + "system": "numpy_bf", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 20:06:17", + "mem_baseline": { + "rss_mb": 35.08203125, + "peak_mb": 35.08203125 + }, + "version": "numpy 2.4.2", + "build": { + "ingest_s": 0.014327999961096793, + "index_s": 3.3999676816165447e-06, + "total_s": 0.01433139992877841, + "first_query_ms": 3.3198000164702535, + "ready_s": 0.017651199945248663 + }, + "mem_after_build": { + "rss_mb": 96.8671875, + "peak_mb": 108.765625 + }, + "mem_after_first_query": { + "rss_mb": 96.66796875, + "peak_mb": 108.765625 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 415.47572300610034, + 313.9052557275998, + 298.1357274927315 + ], + "qps_1t": 313.9052557275998, + "qps_1t_min": 298.1357274927315, + "qps_1t_max": 415.47572300610034, + "median_ms": 3.1363500165753067, + "p95_ms": 3.7269550288328896, + "p99_ms": 4.243831008789129, + "mean_ms": 2.9822436996231168, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 785.7773050635615, + 800.5578927826007, + 771.8811120966224 + ], + "qps_batch": 785.7773050635615, + "qps_batch_min": 771.8811120966224, + "qps_batch_max": 800.5578927826007, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 682, + "threads": 32, + "qps": 225.56059989829527, + "median_ms": 4.249799996614456, + "p95_ms": 6.834149986389093, + "p99_ms": 8.911755014560185, + "mean_ms": 4.412365494288444 + }, + "mem_after_sweep": { + "rss_mb": 103.56640625, + "peak_mb": 1629.66796875 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 101.80078125, + "peak_mb": 1629.66796875 + }, + "status": "ok", + "finished": "2026-11-16 20:06:39" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r2.load.json b/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r2.load.json new file mode 100644 index 0000000..b7dba9c --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r2.load.json @@ -0,0 +1 @@ +{"total_pct": 11.46875, "pin_cpu_pct": 0.0, "max_cpu_pct": 53.7, "noisy": false, "seconds": 22.398793697357178, "after": 13.6} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r2.txt b/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r2.txt new file mode 100644 index 0000000..19ee18b --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r2.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system numpy_bf --ds sift --n 100000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\numpy_bf_sift_100000_r2.json + numpy_bf sift 100000 knob=None recall=1.0000 qps1t=314 med=3.136ms batch=786 diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r3.json b/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r3.json new file mode 100644 index 0000000..e1adda8 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r3.json @@ -0,0 +1,87 @@ +{ + "system": "numpy_bf", + "ds": "sift", + "n": 100000, + "d": 128, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 21:10:19", + "mem_baseline": { + "rss_mb": 35.15234375, + "peak_mb": 35.15234375 + }, + "version": "numpy 2.4.2", + "build": { + "ingest_s": 0.01614479999989271, + "index_s": 1.9000144675374031e-06, + "total_s": 0.01614670001436025, + "first_query_ms": 2.965899999253452, + "ready_s": 0.0191126000136137 + }, + "mem_after_build": { + "rss_mb": 96.9296875, + "peak_mb": 108.828125 + }, + "mem_after_first_query": { + "rss_mb": 96.76171875, + "peak_mb": 108.828125 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 1.0, + "qps_1t_runs": [ + 379.06617658354673, + 394.9652828866279, + 385.5058585640153 + ], + "qps_1t": 385.5058585640153, + "qps_1t_min": 379.06617658354673, + "qps_1t_max": 394.9652828866279, + "median_ms": 2.5368999922648072, + "p95_ms": 3.0597650154959406, + "p99_ms": 3.6420350248226887, + "mean_ms": 2.58797473316857, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 1536.6090971073654, + 1484.1331326739305, + 1511.5238579228217 + ], + "qps_batch": 1511.5238579228217, + "qps_batch_min": 1484.1331326739305, + "qps_batch_max": 1536.6090971073654, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 1000, + "threads": 32, + "qps": 649.7691170013476, + "median_ms": 1.5204500232357532, + "p95_ms": 1.9721349992323665, + "p99_ms": 2.3611349920975035, + "mean_ms": 1.5740484996155526 + }, + "mem_after_sweep": { + "rss_mb": 103.640625, + "peak_mb": 1629.734375 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 101.859375, + "peak_mb": 1629.734375 + }, + "status": "ok", + "finished": "2026-11-16 21:10:34" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r3.load.json b/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r3.load.json new file mode 100644 index 0000000..89be60d --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r3.load.json @@ -0,0 +1 @@ +{"total_pct": 15.05625, "pin_cpu_pct": 0.0, "max_cpu_pct": 96.5, "noisy": false, "seconds": 14.941683292388916, "after": 18.2} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r3.txt b/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r3.txt new file mode 100644 index 0000000..d615163 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_sift_100000_r3.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system numpy_bf --ds sift --n 100000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\numpy_bf_sift_100000_r3.json + numpy_bf sift 100000 knob=None recall=1.0000 qps1t=386 med=2.537ms batch=1512 diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_text384_1000000_r1.json b/benchmarks/rust-stage-c/raw/numpy_bf_text384_1000000_r1.json new file mode 100644 index 0000000..40859de --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_text384_1000000_r1.json @@ -0,0 +1,87 @@ +{ + "system": "numpy_bf", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 20:16:30", + "mem_baseline": { + "rss_mb": 37.12109375, + "peak_mb": 37.12109375 + }, + "version": "numpy 2.4.2", + "build": { + "ingest_s": 0.3111884000245482, + "index_s": 2.400018274784088e-06, + "total_s": 0.31119080004282296, + "first_query_ms": 71.9890000182204, + "ready_s": 0.38317980006104335 + }, + "mem_after_build": { + "rss_mb": 1612.765625, + "peak_mb": 1642.06640625 + }, + "mem_after_first_query": { + "rss_mb": 1612.91796875, + "peak_mb": 1642.06640625 + }, + "sweep": [ + { + "nq": 32, + "recall_queries": 32, + "recall": 1.0, + "qps_1t_runs": [ + 11.14921782153486, + 11.065075402665316, + 11.063615929794848 + ], + "qps_1t": 11.065075402665316, + "qps_1t_min": 11.063615929794848, + "qps_1t_max": 11.14921782153486, + "median_ms": 90.17619999940507, + "p95_ms": 98.30877502099611, + "p99_ms": 101.68980499147436, + "mean_ms": 90.15106458355149, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 53.71628864420876, + 60.7480748785275, + 61.62862288634553 + ], + "qps_batch": 60.7480748785275, + "qps_batch_min": 53.71628864420876, + "qps_batch_max": 61.62862288634553, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 62, + "threads": 32, + "qps": 20.182210199196195, + "median_ms": 48.848650010768324, + "p95_ms": 57.46750000980683, + "p99_ms": 65.1828249858227, + "mean_ms": 50.063688711385915 + }, + "mem_after_sweep": { + "rss_mb": 1621.55078125, + "peak_mb": 16880.2421875 + }, + "ref_knob": null, + "ref_recall_pre": 1.0, + "mem_end": { + "rss_mb": 1617.37890625, + "peak_mb": 16880.2421875 + }, + "status": "ok", + "finished": "2026-11-16 20:17:46" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_text384_1000000_r1.load.json b/benchmarks/rust-stage-c/raw/numpy_bf_text384_1000000_r1.load.json new file mode 100644 index 0000000..e716819 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_text384_1000000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 14.534375, "pin_cpu_pct": 0.0, "max_cpu_pct": 64.1, "noisy": false, "seconds": 76.22887468338013, "after": 25.2} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_text384_1000000_r1.txt b/benchmarks/rust-stage-c/raw/numpy_bf_text384_1000000_r1.txt new file mode 100644 index 0000000..56f8af5 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_text384_1000000_r1.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system numpy_bf --ds text384 --n 1000000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\numpy_bf_text384_1000000_r1.json + numpy_bf text384 1000000 knob=None recall=1.0000 qps1t=11 med=90.176ms batch=61 diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r1.json b/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r1.json new file mode 100644 index 0000000..0971906 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r1.json @@ -0,0 +1,87 @@ +{ + "system": "numpy_bf", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 19:08:37", + "mem_baseline": { + "rss_mb": 37.046875, + "peak_mb": 37.046875 + }, + "version": "numpy 2.4.2", + "build": { + "ingest_s": 0.03482420003274456, + "index_s": 3.7999707274138927e-06, + "total_s": 0.03482800000347197, + "first_query_ms": 8.587200019974262, + "ready_s": 0.04341520002344623 + }, + "mem_after_build": { + "rss_mb": 196.42578125, + "peak_mb": 227.85546875 + }, + "mem_after_first_query": { + "rss_mb": 196.2421875, + "peak_mb": 227.85546875 + }, + "sweep": [ + { + "nq": 442, + "recall_queries": 1000, + "recall": 0.9999, + "qps_1t_runs": [ + 117.31331071483481, + 110.31427763208269, + 112.23707069649419 + ], + "qps_1t": 112.23707069649419, + "qps_1t_min": 110.31427763208269, + "qps_1t_max": 117.31331071483481, + "median_ms": 8.884249982656911, + "p95_ms": 10.39055000001099, + "p99_ms": 11.667725004372187, + "mean_ms": 8.832967949263487, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 672.7372683713093, + 668.643399439229, + 712.2590828314577 + ], + "qps_batch": 672.7372683713093, + "qps_batch_min": 668.643399439229, + "qps_batch_max": 712.2590828314577, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 545, + "threads": 32, + "qps": 174.3593086622107, + "median_ms": 5.529900023248047, + "p95_ms": 6.799999997019767, + "p99_ms": 9.408491995418457, + "mean_ms": 5.699814373302623 + }, + "mem_after_sweep": { + "rss_mb": 205.09375, + "peak_mb": 1730.796875 + }, + "ref_knob": null, + "ref_recall_pre": 0.9999, + "mem_end": { + "rss_mb": 201.0078125, + "peak_mb": 1730.796875 + }, + "status": "ok", + "finished": "2026-11-16 19:09:08" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r1.load.json b/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r1.load.json new file mode 100644 index 0000000..29dad80 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 23.971875, "pin_cpu_pct": 0.8, "max_cpu_pct": 74.9, "noisy": false, "seconds": 31.695981979370117, "after": 30.3} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r1.txt b/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r1.txt new file mode 100644 index 0000000..e59a55a --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r1.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system numpy_bf --ds text384 --n 100000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\numpy_bf_text384_100000_r1.json + numpy_bf text384 100000 knob=None recall=0.9999 qps1t=112 med=8.884ms batch=673 diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r2.json b/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r2.json new file mode 100644 index 0000000..b9a1643 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r2.json @@ -0,0 +1,87 @@ +{ + "system": "numpy_bf", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 19:30:23", + "mem_baseline": { + "rss_mb": 37.0703125, + "peak_mb": 37.0703125 + }, + "version": "numpy 2.4.2", + "build": { + "ingest_s": 0.030586799955926836, + "index_s": 2.200016751885414e-06, + "total_s": 0.03058899997267872, + "first_query_ms": 7.012900023255497, + "ready_s": 0.03760189999593422 + }, + "mem_after_build": { + "rss_mb": 196.55078125, + "peak_mb": 227.98046875 + }, + "mem_after_first_query": { + "rss_mb": 196.38671875, + "peak_mb": 227.98046875 + }, + "sweep": [ + { + "nq": 381, + "recall_queries": 1000, + "recall": 0.9999, + "qps_1t_runs": [ + 135.75819079734285, + 114.48699849365319, + 121.32483276831196 + ], + "qps_1t": 121.32483276831196, + "qps_1t_min": 114.48699849365319, + "qps_1t_max": 135.75819079734285, + "median_ms": 7.372700027190149, + "p95_ms": 12.113899999530982, + "p99_ms": 15.233955999137825, + "mean_ms": 8.11433000866917, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 982.4548304011288, + 1051.8285355349756, + 1234.344195383148 + ], + "qps_batch": 1051.8285355349756, + "qps_batch_min": 982.4548304011288, + "qps_batch_max": 1234.344195383148, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 590, + "threads": 32, + "qps": 183.42864010001793, + "median_ms": 5.179949977900833, + "p95_ms": 7.429124985355883, + "p99_ms": 10.045870013418606, + "mean_ms": 5.5044102257559615 + }, + "mem_after_sweep": { + "rss_mb": 204.85546875, + "peak_mb": 1731.03515625 + }, + "ref_knob": null, + "ref_recall_pre": 0.9999, + "mem_end": { + "rss_mb": 201.1328125, + "peak_mb": 1731.03515625 + }, + "status": "ok", + "finished": "2026-11-16 19:30:50" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r2.load.json b/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r2.load.json new file mode 100644 index 0000000..1cc3c2b --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r2.load.json @@ -0,0 +1 @@ +{"total_pct": 9.16875, "pin_cpu_pct": 0.0, "max_cpu_pct": 86.4, "noisy": false, "seconds": 27.071917295455933, "after": 25.8} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r2.txt b/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r2.txt new file mode 100644 index 0000000..2cf0419 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r2.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system numpy_bf --ds text384 --n 100000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\numpy_bf_text384_100000_r2.json + numpy_bf text384 100000 knob=None recall=0.9999 qps1t=121 med=7.373ms batch=1052 diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r3.json b/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r3.json new file mode 100644 index 0000000..acce085 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r3.json @@ -0,0 +1,87 @@ +{ + "system": "numpy_bf", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core" + ], + "started": "2026-11-16 19:44:15", + "mem_baseline": { + "rss_mb": 37.13671875, + "peak_mb": 37.13671875 + }, + "version": "numpy 2.4.2", + "build": { + "ingest_s": 0.030570300004910678, + "index_s": 1.9000144675374031e-06, + "total_s": 0.030572200019378215, + "first_query_ms": 6.746399973053485, + "ready_s": 0.0373185999924317 + }, + "mem_after_build": { + "rss_mb": 196.48828125, + "peak_mb": 227.91796875 + }, + "mem_after_first_query": { + "rss_mb": 196.28125, + "peak_mb": 227.91796875 + }, + "sweep": [ + { + "nq": 403, + "recall_queries": 1000, + "recall": 0.9999, + "qps_1t_runs": [ + 139.5783673404106, + 109.16552384793803, + 112.69994418654542 + ], + "qps_1t": 112.69994418654542, + "qps_1t_min": 109.16552384793803, + "qps_1t_max": 139.5783673404106, + "median_ms": 8.666599984280765, + "p95_ms": 10.002819972578436, + "p99_ms": 10.331163969822228, + "mean_ms": 8.39931803090672, + "knob": null, + "batch": { + "recall": 1.0, + "qps_batch_runs": [ + 495.9190080944104, + 543.0228594652218, + 573.4600290038417 + ], + "qps_batch": 543.0228594652218, + "qps_batch_min": 495.9190080944104, + "qps_batch_max": 573.4600290038417, + "threads": 32 + } + } + ], + "knob_name": null, + "loop_allthreads": { + "nq": 587, + "threads": 32, + "qps": 175.44856468949789, + "median_ms": 5.557700002100319, + "p95_ms": 6.743099947925657, + "p99_ms": 7.9210800002329105, + "mean_ms": 5.661936399811421 + }, + "mem_after_sweep": { + "rss_mb": 204.76953125, + "peak_mb": 1730.81640625 + }, + "ref_knob": null, + "ref_recall_pre": 0.9999, + "mem_end": { + "rss_mb": 201.046875, + "peak_mb": 1730.81640625 + }, + "status": "ok", + "finished": "2026-11-16 19:44:47" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r3.load.json b/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r3.load.json new file mode 100644 index 0000000..113f9f7 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r3.load.json @@ -0,0 +1 @@ +{"total_pct": 14.8, "pin_cpu_pct": 10.7, "max_cpu_pct": 81.0, "noisy": false, "seconds": 32.20750880241394, "after": 18.2} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r3.txt b/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r3.txt new file mode 100644 index 0000000..0a636a6 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/numpy_bf_text384_100000_r3.txt @@ -0,0 +1,2 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system numpy_bf --ds text384 --n 100000 --exp core --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\numpy_bf_text384_100000_r3.json + numpy_bf text384 100000 knob=None recall=0.9999 qps1t=113 med=8.667ms batch=543 diff --git a/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r1.json b/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r1.json new file mode 100644 index 0000000..c740bec --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r1.json @@ -0,0 +1,382 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_exact_query_ms": 3.2076000000000002, + "ingest_s": 0.0500061 + }, + "d": 100, + "ds": "glove", + "exps": [ + "core", + "int8" + ], + "first_exact_query_ms": 3.2076000000000002, + "int8_rerank": 0, + "int8_study": { + "by_factor": [ + { + "applies": true, + "auto_threads": { + "max_ms": 4.7166999999999994, + "mean_ms": 0.18555289999999994, + "median_ms": 0.1706, + "nq": 1000, + "p95_ms": 0.2233, + "p99_ms": 0.2973, + "qps_1t": 5383.745288549905, + "qps_1t_max": 5619.0508299338135, + "qps_1t_min": 5182.732792808852, + "qps_1t_runs": [ + 5182.732792808852, + 5619.0508299338135, + 5383.745288549905 + ], + "recall": 0.9832 + }, + "batch": { + "nq": 1000, + "qps_batch": 15858.063984116563, + "qps_batch_max": 19024.065442785122, + "qps_batch_min": 14901.464068844763, + "qps_batch_runs": [ + 19024.065442785122, + 15858.063984116563, + 14901.464068844763 + ], + "recall": 0.9832, + "threads": 32 + }, + "min_recall_vs_f32_scan": 0.8, + "one_thread": { + "max_ms": 1.1254, + "mean_ms": 0.5099766333333324, + "median_ms": 0.5045, + "nq": 1000, + "p95_ms": 0.5317, + "p99_ms": 0.6194, + "qps_1t": 1971.3744600158936, + "qps_1t_max": 1971.6379875491089, + "qps_1t_min": 1939.9503877087825, + "qps_1t_runs": [ + 1939.9503877087825, + 1971.3744600158936, + 1971.6379875491089 + ], + "recall": 0.9832 + }, + "recall_vs_f32_scan": 0.9831999999999973, + "report_f32_ms": 0.9112, + "report_int8_ms": 0.1988, + "report_queries": 1000, + "rerank_factor": 1, + "scores_exact": true + }, + { + "applies": true, + "auto_threads": { + "max_ms": 3.7199, + "mean_ms": 0.1946509666666668, + "median_ms": 0.1863, + "nq": 1000, + "p95_ms": 0.217, + "p99_ms": 0.2743, + "qps_1t": 4921.393049516598, + "qps_1t_max": 5871.276949895104, + "qps_1t_min": 4752.000235699213, + "qps_1t_runs": [ + 5871.276949895104, + 4752.000235699213, + 4921.393049516598 + ], + "recall": 1.0 + }, + "batch": { + "nq": 1000, + "qps_batch": 13111.366637078683, + "qps_batch_max": 13704.393491509443, + "qps_batch_min": 11755.660938524947, + "qps_batch_runs": [ + 13704.393491509443, + 11755.660938524947, + 13111.366637078683 + ], + "recall": 1.0, + "threads": 32 + }, + "min_recall_vs_f32_scan": 1.0, + "one_thread": { + "max_ms": 4.6588, + "mean_ms": 0.5360446999999996, + "median_ms": 0.5099, + "nq": 1000, + "p95_ms": 0.6201, + "p99_ms": 0.7711, + "qps_1t": 1947.3237248486082, + "qps_1t_max": 1957.1827137708374, + "qps_1t_min": 1713.2960166038933, + "qps_1t_runs": [ + 1713.2960166038933, + 1947.3237248486082, + 1957.1827137708374 + ], + "recall": 1.0 + }, + "recall_vs_f32_scan": 1.0, + "report_f32_ms": 0.9112, + "report_int8_ms": 0.1777, + "report_queries": 1000, + "rerank_factor": 2, + "scores_exact": true + }, + { + "applies": true, + "auto_threads": { + "max_ms": 5.2120999999999995, + "mean_ms": 0.19008466666666615, + "median_ms": 0.1819, + "nq": 1000, + "p95_ms": 0.2132, + "p99_ms": 0.25070000000000003, + "qps_1t": 5281.060690477559, + "qps_1t_max": 5307.480947470253, + "qps_1t_min": 5195.215414411943, + "qps_1t_runs": [ + 5281.060690477559, + 5307.480947470253, + 5195.215414411943 + ], + "recall": 1.0 + }, + "batch": { + "nq": 1000, + "qps_batch": 10625.705281188038, + "qps_batch_max": 12144.308387709474, + "qps_batch_min": 10540.38442890089, + "qps_batch_runs": [ + 12144.308387709474, + 10540.38442890089, + 10625.705281188038 + ], + "recall": 1.0, + "threads": 32 + }, + "min_recall_vs_f32_scan": 1.0, + "one_thread": { + "max_ms": 0.7825, + "mean_ms": 0.5231933000000009, + "median_ms": 0.516, + "nq": 1000, + "p95_ms": 0.5659000000000001, + "p99_ms": 0.6774, + "qps_1t": 1922.2488774066567, + "qps_1t_max": 1929.6864703313602, + "qps_1t_min": 1882.7534139026268, + "qps_1t_runs": [ + 1882.7534139026268, + 1922.2488774066567, + 1929.6864703313602 + ], + "recall": 1.0 + }, + "recall_vs_f32_scan": 1.0, + "report_f32_ms": 0.9112, + "report_int8_ms": 0.18595, + "report_queries": 1000, + "rerank_factor": 4, + "scores_exact": true + }, + { + "applies": true, + "auto_threads": { + "max_ms": 4.3006, + "mean_ms": 0.2371626666666668, + "median_ms": 0.21660000000000001, + "nq": 1000, + "p95_ms": 0.2689, + "p99_ms": 0.404, + "qps_1t": 4377.022731629854, + "qps_1t_max": 4490.049377072999, + "qps_1t_min": 3841.61040308097, + "qps_1t_runs": [ + 4377.022731629854, + 4490.049377072999, + 3841.61040308097 + ], + "recall": 1.0 + }, + "batch": { + "nq": 1000, + "qps_batch": 9988.553118126627, + "qps_batch_max": 10144.13805766131, + "qps_batch_min": 8612.886083384796, + "qps_batch_runs": [ + 9988.553118126627, + 8612.886083384796, + 10144.13805766131 + ], + "recall": 1.0, + "threads": 32 + }, + "min_recall_vs_f32_scan": 1.0, + "one_thread": { + "max_ms": 0.8030999999999999, + "mean_ms": 0.5397129999999994, + "median_ms": 0.5354, + "nq": 1000, + "p95_ms": 0.5614, + "p99_ms": 0.6196, + "qps_1t": 1851.354043320202, + "qps_1t_max": 1856.1181738469977, + "qps_1t_min": 1851.0463039232911, + "qps_1t_runs": [ + 1851.0463039232911, + 1856.1181738469977, + 1851.354043320202 + ], + "recall": 1.0 + }, + "recall_vs_f32_scan": 1.0, + "report_f32_ms": 0.9112, + "report_int8_ms": 0.2186, + "report_queries": 1000, + "rerank_factor": 8, + "scores_exact": true + } + ], + "f32": { + "auto_threads": { + "max_ms": 5.7409, + "mean_ms": 0.6797377666666665, + "median_ms": 0.5955, + "nq": 1000, + "p95_ms": 1.072, + "p99_ms": 2.4258, + "qps_1t": 1475.9204319487767, + "qps_1t_max": 1490.035610361053, + "qps_1t_min": 1448.1313385613762, + "qps_1t_runs": [ + 1448.1313385613762, + 1490.035610361053, + 1475.9204319487767 + ], + "recall": 1.0 + }, + "batch": { + "nq": 1000, + "qps_batch": 25816.77832423292, + "qps_batch_max": 28595.93937660852, + "qps_batch_min": 25509.358108021926, + "qps_batch_runs": [ + 28595.93937660852, + 25509.358108021926, + 25816.77832423292 + ], + "recall": 1.0, + "threads": 32 + }, + "mem": { + "peak_mb": 67.0390625, + "rss_mb": 63.83203125 + }, + "one_thread": { + "max_ms": 7.6082, + "mean_ms": 1.928739800000002, + "median_ms": 1.7993, + "nq": 1000, + "p95_ms": 2.6473999999999998, + "p99_ms": 4.2631, + "qps_1t": 545.5215057397323, + "qps_1t_max": 569.8441766797307, + "qps_1t_min": 454.90826456409275, + "qps_1t_runs": [ + 454.90826456409275, + 569.8441766797307, + 545.5215057397323 + ], + "recall": 1.0 + } + }, + "mem_after_int8": { + "peak_mb": 72.0078125, + "rss_mb": 72.0078125 + }, + "quant_build_s": 0.032329, + "quant_bytes_mb": 10.0703125, + "rss_delta_mb": 8.17578125 + }, + "k": 10, + "knob_name": "", + "mem_after_first_exact_query": { + "peak_mb": 98.875, + "rss_mb": 70.83984375 + }, + "mem_after_ingest": { + "peak_mb": 64.50390625, + "rss_mb": 58.91796875 + }, + "mem_after_sweep": { + "peak_mb": 98.875, + "rss_mb": 71.3359375 + }, + "mem_baseline": { + "peak_mb": 5.70703125, + "rss_mb": 5.70703125 + }, + "mem_end": { + "peak_mb": 98.875, + "rss_mb": 71.34765625 + }, + "mode": "exact", + "n": 100000, + "queries": 1000, + "raw_vectors_mb": 38.14697265625, + "ref_knob": null, + "started": "SystemTime { intervals: 134393582612441851 }", + "status": "ok", + "sweep": [ + { + "auto_threads": { + "median_ms": 0.5387000000000001, + "p99_ms": 2.2430000000000003, + "qps": 1668.5098138410224, + "runs": [ + 1584.6730423345396, + 1668.5098138410224, + 1765.1867399878179 + ] + }, + "batch": { + "nq": 1000, + "qps_batch": 19239.241896912295, + "qps_batch_max": 20954.608127873402, + "qps_batch_min": 17024.638056194923, + "qps_batch_runs": [ + 20954.608127873402, + 19239.241896912295, + 17024.638056194923 + ], + "recall": 1.0, + "threads": 32 + }, + "knob": null, + "max_ms": 3.8682, + "mean_ms": 1.8209415000000027, + "median_ms": 1.7546, + "nq": 1000, + "p95_ms": 2.3094, + "p99_ms": 2.8266, + "qps_1t": 548.4488768863897, + "qps_1t_max": 573.7615514695963, + "qps_1t_min": 527.2548025267315, + "qps_1t_runs": [ + 548.4488768863897, + 573.7615514695963, + 527.2548025267315 + ], + "recall": 1.0 + } + ], + "system": "rust_exact", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r1.load.json b/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r1.load.json new file mode 100644 index 0000000..2a7cfcc --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 16.6375, "pin_cpu_pct": 0.0, "max_cpu_pct": 85.5, "noisy": false, "seconds": 28.22788119316101, "after": 27.2} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r1.txt b/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r1.txt new file mode 100644 index 0000000..5d721ef --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r1.txt @@ -0,0 +1,13 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/glove --ds glove --n 100000 --mode exact --exp core,int8 --queries 1000 --budget 3 --label rust_exact --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_exact_glove_100000_r1.json +rust_exact: ds=glove n=100000 d=100 mode=exact int8=0 queries=1000 cpus=32 exps=["core", "int8"] + ingest 0.05s rss 59 MB + int8 study: f32 1T 545.5 qps, auto 1475.9 qps, batch 25817 qps + rerank x1: recall vs f32 0.9832 (worst 0.80) exact_scores=true applies=true int8 0.199 ms f32 0.911 ms + 1T 1971.4 qps (recall vs float64 truth 0.9832), auto 5383.7 qps, batch 15858 qps + rerank x2: recall vs f32 1.0000 (worst 1.00) exact_scores=true applies=true int8 0.178 ms f32 0.911 ms + 1T 1947.3 qps (recall vs float64 truth 1.0000), auto 4921.4 qps, batch 13111 qps + rerank x4: recall vs f32 1.0000 (worst 1.00) exact_scores=true applies=true int8 0.186 ms f32 0.911 ms + 1T 1922.2 qps (recall vs float64 truth 1.0000), auto 5281.1 qps, batch 10626 qps + rerank x8: recall vs f32 1.0000 (worst 1.00) exact_scores=true applies=true int8 0.219 ms f32 0.911 ms + 1T 1851.4 qps (recall vs float64 truth 1.0000), auto 4377.0 qps, batch 9989 qps + exact 1T recall=1.0000 qps=548.4 median=1.755ms | auto 1668.5 qps | batch 19239 qps diff --git a/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r2.json b/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r2.json new file mode 100644 index 0000000..80345c0 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r2.json @@ -0,0 +1,89 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_exact_query_ms": 1.8423999999999998, + "ingest_s": 0.0489373 + }, + "d": 100, + "ds": "glove", + "exps": [ + "core" + ], + "first_exact_query_ms": 1.8423999999999998, + "int8_rerank": 0, + "k": 10, + "knob_name": "", + "mem_after_first_exact_query": { + "peak_mb": 64.5078125, + "rss_mb": 60.5625 + }, + "mem_after_ingest": { + "peak_mb": 64.5078125, + "rss_mb": 58.921875 + }, + "mem_after_sweep": { + "peak_mb": 66.8671875, + "rss_mb": 63.55078125 + }, + "mem_baseline": { + "peak_mb": 5.7109375, + "rss_mb": 5.7109375 + }, + "mem_end": { + "peak_mb": 66.8671875, + "rss_mb": 63.57421875 + }, + "mode": "exact", + "n": 100000, + "queries": 1000, + "raw_vectors_mb": 38.14697265625, + "ref_knob": null, + "started": "SystemTime { intervals: 134393541816261769 }", + "status": "ok", + "sweep": [ + { + "auto_threads": { + "median_ms": 1.4115, + "p99_ms": 2.0872, + "qps": 707.6954019543574, + "runs": [ + 707.6954019543574, + 676.8370915389675, + 710.269142286087 + ] + }, + "batch": { + "nq": 1000, + "qps_batch": 10552.029419058019, + "qps_batch_max": 10653.961462490599, + "qps_batch_min": 10508.396208570648, + "qps_batch_runs": [ + 10508.396208570648, + 10552.029419058019, + 10653.961462490599 + ], + "recall": 1.0, + "threads": 32 + }, + "knob": null, + "max_ms": 3.7457, + "mean_ms": 1.9214546666666632, + "median_ms": 1.7194, + "nq": 1000, + "p95_ms": 2.5465, + "p99_ms": 2.7186, + "qps_1t": 537.1059367876796, + "qps_1t_max": 622.8444133409778, + "qps_1t_min": 435.35112047406295, + "qps_1t_runs": [ + 622.8444133409778, + 537.1059367876796, + 435.35112047406295 + ], + "recall": 1.0 + } + ], + "system": "rust_exact", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r2.load.json b/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r2.load.json new file mode 100644 index 0000000..6831c85 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r2.load.json @@ -0,0 +1 @@ +{"total_pct": 13.21875, "pin_cpu_pct": 0.0, "max_cpu_pct": 71.2, "noisy": false, "seconds": 10.765508890151978, "after": 16.1} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r2.txt b/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r2.txt new file mode 100644 index 0000000..449319c --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r2.txt @@ -0,0 +1,4 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/glove --ds glove --n 100000 --mode exact --exp core --queries 1000 --budget 3 --label rust_exact --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_exact_glove_100000_r2.json +rust_exact: ds=glove n=100000 d=100 mode=exact int8=0 queries=1000 cpus=32 exps=["core"] + ingest 0.05s rss 59 MB + exact 1T recall=1.0000 qps=537.1 median=1.719ms | auto 707.7 qps | batch 10552 qps diff --git a/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r3.json b/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r3.json new file mode 100644 index 0000000..61d3a16 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r3.json @@ -0,0 +1,89 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_exact_query_ms": 2.5937, + "ingest_s": 0.0558751 + }, + "d": 100, + "ds": "glove", + "exps": [ + "core" + ], + "first_exact_query_ms": 2.5937, + "int8_rerank": 0, + "k": 10, + "knob_name": "", + "mem_after_first_exact_query": { + "peak_mb": 64.46484375, + "rss_mb": 60.58984375 + }, + "mem_after_ingest": { + "peak_mb": 64.46484375, + "rss_mb": 58.9453125 + }, + "mem_after_sweep": { + "peak_mb": 66.96875, + "rss_mb": 63.63671875 + }, + "mem_baseline": { + "peak_mb": 5.734375, + "rss_mb": 5.734375 + }, + "mem_end": { + "peak_mb": 66.96875, + "rss_mb": 63.66015625 + }, + "mode": "exact", + "n": 100000, + "queries": 1000, + "raw_vectors_mb": 38.14697265625, + "ref_knob": null, + "started": "SystemTime { intervals: 134393583987056697 }", + "status": "ok", + "sweep": [ + { + "auto_threads": { + "median_ms": 0.618, + "p99_ms": 2.6595999999999997, + "qps": 1568.031967781332, + "runs": [ + 1568.031967781332, + 1099.9715767344564, + 1593.798974454111 + ] + }, + "batch": { + "nq": 1000, + "qps_batch": 25285.409054704982, + "qps_batch_max": 25309.791852271806, + "qps_batch_min": 23426.335886803943, + "qps_batch_runs": [ + 25285.409054704982, + 23426.335886803943, + 25309.791852271806 + ], + "recall": 1.0, + "threads": 32 + }, + "knob": null, + "max_ms": 4.4799999999999995, + "mean_ms": 1.8527472666666653, + "median_ms": 1.7778, + "nq": 1000, + "p95_ms": 2.3674999999999997, + "p99_ms": 2.8139, + "qps_1t": 548.2434799732961, + "qps_1t_max": 563.3544467312885, + "qps_1t_min": 510.4245798427313, + "qps_1t_runs": [ + 548.2434799732961, + 510.4245798427313, + 563.3544467312885 + ], + "recall": 1.0 + } + ], + "system": "rust_exact", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r3.load.json b/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r3.load.json new file mode 100644 index 0000000..7702e15 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r3.load.json @@ -0,0 +1 @@ +{"total_pct": 14.88125, "pin_cpu_pct": 0.4, "max_cpu_pct": 87.5, "noisy": false, "seconds": 8.145856380462646, "after": 25.3} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r3.txt b/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r3.txt new file mode 100644 index 0000000..fdd4b30 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_glove_100000_r3.txt @@ -0,0 +1,4 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/glove --ds glove --n 100000 --mode exact --exp core --queries 1000 --budget 3 --label rust_exact --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_exact_glove_100000_r3.json +rust_exact: ds=glove n=100000 d=100 mode=exact int8=0 queries=1000 cpus=32 exps=["core"] + ingest 0.06s rss 59 MB + exact 1T recall=1.0000 qps=548.2 median=1.778ms | auto 1568.0 qps | batch 25285 qps diff --git a/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r1.json b/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r1.json new file mode 100644 index 0000000..50c97c1 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r1.json @@ -0,0 +1,382 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_exact_query_ms": 6.5437, + "ingest_s": 0.052656499999999995 + }, + "d": 128, + "ds": "sift", + "exps": [ + "core", + "int8" + ], + "first_exact_query_ms": 6.5437, + "int8_rerank": 0, + "int8_study": { + "by_factor": [ + { + "applies": true, + "auto_threads": { + "max_ms": 0.9874, + "mean_ms": 0.35803386666666726, + "median_ms": 0.34650000000000003, + "nq": 1000, + "p95_ms": 0.5084, + "p99_ms": 0.6311, + "qps_1t": 2755.111627480261, + "qps_1t_max": 2969.5452349340844, + "qps_1t_min": 2671.025780740836, + "qps_1t_runs": [ + 2755.111627480261, + 2671.025780740836, + 2969.5452349340844 + ], + "recall": 0.9763 + }, + "batch": { + "nq": 1000, + "qps_batch": 9139.497985654643, + "qps_batch_max": 9209.502364539732, + "qps_batch_min": 9088.44694800863, + "qps_batch_runs": [ + 9088.44694800863, + 9139.497985654643, + 9209.502364539732 + ], + "recall": 0.9763, + "threads": 32 + }, + "min_recall_vs_f32_scan": 0.8, + "one_thread": { + "max_ms": 1.4687999999999999, + "mean_ms": 0.8270260666666646, + "median_ms": 0.8284, + "nq": 1000, + "p95_ms": 0.9163, + "p99_ms": 1.026, + "qps_1t": 1206.2467175011213, + "qps_1t_max": 1217.0531540662714, + "qps_1t_min": 1204.2337003355944, + "qps_1t_runs": [ + 1204.2337003355944, + 1206.2467175011213, + 1217.0531540662714 + ], + "recall": 0.9763 + }, + "recall_vs_f32_scan": 0.9762999999999962, + "report_f32_ms": 1.4723499999999998, + "report_int8_ms": 0.27574999999999994, + "report_queries": 1000, + "rerank_factor": 1, + "scores_exact": true + }, + { + "applies": true, + "auto_threads": { + "max_ms": 1.4504, + "mean_ms": 0.31927123333333335, + "median_ms": 0.3126, + "nq": 1000, + "p95_ms": 0.4555, + "p99_ms": 0.5688, + "qps_1t": 3046.7452991005057, + "qps_1t_max": 3569.0143881246017, + "qps_1t_min": 2862.0066329865695, + "qps_1t_runs": [ + 3569.0143881246017, + 2862.0066329865695, + 3046.7452991005057 + ], + "recall": 1.0 + }, + "batch": { + "nq": 1000, + "qps_batch": 8480.124284701516, + "qps_batch_max": 8521.21911377617, + "qps_batch_min": 8056.057268899914, + "qps_batch_runs": [ + 8056.057268899914, + 8480.124284701516, + 8521.21911377617 + ], + "recall": 1.0, + "threads": 32 + }, + "min_recall_vs_f32_scan": 1.0, + "one_thread": { + "max_ms": 1.1993, + "mean_ms": 0.8398882666666685, + "median_ms": 0.8419, + "nq": 1000, + "p95_ms": 0.914, + "p99_ms": 0.9944999999999999, + "qps_1t": 1190.7869291605618, + "qps_1t_max": 1196.578598880648, + "qps_1t_min": 1184.5984667268192, + "qps_1t_runs": [ + 1190.7869291605618, + 1184.5984667268192, + 1196.578598880648 + ], + "recall": 1.0 + }, + "recall_vs_f32_scan": 1.0, + "report_f32_ms": 1.4723499999999998, + "report_int8_ms": 0.28324999999999995, + "report_queries": 1000, + "rerank_factor": 2, + "scores_exact": true + }, + { + "applies": true, + "auto_threads": { + "max_ms": 0.8999, + "mean_ms": 0.30989616666666725, + "median_ms": 0.3009, + "nq": 1000, + "p95_ms": 0.4283, + "p99_ms": 0.5112, + "qps_1t": 3249.833202310891, + "qps_1t_max": 3511.0103529162293, + "qps_1t_min": 2965.9324105526703, + "qps_1t_runs": [ + 2965.9324105526703, + 3249.833202310891, + 3511.0103529162293 + ], + "recall": 1.0 + }, + "batch": { + "nq": 1000, + "qps_batch": 7078.844166324522, + "qps_batch_max": 7170.211499728607, + "qps_batch_min": 7060.262161654586, + "qps_batch_runs": [ + 7060.262161654586, + 7078.844166324522, + 7170.211499728607 + ], + "recall": 1.0, + "threads": 32 + }, + "min_recall_vs_f32_scan": 1.0, + "one_thread": { + "max_ms": 9.8477, + "mean_ms": 0.8547541999999976, + "median_ms": 0.8564, + "nq": 1000, + "p95_ms": 0.9656, + "p99_ms": 1.2065000000000001, + "qps_1t": 1169.695163404074, + "qps_1t_max": 1173.3150609067852, + "qps_1t_min": 1166.7889850452652, + "qps_1t_runs": [ + 1166.7889850452652, + 1173.3150609067852, + 1169.695163404074 + ], + "recall": 1.0 + }, + "recall_vs_f32_scan": 1.0, + "report_f32_ms": 1.4723499999999998, + "report_int8_ms": 0.2796, + "report_queries": 1000, + "rerank_factor": 4, + "scores_exact": true + }, + { + "applies": true, + "auto_threads": { + "max_ms": 0.846, + "mean_ms": 0.3479783666666659, + "median_ms": 0.3441, + "nq": 1000, + "p95_ms": 0.4646, + "p99_ms": 0.5514, + "qps_1t": 2913.509272388932, + "qps_1t_max": 3079.9926449775603, + "qps_1t_min": 2659.360163263439, + "qps_1t_runs": [ + 2659.360163263439, + 2913.509272388932, + 3079.9926449775603 + ], + "recall": 1.0 + }, + "batch": { + "nq": 1000, + "qps_batch": 5813.0173222103185, + "qps_batch_max": 6120.737671304146, + "qps_batch_min": 5766.896285253758, + "qps_batch_runs": [ + 5766.896285253758, + 6120.737671304146, + 5813.0173222103185 + ], + "recall": 1.0, + "threads": 32 + }, + "min_recall_vs_f32_scan": 1.0, + "one_thread": { + "max_ms": 1.5177999999999998, + "mean_ms": 0.8396720333333336, + "median_ms": 0.8819, + "nq": 1000, + "p95_ms": 1.0125, + "p99_ms": 1.1049, + "qps_1t": 1176.6085975025562, + "qps_1t_max": 1256.8029170898435, + "qps_1t_min": 1144.8903372514906, + "qps_1t_runs": [ + 1144.8903372514906, + 1176.6085975025562, + 1256.8029170898435 + ], + "recall": 1.0 + }, + "recall_vs_f32_scan": 1.0, + "report_f32_ms": 1.4723499999999998, + "report_int8_ms": 0.35535000000000005, + "report_queries": 1000, + "rerank_factor": 8, + "scores_exact": true + } + ], + "f32": { + "auto_threads": { + "max_ms": 2.3724000000000003, + "mean_ms": 1.4735060666666653, + "median_ms": 1.4543, + "nq": 1000, + "p95_ms": 1.8296000000000001, + "p99_ms": 1.9918000000000002, + "qps_1t": 676.578588087385, + "qps_1t_max": 692.7757689204851, + "qps_1t_min": 667.1003708544382, + "qps_1t_runs": [ + 676.578588087385, + 692.7757689204851, + 667.1003708544382 + ], + "recall": 1.0 + }, + "batch": { + "nq": 1000, + "qps_batch": 11132.956445647793, + "qps_batch_max": 11203.67839168956, + "qps_batch_min": 10992.248266522447, + "qps_batch_runs": [ + 11203.67839168956, + 11132.956445647793, + 10992.248266522447 + ], + "recall": 1.0, + "threads": 32 + }, + "mem": { + "peak_mb": 77.92578125, + "rss_mb": 73.98046875 + }, + "one_thread": { + "max_ms": 4.7387999999999995, + "mean_ms": 2.560785566666669, + "median_ms": 2.5959, + "nq": 1000, + "p95_ms": 3.234, + "p99_ms": 3.5539, + "qps_1t": 389.00594510005806, + "qps_1t_max": 458.820135762125, + "qps_1t_min": 341.0410195271221, + "qps_1t_runs": [ + 458.820135762125, + 389.00594510005806, + 341.0410195271221 + ], + "recall": 1.0 + } + }, + "mem_after_int8": { + "peak_mb": 85.6875, + "rss_mb": 85.6875 + }, + "quant_build_s": 0.1178652, + "quant_bytes_mb": 12.75, + "rss_delta_mb": 11.70703125 + }, + "k": 10, + "knob_name": "", + "mem_after_first_exact_query": { + "peak_mb": 113.0859375, + "rss_mb": 82.22265625 + }, + "mem_after_ingest": { + "peak_mb": 77.65625, + "rss_mb": 69.9921875 + }, + "mem_after_sweep": { + "peak_mb": 113.0859375, + "rss_mb": 82.16796875 + }, + "mem_baseline": { + "peak_mb": 5.84375, + "rss_mb": 5.84375 + }, + "mem_end": { + "peak_mb": 113.0859375, + "rss_mb": 82.18359375 + }, + "mode": "exact", + "n": 100000, + "queries": 1000, + "raw_vectors_mb": 48.828125, + "ref_knob": null, + "started": "SystemTime { intervals: 134393545378266934 }", + "status": "ok", + "sweep": [ + { + "auto_threads": { + "median_ms": 1.3504999999999998, + "p99_ms": 2.1935000000000002, + "qps": 722.0704590577388, + "runs": [ + 701.5239415038468, + 727.7869418283533, + 722.0704590577388 + ] + }, + "batch": { + "nq": 1000, + "qps_batch": 10910.134314663548, + "qps_batch_max": 12056.181807221652, + "qps_batch_min": 8615.498074866955, + "qps_batch_runs": [ + 8615.498074866955, + 10910.134314663548, + 12056.181807221652 + ], + "recall": 1.0, + "threads": 32 + }, + "knob": null, + "max_ms": 5.0505, + "mean_ms": 2.8641997, + "median_ms": 2.8461, + "nq": 1000, + "p95_ms": 3.2809, + "p99_ms": 3.5549, + "qps_1t": 349.9850871354372, + "qps_1t_max": 355.39324351242954, + "qps_1t_min": 342.2840615426739, + "qps_1t_runs": [ + 355.39324351242954, + 342.2840615426739, + 349.9850871354372 + ], + "recall": 1.0 + } + ], + "system": "rust_exact", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r1.load.json b/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r1.load.json new file mode 100644 index 0000000..0e2a699 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 17.759375, "pin_cpu_pct": 0.0, "max_cpu_pct": 69.4, "noisy": false, "seconds": 45.67226958274841, "after": 26.9} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r1.txt b/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r1.txt new file mode 100644 index 0000000..4fd6f54 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r1.txt @@ -0,0 +1,13 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/sift --ds sift --n 100000 --mode exact --exp core,int8 --queries 1000 --budget 3 --label rust_exact --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_exact_sift_100000_r1.json +rust_exact: ds=sift n=100000 d=128 mode=exact int8=0 queries=1000 cpus=32 exps=["core", "int8"] + ingest 0.05s rss 70 MB + int8 study: f32 1T 389.0 qps, auto 676.6 qps, batch 11133 qps + rerank x1: recall vs f32 0.9763 (worst 0.80) exact_scores=true applies=true int8 0.276 ms f32 1.472 ms + 1T 1206.2 qps (recall vs float64 truth 0.9763), auto 2755.1 qps, batch 9139 qps + rerank x2: recall vs f32 1.0000 (worst 1.00) exact_scores=true applies=true int8 0.283 ms f32 1.472 ms + 1T 1190.8 qps (recall vs float64 truth 1.0000), auto 3046.7 qps, batch 8480 qps + rerank x4: recall vs f32 1.0000 (worst 1.00) exact_scores=true applies=true int8 0.280 ms f32 1.472 ms + 1T 1169.7 qps (recall vs float64 truth 1.0000), auto 3249.8 qps, batch 7079 qps + rerank x8: recall vs f32 1.0000 (worst 1.00) exact_scores=true applies=true int8 0.355 ms f32 1.472 ms + 1T 1176.6 qps (recall vs float64 truth 1.0000), auto 2913.5 qps, batch 5813 qps + exact 1T recall=1.0000 qps=350.0 median=2.846ms | auto 722.1 qps | batch 10910 qps diff --git a/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r2.json b/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r2.json new file mode 100644 index 0000000..e6eca59 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r2.json @@ -0,0 +1,89 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_exact_query_ms": 2.3133, + "ingest_s": 0.052445 + }, + "d": 128, + "ds": "sift", + "exps": [ + "core" + ], + "first_exact_query_ms": 2.3133, + "int8_rerank": 0, + "k": 10, + "knob_name": "", + "mem_after_first_exact_query": { + "peak_mb": 77.484375, + "rss_mb": 70.9765625 + }, + "mem_after_ingest": { + "peak_mb": 77.484375, + "rss_mb": 69.984375 + }, + "mem_after_sweep": { + "peak_mb": 78.0625, + "rss_mb": 74.08203125 + }, + "mem_baseline": { + "peak_mb": 5.85546875, + "rss_mb": 5.85546875 + }, + "mem_end": { + "peak_mb": 78.0625, + "rss_mb": 74.10546875 + }, + "mode": "exact", + "n": 100000, + "queries": 1000, + "raw_vectors_mb": 48.828125, + "ref_knob": null, + "started": "SystemTime { intervals: 134393585175266885 }", + "status": "ok", + "sweep": [ + { + "auto_threads": { + "median_ms": 0.5863, + "p99_ms": 1.4699, + "qps": 1545.8127102595142, + "runs": [ + 1681.0469022172006, + 1528.8595945800696, + 1545.8127102595142 + ] + }, + "batch": { + "nq": 1000, + "qps_batch": 28197.44982263804, + "qps_batch_max": 29288.065698988972, + "qps_batch_min": 26724.96271867701, + "qps_batch_runs": [ + 29288.065698988972, + 28197.44982263804, + 26724.96271867701 + ], + "recall": 1.0, + "threads": 32 + }, + "knob": null, + "max_ms": 5.0104, + "mean_ms": 2.1788779333333332, + "median_ms": 2.1234, + "nq": 1000, + "p95_ms": 2.6664000000000003, + "p99_ms": 3.0552, + "qps_1t": 470.53002242687194, + "qps_1t_max": 477.2361915883782, + "qps_1t_min": 431.7840748015824, + "qps_1t_runs": [ + 477.2361915883782, + 431.7840748015824, + 470.53002242687194 + ], + "recall": 1.0 + } + ], + "system": "rust_exact", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r2.load.json b/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r2.load.json new file mode 100644 index 0000000..4e764cc --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r2.load.json @@ -0,0 +1 @@ +{"total_pct": 13.803125, "pin_cpu_pct": 0.0, "max_cpu_pct": 97.3, "noisy": false, "seconds": 8.862428188323975, "after": 20.1} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r2.txt b/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r2.txt new file mode 100644 index 0000000..be9d193 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r2.txt @@ -0,0 +1,4 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/sift --ds sift --n 100000 --mode exact --exp core --queries 1000 --budget 3 --label rust_exact --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_exact_sift_100000_r2.json +rust_exact: ds=sift n=100000 d=128 mode=exact int8=0 queries=1000 cpus=32 exps=["core"] + ingest 0.05s rss 70 MB + exact 1T recall=1.0000 qps=470.5 median=2.123ms | auto 1545.8 qps | batch 28197 qps diff --git a/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r3.json b/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r3.json new file mode 100644 index 0000000..7a7421a --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r3.json @@ -0,0 +1,89 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_exact_query_ms": 2.285, + "ingest_s": 0.05209060000000001 + }, + "d": 128, + "ds": "sift", + "exps": [ + "core" + ], + "first_exact_query_ms": 2.285, + "int8_rerank": 0, + "k": 10, + "knob_name": "", + "mem_after_first_exact_query": { + "peak_mb": 77.51953125, + "rss_mb": 70.9765625 + }, + "mem_after_ingest": { + "peak_mb": 77.51953125, + "rss_mb": 69.984375 + }, + "mem_after_sweep": { + "peak_mb": 77.51953125, + "rss_mb": 72.88671875 + }, + "mem_baseline": { + "peak_mb": 5.859375, + "rss_mb": 5.859375 + }, + "mem_end": { + "peak_mb": 77.51953125, + "rss_mb": 72.91015625 + }, + "mode": "exact", + "n": 100000, + "queries": 1000, + "raw_vectors_mb": 48.828125, + "ref_knob": null, + "started": "SystemTime { intervals: 134393548973732569 }", + "status": "ok", + "sweep": [ + { + "auto_threads": { + "median_ms": 1.5138, + "p99_ms": 2.1338, + "qps": 645.0379424218492, + "runs": [ + 645.0379424218492, + 689.2154529548076, + 634.7206273984905 + ] + }, + "batch": { + "nq": 1000, + "qps_batch": 11456.905993794939, + "qps_batch_max": 11599.0711463826, + "qps_batch_min": 11353.08315679289, + "qps_batch_runs": [ + 11599.0711463826, + 11353.08315679289, + 11456.905993794939 + ], + "recall": 1.0, + "threads": 32 + }, + "knob": null, + "max_ms": 3.5967, + "mean_ms": 2.437289366666663, + "median_ms": 2.5752, + "nq": 1000, + "p95_ms": 2.9753000000000003, + "p99_ms": 3.1638, + "qps_1t": 396.93923332695437, + "qps_1t_max": 500.5437657198895, + "qps_1t_min": 357.8120295330891, + "qps_1t_runs": [ + 500.5437657198895, + 396.93923332695437, + 357.8120295330891 + ], + "recall": 1.0 + } + ], + "system": "rust_exact", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r3.load.json b/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r3.load.json new file mode 100644 index 0000000..c77473d --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r3.load.json @@ -0,0 +1 @@ +{"total_pct": 11.55625, "pin_cpu_pct": 0.0, "max_cpu_pct": 62.7, "noisy": false, "seconds": 12.619571447372437, "after": 15.3} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r3.txt b/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r3.txt new file mode 100644 index 0000000..9552072 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_sift_100000_r3.txt @@ -0,0 +1,4 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/sift --ds sift --n 100000 --mode exact --exp core --queries 1000 --budget 3 --label rust_exact --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_exact_sift_100000_r3.json +rust_exact: ds=sift n=100000 d=128 mode=exact int8=0 queries=1000 cpus=32 exps=["core"] + ingest 0.05s rss 70 MB + exact 1T recall=1.0000 qps=396.9 median=2.575ms | auto 645.0 qps | batch 11457 qps diff --git a/benchmarks/rust-stage-c/raw/rust_exact_text384_1000000_persist_r1.json b/benchmarks/rust-stage-c/raw/rust_exact_text384_1000000_persist_r1.json new file mode 100644 index 0000000..64b8a1b --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_text384_1000000_persist_r1.json @@ -0,0 +1,64 @@ +{ + "all_threads": 32, + "budget": 3.0, + "d": 384, + "ds": "text384", + "exps": [ + "persist" + ], + "ingest_s": 1.0799625000000006, + "int8_rerank": 0, + "k": 10, + "mem_after_ingest": { + "peak_mb": 1656.76171875, + "rss_mb": 1629.4921875 + }, + "mem_baseline": { + "peak_mb": 7.87890625, + "rss_mb": 7.87890625 + }, + "mem_end": { + "peak_mb": 11505.41796875, + "rss_mb": 23.1171875 + }, + "mode": "exact", + "n": 1000000, + "persist": { + "json_answers_equal_original": true, + "json_first_query_ms": 50.2532, + "json_load_peak_mb": 11505.41796875, + "json_load_s": 30.1224564, + "json_load_s_runs": [ + 30.1224564, + 30.4150109, + 26.3797141 + ], + "json_mb": 4534.09522819519, + "json_save_s": 42.8566, + "sidecar_fingerprint_s": 0.0033589, + "snapshot_answers_equal_original": true, + "snapshot_first_query_ms": 43.6024, + "snapshot_load_s": 4.5606966, + "snapshot_load_s_runs": [ + 4.395748, + 4.6875271, + 4.5606966 + ], + "snapshot_mb": 1488.5797986984253, + "snapshot_save_s": 7.6825872, + "time_to_ready_after_snapshot_load": { + "ivf": { + "build_s": 17.03469, + "first_query_ms": 31.1386, + "load_s": 4.5588184, + "ready_s": 21.624647 + } + } + }, + "queries": 1000, + "raw_vectors_mb": 1464.84375, + "started": "SystemTime { intervals: 134393560973031643 }", + "status": "ok", + "system": "rust_exact", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_exact_text384_1000000_persist_r1.load.json b/benchmarks/rust-stage-c/raw/rust_exact_text384_1000000_persist_r1.load.json new file mode 100644 index 0000000..b6f3c81 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_text384_1000000_persist_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 14.896875, "pin_cpu_pct": 0.0, "max_cpu_pct": 56.4, "noisy": false, "seconds": 184.654887676239, "after": 19.1} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_exact_text384_1000000_persist_r1.txt b/benchmarks/rust-stage-c/raw/rust_exact_text384_1000000_persist_r1.txt new file mode 100644 index 0000000..fcde79d --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_text384_1000000_persist_r1.txt @@ -0,0 +1,7 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/text384 --ds text384 --n 1000000 --mode exact --exp persist --queries 1000 --budget 3 --label rust_exact --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_exact_text384_1000000_persist_r1.json --ready ivf +rust_exact: ds=text384 n=1000000 d=384 mode=exact int8=0 queries=1000 cpus=32 exps=["persist"] + save: JSON 42.86s 4534 MB | snapshot 7.68s 1489 MB (+ sidecar fingerprint 0.00s) + load rep 0: JSON 30.12s (first query 37.9 ms) | snapshot 4.40s (first query 34.6 ms) | answers equal ref: json=true snap=true + load rep 1: JSON 30.42s (first query 41.0 ms) | snapshot 4.69s (first query 37.1 ms) | answers equal ref: json=true snap=true + load rep 2: JSON 26.38s (first query 50.3 ms) | snapshot 4.56s (first query 43.6 ms) | answers equal ref: json=true snap=true + ready ivf: load 4.56s + build 17.03s + first query 31.1 ms = 21.62s diff --git a/benchmarks/rust-stage-c/raw/rust_exact_text384_1000000_r1.json b/benchmarks/rust-stage-c/raw/rust_exact_text384_1000000_r1.json new file mode 100644 index 0000000..a303f6b --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_text384_1000000_r1.json @@ -0,0 +1,405 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_exact_query_ms": 192.6696, + "ingest_s": 1.1387568000000001 + }, + "concurrent_clients": [ + { + "clients": 1, + "qps": 4.929922790835865 + }, + { + "clients": 2, + "qps": 8.229442685998936 + }, + { + "clients": 4, + "qps": 17.26859861991827 + }, + { + "clients": 8, + "qps": 37.009953617838825 + }, + { + "clients": 16, + "qps": 60.15478766785496 + } + ], + "d": 384, + "ds": "text384", + "exps": [ + "core", + "int8", + "conc" + ], + "first_exact_query_ms": 192.6696, + "int8_rerank": 0, + "int8_study": { + "by_factor": [ + { + "applies": true, + "auto_threads": { + "max_ms": 18.2565, + "mean_ms": 12.807307304116852, + "median_ms": 12.7149, + "nq": 251, + "p95_ms": 14.798, + "p99_ms": 16.0047, + "qps_1t": 78.98177824295567, + "qps_1t_max": 79.12257540137136, + "qps_1t_min": 76.20699045205352, + "qps_1t_runs": [ + 79.12257540137136, + 76.20699045205352, + 78.98177824295567 + ], + "recall": 0.9912350597609562 + }, + "batch": { + "nq": 1000, + "qps_batch": 547.2865124249089, + "qps_batch_max": 553.0399917044002, + "qps_batch_min": 544.2834157768494, + "qps_batch_runs": [ + 553.0399917044002, + 544.2834157768494, + 547.2865124249089 + ], + "recall": 0.9899, + "threads": 32 + }, + "min_recall_vs_f32_scan": 0.9, + "one_thread": { + "max_ms": 32.5515, + "mean_ms": 30.28803780068726, + "median_ms": 30.857699999999998, + "nq": 97, + "p95_ms": 31.506100000000004, + "p99_ms": 32.0524, + "qps_1t": 33.047523360681005, + "qps_1t_max": 33.199058543151395, + "qps_1t_min": 32.80482172710014, + "qps_1t_runs": [ + 33.199058543151395, + 33.047523360681005, + 32.80482172710014 + ], + "recall": 0.9907216494845361 + }, + "recall_vs_f32_scan": 0.9916666666666667, + "report_f32_ms": 39.077799999999996, + "report_int8_ms": 12.10835, + "report_queries": 300, + "rerank_factor": 1, + "scores_exact": true + }, + { + "applies": true, + "auto_threads": { + "max_ms": 18.7873, + "mean_ms": 12.45522673733804, + "median_ms": 12.4061, + "nq": 283, + "p95_ms": 14.288, + "p99_ms": 15.4891, + "qps_1t": 80.01114840849483, + "qps_1t_max": 82.01887222369744, + "qps_1t_min": 78.89480367599602, + "qps_1t_runs": [ + 82.01887222369744, + 78.89480367599602, + 80.01114840849483 + ], + "recall": 1.0 + }, + "batch": { + "nq": 1000, + "qps_batch": 548.6725799671082, + "qps_batch_max": 551.2159548355695, + "qps_batch_min": 541.3969720208752, + "qps_batch_runs": [ + 548.6725799671082, + 551.2159548355695, + 541.3969720208752 + ], + "recall": 1.0, + "threads": 32 + }, + "min_recall_vs_f32_scan": 1.0, + "one_thread": { + "max_ms": 49.4542, + "mean_ms": 28.394935374149643, + "median_ms": 27.8479, + "nq": 98, + "p95_ms": 31.369300000000003, + "p99_ms": 31.628200000000003, + "qps_1t": 36.17089181437585, + "qps_1t_max": 36.298427148408535, + "qps_1t_min": 33.34571207828646, + "qps_1t_runs": [ + 33.34571207828646, + 36.298427148408535, + 36.17089181437585 + ], + "recall": 1.0 + }, + "recall_vs_f32_scan": 1.0, + "report_f32_ms": 39.077799999999996, + "report_int8_ms": 12.97385, + "report_queries": 300, + "rerank_factor": 2, + "scores_exact": true + }, + { + "applies": true, + "auto_threads": { + "max_ms": 18.0495, + "mean_ms": 12.604676537911319, + "median_ms": 12.5652, + "nq": 233, + "p95_ms": 14.5899, + "p99_ms": 16.043100000000003, + "qps_1t": 79.53113630332858, + "qps_1t_max": 81.52664343192318, + "qps_1t_min": 77.07480155802244, + "qps_1t_runs": [ + 79.53113630332858, + 81.52664343192318, + 77.07480155802244 + ], + "recall": 1.0 + }, + "batch": { + "nq": 1000, + "qps_batch": 536.9583318723592, + "qps_batch_max": 540.8052135353162, + "qps_batch_min": 534.5763445049454, + "qps_batch_runs": [ + 536.9583318723592, + 534.5763445049454, + 540.8052135353162 + ], + "recall": 1.0, + "threads": 32 + }, + "min_recall_vs_f32_scan": 1.0, + "one_thread": { + "max_ms": 38.0428, + "mean_ms": 30.58792569444443, + "median_ms": 30.9771, + "nq": 96, + "p95_ms": 32.698100000000004, + "p99_ms": 34.158300000000004, + "qps_1t": 32.97960644198896, + "qps_1t_max": 33.174153837052195, + "qps_1t_min": 31.950862768148866, + "qps_1t_runs": [ + 32.97960644198896, + 31.950862768148866, + 33.174153837052195 + ], + "recall": 1.0 + }, + "recall_vs_f32_scan": 1.0, + "report_f32_ms": 39.077799999999996, + "report_int8_ms": 12.642850000000001, + "report_queries": 300, + "rerank_factor": 4, + "scores_exact": true + }, + { + "applies": true, + "auto_threads": { + "max_ms": 17.711399999999998, + "mean_ms": 12.477422916666669, + "median_ms": 12.4673, + "nq": 256, + "p95_ms": 14.365599999999999, + "p99_ms": 15.601, + "qps_1t": 80.68186004842612, + "qps_1t_max": 81.50780265279879, + "qps_1t_min": 78.31377951446748, + "qps_1t_runs": [ + 81.50780265279879, + 80.68186004842612, + 78.31377951446748 + ], + "recall": 1.0 + }, + "batch": { + "nq": 1000, + "qps_batch": 519.7170244745141, + "qps_batch_max": 520.4530231294528, + "qps_batch_min": 516.2989379111289, + "qps_batch_runs": [ + 520.4530231294528, + 519.7170244745141, + 516.2989379111289 + ], + "recall": 1.0, + "threads": 32 + }, + "min_recall_vs_f32_scan": 1.0, + "one_thread": { + "max_ms": 36.7782, + "mean_ms": 30.695418181818184, + "median_ms": 30.9159, + "nq": 99, + "p95_ms": 32.977, + "p99_ms": 34.6354, + "qps_1t": 32.797795034056016, + "qps_1t_max": 33.29963791184632, + "qps_1t_min": 31.67961173467858, + "qps_1t_runs": [ + 32.797795034056016, + 31.67961173467858, + 33.29963791184632 + ], + "recall": 1.0 + }, + "recall_vs_f32_scan": 1.0, + "report_f32_ms": 39.077799999999996, + "report_int8_ms": 13.125850000000002, + "report_queries": 300, + "rerank_factor": 8, + "scores_exact": true + } + ], + "f32": { + "auto_threads": { + "max_ms": 43.4613, + "mean_ms": 39.52294692982457, + "median_ms": 39.557, + "nq": 76, + "p95_ms": 41.8667, + "p99_ms": 42.912800000000004, + "qps_1t": 25.23345678802066, + "qps_1t_max": 25.549543792724887, + "qps_1t_min": 25.12608667018785, + "qps_1t_runs": [ + 25.12608667018785, + 25.23345678802066, + 25.549543792724887 + ], + "recall": 1.0 + }, + "batch": { + "nq": 1000, + "qps_batch": 559.8880626994006, + "qps_batch_max": 560.9546505748972, + "qps_batch_min": 558.7020100813307, + "qps_batch_runs": [ + 559.8880626994006, + 560.9546505748972, + 558.7020100813307 + ], + "recall": 1.0, + "threads": 32 + }, + "mem": { + "peak_mb": 1656.38671875, + "rss_mb": 1634.984375 + }, + "one_thread": { + "max_ms": 94.43769999999999, + "mean_ms": 90.09285625000001, + "median_ms": 91.7648, + "nq": 32, + "p95_ms": 94.1585, + "p99_ms": 94.3586, + "qps_1t": 11.123041345387467, + "qps_1t_max": 11.213182217014321, + "qps_1t_min": 10.96559161834039, + "qps_1t_runs": [ + 11.123041345387467, + 11.213182217014321, + 10.96559161834039 + ], + "recall": 1.0 + } + }, + "mem_after_int8": { + "peak_mb": 2005.66796875, + "rss_mb": 2005.66796875 + }, + "quant_build_s": 2.2619151, + "quant_bytes_mb": 370.375, + "rss_delta_mb": 370.68359375 + }, + "k": 10, + "knob_name": "", + "mem_after_first_exact_query": { + "peak_mb": 2032.296875, + "rss_mb": 1645.29296875 + }, + "mem_after_ingest": { + "peak_mb": 1656.38671875, + "rss_mb": 1628.953125 + }, + "mem_after_sweep": { + "peak_mb": 2032.296875, + "rss_mb": 1650.08203125 + }, + "mem_baseline": { + "peak_mb": 7.86328125, + "rss_mb": 7.86328125 + }, + "mem_end": { + "peak_mb": 2032.296875, + "rss_mb": 1643.859375 + }, + "mode": "exact", + "n": 1000000, + "queries": 1000, + "raw_vectors_mb": 1464.84375, + "ref_knob": null, + "started": "SystemTime { intervals: 134393551448970200 }", + "status": "ok", + "sweep": [ + { + "auto_threads": { + "median_ms": 39.5732, + "p99_ms": 46.235699999999994, + "qps": 25.18870453854444, + "runs": [ + 25.18870453854444, + 25.34189168436234, + 24.873283884360603 + ] + }, + "batch": { + "nq": 1000, + "qps_batch": 560.2943069111799, + "qps_batch_max": 561.0075516104507, + "qps_batch_min": 557.1075643786537, + "qps_batch_runs": [ + 561.0075516104507, + 560.2943069111799, + 557.1075643786537 + ], + "recall": 1.0, + "threads": 32 + }, + "knob": null, + "max_ms": 102.7707, + "mean_ms": 90.02171354166666, + "median_ms": 90.4063, + "nq": 32, + "p95_ms": 93.31490000000001, + "p99_ms": 99.2705, + "qps_1t": 11.07807843528789, + "qps_1t_max": 11.19283591344524, + "qps_1t_min": 11.055353359668334, + "qps_1t_runs": [ + 11.055353359668334, + 11.19283591344524, + 11.07807843528789 + ], + "recall": 1.0 + } + ], + "system": "rust_exact", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_exact_text384_1000000_r1.load.json b/benchmarks/rust-stage-c/raw/rust_exact_text384_1000000_r1.load.json new file mode 100644 index 0000000..78ddd1e --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_text384_1000000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 10.746875, "pin_cpu_pct": 0.0, "max_cpu_pct": 56.4, "noisy": false, "seconds": 241.99774646759033, "after": 29.8} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_exact_text384_1000000_r1.txt b/benchmarks/rust-stage-c/raw/rust_exact_text384_1000000_r1.txt new file mode 100644 index 0000000..bf319fd --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_text384_1000000_r1.txt @@ -0,0 +1,18 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/text384 --ds text384 --n 1000000 --mode exact --exp core,int8,conc --queries 1000 --budget 3 --label rust_exact --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_exact_text384_1000000_r1.json +rust_exact: ds=text384 n=1000000 d=384 mode=exact int8=0 queries=1000 cpus=32 exps=["core", "int8", "conc"] + ingest 1.14s rss 1629 MB + int8 study: f32 1T 11.1 qps, auto 25.2 qps, batch 560 qps + rerank x1: recall vs f32 0.9917 (worst 0.90) exact_scores=true applies=true int8 12.108 ms f32 39.078 ms + 1T 33.0 qps (recall vs float64 truth 0.9907), auto 79.0 qps, batch 547 qps + rerank x2: recall vs f32 1.0000 (worst 1.00) exact_scores=true applies=true int8 12.974 ms f32 39.078 ms + 1T 36.2 qps (recall vs float64 truth 1.0000), auto 80.0 qps, batch 549 qps + rerank x4: recall vs f32 1.0000 (worst 1.00) exact_scores=true applies=true int8 12.643 ms f32 39.078 ms + 1T 33.0 qps (recall vs float64 truth 1.0000), auto 79.5 qps, batch 537 qps + rerank x8: recall vs f32 1.0000 (worst 1.00) exact_scores=true applies=true int8 13.126 ms f32 39.078 ms + 1T 32.8 qps (recall vs float64 truth 1.0000), auto 80.7 qps, batch 520 qps + exact 1T recall=1.0000 qps=11.1 median=90.406ms | auto 25.2 qps | batch 560 qps + concurrent clients=1 (1 scan thread each): 4.9 qps + concurrent clients=2 (1 scan thread each): 8.2 qps + concurrent clients=4 (1 scan thread each): 17.3 qps + concurrent clients=8 (1 scan thread each): 37.0 qps + concurrent clients=16 (1 scan thread each): 60.2 qps diff --git a/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_persist_r1.json b/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_persist_r1.json new file mode 100644 index 0000000..2c78d3e --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_persist_r1.json @@ -0,0 +1,70 @@ +{ + "all_threads": 32, + "budget": 3.0, + "d": 384, + "ds": "text384", + "exps": [ + "persist" + ], + "ingest_s": 0.09835829999999998, + "int8_rerank": 0, + "k": 10, + "mem_after_ingest": { + "peak_mb": 196.3515625, + "rss_mb": 169.05859375 + }, + "mem_baseline": { + "peak_mb": 7.8671875, + "rss_mb": 7.8671875 + }, + "mem_end": { + "peak_mb": 849.90625, + "rss_mb": 32.18359375 + }, + "mode": "exact", + "n": 100000, + "persist": { + "json_answers_equal_original": true, + "json_first_query_ms": 3.7659, + "json_load_peak_mb": 849.90625, + "json_load_s": 1.2717076, + "json_load_s_runs": [ + 1.2658816, + 1.2717076, + 1.2847952 + ], + "json_mb": 453.31162452697754, + "json_save_s": 1.6749711999999999, + "sidecar_fingerprint_s": 0.0009438, + "snapshot_answers_equal_original": true, + "snapshot_first_query_ms": 4.0445, + "snapshot_load_s": 0.1926398, + "snapshot_load_s_runs": [ + 0.1926398, + 0.1924933, + 0.1992911 + ], + "snapshot_mb": 148.76275157928467, + "snapshot_save_s": 0.1851224, + "time_to_ready_after_snapshot_load": { + "hnsw": { + "build_s": 5.9335669, + "first_query_ms": 0.235, + "load_s": 0.1864241, + "ready_s": 6.120226 + }, + "ivf": { + "build_s": 0.503499, + "first_query_ms": 0.5793, + "load_s": 0.2012672, + "ready_s": 0.7053455 + } + } + }, + "queries": 1000, + "raw_vectors_mb": 146.484375, + "started": "SystemTime { intervals: 134393586383314318 }", + "status": "ok", + "system": "rust_exact", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_persist_r1.load.json b/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_persist_r1.load.json new file mode 100644 index 0000000..ff06a75 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_persist_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 13.28125, "pin_cpu_pct": 0.0, "max_cpu_pct": 96.5, "noisy": false, "seconds": 14.082648038864136, "after": 45.3} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_persist_r1.txt b/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_persist_r1.txt new file mode 100644 index 0000000..ff84975 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_persist_r1.txt @@ -0,0 +1,8 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/text384 --ds text384 --n 100000 --mode exact --exp persist --queries 1000 --budget 3 --label rust_exact --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_exact_text384_100000_persist_r1.json --ready ivf,hnsw +rust_exact: ds=text384 n=100000 d=384 mode=exact int8=0 queries=1000 cpus=32 exps=["persist"] + save: JSON 1.67s 453 MB | snapshot 0.19s 149 MB (+ sidecar fingerprint 0.00s) + load rep 0: JSON 1.27s (first query 3.8 ms) | snapshot 0.19s (first query 4.2 ms) | answers equal ref: json=true snap=true + load rep 1: JSON 1.27s (first query 3.9 ms) | snapshot 0.19s (first query 3.7 ms) | answers equal ref: json=true snap=true + load rep 2: JSON 1.28s (first query 3.8 ms) | snapshot 0.20s (first query 4.0 ms) | answers equal ref: json=true snap=true + ready ivf: load 0.20s + build 0.50s + first query 0.6 ms = 0.71s + ready hnsw: load 0.19s + build 5.93s + first query 0.2 ms = 6.12s diff --git a/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r1.json b/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r1.json new file mode 100644 index 0000000..96bdc33 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r1.json @@ -0,0 +1,405 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_exact_query_ms": 16.368199999999998, + "ingest_s": 0.1100465 + }, + "concurrent_clients": [ + { + "clients": 1, + "qps": 56.990295447300426 + }, + { + "clients": 2, + "qps": 82.77841717666725 + }, + { + "clients": 4, + "qps": 183.1750528320655 + }, + { + "clients": 8, + "qps": 340.82928090979584 + }, + { + "clients": 16, + "qps": 603.8156581251359 + } + ], + "d": 384, + "ds": "text384", + "exps": [ + "core", + "int8", + "conc" + ], + "first_exact_query_ms": 16.368199999999998, + "int8_rerank": 0, + "int8_study": { + "by_factor": [ + { + "applies": true, + "auto_threads": { + "max_ms": 2.6828, + "mean_ms": 0.9062141666666662, + "median_ms": 0.856, + "nq": 1000, + "p95_ms": 1.3541, + "p99_ms": 1.6987, + "qps_1t": 1105.3870485115679, + "qps_1t_max": 1224.806045838612, + "qps_1t_min": 1002.4799348628636, + "qps_1t_runs": [ + 1224.806045838612, + 1002.4799348628636, + 1105.3870485115679 + ], + "recall": 0.9905 + }, + "batch": { + "nq": 1000, + "qps_batch": 5105.669488570193, + "qps_batch_max": 5150.526718614879, + "qps_batch_min": 5001.745609217617, + "qps_batch_runs": [ + 5001.745609217617, + 5105.669488570193, + 5150.526718614879 + ], + "recall": 0.9905, + "threads": 32 + }, + "min_recall_vs_f32_scan": 0.8, + "one_thread": { + "max_ms": 14.1748, + "mean_ms": 2.9078288666666663, + "median_ms": 2.7335, + "nq": 1000, + "p95_ms": 4.0469, + "p99_ms": 5.5261, + "qps_1t": 354.1004889738238, + "qps_1t_max": 363.93137302427067, + "qps_1t_min": 317.2932399057045, + "qps_1t_runs": [ + 363.93137302427067, + 317.2932399057045, + 354.1004889738238 + ], + "recall": 0.9905 + }, + "recall_vs_f32_scan": 0.9903999999999983, + "report_f32_ms": 4.1552, + "report_int8_ms": 0.6925, + "report_queries": 1000, + "rerank_factor": 1, + "scores_exact": true + }, + { + "applies": true, + "auto_threads": { + "max_ms": 2.7036000000000002, + "mean_ms": 0.8003255000000007, + "median_ms": 0.748, + "nq": 1000, + "p95_ms": 1.2464, + "p99_ms": 1.6523, + "qps_1t": 1216.2758630419871, + "qps_1t_max": 1344.988103580223, + "qps_1t_min": 1197.1838405082942, + "qps_1t_runs": [ + 1197.1838405082942, + 1344.988103580223, + 1216.2758630419871 + ], + "recall": 0.9999 + }, + "batch": { + "nq": 1000, + "qps_batch": 4780.5237637446035, + "qps_batch_max": 4866.492640889828, + "qps_batch_min": 4723.375513076665, + "qps_batch_runs": [ + 4780.5237637446035, + 4723.375513076665, + 4866.492640889828 + ], + "recall": 0.9999, + "threads": 32 + }, + "min_recall_vs_f32_scan": 1.0, + "one_thread": { + "max_ms": 9.2075, + "mean_ms": 2.8787289112227774, + "median_ms": 2.7604, + "nq": 995, + "p95_ms": 3.8167, + "p99_ms": 4.7219999999999995, + "qps_1t": 358.5227192063757, + "qps_1t_max": 372.3883924853898, + "qps_1t_min": 316.2960593420998, + "qps_1t_runs": [ + 358.5227192063757, + 316.2960593420998, + 372.3883924853898 + ], + "recall": 0.9998994974874372 + }, + "recall_vs_f32_scan": 1.0, + "report_f32_ms": 4.1552, + "report_int8_ms": 0.7208, + "report_queries": 1000, + "rerank_factor": 2, + "scores_exact": true + }, + { + "applies": true, + "auto_threads": { + "max_ms": 2.3789000000000002, + "mean_ms": 0.8121089999999985, + "median_ms": 0.7886, + "nq": 1000, + "p95_ms": 1.0944, + "p99_ms": 1.3117999999999999, + "qps_1t": 1225.847133920368, + "qps_1t_max": 1242.6716545849988, + "qps_1t_min": 1225.7203190599007, + "qps_1t_runs": [ + 1225.7203190599007, + 1242.6716545849988, + 1225.847133920368 + ], + "recall": 0.9999 + }, + "batch": { + "nq": 1000, + "qps_batch": 4362.034941644696, + "qps_batch_max": 4376.951846964256, + "qps_batch_min": 4211.696724142288, + "qps_batch_runs": [ + 4211.696724142288, + 4362.034941644696, + 4376.951846964256 + ], + "recall": 0.9999, + "threads": 32 + }, + "min_recall_vs_f32_scan": 1.0, + "one_thread": { + "max_ms": 10.8589, + "mean_ms": 2.703215500000003, + "median_ms": 2.6065, + "nq": 1000, + "p95_ms": 3.3514999999999997, + "p99_ms": 4.1691, + "qps_1t": 364.01354785622357, + "qps_1t_max": 386.55335499320665, + "qps_1t_min": 360.291418110625, + "qps_1t_runs": [ + 386.55335499320665, + 364.01354785622357, + 360.291418110625 + ], + "recall": 0.9999 + }, + "recall_vs_f32_scan": 1.0, + "report_f32_ms": 4.1552, + "report_int8_ms": 0.73585, + "report_queries": 1000, + "rerank_factor": 4, + "scores_exact": true + }, + { + "applies": true, + "auto_threads": { + "max_ms": 3.2521999999999998, + "mean_ms": 1.0883916666666666, + "median_ms": 0.9602999999999999, + "nq": 1000, + "p95_ms": 1.9591999999999998, + "p99_ms": 2.3726, + "qps_1t": 1005.5015009120903, + "qps_1t_max": 1080.3200685873587, + "qps_1t_min": 743.4972982795175, + "qps_1t_runs": [ + 1080.3200685873587, + 743.4972982795175, + 1005.5015009120903 + ], + "recall": 0.9999 + }, + "batch": { + "nq": 1000, + "qps_batch": 3675.6327601796647, + "qps_batch_max": 3701.597905783969, + "qps_batch_min": 3666.1447298619587, + "qps_batch_runs": [ + 3666.1447298619587, + 3675.6327601796647, + 3701.597905783969 + ], + "recall": 0.9999, + "threads": 32 + }, + "min_recall_vs_f32_scan": 1.0, + "one_thread": { + "max_ms": 6.0645999999999995, + "mean_ms": 2.6938994333333297, + "median_ms": 2.6725, + "nq": 1000, + "p95_ms": 3.0995, + "p99_ms": 3.5435000000000003, + "qps_1t": 368.62387142576273, + "qps_1t_max": 381.3729808684242, + "qps_1t_min": 364.0598645478873, + "qps_1t_runs": [ + 368.62387142576273, + 381.3729808684242, + 364.0598645478873 + ], + "recall": 0.9999 + }, + "recall_vs_f32_scan": 1.0, + "report_f32_ms": 4.1552, + "report_int8_ms": 0.8069999999999999, + "report_queries": 1000, + "rerank_factor": 8, + "scores_exact": true + } + ], + "f32": { + "auto_threads": { + "max_ms": 7.629300000000001, + "mean_ms": 4.3308811294765865, + "median_ms": 4.1784, + "nq": 726, + "p95_ms": 5.3377, + "p99_ms": 6.023, + "qps_1t": 232.53626244465352, + "qps_1t_max": 233.511318705318, + "qps_1t_min": 226.76810718347647, + "qps_1t_runs": [ + 232.53626244465352, + 226.76810718347647, + 233.511318705318 + ], + "recall": 0.999862258953168 + }, + "batch": { + "nq": 1000, + "qps_batch": 5559.870632930113, + "qps_batch_max": 5589.155696551324, + "qps_batch_min": 5382.386248649022, + "qps_batch_runs": [ + 5382.386248649022, + 5589.155696551324, + 5559.870632930113 + ], + "recall": 0.9999, + "threads": 32 + }, + "mem": { + "peak_mb": 196.25390625, + "rss_mb": 173.16015625 + }, + "one_thread": { + "max_ms": 17.7622, + "mean_ms": 8.117735714285702, + "median_ms": 8.3376, + "nq": 434, + "p95_ms": 9.314, + "p99_ms": 10.3004, + "qps_1t": 116.85410756592924, + "qps_1t_max": 138.90356693478049, + "qps_1t_min": 116.32927424578163, + "qps_1t_runs": [ + 138.90356693478049, + 116.85410756592924, + 116.32927424578163 + ], + "recall": 1.0 + } + }, + "mem_after_int8": { + "peak_mb": 210.7578125, + "rss_mb": 210.7578125 + }, + "quant_build_s": 0.204982, + "quant_bytes_mb": 37.25, + "rss_delta_mb": 37.59765625 + }, + "k": 10, + "knob_name": "", + "mem_after_first_exact_query": { + "peak_mb": 238.4375, + "rss_mb": 182.7578125 + }, + "mem_after_ingest": { + "peak_mb": 196.25390625, + "rss_mb": 169.00390625 + }, + "mem_after_sweep": { + "peak_mb": 238.4375, + "rss_mb": 182.4609375 + }, + "mem_baseline": { + "peak_mb": 7.875, + "rss_mb": 7.875 + }, + "mem_end": { + "peak_mb": 238.4375, + "rss_mb": 178.13671875 + }, + "mode": "exact", + "n": 100000, + "queries": 1000, + "raw_vectors_mb": 146.484375, + "ref_knob": null, + "started": "SystemTime { intervals: 134393511981216549 }", + "status": "ok", + "sweep": [ + { + "auto_threads": { + "median_ms": 4.3793, + "p99_ms": 6.916099999999999, + "qps": 215.62869058930434, + "runs": [ + 215.62869058930434, + 204.2992178114394, + 239.28255409607416 + ] + }, + "batch": { + "nq": 1000, + "qps_batch": 5319.762972640991, + "qps_batch_max": 5329.567105911822, + "qps_batch_min": 4877.910771305007, + "qps_batch_runs": [ + 5319.762972640991, + 4877.910771305007, + 5329.567105911822 + ], + "recall": 0.9999, + "threads": 32 + }, + "knob": null, + "max_ms": 14.3822, + "mean_ms": 8.788671250000002, + "median_ms": 8.8023, + "nq": 320, + "p95_ms": 9.4652, + "p99_ms": 10.5773, + "qps_1t": 113.97457184561665, + "qps_1t_max": 114.17387689565653, + "qps_1t_min": 113.20469435811496, + "qps_1t_runs": [ + 113.97457184561665, + 113.20469435811496, + 114.17387689565653 + ], + "recall": 1.0 + } + ], + "system": "rust_exact", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r1.load.json b/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r1.load.json new file mode 100644 index 0000000..f339245 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 16.903125, "pin_cpu_pct": 0.0, "max_cpu_pct": 74.1, "noisy": false, "seconds": 115.10153079032898, "after": 33.4} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r1.txt b/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r1.txt new file mode 100644 index 0000000..0f7f872 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r1.txt @@ -0,0 +1,18 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/text384 --ds text384 --n 100000 --mode exact --exp core,int8,conc --queries 1000 --budget 3 --label rust_exact --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_exact_text384_100000_r1.json +rust_exact: ds=text384 n=100000 d=384 mode=exact int8=0 queries=1000 cpus=32 exps=["core", "int8", "conc"] + ingest 0.11s rss 169 MB + int8 study: f32 1T 116.9 qps, auto 232.5 qps, batch 5560 qps + rerank x1: recall vs f32 0.9904 (worst 0.80) exact_scores=true applies=true int8 0.693 ms f32 4.155 ms + 1T 354.1 qps (recall vs float64 truth 0.9905), auto 1105.4 qps, batch 5106 qps + rerank x2: recall vs f32 1.0000 (worst 1.00) exact_scores=true applies=true int8 0.721 ms f32 4.155 ms + 1T 358.5 qps (recall vs float64 truth 0.9999), auto 1216.3 qps, batch 4781 qps + rerank x4: recall vs f32 1.0000 (worst 1.00) exact_scores=true applies=true int8 0.736 ms f32 4.155 ms + 1T 364.0 qps (recall vs float64 truth 0.9999), auto 1225.8 qps, batch 4362 qps + rerank x8: recall vs f32 1.0000 (worst 1.00) exact_scores=true applies=true int8 0.807 ms f32 4.155 ms + 1T 368.6 qps (recall vs float64 truth 0.9999), auto 1005.5 qps, batch 3676 qps + exact 1T recall=1.0000 qps=114.0 median=8.802ms | auto 215.6 qps | batch 5320 qps + concurrent clients=1 (1 scan thread each): 57.0 qps + concurrent clients=2 (1 scan thread each): 82.8 qps + concurrent clients=4 (1 scan thread each): 183.2 qps + concurrent clients=8 (1 scan thread each): 340.8 qps + concurrent clients=16 (1 scan thread each): 603.8 qps diff --git a/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r2.json b/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r2.json new file mode 100644 index 0000000..489c69b --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r2.json @@ -0,0 +1,89 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_exact_query_ms": 6.774299999999999, + "ingest_s": 0.10413099999999999 + }, + "d": 384, + "ds": "text384", + "exps": [ + "core" + ], + "first_exact_query_ms": 6.774299999999999, + "int8_rerank": 0, + "k": 10, + "knob_name": "", + "mem_after_first_exact_query": { + "peak_mb": 196.2734375, + "rss_mb": 171.48046875 + }, + "mem_after_ingest": { + "peak_mb": 196.2734375, + "rss_mb": 168.99609375 + }, + "mem_after_sweep": { + "peak_mb": 196.2734375, + "rss_mb": 173.4453125 + }, + "mem_baseline": { + "peak_mb": 7.86328125, + "rss_mb": 7.86328125 + }, + "mem_end": { + "peak_mb": 196.2734375, + "rss_mb": 173.48046875 + }, + "mode": "exact", + "n": 100000, + "queries": 1000, + "raw_vectors_mb": 146.484375, + "ref_knob": null, + "started": "SystemTime { intervals: 134393576601872412 }", + "status": "ok", + "sweep": [ + { + "auto_threads": { + "median_ms": 3.5691, + "p99_ms": 6.9918000000000005, + "qps": 284.06871622245416, + "runs": [ + 241.9578239010058, + 284.06871622245416, + 293.8335755088641 + ] + }, + "batch": { + "nq": 1000, + "qps_batch": 5206.188492136833, + "qps_batch_max": 5318.130570741772, + "qps_batch_min": 5124.938308555111, + "qps_batch_runs": [ + 5206.188492136833, + 5318.130570741772, + 5124.938308555111 + ], + "recall": 0.9999, + "threads": 32 + }, + "knob": null, + "max_ms": 22.3915, + "mean_ms": 6.967506244725744, + "median_ms": 6.6138, + "nq": 395, + "p95_ms": 8.660400000000001, + "p99_ms": 11.6247, + "qps_1t": 145.7347640576308, + "qps_1t_max": 147.9401247738248, + "qps_1t_min": 137.33912025914046, + "qps_1t_runs": [ + 145.7347640576308, + 137.33912025914046, + 147.9401247738248 + ], + "recall": 1.0 + } + ], + "system": "rust_exact", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r2.load.json b/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r2.load.json new file mode 100644 index 0000000..043ee0b --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r2.load.json @@ -0,0 +1 @@ +{"total_pct": 15.546875, "pin_cpu_pct": 0.0, "max_cpu_pct": 79.5, "noisy": false, "seconds": 18.377028942108154, "after": 22.9} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r2.txt b/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r2.txt new file mode 100644 index 0000000..c44133f --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r2.txt @@ -0,0 +1,4 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/text384 --ds text384 --n 100000 --mode exact --exp core --queries 1000 --budget 3 --label rust_exact --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_exact_text384_100000_r2.json +rust_exact: ds=text384 n=100000 d=384 mode=exact int8=0 queries=1000 cpus=32 exps=["core"] + ingest 0.10s rss 169 MB + exact 1T recall=1.0000 qps=145.7 median=6.614ms | auto 284.1 qps | batch 5206 qps diff --git a/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r3.json b/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r3.json new file mode 100644 index 0000000..a9959b2 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r3.json @@ -0,0 +1,89 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_exact_query_ms": 6.902100000000001, + "ingest_s": 0.10387149999999999 + }, + "d": 384, + "ds": "text384", + "exps": [ + "core" + ], + "first_exact_query_ms": 6.902100000000001, + "int8_rerank": 0, + "k": 10, + "knob_name": "", + "mem_after_first_exact_query": { + "peak_mb": 196.23046875, + "rss_mb": 171.46484375 + }, + "mem_after_ingest": { + "peak_mb": 196.23046875, + "rss_mb": 168.98046875 + }, + "mem_after_sweep": { + "peak_mb": 196.23046875, + "rss_mb": 172.1796875 + }, + "mem_baseline": { + "peak_mb": 7.85546875, + "rss_mb": 7.85546875 + }, + "mem_end": { + "peak_mb": 196.23046875, + "rss_mb": 172.203125 + }, + "mode": "exact", + "n": 100000, + "queries": 1000, + "raw_vectors_mb": 146.484375, + "ref_knob": null, + "started": "SystemTime { intervals: 134393579896618321 }", + "status": "ok", + "sweep": [ + { + "auto_threads": { + "median_ms": 3.5449, + "p99_ms": 6.243, + "qps": 268.7639789184759, + "runs": [ + 257.66038495801337, + 278.13183187386926, + 268.7639789184759 + ] + }, + "batch": { + "nq": 1000, + "qps_batch": 12743.187810376268, + "qps_batch_max": 13796.433621908736, + "qps_batch_min": 12697.542009817738, + "qps_batch_runs": [ + 12743.187810376268, + 12697.542009817738, + 13796.433621908736 + ], + "recall": 0.9999, + "threads": 32 + }, + "knob": null, + "max_ms": 11.884799999999998, + "mean_ms": 7.106885222502086, + "median_ms": 6.9072, + "nq": 397, + "p95_ms": 8.6987, + "p99_ms": 10.4641, + "qps_1t": 142.61781794451153, + "qps_1t_max": 146.77568390536598, + "qps_1t_min": 133.40819426061137, + "qps_1t_runs": [ + 146.77568390536598, + 133.40819426061137, + 142.61781794451153 + ], + "recall": 1.0 + } + ], + "system": "rust_exact", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r3.load.json b/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r3.load.json new file mode 100644 index 0000000..8202cfc --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r3.load.json @@ -0,0 +1 @@ +{"total_pct": 17.565625, "pin_cpu_pct": 83.1, "max_cpu_pct": 85.2, "noisy": true, "seconds": 20.83128786087036, "after": 25.1} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r3.txt b/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r3.txt new file mode 100644 index 0000000..203cf98 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_exact_text384_100000_r3.txt @@ -0,0 +1,4 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/text384 --ds text384 --n 100000 --mode exact --exp core --queries 1000 --budget 3 --label rust_exact --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_exact_text384_100000_r3.json +rust_exact: ds=text384 n=100000 d=384 mode=exact int8=0 queries=1000 cpus=32 exps=["core"] + ingest 0.10s rss 169 MB + exact 1T recall=1.0000 qps=142.6 median=6.907ms | auto 268.8 qps | batch 12743 qps diff --git a/benchmarks/rust-stage-c/raw/rust_hnsw_i8x4_text384_1000000_r1.json b/benchmarks/rust-stage-c/raw/rust_hnsw_i8x4_text384_1000000_r1.json new file mode 100644 index 0000000..54ef137 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_hnsw_i8x4_text384_1000000_r1.json @@ -0,0 +1,136 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_query_ms": 0.6648000000000001, + "index_s": 144.7168199, + "ingest_s": 1.1461203, + "stats_build_seconds": 144.7156143, + "total_s": 145.8629402 + }, + "d": 384, + "ds": "text384", + "exps": [ + "core" + ], + "first_exact_query_ms": 30.6539, + "index_stats": { + "ef_search_default": 192, + "graph_nodes": 1000000, + "index_bytes": 146214912, + "max_level": 5, + "nlist": null, + "trained_n": 1000000 + }, + "int8_rerank": 4, + "k": 10, + "knob_name": "ef_search", + "mem_after_build": { + "peak_mb": 2171.609375, + "rss_mb": 2163.57421875 + }, + "mem_after_first_exact_query": { + "peak_mb": 2000.55078125, + "rss_mb": 2000.55078125 + }, + "mem_after_ingest": { + "peak_mb": 1656.42578125, + "rss_mb": 1629.9375 + }, + "mem_after_sweep": { + "peak_mb": 2171.609375, + "rss_mb": 2163.77734375 + }, + "mem_baseline": { + "peak_mb": 7.85546875, + "rss_mb": 7.85546875 + }, + "mem_end": { + "peak_mb": 2171.609375, + "rss_mb": 2163.80078125 + }, + "memory_stats": { + "index_bytes": 146214912, + "matrix_bytes": 1536688128, + "metadata_bytes": 130484346, + "quant_bytes": 388366336, + "total_bytes": 1813387386 + }, + "mode": "hnsw", + "n": 1000000, + "quant_build_s": 1.1209916, + "quant_bytes_mb": 370.375, + "queries": 1000, + "raw_vectors_mb": 1464.84375, + "ref_knob": 192, + "ref_recall_pre": 0.9549, + "rss_delta_int8_mb": 368.86328125, + "started": "SystemTime { intervals: 134393573059240073 }", + "status": "ok", + "sweep": [ + { + "batch": { + "nq": 1000, + "qps_batch": 6295.62756074651, + "qps_batch_max": 6463.802383462491, + "qps_batch_min": 6251.4534629301315, + "qps_batch_runs": [ + 6463.802383462491, + 6295.62756074651, + 6251.4534629301315 + ], + "recall": 0.9549, + "threads": 32 + }, + "knob": 192, + "max_ms": 1.1743000000000001, + "mean_ms": 0.5585331333333338, + "median_ms": 0.5569000000000001, + "nq": 1000, + "p95_ms": 0.6959, + "p99_ms": 0.8284, + "qps_1t": 1784.152196746313, + "qps_1t_max": 1833.9063985177663, + "qps_1t_min": 1754.9247576010177, + "qps_1t_runs": [ + 1833.9063985177663, + 1784.152196746313, + 1754.9247576010177 + ], + "recall": 0.9549 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 4938.888661151245, + "qps_batch_max": 5074.787138053494, + "qps_batch_min": 4849.477080886368, + "qps_batch_runs": [ + 4849.477080886368, + 4938.888661151245, + 5074.787138053494 + ], + "recall": 0.9685, + "threads": 32 + }, + "knob": 256, + "max_ms": 5.1765, + "mean_ms": 0.7278126666666671, + "median_ms": 0.713, + "nq": 1000, + "p95_ms": 0.929, + "p99_ms": 1.2015, + "qps_1t": 1371.3318415985568, + "qps_1t_max": 1451.3979792766486, + "qps_1t_min": 1306.798357093104, + "qps_1t_runs": [ + 1306.798357093104, + 1371.3318415985568, + 1451.3979792766486 + ], + "recall": 0.9685 + } + ], + "system": "rust_hnsw_i8x4", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_hnsw_i8x4_text384_1000000_r1.load.json b/benchmarks/rust-stage-c/raw/rust_hnsw_i8x4_text384_1000000_r1.load.json new file mode 100644 index 0000000..4b1fd61 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_hnsw_i8x4_text384_1000000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 13.903125, "pin_cpu_pct": 4.6, "max_cpu_pct": 82.8, "noisy": false, "seconds": 153.93160438537598, "after": 91.3} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_hnsw_i8x4_text384_1000000_r1.txt b/benchmarks/rust-stage-c/raw/rust_hnsw_i8x4_text384_1000000_r1.txt new file mode 100644 index 0000000..3284b61 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_hnsw_i8x4_text384_1000000_r1.txt @@ -0,0 +1,6 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/text384 --ds text384 --n 1000000 --mode hnsw --exp core --queries 1000 --budget 3 --label rust_hnsw_i8x4 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_hnsw_i8x4_text384_1000000_r1.json --int8 4 --knobs 192,256 +rust_hnsw_i8x4: ds=text384 n=1000000 d=384 mode=hnsw int8=4 queries=1000 cpus=32 exps=["core"] + ingest 1.15s rss 1630 MB + hnsw build 144.72s (stats Some(144.7156143)) rss 2164 MB (peak 2172) + hnsw+int8x4 knob=192 recall=0.9549 qps1t=1784 median=0.557ms batch=6296 + hnsw+int8x4 knob=256 recall=0.9685 qps1t=1371 median=0.713ms batch=4939 diff --git a/benchmarks/rust-stage-c/raw/rust_hnsw_i8x4_text384_100000_r1.json b/benchmarks/rust-stage-c/raw/rust_hnsw_i8x4_text384_100000_r1.json new file mode 100644 index 0000000..566e645 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_hnsw_i8x4_text384_100000_r1.json @@ -0,0 +1,198 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_query_ms": 0.3595, + "index_s": 5.6328688, + "ingest_s": 0.0975015, + "stats_build_seconds": 5.6327837, + "total_s": 5.7303703 + }, + "d": 384, + "ds": "text384", + "exps": [ + "core" + ], + "first_exact_query_ms": 2.4186, + "index_stats": { + "ef_search_default": 96, + "graph_nodes": 100000, + "index_bytes": 14458880, + "max_level": 4, + "nlist": null, + "trained_n": 100000 + }, + "int8_rerank": 4, + "k": 10, + "knob_name": "ef_search", + "mem_after_build": { + "peak_mb": 247.109375, + "rss_mb": 245.6875 + }, + "mem_after_first_exact_query": { + "peak_mb": 207.73046875, + "rss_mb": 207.73046875 + }, + "mem_after_ingest": { + "peak_mb": 196.2421875, + "rss_mb": 168.98828125 + }, + "mem_after_sweep": { + "peak_mb": 248.05078125, + "rss_mb": 248.05078125 + }, + "mem_baseline": { + "peak_mb": 7.859375, + "rss_mb": 7.859375 + }, + "mem_end": { + "peak_mb": 248.07421875, + "rss_mb": 248.07421875 + }, + "memory_stats": { + "index_bytes": 14458880, + "matrix_bytes": 154140672, + "metadata_bytes": 12696122, + "quant_bytes": 39059456, + "total_bytes": 181295674 + }, + "mode": "hnsw", + "n": 100000, + "quant_build_s": 0.1089449, + "quant_bytes_mb": 37.25, + "queries": 1000, + "raw_vectors_mb": 146.484375, + "ref_knob": 128, + "ref_recall_pre": 0.9626, + "rss_delta_int8_mb": 36.98046875, + "started": "SystemTime { intervals: 134393576192811456 }", + "status": "ok", + "sweep": [ + { + "batch": { + "nq": 1000, + "qps_batch": 33876.027290527585, + "qps_batch_max": 35578.32568399331, + "qps_batch_min": 33752.99557835758, + "qps_batch_runs": [ + 35578.32568399331, + 33752.99557835758, + 33876.027290527585 + ], + "recall": 0.9145, + "threads": 32 + }, + "knob": 64, + "max_ms": 0.5509, + "mean_ms": 0.16577613333333346, + "median_ms": 0.1466, + "nq": 1000, + "p95_ms": 0.2882, + "p99_ms": 0.4052, + "qps_1t": 7033.842630432062, + "qps_1t_max": 7148.923515097102, + "qps_1t_min": 4645.1694418457255, + "qps_1t_runs": [ + 4645.1694418457255, + 7033.842630432062, + 7148.923515097102 + ], + "recall": 0.9145 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 26517.18048123379, + "qps_batch_max": 26518.375908585855, + "qps_batch_min": 26296.344019290995, + "qps_batch_runs": [ + 26518.375908585855, + 26296.344019290995, + 26517.18048123379 + ], + "recall": 0.9476, + "threads": 32 + }, + "knob": 96, + "max_ms": 0.4948, + "mean_ms": 0.20961716666666635, + "median_ms": 0.20359999999999998, + "nq": 1000, + "p95_ms": 0.2727, + "p99_ms": 0.3544, + "qps_1t": 4773.299309255862, + "qps_1t_max": 5077.17560782677, + "qps_1t_min": 4496.546427516349, + "qps_1t_runs": [ + 4773.299309255862, + 4496.546427516349, + 5077.17560782677 + ], + "recall": 0.9476 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 19502.605548101226, + "qps_batch_max": 20211.49306341558, + "qps_batch_min": 18346.267451886913, + "qps_batch_runs": [ + 20211.49306341558, + 18346.267451886913, + 19502.605548101226 + ], + "recall": 0.9626, + "threads": 32 + }, + "knob": 128, + "max_ms": 0.8796, + "mean_ms": 0.2715536333333334, + "median_ms": 0.2635, + "nq": 1000, + "p95_ms": 0.3635, + "p99_ms": 0.46959999999999996, + "qps_1t": 3641.5836810438773, + "qps_1t_max": 3780.4396273242596, + "qps_1t_min": 3629.295089273404, + "qps_1t_runs": [ + 3780.4396273242596, + 3629.295089273404, + 3641.5836810438773 + ], + "recall": 0.9626 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 14897.157473382504, + "qps_batch_max": 15079.20351647026, + "qps_batch_min": 14735.526765610817, + "qps_batch_runs": [ + 15079.20351647026, + 14897.157473382504, + 14735.526765610817 + ], + "recall": 0.9796, + "threads": 32 + }, + "knob": 192, + "max_ms": 0.7627, + "mean_ms": 0.3667500333333329, + "median_ms": 0.36160000000000003, + "nq": 1000, + "p95_ms": 0.4613, + "p99_ms": 0.5549, + "qps_1t": 2727.5231634904662, + "qps_1t_max": 2734.230598446517, + "qps_1t_min": 2718.251837878021, + "qps_1t_runs": [ + 2734.230598446517, + 2727.5231634904662, + 2718.251837878021 + ], + "recall": 0.9796 + } + ], + "system": "rust_hnsw_i8x4", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_hnsw_i8x4_text384_100000_r1.load.json b/benchmarks/rust-stage-c/raw/rust_hnsw_i8x4_text384_100000_r1.load.json new file mode 100644 index 0000000..ef178e6 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_hnsw_i8x4_text384_100000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 11.746875, "pin_cpu_pct": 74.7, "max_cpu_pct": 74.7, "noisy": true, "seconds": 9.904262781143188, "after": 23.0} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_hnsw_i8x4_text384_100000_r1.txt b/benchmarks/rust-stage-c/raw/rust_hnsw_i8x4_text384_100000_r1.txt new file mode 100644 index 0000000..7585f21 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_hnsw_i8x4_text384_100000_r1.txt @@ -0,0 +1,8 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/text384 --ds text384 --n 100000 --mode hnsw --exp core --queries 1000 --budget 3 --label rust_hnsw_i8x4 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_hnsw_i8x4_text384_100000_r1.json --int8 4 --knobs 64,96,128,192 +rust_hnsw_i8x4: ds=text384 n=100000 d=384 mode=hnsw int8=4 queries=1000 cpus=32 exps=["core"] + ingest 0.10s rss 169 MB + hnsw build 5.63s (stats Some(5.6327837)) rss 246 MB (peak 247) + hnsw+int8x4 knob=64 recall=0.9145 qps1t=7034 median=0.147ms batch=33876 + hnsw+int8x4 knob=96 recall=0.9476 qps1t=4773 median=0.204ms batch=26517 + hnsw+int8x4 knob=128 recall=0.9626 qps1t=3642 median=0.264ms batch=19503 + hnsw+int8x4 knob=192 recall=0.9796 qps1t=2728 median=0.362ms batch=14897 diff --git a/benchmarks/rust-stage-c/raw/rust_hnsw_text384_1000000_r1.json b/benchmarks/rust-stage-c/raw/rust_hnsw_text384_1000000_r1.json new file mode 100644 index 0000000..b28ba17 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_hnsw_text384_1000000_r1.json @@ -0,0 +1,413 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_query_ms": 1.7906, + "index_s": 171.5653783, + "ingest_s": 1.1454275999999999, + "stats_build_seconds": 171.5633542, + "total_s": 172.7108059 + }, + "concurrent_clients": [ + { + "clients": 1, + "qps": 1335.8344455843774 + }, + { + "clients": 2, + "qps": 3037.459635930768 + }, + { + "clients": 4, + "qps": 3765.0390365365747 + }, + { + "clients": 8, + "qps": 5535.439251710136 + }, + { + "clients": 16, + "qps": 5343.1647564852665 + } + ], + "d": 384, + "ds": "text384", + "exps": [ + "core", + "conc", + "writes" + ], + "first_exact_query_ms": 74.6027, + "index_stats": { + "ef_search_default": 192, + "graph_nodes": 1000000, + "index_bytes": 146214912, + "max_level": 5, + "nlist": null, + "trained_n": 1000000 + }, + "int8_rerank": 0, + "k": 10, + "knob_name": "ef_search", + "mem_after_build": { + "peak_mb": 1801.4375, + "rss_mb": 1793.42578125 + }, + "mem_after_first_exact_query": { + "peak_mb": 1656.4921875, + "rss_mb": 1631.35546875 + }, + "mem_after_ingest": { + "peak_mb": 1656.4921875, + "rss_mb": 1629.15625 + }, + "mem_after_sweep": { + "peak_mb": 1802.859375, + "rss_mb": 1802.84375 + }, + "mem_baseline": { + "peak_mb": 7.875, + "rss_mb": 7.875 + }, + "mem_end": { + "peak_mb": 1852.703125, + "rss_mb": 1822.70703125 + }, + "memory_stats": { + "index_bytes": 146214912, + "matrix_bytes": 1536688128, + "metadata_bytes": 130484346, + "quant_bytes": 0, + "total_bytes": 1813387386 + }, + "mode": "hnsw", + "mutation": { + "after": { + "max_ms": 11.4099, + "mean_ms": 0.6063884666666659, + "median_ms": 0.5894, + "nq": 1000, + "p95_ms": 0.7961, + "p99_ms": 0.9555, + "qps_1t": 1603.1336774620083, + "qps_1t_max": 1772.4352640175264, + "qps_1t_min": 1584.3049875822182, + "qps_1t_runs": [ + 1772.4352640175264, + 1584.3049875822182, + 1603.1336774620083 + ], + "recall": 0.955 + }, + "delete_10k": { + "lat": { + "max_ms": 0.0163, + "mean_ms": 0.0006905799999999857, + "median_ms": 0.0006, + "p95_ms": 0.001, + "p99_ms": 0.0013 + }, + "per_s": 1382170.00691085, + "seconds": 0.007235 + }, + "first_query_after_ms": 0.5709000000000001, + "index_stats_after": { + "churn": 20000, + "graph_dead": 10000, + "graph_nodes": 1010000, + "inserts": 20000, + "pending": 0, + "trained_n": 1010000 + }, + "insert_10k": { + "lat": { + "max_ms": 4.6811, + "mean_ms": 0.7176862199999973, + "median_ms": 0.6757, + "p95_ms": 1.0825, + "p99_ms": 1.4894 + }, + "per_s": 1392.9937512668407, + "seconds": 7.1787831 + }, + "mem_after": { + "peak_mb": 1852.703125, + "rss_mb": 1822.70703125 + }, + "mode": "hnsw", + "qps_after": 1603.1336774620083, + "recall_after": 0.955, + "recall_before": 0.9549, + "ref_knob": 192, + "update_10k": { + "lat": { + "max_ms": 6.6285, + "mean_ms": 0.6943683699999987, + "median_ms": 0.6582, + "p95_ms": 1.028, + "p99_ms": 1.3500999999999999 + }, + "per_s": 1439.8539596444643, + "seconds": 6.9451488 + } + }, + "n": 1000000, + "queries": 1000, + "raw_vectors_mb": 1464.84375, + "ref_knob": 192, + "ref_recall_pre": 0.9549, + "started": "SystemTime { intervals: 134393588574625408 }", + "status": "ok", + "sweep": [ + { + "batch": { + "nq": 1000, + "qps_batch": 17715.388117580573, + "qps_batch_max": 18199.686965384193, + "qps_batch_min": 17346.74583721467, + "qps_batch_runs": [ + 17346.74583721467, + 18199.686965384193, + 17715.388117580573 + ], + "recall": 0.8342, + "threads": 32 + }, + "knob": 48, + "max_ms": 1.9324, + "mean_ms": 0.23011150000000066, + "median_ms": 0.2105, + "nq": 1000, + "p95_ms": 0.3679, + "p99_ms": 0.4858, + "qps_1t": 4598.974060866497, + "qps_1t_max": 4856.016677503681, + "qps_1t_min": 3745.8149882044318, + "qps_1t_runs": [ + 3745.8149882044318, + 4856.016677503681, + 4598.974060866497 + ], + "recall": 0.8342 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 13993.431483261757, + "qps_batch_max": 14706.531170492815, + "qps_batch_min": 13580.0373451027, + "qps_batch_runs": [ + 13580.0373451027, + 13993.431483261757, + 14706.531170492815 + ], + "recall": 0.8678, + "threads": 32 + }, + "knob": 64, + "max_ms": 0.5728000000000001, + "mean_ms": 0.23387986666666716, + "median_ms": 0.2277, + "nq": 1000, + "p95_ms": 0.3091, + "p99_ms": 0.3741, + "qps_1t": 4305.773654951661, + "qps_1t_max": 4377.434674447996, + "qps_1t_min": 4150.255863273969, + "qps_1t_runs": [ + 4150.255863273969, + 4377.434674447996, + 4305.773654951661 + ], + "recall": 0.8678 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 10361.631293784369, + "qps_batch_max": 10490.592561120815, + "qps_batch_min": 9639.241738687868, + "qps_batch_runs": [ + 9639.241738687868, + 10361.631293784369, + 10490.592561120815 + ], + "recall": 0.9062, + "threads": 32 + }, + "knob": 96, + "max_ms": 4.9799, + "mean_ms": 0.3514768000000008, + "median_ms": 0.3327, + "nq": 1000, + "p95_ms": 0.46349999999999997, + "p99_ms": 0.5777, + "qps_1t": 2898.0945897500815, + "qps_1t_max": 2898.616258570483, + "qps_1t_min": 2744.358353922884, + "qps_1t_runs": [ + 2744.358353922884, + 2898.0945897500815, + 2898.616258570483 + ], + "recall": 0.9062 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 8200.102173273079, + "qps_batch_max": 8493.210951825658, + "qps_batch_min": 7987.877596958856, + "qps_batch_runs": [ + 8493.210951825658, + 8200.102173273079, + 7987.877596958856 + ], + "recall": 0.9287, + "threads": 32 + }, + "knob": 128, + "max_ms": 1.1229, + "mean_ms": 0.4332562666666668, + "median_ms": 0.42069999999999996, + "nq": 1000, + "p95_ms": 0.5829, + "p99_ms": 0.7369, + "qps_1t": 2312.3969682625807, + "qps_1t_max": 2463.3690858511213, + "qps_1t_min": 2167.4624866430127, + "qps_1t_runs": [ + 2463.3690858511213, + 2312.3969682625807, + 2167.4624866430127 + ], + "recall": 0.9287 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 5582.615734602448, + "qps_batch_max": 5652.924031484526, + "qps_batch_min": 5526.7855662468155, + "qps_batch_runs": [ + 5526.7855662468155, + 5652.924031484526, + 5582.615734602448 + ], + "recall": 0.9549, + "threads": 32 + }, + "knob": 192, + "max_ms": 1.4041, + "mean_ms": 0.5814919666666661, + "median_ms": 0.5756, + "nq": 1000, + "p95_ms": 0.7435, + "p99_ms": 0.9032, + "qps_1t": 1714.1515207095206, + "qps_1t_max": 1754.2945569327367, + "qps_1t_min": 1691.8550039657077, + "qps_1t_runs": [ + 1691.8550039657077, + 1754.2945569327367, + 1714.1515207095206 + ], + "recall": 0.9549 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 4703.631250361592, + "qps_batch_max": 4729.675402377135, + "qps_batch_min": 4432.490510037818, + "qps_batch_runs": [ + 4432.490510037818, + 4703.631250361592, + 4729.675402377135 + ], + "recall": 0.9685, + "threads": 32 + }, + "knob": 256, + "max_ms": 1.9627999999999999, + "mean_ms": 0.7430227999999995, + "median_ms": 0.741, + "nq": 1000, + "p95_ms": 0.9315, + "p99_ms": 1.0846, + "qps_1t": 1357.2467543819048, + "qps_1t_max": 1369.5011277157032, + "qps_1t_min": 1312.181320364082, + "qps_1t_runs": [ + 1357.2467543819048, + 1312.181320364082, + 1369.5011277157032 + ], + "recall": 0.9685 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 3095.3475378831845, + "qps_batch_max": 3163.1695718270407, + "qps_batch_min": 3037.5212854304077, + "qps_batch_runs": [ + 3163.1695718270407, + 3037.5212854304077, + 3095.3475378831845 + ], + "recall": 0.9825, + "threads": 32 + }, + "knob": 400, + "max_ms": 1.8273, + "mean_ms": 1.0451551666666663, + "median_ms": 1.0505, + "nq": 1000, + "p95_ms": 1.2894, + "p99_ms": 1.4321, + "qps_1t": 960.6595504209374, + "qps_1t_max": 988.9363732182578, + "qps_1t_min": 923.0826603906885, + "qps_1t_runs": [ + 923.0826603906885, + 960.6595504209374, + 988.9363732182578 + ], + "recall": 0.9825 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 2082.6065904085553, + "qps_batch_max": 2151.3708857989945, + "qps_batch_min": 2081.453520622521, + "qps_batch_runs": [ + 2082.6065904085553, + 2151.3708857989945, + 2081.453520622521 + ], + "recall": 0.9901, + "threads": 32 + }, + "knob": 640, + "max_ms": 3.1227, + "mean_ms": 1.5943687000000026, + "median_ms": 1.5978999999999999, + "nq": 1000, + "p95_ms": 2.0113000000000003, + "p99_ms": 2.323, + "qps_1t": 638.0589939136828, + "qps_1t_max": 648.3209136656967, + "qps_1t_min": 597.5832895155386, + "qps_1t_runs": [ + 638.0589939136828, + 648.3209136656967, + 597.5832895155386 + ], + "recall": 0.9901 + } + ], + "system": "rust_hnsw", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_hnsw_text384_1000000_r1.load.json b/benchmarks/rust-stage-c/raw/rust_hnsw_text384_1000000_r1.load.json new file mode 100644 index 0000000..b3d9d79 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_hnsw_text384_1000000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 18.2625, "pin_cpu_pct": 0.0, "max_cpu_pct": 75.2, "noisy": false, "seconds": 227.6128692626953, "after": 61.6} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_hnsw_text384_1000000_r1.txt b/benchmarks/rust-stage-c/raw/rust_hnsw_text384_1000000_r1.txt new file mode 100644 index 0000000..dcecbdd --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_hnsw_text384_1000000_r1.txt @@ -0,0 +1,18 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/text384 --ds text384 --n 1000000 --mode hnsw --exp core,conc,writes --queries 1000 --budget 3 --label rust_hnsw --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_hnsw_text384_1000000_r1.json --knobs 48,64,96,128,192,256,400,640 +rust_hnsw: ds=text384 n=1000000 d=384 mode=hnsw int8=0 queries=1000 cpus=32 exps=["core", "conc", "writes"] + ingest 1.15s rss 1629 MB + hnsw build 171.57s (stats Some(171.5633542)) rss 1793 MB (peak 1801) + hnsw knob=48 recall=0.8342 qps1t=4599 median=0.210ms batch=17715 + hnsw knob=64 recall=0.8678 qps1t=4306 median=0.228ms batch=13993 + hnsw knob=96 recall=0.9062 qps1t=2898 median=0.333ms batch=10362 + hnsw knob=128 recall=0.9287 qps1t=2312 median=0.421ms batch=8200 + hnsw knob=192 recall=0.9549 qps1t=1714 median=0.576ms batch=5583 + hnsw knob=256 recall=0.9685 qps1t=1357 median=0.741ms batch=4704 + hnsw knob=400 recall=0.9825 qps1t=961 median=1.050ms batch=3095 + hnsw knob=640 recall=0.9901 qps1t=638 median=1.598ms batch=2083 + concurrent clients=1 (1 scan thread each): 1335.8 qps + concurrent clients=2 (1 scan thread each): 3037.5 qps + concurrent clients=4 (1 scan thread each): 3765.0 qps + concurrent clients=8 (1 scan thread each): 5535.4 qps + concurrent clients=16 (1 scan thread each): 5343.2 qps + writes: ins 1393/s upd 1440/s del 1382170/s first-query 0.6 ms recall_after=0.955 diff --git a/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r1.json b/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r1.json new file mode 100644 index 0000000..bb59f20 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r1.json @@ -0,0 +1,413 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_query_ms": 0.5730999999999999, + "index_s": 7.2630428, + "ingest_s": 0.099744, + "stats_build_seconds": 7.2629425, + "total_s": 7.3627868 + }, + "concurrent_clients": [ + { + "clients": 1, + "qps": 1601.2402479002553 + }, + { + "clients": 2, + "qps": 2994.4593504642444 + }, + { + "clients": 4, + "qps": 5727.596153420001 + }, + { + "clients": 8, + "qps": 10038.675190776814 + }, + { + "clients": 16, + "qps": 14915.951868307522 + } + ], + "d": 384, + "ds": "text384", + "exps": [ + "core", + "conc", + "writes" + ], + "first_exact_query_ms": 6.0464, + "index_stats": { + "ef_search_default": 96, + "graph_nodes": 100000, + "index_bytes": 14458880, + "max_level": 4, + "nlist": null, + "trained_n": 100000 + }, + "int8_rerank": 0, + "k": 10, + "knob_name": "ef_search", + "mem_after_build": { + "peak_mb": 208.6640625, + "rss_mb": 205.54296875 + }, + "mem_after_first_exact_query": { + "peak_mb": 196.28515625, + "rss_mb": 170.7109375 + }, + "mem_after_ingest": { + "peak_mb": 196.28515625, + "rss_mb": 168.9765625 + }, + "mem_after_sweep": { + "peak_mb": 211.44921875, + "rss_mb": 211.44921875 + }, + "mem_baseline": { + "peak_mb": 7.859375, + "rss_mb": 7.859375 + }, + "mem_end": { + "peak_mb": 254.48046875, + "rss_mb": 223.76171875 + }, + "memory_stats": { + "index_bytes": 14458880, + "matrix_bytes": 154140672, + "metadata_bytes": 12696122, + "quant_bytes": 0, + "total_bytes": 181295674 + }, + "mode": "hnsw", + "mutation": { + "after": { + "max_ms": 2.144, + "mean_ms": 0.31776926666666644, + "median_ms": 0.31310000000000004, + "nq": 1000, + "p95_ms": 0.39149999999999996, + "p99_ms": 0.4608, + "qps_1t": 3164.025285624469, + "qps_1t_max": 3187.227631478723, + "qps_1t_min": 3091.1671752667066, + "qps_1t_runs": [ + 3164.025285624469, + 3187.227631478723, + 3091.1671752667066 + ], + "recall": 0.9582 + }, + "delete_10k": { + "lat": { + "max_ms": 0.0135, + "mean_ms": 0.0008667400000000518, + "median_ms": 0.0009, + "p95_ms": 0.0011, + "p99_ms": 0.0013 + }, + "per_s": 1069038.5067670138, + "seconds": 0.0093542 + }, + "first_query_after_ms": 0.7369, + "index_stats_after": { + "churn": 20000, + "graph_dead": 10000, + "graph_nodes": 110000, + "inserts": 20000, + "pending": 0, + "trained_n": 110000 + }, + "insert_10k": { + "lat": { + "max_ms": 2.7957, + "mean_ms": 1.0593985199999978, + "median_ms": 1.0259, + "p95_ms": 1.5459, + "p99_ms": 1.8936 + }, + "per_s": 943.7420544586919, + "seconds": 10.5961157 + }, + "mem_after": { + "peak_mb": 254.48046875, + "rss_mb": 225.05078125 + }, + "mode": "hnsw", + "qps_after": 3164.025285624469, + "recall_after": 0.9582, + "recall_before": 0.9626, + "ref_knob": 128, + "update_10k": { + "lat": { + "max_ms": 3.3960000000000004, + "mean_ms": 1.067913180000001, + "median_ms": 1.0307000000000002, + "p95_ms": 1.5689, + "p99_ms": 1.9529 + }, + "per_s": 936.2149981043519, + "seconds": 10.6813072 + } + }, + "n": 100000, + "queries": 1000, + "raw_vectors_mb": 146.484375, + "ref_knob": 128, + "ref_recall_pre": 0.9626, + "started": "SystemTime { intervals: 134393517774015254 }", + "status": "ok", + "sweep": [ + { + "batch": { + "nq": 1000, + "qps_batch": 45645.6346797274, + "qps_batch_max": 47261.65945138666, + "qps_batch_min": 35973.42283520935, + "qps_batch_runs": [ + 47261.65945138666, + 45645.6346797274, + 35973.42283520935 + ], + "recall": 0.7816, + "threads": 32 + }, + "knob": 24, + "max_ms": 2.8998, + "mean_ms": 0.10433073333333323, + "median_ms": 0.0886, + "nq": 1000, + "p95_ms": 0.1719, + "p99_ms": 0.3026, + "qps_1t": 8822.737960271223, + "qps_1t_max": 13058.393216948238, + "qps_1t_min": 8125.483466266242, + "qps_1t_runs": [ + 8125.483466266242, + 8822.737960271223, + 13058.393216948238 + ], + "recall": 0.7816 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 28371.772356247322, + "qps_batch_max": 29213.712916843167, + "qps_batch_min": 28168.537996540905, + "qps_batch_runs": [ + 28168.537996540905, + 28371.772356247322, + 29213.712916843167 + ], + "recall": 0.885, + "threads": 32 + }, + "knob": 48, + "max_ms": 0.4164, + "mean_ms": 0.13277436666666664, + "median_ms": 0.1264, + "nq": 1000, + "p95_ms": 0.183, + "p99_ms": 0.2096, + "qps_1t": 7752.460824877345, + "qps_1t_max": 7806.882703929444, + "qps_1t_min": 7080.162305640702, + "qps_1t_runs": [ + 7806.882703929444, + 7752.460824877345, + 7080.162305640702 + ], + "recall": 0.885 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 22433.99917442883, + "qps_batch_max": 22910.609674234038, + "qps_batch_min": 20353.582434044216, + "qps_batch_runs": [ + 22910.609674234038, + 22433.99917442883, + 20353.582434044216 + ], + "recall": 0.9145, + "threads": 32 + }, + "knob": 64, + "max_ms": 1.06, + "mean_ms": 0.2227675666666672, + "median_ms": 0.2119, + "nq": 1000, + "p95_ms": 0.31520000000000004, + "p99_ms": 0.453, + "qps_1t": 4480.240570997695, + "qps_1t_max": 4807.821749047086, + "qps_1t_min": 4217.522964412534, + "qps_1t_runs": [ + 4807.821749047086, + 4217.522964412534, + 4480.240570997695 + ], + "recall": 0.9145 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 16667.8611967191, + "qps_batch_max": 17055.88189142908, + "qps_batch_min": 16615.96029450128, + "qps_batch_runs": [ + 16667.8611967191, + 17055.88189142908, + 16615.96029450128 + ], + "recall": 0.9476, + "threads": 32 + }, + "knob": 96, + "max_ms": 1.1111, + "mean_ms": 0.24545139999999988, + "median_ms": 0.23149999999999998, + "nq": 1000, + "p95_ms": 0.34890000000000004, + "p99_ms": 0.4837, + "qps_1t": 3966.6594341242944, + "qps_1t_max": 4350.245201570783, + "qps_1t_min": 3931.1142979344377, + "qps_1t_runs": [ + 3931.1142979344377, + 3966.6594341242944, + 4350.245201570783 + ], + "recall": 0.9476 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 13918.794966406987, + "qps_batch_max": 14280.32856179955, + "qps_batch_min": 13740.944717431214, + "qps_batch_runs": [ + 14280.32856179955, + 13918.794966406987, + 13740.944717431214 + ], + "recall": 0.9626, + "threads": 32 + }, + "knob": 128, + "max_ms": 0.9382, + "mean_ms": 0.2861439000000001, + "median_ms": 0.2752, + "nq": 1000, + "p95_ms": 0.3839, + "p99_ms": 0.4767, + "qps_1t": 3525.671649263362, + "qps_1t_max": 3529.848220056381, + "qps_1t_min": 3430.5376167026006, + "qps_1t_runs": [ + 3525.671649263362, + 3430.5376167026006, + 3529.848220056381 + ], + "recall": 0.9626 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 10022.530648898724, + "qps_batch_max": 10691.48230987337, + "qps_batch_min": 9921.21562670831, + "qps_batch_runs": [ + 10691.48230987337, + 10022.530648898724, + 9921.21562670831 + ], + "recall": 0.9796, + "threads": 32 + }, + "knob": 192, + "max_ms": 2.3179, + "mean_ms": 0.3994103666666667, + "median_ms": 0.3855, + "nq": 1000, + "p95_ms": 0.5265, + "p99_ms": 0.67, + "qps_1t": 2502.1224253473006, + "qps_1t_max": 2537.2248950476896, + "qps_1t_min": 2472.5607569992003, + "qps_1t_runs": [ + 2472.5607569992003, + 2537.2248950476896, + 2502.1224253473006 + ], + "recall": 0.9796 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 8116.395606108075, + "qps_batch_max": 8368.908167133792, + "qps_batch_min": 8026.216834668762, + "qps_batch_runs": [ + 8116.395606108075, + 8368.908167133792, + 8026.216834668762 + ], + "recall": 0.9863, + "threads": 32 + }, + "knob": 256, + "max_ms": 4.5257, + "mean_ms": 0.5098376666666664, + "median_ms": 0.49229999999999996, + "nq": 1000, + "p95_ms": 0.6648000000000001, + "p99_ms": 0.8375, + "qps_1t": 1928.936066378547, + "qps_1t_max": 2030.4486073153003, + "qps_1t_min": 1928.3037386917024, + "qps_1t_runs": [ + 1928.3037386917024, + 2030.4486073153003, + 1928.936066378547 + ], + "recall": 0.9863 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 5509.496167594466, + "qps_batch_max": 5642.547971532217, + "qps_batch_min": 5457.24927350369, + "qps_batch_runs": [ + 5457.24927350369, + 5509.496167594466, + 5642.547971532217 + ], + "recall": 0.9936, + "threads": 32 + }, + "knob": 400, + "max_ms": 1.3542999999999998, + "mean_ms": 0.7178902666666662, + "median_ms": 0.7079, + "nq": 1000, + "p95_ms": 0.9183, + "p99_ms": 1.0484, + "qps_1t": 1398.5390022459153, + "qps_1t_max": 1405.069236894251, + "qps_1t_min": 1375.6478957677077, + "qps_1t_runs": [ + 1398.5390022459153, + 1375.6478957677077, + 1405.069236894251 + ], + "recall": 0.9936 + } + ], + "system": "rust_hnsw", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r1.load.json b/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r1.load.json new file mode 100644 index 0000000..4b6181b --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 15.6, "pin_cpu_pct": 0.0, "max_cpu_pct": 58.7, "noisy": false, "seconds": 55.53210425376892, "after": 23.7} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r1.txt b/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r1.txt new file mode 100644 index 0000000..eac5b74 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r1.txt @@ -0,0 +1,18 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/text384 --ds text384 --n 100000 --mode hnsw --exp core,conc,writes --queries 1000 --budget 3 --label rust_hnsw --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_hnsw_text384_100000_r1.json +rust_hnsw: ds=text384 n=100000 d=384 mode=hnsw int8=0 queries=1000 cpus=32 exps=["core", "conc", "writes"] + ingest 0.10s rss 169 MB + hnsw build 7.26s (stats Some(7.2629425)) rss 206 MB (peak 209) + hnsw knob=24 recall=0.7816 qps1t=8823 median=0.089ms batch=45646 + hnsw knob=48 recall=0.8850 qps1t=7752 median=0.126ms batch=28372 + hnsw knob=64 recall=0.9145 qps1t=4480 median=0.212ms batch=22434 + hnsw knob=96 recall=0.9476 qps1t=3967 median=0.231ms batch=16668 + hnsw knob=128 recall=0.9626 qps1t=3526 median=0.275ms batch=13919 + hnsw knob=192 recall=0.9796 qps1t=2502 median=0.386ms batch=10023 + hnsw knob=256 recall=0.9863 qps1t=1929 median=0.492ms batch=8116 + hnsw knob=400 recall=0.9936 qps1t=1399 median=0.708ms batch=5509 + concurrent clients=1 (1 scan thread each): 1601.2 qps + concurrent clients=2 (1 scan thread each): 2994.5 qps + concurrent clients=4 (1 scan thread each): 5727.6 qps + concurrent clients=8 (1 scan thread each): 10038.7 qps + concurrent clients=16 (1 scan thread each): 14916.0 qps + writes: ins 944/s upd 936/s del 1069039/s first-query 0.7 ms recall_after=0.9582 diff --git a/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r2.json b/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r2.json new file mode 100644 index 0000000..b0fd72f --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r2.json @@ -0,0 +1,319 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_query_ms": 0.5487, + "index_s": 6.8243642, + "ingest_s": 0.1068286, + "stats_build_seconds": 6.824207, + "total_s": 6.9311928 + }, + "d": 384, + "ds": "text384", + "exps": [ + "core" + ], + "first_exact_query_ms": 6.3532, + "index_stats": { + "ef_search_default": 96, + "graph_nodes": 100000, + "index_bytes": 14458880, + "max_level": 4, + "nlist": null, + "trained_n": 100000 + }, + "int8_rerank": 0, + "k": 10, + "knob_name": "ef_search", + "mem_after_build": { + "peak_mb": 209.01171875, + "rss_mb": 207.78125 + }, + "mem_after_first_exact_query": { + "peak_mb": 196.51171875, + "rss_mb": 171.6015625 + }, + "mem_after_ingest": { + "peak_mb": 196.51171875, + "rss_mb": 169.109375 + }, + "mem_after_sweep": { + "peak_mb": 211.98046875, + "rss_mb": 208.59375 + }, + "mem_baseline": { + "peak_mb": 7.85546875, + "rss_mb": 7.85546875 + }, + "mem_end": { + "peak_mb": 211.98046875, + "rss_mb": 208.6171875 + }, + "memory_stats": { + "index_bytes": 14458880, + "matrix_bytes": 154140672, + "metadata_bytes": 12696122, + "quant_bytes": 0, + "total_bytes": 181295674 + }, + "mode": "hnsw", + "n": 100000, + "queries": 1000, + "raw_vectors_mb": 146.484375, + "ref_knob": 128, + "ref_recall_pre": 0.9626, + "started": "SystemTime { intervals: 134393531044277676 }", + "status": "ok", + "sweep": [ + { + "batch": { + "nq": 1000, + "qps_batch": 45956.72714571959, + "qps_batch_max": 49179.924754715124, + "qps_batch_min": 42339.332816793474, + "qps_batch_runs": [ + 49179.924754715124, + 45956.72714571959, + 42339.332816793474 + ], + "recall": 0.7816, + "threads": 32 + }, + "knob": 24, + "max_ms": 0.24420000000000003, + "mean_ms": 0.09157146666666656, + "median_ms": 0.0896, + "nq": 1000, + "p95_ms": 0.1206, + "p99_ms": 0.1409, + "qps_1t": 10823.75250840464, + "qps_1t_max": 11119.945289869185, + "qps_1t_min": 10822.920781631348, + "qps_1t_runs": [ + 11119.945289869185, + 10823.75250840464, + 10822.920781631348 + ], + "recall": 0.7816 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 27571.45141634546, + "qps_batch_max": 28616.233416892734, + "qps_batch_min": 27004.40171747995, + "qps_batch_runs": [ + 27004.40171747995, + 27571.45141634546, + 28616.233416892734 + ], + "recall": 0.885, + "threads": 32 + }, + "knob": 48, + "max_ms": 1.4046, + "mean_ms": 0.16386806666666667, + "median_ms": 0.1631, + "nq": 1000, + "p95_ms": 0.202, + "p99_ms": 0.2319, + "qps_1t": 6040.4383183661275, + "qps_1t_max": 6277.396914157222, + "qps_1t_min": 5996.94395735933, + "qps_1t_runs": [ + 6277.396914157222, + 6040.4383183661275, + 5996.94395735933 + ], + "recall": 0.885 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 23646.252069047056, + "qps_batch_max": 23648.0973923243, + "qps_batch_min": 22909.192542599645, + "qps_batch_runs": [ + 22909.192542599645, + 23646.252069047056, + 23648.0973923243 + ], + "recall": 0.9145, + "threads": 32 + }, + "knob": 64, + "max_ms": 0.447, + "mean_ms": 0.20785226666666656, + "median_ms": 0.2068, + "nq": 1000, + "p95_ms": 0.2528, + "p99_ms": 0.2802, + "qps_1t": 4798.706268789932, + "qps_1t_max": 4895.958435271266, + "qps_1t_min": 4741.1970194939095, + "qps_1t_runs": [ + 4895.958435271266, + 4741.1970194939095, + 4798.706268789932 + ], + "recall": 0.9145 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 18789.822680443365, + "qps_batch_max": 18835.052031831237, + "qps_batch_min": 17690.629804111657, + "qps_batch_runs": [ + 17690.629804111657, + 18835.052031831237, + 18789.822680443365 + ], + "recall": 0.9476, + "threads": 32 + }, + "knob": 96, + "max_ms": 2.4535, + "mean_ms": 0.28702160000000015, + "median_ms": 0.28659999999999997, + "nq": 1000, + "p95_ms": 0.3436, + "p99_ms": 0.3819, + "qps_1t": 3435.5123413909846, + "qps_1t_max": 3586.103418919282, + "qps_1t_min": 3434.8538744464768, + "qps_1t_runs": [ + 3586.103418919282, + 3434.8538744464768, + 3435.5123413909846 + ], + "recall": 0.9476 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 14607.731287861414, + "qps_batch_max": 14784.173246855775, + "qps_batch_min": 14196.51958125946, + "qps_batch_runs": [ + 14784.173246855775, + 14196.51958125946, + 14607.731287861414 + ], + "recall": 0.9626, + "threads": 32 + }, + "knob": 128, + "max_ms": 1.716, + "mean_ms": 0.3650255333333338, + "median_ms": 0.3659, + "nq": 1000, + "p95_ms": 0.4348, + "p99_ms": 0.47150000000000003, + "qps_1t": 2742.3759890721776, + "qps_1t_max": 2759.33925965824, + "qps_1t_min": 2717.216309710491, + "qps_1t_runs": [ + 2759.33925965824, + 2742.3759890721776, + 2717.216309710491 + ], + "recall": 0.9626 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 10110.569184602817, + "qps_batch_max": 10153.356293456365, + "qps_batch_min": 10060.665814863627, + "qps_batch_runs": [ + 10110.569184602817, + 10060.665814863627, + 10153.356293456365 + ], + "recall": 0.9796, + "threads": 32 + }, + "knob": 192, + "max_ms": 1.5215999999999998, + "mean_ms": 0.5272349000000017, + "median_ms": 0.5282, + "nq": 1000, + "p95_ms": 0.6317999999999999, + "p99_ms": 0.6843, + "qps_1t": 1889.948309913724, + "qps_1t_max": 1941.5522865855412, + "qps_1t_min": 1860.333941104433, + "qps_1t_runs": [ + 1941.5522865855412, + 1889.948309913724, + 1860.333941104433 + ], + "recall": 0.9796 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 8042.509488150568, + "qps_batch_max": 8221.236769974728, + "qps_batch_min": 7960.040596207041, + "qps_batch_runs": [ + 7960.040596207041, + 8042.509488150568, + 8221.236769974728 + ], + "recall": 0.9863, + "threads": 32 + }, + "knob": 256, + "max_ms": 2.6725, + "mean_ms": 0.6679123333333326, + "median_ms": 0.6721, + "nq": 1000, + "p95_ms": 0.8062, + "p99_ms": 0.8901, + "qps_1t": 1473.8966998863295, + "qps_1t_max": 1550.490482159127, + "qps_1t_min": 1469.9263361115911, + "qps_1t_runs": [ + 1550.490482159127, + 1473.8966998863295, + 1469.9263361115911 + ], + "recall": 0.9863 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 5653.547233408393, + "qps_batch_max": 5838.294424020145, + "qps_batch_min": 5546.939309827623, + "qps_batch_runs": [ + 5838.294424020145, + 5546.939309827623, + 5653.547233408393 + ], + "recall": 0.9936, + "threads": 32 + }, + "knob": 400, + "max_ms": 1.5977000000000001, + "mean_ms": 0.9573524666666667, + "median_ms": 0.9616, + "nq": 1000, + "p95_ms": 1.1589, + "p99_ms": 1.2366, + "qps_1t": 1050.63217063023, + "qps_1t_max": 1051.3415275592874, + "qps_1t_min": 1031.9025042003589, + "qps_1t_runs": [ + 1051.3415275592874, + 1031.9025042003589, + 1050.63217063023 + ], + "recall": 0.9936 + } + ], + "system": "rust_hnsw", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r2.load.json b/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r2.load.json new file mode 100644 index 0000000..9766ac2 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r2.load.json @@ -0,0 +1 @@ +{"total_pct": 21.265625, "pin_cpu_pct": 0.0, "max_cpu_pct": 71.2, "noisy": false, "seconds": 19.68856143951416, "after": 25.1} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r2.txt b/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r2.txt new file mode 100644 index 0000000..7a28392 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r2.txt @@ -0,0 +1,12 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/text384 --ds text384 --n 100000 --mode hnsw --exp core --queries 1000 --budget 3 --label rust_hnsw --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_hnsw_text384_100000_r2.json +rust_hnsw: ds=text384 n=100000 d=384 mode=hnsw int8=0 queries=1000 cpus=32 exps=["core"] + ingest 0.11s rss 169 MB + hnsw build 6.82s (stats Some(6.824207)) rss 208 MB (peak 209) + hnsw knob=24 recall=0.7816 qps1t=10824 median=0.090ms batch=45957 + hnsw knob=48 recall=0.8850 qps1t=6040 median=0.163ms batch=27571 + hnsw knob=64 recall=0.9145 qps1t=4799 median=0.207ms batch=23646 + hnsw knob=96 recall=0.9476 qps1t=3436 median=0.287ms batch=18790 + hnsw knob=128 recall=0.9626 qps1t=2742 median=0.366ms batch=14608 + hnsw knob=192 recall=0.9796 qps1t=1890 median=0.528ms batch=10111 + hnsw knob=256 recall=0.9863 qps1t=1474 median=0.672ms batch=8043 + hnsw knob=400 recall=0.9936 qps1t=1051 median=0.962ms batch=5654 diff --git a/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r3.json b/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r3.json new file mode 100644 index 0000000..5bfcd6d --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r3.json @@ -0,0 +1,319 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_query_ms": 0.6376, + "index_s": 7.0329798, + "ingest_s": 0.09502379999999999, + "stats_build_seconds": 7.0328157000000004, + "total_s": 7.1280035999999996 + }, + "d": 384, + "ds": "text384", + "exps": [ + "core" + ], + "first_exact_query_ms": 6.0155, + "index_stats": { + "ef_search_default": 96, + "graph_nodes": 100000, + "index_bytes": 14458880, + "max_level": 4, + "nlist": null, + "trained_n": 100000 + }, + "int8_rerank": 0, + "k": 10, + "knob_name": "ef_search", + "mem_after_build": { + "peak_mb": 208.484375, + "rss_mb": 206.72265625 + }, + "mem_after_first_exact_query": { + "peak_mb": 196.50390625, + "rss_mb": 170.8515625 + }, + "mem_after_ingest": { + "peak_mb": 196.50390625, + "rss_mb": 169.11328125 + }, + "mem_after_sweep": { + "peak_mb": 212.6171875, + "rss_mb": 212.6171875 + }, + "mem_baseline": { + "peak_mb": 7.86328125, + "rss_mb": 7.86328125 + }, + "mem_end": { + "peak_mb": 212.640625, + "rss_mb": 212.640625 + }, + "memory_stats": { + "index_bytes": 14458880, + "matrix_bytes": 154140672, + "metadata_bytes": 12696122, + "quant_bytes": 0, + "total_bytes": 181295674 + }, + "mode": "hnsw", + "n": 100000, + "queries": 1000, + "raw_vectors_mb": 146.484375, + "ref_knob": 128, + "ref_recall_pre": 0.9626, + "started": "SystemTime { intervals: 134393537026653284 }", + "status": "ok", + "sweep": [ + { + "batch": { + "nq": 1000, + "qps_batch": 42512.3073129671, + "qps_batch_max": 45602.546446193555, + "qps_batch_min": 31859.512294585795, + "qps_batch_runs": [ + 45602.546446193555, + 42512.3073129671, + 31859.512294585795 + ], + "recall": 0.7816, + "threads": 32 + }, + "knob": 24, + "max_ms": 0.2301, + "mean_ms": 0.07978470000000003, + "median_ms": 0.0769, + "nq": 1000, + "p95_ms": 0.1085, + "p99_ms": 0.13290000000000002, + "qps_1t": 12519.734231081757, + "qps_1t_max": 12737.393165114829, + "qps_1t_min": 12350.070148398449, + "qps_1t_runs": [ + 12350.070148398449, + 12519.734231081757, + 12737.393165114829 + ], + "recall": 0.7816 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 24772.03534474003, + "qps_batch_max": 28554.459064327486, + "qps_batch_min": 20877.08823075028, + "qps_batch_runs": [ + 28554.459064327486, + 20877.08823075028, + 24772.03534474003 + ], + "recall": 0.885, + "threads": 32 + }, + "knob": 48, + "max_ms": 0.316, + "mean_ms": 0.12189133333333332, + "median_ms": 0.1168, + "nq": 1000, + "p95_ms": 0.17129999999999998, + "p99_ms": 0.1949, + "qps_1t": 8199.113839776195, + "qps_1t_max": 8352.20932641102, + "qps_1t_min": 8065.765021680772, + "qps_1t_runs": [ + 8352.20932641102, + 8065.765021680772, + 8199.113839776195 + ], + "recall": 0.885 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 22975.35433740227, + "qps_batch_max": 23287.658472515905, + "qps_batch_min": 22870.263854234086, + "qps_batch_runs": [ + 23287.658472515905, + 22870.263854234086, + 22975.35433740227 + ], + "recall": 0.9145, + "threads": 32 + }, + "knob": 64, + "max_ms": 0.6956, + "mean_ms": 0.1560222000000002, + "median_ms": 0.148, + "nq": 1000, + "p95_ms": 0.21939999999999998, + "p99_ms": 0.2434, + "qps_1t": 6388.376221697104, + "qps_1t_max": 6548.381076488362, + "qps_1t_min": 6296.32521275283, + "qps_1t_runs": [ + 6296.32521275283, + 6388.376221697104, + 6548.381076488362 + ], + "recall": 0.9145 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 17621.51798804556, + "qps_batch_max": 17979.176517757132, + "qps_batch_min": 17426.97661125469, + "qps_batch_runs": [ + 17426.97661125469, + 17979.176517757132, + 17621.51798804556 + ], + "recall": 0.9476, + "threads": 32 + }, + "knob": 96, + "max_ms": 0.6939, + "mean_ms": 0.21256300000000034, + "median_ms": 0.2082, + "nq": 1000, + "p95_ms": 0.27559999999999996, + "p99_ms": 0.3289, + "qps_1t": 4667.494680222941, + "qps_1t_max": 4903.479901616579, + "qps_1t_min": 4555.715258684898, + "qps_1t_runs": [ + 4667.494680222941, + 4903.479901616579, + 4555.715258684898 + ], + "recall": 0.9476 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 13989.28141258168, + "qps_batch_max": 14137.172989517285, + "qps_batch_min": 13748.009975556037, + "qps_batch_runs": [ + 14137.172989517285, + 13748.009975556037, + 13989.28141258168 + ], + "recall": 0.9626, + "threads": 32 + }, + "knob": 128, + "max_ms": 0.7575000000000001, + "mean_ms": 0.2744159999999997, + "median_ms": 0.26449999999999996, + "nq": 1000, + "p95_ms": 0.3767, + "p99_ms": 0.41829999999999995, + "qps_1t": 3643.7239587603303, + "qps_1t_max": 3663.0975885828593, + "qps_1t_min": 3625.6777751390905, + "qps_1t_runs": [ + 3643.7239587603303, + 3625.6777751390905, + 3663.0975885828593 + ], + "recall": 0.9626 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 10334.272374216403, + "qps_batch_max": 10538.362802108515, + "qps_batch_min": 10192.93181336334, + "qps_batch_runs": [ + 10538.362802108515, + 10334.272374216403, + 10192.93181336334 + ], + "recall": 0.9796, + "threads": 32 + }, + "knob": 192, + "max_ms": 0.7049, + "mean_ms": 0.390504833333334, + "median_ms": 0.3772, + "nq": 1000, + "p95_ms": 0.5328, + "p99_ms": 0.5902000000000001, + "qps_1t": 2543.586905207637, + "qps_1t_max": 2599.191651396418, + "qps_1t_min": 2540.431604086946, + "qps_1t_runs": [ + 2540.431604086946, + 2543.586905207637, + 2599.191651396418 + ], + "recall": 0.9796 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 8246.062505153788, + "qps_batch_max": 8277.433068676206, + "qps_batch_min": 7830.718656373971, + "qps_batch_runs": [ + 7830.718656373971, + 8246.062505153788, + 8277.433068676206 + ], + "recall": 0.9863, + "threads": 32 + }, + "knob": 256, + "max_ms": 0.9577, + "mean_ms": 0.6207396999999995, + "median_ms": 0.6399, + "nq": 1000, + "p95_ms": 0.7784, + "p99_ms": 0.8297, + "qps_1t": 1521.3690739441663, + "qps_1t_max": 1863.2768443692416, + "qps_1t_min": 1496.4963279721348, + "qps_1t_runs": [ + 1863.2768443692416, + 1521.3690739441663, + 1496.4963279721348 + ], + "recall": 0.9863 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 5513.235348439231, + "qps_batch_max": 5539.2087351106065, + "qps_batch_min": 5506.262272082039, + "qps_batch_runs": [ + 5513.235348439231, + 5539.2087351106065, + 5506.262272082039 + ], + "recall": 0.9936, + "threads": 32 + }, + "knob": 400, + "max_ms": 2.381, + "mean_ms": 0.9893443666666678, + "median_ms": 0.9890999999999999, + "nq": 1000, + "p95_ms": 1.1897, + "p99_ms": 1.2922, + "qps_1t": 1014.4086606153811, + "qps_1t_max": 1025.0067829823884, + "qps_1t_min": 993.4099172906774, + "qps_1t_runs": [ + 1025.0067829823884, + 1014.4086606153811, + 993.4099172906774 + ], + "recall": 0.9936 + } + ], + "system": "rust_hnsw", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r3.load.json b/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r3.load.json new file mode 100644 index 0000000..c55dd7a --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r3.load.json @@ -0,0 +1 @@ +{"total_pct": 9.421875, "pin_cpu_pct": 0.0, "max_cpu_pct": 57.5, "noisy": false, "seconds": 18.611032485961914, "after": 28.8} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r3.txt b/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r3.txt new file mode 100644 index 0000000..126dfa3 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_hnsw_text384_100000_r3.txt @@ -0,0 +1,12 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/text384 --ds text384 --n 100000 --mode hnsw --exp core --queries 1000 --budget 3 --label rust_hnsw --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_hnsw_text384_100000_r3.json +rust_hnsw: ds=text384 n=100000 d=384 mode=hnsw int8=0 queries=1000 cpus=32 exps=["core"] + ingest 0.10s rss 169 MB + hnsw build 7.03s (stats Some(7.0328157000000004)) rss 207 MB (peak 208) + hnsw knob=24 recall=0.7816 qps1t=12520 median=0.077ms batch=42512 + hnsw knob=48 recall=0.8850 qps1t=8199 median=0.117ms batch=24772 + hnsw knob=64 recall=0.9145 qps1t=6388 median=0.148ms batch=22975 + hnsw knob=96 recall=0.9476 qps1t=4667 median=0.208ms batch=17622 + hnsw knob=128 recall=0.9626 qps1t=3644 median=0.264ms batch=13989 + hnsw knob=192 recall=0.9796 qps1t=2544 median=0.377ms batch=10334 + hnsw knob=256 recall=0.9863 qps1t=1521 median=0.640ms batch=8246 + hnsw knob=400 recall=0.9936 qps1t=1014 median=0.989ms batch=5513 diff --git a/benchmarks/rust-stage-c/raw/rust_ivf_i8x4_text384_1000000_r1.json b/benchmarks/rust-stage-c/raw/rust_ivf_i8x4_text384_1000000_r1.json new file mode 100644 index 0000000..b28b136 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_ivf_i8x4_text384_1000000_r1.json @@ -0,0 +1,198 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_query_ms": 5.3808, + "index_s": 6.9130454, + "ingest_s": 1.1467595, + "stats_build_seconds": 6.911995, + "total_s": 8.0598049 + }, + "d": 384, + "ds": "text384", + "exps": [ + "core" + ], + "first_exact_query_ms": 27.6162, + "index_stats": { + "ef_search_default": null, + "graph_nodes": 0, + "index_bytes": 18240000, + "max_level": null, + "nlist": 4000, + "trained_n": 1000000 + }, + "int8_rerank": 4, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 2262.671875, + "rss_mb": 2019.17578125 + }, + "mem_after_first_exact_query": { + "peak_mb": 2000.5234375, + "rss_mb": 2000.5234375 + }, + "mem_after_ingest": { + "peak_mb": 1656.6015625, + "rss_mb": 1629.1953125 + }, + "mem_after_sweep": { + "peak_mb": 2262.671875, + "rss_mb": 2066.3046875 + }, + "mem_baseline": { + "peak_mb": 7.85546875, + "rss_mb": 7.85546875 + }, + "mem_end": { + "peak_mb": 2262.671875, + "rss_mb": 2066.328125 + }, + "memory_stats": { + "index_bytes": 18240000, + "matrix_bytes": 1536688128, + "metadata_bytes": 130484346, + "quant_bytes": 388366336, + "total_bytes": 1685412474 + }, + "mode": "ivf", + "n": 1000000, + "quant_build_s": 1.1329802, + "quant_bytes_mb": 370.375, + "queries": 1000, + "raw_vectors_mb": 1464.84375, + "ref_knob": 256, + "ref_recall_pre": 0.9531951640759931, + "rss_delta_int8_mb": 369.08203125, + "started": "SystemTime { intervals: 134393594631932265 }", + "status": "ok", + "sweep": [ + { + "batch": { + "nq": 1000, + "qps_batch": 16586.44356794305, + "qps_batch_max": 17375.439815820337, + "qps_batch_min": 16283.64812339097, + "qps_batch_runs": [ + 16283.64812339097, + 17375.439815820337, + 16586.44356794305 + ], + "recall": 0.8579, + "threads": 32 + }, + "knob": 64, + "max_ms": 2.702, + "mean_ms": 1.3086219000000017, + "median_ms": 1.2937999999999998, + "nq": 1000, + "p95_ms": 1.6012000000000002, + "p99_ms": 1.812, + "qps_1t": 770.6866556068009, + "qps_1t_max": 782.1936432999884, + "qps_1t_min": 740.8144380585419, + "qps_1t_runs": [ + 740.8144380585419, + 770.6866556068009, + 782.1936432999884 + ], + "recall": 0.8579 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 12031.797634789222, + "qps_batch_max": 12256.208688916588, + "qps_batch_min": 10045.455686983601, + "qps_batch_runs": [ + 12256.208688916588, + 12031.797634789222, + 10045.455686983601 + ], + "recall": 0.9151, + "threads": 32 + }, + "knob": 128, + "max_ms": 5.2862, + "mean_ms": 2.3656116999999925, + "median_ms": 2.348, + "nq": 1000, + "p95_ms": 2.8092, + "p99_ms": 3.0956, + "qps_1t": 423.5639466773652, + "qps_1t_max": 425.8831506954738, + "qps_1t_min": 418.7859679563419, + "qps_1t_runs": [ + 425.8831506954738, + 423.5639466773652, + 418.7859679563419 + ], + "recall": 0.9151 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 6817.882761051617, + "qps_batch_max": 7284.461878589874, + "qps_batch_min": 6171.2090879693515, + "qps_batch_runs": [ + 7284.461878589874, + 6817.882761051617, + 6171.2090879693515 + ], + "recall": 0.9541, + "threads": 32 + }, + "knob": 256, + "max_ms": 8.407599999999999, + "mean_ms": 4.501758318940695, + "median_ms": 4.4242, + "nq": 579, + "p95_ms": 5.2927, + "p99_ms": 6.2167, + "qps_1t": 222.77476005830957, + "qps_1t_max": 222.97386365067362, + "qps_1t_min": 220.6723409821392, + "qps_1t_runs": [ + 222.77476005830957, + 222.97386365067362, + 220.6723409821392 + ], + "recall": 0.9531951640759931 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 4695.472766023184, + "qps_batch_max": 5402.893789913878, + "qps_batch_min": 4411.732915453992, + "qps_batch_runs": [ + 5402.893789913878, + 4695.472766023184, + 4411.732915453992 + ], + "recall": 0.9799, + "threads": 32 + }, + "knob": 512, + "max_ms": 14.365400000000001, + "mean_ms": 8.254540139616049, + "median_ms": 8.155100000000001, + "nq": 382, + "p95_ms": 9.4377, + "p99_ms": 10.7768, + "qps_1t": 120.698702672206, + "qps_1t_max": 122.33768223510813, + "qps_1t_min": 120.41764369402743, + "qps_1t_runs": [ + 122.33768223510813, + 120.41764369402743, + 120.698702672206 + ], + "recall": 0.9780104712041885 + } + ], + "system": "rust_ivf_i8x4", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_ivf_i8x4_text384_1000000_r1.load.json b/benchmarks/rust-stage-c/raw/rust_ivf_i8x4_text384_1000000_r1.load.json new file mode 100644 index 0000000..9302400 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_ivf_i8x4_text384_1000000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 18.646875, "pin_cpu_pct": 0.0, "max_cpu_pct": 97.0, "noisy": false, "seconds": 41.99746346473694, "after": 31.3} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_ivf_i8x4_text384_1000000_r1.txt b/benchmarks/rust-stage-c/raw/rust_ivf_i8x4_text384_1000000_r1.txt new file mode 100644 index 0000000..2af09b9 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_ivf_i8x4_text384_1000000_r1.txt @@ -0,0 +1,8 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/text384 --ds text384 --n 1000000 --mode ivf --exp core --queries 1000 --budget 3 --label rust_ivf_i8x4 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_ivf_i8x4_text384_1000000_r1.json --int8 4 --knobs 64,128,256,512 +rust_ivf_i8x4: ds=text384 n=1000000 d=384 mode=ivf int8=4 queries=1000 cpus=32 exps=["core"] + ingest 1.15s rss 1629 MB + ivf build 6.91s (stats Some(6.911995)) rss 2019 MB (peak 2263) + ivf+int8x4 knob=64 recall=0.8579 qps1t=771 median=1.294ms batch=16586 + ivf+int8x4 knob=128 recall=0.9151 qps1t=424 median=2.348ms batch=12032 + ivf+int8x4 knob=256 recall=0.9532 qps1t=223 median=4.424ms batch=6818 + ivf+int8x4 knob=512 recall=0.9780 qps1t=121 median=8.155ms batch=4695 diff --git a/benchmarks/rust-stage-c/raw/rust_ivf_i8x4_text384_100000_r1.json b/benchmarks/rust-stage-c/raw/rust_ivf_i8x4_text384_100000_r1.json new file mode 100644 index 0000000..9d5e547 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_ivf_i8x4_text384_100000_r1.json @@ -0,0 +1,260 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_query_ms": 0.521, + "index_s": 0.5086548, + "ingest_s": 0.1027294, + "stats_build_seconds": 0.508485, + "total_s": 0.6113841999999999 + }, + "d": 384, + "ds": "text384", + "exps": [ + "core" + ], + "first_exact_query_ms": 2.489, + "index_stats": { + "ef_search_default": null, + "graph_nodes": 0, + "index_bytes": 3173400, + "max_level": null, + "nlist": 1265, + "trained_n": 100000 + }, + "int8_rerank": 4, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 289.76171875, + "rss_mb": 212.6171875 + }, + "mem_after_first_exact_query": { + "peak_mb": 207.796875, + "rss_mb": 207.796875 + }, + "mem_after_ingest": { + "peak_mb": 196.2734375, + "rss_mb": 169.04296875 + }, + "mem_after_sweep": { + "peak_mb": 289.76171875, + "rss_mb": 221.1953125 + }, + "mem_baseline": { + "peak_mb": 7.859375, + "rss_mb": 7.859375 + }, + "mem_end": { + "peak_mb": 289.76171875, + "rss_mb": 221.21875 + }, + "memory_stats": { + "index_bytes": 3173400, + "matrix_bytes": 154140672, + "metadata_bytes": 12696122, + "quant_bytes": 39059456, + "total_bytes": 170010194 + }, + "mode": "ivf", + "n": 100000, + "quant_build_s": 0.105749, + "quant_bytes_mb": 37.25, + "queries": 1000, + "raw_vectors_mb": 146.484375, + "ref_knob": 128, + "ref_recall_pre": 0.9559, + "rss_delta_int8_mb": 37.00390625, + "started": "SystemTime { intervals: 134393522618715131 }", + "status": "ok", + "sweep": [ + { + "batch": { + "nq": 1000, + "qps_batch": 44722.319120580316, + "qps_batch_max": 44866.3878968432, + "qps_batch_min": 43898.15627743634, + "qps_batch_runs": [ + 44866.3878968432, + 43898.15627743634, + 44722.319120580316 + ], + "recall": 0.7503, + "threads": 32 + }, + "knob": 16, + "max_ms": 0.3995, + "mean_ms": 0.1314181666666666, + "median_ms": 0.1266, + "nq": 1000, + "p95_ms": 0.1706, + "p99_ms": 0.2438, + "qps_1t": 7552.202713204349, + "qps_1t_max": 7859.027901120861, + "qps_1t_min": 7429.387387574798, + "qps_1t_runs": [ + 7552.202713204349, + 7429.387387574798, + 7859.027901120861 + ], + "recall": 0.7503 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 32989.255399516376, + "qps_batch_max": 38895.97660018048, + "qps_batch_min": 32218.363178277094, + "qps_batch_runs": [ + 32218.363178277094, + 32989.255399516376, + 38895.97660018048 + ], + "recall": 0.8401, + "threads": 32 + }, + "knob": 32, + "max_ms": 0.7867, + "mean_ms": 0.19099263333333322, + "median_ms": 0.18450000000000003, + "nq": 1000, + "p95_ms": 0.24810000000000001, + "p99_ms": 0.3411, + "qps_1t": 5315.501597308232, + "qps_1t_max": 5864.851534215846, + "qps_1t_min": 4665.449917328221, + "qps_1t_runs": [ + 5864.851534215846, + 5315.501597308232, + 4665.449917328221 + ], + "recall": 0.8401 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 18057.451587972293, + "qps_batch_max": 18166.32725911904, + "qps_batch_min": 15400.442300702876, + "qps_batch_runs": [ + 18057.451587972293, + 15400.442300702876, + 18166.32725911904 + ], + "recall": 0.9083, + "threads": 32 + }, + "knob": 64, + "max_ms": 0.9566, + "mean_ms": 0.3049193999999999, + "median_ms": 0.2999, + "nq": 1000, + "p95_ms": 0.3905, + "p99_ms": 0.46599999999999997, + "qps_1t": 3251.230021597922, + "qps_1t_max": 3445.2191211039294, + "qps_1t_min": 3155.3206594620215, + "qps_1t_runs": [ + 3155.3206594620215, + 3251.230021597922, + 3445.2191211039294 + ], + "recall": 0.9083 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 11850.475441074697, + "qps_batch_max": 13342.423971532604, + "qps_batch_min": 10507.391424497564, + "qps_batch_runs": [ + 11850.475441074697, + 10507.391424497564, + 13342.423971532604 + ], + "recall": 0.9559, + "threads": 32 + }, + "knob": 128, + "max_ms": 2.1215, + "mean_ms": 0.5183298000000013, + "median_ms": 0.5057, + "nq": 1000, + "p95_ms": 0.6690999999999999, + "p99_ms": 0.8058, + "qps_1t": 1930.4590728198116, + "qps_1t_max": 1932.9778587051162, + "qps_1t_min": 1924.4040216964993, + "qps_1t_runs": [ + 1924.4040216964993, + 1932.9778587051162, + 1930.4590728198116 + ], + "recall": 0.9559 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 9071.24099112379, + "qps_batch_max": 9437.692823861507, + "qps_batch_min": 9062.986852324973, + "qps_batch_runs": [ + 9071.24099112379, + 9062.986852324973, + 9437.692823861507 + ], + "recall": 0.984, + "threads": 32 + }, + "knob": 256, + "max_ms": 20.1362, + "mean_ms": 1.1544370333333336, + "median_ms": 1.0532, + "nq": 1000, + "p95_ms": 1.7231, + "p99_ms": 2.7669, + "qps_1t": 847.0208371361043, + "qps_1t_max": 998.2722901474401, + "qps_1t_min": 780.6572493900919, + "qps_1t_runs": [ + 847.0208371361043, + 780.6572493900919, + 998.2722901474401 + ], + "recall": 0.984 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 5890.1903591720275, + "qps_batch_max": 5923.465274573347, + "qps_batch_min": 5478.575482717286, + "qps_batch_runs": [ + 5923.465274573347, + 5890.1903591720275, + 5478.575482717286 + ], + "recall": 0.9953, + "threads": 32 + }, + "knob": 512, + "max_ms": 8.711699999999999, + "mean_ms": 2.153169966666669, + "median_ms": 1.9751, + "nq": 1000, + "p95_ms": 3.31, + "p99_ms": 4.9854, + "qps_1t": 503.0871945694756, + "qps_1t_max": 525.4946901914525, + "qps_1t_min": 389.2846897718291, + "qps_1t_runs": [ + 389.2846897718291, + 503.0871945694756, + 525.4946901914525 + ], + "recall": 0.9953 + } + ], + "system": "rust_ivf_i8x4", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_ivf_i8x4_text384_100000_r1.load.json b/benchmarks/rust-stage-c/raw/rust_ivf_i8x4_text384_100000_r1.load.json new file mode 100644 index 0000000..b215d1f --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_ivf_i8x4_text384_100000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 9.825, "pin_cpu_pct": 0.0, "max_cpu_pct": 54.6, "noisy": false, "seconds": 16.520529985427856, "after": 21.9} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_ivf_i8x4_text384_100000_r1.txt b/benchmarks/rust-stage-c/raw/rust_ivf_i8x4_text384_100000_r1.txt new file mode 100644 index 0000000..17d5524 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_ivf_i8x4_text384_100000_r1.txt @@ -0,0 +1,10 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/text384 --ds text384 --n 100000 --mode ivf --exp core --queries 1000 --budget 3 --label rust_ivf_i8x4 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_ivf_i8x4_text384_100000_r1.json --int8 4 +rust_ivf_i8x4: ds=text384 n=100000 d=384 mode=ivf int8=4 queries=1000 cpus=32 exps=["core"] + ingest 0.10s rss 169 MB + ivf build 0.51s (stats Some(0.508485)) rss 213 MB (peak 290) + ivf+int8x4 knob=16 recall=0.7503 qps1t=7552 median=0.127ms batch=44722 + ivf+int8x4 knob=32 recall=0.8401 qps1t=5316 median=0.185ms batch=32989 + ivf+int8x4 knob=64 recall=0.9083 qps1t=3251 median=0.300ms batch=18057 + ivf+int8x4 knob=128 recall=0.9559 qps1t=1930 median=0.506ms batch=11850 + ivf+int8x4 knob=256 recall=0.9840 qps1t=847 median=1.053ms batch=9071 + ivf+int8x4 knob=512 recall=0.9953 qps1t=503 median=1.975ms batch=5890 diff --git a/benchmarks/rust-stage-c/raw/rust_ivf_text384_1000000_r1.json b/benchmarks/rust-stage-c/raw/rust_ivf_text384_1000000_r1.json new file mode 100644 index 0000000..604d307 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_ivf_text384_1000000_r1.json @@ -0,0 +1,351 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_query_ms": 9.3968, + "index_s": 6.5399216, + "ingest_s": 1.0997340999999998, + "stats_build_seconds": 6.5388839, + "total_s": 7.6396557 + }, + "concurrent_clients": [ + { + "clients": 1, + "qps": 128.96041345175075 + }, + { + "clients": 2, + "qps": 216.49973406449715 + }, + { + "clients": 4, + "qps": 334.6040756570717 + }, + { + "clients": 8, + "qps": 418.9263627704457 + }, + { + "clients": 16, + "qps": 385.376450986957 + } + ], + "d": 384, + "ds": "text384", + "exps": [ + "core", + "conc", + "writes" + ], + "first_exact_query_ms": 60.964, + "index_stats": { + "ef_search_default": null, + "graph_nodes": 0, + "index_bytes": 18240000, + "max_level": null, + "nlist": 4000, + "trained_n": 1000000 + }, + "int8_rerank": 0, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 1894.31640625, + "rss_mb": 1650.953125 + }, + "mem_after_first_exact_query": { + "peak_mb": 1656.41796875, + "rss_mb": 1631.3828125 + }, + "mem_after_ingest": { + "peak_mb": 1656.41796875, + "rss_mb": 1629.17578125 + }, + "mem_after_sweep": { + "peak_mb": 1894.31640625, + "rss_mb": 1660.73046875 + }, + "mem_baseline": { + "peak_mb": 7.8671875, + "rss_mb": 7.8671875 + }, + "mem_end": { + "peak_mb": 1894.31640625, + "rss_mb": 1678.0 + }, + "memory_stats": { + "index_bytes": 18240000, + "matrix_bytes": 1536688128, + "metadata_bytes": 130484346, + "quant_bytes": 0, + "total_bytes": 1685412474 + }, + "mode": "ivf", + "mutation": { + "after": { + "max_ms": 18.4266, + "mean_ms": 7.666178183292781, + "median_ms": 7.593, + "nq": 411, + "p95_ms": 8.8956, + "p99_ms": 9.9084, + "qps_1t": 129.09832609100096, + "qps_1t_max": 133.4516279734874, + "qps_1t_min": 128.88011458853276, + "qps_1t_runs": [ + 129.09832609100096, + 128.88011458853276, + 133.4516279734874 + ], + "recall": 0.9530413625304136 + }, + "delete_10k": { + "lat": { + "max_ms": 0.0098, + "mean_ms": 0.0008786699999999804, + "median_ms": 0.0007999999999999999, + "p95_ms": 0.0012, + "p99_ms": 0.0014 + }, + "per_s": 1097249.1962649636, + "seconds": 0.0091137 + }, + "first_query_after_ms": 7.959499999999999, + "index_stats_after": { + "churn": 30000, + "graph_dead": 0, + "graph_nodes": 0, + "inserts": 0, + "pending": 32, + "trained_n": 1000000 + }, + "insert_10k": { + "lat": { + "max_ms": 2.5635, + "mean_ms": 0.007208919999999992, + "median_ms": 0.0009, + "p95_ms": 0.0019, + "p99_ms": 0.005699999999999999 + }, + "per_s": 137935.790889341, + "seconds": 0.0724975 + }, + "mem_after": { + "peak_mb": 1894.31640625, + "rss_mb": 1678.0 + }, + "mode": "ivf", + "qps_after": 129.09832609100096, + "recall_after": 0.9530413625304136, + "recall_before": 0.9525547445255474, + "ref_knob": 256, + "update_10k": { + "lat": { + "max_ms": 3.0392, + "mean_ms": 0.007482409999999828, + "median_ms": 0.0015999999999999999, + "p95_ms": 0.002, + "p99_ms": 0.0053 + }, + "per_s": 133013.39178828525, + "seconds": 0.0751804 + } + }, + "n": 1000000, + "queries": 1000, + "raw_vectors_mb": 1464.84375, + "ref_knob": 256, + "ref_recall_pre": 0.9525547445255474, + "started": "SystemTime { intervals: 134393590980783737 }", + "status": "ok", + "sweep": [ + { + "batch": { + "nq": 1000, + "qps_batch": 19772.067604653552, + "qps_batch_max": 19818.030840819596, + "qps_batch_min": 19025.440819463787, + "qps_batch_runs": [ + 19772.067604653552, + 19025.440819463787, + 19818.030840819596 + ], + "recall": 0.6991, + "threads": 32 + }, + "knob": 16, + "max_ms": 1.2993, + "mean_ms": 0.7137250666666674, + "median_ms": 0.7057, + "nq": 1000, + "p95_ms": 0.9147, + "p99_ms": 1.0221, + "qps_1t": 1412.479653230596, + "qps_1t_max": 1444.4191831578426, + "qps_1t_min": 1349.7451546173554, + "qps_1t_runs": [ + 1349.7451546173554, + 1412.479653230596, + 1444.4191831578426 + ], + "recall": 0.6991 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 16139.783436385851, + "qps_batch_max": 16237.588393371816, + "qps_batch_min": 16127.91370921049, + "qps_batch_runs": [ + 16127.91370921049, + 16139.783436385851, + 16237.588393371816 + ], + "recall": 0.7841, + "threads": 32 + }, + "knob": 32, + "max_ms": 2.1309, + "mean_ms": 1.224272266666668, + "median_ms": 1.2046999999999999, + "nq": 1000, + "p95_ms": 1.5315999999999999, + "p99_ms": 1.7568, + "qps_1t": 829.0279150279533, + "qps_1t_max": 834.6070828429298, + "qps_1t_min": 788.38475888277, + "qps_1t_runs": [ + 829.0279150279533, + 788.38475888277, + 834.6070828429298 + ], + "recall": 0.7841 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 14671.038636180248, + "qps_batch_max": 14817.163609638861, + "qps_batch_min": 14270.363451886757, + "qps_batch_runs": [ + 14817.163609638861, + 14270.363451886757, + 14671.038636180248 + ], + "recall": 0.8579, + "threads": 32 + }, + "knob": 64, + "max_ms": 5.423500000000001, + "mean_ms": 2.2403532333333334, + "median_ms": 2.2186, + "nq": 1000, + "p95_ms": 2.7062000000000004, + "p99_ms": 3.0867999999999998, + "qps_1t": 444.2577377258688, + "qps_1t_max": 453.3934025184462, + "qps_1t_min": 441.59390671063767, + "qps_1t_runs": [ + 441.59390671063767, + 453.3934025184462, + 444.2577377258688 + ], + "recall": 0.8579 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 12658.16375613288, + "qps_batch_max": 12746.907600216187, + "qps_batch_min": 12417.70168210187, + "qps_batch_runs": [ + 12658.16375613288, + 12417.70168210187, + 12746.907600216187 + ], + "recall": 0.9151, + "threads": 32 + }, + "knob": 128, + "max_ms": 7.2577, + "mean_ms": 4.119591492329154, + "median_ms": 4.0908999999999995, + "nq": 717, + "p95_ms": 4.9607, + "p99_ms": 5.6181, + "qps_1t": 243.74408869592432, + "qps_1t_max": 243.75665932484293, + "qps_1t_min": 240.7515962636692, + "qps_1t_runs": [ + 240.7515962636692, + 243.74408869592432, + 243.75665932484293 + ], + "recall": 0.9156206415620641 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 10067.715454144576, + "qps_batch_max": 10130.829532583788, + "qps_batch_min": 9901.695962484455, + "qps_batch_runs": [ + 10130.829532583788, + 9901.695962484455, + 10067.715454144576 + ], + "recall": 0.9541, + "threads": 32 + }, + "knob": 256, + "max_ms": 12.440700000000001, + "mean_ms": 7.616362530413627, + "median_ms": 7.57, + "nq": 411, + "p95_ms": 8.6946, + "p99_ms": 9.3189, + "qps_1t": 131.28368097209477, + "qps_1t_max": 131.64685701777216, + "qps_1t_min": 130.9600733962703, + "qps_1t_runs": [ + 130.9600733962703, + 131.28368097209477, + 131.64685701777216 + ], + "recall": 0.9525547445255474 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 6771.310499120068, + "qps_batch_max": 6785.056862169034, + "qps_batch_min": 6529.6527857131205, + "qps_batch_runs": [ + 6771.310499120068, + 6529.6527857131205, + 6785.056862169034 + ], + "recall": 0.9799, + "threads": 32 + }, + "knob": 512, + "max_ms": 19.3372, + "mean_ms": 14.776854629629641, + "median_ms": 14.7035, + "nq": 180, + "p95_ms": 16.6358, + "p99_ms": 18.1961, + "qps_1t": 66.98804304368811, + "qps_1t_max": 69.87200613010404, + "qps_1t_min": 66.26622721849944, + "qps_1t_runs": [ + 66.98804304368811, + 66.26622721849944, + 69.87200613010404 + ], + "recall": 0.9772222222222222 + } + ], + "system": "rust_ivf", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_ivf_text384_1000000_r1.load.json b/benchmarks/rust-stage-c/raw/rust_ivf_text384_1000000_r1.load.json new file mode 100644 index 0000000..8dc0c31 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_ivf_text384_1000000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 13.796875, "pin_cpu_pct": 0.0, "max_cpu_pct": 79.1, "noisy": false, "seconds": 77.71169972419739, "after": 29.5} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_ivf_text384_1000000_r1.txt b/benchmarks/rust-stage-c/raw/rust_ivf_text384_1000000_r1.txt new file mode 100644 index 0000000..3b83682 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_ivf_text384_1000000_r1.txt @@ -0,0 +1,16 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/text384 --ds text384 --n 1000000 --mode ivf --exp core,conc,writes --queries 1000 --budget 3 --label rust_ivf --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_ivf_text384_1000000_r1.json +rust_ivf: ds=text384 n=1000000 d=384 mode=ivf int8=0 queries=1000 cpus=32 exps=["core", "conc", "writes"] + ingest 1.10s rss 1629 MB + ivf build 6.54s (stats Some(6.5388839)) rss 1651 MB (peak 1894) + ivf knob=16 recall=0.6991 qps1t=1412 median=0.706ms batch=19772 + ivf knob=32 recall=0.7841 qps1t=829 median=1.205ms batch=16140 + ivf knob=64 recall=0.8579 qps1t=444 median=2.219ms batch=14671 + ivf knob=128 recall=0.9156 qps1t=244 median=4.091ms batch=12658 + ivf knob=256 recall=0.9526 qps1t=131 median=7.570ms batch=10068 + ivf knob=512 recall=0.9772 qps1t=67 median=14.704ms batch=6771 + concurrent clients=1 (1 scan thread each): 129.0 qps + concurrent clients=2 (1 scan thread each): 216.5 qps + concurrent clients=4 (1 scan thread each): 334.6 qps + concurrent clients=8 (1 scan thread each): 418.9 qps + concurrent clients=16 (1 scan thread each): 385.4 qps + writes: ins 137936/s upd 133013/s del 1097249/s first-query 8.0 ms recall_after=0.9530413625304136 diff --git a/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r1.json b/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r1.json new file mode 100644 index 0000000..f9cf80b --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r1.json @@ -0,0 +1,351 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_query_ms": 0.7931, + "index_s": 0.4762037, + "ingest_s": 0.10406269999999998, + "stats_build_seconds": 0.475795, + "total_s": 0.5802664 + }, + "concurrent_clients": [ + { + "clients": 1, + "qps": 364.2902377982018 + }, + { + "clients": 2, + "qps": 681.5437973002139 + }, + { + "clients": 4, + "qps": 1292.2841708911258 + }, + { + "clients": 8, + "qps": 2284.675265973058 + }, + { + "clients": 16, + "qps": 4370.6897576341335 + } + ], + "d": 384, + "ds": "text384", + "exps": [ + "core", + "conc", + "writes" + ], + "first_exact_query_ms": 6.5432, + "index_stats": { + "ef_search_default": null, + "graph_nodes": 0, + "index_bytes": 3173400, + "max_level": null, + "nlist": 1265, + "trained_n": 100000 + }, + "int8_rerank": 0, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 253.6171875, + "rss_mb": 174.15234375 + }, + "mem_after_first_exact_query": { + "peak_mb": 196.27734375, + "rss_mb": 171.49609375 + }, + "mem_after_ingest": { + "peak_mb": 196.27734375, + "rss_mb": 169.01171875 + }, + "mem_after_sweep": { + "peak_mb": 253.6171875, + "rss_mb": 181.453125 + }, + "mem_baseline": { + "peak_mb": 7.859375, + "rss_mb": 7.859375 + }, + "mem_end": { + "peak_mb": 253.6171875, + "rss_mb": 196.62109375 + }, + "memory_stats": { + "index_bytes": 3173400, + "matrix_bytes": 154140672, + "metadata_bytes": 12696122, + "quant_bytes": 0, + "total_bytes": 170010194 + }, + "mode": "ivf", + "mutation": { + "after": { + "max_ms": 3.9468000000000005, + "mean_ms": 1.175536566666667, + "median_ms": 1.1340000000000001, + "nq": 1000, + "p95_ms": 1.5384, + "p99_ms": 2.0273000000000003, + "qps_1t": 857.1779524573111, + "qps_1t_max": 865.9507390066912, + "qps_1t_min": 829.7442056265456, + "qps_1t_runs": [ + 865.9507390066912, + 829.7442056265456, + 857.1779524573111 + ], + "recall": 0.9529 + }, + "delete_10k": { + "lat": { + "max_ms": 0.0143, + "mean_ms": 0.0010295100000001479, + "median_ms": 0.001, + "p95_ms": 0.0013, + "p99_ms": 0.0015999999999999999 + }, + "per_s": 911294.5850875755, + "seconds": 0.0109734 + }, + "first_query_after_ms": 3.8362000000000003, + "index_stats_after": { + "churn": 30000, + "graph_dead": 0, + "graph_nodes": 0, + "inserts": 0, + "pending": 32, + "trained_n": 100000 + }, + "insert_10k": { + "lat": { + "max_ms": 2.0631999999999997, + "mean_ms": 0.006330839999999924, + "median_ms": 0.0013, + "p95_ms": 0.0028, + "p99_ms": 0.0134 + }, + "per_s": 156139.97011480972, + "seconds": 0.0640451 + }, + "mem_after": { + "peak_mb": 253.6171875, + "rss_mb": 196.61328125 + }, + "mode": "ivf", + "qps_after": 857.1779524573111, + "recall_after": 0.9529, + "recall_before": 0.9559, + "ref_knob": 128, + "update_10k": { + "lat": { + "max_ms": 2.2239, + "mean_ms": 0.007674539999999998, + "median_ms": 0.0022, + "p95_ms": 0.004, + "p99_ms": 0.0072 + }, + "per_s": 128695.65665028371, + "seconds": 0.0777027 + } + }, + "n": 100000, + "queries": 1000, + "raw_vectors_mb": 146.484375, + "ref_knob": 128, + "ref_recall_pre": 0.9559, + "started": "SystemTime { intervals: 134393515368474485 }", + "status": "ok", + "sweep": [ + { + "batch": { + "nq": 1000, + "qps_batch": 76515.7775533315, + "qps_batch_max": 79262.54131559966, + "qps_batch_min": 75700.22710068131, + "qps_batch_runs": [ + 75700.22710068131, + 79262.54131559966, + 76515.7775533315 + ], + "recall": 0.7503, + "threads": 32 + }, + "knob": 16, + "max_ms": 0.6472, + "mean_ms": 0.17145246666666636, + "median_ms": 0.1688, + "nq": 1000, + "p95_ms": 0.2139, + "p99_ms": 0.2364, + "qps_1t": 5847.59073414162, + "qps_1t_max": 5893.727835899759, + "qps_1t_min": 5757.884415076675, + "qps_1t_runs": [ + 5757.884415076675, + 5847.59073414162, + 5893.727835899759 + ], + "recall": 0.7503 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 74357.73506338996, + "qps_batch_max": 74582.33890214797, + "qps_batch_min": 72518.94557453135, + "qps_batch_runs": [ + 72518.94557453135, + 74357.73506338996, + 74582.33890214797 + ], + "recall": 0.8401, + "threads": 32 + }, + "knob": 32, + "max_ms": 0.737, + "mean_ms": 0.3050216999999996, + "median_ms": 0.2986, + "nq": 1000, + "p95_ms": 0.38220000000000004, + "p99_ms": 0.4528, + "qps_1t": 3334.77284361083, + "qps_1t_max": 3353.1144733162473, + "qps_1t_min": 3154.9284399131257, + "qps_1t_runs": [ + 3334.77284361083, + 3154.9284399131257, + 3353.1144733162473 + ], + "recall": 0.8401 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 24238.193575909176, + "qps_batch_max": 27758.192830613956, + "qps_batch_min": 23917.036583499157, + "qps_batch_runs": [ + 27758.192830613956, + 23917.036583499157, + 24238.193575909176 + ], + "recall": 0.9083, + "threads": 32 + }, + "knob": 64, + "max_ms": 1.0917999999999999, + "mean_ms": 0.5617666333333337, + "median_ms": 0.5523, + "nq": 1000, + "p95_ms": 0.7181000000000001, + "p99_ms": 0.8084, + "qps_1t": 1767.667036580453, + "qps_1t_max": 1835.8955397468603, + "qps_1t_min": 1739.4656222451204, + "qps_1t_runs": [ + 1835.8955397468603, + 1767.667036580453, + 1739.4656222451204 + ], + "recall": 0.9083 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 21140.531684371865, + "qps_batch_max": 24188.944684721293, + "qps_batch_min": 18413.086548872012, + "qps_batch_runs": [ + 21140.531684371865, + 24188.944684721293, + 18413.086548872012 + ], + "recall": 0.9559, + "threads": 32 + }, + "knob": 128, + "max_ms": 10.0177, + "mean_ms": 1.206331900000001, + "median_ms": 1.1614, + "nq": 1000, + "p95_ms": 1.5769, + "p99_ms": 1.7577999999999998, + "qps_1t": 888.9671179952729, + "qps_1t_max": 907.3714402117071, + "qps_1t_min": 718.385540565724, + "qps_1t_runs": [ + 907.3714402117071, + 888.9671179952729, + 718.385540565724 + ], + "recall": 0.9559 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 16444.17695249935, + "qps_batch_max": 16571.98539013768, + "qps_batch_min": 15306.786263690006, + "qps_batch_runs": [ + 16444.17695249935, + 15306.786263690006, + 16571.98539013768 + ], + "recall": 0.984, + "threads": 32 + }, + "knob": 256, + "max_ms": 4.584099999999999, + "mean_ms": 2.4544026666666663, + "median_ms": 2.4616, + "nq": 1000, + "p95_ms": 3.0417, + "p99_ms": 3.1740000000000004, + "qps_1t": 409.414536394128, + "qps_1t_max": 463.46726254961516, + "qps_1t_min": 361.9194048191967, + "qps_1t_runs": [ + 463.46726254961516, + 409.414536394128, + 361.9194048191967 + ], + "recall": 0.984 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 9255.265783467503, + "qps_batch_max": 10155.078199179672, + "qps_batch_min": 9242.426062902103, + "qps_batch_runs": [ + 10155.078199179672, + 9255.265783467503, + 9242.426062902103 + ], + "recall": 0.9953, + "threads": 32 + }, + "knob": 512, + "max_ms": 12.4969, + "mean_ms": 5.355488457008247, + "median_ms": 5.3503, + "nq": 566, + "p95_ms": 6.6727, + "p99_ms": 8.1777, + "qps_1t": 185.80865700260304, + "qps_1t_max": 196.41267176826042, + "qps_1t_min": 178.7864958200601, + "qps_1t_runs": [ + 185.80865700260304, + 178.7864958200601, + 196.41267176826042 + ], + "recall": 0.9952296819787986 + } + ], + "system": "rust_ivf", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r1.load.json b/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r1.load.json new file mode 100644 index 0000000..4c8f547 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 11.63125, "pin_cpu_pct": 2.4, "max_cpu_pct": 61.0, "noisy": false, "seconds": 44.84973216056824, "after": 23.5} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r1.txt b/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r1.txt new file mode 100644 index 0000000..06360c0 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r1.txt @@ -0,0 +1,16 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/text384 --ds text384 --n 100000 --mode ivf --exp core,conc,writes --queries 1000 --budget 3 --label rust_ivf --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_ivf_text384_100000_r1.json +rust_ivf: ds=text384 n=100000 d=384 mode=ivf int8=0 queries=1000 cpus=32 exps=["core", "conc", "writes"] + ingest 0.10s rss 169 MB + ivf build 0.48s (stats Some(0.475795)) rss 174 MB (peak 254) + ivf knob=16 recall=0.7503 qps1t=5848 median=0.169ms batch=76516 + ivf knob=32 recall=0.8401 qps1t=3335 median=0.299ms batch=74358 + ivf knob=64 recall=0.9083 qps1t=1768 median=0.552ms batch=24238 + ivf knob=128 recall=0.9559 qps1t=889 median=1.161ms batch=21141 + ivf knob=256 recall=0.9840 qps1t=409 median=2.462ms batch=16444 + ivf knob=512 recall=0.9952 qps1t=186 median=5.350ms batch=9255 + concurrent clients=1 (1 scan thread each): 364.3 qps + concurrent clients=2 (1 scan thread each): 681.5 qps + concurrent clients=4 (1 scan thread each): 1292.3 qps + concurrent clients=8 (1 scan thread each): 2284.7 qps + concurrent clients=16 (1 scan thread each): 4370.7 qps + writes: ins 156140/s upd 128696/s del 911295/s first-query 3.8 ms recall_after=0.9529 diff --git a/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r2.json b/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r2.json new file mode 100644 index 0000000..865f17c --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r2.json @@ -0,0 +1,257 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_query_ms": 1.2001, + "index_s": 0.4975752, + "ingest_s": 0.09910169999999999, + "stats_build_seconds": 0.4971604, + "total_s": 0.5966769 + }, + "d": 384, + "ds": "text384", + "exps": [ + "core" + ], + "first_exact_query_ms": 7.011, + "index_stats": { + "ef_search_default": null, + "graph_nodes": 0, + "index_bytes": 3173400, + "max_level": null, + "nlist": 1265, + "trained_n": 100000 + }, + "int8_rerank": 0, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 253.484375, + "rss_mb": 174.15625 + }, + "mem_after_first_exact_query": { + "peak_mb": 196.515625, + "rss_mb": 171.546875 + }, + "mem_after_ingest": { + "peak_mb": 196.515625, + "rss_mb": 169.0546875 + }, + "mem_after_sweep": { + "peak_mb": 253.484375, + "rss_mb": 181.64453125 + }, + "mem_baseline": { + "peak_mb": 7.859375, + "rss_mb": 7.859375 + }, + "mem_end": { + "peak_mb": 253.484375, + "rss_mb": 181.66796875 + }, + "memory_stats": { + "index_bytes": 3173400, + "matrix_bytes": 154140672, + "metadata_bytes": 12696122, + "quant_bytes": 0, + "total_bytes": 170010194 + }, + "mode": "ivf", + "n": 100000, + "queries": 1000, + "raw_vectors_mb": 146.484375, + "ref_knob": 128, + "ref_recall_pre": 0.9559, + "started": "SystemTime { intervals: 134393576825989414 }", + "status": "ok", + "sweep": [ + { + "batch": { + "nq": 1000, + "qps_batch": 75992.46154781446, + "qps_batch_max": 78528.68652918912, + "qps_batch_min": 74198.8380461962, + "qps_batch_runs": [ + 75992.46154781446, + 74198.8380461962, + 78528.68652918912 + ], + "recall": 0.7503, + "threads": 32 + }, + "knob": 16, + "max_ms": 0.6322, + "mean_ms": 0.16981736666666652, + "median_ms": 0.1666, + "nq": 1000, + "p95_ms": 0.2125, + "p99_ms": 0.244, + "qps_1t": 5921.648302648353, + "qps_1t_max": 6046.471974300077, + "qps_1t_min": 5707.94174474654, + "qps_1t_runs": [ + 5707.94174474654, + 5921.648302648353, + 6046.471974300077 + ], + "recall": 0.7503 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 70968.29136741704, + "qps_batch_max": 71197.18059164856, + "qps_batch_min": 68273.83268814562, + "qps_batch_runs": [ + 70968.29136741704, + 68273.83268814562, + 71197.18059164856 + ], + "recall": 0.8401, + "threads": 32 + }, + "knob": 32, + "max_ms": 0.9356, + "mean_ms": 0.29964830000000064, + "median_ms": 0.2943, + "nq": 1000, + "p95_ms": 0.3728, + "p99_ms": 0.4251, + "qps_1t": 3350.563045366958, + "qps_1t_max": 3383.1456420516865, + "qps_1t_min": 3279.713300582118, + "qps_1t_runs": [ + 3350.563045366958, + 3383.1456420516865, + 3279.713300582118 + ], + "recall": 0.8401 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 26841.746431389813, + "qps_batch_max": 27475.244804431204, + "qps_batch_min": 26507.831738887253, + "qps_batch_runs": [ + 26507.831738887253, + 27475.244804431204, + 26841.746431389813 + ], + "recall": 0.9083, + "threads": 32 + }, + "knob": 64, + "max_ms": 3.0759, + "mean_ms": 0.7065145000000003, + "median_ms": 0.7081000000000001, + "nq": 1000, + "p95_ms": 0.9184, + "p99_ms": 1.0763, + "qps_1t": 1344.7392322023095, + "qps_1t_max": 1586.1097391261862, + "qps_1t_min": 1341.5048867669263, + "qps_1t_runs": [ + 1586.1097391261862, + 1344.7392322023095, + 1341.5048867669263 + ], + "recall": 0.9083 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 24360.417245226574, + "qps_batch_max": 25462.072969208715, + "qps_batch_min": 17228.814157950324, + "qps_batch_runs": [ + 17228.814157950324, + 25462.072969208715, + 24360.417245226574 + ], + "recall": 0.9559, + "threads": 32 + }, + "knob": 128, + "max_ms": 2.2752, + "mean_ms": 1.4433016000000032, + "median_ms": 1.4438, + "nq": 1000, + "p95_ms": 1.7207, + "p99_ms": 1.9143000000000001, + "qps_1t": 692.2305565242935, + "qps_1t_max": 696.5762095122908, + "qps_1t_min": 689.7948915583802, + "qps_1t_runs": [ + 696.5762095122908, + 689.7948915583802, + 692.2305565242935 + ], + "recall": 0.9559 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 14400.341000074883, + "qps_batch_max": 14658.543878885246, + "qps_batch_min": 13468.79347894895, + "qps_batch_runs": [ + 13468.79347894895, + 14658.543878885246, + 14400.341000074883 + ], + "recall": 0.984, + "threads": 32 + }, + "knob": 256, + "max_ms": 6.1386, + "mean_ms": 2.7848689333333283, + "median_ms": 2.7717, + "nq": 1000, + "p95_ms": 3.2862999999999998, + "p99_ms": 3.6715, + "qps_1t": 359.53100331117236, + "qps_1t_max": 363.304250706963, + "qps_1t_min": 354.5229829450334, + "qps_1t_runs": [ + 354.5229829450334, + 363.304250706963, + 359.53100331117236 + ], + "recall": 0.984 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 9164.835990677528, + "qps_batch_max": 9613.831611816551, + "qps_batch_min": 7589.084468027946, + "qps_batch_runs": [ + 7589.084468027946, + 9613.831611816551, + 9164.835990677528 + ], + "recall": 0.9953, + "threads": 32 + }, + "knob": 512, + "max_ms": 12.5402, + "mean_ms": 5.485680896358537, + "median_ms": 5.413, + "nq": 595, + "p95_ms": 6.4157, + "p99_ms": 7.5737, + "qps_1t": 182.40600572540268, + "qps_1t_max": 183.3746105986213, + "qps_1t_min": 181.11186491526516, + "qps_1t_runs": [ + 181.11186491526516, + 183.3746105986213, + 182.40600572540268 + ], + "recall": 0.9951260504201681 + } + ], + "system": "rust_ivf", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r2.load.json b/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r2.load.json new file mode 100644 index 0000000..26c5ae0 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r2.load.json @@ -0,0 +1 @@ +{"total_pct": 15.503125, "pin_cpu_pct": 0.0, "max_cpu_pct": 62.0, "noisy": false, "seconds": 28.592809438705444, "after": 23.2} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r2.txt b/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r2.txt new file mode 100644 index 0000000..b2ab624 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r2.txt @@ -0,0 +1,10 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/text384 --ds text384 --n 100000 --mode ivf --exp core --queries 1000 --budget 3 --label rust_ivf --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_ivf_text384_100000_r2.json +rust_ivf: ds=text384 n=100000 d=384 mode=ivf int8=0 queries=1000 cpus=32 exps=["core"] + ingest 0.10s rss 169 MB + ivf build 0.50s (stats Some(0.4971604)) rss 174 MB (peak 253) + ivf knob=16 recall=0.7503 qps1t=5922 median=0.167ms batch=75992 + ivf knob=32 recall=0.8401 qps1t=3351 median=0.294ms batch=70968 + ivf knob=64 recall=0.9083 qps1t=1345 median=0.708ms batch=26842 + ivf knob=128 recall=0.9559 qps1t=692 median=1.444ms batch=24360 + ivf knob=256 recall=0.9840 qps1t=360 median=2.772ms batch=14400 + ivf knob=512 recall=0.9951 qps1t=182 median=5.413ms batch=9165 diff --git a/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r3.json b/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r3.json new file mode 100644 index 0000000..3044a8b --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r3.json @@ -0,0 +1,257 @@ +{ + "all_threads": 32, + "budget": 3.0, + "build": { + "first_query_ms": 0.7523, + "index_s": 0.4718371, + "ingest_s": 0.0995579, + "stats_build_seconds": 0.4714274, + "total_s": 0.571395 + }, + "d": 384, + "ds": "text384", + "exps": [ + "core" + ], + "first_exact_query_ms": 6.0479, + "index_stats": { + "ef_search_default": null, + "graph_nodes": 0, + "index_bytes": 3173400, + "max_level": null, + "nlist": 1265, + "trained_n": 100000 + }, + "int8_rerank": 0, + "k": 10, + "knob_name": "nprobe", + "mem_after_build": { + "peak_mb": 253.49609375, + "rss_mb": 174.16015625 + }, + "mem_after_first_exact_query": { + "peak_mb": 196.484375, + "rss_mb": 171.55078125 + }, + "mem_after_ingest": { + "peak_mb": 196.484375, + "rss_mb": 169.05859375 + }, + "mem_after_sweep": { + "peak_mb": 253.49609375, + "rss_mb": 180.890625 + }, + "mem_baseline": { + "peak_mb": 7.86328125, + "rss_mb": 7.86328125 + }, + "mem_end": { + "peak_mb": 253.49609375, + "rss_mb": 180.9140625 + }, + "memory_stats": { + "index_bytes": 3173400, + "matrix_bytes": 154140672, + "metadata_bytes": 12696122, + "quant_bytes": 0, + "total_bytes": 170010194 + }, + "mode": "ivf", + "n": 100000, + "queries": 1000, + "raw_vectors_mb": 146.484375, + "ref_knob": 128, + "ref_recall_pre": 0.9559, + "started": "SystemTime { intervals: 134393535784609787 }", + "status": "ok", + "sweep": [ + { + "batch": { + "nq": 1000, + "qps_batch": 71437.75628295066, + "qps_batch_max": 79502.63153710388, + "qps_batch_min": 63786.49384779266, + "qps_batch_runs": [ + 63786.49384779266, + 71437.75628295066, + 79502.63153710388 + ], + "recall": 0.7503, + "threads": 32 + }, + "knob": 16, + "max_ms": 0.3678, + "mean_ms": 0.1764951666666665, + "median_ms": 0.1747, + "nq": 1000, + "p95_ms": 0.2188, + "p99_ms": 0.2422, + "qps_1t": 5646.6836180109, + "qps_1t_max": 5731.286347101601, + "qps_1t_min": 5620.83531233577, + "qps_1t_runs": [ + 5620.83531233577, + 5731.286347101601, + 5646.6836180109 + ], + "recall": 0.7503 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 65618.95075297746, + "qps_batch_max": 71352.63184182548, + "qps_batch_min": 64359.58990069315, + "qps_batch_runs": [ + 64359.58990069315, + 71352.63184182548, + 65618.95075297746 + ], + "recall": 0.8401, + "threads": 32 + }, + "knob": 32, + "max_ms": 0.5165000000000001, + "mean_ms": 0.3053721333333329, + "median_ms": 0.30069999999999997, + "nq": 1000, + "p95_ms": 0.3776, + "p99_ms": 0.4213, + "qps_1t": 3243.9432335885663, + "qps_1t_max": 3393.79722468838, + "qps_1t_min": 3192.9051094144734, + "qps_1t_runs": [ + 3243.9432335885663, + 3393.79722468838, + 3192.9051094144734 + ], + "recall": 0.8401 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 28068.520873155547, + "qps_batch_max": 29620.76534133489, + "qps_batch_min": 24537.227882142786, + "qps_batch_runs": [ + 28068.520873155547, + 24537.227882142786, + 29620.76534133489 + ], + "recall": 0.9083, + "threads": 32 + }, + "knob": 64, + "max_ms": 1.189, + "mean_ms": 0.6739389999999992, + "median_ms": 0.6824, + "nq": 1000, + "p95_ms": 0.8784, + "p99_ms": 0.9571000000000001, + "qps_1t": 1385.082029405571, + "qps_1t_max": 1821.9887809218824, + "qps_1t_min": 1331.5804408410106, + "qps_1t_runs": [ + 1821.9887809218824, + 1385.082029405571, + 1331.5804408410106 + ], + "recall": 0.9083 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 24902.506686323046, + "qps_batch_max": 27160.84926543483, + "qps_batch_min": 16246.082663317808, + "qps_batch_runs": [ + 16246.082663317808, + 27160.84926543483, + 24902.506686323046 + ], + "recall": 0.9559, + "threads": 32 + }, + "knob": 128, + "max_ms": 3.4328000000000003, + "mean_ms": 1.4417884, + "median_ms": 1.4498, + "nq": 1000, + "p95_ms": 1.6714, + "p99_ms": 1.8121999999999998, + "qps_1t": 697.2317737164961, + "qps_1t_max": 697.6260552727738, + "qps_1t_min": 686.0172936727529, + "qps_1t_runs": [ + 697.6260552727738, + 686.0172936727529, + 697.2317737164961 + ], + "recall": 0.9559 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 16415.88663845323, + "qps_batch_max": 16804.971582793052, + "qps_batch_min": 15395.62918087555, + "qps_batch_runs": [ + 15395.62918087555, + 16804.971582793052, + 16415.88663845323 + ], + "recall": 0.984, + "threads": 32 + }, + "knob": 256, + "max_ms": 4.3371, + "mean_ms": 2.8034798333333315, + "median_ms": 2.8016, + "nq": 1000, + "p95_ms": 3.1716, + "p99_ms": 3.3443, + "qps_1t": 355.8095498073409, + "qps_1t_max": 359.26852066947174, + "qps_1t_min": 355.04885507750765, + "qps_1t_runs": [ + 359.26852066947174, + 355.04885507750765, + 355.8095498073409 + ], + "recall": 0.984 + }, + { + "batch": { + "nq": 1000, + "qps_batch": 9225.832700594974, + "qps_batch_max": 9425.434913130479, + "qps_batch_min": 9069.07734834689, + "qps_batch_runs": [ + 9225.832700594974, + 9425.434913130479, + 9069.07734834689 + ], + "recall": 0.9953, + "threads": 32 + }, + "knob": 512, + "max_ms": 7.6560999999999995, + "mean_ms": 5.473558195211785, + "median_ms": 5.4916, + "nq": 543, + "p95_ms": 6.0105, + "p99_ms": 6.2293, + "qps_1t": 182.090179977196, + "qps_1t_max": 184.07581482786617, + "qps_1t_min": 181.9390541695557, + "qps_1t_runs": [ + 181.9390541695557, + 182.090179977196, + 184.07581482786617 + ], + "recall": 0.9950276243093923 + } + ], + "system": "rust_ivf", + "version": "isongraph-vector-rs HEAD (cargo release, default flags)" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r3.load.json b/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r3.load.json new file mode 100644 index 0000000..991d4ec --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r3.load.json @@ -0,0 +1 @@ +{"total_pct": 16.88125, "pin_cpu_pct": 0.0, "max_cpu_pct": 76.6, "noisy": false, "seconds": 27.626598596572876, "after": 20.0} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r3.txt b/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r3.txt new file mode 100644 index 0000000..8553f9a --- /dev/null +++ b/benchmarks/rust-stage-c/raw/rust_ivf_text384_100000_r3.txt @@ -0,0 +1,10 @@ +D:\Goldmine\ISON\bench_data\rs_target_stagec\release\examples\stage_c_bench.exe --ds-dir D:\Goldmine\ISON\bench_data\prep/text384 --ds text384 --n 100000 --mode ivf --exp core --queries 1000 --budget 3 --label rust_ivf --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\rust_ivf_text384_100000_r3.json +rust_ivf: ds=text384 n=100000 d=384 mode=ivf int8=0 queries=1000 cpus=32 exps=["core"] + ingest 0.10s rss 169 MB + ivf build 0.47s (stats Some(0.4714274)) rss 174 MB (peak 253) + ivf knob=16 recall=0.7503 qps1t=5647 median=0.175ms batch=71438 + ivf knob=32 recall=0.8401 qps1t=3244 median=0.301ms batch=65619 + ivf knob=64 recall=0.9083 qps1t=1385 median=0.682ms batch=28069 + ivf knob=128 recall=0.9559 qps1t=697 median=1.450ms batch=24903 + ivf knob=256 recall=0.9840 qps1t=356 median=2.802ms batch=16416 + ivf knob=512 recall=0.9950 qps1t=182 median=5.492ms batch=9226 diff --git a/benchmarks/rust-stage-c/raw/usearch_text384_1000000_r1.json b/benchmarks/rust-stage-c/raw/usearch_text384_1000000_r1.json new file mode 100644 index 0000000..3593548 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/usearch_text384_1000000_r1.json @@ -0,0 +1,369 @@ +{ + "system": "usearch", + "ds": "text384", + "n": 1000000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 20:42:19", + "mem_baseline": { + "rss_mb": 37.15625, + "peak_mb": 37.15625 + }, + "version": "usearch 2.26.0", + "build": { + "ingest_s": 215.80608449981082, + "index_s": 3.3999676816165447e-06, + "total_s": 215.8060878997785, + "first_query_ms": 4.411200003232807, + "ready_s": 215.81049909978174 + }, + "mem_after_build": { + "rss_mb": 1751.06640625, + "peak_mb": 1781.09375 + }, + "mem_after_first_query": { + "rss_mb": 1751.09765625, + "peak_mb": 1781.09375 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.7263, + "qps_1t_runs": [ + 2904.5534658953216, + 2958.1169117713853, + 2924.474282648109 + ], + "qps_1t": 2924.474282648109, + "qps_1t_min": 2904.5534658953216, + "qps_1t_max": 2958.1169117713853, + "median_ms": 0.31564998789690435, + "p95_ms": 0.5032249900978057, + "p99_ms": 0.7969200127990864, + "mean_ms": 0.3414272333417709, + "knob": 24, + "batch": { + "recall": 0.7263, + "qps_batch_runs": [ + 22218.568482297716, + 21557.671076593324, + 20183.79361906828 + ], + "qps_batch": 21557.671076593324, + "qps_batch_min": 20183.79361906828, + "qps_batch_max": 22218.568482297716, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.7728, + "qps_1t_runs": [ + 2496.5073834079276, + 2374.2592305928783, + 2407.5025487441217 + ], + "qps_1t": 2407.5025487441217, + "qps_1t_min": 2374.2592305928783, + "qps_1t_max": 2496.5073834079276, + "median_ms": 0.3838500124402344, + "p95_ms": 0.6040950014721597, + "p99_ms": 0.9179270040476688, + "mean_ms": 0.4123706001555547, + "knob": 32, + "batch": { + "recall": 0.7728, + "qps_batch_runs": [ + 14779.672038111534, + 14009.035828416192, + 17967.5793061287 + ], + "qps_batch": 14779.672038111534, + "qps_batch_min": 14009.035828416192, + "qps_batch_max": 17967.5793061287, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8304, + "qps_1t_runs": [ + 1769.9246625738135, + 1761.8875893865413, + 1581.5472656563716 + ], + "qps_1t": 1761.8875893865413, + "qps_1t_min": 1581.5472656563716, + "qps_1t_max": 1769.9246625738135, + "median_ms": 0.538699998287484, + "p95_ms": 0.9367450315039597, + "p99_ms": 1.383335996652019, + "mean_ms": 0.588287033024244, + "knob": 48, + "batch": { + "recall": 0.8304, + "qps_batch_runs": [ + 12522.50921515546, + 12115.998570620053, + 12921.900040179744 + ], + "qps_batch": 12522.50921515546, + "qps_batch_min": 12115.998570620053, + "qps_batch_max": 12921.900040179744, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8664, + "qps_1t_runs": [ + 1433.3356331425962, + 1409.3556683574302, + 1311.637449727638 + ], + "qps_1t": 1409.3556683574302, + "qps_1t_min": 1311.637449727638, + "qps_1t_max": 1433.3356331425962, + "median_ms": 0.6657500052824616, + "p95_ms": 1.0281150287482888, + "p99_ms": 1.525818033842367, + "mean_ms": 0.7232077334386607, + "knob": 64, + "batch": { + "recall": 0.8664, + "qps_batch_runs": [ + 9055.411880325639, + 9527.69319260648, + 9552.476052423246 + ], + "qps_batch": 9527.69319260648, + "qps_batch_min": 9055.411880325639, + "qps_batch_max": 9552.476052423246, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9104, + "qps_1t_runs": [ + 1054.292812429506, + 1051.372476757281, + 1046.1137394121267 + ], + "qps_1t": 1051.372476757281, + "qps_1t_min": 1046.1137394121267, + "qps_1t_max": 1054.292812429506, + "median_ms": 0.9253000025637448, + "p95_ms": 1.316285011125728, + "p99_ms": 1.6610029590083286, + "mean_ms": 0.9518532666843385, + "knob": 96, + "batch": { + "recall": 0.9104, + "qps_batch_runs": [ + 7263.31401618982, + 7707.4736267445705, + 7888.133644286603 + ], + "qps_batch": 7707.4736267445705, + "qps_batch_min": 7263.31401618982, + "qps_batch_max": 7888.133644286603, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.931, + "qps_1t_runs": [ + 810.1137838993487, + 803.1769182138609, + 791.272016572566 + ], + "qps_1t": 803.1769182138609, + "qps_1t_min": 791.272016572566, + "qps_1t_max": 810.1137838993487, + "median_ms": 1.20749999769032, + "p95_ms": 1.7414149886462837, + "p99_ms": 2.2719680116279033, + "mean_ms": 1.2477460335358046, + "knob": 128, + "batch": { + "recall": 0.931, + "qps_batch_runs": [ + 5892.741498786665, + 6244.914296544523, + 6008.452691257235 + ], + "qps_batch": 6008.452691257235, + "qps_batch_min": 5892.741498786665, + "qps_batch_max": 6244.914296544523, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.956, + "qps_1t_runs": [ + 567.7671216281551, + 561.1985312602646, + 521.278782309747 + ], + "qps_1t": 561.1985312602646, + "qps_1t_min": 521.278782309747, + "qps_1t_max": 567.7671216281551, + "median_ms": 1.7574999947100878, + "p95_ms": 2.4894300353480503, + "p99_ms": 3.167460020049474, + "mean_ms": 1.8205151343912196, + "knob": 192, + "batch": { + "recall": 0.956, + "qps_batch_runs": [ + 3665.749618291206, + 3973.1399843644836, + 4004.6742559520235 + ], + "qps_batch": 3973.1399843644836, + "qps_batch_min": 3665.749618291206, + "qps_batch_max": 4004.6742559520235, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9677, + "qps_1t_runs": [ + 433.5724701135933, + 432.0233320850564, + 428.6042779857604 + ], + "qps_1t": 432.0233320850564, + "qps_1t_min": 428.6042779857604, + "qps_1t_max": 433.5724701135933, + "median_ms": 2.301850006915629, + "p95_ms": 3.0100999865680933, + "p99_ms": 3.5518599790520953, + "mean_ms": 2.3180879335074374, + "knob": 256, + "batch": { + "recall": 0.9677, + "qps_batch_runs": [ + 3379.980443311785, + 3287.2245304121666, + 2943.731456414279 + ], + "qps_batch": 3287.2245304121666, + "qps_batch_min": 2943.731456414279, + "qps_batch_max": 3379.980443311785, + "threads": 32 + } + }, + { + "nq": 851, + "recall_queries": 1000, + "recall": 0.9818, + "qps_1t_runs": [ + 288.57094922989194, + 282.2658845813054, + 287.6903421044477 + ], + "qps_1t": 287.6903421044477, + "qps_1t_min": 282.2658845813054, + "qps_1t_max": 288.57094922989194, + "median_ms": 3.4918999881483614, + "p95_ms": 4.508839990012348, + "p99_ms": 5.491079990752041, + "mean_ms": 3.4946902859396065, + "knob": 400, + "batch": { + "recall": 0.9818, + "qps_batch_runs": [ + 2067.600188030822, + 2108.253771231737, + 2014.5267523306359 + ], + "qps_batch": 2067.600188030822, + "qps_batch_min": 2014.5267523306359, + "qps_batch_max": 2108.253771231737, + "threads": 32 + } + } + ], + "knob_name": "expansion_search", + "mem_after_sweep": { + "rss_mb": 1752.7578125, + "peak_mb": 1781.09375 + }, + "ref_knob": 192, + "ref_recall_pre": 0.956, + "mutation": { + "ref_knob": 192, + "recall_before": 0.956, + "insert_10k": { + "seconds": 2.6394385999883525, + "per_s": 3788.6844573857975, + "batch": 1000 + }, + "update_10k": { + "seconds": 2.9290166999562643, + "per_s": 3414.1150510167176, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.005225400032941252, + "per_s": 1913729.0804453953, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 1798.57421875, + "peak_mb": 1831.046875 + }, + "first_query_after_ms": 27.772400004323572, + "after": { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9562, + "qps_1t_runs": [ + 555.1835826707119, + 535.6785482331195, + 632.7026403340772 + ], + "qps_1t": 555.1835826707119, + "qps_1t_min": 535.6785482331195, + "qps_1t_max": 632.7026403340772, + "median_ms": 1.6880999901331961, + "p95_ms": 2.4401400092756362, + "p99_ms": 3.306578031042589, + "mean_ms": 1.749506166825692 + }, + "recall_after": 0.9562 + }, + "saveload": { + "save_s": 3.246664500038605, + "disk_mb": 1622.6366424560547, + "load_s": 1.2830799999646842, + "first_query_after_load_ms": 1.7958000535145402, + "time_to_ready_s": 1.2848758000181988 + }, + "mem_end": { + "rss_mb": 1763.57421875, + "peak_mb": 1831.078125 + }, + "status": "ok", + "finished": "2026-11-16 20:46:52" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/usearch_text384_1000000_r1.load.json b/benchmarks/rust-stage-c/raw/usearch_text384_1000000_r1.load.json new file mode 100644 index 0000000..ef368be --- /dev/null +++ b/benchmarks/rust-stage-c/raw/usearch_text384_1000000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 13.096875, "pin_cpu_pct": 0.0, "max_cpu_pct": 75.2, "noisy": false, "seconds": 272.9584047794342, "after": 73.1} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/usearch_text384_1000000_r1.txt b/benchmarks/rust-stage-c/raw/usearch_text384_1000000_r1.txt new file mode 100644 index 0000000..8633956 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/usearch_text384_1000000_r1.txt @@ -0,0 +1,10 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system usearch --ds text384 --n 1000000 --exp core,mutation,saveload --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\usearch_text384_1000000_r1.json + usearch text384 1000000 knob=24 recall=0.7263 qps1t=2924 med=0.316ms batch=21558 + usearch text384 1000000 knob=32 recall=0.7728 qps1t=2408 med=0.384ms batch=14780 + usearch text384 1000000 knob=48 recall=0.8304 qps1t=1762 med=0.539ms batch=12523 + usearch text384 1000000 knob=64 recall=0.8664 qps1t=1409 med=0.666ms batch=9528 + usearch text384 1000000 knob=96 recall=0.9104 qps1t=1051 med=0.925ms batch=7707 + usearch text384 1000000 knob=128 recall=0.9310 qps1t=803 med=1.207ms batch=6008 + usearch text384 1000000 knob=192 recall=0.9560 qps1t=561 med=1.757ms batch=3973 + usearch text384 1000000 knob=256 recall=0.9677 qps1t=432 med=2.302ms batch=3287 + usearch text384 1000000 knob=400 recall=0.9818 qps1t=288 med=3.492ms batch=2068 diff --git a/benchmarks/rust-stage-c/raw/usearch_text384_100000_r1.json b/benchmarks/rust-stage-c/raw/usearch_text384_100000_r1.json new file mode 100644 index 0000000..e9bd300 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/usearch_text384_100000_r1.json @@ -0,0 +1,369 @@ +{ + "system": "usearch", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 20:52:19", + "mem_baseline": { + "rss_mb": 36.98828125, + "peak_mb": 36.98828125 + }, + "version": "usearch 2.26.0", + "build": { + "ingest_s": 6.106592199997976, + "index_s": 6.999995093792677e-06, + "total_s": 6.10659919999307, + "first_query_ms": 2.4688999983482063, + "ready_s": 6.109068099991418 + }, + "mem_after_build": { + "rss_mb": 219.61328125, + "peak_mb": 250.0078125 + }, + "mem_after_first_query": { + "rss_mb": 219.703125, + "peak_mb": 250.0078125 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.7812, + "qps_1t_runs": [ + 6607.887291329884, + 6874.723299133796, + 5825.866038748371 + ], + "qps_1t": 6607.887291329884, + "qps_1t_min": 5825.866038748371, + "qps_1t_max": 6874.723299133796, + "median_ms": 0.145500001963228, + "p95_ms": 0.21341999527066935, + "p99_ms": 0.29471003159414955, + "mean_ms": 0.15614766664414978, + "knob": 24, + "batch": { + "recall": 0.7812, + "qps_batch_runs": [ + 43757.73958877585, + 39774.8742202054, + 42870.432752565306 + ], + "qps_batch": 42870.432752565306, + "qps_batch_min": 39774.8742202054, + "qps_batch_max": 43757.73958877585, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8274, + "qps_1t_runs": [ + 4741.239727139687, + 4881.129849946016, + 4949.5933490458265 + ], + "qps_1t": 4881.129849946016, + "qps_1t_min": 4741.239727139687, + "qps_1t_max": 4949.5933490458265, + "median_ms": 0.19395002163946629, + "p95_ms": 0.29387497052084616, + "p99_ms": 0.4159249662188807, + "mean_ms": 0.20594089986601224, + "knob": 32, + "batch": { + "recall": 0.8274, + "qps_batch_runs": [ + 33557.83523640003, + 44794.438280193775, + 37245.61244645285 + ], + "qps_batch": 37245.61244645285, + "qps_batch_min": 33557.83523640003, + "qps_batch_max": 44794.438280193775, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8781, + "qps_1t_runs": [ + 3805.61670510333, + 4313.615373388291, + 3984.0431068509415 + ], + "qps_1t": 3984.0431068509415, + "qps_1t_min": 3805.61670510333, + "qps_1t_max": 4313.615373388291, + "median_ms": 0.23800000781193376, + "p95_ms": 0.33721000654622907, + "p99_ms": 0.4619460244430218, + "mean_ms": 0.24853163338654363, + "knob": 48, + "batch": { + "recall": 0.8781, + "qps_batch_runs": [ + 28619.099818008293, + 30508.081595526146, + 24767.372461808667 + ], + "qps_batch": 28619.099818008293, + "qps_batch_min": 24767.372461808667, + "qps_batch_max": 30508.081595526146, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9147, + "qps_1t_runs": [ + 3200.6216934915888, + 3061.7912290636473, + 2953.3564575398145 + ], + "qps_1t": 3061.7912290636473, + "qps_1t_min": 2953.3564575398145, + "qps_1t_max": 3200.6216934915888, + "median_ms": 0.31269999453797936, + "p95_ms": 0.44092503667343397, + "p99_ms": 0.6278370070504016, + "mean_ms": 0.32588110010450083, + "knob": 64, + "batch": { + "recall": 0.9147, + "qps_batch_runs": [ + 25865.39134722419, + 23809.07029099465, + 26041.05631156196 + ], + "qps_batch": 25865.39134722419, + "qps_batch_min": 23809.07029099465, + "qps_batch_max": 26041.05631156196, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9475, + "qps_1t_runs": [ + 2238.296675174954, + 2145.1963275890166, + 2002.644693603007 + ], + "qps_1t": 2145.1963275890166, + "qps_1t_min": 2002.644693603007, + "qps_1t_max": 2238.296675174954, + "median_ms": 0.44475001050159335, + "p95_ms": 0.6195200025103985, + "p99_ms": 0.9771079896017907, + "mean_ms": 0.4707552664137135, + "knob": 96, + "batch": { + "recall": 0.9475, + "qps_batch_runs": [ + 20332.8900832116, + 20682.822712347508, + 21562.36591264411 + ], + "qps_batch": 20682.822712347508, + "qps_batch_min": 20332.8900832116, + "qps_batch_max": 21562.36591264411, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9626, + "qps_1t_runs": [ + 1797.8837142630448, + 1729.6351886078144, + 1697.4424307951804 + ], + "qps_1t": 1729.6351886078144, + "qps_1t_min": 1697.4424307951804, + "qps_1t_max": 1797.8837142630448, + "median_ms": 0.5539499979931861, + "p95_ms": 0.7211850170278917, + "p99_ms": 1.3385120057500897, + "mean_ms": 0.5744958995492198, + "knob": 128, + "batch": { + "recall": 0.9626, + "qps_batch_runs": [ + 15407.584532535635, + 16567.34543057706, + 14949.381399850738 + ], + "qps_batch": 15407.584532535635, + "qps_batch_min": 14949.381399850738, + "qps_batch_max": 16567.34543057706, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9781, + "qps_1t_runs": [ + 1066.2900810926733, + 914.8626937480899, + 965.69503593092 + ], + "qps_1t": 965.69503593092, + "qps_1t_min": 914.8626937480899, + "qps_1t_max": 1066.2900810926733, + "median_ms": 0.9913000103551894, + "p95_ms": 1.2991200521355495, + "p99_ms": 2.294763948884777, + "mean_ms": 1.0221383004294087, + "knob": 192, + "batch": { + "recall": 0.9781, + "qps_batch_runs": [ + 6012.856690073487, + 7706.00066319477, + 7086.509270967886 + ], + "qps_batch": 7086.509270967886, + "qps_batch_min": 6012.856690073487, + "qps_batch_max": 7706.00066319477, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9854, + "qps_1t_runs": [ + 745.2865471092481, + 744.3153289730236, + 869.4763039538944 + ], + "qps_1t": 745.2865471092481, + "qps_1t_min": 744.3153289730236, + "qps_1t_max": 869.4763039538944, + "median_ms": 1.261600002180785, + "p95_ms": 1.6829000087454915, + "p99_ms": 2.1001879818504583, + "mean_ms": 1.2784666665829718, + "knob": 256, + "batch": { + "recall": 0.9854, + "qps_batch_runs": [ + 9140.492107262766, + 6843.882150275613, + 6988.13973005517 + ], + "qps_batch": 6988.13973005517, + "qps_batch_min": 6843.882150275613, + "qps_batch_max": 9140.492107262766, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.992, + "qps_1t_runs": [ + 676.5383071754719, + 663.0732960062523, + 607.7825094439603 + ], + "qps_1t": 663.0732960062523, + "qps_1t_min": 607.7825094439603, + "qps_1t_max": 676.5383071754719, + "median_ms": 1.4943999995011836, + "p95_ms": 2.0308999781263983, + "p99_ms": 2.6966649858513834, + "mean_ms": 1.5438557332187581, + "knob": 400, + "batch": { + "recall": 0.992, + "qps_batch_runs": [ + 5530.762376103017, + 5778.065653649155, + 5517.740639253556 + ], + "qps_batch": 5530.762376103017, + "qps_batch_min": 5517.740639253556, + "qps_batch_max": 5778.065653649155, + "threads": 32 + } + } + ], + "knob_name": "expansion_search", + "mem_after_sweep": { + "rss_mb": 221.83203125, + "peak_mb": 250.0078125 + }, + "ref_knob": 128, + "ref_recall_pre": 0.9626, + "mutation": { + "ref_knob": 128, + "recall_before": 0.9626, + "insert_10k": { + "seconds": 1.0880211999756284, + "per_s": 9190.997381506904, + "batch": 1000 + }, + "update_10k": { + "seconds": 1.0574874999583699, + "per_s": 9456.376553286605, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.003258500015363097, + "per_s": 3068896.7171558207, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 267.625, + "peak_mb": 271.71875 + }, + "first_query_after_ms": 3.66389995906502, + "after": { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9584, + "qps_1t_runs": [ + 1643.0753913841122, + 1687.8733903285686, + 1694.6527272816509 + ], + "qps_1t": 1687.8733903285686, + "qps_1t_min": 1643.0753913841122, + "qps_1t_max": 1694.6527272816509, + "median_ms": 0.5902000120840967, + "p95_ms": 0.7337049988564104, + "p99_ms": 0.8924139849841591, + "mean_ms": 0.5970558996583956 + }, + "recall_after": 0.9584 + }, + "saveload": { + "save_s": 0.15978739998536184, + "disk_mb": 176.69028091430664, + "load_s": 0.13478169997688383, + "first_query_after_load_ms": 0.9799000108614564, + "time_to_ready_s": 0.13576159998774529 + }, + "mem_end": { + "rss_mb": 232.23828125, + "peak_mb": 271.71875 + }, + "status": "ok", + "finished": "2026-11-16 20:52:50" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/usearch_text384_100000_r1.load.json b/benchmarks/rust-stage-c/raw/usearch_text384_100000_r1.load.json new file mode 100644 index 0000000..ea01761 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/usearch_text384_100000_r1.load.json @@ -0,0 +1 @@ +{"total_pct": 10.00625, "pin_cpu_pct": 0.0, "max_cpu_pct": 76.6, "noisy": false, "seconds": 30.838273763656616, "after": 25.6} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/usearch_text384_100000_r1.txt b/benchmarks/rust-stage-c/raw/usearch_text384_100000_r1.txt new file mode 100644 index 0000000..134ebe1 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/usearch_text384_100000_r1.txt @@ -0,0 +1,10 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system usearch --ds text384 --n 100000 --exp core,mutation,saveload --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\usearch_text384_100000_r1.json + usearch text384 100000 knob=24 recall=0.7812 qps1t=6608 med=0.146ms batch=42870 + usearch text384 100000 knob=32 recall=0.8274 qps1t=4881 med=0.194ms batch=37246 + usearch text384 100000 knob=48 recall=0.8781 qps1t=3984 med=0.238ms batch=28619 + usearch text384 100000 knob=64 recall=0.9147 qps1t=3062 med=0.313ms batch=25865 + usearch text384 100000 knob=96 recall=0.9475 qps1t=2145 med=0.445ms batch=20683 + usearch text384 100000 knob=128 recall=0.9626 qps1t=1730 med=0.554ms batch=15408 + usearch text384 100000 knob=192 recall=0.9781 qps1t=966 med=0.991ms batch=7087 + usearch text384 100000 knob=256 recall=0.9854 qps1t=745 med=1.262ms batch=6988 + usearch text384 100000 knob=400 recall=0.9920 qps1t=663 med=1.494ms batch=5531 diff --git a/benchmarks/rust-stage-c/raw/usearch_text384_100000_r2.json b/benchmarks/rust-stage-c/raw/usearch_text384_100000_r2.json new file mode 100644 index 0000000..9d0f0b3 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/usearch_text384_100000_r2.json @@ -0,0 +1,369 @@ +{ + "system": "usearch", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 20:58:26", + "mem_baseline": { + "rss_mb": 37.05078125, + "peak_mb": 37.05078125 + }, + "version": "usearch 2.26.0", + "build": { + "ingest_s": 8.21301859995583, + "index_s": 3.00002284348011e-06, + "total_s": 8.213021599978674, + "first_query_ms": 2.091800037305802, + "ready_s": 8.21511340001598 + }, + "mem_after_build": { + "rss_mb": 216.95703125, + "peak_mb": 247.3203125 + }, + "mem_after_first_query": { + "rss_mb": 217.05078125, + "peak_mb": 247.3203125 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.7851, + "qps_1t_runs": [ + 6647.5483052709205, + 6047.898150118792, + 6752.595698586662 + ], + "qps_1t": 6647.5483052709205, + "qps_1t_min": 6047.898150118792, + "qps_1t_max": 6752.595698586662, + "median_ms": 0.14949997421354055, + "p95_ms": 0.20501504477579138, + "p99_ms": 0.2405150263803076, + "mean_ms": 0.15462310002961507, + "knob": 24, + "batch": { + "recall": 0.7851, + "qps_batch_runs": [ + 50052.55524522929, + 47658.309004686336, + 51150.63358193882 + ], + "qps_batch": 50052.55524522929, + "qps_batch_min": 47658.309004686336, + "qps_batch_max": 51150.63358193882, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8313, + "qps_1t_runs": [ + 4526.55617262603, + 4932.817498496954, + 5202.101647869569 + ], + "qps_1t": 4932.817498496954, + "qps_1t_min": 4526.55617262603, + "qps_1t_max": 5202.101647869569, + "median_ms": 0.19180000526830554, + "p95_ms": 0.27242497890256334, + "p99_ms": 0.4333189810859011, + "mean_ms": 0.20529079994109148, + "knob": 32, + "batch": { + "recall": 0.8313, + "qps_batch_runs": [ + 37323.181431788755, + 29937.789295042607, + 46424.17760611787 + ], + "qps_batch": 37323.181431788755, + "qps_batch_min": 29937.789295042607, + "qps_batch_max": 46424.17760611787, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8831, + "qps_1t_runs": [ + 4168.158517438059, + 4025.4035263102373, + 3608.9708885062196 + ], + "qps_1t": 4025.4035263102373, + "qps_1t_min": 3608.9708885062196, + "qps_1t_max": 4168.158517438059, + "median_ms": 0.24209998082369566, + "p95_ms": 0.33929996425285935, + "p99_ms": 0.5337009893264621, + "mean_ms": 0.255141233249257, + "knob": 48, + "batch": { + "recall": 0.8831, + "qps_batch_runs": [ + 28774.825471231303, + 28504.808796201003, + 28387.235950636037 + ], + "qps_batch": 28504.808796201003, + "qps_batch_min": 28387.235950636037, + "qps_batch_max": 28774.825471231303, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9152, + "qps_1t_runs": [ + 3172.2869068244972, + 3179.863579333969, + 2993.46167808794 + ], + "qps_1t": 3172.2869068244972, + "qps_1t_min": 2993.46167808794, + "qps_1t_max": 3179.863579333969, + "median_ms": 0.3082000184804201, + "p95_ms": 0.41183499270118756, + "p99_ms": 0.7848419807851305, + "mean_ms": 0.32125676654201624, + "knob": 64, + "batch": { + "recall": 0.9152, + "qps_batch_runs": [ + 23779.346711095866, + 24888.189801967743, + 22803.977027417186 + ], + "qps_batch": 23779.346711095866, + "qps_batch_min": 22803.977027417186, + "qps_batch_max": 24888.189801967743, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9476, + "qps_1t_runs": [ + 2244.6966228158685, + 2331.927491409867, + 2186.0995988422737 + ], + "qps_1t": 2244.6966228158685, + "qps_1t_min": 2186.0995988422737, + "qps_1t_max": 2331.927491409867, + "median_ms": 0.42450003093108535, + "p95_ms": 0.560340014635585, + "p99_ms": 0.742226947913877, + "mean_ms": 0.44392000004881993, + "knob": 96, + "batch": { + "recall": 0.9476, + "qps_batch_runs": [ + 17700.24285402348, + 18467.152481597008, + 20370.000692751295 + ], + "qps_batch": 18467.152481597008, + "qps_batch_min": 17700.24285402348, + "qps_batch_max": 20370.000692751295, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9634, + "qps_1t_runs": [ + 1829.2066086074453, + 1815.826855995592, + 1769.686611556895 + ], + "qps_1t": 1815.826855995592, + "qps_1t_min": 1769.686611556895, + "qps_1t_max": 1829.2066086074453, + "median_ms": 0.5469000025186688, + "p95_ms": 0.6864200433483346, + "p99_ms": 0.8433890185551697, + "mean_ms": 0.5541567339096218, + "knob": 128, + "batch": { + "recall": 0.9634, + "qps_batch_runs": [ + 13656.2396701793, + 13595.379745153063, + 14654.65570355866 + ], + "qps_batch": 13656.2396701793, + "qps_batch_min": 13595.379745153063, + "qps_batch_max": 14654.65570355866, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9775, + "qps_1t_runs": [ + 1183.121261752277, + 1174.9005056687865, + 1196.1155902508408 + ], + "qps_1t": 1183.121261752277, + "qps_1t_min": 1174.9005056687865, + "qps_1t_max": 1196.1155902508408, + "median_ms": 0.8225000055972487, + "p95_ms": 1.0879800189286466, + "p99_ms": 1.3895160442916672, + "mean_ms": 0.8441324667461837, + "knob": 192, + "batch": { + "recall": 0.9775, + "qps_batch_runs": [ + 8659.635618065844, + 10505.68042185185, + 9779.214670702399 + ], + "qps_batch": 9779.214670702399, + "qps_batch_min": 8659.635618065844, + "qps_batch_max": 10505.68042185185, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9855, + "qps_1t_runs": [ + 837.5629558757945, + 937.3861664225809, + 949.7025200789334 + ], + "qps_1t": 937.3861664225809, + "qps_1t_min": 837.5629558757945, + "qps_1t_max": 949.7025200789334, + "median_ms": 1.059149974025786, + "p95_ms": 1.4945900271413843, + "p99_ms": 2.1638800023356386, + "mean_ms": 1.1045658667959894, + "knob": 256, + "batch": { + "recall": 0.9855, + "qps_batch_runs": [ + 6725.827041845155, + 7403.639776918154, + 7370.321043740647 + ], + "qps_batch": 7370.321043740647, + "qps_batch_min": 6725.827041845155, + "qps_batch_max": 7403.639776918154, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9918, + "qps_1t_runs": [ + 617.4202028046799, + 558.06884959913, + 602.9222312447745 + ], + "qps_1t": 602.9222312447745, + "qps_1t_min": 558.06884959913, + "qps_1t_max": 617.4202028046799, + "median_ms": 1.636000000871718, + "p95_ms": 2.3044099711114536, + "p99_ms": 3.1154029734898336, + "mean_ms": 1.6900415997370146, + "knob": 400, + "batch": { + "recall": 0.9918, + "qps_batch_runs": [ + 4693.368176231501, + 4873.415470187277, + 4972.521843168958 + ], + "qps_batch": 4873.415470187277, + "qps_batch_min": 4693.368176231501, + "qps_batch_max": 4972.521843168958, + "threads": 32 + } + } + ], + "knob_name": "expansion_search", + "mem_after_sweep": { + "rss_mb": 220.12109375, + "peak_mb": 247.3203125 + }, + "ref_knob": 128, + "ref_recall_pre": 0.9634, + "mutation": { + "ref_knob": 128, + "recall_before": 0.9634, + "insert_10k": { + "seconds": 1.4394873999990523, + "per_s": 6946.917354057135, + "batch": 1000 + }, + "update_10k": { + "seconds": 1.662352400016971, + "per_s": 6015.571668136016, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.005570899986196309, + "per_s": 1795042.0981848903, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 266.17578125, + "peak_mb": 270.203125 + }, + "first_query_after_ms": 6.5734999952837825, + "after": { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9568, + "qps_1t_runs": [ + 1095.7387500273028, + 1462.0199384172429, + 1340.3850093170106 + ], + "qps_1t": 1340.3850093170106, + "qps_1t_min": 1095.7387500273028, + "qps_1t_max": 1462.0199384172429, + "median_ms": 0.6793999928049743, + "p95_ms": 1.2473200098611414, + "p99_ms": 2.8570339846191897, + "mean_ms": 0.7808885996734413 + }, + "recall_after": 0.9568 + }, + "saveload": { + "save_s": 0.3307234999956563, + "disk_mb": 176.69028091430664, + "load_s": 0.23086139996303245, + "first_query_after_load_ms": 1.0514999739825726, + "time_to_ready_s": 0.23191289993701503 + }, + "mem_end": { + "rss_mb": 230.97265625, + "peak_mb": 270.2421875 + }, + "status": "ok", + "finished": "2026-11-16 20:59:00" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/usearch_text384_100000_r2.load.json b/benchmarks/rust-stage-c/raw/usearch_text384_100000_r2.load.json new file mode 100644 index 0000000..23484bd --- /dev/null +++ b/benchmarks/rust-stage-c/raw/usearch_text384_100000_r2.load.json @@ -0,0 +1 @@ +{"total_pct": 23.153125, "pin_cpu_pct": 1.2, "max_cpu_pct": 72.8, "noisy": false, "seconds": 33.99876093864441, "after": 34.8} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/usearch_text384_100000_r2.txt b/benchmarks/rust-stage-c/raw/usearch_text384_100000_r2.txt new file mode 100644 index 0000000..f469dd5 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/usearch_text384_100000_r2.txt @@ -0,0 +1,10 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system usearch --ds text384 --n 100000 --exp core,mutation,saveload --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\usearch_text384_100000_r2.json + usearch text384 100000 knob=24 recall=0.7851 qps1t=6648 med=0.149ms batch=50053 + usearch text384 100000 knob=32 recall=0.8313 qps1t=4933 med=0.192ms batch=37323 + usearch text384 100000 knob=48 recall=0.8831 qps1t=4025 med=0.242ms batch=28505 + usearch text384 100000 knob=64 recall=0.9152 qps1t=3172 med=0.308ms batch=23779 + usearch text384 100000 knob=96 recall=0.9476 qps1t=2245 med=0.425ms batch=18467 + usearch text384 100000 knob=128 recall=0.9634 qps1t=1816 med=0.547ms batch=13656 + usearch text384 100000 knob=192 recall=0.9775 qps1t=1183 med=0.823ms batch=9779 + usearch text384 100000 knob=256 recall=0.9855 qps1t=937 med=1.059ms batch=7370 + usearch text384 100000 knob=400 recall=0.9918 qps1t=603 med=1.636ms batch=4873 diff --git a/benchmarks/rust-stage-c/raw/usearch_text384_100000_r3.json b/benchmarks/rust-stage-c/raw/usearch_text384_100000_r3.json new file mode 100644 index 0000000..201110a --- /dev/null +++ b/benchmarks/rust-stage-c/raw/usearch_text384_100000_r3.json @@ -0,0 +1,369 @@ +{ + "system": "usearch", + "ds": "text384", + "n": 100000, + "d": 384, + "k": 10, + "queries": 1000, + "all_threads": 32, + "exps": [ + "core", + "mutation", + "saveload" + ], + "started": "2026-11-16 21:03:37", + "mem_baseline": { + "rss_mb": 37.06640625, + "peak_mb": 37.06640625 + }, + "version": "usearch 2.26.0", + "build": { + "ingest_s": 6.788176600006409, + "index_s": 2.900022082030773e-06, + "total_s": 6.788179500028491, + "first_query_ms": 2.1531000384129584, + "ready_s": 6.790332600066904 + }, + "mem_after_build": { + "rss_mb": 218.56640625, + "peak_mb": 248.5625 + }, + "mem_after_first_query": { + "rss_mb": 218.609375, + "peak_mb": 248.5625 + }, + "sweep": [ + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.7801, + "qps_1t_runs": [ + 6921.928965135734, + 6069.147017962103, + 6500.50505733212 + ], + "qps_1t": 6500.50505733212, + "qps_1t_min": 6069.147017962103, + "qps_1t_max": 6921.928965135734, + "median_ms": 0.14720001490786672, + "p95_ms": 0.21071002993267024, + "p99_ms": 0.2597019984386861, + "mean_ms": 0.15435680001974106, + "knob": 24, + "batch": { + "recall": 0.7801, + "qps_batch_runs": [ + 48816.20687449892, + 46644.401797404396, + 43763.1015579286 + ], + "qps_batch": 46644.401797404396, + "qps_batch_min": 43763.1015579286, + "qps_batch_max": 48816.20687449892, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8275, + "qps_1t_runs": [ + 5634.285304300022, + 5255.073390023193, + 5342.311279147823 + ], + "qps_1t": 5342.311279147823, + "qps_1t_min": 5255.073390023193, + "qps_1t_max": 5634.285304300022, + "median_ms": 0.1770000089891255, + "p95_ms": 0.24320998927578324, + "p99_ms": 0.3076340135885395, + "mean_ms": 0.1849873330211267, + "knob": 32, + "batch": { + "recall": 0.8275, + "qps_batch_runs": [ + 35695.54549568839, + 41090.20536414377, + 40580.133623697184 + ], + "qps_batch": 40580.133623697184, + "qps_batch_min": 35695.54549568839, + "qps_batch_max": 41090.20536414377, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.8822, + "qps_1t_runs": [ + 3205.268951821386, + 3999.824001747423, + 4110.677530441585 + ], + "qps_1t": 3999.824001747423, + "qps_1t_min": 3205.268951821386, + "qps_1t_max": 4110.677530441585, + "median_ms": 0.2466500154696405, + "p95_ms": 0.37560000782832503, + "p99_ms": 0.5130080063827334, + "mean_ms": 0.2684220666221033, + "knob": 48, + "batch": { + "recall": 0.8822, + "qps_batch_runs": [ + 29901.802488287172, + 26800.0235760885, + 28265.275300802114 + ], + "qps_batch": 28265.275300802114, + "qps_batch_min": 26800.0235760885, + "qps_batch_max": 29901.802488287172, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9155, + "qps_1t_runs": [ + 3211.6719937466078, + 3098.6268134301176, + 3141.462242282425 + ], + "qps_1t": 3141.462242282425, + "qps_1t_min": 3098.6268134301176, + "qps_1t_max": 3211.6719937466078, + "median_ms": 0.30664997757412493, + "p95_ms": 0.41700497386045754, + "p99_ms": 0.5547410325380032, + "mean_ms": 0.31747033313149586, + "knob": 64, + "batch": { + "recall": 0.9155, + "qps_batch_runs": [ + 20548.648922384884, + 23591.30424220951, + 21437.46797207915 + ], + "qps_batch": 21437.46797207915, + "qps_batch_min": 20548.648922384884, + "qps_batch_max": 23591.30424220951, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.947, + "qps_1t_runs": [ + 2263.6556741977083, + 2064.577503669301, + 2004.7866288276186 + ], + "qps_1t": 2064.577503669301, + "qps_1t_min": 2004.7866288276186, + "qps_1t_max": 2263.6556741977083, + "median_ms": 0.4463499935809523, + "p95_ms": 0.619004998588934, + "p99_ms": 1.0020590032217962, + "mean_ms": 0.47497670012914267, + "knob": 96, + "batch": { + "recall": 0.947, + "qps_batch_runs": [ + 16084.92842461784, + 15976.354993115134, + 17444.761166073782 + ], + "qps_batch": 16084.92842461784, + "qps_batch_min": 15976.354993115134, + "qps_batch_max": 17444.761166073782, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9643, + "qps_1t_runs": [ + 1672.902409907848, + 1859.0726269895745, + 1780.4311740769942 + ], + "qps_1t": 1780.4311740769942, + "qps_1t_min": 1672.902409907848, + "qps_1t_max": 1859.0726269895745, + "median_ms": 0.5460999673232436, + "p95_ms": 0.692599979811348, + "p99_ms": 0.887464004918, + "mean_ms": 0.5657759334620399, + "knob": 128, + "batch": { + "recall": 0.9643, + "qps_batch_runs": [ + 12157.86255234143, + 13727.325893789439, + 13468.829755453628 + ], + "qps_batch": 13468.829755453628, + "qps_batch_min": 12157.86255234143, + "qps_batch_max": 13727.325893789439, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.979, + "qps_1t_runs": [ + 1272.3998026788836, + 1255.00684870086, + 1252.1613866949524 + ], + "qps_1t": 1255.00684870086, + "qps_1t_min": 1252.1613866949524, + "qps_1t_max": 1272.3998026788836, + "median_ms": 0.7817000150680542, + "p95_ms": 0.9923399891704319, + "p99_ms": 1.2210739840520528, + "mean_ms": 0.7937813337775879, + "knob": 192, + "batch": { + "recall": 0.979, + "qps_batch_runs": [ + 9600.09830754337, + 10217.646079442486, + 9668.271919672316 + ], + "qps_batch": 9668.271919672316, + "qps_batch_min": 9600.09830754337, + "qps_batch_max": 10217.646079442486, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9863, + "qps_1t_runs": [ + 956.6206966323663, + 852.5739382119261, + 893.4389946075698 + ], + "qps_1t": 893.4389946075698, + "qps_1t_min": 852.5739382119261, + "qps_1t_max": 956.6206966323663, + "median_ms": 1.0564500116743147, + "p95_ms": 1.5494349820073692, + "p99_ms": 2.276005984749645, + "mean_ms": 1.1125119328110789, + "knob": 256, + "batch": { + "recall": 0.9863, + "qps_batch_runs": [ + 7091.524632879479, + 7357.718281928675, + 6952.019248810059 + ], + "qps_batch": 7091.524632879479, + "qps_batch_min": 6952.019248810059, + "qps_batch_max": 7357.718281928675, + "threads": 32 + } + }, + { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9927, + "qps_1t_runs": [ + 662.9020454270466, + 614.0943372106852, + 616.597586314985 + ], + "qps_1t": 616.597586314985, + "qps_1t_min": 614.0943372106852, + "qps_1t_max": 662.9020454270466, + "median_ms": 1.5398499963339418, + "p95_ms": 2.1161999902687962, + "p99_ms": 2.7727659809170206, + "mean_ms": 1.5862453667796217, + "knob": 400, + "batch": { + "recall": 0.9927, + "qps_batch_runs": [ + 5098.479684577891, + 5030.882070177561, + 5223.909835665954 + ], + "qps_batch": 5098.479684577891, + "qps_batch_min": 5030.882070177561, + "qps_batch_max": 5223.909835665954, + "threads": 32 + } + } + ], + "knob_name": "expansion_search", + "mem_after_sweep": { + "rss_mb": 221.203125, + "peak_mb": 248.5625 + }, + "ref_knob": 128, + "ref_recall_pre": 0.9643, + "mutation": { + "ref_knob": 128, + "recall_before": 0.9643, + "insert_10k": { + "seconds": 1.2999302000389434, + "per_s": 7692.72073200578, + "batch": 1000 + }, + "update_10k": { + "seconds": 1.3017172000254504, + "per_s": 7682.160149535157, + "batch": 1000 + }, + "delete_10k": { + "seconds": 0.0030513000092469156, + "per_s": 3277291.636251814, + "batch": 1000 + }, + "mem_after": { + "rss_mb": 267.12890625, + "peak_mb": 271.12109375 + }, + "first_query_after_ms": 3.566899977158755, + "after": { + "nq": 1000, + "recall_queries": 1000, + "recall": 0.9573, + "qps_1t_runs": [ + 1494.113047102676, + 1465.7678006691626, + 1530.133141718449 + ], + "qps_1t": 1494.113047102676, + "qps_1t_min": 1465.7678006691626, + "qps_1t_max": 1530.133141718449, + "median_ms": 0.6423500017262995, + "p95_ms": 0.8852099912473932, + "p99_ms": 1.2394460063660482, + "mean_ms": 0.6683558661219043 + }, + "recall_after": 0.9573 + }, + "saveload": { + "save_s": 0.18174650002038106, + "disk_mb": 176.69028091430664, + "load_s": 0.13864130002912134, + "first_query_after_load_ms": 0.957000011112541, + "time_to_ready_s": 0.13959830004023388 + }, + "mem_end": { + "rss_mb": 231.55078125, + "peak_mb": 271.1953125 + }, + "status": "ok", + "finished": "2026-11-16 21:04:08" +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/usearch_text384_100000_r3.load.json b/benchmarks/rust-stage-c/raw/usearch_text384_100000_r3.load.json new file mode 100644 index 0000000..6d88785 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/usearch_text384_100000_r3.load.json @@ -0,0 +1 @@ +{"total_pct": 11.293750000000001, "pin_cpu_pct": 61.6, "max_cpu_pct": 84.0, "noisy": true, "seconds": 31.060250520706177, "after": 28.0} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/raw/usearch_text384_100000_r3.txt b/benchmarks/rust-stage-c/raw/usearch_text384_100000_r3.txt new file mode 100644 index 0000000..ad9c4f8 --- /dev/null +++ b/benchmarks/rust-stage-c/raw/usearch_text384_100000_r3.txt @@ -0,0 +1,10 @@ +C:\Program Files\Python312\python.exe D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\bench_c.py --system usearch --ds text384 --n 100000 --exp core,mutation,saveload --queries 1000 --budget 3 --out D:\Goldmine\ISON\ison_dev\isongraphossworking\isongraph-vector\benchmarks\rust-stage-c\raw\usearch_text384_100000_r3.json + usearch text384 100000 knob=24 recall=0.7801 qps1t=6501 med=0.147ms batch=46644 + usearch text384 100000 knob=32 recall=0.8275 qps1t=5342 med=0.177ms batch=40580 + usearch text384 100000 knob=48 recall=0.8822 qps1t=4000 med=0.247ms batch=28265 + usearch text384 100000 knob=64 recall=0.9155 qps1t=3141 med=0.307ms batch=21437 + usearch text384 100000 knob=96 recall=0.9470 qps1t=2065 med=0.446ms batch=16085 + usearch text384 100000 knob=128 recall=0.9643 qps1t=1780 med=0.546ms batch=13469 + usearch text384 100000 knob=192 recall=0.9790 qps1t=1255 med=0.782ms batch=9668 + usearch text384 100000 knob=256 recall=0.9863 qps1t=893 med=1.056ms batch=7092 + usearch text384 100000 knob=400 recall=0.9927 qps1t=617 med=1.540ms batch=5098 diff --git a/benchmarks/rust-stage-c/rerun_noisy.py b/benchmarks/rust-stage-c/rerun_noisy.py new file mode 100644 index 0000000..8d7982b --- /dev/null +++ b/benchmarks/rust-stage-c/rerun_noisy.py @@ -0,0 +1,25 @@ +"""Move every 100k job that started while the pinned CPU was busy (flagged noisy in raw/*.load.json) to +raw/discarded_noisy/ so that `run_stage_c.py --phase 1 --phase 2 --phase 4` runs it again. The first attempts are kept, not deleted. +Usage: python rerun_noisy.py && python -u run_stage_c.py --phase 1 --phase 2 --phase 4 >> run_rerun.log""" +import glob +import json +import os +import shutil + +RAW = os.path.join(os.path.dirname(os.path.abspath(__file__)), "raw") +DIS = os.path.join(RAW, "discarded_noisy") +os.makedirs(DIS, exist_ok=True) +moved = [] +for lp in sorted(glob.glob(os.path.join(RAW, "*.load.json"))): + base = os.path.basename(lp)[:-len(".load.json")] + if "_100000_" not in base: + continue + ld = json.load(open(lp)) + if not ld.get("noisy"): + continue + for ext in (".json", ".txt", ".load.json"): + p = os.path.join(RAW, base + ext) + if os.path.exists(p): + shutil.move(p, os.path.join(DIS, base + ext)) + moved.append(base) +print(len(moved), "moved:", " ".join(moved)) diff --git a/benchmarks/rust-stage-c/run_all.log b/benchmarks/rust-stage-c/run_all.log new file mode 100644 index 0000000..87396d7 --- /dev/null +++ b/benchmarks/rust-stage-c/run_all.log @@ -0,0 +1,106 @@ +skip rust_exact_text384_100000_r1 (done) +skip numpy_bf_text384_100000_r1 (done) +skip faiss_flat_text384_100000_r1 (done) +skip rust_ivf_text384_100000_r1 (done) +skip faiss_ivfflat_text384_100000_r1 (done) +skip rust_hnsw_text384_100000_r1 (done) +skip faiss_hnsw_text384_100000_r1 (done) +skip hnswlib_text384_100000_r1 (done) +skip usearch_text384_100000_r1 (done) +skip rust_ivf_i8x4_text384_100000_r1 (done) +skip rust_hnsw_i8x4_text384_100000_r1 (done) +skip rust_exact_text384_100000_r2 (done) +skip numpy_bf_text384_100000_r2 (done) +skip faiss_flat_text384_100000_r2 (done) +== rust_ivf_text384_100000_r2 load total 13% pin-cpu 53% noisy=True + done in 28s ok=True +== faiss_ivfflat_text384_100000_r2 load total 13% pin-cpu 33% noisy=True + done in 35s ok=True +== rust_hnsw_text384_100000_r2 load total 21% pin-cpu 0% noisy=False + done in 20s ok=True +== faiss_hnsw_text384_100000_r2 load total 14% pin-cpu 28% noisy=True + done in 33s ok=True +== hnswlib_text384_100000_r2 load total 21% pin-cpu 0% noisy=False + done in 36s ok=True +== usearch_text384_100000_r2 load total 20% pin-cpu 24% noisy=True + done in 41s ok=True +== rust_exact_text384_100000_r3 load total 17% pin-cpu 30% noisy=True + done in 22s ok=True +== numpy_bf_text384_100000_r3 load total 15% pin-cpu 11% noisy=False + done in 32s ok=True +== faiss_flat_text384_100000_r3 load total 15% pin-cpu 32% noisy=True + done in 29s ok=True +== rust_ivf_text384_100000_r3 load total 17% pin-cpu 0% noisy=False + done in 28s ok=True +== faiss_ivfflat_text384_100000_r3 load total 14% pin-cpu 27% noisy=True + done in 35s ok=True +== rust_hnsw_text384_100000_r3 load total 9% pin-cpu 0% noisy=False + done in 19s ok=True +== faiss_hnsw_text384_100000_r3 load total 13% pin-cpu 42% noisy=True + done in 33s ok=True +== hnswlib_text384_100000_r3 load total 11% pin-cpu 4% noisy=False + done in 35s ok=True +== usearch_text384_100000_r3 load total 17% pin-cpu 27% noisy=True + done in 42s ok=True +== rust_exact_glove_100000_r1 load total 21% pin-cpu 65% noisy=True + done in 43s ok=True +== numpy_bf_glove_100000_r1 load total 14% pin-cpu 1% noisy=False + done in 21s ok=True +== faiss_flat_glove_100000_r1 load total 14% pin-cpu 26% noisy=True + done in 15s ok=True +== rust_exact_glove_100000_r2 load total 13% pin-cpu 0% noisy=False + done in 11s ok=True +== numpy_bf_glove_100000_r2 load total 12% pin-cpu 35% noisy=True + done in 19s ok=True +== faiss_flat_glove_100000_r2 load total 11% pin-cpu 0% noisy=False + done in 17s ok=True +== rust_exact_glove_100000_r3 load total 17% pin-cpu 37% noisy=True + done in 12s ok=True +== numpy_bf_glove_100000_r3 load total 25% pin-cpu 7% noisy=False + done in 20s ok=True +== faiss_flat_glove_100000_r3 load total 27% pin-cpu 73% noisy=True + done in 18s ok=True +== rust_exact_sift_100000_r1 load total 18% pin-cpu 0% noisy=False + done in 46s ok=True +== numpy_bf_sift_100000_r1 load total 10% pin-cpu 34% noisy=True + done in 22s ok=True +== faiss_flat_sift_100000_r1 load total 16% pin-cpu 0% noisy=False + done in 18s ok=True +== rust_exact_sift_100000_r2 load total 10% pin-cpu 26% noisy=True + done in 13s ok=True +== numpy_bf_sift_100000_r2 load total 11% pin-cpu 0% noisy=False + done in 22s ok=True +== faiss_flat_sift_100000_r2 load total 13% pin-cpu 38% noisy=True + done in 18s ok=True +== rust_exact_sift_100000_r3 load total 12% pin-cpu 0% noisy=False + done in 13s ok=True +== numpy_bf_sift_100000_r3 load total 9% pin-cpu 26% noisy=True + done in 25s ok=True +== faiss_flat_sift_100000_r3 load total 11% pin-cpu 0% noisy=False + done in 17s ok=True +== rust_exact_text384_100000_persist_r1 load total 10% pin-cpu 23% noisy=True + done in 24s ok=True +== rust_exact_text384_1000000_r1 load total 11% pin-cpu 0% noisy=False + done in 242s ok=True +== numpy_bf_text384_1000000_r1 load total 15% pin-cpu 0% noisy=False + done in 76s ok=True +== faiss_flat_text384_1000000_r1 load total 10% pin-cpu 4% noisy=False + done in 38s ok=True +== rust_hnsw_text384_1000000_r1 load total 17% pin-cpu 62% noisy=True + done in 228s ok=True +== hnswlib_text384_1000000_r1 load total 8% pin-cpu 0% noisy=False + done in 195s ok=True +== rust_exact_text384_1000000_persist_r1 load total 15% pin-cpu 0% noisy=False + done in 185s ok=True +== rust_ivf_text384_1000000_r1 load total 17% pin-cpu 61% noisy=True + done in 99s ok=True +== faiss_ivfflat_text384_1000000_r1 load total 9% pin-cpu 0% noisy=False + done in 129s ok=True +== faiss_hnsw_text384_1000000_r1 load total 13% pin-cpu 71% noisy=True + done in 251s ok=True +== usearch_text384_1000000_r1 load total 13% pin-cpu 0% noisy=False + done in 273s ok=True +== rust_ivf_i8x4_text384_1000000_r1 load total 11% pin-cpu 73% noisy=True + done in 41s ok=True +== rust_hnsw_i8x4_text384_1000000_r1 load total 14% pin-cpu 5% noisy=False + done in 154s ok=True diff --git a/benchmarks/rust-stage-c/run_phase12.log b/benchmarks/rust-stage-c/run_phase12.log new file mode 100644 index 0000000..bac6e12 --- /dev/null +++ b/benchmarks/rust-stage-c/run_phase12.log @@ -0,0 +1,29 @@ +== rust_exact_text384_100000_r1 load total 17% pin-cpu 0% noisy=False + done in 115s ok=True +== numpy_bf_text384_100000_r1 load total 24% pin-cpu 1% noisy=False + done in 32s ok=True +== faiss_flat_text384_100000_r1 load total 17% pin-cpu 0% noisy=False + done in 27s ok=True +== rust_ivf_text384_100000_r1 load total 12% pin-cpu 2% noisy=False + done in 45s ok=True +== faiss_ivfflat_text384_100000_r1 load total 16% pin-cpu 0% noisy=False + done in 35s ok=True +== rust_hnsw_text384_100000_r1 load total 16% pin-cpu 0% noisy=False + done in 56s ok=True +== faiss_hnsw_text384_100000_r1 load total 13% pin-cpu 0% noisy=False + done in 32s ok=True +== hnswlib_text384_100000_r1 load total 15% pin-cpu 0% noisy=False + done in 35s ok=True +== usearch_text384_100000_r1 load total 11% pin-cpu 27% noisy=True + done in 39s ok=True +== rust_ivf_i8x4_text384_100000_r1 load total 10% pin-cpu 0% noisy=False + done in 17s ok=True +== rust_hnsw_i8x4_text384_100000_r1 load total 27% pin-cpu 71% noisy=True + done in 12s ok=True +== rust_exact_text384_100000_r2 load total 13% pin-cpu 69% noisy=True + done in 104s ok=True +== numpy_bf_text384_100000_r2 load total 9% pin-cpu 0% noisy=False + done in 27s ok=True +== faiss_flat_text384_100000_r2 load total 20% pin-cpu 0% noisy=False + done in 27s ok=True +== rust_ivf_text384_100000_r2 load total 12% pin-cpu 11% noisy=False diff --git a/benchmarks/rust-stage-c/run_rerun.log b/benchmarks/rust-stage-c/run_rerun.log new file mode 100644 index 0000000..ecd2afb --- /dev/null +++ b/benchmarks/rust-stage-c/run_rerun.log @@ -0,0 +1,70 @@ +skip rust_exact_text384_100000_r1 (done) +skip numpy_bf_text384_100000_r1 (done) +skip faiss_flat_text384_100000_r1 (done) +skip rust_ivf_text384_100000_r1 (done) +skip faiss_ivfflat_text384_100000_r1 (done) +skip rust_hnsw_text384_100000_r1 (done) +skip faiss_hnsw_text384_100000_r1 (done) +skip hnswlib_text384_100000_r1 (done) +== usearch_text384_100000_r1 load total 10% pin-cpu 0% noisy=False + done in 31s ok=True +skip rust_ivf_i8x4_text384_100000_r1 (done) +== rust_hnsw_i8x4_text384_100000_r1 load total 12% pin-cpu 75% noisy=True + done in 10s ok=True +== rust_exact_text384_100000_r2 load total 16% pin-cpu 0% noisy=False + done in 18s ok=True +skip numpy_bf_text384_100000_r2 (done) +skip faiss_flat_text384_100000_r2 (done) +== rust_ivf_text384_100000_r2 load total 16% pin-cpu 0% noisy=False + done in 29s ok=True +== faiss_ivfflat_text384_100000_r2 load total 19% pin-cpu 69% noisy=True + done in 34s ok=True +skip rust_hnsw_text384_100000_r2 (done) +== faiss_hnsw_text384_100000_r2 load total 13% pin-cpu 69% noisy=True + done in 23s ok=True +skip hnswlib_text384_100000_r2 (done) +== usearch_text384_100000_r2 load total 23% pin-cpu 1% noisy=False + done in 34s ok=True +== rust_exact_text384_100000_r3 load total 18% pin-cpu 83% noisy=True + done in 21s ok=True +skip numpy_bf_text384_100000_r3 (done) +== faiss_flat_text384_100000_r3 load total 13% pin-cpu 3% noisy=False + done in 25s ok=True +skip rust_ivf_text384_100000_r3 (done) +== faiss_ivfflat_text384_100000_r3 load total 21% pin-cpu 72% noisy=True + done in 32s ok=True +skip rust_hnsw_text384_100000_r3 (done) +== faiss_hnsw_text384_100000_r3 load total 13% pin-cpu 0% noisy=False + done in 26s ok=True +skip hnswlib_text384_100000_r3 (done) +== usearch_text384_100000_r3 load total 11% pin-cpu 62% noisy=True + done in 31s ok=True +== rust_exact_glove_100000_r1 load total 17% pin-cpu 0% noisy=False + done in 28s ok=True +skip numpy_bf_glove_100000_r1 (done) +== faiss_flat_glove_100000_r1 load total 19% pin-cpu 77% noisy=True + done in 12s ok=True +skip rust_exact_glove_100000_r2 (done) +== numpy_bf_glove_100000_r2 load total 22% pin-cpu 0% noisy=False + done in 13s ok=True +skip faiss_flat_glove_100000_r2 (done) +== rust_exact_glove_100000_r3 load total 15% pin-cpu 0% noisy=False + done in 8s ok=True +skip numpy_bf_glove_100000_r3 (done) +== faiss_flat_glove_100000_r3 load total 18% pin-cpu 95% noisy=True + done in 11s ok=True +skip rust_exact_sift_100000_r1 (done) +== numpy_bf_sift_100000_r1 load total 16% pin-cpu 0% noisy=False + done in 15s ok=True +skip faiss_flat_sift_100000_r1 (done) +== rust_exact_sift_100000_r2 load total 14% pin-cpu 0% noisy=False + done in 9s ok=True +skip numpy_bf_sift_100000_r2 (done) +== faiss_flat_sift_100000_r2 load total 17% pin-cpu 96% noisy=True + done in 13s ok=True +skip rust_exact_sift_100000_r3 (done) +== numpy_bf_sift_100000_r3 load total 15% pin-cpu 0% noisy=False + done in 15s ok=True +skip faiss_flat_sift_100000_r3 (done) +== rust_exact_text384_100000_persist_r1 load total 13% pin-cpu 0% noisy=False + done in 14s ok=True diff --git a/benchmarks/rust-stage-c/run_rerun1m.log b/benchmarks/rust-stage-c/run_rerun1m.log new file mode 100644 index 0000000..0f726bb --- /dev/null +++ b/benchmarks/rust-stage-c/run_rerun1m.log @@ -0,0 +1,16 @@ +skip rust_exact_text384_1000000_r1 (done) +skip numpy_bf_text384_1000000_r1 (done) +skip faiss_flat_text384_1000000_r1 (done) +== rust_hnsw_text384_1000000_r1 load total 18% pin-cpu 0% noisy=False + done in 228s ok=True +skip hnswlib_text384_1000000_r1 (done) +skip rust_exact_text384_1000000_persist_r1 (done) +== rust_ivf_text384_1000000_r1 load total 14% pin-cpu 0% noisy=False + done in 78s ok=True +skip faiss_ivfflat_text384_1000000_r1 (done) +== faiss_hnsw_text384_1000000_r1 load total 13% pin-cpu 0% noisy=False + done in 216s ok=True +skip usearch_text384_1000000_r1 (done) +== rust_ivf_i8x4_text384_1000000_r1 load total 19% pin-cpu 0% noisy=False + done in 42s ok=True +skip rust_hnsw_i8x4_text384_1000000_r1 (done) diff --git a/benchmarks/rust-stage-c/run_stage_c.py b/benchmarks/rust-stage-c/run_stage_c.py new file mode 100644 index 0000000..411f4e5 --- /dev/null +++ b/benchmarks/rust-stage-c/run_stage_c.py @@ -0,0 +1,149 @@ +"""Stage C driver: runs every cell sequentially (never two timed jobs at once), resumable, logs machine load. + + python run_stage_c.py --list + python run_stage_c.py --phase 1 # text384 100k, 3 interleaved rounds + python run_stage_c.py --phase 2 # glove / sift 100k exact (Q1, Q4) + python run_stage_c.py --phase 3 # text384 1M + python run_stage_c.py --phase 4 # persistence 100k / 1M (Rust) + +Before every job it samples total CPU load and the load on the pinned CPU; if the machine is busy it waits up to +WAIT_MAX seconds. The load it saw is stored in raw/<job>.load.json; a job started at high load is flagged noisy +and should be re-run (rm its json and run again: finished jobs are skipped). +""" +import argparse +import json +import os +import subprocess +import sys +import time + +import psutil + +HERE = os.path.dirname(os.path.abspath(__file__)) +RAW = os.path.join(HERE, "raw") +DATA = r"D:\Goldmine\ISON\bench_data" +PREP = DATA + r"\prep" +RS = DATA + r"\rs_target_stagec\release\examples\stage_c_bench.exe" +PY = sys.executable +PIN = int(os.environ.get("BENCH_PIN_CPU", "8")) +LOAD_OK = float(os.environ.get("STAGEC_LOAD_OK", "30")) # total % CPU considered quiet enough +WAIT_MAX = 45 + + +def measure_load(sec=4.0): + per = psutil.cpu_percent(interval=sec, percpu=True) + return {"total_pct": sum(per) / len(per), "pin_cpu_pct": per[PIN], "max_cpu_pct": max(per)} + + +def wait_quiet(): + t0 = time.time() + while True: + ld = measure_load() + if ld["total_pct"] < LOAD_OK and ld["pin_cpu_pct"] < 15: + ld["noisy"] = False + return ld + if time.time() - t0 > WAIT_MAX: + ld["noisy"] = True + return ld + time.sleep(5) + + +def rust(mode, ds, n, exp, rep, int8=0, extra=None, tag=None): + name = f"rust_{mode}{'_i8x%d' % int8 if int8 else ''}_{ds}_{n}{'_' + tag if tag else ''}_r{rep}" + cmd = [RS, "--ds-dir", f"{PREP}/{ds}", "--ds", ds, "--n", str(n), "--mode", mode, "--exp", exp, "--queries", "1000", + "--budget", "3", "--label", f"rust_{mode}{'_i8x%d' % int8 if int8 else ''}", "--out", os.path.join(RAW, name + ".json")] + if int8: + cmd += ["--int8", str(int8)] + cmd += extra or [] + return name, cmd + + +def py(system, ds, n, exp, rep): + name = f"{system}_{ds}_{n}_r{rep}" + cmd = [PY, os.path.join(HERE, "bench_c.py"), "--system", system, "--ds", ds, "--n", str(n), "--exp", exp, + "--queries", "1000", "--budget", "3", "--out", os.path.join(RAW, name + ".json")] + return name, cmd + + +def jobs(phase): + out = [] + if phase == 1: + ds, n = "text384", 100_000 + for r in (1, 2, 3): + first = r == 1 # int8 study, concurrent clients and 10k writes once; the timed sweeps three times + out += [rust("exact", ds, n, "core,int8,conc" if first else "core", r), + py("numpy_bf", ds, n, "core", r), + py("faiss_flat", ds, n, "core", r), + rust("ivf", ds, n, "core,conc,writes" if first else "core", r), + py("faiss_ivfflat", ds, n, "core,mutation,saveload", r), + rust("hnsw", ds, n, "core,conc,writes" if first else "core", r), + py("faiss_hnsw", ds, n, "core,mutation,saveload", r), + py("hnswlib", ds, n, "core,mutation,saveload", r), + py("usearch", ds, n, "core,mutation,saveload", r)] + if r == 1: + out += [rust("ivf", ds, n, "core", r, int8=4), + rust("hnsw", ds, n, "core", r, int8=4, extra=["--knobs", "64,96,128,192"])] + elif phase == 2: + for ds in ("glove", "sift"): + for r in (1, 2, 3): + out += [rust("exact", ds, 100_000, "core,int8" if r == 1 else "core", r), + py("numpy_bf", ds, 100_000, "core", r), + py("faiss_flat", ds, 100_000, "core", r)] + elif phase == 3: + ds, n = "text384", 1_000_000 + out += [rust("exact", ds, n, "core,int8,conc", 1), + py("numpy_bf", ds, n, "core", 1), + py("faiss_flat", ds, n, "core", 1), + rust("hnsw", ds, n, "core,conc,writes", 1, extra=["--knobs", "48,64,96,128,192,256,400,640"]), + py("hnswlib", ds, n, "core,mutation,saveload", 1), + rust("exact", ds, n, "persist", 1, extra=["--ready", "ivf"], tag="persist"), + rust("ivf", ds, n, "core,conc,writes", 1), + py("faiss_ivfflat", ds, n, "core,mutation,saveload", 1), + py("faiss_hnsw", ds, n, "core,mutation,saveload", 1), + py("usearch", ds, n, "core,mutation,saveload", 1), + rust("ivf", ds, n, "core", 1, int8=4, extra=["--knobs", "64,128,256,512"]), + rust("hnsw", ds, n, "core", 1, int8=4, extra=["--knobs", "192,256"])] + elif phase == 4: + out += [rust("exact", "text384", 100_000, "persist", 1, extra=["--ready", "ivf,hnsw"], tag="persist")] + return out + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--phase", type=int, action="append") + ap.add_argument("--list", action="store_true") + ap.add_argument("--only", default="") + a = ap.parse_args() + os.makedirs(RAW, exist_ok=True) + js = [] + for ph in a.phase or [1, 2, 3, 4]: + js += jobs(ph) + if a.only: + js = [j for j in js if a.only in j[0]] + if a.list: + for nm, cmd in js: + print(nm) + return + for nm, cmd in js: + outp = os.path.join(RAW, nm + ".json") + if os.path.exists(outp) and json.load(open(outp)).get("status") == "ok": + print(f"skip {nm} (done)", flush=True) + continue + ld = wait_quiet() + print(f"== {nm} load total {ld['total_pct']:.0f}% pin-cpu {ld['pin_cpu_pct']:.0f}% noisy={ld['noisy']}", flush=True) + t0 = time.time() + with open(os.path.join(RAW, nm + ".txt"), "w") as fh: + fh.write(" ".join(cmd) + "\n") + fh.flush() + try: + subprocess.run(cmd, stdout=fh, stderr=subprocess.STDOUT, timeout=3600) + except subprocess.TimeoutExpired: + fh.write("TIMEOUT\n") + ld["seconds"] = time.time() - t0 + ld["after"] = psutil.cpu_percent(interval=None) + json.dump(ld, open(os.path.join(RAW, nm + ".load.json"), "w")) + print(f" done in {ld['seconds']:.0f}s ok={os.path.exists(outp) and json.load(open(outp)).get("status") == "ok"}", flush=True) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/rust-stage-c/summarize.py b/benchmarks/rust-stage-c/summarize.py new file mode 100644 index 0000000..3f4f6c6 --- /dev/null +++ b/benchmarks/rust-stage-c/summarize.py @@ -0,0 +1,425 @@ +"""Read raw/*.json (Rust stage_c_bench and Python bench_c.py results) and write tables.md + summary.json. + + python summarize.py # from benchmarks/rust-stage-c/ + +Rules: medians over the repeated runs of a cell (min-max shown); QPS at recall 0.95 is log-linearly interpolated +between the two sweep points that bracket it (no extrapolation; '-' = never reached); memory = working set after +the build minus the baseline taken after imports (Python) / before any data (Rust). +""" +import glob +import json +import math +import os +import re +import statistics as st + +HERE = os.path.dirname(os.path.abspath(__file__)) +RAW = os.path.join(HERE, "raw") + +PRETTY = { + "rust_exact": "ISONGraph Rust exact", "rust_ivf": "ISONGraph Rust IVF", "rust_hnsw": "ISONGraph Rust HNSW", + "rust_ivf_i8x4": "ISONGraph Rust IVF + int8 x4", "rust_hnsw_i8x4": "ISONGraph Rust HNSW + int8 x4 (setting on)", + "numpy_bf": "numpy brute force", "faiss_flat": "faiss IndexFlatIP", "faiss_ivfflat": "faiss IVFFlat", + "faiss_hnsw": "faiss HNSWFlat", "hnswlib": "hnswlib", "usearch": "usearch", +} + + +def load_all(): + runs = {} + for p in sorted(glob.glob(os.path.join(RAW, "*.json"))): + if p.endswith(".load.json"): + continue + try: + d = json.load(open(p)) + except Exception: # noqa: BLE001 + continue + if d.get("status") != "ok": + continue + sysname = d.get("system", "") + if d.get("exps") == ["persist"]: + sysname = "rust_persist" + ld = {} + lp = p[:-5] + ".load.json" + if os.path.exists(lp): + ld = json.load(open(lp)) + d["_file"] = os.path.basename(p) + d["_load"] = ld + runs.setdefault((sysname, d["ds"], d["n"]), []).append(d) + return runs + + +def med(xs): + xs = [x for x in xs if x is not None] + return st.median(xs) if xs else None + + +def rng(xs): + xs = [x for x in xs if x is not None] + return (min(xs), max(xs)) if xs else (None, None) + + +def f(x, n=1, comma=True): + if x is None: + return "-" + if isinstance(x, float) and (math.isnan(x) or math.isinf(x)): + return "-" + return f"{x:,.{n}f}" if comma else f"{x:.{n}f}" + + +def cell(xs, n=1): + """median (min-max) across runs""" + m = med(xs) + lo, hi = rng(xs) + if m is None: + return "-" + if len(xs) < 2: + return f"{f(m, n)} (1 run)" + return f"{f(m, n)} ({f(lo, n)}-{f(hi, n)})" + + +def qps_of(pt): + return pt.get("qps_1t") + + +def batch_of(pt): + b = pt.get("batch") + return None if not b else b.get("qps_batch") + + +def interp(sweep, key, target=0.95): + """log-linear interpolation of key(pt) at recall=target; None if not bracketed.""" + pts = [p for p in sweep if p.get("knob") is not None and key(p) is not None] + pts.sort(key=lambda p: p["recall"]) + for a, b in zip(pts, pts[1:]): + if a["recall"] <= target <= b["recall"] and b["recall"] > a["recall"]: + t = (target - a["recall"]) / (b["recall"] - a["recall"]) + return math.exp(math.log(key(a)) * (1 - t) + math.log(key(b)) * t) + hit = [p for p in pts if p["recall"] >= target] + if len(pts) and pts[0]["recall"] >= target: + return None # target below the first measured point: not bracketed + return None + + +def ref_point(sweep, target=0.95): + for p in sorted([p for p in sweep if p.get("knob") is not None], key=lambda p: p["knob"]): + if p["recall"] >= target: + return p + return None + + +def mem_delta(d, which="mem_after_build"): + b = d.get("mem_baseline", {}).get("rss_mb") + a = d.get(which, {}).get("rss_mb") + return None if a is None or b is None else a - b + + +def peak_delta(d, which="mem_after_build"): + b = d.get("mem_baseline", {}).get("rss_mb") + a = d.get(which, {}).get("peak_mb") + return None if a is None or b is None else a - b + + +def build_s(d): + b = d.get("build", {}) + if "total_s" in b: + return b["total_s"] + return b.get("ingest_s") + + +def index_s(d): + return d.get("build", {}).get("index_s") + + +def exact_pt(d): + sw = d.get("sweep", []) + return sw[0] if sw else None + + +def auto_qps(d): + if "loop_allthreads" in d: + return d["loop_allthreads"].get("qps") + pt = exact_pt(d) + if pt and "auto_threads" in pt: + return pt["auto_threads"].get("qps") + return None + + +def exact_batch(d): + pt = exact_pt(d) + if pt and pt.get("batch"): + return pt["batch"].get("qps_batch") + return None + + +def main(): + R = load_all() + md = [] + S = {} # summary.json + out = md.append + + def runs(sysn, ds, n): + return R.get((sysn, ds, n), []) + + # ---------------------------------------------------------------- Q1 exact + out("## Q1. Exact scan: ISONGraph Rust vs numpy vs faiss IndexFlatIP (single thread, one pinned P-core)\n") + out("QPS, median over runs (min-max). `ratio` = Rust / other, > 1 means Rust is faster. Rounds were interleaved " + "(Rust, numpy, faiss, Rust, ...) in one session.\n") + out("| cell | runs | Rust exact | numpy | faiss Flat | Rust/numpy | Rust/faiss | Rust auto-thread loop | faiss Flat batch (32 thr) | numpy batch | Rust batch |") + out("|---|---|---|---|---|---|---|---|---|---|---|") + S["q1"] = {} + for ds, n in (("text384", 100_000), ("text384", 1_000_000), ("glove", 100_000), ("sift", 100_000)): + rr, nn, ff = runs("rust_exact", ds, n), runs("numpy_bf", ds, n), runs("faiss_flat", ds, n) + if not rr: + continue + q = lambda L: [qps_of(exact_pt(d)) for d in L if exact_pt(d)] # noqa: E731 + mr, mn, mf = med(q(rr)), med(q(nn)), med(q(ff)) + S["q1"][f"{ds}_{n}"] = {"rust": mr, "numpy": mn, "faiss_flat": mf, "rust_runs": q(rr), "numpy_runs": q(nn), "faiss_runs": q(ff), + "ratio_numpy": (mr / mn if mr and mn else None), "ratio_faiss": (mr / mf if mr and mf else None)} + out(f"| {ds} {n:,} | {len(rr)}/{len(nn)}/{len(ff)} | {cell(q(rr))} | {cell(q(nn))} | {cell(q(ff))} | " + f"{f(mr / mn, 2) if mr and mn else '-'}x | {f(mr / mf, 2) if mr and mf else '-'}x | " + f"{cell([auto_qps(d) for d in rr], 0)} | {cell([exact_batch(d) for d in ff], 0)} | {cell([exact_batch(d) for d in nn], 0)} | {cell([exact_batch(d) for d in rr], 0)} |") + out("") + + # ---------------------------------------------------------------- Q2 ANN at recall 0.95 + ann_sys = ["rust_hnsw", "rust_ivf", "rust_ivf_i8x4", "rust_hnsw_i8x4", "hnswlib", "faiss_hnsw", "usearch", "faiss_ivfflat"] + out("## Q2. Approximate search at recall@10 = 0.95, single thread (pinned), text384\n") + out("QPS interpolated (log-linear) between the two sweep points that bracket recall 0.95; `-` = not bracketed by the " + "sweep (recall 0.95 not reached, or the lowest point already above it: then the lowest measured point is shown in the " + "`first point` column). Build = ingest + index build on the library's own path with 32 threads. Memory = working set " + "after the build minus baseline.\n") + for ds, n in (("text384", 100_000), ("text384", 1_000_000)): + out(f"### {ds} {n:,}\n") + out("| system | runs | QPS @0.95 (1T) | knob at first recall>=0.95 | recall at that knob | p99 ms there | build s | of which index s | RSS after build MB | peak MB | first point (knob: recall, QPS) | best recall reached |") + out("|---|---|---|---|---|---|---|---|---|---|---|---|") + S.setdefault("q2", {})[f"{ds}_{n}"] = {} + for sn in ann_sys + ["rust_exact", "faiss_flat"]: + rs = runs(sn, ds, n) + if not rs: + continue + if sn in ("rust_exact", "faiss_flat"): + q1 = [qps_of(exact_pt(d)) for d in rs if exact_pt(d)] + out(f"| {PRETTY[sn]} (exact, recall 1.0) | {len(rs)} | {cell(q1)} | - | 1.000 | {f(med([exact_pt(d)['p99_ms'] for d in rs if exact_pt(d)]), 2)} | {cell([build_s(d) for d in rs], 2)} | 0 | {cell([mem_delta(d) for d in rs], 0)} | - | - | 1.000 |") + continue + qs = [interp(d["sweep"], qps_of) for d in rs] + refs = [ref_point(d["sweep"]) for d in rs] + kn = [r["knob"] for r in refs if r] + rc = [r["recall"] for r in refs if r] + p99 = [r["p99_ms"] for r in refs if r] + fp = rs[0]["sweep"][0] if rs[0].get("sweep") else None + fps = f"{fp['knob']}: {fp['recall']:.3f}, {fp['qps_1t']:,.0f}" if fp else "-" + best = max((p["recall"] for d in rs for p in d["sweep"]), default=None) + out(f"| {PRETTY[sn]} | {len(rs)} | {cell([x for x in qs if x], 0)} | {f(med(kn), 0) if kn else '-'} | {f(med(rc), 4) if rc else '-'} | {f(med(p99), 2)} | " + f"{cell([build_s(d) for d in rs], 1)} | {cell([index_s(d) for d in rs], 1)} | {cell([mem_delta(d) for d in rs], 0)} | {cell([peak_delta(d) for d in rs], 0)} | {fps} | {f(best, 4)} |") + S["q2"][f"{ds}_{n}"][sn] = {"qps_at_095": med([x for x in qs if x]), "runs": [x for x in qs if x], "build_s": med([build_s(d) for d in rs]), + "rss_after_build_mb": med([mem_delta(d) for d in rs]), "best_recall": best} + out("") + # sweeps + out("### Sweeps (median over runs; recall / QPS 1T)\n") + for ds, n in (("text384", 100_000), ("text384", 1_000_000)): + out(f"**{ds} {n:,}**\n") + for sn in ann_sys: + rs = runs(sn, ds, n) + if not rs: + continue + knobs = sorted({p["knob"] for d in rs for p in d["sweep"]}) + cells = [] + for kn in knobs: + pts = [p for d in rs for p in d["sweep"] if p["knob"] == kn] + cells.append(f"{kn}: {med([p['recall'] for p in pts]):.3f} / {med([qps_of(p) for p in pts]):,.0f}") + out(f"* {PRETTY[sn]} ({rs[0].get('knob_name', '')}): " + "; ".join(cells)) + out("") + + # ---------------------------------------------------------------- Q3 batch / all-core + out("## Q3. Batch / all-core throughput, text384 100k (32 logical CPUs, 1,000 queries per batch, k=10)\n") + out("Batch QPS is the median of 3 timed batches after a warm-up batch, at the sweep point nearest recall 0.95 " + "(first knob with recall >= 0.95; its recall is shown). Exact rows are recall 1.0.\n") + out("| system | runs | knob | recall there | batch QPS (32 thr) | 1T QPS there | batch / 1T |") + out("|---|---|---|---|---|---|---|") + S["q3"] = {} + for ds, n in (("text384", 100_000), ("text384", 1_000_000)): + out(f"| **{ds} {n:,}** | | | | | | |") + for sn in ["rust_exact", "faiss_flat", "numpy_bf", "rust_ivf", "rust_ivf_i8x4", "faiss_ivfflat", "rust_hnsw", "faiss_hnsw", "hnswlib", "usearch"]: + rs = runs(sn, ds, n) + if not rs: + continue + if sn in ("rust_exact", "faiss_flat", "numpy_bf"): + b = [exact_batch(d) for d in rs] + o = [qps_of(exact_pt(d)) for d in rs if exact_pt(d)] + out(f"| {PRETTY[sn]} | {len(rs)} | - | 1.000 | {cell(b, 0)} | {cell(o, 0)} | {f(med(b) / med(o), 0) + 'x' if med(b) and med(o) else '-'} |") + S["q3"][f"{sn}_{ds}_{n}"] = {"batch_qps": med(b)} + continue + pts = [ref_point(d["sweep"]) for d in rs] + pts = [p for p in pts if p] + if not pts: + continue + b = [batch_of(p) for p in pts] + o = [qps_of(p) for p in pts] + out(f"| {PRETTY[sn]} | {len(rs)} | {pts[0]['knob']} | {f(med([p['recall'] for p in pts]), 3)} | {cell(b, 0)} | {cell(o, 0)} | {f(med(b) / med(o), 0) + 'x' if med(b) and med(o) else '-'} |") + S["q3"][f"{sn}_{ds}_{n}"] = {"batch_qps": med(b), "knob": pts[0]["knob"]} + out("") + out("Rust concurrent clients (each caller runs single-scan-thread searches against one shared store; 3 s per client count, not pinned), QPS:\n") + out("| cell | mode | 1 | 2 | 4 | 8 | 16 |") + out("|---|---|---|---|---|---|---|") + for ds, n in (("text384", 100_000), ("text384", 1_000_000)): + for sn in ("rust_exact", "rust_ivf", "rust_hnsw"): + rs = [d for d in runs(sn, ds, n) if "concurrent_clients" in d] + if not rs: + continue + row = [] + for i in range(5): + row.append(f(med([d["concurrent_clients"][i]["qps"] for d in rs]), 0)) + kn = rs[0].get("ref_knob") + out(f"| {ds} {n:,} | {PRETTY[sn]}{' (knob ' + str(kn) + ')' if kn else ''} | " + " | ".join(row) + " |") + out("") + + # ---------------------------------------------------------------- Q4 int8 + out("## Q4. int8 scan with exact rerank on real embeddings (exact search, Rust)\n") + out("`recall vs f32 scan` is `int8_recall_report` (int8 top-10 vs the library's own f32 scan, first 1,000 queries; 300 at 1M); " + "`recall vs truth` is against the float64 ground truth in the timed loop. Speedup = int8 QPS / f32 QPS from the same process " + "(1T = pinned single thread; auto = one query at a time on the automatic thread count; batch = 1,000-query batch, 32 threads). " + "Medians over runs where several exist.\n") + out("| dataset | factor | recall vs f32 scan (mean / worst query) | recall vs truth | f32 1T QPS | int8 1T QPS | 1T speedup | auto-thread speedup | batch speedup | exact scores |") + out("|---|---|---|---|---|---|---|---|---|---|") + S["q4"] = {} + for ds, n in (("text384", 100_000), ("text384", 1_000_000), ("glove", 100_000), ("sift", 100_000)): + rs = [d for d in runs("rust_exact", ds, n) if "int8_study" in d] + if not rs: + continue + S["q4"][f"{ds}_{n}"] = {} + for fac in (1, 2, 4, 8): + by = [next(b for b in d["int8_study"]["by_factor"] if b["rerank_factor"] == fac) for d in rs] + f1 = [d["int8_study"]["f32"]["one_thread"]["qps_1t"] for d in rs] + fa = [d["int8_study"]["f32"]["auto_threads"]["qps_1t"] for d in rs] + fb = [d["int8_study"]["f32"]["batch"]["qps_batch"] for d in rs] + i1 = [b["one_thread"]["qps_1t"] for b in by] + ia = [b["auto_threads"]["qps_1t"] for b in by] + ib = [b["batch"]["qps_batch"] for b in by] + rec = med([b["recall_vs_f32_scan"] for b in by]) + mn = min(b["min_recall_vs_f32_scan"] for b in by) + rt = med([b["one_thread"]["recall"] for b in by]) + sp1 = med([a / b for a, b in zip(i1, f1)]) + spa = med([a / b for a, b in zip(ia, fa)]) + spb = med([a / b for a, b in zip(ib, fb)]) + ex = all(b["scores_exact"] for b in by) + out(f"| {ds} {n:,} | x{fac} | {f(rec, 4)} / {f(mn, 2)} | {f(rt, 4)} | {f(med(f1), 0)} | {f(med(i1), 0)} | {f(sp1, 2)}x | {f(spa, 2)}x | {f(spb, 2)}x | {ex} |") + S["q4"][f"{ds}_{n}"][f"x{fac}"] = {"recall_vs_f32": rec, "worst": mn, "recall_vs_truth": rt, "speedup_1t": sp1, "speedup_auto": spa, "speedup_batch": spb} + mm = med([d["int8_study"]["rss_delta_mb"] for d in rs]) + out(f"| {ds} {n:,} | memory | int8 copy {f(med([d['int8_study']['quant_bytes_mb'] for d in rs]), 0)} MB; working-set delta {f(mm, 0)} MB | | | | | | | |") + out("") + out("IVF and HNSW combined with int8 (setting `ScanPrecision::Int8 { rerank_factor: 4 }` before the build) are in the Q2 table (`+ int8 x4` rows) with the sweeps above.\n") + + # ---------------------------------------------------------------- Q5 persistence + out("## Q5. Persistence: ISONGraph JSON sidecar vs binary snapshot vs faiss / hnswlib / usearch index files\n") + out("ISONGraph = store only (vectors + keys), index not persisted. `ready` for the ANN rows adds the index rebuild " + "(and a first query) that every ISONGraph load needs. Library rows are native `write_index` / `save_index` / `load`; " + "their load time is to a usable index (the graph is inside the file).\n") + out("| N | system | save s | file MB | load s | first query after load ms | time to ready s | note |") + out("|---|---|---|---|---|---|---|---|") + S["q5"] = {} + for n in (100_000, 1_000_000): + rs = [d for d in runs("rust_persist", "text384", n)] + if rs: + p = rs[0]["persist"] + rd = p.get("time_to_ready_after_snapshot_load", {}) + S["q5"][f"rust_{n}"] = p + out(f"| {n:,} | ISONGraph Rust JSON sidecar | {f(p['json_save_s'], 2)} | {f(p['json_mb'], 0)} | {f(p['json_load_s'], 2)} (runs {', '.join(f(x, 2) for x in p['json_load_s_runs'])}) | {f(p['json_first_query_ms'], 1)} | {f(p['json_load_s'] + p['json_first_query_ms'] / 1000, 2)} | exact search ready; answers equal original: {p['json_answers_equal_original']} |") + out(f"| {n:,} | ISONGraph Rust binary snapshot | {f(p['snapshot_save_s'], 2)} (+{f(p['sidecar_fingerprint_s'], 2)} sidecar fingerprint) | {f(p['snapshot_mb'], 0)} | {f(p['snapshot_load_s'], 2)} (runs {', '.join(f(x, 2) for x in p['snapshot_load_s_runs'])}) | {f(p['snapshot_first_query_ms'], 1)} | {f(p['snapshot_load_s'] + p['snapshot_first_query_ms'] / 1000, 2)} | exact search ready; answers equal original: {p['snapshot_answers_equal_original']} |") + for md_, v in rd.items(): + out(f"| {n:,} | ISONGraph Rust snapshot + {md_.upper()} rebuild | - | - | {f(v['load_s'], 2)} | {f(v['first_query_ms'], 1)} | {f(v['ready_s'], 2)} | load + build {f(v['build_s'], 1)} s + first query |") + for sn in ("faiss_flat", "faiss_ivfflat", "faiss_hnsw", "hnswlib", "usearch"): + rr = [d for d in runs(sn, "text384", n) if "saveload" in d and "save_s" in d["saveload"]] + if not rr: + continue + sl = [d["saveload"] for d in rr] + out(f"| {n:,} | {PRETTY[sn]} | {cell([s['save_s'] for s in sl], 3)} | {f(med([s['disk_mb'] for s in sl]), 0)} | {cell([s['load_s'] for s in sl], 3)} | {f(med([s['first_query_after_load_ms'] for s in sl]), 2)} | {f(med([s['time_to_ready_s'] for s in sl]), 3)} | native index file |") + out("") + + # ---------------------------------------------------------------- Q6 writes + out("## Q6. Writes and the first query after writes (text384; index built, recall ~0.95 operating point)\n") + out("10k inserts, then 10k updates (new vector under an existing id), then 10k deletes, one call at a time on the built index " + "(Rust: `add` / `remove`; libraries as in the competitive harness: calls of 1,000). Then the first search and recall@10 against " + "the exact post-mutation float64 truth at the same knob. `p99/max` are per-write latencies (Rust only).\n") + out("| N | system | knob | insert/s | update/s | delete/s | insert p99 / max ms | update p99 / max ms | first query after ms | recall before | recall after | QPS 1T after |") + out("|---|---|---|---|---|---|---|---|---|---|---|---|") + S["q6"] = {} + for n in (100_000, 1_000_000): + for sn in ("rust_ivf", "rust_hnsw", "faiss_ivfflat", "hnswlib", "usearch", "faiss_hnsw"): + rs = [d for d in runs(sn, "text384", n) if isinstance(d.get("mutation"), dict) and "insert_10k" in d["mutation"]] + if not rs: + continue + ms = [d["mutation"] for d in rs] + def g(key, sub="per_s"): + return med([m_[key].get(sub) for m_ in ms if isinstance(m_.get(key), dict) and sub in m_[key]]) + def lat(key): + a = med([m_[key]["lat"]["p99_ms"] for m_ in ms if "lat" in m_.get(key, {})]) + b = med([m_[key]["lat"]["max_ms"] for m_ in ms if "lat" in m_.get(key, {})]) + return "-" if a is None else f"{f(a, 2)} / {f(b, 1)}" + def un(key): + return any(isinstance(m_.get(key), dict) and ("unsupported" in m_[key] or "error" in m_[key]) for m_ in ms) + fqa = med([m_.get("first_query_after_ms") for m_ in ms]) + ra = med([m_.get("recall_after") for m_ in ms]) + rb = med([m_.get("recall_before") for m_ in ms]) + qa = med([(m_.get("after") or {}).get("qps_1t") for m_ in ms]) + out(f"| {n:,} | {PRETTY[sn]} ({len(rs)}) | {ms[0].get('ref_knob')} | {f(g('insert_10k'), 0)} | {f(g('update_10k'), 0)}{' (unsupported)' if un('update_10k') else ''} | {f(g('delete_10k'), 0)}{' (unsupported)' if un('delete_10k') else ''} | {lat('insert_10k')} | {lat('update_10k')} | {f(fqa, 1)} | {f(rb, 4)} | {f(ra, 4)} | {f(qa, 0)} |") + S["q6"][f"{sn}_{n}"] = {"insert_s": g("insert_10k"), "update_s": g("update_10k"), "delete_s": g("delete_10k"), "first_query_after_ms": fqa, "recall_after": ra} + out("") + + # ---------------------------------------------------------------- Q7 memory + out("## Q7. Memory at 1M x 384 (working set, MB; baseline subtracted)\n") + out("Raw vectors = 1,000,000 x 384 x 4 = 1,465 MB. Each Rust row is its own process (ingest, then that mode only). " + "Python rows: working set after the index is built and again after the sweep, with the harness' preallocation (hnswlib `max_elements` = N + 20k).\n") + out("| system | after ingest | after build / first query | peak during build | after the sweep | x raw vectors (after build) |") + out("|---|---|---|---|---|---|") + S["q7"] = {} + raw_mb = 1_000_000 * 384 * 4 / 2**20 + for n in (100_000, 1_000_000): + raw_mb = n * 384 * 4 / 2**20 + out(f"| **text384 {n:,}** (raw {f(raw_mb, 0)} MB) | | | | | |") + for sn in ["rust_exact", "rust_ivf", "rust_hnsw", "rust_ivf_i8x4", "rust_hnsw_i8x4", "faiss_flat", "numpy_bf", "faiss_ivfflat", "faiss_hnsw", "hnswlib", "usearch"]: + rs = runs(sn, "text384", n) + if not rs: + continue + ingest = med([mem_delta(d, "mem_after_ingest") for d in rs]) if sn.startswith("rust") else None + if sn.startswith("rust"): + aft = [mem_delta(d, "mem_after_build") if "mem_after_build" in d else mem_delta(d, "mem_after_first_exact_query") for d in rs] + # the exact process runs the int8 study before it measures, so its own peak would include the int8 copy: use the peak after ingest + pk = [peak_delta(d, "mem_after_build") if "mem_after_build" in d else peak_delta(d, "mem_after_ingest") for d in rs] + else: + aft = [mem_delta(d, "mem_after_first_query") for d in rs] + pk = [peak_delta(d, "mem_after_first_query") for d in rs] + sweep = [mem_delta(d, "mem_after_sweep") for d in rs] + a = med(aft) + out(f"| {PRETTY[sn]} | {f(ingest, 0)} | {f(a, 0)} | {f(med(pk), 0)} | {f(med(sweep), 0)} | {f(a / raw_mb, 2) + 'x' if a else '-'} |") + S["q7"][f"{sn}_{n}"] = {"after_build_mb": a, "peak_mb": med(pk), "x_raw": (a / raw_mb if a else None)} + ex = [d for d in runs("rust_exact", "text384", n) if "int8_study" in d] + if ex: + a8 = med([d["int8_study"]["mem_after_int8"]["rss_mb"] - d["mem_baseline"]["rss_mb"] for d in ex]) + out(f"| ISONGraph Rust exact + int8 copy (any factor) | - | {f(a8, 0)} | - | - | {f(a8 / raw_mb, 2)}x |") + S["q7"][f"rust_exact_int8_{n}"] = {"after_build_mb": a8, "x_raw": a8 / raw_mb} + out("") + # Rust lib-reported breakdown + out("Rust library-reported breakdown (`memory_stats`, 1M):\n") + for sn in ("rust_ivf", "rust_hnsw", "rust_ivf_i8x4", "rust_hnsw_i8x4"): + rs = [d for d in runs(sn, "text384", 1_000_000) if "memory_stats" in d] + if rs: + m_ = rs[0]["memory_stats"] + out(f"* {PRETTY[sn]}: matrix {m_['matrix_bytes'] / 2**20:,.0f} MB, metadata {m_['metadata_bytes'] / 2**20:,.0f} MB, index {m_['index_bytes'] / 2**20:,.0f} MB, int8 copy {m_['quant_bytes'] / 2**20:,.0f} MB") + out("") + + # ---------------------------------------------------------------- build time / noise + out("## Appendix: noisy runs and machine load at job start\n") + out("| job file | total CPU % | pinned-CPU % | flagged noisy |") + out("|---|---|---|---|") + for (sn, ds, n), L in sorted(R.items()): + for d in L: + ld = d.get("_load", {}) + if ld.get("noisy") or ld.get("total_pct", 0) > 25: + out(f"| {d['_file']} | {f(ld.get('total_pct'), 0)} | {f(ld.get('pin_cpu_pct'), 0)} | {ld.get('noisy')} |") + out("") + open(os.path.join(HERE, "tables.md"), "w").write("\n".join(md)) + json.dump(S, open(os.path.join(HERE, "summary.json"), "w"), indent=1, default=float) + print("\n".join(md)) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/rust-stage-c/summary.json b/benchmarks/rust-stage-c/summary.json new file mode 100644 index 0000000..6389bf0 --- /dev/null +++ b/benchmarks/rust-stage-c/summary.json @@ -0,0 +1,738 @@ +{ + "q1": { + "text384_100000": { + "rust": 142.61781794451153, + "numpy": 112.69994418654542, + "faiss_flat": 117.76265279742822, + "rust_runs": [ + 113.97457184561665, + 145.7347640576308, + 142.61781794451153 + ], + "numpy_runs": [ + 112.23707069649419, + 121.32483276831196, + 112.69994418654542 + ], + "faiss_runs": [ + 117.76265279742822, + 116.54350928026874, + 135.50167209408608 + ], + "ratio_numpy": 1.2654648498178922, + "ratio_faiss": 1.2110615255062096 + }, + "text384_1000000": { + "rust": 11.07807843528789, + "numpy": 11.065075402665316, + "faiss_flat": 11.368230900068069, + "rust_runs": [ + 11.07807843528789 + ], + "numpy_runs": [ + 11.065075402665316 + ], + "faiss_runs": [ + 11.368230900068069 + ], + "ratio_numpy": 1.0011751417997063, + "ratio_faiss": 0.9744769025778285 + }, + "glove_100000": { + "rust": 548.2434799732961, + "numpy": 376.9976257074899, + "faiss_flat": 605.0412763708653, + "rust_runs": [ + 548.4488768863897, + 537.1059367876796, + 548.2434799732961 + ], + "numpy_runs": [ + 376.9976257074899, + 456.60346991576137, + 323.3826549125653 + ], + "faiss_runs": [ + 603.5059472450935, + 605.0412763708653, + 621.6223379371471 + ], + "ratio_numpy": 1.4542358959010389, + "ratio_faiss": 0.9061257494063025 + }, + "sift_100000": { + "rust": 396.93923332695437, + "numpy": 385.5058585640153, + "faiss_flat": 431.6622346944021, + "rust_runs": [ + 349.9850871354372, + 470.53002242687194, + 396.93923332695437 + ], + "numpy_runs": [ + 389.30091610454076, + 313.9052557275998, + 385.5058585640153 + ], + "faiss_runs": [ + 431.6622346944021, + 521.5784570587275, + 421.52335367303846 + ], + "ratio_numpy": 1.0296581089727863, + "ratio_faiss": 0.9195597887037069 + } + }, + "q2": { + "text384_100000": { + "rust_hnsw": { + "qps_at_095": 3892.5627880837083, + "runs": [ + 3892.5627880837083, + 3313.8526262972796, + 4486.190315407603 + ], + "build_s": 7.1280035999999996, + "rss_after_build_mb": 198.859375, + "best_recall": 0.9936 + }, + "rust_ivf": { + "qps_at_095": 759.1477903772653, + "runs": [ + 968.0247849899364, + 751.6164072736235, + 759.1477903772653 + ], + "build_s": 0.5802664, + "rss_after_build_mb": 166.296875, + "best_recall": 0.9952296819787986 + }, + "rust_ivf_i8x4": { + "qps_at_095": 2059.3074200847327, + "runs": [ + 2059.3074200847327 + ], + "build_s": 0.6113841999999999, + "rss_after_build_mb": 204.7578125, + "best_recall": 0.9953 + }, + "rust_hnsw_i8x4": { + "qps_at_095": 4571.0305921424415, + "runs": [ + 4571.0305921424415 + ], + "build_s": 5.7303703, + "rss_after_build_mb": 237.828125, + "best_recall": 0.9796 + }, + "hnswlib": { + "qps_at_095": 2723.145193021137, + "runs": [ + 2723.145193021137, + 2751.4155852591994, + 2479.804927704833 + ], + "build_s": 8.380476800026372, + "rss_after_build_mb": 200.875, + "best_recall": 0.993 + }, + "faiss_hnsw": { + "qps_at_095": 3964.747552352872, + "runs": [ + 3399.6842989183237, + 4117.024698676574, + 3964.747552352872 + ], + "build_s": 9.61685940012103, + "rss_after_build_mb": 206.68359375, + "best_recall": 0.9971 + }, + "usearch": { + "qps_at_095": 2070.0690749805694, + "runs": [ + 2070.0690749805694, + 2173.553472326835, + 2012.240386246597 + ], + "build_s": 6.788179500028491, + "rss_after_build_mb": 181.5, + "best_recall": 0.9927 + }, + "faiss_ivfflat": { + "qps_at_095": 1065.4716029008105, + "runs": [ + 1096.1413481327613, + 1065.4716029008105, + 1027.782299058379 + ], + "build_s": 1.4772849000873975, + "rss_after_build_mb": 240.13671875, + "best_recall": 0.9958 + } + }, + "text384_1000000": { + "rust_hnsw": { + "qps_at_095": 1812.8616045254419, + "runs": [ + 1812.8616045254419 + ], + "build_s": 172.7108059, + "rss_after_build_mb": 1785.55078125, + "best_recall": 0.9901 + }, + "rust_ivf": { + "qps_at_095": 137.02456462635791, + "runs": [ + 137.02456462635791 + ], + "build_s": 7.6396557, + "rss_after_build_mb": 1643.0859375, + "best_recall": 0.9772222222222222 + }, + "rust_ivf_i8x4": { + "qps_at_095": 235.10997666022985, + "runs": [ + 235.10997666022985 + ], + "build_s": 8.0598049, + "rss_after_build_mb": 2011.3203125, + "best_recall": 0.9780104712041885 + }, + "rust_hnsw_i8x4": { + "qps_at_095": null, + "runs": [], + "build_s": 145.8629402, + "rss_after_build_mb": 2155.71875, + "best_recall": 0.9685 + }, + "hnswlib": { + "qps_at_095": 1516.618902315218, + "runs": [ + 1516.618902315218 + ], + "build_s": 157.21686790010426, + "rss_after_build_mb": 1876.30859375, + "best_recall": 0.9817 + }, + "faiss_hnsw": { + "qps_at_095": 1773.218758405916, + "runs": [ + 1773.218758405916 + ], + "build_s": 189.53744800016284, + "rss_after_build_mb": 1814.55078125, + "best_recall": 0.9885 + }, + "usearch": { + "qps_at_095": 611.6222163248258, + "runs": [ + 611.6222163248258 + ], + "build_s": 215.8060878997785, + "rss_after_build_mb": 1713.91015625, + "best_recall": 0.9818 + }, + "faiss_ivfflat": { + "qps_at_095": 179.58498142709087, + "runs": [ + 179.58498142709087 + ], + "build_s": 35.34201039990876, + "rss_after_build_mb": 1881.44921875, + "best_recall": 0.9801 + } + } + }, + "q3": { + "rust_exact_text384_100000": { + "batch_qps": 5319.762972640991 + }, + "faiss_flat_text384_100000": { + "batch_qps": 5085.293077584747 + }, + "numpy_bf_text384_100000": { + "batch_qps": 672.7372683713093 + }, + "rust_ivf_text384_100000": { + "batch_qps": 24360.417245226574, + "knob": 128 + }, + "rust_ivf_i8x4_text384_100000": { + "batch_qps": 11850.475441074697, + "knob": 128 + }, + "faiss_ivfflat_text384_100000": { + "batch_qps": 3014.384340501241, + "knob": 128 + }, + "rust_hnsw_text384_100000": { + "batch_qps": 13989.28141258168, + "knob": 128 + }, + "faiss_hnsw_text384_100000": { + "batch_qps": 16693.515414707883, + "knob": 96 + }, + "hnswlib_text384_100000": { + "batch_qps": 12743.269000605947, + "knob": 128 + }, + "usearch_text384_100000": { + "batch_qps": 13656.2396701793, + "knob": 128 + }, + "rust_exact_text384_1000000": { + "batch_qps": 560.2943069111799 + }, + "faiss_flat_text384_1000000": { + "batch_qps": 297.6627048142383 + }, + "numpy_bf_text384_1000000": { + "batch_qps": 60.7480748785275 + }, + "rust_ivf_text384_1000000": { + "batch_qps": 10067.715454144576, + "knob": 256 + }, + "rust_ivf_i8x4_text384_1000000": { + "batch_qps": 6817.882761051617, + "knob": 256 + }, + "faiss_ivfflat_text384_1000000": { + "batch_qps": 470.62373693374275, + "knob": 256 + }, + "rust_hnsw_text384_1000000": { + "batch_qps": 5582.615734602448, + "knob": 192 + }, + "faiss_hnsw_text384_1000000": { + "batch_qps": 4967.3989593157485, + "knob": 192 + }, + "hnswlib_text384_1000000": { + "batch_qps": 6679.834845899164, + "knob": 192 + }, + "usearch_text384_1000000": { + "batch_qps": 3973.1399843644836, + "knob": 192 + } + }, + "q4": { + "text384_100000": { + "x1": { + "recall_vs_f32": 0.9903999999999983, + "worst": 0.8, + "recall_vs_truth": 0.9905, + "speedup_1t": 3.030278492983568, + "speedup_auto": 4.753611487905735, + "speedup_batch": 0.9183072459150814 + }, + "x2": { + "recall_vs_f32": 1.0, + "worst": 1.0, + "recall_vs_truth": 0.9998994974874372, + "speedup_1t": 3.068122521958389, + "speedup_auto": 5.230478249952417, + "speedup_batch": 0.8598264383042334 + }, + "x4": { + "recall_vs_f32": 1.0, + "worst": 1.0, + "recall_vs_truth": 0.9999, + "speedup_1t": 3.1151112736952498, + "speedup_auto": 5.271638586743582, + "speedup_batch": 0.7845569132146976 + }, + "x8": { + "recall_vs_f32": 1.0, + "worst": 1.0, + "recall_vs_truth": 0.9999, + "speedup_1t": 3.1545649451627935, + "speedup_auto": 4.324063224983725, + "speedup_batch": 0.6611004109357426 + } + }, + "text384_1000000": { + "x1": { + "recall_vs_f32": 0.9916666666666667, + "worst": 0.9, + "recall_vs_truth": 0.9907216494845361, + "speedup_1t": 2.9710869837218796, + "speedup_auto": 3.130041948134966, + "speedup_batch": 0.9774927327192232 + }, + "x2": { + "recall_vs_f32": 1.0, + "worst": 1.0, + "recall_vs_truth": 1.0, + "speedup_1t": 3.2518886418933697, + "speedup_auto": 3.1708358105925205, + "speedup_batch": 0.9799683481762068 + }, + "x4": { + "recall_vs_f32": 1.0, + "worst": 1.0, + "recall_vs_truth": 1.0, + "speedup_1t": 2.964981017144653, + "speedup_auto": 3.15181296686569, + "speedup_batch": 0.959045865853096 + }, + "x8": { + "recall_vs_f32": 1.0, + "worst": 1.0, + "recall_vs_truth": 1.0, + "speedup_1t": 2.94863554091316, + "speedup_auto": 3.197416062579624, + "speedup_batch": 0.9282516615353272 + } + }, + "glove_100000": { + "x1": { + "recall_vs_f32": 0.9831999999999973, + "worst": 0.8, + "recall_vs_truth": 0.9832, + "speedup_1t": 3.6137428850631492, + "speedup_auto": 3.6477205491635565, + "speedup_batch": 0.614254179392763 + }, + "x2": { + "recall_vs_f32": 1.0, + "worst": 1.0, + "recall_vs_truth": 1.0, + "speedup_1t": 3.5696552828068966, + "speedup_auto": 3.3344568873665406, + "speedup_batch": 0.5078622310039242 + }, + "x4": { + "recall_vs_f32": 1.0, + "worst": 1.0, + "recall_vs_truth": 1.0, + "speedup_1t": 3.5236903718398214, + "speedup_auto": 3.5781472877264453, + "speedup_batch": 0.41158138121417803 + }, + "x8": { + "recall_vs_f32": 1.0, + "worst": 1.0, + "recall_vs_truth": 1.0, + "speedup_1t": 3.3937324630488184, + "speedup_auto": 2.965622425763507, + "speedup_batch": 0.3869016107540758 + } + }, + "sift_100000": { + "x1": { + "recall_vs_f32": 0.9762999999999962, + "worst": 0.8, + "recall_vs_truth": 0.9763, + "speedup_1t": 3.1008439143284994, + "speedup_auto": 4.072123587695357, + "speedup_batch": 0.8209407833646514 + }, + "x2": { + "recall_vs_f32": 1.0, + "worst": 1.0, + "recall_vs_truth": 1.0, + "speedup_1t": 3.061102135223856, + "speedup_auto": 4.503165416028502, + "speedup_batch": 0.7617135956744582 + }, + "x4": { + "recall_vs_f32": 1.0, + "worst": 1.0, + "recall_vs_truth": 1.0, + "speedup_1t": 3.006882486341465, + "speedup_auto": 4.803334393862242, + "speedup_batch": 0.6358458510894341 + }, + "x8": { + "recall_vs_f32": 1.0, + "worst": 1.0, + "recall_vs_truth": 1.0, + "speedup_1t": 3.024654538891212, + "speedup_auto": 4.306239250971731, + "speedup_batch": 0.5221449801398264 + } + } + }, + "q5": { + "rust_100000": { + "json_answers_equal_original": true, + "json_first_query_ms": 3.7659, + "json_load_peak_mb": 849.90625, + "json_load_s": 1.2717076, + "json_load_s_runs": [ + 1.2658816, + 1.2717076, + 1.2847952 + ], + "json_mb": 453.31162452697754, + "json_save_s": 1.6749711999999999, + "sidecar_fingerprint_s": 0.0009438, + "snapshot_answers_equal_original": true, + "snapshot_first_query_ms": 4.0445, + "snapshot_load_s": 0.1926398, + "snapshot_load_s_runs": [ + 0.1926398, + 0.1924933, + 0.1992911 + ], + "snapshot_mb": 148.76275157928467, + "snapshot_save_s": 0.1851224, + "time_to_ready_after_snapshot_load": { + "hnsw": { + "build_s": 5.9335669, + "first_query_ms": 0.235, + "load_s": 0.1864241, + "ready_s": 6.120226 + }, + "ivf": { + "build_s": 0.503499, + "first_query_ms": 0.5793, + "load_s": 0.2012672, + "ready_s": 0.7053455 + } + } + }, + "rust_1000000": { + "json_answers_equal_original": true, + "json_first_query_ms": 50.2532, + "json_load_peak_mb": 11505.41796875, + "json_load_s": 30.1224564, + "json_load_s_runs": [ + 30.1224564, + 30.4150109, + 26.3797141 + ], + "json_mb": 4534.09522819519, + "json_save_s": 42.8566, + "sidecar_fingerprint_s": 0.0033589, + "snapshot_answers_equal_original": true, + "snapshot_first_query_ms": 43.6024, + "snapshot_load_s": 4.5606966, + "snapshot_load_s_runs": [ + 4.395748, + 4.6875271, + 4.5606966 + ], + "snapshot_mb": 1488.5797986984253, + "snapshot_save_s": 7.6825872, + "time_to_ready_after_snapshot_load": { + "ivf": { + "build_s": 17.03469, + "first_query_ms": 31.1386, + "load_s": 4.5588184, + "ready_s": 21.624647 + } + } + } + }, + "q6": { + "rust_ivf_100000": { + "insert_s": 156139.97011480972, + "update_s": 128695.65665028371, + "delete_s": 911294.5850875755, + "first_query_after_ms": 3.8362000000000003, + "recall_after": 0.9529 + }, + "rust_hnsw_100000": { + "insert_s": 943.7420544586919, + "update_s": 936.2149981043519, + "delete_s": 1069038.5067670138, + "first_query_after_ms": 0.7369, + "recall_after": 0.9582 + }, + "faiss_ivfflat_100000": { + "insert_s": 235791.2210345641, + "update_s": 263826.4864300136, + "delete_s": 2687521.8464886155, + "first_query_after_ms": 0.9569000103510916, + "recall_after": 0.9521 + }, + "hnswlib_100000": { + "insert_s": 6438.298916415511, + "update_s": 1566.8058763162978, + "delete_s": 375037.50368665985, + "first_query_after_ms": 0.9597000316716731, + "recall_after": 0.9675 + }, + "usearch_100000": { + "insert_s": 7692.72073200578, + "update_s": 7682.160149535157, + "delete_s": 3068896.7171558207, + "first_query_after_ms": 3.66389995906502, + "recall_after": 0.9573 + }, + "faiss_hnsw_100000": { + "insert_s": 4351.938479551619, + "update_s": null, + "delete_s": null, + "first_query_after_ms": null, + "recall_after": null + }, + "rust_ivf_1000000": { + "insert_s": 137935.790889341, + "update_s": 133013.39178828525, + "delete_s": 1097249.1962649636, + "first_query_after_ms": 7.959499999999999, + "recall_after": 0.9530413625304136 + }, + "rust_hnsw_1000000": { + "insert_s": 1392.9937512668407, + "update_s": 1439.8539596444643, + "delete_s": 1382170.00691085, + "first_query_after_ms": 0.5709000000000001, + "recall_after": 0.955 + }, + "faiss_ivfflat_1000000": { + "insert_s": 86801.63743997154, + "update_s": 68770.82466479427, + "delete_s": 1236032.8258859534, + "first_query_after_ms": 8.086899993941188, + "recall_after": 0.9571000000000001 + }, + "hnswlib_1000000": { + "insert_s": 6004.713820335999, + "update_s": 1695.5782540630792, + "delete_s": 355493.62078393926, + "first_query_after_ms": 1.2923000031150877, + "recall_after": 0.9557 + }, + "usearch_1000000": { + "insert_s": 3788.6844573857975, + "update_s": 3414.1150510167176, + "delete_s": 1913729.0804453953, + "first_query_after_ms": 27.772400004323572, + "recall_after": 0.9562 + }, + "faiss_hnsw_1000000": { + "insert_s": 2346.823642249814, + "update_s": null, + "delete_s": null, + "first_query_after_ms": null, + "recall_after": null + } + }, + "q7": { + "rust_exact_100000": { + "after_build_mb": 163.6171875, + "peak_mb": 188.37890625, + "x_raw": 1.11696 + }, + "rust_ivf_100000": { + "after_build_mb": 166.296875, + "peak_mb": 245.6328125, + "x_raw": 1.1352533333333332 + }, + "rust_hnsw_100000": { + "after_build_mb": 198.859375, + "peak_mb": 200.8046875, + "x_raw": 1.3575466666666667 + }, + "rust_ivf_i8x4_100000": { + "after_build_mb": 204.7578125, + "peak_mb": 281.90234375, + "x_raw": 1.3978133333333334 + }, + "rust_hnsw_i8x4_100000": { + "after_build_mb": 237.828125, + "peak_mb": 239.25, + "x_raw": 1.6235733333333333 + }, + "faiss_flat_100000": { + "after_build_mb": 173.92578125, + "peak_mb": 319.47265625, + "x_raw": 1.1873333333333334 + }, + "numpy_bf_100000": { + "after_build_mb": 159.1953125, + "peak_mb": 190.80859375, + "x_raw": 1.0867733333333334 + }, + "faiss_ivfflat_100000": { + "after_build_mb": 240.21484375, + "peak_mb": 460.76171875, + "x_raw": 1.6398666666666666 + }, + "faiss_hnsw_100000": { + "after_build_mb": 206.84765625, + "peak_mb": 348.25, + "x_raw": 1.41208 + }, + "hnswlib_100000": { + "after_build_mb": 200.890625, + "peak_mb": 230.671875, + "x_raw": 1.3714133333333334 + }, + "usearch_100000": { + "after_build_mb": 181.54296875, + "peak_mb": 211.49609375, + "x_raw": 1.2393333333333334 + }, + "rust_exact_int8_100000": { + "after_build_mb": 202.8828125, + "x_raw": 1.3850133333333334 + }, + "rust_exact_1000000": { + "after_build_mb": 1637.4296875, + "peak_mb": 1648.5234375, + "x_raw": 1.1178186666666667 + }, + "rust_ivf_1000000": { + "after_build_mb": 1643.0859375, + "peak_mb": 1886.44921875, + "x_raw": 1.12168 + }, + "rust_hnsw_1000000": { + "after_build_mb": 1785.55078125, + "peak_mb": 1793.5625, + "x_raw": 1.218936 + }, + "rust_ivf_i8x4_1000000": { + "after_build_mb": 2011.3203125, + "peak_mb": 2254.81640625, + "x_raw": 1.3730613333333332 + }, + "rust_hnsw_i8x4_1000000": { + "after_build_mb": 2155.71875, + "peak_mb": 2163.75390625, + "x_raw": 1.4716373333333332 + }, + "faiss_flat_1000000": { + "after_build_mb": 1555.1484375, + "peak_mb": 2123.75390625, + "x_raw": 1.061648 + }, + "numpy_bf_1000000": { + "after_build_mb": 1575.796875, + "peak_mb": 1604.9453125, + "x_raw": 1.075744 + }, + "faiss_ivfflat_1000000": { + "after_build_mb": 1881.52734375, + "peak_mb": 3588.3359375, + "x_raw": 1.284456 + }, + "faiss_hnsw_1000000": { + "after_build_mb": 1815.57421875, + "peak_mb": 2297.890625, + "x_raw": 1.239432 + }, + "hnswlib_1000000": { + "after_build_mb": 1876.32421875, + "peak_mb": 1906.10546875, + "x_raw": 1.280904 + }, + "usearch_1000000": { + "after_build_mb": 1713.94140625, + "peak_mb": 1743.9375, + "x_raw": 1.1700506666666666 + }, + "rust_exact_int8_1000000": { + "after_build_mb": 1997.8046875, + "x_raw": 1.3638346666666668 + } + } +} \ No newline at end of file diff --git a/benchmarks/rust-stage-c/tables.md b/benchmarks/rust-stage-c/tables.md new file mode 100644 index 0000000..946a4dc --- /dev/null +++ b/benchmarks/rust-stage-c/tables.md @@ -0,0 +1,233 @@ +## Q1. Exact scan: ISONGraph Rust vs numpy vs faiss IndexFlatIP (single thread, one pinned P-core) + +QPS, median over runs (min-max). `ratio` = Rust / other, > 1 means Rust is faster. Rounds were interleaved (Rust, numpy, faiss, Rust, ...) in one session. + +| cell | runs | Rust exact | numpy | faiss Flat | Rust/numpy | Rust/faiss | Rust auto-thread loop | faiss Flat batch (32 thr) | numpy batch | Rust batch | +|---|---|---|---|---|---|---|---|---|---|---| +| text384 100,000 | 3/3/3 | 142.6 (114.0-145.7) | 112.7 (112.2-121.3) | 117.8 (116.5-135.5) | 1.27x | 1.21x | 269 (216-284) | 5,085 (3,070-6,552) | 673 (543-1,052) | 5,320 (5,206-12,743) | +| text384 1,000,000 | 1/1/1 | 11.1 (1 run) | 11.1 (1 run) | 11.4 (1 run) | 1.00x | 0.97x | 25 (1 run) | 298 (1 run) | 61 (1 run) | 560 (1 run) | +| glove 100,000 | 3/3/3 | 548.2 (537.1-548.4) | 377.0 (323.4-456.6) | 605.0 (603.5-621.6) | 1.45x | 0.91x | 1,568 (708-1,669) | 14,934 (5,444-15,184) | 857 (783-1,434) | 19,239 (10,552-25,285) | +| sift 100,000 | 3/3/3 | 396.9 (350.0-470.5) | 385.5 (313.9-389.3) | 431.7 (421.5-521.6) | 1.03x | 0.92x | 722 (645-1,546) | 5,898 (5,075-14,071) | 1,364 (786-1,512) | 11,457 (10,910-28,197) | + +## Q2. Approximate search at recall@10 = 0.95, single thread (pinned), text384 + +QPS interpolated (log-linear) between the two sweep points that bracket recall 0.95; `-` = not bracketed by the sweep (recall 0.95 not reached, or the lowest point already above it: then the lowest measured point is shown in the `first point` column). Build = ingest + index build on the library's own path with 32 threads. Memory = working set after the build minus baseline. + +### text384 100,000 + +| system | runs | QPS @0.95 (1T) | knob at first recall>=0.95 | recall at that knob | p99 ms there | build s | of which index s | RSS after build MB | peak MB | first point (knob: recall, QPS) | best recall reached | +|---|---|---|---|---|---|---|---|---|---|---|---| +| ISONGraph Rust HNSW | 3 | 3,893 (3,314-4,486) | 128 | 0.9626 | 0.47 | 7.1 (6.9-7.4) | 7.0 (6.8-7.3) | 199 (198-200) | 201 (201-201) | 24: 0.782, 8,823 | 0.9936 | +| ISONGraph Rust IVF | 3 | 759 (752-968) | 128 | 0.9559 | 1.81 | 0.6 (0.6-0.6) | 0.5 (0.5-0.5) | 166 (166-166) | 246 (246-246) | 16: 0.750, 5,848 | 0.9952 | +| ISONGraph Rust IVF + int8 x4 | 1 | 2,059 (1 run) | 128 | 0.9559 | 0.81 | 0.6 (1 run) | 0.5 (1 run) | 205 (1 run) | 282 (1 run) | 16: 0.750, 7,552 | 0.9953 | +| ISONGraph Rust HNSW + int8 x4 (setting on) | 1 | 4,571 (1 run) | 128 | 0.9626 | 0.47 | 5.7 (1 run) | 5.6 (1 run) | 238 (1 run) | 239 (1 run) | 64: 0.914, 7,034 | 0.9796 | +| hnswlib | 3 | 2,723 (2,480-2,751) | 128 | 0.9631 | 0.66 | 8.4 (8.0-8.7) | 0.0 (0.0-0.0) | 201 (201-201) | 231 (231-231) | 24: 0.785, 7,766 | 0.9930 | +| faiss HNSWFlat | 3 | 3,965 (3,400-4,117) | 96 | 0.9639 | 0.48 | 9.6 (7.6-12.2) | 0.0 (0.0-0.0) | 207 (206-207) | 348 (347-349) | 24: 0.810, 8,382 | 0.9971 | +| usearch | 3 | 2,070 (2,012-2,174) | 128 | 0.9634 | 0.89 | 6.8 (6.1-8.2) | 0.0 (0.0-0.0) | 182 (180-183) | 211 (210-213) | 24: 0.781, 6,608 | 0.9927 | +| faiss IVFFlat | 3 | 1,065 (1,028-1,096) | 128 | 0.9553 | 1.76 | 1.5 (1.4-1.6) | 1.5 (1.4-1.6) | 240 (240-242) | 461 (461-463) | 16: 0.751, 7,235 | 0.9958 | +| ISONGraph Rust exact (exact, recall 1.0) | 3 | 142.6 (114.0-145.7) | - | 1.000 | 10.58 | 0.10 (0.10-0.11) | 0 | - | - | - | 1.000 | +| faiss IndexFlatIP (exact, recall 1.0) | 3 | 117.8 (116.5-135.5) | - | 1.000 | 11.04 | 0.11 (0.10-0.13) | 0 | 174 (174-174) | - | - | 1.000 | + +### text384 1,000,000 + +| system | runs | QPS @0.95 (1T) | knob at first recall>=0.95 | recall at that knob | p99 ms there | build s | of which index s | RSS after build MB | peak MB | first point (knob: recall, QPS) | best recall reached | +|---|---|---|---|---|---|---|---|---|---|---|---| +| ISONGraph Rust HNSW | 1 | 1,813 (1 run) | 192 | 0.9549 | 0.90 | 172.7 (1 run) | 171.6 (1 run) | 1,786 (1 run) | 1,794 (1 run) | 48: 0.834, 4,599 | 0.9901 | +| ISONGraph Rust IVF | 1 | 137 (1 run) | 256 | 0.9526 | 9.32 | 7.6 (1 run) | 6.5 (1 run) | 1,643 (1 run) | 1,886 (1 run) | 16: 0.699, 1,412 | 0.9772 | +| ISONGraph Rust IVF + int8 x4 | 1 | 235 (1 run) | 256 | 0.9532 | 6.22 | 8.1 (1 run) | 6.9 (1 run) | 2,011 (1 run) | 2,255 (1 run) | 64: 0.858, 771 | 0.9780 | +| ISONGraph Rust HNSW + int8 x4 (setting on) | 1 | - | 192 | 0.9549 | 0.83 | 145.9 (1 run) | 144.7 (1 run) | 2,156 (1 run) | 2,164 (1 run) | 192: 0.955, 1,784 | 0.9685 | +| hnswlib | 1 | 1,517 (1 run) | 192 | 0.9552 | 0.99 | 157.2 (1 run) | 0.0 (1 run) | 1,876 (1 run) | 1,906 (1 run) | 24: 0.728, 6,005 | 0.9817 | +| faiss HNSWFlat | 1 | 1,773 (1 run) | 192 | 0.9668 | 1.03 | 189.5 (1 run) | 0.0 (1 run) | 1,815 (1 run) | 2,298 (1 run) | 24: 0.760, 6,168 | 0.9885 | +| usearch | 1 | 612 (1 run) | 192 | 0.9560 | 3.17 | 215.8 (1 run) | 0.0 (1 run) | 1,714 (1 run) | 1,744 (1 run) | 24: 0.726, 2,924 | 0.9818 | +| faiss IVFFlat | 1 | 180 (1 run) | 256 | 0.9561 | 15.10 | 35.3 (1 run) | 35.3 (1 run) | 1,881 (1 run) | 3,588 (1 run) | 16: 0.703, 1,561 | 0.9801 | +| ISONGraph Rust exact (exact, recall 1.0) | 1 | 11.1 (1 run) | - | 1.000 | 99.27 | 1.14 (1 run) | 0 | - | - | - | 1.000 | +| faiss IndexFlatIP (exact, recall 1.0) | 1 | 11.4 (1 run) | - | 1.000 | 95.71 | 1.05 (1 run) | 0 | 1,555 (1 run) | - | - | 1.000 | + +### Sweeps (median over runs; recall / QPS 1T) + +**text384 100,000** + +* ISONGraph Rust HNSW (ef_search): 24: 0.782 / 10,824; 48: 0.885 / 7,752; 64: 0.914 / 4,799; 96: 0.948 / 3,967; 128: 0.963 / 3,526; 192: 0.980 / 2,502; 256: 0.986 / 1,521; 400: 0.994 / 1,051 +* ISONGraph Rust IVF (nprobe): 16: 0.750 / 5,848; 32: 0.840 / 3,335; 64: 0.908 / 1,385; 128: 0.956 / 697; 256: 0.984 / 360; 512: 0.995 / 182 +* ISONGraph Rust IVF + int8 x4 (nprobe): 16: 0.750 / 7,552; 32: 0.840 / 5,316; 64: 0.908 / 3,251; 128: 0.956 / 1,930; 256: 0.984 / 847; 512: 0.995 / 503 +* ISONGraph Rust HNSW + int8 x4 (setting on) (ef_search): 64: 0.914 / 7,034; 96: 0.948 / 4,773; 128: 0.963 / 3,642; 192: 0.980 / 2,728 +* hnswlib (ef): 24: 0.786 / 7,766; 32: 0.831 / 6,152; 48: 0.885 / 4,375; 64: 0.914 / 3,843; 96: 0.949 / 2,789; 128: 0.963 / 2,175; 192: 0.980 / 1,572; 256: 0.986 / 1,243; 400: 0.993 / 813 +* faiss HNSWFlat (efSearch): 24: 0.811 / 8,382; 32: 0.857 / 8,922; 48: 0.907 / 6,351; 64: 0.936 / 4,365; 96: 0.964 / 3,579; 128: 0.975 / 2,808; 192: 0.987 / 1,945; 256: 0.992 / 1,518; 400: 0.997 / 965 +* usearch (expansion_search): 24: 0.781 / 6,608; 32: 0.828 / 4,933; 48: 0.882 / 4,000; 64: 0.915 / 3,141; 96: 0.948 / 2,145; 128: 0.963 / 1,780; 192: 0.978 / 1,183; 256: 0.986 / 893; 400: 0.992 / 617 +* faiss IVFFlat (nprobe): 16: 0.751 / 6,404; 32: 0.840 / 3,363; 64: 0.907 / 1,885; 128: 0.955 / 985; 256: 0.986 / 483; 512: 0.996 / 266 + +**text384 1,000,000** + +* ISONGraph Rust HNSW (ef_search): 48: 0.834 / 4,599; 64: 0.868 / 4,306; 96: 0.906 / 2,898; 128: 0.929 / 2,312; 192: 0.955 / 1,714; 256: 0.969 / 1,357; 400: 0.983 / 961; 640: 0.990 / 638 +* ISONGraph Rust IVF (nprobe): 16: 0.699 / 1,412; 32: 0.784 / 829; 64: 0.858 / 444; 128: 0.916 / 244; 256: 0.953 / 131; 512: 0.977 / 67 +* ISONGraph Rust IVF + int8 x4 (nprobe): 64: 0.858 / 771; 128: 0.915 / 424; 256: 0.953 / 223; 512: 0.978 / 121 +* ISONGraph Rust HNSW + int8 x4 (setting on) (ef_search): 192: 0.955 / 1,784; 256: 0.969 / 1,371 +* hnswlib (ef): 24: 0.728 / 6,005; 32: 0.774 / 6,181; 48: 0.832 / 4,509; 64: 0.868 / 3,471; 96: 0.905 / 2,659; 128: 0.928 / 1,772; 192: 0.955 / 1,461; 256: 0.967 / 1,136; 400: 0.982 / 760 +* faiss HNSWFlat (efSearch): 24: 0.760 / 6,168; 32: 0.802 / 5,394; 48: 0.859 / 4,129; 64: 0.892 / 3,336; 96: 0.929 / 2,386; 128: 0.947 / 1,872; 192: 0.967 / 1,357; 256: 0.977 / 1,117; 400: 0.989 / 709 +* usearch (expansion_search): 24: 0.726 / 2,924; 32: 0.773 / 2,408; 48: 0.830 / 1,762; 64: 0.866 / 1,409; 96: 0.910 / 1,051; 128: 0.931 / 803; 192: 0.956 / 561; 256: 0.968 / 432; 400: 0.982 / 288 +* faiss IVFFlat (nprobe): 16: 0.703 / 1,561; 32: 0.792 / 905; 64: 0.863 / 567; 128: 0.918 / 283; 256: 0.956 / 165; 512: 0.980 / 100 + +## Q3. Batch / all-core throughput, text384 100k (32 logical CPUs, 1,000 queries per batch, k=10) + +Batch QPS is the median of 3 timed batches after a warm-up batch, at the sweep point nearest recall 0.95 (first knob with recall >= 0.95; its recall is shown). Exact rows are recall 1.0. + +| system | runs | knob | recall there | batch QPS (32 thr) | 1T QPS there | batch / 1T | +|---|---|---|---|---|---|---| +| **text384 100,000** | | | | | | | +| ISONGraph Rust exact | 3 | - | 1.000 | 5,320 (5,206-12,743) | 143 (114-146) | 37x | +| faiss IndexFlatIP | 3 | - | 1.000 | 5,085 (3,070-6,552) | 118 (117-136) | 43x | +| numpy brute force | 3 | - | 1.000 | 673 (543-1,052) | 113 (112-121) | 6x | +| ISONGraph Rust IVF | 3 | 128 | 0.956 | 24,360 (21,141-24,903) | 697 (692-889) | 35x | +| ISONGraph Rust IVF + int8 x4 | 1 | 128 | 0.956 | 11,850 (1 run) | 1,930 (1 run) | 6x | +| faiss IVFFlat | 3 | 128 | 0.955 | 3,014 (2,922-3,561) | 985 (955-1,025) | 3x | +| ISONGraph Rust HNSW | 3 | 128 | 0.963 | 13,989 (13,919-14,608) | 3,526 (2,742-3,644) | 4x | +| faiss HNSWFlat | 3 | 96 | 0.964 | 16,694 (14,638-19,818) | 3,579 (2,966-3,622) | 5x | +| hnswlib | 3 | 128 | 0.963 | 12,743 (12,607-12,757) | 2,175 (2,074-2,226) | 6x | +| usearch | 3 | 128 | 0.963 | 13,656 (13,469-15,408) | 1,780 (1,730-1,816) | 8x | +| **text384 1,000,000** | | | | | | | +| ISONGraph Rust exact | 1 | - | 1.000 | 560 (1 run) | 11 (1 run) | 51x | +| faiss IndexFlatIP | 1 | - | 1.000 | 298 (1 run) | 11 (1 run) | 26x | +| numpy brute force | 1 | - | 1.000 | 61 (1 run) | 11 (1 run) | 5x | +| ISONGraph Rust IVF | 1 | 256 | 0.953 | 10,068 (1 run) | 131 (1 run) | 77x | +| ISONGraph Rust IVF + int8 x4 | 1 | 256 | 0.953 | 6,818 (1 run) | 223 (1 run) | 31x | +| faiss IVFFlat | 1 | 256 | 0.956 | 471 (1 run) | 165 (1 run) | 3x | +| ISONGraph Rust HNSW | 1 | 192 | 0.955 | 5,583 (1 run) | 1,714 (1 run) | 3x | +| faiss HNSWFlat | 1 | 192 | 0.967 | 4,967 (1 run) | 1,357 (1 run) | 4x | +| hnswlib | 1 | 192 | 0.955 | 6,680 (1 run) | 1,461 (1 run) | 5x | +| usearch | 1 | 192 | 0.956 | 3,973 (1 run) | 561 (1 run) | 7x | + +Rust concurrent clients (each caller runs single-scan-thread searches against one shared store; 3 s per client count, not pinned), QPS: + +| cell | mode | 1 | 2 | 4 | 8 | 16 | +|---|---|---|---|---|---|---| +| text384 100,000 | ISONGraph Rust exact | 57 | 83 | 183 | 341 | 604 | +| text384 100,000 | ISONGraph Rust IVF (knob 128) | 364 | 682 | 1,292 | 2,285 | 4,371 | +| text384 100,000 | ISONGraph Rust HNSW (knob 128) | 1,601 | 2,994 | 5,728 | 10,039 | 14,916 | +| text384 1,000,000 | ISONGraph Rust exact | 5 | 8 | 17 | 37 | 60 | +| text384 1,000,000 | ISONGraph Rust IVF (knob 256) | 129 | 216 | 335 | 419 | 385 | +| text384 1,000,000 | ISONGraph Rust HNSW (knob 192) | 1,336 | 3,037 | 3,765 | 5,535 | 5,343 | + +## Q4. int8 scan with exact rerank on real embeddings (exact search, Rust) + +`recall vs f32 scan` is `int8_recall_report` (int8 top-10 vs the library's own f32 scan, first 1,000 queries; 300 at 1M); `recall vs truth` is against the float64 ground truth in the timed loop. Speedup = int8 QPS / f32 QPS from the same process (1T = pinned single thread; auto = one query at a time on the automatic thread count; batch = 1,000-query batch, 32 threads). Medians over runs where several exist. + +| dataset | factor | recall vs f32 scan (mean / worst query) | recall vs truth | f32 1T QPS | int8 1T QPS | 1T speedup | auto-thread speedup | batch speedup | exact scores | +|---|---|---|---|---|---|---|---|---|---| +| text384 100,000 | x1 | 0.9904 / 0.80 | 0.9905 | 117 | 354 | 3.03x | 4.75x | 0.92x | True | +| text384 100,000 | x2 | 1.0000 / 1.00 | 0.9999 | 117 | 359 | 3.07x | 5.23x | 0.86x | True | +| text384 100,000 | x4 | 1.0000 / 1.00 | 0.9999 | 117 | 364 | 3.12x | 5.27x | 0.78x | True | +| text384 100,000 | x8 | 1.0000 / 1.00 | 0.9999 | 117 | 369 | 3.15x | 4.32x | 0.66x | True | +| text384 100,000 | memory | int8 copy 37 MB; working-set delta 38 MB | | | | | | | | +| text384 1,000,000 | x1 | 0.9917 / 0.90 | 0.9907 | 11 | 33 | 2.97x | 3.13x | 0.98x | True | +| text384 1,000,000 | x2 | 1.0000 / 1.00 | 1.0000 | 11 | 36 | 3.25x | 3.17x | 0.98x | True | +| text384 1,000,000 | x4 | 1.0000 / 1.00 | 1.0000 | 11 | 33 | 2.96x | 3.15x | 0.96x | True | +| text384 1,000,000 | x8 | 1.0000 / 1.00 | 1.0000 | 11 | 33 | 2.95x | 3.20x | 0.93x | True | +| text384 1,000,000 | memory | int8 copy 370 MB; working-set delta 371 MB | | | | | | | | +| glove 100,000 | x1 | 0.9832 / 0.80 | 0.9832 | 546 | 1,971 | 3.61x | 3.65x | 0.61x | True | +| glove 100,000 | x2 | 1.0000 / 1.00 | 1.0000 | 546 | 1,947 | 3.57x | 3.33x | 0.51x | True | +| glove 100,000 | x4 | 1.0000 / 1.00 | 1.0000 | 546 | 1,922 | 3.52x | 3.58x | 0.41x | True | +| glove 100,000 | x8 | 1.0000 / 1.00 | 1.0000 | 546 | 1,851 | 3.39x | 2.97x | 0.39x | True | +| glove 100,000 | memory | int8 copy 10 MB; working-set delta 8 MB | | | | | | | | +| sift 100,000 | x1 | 0.9763 / 0.80 | 0.9763 | 389 | 1,206 | 3.10x | 4.07x | 0.82x | True | +| sift 100,000 | x2 | 1.0000 / 1.00 | 1.0000 | 389 | 1,191 | 3.06x | 4.50x | 0.76x | True | +| sift 100,000 | x4 | 1.0000 / 1.00 | 1.0000 | 389 | 1,170 | 3.01x | 4.80x | 0.64x | True | +| sift 100,000 | x8 | 1.0000 / 1.00 | 1.0000 | 389 | 1,177 | 3.02x | 4.31x | 0.52x | True | +| sift 100,000 | memory | int8 copy 13 MB; working-set delta 12 MB | | | | | | | | + +IVF and HNSW combined with int8 (setting `ScanPrecision::Int8 { rerank_factor: 4 }` before the build) are in the Q2 table (`+ int8 x4` rows) with the sweeps above. + +## Q5. Persistence: ISONGraph JSON sidecar vs binary snapshot vs faiss / hnswlib / usearch index files + +ISONGraph = store only (vectors + keys), index not persisted. `ready` for the ANN rows adds the index rebuild (and a first query) that every ISONGraph load needs. Library rows are native `write_index` / `save_index` / `load`; their load time is to a usable index (the graph is inside the file). + +| N | system | save s | file MB | load s | first query after load ms | time to ready s | note | +|---|---|---|---|---|---|---|---| +| 100,000 | ISONGraph Rust JSON sidecar | 1.67 | 453 | 1.27 (runs 1.27, 1.27, 1.28) | 3.8 | 1.28 | exact search ready; answers equal original: True | +| 100,000 | ISONGraph Rust binary snapshot | 0.19 (+0.00 sidecar fingerprint) | 149 | 0.19 (runs 0.19, 0.19, 0.20) | 4.0 | 0.20 | exact search ready; answers equal original: True | +| 100,000 | ISONGraph Rust snapshot + HNSW rebuild | - | - | 0.19 | 0.2 | 6.12 | load + build 5.9 s + first query | +| 100,000 | ISONGraph Rust snapshot + IVF rebuild | - | - | 0.20 | 0.6 | 0.71 | load + build 0.5 s + first query | +| 100,000 | faiss IVFFlat | 0.061 (0.046-0.098) | 149 | 0.094 (0.087-0.170) | 1.24 | 0.095 | native index file | +| 100,000 | faiss HNSWFlat | 0.134 (0.128-0.253) | 176 | 0.091 (0.070-0.158) | 0.46 | 0.091 | native index file | +| 100,000 | hnswlib | 0.312 (0.290-0.367) | 177 | 0.595 (0.572-0.605) | 1.02 | 0.596 | native index file | +| 100,000 | usearch | 0.182 (0.160-0.331) | 177 | 0.139 (0.135-0.231) | 0.98 | 0.140 | native index file | +| 1,000,000 | ISONGraph Rust JSON sidecar | 42.86 | 4,534 | 30.12 (runs 30.12, 30.42, 26.38) | 50.3 | 30.17 | exact search ready; answers equal original: True | +| 1,000,000 | ISONGraph Rust binary snapshot | 7.68 (+0.00 sidecar fingerprint) | 1,489 | 4.56 (runs 4.40, 4.69, 4.56) | 43.6 | 4.60 | exact search ready; answers equal original: True | +| 1,000,000 | ISONGraph Rust snapshot + IVF rebuild | - | - | 4.56 | 31.1 | 21.62 | load + build 17.0 s + first query | +| 1,000,000 | faiss IVFFlat | 0.475 (1 run) | 1,478 | 0.848 (1 run) | 5.81 | 0.854 | native index file | +| 1,000,000 | faiss HNSWFlat | 1.154 (1 run) | 1,618 | 0.888 (1 run) | 1.08 | 0.889 | native index file | +| 1,000,000 | hnswlib | 2.529 (1 run) | 1,626 | 6.048 (1 run) | 3.08 | 6.051 | native index file | +| 1,000,000 | usearch | 3.247 (1 run) | 1,623 | 1.283 (1 run) | 1.80 | 1.285 | native index file | + +## Q6. Writes and the first query after writes (text384; index built, recall ~0.95 operating point) + +10k inserts, then 10k updates (new vector under an existing id), then 10k deletes, one call at a time on the built index (Rust: `add` / `remove`; libraries as in the competitive harness: calls of 1,000). Then the first search and recall@10 against the exact post-mutation float64 truth at the same knob. `p99/max` are per-write latencies (Rust only). + +| N | system | knob | insert/s | update/s | delete/s | insert p99 / max ms | update p99 / max ms | first query after ms | recall before | recall after | QPS 1T after | +|---|---|---|---|---|---|---|---|---|---|---|---| +| 100,000 | ISONGraph Rust IVF (1) | 128 | 156,140 | 128,696 | 911,295 | 0.01 / 2.1 | 0.01 / 2.2 | 3.8 | 0.9559 | 0.9529 | 857 | +| 100,000 | ISONGraph Rust HNSW (1) | 128 | 944 | 936 | 1,069,039 | 1.89 / 2.8 | 1.95 / 3.4 | 0.7 | 0.9626 | 0.9582 | 3,164 | +| 100,000 | faiss IVFFlat (3) | 128 | 235,791 | 263,826 | 2,687,522 | - | - | 1.0 | 0.9553 | 0.9521 | 1,185 | +| 100,000 | hnswlib (3) | 128 | 6,438 | 1,567 | 375,038 | - | - | 1.0 | 0.9631 | 0.9675 | 1,780 | +| 100,000 | usearch (3) | 128 | 7,693 | 7,682 | 3,068,897 | - | - | 3.7 | 0.9634 | 0.9573 | 1,494 | +| 100,000 | faiss HNSWFlat (3) | 96 | 4,352 | - (unsupported) | - (unsupported) | - | - | - | 0.9639 | - | - | +| 1,000,000 | ISONGraph Rust IVF (1) | 256 | 137,936 | 133,013 | 1,097,249 | 0.01 / 2.6 | 0.01 / 3.0 | 8.0 | 0.9526 | 0.9530 | 129 | +| 1,000,000 | ISONGraph Rust HNSW (1) | 192 | 1,393 | 1,440 | 1,382,170 | 1.49 / 4.7 | 1.35 / 6.6 | 0.6 | 0.9549 | 0.9550 | 1,603 | +| 1,000,000 | faiss IVFFlat (1) | 256 | 86,802 | 68,771 | 1,236,033 | - | - | 8.1 | 0.9561 | 0.9571 | 154 | +| 1,000,000 | hnswlib (1) | 192 | 6,005 | 1,696 | 355,494 | - | - | 1.3 | 0.9552 | 0.9557 | 1,377 | +| 1,000,000 | usearch (1) | 192 | 3,789 | 3,414 | 1,913,729 | - | - | 27.8 | 0.9560 | 0.9562 | 555 | +| 1,000,000 | faiss HNSWFlat (1) | 192 | 2,347 | - (unsupported) | - (unsupported) | - | - | - | 0.9668 | - | - | + +## Q7. Memory at 1M x 384 (working set, MB; baseline subtracted) + +Raw vectors = 1,000,000 x 384 x 4 = 1,465 MB. Each Rust row is its own process (ingest, then that mode only). Python rows: working set after the index is built and again after the sweep, with the harness' preallocation (hnswlib `max_elements` = N + 20k). + +| system | after ingest | after build / first query | peak during build | after the sweep | x raw vectors (after build) | +|---|---|---|---|---|---| +| **text384 100,000** (raw 146 MB) | | | | | | +| ISONGraph Rust exact | 161 | 164 | 188 | 166 | 1.12x | +| ISONGraph Rust IVF | 161 | 166 | 246 | 174 | 1.14x | +| ISONGraph Rust HNSW | 161 | 199 | 201 | 204 | 1.36x | +| ISONGraph Rust IVF + int8 x4 | 161 | 205 | 282 | 213 | 1.40x | +| ISONGraph Rust HNSW + int8 x4 (setting on) | 161 | 238 | 239 | 240 | 1.62x | +| faiss IndexFlatIP | - | 174 | 319 | 185 | 1.19x | +| numpy brute force | - | 159 | 191 | 168 | 1.09x | +| faiss IVFFlat | - | 240 | 461 | 232 | 1.64x | +| faiss HNSWFlat | - | 207 | 348 | 209 | 1.41x | +| hnswlib | - | 201 | 231 | 204 | 1.37x | +| usearch | - | 182 | 211 | 184 | 1.24x | +| ISONGraph Rust exact + int8 copy (any factor) | - | 203 | - | - | 1.39x | +| **text384 1,000,000** (raw 1,465 MB) | | | | | | +| ISONGraph Rust exact | 1,621 | 1,637 | 1,649 | 1,642 | 1.12x | +| ISONGraph Rust IVF | 1,621 | 1,643 | 1,886 | 1,653 | 1.12x | +| ISONGraph Rust HNSW | 1,621 | 1,786 | 1,794 | 1,795 | 1.22x | +| ISONGraph Rust IVF + int8 x4 | 1,621 | 2,011 | 2,255 | 2,058 | 1.37x | +| ISONGraph Rust HNSW + int8 x4 (setting on) | 1,622 | 2,156 | 2,164 | 2,156 | 1.47x | +| faiss IndexFlatIP | - | 1,555 | 2,124 | 1,565 | 1.06x | +| numpy brute force | - | 1,576 | 1,605 | 1,584 | 1.08x | +| faiss IVFFlat | - | 1,882 | 3,588 | 1,884 | 1.28x | +| faiss HNSWFlat | - | 1,816 | 2,298 | 1,827 | 1.24x | +| hnswlib | - | 1,876 | 1,906 | 1,878 | 1.28x | +| usearch | - | 1,714 | 1,744 | 1,716 | 1.17x | +| ISONGraph Rust exact + int8 copy (any factor) | - | 1,998 | - | - | 1.36x | + +Rust library-reported breakdown (`memory_stats`, 1M): + +* ISONGraph Rust IVF: matrix 1,466 MB, metadata 124 MB, index 17 MB, int8 copy 0 MB +* ISONGraph Rust HNSW: matrix 1,466 MB, metadata 124 MB, index 139 MB, int8 copy 0 MB +* ISONGraph Rust IVF + int8 x4: matrix 1,466 MB, metadata 124 MB, index 17 MB, int8 copy 370 MB +* ISONGraph Rust HNSW + int8 x4 (setting on): matrix 1,466 MB, metadata 124 MB, index 139 MB, int8 copy 370 MB + +## Appendix: noisy runs and machine load at job start + +| job file | total CPU % | pinned-CPU % | flagged noisy | +|---|---|---|---| +| faiss_flat_glove_100000_r1.json | 19 | 77 | True | +| faiss_flat_glove_100000_r3.json | 18 | 95 | True | +| faiss_flat_sift_100000_r2.json | 17 | 96 | True | +| faiss_hnsw_text384_100000_r2.json | 13 | 69 | True | +| faiss_ivfflat_text384_100000_r2.json | 19 | 69 | True | +| faiss_ivfflat_text384_100000_r3.json | 21 | 72 | True | +| numpy_bf_glove_100000_r3.json | 25 | 7 | False | +| rust_exact_text384_100000_r3.json | 18 | 83 | True | +| rust_hnsw_i8x4_text384_100000_r1.json | 12 | 75 | True | +| usearch_text384_100000_r3.json | 11 | 62 | True | diff --git a/isongraph-vector-cpp/CMakeLists.txt b/isongraph-vector-cpp/CMakeLists.txt index 59ca792..d73b753 100644 --- a/isongraph-vector-cpp/CMakeLists.txt +++ b/isongraph-vector-cpp/CMakeLists.txt @@ -22,6 +22,15 @@ target_include_directories(isongraph_vector INTERFACE $<INSTALL_INTERFACE:include> ) +# The approximate index (isongraph_vector_ivf.hpp) trains on several threads, so +# link the platform thread library when building here. BUILD_INTERFACE only: an +# installed consumer on a recent libc needs nothing, and where -pthread is +# needed the header falls back to one thread instead of failing. +find_package(Threads) +if(Threads_FOUND) + target_link_libraries(isongraph_vector INTERFACE $<BUILD_INTERFACE:Threads::Threads>) +endif() + # Install rules. The interface above already declares an INSTALL_INTERFACE # include path; without these nothing was ever placed there, so a consumer # could not find_package() this library. @@ -95,6 +104,24 @@ if(ISONGRAPH_VECTOR_CURL) target_compile_definitions(isongraph_vector_curl INTERFACE ISONGRAPH_VECTOR_CURL=1) endif() +# Optional benchmark of the approximate index (bench/ann_bench.cpp). Off by +# default; it is an executable for people measuring, not part of the tests. +option(ISONGRAPH_VECTOR_BENCH "Build the approximate-index benchmark (bench/ann_bench.cpp)" OFF) +option(ISONGRAPH_VECTOR_BENCH_NATIVE + "Compile the benchmark for the build machine (AVX2/FMA or -march=native) instead of baseline SSE2" OFF) + +if(ISONGRAPH_VECTOR_BENCH) + add_executable(ann_bench bench/ann_bench.cpp) + target_link_libraries(ann_bench PRIVATE isongraph_vector) + if(ISONGRAPH_VECTOR_BENCH_NATIVE) + if(MSVC) + target_compile_options(ann_bench PRIVATE /arch:AVX2) + else() + target_compile_options(ann_bench PRIVATE -march=native) + endif() + endif() +endif() + # Tests. Skipped when this project is consumed via add_subdirectory. if(CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR) enable_testing() diff --git a/isongraph-vector-cpp/README.md b/isongraph-vector-cpp/README.md index d8b3405..4fc960e 100644 --- a/isongraph-vector-cpp/README.md +++ b/isongraph-vector-cpp/README.md @@ -295,6 +295,77 @@ The fetched SQLite targets are build-tree only and are not installed. A consumer who wants the store either builds with the option on, or links their own SQLite and includes `isongraph_vector_sqlite.hpp` directly. +## Approximate search (opt-in) + +Search is an exact scan by default. For stores past a few hundred thousand +vectors, `EmbeddingStore::enableAnn()` (or the `ann` argument of `SemanticGraph`'s +constructor and `load`) answers bounded top-k searches from an in-memory +IVF-flat index instead: the vectors are clustered into `nlist` cells and only +the `nprobe` cells nearest the query are scored. It is the same index the +Python port has, with the same policy. + +```cpp +AnnOptions ann; +ann.nprobe = 32; // nlist 0 = automatic: ~4*sqrt(N); nprobe 0 = ~5% of nlist +store.enableAnn(ann); // off by default; nothing else changes + +auto hits = store.similaritySearchVector(query, 10); // from the index +auto truth = store.similaritySearchVector(query, 10, "", 0.0f, /*exact=*/true); +RecallReport r = store.recallCheck(myQueries, 10); // recall@10 against the exact scan +store.setAnnNprobe(64); // trade speed for recall, no rebuild +AnnStats stats = store.annStats(); // nlist, pending, churn, memory, counters +``` + +It is **approximate**: a true neighbour in an unprobed cell is missed, and how +often depends on how your data clusters. Measure with `recallCheck` on real +queries before relying on a setting, and raise `nprobe` until the recall you +need holds (`nprobe >= nlist` is the exact scan). What stays the same: + +- **Scores** are the exact cosine values (the top hits are re-scored with the + exact scan's own formula), so a hit scores the same float either way. +- **`node_type`, `threshold` and `NodeFilterOptions`** apply to the probed + rows. If fewer than k survive, the probe is widened 4x and retried, ending in + the exact scan, so a rare type or a selective filter costs time, not results. +- **Never exact-by-accident:** `exact = true`, a `topK` that asks for every row, + a zero query vector, a store under `minSize` (default 50,000) and + `nprobe >= nlist` all use the plain scan. +- **Nothing is persisted.** The graph file, the embeddings sidecar and + `graphDigest()` are byte-for-byte what they are without it; the index is + rebuilt lazily from the loaded vectors on the first search. +- **Writes** are visible immediately: a new or updated row is scored exactly + until enough have piled up to fold into their cells; a delete is skipped; + when writes and deletes since training pass half the trained size the index + retrains. + +The index keeps its own contiguous copy of the vectors (`AnnStats::vectorBytes`), +so an indexed store holds them twice: 1.5 GB more at 1M x 384. It builds on all +hardware threads by default (`AnnOptions::threads`; the result does not depend on +the count) and is seeded, so the same data gives the same index however it was +inserted. Searches on one store take turns on the index. + +Measured with `bench/ann_bench.cpp` (build with `-DISONGRAPH_VECTOR_BENCH=ON`; +384-d synthetic clustered data, top-10, single-threaded queries, a 32-thread +machine that was also busy with other work, so treat the absolute times as +rough). "Flat SIMD scan" is a contiguous single-thread brute force over the same +vectors - what an exact search costs when it is written for speed; the +store's own exact scan walks a hash map and is several times slower: + +| data | N | nprobe (of nlist) | recall@10 | index | flat SIMD scan | +|---|---|---|---|---|---| +| clustered | 100k | 63 (auto, of 1265) | 1.000 | 1.9 ms | 22 ms | +| clustered | 1M | 8 (of 4000) | 0.982 | 0.66 ms | 83 ms | +| clustered | 1M | 200 (auto) | 0.995 | 8.9 ms | 83 ms | +| diffuse (hard) | 1M | 128 | 0.927 | 8.2 ms | 81 ms | +| diffuse (hard) | 1M | 200 (auto) | 0.984 | 10.2 ms | 81 ms | + +Building the index at 1M x 384 took 14-17 s on 32 threads (about 9 s with +`-DISONGRAPH_VECTOR_BENCH_NATIVE=ON`'s AVX2 build, 186 s on one thread). On hard, +diffuse data the speed-up shrinks to about 8x over a flat scan at good recall, +and below ~100k vectors a flat scan is already a few milliseconds, which is why +this is opt-in and `minSize` defaults to 50,000. Queries are memory-bound, so +AVX2 barely changes them; it mainly speeds the build. The in-memory store only: +`SqliteEmbeddingStore` has no approximate index (its `vec0` option is exact). + ## Embedding over HTTP `HttpEncoder` (`isongraph_vector_http.hpp`) is an `EmbeddingEncoder` that gets @@ -356,8 +427,15 @@ A ready-made libcurl transport, `CurlTransport`, is compiled in when default. With CMake, `-DISONGRAPH_VECTOR_CURL=ON` finds libcurl and gives you an `isongraph_vector_curl` target to link (it defines the macro). It does not follow redirects (so the key is never forwarded), allows only http and https, -and caps a response at 64 MiB. It is off by default and nothing else needs -libcurl. +sends no `Expect: 100-continue` (a large batch would otherwise wait a second +for a server that does not answer it), and caps a response at 64 MiB. It is off +by default and nothing else needs libcurl. Its tests run against a small HTTP +server inside the test process on 127.0.0.1 (OpenAI- and Ollama-shaped answers, +error statuses, retries, a stall and timeout, a redirect that must not be +followed, an oversized body, API-key redaction); no internet is used. On +Windows, `#define NOMINMAX` before including `<windows.h>`, `<winsock2.h>` or +`<curl/curl.h>` in your own code, as `windows.h`'s `min`/`max` macros break the +library headers. ## One format, six languages @@ -370,9 +448,11 @@ row written by Python byte for byte. ## Things that will bite you -- **Search is brute force**, sqlite-vec included: linear in the number of - nodes. This is built for an application's own knowledge graph, not as a - vector-database replacement. +- **Search is brute force by default**, sqlite-vec included: linear in the + number of nodes. `enableAnn()` adds an opt-in approximate index for large + in-memory stores (see above), at the price of recall you have to measure. This + is built for an application's own knowledge graph, not as a vector-database + replacement. - **One embedding per node**, from one text field or a concatenation. No multi-vector nodes, no edge embeddings. - **`MockEncoder` has no semantic structure.** Use it to test plumbing, never @@ -405,13 +485,16 @@ row written by Python byte for byte. ```bash cmake -S . -B build && cmake --build build -./build/test_isongraph_vector # 167 tests +./build/test_isongraph_vector # 188 tests, 21 of them the approximate index cmake -S . -B build -DISONGRAPH_VECTOR_SQLITE=ON && cmake --build build ./build/test_isongraph_vector_sqlite # + 37 SQLite tests cmake -S . -B build -DISONGRAPH_VECTOR_CURL=ON && cmake --build build -./build/test_isongraph_vector_curl # + 3 libcurl smoke tests (loopback only) +./build/test_isongraph_vector_curl # + 21 libcurl transport tests (loopback server, no internet) + +cmake -S . -B build -DISONGRAPH_VECTOR_BENCH=ON -DCMAKE_BUILD_TYPE=Release && cmake --build build --config Release +./build/ann_bench --sizes 10000,100000,1000000 # exact vs approximate latency and recall; --json out.jsonl ``` On a multi-config generator such as Visual Studio the binaries land in @@ -421,6 +504,13 @@ On a multi-config generator such as Visual Studio the binaries land in Nothing from 1.0.0 has shipped, so these are breaking changes made in place: +- New, opt-in and additive: `EmbeddingStore::enableAnn / disableAnn / annEnabled / + setAnnNprobe / buildAnnIndex / annStats / recallCheck / recallCheckTexts`, + the `AnnOptions`, `AnnStats` and `RecallReport` types, an `exact` flag on + `similaritySearch` / `similaritySearchVector`, and on `SemanticGraph` an + `ann` constructor / `load` argument, `similaritySearchVector`, an `exact` flag + and `recallCheck`. Existing calls behave exactly as before. + - `SemanticGraph::graph()` is `const ISONGraph&` only. Replace `graph.graph().addEdge(...)` / `removeEdge(...)` with `graph.addEdge(...)` / `graph.removeEdge(...)`; `graph().addNode/updateNode/removeNode` are gone diff --git a/isongraph-vector-cpp/bench/ann_bench.cpp b/isongraph-vector-cpp/bench/ann_bench.cpp new file mode 100644 index 0000000..823d6ce --- /dev/null +++ b/isongraph-vector-cpp/bench/ann_bench.cpp @@ -0,0 +1,334 @@ +/** + * Approximate-index benchmark for the C++ port: latency and recall@10 of the + * opt-in IVF index (EmbeddingStore::enableAnn) against the exact scan, on + * synthetic clustered 384-d unit vectors. + * + * This is the C++ counterpart of benchmarks/ann.py and uses the same two data + * regimes (the generators are not bit-identical to numpy's, so the datasets + * differ; their structure does not): + * + * clustered N/100 Gaussian clusters, noise norm 1.0: well separated topics + * diffuse N/40 clusters grouped under 20 super-topics, noise norm 2.0 and + * a super-topic spread of 1.0: hierarchical, overlapping, harder + * + * Both are synthetic. An IVF index's recall depends entirely on how the data + * clusters; use EmbeddingStore::recallCheck on your own queries before trusting + * a setting. + * + * Build (not part of the default build): + * cmake -S . -B build -DISONGRAPH_VECTOR_BENCH=ON -DCMAKE_BUILD_TYPE=Release + * cmake --build build --config Release --target ann_bench + * Add -DISONGRAPH_VECTOR_BENCH_NATIVE=ON to compile it with AVX2/FMA (or + * -march=native); the default is the portable SSE2 baseline. + * + * Run: + * ann_bench # 10k and 100k, both regimes + * ann_bench --sizes 10000,100000,1000000 # adds 1M (needs ~4 GB RAM) + * ann_bench --sizes 1000000 --regime clustered --queries 100 --json out.jsonl + * ann_bench --threads 1 # single-threaded index build + * + * Output is a markdown table per (regime, size) on stdout; --json also appends + * one JSON object per row to the given file for the benchmark wave to collect. + */ + +#include <algorithm> +#include <chrono> +#include <cmath> +#include <cstdint> +#include <cstdio> +#include <cstdlib> +#include <cstring> +#include <fstream> +#include <iostream> +#include <random> +#include <set> +#include <thread> +#include <sstream> +#include <string> +#include <vector> + +#include "../include/isongraph_vector.hpp" + +using namespace isongraph_vector; +using Clock = std::chrono::steady_clock; + +namespace { + +constexpr size_t DIM = 384; +constexpr size_t K = 10; +constexpr uint64_t SEED = 4242; + +struct Regime { + const char* name; + size_t perCluster; + double noise; + size_t supers; // 0 = none + double spread; +}; + +const Regime REGIMES[] = { + {"clustered", 100, 1.0, 0, 0.0}, + {"diffuse", 40, 2.0, 20, 1.0}, +}; + +double millis(Clock::time_point a, Clock::time_point b) { + return std::chrono::duration<double, std::milli>(b - a).count(); +} + +double percentile(std::vector<double> v, double p) { + if (v.empty()) return 0.0; + std::sort(v.begin(), v.end()); + double pos = p * (v.size() - 1); + size_t lo = static_cast<size_t>(pos), hi = std::min(v.size() - 1, lo + 1); + return v[lo] + (v[hi] - v[lo]) * (pos - lo); +} + +struct Gaussian { + std::mt19937_64 rng; + explicit Gaussian(uint64_t seed) : rng(seed) {} + double next() { + double u1 = (static_cast<double>(rng() >> 11) + 1.0) / 9007199254740993.0; + double u2 = static_cast<double>(rng() >> 11) / 9007199254740992.0; + return std::sqrt(-2.0 * std::log(u1)) * std::cos(6.283185307179586 * u2); + } +}; + +void normalise(std::vector<float>& v) { + double sq = 0; + for (float x : v) sq += double(x) * x; + double inv = sq > 0 ? 1.0 / std::sqrt(sq) : 0.0; + for (float& x : v) x = static_cast<float>(x * inv); +} + +class Generator { +public: + Generator(const Regime& regime, size_t n) : regime_(regime) { + Gaussian g(SEED ^ (n * 1000003u) ^ regime.perCluster); + size_t clusters = std::max<size_t>(8, n / regime.perCluster); + std::vector<std::vector<float>> supers; + for (size_t s = 0; s < regime.supers; ++s) supers.push_back(randomUnit(g)); + for (size_t c = 0; c < clusters; ++c) { + std::vector<float> center = randomUnit(g); + if (regime.supers) { + const auto& parent = supers[g.rng() % supers.size()]; + for (size_t d = 0; d < DIM; ++d) center[d] = parent[d] + static_cast<float>(regime.spread) * center[d]; + normalise(center); + } + centers_.push_back(std::move(center)); + } + } + + /** `count` vectors of stream `stream` (a different stream gives different vectors). */ + std::vector<std::vector<float>> sample(size_t count, uint64_t stream) const { + Gaussian g(SEED * 31 + stream); + std::vector<std::vector<float>> out; + out.reserve(count); + const double scale = regime_.noise / std::sqrt(static_cast<double>(DIM)); + for (size_t i = 0; i < count; ++i) { + const auto& center = centers_[g.rng() % centers_.size()]; + std::vector<float> v(DIM); + for (size_t d = 0; d < DIM; ++d) v[d] = center[d] + static_cast<float>(scale * g.next()); + normalise(v); + out.push_back(std::move(v)); + } + return out; + } + +private: + static std::vector<float> randomUnit(Gaussian& g) { + std::vector<float> v(DIM); + for (float& x : v) x = static_cast<float>(g.next()); + normalise(v); + return v; + } + Regime regime_; + std::vector<std::vector<float>> centers_; +}; + +struct Options { + std::vector<size_t> sizes = {10000, 100000}; + std::vector<std::string> regimes = {"clustered", "diffuse"}; + size_t queries = 100; + size_t threads = 0; + std::vector<size_t> nprobes = {1, 2, 4, 8, 16, 32, 64}; + std::string json; +}; + +std::vector<size_t> parseList(const std::string& text) { + std::vector<size_t> out; + std::stringstream ss(text); + std::string item; + while (std::getline(ss, item, ',')) out.push_back(static_cast<size_t>(std::stoull(item))); + return out; +} + +Options parse(int argc, char** argv) { + Options o; + for (int i = 1; i < argc; ++i) { + std::string a = argv[i]; + auto next = [&]() -> std::string { + if (i + 1 >= argc) { std::cerr << "missing value for " << a << "\n"; std::exit(2); } + return argv[++i]; + }; + if (a == "--sizes") o.sizes = parseList(next()); + else if (a == "--regime") { + std::string r = next(); + o.regimes = r == "both" ? std::vector<std::string>{"clustered", "diffuse"} : std::vector<std::string>{r}; + } + else if (a == "--queries") o.queries = static_cast<size_t>(std::stoull(next())); + else if (a == "--threads") o.threads = static_cast<size_t>(std::stoull(next())); + else if (a == "--nprobe") o.nprobes = parseList(next()); + else if (a == "--json") o.json = next(); + else if (a == "--help" || a == "-h") { + std::cout << "ann_bench [--sizes 10000,100000] [--regime clustered|diffuse|both] [--queries 100]\n" + " [--threads N] [--nprobe 1,2,4,...] [--json file]\n"; + std::exit(0); + } + else { std::cerr << "unknown argument " << a << "\n"; std::exit(2); } + } + return o; +} + +const Regime& regimeNamed(const std::string& name) { + for (const auto& r : REGIMES) if (name == r.name) return r; + std::cerr << "unknown regime " << name << "\n"; + std::exit(2); +} + +} // namespace + +int main(int argc, char** argv) { + Options opt = parse(argc, argv); + std::ofstream json; + if (!opt.json.empty()) json.open(opt.json, std::ios::app); + + std::cout << "# ann_bench: IVF-flat vs exact scan, dim " << DIM << ", top_k " << K << ", " + << opt.queries << " queries\n"; + std::cout << "SIMD backend lanes: " << detail::simd::W << " floats; hardware threads: " + << std::thread::hardware_concurrency() << "; index build threads: " + << (opt.threads ? std::to_string(opt.threads) : std::string("auto")) << "\n\n"; + + for (const std::string& regimeName : opt.regimes) { + const Regime& regime = regimeNamed(regimeName); + for (size_t n : opt.sizes) { + std::cout << "## " << regime.name << ", N = " << n << "\n\n"; + Generator gen(regime, n); + + EmbeddingStore store; + AnnOptions ann; + ann.minSize = 0; + ann.threads = opt.threads; + store.enableAnn(ann); + + // The same vectors, kept flat for the contiguous brute-force baseline. + std::vector<float> flat; + flat.reserve(n * DIM); + double added = 0; + const size_t chunk = 5000; + for (size_t start = 0; start < n; start += chunk) { + size_t count = std::min(chunk, n - start); + auto vectors = gen.sample(count, 1000 + start / chunk); + auto a = Clock::now(); + for (size_t i = 0; i < count; ++i) { + store.add(NodeRef("doc", std::to_string(start + i)), "", vectors[i]); + } + added += millis(a, Clock::now()); + for (const auto& v : vectors) flat.insert(flat.end(), v.begin(), v.end()); + } + std::cout << "* ingest: " << static_cast<size_t>(n / (added / 1000.0)) << " vectors/s (index not built yet)\n"; + + auto queries = gen.sample(opt.queries, 7); + + // exact scan: warm latency, and the ground truth for recall + std::vector<std::set<std::string>> truth(queries.size()); + std::vector<double> exactMs; + store.similaritySearchVector(queries[0], K, "", 0.0f, true); // warm up + for (size_t q = 0; q < queries.size(); ++q) { + auto a = Clock::now(); + auto res = store.similaritySearchVector(queries[q], K, "", 0.0f, true); + auto b = Clock::now(); + exactMs.push_back(millis(a, b)); + for (const auto& r : res) truth[q].insert(r.node_ref.id); + } + double exactMedian = percentile(exactMs, 0.5); + std::cout << "* exact scan: median " << exactMedian << " ms, p95 " << percentile(exactMs, 0.95) << " ms\n"; + + // Contiguous single-threaded SIMD scan of the same vectors (they are + // unit length, so the dot product is the cosine): what an exact + // search costs when it is written for speed rather than as a loop + // over a hash map. This is the fair yardstick for the index. + std::vector<double> flatMs; + size_t flatAgree = 0; + for (size_t q = 0; q < queries.size(); ++q) { + auto a = Clock::now(); + std::vector<detail::IvfHit> heap; + for (size_t row = 0; row < n; ++row) { + float s = detail::simd::dot(flat.data() + row * DIM, queries[q].data(), DIM); + detail::IvfHit hit{s, static_cast<uint32_t>(row)}; + if (heap.size() < K) { heap.push_back(hit); std::push_heap(heap.begin(), heap.end(), detail::hitBefore); } + else if (detail::hitBefore(hit, heap.front())) { + std::pop_heap(heap.begin(), heap.end(), detail::hitBefore); + heap.back() = hit; + std::push_heap(heap.begin(), heap.end(), detail::hitBefore); + } + } + flatMs.push_back(millis(a, Clock::now())); + for (const auto& h : heap) flatAgree += truth[q].count(std::to_string(h.slot)); + } + double flatMedian = percentile(flatMs, 0.5); + std::cout << "* flat SIMD scan (contiguous, one thread, no index): median " << flatMedian + << " ms, p95 " << percentile(flatMs, 0.95) << " ms (agrees with exact on " + << flatAgree << "/" << queries.size() * K << " top-10 entries)\n"; + + AnnStats built = store.buildAnnIndex(); + std::cout << "* index build: " << built.buildSeconds << " s, nlist " << built.nlist + << " (automatic nprobe " << built.nprobe << "), index overhead " + << built.indexBytes / 1048576.0 << " MB plus a " << built.vectorBytes / 1048576.0 + << " MB contiguous copy of the vectors\n\n"; + + std::cout << "| nprobe | % of cells | recall@10 mean | worst | median ms | p95 ms | vs exact scan | vs flat SIMD scan |\n" + "|---|---|---|---|---|---|---|---|\n"; + std::vector<size_t> probes = opt.nprobes; + probes.push_back(0); // the automatic default + for (size_t probe : probes) { + if (probe != 0 && probe >= built.nlist) continue; + store.setAnnNprobe(probe); + size_t effective = store.annStats().nprobe; + // one untimed pass so the first measured query is not a cold one + store.similaritySearchVector(queries[0], K, "", 0.0f); + std::vector<double> ms, recalls; + for (size_t q = 0; q < queries.size(); ++q) { + auto a = Clock::now(); + auto res = store.similaritySearchVector(queries[q], K, "", 0.0f); + auto b = Clock::now(); + ms.push_back(millis(a, b)); + size_t hit = 0; + for (const auto& r : res) hit += truth[q].count(r.node_ref.id); + recalls.push_back(truth[q].empty() ? 1.0 : double(hit) / truth[q].size()); + } + double mean = 0, worst = 1; + for (double r : recalls) { mean += r; worst = std::min(worst, r); } + mean /= recalls.size(); + double median = percentile(ms, 0.5), p95 = percentile(ms, 0.95); + char line[256]; + std::snprintf(line, sizeof line, "| %zu%s | %.1f | %.3f | %.2f | %.2f | %.2f | %.1fx | %.1fx |", + effective, probe == 0 ? " (auto)" : "", + 100.0 * effective / built.nlist, mean, worst, median, p95, exactMedian / median, + flatMedian / median); + std::cout << line << "\n"; + if (json.is_open()) { + json << "{\"port\":\"cpp\",\"regime\":\"" << regime.name << "\",\"n\":" << n + << ",\"dim\":" << DIM << ",\"k\":" << K << ",\"queries\":" << queries.size() + << ",\"nlist\":" << built.nlist << ",\"nprobe\":" << effective + << ",\"recall_mean\":" << mean << ",\"recall_worst\":" << worst + << ",\"ann_median_ms\":" << median << ",\"ann_p95_ms\":" << p95 + << ",\"exact_median_ms\":" << exactMedian << ",\"flat_simd_median_ms\":" << flatMedian + << ",\"build_seconds\":" << built.buildSeconds + << ",\"simd_lanes\":" << detail::simd::W << "}\n"; + } + } + std::cout << "\n"; + } + } + return 0; +} diff --git a/isongraph-vector-cpp/bench/competitive_bench.cpp b/isongraph-vector-cpp/bench/competitive_bench.cpp new file mode 100644 index 0000000..ab8fadb --- /dev/null +++ b/isongraph-vector-cpp/bench/competitive_bench.cpp @@ -0,0 +1,449 @@ +/** + * Loader and measurement harness for benchmarks/competitive/ (the cross-library comparison). + * + * Reads the same raw float32 vectors, queries and float64 ground truth the Python harness feeds the + * other systems, so every system sees identical data. Bench-only: the library is not touched. + * + * Build (MSVC, from a vcvars64 prompt; header-only library): + * cl /std:c++17 /O2 /EHsc /MD /DNDEBUG [/arch:AVX2] /I include /I vendor/ison-graph-cpp + * bench/competitive_bench.cpp /Fe:competitive_bench.exe /link psapi.lib + * + * Run: + * competitive_bench --ds-dir D:/.../prep/text384 --ds text384 --n 100000 --exp core,mutation,saveload + * --queries 2000 --label cpp --out result.json + * competitive_bench ... --exp filter + * + * `core` times the exact scan (the library's hash-map loop, there is no scan cache in this port) and + * the opt-in IVF index (nprobe sweep, 3 timed runs per point). + */ +#define NOMINMAX +#include <windows.h> +#include <psapi.h> + +#include <algorithm> +#include <chrono> +#include <cmath> +#include <cstdint> +#include <cstdio> +#include <cstdlib> +#include <fstream> +#include <iostream> +#include <set> +#include <sstream> +#include <string> +#include <thread> +#include <unordered_set> +#include <vector> + +#include "../include/isongraph_vector.hpp" + +using namespace isongraph_vector; +using Clock = std::chrono::steady_clock; + +namespace { + +double now_s() { + return std::chrono::duration<double>(Clock::now().time_since_epoch()).count(); +} + +std::string fnum(double v) { + char b[64]; + std::snprintf(b, sizeof b, "%.9g", v); + return b; +} +std::string farr(const std::vector<double>& v) { + std::string s = "["; + for (size_t i = 0; i < v.size(); ++i) s += (i ? "," : "") + fnum(v[i]); + return s + "]"; +} + +std::string mem_json() { + PROCESS_MEMORY_COUNTERS pmc; + GetProcessMemoryInfo(GetCurrentProcess(), &pmc, sizeof pmc); + return "{\"rss_mb\":" + fnum(pmc.WorkingSetSize / 1048576.0) + ",\"peak_mb\":" + fnum(pmc.PeakWorkingSetSize / 1048576.0) + "}"; +} + +std::vector<float> read_f32(const std::string& path, size_t start, size_t count, size_t d) { + std::ifstream f(path, std::ios::binary); + if (!f) { std::cerr << "cannot open " << path << "\n"; std::exit(2); } + f.seekg(static_cast<std::streamoff>(start * d * 4)); + std::vector<float> v(count * d); + f.read(reinterpret_cast<char*>(v.data()), static_cast<std::streamsize>(count * d * 4)); + return v; +} +std::vector<int32_t> read_i32(const std::string& path) { + std::ifstream f(path, std::ios::binary | std::ios::ate); + if (!f) { std::cerr << "cannot open " << path << "\n"; std::exit(2); } + size_t n = static_cast<size_t>(f.tellg()) / 4; + f.seekg(0); + std::vector<int32_t> v(n); + f.read(reinterpret_cast<char*>(v.data()), static_cast<std::streamsize>(n * 4)); + return v; +} +std::vector<uint8_t> read_u8(const std::string& path) { + std::ifstream f(path, std::ios::binary | std::ios::ate); + if (!f) { std::cerr << "cannot open " << path << "\n"; std::exit(2); } + size_t n = static_cast<size_t>(f.tellg()); + f.seekg(0); + std::vector<uint8_t> v(n); + f.read(reinterpret_cast<char*>(v.data()), static_cast<std::streamsize>(n)); + return v; +} + +double pct(std::vector<double> s, double p) { + if (s.empty()) return 0; + std::sort(s.begin(), s.end()); + return s[static_cast<size_t>(std::llround((s.size() - 1) * p))]; +} +double median(const std::vector<double>& v) { return pct(v, 0.5); } + +struct Ctx { + std::string dir; + size_t d = 0, n = 0, k = 10; + std::vector<std::vector<float>> queries; + std::vector<std::vector<float>> base(size_t start, size_t count) const { + auto flat = read_f32(dir + "/base.f32", start, count, d); + std::vector<std::vector<float>> rows(count); + for (size_t i = 0; i < count; ++i) rows[i].assign(flat.begin() + i * d, flat.begin() + (i + 1) * d); + return rows; + } + std::vector<std::vector<int32_t>> gt(const std::string& kind) const { + auto flat = read_i32(dir + "/gt_" + kind + std::to_string(n) + ".i32"); + std::vector<std::vector<int32_t>> rows(flat.size() / 100); + for (size_t i = 0; i < rows.size(); ++i) rows[i].assign(flat.begin() + i * 100, flat.begin() + (i + 1) * 100); + return rows; + } +}; + +double recall(const std::vector<std::vector<size_t>>& found, const std::vector<std::vector<int32_t>>& truth, size_t k) { + size_t hit = 0; + for (size_t i = 0; i < found.size(); ++i) { + std::unordered_set<int32_t> want(truth[i].begin(), truth[i].begin() + k); + for (size_t j = 0; j < std::min(k, found[i].size()); ++j) hit += want.count(static_cast<int32_t>(found[i][j])); + } + return static_cast<double>(hit) / (k * std::max<size_t>(found.size(), 1)); +} + +enum class Mode { Exact, Ann }; + +std::vector<size_t> do_search(const EmbeddingStore& s, const std::vector<float>& q, size_t k, Mode p, const std::string& ty) { + auto r = s.similaritySearchVector(q, k, ty, 0.0f, p == Mode::Exact); + std::vector<size_t> ids; + ids.reserve(r.size()); + for (auto& x : r) ids.push_back(static_cast<size_t>(std::stoull(x.node_ref.id))); + return ids; +} + +void run_loop(const EmbeddingStore& s, const std::vector<std::vector<float>>& qs, size_t nq, size_t k, Mode p, const std::string& ty, + std::vector<double>& lat, std::vector<std::vector<size_t>>* out) { + for (size_t i = 0; i < nq; ++i) { + double t = now_s(); + auto r = do_search(s, qs[i], k, p, ty); + lat.push_back(now_s() - t); + if (out) out->push_back(std::move(r)); + } +} + +/** warm up, size the query count to `budget` seconds, then 3 timed runs. Returns a JSON object body (no braces). */ +std::string point_impl(const EmbeddingStore& s, const Ctx& c, const std::vector<std::vector<int32_t>>& truth, Mode p, const std::string& ty, + double budget, double* qps_out, double* rec_out) { + size_t nqall = c.queries.size(); + { + double w0 = now_s(); + for (size_t i = 0; i < std::min<size_t>(50, nqall); ++i) { + do_search(s, c.queries[i], c.k, p, ty); + if (now_s() - w0 > 3.0) break; + } + } + std::vector<double> probe; + run_loop(s, c.queries, std::min<size_t>(5, nqall), c.k, p, ty, probe, nullptr); + double est = std::max(median(probe), 1e-7); + size_t nq = std::min(nqall, std::max<size_t>(30, static_cast<size_t>(budget / est))); + std::vector<double> runs, lats; + double rec = 0; + for (int r = 0; r < 3; ++r) { + std::vector<double> lat; + std::vector<std::vector<size_t>> res; + run_loop(s, c.queries, nq, c.k, p, ty, lat, r == 0 ? &res : nullptr); + if (r == 0) rec = recall(res, truth, c.k); + double total = 0; + for (double x : lat) total += x; + runs.push_back(nq / total); + lats.insert(lats.end(), lat.begin(), lat.end()); + if (total > budget * 1.5) break; + } + double mean = 0; + for (double x : lats) mean += x; + mean = mean / std::max<size_t>(lats.size(), 1); + double qps = median(runs); + if (qps_out) *qps_out = qps; + if (rec_out) *rec_out = rec; + std::ostringstream o; + o << "\"nq\":" << nq << ",\"recall\":" << fnum(rec) << ",\"qps_1t\":" << fnum(qps) << ",\"qps_1t_min\":" << fnum(*std::min_element(runs.begin(), runs.end())) + << ",\"qps_1t_max\":" << fnum(*std::max_element(runs.begin(), runs.end())) << ",\"qps_1t_runs\":" << farr(runs) + << ",\"median_ms\":" << fnum(pct(lats, 0.5) * 1000) << ",\"p95_ms\":" << fnum(pct(lats, 0.95) * 1000) << ",\"p99_ms\":" << fnum(pct(lats, 0.99) * 1000) + << ",\"mean_ms\":" << fnum(mean * 1000); + return o.str(); +} + +size_t nlist_for(size_t n) { return static_cast<size_t>(std::min(std::max(std::nearbyint(4.0 * std::sqrt(static_cast<double>(n))), 16.0), 65536.0)); } +std::vector<size_t> knobs(size_t n) { + size_t nl = nlist_for(n); + std::vector<size_t> v; + for (size_t x : {1, 2, 4, 8, 16, 32, 64, 128, 256}) if (x < nl) v.push_back(x); + if (n <= 200000) v.push_back(nl); // nprobe == nlist is the exact scan; measured separately at scale + return v; +} + +double ingest(EmbeddingStore& s, const Ctx& c, size_t n, const std::vector<uint8_t>* grp) { + double secs = 0; + static const char* names[] = {"t10", "t01", "rest"}; + for (size_t at = 0; at < n; at += 20000) { + size_t m = std::min<size_t>(20000, n - at); + auto rows = c.base(at, m); + double t = now_s(); + for (size_t i = 0; i < m; ++i) { + const char* ty = grp ? names[(*grp)[at + i]] : "doc"; + s.add(NodeRef(ty, std::to_string(at + i)), "", rows[i]); + } + secs += now_s() - t; + } + return secs; +} + +/** Pin the process to one P-core (logical CPU 8 on the i9-14900HX; 0-15 are P-cores) around a single-threaded loop. + * Unpinned single-thread loops varied up to 3x between processes on this hybrid CPU. */ +struct Pin { + DWORD_PTR old = 0; + Pin() { + DWORD_PTR sys = 0; + GetProcessAffinityMask(GetCurrentProcess(), &old, &sys); + const char* e = std::getenv("BENCH_PIN_CPU"); + int cpu = e ? std::atoi(e) : 8; + SetPriorityClass(GetCurrentProcess(), HIGH_PRIORITY_CLASS); + SetProcessAffinityMask(GetCurrentProcess(), static_cast<DWORD_PTR>(1) << cpu); + } + ~Pin() { SetProcessAffinityMask(GetCurrentProcess(), old); } +}; + +std::string point(const EmbeddingStore& s, const Ctx& c, const std::vector<std::vector<int32_t>>& truth, Mode p, const std::string& ty, + double budget, double* qps_out = nullptr, double* rec_out = nullptr) { + Pin pin; + return point_impl(s, c, truth, p, ty, budget, qps_out, rec_out); +} + +std::string arg(int argc, char** argv, const std::string& name, const std::string& def) { + for (int i = 1; i + 1 < argc; ++i) if (name == argv[i]) return argv[i + 1]; + return def; +} + +} // namespace + +int main(int argc, char** argv) { + std::string dir = arg(argc, argv, "--ds-dir", ""), ds = arg(argc, argv, "--ds", ""), label = arg(argc, argv, "--label", "cpp"); + size_t n = std::stoull(arg(argc, argv, "--n", "100000")), nq = std::stoull(arg(argc, argv, "--queries", "1000")); + double budget = std::stod(arg(argc, argv, "--budget", "15")); + std::string exps = "," + arg(argc, argv, "--exp", "core") + ","; + std::string out_path = arg(argc, argv, "--out", ""); + auto has = [&](const char* e) { return exps.find(std::string(",") + e + ",") != std::string::npos; }; + + std::string meta; + { std::ifstream f(dir + "/meta.json"); std::ostringstream o; o << f.rdbuf(); meta = o.str(); } + auto pos = meta.find("\"d\":"); + Ctx c; + c.dir = dir; + c.d = std::stoull(meta.substr(pos + 4)); + c.n = n; + auto qflat = read_f32(dir + "/queries.f32", 0, nq, c.d); + for (size_t i = 0; i < nq; ++i) c.queries.emplace_back(qflat.begin() + i * c.d, qflat.begin() + (i + 1) * c.d); + unsigned cpus = std::thread::hardware_concurrency(); + std::vector<std::string> runs; + std::string mem_base = mem_json(); + std::cout << label << ": ds=" << ds << " n=" << n << " d=" << c.d << " queries=" << nq << " cpus=" << cpus << std::endl; + + auto header = [&](const std::string& sys) { + std::ostringstream o; + o << "{\"system\":\"" << sys << "\",\"ds\":\"" << ds << "\",\"n\":" << n << ",\"d\":" << c.d << ",\"k\":10,\"queries\":" << nq + << ",\"all_threads\":" << cpus << ",\"version\":\"isongraph-vector-cpp (repo HEAD, MSVC /O2)\""; + return o.str(); + }; + + if (has("core") || has("mutation") || has("saveload")) { + EmbeddingStore store; + double ingest_s = ingest(store, c, n, nullptr); + std::string mem_ing = mem_json(); + std::cout << " ingest " << ingest_s << " s" << std::endl; + auto truth = c.gt(""); + // ---- exact (hash-map scan; there is no cache in this port) ---- + double t0 = now_s(); + store.similaritySearchVector(c.queries[0], c.k, "", 0.0f, true); + double first_ms = (now_s() - t0) * 1000; + std::ostringstream ex; + ex << header(label + "_exact") << ",\"build\":{\"ingest_s\":" << fnum(ingest_s) << ",\"index_s\":0,\"total_s\":" << fnum(ingest_s) + << ",\"first_query_ms\":" << fnum(first_ms) << ",\"ready_s\":" << fnum(ingest_s + first_ms / 1000) << "},\"mem_baseline\":" << mem_base + << ",\"mem_after_ingest\":" << mem_ing << ",\"mem_after_build\":" << mem_json() << ",\"mem_after_first_query\":" << mem_json(); + double qps_ex = 0; + std::string p1 = point(store, c, truth, Mode::Exact, "", budget, &qps_ex); + std::cout << " exact qps=" << qps_ex << std::endl; + ex << ",\"sweep\":[{" << p1 << ",\"knob\":null}],\"mem_after_sweep\":" << mem_json() << "}"; + runs.push_back(ex.str()); + + // ---- ivf ---- + AnnOptions opt; + opt.minSize = 0; + store.enableAnn(opt); + double t1 = now_s(); + AnnStats st = store.buildAnnIndex(); + double idx_s = now_s() - t1; + std::cout << " ivf build " << idx_s << " s nlist=" << st.nlist << std::endl; + std::ostringstream iv; + iv << header(label + "_ivf") << ",\"knob_name\":\"nprobe\",\"build\":{\"ingest_s\":" << fnum(ingest_s) << ",\"index_s\":" << fnum(idx_s) + << ",\"total_s\":" << fnum(ingest_s + idx_s) << ",\"first_query_ms\":0,\"ready_s\":" << fnum(ingest_s + idx_s) << "},\"mem_baseline\":" << mem_base + << ",\"mem_after_ingest\":" << mem_ing << ",\"mem_after_build\":" << mem_json() << ",\"index_stats\":{\"nlist\":" << st.nlist + << ",\"trained_n\":" << st.trainedN << ",\"index_bytes\":" << st.indexBytes << ",\"vector_bytes\":" << st.vectorBytes << ",\"build_seconds\":" << fnum(st.buildSeconds) << "}"; + std::vector<std::string> pts; + long ref_knob = -1; + double ref_recall = 0; + for (size_t kn : knobs(n)) { + store.setAnnNprobe(kn); + double qps = 0, rec = 0; + std::string p = point(store, c, truth, Mode::Ann, "", budget, &qps, &rec); + std::cout << " ivf nprobe=" << kn << " recall=" << rec << " qps1t=" << qps << std::endl; + pts.push_back("{" + p + ",\"knob\":" + std::to_string(kn) + "}"); + if (ref_knob < 0 && rec >= 0.95) { ref_knob = static_cast<long>(kn); ref_recall = rec; } + } + if (ref_knob < 0) ref_knob = static_cast<long>(knobs(n).back()); + iv << ",\"sweep\":["; + for (size_t i = 0; i < pts.size(); ++i) iv << (i ? "," : "") << pts[i]; + iv << "],\"ref_knob\":" << ref_knob << ",\"ref_recall_pre\":" << fnum(ref_recall) << ",\"mem_after_sweep\":" << mem_json(); + // concurrent clients: searches take turns on the index; measured not assumed + { + store.setAnnNprobe(static_cast<size_t>(ref_knob)); + unsigned conc = std::min(8u, cpus); + size_t total = std::min<size_t>(nq, 400); + double t = now_s(); + std::vector<std::thread> th; + for (unsigned w = 0; w < conc; ++w) + th.emplace_back([&, w] { for (size_t i = w; i < total; i += conc) do_search(store, c.queries[i], c.k, Mode::Ann, ""); }); + for (auto& x : th) x.join(); + iv << ",\"concurrent_clients\":{\"clients\":" << conc << ",\"qps\":" << fnum(total / (now_s() - t)) << "}"; + } + + // ---- mutation ---- + if (has("mutation")) { + std::string gtm = dir + "/gt_mut_" + std::to_string(n) + ".i32"; + if (std::ifstream(gtm).good()) { + auto upd = read_i32(dir + "/mut_" + std::to_string(n) + "_upd.i32"); + auto del = read_i32(dir + "/mut_" + std::to_string(n) + "_del.i32"); + auto flat = read_i32(gtm); + std::vector<std::vector<int32_t>> truth_m(std::min(nq, flat.size() / 100)); + for (size_t i = 0; i < truth_m.size(); ++i) truth_m[i].assign(flat.begin() + i * 100, flat.begin() + (i + 1) * 100); + store.setAnnNprobe(static_cast<size_t>(ref_knob)); + iv << ",\"mutation\":{\"ref_knob\":" << ref_knob << ",\"recall_before\":" << fnum(ref_recall); + auto ins = c.base(n, 10000); + double t = now_s(); + for (size_t i = 0; i < ins.size(); ++i) store.add(NodeRef("doc", std::to_string(n + i)), "", ins[i]); + double dt = now_s() - t; + iv << ",\"insert_10k\":{\"seconds\":" << fnum(dt) << ",\"per_s\":" << fnum(10000 / dt) << "}"; + auto nv = c.base(n + 10000, 10000); + t = now_s(); + for (size_t j = 0; j < nv.size(); ++j) store.add(NodeRef("doc", std::to_string(upd[j])), "", nv[j]); + dt = now_s() - t; + iv << ",\"update_10k\":{\"seconds\":" << fnum(dt) << ",\"per_s\":" << fnum(10000 / dt) << "}"; + t = now_s(); + for (int32_t id : del) store.remove(NodeRef("doc", std::to_string(id))); + dt = now_s() - t; + iv << ",\"delete_10k\":{\"seconds\":" << fnum(dt) << ",\"per_s\":" << fnum(10000 / dt) << "}"; + t = now_s(); + do_search(store, c.queries[0], c.k, Mode::Ann, ""); + iv << ",\"first_query_after_ms\":" << fnum((now_s() - t) * 1000); + double qps = 0, rec = 0; + std::string p = point(store, c, truth_m, Mode::Ann, "", budget, &qps, &rec); + AnnStats s2 = store.annStats(); + iv << ",\"recall_after\":" << fnum(rec) << ",\"after\":{" << p << "},\"index_stats_after\":{\"pending\":" << s2.pending << ",\"churn\":" << s2.churn + << ",\"trained_n\":" << s2.trainedN << ",\"builds\":" << s2.builds << "},\"mem_after\":" << mem_json() << "}"; + std::cout << " mutation recall_after=" << rec << std::endl; + } else { + iv << ",\"mutation\":{\"skipped\":\"dataset has no spare rows for this n\"}"; + } + } + + // ---- save / load (JSON sidecar) ---- + if (has("saveload")) { + std::string path = dir + "/../../work/cpp_" + ds + "_" + std::to_string(n) + ".json"; + CreateDirectoryA((dir + "/../../work").c_str(), nullptr); + iv << ",\"saveload\":{"; + try { + double t = now_s(); + store.saveEmbeddings(path); + double save_s = now_s() - t; + std::ifstream f(path, std::ios::binary | std::ios::ate); + double mb = static_cast<double>(f.tellg()) / 1048576.0; + f.close(); + EmbeddingStore s2; + t = now_s(); + s2.loadEmbeddings(path); + double load_s = now_s() - t; + AnnOptions o2; + o2.minSize = 0; + o2.nprobe = static_cast<size_t>(ref_knob); + s2.enableAnn(o2); + t = now_s(); + do_search(s2, c.queries[0], c.k, Mode::Ann, ""); + double fq = (now_s() - t) * 1000; + iv << "\"save_s\":" << fnum(save_s) << ",\"disk_mb\":" << fnum(mb) << ",\"load_s\":" << fnum(load_s) << ",\"first_query_after_load_ms\":" << fnum(fq) + << ",\"time_to_ready_s\":" << fnum(load_s + fq / 1000) << ",\"note\":\"load = JSON parse + insert; first query builds the IVF index (never persisted)\""; + std::remove(path.c_str()); + } catch (const std::exception& e) { + iv << "\"error\":\"" << e.what() << "\""; + } + iv << "}"; + } + iv << "}"; + runs.push_back(iv.str()); + } + + if (has("filter")) { + auto g = read_u8(dir + "/grp_" + std::to_string(n) + ".u8"); + EmbeddingStore store; + double ingest_s = ingest(store, c, n, &g); + AnnOptions opt; + opt.minSize = 0; + store.enableAnn(opt); + double t = now_s(); + store.buildAnnIndex(); + double idx_s = now_s() - t; + std::ostringstream o; + o << header(label + "_ivf") << ",\"filter\":{\"build_with_metadata_s\":" << fnum(ingest_s + idx_s) << ",\"mem\":" << mem_json() << ",\"subsets\":{"; + auto kn_all = knobs(n); + std::set<size_t> idxs = {1, kn_all.size() / 3, kn_all.size() / 2, 2 * kn_all.size() / 3, kn_all.size() - 1}; + bool firstsub = true; + for (auto [ty, lab] : {std::pair<const char*, const char*>{"t10", "f10"}, {"t01", "f01"}}) { + auto truth = c.gt(std::string(lab) + "_"); + o << (firstsub ? "" : ",") << "\"" << lab << "\":["; + firstsub = false; + bool first = true; + for (size_t i : idxs) { + store.setAnnNprobe(kn_all[i]); + double qps = 0, rec = 0; + std::string p = point(store, c, truth, Mode::Ann, ty, budget, &qps, &rec); + std::cout << " filter " << lab << " nprobe=" << kn_all[i] << " recall=" << rec << " qps=" << qps << std::endl; + o << (first ? "" : ",") << "{" << p << ",\"knob\":" << kn_all[i] << "}"; + first = false; + } + std::string pe = point(store, c, truth, Mode::Exact, ty, budget); + o << ",{" << pe << ",\"knob\":\"exact\"}]"; + } + o << "}}}"; + runs.push_back(o.str()); + } + + if (!out_path.empty()) { + std::ofstream f(out_path); + f << "{\"status\":\"ok\",\"label\":\"" << label << "\",\"runs\":["; + for (size_t i = 0; i < runs.size(); ++i) f << (i ? "," : "") << runs[i]; + f << "]}"; + std::cout << "wrote " << out_path << std::endl; + } + return 0; +} diff --git a/isongraph-vector-cpp/include/isongraph_vector.hpp b/isongraph-vector-cpp/include/isongraph_vector.hpp index 9a4ad40..b21dffc 100644 --- a/isongraph-vector-cpp/include/isongraph_vector.hpp +++ b/isongraph-vector-cpp/include/isongraph_vector.hpp @@ -48,7 +48,9 @@ #include <chrono> #include <random> #include <cstdio> +#include <mutex> #include "ison_graph.hpp" +#include "isongraph_vector_ivf.hpp" namespace isongraph_vector { @@ -306,6 +308,12 @@ struct EmbeddingRecord { std::string text; /** Model that produced the vector: the encoder's modelName(), "provided" or "unknown". */ std::string model = "unknown"; + /** + * Internal: this record's row in the approximate index, valid only while + * an index exists (see EmbeddingStore::enableAnn). Not part of the data; + * never saved, copied between stores or compared. + */ + mutable uint32_t ann_slot = detail::IvfIndex::kNoSlot; }; // ============================================================================= @@ -1016,6 +1024,143 @@ inline std::string graphDigest( return detail::finishDigest(graph.directed, nodes, edges); } +// ============================================================================= +// Approximate search (opt-in) +// ============================================================================= + +/** + * Configuration of the opt-in IVF-flat approximate index (see + * EmbeddingStore::enableAnn). Defaults match the Python port. + */ +struct AnnOptions { + /** Cells in the index; 0 = automatic, about 4 * sqrt(N) (see autoNlist) when the index is built. */ + size_t nlist = 0; + /** Cells scanned per query; 0 = automatic, about 5% of nlist. >= nlist is the exact scan. */ + size_t nprobe = 0; + /** Below this many vectors the index is not used and search stays exact. */ + size_t minSize = 50000; + /** Seed of the k-means initialisation: the same data gives the same index. */ + uint64_t seed = 0; + /** + * Threads used to train and fill the index (0 = hardware concurrency, 1 = + * none). Queries are always single-threaded. The thread count never + * changes the resulting index. + */ + size_t threads = 0; + + /** The automatic nlist for a store of n vectors: round(4 * sqrt(n)) clamped to [16, 65536]. */ + static size_t autoNlist(size_t n) { + double wanted = std::nearbyint(4.0 * std::sqrt(static_cast<double>(n))); + return static_cast<size_t>(std::min(std::max(wanted, 16.0), 65536.0)); + } +}; + +/** What the approximate index looks like right now (EmbeddingStore::annStats). Never builds anything. */ +struct AnnStats { + bool enabled = false; + /** An index exists and will answer searches. */ + bool active = false; + size_t nlist = 0; + size_t nprobe = 0; + /** Vectors present when the centroids were trained. */ + size_t trainedN = 0; + /** Rows scored exactly on every query until folded into a cell. */ + size_t pending = 0; + /** Writes + deletes since training. */ + size_t churn = 0; + double buildSeconds = 0.0; + /** Memory held by the index on top of the vectors it scans. */ + size_t indexBytes = 0; + /** Memory of the contiguous copy of the vectors the index scans. */ + size_t vectorBytes = 0; + size_t minSize = 0; + /** Searches answered from the index, from a widened probe, by the full scan, and index builds, since enableAnn. */ + size_t approximateSearches = 0; + size_t widenedSearches = 0; + size_t exactSearches = 0; + size_t builds = 0; +}; + +/** + * Recall of the approximate index against the exact scan, over some queries + * (EmbeddingStore::recallCheck). `recall` is the mean over queries of + * |approximate top-k & exact top-k| / |exact top-k|; `minRecall` the worst + * query. `approximate` is false when the index was never consulted (it is + * off, the store is under minSize, or nprobe >= nlist), in which case recall + * is 1.0 by construction. Times are per-query medians in milliseconds. + */ +struct RecallReport { + double recall = 1.0; + double minRecall = 1.0; + size_t queries = 0; + size_t k = 0; + size_t nlist = 0; + size_t nprobe = 0; + bool approximate = false; + double annMs = 0.0; + double exactMs = 0.0; +}; + +namespace detail { + +/** + * Everything the store keeps for approximate search. Copying drops the index + * (its slot table points into the source's records) and the counters; moving + * carries the index along, since moving the record map keeps its nodes. + */ +struct AnnState { + mutable std::mutex mutex; + std::optional<AnnOptions> options; + std::unique_ptr<IvfIndex> index; + std::vector<const EmbeddingRecord*> recs; // slot -> record, null once dropped + std::unordered_map<std::string, uint32_t> codes; // node type -> small integer + std::vector<IvfHit> hits; // per-query scratch + size_t approximate = 0, widened = 0, exact = 0, builds = 0; + + AnnState() = default; + AnnState(const AnnState& other) : options(other.options) {} + AnnState(AnnState&& other) noexcept { takeFrom(other); } + AnnState& operator=(const AnnState& other) { + if (this != &other) { + std::lock_guard<std::mutex> lock(mutex); + reset(); + options = other.options; + approximate = widened = exact = builds = 0; + } + return *this; + } + AnnState& operator=(AnnState&& other) noexcept { + if (this != &other) { + std::lock_guard<std::mutex> lock(mutex); + takeFrom(other); + } + return *this; + } + + void reset() { + index.reset(); + recs.clear(); + recs.shrink_to_fit(); + codes.clear(); + hits.clear(); + } + +private: + void takeFrom(AnnState& other) { + options = std::move(other.options); + index = std::move(other.index); + recs = std::move(other.recs); + codes = std::move(other.codes); + hits = std::move(other.hits); + approximate = other.approximate; widened = other.widened; + exact = other.exact; builds = other.builds; + other.options.reset(); + other.reset(); + } +}; + +} // namespace detail + // ============================================================================= // Embedding Store // ============================================================================= @@ -1029,6 +1174,7 @@ class EmbeddingStore { ModelMismatchMode onModelMismatch_ = ModelMismatchMode::Warn; mutable std::set<std::string> warnedMismatch_; EmbeddingMeta lastImportMeta_; + mutable detail::AnnState ann_; std::string nodeRefToKey(const NodeRef& ref) const { return ref.type + ":" + ref.id; @@ -1097,6 +1243,17 @@ class EmbeddingStore { } } + // One out-of-line copy on purpose: the exact scan, the approximate path's + // re-score and the tests all compare these scores for bit equality, and + // two inlined copies are free to be contracted or vectorised differently + // by an optimiser (observed as last-bit differences on GCC 15). +#if defined(__clang__) + __attribute__((noinline)) +#elif defined(__GNUC__) + __attribute__((noinline, noclone)) +#elif defined(_MSC_VER) + __declspec(noinline) +#endif float cosineSimilarity(const std::vector<float>& v1, const std::vector<float>& v2) const { float dot = 0.0f; float norm1 = 0.0f; @@ -1115,6 +1272,205 @@ class EmbeddingStore { return dot / (norm1 * norm2); } + // -- approximate index upkeep (all of it under ann_.mutex) ----------------- + + using Ivf = detail::IvfIndex; + + uint32_t annCodeLocked(const std::string& type) const { + auto it = ann_.codes.find(type); + if (it != ann_.codes.end()) return it->second; + uint32_t code = static_cast<uint32_t>(ann_.codes.size()); + ann_.codes.emplace(type, code); + return code; + } + + size_t annThreads() const { return ann_.options ? ann_.options->threads : 1; } + + /** Cells probed at the configured nprobe. */ + size_t annProbeLocked(const Ivf& ix) const { + size_t wanted = ann_.options->nprobe; + if (wanted == 0) { + wanted = std::max<size_t>(1, static_cast<size_t>(std::nearbyint(0.05 * static_cast<double>(ix.nlist)))); + } + return std::min(wanted, ix.nlist); + } + + /** + * Cells to probe at widening `scale`; 0 = widened enough that the full scan + * costs about the same. Scale 1 honours nprobe as configured (nprobe >= + * nlist is the exact scan); later steps multiply it by 4 and give up on the + * index once they would touch half the cells. + */ + size_t annListsLocked(const Ivf& ix, size_t scale) const { + size_t lists = annProbeLocked(ix) * scale; + if (scale == 1) return lists >= ix.nlist ? 0 : lists; + return lists * 2 >= ix.nlist ? 0 : lists; + } + + void annDropIndexLocked() const { ann_.reset(); } + + void annBuildLocked() const { + std::vector<const EmbeddingRecord*> recs; + recs.reserve(embeddings_.size()); + for (const auto& entry : embeddings_) recs.push_back(&entry.second); + // Slots are numbered by key, not by hash-table order, so the same data + // gives the same index however (and on whichever standard library) it + // was inserted. + std::sort(recs.begin(), recs.end(), + [](const EmbeddingRecord* a, const EmbeddingRecord* b) { return a->id < b->id; }); + + const size_t n = recs.size(); + const size_t dim = *dimension_; + std::vector<float> data(n * dim); + std::vector<uint32_t> codes(n); + ann_.codes.clear(); + for (size_t i = 0; i < n; ++i) { + const EmbeddingRecord& r = *recs[i]; + if (r.vector.size() != dim) { + throw EmbeddingError("cannot index " + r.id + ": it has " + + std::to_string(r.vector.size()) + " dimensions, store holds " + + std::to_string(dim)); + } + std::copy(r.vector.begin(), r.vector.end(), data.begin() + static_cast<std::ptrdiff_t>(i * dim)); + codes[i] = annCodeLocked(r.node_ref.type); + r.ann_slot = static_cast<uint32_t>(i); + } + size_t nlist = ann_.options->nlist ? ann_.options->nlist : AnnOptions::autoNlist(n); + ann_.index = std::make_unique<Ivf>(Ivf::build(dim, std::move(data), std::move(codes), nlist, + ann_.options->seed, annThreads())); + ann_.recs = std::move(recs); + ++ann_.builds; + } + + /** + * The index for the store as it is now, built or maintained as the policy + * says; null means "answer exactly". Policy (as in Python): + * - built lazily by the first eligible search, never persisted; + * - new and updated rows are scored exactly until max(512, 2%) of the + * indexed rows are pending, then folded into their nearest cell; + * - deletes tombstone; a quarter dead and the arrays are compacted; + * - once writes + deletes since training exceed 50% of the trained size + * the index is retrained from the current data; + * - a store under minSize, a clear() or a dimension change drops it. + */ + Ivf* annIndexLocked() const { + if (!ann_.options) return nullptr; + const size_t live = embeddings_.size(); + if (live < std::max<size_t>(2, ann_.options->minSize) || !dimension_.has_value()) { + annDropIndexLocked(); + return nullptr; + } + if (ann_.index && static_cast<double>(ann_.index->churn) > 0.5 * static_cast<double>(ann_.index->trainedN)) { + annDropIndexLocked(); + } + if (!ann_.index) { + annBuildLocked(); + } else if (static_cast<double>(ann_.index->pendingLive) >= + std::max(512.0, 0.02 * static_cast<double>(ann_.index->trainedN))) { + ann_.index->fold(annThreads()); + } + return ann_.index.get(); + } + + /** A record was written (inserted or replaced). */ + void annAfterPut(const EmbeddingRecord& record) { + std::lock_guard<std::mutex> lock(ann_.mutex); + if (!ann_.index) return; + uint32_t code = annCodeLocked(record.node_ref.type); + if (record.ann_slot != Ivf::kNoSlot) { + ann_.index->overwrite(record.ann_slot, record.vector.data(), code); + ann_.recs[record.ann_slot] = &record; + } else { + record.ann_slot = ann_.index->append(record.vector.data(), code); + ann_.recs.push_back(&record); + } + } + + /** A record is about to be erased. */ + void annBeforeErase(const EmbeddingRecord& record) { + std::lock_guard<std::mutex> lock(ann_.mutex); + if (!ann_.index || record.ann_slot == Ivf::kNoSlot) return; + ann_.index->drop(record.ann_slot); + ann_.recs[record.ann_slot] = nullptr; + if (ann_.index->needsCompaction()) { + std::vector<uint32_t> remap = ann_.index->compact(); + std::vector<const EmbeddingRecord*> kept(ann_.index->slots(), nullptr); + for (size_t old = 0; old < remap.size(); ++old) { + if (remap[old] == Ivf::kNoSlot) continue; + kept[remap[old]] = ann_.recs[old]; + if (kept[remap[old]] != nullptr) kept[remap[old]]->ann_slot = remap[old]; + } + ann_.recs = std::move(kept); + } + } + + /** Insert or replace the record for `key`, keeping the approximate index in step. */ + void putRecord(const std::string& key, EmbeddingRecord&& record) { + auto it = embeddings_.find(key); + uint32_t slot = Ivf::kNoSlot; + if (it == embeddings_.end()) { + it = embeddings_.emplace(key, std::move(record)).first; + } else { + slot = it->second.ann_slot; + it->second = std::move(record); + } + it->second.ann_slot = slot; + annAfterPut(it->second); + } + + /** + * Top-k from the index, or nullopt to use the exact scan (counted as an + * exact search here). Filters (type, threshold) apply to the probed rows; + * if fewer than topK survive, the probe is widened 4x and tried again, and + * past half the cells the exact scan answers instead, so a rare type or a + * high threshold costs time rather than silently returning a short list. + * Scores in the result are recomputed with the exact cosine formula, so + * they equal the exact scan's bit for bit. + */ + std::optional<std::vector<SimilarityResult>> annSearch( + const std::vector<float>& query, size_t topK, const std::string& nodeType, + float threshold) const { + std::lock_guard<std::mutex> lock(ann_.mutex); + auto fallBack = [&]() -> std::optional<std::vector<SimilarityResult>> { + ++ann_.exact; + return std::nullopt; + }; + Ivf* ix = annIndexLocked(); + if (ix == nullptr) return fallBack(); + const float norm = std::sqrt(detail::simd::dot(query.data(), query.data(), query.size())); + if (!(norm > 0.0f)) return fallBack(); // a zero query has no direction to rank cells by + if (topK >= ix->live()) return fallBack(); // nothing to narrow down + + uint32_t code = Ivf::kAnyCode; + if (!nodeType.empty()) { + auto found = ann_.codes.find(nodeType); + if (found == ann_.codes.end()) return std::vector<SimilarityResult>(); // no such type at all + code = found->second; + } + // The fast scores differ from the exact ones in the last bits: admit + // rows that close to the threshold and re-check them below. + const float slack = threshold - 1e-5f; + for (size_t scale = 1;; scale *= 4) { + size_t lists = annListsLocked(*ix, scale); + if (lists == 0) return fallBack(); + ix->search(query.data(), 1.0f / norm, code, slack, lists, topK, ann_.hits); + if (ann_.hits.size() < topK) continue; + std::vector<SimilarityResult> results; + results.reserve(ann_.hits.size()); + for (const detail::IvfHit& hit : ann_.hits) { + const EmbeddingRecord& r = *ann_.recs[hit.slot]; + float score = cosineSimilarity(query, r.vector); + if (score >= threshold) results.push_back(SimilarityResult{r.node_ref, score, r.text}); + } + if (results.size() >= topK) { + std::stable_sort(results.begin(), results.end()); + ++ann_.approximate; + if (scale > 1) ++ann_.widened; + return results; + } + } + } + public: /** * @param encoder encoder for text; may be null when only vectors are added @@ -1225,8 +1581,7 @@ class EmbeddingStore { checkDimension(vec, "vector"); std::string key = nodeRefToKey(node_ref); - embeddings_[key] = EmbeddingRecord{key, node_ref, std::move(vec), text, - currentModelStamp()}; + putRecord(key, EmbeddingRecord{key, node_ref, std::move(vec), text, currentModelStamp()}); return key; } @@ -1266,27 +1621,58 @@ class EmbeddingStore { bool remove(const NodeRef& node_ref) { std::string key = nodeRefToKey(node_ref); - return embeddings_.erase(key) > 0; + auto it = embeddings_.find(key); + if (it == embeddings_.end()) return false; + annBeforeErase(it->second); + embeddings_.erase(it); + return true; } + /** + * @param exact skip the approximate index and scan every vector (no effect + * unless enableAnn() was called) + */ std::vector<SimilarityResult> similaritySearch( const std::string& query, size_t topK = 10, const std::string& nodeType = "", - float threshold = 0.0f + float threshold = 0.0f, + bool exact = false ) const { auto queryVector = encodeQuery(query); - return similaritySearchVector(queryVector, topK, nodeType, threshold); + return similaritySearchVector(queryVector, topK, nodeType, threshold, exact); } + /** + * Cosine top-k of a query vector. With enableAnn() a bounded topK on a + * large enough store is answered from the approximate index (see there); + * `exact` forces the full scan. + */ std::vector<SimilarityResult> similaritySearchVector( const std::vector<float>& queryVector, size_t topK = 10, const std::string& nodeType = "", - float threshold = 0.0f + float threshold = 0.0f, + bool exact = false ) const { checkQuery(queryVector); + if (topK > 0) { + // A store without enableAnn() only reads its (empty) options under a + // brief lock here; the scan below is what it always was. + bool enabled; + { + std::lock_guard<std::mutex> lock(ann_.mutex); + enabled = ann_.options.has_value(); + if (enabled && exact) ++ann_.exact; + } + if (enabled && !exact) { + if (auto answer = annSearch(queryVector, topK, nodeType, threshold)) { + return std::move(*answer); + } + } + } + std::vector<SimilarityResult> results; for (const auto& [key, record] : embeddings_) { @@ -1331,6 +1717,209 @@ class EmbeddingStore { return scores; } + // -- approximate (IVF) search ------------------------------------------------ + + /** + * Opt in to approximate search. Off by default, and then search is the + * exact scan, unchanged. + * + * With it on, a bounded top-k search on a store of at least + * `options.minSize` vectors is answered from an in-memory IVF-flat index: + * the vectors are clustered into `nlist` cells and only the `nprobe` cells + * nearest the query are scored. It is approximate - a true neighbour in an + * unprobed cell is missed - so measure it with recallCheck() on your own + * queries before relying on a setting, and raise nprobe until the recall + * you need holds (nprobe >= nlist is the exact scan). + * + * The index is built lazily by the first eligible search (or by + * buildAnnIndex()), kept current through add/update/remove, and never + * written to disk: the graph file, the embeddings sidecar and the graph + * digest are exactly what they are without it. Scores are the same cosine + * values the exact scan computes. searches with `exact = true`, an + * unbounded topK (one that asks for every row), a zero query vector, or a + * store under minSize use the exact scan. + * + * The index keeps its own contiguous copy of the vectors (see + * AnnStats::vectorBytes), so an indexed store holds them twice. + * Concurrent searches are safe; they take turns on the index. Calling this + * again replaces the options and drops the current index. + */ + void enableAnn(AnnOptions options = AnnOptions()) { + std::lock_guard<std::mutex> lock(ann_.mutex); + ann_.reset(); + ann_.options = options; + } + + /** Turn approximate search off and free the index. */ + void disableAnn() { + std::lock_guard<std::mutex> lock(ann_.mutex); + ann_.reset(); + ann_.options.reset(); + } + + bool annEnabled() const { + std::lock_guard<std::mutex> lock(ann_.mutex); + return ann_.options.has_value(); + } + + /** Cells scanned per query from now on (0 = automatic). Takes effect immediately; no rebuild. */ + void setAnnNprobe(size_t nprobe) { + std::lock_guard<std::mutex> lock(ann_.mutex); + if (!ann_.options) throw EmbeddingError("this store has no approximate index; call enableAnn() first"); + ann_.options->nprobe = nprobe; + } + + /** + * Build the index now instead of on the first search; with `force`, + * retrain even if one exists. Does nothing (and says so in the stats) while + * the store is under minSize. + * + * @throws EmbeddingError when enableAnn() was not called + */ + AnnStats buildAnnIndex(bool force = false) { + { + std::lock_guard<std::mutex> lock(ann_.mutex); + if (!ann_.options) throw EmbeddingError("this store has no approximate index; call enableAnn() first"); + if (force) ann_.reset(); + Ivf* ix = annIndexLocked(); + if (ix != nullptr && ix->pendingLive > 0) ix->fold(annThreads()); + } + return annStats(); + } + + /** What the approximate index looks like right now. Never builds anything. */ + AnnStats annStats() const { + std::lock_guard<std::mutex> lock(ann_.mutex); + AnnStats stats; + stats.enabled = ann_.options.has_value(); + stats.minSize = ann_.options ? ann_.options->minSize : 0; + stats.approximateSearches = ann_.approximate; + stats.widenedSearches = ann_.widened; + stats.exactSearches = ann_.exact; + stats.builds = ann_.builds; + const Ivf* ix = ann_.index.get(); + stats.active = ix != nullptr; + if (ix != nullptr) { + stats.nlist = ix->nlist; + stats.nprobe = annProbeLocked(*ix); + stats.trainedN = ix->trainedN; + stats.pending = ix->pendingLive; + stats.churn = ix->churn; + stats.buildSeconds = ix->buildSeconds; + stats.indexBytes = ix->overheadBytes(); + stats.vectorBytes = ix->vectorBytes(); + } else if (ann_.options) { + stats.nprobe = ann_.options->nprobe; + } + return stats; + } + + /** + * Recall@k of the approximate index against the exact scan: runs every + * query both ways and compares the sets of top-k nodes. Use real queries + * drawn from your own workload - recall depends on how the data clusters, + * and a score measured on unrelated queries or data does not transfer. + * + * Without an index in use (disabled, under minSize, or nprobe >= nlist) + * both answers are the exact scan, recall is 1.0 and `approximate` false. + * + * @throws EmbeddingError when k is 0 + */ + RecallReport recallCheck(const std::vector<std::vector<float>>& queries, size_t k = 10, + const std::string& nodeType = "", float threshold = 0.0f) const { + if (k == 0) throw EmbeddingError("k must be positive"); + using Clock = std::chrono::steady_clock; + std::vector<double> recalls, annTimes, exactTimes; + bool used = false; + for (const auto& query : queries) { + size_t before = annStats().approximateSearches; + auto t0 = Clock::now(); + auto found = similaritySearchVector(query, k, nodeType, threshold); + auto t1 = Clock::now(); + used = used || annStats().approximateSearches > before; + auto truth = similaritySearchVector(query, k, nodeType, threshold, /*exact=*/true); + auto t2 = Clock::now(); + annTimes.push_back(std::chrono::duration<double, std::milli>(t1 - t0).count()); + exactTimes.push_back(std::chrono::duration<double, std::milli>(t2 - t1).count()); + if (truth.empty()) continue; + std::unordered_set<std::string> wanted; + for (const auto& r : truth) wanted.insert(nodeRefToKey(r.node_ref)); + size_t hit = 0; + for (const auto& r : found) hit += wanted.count(nodeRefToKey(r.node_ref)); + recalls.push_back(static_cast<double>(hit) / static_cast<double>(wanted.size())); + } + auto median = [](std::vector<double> v) { + if (v.empty()) return 0.0; + std::sort(v.begin(), v.end()); + size_t mid = v.size() / 2; + return v.size() % 2 ? v[mid] : 0.5 * (v[mid - 1] + v[mid]); + }; + RecallReport report; + report.queries = queries.size(); + report.k = k; + report.approximate = used; + report.annMs = median(annTimes); + report.exactMs = median(exactTimes); + if (!recalls.empty()) { + report.recall = std::accumulate(recalls.begin(), recalls.end(), 0.0) / static_cast<double>(recalls.size()); + report.minRecall = *std::min_element(recalls.begin(), recalls.end()); + } + AnnStats stats = annStats(); + report.nlist = stats.nlist; + report.nprobe = stats.nprobe; + return report; + } + + /** recallCheck() with text queries (needs an encoder). */ + RecallReport recallCheckTexts(const std::vector<std::string>& queries, size_t k = 10, + const std::string& nodeType = "", float threshold = 0.0f) const { + std::vector<std::vector<float>> vectors; + vectors.reserve(queries.size()); + for (const auto& query : queries) vectors.push_back(encodeQuery(query)); + return recallCheck(vectors, k, nodeType, threshold); + } + + /** + * Visit the matches of the probed cells best first, for a caller that + * filters further (SemanticGraph's node filter). `visit` returns false to + * stop. Returns false, without visiting anything, when the index cannot + * answer at this widening `scale` (1, 4, 16, ...) and the caller should use + * the exact scan. Matches are ordered by the index's fast scores; the + * score handed to `visit` is the exact cosine. + */ + bool annVisit(const std::vector<float>& query, const std::string& nodeType, float threshold, + size_t scale, const std::function<bool(SimilarityResult&&)>& visit) const { + checkQuery(query); + std::vector<const EmbeddingRecord*> order; + { + std::lock_guard<std::mutex> lock(ann_.mutex); + Ivf* ix = annIndexLocked(); + if (ix == nullptr) return false; + const float norm = std::sqrt(detail::simd::dot(query.data(), query.data(), query.size())); + if (!(norm > 0.0f)) return false; + uint32_t code = Ivf::kAnyCode; + if (!nodeType.empty()) { + auto found = ann_.codes.find(nodeType); + if (found == ann_.codes.end()) return true; // nothing of that type exists + code = found->second; + } + size_t lists = annListsLocked(*ix, scale); + if (lists == 0) return false; + ix->candidates(query.data(), 1.0f / norm, code, threshold - 1e-5f, lists, ann_.hits); + order.reserve(ann_.hits.size()); + for (const detail::IvfHit& hit : ann_.hits) order.push_back(ann_.recs[hit.slot]); + ++ann_.approximate; + if (scale > 1) ++ann_.widened; + } + // Outside the lock: `visit` is user code and may search this store again. + for (const EmbeddingRecord* r : order) { + float score = cosineSimilarity(query, r->vector); + if (!(score >= threshold)) continue; + if (!visit(SimilarityResult{r->node_ref, score, r->text})) break; + } + return true; + } + size_t count() const { return embeddings_.size(); } @@ -1345,6 +1934,10 @@ class EmbeddingStore { * parameter existed, clear() always reset it.) */ void clear(bool resetDimension = false) { + { + std::lock_guard<std::mutex> lock(ann_.mutex); + ann_.reset(); + } embeddings_.clear(); if (resetDimension) dimension_.reset(); } @@ -1413,8 +2006,8 @@ class EmbeddingStore { for (const auto& item : payload.embeddings) { checkDimension(item.vector, "embedding for :" + item.type + ":" + item.id); std::string key = nodeRefToKey(NodeRef(item.type, item.id)); - embeddings_[key] = EmbeddingRecord{key, NodeRef(item.type, item.id), item.vector, - item.text, item.model}; + putRecord(key, EmbeddingRecord{key, NodeRef(item.type, item.id), item.vector, + item.text, item.model}); ++written; } return written; @@ -1865,16 +2458,33 @@ class SemanticGraph { * With a gate the store cannot truncate first: the best k *unfiltered* * matches may all be filtered out, so every match is scored, filtered, and * only then cut to k. + * + * With the store's approximate index the candidates are the probed cells + * only. If fewer than k of them pass the gate, the probe is widened (4x + * each step) and the search retried, ending in the exact scan, so a + * selective filter costs time rather than results. */ std::vector<SimilarityResult> seedSearchVector( const std::vector<float>& queryVector, size_t topK, const std::string& nodeType, - float threshold, const NodeGate& gate + float threshold, const NodeGate& gate, bool exact = false ) const { if (!gate) { - return embedding_store_.similaritySearchVector(queryVector, topK, nodeType, threshold); + return embedding_store_.similaritySearchVector(queryVector, topK, nodeType, threshold, exact); + } + if (!exact && topK > 0 && topK < embedding_store_.count() && embedding_store_.annEnabled()) { + for (size_t scale = 1;; scale *= 4) { + std::vector<SimilarityResult> kept; + bool answered = embedding_store_.annVisit( + queryVector, nodeType, threshold, scale, [&](SimilarityResult&& result) { + if (gate(result.node_ref)) kept.push_back(std::move(result)); + return kept.size() < topK; + }); + if (!answered) break; + if (kept.size() >= topK) return kept; + } } auto all = embedding_store_.similaritySearchVector( - queryVector, std::numeric_limits<size_t>::max(), nodeType, threshold); + queryVector, std::numeric_limits<size_t>::max(), nodeType, threshold, exact); std::vector<SimilarityResult> kept; for (auto& result : all) { if (kept.size() >= topK) break; @@ -1889,6 +2499,9 @@ class SemanticGraph { * differs from the model that produced the stored vectors * @param version encoder/model version recorded in the sidecar * (default: the encoder's own version(), if any) + * @param ann opt in to approximate search for the graph's + * embedding store (see EmbeddingStore::enableAnn); nullopt (the + * default) keeps search exact */ SemanticGraph( const std::string& name = "semantic_graph", @@ -1896,7 +2509,8 @@ class SemanticGraph { bool autoEmbed = true, std::vector<std::string> embedFields = {"name", "description", "content", "text"}, ModelMismatchMode onModelMismatch = ModelMismatchMode::Warn, - std::optional<std::string> version = std::nullopt + std::optional<std::string> version = std::nullopt, + std::optional<AnnOptions> ann = std::nullopt ) : graph_(name) , embedding_store_(encoder, onModelMismatch) @@ -1904,6 +2518,7 @@ class SemanticGraph { , embed_fields_(std::move(embedFields)) { if (version.has_value()) embedding_store_.setVersion(std::move(version)); + if (ann.has_value()) embedding_store_.enableAnn(*ann); } /** @@ -2132,20 +2747,51 @@ class SemanticGraph { * applied *before* topK is taken, so you still get topK matching nodes when * that many exist. A text query is checked against the model that made the * stored vectors (see ModelMismatchMode). + * + * With approximate search enabled on the embedding store the candidates + * come from the probed cells; `exact` forces the full scan (no effect + * otherwise). A filter that few nodes pass widens the probe, ending in the + * full scan, so you still get topK matching nodes when that many exist. */ std::vector<SimilarityResult> similaritySearch( const std::string& query, size_t topK = 10, const std::string& nodeType = "", float threshold = 0.0f, - const NodeFilterOptions& filter = NodeFilterOptions() + const NodeFilterOptions& filter = NodeFilterOptions(), + bool exact = false ) const { NodeGate gate = makeGate(filter); if (!gate) { - return embedding_store_.similaritySearch(query, topK, nodeType, threshold); + return embedding_store_.similaritySearch(query, topK, nodeType, threshold, exact); } return seedSearchVector(embedding_store_.encodeQuery(query), topK, nodeType, - threshold, gate); + threshold, gate, exact); + } + + /** similaritySearch() with a ready-made query vector (never checked against the encoder's model). */ + std::vector<SimilarityResult> similaritySearchVector( + const std::vector<float>& queryVector, + size_t topK = 10, + const std::string& nodeType = "", + float threshold = 0.0f, + const NodeFilterOptions& filter = NodeFilterOptions(), + bool exact = false + ) const { + NodeGate gate = makeGate(filter); + return seedSearchVector(queryVector, topK, nodeType, threshold, gate, exact); + } + + /** Recall@k of the approximate index against the exact scan (see EmbeddingStore::recallCheck). */ + RecallReport recallCheck(const std::vector<std::vector<float>>& queries, size_t k = 10, + const std::string& nodeType = "", float threshold = 0.0f) const { + return embedding_store_.recallCheck(queries, k, nodeType, threshold); + } + + /** recallCheck() with text queries (needs an encoder). */ + RecallReport recallCheckTexts(const std::vector<std::string>& queries, size_t k = 10, + const std::string& nodeType = "", float threshold = 0.0f) const { + return embedding_store_.recallCheckTexts(queries, k, nodeType, threshold); } /** @@ -2763,6 +3409,9 @@ class SemanticGraph { * sidecar's encoder_version are set and differ, a warning is issued. * @param onModelMismatch for text searches whose encoder model differs from * the stored vectors' model (see EmbeddingStore) + * @param ann opt in to approximate search (see EmbeddingStore::enableAnn). + * The index is never saved; it is rebuilt lazily from the loaded + * vectors on the first search. */ static SemanticGraph load( const std::string& path, @@ -2773,7 +3422,8 @@ class SemanticGraph { std::vector<std::string> embedFields = {"name", "description", "content", "text"}, GraphMismatchMode onMismatch = GraphMismatchMode::Error, std::optional<std::string> version = std::nullopt, - ModelMismatchMode onModelMismatch = ModelMismatchMode::Warn + ModelMismatchMode onModelMismatch = ModelMismatchMode::Warn, + std::optional<AnnOptions> ann = std::nullopt ) { // Read and parse the text here rather than through ISONGraph::load: the // digest has to be taken from the document (it records which ids were @@ -2806,7 +3456,7 @@ class SemanticGraph { ISONGraph base = ISONGraph::buildFromDocument(doc, graphName); SemanticGraph result(base.name, std::move(encoder), autoEmbed, std::move(embedFields), - onModelMismatch, std::move(version)); + onModelMismatch, std::move(version), std::move(ann)); result.graph_ = std::move(base); if (embeddings) { diff --git a/isongraph-vector-cpp/include/isongraph_vector_http.hpp b/isongraph-vector-cpp/include/isongraph_vector_http.hpp index 29cc315..bcbd6d3 100644 --- a/isongraph-vector-cpp/include/isongraph_vector_http.hpp +++ b/isongraph-vector-cpp/include/isongraph_vector_http.hpp @@ -123,8 +123,9 @@ using HttpTransport = std::function<HttpResponse(const HttpRequest&)>; * * One easy handle per request (no connection reuse - an embeddings call is * long next to a TLS handshake). Redirects are not followed, so the API key is - * never forwarded to another host; only http and https are allowed; the - * response body is capped at 64 MiB. The first use initialises libcurl + * never forwarded to another host; only http and https are allowed; no + * `Expect: 100-continue` is sent; the response body is capped at 64 MiB (a + * larger one is an HttpError). The first use initialises libcurl * globally (curl_global_init is not thread-safe before that point, so make a * first call from one thread before going concurrent). */ @@ -143,18 +144,24 @@ class CurlTransport { std::unique_ptr<CURL, void (*)(CURL*)> easy(raw, curl_easy_cleanup); curl_slist* rawList = nullptr; - for (const auto& [name, value] : request.headers) { - std::string line = name + ": " + value; + auto appendHeader = [&](const std::string& line) { curl_slist* next = curl_slist_append(rawList, line.c_str()); if (next == nullptr) { curl_slist_free_all(rawList); throw HttpError("out of memory building request headers"); } rawList = next; - } + }; + for (const auto& [name, value] : request.headers) appendHeader(name + ": " + value); + // "Expect:" with no value stops libcurl asking for a 100-continue before + // a large body: servers that do not answer it would stall the request + // for a second, and an embeddings call gains nothing from the handshake. + appendHeader("Expect:"); std::unique_ptr<curl_slist, void (*)(curl_slist*)> list(rawList, curl_slist_free_all); HttpResponse response; + BodySink sink; + sink.body = &response.body; char errorBuffer[CURL_ERROR_SIZE] = {0}; auto set = [&](CURLoption option, auto value) { @@ -183,11 +190,14 @@ class CurlTransport { set(CURLOPT_ACCEPT_ENCODING, ""); set(CURLOPT_ERRORBUFFER, errorBuffer); set(CURLOPT_WRITEFUNCTION, &CurlTransport::onBody); - set(CURLOPT_WRITEDATA, &response.body); + set(CURLOPT_WRITEDATA, &sink); set(CURLOPT_HEADERFUNCTION, &CurlTransport::onHeader); set(CURLOPT_HEADERDATA, &response.headers); CURLcode rc = curl_easy_perform(easy.get()); + if (rc != CURLE_OK && sink.overflowed) { + throw HttpError("HTTP response body exceeds the 64 MiB limit"); + } if (rc != CURLE_OK) { // libcurl's own text: names the failure and the host, never a header. throw HttpError(std::string("HTTP request failed: ") + @@ -212,11 +222,19 @@ class CurlTransport { ~GlobalInit() { curl_global_cleanup(); } }; + struct BodySink { + std::string* body = nullptr; + bool overflowed = false; + }; + static size_t onBody(char* data, size_t size, size_t count, void* userdata) { - auto* body = static_cast<std::string*>(userdata); + auto* sink = static_cast<BodySink*>(userdata); size_t bytes = size * count; - if (body->size() + bytes > MAX_BODY_BYTES) return 0; // aborts the transfer - body->append(data, bytes); + if (sink->body->size() + bytes > MAX_BODY_BYTES) { + sink->overflowed = true; + return 0; // aborts the transfer + } + sink->body->append(data, bytes); return bytes; } diff --git a/isongraph-vector-cpp/include/isongraph_vector_ivf.hpp b/isongraph-vector-cpp/include/isongraph_vector_ivf.hpp new file mode 100644 index 0000000..bbaf9fb --- /dev/null +++ b/isongraph-vector-cpp/include/isongraph_vector_ivf.hpp @@ -0,0 +1,715 @@ +/** + * ISONGraph Vector - IVF-flat approximate index (numerics only) + * + * The inverted-file index behind EmbeddingStore::enableAnn(). It knows nothing + * about node references or the store: it holds rows of floats, addressed by + * slot number, and answers "which slots score highest against this query". + * isongraph_vector.hpp owns the slot <-> record mapping and the policy + * (when to build, fold, retrain); this file is the arithmetic. + * + * Layout (all contiguous, nothing allocated per query): + * data slots x dim floats, the vectors as given (not normalised) + * invNorm 1 / |row| per slot (0 for a zero row, which then scores 0) + * assign cell of each slot, -1 = in no cell yet (new/updated) or dead + * order the slots of every cell, grouped by cell + * offsets order[offsets[c] .. offsets[c+1]) are the slots of cell c + * centroids nlist x dim unit vectors + * + * Training is spherical k-means (centroids are unit vectors and rows are + * compared by direction, which is what cosine similarity measures) on a + * sample of at most 40 x nlist rows, 8 iterations, with a seeded Mersenne + * Twister. For a given build the result does not depend on the number of + * threads used: every row's cell is independent of the others, and a + * centroid sums its rows in ascending row order. + * + * Not thread-safe by itself; EmbeddingStore serialises access. + */ + +#ifndef ISONGRAPH_VECTOR_IVF_HPP +#define ISONGRAPH_VECTOR_IVF_HPP + +#include <algorithm> +#include <atomic> +#include <chrono> +#include <cmath> +#include <cstddef> +#include <cstdint> +#include <exception> +#include <limits> +#include <numeric> +#include <random> +#include <system_error> +#include <thread> +#include <vector> + +// The kernels use SIMD where the compiler enables it: AVX2+FMA (compile with +// -mavx2 -mfma, /arch:AVX2 or -march=native), else SSE2 (every x86-64), else +// NEON (AArch64), else plain scalar code. Define ISONGRAPH_VECTOR_NO_SIMD to +// force the scalar path. +#if defined(ISONGRAPH_VECTOR_NO_SIMD) +// scalar +#elif defined(__AVX2__) && (defined(__FMA__) || defined(_MSC_VER)) // MSVC has no __FMA__; /arch:AVX2 allows FMA +#include <immintrin.h> +#define ISONGRAPH_VECTOR_SIMD_AVX2 1 +#elif defined(__SSE2__) || defined(_M_X64) || (defined(_M_IX86_FP) && _M_IX86_FP >= 2) +#include <emmintrin.h> +#define ISONGRAPH_VECTOR_SIMD_SSE2 1 +#elif defined(__ARM_NEON) +#include <arm_neon.h> +#define ISONGRAPH_VECTOR_SIMD_NEON 1 +#endif + +namespace isongraph_vector { +namespace detail { + +// ============================================================================= +// SIMD lane abstraction. One implementation of each kernel serves every +// backend; the scalar fallback is the same code with one-float lanes. +// ============================================================================= + +namespace simd { + +#if defined(ISONGRAPH_VECTOR_SIMD_AVX2) +using V = __m256; +constexpr size_t W = 8; +inline V zero() { return _mm256_setzero_ps(); } +inline V load(const float* p) { return _mm256_loadu_ps(p); } +inline V fma(V a, V b, V c) { return _mm256_fmadd_ps(a, b, c); } +inline V add(V a, V b) { return _mm256_add_ps(a, b); } +inline float hsum(V v) { + __m128 lo = _mm256_castps256_ps128(v); + __m128 hi = _mm256_extractf128_ps(v, 1); + __m128 s = _mm_add_ps(lo, hi); + s = _mm_add_ps(s, _mm_movehl_ps(s, s)); + s = _mm_add_ss(s, _mm_shuffle_ps(s, s, 1)); + return _mm_cvtss_f32(s); +} +inline void prefetch(const float* p) { _mm_prefetch(reinterpret_cast<const char*>(p), _MM_HINT_T0); } +#elif defined(ISONGRAPH_VECTOR_SIMD_SSE2) +using V = __m128; +constexpr size_t W = 4; +inline V zero() { return _mm_setzero_ps(); } +inline V load(const float* p) { return _mm_loadu_ps(p); } +inline V fma(V a, V b, V c) { return _mm_add_ps(_mm_mul_ps(a, b), c); } +inline V add(V a, V b) { return _mm_add_ps(a, b); } +inline float hsum(V v) { + __m128 s = _mm_add_ps(v, _mm_movehl_ps(v, v)); + s = _mm_add_ss(s, _mm_shuffle_ps(s, s, 1)); + return _mm_cvtss_f32(s); +} +inline void prefetch(const float* p) { _mm_prefetch(reinterpret_cast<const char*>(p), _MM_HINT_T0); } +#elif defined(ISONGRAPH_VECTOR_SIMD_NEON) +using V = float32x4_t; +constexpr size_t W = 4; +inline V zero() { return vdupq_n_f32(0.0f); } +inline V load(const float* p) { return vld1q_f32(p); } +inline V fma(V a, V b, V c) { return vmlaq_f32(c, a, b); } +inline V add(V a, V b) { return vaddq_f32(a, b); } +inline float hsum(V v) { return vaddvq_f32(v); } +inline void prefetch(const float*) {} +#else +using V = float; +constexpr size_t W = 1; +inline V zero() { return 0.0f; } +inline V load(const float* p) { return *p; } +inline V fma(V a, V b, V c) { return a * b + c; } +inline V add(V a, V b) { return a + b; } +inline float hsum(V v) { return v; } +inline void prefetch(const float*) {} +#endif + +/** Dot product of two n-float rows. Fixed summation order for a given backend. */ +inline float dot(const float* a, const float* b, size_t n) { + V s0 = zero(), s1 = zero(), s2 = zero(), s3 = zero(); + size_t i = 0; + for (; i + 4 * W <= n; i += 4 * W) { + s0 = fma(load(a + i), load(b + i), s0); + s1 = fma(load(a + i + W), load(b + i + W), s1); + s2 = fma(load(a + i + 2 * W), load(b + i + 2 * W), s2); + s3 = fma(load(a + i + 3 * W), load(b + i + 3 * W), s3); + } + for (; i + W <= n; i += W) s0 = fma(load(a + i), load(b + i), s0); + float total = hsum(add(add(s0, s1), add(s2, s3))); + for (; i < n; ++i) total += a[i] * b[i]; + return total; +} + +/** + * Four rows against two centroids at once (eight dot products): each centroid + * chunk is loaded once for four rows, which is what makes assigning a million + * rows to thousands of cells compute-bound rather than memory-bound. + * `rows` points at four consecutive rows of length `n`. + */ +inline void tile4x2(const float* rows, size_t n, const float* c0, const float* c1, + float* out /* [4][2] */) { + const float* r0 = rows; + const float* r1 = rows + n; + const float* r2 = rows + 2 * n; + const float* r3 = rows + 3 * n; + V a00 = zero(), a01 = zero(), a10 = zero(), a11 = zero(); + V a20 = zero(), a21 = zero(), a30 = zero(), a31 = zero(); + size_t i = 0; + for (; i + W <= n; i += W) { + V b0 = load(c0 + i); + V b1 = load(c1 + i); + V x0 = load(r0 + i); + V x1 = load(r1 + i); + V x2 = load(r2 + i); + V x3 = load(r3 + i); + a00 = fma(x0, b0, a00); a01 = fma(x0, b1, a01); + a10 = fma(x1, b0, a10); a11 = fma(x1, b1, a11); + a20 = fma(x2, b0, a20); a21 = fma(x2, b1, a21); + a30 = fma(x3, b0, a30); a31 = fma(x3, b1, a31); + } + out[0] = hsum(a00); out[1] = hsum(a01); + out[2] = hsum(a10); out[3] = hsum(a11); + out[4] = hsum(a20); out[5] = hsum(a21); + out[6] = hsum(a30); out[7] = hsum(a31); + for (; i < n; ++i) { + out[0] += r0[i] * c0[i]; out[1] += r0[i] * c1[i]; + out[2] += r1[i] * c0[i]; out[3] += r1[i] * c1[i]; + out[4] += r2[i] * c0[i]; out[5] += r2[i] * c1[i]; + out[6] += r3[i] * c0[i]; out[7] += r3[i] * c1[i]; + } +} + +} // namespace simd + +// ============================================================================= +// Small helpers +// ============================================================================= + +/** Run fn(0..tasks) on up to `threads` threads; serial when that is 1 or the work is tiny. */ +template <class F> +void parallelFor(size_t tasks, size_t threads, F&& fn) { + if (tasks == 0) return; + if (threads > tasks) threads = tasks; + if (threads <= 1) { + for (size_t i = 0; i < tasks; ++i) fn(i); + return; + } + std::atomic<size_t> next{0}; + std::exception_ptr failure; + std::atomic<bool> failed{false}; + auto worker = [&]() { + try { + for (;;) { + size_t i = next.fetch_add(1); + if (i >= tasks || failed.load()) break; + fn(i); + } + } catch (...) { + if (!failed.exchange(true)) failure = std::current_exception(); + } + }; + std::vector<std::thread> pool; + pool.reserve(threads - 1); + try { + for (size_t t = 1; t < threads; ++t) pool.emplace_back(worker); + } catch (const std::system_error&) { + // No threads available (for example a static build without -pthread): + // the caller's thread does the rest, same result. + } + worker(); + for (auto& t : pool) t.join(); + if (failure) std::rethrow_exception(failure); +} + +inline size_t autoThreads() { + unsigned n = std::thread::hardware_concurrency(); + return n == 0 ? 1 : n; +} + +/** One search hit: higher score first, lower slot first on a tie. */ +struct IvfHit { + float score; + uint32_t slot; +}; + +inline bool hitBefore(const IvfHit& a, const IvfHit& b) { + return a.score > b.score || (a.score == b.score && a.slot < b.slot); +} + +// ============================================================================= +// The index +// ============================================================================= + +class IvfIndex { +public: + static constexpr uint32_t kNoSlot = std::numeric_limits<uint32_t>::max(); + + size_t dim = 0; + size_t nlist = 0; + size_t trainedN = 0; ///< live rows when the centroids were trained + size_t churn = 0; ///< writes + deletes since then + size_t pendingLive = 0; ///< live rows not in any cell yet + double buildSeconds = 0.0; + + size_t slots() const { return alive_.size(); } + size_t live() const { return alive_.size() - dead_; } + size_t dead() const { return dead_; } + + /** + * Train on `n` rows laid out contiguously in `data` (slot i = row i, all + * alive), then put every row in its nearest cell. `codes` gives each row + * a small integer type code for filtered searches. + */ + static IvfIndex build(size_t dim, std::vector<float>&& data, std::vector<uint32_t>&& codes, + size_t nlistWanted, uint64_t seed, size_t threads, int iterations = 8) { + auto started = std::chrono::steady_clock::now(); + IvfIndex index; + index.dim = dim; + index.data_ = std::move(data); + index.codes_ = std::move(codes); + const size_t n = index.codes_.size(); + index.alive_.assign(n, 1); + index.pend_.assign(n, 0); + index.assign_.assign(n, -1); + index.invNorm_.resize(n); + if (threads == 0) threads = autoThreads(); + parallelFor((n + 8191) / 8192, threads, [&](size_t block) { + size_t lo = block * 8192, hi = std::min(n, lo + 8192); + for (size_t i = lo; i < hi; ++i) index.invNorm_[i] = inverseNorm(index.row(i), dim); + }); + + index.nlist = std::max<size_t>(1, std::min(nlistWanted, n)); + index.trainedN = n; + index.train(seed, threads, iterations); + index.assignAll(threads); + index.relist(); + index.buildSeconds = std::chrono::duration<double>( + std::chrono::steady_clock::now() - started).count(); + return index; + } + + // -- writes ----------------------------------------------------------------- + + /** A new row; it is scored exactly by every query until folded into a cell. */ + uint32_t append(const float* v, uint32_t code) { + uint32_t slot = static_cast<uint32_t>(alive_.size()); + data_.insert(data_.end(), v, v + dim); + invNorm_.push_back(inverseNorm(v, dim)); + assign_.push_back(-1); + alive_.push_back(1); + pend_.push_back(1); + codes_.push_back(code); + pending_.push_back(slot); + ++pendingLive; + ++churn; + return slot; + } + + /** Replace a row in place: it leaves its cell and is pending again. */ + void overwrite(uint32_t slot, const float* v, uint32_t code) { + std::copy(v, v + dim, data_.begin() + static_cast<std::ptrdiff_t>(slot) * dim); + invNorm_[slot] = inverseNorm(v, dim); + codes_[slot] = code; + assign_[slot] = -1; + if (!pend_[slot]) { + pend_[slot] = 1; + pending_.push_back(slot); + ++pendingLive; + } + ++churn; + } + + /** Tombstone a row. Its stale entry in a cell list is skipped at query time. */ + void drop(uint32_t slot) { + if (!alive_[slot]) return; + alive_[slot] = 0; + assign_[slot] = -1; + if (pend_[slot]) { + pend_[slot] = 0; + --pendingLive; + } + ++dead_; + ++churn; + } + + bool needsCompaction() const { return dead_ >= 64 && dead_ * 4 > alive_.size(); } + + /** + * Squeeze out the tombstones, keeping live rows in slot order and their + * cells. Returns, for every old slot, its new slot (kNoSlot when dead); + * the owner uses it to renumber its own slot tables. + */ + std::vector<uint32_t> compact() { + const size_t n = alive_.size(); + std::vector<uint32_t> remap(n, kNoSlot); + size_t kept = 0; + for (size_t i = 0; i < n; ++i) { + if (!alive_[i]) continue; + remap[i] = static_cast<uint32_t>(kept); + if (kept != i) { + std::copy(row(i), row(i) + dim, data_.begin() + static_cast<std::ptrdiff_t>(kept) * dim); + invNorm_[kept] = invNorm_[i]; + assign_[kept] = assign_[i]; + pend_[kept] = pend_[i]; + codes_[kept] = codes_[i]; + } + ++kept; + } + data_.resize(kept * dim); + invNorm_.resize(kept); + assign_.resize(kept); + pend_.resize(kept); + codes_.resize(kept); + alive_.assign(kept, 1); + dead_ = 0; + std::vector<uint32_t> pending; + pending.reserve(pendingLive); + for (uint32_t old : pending_) { + if (old < n && remap[old] != kNoSlot && pend_[remap[old]]) { + pending.push_back(remap[old]); + } + } + // a slot can be listed twice only if it was dropped and re-listed, which + // overwrite() prevents; sort+unique keeps the list a set regardless + std::sort(pending.begin(), pending.end()); + pending.erase(std::unique(pending.begin(), pending.end()), pending.end()); + pending_ = std::move(pending); + relist(); + return remap; + } + + /** Put every pending row in its nearest cell and regroup. */ + void fold(size_t threads) { + std::vector<uint32_t> ids; + ids.reserve(pendingLive); + for (uint32_t slot : pending_) { + if (alive_[slot] && pend_[slot]) ids.push_back(slot); + } + std::sort(ids.begin(), ids.end()); + ids.erase(std::unique(ids.begin(), ids.end()), ids.end()); + if (!ids.empty()) { + std::vector<float> gathered(ids.size() * dim); + for (size_t i = 0; i < ids.size(); ++i) { + std::copy(row(ids[i]), row(ids[i]) + dim, gathered.begin() + static_cast<std::ptrdiff_t>(i) * dim); + } + std::vector<int32_t> cells(ids.size()); + if (threads == 0) threads = autoThreads(); + nearest(gathered.data(), ids.size(), cells.data(), threads); + for (size_t i = 0; i < ids.size(); ++i) { + assign_[ids[i]] = cells[i]; + pend_[ids[i]] = 0; + } + } + pending_.clear(); + pendingLive = 0; + relist(); + } + + // -- queries ---------------------------------------------------------------- + + /** + * Score the rows of the `lists` cells whose centroids are nearest `q`, plus + * every pending row, and keep the best `k` that score at least `threshold` + * (and have type `code`, unless it is kAnyCode). `out` is filled best + * first. Scores are cosine similarities computed from the cached inverse + * norms. + */ + static constexpr uint32_t kAnyCode = std::numeric_limits<uint32_t>::max(); + + void search(const float* q, float invQ, uint32_t code, float threshold, size_t lists, + size_t k, std::vector<IvfHit>& out) { + out.clear(); + if (k == 0 || live() == 0) return; + chooseCells(q, lists); + heap_.clear(); + heap_.reserve(k); + scan(q, invQ, code, threshold, k, /*collectAll=*/false); + out.assign(heap_.begin(), heap_.end()); + std::sort(out.begin(), out.end(), hitBefore); + } + + /** Like search() but returns every candidate that passes the filters, best first. */ + void candidates(const float* q, float invQ, uint32_t code, float threshold, size_t lists, + std::vector<IvfHit>& out) { + out.clear(); + if (live() == 0) return; + chooseCells(q, lists); + heap_.clear(); + scan(q, invQ, code, threshold, 0, /*collectAll=*/true); + out.assign(heap_.begin(), heap_.end()); + std::sort(out.begin(), out.end(), hitBefore); + } + + /** Bytes held by the index apart from the vectors themselves. */ + size_t overheadBytes() const { + return centroids_.capacity() * sizeof(float) + assign_.capacity() * sizeof(int32_t) + + order_.capacity() * sizeof(uint32_t) + offsets_.capacity() * sizeof(uint32_t) + + invNorm_.capacity() * sizeof(float) + codes_.capacity() * sizeof(uint32_t) + + alive_.capacity() + pend_.capacity() + pending_.capacity() * sizeof(uint32_t); + } + /** Bytes of the contiguous vector copy the index scans. */ + size_t vectorBytes() const { return data_.capacity() * sizeof(float); } + + static float inverseNorm(const float* v, size_t n) { + float s = simd::dot(v, v, n); + return s > 0.0f ? 1.0f / std::sqrt(s) : 0.0f; + } + +private: + std::vector<float> data_; + std::vector<float> invNorm_; + std::vector<int32_t> assign_; + std::vector<uint8_t> alive_; + std::vector<uint8_t> pend_; + std::vector<uint32_t> codes_; + std::vector<uint32_t> pending_; + std::vector<uint32_t> order_; + std::vector<uint32_t> offsets_; + std::vector<float> centroids_; + size_t dead_ = 0; + + // per-query scratch, sized once + std::vector<float> cellScore_; + std::vector<uint32_t> cellIdx_; + std::vector<uint32_t> chosen_; + std::vector<IvfHit> heap_; + + const float* row(size_t slot) const { return data_.data() + slot * dim; } + + // -- training ----------------------------------------------------------------- + + static uint64_t below(std::mt19937_64& rng, uint64_t bound) { return rng() % bound; } + + /** `count` distinct values of [0, bound) by a partial Fisher-Yates shuffle. */ + static std::vector<uint32_t> pick(std::mt19937_64& rng, size_t bound, size_t count) { + std::vector<uint32_t> all(bound); + std::iota(all.begin(), all.end(), 0u); + for (size_t i = 0; i < count; ++i) { + size_t j = i + static_cast<size_t>(below(rng, bound - i)); + std::swap(all[i], all[j]); + } + all.resize(count); + return all; + } + + /** Nearest centroid (by dot product) of each of `count` rows at `rows`. */ + void nearest(const float* rows, size_t count, int32_t* out, size_t threads) const { + constexpr size_t kChunk = 256; + parallelFor((count + kChunk - 1) / kChunk, threads, [&](size_t chunk) { + size_t lo = chunk * kChunk; + nearestRange(rows + lo * dim, std::min(kChunk, count - lo), out + lo); + }); + } + + /** Blocked nearest-centroid kernel: 4 rows x 2 centroids per step, centroids in L2-sized panels. */ + void nearestRange(const float* rows, size_t count, int32_t* out) const { + constexpr size_t kRowBlock = 32; + constexpr size_t kCentroidPanel = 64; + float best[kRowBlock]; + for (size_t rb = 0; rb < count; rb += kRowBlock) { + const size_t rows_here = std::min(kRowBlock, count - rb); + for (size_t j = 0; j < rows_here; ++j) { + best[j] = -std::numeric_limits<float>::infinity(); + out[rb + j] = 0; + } + for (size_t cp = 0; cp < nlist; cp += kCentroidPanel) { + const size_t cEnd = std::min(nlist, cp + kCentroidPanel); + size_t j = 0; + for (; j + 4 <= rows_here; j += 4) { + const float* block = rows + (rb + j) * dim; + size_t c = cp; + for (; c + 2 <= cEnd; c += 2) { + float s[8]; + simd::tile4x2(block, dim, centroid(c), centroid(c + 1), s); + for (size_t r = 0; r < 4; ++r) { + if (s[2 * r] > best[j + r]) { best[j + r] = s[2 * r]; out[rb + j + r] = static_cast<int32_t>(c); } + if (s[2 * r + 1] > best[j + r]) { best[j + r] = s[2 * r + 1]; out[rb + j + r] = static_cast<int32_t>(c + 1); } + } + } + for (; c < cEnd; ++c) { + for (size_t r = 0; r < 4; ++r) { + float s = simd::dot(block + r * dim, centroid(c), dim); + if (s > best[j + r]) { best[j + r] = s; out[rb + j + r] = static_cast<int32_t>(c); } + } + } + } + for (; j < rows_here; ++j) { + const float* r = rows + (rb + j) * dim; + for (size_t c = cp; c < cEnd; ++c) { + float s = simd::dot(r, centroid(c), dim); + if (s > best[j]) { best[j] = s; out[rb + j] = static_cast<int32_t>(c); } + } + } + } + } + } + + const float* centroid(size_t c) const { return centroids_.data() + c * dim; } + float* centroid(size_t c) { return centroids_.data() + c * dim; } + + void train(uint64_t seed, size_t threads, int iterations) { + const size_t total = alive_.size(); + std::mt19937_64 rng(seed); + const size_t sampleN = std::min(total, 40 * nlist); + + std::vector<uint32_t> sample; + if (sampleN == total) { + sample.resize(total); + std::iota(sample.begin(), sample.end(), 0u); + } else { + sample = pick(rng, total, sampleN); + std::sort(sample.begin(), sample.end()); + } + + // unit-length copy of the sample (a zero row stays zero) + std::vector<float> x(sampleN * dim); + parallelFor((sampleN + 4095) / 4096, threads, [&](size_t block) { + size_t lo = block * 4096, hi = std::min(sampleN, lo + 4096); + for (size_t i = lo; i < hi; ++i) { + const float* src = row(sample[i]); + float inv = invNorm_[sample[i]]; + float* dst = x.data() + i * dim; + for (size_t d = 0; d < dim; ++d) dst[d] = src[d] * inv; + } + }); + + centroids_.assign(nlist * dim, 0.0f); + { + std::vector<uint32_t> init = pick(rng, sampleN, nlist); + for (size_t c = 0; c < nlist; ++c) { + std::copy(x.data() + static_cast<size_t>(init[c]) * dim, + x.data() + (static_cast<size_t>(init[c]) + 1) * dim, centroid(c)); + } + } + + std::vector<int32_t> cell(sampleN); + std::vector<uint32_t> start(nlist + 1), byCell(sampleN), fill(nlist); + std::vector<char> reseed(nlist); + for (int it = 0; it < iterations; ++it) { + nearest(x.data(), sampleN, cell.data(), threads); + + // rows of each cell in ascending row order (counting sort) + std::fill(start.begin(), start.end(), 0u); + for (size_t i = 0; i < sampleN; ++i) ++start[static_cast<size_t>(cell[i]) + 1]; + for (size_t c = 0; c < nlist; ++c) start[c + 1] += start[c]; + std::copy(start.begin(), start.end() - 1, fill.begin()); + for (size_t i = 0; i < sampleN; ++i) byCell[fill[static_cast<size_t>(cell[i])]++] = static_cast<uint32_t>(i); + + constexpr size_t kCellsPerTask = 16; + parallelFor((nlist + kCellsPerTask - 1) / kCellsPerTask, threads, [&](size_t task) { + std::vector<double> sum(dim); + size_t cLo = task * kCellsPerTask, cHi = std::min(nlist, cLo + kCellsPerTask); + for (size_t c = cLo; c < cHi; ++c) { + std::fill(sum.begin(), sum.end(), 0.0); + for (uint32_t p = start[c]; p < start[c + 1]; ++p) { + const float* r = x.data() + static_cast<size_t>(byCell[p]) * dim; + for (size_t d = 0; d < dim; ++d) sum[d] += r[d]; + } + float* out = centroid(c); + double sq = 0.0; + for (size_t d = 0; d < dim; ++d) { + out[d] = static_cast<float>(sum[d]); + sq += static_cast<double>(out[d]) * out[d]; + } + double length = std::sqrt(sq); + reseed[c] = (start[c + 1] == start[c] || length == 0.0) ? 1 : 0; + if (!reseed[c]) { + float inv = static_cast<float>(1.0 / length); + for (size_t d = 0; d < dim; ++d) out[d] *= inv; + } + } + }); + // a cell that lost all its rows restarts on a random sample point + for (size_t c = 0; c < nlist; ++c) { + if (!reseed[c]) continue; + size_t pickRow = static_cast<size_t>(below(rng, sampleN)); + std::copy(x.data() + pickRow * dim, x.data() + (pickRow + 1) * dim, centroid(c)); + } + } + } + + void assignAll(size_t threads) { + const size_t n = alive_.size(); + nearest(data_.data(), n, assign_.data(), threads); + for (size_t i = 0; i < n; ++i) { + if (!alive_[i]) assign_[i] = -1; + } + pending_.clear(); + pendingLive = 0; + std::fill(pend_.begin(), pend_.end(), uint8_t(0)); + } + + /** Regroup the live rows by cell (a stable counting sort of assign_). */ + void relist() { + offsets_.assign(nlist + 1, 0u); + const size_t n = alive_.size(); + for (size_t i = 0; i < n; ++i) { + if (assign_[i] >= 0) ++offsets_[static_cast<size_t>(assign_[i]) + 1]; + } + for (size_t c = 0; c < nlist; ++c) offsets_[c + 1] += offsets_[c]; + order_.resize(offsets_[nlist]); + std::vector<uint32_t> fill(offsets_.begin(), offsets_.end() - 1); + for (size_t i = 0; i < n; ++i) { + if (assign_[i] >= 0) order_[fill[static_cast<size_t>(assign_[i])]++] = static_cast<uint32_t>(i); + } + } + + // -- query internals --------------------------------------------------------- + + void chooseCells(const float* q, size_t lists) { + if (cellScore_.size() != nlist) { + cellScore_.assign(nlist, 0.0f); + cellIdx_.resize(nlist); + } + chosen_.clear(); + if (lists >= nlist) { + for (size_t c = 0; c < nlist; ++c) chosen_.push_back(static_cast<uint32_t>(c)); + return; + } + for (size_t c = 0; c < nlist; ++c) { + cellScore_[c] = simd::dot(q, centroid(c), dim); + cellIdx_[c] = static_cast<uint32_t>(c); + } + auto better = [this](uint32_t a, uint32_t b) { + return cellScore_[a] > cellScore_[b] || (cellScore_[a] == cellScore_[b] && a < b); + }; + std::nth_element(cellIdx_.begin(), cellIdx_.begin() + static_cast<std::ptrdiff_t>(lists - 1), + cellIdx_.end(), better); + chosen_.assign(cellIdx_.begin(), cellIdx_.begin() + static_cast<std::ptrdiff_t>(lists)); + std::sort(chosen_.begin(), chosen_.end()); + } + + void scan(const float* q, float invQ, uint32_t code, float threshold, size_t k, bool collectAll) { + const float* base = data_.data(); + auto take = [&](uint32_t slot) { + float s = simd::dot(base + static_cast<size_t>(slot) * dim, q, dim) * invNorm_[slot] * invQ; + if (!(s >= threshold)) return; + IvfHit hit{s, slot}; + if (collectAll) { + heap_.push_back(hit); + } else if (heap_.size() < k) { + heap_.push_back(hit); + std::push_heap(heap_.begin(), heap_.end(), hitBefore); + } else if (hitBefore(hit, heap_.front())) { + std::pop_heap(heap_.begin(), heap_.end(), hitBefore); + heap_.back() = hit; + std::push_heap(heap_.begin(), heap_.end(), hitBefore); + } + }; + + for (uint32_t c : chosen_) { + const uint32_t lo = offsets_[c], hi = offsets_[c + 1]; + for (uint32_t p = lo; p < hi; ++p) { + if (p + 2 < hi) simd::prefetch(base + static_cast<size_t>(order_[p + 2]) * dim); + uint32_t slot = order_[p]; + if (assign_[slot] != static_cast<int32_t>(c)) continue; // moved or deleted since + if (code != kAnyCode && codes_[slot] != code) continue; + take(slot); + } + } + for (uint32_t slot : pending_) { + if (!alive_[slot] || !pend_[slot]) continue; + if (code != kAnyCode && codes_[slot] != code) continue; + take(slot); + } + } +}; + +} // namespace detail +} // namespace isongraph_vector + +#endif // ISONGRAPH_VECTOR_IVF_HPP diff --git a/isongraph-vector-cpp/tests/test_isongraph_vector.cpp b/isongraph-vector-cpp/tests/test_isongraph_vector.cpp index e1a5eb0..e20a12b 100644 --- a/isongraph-vector-cpp/tests/test_isongraph_vector.cpp +++ b/isongraph-vector-cpp/tests/test_isongraph_vector.cpp @@ -17,6 +17,9 @@ #include <functional> #include <map> #include <type_traits> +#include <thread> +#include <atomic> +#include <limits> #include "../include/isongraph_vector.hpp" #include "../include/isongraph_vector_http.hpp" @@ -2886,6 +2889,617 @@ TEST(a_node_added_while_the_endpoint_is_down_is_rolled_back) { ASSERT_EQ(graph.embeddingStore().count(), 1u); } +// ============================================================================= +// Approximate (IVF) search +// ============================================================================= + +namespace { + +/// Recall@10 the tests demand of the index on the clustered data below (4,000 +/// vectors, 40 clusters, 64 cells, nprobe 8 = 12.5% of the cells). Measured +/// 1.000 on every seed tried; the floor leaves room for rounding differences +/// between compilers, not for a worse algorithm. +const double ANN_RECALL_FLOOR = 0.95; + +/// Clustered unit-ish vectors and a way to draw queries from the same clusters +/// (the shape the Python tests use: an IVF index is meaningless on uniform noise). +struct ClusteredData { + size_t dim; + double noise; + std::vector<std::vector<float>> centers; + std::mt19937_64 rng; + + ClusteredData(size_t clusters, size_t dimension, uint64_t seed, double noiseNorm = 1.0) + : dim(dimension), noise(noiseNorm), rng(seed) { + for (size_t c = 0; c < clusters; ++c) { + std::vector<float> center(dim); + double sq = 0; + for (auto& x : center) { x = static_cast<float>(gauss()); sq += double(x) * x; } + for (auto& x : center) x = static_cast<float>(x / std::sqrt(sq)); + centers.push_back(std::move(center)); + } + } + + double gauss() { + double u1 = (static_cast<double>(rng() >> 11) + 1.0) / 9007199254740993.0; + double u2 = static_cast<double>(rng() >> 11) / 9007199254740992.0; + return std::sqrt(-2.0 * std::log(u1)) * std::cos(6.283185307179586 * u2); + } + + std::vector<float> draw() { + const auto& center = centers[rng() % centers.size()]; + std::vector<float> v(dim); + for (size_t d = 0; d < dim; ++d) { + v[d] = center[d] + static_cast<float>(noise * gauss() / std::sqrt(double(dim))); + } + return v; + } + + std::vector<std::vector<float>> drawMany(size_t count) { + std::vector<std::vector<float>> out; + for (size_t i = 0; i < count; ++i) out.push_back(draw()); + return out; + } +}; + +std::string annKey(const SimilarityResult& r) { return r.node_ref.type + ":" + r.node_ref.id; } + +std::vector<std::string> annKeys(const std::vector<SimilarityResult>& results) { + std::vector<std::string> keys; + for (const auto& r : results) keys.push_back(annKey(r)); + return keys; +} + +AnnOptions annOptions(size_t nlist = 64, size_t nprobe = 8, size_t minSize = 0) { + AnnOptions o; + o.nlist = nlist; + o.nprobe = nprobe; + o.minSize = minSize; + return o; +} + +void fillStore(EmbeddingStore& store, const std::vector<std::vector<float>>& data, + const std::function<std::string(size_t)>& typeOf = nullptr) { + for (size_t i = 0; i < data.size(); ++i) { + store.add(NodeRef(typeOf ? typeOf(i) : "doc", std::to_string(i)), "text " + std::to_string(i), data[i]); + } +} + +/// Same vectors, no index: the brute-force reference. +EmbeddingStore plainStore(const std::vector<std::vector<float>>& data, + const std::function<std::string(size_t)>& typeOf = nullptr) { + EmbeddingStore store; + fillStore(store, data, typeOf); + return store; +} + +} // namespace + +TEST(ann_is_off_by_default_and_exact_search_is_untouched) { + ClusteredData gen(40, 32, 1); + auto data = gen.drawMany(1500); + EmbeddingStore store = plainStore(data); + ASSERT_FALSE(store.annStats().enabled); + ASSERT_FALSE(store.annStats().active); + ASSERT_THROWS(store.buildAnnIndex(), EmbeddingError); + ASSERT_THROWS(store.setAnnNprobe(4), EmbeddingError); + + // exact=true on a store without an index is the same scan + for (const auto& q : gen.drawMany(5)) { + auto a = store.similaritySearchVector(q, 10, "", -1.0f); + auto b = store.similaritySearchVector(q, 10, "", -1.0f, /*exact=*/true); + ASSERT_EQ(a.size(), b.size()); + for (size_t i = 0; i < a.size(); ++i) { + ASSERT_EQ(annKey(a[i]), annKey(b[i])); + ASSERT_EQ(a[i].score, b[i].score); + } + } + ASSERT_EQ(store.annStats().exactSearches, 0u); // not counted while disabled +} + +TEST(ann_recall_floor_on_clustered_data) { + ClusteredData gen(40, 32, 1); + auto data = gen.drawMany(4000); + EmbeddingStore store; + store.enableAnn(annOptions(64, 8, 0)); + fillStore(store, data); + + RecallReport report = store.recallCheck(gen.drawMany(60), 10); + ASSERT_TRUE(report.approximate); + ASSERT_EQ(report.nlist, 64u); + ASSERT_EQ(report.nprobe, 8u); + ASSERT_EQ(report.queries, 60u); + ASSERT_GE(report.recall, ANN_RECALL_FLOOR); + ASSERT_TRUE(report.minRecall >= 0.0 && report.minRecall <= report.recall && report.recall <= 1.0); + ASSERT_TRUE(report.annMs >= 0.0 && report.exactMs >= 0.0); + + AnnStats stats = store.annStats(); + ASSERT_TRUE(stats.enabled && stats.active); + ASSERT_EQ(stats.nlist, 64u); + ASSERT_EQ(stats.trainedN, 4000u); + ASSERT_GT(stats.indexBytes, 0u); + ASSERT_GE(stats.approximateSearches, 60u); + ASSERT_EQ(stats.builds, 1u); +} + +TEST(ann_results_carry_the_exact_cosine_score) { + ClusteredData gen(40, 32, 2); + auto data = gen.drawMany(3000); + EmbeddingStore store; + store.enableAnn(annOptions()); + fillStore(store, data); + EmbeddingStore reference = plainStore(data); + + for (const auto& q : gen.drawMany(10)) { + std::map<std::string, float> truth; + for (const auto& r : reference.similaritySearchVector(q, 3000, "", -1.0f)) truth[annKey(r)] = r.score; + auto got = store.similaritySearchVector(q, 10, "", -1.0f); + ASSERT_EQ(got.size(), 10u); + for (size_t i = 0; i < got.size(); ++i) { + ASSERT_EQ(got[i].score, truth.at(annKey(got[i]))); // same float, not merely close + if (i > 0) ASSERT_TRUE(got[i - 1].score >= got[i].score); + } + } +} + +TEST(ann_exact_override_equals_brute_force) { + ClusteredData gen(40, 32, 3); + auto data = gen.drawMany(3000); + EmbeddingStore store; + store.enableAnn(annOptions(64, 2, 0)); // a narrow probe, so approximate answers would differ + fillStore(store, data); + EmbeddingStore reference = plainStore(data); + + for (const auto& q : gen.drawMany(15)) { + auto want = reference.similaritySearchVector(q, 10, "", 0.0f); + auto got = store.similaritySearchVector(q, 10, "", 0.0f, /*exact=*/true); + ASSERT_EQ(got.size(), want.size()); + for (size_t i = 0; i < want.size(); ++i) { + ASSERT_EQ(annKey(got[i]), annKey(want[i])); + ASSERT_EQ(got[i].score, want[i].score); + ASSERT_EQ(got[i].text, want[i].text); + } + } + // the override never consulted the index + ASSERT_EQ(store.annStats().approximateSearches, 0u); + ASSERT_GE(store.annStats().exactSearches, 15u); +} + +TEST(ann_probing_every_cell_is_the_exact_scan) { + ClusteredData gen(40, 32, 4); + auto data = gen.drawMany(2000); + EmbeddingStore store; + store.enableAnn(annOptions(32, 32, 0)); // nprobe == nlist + fillStore(store, data); + EmbeddingStore reference = plainStore(data); + for (const auto& q : gen.drawMany(8)) { + auto want = reference.similaritySearchVector(q, 10, "", 0.0f); + auto got = store.similaritySearchVector(q, 10, "", 0.0f); + ASSERT_EQ(annKeys(got), annKeys(want)); + for (size_t i = 0; i < want.size(); ++i) ASSERT_EQ(got[i].score, want[i].score); + } + RecallReport report = store.recallCheck(gen.drawMany(5)); + ASSERT_FALSE(report.approximate); + ASSERT_EQ(report.recall, 1.0); + ASSERT_EQ(store.annStats().approximateSearches, 0u); +} + +TEST(ann_small_store_falls_back_to_the_exact_scan) { + ClusteredData gen(10, 16, 5); + auto data = gen.drawMany(500); + EmbeddingStore store; + store.enableAnn(annOptions(16, 2, /*minSize=*/1000)); + fillStore(store, data); + EmbeddingStore reference = plainStore(data); + auto q = gen.draw(); + ASSERT_EQ(annKeys(store.similaritySearchVector(q, 10)), annKeys(reference.similaritySearchVector(q, 10))); + ASSERT_FALSE(store.annStats().active); + ASSERT_EQ(store.annStats().builds, 0u); + ASSERT_FALSE(store.recallCheck({q}).approximate); + + // growing past minSize switches the index on, shrinking below drops it again + auto more = gen.drawMany(600); + for (size_t i = 0; i < more.size(); ++i) store.add(NodeRef("late", std::to_string(i)), "", more[i]); + store.similaritySearchVector(q, 10); + ASSERT_TRUE(store.annStats().active); + for (size_t i = 0; i < 200; ++i) store.remove(NodeRef("late", std::to_string(i))); + store.similaritySearchVector(q, 10); + ASSERT_FALSE(store.annStats().active); +} + +TEST(ann_default_policy_matches_python) { + AnnOptions defaults; + ASSERT_EQ(defaults.minSize, 50000u); + ASSERT_EQ(defaults.nlist, 0u); // automatic: ~4*sqrt(N), 16..65536 + ASSERT_EQ(defaults.nprobe, 0u); // automatic: ~5% of nlist + ASSERT_EQ(defaults.seed, 0u); + ASSERT_EQ(AnnOptions::autoNlist(10000), 400u); + ASSERT_EQ(AnnOptions::autoNlist(100), 40u); + ASSERT_EQ(AnnOptions::autoNlist(4), 16u); + ASSERT_EQ(AnnOptions::autoNlist(1000000), 4000u); + ASSERT_EQ(AnnOptions::autoNlist(100000000), 40000u); + ASSERT_EQ(AnnOptions::autoNlist(1000000000000ull), 65536u); + + // automatic nlist/nprobe on a real index + ClusteredData gen(25, 16, 6); + auto data = gen.drawMany(2500); + EmbeddingStore store; + AnnOptions o; + o.minSize = 0; + store.enableAnn(o); + fillStore(store, data); + AnnStats stats = store.buildAnnIndex(); + ASSERT_EQ(stats.nlist, 200u); // round(4*sqrt(2500)) + ASSERT_EQ(stats.nprobe, 10u); // round(5% of 200) +} + +TEST(ann_same_seed_builds_the_same_index) { + ClusteredData gen(40, 32, 7); + auto data = gen.drawMany(3000); + auto queries = gen.drawMany(30); + + EmbeddingStore a, b, c, d; + a.enableAnn(annOptions(48, 4, 0)); + b.enableAnn(annOptions(48, 4, 0)); + AnnOptions threaded = annOptions(48, 4, 0); + threaded.threads = 4; + c.enableAnn(threaded); + AnnOptions single = annOptions(48, 4, 0); + single.threads = 1; + d.enableAnn(single); + fillStore(a, data); + // b gets the same rows in the opposite insertion order + for (size_t i = data.size(); i-- > 0;) b.add(NodeRef("doc", std::to_string(i)), "text " + std::to_string(i), data[i]); + fillStore(c, data); + fillStore(d, data); + + for (const auto& q : queries) { + auto ra = a.similaritySearchVector(q, 10, "", -1.0f); + auto rb = b.similaritySearchVector(q, 10, "", -1.0f); + auto rc = c.similaritySearchVector(q, 10, "", -1.0f); + auto rd = d.similaritySearchVector(q, 10, "", -1.0f); + ASSERT_EQ(annKeys(ra), annKeys(rb)); // insertion order does not matter + ASSERT_EQ(annKeys(ra), annKeys(rc)); // neither does the thread count + ASSERT_EQ(annKeys(ra), annKeys(rd)); + for (size_t i = 0; i < ra.size(); ++i) ASSERT_EQ(ra[i].score, rd[i].score); + } +} + +TEST(ann_writes_are_visible_immediately) { + ClusteredData gen(40, 32, 8); + auto data = gen.drawMany(4000); + EmbeddingStore store; + store.enableAnn(annOptions()); + fillStore(store, data); + store.buildAnnIndex(); + + auto fresh = gen.draw(); + store.add(NodeRef("new", "1"), "fresh", fresh); + auto top = store.similaritySearchVector(fresh, 3); + ASSERT_EQ(annKey(top[0]), "new:1"); + ASSERT_NEAR(top[0].score, 1.0f, 1e-5f); + ASSERT_EQ(store.annStats().pending, 1u); + + // an in-place update moves the row: found at its new place, scored from its + // new vector everywhere, and listed once + auto target = gen.draw(); + store.add(NodeRef("doc", "7"), "moved", target); + auto hit = store.similaritySearchVector(target, 1); + ASSERT_EQ(annKey(hit[0]), "doc:7"); + ASSERT_NEAR(hit[0].score, 1.0f, 1e-5f); + ASSERT_EQ(hit[0].text, "moved"); + ASSERT_EQ(store.count(), 4001u); + + EmbeddingStore truthStore = plainStore(data); + truthStore.add(NodeRef("new", "1"), "fresh", fresh); + truthStore.add(NodeRef("doc", "7"), "moved", target); + for (const auto& q : {data[7], target}) { + auto res = store.similaritySearchVector(q, 400, "", -1.0f); + std::set<std::string> seen; + for (const auto& r : res) ASSERT_TRUE(seen.insert(annKey(r)).second); // no stale second entry + std::map<std::string, float> truth; + for (const auto& r : truthStore.similaritySearchVector(q, 5000, "", -1.0f)) truth[annKey(r)] = r.score; + for (const auto& r : res) ASSERT_EQ(r.score, truth.at(annKey(r))); + } + // updating a row twice before it is folded does not list it twice + store.add(NodeRef("doc", "7"), "moved again", target); + ASSERT_EQ(store.annStats().pending, 2u); +} + +TEST(ann_deleted_rows_never_come_back) { + ClusteredData gen(40, 32, 9); + auto data = gen.drawMany(4000); + EmbeddingStore store; + store.enableAnn(annOptions()); + fillStore(store, data); + store.buildAnnIndex(); + ASSERT_EQ(store.annStats().builds, 1u); + + std::set<std::string> gone; + for (size_t i = 0; i < 4000; i += 3) { // a third of them: forces compaction + ASSERT_TRUE(store.remove(NodeRef("doc", std::to_string(i)))); + gone.insert("doc:" + std::to_string(i)); + } + for (const auto& q : gen.drawMany(15)) { + for (const auto& r : store.similaritySearchVector(q, 10)) ASSERT_EQ(gone.count(annKey(r)), 0u); + // tombstones would score 0.0, which beats any negative cosine + for (const auto& r : store.similaritySearchVector(q, 250, "", -1.0f)) ASSERT_EQ(gone.count(annKey(r)), 0u); + } + RecallReport report = store.recallCheck(gen.drawMany(40)); + ASSERT_GE(report.recall, ANN_RECALL_FLOOR); + ASSERT_EQ(store.annStats().builds, 1u); // the index followed the compaction, it was not retrained + ASSERT_EQ(store.count(), 4000u - gone.size()); + ASSERT_FALSE(store.remove(NodeRef("doc", "0"))); +} + +TEST(ann_pending_rows_fold_and_heavy_churn_retrains) { + ClusteredData gen(40, 32, 10); + auto data = gen.drawMany(3000); + EmbeddingStore store; + store.enableAnn(annOptions()); + fillStore(store, data); + store.buildAnnIndex(); + ASSERT_EQ(store.annStats().builds, 1u); + + auto extra = gen.drawMany(700); + for (size_t i = 0; i < extra.size(); ++i) store.add(NodeRef("late", std::to_string(i)), "", extra[i]); + ASSERT_EQ(store.annStats().pending, 700u); + store.similaritySearchVector(extra[0], 5); // >= 512 pending: folded + ASSERT_EQ(store.annStats().pending, 0u); + ASSERT_EQ(store.annStats().builds, 1u); + + auto more = gen.drawMany(1200); + for (size_t i = 0; i < more.size(); ++i) store.add(NodeRef("later", std::to_string(i)), "", more[i]); + store.similaritySearchVector(more[0], 5); // churn 1900 > 50% of 3000 + ASSERT_EQ(store.annStats().builds, 2u); + ASSERT_EQ(store.annStats().trainedN, 4900u); + ASSERT_GE(store.recallCheck(gen.drawMany(30)).recall, ANN_RECALL_FLOOR); +} + +TEST(ann_clear_drops_the_index_and_a_new_dimension_rebuilds) { + ClusteredData gen(15, 16, 11); + auto data = gen.drawMany(1500); + EmbeddingStore store; + store.enableAnn(annOptions(16, 4, 0)); + fillStore(store, data); + store.buildAnnIndex(); + ASSERT_TRUE(store.annStats().active); + store.clear(); + ASSERT_TRUE(store.similaritySearchVector(data[0], 5).empty()); + ASSERT_FALSE(store.annStats().active); + + store.clear(/*resetDimension=*/true); + ClusteredData small(12, 8, 12); + auto fresh = small.drawMany(1200); + fillStore(store, fresh); + auto res = store.similaritySearchVector(fresh[5], 5); + ASSERT_EQ(annKey(res[0]), "doc:5"); + ASSERT_TRUE(store.annStats().active); + ASSERT_EQ(*store.dimension(), 8u); +} + +TEST(ann_node_type_filter_widens_instead_of_returning_a_short_list) { + ClusteredData gen(40, 32, 13); + auto data = gen.drawMany(4000); + auto typeOf = [](size_t i) { return std::string(i % 200 == 0 ? "rare" : "common"); }; + EmbeddingStore store; + store.enableAnn(annOptions(64, 2, 0)); + fillStore(store, data, typeOf); + EmbeddingStore reference = plainStore(data, typeOf); + auto q = gen.draw(); + + auto common = store.similaritySearchVector(q, 10, "common"); + ASSERT_EQ(common.size(), 10u); + for (const auto& r : common) ASSERT_EQ(r.node_ref.type, "common"); + + // 20 rare rows in the whole store, a k of 15: two probed cells cannot hold 15 + // of them, so the probe must widen (ending in the full scan) + AnnStats before = store.annStats(); + auto rare = store.similaritySearchVector(q, 15, "rare", -1.0f); + ASSERT_EQ(rare.size(), 15u); + for (const auto& r : rare) ASSERT_EQ(r.node_ref.type, "rare"); + ASSERT_EQ(rare.size(), reference.similaritySearchVector(q, 15, "rare", -1.0f).size()); + AnnStats after = store.annStats(); + ASSERT_TRUE(after.widenedSearches > before.widenedSearches || after.exactSearches > before.exactSearches); + ASSERT_TRUE(store.similaritySearchVector(q, 5, "nope").empty()); +} + +TEST(ann_threshold_is_respected) { + ClusteredData gen(40, 32, 14); + auto data = gen.drawMany(4000); + EmbeddingStore store; + store.enableAnn(annOptions()); + fillStore(store, data); + auto q = gen.draw(); + float best = store.similaritySearchVector(q, 1, "", 0.0f, true)[0].score; + for (const auto& r : store.similaritySearchVector(q, 10, "", best - 1e-3f)) ASSERT_GE(r.score, best - 1e-3f); + ASSERT_TRUE(store.similaritySearchVector(q, 10, "", 2.0f).empty()); + // fewer than k above a threshold: the same short list as the scan + float cut = store.similaritySearchVector(q, 10, "", 0.0f, true)[4].score; + auto got = annKeys(store.similaritySearchVector(q, 10, "", cut)); + auto want = annKeys(store.similaritySearchVector(q, 10, "", cut, true)); + ASSERT_EQ(std::set<std::string>(got.begin(), got.end()), std::set<std::string>(want.begin(), want.end())); +} + +TEST(ann_zero_query_and_unbounded_k_use_the_exact_scan) { + ClusteredData gen(40, 32, 15); + auto data = gen.drawMany(2000); + EmbeddingStore store; + store.enableAnn(annOptions()); + fillStore(store, data); + auto zero = store.similaritySearchVector(std::vector<float>(32, 0.0f), 5, "", 0.0f); + ASSERT_EQ(zero.size(), 5u); + for (const auto& r : zero) ASSERT_EQ(r.score, 0.0f); + // asking for at least every row cannot be narrowed by an index + AnnStats before = store.annStats(); + ASSERT_EQ(store.similaritySearchVector(gen.draw(), 2000, "", -1.0f).size(), 2000u); + ASSERT_EQ(store.similaritySearchVector(gen.draw(), std::numeric_limits<size_t>::max(), "", -1.0f).size(), 2000u); + ASSERT_EQ(store.annStats().approximateSearches, before.approximateSearches); + ASSERT_TRUE(store.similaritySearchVector(gen.draw(), 0).empty()); +} + +TEST(ann_nprobe_can_be_changed_and_recall_follows) { + ClusteredData gen(40, 32, 16); + auto data = gen.drawMany(4000); + EmbeddingStore store; + store.enableAnn(annOptions(64, 1, 0)); + fillStore(store, data); + auto queries = gen.drawMany(60); + double narrow = store.recallCheck(queries).recall; + store.setAnnNprobe(16); + ASSERT_EQ(store.annStats().nprobe, 16u); + double wide = store.recallCheck(queries).recall; + ASSERT_GE(wide, narrow); + ASSERT_GE(wide, ANN_RECALL_FLOOR); +} + +TEST(ann_recall_check_arguments_and_text_queries) { + ClusteredData gen(40, 32, 17); + auto data = gen.drawMany(1500); + EmbeddingStore store(std::make_shared<MockEncoder>(32)); + store.enableAnn(annOptions(16, 4, 0)); + fillStore(store, data); + ASSERT_THROWS(store.recallCheck(gen.drawMany(2), 0), EmbeddingError); + ASSERT_EQ(store.recallCheck(gen.drawMany(3)).queries, 3u); + RecallReport texts = store.recallCheckTexts({"alpha", "beta", "gamma"}, 5); + ASSERT_EQ(texts.queries, 3u); + ASSERT_EQ(texts.k, 5u); + ASSERT_TRUE(texts.recall >= 0.0 && texts.recall <= 1.0); + + EmbeddingStore off = plainStore(data); + RecallReport r = off.recallCheck(gen.drawMany(3)); + ASSERT_EQ(r.recall, 1.0); + ASSERT_FALSE(r.approximate); +} + +TEST(ann_does_not_change_what_is_saved_or_the_graph_digest) { + ClusteredData gen(20, 16, 18); + auto data = gen.drawMany(1200); + + auto build = [&](bool withAnn) { + SemanticGraph graph("g", nullptr, false, {"name"}, ModelMismatchMode::Warn, std::nullopt, + withAnn ? std::optional<AnnOptions>(annOptions(16, 4, 0)) : std::nullopt); + for (size_t i = 0; i < data.size(); ++i) { + graph.addNode("item", std::to_string(i), {{"name", "n" + std::to_string(i)}}); + graph.embeddingStore().add(NodeRef("item", std::to_string(i)), "n" + std::to_string(i), data[i]); + } + return graph; + }; + SemanticGraph plain = build(false); + SemanticGraph indexed = build(true); + indexed.embeddingStore().buildAnnIndex(); + ASSERT_TRUE(indexed.embeddingStore().annStats().active); + indexed.similaritySearchVector(data[3], 5); + + ASSERT_EQ(graphDigest(plain.graph()), graphDigest(indexed.graph())); + ASSERT_EQ(plain.embeddingStore().toJson(), indexed.embeddingStore().toJson()); + + TempDir dir; + std::string a = dir.file("plain.ison"), b = dir.file("indexed.ison"); + plain.save(a); + indexed.save(b); + ASSERT_EQ(readWhole(a), readWhole(b)); + std::string sideA = readWhole(SemanticGraph::embeddingsPathFor(a)); + std::string sideB = readWhole(SemanticGraph::embeddingsPathFor(b)); + ASSERT_EQ(sideA, sideB); + ASSERT_TRUE(sideA.find("ivf") == std::string::npos && sideA.find("nlist") == std::string::npos); + + // and a graph loaded from either sidecar searches the same with the index on + SemanticGraph loaded = SemanticGraph::load(b, nullptr, "auto", true, false, {"name"}); + ASSERT_EQ(loaded.embeddingStore().count(), 1200u); + ASSERT_FALSE(loaded.embeddingStore().annStats().enabled); // never persisted + loaded.embeddingStore().enableAnn(annOptions(16, 16, 0)); + ASSERT_EQ(annKeys(loaded.similaritySearchVector(data[9], 8, "", -1.0f)), + annKeys(plain.similaritySearchVector(data[9], 8, "", -1.0f))); +} + +TEST(ann_through_the_graph_node_filter_and_exact_flag) { + ClusteredData gen(40, 32, 19); + auto data = gen.drawMany(3000); + SemanticGraph graph("g", nullptr, false, {"name"}, ModelMismatchMode::Warn, std::nullopt, + std::optional<AnnOptions>(annOptions(48, 6, 0))); + for (size_t i = 0; i < data.size(); ++i) { + graph.addNode("item", std::to_string(i), {{"group", static_cast<int>(i % 50)}}); + graph.embeddingStore().add(NodeRef("item", std::to_string(i)), "", data[i]); + } + auto q = gen.draw(); + + auto plainHits = graph.similaritySearchVector(q, 5); + ASSERT_EQ(plainHits.size(), 5u); + ASSERT_GE(graph.embeddingStore().annStats().approximateSearches, 1u); + + AnnStats before = graph.embeddingStore().annStats(); + graph.similaritySearchVector(q, 5, "", 0.0f, NodeFilterOptions(), /*exact=*/true); + ASSERT_EQ(graph.embeddingStore().annStats().approximateSearches, before.approximateSearches); + + // 1 node in 50 passes the filter: the 6 probed cells hold few of them, so the + // search must widen and still hand back k (the same k the full scan finds) + NodeFilterOptions filter; + filter.nodeFilter = [](const Node& n) { return n.properties.at("group").as_int() == 7; }; + auto filtered = graph.similaritySearchVector(q, 10, "", -1.0f, filter); + auto truth = graph.similaritySearchVector(q, 10, "", -1.0f, filter, /*exact=*/true); + ASSERT_EQ(filtered.size(), 10u); + ASSERT_EQ(truth.size(), 10u); + for (const auto& r : filtered) { + ASSERT_EQ(graph.graph().getNode(r.node_ref.type, r.node_ref.id).properties.at("group").as_int(), 7); + } + + auto near = graph.similarToNode(NodeRef("item", "3"), 5); + ASSERT_EQ(near.size(), 5u); + for (const auto& r : near) ASSERT_NE(r.node_ref.id, "3"); + ASSERT_TRUE(graph.recallCheck(gen.drawMany(20)).recall >= ANN_RECALL_FLOOR); +} + +TEST(ann_store_survives_copy_and_move) { + ClusteredData gen(20, 16, 20); + auto data = gen.drawMany(1500); + EmbeddingStore store; + store.enableAnn(annOptions(16, 16, 0)); + fillStore(store, data); + store.buildAnnIndex(); + auto q = gen.draw(); + auto want = annKeys(store.similaritySearchVector(q, 10, "", -1.0f)); + + EmbeddingStore copy = store; // the copy's vectors live elsewhere: it rebuilds lazily + ASSERT_FALSE(copy.annStats().active); + ASSERT_TRUE(copy.annStats().enabled); + ASSERT_EQ(annKeys(copy.similaritySearchVector(q, 10, "", -1.0f)), want); + copy.add(NodeRef("extra", "1"), "", q); + ASSERT_EQ(annKey(copy.similaritySearchVector(q, 1)[0]), "extra:1"); + ASSERT_EQ(store.count(), 1500u); // the original is untouched + + EmbeddingStore moved = std::move(store); + ASSERT_EQ(annKeys(moved.similaritySearchVector(q, 10, "", -1.0f)), want); + moved.remove(NodeRef("doc", "0")); + ASSERT_EQ(moved.count(), 1499u); +} + +TEST(ann_concurrent_searches_are_safe) { + ClusteredData gen(30, 24, 21); + auto data = gen.drawMany(3000); + EmbeddingStore store; + store.enableAnn(annOptions(32, 4, 0)); + fillStore(store, data); + auto queries = gen.drawMany(40); + std::atomic<int> bad{0}; + auto reader = [&](int seed) { + for (int i = 0; i < 100; ++i) { + auto res = store.similaritySearchVector(queries[(i + seed) % 40], 8, "", -1.0f); + std::set<std::string> seen; + bool ok = res.size() == 8; + for (size_t j = 0; j < res.size(); ++j) { + ok = ok && seen.insert(annKey(res[j])).second && (j == 0 || res[j - 1].score >= res[j].score); + } + if (!ok) ++bad; + } + }; + std::vector<std::thread> pool; + for (int t = 0; t < 4; ++t) pool.emplace_back(reader, t); + for (auto& t : pool) t.join(); + ASSERT_EQ(bad.load(), 0); + ASSERT_EQ(store.annStats().builds, 1u); // four threads, one lazy build +} + // ============================================================================= // Main // ============================================================================= diff --git a/isongraph-vector-cpp/tests/test_isongraph_vector_curl.cpp b/isongraph-vector-cpp/tests/test_isongraph_vector_curl.cpp index c1605d2..011cb17 100644 --- a/isongraph-vector-cpp/tests/test_isongraph_vector_curl.cpp +++ b/isongraph-vector-cpp/tests/test_isongraph_vector_curl.cpp @@ -1,13 +1,60 @@ /** - * libcurl transport smoke test. Built only with -DISONGRAPH_VECTOR_CURL=ON. + * libcurl transport tests. Built only with -DISONGRAPH_VECTOR_CURL=ON. * - * No network: the request goes to a port on the loopback interface where - * nothing listens, so what is checked is that the transport is wired up and - * reports a refused connection as an HttpError rather than crashing. + * No internet: every request goes to a tiny HTTP/1.1 server started inside the + * test process, listening on 127.0.0.1 on an ephemeral port. It answers with + * what OpenAI-compatible and Ollama endpoints answer, and misbehaves on demand + * (error statuses, a stall, a redirect, an empty reply, a chunked or enormous + * body), so what is checked is real libcurl talking real HTTP: the request the + * server sees, the response the transport hands back, and the errors it + * raises - including that the API key never ends up in one. */ +#include <atomic> +#include <chrono> +#include <condition_variable> +#include <cstdio> +#include <cstdlib> +#include <cstring> +#include <functional> #include <iostream> +#include <mutex> +#include <sstream> #include <string> +#include <thread> +#include <vector> + +#ifdef _WIN32 +// winsock2.h pulls in windows.h, whose min/max macros break std::min/std::max +// (and numeric_limits<T>::max()) in every header that follows. +#ifndef NOMINMAX +#define NOMINMAX +#endif +#ifndef WIN32_LEAN_AND_MEAN +#define WIN32_LEAN_AND_MEAN +#endif +#include <winsock2.h> +#include <ws2tcpip.h> +using socket_t = SOCKET; +static const socket_t BAD_SOCKET = INVALID_SOCKET; +static void closeSocket(socket_t s) { closesocket(s); } +static const int SEND_FLAGS = 0; +#else +#include <arpa/inet.h> +#include <csignal> +#include <netinet/in.h> +#include <sys/select.h> +#include <sys/socket.h> +#include <unistd.h> +using socket_t = int; +static const socket_t BAD_SOCKET = -1; +static void closeSocket(socket_t s) { close(s); } +#ifdef MSG_NOSIGNAL +static const int SEND_FLAGS = MSG_NOSIGNAL; +#else +static const int SEND_FLAGS = 0; // macOS: SO_NOSIGPIPE is set on each socket instead +#endif +#endif #include "../include/isongraph_vector_http.hpp" @@ -21,7 +68,7 @@ int tests_passed = 0; struct TestRunner_##name { \ TestRunner_##name() { \ tests_run++; \ - std::cout << "Running: " << #name << "... "; \ + std::cout << "Running: " << #name << "... " << std::flush; \ try { \ test_##name(); \ tests_passed++; \ @@ -37,46 +84,765 @@ int tests_passed = 0; #define ASSERT_TRUE(expr) \ if (!(expr)) throw std::runtime_error("Assertion failed: " #expr); +#define ASSERT_FALSE(expr) \ + if (expr) throw std::runtime_error("Assertion failed: " #expr " should be false"); +#define ASSERT_EQ(a, b) \ + if (!((a) == (b))) throw std::runtime_error("Assertion failed: " #a " != " #b); #define ASSERT_THROWS(expr, exception_type) \ { \ bool threw = false; \ try { expr; } catch (const exception_type&) { threw = true; } \ - if (!threw) throw std::runtime_error("Assertion failed: " #expr " did not throw"); \ + if (!threw) throw std::runtime_error("Assertion failed: " #expr " did not throw " #exception_type); \ + } + +namespace { + +// ============================================================================= +// A small in-process HTTP/1.1 server +// ============================================================================= + +struct Received { + std::string method; + std::string target; + std::map<std::string, std::string> headers; // names lower-cased + std::string body; + + std::string header(const std::string& name) const { + auto it = headers.find(name); + return it == headers.end() ? std::string() : it->second; + } +}; + +struct Reply { + enum class Mode { + Normal, // status line, headers, Content-Length body + Chunked, // same, Transfer-Encoding: chunked in three pieces + NoReply, // close the connection without answering + Stall, // read the request, then say nothing until the test ends + Huge, // announce and stream `hugeBytes` of body + }; + Mode mode = Mode::Normal; + int status = 200; + std::vector<std::pair<std::string, std::string>> headers; + std::string body; + size_t hugeBytes = 0; +}; + +Reply json(int status, const std::string& body) { + Reply r; + r.status = status; + r.headers.emplace_back("Content-Type", "application/json"); + r.body = body; + return r; +} + +const char* reasonFor(int status) { + switch (status) { + case 200: return "OK"; + case 301: return "Moved Permanently"; + case 302: return "Found"; + case 401: return "Unauthorized"; + case 404: return "Not Found"; + case 429: return "Too Many Requests"; + case 500: return "Internal Server Error"; + case 503: return "Service Unavailable"; + default: return "Status"; + } +} + +class LoopbackServer { +public: + using Handler = std::function<Reply(const Received&, size_t requestNumber)>; + + explicit LoopbackServer(Handler handler) : handler_(std::move(handler)) { +#ifdef _WIN32 + WSADATA data; + WSAStartup(MAKEWORD(2, 2), &data); +#else + std::signal(SIGPIPE, SIG_IGN); +#endif + listener_ = ::socket(AF_INET, SOCK_STREAM, 0); + if (listener_ == BAD_SOCKET) throw std::runtime_error("cannot create a socket"); + int yes = 1; + setsockopt(listener_, SOL_SOCKET, SO_REUSEADDR, reinterpret_cast<const char*>(&yes), sizeof yes); + sockaddr_in address; + std::memset(&address, 0, sizeof address); + address.sin_family = AF_INET; + address.sin_addr.s_addr = htonl(INADDR_LOOPBACK); + address.sin_port = 0; + if (::bind(listener_, reinterpret_cast<sockaddr*>(&address), sizeof address) != 0 || + ::listen(listener_, 8) != 0) { + closeSocket(listener_); + throw std::runtime_error("cannot listen on the loopback interface"); + } + socklen_t length = sizeof address; + getsockname(listener_, reinterpret_cast<sockaddr*>(&address), &length); + port_ = ntohs(address.sin_port); + thread_ = std::thread([this] { run(); }); + } + + ~LoopbackServer() { + stop_ = true; + if (thread_.joinable()) thread_.join(); + closeSocket(listener_); + } + + int port() const { return port_; } + std::string url(const std::string& path = "/v1/embeddings") const { + return "http://127.0.0.1:" + std::to_string(port_) + path; + } + + size_t requestCount() const { + std::lock_guard<std::mutex> lock(mutex_); + return requests_.size(); + } + Received request(size_t index) const { + std::lock_guard<std::mutex> lock(mutex_); + return requests_.at(index); + } + +private: + Handler handler_; + socket_t listener_ = BAD_SOCKET; + int port_ = 0; + std::atomic<bool> stop_{false}; + std::thread thread_; + mutable std::mutex mutex_; + std::vector<Received> requests_; + + void run() { + while (!stop_) { + fd_set readable; + FD_ZERO(&readable); + FD_SET(listener_, &readable); + timeval wait; + wait.tv_sec = 0; + wait.tv_usec = 50 * 1000; + int ready = select(static_cast<int>(listener_) + 1, &readable, nullptr, nullptr, &wait); + if (ready <= 0) continue; + socket_t client = ::accept(listener_, nullptr, nullptr); + if (client == BAD_SOCKET) continue; +#ifdef SO_NOSIGPIPE + int one = 1; + setsockopt(client, SOL_SOCKET, SO_NOSIGPIPE, &one, sizeof one); +#endif + try { + serve(client); + } catch (...) { + // a client that gave up mid-exchange is not a server failure + } + closeSocket(client); + } + } + + static bool sendAll(socket_t s, const char* data, size_t size) { + while (size > 0) { + int sent = static_cast<int>(::send(s, data, static_cast<int>(std::min<size_t>(size, 1 << 20)), + SEND_FLAGS)); + if (sent <= 0) return false; + data += sent; + size -= static_cast<size_t>(sent); + } + return true; } -TEST(refused_connection_is_an_http_error) { + /** Reads one request, or false when the client closed first. */ + bool readRequest(socket_t s, Received& out) { + std::string buffer; + size_t headerEnd = std::string::npos; + char chunk[16384]; + while ((headerEnd = buffer.find("\r\n\r\n")) == std::string::npos) { + int got = static_cast<int>(::recv(s, chunk, sizeof chunk, 0)); + if (got <= 0) return false; + buffer.append(chunk, static_cast<size_t>(got)); + if (buffer.size() > (1u << 20)) return false; + } + std::istringstream head(buffer.substr(0, headerEnd)); + std::string line; + std::getline(head, line); + { + std::istringstream request(line); + std::string version; + request >> out.method >> out.target >> version; + } + while (std::getline(head, line)) { + if (!line.empty() && line.back() == '\r') line.pop_back(); + size_t colon = line.find(':'); + if (colon == std::string::npos) continue; + std::string name = line.substr(0, colon); + for (char& c : name) c = static_cast<char>(std::tolower(static_cast<unsigned char>(c))); + std::string value = line.substr(colon + 1); + while (!value.empty() && value.front() == ' ') value.erase(value.begin()); + out.headers[name] = value; + } + if (out.header("expect") == "100-continue") { + const char* go = "HTTP/1.1 100 Continue\r\n\r\n"; + sendAll(s, go, std::strlen(go)); + } + size_t length = 0; + if (!out.header("content-length").empty()) length = std::strtoull(out.header("content-length").c_str(), nullptr, 10); + out.body = buffer.substr(headerEnd + 4); + while (out.body.size() < length) { + int got = static_cast<int>(::recv(s, chunk, sizeof chunk, 0)); + if (got <= 0) return false; + out.body.append(chunk, static_cast<size_t>(got)); + } + out.body.resize(length); + return true; + } + + void serve(socket_t s) { + Received received; + if (!readRequest(s, received)) return; + size_t number; + { + std::lock_guard<std::mutex> lock(mutex_); + requests_.push_back(received); + number = requests_.size() - 1; + } + Reply reply = handler_(received, number); + + if (reply.mode == Reply::Mode::NoReply) return; + if (reply.mode == Reply::Mode::Stall) { + // Say nothing until the client gives up (so the next connection is + // served at once) or the test ends. + for (int i = 0; i < 400 && !stop_; ++i) { + fd_set readable; + FD_ZERO(&readable); + FD_SET(s, &readable); + timeval wait; + wait.tv_sec = 0; + wait.tv_usec = 10 * 1000; + if (select(static_cast<int>(s) + 1, &readable, nullptr, nullptr, &wait) > 0) { + char probe; + if (::recv(s, &probe, 1, MSG_PEEK) <= 0) return; // closed by the client + } + } + return; + } + + std::string head = "HTTP/1.1 " + std::to_string(reply.status) + " " + reasonFor(reply.status) + "\r\n"; + for (const auto& [name, value] : reply.headers) head += name + ": " + value + "\r\n"; + head += "Connection: close\r\n"; + if (reply.mode == Reply::Mode::Huge) { + head += "Content-Length: " + std::to_string(reply.hugeBytes) + "\r\n\r\n"; + if (!sendAll(s, head.data(), head.size())) return; + std::string block(1 << 20, 'a'); + for (size_t sent = 0; sent < reply.hugeBytes;) { + size_t n = std::min(block.size(), reply.hugeBytes - sent); + if (!sendAll(s, block.data(), n)) return; // the client stopped reading, as it should + sent += n; + } + } else if (reply.mode == Reply::Mode::Chunked) { + head += "Transfer-Encoding: chunked\r\n\r\n"; + sendAll(s, head.data(), head.size()); + size_t third = reply.body.size() / 3; + std::string parts[3] = {reply.body.substr(0, third), reply.body.substr(third, third), + reply.body.substr(2 * third)}; + for (const auto& part : parts) { + char size[32]; + std::snprintf(size, sizeof size, "%zx\r\n", part.size()); + std::string piece = std::string(size) + part + "\r\n"; + sendAll(s, piece.data(), piece.size()); + } + sendAll(s, "0\r\n\r\n", 5); + } else { + head += "Content-Length: " + std::to_string(reply.body.size()) + "\r\n\r\n"; + std::string whole = head + reply.body; + sendAll(s, whole.data(), whole.size()); + } +#ifdef _WIN32 + shutdown(s, SD_SEND); +#else + shutdown(s, SHUT_WR); +#endif + } +}; + +// ============================================================================= +// Canned responses and helpers +// ============================================================================= + +const char* const KEY = "sk-test-0123456789abcdef"; + +/** OpenAI-shaped answer for two inputs, deliberately out of order (index says which is which). */ +const char* const OPENAI_TWO = + "{\"object\":\"list\",\"data\":[" + "{\"object\":\"embedding\",\"index\":1,\"embedding\":[0.0,1.0,0.0,0.0]}," + "{\"object\":\"embedding\",\"index\":0,\"embedding\":[1.0,0.0,0.0,0.0]}]," + "\"model\":\"text-embedding-3-small\",\"usage\":{\"prompt_tokens\":4,\"total_tokens\":4}}"; + +const char* const OLLAMA_TWO = + "{\"model\":\"nomic-embed-text\",\"embeddings\":[[1.0,0.0,0.0,0.0],[0.0,1.0,0.0,0.0]]," + "\"total_duration\":123}"; + +HttpEncoderOptions optionsFor(const LoopbackServer& server, const std::string& path = "/v1/embeddings") { + HttpEncoderOptions options; + options.url = server.url(path); + options.model = "test-embed"; + options.dimension = 4; + options.maxRetries = 0; + options.timeoutMs = 5000; + options.sleeper = [](std::chrono::milliseconds) {}; + return options; +} + +bool contains(const std::string& haystack, const std::string& needle) { + return haystack.find(needle) != std::string::npos; +} + +std::string lower(std::string s) { + for (char& c : s) c = static_cast<char>(std::tolower(static_cast<unsigned char>(c))); + return s; +} + +} // namespace + +// ============================================================================= +// Shapes: OpenAI and Ollama, through the default (libcurl) transport +// ============================================================================= + +TEST(openai_shaped_response_is_decoded_in_index_order_and_the_request_is_well_formed) { + LoopbackServer server([](const Received&, size_t) { return json(200, OPENAI_TWO); }); + auto options = optionsFor(server); + options.apiKey = KEY; + HttpEncoder encoder(options); // no transport given: CurlTransport + + auto vectors = encoder.encodeBatch({"first", "second"}); + ASSERT_EQ(vectors.size(), 2u); + ASSERT_EQ(vectors[0], (std::vector<float>{1.0f, 0.0f, 0.0f, 0.0f})); // index 0, though listed second + ASSERT_EQ(vectors[1], (std::vector<float>{0.0f, 1.0f, 0.0f, 0.0f})); + + ASSERT_EQ(server.requestCount(), 1u); + Received seen = server.request(0); + ASSERT_EQ(seen.method, "POST"); + ASSERT_EQ(seen.target, "/v1/embeddings"); + ASSERT_EQ(seen.header("content-type"), "application/json"); + ASSERT_EQ(seen.header("accept"), "application/json"); + ASSERT_EQ(seen.header("authorization"), std::string("Bearer ") + KEY); + ASSERT_EQ(seen.body, "{\"model\":\"test-embed\",\"input\":[\"first\",\"second\"]}"); + ASSERT_EQ(seen.header("content-length"), std::to_string(seen.body.size())); + ASSERT_EQ(seen.header("expect"), ""); // no 100-continue round trip +} + +TEST(ollama_shaped_response_is_decoded_and_no_credential_is_sent_without_a_key) { + LoopbackServer server([](const Received&, size_t) { return json(200, OLLAMA_TWO); }); + HttpEncoder encoder(optionsFor(server, "/api/embed")); + + auto vectors = encoder.encodeBatch({"a", "b"}); + ASSERT_EQ(vectors.size(), 2u); + ASSERT_EQ(vectors[1], (std::vector<float>{0.0f, 1.0f, 0.0f, 0.0f})); + Received seen = server.request(0); + ASSERT_EQ(seen.target, "/api/embed"); + ASSERT_EQ(seen.header("authorization"), ""); +} + +TEST(a_custom_key_header_and_extra_headers_reach_the_server) { + LoopbackServer server([](const Received&, size_t) { return json(200, OPENAI_TWO); }); + auto options = optionsFor(server); + options.apiKey = KEY; + options.apiKeyHeader = "api-key"; + options.apiKeyPrefix = ""; + options.headers["X-Tenant"] = "acme"; + HttpEncoder(options).encodeBatch({"a", "b"}); + Received seen = server.request(0); + ASSERT_EQ(seen.header("api-key"), KEY); + ASSERT_EQ(seen.header("x-tenant"), "acme"); + ASSERT_EQ(seen.header("authorization"), ""); +} + +TEST(a_chunked_response_is_reassembled) { + LoopbackServer server([](const Received&, size_t) { + Reply r = json(200, OLLAMA_TWO); + r.mode = Reply::Mode::Chunked; + return r; + }); + auto vectors = HttpEncoder(optionsFor(server)).encodeBatch({"a", "b"}); + ASSERT_EQ(vectors.size(), 2u); + ASSERT_EQ(vectors[0], (std::vector<float>{1.0f, 0.0f, 0.0f, 0.0f})); +} + +TEST(a_large_request_body_arrives_whole_and_without_waiting_for_100_continue) { + LoopbackServer server([](const Received&, size_t) { return json(200, OLLAMA_TWO); }); + auto options = optionsFor(server); + options.batchSize = 2; + std::string big(3 * 1024 * 1024, 'x'); + auto started = std::chrono::steady_clock::now(); + HttpEncoder(options).encodeBatch({big, "y"}); + auto elapsed = std::chrono::steady_clock::now() - started; + Received seen = server.request(0); + ASSERT_EQ(seen.body.size(), big.size() + std::string("{\"model\":\"test-embed\",\"input\":[\"\",\"y\"]}").size()); + ASSERT_EQ(seen.header("expect"), ""); + ASSERT_TRUE(elapsed < std::chrono::milliseconds(900)); // libcurl would wait 1 s for a 100-continue +} + +TEST(the_transport_reports_status_body_and_lower_cased_headers) { + LoopbackServer server([](const Received&, size_t) { + Reply r = json(429, "{\"error\":\"slow down\"}"); + r.headers.emplace_back("Retry-After", "7"); + r.headers.emplace_back("X-Request-Id", "abc"); + return r; + }); HttpRequest request; - request.url = "http://127.0.0.1:1/v1/embeddings"; + request.url = server.url(); request.body = "{}"; request.headers["Content-Type"] = "application/json"; - request.timeoutMs = 2000; + HttpResponse response = CurlTransport()(request); // any status is a response, not an error + ASSERT_EQ(response.status, 429); + ASSERT_EQ(response.body, "{\"error\":\"slow down\"}"); + ASSERT_EQ(response.headers.at("retry-after"), "7"); + ASSERT_EQ(response.headers.at("x-request-id"), "abc"); +} + +// ============================================================================= +// Failures: statuses, retries, timeouts, refused connections, empty replies +// ============================================================================= + +TEST(a_client_error_is_an_http_error_with_the_server_text_and_is_not_retried) { + LoopbackServer server([](const Received&, size_t) { + return json(404, "{\"error\":{\"message\":\"model not found\"}}"); + }); + auto options = optionsFor(server); + options.maxRetries = 3; + HttpEncoder encoder(options); + try { + encoder.encode("x"); + throw std::runtime_error("expected an HttpError"); + } catch (const HttpError& error) { + ASSERT_EQ(error.status(), 404); + ASSERT_TRUE(contains(error.what(), "HTTP 404")); + ASSERT_TRUE(contains(error.what(), "model not found")); + } + ASSERT_EQ(server.requestCount(), 1u); +} + +TEST(a_server_error_is_retried_and_a_later_success_is_returned) { + LoopbackServer server([](const Received&, size_t number) { + return number < 2 ? json(503, "{\"error\":\"overloaded\"}") : json(200, OLLAMA_TWO); + }); + auto options = optionsFor(server); + options.maxRetries = 3; + HttpEncoder encoder(options); + auto vectors = encoder.encodeBatch({"a", "b"}); + ASSERT_EQ(vectors.size(), 2u); + ASSERT_EQ(server.requestCount(), 3u); +} - CurlTransport transport; - ASSERT_THROWS(transport(request), HttpError); +TEST(a_persistent_server_error_gives_up_after_the_configured_retries) { + LoopbackServer server([](const Received&, size_t) { return json(500, "{\"error\":\"boom\"}"); }); + auto options = optionsFor(server); + options.maxRetries = 2; + HttpEncoder encoder(options); + ASSERT_THROWS(encoder.encode("x"), HttpError); + ASSERT_EQ(server.requestCount(), 3u); } -TEST(encoder_uses_curl_when_no_transport_is_given) { +TEST(retry_after_from_a_real_response_sets_the_wait) { + LoopbackServer server([](const Received&, size_t number) { + if (number > 0) return json(200, OLLAMA_TWO); + Reply r = json(429, "{\"error\":\"rate limited\"}"); + r.headers.emplace_back("Retry-After", "3"); + return r; + }); + auto options = optionsFor(server); + options.maxRetries = 1; + options.maxBackoffMs = 60000; + std::vector<long long> waits; + options.sleeper = [&](std::chrono::milliseconds ms) { waits.push_back(ms.count()); }; + HttpEncoder(options).encodeBatch({"a", "b"}); + ASSERT_EQ(waits.size(), 1u); + ASSERT_EQ(waits[0], 3000); +} + +TEST(a_stalled_server_times_out_within_the_budget) { + LoopbackServer server([](const Received&, size_t) { + Reply r; + r.mode = Reply::Mode::Stall; + return r; + }); + auto options = optionsFor(server); + options.timeoutMs = 300; + HttpEncoder encoder(options); + auto started = std::chrono::steady_clock::now(); + try { + encoder.encode("x"); + throw std::runtime_error("expected an HttpError"); + } catch (const HttpError& error) { + ASSERT_EQ(error.status(), 0); // no response at all + ASSERT_TRUE(contains(lower(error.what()), "time")); + } + auto elapsed = std::chrono::steady_clock::now() - started; + ASSERT_TRUE(elapsed < std::chrono::milliseconds(3000)); +} + +TEST(a_timeout_is_retried_like_any_transport_failure) { + LoopbackServer server([](const Received&, size_t number) { + if (number == 0) { + Reply r; + r.mode = Reply::Mode::Stall; + return r; + } + return json(200, OLLAMA_TWO); + }); + auto options = optionsFor(server); + options.timeoutMs = 300; + options.maxRetries = 1; + auto vectors = HttpEncoder(options).encodeBatch({"a", "b"}); + ASSERT_EQ(vectors.size(), 2u); + ASSERT_EQ(server.requestCount(), 2u); +} + +TEST(a_refused_connection_is_an_http_error) { + int deadPort; + { + LoopbackServer gone([](const Received&, size_t) { return json(200, "{}"); }); + deadPort = gone.port(); + } // nothing listens there any more HttpEncoderOptions options; - options.url = "http://127.0.0.1:1/v1/embeddings"; + options.url = "http://127.0.0.1:" + std::to_string(deadPort) + "/v1/embeddings"; options.model = "m"; options.dimension = 4; options.maxRetries = 0; options.timeoutMs = 2000; + HttpEncoder encoder(options); + try { + encoder.encode("hello"); + throw std::runtime_error("expected an HttpError"); + } catch (const HttpError& error) { + ASSERT_EQ(error.status(), 0); + ASSERT_TRUE(contains(error.what(), "HTTP request failed")); + } +} + +TEST(a_connection_closed_without_a_reply_is_an_http_error) { + LoopbackServer server([](const Received&, size_t) { + Reply r; + r.mode = Reply::Mode::NoReply; + return r; + }); + HttpEncoder encoder(optionsFor(server)); + try { + encoder.encode("x"); + throw std::runtime_error("expected an HttpError"); + } catch (const HttpError& error) { + ASSERT_EQ(error.status(), 0); + } +} + +TEST(https_to_a_plain_http_port_is_a_transport_error_that_does_not_leak_the_key) { + LoopbackServer server([](const Received&, size_t) { return json(200, OLLAMA_TWO); }); + auto options = optionsFor(server); + options.url = "https://127.0.0.1:" + std::to_string(server.port()) + "/v1/embeddings"; + options.apiKey = KEY; + options.timeoutMs = 1500; // the plain server never answers a TLS hello; some TLS stacks wait + try { + HttpEncoder(options).encode("x"); + throw std::runtime_error("expected an HttpError"); + } catch (const HttpError& error) { + ASSERT_EQ(error.status(), 0); + ASSERT_FALSE(contains(error.what(), KEY)); + } +} + +TEST(a_malformed_success_body_is_an_embedding_error_not_a_crash) { + LoopbackServer server([](const Received&, size_t) { return json(200, "{\"data\": [ not json"); }); + HttpEncoder encoder(optionsFor(server)); + ASSERT_THROWS(encoder.encode("x"), EmbeddingError); + LoopbackServer other([](const Received&, size_t) { return json(200, "{\"unexpected\":true}"); }); + ASSERT_THROWS(HttpEncoder(optionsFor(other)).encode("x"), EmbeddingError); +} + +TEST(a_response_body_past_the_cap_is_refused_with_a_clear_message) { + LoopbackServer server([](const Received&, size_t) { + Reply r = json(200, ""); + r.mode = Reply::Mode::Huge; + r.hugeBytes = 65u * 1024u * 1024u; // one MiB over the transport's 64 MiB limit + return r; + }); + HttpRequest request; + request.url = server.url(); + request.body = "{}"; + request.timeoutMs = 20000; + try { + CurlTransport()(request); + throw std::runtime_error("expected an HttpError"); + } catch (const HttpError& error) { + ASSERT_TRUE(contains(error.what(), "64 MiB")); + } +} - HttpEncoder encoder(options); // would throw "no transport" without ISONGRAPH_VECTOR_CURL - ASSERT_THROWS(encoder.encode("hello"), HttpError); +// ============================================================================= +// Safety: redirects, protocols, and the API key +// ============================================================================= + +TEST(a_redirect_is_not_followed_and_the_key_is_never_forwarded) { + LoopbackServer elsewhere([](const Received&, size_t) { return json(200, OLLAMA_TWO); }); + std::string target = elsewhere.url("/stolen"); + LoopbackServer origin([&](const Received&, size_t) { + Reply r = json(302, ""); + r.headers.emplace_back("Location", target); + return r; + }); + auto options = optionsFor(origin); + options.apiKey = KEY; + HttpEncoder encoder(options); + try { + encoder.encode("x"); + throw std::runtime_error("expected an HttpError"); + } catch (const HttpError& error) { + ASSERT_EQ(error.status(), 302); + ASSERT_FALSE(contains(error.what(), KEY)); + } + ASSERT_EQ(origin.requestCount(), 1u); + ASSERT_EQ(elsewhere.requestCount(), 0u); // never contacted + + // and at the transport level the 302 is simply returned + HttpRequest request; + request.url = origin.url(); + request.body = "{}"; + request.headers["Authorization"] = std::string("Bearer ") + KEY; + HttpResponse response = CurlTransport()(request); + ASSERT_EQ(response.status, 302); + ASSERT_EQ(elsewhere.requestCount(), 0u); } -TEST(rejects_a_non_post_request) { +TEST(only_http_and_https_are_allowed) { HttpRequest request; + request.url = "file:///etc/hostname"; + request.body = "{}"; + ASSERT_THROWS(CurlTransport()(request), HttpError); + request.url = "ftp://127.0.0.1:1/x"; + ASSERT_THROWS(CurlTransport()(request), HttpError); + request.url = "gopher://127.0.0.1:1/x"; + ASSERT_THROWS(CurlTransport()(request), HttpError); request.method = "GET"; request.url = "http://127.0.0.1:1/"; - CurlTransport transport; - ASSERT_THROWS(transport(request), HttpError); + ASSERT_THROWS(CurlTransport()(request), HttpError); +} + +TEST(the_api_key_never_appears_in_any_error) { + // 1. an error page that echoes the credential back, in each shape the encoder reads + const std::string echoed = std::string("invalid api key ") + KEY; + std::vector<std::string> bodies = { + "{\"error\":\"" + echoed + "\"}", + "{\"error\":{\"message\":\"" + echoed + "\"}}", + "{\"message\":\"" + echoed + "\"}", + "<html>" + echoed + "</html>", + }; + for (const auto& body : bodies) { + LoopbackServer server([&](const Received&, size_t) { return json(401, body); }); + auto options = optionsFor(server); + options.apiKey = KEY; + try { + HttpEncoder(options).encode("x"); + throw std::runtime_error("expected an HttpError"); + } catch (const HttpError& error) { + ASSERT_EQ(error.status(), 401); + ASSERT_FALSE(contains(error.what(), KEY)); + ASSERT_TRUE(contains(error.what(), "[redacted]")); + } + } + + // 2. a 200 whose error object echoes it + { + LoopbackServer server([&](const Received&, size_t) { + return json(200, "{\"error\":{\"message\":\"" + echoed + "\"}}"); + }); + auto options = optionsFor(server); + options.apiKey = KEY; + try { + HttpEncoder(options).encode("x"); + throw std::runtime_error("expected an error"); + } catch (const EmbeddingError& error) { + ASSERT_FALSE(contains(error.what(), KEY)); + } + } + + // 3. transport-level failures: the key travels in a header and a header is + // never part of libcurl's error text + { + LoopbackServer server([](const Received&, size_t) { + Reply r; + r.mode = Reply::Mode::NoReply; + return r; + }); + auto options = optionsFor(server); + options.apiKey = KEY; + try { + HttpEncoder(options).encode("x"); + throw std::runtime_error("expected an HttpError"); + } catch (const HttpError& error) { + ASSERT_FALSE(contains(error.what(), KEY)); + } + ASSERT_EQ(server.request(0).header("authorization"), std::string("Bearer ") + KEY); // it was sent... + } + { + LoopbackServer stall([](const Received&, size_t) { + Reply r; + r.mode = Reply::Mode::Stall; + return r; + }); + auto options = optionsFor(stall); + options.apiKey = KEY; + options.timeoutMs = 200; + try { + HttpEncoder(options).encode("x"); + throw std::runtime_error("expected an HttpError"); + } catch (const HttpError& error) { + ASSERT_FALSE(contains(error.what(), KEY)); // ...and still not in the timeout text + } + } + + // 4. a key that is also part of the URL (a query-string credential) is scrubbed from curl's text + { + int deadPort; + { + LoopbackServer gone([](const Received&, size_t) { return json(200, "{}"); }); + deadPort = gone.port(); + } + HttpEncoderOptions options; + options.url = "http://127.0.0.1:" + std::to_string(deadPort) + "/v1/embeddings?key=" + KEY; + options.model = "m"; + options.apiKey = KEY; + options.maxRetries = 0; + options.timeoutMs = 1000; + try { + HttpEncoder(options).encode("x"); + throw std::runtime_error("expected an HttpError"); + } catch (const HttpError& error) { + ASSERT_FALSE(contains(error.what(), KEY)); + } + } +} + +TEST(the_encoder_can_drive_a_semantic_graph_over_real_http) { + LoopbackServer server([](const Received& request, size_t) { + // one vector per input: inputs are the strings of the "input" array + // (none of these texts contains a comma) + size_t start = request.body.find("\"input\":["); + size_t count = 1; + for (size_t at = request.body.find("\",\"", start); at != std::string::npos; + at = request.body.find("\",\"", at + 1)) { + ++count; + } + std::string reply = "{\"data\":["; + for (size_t i = 0; i < count; ++i) { + if (i > 0) reply += ","; + reply += "{\"index\":" + std::to_string(i) + ",\"embedding\":[" + + (i % 2 == 0 ? "1.0,0.0,0.0,0.0" : "0.0,1.0,0.0,0.0") + "]}"; + } + return json(200, reply + "]}"); + }); + auto options = optionsFor(server); + auto encoder = std::make_shared<HttpEncoder>(options); + SemanticGraph graph("remote", encoder); + graph.addNode("person", "1", {{"name", "Alice"}}); + graph.addNode("person", "2", {{"name", "Bob"}}); + ASSERT_EQ(graph.embeddingStore().count(), 2u); + ASSERT_EQ(graph.getEmbedding(NodeRef("person", "1"))->model, "test-embed"); } int main() { - std::cout << "\n=== isongraph-vector libcurl transport tests ===" << std::endl << std::endl; std::cout << std::endl << "=== Results ===" << std::endl; std::cout << "Tests passed: " << tests_passed << "/" << tests_run << std::endl; return (tests_passed == tests_run) ? 0 : 1; diff --git a/isongraph-vector-csharp/README.md b/isongraph-vector-csharp/README.md index 1dc1dbf..adac3b0 100644 --- a/isongraph-vector-csharp/README.md +++ b/isongraph-vector-csharp/README.md @@ -323,6 +323,54 @@ search rather than filtering its results. `Dispose` closes the connection and clears the ADO.NET pool, so the database file is genuinely released and can be moved or deleted. +## Approximate search (opt-in) + +Search is exact by default. For a store past a few hundred thousand vectors you +can opt in to an in-memory IVF-flat index: the vectors are clustered by direction +and a query scores only the cells nearest to it. + +```csharp +var store = new EmbeddingStore(encoder, ann: AnnMode.Ivf); // or SemanticGraph(..., ann: AnnMode.Ivf) +var hits = store.SimilaritySearch("…", topK: 10); // from the index (once ≥ 50,000 vectors) +var truth = store.SimilaritySearch("…", topK: 10, exact: true); // per-call full scan + +RecallReport report = store.RecallCheck(myRealQueries, k: 10); // recall@10 against the exact scan +// report.Recall, report.MinRecall, report.AnnMs, report.ExactMs +store.AnnNprobe = 64; // raise until the recall you need holds +``` + +`annNlist` (default about 4 x sqrt(N)), `annNprobe` (default 5% of the cells), +`annMinSize` (default 50,000; below it every search is exact) and `annSeed` +(seeds the k-means, so the same data gives the same index) tune it. +`GetAnnStats()` shows what it is doing; `BuildAnnIndex()` builds it ahead of the +first search. + +It is approximate: a true neighbour in a cell that is not probed is missed, and +how many depends on how your data clusters, so measure with `RecallCheck` on real +queries. Scores are the same cosine values the exact scan returns. It applies to +bounded top-k searches; `topK: null`, `exact: true`, a zero query and +`ScoreMap` (and so `SemanticMultiHop` with `Blend > 0`) use the exact scan. +`nodeType`, `threshold` and `nodeFilter` apply to the probed rows; when fewer than +`topK` survive the probe is widened 4x and retried, ending in the exact scan, +so a selective filter costs time, not results. + +The index is built lazily by the first search, never saved (nothing about it is +in a graph file or sidecar, and the digest is unchanged), and kept current: +added and updated vectors are visible at once and folded into cells in batches, +removed ones never come back, and heavy churn retrains it. It holds its own +contiguous copy of the vectors, so budget about one extra copy of them. Writes and +searches share the store's lock, as everywhere else in the store. It is not +available on `SqliteEmbeddingStore`. + +Measured on synthetic clustered 384-d vectors (32 cores, .NET 8, top-10, warm medians): +1,000,000 vectors, 4,000 cells, built in 11 s - recall@10 0.963 at `nprobe` 1 +(0.7 ms), 0.983 at 64 (4.6 ms), 0.991 at the automatic 200 (10.6 ms), against +293 ms for this port's exact scan; 100,000 vectors: recall 1.000 from `nprobe` 4 +(0.14 ms) against 26 ms. Real embeddings will differ; use `RecallCheck`. + +`bench/IsonGraph.Vector.Bench` measures exact against indexed latency and recall: +`dotnet run -c Release --project bench/IsonGraph.Vector.Bench`. + ## One format, six languages The SQLite schema and the portable JSON payload are shared by every port — @@ -340,8 +388,8 @@ store.FromJson(json); ## Things that will bite you -- **Search is brute force**, sqlite-vec included: linear in the number of - nodes. This is built for an application's own knowledge graph, not as a +- **Search is brute force** by default, sqlite-vec included: linear in the number of + nodes (an opt-in approximate index for the in-memory store is described above). This is built for an application's own knowledge graph, not as a vector-database replacement. - **One embedding per node**, from one text field or a concatenation. No multi-vector nodes, no edge embeddings. @@ -376,7 +424,7 @@ store.FromJson(json); ## Tests ```bash -dotnet test # 222 tests +dotnet test # 250 tests ``` ## Links diff --git a/isongraph-vector-csharp/bench/IsonGraph.Vector.Bench/IsonGraph.Vector.Bench.csproj b/isongraph-vector-csharp/bench/IsonGraph.Vector.Bench/IsonGraph.Vector.Bench.csproj new file mode 100644 index 0000000..7fec73e --- /dev/null +++ b/isongraph-vector-csharp/bench/IsonGraph.Vector.Bench/IsonGraph.Vector.Bench.csproj @@ -0,0 +1,26 @@ +<Project Sdk="Microsoft.NET.Sdk"> + + <!-- + Approximate-index benchmark: exact scan vs IVF latency and recall@10 on + synthetic clustered vectors. Not part of the solution or the test run. + Run it with dotnet run, Release configuration, from the csharp directory; + the options are listed at the top of Program.cs. + --> + <PropertyGroup> + <OutputType>Exe</OutputType> + <TargetFramework>net8.0</TargetFramework> + <LangVersion>latest</LangVersion> + <Nullable>enable</Nullable> + <ImplicitUsings>enable</ImplicitUsings> + <TreatWarningsAsErrors>true</TreatWarningsAsErrors> + <IsPackable>false</IsPackable> + <ServerGarbageCollector>false</ServerGarbageCollector> + <ConcurrentGarbageCollection>false</ConcurrentGarbageCollection> + <InvariantGlobalization>true</InvariantGlobalization> + </PropertyGroup> + + <ItemGroup> + <ProjectReference Include="..\..\src\IsonGraph.Vector\IsonGraph.Vector.csproj" /> + </ItemGroup> + +</Project> diff --git a/isongraph-vector-csharp/bench/IsonGraph.Vector.Bench/Program.cs b/isongraph-vector-csharp/bench/IsonGraph.Vector.Bench/Program.cs new file mode 100644 index 0000000..ffa7a9b --- /dev/null +++ b/isongraph-vector-csharp/bench/IsonGraph.Vector.Bench/Program.cs @@ -0,0 +1,212 @@ +using System.Diagnostics; +using System.Globalization; +using System.Runtime.InteropServices; +using System.Text; +using IsonGraph; +using IsonGraph.Vector; + +// Exact scan vs the opt-in IVF index: latency and recall@10 on synthetic +// clustered unit vectors (default 384-d at 10k / 100k / 1M), mirroring +// benchmarks/ann.py so the numbers line up across ports. +// +// dotnet run -c Release --project bench/IsonGraph.Vector.Bench +// dotnet run -c Release --project bench/IsonGraph.Vector.Bench -- --sizes 1000000 --regimes clustered,diffuse --out ann_cs.md +// +// Options: --sizes a,b,c --regimes clustered[,diffuse] --dim 384 --queries 200 +// --nprobes 1,2,4,... --out file.md + +var options = Options.Parse(args); +var report = new StringBuilder(); +void Say(string line = "") { Console.WriteLine(line); report.AppendLine(line); } + +Say("# C# approximate index (IVF-flat) benchmark"); +Say(); +Say($"Machine: {Environment.ProcessorCount} logical CPUs, {RuntimeInformation.OSDescription}, " + + $"{RuntimeInformation.FrameworkDescription}, Vector<float>.Count = {System.Numerics.Vector<float>.Count}. " + + $"{options.Dim}-d unit vectors, top_k=10, {options.Queries} queries per cell, warm single-thread medians " + + "(index builds use all cores). Synthetic clustered data: recall depends on how the data clusters, " + + "so treat it as a property of these regimes, not of real embeddings."); +Say(); + +foreach (var n in options.Sizes) + foreach (var regimeName in options.Regimes) + RunOne(n, regimeName, options, Say); + +if (options.Out is not null) File.WriteAllText(options.Out, report.ToString()); +return 0; + +static void RunOne(int n, string regimeName, Options o, Action<string> say) +{ + var regime = new Regime(regimeName, n, o.Dim); + Console.Error.WriteLine($"[{regimeName} {n:N0}] generating ..."); + var data = regime.Sample(n, stream: 1); + var queries = regime.Sample(o.Queries, stream: 7); + + var store = new EmbeddingStore(ann: AnnMode.Ivf, annMinSize: 0, annSeed: 0); + var sw = Stopwatch.StartNew(); + for (int i = 0; i < n; i++) + store.Add(new NodeRef("doc", i.ToString(CultureInfo.InvariantCulture)), "", data[i]); + double ingest = n / sw.Elapsed.TotalSeconds; + Console.Error.WriteLine($"[{regimeName} {n:N0}] ingested ({ingest:N0}/s); exact baseline ..."); + + var (exactMs, truth) = Timed(store, queries, exact: true); + Console.Error.WriteLine($"[{regimeName} {n:N0}] exact median {Pct(exactMs, 0.5):F2} ms; building ..."); + + var stats = store.BuildAnnIndex(); + long gcAfter = GC.GetTotalMemory(false); + say($"## {regimeName}, N = {n:N0}"); + say(""); + say($"* exact scan: warm median {Pct(exactMs, 0.5):F2} ms, p95 {Pct(exactMs, 0.95):F2} ms"); + say($"* index build: {stats.BuildSeconds:F2} s for nlist={stats.Nlist} (automatic nprobe={stats.Nprobe}); " + + $"cells {stats.IndexBytes / 1048576.0:F1} MB plus {stats.VectorBytes / 1048576.0:F0} MB for the index's " + + $"contiguous copy of the vectors (the store's own {(long)n * o.Dim * 4 / 1048576} MB is separate); " + + $"managed heap {gcAfter / 1048576} MB"); + say($"* ingest {ingest:N0} vectors/s (the index is built lazily; ingest is unaffected)"); + say(""); + say("| nprobe | % of cells | recall@10 mean | p5 | worst | median ms | p95 ms | speed-up |"); + say("|---|---|---|---|---|---|---|---|"); + + int nlist = stats.Nlist!.Value; + int auto = stats.Nprobe!.Value; + var probes = o.Nprobes.Where(p => p < nlist).Append(auto).Distinct().OrderBy(p => p).ToList(); + foreach (int p in probes) + { + store.AnnNprobe = p; + var (ms, found) = Timed(store, queries, exact: false); + var recalls = found.Select((f, i) => truth[i].Count == 0 ? 1.0 + : (double)f.Intersect(truth[i]).Count() / truth[i].Count).ToList(); + double med = Pct(ms, 0.5); + say($"| {p}{(p == auto ? " (auto)" : "")} | {100.0 * p / nlist:F1} | {recalls.Average():F3} | " + + $"{Pct(recalls, 0.05):F2} | {recalls.Min():F2} | {med:F2} | {Pct(ms, 0.95):F2} | {Pct(exactMs, 0.5) / med:F1}x |"); + } + say(""); + var final = store.GetAnnStats(); + Console.Error.WriteLine( + $"[{regimeName} {n:N0}] done; searches approximate={final.Searches.Approximate} " + + $"exact={final.Searches.Exact} widened={final.Searches.Widened} builds={final.Searches.Builds}"); + GC.KeepAlive(data); +} + +static (List<double> Ms, List<HashSet<NodeRef>> Found) Timed(EmbeddingStore store, float[][] queries, bool exact) +{ + for (int i = 0; i < Math.Min(3, queries.Length); i++) // warm up (JIT, caches) + store.SimilaritySearchVector(queries[i], 10, exact: exact); + var ms = new List<double>(queries.Length); + var found = new List<HashSet<NodeRef>>(queries.Length); + foreach (var q in queries) + { + long t0 = Stopwatch.GetTimestamp(); + var res = store.SimilaritySearchVector(q, 10, exact: exact); + ms.Add(Stopwatch.GetElapsedTime(t0).TotalMilliseconds); + found.Add(res.Select(r => r.NodeRef).ToHashSet()); + } + return (ms, found); +} + +static double Pct(List<double> values, double q) +{ + var ordered = values.OrderBy(v => v).ToList(); + return ordered[Math.Min(ordered.Count - 1, (int)Math.Round(q * (ordered.Count - 1)))]; +} + +/// <summary>Deterministic generator of clustered unit vectors (the "clustered" and "diffuse" regimes of benchmarks/ann.py).</summary> +sealed class Regime +{ + private readonly int _dim; + private readonly float _noise; + private readonly float[][] _centers; + + public Regime(string name, int n, int dim) + { + _dim = dim; + bool diffuse = name == "diffuse"; + int perCluster = diffuse ? 40 : 100; + _noise = diffuse ? 2.0f : 1.0f; + int clusters = Math.Max(8, n / perCluster); + var rng = new Random(4242 + name.Length); + _centers = new float[clusters][]; + float[][]? supers = null; + if (diffuse) + { + supers = new float[20][]; + for (int s = 0; s < 20; s++) supers[s] = UnitGaussian(rng, dim); + } + for (int c = 0; c < clusters; c++) + { + var v = UnitGaussian(rng, dim); + if (supers is not null) + { + var parent = supers[rng.Next(supers.Length)]; + for (int d = 0; d < dim; d++) v[d] = parent[d] + 1.0f * v[d]; + Normalize(v); + } + _centers[c] = v; + } + } + + private static float Gauss(Random r) + { + double u1 = 1.0 - r.NextDouble(), u2 = r.NextDouble(); + return (float)(Math.Sqrt(-2.0 * Math.Log(u1)) * Math.Cos(2.0 * Math.PI * u2)); + } + + private static float[] UnitGaussian(Random r, int dim) + { + var v = new float[dim]; + for (int d = 0; d < dim; d++) v[d] = Gauss(r); + Normalize(v); + return v; + } + + private static void Normalize(float[] v) + { + double norm = 0; + foreach (float x in v) norm += (double)x * x; + float inv = (float)(1.0 / Math.Sqrt(norm)); + for (int d = 0; d < v.Length; d++) v[d] *= inv; + } + + public float[][] Sample(int count, int stream) + { + var all = new float[count][]; + const int Chunk = 4096; + Parallel.For(0, (count + Chunk - 1) / Chunk, chunk => + { + var r = new Random(unchecked(stream * 7919 + chunk * 104729 + 13)); + float scale = _noise / MathF.Sqrt(_dim); + for (int i = chunk * Chunk, end = Math.Min(count, i + Chunk); i < end; i++) + { + var center = _centers[r.Next(_centers.Length)]; + var v = new float[_dim]; + for (int d = 0; d < _dim; d++) v[d] = center[d] + scale * Gauss(r); + Normalize(v); + all[i] = v; + } + }); + return all; + } +} + +sealed record Options(int[] Sizes, string[] Regimes, int Dim, int Queries, int[] Nprobes, string? Out) +{ + public static Options Parse(string[] args) + { + var o = new Options(new[] { 10_000, 100_000, 1_000_000 }, new[] { "clustered" }, 384, 200, + new[] { 1, 2, 4, 8, 16, 32, 64, 128 }, null); + for (int i = 0; i < args.Length; i++) + { + string Next() => i + 1 < args.Length ? args[++i] : throw new ArgumentException($"{args[i]} needs a value"); + switch (args[i]) + { + case "--sizes": o = o with { Sizes = Next().Split(',').Select(int.Parse).ToArray() }; break; + case "--regimes": o = o with { Regimes = Next().Split(',') }; break; + case "--dim": o = o with { Dim = int.Parse(Next()) }; break; + case "--queries": o = o with { Queries = int.Parse(Next()) }; break; + case "--nprobes": o = o with { Nprobes = Next().Split(',').Select(int.Parse).ToArray() }; break; + case "--out": o = o with { Out = Next() }; break; + default: throw new ArgumentException($"unknown option {args[i]}"); + } + } + return o; + } +} diff --git a/isongraph-vector-csharp/bench/IsonGraph.Vector.CompetitiveBench/IsonGraph.Vector.CompetitiveBench.csproj b/isongraph-vector-csharp/bench/IsonGraph.Vector.CompetitiveBench/IsonGraph.Vector.CompetitiveBench.csproj new file mode 100644 index 0000000..cfee299 --- /dev/null +++ b/isongraph-vector-csharp/bench/IsonGraph.Vector.CompetitiveBench/IsonGraph.Vector.CompetitiveBench.csproj @@ -0,0 +1,26 @@ +<Project Sdk="Microsoft.NET.Sdk"> + + <!-- + Loader and measurement harness for benchmarks/competitive/ (the cross-library comparison). + Reads the same raw float32 vectors, queries and float64 ground truth as the other harnesses. + Bench-only; not part of the solution or the test run. Build with + dotnet build bench/IsonGraph.Vector.CompetitiveBench -c Release -o OUTDIR and run the dll + with the options listed in Program.cs (ds-dir, ds, n, exp, queries, label, out). + --> + <PropertyGroup> + <OutputType>Exe</OutputType> + <TargetFramework>net8.0</TargetFramework> + <LangVersion>latest</LangVersion> + <Nullable>enable</Nullable> + <ImplicitUsings>enable</ImplicitUsings> + <IsPackable>false</IsPackable> + <ServerGarbageCollector>false</ServerGarbageCollector> + <ConcurrentGarbageCollection>false</ConcurrentGarbageCollection> + <InvariantGlobalization>true</InvariantGlobalization> + </PropertyGroup> + + <ItemGroup> + <ProjectReference Include="..\..\src\IsonGraph.Vector\IsonGraph.Vector.csproj" /> + </ItemGroup> + +</Project> diff --git a/isongraph-vector-csharp/bench/IsonGraph.Vector.CompetitiveBench/Program.cs b/isongraph-vector-csharp/bench/IsonGraph.Vector.CompetitiveBench/Program.cs new file mode 100644 index 0000000..016c9e1 --- /dev/null +++ b/isongraph-vector-csharp/bench/IsonGraph.Vector.CompetitiveBench/Program.cs @@ -0,0 +1,304 @@ +// Loader and measurement harness for benchmarks/competitive/ - see the csproj header. +using System.Diagnostics; +using System.Text.Json; +using NodeRef = IsonGraph.NodeRef; +using IsonGraph.Vector; + +using J = System.Collections.Generic.Dictionary<string, object?>; + +static string Arg(string[] a, string name, string def) +{ + for (int i = 0; i + 1 < a.Length; i++) if (a[i] == name) return a[i + 1]; + return def; +} + +static float[] ReadF32(string path, long start, int count, int d) +{ + using var f = File.OpenRead(path); + f.Seek(start * d * 4, SeekOrigin.Begin); + var bytes = new byte[(long)count * d * 4]; + f.ReadExactly(bytes); + var v = new float[(long)count * d]; + Buffer.BlockCopy(bytes, 0, v, 0, bytes.Length); + return v; +} +static int[] ReadI32(string path) +{ + var b = File.ReadAllBytes(path); + var v = new int[b.Length / 4]; + Buffer.BlockCopy(b, 0, v, 0, b.Length); + return v; +} + +static double Pct(IEnumerable<double> xs, double p) +{ + var s = xs.OrderBy(x => x).ToArray(); + return s.Length == 0 ? 0 : s[(int)Math.Round((s.Length - 1) * p)]; +} +static double Now() => Stopwatch.GetTimestamp() / (double)Stopwatch.Frequency; + +static J Mem() +{ + var p = Process.GetCurrentProcess(); + p.Refresh(); + return new J { ["rss_mb"] = p.WorkingSet64 / 1048576.0, ["peak_mb"] = p.PeakWorkingSet64 / 1048576.0 }; +} + +string dir = Arg(args, "--ds-dir", ""), ds = Arg(args, "--ds", ""), label = Arg(args, "--label", "csharp"); +int n = int.Parse(Arg(args, "--n", "100000")), nq = int.Parse(Arg(args, "--queries", "1000")); +double budget = double.Parse(Arg(args, "--budget", "15")); +var exps = Arg(args, "--exp", "core").Split(',').ToHashSet(); +string outPath = Arg(args, "--out", ""); +var meta = JsonDocument.Parse(File.ReadAllText(System.IO.Path.Combine(dir, "meta.json"))).RootElement; +int d = meta.GetProperty("d").GetInt32(); +var qflat = ReadF32(System.IO.Path.Combine(dir, "queries.f32"), 0, nq, d); +var queries = Enumerable.Range(0, nq).Select(i => qflat.AsSpan(i * d, d).ToArray()).ToArray(); +const int K = 10; +int cpus = Environment.ProcessorCount; +Console.WriteLine($"{label}: ds={ds} n={n} d={d} queries={nq} cpus={cpus}"); + +float[][] Base(int start, int count) +{ + var flat = ReadF32(System.IO.Path.Combine(dir, "base.f32"), start, count, d); + return Enumerable.Range(0, count).Select(i => flat.AsSpan(i * d, d).ToArray()).ToArray(); +} +int[][] Gt(string kind) +{ + var f = ReadI32(System.IO.Path.Combine(dir, $"gt_{kind}{n}.i32")); + return Enumerable.Range(0, f.Length / 100).Select(i => f.AsSpan(i * 100, 100).ToArray()).ToArray(); +} +double Recall(List<int[]> found, int[][] truth) +{ + long hit = 0; + for (int i = 0; i < found.Count; i++) + { + var want = truth[i].Take(K).ToHashSet(); + hit += found[i].Take(K).Count(x => want.Contains(x)); + } + return hit / (double)(K * Math.Max(1, found.Count)); +} +int[] Search(EmbeddingStore s, float[] q, bool exact, string? ty) +{ + var r = s.SimilaritySearchVector(q, K, ty, 0.0f, exact); + var ids = new int[r.Count]; + for (int i = 0; i < ids.Length; i++) ids[i] = int.Parse(r[i].NodeRef.Id.ToString()!); + return ids; +} +J Point(EmbeddingStore s, int[][] truth, bool exact, string? ty) +{ + // Pin to one P-core (logical CPU 8 on the i9-14900HX) for the single-threaded loop: unpinned loops varied up to 3x. + var proc = Process.GetCurrentProcess(); + var oldMask = proc.ProcessorAffinity; + int pinCpu = int.Parse(Environment.GetEnvironmentVariable("BENCH_PIN_CPU") ?? "8"); + try { proc.PriorityClass = ProcessPriorityClass.High; proc.ProcessorAffinity = (IntPtr)(1L << pinCpu); } catch { } + try { return PointPinned(s, truth, exact, ty); } + finally { try { proc.ProcessorAffinity = oldMask; } catch { } } +} +J PointPinned(EmbeddingStore s, int[][] truth, bool exact, string? ty) +{ + double w0 = Now(); + for (int i = 0; i < Math.Min(50, nq); i++) { Search(s, queries[i], exact, ty); if (Now() - w0 > 3.0) break; } + var probe = new List<double>(); + for (int i = 0; i < Math.Min(5, nq); i++) { double t = Now(); Search(s, queries[i], exact, ty); probe.Add(Now() - t); } + double est = Math.Max(Pct(probe, 0.5), 1e-7); + int m = (int)Math.Min(nq, Math.Max(30, budget / est)); + var runs = new List<double>(); + var lats = new List<double>(); + double rec = 0; + for (int r = 0; r < 3; r++) + { + var lat = new List<double>(m); + var res = new List<int[]>(m); + for (int i = 0; i < m; i++) + { + double t = Now(); + var ids = Search(s, queries[i], exact, ty); + lat.Add(Now() - t); + if (r == 0) res.Add(ids); + } + if (r == 0) rec = Recall(res, truth.Take(m).ToArray()); + double total = lat.Sum(); + runs.Add(m / total); + lats.AddRange(lat); + if (total > budget * 1.5) break; + } + return new J + { + ["nq"] = m, ["recall"] = rec, ["qps_1t"] = Pct(runs, 0.5), ["qps_1t_min"] = runs.Min(), ["qps_1t_max"] = runs.Max(), ["qps_1t_runs"] = runs, + ["median_ms"] = Pct(lats, 0.5) * 1000, ["p95_ms"] = Pct(lats, 0.95) * 1000, ["p99_ms"] = Pct(lats, 0.99) * 1000, ["mean_ms"] = lats.Average() * 1000, + }; +} +int NlistFor(int x) => (int)Math.Clamp(Math.Round(4 * Math.Sqrt(x)), 16, 65536); +List<int> Knobs() +{ + int nl = NlistFor(n); + var v = new[] { 1, 2, 4, 8, 16, 32, 64, 128, 256 }.Where(x => x < nl).ToList(); + if (n <= 200_000) v.Add(nl); + return v; +} +double Ingest(EmbeddingStore s, byte[]? grp) +{ + double secs = 0; + string[] names = { "t10", "t01", "rest" }; + for (int at = 0; at < n; at += 20000) + { + int m = Math.Min(20000, n - at); + var rows = Base(at, m); + double t = Now(); + for (int i = 0; i < m; i++) s.Add(new NodeRef(grp == null ? "doc" : names[grp[at + i]], (at + i).ToString()), "", rows[i]); + secs += Now() - t; + } + return secs; +} +J Header(string sys) => new J +{ + ["system"] = sys, ["ds"] = ds, ["n"] = n, ["d"] = d, ["k"] = K, ["queries"] = nq, ["all_threads"] = cpus, + ["version"] = "isongraph-vector-csharp (repo HEAD, .NET 8 Release)", +}; + +var runsOut = new List<J>(); +var memBase = Mem(); +if (exps.Contains("core") || exps.Contains("mutation") || exps.Contains("saveload")) +{ + var store = new EmbeddingStore(ann: AnnMode.None); + double ingestS = Ingest(store, null); + var memIng = Mem(); + Console.WriteLine($" ingest {ingestS:F2}s"); + var truth = Gt(""); + double t0 = Now(); + store.SimilaritySearchVector(queries[0], K, null, 0f, true); + double firstMs = (Now() - t0) * 1000; + var ex = Header(label + "_exact"); + ex["build"] = new J { ["ingest_s"] = ingestS, ["index_s"] = 0.0, ["total_s"] = ingestS, ["first_query_ms"] = firstMs, ["ready_s"] = ingestS + firstMs / 1000 }; + ex["mem_baseline"] = memBase; ex["mem_after_ingest"] = memIng; ex["mem_after_build"] = Mem(); ex["mem_after_first_query"] = Mem(); + var p1 = Point(store, truth, true, null); p1["knob"] = null; + Console.WriteLine($" exact recall={p1["recall"]} qps={p1["qps_1t"]:F1}"); + ex["sweep"] = new List<J> { p1 }; + ex["mem_after_sweep"] = Mem(); + runsOut.Add(ex); + + // IVF: a fresh store with the index on (the port enables it at construction) + store = null!; + GC.Collect(); + var st = new EmbeddingStore(ann: AnnMode.Ivf, annMinSize: 0, annSeed: 0); + ingestS = Ingest(st, null); + memIng = Mem(); + t0 = Now(); + var stats = st.BuildAnnIndex(); + double idxS = Now() - t0; + Console.WriteLine($" ivf build {idxS:F2}s nlist={stats.Nlist}"); + var iv = Header(label + "_ivf"); + iv["knob_name"] = "nprobe"; + iv["build"] = new J { ["ingest_s"] = ingestS, ["index_s"] = idxS, ["total_s"] = ingestS + idxS, ["first_query_ms"] = 0.0, ["ready_s"] = ingestS + idxS }; + iv["mem_baseline"] = memBase; iv["mem_after_ingest"] = memIng; iv["mem_after_build"] = Mem(); + iv["index_stats"] = new J { ["nlist"] = stats.Nlist, ["trained_n"] = stats.TrainedN, ["index_bytes"] = stats.IndexBytes, ["vector_bytes"] = stats.VectorBytes, ["build_seconds"] = stats.BuildSeconds }; + var pts = new List<J>(); + int refKnob = -1; double refRecall = 0; + foreach (int kn in Knobs()) + { + st.AnnNprobe = kn; + var p = Point(st, truth, false, null); p["knob"] = kn; + Console.WriteLine($" ivf nprobe={kn} recall={p["recall"]:F4} qps1t={p["qps_1t"]:F0}"); + if (refKnob < 0 && (double)p["recall"]! >= 0.95) { refKnob = kn; refRecall = (double)p["recall"]!; } + pts.Add(p); + } + if (refKnob < 0) refKnob = Knobs().Last(); + iv["sweep"] = pts; iv["ref_knob"] = refKnob; iv["ref_recall_pre"] = refRecall; iv["mem_after_sweep"] = Mem(); + + if (exps.Contains("mutation")) + { + string gtm = System.IO.Path.Combine(dir, $"gt_mut_{n}.i32"); + if (File.Exists(gtm)) + { + var upd = ReadI32(System.IO.Path.Combine(dir, $"mut_{n}_upd.i32")); + var del = ReadI32(System.IO.Path.Combine(dir, $"mut_{n}_del.i32")); + var f = ReadI32(gtm); + var truthM = Enumerable.Range(0, f.Length / 100).Select(i => f.AsSpan(i * 100, 100).ToArray()).Take(nq).ToArray(); + st.AnnNprobe = refKnob; + var m = new J { ["ref_knob"] = refKnob, ["recall_before"] = refRecall }; + var ins = Base(n, 10000); + double t = Now(); + for (int i = 0; i < ins.Length; i++) st.Add(new NodeRef("doc", (n + i).ToString()), "", ins[i]); + double dt = Now() - t; m["insert_10k"] = new J { ["seconds"] = dt, ["per_s"] = 10000 / dt }; + var nv = Base(n + 10000, 10000); + t = Now(); + for (int j = 0; j < nv.Length; j++) st.Add(new NodeRef("doc", upd[j].ToString()), "", nv[j]); + dt = Now() - t; m["update_10k"] = new J { ["seconds"] = dt, ["per_s"] = 10000 / dt }; + t = Now(); + foreach (int id in del) st.Remove(new NodeRef("doc", id.ToString())); + dt = Now() - t; m["delete_10k"] = new J { ["seconds"] = dt, ["per_s"] = 10000 / dt }; + t = Now(); + Search(st, queries[0], false, null); + m["first_query_after_ms"] = (Now() - t) * 1000; + var p = Point(st, truthM, false, null); + m["recall_after"] = p["recall"]; m["after"] = p; m["mem_after"] = Mem(); + var s2 = st.GetAnnStats(); + m["index_stats_after"] = new J { ["pending"] = s2.Pending, ["churn"] = s2.Churn, ["trained_n"] = s2.TrainedN }; + Console.WriteLine($" mutation recall_after={p["recall"]}"); + iv["mutation"] = m; + } + else iv["mutation"] = new J { ["skipped"] = "dataset has no spare rows for this n" }; + } + if (exps.Contains("saveload")) + { + string path = System.IO.Path.GetFullPath(System.IO.Path.Combine(dir, "..", "..", "work", $"cs_{ds}_{n}.json")); + Directory.CreateDirectory(System.IO.Path.GetDirectoryName(path)!); + var sl = new J(); + try + { + double t = Now(); + st.SaveEmbeddings(path); + sl["save_s"] = Now() - t; + sl["disk_mb"] = new FileInfo(path).Length / 1048576.0; + var s2 = new EmbeddingStore(ann: AnnMode.Ivf, annMinSize: 0, annNprobe: refKnob); + t = Now(); + s2.LoadEmbeddings(path); + double loadS = Now() - t; + sl["load_s"] = loadS; + t = Now(); + Search(s2, queries[0], false, null); + double fq = (Now() - t) * 1000; + sl["first_query_after_load_ms"] = fq; sl["time_to_ready_s"] = loadS + fq / 1000; + File.Delete(path); + } + catch (Exception e) { sl["error"] = e.Message; } + iv["saveload"] = sl; + } + runsOut.Add(iv); +} + +if (exps.Contains("filter")) +{ + var g = File.ReadAllBytes(System.IO.Path.Combine(dir, $"grp_{n}.u8")); + var st = new EmbeddingStore(ann: AnnMode.Ivf, annMinSize: 0, annSeed: 0); + double ingestS = Ingest(st, g); + double t0 = Now(); + st.BuildAnnIndex(); + double idxS = Now() - t0; + var res = Header(label + "_ivf"); + var subsets = new J(); + res["filter"] = new J { ["build_with_metadata_s"] = ingestS + idxS, ["mem"] = Mem(), ["subsets"] = subsets }; + var kn = Knobs(); + var pick = new SortedSet<int> { 1, kn.Count / 3, kn.Count / 2, 2 * kn.Count / 3, kn.Count - 1 }; + foreach (var (ty, lab) in new[] { ("t10", "f10"), ("t01", "f01") }) + { + var truth = Gt(lab + "_"); + var pts = new List<J>(); + foreach (int i in pick) + { + st.AnnNprobe = kn[i]; + var p = Point(st, truth, false, ty); p["knob"] = kn[i]; + Console.WriteLine($" filter {lab} nprobe={kn[i]} recall={p["recall"]:F4} qps={p["qps_1t"]:F0}"); + pts.Add(p); + } + var pe = Point(st, truth, true, ty); pe["knob"] = "exact"; pts.Add(pe); + subsets[lab] = pts; + } + runsOut.Add(res); +} + +if (outPath.Length > 0) +{ + File.WriteAllText(outPath, JsonSerializer.Serialize(new J { ["status"] = "ok", ["label"] = label, ["runs"] = runsOut }, new JsonSerializerOptions { WriteIndented = true })); + Console.WriteLine($"wrote {outPath}"); +} diff --git a/isongraph-vector-csharp/src/IsonGraph.Vector/AnnTypes.cs b/isongraph-vector-csharp/src/IsonGraph.Vector/AnnTypes.cs new file mode 100644 index 0000000..62cec9f --- /dev/null +++ b/isongraph-vector-csharp/src/IsonGraph.Vector/AnnTypes.cs @@ -0,0 +1,65 @@ +namespace IsonGraph.Vector; + +/// <summary>Approximate nearest-neighbour index a store may keep (see <see cref="EmbeddingStore"/>).</summary> +public enum AnnMode +{ + /// <summary>No index: every search is the exact scan (default).</summary> + None, + /// <summary> + /// In-memory inverted-file ("IVF-flat") index: vectors are clustered by + /// direction and only the cells nearest the query are scored. + /// </summary> + Ivf, +} + +/// <summary>How many searches each route answered, since the store was created.</summary> +/// <param name="Approximate">Answered from the index (including widened probes).</param> +/// <param name="Exact">Answered by the full scan.</param> +/// <param name="Widened">Answered from the index only after widening the probe.</param> +/// <param name="Builds">Times the index was trained (first build and retrains).</param> +public sealed record AnnSearchCounts(long Approximate, long Exact, long Widened, long Builds); + +/// <summary> +/// What the approximate index looks like right now; see +/// <see cref="EmbeddingStore.GetAnnStats"/>. +/// </summary> +/// <param name="Enabled">The store was created with an index mode.</param> +/// <param name="Mode">The mode.</param> +/// <param name="Active">An index exists and will answer searches.</param> +/// <param name="Nlist">Cells in the index (null while none).</param> +/// <param name="Nprobe">Cells scanned per query at present (the configured value, or the automatic one).</param> +/// <param name="TrainedN">Vectors the centroids were trained on (null while none).</param> +/// <param name="Pending">Rows scored exactly until folded into a cell.</param> +/// <param name="Churn">Writes plus deletes since training.</param> +/// <param name="BuildSeconds">Time the last training took (null while none).</param> +/// <param name="IndexBytes">Memory held by centroids and cell lists, on top of <paramref name="VectorBytes"/>.</param> +/// <param name="VectorBytes">Memory held by the index's own contiguous copy of the vectors.</param> +/// <param name="MinSize">Below this many vectors search stays exact.</param> +/// <param name="Searches">Per-route search counts.</param> +public sealed record AnnStats( + bool Enabled, AnnMode Mode, bool Active, int? Nlist, int? Nprobe, int? TrainedN, + int Pending, long Churn, double? BuildSeconds, long IndexBytes, long VectorBytes, + int MinSize, AnnSearchCounts Searches); + +/// <summary> +/// Recall of the approximate index against the exact scan, over some queries. +/// </summary> +/// <param name="Recall"> +/// Mean over queries of |approximate top-k ∩ exact top-k| / |exact top-k| +/// (1.0 = identical sets). +/// </param> +/// <param name="MinRecall">The worst single query.</param> +/// <param name="Queries">Queries run.</param> +/// <param name="K">Results compared per query.</param> +/// <param name="Nlist">Cells in the index (0 when none).</param> +/// <param name="Nprobe">Cells scanned per query.</param> +/// <param name="Approximate"> +/// False when the index was not actually consulted (it is off, the store is +/// smaller than the minimum, or nprobe >= nlist); recall is then 1.0 by +/// construction. +/// </param> +/// <param name="AnnMs">Median per-query latency of the indexed search, milliseconds.</param> +/// <param name="ExactMs">Median per-query latency of the exact scan, milliseconds.</param> +public sealed record RecallReport( + double Recall, double MinRecall, int Queries, int K, int Nlist, int Nprobe, + bool Approximate, double AnnMs, double ExactMs); diff --git a/isongraph-vector-csharp/src/IsonGraph.Vector/EmbeddingStore.Ann.cs b/isongraph-vector-csharp/src/IsonGraph.Vector/EmbeddingStore.Ann.cs new file mode 100644 index 0000000..2a4a2d0 --- /dev/null +++ b/isongraph-vector-csharp/src/IsonGraph.Vector/EmbeddingStore.Ann.cs @@ -0,0 +1,250 @@ +namespace IsonGraph.Vector; + +// Opt-in approximate (IVF) search for the in-memory store. The exact scan in +// IsonGraphVector.cs is untouched; everything here runs under the same +// coarse lock as every other store operation. +public sealed partial class EmbeddingStore +{ + private const int AnnFoldMin = 512; // fold pending rows into cells at >= this many ... + private const double AnnFoldFraction = 0.02; // ... and at least this fraction of the indexed rows + private const double AnnRetrainFraction = 0.5; // retrain once writes+deletes exceed this x trained size + + private static int AnnAutoNlist(int n) => + (int)Math.Min(Math.Max(Math.Round(4 * Math.Sqrt(n)), 16), 65536); + + /// <summary> + /// The index, built or maintained as the policy says; null means "answer + /// exactly". Caller holds the lock. + /// + /// Policy: built lazily by the first qualifying search (never persisted), + /// k-means on a sample of up to 40 x nlist vectors, then every row + /// assigned. New and updated rows are scored exactly until at least + /// max(512, 2%) of the indexed rows are pending, then folded into their + /// nearest cell. Deletes just drop the row. Once writes + deletes since + /// training exceed 50% of the trained size the index is retrained from the + /// current data (with an automatic nlist, re-chosen for the new size). A + /// store that shrinks below the minimum size, or is cleared, drops it. + /// </summary> + private IvfIndex? AnnIndexForSearch() + { + if (_ann == AnnMode.None) return null; + int live = _embeddings.Count; + if (live < Math.Max(2, _annMinSize)) + { + _ivf = null; + return null; + } + if (_ivf is null) + { + _ivf = IvfIndex.FromRecords(_embeddings.Values, _dimension!.Value, live); + TrainIndex(_ivf); + } + else if (_ivf.Churn > AnnRetrainFraction * _ivf.TrainedN) + { + TrainIndex(_ivf); + } + else if (_ivf.PendingCount >= Math.Max(AnnFoldMin, AnnFoldFraction * _ivf.TrainedN)) + { + _ivf.Fold(); + } + return _ivf; + } + + private void TrainIndex(IvfIndex ivf) + { + int nlist = _annNlist ?? AnnAutoNlist(ivf.LiveCount); + ivf.Train(nlist, _annSeed); + _annBuilds++; + } + + private int AnnProbe(IvfIndex ivf) + { + int wanted = _annNprobe ?? Math.Max(1, (int)Math.Round(0.05 * ivf.Nlist)); + return Math.Min(wanted, ivf.Nlist); + } + + /// <summary> + /// Cells to probe at widening <paramref name="scale"/>; null = widened + /// enough to just scan. Scale 1 honours nprobe as configured (nprobe >= + /// nlist is the exact scan). Later steps multiply it by 4 and give up on + /// the index once they would touch half the cells, where scanning + /// everything costs about the same. + /// </summary> + private int? AnnLists(IvfIndex ivf, long scale) + { + long lists = AnnProbe(ivf) * scale; + if (scale == 1) return lists >= ivf.Nlist ? null : (int)lists; + return lists * 2 >= ivf.Nlist ? null : (int)lists; + } + + /// <summary> + /// Top-k from the index, or null to fall back to the exact scan. Caller + /// holds the lock. Filters apply to the probed rows; if fewer than topK + /// survive, the probe is widened 4x and tried again, ending in the exact + /// scan, so a rare type or a high threshold costs time rather than + /// silently returning a short list. + /// </summary> + private IReadOnlyList<SimilarityResult>? AnnSearch( + float[] queryVector, int topK, string? nodeType, float threshold) + { + var ivf = AnnIndexForSearch(); + if (ivf is null) return null; + for (long scale = 1; ; scale *= 4) + { + var lists = AnnLists(ivf, scale); + if (lists is null) return null; + if (!ivf.Scan(queryVector, nodeType, threshold, lists.Value, out int passing)) + return null; // a zero query has no direction to rank cells by + if (passing >= topK) + { + _annApproximate++; + if (scale > 1) _annWidened++; + return ivf.TopK(topK); + } + } + } + + /// <summary> + /// Every match in the probed cells, best first, for callers that apply a + /// predicate before taking k (the graph's nodeFilter). <c>Approximate</c> + /// is false when the answer is exact (the full scan) and there is nothing + /// left to widen to. <paramref name="scale"/> is the widening step + /// (1, 4, 16, ...). The sequence is a snapshot: it may be enumerated after + /// the lock is released, with the store changing underneath. + /// </summary> + internal (IEnumerable<SimilarityResult> Ranked, bool Approximate) RankedForGate( + float[] queryVector, string? nodeType, float threshold, long scale) + { + lock (_lock) + { + CheckQuery(queryVector); + var ivf = AnnIndexForSearch(); + var lists = ivf is null ? null : AnnLists(ivf, scale); + if (ivf is not null && lists is not null && + ivf.Scan(queryVector, nodeType, threshold, lists.Value, out _)) + { + _annApproximate++; + if (scale > 1) _annWidened++; + return (ivf.RankedSnapshot(), true); + } + return (SimilaritySearchVector(queryVector, null, nodeType, threshold, exact: true), false); + } + } + + /// <summary>Resolve a text query to its vector (encoder, model check and dimension check).</summary> + internal float[] EncodeQuery(string query) => ResolveQuery(query); + + /// <summary> + /// Build the approximate index now instead of on the first search. Returns + /// <see cref="GetAnnStats"/>. <paramref name="force"/> retrains even if an + /// index exists. Does nothing (and says so in the stats) while the store is + /// smaller than the minimum size. + /// </summary> + /// <exception cref="EmbeddingException">The store was created without an index mode.</exception> + public AnnStats BuildAnnIndex(bool force = false) + { + if (_ann == AnnMode.None) + throw new EmbeddingException( + "this store has no approximate index; create it with ann: AnnMode.Ivf"); + lock (_lock) + { + if (force) _ivf = null; + AnnIndexForSearch()?.Fold(); + return GetAnnStats(); + } + } + + /// <summary>What the approximate index looks like right now. Never builds anything.</summary> + public AnnStats GetAnnStats() + { + lock (_lock) + { + var ivf = _ivf; + return new AnnStats( + Enabled: _ann != AnnMode.None, + Mode: _ann, + Active: ivf is not null, + Nlist: ivf?.Nlist, + Nprobe: ivf is not null ? AnnProbe(ivf) : _annNprobe, + TrainedN: ivf?.TrainedN, + Pending: ivf?.PendingCount ?? 0, + Churn: ivf?.Churn ?? 0, + BuildSeconds: ivf?.BuildSeconds, + IndexBytes: ivf?.IndexBytes ?? 0, + VectorBytes: ivf?.VectorBytes ?? 0, + MinSize: _annMinSize, + Searches: new AnnSearchCounts(_annApproximate, _annExact, _annWidened, _annBuilds)); + } + } + + /// <summary>Recall@k for text queries; see the vector overload.</summary> + public RecallReport RecallCheck( + IEnumerable<string> queries, int k = 10, string? nodeType = null, float threshold = 0.0f) => + RecallCheck(queries.Select(ResolveQuery).ToList(), k, nodeType, threshold); + + /// <summary>Recall@k for a single text query.</summary> + public RecallReport RecallCheck(string query, int k = 10, string? nodeType = null, float threshold = 0.0f) => + RecallCheck(new[] { ResolveQuery(query) }, k, nodeType, threshold); + + /// <summary>Recall@k for a single query vector.</summary> + public RecallReport RecallCheck(float[] query, int k = 10, string? nodeType = null, float threshold = 0.0f) => + RecallCheck(new[] { query }, k, nodeType, threshold); + + /// <summary> + /// Recall@k of the approximate index against the exact scan. + /// + /// Runs every query both ways and compares the sets of top-k nodes. Use + /// real queries drawn from your own workload: recall depends on how the data + /// clusters, and a score measured on unrelated queries or data does not + /// transfer. Raise <see cref="AnnNprobe"/> until the recall you need holds. + /// Latencies are per-query medians of the search itself (a text query's + /// encoding is not timed). + /// + /// Without an index in use (none requested, a store under the minimum size, + /// or nprobe >= nlist) both answers are the exact scan, recall is 1.0 and + /// <see cref="RecallReport.Approximate"/> is false. + /// </summary> + public RecallReport RecallCheck( + IEnumerable<float[]> queries, int k = 10, string? nodeType = null, float threshold = 0.0f) + { + if (k <= 0) throw new ArgumentOutOfRangeException(nameof(k), k, "k must be positive"); + var batch = queries.ToList(); + var recalls = new List<double>(); + var annTimes = new List<double>(); + var exactTimes = new List<double>(); + bool used = false; + foreach (var query in batch) + { + long before = GetAnnStats().Searches.Approximate; + long t0 = System.Diagnostics.Stopwatch.GetTimestamp(); + var found = SimilaritySearchVector(query, k, nodeType, threshold); + annTimes.Add(System.Diagnostics.Stopwatch.GetElapsedTime(t0).TotalMilliseconds); + used |= GetAnnStats().Searches.Approximate > before; + + t0 = System.Diagnostics.Stopwatch.GetTimestamp(); + var truth = SimilaritySearchVector(query, k, nodeType, threshold, exact: true); + exactTimes.Add(System.Diagnostics.Stopwatch.GetElapsedTime(t0).TotalMilliseconds); + if (truth.Count > 0) + { + var wanted = new HashSet<NodeRef>(truth.Select(r => r.NodeRef)); + int hit = found.Count(r => wanted.Contains(r.NodeRef)); + recalls.Add((double)hit / wanted.Count); + } + } + var stats = GetAnnStats(); + return new RecallReport( + Recall: recalls.Count > 0 ? recalls.Average() : 1.0, + MinRecall: recalls.Count > 0 ? recalls.Min() : 1.0, + Queries: batch.Count, K: k, + Nlist: stats.Nlist ?? 0, Nprobe: stats.Nprobe ?? 0, + Approximate: used, AnnMs: Median(annTimes), ExactMs: Median(exactTimes)); + } + + private static double Median(List<double> values) + { + if (values.Count == 0) return 0.0; + var ordered = values.OrderBy(v => v).ToList(); + int mid = ordered.Count / 2; + return ordered.Count % 2 == 1 ? ordered[mid] : 0.5 * (ordered[mid - 1] + ordered[mid]); + } +} diff --git a/isongraph-vector-csharp/src/IsonGraph.Vector/IsonGraphVector.cs b/isongraph-vector-csharp/src/IsonGraph.Vector/IsonGraphVector.cs index c68dc7d..8ff525d 100644 --- a/isongraph-vector-csharp/src/IsonGraph.Vector/IsonGraphVector.cs +++ b/isongraph-vector-csharp/src/IsonGraph.Vector/IsonGraphVector.cs @@ -381,7 +381,7 @@ public sealed class ExportedEmbeddings /// <summary> /// In-memory embedding storage with brute-force cosine similarity search. /// </summary> -public sealed class EmbeddingStore +public sealed partial class EmbeddingStore { private readonly Dictionary<string, EmbeddingRecord> _embeddings = new(); private readonly object _lock = new(); @@ -393,6 +393,19 @@ public sealed class EmbeddingStore private readonly HashSet<string> _warnedMismatch = new(); private IReadOnlyDictionary<string, string> _lastImportMeta = new Dictionary<string, string>(); + // Opt-in approximate index (see AnnMode). _ivf is built lazily by the first + // search that qualifies, kept current by Put/Remove/Clear, and never saved. + private readonly AnnMode _ann; + private readonly int? _annNlist; + private int? _annNprobe; + private readonly int _annMinSize; + private readonly int _annSeed; + private IvfIndex? _ivf; + private long _annApproximate, _annExact, _annWidened, _annBuilds; + + /// <summary>Below this many vectors the index is not used by default.</summary> + public const int DefaultAnnMinSize = 50_000; + /// <param name="encoder">Encoder used when a text is added or queried without a vector.</param> /// <param name="version"> /// Free-form version of the encoder/model that made these vectors, @@ -404,15 +417,68 @@ public sealed class EmbeddingStore /// "unknown"/"provided" never trigger it, and vector queries are never /// checked. /// </param> + /// <param name="ann"> + /// <see cref="AnnMode.None"/> (default) keeps search exact, exactly as it + /// always was. <see cref="AnnMode.Ivf"/> answers bounded top-k searches + /// from an in-memory IVF-flat index: vectors are clustered into + /// <paramref name="annNlist"/> cells and only the <paramref name="annNprobe"/> + /// cells nearest the query are scored. It is approximate - a true neighbour + /// in an unprobed cell is missed - so measure it with + /// <see cref="RecallCheck(IEnumerable{float[]}, int, string?, float)"/>. The + /// index is built lazily by the first search, never persisted, and kept + /// current through writes and deletes. It holds its own contiguous copy of + /// the vectors (about the size of the vectors again). + /// </param> + /// <param name="annNlist">Cells in the index. Null picks about 4 * sqrt(N) when it is built.</param> + /// <param name="annNprobe"> + /// Cells scanned per query. Null picks about 5% of nlist. More cells = higher + /// recall, slower; nprobe >= nlist is the exact scan. Adjustable later via + /// <see cref="AnnNprobe"/>. + /// </param> + /// <param name="annMinSize">Below this many vectors search stays exact. Default 50,000.</param> + /// <param name="annSeed">Seed for the k-means initialisation, so the same data gives the same index.</param> public EmbeddingStore( IEmbeddingEncoder? encoder = null, string? version = null, - ModelMismatchMode onModelMismatch = ModelMismatchMode.Warn) + ModelMismatchMode onModelMismatch = ModelMismatchMode.Warn, + AnnMode ann = AnnMode.None, + int? annNlist = null, + int? annNprobe = null, + int annMinSize = DefaultAnnMinSize, + int annSeed = 0) { CheckMode(onModelMismatch); + if (!Enum.IsDefined(ann)) + throw new ArgumentOutOfRangeException(nameof(ann), ann, "unknown AnnMode"); + if (annNlist is < 1) + throw new ArgumentOutOfRangeException(nameof(annNlist), annNlist, "annNlist must be >= 1"); + if (annNprobe is < 1) + throw new ArgumentOutOfRangeException(nameof(annNprobe), annNprobe, "annNprobe must be >= 1"); + if (annMinSize < 0) + throw new ArgumentOutOfRangeException(nameof(annMinSize), annMinSize, "annMinSize must be >= 0"); _encoder = encoder; _version = version; _onModelMismatch = onModelMismatch; + _ann = ann; + _annNlist = annNlist; + _annNprobe = annNprobe; + _annMinSize = annMinSize; + _annSeed = annSeed; + } + + /// <summary>The approximate index in use, or <see cref="AnnMode.None"/> for exact search.</summary> + public AnnMode Ann => _ann; + + /// <summary>Cells scanned per query; null means automatic (about 5% of nlist).</summary> + public int? AnnNprobe + { + get { lock (_lock) return _annNprobe; } + set + { + if (value is < 1) + throw new ArgumentOutOfRangeException(nameof(value), value, "AnnNprobe must be >= 1"); + lock (_lock) _annNprobe = value; + } } private static void CheckMode(ModelMismatchMode mode) @@ -662,8 +728,10 @@ public string Add(NodeRef nodeRef, string text, float[]? vector = null) private string Put(NodeRef nodeRef, string text, float[] vector, string model) { var key = Key(nodeRef); - _embeddings[key] = new EmbeddingRecord(key, nodeRef, vector, text, model); + var record = new EmbeddingRecord(key, nodeRef, vector, text, model); + _embeddings[key] = record; _modelsCache = null; + _ivf?.Put(record); return key; } @@ -705,7 +773,10 @@ public bool Remove(NodeRef nodeRef) lock (_lock) { _modelsCache = null; - return _embeddings.Remove(Key(nodeRef)); + var key = Key(nodeRef); + bool removed = _embeddings.Remove(key); + if (removed) _ivf?.Drop(key); + return removed; } } @@ -730,21 +801,49 @@ public void Clear(bool resetDimension = true) { _embeddings.Clear(); _modelsCache = null; + _ivf = null; if (resetDimension) _dimension = null; } } + /// <summary> + /// Find the most similar embeddings to a text query. + /// + /// With an approximate index (<see cref="Ann"/>) a bounded <paramref name="topK"/> + /// on a store of at least the minimum size is answered from the index + /// (scores are the same cosine values, but a true neighbour in an unprobed + /// cell can be missing). <paramref name="exact"/> forces the full scan. + /// </summary> + /// <param name="query">Query text.</param> + /// <param name="topK">Number of results (null returns every match).</param> + /// <param name="nodeType">Keep only embeddings of this node type.</param> + /// <param name="threshold">Minimum similarity.</param> + /// <param name="exact">Skip the approximate index and scan everything (no effect without one).</param> public IReadOnlyList<SimilarityResult> SimilaritySearch( - string query, int? topK = 10, string? nodeType = null, float threshold = 0.0f) - => SimilaritySearchVector(ResolveQuery(query), topK, nodeType, threshold); + string query, int? topK = 10, string? nodeType = null, float threshold = 0.0f, bool exact = false) + => SimilaritySearchVector(ResolveQuery(query), topK, nodeType, threshold, exact); + /// <summary>Find the most similar embeddings to a query vector; see <see cref="SimilaritySearch"/>.</summary> public IReadOnlyList<SimilarityResult> SimilaritySearchVector( - float[] queryVector, int? topK = 10, string? nodeType = null, float threshold = 0.0f) + float[] queryVector, int? topK = 10, string? nodeType = null, float threshold = 0.0f, + bool exact = false) { lock (_lock) { CheckQuery(queryVector); + if (_ann != AnnMode.None) + { + if (exact || topK is null || topK <= 0) + _annExact++; + else + { + var answer = AnnSearch(queryVector, topK.Value, nodeType, threshold); + if (answer is not null) return answer; + _annExact++; + } + } + var results = new List<SimilarityResult>(); foreach (var record in _embeddings.Values) { @@ -977,6 +1076,11 @@ public sealed class SemanticGraph /// <param name="directed">Whether edges are directed.</param> /// <param name="version">Encoder/model version recorded in the sidecar (default: the encoder's own Version).</param> /// <param name="onModelMismatch">What a text search does when the encoder's model differs from the stored vectors' model.</param> + /// <param name="ann">Opt-in approximate index for the graph's embedding store (see <see cref="EmbeddingStore"/>); default is exact search.</param> + /// <param name="annNlist">Cells in the index; null picks about 4 * sqrt(N).</param> + /// <param name="annNprobe">Cells scanned per query; null picks about 5% of nlist.</param> + /// <param name="annMinSize">Below this many vectors search stays exact.</param> + /// <param name="annSeed">Seed for the k-means initialisation.</param> public SemanticGraph( string name = "semantic_graph", IEmbeddingEncoder? encoder = null, @@ -984,11 +1088,16 @@ public SemanticGraph( IEnumerable<string>? embedFields = null, bool directed = true, string? version = null, - ModelMismatchMode onModelMismatch = ModelMismatchMode.Warn) + ModelMismatchMode onModelMismatch = ModelMismatchMode.Warn, + AnnMode ann = AnnMode.None, + int? annNlist = null, + int? annNprobe = null, + int annMinSize = EmbeddingStore.DefaultAnnMinSize, + int annSeed = 0) { _graph = new ISONGraph(name, directed); _view = new ReadOnlyGraphView(_graph); - _store = new EmbeddingStore(encoder, version, onModelMismatch); + _store = new EmbeddingStore(encoder, version, onModelMismatch, ann, annNlist, annNprobe, annMinSize, annSeed); _autoEmbed = autoEmbed; _embedFields = (embedFields ?? DefaultEmbedFields).ToList(); } @@ -1184,15 +1293,49 @@ public Node UpdateNode( /// With a gate the store cannot truncate first: the best k unfiltered /// matches may all be filtered out, so every match is scored, filtered, and /// only then cut to k. + /// + /// With the store's approximate index (<paramref name="ann"/> given) the + /// candidates are the probed cells only. If fewer than k of them pass the + /// gate, the probe is widened (4x each step) and the search retried, + /// ending in the exact scan, so a selective filter costs time rather than + /// results. /// </summary> private IReadOnlyList<SimilarityResult> SeedSearch( - Func<int?, IReadOnlyList<SimilarityResult>> search, int? topK, Func<NodeRef, bool>? gate) + Func<int?, IReadOnlyList<SimilarityResult>> search, int? topK, Func<NodeRef, bool>? gate, + Func<long, (IEnumerable<SimilarityResult> Ranked, bool Approximate)>? ann = null) { if (gate is null) return search(topK); - var passing = search(null).Where(r => gate(r.NodeRef)).ToList(); - return topK is null ? passing : passing.Take(Math.Max(topK.Value, 0)).ToList(); + if (ann is null || topK is not > 0) + { + var passing = search(null).Where(r => gate(r.NodeRef)).ToList(); + return topK is null ? passing : passing.Take(Math.Max(topK.Value, 0)).ToList(); + } + + int limit = topK.Value; + for (long scale = 1; ; scale *= 4) + { + var (ranked, approximate) = ann(scale); + // Best first, and stop as soon as k have passed: the gate (a user + // callback that may touch the graph) runs on the matches that are + // needed, not on every stored vector. + var results = new List<SimilarityResult>(); + foreach (var result in ranked) + { + if (!gate(result.NodeRef)) continue; + results.Add(result); + if (results.Count >= limit) break; + } + if (!approximate || results.Count >= limit) return results; + } } + /// <summary>The widening search for <see cref="SeedSearch"/>, or null when the store has no index in play.</summary> + private Func<long, (IEnumerable<SimilarityResult>, bool)>? AnnRanked( + float[] queryVector, string? nodeType, float threshold, bool exact = false) => + _store.Ann == AnnMode.None || exact + ? null + : scale => _store.RankedForGate(queryVector, nodeType, threshold, scale); + /// <summary> /// Search by embedding similarity. /// </summary> @@ -1209,14 +1352,36 @@ private IReadOnlyList<SimilarityResult> SeedSearch( /// Keep only nodes whose type is in this collection (combines with /// nodeType and nodeFilter by AND; empty matches nothing). /// </param> + /// <param name="exact"> + /// Force the full scan even when the graph was created with an approximate + /// index (<c>ann</c>); no effect otherwise. + /// </param> public IReadOnlyList<SimilarityResult> SimilaritySearch( string query, int? topK = 10, string? nodeType = null, float threshold = 0.0f, - Func<Node, bool>? nodeFilter = null, IEnumerable<string>? nodeTypes = null) + Func<Node, bool>? nodeFilter = null, IEnumerable<string>? nodeTypes = null, bool exact = false) { var gate = NodeGate(nodeFilter, nodeTypes); - return SeedSearch(k => _store.SimilaritySearch(query, k, nodeType, threshold), topK, gate); + if (gate is null) + return _store.SimilaritySearch(query, topK, nodeType, threshold, exact); + var queryVector = _store.EncodeQuery(query); + return SeedSearch( + k => _store.SimilaritySearchVector(queryVector, k, nodeType, threshold, exact), topK, gate, + AnnRanked(queryVector, nodeType, threshold, exact)); } + /// <summary> + /// Recall@k of the approximate index against the exact scan, over text + /// queries (see <see cref="EmbeddingStore.RecallCheck(IEnumerable{float[]}, int, string?, float)"/>). + /// </summary> + public RecallReport RecallCheck( + IEnumerable<string> queries, int k = 10, string? nodeType = null, float threshold = 0.0f) => + _store.RecallCheck(queries, k, nodeType, threshold); + + /// <summary>Recall@k over query vectors.</summary> + public RecallReport RecallCheck( + IEnumerable<float[]> queries, int k = 10, string? nodeType = null, float threshold = 0.0f) => + _store.RecallCheck(queries, k, nodeType, threshold); + /// <summary> /// Find nodes semantically closest to one already in the graph. /// @@ -1317,9 +1482,12 @@ public IReadOnlyList<SemanticSearchResult> SemanticMultiHop(string query, MultiH if (scores.Count == 0) return Array.Empty<SemanticSearchResult>(); } + // With blend the scan above already scored every node, so the seeds come + // from the exact scan, never the approximate index (as in the Python port). + bool exactSeeds = options.Blend > 0f; var seeds = SeedSearch( - k => _store.SimilaritySearchVector(queryVector, k, null, options.Threshold), - options.TopKSeeds, gate); + k => _store.SimilaritySearchVector(queryVector, k, null, options.Threshold, exactSeeds), + options.TopKSeeds, gate, AnnRanked(queryVector, null, options.Threshold, exactSeeds)); if (seeds.Count == 0) return Array.Empty<SemanticSearchResult>(); float Combined(float baseScore, string key) => @@ -1468,7 +1636,14 @@ float Combined(float baseScore, string key) => var gate = NodeGate(nodeFilter, nodeTypes); if (gate is not null && !gate(targetRef)) return null; - var seeds = SeedSearch(k => _store.SimilaritySearch(query, k, null, threshold), topKSeeds, gate); + float[]? pathQuery = null; + if (gate is not null && _store.Ann != AnnMode.None) pathQuery = _store.EncodeQuery(query); + var seeds = SeedSearch( + k => pathQuery is null + ? _store.SimilaritySearch(query, k, null, threshold) + : _store.SimilaritySearchVector(pathQuery, k, null, threshold), + topKSeeds, gate, + pathQuery is null ? null : AnnRanked(pathQuery, null, threshold)); SemanticSearchResult? best = null; foreach (var seed in seeds) @@ -1845,6 +2020,14 @@ public void Save(string path, bool embeddings = true) /// Receives warnings raised while loading and afterwards; null sends them to /// standard error. /// </param> + /// <param name="ann"> + /// Opt-in approximate index for the loaded graph's store. Nothing about it + /// is saved or loaded: the index is built lazily from the loaded vectors. + /// </param> + /// <param name="annNlist">Cells in the index; null picks about 4 * sqrt(N).</param> + /// <param name="annNprobe">Cells scanned per query; null picks about 5% of nlist.</param> + /// <param name="annMinSize">Below this many vectors search stays exact.</param> + /// <param name="annSeed">Seed for the k-means initialisation.</param> /// <exception cref="GraphEmbeddingMismatchException">Digest mismatch with <see cref="GraphMismatchMode.Error"/>.</exception> public static SemanticGraph Load( string path, @@ -1855,7 +2038,12 @@ public static SemanticGraph Load( GraphMismatchMode onMismatch = GraphMismatchMode.Error, string? version = null, ModelMismatchMode onModelMismatch = ModelMismatchMode.Warn, - Action<string>? onWarning = null) + Action<string>? onWarning = null, + AnnMode ann = AnnMode.None, + int? annNlist = null, + int? annNprobe = null, + int annMinSize = EmbeddingStore.DefaultAnnMinSize, + int annSeed = 0) { if (!Enum.IsDefined(onMismatch)) throw new ArgumentOutOfRangeException(nameof(onMismatch), onMismatch, "unknown GraphMismatchMode"); @@ -1866,7 +2054,8 @@ public static SemanticGraph Load( var loaded = isonl ? ISONGraph.FromIsonl(graphText, name) : ISONGraph.FromIson(graphText, name); var graph = new SemanticGraph( - loaded.Name, encoder, autoEmbed, embedFields, loaded.Directed, version, onModelMismatch); + loaded.Name, encoder, autoEmbed, embedFields, loaded.Directed, version, onModelMismatch, + ann, annNlist, annNprobe, annMinSize, annSeed); if (onWarning is not null) graph._store.Warning += onWarning; foreach (var node in loaded.Nodes()) diff --git a/isongraph-vector-csharp/src/IsonGraph.Vector/IvfIndex.cs b/isongraph-vector-csharp/src/IsonGraph.Vector/IvfIndex.cs new file mode 100644 index 0000000..a3ce1f5 --- /dev/null +++ b/isongraph-vector-csharp/src/IsonGraph.Vector/IvfIndex.cs @@ -0,0 +1,671 @@ +using System.Diagnostics; +using System.Numerics; +using NVec = System.Numerics.Vector; +using System.Runtime.InteropServices; + +namespace IsonGraph.Vector; + +/// <summary> +/// Inverted-file ("IVF-flat") index over a contiguous copy of a store's vectors. +/// +/// Spherical k-means partitions the rows into <c>Nlist</c> cells by direction, +/// which is what cosine similarity compares. A query scores the <c>lists</c> +/// cells whose centroids are nearest and only the rows inside them, with the +/// same cosine formula the exact scan uses. +/// +/// Storage is struct-of-arrays: one <c>float[]</c> matrix (row stride padded +/// to a whole number of SIMD lanes, so the dot product has no scalar tail), +/// the row norms, a type code per row and the owning record. The cells are a +/// per-row cell number (<c>_assign</c>, -1 = none) plus the rows grouped by +/// cell (<c>_order</c>/<c>_offsets</c>). +/// +/// It follows the store it belongs to instead of being rebuilt on every write: +/// * a new or updated row is "pending": it has no cell, and every query +/// scores the pending rows exactly, so a write is visible immediately; +/// * pending rows are folded into their nearest cell in one pass once there +/// are enough of them (the store decides when); +/// * a delete clears the row's cell; stale entries in <c>_order</c> are +/// skipped at query time (a row is only taken from the cell it is assigned to); +/// * compaction (when a quarter of the slots are dead) remaps row numbers and +/// carries the cells across without retraining; +/// * <c>Churn</c> counts every write and delete since the centroids were +/// trained; the store retrains when it passes a fraction of the trained size. +/// +/// Not thread-safe: the owning store calls it only under its lock. Queries +/// reuse scratch buffers owned by the index, so a query allocates nothing +/// per scanned row (only its result objects). +/// </summary> +internal sealed class IvfIndex +{ + // -- rows ------------------------------------------------------------------- + private readonly int _dim; + private readonly int _stride; // dim rounded up to the SIMD width + private float[] _matrix; + private float[] _norms; + private int[] _codes; + private bool[] _alive; + private EmbeddingRecord?[] _recs; + private int[] _assign; + private int _n; // slots in use, tombstones included + private int _dead; + private readonly Dictionary<string, int> _pos = new(); // record id -> slot + private readonly Dictionary<string, int> _codeOf = new(); // node type -> code + + // -- cells ------------------------------------------------------------------ + private float[] _centroids = Array.Empty<float>(); + private int[] _order = Array.Empty<int>(); + private int[] _offsets = Array.Empty<int>(); + private readonly HashSet<int> _pending = new(); + + public int Nlist { get; private set; } + public long Churn { get; private set; } + public int TrainedN { get; private set; } + public double BuildSeconds { get; private set; } + public int PendingCount => _pending.Count; + public int LiveCount => _n - _dead; + public long VectorBytes => ((long)_matrix.Length + _norms.Length) * sizeof(float); + public long IndexBytes => + ((long)_centroids.Length + _assign.Length + _order.Length + _offsets.Length) * sizeof(int); + + // -- scratch (single-threaded, under the store lock) ------------------------ + private float[] _q; + private float[] _cKey = Array.Empty<float>(); + private int[] _cId = Array.Empty<int>(); + private int[] _candRows = new int[256]; + private float[] _candScores = new float[256]; + private int _candCount; + private int[] _topRows = new int[16]; + private float[] _topScores = new float[16]; + + private IvfIndex(int dim, int capacity) + { + _dim = dim; + int lanes = Vector<float>.Count; + _stride = (dim + lanes - 1) / lanes * lanes; + long cells = (long)capacity * _stride; + if (cells > Array.MaxLength) + throw new EmbeddingException( + $"approximate index would need {cells} floats ({capacity} vectors x {_stride}); " + + "the contiguous copy is limited to about 2 billion values"); + _matrix = new float[cells]; + _norms = new float[capacity]; + _codes = new int[capacity]; + _alive = new bool[capacity]; + _recs = new EmbeddingRecord?[capacity]; + _assign = new int[capacity]; + Array.Fill(_assign, -1); + _q = new float[_stride]; + } + + /// <summary>Copy <paramref name="records"/> into a fresh, untrained index. Call <see cref="Train"/> next.</summary> + public static IvfIndex FromRecords(IEnumerable<EmbeddingRecord> records, int dim, int count) + { + var index = new IvfIndex(dim, count + count / 8 + 16); + foreach (var record in records) index.Append(record); + return index; + } + + // -- writes ----------------------------------------------------------------- + + private int CodeFor(string nodeType) + { + if (!_codeOf.TryGetValue(nodeType, out int code)) + _codeOf[nodeType] = code = _codeOf.Count; + return code; + } + + private void Grow() + { + int capacity = Math.Max(_alive.Length + _alive.Length / 2, _alive.Length + 64); + if ((long)capacity * _stride > Array.MaxLength) + capacity = Array.MaxLength / _stride; + if (capacity <= _alive.Length) + throw new EmbeddingException("approximate index is full (contiguous copy limited to about 2 billion values)"); + int old = _alive.Length; + Array.Resize(ref _matrix, capacity * _stride); + Array.Resize(ref _norms, capacity); + Array.Resize(ref _codes, capacity); + Array.Resize(ref _alive, capacity); + Array.Resize(ref _recs, capacity); + Array.Resize(ref _assign, capacity); + Array.Fill(_assign, -1, old, capacity - old); + } + + private int Append(EmbeddingRecord record) + { + if (_n >= _alive.Length) Grow(); + int i = _n++; + _pos[record.Id] = i; + _alive[i] = true; + Store(i, record); + return i; + } + + private void Store(int i, EmbeddingRecord record) + { + var dst = new Span<float>(_matrix, i * _stride, _stride); + record.Vector.AsSpan(0, _dim).CopyTo(dst); + dst.Slice(_dim).Clear(); + _norms[i] = MathF.Sqrt(Dot(dst, dst)); + _codes[i] = CodeFor(record.NodeRef.Type); + _recs[i] = record; + } + + /// <summary>A new or replaced vector: visible to the next query, folded into a cell later.</summary> + public void Put(EmbeddingRecord record) + { + if (_pos.TryGetValue(record.Id, out int i)) + Store(i, record); // overwrite in place: keeps its position + else + i = Append(record); + _assign[i] = -1; + _pending.Add(i); + Churn++; + } + + /// <summary>A deleted vector: the slot becomes a tombstone no query will read.</summary> + public void Drop(string id) + { + if (!_pos.Remove(id, out int i)) return; + _norms[i] = 0f; + _alive[i] = false; + _recs[i] = null; + _assign[i] = -1; + _pending.Remove(i); + _dead++; + Churn++; + while (_n > 0 && !_alive[_n - 1]) // trailing tombstones just shorten the table + { + _n--; + _dead--; + } + if (_dead > 32 && _dead * 4 > _n) Compact(); + } + + /// <summary>Squeeze tombstones out, keeping the live rows in order, and carry the cells across.</summary> + private void Compact() + { + var remap = new int[_n]; + int kept = 0; + for (int i = 0; i < _n; i++) + { + if (!_alive[i]) { remap[i] = -1; continue; } + remap[i] = kept; + if (kept != i) // destination never overtakes the source + { + Array.Copy(_matrix, i * _stride, _matrix, kept * _stride, _stride); + _norms[kept] = _norms[i]; + _codes[kept] = _codes[i]; + _recs[kept] = _recs[i]; + _assign[kept] = _assign[i]; + } + kept++; + } + if (_pending.Count > 0) + { + var moved = new List<int>(_pending.Count); + foreach (int old in _pending) moved.Add(remap[old]); + _pending.Clear(); + foreach (int row in moved) _pending.Add(row); + } + for (int i = kept; i < _n; i++) + { + _recs[i] = null; + _alive[i] = false; + _assign[i] = -1; + _norms[i] = 0f; + } + for (int i = 0; i < kept; i++) + { + _alive[i] = true; + _pos[_recs[i]!.Id] = i; + } + _n = kept; + _dead = 0; + Relist(); + } + + // -- training --------------------------------------------------------------- + + private struct SplitMix64 + { + private ulong _s; + public SplitMix64(int seed) => _s = unchecked((ulong)(long)seed * 0x9E3779B97F4A7C15UL + 0x632BE59BD9B4E019UL); + public ulong Next() + { + unchecked + { + _s += 0x9E3779B97F4A7C15UL; + ulong z = _s; + z = (z ^ (z >> 30)) * 0xBF58476D1CE4E5B9UL; + z = (z ^ (z >> 27)) * 0x94D049BB133111EBUL; + return z ^ (z >> 31); + } + } + public int NextInt(int bound) => (int)(((Next() >> 32) * (ulong)bound) >> 32); + } + + /// <summary> + /// Train centroids on a sample, then assign every live row. Deterministic + /// for a given data set, <paramref name="nlist"/>, <paramref name="seed"/> + /// and SIMD width (the assignment passes run in parallel but every row's + /// answer is independent of the others, and the centroid sums are + /// accumulated in a fixed order). + /// </summary> + public void Train(int nlist, int seed, int iters = 8) + { + var started = Stopwatch.GetTimestamp(); + var live = new int[LiveCount]; + for (int i = 0, j = 0; i < _n; i++) + if (_alive[i]) live[j++] = i; + int total = live.Length; + nlist = Math.Max(1, Math.Min(nlist, total)); + var rng = new SplitMix64(seed); + + // Sample up to 40 x nlist live rows (kept in row order). + int sampleN = (int)Math.Min(total, 40L * nlist); + int[] sample; + if (sampleN == total) + sample = live; + else + { + for (int i = 0; i < sampleN; i++) // partial Fisher-Yates + { + int j = i + rng.NextInt(total - i); + (live[i], live[j]) = (live[j], live[i]); + } + sample = new int[sampleN]; + Array.Copy(live, sample, sampleN); + Array.Sort(sample); + } + + // Unit-length sample rows (a zero row stays zero). + var x = new float[(long)sampleN * _stride]; + for (int r = 0; r < sampleN; r++) + { + int row = sample[r]; + float norm = _norms[row]; + float scale = norm > 0f ? 1f / norm : 0f; + var src = new ReadOnlySpan<float>(_matrix, row * _stride, _stride); + var dst = new Span<float>(x, r * _stride, _stride); + for (int d = 0; d < _stride; d++) dst[d] = src[d] * scale; + } + + // Initial centroids: nlist distinct sample rows. + var cent = new float[(long)nlist * _stride]; + var pick = new int[sampleN]; + for (int i = 0; i < sampleN; i++) pick[i] = i; + for (int c = 0; c < nlist; c++) + { + int j = c + rng.NextInt(sampleN - c); + (pick[c], pick[j]) = (pick[j], pick[c]); + Array.Copy(x, pick[c] * _stride, cent, c * _stride, _stride); + } + + var cell = new int[sampleN]; + var counts = new int[nlist]; + var sums = new double[(long)nlist * _stride]; + for (int it = 0; it < iters; it++) + { + AssignNearest(x, null, sampleN, cent, nlist, cell); + Array.Clear(counts); + Array.Clear(sums); + for (int r = 0; r < sampleN; r++) // sequential: a fixed summation order + { + int c = cell[r]; + counts[c]++; + var src = new ReadOnlySpan<float>(x, r * _stride, _stride); + var acc = new Span<double>(sums, c * _stride, _stride); + for (int d = 0; d < _stride; d++) acc[d] += src[d]; + } + for (int c = 0; c < nlist; c++) + { + var dst = new Span<float>(cent, c * _stride, _stride); + var acc = new ReadOnlySpan<double>(sums, c * _stride, _stride); + for (int d = 0; d < _stride; d++) dst[d] = (float)acc[d]; + float length = MathF.Sqrt(Dot(dst, dst)); + if (counts[c] == 0 || length == 0f) // re-seed a dead centroid on a random point + { + Array.Copy(x, rng.NextInt(sampleN) * _stride, cent, c * _stride, _stride); + length = MathF.Sqrt(Dot(dst, dst)); + } + if (length > 0f) + for (int d = 0; d < _stride; d++) dst[d] /= length; + } + } + + _centroids = cent; + Nlist = nlist; + _cKey = new float[nlist]; + _cId = new int[nlist]; + + // Every row, in the order of the table. (Raw rows rank centroids the same + // as unit rows: scaling by a positive norm does not change the argmax.) + AssignNearest(_matrix, null, _n, cent, nlist, _assign); + for (int i = 0; i < _n; i++) + if (!_alive[i]) _assign[i] = -1; + _pending.Clear(); + Churn = 0; + TrainedN = total; + Relist(); + BuildSeconds = Stopwatch.GetElapsedTime(started).TotalSeconds; + } + + /// <summary>Regroup the rows by cell (a stable counting sort of <c>_assign</c>).</summary> + private void Relist() + { + if (_offsets.Length != Nlist + 1) _offsets = new int[Nlist + 1]; + else Array.Clear(_offsets); + int assigned = 0; + for (int i = 0; i < _n; i++) + { + int c = _assign[i]; + if (c >= 0) { _offsets[c + 1]++; assigned++; } + } + for (int c = 0; c < Nlist; c++) _offsets[c + 1] += _offsets[c]; + if (_order.Length < assigned) _order = new int[assigned + assigned / 8 + 16]; + var cursor = new int[Nlist]; + Array.Copy(_offsets, cursor, Nlist); + for (int i = 0; i < _n; i++) + { + int c = _assign[i]; + if (c >= 0) _order[cursor[c]++] = i; + } + } + + /// <summary>Give every pending row its nearest cell and regroup.</summary> + public void Fold() + { + if (_pending.Count == 0) return; + var ids = new int[_pending.Count]; + _pending.CopyTo(ids); + Array.Sort(ids); + var cells = new int[ids.Length]; + AssignNearest(_matrix, ids, ids.Length, _centroids, Nlist, cells); + for (int i = 0; i < ids.Length; i++) _assign[ids[i]] = cells[i]; + _pending.Clear(); + Relist(); + } + + // -- nearest-centroid kernel ------------------------------------------------ + + private const int BlockRows = 256; + private const int CentroidTile = 64; + + /// <summary> + /// For <paramref name="count"/> rows (consecutive from 0, or the rows named + /// by <paramref name="ids"/>) of <paramref name="data"/>, the index of the + /// centroid with the largest dot product (the lowest index wins a tie). + /// <paramref name="output"/> is indexed like the rows, or by the row itself + /// when <paramref name="ids"/> is null. + /// </summary> + private void AssignNearest(float[] data, int[]? ids, int count, float[] cent, int nlist, int[] output) + { + if (count == 0) return; + int stride = _stride; + int blocks = (count + BlockRows - 1) / BlockRows; + if (blocks == 1) + { + NearestBlock(data, ids, 0, count, cent, nlist, stride, output); + return; + } + Parallel.For(0, blocks, b => + { + int start = b * BlockRows; + NearestBlock(data, ids, start, Math.Min(BlockRows, count - start), cent, nlist, stride, output); + }); + } + + private static void NearestBlock( + float[] data, int[]? ids, int start, int count, float[] cent, int nlist, int stride, int[] output) + { + Span<float> best = stackalloc float[BlockRows]; + Span<int> bestIdx = stackalloc int[BlockRows]; + Span<int> offs = stackalloc int[BlockRows]; + best.Slice(0, count).Fill(float.NegativeInfinity); + bestIdx.Slice(0, count).Clear(); + for (int i = 0; i < count; i++) + offs[i] = (ids is null ? start + i : ids[start + i]) * stride; + + for (int tile = 0; tile < nlist; tile += CentroidTile) + { + int tileEnd = Math.Min(nlist, tile + CentroidTile); + int g = 0; + for (; g + 4 <= count; g += 4) + Update4(data, offs[g], offs[g + 1], offs[g + 2], offs[g + 3], cent, tile, tileEnd, + stride, best, bestIdx, g); + for (; g < count; g++) + { + var row = new ReadOnlySpan<float>(data, offs[g], stride); + for (int c = tile; c < tileEnd; c++) + { + float s = Dot(row, new ReadOnlySpan<float>(cent, c * stride, stride)); + if (s > best[g]) { best[g] = s; bestIdx[g] = c; } + } + } + } + for (int i = 0; i < count; i++) + output[start + i] = bestIdx[i]; + } + + /// <summary>Four rows against centroids [from, to): two centroids per step, eight independent accumulators.</summary> + private static void Update4( + float[] data, int o0, int o1, int o2, int o3, float[] cent, int from, int to, int stride, + Span<float> best, Span<int> bestIdx, int g) + { + var r0 = MemoryMarshal.Cast<float, Vector<float>>(new ReadOnlySpan<float>(data, o0, stride)); + var r1 = MemoryMarshal.Cast<float, Vector<float>>(new ReadOnlySpan<float>(data, o1, stride)); + var r2 = MemoryMarshal.Cast<float, Vector<float>>(new ReadOnlySpan<float>(data, o2, stride)); + var r3 = MemoryMarshal.Cast<float, Vector<float>>(new ReadOnlySpan<float>(data, o3, stride)); + int n = r0.Length; + int c = from; + for (; c + 1 < to; c += 2) + { + var ca = MemoryMarshal.Cast<float, Vector<float>>(new ReadOnlySpan<float>(cent, c * stride, stride)); + var cb = MemoryMarshal.Cast<float, Vector<float>>(new ReadOnlySpan<float>(cent, (c + 1) * stride, stride)); + Vector<float> a0 = default, a1 = default, a2 = default, a3 = default; + Vector<float> b0 = default, b1 = default, b2 = default, b3 = default; + for (int i = 0; i < n; i++) + { + var x = ca[i]; + var y = cb[i]; + var v0 = r0[i]; a0 += v0 * x; b0 += v0 * y; + var v1 = r1[i]; a1 += v1 * x; b1 += v1 * y; + var v2 = r2[i]; a2 += v2 * x; b2 += v2 * y; + var v3 = r3[i]; a3 += v3 * x; b3 += v3 * y; + } + Offer(best, bestIdx, g, NVec.Sum(a0), c); + Offer(best, bestIdx, g, NVec.Sum(b0), c + 1); + Offer(best, bestIdx, g + 1, NVec.Sum(a1), c); + Offer(best, bestIdx, g + 1, NVec.Sum(b1), c + 1); + Offer(best, bestIdx, g + 2, NVec.Sum(a2), c); + Offer(best, bestIdx, g + 2, NVec.Sum(b2), c + 1); + Offer(best, bestIdx, g + 3, NVec.Sum(a3), c); + Offer(best, bestIdx, g + 3, NVec.Sum(b3), c + 1); + } + if (c < to) + { + var ca = MemoryMarshal.Cast<float, Vector<float>>(new ReadOnlySpan<float>(cent, c * stride, stride)); + Vector<float> a0 = default, a1 = default, a2 = default, a3 = default; + for (int i = 0; i < n; i++) + { + var x = ca[i]; + a0 += r0[i] * x; a1 += r1[i] * x; a2 += r2[i] * x; a3 += r3[i] * x; + } + Offer(best, bestIdx, g, NVec.Sum(a0), c); + Offer(best, bestIdx, g + 1, NVec.Sum(a1), c); + Offer(best, bestIdx, g + 2, NVec.Sum(a2), c); + Offer(best, bestIdx, g + 3, NVec.Sum(a3), c); + } + } + + private static void Offer(Span<float> best, Span<int> bestIdx, int g, float score, int c) + { + if (score > best[g]) { best[g] = score; bestIdx[g] = c; } + } + + /// <summary>Dot product of two equal-length, lane-padded spans.</summary> + private static float Dot(ReadOnlySpan<float> a, ReadOnlySpan<float> b) + { + var va = MemoryMarshal.Cast<float, Vector<float>>(a); + var vb = MemoryMarshal.Cast<float, Vector<float>>(b); + int n = Math.Min(va.Length, vb.Length); + Vector<float> s0 = default, s1 = default, s2 = default, s3 = default; + int i = 0; + for (; i + 4 <= n; i += 4) + { + s0 += va[i] * vb[i]; + s1 += va[i + 1] * vb[i + 1]; + s2 += va[i + 2] * vb[i + 2]; + s3 += va[i + 3] * vb[i + 3]; + } + for (; i < n; i++) s0 += va[i] * vb[i]; + return NVec.Sum((s0 + s1) + (s2 + s3)); + } + + // -- query ------------------------------------------------------------------ + + /// <summary> + /// Score the rows in the <paramref name="lists"/> cells nearest the query, + /// plus every pending row, into the candidate buffers (rows that pass the + /// type filter and <paramref name="threshold"/>). Returns false for a zero + /// query, which has no direction to rank cells by. Allocates nothing per row. + /// </summary> + public bool Scan(float[] query, string? nodeType, float threshold, int lists, out int passing) + { + passing = 0; + _candCount = 0; + query.AsSpan(0, _dim).CopyTo(_q); + float qnorm = MathF.Sqrt(Dot(_q, _q)); + if (qnorm == 0f) return false; + int code = -1; + if (nodeType is not null && !_codeOf.TryGetValue(nodeType, out code)) + return true; // no row has this type + + int stride = _stride; + int nlist = Nlist; + var centroids = _centroids; + for (int c = 0; c < nlist; c++) + { + _cKey[c] = -Dot(_q, new ReadOnlySpan<float>(centroids, c * stride, stride)); + _cId[c] = c; + } + if (lists < nlist) + MemoryExtensions.Sort(_cKey.AsSpan(0, nlist), _cId.AsSpan(0, nlist)); + else + lists = nlist; + + var matrix = _matrix; + var norms = _norms; + var codes = _codes; + var assign = _assign; + var order = _order; + var offsets = _offsets; + for (int l = 0; l < lists; l++) + { + int cell = _cId[l]; + for (int p = offsets[cell], end = offsets[cell + 1]; p < end; p++) + { + int row = order[p]; + if (assign[row] != cell) continue; // deleted or updated since the last regroup + if (code >= 0 && codes[row] != code) continue; + Consider(row, qnorm, threshold); + } + } + foreach (int row in _pending) + { + if (code >= 0 && codes[row] != code) continue; + Consider(row, qnorm, threshold); + } + passing = _candCount; + return true; + } + + private void Consider(int row, float qnorm, float threshold) + { + float dot = Dot(_q, new ReadOnlySpan<float>(_matrix, row * _stride, _stride)); + float denom = _norms[row] * qnorm; + float score = denom > 0f ? dot / denom : 0f; + if (!(score >= threshold)) return; + if (_candCount == _candRows.Length) + { + Array.Resize(ref _candRows, _candCount * 2); + Array.Resize(ref _candScores, _candCount * 2); + } + _candRows[_candCount] = row; + _candScores[_candCount++] = score; + } + + private static bool Better(float s1, int r1, float s2, int r2) => s1 > s2 || (s1 == s2 && r1 < r2); + + /// <summary> + /// The best <paramref name="k"/> of the last <see cref="Scan"/>, best first; + /// equal scores keep row (insertion) order, as in the exact scan. + /// </summary> + public List<SimilarityResult> TopK(int k) + { + if (_topRows.Length < k) + { + _topRows = new int[k]; + _topScores = new float[k]; + } + var topRows = _topRows; + var topScores = _topScores; + int m = 0; + for (int i = 0; i < _candCount; i++) + { + float s = _candScores[i]; + int r = _candRows[i]; + if (m == k && !Better(s, r, topScores[k - 1], topRows[k - 1])) continue; + int pos = m < k ? m++ : k - 1; + while (pos > 0 && Better(s, r, topScores[pos - 1], topRows[pos - 1])) + { + topScores[pos] = topScores[pos - 1]; + topRows[pos] = topRows[pos - 1]; + pos--; + } + topScores[pos] = s; + topRows[pos] = r; + } + var results = new List<SimilarityResult>(m); + for (int i = 0; i < m; i++) + { + var rec = _recs[topRows[i]]!; + results.Add(new SimilarityResult(rec.NodeRef, topScores[i], rec.Text)); + } + return results; + } + + /// <summary> + /// Every candidate of the last <see cref="Scan"/>, best first, as a + /// snapshot that stays valid however the index changes afterwards. + /// </summary> + public IEnumerable<SimilarityResult> RankedSnapshot() + { + int m = _candCount; + var keys = new ulong[m]; + for (int i = 0; i < m; i++) + { + int bits = BitConverter.SingleToInt32Bits(_candScores[i]); + uint ordered = bits < 0 ? ~(uint)bits : (uint)bits | 0x80000000u; // ascending = ascending score + keys[i] = ((ulong)~ordered << 32) | (uint)_candRows[i]; // best score, then lowest row, first + } + Array.Sort(keys); + var recs = new EmbeddingRecord[m]; + var scores = new float[m]; + for (int i = 0; i < m; i++) + { + uint ordered = ~(uint)(keys[i] >> 32); + int bits = (ordered & 0x80000000u) != 0 ? (int)(ordered & 0x7FFFFFFFu) : (int)~ordered; + scores[i] = BitConverter.Int32BitsToSingle(bits); + recs[i] = _recs[(int)(keys[i] & 0xFFFFFFFFu)]!; + } + return Enumerate(recs, scores); + } + + private static IEnumerable<SimilarityResult> Enumerate(EmbeddingRecord[] recs, float[] scores) + { + for (int i = 0; i < recs.Length; i++) + yield return new SimilarityResult(recs[i].NodeRef, scores[i], recs[i].Text); + } +} diff --git a/isongraph-vector-csharp/tests/IsonGraph.Vector.Tests/AnnTests.cs b/isongraph-vector-csharp/tests/IsonGraph.Vector.Tests/AnnTests.cs new file mode 100644 index 0000000..585a056 --- /dev/null +++ b/isongraph-vector-csharp/tests/IsonGraph.Vector.Tests/AnnTests.cs @@ -0,0 +1,681 @@ +using IsonGraph; +using IsonGraph.Vector; +using Xunit; + +namespace IsonGraph.Vector.Tests; + +/// <summary>Seeded clustered unit-ish vectors, the shape of real embeddings: tight groups around a few centres.</summary> +internal sealed class Clustered +{ + private readonly Random _rng; + private readonly float[][] _centers; + private readonly int _dim; + private readonly float _noise; + + public Clustered(int dim, int clusters = 40, int seed = 1, float noise = 0.35f) + { + _rng = new Random(seed); + _dim = dim; + _noise = noise; + _centers = new float[clusters][]; + for (int c = 0; c < clusters; c++) + { + var v = new float[dim]; + double norm = 0; + for (int d = 0; d < dim; d++) { v[d] = (float)Gauss(); norm += v[d] * v[d]; } + float inv = (float)(1 / Math.Sqrt(norm)); + for (int d = 0; d < dim; d++) v[d] *= inv; + _centers[c] = v; + } + } + + private double Gauss() + { + double u1 = 1.0 - _rng.NextDouble(), u2 = _rng.NextDouble(); + return Math.Sqrt(-2.0 * Math.Log(u1)) * Math.Cos(2.0 * Math.PI * u2); + } + + public float[] Draw() + { + var center = _centers[_rng.Next(_centers.Length)]; + var v = new float[_dim]; + float scale = _noise / MathF.Sqrt(_dim); + for (int d = 0; d < _dim; d++) v[d] = center[d] + scale * (float)Gauss(); + return v; + } + + /// <summary>A vector from the same clusters, drawn from its own stream (so it does not disturb <see cref="Draw()"/>).</summary> + public float[] Query(int salt) + { + var r = new Random(salt); + var center = _centers[r.Next(_centers.Length)]; + var v = new float[_dim]; + float scale = _noise / MathF.Sqrt(_dim); + for (int d = 0; d < _dim; d++) + { + double u1 = 1.0 - r.NextDouble(), u2 = r.NextDouble(); + v[d] = center[d] + scale * (float)(Math.Sqrt(-2.0 * Math.Log(u1)) * Math.Cos(2.0 * Math.PI * u2)); + } + return v; + } + + public float[][] Draw(int count) + { + var all = new float[count][]; + for (int i = 0; i < count; i++) all[i] = Draw(); + return all; + } +} + +/// <summary>Encodes "item N" as the N-th clustered vector and any other text as a clustered query vector.</summary> +internal sealed class ClusteredEncoder : IEmbeddingEncoder +{ + private readonly Clustered _gen; + private readonly float[][] _items; + public ClusteredEncoder(int dim, int items) + { + Dimension = dim; + _gen = new Clustered(dim, seed: 5); + _items = _gen.Draw(items); + } + public int Dimension { get; } + public float[] Encode(string text) => + text.StartsWith("item ") ? _items[int.Parse(text.AsSpan(5))] : _gen.Query(text.Aggregate(17, (h, c) => unchecked(h * 31 + c))); + public IReadOnlyList<float[]> EncodeBatch(IReadOnlyList<string> texts) => texts.Select(Encode).ToList(); +} + +public class AnnTests +{ + private const double RecallFloor = 0.95; + + private static void Close(double expected, double actual, double tol = 1e-5) => + Assert.True(Math.Abs(expected - actual) <= tol, $"expected {expected} but was {actual}"); + + private static NodeRef Ref(int i, string type = "doc") => new(type, i.ToString()); + + private static (EmbeddingStore Store, float[][] Data, Clustered Gen) Build( + int n = 4000, int dim = 32, int seed = 1, Func<int, string>? types = null, + AnnMode ann = AnnMode.Ivf, int? nlist = 64, int? nprobe = 8, int minSize = 0, int annSeed = 0) + { + var gen = new Clustered(dim, seed: seed); + var data = gen.Draw(n); + var store = new EmbeddingStore( + ann: ann, annNlist: nlist, annNprobe: nprobe, annMinSize: minSize, annSeed: annSeed); + for (int i = 0; i < n; i++) + store.Add(Ref(i, types?.Invoke(i) ?? "doc"), "text " + i, data[i]); + return (store, data, gen); + } + + /// <summary>The definition of exact: cosine over every vector, best first, ties in insertion order.</summary> + private static List<(NodeRef Ref, float Score)> BruteForce( + IReadOnlyList<float[]> data, IReadOnlyList<NodeRef> refs, float[] q, int k, float threshold = 0f) + { + double qn = Math.Sqrt(q.Sum(x => (double)x * x)); + var scored = new List<(NodeRef, float, int)>(); + for (int i = 0; i < data.Count; i++) + { + double dot = 0, vn = 0; + for (int d = 0; d < q.Length; d++) { dot += (double)q[d] * data[i][d]; vn += (double)data[i][d] * data[i][d]; } + float s = vn == 0 || qn == 0 ? 0f : (float)(dot / (Math.Sqrt(vn) * qn)); + if (s >= threshold) scored.Add((refs[i], s, i)); + } + return scored.OrderByDescending(t => t.Item2).ThenBy(t => t.Item3).Take(k) + .Select(t => (t.Item1, t.Item2)).ToList(); + } + + private static List<NodeRef> Refs(int n, Func<int, string>? types = null) => + Enumerable.Range(0, n).Select(i => Ref(i, types?.Invoke(i) ?? "doc")).ToList(); + + // -- off by default ----------------------------------------------------------- + + [Fact] + public void Off_by_default_is_exact_and_has_no_index() + { + var (store, data, gen) = Build(ann: AnnMode.None); + Assert.Equal(AnnMode.None, store.Ann); + var stats = store.GetAnnStats(); + Assert.False(stats.Enabled); + Assert.False(stats.Active); + Assert.Throws<EmbeddingException>(() => store.BuildAnnIndex()); + + var refs = Refs(data.Length); + foreach (var q in gen.Draw(5)) + { + var got = store.SimilaritySearchVector(q, 10); + var want = BruteForce(data, refs, q, 10); + Assert.Equal(want.Select(w => w.Ref), got.Select(g => g.NodeRef)); + } + Assert.Equal(0, store.GetAnnStats().Searches.Approximate); + Assert.Equal(0, store.GetAnnStats().Searches.Builds); + } + + [Fact] + public void Exact_results_are_identical_with_or_without_the_flag_on() + { + var (off, _, gen) = Build(ann: AnnMode.None); + var (on, _, _) = Build(ann: AnnMode.Ivf); + foreach (var q in gen.Draw(5)) + { + var a = off.SimilaritySearchVector(q, 10); + var b = on.SimilaritySearchVector(q, 10, exact: true); + Assert.Equal(a.Select(r => (r.NodeRef, r.Score)), b.Select(r => (r.NodeRef, r.Score))); + } + } + + [Fact] + public void Argument_validation() + { + Assert.Throws<ArgumentOutOfRangeException>(() => new EmbeddingStore(ann: (AnnMode)99)); + Assert.Throws<ArgumentOutOfRangeException>(() => new EmbeddingStore(ann: AnnMode.Ivf, annNlist: 0)); + Assert.Throws<ArgumentOutOfRangeException>(() => new EmbeddingStore(ann: AnnMode.Ivf, annNprobe: 0)); + Assert.Throws<ArgumentOutOfRangeException>(() => new EmbeddingStore(ann: AnnMode.Ivf, annMinSize: -1)); + var store = new EmbeddingStore(ann: AnnMode.Ivf); + Assert.Throws<ArgumentOutOfRangeException>(() => store.AnnNprobe = 0); + store.AnnNprobe = 5; + Assert.Equal(5, store.AnnNprobe); + Assert.Equal(EmbeddingStore.DefaultAnnMinSize, store.GetAnnStats().MinSize); + Assert.Equal(50_000, EmbeddingStore.DefaultAnnMinSize); + } + + // -- recall, scores, exact override ------------------------------------------- + + [Fact] + public void Recall_floor_on_clustered_data() + { + var (store, _, gen) = Build(); + var report = store.RecallCheck(gen.Draw(60), k: 10); + Assert.True(report.Approximate); + Assert.Equal(64, report.Nlist); + Assert.Equal(8, report.Nprobe); + Assert.Equal(60, report.Queries); + Assert.True(report.Recall >= RecallFloor, $"recall {report.Recall:F3}"); + Assert.InRange(report.MinRecall, 0.0, report.Recall); + Assert.True(report.Recall <= 1.0); + var stats = store.GetAnnStats(); + Assert.True(stats.Active); + Assert.Equal(64, stats.Nlist); + Assert.Equal(4000, stats.TrainedN); + Assert.True(stats.IndexBytes > 0 && stats.VectorBytes > 0); + Assert.True(stats.Searches.Approximate >= 60); + Assert.Equal(1, stats.Searches.Builds); + } + + [Fact] + public void Index_actually_narrows_the_search() + { + // With 8 of 64 cells probed the index must not be reading every row: + // an index that silently scanned everything would score recall 1.0. + var (store, _, gen) = Build(nprobe: 1); + var report = store.RecallCheck(gen.Draw(80), k: 10); + Assert.True(report.Approximate); + Assert.True(report.Recall < 1.0, "nprobe=1 of 64 cells should miss something"); + Assert.True(report.Recall > 0.3); + } + + [Fact] + public void Exact_override_equals_brute_force_and_skips_the_index() + { + var (store, data, gen) = Build(); + var refs = Refs(data.Length); + store.SimilaritySearchVector(data[0], 5); // warm the index + long before = store.GetAnnStats().Searches.Approximate; + foreach (var q in gen.Draw(10)) + { + var got = store.SimilaritySearchVector(q, 10, exact: true); + var want = BruteForce(data, refs, q, 10); + Assert.Equal(want.Select(w => w.Ref), got.Select(g => g.NodeRef)); + for (int i = 0; i < want.Count; i++) Close(want[i].Score, got[i].Score); + } + Assert.Equal(before, store.GetAnnStats().Searches.Approximate); + } + + [Fact] + public void Results_carry_exact_scores_sorted_descending() + { + var (store, data, gen) = Build(); + var refs = Refs(data.Length); + foreach (var q in gen.Draw(10)) + { + var truth = BruteForce(data, refs, q, data.Length, -1f).ToDictionary(t => t.Ref, t => t.Score); + var got = store.SimilaritySearchVector(q, 10); + Assert.Equal(10, got.Count); + foreach (var r in got) Close(truth[r.NodeRef], r.Score); + Assert.Equal(got.Select(r => r.Score).OrderByDescending(s => s), got.Select(r => r.Score)); + Assert.Equal(10, got.Select(r => r.NodeRef).Distinct().Count()); + Assert.All(got, r => Assert.StartsWith("text ", r.Text)); + } + } + + [Fact] + public void Probing_every_cell_is_the_exact_scan() + { + var (store, data, gen) = Build(nprobe: 64); // nprobe == nlist + var refs = Refs(data.Length); + foreach (var q in gen.Draw(8)) + { + var got = store.SimilaritySearchVector(q, 10); + var want = BruteForce(data, refs, q, 10); + Assert.Equal(want.Select(w => w.Ref), got.Select(g => g.NodeRef)); + } + var report = store.RecallCheck(gen.Draw(5)); + Assert.False(report.Approximate); + Assert.Equal(1.0, report.Recall); + Assert.Equal(0, store.GetAnnStats().Searches.Approximate); + } + + [Fact] + public void Small_store_falls_back_to_exact() + { + var (store, data, gen) = Build(n: 500, minSize: 1000); + var refs = Refs(data.Length); + var q = gen.Draw(); + Assert.Equal(BruteForce(data, refs, q, 10).Select(w => w.Ref), + store.SimilaritySearchVector(q, 10).Select(r => r.NodeRef)); + Assert.False(store.GetAnnStats().Active); + Assert.Equal(0, store.GetAnnStats().Searches.Builds); + Assert.False(store.RecallCheck(q).Approximate); + // BuildAnnIndex on a small store says so in the stats and builds nothing. + Assert.False(store.BuildAnnIndex().Active); + } + + [Fact] + public void Index_appears_when_a_growing_store_reaches_the_minimum_size() + { + var (store, data, gen) = Build(n: 90, minSize: 100, nlist: 8, nprobe: 2); + store.SimilaritySearchVector(gen.Draw(), 5); + Assert.False(store.GetAnnStats().Active); + foreach (var v in gen.Draw(20)) store.Add(Ref(1000 + store.Count), "x", v); + store.SimilaritySearchVector(gen.Draw(), 5); + Assert.True(store.GetAnnStats().Active); + // ... and goes away again when it shrinks below it. + for (int i = 0; i < 30; i++) store.Remove(Ref(i)); + store.SimilaritySearchVector(gen.Draw(), 5); + Assert.False(store.GetAnnStats().Active); + } + + [Fact] + public void TopK_null_and_zero_are_not_approximate_and_zero_query_is_exact() + { + var (store, data, gen) = Build(); + var q = gen.Draw(); + long before = store.GetAnnStats().Searches.Approximate; + Assert.Equal(4000, store.SimilaritySearchVector(q, null, threshold: -1f).Count); + Assert.Empty(store.SimilaritySearchVector(q, 0)); + Assert.Equal(before, store.GetAnnStats().Searches.Approximate); + + var zero = store.SimilaritySearchVector(new float[32], 5, threshold: 0f); + Assert.Equal(5, zero.Count); + Assert.All(zero, r => Assert.Equal(0f, r.Score)); + } + + // -- writes: add / update / remove --------------------------------------------- + + [Fact] + public void Added_vectors_are_visible_immediately() + { + var (store, data, gen) = Build(); + store.BuildAnnIndex(); + var fresh = gen.Draw(); + store.Add(new NodeRef("new", "1"), "fresh", fresh); + var top = store.SimilaritySearchVector(fresh, 3); + Assert.Equal(new NodeRef("new", "1"), top[0].NodeRef); + Assert.Equal(1.0, top[0].Score, 4); + Assert.Equal(1, store.GetAnnStats().Pending); + } + + [Fact] + public void Updated_vectors_move_and_are_listed_once() + { + var (store, data, gen) = Build(); + store.BuildAnnIndex(); + var target = gen.Draw(); + store.Add(Ref(7), "moved", target); + var hit = store.SimilaritySearchVector(target, 1)[0]; + Assert.Equal(Ref(7), hit.NodeRef); + Assert.Equal(1.0, hit.Score, 4); + Assert.Equal("moved", hit.Text); + + // Scored from the new vector everywhere, and no stale second entry. + foreach (var q in new[] { data[7], target }) + { + var res = store.SimilaritySearchVector(q, 400, threshold: -1f); + Assert.Equal(res.Count, res.Select(r => r.NodeRef).Distinct().Count()); + var truth = store.SimilaritySearchVector(q, null, threshold: -1f).ToDictionary(r => r.NodeRef, r => r.Score); + Assert.All(res, r => Close(truth[r.NodeRef], r.Score)); + } + Assert.Equal(4000, store.Count); + Assert.True(store.GetAnnStats().Searches.Approximate > 0); // answered by the index + } + + [Fact] + public void Removed_vectors_never_come_back_and_compaction_keeps_the_index() + { + var (store, data, gen) = Build(); + store.BuildAnnIndex(); + var gone = new HashSet<NodeRef>(); + for (int i = 0; i < 4000; i += 3) // a third: forces compaction + { + Assert.True(store.Remove(Ref(i))); + gone.Add(Ref(i)); + } + long builds = store.GetAnnStats().Searches.Builds; + foreach (var q in gen.Draw(15)) + { + Assert.DoesNotContain(store.SimilaritySearchVector(q, 10).Select(r => r.NodeRef), gone.Contains); + // A negative threshold with many results would surface a tombstone scoring 0. + Assert.DoesNotContain(store.SimilaritySearchVector(q, 250, threshold: -1f).Select(r => r.NodeRef), gone.Contains); + } + var report = store.RecallCheck(gen.Draw(40)); + Assert.True(report.Recall >= RecallFloor, $"recall {report.Recall:F3}"); + Assert.Equal(builds, store.GetAnnStats().Searches.Builds); // followed, not retrained + Assert.True(store.GetAnnStats().Active); + Assert.True(report.Approximate); + Assert.Equal(2666, store.Count); + } + + [Fact] + public void Pending_rows_fold_and_heavy_churn_retrains() + { + var (store, data, gen) = Build(n: 3000); + store.BuildAnnIndex(); + long builds = store.GetAnnStats().Searches.Builds; + var extra = gen.Draw(700); + for (int i = 0; i < 700; i++) store.Add(Ref(i, "late"), "", extra[i]); + Assert.Equal(700, store.GetAnnStats().Pending); + store.SimilaritySearchVector(extra[0], 5); // >= 512 pending: folded + var stats = store.GetAnnStats(); + Assert.Equal(0, stats.Pending); + Assert.Equal(builds, stats.Searches.Builds); + + var more = gen.Draw(1200); + for (int i = 0; i < 1200; i++) store.Add(Ref(i, "later"), "", more[i]); + store.SimilaritySearchVector(more[0], 5); // churn 1900 > 50% of 3000 + Assert.Equal(builds + 1, store.GetAnnStats().Searches.Builds); + Assert.Equal(4900, store.GetAnnStats().TrainedN); + Assert.True(store.RecallCheck(gen.Draw(30)).Recall >= RecallFloor); + } + + [Fact] + public void Clear_drops_the_index_and_a_new_dimension_gets_a_new_one() + { + var (store, data, gen) = Build(n: 1500, nlist: 16, nprobe: 4); + store.BuildAnnIndex(); + Assert.True(store.GetAnnStats().Active); + store.Clear(); + Assert.Empty(store.SimilaritySearchVector(data[0], 5)); // dimension reset: no mismatch either + Assert.False(store.GetAnnStats().Active); + + var small = new Clustered(8, clusters: 12, seed: 3).Draw(1200); + for (int i = 0; i < small.Length; i++) store.Add(Ref(i), "", small[i]); + var res = store.SimilaritySearchVector(small[5], 5); + Assert.Equal(Ref(5), res[0].NodeRef); + Assert.True(store.GetAnnStats().Active); + Assert.Equal(8, store.Dimension); + } + + [Fact] + public void Importing_with_replace_swaps_the_indexed_data() + { + var (store, data, gen) = Build(n: 1500, nlist: 16, nprobe: 4); + store.BuildAnnIndex(); + var other = new EmbeddingStore(); + var fresh = gen.Draw(1200); + for (int i = 0; i < fresh.Length; i++) other.Add(Ref(i, "fresh"), "f" + i, fresh[i]); + store.ImportEmbeddings(other.ExportEmbeddings(), replace: true); + Assert.Equal(1200, store.Count); + var res = store.SimilaritySearchVector(fresh[3], 5); + Assert.Equal(Ref(3, "fresh"), res[0].NodeRef); + Assert.All(res, r => Assert.Equal("fresh", r.NodeRef.Type)); + } + + // -- filters -------------------------------------------------------------------- + + [Fact] + public void NodeType_filter_widens_for_a_rare_type() + { + var (store, data, gen) = Build(types: i => i % 200 == 0 ? "rare" : "common", nprobe: 2); + var q = gen.Draw(); + var common = store.SimilaritySearchVector(q, 10, nodeType: "common"); + Assert.Equal(10, common.Count); + Assert.All(common, r => Assert.Equal("common", r.NodeRef.Type)); + Assert.True(store.GetAnnStats().Searches.Approximate > 0); + + // 20 rare rows in the store, top-15: the two probed cells cannot hold 15 of them. + var rare = store.SimilaritySearchVector(q, 15, nodeType: "rare", threshold: -1f); + Assert.Equal(15, rare.Count); + Assert.All(rare, r => Assert.Equal("rare", r.NodeRef.Type)); + var counts = store.GetAnnStats().Searches; + Assert.True(counts.Widened > 0 || counts.Exact > 0); + + Assert.Empty(store.SimilaritySearchVector(q, 5, nodeType: "nope")); + } + + [Fact] + public void Threshold_is_respected() + { + var (store, data, gen) = Build(); + var q = gen.Draw(); + float best = store.SimilaritySearchVector(q, 1, exact: true)[0].Score; + var res = store.SimilaritySearchVector(q, 10, threshold: best - 1e-3f); + Assert.All(res, r => Assert.True(r.Score >= best - 1e-3f)); + Assert.Empty(store.SimilaritySearchVector(q, 10, threshold: 2f)); + + float cut = store.SimilaritySearchVector(q, 10, exact: true)[4].Score; + var got = store.SimilaritySearchVector(q, 10, threshold: cut); + var want = store.SimilaritySearchVector(q, 10, threshold: cut, exact: true); + Assert.Equal(want.Select(r => r.NodeRef).ToHashSet(), got.Select(r => r.NodeRef).ToHashSet()); + // A threshold that fewer than k rows clear widens the probe until the scan answers. + store.SimilaritySearchVector(q, 10, threshold: 0f); + Assert.True(store.GetAnnStats().Searches.Approximate > 0); + } + + // -- determinism --------------------------------------------------------------- + + [Fact] + public void Same_data_and_seed_give_the_same_index_and_answers() + { + var (a, _, gen) = Build(annSeed: 7, nprobe: 3); + var (b, _, _) = Build(annSeed: 7, nprobe: 3); + var queries = gen.Draw(30); + foreach (var q in queries) + { + var ra = a.SimilaritySearchVector(q, 10); + var rb = b.SimilaritySearchVector(q, 10); + Assert.Equal(ra.Select(r => (r.NodeRef, r.Score)), rb.Select(r => (r.NodeRef, r.Score))); + } + // ... and again from the same store, and after a forced retrain on unchanged data. + var first = queries.Select(q => a.SimilaritySearchVector(q, 10).Select(r => r.NodeRef).ToList()).ToList(); + a.BuildAnnIndex(force: true); + var second = queries.Select(q => a.SimilaritySearchVector(q, 10).Select(r => r.NodeRef).ToList()).ToList(); + Assert.Equal(first, second); + Assert.Equal(a.GetAnnStats().TrainedN, b.GetAnnStats().TrainedN); + Assert.True(a.GetAnnStats().Active && b.GetAnnStats().Active); + Assert.Equal(2, a.GetAnnStats().Searches.Builds); // first build + forced retrain + } + + // -- persistence: nothing changes ------------------------------------------------ + + private static SemanticGraph Graph(AnnMode ann, int n = 1500, int dim = 32) + { + var encoder = new ClusteredEncoder(dim, n); + var graph = new SemanticGraph( + "ann", encoder, autoEmbed: false, + ann: ann, annMinSize: 0, annNlist: 32, annNprobe: 4); + for (int i = 0; i < n; i++) + graph.AddNode("item", i.ToString(), new() { ["group"] = (long)(i % 50) }, embedText: "item " + i); + return graph; + } + + [Fact] + public void Graph_file_sidecar_and_digest_are_unchanged_by_the_index() + { + string dir = System.IO.Path.Combine(System.IO.Path.GetTempPath(), "ivf-" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(dir); + try + { + var plain = Graph(AnnMode.None); + var indexed = Graph(AnnMode.Ivf); + indexed.EmbeddingStore.BuildAnnIndex(); // an index definitely exists + Assert.True(indexed.EmbeddingStore.GetAnnStats().Active); + indexed.EmbeddingStore.SimilaritySearchVector(new float[32].Select((_, i) => (float)(i + 1)).ToArray(), 5); + + string pa = System.IO.Path.Combine(dir, "plain.ison"), pb = System.IO.Path.Combine(dir, "indexed.ison"); + plain.Save(pa); + indexed.Save(pb); + + Assert.Equal(File.ReadAllBytes(pa), File.ReadAllBytes(pb)); + var sa = File.ReadAllText(SemanticGraph.EmbeddingsPathFor(pa)); + var sb = File.ReadAllText(SemanticGraph.EmbeddingsPathFor(pb)); + Assert.Equal(sa, sb); + Assert.DoesNotContain("ivf", sb, StringComparison.OrdinalIgnoreCase); + Assert.DoesNotContain("centroid", sb, StringComparison.OrdinalIgnoreCase); + Assert.Equal( + plain.EmbeddingStore.ExportEmbeddings("d").GraphDigest, + indexed.EmbeddingStore.ExportEmbeddings("d").GraphDigest); + + // Loading builds nothing; the index appears lazily on the first search. + var loaded = SemanticGraph.Load( + pb, new MockEncoder(32), ann: AnnMode.Ivf, annMinSize: 0, annNlist: 32, annNprobe: 4); + Assert.Equal(AnnMode.Ivf, loaded.EmbeddingStore.Ann); + Assert.False(loaded.EmbeddingStore.GetAnnStats().Active); + Assert.Equal(5, loaded.EmbeddingStore.SimilaritySearchVector( + new float[32].Select((_, i) => (float)(i + 1)).ToArray(), 5).Count); + Assert.True(loaded.EmbeddingStore.GetAnnStats().Active); + } + finally { Directory.Delete(dir, recursive: true); } + } + + // -- through the graph -------------------------------------------------------------- + + [Fact] + public void Graph_passes_the_options_through_and_exact_forces_the_scan() + { + var graph = Graph(AnnMode.Ivf); + var store = graph.EmbeddingStore; + Assert.Equal(AnnMode.Ivf, store.Ann); + Assert.Equal(AnnMode.None, new SemanticGraph().EmbeddingStore.Ann); + Assert.Equal(4, store.AnnNprobe); + + var res = graph.SimilaritySearch("anything", 5); + Assert.Equal(5, res.Count); + Assert.True(store.GetAnnStats().Active); + + long before = store.GetAnnStats().Searches.Approximate; + graph.SimilaritySearch("anything", 5, exact: true); + graph.SimilaritySearch("anything", 5, exact: true, nodeFilter: _ => true); + Assert.Equal(before, store.GetAnnStats().Searches.Approximate); + + var near = graph.SimilarToNode(new NodeRef("item", "3"), topK: 5); + Assert.Equal(5, near.Count); + Assert.DoesNotContain(near, r => r.NodeRef == new NodeRef("item", "3")); + Assert.True(graph.RecallCheck(new[] { "a", "b", "c" }).Queries == 3); + } + + [Fact] + public void NodeFilter_still_returns_k_results_when_selective() + { + var graph = Graph(AnnMode.Ivf); + long Group(Node n) => (long)n.Properties["group"]!; + + // 1 node in 50 passes: the probed cells hold few of them, so the search widens. + var res = graph.SimilaritySearch("q", 10, nodeFilter: n => Group(n) == 7); + Assert.Equal(10, res.Count); + Assert.All(res, r => Assert.Equal(7, int.Parse(r.NodeRef.Id) % 50)); + Assert.Equal(res.Select(r => r.Score).OrderByDescending(s => s), res.Select(r => r.Score)); + var want = graph.SimilaritySearch("q", 10, nodeFilter: n => Group(n) == 7, exact: true); + Assert.True(res.Select(r => r.NodeRef).Intersect(want.Select(r => r.NodeRef)).Count() >= 5); + + Assert.Empty(graph.SimilaritySearch("q", 10, nodeFilter: _ => false)); + var even = graph.SimilaritySearch("q", 10, nodeFilter: n => Group(n) % 2 == 0); + var evenExact = graph.SimilaritySearch("q", 10, nodeFilter: n => Group(n) % 2 == 0, exact: true); + Assert.True(even.Select(r => r.NodeRef).Intersect(evenExact.Select(r => r.NodeRef)).Count() >= 6); + } + + [Fact] + public void MultiHop_uses_the_index_for_seeds_but_blend_does_not() + { + var graph = Graph(AnnMode.Ivf, n: 1200); + for (int i = 0; i < 1199; i++) + graph.AddEdge("NEXT", new NodeRef("item", i.ToString()), new NodeRef("item", (i + 1).ToString())); + var store = graph.EmbeddingStore; + var found = graph.SemanticMultiHop("q", new MultiHopOptions + { RelType = "NEXT", MaxHops = 2, TopKSeeds = 3, Threshold = -1f }); + Assert.NotEmpty(found); + Assert.True(store.GetAnnStats().Searches.Approximate >= 1); + + long before = store.GetAnnStats().Searches.Approximate; + graph.SemanticMultiHop("q", new MultiHopOptions + { RelType = "NEXT", MaxHops = 1, Blend = 0.3f, Threshold = -1f }); + Assert.Equal(before, store.GetAnnStats().Searches.Approximate); + } + + // -- concurrency -------------------------------------------------------------------- + + [Fact] + public void Concurrent_searches_and_writes_stay_consistent() + { + var (store, data, gen) = Build(n: 3000, nlist: 32, nprobe: 4); + var extra = gen.Draw(600); + var queries = gen.Draw(40); + var errors = new System.Collections.Concurrent.ConcurrentBag<string>(); + + void Reader(int seed) + { + try + { + for (int i = 0; i < 120; i++) + { + var res = store.SimilaritySearchVector(queries[(i + seed) % 40], 8, threshold: -1f); + if (res.Count != 8) errors.Add($"short result {res.Count}"); + if (!res.Select(r => r.Score).SequenceEqual(res.Select(r => r.Score).OrderByDescending(s => s))) + errors.Add("unsorted"); + if (res.Select(r => r.NodeRef).Distinct().Count() != res.Count) errors.Add("duplicate"); + } + } + catch (Exception ex) { errors.Add(ex.ToString()); } + } + + void Writer(int seed) + { + try + { + for (int i = 0; i < 100; i++) + { + store.Add(new NodeRef("w", (seed * 1000 + i).ToString()), "", extra[(seed * 100 + i) % 600]); + if (i % 4 == 0) store.Remove(Ref(seed * 300 + i)); + if (i % 20 == 0) store.BuildAnnIndex(); + } + } + catch (Exception ex) { errors.Add(ex.ToString()); } + } + + var threads = Enumerable.Range(0, 3).Select(t => new Thread(() => Reader(t))) + .Concat(Enumerable.Range(0, 2).Select(t => new Thread(() => Writer(t)))).ToList(); + threads.ForEach(t => t.Start()); + threads.ForEach(t => t.Join()); + Assert.Empty(errors); + Assert.True(store.RecallCheck(queries.Take(20)).Recall >= RecallFloor); + } + + // -- the vector kernel against awkward dimensions ------------------------------------ + + [Theory] + [InlineData(3)] + [InlineData(10)] + [InlineData(33)] + public void Odd_dimensions_work(int dim) + { + var (store, data, gen) = Build(n: 1200, dim: dim, nlist: 16, nprobe: 16); // all cells: must equal brute force + var refs = Refs(data.Length); + foreach (var q in gen.Draw(5)) + { + var got = store.SimilaritySearchVector(q, 10); + var want = BruteForce(data, refs, q, 10); + Assert.Equal(want.Select(w => w.Ref), got.Select(g => g.NodeRef)); + } + store.AnnNprobe = 4; + var report = store.RecallCheck(gen.Draw(30)); + Assert.True(report.Approximate); + Assert.True(report.Recall >= RecallFloor); + } +} diff --git a/isongraph-vector-js/README.md b/isongraph-vector-js/README.md index 109f85e..c71663d 100644 --- a/isongraph-vector-js/README.md +++ b/isongraph-vector-js/README.md @@ -261,9 +261,63 @@ an integral float (`1.0`) or an integer beyond 2^53 may digest differently here. When a mismatch is found and the file holds such literals, `loadGraph` warns and uses the vectors instead of throwing. +## Approximate search (opt-in) + +Search is exact by default. For larger stores, `ann: 'ivf'` answers bounded top-k searches from an in-memory IVF-flat index: +the vectors are clustered into cells with seeded spherical k-means, and a query +scores only the `annNprobe` cells nearest to it, with the same cosine formula +as the exact scan. + +```javascript +import { SemanticGraph, EmbeddingStore } from 'isongraph-vector-js'; + +const graph = new SemanticGraph({ encoder, ann: 'ivf' }); // also on EmbeddingStore +graph.similaritySearch(query, 10); // approximate once the store is large enough +graph.similaritySearch(query, 10, undefined, 0, undefined, undefined, true); // exact: true forces the full scan +graph.recallCheck(sampleQueries, 10); // recall@10 against the exact scan, with timings +graph.embeddingStore.annStats(); // cells, pending rows, memory, counters +``` + +Options (also `EmbeddingStore`'s second argument): `ann`, `annNlist` (default +about `4 * sqrt(N)`, between 16 and 65536), `annNprobe` (default 5% of the +cells; settable later as `store.annNprobe`), `annMinSize` (default 50000: below +it search stays exact) and `annSeed` (the same data and seed give the same +index). `annNprobe >= annNlist` is the exact scan. + +- **It is approximate.** A true neighbour in an unprobed cell is missed. Run + `recallCheck()` on queries from your own workload and raise `annNprobe` until + the recall holds; the figure depends on how your data clusters, and a number + measured on other data does not transfer. +- **Scores are exact.** The index chooses the candidates with float32 copies of + the vectors; the `topK` it returns are re-scored with the same float64 cosine + the exact scan uses, so a score never differs between the two paths. +- **Lazy, and never persisted.** The first search builds it (or call + `buildAnnIndex()` to choose the moment); a saved graph, its sidecar and the + graph digest are byte-for-byte what they would be without it. Reload a graph + with `loadGraph(path, { ann: 'ivf' })` and it is rebuilt on demand. +- **Kept current.** New and updated nodes are scored exactly until enough have + accumulated to be folded into cells; deletes never come back; the index + retrains by itself once churn passes half the trained size. Filters + (`nodeType`, `threshold`, `nodeFilter`) are applied to the probed cells, and + the probe widens (4x per step, ending in the exact scan) when too few + candidates survive, so a selective filter costs time, not results. +- **Costs memory.** A Float32Array copy of the vectors (4 bytes per component), + next to the stored ones, plus about 8 bytes per vector for the cell lists. +- **Faster to query, slow to build.** Without BLAS the exact scan of 384-d + vectors takes about 5 ms at 10k, 50 ms at 100k and 0.5 s at 1M; the index + answers in 0.2-3 ms at 10k-100k (recall@10 1.00 at the default probe) and + in 1.4-27 ms at 1M (recall 0.75 at `annNprobe` 1, 0.94 at the default 200), + on synthetic clustered data. Building it runs k-means in plain JavaScript + and takes about 8 s at 10k, 55 s at 100k and 8 min at 1M, so call + `buildAnnIndex()` at startup instead of letting the first search pay for it. + It is not worth it much below 10k vectors. `isongraph-vector-js/bench/ann.mjs` + in the repository reproduces the numbers (`npm run bench:ann`). +- The index is for the in-memory `EmbeddingStore` (and so for `SemanticGraph`); + `SqliteEmbeddingStore` has no `ann` option. + ## Embeddings from an HTTP endpoint -`HttpEncoder` is an `EmbeddingEncoder` that calls an OpenAI-compatible +`HttpEncoder` is an `AsyncEncoder` (and an `EmbeddingEncoder`) that calls an OpenAI-compatible `/v1/embeddings` endpoint or Ollama's `/api/embed`. It uses the platform `fetch` - no dependency, no `node:` import, so it lives in the main entry and runs on Node 18+ and in a browser. @@ -273,7 +327,7 @@ search queries alike. Point it at a service you are willing to send that data to (a local Ollama keeps it on your machine). ```javascript -import { HttpEncoder, EmbeddingStore } from 'isongraph-vector-js'; +import { HttpEncoder, SemanticGraph, EmbeddingStore } from 'isongraph-vector-js'; const encoder = new HttpEncoder({ url: 'https://api.openai.com/v1/embeddings', // or http://localhost:11434/api/embed @@ -282,34 +336,58 @@ const encoder = new HttpEncoder({ apiKey: process.env.OPENAI_API_KEY, // sent as a Bearer token }); -const texts = ['Alice is an engineer', 'Bob plans roadmaps']; -await encoder.prefetch([...texts, 'engineering']); // the network calls happen here +// A graph embeds its nodes over the network with the Async methods - no prefetch. +const graph = new SemanticGraph({ encoder }); +await graph.addNodeAsync('person', 1, { name: 'Alice is an engineer' }); +await graph.addNodeAsync('person', 2, { name: 'Bob plans roadmaps' }); +const hits = await graph.similaritySearchAsync('engineering', 1); +// The bare store has them too. const store = new EmbeddingStore(encoder); -store.addBatch([[['person', 1], texts[0]], [['person', 2], texts[1]]]); -store.similaritySearch('engineering', 1); // served from the prefetched vectors +await store.addBatchAsync([[['person', 1], 'Alice is an engineer'], [['person', 2], 'Bob plans roadmaps']]); +await store.similaritySearchAsync('engineering', 1); ``` Options: `url`, `model` (also the `modelName` recorded on every vector), `api` (`'openai'` default, or `'ollama'`), `apiKey`, `dimension` (otherwise learned from the first response; `NaN` until then), `batchSize` (64), `timeoutMs` (30000), `headers`, `version`, `maxRetries` (2), `retryDelayMs` -(500), `maxCacheEntries` (100000) and `fetch` (a replacement for the global). +(500), `maxCacheEntries` (100000), `concurrency` (1) and `fetch` (a replacement for +the global). 429 and 5xx answers, timeouts and network errors are retried a bounded number of times with backoff (a `Retry-After` header is honoured, up to 30 s); other errors throw an `HttpEncoderError` carrying the HTTP `status`. Errors name the endpoint's origin and path, never the API key or the query string. -**It is not synchronous, and `EmbeddingEncoder` is.** The stores call -`encode` / `encodeBatch` and expect a vector back, so `HttpEncoder` serves those -from a cache and throws an `HttpEncoderError` for a text it has not fetched. -Fetch first with `await encoder.prefetch(texts)` (or `encodeAsync` / -`encodeBatchAsync`, which also return the vectors). In practice that means -`SemanticGraph` auto-embedding does not work with it - the graph builds the -text itself and asks for the vector at once - so use it with the stores and -`autoEmbed: false`, prefetching the texts you pass, or embed with -`encodeBatchAsync` and hand the vectors in. A bounded cache (`maxCacheEntries`) -evicts the oldest vectors, so prefetch no more than you are about to use. +**It is asynchronous, and the Async methods are how you use it.** +`HttpEncoder` implements `AsyncEncoder` (`encodeAsync` / `encodeBatchAsync`), +and the store and graph have an `...Async` method wherever the encoder is +called: `addAsync`, `addBatchAsync`, `similaritySearchAsync` on the store, and +on the graph `addNodeAsync`, `addNodeWithEmbedAsync`, `updateNodeAsync`, +`embedNodeAsync`, `embedNodesAsync`, `embedAllNodesAsync`, `reembedStaleAsync`, +`similaritySearchAsync`, `semanticMultiHopAsync`, `semanticPathAsync`, +`semanticSubgraphAsync` and `recallCheckAsync`. Each awaits the network and +then does exactly what its synchronous namesake does, with the same model +metadata (`modelName` on every vector, `staleEmbeddings()`, `onModelMismatch`) +and the same key redaction. They also work with any synchronous encoder, and +`isAsyncEncoder(x)` tells the two kinds apart. Your own encoder can be +async-only: implement `AsyncEncoder` and pass it where an encoder goes. + +Failures are atomic. `addNodeAsync` embeds the text *before* it adds the node, +`updateNodeAsync` before it changes the properties, and `reembedStaleAsync` +encodes every stale text before it writes one, so a timeout or a 5xx leaves +the graph exactly as it was and the call can be retried. + +For bulk work, `batchSize` texts go in each request, `concurrency` (default 1: +one after another) requests run at once, and texts asked for at the same +moment share a single request. + +The synchronous interface is unchanged: `encode` / `encodeBatch` serve from a +cache and throw an `HttpEncoderError` for a text that has not been fetched, so +the prefetch workflow (`await encoder.prefetch(texts)`, then `addNode`, +`addBatch`, `similaritySearch`) still works. Every `...Async` call fills that +cache too. A bounded cache (`maxCacheEntries`) evicts the oldest vectors, so +prefetch no more than you are about to use. ## One format, six languages @@ -327,18 +405,19 @@ store.importEmbeddings(payload); ## Things that will bite you -- **Search is brute force**, sqlite-vec included: linear in the number of - nodes. This is built for an application's own knowledge graph, not as a - vector-database replacement. +- **Search is brute force** unless you opt into `ann: 'ivf'` (sqlite-vec is + exact too): linear in the number of nodes. This is built for an application's + own knowledge graph, not as a vector-database replacement. - **One embedding per node**, from one text field or a concatenation. No multi-vector nodes, no edge embeddings. - **`MockEncoder` has no semantic structure.** Use it to test plumbing, never - to judge relevance quality. For real vectors use `HttpEncoder` (below, - prefetch-then-search) or implement `EmbeddingEncoder` over the model of your - choice. + to judge relevance quality. For real vectors use `HttpEncoder` (below, with + the `...Async` methods) or implement `EmbeddingEncoder` / `AsyncEncoder` over + the model of your choice. - **`HttpEncoder` sends your texts to the endpoint you configure**, and its - `encode` only serves texts fetched beforehand - see *Embeddings from an HTTP - endpoint*. + synchronous `encode` only serves texts fetched beforehand - auto-embedding a + graph over the network is `addNodeAsync`, not `addNode`. See *Embeddings from + an HTTP endpoint*. - **Auto-embed renders typed properties.** Since ISONGraph 1.4.0 a property can be a number, a bool or null. Nulls are skipped and booleans render as `true` and `false`, so the same graph produces the same text in every port. @@ -349,8 +428,8 @@ store.importEmbeddings(payload); ## Tests ```bash -npm test # 193 tests, needs Node 24 for the SQLite half -npm run test:core # 68 tests, main entry only, Node 18 and up +npm test # 254 tests, needs Node 24 for the SQLite half +npm run test:core # 129 tests, main entry only, Node 18 and up ``` ## Links diff --git a/isongraph-vector-js/bench/ann.mjs b/isongraph-vector-js/bench/ann.mjs new file mode 100644 index 0000000..95e7355 --- /dev/null +++ b/isongraph-vector-js/bench/ann.mjs @@ -0,0 +1,191 @@ +#!/usr/bin/env node +/** + * Exact vs IVF approximate search: latency and recall@k on synthetic clustered + * vectors (default 384-d, 10k / 100k / 1M). + * + * node --max-old-space-size=16384 bench/ann.mjs # all three sizes + * node bench/ann.mjs --sizes 10000,100000 # smaller, default heap + * node bench/ann.mjs --sizes 100000 --queries 200 --nprobe 1,8,32 # custom + * + * The vectors are CLUSTERED (n/100 Gaussian clusters, 100 points each, unit + * length), the same regime as benchmarks/ann.py "clustered": an IVF index's + * recall depends entirely on how data clusters, and uniform random vectors + * (where it degenerates to a coin flip) would say nothing. Real embeddings sit + * somewhere else - run store.recallCheck() on your own queries before relying + * on a setting. + * + * Stored vectors are plain JS arrays of doubles (8 bytes per component) plus + * the index's Float32Array copy (4 bytes), so 1M x 384 needs about 12 GB of + * heap: run it with --max-old-space-size. A size that will not fit is skipped + * with a message instead of crashing. + * + * To measure the TypeScript build instead, pass --port ts (needs `npm run + * build` in isongraph-vector-ts first); --entry <file> imports any other build. + */ +import { getHeapStatistics } from 'node:v8'; +import { cpus, totalmem } from 'node:os'; +import { pathToFileURL } from 'node:url'; +import { resolve, dirname } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const here = dirname(fileURLToPath(import.meta.url)); + +function args() { + const out = { + sizes: [10000, 100000, 1000000], dim: 384, queries: 100, k: 10, nprobe: null, port: 'js', seed: 4242, + }; + const argv = process.argv.slice(2); + for (let i = 0; i < argv.length; i++) { + const [flag, inline] = argv[i].split('='); + const value = () => inline ?? argv[++i]; + if (flag === '--sizes') out.sizes = value().split(',').map(Number); + else if (flag === '--dim') out.dim = Number(value()); + else if (flag === '--queries') out.queries = Number(value()); + else if (flag === '--k') out.k = Number(value()); + else if (flag === '--nprobe') out.nprobe = value().split(',').map(Number); + else if (flag === '--port') out.port = value(); + else if (flag === '--entry') out.entry = value(); + else if (flag === '--seed') out.seed = Number(value()); + else if (flag === '--help' || flag === '-h') { + console.log('usage: node bench/ann.mjs [--sizes 10000,100000,1000000] [--dim 384] [--queries 100] [--k 10] [--nprobe 1,8,32] [--port js|ts] [--entry file]'); + process.exit(0); + } else throw new Error(`unknown argument ${argv[i]}`); + } + return out; +} + +const opts = args(); +const entry = opts.entry ? resolve(opts.entry) : opts.port === 'ts' + ? resolve(here, '../../isongraph-vector-ts/dist/index.js') + : resolve(here, '../src/index.js'); +const { EmbeddingStore } = await import(pathToFileURL(entry).href); + +// -- data ---------------------------------------------------------------------- + +function rng(seed) { + let a = seed >>> 0; + return () => { + a = (a + 0x6d2b79f5) >>> 0; + let t = a; + t = Math.imul(t ^ (t >>> 15), t | 1); + t ^= t + Math.imul(t ^ (t >>> 7), t | 61); + return ((t ^ (t >>> 14)) >>> 0) / 4294967296; + }; +} + +class Clustered { + constructor(n, dim, seed) { + this.dim = dim; + this.next = rng(seed ^ n); + const clusters = Math.max(8, Math.floor(n / 100)); + this.centers = Array.from({ length: clusters }, () => { + const v = Float64Array.from({ length: dim }, () => this.gauss()); + const norm = Math.hypot(...v); + return v.map(x => x / norm); + }); + } + + gauss() { + return Math.sqrt(-2 * Math.log(1 - this.next())) * Math.cos(2 * Math.PI * this.next()); + } + + /** One unit vector near a random cluster centre (noise norm 1 across the vector). */ + draw() { + const { dim } = this; + const center = this.centers[Math.floor(this.next() * this.centers.length)]; + const v = new Array(dim); + let sum = 0; + for (let j = 0; j < dim; j++) { + v[j] = center[j] + this.gauss() / Math.sqrt(dim); + sum += v[j] * v[j]; + } + const norm = Math.sqrt(sum); + for (let j = 0; j < dim; j++) v[j] /= norm; + return v; + } +} + +// -- measuring ------------------------------------------------------------------- + +const median = values => { + const s = [...values].sort((a, b) => a - b); + return s[Math.floor(s.length / 2)]; +}; +const percentile = (values, q) => { + const s = [...values].sort((a, b) => a - b); + return s[Math.min(s.length - 1, Math.round(q * (s.length - 1)))]; +}; +const fmt = ms => (ms >= 100 ? ms.toFixed(0) : ms >= 10 ? ms.toFixed(1) : ms.toFixed(2)); +const keyOf = r => `${r.nodeRef[0]}:${r.nodeRef[1]}`; + +function timed(store, queries, k, exact) { + for (const q of queries.slice(0, 3)) store.similaritySearch(q, k, undefined, 0, exact); // warm up + const times = []; + const found = []; + for (const q of queries) { + const t0 = performance.now(); + const res = store.similaritySearch(q, k, undefined, 0, exact); + times.push(performance.now() - t0); + found.push(new Set(res.map(keyOf))); + } + return { times, found }; +} + +function recall(found, truth) { + const per = found.map((f, i) => { + let hit = 0; + for (const key of truth[i]) if (f.has(key)) hit++; + return truth[i].size ? hit / truth[i].size : 1; + }); + return { mean: per.reduce((a, b) => a + b, 0) / per.length, min: Math.min(...per) }; +} + +// -- run ------------------------------------------------------------------------- + +const heapLimit = getHeapStatistics().heap_size_limit; +console.log(`# exact vs IVF (${opts.port}), dim ${opts.dim}, k ${opts.k}, ${opts.queries} queries, warm single-thread`); +console.log(`node ${process.version}, ${cpus().length} logical CPUs, ${(totalmem() / 2 ** 30).toFixed(1)} GB RAM, heap limit ${(heapLimit / 2 ** 30).toFixed(1)} GB\n`); + +for (const n of opts.sizes) { + const need = n * opts.dim * 13 * 1.2; // doubles + float32 copy + slack + if (need > heapLimit * 0.9) { + console.log(`## N = ${n.toLocaleString('en-US')}: skipped (needs about ${(need / 2 ** 30).toFixed(1)} GB of heap, limit ${(heapLimit / 2 ** 30).toFixed(1)} GB; rerun with --max-old-space-size)\n`); + continue; + } + console.log(`## N = ${n.toLocaleString('en-US')}`); + const data = new Clustered(n, opts.dim, opts.seed); + const store = new EmbeddingStore(undefined, { ann: 'ivf', annMinSize: 0, annSeed: 0 }); + + let t0 = performance.now(); + const CHUNK = 5000; + for (let start = 0; start < n; start += CHUNK) { + const entries = []; + for (let i = start; i < Math.min(n, start + CHUNK); i++) entries.push([['doc', i], '', data.draw()]); + store.addBatch(entries); + } + console.log(`ingest ${(((n) / (performance.now() - t0)) * 1000).toFixed(0)} vectors/s`); + + const queries = Array.from({ length: opts.queries }, () => data.draw()); + const exact = timed(store, queries, opts.k, true); + console.log(`exact: median ${fmt(median(exact.times))} ms, p95 ${fmt(percentile(exact.times, 0.95))} ms`); + + t0 = performance.now(); + const stats = store.buildAnnIndex(); + const buildSeconds = (performance.now() - t0) / 1000; + console.log(`index build ${buildSeconds.toFixed(1)} s: nlist ${stats.nlist}, automatic nprobe ${stats.nprobe}, ` + + `index ${(stats.indexBytes / 2 ** 20).toFixed(1)} MB + float32 copy ${(stats.cacheBytes / 2 ** 20).toFixed(0)} MB\n`); + + const sweep = opts.nprobe ?? [...new Set([1, 4, 16, 64, stats.nprobe])].sort((a, b) => a - b); + console.log('| nprobe | % of cells | recall@k mean | worst | median ms | p95 ms | speed-up |'); + console.log('|---|---|---|---|---|---|---|'); + const exactMedian = median(exact.times); + for (const nprobe of sweep) { + store.annNprobe = nprobe; + const run = timed(store, queries, opts.k, false); + const r = recall(run.found, exact.found); + const label = nprobe === stats.nprobe ? `${nprobe} (auto)` : `${nprobe}`; + console.log(`| ${label} | ${(100 * nprobe / stats.nlist).toFixed(1)} | ${r.mean.toFixed(3)} | ${r.min.toFixed(2)} | ` + + `${fmt(median(run.times))} | ${fmt(percentile(run.times, 0.95))} | ${(exactMedian / median(run.times)).toFixed(1)}x |`); + } + console.log(''); +} diff --git a/isongraph-vector-js/bench/competitive.mjs b/isongraph-vector-js/bench/competitive.mjs new file mode 100644 index 0000000..0b7c233 --- /dev/null +++ b/isongraph-vector-js/bench/competitive.mjs @@ -0,0 +1,193 @@ +#!/usr/bin/env node +/** + * Loader and measurement harness for benchmarks/competitive/ (the cross-library comparison). + * Reads the same raw float32 vectors, queries and float64 ground truth the other harnesses feed their + * systems. Bench-only: the library is not touched. + * + * node --max-old-space-size=24000 bench/competitive.mjs --ds-dir <prep>/text384 --ds text384 --n 100000 \ + * --exp core,mutation --queries 2000 --label js --out result.json + * ... --exp filter + * (No save/load experiment: this port has no vector-persistence call besides exportEmbeddings.) + */ +import { openSync, readSync, closeSync, readFileSync, writeFileSync, existsSync } from 'node:fs'; +import { cpus } from 'node:os'; +import { pathToFileURL, fileURLToPath } from 'node:url'; +import { resolve, dirname, join } from 'node:path'; + +const here = dirname(fileURLToPath(import.meta.url)); +const argv = process.argv.slice(2); +const arg = (name, def) => { const i = argv.indexOf(name); return i >= 0 ? argv[i + 1] : def; }; +const dir = arg('--ds-dir', ''), ds = arg('--ds', ''), label = arg('--label', 'js'); +const n = Number(arg('--n', '100000')), nq = Number(arg('--queries', '1000')), budget = Number(arg('--budget', '15')); +const exps = new Set(arg('--exp', 'core').split(',')); +const outPath = arg('--out', ''); +const entry = resolve(here, '../src/index.js'); +const { EmbeddingStore } = await import(pathToFileURL(entry).href); + +const K = 10; +const meta = JSON.parse(readFileSync(join(dir, 'meta.json'), 'utf8')); +const d = meta.d; + +function readF32(path, start, count) { + const fd = openSync(path, 'r'); + const buf = Buffer.alloc(count * d * 4); + readSync(fd, buf, 0, buf.length, start * d * 4); + closeSync(fd); + const f = new Float32Array(buf.buffer, buf.byteOffset, count * d); + const rows = new Array(count); + for (let i = 0; i < count; i++) rows[i] = Array.from(f.subarray(i * d, (i + 1) * d)); + return rows; +} +const readI32 = path => { const b = readFileSync(path); return new Int32Array(b.buffer, b.byteOffset, b.length / 4); }; +const base = (start, count) => readF32(join(dir, 'base.f32'), start, count); +const gt = kind => { const f = readI32(join(dir, `gt_${kind}${n}.i32`)); const out = []; for (let i = 0; i < f.length / 100; i++) out.push(Array.from(f.subarray(i * 100, i * 100 + K))); return out; }; +const queries = readF32(join(dir, 'queries.f32'), 0, nq); + +const now = () => Number(process.hrtime.bigint()) / 1e9; +const pct = (xs, p) => { const s = [...xs].sort((a, b) => a - b); return s.length ? s[Math.round((s.length - 1) * p)] : 0; }; +const mem = () => { const m = process.memoryUsage(); return { rss_mb: m.rss / 1048576, peak_mb: m.rss / 1048576, heap_mb: m.heapUsed / 1048576 }; }; +let peakRss = 0; +const memP = () => { const m = mem(); peakRss = Math.max(peakRss, m.rss_mb); return { ...m, peak_mb: peakRss }; }; +const idOf = r => Number(r.nodeRef[1]); +const search = (s, q, exact, ty) => s.similaritySearch(q, K, ty, 0.0, exact).map(idOf); +const recallOf = (found, truth) => { let hit = 0; for (let i = 0; i < found.length; i++) { const want = new Set(truth[i]); for (const x of found[i].slice(0, K)) if (want.has(x)) hit++; } return hit / (K * Math.max(1, found.length)); }; + +function point(s, truth, exact, ty) { + let w0 = now(); + for (let i = 0; i < Math.min(50, nq); i++) { search(s, queries[i], exact, ty); if (now() - w0 > 3) break; } + const probe = []; + for (let i = 0; i < Math.min(5, nq); i++) { const t = now(); search(s, queries[i], exact, ty); probe.push(now() - t); } + const est = Math.max(pct(probe, 0.5), 1e-7); + const m = Math.floor(Math.min(nq, Math.max(30, budget / est))); + const runs = [], lats = []; + let rec = 0; + for (let r = 0; r < 3; r++) { + const lat = [], res = []; + for (let i = 0; i < m; i++) { const t = now(); const ids = search(s, queries[i], exact, ty); lat.push(now() - t); if (r === 0) res.push(ids); } + if (r === 0) rec = recallOf(res, truth.slice(0, m)); + const total = lat.reduce((a, b) => a + b, 0); + runs.push(m / total); lats.push(...lat); + if (total > budget * 1.5) break; + } + return { nq: m, recall: rec, qps_1t: pct(runs, 0.5), qps_1t_min: Math.min(...runs), qps_1t_max: Math.max(...runs), qps_1t_runs: runs, + median_ms: pct(lats, 0.5) * 1000, p95_ms: pct(lats, 0.95) * 1000, p99_ms: pct(lats, 0.99) * 1000, mean_ms: lats.reduce((a, b) => a + b, 0) / lats.length * 1000 }; +} +const nlistFor = x => Math.min(Math.max(Math.round(4 * Math.sqrt(x)), 16), 65536); +function knobs() { const nl = nlistFor(n); const v = [1, 2, 4, 8, 16, 32, 64, 128, 256].filter(x => x < nl); if (n <= 200000) v.push(nl); return v; } +function ingest(s, grp) { + let secs = 0; + const names = ['t10', 't01', 'rest']; + for (let at = 0; at < n; at += 20000) { + const m = Math.min(20000, n - at); + const rows = base(at, m); + const t = now(); + const entries = rows.map((v, i) => [[grp ? names[grp[at + i]] : 'doc', at + i], '', v]); + s.addBatch(entries); + secs += now() - t; + } + return secs; +} +const header = sys => ({ system: sys, ds, n, d, k: K, queries: nq, all_threads: cpus().length, version: 'isongraph-vector-js (repo HEAD, Node ' + process.version + ')' }); + +const runs = []; +const memBase = memP(); +console.log(`${label}: ds=${ds} n=${n} d=${d} queries=${nq}`); +if (exps.has('core') || exps.has('mutation')) { + let s = new EmbeddingStore(undefined, {}); + let ingestS = ingest(s, null); + let memIng = memP(); + console.log(` ingest ${ingestS.toFixed(2)}s`); + const truth = gt(''); + let t0 = now(); + search(s, queries[0], true); + const firstMs = (now() - t0) * 1000; + const ex = header(`${label}_exact`); + ex.build = { ingest_s: ingestS, index_s: 0, total_s: ingestS, first_query_ms: firstMs, ready_s: ingestS + firstMs / 1000 }; + ex.mem_baseline = memBase; ex.mem_after_ingest = memIng; ex.mem_after_build = memP(); ex.mem_after_first_query = memP(); + const p1 = point(s, truth, true); p1.knob = null; + console.log(` exact recall=${p1.recall} qps=${p1.qps_1t.toFixed(1)}`); + ex.sweep = [p1]; ex.mem_after_sweep = memP(); + runs.push(ex); + s = null; + if (global.gc) global.gc(); + + const st = new EmbeddingStore(undefined, { ann: 'ivf', annMinSize: 0, annSeed: 0 }); + ingestS = ingest(st, null); + memIng = memP(); + t0 = now(); + const stats = st.buildAnnIndex(); + const idxS = now() - t0; + console.log(` ivf build ${idxS.toFixed(2)}s nlist=${stats.nlist}`); + const iv = header(`${label}_ivf`); + iv.knob_name = 'nprobe'; + iv.build = { ingest_s: ingestS, index_s: idxS, total_s: ingestS + idxS, first_query_ms: 0, ready_s: ingestS + idxS }; + iv.mem_baseline = memBase; iv.mem_after_ingest = memIng; iv.mem_after_build = memP(); + iv.index_stats = { nlist: stats.nlist, trained_n: stats.trainedN, index_bytes: stats.indexBytes, build_seconds: stats.buildSeconds }; + const pts = []; + let refKnob = null, refRecall = 0; + for (const kn of knobs()) { + st.annNprobe = kn; + const p = point(st, truth, false); p.knob = kn; + console.log(` ivf nprobe=${kn} recall=${p.recall.toFixed(4)} qps1t=${p.qps_1t.toFixed(0)}`); + if (refKnob === null && p.recall >= 0.95) { refKnob = kn; refRecall = p.recall; } + pts.push(p); + } + if (refKnob === null) refKnob = knobs().slice(-1)[0]; + iv.sweep = pts; iv.ref_knob = refKnob; iv.ref_recall_pre = refRecall; iv.mem_after_sweep = memP(); + if (exps.has('mutation')) { + const gtm = join(dir, `gt_mut_${n}.i32`); + if (existsSync(gtm)) { + const upd = readI32(join(dir, `mut_${n}_upd.i32`)), del = readI32(join(dir, `mut_${n}_del.i32`)); + const f = readI32(gtm); const truthM = []; + for (let i = 0; i < Math.min(nq, f.length / 100); i++) truthM.push(Array.from(f.subarray(i * 100, i * 100 + K))); + st.annNprobe = refKnob; + const m = { ref_knob: refKnob, recall_before: refRecall }; + const ins = base(n, 10000); + let t = now(); + for (let i = 0; i < ins.length; i++) st.add(['doc', n + i], '', ins[i]); + let dt = now() - t; m.insert_10k = { seconds: dt, per_s: 10000 / dt }; + const nv = base(n + 10000, 10000); + t = now(); + for (let j = 0; j < nv.length; j++) st.add(['doc', upd[j]], '', nv[j]); + dt = now() - t; m.update_10k = { seconds: dt, per_s: 10000 / dt }; + t = now(); + for (const id of del) st.remove(['doc', id]); + dt = now() - t; m.delete_10k = { seconds: dt, per_s: 10000 / dt }; + t = now(); search(st, queries[0], false); m.first_query_after_ms = (now() - t) * 1000; + const p = point(st, truthM, false); + m.recall_after = p.recall; m.after = p; m.mem_after = memP(); + const s2 = st.annStats(); + m.index_stats_after = { pending: s2.pending, churn: s2.churn, trained_n: s2.trainedN }; + console.log(` mutation recall_after=${p.recall}`); + iv.mutation = m; + } else iv.mutation = { skipped: 'dataset has no spare rows for this n' }; + } + runs.push(iv); +} +if (exps.has('filter')) { + const g = readFileSync(join(dir, `grp_${n}.u8`)); + const st = new EmbeddingStore(undefined, { ann: 'ivf', annMinSize: 0, annSeed: 0 }); + const ingestS = ingest(st, g); + const t0 = now(); + st.buildAnnIndex(); + const idxS = now() - t0; + const res = header(`${label}_ivf`); + const subsets = {}; + res.filter = { build_with_metadata_s: ingestS + idxS, mem: memP(), subsets }; + const kn = knobs(); + const pick = [...new Set([1, Math.floor(kn.length / 3), Math.floor(kn.length / 2), Math.floor(2 * kn.length / 3), kn.length - 1])].sort((a, b) => a - b); + for (const [ty, lab] of [['t10', 'f10'], ['t01', 'f01']]) { + const truth = gt(lab + '_'); + const pts = []; + for (const i of pick) { + st.annNprobe = kn[i]; + const p = point(st, truth, false, ty); p.knob = kn[i]; + console.log(` filter ${lab} nprobe=${kn[i]} recall=${p.recall.toFixed(4)} qps=${p.qps_1t.toFixed(0)}`); + pts.push(p); + } + const pe = point(st, truth, true, ty); pe.knob = 'exact'; pts.push(pe); + subsets[lab] = pts; + } + runs.push(res); +} +if (outPath) { writeFileSync(outPath, JSON.stringify({ status: 'ok', label, runs }, null, 1)); console.log(`wrote ${outPath}`); } diff --git a/isongraph-vector-js/bench/results.md b/isongraph-vector-js/bench/results.md new file mode 100644 index 0000000..8c600c9 --- /dev/null +++ b/isongraph-vector-js/bench/results.md @@ -0,0 +1,46 @@ +# bench/ann.mjs output + +One run of `npm run bench:ann` (JavaScript build; the TypeScript build is the same code). Synthetic clustered unit vectors, 384-d, k=10, 100 queries, single thread, while other jobs shared the machine, so treat timings as indicative. Real embeddings cluster differently: run `recallCheck()` on your own queries. + +## Run: exact vs IVF (js), dim 384, k 10, 100 queries, warm single-thread +node v24.18.0, 32 logical CPUs, 47.7 GB RAM, heap limit 19.7 GB + +## N = 10,000 +ingest 66276 vectors/s +exact: median 5.10 ms, p95 5.96 ms +index build 7.9 s: nlist 400, automatic nprobe 20, index 0.7 MB + float32 copy 16 MB + +| nprobe | % of cells | recall@k mean | worst | median ms | p95 ms | speed-up | +|---|---|---|---|---|---|---| +| 1 | 0.3 | 0.581 | 0.10 | 0.21 | 0.25 | 24.2x | +| 4 | 1.0 | 0.984 | 0.80 | 0.23 | 0.27 | 22.3x | +| 16 | 4.0 | 1.000 | 1.00 | 0.33 | 0.43 | 15.6x | +| 20 (auto) | 5.0 | 1.000 | 1.00 | 0.32 | 0.39 | 16.0x | +| 64 | 16.0 | 1.000 | 1.00 | 0.72 | 0.89 | 7.0x | + +## N = 100,000 +ingest 58092 vectors/s +exact: median 54.5 ms, p95 60.8 ms +index build 53.7 s: nlist 1265, automatic nprobe 63, index 2.6 MB + float32 copy 157 MB + +| nprobe | % of cells | recall@k mean | worst | median ms | p95 ms | speed-up | +|---|---|---|---|---|---|---| +| 1 | 0.1 | 0.944 | 0.20 | 0.47 | 0.53 | 116.8x | +| 4 | 0.3 | 1.000 | 1.00 | 0.58 | 0.65 | 94.4x | +| 16 | 1.3 | 1.000 | 1.00 | 1.09 | 1.36 | 49.9x | +| 63 (auto) | 5.0 | 1.000 | 1.00 | 2.97 | 3.31 | 18.3x | +| 64 | 5.1 | 1.000 | 1.00 | 2.68 | 3.28 | 20.3x | + +## N = 1,000,000 +ingest 57685 vectors/s +exact: median 543 ms, p95 574 ms +index build 502.7 s: nlist 4000, automatic nprobe 200, index 13.7 MB + float32 copy 1570 MB + +| nprobe | % of cells | recall@k mean | worst | median ms | p95 ms | speed-up | +|---|---|---|---|---|---|---| +| 1 | 0.0 | 0.753 | 0.00 | 1.43 | 1.57 | 380.2x | +| 4 | 0.1 | 0.803 | 0.00 | 1.81 | 2.01 | 299.7x | +| 16 | 0.4 | 0.861 | 0.10 | 3.41 | 4.10 | 159.3x | +| 64 | 1.6 | 0.904 | 0.20 | 9.49 | 10.5 | 57.2x | +| 200 (auto) | 5.0 | 0.939 | 0.40 | 27.3 | 29.6 | 19.9x | + diff --git a/isongraph-vector-js/package.json b/isongraph-vector-js/package.json index 8c9470a..dbcdc87 100644 --- a/isongraph-vector-js/package.json +++ b/isongraph-vector-js/package.json @@ -18,7 +18,8 @@ ], "scripts": { "test": "vitest run", - "test:core": "vitest run src/index.test.js src/http-encoder.test.js" + "test:core": "vitest run src/index.test.js src/http-encoder.test.js src/async-encoder.test.js src/ann.test.js", + "bench:ann": "node --max-old-space-size=16384 bench/ann.mjs" }, "keywords": [ "ison", diff --git a/isongraph-vector-js/src/ann.test.js b/isongraph-vector-js/src/ann.test.js new file mode 100644 index 0000000..52b2408 --- /dev/null +++ b/isongraph-vector-js/src/ann.test.js @@ -0,0 +1,517 @@ +/** + * The opt-in IVF approximate index. Mirrors the Python port's + * TestApproximateIndex / TestApproximateIndexInGraph: recall on clustered + * data, exactness when every cell is probed, upkeep under writes and deletes, + * filters, and that nothing about the index reaches a saved file. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import { mkdtempSync, rmSync, readFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { + EmbeddingStore, SemanticGraph, MockEncoder, EmbeddingError, graphDigest, +} from './index.js'; +import { saveGraph, loadGraph, embeddingsPathFor } from './node.js'; + +/** Recall@10 demanded on the clustered data below (4000 vectors, 40 clusters, 64 cells, nprobe 8). */ +const RECALL_FLOOR = 0.95; + +/** Small seeded generator, so a failure reproduces. */ +function rng(seed) { + let a = seed >>> 0; + return () => { + a = (a + 0x6d2b79f5) >>> 0; + let t = a; + t = Math.imul(t ^ (t >>> 15), t | 1); + t ^= t + Math.imul(t ^ (t >>> 7), t | 61); + return ((t ^ (t >>> 14)) >>> 0) / 4294967296; + }; +} + +function gaussian(next) { + return Math.sqrt(-2 * Math.log(1 - next())) * Math.cos(2 * Math.PI * next()); +} + +/** Clustered, roughly unit vectors and a function drawing queries from the same clusters. */ +function clustered(n, dim = 32, clusters = 40, seed = 1, noise = 1.0) { + const next = rng(seed); + const centers = []; + for (let c = 0; c < clusters; c++) { + const v = Array.from({ length: dim }, () => gaussian(next)); + const norm = Math.hypot(...v); + centers.push(v.map(x => x / norm)); + } + const draw = (count) => + Array.from({ length: count }, () => { + const center = centers[Math.floor(next() * clusters)]; + return center.map(x => x + (noise * gaussian(next)) / Math.sqrt(dim)); + }); + return { data: draw(n), draw }; +} + +function annStore( + { n = 4000, dim = 32, seed = 1, types, ...options } = {} +) { + const { data, draw } = clustered(n, dim, 40, seed); + const store = new EmbeddingStore(undefined, { + ann: 'ivf', annMinSize: 0, annNlist: 64, annNprobe: 8, ...options, + }); + store.addBatch(data.map((v, i) => [[types ? types(i) : 'doc', i], `text ${i}`, v])); + return { store, data, draw }; +} + +const refs = results => results.map(r => `${r.nodeRef[0]}:${r.nodeRef[1]}`); + +describe('approximate index: store', () => { + it('is off by default and exact search is untouched', () => { + const { store, data, draw } = annStore({ ann: null }); + expect(store.ann).toBeNull(); + expect(store.annStats().enabled).toBe(false); + expect(() => store.buildAnnIndex()).toThrow(/ann/); + + // identical (not merely close) to the same store with ann on and exact: true + const { store: other } = annStore(); + for (const q of draw(5)) { + const a = store.similaritySearch(q, 10); + const b = other.similaritySearch(q, 10, undefined, 0, true); + expect(a.map(r => [r.nodeRef, r.score])).toEqual(b.map(r => [r.nodeRef, r.score])); + } + expect(data.length).toBe(4000); + }); + + it('validates its arguments', () => { + expect(() => new EmbeddingStore(undefined, { ann: 'hnsw' })).toThrow(EmbeddingError); + expect(() => new EmbeddingStore(undefined, { annNprobe: 0 })).toThrow(/annNprobe/); + expect(() => new EmbeddingStore(undefined, { ann: 'ivf', annNlist: 0 })).toThrow(/annNlist/); + expect(() => new EmbeddingStore(undefined, { ann: 'ivf', annMinSize: -1 })).toThrow(/annMinSize/); + expect(() => new EmbeddingStore(undefined, { ann: 'ivf', annNlist: 1.5 })).toThrow(/annNlist/); + const store = new EmbeddingStore(undefined, { ann: 'ivf' }); + expect(() => { store.annNprobe = 0; }).toThrow(EmbeddingError); + store.annNprobe = 5; + expect(store.annNprobe).toBe(5); + store.annNprobe = null; + expect(store.annNprobe).toBeNull(); + }); + + it('reaches the recall floor on clustered data and reports it', () => { + const { store, draw } = annStore(); + const report = store.recallCheck(draw(60), 10); + expect(report.approximate).toBe(true); + expect(report.nlist).toBe(64); + expect(report.nprobe).toBe(8); + expect(report.queries).toBe(60); + expect(report.k).toBe(10); + expect(report.recall).toBeGreaterThanOrEqual(RECALL_FLOOR); + expect(report.minRecall).toBeLessThanOrEqual(report.recall); + expect(report.minRecall).toBeGreaterThanOrEqual(0); + expect(report.annMs).toBeGreaterThanOrEqual(0); + expect(report.exactMs).toBeGreaterThanOrEqual(0); + const stats = store.annStats(); + expect(stats.active).toBe(true); + expect(stats.nlist).toBe(64); + expect(stats.trainedN).toBe(4000); + expect(stats.indexBytes).toBeGreaterThan(0); + expect(stats.searches.approximate).toBeGreaterThanOrEqual(60); + }); + + it('returns the exact cosine score of each node, best first', () => { + const { store, draw } = annStore(); + for (const q of draw(10)) { + const truth = new Map(store.similaritySearch(q, 100, undefined, 0, true).map(r => [`${r.nodeRef}`, r.score])); + const got = store.similaritySearch(q, 10); + for (const r of got) if (truth.has(`${r.nodeRef}`)) expect(r.score).toBe(truth.get(`${r.nodeRef}`)); + const scores = got.map(r => r.score); + expect(scores).toEqual([...scores].sort((a, b) => b - a)); + expect(got.every(r => typeof r.text === 'string')).toBe(true); + } + expect(store.annStats().searches.approximate).toBe(10); // the index answered, not the scan + }); + + it('probing every cell is the exact scan', () => { + const { store, data, draw } = annStore({ annNprobe: 64 }); // nprobe == nlist + const reference = new EmbeddingStore(); + reference.addBatch(data.map((v, i) => [['doc', i], `text ${i}`, v])); + for (const q of draw(8)) { + expect(store.similaritySearch(q, 10)).toEqual(reference.similaritySearch(q, 10)); + } + expect(store.recallCheck(draw(5)).approximate).toBe(false); + expect(store.annStats().searches.approximate).toBe(0); + }); + + it('stays exact under annMinSize, and the default minimum is 50000', () => { + const { store, draw } = annStore({ n: 500, annMinSize: 1000 }); + const q = draw(1)[0]; + expect(refs(store.similaritySearch(q, 10))).toEqual(refs(store.similaritySearch(q, 10, undefined, 0, true))); + expect(store.annStats().active).toBe(false); + expect(store.recallCheck([q]).approximate).toBe(false); + + const dflt = new EmbeddingStore(undefined, { ann: 'ivf' }); + expect(dflt.annStats().minSize).toBe(50000); + expect(dflt.annStats().mode).toBe('ivf'); + dflt.addBatch(clustered(2000).data.map((v, i) => [['doc', i], '', v])); + dflt.similaritySearch(clustered(1).data[0], 5); + expect(dflt.annStats().active).toBe(false); + }); + + it('chooses nlist and nprobe automatically', () => { + const { store, draw } = annStore({ n: 2500, annNlist: undefined, annNprobe: undefined }); + store.similaritySearch(draw(1)[0], 5); + const stats = store.annStats(); + expect(stats.nlist).toBe(200); // 4 * sqrt(2500) + expect(stats.nprobe).toBe(10); // 5% of the cells + expect(stats.active).toBe(true); + + const mid = annStore({ n: 100, annNlist: undefined, annNprobe: undefined }); + mid.store.similaritySearch(mid.draw(1)[0], 5); + expect(mid.store.annStats().nlist).toBe(40); // 4 * sqrt(100) + expect(mid.store.annStats().nprobe).toBe(2); + + const tiny = annStore({ n: 9, annNlist: undefined, annNprobe: undefined }); + tiny.store.similaritySearch(tiny.draw(1)[0], 5); + expect(tiny.store.annStats().nlist).toBe(9); // at least 16, but never more cells than vectors + expect(tiny.store.annStats().nprobe).toBe(1); + }); + + it('topK null and exact: true scan everything; topK 0 is empty', () => { + const { store, draw } = annStore(); + const q = draw(1)[0]; + const before = store.annStats().searches.approximate; + expect(store.similaritySearch(q, null, undefined, -1).length).toBe(4000); + store.similaritySearch(q, 5, undefined, 0, true); + expect(store.annStats().searches.approximate).toBe(before); + expect(store.similaritySearch(q, 0)).toEqual([]); + }); + + it('a zero query falls back to the exact scan', () => { + const { store } = annStore(); + const res = store.similaritySearch(new Array(32).fill(0), 5, undefined, 0); + expect(res.map(r => r.score)).toEqual([0, 0, 0, 0, 0]); + expect(store.annStats().searches).toMatchObject({ approximate: 0, exact: 1 }); + }); + + it('shows writes immediately, and an update moves the row', () => { + const { store, draw } = annStore(); + store.buildAnnIndex(); + const fresh = draw(1)[0]; + store.add(['new', 1], 'fresh', fresh); + const top = store.similaritySearch(fresh, 3); + expect(top[0].nodeRef).toEqual(['new', 1]); + expect(top[0].score).toBeCloseTo(1, 6); + expect(store.annStats().pending).toBe(1); + + const target = draw(1)[0]; + store.add(['doc', 7], 'moved', target); + const hit = store.similaritySearch(target, 1)[0]; + expect(hit.nodeRef).toEqual(['doc', 7]); + expect(hit.score).toBeCloseTo(1, 6); + for (const q of [target, fresh]) { + const res = store.similaritySearch(q, 400, undefined, -1); + expect(new Set(refs(res)).size).toBe(res.length); // no stale second entry + const truth = new Map(store.similaritySearch(q, null, undefined, -1).map(r => [`${r.nodeRef}`, r.score])); + for (const r of res) expect(r.score).toBeCloseTo(truth.get(`${r.nodeRef}`), 9); + } + }); + + it('never returns a deleted node, follows compaction without retraining', () => { + const { store, draw } = annStore(); + store.buildAnnIndex(); + const gone = new Set(); + for (let i = 0; i < 4000; i += 3) { // a third: forces compaction + store.remove(['doc', i]); + gone.add(`doc:${i}`); + } + const builds = store.annStats().searches.builds; + for (const q of draw(15)) { + expect(refs(store.similaritySearch(q, 10)).filter(r => gone.has(r))).toEqual([]); + // a tombstone would score 0, which beats any negative cosine + expect(refs(store.similaritySearch(q, 250, undefined, -1)).filter(r => gone.has(r))).toEqual([]); + } + expect(store.recallCheck(draw(40)).recall).toBeGreaterThanOrEqual(RECALL_FLOOR); + expect(store.annStats().searches.builds).toBe(builds); // followed, not retrained + expect(store.count()).toBe(4000 - gone.size); + }); + + it('folds pending rows into cells and retrains after heavy churn', () => { + const { store, draw } = annStore({ n: 3000 }); + store.buildAnnIndex(); + const builds = store.annStats().searches.builds; + const extra = draw(700); + store.addBatch(extra.map((v, i) => [['late', i], '', v])); + expect(store.annStats().pending).toBe(700); + store.similaritySearch(extra[0], 5); // >= 512 pending: folded + let stats = store.annStats(); + expect(stats.pending).toBe(0); + expect(stats.searches.builds).toBe(builds); + + const more = draw(1200); + store.addBatch(more.map((v, i) => [['later', i], '', v])); + store.similaritySearch(more[0], 5); // churn 1900 > 50% of 3000 + stats = store.annStats(); + expect(stats.searches.builds).toBe(builds + 1); + expect(stats.trainedN).toBe(4900); + expect(store.recallCheck(draw(30)).recall).toBeGreaterThanOrEqual(RECALL_FLOOR); + }); + + it('drops the index with a cleared store, and rebuilds for a new dimension', () => { + const { store, data } = annStore({ n: 1500, annNlist: 16, annNprobe: 4 }); + store.buildAnnIndex(); + store.clear(); + expect(store.similaritySearch(data[0], 5)).toEqual([]); + expect(store.annStats().active).toBe(false); + store.clear(true); + const small = clustered(1200, 8, 12, 3); + store.addBatch(small.data.map((v, i) => [['doc', i], '', v])); + const res = store.similaritySearch(small.data[5], 5); + expect(res[0].nodeRef).toEqual(['doc', 5]); + expect(store.annStats().active).toBe(true); + expect(store.dimension).toBe(8); + }); + + it('applies nodeType, widening the probe for a rare type', () => { + const { store, draw } = annStore({ types: i => (i % 200 === 0 ? 'rare' : 'common'), annNprobe: 2 }); + const q = draw(1)[0]; + const common = store.similaritySearch(q, 10, 'common'); + expect(common).toHaveLength(10); + expect(new Set(common.map(r => r.nodeRef[0]))).toEqual(new Set(['common'])); + // 20 rare rows in the whole store and a topK of 15: two cells cannot hold them + const widenedBefore = store.annStats().searches.widened; + const rare = store.similaritySearch(q, 15, 'rare', -1); + expect(rare).toHaveLength(15); + expect(new Set(rare.map(r => r.nodeRef[0]))).toEqual(new Set(['rare'])); + const counts = store.annStats().searches; + expect(counts.widened > widenedBefore || counts.exact > 0).toBe(true); + expect(store.similaritySearch(q, 5, 'nope')).toEqual([]); + }); + + it('respects the threshold', () => { + const { store, draw } = annStore(); + const q = draw(1)[0]; + const best = store.similaritySearch(q, 1, undefined, 0, true)[0].score; + const res = store.similaritySearch(q, 10, undefined, best - 1e-3); + expect(res.every(r => r.score >= best - 1e-3)).toBe(true); + expect(store.similaritySearch(q, 10, undefined, 2)).toEqual([]); + const cut = store.similaritySearch(q, 10, undefined, 0, true)[4].score; + const got = store.similaritySearch(q, 10, undefined, cut); + const want = store.similaritySearch(q, 10, undefined, cut, true); + expect(new Set(refs(got))).toEqual(new Set(refs(want))); + // a threshold that plenty clear is applied inside the probed cells: no scan needed + const before = store.annStats().searches.approximate; + const floor = store.similaritySearch(q, 30, undefined, 0, true)[29].score; + const above = store.similaritySearch(q, 10, undefined, floor); + expect(above).toHaveLength(10); + expect(above.every(r => r.score >= floor)).toBe(true); + expect(store.annStats().searches.approximate).toBe(before + 1); + }); + + it('breaks ties in insertion order, like the exact scan', () => { + const store = new EmbeddingStore(undefined, { ann: 'ivf', annMinSize: 0, annNlist: 4, annNprobe: 2 }); + const same = [1, 0.5, 0.25, 0]; + store.addBatch([...Array.from({ length: 40 }, (_, i) => [['t', i], '', same]), + [['x', 1], '', [0, 1, 0, 0]], [['x', 2], '', [0, 0, 1, 0]], [['x', 3], '', [0, 0, 0, 1]], [['x', 4], '', [-1, 0, 0, 0]]]); + expect(refs(store.similaritySearch(same, 5))).toEqual(['t:0', 't:1', 't:2', 't:3', 't:4']); + expect(store.annStats().searches.approximate).toBe(1); + }); + + it('recallCheck takes one text-free query, a list, or a typed array', () => { + const { store, data, draw } = annStore(); + const q = draw(3); + expect(store.recallCheck(q[0]).queries).toBe(1); // one flat vector + expect(store.recallCheck(q).queries).toBe(3); // a list of them + expect(store.recallCheck(new Float32Array(q[0])).queries).toBe(1); + const off = new EmbeddingStore(); + off.addBatch(data.slice(0, 100).map((v, i) => [['doc', i], '', v])); + const r = off.recallCheck(q); + expect(r.recall).toBe(1); + expect(r.approximate).toBe(false); + expect(() => store.recallCheck(q, 0)).toThrow(/k/); + }); + + it('recallCheck takes text queries through the encoder', async () => { + const encoder = new MockEncoder(32); + const store = new EmbeddingStore(encoder, { ann: 'ivf', annMinSize: 0, annNlist: 16, annNprobe: 16 }); + store.addBatch(Array.from({ length: 300 }, (_, i) => [['doc', i], `word${i} other${i % 7}`])); + expect(store.recallCheck('word5 other5').queries).toBe(1); + expect((await store.recallCheckAsync(['word5', 'word9'])).queries).toBe(2); + }); + + it('is deterministic for a seed', () => { + const run = annSeed => { + const { store, draw } = annStore({ annSeed }); + const out = draw(20).map(q => refs(store.similaritySearch(q, 10))); + expect(store.annStats().searches.approximate).toBe(20); + return out; + }; + expect(run(0)).toEqual(run(0)); + expect(run(7)).toEqual(run(7)); + }); + + it('survives an import and keeps every row findable', () => { + const { store, draw } = annStore({ n: 1200, annNlist: 16, annNprobe: 4 }); + store.buildAnnIndex(); + const q = draw(1)[0]; + const payload = JSON.parse(JSON.stringify(store.exportEmbeddings())); + payload.embeddings.push({ type: 'imp', id: 1, id_type: 'int', text: 'imported', vector: q }); + payload.count += 1; + store.importEmbeddings(payload); + expect(store.similaritySearch(q, 1)[0].nodeRef).toEqual(['imp', 1]); + const other = new EmbeddingStore(undefined, { ann: 'ivf', annMinSize: 0, annNlist: 16, annNprobe: 4 }); + other.similaritySearch(q, 1); // empty: nothing built, no error + other.importEmbeddings(payload, true); + expect(other.similaritySearch(q, 1)[0].nodeRef).toEqual(['imp', 1]); + expect(other.annStats().active).toBe(true); + }); + + it('never changes what is exported', () => { + const { store, data } = annStore({ n: 800, annNlist: 16, annNprobe: 4 }); + const plain = new EmbeddingStore(); + plain.addBatch(data.slice(0, 800).map((v, i) => [['doc', i], `text ${i}`, v])); + store.buildAnnIndex(); + store.similaritySearch(data[0], 5); + expect(JSON.stringify(store.exportEmbeddings())).toBe(JSON.stringify(plain.exportEmbeddings())); + }); + + it('survives writes and searches interleaved (single thread, async)', async () => { + const { store, draw } = annStore({ n: 3000, annNlist: 32, annNprobe: 4 }); + const extra = draw(600); + const queries = draw(40); + await Promise.all([ + (async () => { + for (let i = 0; i < 100; i++) { + store.add(['w', i], '', extra[i % 600]); + if (i % 4 === 0) store.remove(['doc', i]); + if (i % 20 === 0) store.buildAnnIndex(); + await Promise.resolve(); + } + })(), + (async () => { + for (let i = 0; i < 120; i++) { + const res = await store.similaritySearchAsync(queries[i % 40], 8, undefined, -1); + expect(res).toHaveLength(8); + expect(new Set(refs(res)).size).toBe(8); + const scores = res.map(r => r.score); + expect(scores).toEqual([...scores].sort((a, b) => b - a)); + } + })(), + ]); + expect(store.recallCheck(queries.slice(0, 20)).recall).toBeGreaterThanOrEqual(RECALL_FLOOR); + }); +}); + +describe('approximate index: in a SemanticGraph', () => { + function graph(n = 3000, dim = 32, extra = {}) { + const { data, draw } = clustered(n, dim); + const g = new SemanticGraph({ + name: 'ann', encoder: new MockEncoder(dim), autoEmbed: false, + ann: 'ivf', annMinSize: 0, annNlist: 48, annNprobe: 6, ...extra, + }); + for (let i = 0; i < n; i++) g.addNode('item', i, { group: i % 50 }); + g.embeddingStore.addBatch(data.map((v, i) => [['item', i], `item ${i}`, v])); + return { g, data, draw }; + } + + it('passes the options to its store, and recallCheck through', () => { + const { g, draw } = graph(); + const store = g.embeddingStore; + expect(store.ann).toBe('ivf'); + expect(store.annNprobe).toBe(6); + expect(g.similaritySearch(draw(1)[0], 5)).toHaveLength(5); + expect(store.annStats().active).toBe(true); + expect(g.recallCheck(draw(20)).recall).toBeGreaterThanOrEqual(RECALL_FLOOR); + expect(new SemanticGraph().embeddingStore.ann).toBeNull(); + }); + + it('exact: true skips the index, and similarToNode works', () => { + const { g, draw } = graph(); + const q = draw(1)[0]; + const before = g.embeddingStore.annStats().searches.approximate; + g.similaritySearch(q, 5, undefined, 0, undefined, undefined, true); + g.similaritySearch(q, 5, undefined, 0, () => true, undefined, true); + expect(g.embeddingStore.annStats().searches.approximate).toBe(before); + const near = g.similarToNode(['item', 3], 5); + expect(near).toHaveLength(5); + expect(refs(near)).not.toContain('item:3'); + }); + + it('a selective nodeFilter still gets topK results', () => { + const { g, draw } = graph(); + const q = draw(1)[0]; + const group7 = n => n.properties.group === 7; // 1 node in 50 + const res = g.similaritySearch(q, 10, undefined, 0, group7); + expect(res).toHaveLength(10); + expect(res.every(r => g.getNodeByRef(r.nodeRef).properties.group === 7)).toBe(true); + const want = g.similaritySearch(q, 10, undefined, 0, group7, undefined, true); + const overlap = refs(res).filter(r => refs(want).includes(r)).length; + expect(overlap).toBeGreaterThanOrEqual(7); + const scores = res.map(r => r.score); + expect(scores).toEqual([...scores].sort((a, b) => b - a)); + expect(g.embeddingStore.annStats().searches.approximate).toBeGreaterThanOrEqual(1); // probed cells, not a scan + // a filter nothing passes terminates with nothing + expect(g.similaritySearch(q, 10, undefined, 0, () => false)).toEqual([]); + // a lenient one agrees closely with the scan + const even = n => n.properties.group % 2 === 0; + const got = g.similaritySearch(q, 10, undefined, 0, even); + const ref = g.similaritySearch(q, 10, undefined, 0, even, undefined, true); + expect(refs(got).filter(r => refs(ref).includes(r)).length).toBeGreaterThanOrEqual(8); + // nodeTypes goes through the same gate + expect(g.similaritySearch(q, 5, undefined, 0, undefined, ['item'])).toHaveLength(5); + expect(g.similaritySearch(q, 5, undefined, 0, undefined, ['other'])).toEqual([]); + }); + + it('semanticMultiHop takes its seeds from the index, blend > 0 does not', () => { + const { g, draw } = graph(2000); + for (let i = 0; i < 1999; i++) g.addEdge('NEXT', ['item', i], ['item', i + 1]); + const q = draw(1)[0]; + const out = g.semanticMultiHop(q, { relType: 'NEXT', maxHops: 2, topKSeeds: 3, threshold: -1 }); + expect(out.length).toBeGreaterThan(0); + expect(out[0].hopCount).toBe(0); + const used = g.embeddingStore.annStats().searches.approximate; + expect(used).toBeGreaterThanOrEqual(1); + g.semanticMultiHop(q, { relType: 'NEXT', maxHops: 1, blend: 0.3, threshold: -1 }); + expect(g.embeddingStore.annStats().searches.approximate).toBe(used + 1); // the seed search only; scoreMap is a full scan + }); + + it('async text search honours exact', async () => { + const encoder = new MockEncoder(32); + const g = new SemanticGraph({ encoder, ann: 'ivf', annMinSize: 0, annNlist: 16, annNprobe: 4 }); + for (let i = 0; i < 400; i++) g.addNode('doc', i, { name: `entry number ${i} about topic ${i % 9}` }); + const before = g.embeddingStore.annStats().searches.approximate; + const hits = await g.similaritySearchAsync('entry number 12 about topic 3', 3); + expect(hits).toHaveLength(3); + const used = g.embeddingStore.annStats().searches.approximate; + expect(used).toBe(before + 1); + await g.similaritySearchAsync('entry number 12 about topic 3', 3, undefined, 0, undefined, undefined, true); + expect(g.embeddingStore.annStats().searches.approximate).toBe(used); + }); + + describe('files', () => { + const dirs = []; + afterEach(() => { while (dirs.length) rmSync(dirs.pop(), { recursive: true, force: true }); }); + + it('loadGraph passes ann along and rebuilds lazily; nothing about it is saved', () => { + const dir = mkdtempSync(join(tmpdir(), 'isongraph-vector-ann-')); + dirs.push(dir); + const { g, draw } = graph(1500); + const path = join(dir, 'g.ison'); + saveGraph(g, path); + + const plain = new SemanticGraph({ name: 'ann', encoder: new MockEncoder(32), autoEmbed: false }); + for (let i = 0; i < 1500; i++) plain.addNode('item', i, { group: i % 50 }); + plain.embeddingStore.importEmbeddings(g.embeddingStore.exportEmbeddings()); + const plainPath = join(dir, 'plain.ison'); + saveGraph(plain, plainPath); + // same graph, same vectors: the ann graph's files are exactly the plain graph's + expect(readFileSync(path, 'utf8')).toBe(readFileSync(plainPath, 'utf8')); + expect(readFileSync(embeddingsPathFor(path), 'utf8')).toBe(readFileSync(embeddingsPathFor(plainPath), 'utf8')); + expect(readFileSync(embeddingsPathFor(path), 'utf8')).not.toMatch(/ivf|nlist|centroid/i); + expect(graphDigest(g)).toBe(graphDigest(plain)); + + const loaded = loadGraph(path, { + encoder: g.encoder, ann: 'ivf', annMinSize: 0, annNlist: 32, annNprobe: 4, + }); + expect(loaded.embeddingStore.ann).toBe('ivf'); + expect(loaded.embeddingStore.annStats().active).toBe(false); // nothing persisted + expect(loaded.similaritySearch(draw(1)[0], 5)).toHaveLength(5); + expect(loaded.embeddingStore.annStats().active).toBe(true); + }); + }); +}); diff --git a/isongraph-vector-js/src/async-encoder.test.js b/isongraph-vector-js/src/async-encoder.test.js new file mode 100644 index 0000000..47a30ef --- /dev/null +++ b/isongraph-vector-js/src/async-encoder.test.js @@ -0,0 +1,512 @@ +/** + * The asynchronous encoder path: AsyncEncoder, the *Async methods on the store + * and the graph, and HttpEncoder driving auto-embedding with no prefetch. + * Everything runs against a local node:http server - never the real network. + */ + +import { describe, it, expect, afterEach, vi } from 'vitest'; +import { createServer } from 'node:http'; +import { + SemanticGraph, EmbeddingStore, MockEncoder, HttpEncoder, HttpEncoderError, EmbeddingError, + DimensionMismatchError, ModelMismatchError, NoEncoderError, Direction, isAsyncEncoder, +} from './index.js'; + +const servers = []; +afterEach(async () => { + vi.restoreAllMocks(); + while (servers.length) { + const server = servers.pop(); + server.closeAllConnections(); + await new Promise(resolve => server.close(() => resolve())); + } +}); + +/** A server whose answers come from `respond(request, n)`; may delay with `wait` ms or `hang`. */ +async function serve(respond) { + const seen = []; + let inflight = 0; + let peak = 0; + const server = createServer((req, res) => { + const chunks = []; + req.on('data', chunk => chunks.push(chunk)); + req.on('end', () => { + const raw = Buffer.concat(chunks).toString('utf8'); + const request = { + method: req.method, url: req.url, headers: req.headers, + body: raw ? JSON.parse(raw) : null, + }; + const n = seen.length; + seen.push(request); + const answer = respond(request, n); + if (answer.hang) return; + inflight++; + peak = Math.max(peak, inflight); + setTimeout(() => { + inflight--; + res.writeHead(answer.status ?? 200, { 'content-type': 'application/json', ...(answer.headers ?? {}) }); + res.end(typeof answer.body === 'string' ? answer.body : JSON.stringify(answer.body)); + }, answer.wait ?? 0); + }); + }); + servers.push(server); + await new Promise(resolve => server.listen(0, '127.0.0.1', () => resolve())); + const { port } = server.address(); + return { base: `http://127.0.0.1:${port}`, seen, peak: () => peak }; +} + +/** Topic vectors: texts that mention a topic word point along its axis. */ +function vectorOf(text) { + const t = text.toLowerCase(); + return [ + t.includes('engineer') ? 1 : 0, + t.includes('manager') ? 1 : 0, + t.includes('design') ? 1 : 0, + 0.1, + ]; +} + +const openaiAnswer = request => ({ + body: { + data: request.body.input.map((text, index) => ({ index, embedding: vectorOf(text) })).reverse(), + }, +}); +const ollamaAnswer = request => ({ body: { embeddings: request.body.input.map(vectorOf) } }); + +const fast = { retryDelayMs: 1 }; + +describe('AsyncEncoder', () => { + it('HttpEncoder is one, MockEncoder is not', () => { + const http = new HttpEncoder({ url: 'http://localhost:1/e', model: 'm' }); + expect(isAsyncEncoder(http)).toBe(true); + expect(isAsyncEncoder(new MockEncoder(8))).toBe(false); + expect(isAsyncEncoder(null)).toBe(false); + expect(isAsyncEncoder({ encodeAsync: () => 1 })).toBe(false); + }); + + const remote = () => ({ + dimension: 4, + modelName: 'only-async', + version: 'v9', + encodeAsync: async text => vectorOf(text), + encodeBatchAsync: async texts => texts.map(vectorOf), + }); + + it('a pure async encoder works with the Async methods and stamps its model', async () => { + const store = new EmbeddingStore(remote()); + expect(store.encoder).not.toBeNull(); + await store.addAsync(['p', 1], 'Alice the engineer'); + await store.addBatchAsync([[['p', 2], 'Bob the manager'], [['p', 3], 'Carol the designer']]); + + expect(store.count()).toBe(3); + expect(store.models()).toEqual({ 'only-async': 3 }); + expect(store.version).toBe('v9'); + const hits = await store.similaritySearchAsync('engineering', 1); + expect(hits[0].nodeRef).toEqual(['p', 1]); + }); + + it('its synchronous face says to use the Async method', async () => { + const store = new EmbeddingStore(remote()); + expect(() => store.add(['p', 1], 'text')).toThrow(EmbeddingError); + expect(() => store.add(['p', 1], 'text')).toThrow(/Async/); + expect(() => store.similaritySearch('text')).toThrow(/Async/); + // vectors you already have never need the encoder + store.add(['p', 1], 'text', [1, 0, 0, 0]); + expect(store.similaritySearch([1, 0, 0, 0], 1).length).toBe(1); + }); + + it('works through a graph and reports stale vectors when the model changes', async () => { + const graph = new SemanticGraph({ encoder: remote() }); + await graph.addNodeAsync('person', 1, { name: 'Alice the engineer' }); + expect(graph.staleEmbeddings()).toEqual([]); + + graph.setEncoder({ ...remote(), modelName: 'newer' }); + expect(graph.staleEmbeddings()).toEqual([['person', 1]]); + expect(await graph.reembedStaleAsync()).toBe(1); + expect(graph.embeddingStore.models()).toEqual({ newer: 1 }); + }); + + it('async methods also work with a synchronous encoder', async () => { + const encoder = new MockEncoder(16); + const sync = new SemanticGraph({ encoder }); + sync.addNode('person', 1, { name: 'Alice' }); + const viaAsync = new SemanticGraph({ encoder }); + await viaAsync.addNodeAsync('person', 1, { name: 'Alice' }); + expect(viaAsync.getEmbedding(['person', 1]).vector).toEqual(sync.getEmbedding(['person', 1]).vector); + const hits = await viaAsync.similaritySearchAsync('Alice', 1); + expect(hits[0].score).toBeCloseTo(1, 6); + }); + + it('refuses a text query with no encoder', async () => { + const store = new EmbeddingStore(); + await expect(store.similaritySearchAsync('text')).rejects.toThrow(NoEncoderError); + await expect(store.addAsync(['p', 1], 'text')).rejects.toThrow(NoEncoderError); + store.add(['p', 1], 'x', [1, 0]); + expect(await store.similaritySearchAsync([1, 0], 1)).toHaveLength(1); + }); + + it('rejects an encoder that returns the wrong number of vectors', async () => { + const broken = { + dimension: 4, encodeAsync: async () => [1, 0, 0, 0], encodeBatchAsync: async () => [[1, 0, 0, 0]], + }; + const store = new EmbeddingStore(broken); + await expect(store.addBatchAsync([[['p', 1], 'a'], [['p', 2], 'b']])).rejects.toThrow(/1 vectors for 2 texts/); + expect(store.count()).toBe(0); + }); +}); + +describe('SemanticGraph auto-embed over HTTP (openai shape)', () => { + it('addNodeAsync embeds with no prefetch; sync addNode still needs one', async () => { + const { base, seen } = await serve(openaiAnswer); + const encoder = new HttpEncoder({ url: `${base}/v1/embeddings`, model: 'remote-4', apiKey: 'sk-test-key' }); + const graph = new SemanticGraph({ encoder }); + + const node = await graph.addNodeAsync('person', 1, { name: 'Alice', description: 'an engineer' }); + expect(node.ref).toEqual(['person', 1]); + expect(seen).toHaveLength(1); + expect(seen[0].body.input).toEqual(['Alice an engineer']); + expect(seen[0].headers.authorization).toBe('Bearer sk-test-key'); + const record = graph.getEmbedding(['person', 1]); + expect(record.vector).toEqual(vectorOf('Alice an engineer')); + expect(record.model).toBe('remote-4'); + expect(graph.embeddingStore.dimension).toBe(4); + + // the synchronous path is unchanged: it needs the text prefetched + expect(() => graph.addNode('person', 2, { name: 'Bob the manager' })).toThrow(HttpEncoderError); + await encoder.prefetch(['Bob the manager']); + graph.addNode('person', 3, { name: 'Bob the manager' }); + expect(graph.getEmbedding(['person', 3])).not.toBeNull(); + }); + + it('a node with nothing to embed makes no request', async () => { + const { base, seen } = await serve(openaiAnswer); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${base}/e`, model: 'm' }) }); + await graph.addNodeAsync('person', 1, { age: 30 }); + expect(seen).toHaveLength(0); + expect(graph.getEmbedding(['person', 1])).toBeNull(); + expect(graph.getNode('person', 1).properties.age).toBe(30); + }); + + it('autoEmbed off leaves the node unembedded; addNodeWithEmbedAsync embeds the given text', async () => { + const { base, seen } = await serve(openaiAnswer); + const encoder = new HttpEncoder({ url: `${base}/e`, model: 'm' }); + const off = new SemanticGraph({ encoder, autoEmbed: false }); + await off.addNodeAsync('person', 1, { name: 'Alice' }); + expect(seen).toHaveLength(0); + await off.addNodeWithEmbedAsync('person', 2, { name: 'Bob' }, 'a manager'); + expect(seen[0].body.input).toEqual(['a manager']); + expect(off.getEmbedding(['person', 2]).text).toBe('a manager'); + await off.addNodeWithEmbedAsync('person', 3, { name: 'Carol the designer' }); // autoEmbed off -> no embed text, no vector + expect(off.getEmbedding(['person', 3])).toBeNull(); + }); + + it('updateNodeAsync re-embeds on an embed-field change and only then', async () => { + const { base, seen } = await serve(openaiAnswer); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${base}/e`, model: 'm' }) }); + await graph.addNodeAsync('person', 1, { name: 'Alice the engineer', age: 30 }); + expect(seen).toHaveLength(1); + + await graph.updateNodeAsync('person', 1, { age: 31 }); + expect(seen).toHaveLength(1); // no embed field touched + + await graph.updateNodeAsync('person', 1, { name: 'Alice the manager' }); + expect(seen).toHaveLength(2); + expect(graph.getEmbedding(['person', 1]).text).toBe('Alice the manager'); + expect(graph.getEmbedding(['person', 1]).vector).toEqual(vectorOf('manager')); + + await graph.updateNodeAsync('person', 1, { name: null }); // nothing left to embed + expect(graph.getEmbedding(['person', 1])).toBeNull(); + expect(seen).toHaveLength(2); + }); + + it('searches by text over the network and ranks by meaning', async () => { + const { base, seen } = await serve(openaiAnswer); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${base}/e`, model: 'm' }) }); + await graph.embedAllNodesAsync(); // empty graph: nothing to do + for (const [id, name] of [[1, 'Alice the engineer'], [2, 'Bob the manager'], [3, 'Carol the designer']]) { + await graph.addNodeAsync('person', id, { name }); + } + const before = seen.length; + const hits = await graph.similaritySearchAsync('who is the engineering lead', 2, undefined, 0); + expect(seen.length).toBe(before + 1); + expect(seen[before].body.input).toEqual(['who is the engineering lead']); + expect(hits[0].nodeRef).toEqual(['person', 1]); + + const filtered = await graph.similaritySearchAsync('engineering', 5, undefined, -1, n => n.id !== 1); + expect(filtered.every(r => r.nodeRef[1] !== 1)).toBe(true); + }); + + it('semanticMultiHopAsync, semanticPathAsync and semanticSubgraphAsync encode the query once', async () => { + const { base, seen } = await serve(openaiAnswer); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${base}/e`, model: 'm' }) }); + await graph.addNodeAsync('person', 1, { name: 'Alice the engineer' }); + await graph.addNodeAsync('person', 2, { name: 'Bob the manager' }); + await graph.addNodeAsync('person', 3, { name: 'Carol the designer' }); + graph.addEdge('KNOWS', ['person', 1], ['person', 2]); + graph.addEdge('KNOWS', ['person', 2], ['person', 3]); + + const mark = seen.length; + const hops = await graph.semanticMultiHopAsync('engineering lead', { relType: 'KNOWS', maxHops: 2, threshold: 0.5 }); + expect(seen.length).toBe(mark + 1); + expect(hops.map(r => r.nodeRef[1])).toContain(2); + + const path = await graph.semanticPathAsync('an engineer', ['person', 3], { relType: 'KNOWS', threshold: 0.5 }); + expect(path.path.map(r => r[1])).toEqual([1, 2, 3]); + expect(seen.length).toBe(mark + 2); + + const sub = await graph.semanticSubgraphAsync('the engineers', { relType: 'KNOWS', maxHops: 1, threshold: 0.5, direction: Direction.OUT }); + expect(sub.nodeCount()).toBe(2); + expect(seen.length).toBe(mark + 3); + + // the sync versions still take a vector, no encoder involved + expect(graph.semanticMultiHop(vectorOf('engineer'), { relType: 'KNOWS', threshold: 0.5 }).length).toBeGreaterThan(0); + expect(seen.length).toBe(mark + 3); + }); + + it('similarToNode needs no encoder: a plain sync call keeps working', async () => { + const { base } = await serve(openaiAnswer); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${base}/e`, model: 'm' }) }); + await graph.addNodeAsync('person', 1, { name: 'Alice the engineer' }); + await graph.addNodeAsync('person', 2, { name: 'Bob the engineer' }); + expect(graph.similarToNode(['person', 1], 1)[0].nodeRef).toEqual(['person', 2]); + }); +}); + +describe('ollama shape', () => { + it('auto-embeds and searches', async () => { + const { base, seen } = await serve(ollamaAnswer); + const encoder = new HttpEncoder({ url: `${base}/api/embed`, model: 'nomic', api: 'ollama' }); + const graph = new SemanticGraph({ encoder }); + await graph.addNodeAsync('person', 1, { name: 'Alice the engineer' }); + await graph.addNodeAsync('person', 2, { name: 'Bob the manager' }); + expect(seen[0].url).toBe('/api/embed'); + expect(seen[0].body).toEqual({ model: 'nomic', input: ['Alice the engineer'] }); + const hits = await graph.similaritySearchAsync('engineer', 1); + expect(hits[0].nodeRef).toEqual(['person', 1]); + expect(graph.embeddingStore.models()).toEqual({ nomic: 2 }); + }); + + it('a vector of the wrong length is a DimensionMismatchError and changes nothing', async () => { + const { base } = await serve(ollamaAnswer); + const encoder = new HttpEncoder({ url: `${base}/api/embed`, model: 'nomic', api: 'ollama', dimension: 8 }); + const graph = new SemanticGraph({ encoder }); + await expect(graph.addNodeAsync('person', 1, { name: 'Alice' })).rejects.toThrow(DimensionMismatchError); + expect(graph.nodeCount()).toBe(0); + }); +}); + +describe('batching and concurrency', () => { + it('embedAllNodesAsync sends the texts in batchSize requests', async () => { + const { base, seen } = await serve(openaiAnswer); + const encoder = new HttpEncoder({ url: `${base}/e`, model: 'm', batchSize: 40 }); + const graph = new SemanticGraph({ encoder, autoEmbed: false }); + for (let i = 0; i < 100; i++) graph.addNode('doc', i, { name: `document ${i} engineer` }); + + expect(await graph.embedAllNodesAsync()).toBe(100); + expect(seen.map(r => r.body.input.length)).toEqual([40, 40, 20]); + expect(graph.embeddingStore.count()).toBe(100); + + // skipExisting: nothing is requested for nodes that already have a vector + expect(await graph.embedAllNodesAsync(undefined, true)).toBe(0); + expect(seen).toHaveLength(3); + }); + + it('concurrency runs chunk requests in parallel, up to the limit', async () => { + const { base, seen, peak } = await serve(request => ({ ...openaiAnswer(request), wait: 40 })); + const encoder = new HttpEncoder({ url: `${base}/e`, model: 'm', batchSize: 1, concurrency: 3 }); + const vectors = await encoder.encodeBatchAsync(['a', 'b', 'c', 'd', 'e', 'f', 'g']); + expect(vectors).toEqual(['a', 'b', 'c', 'd', 'e', 'f', 'g'].map(vectorOf)); + expect(seen).toHaveLength(7); + expect(peak()).toBe(3); + + const serial = await serve(request => ({ ...openaiAnswer(request), wait: 10 })); + await new HttpEncoder({ url: `${serial.base}/e`, model: 'm', batchSize: 1 }).encodeBatchAsync(['a', 'b', 'c']); + expect(serial.peak()).toBe(1); // default stays sequential + }); + + it('validates concurrency', () => { + expect(() => new HttpEncoder({ url: 'http://localhost:1/e', model: 'm', concurrency: 0 })).toThrow(/concurrency/); + expect(() => new HttpEncoder({ url: 'http://localhost:1/e', model: 'm', concurrency: 1.5 })).toThrow(/concurrency/); + }); + + it('identical texts requested at the same moment share one request', async () => { + const { base, seen } = await serve(request => ({ ...openaiAnswer(request), wait: 30 })); + const encoder = new HttpEncoder({ url: `${base}/e`, model: 'm' }); + const results = await Promise.all([ + encoder.encodeAsync('same engineer'), encoder.encodeAsync('same engineer'), encoder.encodeAsync('same engineer'), + ]); + expect(seen).toHaveLength(1); + expect(results[0]).toEqual(vectorOf('same engineer')); + expect(results[1]).not.toBe(results[0]); // each caller owns its copy + }); + + it('many nodes added at once all land', async () => { + const { base, seen } = await serve(request => ({ ...openaiAnswer(request), wait: 5 })); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${base}/e`, model: 'm' }) }); + await Promise.all(Array.from({ length: 25 }, (_, i) => graph.addNodeAsync('doc', i, { name: `doc ${i} designer` }))); + expect(graph.nodeCount()).toBe(25); + expect(graph.embeddingStore.count()).toBe(25); + expect(seen.length).toBeGreaterThan(0); + expect((await graph.similaritySearchAsync('design', 25, undefined, 0.5)).length).toBe(25); + }); + + it('a failed request in the middle of a batch rejects the whole call and stores nothing', async () => { + const { base } = await serve((request, n) => + n === 1 ? { status: 400, body: 'bad' } : openaiAnswer(request)); + const encoder = new HttpEncoder({ url: `${base}/e`, model: 'm', batchSize: 2, ...fast }); + const graph = new SemanticGraph({ encoder, autoEmbed: false }); + for (let i = 0; i < 6; i++) graph.addNode('doc', i, { name: `doc ${i}` }); + await expect(graph.embedAllNodesAsync()).rejects.toThrow(HttpEncoderError); + expect(graph.embeddingStore.count()).toBe(0); + }); +}); + +describe('stale detection and re-embedding over the network', () => { + it('reembedStaleAsync rewrites vectors from another model, in batches', async () => { + const { base, seen } = await serve(openaiAnswer); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${base}/e`, model: 'old', batchSize: 100 }) }); + for (let i = 0; i < 5; i++) await graph.addNodeAsync('doc', i, { name: `doc ${i} engineer` }); + expect(graph.staleEmbeddings()).toEqual([]); + + graph.setEncoder(new HttpEncoder({ url: `${base}/e`, model: 'new', batchSize: 100 })); + expect(graph.staleEmbeddings()).toHaveLength(5); + const mark = seen.length; + expect(await graph.reembedStaleAsync(false, 2)).toBe(5); + expect(seen.length - mark).toBe(3); // 2 + 2 + 1 texts + expect(graph.embeddingStore.models()).toEqual({ new: 5 }); + expect(graph.staleEmbeddings()).toEqual([]); + expect(seen[mark].body.model).toBe('new'); + }); + + it('a different dimension empties the store and re-embeds everything, learning it from the answer', async () => { + const wide = await serve(request => ({ + body: { data: request.body.input.map((t, index) => ({ index, embedding: [...vectorOf(t), 0.5] })) }, + })); + const { base } = await serve(openaiAnswer); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${base}/e`, model: 'four' }) }); + await graph.addNodeAsync('doc', 1, { name: 'Alice the engineer' }); + expect(graph.embeddingStore.dimension).toBe(4); + + graph.setEncoder(new HttpEncoder({ url: `${wide.base}/e`, model: 'five' })); // dimension not known yet + expect(await graph.reembedStaleAsync()).toBe(1); + expect(graph.embeddingStore.dimension).toBe(5); + expect(graph.getEmbedding(['doc', 1]).vector).toHaveLength(5); + }); + + it('a failing re-embed leaves the old vectors alone', async () => { + const { base } = await serve((request, n) => n === 0 ? openaiAnswer(request) : { status: 400, body: 'no' }); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${base}/e`, model: 'old', ...fast }) }); + await graph.addNodeAsync('doc', 1, { name: 'Alice the engineer' }); + graph.setEncoder(new HttpEncoder({ url: `${base}/e`, model: 'new', ...fast })); + await expect(graph.reembedStaleAsync()).rejects.toThrow(HttpEncoderError); + expect(graph.getEmbedding(['doc', 1]).model).toBe('old'); + expect(graph.staleEmbeddings()).toHaveLength(1); + }); + + it('a text query from another model follows onModelMismatch', async () => { + const { base } = await serve(openaiAnswer); + const store = new EmbeddingStore(new HttpEncoder({ url: `${base}/e`, model: 'old' }), { onModelMismatch: 'error' }); + await store.addAsync(['p', 1], 'Alice the engineer'); + store.encoder = new HttpEncoder({ url: `${base}/e`, model: 'new' }); + await expect(store.similaritySearchAsync('engineer')).rejects.toThrow(ModelMismatchError); + expect(await store.similaritySearchAsync([1, 0, 0, 0.1], 1)).toHaveLength(1); // vectors are never checked + + store.onModelMismatch = 'warn'; + const warn = vi.spyOn(console, 'warn').mockImplementation(() => {}); + await store.similaritySearchAsync('engineer'); + await store.similaritySearchAsync('engineer'); + expect(warn).toHaveBeenCalledTimes(1); + }); +}); + +describe('failures from the network', () => { + it('a server error leaves the graph as it was', async () => { + const { base } = await serve(() => ({ status: 500, body: 'boom' })); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${base}/e`, model: 'm', maxRetries: 0 }) }); + const error = await graph.addNodeAsync('person', 1, { name: 'Alice' }).catch(e => e); + expect(error).toBeInstanceOf(HttpEncoderError); + expect(error.status).toBe(500); + expect(graph.nodeCount()).toBe(0); + expect(graph.embeddingStore.count()).toBe(0); + // ... so the call can simply be retried + await expect(graph.addNodeAsync('person', 1, { name: 'Alice' })).rejects.toThrow(HttpEncoderError); + }); + + it('updateNodeAsync that cannot embed keeps the old properties and vector', async () => { + const { base } = await serve((request, n) => n === 0 ? openaiAnswer(request) : { status: 503, body: 'down' }); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${base}/e`, model: 'm', maxRetries: 0 }) }); + await graph.addNodeAsync('person', 1, { name: 'Alice the engineer' }); + await expect(graph.updateNodeAsync('person', 1, { name: 'Alice the manager' })).rejects.toThrow(HttpEncoderError); + expect(graph.getNode('person', 1).properties.name).toBe('Alice the engineer'); + expect(graph.getEmbedding(['person', 1]).text).toBe('Alice the engineer'); + }); + + it('retries a 429 then succeeds', async () => { + const { base, seen } = await serve((request, n) => + n === 0 ? { status: 429, headers: { 'retry-after': '0' }, body: 'slow' } : openaiAnswer(request)); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${base}/e`, model: 'm', ...fast }) }); + await graph.addNodeAsync('person', 1, { name: 'Alice' }); + expect(seen).toHaveLength(2); + expect(graph.embeddingStore.count()).toBe(1); + }); + + it('a request that never completes times out, and the node is not added', async () => { + const { base, seen } = await serve(() => ({ hang: true })); + const graph = new SemanticGraph({ + encoder: new HttpEncoder({ url: `${base}/e`, model: 'm', timeoutMs: 50, maxRetries: 1, ...fast }), + }); + const error = await graph.addNodeAsync('person', 1, { name: 'Alice' }).catch(e => e); + expect(error).toBeInstanceOf(HttpEncoderError); + expect(error.message).toContain('timed out'); + expect(seen).toHaveLength(2); + expect(graph.nodeCount()).toBe(0); + await expect(graph.similaritySearchAsync('text')).rejects.toThrow(/timed out/); + }); + + it('a refused connection is reported, not hung', async () => { + const { base } = await serve(openaiAnswer); + const dead = base.replace(/:\d+$/, ':1'); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${dead}/e`, model: 'm', maxRetries: 0 }) }); + await expect(graph.addNodeAsync('person', 1, { name: 'Alice' })).rejects.toThrow(/failed/); + }); + + it('the api key never appears in an error from any async path', async () => { + const secret = 'sk-do-not-leak-0123456789'; + // a server that echoes the credentials back, the worst case + const { base } = await serve(request => ({ status: 401, body: `rejected ${request.headers.authorization}` })); + const encoder = new HttpEncoder({ + url: `${base}/e?api_key=query-secret`, model: 'm', apiKey: secret, headers: { 'X-Api-Token': 'header-secret' }, ...fast, + }); + const graph = new SemanticGraph({ encoder }); + const errors = []; + errors.push(await graph.addNodeAsync('person', 1, { name: 'Alice' }).catch(e => e)); + errors.push(await graph.similaritySearchAsync('Alice').catch(e => e)); + const manual = new SemanticGraph({ encoder, autoEmbed: false }); + manual.addNode('person', 1, { name: 'Alice' }); + errors.push(await manual.embedAllNodesAsync().catch(e => e)); + errors.push(await graph.embeddingStore.addBatchAsync([[['p', 1], 'x']]).catch(e => e)); + for (const error of errors) { + expect(error).toBeInstanceOf(Error); + for (const leaked of [secret, 'query-secret', 'header-secret']) { + expect(String(error.message)).not.toContain(leaked); + expect(String(error.stack)).not.toContain(leaked); + } + } + expect(errors.some(e => e instanceof HttpEncoderError && e.status === 401)).toBe(true); + expect(JSON.stringify(graph.encoder)).not.toContain(secret); + }); + + it('sends the key as a bearer token and lets custom headers win', async () => { + const { base, seen } = await serve(openaiAnswer); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${base}/e`, model: 'm', apiKey: 'sk-abc' }) }); + await graph.addNodeAsync('person', 1, { name: 'Alice' }); + await graph.similaritySearchAsync('Alice'); + expect(seen.every(r => r.headers.authorization === 'Bearer sk-abc')).toBe(true); + + const custom = await serve(openaiAnswer); + const g2 = new SemanticGraph({ + encoder: new HttpEncoder({ url: `${custom.base}/e`, model: 'm', apiKey: 'ignored', headers: { Authorization: 'Token zzz' } }), + }); + await g2.addNodeAsync('person', 1, { name: 'Alice' }); + expect(custom.seen[0].headers.authorization).toBe('Token zzz'); + }); +}); diff --git a/isongraph-vector-js/src/index.js b/isongraph-vector-js/src/index.js index e01152c..2941e52 100644 --- a/isongraph-vector-js/src/index.js +++ b/isongraph-vector-js/src/index.js @@ -89,6 +89,55 @@ export class GraphEmbeddingMismatchError extends EmbeddingError { this.name = 'GraphEmbeddingMismatchError'; } } +/** True for an object with `encodeAsync` and `encodeBatchAsync` methods. */ +export function isAsyncEncoder(value) { + const v = value; + return (v !== null && v !== undefined && typeof v === 'object' && + typeof v.encodeAsync === 'function' && typeof v.encodeBatchAsync === 'function'); +} +/** + * The synchronous face of an encoder that only has an asynchronous one, so the + * stores (whose `encoder` is an EmbeddingEncoder) can hold it. Its metadata is + * read live from the wrapped encoder; its synchronous methods refuse, naming + * the asynchronous ones. + */ +class AsyncOnlyEncoder { + constructor(inner) { + this.inner = inner; + } + get dimension() { + return this.inner.dimension; + } + get modelName() { + return this.inner.modelName; + } + get version() { + return this.inner.version; + } + encode(_text) { + throw new EmbeddingError('this encoder is asynchronous: use the Async methods (addAsync, addNodeAsync, ' + + 'similaritySearchAsync, ...) or pass the vector yourself'); + } + encodeBatch(texts) { + return texts.map(text => this.encode(text)); + } + encodeAsync(text) { + return this.inner.encodeAsync(text); + } + encodeBatchAsync(texts) { + return this.inner.encodeBatchAsync(texts); + } +} +/** What a store or graph holds: sync encoders as they are, async-only ones wrapped. */ +function adoptEncoder(encoder) { + if (encoder === null || encoder === undefined) + return null; + const sync = encoder; + if (typeof sync.encode === 'function' && typeof sync.encodeBatch === 'function') { + return encoder; + } + return isAsyncEncoder(encoder) ? new AsyncOnlyEncoder(encoder) : encoder; +} /** * Deterministic encoder with no dependencies, for tests and offline work. * @@ -202,20 +251,29 @@ const ERROR_SNIPPET_CHARS = 200; * **Every text you embed is sent to the configured endpoint**, including the * text of every node a graph embeds and every search query. * - * `EmbeddingEncoder` is synchronous and a network call is not, so the encoder - * works in two steps: fetch the vectors with `await encoder.prefetch(texts)` - * (or `encodeAsync` / `encodeBatchAsync`, which fetch and return them), then - * use the synchronous `encode` / `encodeBatch` the stores call. Those serve - * from a cache and throw an HttpEncoderError for a text that was not fetched - * first. That makes `SemanticGraph` auto-embedding unusable with this encoder - * unless the node texts are prefetched; the plain stores (`addBatch`, - * `similaritySearch` with a prefetched query) are the intended fit. + * It is an {@link AsyncEncoder} (`encodeAsync` / `encodeBatchAsync`), and that + * is the way to use it: give it to a store or graph and call the `...Async` + * methods (`addNodeAsync`, `updateNodeAsync`, `embedAllNodesAsync`, + * `similaritySearchAsync`, ...), which await the network themselves. Identical + * texts requested at the same moment share one request. + * + * `EmbeddingEncoder` is synchronous and a network call is not, so the + * synchronous `encode` / `encodeBatch` serve from a cache: fetch first with + * `await encoder.prefetch(texts)` (every `...Async` call fills it too), and a + * text that was not fetched makes them throw an HttpEncoderError. That keeps + * the plain synchronous methods (`addNode`, `addBatch`, `similaritySearch`) + * usable for prefetched texts. * * @example * ```typescript * const encoder = new HttpEncoder({ * url: 'http://localhost:11434/api/embed', model: 'nomic-embed-text', api: 'ollama', * }); + * const graph = new SemanticGraph({ encoder }); + * await graph.addNodeAsync('person', 1, { name: 'Alice' }); // embeds over the network + * const hits = await graph.similaritySearchAsync('who is Alice'); + * + * // or, synchronously, after fetching the texts: * const texts = ['Alice is an engineer', 'Bob plans roadmaps']; * await encoder.prefetch([...texts, 'engineering']); * const store = new EmbeddingStore(encoder); @@ -226,6 +284,7 @@ const ERROR_SNIPPET_CHARS = 200; export class HttpEncoder { constructor(options) { this._cache = new Map(); + this._inflight = new Map(); if (!options || typeof options.url !== 'string' || typeof options.model !== 'string' || !options.model) { throw new EmbeddingError('HttpEncoder needs a url and a model'); } @@ -275,6 +334,7 @@ export class HttpEncoder { this._maxRetries = Math.floor(nonNegative(options.maxRetries, 2, 'maxRetries')); this._retryDelayMs = nonNegative(options.retryDelayMs, 500, 'retryDelayMs'); this._maxCache = Math.floor(positive(options.maxCacheEntries, 100000, 'maxCacheEntries', false)); + this._concurrency = positive(options.concurrency, 1, 'concurrency', true); const fetchFn = options.fetch ?? globalThis.fetch; if (typeof fetchFn !== 'function') { throw new EmbeddingError('HttpEncoder needs a global fetch (Node 18+) or a fetch option'); @@ -317,28 +377,69 @@ export class HttpEncoder { async encodeAsync(text) { return (await this.encodeBatchAsync([text]))[0]; } - /** Fetch vectors for many texts, in order; only uncached texts hit the network. */ + /** + * Fetch vectors for many texts, in order. Only texts that are neither cached + * nor already being fetched hit the network, `batchSize` per request and + * `concurrency` requests at a time; the first failure rejects the call. + */ async encodeBatchAsync(texts) { const found = new Map(); - const missing = new Set(); + const fresh = []; + const waits = []; + const considered = new Set(); for (const text of texts) { - if (found.has(text) || missing.has(text)) + if (considered.has(text)) continue; + considered.add(text); const hit = this._cache.get(text); - if (hit) + if (hit) { found.set(text, hit); - else - missing.add(text); - } - const pending = [...missing]; - for (let start = 0; start < pending.length; start += this._batchSize) { - const chunk = pending.slice(start, start + this._batchSize); - const vectors = await this._request(chunk); - chunk.forEach((text, i) => { - found.set(text, vectors[i]); - this._remember(text, vectors[i]); - }); - } + continue; + } + const shared = this._inflight.get(text); + if (shared) { + waits.push(shared.then(fetched => { found.set(text, fetched.get(text)); })); + continue; + } + fresh.push(text); + } + const chunks = []; + for (let start = 0; start < fresh.length; start += this._batchSize) { + chunks.push(fresh.slice(start, start + this._batchSize)); + } + let next = 0; + let failed = false; + const worker = async () => { + while (!failed && next < chunks.length) { + const chunk = chunks[next++]; + const job = this._request(chunk).then(vectors => { + const fetched = new Map(); + chunk.forEach((text, i) => { + fetched.set(text, vectors[i]); + this._remember(text, vectors[i]); + }); + return fetched; + }); + for (const text of chunk) + this._inflight.set(text, job); + const release = () => { + for (const text of chunk) + if (this._inflight.get(text) === job) + this._inflight.delete(text); + }; + job.then(release, release); + try { + for (const [text, vector] of await job) + found.set(text, vector); + } + catch (error) { + failed = true; + throw error; + } + } + }; + const workers = Array.from({ length: Math.min(this._concurrency, chunks.length) }, worker); + await Promise.all([...waits, ...workers]); return texts.map(text => found.get(text).slice()); } /** Synchronous: serves a prefetched text, otherwise throws HttpEncoderError. */ @@ -487,8 +588,736 @@ export function selectTopK(results, k) { } return top; } +// ============================================================================= +// Approximate (IVF) search +// ============================================================================= +/** Stores below this many vectors answer exactly even with `ann` on. */ +const ANN_MIN_SIZE = 50000; +/** New or updated rows are scored exactly until this many (or this fraction of the trained size) are pending. */ +const ANN_FOLD_MIN = 512; +const ANN_FOLD_FRACTION = 0.02; +/** Retrain once writes + deletes since training exceed this fraction of the trained size. */ +const ANN_RETRAIN_FRACTION = 0.5; +/** + * k-means trains on up to this many sampled vectors per cell, for this many + * rounds. Python samples 40 per cell; with no BLAS here the build cost matters, + * and on 600k clustered vectors 24 per cell took 296 s against 390 s for 40 + * with the same recall at the default probe (0.978) and a little less at + * `annNprobe` 1 (0.93 against 0.96). + */ +const ANN_SAMPLE_PER_CELL = 24; +const ANN_KMEANS_ITERS = 8; +/** From this many cells on, nearest-cell lookups go through a coarse layer of about 3 x sqrt(nlist) super-cells. */ +const ANN_COARSE_MIN_CELLS = 256; +/** + * ... and look inside this many of the super-cells nearest the vector. Measured + * on 100k-300k clustered vectors: this finds the true nearest cell for 95-99% + * of them at about a third of the brute-force cost; looking inside only 4 + * left 2-20% in a different cell, which capped recall whatever `annNprobe`. + */ +const ANN_COARSE_PROBE = 24; +/** k-means rounds that place the super-cells over the centroids. */ +const ANN_COARSE_ROUNDS = 3; +/** Super-cells per sqrt(nlist): finer ones find the nearest cell more often for the same work. */ +const ANN_COARSE_FACTOR = 3; +/** float32 rows vs the float64 exact score differ by about 1e-7; candidates this close under the threshold are re-scored. */ +const ANN_THRESHOLD_SLACK = 1e-6; +/** Banker's rounding, as Python's `round`, so the default cell counts agree across ports. */ +function roundHalfEven(x) { + const floor = Math.floor(x); + const diff = x - floor; + if (diff < 0.5) + return floor; + if (diff > 0.5) + return floor + 1; + return floor % 2 === 0 ? floor : floor + 1; +} +const nowMs = () => typeof performance !== 'undefined' ? performance.now() : Date.now(); +/** Small seeded generator (mulberry32): the same seed gives the same index. */ +class Rng { + constructor(seed) { + this._a = (Math.trunc(seed) ^ 0x9e3779b9) >>> 0; + } + next() { + this._a = (this._a + 0x6d2b79f5) >>> 0; + let t = this._a; + t = Math.imul(t ^ (t >>> 15), t | 1); + t ^= t + Math.imul(t ^ (t >>> 7), t | 61); + return ((t ^ (t >>> 14)) >>> 0) / 4294967296; + } + /** Integer in [0, n). */ + below(n) { + return Math.floor(this.next() * n); + } +} +/** Dot product of a[ao..ao+dim) and b[bo..bo+dim), four accumulators wide. */ +function dot2(a, ao, b, bo, dim) { + let s0 = 0; + let s1 = 0; + let s2 = 0; + let s3 = 0; + let j = 0; + const lim = dim - 3; + for (; j < lim; j += 4) { + s0 += a[ao + j] * b[bo + j]; + s1 += a[ao + j + 1] * b[bo + j + 1]; + s2 += a[ao + j + 2] * b[bo + j + 2]; + s3 += a[ao + j + 3] * b[bo + j + 3]; + } + for (; j < dim; j++) + s0 += a[ao + j] * b[bo + j]; + return s0 + s1 + s2 + s3; +} +/** + * Nearest unit centroid to a vector, by dot product. + * + * With many cells a brute-force pass over all of them costs nlist x dim per + * vector, which is what makes k-means and assignment slow in a language with no + * BLAS. Past `ANN_COARSE_MIN_CELLS` the centroids are themselves grouped into + * about sqrt(nlist) super-cells and only the members of the few super-cells + * nearest the vector are compared: roughly 10x less work, and the cell it picks + * is the true nearest one all but rarely. A vector put in a near-miss cell is + * still found by any query that probes that cell. + */ +class CentroidSearch { + constructor(centroids, nlist, dim, coarseFactor = ANN_COARSE_FACTOR, probe = ANN_COARSE_PROBE) { + this._coarse = null; + this._ncoarse = 0; + this._members = new Int32Array(0); + this._memberOffsets = new Int32Array(0); + this._coarseScore = new Float64Array(0); + this._pickIdx = new Int32Array(0); + this._pickScore = new Float64Array(0); + this._nlist = nlist; + this._dim = dim; + this._centroids = centroids; + this._probe = probe; + if (nlist >= ANN_COARSE_MIN_CELLS) + this._buildCoarse(coarseFactor); + } + _buildCoarse(coarseFactor) { + const nlist = this._nlist; + const dim = this._dim; + const centroids = this._centroids; + const ncoarse = Math.max(2, Math.round(coarseFactor * Math.sqrt(nlist))); + const coarse = new Float32Array(ncoarse * dim); + for (let i = 0; i < ncoarse; i++) { + const src = Math.floor((i * nlist) / ncoarse) * dim; + coarse.set(centroids.subarray(src, src + dim), i * dim); + } + const owner = new Int32Array(nlist); + const sums = new Float64Array(ncoarse * dim); + const counts = new Int32Array(ncoarse); + for (let round = 0; round < ANN_COARSE_ROUNDS; round++) { + for (let c = 0; c < nlist; c++) { + let best = 0; + let bestScore = -Infinity; + for (let k = 0; k < ncoarse; k++) { + const s = dot2(centroids, c * dim, coarse, k * dim, dim); + if (s > bestScore) { + bestScore = s; + best = k; + } + } + owner[c] = best; + } + if (round === ANN_COARSE_ROUNDS - 1) + break; + sums.fill(0); + counts.fill(0); + for (let c = 0; c < nlist; c++) { + const k = owner[c]; + counts[k]++; + const base = k * dim; + const src = c * dim; + for (let j = 0; j < dim; j++) + sums[base + j] += centroids[src + j]; + } + for (let k = 0; k < ncoarse; k++) { + if (counts[k] === 0) + continue; + let len = 0; + for (let j = 0; j < dim; j++) + len += sums[k * dim + j] * sums[k * dim + j]; + len = Math.sqrt(len); + if (len === 0) + continue; + for (let j = 0; j < dim; j++) + coarse[k * dim + j] = sums[k * dim + j] / len; + } + } + const memberOffsets = new Int32Array(ncoarse + 1); + for (let c = 0; c < nlist; c++) + memberOffsets[owner[c] + 1]++; + for (let k = 0; k < ncoarse; k++) + memberOffsets[k + 1] += memberOffsets[k]; + const fill = memberOffsets.slice(0, ncoarse); + const members = new Int32Array(nlist); + for (let c = 0; c < nlist; c++) + members[fill[owner[c]]++] = c; + this._coarse = coarse; + this._ncoarse = ncoarse; + this._members = members; + this._memberOffsets = memberOffsets; + this._coarseScore = new Float64Array(ncoarse); + const probe = Math.min(this._probe, ncoarse); + this._pickIdx = new Int32Array(probe); + this._pickScore = new Float64Array(probe); + } + /** Index of the centroid most similar to v[off..off+dim). */ + nearest(v, off) { + const dim = this._dim; + const centroids = this._centroids; + let best = 0; + let bestScore = -Infinity; + const coarse = this._coarse; + if (coarse === null) { + for (let c = 0; c < this._nlist; c++) { + const s = dot2(centroids, c * dim, v, off, dim); + if (s > bestScore) { + bestScore = s; + best = c; + } + } + return best; + } + const pickIdx = this._pickIdx; + const pickScore = this._pickScore; + const probe = pickIdx.length; + let picked = 0; + for (let k = 0; k < this._ncoarse; k++) { + const s = dot2(coarse, k * dim, v, off, dim); + if (picked === probe && s <= pickScore[probe - 1]) + continue; + let i = picked < probe ? picked++ : probe - 1; + while (i > 0 && pickScore[i - 1] < s) { + pickScore[i] = pickScore[i - 1]; + pickIdx[i] = pickIdx[i - 1]; + i--; + } + pickScore[i] = s; + pickIdx[i] = k; + } + const members = this._members; + const offsets = this._memberOffsets; + for (let p = 0; p < picked; p++) { + const k = pickIdx[p]; + for (let m = offsets[k]; m < offsets[k + 1]; m++) { + const c = members[m]; + const s = dot2(centroids, c * dim, v, off, dim); + if (s > bestScore) { + bestScore = s; + best = c; + } + } + } + return best; + } +} +/** + * Float32Array snapshot of the store's vectors, kept in step with its writes. + * + * One contiguous matrix with spare row capacity, the vector norms, a type code + * per row and a liveness flag. A write patches it in place: an insert appends + * a row, an update overwrites one, a delete tombstones one (squeezed out once + * a quarter of the rows are dead). Rows stay in insertion order, which is the + * order the store's Map iterates, so ties break exactly as the exact scan's do. + * Built lazily by the first approximate search, never persisted. + */ +class ScanCache { + constructor(dim, capacity) { + /** Rows in use, tombstones included, and how many of those are tombstones. */ + this.n = 0; + this.dead = 0; + this.keys = []; + this.pos = new Map(); + this.codeOf = new Map(); + this.ivf = null; + this.dim = dim; + this.matrix = new Float32Array(capacity * dim); + this.norms = new Float64Array(capacity); + this.codes = new Int32Array(capacity); + this.alive = new Uint8Array(capacity); + this.query = new Float32Array(dim); + } + static build(records, count, dim) { + const cache = new ScanCache(dim, count + Math.max(1024, count >> 4)); + for (const record of records) + cache.put(record.id, record); + return cache; + } + get capacity() { + return this.norms.length; + } + _grow() { + const capacity = this.capacity + Math.max(1024, this.capacity >> 2); + const matrix = new Float32Array(capacity * this.dim); + matrix.set(this.matrix.subarray(0, this.n * this.dim)); + const norms = new Float64Array(capacity); + norms.set(this.norms.subarray(0, this.n)); + const codes = new Int32Array(capacity); + codes.set(this.codes.subarray(0, this.n)); + const alive = new Uint8Array(capacity); + alive.set(this.alive.subarray(0, this.n)); + this.matrix = matrix; + this.norms = norms; + this.codes = codes; + this.alive = alive; + } + /** Mirror one upserted record. */ + put(key, record) { + const dim = this.dim; + let i = this.pos.get(key); + if (i === undefined) { + if (this.n >= this.capacity) + this._grow(); + i = this.n++; + this.keys[i] = key; + this.pos.set(key, i); + const type = record.nodeRef[0]; + let code = this.codeOf.get(type); + if (code === undefined) { + code = this.codeOf.size; + this.codeOf.set(type, code); + } + this.codes[i] = code; + this.alive[i] = 1; + } + const vector = record.vector; + const base = i * dim; + let sum = 0; + for (let j = 0; j < dim; j++) { + const f = Math.fround(vector[j]); + this.matrix[base + j] = f; + sum += f * f; + } + this.norms[i] = Math.sqrt(sum); + if (this.ivf !== null) + this.ivf.notePut(i, this.capacity); + } + /** Mirror one deleted record. */ + drop(key) { + const i = this.pos.get(key); + if (i === undefined) + return; + this.pos.delete(key); + this.matrix.fill(0, i * this.dim, (i + 1) * this.dim); + this.norms[i] = 0; + this.alive[i] = 0; + this.keys[i] = null; + this.dead++; + if (this.ivf !== null) + this.ivf.noteDrop(i); + while (this.n > 0 && this.alive[this.n - 1] === 0) { // trailing tombstones + this.n--; + this.keys.pop(); + this.dead--; + } + if (this.dead > 32 && this.dead * 4 > this.n) + this._compact(); + } + /** Squeeze the tombstones out, keeping the live rows in their order. */ + _compact() { + const dim = this.dim; + const live = new Int32Array(this.n - this.dead); + let kept = 0; + for (let r = 0; r < this.n; r++) + if (this.alive[r] === 1) + live[kept++] = r; + if (this.ivf !== null) + this.ivf.compact(live); + for (let w = 0; w < kept; w++) { + const r = live[w]; + if (r !== w) { + this.matrix.copyWithin(w * dim, r * dim, (r + 1) * dim); + this.norms[w] = this.norms[r]; + this.codes[w] = this.codes[r]; + const key = this.keys[r]; + this.keys[w] = key; + this.pos.set(key, w); + } + } + this.alive.fill(1, 0, kept); + this.alive.fill(0, kept, this.n); + this.keys.length = kept; + this.n = kept; + this.dead = 0; + } + bytes() { + return this.matrix.byteLength + this.norms.byteLength + this.codes.byteLength + this.alive.byteLength; + } +} +/** + * Inverted-file ("IVF-flat") index over a {@link ScanCache}. + * + * Spherical k-means partitions the rows into `nlist` cells by direction, which + * is what cosine similarity compares. A query scores the `nprobe` cells whose + * centroids are nearest and only the rows inside them, with the same cosine + * formula as the exact scan. The vectors are not copied again: the index is a + * cell number per row (`assign`) plus the rows grouped by cell + * (`order`/`offsets`), about 8 bytes per vector plus the centroids. + * + * It follows its cache instead of being rebuilt on every write: + * - a **new** or **updated** row is *pending* (cell -1): every query scores + * the pending rows exactly, so a write is visible at once and costs the + * index nothing; + * - pending rows are folded into their nearest cell in one pass once there + * are enough of them (see `EmbeddingStore._annIndex`); + * - a **delete** clears the row's cell; stale entries in `order` are skipped + * at query time; + * - the cache's compaction renumbers rows, and `compact` carries the cells + * across without retraining; + * - `churn` counts writes and deletes since training; the store retrains when + * it passes a fraction of the trained size. + */ +class IvfIndex { + constructor(centroids, nlist, dim, assign) { + this.order = new Int32Array(0); + this.offsets = new Int32Array(0); + this.pending = new Set(); + this.churn = 0; + this.trainedN = 0; + this.buildSeconds = 0; + // Scratch kept between queries: a search allocates nothing proportional to the store. + this._chosen = new Int32Array(0); + this._chosenScore = new Float64Array(0); + this.candRows = new Int32Array(0); + this.candScores = new Float64Array(0); + this.nlist = nlist; + this.dim = dim; + this.centroids = centroids; + this.assign = assign; + this._search = new CentroidSearch(centroids, nlist, dim); + } + /** Train centroids on a sample, then give every live row its cell. */ + static build(cache, nlistWanted, seed) { + const started = nowMs(); + const { n, dim, matrix, norms, alive } = cache; + const live = new Int32Array(n - cache.dead); + let total = 0; + for (let r = 0; r < n; r++) + if (alive[r] === 1) + live[total++] = r; + const nlist = Math.max(1, Math.min(Math.floor(nlistWanted), total)); + const rng = new Rng(seed); + // Sample up to 40 vectors per cell (a partial Fisher-Yates), as unit rows. + const sampleN = Math.min(total, ANN_SAMPLE_PER_CELL * nlist); + let sample; + if (sampleN === total) { + sample = live; + } + else { + const pool = live.slice(); + for (let i = 0; i < sampleN; i++) { + const j = i + rng.below(total - i); + const t = pool[i]; + pool[i] = pool[j]; + pool[j] = t; + } + sample = pool.subarray(0, sampleN); + sample.sort(); + } + const x = new Float32Array(sampleN * dim); + for (let i = 0; i < sampleN; i++) { + const r = sample[i]; + const inv = norms[r] > 0 ? 1 / norms[r] : 0; + const src = r * dim; + const dst = i * dim; + for (let j = 0; j < dim; j++) + x[dst + j] = matrix[src + j] * inv; + } + // Initial centroids: distinct sample points. + const centroids = new Float32Array(nlist * dim); + const picks = new Int32Array(sampleN); + for (let i = 0; i < sampleN; i++) + picks[i] = i; + for (let c = 0; c < nlist; c++) { + const j = c + rng.below(sampleN - c); + const t = picks[c]; + picks[c] = picks[j]; + picks[j] = t; + centroids.set(x.subarray(picks[c] * dim, (picks[c] + 1) * dim), c * dim); + } + // Spherical k-means: assign to the most similar centroid, re-centre on the sum, re-normalise. + const cell = new Int32Array(sampleN); + const counts = new Int32Array(nlist); + const sums = new Float64Array(nlist * dim); + for (let iter = 0; iter < ANN_KMEANS_ITERS; iter++) { + const search = new CentroidSearch(centroids, nlist, dim); + counts.fill(0); + sums.fill(0); + for (let i = 0; i < sampleN; i++) { + const c = search.nearest(x, i * dim); + cell[i] = c; + counts[c]++; + const base = c * dim; + const src = i * dim; + for (let j = 0; j < dim; j++) + sums[base + j] += x[src + j]; + } + for (let c = 0; c < nlist; c++) { + let len = 0; + if (counts[c] > 0) { + for (let j = 0; j < dim; j++) + len += sums[c * dim + j] * sums[c * dim + j]; + len = Math.sqrt(len); + } + if (counts[c] === 0 || len === 0) { // a dead centroid: re-seed on a random point + const i = rng.below(sampleN); + centroids.set(x.subarray(i * dim, (i + 1) * dim), c * dim); + } + else { + for (let j = 0; j < dim; j++) + centroids[c * dim + j] = sums[c * dim + j] / len; + } + } + } + const index = new IvfIndex(centroids, nlist, dim, new Int32Array(cache.capacity).fill(-1)); + index.trainedN = total; + for (let t = 0; t < total; t++) { + const r = live[t]; + index.assign[r] = index._search.nearest(matrix, r * dim); + } + index.relist(n); + index.buildSeconds = (nowMs() - started) / 1000; + return index; + } + /** Regroup the rows by cell (a stable counting sort of `assign`). */ + relist(n) { + const assign = this.assign; + const offsets = new Int32Array(this.nlist + 1); + let total = 0; + for (let r = 0; r < n; r++) { + const c = assign[r]; + if (c >= 0) { + offsets[c + 1]++; + total++; + } + } + for (let c = 0; c < this.nlist; c++) + offsets[c + 1] += offsets[c]; + const fill = offsets.slice(0, this.nlist); + const order = new Int32Array(total); + for (let r = 0; r < n; r++) { + const c = assign[r]; + if (c >= 0) + order[fill[c]++] = r; + } + this.order = order; + this.offsets = offsets; + } + /** Give every pending row its nearest cell and regroup. */ + fold(cache) { + if (this.pending.size > 0) { + for (const r of this.pending) { + this.assign[r] = this._search.nearest(cache.matrix, r * this.dim); + } + this.pending.clear(); + } + this.relist(cache.n); + } + notePut(i, capacity) { + if (i >= this.assign.length) { + const grown = new Int32Array(Math.max(capacity, i + 1)).fill(-1); + grown.set(this.assign); + this.assign = grown; + } + this.assign[i] = -1; + this.pending.add(i); + this.churn++; + } + noteDrop(i) { + if (i < this.assign.length) + this.assign[i] = -1; + this.pending.delete(i); + this.churn++; + } + /** Follow the cache's compaction (`live` = the old rows kept, in order). */ + compact(live) { + const kept = live.length; + const moved = new Int32Array(kept); + for (let w = 0; w < kept; w++) + moved[w] = this.assign[live[w]]; + this.assign.fill(-1); + this.assign.set(moved); + if (this.pending.size > 0) { + const old = Array.from(this.pending).sort((a, b) => a - b); + this.pending = new Set(old.map(r => lowerBound(live, r))); + } + this.relist(kept); + } + bytes() { + return this.centroids.byteLength + this.assign.byteLength + this.order.byteLength + this.offsets.byteLength; + } + /** + * Score the rows of the `lists` cells nearest `q`, plus every pending row, + * into `candRows`/`candScores`; returns how many were kept. `code` < 0 means + * any node type; rows scoring under `floor` are left out. + */ + gather(cache, q, qnorm, code, floor, lists) { + const { nlist, dim, centroids } = this; + if (this._chosen.length < lists) { + this._chosen = new Int32Array(lists); + this._chosenScore = new Float64Array(lists); + } + const chosen = this._chosen; + let nChosen = 0; + if (lists >= nlist) { + for (let c = 0; c < nlist; c++) + chosen[c] = c; + nChosen = nlist; + } + else { + const chosenScore = this._chosenScore; + for (let c = 0; c < nlist; c++) { + const s = dot2(centroids, c * dim, q, 0, dim); + if (nChosen === lists && s <= chosenScore[lists - 1]) + continue; + let i = nChosen < lists ? nChosen++ : lists - 1; + while (i > 0 && chosenScore[i - 1] < s) { + chosenScore[i] = chosenScore[i - 1]; + chosen[i] = chosen[i - 1]; + i--; + } + chosenScore[i] = s; + chosen[i] = c; + } + } + const order = this.order; + const offsets = this.offsets; + let bound = this.pending.size; + for (let k = 0; k < nChosen; k++) + bound += offsets[chosen[k] + 1] - offsets[chosen[k]]; + if (this.candRows.length < bound) { + const size = Math.max(bound, this.candRows.length * 2, 256); + this.candRows = new Int32Array(size); + this.candScores = new Float64Array(size); + } + const candRows = this.candRows; + const candScores = this.candScores; + const { matrix, norms, codes } = cache; + const assign = this.assign; + let m = 0; + for (let k = 0; k < nChosen; k++) { + const c = chosen[k]; + for (let p = offsets[c], end = offsets[c + 1]; p < end; p++) { + const r = order[p]; + if (assign[r] < 0) + continue; // deleted or updated since the last regroup + if (code >= 0 && codes[r] !== code) + continue; + const denom = norms[r] * qnorm; + const s = denom > 0 ? dot2(matrix, r * dim, q, 0, dim) / denom : 0; + if (s < floor) + continue; + candRows[m] = r; + candScores[m] = s; + m++; + } + } + for (const r of this.pending) { + if (code >= 0 && codes[r] !== code) + continue; + const denom = norms[r] * qnorm; + const s = denom > 0 ? dot2(matrix, r * dim, q, 0, dim) / denom : 0; + if (s < floor) + continue; + candRows[m] = r; + candScores[m] = s; + m++; + } + return m; + } + /** The `k` best of the first `m` candidates: score descending, row ascending on ties. */ + best(m, k) { + const rows = this.candRows; + const scores = this.candScores; + const size = Math.min(k, m); + // Min-heap of candidate slots whose root is the worst kept one. + const heap = new Int32Array(size); + const worse = (a, b) => scores[a] < scores[b] || (scores[a] === scores[b] && rows[a] > rows[b]); + let count = 0; + for (let i = 0; i < m; i++) { + if (count === size) { + const root = heap[0]; + if (!(scores[i] > scores[root] || (scores[i] === scores[root] && rows[i] < rows[root]))) + continue; + heap[0] = i; + let at = 0; + for (;;) { + const l = 2 * at + 1; + if (l >= size) + break; + const r = l + 1; + const child = r < size && worse(heap[r], heap[l]) ? r : l; + if (!worse(heap[child], heap[at])) + break; + const t = heap[at]; + heap[at] = heap[child]; + heap[child] = t; + at = child; + } + } + else { + let at = count++; + heap[at] = i; + while (at > 0) { + const parent = (at - 1) >> 1; + if (!worse(heap[at], heap[parent])) + break; + const t = heap[at]; + heap[at] = heap[parent]; + heap[parent] = t; + at = parent; + } + } + } + // Drain worst-first into the back of the output. + const out = new Int32Array(count); + for (let o = count - 1; o >= 0; o--) { + out[o] = heap[0]; + const last = heap[o]; + if (o === 0) + break; + heap[0] = last; + let at = 0; + for (;;) { + const l = 2 * at + 1; + if (l >= o) + break; + const r = l + 1; + const child = r < o && worse(heap[r], heap[l]) ? r : l; + if (!worse(heap[child], heap[at])) + break; + const t = heap[at]; + heap[at] = heap[child]; + heap[child] = t; + at = child; + } + } + return out; + } +} +/** Index of the first element of sorted `values` that is >= `target`. */ +function lowerBound(values, target) { + let lo = 0; + let hi = values.length; + while (lo < hi) { + const mid = (lo + hi) >> 1; + if (values[mid] < target) + lo = mid + 1; + else + hi = mid; + } + return lo; +} /** * In-memory embedding storage with brute-force cosine similarity search. + * + * Search is exact by default. For large stores an opt-in approximate index + * (`ann: 'ivf'`, see {@link AnnOptions}) trades a little recall for much lower + * latency; this is built for an application's own knowledge graph, not as a + * vector-database replacement. */ export class EmbeddingStore { constructor(encoder, options = {}) { @@ -496,17 +1325,55 @@ export class EmbeddingStore { this._dimension = null; this._onModelMismatch = 'warn'; this._warnedMismatch = new Set(); + this._ann = null; + this._annNlist = null; + this._annNprobe = null; + this._annMinSize = ANN_MIN_SIZE; + this._annSeed = 0; + this._annCounts = { approximate: 0, exact: 0, widened: 0, builds: 0 }; + this._scan = null; /** Metadata of the most recent imported sidecar (empty before any). */ this.lastImportMeta = {}; - this._encoder = encoder || null; + this._encoder = adoptEncoder(encoder); this._version = options.version ?? null; this.onModelMismatch = options.onModelMismatch ?? 'warn'; + const ann = options.ann ?? null; + if (ann !== null && ann !== 'ivf') { + throw new EmbeddingError(`ann must be null or 'ivf', got ${JSON.stringify(ann)}`); + } + const whole = (label, value, floor) => { + if (value === undefined || value === null) + return null; + if (typeof value !== 'number' || !Number.isInteger(value) || value < floor) { + throw new EmbeddingError(`${label} must be an integer >= ${floor}, got ${value}`); + } + return value; + }; + this._ann = ann; + this._annNlist = whole('annNlist', options.annNlist, 1); + this._annNprobe = whole('annNprobe', options.annNprobe, 1); + this._annMinSize = whole('annMinSize', options.annMinSize, 0) ?? ANN_MIN_SIZE; + this._annSeed = Math.trunc(Number(options.annSeed ?? 0)) || 0; + } + /** The approximate index in use (`'ivf'`), or null when search is exact. */ + get ann() { + return this._ann; + } + /** Cells scanned per query; null means "automatic" (about 5% of the cells). */ + get annNprobe() { + return this._annNprobe; + } + set annNprobe(value) { + if (value !== null && (typeof value !== 'number' || !Number.isInteger(value) || value < 1)) { + throw new EmbeddingError(`annNprobe must be an integer >= 1, got ${value}`); + } + this._annNprobe = value; } get encoder() { return this._encoder; } set encoder(encoder) { - this._encoder = encoder; + this._encoder = adoptEncoder(encoder); } /** Vector length this store holds, or null while it is still empty. */ get dimension() { @@ -573,6 +1440,9 @@ export class EmbeddingStore { else { vector = query; } + return this._checkQuery(vector); + } + _checkQuery(vector) { if (vector.length === 0) { throw new DimensionMismatchError("query vector is empty"); } @@ -673,9 +1543,15 @@ export class EmbeddingStore { } this._checkDimension(vector, "vector"); const key = this._nodeRefToKey(nodeRef); - this._embeddings.set(key, { id: key, nodeRef, vector, text, model: this._modelName() }); + this._set(key, { id: key, nodeRef, vector, text, model: this._modelName() }); return key; } + /** Every write goes through here, so the approximate index's cache stays in step. */ + _set(key, record) { + this._embeddings.set(key, record); + if (this._scan !== null) + this._scan.put(key, record); + } /** Add many embeddings, encoding their texts in one batched call. */ addBatch(entries) { if (entries.length === 0) @@ -701,16 +1577,106 @@ export class EmbeddingStore { }); return entries.length; } + /** + * Encode texts with the store's encoder, awaiting it: an {@link AsyncEncoder} + * (such as HttpEncoder) over the network, any other encoder as `encodeBatch`. + * Nothing is stored. Rejects with NoEncoderError when there is no encoder. + */ + async encodeTextsAsync(texts) { + const encoder = this._encoder; + if (!encoder) { + throw new NoEncoderError("No encoder provided and no vectors given"); + } + const vectors = isAsyncEncoder(encoder) + ? await encoder.encodeBatchAsync(texts) + : encoder.encodeBatch(texts); + if (vectors.length !== texts.length) { + throw new EmbeddingError(`encoder returned ${vectors.length} vectors for ${texts.length} texts`); + } + return vectors; + } + /** + * Like {@link add}, but awaits the encoder, so it works with one that makes + * a network call. On a failure nothing is stored. + */ + async addAsync(nodeRef, text, vector) { + if (vector === undefined) { + vector = (await this.encodeTextsAsync([text]))[0]; + } + return this.add(nodeRef, text, vector); + } + /** + * Like {@link addBatch}, but awaits the encoder: the texts that need + * encoding go out in one `encodeBatchAsync` call (HttpEncoder splits it into + * `batchSize` requests). All entries are stored only once every vector is in. + */ + async addBatchAsync(entries) { + if (entries.length === 0) + return 0; + const needsEncoding = entries + .map((e, i) => (e.length < 3 || e[2] === undefined ? i : -1)) + .filter(i => i >= 0); + const vectorFor = new Map(); + if (needsEncoding.length > 0) { + const encoded = await this.encodeTextsAsync(needsEncoding.map(i => entries[i][1])); + needsEncoding.forEach((slot, i) => vectorFor.set(slot, encoded[i])); + } + entries.forEach((entry, i) => { + this.add(entry[0], entry[1], vectorFor.get(i) ?? entry[2]); + }); + return entries.length; + } + /** + * The vector a query stands for: a vector as it is, a text through the + * encoder (awaited), with the same model-mismatch check as a synchronous + * text query. Used by the `...Async` searches. + */ + async resolveQueryAsync(query) { + if (typeof query !== 'string') + return this._checkQuery(query); + if (!this._encoder) { + throw new NoEncoderError("No encoder provided for text query"); + } + this._checkModelCompat(); + return this._checkQuery((await this.encodeTextsAsync([query]))[0]); + } get(nodeRef) { const key = this._nodeRefToKey(nodeRef); return this._embeddings.get(key) || null; } remove(nodeRef) { const key = this._nodeRefToKey(nodeRef); - return this._embeddings.delete(key); + const removed = this._embeddings.delete(key); + if (removed && this._scan !== null) + this._scan.drop(key); + return removed; } - similaritySearch(query, topK = 10, nodeType, threshold = 0.0) { + /** + * Find the most similar nodes by embedding. + * + * Exact by default. With `ann: 'ivf'`, a bounded `topK` on a store of at + * least `annMinSize` vectors is answered from the approximate index (the + * scores are the same cosine values, but a true neighbour in an unprobed + * cell can be missing); `exact: true` forces the full scan, and has no effect + * on a store without `ann`. `topK: null` always scans everything. + */ + similaritySearch(query, topK = 10, nodeType, threshold = 0.0, exact = false) { const queryVector = this._resolveQuery(query); + if (this._ann !== null) { + if (exact || topK === null || topK <= 0) { + this._annCounts.exact++; + } + else { + const answer = this._annSearch(queryVector, topK, nodeType, threshold); + if (answer !== null) + return answer; + this._annCounts.exact++; + } + } + return this._scanExact(queryVector, topK, nodeType, threshold); + } + /** The brute-force scan every search used before the approximate index existed. */ + _scanExact(queryVector, topK, nodeType, threshold) { const results = []; for (const record of this._embeddings.values()) { if (nodeType && record.nodeRef[0] !== nodeType) @@ -724,6 +1690,305 @@ export class EmbeddingStore { return results.sort((a, b) => b.score - a.score); return selectTopK(results, topK); } + /** {@link similaritySearch} with a text query encoded by awaiting the encoder. */ + async similaritySearchAsync(query, topK = 10, nodeType, threshold = 0.0, exact = false) { + return this.similaritySearch(await this.resolveQueryAsync(query), topK, nodeType, threshold, exact); + } + /** The vector a query stands for (a text goes through the synchronous encoder). */ + resolveQuery(query) { + return this._resolveQuery(query); + } + // -- approximate (IVF) search ------------------------------------------------- + /** Cells for a store of `n` vectors: about 4 * sqrt(n), within [16, 65536]. */ + static _annAutoNlist(n) { + return Math.min(Math.max(roundHalfEven(4 * Math.sqrt(n)), 16), 65536); + } + /** + * The cache holding the index, with the index built or maintained as the + * policy says; null means "answer exactly": `ann` is off, or the store holds + * fewer than `annMinSize` vectors. + * + * Policy: + * - built lazily by the first search (never persisted): k-means on a sample + * of up to 40 x nlist vectors, then every row assigned a cell; + * - new and updated rows are scored exactly until at least + * `max(512, 2%)` of the indexed rows are pending, then folded into cells; + * - deletes tombstone (nothing to do but skip them); + * - once writes + deletes since training exceed 50% of the trained size, + * the index is retrained from the current data, with an automatic `nlist` + * re-chosen for the new size; + * - a cleared store, a dimension change, or a store that shrinks below + * `annMinSize` drops the index with its cache. + */ + _annIndex() { + if (this._ann === null || this._dimension === null) + return null; + const live = this._embeddings.size; + if (live < Math.max(2, this._annMinSize)) { + this._scan = null; + return null; + } + let scan = this._scan; + if (scan === null) { + scan = this._scan = ScanCache.build(this._embeddings.values(), live, this._dimension); + } + let ivf = scan.ivf; + if (ivf !== null && ivf.churn > ANN_RETRAIN_FRACTION * ivf.trainedN) + ivf = scan.ivf = null; + if (ivf === null) { + const nlist = this._annNlist ?? EmbeddingStore._annAutoNlist(live); + scan.ivf = IvfIndex.build(scan, nlist, this._annSeed); + this._annCounts.builds++; + } + else if (ivf.pending.size >= Math.max(ANN_FOLD_MIN, ANN_FOLD_FRACTION * ivf.trainedN)) { + ivf.fold(scan); + } + return scan; + } + _annProbe(ivf) { + const wanted = this._annNprobe ?? Math.max(1, roundHalfEven(0.05 * ivf.nlist)); + return Math.min(wanted, ivf.nlist); + } + /** + * Cells to probe at widening `scale`; null = widened enough to just scan. + * Scale 1 honours `annNprobe` as configured (`annNprobe >= nlist` is the + * exact scan); later steps multiply it by 4 and give up on the index once + * they would touch half the cells, where scanning everything costs about the + * same. + */ + _annLists(ivf, scale) { + const lists = this._annProbe(ivf) * scale; + if (scale === 1) + return lists >= ivf.nlist ? null : lists; + return lists * 2 >= ivf.nlist ? null : lists; + } + /** Load the query into the cache's float32 buffer; null when it has no direction (all zeros). */ + _annQuery(scan, queryVector, nodeType) { + const q = scan.query; + let sum = 0; + for (let j = 0; j < scan.dim; j++) { + const f = Math.fround(queryVector[j]); + q[j] = f; + sum += f * f; + } + const qnorm = Math.sqrt(sum); + if (qnorm === 0) + return null; + let code = -1; + if (nodeType) + code = scan.codeOf.get(nodeType) ?? -2; // -2: no node of that type exists + return { qnorm, code }; + } + /** + * Top-k from the index, or null to fall back to the exact scan. + * + * Filters (`nodeType`, `threshold`) apply to the probed rows. If fewer than + * `topK` survive, the probe is widened 4x and tried again, and past half the + * cells the exact scan answers instead - so a rare type or a high threshold + * costs time rather than silently returning a short list. + */ + _annSearch(queryVector, topK, nodeType, threshold) { + const scan = this._annIndex(); + if (scan === null) + return null; + const ivf = scan.ivf; + const prepared = this._annQuery(scan, queryVector, nodeType); + if (prepared === null) + return null; + if (prepared.code === -2) + return []; + const floor = threshold - ANN_THRESHOLD_SLACK; + for (let scale = 1;; scale *= 4) { + const lists = this._annLists(ivf, scale); + if (lists === null) + return null; + const m = ivf.gather(scan, scan.query, prepared.qnorm, prepared.code, floor, lists); + if (m < topK) + continue; + const best = ivf.best(m, topK); + const results = []; + for (let i = 0; i < best.length; i++) { + const record = this._embeddings.get(scan.keys[ivf.candRows[best[i]]]); + results.push({ nodeRef: record.nodeRef, score: ivf.candScores[best[i]], text: record.text }); + } + this._annCounts.approximate++; + if (scale > 1) + this._annCounts.widened++; + return this.annPolish(results, queryVector, threshold); + } + } + /** + * Re-score approximate results with the exact float64 cosine the full scan + * uses (the index scores float32 copies), drop any that fall under + * `threshold` once exact, and order them best first. + */ + annPolish(results, queryVector, threshold) { + const out = []; + for (const result of results) { + const record = this._embeddings.get(this._nodeRefToKey(result.nodeRef)); + if (!record) + continue; + const score = this._cosineSimilarity(queryVector, record.vector); + if (score >= threshold) + out.push({ nodeRef: result.nodeRef, score, text: result.text }); + } + return out.sort((a, b) => b.score - a.score); + } + /** + * Candidates for a search that has to filter them itself (a graph's + * `nodeFilter`): every row of the probed cells that clears `threshold`, best + * first, lazily, with *approximate* (float32) scores - pass what you keep to + * {@link annPolish}. `scale` is the widening step (1, 4, 16, ...); null asks + * for the exact scan. `approximate` is false when the answer is exact - the + * caller has nothing left to widen to. + */ + annRanked(queryVector, nodeType, threshold, scale) { + if (scale !== null && this._ann !== null) { + const scan = this._annIndex(); + const ivf = scan === null ? null : scan.ivf; + const lists = ivf === null ? null : this._annLists(ivf, scale); + const prepared = scan === null || lists === null ? null : this._annQuery(scan, queryVector, nodeType); + if (scan !== null && ivf !== null && lists !== null && prepared !== null) { + if (prepared.code === -2) + return { ranked: [], approximate: false }; + const m = ivf.gather(scan, scan.query, prepared.qnorm, prepared.code, threshold - ANN_THRESHOLD_SLACK, lists); + this._annCounts.approximate++; + if (scale > 1) + this._annCounts.widened++; + const rows = ivf.candRows.slice(0, m); + const scores = ivf.candScores.slice(0, m); + const slots = new Int32Array(m); + for (let i = 0; i < m; i++) + slots[i] = i; + slots.sort((a, b) => scores[b] - scores[a] || rows[a] - rows[b]); + const keys = scan.keys; + const embeddings = this._embeddings; + const ranked = (function* () { + for (let i = 0; i < m; i++) { + const record = embeddings.get(keys[rows[slots[i]]]); + yield { nodeRef: record.nodeRef, score: scores[slots[i]], text: record.text }; + } + })(); + return { ranked, approximate: true }; + } + } + return { ranked: this._scanExact(queryVector, null, nodeType, threshold), approximate: false }; + } + /** + * Build the approximate index now instead of on the first search; returns + * {@link annStats}. `force` retrains even if an index exists. Does nothing + * (and says so in the stats) while the store is smaller than `annMinSize`. + * + * @throws EmbeddingError when the store was created without `ann` + */ + buildAnnIndex(force = false) { + if (this._ann === null) { + throw new EmbeddingError("this store has no approximate index; create it with ann: 'ivf'"); + } + if (force && this._scan !== null) + this._scan.ivf = null; + const scan = this._annIndex(); + if (scan !== null && scan.ivf !== null && scan.ivf.pending.size > 0) + scan.ivf.fold(scan); + return this.annStats(); + } + /** What the approximate index looks like right now. Never builds anything. */ + annStats() { + const scan = this._scan; + const ivf = scan === null ? null : scan.ivf; + return { + enabled: this._ann !== null, + mode: this._ann, + active: ivf !== null, + nlist: ivf === null ? null : ivf.nlist, + nprobe: ivf === null ? this._annNprobe : this._annProbe(ivf), + trainedN: ivf === null ? null : ivf.trainedN, + pending: ivf === null ? 0 : ivf.pending.size, + churn: ivf === null ? 0 : ivf.churn, + buildSeconds: ivf === null ? null : ivf.buildSeconds, + indexBytes: ivf === null ? 0 : ivf.bytes(), + cacheBytes: scan === null ? 0 : scan.bytes(), + minSize: this._annMinSize, + searches: { ...this._annCounts }, + }; + } + /** + * Recall@k of the approximate index against the exact scan. + * + * Runs every query both ways and compares the *sets* of top-k nodes. + * `queries` is a text, a vector, or a list of either. Use real queries from + * your own workload: recall depends on how the data clusters, and a score + * measured on unrelated queries or data does not transfer. Raise `annNprobe` + * until the recall you need holds. + * + * Without an index in use (`ann` off, a store under `annMinSize`, or + * `annNprobe >= nlist`) both answers are the exact scan, recall is 1 and + * `approximate` is false. + */ + recallCheck(queries, k = 10, nodeType, threshold = 0.0) { + if (!Number.isInteger(k) || k <= 0) + throw new EmbeddingError(`k must be positive, got ${k}`); + const vectors = EmbeddingStore._queryList(queries).map(q => this._resolveQuery(q)); + return this._recall(vectors, k, nodeType, threshold); + } + /** {@link recallCheck} with text queries encoded by awaiting the encoder. */ + async recallCheckAsync(queries, k = 10, nodeType, threshold = 0.0) { + if (!Number.isInteger(k) || k <= 0) + throw new EmbeddingError(`k must be positive, got ${k}`); + const vectors = []; + for (const query of EmbeddingStore._queryList(queries)) + vectors.push(await this.resolveQueryAsync(query)); + return this._recall(vectors, k, nodeType, threshold); + } + /** One query or many, as a list of texts and plain number arrays. */ + static _queryList(queries) { + if (typeof queries === 'string') + return [queries]; + const list = Array.from(queries); + if (list.length > 0 && typeof list[0] === 'number') + return [list]; + return list.map(q => (typeof q === 'string' ? q : Array.from(q))); + } + _recall(vectors, k, nodeType, threshold) { + const recalls = []; + const annTimes = []; + const exactTimes = []; + let used = false; + for (const vector of vectors) { + const before = this._annCounts.approximate; + let started = nowMs(); + const found = this.similaritySearch(vector, k, nodeType, threshold); + annTimes.push(nowMs() - started); + used = used || this._annCounts.approximate > before; + started = nowMs(); + const truth = this.similaritySearch(vector, k, nodeType, threshold, true); + exactTimes.push(nowMs() - started); + if (truth.length > 0) { + const wanted = new Set(truth.map(r => this._nodeRefToKey(r.nodeRef))); + const hits = found.filter(r => wanted.has(this._nodeRefToKey(r.nodeRef))).length; + recalls.push(hits / wanted.size); + } + } + const median = (values) => { + if (values.length === 0) + return 0; + const ordered = [...values].sort((a, b) => a - b); + const mid = ordered.length >> 1; + return ordered.length % 2 ? ordered[mid] : 0.5 * (ordered[mid - 1] + ordered[mid]); + }; + const ivf = this._scan === null ? null : this._scan.ivf; + return { + recall: recalls.length ? recalls.reduce((a, b) => a + b, 0) / recalls.length : 1, + minRecall: recalls.length ? Math.min(...recalls) : 1, + queries: vectors.length, + k, + nlist: ivf === null ? 0 : ivf.nlist, + nprobe: ivf === null ? this._annNprobe ?? 0 : this._annProbe(ivf), + approximate: used, + annMs: median(annTimes), + exactMs: median(exactTimes), + }; + } /** * Cosine score for every embedded node, keyed by `type:id`. * @@ -753,6 +2018,7 @@ export class EmbeddingStore { */ clear(resetDimension = false) { this._embeddings.clear(); + this._scan = null; if (resetDimension) this._dimension = null; } @@ -820,7 +2086,7 @@ export class EmbeddingStore { const ref = [item.type, id]; this._checkDimension(item.vector, `embedding for :${item.type}:${id}`); const key = this._nodeRefToKey(ref); - this._embeddings.set(key, { + this._set(key, { id: key, nodeRef: ref, vector: item.vector, @@ -1049,10 +2315,15 @@ function edgeWeight(edge, weightField) { export class SemanticGraph extends ISONGraph { constructor(options = {}) { super(options.name || "semantic_graph", options.directed ?? true); - this._encoder = options.encoder || null; + this._encoder = adoptEncoder(options.encoder); this._embeddingStore = new EmbeddingStore(this._encoder || undefined, { version: options.version, onModelMismatch: options.onModelMismatch, + ann: options.ann, + annNlist: options.annNlist, + annNprobe: options.annNprobe, + annMinSize: options.annMinSize, + annSeed: options.annSeed, }); this._autoEmbed = options.autoEmbed ?? true; this._embedFields = options.embedFields || ['name', 'description', 'content', 'text']; @@ -1064,8 +2335,8 @@ export class SemanticGraph extends ISONGraph { return this._encoder; } setEncoder(encoder) { - this._encoder = encoder; - this._embeddingStore.encoder = encoder; + this._encoder = adoptEncoder(encoder); + this._embeddingStore.encoder = this._encoder; } /** * Render one property value as embedding text. @@ -1119,12 +2390,53 @@ export class SemanticGraph extends ISONGraph { this.embedNode(node.ref, embedText); return node; } + /** + * Like {@link addNode}, but awaits the encoder, so auto-embedding works with + * one that makes a network call (an HttpEncoder needs no prefetch). + * + * The text is embedded *before* the node is added: if the encoder fails the + * graph is untouched and the call can simply be retried. + */ + async addNodeAsync(nodeType, nodeId, properties = {}) { + let text = ''; + let vector = null; + if (this._autoEmbed && this._encoder) { + text = this.embedTextFor(properties); + if (text) + vector = (await this._embeddingStore.encodeTextsAsync([text]))[0]; + } + const node = super.addNode(nodeType, nodeId, properties); + if (vector !== null) + this._embeddingStore.add(node.ref, text, vector); + return node; + } + /** Like {@link addNodeWithEmbed}, awaiting the encoder (see {@link addNodeAsync}). */ + async addNodeWithEmbedAsync(nodeType, nodeId, properties = {}, embedText) { + if (embedText === undefined) { + return this.addNodeAsync(nodeType, nodeId, properties); + } + const vector = (await this._embeddingStore.encodeTextsAsync([embedText]))[0]; + const node = super.addNode(nodeType, nodeId, properties); + this._embeddingStore.add(node.ref, embedText, vector); + return node; + } /** * Embed a node with given text. */ embedNode(nodeRef, text) { return this._embeddingStore.add(nodeRef, text); } + /** Like {@link embedNode}, awaiting the encoder. */ + embedNodeAsync(nodeRef, text) { + return this._embeddingStore.addAsync(nodeRef, text); + } + /** Like {@link embedNodes}, awaiting the encoder; stores nothing if it fails. */ + async embedNodesAsync(items) { + const pairs = items.filter(([, text]) => Boolean(text)); + if (pairs.length === 0) + return 0; + return this._embeddingStore.addBatchAsync(pairs); + } /** Embed many nodes in a single batched encoder call. */ embedNodes(items) { const pairs = items.filter(([, text]) => Boolean(text)); @@ -1161,6 +2473,38 @@ export class SemanticGraph extends ISONGraph { } return node; } + /** + * Like {@link updateNode}, but awaits the encoder when a re-embed is needed. + * + * The new text is embedded before anything changes, so an encoder failure + * leaves the node's properties and vector exactly as they were. + */ + async updateNodeAsync(nodeType, nodeId, properties) { + const current = this.getNode(nodeType, nodeId); + const reembed = this._autoEmbed && this._embedFields.some(f => f in (properties ?? {})); + let text = ''; + let vector = null; + if (reembed) { + text = this.embedTextFor({ ...current.properties, ...properties }); + if (text && this._encoder) + vector = (await this._embeddingStore.encodeTextsAsync([text]))[0]; + } + const node = super.updateNode(nodeType, nodeId, properties); + if (reembed) { + // Another update may have landed while the encoder was working. + const latest = this.embedTextFor(node.properties); + if (!latest) { + this._embeddingStore.remove(node.ref); + } + else if (this._encoder) { + if (vector === null || latest !== text) { + vector = (await this._embeddingStore.encodeTextsAsync([latest]))[0]; + } + this._embeddingStore.add(node.ref, latest, vector); + } + } + return node; + } /** * Remove node and its embedding. */ @@ -1208,15 +2552,39 @@ export class SemanticGraph extends ISONGraph { * With a gate the store cannot truncate first: the best k *unfiltered* * matches may all be filtered out, so every match is scored, filtered, and * only then cut to k. + * + * With the store's approximate index the candidates are the probed cells + * only. If fewer than k of them pass the gate, the probe is widened (4x each + * step) and the search retried, ending in the exact scan, so a selective + * filter costs time rather than results. */ - _seedSearch(query, topK, nodeType, threshold, gate) { + _seedSearch(query, topK, nodeType, threshold, gate, exact = false) { + const store = this._embeddingStore; if (gate === null) { - return this._embeddingStore.similaritySearch(query, topK, nodeType, threshold); + return store.similaritySearch(query, topK, nodeType, threshold, exact); + } + if (store.ann === null || exact || topK === null || topK <= 0) { + const results = store + .similaritySearch(query, null, nodeType, threshold) + .filter(r => gate(r.nodeRef)); + return topK === null ? results : results.slice(0, Math.max(topK, 0)); + } + const queryVector = store.resolveQuery(query); + for (let scale = 1;; scale *= 4) { + const { ranked, approximate } = store.annRanked(queryVector, nodeType, threshold, scale); + const results = []; + for (const result of ranked) { + if (gate(result.nodeRef)) { + results.push(result); + if (results.length >= topK) + break; + } + } + if (!approximate) + return results; + if (results.length >= topK) + return store.annPolish(results, queryVector, threshold); } - const results = this._embeddingStore - .similaritySearch(query, null, nodeType, threshold) - .filter(r => gate(r.nodeRef)); - return topK === null ? results : results.slice(0, Math.max(topK, 0)); } /** * Find similar nodes by embedding. @@ -1227,10 +2595,25 @@ export class SemanticGraph extends ISONGraph { * the filter. * @param nodeTypes Keep only nodes whose type is in this list (a bare string * means one type). Combines with `nodeType` and `nodeFilter` by AND. + * @param exact Force the full scan even when the graph was created with an + * approximate index (`ann`); no effect otherwise. */ - similaritySearch(query, topK = 10, nodeType, threshold = 0.0, nodeFilter, nodeTypes) { + similaritySearch(query, topK = 10, nodeType, threshold = 0.0, nodeFilter, nodeTypes, exact = false) { const gate = this._nodeGate(nodeFilter, nodeTypes); - return this._seedSearch(query, topK, nodeType, threshold, gate); + return this._seedSearch(query, topK, nodeType, threshold, gate, exact); + } + /** Recall@k of the approximate index against the exact scan (see {@link EmbeddingStore.recallCheck}). */ + recallCheck(queries, k = 10, nodeType, threshold = 0.0) { + return this._embeddingStore.recallCheck(queries, k, nodeType, threshold); + } + /** {@link recallCheck} with text queries encoded by awaiting the encoder. */ + recallCheckAsync(queries, k = 10, nodeType, threshold = 0.0) { + return this._embeddingStore.recallCheckAsync(queries, k, nodeType, threshold); + } + /** {@link similaritySearch} with a text query encoded by awaiting the encoder. */ + async similaritySearchAsync(query, topK = 10, nodeType, threshold = 0.0, nodeFilter, nodeTypes, exact = false) { + const vector = await this._embeddingStore.resolveQueryAsync(query); + return this.similaritySearch(vector, topK, nodeType, threshold, nodeFilter, nodeTypes, exact); } /** * Find nodes semantically closest to one already in the graph. @@ -1599,6 +2982,18 @@ export class SemanticGraph extends ISONGraph { } return null; } + /** {@link semanticMultiHop} with a text query encoded by awaiting the encoder (options form). */ + async semanticMultiHopAsync(query, options = {}) { + return this.semanticMultiHop(await this._embeddingStore.resolveQueryAsync(query), options); + } + /** {@link semanticPath} with a text query encoded by awaiting the encoder (options form). */ + async semanticPathAsync(query, targetRef, options = {}) { + return this.semanticPath(await this._embeddingStore.resolveQueryAsync(query), targetRef, options); + } + /** {@link semanticSubgraph} with a text query encoded by awaiting the encoder. */ + async semanticSubgraphAsync(query, options = {}) { + return this.semanticSubgraph(await this._embeddingStore.resolveQueryAsync(query), options); + } /** * Extract the slice of the graph a query is actually about. * @@ -1681,6 +3076,19 @@ export class SemanticGraph extends ISONGraph { } return this.embedNodes(batch); } + /** {@link embedAllNodes} awaiting the encoder: all the texts, then the stores. */ + async embedAllNodesAsync(textFn, skipExisting = false) { + const fn = textFn || ((node) => this.embedTextFor(node.properties) || `${node.type}:${node.id}`); + const batch = []; + for (const node of this.nodes()) { + if (skipExisting && this._embeddingStore.get(node.ref)) + continue; + const text = fn(node); + if (text) + batch.push([node.ref, text]); + } + return this.embedNodesAsync(batch); + } /** * Nodes whose stored vector was not made by the current encoder. * @@ -1737,6 +3145,45 @@ export class SemanticGraph extends ISONGraph { } return count; } + /** + * {@link reembedStale} awaiting the encoder. + * + * Every stale text is encoded (`batchSize` per encoder call) before anything + * is written, so a failure partway leaves the store as it was. The new + * vectors' length decides whether the pinned dimension changes, which also + * works for an encoder that has not learned its dimension yet. + */ + async reembedStaleAsync(includeUnknown = false, batchSize = 256) { + const stale = this.staleEmbeddings(includeUnknown); + if (stale.length === 0) + return 0; + const store = this._embeddingStore; + const pairs = []; + for (const ref of stale) { + const record = store.get(ref); + if (!record) + continue; + let text = record.text; + if (!text) { + const node = this._resolveNode(ref); + text = (node ? this.embedTextFor(node.properties) : '') || `${ref[0]}:${ref[1]}`; + } + pairs.push([ref, text]); + } + if (!this._encoder) + throw new NoEncoderError("No encoder provided to re-embed with"); + const step = Math.max(1, Math.floor(batchSize)); + const vectors = []; + for (let start = 0; start < pairs.length; start += step) { + const chunk = pairs.slice(start, start + step); + vectors.push(...await store.encodeTextsAsync(chunk.map(([, text]) => text))); + } + if (vectors.length > 0 && store.dimension !== null && store.dimension !== vectors[0].length) { + store.clear(true); + } + pairs.forEach(([ref, text], i) => store.add(ref, text, vectors[i])); + return pairs.length; + } /** * Build a SemanticGraph from a plain ISONGraph (for example one parsed with * ISONGraph.fromIson), copying its nodes and edges. diff --git a/isongraph-vector-js/src/node.js b/isongraph-vector-js/src/node.js index 1f3b58b..e4367fa 100644 --- a/isongraph-vector-js/src/node.js +++ b/isongraph-vector-js/src/node.js @@ -170,6 +170,7 @@ export function loadGraph(path, options = {}) { const { format = 'auto', embeddings = true, encoder, autoEmbed, embedFields, version, onModelMismatch, onMismatch = 'error', + ann, annNlist, annNprobe, annMinSize, annSeed, } = options; if (onMismatch !== 'error' && onMismatch !== 'warn' && onMismatch !== 'reembed') { throw new EmbeddingError(`onMismatch must be 'error', 'warn' or 'reembed', got ${JSON.stringify(onMismatch)}`); @@ -189,6 +190,11 @@ export function loadGraph(path, options = {}) { embedFields, version, onModelMismatch, + ann, + annNlist, + annNprobe, + annMinSize, + annSeed, }); for (const node of base.nodes()) { ISONGraph.prototype.addNode.call(graph, node.type, node.id, node.properties); diff --git a/isongraph-vector-rs/DESIGN.md b/isongraph-vector-rs/DESIGN.md new file mode 100644 index 0000000..ae38be3 --- /dev/null +++ b/isongraph-vector-rs/DESIGN.md @@ -0,0 +1,597 @@ +# Search engine design (reference implementation) + +This is the specification the Rust crate implements and the C++, C#, Python and +JS ports should follow. Code: `src/table.rs` (data), `src/ann.rs` (scores, +search, index), `src/pool.rs` (threads), `src/shared.rs` (reader/writer handle). + +## 1. Data layout: one copy of every vector + +A **table** holds the whole store. A record *is a slot number*; nothing else +owns the floats. + +| part | shape | notes | +|---|---|---| +| matrix | chunks of `CHUNK_ROWS = 1024` rows, `dim` f32 each, row-major | chunked so growth never copies stored data and a scan block never straddles chunks. Chunk 0 grows like a vector; later chunks are allocated whole. | +| `norms[slot]` | f32 | Euclidean norm; **NaN = tombstone** (a NaN score is dropped by the collector, so scans need no liveness branch) | +| `codes[slot]` | u32 | node type as an index into a type-name table (never shrinks) | +| `meta[slot]` | optional {key `"type:id"`, text, model} | `None` = tombstone. Model names are interned. | +| `index` | hash map key -> slot | the only lookup structure; the key string is shared with `meta` | + +Slots are in arrival order. An update overwrites its slot in place; an insert +takes `n`; a delete tombstones (`norm = NaN`, `meta = None`, key removed). +When more than 25% of slots are dead, **compaction** moves live rows down, +preserving order, and the index follows (section 5). Equal scores are ordered by +slot, so answers are reproducible for any thread count. +Measured: 1M x 384 = 1.12x the raw vector bytes including the IVF index. + +## 2. One definition of a score + +`score = dot(q, row) / (norm(row) * norm(q))` (0 if the product is 0). `dot` has +**one fixed summation order**: one 8-lane accumulator walked in ascending +dimension (`acc[l] += q[i+l]*row[i+l]`, fused multiply-add where the CPU has it), +reduced as `t[l] = acc[l] + acc[l+4]`, `s = (t0 + t2) + (t1 + t3)`, then the +`dim % 8` tail added one element at a time. Every kernel (1 query x 4 rows, 4 +queries x 2 rows, k-means 2 points x 4 centroids, a lone dot) computes each +(query, row) pair in this order; they differ only in how many pairs they hold in +registers. Consequences: **a batch answer is bit-identical to the one-query +answer**, and does not depend on tiling or thread count. A port may pick another +order, but must use it everywhere. + +## 3. Search paths + +* **Exact, one query**: blocks of 1024 rows claimed from an atomic counter by up + to `min(pool, 8, n*dim / 2^21)` participants; each keeps a bounded heap + (`TopK`: min-heap, root = worst kept, `bar` = its score so most rows are + rejected by one compare; ordering `(score desc, slot asc)`); heaps merge. +* **Exact, batch**: tasks = (block of rows) x (tile of `BATCH_TILE = 16` + queries), block-major so threads share a block in cache. Inner kernel: 4 + queries x 2 rows (8 accumulators, 6 loads per 8 FMAs). Per-thread heaps per + query, merged at the end. Big batches run in passes of `BATCH_CHUNK = 4096`. +* **IVF, one query**: rank `nlist` centroids by dot, take the `nprobe` best + (cell ids ascending), score those cells' rows and the pending rows with the + 1 x 4 gather kernel. Fewer than `k` survivors (filters) -> widen `nprobe` x4, + ending in the exact scan. +* **IVF, batch (cell-major)**: choose cells per query, invert to "queries that + probe cell c", score each probed cell **once** against all its queries with the + 4 x 2 kernel. Exactly the rows the single path scores, hence the same answer; + short queries are re-run on the single path. +* A type filter is a compare of `codes[slot]`; a threshold is the initial `bar`. + +## 4. Concurrency model + +* **Reads are lock-free.** A search borrows the table immutably and touches no + shared mutable state: counters are relaxed atomics, scratch is per call. Any + number run at once. The only lock on the read path is a shared read lock on the + lazily built index (taken for write only by a build or retrain). +* **Writes are exclusive** (`&mut self` in Rust). A search therefore sees the + store before a write or after it, never between: *no torn reads*, by + construction, not by timing. +* **Mixed readers/writers**: `SharedEmbeddingStore` = reader/writer lock. + Searches hold the read lock for their scan (ms); a write holds the write lock + for one row copy, a few index updates and (every 256th) one bounded fold + (~1-3 ms at 1M x 384). Encoding happens before the lock is taken. + *Rejected*: copy-on-write snapshots. Per-write copying of GBs is impossible; + chunk-level sharing saves readers nothing the lock does not, and is much + harder to port. +* **Pool**: one persistent set of workers (`SearchPool`, default = logical CPUs, + override `ISONGRAPH_VECTOR_THREADS` or `set_search_pool`). `run(n, f)` runs `f` + on the caller plus up to `n-1` idle workers; `f` must be work-stealing (claim + units from a shared counter), so a call completes even if no worker is free. + Callers share the pool without deadlock. `set_scan_threads(t)` caps one call + (0 = auto, 1 = never touch the pool). + +## 5. Write path + +* **put(key, type, vector)**: copy into slot (existing key -> overwrite; else + append), set norm/code/meta/index; if an index exists: detach the slot from its + cell or pending list (O(1), see below), push it on `pending`; **if + `pending.len() == FOLD_BATCH (256)`: fold**. +* **fold**: gather the pending rows, find each one's nearest centroid + (`nlist x dim` MACs per row, parallel in blocks of 16-64 points), append the + slot to that cell's list. Cost ~6-8 us/row at nlist 4000, dim 384 on 32 threads; + a pause of ~1.5 ms every 256 writes. Until folded a row is scored exactly by + every query, so a search never waits for, and is never wrong because of, a fold. +* **remove**: tombstone; detach from its cell/pending list (swap-remove using + `pos[slot]`, fixing the moved element's `pos`); `churn++`. Drop the index if + live rows fall under `min_size`. +* **Index state**: `centroids[nlist][dim]`, `assign[slot]` (cell, `PENDING`, or + none), `pos[slot]` (position in its list), `lists[cell]`, `pending`. Lists hold + exactly the live folded rows: no stale entries. +* **Compaction** (dead > 25% of slots, or explicit): O(n) row moves, order + preserving; the index remaps `assign`, rebuilds lists, remaps `pending`. + This is the one unbounded write section. +* **Retrain**: `churn > 0.5 * trained_n` -> the next search that needs the index + rebuilds it under the write lock (k-means on `min(40*nlist, n)` live rows, 8 + iterations, seeded; `nlist = 4*sqrt(n)` in [16, 65536]). Build is also + `build_ann_index()`; call it off the hot path. Never persisted. + +## 6. Snapshot semantics + +A search observes the state after the last write that completed before it +started (and, with `SharedEmbeddingStore`, possibly later writes that complete +while it waits for the lock, but none partially). A batch observes **one** state +for all its queries. Results never contain a deleted row, a row twice, or a row +whose score is not the cosine of one stored version of it. + +## 7. Invariants (checked by `verify_invariants()` and the tests) + +1. `norms`, `codes`, `meta` have `n` entries; matrix has `n*dim` floats, all + chunks but the last full. +2. `dead` = count of `meta == None`; those and only those have NaN norm. +3. `index` has one entry per live slot, and `meta[index[k]].key == k`. +4. Every key is `type + ":" + id` for the slot's type code. +5. Per-model counts equal the rows carrying each model. +6. Index: `assign.len() == n`; a live slot is in exactly one list (cell or + pending) at `pos[slot]`; dead slots are in none; `pending.len() < 256` between + writes. +7. Batch == single, bit for bit (same ids, scores, order). + +## 8. Knobs and defaults + +| knob | default | meaning | +|---|---|---| +| `CHUNK_ROWS` | 1024 | rows per matrix chunk (= scan block) | +| `FOLD_BATCH` | 256 | pending rows that trigger a fold | +| retrain threshold | 0.5 x trained size | churn before the next search retrains | +| compaction threshold | 25% dead | | +| `AnnConfig.nlist / nprobe / min_size / seed` | auto (4 sqrt n) / 5% of nlist / 50,000 / 0 | | +| `set_scan_threads` | 0 | per-call cap; auto = 8 for one scan, whole pool for batch/build | +| pool size | logical CPUs | `ISONGRAPH_VECTOR_THREADS`, `SearchPool::new(n)` | +| `BATCH_TILE` / `BATCH_CHUNK` | 16 / 4096 queries | | + +## 9. Portable vs language-specific + +*Portable (copy exactly)*: chunked matrix + parallel arrays, tombstone-by-NaN, +slot order and tie-break, the score definition, the pending/fold policy with +`FOLD_BATCH`, `pos`-based O(1) list removal, compaction, retrain rule, +cell-major batch, the invariants. +*Language-specific*: SIMD (Rust: AVX2+FMA intrinsics picked at run time, portable +fallback with the same order; C++: intrinsics/`#pragma omp simd`; C#: +`System.Numerics.Vector<T>` or `Avx2`; Python: numpy/BLAS with the batch as a +matrix product, bit-identity then holds per kernel, not across them; JS: +`Float32Array` loops or WASM SIMD); the lock (Rust `&mut`/`&` + `RwLock`; C++ +`shared_mutex`; C# `ReaderWriterLockSlim`; Python a lock around writes with +searches in numpy outside the GIL; JS single-threaded, workers need +`SharedArrayBuffer`); the pool (Rust hand-written, one `unsafe` lifetime erasure; +others use their standard pool); hash map and heap (any). + +## 10. Int8 scalar-quantised scan with exact rerank (opt-in) + +Code: `src/quant.rs` (child module of `ann`), hooks in `ann.rs`, storage in +`table.rs`. Off by default; `ScanPrecision::Int8 { rerank_factor }` via +`EmbeddingStore::set_scan_precision` (default factor `DEFAULT_RERANK_FACTOR = 4`). +With it off nothing below exists (no memory, no code path). + +### 10.1 The copy + +Beside every f32 row the table keeps `dim` int8 codes and one f32 scale, slot for +slot, in chunks that mirror the f32 chunks (`CHUNK_ROWS` rows). It is **derived +data**: never persisted, never in a sidecar or digest, rebuilt in one pass by +`set_scan_precision`. Cost: `dim + 4` bytes per row = 0.25x the f32 matrix +(+4 bytes/row). + +Quantisation (the same function for rows and queries, so a port must use one +definition everywhere): + +``` +m = max over i of |x_i| (a NaN component is ignored) +if m == 0 or m is not finite: scale = 0, every code = 0 +else: inv = 127 / m (f32); code_i = clamp(round_half_away(x_i * inv), -127, 127) + scale = m / 127 (f32) +``` +Symmetric, no zero point, **-128 is never produced** (the AVX2 kernel relies on +it). A zero scale makes the row's approximate dot 0, like its exact one. + +Maintenance (all inside the table, so it cannot go stale): a put overwrites or +appends the codes with the f32 row; a delete leaves them (the NaN norm already +drops the row); compaction moves codes and scales with the rows; `clear` empties +them. Invariant 8 (checked by `verify_invariants`): for **every** slot, tombstones +included, codes and scale equal `quantize(f32 row)` bit for bit. + +### 10.2 The search + +1. Quantise the query. A query with no direction (zero or non-finite norm) is + answered by the f32 scan. +2. First pass over exactly the rows the f32 path would score (all rows of the + blocks, or the probed cells plus the pending rows), with the type filter: + `d = sum_i q_i * r_i` as an **integer** (exact in i32 for any realistic `dim`: + 384 * 127 * 127 < 2^23), then + `approx = cosine(float(d) * row_scale * query_scale, row_norm, query_norm)` + where `cosine` is the section-2 function (zero denominator -> 0; a tombstone's + NaN norm -> NaN -> dropped by the collector). Keep the best `kc = k * + rerank_factor` in the section-3 heap (order: score desc, slot asc), with the + bar starting at `threshold - 0.05` (so quantisation noise cannot lose a row + that passes the exact threshold). +3. **Exact rerank**: score those `kc` slots with the exact f32 kernel (section 2, + same summation order as the f32 scan), apply the exact threshold, keep the best + `k`. Returned scores are therefore exactly the cosines the f32 scan returns; + only *which* rows reach the rerank is approximate. +4. If `kc >= rows the first pass would score` (live rows for a full scan; the + probed rows for a cell scan), int8 cannot skip anything: answer in f32. The + graph's unbounded filtered search (`k = usize::MAX`) therefore stays f32. + +Because step 2 is integer arithmetic (associative: lane count, tiling, kernel, +thread count do not matter) and selection is by the total order (score, slot), the +candidate set of a query is a function of that query and the store only. So **batch +== single bit for bit still holds**, and the answer does not depend on the thread +count. The exact batch tiles (block of rows) x (16 queries) and runs the same +per-row first pass; the IVF batch is cell-major as in section 3 (each probed +cell's int8 rows are scored once against all queries that probe it, into +per-query `kc` heaps); a query that comes up short or whose probe holds `<= kc` +rows is re-run on the single-query path. + +Kernel (Rust): AVX2 `sign(r, q)` -> `maddubs(|q|, .)` -> `madd(., 1)` -> add: 32 +products per 4 instructions; a pair sum is at most 2*127*127 < 2^15 so the 16-bit +step cannot saturate. The `unsafe` block sits behind an `assert!` that every +operand has `dim` bytes; a portable loop is the fallback and the specification +(they are tested equal on every dimension 1..1000 sampled, including +-127 +extremes). A port may use any integer kernel (NEON `sdot`, VNNI): the result is +identical by construction. + +### 10.3 Reporting recall + +`int8_recall_report(queries, k, node_type, threshold, rerank_factors)` runs the +full scan in f32 and, per factor, in int8 + rerank, and returns the mean and worst +per-query overlap of the two top-k sets, whether every returned score is +bit-identical to the exact cosine of its row, whether int8 applied (`k * factor < +live`), and median times. It measures full scans; an IVF probe adds its own +approximation on top (measure it with `recall_check`; the int8 error is the same +mechanism on a smaller candidate set). + +### 10.4 Measured (indicative; this machine was shared with another agent's work) + +Clustered synthetic data, 384-d, k = 10, rerank factor 4. Timings are the fastest +of 180 searches (5 batches): medians swung 3x run to run on this machine, minimums +were stable to ~5%. + +| 1M x 384 | f32 | int8 | | +|---|---|---|---| +| exact scan, 1 thread | 155 ms | 43 ms | 3.6x | +| exact scan, auto threads | 31.9 ms | 11.2 ms | 2.9x | +| IVF (nlist 4000, nprobe 200), 1 thread | 10.2 ms | 2.9 ms | 3.5x | +| IVF, auto threads | 2.87 ms | 1.75 ms | 1.6x | +| exact batch of 128, auto / 1 thread | 309 / 3202 ms | 240 / 2947 ms | 1.3x / 1.1x | +| IVF batch of 128, auto / 1 thread | 75 / 502 ms | 69 / 431 ms | 1.1x / 1.2x | + +100k x 384: exact 1 thread 5.7 -> 1.85 ms (3.1x), auto threads 3.3 -> 0.58 ms +(5.7x), batch 1.0-1.1x. Recall@10 against the f32 scan (128 queries): factor 1 +0.970 (1M) / 0.976 (100k), factor >= 2 **1.000**, worst single query 1.00 from +factor 2. Memory (working set, 1M x 384): 1658 -> 2033 MiB, **+375 MiB (+23%)**; +the copy itself is 370 MiB against a 1466 MiB matrix. + +Honest reading: int8 buys **single-query latency** (2-4x on exact scans, 1.6-3.5x on +IVF cells) because a scan is memory-bandwidth bound and the copy is 4x smaller. It +buys **nothing for batch throughput** (1.0-1.3x): the f32 batch kernel (4 queries x +2 rows in registers) is compute-bound, and an AVX2 int8 dot does the same number of +MACs per instruction as an f32 FMA; only VNNI (`vpdpbusd`, not assumed here) would +change that. It costs +23% RSS at 384-d. Recommendation: keep it as an opt-in for +latency-sensitive single-query serving of large stores; do not enable it for +batch/offline workloads; the recall figure is for synthetic clustered data and +must be re-measured with `int8_recall_report` on real embeddings before relying on +factor 4 (use a larger factor if it shows < 0.99). + +Ports: implement 10.1 and 10.2 exactly (including the quantisation rounding rule +and the `0.05` slack) and the invariant; kernels and threading are free. + +## 11. Binary snapshot (optional, beside the JSON sidecar) + +Code: `src/snapshot.rs`; glue in `SemanticGraph::{save_with_snapshot, load_with_report}`. + +Evidence for it: at 1M x 384 the JSON sidecar is 4.5 GB and takes 30-50 s to write +and 20-30 s to read here; a snapshot of the same store is 1.5 GB and +saves in ~3.5 s / loads in ~4.3 s. The JSON sidecar is **unchanged and remains the +portable, authoritative format**; the snapshot is a cache of it. Graph files, the +sidecar bytes and the graph digest are byte-identical with or without a snapshot. + +### 11.1 Files and protocol + +* Path: `<graph file>.embeddings.snap` (`.embeddings.json` is the sidecar). +* **Save with snapshot**: write the graph file, then the JSON sidecar (both atomic, + as before), then the snapshot: temp file in the same directory, flush + fsync, + rename over the old one. A failure at any point leaves the previous file + untouched and removes the temp file. If the snapshot cannot be written the pair + is still saved, a stale `.snap` is removed, and the error is returned. A save + **without** snapshot deletes any existing `.snap` first (the sidecar is about to + change). +* **Load**: read the graph; if a `.snap` exists, take the cheap identity checks + (below), and only then stream the body into a *staging* store, verify the + trailer checksum, and install the staging store; **any** failure (missing file, + bad magic/version/flags, identity mismatch, truncation, malformed body, + duplicate key, checksum) leaves the store untouched and falls back to the JSON + sidecar, exactly as without a snapshot (its own digest check and + `GraphMismatch` policy apply). The load reports which path ran + (`LoadReport { source: Snapshot | Json | NoEmbeddings, snapshot_skipped: + Option<reason>, embeddings, seconds }`). +* **Identity** (all required): `graph_digest` in the header equals the digest of the + graph file read now (same `GRAPH_DIGEST_BASIS`; no Python-view leniency), and + the header's sidecar identity equals the identity of the sidecar file on disk. + A snapshot is never used when the sidecar is missing. +* Sidecar identity = `len` (u64) and `fingerprint` = SHA-256 over: `len` as u64 LE, + the first 65,536 bytes, the last 65,536 bytes, and for i in 1..=64 the 4,096 + bytes at offset `floor(len * i / 65)` (each clipped to the file end). Any + serializer rewriting the sidecar changes it; **limit**: an in-place edit that + keeps the length and touches no sampled byte is not detected. Tools that rewrite + the sidecar must delete the `.snap` (this crate's `save` does). + +### 11.2 Byte layout, version 1 (all integers little endian) + +``` +header + 0 8 magic "ISONVSNP" + 8 4 version u32 = 1 (reader rejects any other) + 12 4 flags u32 bit 0: sidecar identity present; other bits must be 0 + 16 4 dim u32 must be > 0 when count > 0 + 20 4 block_rows u32 (1..=2^20) writer uses 1024; reader honours the file's + 24 8 count u64 live rows (<= 2^32-1) + 32 . graph_digest opt string + graph_digest_basis opt string ("nodes-edges-json-v1" when a digest is given) + encoder_model opt string + encoder_version opt string + [if flags bit 0] sidecar_len u64, sidecar_fingerprint 32 bytes + n_models u32, then n_models x string (model names, order of first use) +block * ceil(count / block_rows) R = min(block_rows, rows left) + R x row metadata: type string, id string, text string, model_index u32 + R x dim x f32 row-major, in the same row order +trailer 32 bytes SHA-256 of every byte before it (from offset 0) +``` +`string` = `u32 length` + that many UTF-8 bytes. `opt string` = the same, with length +`0xFFFFFFFF` meaning absent. Rows are in the **sidecar's order**: live rows only, +sorted by node type then id (byte order), so loading a snapshot yields the same +slots, tombstone-free, as loading the JSON (equal score ties resolve identically). +`key = type ":" id`. A reader must reject: a length past the end of the file, a +model index >= n_models, a repeated key, `count * dim * 4` larger than the file, +bytes between the last block and the trailer or after it, a trailer mismatch. +The reader sets `dimension` from the first row (as the JSON import does), and the +import metadata (encoder model/version, digest, basis) from the header. + +Smallest example (1 row: type `t`, id `a`, text `x`, model `provided`, vector +`[1.0, -2.0]`, no digest, no sidecar), 123 bytes: +``` +49 53 4f 4e 56 53 4e 50 01 00 00 00 00 00 00 00 magic, version 1, flags 0 +02 00 00 00 00 04 00 00 01 00 00 00 00 00 00 00 dim 2, block_rows 1024, count 1 +ff ff ff ff x4 four absent strings +01 00 00 00 08 00 00 00 "provided" 1 model +01 00 00 00 "t" 01 00 00 00 "a" 01 00 00 00 "x" 00 00 00 00 row metadata, model 0 +00 00 80 3f 00 00 00 c0 1.0, -2.0 +<32-byte SHA-256 of the 91 bytes above> +``` +(A port that has no float byte order issue reads/writes f32 as IEEE-754 LE.) + +### 11.3 What is deliberately not in it + +* **IVF state.** The index is rebuilt (lazily, or by `build_ann_index`). A rebuild is + deterministic for a given store and seed, but it is *not* the saved store's index: + the saved index depends on the slot order and fold history of the live store, + while a canonical-order file loads into a different slot order. Reproducing the + saved index's answers after a round trip cannot be proved without storing the + slot permutation and the pending list and re-validating them, so it is not + attempted: search results after load equal those of a JSON load, and an index + built after load equals one built after a JSON load. +* **The int8 copy** (derived; section 10). +* Warnings, the pool, per-store settings. + +### 11.4 Measured (indicative) + +| 384-d | rows | JSON save / size / load | snapshot save / size / load | +|---|---|---|---| +| 100k | 100,000 | 1.7 s / 451 MiB / 2.5 s | 0.21 s / 150 MiB / 0.34 s | +| 1M | 1,000,000 | 49.8 s / 4513 MiB / 30.3 s | 3.55 s / 1503 MiB / 4.33 s | + +The snapshot is 1.03x the raw f32 matrix (1465 MiB at 1M), 3.0x smaller than the +JSON, loads 7x faster and saves 14x faster. It is **not** mmap-class: faiss and +hnswlib save in 0.06-0.2 s because they write one buffer; this loader reads, +checksums (SHA-256 ~2 GB/s here, ~0.7 s of the 4.3 s), copies each row into the +chunked matrix, computes its norm and builds the key map, which is most of the +time. Making load mmap-fast would need a layout that is the in-memory layout +(aligned chunks, no per-row metadata interleaved), a different trade-off than the +portability chosen here. + +Ports: write `.snap` byte-for-byte as in 11.2 or skip the feature; a port that does +not know the file simply loads the JSON. A port reading a `.snap` must apply every +rejection rule above and fall back to the JSON on any failure. + +## 12. HNSW (opt-in, beside IVF) + +`AnnConfig.mode = Hnsw` (`AnnConfig::hnsw()`) replaces IVF as the approximate +index; exact search stays the default and IVF is untouched. Code: +`src/hnsw.rs`. Like IVF it is built lazily by the first search that wants it (or +`build_ann_index`), is **never persisted** (a loaded store rebuilds it; the graph +file, the JSON sidecar and the graph digest are byte-identical with it on or +off), and is used only for a bounded `top_k` on a store of at least `min_size` +live rows. `exact`, `recall_check`, `set_scan_threads` and the pool mean what +they mean for IVF. Why: at equal recall@10 of about 0.95 on real text-384 +vectors, IVF answers a single query about 4x (100k) to 10x (1M) slower than a +graph walk, because a graph touches far fewer rows per query than the probed +cells do. + +### 12.1 Parameters + +| knob | default | meaning | +|---|---|---| +| `hnsw.m` | 16 | links per node on layers >= 1; layer 0 holds `2m` (2..64) | +| `hnsw.ef_construction` | 200 | candidates while linking a row (used as `max(efc, m)`) | +| `hnsw.ef_search` | auto | candidates per query, `max(ef, k)`; auto by live rows: 64 (<10k), 96 (<200k), 128 (<600k), 192 | +| `seed` | 0 | seeds the level generator (shared with IVF's k-means seed) | +| `min_size` | 50,000 | same threshold as IVF: below it, or `top_k >= live`, search is exact | + +`ef_search` can be changed without a rebuild (`set_ann_ef_search`) or for one +call (`similarity_search_vector_ef`). + +### 12.2 Layout: links only, no second copy of any vector + +Scores are computed from the one stored matrix with the section 2 kernels +(1 query x 4 rows, then 1 x 1), so a hit's score is **bit-identical** to the +exact scan's, and ties order by slot as everywhere else. The graph holds, per +table slot: + +* `level[slot]` (u8; 255 = not in the graph), +* layer 0: one block of `2m + 1` u32 (link count, then links), in chunks of + `CHUNK_ROWS` nodes so growth never copies, +* `upper_at[slot]` (u32) into an arena holding, for a node of level `L`, `L` + consecutive blocks of `m + 1` u32 for layers `1..L`, +* the entry slot and its level. + +That is `4(2m+1) + 5` bytes plus about `4(m+1)/m` of upper layers per node: +**146 B/node at m = 16**, 9.5% of the raw 384-d vectors; the whole store +(matrix, metadata, graph) measured **1.18x** the raw vector bytes at 1M x 384. +Searches also use a per-thread visited bitmap (1 bit per slot, cleared by +touched words, so a query pays for what it visited). + +### 12.3 Level draw and search (portable, integers only) + +* **Level**: `level = 0; while level < 24 and rng.below(m) == 0: level += 1` + with the SplitMix64 generator of the IVF code (`state += 0x9E3779B97F4A7C15`, + then the usual mix; `below(n)` = high 64 bits of `next * n`), seeded `seed XOR + 0x4857535748575357`, one draw sequence in insertion order. So + `P(level >= l) = m^-l`, and no logarithm is needed (ports must not use one). +* **Order** ("better is greater"): higher score, then lower slot. Every heap and + every tie in the graph uses it. +* **Query**: greedy descent (ef = 1) from the entry through layers `top..1`, + then a best-first search of layer 0: candidates in a max-heap, the best `ef` + *admitted* nodes in a min-heap; pop the best candidate; stop when it is worse + than the worst admitted node and `ef` are kept; score its unvisited links + (4 at a time) and push those that beat the worst kept (or any, while fewer + than `ef` are kept). **Admitted** = live (norm not NaN), type code matches the + filter, score >= threshold. Tombstones and non-admitted nodes are routed + through but never kept. NaN scores rank last. The answer is the best `k` kept, + sorted by (score desc, slot asc). +* **Filters** (type, threshold, a graph `node_filter` gate): if fewer than `k` + are kept, retry with `ef x 4`, repeating; give up to the **exact scan** when + `ef >= live` (first try) / `2 ef >= live` (later tries), or when one search + scores more than `max(live / 8, 16 ef)` nodes (a filter so selective that + scanning is cheaper). The graph-gate path asks for the best `ef x scale` + admitted nodes and widens the same way. +* **Batch**: queries are spread over the pool, each runs the single-query path + above, so a batch answer equals the one-query answer bit for bit. (No + cell-major sharing exists for a graph: see 12.8.) +* **Concurrency**: a search takes `&self` plus a shared read lock on the lazily + built graph, like IVF; all scratch is per thread. Writes are `&mut self`. + +### 12.4 Linking (one routine for build, insert and update) + +`link(jobs)`: for each job (a new row with its drawn level, or an existing node +to relink) **plan** against the graph as it is, then **apply** the plans in job +order. + +* **Plan** (read-only, parallel over jobs): descend greedily to the job's level; + on each layer `min(level, top)..0` run the layer search with `ef_construction` + admitting only live nodes other than the job itself, and pass the kept set as + entry points to the next layer down; add the other *new* jobs of the same + batch with level >= the layer as extra candidates (scored by brute force; this + is how a batch sees itself); sort candidates best first; pick `m` with the + **diversity heuristic**: keep a candidate only if it scores no better with any + already-kept node than with the base (`sim(c, kept) > sim(c, base)` rejects); + with `<= m` candidates, keep all. +* **Apply**: write each job's lists (new nodes get their blocks; an update + overwrites its own lists); then add the reverse links. Reverse links are + grouped by (layer, target) in a fixed order; a target whose list stays within + capacity (`2m` on layer 0, `m` above) just appends (deduplicated); an overflow + first drops tombstones, and if the list is still over capacity **re-prunes** + it with the same heuristic over the old links plus the new ones. Groups are independent, so + they run in parallel and are written back in group order. The entry point is + the first new node of the highest level. +* A job with a non-finite norm is not linked (the exact scan never returns it + either). + +### 12.5 Maintenance + +* **Build**: insert live rows in slot order in batches of `size / 16` (1 at the + start, at most 512), each batch one `link` call. Because plans read a frozen + graph and are applied in order, **the graph depends on the seed, the data and + the insertion order, not on the thread count** (checked by the tests). The + price is that rows of one batch find each other only by brute force, not + through the graph; at most 6% of the graph, this costs no measurable recall + (the recall-vs-ef curve equals hnswlib's on text-384 at 100k). +* **Insert**: the write that stores a row links it (`link` of one job, no + parallelism). No pending list, so a search never waits for a fold and never + scans an unindexed tail. Cost 0.4-1.5 ms per insert at 100k-1M x 384 + (dominated by the layer search at `ef_construction` 200 and the re-prunes). +* **Update** (same slot, new vector): the node keeps its level; its own links + are recomputed from the new vector and the new neighbours link back. Links + other nodes held to the old position stay: they are only hints, every score + comes from the current vector, and a list drops them when it next overflows. + Counts as churn. +* **Delete**: tombstone only (the table's NaN norm already hides the row). The + node and its links stay, so routes through it keep working; its floats are + still in the matrix until compaction. A tombstone leaves a list when that list + is re-pruned. No work at delete time (about a microsecond). +* **Compaction** (the table's, > 25% dead, or explicit): the graph follows the + table's old-slot -> new-slot map in one serial pass: tombstones leave, every + link is renumbered, and a node that lost links takes up the links of the + tombstones it pointed to ("bridging": walk its lost neighbours in order and + append their surviving links not already present, up to capacity, no distance + computed). The entry point becomes the first node of the highest level. An + O(n x m) pause, the same order as the table's own compaction. +* **Rebuild**: updates plus deletes since the build exceeding the node count + mark the graph stale; the next search that needs it rebuilds it under the + write lock (the build time below; searches wait). Call + `build_ann_index(true)` off the hot path to choose when. Inserts do not count. +* Dropping the index when live rows fall below `min_size` (or 2), `clear`, and + `set_ann` follow IVF. + +### 12.6 Invariants (added to section 7; checked by `verify_invariants`) + +8. `level.len() == slots`; every live row with a finite norm is in the graph; + tombstoned rows may be (until compaction). +9. A node's lists exist for layers `0..=level`, hold no more than the layer's + capacity, no self link, no duplicate, and every target is in the graph with + level >= the layer. +10. The entry point is in the graph at the top level, and no node is higher. + +### 12.7 Measured (real text-384, MiniLM, HotpotQA; `examples/hnsw_bench.rs`) + +32-thread laptop with other work running, so **timings are indicative**; recall +and memory are exact. Single-thread loops pinned to one CPU, `k = 10`, 1,000 +real HotpotQA questions, ground truth the float64 cosine top-10 +(`benchmarks/competitive/`), `m` 16, `ef_construction` 200, build on all 32 +threads. IVF is the default `AnnConfig` with `scan_threads(1)` in the 1T loop. + +| | 100,000 rows | 1,000,000 rows | +|---|---|---| +| HNSW build | 4.9-6.1 s | 142-170 s | +| IVF build | 0.6-1.2 s | 8.4 s | +| graph size | 13.8 MB (145 B/node) | 139 MB (146 B/node) | +| whole store / raw vectors | 1.18x | 1.18x | +| insert / update (median, p99) | 0.44 / 0.81 ms | 0.65-1.5 / 1.4-2.7 ms | +| delete | < 2 us | < 2 us | +| HNSW recall@10 by `ef` | 64: .915, 96: .948, 128: .963, 192: .980 | 96: .906, 128: .929, 192: .955, 256: .969 | +| **HNSW** at the auto `ef` (96 / 192): recall, QPS 1T, QPS batch | .948, 5,021, 28,925 | .955, 1,539-1,782, 6,225 | +| **IVF** near the same recall (`nprobe` 101 / 200): recall, QPS 1T, QPS batch | .943, 1,186, 49,996 | .941, 159, 8,570 | +| single-thread speed-up over IVF | about 4x | about 10x | + +For reference, `benchmarks/competitive/` measured hnswlib at recall 0.949, 3,563 +QPS (100k, `ef` 96) and 1,486 QPS at recall 0.95 (1M) on a quiet machine, and its +`ef`-vs-recall curve matches the one above (0.916 at 64, 0.964 at 128), so the +graph quality is the same; the batch column is where IVF is not beaten (12.8). + +### 12.8 Not done, on purpose + +* **No persistence.** The graph is rebuilt after load, like IVF. A saved graph + would change the file format and the digest contract; loading 1M rows takes + seconds, rebuilding the graph minutes. +* **Slow build.** HNSW cannot be built as fast as k-means cells. The batched + parallel build is the mitigation; incremental inserts are the other (build once + from the loaded rows, then writes keep it current). +* **Update leaves stale in-links** (12.5) and delete leaves tombstones until + compaction; heavy churn rebuilds instead of repairing. +* **No cell-major batch.** IVF's batch scores each probed cell once for all + queries that probe it; a graph has no such shared structure, so a batch is + throughput-parallel single queries. At 100k text-384 (recall about 0.95) IVF's + batch measured faster than HNSW's; at 1M they are comparable. Choose HNSW for + single-query latency and QPS, IVF for fast builds, small memory and big + batches. +* **No quantisation, no filter-aware graph (ACORN style)**: a selective type + filter widens `ef` and may end in the scan. +* **No online graph repair** other than the bridging at compaction. + +### 12.9 Portable vs language-specific + +*Portable (copy exactly)*: the level draw (integer, seeded), the (score, slot) +order, the layout sizes, the plan/apply split with in-batch brute-force +candidates and fixed apply order (this is what makes the build deterministic and +parallel in any language), the diversity heuristic and overflow re-prune, the +widening/fallback rules, tombstone-then-bridge compaction, the rebuild rule. +*Language-specific*: threads and the pool (plans are independent; any parallel +map with ordered results works; a single-threaded port is just slower), the +visited set (bitmap per thread, or a stamp array), prefetching, heaps, SIMD +(section 2: one fixed summation order everywhere). diff --git a/isongraph-vector-rs/README.md b/isongraph-vector-rs/README.md index 900eabc..fc7781b 100644 --- a/isongraph-vector-rs/README.md +++ b/isongraph-vector-rs/README.md @@ -246,7 +246,8 @@ get back: `vec0` runs an exact brute-force KNN in C, not an approximate index — measured identical ordering and scores within 2e-07 of the scan. Measured over 20,000 vectors of 384 dimensions, top-10, median of five runs: **19.0 ms** for the SQLite scan, **6.9 ms** through the index, and **9.9 ms** -for the in-memory store. Recall of the index against the exact scan is +for the in-memory store as it was then (its scan has since been rewritten; see +*Searching large stores*). Recall of the index against the exact scan is **1.000** — the same answers, faster. Below a few thousand vectors the scan wins outright, which is why the index is opt-in. @@ -255,6 +256,110 @@ them, which is where a third of the query time used to go. Filtering by node type stays exact too: the type is a `vec0` metadata column, so it narrows the search rather than filtering its results. +## Searching large stores + +Search is exact unless you switch the approximate index on, and the exact path +is fast. The vectors live **once**, in one contiguous chunked `f32` matrix that +is the store itself (a record is a row of it; there is no second copy, about +1.1x the raw vector bytes at a million 384-d vectors). A search is a single pass +over it: a SIMD dot product per row (AVX2 + FMA when the CPU has them, checked +at run time), a bounded heap for the top k, and, for a big store, up to eight +threads of a persistent pool (`set_scan_threads(1)` keeps it on one). Equal +scores come back in the order the vectors were stored, so a search gives the +same answer on every run and for any thread count. See `DESIGN.md` for the +layout, the concurrency model and the write path. + +### Many queries, many threads + +```rust +// one call, many queries: ~10x the throughput of a loop on a big store, and +// every answer is identical (ids, scores, order) to the one-query call +let answers = store.similarity_search_vector_batch(&queries, 10, None, 0.0)?; +let by_text = store.similarity_search_batch(&["first", "second"], 10, None, 0.0)?; + +// searches take &self, never lock the data, and scale across threads +std::thread::scope(|s| { + for q in &queries { let store = &store; s.spawn(move || store.similarity_search_vector(q, 10, None, 0.0)); } +}); + +// readers and a writer on different threads: a reader/writer lock with short write sections +let shared = SharedEmbeddingStore::new(store); +let writer = shared.clone(); +std::thread::spawn(move || writer.add(NodeId::new("doc", "9"), "new text", None)); +let hits = shared.similarity_search_vector(&q, 10, None, 0.0)?; // never sees a half-applied write +``` + +Batch calls run on a persistent thread pool shared by all stores (one worker per +logical CPU; `ISONGRAPH_VECTOR_THREADS=n` or `store.set_search_pool(Some(Arc::new(SearchPool::new(n))))` +changes it, `set_scan_threads(n)` caps one call). The exact batch scores 16 +queries against each block of rows while it is in cache; with the index on, the +batch scores each probed cell once for all the queries that probe it. + +For stores past a few hundred thousand vectors there is an opt-in approximate +index, IVF-flat, matching the Python port: + +```rust +use isongraph_vector_rs::{AnnConfig, EmbeddingStore}; + +graph.set_ann(Some(AnnConfig::default()))?; // or store.set_ann(..) / EmbeddingStore::with_ann(..) +let hits = graph.similarity_search("query", 10, None, 0.0)?; // from the index +let exact = graph.similarity_search_with( + "query", + &SimilarityOptions { exact: true, ..Default::default() }, // the full scan +)?; +let report = graph.recall_check(&my_queries, 10, None, 0.0)?; // measure it on YOUR queries +println!("recall@10 = {:.3}, nprobe {} of {} cells", report.recall, report.nprobe, report.nlist); +``` + +Vectors are clustered into `nlist` cells (default about `4 * sqrt(N)`) by a +seeded spherical k-means, and a query scores only the `nprobe` nearest cells +(default 5% of them), with the same cosine as the exact scan. It is built by the +first search that wants it, is never saved (a saved graph, its sidecar and its +digest are identical with it on or off), and is used only for a bounded `top_k` +on a store of at least `AnnConfig::min_size` (default 50,000) vectors. Writes +are visible at once: new and updated vectors are scored exactly until 256 of +them have piled up, and the write that makes it 256 folds them into their cells +(a pause of a millisecond or two at a million vectors), so a search never pays +for a backlog; deletes stop being returned, and heavy churn retrains on the next +search (call `build_ann_index(true)` yourself, off the hot path, to choose when). A `node_type`, `threshold` or `NodeFilter` applies to the probed rows; +if fewer than `top_k` survive, the probe widens 4x and tries again, ending in +the full scan. `build_ann_index`, `ann_stats` and `recall_check` are on both +`EmbeddingStore` and `SemanticGraph`. The SQLite store has no approximate index +(load its `export_embeddings` into an in-memory store to use one). + +Measured with `examples/ann_bench.rs` on a 32-thread i9-14900HX laptop with +other work running, 384-d unit vectors, top 10, median of 200 queries +(synthetic clustered data; the index's recall depends entirely on how your data +clusters, so measure your own): + +| vectors | exact, 1 thread | exact, auto threads | index build | IVF at default nprobe | recall@10 | +|---|---|---|---|---|---| +| 10,000 | 0.44 ms | 0.30 ms | 0.07 s | 0.043 ms | 1.000 | +| 100,000 | 6.8 ms | 3.8 ms | 0.6 s | 0.64 ms | 1.000 | +| 1,000,000 | 68 ms | 28 ms | 8.3 s | 2.4 ms | 0.993 | + +On the harder, overlapping "diffuse" data the same default gives 0.986 recall +at 100,000 vectors (0.62 ms) and 0.991 at 1,000,000 (2.6 ms); lower `nprobe` +values are faster and much less accurate there (`nprobe = 16` of 4,000 cells is +0.30 recall), and below 100,000 vectors the exact scan is usually the better +choice. Run `cargo run --release --example ann_bench -- --help` to reproduce. + +**HNSW (opt-in alternative to IVF).** `AnnConfig::hnsw()` (or `mode: AnnMode::Hnsw`) +builds a hierarchical navigable small-world graph instead: `m` 16, `ef_construction` +200, `ef_search` chosen by store size (about 0.95 recall@10 on 384-d text +embeddings; override with `set_ann_ef_search` or per call with +`similarity_search_vector_ef`). It stores slot numbers and links only (about 146 +bytes per vector, the whole store 1.18x the raw vectors at 1M x 384), returns the +exact scan's scores for the hits it finds, links each row as it is written +(0.5-1.5 ms), tombstones deletes and bridges over them at compaction, and is +never saved. `AnnConfig::default()` stays IVF, and exact search stays the +default. The trade: on real text-384 vectors at recall about 0.95 a single query +is about 4x (100k) to 10x (1M) faster than IVF, but the build is far slower +(142-170 s at 1M against IVF's 8 s; same graph for any thread count) and IVF's +big batches stay faster at 100k. `examples/hnsw_bench.rs` sweeps `ef_search` +(recall against queries per second, build time, memory); `DESIGN.md` section 10 +is the specification. + ## Embedding over HTTP With the `http-encoder` feature, `HttpEncoder` is an `EmbeddingEncoder` that @@ -341,9 +446,14 @@ store.from_json(&json, false)?; ## Things that will bite you -- **Search is brute force**, sqlite-vec included: linear in the number of +- **Search is brute force** unless you switch on the opt-in IVF index (see + *Searching large stores*), sqlite-vec included: linear in the number of nodes. This is built for an application's own knowledge graph, not as a vector-database replacement. +- **The approximate index is approximate.** A true neighbour in an unprobed cell + is missed; recall was measured on synthetic data only. Its k-means uses this + port's own seeded generator, so the same `seed` builds a different (equally + valid) index than the Python port does. - **One embedding per node**, from one text field or a concatenation. No multi-vector nodes, no edge embeddings. - **`MockEncoder` has no semantic structure.** Use it to test plumbing, never @@ -353,9 +463,15 @@ store.from_json(&json, false)?; - **`NodeId` is string-typed here.** The shared payload format carries an `id_type` of `str` or `int`; this port reads and preserves `int` written by another, but every id it originates is a string. -- **Seed order among equal scores is not deterministic.** The store is a - `HashMap`, so ties can be broken differently between runs. It only shows up - with hand-crafted identical vectors; distinct texts give distinct scores. +- **Equal scores are ordered by arrival.** Searches do not depend on `HashMap` + order: rows are kept in the order they were stored (an update keeps its row, + a delete and re-add goes to the end), and ties come back in that order. +- **`EmbeddingStore::get` returns an owned `EmbeddingRecord`** (the vector is a + row of the shared matrix, so the record is built on request). Use `get_vector` + to borrow the floats without a copy. +- **Compaction is a pause.** A store whose rows are a quarter deleted squeezes + the matrix (order preserving), a pause proportional to its size; call + `compact()` at a quiet moment to choose when. - **The graph digest sees integer ids only through the file text.** `NodeId` is a string here, so `graph_digest(&graph)` hashes every id as a string (right for anything built in Rust); `save` and `load` read the id column's types from @@ -370,8 +486,8 @@ store.from_json(&json, false)?; ## Tests ```bash -cargo test # 122 unit tests + 2 doctests -cargo test --features sqlite # + 22 SQLite tests +cargo test # 127 unit tests + 32 approximate-index tests + 2 doctests +cargo test --features sqlite # + 22 SQLite tests + 3 index/SQLite tests cargo test --features http-encoder # + 18 HTTP encoder tests (local mock server) cargo clippy --all-targets --features sqlite -- -D warnings cargo fmt --check diff --git a/isongraph-vector-rs/examples/ann_bench.rs b/isongraph-vector-rs/examples/ann_bench.rs new file mode 100644 index 0000000..6a42cde --- /dev/null +++ b/isongraph-vector-rs/examples/ann_bench.rs @@ -0,0 +1,415 @@ +//! Exact search against the opt-in IVF index on synthetic clustered vectors. +//! +//! For each size and data regime it prints: the ingest rate, the first query +//! (nothing to build: the vectors are already one matrix), warm exact latency on one thread and on all +//! of them, the index build time and size, and a sweep over `nprobe` with +//! recall@10 against the exact scan and the speed-up. +//! +//! The data mirror `benchmarks/ann.py` in the repository root (384-d unit +//! vectors; "clustered" = well-separated topics, N/100 clusters; "diffuse" = +//! overlapping, hierarchical, N/40 clusters in 20 super-topics), so the numbers +//! can sit next to the Python ones. They are generated here, with this port's +//! own random numbers, so they are *the same kind* of data, not the same +//! vectors. Synthetic clustered data says what the index does on clustered +//! data and nothing about your embeddings: use `recall_check` on your own +//! queries. +//! +//! ```text +//! cargo run --release --example ann_bench # 10k and 100k +//! cargo run --release --example ann_bench -- --sizes 10000,100000,1000000 +//! cargo run --release --example ann_bench -- --sizes 1000000 --regimes clustered --json out.json +//! ``` +//! +//! Options: `--sizes a,b,c` `--regimes clustered,diffuse` `--queries N` +//! `--dim D` `--nprobe 1,4,16,64,auto` `--k K` `--json FILE` `--no-index`. +//! Needs about 1.7 GB of RAM per million vectors (the store keeps the vectors +//! once, in one matrix). + +use isongraph_vector_rs::{AnnConfig, EmbeddingStore, NodeId, SimilarityResult}; +use std::collections::HashSet; +use std::time::Instant; + +struct Rng(u64); + +impl Rng { + fn new(seed: u64) -> Self { + Rng(seed) + } + fn next_u64(&mut self) -> u64 { + self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } + fn uniform(&mut self) -> f32 { + ((self.next_u64() >> 40) as f32 + 0.5) / (1u64 << 24) as f32 + } + fn below(&mut self, n: usize) -> usize { + ((self.next_u64() as u128 * n as u128) >> 64) as usize + } + fn fill_gauss(&mut self, out: &mut [f32]) { + for pair in out.chunks_mut(2) { + let r = (-2.0 * self.uniform().ln()).sqrt(); + let a = std::f32::consts::TAU * self.uniform(); + pair[0] = r * a.cos(); + if pair.len() == 2 { + pair[1] = r * a.sin(); + } + } + } +} + +fn unit(v: &mut [f32]) { + let norm = v.iter().map(|x| x * x).sum::<f32>().sqrt(); + if norm > 0.0 { + v.iter_mut().for_each(|x| *x /= norm); + } +} + +struct Regime { + centers: Vec<Vec<f32>>, + noise: f32, + dim: usize, +} + +impl Regime { + fn new(name: &str, n: usize, dim: usize) -> Self { + let mut rng = Rng::new(4242 ^ (n as u64).wrapping_mul(0x9E37) ^ name.len() as u64); + let (per_cluster, noise, supers, spread) = match name { + "clustered" => (100, 1.0, 0, 0.0), + "diffuse" => (40, 2.0, 20, 1.0), + other => panic!("unknown regime {other:?} (clustered, diffuse)"), + }; + let clusters = (n / per_cluster).max(8); + let gauss_unit = |rng: &mut Rng| { + let mut v = vec![0.0f32; dim]; + rng.fill_gauss(&mut v); + unit(&mut v); + v + }; + let super_centers: Vec<Vec<f32>> = (0..supers).map(|_| gauss_unit(&mut rng)).collect(); + let centers = (0..clusters) + .map(|_| { + let mut c = gauss_unit(&mut rng); + if supers > 0 { + let parent = &super_centers[rng.below(supers)]; + for (x, p) in c.iter_mut().zip(parent) { + *x = p + spread * *x; + } + unit(&mut c); + } + c + }) + .collect(); + Self { + centers, + noise, + dim, + } + } + + fn sample(&self, count: usize, stream: u64) -> Vec<Vec<f32>> { + let mut rng = Rng::new(0xA5A5_0000 ^ stream.wrapping_mul(0x2545_F491_4F6C_DD1D)); + let scale = self.noise / (self.dim as f32).sqrt(); + (0..count) + .map(|_| { + let mut v = vec![0.0f32; self.dim]; + rng.fill_gauss(&mut v); + let c = &self.centers[rng.below(self.centers.len())]; + for (x, c) in v.iter_mut().zip(c) { + *x = c + *x * scale; + } + unit(&mut v); + v + }) + .collect() + } +} + +struct Opts { + sizes: Vec<usize>, + regimes: Vec<String>, + queries: usize, + dim: usize, + k: usize, + nprobe: Vec<Option<usize>>, + json: Option<String>, + index: bool, +} + +fn parse() -> Opts { + let mut o = Opts { + sizes: vec![10_000, 100_000], + regimes: vec!["clustered".into(), "diffuse".into()], + queries: 200, + dim: 384, + k: 10, + nprobe: vec![Some(1), Some(4), Some(16), Some(64), None], + json: None, + index: true, + }; + let args: Vec<String> = std::env::args().skip(1).collect(); + let mut i = 0; + while i < args.len() { + let value = |i: usize| { + args.get(i + 1) + .cloned() + .unwrap_or_else(|| panic!("{} needs a value", args[i])) + }; + match args[i].as_str() { + "--sizes" => { + o.sizes = value(i) + .split(',') + .map(|s| s.trim().parse().expect("size")) + .collect(); + i += 1; + } + "--regimes" => { + o.regimes = value(i).split(',').map(|s| s.trim().to_string()).collect(); + i += 1; + } + "--queries" => { + o.queries = value(i).parse().expect("queries"); + i += 1; + } + "--dim" => { + o.dim = value(i).parse().expect("dim"); + i += 1; + } + "--k" => { + o.k = value(i).parse().expect("k"); + i += 1; + } + "--nprobe" => { + o.nprobe = value(i) + .split(',') + .map(|s| match s.trim() { + "auto" => None, + n => Some(n.parse().expect("nprobe")), + }) + .collect(); + i += 1; + } + "--json" => { + o.json = Some(value(i)); + i += 1; + } + "--no-index" => o.index = false, + "-h" | "--help" => { + println!( + "{}", + include_str!("ann_bench.rs") + .lines() + .take_while(|l| l.starts_with("//!")) + .map(|l| l.trim_start_matches("//!").trim_start()) + .collect::<Vec<_>>() + .join("\n") + ); + std::process::exit(0); + } + other => panic!("unknown option {other}"), + } + i += 1; + } + o +} + +fn percentile(sorted: &[f64], p: f64) -> f64 { + if sorted.is_empty() { + return 0.0; + } + let at = ((sorted.len() - 1) as f64 * p).round() as usize; + sorted[at] +} + +struct Timing { + median_ms: f64, + p95_ms: f64, +} + +fn time_queries( + queries: &[Vec<f32>], + mut run: impl FnMut(&[f32]) -> Vec<SimilarityResult>, +) -> (Timing, Vec<Vec<SimilarityResult>>) { + let mut times = Vec::with_capacity(queries.len()); + let mut out = Vec::with_capacity(queries.len()); + for q in queries { + let t = Instant::now(); + let r = run(q); + times.push(t.elapsed().as_secs_f64() * 1000.0); + out.push(r); + } + times.sort_by(f64::total_cmp); + ( + Timing { + median_ms: percentile(×, 0.5), + p95_ms: percentile(×, 0.95), + }, + out, + ) +} + +fn recall(found: &[Vec<SimilarityResult>], truth: &[Vec<SimilarityResult>]) -> (f64, f64) { + let mut sum = 0.0; + let mut worst: f64 = 1.0; + for (f, t) in found.iter().zip(truth) { + let want: HashSet<&NodeId> = t.iter().map(|r| &r.node_id).collect(); + let r = f.iter().filter(|x| want.contains(&x.node_id)).count() as f64 + / want.len().max(1) as f64; + sum += r; + worst = worst.min(r); + } + (sum / found.len().max(1) as f64, worst) +} + +fn main() { + let o = parse(); + let cpus = std::thread::available_parallelism() + .map(|n| n.get()) + .unwrap_or(1); + println!( + "isongraph-vector-rs ann_bench: {} logical CPUs, dim {}, top_k {}, {} queries per cell, release={}", + cpus, + o.dim, + o.k, + o.queries, + !cfg!(debug_assertions) + ); + let mut rows = Vec::new(); + for regime_name in &o.regimes { + for &n in &o.sizes { + println!("\n## {regime_name}, N = {n}"); + let regime = Regime::new(regime_name, n, o.dim); + + let mut store = EmbeddingStore::new(None); + let started = Instant::now(); + let chunk = 10_000; + let mut added = 0; + while added < n { + let count = chunk.min(n - added); + for (i, v) in regime + .sample(count, 1000 + (added / chunk) as u64) + .into_iter() + .enumerate() + { + store + .add(NodeId::new("doc", (added + i).to_string()), "", Some(v)) + .unwrap(); + } + added += count; + } + let ingest = n as f64 / started.elapsed().as_secs_f64(); + println!("* ingest {:.0} vectors/s", ingest); + + let queries = regime.sample(o.queries, 7); + // first query: no cache to build, since the matrix is the store + let t = Instant::now(); + store + .similarity_search_vector_exact(&queries[0], o.k, None, 0.0) + .unwrap(); + let first_ms = t.elapsed().as_secs_f64() * 1000.0; + + store.set_scan_threads(1); + let (exact_1, truth) = time_queries(&queries, |q| { + store + .similarity_search_vector_exact(q, o.k, None, 0.0) + .unwrap() + }); + // exact latency by thread count, so the automatic choice can be checked + let mut by_threads = Vec::new(); + for t in [2usize, 4, 8, 16, 32].into_iter().filter(|t| *t <= cpus) { + store.set_scan_threads(t); + let (timing, _) = time_queries(&queries[..queries.len().min(50)], |q| { + store + .similarity_search_vector_exact(q, o.k, None, 0.0) + .unwrap() + }); + by_threads.push((t, timing.median_ms)); + } + println!( + "* exact median by thread count: 1 -> {:.2} ms, {}", + exact_1.median_ms, + by_threads + .iter() + .map(|(t, ms)| format!("{t} -> {ms:.2} ms")) + .collect::<Vec<_>>() + .join(", ") + ); + store.set_scan_threads(0); + let (exact_n, _) = time_queries(&queries, |q| { + store + .similarity_search_vector_exact(q, o.k, None, 0.0) + .unwrap() + }); + println!( + "* exact scan: first query {first_ms:.0} ms (builds the cache); warm median {:.2} ms / p95 {:.2} ms on 1 thread, {:.2} ms / {:.2} ms on auto threads", + exact_1.median_ms, exact_1.p95_ms, exact_n.median_ms, exact_n.p95_ms + ); + let mut row = serde_json::json!({ + "regime": regime_name, "n": n, "dim": o.dim, "k": o.k, "queries": o.queries, "cpus": cpus, + "ingest_per_s": ingest, "first_query_ms": first_ms, + "exact_1thread_median_ms": exact_1.median_ms, "exact_1thread_p95_ms": exact_1.p95_ms, + "exact_auto_median_ms": exact_n.median_ms, "exact_auto_p95_ms": exact_n.p95_ms, + }); + + if o.index { + store + .set_ann(Some(AnnConfig { + min_size: 0, + ..AnnConfig::default() + })) + .unwrap(); + let t = Instant::now(); + let stats = store.build_ann_index(false).unwrap(); + let build = t.elapsed().as_secs_f64(); + println!( + "* index build {build:.2} s for nlist={} (automatic nprobe={}); index {:.1} MB on top of {:.0} MB of cached vectors", + stats.nlist.unwrap_or(0), + stats.nprobe.unwrap_or(0), + stats.index_bytes as f64 / 1e6, + (n * o.dim * 4) as f64 / 1e6 + ); + row["index_build_s"] = build.into(); + row["nlist"] = stats.nlist.into(); + row["index_mb"] = (stats.index_bytes as f64 / 1e6).into(); + + println!("\n| nprobe | % cells | recall@{} mean | worst | median ms | p95 ms | vs exact 1T | vs exact auto |", o.k); + println!("|---|---|---|---|---|---|---|---|"); + let mut sweep = Vec::new(); + for nprobe in &o.nprobe { + store.set_ann_nprobe(*nprobe).unwrap(); + let (ann, found) = time_queries(&queries, |q| { + store.similarity_search_vector(q, o.k, None, 0.0).unwrap() + }); + let effective = store.ann_stats().nprobe.unwrap_or(0); + let nlist = stats.nlist.unwrap_or(1); + let (mean, worst) = recall(&found, &truth); + let label = match nprobe { + Some(p) => p.to_string(), + None => format!("{effective} (auto)"), + }; + println!( + "| {label} | {:.1} | {mean:.3} | {worst:.2} | {:.3} | {:.3} | {:.1}x | {:.1}x |", + 100.0 * effective as f64 / nlist as f64, + ann.median_ms, + ann.p95_ms, + exact_1.median_ms / ann.median_ms, + exact_n.median_ms / ann.median_ms + ); + sweep.push(serde_json::json!({ + "nprobe": effective, "recall_mean": mean, "recall_worst": worst, + "median_ms": ann.median_ms, "p95_ms": ann.p95_ms, + })); + } + row["sweep"] = sweep.into(); + store.set_ann(None).unwrap(); + } + rows.push(row); + } + } + if let Some(path) = o.json { + std::fs::write(&path, serde_json::to_string_pretty(&rows).unwrap()).unwrap(); + println!("\nwrote {path}"); + } +} diff --git a/isongraph-vector-rs/examples/competitive_bench.rs b/isongraph-vector-rs/examples/competitive_bench.rs new file mode 100644 index 0000000..fd8c18f --- /dev/null +++ b/isongraph-vector-rs/examples/competitive_bench.rs @@ -0,0 +1,649 @@ +//! Loader and measurement harness for `benchmarks/competitive/` (the cross-library comparison). +//! +//! It reads the same raw float32 vectors, queries and float64 ground truth that the Python harness +//! feeds the other systems, so every system sees identical data. Bench-only: nothing in the library +//! is touched. +//! +//! ```text +//! cargo run --release --example competitive_bench -- --ds-dir D:/.../prep/text384 --ds text384 --n 100000 \ +//! --exp core,mutation,saveload --queries 2000 --out result.json +//! cargo run --release --example competitive_bench -- ... --exp filter +//! ``` +//! `core` measures the exact scan (1 thread, then automatic threads) and the opt-in IVF index +//! (nprobe sweep, 3 timed runs each); `mutation` inserts, updates and removes 10k vectors and +//! measures recall again; `saveload` times the JSON sidecar; `filter` restricts to a node type +//! holding a random 10% / 1% of the vectors. + +use isongraph_vector_rs::{AnnConfig, EmbeddingStore, NodeId, SimilarityResult}; +use serde_json::{json, Value}; +use std::collections::HashSet; +use std::fs::File; +use std::io::{Read, Seek, SeekFrom}; +use std::time::Instant; + +// ---- process memory (Windows working set), no extra crates ------------------------------------- +#[cfg(windows)] +#[repr(C)] +struct Pmc { + cb: u32, + page_fault_count: u32, + peak_working_set_size: usize, + working_set_size: usize, + quota_peak_paged_pool_usage: usize, + quota_paged_pool_usage: usize, + quota_peak_non_paged_pool_usage: usize, + quota_non_paged_pool_usage: usize, + pagefile_usage: usize, + peak_pagefile_usage: usize, +} +#[cfg(windows)] +#[link(name = "kernel32")] +extern "system" { + fn GetCurrentProcess() -> isize; + fn K32GetProcessMemoryInfo(h: isize, p: *mut Pmc, cb: u32) -> i32; + fn SetProcessAffinityMask(h: isize, mask: usize) -> i32; + fn SetPriorityClass(h: isize, class: u32) -> i32; +} + +/// Pin the process to one P-core (logical CPU 8 on the i9-14900HX; 0-15 are P-cores) for a single-threaded timed +/// loop, then release it. Unpinned single-thread loops varied up to 3x between processes on this hybrid CPU. +fn pinned<T>(f: impl FnOnce() -> T) -> T { + #[cfg(windows)] + { + let cpu: usize = std::env::var("BENCH_PIN_CPU") + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(8); + let all = usize::MAX + >> (usize::BITS as usize + - std::thread::available_parallelism() + .map(|n| n.get()) + .unwrap_or(32)); + // SAFETY: plain kernel32 calls on the current process. + unsafe { + SetPriorityClass(GetCurrentProcess(), 0x80); // HIGH_PRIORITY_CLASS + SetProcessAffinityMask(GetCurrentProcess(), 1usize << cpu); + } + let r = f(); + unsafe { SetProcessAffinityMask(GetCurrentProcess(), all) }; + r + } + #[cfg(not(windows))] + f() +} +fn mem() -> Value { + #[cfg(windows)] + { + // SAFETY: Pmc matches PROCESS_MEMORY_COUNTERS and `cb` is its size. + let p = unsafe { + let mut p: Pmc = std::mem::zeroed(); + p.cb = std::mem::size_of::<Pmc>() as u32; + K32GetProcessMemoryInfo(GetCurrentProcess(), &mut p, p.cb); + p + }; + json!({"rss_mb": p.working_set_size as f64 / 1048576.0, "peak_mb": p.peak_working_set_size as f64 / 1048576.0}) + } + #[cfg(not(windows))] + json!({"rss_mb": 0.0, "peak_mb": 0.0}) +} + +// ---- data --------------------------------------------------------------------------------------- +fn read_f32(path: &str, start: usize, count: usize, d: usize) -> Vec<f32> { + let mut f = File::open(path).unwrap_or_else(|e| panic!("{path}: {e}")); + f.seek(SeekFrom::Start((start * d * 4) as u64)).unwrap(); + let mut buf = vec![0u8; count * d * 4]; + f.read_exact(&mut buf).unwrap(); + (0..buf.len() / 4) + .map(|i| f32::from_le_bytes(buf[4 * i..4 * i + 4].try_into().unwrap())) + .collect() +} +fn read_i32(path: &str) -> Vec<i32> { + let mut b = Vec::new(); + File::open(path) + .unwrap_or_else(|e| panic!("{path}: {e}")) + .read_to_end(&mut b) + .unwrap(); + (0..b.len() / 4) + .map(|i| i32::from_le_bytes(b[4 * i..4 * i + 4].try_into().unwrap())) + .collect() +} +fn read_u8(path: &str) -> Vec<u8> { + let mut b = Vec::new(); + File::open(path) + .unwrap_or_else(|e| panic!("{path}: {e}")) + .read_to_end(&mut b) + .unwrap(); + b +} + +struct Ctx { + dir: String, + #[allow(dead_code)] + ds: String, + d: usize, + n: usize, + k: usize, + queries: Vec<Vec<f32>>, +} +impl Ctx { + fn base(&self, start: usize, count: usize) -> Vec<Vec<f32>> { + read_f32(&format!("{}/base.f32", self.dir), start, count, self.d) + .chunks(self.d) + .map(|c| c.to_vec()) + .collect() + } + fn gt(&self, kind: &str) -> Vec<Vec<i32>> { + read_i32(&format!("{}/gt_{}{}.i32", self.dir, kind, self.n)) + .chunks(100) + .map(|c| c.to_vec()) + .collect() + } +} + +// ---- stats -------------------------------------------------------------------------------------- +fn pct(sorted: &[f64], p: f64) -> f64 { + if sorted.is_empty() { + return 0.0; + } + sorted[((sorted.len() - 1) as f64 * p).round() as usize] +} +fn lat_stats(lat_s: &[f64]) -> Value { + let mut ms: Vec<f64> = lat_s.iter().map(|x| x * 1000.0).collect(); + ms.sort_by(f64::total_cmp); + let mean = ms.iter().sum::<f64>() / ms.len().max(1) as f64; + json!({"median_ms": pct(&ms, 0.5), "p95_ms": pct(&ms, 0.95), "p99_ms": pct(&ms, 0.99), "mean_ms": mean}) +} +fn median(v: &[f64]) -> f64 { + let mut s = v.to_vec(); + s.sort_by(f64::total_cmp); + pct(&s, 0.5) +} +fn recall(found: &[Vec<usize>], truth: &[Vec<i32>], k: usize) -> f64 { + let mut hit = 0usize; + for (f, t) in found.iter().zip(truth) { + let want: HashSet<i32> = t.iter().take(k).cloned().collect(); + hit += f + .iter() + .take(k) + .filter(|x| want.contains(&(**x as i32))) + .count(); + } + hit as f64 / (k * found.len().max(1)) as f64 +} +fn ids(r: &[SimilarityResult]) -> Vec<usize> { + r.iter() + .map(|x| x.node_id.id.parse::<usize>().unwrap()) + .collect() +} + +#[derive(Clone, Copy, PartialEq)] +enum Path { + Exact, + Ann, +} +fn search( + s: &EmbeddingStore, + q: &[f32], + k: usize, + p: Path, + ty: Option<&str>, +) -> Vec<SimilarityResult> { + match p { + Path::Exact => s.similarity_search_vector_exact(q, k, ty, 0.0).unwrap(), + Path::Ann => s.similarity_search_vector(q, k, ty, 0.0).unwrap(), + } +} +fn run_loop( + s: &EmbeddingStore, + qs: &[Vec<f32>], + k: usize, + p: Path, + ty: Option<&str>, +) -> (Vec<f64>, Vec<Vec<usize>>) { + let mut lat = Vec::with_capacity(qs.len()); + let mut out = Vec::with_capacity(qs.len()); + for q in qs { + let t = Instant::now(); + let r = search(s, q, k, p, ty); + lat.push(t.elapsed().as_secs_f64()); + out.push(ids(&r)); + } + (lat, out) +} + +/// The batched call over all the pool's threads: warm up once, then 3 timed runs of the whole query set. +fn batch_point(s: &EmbeddingStore, c: &Ctx, truth: &[Vec<i32>], p: Path) -> Value { + let run = || { + let t = Instant::now(); + let r = match p { + Path::Exact => s.similarity_search_vector_exact_batch(&c.queries, c.k, None, 0.0), + Path::Ann => s.similarity_search_vector_batch(&c.queries, c.k, None, 0.0), + } + .unwrap(); + (c.queries.len() as f64 / t.elapsed().as_secs_f64(), r) + }; + let _ = run(); + let mut qps = Vec::new(); + let mut found = Vec::new(); + for _ in 0..3 { + let (q, r) = run(); + qps.push(q); + found = r; + } + let found: Vec<Vec<usize>> = found.iter().map(|r| ids(r)).collect(); + json!({"nq": c.queries.len(), "threads": s.search_pool().threads(), "qps": median(&qps), "qps_runs": qps, "recall": recall(&found, &truth[..found.len()], c.k)}) +} + +/// Warm up, size the query count to `budget` seconds, then 3 timed runs. +fn point( + s: &EmbeddingStore, + c: &Ctx, + truth: &[Vec<i32>], + p: Path, + ty: Option<&str>, + budget: f64, +) -> Value { + let k = c.k; + let nqall = c.queries.len(); + let w0 = Instant::now(); + for q in c.queries.iter().take(50) { + let _ = search(s, q, k, p, ty); + if w0.elapsed().as_secs_f64() > 3.0 { + break; + } + } + let (probe, _) = run_loop(s, &c.queries[..5.min(nqall)], k, p, ty); + let est = median(&probe).max(1e-7); + let nq = ((budget / est) as usize).clamp(30.min(nqall), nqall); + let qs = &c.queries[..nq]; + let mut runs = Vec::new(); + let mut lats: Vec<f64> = Vec::new(); + let mut rec = 0.0; + for r in 0..3 { + let (lat, res) = run_loop(s, qs, k, p, ty); + if r == 0 { + rec = recall(&res, &truth[..nq], k); + } + let total: f64 = lat.iter().sum(); + runs.push(nq as f64 / total); + lats.extend(lat); + if total > budget * 1.5 { + break; + } + } + let mut v = lat_stats(&lats); + v["nq"] = json!(nq); + v["recall"] = json!(rec); + v["qps_1t"] = json!(median(&runs)); + v["qps_1t_min"] = json!(runs.iter().cloned().fold(f64::MAX, f64::min)); + v["qps_1t_max"] = json!(runs.iter().cloned().fold(0.0, f64::max)); + v["qps_1t_runs"] = json!(runs); + v +} + +fn nlist_for(n: usize) -> usize { + ((4.0 * (n as f64).sqrt()).round() as usize).clamp(16, 65536) +} +fn knobs(n: usize) -> Vec<usize> { + let nl = nlist_for(n); + let mut v: Vec<usize> = [1, 2, 4, 8, 16, 32, 64, 128, 256] + .into_iter() + .filter(|x| *x < nl) + .collect(); + if n <= 200_000 { + v.push(nl); // nprobe == nlist is the exact scan; measured separately at scale + } + v +} + +fn ingest(s: &mut EmbeddingStore, c: &Ctx, n: usize, grp: Option<&[u8]>) -> f64 { + let mut secs = 0.0; + let step = 20000; + let mut at = 0; + while at < n { + let m = step.min(n - at); + let rows = c.base(at, m); + let t = Instant::now(); + for (i, v) in rows.into_iter().enumerate() { + let id = at + i; + let ty = match grp { + None => "doc", + Some(g) => ["t10", "t01", "rest"][g[id] as usize], + }; + s.add(NodeId::new(ty, id.to_string()), "", Some(v)).unwrap(); + } + secs += t.elapsed().as_secs_f64(); + at += m; + } + secs +} + +fn arg(args: &[String], name: &str, default: &str) -> String { + args.iter() + .position(|a| a == name) + .map(|i| args[i + 1].clone()) + .unwrap_or(default.to_string()) +} + +fn main() { + let args: Vec<String> = std::env::args().skip(1).collect(); + let dir = arg(&args, "--ds-dir", ""); + let ds = arg(&args, "--ds", ""); + let n: usize = arg(&args, "--n", "100000").parse().unwrap(); + let nq: usize = arg(&args, "--queries", "1000").parse().unwrap(); + let budget: f64 = arg(&args, "--budget", "15").parse().unwrap(); + let exps: Vec<String> = arg(&args, "--exp", "core") + .split(',') + .map(|s| s.to_string()) + .collect(); + let out_path = arg(&args, "--out", ""); + let label = arg(&args, "--label", "rust"); + let meta: Value = + serde_json::from_str(&std::fs::read_to_string(format!("{dir}/meta.json")).unwrap()) + .unwrap(); + let d = meta["d"].as_u64().unwrap() as usize; + let qall = read_f32(&format!("{dir}/queries.f32"), 0, nq, d); + let c = Ctx { + dir: dir.clone(), + ds: ds.clone(), + d, + n, + k: 10, + queries: qall.chunks(d).map(|x| x.to_vec()).collect(), + }; + let cpus = std::thread::available_parallelism() + .map(|x| x.get()) + .unwrap_or(1); + let mut runs: Vec<Value> = Vec::new(); + println!( + "{label}: ds={ds} n={n} d={d} queries={nq} cpus={cpus} release={}", + !cfg!(debug_assertions) + ); + let mem_base = mem(); + + if exps + .iter() + .any(|e| e == "core" || e == "mutation" || e == "saveload") + { + let mut store = EmbeddingStore::new(None); + let ingest_s = ingest(&mut store, &c, n, None); + let mem_after_ingest = mem(); + println!(" ingest {:.2}s", ingest_s); + let truth = c.gt(""); + // ---- exact ---- + store.set_scan_threads(1); + let t = Instant::now(); + store + .similarity_search_vector_exact(&c.queries[0], c.k, None, 0.0) + .unwrap(); + let first_ms = t.elapsed().as_secs_f64() * 1000.0; + let mem_after_first = mem(); + let mut exact = json!({ + "system": format!("{label}_exact"), "ds": ds, "n": n, "d": d, "k": c.k, "queries": nq, "all_threads": cpus, + "version": "isongraph-vector-rs (repo HEAD, cargo release)", + "build": {"ingest_s": ingest_s, "index_s": 0.0, "total_s": ingest_s, "first_query_ms": first_ms, "ready_s": ingest_s + first_ms / 1000.0}, + "mem_baseline": mem_base, "mem_after_ingest": mem_after_ingest, "mem_after_build": mem_after_first, "mem_after_first_query": mem_after_first, + }); + let mut p1 = pinned(|| point(&store, &c, &truth, Path::Exact, None, budget)); + p1["knob"] = Value::Null; + println!( + " exact 1T recall={} qps={:.1} median={:.3}ms", + p1["recall"], + p1["qps_1t"].as_f64().unwrap(), + p1["median_ms"].as_f64().unwrap() + ); + exact["sweep"] = json!([p1]); + store.set_scan_threads(0); + let pa = point(&store, &c, &truth, Path::Exact, None, budget); + println!( + " exact auto-threads qps={:.1} median={:.3}ms", + pa["qps_1t"].as_f64().unwrap(), + pa["median_ms"].as_f64().unwrap() + ); + exact["loop_allthreads"] = json!({"nq": pa["nq"], "threads": "auto", "qps": pa["qps_1t"], "median_ms": pa["median_ms"], "p95_ms": pa["p95_ms"], "p99_ms": pa["p99_ms"], "mean_ms": pa["mean_ms"], "qps_runs": pa["qps_1t_runs"]}); + exact["batch_allthreads"] = batch_point(&store, &c, &truth, Path::Exact); + println!( + " exact batch qps={:.1}", + exact["batch_allthreads"]["qps"].as_f64().unwrap() + ); + // concurrent clients, each running single-threaded scans against the same store: measured, not assumed + store.set_scan_threads(1); + let conc = 8usize.min(cpus); + let qs = &c.queries[..c.queries.len().min(400)]; + let t = Instant::now(); + std::thread::scope(|sc| { + for w in 0..conc { + let st = &store; + sc.spawn(move || { + for q in qs.iter().skip(w).step_by(conc) { + st.similarity_search_vector_exact(q, 10, None, 0.0).unwrap(); + } + }); + } + }); + let conc_qps = qs.len() as f64 / t.elapsed().as_secs_f64(); + exact["concurrent_clients"] = json!({"clients": conc, "scan_threads": 1, "qps": conc_qps}); + println!( + " exact {conc} concurrent clients (scan threads 1): {:.1} qps", + conc_qps + ); + exact["mem_after_sweep"] = mem(); + runs.push(exact); + + // ---- ivf ---- + store.set_scan_threads(0); + store + .set_ann(Some(AnnConfig { + min_size: 0, + ..AnnConfig::default() + })) + .unwrap(); + let t = Instant::now(); + let stats = store.build_ann_index(false).unwrap(); + let idx_s = t.elapsed().as_secs_f64(); + let mem_after_build = mem(); + println!(" ivf build {:.2}s nlist={:?}", idx_s, stats.nlist); + let mut ivf = json!({ + "system": format!("{label}_ivf"), "ds": ds, "n": n, "d": d, "k": c.k, "queries": nq, "all_threads": cpus, + "version": "isongraph-vector-rs (repo HEAD, cargo release)", "knob_name": "nprobe", + "build": {"ingest_s": ingest_s, "cache_s": first_ms / 1000.0, "index_s": idx_s, "total_s": ingest_s + first_ms / 1000.0 + idx_s, + "first_query_ms": 0.0, "ready_s": ingest_s + first_ms / 1000.0 + idx_s}, + "mem_baseline": mem_base, "mem_after_ingest": mem_after_ingest, "mem_after_build": mem_after_build, + "index_stats": {"nlist": stats.nlist, "trained_n": stats.trained_n, "index_bytes": stats.index_bytes, "build_seconds": stats.build_seconds}, + }); + let mut pts = Vec::new(); + let mut ref_knob: Option<usize> = None; + let mut ref_recall = Value::Null; + for kn in knobs(n) { + store.set_ann_nprobe(Some(kn)).unwrap(); + store.set_scan_threads(1); + let mut p = pinned(|| point(&store, &c, &truth, Path::Ann, None, budget)); + // all-cores: the same loop with the library's automatic thread choice + store.set_scan_threads(0); + let pa = point(&store, &c, &truth, Path::Ann, None, budget.min(5.0)); + p["knob"] = json!(kn); + p["batch_allthreads"] = batch_point(&store, &c, &truth, Path::Ann); + p["allthreads"] = + json!({"qps": pa["qps_1t"], "median_ms": pa["median_ms"], "recall": pa["recall"]}); + println!( + " ivf nprobe={kn} recall={:.4} qps1t={:.0} median={:.3}ms allthreads_qps={:.0} batch_qps={:.0}", + p["recall"].as_f64().unwrap(), + p["qps_1t"].as_f64().unwrap(), + p["median_ms"].as_f64().unwrap(), + pa["qps_1t"].as_f64().unwrap(), + p["batch_allthreads"]["qps"].as_f64().unwrap() + ); + if ref_knob.is_none() && p["recall"].as_f64().unwrap() >= 0.95 { + ref_knob = Some(kn); + ref_recall = p["recall"].clone(); + } + pts.push(p); + } + if ref_knob.is_none() { + ref_knob = knobs(n).last().cloned(); + } + ivf["sweep"] = json!(pts); + ivf["ref_knob"] = json!(ref_knob); + ivf["ref_recall_pre"] = ref_recall.clone(); + ivf["mem_after_sweep"] = mem(); + + // ---- mutation ---- + if exps.iter().any(|e| e == "mutation") { + let gtm = format!("{dir}/gt_mut_{n}.i32"); + if std::path::Path::new(>m).exists() { + let upd: Vec<i32> = read_i32(&format!("{dir}/mut_{n}_upd.i32")); + let del: Vec<i32> = read_i32(&format!("{dir}/mut_{n}_del.i32")); + let truth_m: Vec<Vec<i32>> = read_i32(>m) + .chunks(100) + .map(|c| c.to_vec()) + .take(nq) + .collect(); + store.set_ann_nprobe(ref_knob).unwrap(); + store.set_scan_threads(0); + let mut m = json!({"ref_knob": ref_knob, "recall_before": ref_recall}); + let ins = c.base(n, 10000); + let t = Instant::now(); + for (i, v) in ins.into_iter().enumerate() { + store + .add(NodeId::new("doc", (n + i).to_string()), "", Some(v)) + .unwrap(); + } + let dt = t.elapsed().as_secs_f64(); + m["insert_10k"] = json!({"seconds": dt, "per_s": 10000.0 / dt}); + let newv = c.base(n + 10000, 10000); + let t = Instant::now(); + for (j, v) in newv.into_iter().enumerate() { + store + .add(NodeId::new("doc", upd[j].to_string()), "", Some(v)) + .unwrap(); + } + let dt = t.elapsed().as_secs_f64(); + m["update_10k"] = json!({"seconds": dt, "per_s": 10000.0 / dt}); + let t = Instant::now(); + for id in &del { + store.remove(&NodeId::new("doc", id.to_string())); + } + let dt = t.elapsed().as_secs_f64(); + m["delete_10k"] = json!({"seconds": dt, "per_s": 10000.0 / dt}); + store.set_scan_threads(1); + let t = Instant::now(); + let _ = store + .similarity_search_vector(&c.queries[0], c.k, None, 0.0) + .unwrap(); + m["first_query_after_ms"] = json!(t.elapsed().as_secs_f64() * 1000.0); + let p = pinned(|| point(&store, &c, &truth_m, Path::Ann, None, budget)); + m["recall_after"] = p["recall"].clone(); + m["after"] = p; + let st = store.ann_stats(); + m["index_stats_after"] = json!({"pending": st.pending, "churn": st.churn, "trained_n": st.trained_n, "builds": st.searches.approximate}); + m["mem_after"] = mem(); + println!( + " mutation: ins {:.2}s upd {:.2}s del {:.2}s recall_after={}", + m["insert_10k"]["seconds"].as_f64().unwrap(), + m["update_10k"]["seconds"].as_f64().unwrap(), + m["delete_10k"]["seconds"].as_f64().unwrap(), + m["recall_after"] + ); + ivf["mutation"] = m; + } else { + ivf["mutation"] = json!({"skipped": "dataset has no spare rows for this n"}); + } + } + + // ---- save / load (JSON sidecar, the only persistence this port has for vectors) ---- + if exps.iter().any(|e| e == "saveload") { + let path = format!("{dir}/../../work/rust_{ds}_{n}.json"); + let _ = std::fs::create_dir_all(format!("{dir}/../../work")); + let mut sl = json!({}); + let t = Instant::now(); + match store.save_embeddings(&path) { + Ok(_) => { + sl["save_s"] = json!(t.elapsed().as_secs_f64()); + sl["disk_mb"] = + json!(std::fs::metadata(&path).unwrap().len() as f64 / 1048576.0); + drop(store); + let mut s2 = EmbeddingStore::new(None); + let t = Instant::now(); + s2.load_embeddings(&path, false).unwrap(); + sl["load_s"] = json!(t.elapsed().as_secs_f64()); + s2.set_ann(Some(AnnConfig { + min_size: 0, + nprobe: ref_knob, + ..AnnConfig::default() + })) + .unwrap(); + let t = Instant::now(); + s2.similarity_search_vector(&c.queries[0], c.k, None, 0.0) + .unwrap(); + sl["first_query_after_load_ms"] = json!(t.elapsed().as_secs_f64() * 1000.0); + sl["time_to_ready_s"] = json!( + sl["load_s"].as_f64().unwrap() + + sl["first_query_after_load_ms"].as_f64().unwrap() / 1000.0 + ); + sl["note"] = json!("load = JSON parse + insert; first query includes the scan-cache build and the IVF index build (the index is never persisted)"); + let _ = std::fs::remove_file(&path); + } + Err(e) => sl["error"] = json!(format!("{e:?}")), + } + ivf["saveload"] = sl; + } + runs.push(ivf); + } + + if exps.iter().any(|e| e == "filter") { + let g = read_u8(&format!("{dir}/grp_{n}.u8")); + let mut store = EmbeddingStore::new(None); + let ingest_s = ingest(&mut store, &c, n, Some(&g)); + store.set_scan_threads(1); + store + .set_ann(Some(AnnConfig { + min_size: 0, + ..AnnConfig::default() + })) + .unwrap(); + let t = Instant::now(); + store.build_ann_index(false).unwrap(); + let idx_s = t.elapsed().as_secs_f64(); + let mut res = json!({"system": format!("{label}_ivf"), "ds": ds, "n": n, "d": d, "filter": {"build_with_metadata_s": ingest_s + idx_s, "mem": mem(), "subsets": {}}}); + let kn_all = knobs(n); + let pick: Vec<usize> = [ + 1usize, + kn_all.len() / 3, + kn_all.len() / 2, + 2 * kn_all.len() / 3, + kn_all.len() - 1, + ] + .iter() + .cloned() + .collect::<std::collections::BTreeSet<_>>() + .into_iter() + .map(|i| kn_all[i]) + .collect(); + for (ty, lab) in [("t10", "f10"), ("t01", "f01")] { + let truth = c.gt(&format!("{lab}_")); + let mut pts = Vec::new(); + for kn in &pick { + store.set_ann_nprobe(Some(*kn)).unwrap(); + let mut p = pinned(|| point(&store, &c, &truth, Path::Ann, Some(ty), budget)); + p["knob"] = json!(kn); + println!( + " filter {lab} nprobe={kn} recall={:.4} qps={:.0}", + p["recall"].as_f64().unwrap(), + p["qps_1t"].as_f64().unwrap() + ); + pts.push(p); + } + let mut pe = pinned(|| point(&store, &c, &truth, Path::Exact, Some(ty), budget)); + pe["knob"] = json!("exact"); + pts.push(pe); + res["filter"]["subsets"][lab] = json!(pts); + } + runs.push(res); + } + + if !out_path.is_empty() { + let doc = json!({"runs": runs, "status": "ok", "label": label}); + std::fs::write(&out_path, serde_json::to_string_pretty(&doc).unwrap()).unwrap(); + println!("wrote {out_path}"); + } +} diff --git a/isongraph-vector-rs/examples/engine_ab.rs b/isongraph-vector-rs/examples/engine_ab.rs new file mode 100644 index 0000000..d2a2796 --- /dev/null +++ b/isongraph-vector-rs/examples/engine_ab.rs @@ -0,0 +1,558 @@ +//! A/B harness for the engine work (concurrent and batch search, one copy of the +//! vectors, no stall after writes, deletes). Bench-only; reads the same raw +//! float32 data and float64 ground truth as `competitive_bench`. +//! +//! It uses only the API that exists both before and after the change, plus (with +//! `RUSTFLAGS="--cfg has_batch"`) the batch search, so the very same source is +//! built against the old commit and the new tree and run back to back. +//! +//! ```text +//! engine_ab --ds-dir D:/.../prep/text384 --n 100000 --exp scaling,batch --queries 2000 +//! engine_ab --ds-dir D:/.../prep/text384 --n 1000000 --exp writes,deletes +//! ``` +//! Experiments: `scaling` (N concurrent single-query callers), `batch` (batch +//! throughput by thread count, equality with per-query answers, recall), +//! `writes` (insert/update/delete throughput and write pauses on a built IVF +//! index, the first query after them), `deletes` (delete cost, by store state). +#![allow(unexpected_cfgs, dead_code)] + +use isongraph_vector_rs::{AnnConfig, EmbeddingStore, NodeId, SimilarityResult}; +use serde_json::{json, Value}; +use std::collections::HashSet; +use std::fs::File; +use std::io::{Read, Seek, SeekFrom}; +use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; +use std::time::{Duration, Instant}; + +#[cfg(windows)] +#[repr(C)] +struct Pmc { + cb: u32, + page_fault_count: u32, + peak_working_set_size: usize, + working_set_size: usize, + quota_peak_paged_pool_usage: usize, + quota_paged_pool_usage: usize, + quota_peak_non_paged_pool_usage: usize, + quota_non_paged_pool_usage: usize, + pagefile_usage: usize, + peak_pagefile_usage: usize, +} +#[cfg(windows)] +#[link(name = "kernel32")] +extern "system" { + fn GetCurrentProcess() -> isize; + fn K32GetProcessMemoryInfo(h: isize, p: *mut Pmc, cb: u32) -> i32; + fn SetProcessAffinityMask(h: isize, mask: usize) -> i32; + fn SetPriorityClass(h: isize, class: u32) -> i32; +} + +/// Pin to one P-core for a single-threaded timed loop (see README_method.md). +fn pinned<T>(f: impl FnOnce() -> T) -> T { + #[cfg(windows)] + unsafe { + let cpu: usize = std::env::var("BENCH_PIN_CPU") + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(8); + SetPriorityClass(GetCurrentProcess(), 0x80); + SetProcessAffinityMask(GetCurrentProcess(), 1usize << cpu); + let r = f(); + SetProcessAffinityMask( + GetCurrentProcess(), + usize::MAX + >> (usize::BITS as usize + - std::thread::available_parallelism() + .map(|n| n.get()) + .unwrap_or(32)), + ); + return r; + } + #[allow(unreachable_code)] + f() +} +fn rss_mb() -> f64 { + #[cfg(windows)] + unsafe { + let mut p: Pmc = std::mem::zeroed(); + p.cb = std::mem::size_of::<Pmc>() as u32; + K32GetProcessMemoryInfo(GetCurrentProcess(), &mut p, p.cb); + return p.working_set_size as f64 / 1048576.0; + } + #[allow(unreachable_code)] + 0.0 +} +fn peak_mb() -> f64 { + #[cfg(windows)] + unsafe { + let mut p: Pmc = std::mem::zeroed(); + p.cb = std::mem::size_of::<Pmc>() as u32; + K32GetProcessMemoryInfo(GetCurrentProcess(), &mut p, p.cb); + return p.peak_working_set_size as f64 / 1048576.0; + } + #[allow(unreachable_code)] + 0.0 +} + +fn read_f32(path: &str, start: usize, count: usize, d: usize) -> Vec<f32> { + let mut f = File::open(path).unwrap_or_else(|e| panic!("{path}: {e}")); + f.seek(SeekFrom::Start((start * d * 4) as u64)).unwrap(); + let mut buf = vec![0u8; count * d * 4]; + f.read_exact(&mut buf).unwrap(); + (0..buf.len() / 4) + .map(|i| f32::from_le_bytes(buf[4 * i..4 * i + 4].try_into().unwrap())) + .collect() +} +fn read_i32(path: &str) -> Vec<i32> { + let mut b = Vec::new(); + File::open(path) + .unwrap_or_else(|e| panic!("{path}: {e}")) + .read_to_end(&mut b) + .unwrap(); + (0..b.len() / 4) + .map(|i| i32::from_le_bytes(b[4 * i..4 * i + 4].try_into().unwrap())) + .collect() +} + +fn arg(args: &[String], name: &str, default: &str) -> String { + args.iter() + .position(|a| a == name) + .map(|i| args[i + 1].clone()) + .unwrap_or(default.to_string()) +} +fn median(v: &[f64]) -> f64 { + let mut s = v.to_vec(); + s.sort_by(f64::total_cmp); + if s.is_empty() { + 0.0 + } else { + s[s.len() / 2] + } +} +fn pct(v: &[f64], p: f64) -> f64 { + let mut s = v.to_vec(); + s.sort_by(f64::total_cmp); + if s.is_empty() { + 0.0 + } else { + s[((s.len() - 1) as f64 * p).round() as usize] + } +} + +struct Ctx { + dir: String, + d: usize, + n: usize, + k: usize, + queries: Vec<Vec<f32>>, +} +impl Ctx { + fn base(&self, start: usize, count: usize) -> Vec<Vec<f32>> { + read_f32(&format!("{}/base.f32", self.dir), start, count, self.d) + .chunks(self.d) + .map(|c| c.to_vec()) + .collect() + } + fn gt(&self) -> Vec<Vec<i32>> { + read_i32(&format!("{}/gt_{}.i32", self.dir, self.n)) + .chunks(100) + .map(|c| c.to_vec()) + .collect() + } +} + +fn ids(r: &[SimilarityResult]) -> Vec<usize> { + r.iter() + .map(|x| x.node_id.id.parse::<usize>().unwrap()) + .collect() +} +fn recall(found: &[Vec<usize>], truth: &[Vec<i32>], k: usize) -> f64 { + let mut hit = 0usize; + for (f, t) in found.iter().zip(truth) { + let want: HashSet<i32> = t.iter().take(k).cloned().collect(); + hit += f + .iter() + .take(k) + .filter(|x| want.contains(&(**x as i32))) + .count(); + } + hit as f64 / (k * found.len().max(1)) as f64 +} + +fn ingest(s: &mut EmbeddingStore, c: &Ctx, n: usize) -> f64 { + let mut secs = 0.0; + let step = 20000; + let mut at = 0; + while at < n { + let m = step.min(n - at); + let rows = c.base(at, m); + let t = Instant::now(); + for (i, v) in rows.into_iter().enumerate() { + s.add(NodeId::new("doc", (at + i).to_string()), "", Some(v)) + .unwrap(); + } + secs += t.elapsed().as_secs_f64(); + at += m; + } + secs +} + +fn built(c: &Ctx, ivf: bool, nprobe: usize) -> EmbeddingStore { + let mut s = EmbeddingStore::new(None); + ingest(&mut s, c, c.n); + if ivf { + s.set_ann(Some(AnnConfig { + min_size: 0, + nprobe: Some(nprobe), + ..AnnConfig::default() + })) + .unwrap(); + s.build_ann_index(false).unwrap(); + } + s +} + +/// Total QPS of `clients` threads each calling the single-query search in a loop for `secs`. +fn clients_qps( + s: &EmbeddingStore, + qs: &[Vec<f32>], + k: usize, + clients: usize, + secs: f64, + exact: bool, +) -> f64 { + let stop = AtomicBool::new(false); + let total = AtomicUsize::new(0); + let t0 = Instant::now(); + std::thread::scope(|sc| { + for w in 0..clients { + let (stop, total) = (&stop, &total); + sc.spawn(move || { + let mut i = w * 7919; + let mut done = 0usize; + while !stop.load(Ordering::Relaxed) { + let q = &qs[i % qs.len()]; + if exact { + s.similarity_search_vector_exact(q, k, None, 0.0).unwrap(); + } else { + s.similarity_search_vector(q, k, None, 0.0).unwrap(); + } + i += 1; + done += 1; + } + total.fetch_add(done, Ordering::Relaxed); + }); + } + std::thread::sleep(Duration::from_secs_f64(secs)); + stop.store(true, Ordering::Relaxed); + }); + total.load(Ordering::Relaxed) as f64 / t0.elapsed().as_secs_f64() +} + +fn main() { + let args: Vec<String> = std::env::args().skip(1).collect(); + let dir = arg(&args, "--ds-dir", ""); + let n: usize = arg(&args, "--n", "100000").parse().unwrap(); + let nq: usize = arg(&args, "--queries", "2000").parse().unwrap(); + let nprobe: usize = arg(&args, "--nprobe", "128").parse().unwrap(); + let label = arg(&args, "--label", "run"); + let secs: f64 = arg(&args, "--secs", "3").parse().unwrap(); + let exps: Vec<String> = arg(&args, "--exp", "scaling") + .split(',') + .map(|s| s.to_string()) + .collect(); + let out_path = arg(&args, "--out", ""); + let clients_list: Vec<usize> = arg(&args, "--clients", "1,2,4,8,16,24,32") + .split(',') + .map(|s| s.parse().unwrap()) + .collect(); + let meta: Value = + serde_json::from_str(&std::fs::read_to_string(format!("{dir}/meta.json")).unwrap()) + .unwrap(); + let d = meta["d"].as_u64().unwrap() as usize; + let qall = read_f32(&format!("{dir}/queries.f32"), 0, nq, d); + let c = Ctx { + dir: dir.clone(), + d, + n, + k: 10, + queries: qall.chunks(d).map(|x| x.to_vec()).collect(), + }; + let mut out = json!({"label": label, "n": n, "d": d, "nprobe": nprobe}); + println!( + "{label}: n={n} d={d} queries={nq} nprobe={nprobe} has_batch={}", + cfg!(has_batch) + ); + let rss0 = rss_mb(); + + if exps.iter().any(|e| e == "scaling") { + let mut res = json!({}); + for (mode, ivf) in [("exact", false), ("ivf", true)] { + let mut s = built(&c, ivf, nprobe); + s.set_scan_threads(1); + let _ = clients_qps(&s, &c.queries, c.k, 1, 0.5, !ivf); // warm + let mut rows = Vec::new(); + let mut base = 0.0; + for &cl in &clients_list { + let runs: Vec<f64> = (0..3) + .map(|_| clients_qps(&s, &c.queries, c.k, cl, secs, !ivf)) + .collect(); + let q = median(&runs); + if cl == 1 { + base = q; + } + println!( + " scaling {mode} clients={cl:2} qps={q:9.1} efficiency={:.2}", + q / (base * cl as f64).max(1e-9) + ); + rows.push(json!({"clients": cl, "qps": q, "runs": runs, "efficiency": q / (base * cl as f64).max(1e-9)})); + } + res[mode] = json!(rows); + } + out["scaling"] = res; + } + + if exps.iter().any(|e| e == "mem") { + // working set after each stage, minus the baseline before any data + let base = rss_mb(); + let raw_mb = (n * d * 4) as f64 / 1048576.0; + let mut s = EmbeddingStore::new(None); + let ingest_s = ingest(&mut s, &c, n); + let after_ingest = rss_mb() - base; + let peak_ingest = peak_mb() - base; + s.similarity_search_vector_exact(&c.queries[0], c.k, None, 0.0) + .unwrap(); + let after_first = rss_mb() - base; + s.set_ann(Some(AnnConfig { + min_size: 0, + nprobe: Some(nprobe), + ..AnnConfig::default() + })) + .unwrap(); + s.build_ann_index(false).unwrap(); + let after_index = rss_mb() - base; + let peak_index = peak_mb() - base; + println!(" mem raw={raw_mb:.0} MB: after ingest {after_ingest:.0} ({:.2}x raw, peak {peak_ingest:.0}), after first query {after_first:.0}, after IVF build {after_index:.0} ({:.2}x raw, peak {peak_index:.0}); ingest {ingest_s:.1}s", after_ingest / raw_mb, after_index / raw_mb); + out["mem"] = json!({"raw_mb": raw_mb, "after_ingest_mb": after_ingest, "peak_after_ingest_mb": peak_ingest, "after_first_query_mb": after_first, "after_ivf_build_mb": after_index, "peak_after_build_mb": peak_index, "ingest_s": ingest_s}); + } + + if exps.iter().any(|e| e == "loop") { + // one-query-at-a-time loops by thread cap: latency of a single scan + let mut s = built(&c, false, nprobe); + let mut rows = Vec::new(); + for threads in [1usize, 2, 4, 8, 0] { + s.set_scan_threads(threads); + let run = |s: &EmbeddingStore| { + let t = Instant::now(); + for q in c.queries.iter().take(150) { + s.similarity_search_vector_exact(q, c.k, None, 0.0).unwrap(); + } + 150.0 / t.elapsed().as_secs_f64() + }; + let _ = run(&s); + let runs: Vec<f64> = (0..5).map(|_| run(&s)).collect(); + println!( + " exact loop threads={threads} qps={:.1} (min {:.1} max {:.1})", + median(&runs), + runs.iter().cloned().fold(f64::MAX, f64::min), + runs.iter().cloned().fold(0.0, f64::max) + ); + rows.push(json!({"threads": threads, "qps": median(&runs), "runs": runs})); + } + out["loop"] = json!(rows); + } + + if exps.iter().any(|e| e == "batch") { + #[cfg(has_batch)] + { + let truth = c.gt(); + let mut res = json!({}); + for (mode, ivf) in [("exact", false), ("ivf", true)] { + let mut s = built(&c, ivf, nprobe); + let mut rows = Vec::new(); + let per_query: Vec<Vec<usize>> = c + .queries + .iter() + .take(200) + .map(|q| { + ids(&if ivf { + s.similarity_search_vector(q, c.k, None, 0.0) + } else { + s.similarity_search_vector_exact(q, c.k, None, 0.0) + } + .unwrap()) + }) + .collect(); + let pool = s.search_pool().threads(); + for threads in [1usize, 2, 4, 8, 16, 24, 32] { + if threads > pool { + continue; + } + s.set_scan_threads(threads); + let run = |s: &EmbeddingStore| { + let t = Instant::now(); + let r = if ivf { + s.similarity_search_vector_batch(&c.queries, c.k, None, 0.0) + } else { + s.similarity_search_vector_exact_batch(&c.queries, c.k, None, 0.0) + } + .unwrap(); + (t.elapsed().as_secs_f64(), r) + }; + let _ = run(&s); // warm + let mut times = Vec::new(); + let mut last = Vec::new(); + for _ in 0..3 { + let (t, r) = run(&s); + times.push(c.queries.len() as f64 / t); + last = r; + } + let found: Vec<Vec<usize>> = last.iter().map(|r| ids(r)).collect(); + let identical = found.iter().take(200).zip(&per_query).all(|(a, b)| a == b); + let rec = recall(&found, &truth, c.k); + let qps = median(×); + println!(" batch {mode} threads={threads:2} qps={qps:9.1} recall={rec:.4} equals_single(first200)={identical}"); + rows.push(json!({"threads": threads, "qps": qps, "runs": times, "recall": rec, "equals_single": identical})); + } + res[mode] = json!(rows); + } + out["batch"] = res; + } + #[cfg(not(has_batch))] + println!(" batch: not available in this build"); + } + + if exps.iter().any(|e| e == "writes") { + // an IVF store at ref nprobe; then 30k pure inserts, 10k updates, 10k deletes + let rss_before = rss_mb(); + let mut s = built(&c, true, nprobe); + let rss_built = rss_mb(); + let mut res = json!({"rss_built_mb": rss_built - rss_before, "peak_mb": peak_mb() - rss0}); + let probe = |s: &EmbeddingStore, label: &str| { + let t = Instant::now(); + let _ = s + .similarity_search_vector(&c.queries[0], c.k, None, 0.0) + .unwrap(); + let first = t.elapsed().as_secs_f64() * 1000.0; + let lat: Vec<f64> = c + .queries + .iter() + .take(300) + .map(|q| { + let t = Instant::now(); + let _ = s.similarity_search_vector(q, c.k, None, 0.0).unwrap(); + t.elapsed().as_secs_f64() * 1000.0 + }) + .collect(); + println!( + " {label}: first query {first:.2} ms, then median {:.3} ms p99 {:.3} ms", + median(&lat), + pct(&lat, 0.99) + ); + json!({"first_ms": first, "median_ms": median(&lat), "p99_ms": pct(&lat, 0.99)}) + }; + res["before"] = probe(&s, "built"); + s.set_scan_threads(1); + res["before_1t"] = probe(&s, "built (1 thread)"); + s.set_scan_threads(0); + let upd: Vec<i32> = read_i32(&format!("{dir}/mut_{n}_upd.i32")); + let del: Vec<i32> = read_i32(&format!("{dir}/mut_{n}_del.i32")); + let batch = 30000usize; + let ins = c.base(n, batch); + let mut per_write = Vec::with_capacity(batch); + let t = Instant::now(); + for (i, v) in ins.into_iter().enumerate() { + let t1 = Instant::now(); + s.add(NodeId::new("doc", (n + i).to_string()), "", Some(v)) + .unwrap(); + per_write.push(t1.elapsed().as_secs_f64() * 1000.0); + } + let dt = t.elapsed().as_secs_f64(); + println!( + " insert 30k: {:.2}s = {:.0}/s, per-write median {:.4} ms p99 {:.4} ms max {:.3} ms", + dt, + batch as f64 / dt, + median(&per_write), + pct(&per_write, 0.99), + pct(&per_write, 1.0) + ); + res["insert_30k"] = json!({"seconds": dt, "per_s": batch as f64 / dt, "median_ms": median(&per_write), "p99_ms": pct(&per_write, 0.99), "max_ms": pct(&per_write, 1.0)}); + s.set_scan_threads(1); + res["after_30k_inserts"] = probe(&s, "after 30k inserts (1 thread)"); + s.set_scan_threads(0); + let newv = c.base(n + 40000, 10000); + let t = Instant::now(); + let mut pw = Vec::new(); + for (j, v) in newv.into_iter().enumerate() { + let t1 = Instant::now(); + s.add(NodeId::new("doc", upd[j].to_string()), "", Some(v)) + .unwrap(); + pw.push(t1.elapsed().as_secs_f64() * 1000.0); + } + let dt = t.elapsed().as_secs_f64(); + println!( + " update 10k: {:.0}/s max {:.3} ms", + 10000.0 / dt, + pct(&pw, 1.0) + ); + res["update_10k"] = json!({"per_s": 10000.0 / dt, "max_ms": pct(&pw, 1.0)}); + let t = Instant::now(); + for id in &del { + s.remove(&NodeId::new("doc", id.to_string())); + } + let dt = t.elapsed().as_secs_f64(); + println!(" delete 10k: {:.0}/s", 10000.0 / dt); + res["delete_10k"] = json!({"per_s": 10000.0 / dt}); + s.set_scan_threads(1); + res["after_all"] = probe(&s, "after 30k writes total (1 thread)"); + res["rss_after_mb"] = json!(rss_mb() - rss0); + out["writes"] = res; + } + + if exps.iter().any(|e| e == "deletes") { + let mut res = json!({}); + // random delete order over the whole store, so every delete is a cache miss + let mut order: Vec<usize> = (0..n).collect(); + let mut x = 0x9E3779B97F4A7C15u64; + for i in (1..order.len()).rev() { + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + order.swap(i, (x % (i as u64 + 1)) as usize); + } + for (label, search_first, ivf) in [ + ("never_searched", false, false), + ("after_exact_search", true, false), + ("ivf_built", true, true), + ] { + let mut s = built(&c, ivf, nprobe); + if search_first { + s.similarity_search_vector_exact(&c.queries[0], c.k, None, 0.0) + .unwrap(); + } + let ndel = 10000usize.min(n / 4); + let keys: Vec<NodeId> = order + .iter() + .take(ndel) + .map(|i| NodeId::new("doc", i.to_string())) + .collect(); + let t = Instant::now(); + for k in &keys { + s.remove(k); + } + let dt = t.elapsed().as_secs_f64(); + println!( + " delete {ndel} ({label}): {:.0}/s = {:.0} ns each", + ndel as f64 / dt, + dt / ndel as f64 * 1e9 + ); + res[label] = json!({"per_s": ndel as f64 / dt, "ns_each": dt / ndel as f64 * 1e9}); + } + out["deletes"] = res; + } + + if !out_path.is_empty() { + std::fs::write(&out_path, serde_json::to_string_pretty(&out).unwrap()).unwrap(); + println!("wrote {out_path}"); + } +} diff --git a/isongraph-vector-rs/examples/hnsw_bench.rs b/isongraph-vector-rs/examples/hnsw_bench.rs new file mode 100644 index 0000000..14603fd --- /dev/null +++ b/isongraph-vector-rs/examples/hnsw_bench.rs @@ -0,0 +1,568 @@ +//! The opt-in HNSW index: recall@10 versus queries/second across `ef_search`, +//! build time, memory, and the cost of one insert / update / delete. +//! +//! Data: with `--ds-dir DIR` the real vectors, queries and float64 ground truth +//! of `benchmarks/competitive/` (the `prep/text384` layout: `base.f32`, +//! `queries.f32`, `gt_<n>.i32`, `meta.json`); without it, synthetic clustered +//! unit vectors (ground truth from the exact scan). +//! +//! ```text +//! cargo run --release --example hnsw_bench -- --n 100000 +//! cargo run --release --example hnsw_bench -- --ds-dir D:/.../prep/text384 --n 100000 --ivf +//! cargo run --release --example hnsw_bench -- --ds-dir D:/.../prep/text384 --n 1000000 --ef 16,32,64,128,256 +//! ``` +//! +//! Options: `--n N` `--dim D` (synthetic) `--queries Q` `--ef a,b,c` `--m M` +//! `--efc E` `--threads T` (build/batch pool cap, 0 = all) `--ivf` (also sweep +//! IVF `nprobe` on the same data; `--ivf-only` skips HNSW) `--budget SECONDS` (per timed loop) `--pin` +//! (single-thread loops on one pinned CPU, Windows). +//! +//! Timings are wall-clock on whatever the machine is doing; when other work +//! runs, read them as indicative. Recall and memory are not affected. + +use isongraph_vector_rs::{AnnConfig, EmbeddingStore, HnswConfig, NodeId}; +use std::collections::HashSet; +use std::fs::File; +use std::io::{Read, Seek, SeekFrom}; +use std::time::Instant; + +// ---- process memory (Windows working set), no extra crates ------------------------------------ +#[cfg(windows)] +mod winmem { + #[repr(C)] + struct Pmc { + cb: u32, + page_fault_count: u32, + peak_working_set_size: usize, + working_set_size: usize, + rest: [usize; 6], + } + #[link(name = "kernel32")] + extern "system" { + fn GetCurrentProcess() -> isize; + fn K32GetProcessMemoryInfo(h: isize, p: *mut Pmc, cb: u32) -> i32; + } + /// (working set, peak working set) in MB. + pub fn mb() -> (f64, f64) { + // SAFETY: Pmc matches PROCESS_MEMORY_COUNTERS and `cb` is its size. + let p = unsafe { + let mut p: Pmc = std::mem::zeroed(); + p.cb = std::mem::size_of::<Pmc>() as u32; + K32GetProcessMemoryInfo(GetCurrentProcess(), &mut p, p.cb); + p + }; + ( + p.working_set_size as f64 / 1048576.0, + p.peak_working_set_size as f64 / 1048576.0, + ) + } +} +#[cfg(not(windows))] +mod winmem { + pub fn mb() -> (f64, f64) { + (0.0, 0.0) + } +} + +/// With `--pin`, run `f` on one logical CPU (`BENCH_PIN_CPU`, default 8) at high +/// priority: single-thread loops vary a lot between runs on a hybrid CPU +/// otherwise. Windows only; elsewhere it just runs `f`. +fn pinned<T>(pin: bool, f: impl FnOnce() -> T) -> T { + #[cfg(windows)] + if pin { + #[link(name = "kernel32")] + extern "system" { + fn GetCurrentProcess() -> isize; + fn SetProcessAffinityMask(h: isize, mask: usize) -> i32; + fn SetPriorityClass(h: isize, class: u32) -> i32; + } + let cpu: usize = std::env::var("BENCH_PIN_CPU") + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(8); + let all = usize::MAX + >> (usize::BITS as usize + - std::thread::available_parallelism() + .map(|n| n.get()) + .unwrap_or(32)); + // SAFETY: plain kernel32 calls on the current process. + unsafe { + SetPriorityClass(GetCurrentProcess(), 0x80); // HIGH_PRIORITY_CLASS + SetProcessAffinityMask(GetCurrentProcess(), 1usize << cpu); + } + let r = f(); + unsafe { SetProcessAffinityMask(GetCurrentProcess(), all) }; + return r; + } + let _ = pin; + f() +} + +// ---- data ------------------------------------------------------------------------------------- +struct Rng(u64); + +impl Rng { + fn next_u64(&mut self) -> u64 { + self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } + fn uniform(&mut self) -> f32 { + ((self.next_u64() >> 40) as f32 + 0.5) / (1u64 << 24) as f32 + } + fn below(&mut self, n: usize) -> usize { + ((self.next_u64() as u128 * n as u128) >> 64) as usize + } + fn gauss(&mut self) -> f32 { + (-2.0 * self.uniform().ln()).sqrt() * (std::f32::consts::TAU * self.uniform()).cos() + } +} + +fn unit(v: &mut [f32]) { + let norm = v.iter().map(|x| x * x).sum::<f32>().sqrt(); + if norm > 0.0 { + v.iter_mut().for_each(|x| *x /= norm); + } +} + +fn read_f32(path: &str, start: usize, count: usize, d: usize) -> Vec<f32> { + let mut f = File::open(path).unwrap_or_else(|e| panic!("{path}: {e}")); + f.seek(SeekFrom::Start((start * d * 4) as u64)).unwrap(); + let mut buf = vec![0u8; count * d * 4]; + f.read_exact(&mut buf).unwrap(); + (0..buf.len() / 4) + .map(|i| f32::from_le_bytes(buf[4 * i..4 * i + 4].try_into().unwrap())) + .collect() +} + +fn read_i32(path: &str) -> Vec<i32> { + let mut b = Vec::new(); + File::open(path) + .unwrap_or_else(|e| panic!("{path}: {e}")) + .read_to_end(&mut b) + .unwrap(); + (0..b.len() / 4) + .map(|i| i32::from_le_bytes(b[4 * i..4 * i + 4].try_into().unwrap())) + .collect() +} + +/// Where the vectors come from. +enum Source { + Files { dir: String, d: usize }, + Synthetic { centers: Vec<Vec<f32>>, d: usize }, +} + +impl Source { + fn dim(&self) -> usize { + match self { + Source::Files { d, .. } | Source::Synthetic { d, .. } => *d, + } + } + + /// Base vectors `start..start + count`. + fn base(&self, start: usize, count: usize) -> Vec<Vec<f32>> { + match self { + Source::Files { dir, d } => read_f32(&format!("{dir}/base.f32"), start, count, *d) + .chunks(*d) + .map(|c| c.to_vec()) + .collect(), + Source::Synthetic { centers, d } => (start..start + count) + .map(|i| Self::synth(centers, *d, 0x1000 + i as u64)) + .collect(), + } + } + + fn queries(&self, count: usize) -> Vec<Vec<f32>> { + match self { + Source::Files { dir, d } => read_f32(&format!("{dir}/queries.f32"), 0, count, *d) + .chunks(*d) + .map(|c| c.to_vec()) + .collect(), + Source::Synthetic { centers, d } => (0..count) + .map(|i| Self::synth(centers, *d, 0xFFFF_0000 + i as u64)) + .collect(), + } + } + + fn synth(centers: &[Vec<f32>], d: usize, key: u64) -> Vec<f32> { + let mut rng = Rng(key.wrapping_mul(0x2545_F491_4F6C_DD1D)); + let c = ¢ers[rng.below(centers.len())]; + let scale = 1.0 / (d as f32).sqrt(); + let mut v: Vec<f32> = c.iter().map(|x| x + rng.gauss() * scale).collect(); + unit(&mut v); + v + } +} + +fn arg(args: &[String], name: &str, default: &str) -> String { + args.iter() + .position(|a| a == name) + .map(|i| args[i + 1].clone()) + .unwrap_or_else(|| default.to_string()) +} + +fn ingest(store: &mut EmbeddingStore, src: &Source, n: usize) -> f64 { + let mut secs = 0.0; + let step = 20_000; + let mut at = 0; + while at < n { + let m = step.min(n - at); + let rows = src.base(at, m); + let t = Instant::now(); + for (i, v) in rows.into_iter().enumerate() { + store + .add(NodeId::new("doc", (at + i).to_string()), "", Some(v)) + .unwrap(); + } + secs += t.elapsed().as_secs_f64(); + at += m; + } + secs +} + +fn ids_of(store: &EmbeddingStore, q: &[f32], k: usize, ef: Option<usize>) -> Vec<usize> { + let r = match ef { + Some(ef) => store.similarity_search_vector_ef(q, k, None, 0.0, ef), + None => store.similarity_search_vector(q, k, None, 0.0), + } + .unwrap(); + r.iter().map(|x| x.node_id.id.parse().unwrap()).collect() +} + +fn recall(found: &[Vec<usize>], truth: &[Vec<usize>], k: usize) -> f64 { + let mut hit = 0usize; + for (f, t) in found.iter().zip(truth) { + let want: HashSet<usize> = t.iter().take(k).copied().collect(); + hit += f.iter().take(k).filter(|x| want.contains(x)).count(); + } + hit as f64 / (k * found.len().max(1)) as f64 +} + +fn pct(sorted: &[f64], p: f64) -> f64 { + if sorted.is_empty() { + return 0.0; + } + sorted[((sorted.len() - 1) as f64 * p).round() as usize] +} + +struct Point { + label: String, + recall: f64, + qps_1t: f64, + median_ms: f64, + p99_ms: f64, + qps_batch: f64, +} + +/// One accuracy setting: recall, single-thread loop (latency + QPS), batch QPS. +#[allow(clippy::too_many_arguments)] +fn measure( + store: &mut EmbeddingStore, + threads: usize, + queries: &[Vec<f32>], + truth: &[Vec<usize>], + k: usize, + ef: Option<usize>, + budget: f64, + pin: bool, + label: String, +) -> Point { + let found: Vec<Vec<usize>> = queries.iter().map(|q| ids_of(store, q, k, ef)).collect(); + let rec = recall(&found, truth, k); + // single-threaded one-query-at-a-time loop, each query timed + let mut lat = Vec::new(); + store.set_scan_threads(1); + let loop_start = Instant::now(); + pinned(pin, || { + 'outer: for _ in 0..3 { + for q in queries { + let t = Instant::now(); + let _ = ids_of(store, q, k, ef); + lat.push(t.elapsed().as_secs_f64()); + if loop_start.elapsed().as_secs_f64() > budget { + break 'outer; + } + } + } + }); + let total = loop_start.elapsed().as_secs_f64(); + store.set_scan_threads(threads); + let mut ms: Vec<f64> = lat.iter().map(|x| x * 1000.0).collect(); + ms.sort_by(f64::total_cmp); + // batch over all threads (HNSW has no per-call ef in a batch: it uses the configured one) + let t = Instant::now(); + let mut done = 0usize; + while t.elapsed().as_secs_f64() < budget.min(3.0) { + let _ = store + .similarity_search_vector_batch(queries, k, None, 0.0) + .unwrap(); + done += queries.len(); + } + let qps_batch = done as f64 / t.elapsed().as_secs_f64(); + Point { + label, + recall: rec, + qps_1t: lat.len() as f64 / total, + median_ms: pct(&ms, 0.5), + p99_ms: pct(&ms, 0.99), + qps_batch, + } +} + +fn print_header(title: &str) { + println!("\n{title}"); + println!( + " {:<12} {:>8} {:>10} {:>10} {:>10} {:>12}", + "setting", "recall", "qps(1T)", "median ms", "p99 ms", "qps(batch)" + ); +} + +fn print_point(p: &Point) { + println!( + " {:<12} {:>8.4} {:>10.0} {:>10.3} {:>10.3} {:>12.0}", + p.label, p.recall, p.qps_1t, p.median_ms, p.p99_ms, p.qps_batch + ); +} + +fn main() { + let args: Vec<String> = std::env::args().skip(1).collect(); + let n: usize = arg(&args, "--n", "50000").parse().unwrap(); + let nq: usize = arg(&args, "--queries", "1000").parse().unwrap(); + let k = 10usize; + let budget: f64 = arg(&args, "--budget", "6").parse().unwrap(); + let m: usize = arg(&args, "--m", "16").parse().unwrap(); + let efc: usize = arg(&args, "--efc", "200").parse().unwrap(); + let threads: usize = arg(&args, "--threads", "0").parse().unwrap(); + let efs: Vec<usize> = arg(&args, "--ef", "10,16,24,32,48,64,96,128,256") + .split(',') + .map(|s| s.parse().unwrap()) + .collect(); + let with_ivf = args.iter().any(|a| a == "--ivf"); + let pin = args.iter().any(|a| a == "--pin"); + let ivf_only = args.iter().any(|a| a == "--ivf-only"); + let dir = arg(&args, "--ds-dir", ""); + let cpus = std::thread::available_parallelism() + .map(|x| x.get()) + .unwrap_or(1); + + let src = if dir.is_empty() { + let d: usize = arg(&args, "--dim", "384").parse().unwrap(); + let mut rng = Rng(4242); + let clusters = (n / 100).max(8); + let centers = (0..clusters) + .map(|_| { + let mut c: Vec<f32> = (0..d).map(|_| rng.gauss()).collect(); + unit(&mut c); + c + }) + .collect(); + Source::Synthetic { centers, d } + } else { + let meta: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(format!("{dir}/meta.json")).unwrap()) + .unwrap(); + Source::Files { + dir: dir.clone(), + d: meta["d"].as_u64().unwrap() as usize, + } + }; + let d = src.dim(); + println!( + "hnsw_bench: n={n} d={d} queries={nq} cpus={cpus} threads={threads} M={m} efC={efc} data={} release={}", + if dir.is_empty() { "synthetic clustered" } else { &dir }, + !cfg!(debug_assertions) + ); + println!("(timings are indicative: the machine is shared)"); + + let mut store = EmbeddingStore::new(None); + store.set_scan_threads(threads); + let rss0 = winmem::mb(); + let ingest_s = ingest(&mut store, &src, n); + let rss_ingest = winmem::mb(); + println!( + "ingest {:.2}s ({:.0} vec/s); working set {:.0} MB (raw vectors {:.0} MB)", + ingest_s, + n as f64 / ingest_s, + rss_ingest.0 - rss0.0, + (n * d * 4) as f64 / 1048576.0 + ); + let queries = src.queries(nq); + + // ground truth: the files' float64 truth, or the exact scan + let truth: Vec<Vec<usize>> = if dir.is_empty() { + let t = Instant::now(); + let truth = queries + .iter() + .map(|q| { + store + .similarity_search_vector_exact(q, 100, None, -2.0) + .unwrap() + .iter() + .map(|r| r.node_id.id.parse().unwrap()) + .collect() + }) + .collect(); + println!( + "exact ground truth ({:.1}s, {:.1} ms/query)", + t.elapsed().as_secs_f64(), + t.elapsed().as_secs_f64() * 1000.0 / nq as f64 + ); + truth + } else { + read_i32(&format!("{dir}/gt_{n}.i32")) + .chunks(100) + .take(nq) + .map(|c| c.iter().map(|&x| x as usize).collect()) + .collect() + }; + + // ---- HNSW ---- + if !ivf_only { + store + .set_ann(Some(AnnConfig { + min_size: 0, + hnsw: HnswConfig { + m, + ef_construction: efc, + ef_search: None, + }, + ..AnnConfig::hnsw() + })) + .unwrap(); + let rss_before = winmem::mb(); + let t = Instant::now(); + let stats = store.build_ann_index(false).unwrap(); + let build_s = t.elapsed().as_secs_f64(); + let rss_after = winmem::mb(); + let mem = store.memory_stats(); + println!( + "\nHNSW build {:.1}s ({:.0} nodes/s), max level {:?}", + build_s, + n as f64 / build_s, + stats.max_level + ); + println!( + " index {:.1} MB = {:.1} B/node = {:.3}x the raw vectors; whole store {:.3}x raw; working set +{:.0} MB, peak {:.0} MB", + mem.index_bytes as f64 / 1048576.0, + mem.index_bytes as f64 / n as f64, + mem.index_bytes as f64 / mem.live_vector_bytes as f64, + mem.total_bytes() as f64 / mem.live_vector_bytes as f64, + rss_after.0 - rss_before.0, + rss_after.1 + ); + + print_header("HNSW: recall@10 vs speed over ef_search"); + let mut points = Vec::new(); + for &ef in &efs { + // the batch path uses the configured ef, so set it for the batch too + store.set_ann_ef_search(Some(ef)).unwrap(); + let p = measure( + &mut store, + threads, + &queries, + &truth, + k, + Some(ef), + budget, + pin, + format!("ef={ef}"), + ); + print_point(&p); + points.push(p); + } + store.set_ann_ef_search(None).unwrap(); + let default = measure( + &mut store, + threads, + &queries, + &truth, + k, + None, + budget, + pin, + "default".into(), + ); + print_point(&default); + if let Some(p) = points.iter().find(|p| p.recall >= 0.95) { + println!( + " first setting at recall >= 0.95: {} ({:.4}), {:.0} qps single-thread, {:.0} qps batch", + p.label, p.recall, p.qps_1t, p.qps_batch + ); + } + + // ---- writes into the built graph ---- + let extra = 2000usize.min(nq.max(500)); + let more = src.base(n, extra); + let mut ins = Vec::new(); + for (i, v) in more.iter().enumerate() { + let t = Instant::now(); + store + .add(NodeId::new("doc", (n + i).to_string()), "", Some(v.clone())) + .unwrap(); + ins.push(t.elapsed().as_secs_f64() * 1000.0); + } + let mut upd = Vec::new(); + for (i, v) in more.iter().enumerate().take(extra / 2) { + let t = Instant::now(); + store + .add(NodeId::new("doc", i.to_string()), "", Some(v.clone())) + .unwrap(); + upd.push(t.elapsed().as_secs_f64() * 1000.0); + } + let mut del = Vec::new(); + for i in 0..extra / 2 { + let t = Instant::now(); + store.remove(&NodeId::new("doc", (n + i).to_string())); + del.push(t.elapsed().as_secs_f64() * 1000.0); + } + for v in [&mut ins, &mut upd, &mut del] { + v.sort_by(f64::total_cmp); + } + println!( + "\nwrites into the built graph (ms): insert median {:.3} p99 {:.3} max {:.3} | update median {:.3} p99 {:.3} | delete median {:.4} p99 {:.4}", + pct(&ins, 0.5), pct(&ins, 0.99), pct(&ins, 1.0), pct(&upd, 0.5), pct(&upd, 0.99), pct(&del, 0.5), pct(&del, 0.99) + ); + store.verify_invariants().unwrap(); + } + + // ---- IVF on the same data, for the same comparison ---- + if with_ivf || ivf_only { + let mut store2 = EmbeddingStore::new(None); + store2.set_scan_threads(threads); + ingest(&mut store2, &src, n); + store2 + .set_ann(Some(AnnConfig { + min_size: 0, + ..AnnConfig::default() + })) + .unwrap(); + let t = Instant::now(); + let st = store2.build_ann_index(false).unwrap(); + println!( + "\nIVF build {:.1}s nlist={:?}; index {:.1} MB", + t.elapsed().as_secs_f64(), + st.nlist, + store2.memory_stats().index_bytes as f64 / 1048576.0 + ); + print_header("IVF-flat: recall@10 vs speed over nprobe"); + let nlist = st.nlist.unwrap_or(16); + for pct_cells in [0.5f64, 1.0, 2.0, 3.0, 5.0, 8.0, 12.0] { + let nprobe = ((nlist as f64 * pct_cells / 100.0).round() as usize).max(1); + store2.set_ann_nprobe(Some(nprobe)).unwrap(); + let p = measure( + &mut store2, + threads, + &queries, + &truth, + k, + None, + budget, + pin, + format!("nprobe={nprobe}"), + ); + print_point(&p); + } + } +} diff --git a/isongraph-vector-rs/examples/stage_b_bench.rs b/isongraph-vector-rs/examples/stage_b_bench.rs new file mode 100644 index 0000000..0b08bb1 --- /dev/null +++ b/isongraph-vector-rs/examples/stage_b_bench.rs @@ -0,0 +1,330 @@ +//! Indicative measurements for the binary snapshot and the int8 scan. +//! +//! ```text +//! cargo run --release --example stage_b_bench -- snapshot 100000 <dir> +//! cargo run --release --example stage_b_bench -- quant 1000000 +//! ``` +//! +//! Timings are wall clock on whatever machine this runs on, with other work +//! going on: read them as indicative, not as a benchmark result. + +use isongraph_vector_rs::{ + AnnConfig, EmbeddingStore, NodeId, ScanPrecision, SidecarId, SnapshotIdentity, +}; +use std::path::PathBuf; +use std::time::Instant; + +const DIM: usize = 384; + +struct Rng(u64); + +impl Rng { + fn next_u64(&mut self) -> u64 { + self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } + fn unit(&mut self) -> f32 { + (self.next_u64() >> 40) as f32 / (1u64 << 24) as f32 + } + fn gauss(&mut self) -> f32 { + let u1 = self.unit().max(1e-7); + let u2 = self.unit(); + (-2.0 * u1.ln()).sqrt() * (std::f32::consts::TAU * u2).cos() + } +} + +/// Clustered unit vectors (200 centres, noise about one unit norm). +fn vectors(n: usize, seed: u64) -> Vec<Vec<f32>> { + let mut rng = Rng(seed); + let centers: Vec<Vec<f32>> = (0..200) + .map(|_| { + let mut c: Vec<f32> = (0..DIM).map(|_| rng.gauss()).collect(); + let norm = c.iter().map(|x| x * x).sum::<f32>().sqrt(); + c.iter_mut().for_each(|x| *x /= norm); + c + }) + .collect(); + let scale = 1.0 / (DIM as f32).sqrt(); + (0..n) + .map(|_| { + let c = ¢ers[(rng.next_u64() % 200) as usize]; + c.iter().map(|x| x + rng.gauss() * scale).collect() + }) + .collect() +} + +fn filled(n: usize) -> EmbeddingStore { + let data = vectors(n, 1); + let mut store = EmbeddingStore::new(None); + for (i, v) in data.into_iter().enumerate() { + store + .add( + NodeId::new("doc", i.to_string()), + &format!("document {i}"), + Some(v), + ) + .unwrap(); + } + store +} + +/// Current and peak working set in MiB (0 where unknown). +fn rss_mib() -> (f64, f64) { + #[cfg(windows)] + { + #[repr(C)] + struct Counters { + cb: u32, + page_faults: u32, + peak_working_set: usize, + working_set: usize, + rest: [usize; 6], + } + extern "system" { + fn GetCurrentProcess() -> isize; + fn K32GetProcessMemoryInfo(h: isize, c: *mut Counters, cb: u32) -> i32; + } + let mut c = Counters { + cb: std::mem::size_of::<Counters>() as u32, + page_faults: 0, + peak_working_set: 0, + working_set: 0, + rest: [0; 6], + }; + // SAFETY: `c` is a correctly sized PROCESS_MEMORY_COUNTERS and `cb` says so. + let ok = unsafe { K32GetProcessMemoryInfo(GetCurrentProcess(), &mut c, c.cb) }; + if ok != 0 { + return ( + c.working_set as f64 / 1048576.0, + c.peak_working_set as f64 / 1048576.0, + ); + } + (0.0, 0.0) + } + #[cfg(not(windows))] + { + let status = std::fs::read_to_string("/proc/self/status").unwrap_or_default(); + let kb = |key: &str| { + status + .lines() + .find(|l| l.starts_with(key)) + .and_then(|l| l.split_whitespace().nth(1)) + .and_then(|v| v.parse::<f64>().ok()) + .unwrap_or(0.0) + / 1024.0 + }; + (kb("VmRSS:"), kb("VmHWM:")) + } +} + +fn mib(bytes: u64) -> f64 { + bytes as f64 / 1048576.0 +} + +fn snapshot(n: usize, dir: PathBuf) { + std::fs::create_dir_all(&dir).unwrap(); + println!("== snapshot: {n} x {DIM} =="); + let t = Instant::now(); + let store = filled(n); + println!("build store {:8.2} s", t.elapsed().as_secs_f64()); + let json = dir.join("bench.embeddings.json"); + let snap = dir.join("bench.embeddings.snap"); + + let t = Instant::now(); + store + .save_embeddings_with(&json, Some("bench-digest")) + .unwrap(); + let json_save = t.elapsed().as_secs_f64(); + let t = Instant::now(); + let identity = SnapshotIdentity { + graph_digest: Some("bench-digest".to_string()), + sidecar: Some(SidecarId::of_file(&json).unwrap()), + }; + let sidecar_id = t.elapsed().as_secs_f64(); + let t = Instant::now(); + let info = store.save_snapshot(&snap, &identity).unwrap(); + let snap_save = t.elapsed().as_secs_f64(); + let json_bytes = std::fs::metadata(&json).unwrap().len(); + println!( + "JSON save {json_save:8.2} s size {:9.1} MiB", + mib(json_bytes) + ); + println!( + "snapshot save {snap_save:8.2} s size {:9.1} MiB ({:.1}x smaller; sidecar identity {:.3} s)", + mib(info.bytes), + json_bytes as f64 / info.bytes as f64, + sidecar_id + ); + let raw = (n * DIM * 4) as f64 / 1048576.0; + println!("raw f32 matrix {raw:9.1} MiB"); + drop(store); + + let mut a = EmbeddingStore::new(None); + let t = Instant::now(); + a.load_embeddings(&json, true).unwrap(); + let json_load = t.elapsed().as_secs_f64(); + let mut b = EmbeddingStore::new(None); + let t = Instant::now(); + b.load_snapshot(&snap).unwrap(); + let snap_load = t.elapsed().as_secs_f64(); + println!("JSON load {json_load:8.2} s"); + println!( + "snapshot load {snap_load:8.2} s ({:.1}x faster)", + json_load / snap_load + ); + // same store? + assert_eq!(a.count(), b.count()); + let q = vectors(5, 99); + for v in &q { + let (ra, rb) = ( + a.similarity_search_vector(v, 10, None, -1.0).unwrap(), + b.similarity_search_vector(v, 10, None, -1.0).unwrap(), + ); + assert_eq!( + ra.iter() + .map(|r| (r.node_id.id.clone(), r.score.to_bits())) + .collect::<Vec<_>>(), + rb.iter() + .map(|r| (r.node_id.id.clone(), r.score.to_bits())) + .collect::<Vec<_>>() + ); + } + println!("loads agree on count and on top-10 bits of 5 queries"); + let (now, peak) = rss_mib(); + println!("working set now {now:.0} MiB, peak {peak:.0} MiB"); + let _ = std::fs::remove_file(&json); + let _ = std::fs::remove_file(&snap); +} + +fn quant(n: usize, ivf_only: bool) { + println!("== int8 scan: {n} x {DIM} =="); + let mut store = filled(n); + let queries = vectors(128, 4242); + let (rss_f32, _) = rss_mib(); + println!("working set, f32 store only: {rss_f32:.0} MiB"); + + // the fastest of 180 searches / of 5 batches: the machine is shared, and + // the minimum is the least disturbed measurement (medians swing 3x here) + let time = |store: &EmbeddingStore, _threads: usize, exact: bool| -> f64 { + (0..3) + .map(|_| { + let mut ts = Vec::new(); + for q in queries.iter().cycle().take(60) { + let t = Instant::now(); + if exact { + store + .similarity_search_vector_exact(q, 10, None, -1.0) + .unwrap(); + } else { + store.similarity_search_vector(q, 10, None, -1.0).unwrap(); + } + ts.push(t.elapsed().as_secs_f64() * 1000.0); + } + ts.into_iter().fold(f64::INFINITY, f64::min) + }) + .fold(f64::INFINITY, f64::min) + }; + let batch = |store: &EmbeddingStore| -> f64 { + (0..5) + .map(|_| { + let t = Instant::now(); + store + .similarity_search_vector_batch(&queries, 10, None, -1.0) + .unwrap(); + t.elapsed().as_secs_f64() * 1000.0 + }) + .fold(f64::INFINITY, f64::min) + }; + + for threads in if ivf_only { vec![] } else { vec![1usize, 0] } { + store.set_scan_threads(threads); + let label = if threads == 1 { + "1 thread " + } else { + "auto thr " + }; + store.set_scan_precision(ScanPrecision::F32).unwrap(); + let f = time(&store, threads, true); + let fb = batch(&store); + store.set_scan_precision(ScanPrecision::int8()).unwrap(); + let i = time(&store, threads, true); + let ib = batch(&store); + println!( + "exact scan, {label}: f32 {f:8.2} ms/query int8(x4) {i:8.2} ms/query ({:.2}x) batch of 128: f32 {fb:8.1} ms int8 {ib:8.1} ms ({:.2}x)", + f / i, + fb / ib + ); + store.set_scan_precision(ScanPrecision::F32).unwrap(); + } + store.set_scan_threads(0); + + // recall and memory + if !ivf_only { + store.set_scan_precision(ScanPrecision::int8()).unwrap(); + let (rss_int8, peak) = rss_mib(); + println!( + "working set with int8 copy: {rss_int8:.0} MiB (+{:.0} MiB; copy holds {:.0} MiB, f32 matrix {:.0} MiB); peak {peak:.0} MiB", + rss_int8 - rss_f32, + store.quant_bytes() as f64 / 1048576.0, + store.memory_stats().matrix_bytes as f64 / 1048576.0 + ); + let report = store + .int8_recall_report(&queries, 10, None, -1.0, &[1, 2, 4, 8, 16]) + .unwrap(); + for r in &report { + println!( + " rerank x{:<2}: recall@10 {:.4} (worst {:.2}) exact scores {} | {:7.2} ms vs f32 {:7.2} ms", + r.rerank_factor, r.recall, r.min_recall, r.scores_exact, r.int8_ms, r.f32_ms + ); + } + } + + // IVF + store.set_scan_precision(ScanPrecision::F32).unwrap(); + store + .set_ann(Some(AnnConfig { + seed: 1, + ..AnnConfig::default() + })) + .unwrap(); + let t = Instant::now(); + let stats = store.build_ann_index(false).unwrap(); + println!( + "IVF build {:.1} s (nlist {:?}, nprobe {:?})", + t.elapsed().as_secs_f64(), + stats.nlist, + stats.nprobe + ); + for threads in [0usize, 1] { + store.set_scan_threads(threads); + store.set_scan_precision(ScanPrecision::F32).unwrap(); + let f = time(&store, 0, false); + let fb = batch(&store); + store.set_scan_precision(ScanPrecision::int8()).unwrap(); + let i = time(&store, 0, false); + let ib = batch(&store); + println!( + "IVF search, threads {threads}: f32 {f:.3} ms/query int8(x4) {i:.3} ms/query ({:.2}x) batch of 128: f32 {fb:.1} ms int8 {ib:.1} ms ({:.2}x)", + f / i, + fb / ib + ); + } +} + +fn main() { + let args: Vec<String> = std::env::args().skip(1).collect(); + let n: usize = args.get(1).and_then(|a| a.parse().ok()).unwrap_or(100_000); + match args.first().map(String::as_str) { + Some("snapshot") => snapshot( + n, + args.get(2) + .map(PathBuf::from) + .unwrap_or_else(|| std::env::temp_dir().join("stage_b_bench")), + ), + Some("quant") => quant(n, args.get(2).is_some_and(|a| a == "ivf-only")), + _ => eprintln!("usage: stage_b_bench snapshot|quant <rows> [dir]"), + } +} diff --git a/isongraph-vector-rs/examples/stage_c_bench.rs b/isongraph-vector-rs/examples/stage_c_bench.rs new file mode 100644 index 0000000..6b0bdbf --- /dev/null +++ b/isongraph-vector-rs/examples/stage_c_bench.rs @@ -0,0 +1,849 @@ +//! Stage C harness: one process = one (dataset, n, mode, precision) cell of the Rust side of the +//! honest re-benchmark in `benchmarks/rust-stage-c/`. It reads the same raw float32 vectors, queries and +//! float64 ground truth as `benchmarks/competitive/` and writes the same JSON shape for sweep points +//! (`knob`, `recall`, `qps_1t`, `batch.qps`, ...), so one summariser reads Rust and Python results. +//! +//! ```text +//! stage_c_bench --ds-dir D:/.../prep/text384 --ds text384 --n 100000 --mode exact|ivf|hnsw \ +//! [--int8 RF] --exp core,int8,conc,writes,persist --queries 1000 --budget 3 --knobs 16,32,64 --out r.json +//! ``` +//! Experiments: `core` (ingest, memory, build, first query, knob sweep: pinned 1-thread loop x3 runs and +//! a batch on every thread), `int8` (mode exact only: rerank factors 1,2,4,8, QPS and recall against the +//! f32 scan and the float64 truth), `conc` (N concurrent clients, one scan thread each, at the reference +//! knob), `writes` (insert / update / delete 10k on the built index, first query after, recall after; +//! ivf and hnsw), `persist` (JSON save/load against the binary snapshot, optional time-to-ready of ivf +//! and hnsw after a load). +//! +//! Single-thread loops pin the process to one P-core (`BENCH_PIN_CPU`, default logical CPU 8) exactly as +//! `competitive_bench` does. Bench-only: nothing in the library is touched. + +use isongraph_vector_rs::{ + AnnConfig, EmbeddingStore, NodeId, ScanPrecision, SidecarId, SimilarityResult, SnapshotIdentity, +}; +use serde_json::{json, Value}; +use std::collections::HashSet; +use std::fs::File; +use std::io::{Read, Seek, SeekFrom}; +use std::time::Instant; + +// ---- process memory + pinning (Windows), no extra crates --------------------------------------- +#[cfg(windows)] +#[repr(C)] +struct Pmc { + cb: u32, + page_fault_count: u32, + peak_working_set_size: usize, + working_set_size: usize, + quota_peak_paged_pool_usage: usize, + quota_paged_pool_usage: usize, + quota_peak_non_paged_pool_usage: usize, + quota_non_paged_pool_usage: usize, + pagefile_usage: usize, + peak_pagefile_usage: usize, +} +#[cfg(windows)] +#[link(name = "kernel32")] +extern "system" { + fn GetCurrentProcess() -> isize; + fn K32GetProcessMemoryInfo(h: isize, p: *mut Pmc, cb: u32) -> i32; + fn SetProcessAffinityMask(h: isize, mask: usize) -> i32; + fn SetPriorityClass(h: isize, class: u32) -> i32; +} + +fn pinned<T>(f: impl FnOnce() -> T) -> T { + #[cfg(windows)] + { + let cpu: usize = std::env::var("BENCH_PIN_CPU") + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(8); + let all = usize::MAX + >> (usize::BITS as usize + - std::thread::available_parallelism() + .map(|n| n.get()) + .unwrap_or(32)); + // SAFETY: plain kernel32 calls on the current process. + unsafe { + SetPriorityClass(GetCurrentProcess(), 0x80); // HIGH_PRIORITY_CLASS + SetProcessAffinityMask(GetCurrentProcess(), 1usize << cpu); + } + let r = f(); + unsafe { SetProcessAffinityMask(GetCurrentProcess(), all) }; + r + } + #[cfg(not(windows))] + f() +} +fn mem() -> Value { + #[cfg(windows)] + { + // SAFETY: Pmc matches PROCESS_MEMORY_COUNTERS and `cb` is its size. + let p = unsafe { + let mut p: Pmc = std::mem::zeroed(); + p.cb = std::mem::size_of::<Pmc>() as u32; + K32GetProcessMemoryInfo(GetCurrentProcess(), &mut p, p.cb); + p + }; + json!({"rss_mb": p.working_set_size as f64 / 1048576.0, "peak_mb": p.peak_working_set_size as f64 / 1048576.0}) + } + #[cfg(not(windows))] + json!({"rss_mb": 0.0, "peak_mb": 0.0}) +} +fn rss(v: &Value) -> f64 { + v["rss_mb"].as_f64().unwrap_or(0.0) +} + +// ---- data --------------------------------------------------------------------------------------- +fn read_f32(path: &str, start: usize, count: usize, d: usize) -> Vec<f32> { + let mut f = File::open(path).unwrap_or_else(|e| panic!("{path}: {e}")); + f.seek(SeekFrom::Start((start * d * 4) as u64)).unwrap(); + let mut buf = vec![0u8; count * d * 4]; + f.read_exact(&mut buf).unwrap(); + (0..buf.len() / 4) + .map(|i| f32::from_le_bytes(buf[4 * i..4 * i + 4].try_into().unwrap())) + .collect() +} +fn read_i32(path: &str) -> Vec<i32> { + let mut b = Vec::new(); + File::open(path) + .unwrap_or_else(|e| panic!("{path}: {e}")) + .read_to_end(&mut b) + .unwrap(); + (0..b.len() / 4) + .map(|i| i32::from_le_bytes(b[4 * i..4 * i + 4].try_into().unwrap())) + .collect() +} + +struct Ctx { + dir: String, + d: usize, + n: usize, + k: usize, + queries: Vec<Vec<f32>>, +} +impl Ctx { + fn base(&self, start: usize, count: usize) -> Vec<Vec<f32>> { + read_f32(&format!("{}/base.f32", self.dir), start, count, self.d) + .chunks(self.d) + .map(|c| c.to_vec()) + .collect() + } + fn gt(&self, kind: &str) -> Vec<Vec<i32>> { + read_i32(&format!("{}/gt_{}{}.i32", self.dir, kind, self.n)) + .chunks(100) + .map(|c| c.to_vec()) + .collect() + } +} + +// ---- stats -------------------------------------------------------------------------------------- +fn pct(sorted: &[f64], p: f64) -> f64 { + if sorted.is_empty() { + return 0.0; + } + sorted[((sorted.len() - 1) as f64 * p).round() as usize] +} +fn lat_stats(lat_s: &[f64]) -> Value { + let mut ms: Vec<f64> = lat_s.iter().map(|x| x * 1000.0).collect(); + ms.sort_by(f64::total_cmp); + let mean = ms.iter().sum::<f64>() / ms.len().max(1) as f64; + json!({"median_ms": pct(&ms, 0.5), "p95_ms": pct(&ms, 0.95), "p99_ms": pct(&ms, 0.99), "max_ms": ms.last().cloned().unwrap_or(0.0), "mean_ms": mean}) +} +fn median(v: &[f64]) -> f64 { + let mut s = v.to_vec(); + s.sort_by(f64::total_cmp); + pct(&s, 0.5) +} +fn recall(found: &[Vec<usize>], truth: &[Vec<i32>], k: usize) -> f64 { + let mut hit = 0usize; + for (f, t) in found.iter().zip(truth) { + let want: HashSet<i32> = t.iter().take(k).cloned().collect(); + hit += f + .iter() + .take(k) + .filter(|x| want.contains(&(**x as i32))) + .count(); + } + hit as f64 / (k * found.len().max(1)) as f64 +} +fn ids(r: &[SimilarityResult]) -> Vec<usize> { + r.iter() + .map(|x| x.node_id.id.parse::<usize>().unwrap()) + .collect() +} + +#[derive(Clone, Copy, PartialEq)] +enum Path { + Exact, + Ann, +} +fn search(s: &EmbeddingStore, q: &[f32], k: usize, p: Path) -> Vec<SimilarityResult> { + match p { + Path::Exact => s.similarity_search_vector_exact(q, k, None, 0.0).unwrap(), + Path::Ann => s.similarity_search_vector(q, k, None, 0.0).unwrap(), + } +} +fn run_loop(s: &EmbeddingStore, qs: &[Vec<f32>], k: usize, p: Path) -> (Vec<f64>, Vec<Vec<usize>>) { + let mut lat = Vec::with_capacity(qs.len()); + let mut out = Vec::with_capacity(qs.len()); + for q in qs { + let t = Instant::now(); + let r = search(s, q, k, p); + lat.push(t.elapsed().as_secs_f64()); + out.push(ids(&r)); + } + (lat, out) +} + +/// Batched call over all the pool's threads: one warm-up run, then 3 timed runs of the whole query set. +fn batch_point(s: &EmbeddingStore, c: &Ctx, truth: &[Vec<i32>], p: Path) -> Value { + let run = || { + let t = Instant::now(); + let r = match p { + Path::Exact => s.similarity_search_vector_exact_batch(&c.queries, c.k, None, 0.0), + Path::Ann => s.similarity_search_vector_batch(&c.queries, c.k, None, 0.0), + } + .unwrap(); + (c.queries.len() as f64 / t.elapsed().as_secs_f64(), r) + }; + let _ = run(); + let mut qps = Vec::new(); + let mut found = Vec::new(); + for _ in 0..3 { + let (q, r) = run(); + qps.push(q); + found = r; + } + let found: Vec<Vec<usize>> = found.iter().map(|r| ids(r)).collect(); + json!({"nq": c.queries.len(), "threads": s.search_pool().threads(), "qps_batch": median(&qps), "qps_batch_runs": qps, + "qps_batch_min": qps.iter().cloned().fold(f64::MAX, f64::min), "qps_batch_max": qps.iter().cloned().fold(0.0, f64::max), + "recall": recall(&found, &truth[..found.len()], c.k)}) +} + +/// Warm up, size the query count to `budget` seconds, then 3 timed runs (one-query-at-a-time loop). +fn point(s: &EmbeddingStore, c: &Ctx, truth: &[Vec<i32>], p: Path, budget: f64) -> Value { + let k = c.k; + let nqall = c.queries.len(); + let w0 = Instant::now(); + for q in c.queries.iter().take(50) { + let _ = search(s, q, k, p); + if w0.elapsed().as_secs_f64() > 3.0 { + break; + } + } + let (probe, _) = run_loop(s, &c.queries[..5.min(nqall)], k, p); + let est = median(&probe).max(1e-7); + let nq = ((budget / est) as usize).clamp(30.min(nqall), nqall); + let qs = &c.queries[..nq]; + let mut runs = Vec::new(); + let mut lats: Vec<f64> = Vec::new(); + let mut rec = 0.0; + for r in 0..3 { + let (lat, res) = run_loop(s, qs, k, p); + if r == 0 { + rec = recall(&res, &truth[..nq], k); + } + let total: f64 = lat.iter().sum(); + runs.push(nq as f64 / total); + lats.extend(lat); + if total > budget * 1.5 { + break; + } + } + let mut v = lat_stats(&lats); + v["nq"] = json!(nq); + v["recall"] = json!(rec); + v["qps_1t"] = json!(median(&runs)); + v["qps_1t_min"] = json!(runs.iter().cloned().fold(f64::MAX, f64::min)); + v["qps_1t_max"] = json!(runs.iter().cloned().fold(0.0, f64::max)); + v["qps_1t_runs"] = json!(runs); + v +} + +fn ingest(s: &mut EmbeddingStore, c: &Ctx, n: usize) -> f64 { + let mut secs = 0.0; + let step = 20000; + let mut at = 0; + while at < n { + let m = step.min(n - at); + let rows = c.base(at, m); + let t = Instant::now(); + for (i, v) in rows.into_iter().enumerate() { + s.add(NodeId::new("doc", (at + i).to_string()), "", Some(v)) + .unwrap(); + } + secs += t.elapsed().as_secs_f64(); + at += m; + } + secs +} + +fn arg(args: &[String], name: &str, default: &str) -> String { + args.iter() + .position(|a| a == name) + .map(|i| args[i + 1].clone()) + .unwrap_or(default.to_string()) +} +fn list(s: &str) -> Vec<usize> { + s.split(',') + .filter(|x| !x.is_empty()) + .map(|x| x.parse().unwrap()) + .collect() +} + +fn set_knob(store: &mut EmbeddingStore, mode: &str, kn: usize) { + match mode { + "ivf" => store.set_ann_nprobe(Some(kn)).unwrap(), + "hnsw" => store.set_ann_ef_search(Some(kn)).unwrap(), + _ => {} + } +} +fn ann_config(mode: &str, m: usize, efc: usize) -> AnnConfig { + let mut cfg = if mode == "hnsw" { + AnnConfig::hnsw() + } else { + AnnConfig::default() + }; + cfg.min_size = 0; + if mode == "hnsw" { + cfg.hnsw.m = m; + cfg.hnsw.ef_construction = efc; + } + cfg +} + +fn main() { + let args: Vec<String> = std::env::args().skip(1).collect(); + let dir = arg(&args, "--ds-dir", ""); + let ds = arg(&args, "--ds", ""); + let n: usize = arg(&args, "--n", "100000").parse().unwrap(); + let nq: usize = arg(&args, "--queries", "1000").parse().unwrap(); + let budget: f64 = arg(&args, "--budget", "3").parse().unwrap(); + let mode = arg(&args, "--mode", "exact"); + let int8: usize = arg(&args, "--int8", "0").parse().unwrap(); + let exps: Vec<String> = arg(&args, "--exp", "core") + .split(',') + .map(|s| s.to_string()) + .collect(); + let out_path = arg(&args, "--out", ""); + let label = arg(&args, "--label", "rust"); + let workdir = arg(&args, "--workdir", "D:/Goldmine/ISON/bench_data/work"); + let ready = arg(&args, "--ready", ""); + let m: usize = arg(&args, "--m", "16").parse().unwrap(); + let efc: usize = arg(&args, "--efc", "200").parse().unwrap(); + let default_knobs = match mode.as_str() { + "ivf" => "16,32,64,128,256,512", + "hnsw" => "24,48,64,96,128,192,256,400", + _ => "", + }; + let knobs = list(&arg(&args, "--knobs", default_knobs)); + let has = |e: &str| exps.iter().any(|x| x == e); + + let meta: Value = + serde_json::from_str(&std::fs::read_to_string(format!("{dir}/meta.json")).unwrap()) + .unwrap(); + let d = meta["d"].as_u64().unwrap() as usize; + let qall = read_f32(&format!("{dir}/queries.f32"), 0, nq, d); + let c = Ctx { + dir: dir.clone(), + d, + n, + k: 10, + queries: qall.chunks(d).map(|x| x.to_vec()).collect(), + }; + let cpus = std::thread::available_parallelism() + .map(|x| x.get()) + .unwrap_or(1); + println!("{label}: ds={ds} n={n} d={d} mode={mode} int8={int8} queries={nq} cpus={cpus} exps={exps:?}"); + let mut out = json!({ + "system": label, "ds": ds, "n": n, "d": d, "k": c.k, "queries": nq, "all_threads": cpus, "mode": mode, + "int8_rerank": int8, "exps": exps, "budget": budget, + "version": "isongraph-vector-rs HEAD (cargo release, default flags)", + "started": format!("{:?}", std::time::SystemTime::now()), + }); + let mem_base = mem(); + out["mem_baseline"] = mem_base.clone(); + let write_out = |out: &Value| { + if !out_path.is_empty() { + std::fs::write(&out_path, serde_json::to_string_pretty(out).unwrap()).unwrap(); + } + }; + + // ---------------------------------------------------------------- persist (own flow) + if has("persist") { + persist(&c, n, &workdir, &ready, &mut out, &mode, m, efc); + write_out(&out); + return; + } + + let mut store = EmbeddingStore::new(None); + let ingest_s = ingest(&mut store, &c, n); + let mem_after_ingest = mem(); + println!( + " ingest {ingest_s:.2}s rss {:.0} MB", + rss(&mem_after_ingest) + ); + out["mem_after_ingest"] = mem_after_ingest.clone(); + let truth = c.gt(""); + let raw_mb = (n * d * 4) as f64 / 1048576.0; + out["raw_vectors_mb"] = json!(raw_mb); + + // ---------------------------------------------------------------- int8 study (mode exact) + // runs first on a pristine store; the f32 baseline is the same process, same pinning + if has("int8") && mode == "exact" { + store.set_scan_threads(1); + let _ = store.similarity_search_vector_exact(&c.queries[0], c.k, None, 0.0); + let mut study = json!({}); + let f32_pin = pinned(|| point(&store, &c, &truth, Path::Exact, budget)); + store.set_scan_threads(0); + let f32_auto = point(&store, &c, &truth, Path::Exact, budget); + let f32_batch = batch_point(&store, &c, &truth, Path::Exact); + let mem_f32 = mem(); + println!( + " int8 study: f32 1T {:.1} qps, auto {:.1} qps, batch {:.0} qps", + f32_pin["qps_1t"].as_f64().unwrap(), + f32_auto["qps_1t"].as_f64().unwrap(), + f32_batch["qps_batch"].as_f64().unwrap() + ); + study["f32"] = json!({"one_thread": f32_pin, "auto_threads": f32_auto, "batch": f32_batch, "mem": mem_f32.clone()}); + let t = Instant::now(); + store.set_scan_precision(ScanPrecision::int8()).unwrap(); + study["quant_build_s"] = json!(t.elapsed().as_secs_f64()); + let mem_i8 = mem(); + study["mem_after_int8"] = mem_i8.clone(); + study["quant_bytes_mb"] = json!(store.quant_bytes() as f64 / 1048576.0); + study["rss_delta_mb"] = json!(rss(&mem_i8) - rss(&mem_f32)); + // recall of int8 + rerank against the f32 scan, on the real queries (cap for time at 1M) + let rq: Vec<Vec<f32>> = c + .queries + .iter() + .take(if n >= 500_000 { 300 } else { 1000 }) + .cloned() + .collect(); + store.set_scan_threads(0); + let rep = store + .int8_recall_report(&rq, c.k, None, 0.0, &[1, 2, 4, 8]) + .unwrap(); + let mut by = Vec::new(); + for r in &rep { + println!( + " rerank x{}: recall vs f32 {:.4} (worst {:.2}) exact_scores={} applies={} int8 {:.3} ms f32 {:.3} ms", + r.rerank_factor, r.recall, r.min_recall, r.scores_exact, r.applies, r.int8_ms, r.f32_ms + ); + store + .set_scan_precision(ScanPrecision::Int8 { + rerank_factor: r.rerank_factor, + }) + .unwrap(); + store.set_scan_threads(1); + let p1 = pinned(|| point(&store, &c, &truth, Path::Exact, budget)); + store.set_scan_threads(0); + let pa = point(&store, &c, &truth, Path::Exact, budget); + let pb = batch_point(&store, &c, &truth, Path::Exact); + println!( + " 1T {:.1} qps (recall vs float64 truth {:.4}), auto {:.1} qps, batch {:.0} qps", + p1["qps_1t"].as_f64().unwrap(), + p1["recall"].as_f64().unwrap(), + pa["qps_1t"].as_f64().unwrap(), + pb["qps_batch"].as_f64().unwrap() + ); + by.push(json!({"rerank_factor": r.rerank_factor, "recall_vs_f32_scan": r.recall, "min_recall_vs_f32_scan": r.min_recall, + "scores_exact": r.scores_exact, "applies": r.applies, "report_queries": r.queries, "report_int8_ms": r.int8_ms, "report_f32_ms": r.f32_ms, + "one_thread": p1, "auto_threads": pa, "batch": pb})); + } + study["by_factor"] = json!(by); + out["int8_study"] = study; + store.set_scan_precision(ScanPrecision::F32).unwrap(); + write_out(&out); + } + + // ---------------------------------------------------------------- core + if has("core") { + if int8 > 0 { + let before = mem(); + let t = Instant::now(); + store + .set_scan_precision(ScanPrecision::Int8 { + rerank_factor: int8, + }) + .unwrap(); + out["quant_build_s"] = json!(t.elapsed().as_secs_f64()); + out["quant_bytes_mb"] = json!(store.quant_bytes() as f64 / 1048576.0); + out["rss_delta_int8_mb"] = json!(rss(&mem()) - rss(&before)); + } + store.set_scan_threads(1); + let t = Instant::now(); + store + .similarity_search_vector_exact(&c.queries[0], c.k, None, 0.0) + .unwrap(); + let first_exact_ms = t.elapsed().as_secs_f64() * 1000.0; + let mem_after_first = mem(); + out["first_exact_query_ms"] = json!(first_exact_ms); + out["mem_after_first_exact_query"] = mem_after_first.clone(); + out["build"] = json!({"ingest_s": ingest_s, "first_exact_query_ms": first_exact_ms}); + + let mut pts = Vec::new(); + if mode == "exact" { + let mut p1 = pinned(|| point(&store, &c, &truth, Path::Exact, budget)); + p1["knob"] = Value::Null; + store.set_scan_threads(0); + let pa = point(&store, &c, &truth, Path::Exact, budget); + p1["auto_threads"] = json!({"qps": pa["qps_1t"], "median_ms": pa["median_ms"], "p99_ms": pa["p99_ms"], "runs": pa["qps_1t_runs"]}); + p1["batch"] = batch_point(&store, &c, &truth, Path::Exact); + println!( + " exact{} 1T recall={:.4} qps={:.1} median={:.3}ms | auto {:.1} qps | batch {:.0} qps", + if int8 > 0 { format!("+int8x{int8}") } else { String::new() }, + p1["recall"].as_f64().unwrap(), + p1["qps_1t"].as_f64().unwrap(), + p1["median_ms"].as_f64().unwrap(), + pa["qps_1t"].as_f64().unwrap(), + p1["batch"]["qps_batch"].as_f64().unwrap() + ); + pts.push(p1); + out["ref_knob"] = Value::Null; + } else { + store.set_scan_threads(0); + store.set_ann(Some(ann_config(&mode, m, efc))).unwrap(); + let t = Instant::now(); + let stats = store.build_ann_index(false).unwrap(); + let idx_s = t.elapsed().as_secs_f64(); + let mem_after_build = mem(); + println!( + " {mode} build {idx_s:.2}s (stats {:?}) rss {:.0} MB (peak {:.0})", + stats.build_seconds, + rss(&mem_after_build), + mem_after_build["peak_mb"].as_f64().unwrap() + ); + out["build"] = json!({"ingest_s": ingest_s, "index_s": idx_s, "total_s": ingest_s + idx_s, + "stats_build_seconds": stats.build_seconds}); + out["mem_after_build"] = mem_after_build; + out["index_stats"] = json!({"nlist": stats.nlist, "trained_n": stats.trained_n, "index_bytes": stats.index_bytes, + "max_level": stats.max_level, "graph_nodes": stats.graph_nodes, "ef_search_default": stats.ef_search}); + store.set_scan_threads(1); + let t = Instant::now(); + let _ = store + .similarity_search_vector(&c.queries[0], c.k, None, 0.0) + .unwrap(); + out["build"]["first_query_ms"] = json!(t.elapsed().as_secs_f64() * 1000.0); + let ms = store.memory_stats(); + out["memory_stats"] = json!({"matrix_bytes": ms.matrix_bytes, "metadata_bytes": ms.metadata_bytes, "index_bytes": ms.index_bytes, + "total_bytes": ms.total_bytes(), "quant_bytes": store.quant_bytes()}); + let mut ref_knob: Option<usize> = None; + for &kn in &knobs { + set_knob(&mut store, &mode, kn); + store.set_scan_threads(1); + let mut p = pinned(|| point(&store, &c, &truth, Path::Ann, budget)); + store.set_scan_threads(0); + p["knob"] = json!(kn); + p["batch"] = batch_point(&store, &c, &truth, Path::Ann); + println!( + " {mode}{} knob={kn} recall={:.4} qps1t={:.0} median={:.3}ms batch={:.0}", + if int8 > 0 { + format!("+int8x{int8}") + } else { + String::new() + }, + p["recall"].as_f64().unwrap(), + p["qps_1t"].as_f64().unwrap(), + p["median_ms"].as_f64().unwrap(), + p["batch"]["qps_batch"].as_f64().unwrap() + ); + if ref_knob.is_none() && p["recall"].as_f64().unwrap() >= 0.95 { + ref_knob = Some(kn); + out["ref_recall_pre"] = p["recall"].clone(); + } + pts.push(p); + } + if ref_knob.is_none() { + ref_knob = knobs.last().cloned(); + out["ref_recall_pre"] = pts + .last() + .map(|p| p["recall"].clone()) + .unwrap_or(Value::Null); + } + out["ref_knob"] = json!(ref_knob); + } + out["knob_name"] = json!(match mode.as_str() { + "ivf" => "nprobe", + "hnsw" => "ef_search", + _ => "", + }); + out["sweep"] = json!(pts); + out["mem_after_sweep"] = mem(); + write_out(&out); + + let ref_knob = out["ref_knob"].as_u64().map(|x| x as usize); + if let Some(rk) = ref_knob { + set_knob(&mut store, &mode, rk); + } + + // ------------------------------------------------------------ concurrent clients + if has("conc") { + let secs = 3.0; + let mut res = Vec::new(); + store.set_scan_threads(1); + for clients in [1usize, 2, 4, 8, 16] { + let path = if mode == "exact" { + Path::Exact + } else { + Path::Ann + }; + // warm + for q in c.queries.iter().take(20) { + let _ = search(&store, q, c.k, path); + } + let t0 = Instant::now(); + let done: usize = std::thread::scope(|sc| { + let hs: Vec<_> = (0..clients) + .map(|w| { + let st = &store; + let qs = &c.queries; + sc.spawn(move || { + let mut cnt = 0usize; + let mut i = w; + while t0.elapsed().as_secs_f64() < secs { + let _ = search(st, &qs[i % qs.len()], 10, path); + i += clients; + cnt += 1; + } + cnt + }) + }) + .collect(); + hs.into_iter().map(|h| h.join().unwrap()).sum() + }); + let qps = done as f64 / t0.elapsed().as_secs_f64(); + println!(" concurrent clients={clients} (1 scan thread each): {qps:.1} qps"); + res.push(json!({"clients": clients, "qps": qps})); + } + out["concurrent_clients"] = json!(res); + write_out(&out); + } + + // ------------------------------------------------------------ writes + if has("writes") && mode != "exact" { + let gtm = format!("{dir}/gt_mut_{n}.i32"); + if std::path::Path::new(>m).exists() { + let upd: Vec<i32> = read_i32(&format!("{dir}/mut_{n}_upd.i32")); + let del: Vec<i32> = read_i32(&format!("{dir}/mut_{n}_del.i32")); + let truth_m: Vec<Vec<i32>> = read_i32(>m) + .chunks(100) + .map(|c| c.to_vec()) + .take(nq) + .collect(); + let mut w = json!({"ref_knob": ref_knob, "recall_before": out["ref_recall_pre"].clone(), "mode": mode}); + store.set_scan_threads(0); + let lat_stats_of = |lat: &[f64]| lat_stats(lat); + let ins = c.base(n, 10000); + let mut lat = Vec::with_capacity(10000); + let t = Instant::now(); + for (i, v) in ins.into_iter().enumerate() { + let t1 = Instant::now(); + store + .add(NodeId::new("doc", (n + i).to_string()), "", Some(v)) + .unwrap(); + lat.push(t1.elapsed().as_secs_f64()); + } + let dt = t.elapsed().as_secs_f64(); + w["insert_10k"] = + json!({"seconds": dt, "per_s": 10000.0 / dt, "lat": lat_stats_of(&lat)}); + let newv = c.base(n + 10000, 10000); + lat.clear(); + let t = Instant::now(); + for (j, v) in newv.into_iter().enumerate() { + let t1 = Instant::now(); + store + .add(NodeId::new("doc", upd[j].to_string()), "", Some(v)) + .unwrap(); + lat.push(t1.elapsed().as_secs_f64()); + } + let dt = t.elapsed().as_secs_f64(); + w["update_10k"] = + json!({"seconds": dt, "per_s": 10000.0 / dt, "lat": lat_stats_of(&lat)}); + lat.clear(); + let t = Instant::now(); + for id in &del { + let t1 = Instant::now(); + store.remove(&NodeId::new("doc", id.to_string())); + lat.push(t1.elapsed().as_secs_f64()); + } + let dt = t.elapsed().as_secs_f64(); + w["delete_10k"] = + json!({"seconds": dt, "per_s": 10000.0 / dt, "lat": lat_stats_of(&lat)}); + store.set_scan_threads(1); + let t = Instant::now(); + let _ = store + .similarity_search_vector(&c.queries[0], c.k, None, 0.0) + .unwrap(); + w["first_query_after_ms"] = json!(t.elapsed().as_secs_f64() * 1000.0); + let p = pinned(|| point(&store, &c, &truth_m, Path::Ann, budget)); + w["recall_after"] = p["recall"].clone(); + w["qps_after"] = p["qps_1t"].clone(); + w["after"] = p; + let st = store.ann_stats(); + w["index_stats_after"] = json!({"pending": st.pending, "churn": st.churn, "trained_n": st.trained_n, + "graph_nodes": st.graph_nodes, "graph_dead": st.graph_dead, "inserts": st.inserts}); + w["mem_after"] = mem(); + println!( + " writes: ins {:.0}/s upd {:.0}/s del {:.0}/s first-query {:.1} ms recall_after={}", + w["insert_10k"]["per_s"].as_f64().unwrap(), + w["update_10k"]["per_s"].as_f64().unwrap(), + w["delete_10k"]["per_s"].as_f64().unwrap(), + w["first_query_after_ms"].as_f64().unwrap(), + w["recall_after"] + ); + out["mutation"] = w; + } else { + out["mutation"] = json!({"skipped": "no spare rows for this n"}); + } + write_out(&out); + } + } + out["mem_end"] = mem(); + out["status"] = json!("ok"); + write_out(&out); +} + +#[allow(clippy::too_many_arguments)] +fn persist( + c: &Ctx, + n: usize, + workdir: &str, + ready: &str, + out: &mut Value, + _mode: &str, + m: usize, + efc: usize, +) { + std::fs::create_dir_all(workdir).unwrap(); + let json_path = format!("{workdir}/stagec_rust_{n}.embeddings.json"); + let snap_path = format!("{workdir}/stagec_rust_{n}.embeddings.snap"); + let mut store = EmbeddingStore::new(None); + let ingest_s = ingest(&mut store, c, n); + out["ingest_s"] = json!(ingest_s); + out["mem_after_ingest"] = mem(); + let raw_mb = (n * c.d * 4) as f64 / 1048576.0; + out["raw_vectors_mb"] = json!(raw_mb); + let mut p = json!({}); + let t = Instant::now(); + store + .save_embeddings_with(&json_path, Some("stagec")) + .unwrap(); + p["json_save_s"] = json!(t.elapsed().as_secs_f64()); + p["json_mb"] = json!(std::fs::metadata(&json_path).unwrap().len() as f64 / 1048576.0); + let t = Instant::now(); + let sc = SidecarId::of_file(&json_path).unwrap(); + p["sidecar_fingerprint_s"] = json!(t.elapsed().as_secs_f64()); + let t = Instant::now(); + let info = store + .save_snapshot( + &snap_path, + &SnapshotIdentity { + graph_digest: Some("stagec".to_string()), + sidecar: Some(sc), + }, + ) + .unwrap(); + p["snapshot_save_s"] = json!(t.elapsed().as_secs_f64()); + p["snapshot_mb"] = json!(info.bytes as f64 / 1048576.0); + println!( + " save: JSON {:.2}s {:.0} MB | snapshot {:.2}s {:.0} MB (+ sidecar fingerprint {:.2}s)", + p["json_save_s"].as_f64().unwrap(), + p["json_mb"].as_f64().unwrap(), + p["snapshot_save_s"].as_f64().unwrap(), + p["snapshot_mb"].as_f64().unwrap(), + p["sidecar_fingerprint_s"].as_f64().unwrap() + ); + let q0 = c.queries[0].clone(); + let ref_ids: Vec<Vec<usize>> = c + .queries + .iter() + .take(20) + .map(|q| { + ids(&store + .similarity_search_vector_exact(q, 10, None, 0.0) + .unwrap()) + }) + .collect(); + drop(store); + let mut runs_json = Vec::new(); + let mut runs_snap = Vec::new(); + for rep in 0..3 { + let mut a = EmbeddingStore::new(None); + let t = Instant::now(); + a.load_embeddings(&json_path, true).unwrap(); + let jl = t.elapsed().as_secs_f64(); + let t = Instant::now(); + let _ = a + .similarity_search_vector_exact(&q0, 10, None, 0.0) + .unwrap(); + let jf = t.elapsed().as_secs_f64() * 1000.0; + let agree_a: Vec<Vec<usize>> = c + .queries + .iter() + .take(20) + .map(|q| ids(&a.similarity_search_vector_exact(q, 10, None, 0.0).unwrap())) + .collect(); + if rep == 0 { + p["json_load_peak_mb"] = mem()["peak_mb"].clone(); + } + drop(a); + let mut b = EmbeddingStore::new(None); + let t = Instant::now(); + b.load_snapshot(&snap_path).unwrap(); + let sl = t.elapsed().as_secs_f64(); + let t = Instant::now(); + let _ = b + .similarity_search_vector_exact(&q0, 10, None, 0.0) + .unwrap(); + let sf = t.elapsed().as_secs_f64() * 1000.0; + let agree_b: Vec<Vec<usize>> = c + .queries + .iter() + .take(20) + .map(|q| ids(&b.similarity_search_vector_exact(q, 10, None, 0.0).unwrap())) + .collect(); + println!(" load rep {rep}: JSON {jl:.2}s (first query {jf:.1} ms) | snapshot {sl:.2}s (first query {sf:.1} ms) | answers equal ref: json={} snap={}", + agree_a == ref_ids, agree_b == ref_ids); + runs_json.push(jl); + runs_snap.push(sl); + p["json_first_query_ms"] = json!(jf); + p["snapshot_first_query_ms"] = json!(sf); + p["json_answers_equal_original"] = json!(agree_a == ref_ids); + p["snapshot_answers_equal_original"] = json!(agree_b == ref_ids); + if rep == 2 && !ready.is_empty() { + // time to ready: load (snapshot) + index build + first indexed query, per requested ANN mode + let mut rd = json!({}); + for md in ready.split(',') { + let mut s = EmbeddingStore::new(None); + let t = Instant::now(); + s.load_snapshot(&snap_path).unwrap(); + let l = t.elapsed().as_secs_f64(); + s.set_ann(Some(ann_config(md, m, efc))).unwrap(); + let t = Instant::now(); + s.build_ann_index(false).unwrap(); + let bs = t.elapsed().as_secs_f64(); + let t = Instant::now(); + let _ = s.similarity_search_vector(&q0, 10, None, 0.0).unwrap(); + let fq = t.elapsed().as_secs_f64(); + println!( + " ready {md}: load {l:.2}s + build {bs:.2}s + first query {:.1} ms = {:.2}s", + fq * 1000.0, + l + bs + fq + ); + rd[md] = json!({"load_s": l, "build_s": bs, "first_query_ms": fq * 1000.0, "ready_s": l + bs + fq}); + } + p["time_to_ready_after_snapshot_load"] = rd; + } + drop(b); + } + p["json_load_s_runs"] = json!(runs_json); + p["snapshot_load_s_runs"] = json!(runs_snap); + p["json_load_s"] = json!(median(&runs_json)); + p["snapshot_load_s"] = json!(median(&runs_snap)); + out["persist"] = p; + out["mem_end"] = mem(); + out["status"] = json!("ok"); + let _ = std::fs::remove_file(&json_path); + let _ = std::fs::remove_file(&snap_path); +} diff --git a/isongraph-vector-rs/src/ann.rs b/isongraph-vector-rs/src/ann.rs new file mode 100644 index 0000000..e575c0e --- /dev/null +++ b/isongraph-vector-rs/src/ann.rs @@ -0,0 +1,2893 @@ +//! Search for [`EmbeddingStore`]: the exact scan, the batched scan, and the +//! opt-in IVF-flat approximate index built on top of the store's matrix (the +//! opt-in HNSW graph index lives in `hnsw.rs`; `AnnConfig::mode` picks one). +//! `DESIGN.md` is the specification; this file is its Rust realisation. +//! +//! # The matrix +//! +//! The store's vectors live once, in a [`VectorTable`] (see `table.rs`): a +//! chunked row-major `f32` matrix with a norm and a type code per row. A search +//! is a pass over that memory: a dot product per row, a bounded heap for the +//! best `k`, no per-row allocation, no hashing, **no lock**. Writes need +//! `&mut self`, so a search can never overlap one (see `shared.rs` for the +//! reader/writer wrapper). +//! +//! # One definition of a score +//! +//! Every cosine in the crate (single query, batch, index probe, fold, k-means) +//! is `dot / (|row| * |query|)` where `dot` has one fixed summation order: a +//! single 8-lane accumulator walked in ascending order, a fixed horizontal-sum +//! tree, then the scalar tail. Kernels differ only in how many (query, row) +//! pairs they compute at once; none changes the order inside a pair, so a +//! batched answer is bit-identical to the one-query answer, and the answer does +//! not depend on thread count. +//! +//! # The IVF-flat index +//! +//! Opt-in through [`AnnConfig`]. Spherical k-means (seeded, on a sample) +//! partitions the rows into `nlist` cells by direction; a query scores the +//! `nprobe` cells whose centroids are nearest and only the rows inside them, +//! with exactly the cosine the exact scan uses. The index stores no vectors: +//! per-cell lists of slot numbers, a cell number per slot, and the centroids. +//! New and updated rows wait in a short `pending` list that every query scores +//! exactly; the write that makes it [`FOLD_BATCH`] long folds it into cells on +//! the spot, so a search never has a backlog to pay for. The index is built +//! lazily by the first search that wants it (or by `build_ann_index`) and is +//! never persisted. +//! +//! The behaviour deliberately mirrors the Python port's `ann="ivf"`. + +use crate::hnsw::HnswConfig; +use crate::pool::SearchPool; +use crate::table::{Rows, VectorTable, CHUNK_ROWS}; +use crate::{EmbeddingEncoder, EmbeddingError, EmbeddingStore, Query, SimilarityResult}; +use std::collections::HashMap; +use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering}; +use std::sync::{Arc, Mutex, RwLock, RwLockReadGuard}; +use std::time::Instant; + +/// Below this many vectors the index is not used and search stays exact. +pub const ANN_MIN_SIZE: usize = 50_000; +/// Rows that are new or rewritten since the last fold wait in `pending`, scored +/// exactly by every query. The write that makes the list this long assigns all +/// of them to cells. Smaller is a shorter write pause and a cheaper query-side +/// exact tail; larger is fewer, bigger pauses. +pub const FOLD_BATCH: usize = 256; +/// Retrain once writes plus deletes exceed this fraction of the trained size. +const RETRAIN_FRACTION: f64 = 0.5; +const KMEANS_ITERATIONS: usize = 8; +const SAMPLE_PER_CELL: usize = 40; +/// Threads one single-query scan uses when none is configured (memory +/// bandwidth, not compute, limits it; see `workers`). +const SCAN_AUTO_THREADS: usize = 8; +/// Rows per scan task. Divides the table's chunk size, so a block never +/// straddles chunks. +const BLOCK_ROWS: usize = 1024; +/// A scan spreads over threads only past this many floats per thread. +const PAR_MIN_FLOATS: usize = 1 << 21; +/// The same for the rows of probed cells, which cost more per float. +const PAR_MIN_CELL_FLOATS: usize = 1 << 20; +/// Points per unit of parallel k-means work; a multiple of `TILE`. +const NEAR_BLOCK: usize = 256; +/// Points per tile of the assignment kernel (the tile stays in L2). +const TILE: usize = 64; +/// Queries scored together against one block of rows in a batch. +const BATCH_TILE: usize = 16; +/// Queries handled per pass of a big batch (bounds the per-thread heaps). +const BATCH_CHUNK: usize = 4096; + +const NO_CELL: i32 = -1; +const PENDING: i32 = -2; + +// ============================================================================= +// Public types +// ============================================================================= + +/// Settings for the opt-in approximate index (IVF-flat by default; HNSW with +/// `mode: AnnMode::Hnsw`, see [`AnnConfig::hnsw`]). +/// +/// Off unless given to [`EmbeddingStore::set_ann`]; with it off every search +/// is exact, as before. With it on, a *bounded* top-k search +/// (`top_k < number of vectors`) on a store of at least `min_size` vectors is +/// answered from the index. The scores are the same cosine values the exact +/// scan gives, but a true neighbour in an unprobed cell can be missing: +/// measure it with [`EmbeddingStore::recall_check`] on your own queries. +/// +/// Policy (the Python port's, except that folding is eager): +/// +/// * the index is built by the first search that wants it (or by +/// `build_ann_index`), from k-means on a sample of up to `40 * nlist` live +/// vectors, and is never persisted; +/// * new and updated vectors are scored exactly until [`FOLD_BATCH`] of them +/// are pending, then the write that reaches that count folds them into their +/// nearest cells; +/// * a delete just stops the row being returned; +/// * once writes plus deletes since training exceed 50% of the trained size +/// the next search retrains the index (with an automatic `nlist`, re-chosen +/// for the new size); +/// * a clear, or a store that shrinks below `min_size`, drops the index; +/// * a filter (`node_type`, `threshold`, a graph `node_filter`) applies to the +/// probed rows; if fewer than `top_k` survive the probe is widened 4x and +/// tried again, ending in the exact scan, so a selective filter costs time +/// rather than results. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct AnnConfig { + /// Cells in the index; `None` picks about `4 * sqrt(N)` (16 to 65,536) + /// when the index is built. + pub nlist: Option<usize>, + /// Cells scanned per query; `None` picks about 5% of `nlist`. More cells + /// is higher recall and slower; `nprobe >= nlist` is the exact scan. + pub nprobe: Option<usize>, + /// Below this many vectors search stays exact (an index only pays off on + /// large stores). Default 50,000. + pub min_size: usize, + /// Seed for the k-means initialisation (IVF) or the level draws (HNSW): the + /// same data in the same order gives the same index. + pub seed: u64, + /// Which index answers bounded searches. `Ivf` (the default) keeps every + /// pre-existing behaviour; `Hnsw` is the graph index of `hnsw.rs`. + pub mode: AnnMode, + /// HNSW settings; ignored unless `mode` is `Hnsw`. + pub hnsw: HnswConfig, +} + +/// Which approximate index an [`AnnConfig`] asks for. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub enum AnnMode { + /// IVF-flat: spherical k-means cells (the default). + #[default] + Ivf, + /// A hierarchical navigable small-world graph: slower to build, several + /// times faster per query at equal recall on large stores. + Hnsw, +} + +impl Default for AnnConfig { + fn default() -> Self { + Self { + nlist: None, + nprobe: None, + min_size: ANN_MIN_SIZE, + seed: 0, + mode: AnnMode::Ivf, + hnsw: HnswConfig::default(), + } + } +} + +impl AnnConfig { + fn validate(&self) -> Result<(), EmbeddingError> { + if self.nlist == Some(0) { + return Err(EmbeddingError::InvalidOption( + "ann nlist must be at least 1".to_string(), + )); + } + if self.nprobe == Some(0) { + return Err(EmbeddingError::InvalidOption( + "ann nprobe must be at least 1".to_string(), + )); + } + self.hnsw.validate() + } + + /// Settings for the HNSW index with its defaults (M 16, ef_construction + /// 200, ef_search chosen per query), otherwise as [`AnnConfig::default`]. + pub fn hnsw() -> Self { + Self { + mode: AnnMode::Hnsw, + ..Self::default() + } + } +} + +/// How many searches each path has answered since the store was created. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct AnnSearchCounts { + /// Answered from the index. + pub approximate: u64, + /// Answered by the full scan while an index was configured. + pub exact: u64, + /// Answered from the index after widening the probe at least once. + pub widened: u64, + /// Times the index was (re)trained. + pub builds: u64, +} + +/// What the approximate index looks like right now; see +/// [`EmbeddingStore::ann_stats`]. Never builds anything. +#[derive(Clone, Debug, Default)] +pub struct AnnStats { + /// An [`AnnConfig`] is set. + pub enabled: bool, + /// An index exists and will answer. + pub active: bool, + pub nlist: Option<usize>, + /// Cells scanned per query (the effective value, once an index exists). + pub nprobe: Option<usize>, + /// Vectors the centroids were trained on. + pub trained_n: Option<usize>, + /// Rows scored exactly until they are folded into cells (always fewer than + /// [`FOLD_BATCH`] between writes). + pub pending: usize, + /// Writes plus deletes since training. + pub churn: usize, + pub build_seconds: Option<f64>, + /// Memory the index holds (it stores no vectors). + pub index_bytes: usize, + pub min_size: Option<usize>, + pub searches: AnnSearchCounts, + /// Fold steps run by writes since the index was built. + pub folds: u64, + /// Total time those steps took. + pub fold_seconds: f64, + /// Which index is configured (`Ivf` when none is). + pub mode: AnnMode, + /// HNSW only: the `ef_search` a query uses when it gives none (before the + /// `max(k)` rule): the configured value, else the size-based default. + pub ef_search: Option<usize>, + /// HNSW only: highest layer of the graph. + pub max_level: Option<usize>, + /// HNSW only: nodes in the graph (live rows plus tombstones awaiting + /// compaction). + pub graph_nodes: usize, + /// HNSW only: tombstones still in the graph. + pub graph_dead: usize, + /// HNSW only: rows linked into the graph by writes since the build, and the + /// time that took. + pub inserts: u64, + pub insert_seconds: f64, +} + +/// Where a store's memory goes; see [`EmbeddingStore::memory_stats`]. The +/// vectors are held once, so `matrix_bytes` is the only part that scales with +/// the dimension. +#[derive(Clone, Copy, Debug, Default)] +pub struct MemoryStats { + /// Row slots in use, tombstones included. + pub rows: usize, + pub live: usize, + pub dead: usize, + pub chunks: usize, + /// Bytes allocated for the matrix (`f32` rows). + pub matrix_bytes: usize, + /// Bytes the live vectors need at minimum: `live * dim * 4`. + pub live_vector_bytes: usize, + /// Norms, type codes, per-row metadata, keys, text, the key index. + pub metadata_bytes: usize, + /// The approximate index, if built. + pub index_bytes: usize, +} + +impl MemoryStats { + pub fn total_bytes(&self) -> usize { + self.matrix_bytes + self.metadata_bytes + self.index_bytes + } +} + +/// Recall of the approximate index against the exact scan, over some queries. +/// +/// `recall` is the mean over queries of `|approximate top-k & exact top-k| / +/// |exact top-k|` (1.0 = identical sets); `min_recall` is the worst single +/// query. `approximate` is false when the index was not actually consulted (it +/// is off, the store is smaller than `min_size`, or `nprobe >= nlist`), in +/// which case recall is 1.0 by construction. Times are per-query medians in +/// milliseconds. +#[derive(Clone, Debug)] +pub struct RecallReport { + pub recall: f64, + pub min_recall: f64, + pub queries: usize, + pub k: usize, + pub nlist: usize, + pub nprobe: usize, + pub approximate: bool, + /// HNSW only: the effective `ef_search` (0 otherwise). + pub ef_search: usize, + pub ann_ms: f64, + pub exact_ms: f64, +} + +/// The search counters, updated without a lock. +#[derive(Default)] +pub(crate) struct Counters { + approximate: AtomicU64, + exact: AtomicU64, + widened: AtomicU64, + pub(crate) builds: AtomicU64, +} + +impl Counters { + /// One search answered from the index (`widened` if it needed a wider probe). + pub(crate) fn note_approximate(&self, widened: bool) { + self.approximate.fetch_add(1, Ordering::Relaxed); + if widened { + self.widened.fetch_add(1, Ordering::Relaxed); + } + } + + pub(crate) fn snapshot(&self) -> AnnSearchCounts { + AnnSearchCounts { + approximate: self.approximate.load(Ordering::Relaxed), + exact: self.exact.load(Ordering::Relaxed), + widened: self.widened.load(Ordering::Relaxed), + builds: self.builds.load(Ordering::Relaxed), + } + } +} + +// ============================================================================= +// Small helpers +// ============================================================================= + +/// SplitMix64: tiny, seedable, and the same on every platform. +pub(crate) struct Rng(pub(crate) u64); + +impl Rng { + pub(crate) fn next_u64(&mut self) -> u64 { + self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } + + /// Uniform in `0..n` (`n > 0`). + pub(crate) fn below(&mut self, n: usize) -> usize { + ((self.next_u64() as u128 * n as u128) >> 64) as usize + } + + /// `count` distinct values of `items`, by a partial Fisher-Yates. + fn choose<T: Copy>(&mut self, items: &[T], count: usize) -> Vec<T> { + let mut pool = items.to_vec(); + let count = count.min(pool.len()); + for i in 0..count { + let j = i + self.below(pool.len() - i); + pool.swap(i, j); + } + pool.truncate(count); + pool + } +} + +/// Round half to even, as Python's `round` does, so the automatic `nlist` and +/// `nprobe` match the reference port on the half-way cases. +fn round_half_even(x: f64) -> f64 { + let floor = x.floor(); + let diff = x - floor; + if diff < 0.5 { + floor + } else if diff > 0.5 { + floor + 1.0 + } else if floor % 2.0 == 0.0 { + floor + } else { + floor + 1.0 + } +} + +fn auto_nlist(n: usize) -> usize { + (round_half_even(4.0 * (n as f64).sqrt()).max(16.0) as usize).min(65_536) +} + +/// How a step may use the pool: the store's configured thread count (0 = auto) +/// and the pool to run on. +#[derive(Clone, Copy)] +pub(crate) struct Par<'a> { + pub(crate) configured: usize, + pub(crate) pool: &'a Option<Arc<SearchPool>>, +} + +impl Par<'_> { + /// Threads available: the pool's size. + pub(crate) fn size(&self) -> usize { + self.pool + .as_ref() + .map_or_else(SearchPool::default_threads, |p| p.threads()) + } + + /// Participants for `tasks` units of work: the configured count, or with 0 + /// the pool size up to `auto_cap`. A single-query scan is limited by memory + /// bandwidth, not compute; on the machine this was tuned on eight threads + /// were within a few percent of thirty-two with a much steadier tail. + pub(crate) fn workers(&self, tasks: usize, auto_cap: usize) -> usize { + let want = if self.configured == 0 { + self.size().min(auto_cap) + } else { + self.configured + }; + want.min(tasks).max(1) + } + + pub(crate) fn run(&self, participants: usize, work: &(dyn Fn(usize) + Sync)) { + if participants <= 1 { + work(0); + return; + } + match self.pool { + Some(pool) => pool.run(participants, work), + None => SearchPool::global().run(participants, work), + } + } +} + +// ============================================================================= +// Kernels +// +// One family, `block::<Q, R, F>`: Q queries against R rows at once, returning +// the Q x R dot products. Each (query, row) pair is summed in the same order +// whatever Q and R are (see the module docs), so callers pick the shape that +// suits the loop they are in. `F` selects fused multiply-add; the `true` +// instantiation is only ever reached through a function that enabled the +// feature, so `mul_add` compiles to one instruction, never a library call. +// ============================================================================= + +#[inline(always)] +fn fma<const F: bool>(a: f32, b: f32, c: f32) -> f32 { + if F { + a.mul_add(b, c) + } else { + a * b + c + } +} + +#[cfg(target_arch = "x86_64")] +mod x86 { + use std::arch::x86_64::*; + + #[inline] + #[target_feature(enable = "avx2,fma")] + unsafe fn hsum(v: __m256) -> f32 { + // (l0+l4, l1+l5, l2+l6, l3+l7), then (t0+t2) + (t1+t3): the tree the + // portable path spells out. + let lo = _mm256_castps256_ps128(v); + let hi = _mm256_extractf128_ps(v, 1); + let s = _mm_add_ps(lo, hi); + let s = _mm_add_ps(s, _mm_movehl_ps(s, s)); + let s = _mm_add_ss(s, _mm_shuffle_ps(s, s, 1)); + _mm_cvtss_f32(s) + } + + /// `Q` queries against `R` rows. + /// + /// # Safety + /// The CPU must support AVX2 and FMA; every slice must be exactly `dim` + /// long (the callers slice rows to `dim`). + #[inline] + #[target_feature(enable = "avx2,fma")] + pub unsafe fn block<const Q: usize, const R: usize>( + q: [&[f32]; Q], + r: [&[f32]; R], + dim: usize, + ) -> [[f32; R]; Q] { + debug_assert!(q.iter().all(|s| s.len() == dim) && r.iter().all(|s| s.len() == dim)); + let qp: [*const f32; Q] = std::array::from_fn(|i| q[i].as_ptr()); + let rp: [*const f32; R] = std::array::from_fn(|i| r[i].as_ptr()); + let mut acc = [[_mm256_setzero_ps(); R]; Q]; + let body = dim & !7; + let mut i = 0; + while i < body { + let mut rv = [_mm256_setzero_ps(); R]; + for j in 0..R { + rv[j] = _mm256_loadu_ps(rp[j].add(i)); + } + for a in 0..Q { + let qv = _mm256_loadu_ps(qp[a].add(i)); + for j in 0..R { + acc[a][j] = _mm256_fmadd_ps(qv, rv[j], acc[a][j]); + } + } + i += 8; + } + let mut out = [[0.0f32; R]; Q]; + for a in 0..Q { + for j in 0..R { + let mut s = hsum(acc[a][j]); + let mut t = body; + while t < dim { + s += *qp[a].add(t) * *rp[j].add(t); + t += 1; + } + out[a][j] = s; + } + } + out + } +} + +#[cfg(target_arch = "x86_64")] +fn has_avx2_fma() -> bool { + std::is_x86_feature_detected!("avx2") && std::is_x86_feature_detected!("fma") +} + +/// Define `$name` as a run-time dispatch between the AVX2+FMA and the portable +/// instantiation of `$imp`. +macro_rules! dispatch { + ($(#[$meta:meta])* $vis:vis fn $name:ident($($arg:ident: $ty:ty),* $(,)?) $(-> $ret:ty)? = $imp:ident) => { + $(#[$meta])* + $vis fn $name($($arg: $ty),*) $(-> $ret)? { + #[cfg(target_arch = "x86_64")] + { + #[target_feature(enable = "avx2,fma")] + unsafe fn fast($($arg: $ty),*) $(-> $ret)? { + $imp::<true>($($arg),*) + } + if has_avx2_fma() { + // SAFETY: the CPU reports AVX2 and FMA, which is all `fast` + // requires; it touches nothing but its arguments. + return unsafe { fast($($arg),*) }; + } + } + $imp::<false>($($arg),*) + } + }; +} + +/// `Q` queries against `R` rows, each exactly `dim` floats. +#[inline(always)] +fn block<const Q: usize, const R: usize, const F: bool>( + q: [&[f32]; Q], + r: [&[f32]; R], +) -> [[f32; R]; Q] { + let dim = q[0].len(); + #[cfg(target_arch = "x86_64")] + if F { + // the kernel reads `dim` floats from every slice: make that a fact, not a hope + assert!(q.iter().all(|s| s.len() == dim) && r.iter().all(|s| s.len() == dim)); + // SAFETY: `F` is true only in the instantiation reached through a + // `dispatch!` wrapper, which has checked for AVX2 and FMA. + return unsafe { x86::block::<Q, R>(q, r, dim) }; + } + const LANES: usize = 8; + let body = dim - dim % LANES; + let mut acc = [[[0.0f32; LANES]; R]; Q]; + let mut i = 0; + while i < body { + for a in 0..Q { + for j in 0..R { + for l in 0..LANES { + acc[a][j][l] = fma::<F>(q[a][i + l], r[j][i + l], acc[a][j][l]); + } + } + } + i += LANES; + } + let mut out = [[0.0f32; R]; Q]; + for a in 0..Q { + for j in 0..R { + let v = &acc[a][j]; + let t = [v[0] + v[4], v[1] + v[5], v[2] + v[6], v[3] + v[7]]; + let mut s = (t[0] + t[2]) + (t[1] + t[3]); + for x in body..dim { + s += q[a][x] * r[j][x]; + } + out[a][j] = s; + } + } + out +} + +/// Dot product in the crate's one summation order. +#[inline(always)] +fn dot_impl<const F: bool>(a: &[f32], b: &[f32]) -> f32 { + block::<1, 1, F>([a], [b])[0][0] +} + +dispatch! { + pub(crate) fn dot_f32(a: &[f32], b: &[f32]) -> f32 = dot_impl +} + +/// `out[i] = dot(query, row i of mat)`. +#[inline(always)] +fn dots_impl<const F: bool>(query: &[f32], mat: &[f32], dim: usize, out: &mut [f32]) { + let n = out.len(); + let row = |i: usize| &mat[i * dim..(i + 1) * dim]; + let mut i = 0; + while i + 4 <= n { + let d = block::<1, 4, F>([query], [row(i), row(i + 1), row(i + 2), row(i + 3)]); + out[i..i + 4].copy_from_slice(&d[0]); + i += 4; + } + while i < n { + out[i] = block::<1, 1, F>([query], [row(i)])[0][0]; + i += 1; + } +} + +dispatch! { + fn dots_into(query: &[f32], mat: &[f32], dim: usize, out: &mut [f32]) = dots_impl +} + +/// For each point in `pts` (row-major, `out.len()` rows of `dim`), the index of +/// the centroid with the largest dot product (the first on a tie). Points are +/// handled in tiles that stay in cache while the centroids stream past. Every +/// point's dot products use the canonical order, so its answer does not depend +/// on which tile or block it is in. +#[inline(always)] +fn nearest_impl<const F: bool>( + pts: &[f32], + dim: usize, + cents: &[f32], + nlist: usize, + out: &mut [i32], +) { + let m = out.len(); + let point = |p: usize| &pts[p * dim..(p + 1) * dim]; + let cent = |c: usize| ¢s[c * dim..(c + 1) * dim]; + let mut best_s = [f32::NEG_INFINITY; TILE]; + let mut best_i = [0i32; TILE]; + let mut p0 = 0; + while p0 < m { + let t = (m - p0).min(TILE); + best_s[..t].fill(f32::NEG_INFINITY); + best_i[..t].fill(0); + let mut c = 0; + while c + 4 <= nlist { + let group = [cent(c), cent(c + 1), cent(c + 2), cent(c + 3)]; + let mut p = 0; + while p + 2 <= t { + let d = block::<2, 4, F>([point(p0 + p), point(p0 + p + 1)], group); + for (pi, row) in d.iter().enumerate() { + for (ci, &s) in row.iter().enumerate() { + if s > best_s[p + pi] { + best_s[p + pi] = s; + best_i[p + pi] = (c + ci) as i32; + } + } + } + p += 2; + } + if p < t { + let d = block::<1, 4, F>([point(p0 + p)], group); + for (ci, &s) in d[0].iter().enumerate() { + if s > best_s[p] { + best_s[p] = s; + best_i[p] = (c + ci) as i32; + } + } + } + c += 4; + } + while c < nlist { + let mut p = 0; + while p + 2 <= t { + let d = block::<2, 1, F>([point(p0 + p), point(p0 + p + 1)], [cent(c)]); + for pi in 0..2 { + if d[pi][0] > best_s[p + pi] { + best_s[p + pi] = d[pi][0]; + best_i[p + pi] = c as i32; + } + } + p += 2; + } + if p < t { + let s = block::<1, 1, F>([point(p0 + p)], [cent(c)])[0][0]; + if s > best_s[p] { + best_s[p] = s; + best_i[p] = c as i32; + } + } + c += 1; + } + out[p0..p0 + t].copy_from_slice(&best_i[..t]); + p0 += t; + } +} + +dispatch! { + fn nearest_block(pts: &[f32], dim: usize, cents: &[f32], nlist: usize, out: &mut [i32]) = nearest_impl +} + +/// `nearest_block` over `m = out.len()` points spread over threads in fixed +/// blocks. `points(lo, len)` returns the contiguous floats of points +/// `lo..lo + len`; a block never straddles a table chunk because the block +/// sizes divide the chunk size. +fn nearest_parallel<'a>( + points: &(dyn Fn(usize, usize) -> &'a [f32] + Sync), + dim: usize, + cents: &[f32], + nlist: usize, + out: &mut [i32], + par: Par, +) { + let m = out.len(); + // big jobs in NEAR_BLOCK units; a small one (a fold) is cut finer, so that + // every thread of the pool gets a piece + let block = if m >= 8 * NEAR_BLOCK { + NEAR_BLOCK + } else { + (m / par.size().max(1)).next_power_of_two().clamp(16, TILE) + }; + let blocks = m.div_ceil(block); + let work = m.saturating_mul(nlist).saturating_mul(dim); + let threads = if work < (1 << 24) { + 1 + } else { + par.workers(blocks, usize::MAX) + }; + if threads <= 1 { + for (b, chunk) in out.chunks_mut(block).enumerate() { + nearest_block(points(b * block, chunk.len()), dim, cents, nlist, chunk); + } + return; + } + let queue = Mutex::new(out.chunks_mut(block).enumerate()); + par.run(threads, &|_| loop { + let next = queue.lock().ok().and_then(|mut q| q.next()); + let Some((b, chunk)) = next else { break }; + nearest_block(points(b * block, chunk.len()), dim, cents, nlist, chunk); + }); +} + +// ============================================================================= +// Top-k selection +// ============================================================================= + +/// A scored row. Ordered by score, then by arrival (lower slot first). +#[derive(Clone, Copy, Debug)] +pub(crate) struct Hit { + pub score: f32, + pub slot: u32, +} + +/// `a` is a worse answer than `b`. +#[inline(always)] +fn worse(a: Hit, b: Hit) -> bool { + a.score < b.score || (a.score == b.score && a.slot > b.slot) +} + +/// The best `k` rows above a threshold: a min-heap whose root is the worst row +/// kept, so most rows are rejected with one comparison against `bar`. Asked +/// for at least as many rows as can exist, it just collects them. +pub(crate) struct TopK { + k: usize, + collect_all: bool, + /// Rows must score at least this much: the threshold until the heap is + /// full, then the worst score kept. + bar: f32, + heap: Vec<Hit>, +} + +impl TopK { + /// `bound` is the most rows that can ever be offered. + pub fn new(k: usize, threshold: f32, bound: usize) -> Self { + let collect_all = k >= bound; + Self { + k, + collect_all, + bar: threshold, + heap: Vec::with_capacity(if collect_all { 0 } else { k.min(1 << 16) }), + } + } + + pub fn len(&self) -> usize { + self.heap.len() + } + + #[inline(always)] + #[allow(clippy::neg_cmp_op_on_partial_ord)] // `!(a >= b)` is deliberately true for NaN + pub fn offer(&mut self, score: f32, slot: u32) { + // False for NaN as well, so tombstones (norm NaN) and corrupt rows + // drop out here. + if !(score >= self.bar) { + return; + } + self.accept(Hit { score, slot }); + } + + #[inline(never)] + fn accept(&mut self, hit: Hit) { + if self.collect_all { + self.heap.push(hit); + } else if self.heap.len() < self.k { + self.heap.push(hit); + self.sift_up(self.heap.len() - 1); + if self.heap.len() == self.k { + self.bar = self.heap[0].score; + } + } else if worse(self.heap[0], hit) { + self.heap[0] = hit; + self.sift_down(0); + self.bar = self.heap[0].score; + } + } + + fn sift_up(&mut self, mut i: usize) { + while i > 0 { + let parent = (i - 1) / 2; + if worse(self.heap[i], self.heap[parent]) { + self.heap.swap(i, parent); + i = parent; + } else { + break; + } + } + } + + fn sift_down(&mut self, mut i: usize) { + let len = self.heap.len(); + loop { + let (l, r) = (2 * i + 1, 2 * i + 2); + let mut m = i; + if l < len && worse(self.heap[l], self.heap[m]) { + m = l; + } + if r < len && worse(self.heap[r], self.heap[m]) { + m = r; + } + if m == i { + break; + } + self.heap.swap(i, m); + i = m; + } + } + + pub fn merge(&mut self, other: TopK) { + for hit in other.heap { + self.offer(hit.score, hit.slot); + } + } + + /// Best first; equal scores in arrival order. + pub fn into_sorted(mut self) -> Vec<Hit> { + self.heap.sort_unstable_by(|a, b| { + b.score + .partial_cmp(&a.score) + .unwrap_or(std::cmp::Ordering::Equal) + .then(a.slot.cmp(&b.slot)) + }); + self.heap + } +} + +// ============================================================================= +// Scoring +// ============================================================================= + +pub(crate) struct QueryVec<'a> { + pub(crate) v: &'a [f32], + pub(crate) norm: f32, +} + +impl<'a> QueryVec<'a> { + pub(crate) fn new(v: &'a [f32]) -> Self { + Self { + v, + norm: dot_f32(v, v).sqrt(), + } + } + + /// A direction to rank cells by. + pub(crate) fn has_direction(&self) -> bool { + self.norm > 0.0 && self.norm.is_finite() + } +} + +/// Cosine from a dot product and the two norms. A zero-length row scores 0, a +/// tombstone (norm NaN) scores NaN and is dropped by the collector. +#[inline(always)] +pub(crate) fn cosine(dot: f32, row_norm: f32, query_norm: f32) -> f32 { + let denom = row_norm * query_norm; + if denom == 0.0 { + 0.0 + } else { + dot / denom + } +} + +/// Rows scored per kernel call by a single query. +const QW: usize = 4; + +/// Collects rows `QW` at a time so one query is scored against `QW` rows per +/// kernel call (`QW` independent dependency chains and memory streams). +struct Quad<'a> { + slots: [usize; QW], + rows: [&'a [f32]; QW], + m: usize, +} + +impl<'a> Quad<'a> { + #[inline(always)] + fn new() -> Self { + Self { + slots: [0; QW], + rows: [&[]; QW], + m: 0, + } + } + + /// Queue `slot`, looking its row up in the (chunked) matrix. + #[inline(always)] + fn push<const F: bool>( + &mut self, + rows: &Rows<'a>, + q: &QueryVec, + slot: usize, + sink: &mut impl FnMut(usize, f32), + ) { + self.push_row::<F>(rows.norms, q, slot, rows.row(slot), sink); + } + + /// Queue `slot` whose row the caller already has in hand. + #[inline(always)] + fn push_row<const F: bool>( + &mut self, + norms: &[f32], + q: &QueryVec, + slot: usize, + row: &'a [f32], + sink: &mut impl FnMut(usize, f32), + ) { + self.slots[self.m] = slot; + self.rows[self.m] = row; + self.m += 1; + if self.m == QW { + self.m = 0; + let d = block::<1, QW, F>([q.v], self.rows); + for i in 0..QW { + sink(self.slots[i], cosine(d[0][i], norms[self.slots[i]], q.norm)); + } + } + } + + #[inline(always)] + fn finish<const F: bool>( + &mut self, + rows: &Rows, + q: &QueryVec, + sink: &mut impl FnMut(usize, f32), + ) { + for i in 0..self.m { + let slot = self.slots[i]; + let d = block::<1, 1, F>([q.v], [self.rows[i]]); + sink(slot, cosine(d[0][0], rows.norms[slot], q.norm)); + } + self.m = 0; + } +} + +/// Score rows `lo..hi`. +#[inline(always)] +fn scan_range_impl<const F: bool>( + rows: Rows, + q: &QueryVec, + code: Option<u32>, + lo: usize, + hi: usize, + top: &mut TopK, +) { + let mut quad = Quad::new(); + let mut sink = |slot: usize, score: f32| top.offer(score, slot as u32); + // walk the matrix one contiguous run (the rest of a chunk) at a time, so a + // row is a slice of the run rather than two dependent lookups + let dim = rows.dim; + let mut at = lo; + while at < hi { + let (flat, count) = rows.run(at, hi); + for i in 0..count { + let slot = at + i; + if let Some(code) = code { + if rows.codes[slot] != code { + continue; + } + } + quad.push_row::<F>( + rows.norms, + q, + slot, + &flat[i * dim..(i + 1) * dim], + &mut sink, + ); + } + at += count; + } + quad.finish::<F>(&rows, q, &mut sink); +} + +dispatch! { + fn scan_range(rows: Rows, q: &QueryVec, code: Option<u32>, lo: usize, hi: usize, top: &mut TopK) = scan_range_impl +} + +/// `out[i]` = cosine of the query with row `slots[i]`, computed exactly as the +/// scan computes it (one query against four rows at a time, then singly), so a +/// score from here is bit-identical to the scan's. A tombstone scores NaN. +#[inline(always)] +fn score_slots_impl<const F: bool>(rows: Rows, q: &QueryVec, slots: &[u32], out: &mut [f32]) { + let row = |i: usize| rows.row(slots[i] as usize); + let norm = |i: usize| rows.norms[slots[i] as usize]; + let mut i = 0; + while i + QW <= slots.len() { + let d = block::<1, QW, F>([q.v], [row(i), row(i + 1), row(i + 2), row(i + 3)]); + for j in 0..QW { + out[i + j] = cosine(d[0][j], norm(i + j), q.norm); + } + i += QW; + } + while i < slots.len() { + let d = block::<1, 1, F>([q.v], [row(i)]); + out[i] = cosine(d[0][0], norm(i), q.norm); + i += 1; + } +} + +dispatch! { + pub(crate) fn score_slots(rows: Rows, q: &QueryVec, slots: &[u32], out: &mut [f32]) = score_slots_impl +} + +/// Score every live row (for the score map) and hand `(slot, score)` on. +#[inline(always)] +fn score_all_impl<const F: bool>(rows: Rows, q: &QueryVec, n: usize, out: &mut Vec<(u32, f32)>) { + let mut quad = Quad::new(); + let mut sink = |slot: usize, score: f32| out.push((slot as u32, score)); + for slot in 0..n { + if rows.norms[slot].is_nan() { + continue; + } + quad.push::<F>(&rows, q, slot, &mut sink); + } + quad.finish::<F>(&rows, q, &mut sink); +} + +dispatch! { + fn score_all(rows: Rows, q: &QueryVec, n: usize, out: &mut Vec<(u32, f32)>) = score_all_impl +} + +/// The parts of an [`Ivf`] a cell scan reads. +#[derive(Clone, Copy)] +struct Cells<'a> { + lists: &'a [Vec<u32>], + pending: &'a [u32], +} + +/// Score the rows of the `chosen` cells, and every pending row if asked. +#[inline(always)] +fn scan_cells_impl<const F: bool>( + rows: Rows, + cells: Cells, + chosen: &[u32], + with_pending: bool, + q: &QueryVec, + code: Option<u32>, + top: &mut TopK, +) { + let mut quad = Quad::new(); + let mut sink = |slot: usize, score: f32| top.offer(score, slot as u32); + for &c in chosen { + for &slot in &cells.lists[c as usize] { + let slot = slot as usize; + if let Some(code) = code { + if rows.codes[slot] != code { + continue; + } + } + quad.push::<F>(&rows, q, slot, &mut sink); + } + } + if with_pending { + for &slot in cells.pending { + let slot = slot as usize; + if let Some(code) = code { + if rows.codes[slot] != code { + continue; + } + } + quad.push::<F>(&rows, q, slot, &mut sink); + } + } + quad.finish::<F>(&rows, q, &mut sink); +} + +dispatch! { + fn scan_cells(rows: Rows, cells: Cells, chosen: &[u32], with_pending: bool, q: &QueryVec, code: Option<u32>, top: &mut TopK) = scan_cells_impl +} + +/// A tile of queries against one block of rows (`slots`, already filtered by +/// type): four queries against two rows per kernel call, so a row loaded from +/// L2 is used by four queries and a query loaded from L1 by two rows. +#[inline(always)] +fn batch_block_impl<const F: bool>(rows: Rows, qs: &[&QueryVec], slots: &[u32], tops: &mut [TopK]) { + let mut base = 0; + for group in qs.chunks(4) { + if group.len() == 4 { + let qv = [group[0].v, group[1].v, group[2].v, group[3].v]; + let mut i = 0; + while i + 2 <= slots.len() { + let (s0, s1) = (slots[i] as usize, slots[i + 1] as usize); + let d = block::<4, 2, F>(qv, [rows.row(s0), rows.row(s1)]); + for a in 0..4 { + let top = &mut tops[base + a]; + top.offer(cosine(d[a][0], rows.norms[s0], group[a].norm), s0 as u32); + top.offer(cosine(d[a][1], rows.norms[s1], group[a].norm), s1 as u32); + } + i += 2; + } + if i < slots.len() { + let s0 = slots[i] as usize; + let d = block::<4, 1, F>(qv, [rows.row(s0)]); + for a in 0..4 { + tops[base + a].offer(cosine(d[a][0], rows.norms[s0], group[a].norm), s0 as u32); + } + } + } else { + for (a, q) in group.iter().enumerate() { + let top = &mut tops[base + a]; + let mut quad = Quad::new(); + let mut sink = |slot: usize, score: f32| top.offer(score, slot as u32); + for &slot in slots { + quad.push::<F>(&rows, q, slot as usize, &mut sink); + } + quad.finish::<F>(&rows, q, &mut sink); + } + } + base += group.len(); + } +} + +dispatch! { + fn batch_block(rows: Rows, qs: &[&QueryVec], slots: &[u32], tops: &mut [TopK]) = batch_block_impl +} + +// ============================================================================= +// The IVF index +// ============================================================================= + +pub(crate) struct Ivf { + /// `nlist` unit vectors, row-major. + centroids: Vec<f32>, + nlist: usize, + dim: usize, + /// Per row slot: its cell, `PENDING`, or `NO_CELL` (deleted). + assign: Vec<i32>, + /// Per row slot: its position in its cell's list (or in `pending`), so a + /// row leaves a list in O(1). + pos: Vec<u32>, + /// The slots of each cell. No stale entries: a list holds exactly the live, + /// folded rows assigned to it. + lists: Vec<Vec<u32>>, + /// Live rows with no cell yet; every query scores them exactly. + pending: Vec<u32>, + churn: usize, + trained_n: usize, + build_seconds: f64, + folds: u64, + fold_seconds: f64, +} + +impl Ivf { + /// Train centroids on a sample, then give every live row its cell. + fn build(table: &VectorTable, nlist: usize, seed: u64, par: Par) -> Ivf { + let started = Instant::now(); + let (n, dim) = (table.slots(), table.dim); + let live: Vec<u32> = table.live_slots().map(|s| s as u32).collect(); + let total = live.len(); + let nlist = nlist.clamp(1, total.max(1)); + let mut rng = Rng(seed); + + let sample_n = total.min(SAMPLE_PER_CELL * nlist); + let sample: Vec<u32> = if sample_n == total { + live + } else { + let mut picked = rng.choose(&live, sample_n); + picked.sort_unstable(); + picked + }; + + // Unit-length copies of the sample (a zero row stays as it is). + let mut x = vec![0.0f32; sample_n * dim]; + for (i, &slot) in sample.iter().enumerate() { + let norm = table.norm(slot as usize); + let scale = if norm > 0.0 && norm.is_finite() { + 1.0 / norm + } else { + 1.0 + }; + for (dst, src) in x[i * dim..(i + 1) * dim] + .iter_mut() + .zip(table.row(slot as usize)) + { + *dst = src * scale; + } + } + + let sample_ids: Vec<usize> = (0..sample_n).collect(); + let mut centroids = vec![0.0f32; nlist * dim]; + for (c, &i) in rng.choose(&sample_ids, nlist).iter().enumerate() { + centroids[c * dim..(c + 1) * dim].copy_from_slice(&x[i * dim..(i + 1) * dim]); + } + + let mut cell = vec![0i32; sample_n]; + let mut sums = vec![0.0f64; nlist * dim]; + let mut counts = vec![0u32; nlist]; + for _ in 0..KMEANS_ITERATIONS { + nearest_parallel( + &|lo, len| &x[lo * dim..(lo + len) * dim], + dim, + ¢roids, + nlist, + &mut cell, + par, + ); + sums.fill(0.0); + counts.fill(0); + for (i, &c) in cell.iter().enumerate() { + let c = c as usize; + counts[c] += 1; + for (s, v) in sums[c * dim..(c + 1) * dim] + .iter_mut() + .zip(&x[i * dim..(i + 1) * dim]) + { + *s += *v as f64; + } + } + for c in 0..nlist { + let sum = &sums[c * dim..(c + 1) * dim]; + let length = sum.iter().map(|v| v * v).sum::<f64>().sqrt(); + let target = &mut centroids[c * dim..(c + 1) * dim]; + if counts[c] == 0 || length == 0.0 || !length.is_finite() { + // A dead centroid is re-seeded on a random sample point. + let i = rng.below(sample_n); + target.copy_from_slice(&x[i * dim..(i + 1) * dim]); + } else { + for (t, s) in target.iter_mut().zip(sum) { + *t = (*s / length) as f32; + } + } + } + } + drop(x); + + // Every row (dead ones too, to keep the kernel's blocks fixed) gets its + // nearest cell; the dead are then taken out again. + let mut assign = vec![0i32; n]; + nearest_parallel( + &|lo, len| table.run(lo, lo + len).0, + dim, + ¢roids, + nlist, + &mut assign, + par, + ); + for (slot, a) in assign.iter_mut().enumerate() { + if !table.is_live(slot) { + *a = NO_CELL; + } + } + let mut index = Ivf { + centroids, + nlist, + dim, + assign, + pos: vec![0; n], + lists: Vec::new(), + pending: Vec::new(), + churn: 0, + trained_n: total, + build_seconds: 0.0, + folds: 0, + fold_seconds: 0.0, + }; + index.relist(); + index.build_seconds = started.elapsed().as_secs_f64(); + index + } + + /// Rebuild every cell list from `assign` (slot order within a cell). + fn relist(&mut self) { + let mut counts = vec![0usize; self.nlist]; + for &c in &self.assign { + if c >= 0 { + counts[c as usize] += 1; + } + } + self.lists = counts.iter().map(|&c| Vec::with_capacity(c)).collect(); + for (slot, &c) in self.assign.iter().enumerate() { + if c >= 0 { + let list = &mut self.lists[c as usize]; + self.pos[slot] = list.len() as u32; + list.push(slot as u32); + } + } + } + + /// Past the churn that calls for retraining. + fn stale(&self) -> bool { + self.churn as f64 > RETRAIN_FRACTION * self.trained_n as f64 + } + + /// Take `slot` out of whatever list holds it. + fn detach(&mut self, slot: u32) { + let s = slot as usize; + let Some(&cell) = self.assign.get(s) else { + return; + }; + let at = self.pos[s] as usize; + let list = match cell { + NO_CELL => return, + PENDING => &mut self.pending, + c => &mut self.lists[c as usize], + }; + let last = list.len() - 1; + list.swap_remove(at); + if at != last { + self.pos[list[at] as usize] = at as u32; + } + self.assign[s] = NO_CELL; + } + + /// A row was written (new, or its vector replaced): it waits in `pending`. + fn note_put(&mut self, slot: u32) { + let s = slot as usize; + if s >= self.assign.len() { + self.assign.resize(s + 1, NO_CELL); + self.pos.resize(s + 1, 0); + } + self.detach(slot); + self.assign[s] = PENDING; + self.pos[s] = self.pending.len() as u32; + self.pending.push(slot); + self.churn += 1; + } + + fn note_drop(&mut self, slot: u32) { + self.detach(slot); + self.churn += 1; + } + + /// Give every pending row its nearest cell. + fn fold(&mut self, table: &VectorTable, par: Par) { + if self.pending.is_empty() { + return; + } + let started = Instant::now(); + let dim = self.dim; + let slots = std::mem::take(&mut self.pending); + let mut gathered = Vec::with_capacity(slots.len() * dim); + for &s in &slots { + gathered.extend_from_slice(table.row(s as usize)); + } + let mut cells = vec![0i32; slots.len()]; + nearest_parallel( + &|lo, len| &gathered[lo * dim..(lo + len) * dim], + dim, + &self.centroids, + self.nlist, + &mut cells, + par, + ); + for (&s, &c) in slots.iter().zip(&cells) { + let list = &mut self.lists[c as usize]; + self.assign[s as usize] = c; + self.pos[s as usize] = list.len() as u32; + list.push(s); + } + self.folds += 1; + self.fold_seconds += started.elapsed().as_secs_f64(); + } + + /// Follow the table's compaction (`live` = old slots kept, in order). + fn compact(&mut self, live: &[u32]) { + let moved: Vec<i32> = live.iter().map(|&old| self.assign[old as usize]).collect(); + let pending: Vec<u32> = self + .pending + .iter() + .filter_map(|old| live.binary_search(old).ok().map(|new| new as u32)) + .collect(); + self.assign = moved; + self.pos = vec![0; live.len()]; + self.relist(); + for (i, &slot) in pending.iter().enumerate() { + self.pos[slot as usize] = i as u32; + } + self.pending = pending; + } + + /// Every invariant of the index against the table it indexes. + fn check(&self, table: &VectorTable, pending_cap: usize) -> Result<(), String> { + let n = table.slots(); + if self.assign.len() != n || self.pos.len() != n { + return Err(format!( + "index covers {} rows (pos {}), table has {n}", + self.assign.len(), + self.pos.len() + )); + } + if self.pending.len() >= pending_cap { + return Err(format!( + "{} rows pending, the cap is {pending_cap}", + self.pending.len() + )); + } + let mut seen = 0usize; + for (c, list) in self.lists.iter().enumerate() { + for (at, &slot) in list.iter().enumerate() { + let s = slot as usize; + if self.assign[s] != c as i32 || self.pos[s] as usize != at { + return Err(format!( + "slot {slot} listed in cell {c} at {at} but assigned {} at {}", + self.assign[s], self.pos[s] + )); + } + if !table.is_live(s) { + return Err(format!("dead slot {slot} still in cell {c}")); + } + seen += 1; + } + } + for (at, &slot) in self.pending.iter().enumerate() { + let s = slot as usize; + if self.assign[s] != PENDING || self.pos[s] as usize != at || !table.is_live(s) { + return Err(format!("pending slot {slot} is inconsistent")); + } + seen += 1; + } + for (slot, &assigned) in self.assign.iter().enumerate().take(n) { + let live = table.is_live(slot); + match (live, assigned) { + (false, NO_CELL) => {} + (false, a) => return Err(format!("dead slot {slot} assigned {a}")), + (true, NO_CELL) => return Err(format!("live slot {slot} is in no cell")), + (true, _) => {} + } + } + if seen != table.live() { + return Err(format!("{seen} rows in lists, {} live", table.live())); + } + Ok(()) + } + + fn nbytes(&self) -> usize { + self.centroids.len() * 4 + + self.assign.len() * 4 + + self.pos.len() * 4 + + self + .lists + .iter() + .map(|l| l.capacity() * 4 + 24) + .sum::<usize>() + + self.pending.capacity() * 4 + } + + /// The `lists` cells whose centroids are most similar to the query + /// (ascending, for memory locality). + fn choose_cells(&self, q: &QueryVec, lists: usize) -> Vec<u32> { + if lists >= self.nlist { + return (0..self.nlist as u32).collect(); + } + let mut scores = vec![0.0f32; self.nlist]; + dots_into(q.v, &self.centroids, self.dim, &mut scores); + let mut order: Vec<u32> = (0..self.nlist as u32).collect(); + order.select_nth_unstable_by(lists - 1, |&a, &b| { + scores[b as usize] + .total_cmp(&scores[a as usize]) + .then(a.cmp(&b)) + }); + order.truncate(lists); + order.sort_unstable(); + order + } +} + +fn effective_nprobe(cfg: &AnnConfig, nlist: usize) -> usize { + cfg.nprobe + .unwrap_or_else(|| (round_half_even(0.05 * nlist as f64) as usize).max(1)) + .min(nlist) +} + +// ============================================================================= +// The store's search surface +// ============================================================================= + +/// Matches from a search, best first, resolved to results one at a time (so a +/// caller that stops early never builds the rest). Returned by +/// `EmbeddingStore::ranked_hits`. +pub(crate) struct RankedHits<'a> { + store: &'a EmbeddingStore, + hits: std::vec::IntoIter<Hit>, +} + +impl Iterator for RankedHits<'_> { + type Item = SimilarityResult; + + fn next(&mut self) -> Option<SimilarityResult> { + for hit in self.hits.by_ref() { + if let Some(result) = self.store.resolve_hit(hit) { + return Some(result); + } + } + None + } +} + +impl EmbeddingStore { + /// A store whose bounded searches use the approximate index. + pub fn with_ann( + encoder: Option<Box<dyn EmbeddingEncoder>>, + config: AnnConfig, + ) -> Result<Self, EmbeddingError> { + let mut store = Self::new(encoder); + store.set_ann(Some(config))?; + Ok(store) + } + + /// Switch the approximate index on (with these settings) or, with `None`, + /// off. Any index already built is dropped. Nothing is saved: a saved + /// graph or sidecar is the same with or without it. + pub fn set_ann(&mut self, config: Option<AnnConfig>) -> Result<(), EmbeddingError> { + if let Some(config) = &config { + config.validate()?; + } + self.ann = config; + *self.ivf_mut() = None; + *self.hnsw_mut() = None; + Ok(()) + } + + /// The approximate-index settings, or `None` while search is exact. + pub fn ann(&self) -> Option<&AnnConfig> { + self.ann.as_ref() + } + + /// Change how many cells a query scans (`None` = about 5% of `nlist`). + /// Takes effect on the next search; the index is not rebuilt. + pub fn set_ann_nprobe(&mut self, nprobe: Option<usize>) -> Result<(), EmbeddingError> { + if nprobe == Some(0) { + return Err(EmbeddingError::InvalidOption( + "ann nprobe must be at least 1".to_string(), + )); + } + match self.ann.as_mut() { + Some(config) => { + config.nprobe = nprobe; + Ok(()) + } + None => Err(EmbeddingError::InvalidOption( + "this store has no approximate index; call set_ann first".to_string(), + )), + } + } + + /// The most threads one search, batch, fold or index build may use (0, the + /// default, picks automatically: a single scan up to 8, a batch or a build + /// the whole pool; 1 never touches the pool). The answer does not depend + /// on it. + pub fn set_scan_threads(&mut self, threads: usize) { + self.scan_threads = threads; + } + + /// The configured thread cap (0 = automatic). + pub fn scan_threads(&self) -> usize { + self.scan_threads + } + + /// Use this pool instead of the process-wide one (`SearchPool::global`), + /// e.g. to give a store its own fixed number of workers. `None` goes back + /// to the shared pool. + pub fn set_search_pool(&mut self, pool: Option<Arc<SearchPool>>) { + self.pool = pool; + } + + /// The pool this store runs parallel work on (the shared one by default). + pub fn search_pool(&self) -> Arc<SearchPool> { + self.pool.clone().unwrap_or_else(SearchPool::global) + } + + pub(crate) fn par(&self) -> Par<'_> { + Par { + configured: self.scan_threads, + pool: &self.pool, + } + } + + pub(crate) fn ivf_mut(&mut self) -> &mut Option<Ivf> { + self.ivf.get_mut().unwrap_or_else(|e| e.into_inner()) + } + + fn ivf_read(&self) -> RwLockReadGuard<'_, Option<Ivf>> { + self.ivf.read().unwrap_or_else(|e| e.into_inner()) + } + + // -- write-side upkeep of the index (the caller holds `&mut self`) -------- + + /// A row was written: it joins `pending`, and a full `pending` is folded + /// now, by the write that filled it. + pub(crate) fn index_row_written(&mut self, slot: u32) { + self.hnsw_row_written(slot); + let (table, pool, configured) = (&self.table, &self.pool, self.scan_threads); + let guard = self.ivf.get_mut().unwrap_or_else(|e| e.into_inner()); + if let Some(ivf) = guard.as_mut() { + ivf.note_put(slot); + if ivf.pending.len() >= FOLD_BATCH { + ivf.fold(table, Par { configured, pool }); + } + } + } + + /// A row was deleted (and the table compacted if it was a quarter dead). + pub(crate) fn index_row_removed(&mut self, slot: u32) { + if let Some(ivf) = self.ivf_mut().as_mut() { + ivf.note_drop(slot); + } + self.hnsw_row_removed(); + if self.table.should_compact() { + self.compact_now(); + } + if let Some(cfg) = &self.ann { + if self.table.live() < cfg.min_size.max(2) { + *self.ivf_mut() = None; + *self.hnsw_mut() = None; + } + } + } + + /// Squeeze the tombstones out of the matrix now (it happens by itself once + /// a quarter of the rows are dead). Order-preserving, so results do not + /// change; a pause proportional to the store's size. + pub fn compact(&mut self) { + if self.table.dead() > 0 { + self.compact_now(); + } + } + + fn compact_now(&mut self) { + let live = self.table.compact(); + if let Some(ivf) = self.ivf_mut().as_mut() { + ivf.compact(&live); + } + self.hnsw_compact(&live); + } + + // -- the index, as the searches see it ------------------------------------ + + /// A read lock on a usable index, building or retraining first if the + /// policy says so; `None` means "answer exactly" (the store is under + /// `min_size`). Concurrent searches share the read lock; only a build + /// takes the write lock. + fn ready_index(&self, cfg: &AnnConfig) -> Option<RwLockReadGuard<'_, Option<Ivf>>> { + if self.table.live() < cfg.min_size.max(2) { + return None; + } + loop { + { + let guard = self.ivf_read(); + if guard.as_ref().is_some_and(|i| !i.stale()) { + return Some(guard); + } + } + let mut guard = self.ivf.write().unwrap_or_else(|e| e.into_inner()); + if guard.as_ref().is_none_or(|i| i.stale()) { + let nlist = cfg.nlist.unwrap_or_else(|| auto_nlist(self.table.live())); + *guard = Some(Ivf::build(&self.table, nlist, cfg.seed, self.par())); + self.counts.builds.fetch_add(1, Ordering::Relaxed); + } + } + } + + /// Top `k` from the index, or `None` to fall back to the scan. + /// + /// `scale` 1 honours `nprobe` as configured (`nprobe >= nlist` is the + /// scan). Later widening steps multiply it by 4 and give up on the index + /// once they would touch half the cells, where scanning everything costs + /// about the same. `threads` is the cap for scanning one query's cells. + #[allow(clippy::too_many_arguments)] + fn approximate_topk( + &self, + cfg: &AnnConfig, + ivf: &Ivf, + q: &QueryVec, + code: Option<u32>, + threshold: f32, + k: usize, + par: Par, + ) -> Option<Vec<Hit>> { + if !q.has_direction() { + return None; + } + let mut scale = 1usize; + loop { + let top = self.probe(cfg, ivf, q, code, threshold, k, scale, par)?; + if top.len() >= k { + self.counts.approximate.fetch_add(1, Ordering::Relaxed); + if scale > 1 { + self.counts.widened.fetch_add(1, Ordering::Relaxed); + } + return Some(top.into_sorted()); + } + scale = scale.saturating_mul(4); + } + } + + #[allow(clippy::too_many_arguments)] + fn probe( + &self, + cfg: &AnnConfig, + ivf: &Ivf, + q: &QueryVec, + code: Option<u32>, + threshold: f32, + k: usize, + scale: usize, + par: Par, + ) -> Option<TopK> { + let nprobe = effective_nprobe(cfg, ivf.nlist); + let lists = nprobe.saturating_mul(scale); + let exhausted = if scale == 1 { + lists >= ivf.nlist + } else { + lists.saturating_mul(2) >= ivf.nlist + }; + if exhausted { + return None; + } + let chosen = ivf.choose_cells(q, lists); + let cells = Cells { + lists: &ivf.lists, + pending: &ivf.pending, + }; + Some(scan_probed( + &self.table, + cells, + &chosen, + q, + code, + threshold, + k, + par, + )) + } + + /// Every row the probed cells yield (for a caller that filters further), + /// best first; `None` to fall back to the scan. + fn approximate_ranked( + &self, + cfg: &AnnConfig, + q: &QueryVec, + code: Option<u32>, + threshold: f32, + scale: usize, + ) -> Option<Vec<Hit>> { + if !q.has_direction() { + return None; + } + if cfg.mode == AnnMode::Hnsw { + return self.hnsw_ranked(cfg, q, code, threshold, scale); + } + let guard = self.ready_index(cfg)?; + let ivf = guard.as_ref()?; + let top = self.probe(cfg, ivf, q, code, threshold, usize::MAX, scale, self.par())?; + self.counts.approximate.fetch_add(1, Ordering::Relaxed); + if scale > 1 { + self.counts.widened.fetch_add(1, Ordering::Relaxed); + } + Some(top.into_sorted()) + } + + // -- exact scans ---------------------------------------------------------- + + /// The full scan, in int8 + exact rerank when that is on and applies. + fn exact_hits(&self, q: &QueryVec, code: Option<u32>, threshold: f32, k: usize) -> Vec<Hit> { + match self.quant_exact(q, code, threshold, k) { + Some(hits) => hits, + None => self.exact_hits_f32(q, code, threshold, k), + } + } + + /// The f32 full scan: every live row, spread over threads for a big store. + fn exact_hits_f32( + &self, + q: &QueryVec, + code: Option<u32>, + threshold: f32, + k: usize, + ) -> Vec<Hit> { + let table = &self.table; + let (n, dim) = (table.slots(), table.dim); + let rows = table.rows(); + let live = table.live(); + let blocks = n.div_ceil(BLOCK_ROWS); + let par = self.par(); + let threads = if par.configured == 0 { + par.workers(blocks, SCAN_AUTO_THREADS) + .min((n * dim / PAR_MIN_FLOATS).max(1)) + } else { + par.workers(blocks, SCAN_AUTO_THREADS) + }; + if threads <= 1 { + let mut top = TopK::new(k, threshold, live); + scan_range(rows, q, code, 0, n, &mut top); + return top.into_sorted(); + } + let next = AtomicUsize::new(0); + let parts = Mutex::new(Vec::with_capacity(threads)); + par.run(threads, &|_| { + let mut top = TopK::new(k, threshold, live); + loop { + let b = next.fetch_add(1, Ordering::Relaxed); + if b >= blocks { + break; + } + scan_range( + rows, + q, + code, + b * BLOCK_ROWS, + ((b + 1) * BLOCK_ROWS).min(n), + &mut top, + ); + } + parts.lock().unwrap_or_else(|e| e.into_inner()).push(top); + }); + let mut parts = parts.into_inner().unwrap_or_else(|e| e.into_inner()); + let mut merged = parts.pop().expect("at least one participant"); + for part in parts { + merged.merge(part); + } + merged.into_sorted() + } + + /// Many queries against the whole matrix at once: the work is cut into + /// (block of rows) x (tile of queries) tasks claimed from one counter, with + /// all tiles of a block adjacent so threads share the block in cache. + fn exact_batch_hits( + &self, + qs: &[QueryVec], + code: Option<u32>, + threshold: f32, + k: usize, + ) -> Vec<Vec<Hit>> { + if let Some(hits) = self.quant_exact_batch(qs, code, threshold, k) { + return hits; + } + let table = &self.table; + let n = table.slots(); + let rows = table.rows(); + let live = table.live(); + let blocks = n.div_ceil(BLOCK_ROWS); + let tiles = qs.len().div_ceil(BATCH_TILE); + let total = blocks * tiles; + let par = self.par(); + let threads = par.workers(total, usize::MAX); + let next = AtomicUsize::new(0); + let parts: Mutex<Vec<Vec<TopK>>> = Mutex::new(Vec::with_capacity(threads)); + par.run(threads, &|_| { + let mut tops: Vec<TopK> = Vec::new(); + let mut slots: Vec<u32> = Vec::with_capacity(BLOCK_ROWS); + loop { + let t = next.fetch_add(1, Ordering::Relaxed); + if t >= total { + break; + } + if tops.is_empty() { + tops = (0..qs.len()) + .map(|_| TopK::new(k, threshold, live)) + .collect(); + } + let (block, tile) = (t / tiles, t % tiles); + let (lo, hi) = (block * BLOCK_ROWS, ((block + 1) * BLOCK_ROWS).min(n)); + slots.clear(); + match code { + None => slots.extend(lo as u32..hi as u32), + Some(c) => { + slots.extend((lo..hi).filter(|&s| rows.codes[s] == c).map(|s| s as u32)) + } + } + let (q_lo, q_hi) = (tile * BATCH_TILE, ((tile + 1) * BATCH_TILE).min(qs.len())); + let tile_queries: Vec<&QueryVec> = qs[q_lo..q_hi].iter().collect(); + batch_block(rows, &tile_queries, &slots, &mut tops[q_lo..q_hi]); + } + if !tops.is_empty() { + parts.lock().unwrap_or_else(|e| e.into_inner()).push(tops); + } + }); + let mut parts = parts.into_inner().unwrap_or_else(|e| e.into_inner()); + let Some(mut merged) = parts.pop() else { + return qs.iter().map(|_| Vec::new()).collect(); + }; + for part in parts { + for (m, p) in merged.iter_mut().zip(part) { + m.merge(p); + } + } + merged.into_iter().map(TopK::into_sorted).collect() + } + + // -- results -------------------------------------------------------------- + + fn resolve_hit(&self, hit: Hit) -> Option<SimilarityResult> { + let slot = hit.slot as usize; + let meta = self.table.meta(slot)?; + Some(SimilarityResult { + node_id: self.table.node_id(slot)?, + score: hit.score, + text: meta.text.to_string(), + }) + } + + fn materialise(&self, hits: &[Hit]) -> Vec<SimilarityResult> { + hits.iter() + .filter_map(|&hit| self.resolve_hit(hit)) + .collect() + } + + /// The type filter as a code: `Ok(None)` no filter, `Err(())` a type the + /// store has never held (nothing can match). + fn type_filter(&self, node_type: Option<&str>) -> Result<Option<u32>, ()> { + match node_type { + None => Ok(None), + Some(t) => self.table.type_code(t).map(Some).ok_or(()), + } + } + + // -- public search surface ------------------------------------------------ + + /// Best `k` matches for a vector, from the index when one is configured + /// and `exact` is false, else from the full scan. + pub(crate) fn search_vector( + &self, + query_vector: &[f32], + top_k: usize, + node_type: Option<&str>, + threshold: f32, + exact: bool, + ) -> Result<Vec<SimilarityResult>, EmbeddingError> { + self.search_vector_with(query_vector, top_k, node_type, threshold, exact, None) + } + + /// `search_vector` with an optional per-call HNSW `ef_search`. + pub(crate) fn search_vector_with( + &self, + query_vector: &[f32], + top_k: usize, + node_type: Option<&str>, + threshold: f32, + exact: bool, + ef: Option<usize>, + ) -> Result<Vec<SimilarityResult>, EmbeddingError> { + self.check_query(query_vector)?; + if top_k == 0 { + return Ok(Vec::new()); + } + let live = self.table.live(); + let Ok(code) = self.type_filter(node_type) else { + return Ok(Vec::new()); + }; + if live == 0 { + return Ok(Vec::new()); + } + let q = QueryVec::new(query_vector); + let hits = self.hits_for(&q, code, threshold, top_k, exact, ef); + Ok(self.materialise(&hits)) + } + + fn hits_for( + &self, + q: &QueryVec, + code: Option<u32>, + threshold: f32, + top_k: usize, + exact: bool, + ef: Option<usize>, + ) -> Vec<Hit> { + if let Some(cfg) = &self.ann { + if exact || top_k >= self.table.live() { + self.counts.exact.fetch_add(1, Ordering::Relaxed); + } else { + let approx = if cfg.mode == AnnMode::Hnsw { + self.ready_hnsw(cfg).and_then(|guard| { + let h = guard.as_ref()?; + self.hnsw_topk(cfg, h, q, code, threshold, top_k, ef) + }) + } else { + self.ready_index(cfg).and_then(|guard| { + let ivf = guard.as_ref()?; + self.approximate_topk(cfg, ivf, q, code, threshold, top_k, self.par()) + }) + }; + match approx { + Some(hits) => return hits, + None => { + self.counts.exact.fetch_add(1, Ordering::Relaxed); + } + } + } + } + self.exact_hits(q, code, threshold, top_k) + } + + /// Search with many vector queries in one call, best `top_k` for each, + /// in query order. Each answer is **identical** (same ids, same bit-exact + /// scores, same order) to `similarity_search_vector` on that query alone; + /// the batch is only faster, because the work is spread over the thread + /// pool and, for the exact scan, queries share every pass over the matrix. + /// + /// With an approximate index configured, queries are answered from it + /// exactly as the single-query path would. The first bad query (wrong + /// dimension, empty) fails the whole call. + pub fn similarity_search_vector_batch<V: AsRef<[f32]> + Sync>( + &self, + queries: &[V], + top_k: usize, + node_type: Option<&str>, + threshold: f32, + ) -> Result<Vec<Vec<SimilarityResult>>, EmbeddingError> { + self.search_batch(queries, top_k, node_type, threshold, false) + } + + /// Like [`similarity_search_vector_batch`](Self::similarity_search_vector_batch) + /// but always the full scan, even with an approximate index configured. + pub fn similarity_search_vector_exact_batch<V: AsRef<[f32]> + Sync>( + &self, + queries: &[V], + top_k: usize, + node_type: Option<&str>, + threshold: f32, + ) -> Result<Vec<Vec<SimilarityResult>>, EmbeddingError> { + self.search_batch(queries, top_k, node_type, threshold, true) + } + + /// Search with many text queries in one call (encoded together with the + /// encoder's `encode_batch`), then as `similarity_search_vector_batch`. + pub fn similarity_search_batch( + &self, + queries: &[&str], + top_k: usize, + node_type: Option<&str>, + threshold: f32, + ) -> Result<Vec<Vec<SimilarityResult>>, EmbeddingError> { + let vectors = self.resolve_text_batch(queries)?; + self.search_batch(&vectors, top_k, node_type, threshold, false) + } + + /// Text batch, always the full scan. + pub fn similarity_search_exact_batch( + &self, + queries: &[&str], + top_k: usize, + node_type: Option<&str>, + threshold: f32, + ) -> Result<Vec<Vec<SimilarityResult>>, EmbeddingError> { + let vectors = self.resolve_text_batch(queries)?; + self.search_batch(&vectors, top_k, node_type, threshold, true) + } + + fn resolve_text_batch(&self, queries: &[&str]) -> Result<Vec<Vec<f32>>, EmbeddingError> { + if queries.is_empty() { + return Ok(Vec::new()); + } + let encoder = self.encoder.as_ref().ok_or(EmbeddingError::NoEncoder)?; + self.check_model_compat()?; + Ok(encoder.encode_batch(queries)) + } + + fn search_batch<V: AsRef<[f32]> + Sync>( + &self, + queries: &[V], + top_k: usize, + node_type: Option<&str>, + threshold: f32, + exact: bool, + ) -> Result<Vec<Vec<SimilarityResult>>, EmbeddingError> { + for q in queries { + self.check_query(q.as_ref())?; + } + let empty = || queries.iter().map(|_| Vec::new()).collect::<Vec<_>>(); + if top_k == 0 || queries.is_empty() { + return Ok(empty()); + } + let live = self.table.live(); + let Ok(code) = self.type_filter(node_type) else { + return Ok(empty()); + }; + if live == 0 { + return Ok(empty()); + } + if top_k >= live { + // "everything" answers hold every row: no shared heaps to bound + return queries + .iter() + .map(|q| self.search_vector(q.as_ref(), top_k, node_type, threshold, exact)) + .collect(); + } + let mut out: Vec<Vec<SimilarityResult>> = Vec::with_capacity(queries.len()); + for chunk in queries.chunks(BATCH_CHUNK) { + let qs: Vec<QueryVec> = chunk.iter().map(|q| QueryVec::new(q.as_ref())).collect(); + let hits = self.batch_hits(&qs, code, threshold, top_k, exact); + out.extend(hits.iter().map(|h| self.materialise(h))); + } + Ok(out) + } + + fn batch_hits( + &self, + qs: &[QueryVec], + code: Option<u32>, + threshold: f32, + top_k: usize, + exact: bool, + ) -> Vec<Vec<Hit>> { + let mut answers: Vec<Option<Vec<Hit>>> = (0..qs.len()).map(|_| None).collect(); + if let Some(cfg) = &self.ann { + if exact { + self.counts + .exact + .fetch_add(qs.len() as u64, Ordering::Relaxed); + } else if cfg.mode == AnnMode::Hnsw { + if let Some(guard) = self.ready_hnsw(cfg) { + if let Some(h) = guard.as_ref() { + self.hnsw_batch(cfg, h, qs, code, threshold, top_k, &mut answers); + } + let fallbacks = answers.iter().filter(|a| a.is_none()).count() as u64; + self.counts.exact.fetch_add(fallbacks, Ordering::Relaxed); + } else { + self.counts + .exact + .fetch_add(qs.len() as u64, Ordering::Relaxed); + } + } else if let Some(guard) = self.ready_index(cfg) { + if let Some(ivf) = guard.as_ref() { + if !self.quant_approximate_batch( + cfg, + ivf, + qs, + code, + threshold, + top_k, + &mut answers, + ) { + self.approximate_batch(cfg, ivf, qs, code, threshold, top_k, &mut answers); + } + } + let fallbacks = answers.iter().filter(|a| a.is_none()).count() as u64; + self.counts.exact.fetch_add(fallbacks, Ordering::Relaxed); + } else { + self.counts + .exact + .fetch_add(qs.len() as u64, Ordering::Relaxed); + } + } + // whatever the index did not answer goes through the batched scan + let todo: Vec<usize> = (0..qs.len()).filter(|&i| answers[i].is_none()).collect(); + if !todo.is_empty() { + let rest: Vec<QueryVec> = todo + .iter() + .map(|&i| QueryVec { + v: qs[i].v, + norm: qs[i].norm, + }) + .collect(); + let exact_hits = self.exact_batch_hits(&rest, code, threshold, top_k); + for (i, hits) in todo.into_iter().zip(exact_hits) { + answers[i] = Some(hits); + } + } + answers.into_iter().map(|a| a.unwrap_or_default()).collect() + } + + /// Answer a batch from the index, **cell-major**: work out which cells each + /// query probes, invert that into "the queries that probe cell c", and score + /// each probed cell once against all of its queries (the same 4-query x + /// 2-row kernel the exact batch uses), instead of once per query. The rows + /// scored for a query are exactly the rows the single-query probe scores + /// (its cells plus the pending rows), so the answer is identical. A query + /// that comes up short (a selective filter) is re-run on the single-query + /// path, which widens. + #[allow(clippy::too_many_arguments)] + fn approximate_batch( + &self, + cfg: &AnnConfig, + ivf: &Ivf, + qs: &[QueryVec], + code: Option<u32>, + threshold: f32, + top_k: usize, + answers: &mut [Option<Vec<Hit>>], + ) { + let nprobe = effective_nprobe(cfg, ivf.nlist); + if nprobe >= ivf.nlist { + return; // probing everything is the exact scan + } + let rows = self.table.rows(); + let live = self.table.live(); + let par = self.par(); + + // 1. the cells each query probes + let mut chosen: Vec<Option<Vec<u32>>> = (0..qs.len()).map(|_| None).collect(); + { + let next = AtomicUsize::new(0); + let done: Mutex<Vec<(usize, Vec<u32>)>> = Mutex::new(Vec::new()); + par.run(par.workers(qs.len(), usize::MAX), &|_| { + let mut local = Vec::new(); + loop { + let i = next.fetch_add(1, Ordering::Relaxed); + if i >= qs.len() { + break; + } + if qs[i].has_direction() { + local.push((i, ivf.choose_cells(&qs[i], nprobe))); + } + } + done.lock().unwrap_or_else(|e| e.into_inner()).extend(local); + }); + for (i, cells) in done.into_inner().unwrap_or_else(|e| e.into_inner()) { + chosen[i] = Some(cells); + } + } + + // 2. invert: the queries of each cell; the extra last entry is the + // pending rows, which every query scores + let mut by_cell: Vec<Vec<u32>> = vec![Vec::new(); ivf.nlist + 1]; + for (i, cells) in chosen.iter().enumerate() { + if let Some(cells) = cells { + for &c in cells { + by_cell[c as usize].push(i as u32); + } + by_cell[ivf.nlist].push(i as u32); + } + } + let members_of = |c: usize| -> &[u32] { + if c == ivf.nlist { + &ivf.pending + } else { + &ivf.lists[c] + } + }; + let tasks: Vec<usize> = (0..=ivf.nlist) + .filter(|&c| !by_cell[c].is_empty() && !members_of(c).is_empty()) + .collect(); + + // 3. score every probed cell once against all of its queries + let next = AtomicUsize::new(0); + let parts: Mutex<Vec<Vec<TopK>>> = Mutex::new(Vec::new()); + par.run(par.workers(tasks.len(), usize::MAX), &|_| { + let mut tops: Vec<TopK> = Vec::new(); + let mut slots: Vec<u32> = Vec::new(); + loop { + let t = next.fetch_add(1, Ordering::Relaxed); + if t >= tasks.len() { + break; + } + if tops.is_empty() { + tops = (0..qs.len()) + .map(|_| TopK::new(top_k, threshold, live)) + .collect(); + } + let c = tasks[t]; + slots.clear(); + slots.extend( + members_of(c) + .iter() + .copied() + .filter(|&s| code.is_none_or(|code| rows.codes[s as usize] == code)), + ); + for group in by_cell[c].chunks(4) { + let refs: Vec<&QueryVec> = group.iter().map(|&i| &qs[i as usize]).collect(); + let mut taken: Vec<TopK> = group + .iter() + .map(|&i| std::mem::replace(&mut tops[i as usize], TopK::new(0, 0.0, 0))) + .collect(); + batch_block(rows, &refs, &slots, &mut taken); + for (&i, top) in group.iter().zip(taken) { + tops[i as usize] = top; + } + } + } + if !tops.is_empty() { + parts.lock().unwrap_or_else(|e| e.into_inner()).push(tops); + } + }); + let mut parts = parts.into_inner().unwrap_or_else(|e| e.into_inner()); + let mut merged: Vec<TopK> = match parts.pop() { + Some(first) => first, + None => (0..qs.len()) + .map(|_| TopK::new(top_k, threshold, live)) + .collect(), + }; + for part in parts { + for (a, b) in merged.iter_mut().zip(part) { + a.merge(b); + } + } + + // 4. answers; the short ones go back through the single-query path + let mut short = Vec::new(); + for (i, top) in merged.into_iter().enumerate() { + if chosen[i].is_none() { + continue; + } + if top.len() >= top_k { + self.counts.approximate.fetch_add(1, Ordering::Relaxed); + answers[i] = Some(top.into_sorted()); + } else { + short.push(i); + } + } + let one_thread = Par { + configured: 1, + pool: &self.pool, + }; + let next = AtomicUsize::new(0); + let redone: Mutex<Vec<(usize, Option<Vec<Hit>>)>> = Mutex::new(Vec::new()); + par.run(par.workers(short.len(), usize::MAX), &|_| { + let mut local = Vec::new(); + loop { + let j = next.fetch_add(1, Ordering::Relaxed); + if j >= short.len() { + break; + } + let i = short[j]; + local.push(( + i, + self.approximate_topk(cfg, ivf, &qs[i], code, threshold, top_k, one_thread), + )); + } + redone + .lock() + .unwrap_or_else(|e| e.into_inner()) + .extend(local); + }); + for (i, hits) in redone.into_inner().unwrap_or_else(|e| e.into_inner()) { + answers[i] = hits; + } + } + + /// Every match above `threshold`, best first, as a lazy iterator. With + /// `scale: Some(s)` and an index configured, only the rows of the probed + /// cells (`nprobe * s` of them); the flag says whether that was so (false + /// means the list is exact and there is nothing left to widen to). Used by + /// the graph's filtered search, which widens until enough matches pass its + /// filter. + pub(crate) fn ranked_hits( + &self, + query_vector: &[f32], + node_type: Option<&str>, + threshold: f32, + scale: Option<usize>, + ) -> Result<(RankedHits<'_>, bool), EmbeddingError> { + self.check_query(query_vector)?; + let empty = || RankedHits { + store: self, + hits: Vec::new().into_iter(), + }; + let Ok(code) = self.type_filter(node_type) else { + return Ok((empty(), false)); + }; + if self.table.live() == 0 { + return Ok((empty(), false)); + } + let q = QueryVec::new(query_vector); + let mut approximate = None; + if let (Some(cfg), Some(scale)) = (&self.ann, scale) { + approximate = self.approximate_ranked(cfg, &q, code, threshold, scale); + if approximate.is_none() { + self.counts.exact.fetch_add(1, Ordering::Relaxed); + } + } + let was_approximate = approximate.is_some(); + let hits = approximate.unwrap_or_else(|| self.exact_hits(&q, code, threshold, usize::MAX)); + Ok(( + RankedHits { + store: self, + hits: hits.into_iter(), + }, + was_approximate, + )) + } + + /// Cosine score of every stored vector, keyed by `type:id`. + pub(crate) fn scan_scores( + &self, + query_vector: &[f32], + ) -> Result<HashMap<String, f32>, EmbeddingError> { + self.check_query(query_vector)?; + let q = QueryVec::new(query_vector); + let table = &self.table; + let mut scored = Vec::with_capacity(table.live()); + score_all(table.rows(), &q, table.slots(), &mut scored); + let mut scores = HashMap::with_capacity(scored.len()); + for (slot, score) in scored { + if let Some(meta) = table.meta(slot as usize) { + scores.insert(meta.key.to_string(), score); + } + } + Ok(scores) + } + + /// Like `similarity_search_vector`, but always the full scan, even with an + /// approximate index configured. + pub fn similarity_search_vector_exact( + &self, + query_vector: &[f32], + top_k: usize, + node_type: Option<&str>, + threshold: f32, + ) -> Result<Vec<SimilarityResult>, EmbeddingError> { + self.search_vector(query_vector, top_k, node_type, threshold, true) + } + + /// Like `similarity_search`, but always the full scan. + pub fn similarity_search_exact( + &self, + query: &str, + top_k: usize, + node_type: Option<&str>, + threshold: f32, + ) -> Result<Vec<SimilarityResult>, EmbeddingError> { + let query_vector = self.resolve_query(Query::Text(query))?; + self.similarity_search_vector_exact(&query_vector, top_k, node_type, threshold) + } + + /// Build the approximate index now instead of on the first search. + /// + /// Returns [`ann_stats`](Self::ann_stats). `force` retrains even if an + /// index exists. Does nothing (and says so in the stats) while the store + /// is smaller than `min_size`. Call this off the hot path after loading a + /// big store: the first search that needs the index otherwise pays for it. + /// + /// # Errors + /// + /// `InvalidOption` when the store has no [`AnnConfig`]. + pub fn build_ann_index(&self, force: bool) -> Result<AnnStats, EmbeddingError> { + let Some(cfg) = &self.ann else { + return Err(EmbeddingError::InvalidOption( + "this store has no approximate index; call set_ann first".to_string(), + )); + }; + if cfg.mode == AnnMode::Hnsw { + if force { + *self.hnsw.write().unwrap_or_else(|e| e.into_inner()) = None; + } + let _ = self.ready_hnsw(cfg); + return Ok(self.ann_stats()); + } + if force { + *self.ivf.write().unwrap_or_else(|e| e.into_inner()) = None; + } + if self.ready_index(cfg).is_some() { + let mut guard = self.ivf.write().unwrap_or_else(|e| e.into_inner()); + if let Some(ivf) = guard.as_mut() { + ivf.fold(&self.table, self.par()); + } + } + Ok(self.ann_stats()) + } + + /// What the approximate index looks like right now. Never builds anything. + pub fn ann_stats(&self) -> AnnStats { + if self.hnsw_is_mode() { + return self.hnsw_ann_stats(); + } + let guard = self.ivf_read(); + let ivf = guard.as_ref(); + AnnStats { + enabled: self.ann.is_some(), + active: ivf.is_some(), + nlist: ivf.map(|i| i.nlist), + nprobe: match (ivf, &self.ann) { + (Some(i), Some(cfg)) => Some(effective_nprobe(cfg, i.nlist)), + (None, Some(cfg)) => cfg.nprobe, + _ => None, + }, + trained_n: ivf.map(|i| i.trained_n), + pending: ivf.map_or(0, |i| i.pending.len()), + churn: ivf.map_or(0, |i| i.churn), + build_seconds: ivf.map(|i| i.build_seconds), + index_bytes: ivf.map_or(0, Ivf::nbytes), + min_size: self.ann.as_ref().map(|c| c.min_size), + searches: self.counts.snapshot(), + folds: ivf.map_or(0, |i| i.folds), + fold_seconds: ivf.map_or(0.0, |i| i.fold_seconds), + mode: AnnMode::Ivf, + ef_search: None, + max_level: None, + graph_nodes: 0, + graph_dead: 0, + inserts: 0, + insert_seconds: 0.0, + } + } + + /// Check every internal invariant (matrix shape, key index, per-model + /// counts, the approximate index against the matrix) and describe the first + /// one that fails. For tests and debugging; it walks the whole store. + pub fn verify_invariants(&self) -> Result<(), String> { + self.table.check()?; + let by_model: usize = self.model_counts.values().sum(); + if by_model != self.table.live() { + return Err(format!( + "model counts add up to {by_model}, {} live rows", + self.table.live() + )); + } + let mut seen: HashMap<&str, usize> = HashMap::new(); + for slot in self.table.live_slots() { + if let Some(meta) = self.table.meta(slot) { + *seen.entry(&meta.model).or_default() += 1; + } + } + for (model, count) in &self.model_counts { + if seen.get(model.as_str()) != Some(count) { + return Err(format!( + "model {model:?} counted {count}, rows say {:?}", + seen.get(model.as_str()) + )); + } + } + if let Some(ivf) = self.ivf_read().as_ref() { + ivf.check(&self.table, FOLD_BATCH)?; + } + if let Some(h) = self.hnsw_read().as_ref() { + h.check(&self.table)?; + } + Ok(()) + } + + /// Where the store's memory goes. The vectors are held once; compare + /// `matrix_bytes` with `live_vector_bytes`. + pub fn memory_stats(&self) -> MemoryStats { + let t = self.table.bytes(); + MemoryStats { + rows: self.table.slots(), + live: self.table.live(), + dead: self.table.dead(), + chunks: self.table.chunk_count(), + matrix_bytes: t.matrix, + live_vector_bytes: self.table.live() * self.table.dim * 4, + metadata_bytes: t.per_row + t.keys_and_text + t.key_index, + index_bytes: self.ivf_read().as_ref().map_or(0, Ivf::nbytes) + + self.hnsw_read().as_ref().map_or(0, |h| h.nbytes()), + } + } + + /// Recall@k of the approximate index against the exact scan. + /// + /// Runs every query both ways and compares the *sets* of top-k nodes. + /// Queries are texts or vectors. Use real queries drawn from your own + /// workload: recall depends on how the data clusters, and a score measured + /// on unrelated queries or data does not transfer. Raise `nprobe` until + /// the recall you need holds. + /// + /// Without an index in use (none configured, a store under `min_size`, or + /// `nprobe >= nlist`) both answers are the exact scan, recall is 1.0 and + /// `approximate` is false. + pub fn recall_check<'q, I, Q>( + &self, + queries: I, + k: usize, + node_type: Option<&str>, + threshold: f32, + ) -> Result<RecallReport, EmbeddingError> + where + I: IntoIterator<Item = Q>, + Q: Into<Query<'q>>, + { + if k == 0 { + return Err(EmbeddingError::InvalidOption( + "k must be positive".to_string(), + )); + } + let mut recalls: Vec<f64> = Vec::new(); + let mut ann_times: Vec<f64> = Vec::new(); + let mut exact_times: Vec<f64> = Vec::new(); + let mut used = false; + let mut count = 0usize; + for query in queries { + count += 1; + let vector = self.resolve_query(query.into())?; + let before = self.ann_stats().searches.approximate; + let started = Instant::now(); + let found = self.search_vector(&vector, k, node_type, threshold, false)?; + ann_times.push(started.elapsed().as_secs_f64()); + used |= self.ann_stats().searches.approximate > before; + let started = Instant::now(); + let truth = self.search_vector(&vector, k, node_type, threshold, true)?; + exact_times.push(started.elapsed().as_secs_f64()); + if !truth.is_empty() { + let wanted: std::collections::HashSet<&crate::NodeId> = + truth.iter().map(|r| &r.node_id).collect(); + let hit = found.iter().filter(|r| wanted.contains(&r.node_id)).count(); + recalls.push(hit as f64 / wanted.len() as f64); + } + } + let stats = self.ann_stats(); + let median = |mut v: Vec<f64>| -> f64 { + if v.is_empty() { + return 0.0; + } + v.sort_by(f64::total_cmp); + let mid = v.len() / 2; + if v.len() % 2 == 1 { + v[mid] + } else { + 0.5 * (v[mid - 1] + v[mid]) + } + }; + Ok(RecallReport { + recall: if recalls.is_empty() { + 1.0 + } else { + recalls.iter().sum::<f64>() / recalls.len() as f64 + }, + min_recall: recalls.iter().copied().fold(1.0, f64::min), + queries: count, + k, + nlist: stats.nlist.unwrap_or(0), + nprobe: stats.nprobe.unwrap_or(0), + approximate: used, + ef_search: stats.ef_search.unwrap_or(0), + ann_ms: 1000.0 * median(ann_times), + exact_ms: 1000.0 * median(exact_times), + }) + } +} + +/// Score the probed cells (and the pending rows). The rows of a cell are +/// scattered through the matrix, so one thread is limited by how many cache +/// lines it can have in flight, not by bandwidth: a large probe is shared out +/// a few cells at a time. The answer does not depend on the thread count. +#[allow(clippy::too_many_arguments)] +fn scan_probed( + table: &VectorTable, + cells: Cells, + chosen: &[u32], + q: &QueryVec, + code: Option<u32>, + threshold: f32, + k: usize, + par: Par, +) -> TopK { + const CELLS_PER_TASK: usize = 4; + if let Some(top) = quant::scan_probed(table, cells, chosen, q, code, threshold, k, par) { + return top; + } + let rows = table.rows(); + let live = table.live(); + let candidates: usize = chosen + .iter() + .map(|&c| cells.lists[c as usize].len()) + .sum::<usize>() + + cells.pending.len(); + let tasks = chosen.len().div_ceil(CELLS_PER_TASK); + let threads = if par.configured == 0 { + par.workers(tasks, SCAN_AUTO_THREADS) + .min((candidates * table.dim / PAR_MIN_CELL_FLOATS).max(1)) + } else { + par.workers(tasks, SCAN_AUTO_THREADS) + }; + let mut top = TopK::new(k, threshold, live); + if threads <= 1 { + scan_cells(rows, cells, chosen, true, q, code, &mut top); + return top; + } + let next = AtomicUsize::new(0); + let parts = Mutex::new(Vec::with_capacity(threads)); + par.run(threads, &|_| { + let mut local = TopK::new(k, threshold, live); + loop { + let t = next.fetch_add(1, Ordering::Relaxed); + if t >= tasks { + break; + } + let lo = t * CELLS_PER_TASK; + let hi = (lo + CELLS_PER_TASK).min(chosen.len()); + // the pending rows ride with the first task only + scan_cells(rows, cells, &chosen[lo..hi], t == 0, q, code, &mut local); + } + parts.lock().unwrap_or_else(|e| e.into_inner()).push(local); + }); + for part in parts.into_inner().unwrap_or_else(|e| e.into_inner()) { + top.merge(part); + } + top +} + +/// Counters, the pool and the lazily built index: the search-side state an +/// [`EmbeddingStore`] carries besides its table. +pub(crate) type IvfSlot = RwLock<Option<Ivf>>; + +#[allow(dead_code, clippy::manual_is_multiple_of)] +const _: () = assert!(CHUNK_ROWS % BLOCK_ROWS == 0 && CHUNK_ROWS % NEAR_BLOCK == 0); + +// Opt-in int8 first-pass scan with an exact rerank (see `quant.rs`). Declared +// after `dispatch!` so the child can use that macro. +#[path = "quant.rs"] +pub(crate) mod quant; + +#[cfg(test)] +mod tests { + use super::*; + + fn naive_dot(a: &[f32], b: &[f32]) -> f64 { + a.iter().zip(b).map(|(x, y)| *x as f64 * *y as f64).sum() + } + + fn pseudo(n: usize, seed: u64) -> Vec<f32> { + let mut rng = Rng(seed); + (0..n) + .map(|_| (rng.next_u64() >> 40) as f32 / (1u64 << 24) as f32 - 0.5) + .collect() + } + + #[test] + fn the_dot_kernel_matches_a_plain_sum_at_awkward_lengths() { + for len in [1usize, 3, 7, 8, 31, 32, 33, 100, 384, 1000] { + let a = pseudo(len, 1); + let b = pseudo(len, 2); + let want = naive_dot(&a, &b); + assert!((dot_f32(&a, &b) as f64 - want).abs() < 1e-4, "len {len}"); + assert!( + (dot_impl::<false>(&a, &b) as f64 - want).abs() < 1e-4, + "len {len}" + ); + } + } + + #[test] + #[allow(clippy::needless_range_loop)] + fn every_kernel_shape_gives_bit_identical_dot_products() { + // The promise behind "a batch equals the single queries exactly": a + // (query, row) pair is summed in the same order whatever block shape + // computes it. + fn check<const F: bool>(dim: usize) { + let q: Vec<Vec<f32>> = (0..4).map(|i| pseudo(dim, 10 + i)).collect(); + let r: Vec<Vec<f32>> = (0..4).map(|i| pseudo(dim, 20 + i)).collect(); + let one = |a: usize, j: usize| block::<1, 1, F>([&q[a]], [&r[j]])[0][0]; + let b42 = block::<4, 2, F>([&q[0], &q[1], &q[2], &q[3]], [&r[0], &r[1]]); + let b14 = block::<1, 4, F>([&q[2]], [&r[0], &r[1], &r[2], &r[3]]); + let b24 = block::<2, 4, F>([&q[1], &q[3]], [&r[0], &r[1], &r[2], &r[3]]); + let b41 = block::<4, 1, F>([&q[0], &q[1], &q[2], &q[3]], [&r[3]]); + for a in 0..4 { + for j in 0..2 { + assert_eq!(b42[a][j].to_bits(), one(a, j).to_bits(), "4x2 dim {dim}"); + } + assert_eq!(b41[a][0].to_bits(), one(a, 3).to_bits(), "4x1 dim {dim}"); + } + for j in 0..4 { + assert_eq!(b14[0][j].to_bits(), one(2, j).to_bits(), "1x4 dim {dim}"); + assert_eq!(b24[0][j].to_bits(), one(1, j).to_bits(), "2x4 dim {dim}"); + assert_eq!(b24[1][j].to_bits(), one(3, j).to_bits(), "2x4 dim {dim}"); + } + } + for dim in [1usize, 5, 8, 9, 32, 100, 384] { + check::<false>(dim); + // the fused-multiply-add instantiation is only valid where the CPU + // has it; the dispatching wrapper checks + let wrapper_ok = { + let a = pseudo(dim, 1); + let b = pseudo(dim, 2); + let fused = dot_f32(&a, &b); + let plain = dot_impl::<false>(&a, &b); + (fused - plain).abs() < 1e-4 + }; + assert!(wrapper_ok); + } + } + + #[test] + fn the_assignment_kernel_matches_a_plain_argmax() { + for (dim, nlist, m) in [ + (5usize, 1usize, 3usize), + (9, 7, 130), + (32, 10, 65), + (384, 13, 70), + ] { + let pts = pseudo(m * dim, 3); + let cents = pseudo(nlist * dim, 4); + let mut got = vec![0i32; m]; + nearest_block(&pts, dim, ¢s, nlist, &mut got); + // the answer for a point does not depend on its neighbours + let mut alone = vec![0i32; m]; + for p in 0..m { + nearest_block( + &pts[p * dim..(p + 1) * dim], + dim, + ¢s, + nlist, + &mut alone[p..p + 1], + ); + } + assert_eq!(got, alone); + let pool = None; + let par = Par { + configured: 1, + pool: &pool, + }; + let mut serial = vec![0i32; m]; + nearest_parallel( + &|lo, len| &pts[lo * dim..(lo + len) * dim], + dim, + ¢s, + nlist, + &mut serial, + par, + ); + assert_eq!(got, serial); + for p in 0..m { + let x = &pts[p * dim..(p + 1) * dim]; + let scores: Vec<f64> = (0..nlist) + .map(|c| naive_dot(x, ¢s[c * dim..(c + 1) * dim])) + .collect(); + let best = scores.iter().cloned().fold(f64::NEG_INFINITY, f64::max); + // an f32 near-tie may legitimately pick the runner-up + assert!( + scores[got[p] as usize] >= best - 1e-4, + "dim {dim} nlist {nlist} point {p}" + ); + } + } + } + + #[test] + fn top_k_keeps_the_best_and_breaks_ties_by_arrival() { + let mut top = TopK::new(3, 0.0, 100); + for (slot, score) in [0.5f32, 0.9, 0.9, 0.1, 0.9, 0.7, f32::NAN, -1.0, 0.9] + .into_iter() + .enumerate() + { + top.offer(score, slot as u32); + } + let got: Vec<(u32, f32)> = top + .into_sorted() + .iter() + .map(|h| (h.slot, h.score)) + .collect(); + assert_eq!(got, vec![(1, 0.9), (2, 0.9), (4, 0.9)]); + + // asked for more than can exist: collects everything above the bar + let mut all = TopK::new(usize::MAX, 0.3, 10); + for (slot, score) in [0.2f32, 0.4, 0.4, 0.9].into_iter().enumerate() { + all.offer(score, slot as u32); + } + let got: Vec<u32> = all.into_sorted().iter().map(|h| h.slot).collect(); + assert_eq!(got, vec![3, 1, 2]); + } + + #[test] + fn merging_partial_results_equals_one_pass() { + let scores = pseudo(500, 9); + let mut whole = TopK::new(17, -1.0, 500); + let (mut a, mut b) = (TopK::new(17, -1.0, 500), TopK::new(17, -1.0, 500)); + for (slot, &s) in scores.iter().enumerate() { + // quantise so there are plenty of ties + let s = (s * 20.0).round() / 20.0; + whole.offer(s, slot as u32); + if slot % 2 == 0 { + a.offer(s, slot as u32); + } else { + b.offer(s, slot as u32); + } + } + a.merge(b); + let w: Vec<(u32, f32)> = whole + .into_sorted() + .iter() + .map(|h| (h.slot, h.score)) + .collect(); + let m: Vec<(u32, f32)> = a.into_sorted().iter().map(|h| (h.slot, h.score)).collect(); + assert_eq!(w, m); + } + + #[test] + fn rounding_matches_pythons_round() { + assert_eq!(round_half_even(0.5), 0.0); + assert_eq!(round_half_even(1.5), 2.0); + assert_eq!(round_half_even(2.5), 2.0); + assert_eq!(round_half_even(2.4), 2.0); + assert_eq!(round_half_even(2.6), 3.0); + assert_eq!(auto_nlist(100), 40); + assert_eq!(auto_nlist(4), 16); + assert_eq!(auto_nlist(1_000_000), 4000); + assert_eq!(auto_nlist(usize::MAX / 4), 65_536); + assert_eq!(effective_nprobe(&AnnConfig::default(), 4000), 200); + assert_eq!(effective_nprobe(&AnnConfig::default(), 10), 1); + assert_eq!(effective_nprobe(&AnnConfig::default(), 50), 2); + assert_eq!( + effective_nprobe( + &AnnConfig { + nprobe: Some(99), + ..AnnConfig::default() + }, + 64 + ), + 64 + ); + } +} diff --git a/isongraph-vector-rs/src/hnsw.rs b/isongraph-vector-rs/src/hnsw.rs new file mode 100644 index 0000000..89475d1 --- /dev/null +++ b/isongraph-vector-rs/src/hnsw.rs @@ -0,0 +1,1616 @@ +//! The opt-in HNSW approximate index (`AnnMode::Hnsw`), beside the IVF-flat one +//! in `ann.rs`. `DESIGN.md` section 10 is the specification; this file is its +//! Rust realisation. +//! +//! # What it is +//! +//! A hierarchical navigable small-world graph over the store's matrix. The +//! graph holds **slot numbers and links only**: every score is computed from +//! the one stored copy of each vector with the crate's canonical kernels, so a +//! hit's score is bit-identical to the exact scan's. Layer 0 holds every node +//! with up to `2 * M` links; a node also lives in layers `1..=level` +//! (`P(level >= l) = M^-l`, drawn from a seeded integer generator) with up to +//! `M` links per layer. A query descends greedily from the entry point to layer +//! 0, then runs a best-first search with `ef` candidates there. +//! +//! # Maintenance +//! +//! * **Insert** happens inside the write that stored the row (no pending list): +//! search for neighbours, choose `M` with the diversity heuristic, link both +//! ways, re-prune any neighbour that overflowed. A few hundred microseconds to +//! a few milliseconds at a million rows. +//! * **Update** (same slot, new vector): the node keeps its level; its own links +//! are recomputed from the new vector and the new neighbours link back. Links +//! that other nodes held to the old position are left (they are only routing +//! hints; scores always come from the current vector) and are trimmed the next +//! time those lists overflow. Updates count as churn. +//! * **Delete** tombstones: the table's NaN norm stops the row from ever being +//! returned, but the graph keeps the node so routes through it still work (its +//! floats stay in the matrix until compaction). A tombstone is dropped from a +//! list whenever that list is re-pruned, and wholesale at compaction. +//! * **Compaction** (the table's, at 25% dead) drops tombstones from the graph, +//! renumbers every link, and refills a node's lost links from the links of the +//! tombstones it pointed to ("bridging"), without computing a distance. +//! * **Rebuild**: once updates plus deletes exceed the number of nodes, the next +//! search that needs the index rebuilds it. +//! +//! # Determinism and parallel build +//! +//! A build inserts in slot order in batches that grow with the graph +//! (`size / 16`, at most 512). Within a batch every member is planned against +//! the same frozen graph in parallel, and the plans are applied in member order, +//! so the result depends on the seed and the insertion order, never on the +//! thread count. Members of one batch see each other by brute force. +//! +//! The index is never persisted; a loaded store builds it on its first search. + +use crate::ann::{ + cosine, dot_f32, score_slots, AnnConfig, AnnMode, AnnStats, Hit, Par, QueryVec, Rng, +}; +use crate::table::{Rows, VectorTable, CHUNK_MASK, CHUNK_ROWS, CHUNK_SHIFT}; +use crate::{EmbeddingError, EmbeddingStore}; +use std::cell::RefCell; +use std::cmp::{Ordering as Cmp, Reverse}; +use std::collections::BinaryHeap; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::{Mutex, RwLock, RwLockReadGuard}; +use std::time::Instant; + +/// Links per node on layers above 0 (layer 0 allows twice this). +pub const HNSW_M: usize = 16; +/// Candidates kept while inserting. +pub const HNSW_EF_CONSTRUCTION: usize = 200; + +const ABSENT: u8 = u8::MAX; +const NONE: u32 = u32::MAX; +const MAX_LEVEL: usize = 24; +/// Largest batch the parallel build plans against one frozen graph. +const BUILD_BATCH: usize = 512; +/// A batch is at most `size / BUILD_RATIO` nodes, so members are few next to +/// the graph they are planned against. +const BUILD_RATIO: usize = 16; +/// Updates plus deletes beyond this multiple of the node count call for a +/// rebuild. +const REBUILD_FRACTION: f64 = 1.0; +/// Most links a layer can hold: `2 * M` with `M` at most 64. +const MAX_CAP: usize = 128; + +/// Settings for the HNSW index; see [`AnnConfig::hnsw`]. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct HnswConfig { + /// Links per node on layers above 0 (layer 0 holds up to `2 * m`). More is + /// better recall and more memory. 2 to 64; default 16. + pub m: usize, + /// Candidates kept while linking a new row. More is a better graph and a + /// slower build and write. Default 200 (never less than `m`). + pub ef_construction: usize, + /// Candidates kept per query (never less than the `top_k` asked for). More + /// is higher recall and slower. `None` picks by store size, aiming at about + /// 0.95 recall@10 on 384-d text embeddings: 64 under 10,000 rows, 96 under + /// 200,000, 128 under 600,000, 192 above (measured 0.95 at 100,000 rows + /// with 96 and 1,000,000 with 192; measure yours with `recall_check`). + pub ef_search: Option<usize>, +} + +impl Default for HnswConfig { + fn default() -> Self { + Self { + m: HNSW_M, + ef_construction: HNSW_EF_CONSTRUCTION, + ef_search: None, + } + } +} + +impl HnswConfig { + pub(crate) fn validate(&self) -> Result<(), EmbeddingError> { + if !(2..=MAX_CAP / 2).contains(&self.m) { + return Err(EmbeddingError::InvalidOption(format!( + "hnsw m must be between 2 and {}", + MAX_CAP / 2 + ))); + } + if self.ef_construction == 0 { + return Err(EmbeddingError::InvalidOption( + "hnsw ef_construction must be at least 1".to_string(), + )); + } + if self.ef_search == Some(0) { + return Err(EmbeddingError::InvalidOption( + "hnsw ef_search must be at least 1".to_string(), + )); + } + Ok(()) + } +} + +// ============================================================================= +// Scores and small helpers +// ============================================================================= + +/// A scored node. The order is "better is greater": higher score, then lower +/// slot (the exact scan's tie-break). +#[derive(Clone, Copy, Debug)] +struct Sc { + score: f32, + slot: u32, +} + +impl PartialEq for Sc { + fn eq(&self, other: &Self) -> bool { + self.cmp(other) == Cmp::Equal + } +} +impl Eq for Sc {} +impl PartialOrd for Sc { + fn partial_cmp(&self, other: &Self) -> Option<Cmp> { + Some(self.cmp(other)) + } +} +impl Ord for Sc { + fn cmp(&self, other: &Self) -> Cmp { + self.score + .total_cmp(&other.score) + .then_with(|| other.slot.cmp(&self.slot)) + } +} + +/// Euclidean norm of a row, for tombstones too (their table norm is NaN). +fn node_norm(rows: &Rows, slot: usize) -> f32 { + let n = rows.norms[slot]; + if n.is_nan() { + let row = rows.row(slot); + dot_f32(row, row).sqrt() + } else { + n + } +} + +fn node_query<'a>(rows: &Rows<'a>, slot: usize) -> QueryVec<'a> { + QueryVec { + v: rows.row(slot), + norm: node_norm(rows, slot), + } +} + +/// Cosine of the query with each of `slots`. Live rows score exactly as the +/// scan scores them; tombstones, which the table marks NaN, are scored from +/// their still-present floats so the search can route through them; anything +/// that stays NaN (a corrupt row) ranks last. +fn score_into(rows: Rows, q: &QueryVec, slots: &[u32], out: &mut [f32]) { + score_slots(rows, q, slots, out); + for (i, &s) in slots.iter().enumerate() { + if out[i].is_nan() { + out[i] = if rows.norms[s as usize].is_nan() { + let row = rows.row(s as usize); + let sc = cosine(dot_f32(q.v, row), dot_f32(row, row).sqrt(), q.norm); + if sc.is_nan() { + f32::NEG_INFINITY + } else { + sc + } + } else { + f32::NEG_INFINITY + }; + } + } +} + +fn score_one(rows: Rows, q: &QueryVec, slot: u32) -> Sc { + let mut out = [0.0f32]; + score_into(rows, q, &[slot], &mut out); + Sc { + score: out[0], + slot, + } +} + +#[inline(always)] +fn prefetch(row: &[f32]) { + #[cfg(target_arch = "x86_64")] + { + use std::arch::x86_64::{_mm_prefetch, _MM_HINT_T0}; + let p = row.as_ptr() as *const i8; + // SAFETY: a prefetch is a hint; it never faults, whatever the address. + #[allow(unused_unsafe)] + unsafe { + _mm_prefetch(p, _MM_HINT_T0); + _mm_prefetch(p.wrapping_add(64), _MM_HINT_T0); + } + } + #[cfg(not(target_arch = "x86_64"))] + let _ = row; +} + +/// Which nodes a layer-0 search may return (the others are only routed +/// through). +#[derive(Clone, Copy)] +struct Admit { + code: Option<u32>, + threshold: f32, + exclude: u32, +} + +impl Admit { + #[inline(always)] + fn ok(&self, rows: &Rows, slot: u32, score: f32) -> bool { + let s = slot as usize; + slot != self.exclude + && !rows.norms[s].is_nan() + && score >= self.threshold + && self.code.is_none_or(|c| rows.codes[s] == c) + } +} + +/// The marks of the nodes a search has touched. Thread-local and reused: a +/// query clears only the words it set, so it costs what it visited, and the +/// memory is one bit per slot per searching thread. +#[derive(Default)] +struct Visited { + bits: Vec<u64>, + words: Vec<u32>, +} + +impl Visited { + fn begin(&mut self, slots: usize) { + self.clear(); + let need = slots.div_ceil(64); + if self.bits.len() < need { + self.bits.resize(need, 0); + } + } + + fn clear(&mut self) { + for &w in &self.words { + self.bits[w as usize] = 0; + } + self.words.clear(); + } + + /// Mark `slot`; true if it already was. + #[inline(always)] + fn test_set(&mut self, slot: u32) -> bool { + let w = (slot >> 6) as usize; + let m = 1u64 << (slot & 63); + let cur = self.bits[w]; + if cur & m != 0 { + return true; + } + if cur == 0 { + self.words.push(w as u32); + } + self.bits[w] = cur | m; + false + } +} + +thread_local! { + static VISITED: RefCell<Visited> = RefCell::new(Visited::default()); +} + +fn with_visited<R>(slots: usize, f: impl FnOnce(&mut Visited) -> R) -> R { + VISITED.with(|v| { + let mut v = v.borrow_mut(); + v.begin(slots); + let out = f(&mut v); + v.clear(); + out + }) +} + +/// What a layer search returns. +pub(crate) struct Found { + /// Admitted nodes, best first. + pub hits: Vec<Hit>, + /// The search gave up after scoring too many nodes (a filter so selective + /// that scanning is cheaper). + pub budget_hit: bool, +} + +// ============================================================================= +// The graph +// ============================================================================= + +pub(crate) struct Hnsw { + m: usize, + m0: usize, + ef_c: usize, + /// Highest layer of each slot's node, or `ABSENT`. + levels: Vec<u8>, + /// Layer 0: `CHUNK_ROWS` nodes per chunk, `m0 + 1` words each (the link + /// count, then the links). + l0: Vec<Vec<u32>>, + /// Per slot: where its upper-layer blocks start in `upper`, or `NONE`. + upper_at: Vec<u32>, + /// A node of level `L` owns `L` consecutive blocks of `m + 1` words. + upper: Vec<u32>, + entry: u32, + top: u8, + rng: Rng, + /// Nodes in the graph, tombstones included. + members: usize, + churn: usize, + build_seconds: f64, + inserts: u64, + insert_seconds: f64, +} + +/// One node to link: a new row, or an existing node whose vector changed. +#[derive(Clone, Copy)] +struct Job { + slot: u32, + level: u8, + relink: bool, +} + +impl Hnsw { + fn new(cfg: &AnnConfig) -> Hnsw { + let h = cfg.hnsw; + Hnsw { + m: h.m, + m0: 2 * h.m, + ef_c: h.ef_construction.max(h.m), + levels: Vec::new(), + l0: Vec::new(), + upper_at: Vec::new(), + upper: Vec::new(), + entry: NONE, + top: 0, + rng: Rng(cfg.seed ^ 0x4857_5357_4857_5357), + members: 0, + churn: 0, + build_seconds: 0.0, + inserts: 0, + insert_seconds: 0.0, + } + } + + fn cap(&self, level: usize) -> usize { + if level == 0 { + self.m0 + } else { + self.m + } + } + + /// Cover slot `s` (new slots start absent, with empty link blocks). + fn ensure_slot(&mut self, s: usize) { + while self.levels.len() <= s { + self.levels.push(ABSENT); + self.upper_at.push(NONE); + } + while self.l0.len() <= s >> CHUNK_SHIFT { + self.l0.push(vec![0u32; CHUNK_ROWS * (self.m0 + 1)]); + } + } + + fn block_at(&self, s: usize, level: usize) -> (bool, usize) { + if level == 0 { + (true, (s & CHUNK_MASK) * (self.m0 + 1)) + } else { + ( + false, + self.upper_at[s] as usize + (level - 1) * (self.m + 1), + ) + } + } + + /// The links of `s` on `level` (`s` must have that layer). + #[inline(always)] + fn list(&self, s: usize, level: usize) -> &[u32] { + let (zero, at) = self.block_at(s, level); + let block = if zero { + &self.l0[s >> CHUNK_SHIFT][at..at + self.m0 + 1] + } else { + &self.upper[at..at + self.m + 1] + }; + &block[1..1 + block[0] as usize] + } + + fn set_list(&mut self, s: usize, level: usize, items: &[u32]) { + debug_assert!(items.len() <= self.cap(level)); + let (zero, at) = self.block_at(s, level); + let (m0, m) = (self.m0, self.m); + let block = if zero { + &mut self.l0[s >> CHUNK_SHIFT][at..at + m0 + 1] + } else { + &mut self.upper[at..at + m + 1] + }; + block[0] = items.len() as u32; + block[1..1 + items.len()].copy_from_slice(items); + } + + fn add_node(&mut self, s: usize, level: u8) { + self.ensure_slot(s); + debug_assert_eq!(self.levels[s], ABSENT); + self.levels[s] = level; + if level >= 1 { + self.upper_at[s] = self.upper.len() as u32; + let grown = self.upper.len() + level as usize * (self.m + 1); + self.upper.resize(grown, 0); + } + self.members += 1; + } + + /// `P(level >= l) = m^-l`, from integer draws only (portable). + fn draw_level(&mut self) -> u8 { + let mut level = 0usize; + while level < MAX_LEVEL && self.rng.below(self.m) == 0 { + level += 1; + } + level as u8 + } + + fn in_graph(&self, s: usize) -> bool { + self.levels.get(s).is_some_and(|&l| l != ABSENT) + } + + pub(crate) fn stale(&self) -> bool { + self.churn as f64 > REBUILD_FRACTION * self.members.max(1) as f64 + } + + // -- search ----------------------------------------------------------------- + + /// Walk to the best node reachable by single steps on `level`. + fn greedy(&self, rows: Rows, q: &QueryVec, mut cur: Sc, level: usize) -> Sc { + let mut scores = [0.0f32; MAX_CAP]; + loop { + let list = self.list(cur.slot as usize, level); + if list.is_empty() { + return cur; + } + for &x in list { + prefetch(rows.row(x as usize)); + } + score_into(rows, q, list, &mut scores[..list.len()]); + let mut best = cur; + for (i, &x) in list.iter().enumerate() { + let c = Sc { + score: scores[i], + slot: x, + }; + if c > best { + best = c; + } + } + if best == cur { + return cur; + } + cur = best; + } + } + + /// Best-first search of one layer from `eps`, keeping the `ef` best + /// admitted nodes. Tombstones and non-admitted nodes are routed through but + /// never kept. Stops when the best unexpanded node is worse than the worst + /// kept one (once `ef` are kept), or after `budget` nodes were scored. + #[allow(clippy::too_many_arguments)] + fn search_layer( + &self, + rows: Rows, + q: &QueryVec, + eps: &[Sc], + ef: usize, + level: usize, + admit: &Admit, + budget: usize, + visited: &mut Visited, + ) -> Found { + visited.clear(); + let mut cand: BinaryHeap<Sc> = BinaryHeap::new(); + let mut kept: BinaryHeap<Reverse<Sc>> = BinaryHeap::with_capacity(ef + 1); + for &e in eps { + if !visited.test_set(e.slot) { + cand.push(e); + if admit.ok(&rows, e.slot, e.score) { + kept.push(Reverse(e)); + if kept.len() > ef { + kept.pop(); + } + } + } + } + let mut fresh: Vec<u32> = Vec::with_capacity(self.m0); + let mut scores = [0.0f32; MAX_CAP]; + let mut scored = 0usize; + let mut budget_hit = false; + while let Some(c) = cand.pop() { + if kept.len() >= ef { + if let Some(Reverse(worst)) = kept.peek() { + if c < *worst { + break; + } + } + } + fresh.clear(); + for &x in self.list(c.slot as usize, level) { + if !visited.test_set(x) { + fresh.push(x); + prefetch(rows.row(x as usize)); + } + } + if fresh.is_empty() { + continue; + } + score_into(rows, q, &fresh, &mut scores[..fresh.len()]); + scored += fresh.len(); + for (i, &x) in fresh.iter().enumerate() { + let node = Sc { + score: scores[i], + slot: x, + }; + let promising = kept.len() < ef || kept.peek().is_some_and(|w| node > w.0); + if promising { + cand.push(node); + if admit.ok(&rows, x, node.score) { + kept.push(Reverse(node)); + if kept.len() > ef { + kept.pop(); + } + } + } + } + if scored > budget { + budget_hit = true; + break; + } + } + let mut best: Vec<Sc> = kept.into_iter().map(|r| r.0).collect(); + best.sort_unstable_by(|a, b| b.cmp(a)); + Found { + hits: best + .into_iter() + .map(|s| Hit { + score: s.score, + slot: s.slot, + }) + .collect(), + budget_hit, + } + } + + /// The `ef` best admitted nodes for a query, best first (ties by slot). + pub(crate) fn search( + &self, + rows: Rows, + q: &QueryVec, + code: Option<u32>, + threshold: f32, + ef: usize, + budget: usize, + ) -> Found { + if self.entry == NONE { + return Found { + hits: Vec::new(), + budget_hit: false, + }; + } + with_visited(self.levels.len(), |visited| { + let mut cur = score_one(rows, q, self.entry); + for level in (1..=self.top as usize).rev() { + cur = self.greedy(rows, q, cur, level); + } + let admit = Admit { + code, + threshold, + exclude: NONE, + }; + let mut found = self.search_layer(rows, q, &[cur], ef, 0, &admit, budget, visited); + // best first, ties by slot, exactly as the scan orders them + found.hits.sort_unstable_by(|a, b| { + b.score + .partial_cmp(&a.score) + .unwrap_or(Cmp::Equal) + .then(a.slot.cmp(&b.slot)) + }); + found + }) + } + + // -- linking ------------------------------------------------------------------ + + /// The diversity heuristic: walk `cands` (best first) and keep one only if + /// it is closer to the base than to every node already kept, until `cap` + /// are kept. Fewer than `cap` candidates are all kept. + fn select(&self, rows: Rows, cands: &[Sc], cap: usize) -> Vec<u32> { + if cands.len() <= cap { + return cands.iter().map(|c| c.slot).collect(); + } + let mut chosen: Vec<u32> = Vec::with_capacity(cap); + let mut scores = [0.0f32; MAX_CAP]; + for c in cands { + if chosen.len() >= cap { + break; + } + let q = node_query(&rows, c.slot as usize); + score_into(rows, &q, &chosen, &mut scores[..chosen.len()]); + if !scores[..chosen.len()].iter().any(|&s| s > c.score) { + chosen.push(c.slot); + } + } + chosen + } + + /// Neighbours chosen for `job` on every layer, against the graph as it is + /// now plus the other new members of its batch (`intra`). + fn plan( + &self, + rows: Rows, + job: &Job, + intra: &[(u32, u8)], + visited: &mut Visited, + ) -> Vec<Vec<u32>> { + let x = job.slot; + let level = job.level as usize; + let q = node_query(&rows, x as usize); + let admit = Admit { + code: None, + threshold: f32::NEG_INFINITY, + exclude: x, + }; + let top = self.top as usize; + let mut lists: Vec<Vec<u32>> = vec![Vec::new(); level + 1]; + let mut eps: Vec<Sc> = Vec::new(); + if self.entry != NONE { + let mut cur = score_one(rows, &q, self.entry); + for l in ((level + 1)..=top).rev() { + cur = self.greedy(rows, &q, cur, l); + } + eps.push(cur); + } + for l in (0..=level).rev() { + let mut cands: Vec<Sc> = Vec::new(); + if l <= top && !eps.is_empty() { + let found = + self.search_layer(rows, &q, &eps, self.ef_c, l, &admit, usize::MAX, visited); + cands = found + .hits + .iter() + .map(|h| Sc { + score: h.score, + slot: h.slot, + }) + .collect(); + if !cands.is_empty() { + eps = cands.clone(); + } + } + let members: Vec<u32> = intra + .iter() + .filter(|&&(s, lv)| s != x && lv as usize >= l) + .map(|&(s, _)| s) + .collect(); + if !members.is_empty() { + let mut scores = vec![0.0f32; members.len()]; + score_into(rows, &q, &members, &mut scores); + cands.extend( + members + .iter() + .zip(&scores) + .map(|(&slot, &score)| Sc { score, slot }), + ); + cands.sort_unstable_by(|a, b| b.cmp(a)); + } + lists[l] = self.select(rows, &cands, self.m); + } + lists + } + + /// `t`'s list on `layer` with `incoming` added, pruned to the layer's + /// capacity when it overflows (tombstones are dropped then). `None` if + /// nothing changes. + fn grown_list(&self, rows: Rows, layer: usize, t: u32, incoming: &[u32]) -> Option<Vec<u32>> { + let cap = self.cap(layer); + let cur = self.list(t as usize, layer); + let mut all: Vec<u32> = cur.to_vec(); + for &n in incoming { + if n != t && !all.contains(&n) { + all.push(n); + } + } + if all.len() == cur.len() { + return None; + } + if all.len() <= cap { + return Some(all); + } + all.retain(|&s| !rows.norms[s as usize].is_nan()); + if all.len() <= cap { + return Some(all); + } + let q = node_query(&rows, t as usize); + let mut scores = vec![0.0f32; all.len()]; + score_into(rows, &q, &all, &mut scores); + let mut cands: Vec<Sc> = all + .iter() + .zip(&scores) + .map(|(&slot, &score)| Sc { score, slot }) + .collect(); + cands.sort_unstable_by(|a, b| b.cmp(a)); + Some(self.select(rows, &cands, cap)) + } + + /// Link `jobs` into the graph: plan every job against the current graph (in + /// parallel, read-only), then apply the plans in job order. + fn link(&mut self, table: &VectorTable, jobs: &[Job], par: Par) { + let rows = table.rows(); + let slots = self.levels.len(); + let intra: Vec<(u32, u8)> = jobs + .iter() + .filter(|j| !j.relink) + .map(|j| (j.slot, j.level)) + .collect(); + + let plans: Vec<Vec<Vec<u32>>> = { + let this: &Hnsw = self; + let workers = if jobs.len() <= 1 { + 1 + } else { + par.workers(jobs.len(), usize::MAX) + }; + if workers <= 1 { + jobs.iter() + .map(|j| with_visited(slots, |v| this.plan(rows, j, &intra, v))) + .collect() + } else { + let next = AtomicUsize::new(0); + let done: Mutex<Vec<(usize, Vec<Vec<u32>>)>> = Mutex::new(Vec::new()); + par.run(workers, &|_| { + let mut local = Vec::new(); + with_visited(slots, |v| loop { + let i = next.fetch_add(1, Ordering::Relaxed); + if i >= jobs.len() { + break; + } + local.push((i, this.plan(rows, &jobs[i], &intra, v))); + }); + done.lock().unwrap_or_else(|e| e.into_inner()).extend(local); + }); + let mut done = done.into_inner().unwrap_or_else(|e| e.into_inner()); + done.sort_unstable_by_key(|d| d.0); + done.into_iter().map(|d| d.1).collect() + } + }; + + // forward links + for (job, lists) in jobs.iter().zip(&plans) { + if !job.relink { + self.add_node(job.slot as usize, job.level); + } + for (l, list) in lists.iter().enumerate() { + self.set_list(job.slot as usize, l, list); + } + } + + // backward links, grouped by (layer, target) in a fixed order + let mut adds: Vec<(u8, u32, u32)> = Vec::new(); + for (job, lists) in jobs.iter().zip(&plans) { + for (l, list) in lists.iter().enumerate() { + for &nb in list { + if !rows.norms[nb as usize].is_nan() { + adds.push((l as u8, nb, job.slot)); + } + } + } + } + adds.sort_by_key(|a| (a.0, a.1)); + let mut groups: Vec<(usize, usize)> = Vec::new(); // (start, end) into `adds` + let mut at = 0; + while at < adds.len() { + let mut end = at + 1; + while end < adds.len() && adds[end].0 == adds[at].0 && adds[end].1 == adds[at].1 { + end += 1; + } + groups.push((at, end)); + at = end; + } + let grown: Vec<Option<Vec<u32>>> = { + let this: &Hnsw = self; + let one = |g: &(usize, usize)| { + let (layer, target) = (adds[g.0].0 as usize, adds[g.0].1); + let incoming: Vec<u32> = adds[g.0..g.1].iter().map(|a| a.2).collect(); + this.grown_list(rows, layer, target, &incoming) + }; + let workers = if groups.len() <= 1 { + 1 + } else { + par.workers(groups.len(), usize::MAX) + }; + if workers <= 1 { + groups.iter().map(one).collect() + } else { + let next = AtomicUsize::new(0); + let done: Mutex<Vec<(usize, Option<Vec<u32>>)>> = Mutex::new(Vec::new()); + par.run(workers, &|_| { + let mut local = Vec::new(); + loop { + let i = next.fetch_add(1, Ordering::Relaxed); + if i >= groups.len() { + break; + } + local.push((i, one(&groups[i]))); + } + done.lock().unwrap_or_else(|e| e.into_inner()).extend(local); + }); + let mut done = done.into_inner().unwrap_or_else(|e| e.into_inner()); + done.sort_unstable_by_key(|d| d.0); + done.into_iter().map(|d| d.1).collect() + } + }; + for (g, list) in groups.iter().zip(grown) { + if let Some(list) = list { + self.set_list(adds[g.0].1 as usize, adds[g.0].0 as usize, &list); + } + } + + // the entry point is the first node of the highest level + for job in jobs.iter().filter(|j| !j.relink) { + if self.entry == NONE || job.level > self.top { + self.entry = job.slot; + self.top = job.level; + } + } + } + + /// A graph over every live row of `table`, built in slot order. + pub(crate) fn build(table: &VectorTable, cfg: &AnnConfig, par: Par) -> Hnsw { + let started = Instant::now(); + let mut h = Hnsw::new(cfg); + let n = table.slots(); + if n > 0 { + h.ensure_slot(n - 1); + } + let order: Vec<u32> = table + .live_slots() + .filter(|&s| table.norm(s).is_finite()) + .map(|s| s as u32) + .collect(); + let mut at = 0; + let mut jobs: Vec<Job> = Vec::new(); + while at < order.len() { + let size = (at / BUILD_RATIO).clamp(1, BUILD_BATCH); + let end = (at + size).min(order.len()); + jobs.clear(); + for &slot in &order[at..end] { + let level = h.draw_level(); + jobs.push(Job { + slot, + level, + relink: false, + }); + } + h.link(table, &jobs, par); + at = end; + } + h.build_seconds = started.elapsed().as_secs_f64(); + h + } + + /// A row was written (new, or its vector replaced). + fn note_put(&mut self, table: &VectorTable, slot: u32, par: Par) { + let started = Instant::now(); + let s = slot as usize; + self.ensure_slot(s); + if !table.norm(s).is_finite() { + if self.in_graph(s) { + self.churn += 1; + } + return; + } + let job = if self.in_graph(s) { + self.churn += 1; + Job { + slot, + level: self.levels[s], + relink: true, + } + } else { + Job { + slot, + level: self.draw_level(), + relink: false, + } + }; + self.link(table, &[job], par); + self.inserts += 1; + self.insert_seconds += started.elapsed().as_secs_f64(); + } + + fn note_drop(&mut self) { + self.churn += 1; + } + + /// Follow the table's compaction (`live` = old slots kept, in order): + /// tombstones leave the graph, links are renumbered, and a node that lost + /// links takes up the links of the tombstones it pointed to. + fn compact(&mut self, live: &[u32]) { + let old_n = self.levels.len(); + let mut new_of = vec![NONE; old_n]; + for (new, &old) in live.iter().enumerate() { + if (old as usize) < old_n { + new_of[old as usize] = new as u32; + } + } + let mut fresh = Hnsw { + m: self.m, + m0: self.m0, + ef_c: self.ef_c, + levels: Vec::new(), + l0: Vec::new(), + upper_at: Vec::new(), + upper: Vec::new(), + entry: NONE, + top: 0, + rng: Rng(self.rng.0), + members: 0, + churn: self.churn, + build_seconds: self.build_seconds, + inserts: self.inserts, + insert_seconds: self.insert_seconds, + }; + if !live.is_empty() { + fresh.ensure_slot(live.len() - 1); + } + let mut lost: Vec<u32> = Vec::new(); + let mut out: Vec<u32> = Vec::new(); + for (new, &old) in live.iter().enumerate() { + let old = old as usize; + let level = self.levels.get(old).copied().unwrap_or(ABSENT); + // Only live rows survive; a tombstone has no new slot, so every + // surviving node is either a live graph member or absent. + if level == ABSENT { + continue; + } + fresh.add_node(new, level); + for l in 0..=level as usize { + let cap = self.cap(l); + out.clear(); + lost.clear(); + for &nb in self.list(old, l) { + match new_of.get(nb as usize) { + Some(&n) if n != NONE => out.push(n), + _ => lost.push(nb), + } + } + 'refill: for &d in &lost { + if out.len() >= cap { + break; + } + let d = d as usize; + if d >= old_n || (self.levels[d] as usize) < l || self.levels[d] == ABSENT { + continue; + } + for &w in self.list(d, l) { + let nw = new_of.get(w as usize).copied().unwrap_or(NONE); + if nw != NONE && nw as usize != new && !out.contains(&nw) { + out.push(nw); + if out.len() >= cap { + break 'refill; + } + } + } + } + fresh.set_list(new, l, &out); + } + if fresh.entry == NONE || level > fresh.top { + fresh.entry = new as u32; + fresh.top = level; + } + } + *self = fresh; + } + + /// Every invariant of the graph against the table it indexes. + pub(crate) fn check(&self, table: &VectorTable) -> Result<(), String> { + let n = table.slots(); + if self.levels.len() != n || self.upper_at.len() != n { + return Err(format!( + "graph covers {} rows, table has {n}", + self.levels.len() + )); + } + if self.l0.len() != n.div_ceil(CHUNK_ROWS) { + return Err(format!("{} link chunks for {n} rows", self.l0.len())); + } + let mut members = 0usize; + let mut max_level = 0u8; + for s in 0..n { + let level = self.levels[s]; + if level == ABSENT { + if table.is_live(s) && table.norm(s).is_finite() { + return Err(format!("live slot {s} is not in the graph")); + } + continue; + } + members += 1; + max_level = max_level.max(level); + for l in 0..=level as usize { + let list = self.list(s, l); + if list.len() > self.cap(l) { + return Err(format!("slot {s} layer {l} holds {} links", list.len())); + } + let mut seen: Vec<u32> = list.to_vec(); + seen.sort_unstable(); + if seen.windows(2).any(|w| w[0] == w[1]) { + return Err(format!("slot {s} layer {l} links a node twice")); + } + for &nb in list { + let t = nb as usize; + if t >= n || nb as usize == s { + return Err(format!("slot {s} layer {l} links {nb}")); + } + if self.levels[t] == ABSENT || (self.levels[t] as usize) < l { + return Err(format!( + "slot {s} layer {l} links {nb}, which is not on that layer" + )); + } + } + } + } + if members != self.members { + return Err(format!( + "member counter {} but {members} nodes", + self.members + )); + } + if members == 0 { + if self.entry != NONE { + return Err("an empty graph has an entry point".to_string()); + } + } else if self.entry == NONE + || self.levels[self.entry as usize] != self.top + || max_level != self.top + { + return Err(format!( + "entry {} (level {:?}) does not sit on top level {} (highest node {max_level})", + self.entry, + self.levels.get(self.entry as usize), + self.top + )); + } + Ok(()) + } + + /// Nodes whose rows are tombstones. + fn dead(&self, table: &VectorTable) -> usize { + (0..self.levels.len()) + .filter(|&s| self.levels[s] != ABSENT && table.norm(s).is_nan()) + .count() + } + + pub(crate) fn nbytes(&self) -> usize { + self.levels.capacity() + + self.l0.iter().map(|c| c.capacity() * 4).sum::<usize>() + + self.upper_at.capacity() * 4 + + self.upper.capacity() * 4 + } +} + +// ============================================================================= +// The store's side +// ============================================================================= + +/// The graph slot an [`EmbeddingStore`] carries: built lazily by a search, +/// maintained by writes through `&mut self`, never persisted. +pub(crate) type HnswSlot = RwLock<Option<Hnsw>>; + +/// `ef_search` by store size when none is configured (see +/// [`HnswConfig::ef_search`]). +pub(crate) fn auto_ef(live: usize) -> usize { + match live { + 0..=9_999 => 64, + 10_000..=199_999 => 96, + 200_000..=599_999 => 128, + _ => 192, + } +} + +/// The `ef` a query uses: the per-call override, else the configured value, +/// else the size-based default; never below `k`. +fn effective_ef(cfg: &AnnConfig, over: Option<usize>, k: usize, live: usize) -> usize { + over.or(cfg.hnsw.ef_search) + .unwrap_or_else(|| auto_ef(live)) + .max(k) + .max(1) +} + +/// Scoring more than this many nodes costs more than the exact scan. +fn budget_for(live: usize, ef: usize) -> usize { + (live / 8).max(ef.saturating_mul(16)) +} + +impl EmbeddingStore { + pub(crate) fn hnsw_mut(&mut self) -> &mut Option<Hnsw> { + self.hnsw.get_mut().unwrap_or_else(|e| e.into_inner()) + } + + pub(crate) fn hnsw_read(&self) -> RwLockReadGuard<'_, Option<Hnsw>> { + self.hnsw.read().unwrap_or_else(|e| e.into_inner()) + } + + /// A row was written: link it into the graph now. + pub(crate) fn hnsw_row_written(&mut self, slot: u32) { + let (table, pool, configured) = (&self.table, &self.pool, self.scan_threads); + let guard = self.hnsw.get_mut().unwrap_or_else(|e| e.into_inner()); + if let Some(h) = guard.as_mut() { + h.note_put(table, slot, Par { configured, pool }); + } + } + + pub(crate) fn hnsw_row_removed(&mut self) { + if let Some(h) = self.hnsw_mut().as_mut() { + h.note_drop(); + } + } + + pub(crate) fn hnsw_compact(&mut self, live: &[u32]) { + if let Some(h) = self.hnsw_mut().as_mut() { + h.compact(live); + } + } + + /// A read lock on a usable graph, building (or rebuilding when stale) + /// first; `None` means "answer exactly" (the store is under `min_size`). + pub(crate) fn ready_hnsw(&self, cfg: &AnnConfig) -> Option<RwLockReadGuard<'_, Option<Hnsw>>> { + if self.table.live() < cfg.min_size.max(2) { + return None; + } + loop { + { + let guard = self.hnsw_read(); + if guard.as_ref().is_some_and(|h| !h.stale()) { + return Some(guard); + } + } + let mut guard = self.hnsw.write().unwrap_or_else(|e| e.into_inner()); + if guard.as_ref().is_none_or(|h| h.stale()) { + *guard = Some(Hnsw::build(&self.table, cfg, self.par())); + self.counts.builds.fetch_add(1, Ordering::Relaxed); + } + } + } + + /// Top `k` from the graph, or `None` to fall back to the scan. Widens `ef` + /// 4x when a filter leaves fewer than `k`, and gives up (scan) once `ef` + /// would cover half the store or the search scores more nodes than the scan + /// would cost. + #[allow(clippy::too_many_arguments)] + pub(crate) fn hnsw_topk( + &self, + cfg: &AnnConfig, + h: &Hnsw, + q: &QueryVec, + code: Option<u32>, + threshold: f32, + k: usize, + over: Option<usize>, + ) -> Option<Vec<Hit>> { + if !q.has_direction() { + return None; + } + let live = self.table.live(); + let base = effective_ef(cfg, over, k, live); + let rows = self.table.rows(); + let mut scale = 1usize; + loop { + let ef = base.saturating_mul(scale); + let exhausted = if scale == 1 { + ef >= live + } else { + ef.saturating_mul(2) >= live + }; + if exhausted { + return None; + } + let found = h.search(rows, q, code, threshold, ef, budget_for(live, ef)); + if found.budget_hit { + return None; + } + if found.hits.len() >= k { + let mut hits = found.hits; + hits.truncate(k); + self.counts.note_approximate(scale > 1); + return Some(hits); + } + scale = scale.saturating_mul(4); + } + } + + /// Every row the widened search keeps (for a caller that filters further), + /// best first; `None` to fall back to the scan. + pub(crate) fn hnsw_ranked( + &self, + cfg: &AnnConfig, + q: &QueryVec, + code: Option<u32>, + threshold: f32, + scale: usize, + ) -> Option<Vec<Hit>> { + if !q.has_direction() { + return None; + } + let guard = self.ready_hnsw(cfg)?; + let h = guard.as_ref()?; + let live = self.table.live(); + let ef = effective_ef(cfg, None, 1, live).saturating_mul(scale); + let exhausted = if scale == 1 { + ef >= live + } else { + ef.saturating_mul(2) >= live + }; + if exhausted { + return None; + } + let found = h.search( + self.table.rows(), + q, + code, + threshold, + ef, + budget_for(live, ef), + ); + if found.budget_hit { + return None; + } + self.counts.note_approximate(scale > 1); + Some(found.hits) + } + + /// Answer a batch from the graph: queries are spread over the pool and each + /// runs the single-query path, so each answer is the one-query answer bit + /// for bit. Queries it cannot answer stay `None` for the exact batch. + #[allow(clippy::too_many_arguments)] + pub(crate) fn hnsw_batch( + &self, + cfg: &AnnConfig, + h: &Hnsw, + qs: &[QueryVec], + code: Option<u32>, + threshold: f32, + k: usize, + answers: &mut [Option<Vec<Hit>>], + ) { + let par = self.par(); + let next = AtomicUsize::new(0); + let done: Mutex<Vec<(usize, Option<Vec<Hit>>)>> = Mutex::new(Vec::new()); + par.run(par.workers(qs.len(), usize::MAX), &|_| { + let mut local = Vec::new(); + loop { + let i = next.fetch_add(1, Ordering::Relaxed); + if i >= qs.len() { + break; + } + local.push((i, self.hnsw_topk(cfg, h, &qs[i], code, threshold, k, None))); + } + done.lock().unwrap_or_else(|e| e.into_inner()).extend(local); + }); + for (i, hits) in done.into_inner().unwrap_or_else(|e| e.into_inner()) { + answers[i] = hits; + } + } + + /// Change `ef_search` (`None` = chosen by store size). Takes effect on the next + /// search; the graph is not rebuilt. + pub fn set_ann_ef_search(&mut self, ef_search: Option<usize>) -> Result<(), EmbeddingError> { + if ef_search == Some(0) { + return Err(EmbeddingError::InvalidOption( + "hnsw ef_search must be at least 1".to_string(), + )); + } + match self.ann.as_mut() { + Some(config) => { + config.hnsw.ef_search = ef_search; + Ok(()) + } + None => Err(EmbeddingError::InvalidOption( + "this store has no approximate index; call set_ann first".to_string(), + )), + } + } + + /// Like `similarity_search_vector`, with `ef_search` for this call only + /// (HNSW mode; ignored by the exact scan and by IVF). + pub fn similarity_search_vector_ef( + &self, + query_vector: &[f32], + top_k: usize, + node_type: Option<&str>, + threshold: f32, + ef_search: usize, + ) -> Result<Vec<crate::SimilarityResult>, EmbeddingError> { + if ef_search == 0 { + return Err(EmbeddingError::InvalidOption( + "hnsw ef_search must be at least 1".to_string(), + )); + } + self.search_vector_with( + query_vector, + top_k, + node_type, + threshold, + false, + Some(ef_search), + ) + } + + /// Graph facts for `ann_stats`. + pub(crate) fn hnsw_stats(&self) -> HnswStats { + let guard = self.hnsw_read(); + match guard.as_ref() { + Some(h) => HnswStats { + max_level: Some(h.top as usize), + nodes: h.members, + dead: h.dead(&self.table), + churn: h.churn, + build_seconds: Some(h.build_seconds), + bytes: h.nbytes(), + inserts: h.inserts, + insert_seconds: h.insert_seconds, + built: true, + }, + None => HnswStats::default(), + } + } + + pub(crate) fn hnsw_is_mode(&self) -> bool { + self.ann.as_ref().is_some_and(|c| c.mode == AnnMode::Hnsw) + } + + /// `ann_stats` for a store in HNSW mode. + pub(crate) fn hnsw_ann_stats(&self) -> AnnStats { + let g = self.hnsw_stats(); + let cfg = self.ann.as_ref(); + AnnStats { + enabled: cfg.is_some(), + active: g.built, + nlist: None, + nprobe: None, + trained_n: g.built.then_some(g.nodes), + pending: 0, + churn: g.churn, + build_seconds: g.build_seconds, + index_bytes: g.bytes, + min_size: cfg.map(|c| c.min_size), + searches: self.counts.snapshot(), + folds: 0, + fold_seconds: 0.0, + mode: AnnMode::Hnsw, + ef_search: cfg.map(|c| { + c.hnsw + .ef_search + .unwrap_or_else(|| auto_ef(self.table.live())) + }), + max_level: g.max_level, + graph_nodes: g.nodes, + graph_dead: g.dead, + inserts: g.inserts, + insert_seconds: g.insert_seconds, + } + } +} + +#[derive(Default)] +pub(crate) struct HnswStats { + pub max_level: Option<usize>, + pub nodes: usize, + pub dead: usize, + pub churn: usize, + pub build_seconds: Option<f64>, + pub bytes: usize, + pub inserts: u64, + pub insert_seconds: f64, + pub built: bool, +} + +#[cfg(test)] +mod tests { + use super::*; + use std::sync::Arc; + + fn table(n: usize, dim: usize) -> VectorTable { + let model: Arc<str> = Arc::from("m"); + let mut rng = Rng(99); + let centers: Vec<Vec<f32>> = (0..8) + .map(|_| { + (0..dim) + .map(|_| (rng.below(2001) as f32 - 1000.0) / 1000.0) + .collect() + }) + .collect(); + let mut t = VectorTable::new(); + for i in 0..n { + let c = ¢ers[rng.below(centers.len())]; + let v: Vec<f32> = c + .iter() + .map(|x| x + (rng.below(2001) as f32 - 1000.0) / 2500.0) + .collect(); + t.put(&format!("doc:{i}"), "doc", &v, "", &model); + } + t + } + + fn cfg() -> AnnConfig { + AnnConfig { + min_size: 0, + ..AnnConfig::hnsw() + } + } + + fn same_graph(a: &Hnsw, b: &Hnsw) -> bool { + a.levels == b.levels + && a.l0 == b.l0 + && a.upper_at == b.upper_at + && a.upper == b.upper + && a.entry == b.entry + && a.top == b.top + && a.members == b.members + } + + #[test] + fn the_built_graph_does_not_depend_on_the_thread_count() { + let t = table(3000, 12); + let pool = None; + let one = Hnsw::build( + &t, + &cfg(), + Par { + configured: 1, + pool: &pool, + }, + ); + let auto = Hnsw::build( + &t, + &cfg(), + Par { + configured: 0, + pool: &pool, + }, + ); + let three = Hnsw::build( + &t, + &cfg(), + Par { + configured: 3, + pool: &pool, + }, + ); + one.check(&t).unwrap(); + assert!(same_graph(&one, &auto), "auto threads changed the graph"); + assert!(same_graph(&one, &three), "three threads changed the graph"); + let again = Hnsw::build( + &t, + &cfg(), + Par { + configured: 0, + pool: &pool, + }, + ); + assert!(same_graph(&one, &again)); + let other = Hnsw::build( + &t, + &AnnConfig { seed: 1, ..cfg() }, + Par { + configured: 1, + pool: &pool, + }, + ); + assert!(!same_graph(&one, &other), "the seed must matter"); + } + + #[test] + fn levels_follow_a_geometric_law() { + let mut h = Hnsw::new(&cfg()); + let draws = 160_000usize; + let mut counts = [0usize; 6]; + for _ in 0..draws { + let l = h.draw_level() as usize; + for c in counts.iter_mut().take(l.min(5) + 1).skip(1) { + *c += 1; // draws with level >= index + } + } + // P(level >= 1) = 1/16, P(level >= 2) = 1/256, ... + let p1 = counts[1] as f64 / draws as f64; + let p2 = counts[2] as f64 / draws as f64; + assert!((p1 - 1.0 / 16.0).abs() < 0.004, "{p1}"); + assert!((p2 - 1.0 / 256.0).abs() < 0.001, "{p2}"); + } + + /// Nodes reachable from the entry point on layer 0, live ones counted. + fn reachable_live(h: &Hnsw, t: &VectorTable) -> usize { + let mut seen = vec![false; t.slots()]; + let mut stack = vec![h.entry as usize]; + seen[h.entry as usize] = true; + while let Some(s) = stack.pop() { + for &nb in h.list(s, 0) { + if !seen[nb as usize] { + seen[nb as usize] = true; + stack.push(nb as usize); + } + } + } + (0..t.slots()).filter(|&s| seen[s] && t.is_live(s)).count() + } + + #[test] + fn compaction_renumbers_and_bridges_over_the_tombstones() { + let mut t = table(4000, 12); + let pool = None; + let par = Par { + configured: 0, + pool: &pool, + }; + let mut h = Hnsw::build(&t, &cfg(), par); + assert_eq!(reachable_live(&h, &t), 4000); + // kill every third row, but keep the entry point's row alive + for i in (0..4000).step_by(3) { + if i as u32 != h.entry { + t.remove(&format!("doc:{i}")); + } + } + for _ in 0..3 { + h.note_drop(); + } + let live = t.compact(); + h.compact(&live); + h.check(&t).unwrap(); + assert_eq!(h.members, t.live()); + assert_eq!(h.dead(&t), 0); + let reach = reachable_live(&h, &t); + assert!( + reach * 100 >= t.live() * 99, + "{reach} of {} live rows reachable after compaction", + t.live() + ); + // lists were refilled from the tombstones' links: the mean degree stays + // within 15% of what it was + let degree = |h: &Hnsw| -> f64 { + let total: usize = (0..h.levels.len()).map(|s| h.list(s, 0).len()).sum(); + total as f64 / h.levels.len() as f64 + }; + let fresh = Hnsw::build(&t, &cfg(), par); + assert!( + degree(&h) >= 0.85 * degree(&fresh), + "{} vs {}", + degree(&h), + degree(&fresh) + ); + } + + #[test] + fn writes_after_a_build_keep_every_invariant() { + let mut t = table(1500, 10); + let pool = None; + let par = Par { + configured: 0, + pool: &pool, + }; + let mut h = Hnsw::build(&t, &cfg(), par); + let model: Arc<str> = Arc::from("m"); + let mut rng = Rng(5); + for i in 0..400 { + let v: Vec<f32> = (0..10) + .map(|_| (rng.below(2001) as f32 - 1000.0) / 1000.0) + .collect(); + match i % 4 { + 0 | 1 => { + let (slot, _) = t.put(&format!("doc:{}", 2000 + i), "doc", &v, "", &model); + h.note_put(&t, slot, par); + } + 2 => { + let (slot, old) = + t.put(&format!("doc:{}", rng.below(1500)), "doc", &v, "", &model); + if old.is_some() { + h.note_put(&t, slot, par); + } + } + _ => { + if t.remove(&format!("doc:{}", rng.below(1500))).is_some() { + h.note_drop(); + } + } + } + } + h.check(&t).unwrap(); + assert!(h.inserts > 0 && h.churn > 0); + } +} diff --git a/isongraph-vector-rs/src/lib.rs b/isongraph-vector-rs/src/lib.rs index 66c7b04..168e4f5 100644 --- a/isongraph-vector-rs/src/lib.rs +++ b/isongraph-vector-rs/src/lib.rs @@ -26,9 +26,28 @@ use serde::{Deserialize, Serialize}; use std::cell::RefCell; use std::collections::{HashMap, HashSet, VecDeque}; use std::path::{Path, PathBuf}; -use std::sync::Mutex; +use std::sync::{Arc, Mutex, RwLock}; use thiserror::Error; +mod ann; +pub use ann::quant::{Int8Recall, ScanPrecision, DEFAULT_RERANK_FACTOR}; +pub use ann::{ + AnnConfig, AnnMode, AnnSearchCounts, AnnStats, MemoryStats, RecallReport, ANN_MIN_SIZE, + FOLD_BATCH, +}; +mod hnsw; +pub use hnsw::{HnswConfig, HNSW_EF_CONSTRUCTION, HNSW_M}; +mod pool; +pub use pool::SearchPool; +mod shared; +pub use shared::SharedEmbeddingStore; +mod snapshot; +pub use snapshot::{ + snapshot_path_for, LoadReport, LoadSource, SidecarId, SnapshotIdentity, SnapshotInfo, + SNAPSHOT_SUFFIX, SNAPSHOT_VERSION, +}; +mod table; +use table::VectorTable; mod digest; pub use digest::{ graph_digest, graph_digest_of_text, graph_digest_with_int_ids, GRAPH_DIGEST_BASIS, @@ -516,8 +535,21 @@ pub(crate) fn atomic_write(path: &Path, contents: &[u8]) -> Result<(), Embedding result } +/// Vectors, their text and model, searchable by cosine similarity. +/// +/// The vectors live once, in a contiguous matrix; a record is a row of it (see +/// `DESIGN.md`). Searches take `&self`, never lock the data and may run from +/// any number of threads at once; writes take `&mut self`, so a search can +/// never see a half-applied write. To read and write from different threads, +/// share the store through [`SharedEmbeddingStore`] (a reader/writer lock with +/// short write sections). pub struct EmbeddingStore { - embeddings: HashMap<String, EmbeddingRecord>, + /// The single copy of every vector, plus the per-row metadata and the key + /// index. Everything else derived from it (the approximate index) refers to + /// rows by slot number. + table: VectorTable, + /// One shared string per distinct model name, so a row holds a pointer. + model_names: HashMap<String, Arc<str>>, encoder: Option<Box<dyn EmbeddingEncoder>>, dimension: Option<usize>, version: Option<String>, @@ -529,12 +561,27 @@ pub struct EmbeddingStore { /// Every warning this store has issued, so callers and tests can read them. warnings: Mutex<Vec<String>>, last_import_meta: ImportMeta, + /// The opt-in approximate index settings (`None` = every search exact). + ann: Option<AnnConfig>, + /// The most threads a search, batch, fold or build may use (0 = automatic). + scan_threads: usize, + /// The pool parallel work runs on (`None` = the process-wide one). + pool: Option<Arc<SearchPool>>, + /// The approximate index: built lazily by a search (the only time the lock + /// is written during a search), maintained by writes through `&mut self` + /// (no locking), never persisted. + ivf: ann::IvfSlot, + /// The HNSW graph (same life cycle as `ivf`; only one of the two is ever + /// built, the one `ann.mode` names). + hnsw: hnsw::HnswSlot, + counts: ann::Counters, } impl EmbeddingStore { pub fn new(encoder: Option<Box<dyn EmbeddingEncoder>>) -> Self { Self { - embeddings: HashMap::new(), + table: VectorTable::new(), + model_names: HashMap::new(), encoder, dimension: None, version: None, @@ -543,6 +590,12 @@ impl EmbeddingStore { warned: Mutex::new(HashSet::new()), warnings: Mutex::new(Vec::new()), last_import_meta: ImportMeta::default(), + ann: None, + scan_threads: 0, + pool: None, + ivf: RwLock::new(None), + hnsw: RwLock::new(None), + counts: ann::Counters::default(), } } @@ -596,6 +649,20 @@ impl EmbeddingStore { self.warnings.lock().map(|w| w.clone()).unwrap_or_default() } + /// Warn when the encoder version in use differs from the one recorded by + /// the last import (JSON or snapshot). + pub(crate) fn warn_on_version_skew(&self) { + let stored = self.last_import_meta().encoder_version.clone(); + if let (Some(stored), Some(current)) = (stored, self.version()) { + if stored != current { + self.warn(format!( + "embeddings were made by encoder version '{stored}', but the \ + encoder in use is version '{current}'" + )); + } + } + } + pub fn clear_warnings(&self) { if let Ok(mut w) = self.warnings.lock() { w.clear(); @@ -650,27 +717,6 @@ impl EmbeddingStore { } } - fn cosine_similarity(v1: &[f32], v2: &[f32]) -> f32 { - let mut dot = 0.0; - let mut norm1 = 0.0; - let mut norm2 = 0.0; - - for (a, b) in v1.iter().zip(v2.iter()) { - dot += a * b; - norm1 += a * a; - norm2 += b * b; - } - - norm1 = norm1.sqrt(); - norm2 = norm2.sqrt(); - - if norm1 == 0.0 || norm2 == 0.0 { - 0.0 - } else { - dot / (norm1 * norm2) - } - } - // -- model lifecycle ------------------------------------------------------ /// Model name stamped on a vector written now: the encoder's, or @@ -751,22 +797,30 @@ impl EmbeddingStore { ) -> Vec<NodeId> { let model = model.map(String::from).or_else(|| self.current_model()); let dimension = dimension.or_else(|| self.encoder.as_ref().map(|e| e.dimension())); + let by_dim = dimension.is_some_and(|d| self.table.dim != d); let mut out: Vec<NodeId> = self - .embeddings - .values() - .filter(|record| { + .table + .live_slots() + .filter(|&slot| { + let meta = self.table.meta(slot).expect("live rows have metadata"); let by_model = model.as_ref().is_some_and(|m| { - record.model != *m && (include_unknown || !is_unknown_model(&record.model)) + *meta.model != **m && (include_unknown || !is_unknown_model(&meta.model)) }); - let by_dim = dimension.is_some_and(|d| record.vector.len() != d); by_model || by_dim }) - .map(|record| record.node_id.clone()) + .filter_map(|slot| self.table.node_id(slot)) .collect(); out.sort_by(|a, b| (&a.node_type, &a.id).cmp(&(&b.node_type, &b.id))); out } + /// Encode one text with the store's encoder (no model-compatibility + /// check; that is for queries, see `resolve_query`). + pub fn encode(&self, text: &str) -> Result<Vec<f32>, EmbeddingError> { + let encoder = self.encoder.as_ref().ok_or(EmbeddingError::NoEncoder)?; + Ok(encoder.encode(text)) + } + /// Turn a query into a vector. Text needs an encoder, and is checked /// against the stored vectors' model; a vector passes through untouched. pub fn resolve_query(&self, query: Query) -> Result<Vec<f32>, EmbeddingError> { @@ -782,33 +836,50 @@ impl EmbeddingStore { // -- writes --------------------------------------------------------------- - /// Insert or replace one record, keeping the per-model counts in step. + /// The shared string for a model name. + fn intern_model(&mut self, model: &str) -> Arc<str> { + if let Some(name) = self.model_names.get(model) { + return name.clone(); + } + let name: Arc<str> = Arc::from(model); + self.model_names.insert(model.to_string(), name.clone()); + name + } + + /// One fewer record carries `model`. + fn uncount_model(&mut self, model: &str) { + if let Some(n) = self.model_counts.get_mut(model) { + *n -= 1; + if *n == 0 { + self.model_counts.remove(model); + } + } + } + + /// Insert or replace one record, keeping the per-model counts in step. An + /// update overwrites the row in place (same slot); an insert takes the next + /// one. The write is complete when this returns: the row, its metadata, the + /// key index and the approximate index (including any fold it triggers). fn put(&mut self, node_id: NodeId, text: &str, vector: Vec<f32>, model: &str) -> String { let key = Self::node_id_to_key(&node_id); - self.forget(&key); + let model_name = self.intern_model(model); + let (slot, replaced) = self + .table + .put(&key, &node_id.node_type, &vector, text, &model_name); + if let Some(old) = replaced { + self.uncount_model(&old); + } *self.model_counts.entry(model.to_string()).or_insert(0) += 1; - self.embeddings.insert( - key.clone(), - EmbeddingRecord { - id: key.clone(), - node_id, - vector, - text: text.to_string(), - model: model.to_string(), - }, - ); + self.index_row_written(slot); key } + /// Remove a record: tombstone its row and take it out of the index. fn forget(&mut self, key: &str) -> bool { - match self.embeddings.remove(key) { - Some(old) => { - if let Some(n) = self.model_counts.get_mut(&old.model) { - *n -= 1; - if *n == 0 { - self.model_counts.remove(&old.model); - } - } + match self.table.remove(key) { + Some((slot, model)) => { + self.uncount_model(&model); + self.index_row_removed(slot); true } None => false, @@ -873,9 +944,18 @@ impl EmbeddingStore { Ok(entries.len()) } - pub fn get(&self, node_id: &NodeId) -> Option<&EmbeddingRecord> { + /// The record for a node, copied out of the store (the vector is a row of + /// the shared matrix, so a record is built on request; use + /// [`get_vector`](Self::get_vector) to borrow it instead). + pub fn get(&self, node_id: &NodeId) -> Option<EmbeddingRecord> { let key = Self::node_id_to_key(node_id); - self.embeddings.get(&key) + self.table.record(self.table.slot_of(&key)? as usize) + } + + /// The stored vector of a node, borrowed from the matrix. + pub fn get_vector(&self, node_id: &NodeId) -> Option<&[f32]> { + let key = Self::node_id_to_key(node_id); + Some(self.table.row(self.table.slot_of(&key)? as usize)) } pub fn remove(&mut self, node_id: &NodeId) -> bool { @@ -894,6 +974,13 @@ impl EmbeddingStore { self.similarity_search_vector(&query_vector, top_k, node_type, threshold) } + /// Best `top_k` matches for a vector, best first; equal scores come back + /// in a stable order (the order the vectors were stored in). + /// + /// `top_k = usize::MAX` returns every match. With an approximate index + /// configured (`set_ann`) a bounded `top_k` on a large enough store is + /// answered from it; use `similarity_search_vector_exact` to force the + /// full scan. pub fn similarity_search_vector( &self, query_vector: &[f32], @@ -901,65 +988,19 @@ impl EmbeddingStore { node_type: Option<&str>, threshold: f32, ) -> Result<Vec<SimilarityResult>, EmbeddingError> { - self.check_query(query_vector)?; - - let mut results: Vec<SimilarityResult> = self - .embeddings - .values() - .filter(|record| node_type.is_none_or(|t| record.node_id.node_type == t)) - .filter_map(|record| { - let score = Self::cosine_similarity(query_vector, &record.vector); - if score >= threshold { - Some(SimilarityResult { - node_id: record.node_id.clone(), - score, - text: record.text.clone(), - }) - } else { - None - } - }) - .collect(); - - // Selecting the top k rather than ordering every candidate: O(n log k) - // against O(n log n), same rows in the same order. - let by_score = |a: &SimilarityResult, b: &SimilarityResult| { - b.score - .partial_cmp(&a.score) - .unwrap_or(std::cmp::Ordering::Equal) - }; - if top_k == 0 { - return Ok(Vec::new()); - } - if results.len() > top_k { - results.select_nth_unstable_by(top_k - 1, by_score); - results.truncate(top_k); - } - results.sort_by(by_score); - - Ok(results) + self.search_vector(query_vector, top_k, node_type, threshold, false) } /// Cosine score for every embedded node, keyed by `type:id`. /// /// One scan serving both seed selection and the per-node relevance that - /// `semantic_multi_hop` blends in, instead of two. + /// `semantic_multi_hop` blends in, instead of two. Always the full scan. pub fn score_map(&self, query_vector: &[f32]) -> Result<HashMap<String, f32>, EmbeddingError> { - self.check_query(query_vector)?; - Ok(self - .embeddings - .iter() - .map(|(key, record)| { - ( - key.clone(), - Self::cosine_similarity(query_vector, &record.vector), - ) - }) - .collect()) + self.scan_scores(query_vector) } pub fn count(&self) -> usize { - self.embeddings.len() + self.table.live() } /// Remove every embedding and forget the pinned dimension, so the next @@ -974,8 +1015,10 @@ impl EmbeddingStore { /// without it the store keeps refusing vectors of the wrong length even /// while empty. pub fn clear_with(&mut self, reset_dimension: bool) { - self.embeddings.clear(); + self.table.clear(); self.model_counts.clear(); + *self.ivf_mut() = None; + *self.hnsw_mut() = None; if reset_dimension { self.dimension = None; } @@ -994,17 +1037,21 @@ impl EmbeddingStore { /// and `graph_digest` / `graph_digest_basis` when a digest is given. pub fn export_embeddings_with(&self, graph_digest: Option<&str>) -> ExportedEmbeddings { let mut embeddings: Vec<ExportedEmbedding> = self - .embeddings - .values() - .map(|record| ExportedEmbedding { - node_type: record.node_id.node_type.clone(), - id: EmbeddingId::Str(record.node_id.id.clone()), - // Rust node ids are always strings; the other ports use this - // to tell a numeric id from a numeric-looking string one. - id_type: "str".to_string(), - text: record.text.clone(), - model: Some(record.model.clone()), - vector: record.vector.clone(), + .table + .live_slots() + .filter_map(|slot| { + let meta = self.table.meta(slot)?; + let node_id = self.table.node_id(slot)?; + Some(ExportedEmbedding { + node_type: node_id.node_type, + id: EmbeddingId::Str(node_id.id), + // Rust node ids are always strings; the other ports use this + // to tell a numeric id from a numeric-looking string one. + id_type: "str".to_string(), + text: meta.text.to_string(), + model: Some(meta.model.to_string()), + vector: self.table.row(slot).to_vec(), + }) }) .collect(); embeddings.sort_by(|a, b| { @@ -1289,6 +1336,9 @@ pub struct SimilarityOptions<'a> { /// Applied *before* `top_k` is taken, so a filtered-out best match does /// not cost a slot. pub filter: NodeFilter<'a>, + /// Force the full scan even when the store has an approximate index + /// (`set_ann`); no effect otherwise. + pub exact: bool, } impl Default for SimilarityOptions<'_> { @@ -1298,6 +1348,7 @@ impl Default for SimilarityOptions<'_> { node_type: None, threshold: 0.0, filter: NodeFilter::default(), + exact: false, } } } @@ -1581,6 +1632,46 @@ impl SemanticGraph { self.embedding_store.encoder() } + /// Switch the approximate (IVF-flat) index on, with these settings, or off + /// with `None`. See `AnnConfig`. Nothing is saved: a saved graph, its + /// sidecar and its digest are the same with or without it, and a loaded + /// graph builds the index again on its first search. + pub fn set_ann(&mut self, config: Option<AnnConfig>) -> Result<(), EmbeddingError> { + self.embedding_store.set_ann(config) + } + + /// The approximate-index settings, or `None` while search is exact. + pub fn ann(&self) -> Option<&AnnConfig> { + self.embedding_store.ann() + } + + /// Build the approximate index now (see `EmbeddingStore::build_ann_index`). + pub fn build_ann_index(&self, force: bool) -> Result<AnnStats, EmbeddingError> { + self.embedding_store.build_ann_index(force) + } + + /// What the approximate index looks like right now. + pub fn ann_stats(&self) -> AnnStats { + self.embedding_store.ann_stats() + } + + /// Recall@k of the approximate index against the exact scan (see + /// `EmbeddingStore::recall_check`). + pub fn recall_check<'q, I, Q>( + &self, + queries: I, + k: usize, + node_type: Option<&str>, + threshold: f32, + ) -> Result<RecallReport, EmbeddingError> + where + I: IntoIterator<Item = Q>, + Q: Into<Query<'q>>, + { + self.embedding_store + .recall_check(queries, k, node_type, threshold) + } + /// Install an encoder, replacing any current one. Vectors already stored /// keep the model that made them; see `stale_embeddings`. pub fn set_encoder(&mut self, encoder: Box<dyn EmbeddingEncoder>) { @@ -1684,7 +1775,7 @@ impl SemanticGraph { self.embedding_store.add_batch(items) } - pub fn get_embedding(&self, node_id: &NodeId) -> Option<&EmbeddingRecord> { + pub fn get_embedding(&self, node_id: &NodeId) -> Option<EmbeddingRecord> { self.embedding_store.get(node_id) } @@ -1750,8 +1841,13 @@ impl SemanticGraph { /// Top-k similarity search among the nodes that pass `gate`. /// /// With a gate the store cannot truncate first: the best k *unfiltered* - /// matches may all be filtered out, so every match is scored, filtered, - /// and only then cut to k. + /// matches may all be filtered out, so matches are walked best first and + /// filtered until k have passed. + /// + /// With the store's approximate index the candidates are the probed cells + /// only. If fewer than k of them pass the gate, the probe is widened (4x + /// each step) and the search retried, ending in the full scan, so a + /// selective filter costs time rather than results. fn seed_search( &self, query_vector: &[f32], @@ -1759,27 +1855,38 @@ impl SemanticGraph { node_type: Option<&str>, threshold: f32, gate: Option<&Gate>, + exact: bool, ) -> Result<Vec<SimilarityResult>, EmbeddingError> { - match gate { - None => self.embedding_store.similarity_search_vector( + let store = &self.embedding_store; + let Some(gate) = gate else { + return store.search_vector( query_vector, top_k.unwrap_or(usize::MAX), node_type, threshold, - ), - Some(gate) => { - let mut results = self.embedding_store.similarity_search_vector( - query_vector, - usize::MAX, - node_type, - threshold, - )?; - results.retain(|r| gate.passes(&r.node_id)); - if let Some(k) = top_k { - results.truncate(k); + exact, + ); + }; + if top_k == Some(0) { + return Ok(Vec::new()); + } + let mut scale = (store.ann().is_some() && !exact && top_k.is_some()).then_some(1usize); + loop { + let (hits, approximate) = + store.ranked_hits(query_vector, node_type, threshold, scale)?; + let mut results = Vec::new(); + for hit in hits { + if gate.passes(&hit.node_id) { + results.push(hit); + if top_k.is_some_and(|k| results.len() >= k) { + break; + } } - Ok(results) } + if !approximate || top_k.is_some_and(|k| results.len() >= k) { + return Ok(results); + } + scale = scale.map(|s| s.saturating_mul(4)); } } @@ -1802,6 +1909,7 @@ impl SemanticGraph { options.node_type, options.threshold, gate.as_ref(), + options.exact, ) } @@ -1817,12 +1925,12 @@ impl SemanticGraph { node_type: Option<&str>, threshold: f32, ) -> Result<Vec<SimilarityResult>, EmbeddingError> { - let record = match self.embedding_store.get(node_id) { - Some(record) => record, + let vector = match self.embedding_store.get_vector(node_id) { + Some(vector) => vector, None => return Ok(vec![]), }; let mut results = self.embedding_store.similarity_search_vector( - &record.vector, + vector, top_k + 1, node_type, threshold, @@ -1887,6 +1995,9 @@ impl SemanticGraph { None, hops.threshold, gate, + // Blending scores every node anyway, so seeds come from the same + // exact scan rather than from the approximate index. + hops.blend > 0.0, )?; if seeds.is_empty() { return Ok(vec![]); @@ -2180,6 +2291,7 @@ impl SemanticGraph { None, options.threshold, gate.as_ref(), + false, )?; let mut best: Option<SemanticSearchResult> = None; @@ -2442,7 +2554,7 @@ impl SemanticGraph { sub.embedding_store.add_with_model( id.clone(), &record.text, - record.vector.clone(), + record.vector, &record.model, )?; } @@ -2596,7 +2708,31 @@ impl SemanticGraph { format: Option<&str>, embeddings: bool, ) -> Result<(), EmbeddingError> { - let path = path.as_ref(); + self.save_impl(path.as_ref(), format, embeddings, false) + } + + /// `save`, and also a binary snapshot of the embeddings at + /// `<path>.embeddings.snap` (see [`snapshot_path_for`]), written after and + /// in addition to the JSON sidecar, which stays the portable copy. `load` + /// uses the snapshot when it provably belongs to the graph and sidecar on + /// disk, and the JSON otherwise. If the snapshot cannot be written the + /// graph and sidecar are saved anyway, a stale snapshot is removed, and the + /// error is returned. + pub fn save_with_snapshot( + &self, + path: impl AsRef<Path>, + format: Option<&str>, + ) -> Result<(), EmbeddingError> { + self.save_impl(path.as_ref(), format, true, true) + } + + fn save_impl( + &self, + path: &Path, + format: Option<&str>, + embeddings: bool, + snapshot: bool, + ) -> Result<(), EmbeddingError> { let isonl = Self::is_isonl(path, format); let text = if isonl { self.graph.to_isonl() @@ -2608,8 +2744,26 @@ impl SemanticGraph { if embeddings { let (reread, int_ids) = digest::parse_graph_text(&text, isonl, Self::graph_name(path))?; let digest = graph_digest_with_int_ids(&reread, &int_ids); + let sidecar = Self::embeddings_path_for(path); + let snap = snapshot_path_for(path); + if !snapshot { + // the sidecar is about to change: a snapshot of the old one is stale + let _ = std::fs::remove_file(&snap); + } self.embedding_store - .save_embeddings_with(Self::embeddings_path_for(path), Some(&digest))?; + .save_embeddings_with(&sidecar, Some(&digest))?; + if snapshot { + let identity = SnapshotIdentity { + graph_digest: Some(digest), + sidecar: Some(SidecarId::of_file(&sidecar).map_err(|e| { + EmbeddingError::Io(format!("reading {}: {e}", sidecar.display())) + })?), + }; + if let Err(e) = self.embedding_store.save_snapshot(&snap, &identity) { + let _ = std::fs::remove_file(&snap); + return Err(e); + } + } } Ok(()) } @@ -2634,6 +2788,30 @@ impl SemanticGraph { encoder: Box<dyn EmbeddingEncoder>, options: &LoadOptions, ) -> Result<Self, EmbeddingError> { + Self::load_with_report(path, encoder, options).map(|(graph, _)| graph) + } + + /// `load_with`, also saying where the embeddings came from. + /// + /// If a binary snapshot (`<path>.embeddings.snap`, written by + /// [`save_with_snapshot`](Self::save_with_snapshot)) sits beside the + /// sidecar, it is used when its header names this graph's digest and the + /// sidecar file as it is now, and its checksum verifies; the result is + /// identical to loading the JSON. Otherwise the JSON sidecar is read, + /// exactly as before, and `LoadReport::snapshot_skipped` says why the + /// snapshot was not used. + pub fn load_with_report( + path: impl AsRef<Path>, + encoder: Box<dyn EmbeddingEncoder>, + options: &LoadOptions, + ) -> Result<(Self, LoadReport), EmbeddingError> { + let started = std::time::Instant::now(); + let report = |source, snapshot_skipped, embeddings: usize| LoadReport { + source, + snapshot_skipped, + embeddings, + seconds: started.elapsed().as_secs_f64(), + }; let path = path.as_ref(); let isonl = Self::is_isonl(path, options.format.as_deref()); let text = std::fs::read_to_string(path) @@ -2665,11 +2843,28 @@ impl SemanticGraph { }; if !options.embeddings { - return Ok(loaded); + return Ok((loaded, report(LoadSource::NoEmbeddings, None, 0))); } let sidecar = Self::embeddings_path_for(path); if !sidecar.is_file() { - return Ok(loaded); + return Ok((loaded, report(LoadSource::NoEmbeddings, None, 0))); + } + + let snap = snapshot_path_for(path); + let mut snapshot_skipped = None; + if snap.is_file() { + let int_ids = digest::int_ids_of_text(&text, isonl)?; + let actual = graph_digest_with_int_ids(&loaded.graph, &int_ids); + match loaded + .embedding_store + .adopt_snapshot_for(&snap, &sidecar, &actual) + { + Ok(info) => { + loaded.embedding_store.warn_on_version_skew(); + return Ok((loaded, report(LoadSource::Snapshot, None, info.count))); + } + Err(reason) => snapshot_skipped = Some(reason), + } } let json = std::fs::read_to_string(&sidecar) @@ -2741,23 +2936,11 @@ impl SemanticGraph { loaded.embedding_store.warn(problem); } loaded.embedding_store.import_embeddings(&payload, false)?; - let stored_version = loaded - .embedding_store - .last_import_meta() - .encoder_version - .clone(); - let current_version = loaded.embedding_store.version(); - if let (Some(stored), Some(current)) = (stored_version, current_version) { - if stored != current { - loaded.embedding_store.warn(format!( - "embeddings were made by encoder version '{stored}', but the \ - encoder in use is version '{current}'" - )); - } - } + loaded.embedding_store.warn_on_version_skew(); } } - Ok(loaded) + let restored = loaded.embedding_store.count(); + Ok((loaded, report(LoadSource::Json, snapshot_skipped, restored))) } /// Embed all nodes in the graph, in one batched encoder call. diff --git a/isongraph-vector-rs/src/pool.rs b/isongraph-vector-rs/src/pool.rs new file mode 100644 index 0000000..a9f7224 --- /dev/null +++ b/isongraph-vector-rs/src/pool.rs @@ -0,0 +1,335 @@ +//! A small persistent thread pool for searches, folds and index builds. +//! +//! The previous code spawned fresh threads (`std::thread::scope`) for every +//! parallel scan. A [`SearchPool`] keeps its workers alive and parked, so a +//! parallel step costs a wake-up instead of a thread creation. +//! +//! The one primitive is [`SearchPool::run`]: "run this closure on up to `n` +//! participants and return when all of them are done". The calling thread is +//! always participant 0, so a call makes progress even when every worker is busy +//! serving another caller (no deadlock, no queueing behind a long job), and many +//! callers can share one pool. The closure is *work-stealing style*: every +//! participant loops claiming units of work from a counter the closure owns, so +//! the result does not depend on how many participants actually joined. +//! +//! No external dependency (rayon would do the same job; this is about 100 lines +//! and easy to port). It contains the crate's only lifetime-erasing `unsafe`, +//! argued at the `SAFETY` comments. + +use std::collections::VecDeque; +use std::panic::{catch_unwind, resume_unwind, AssertUnwindSafe}; +use std::sync::{Arc, Condvar, Mutex, MutexGuard, OnceLock}; +use std::thread::JoinHandle; + +type Work<'a> = dyn Fn(usize) + Sync + 'a; + +/// One `run` call: the closure (lifetime erased) and who is executing it. +struct Job { + /// Points at the caller's closure. Only dereferenced by a thread that + /// registered itself in `state.running` while `state.open` was true; the + /// caller does not return before `running` is back to zero. + work: *const Work<'static>, + state: Mutex<JobState>, + idle: Condvar, +} + +struct JobState { + /// Helpers may still join. + open: bool, + /// Participants currently inside the closure (helpers only). + running: usize, + /// Participant numbers handed out so far (0 is the caller). + started: usize, + panic: Option<Box<dyn std::any::Any + Send>>, +} + +// SAFETY: `work` is a pointer to a `Sync` closure and is only used under the +// protocol described on `Job::work`; everything else is Mutex/Condvar. +unsafe impl Send for Job {} +unsafe impl Sync for Job {} + +struct Queue { + tickets: VecDeque<Arc<Job>>, + shutdown: bool, +} + +struct Shared { + queue: Mutex<Queue>, + wake: Condvar, +} + +fn lock<T>(m: &Mutex<T>) -> MutexGuard<'_, T> { + // A panic inside a participant is caught and re-raised by `run`; the + // bookkeeping itself never panics, so a poisoned lock is still consistent. + m.lock().unwrap_or_else(|p| p.into_inner()) +} + +/// A fixed set of parked worker threads shared by the searches of any number of +/// stores. See the module docs. +pub struct SearchPool { + shared: Arc<Shared>, + workers: Vec<JoinHandle<()>>, + threads: usize, +} + +impl SearchPool { + /// A pool that can run `threads` participants at once, the caller of + /// [`run`](Self::run) included, so it starts `threads - 1` workers. `1` (or + /// `0`) starts none and every `run` is a plain call. + pub fn new(threads: usize) -> Self { + let threads = threads.max(1); + let shared = Arc::new(Shared { + queue: Mutex::new(Queue { + tickets: VecDeque::new(), + shutdown: false, + }), + wake: Condvar::new(), + }); + let workers = (1..threads) + .map(|i| { + let shared = shared.clone(); + std::thread::Builder::new() + .name(format!("isongraph-search-{i}")) + .spawn(move || worker_loop(&shared)) + .expect("could not start a search worker thread") + }) + .collect(); + Self { + shared, + workers, + threads, + } + } + + /// The process-wide pool every store uses unless given its own. Created on + /// first use with `ISONGRAPH_VECTOR_THREADS` threads, or one per logical + /// CPU when that is unset. + pub fn global() -> Arc<SearchPool> { + static GLOBAL: OnceLock<Arc<SearchPool>> = OnceLock::new(); + GLOBAL + .get_or_init(|| Arc::new(SearchPool::new(Self::default_threads()))) + .clone() + } + + /// The size the shared pool has (or will have): `ISONGRAPH_VECTOR_THREADS` + /// if set to a positive number, else one per logical CPU. Cheap; does not + /// start the pool. + pub fn default_threads() -> usize { + static N: OnceLock<usize> = OnceLock::new(); + *N.get_or_init(|| { + std::env::var("ISONGRAPH_VECTOR_THREADS") + .ok() + .and_then(|v| v.trim().parse::<usize>().ok()) + .filter(|&n| n > 0) + .unwrap_or_else(|| { + std::thread::available_parallelism() + .map(|n| n.get()) + .unwrap_or(1) + }) + }) + } + + /// How many participants one `run` can have at most. + pub fn threads(&self) -> usize { + self.threads + } + + /// Run `work(i)` on up to `participants` threads (the caller is `i == 0`; + /// helpers get 1, 2, ...) and return when every one that started is done. + /// + /// Helpers join only if a worker is free before the caller finishes, so + /// `work` must be able to complete the whole job on a single participant + /// (claim work units from a shared counter). A panic in any participant is + /// re-raised here after all of them have stopped. + pub fn run(&self, participants: usize, work: &(dyn Fn(usize) + Sync)) { + let helpers = participants.min(self.threads).saturating_sub(1); + if helpers == 0 { + work(0); + return; + } + // SAFETY: only the lifetime is erased. `work` outlives every use of the + // pointer because this function does not return (or unwind) until + // `state.open` is false and `state.running` is zero, and a helper only + // dereferences the pointer after incrementing `running` while `open`. + let erased: *const Work<'static> = + unsafe { std::mem::transmute::<*const Work<'_>, *const Work<'static>>(work) }; + let job = Arc::new(Job { + work: erased, + state: Mutex::new(JobState { + open: true, + running: 0, + started: 1, + panic: None, + }), + idle: Condvar::new(), + }); + { + let mut q = lock(&self.shared.queue); + for _ in 0..helpers { + q.tickets.push_back(job.clone()); + } + } + if helpers == 1 { + self.shared.wake.notify_one(); + } else { + self.shared.wake.notify_all(); + } + + let mine = catch_unwind(AssertUnwindSafe(|| work(0))); + + let mut state = lock(&job.state); + state.open = false; + while state.running > 0 { + state = job.idle.wait(state).unwrap_or_else(|p| p.into_inner()); + } + let theirs = state.panic.take(); + drop(state); + if let Err(payload) = mine { + resume_unwind(payload); + } + if let Some(payload) = theirs { + resume_unwind(payload); + } + } +} + +fn worker_loop(shared: &Shared) { + loop { + let job = { + let mut q = lock(&shared.queue); + loop { + if let Some(job) = q.tickets.pop_front() { + break job; + } + if q.shutdown { + return; + } + q = shared.wake.wait(q).unwrap_or_else(|p| p.into_inner()); + } + }; + // Join the job only if its caller has not already finished. + let id = { + let mut s = lock(&job.state); + if !s.open { + continue; + } + s.running += 1; + s.started += 1; + s.started - 1 + }; + // SAFETY: `running` was incremented while `open`, so the caller is + // still waiting in `run` and the closure behind `work` is alive. + let outcome = catch_unwind(AssertUnwindSafe(|| unsafe { (*job.work)(id) })); + let mut s = lock(&job.state); + if let Err(payload) = outcome { + s.panic.get_or_insert(payload); + } + s.running -= 1; + if s.running == 0 { + job.idle.notify_all(); + } + } +} + +impl Drop for SearchPool { + fn drop(&mut self) { + lock(&self.shared.queue).shutdown = true; + self.shared.wake.notify_all(); + for worker in self.workers.drain(..) { + let _ = worker.join(); + } + } +} + +impl std::fmt::Debug for SearchPool { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("SearchPool") + .field("threads", &self.threads) + .finish() + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::sync::atomic::{AtomicUsize, Ordering}; + + #[test] + fn every_unit_of_work_runs_exactly_once_whoever_joins() { + let pool = SearchPool::new(6); + for participants in [1usize, 2, 6, 64] { + let next = AtomicUsize::new(0); + let hits: Vec<AtomicUsize> = (0..1000).map(|_| AtomicUsize::new(0)).collect(); + pool.run(participants, &|_id| loop { + let i = next.fetch_add(1, Ordering::Relaxed); + if i >= hits.len() { + break; + } + hits[i].fetch_add(1, Ordering::Relaxed); + }); + assert!(hits.iter().all(|h| h.load(Ordering::Relaxed) == 1)); + } + } + + #[test] + fn many_callers_share_one_pool_without_deadlock() { + let pool = Arc::new(SearchPool::new(4)); + std::thread::scope(|s| { + for t in 0..12 { + let pool = pool.clone(); + s.spawn(move || { + for round in 0..50 { + let sum = AtomicUsize::new(0); + let next = AtomicUsize::new(0); + pool.run(4, &|_| loop { + let i = next.fetch_add(1, Ordering::Relaxed); + if i >= 100 { + break; + } + sum.fetch_add(i, Ordering::Relaxed); + }); + assert_eq!( + sum.load(Ordering::Relaxed), + 4950, + "caller {t} round {round}" + ); + } + }); + } + }); + } + + #[test] + fn a_panicking_participant_is_reraised_and_the_pool_survives() { + let pool = SearchPool::new(4); + let next = AtomicUsize::new(0); + let result = catch_unwind(AssertUnwindSafe(|| { + pool.run(4, &|_| loop { + let i = next.fetch_add(1, Ordering::Relaxed); + if i >= 200 { + break; + } + if i == 77 { + panic!("boom"); + } + }); + })); + assert!(result.is_err()); + let ok = AtomicUsize::new(0); + pool.run(4, &|_| { + ok.fetch_add(1, Ordering::Relaxed); + }); + assert!(ok.load(Ordering::Relaxed) >= 1); + } + + #[test] + fn a_single_thread_pool_is_a_plain_call() { + let pool = SearchPool::new(1); + let ran = AtomicUsize::new(0); + pool.run(8, &|id| { + assert_eq!(id, 0); + ran.fetch_add(1, Ordering::Relaxed); + }); + assert_eq!(ran.load(Ordering::Relaxed), 1); + } +} diff --git a/isongraph-vector-rs/src/quant.rs b/isongraph-vector-rs/src/quant.rs new file mode 100644 index 0000000..02d4993 --- /dev/null +++ b/isongraph-vector-rs/src/quant.rs @@ -0,0 +1,1290 @@ +//! Opt-in int8 scalar-quantised scan with an exact f32 rerank. +//! +//! This is a child module of `ann` (it reuses the kernels, `TopK`, `Quad` and +//! the pool wrapper without widening their visibility). `DESIGN.md` section 11 +//! is the specification. +//! +//! # What is stored +//! +//! With [`ScanPrecision::Int8`] the table keeps, beside every f32 row, an int8 +//! copy of it and a per-row scale: `scale = max|x| / 127`, `q = round(x / +//! scale)` clamped to +-127 (symmetric, no zero point; an all-zero or +//! non-finite row has scale 0 and codes 0). The copy lives in chunks that +//! mirror the f32 matrix chunk for chunk, so a write, an update, a delete and a +//! compaction maintain it in the same place and at the same moment as the f32 +//! row: it can never be stale. Cost: `dim` bytes + 4 per row = 0.25x the f32 +//! matrix (+ 4 bytes/row). +//! +//! # How a search uses it +//! +//! 1. The query is quantised the same way. +//! 2. The scan (full scans and IVF probed cells alike) scores rows with an +//! **integer** dot product `sum(q_i * r_i)` (exact in i32, so it does not +//! depend on summation order, lane count, kernel or thread) and turns it +//! into an approximate cosine `dot * row_scale * query_scale / (|row| * +//! |query|)` using the exact f32 norms (a tombstone's NaN norm still drops +//! it). +//! 3. The best `k * rerank_factor` rows by approximate score are kept. +//! 4. Those rows are re-scored with the exact f32 kernel (the same one the +//! f32 scan uses, same summation order) and the best `k` of them are +//! returned. **Returned scores are exact cosines**; only the choice of +//! candidates is approximate, so a result differs from the f32 scan only by +//! a true top-k row that ranked below `k * rerank_factor` by approximate +//! score. +//! +//! Because step 2 is integer arithmetic and step 3 selects by the total order +//! (score, slot), the candidate set depends on neither threading, nor tiling, +//! nor batching: **a batch answer is bit-identical to the one-query answer**. +//! +//! When `k * rerank_factor` is not smaller than the number of rows that would +//! be scored (a huge `k`, a small store, the graph's unbounded filtered search) +//! the int8 pass could not skip anything, so the f32 path answers. + +#[cfg(target_arch = "x86_64")] +use super::has_avx2_fma; // named by the `dispatch!` expansions below +use super::{ + cosine, Cells, Hit, Ivf, Par, Quad, QueryVec, TopK, BATCH_TILE, BLOCK_ROWS, + PAR_MIN_CELL_FLOATS, PAR_MIN_FLOATS, SCAN_AUTO_THREADS, +}; +use crate::table::{Rows, VectorTable, CHUNK_MASK, CHUNK_ROWS, CHUNK_SHIFT}; +use crate::{AnnConfig, EmbeddingError, EmbeddingStore}; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::Mutex; +use std::time::Instant; + +/// The rerank factor `ScanPrecision::int8()` uses. +pub const DEFAULT_RERANK_FACTOR: usize = 4; + +/// First-pass candidates are kept down to `threshold - THRESHOLD_SLACK`, so a +/// row whose exact score passes the threshold is not lost to quantisation noise +/// (the noise is about 0.01 for unit vectors at 384-d; the exact threshold is +/// applied at the rerank). +const THRESHOLD_SLACK: f32 = 0.05; + +/// How the scan scores rows. Set with [`EmbeddingStore::set_scan_precision`]. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Default)] +pub enum ScanPrecision { + /// Score every scanned row with the exact f32 kernel (the default). + #[default] + F32, + /// First-pass scan on a per-row int8 copy, then re-score the best + /// `k * rerank_factor` rows in f32. Adds 0.25x the raw vector memory. + Int8 { + /// Candidates kept per requested result (>= 1). + rerank_factor: usize, + }, +} + +impl ScanPrecision { + /// `Int8` with [`DEFAULT_RERANK_FACTOR`]. + pub fn int8() -> Self { + ScanPrecision::Int8 { + rerank_factor: DEFAULT_RERANK_FACTOR, + } + } +} + +// ============================================================================= +// Quantisation +// ============================================================================= + +/// Quantise `x` into `out` (same length) and return the scale. Deterministic: +/// the table's copy and a query use this one function. +pub(crate) fn quantize(x: &[f32], out: &mut [i8]) -> f32 { + debug_assert_eq!(x.len(), out.len()); + let mut m = 0.0f32; + for &v in x { + m = m.max(v.abs()); // f32::max ignores a NaN operand + } + if m.is_nan() || m <= 0.0 || !m.is_finite() { + out.fill(0); + return 0.0; + } + let inv = 127.0 / m; + for (o, &v) in out.iter_mut().zip(x) { + *o = (v * inv).round().clamp(-127.0, 127.0) as i8; + } + m / 127.0 +} + +/// A quantised query: signed codes, their absolute values (the unsigned +/// operand of the AVX2 kernel) and the scale. +pub(crate) struct QQuery { + q: Vec<i8>, + #[cfg_attr(not(target_arch = "x86_64"), allow(dead_code))] + abs: Vec<i8>, + scale: f32, +} + +impl QQuery { + fn new(v: &[f32]) -> Self { + let mut q = vec![0i8; v.len()]; + let scale = quantize(v, &mut q); + let abs = q.iter().map(|c| c.abs()).collect(); + Self { q, abs, scale } + } +} + +// ============================================================================= +// The int8 copy +// ============================================================================= + +/// Int8 rows and scales, slot for slot with the table's f32 matrix. +pub(crate) struct QuantRows { + chunks: Vec<Vec<i8>>, + scales: Vec<f32>, + pub rerank_factor: usize, +} + +/// A read-only view for scan kernels. +#[derive(Clone, Copy)] +pub(crate) struct QRows<'a> { + chunks: &'a [Vec<i8>], + scales: &'a [f32], + dim: usize, +} + +impl<'a> QRows<'a> { + #[inline(always)] + fn row(&self, slot: usize) -> &'a [i8] { + let chunk: &'a Vec<i8> = &self.chunks[slot >> CHUNK_SHIFT]; + let at = (slot & CHUNK_MASK) * self.dim; + &chunk[at..at + self.dim] + } + + #[inline(always)] + fn run(&self, slot: usize, hi: usize) -> (&'a [i8], usize) { + let count = (CHUNK_ROWS - (slot & CHUNK_MASK)).min(hi - slot); + let at = (slot & CHUNK_MASK) * self.dim; + let chunk: &'a Vec<i8> = &self.chunks[slot >> CHUNK_SHIFT]; + (&chunk[at..at + count * self.dim], count) + } +} + +impl QuantRows { + pub fn new(rerank_factor: usize) -> Self { + Self { + chunks: Vec::new(), + scales: Vec::new(), + rerank_factor, + } + } + + pub fn clear(&mut self) { + self.chunks = Vec::new(); + self.scales = Vec::new(); + } + + fn view(&self, dim: usize) -> QRows<'_> { + QRows { + chunks: &self.chunks, + scales: &self.scales, + dim, + } + } + + /// Quantise every row of the f32 matrix (tombstones included, so slots + /// line up). + pub fn rebuild(&mut self, rows: &Rows, n: usize) { + self.clear(); + for slot in 0..n { + self.push(slot, rows.row(slot)); + } + } + + /// Append the row for `slot` (which must be the next slot). + pub fn push(&mut self, slot: usize, vector: &[f32]) { + debug_assert_eq!(slot, self.scales.len()); + if slot & CHUNK_MASK == 0 { + self.chunks.push(if self.chunks.is_empty() { + Vec::new() + } else { + Vec::with_capacity(CHUNK_ROWS * vector.len()) + }); + } + let chunk = self.chunks.last_mut().expect("a chunk exists"); + let at = chunk.len(); + chunk.resize(at + vector.len(), 0); + let scale = quantize(vector, &mut chunk[at..]); + self.scales.push(scale); + } + + /// Overwrite the row for an existing `slot`. + pub fn set(&mut self, slot: usize, vector: &[f32]) { + let dim = vector.len(); + let at = (slot & CHUNK_MASK) * dim; + self.scales[slot] = quantize(vector, &mut self.chunks[slot >> CHUNK_SHIFT][at..at + dim]); + } + + /// Follow the table's order-preserving compaction (`live` = old slot of + /// every survivor). + pub fn compact(&mut self, live: &[u32], dim: usize) { + for (new, &old) in live.iter().enumerate() { + let old = old as usize; + if new == old { + continue; + } + let (oc, oa) = (old >> CHUNK_SHIFT, (old & CHUNK_MASK) * dim); + let (nc, na) = (new >> CHUNK_SHIFT, (new & CHUNK_MASK) * dim); + if oc == nc { + self.chunks[oc].copy_within(oa..oa + dim, na); + } else { + let (head, tail) = self.chunks.split_at_mut(oc); + head[nc][na..na + dim].copy_from_slice(&tail[0][oa..oa + dim]); + } + self.scales[new] = self.scales[old]; + } + let kept = live.len(); + self.scales.truncate(kept); + let chunks = kept.div_ceil(CHUNK_ROWS); + self.chunks.truncate(chunks); + if let Some(last) = self.chunks.last_mut() { + last.truncate((kept - (chunks - 1) * CHUNK_ROWS) * dim); + } + } + + pub fn bytes(&self) -> usize { + self.chunks.iter().map(|c| c.capacity()).sum::<usize>() + self.scales.capacity() * 4 + } + + /// The copy agrees with the f32 matrix: same shape, and every row (dead + /// ones too) is exactly what `quantize` makes of its f32 row. + pub fn check(&self, rows: &Rows, n: usize) -> Result<(), String> { + if self.scales.len() != n { + return Err(format!( + "int8 copy has {} scales for {n} rows", + self.scales.len() + )); + } + if self.chunks.len() != n.div_ceil(CHUNK_ROWS) { + return Err(format!( + "int8 copy has {} chunks for {n} rows", + self.chunks.len() + )); + } + let total: usize = self.chunks.iter().map(|c| c.len()).sum(); + if total != n * rows.dim { + return Err(format!( + "int8 copy holds {total} codes, expected {}", + n * rows.dim + )); + } + let view = self.view(rows.dim); + let mut fresh = vec![0i8; rows.dim]; + for (slot, &stored) in self.scales.iter().enumerate().take(n) { + let scale = quantize(rows.row(slot), &mut fresh); + if scale.to_bits() != stored.to_bits() || view.row(slot) != &fresh[..] { + return Err(format!( + "int8 row {slot} is not the quantisation of its f32 row" + )); + } + } + Ok(()) + } +} + +// ============================================================================= +// Kernels: integer dot products +// ============================================================================= + +#[cfg(target_arch = "x86_64")] +mod x86q { + use std::arch::x86_64::*; + + /// `R` int8 rows against one query: `sum(q_i * r_i)` per row, exact. + /// + /// `sign(r, q) * |q|` is `r * q` with the unsigned operand `maddubs` + /// wants; a pair sum is at most 2 * 127 * 127 < 2^15, so the 16-bit step + /// cannot saturate (codes are clamped to +-127, never -128). + /// + /// # Safety + /// The CPU must support AVX2; `q`, `abs` and every row must be `dim` long. + #[inline] + #[target_feature(enable = "avx2")] + pub unsafe fn dots<const R: usize>( + q: *const i8, + abs: *const i8, + rows: [*const i8; R], + dim: usize, + ) -> [i32; R] { + let ones = _mm256_set1_epi16(1); + let mut acc = [_mm256_setzero_si256(); R]; + let body = dim & !31; + let mut i = 0; + while i < body { + let qv = _mm256_loadu_si256(q.add(i) as *const __m256i); + let av = _mm256_loadu_si256(abs.add(i) as *const __m256i); + for j in 0..R { + let rv = _mm256_loadu_si256(rows[j].add(i) as *const __m256i); + let pairs = _mm256_maddubs_epi16(av, _mm256_sign_epi8(rv, qv)); + acc[j] = _mm256_add_epi32(acc[j], _mm256_madd_epi16(pairs, ones)); + } + i += 32; + } + let mut out = [0i32; R]; + for j in 0..R { + let s = _mm_add_epi32( + _mm256_castsi256_si128(acc[j]), + _mm256_extracti128_si256(acc[j], 1), + ); + let s = _mm_add_epi32(s, _mm_shuffle_epi32(s, 0b01_00_11_10)); + let s = _mm_add_epi32(s, _mm_shuffle_epi32(s, 0b10_11_00_01)); + let mut sum = _mm_cvtsi128_si32(s); + let mut t = body; + while t < dim { + sum += *q.add(t) as i32 * *rows[j].add(t) as i32; + t += 1; + } + out[j] = sum; + } + out + } +} + +/// The portable integer dot (also the specification the AVX2 kernel is tested +/// against). +#[inline(always)] +pub(crate) fn dot_i8_portable(q: &[i8], r: &[i8]) -> i32 { + q.iter().zip(r).map(|(a, b)| *a as i32 * *b as i32).sum() +} + +/// `R` rows against one quantised query. `F` selects the AVX2 kernel; the +/// `true` instantiation is only reached through a `dispatch!` wrapper that +/// checked the CPU. +#[inline(always)] +fn dots<const R: usize, const F: bool>(qq: &QQuery, rows: [&[i8]; R]) -> [i32; R] { + #[cfg(target_arch = "x86_64")] + if F { + let dim = qq.q.len(); + // the kernel reads `dim` bytes from every operand: make that a fact + assert!(qq.abs.len() == dim && rows.iter().all(|r| r.len() == dim)); + // SAFETY: `F` is true only in the instantiation reached through a + // `dispatch!` wrapper, which has checked for AVX2 (and FMA); the + // assert above guarantees every pointer has `dim` readable bytes. + return unsafe { + x86q::dots::<R>( + qq.q.as_ptr(), + qq.abs.as_ptr(), + std::array::from_fn(|j| rows[j].as_ptr()), + dim, + ) + }; + } + std::array::from_fn(|j| dot_i8_portable(&qq.q, rows[j])) +} + +/// Int8 dot of two vectors by whichever kernel the CPU has (for tests). +#[cfg(test)] +pub(crate) fn dot_i8(q: &[i8], r: &[i8]) -> i32 { + let qq = QQuery { + q: q.to_vec(), + abs: q.iter().map(|c| c.abs()).collect(), + scale: 1.0, + }; + dot_i8_dispatch(&qq, r) +} + +#[cfg(test)] +dispatch! { + fn dot_i8_dispatch(qq: &QQuery, r: &[i8]) -> i32 = dot_one_impl +} + +#[cfg(test)] +#[inline(always)] +fn dot_one_impl<const F: bool>(qq: &QQuery, r: &[i8]) -> i32 { + dots::<1, F>(qq, [r])[0] +} + +/// Approximate cosine of a row from its integer dot. +#[inline(always)] +fn approx(d: i32, row_scale: f32, q_scale: f32, row_norm: f32, q_norm: f32) -> f32 { + cosine(d as f32 * row_scale * q_scale, row_norm, q_norm) +} + +/// Everything a first-pass kernel reads. +#[derive(Clone, Copy)] +struct View<'a> { + rows: Rows<'a>, + q: QRows<'a>, +} + +/// Four-at-a-time batching of int8 rows (the f32 `Quad`'s counterpart). +struct Quad8<'a> { + slots: [usize; 4], + rows: [&'a [i8]; 4], + m: usize, +} + +impl<'a> Quad8<'a> { + #[inline(always)] + fn new() -> Self { + Self { + slots: [0; 4], + rows: [&[]; 4], + m: 0, + } + } + + #[inline(always)] + fn push<const F: bool>( + &mut self, + v: &View, + qq: &QQuery, + q: &QueryVec, + slot: usize, + row: &'a [i8], + top: &mut TopK, + ) { + self.slots[self.m] = slot; + self.rows[self.m] = row; + self.m += 1; + if self.m == 4 { + self.m = 0; + let d = dots::<4, F>(qq, self.rows); + for (&s, &dot) in self.slots.iter().zip(&d) { + top.offer( + approx(dot, v.q.scales[s], qq.scale, v.rows.norms[s], q.norm), + s as u32, + ); + } + } + } + + #[inline(always)] + fn finish<const F: bool>(&mut self, v: &View, qq: &QQuery, q: &QueryVec, top: &mut TopK) { + for i in 0..self.m { + let s = self.slots[i]; + let d = dots::<1, F>(qq, [self.rows[i]])[0]; + top.offer( + approx(d, v.q.scales[s], qq.scale, v.rows.norms[s], q.norm), + s as u32, + ); + } + self.m = 0; + } +} + +/// First pass over rows `lo..hi`. +#[inline(always)] +fn scan_range_impl<const F: bool>( + v: View, + qq: &QQuery, + q: &QueryVec, + code: Option<u32>, + lo: usize, + hi: usize, + top: &mut TopK, +) { + let mut quad = Quad8::new(); + let dim = v.q.dim; + let mut at = lo; + while at < hi { + let (flat, count) = v.q.run(at, hi); + for i in 0..count { + let slot = at + i; + if let Some(code) = code { + if v.rows.codes[slot] != code { + continue; + } + } + quad.push::<F>(&v, qq, q, slot, &flat[i * dim..(i + 1) * dim], top); + } + at += count; + } + quad.finish::<F>(&v, qq, q, top); +} + +dispatch! { + fn scan_range(v: View, qq: &QQuery, q: &QueryVec, code: Option<u32>, lo: usize, hi: usize, top: &mut TopK) = scan_range_impl +} + +/// First pass over the rows of the `chosen` cells (and the pending rows). +#[inline(always)] +#[allow(clippy::too_many_arguments)] +fn scan_cells_impl<const F: bool>( + v: View, + cells: Cells, + chosen: &[u32], + with_pending: bool, + qq: &QQuery, + q: &QueryVec, + code: Option<u32>, + top: &mut TopK, +) { + let mut quad = Quad8::new(); + let mut visit = |slot: u32| { + let slot = slot as usize; + if let Some(code) = code { + if v.rows.codes[slot] != code { + return; + } + } + quad.push::<F>(&v, qq, q, slot, v.q.row(slot), top); + }; + for &c in chosen { + for &slot in &cells.lists[c as usize] { + visit(slot); + } + } + if with_pending { + for &slot in cells.pending { + visit(slot); + } + } + quad.finish::<F>(&v, qq, q, top); +} + +dispatch! { + #[allow(clippy::too_many_arguments)] + fn scan_cells(v: View, cells: Cells, chosen: &[u32], with_pending: bool, qq: &QQuery, q: &QueryVec, code: Option<u32>, top: &mut TopK) = scan_cells_impl +} + +/// First pass over an explicit list of slots (a cell's rows). +#[inline(always)] +fn scan_slots_impl<const F: bool>( + v: View, + slots: &[u32], + qq: &QQuery, + q: &QueryVec, + top: &mut TopK, +) { + let mut quad = Quad8::new(); + for &slot in slots { + quad.push::<F>(&v, qq, q, slot as usize, v.q.row(slot as usize), top); + } + quad.finish::<F>(&v, qq, q, top); +} + +dispatch! { + fn scan_slots(v: View, slots: &[u32], qq: &QQuery, q: &QueryVec, top: &mut TopK) = scan_slots_impl +} + +/// Exact f32 scores of `slots`, offered to `top`. +#[inline(always)] +fn rerank_impl<const F: bool>(rows: Rows, q: &QueryVec, slots: &[u32], top: &mut TopK) { + let mut quad = Quad::new(); + let mut sink = |slot: usize, score: f32| top.offer(score, slot as u32); + for &slot in slots { + quad.push::<F>(&rows, q, slot as usize, &mut sink); + } + quad.finish::<F>(&rows, q, &mut sink); +} + +dispatch! { + fn rerank(rows: Rows, q: &QueryVec, slots: &[u32], top: &mut TopK) = rerank_impl +} + +/// The exact top `k` of a first-pass candidate heap. +fn rerank_candidates(rows: Rows, q: &QueryVec, cand: TopK, threshold: f32, k: usize) -> TopK { + let mut slots: Vec<u32> = cand.into_sorted().into_iter().map(|h| h.slot).collect(); + slots.sort_unstable(); // walk the matrix in order + let mut top = TopK::new(k, threshold, slots.len()); + rerank(rows, q, &slots, &mut top); + top +} + +// ============================================================================= +// Searches (called from the hooks in ann.rs) +// ============================================================================= + +/// First-pass scan of a probed set of cells, then the exact rerank. `None` +/// when int8 cannot skip any work here (the caller scores in f32). +#[allow(clippy::too_many_arguments)] +pub(super) fn scan_probed( + table: &VectorTable, + cells: Cells, + chosen: &[u32], + q: &QueryVec, + code: Option<u32>, + threshold: f32, + k: usize, + par: Par, +) -> Option<TopK> { + const CELLS_PER_TASK: usize = 4; + let qr = table.quant()?; + if !q.has_direction() { + return None; + } + let candidates: usize = chosen + .iter() + .map(|&c| cells.lists[c as usize].len()) + .sum::<usize>() + + cells.pending.len(); + let kc = k.saturating_mul(qr.rerank_factor); + if kc >= candidates { + return None; + } + let dim = table.dim; + let view = View { + rows: table.rows(), + q: qr.view(dim), + }; + let qq = QQuery::new(q.v); + let live = table.live(); + let bar = threshold - THRESHOLD_SLACK; + let tasks = chosen.len().div_ceil(CELLS_PER_TASK); + let threads = if par.configured == 0 { + par.workers(tasks, SCAN_AUTO_THREADS) + .min((candidates * dim / PAR_MIN_CELL_FLOATS).max(1)) + } else { + par.workers(tasks, SCAN_AUTO_THREADS) + }; + let mut top = TopK::new(kc, bar, live); + if threads <= 1 { + scan_cells(view, cells, chosen, true, &qq, q, code, &mut top); + } else { + let next = AtomicUsize::new(0); + let parts = Mutex::new(Vec::with_capacity(threads)); + par.run(threads, &|_| { + let mut local = TopK::new(kc, bar, live); + loop { + let t = next.fetch_add(1, Ordering::Relaxed); + if t >= tasks { + break; + } + let lo = t * CELLS_PER_TASK; + let hi = (lo + CELLS_PER_TASK).min(chosen.len()); + scan_cells( + view, + cells, + &chosen[lo..hi], + t == 0, + &qq, + q, + code, + &mut local, + ); + } + parts.lock().unwrap_or_else(|e| e.into_inner()).push(local); + }); + for part in parts.into_inner().unwrap_or_else(|e| e.into_inner()) { + top.merge(part); + } + } + Some(rerank_candidates(view.rows, q, top, threshold, k)) +} + +impl EmbeddingStore { + /// Choose how scans score rows. `Int8` builds an int8 copy of every row + /// now (one pass over the matrix) and keeps it current through every + /// write; `F32` drops it. Searches are unchanged in what they return + /// (exact cosines) but, with `Int8`, a bounded search may miss a true + /// top-`k` row that ranks below `k * rerank_factor` by the int8 score: + /// measure with [`int8_recall_report`](Self::int8_recall_report). + /// + /// With [`AnnMode::Hnsw`](crate::AnnMode::Hnsw) the int8 copy is **not** + /// used by graph search (the graph scores candidates with the f32 + /// kernels); it only speeds the full scans HNSW falls back to (exact + /// override, selective filters). It still costs 0.25x the raw vectors in + /// memory, so leave `F32` on when HNSW is the only search you use. + /// + /// # Errors + /// `InvalidOption` for a rerank factor of 0. + pub fn set_scan_precision(&mut self, precision: ScanPrecision) -> Result<(), EmbeddingError> { + match precision { + ScanPrecision::F32 => self.table.set_quant(None), + ScanPrecision::Int8 { rerank_factor } => { + if rerank_factor == 0 { + return Err(EmbeddingError::InvalidOption( + "int8 rerank_factor must be at least 1".to_string(), + )); + } + self.table.set_quant(Some(rerank_factor)); + } + } + Ok(()) + } + + /// The current scan precision. + pub fn scan_precision(&self) -> ScanPrecision { + match self.table.quant() { + None => ScanPrecision::F32, + Some(q) => ScanPrecision::Int8 { + rerank_factor: q.rerank_factor, + }, + } + } + + /// Bytes held by the int8 copy (0 under `F32`). + pub fn quant_bytes(&self) -> usize { + self.table.quant().map_or(0, QuantRows::bytes) + } + + /// Full scan with the int8 first pass and exact rerank, for a given + /// `factor`. `None` when it cannot apply (no int8 copy, a query with no + /// direction, or `k * factor` not smaller than the live rows). + pub(super) fn q8_exact_hits( + &self, + q: &QueryVec, + code: Option<u32>, + threshold: f32, + k: usize, + factor: usize, + ) -> Option<Vec<Hit>> { + let table = &self.table; + let qr = table.quant()?; + let live = table.live(); + let kc = k.saturating_mul(factor); + if kc >= live || !q.has_direction() { + return None; + } + let (n, dim) = (table.slots(), table.dim); + let view = View { + rows: table.rows(), + q: qr.view(dim), + }; + let qq = QQuery::new(q.v); + let bar = threshold - THRESHOLD_SLACK; + let blocks = n.div_ceil(BLOCK_ROWS); + let par = self.par(); + let threads = if par.configured == 0 { + par.workers(blocks, SCAN_AUTO_THREADS) + .min((n * dim / PAR_MIN_FLOATS).max(1)) + } else { + par.workers(blocks, SCAN_AUTO_THREADS) + }; + let cand = if threads <= 1 { + let mut top = TopK::new(kc, bar, live); + scan_range(view, &qq, q, code, 0, n, &mut top); + top + } else { + let next = AtomicUsize::new(0); + let parts = Mutex::new(Vec::with_capacity(threads)); + par.run(threads, &|_| { + let mut top = TopK::new(kc, bar, live); + loop { + let b = next.fetch_add(1, Ordering::Relaxed); + if b >= blocks { + break; + } + let (lo, hi) = (b * BLOCK_ROWS, ((b + 1) * BLOCK_ROWS).min(n)); + scan_range(view, &qq, q, code, lo, hi, &mut top); + } + parts.lock().unwrap_or_else(|e| e.into_inner()).push(top); + }); + let mut parts = parts.into_inner().unwrap_or_else(|e| e.into_inner()); + let mut merged = parts.pop().expect("at least one participant"); + for part in parts { + merged.merge(part); + } + merged + }; + Some(rerank_candidates(view.rows, q, cand, threshold, k).into_sorted()) + } + + /// The hook `exact_hits` calls first: int8 when it applies. + pub(super) fn quant_exact( + &self, + q: &QueryVec, + code: Option<u32>, + threshold: f32, + k: usize, + ) -> Option<Vec<Hit>> { + let factor = self.table.quant()?.rerank_factor; + self.q8_exact_hits(q, code, threshold, k, factor) + } + + /// The hook `exact_batch_hits` calls first: queries x blocks tasks, a + /// block of int8 rows (<= 512 KB at 512-d) staying in cache across a tile + /// of 16 queries. `None` when int8 cannot apply. + pub(super) fn quant_exact_batch( + &self, + qs: &[QueryVec], + code: Option<u32>, + threshold: f32, + k: usize, + ) -> Option<Vec<Vec<Hit>>> { + let table = &self.table; + let qr = table.quant()?; + let live = table.live(); + let kc = k.saturating_mul(qr.rerank_factor); + if kc >= live || qs.is_empty() || !qs.iter().all(QueryVec::has_direction) { + return None; + } + let (n, dim) = (table.slots(), table.dim); + let view = View { + rows: table.rows(), + q: qr.view(dim), + }; + let qqs: Vec<QQuery> = qs.iter().map(|q| QQuery::new(q.v)).collect(); + let bar = threshold - THRESHOLD_SLACK; + let blocks = n.div_ceil(BLOCK_ROWS); + let tiles = qs.len().div_ceil(BATCH_TILE); + let total = blocks * tiles; + let par = self.par(); + let threads = par.workers(total, usize::MAX); + let next = AtomicUsize::new(0); + let parts: Mutex<Vec<Vec<TopK>>> = Mutex::new(Vec::with_capacity(threads)); + par.run(threads, &|_| { + let mut tops: Vec<TopK> = Vec::new(); + loop { + let t = next.fetch_add(1, Ordering::Relaxed); + if t >= total { + break; + } + if tops.is_empty() { + tops = (0..qs.len()).map(|_| TopK::new(kc, bar, live)).collect(); + } + let (block, tile) = (t / tiles, t % tiles); + let (lo, hi) = (block * BLOCK_ROWS, ((block + 1) * BLOCK_ROWS).min(n)); + let (q_lo, q_hi) = (tile * BATCH_TILE, ((tile + 1) * BATCH_TILE).min(qs.len())); + for i in q_lo..q_hi { + scan_range(view, &qqs[i], &qs[i], code, lo, hi, &mut tops[i]); + } + } + if !tops.is_empty() { + parts.lock().unwrap_or_else(|e| e.into_inner()).push(tops); + } + }); + let mut parts = parts.into_inner().unwrap_or_else(|e| e.into_inner()); + let mut merged = parts.pop()?; + for part in parts { + for (m, p) in merged.iter_mut().zip(part) { + m.merge(p); + } + } + // rerank every query exactly, in parallel + let cands: Vec<Mutex<Option<TopK>>> = + merged.into_iter().map(|t| Mutex::new(Some(t))).collect(); + let next = AtomicUsize::new(0); + let done: Mutex<Vec<(usize, Vec<Hit>)>> = Mutex::new(Vec::with_capacity(qs.len())); + par.run(par.workers(qs.len(), usize::MAX), &|_| { + let mut local = Vec::new(); + loop { + let i = next.fetch_add(1, Ordering::Relaxed); + if i >= qs.len() { + break; + } + let cand = cands[i] + .lock() + .unwrap_or_else(|e| e.into_inner()) + .take() + .expect("each query is reranked once"); + local.push(( + i, + rerank_candidates(view.rows, &qs[i], cand, threshold, k).into_sorted(), + )); + } + done.lock().unwrap_or_else(|e| e.into_inner()).extend(local); + }); + let mut out: Vec<Vec<Hit>> = vec![Vec::new(); qs.len()]; + for (i, hits) in done.into_inner().unwrap_or_else(|e| e.into_inner()) { + out[i] = hits; + } + Some(out) + } + + /// The hook `batch_hits` calls before the f32 cell-major batch. Like it, + /// this works **cell-major**: each probed cell's int8 rows are scored once + /// against every query that probes the cell (the rows stay in cache), into + /// per-query candidate heaps of `k * rerank_factor`; then each query's + /// candidates are re-scored exactly. A query's candidate set is the top + /// `kc` of exactly the rows the single-query probe scans, under the same + /// total order, so the answer is identical to the single-query answer. A + /// query that comes up short, or whose probe is too small for int8 to skip + /// anything, is re-run on the single-query path. Returns whether it handled + /// the batch (even by leaving `answers` empty for the exact scan). + #[allow(clippy::too_many_arguments)] + pub(super) fn quant_approximate_batch( + &self, + cfg: &AnnConfig, + ivf: &Ivf, + qs: &[QueryVec], + code: Option<u32>, + threshold: f32, + top_k: usize, + answers: &mut [Option<Vec<Hit>>], + ) -> bool { + let table = &self.table; + let Some(qr) = table.quant() else { + return false; + }; + let nprobe = super::effective_nprobe(cfg, ivf.nlist); + if nprobe >= ivf.nlist { + return true; // probing everything is the exact scan + } + let kc = top_k.saturating_mul(qr.rerank_factor); + let live = table.live(); + let view = View { + rows: table.rows(), + q: qr.view(table.dim), + }; + let bar = threshold - THRESHOLD_SLACK; + let par = self.par(); + + // 1. the cells each query probes (queries with no direction go to the + // exact scan, as in the f32 batch) + let mut chosen: Vec<Option<Vec<u32>>> = (0..qs.len()).map(|_| None).collect(); + { + let next = AtomicUsize::new(0); + let done: Mutex<Vec<(usize, Vec<u32>)>> = Mutex::new(Vec::new()); + par.run(par.workers(qs.len(), usize::MAX), &|_| { + let mut local = Vec::new(); + loop { + let i = next.fetch_add(1, Ordering::Relaxed); + if i >= qs.len() { + break; + } + if qs[i].has_direction() { + local.push((i, ivf.choose_cells(&qs[i], nprobe))); + } + } + done.lock().unwrap_or_else(|e| e.into_inner()).extend(local); + }); + for (i, cells) in done.into_inner().unwrap_or_else(|e| e.into_inner()) { + chosen[i] = Some(cells); + } + } + // a probe with no more rows than `kc` cannot be thinned by int8: the + // single-query path answers it in f32 + let mut single = Vec::new(); + for (i, cells) in chosen.iter_mut().enumerate() { + if let Some(c) = cells { + let rows: usize = c + .iter() + .map(|&c| ivf.lists[c as usize].len()) + .sum::<usize>() + + ivf.pending.len(); + if kc >= rows { + single.push(i); + *cells = None; + } + } + } + + // 2. invert: the queries of each cell (the extra last entry is the + // pending rows, which every query scores) + let mut by_cell: Vec<Vec<u32>> = vec![Vec::new(); ivf.nlist + 1]; + for (i, cells) in chosen.iter().enumerate() { + if let Some(cells) = cells { + for &c in cells { + by_cell[c as usize].push(i as u32); + } + by_cell[ivf.nlist].push(i as u32); + } + } + let members_of = |c: usize| -> &[u32] { + if c == ivf.nlist { + &ivf.pending + } else { + &ivf.lists[c] + } + }; + let tasks: Vec<usize> = (0..=ivf.nlist) + .filter(|&c| !by_cell[c].is_empty() && !members_of(c).is_empty()) + .collect(); + + // 3. score every probed cell once against all of its queries + let qqs: Vec<Option<QQuery>> = chosen + .iter() + .zip(qs) + .map(|(c, q)| c.as_ref().map(|_| QQuery::new(q.v))) + .collect(); + let next = AtomicUsize::new(0); + let parts: Mutex<Vec<Vec<TopK>>> = Mutex::new(Vec::new()); + par.run(par.workers(tasks.len(), usize::MAX), &|_| { + let mut tops: Vec<TopK> = Vec::new(); + let mut slots: Vec<u32> = Vec::new(); + loop { + let t = next.fetch_add(1, Ordering::Relaxed); + if t >= tasks.len() { + break; + } + if tops.is_empty() { + tops = (0..qs.len()).map(|_| TopK::new(kc, bar, live)).collect(); + } + let c = tasks[t]; + slots.clear(); + slots.extend( + members_of(c) + .iter() + .copied() + .filter(|&s| code.is_none_or(|code| view.rows.codes[s as usize] == code)), + ); + for &i in &by_cell[c] { + let i = i as usize; + let qq = qqs[i] + .as_ref() + .expect("a probing query has a quantised copy"); + scan_slots(view, &slots, qq, &qs[i], &mut tops[i]); + } + } + if !tops.is_empty() { + parts.lock().unwrap_or_else(|e| e.into_inner()).push(tops); + } + }); + let mut parts = parts.into_inner().unwrap_or_else(|e| e.into_inner()); + let mut merged: Vec<TopK> = match parts.pop() { + Some(first) => first, + None => (0..qs.len()).map(|_| TopK::new(kc, bar, live)).collect(), + }; + for part in parts { + for (a, b) in merged.iter_mut().zip(part) { + a.merge(b); + } + } + + // 4. exact rerank; the short ones go back through the single path + for (i, cand) in merged.into_iter().enumerate() { + if chosen[i].is_none() { + continue; + } + let top = rerank_candidates(view.rows, &qs[i], cand, threshold, top_k); + if top.len() >= top_k { + self.counts.approximate.fetch_add(1, Ordering::Relaxed); + answers[i] = Some(top.into_sorted()); + } else { + single.push(i); + } + } + let one_thread = Par { + configured: 1, + pool: &self.pool, + }; + let next = AtomicUsize::new(0); + let redone: Mutex<Vec<(usize, Option<Vec<Hit>>)>> = Mutex::new(Vec::new()); + par.run(par.workers(single.len(), usize::MAX), &|_| { + let mut local = Vec::new(); + loop { + let j = next.fetch_add(1, Ordering::Relaxed); + if j >= single.len() { + break; + } + let i = single[j]; + local.push(( + i, + self.approximate_topk(cfg, ivf, &qs[i], code, threshold, top_k, one_thread), + )); + } + redone + .lock() + .unwrap_or_else(|e| e.into_inner()) + .extend(local); + }); + for (i, hits) in redone.into_inner().unwrap_or_else(|e| e.into_inner()) { + answers[i] = hits; + } + true + } + + /// Recall of the int8 scan against the exact f32 scan, at each of the + /// given rerank factors, over `queries` (use real queries from your own + /// workload; recall depends on the data). + /// + /// Full scans only (an IVF probe adds its own approximation; measure that + /// with [`recall_check`](Self::recall_check)). For each factor the report + /// gives the mean and worst per-query overlap between the int8 top `k` and + /// the f32 top `k`, whether every returned score is bit-identical to the + /// exact cosine of that row, and median per-query times. + /// + /// # Errors + /// `InvalidOption` unless the store is on `ScanPrecision::Int8`, `k` is 0, + /// or a factor is 0; a query of the wrong width. + pub fn int8_recall_report<V: AsRef<[f32]>>( + &self, + queries: &[V], + k: usize, + node_type: Option<&str>, + threshold: f32, + rerank_factors: &[usize], + ) -> Result<Vec<Int8Recall>, EmbeddingError> { + if self.table.quant().is_none() { + return Err(EmbeddingError::InvalidOption( + "scan precision is not Int8; call set_scan_precision first".to_string(), + )); + } + if k == 0 || rerank_factors.contains(&0) { + return Err(EmbeddingError::InvalidOption( + "k and every rerank factor must be positive".to_string(), + )); + } + for q in queries { + self.check_query(q.as_ref())?; + } + let Ok(code) = self.type_filter(node_type) else { + return Ok(Vec::new()); + }; + let qvs: Vec<QueryVec> = queries.iter().map(|q| QueryVec::new(q.as_ref())).collect(); + let live = self.table.live(); + let mut truth: Vec<Vec<Hit>> = Vec::new(); + let mut f32_times = Vec::new(); + for q in &qvs { + let started = Instant::now(); + truth.push(self.exact_hits_f32(q, code, threshold, k)); + f32_times.push(started.elapsed().as_secs_f64()); + } + let f32_ms = 1000.0 * median(f32_times); + let mut out = Vec::new(); + for &factor in rerank_factors { + let (mut recalls, mut times) = (Vec::new(), Vec::new()); + let mut exact = true; + for (q, truth) in qvs.iter().zip(&truth) { + let started = Instant::now(); + let found = self + .q8_exact_hits(q, code, threshold, k, factor) + .unwrap_or_else(|| self.exact_hits_f32(q, code, threshold, k)); + times.push(started.elapsed().as_secs_f64()); + let wanted: std::collections::HashSet<u32> = truth.iter().map(|h| h.slot).collect(); + if !wanted.is_empty() { + let hit = found.iter().filter(|h| wanted.contains(&h.slot)).count(); + recalls.push(hit as f64 / wanted.len() as f64); + } + for h in &found { + let slot = h.slot as usize; + let d = crate::ann::dot_f32(q.v, self.table.row(slot)); + exact &= + cosine(d, self.table.norm(slot), q.norm).to_bits() == h.score.to_bits(); + } + } + out.push(Int8Recall { + rerank_factor: factor, + k, + queries: qvs.len(), + recall: if recalls.is_empty() { + 1.0 + } else { + recalls.iter().sum::<f64>() / recalls.len() as f64 + }, + min_recall: recalls.iter().copied().fold(1.0, f64::min), + scores_exact: exact, + applies: k.saturating_mul(factor) < live, + int8_ms: 1000.0 * median(times), + f32_ms, + }); + } + Ok(out) + } +} + +fn median(mut v: Vec<f64>) -> f64 { + if v.is_empty() { + return 0.0; + } + v.sort_by(f64::total_cmp); + let mid = v.len() / 2; + if v.len() % 2 == 1 { + v[mid] + } else { + 0.5 * (v[mid - 1] + v[mid]) + } +} + +/// Recall of the int8 scan at one rerank factor; see +/// [`EmbeddingStore::int8_recall_report`]. +#[derive(Clone, Debug)] +pub struct Int8Recall { + pub rerank_factor: usize, + pub k: usize, + pub queries: usize, + /// Mean over queries of `|int8 top-k & f32 top-k| / |f32 top-k|`. + pub recall: f64, + /// The worst single query. + pub min_recall: f64, + /// Every returned score was bit-identical to the exact f32 cosine of its + /// row (true by construction; this checks it). + pub scores_exact: bool, + /// `k * rerank_factor` is below the row count, so the int8 pass really + /// ran (otherwise the answer is the f32 scan and recall is 1.0). + pub applies: bool, + /// Median per-query time of the int8 search, milliseconds. + pub int8_ms: f64, + /// Median per-query time of the f32 scan, milliseconds. + pub f32_ms: f64, +} + +#[cfg(test)] +mod tests { + use super::*; + + struct Rng(u64); + impl Rng { + fn next(&mut self) -> u64 { + self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } + } + + #[test] + fn quantize_is_symmetric_and_bounded() { + let mut out = vec![0i8; 4]; + let s = quantize(&[1.0, -0.5, 0.0, 0.25], &mut out); + assert_eq!(out, vec![127, -64, 0, 32]); + assert!((s - 1.0 / 127.0).abs() < 1e-9); + assert_eq!(quantize(&[0.0; 4], &mut out), 0.0); + assert_eq!(out, vec![0; 4]); + assert_eq!(quantize(&[f32::NAN, 1.0, 0.0, 0.0], &mut out), 1.0 / 127.0); + assert_eq!(out[0], 0); + assert_eq!(quantize(&[f32::INFINITY, 1.0, 0.0, 0.0], &mut out), 0.0); + assert!(out.iter().all(|&c| c != i8::MIN)); + } + + #[test] + fn avx2_kernel_matches_the_portable_dot_exactly() { + let mut rng = Rng(7); + for dim in [1usize, 7, 16, 31, 32, 33, 63, 64, 100, 384, 385, 1000] { + for trial in 0..20 { + let mut gen = |extreme: bool| -> Vec<i8> { + (0..dim) + .map(|_| { + if extreme { + if rng.next() & 1 == 0 { + 127 + } else { + -127 + } + } else { + ((rng.next() % 255) as i32 - 127) as i8 + } + }) + .collect() + }; + let extreme = trial % 5 == 0; + let q = gen(extreme); + let rows: Vec<Vec<i8>> = (0..4).map(|_| gen(extreme)).collect(); + let qq = QQuery { + abs: q.iter().map(|c| c.abs()).collect(), + q: q.clone(), + scale: 1.0, + }; + let r4 = [&rows[0][..], &rows[1][..], &rows[2][..], &rows[3][..]]; + let want: Vec<i32> = rows.iter().map(|r| dot_i8_portable(&q, r)).collect(); + assert_eq!(dots::<4, false>(&qq, r4).to_vec(), want); + assert_eq!(dot_i8_dispatch(&qq, &rows[0]), want[0], "dim {dim}"); + assert_eq!(dot_i8(&q, &rows[1]), want[1]); + #[cfg(target_arch = "x86_64")] + if has_avx2_fma() { + #[target_feature(enable = "avx2,fma")] + unsafe fn fast(qq: &QQuery, r4: [&[i8]; 4]) -> [i32; 4] { + dots::<4, true>(qq, r4) + } + // SAFETY: AVX2 and FMA were just detected. + let got = unsafe { fast(&qq, r4) }; + assert_eq!(got.to_vec(), want, "dim {dim} trial {trial}"); + } + } + } + } + + #[test] + #[cfg(target_arch = "x86_64")] + fn avx2_kernel_refuses_a_short_row() { + if !has_avx2_fma() { + return; + } + #[target_feature(enable = "avx2,fma")] + unsafe fn fast(qq: &QQuery, r: [&[i8]; 1]) -> [i32; 1] { + dots::<1, true>(qq, r) + } + let qq = QQuery::new(&[0.5; 64]); + let short = [1i8; 63]; + let result = std::panic::catch_unwind(|| { + // SAFETY: AVX2 detected; the call must panic in the assert before + // any read. + unsafe { fast(&qq, [&short[..]]) } + }); + assert!(result.is_err()); + } +} diff --git a/isongraph-vector-rs/src/shared.rs b/isongraph-vector-rs/src/shared.rs new file mode 100644 index 0000000..d9c7e51 --- /dev/null +++ b/isongraph-vector-rs/src/shared.rs @@ -0,0 +1,101 @@ +//! Readers and a writer on different threads. +//! +//! [`EmbeddingStore`] is already safe to *read* from any number of threads +//! (`&self` searches never lock the data), and its writes need `&mut self`, so +//! a search cannot overlap a write: the borrow checker is the isolation +//! mechanism. [`SharedEmbeddingStore`] is the thin layer that lets different +//! threads hold the two roles at once: a cloneable handle around +//! `Arc<RwLock<EmbeddingStore>>`. +//! +//! * A search holds the read lock for the length of the scan. Any number run +//! at once. +//! * A write holds the write lock for one `put`/`remove`: a row copy and a few +//! index updates, plus (every [`FOLD_BATCH`](crate::FOLD_BATCH) writes) one +//! bounded fold. Encoding a text happens *before* the lock is taken. +//! * A reader therefore sees the store before a write or after it, never in +//! between; a writer waits for the scans in flight (milliseconds) and holds +//! readers off for microseconds. +//! +//! Rejected alternative: copy-on-write snapshots (readers pin an `Arc` of the +//! data, writers publish a new one). With a matrix of gigabytes a writer cannot +//! copy it per write, and chunked structural sharing buys a reader nothing the +//! lock does not already give, at the price of a far harder-to-port design. + +use crate::{EmbeddingError, EmbeddingRecord, EmbeddingStore, NodeId, SimilarityResult}; +use std::sync::{Arc, RwLock, RwLockReadGuard, RwLockWriteGuard}; + +/// A cloneable, thread-safe handle to one [`EmbeddingStore`]. See the module +/// docs for the guarantees. +#[derive(Clone)] +pub struct SharedEmbeddingStore { + inner: Arc<RwLock<EmbeddingStore>>, +} + +impl SharedEmbeddingStore { + pub fn new(store: EmbeddingStore) -> Self { + Self { + inner: Arc::new(RwLock::new(store)), + } + } + + /// Shared access, for calling any `&self` method (several threads at once). + /// A poisoned lock (a thread panicked inside a write, which is a bug) is + /// still handed out rather than turning every later call into a panic. + pub fn read(&self) -> RwLockReadGuard<'_, EmbeddingStore> { + self.inner.read().unwrap_or_else(|e| e.into_inner()) + } + + /// Exclusive access, for any `&mut self` method. Keep it short. + pub fn write(&self) -> RwLockWriteGuard<'_, EmbeddingStore> { + self.inner.write().unwrap_or_else(|e| e.into_inner()) + } + + /// Add or replace one embedding. With no vector the text is encoded first, + /// under the read lock, so a slow encoder never holds up the writer queue. + pub fn add( + &self, + node_id: NodeId, + text: &str, + vector: Option<Vec<f32>>, + ) -> Result<String, EmbeddingError> { + let vector = match vector { + Some(vector) => vector, + None => self.read().encode(text)?, + }; + self.write().add(node_id, text, Some(vector)) + } + + pub fn remove(&self, node_id: &NodeId) -> bool { + self.write().remove(node_id) + } + + pub fn get(&self, node_id: &NodeId) -> Option<EmbeddingRecord> { + self.read().get(node_id) + } + + pub fn count(&self) -> usize { + self.read().count() + } + + pub fn similarity_search_vector( + &self, + query_vector: &[f32], + top_k: usize, + node_type: Option<&str>, + threshold: f32, + ) -> Result<Vec<SimilarityResult>, EmbeddingError> { + self.read() + .similarity_search_vector(query_vector, top_k, node_type, threshold) + } + + pub fn similarity_search_vector_batch<V: AsRef<[f32]> + Sync>( + &self, + queries: &[V], + top_k: usize, + node_type: Option<&str>, + threshold: f32, + ) -> Result<Vec<Vec<SimilarityResult>>, EmbeddingError> { + self.read() + .similarity_search_vector_batch(queries, top_k, node_type, threshold) + } +} diff --git a/isongraph-vector-rs/src/snapshot.rs b/isongraph-vector-rs/src/snapshot.rs new file mode 100644 index 0000000..44b4099 --- /dev/null +++ b/isongraph-vector-rs/src/snapshot.rs @@ -0,0 +1,722 @@ +//! Optional binary snapshot of an [`EmbeddingStore`], written next to (never +//! instead of) the JSON embeddings sidecar. +//! +//! The JSON sidecar stays the portable, authoritative format. A snapshot is a +//! cache of it: the same rows in the same order (the sidecar's: sorted by node +//! type then id, live rows only), as little-endian binary, so a load is a read +//! and a copy instead of a parse. A load takes it only when it provably belongs +//! to the graph and sidecar on disk and its checksum verifies; otherwise the +//! caller falls back to the JSON sidecar and is told why ([`LoadReport`]). +//! +//! `DESIGN.md` section 12 gives the exact byte layout. In short: +//! +//! ```text +//! header | block* | sha256(everything before it) +//! block = metadata of R rows, then R * dim f32 (R = 1024 except the last) +//! ``` +//! +//! Not stored, on purpose: the IVF index (see DESIGN.md: a rebuilt index is +//! deterministic but only reproduces a *saved* index when the slot order and +//! fold history are, which a canonical-order file does not keep) and the int8 +//! copy (derived data, rebuilt by `set_scan_precision`). + +use crate::digest::GRAPH_DIGEST_BASIS; +use crate::{EmbeddingError, EmbeddingStore, ImportMeta}; +use sha2::{Digest, Sha256}; +use std::io::{self, BufReader, BufWriter, Read, Seek, SeekFrom, Write}; +use std::path::{Path, PathBuf}; + +/// Appended to the graph file's name: `<graph>.embeddings.snap`. +pub const SNAPSHOT_SUFFIX: &str = ".embeddings.snap"; +/// The only layout version this build reads and writes. +pub const SNAPSHOT_VERSION: u32 = 1; +const MAGIC: &[u8; 8] = b"ISONVSNP"; +/// Rows per block (a writer's choice, recorded in the header; readers honour it). +const BLOCK_ROWS: u32 = 1024; +/// Length value of an absent optional string. +const ABSENT: u32 = u32::MAX; +/// Header flag: the header carries the sidecar's identity. +const FLAG_SIDECAR: u32 = 1; +const KNOWN_FLAGS: u32 = FLAG_SIDECAR; +const TRAILER: u64 = 32; +/// Bytes sampled from the head and the tail of the sidecar, and per probe. +const FP_EDGE: u64 = 64 * 1024; +const FP_PROBE: u64 = 4 * 1024; +const FP_PROBES: u64 = 64; + +// ============================================================================= +// Public types +// ============================================================================= + +/// A cheap identity for a sidecar file: its length and a SHA-256 over its +/// length, its first and last 64 KiB and 64 probes of 4 KiB spread over the +/// rest. Any rewrite of the sidecar by a serializer (every float re-rendered) +/// changes it; a hand edit that keeps the length and avoids every sampled byte +/// does not, which is why a save that does not write a snapshot deletes the old +/// one. See DESIGN.md. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct SidecarId { + pub len: u64, + pub fingerprint: [u8; 32], +} + +impl SidecarId { + /// Read the identity of the file at `path`. + pub fn of_file(path: impl AsRef<Path>) -> io::Result<Self> { + let mut file = std::fs::File::open(path.as_ref())?; + let len = file.metadata()?.len(); + let mut hasher = Sha256::new(); + hasher.update(len.to_le_bytes()); + let mut buf = Vec::new(); + let mut take = |file: &mut std::fs::File, at: u64, n: u64| -> io::Result<()> { + let n = n.min(len.saturating_sub(at)); + buf.resize(n as usize, 0); + file.seek(SeekFrom::Start(at))?; + file.read_exact(&mut buf)?; + hasher.update(&buf); + Ok(()) + }; + take(&mut file, 0, FP_EDGE)?; + take(&mut file, len.saturating_sub(FP_EDGE), FP_EDGE)?; + for i in 1..=FP_PROBES { + take( + &mut file, + (len as u128 * i as u128 / (FP_PROBES + 1) as u128) as u64, + FP_PROBE, + )?; + } + Ok(Self { + len, + fingerprint: hasher.finalize().into(), + }) + } +} + +/// What a snapshot is tied to. +#[derive(Clone, Debug, Default)] +pub struct SnapshotIdentity { + /// `graph_digest` of the graph file (the JSON sidecar records the same). + pub graph_digest: Option<String>, + /// The JSON sidecar written for the same save. + pub sidecar: Option<SidecarId>, +} + +/// The header of a snapshot, and its size. +#[derive(Clone, Debug)] +pub struct SnapshotInfo { + pub version: u32, + pub dimension: usize, + pub count: usize, + pub graph_digest: Option<String>, + pub graph_digest_basis: Option<String>, + pub encoder_model: Option<String>, + pub encoder_version: Option<String>, + pub sidecar: Option<SidecarId>, + /// File size in bytes. + pub bytes: u64, +} + +/// Where a load's embeddings came from. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum LoadSource { + /// The binary snapshot. + Snapshot, + /// The JSON sidecar. + Json, + /// Nothing was restored (embeddings off, or no sidecar). + NoEmbeddings, +} + +/// What `SemanticGraph::load_with_report` did. +#[derive(Clone, Debug)] +pub struct LoadReport { + pub source: LoadSource, + /// Why a snapshot that exists was not used (and the JSON sidecar was read + /// instead); `None` when it was used or there is none. + pub snapshot_skipped: Option<String>, + /// Embeddings restored. + pub embeddings: usize, + /// Wall time of the whole load. + pub seconds: f64, +} + +/// `<graph path>.embeddings.snap`. +pub fn snapshot_path_for(graph_path: impl AsRef<Path>) -> PathBuf { + let mut name = graph_path.as_ref().as_os_str().to_os_string(); + name.push(SNAPSHOT_SUFFIX); + PathBuf::from(name) +} + +// ============================================================================= +// Errors, hashing wrappers +// ============================================================================= + +enum Fail { + /// The file cannot be read or written. + Io(String), + /// The file is not a valid snapshot of the right identity. + Bad(String), +} + +impl Fail { + fn into_error(self) -> EmbeddingError { + match self { + Fail::Io(m) => EmbeddingError::Io(m), + Fail::Bad(m) => EmbeddingError::Payload(format!("invalid snapshot: {m}")), + } + } + + fn message(self) -> String { + match self { + Fail::Io(m) | Fail::Bad(m) => m, + } + } +} + +fn io_fail(what: &str, e: io::Error) -> Fail { + if e.kind() == io::ErrorKind::UnexpectedEof { + Fail::Bad("truncated".to_string()) + } else { + Fail::Io(format!("{what}: {e}")) + } +} + +/// Hashes everything written through it. +struct HashWriter<W: Write> { + inner: W, + hasher: Sha256, + written: u64, +} + +impl<W: Write> Write for HashWriter<W> { + fn write(&mut self, buf: &[u8]) -> io::Result<usize> { + let n = self.inner.write(buf)?; + self.hasher.update(&buf[..n]); + self.written += n as u64; + Ok(n) + } + fn flush(&mut self) -> io::Result<()> { + self.inner.flush() + } +} + +/// Hashes everything read through it and tracks how much of the body is left. +struct HashReader<R: Read> { + inner: R, + hasher: Sha256, + consumed: u64, + /// Bytes before the trailer. + body: u64, +} + +impl<R: Read> HashReader<R> { + fn remaining(&self) -> u64 { + self.body.saturating_sub(self.consumed) + } + + fn exact(&mut self, buf: &mut [u8]) -> Result<(), Fail> { + if buf.len() as u64 > self.remaining() { + return Err(Fail::Bad("truncated".to_string())); + } + self.inner + .read_exact(buf) + .map_err(|e| io_fail("reading snapshot", e))?; + self.hasher.update(&*buf); + self.consumed += buf.len() as u64; + Ok(()) + } + + fn u32(&mut self) -> Result<u32, Fail> { + let mut b = [0u8; 4]; + self.exact(&mut b)?; + Ok(u32::from_le_bytes(b)) + } + + fn u64(&mut self) -> Result<u64, Fail> { + let mut b = [0u8; 8]; + self.exact(&mut b)?; + Ok(u64::from_le_bytes(b)) + } + + /// A length-prefixed string; `optional` lets the length be `ABSENT`. + fn string(&mut self, optional: bool) -> Result<Option<String>, Fail> { + let len = self.u32()?; + if len == ABSENT { + return if optional { + Ok(None) + } else { + Err(Fail::Bad("a required string is marked absent".to_string())) + }; + } + if len as u64 > self.remaining() { + return Err(Fail::Bad("a string is longer than the file".to_string())); + } + let mut b = vec![0u8; len as usize]; + self.exact(&mut b)?; + String::from_utf8(b) + .map(Some) + .map_err(|_| Fail::Bad("a string is not UTF-8".to_string())) + } +} + +// ============================================================================= +// Writing +// ============================================================================= + +fn put_u32(out: &mut Vec<u8>, v: u32) { + out.extend_from_slice(&v.to_le_bytes()); +} + +fn put_str(out: &mut Vec<u8>, s: &str) { + put_u32(out, s.len() as u32); + out.extend_from_slice(s.as_bytes()); +} + +fn put_opt(out: &mut Vec<u8>, s: Option<&str>) { + match s { + Some(s) => put_str(out, s), + None => put_u32(out, ABSENT), + } +} + +/// Write `fill` to a temp file beside `path`, flush it to disk and rename it +/// over `path`, so a reader sees the old file or the new one, never a partial +/// one. A failure removes the temp file and leaves `path` as it was. +fn write_atomic<T>( + path: &Path, + fill: impl FnOnce(&mut BufWriter<std::fs::File>) -> Result<T, Fail>, +) -> Result<T, Fail> { + use std::sync::atomic::{AtomicU64, Ordering}; + static COUNTER: AtomicU64 = AtomicU64::new(0); + let name = path + .file_name() + .map(|n| n.to_string_lossy().into_owned()) + .unwrap_or_else(|| "snapshot".to_string()); + let tmp = path.with_file_name(format!( + ".{}.{}.{}.tmp", + name, + std::process::id(), + COUNTER.fetch_add(1, Ordering::Relaxed) + )); + let result = (|| -> Result<T, Fail> { + let file = std::fs::File::create(&tmp).map_err(|e| io_fail("creating snapshot", e))?; + let mut writer = BufWriter::with_capacity(4 << 20, file); + let value = fill(&mut writer)?; + writer.flush().map_err(|e| io_fail("writing snapshot", e))?; + let file = writer + .into_inner() + .map_err(|e| io_fail("writing snapshot", e.into_error()))?; + // A filesystem that cannot fsync still gets the atomic rename. + let _ = file.sync_all(); + drop(file); + std::fs::rename(&tmp, path).map_err(|e| io_fail("replacing snapshot", e))?; + Ok(value) + })(); + if result.is_err() { + let _ = std::fs::remove_file(&tmp); + } + result +} + +impl EmbeddingStore { + /// Write a binary snapshot of every live embedding to `path`, atomically + /// (a temp file, flushed, renamed over `path`: a failure or crash leaves + /// the previous file untouched). Rows are written in the order the JSON + /// sidecar uses (sorted by node type, then id), so loading a snapshot gives + /// the same store, slot for slot, as loading the sidecar. + /// + /// `identity` ties it to the graph and sidecar it was saved with; a load + /// that finds a different identity refuses the snapshot. + pub fn save_snapshot( + &self, + path: impl AsRef<Path>, + identity: &SnapshotIdentity, + ) -> Result<SnapshotInfo, EmbeddingError> { + write_atomic(path.as_ref(), |w| self.write_snapshot(w, identity)).map_err(Fail::into_error) + } + + /// Replace the store's embeddings with those of a snapshot file. Nothing + /// changes unless the whole file verifies (structure and checksum); the + /// scan precision and approximate-index settings are kept (the index is + /// rebuilt lazily). Does not check identity: use `SemanticGraph::load` for + /// that. + pub fn load_snapshot( + &mut self, + path: impl AsRef<Path>, + ) -> Result<SnapshotInfo, EmbeddingError> { + let (stage, info) = read_snapshot(path.as_ref(), &|_| Ok(())).map_err(Fail::into_error)?; + self.adopt(stage, &info); + Ok(info) + } + + fn write_snapshot<W: Write>( + &self, + w: &mut W, + identity: &SnapshotIdentity, + ) -> Result<SnapshotInfo, Fail> { + let table = &self.table; + let dim = table.dim; + // the sidecar's order: node type, then id + let mut rows: Vec<(&str, &str, usize)> = table + .live_slots() + .filter_map(|s| table.type_and_id(s).map(|(t, i)| (t, i, s))) + .collect(); + rows.sort_unstable_by(|a, b| (a.0, a.1).cmp(&(b.0, b.1))); + let count = rows.len(); + // models, in order of first appearance + let mut models: Vec<&str> = Vec::new(); + let mut model_ix: std::collections::HashMap<&str, u32> = std::collections::HashMap::new(); + for &(_, _, s) in &rows { + let m: &str = &table.meta(s).expect("live").model; + if !model_ix.contains_key(m) { + model_ix.insert(m, models.len() as u32); + models.push(m); + } + } + + let basis = identity.graph_digest.as_ref().map(|_| GRAPH_DIGEST_BASIS); + let encoder_model = self.current_model(); + let encoder_version = self.version(); + let mut head = Vec::with_capacity(512); + head.extend_from_slice(MAGIC); + put_u32(&mut head, SNAPSHOT_VERSION); + put_u32( + &mut head, + if identity.sidecar.is_some() { + FLAG_SIDECAR + } else { + 0 + }, + ); + put_u32(&mut head, dim as u32); + put_u32(&mut head, BLOCK_ROWS); + head.extend_from_slice(&(count as u64).to_le_bytes()); + put_opt(&mut head, identity.graph_digest.as_deref()); + put_opt(&mut head, basis); + put_opt(&mut head, encoder_model.as_deref()); + put_opt(&mut head, encoder_version.as_deref()); + if let Some(id) = &identity.sidecar { + head.extend_from_slice(&id.len.to_le_bytes()); + head.extend_from_slice(&id.fingerprint); + } + put_u32(&mut head, models.len() as u32); + for m in &models { + put_str(&mut head, m); + } + + let mut out = HashWriter { + inner: w, + hasher: Sha256::new(), + written: 0, + }; + let werr = |e: io::Error| io_fail("writing snapshot", e); + out.write_all(&head).map_err(werr)?; + let mut buf: Vec<u8> = Vec::with_capacity(BLOCK_ROWS as usize * (dim * 4 + 96)); + for block in rows.chunks(BLOCK_ROWS as usize) { + buf.clear(); + for &(ty, id, s) in block { + let meta = table.meta(s).expect("live"); + put_str(&mut buf, ty); + put_str(&mut buf, id); + put_str(&mut buf, &meta.text); + put_u32(&mut buf, model_ix[&*meta.model]); + } + for &(_, _, s) in block { + for f in table.row(s) { + buf.extend_from_slice(&f.to_le_bytes()); + } + } + out.write_all(&buf).map_err(werr)?; + } + let digest: [u8; 32] = out.hasher.clone().finalize().into(); + let body = out.written; + out.inner.write_all(&digest).map_err(werr)?; + Ok(SnapshotInfo { + version: SNAPSHOT_VERSION, + dimension: dim, + count, + graph_digest: identity.graph_digest.clone(), + graph_digest_basis: basis.map(String::from), + encoder_model, + encoder_version, + sidecar: identity.sidecar, + bytes: body + TRAILER, + }) + } + + /// Add one row of a snapshot to a staging store. False for a repeated key. + fn put_snapshot_row( + &mut self, + ty: &str, + id: &str, + text: &str, + model: &str, + vector: &[f32], + ) -> bool { + let key = format!("{ty}:{id}"); + let model_name = self.intern_model(model); + let (_, replaced) = self.table.put(&key, ty, vector, text, &model_name); + if replaced.is_some() { + return false; + } + self.dimension.get_or_insert(vector.len()); + match self.model_counts.get_mut(model) { + Some(n) => *n += 1, + None => { + self.model_counts.insert(model.to_string(), 1); + } + } + true + } + + /// Install a verified staging store's rows in this one. + fn adopt(&mut self, mut stage: EmbeddingStore, info: &SnapshotInfo) { + let quant = self.table.quant().map(|q| q.rerank_factor); + self.table = std::mem::replace(&mut stage.table, crate::table::VectorTable::new()); + self.table.set_quant(quant); + self.model_names = std::mem::take(&mut stage.model_names); + self.model_counts = std::mem::take(&mut stage.model_counts); + self.dimension = stage.dimension; + self.last_import_meta = ImportMeta { + encoder_model: info.encoder_model.clone(), + encoder_version: info.encoder_version.clone(), + graph_digest: info.graph_digest.clone(), + graph_digest_basis: info.graph_digest_basis.clone(), + }; + *self.ivf_mut() = None; + } + + /// Use the snapshot at `snap` for the graph whose digest is `graph_digest` + /// and whose sidecar is at `sidecar`, if it belongs to them. `Err` is the + /// reason it was not used (the store is untouched). + pub(crate) fn adopt_snapshot_for( + &mut self, + snap: &Path, + sidecar: &Path, + graph_digest: &str, + ) -> Result<SnapshotInfo, String> { + let current = SidecarId::of_file(sidecar) + .map_err(|e| format!("cannot read the sidecar {}: {e}", sidecar.display()))?; + let (stage, info) = read_snapshot(snap, &|h| { + if h.graph_digest.as_deref() != Some(graph_digest) + || h.graph_digest_basis.as_deref() != Some(GRAPH_DIGEST_BASIS) + { + return Err("it was saved for a different graph (digest differs)".to_string()); + } + match &h.sidecar { + None => Err("it records no sidecar identity".to_string()), + Some(id) if *id != current => Err( + "the JSON sidecar changed since the snapshot was written (stale snapshot)" + .to_string(), + ), + Some(_) => Ok(()), + } + }) + .map_err(Fail::message)?; + self.adopt(stage, &info); + Ok(info) + } +} + +// ============================================================================= +// Reading +// ============================================================================= + +/// Read and fully verify the snapshot at `path`, after `accept` has approved +/// its header. Returns the rows in a fresh staging store. +fn read_snapshot( + path: &Path, + accept: &dyn Fn(&SnapshotInfo) -> Result<(), String>, +) -> Result<(EmbeddingStore, SnapshotInfo), Fail> { + let file = std::fs::File::open(path) + .map_err(|e| Fail::Io(format!("opening snapshot {}: {e}", path.display())))?; + let file_len = file + .metadata() + .map_err(|e| io_fail("reading snapshot", e))? + .len(); + if file_len < TRAILER + MAGIC.len() as u64 { + return Err(Fail::Bad("file too short".to_string())); + } + let mut r = HashReader { + inner: BufReader::with_capacity(4 << 20, file), + hasher: Sha256::new(), + consumed: 0, + body: file_len - TRAILER, + }; + let mut magic = [0u8; 8]; + r.exact(&mut magic)?; + if &magic != MAGIC { + return Err(Fail::Bad("not a snapshot (bad magic)".to_string())); + } + let version = r.u32()?; + if version != SNAPSHOT_VERSION { + return Err(Fail::Bad(format!( + "unsupported snapshot version {version} (this build reads {SNAPSHOT_VERSION})" + ))); + } + let flags = r.u32()?; + if flags & !KNOWN_FLAGS != 0 { + return Err(Fail::Bad(format!("unknown flags {flags:#x}"))); + } + let dim = r.u32()? as usize; + let block_rows = r.u32()? as usize; + let count = r.u64()?; + if block_rows == 0 || block_rows > 1 << 20 { + return Err(Fail::Bad(format!("implausible block size {block_rows}"))); + } + if count > u32::MAX as u64 || (count > 0 && dim == 0) { + return Err(Fail::Bad(format!("implausible shape {count} x {dim}"))); + } + if (count as u128) * (dim as u128) * 4 > r.remaining() as u128 { + return Err(Fail::Bad("fewer bytes than the rows need".to_string())); + } + let graph_digest = r.string(true)?; + let graph_digest_basis = r.string(true)?; + let encoder_model = r.string(true)?; + let encoder_version = r.string(true)?; + let sidecar = if flags & FLAG_SIDECAR != 0 { + let len = r.u64()?; + let mut fingerprint = [0u8; 32]; + r.exact(&mut fingerprint)?; + Some(SidecarId { len, fingerprint }) + } else { + None + }; + let model_count = r.u32()? as usize; + if model_count as u64 > r.remaining() / 4 { + return Err(Fail::Bad("more models than the file can hold".to_string())); + } + let mut models = Vec::with_capacity(model_count); + for _ in 0..model_count { + models.push(r.string(false)?.unwrap_or_default()); + } + let info = SnapshotInfo { + version, + dimension: dim, + count: count as usize, + graph_digest, + graph_digest_basis, + encoder_model, + encoder_version, + sidecar, + bytes: file_len, + }; + accept(&info).map_err(Fail::Bad)?; + + let mut stage = EmbeddingStore::new(None); + let mut left = count as usize; + let (mut bytes, mut floats): (Vec<u8>, Vec<f32>) = (Vec::new(), Vec::new()); + while left > 0 { + let rows = left.min(block_rows); + let mut meta: Vec<(String, String, String, usize)> = Vec::with_capacity(rows); + for _ in 0..rows { + let ty = r.string(false)?.unwrap_or_default(); + let id = r.string(false)?.unwrap_or_default(); + let text = r.string(false)?.unwrap_or_default(); + let model = r.u32()? as usize; + if model >= models.len() { + return Err(Fail::Bad( + "a row names a model that is not listed".to_string(), + )); + } + meta.push((ty, id, text, model)); + } + let need = rows * dim * 4; + if need as u64 > r.remaining() { + return Err(Fail::Bad("truncated".to_string())); + } + bytes.resize(need, 0); + r.exact(&mut bytes)?; + floats.clear(); + floats.extend((0..bytes.len() / 4).map(|i| { + let mut four = [0u8; 4]; + four.copy_from_slice(&bytes[4 * i..4 * i + 4]); + f32::from_le_bytes(four) + })); + for (i, (ty, id, text, model)) in meta.iter().enumerate() { + if !stage.put_snapshot_row( + ty, + id, + text, + &models[*model], + &floats[i * dim..(i + 1) * dim], + ) { + return Err(Fail::Bad(format!("the key {ty}:{id} appears twice"))); + } + } + left -= rows; + } + if r.remaining() != 0 { + return Err(Fail::Bad( + "bytes left over after the last block".to_string(), + )); + } + let expected: [u8; 32] = r.hasher.clone().finalize().into(); + let mut stored = [0u8; 32]; + r.inner + .read_exact(&mut stored) + .map_err(|e| io_fail("reading snapshot", e))?; + if stored != expected { + return Err(Fail::Bad("checksum mismatch".to_string())); + } + let mut extra = [0u8; 1]; + if matches!(r.inner.read(&mut extra), Ok(n) if n > 0) { + return Err(Fail::Bad("bytes after the checksum".to_string())); + } + Ok((stage, info)) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn a_failed_atomic_write_keeps_the_previous_file_and_leaves_no_temp() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("x.snap"); + std::fs::write(&path, b"previous").unwrap(); + let result: Result<(), Fail> = write_atomic(&path, |w| { + w.write_all(b"half of the new contents") + .map_err(|e| io_fail("w", e))?; + Err(Fail::Io("disk full".to_string())) + }); + assert!(result.is_err()); + assert_eq!(std::fs::read(&path).unwrap(), b"previous"); + let left: Vec<_> = std::fs::read_dir(dir.path()).unwrap().collect(); + assert_eq!(left.len(), 1, "only the previous file remains"); + write_atomic(&path, |w| w.write_all(b"new").map_err(|e| io_fail("w", e))) + .unwrap_or_else(|_| panic!("the next write works")); + assert_eq!(std::fs::read(&path).unwrap(), b"new"); + } + + #[test] + fn a_failing_writer_fails_the_snapshot_without_touching_the_file() { + struct Failing(usize); + impl Write for Failing { + fn write(&mut self, b: &[u8]) -> io::Result<usize> { + if self.0 < b.len() { + return Err(io::Error::other("injected")); + } + self.0 -= b.len(); + Ok(b.len()) + } + fn flush(&mut self) -> io::Result<()> { + Ok(()) + } + } + let mut store = EmbeddingStore::new(None); + for i in 0..50 { + store + .add( + crate::NodeId::new("doc", i.to_string()), + "t", + Some(vec![i as f32 + 1.0; 8]), + ) + .unwrap(); + } + let mut sink = Failing(300); + assert!(store + .write_snapshot(&mut sink, &SnapshotIdentity::default()) + .is_err()); + } +} diff --git a/isongraph-vector-rs/src/table.rs b/isongraph-vector-rs/src/table.rs new file mode 100644 index 0000000..3e5acd2 --- /dev/null +++ b/isongraph-vector-rs/src/table.rs @@ -0,0 +1,453 @@ +//! The single source of truth for a store's vectors: one contiguous matrix. +//! +//! A [`VectorTable`] holds every vector exactly once, in row-major chunks of +//! [`CHUNK_ROWS`] rows, with the per-row facts a scan needs (norm, type code) +//! in parallel arrays and the per-row metadata a result needs (key, text, +//! model) beside them. A record is *a slot number*: nothing else owns a copy of +//! the floats. Searches read the table directly; the approximate index stores +//! only slot numbers. +//! +//! * Rows are in arrival order. An update overwrites its slot, an insert takes +//! the next one, a delete tombstones one (norm NaN, metadata dropped) and a +//! quarter of dead rows triggers an order-preserving compaction. +//! * The matrix is chunked so growth never copies what is already stored (a +//! single `Vec` would copy up to the whole matrix on a doubling) and so a +//! scan splits into cache-sized blocks that never straddle a chunk. + +use crate::ann::dot_f32; +use crate::ann::quant::QuantRows; +use crate::{EmbeddingRecord, NodeId}; +use std::collections::HashMap; +use std::sync::Arc; + +/// Rows per chunk (a power of two: slot -> chunk is a shift). +pub(crate) const CHUNK_SHIFT: usize = 10; +pub(crate) const CHUNK_ROWS: usize = 1 << CHUNK_SHIFT; +pub(crate) const CHUNK_MASK: usize = CHUNK_ROWS - 1; + +/// What a live row remembers besides its floats. +pub(crate) struct RowMeta { + /// `"<type>:<id>"`, the record's key (shared with the key index). + pub key: Arc<str>, + pub text: Box<str>, + pub model: Arc<str>, +} + +/// The matrix and everything indexed by slot. +pub(crate) struct VectorTable { + pub dim: usize, + chunks: Vec<Vec<f32>>, + /// Euclidean norm per row; NaN marks a tombstone. + norms: Vec<f32>, + /// Node-type code per row (index into `types`). + codes: Vec<u32>, + meta: Vec<Option<RowMeta>>, + index: HashMap<Arc<str>, u32>, + types: Vec<Arc<str>>, + code_of: HashMap<Arc<str>, u32>, + /// Slots in use, tombstones included. + n: usize, + dead: usize, + /// The opt-in int8 copy of every row (see `ann/quant.rs`), kept slot for + /// slot beside the f32 matrix so it follows every write and compaction. + quant: Option<QuantRows>, +} + +/// A read-only view of the matrix side, handed to scan kernels. +#[derive(Clone, Copy)] +pub(crate) struct Rows<'a> { + pub chunks: &'a [Vec<f32>], + pub norms: &'a [f32], + pub codes: &'a [u32], + pub dim: usize, +} + +impl<'a> Rows<'a> { + #[inline(always)] + pub fn row(&self, slot: usize) -> &'a [f32] { + let chunk: &'a Vec<f32> = &self.chunks[slot >> CHUNK_SHIFT]; + let at = (slot & CHUNK_MASK) * self.dim; + &chunk[at..at + self.dim] + } + + /// The contiguous rows from `slot` to the end of its chunk or `hi`, + /// whichever comes first: `(floats, row count)`. + #[inline(always)] + pub fn run(&self, slot: usize, hi: usize) -> (&'a [f32], usize) { + let count = (CHUNK_ROWS - (slot & CHUNK_MASK)).min(hi - slot); + let at = (slot & CHUNK_MASK) * self.dim; + let chunk: &'a Vec<f32> = &self.chunks[slot >> CHUNK_SHIFT]; + (&chunk[at..at + count * self.dim], count) + } +} + +impl VectorTable { + pub fn new() -> Self { + Self { + dim: 0, + chunks: Vec::new(), + norms: Vec::new(), + codes: Vec::new(), + meta: Vec::new(), + index: HashMap::new(), + types: Vec::new(), + code_of: HashMap::new(), + n: 0, + dead: 0, + quant: None, + } + } + + /// Forget every row (the type codes are kept; they are tiny and stable). + pub fn clear(&mut self) { + self.chunks = Vec::new(); + self.norms = Vec::new(); + self.codes = Vec::new(); + self.meta = Vec::new(); + self.index = HashMap::new(); + self.n = 0; + self.dead = 0; + self.dim = 0; + if let Some(q) = &mut self.quant { + q.clear(); + } + } + + /// The int8 copy, when scan precision is `Int8`. + pub fn quant(&self) -> Option<&QuantRows> { + self.quant.as_ref() + } + + /// Keep (or, with `None`, drop) an int8 copy of every row. Building walks + /// the whole matrix once. + pub fn set_quant(&mut self, rerank_factor: Option<usize>) { + match (rerank_factor, self.quant.as_mut()) { + (None, _) => self.quant = None, + (Some(f), Some(q)) => q.rerank_factor = f, + (Some(f), None) => { + let mut q = QuantRows::new(f); + q.rebuild(&self.rows(), self.n); + self.quant = Some(q); + } + } + } + + /// Node type name and id of a live row, borrowed. + pub fn type_and_id(&self, slot: usize) -> Option<(&str, &str)> { + let meta = self.meta(slot)?; + let ty = &*self.types[self.codes[slot] as usize]; + Some((ty, &meta.key[ty.len() + 1..])) + } + + /// Rows in use, tombstones included. Slots are `0..slots()`. + pub fn slots(&self) -> usize { + self.n + } + + pub fn live(&self) -> usize { + self.n - self.dead + } + + pub fn dead(&self) -> usize { + self.dead + } + + pub fn chunk_count(&self) -> usize { + self.chunks.len() + } + + pub fn rows(&self) -> Rows<'_> { + Rows { + chunks: &self.chunks, + norms: &self.norms, + codes: &self.codes, + dim: self.dim, + } + } + + pub fn row(&self, slot: usize) -> &[f32] { + self.rows().row(slot) + } + + pub fn norm(&self, slot: usize) -> f32 { + self.norms[slot] + } + + pub fn is_live(&self, slot: usize) -> bool { + self.meta.get(slot).is_some_and(|m| m.is_some()) + } + + pub fn meta(&self, slot: usize) -> Option<&RowMeta> { + self.meta.get(slot)?.as_ref() + } + + pub fn slot_of(&self, key: &str) -> Option<u32> { + self.index.get(key).copied() + } + + /// The slot's contiguous run of rows from `slot` to the end of its chunk + /// or `hi`, whichever is first: `(rows as a flat slice, row count)`. + pub fn run(&self, slot: usize, hi: usize) -> (&[f32], usize) { + let in_chunk = CHUNK_ROWS - (slot & CHUNK_MASK); + let count = in_chunk.min(hi - slot); + let at = (slot & CHUNK_MASK) * self.dim; + ( + &self.chunks[slot >> CHUNK_SHIFT][at..at + count * self.dim], + count, + ) + } + + pub fn type_code(&self, node_type: &str) -> Option<u32> { + self.code_of.get(node_type).copied() + } + + fn code_for(&mut self, node_type: &str) -> u32 { + if let Some(&code) = self.code_of.get(node_type) { + return code; + } + let code = self.types.len() as u32; + let name: Arc<str> = Arc::from(node_type); + self.types.push(name.clone()); + self.code_of.insert(name, code); + code + } + + pub fn node_id(&self, slot: usize) -> Option<NodeId> { + let meta = self.meta(slot)?; + let ty = &self.types[self.codes[slot] as usize]; + Some(NodeId::new(&**ty, &meta.key[ty.len() + 1..])) + } + + pub fn record(&self, slot: usize) -> Option<EmbeddingRecord> { + let meta = self.meta(slot)?; + Some(EmbeddingRecord { + id: meta.key.to_string(), + node_id: self.node_id(slot)?, + vector: self.row(slot).to_vec(), + text: meta.text.to_string(), + model: meta.model.to_string(), + }) + } + + /// Live slots in arrival order. + pub fn live_slots(&self) -> impl Iterator<Item = usize> + '_ { + (0..self.n).filter(|&s| self.meta[s].is_some()) + } + + /// Insert or overwrite the row for `key`. Returns its slot and the model + /// of the record it replaced, if any. + pub fn put( + &mut self, + key: &str, + node_type: &str, + vector: &[f32], + text: &str, + model: &Arc<str>, + ) -> (u32, Option<Arc<str>>) { + if self.live() == 0 && self.dim != vector.len() { + // empty (possibly after deletes): the first vector fixes the width + self.clear(); + self.dim = vector.len(); + } + debug_assert_eq!(self.dim, vector.len()); + let code = self.code_for(node_type); + let norm = dot_f32(vector, vector).sqrt(); + if let Some(&slot) = self.index.get(key) { + let s = slot as usize; + let at = (s & CHUNK_MASK) * self.dim; + self.chunks[s >> CHUNK_SHIFT][at..at + self.dim].copy_from_slice(vector); + if let Some(q) = &mut self.quant { + q.set(s, vector); + } + self.norms[s] = norm; + self.codes[s] = code; + let meta = self.meta[s].as_mut().expect("indexed rows are live"); + meta.text = text.into(); + let old = std::mem::replace(&mut meta.model, model.clone()); + return (slot, Some(old)); + } + let slot = self.n; + if slot & CHUNK_MASK == 0 { + // later chunks are reserved whole so they never reallocate; the + // first grows like a Vec so a tiny store stays tiny + self.chunks.push(if self.chunks.is_empty() { + Vec::new() + } else { + Vec::with_capacity(CHUNK_ROWS * self.dim) + }); + } + self.chunks + .last_mut() + .expect("a chunk exists") + .extend_from_slice(vector); + if let Some(q) = &mut self.quant { + q.push(slot, vector); + } + self.norms.push(norm); + self.codes.push(code); + let key: Arc<str> = Arc::from(key); + self.meta.push(Some(RowMeta { + key: key.clone(), + text: text.into(), + model: model.clone(), + })); + self.index.insert(key, slot as u32); + self.n += 1; + (slot as u32, None) + } + + /// Tombstone the row for `key`. Returns its slot and the model it had. + pub fn remove(&mut self, key: &str) -> Option<(u32, Arc<str>)> { + let slot = self.index.remove(key)?; + let s = slot as usize; + self.norms[s] = f32::NAN; + self.dead += 1; + let meta = self.meta[s].take().expect("indexed rows are live"); + Some((slot, meta.model)) + } + + pub fn should_compact(&self) -> bool { + self.dead > 0 && self.dead * 4 > self.n + } + + /// Drop the tombstones, keeping row order. Returns the old slot of every + /// surviving row, in order, so an index can follow the move. + pub fn compact(&mut self) -> Vec<u32> { + let dim = self.dim; + let live: Vec<u32> = (0..self.n as u32) + .filter(|&s| self.meta[s as usize].is_some()) + .collect(); + for (new, &old) in live.iter().enumerate() { + let old = old as usize; + if new == old { + continue; + } + let (oc, oa) = (old >> CHUNK_SHIFT, (old & CHUNK_MASK) * dim); + let (nc, na) = (new >> CHUNK_SHIFT, (new & CHUNK_MASK) * dim); + if oc == nc { + self.chunks[oc].copy_within(oa..oa + dim, na); + } else { + // new < old, so the destination chunk is the earlier one + let (head, tail) = self.chunks.split_at_mut(oc); + head[nc][na..na + dim].copy_from_slice(&tail[0][oa..oa + dim]); + } + self.norms[new] = self.norms[old]; + self.codes[new] = self.codes[old]; + self.meta.swap(new, old); + if let Some(meta) = &self.meta[new] { + if let Some(slot) = self.index.get_mut(&*meta.key) { + *slot = new as u32; + } + } + } + let kept = live.len(); + self.norms.truncate(kept); + self.codes.truncate(kept); + self.meta.truncate(kept); + let chunks = kept.div_ceil(CHUNK_ROWS); + self.chunks.truncate(chunks); + if let Some(last) = self.chunks.last_mut() { + last.truncate((kept - (chunks - 1) * CHUNK_ROWS) * dim); + } + self.n = kept; + self.dead = 0; + if let Some(q) = &mut self.quant { + q.compact(&live, dim); + } + live + } + + /// Every structural invariant of the table, or the first one broken. + pub fn check(&self) -> Result<(), String> { + let n = self.n; + if self.norms.len() != n || self.codes.len() != n || self.meta.len() != n { + return Err(format!( + "parallel arrays disagree: n={n} norms={} codes={} meta={}", + self.norms.len(), + self.codes.len(), + self.meta.len() + )); + } + let floats: usize = self.chunks.iter().map(|c| c.len()).sum(); + if floats != n * self.dim { + return Err(format!( + "matrix holds {floats} floats, expected {}", + n * self.dim + )); + } + if self.chunks.len() != n.div_ceil(CHUNK_ROWS) { + return Err(format!("{} chunks for {n} rows", self.chunks.len())); + } + for (i, chunk) in self.chunks.iter().enumerate() { + if i + 1 < self.chunks.len() && chunk.len() != CHUNK_ROWS * self.dim { + return Err(format!("chunk {i} is not full ({} floats)", chunk.len())); + } + } + let dead = self.meta.iter().filter(|m| m.is_none()).count(); + if dead != self.dead { + return Err(format!("dead counter {} but {dead} tombstones", self.dead)); + } + if self.index.len() != n - dead { + return Err(format!( + "index holds {} keys for {} live rows", + self.index.len(), + n - dead + )); + } + for (slot, meta) in self.meta.iter().enumerate() { + match meta { + None => { + if !self.norms[slot].is_nan() { + return Err(format!("tombstone {slot} has norm {}", self.norms[slot])); + } + } + Some(m) => { + if self.index.get(&*m.key).copied() != Some(slot as u32) { + return Err(format!("key {:?} does not map back to slot {slot}", m.key)); + } + let ty = self + .types + .get(self.codes[slot] as usize) + .ok_or_else(|| format!("slot {slot} has an unknown type code"))?; + if !m.key.starts_with(&**ty) || m.key.as_bytes().get(ty.len()) != Some(&b':') { + return Err(format!("key {:?} does not start with type {ty:?}", m.key)); + } + } + } + } + if let Some(q) = &self.quant { + q.check(&self.rows(), n)?; + } + Ok(()) + } + + /// Bytes held by the table, by part (capacity, i.e. what is allocated). + pub fn bytes(&self) -> TableBytes { + let matrix: usize = self.chunks.iter().map(|c| c.capacity() * 4).sum(); + let per_row_arrays = self.norms.capacity() * 4 + + self.codes.capacity() * 4 + + self.meta.capacity() * std::mem::size_of::<Option<RowMeta>>(); + let mut strings = 0usize; + for m in self.meta.iter().flatten() { + // key (shared with the index), text, and the allocator's headers + strings += 16 + m.key.len().max(1) + m.text.len(); + } + // hashbrown: 1 control byte + (Arc<str>, u32) per bucket, ~7/8 load + let index = self.index.capacity() * (std::mem::size_of::<(Arc<str>, u32)>() + 1); + TableBytes { + matrix, + per_row: per_row_arrays, + keys_and_text: strings, + key_index: index, + } + } +} + +/// Where a table's memory goes. +#[derive(Clone, Copy, Debug, Default)] +pub(crate) struct TableBytes { + pub matrix: usize, + pub per_row: usize, + pub keys_and_text: usize, + pub key_index: usize, +} diff --git a/isongraph-vector-rs/tests/ann.rs b/isongraph-vector-rs/tests/ann.rs new file mode 100644 index 0000000..cb18dee --- /dev/null +++ b/isongraph-vector-rs/tests/ann.rs @@ -0,0 +1,1337 @@ +//! The opt-in IVF-flat approximate index: recall, exactness where it must be +//! exact, upkeep under writes, filters, determinism, and the guarantee that +//! nothing it builds ever reaches a graph file, a sidecar or the graph digest. +//! +//! Test data are clustered unit-ish vectors drawn from a seeded generator, so +//! every run sees the same numbers. + +use ison_graph_rs::Node; +use isongraph_vector_rs::{ + graph_digest, AnnConfig, EmbeddingError, EmbeddingStore, MockEncoder, MultiHopOptions, + NodeFilter, NodeId, PropertyValue, SearchOptions, SemanticGraph, SimilarityOptions, + SimilarityResult, FOLD_BATCH, +}; +use std::collections::HashSet; + +/// Recall@10 demanded of the index on the clustered data below (4,000 vectors, +/// 40 clusters, 64 cells, nprobe 8 = 12.5% of the cells). The same floor the +/// Python port's tests use; measured 1.000 on every seed tried. +const RECALL_FLOOR: f64 = 0.95; + +// -- data --------------------------------------------------------------------- + +struct Rng(u64); + +impl Rng { + fn next_u64(&mut self) -> u64 { + self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } + fn unit(&mut self) -> f32 { + (self.next_u64() >> 40) as f32 / (1u64 << 24) as f32 + } + fn gauss(&mut self) -> f32 { + // Box-Muller + let u1 = self.unit().max(1e-7); + let u2 = self.unit(); + (-2.0 * u1.ln()).sqrt() * (std::f32::consts::TAU * u2).cos() + } + fn below(&mut self, n: usize) -> usize { + (self.next_u64() % n as u64) as usize + } +} + +struct Clusters { + centers: Vec<Vec<f32>>, + dim: usize, +} + +impl Clusters { + fn new(dim: usize, clusters: usize, seed: u64) -> Self { + let mut rng = Rng(seed); + let centers = (0..clusters) + .map(|_| { + let mut c: Vec<f32> = (0..dim).map(|_| rng.gauss()).collect(); + let norm = c.iter().map(|x| x * x).sum::<f32>().sqrt(); + c.iter_mut().for_each(|x| *x /= norm); + c + }) + .collect(); + Self { centers, dim } + } + + fn draw(&self, count: usize, rng: &mut Rng) -> Vec<Vec<f32>> { + let scale = 1.0 / (self.dim as f32).sqrt(); + (0..count) + .map(|_| { + let c = &self.centers[rng.below(self.centers.len())]; + c.iter().map(|x| x + rng.gauss() * scale).collect() + }) + .collect() + } +} + +fn cosine(a: &[f32], b: &[f32]) -> f64 { + let (mut dot, mut na, mut nb) = (0.0f64, 0.0f64, 0.0f64); + for (x, y) in a.iter().zip(b) { + dot += *x as f64 * *y as f64; + na += *x as f64 * *x as f64; + nb += *y as f64 * *y as f64; + } + if na == 0.0 || nb == 0.0 { + 0.0 + } else { + dot / (na.sqrt() * nb.sqrt()) + } +} + +fn doc(i: usize) -> NodeId { + NodeId::new("doc", i.to_string()) +} + +fn ref_of(r: &SimilarityResult) -> (String, String) { + (r.node_id.node_type.clone(), r.node_id.id.clone()) +} + +fn refs(results: &[SimilarityResult]) -> Vec<(String, String)> { + results.iter().map(ref_of).collect() +} + +fn ref_set(results: &[SimilarityResult]) -> HashSet<(String, String)> { + results.iter().map(ref_of).collect() +} + +/// nlist 64, nprobe 8, no minimum size: the configuration the recall floor is +/// stated for. +fn config() -> AnnConfig { + AnnConfig { + nlist: Some(64), + nprobe: Some(8), + min_size: 0, + ..AnnConfig::default() + } +} + +struct Fixture { + store: EmbeddingStore, + data: Vec<Vec<f32>>, + clusters: Clusters, + rng: Rng, +} + +impl Fixture { + fn queries(&mut self, count: usize) -> Vec<Vec<f32>> { + self.clusters.draw(count, &mut self.rng) + } +} + +fn fixture_with( + n: usize, + dim: usize, + ann: Option<AnnConfig>, + type_of: impl Fn(usize) -> String, +) -> Fixture { + let clusters = Clusters::new(dim, 40, 1); + let mut rng = Rng(77); + let data = clusters.draw(n, &mut rng); + let mut store = EmbeddingStore::new(None); + for (i, v) in data.iter().enumerate() { + store + .add( + NodeId::new(type_of(i), i.to_string()), + &format!("text {i}"), + Some(v.clone()), + ) + .unwrap(); + } + if let Some(cfg) = ann { + store.set_ann(Some(cfg)).unwrap(); + } + Fixture { + store, + data, + clusters, + rng, + } +} + +fn fixture(n: usize) -> Fixture { + fixture_with(n, 32, Some(config()), |_| "doc".to_string()) +} + +fn brute(data: &[Vec<f32>], query: &[f32], k: usize) -> Vec<(usize, f64)> { + let mut all: Vec<(usize, f64)> = data + .iter() + .enumerate() + .map(|(i, v)| (i, cosine(query, v))) + .collect(); + all.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap().then(a.0.cmp(&b.0))); + all.truncate(k); + all +} + +// -- configuration ------------------------------------------------------------ + +#[test] +fn off_by_default_and_exact_is_untouched() { + let mut fx = fixture_with(1500, 32, None, |_| "doc".to_string()); + assert!(fx.store.ann().is_none()); + let stats = fx.store.ann_stats(); + assert!(!stats.enabled && !stats.active); + assert!(matches!( + fx.store.build_ann_index(false), + Err(EmbeddingError::InvalidOption(_)) + )); + + // The same store with the index switched on and exact forced: identical, + // not merely close, because the exact path is the same code either way. + let mut other = fixture_with(1500, 32, Some(config()), |_| "doc".to_string()); + for q in fx.queries(5) { + let a = fx + .store + .similarity_search_vector(&q, 10, None, -1.0) + .unwrap(); + let b = other + .store + .similarity_search_vector_exact(&q, 10, None, -1.0) + .unwrap(); + assert_eq!(refs(&a), refs(&b)); + assert_eq!( + a.iter().map(|r| r.score).collect::<Vec<_>>(), + b.iter().map(|r| r.score).collect::<Vec<_>>() + ); + } + // and a store with ann off never records an approximate answer + assert_eq!(fx.store.ann_stats().searches.approximate, 0); + let _ = other.queries(1); +} + +#[test] +fn argument_validation() { + let mut store = EmbeddingStore::new(None); + let bad = [ + AnnConfig { + nlist: Some(0), + ..AnnConfig::default() + }, + AnnConfig { + nprobe: Some(0), + ..AnnConfig::default() + }, + ]; + for cfg in bad { + assert!(matches!( + store.set_ann(Some(cfg.clone())), + Err(EmbeddingError::InvalidOption(_)) + )); + assert!(EmbeddingStore::with_ann(None, cfg).is_err()); + } + assert!(store.ann().is_none(), "a refused config must not be kept"); + + store.set_ann(Some(AnnConfig::default())).unwrap(); + let cfg = store.ann().unwrap(); + assert_eq!(cfg.min_size, 50_000, "default minimum size matches Python"); + assert_eq!(cfg.seed, 0); + assert!(cfg.nlist.is_none() && cfg.nprobe.is_none()); + + assert!(store.set_ann_nprobe(Some(0)).is_err()); + store.set_ann_nprobe(Some(5)).unwrap(); + assert_eq!(store.ann().unwrap().nprobe, Some(5)); + store.set_ann(None).unwrap(); + assert!(store.ann().is_none()); +} + +// -- recall and exactness ----------------------------------------------------- + +#[test] +fn recall_floor_on_clustered_data() { + let mut fx = fixture(4000); + let queries = fx.queries(60); + let report = fx.store.recall_check(&queries, 10, None, 0.0).unwrap(); + assert!(report.approximate, "{report:?}"); + assert_eq!((report.nlist, report.nprobe), (64, 8)); + assert_eq!((report.queries, report.k), (60, 10)); + assert!(report.recall >= RECALL_FLOOR, "{report:?}"); + assert!(0.0 <= report.min_recall && report.min_recall <= report.recall && report.recall <= 1.0); + let stats = fx.store.ann_stats(); + assert!(stats.enabled && stats.active); + assert_eq!(stats.nlist, Some(64)); + assert_eq!(stats.trained_n, Some(4000)); + assert!(stats.index_bytes > 0); + assert!(stats.searches.approximate >= 60); +} + +#[test] +fn results_carry_exact_cosine_scores() { + let mut fx = fixture(4000); + for q in fx.queries(10) { + let got = fx + .store + .similarity_search_vector(&q, 10, None, 0.0) + .unwrap(); + assert_eq!(got.len(), 10); + for r in &got { + let i: usize = r.node_id.id.parse().unwrap(); + let want = cosine(&q, &fx.data[i]); + assert!( + (r.score as f64 - want).abs() < 1e-5, + "{} vs {}", + r.score, + want + ); + } + let scores: Vec<f32> = got.iter().map(|r| r.score).collect(); + assert!(scores.windows(2).all(|w| w[0] >= w[1])); + assert!(fx.store.ann_stats().searches.approximate > 0); + } +} + +#[test] +fn exact_true_equals_brute_force() { + let mut fx = fixture(4000); + for q in fx.queries(12) { + let got = fx + .store + .similarity_search_vector_exact(&q, 10, None, -1.0) + .unwrap(); + let want = brute(&fx.data, &q, 10); + assert_eq!(got.len(), 10); + let got_ids: HashSet<usize> = got.iter().map(|r| r.node_id.id.parse().unwrap()).collect(); + let want_ids: HashSet<usize> = want.iter().map(|w| w.0).collect(); + assert_eq!(got_ids, want_ids); + for (r, (_, score)) in got.iter().zip(&want) { + assert!((r.score as f64 - score).abs() < 1e-5); + } + } + assert_eq!( + fx.store.ann_stats().searches.approximate, + 0, + "exact must never consult the index" + ); + assert!(fx.store.ann_stats().searches.exact >= 12); +} + +#[test] +fn text_queries_take_the_same_path() { + let mut store = EmbeddingStore::new(Some(Box::new(MockEncoder::new(16)))); + for i in 0..300 { + store + .add(doc(i), &format!("item number {i}"), None) + .unwrap(); + } + store + .set_ann(Some(AnnConfig { + nlist: Some(8), + nprobe: Some(8), // every cell: must equal the exact answer + min_size: 0, + ..AnnConfig::default() + })) + .unwrap(); + let a = store + .similarity_search("item number 42", 5, None, -1.0) + .unwrap(); + let b = store + .similarity_search_exact("item number 42", 5, None, -1.0) + .unwrap(); + assert_eq!(refs(&a), refs(&b)); + assert_eq!(a[0].node_id, doc(42)); + assert!((a[0].score - 1.0).abs() < 1e-5); +} + +#[test] +fn probing_every_cell_is_the_exact_scan() { + let mut fx = fixture(4000); + fx.store.set_ann_nprobe(Some(64)).unwrap(); // nprobe == nlist + let reference = fixture_with(4000, 32, None, |_| "doc".to_string()); + for q in fx.queries(8) { + let got = fx + .store + .similarity_search_vector(&q, 10, None, 0.0) + .unwrap(); + let want = reference + .store + .similarity_search_vector(&q, 10, None, 0.0) + .unwrap(); + assert_eq!(refs(&got), refs(&want)); + assert_eq!( + got.iter().map(|r| r.score).collect::<Vec<_>>(), + want.iter().map(|r| r.score).collect::<Vec<_>>() + ); + } + let queries = fx.queries(5); + let report = fx.store.recall_check(&queries, 10, None, 0.0).unwrap(); + assert!(!report.approximate); + assert_eq!(report.recall, 1.0); +} + +#[test] +fn under_min_size_stays_exact() { + let mut fx = fixture_with( + 500, + 32, + Some(AnnConfig { + nlist: Some(16), + nprobe: Some(2), + min_size: 1000, + ..AnnConfig::default() + }), + |_| "doc".to_string(), + ); + let q = fx.queries(1).remove(0); + let a = fx + .store + .similarity_search_vector(&q, 10, None, 0.0) + .unwrap(); + let b = fx + .store + .similarity_search_vector_exact(&q, 10, None, 0.0) + .unwrap(); + assert_eq!(refs(&a), refs(&b)); + let stats = fx.store.ann_stats(); + assert!(!stats.active); + assert_eq!(stats.searches.approximate, 0); + assert_eq!(stats.searches.builds, 0, "no index is built below min_size"); + let report = fx.store.recall_check(&[q], 10, None, 0.0).unwrap(); + assert!(!report.approximate); + assert_eq!(report.recall, 1.0); + // build_ann_index says so in the stats instead of building + assert!(!fx.store.build_ann_index(false).unwrap().active); +} + +#[test] +fn unbounded_zero_and_oversized_k_are_exact() { + let mut fx = fixture(2000); + let q = fx.queries(1).remove(0); + let before = fx.store.ann_stats().searches.approximate; + let all = fx + .store + .similarity_search_vector(&q, usize::MAX, None, -1.0) + .unwrap(); + assert_eq!(all.len(), 2000); + assert!(all.windows(2).all(|w| w[0].score >= w[1].score)); + let k_ge_live = fx + .store + .similarity_search_vector(&q, 2500, None, -1.0) + .unwrap(); + assert_eq!(k_ge_live.len(), 2000); + assert!(fx + .store + .similarity_search_vector(&q, 0, None, -1.0) + .unwrap() + .is_empty()); + assert_eq!(fx.store.ann_stats().searches.approximate, before); +} + +#[test] +fn zero_query_falls_back_to_exact() { + let fx = fixture(2000); + let res = fx + .store + .similarity_search_vector(&[0.0; 32], 5, None, 0.0) + .unwrap(); + assert_eq!(res.len(), 5); + assert!(res.iter().all(|r| r.score == 0.0)); + assert_eq!(fx.store.ann_stats().searches.approximate, 0); +} + +#[test] +fn wrong_dimension_is_still_an_error() { + let fx = fixture(100); + assert!(matches!( + fx.store.similarity_search_vector(&[1.0, 2.0], 3, None, 0.0), + Err(EmbeddingError::DimensionMismatch { .. }) + )); + assert!(matches!( + fx.store + .similarity_search_vector_exact(&[1.0, 2.0], 3, None, 0.0), + Err(EmbeddingError::DimensionMismatch { .. }) + )); +} + +// -- upkeep under writes ------------------------------------------------------ + +#[test] +fn writes_are_visible_immediately() { + let mut fx = fixture(4000); + fx.store.build_ann_index(false).unwrap(); + let fresh = fx.queries(1).remove(0); + fx.store + .add(NodeId::new("new", "1"), "fresh", Some(fresh.clone())) + .unwrap(); + let top = fx + .store + .similarity_search_vector(&fresh, 3, None, 0.0) + .unwrap(); + assert_eq!(top[0].node_id, NodeId::new("new", "1")); + assert!((top[0].score - 1.0).abs() < 1e-5); + assert_eq!(fx.store.ann_stats().pending, 1); + + // An in-place update moves the row: found at its new place, scored from + // its new vector everywhere, and listed once. + let target = fx.queries(1).remove(0); + fx.store.add(doc(7), "moved", Some(target.clone())).unwrap(); + let hit = fx + .store + .similarity_search_vector(&target, 1, None, 0.0) + .unwrap(); + assert_eq!(hit[0].node_id, doc(7)); + assert!((hit[0].score - 1.0).abs() < 1e-5); + assert_eq!(hit[0].text, "moved"); + let old = fx.data[7].clone(); + for q in [old, target] { + let res = fx + .store + .similarity_search_vector(&q, 400, None, -1.0) + .unwrap(); + let ids = refs(&res); + assert_eq!(ids.len(), ref_set(&res).len(), "a stale second entry"); + let truth = fx + .store + .similarity_search_vector_exact(&q, usize::MAX, None, -1.0) + .unwrap(); + let truth: std::collections::HashMap<_, _> = + truth.iter().map(|r| (ref_of(r), r.score)).collect(); + for r in &res { + assert!((r.score - truth[&ref_of(r)]).abs() < 1e-5); + } + } + assert_eq!(fx.store.count(), 4001); +} + +#[test] +fn deletes_never_come_back() { + let mut fx = fixture(4000); + fx.store.build_ann_index(false).unwrap(); + let builds = fx.store.ann_stats().searches.builds; + let mut gone = HashSet::new(); + for i in (0..4000).step_by(3) { + // a third of them: forces the scan cache to compact + assert!(fx.store.remove(&doc(i))); + gone.insert(("doc".to_string(), i.to_string())); + } + for q in fx.queries(15) { + let got = ref_set( + &fx.store + .similarity_search_vector(&q, 10, None, 0.0) + .unwrap(), + ); + assert!(got.is_disjoint(&gone)); + // a tombstone must not score 0.0 and beat a negative cosine + let deep = ref_set( + &fx.store + .similarity_search_vector(&q, 250, None, -1.0) + .unwrap(), + ); + assert!(deep.is_disjoint(&gone)); + let every = ref_set( + &fx.store + .similarity_search_vector_exact(&q, usize::MAX, None, -1.0) + .unwrap(), + ); + assert!(every.is_disjoint(&gone)); + assert_eq!(every.len(), fx.store.count()); + } + let queries = fx.queries(40); + let report = fx.store.recall_check(&queries, 10, None, 0.0).unwrap(); + assert!(report.recall >= RECALL_FLOOR, "{report:?}"); + // 1,334 deletes is below the retrain threshold: followed, not retrained + assert_eq!(fx.store.ann_stats().searches.builds, builds); + assert!(fx.store.ann_stats().active); +} + +#[test] +fn pending_rows_fold_and_heavy_churn_retrains() { + let mut fx = fixture(3000); + fx.store.build_ann_index(false).unwrap(); + let builds = fx.store.ann_stats().searches.builds; + let extra = fx.queries(700); + for (i, v) in extra.iter().enumerate() { + fx.store + .add(NodeId::new("late", i.to_string()), "", Some(v.clone())) + .unwrap(); + } + // folding is done by the writes themselves, FOLD_BATCH rows at a time: 700 + // writes made two folds and left 188 rows waiting + let stats = fx.store.ann_stats(); + assert_eq!(stats.pending, 700 % FOLD_BATCH); + assert_eq!(stats.folds, (700 / FOLD_BATCH) as u64); + fx.store + .similarity_search_vector(&extra[0], 5, None, 0.0) + .unwrap(); // a search folds nothing and rebuilds nothing + let stats = fx.store.ann_stats(); + assert_eq!(stats.pending, 700 % FOLD_BATCH); + assert_eq!(stats.folds, (700 / FOLD_BATCH) as u64); + assert_eq!(stats.searches.builds, builds); + fx.store.build_ann_index(false).unwrap(); // folds the remainder on request + assert_eq!(fx.store.ann_stats().pending, 0); + + let more = fx.queries(1200); + for (i, v) in more.iter().enumerate() { + fx.store + .add(NodeId::new("later", i.to_string()), "", Some(v.clone())) + .unwrap(); + } + fx.store + .similarity_search_vector(&more[0], 5, None, 0.0) + .unwrap(); // churn 1900 > 50% of 3000 + assert_eq!(fx.store.ann_stats().searches.builds, builds + 1); + assert_eq!(fx.store.ann_stats().trained_n, Some(4900)); + let queries = fx.queries(30); + assert!( + fx.store + .recall_check(&queries, 10, None, 0.0) + .unwrap() + .recall + >= RECALL_FLOOR + ); +} + +#[test] +fn small_writes_stay_pending_until_enough() { + let mut fx = fixture(2000); + fx.store.build_ann_index(false).unwrap(); + for (i, v) in fx.queries(100).iter().enumerate() { + fx.store + .add(NodeId::new("w", i.to_string()), "", Some(v.clone())) + .unwrap(); + } + let q = fx.queries(1).remove(0); + fx.store.similarity_search_vector(&q, 5, None, 0.0).unwrap(); + // 100 < FOLD_BATCH: scored exactly, not yet folded, and no rebuild + let stats = fx.store.ann_stats(); + assert_eq!(stats.pending, 100); + assert_eq!(stats.churn, 100); + assert_eq!(stats.searches.builds, 1); +} + +#[test] +fn clear_and_dimension_change() { + let mut fx = fixture_with( + 1500, + 32, + Some(AnnConfig { + nlist: Some(16), + nprobe: Some(4), + min_size: 0, + ..AnnConfig::default() + }), + |_| "doc".to_string(), + ); + fx.store.build_ann_index(false).unwrap(); + let q = fx.data[0].clone(); + fx.store.clear_with(false); + assert!(fx + .store + .similarity_search_vector(&q, 5, None, 0.0) + .unwrap() + .is_empty()); + assert!(!fx.store.ann_stats().active); + + fx.store.clear(); + let small = Clusters::new(8, 12, 3); + let mut rng = Rng(5); + let vectors = small.draw(1200, &mut rng); + for (i, v) in vectors.iter().enumerate() { + fx.store.add(doc(i), "", Some(v.clone())).unwrap(); + } + let res = fx + .store + .similarity_search_vector(&vectors[5], 5, None, 0.0) + .unwrap(); + assert_eq!(res[0].node_id, doc(5)); + assert!(fx.store.ann_stats().active); + assert_eq!(fx.store.dimension(), Some(8)); +} + +#[test] +fn importing_a_payload_keeps_the_index_honest() { + let mut fx = fixture(3000); + fx.store.build_ann_index(false).unwrap(); + let payload = fx.store.export_embeddings(); + // a replace import discards the old rows (and so the old index) + fx.store.import_embeddings(&payload, true).unwrap(); + assert_eq!(fx.store.count(), 3000); + let q = fx.queries(1).remove(0); + let got = fx + .store + .similarity_search_vector(&q, 10, None, 0.0) + .unwrap(); + assert_eq!(got.len(), 10); + assert!(fx.store.ann_stats().active); + let queries = fx.queries(30); + assert!( + fx.store + .recall_check(&queries, 10, None, 0.0) + .unwrap() + .recall + >= RECALL_FLOOR + ); +} + +// -- filters ------------------------------------------------------------------ + +#[test] +fn node_type_filter_and_widening() { + let mut cfg = config(); + cfg.nprobe = Some(2); + let mut fx = fixture_with(4000, 32, Some(cfg), |i| { + if i % 200 == 0 { "rare" } else { "common" }.to_string() + }); + let q = fx.queries(1).remove(0); + let common = fx + .store + .similarity_search_vector(&q, 10, Some("common"), 0.0) + .unwrap(); + assert_eq!(common.len(), 10); + assert!(common.iter().all(|r| r.node_id.node_type == "common")); + + // 20 rare rows in the whole store and a top_k of 15: two probed cells + // cannot hold 15 of them, so the probe widens (ending in the scan). + let widened = fx.store.ann_stats().searches.widened; + let exact_before = fx.store.ann_stats().searches.exact; + let rare = fx + .store + .similarity_search_vector(&q, 15, Some("rare"), -1.0) + .unwrap(); + assert_eq!(rare.len(), 15); + assert!(rare.iter().all(|r| r.node_id.node_type == "rare")); + let truth = fx + .store + .similarity_search_vector_exact(&q, 15, Some("rare"), -1.0) + .unwrap(); + assert_eq!(rare.len(), truth.len()); + let stats = fx.store.ann_stats(); + assert!(stats.searches.widened > widened || stats.searches.exact > exact_before + 1); + + assert!(fx + .store + .similarity_search_vector(&q, 5, Some("nope"), -1.0) + .unwrap() + .is_empty()); +} + +#[test] +fn threshold_is_respected() { + let mut fx = fixture(4000); + let q = fx.queries(1).remove(0); + let best = fx + .store + .similarity_search_vector_exact(&q, 1, None, -1.0) + .unwrap()[0] + .score; + let res = fx + .store + .similarity_search_vector(&q, 10, None, best - 1e-3) + .unwrap(); + assert!(!res.is_empty() && res.iter().all(|r| r.score >= best - 1e-3)); + assert!(fx + .store + .similarity_search_vector(&q, 10, None, 2.0) + .unwrap() + .is_empty()); + // fewer than k above a threshold: the same short list as the scan + let cut = fx + .store + .similarity_search_vector_exact(&q, 10, None, -1.0) + .unwrap()[4] + .score; + let got = fx + .store + .similarity_search_vector(&q, 10, None, cut) + .unwrap(); + let want = fx + .store + .similarity_search_vector_exact(&q, 10, None, cut) + .unwrap(); + assert_eq!(ref_set(&got), ref_set(&want)); +} + +// -- determinism -------------------------------------------------------------- + +#[test] +fn a_fixed_seed_gives_the_same_index_every_time() { + let one_cell = |seed: u64| { + let mut cfg = config(); + cfg.nprobe = Some(1); + cfg.seed = seed; + fixture_with(3000, 32, Some(cfg), |_| "doc".to_string()) + }; + let mut a = one_cell(9); + let b = one_cell(9); + let queries = a.queries(40); + for q in &queries { + let ra = a.store.similarity_search_vector(q, 10, None, -1.0).unwrap(); + let rb = b.store.similarity_search_vector(q, 10, None, -1.0).unwrap(); + assert_eq!(refs(&ra), refs(&rb)); + assert_eq!( + ra.iter().map(|r| r.score).collect::<Vec<_>>(), + rb.iter().map(|r| r.score).collect::<Vec<_>>() + ); + } + let (sa, sb) = (a.store.ann_stats(), b.store.ann_stats()); + assert_eq!( + (sa.nlist, sa.trained_n, sa.index_bytes), + (sb.nlist, sb.trained_n, sb.index_bytes) + ); + + // retraining the same data with the same seed reproduces the answers + let before: Vec<_> = queries + .iter() + .map(|q| refs(&a.store.similarity_search_vector(q, 10, None, -1.0).unwrap())) + .collect(); + a.store.build_ann_index(true).unwrap(); + let after: Vec<_> = queries + .iter() + .map(|q| refs(&a.store.similarity_search_vector(q, 10, None, -1.0).unwrap())) + .collect(); + assert_eq!(before, after); + assert_eq!(a.store.ann_stats().searches.builds, 2); +} + +#[test] +fn the_thread_count_never_changes_an_answer() { + // 256 dimensions put 2,048 rows in a scan block, so 7,000 rows are four + // blocks and eight threads really do share the work. + let mut cfg = config(); + cfg.nprobe = Some(3); + let build = || fixture_with(7000, 256, Some(cfg.clone()), |_| "doc".to_string()); + let (mut single, mut many) = (build(), build()); + single.store.set_scan_threads(1); + many.store.set_scan_threads(8); + assert_eq!( + (single.store.scan_threads(), many.store.scan_threads()), + (1, 8) + ); + for q in single.queries(10) { + for k in [10, 25, 3000] { + let a = single + .store + .similarity_search_vector_exact(&q, k, None, -1.0) + .unwrap(); + let b = many + .store + .similarity_search_vector_exact(&q, k, None, -1.0) + .unwrap(); + assert_eq!(refs(&a), refs(&b), "k={k}"); + assert_eq!( + a.iter().map(|r| r.score).collect::<Vec<_>>(), + b.iter().map(|r| r.score).collect::<Vec<_>>() + ); + } + let a = single + .store + .similarity_search_vector(&q, 10, None, -1.0) + .unwrap(); + let b = many + .store + .similarity_search_vector(&q, 10, None, -1.0) + .unwrap(); + assert_eq!(refs(&a), refs(&b)); + } + assert_eq!(single.store.ann_stats().nlist, many.store.ann_stats().nlist); +} + +#[test] +fn awkward_dimensions_work_end_to_end() { + // 37 is neither a multiple of 8 nor of 32, and nlist 13 is not a multiple + // of the kernel's group of four. + let mut fx = fixture_with( + 1200, + 37, + Some(AnnConfig { + nlist: Some(13), + nprobe: Some(13), + min_size: 0, + ..AnnConfig::default() + }), + |_| "doc".to_string(), + ); + for q in fx.queries(6) { + let got = fx + .store + .similarity_search_vector_exact(&q, 10, None, -1.0) + .unwrap(); + let want = brute(&fx.data, &q, 10); + let ids: HashSet<usize> = got.iter().map(|r| r.node_id.id.parse().unwrap()).collect(); + assert_eq!(ids, want.iter().map(|w| w.0).collect::<HashSet<_>>()); + } + // Thirteen coarse cells over forty clusters: recall climbs with nprobe. + let queries = fx.queries(30); + let mut last = 0.0; + for nprobe in [2, 6, 10] { + fx.store.set_ann_nprobe(Some(nprobe)).unwrap(); + let report = fx.store.recall_check(&queries, 10, None, 0.0).unwrap(); + assert!( + report.approximate && report.nlist == 13 && report.nprobe == nprobe, + "{report:?}" + ); + assert!( + report.recall >= last, + "recall fell as nprobe rose: {report:?}" + ); + last = report.recall; + } + assert!(last >= RECALL_FLOOR, "nprobe 10 of 13 gave {last}"); +} + +#[test] +fn equal_scores_break_in_a_stable_order() { + // Identical vectors score identically; the answer must not depend on the + // run (the store used to iterate a HashMap, so it did). + let mut order = None; + for _ in 0..5 { + let mut store = EmbeddingStore::new(None); + for i in 0..50 { + store + .add(doc(i), "", Some(vec![1.0, 2.0, 3.0, 4.0])) + .unwrap(); + } + let got = refs( + &store + .similarity_search_vector(&[1.0, 2.0, 3.0, 4.0], 7, None, 0.0) + .unwrap(), + ); + match &order { + None => order = Some(got), + Some(first) => assert_eq!(&got, first), + } + } + assert_eq!(order.unwrap().len(), 7); +} + +// -- recall_check ------------------------------------------------------------- + +#[test] +fn recall_check_shapes_and_errors() { + let mut fx = fixture(2000); + let q = fx.queries(3); + assert_eq!( + fx.store + .recall_check(&q[..1], 10, None, 0.0) + .unwrap() + .queries, + 1 + ); + assert_eq!(fx.store.recall_check(&q, 10, None, 0.0).unwrap().queries, 3); + assert_eq!( + fx.store + .recall_check(q.iter().map(|v| v.as_slice()), 5, None, 0.0) + .unwrap() + .k, + 5 + ); + let empty: [&[f32]; 0] = []; + let none = fx.store.recall_check(empty, 10, None, 0.0).unwrap(); + assert_eq!((none.queries, none.recall, none.min_recall), (0, 1.0, 1.0)); + assert!(matches!( + fx.store.recall_check(&q, 0, None, 0.0), + Err(EmbeddingError::InvalidOption(_)) + )); + + // without an index in use both answers are the scan + let off = fixture_with(100, 32, None, |_| "doc".to_string()); + let r = off.store.recall_check(&q, 10, None, 0.0).unwrap(); + assert_eq!(r.recall, 1.0); + assert!(!r.approximate); + assert!(r.ann_ms >= 0.0 && r.exact_ms >= 0.0); +} + +#[test] +fn concurrent_readers_agree() { + let mut fx = fixture(3000); + fx.store.build_ann_index(false).unwrap(); + let queries = fx.queries(24); + let store = std::sync::Arc::new(fx.store); + let want: Vec<_> = queries + .iter() + .map(|q| refs(&store.similarity_search_vector(q, 8, None, -1.0).unwrap())) + .collect(); + std::thread::scope(|s| { + for t in 0..4 { + let (store, queries, want) = (&store, &queries, &want); + s.spawn(move || { + for i in 0..60 { + let at = (i + t) % queries.len(); + let res = store + .similarity_search_vector(&queries[at], 8, None, -1.0) + .unwrap(); + assert_eq!(refs(&res), want[at]); + } + }); + } + }); +} + +// -- through the graph -------------------------------------------------------- + +fn graph_with( + n: usize, + dim: usize, + ann: Option<AnnConfig>, +) -> (SemanticGraph, Vec<Vec<f32>>, Clusters, Rng) { + let clusters = Clusters::new(dim, 40, 1); + let mut rng = Rng(11); + let data = clusters.draw(n, &mut rng); + let mut graph = SemanticGraph::with_options( + "ann", + Box::new(MockEncoder::new(dim)), + false, + vec!["name".to_string()], + ); + for (i, v) in data.iter().enumerate() { + graph + .add_node( + "item", + &i.to_string(), + vec![("group", PropertyValue::Int((i % 50) as i64))], + None, + ) + .unwrap(); + graph + .embedding_store_mut() + .add( + NodeId::new("item", i.to_string()), + &format!("item {i}"), + Some(v.clone()), + ) + .unwrap(); + } + if let Some(cfg) = ann { + graph.set_ann(Some(cfg)).unwrap(); + } + (graph, data, clusters, rng) +} + +fn graph_config() -> AnnConfig { + AnnConfig { + nlist: Some(48), + nprobe: Some(6), + min_size: 0, + ..AnnConfig::default() + } +} + +fn group_of(graph: &SemanticGraph, r: &SimilarityResult) -> i64 { + match graph + .graph() + .get_node(&r.node_id.node_type, &r.node_id.id) + .unwrap() + .properties + .get("group") + { + Some(PropertyValue::Int(i)) => *i, + other => panic!("group was {other:?}"), + } +} + +#[test] +fn graph_passthrough_and_recall() { + let (graph, _, clusters, mut rng) = graph_with(3000, 32, Some(graph_config())); + assert_eq!(graph.ann().unwrap().nprobe, Some(6)); + let q = clusters.draw(1, &mut rng).remove(0); + let res = graph + .similarity_search_with(&q, &SimilarityOptions::default()) + .unwrap(); + assert_eq!(res.len(), 10); + assert!(graph.ann_stats().active); + let queries = clusters.draw(20, &mut rng); + assert!(graph.recall_check(&queries, 10, None, 0.0).unwrap().recall >= RECALL_FLOOR); + + let (plain, ..) = graph_with(50, 32, None); + assert!(plain.ann().is_none()); +} + +#[test] +fn graph_exact_flag_and_similar_to_node() { + let (graph, _, clusters, mut rng) = graph_with(3000, 32, Some(graph_config())); + let q = clusters.draw(1, &mut rng).remove(0); + let before = graph.ann_stats().searches.approximate; + let exact = SimilarityOptions { + top_k: Some(5), + exact: true, + ..SimilarityOptions::default() + }; + graph.similarity_search_with(&q, &exact).unwrap(); + let pred = |_: &Node| true; + graph + .similarity_search_with( + &q, + &SimilarityOptions { + filter: NodeFilter::new().predicate(&pred), + ..exact.clone() + }, + ) + .unwrap(); + assert_eq!(graph.ann_stats().searches.approximate, before); + let near = graph + .similar_to_node(&NodeId::new("item", "3"), 5, None, 0.0) + .unwrap(); + assert_eq!(near.len(), 5); + assert!(near.iter().all(|r| r.node_id != NodeId::new("item", "3"))); +} + +#[test] +fn node_filter_gets_k_results_even_when_selective() { + let (graph, _, clusters, mut rng) = graph_with(3000, 32, Some(graph_config())); + let q = clusters.draw(1, &mut rng).remove(0); + let seven = |n: &Node| matches!(n.properties.get("group"), Some(PropertyValue::Int(7))); + // 1 node in 50 passes: the probed cells hold few, so the search widens + let filter = NodeFilter::new().predicate(&seven); + let res = graph + .similarity_search_with( + &q, + &SimilarityOptions { + filter: filter.clone(), + ..SimilarityOptions::default() + }, + ) + .unwrap(); + assert_eq!(res.len(), 10); + assert!(res.iter().all(|r| group_of(&graph, r) == 7)); + assert!(res.windows(2).all(|w| w[0].score >= w[1].score)); + let want = graph + .similarity_search_with( + &q, + &SimilarityOptions { + filter: filter.clone(), + exact: true, + ..SimilarityOptions::default() + }, + ) + .unwrap(); + assert!(!want.is_empty()); + // The widened search finds most of the exact answers: judged over many + // queries, because one query's overlap depends on which sample the index + // was trained on (the old single-query bound of 7 of 10 held only for the + // sample that the sorted-key row order happened to give). + let (mut overlap, mut asked) = (0, 0); + for q in clusters.draw(10, &mut rng) { + let res = graph + .similarity_search_with( + &q, + &SimilarityOptions { + filter: filter.clone(), + ..SimilarityOptions::default() + }, + ) + .unwrap(); + let want = graph + .similarity_search_with( + &q, + &SimilarityOptions { + filter: filter.clone(), + exact: true, + ..SimilarityOptions::default() + }, + ) + .unwrap(); + overlap += ref_set(&res).intersection(&ref_set(&want)).count(); + asked += want.len(); + } + assert!(overlap * 100 >= asked * 80, "{overlap} of {asked}"); + + // a filter nothing passes terminates, empty + let never = |_: &Node| false; + let none = graph + .similarity_search_with( + &q, + &SimilarityOptions { + filter: NodeFilter::new().predicate(&never), + ..SimilarityOptions::default() + }, + ) + .unwrap(); + assert!(none.is_empty()); +} + +#[test] +fn multi_hop_seeds_use_the_index_unless_blending() { + let (mut graph, _, clusters, mut rng) = graph_with(2000, 32, Some(graph_config())); + for i in 0..1999 { + graph + .add_edge( + "NEXT", + ("item", &i.to_string()), + ("item", &(i + 1).to_string()), + vec![], + ) + .unwrap(); + } + let q = clusters.draw(1, &mut rng).remove(0); + let opts = MultiHopOptions { + rel_type: Some("NEXT"), + max_hops: 2, + top_k_seeds: 3, + threshold: -1.0, + ..MultiHopOptions::default() + }; + let out = graph + .semantic_multi_hop_search(&q, &SearchOptions::from(opts.clone())) + .unwrap(); + assert!(!out.is_empty() && out[0].hop_count == 0); + assert!(graph.ann_stats().searches.approximate >= 1); + + // blend > 0 scores every node and must not touch the approximate path + let before = graph.ann_stats().searches.approximate; + let blended = MultiHopOptions { + blend: 0.3, + max_hops: 1, + ..opts + }; + graph + .semantic_multi_hop_search(&q, &SearchOptions::from(blended)) + .unwrap(); + assert_eq!(graph.ann_stats().searches.approximate, before); +} + +#[test] +fn a_loaded_graph_rebuilds_lazily_and_persists_nothing() { + let (graph, _, clusters, mut rng) = graph_with(1500, 32, Some(graph_config())); + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("g.ison"); + graph.save(&path, None, true).unwrap(); + let mut loaded = SemanticGraph::load(&path, Box::new(MockEncoder::new(32))).unwrap(); + loaded.set_ann(Some(graph_config())).unwrap(); + assert!(!loaded.ann_stats().active, "nothing was persisted"); + let q = clusters.draw(1, &mut rng).remove(0); + assert_eq!( + loaded + .similarity_search_with(&q, &SimilarityOptions::default()) + .unwrap() + .len(), + 10 + ); + assert!(loaded.ann_stats().active); +} + +// -- the index never reaches a file or the digest ----------------------------- + +#[test] +fn graph_files_sidecar_and_digest_are_the_same_with_ann_on() { + let (off, ..) = graph_with(600, 16, None); + let (on, _, clusters, mut rng) = graph_with(600, 16, Some(graph_config())); + // exercise the index hard before saving: build, write, delete, search + let q = clusters.draw(1, &mut rng).remove(0); + on.similarity_search_with(&q, &SimilarityOptions::default()) + .unwrap(); + assert!(on.ann_stats().active); + + let dir = tempfile::tempdir().unwrap(); + let (p_off, p_on) = (dir.path().join("off.ison"), dir.path().join("on.ison")); + off.save(&p_off, None, true).unwrap(); + on.save(&p_on, None, true).unwrap(); + assert_eq!( + std::fs::read(&p_off).unwrap(), + std::fs::read(&p_on).unwrap() + ); + + let (s_off, s_on) = ( + SemanticGraph::embeddings_path_for(&p_off), + SemanticGraph::embeddings_path_for(&p_on), + ); + let (side_off, side_on) = ( + std::fs::read(&s_off).unwrap(), + std::fs::read(&s_on).unwrap(), + ); + assert_eq!(side_off, side_on, "the sidecar must be byte-identical"); + let text = String::from_utf8(side_on).unwrap().to_lowercase(); + for word in ["ivf", "nlist", "nprobe", "centroid", "\"ann"] { + assert!(!text.contains(word), "sidecar mentions {word}"); + } + assert_eq!(graph_digest(off.graph()), graph_digest(on.graph())); + + // and the directory holds exactly the two files, nothing for the index + let mut names: Vec<_> = std::fs::read_dir(dir.path()) + .unwrap() + .map(|e| e.unwrap().file_name().to_string_lossy().into_owned()) + .collect(); + names.sort(); + assert_eq!( + names, + vec![ + "off.ison", + "off.ison.embeddings.json", + "on.ison", + "on.ison.embeddings.json" + ] + ); + + // the exported payload is also unchanged by building the index + let a = off.embedding_store().to_json().unwrap(); + let b = on.embedding_store().to_json().unwrap(); + assert_eq!(a, b); +} + +// -- the graph keeps the index honest ----------------------------------------- + +#[test] +fn graph_edits_reach_the_index() { + // Auto-embedded nodes: add, update (re-embeds), remove, with the index on. + let mut graph = SemanticGraph::new("g", Box::new(MockEncoder::new(48))); + for i in 0..300 { + graph + .add_node( + "doc", + &i.to_string(), + vec![( + "name", + PropertyValue::String(format!("document number {i}")), + )], + None, + ) + .unwrap(); + } + graph + .set_ann(Some(AnnConfig { + nlist: Some(10), + nprobe: Some(10), // every cell: the index must agree with the scan + min_size: 0, + ..AnnConfig::default() + })) + .unwrap(); + let top = |graph: &SemanticGraph, text: &str| { + graph + .similarity_search_with( + text, + &SimilarityOptions { + top_k: Some(3), + threshold: -1.0, + ..SimilarityOptions::default() + }, + ) + .unwrap() + }; + assert_eq!(top(&graph, "document number 5")[0].node_id, doc(5)); + + // an edit moves the node's vector: found by its new text, not its old + graph + .update_node( + "doc", + "5", + vec![( + "name", + PropertyValue::String("completely different words".into()), + )], + ) + .unwrap(); + assert_eq!(top(&graph, "completely different words")[0].node_id, doc(5)); + assert_ne!(top(&graph, "document number 5")[0].node_id, doc(5)); + + // a removal is gone from every result + graph.remove_node("doc", "9").unwrap(); + let all = graph + .similarity_search_with( + "document number 9", + &SimilarityOptions { + top_k: None, + threshold: -1.0, + ..SimilarityOptions::default() + }, + ) + .unwrap(); + assert_eq!(all.len(), 299); + assert!(all.iter().all(|r| r.node_id != doc(9))); +} diff --git a/isongraph-vector-rs/tests/ann_sqlite.rs b/isongraph-vector-rs/tests/ann_sqlite.rs new file mode 100644 index 0000000..94fc68f --- /dev/null +++ b/isongraph-vector-rs/tests/ann_sqlite.rs @@ -0,0 +1,173 @@ +//! The approximate index and the SQLite store, behind the `sqlite` feature: +//! `cargo test --features sqlite`. +//! +//! `SqliteEmbeddingStore` is a separate, persistent store with no approximate +//! index of its own (its search is a scan, or sqlite-vec's exact KNN). What is +//! worth pinning down is the seam between the two: vectors that live in SQLite +//! can be loaded into an in-memory store with `ann` on, the index answers +//! from the same scores the SQLite store gives, and nothing about the index +//! leaks back into the portable payload. + +#![cfg(feature = "sqlite")] + +use ison_graph_rs::NodeId; +use isongraph_vector_rs::{AnnConfig, EmbeddingStore, ExportedEmbedding, SqliteEmbeddingStore}; + +fn vector(i: usize, dim: usize) -> Vec<f32> { + // Ten loose clusters: a fixed direction per cluster plus a small, + // deterministic wobble per vector. + let mut state = (i as u64 + 1).wrapping_mul(0x9E37_79B9_7F4A_7C15); + let mut next = || { + state ^= state << 13; + state ^= state >> 7; + state ^= state << 17; + (state >> 40) as f32 / (1u64 << 24) as f32 - 0.5 + }; + let cluster = i % 10; + (0..dim) + .map(|d| if d % 10 == cluster { 3.0 } else { 0.0 } + 0.4 * next()) + .collect() +} + +#[test] +fn vectors_from_sqlite_are_searched_by_the_index_with_the_same_scores() { + let dim = 40; + let mut sqlite = SqliteEmbeddingStore::open_in_memory(None, false).unwrap(); + for i in 0..1500 { + sqlite + .add(NodeId::new("doc", i.to_string()), "", Some(vector(i, dim))) + .unwrap(); + } + let payload = sqlite.export_embeddings().unwrap(); + + let mut indexed = EmbeddingStore::with_ann( + None, + AnnConfig { + nlist: Some(20), + nprobe: Some(4), + min_size: 0, + ..AnnConfig::default() + }, + ) + .unwrap(); + assert_eq!(indexed.import_embeddings(&payload, true).unwrap(), 1500); + assert_eq!(indexed.count(), sqlite.count().unwrap()); + + let queries: Vec<Vec<f32>> = (0..20).map(|i| vector(5000 + i, dim)).collect(); + let report = indexed.recall_check(&queries, 10, None, 0.0).unwrap(); + assert!(report.approximate, "{report:?}"); + assert!(report.recall >= 0.95, "{report:?}"); + + for q in &queries { + let want = sqlite.similarity_search_vector(q, 10, None, 0.0).unwrap(); + // the full scan agrees with SQLite on the whole ranking, scores included + let exact = indexed + .similarity_search_vector_exact(q, 10, None, 0.0) + .unwrap(); + assert_eq!(want.len(), exact.len()); + for (a, b) in want.iter().zip(&exact) { + assert!( + (a.score - b.score).abs() < 1e-5, + "{} vs {}", + a.score, + b.score + ); + } + // and every approximate hit carries the score SQLite gives that node + let truth: std::collections::HashMap<_, _> = sqlite + .similarity_search_vector(q, usize::MAX, None, -1.0) + .unwrap() + .into_iter() + .map(|r| (r.node_id, r.score)) + .collect(); + for r in indexed.similarity_search_vector(q, 10, None, 0.0).unwrap() { + assert!((r.score - truth[&r.node_id]).abs() < 1e-5); + } + } +} + +#[test] +fn the_index_changes_nothing_that_is_exported() { + let dim = 24; + let mut sqlite = SqliteEmbeddingStore::open_in_memory(None, false).unwrap(); + let mut plain = EmbeddingStore::new(None); + let mut indexed = EmbeddingStore::with_ann( + None, + AnnConfig { + nlist: Some(8), + nprobe: Some(2), + min_size: 0, + ..AnnConfig::default() + }, + ) + .unwrap(); + for i in 0..400 { + let id = NodeId::new("doc", i.to_string()); + let v = vector(i, dim); + sqlite.add(id.clone(), "t", Some(v.clone())).unwrap(); + plain.add(id.clone(), "t", Some(v.clone())).unwrap(); + indexed.add(id, "t", Some(v)).unwrap(); + } + // build the index, then write and delete through it + indexed + .similarity_search_vector(&vector(3, dim), 5, None, 0.0) + .unwrap(); + assert!(indexed.ann_stats().active); + for store in [&mut plain, &mut indexed] { + store + .add(NodeId::new("doc", "7"), "t", Some(vector(900, dim))) + .unwrap(); + store.remove(&NodeId::new("doc", "8")); + } + sqlite + .add(NodeId::new("doc", "7"), "t", Some(vector(900, dim))) + .unwrap(); + sqlite.remove(&NodeId::new("doc", "8")).unwrap(); + + assert_eq!(plain.to_json().unwrap(), indexed.to_json().unwrap()); + // the SQLite store exports the same set of vectors + let (a, b) = ( + indexed.export_embeddings(), + sqlite.export_embeddings().unwrap(), + ); + assert_eq!(a.count, b.count); + let rows = |e: &[ExportedEmbedding]| { + let mut rows: Vec<_> = e + .iter() + .map(|e| ((e.node_type.clone(), e.id.as_string()), e.vector.clone())) + .collect(); + rows.sort_by(|x, y| x.0.cmp(&y.0)); + rows + }; + assert_eq!(rows(&a.embeddings), rows(&b.embeddings)); +} + +#[test] +fn ann_and_the_sqlite_vec_index_are_independent() { + // The Python port refuses ann together with sqlite_vec because one store + // would own two indexes. Here they are different types: a SQLite store + // (with or without vec0) never consults the in-memory index, and turning + // the index on in memory does not touch it. + let mut sqlite = SqliteEmbeddingStore::open_in_memory(None, true).unwrap(); + let mut indexed = EmbeddingStore::new(None); + for i in 0..300 { + let id = NodeId::new("doc", i.to_string()); + sqlite.add(id.clone(), "", Some(vector(i, 16))).unwrap(); + indexed.add(id, "", Some(vector(i, 16))).unwrap(); + } + indexed + .set_ann(Some(AnnConfig { + min_size: 0, + nlist: Some(10), + nprobe: Some(10), + ..AnnConfig::default() + })) + .unwrap(); + let q = vector(12, 16); + let a = sqlite.similarity_search_vector(&q, 5, None, 0.0).unwrap(); + let b = indexed.similarity_search_vector(&q, 5, None, 0.0).unwrap(); + assert_eq!( + a.iter().map(|r| r.node_id.clone()).collect::<Vec<_>>(), + b.iter().map(|r| r.node_id.clone()).collect::<Vec<_>>() + ); +} diff --git a/isongraph-vector-rs/tests/engine.rs b/isongraph-vector-rs/tests/engine.rs new file mode 100644 index 0000000..e641a0a --- /dev/null +++ b/isongraph-vector-rs/tests/engine.rs @@ -0,0 +1,836 @@ +//! The engine guarantees: batch search equals per-query search exactly, +//! readers run concurrently and never see a half-applied write, a write leaves +//! no backlog for the next search, and deletes stay deleted. DESIGN.md states +//! the guarantees; these tests hold the code to them. + +use isongraph_vector_rs::{ + AnnConfig, EmbeddingStore, MockEncoder, NodeId, SearchPool, SharedEmbeddingStore, + SimilarityResult, FOLD_BATCH, +}; +use std::collections::{HashMap, HashSet}; +use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; +use std::sync::Arc; +use std::time::{Duration, Instant}; + +// -- data --------------------------------------------------------------------- + +struct Rng(u64); + +impl Rng { + fn next_u64(&mut self) -> u64 { + self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } + fn unit(&mut self) -> f32 { + (self.next_u64() >> 40) as f32 / (1u64 << 24) as f32 + } + fn below(&mut self, n: usize) -> usize { + (self.next_u64() % n as u64) as usize + } + fn vector(&mut self, dim: usize) -> Vec<f32> { + (0..dim).map(|_| self.unit() * 2.0 - 1.0).collect() + } +} + +fn id(i: usize) -> NodeId { + let ty = ["alpha", "beta", "gamma"][i % 3]; + NodeId::new(ty, i.to_string()) +} + +fn key(r: &SimilarityResult) -> (String, String) { + (r.node_id.node_type.clone(), r.node_id.id.clone()) +} + +fn keys(rs: &[SimilarityResult]) -> Vec<(String, String)> { + rs.iter().map(key).collect() +} + +fn scores_bits(rs: &[SimilarityResult]) -> Vec<u32> { + rs.iter().map(|r| r.score.to_bits()).collect() +} + +fn cosine(a: &[f32], b: &[f32]) -> f64 { + let (mut d, mut na, mut nb) = (0.0f64, 0.0f64, 0.0f64); + for (x, y) in a.iter().zip(b) { + d += *x as f64 * *y as f64; + na += *x as f64 * *x as f64; + nb += *y as f64 * *y as f64; + } + if na == 0.0 || nb == 0.0 { + 0.0 + } else { + d / (na.sqrt() * nb.sqrt()) + } +} + +/// A store with `n` vectors in three node types, some identical (ties), some +/// updated and some deleted, so tombstones and rewritten rows are in play. +fn messy_store(n: usize, dim: usize, ann: Option<AnnConfig>, seed: u64) -> EmbeddingStore { + let mut rng = Rng(seed); + let mut store = EmbeddingStore::new(None); + for i in 0..n { + let v = if i % 17 == 0 && i > 0 { + // an exact duplicate of an earlier row: equal scores for every query + store.get_vector(&id(i - 1)).unwrap().to_vec() + } else { + rng.vector(dim) + }; + store.add(id(i), &format!("text {i}"), Some(v)).unwrap(); + } + for i in (0..n).step_by(11) { + store + .add(id(i), "rewritten", Some(rng.vector(dim))) + .unwrap(); + } + for i in (3..n).step_by(13) { + store.remove(&id(i)); + } + if let Some(cfg) = ann { + store.set_ann(Some(cfg)).unwrap(); + store.build_ann_index(false).unwrap(); + // writes after the build leave rows pending and rows folded + for i in n..n + 300 { + store.add(id(i), "late", Some(rng.vector(dim))).unwrap(); + } + for i in (0..n).step_by(29) { + store.remove(&id(i)); + } + } + store.verify_invariants().unwrap(); + store +} + +fn queries(count: usize, dim: usize, seed: u64) -> Vec<Vec<f32>> { + let mut rng = Rng(seed); + let mut qs: Vec<Vec<f32>> = (0..count).map(|_| rng.vector(dim)).collect(); + qs[1] = vec![0.0; dim]; // a query with no direction + qs[2] = qs[3].clone(); // two identical queries + qs +} + +// -- batch search equals per-query search -------------------------------------- + +#[test] +fn batch_answers_equal_single_answers_bit_for_bit() { + let ann = AnnConfig { + nlist: Some(24), + nprobe: Some(3), + min_size: 0, + ..AnnConfig::default() + }; + for dim in [8usize, 37, 100] { + for use_ann in [false, true] { + let mut store = messy_store(1500, dim, use_ann.then(|| ann.clone()), 7 + dim as u64); + let qs = queries(37, dim, 99); + for threads in [1usize, 0, 8] { + store.set_scan_threads(threads); + for exact in [false, true] { + for (k, ty, thr) in [ + (1usize, None, -1.0f32), + (10, None, -1.0), + (10, Some("beta"), -1.0), + (25, None, 0.15), + (7, Some("nowhere"), -1.0), + (100_000, None, -1.0), // "everything" + ] { + let batch = if exact { + store.similarity_search_vector_exact_batch(&qs, k, ty, thr) + } else { + store.similarity_search_vector_batch(&qs, k, ty, thr) + } + .unwrap(); + assert_eq!(batch.len(), qs.len()); + for (i, q) in qs.iter().enumerate() { + let single = if exact { + store.similarity_search_vector_exact(q, k, ty, thr) + } else { + store.similarity_search_vector(q, k, ty, thr) + } + .unwrap(); + let ctx = format!( + "dim {dim} ann {use_ann} threads {threads} exact {exact} k {k} type {ty:?} q {i}" + ); + assert_eq!(keys(&batch[i]), keys(&single), "{ctx}"); + assert_eq!(scores_bits(&batch[i]), scores_bits(&single), "{ctx}"); + assert_eq!( + batch[i].iter().map(|r| &r.text).collect::<Vec<_>>(), + single.iter().map(|r| &r.text).collect::<Vec<_>>(), + "{ctx}" + ); + } + } + } + } + } + } +} + +#[test] +fn text_batches_encode_together_and_match_single_searches() { + let mut store = EmbeddingStore::new(Some(Box::new(MockEncoder::new(32)))); + for i in 0..400 { + store + .add( + id(i), + &format!("document number {i} about topic {}", i % 9), + None, + ) + .unwrap(); + } + let texts = [ + "topic 3", + "document number 77", + "nothing like it", + "topic 3", + ]; + let batch = store + .similarity_search_batch(&texts, 5, None, -1.0) + .unwrap(); + assert_eq!(batch.len(), texts.len()); + for (t, got) in texts.iter().zip(&batch) { + let single = store.similarity_search(t, 5, None, -1.0).unwrap(); + assert_eq!(keys(got), keys(&single)); + assert_eq!(scores_bits(got), scores_bits(&single)); + } + assert!(store + .similarity_search_batch(&[], 5, None, -1.0) + .unwrap() + .is_empty()); + let none = EmbeddingStore::new(None); + assert!(none.similarity_search_batch(&["x"], 5, None, -1.0).is_err()); +} + +#[test] +fn batch_edge_cases() { + let store = messy_store(200, 12, None, 5); + let qs = queries(5, 12, 1); + // zero results asked for, and queries of the wrong width + let none = store + .similarity_search_vector_batch(&qs, 0, None, -1.0) + .unwrap(); + assert!(none.iter().all(|r| r.is_empty())); + let bad = vec![vec![1.0f32; 3]]; + assert!(store + .similarity_search_vector_batch(&bad, 5, None, -1.0) + .is_err()); + let empty: Vec<Vec<f32>> = Vec::new(); + assert!(store + .similarity_search_vector_batch(&empty, 5, None, -1.0) + .unwrap() + .is_empty()); + // an empty store answers every query with nothing + let blank = EmbeddingStore::new(None); + let got = blank.similarity_search_vector_batch(&[vec![1.0f32; 4]], 5, None, -1.0); + assert!(got.unwrap()[0].is_empty()); + // more queries than one pass handles + let many: Vec<Vec<f32>> = queries(4200, 12, 3); + let big = store + .similarity_search_vector_batch(&many, 3, None, -1.0) + .unwrap(); + assert_eq!(big.len(), 4200); + for i in [0usize, 1, 2049, 4095, 4096, 4199] { + let single = store + .similarity_search_vector(&many[i], 3, None, -1.0) + .unwrap(); + assert_eq!(keys(&big[i]), keys(&single), "query {i}"); + } +} + +#[test] +fn a_dedicated_pool_gives_the_same_answers() { + let mut store = messy_store(1200, 24, None, 3); + let qs = queries(20, 24, 4); + let want = store + .similarity_search_vector_batch(&qs, 10, None, -1.0) + .unwrap(); + store.set_search_pool(Some(Arc::new(SearchPool::new(3)))); + assert_eq!(store.search_pool().threads(), 3); + let got = store + .similarity_search_vector_batch(&qs, 10, None, -1.0) + .unwrap(); + for (a, b) in want.iter().zip(&got) { + assert_eq!(keys(a), keys(b)); + assert_eq!(scores_bits(a), scores_bits(b)); + } +} + +// -- readers run concurrently --------------------------------------------------- + +/// Queries per second from `threads` callers of the single-query search. +fn throughput(store: &EmbeddingStore, qs: &[Vec<f32>], threads: usize, window: Duration) -> f64 { + let stop = AtomicBool::new(false); + let done = AtomicUsize::new(0); + let started = Instant::now(); + std::thread::scope(|s| { + for t in 0..threads { + let (stop, done) = (&stop, &done); + s.spawn(move || { + let mut i = t * 31; + let mut mine = 0usize; + while !stop.load(Ordering::Relaxed) { + store + .similarity_search_vector_exact(&qs[i % qs.len()], 10, None, -1.0) + .unwrap(); + i += 1; + mine += 1; + } + done.fetch_add(mine, Ordering::Relaxed); + }); + } + std::thread::sleep(window); + stop.store(true, Ordering::Relaxed); + }); + done.load(Ordering::Relaxed) as f64 / started.elapsed().as_secs_f64() +} + +#[test] +fn concurrent_searches_scale_instead_of_queueing_on_one_lock() { + let cpus = std::thread::available_parallelism().map_or(1, |n| n.get()); + if cpus < 4 { + eprintln!("skipped: needs 4 CPUs, have {cpus}"); + return; + } + // small enough to sit in cache, so the scan is compute-bound and a lock + // (not memory bandwidth) is the only thing that could stop it scaling + let mut store = EmbeddingStore::new(None); + let mut rng = Rng(21); + for i in 0..6000 { + store.add(id(i), "", Some(rng.vector(64))).unwrap(); + } + store.set_scan_threads(1); + let qs = queries(64, 64, 8); + throughput(&store, &qs, 2, Duration::from_millis(100)); // warm up + let mut best = 0.0f64; + for _ in 0..4 { + let one = throughput(&store, &qs, 1, Duration::from_millis(400)); + let four = throughput(&store, &qs, 4, Duration::from_millis(400)); + best = best.max(four / one); + if best >= 2.0 { + break; + } + } + // serialised callers give 1.0x; independent ones approach 4x + eprintln!("4 concurrent callers: {best:.2}x one caller"); + assert!( + best >= 2.0, + "4 concurrent callers ran at {best:.2}x one caller" + ); +} + +// -- readers and a writer: no torn, duplicated or deleted results -------------- + +#[derive(Clone)] +enum Op { + Put(usize, Vec<f32>), + Remove(usize), +} + +/// For each id, `(first state it holds this value in, value)`; `None` is a delete. +type History = HashMap<usize, Vec<(usize, Option<Vec<f32>>)>>; + +/// The serial oracle: the state after each prefix of the write sequence. +struct Oracle { + versions: History, +} + +impl Oracle { + fn new(initial: &[(usize, Vec<f32>)], ops: &[Op]) -> Self { + let mut versions: History = HashMap::new(); + for (i, v) in initial { + versions.entry(*i).or_default().push((0, Some(v.clone()))); + } + for (n, op) in ops.iter().enumerate() { + let state = n + 1; + match op { + Op::Put(i, v) => versions + .entry(*i) + .or_default() + .push((state, Some(v.clone()))), + Op::Remove(i) => { + if versions + .get(i) + .and_then(|h| h.last()) + .is_some_and(|(_, v)| v.is_some()) + { + versions.entry(*i).or_default().push((state, None)); + } + } + } + } + Self { versions } + } + + fn value_at(&self, i: usize, state: usize) -> Option<&Vec<f32>> { + let history = self.versions.get(&i)?; + history + .iter() + .rev() + .find(|(from, _)| *from <= state) + .and_then(|(_, v)| v.as_ref()) + } + + /// The exact top-k ids of a query in `state`. + fn top(&self, q: &[f32], k: usize, state: usize) -> Vec<(usize, f64)> { + let mut all: Vec<(usize, f64)> = self + .versions + .keys() + .filter_map(|&i| self.value_at(i, state).map(|v| (i, cosine(q, v)))) + .collect(); + all.sort_by(|a, b| b.1.total_cmp(&a.1).then(a.0.cmp(&b.0))); + all.truncate(k); + all + } +} + +struct Seen { + query: Vec<f32>, + lo: usize, + hi: usize, + exact: bool, + k: usize, + result: Vec<(usize, f32)>, +} + +fn raw_id(r: &SimilarityResult) -> usize { + r.node_id.id.parse().unwrap() +} + +fn run_stress(approximate: bool) { + const DIM: usize = 16; + const N0: usize = 400; + const OPS: usize = 2500; + let mut rng = Rng(if approximate { 5 } else { 6 }); + let initial: Vec<(usize, Vec<f32>)> = (0..N0).map(|i| (i, rng.vector(DIM))).collect(); + let mut ops = Vec::new(); + let mut live: Vec<usize> = (0..N0).collect(); + let mut next_id = N0; + for _ in 0..OPS { + match rng.below(10) { + 0..=3 => { + // a new id + ops.push(Op::Put(next_id, rng.vector(DIM))); + live.push(next_id); + next_id += 1; + } + 4..=6 => { + let i = live[rng.below(live.len())]; + ops.push(Op::Put(i, rng.vector(DIM))); + } + _ => { + if live.len() > 60 { + let at = rng.below(live.len()); + ops.push(Op::Remove(live.swap_remove(at))); + } + } + } + } + let oracle = Oracle::new(&initial, &ops); + + let mut base = EmbeddingStore::new(None); + for (i, v) in &initial { + base.add(NodeId::new("n", i.to_string()), "", Some(v.clone())) + .unwrap(); + } + if approximate { + base.set_ann(Some(AnnConfig { + nlist: Some(12), + nprobe: Some(4), + min_size: 0, + ..AnnConfig::default() + })) + .unwrap(); + base.build_ann_index(false).unwrap(); + } + let shared = SharedEmbeddingStore::new(base); + + let started = AtomicUsize::new(0); // ops begun + let finished = AtomicUsize::new(0); // ops completed + let stop = AtomicBool::new(false); + let mut seen: Vec<Seen> = Vec::new(); + + std::thread::scope(|s| { + let writer = s.spawn(|| { + for (n, op) in ops.iter().enumerate() { + started.store(n + 1, Ordering::SeqCst); + match op { + Op::Put(i, v) => { + shared + .add(NodeId::new("n", i.to_string()), "", Some(v.clone())) + .unwrap(); + } + Op::Remove(i) => { + shared.remove(&NodeId::new("n", i.to_string())); + } + } + finished.store(n + 1, Ordering::SeqCst); + if n % 50 == 0 { + std::thread::yield_now(); + } + } + stop.store(true, Ordering::SeqCst); + }); + let readers: Vec<_> = (0..3) + .map(|r| { + let (shared, started, finished, stop) = (&shared, &started, &finished, &stop); + s.spawn(move || { + let mut rng = Rng(1000 + r as u64); + let mut mine = Vec::new(); + while !stop.load(Ordering::SeqCst) && mine.len() < 700 { + let k = 1 + rng.below(12); + let batch = r == 2; + let exact = !approximate || rng.below(2) == 0; + let qs: Vec<Vec<f32>> = (0..if batch { 3 } else { 1 }) + .map(|_| rng.vector(DIM)) + .collect(); + let lo = finished.load(Ordering::SeqCst); + let results: Vec<Vec<SimilarityResult>> = if batch { + let guard = shared.read(); + if exact { + guard.similarity_search_vector_exact_batch(&qs, k, None, -2.0) + } else { + guard.similarity_search_vector_batch(&qs, k, None, -2.0) + } + .unwrap() + } else if exact { + vec![shared + .read() + .similarity_search_vector_exact(&qs[0], k, None, -2.0) + .unwrap()] + } else { + vec![shared + .similarity_search_vector(&qs[0], k, None, -2.0) + .unwrap()] + }; + let hi = started.load(Ordering::SeqCst); + for (q, res) in qs.into_iter().zip(results) { + mine.push(Seen { + query: q, + lo, + hi, + exact, + k, + result: res.iter().map(|x| (raw_id(x), x.score)).collect(), + }); + } + } + mine + }) + }) + .collect(); + writer.join().unwrap(); + for r in readers { + seen.extend(r.join().unwrap()); + } + }); + + assert!( + seen.len() > 200, + "readers only managed {} searches", + seen.len() + ); + let mut straddled = 0usize; + for (n, obs) in seen.iter().enumerate() { + let states = obs.lo..=obs.hi.min(ops.len()); + if obs.hi >= obs.lo + 2 { + straddled += 1; + } + // no duplicates, best first, no more than asked for + let ids: HashSet<usize> = obs.result.iter().map(|(i, _)| *i).collect(); + assert_eq!(ids.len(), obs.result.len(), "search {n}: duplicate ids"); + assert!(obs.result.len() <= obs.k, "search {n}: too many results"); + assert!( + obs.result.windows(2).all(|w| w[0].1 >= w[1].1), + "search {n}: not sorted" + ); + if obs.exact { + // exactly the serial oracle's answer in *some* state the write + // sequence passed through while the search ran + let ok = states.clone().any(|state| { + let want = oracle.top(&obs.query, obs.k, state); + want.len() == obs.result.len() + && want + .iter() + .zip(&obs.result) + .all(|((wi, ws), (gi, gs))| wi == gi && (ws - *gs as f64).abs() < 1e-4) + }); + assert!( + ok, + "search {n} (states {}..={}): not any oracle state's top-{}", + obs.lo, obs.hi, obs.k + ); + } else { + // every hit is a live row with that exact value in some state of the window + for (i, score) in &obs.result { + let ok = states.clone().any(|state| { + oracle + .value_at(*i, state) + .is_some_and(|v| (cosine(&obs.query, v) - *score as f64).abs() < 1e-4) + }); + assert!( + ok, + "search {n}: hit {i} (score {score}) matches no state in {}..={}", + obs.lo, obs.hi + ); + } + } + } + eprintln!( + "stress approximate={approximate}: {} searches, {straddled} overlapped writes", + seen.len() + ); + + // the end state is the oracle's end state, and the store is internally sound + let guard = shared.read(); + guard.verify_invariants().unwrap(); + let final_state = ops.len(); + let mut want: Vec<usize> = oracle + .versions + .keys() + .copied() + .filter(|&i| oracle.value_at(i, final_state).is_some()) + .collect(); + want.sort_unstable(); + assert_eq!(guard.count(), want.len()); + let mut got: Vec<usize> = guard + .similarity_search_vector_exact(&[1.0; DIM], usize::MAX, None, -2.0) + .unwrap() + .iter() + .map(raw_id) + .collect(); + got.sort_unstable(); + assert_eq!(got, want); +} + +#[test] +fn exact_readers_and_a_writer_match_a_serial_oracle() { + run_stress(false); +} + +#[test] +fn indexed_readers_and_a_writer_never_see_torn_or_deleted_rows() { + run_stress(true); +} + +// -- no backlog after writes ---------------------------------------------------- + +#[test] +fn writes_leave_no_backlog_for_the_next_search() { + let dim = 48; + let mut rng = Rng(77); + let mut store = EmbeddingStore::with_ann( + None, + AnnConfig { + min_size: 0, + ..AnnConfig::default() + }, + ) + .unwrap(); + for i in 0..20_000 { + store.add(id(i), "", Some(rng.vector(dim))).unwrap(); + } + store.build_ann_index(false).unwrap(); + let q = rng.vector(dim); + let steady = { + let t = Instant::now(); + for _ in 0..20 { + store.similarity_search_vector(&q, 10, None, -1.0).unwrap(); + } + t.elapsed() / 20 + }; + + // inserts, updates and deletes: more than the whole index's 2% policy + let before = store.ann_stats(); + for i in 20_000..25_000 { + store.add(id(i), "", Some(rng.vector(dim))).unwrap(); + } + for i in (0..20_000).step_by(7) { + store.add(id(i), "", Some(rng.vector(dim))).unwrap(); + } + for i in (1..20_000).step_by(11) { + store.remove(&id(i)); + } + let after = store.ann_stats(); + assert!( + after.pending < FOLD_BATCH, + "{} rows are waiting to be folded after the writes", + after.pending + ); + assert!(after.folds > before.folds, "the writes folded nothing"); + assert_eq!( + after.searches.builds, before.searches.builds, + "writes retrained" + ); + + // the first search folds nothing, rebuilds nothing, and is not slow + let folds = store.ann_stats().folds; + let t = Instant::now(); + let first = store.similarity_search_vector(&q, 10, None, -1.0).unwrap(); + let first_took = t.elapsed(); + assert_eq!(first.len(), 10); + let now = store.ann_stats(); + assert_eq!(now.folds, folds); + assert_eq!(now.pending, after.pending); + assert_eq!(now.searches.builds, before.searches.builds); + // generous: an order of magnitude over a steady query, plus a fixed allowance + assert!( + first_took < steady * 20 + Duration::from_millis(100), + "first search {first_took:?} against a steady {steady:?}" + ); + store.verify_invariants().unwrap(); +} + +// -- deletes stay deleted --------------------------------------------------------- + +#[test] +fn deleted_rows_never_come_back_through_any_path() { + let dim = 10; + for use_ann in [false, true] { + let mut rng = Rng(if use_ann { 31 } else { 32 }); + let mut store = EmbeddingStore::new(None); + if use_ann { + store + .set_ann(Some(AnnConfig { + nlist: Some(10), + nprobe: Some(4), + min_size: 0, + ..AnnConfig::default() + })) + .unwrap(); + } + let mut model: HashMap<usize, Vec<f32>> = HashMap::new(); + let mut ever_deleted: HashSet<usize> = HashSet::new(); + let mut next = 0usize; + for step in 0..4000 { + match rng.below(10) { + 0..=4 => { + let v = rng.vector(dim); + store.add(id(next), "", Some(v.clone())).unwrap(); + model.insert(next, v); + next += 1; + } + 5 | 6 if next > 0 => { + let i = rng.below(next); + let v = rng.vector(dim); + store.add(id(i), "", Some(v.clone())).unwrap(); + model.insert(i, v); + ever_deleted.remove(&i); + } + _ if next > 0 => { + let i = rng.below(next); + let was = model.remove(&i).is_some(); + assert_eq!(store.remove(&id(i)), was); + if was { + ever_deleted.insert(i); + } + } + _ => {} + } + if step % 97 == 0 { + store.verify_invariants().unwrap(); + assert_eq!(store.count(), model.len()); + let q = rng.vector(dim); + // the exact path returns precisely the live rows + let all = store + .similarity_search_vector_exact(&q, usize::MAX, None, -2.0) + .unwrap(); + let got: HashSet<usize> = all.iter().map(raw_id).collect(); + let want: HashSet<usize> = model.keys().copied().collect(); + assert_eq!(got, want, "step {step}"); + // no path returns a row deleted since it was last written + for res in [ + store.similarity_search_vector(&q, 20, None, -2.0).unwrap(), + store + .similarity_search_vector_exact(&q, 20, None, -2.0) + .unwrap(), + store + .similarity_search_vector_batch(std::slice::from_ref(&q), 20, None, -2.0) + .unwrap() + .remove(0), + ] { + for r in &res { + let i = raw_id(r); + assert!( + model.contains_key(&i), + "step {step}: deleted row {i} came back" + ); + assert!(!ever_deleted.contains(&i)); + // and it comes back with its current vector's score + assert!( + (cosine(&q, &model[&i]) - r.score as f64).abs() < 1e-4, + "step {step}: stale score for {i}" + ); + } + } + } + } + assert!(store.memory_stats().dead < store.memory_stats().rows / 4 + 1); + } +} + +#[test] +fn compaction_keeps_answers_and_the_index() { + let dim = 12; + let mut rng = Rng(8); + let mut store = EmbeddingStore::new(None); + store + .set_ann(Some(AnnConfig { + nlist: Some(16), + nprobe: Some(16), + min_size: 0, + ..AnnConfig::default() + })) + .unwrap(); + for i in 0..3000 { + store.add(id(i), "", Some(rng.vector(dim))).unwrap(); + } + store.build_ann_index(false).unwrap(); + let q = rng.vector(dim); + for i in (0..3000).filter(|i| i % 3 != 0).take(1200) { + store.remove(&id(i)); + } + let before = store + .similarity_search_vector_exact(&q, 15, None, -2.0) + .unwrap(); + store.compact(); + store.verify_invariants().unwrap(); + assert_eq!(store.memory_stats().dead, 0); + let after = store + .similarity_search_vector_exact(&q, 15, None, -2.0) + .unwrap(); + assert_eq!(keys(&before), keys(&after)); + assert_eq!(scores_bits(&before), scores_bits(&after)); + let indexed = store.similarity_search_vector(&q, 15, None, -2.0).unwrap(); + assert_eq!(keys(&indexed), keys(&after)); // nprobe covers every cell +} + +// -- the memory account --------------------------------------------------------- + +#[test] +fn memory_stats_account_for_one_copy_of_the_matrix() { + let (n, dim) = (5000usize, 64usize); + let store = messy_store(n, dim, None, 12); + let stats = store.memory_stats(); + assert_eq!(stats.live, store.count()); + assert_eq!(stats.live_vector_bytes, store.count() * dim * 4); + // the matrix is the vectors (dead rows and a chunk's tail included): one copy + assert!(stats.matrix_bytes >= stats.rows * dim * 4); + assert!((stats.matrix_bytes as f64) < (stats.rows * dim * 4) as f64 * 1.1 + 1e5); + assert!(stats.chunks >= 1 && stats.dead <= stats.rows / 4 + 1); + // a search builds nothing that holds vectors + let before = store.memory_stats().total_bytes(); + store + .similarity_search_vector(&[0.5; 64], 10, None, -1.0) + .unwrap(); + assert_eq!(store.memory_stats().total_bytes(), before); + // borrowed and copied vectors are the same row + let someone = (0..n).find(|&i| store.get(&id(i)).is_some()).unwrap(); + assert_eq!( + store.get(&id(someone)).unwrap().vector, + store.get_vector(&id(someone)).unwrap() + ); + assert!(store.get_vector(&NodeId::new("nobody", "1")).is_none()); + store.verify_invariants().unwrap(); +} diff --git a/isongraph-vector-rs/tests/hnsw.rs b/isongraph-vector-rs/tests/hnsw.rs new file mode 100644 index 0000000..5451314 --- /dev/null +++ b/isongraph-vector-rs/tests/hnsw.rs @@ -0,0 +1,1356 @@ +//! The opt-in HNSW approximate index: recall, exactness where it must be exact, +//! upkeep under writes (insert, update, delete, compaction) checked against a +//! model through every search path, filters, concurrency, determinism, and the +//! guarantee that nothing it builds ever reaches a graph file, a sidecar or the +//! graph digest. +//! +//! Test data are clustered unit-ish vectors from a seeded generator, so every +//! run sees the same numbers. + +use ison_graph_rs::Node; +use isongraph_vector_rs::{ + graph_digest, AnnConfig, AnnMode, EmbeddingError, EmbeddingStore, HnswConfig, MockEncoder, + NodeFilter, NodeId, PropertyValue, SemanticGraph, SharedEmbeddingStore, SimilarityOptions, + SimilarityResult, +}; +use std::collections::{HashMap, HashSet}; +use std::sync::atomic::{AtomicBool, Ordering}; + +/// Recall@10 demanded at the defaults on the clustered data below. +const RECALL_FLOOR: f64 = 0.9; + +// -- data --------------------------------------------------------------------- + +struct Rng(u64); + +impl Rng { + fn next_u64(&mut self) -> u64 { + self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } + fn unit(&mut self) -> f32 { + (self.next_u64() >> 40) as f32 / (1u64 << 24) as f32 + } + fn gauss(&mut self) -> f32 { + let u1 = self.unit().max(1e-7); + let u2 = self.unit(); + (-2.0 * u1.ln()).sqrt() * (std::f32::consts::TAU * u2).cos() + } + fn below(&mut self, n: usize) -> usize { + (self.next_u64() % n as u64) as usize + } +} + +struct Clusters { + centers: Vec<Vec<f32>>, + dim: usize, +} + +impl Clusters { + fn new(dim: usize, clusters: usize, seed: u64) -> Self { + let mut rng = Rng(seed); + let centers = (0..clusters) + .map(|_| { + let mut c: Vec<f32> = (0..dim).map(|_| rng.gauss()).collect(); + let norm = c.iter().map(|x| x * x).sum::<f32>().sqrt(); + c.iter_mut().for_each(|x| *x /= norm); + c + }) + .collect(); + Self { centers, dim } + } + + fn draw(&self, count: usize, rng: &mut Rng) -> Vec<Vec<f32>> { + let scale = 1.0 / (self.dim as f32).sqrt(); + (0..count) + .map(|_| { + let c = &self.centers[rng.below(self.centers.len())]; + c.iter().map(|x| x + rng.gauss() * scale).collect() + }) + .collect() + } +} + +fn cosine(a: &[f32], b: &[f32]) -> f64 { + let (mut dot, mut na, mut nb) = (0.0f64, 0.0f64, 0.0f64); + for (x, y) in a.iter().zip(b) { + dot += *x as f64 * *y as f64; + na += *x as f64 * *x as f64; + nb += *y as f64 * *y as f64; + } + if na == 0.0 || nb == 0.0 { + 0.0 + } else { + dot / (na.sqrt() * nb.sqrt()) + } +} + +fn doc(i: usize) -> NodeId { + NodeId::new("doc", i.to_string()) +} + +fn ids(results: &[SimilarityResult]) -> Vec<String> { + results + .iter() + .map(|r| format!("{}:{}", r.node_id.node_type, r.node_id.id)) + .collect() +} + +fn id_set(results: &[SimilarityResult]) -> HashSet<String> { + ids(results).into_iter().collect() +} + +fn bits(results: &[SimilarityResult]) -> Vec<(String, u32)> { + results + .iter() + .map(|r| { + ( + format!("{}:{}", r.node_id.node_type, r.node_id.id), + r.score.to_bits(), + ) + }) + .collect() +} + +/// HNSW with the defaults, no minimum size. +fn hnsw_config() -> AnnConfig { + AnnConfig { + min_size: 0, + ..AnnConfig::hnsw() + } +} + +struct Fixture { + store: EmbeddingStore, + data: Vec<Vec<f32>>, + clusters: Clusters, + rng: Rng, +} + +impl Fixture { + fn queries(&mut self, count: usize) -> Vec<Vec<f32>> { + self.clusters.draw(count, &mut self.rng) + } +} + +fn fixture_with( + n: usize, + dim: usize, + ann: Option<AnnConfig>, + type_of: impl Fn(usize) -> String, +) -> Fixture { + let clusters = Clusters::new(dim, 40, 1); + let mut rng = Rng(77); + let data = clusters.draw(n, &mut rng); + let mut store = EmbeddingStore::new(None); + for (i, v) in data.iter().enumerate() { + store + .add( + NodeId::new(type_of(i), i.to_string()), + &format!("text {i}"), + Some(v.clone()), + ) + .unwrap(); + } + if let Some(cfg) = ann { + store.set_ann(Some(cfg)).unwrap(); + } + Fixture { + store, + data, + clusters, + rng, + } +} + +fn fixture(n: usize) -> Fixture { + fixture_with(n, 32, Some(hnsw_config()), |_| "doc".to_string()) +} + +fn recall_of(store: &EmbeddingStore, queries: &[Vec<f32>], k: usize) -> f64 { + let mut sum = 0.0; + for q in queries { + let got = id_set(&store.similarity_search_vector(q, k, None, -1.0).unwrap()); + let want = id_set( + &store + .similarity_search_vector_exact(q, k, None, -1.0) + .unwrap(), + ); + sum += got.intersection(&want).count() as f64 / want.len() as f64; + } + sum / queries.len() as f64 +} + +// -- configuration ------------------------------------------------------------ + +#[test] +fn defaults_leave_ivf_alone_and_hnsw_is_opt_in() { + let cfg = AnnConfig::default(); + assert_eq!(cfg.mode, AnnMode::Ivf); + assert_eq!(cfg.hnsw, HnswConfig::default()); + assert_eq!( + (cfg.hnsw.m, cfg.hnsw.ef_construction, cfg.hnsw.ef_search), + (16, 200, None) + ); + let h = AnnConfig::hnsw(); + assert_eq!(h.mode, AnnMode::Hnsw); + assert_eq!(h.min_size, 50_000, "the same minimum size as IVF"); + assert_eq!(h.seed, 0); + + // an IVF store never builds a graph and reports IVF facts + let mut fx = fixture_with( + 1200, + 16, + Some(AnnConfig { + nlist: Some(16), + nprobe: Some(4), + min_size: 0, + ..AnnConfig::default() + }), + |_| "doc".to_string(), + ); + let q = fx.queries(1).remove(0); + fx.store.similarity_search_vector(&q, 5, None, 0.0).unwrap(); + let stats = fx.store.ann_stats(); + assert_eq!(stats.mode, AnnMode::Ivf); + assert_eq!(stats.nlist, Some(16)); + assert_eq!(stats.graph_nodes, 0); +} + +#[test] +fn argument_validation() { + let mut store = EmbeddingStore::new(None); + let bad = [ + HnswConfig { + m: 1, + ..HnswConfig::default() + }, + HnswConfig { + m: 65, + ..HnswConfig::default() + }, + HnswConfig { + ef_construction: 0, + ..HnswConfig::default() + }, + HnswConfig { + ef_search: Some(0), + ..HnswConfig::default() + }, + ]; + for hnsw in bad { + let cfg = AnnConfig { + hnsw, + ..AnnConfig::hnsw() + }; + assert!(matches!( + store.set_ann(Some(cfg.clone())), + Err(EmbeddingError::InvalidOption(_)) + )); + assert!(EmbeddingStore::with_ann(None, cfg).is_err()); + } + assert!(store.ann().is_none(), "a refused config must not be kept"); + + assert!(store.set_ann_ef_search(Some(10)).is_err(), "no ann yet"); + store.set_ann(Some(AnnConfig::hnsw())).unwrap(); + assert!(store.set_ann_ef_search(Some(0)).is_err()); + store.set_ann_ef_search(Some(33)).unwrap(); + assert_eq!(store.ann().unwrap().hnsw.ef_search, Some(33)); + assert_eq!(store.ann_stats().ef_search, Some(33)); + assert!(store + .similarity_search_vector_ef(&[1.0], 1, None, 0.0, 0) + .is_err()); +} + +// -- recall and exactness ----------------------------------------------------- + +#[test] +fn recall_floor_at_defaults_and_recall_check_agrees() { + let mut fx = fixture(4000); + let queries = fx.queries(80); + let report = fx.store.recall_check(&queries, 10, None, 0.0).unwrap(); + assert!(report.approximate, "{report:?}"); + assert_eq!((report.queries, report.k), (80, 10)); + assert_eq!(report.ef_search, 64); + assert!(report.recall >= RECALL_FLOOR, "{report:?}"); + assert!(report.min_recall <= report.recall && report.recall <= 1.0); + + // the report's mean is the mean of the per-query overlaps computed here + let mine = recall_of(&fx.store, &queries, 10); + assert!( + (report.recall - mine).abs() < 1e-9, + "{} vs {mine}", + report.recall + ); + + let stats = fx.store.ann_stats(); + assert!(stats.enabled && stats.active); + assert_eq!(stats.mode, AnnMode::Hnsw); + assert_eq!(stats.graph_nodes, 4000); + assert_eq!(stats.graph_dead, 0); + assert!(stats.max_level.is_some() && stats.build_seconds.is_some()); + assert!(stats.index_bytes > 0); + assert!(stats.searches.approximate >= 80, "{stats:?}"); + assert_eq!(stats.searches.builds, 1); + fx.store.verify_invariants().unwrap(); +} + +#[test] +fn more_ef_never_hurts_much_and_ef_one_still_answers() { + let mut fx = fixture(3000); + let queries = fx.queries(40); + let mean = |ef: usize| -> f64 { + let mut sum = 0.0; + for q in &queries { + let got = id_set( + &fx.store + .similarity_search_vector_ef(q, 10, None, -1.0, ef) + .unwrap(), + ); + let want = id_set( + &fx.store + .similarity_search_vector_exact(q, 10, None, -1.0) + .unwrap(), + ); + sum += got.intersection(&want).count() as f64 / 10.0; + } + sum / queries.len() as f64 + }; + let (low, mid, high) = (mean(10), mean(64), mean(400)); + assert!( + low <= mid + 0.02 && mid <= high + 0.02, + "{low} {mid} {high}" + ); + assert!(high >= 0.98, "{high}"); + // ef below k is raised to k + let res = fx + .store + .similarity_search_vector_ef(&queries[0], 10, None, -1.0, 1) + .unwrap(); + assert_eq!(res.len(), 10); +} + +#[test] +fn scores_are_the_exact_scans_and_hits_come_back_sorted() { + let mut fx = fixture(2500); + let queries = fx.queries(30); + for q in &queries { + let exact_all = fx + .store + .similarity_search_vector_exact(q, usize::MAX, None, -2.0) + .unwrap(); + let by_id: HashMap<String, u32> = bits(&exact_all).into_iter().collect(); + let res = fx + .store + .similarity_search_vector(q, 15, None, -1.0) + .unwrap(); + assert_eq!(res.len(), 15); + for (id, score_bits) in bits(&res) { + assert_eq!(by_id[&id], score_bits, "{id}: score differs from the scan"); + } + assert!(res.windows(2).all(|w| w[0].score >= w[1].score)); + } + assert!(fx.store.ann_stats().searches.approximate >= 30); + let _ = &fx.data; +} + +#[test] +fn exact_true_and_big_k_and_small_stores_stay_exact() { + let mut fx = fixture(1500); + let q = fx.queries(1).remove(0); + let before = fx.store.ann_stats().searches; + let a = fx + .store + .similarity_search_vector_exact(&q, 10, None, 0.0) + .unwrap(); + let b = fx + .store + .similarity_search_vector(&q, usize::MAX, None, -2.0) + .unwrap(); + assert_eq!(a.len(), 10); + assert_eq!(b.len(), 1500); + assert_eq!(bits(&a), bits(&b[..10])); + let after = fx.store.ann_stats().searches; + assert_eq!(after.approximate, before.approximate); + assert_eq!(after.exact, before.exact + 2); + + // under min_size nothing is built + let small = fixture_with( + 400, + 16, + Some(AnnConfig { + min_size: 1000, + ..AnnConfig::hnsw() + }), + |_| "doc".to_string(), + ); + let q = vec![0.5f32; 16]; + small + .store + .similarity_search_vector(&q, 5, None, 0.0) + .unwrap(); + let stats = small.store.ann_stats(); + assert!(!stats.active && stats.searches.approximate == 0 && stats.searches.exact == 1); + + // a store smaller than the ef window answers exactly (the graph would + // cost more than the scan) + let tiny = fixture_with(40, 16, Some(hnsw_config()), |_| "doc".to_string()); + tiny.store + .similarity_search_vector(&q, 5, None, 0.0) + .unwrap(); + assert_eq!(tiny.store.ann_stats().searches.approximate, 0); +} + +#[test] +fn zero_query_and_wrong_dimension() { + let fx = fixture(1200); + let zero = vec![0.0f32; 32]; + let res = fx + .store + .similarity_search_vector(&zero, 5, None, -1.0) + .unwrap(); + assert_eq!(res.len(), 5, "the scan answers a zero query"); + assert!(matches!( + fx.store.similarity_search_vector(&[1.0, 2.0], 5, None, 0.0), + Err(EmbeddingError::DimensionMismatch { .. }) + )); +} + +#[test] +fn threshold_is_respected() { + let mut fx = fixture(2500); + let queries = fx.queries(10); + for q in &queries { + let exact = fx + .store + .similarity_search_vector_exact(q, usize::MAX, None, 0.6) + .unwrap(); + let res = fx.store.similarity_search_vector(q, 5, None, 0.6).unwrap(); + assert!(res.iter().all(|r| r.score >= 0.6)); + assert_eq!(res.len(), exact.len().min(5)); + } + // nothing passes + let none = fx + .store + .similarity_search_vector(&queries[0], 5, None, 1.01) + .unwrap(); + assert!(none.is_empty()); +} + +// -- batches ------------------------------------------------------------------ + +#[test] +fn batch_equals_single_bit_for_bit() { + let mut fx = fixture_with(3000, 32, Some(hnsw_config()), |i| { + if i % 4 == 0 { "a" } else { "b" }.to_string() + }); + let queries = fx.queries(50); + for node_type in [None, Some("a")] { + let batch = fx + .store + .similarity_search_vector_batch(&queries, 10, node_type, 0.0) + .unwrap(); + for (q, got) in queries.iter().zip(&batch) { + let one = fx + .store + .similarity_search_vector(q, 10, node_type, 0.0) + .unwrap(); + assert_eq!(bits(got), bits(&one)); + } + } + // the exact batch is exact, and different calls agree on thread counts + let exact = fx + .store + .similarity_search_vector_exact_batch(&queries, 10, None, 0.0) + .unwrap(); + for (q, got) in queries.iter().zip(&exact) { + let one = fx + .store + .similarity_search_vector_exact(q, 10, None, 0.0) + .unwrap(); + assert_eq!(bits(got), bits(&one)); + } + let counts = fx.store.ann_stats().searches; + assert!(counts.approximate >= 100, "{counts:?}"); +} + +// -- filters ------------------------------------------------------------------ + +#[test] +fn node_type_filter_returns_k_even_when_selective() { + // 1 row in 50 has the rare type: a plain ef window holds almost none + let mut fx = fixture_with(4000, 32, Some(hnsw_config()), |i| { + if i % 50 == 7 { "rare" } else { "common" }.to_string() + }); + let queries = fx.queries(30); + let (mut overlap, mut asked) = (0usize, 0usize); + for q in &queries { + let res = fx + .store + .similarity_search_vector(q, 10, Some("rare"), -1.0) + .unwrap(); + assert_eq!(res.len(), 10, "a selective filter must still give k"); + assert!(res.iter().all(|r| r.node_id.node_type == "rare")); + assert!(res.windows(2).all(|w| w[0].score >= w[1].score)); + let want = fx + .store + .similarity_search_vector_exact(q, 10, Some("rare"), -1.0) + .unwrap(); + overlap += id_set(&res).intersection(&id_set(&want)).count(); + asked += want.len(); + } + assert!(overlap * 100 >= asked * 80, "{overlap} of {asked}"); + let counts = fx.store.ann_stats().searches; + assert!(counts.widened > 0 || counts.exact > 0, "{counts:?}"); + + // a common type is answered by the graph itself + let before = fx.store.ann_stats().searches.approximate; + fx.store + .similarity_search_vector(&queries[0], 10, Some("common"), -1.0) + .unwrap(); + assert_eq!(fx.store.ann_stats().searches.approximate, before + 1); + + // a type nothing has, and one that is too rare to ever fill k + assert!(fx + .store + .similarity_search_vector(&queries[0], 10, Some("nope"), -1.0) + .unwrap() + .is_empty()); + let mut store = fx.store; + for i in 0..4000 { + if i % 50 != 7 { + continue; + } + if i > 7 + 50 * 3 { + store.remove(&NodeId::new("rare", i.to_string())); + } + } + let res = store + .similarity_search_vector(&queries[1], 10, Some("rare"), -1.0) + .unwrap(); + assert_eq!(res.len(), 4, "only four rare rows remain"); + store.verify_invariants().unwrap(); +} + +fn graph_with(n: usize, dim: usize, ann: Option<AnnConfig>) -> (SemanticGraph, Clusters, Rng) { + let clusters = Clusters::new(dim, 40, 1); + let mut rng = Rng(11); + let data = clusters.draw(n, &mut rng); + let mut graph = SemanticGraph::with_options( + "ann", + Box::new(MockEncoder::new(dim)), + false, + vec!["name".to_string()], + ); + for (i, v) in data.iter().enumerate() { + graph + .add_node( + "item", + &i.to_string(), + vec![("group", PropertyValue::Int((i % 50) as i64))], + None, + ) + .unwrap(); + graph + .embedding_store_mut() + .add( + NodeId::new("item", i.to_string()), + &format!("item {i}"), + Some(v.clone()), + ) + .unwrap(); + } + if let Some(cfg) = ann { + graph.set_ann(Some(cfg)).unwrap(); + } + (graph, clusters, rng) +} + +fn group_of(graph: &SemanticGraph, r: &SimilarityResult) -> i64 { + match graph + .graph() + .get_node(&r.node_id.node_type, &r.node_id.id) + .unwrap() + .properties + .get("group") + { + Some(PropertyValue::Int(i)) => *i, + other => panic!("group was {other:?}"), + } +} + +#[test] +fn graph_node_filter_gets_k_results_even_when_selective() { + let (graph, clusters, mut rng) = graph_with(3000, 32, Some(hnsw_config())); + let seven = |n: &Node| matches!(n.properties.get("group"), Some(PropertyValue::Int(7))); + let filter = NodeFilter::new().predicate(&seven); + let (mut overlap, mut asked) = (0, 0); + for q in clusters.draw(15, &mut rng) { + let opts = SimilarityOptions { + filter: filter.clone(), + ..SimilarityOptions::default() + }; + let res = graph.similarity_search_with(&q, &opts).unwrap(); + assert_eq!(res.len(), 10); + assert!(res.iter().all(|r| group_of(&graph, r) == 7)); + let want = graph + .similarity_search_with( + &q, + &SimilarityOptions { + exact: true, + ..opts + }, + ) + .unwrap(); + overlap += id_set(&res).intersection(&id_set(&want)).count(); + asked += want.len(); + } + assert!(overlap * 100 >= asked * 80, "{overlap} of {asked}"); + assert!(graph.ann_stats().active); + + // a filter nothing passes terminates, empty + let never = |_: &Node| false; + let q = clusters.draw(1, &mut rng).remove(0); + let none = graph + .similarity_search_with( + &q, + &SimilarityOptions { + filter: NodeFilter::new().predicate(&never), + ..SimilarityOptions::default() + }, + ) + .unwrap(); + assert!(none.is_empty()); + graph.embedding_store().verify_invariants().unwrap(); +} + +// -- writes, checked against a model ------------------------------------------ + +/// What the store should hold: every id's current vector, and every vector it +/// ever had (a concurrent reader may see any of them). +#[derive(Default)] +struct Model { + current: HashMap<usize, Vec<f32>>, + history: HashMap<usize, Vec<Vec<f32>>>, +} + +impl Model { + fn put(&mut self, id: usize, v: Vec<f32>) { + self.history.entry(id).or_default().push(v.clone()); + self.current.insert(id, v); + } + fn remove(&mut self, id: usize) { + self.current.remove(&id); + } +} + +fn raw_id(r: &SimilarityResult) -> usize { + r.node_id.id.parse().unwrap() +} + +/// Every result is a live id whose score is the cosine of its current vector. +fn check_against_model(model: &Model, q: &[f32], res: &[SimilarityResult], what: &str) { + let mut seen = HashSet::new(); + for r in res { + let id = raw_id(r); + let v = model + .current + .get(&id) + .unwrap_or_else(|| panic!("{what}: returned deleted id {id}")); + assert!(seen.insert(id), "{what}: id {id} twice"); + let want = cosine(q, v); + assert!( + (r.score as f64 - want).abs() < 2e-5, + "{what}: id {id} scored {} but its vector says {want}", + r.score + ); + } + assert!( + res.windows(2).all(|w| w[0].score >= w[1].score), + "{what}: not sorted" + ); +} + +#[test] +fn inserts_updates_and_deletes_agree_with_a_model_on_every_path() { + const DIM: usize = 24; + let clusters = Clusters::new(DIM, 12, 5); + let mut rng = Rng(9); + let mut model = Model::default(); + let mut store = EmbeddingStore::new(None); + let initial = clusters.draw(1500, &mut rng); + for (i, v) in initial.iter().enumerate() { + store.add(doc(i), "", Some(v.clone())).unwrap(); + model.put(i, v.clone()); + } + store.set_ann(Some(hnsw_config())).unwrap(); + store.build_ann_index(false).unwrap(); + assert!(store.ann_stats().active); + let mut next_id = 1500; + let mut found_after_update = 0usize; + let mut updated = 0usize; + + for step in 0..900 { + match rng.below(10) { + 0..=3 => { + let v = clusters.draw(1, &mut rng).remove(0); + store.add(doc(next_id), "", Some(v.clone())).unwrap(); + model.put(next_id, v); + next_id += 1; + } + 4..=6 => { + // an update may move the row anywhere + let live: Vec<usize> = model.current.keys().copied().collect(); + let id = live[rng.below(live.len())]; + let v = clusters.draw(1, &mut rng).remove(0); + store.add(doc(id), "", Some(v.clone())).unwrap(); + model.put(id, v.clone()); + // it must now be found at its new position, with a high ef + let top = store + .similarity_search_vector_ef(&v, 3, None, -1.0, 200) + .unwrap(); + updated += 1; + if top.iter().any(|r| raw_id(r) == id) { + found_after_update += 1; + } + } + _ => { + let live: Vec<usize> = model.current.keys().copied().collect(); + if live.len() > 600 { + let id = live[rng.below(live.len())]; + assert!(store.remove(&doc(id))); + model.remove(id); + } + } + } + if step % 30 == 0 { + store.verify_invariants().unwrap(); + for q in clusters.draw(3, &mut rng) { + let single = store.similarity_search_vector(&q, 10, None, -1.0).unwrap(); + check_against_model(&model, &q, &single, "single"); + let exact = store + .similarity_search_vector_exact(&q, 10, None, -1.0) + .unwrap(); + check_against_model(&model, &q, &exact, "exact"); + let batch = store + .similarity_search_vector_batch(&[q.clone(), q.clone()], 10, None, -1.0) + .unwrap(); + for b in &batch { + check_against_model(&model, &q, b, "batch"); + assert_eq!(bits(b), bits(&single)); + } + let all = store + .similarity_search_vector(&q, usize::MAX, None, -1.0) + .unwrap(); + assert_eq!(all.len(), model.current.len()); + check_against_model(&model, &q, &all, "unbounded"); + // the exact top-1 is nearly always inside the approximate top 10 + let overlap = id_set(&single).intersection(&id_set(&exact)).count(); + assert!(overlap >= 6, "step {step}: only {overlap} of 10 agree"); + } + } + } + assert_eq!(store.count(), model.current.len()); + assert!( + found_after_update * 100 >= updated * 98, + "{found_after_update} of {updated} updated rows found at their new position" + ); + store.verify_invariants().unwrap(); + let stats = store.ann_stats(); + assert!(stats.inserts > 0 && stats.churn > 0, "{stats:?}"); + + // and the graph still has good recall after all of that + let queries = clusters.draw(60, &mut rng); + let recall = recall_of(&store, &queries, 10); + assert!(recall >= RECALL_FLOOR, "recall {recall} after churn"); +} + +#[test] +fn deletes_never_come_back_and_compaction_keeps_the_graph_whole() { + const DIM: usize = 24; + let clusters = Clusters::new(DIM, 12, 6); + let mut rng = Rng(21); + let data = clusters.draw(3000, &mut rng); + let mut store = EmbeddingStore::new(None); + for (i, v) in data.iter().enumerate() { + store.add(doc(i), "", Some(v.clone())).unwrap(); + } + store.set_ann(Some(hnsw_config())).unwrap(); + store.build_ann_index(false).unwrap(); + let mut gone: HashSet<usize> = HashSet::new(); + + // delete a fifth: tombstones stay in the graph (below the 25% compaction) + for i in (0..3000).step_by(5) { + assert!(store.remove(&doc(i))); + gone.insert(i); + } + let stats = store.ann_stats(); + assert_eq!(stats.graph_dead, 600, "tombstones are kept for routing"); + assert_eq!(stats.graph_nodes, 3000); + store.verify_invariants().unwrap(); + let queries = clusters.draw(40, &mut rng); + for q in &queries { + for r in store.similarity_search_vector(q, 10, None, -1.0).unwrap() { + assert!(!gone.contains(&raw_id(&r))); + } + } + let recall_with_tombstones = recall_of(&store, &queries, 10); + assert!( + recall_with_tombstones >= RECALL_FLOOR, + "{recall_with_tombstones}" + ); + + // cross 25% dead: the table compacts, and the graph follows + for i in (1..3000).step_by(9) { + if store.remove(&doc(i)) { + gone.insert(i); + } + } + let stats = store.ann_stats(); + assert!( + stats.graph_dead < 600, + "compaction dropped the tombstones: {} left", + stats.graph_dead + ); + assert_eq!(stats.graph_nodes, store.count() + stats.graph_dead); + assert_eq!(store.count(), 3000 - gone.len()); + store.compact(); + let stats = store.ann_stats(); + assert_eq!((stats.graph_dead, stats.graph_nodes), (0, store.count())); + store.verify_invariants().unwrap(); + for q in &queries { + let res = store.similarity_search_vector(q, 10, None, -1.0).unwrap(); + assert_eq!(res.len(), 10); + for r in &res { + assert!(!gone.contains(&raw_id(r))); + } + } + let recall = recall_of(&store, &queries, 10); + assert!(recall >= RECALL_FLOOR, "recall {recall} after compaction"); + + // explicit compaction of a few more, then writes into the compacted store + for i in (2..400).step_by(7) { + if store.remove(&doc(i)) { + gone.insert(i); + } + } + store.compact(); + store.verify_invariants().unwrap(); + for (n, v) in clusters.draw(50, &mut rng).iter().enumerate() { + store.add(doc(10_000 + n), "", Some(v.clone())).unwrap(); + let top = store + .similarity_search_vector_ef(v, 3, None, -1.0, 200) + .unwrap(); + assert!(top.iter().any(|r| raw_id(r) == 10_000 + n)); + } + store.verify_invariants().unwrap(); + + // delete almost everything: the index goes away under min_size 2 and + // answers come from the scan + let live: Vec<usize> = (0..3000) + .filter(|i| !gone.contains(i)) + .chain(10_000..10_050) + .collect(); + for &i in &live[..live.len() - 1] { + store.remove(&doc(i)); + } + assert_eq!(store.count(), 1); + assert!(!store.ann_stats().active); + let res = store + .similarity_search_vector(&queries[0], 3, None, -1.0) + .unwrap(); + assert_eq!(res.len(), 1); + store.verify_invariants().unwrap(); +} + +#[test] +fn heavy_churn_rebuilds_the_graph() { + let mut fx = fixture(1500); + fx.store.build_ann_index(false).unwrap(); + assert_eq!(fx.store.ann_stats().searches.builds, 1); + let mut rng = Rng(3); + // rewrite every row once and a half times: churn passes the node count + for _round in 0..3 { + for i in 0..750 { + let v = fx.clusters.draw(1, &mut rng).remove(0); + fx.store + .add(NodeId::new("doc", i.to_string()), "", Some(v)) + .unwrap(); + } + } + let q = fx.queries(1).remove(0); + fx.store + .similarity_search_vector(&q, 10, None, 0.0) + .unwrap(); + let stats = fx.store.ann_stats(); + assert_eq!(stats.searches.builds, 2, "{stats:?}"); + assert_eq!(stats.churn, 0); + fx.store.verify_invariants().unwrap(); + let queries = fx.queries(40); + assert!(recall_of(&fx.store, &queries, 10) >= RECALL_FLOOR); +} + +#[test] +fn clear_and_dimension_change() { + let mut fx = fixture(1200); + fx.store.build_ann_index(false).unwrap(); + assert!(fx.store.ann_stats().active); + fx.store.clear(); + assert!(!fx.store.ann_stats().active); + assert_eq!(fx.store.count(), 0); + // a different width afterwards is fine; the index is rebuilt lazily + let mut rng = Rng(5); + for i in 0..300 { + let v: Vec<f32> = (0..8).map(|_| rng.gauss()).collect(); + fx.store.add(doc(i), "", Some(v)).unwrap(); + } + let q = vec![0.3f32; 8]; + let res = fx + .store + .similarity_search_vector(&q, 5, None, -2.0) + .unwrap(); + assert_eq!(res.len(), 5); + assert!(fx.store.ann_stats().active); + fx.store.verify_invariants().unwrap(); +} + +// -- determinism -------------------------------------------------------------- + +#[test] +fn same_seed_same_order_gives_the_same_answers_whatever_the_threads() { + let build = |seed: u64, threads: usize| -> Vec<Vec<(String, u32)>> { + let mut fx = fixture_with( + 2500, + 32, + Some(AnnConfig { + seed, + ..hnsw_config() + }), + |_| "doc".to_string(), + ); + fx.store.set_scan_threads(threads); + // some incremental writes after the build too + fx.store.build_ann_index(false).unwrap(); + let mut rng = Rng(100); + for i in 0..60 { + let v = fx.clusters.draw(1, &mut rng).remove(0); + fx.store.add(doc(5000 + i), "", Some(v)).unwrap(); + } + let mut qrng = Rng(55); + let queries = fx.clusters.draw(30, &mut qrng); + queries + .iter() + .map(|q| { + bits( + &fx.store + .similarity_search_vector(q, 10, None, -1.0) + .unwrap(), + ) + }) + .collect() + }; + let a = build(7, 0); + assert_eq!(a, build(7, 0), "same seed, same answers"); + assert_eq!(a, build(7, 1), "one thread builds the same graph"); + assert_eq!(a, build(7, 3), "three threads build the same graph"); + // another seed is another graph (almost surely different somewhere), but + // the same quality; compare against exact only + let b = build(8, 0); + assert_eq!(a.len(), b.len()); +} + +// -- awkward shapes ----------------------------------------------------------- + +#[test] +fn awkward_dimensions_and_duplicates_work() { + for dim in [1usize, 3, 7, 9, 31, 33] { + let mut rng = Rng(dim as u64); + let mut store = EmbeddingStore::with_ann(None, hnsw_config()).unwrap(); + for i in 0..600 { + let v: Vec<f32> = (0..dim).map(|_| rng.gauss()).collect(); + store.add(doc(i), "", Some(v)).unwrap(); + } + // exact duplicates, and a zero vector + for i in 600..650 { + store.add(doc(i), "", Some(vec![1.0; dim])).unwrap(); + } + store.add(doc(650), "", Some(vec![0.0; dim])).unwrap(); + let q: Vec<f32> = (0..dim).map(|_| rng.gauss()).collect(); + let res = store.similarity_search_vector(&q, 10, None, -2.0).unwrap(); + assert_eq!(res.len(), 10, "dim {dim}"); + store.verify_invariants().unwrap(); + // ties break by arrival, as in the scan + let dup = store + .similarity_search_vector(&vec![1.0; dim], 20, None, 0.999) + .unwrap(); + let order: Vec<usize> = dup.iter().map(raw_id).collect(); + let mut sorted = order.clone(); + sorted.sort_unstable(); + assert_eq!( + order, sorted, + "dim {dim}: equal scores must be in slot order" + ); + } +} + +#[test] +fn non_finite_vectors_do_not_break_the_graph() { + let mut fx = fixture(1200); + fx.store.build_ann_index(false).unwrap(); + let mut bad = vec![0.5f32; 32]; + bad[3] = f32::NAN; + fx.store.add(doc(5000), "", Some(bad)).unwrap(); + let mut inf = vec![0.5f32; 32]; + inf[0] = f32::INFINITY; + fx.store.add(doc(5001), "", Some(inf)).unwrap(); + let q = fx.queries(1).remove(0); + let res = fx + .store + .similarity_search_vector(&q, 10, None, -2.0) + .unwrap(); + assert_eq!(res.len(), 10); + assert!(res.iter().all(|r| raw_id(r) < 5000)); + fx.store.verify_invariants().unwrap(); + // a healthy vector can replace a bad one + fx.store.add(doc(5000), "", Some(vec![0.25; 32])).unwrap(); + fx.store.verify_invariants().unwrap(); +} + +// -- memory ------------------------------------------------------------------- + +#[test] +fn the_graph_is_links_only() { + let fx = fixture_with(5000, 128, Some(hnsw_config()), |_| "doc".to_string()); + let before = fx.store.memory_stats(); + assert_eq!(before.index_bytes, 0); + fx.store.build_ann_index(false).unwrap(); + let after = fx.store.memory_stats(); + assert_eq!(after.matrix_bytes, before.matrix_bytes, "no second copy"); + // 132 bytes of layer-0 links + about 10 more per node at M = 16 + let per_node = after.index_bytes as f64 / 5000.0; + assert!( + (130.0..170.0).contains(&per_node), + "{per_node} bytes per node" + ); + // far less than the vectors themselves (512 bytes each at dim 128) + assert!(after.index_bytes * 3 < after.live_vector_bytes); +} + +// -- concurrency -------------------------------------------------------------- + +#[test] +fn concurrent_readers_and_one_writer_never_see_a_torn_graph() { + const DIM: usize = 16; + let clusters = Clusters::new(DIM, 10, 8); + let mut rng = Rng(31); + let initial = clusters.draw(800, &mut rng); + let mut model = Model::default(); + let mut base = EmbeddingStore::new(None); + for (i, v) in initial.iter().enumerate() { + base.add(doc(i), "", Some(v.clone())).unwrap(); + model.put(i, v.clone()); + } + base.set_ann(Some(hnsw_config())).unwrap(); + base.build_ann_index(false).unwrap(); + + // the writer's script, with the model of every version known up front + enum Op { + Put(usize, Vec<f32>), + Remove(usize), + } + let mut ops = Vec::new(); + let mut live: Vec<usize> = (0..800).collect(); + let mut next_id = 800; + for _ in 0..1500 { + match rng.below(10) { + 0..=3 => { + let v = clusters.draw(1, &mut rng).remove(0); + model.put(next_id, v.clone()); + ops.push(Op::Put(next_id, v)); + live.push(next_id); + next_id += 1; + } + 4..=6 => { + let id = live[rng.below(live.len())]; + let v = clusters.draw(1, &mut rng).remove(0); + model.put(id, v.clone()); + ops.push(Op::Put(id, v)); + } + _ => { + if live.len() > 300 { + let at = rng.below(live.len()); + ops.push(Op::Remove(live.swap_remove(at))); + } + } + } + } + let final_live: HashSet<usize> = live.iter().copied().collect(); + + let shared = SharedEmbeddingStore::new(base); + let stop = AtomicBool::new(false); + type Observed = (Vec<f32>, usize, Vec<(usize, f32)>); + let observed: Vec<Observed> = std::thread::scope(|s| { + let writer = s.spawn(|| { + for (n, op) in ops.iter().enumerate() { + match op { + Op::Put(i, v) => { + shared.add(doc(*i), "", Some(v.clone())).unwrap(); + } + Op::Remove(i) => { + shared.remove(&doc(*i)); + } + } + if n % 40 == 0 { + std::thread::yield_now(); + } + } + stop.store(true, Ordering::SeqCst); + }); + let readers: Vec<_> = (0..3) + .map(|r| { + let (shared, stop) = (&shared, &stop); + let clusters = &clusters; + s.spawn(move || { + let mut rng = Rng(900 + r as u64); + let mut mine = Vec::new(); + while !stop.load(Ordering::SeqCst) && mine.len() < 500 { + let k = 1 + rng.below(12); + let qs = clusters.draw(if r == 2 { 3 } else { 1 }, &mut rng); + let results = if r == 2 { + shared + .similarity_search_vector_batch(&qs, k, None, -2.0) + .unwrap() + } else { + vec![shared + .similarity_search_vector(&qs[0], k, None, -2.0) + .unwrap()] + }; + for (q, res) in qs.into_iter().zip(results) { + mine.push(( + q, + k, + res.iter().map(|x| (raw_id(x), x.score)).collect::<Vec<_>>(), + )); + } + } + mine + }) + }) + .collect(); + writer.join().unwrap(); + readers + .into_iter() + .flat_map(|r| r.join().unwrap()) + .collect() + }); + + assert!(observed.len() > 100, "readers managed {}", observed.len()); + for (q, k, res) in &observed { + // sorted, no duplicates, no more than asked, and every score is the + // cosine of *some* version the row ever had (never a blend) + assert!(res.len() <= *k); + assert!(res.windows(2).all(|w| w[0].1 >= w[1].1)); + let uniq: HashSet<usize> = res.iter().map(|r| r.0).collect(); + assert_eq!(uniq.len(), res.len()); + for &(id, score) in res { + let versions = &model.history[&id]; + assert!( + versions + .iter() + .any(|v| (cosine(q, v) - score as f64).abs() < 2e-5), + "id {id} scored {score}, matching none of its {} versions", + versions.len() + ); + } + } + + // the finished store: sound, and exactly the model + let store = shared.read(); + store.verify_invariants().unwrap(); + assert_eq!(store.count(), final_live.len()); + for q in clusters.draw(20, &mut rng) { + let res = store.similarity_search_vector(&q, 10, None, -2.0).unwrap(); + let mut final_model = Model::default(); + for &id in &final_live { + final_model.put(id, model.history[&id].last().unwrap().clone()); + } + check_against_model(&final_model, &q, &res, "final"); + } +} + +#[test] +fn concurrent_readers_agree_with_each_other() { + let mut fx = fixture(3000); + fx.store.build_ann_index(false).unwrap(); + let queries = fx.queries(24); + let store = std::sync::Arc::new(fx.store); + let want: Vec<_> = queries + .iter() + .map(|q| bits(&store.similarity_search_vector(q, 8, None, -1.0).unwrap())) + .collect(); + std::thread::scope(|s| { + for t in 0..4 { + let (store, queries, want) = (&store, &queries, &want); + s.spawn(move || { + for i in 0..60 { + let at = (i + t) % queries.len(); + let res = store + .similarity_search_vector(&queries[at], 8, None, -1.0) + .unwrap(); + assert_eq!(bits(&res), want[at]); + if i % 20 == 0 { + let b = store + .similarity_search_vector_batch(&queries[..5], 8, None, -1.0) + .unwrap(); + for (j, got) in b.iter().enumerate() { + assert_eq!(bits(got), want[j]); + } + } + } + }); + } + }); +} + +// -- the index never reaches a file or the digest ----------------------------- + +#[test] +fn a_loaded_graph_rebuilds_lazily_and_persists_nothing() { + let (graph, clusters, mut rng) = graph_with(1500, 32, Some(hnsw_config())); + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("g.ison"); + graph.save(&path, None, true).unwrap(); + let mut loaded = SemanticGraph::load(&path, Box::new(MockEncoder::new(32))).unwrap(); + loaded.set_ann(Some(hnsw_config())).unwrap(); + assert!(!loaded.ann_stats().active, "nothing was persisted"); + let q = clusters.draw(1, &mut rng).remove(0); + let res = loaded + .similarity_search_with(&q, &SimilarityOptions::default()) + .unwrap(); + assert_eq!(res.len(), 10); + let stats = loaded.ann_stats(); + assert!(stats.active && stats.mode == AnnMode::Hnsw); + assert_eq!(stats.graph_nodes, 1500); + // the answers equal the original's: same data, same slot order, same seed + let original = graph + .similarity_search_with(&q, &SimilarityOptions::default()) + .unwrap(); + assert_eq!(bits(&res), bits(&original)); + // and the rebuilt graph is as good + let queries = clusters.draw(30, &mut rng); + assert!(loaded.recall_check(&queries, 10, None, 0.0).unwrap().recall >= RECALL_FLOOR); + loaded.embedding_store().verify_invariants().unwrap(); +} + +#[test] +fn graph_files_sidecar_and_digest_are_the_same_with_hnsw_on() { + let (off, ..) = graph_with(600, 16, None); + let (on, clusters, mut rng) = graph_with(600, 16, Some(hnsw_config())); + // exercise the index before saving: build, search, filter + let q = clusters.draw(1, &mut rng).remove(0); + on.similarity_search_with(&q, &SimilarityOptions::default()) + .unwrap(); + assert!(on.ann_stats().active); + + let dir = tempfile::tempdir().unwrap(); + let (p_off, p_on) = (dir.path().join("off.ison"), dir.path().join("on.ison")); + off.save(&p_off, None, true).unwrap(); + on.save(&p_on, None, true).unwrap(); + assert_eq!( + std::fs::read(&p_off).unwrap(), + std::fs::read(&p_on).unwrap(), + "the graph file must be byte-identical" + ); + let (s_off, s_on) = ( + SemanticGraph::embeddings_path_for(&p_off), + SemanticGraph::embeddings_path_for(&p_on), + ); + let (side_off, side_on) = ( + std::fs::read(&s_off).unwrap(), + std::fs::read(&s_on).unwrap(), + ); + assert_eq!(side_off, side_on, "the sidecar must be byte-identical"); + let text = String::from_utf8(side_on).unwrap().to_lowercase(); + for word in ["hnsw", "ef_search", "ef_construction", "neighbor", "\"ann"] { + assert!(!text.contains(word), "sidecar mentions {word}"); + } + assert_eq!(graph_digest(off.graph()), graph_digest(on.graph())); + let mut names: Vec<_> = std::fs::read_dir(dir.path()) + .unwrap() + .map(|e| e.unwrap().file_name().to_string_lossy().into_owned()) + .collect(); + names.sort(); + assert_eq!(names.len(), 4, "nothing extra for the index: {names:?}"); + assert_eq!( + off.embedding_store().to_json().unwrap(), + on.embedding_store().to_json().unwrap() + ); +} + +#[test] +fn graph_edits_reach_the_index() { + let mut graph = SemanticGraph::new("g", Box::new(MockEncoder::new(48))); + for i in 0..300 { + graph + .add_node( + "doc", + &i.to_string(), + vec![( + "name", + PropertyValue::String(format!("document number {i}")), + )], + None, + ) + .unwrap(); + } + graph + .set_ann(Some(AnnConfig { + hnsw: HnswConfig { + ef_search: Some(150), // half the store: the graph still answers + ..HnswConfig::default() + }, + ..hnsw_config() + })) + .unwrap(); + let top = |graph: &SemanticGraph, text: &str| { + graph + .similarity_search_with( + text, + &SimilarityOptions { + top_k: Some(3), + threshold: -1.0, + ..SimilarityOptions::default() + }, + ) + .unwrap() + }; + assert_eq!(top(&graph, "document number 5")[0].node_id, doc(5)); + graph + .update_node( + "doc", + "5", + vec![( + "name", + PropertyValue::String("completely different words".into()), + )], + ) + .unwrap(); + assert_eq!(top(&graph, "completely different words")[0].node_id, doc(5)); + assert_ne!(top(&graph, "document number 5")[0].node_id, doc(5)); + graph.remove_node("doc", "9").unwrap(); + let all = graph + .similarity_search_with( + "document number 9", + &SimilarityOptions { + top_k: None, + threshold: -1.0, + ..SimilarityOptions::default() + }, + ) + .unwrap(); + assert_eq!(all.len(), 299); + assert!(all.iter().all(|r| r.node_id != doc(9))); +} diff --git a/isongraph-vector-rs/tests/memory_layout.rs b/isongraph-vector-rs/tests/memory_layout.rs new file mode 100644 index 0000000..ba79489 --- /dev/null +++ b/isongraph-vector-rs/tests/memory_layout.rs @@ -0,0 +1,106 @@ +//! The memory layout: every vector is stored once. A counting allocator +//! measures the live heap, so the test sees the real cost of a store (and would +//! see a second copy of the vectors, which an `EmbeddingStore` method could not +//! report on its own behalf). +//! +//! This file holds a single test so no other test thread allocates meanwhile. + +use isongraph_vector_rs::{AnnConfig, EmbeddingStore, NodeId}; +use std::alloc::{GlobalAlloc, Layout, System}; +use std::sync::atomic::{AtomicUsize, Ordering}; + +struct Counting; + +static LIVE: AtomicUsize = AtomicUsize::new(0); + +unsafe impl GlobalAlloc for Counting { + unsafe fn alloc(&self, layout: Layout) -> *mut u8 { + LIVE.fetch_add(layout.size(), Ordering::Relaxed); + System.alloc(layout) + } + unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) { + LIVE.fetch_sub(layout.size(), Ordering::Relaxed); + System.dealloc(ptr, layout) + } + unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 { + if new_size >= layout.size() { + LIVE.fetch_add(new_size - layout.size(), Ordering::Relaxed); + } else { + LIVE.fetch_sub(layout.size() - new_size, Ordering::Relaxed); + } + System.realloc(ptr, layout, new_size) + } +} + +#[global_allocator] +static ALLOC: Counting = Counting; + +fn vector(i: usize, dim: usize) -> Vec<f32> { + let mut x = (i as u64 + 1).wrapping_mul(0x9E37_79B9_7F4A_7C15); + (0..dim) + .map(|_| { + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + (x >> 40) as f32 / (1u64 << 24) as f32 - 0.5 + }) + .collect() +} + +#[test] +fn the_vectors_are_held_once() { + const N: usize = 20_000; + const DIM: usize = 384; + let raw = N * DIM * 4; + + let before = LIVE.load(Ordering::Relaxed); + let mut store = EmbeddingStore::new(None); + for i in 0..N { + store + .add(NodeId::new("doc", i.to_string()), "", Some(vector(i, DIM))) + .unwrap(); + } + // the first search used to build a second, contiguous copy + let q = vector(N + 1, DIM); + store.similarity_search_vector(&q, 10, None, -1.0).unwrap(); + store + .set_ann(Some(AnnConfig { + nlist: Some(64), + min_size: 0, + ..AnnConfig::default() + })) + .unwrap(); + store.build_ann_index(false).unwrap(); + store.similarity_search_vector(&q, 10, None, -1.0).unwrap(); + let held = LIVE.load(Ordering::Relaxed) - before; + let ratio = held as f64 / raw as f64; + eprintln!("{N} x {DIM}: raw {raw} bytes, store holds {held} ({ratio:.3}x)"); + + // one copy of the floats plus per-record metadata and the index: under 1.2x + // (the old layout, records plus a scan cache, was about 2.1x) + assert!( + ratio < 1.2, + "the store holds {ratio:.2}x the raw vector bytes" + ); + + // a record is a row of the matrix: the record built on request carries the vector + let id = NodeId::new("doc", "123"); + assert_eq!(store.get(&id).unwrap().vector, vector(123, DIM)); + + // an update overwrites its row in place: the heap does not grow + let steady = LIVE.load(Ordering::Relaxed); + for i in 0..2_000 { + store + .add( + NodeId::new("doc", i.to_string()), + "", + Some(vector(i + 7, DIM)), + ) + .unwrap(); + } + let grown = LIVE.load(Ordering::Relaxed).abs_diff(steady); + assert!( + grown < 2_000 * 400, + "2,000 in-place updates grew the heap by {grown} bytes" + ); +} diff --git a/isongraph-vector-rs/tests/quant.rs b/isongraph-vector-rs/tests/quant.rs new file mode 100644 index 0000000..7cbc0a4 --- /dev/null +++ b/isongraph-vector-rs/tests/quant.rs @@ -0,0 +1,578 @@ +//! The opt-in int8 scan with exact rerank: recall, exactness of returned +//! scores, batch == single bit for bit, consistency of the int8 copy under +//! writes (against a twin store and a model), concurrency, filters and +//! tombstones, and that nothing it keeps reaches a sidecar or digest. + +#![allow(clippy::manual_is_multiple_of)] + +use isongraph_vector_rs::{ + AnnConfig, EmbeddingStore, NodeId, ScanPrecision, SearchPool, SharedEmbeddingStore, + SimilarityResult, DEFAULT_RERANK_FACTOR, +}; +use std::collections::{HashMap, HashSet}; +use std::sync::Arc; + +struct Rng(u64); + +impl Rng { + fn next_u64(&mut self) -> u64 { + self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } + fn unit(&mut self) -> f32 { + (self.next_u64() >> 40) as f32 / (1u64 << 24) as f32 + } + fn gauss(&mut self) -> f32 { + let u1 = self.unit().max(1e-7); + let u2 = self.unit(); + (-2.0 * u1.ln()).sqrt() * (std::f32::consts::TAU * u2).cos() + } + fn below(&mut self, n: usize) -> usize { + (self.next_u64() % n as u64) as usize + } +} + +/// Clustered unit-ish vectors from a seeded generator (same recipe as the IVF +/// tests): 40 centres, noise of about one unit norm. +struct Clusters { + centers: Vec<Vec<f32>>, + dim: usize, +} + +impl Clusters { + fn new(dim: usize, clusters: usize, seed: u64) -> Self { + let mut rng = Rng(seed); + let centers = (0..clusters) + .map(|_| { + let mut c: Vec<f32> = (0..dim).map(|_| rng.gauss()).collect(); + let norm = c.iter().map(|x| x * x).sum::<f32>().sqrt(); + c.iter_mut().for_each(|x| *x /= norm); + c + }) + .collect(); + Self { centers, dim } + } + + fn draw(&self, count: usize, rng: &mut Rng) -> Vec<Vec<f32>> { + let scale = 1.0 / (self.dim as f32).sqrt(); + (0..count) + .map(|_| { + let c = &self.centers[rng.below(self.centers.len())]; + c.iter().map(|x| x + rng.gauss() * scale).collect() + }) + .collect() + } +} + +fn doc(ty: &str, i: usize) -> NodeId { + NodeId::new(ty, i.to_string()) +} + +fn key(r: &SimilarityResult) -> String { + format!("{}:{}", r.node_id.node_type, r.node_id.id) +} + +fn keys(results: &[SimilarityResult]) -> Vec<String> { + results.iter().map(key).collect() +} + +fn bits(results: &[SimilarityResult]) -> Vec<(String, u32)> { + results + .iter() + .map(|r| (key(r), r.score.to_bits())) + .collect() +} + +fn filled(data: &[Vec<f32>], type_of: impl Fn(usize) -> &'static str) -> EmbeddingStore { + let mut store = EmbeddingStore::new(None); + for (i, v) in data.iter().enumerate() { + store + .add(doc(type_of(i), i), "", Some(v.clone())) + .expect("add"); + } + store +} + +fn int8(factor: usize) -> ScanPrecision { + ScanPrecision::Int8 { + rerank_factor: factor, + } +} + +#[test] +fn default_rerank_reaches_99_percent_recall_on_clustered_data() { + let (dim, n) = (128, 12_000); + let clusters = Clusters::new(dim, 40, 1); + let mut rng = Rng(77); + let data = clusters.draw(n, &mut rng); + let queries = clusters.draw(200, &mut rng); + let mut store = filled(&data, |_| "doc"); + store.set_scan_precision(ScanPrecision::int8()).unwrap(); + assert_eq!( + store.scan_precision(), + int8(DEFAULT_RERANK_FACTOR), + "default factor" + ); + let report = store + .int8_recall_report(&queries, 10, None, -1.0, &[1, 2, DEFAULT_RERANK_FACTOR, 8]) + .unwrap(); + for r in &report { + eprintln!( + "factor {}: recall {:.4} (worst {:.2}) int8 {:.3} ms f32 {:.3} ms", + r.rerank_factor, r.recall, r.min_recall, r.int8_ms, r.f32_ms + ); + assert!(r.scores_exact, "factor {}", r.rerank_factor); + assert!(r.applies); + } + let at_default = report + .iter() + .find(|r| r.rerank_factor == DEFAULT_RERANK_FACTOR) + .unwrap(); + assert!(at_default.recall >= 0.99, "recall {}", at_default.recall); + assert!( + report[3].recall >= at_default.recall - 1e-12 && report[3].recall >= 0.995, + "a larger factor does not lose recall" + ); +} + +#[test] +fn returned_scores_are_the_exact_f32_cosines_and_ties_keep_slot_order() { + let (dim, n) = (100, 3000); + let clusters = Clusters::new(dim, 12, 3); + let mut rng = Rng(5); + let mut data = clusters.draw(n, &mut rng); + // exact duplicates, to exercise the tie order + for i in 0..40 { + data[n - 1 - i] = data[i].clone(); + } + let exact = filled(&data, |_| "doc"); + let mut quant = filled(&data, |_| "doc"); + quant.set_scan_precision(int8(4)).unwrap(); + for q in clusters.draw(50, &mut rng).iter().chain(&data[..10]) { + let want = exact.score_map(q).unwrap(); + let got = quant.similarity_search_vector(q, 10, None, -1.0).unwrap(); + assert_eq!(got.len(), 10); + for r in &got { + assert_eq!( + r.score.to_bits(), + want[&key(r)].to_bits(), + "{} is not the exact cosine", + key(r) + ); + } + assert!(got.windows(2).all(|w| { + w[0].score > w[1].score + || (w[0].score == w[1].score + && w[0].node_id.id.parse::<usize>().unwrap() + < w[1].node_id.id.parse::<usize>().unwrap()) + })); + } + // a vector queried with itself: its duplicate ties with it, lower slot first + let got = quant + .similarity_search_vector(&data[3], 2, None, -1.0) + .unwrap(); + assert_eq!( + keys(&got), + vec!["doc:3".to_string(), format!("doc:{}", n - 1 - 3)] + ); + assert_eq!(got[0].score.to_bits(), got[1].score.to_bits()); +} + +#[test] +fn batch_equals_single_bit_for_bit_exact_and_ivf_and_across_threads() { + let (dim, n) = (96, 6000); + let clusters = Clusters::new(dim, 30, 9); + let mut rng = Rng(11); + let data = clusters.draw(n, &mut rng); + let queries = clusters.draw(70, &mut rng); + let ann = AnnConfig { + nlist: Some(48), + nprobe: Some(6), + min_size: 0, + ..AnnConfig::default() + }; + for config in [None, Some(ann)] { + let mut store = filled(&data, |i| match i { + _ if i % 500 == 0 => "rare", // a dozen rows: probes come up short + _ if i % 3 == 0 => "a", + _ => "b", + }); + if let Some(c) = config.clone() { + store.set_ann(Some(c)).unwrap(); + } + store.set_scan_precision(int8(4)).unwrap(); + for filter in [None, Some("a"), Some("rare")] { + let singles: Vec<_> = queries + .iter() + .map(|q| store.similarity_search_vector(q, 10, filter, -1.0).unwrap()) + .collect(); + for threads in [1usize, 3, 0] { + store.set_scan_threads(threads); + if threads == 3 { + store.set_search_pool(Some(Arc::new(SearchPool::new(3)))); + } + let batch = store + .similarity_search_vector_batch(&queries, 10, filter, -1.0) + .unwrap(); + for (i, (b, s)) in batch.iter().zip(&singles).enumerate() { + assert_eq!(bits(b), bits(s), "query {i}, threads {threads}, {config:?}"); + } + // and one query is the same on any thread count + let alone = store + .similarity_search_vector(&queries[0], 10, filter, -1.0) + .unwrap(); + assert_eq!(bits(&alone), bits(&singles[0])); + } + store.set_scan_threads(0); + store.set_search_pool(None); + } + } +} + +#[test] +fn ivf_with_int8_keeps_the_ivf_recall_and_exact_scores() { + let (dim, n) = (96, 8000); + let clusters = Clusters::new(dim, 40, 2); + let mut rng = Rng(21); + let data = clusters.draw(n, &mut rng); + let queries = clusters.draw(100, &mut rng); + let ann = AnnConfig { + nlist: Some(64), + nprobe: Some(8), + min_size: 0, + ..AnnConfig::default() + }; + let mut plain = filled(&data, |_| "doc"); + plain.set_ann(Some(ann.clone())).unwrap(); + let mut quant = filled(&data, |_| "doc"); + quant.set_ann(Some(ann)).unwrap(); + quant.set_scan_precision(ScanPrecision::int8()).unwrap(); + let (mut same, mut total) = (0usize, 0usize); + for q in &queries { + let a = plain.similarity_search_vector(q, 10, None, -1.0).unwrap(); + let b = quant.similarity_search_vector(q, 10, None, -1.0).unwrap(); + let wanted: HashSet<String> = keys(&a).into_iter().collect(); + total += wanted.len(); + same += keys(&b).iter().filter(|k| wanted.contains(*k)).count(); + let scores = plain.score_map(q).unwrap(); + for r in &b { + assert_eq!(r.score.to_bits(), scores[&key(r)].to_bits()); + } + } + let recall = same as f64 / total as f64; + eprintln!("IVF f32 vs IVF int8 set overlap: {recall:.4}"); + assert!(recall >= 0.99, "{recall}"); + let stats = quant.ann_stats(); + assert!(stats.active && stats.searches.approximate >= queries.len() as u64); +} + +#[test] +fn int8_copy_follows_writes_updates_deletes_and_compaction() { + let (dim, n) = (50, 600); // 50 is not a multiple of 32: the scalar tail runs + let clusters = Clusters::new(dim, 8, 4); + let mut rng = Rng(8); + let data = clusters.draw(n + 2000, &mut rng); + let mut twin = EmbeddingStore::new(None); + let mut store = EmbeddingStore::new(None); + store.set_scan_precision(int8(8)).unwrap(); // enabled while empty + let mut model: HashMap<usize, usize> = HashMap::new(); // id -> data index + let mut next = 0usize; + let (mut hit, mut want_total) = (0usize, 0usize); + for step in 0..2500 { + match rng.below(10) { + 0..=4 => { + // insert + if next < data.len() { + for s in [&mut twin, &mut store] { + s.add(doc("doc", next), "", Some(data[next].clone())) + .unwrap(); + } + model.insert(next, next); + next += 1; + } + } + 5 | 6 => { + // update an existing id with another vector + if let Some(&id) = model.keys().next() { + let v = rng.below(data.len()); + for s in [&mut twin, &mut store] { + s.add(doc("doc", id), "", Some(data[v].clone())).unwrap(); + } + model.insert(id, v); + } + } + 7 | 8 => { + // delete + if let Some(&id) = model.keys().nth(rng.below(model.len().max(1))) { + for s in [&mut twin, &mut store] { + assert!(s.remove(&doc("doc", id))); + } + model.remove(&id); + } + } + _ => { + if step % 4 == 0 { + twin.compact(); + store.compact(); + } + } + } + if step % 25 == 0 { + store.verify_invariants().unwrap(); + assert_eq!(store.count(), model.len()); + let q = &data[rng.below(data.len())]; + let want = twin.score_map(q).unwrap(); + let got = store.similarity_search_vector(q, 3, None, -1.0).unwrap(); + let live: HashSet<String> = model.keys().map(|i| format!("doc:{i}")).collect(); + for r in &got { + assert!( + live.contains(&key(r)), + "{} was deleted or never stored", + key(r) + ); + assert_eq!( + r.score.to_bits(), + want[&key(r)].to_bits(), + "stale or inexact score" + ); + } + let truth = twin + .similarity_search_vector_exact(q, 3, None, -1.0) + .unwrap(); + want_total += truth.len(); + let wanted: HashSet<String> = keys(&truth).into_iter().collect(); + hit += keys(&got).iter().filter(|k| wanted.contains(*k)).count(); + assert_eq!(got.len(), truth.len()); + } + } + store.verify_invariants().unwrap(); + let recall = hit as f64 / want_total as f64; + eprintln!("recall against the twin during the churn: {recall:.4}"); + assert!(recall >= 0.95, "{recall}"); + // every surviving row: its int8 copy was verified above; its stored vector + // is the model's + for (&id, &v) in &model { + assert_eq!(store.get_vector(&doc("doc", id)).unwrap(), &data[v][..]); + } +} + +#[test] +fn switching_precision_is_clean_and_costs_a_quarter_of_the_matrix() { + let (dim, n) = (384, 3000); + let clusters = Clusters::new(dim, 20, 6); + let mut rng = Rng(2); + let data = clusters.draw(n, &mut rng); + let mut store = filled(&data, |_| "doc"); + assert_eq!(store.quant_bytes(), 0); + let q = &data[7]; + let before = store.similarity_search_vector(q, 5, None, -1.0).unwrap(); + store.set_scan_precision(int8(4)).unwrap(); + store.verify_invariants().unwrap(); + let matrix = store.memory_stats().matrix_bytes as f64; + let ratio = store.quant_bytes() as f64 / matrix; + eprintln!("int8 copy / f32 matrix = {ratio:.4}"); + assert!((0.25..0.27).contains(&ratio), "{ratio}"); + let with = store.similarity_search_vector(q, 5, None, -1.0).unwrap(); + assert_eq!( + bits(&before), + bits(&with), + "the query's own neighbours survive" + ); + store.set_scan_precision(ScanPrecision::F32).unwrap(); + assert_eq!(store.quant_bytes(), 0); + assert_eq!(store.scan_precision(), ScanPrecision::F32); + let after = store.similarity_search_vector(q, 5, None, -1.0).unwrap(); + assert_eq!(bits(&before), bits(&after)); + assert!(store.set_scan_precision(int8(0)).is_err()); + // clearing and refilling with another width keeps the copy in step + store.set_scan_precision(int8(2)).unwrap(); + store.clear(); + store.add(doc("doc", 1), "", Some(vec![1.0; 40])).unwrap(); + store.add(doc("doc", 2), "", Some(vec![-1.0; 40])).unwrap(); + store.verify_invariants().unwrap(); +} + +#[test] +fn filters_thresholds_and_tombstones() { + let (dim, n) = (64, 4000); + let clusters = Clusters::new(dim, 16, 12); + let mut rng = Rng(31); + let data = clusters.draw(n, &mut rng); + let kind = |i: usize| match i % 4 { + 0 => "alpha", + 1 => "beta", + _ => "gamma", + }; + let mut twin = filled(&data, kind); + let mut store = filled(&data, kind); + store.set_scan_precision(int8(6)).unwrap(); + // tombstone a third of the rows, but stay under the compaction threshold + // for part of the test + let mut dead = HashSet::new(); + for i in (0..n).step_by(5) { + let id = doc(kind(i), i); + twin.remove(&id); + store.remove(&id); + dead.insert(format!("{}:{}", kind(i), i)); + } + store.verify_invariants().unwrap(); + for ty in [None, Some("alpha"), Some("gamma"), Some("missing")] { + for threshold in [-1.0f32, 0.3, 0.6] { + for q in clusters.draw(15, &mut rng) { + let got = store + .similarity_search_vector(&q, 8, ty, threshold) + .unwrap(); + let want = twin + .similarity_search_vector_exact(&q, 8, ty, threshold) + .unwrap(); + let scores = twin.score_map(&q).unwrap(); + for r in &got { + assert!(!dead.contains(&key(r))); + assert!(ty.is_none_or(|t| r.node_id.node_type == t)); + assert!(r.score >= threshold); + assert_eq!(r.score.to_bits(), scores[&key(r)].to_bits()); + } + if ty == Some("missing") { + assert!(got.is_empty()); + } + // the same rows when the threshold leaves few candidates, and a + // large overlap otherwise + let wanted: HashSet<String> = keys(&want).into_iter().collect(); + let hit = keys(&got).iter().filter(|k| wanted.contains(*k)).count(); + assert!( + hit as f64 >= 0.75 * wanted.len() as f64, + "{ty:?} thr {threshold}: {hit}/{}", + wanted.len() + ); + if want.len() < 8 { + assert_eq!(keys(&got), keys(&want), "short answers are exact"); + } + } + } + } +} + +#[test] +fn odd_inputs_fall_back_or_work() { + let mut store = EmbeddingStore::new(None); + store.set_scan_precision(int8(4)).unwrap(); + for i in 0..50 { + let mut v = vec![0.0f32; 7]; + v[i % 7] = 1.0 + (i / 7) as f32; + store.add(doc("doc", i), "", Some(v)).unwrap(); + } + store.add(doc("zero", 0), "", Some(vec![0.0; 7])).unwrap(); + store.verify_invariants().unwrap(); + // a zero query has no direction: every score is 0, as in the f32 scan + let z = store + .similarity_search_vector(&[0.0; 7], 3, None, -1.0) + .unwrap(); + assert_eq!(z.len(), 3); + assert!(z.iter().all(|r| r.score == 0.0)); + // k >= live: the f32 path, everything comes back + let all = store + .similarity_search_vector(&[1.0; 7], usize::MAX, None, -1.0) + .unwrap(); + assert_eq!(all.len(), 51); + let some = store.similarity_search_vector(&[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 5, None, -1.0); + assert_eq!(some.unwrap().len(), 5); +} + +#[test] +fn concurrent_readers_and_a_writer_stay_consistent() { + let (dim, n) = (64, 3000); + let clusters = Clusters::new(dim, 16, 14); + let mut rng = Rng(3); + let data = Arc::new(clusters.draw(n + 4000, &mut rng)); + let mut base = filled(&data[..n], |_| "doc"); + base.set_scan_precision(int8(4)).unwrap(); + base.set_ann(Some(AnnConfig { + nlist: Some(32), + nprobe: Some(4), + min_size: 0, + ..AnnConfig::default() + })) + .unwrap(); + let shared = SharedEmbeddingStore::new(base); + let queries = Arc::new(clusters.draw(64, &mut rng)); + let stop = Arc::new(std::sync::atomic::AtomicBool::new(false)); + + let writer = { + let (shared, data, stop) = (shared.clone(), data.clone(), stop.clone()); + std::thread::spawn(move || { + let mut rng = Rng(99); + for i in 0..4000usize { + match rng.below(3) { + 0 | 1 => { + let id = n + i; + shared + .add(doc("doc", id), "", Some(data[id].clone())) + .unwrap(); + } + _ => { + shared.remove(&doc("doc", rng.below(n + i))); + } + } + if i % 500 == 0 { + shared.write().compact(); + } + } + stop.store(true, std::sync::atomic::Ordering::SeqCst); + }) + }; + let readers: Vec<_> = (0..4) + .map(|t| { + let (shared, queries, stop) = (shared.clone(), queries.clone(), stop.clone()); + std::thread::spawn(move || { + let mut searches = 0usize; + while !stop.load(std::sync::atomic::Ordering::SeqCst) || searches < 20 { + let q = &queries[(searches + t) % queries.len()]; + let results = if searches % 3 == 0 { + shared + .similarity_search_vector_batch(&queries[..8], 5, None, -1.0) + .unwrap() + } else { + vec![shared.similarity_search_vector(q, 5, None, -1.0).unwrap()] + }; + for r in results { + let ks = keys(&r); + let unique: HashSet<&String> = ks.iter().collect(); + assert_eq!(unique.len(), ks.len(), "a row twice"); + assert!(r.windows(2).all(|w| w[0].score >= w[1].score)); + assert!(r + .iter() + .all(|x| x.score.is_finite() && x.score.abs() <= 1.0001)); + } + searches += 1; + } + searches + }) + }) + .collect(); + writer.join().unwrap(); + let ran: usize = readers.into_iter().map(|h| h.join().unwrap()).sum(); + assert!(ran >= 80); + shared.read().verify_invariants().unwrap(); +} + +#[test] +fn the_int8_copy_never_reaches_a_sidecar_or_the_digest() { + let (dim, n) = (48, 200); + let clusters = Clusters::new(dim, 5, 15); + let mut rng = Rng(4); + let data = clusters.draw(n, &mut rng); + let plain = filled(&data, |_| "doc"); + let mut quant = filled(&data, |_| "doc"); + quant.set_scan_precision(ScanPrecision::int8()).unwrap(); + assert_eq!( + plain.export_embeddings_with(Some("d")).embeddings.len(), + quant.export_embeddings_with(Some("d")).embeddings.len() + ); + assert_eq!( + serde_json::to_string(&plain.export_embeddings_with(Some("d"))).unwrap(), + serde_json::to_string(&quant.export_embeddings_with(Some("d"))).unwrap() + ); +} diff --git a/isongraph-vector-rs/tests/snapshot.rs b/isongraph-vector-rs/tests/snapshot.rs new file mode 100644 index 0000000..3f4a23f --- /dev/null +++ b/isongraph-vector-rs/tests/snapshot.rs @@ -0,0 +1,825 @@ +//! The optional binary snapshot beside the JSON sidecar: a load from it is the +//! JSON load bit for bit; anything doubtful (corrupt, truncated, stale, for a +//! different graph) is refused and the JSON is read instead; the graph file and +//! the sidecar bytes are what they always were. + +use isongraph_vector_rs::{ + snapshot_path_for, AnnConfig, EmbeddingError, EmbeddingStore, GraphMismatch, LoadOptions, + LoadSource, MockEncoder, NodeId, ScanPrecision, SemanticGraph, SidecarId, SnapshotIdentity, + SNAPSHOT_SUFFIX, +}; +use sha2::{Digest, Sha256}; +use std::path::{Path, PathBuf}; + +struct Rng(u64); + +impl Rng { + fn next_u64(&mut self) -> u64 { + self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } + fn f(&mut self) -> f32 { + ((self.next_u64() >> 40) as f32 / (1u64 << 23) as f32) - 1.0 + } +} + +const DIM: usize = 12; + +fn encoder() -> Box<dyn isongraph_vector_rs::EmbeddingEncoder> { + Box::new(MockEncoder::new(DIM)) +} + +/// A graph whose store holds `n` rows of three types, ids with colons and +/// unicode, text of every shape (empty, quotes, newlines), exact duplicates +/// (ties), two models, updates, and tombstoned rows left behind by deletes. +fn graph(n: usize, seed: u64) -> SemanticGraph { + let mut g = SemanticGraph::new("g", encoder()); + g.add_node_with_embed("person", "1", vec![("name", "Alice")], None) + .unwrap(); + g.add_node_with_embed("person", "2", vec![("name", "Bob")], None) + .unwrap(); + g.add_edge("KNOWS", ("person", "1"), ("person", "2"), vec![]) + .unwrap(); + let mut rng = Rng(seed); + let types = ["doc", "note", "a:b"]; + let mut vectors: Vec<Vec<f32>> = Vec::new(); + for i in 0..n { + let v: Vec<f32> = if i % 17 == 5 && !vectors.is_empty() { + vectors[i / 2].clone() // an exact duplicate: a tie + } else if i % 41 == 7 { + vec![0.0; DIM] // a zero vector + } else { + (0..DIM).map(|_| rng.f()).collect() + }; + vectors.push(v.clone()); + if i == n / 2 { + g.set_encoder(Box::new( + MockEncoder::new(DIM).with_model_name("second-model"), + )); + } + let text = match i % 5 { + 0 => String::new(), + 1 => format!("quote \" and \\ and\nnewline {i}"), + 2 => format!("unicode \u{1F600} \u{00E9}\u{4E2D} {i}"), + _ => format!("text {i}"), + }; + let id = if i % 9 == 0 { + format!("id:with:colons:{i}") + } else { + format!("{i:05}") + }; + g.embedding_store_mut() + .add(NodeId::new(types[i % 3], id), &text, Some(v)) + .unwrap(); + } + // updates and deletes leave rewritten rows and tombstones behind + for i in (0..n).step_by(11) { + let id = if i % 9 == 0 { + format!("id:with:colons:{i}") + } else { + format!("{i:05}") + }; + let v: Vec<f32> = (0..DIM).map(|_| rng.f()).collect(); + g.embedding_store_mut() + .add(NodeId::new(types[i % 3], id), "updated", Some(v)) + .unwrap(); + } + for i in (3..n).step_by(7) { + let id = if i % 9 == 0 { + format!("id:with:colons:{i}") + } else { + format!("{i:05}") + }; + g.embedding_store_mut() + .remove(&NodeId::new(types[i % 3], id)); + } + g +} + +fn path_in(dir: &tempfile::TempDir, name: &str) -> PathBuf { + dir.path().join(name) +} + +/// Copy a saved pair (graph + sidecar, no snapshot) to a fresh name, so it can +/// only be loaded from the JSON. +fn json_only_copy(dir: &tempfile::TempDir, from: &Path, name: &str) -> PathBuf { + let to = path_in(dir, name); + std::fs::copy(from, &to).unwrap(); + std::fs::copy( + SemanticGraph::embeddings_path_for(from), + SemanticGraph::embeddings_path_for(&to), + ) + .unwrap(); + to +} + +fn load(path: &Path) -> (SemanticGraph, isongraph_vector_rs::LoadReport) { + SemanticGraph::load_with_report(path, encoder(), &LoadOptions::default()).unwrap() +} + +fn queries(count: usize) -> Vec<Vec<f32>> { + let mut rng = Rng(555); + (0..count) + .map(|_| (0..DIM).map(|_| rng.f()).collect()) + .collect() +} + +fn search_bits( + store: &EmbeddingStore, + q: &[f32], + k: usize, + ty: Option<&str>, +) -> Vec<(String, u32, String)> { + store + .similarity_search_vector(q, k, ty, -1.0) + .unwrap() + .into_iter() + .map(|r| { + ( + format!("{}:{}", r.node_id.node_type, r.node_id.id), + r.score.to_bits(), + r.text, + ) + }) + .collect() +} + +/// Everything observable about two stores is the same. +fn assert_same_store(a: &EmbeddingStore, b: &EmbeddingStore) { + a.verify_invariants().unwrap(); + b.verify_invariants().unwrap(); + assert_eq!(a.count(), b.count()); + assert_eq!(a.dimension(), b.dimension()); + assert_eq!(a.models(), b.models()); + assert_eq!(a.last_import_meta(), b.last_import_meta()); + assert_eq!( + serde_json::to_string(&a.export_embeddings_with(Some("x"))).unwrap(), + serde_json::to_string(&b.export_embeddings_with(Some("x"))).unwrap(), + "export differs" + ); + let ea = a.export_embeddings(); + for e in &ea.embeddings { + let id = NodeId::new(&e.node_type, e.id.as_string()); + let (ra, rb) = (a.get(&id).unwrap(), b.get(&id).unwrap()); + assert_eq!(ra.text, rb.text); + assert_eq!(ra.model, rb.model); + let (va, vb): (Vec<u32>, Vec<u32>) = ( + ra.vector.iter().map(|x| x.to_bits()).collect(), + rb.vector.iter().map(|x| x.to_bits()).collect(), + ); + assert_eq!(va, vb, "vector bits of {:?}", id); + } + for q in queries(30) { + for ty in [None, Some("doc"), Some("a:b")] { + for k in [1, 5, 40, usize::MAX] { + assert_eq!( + search_bits(a, &q, k, ty), + search_bits(b, &q, k, ty), + "k {k} {ty:?}" + ); + } + } + let (ma, mb) = (a.score_map(&q).unwrap(), b.score_map(&q).unwrap()); + assert_eq!(ma.len(), mb.len()); + for (k, v) in ma { + assert_eq!(v.to_bits(), mb[&k].to_bits()); + } + } +} + +#[test] +fn a_snapshot_load_equals_the_json_load_bit_for_bit() { + let dir = tempfile::tempdir().unwrap(); + // 2600 rows: two full blocks of 1024 and a partial one, with holes and ties + let g = graph(2600, 1); + assert!(g.embedding_store().verify_invariants().is_ok()); + let path = path_in(&dir, "g.ison"); + g.save_with_snapshot(&path, None).unwrap(); + assert!(snapshot_path_for(&path).is_file()); + assert!(snapshot_path_for(&path) + .to_string_lossy() + .ends_with(SNAPSHOT_SUFFIX)); + + let (from_snap, report) = load(&path); + assert_eq!( + report.source, + LoadSource::Snapshot, + "{:?}", + report.snapshot_skipped + ); + assert_eq!(report.snapshot_skipped, None); + assert_eq!(report.embeddings, g.embedding_store().count()); + + let json_path = json_only_copy(&dir, &path, "copy.ison"); + let (from_json, jreport) = load(&json_path); + assert_eq!(jreport.source, LoadSource::Json); + assert_eq!( + jreport.snapshot_skipped, None, + "no snapshot beside the copy" + ); + + assert_same_store(from_snap.embedding_store(), from_json.embedding_store()); + // the loaded graph is the same too + assert_eq!( + isongraph_vector_rs::graph_digest(from_snap.graph()), + isongraph_vector_rs::graph_digest(from_json.graph()) + ); +} + +#[test] +fn snapshot_loaded_stores_search_the_same_with_ivf_and_int8_on() { + let dir = tempfile::tempdir().unwrap(); + let g = graph(2200, 2); + let path = path_in(&dir, "g.ison"); + g.save_with_snapshot(&path, None).unwrap(); + let (mut a, ra) = load(&path); + let (mut b, rb) = load(&json_only_copy(&dir, &path, "c.ison")); + assert_eq!( + (ra.source, rb.source), + (LoadSource::Snapshot, LoadSource::Json) + ); + let ann = AnnConfig { + nlist: Some(16), + nprobe: Some(4), + min_size: 0, + ..AnnConfig::default() + }; + for g in [&mut a, &mut b] { + g.set_ann(Some(ann.clone())).unwrap(); + g.embedding_store_mut() + .set_scan_precision(ScanPrecision::int8()) + .unwrap(); + } + for q in queries(40) { + assert_eq!( + search_bits(a.embedding_store(), &q, 7, None), + search_bits(b.embedding_store(), &q, 7, None) + ); + } + assert_same_store(a.embedding_store(), b.embedding_store()); +} + +#[test] +fn the_graph_file_and_the_json_sidecar_are_byte_identical_with_or_without_a_snapshot() { + let dir = tempfile::tempdir().unwrap(); + let g = graph(300, 3); + let plain = path_in(&dir, "plain.ison"); + let snap = path_in(&dir, "snap.ison"); + g.save(&plain, None, true).unwrap(); + g.save_with_snapshot(&snap, None).unwrap(); + assert!(!snapshot_path_for(&plain).exists()); + assert_eq!( + std::fs::read(&plain).unwrap(), + std::fs::read(&snap).unwrap() + ); + let (sj_plain, sj_snap) = ( + std::fs::read(SemanticGraph::embeddings_path_for(&plain)).unwrap(), + std::fs::read(SemanticGraph::embeddings_path_for(&snap)).unwrap(), + ); + // the digest differs only through the file name (graph name), so compare + // after normalising: the sidecars are otherwise the same bytes + assert_eq!(sj_plain.len(), sj_snap.len()); + let text = |b: &[u8]| String::from_utf8(b.to_vec()).unwrap(); + assert_eq!(text(&sj_plain), text(&sj_snap)); + // the snapshot is much smaller than the JSON + let snap_len = std::fs::metadata(snapshot_path_for(&snap)).unwrap().len(); + eprintln!("json {} bytes, snapshot {} bytes", sj_snap.len(), snap_len); + assert!(snap_len < sj_snap.len() as u64); +} + +fn saved(dir: &tempfile::TempDir, n: usize) -> (SemanticGraph, PathBuf) { + let g = graph(n, 4); + let path = path_in(dir, "g.ison"); + g.save_with_snapshot(&path, None).unwrap(); + (g, path) +} + +/// Load after `damage` was applied to the snapshot file: the JSON must be +/// used, with a reason, and equal the undamaged load. +fn assert_falls_back(dir: &tempfile::TempDir, path: &Path, reason_has: &str) { + let (loaded, report) = load(path); + assert_eq!(report.source, LoadSource::Json, "damage accepted"); + let why = report.snapshot_skipped.clone().expect("a reason"); + assert!( + why.contains(reason_has), + "reason {why:?} should mention {reason_has:?}" + ); + let reference = json_only_copy(dir, path, &format!("ref-{}.ison", reason_has.len())); + let (expected, _) = load(&reference); + assert_same_store(loaded.embedding_store(), expected.embedding_store()); +} + +#[test] +fn a_flipped_byte_anywhere_is_caught_by_the_checksum() { + let dir = tempfile::tempdir().unwrap(); + let (_g, path) = saved(&dir, 1500); + let snap = snapshot_path_for(&path); + let original = std::fs::read(&snap).unwrap(); + let mut at = vec![ + 0usize, + 7, + 9, + 13, + 20, + 40, + original.len() / 3, + original.len() / 2, + ]; + at.extend([ + original.len() - 40, + original.len() - 33, + original.len() - 32, + original.len() - 1, + ]); + for position in at { + let mut damaged = original.clone(); + damaged[position] ^= 0x10; + std::fs::write(&snap, &damaged).unwrap(); + let (_, report) = load(&path); + assert_eq!( + report.source, + LoadSource::Json, + "flip at {position} accepted" + ); + assert!(report.snapshot_skipped.is_some()); + } + std::fs::write(&snap, &original).unwrap(); + let (_, report) = load(&path); + assert_eq!( + report.source, + LoadSource::Snapshot, + "{:?}", + report.snapshot_skipped + ); +} + +#[test] +fn a_float_flip_reports_a_checksum_mismatch_and_loads_the_json() { + let dir = tempfile::tempdir().unwrap(); + let (_g, path) = saved(&dir, 1500); + let snap = snapshot_path_for(&path); + let mut bytes = std::fs::read(&snap).unwrap(); + let mid = bytes.len() / 2; + bytes[mid] ^= 0x01; + std::fs::write(&snap, &bytes).unwrap(); + assert_falls_back(&dir, &path, "checksum"); +} + +#[test] +fn truncated_snapshots_are_refused_at_every_length() { + let dir = tempfile::tempdir().unwrap(); + let (_g, path) = saved(&dir, 1300); + let snap = snapshot_path_for(&path); + let original = std::fs::read(&snap).unwrap(); + let len = original.len(); + for cut in [ + 0, + 1, + 7, + 8, + 11, + 31, + 33, + 100, + 500, + len / 2, + len - 33, + len - 32, + len - 31, + len - 1, + ] { + std::fs::write(&snap, &original[..cut]).unwrap(); + let (_, report) = load(&path); + assert_eq!( + report.source, + LoadSource::Json, + "cut at {cut} of {len} accepted" + ); + assert!(report.snapshot_skipped.is_some(), "cut {cut}"); + } + // and trailing bytes + let mut longer = original.clone(); + longer.push(0); + std::fs::write(&snap, &longer).unwrap(); + assert_eq!(load(&path).1.source, LoadSource::Json); +} + +fn reseal(mut bytes: Vec<u8>) -> Vec<u8> { + let n = bytes.len() - 32; + let sum = Sha256::digest(&bytes[..n]); + bytes[n..].copy_from_slice(&sum); + bytes +} + +#[test] +fn a_well_formed_looking_but_wrong_snapshot_is_refused_even_with_a_valid_checksum() { + let dir = tempfile::tempdir().unwrap(); + let (_g, path) = saved(&dir, 1200); + let snap = snapshot_path_for(&path); + let original = std::fs::read(&snap).unwrap(); + let patch = |offset: usize, with: &[u8]| { + let mut b = original.clone(); + b[offset..offset + with.len()].copy_from_slice(with); + reseal(b) + }; + // header: magic 0..8, version 8..12, flags 12..16, dim 16..20, block 20..24, count 24..32 + let cases: Vec<(&str, Vec<u8>)> = vec![ + ("bad magic", patch(0, b"NOTASNAP")), + ("version 2", patch(8, &2u32.to_le_bytes())), + ("unknown flag", patch(12, &0x8000_0001u32.to_le_bytes())), + ("zero block size", patch(20, &0u32.to_le_bytes())), + ("huge count", patch(24, &u64::MAX.to_le_bytes())), + ( + "count past the data", + patch(24, &(5_000_000u64).to_le_bytes()), + ), + ("count short by one", { + let count = u64::from_le_bytes(original[24..32].try_into().unwrap()) - 1; + patch(24, &count.to_le_bytes()) + }), + ("dim changed", patch(16, &((DIM as u32) + 1).to_le_bytes())), + ]; + for (name, bytes) in cases { + std::fs::write(&snap, bytes).unwrap(); + let (_, report) = load(&path); + assert_eq!(report.source, LoadSource::Json, "{name} accepted"); + assert!(report.snapshot_skipped.is_some(), "{name}"); + } + // a store loaded straight from a damaged file is untouched by the failure + let mut store = EmbeddingStore::new(None); + store + .add(NodeId::new("keep", "me"), "t", Some(vec![1.0; 4])) + .unwrap(); + let mut b = original.clone(); + let mid = b.len() / 2; + b[mid] ^= 0xFF; + std::fs::write(&snap, b).unwrap(); + assert!(matches!( + store.load_snapshot(&snap), + Err(EmbeddingError::Payload(_)) + )); + assert_eq!(store.count(), 1); + assert!(store.get(&NodeId::new("keep", "me")).is_some()); + store.verify_invariants().unwrap(); + assert!(matches!( + store.load_snapshot(path_in(&dir, "missing.snap")), + Err(EmbeddingError::Io(_)) + )); +} + +#[test] +fn a_stale_snapshot_is_refused() { + let dir = tempfile::tempdir().unwrap(); + let (g, path) = saved(&dir, 800); + let snap = snapshot_path_for(&path); + let sidecar = SemanticGraph::embeddings_path_for(&path); + + // 1. another save of the same graph with other vectors rewrote the JSON + // only (as another port or tool might) + let mut other = EmbeddingStore::new(Some(encoder())); + let mut rng = Rng(9); + for e in g.embedding_store().export_embeddings().embeddings { + let v: Vec<f32> = (0..DIM).map(|_| rng.f()).collect(); + other + .add( + NodeId::new(&e.node_type, e.id.as_string()), + &e.text, + Some(v), + ) + .unwrap(); + } + let old: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(&sidecar).unwrap()).unwrap(); + let recorded = old["graph_digest"].as_str().unwrap().to_string(); + other + .save_embeddings_with(&sidecar, Some(&recorded)) + .unwrap(); + let (loaded, report) = load(&path); + assert_eq!(report.source, LoadSource::Json); + assert!( + report + .snapshot_skipped + .as_deref() + .unwrap() + .contains("stale"), + "{:?}", + report.snapshot_skipped + ); + // the new vectors were used, not the snapshot's + let mut expected = EmbeddingStore::new(Some(encoder())); + expected.load_embeddings(&sidecar, true).unwrap(); + assert_same_store(loaded.embedding_store(), &expected); + + // 2. a graph saved without a snapshot removes the old snapshot + g.save_with_snapshot(&path, None).unwrap(); + assert!(snap.is_file()); + assert_eq!(load(&path).1.source, LoadSource::Snapshot); + g.save(&path, None, true).unwrap(); + assert!( + !snap.exists(), + "a plain save must not leave a stale snapshot" + ); + + // 3. a snapshot copied beside another graph's file is for a different graph + let (_g2, path2) = { + let mut g2 = graph(800, 4); + g2.add_node_with_embed("person", "3", vec![("name", "Carol")], None) + .unwrap(); + let p = path_in(&dir, "other.ison"); + g2.save_with_snapshot(&p, None).unwrap(); + (g2, p) + }; + g.save_with_snapshot(&path, None).unwrap(); + std::fs::copy(snapshot_path_for(&path), snapshot_path_for(&path2)).unwrap(); + let (_, report) = load(&path2); + assert_eq!(report.source, LoadSource::Json); + assert!( + report + .snapshot_skipped + .as_deref() + .unwrap() + .contains("different graph"), + "{:?}", + report.snapshot_skipped + ); +} + +#[test] +fn an_edited_graph_still_follows_the_graph_mismatch_policy() { + let dir = tempfile::tempdir().unwrap(); + let (_g, path) = saved(&dir, 400); + let text = std::fs::read_to_string(&path) + .unwrap() + .replace("Bob", "Rob"); + std::fs::write(&path, text).unwrap(); + // default: the JSON digest check fails the load, as before + assert!(matches!( + SemanticGraph::load_with_report(&path, encoder(), &LoadOptions::default()), + Err(EmbeddingError::GraphEmbeddingMismatch(_)) + )); + // warn: loads the JSON, the snapshot (for the old graph) is not used + let opts = LoadOptions { + on_mismatch: GraphMismatch::Warn, + ..LoadOptions::default() + }; + let (g, report) = SemanticGraph::load_with_report(&path, encoder(), &opts).unwrap(); + assert_eq!(report.source, LoadSource::Json); + assert!(report.snapshot_skipped.unwrap().contains("different graph")); + assert!(!g.embedding_store().warnings().is_empty()); +} + +#[test] +fn a_snapshot_that_cannot_be_written_leaves_the_json_pair_and_a_clear_error() { + let dir = tempfile::tempdir().unwrap(); + let g = graph(100, 5); + let path = path_in(&dir, "g.ison"); + // a directory sits where the snapshot goes: the rename must fail + let snap = snapshot_path_for(&path); + std::fs::create_dir(&snap).unwrap(); + std::fs::write(snap.join("inside"), "x").unwrap(); + let err = g.save_with_snapshot(&path, None).unwrap_err(); + assert!(matches!(err, EmbeddingError::Io(_)), "{err}"); + assert!(snap.join("inside").is_file(), "what was there is untouched"); + let leftovers: Vec<_> = std::fs::read_dir(dir.path()) + .unwrap() + .map(|e| e.unwrap().file_name().to_string_lossy().into_owned()) + .filter(|n| n.ends_with(".tmp")) + .collect(); + assert!(leftovers.is_empty(), "{leftovers:?}"); + // the JSON pair is complete and loads + let (loaded, report) = load(&path); + assert_eq!(report.source, LoadSource::Json); + assert_eq!( + loaded.embedding_store().count(), + g.embedding_store().count() + ); +} + +#[test] +fn a_failed_snapshot_write_keeps_the_previous_snapshot() { + let dir = tempfile::tempdir().unwrap(); + let mut store = EmbeddingStore::new(None); + for i in 0..40 { + store + .add( + NodeId::new("doc", i.to_string()), + "t", + Some(vec![i as f32 + 1.0; 6]), + ) + .unwrap(); + } + let snap = path_in(&dir, "s.snap"); + let info = store + .save_snapshot(&snap, &SnapshotIdentity::default()) + .unwrap(); + assert_eq!(info.count, 40); + assert_eq!(info.bytes, std::fs::metadata(&snap).unwrap().len()); + let before = std::fs::read(&snap).unwrap(); + // writing into a directory that does not exist fails before touching it + let bad = dir.path().join("no-such-dir").join("s.snap"); + assert!(store + .save_snapshot(&bad, &SnapshotIdentity::default()) + .is_err()); + assert_eq!(std::fs::read(&snap).unwrap(), before); + // and a rename onto a non-empty directory fails after the temp was written + let blocked = path_in(&dir, "blocked"); + std::fs::create_dir(&blocked).unwrap(); + std::fs::write(blocked.join("inside"), "x").unwrap(); + assert!(store + .save_snapshot(&blocked, &SnapshotIdentity::default()) + .is_err()); + assert!(blocked.join("inside").is_file()); + assert!(std::fs::read_dir(dir.path()).unwrap().all(|e| !e + .unwrap() + .file_name() + .to_string_lossy() + .ends_with(".tmp"))); +} + +#[test] +fn readers_never_see_a_torn_snapshot_while_it_is_replaced() { + let dir = tempfile::tempdir().unwrap(); + let snap = path_in(&dir, "s.snap"); + let build = |scale: f32, n: usize| { + let mut s = EmbeddingStore::new(None); + for i in 0..n { + s.add( + NodeId::new("doc", i.to_string()), + "t", + Some(vec![scale * (i as f32 + 1.0); 16]), + ) + .unwrap(); + } + s + }; + let (a, b) = (build(1.0, 3000), build(-2.0, 2100)); + a.save_snapshot(&snap, &SnapshotIdentity::default()) + .unwrap(); + let stop = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false)); + let writer = { + let (snap, stop) = (snap.clone(), stop.clone()); + std::thread::spawn(move || { + for i in 0..40 { + let s = if i % 2 == 0 { &b } else { &a }; + s.save_snapshot(&snap, &SnapshotIdentity::default()) + .unwrap(); + } + stop.store(true, std::sync::atomic::Ordering::SeqCst); + }) + }; + let readers: Vec<_> = (0..3) + .map(|_| { + let (snap, stop) = (snap.clone(), stop.clone()); + std::thread::spawn(move || { + let (mut ok, mut transient) = (0usize, 0usize); + while !stop.load(std::sync::atomic::Ordering::SeqCst) || ok == 0 { + let mut s = EmbeddingStore::new(None); + match s.load_snapshot(&snap) { + Ok(info) => { + // wholly one of the two stores + let (scale, n) = match info.count { + 3000 => (1.0f32, 3000usize), + 2100 => (-2.0, 2100), + other => panic!("a torn count {other}"), + }; + assert_eq!(s.count(), n); + let v = s + .get_vector(&NodeId::new("doc", (n - 1).to_string())) + .unwrap(); + assert_eq!(v[0], scale * n as f32); + s.verify_invariants().unwrap(); + ok += 1; + } + // a sharing violation while the file is swapped is not a torn read + Err(EmbeddingError::Io(_)) => transient += 1, + Err(e) => panic!("a reader saw an invalid snapshot: {e}"), + } + } + (ok, transient) + }) + }) + .collect(); + writer.join().unwrap(); + for r in readers { + let (ok, transient) = r.join().unwrap(); + eprintln!("reader: {ok} verified loads, {transient} transient open errors"); + assert!(ok > 0); + } +} + +#[test] +fn empty_and_tiny_stores_round_trip_and_options_are_respected() { + let dir = tempfile::tempdir().unwrap(); + // empty embeddings + let mut g = SemanticGraph::new("g", encoder()); + g.add_node_with_embed("person", "1", vec![("name", "Alice")], None) + .unwrap(); + g.embedding_store_mut().clear(); + let path = path_in(&dir, "e.ison"); + g.save_with_snapshot(&path, None).unwrap(); + let (loaded, report) = load(&path); + assert_eq!( + report.source, + LoadSource::Snapshot, + "{:?}", + report.snapshot_skipped + ); + assert_eq!(loaded.embedding_store().count(), 0); + loaded.embedding_store().verify_invariants().unwrap(); + // one row, jsonl format + let mut g = SemanticGraph::new("g", encoder()); + g.embedding_store_mut() + .add(NodeId::new("t", "only"), "one", Some(vec![0.5; DIM])) + .unwrap(); + let path = path_in(&dir, "one.isonl"); + g.save_with_snapshot(&path, None).unwrap(); + let (loaded, report) = load(&path); + assert_eq!( + report.source, + LoadSource::Snapshot, + "{:?}", + report.snapshot_skipped + ); + assert_eq!(loaded.embedding_store().count(), 1); + // embeddings off: nothing restored, no snapshot read + let opts = LoadOptions { + embeddings: false, + ..LoadOptions::default() + }; + let (none, report) = SemanticGraph::load_with_report(&path, encoder(), &opts).unwrap(); + assert_eq!(report.source, LoadSource::NoEmbeddings); + assert_eq!(none.embedding_store().count(), 0); + // plain load() / load_with() use the snapshot too + let plain = SemanticGraph::load(&path, encoder()).unwrap(); + assert_eq!(plain.embedding_store().count(), 1); +} + +#[test] +fn the_sidecar_identity_sees_a_rewritten_file() { + let dir = tempfile::tempdir().unwrap(); + let p = path_in(&dir, "f.json"); + std::fs::write(&p, vec![b'a'; 1_000_000]).unwrap(); + let id = SidecarId::of_file(&p).unwrap(); + assert_eq!(id, SidecarId::of_file(&p).unwrap()); + assert_eq!(id.len, 1_000_000); + let mut changed = vec![b'a'; 1_000_000]; + changed[999_999] = b'b'; // the last byte + std::fs::write(&p, &changed).unwrap(); + assert_ne!(id, SidecarId::of_file(&p).unwrap()); + changed[999_999] = b'a'; + changed[0] = b'b'; // the first byte + std::fs::write(&p, &changed).unwrap(); + assert_ne!(id, SidecarId::of_file(&p).unwrap()); + std::fs::write(&p, vec![b'a'; 1_000_001]).unwrap(); + assert_ne!(id, SidecarId::of_file(&p).unwrap()); + // a tiny file + std::fs::write(&p, b"x").unwrap(); + SidecarId::of_file(&p).unwrap(); +} + +#[test] +fn a_store_keeps_its_scan_precision_and_index_settings_across_a_snapshot_load() { + let dir = tempfile::tempdir().unwrap(); + let g = graph(1500, 6); + let snap = path_in(&dir, "s.snap"); + g.embedding_store() + .save_snapshot(&snap, &SnapshotIdentity::default()) + .unwrap(); + + let mut store = EmbeddingStore::new(None); + store.set_scan_precision(ScanPrecision::int8()).unwrap(); + store + .set_ann(Some(AnnConfig { + nlist: Some(16), + nprobe: Some(4), + min_size: 0, + ..AnnConfig::default() + })) + .unwrap(); + store + .add(NodeId::new("old", "row"), "t", Some(vec![1.0; DIM])) + .unwrap(); + let info = store.load_snapshot(&snap).unwrap(); + assert_eq!(info.count, g.embedding_store().count()); + assert!(store.get(&NodeId::new("old", "row")).is_none(), "replaced"); + assert_eq!(store.scan_precision(), ScanPrecision::int8()); + assert!(store.quant_bytes() > 0); + assert!(store.ann().is_some()); + store.verify_invariants().unwrap(); + // and it searches like a JSON-loaded store with the same settings + let mut json = EmbeddingStore::new(None); + let side = path_in(&dir, "s.json"); + g.embedding_store().save_embeddings(&side).unwrap(); + json.load_embeddings(&side, true).unwrap(); + json.set_scan_precision(ScanPrecision::int8()).unwrap(); + json.set_ann(store.ann().cloned()).unwrap(); + for q in queries(20) { + assert_eq!( + search_bits(&store, &q, 6, None), + search_bits(&json, &q, 6, None) + ); + } +} diff --git a/isongraph-vector-server/Dockerfile b/isongraph-vector-server/Dockerfile new file mode 100644 index 0000000..5b51e70 --- /dev/null +++ b/isongraph-vector-server/Dockerfile @@ -0,0 +1,57 @@ +# Build from the REPOSITORY ROOT (the server depends on the sibling library directory, which is +# not on PyPI until 1.0.0 is published): +# +# docker build -f isongraph-vector-server/Dockerfile -t igv-server . +# docker build -f isongraph-vector-server/Dockerfile --build-arg WITH_SENTENCE_TRANSFORMERS=1 -t igv-server:st . +# +# The default image is small (no torch): use encoder.kind = "http" (an OpenAI/Ollama-style endpoint) +# or "mock" (tests only), or build the :st variant to run a local sentence-transformers model. +# +# Run: +# docker run -v igv-data:/data -p 8080:8080 igv-server serve --config /etc/igv/config.json +# docker run -v igv-data:/data igv-server tenant create acme +# docker run -v igv-data:/data igv-server key create acme # prints the key once +# +# This image speaks plain HTTP. Put a TLS-terminating proxy in front of it; see README "Threat model". + +FROM python:3.12-slim + +ENV PYTHONDONTWRITEBYTECODE=1 \ + PYTHONUNBUFFERED=1 \ + PIP_NO_CACHE_DIR=1 \ + PIP_DISABLE_PIP_VERSION_CHECK=1 + +RUN useradd --system --uid 10001 --no-create-home --shell /usr/sbin/nologin igv \ + && mkdir /data && chown igv:igv /data + +WORKDIR /src +COPY pyproject.toml LICENSE README.md ./ +COPY isongraph_vector-py ./isongraph_vector-py +COPY isongraph-vector-server ./isongraph-vector-server + +ARG WITH_SENTENCE_TRANSFORMERS=0 +# ison-graph (the library's graph engine) and numpy (vectorised search, ANN) are the only runtime +# dependencies; the library and the server are installed --no-deps so the library's hard +# sentence-transformers requirement is only pulled in when asked for. +RUN pip install ison-graph numpy \ + && pip install --no-deps . \ + && pip install --no-deps ./isongraph-vector-server \ + && if [ "$WITH_SENTENCE_TRANSFORMERS" = "1" ]; then pip install sentence-transformers; fi \ + && rm -rf /src + +USER igv +WORKDIR /data +VOLUME /data +EXPOSE 8080 + +ENV IGV_DATA_DIR=/data \ + IGV_HOST=0.0.0.0 \ + IGV_PORT=8080 + +HEALTHCHECK --interval=15s --timeout=4s --start-period=20s --retries=3 \ + CMD python -c "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8080/healthz', timeout=3).status == 200 else 1)" + +# SIGTERM (docker stop) triggers a graceful shutdown: stop accepting, drain, final snapshot. +STOPSIGNAL SIGTERM +ENTRYPOINT ["igv-server"] +CMD ["serve"] diff --git a/isongraph-vector-server/Dockerfile.dockerignore b/isongraph-vector-server/Dockerfile.dockerignore new file mode 100644 index 0000000..d6341d9 --- /dev/null +++ b/isongraph-vector-server/Dockerfile.dockerignore @@ -0,0 +1,11 @@ +* +!pyproject.toml +!LICENSE +!README.md +!isongraph_vector-py +!isongraph-vector-server +**/__pycache__ +**/.pytest_cache +**/*.pyc +isongraph-vector-server/tests +isongraph-vector-server/scripts/results diff --git a/isongraph-vector-server/README.md b/isongraph-vector-server/README.md new file mode 100644 index 0000000..a2e5d18 --- /dev/null +++ b/isongraph-vector-server/README.md @@ -0,0 +1,360 @@ +# isongraph-vector-server + +A multi-tenant HTTP service, a tenant-sharding router and snapshot read replicas around +[`isongraph-vector`](../README.md). The library is deliberately an embedded component - +its roadmap lists "multi-tenant access control and a service layer" as out of scope - so this +is a **separate package**: the library is used unmodified, as a dependency. + +**Status: 0.1.0, alpha.** It has real tests (see [Testing](#testing)), but it is new code and has not +been run in production. Read [What is NOT provided](#what-is-not-provided) before relying on it. + +The only runtime dependency is the library itself (and through it `ison-graph`). The HTTP server, +router, replication and CLI are standard library only. `numpy` is optional (vectorised search and the +opt-in IVF index come from the library). + +## What it is + +``` + clients (HTTPS, via your TLS proxy) + | + +---------v---------+ + | igv-server router | stateless; tenant -> shard by rendezvous hash + +----+---------+----+ (reads may go to a replica; writes never retried) + | | + +------------v--+ +--v------------+ + | shard s0 | | shard s1 | one primary per shard (no failover) + | primary | | primary | + | tenants A,C.. | | tenants B,D.. | each tenant lives on exactly one shard + +------+--------+ +---------------+ + | pull snapshots (generation + sha256 + graph digest) + +------v--------+ + | read replica | serves a *saved generation*, read-only, bounded staleness + +---------------+ +``` + +* **Tenancy.** Every request is bound to a tenant by its API key. The tenant is derived from the + verified key and from nothing else (no header, query parameter, body field or path segment can + select or override it). Each tenant has its own graphs, quotas, rate limit and directory. +* **API.** JSON over HTTP: create/delete graphs; upsert/patch/delete nodes and edges; semantic search, + `multi_hop`, `path`, `subgraph`, `similar`; declarative filters and edge weights; stale-embedding + detection and re-embedding; usage, health and metrics. See [API](#api). +* **Durability.** Atomic, checksummed, digest-verified snapshot *generations* with automatic fallback; + crash-tested with real `SIGKILL`. +* **Scale-out.** Hash-shard tenants across nodes behind a stateless router; add snapshot-based read + replicas. See [Scale-out and consistency](#scale-out-and-consistency) for exactly what that does and + does not give you. + +## Quick start + +```bash +pip install ./isongraph-vector-server # from a checkout; brings the library too +export IGV_DATA_DIR=./igv-data + +igv-server tenant create acme --quota max_nodes=50000 +igv-server key create acme --role rw --label ci # prints the key ONCE, stores only a salted hash +# igv_acme.3f9a.... + +cat > config.json <<'EOF' +{ "data_dir": "./igv-data", "port": 8080, + "encoder": {"kind": "mock", "dimension": 64} } # "mock" is plumbing only; see config.example.json +EOF +igv-server serve --config config.json & + +KEY=igv_acme.3f9a... +curl -s -X POST localhost:8080/v1/graphs -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \ + -d '{"name":"kb"}' +curl -s -X POST localhost:8080/v1/graphs/kb/nodes -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \ + -d '{"nodes":[{"type":"person","id":1,"properties":{"name":"Alice","description":"builds payment APIs"}}]}' +curl -s -X POST localhost:8080/v1/graphs/kb/search -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \ + -d '{"query":"who builds APIs?","top_k":3}' +``` + +Docker: `docker build -f isongraph-vector-server/Dockerfile -t igv-server .` from the repository root +(see the comments in the Dockerfile; the image has no torch unless you pass +`--build-arg WITH_SENTENCE_TRANSFORMERS=1`). + +## API + +All paths are under `/v1` and need `Authorization: Bearer <key>` unless noted. Bodies are JSON +(`Content-Type: application/json`), validated against an explicit schema (`schemas.py`): unknown fields, +duplicate keys, `NaN`/`Infinity`, nested property values and invalid UTF-8 are rejected. + +| Method and path | Purpose | +| --- | --- | +| `GET /healthz`, `GET /readyz` (no key) | liveness; readiness (`503` until startup verification finished, and on a replica past its staleness bound) | +| `GET /metrics` (operator token) | Prometheus text, aggregate only. Exists only if `IGV_METRICS_TOKEN` is set | +| `GET /v1/usage` | this tenant's quotas and usage | +| `GET /v1/graphs`, `POST /v1/graphs` | list (paginated) / create `{name, embedding:{mode,dimension}, embed_fields, ann}` | +| `GET /v1/graphs/{g}`, `DELETE /v1/graphs/{g}` | info (counts, generation, `snapshot_digest`) / delete | +| `POST /v1/graphs/{g}/save` | force a snapshot now | +| `GET /v1/graphs/{g}/nodes` | list nodes (`type`, `limit`, `cursor`) | +| `POST /v1/graphs/{g}/nodes` | batch upsert (create or **replace**); `nodes/delete` for batch delete | +| `GET/PUT/PATCH/DELETE /v1/graphs/{g}/nodes/{type}/{id}` | one node; `PATCH` merges (`remove:[...]` deletes properties) | +| `GET /v1/graphs/{g}/edges`, `POST .../edges`, `POST .../edges/delete` | list / batch upsert / batch delete | +| `POST /v1/graphs/{g}/search` | semantic search: `query` (text) or `vector`, `top_k`, `threshold`, `node_type`, `filter`, `exact` | +| `POST /v1/graphs/{g}/multi_hop` | seed by similarity, traverse with decay: `rel_type`, `max_hops`, `direction`, `decay`, `blend`, `weight_field`, `filter`, `strict_filter` | +| `POST /v1/graphs/{g}/path`, `.../subgraph`, `.../similar` | `semantic_path`, `semantic_subgraph` (returned as nodes+edges JSON), `similar_to_node` | +| `GET /v1/graphs/{g}/stale`, `POST .../reembed` | model lifecycle (server-mode graphs) | + +* **Embedding modes.** `server` (default): the server embeds text with its configured encoder (node + `properties` named in `embed_fields`, or an explicit `embed_text`). `client`: you create the graph with a + `dimension` and send `vector`s; text queries are refused. A graph never mixes the two, so a stored vector + always has a known origin. +* **Filters** are data, never code: `{"types":[...], "where":[{"field":"age","op":"gte","value":30}]}` + with `eq ne lt lte gt gte in nin exists prefix contains`, ANDed, at most 16 conditions. +* **Ids.** Integer, or a string matching `[A-Za-z0-9_][A-Za-z0-9_.@+=~-]{0,127}`. A string that spells a + canonical integer *is* that integer (`"5"` and `5` are one node - the library cannot hold both). +* **Pagination.** `limit` (default 100, max `max_page`) and an opaque `cursor`; responses carry + `next_cursor`. Node cursors are keyset positions; edge and stale cursors are offsets (best effort if the + graph changes between pages). +* **Errors** always look like `{"error": {"code", "message", "request_id", "details"?}}`. Stable codes + include `validation_failed`, `unauthorized`, `forbidden`, `read_only_key`, `read_only_replica`, + `not_found`, `already_exists`, `conflict`, `id_conflict`, `dimension_mismatch`, `model_mismatch`, + `quota_exceeded`, `rate_limited`, `overloaded`, `tenant_overloaded`, `busy`, `deadline_exceeded`, + `graph_quarantined`, `replica_stale`, `encoder_failed`, `internal_error`. Internal errors are logged + server-side with the same `request_id`; clients never see exception text, paths or stack traces. +* **Null and absent are the same.** The ISON text format is columnar and cannot tell a missing property + from a `null` one (the library's own round trip turns one into the other), so the service never stores or + returns null properties; sending `null` in a `PATCH` removes the property. + +Not supported on purpose: undirected graphs (the file format cannot record directedness; use +`"direction": "both"`), property values that are lists or objects, and ids/types/property names outside the +whitelists above (they are exactly what survives the ISON round trip). + +## Threat model + +Assume: a hostile or buggy tenant holding a valid key (including a read-only one); anonymous network +callers; a tenant trying to infer other tenants' existence or data. Assume the operator and the host are +trusted, and that TLS is terminated in front of this process. + +| Threat | Mitigation | Evidence | +| --- | --- | --- | +| Tenant B reads/writes/enumerates tenant A | The tenant is derived from the verified key only; a handler receives only that tenant's runtime; there is no global name-keyed map | `tests/test_isolation.py` - colliding ids and graph names, every endpoint incl. batch, search, subgraph, cursors; 3 deliberate code mutations each make it fail | +| Inference ("does graph X exist for someone else?") | A name that is not yours is simply absent: identical status, code and message to a name that exists nowhere; quota/usage errors report only your own numbers; metrics are aggregate with no tenant labels | same file | +| Forged / mixed credentials | Format `igv_<tenant>.<key-id>.<256-bit secret>`; the tenant in the key is only a lookup hint, authority is the HMAC check against that tenant's record; duplicate `Authorization` headers rejected | forgery, truncation, case, whitespace, duplicate-header and cross-tenant tests | +| Key theft from disk | Only `HMAC-SHA256(salt, secret)` and the salt are stored; `compare_digest`; unknown tenant/key id/revoked key still do one dummy HMAC; `tenant.json` is `0600` on POSIX | tests assert the secret is not in the file and is never logged | +| Brute force | Failed-auth throttle per peer address (`429`); 256-bit secrets | throttle test | +| Header/query/body tricks to select a tenant | None of them is read. Unknown query parameters and body fields are errors | `X-Tenant*`, `Host`, `Forwarded`, cookies, `?tenant=`, `?api_key=` tests | +| Path traversal / aliasing via names | Names are whitelist-matched (lowercase `[a-z0-9_-]`, so case-insensitive file systems cannot alias two names); the directory is `g-<name>` derived from the *validated* name and re-checked to be inside the base | 40 crafted names x GET/DELETE/POST/create, file tree compared before/after; traversal in replication routes | +| Request smuggling / framing | `Content-Length` only; chunked, duplicate or malformed lengths refused | `test_limits.py` raw-socket tests | +| Resource exhaustion | Body/header/URI caps; socket timeouts; bounded worker pool and accept queue (`503` when full); global and per-tenant concurrency gates; per-tenant token bucket; node/edge/graph/dimension/batch quotas; `top_k`/`max_hops` caps; response size cap; request deadlines; lock-wait timeouts | `test_limits.py` | +| Stack traces / secrets in responses and logs | Generic `internal_error` + request id; encoder errors are replaced by a generic message; log records pass a redaction filter and never include headers or bodies | tests | +| Corrupted or tampered snapshot | sha256 + size per file, library digest check, manifest-digest cross-check; fallback to the last good generation or quarantine | `test_persistence.py`, `test_replication.py` | +| Malicious replica source | A replica verifies everything before installing and refuses (keeps serving what it had). **It trusts the primary and the replication token.** | 6 tampering variants | + +### What is NOT provided + +Say these out loud before deploying: + +* **No TLS.** The server speaks plain HTTP. Terminate TLS in a reverse proxy. API keys travel in a header. +* **Not hardened for the open internet.** It is a standard-library `socketserver` with sensible limits, + not nginx. Slow-connection floods beyond the per-socket timeouts and the pool bound, HTTP/2, connection + limits per address, and request-header memory (up to ~6 MB per connection in the worst case) are the + proxy's job. +* **No remote admin API.** Tenants and keys are managed by the CLI on the data directory. Anyone with + filesystem access to the data directory can mint keys. +* **No per-user permissions inside a tenant** beyond the `rw`/`ro` key role. No audit log beyond the + request log. No key expiry (revoke manually). No encryption at rest. +* **Rate limiting is per tenant, in memory, per node.** Behind a router with several shards each tenant is + on one shard, so its limit is exact; it is *not* shared across replicas. +* **A running computation cannot be interrupted.** Deadlines are checked between phases, and lock waits + and encoder calls are bounded; a pathological traversal that is already running holds its worker until it + finishes. It is bounded by the quotas (graph size, `max_hops`, `top_k`) and cannot stall other graphs. +* **Tenant existence is not a secret** (it is a prefix of the key, and registry lookups are slightly + faster for known tenants). Key *validity* is. +* **The embedding endpoint sees every text.** With `encoder.kind = "http"` all node text and text queries + go to that endpoint; with `mock` the "embeddings" are noise. +* **The metrics and replication tokens are single shared secrets** read from the environment + (`IGV_METRICS_TOKEN`, `IGV_REPLICATION_TOKEN`, >= 24 characters). The replication token reads every + tenant's data and key hashes: treat it like a database password. + +## Persistence and durability + +Per graph, under `<data>/tenants/<tenant>/graphs/g-<name>/`: + +``` +meta.json immutable configuration +CURRENT the generation to serve +gen-000007/ graph.isong + graph.isong.embeddings.json + MANIFEST.json (sha256, sizes, graph_digest) +``` + +A save builds the whole pair in a temp directory, re-parses it and checks it equals memory, fsyncs, +renames the directory into place, then atomically flips `CURRENT`. The library alone writes two separate +atomic files, so a crash between them leaves a mismatched pair (it detects that on load, but has nothing to +fall back to); generations fix that. On start every graph is verified (checksums, the library's +sidecar-vs-graph digest check, manifest digest). If `CURRENT`'s generation fails, the newest older one that +passes is served and the graph is reported `degraded`; if none passes the graph is **quarantined** +(`503 graph_quarantined`, files left untouched, other graphs unaffected). `strict_startup` makes any such +failure abort startup instead. `igv-server verify` does the same check offline. + +* `durability = "snapshot"` (default): a write is acknowledged after it is applied in memory. A snapshot is + written every `autosave_interval_seconds` (30 s), on `POST .../save`, and at graceful shutdown. **A crash + (not a graceful stop) loses up to one autosave interval of acknowledged writes.** There is no write-ahead log. +* `durability = "sync"`: every mutating request writes a snapshot before it is acknowledged. No acknowledged + write is lost (tested with `SIGKILL`), but each write costs O(graph size). Use it for small graphs. +* Snapshots hold the graph's read lock only while capturing the image (writers to *that graph* wait; readers + do not): measured 0.20 s for 20,000 nodes with 64-dimensional vectors; the re-parse, checksums and fsync + (a further 0.78 s there) run without any lock. Loading plus full verification of that graph takes 0.84 s. +* A data directory belongs to one process (an OS file lock enforces it). + +## Scale-out and consistency + +**Router** (`igv-server router --config ...`). Stateless; no credentials, no data. It reads the tenant id out +of the key, maps it to a shard with rendezvous hashing (or `router.pins`), and forwards the request unchanged. +The backend authenticates, so the router cannot widen access: a key naming tenant A but made for B is +forwarded to A's shard and fails there. Only `/v1/*` is proxied; `/metrics` and `/_repl` never are. +Reads (`GET`, and the five search `POST`s) may go to a replica if `read_preference = replica_preferred` +(fall back to the primary if the replica errors or reports `replica_stale`); a client can force the primary with +`X-Igv-Consistency: primary`. Writes always go to the primary and are **never retried**. + +**Replicas** (`role = "replica"`, `replica.source = "dir:/path"` or `http://primary:8080` with +`IGV_REPLICATION_TOKEN`). A replica pulls saved generations and installs them only after verifying every +checksum and digest (see `replication.py`). Honest consistency statement: + +1. A replica serves what the primary has *saved*. A read can be stale by up to + `primary autosave interval + replica sync interval + sync time` (default 30 + 15 s plus transfer). +2. If the last fully successful sync is older than `replica.max_staleness_seconds` the replica answers + `503 replica_stale` and reports not-ready, rather than serving unboundedly old data. Staleness survives a + replica restart. +3. Graphs are synced independently: two graphs can reflect different primary moments. There is no + cross-graph snapshot. +4. Read-your-writes is not provided on a replica. Revoking a key or disabling a tenant reaches replicas on + the next sync, so a revoked key can keep working on a replica for up to the sync interval. +5. A snapshot that fails verification is refused and counted (`igv_replica_refused_total`); the replica keeps + serving the previous generation, and the cycle is not counted as fresh, so repeated refusals eventually + trip the staleness bound. + +**Not provided:** automatic failover or promotion (a dead primary means its tenants get `502`/`503`); online +resharding or tenant migration (changing the shard list remaps tenants, but moving their data is manual: copy +the tenant directory to the new owner, or pin the tenant with `router.pins` meanwhile - `igv-server shard-of +<tenant> --config ...` tells you the owner); cross-tenant or cross-shard queries; streaming replication or a +write-ahead log; multi-primary writes. Capacity scales by *adding tenants' homes*, not by splitting one tenant +across nodes - one tenant's graphs live on one node and must fit its memory. + +## Configuration + +A JSON file (`--config` or `IGV_CONFIG`), see `config.example.json`, with environment overrides for the +common scalars (`IGV_DATA_DIR`, `IGV_HOST`, `IGV_PORT`, `IGV_ROLE`, `IGV_LOG_LEVEL`, `IGV_DURABILITY`, +`IGV_AUTOSAVE_INTERVAL_SECONDS`, `IGV_MAX_WORKERS`, `IGV_MAX_BODY_BYTES`, `IGV_STRICT_STARTUP`, +`IGV_REPLICA_SOURCE`). Secrets are environment-only and are *refused* in the file: `IGV_METRICS_TOKEN`, +`IGV_REPLICATION_TOKEN`, and the embedding endpoint's key (named by `encoder.api_key_env`). Unknown keys are +errors. Per-tenant quota overrides live in the tenant's `tenant.json` (`igv-server tenant create/set-quota +--quota name=value`) and take effect within `registry_recheck_seconds` (1 s). + +## Operations + +* **Logs.** JSON lines on stderr: `ts, level, msg`, and for requests `method, route (template), status, ms, + tenant, request_id`. No headers, bodies, query text or ids. A redaction filter removes anything shaped like + a key as a backstop. +* **Metrics.** `GET /metrics` (token): request counters by route and status class, a latency histogram per + route, in-flight, auth failures/throttles, rate-limited, snapshots/failures, quarantined/degraded graphs, + replica staleness/refusals. Aggregate only. +* **Shutdown.** `SIGTERM`/`SIGINT` (`CTRL_BREAK` on Windows): stop accepting, finish in-flight requests, + close idle connections, write a final snapshot of every dirty graph, exit 0. +* **Backups.** Generations are immutable directories; copy `CURRENT` and the generation it names (or the + whole data directory while the node is stopped) and `igv-server verify` the copy. +* **CLI.** `serve`, `router`, `tenant create|list|show|disable|enable|set-quota`, `key create|revoke`, + `shard-of`, `verify`, `replica-sync` (one cycle, cron-friendly). + +## Testing + +```bash +cd isongraph-vector-server +pip install pytest numpy +python -m pytest tests # ~1 minute; no network, no model download (MockEncoder) +``` + +147 tests (146 pass, 1 skipped on Windows: the POSIX file-mode check), about 70 s of test time on a quiet machine: + +| file | tests | covers | +| --- | ---: | --- | +| `test_api_basic.py` | 16 | every endpoint's happy path, validation, error model, pagination, id rules | +| `test_isolation.py` | 30 | tenant isolation, credentials, crafted names, replication/metrics endpoints, log secrecy | +| `test_limits.py` | 23 | quotas (incl. a concurrent quota race), protocol limits over raw sockets, load shedding, deadlines, lock timeouts, prompt graceful stop, pipelining, byte-drip cut-off, `RWLock`/token bucket | +| `test_persistence.py` | 21 | snapshots, corruption fallback, quarantine, restart, `sync` durability, round-trip fuzzing, graceful shutdown | +| `test_crash.py` | 10 | real `SIGKILL` at five points inside a save, 10 random kills, `SIGKILL` of a whole server in `sync` mode, signal-driven shutdown | +| `test_replication.py` | 19 | dir and HTTP sources, incremental sync, six tampering variants refused, staleness bound, revocation, deletions | +| `test_router.py` | 13 | hashing properties, routing, forged keys, 502 without failover, writes never retried, replica reads and fallback | +| `test_lifecycle.py` | 15 | model change / `reembed`, traversal options, filters, ANN opt-in, CLI, config, logs, generic 500s | + +See [Tested versus only claimed](#tested-versus-only-claimed) for exactly which properties are covered. + +## Load test + +`scripts/load_test.py` creates N tenants with M vectors each, starts the server (and optionally K shards and +a router) as separate processes, and drives closed-loop search load, reporting throughput and latency +percentiles as a table and JSON. Results from one development machine (Windows 11, 32 logical cores, +Python 3.12, MockEncoder, client-mode graphs, `top_k = 10`, vector queries) are in `scripts/results/`: + +Single node, search throughput (`scripts/results/*.json`; "gen" = load-generator processes): + +| setup | tenants x vectors | clients | req/s | p50 ms | p95 ms | p99 ms | +| --- | --- | ---: | ---: | ---: | ---: | ---: | +| 1 node, 1 generator | 8 x 2,000 (dim 64) | 1 | 1,609 | 0.57 | 0.75 | 1.2 | +| 1 node, 1 generator | 8 x 2,000 | 4 | 1,949 | 1.9 | 3.3 | 5.0 | +| 1 node, 1 generator | 8 x 2,000 | 16 | 2,000 | 6.9 | 16.4 | 22.1 | +| 1 node, 1 generator | 8 x 2,000 | 64 | 1,624 | 31.8 | 99.2 | 125.2 | +| 1 node, 1 generator | 32 x 2,000 | 16 | 1,895 | 7.7 | 15.5 | 20.7 | +| 1 node, 1 generator | 4 x 20,000 | 16 | 1,472 | 9.6 | 23.2 | 28.3 | +| 1 node, 1 generator, text queries | 8 x 2,000 | 16 | 1,945 | 6.5 | 19.3 | 24.9 | +| server inside the generator's process | 8 x 2,000 | 16 | 1,397 | 10.1 | 22.7 | 29.8 | + +Scale-out (16 tenants x 2,000 vectors, 32 clients over 8 generator processes, nodes are separate processes): + +| topology | req/s | p50 ms | p95 ms | p99 ms | vs. 1 node | +| --- | ---: | ---: | ---: | ---: | ---: | +| 1 node, direct | 2,102 | 12.5 | 34.6 | 49.5 | 1.00x | +| router -> 1 node | 1,818 | 14.9 | 38.1 | 57.9 | 0.86x (router overhead) | +| router -> 2 nodes | 2,977 | 8.4 | 28.8 | 42.1 | 1.42x | +| router -> 4 nodes | 3,333 | 9.2 | 15.4 | 19.1 | 1.59x | + +Reading these honestly: a node tops out around 2,000 small searches per second (one Python process, GIL). +Adding shards helps but sub-linearly here, because the router is itself a single Python process, the +generators and four servers shared one 32-thread desktop, and every request pays an extra hop. If the router +becomes the limit, run several routers (they are stateless). Zero errors in every run. + +These are *this machine, this workload* numbers for a GIL-bound pure-Python server; they say nothing about +other hardware, and the load generator shares the machine with the server. Use the script on your own. + +## Known limits + +* Throughput per node is bounded by the GIL (a few thousand small searches per second); scale by adding + shard processes, not threads. +* All of a node's graphs stay in memory (loaded at startup, never evicted). Memory is bounded only by the + quotas you set. +* A snapshot rewrites the whole graph; very large graphs make `sync` durability and snapshot lock time + expensive. The library targets small-to-medium graphs, and so does this. +* `reembed` holds the graph's write lock while it calls the encoder. +* The ISON file format forces: directed graphs only; scalar property values; no null/absent distinction; + no unicode whitespace at value edges (rejected up front). +* The standard-library server is single-process, HTTP/1.1, no TLS. + +## Tested versus only claimed + +**Exercised by a test that fails if the property breaks** (three deliberate mutations of the source - a +shared tenant runtime, a loosened graph-name regex, a key lookup that ignores the tenant - each made the +isolation suite fail, then were reverted): + +* tenant isolation on every endpoint including batch, search, subgraph, cursors, with identical ids and graph + names in several tenants, concurrently; uniform not-found for foreign and absent names; own-only usage in + quota errors, listings and usage; read-only keys; revocation and disabling +* key forgery / truncation / case / duplicate-header variants; key never in file, response or log +* 40+ crafted names and ids against path traversal (file tree compared before and after); traversal in the + replication file routes; the replication and metrics tokens are not tenant keys and vice versa +* body/header/URI limits, chunked and conflicting `Content-Length`, slow clients, pool exhaustion, per-tenant + and global load shedding, deadlines, lock timeouts, exact and race-free quotas +* atomic snapshots under `SIGKILL` at each step, random kills, `SIGKILL` of the real server with `sync` + durability, corruption fallback, quarantine isolation, round-trip fidelity of everything validation admits +* replica refusal on six distinct tampering patterns including a manifest whose checksums were re-written to + match; staleness cut-off; deletions; "unreadable is not deleted" +* router: forged tenant prefix, no failover, no write retry, replica fallback, the hash's balance and + minimal-disruption properties + +**Claimed but not tested here:** behaviour under TLS or behind a particular proxy; HTTP encoder end to end (the +library's `HttpEncoder` is used as-is; this repo only tests encoder *failures*); a real +sentence-transformers model; POSIX-only paths (`fsync` of directories, `0600` mode - the mode check is +skipped on Windows, SIGTERM handling is covered via `CTRL_BREAK` on Windows); the Dockerfile (no Docker +daemon was available, so it was not built; the two `pip install --no-deps` steps it runs were checked +separately); timing-side-channel resistance (the code takes the same path for unknown tenant, unknown key id +and wrong secret and the responses are byte-identical, but no timing measurement was made); memory limits; +long soak runs; multi-day replica drift. diff --git a/isongraph-vector-server/config.example.json b/isongraph-vector-server/config.example.json new file mode 100644 index 0000000..9bc29ec --- /dev/null +++ b/isongraph-vector-server/config.example.json @@ -0,0 +1,57 @@ +{ + "role": "primary", + "data_dir": "/data", + "host": "0.0.0.0", + "port": 8080, + "log_level": "INFO", + + "max_workers": 32, + "accept_queue": 64, + "socket_timeout_seconds": 10, + "max_body_bytes": 1048576, + "max_concurrent_requests": 32, + "queue_wait_seconds": 0.5, + "request_deadline_seconds": 30, + "lock_timeout_seconds": 10, + "failed_auth_per_minute": 30, + "trusted_proxies": [], + + "durability": "snapshot", + "autosave_interval_seconds": 30, + "keep_generations": 3, + "strict_startup": false, + + "quotas": { + "max_graphs": 20, + "max_nodes": 100000, + "max_edges": 500000, + "max_dimension": 1536, + "max_batch_items": 500, + "max_top_k": 100, + "max_hops": 5, + "rate_per_second": 50, + "rate_burst": 100, + "max_inflight": 8 + }, + + "encoder": { + "kind": "http", + "url": "http://embeddings.internal:8000/v1/embeddings", + "model": "my-embedding-model", + "api": "openai", + "api_key_env": "EMBEDDING_API_KEY", + "timeout": 30, + "max_concurrency": 4 + }, + + "replica": { + "source": "", + "sync_interval_seconds": 15, + "max_staleness_seconds": 120 + }, + + "router": { + "shards": [], + "read_preference": "primary" + } +} diff --git a/isongraph-vector-server/isongraph_vector_server/__init__.py b/isongraph-vector-server/isongraph_vector_server/__init__.py new file mode 100644 index 0000000..85dade7 --- /dev/null +++ b/isongraph-vector-server/isongraph_vector_server/__init__.py @@ -0,0 +1,6 @@ +"""isongraph-vector-server: a multi-tenant HTTP service around isongraph-vector. + +See README.md for what it provides and, just as important, what it does not. +""" + +__version__ = "0.1.0" diff --git a/isongraph-vector-server/isongraph_vector_server/__main__.py b/isongraph-vector-server/isongraph_vector_server/__main__.py new file mode 100644 index 0000000..dd8a8c9 --- /dev/null +++ b/isongraph-vector-server/isongraph_vector_server/__main__.py @@ -0,0 +1,5 @@ +import sys + +from .cli import main + +sys.exit(main()) diff --git a/isongraph-vector-server/isongraph_vector_server/auth.py b/isongraph-vector-server/isongraph_vector_server/auth.py new file mode 100644 index 0000000..a71b41a --- /dev/null +++ b/isongraph-vector-server/isongraph_vector_server/auth.py @@ -0,0 +1,394 @@ +"""API keys, tenant records and the registry that authenticates requests. + +Key format:: + + igv_<tenant-id>.<key-id>.<secret> + lowercase 16 hex 43 chars urlsafe base64 (256 random bits) + +* The server stores only ``HMAC-SHA256(salt, secret)`` plus the salt, never the + secret. A plain salted fast hash is the right tool here (the same reasoning + as for GitHub/Stripe style tokens): the secret is 256 random bits, so there + is nothing for a slow password KDF to protect against, and a per-request KDF + would cap throughput at a few dozen requests per second per core. +* Verification uses :func:`hmac.compare_digest`. An unknown tenant, unknown + key id or revoked key still runs one dummy HMAC + compare, so the three + failure paths take about the same time as a wrong secret. +* The tenant id inside the key is *only a lookup hint* (it lets the stateless + router pick a shard without a registry). Authority comes from the hash + check against that tenant's own record: a key naming tenant B but made for + tenant A fails here. +* A key whose tenant is disabled, or that is revoked, stops working within + ``recheck_seconds`` (the registry re-stats ``tenant.json`` that often). +""" + +from __future__ import annotations + +import hashlib +import hmac +import json +import os +import re +import secrets +import threading +import time +import uuid +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Dict, List, Optional, Tuple + +from . import names +from .config import ConfigError, Quotas + +KEY_RE = re.compile(r"^igv_([a-z0-9][a-z0-9-]{0,30}[a-z0-9])\.([0-9a-f]{16})\.([A-Za-z0-9_-]{43})$") +TENANT_FORMAT = "igv-tenant" +TENANT_FORMAT_VERSION = 1 +ROLES = ("rw", "ro") + +_DUMMY_SALT = b"\x00" * 16 +_DUMMY_HASH = hmac.new(_DUMMY_SALT, b"dummy", hashlib.sha256).hexdigest() + + +def _hash_secret(salt: bytes, secret: str) -> str: + return hmac.new(salt, secret.encode("ascii", "strict"), hashlib.sha256).hexdigest() + + +def parse_key(token: str) -> Optional[Tuple[str, str, str]]: + """Split a presented key into (tenant_id, key_id, secret), or None.""" + if not isinstance(token, str) or len(token) > 200: + return None + m = KEY_RE.match(token) + return (m.group(1), m.group(2), m.group(3)) if m else None + + +def tenant_id_from_key(token: str) -> Optional[str]: + parsed = parse_key(token) + return parsed[0] if parsed else None + + +def new_key_record(role: str = "rw", label: str = "") -> Tuple[Dict[str, Any], str, str]: + """Return (record_without_tenant, key_id, secret). The caller assembles the token.""" + if role not in ROLES: + raise ValueError("role must be 'rw' or 'ro'") + key_id = secrets.token_hex(8) + secret = secrets.token_urlsafe(32) + salt = secrets.token_bytes(16) + record = { + "key_id": key_id, + "label": str(label)[:64], + "role": role, + "algo": "hmac-sha256", + "salt": salt.hex(), + "hash": _hash_secret(salt, secret), + "created_at": time.time(), + "revoked_at": None, + } + return record, key_id, secret + + +def make_token(tenant_id: str, key_id: str, secret: str) -> str: + return f"igv_{tenant_id}.{key_id}.{secret}" + + +@dataclass(frozen=True) +class Principal: + """The authenticated caller. Derived only from a verified key.""" + tenant_id: str + key_id: str + role: str + quotas: Quotas + + +@dataclass +class TenantRecord: + tenant_id: str + disabled: bool + quota_overrides: Dict[str, Any] + keys: Dict[str, Dict[str, Any]] + raw: Dict[str, Any] = field(repr=False, default_factory=dict) + stamp: Tuple[int, int] = (0, 0) + + +def _atomic_write_json(path: Path, obj: Any, mode: int = 0o600) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_name(f".{path.name}.{os.getpid()}.{uuid.uuid4().hex[:8]}.tmp") + data = json.dumps(obj, indent=2, sort_keys=True).encode("utf-8") + try: + fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL, mode) + with os.fdopen(fd, "wb") as handle: + handle.write(data) + handle.flush() + os.fsync(handle.fileno()) + os.replace(tmp, path) + except BaseException: + try: + tmp.unlink() + except OSError: + pass + raise + + +def parse_tenant_file(data: Any, expect_id: str) -> TenantRecord: + """Validate a tenant.json document. Raises ValueError on any problem.""" + if not isinstance(data, dict) or data.get("format") != TENANT_FORMAT: + raise ValueError("not a tenant file") + if data.get("version") != TENANT_FORMAT_VERSION: + raise ValueError("unsupported tenant file version") + tid = data.get("tenant_id") + if tid != expect_id or not names.is_tenant_id(tid): + raise ValueError("tenant id does not match its directory") + keys: Dict[str, Dict[str, Any]] = {} + for rec in data.get("keys", []): + if not isinstance(rec, dict): + raise ValueError("bad key record") + kid = rec.get("key_id") + if not isinstance(kid, str) or not re.fullmatch(r"[0-9a-f]{16}", kid): + raise ValueError("bad key id") + if rec.get("role") not in ROLES or rec.get("algo") != "hmac-sha256": + raise ValueError("bad key role/algo") + bytes.fromhex(rec["salt"]) + int(rec["hash"], 16) + keys[kid] = rec + overrides = data.get("quotas", {}) + if not isinstance(overrides, dict): + raise ValueError("quotas must be an object") + return TenantRecord( + tenant_id=tid, + disabled=bool(data.get("disabled", False)), + quota_overrides=overrides, + keys=keys, + raw=data, + ) + + +class TenantRegistry: + """Read side: authenticate keys against ``<data_dir>/tenants/*/tenant.json``. + + The CLI (another process) is the only writer. The registry notices + changes by stat'ing each tenant file at most every ``recheck_seconds`` + and re-scans the directory (at most that often) when it sees an unknown + tenant id, so a freshly created tenant works without a restart while a + flood of bogus tenant ids cannot turn into a flood of directory scans. + """ + + def __init__(self, data_dir: str, default_quotas: Quotas, recheck_seconds: float = 1.0, + clock=time.monotonic): + self.data_dir = Path(data_dir) + self.default_quotas = default_quotas + self.recheck_seconds = recheck_seconds + self._clock = clock + self._lock = threading.Lock() + self._records: Dict[str, TenantRecord] = {} + self._checked_at: Dict[str, float] = {} + self._last_scan = -1e9 + self.load_errors: Dict[str, str] = {} + self.rescan(force=True) + + # -- loading ------------------------------------------------------------ + + def _tenant_file(self, tid: str) -> Path: + return names.tenant_dir(self.data_dir, tid) / "tenant.json" + + def _load_one(self, tid: str) -> Optional[TenantRecord]: + path = self._tenant_file(tid) + try: + st = path.stat() + stamp = (st.st_mtime_ns, st.st_size) + with open(path, "r", encoding="utf-8") as handle: + data = json.load(handle) + rec = parse_tenant_file(data, tid) + rec.stamp = stamp + self.load_errors.pop(tid, None) + return rec + except FileNotFoundError: + return None + except (OSError, ValueError, KeyError, TypeError) as exc: + # Fail closed: a corrupt tenant file authenticates nobody. + self.load_errors[tid] = f"{type(exc).__name__}" + return None + + def rescan(self, force: bool = False) -> None: + now = self._clock() + with self._lock: + if not force and now - self._last_scan < self.recheck_seconds: + return + self._last_scan = now + base = self.data_dir / "tenants" + found = set() + if base.is_dir(): + for entry in os.scandir(base): + if entry.is_dir(follow_symlinks=False) and names.is_tenant_id(entry.name): + found.add(entry.name) + for tid in list(self._records): + if tid not in found: + del self._records[tid] + for tid in found: + rec = self._load_one(tid) + if rec is None: + self._records.pop(tid, None) + else: + self._records[tid] = rec + self._checked_at[tid] = now + + def _fresh(self, tid: str, force: bool = False) -> Optional[TenantRecord]: + """The current record for ``tid``, re-reading ``tenant.json`` if it changed. + + An unknown tenant costs one targeted file open (not a directory scan); + a *known* tenant is re-stat'ed at most every ``recheck_seconds`` unless + ``force`` (used when a presented key id is not in the cached record, so + a key created a moment ago works immediately). Callers that fail + authentication are throttled per peer before this is reached. + """ + now = self._clock() + with self._lock: + rec = self._records.get(tid) + if rec is not None and not force and now - self._checked_at.get(tid, -1e9) < self.recheck_seconds: + return rec + try: + st = self._tenant_file(tid).stat() + stamp = (st.st_mtime_ns, st.st_size) + except OSError: + stamp = None + with self._lock: + self._checked_at[tid] = now + cur = self._records.get(tid) + if cur is not None and stamp is not None and stamp == cur.stamp: + return cur + if stamp is None: + self._records.pop(tid, None) + return None + new = self._load_one(tid) + with self._lock: + if new is None: + self._records.pop(tid, None) + else: + self._records[tid] = new + return new + + def tenant_ids(self) -> List[str]: + with self._lock: + return sorted(self._records) + + def get(self, tid: str) -> Optional[TenantRecord]: + return self._fresh(tid) if names.is_tenant_id(tid) else None + + def quotas_for(self, rec: TenantRecord) -> Quotas: + return self.default_quotas.merged(rec.quota_overrides) + + # -- authentication -------------------------------------------------------- + + def authenticate(self, token: str) -> Optional[Principal]: + """Return the Principal for a valid key, else None. Constant-time compare.""" + parsed = parse_key(token) + if parsed is None: + # Still burn one HMAC so malformed tokens are not a timing oracle + # for "your token had the right shape". + hmac.compare_digest(_hash_secret(_DUMMY_SALT, "x" * 43), _DUMMY_HASH) + return None + tid, kid, secret = parsed + rec = self._fresh(tid) + key = rec.keys.get(kid) if rec is not None else None + if rec is not None and key is None: + rec = self._fresh(tid, force=True) # maybe the key was created just now + key = rec.keys.get(kid) if rec is not None else None + if key is None or key.get("revoked_at") is not None: + salt, expected = _DUMMY_SALT, _DUMMY_HASH + valid_shape = False + else: + salt, expected = bytes.fromhex(key["salt"]), key["hash"] + valid_shape = True + presented = _hash_secret(salt, secret) + match = hmac.compare_digest(presented, expected) + if not (match and valid_shape): + return None + assert rec is not None and key is not None + if rec.disabled: + return None + try: + quotas = self.quotas_for(rec) + except ConfigError: + return None # corrupt quota override: fail closed + return Principal(tenant_id=tid, key_id=kid, role=key["role"], quotas=quotas) + + +# -- writer side (used by the CLI) ------------------------------------------------ + + +def tenant_file_path(data_dir: str, tid: str) -> Path: + return names.tenant_dir(data_dir, tid) / "tenant.json" + + +def create_tenant(data_dir: str, tid: str, quotas: Optional[Dict[str, Any]] = None) -> Path: + if not names.is_tenant_id(tid): + raise ValueError("tenant id must match [a-z0-9][a-z0-9-]{0,30}[a-z0-9]") + path = tenant_file_path(data_dir, tid) + if path.exists(): + raise FileExistsError(f"tenant {tid} already exists") + if quotas: + Quotas().merged(quotas) # validates names and values + doc = { + "format": TENANT_FORMAT, + "version": TENANT_FORMAT_VERSION, + "tenant_id": tid, + "created_at": time.time(), + "disabled": False, + "quotas": quotas or {}, + "keys": [], + } + (path.parent / "graphs").mkdir(parents=True, exist_ok=True) + _atomic_write_json(path, doc) + return path + + +def _load_raw(data_dir: str, tid: str) -> Tuple[Path, Dict[str, Any]]: + path = tenant_file_path(data_dir, tid) + with open(path, "r", encoding="utf-8") as handle: + data = json.load(handle) + parse_tenant_file(data, tid) + return path, data + + +def add_key(data_dir: str, tid: str, role: str = "rw", label: str = "") -> str: + """Create a key and return the full token. This is the only time it exists.""" + path, data = _load_raw(data_dir, tid) + record, kid, secret = new_key_record(role, label) + data["keys"].append(record) + _atomic_write_json(path, data) + return make_token(tid, kid, secret) + + +def revoke_key(data_dir: str, tid: str, key_id: str) -> None: + path, data = _load_raw(data_dir, tid) + for rec in data["keys"]: + if rec["key_id"] == key_id: + if rec["revoked_at"] is None: + rec["revoked_at"] = time.time() + _atomic_write_json(path, data) + return + raise KeyError(key_id) + + +def set_disabled(data_dir: str, tid: str, disabled: bool) -> None: + path, data = _load_raw(data_dir, tid) + data["disabled"] = disabled + _atomic_write_json(path, data) + + +def set_quotas(data_dir: str, tid: str, quotas: Dict[str, Any]) -> None: + path, data = _load_raw(data_dir, tid) + merged = dict(data.get("quotas", {})) + merged.update(quotas) + Quotas().merged(merged) + data["quotas"] = merged + _atomic_write_json(path, data) + + +def describe_tenant(data_dir: str, tid: str) -> Dict[str, Any]: + """Tenant summary for the CLI. Never contains salts or hashes.""" + _, data = _load_raw(data_dir, tid) + return { + "tenant_id": tid, + "disabled": data.get("disabled", False), + "quotas": data.get("quotas", {}), + "keys": [{k: r[k] for k in ("key_id", "label", "role", "created_at", "revoked_at")} + for r in data["keys"]], + } diff --git a/isongraph-vector-server/isongraph_vector_server/cli.py b/isongraph-vector-server/isongraph_vector_server/cli.py new file mode 100644 index 0000000..518b59f --- /dev/null +++ b/isongraph-vector-server/isongraph_vector_server/cli.py @@ -0,0 +1,247 @@ +"""igv-server command line: run nodes and the router, manage tenants and keys. + +Tenant and key management is a *local, offline* operation on the data +directory (it edits ``tenant.json``); there is deliberately no HTTP admin +API, so there is no remotely reachable credential that can mint other +credentials. A running server notices the change within a second. +""" + +from __future__ import annotations + +import argparse +import json +import os +import signal +import sys +import threading +from pathlib import Path +from typing import Any, Dict, List, Optional + +from . import auth, names +from .config import ConfigError, load_config +from .router import rendezvous + + +def _data_dir(args) -> str: + return args.data_dir or os.environ.get("IGV_DATA_DIR") or "./igv-data" + + +def _parse_quotas(items: Optional[List[str]]) -> Dict[str, Any]: + out: Dict[str, Any] = {} + for item in items or []: + if "=" not in item: + raise SystemExit(f"quota must look like name=value, got {item!r}") + k, v = item.split("=", 1) + try: + out[k] = json.loads(v) + except ValueError: + raise SystemExit(f"quota value for {k} is not a number: {v!r}") from None + return out + + +def _wait_for_signal(stop: threading.Event) -> None: + def handler(signum, frame): + stop.set() + for name in ("SIGINT", "SIGTERM", "SIGBREAK"): + sig = getattr(signal, name, None) + if sig is not None: + try: + signal.signal(sig, handler) + except (ValueError, OSError): + pass + while not stop.wait(0.5): + pass + + +def cmd_serve(args) -> int: + from .server import Server + cfg = load_config(args.config) + for attr, key in (("host", "host"), ("port", "port"), ("data_dir", "data_dir"), ("role", "role")): + value = getattr(args, attr, None) + if value is not None: + setattr(cfg, key, value) + cfg.validate() + server = Server(cfg).start() + stop = threading.Event() + _wait_for_signal(stop) + server.stop() + return 0 + + +def cmd_router(args) -> int: + from .server import RouterServer + cfg = load_config(args.config) + if args.host is not None: + cfg.host = args.host + if args.port is not None: + cfg.port = args.port + server = RouterServer(cfg).start() + stop = threading.Event() + _wait_for_signal(stop) + server.stop() + return 0 + + +def cmd_tenant(args) -> int: + d = _data_dir(args) + if args.action == "create": + auth.create_tenant(d, args.tenant_id, _parse_quotas(args.quota)) + print(f"created tenant {args.tenant_id}") + elif args.action == "list": + base = Path(d) / "tenants" + for entry in sorted(os.scandir(base), key=lambda e: e.name) if base.is_dir() else []: + if names.is_tenant_id(entry.name) and (Path(entry.path) / "tenant.json").is_file(): + info = auth.describe_tenant(d, entry.name) + active = sum(1 for k in info["keys"] if k["revoked_at"] is None) + print(f"{entry.name}\t{'disabled' if info['disabled'] else 'active'}\t{active} key(s)") + elif args.action == "show": + print(json.dumps(auth.describe_tenant(d, args.tenant_id), indent=2)) + elif args.action in ("disable", "enable"): + auth.set_disabled(d, args.tenant_id, args.action == "disable") + print(f"{args.action}d tenant {args.tenant_id}") + elif args.action == "set-quota": + auth.set_quotas(d, args.tenant_id, _parse_quotas(args.quota)) + print("quotas updated") + return 0 + + +def cmd_key(args) -> int: + d = _data_dir(args) + if args.action == "create": + token = auth.add_key(d, args.tenant_id, args.role, args.label or "") + print("API key (shown once; only a salted hash is stored):", file=sys.stderr) + print(token) + elif args.action == "revoke": + auth.revoke_key(d, args.tenant_id, args.key_id) + print(f"revoked key {args.key_id}") + return 0 + + +def cmd_shard_of(args) -> int: + cfg = load_config(args.config) + shard_names = [s["name"] for s in cfg.router.shards] + pinned = cfg.router.pins.get(args.tenant_id) + print(pinned or rendezvous(args.tenant_id, shard_names)) + return 0 + + +def cmd_verify(args) -> int: + """Offline integrity check of every graph under a data dir. Exit 1 if any fails.""" + from .storage import GraphStorage, SnapshotError + base = Path(_data_dir(args)) / "tenants" + bad = 0 + total = 0 + for t in sorted(os.scandir(base), key=lambda e: e.name) if base.is_dir() else []: + if not names.is_tenant_id(t.name): + continue + gdir = Path(t.path) / "graphs" + for g in sorted(os.scandir(gdir), key=lambda e: e.name) if gdir.is_dir() else []: + if not (g.name.startswith("g-") and names.is_graph_name(g.name[2:])): + continue + total += 1 + st = GraphStorage(Path(g.path)) + try: + meta = st.read_meta() + from .encoders import build_encoder + graph, manifest, degraded, _ = st.load(meta, None) + nodes = graph.node_count() + graph.close() + status = "DEGRADED (older generation)" if degraded else "ok" + if degraded: + bad += 1 + print(f"{t.name}/{g.name[2:]}\t{status}\t{nodes} nodes\tgeneration " + f"{manifest.generation if manifest else 0}") + except (SnapshotError, OSError) as exc: + bad += 1 + print(f"{t.name}/{g.name[2:]}\tFAILED\t{exc}") + print(f"{total} graph(s) checked, {bad} problem(s)") + return 1 if bad else 0 + + +def cmd_replica_sync(args) -> int: + """Run one replica sync cycle and exit (cron-friendly).""" + from .auth import TenantRegistry + from .config import ServerConfig + from .engine import Engine + from .encoders import build_encoder + from .metrics import Metrics + from .replication import ReplicaSyncer + cfg = load_config(args.config) + cfg.role = "replica" + cfg.validate() + metrics = Metrics() + engine = Engine(cfg, build_encoder(cfg.encoder), metrics) + engine.start() + try: + syncer = ReplicaSyncer(cfg, engine, TenantRegistry(cfg.data_dir, cfg.quotas), metrics) + report = syncer.sync_once() + print(json.dumps(report, indent=2)) + return 0 if not report["errors"] and not report["refused"] else 1 + finally: + engine.shutdown() + + +def build_parser() -> argparse.ArgumentParser: + p = argparse.ArgumentParser(prog="igv-server", description=__doc__.split("\n\n")[0]) + sub = p.add_subparsers(dest="cmd", required=True) + + s = sub.add_parser("serve", help="run a storage node (primary or replica)") + s.add_argument("--config") + s.add_argument("--host") + s.add_argument("--port", type=int) + s.add_argument("--data-dir", dest="data_dir") + s.add_argument("--role", choices=["primary", "replica"]) + s.set_defaults(fn=cmd_serve) + + r = sub.add_parser("router", help="run the stateless tenant-sharding router") + r.add_argument("--config", required=True) + r.add_argument("--host") + r.add_argument("--port", type=int) + r.set_defaults(fn=cmd_router) + + t = sub.add_parser("tenant", help="manage tenants (offline, on the data dir)") + t.add_argument("action", choices=["create", "list", "show", "disable", "enable", "set-quota"]) + t.add_argument("tenant_id", nargs="?") + t.add_argument("--quota", action="append", metavar="NAME=VALUE") + t.add_argument("--data-dir", dest="data_dir") + t.set_defaults(fn=cmd_tenant) + + k = sub.add_parser("key", help="create or revoke API keys") + k.add_argument("action", choices=["create", "revoke"]) + k.add_argument("tenant_id") + k.add_argument("key_id", nargs="?") + k.add_argument("--role", choices=["rw", "ro"], default="rw") + k.add_argument("--label") + k.add_argument("--data-dir", dest="data_dir") + k.set_defaults(fn=cmd_key) + + sh = sub.add_parser("shard-of", help="which router shard owns a tenant") + sh.add_argument("tenant_id") + sh.add_argument("--config", required=True) + sh.set_defaults(fn=cmd_shard_of) + + v = sub.add_parser("verify", help="offline integrity check of every snapshot in a data dir") + v.add_argument("--data-dir", dest="data_dir") + v.set_defaults(fn=cmd_verify) + + rs = sub.add_parser("replica-sync", help="run one replica sync cycle and exit") + rs.add_argument("--config", required=True) + rs.set_defaults(fn=cmd_replica_sync) + return p + + +def main(argv: Optional[List[str]] = None) -> int: + args = build_parser().parse_args(argv) + try: + if args.cmd in ("tenant", "key") and args.action not in ("list",) and not args.tenant_id: + raise SystemExit("a tenant id is required") + if args.cmd == "key" and args.action == "revoke" and not args.key_id: + raise SystemExit("a key id is required") + return args.fn(args) + except (ConfigError, ValueError, FileExistsError, FileNotFoundError, KeyError) as exc: + print(f"error: {exc}", file=sys.stderr) + return 2 + + +if __name__ == "__main__": # pragma: no cover + sys.exit(main()) diff --git a/isongraph-vector-server/isongraph_vector_server/config.py b/isongraph-vector-server/isongraph_vector_server/config.py new file mode 100644 index 0000000..af2a639 --- /dev/null +++ b/isongraph-vector-server/isongraph_vector_server/config.py @@ -0,0 +1,251 @@ +"""Configuration: a JSON file, overridden by ``IGV_*`` environment variables. + +Secrets (the metrics token, the replication token, an embedding endpoint's API +key) are read from the environment only and are never part of the config +file, so a config file can be committed, logged or mounted read-only without +leaking anything. Tenant API keys are not configuration at all: they live, +as salted hashes, in each tenant's ``tenant.json`` (see ``auth.py``). +""" + +from __future__ import annotations + +import dataclasses +import json +import os +from dataclasses import dataclass, field +from typing import Any, Dict, Mapping, Optional + + +class ConfigError(ValueError): + pass + + +@dataclass +class Quotas: + """Per-tenant limits. A tenant's ``tenant.json`` may override any field.""" + max_graphs: int = 20 + max_nodes: int = 100_000 # across all of the tenant's graphs + max_edges: int = 500_000 # stored edges (an undirected edge counts twice) + max_dimension: int = 1536 # vector length a graph of this tenant may use + max_batch_items: int = 500 # nodes/edges per request + max_top_k: int = 100 # top_k / top_k_results / top_k_seeds cap + max_hops: int = 5 + max_text_chars: int = 20_000 # one embed text / one string property + max_properties: int = 64 # properties per node/edge + rate_per_second: float = 50.0 # sustained requests/s (token bucket refill) + rate_burst: int = 100 # bucket capacity + max_inflight: int = 8 # concurrent requests from this tenant + + def merged(self, overrides: Optional[Mapping[str, Any]]) -> "Quotas": + if not overrides: + return self + data = dataclasses.asdict(self) + for key, value in overrides.items(): + if key not in data: + raise ConfigError(f"unknown quota {key!r}") + data[key] = value + return Quotas(**data).checked() + + def checked(self) -> "Quotas": + for f in dataclasses.fields(self): + value = getattr(self, f.name) + if isinstance(value, bool) or not isinstance(value, (int, float)): + raise ConfigError(f"quota {f.name} must be a number") + if value <= 0: + raise ConfigError(f"quota {f.name} must be positive") + for name in ("max_graphs", "max_nodes", "max_edges", "max_dimension", + "max_batch_items", "max_top_k", "max_hops", "max_text_chars", + "max_properties", "rate_burst", "max_inflight"): + if not isinstance(getattr(self, name), int): + raise ConfigError(f"quota {name} must be an integer") + return self + + +@dataclass +class EncoderConfig: + kind: str = "none" # none | mock | http | sentence-transformers + dimension: Optional[int] = None # mock: required; http: optional (probed) + model: Optional[str] = None + url: Optional[str] = None # http + api: str = "openai" # http: openai | ollama + api_key_env: Optional[str] = None # http: NAME of the env var holding the key + version: Optional[str] = None + timeout: float = 30.0 + max_concurrency: int = 4 # simultaneous encoder calls, all tenants + + +@dataclass +class ReplicaConfig: + source: str = "" # "http://primary:8080" or "dir:/path/to/primary/data" + sync_interval_seconds: float = 15.0 + max_staleness_seconds: float = 120.0 # refuse reads when the last good sync is older + fetch_timeout_seconds: float = 30.0 + max_file_bytes: int = 2 * 1024 ** 3 + autostart: bool = True # run the background sync loop (tests drive sync_once() by hand) + + +@dataclass +class RouterConfig: + #: [{"name": "s0", "primary": "http://host:8080", "replicas": ["http://host:8081"]}, ...] + shards: list = field(default_factory=list) + #: tenant id -> shard name, overriding the hash (used while moving a tenant by hand) + pins: dict = field(default_factory=dict) + read_preference: str = "primary" # primary | replica_preferred + upstream_timeout_seconds: float = 30.0 + expose_backend_header: bool = False # add X-Igv-Backend: <shard>/<primary|replica> + + +@dataclass +class ServerConfig: + role: str = "primary" # primary | replica + data_dir: str = "./igv-data" + host: str = "127.0.0.1" + port: int = 8080 + log_level: str = "INFO" + + # connection / request handling + max_workers: int = 32 # worker threads == max simultaneous connections served + accept_queue: int = 64 # accepted-but-waiting connections before 503 + socket_timeout_seconds: float = 10.0 # idle / slow-read limit per socket operation + max_body_bytes: int = 1_048_576 + max_header_bytes: int = 16_384 + max_requests_per_connection: int = 1000 + max_concurrent_requests: int = 32 # requests inside handlers (all tenants) + queue_wait_seconds: float = 0.5 # wait for a concurrency slot before 503 + request_deadline_seconds: float = 30.0 + lock_timeout_seconds: float = 10.0 # wait for a graph lock before 503 + failed_auth_per_minute: int = 30 # per peer address, before 429 on bad keys + registry_recheck_seconds: float = 1.0 # how soon a revoked key / disabled tenant takes effect + max_page: int = 1000 + max_response_bytes: int = 16 * 1024 * 1024 + # Peer addresses allowed to tell us the real client via X-Forwarded-For + # (the router, a reverse proxy). Used only for the failed-auth throttle. + trusted_proxies: list = field(default_factory=list) + + # persistence + durability: str = "snapshot" # snapshot (periodic) | sync (save before every ack) + autosave_interval_seconds: float = 30.0 + keep_generations: int = 3 + strict_startup: bool = False # refuse to start when a graph fails verification + + quotas: Quotas = field(default_factory=Quotas) + encoder: EncoderConfig = field(default_factory=EncoderConfig) + replica: ReplicaConfig = field(default_factory=ReplicaConfig) + router: RouterConfig = field(default_factory=RouterConfig) + + # secrets, from the environment only (never read from the file) + metrics_token: Optional[str] = field(default=None, repr=False) + replication_token: Optional[str] = field(default=None, repr=False) + + def validate(self) -> "ServerConfig": + if self.role not in ("primary", "replica"): + raise ConfigError("role must be 'primary' or 'replica'") + if self.durability not in ("snapshot", "sync"): + raise ConfigError("durability must be 'snapshot' or 'sync'") + if self.role == "replica" and self.durability == "sync": + self.durability = "snapshot" + if not (0 <= self.port <= 65535): + raise ConfigError("port out of range") + for name in ("max_workers", "accept_queue", "max_body_bytes", "max_header_bytes", + "max_concurrent_requests", "keep_generations", "max_page", + "max_requests_per_connection", "failed_auth_per_minute", "max_response_bytes"): + value = getattr(self, name) + if isinstance(value, bool) or not isinstance(value, int) or value < 1: + raise ConfigError(f"{name} must be an integer >= 1") + for name in ("socket_timeout_seconds", "queue_wait_seconds", + "request_deadline_seconds", "lock_timeout_seconds"): + value = getattr(self, name) + if isinstance(value, bool) or not isinstance(value, (int, float)) or value <= 0: + raise ConfigError(f"{name} must be a positive number") + if self.autosave_interval_seconds < 0: + raise ConfigError("autosave_interval_seconds must be >= 0 (0 disables autosave)") + if self.keep_generations < 2: + raise ConfigError("keep_generations must be >= 2 (the previous one is the fallback)") + if self.encoder.kind not in ("none", "mock", "http", "sentence-transformers"): + raise ConfigError("encoder.kind must be none, mock, http or sentence-transformers") + if self.encoder.kind == "mock" and not self.encoder.dimension: + raise ConfigError("encoder.dimension is required for the mock encoder") + if self.encoder.kind == "http" and not self.encoder.url: + raise ConfigError("encoder.url is required for the http encoder") + if self.encoder.kind in ("http",) and not self.encoder.model: + raise ConfigError("encoder.model is required for the http encoder") + if self.role == "replica" and not self.replica.source: + raise ConfigError("replica.source is required when role is 'replica'") + for tok_name in ("metrics_token", "replication_token"): + tok = getattr(self, tok_name) + if tok is not None and len(tok) < 24: + raise ConfigError(f"{tok_name} (IGV_{tok_name.upper()}) must be at least 24 characters") + self.quotas.checked() + if self.router.read_preference not in ("primary", "replica_preferred"): + raise ConfigError("router.read_preference must be 'primary' or 'replica_preferred'") + seen = set() + for sh in self.router.shards: + if (not isinstance(sh, dict) or not isinstance(sh.get("name"), str) + or not isinstance(sh.get("primary"), str) + or not isinstance(sh.get("replicas", []), list)): + raise ConfigError("each router shard needs a name, a primary URL and a replicas list") + if sh["name"] in seen: + raise ConfigError("duplicate shard name") + seen.add(sh["name"]) + for tid, shard in self.router.pins.items(): + if shard not in seen: + raise ConfigError(f"pin for {tid!r} names an unknown shard") + return self + + +def _coerce(target: Any, data: Mapping[str, Any], where: str) -> None: + fields = {f.name: f for f in dataclasses.fields(target)} + for key, value in data.items(): + if key not in fields: + raise ConfigError(f"unknown config key {where}{key!r}") + if key in ("metrics_token", "replication_token"): + raise ConfigError(f"{key} is a secret: set it via the IGV_{key.upper()} environment variable") + current = getattr(target, key) + if dataclasses.is_dataclass(current): + if not isinstance(value, dict): + raise ConfigError(f"{where}{key} must be an object") + _coerce(current, value, f"{where}{key}.") + else: + setattr(target, key, value) + + +_ENV_SCALARS = { + "IGV_ROLE": ("role", str), + "IGV_DATA_DIR": ("data_dir", str), + "IGV_HOST": ("host", str), + "IGV_PORT": ("port", int), + "IGV_LOG_LEVEL": ("log_level", str), + "IGV_DURABILITY": ("durability", str), + "IGV_AUTOSAVE_INTERVAL_SECONDS": ("autosave_interval_seconds", float), + "IGV_MAX_WORKERS": ("max_workers", int), + "IGV_MAX_BODY_BYTES": ("max_body_bytes", int), + "IGV_STRICT_STARTUP": ("strict_startup", lambda s: s.lower() in ("1", "true", "yes")), +} + + +def load_config(path: Optional[str] = None, + env: Optional[Mapping[str, str]] = None) -> ServerConfig: + """Build a validated config from an optional JSON file plus ``IGV_*`` env vars.""" + env = os.environ if env is None else env + cfg = ServerConfig() + path = path or env.get("IGV_CONFIG") + if path: + try: + with open(path, "r", encoding="utf-8") as handle: + data = json.load(handle) + except (OSError, ValueError) as exc: + raise ConfigError(f"cannot read config {path}: {exc}") from None + if not isinstance(data, dict): + raise ConfigError("config file must contain a JSON object") + _coerce(cfg, data, "") + for var, (attr, conv) in _ENV_SCALARS.items(): + if var in env: + try: + setattr(cfg, attr, conv(env[var])) + except ValueError: + raise ConfigError(f"{var} has an invalid value") from None + if env.get("IGV_REPLICA_SOURCE"): + cfg.replica.source = env["IGV_REPLICA_SOURCE"] + cfg.metrics_token = env.get("IGV_METRICS_TOKEN") or None + cfg.replication_token = env.get("IGV_REPLICATION_TOKEN") or None + return cfg.validate() diff --git a/isongraph-vector-server/isongraph_vector_server/encoders.py b/isongraph-vector-server/isongraph_vector_server/encoders.py new file mode 100644 index 0000000..485b019 --- /dev/null +++ b/isongraph-vector-server/isongraph_vector_server/encoders.py @@ -0,0 +1,87 @@ +"""Build the one shared encoder from config and bound how it is used. + +The encoder is shared by every tenant, so it is the one place where tenants +compete for a resource. :class:`BoundedEncoder` caps simultaneous calls and +makes a caller wait only a bounded time for a slot (raising ``Overloaded``), +so a slow remote embedding endpoint cannot absorb every worker thread. +Texts are encoded *outside* any graph lock: a slow encoder stalls the one +request, never a whole graph. +""" + +from __future__ import annotations + +import os +import threading +from typing import List, Optional + +from isongraph_vector import EmbeddingEncoder, EmbeddingError + +from .config import EncoderConfig +from .errors import ApiError, Overloaded, Unavailable + +ENCODE_CHUNK = 64 + + +class BoundedEncoder(EmbeddingEncoder): + def __init__(self, inner: EmbeddingEncoder, max_concurrency: int, wait: float = 5.0, + serialize: bool = False): + self._inner = inner + self._sem = threading.BoundedSemaphore(max(1, max_concurrency)) + self._wait = wait + self._serial = threading.Lock() if serialize else None + self.model_name = getattr(inner, "model_name", "unknown") + self.version = getattr(inner, "version", None) + + @property + def dimension(self) -> int: + return int(self._inner.dimension) + + def _call(self, fn, *args): + if not self._sem.acquire(timeout=self._wait): + raise Overloaded("embedding capacity exhausted; retry shortly", + headers={"Retry-After": "1"}) + try: + if self._serial is not None: + with self._serial: + return fn(*args) + return fn(*args) + except ApiError: + raise + except EmbeddingError as exc: + # Do not echo the encoder's message: it can quote the upstream URL or body. + raise Unavailable("the embedding service failed", code="encoder_failed") from exc + finally: + self._sem.release() + + def encode(self, text: str) -> List[float]: + return self._call(self._inner.encode, text) + + def encode_batch(self, texts: List[str]) -> List[List[float]]: + texts = list(texts) + out: List[List[float]] = [] + for i in range(0, len(texts), ENCODE_CHUNK): + out.extend(self._call(self._inner.encode_batch, texts[i:i + ENCODE_CHUNK])) + return out + + +def build_encoder(cfg: EncoderConfig, env=None) -> Optional[EmbeddingEncoder]: + env = os.environ if env is None else env + if cfg.kind == "none": + return None + if cfg.kind == "mock": + from isongraph_vector import MockEncoder + inner: EmbeddingEncoder = MockEncoder(cfg.dimension, version=cfg.version) + serialize = False + elif cfg.kind == "http": + from isongraph_vector import HttpEncoder + key = env.get(cfg.api_key_env) if cfg.api_key_env else None + inner = HttpEncoder(cfg.url, cfg.model, api_key=key, dimension=cfg.dimension, + timeout=cfg.timeout, api=cfg.api, version=cfg.version) + serialize = False + elif cfg.kind == "sentence-transformers": + from isongraph_vector import SentenceTransformerEncoder + inner = SentenceTransformerEncoder(cfg.model or "all-MiniLM-L6-v2", version=cfg.version) + serialize = True # one model instance; do not assume concurrent encode() is safe + else: # pragma: no cover - validated in config + raise ValueError(cfg.kind) + return BoundedEncoder(inner, cfg.max_concurrency, serialize=serialize) diff --git a/isongraph-vector-server/isongraph_vector_server/engine.py b/isongraph-vector-server/isongraph_vector_server/engine.py new file mode 100644 index 0000000..2b790a0 --- /dev/null +++ b/isongraph-vector-server/isongraph_vector_server/engine.py @@ -0,0 +1,419 @@ +"""In-memory state: tenants, their graphs, quotas, and the save/load lifecycle. + +Isolation is structural. There is no process-wide map keyed by a string that +mixes tenant and graph names: a :class:`TenantRuntime` owns its own +``graphs`` dict, handlers receive the runtime of the *authenticated* tenant +only, and the only way to reach a graph is ``tenant.graphs[name]``. A request +cannot name another tenant's runtime because no request field selects one. +""" + +from __future__ import annotations + +import logging +import os +import shutil +import threading +import time +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Dict, List, Optional + +from isongraph_vector import EmbeddingEncoder, SemanticGraph + +from . import names +from .config import Quotas, ServerConfig +from .errors import (Conflict, DeadlineExceeded, NotFound, QuotaExceeded, Unavailable) +from .limits import Gate, LockTimeout, RWLock, TokenBucket +from .logs import log +from .metrics import Metrics +from .storage import (GraphMeta, GraphStorage, SnapshotError, acquire_dir_lock, new_semantic_graph, + serialize) + + +@dataclass +class OpContext: + """Deadline bookkeeping for one request.""" + deadline: float + lock_timeout: float + + def remaining(self) -> float: + return self.deadline - time.monotonic() + + def check(self) -> None: + if self.remaining() <= 0: + raise DeadlineExceeded("request deadline exceeded") + + def lock_wait(self) -> float: + left = self.remaining() + if left <= 0: + raise DeadlineExceeded("request deadline exceeded") + return min(self.lock_timeout, left) + + +class GraphHandle: + """One graph of one tenant: the library object plus its lock and bookkeeping.""" + + def __init__(self, name: str, meta: Optional[GraphMeta], storage: Optional[GraphStorage], + graph: Optional[SemanticGraph]): + self.name = name + self.meta = meta + self.storage = storage + self.graph = graph + self.lock = RWLock() + self.save_lock = threading.Lock() + self.version = 0 # bumped (under the write lock) by every mutation + self.saved_version = 0 + self.generation = 0 + self.manifest = None + self.degraded = False + self.quarantined: Optional[str] = None + self.deleted = False + self.last_save_ts: Optional[float] = None + self.last_save_error: Optional[str] = None + self._cache_lock = threading.Lock() + self._sorted_nodes = None # (version, [keys]) + self._sorted_edges = None + self._digest_cache = None # (version, digest) + self.synced_at: Optional[float] = None # replicas: wall time of the last verified sync + + @property + def dirty(self) -> bool: + return self.version != self.saved_version + + def bump(self) -> None: + """Record a mutation. Call with the write lock held.""" + self.version += 1 + + +class TenantRuntime: + def __init__(self, tenant_id: str, quotas: Quotas): + self.tenant_id = tenant_id + self.graphs: Dict[str, GraphHandle] = {} + self.glock = threading.Lock() # guards the graphs dict and creating-state + self._creating: set = set() + self.quota_lock = threading.Lock() + self.used_nodes = 0 + self.used_edges = 0 + self.bucket = TokenBucket(quotas.rate_per_second, quotas.rate_burst) + self._rate_cfg = (quotas.rate_per_second, quotas.rate_burst) + self.gate = Gate(quotas.max_inflight) + self.requests = 0 + self.rate_limited = 0 + + def apply_quotas(self, quotas: Quotas) -> None: + cfg = (quotas.rate_per_second, quotas.rate_burst) + if cfg != self._rate_cfg: + self.bucket.reconfigure(*cfg) + self._rate_cfg = cfg + if self.gate.size != quotas.max_inflight: + self.gate = Gate(quotas.max_inflight) + + # quota accounting: reserve before applying, reconcile after. + def reserve(self, quotas: Quotas, nodes: int = 0, edges: int = 0) -> None: + with self.quota_lock: + if nodes > 0 and self.used_nodes + nodes > quotas.max_nodes: + raise QuotaExceeded( + f"node quota of {quotas.max_nodes} would be exceeded " + f"({self.used_nodes} used, {nodes} requested)") + if edges > 0 and self.used_edges + edges > quotas.max_edges: + raise QuotaExceeded( + f"edge quota of {quotas.max_edges} would be exceeded " + f"({self.used_edges} used, {edges} requested)") + self.used_nodes += nodes + self.used_edges += edges + + def adjust(self, nodes: int = 0, edges: int = 0) -> None: + with self.quota_lock: + self.used_nodes = max(0, self.used_nodes + nodes) + self.used_edges = max(0, self.used_edges + edges) + + +class Engine: + def __init__(self, cfg: ServerConfig, encoder: Optional[EmbeddingEncoder], metrics: Metrics): + self.cfg = cfg + self.encoder = encoder + self.metrics = metrics + self.data_dir = Path(cfg.data_dir) + self.tenants: Dict[str, TenantRuntime] = {} + self._tenants_lock = threading.Lock() + self.ready = False + self._stop = threading.Event() + self._autosave_thread: Optional[threading.Thread] = None + self._dir_lock = None + self.startup_report: Dict[str, Any] = {"loaded": 0, "quarantined": 0, "degraded": 0} + + # -- tenants --------------------------------------------------------------------- + + def runtime(self, tenant_id: str, quotas: Quotas) -> TenantRuntime: + with self._tenants_lock: + rt = self.tenants.get(tenant_id) + if rt is None: + rt = self.tenants[tenant_id] = TenantRuntime(tenant_id, quotas) + rt.apply_quotas(quotas) + return rt + + def _graphs_parent(self, tenant_id: str) -> Path: + return names.tenant_dir(self.data_dir, tenant_id) / "graphs" + + # -- startup ----------------------------------------------------------------------- + + def start(self) -> None: + self.data_dir.mkdir(parents=True, exist_ok=True) + (self.data_dir / "tenants").mkdir(exist_ok=True) + self._dir_lock = acquire_dir_lock(self.data_dir) + self.load_all() + self.ready = True + if self.cfg.autosave_interval_seconds > 0 and self.cfg.role == "primary": + self._autosave_thread = threading.Thread( + target=self._autosave_loop, name="igv-autosave", daemon=True) + self._autosave_thread.start() + + def load_all(self) -> None: + base = self.data_dir / "tenants" + for entry in sorted(os.scandir(base), key=lambda e: e.name): + if not (entry.is_dir(follow_symlinks=False) and names.is_tenant_id(entry.name)): + continue + tid = entry.name + parent = self._graphs_parent(tid) + if not parent.is_dir(): + continue + rt = self.runtime(tid, self.cfg.quotas) + for g in sorted(os.scandir(parent), key=lambda e: e.name): + if g.name.startswith((".tmp-", ".trash-")): + shutil.rmtree(g.path, ignore_errors=True) + continue + if not (g.is_dir(follow_symlinks=False) and g.name.startswith("g-") + and names.is_graph_name(g.name[2:])): + continue + handle = self._load_handle(tid, g.name[2:], Path(g.path)) + rt.graphs[handle.name] = handle + if handle.graph is not None: + rt.used_nodes += handle.graph.node_count() + rt.used_edges += handle.graph.edge_count() + self._update_gauges() + + def _load_handle(self, tid: str, name: str, path: Path) -> GraphHandle: + storage = GraphStorage(path, self.cfg.keep_generations) + storage.clean_tmp() + try: + meta = storage.read_meta() + graph, manifest, degraded, caught = storage.load(meta, self.encoder) + except (SnapshotError, OSError, ValueError) as exc: + handle = GraphHandle(name, None, storage, None) + handle.quarantined = f"{type(exc).__name__}" + self.startup_report["quarantined"] += 1 + log(logging.ERROR, "graph failed verification and is quarantined", + tenant=tid, graph=name, reason=str(exc)[:300]) + if self.cfg.strict_startup: + raise SnapshotError(f"{tid}/{name}: {exc}") from None + return handle + handle = GraphHandle(name, meta, storage, graph) + if manifest is not None: + handle.generation = manifest.generation + handle.manifest = manifest + handle.degraded = degraded + self.startup_report["loaded"] += 1 + if degraded: + self.startup_report["degraded"] += 1 + log(logging.ERROR, "current generation failed verification; serving an older one", + tenant=tid, graph=name, generation=handle.generation) + for w in caught: + log(logging.WARNING, "library warning while loading", tenant=tid, graph=name, warning=w) + return handle + + def _update_gauges(self) -> None: + q = deg = 0 + for rt in list(self.tenants.values()): + for h in list(rt.graphs.values()): + q += h.quarantined is not None + deg += bool(h.degraded) + self.metrics.gauge("igv_graphs_quarantined", q) + self.metrics.gauge("igv_graphs_degraded", deg) + + # -- graph lifecycle ------------------------------------------------------------------ + + def get_handle(self, rt: TenantRuntime, name: str, allow_quarantined: bool = False) -> GraphHandle: + with rt.glock: + handle = rt.graphs.get(name) + if handle is None or handle.deleted: + raise NotFound("graph not found") + if handle.quarantined is not None and not allow_quarantined: + raise Unavailable("graph failed integrity verification and is unavailable", + code="graph_quarantined") + return handle + + def create_graph(self, rt: TenantRuntime, quotas: Quotas, meta: GraphMeta) -> GraphHandle: + with rt.glock: + if meta.name in rt.graphs or meta.name in rt._creating: + raise Conflict("a graph with that name already exists", code="already_exists") + if len(rt.graphs) + len(rt._creating) >= quotas.max_graphs: + raise QuotaExceeded(f"graph quota of {quotas.max_graphs} reached") + rt._creating.add(meta.name) + try: + storage = GraphStorage.create(self._graphs_parent(rt.tenant_id), meta, + self.cfg.keep_generations) + graph = new_semantic_graph(meta, self.encoder) + handle = GraphHandle(meta.name, meta, storage, graph) + with rt.glock: + rt.graphs[meta.name] = handle + finally: + with rt.glock: + rt._creating.discard(meta.name) + return handle + + def delete_graph(self, rt: TenantRuntime, name: str, ctx: OpContext) -> None: + with rt.glock: + handle = rt.graphs.get(name) + if handle is None: + raise NotFound("graph not found") + del rt.graphs[name] + # Wait for in-flight operations, then keep newcomers out. + try: + handle.lock.acquire_write(ctx.lock_wait()) + except LockTimeout: + with rt.glock: + rt.graphs[name] = handle + raise Unavailable("graph is busy; retry", code="busy", headers={"Retry-After": "1"}) from None + try: + handle.deleted = True + nodes = edges = 0 + if handle.graph is not None: + nodes, edges = handle.graph.node_count(), handle.graph.edge_count() + finally: + handle.lock.release_write() + with handle.save_lock: + if handle.storage is not None: + handle.storage.remove() + if handle.graph is not None: + handle.graph.close() + rt.adjust(nodes=-nodes, edges=-edges) + self._update_gauges() + + # -- saving -------------------------------------------------------------------------------- + + def save_graph(self, handle: GraphHandle, ctx: Optional[OpContext] = None, force: bool = False) -> bool: + """Snapshot a dirty graph. Returns True if a generation was written.""" + if handle.graph is None or handle.storage is None: + return False + timeout = ctx.lock_wait() if ctx else self.cfg.lock_timeout_seconds + with handle.save_lock: + if handle.deleted: + return False + if not (handle.dirty or force): + return False + try: + try: + handle.lock.acquire_read(timeout) + except LockTimeout: + raise Unavailable("graph is busy; retry", code="busy", + headers={"Retry-After": "1"}) from None + try: + version = handle.version + image = serialize(handle.graph) + finally: + handle.lock.release_read() + manifest = handle.storage.write_generation(image) + except Unavailable: + raise + except Exception as exc: + handle.last_save_error = type(exc).__name__ + self.metrics.inc("igv_snapshot_failures_total") + log(logging.ERROR, "snapshot failed", graph=handle.name, error=str(exc)[:300]) + raise + handle.saved_version = version + handle.generation = manifest.generation + handle.manifest = manifest + handle.degraded = False + handle.last_save_ts = time.time() + handle.last_save_error = None + self.metrics.inc("igv_snapshots_total") + return True + + def save_all(self) -> int: + n = 0 + for rt in list(self.tenants.values()): + for handle in list(rt.graphs.values()): + if handle.graph is not None and handle.dirty and not handle.deleted: + try: + if self.save_graph(handle): + n += 1 + except Exception: + pass # already logged and counted + return n + + def _autosave_loop(self) -> None: + while not self._stop.wait(self.cfg.autosave_interval_seconds): + try: + self.save_all() + except Exception as exc: # pragma: no cover - defensive + log(logging.ERROR, "autosave loop error", error=str(exc)[:200]) + + # -- replica hook ------------------------------------------------------------------------------ + + def swap_graph(self, rt: TenantRuntime, name: str, meta: GraphMeta, storage: GraphStorage, + graph: SemanticGraph, manifest) -> None: + """Atomically replace (or create) a graph with a verified one. Replica sync only.""" + with rt.glock: + handle = rt.graphs.get(name) + if handle is None or handle.quarantined is not None: + handle = GraphHandle(name, meta, storage, graph) + rt.graphs[name] = handle + old = None + old_counts = (0, 0) + else: + old = handle.graph + old_counts = (old.node_count(), old.edge_count()) if old is not None else (0, 0) + if old is not None: + handle.lock.acquire_write(self.cfg.lock_timeout_seconds * 3) + try: + handle.graph, handle.meta, handle.storage = graph, meta, storage + handle.bump() + handle.saved_version = handle.version + finally: + handle.lock.release_write() + old.close() + handle.generation = manifest.generation if manifest is not None else 0 + handle.manifest = manifest + handle.degraded = False + handle.quarantined = None + handle.synced_at = time.time() + rt.adjust(nodes=graph.node_count() - old_counts[0], edges=graph.edge_count() - old_counts[1]) + self._update_gauges() + + def drop_graph_local(self, rt: TenantRuntime, name: str) -> None: + """Replica: the primary no longer has this graph.""" + with rt.glock: + handle = rt.graphs.pop(name, None) + if handle is None: + return + handle.lock.acquire_write(self.cfg.lock_timeout_seconds * 3) + try: + handle.deleted = True + finally: + handle.lock.release_write() + if handle.graph is not None: + rt.adjust(nodes=-handle.graph.node_count(), edges=-handle.graph.edge_count()) + handle.graph.close() + if handle.storage is not None and handle.storage.dir.exists(): + handle.storage.remove() + self._update_gauges() + + # -- shutdown -------------------------------------------------------------------------------------- + + def shutdown(self) -> None: + """Stop autosave, write a final snapshot of every dirty graph, release resources.""" + self._stop.set() + if self._autosave_thread is not None: + self._autosave_thread.join(timeout=30) + if self.cfg.role == "primary": + self.save_all() + for rt in list(self.tenants.values()): + for handle in list(rt.graphs.values()): + if handle.graph is not None: + handle.graph.close() + if self._dir_lock is not None: + try: + self._dir_lock.close() + except OSError: + pass + self._dir_lock = None diff --git a/isongraph-vector-server/isongraph_vector_server/errors.py b/isongraph-vector-server/isongraph_vector_server/errors.py new file mode 100644 index 0000000..78d90cd --- /dev/null +++ b/isongraph-vector-server/isongraph_vector_server/errors.py @@ -0,0 +1,126 @@ +"""The error model. + +Every failure a client can see is an :class:`ApiError`, rendered as + + {"error": {"code": "...", "message": "...", "request_id": "...", "details": [...]}} + +Anything that is not an ApiError is logged server-side (with a traceback) and +answered with the generic ``internal_error``: no exception text, no stack +trace and no file path ever reaches a client. +""" + +from __future__ import annotations + +from typing import Any, Dict, List, Optional + + +class ApiError(Exception): + status = 500 + code = "internal_error" + + def __init__(self, message: str = "", *, code: Optional[str] = None, + status: Optional[int] = None, + details: Optional[List[Dict[str, Any]]] = None, + headers: Optional[Dict[str, str]] = None): + super().__init__(message) + self.message = message or self.code + if code is not None: + self.code = code + if status is not None: + self.status = status + self.details = details + self.headers = headers or {} + + def to_body(self, request_id: str) -> Dict[str, Any]: + err: Dict[str, Any] = { + "code": self.code, + "message": self.message, + "request_id": request_id, + } + if self.details: + err["details"] = self.details + return {"error": err} + + +class BadRequest(ApiError): + status = 400 + code = "invalid_request" + + +class Unauthorized(ApiError): + status = 401 + code = "unauthorized" + + +class Forbidden(ApiError): + status = 403 + code = "forbidden" + + +class NotFound(ApiError): + status = 404 + code = "not_found" + + +class MethodNotAllowed(ApiError): + status = 405 + code = "method_not_allowed" + + +class Conflict(ApiError): + status = 409 + code = "conflict" + + +class LengthRequired(ApiError): + status = 411 + code = "length_required" + + +class PayloadTooLarge(ApiError): + status = 413 + code = "payload_too_large" + + +class UnsupportedMediaType(ApiError): + status = 415 + code = "unsupported_media_type" + + +class QuotaExceeded(ApiError): + status = 403 + code = "quota_exceeded" + + +class RateLimited(ApiError): + status = 429 + code = "rate_limited" + + +class Overloaded(ApiError): + status = 503 + code = "overloaded" + + +class Unavailable(ApiError): + status = 503 + code = "unavailable" + + +class DeadlineExceeded(ApiError): + status = 504 + code = "deadline_exceeded" + + +class BadGateway(ApiError): + status = 502 + code = "bad_gateway" + + +def validation_error(path: str, message: str) -> BadRequest: + """A schema violation. ``path`` is a JSONPath-ish pointer into the body.""" + return BadRequest( + f"{path}: {message}", + code="validation_failed", + details=[{"path": path, "message": message}], + ) diff --git a/isongraph-vector-server/isongraph_vector_server/filters.py b/isongraph-vector-server/isongraph_vector_server/filters.py new file mode 100644 index 0000000..fc63f0e --- /dev/null +++ b/isongraph-vector-server/isongraph_vector_server/filters.py @@ -0,0 +1,110 @@ +"""Declarative node filters. + +The library takes ``node_filter`` as a Python callback. Over HTTP that has to +be data, never code, so a filter is a list of conditions (ANDed) plus an +optional list of allowed node types: + + {"types": ["person"], "where": [{"field": "age", "op": "gte", "value": 30}, + {"field": "team", "op": "in", "value": ["a", "b"]}]} + +Semantics: a missing field fails every condition except ``ne``, ``nin`` and +``exists`` with ``false``. Values of different types never compare equal and +never order (no exception, just "no match"); booleans are not numbers. +""" + +from __future__ import annotations + +import math +from typing import Any, Callable, List, Optional, Sequence, Tuple + +from . import validation +from .errors import validation_error + +MAX_IN = 64 + + +def _num(v: Any) -> bool: + return isinstance(v, (int, float)) and not isinstance(v, bool) + + +def _comparable(a: Any, b: Any) -> bool: + if _num(a) and _num(b): + return True + return isinstance(a, str) and isinstance(b, str) + + +def _eq(a: Any, b: Any) -> bool: + if isinstance(a, bool) or isinstance(b, bool): + return isinstance(a, bool) and isinstance(b, bool) and a == b + if _num(a) and _num(b): + return a == b + return type(a) is type(b) and a == b + + +def _make_test(op: str, value: Any) -> Callable[[bool, Any], bool]: + """Return ``test(present, actual)``.""" + if op == "exists": + want = bool(value) + return lambda present, actual: present == want + if op == "eq": + return lambda present, actual: present and _eq(actual, value) + if op == "ne": + return lambda present, actual: (not present) or not _eq(actual, value) + if op in ("lt", "lte", "gt", "gte"): + cmp = {"lt": lambda a, b: a < b, "lte": lambda a, b: a <= b, + "gt": lambda a, b: a > b, "gte": lambda a, b: a >= b}[op] + return lambda present, actual: present and _comparable(actual, value) and cmp(actual, value) + if op == "in": + return lambda present, actual: present and any(_eq(actual, v) for v in value) + if op == "nin": + return lambda present, actual: (not present) or not any(_eq(actual, v) for v in value) + if op == "prefix": + return lambda present, actual: present and isinstance(actual, str) and actual.startswith(value) + if op == "contains": + return lambda present, actual: present and isinstance(actual, str) and value in actual + raise AssertionError(op) + + +def compile_filter(spec: Optional[dict], path: str = "$.filter" + ) -> Tuple[Optional[Callable[[Any], bool]], Optional[List[str]]]: + """Return ``(node_predicate or None, node_types or None)`` for the library.""" + if not spec: + return None, None + types = spec.get("types") + tests: List[Tuple[str, Callable[[bool, Any], bool]]] = [] + for i, cond in enumerate(spec.get("where", [])): + where = f"{path}.where[{i}]" + op = cond["op"] + value = cond.get("value") + if op == "exists": + if "value" in cond and not isinstance(value, bool): + raise validation_error(where + ".value", "must be a boolean") + value = True if "value" not in cond else value + elif op in ("in", "nin"): + if not isinstance(value, list) or len(value) > MAX_IN: + raise validation_error(where + ".value", f"must be an array of at most {MAX_IN} scalars") + for j, item in enumerate(value): + validation.scalar_property_value(item, f"{where}.value[{j}]") + else: + if "value" not in cond: + raise validation_error(where, "missing 'value'") + validation.scalar_property_value(value, where + ".value") + if op in ("lt", "lte", "gt", "gte") and not (_num(value) or isinstance(value, str)): + raise validation_error(where + ".value", "ordering needs a number or a string") + if op in ("prefix", "contains") and not isinstance(value, str): + raise validation_error(where + ".value", "must be a string") + tests.append((cond["field"], _make_test(op, value))) + + if not tests: + return None, (list(types) if types is not None else None) + + def predicate(node: Any) -> bool: + props = node.properties + for field_name, test in tests: + actual = props.get(field_name) + # null and absent are the same thing (see ops.py) + if not test(actual is not None, actual): + return False + return True + + return predicate, (list(types) if types is not None else None) diff --git a/isongraph-vector-server/isongraph_vector_server/httpd.py b/isongraph-vector-server/isongraph_vector_server/httpd.py new file mode 100644 index 0000000..c2e9615 --- /dev/null +++ b/isongraph-vector-server/isongraph_vector_server/httpd.py @@ -0,0 +1,416 @@ +"""A small, bounded HTTP/1.1 server on the standard library. + +What it does to stay correct under load and abuse: + +* A fixed worker pool (``max_workers``) and a bounded accept queue + (``accept_queue``). A connection that arrives when both are full is answered + ``503`` immediately and closed; there is no unbounded thread-per-connection. +* Every socket operation has a timeout (``socket_timeout_seconds``), body reads + also have a total deadline, so a slow-loris client cannot hold a worker. +* The request line and headers are bounded; ``Transfer-Encoding`` (chunked + bodies) is refused so there is a single, unambiguous framing (Content-Length); + duplicate or malformed Content-Length is refused; the body is read only after + the declared length passes the size limit. +* Graceful shutdown: stop accepting, let in-flight requests finish, close idle + keep-alive connections. + +It is *not* a hardened internet-facing server: no TLS, no HTTP/2, no +protection against many slow connections beyond the limits above. Run it behind a +TLS-terminating reverse proxy (see README). +""" + +from __future__ import annotations + +import io +import json +import logging +import select +import socket +import socketserver +import threading +import time +from concurrent.futures import ThreadPoolExecutor +from dataclasses import dataclass, field +from http.server import BaseHTTPRequestHandler +from typing import Any, Dict, Iterator, List, Optional, Tuple + +from .config import ServerConfig +from .errors import ApiError +from .logs import log + + +@dataclass +class Request: + method: str + target: str # raw request-target (path + query) + headers: List[Tuple[str, str]] # (lowercased name, value), in arrival order + body: bytes + peer: str + + def header_all(self, name: str) -> List[str]: + name = name.lower() + return [v for k, v in self.headers if k == name] + + def header(self, name: str) -> Optional[str]: + vals = self.header_all(name) + return vals[0] if vals else None + + +@dataclass +class Response: + status: int + headers: Dict[str, str] = field(default_factory=dict) + body: bytes = b"" + stream: Optional[Iterator[bytes]] = None + length: Optional[int] = None # required with ``stream`` + + @staticmethod + def json(status: int, obj: Any, headers: Optional[Dict[str, str]] = None) -> "Response": + data = json.dumps(obj, separators=(",", ":"), allow_nan=False).encode("utf-8") + h = {"Content-Type": "application/json"} + h.update(headers or {}) + return Response(status, h, data) + + +_REASONS = { + 200: "OK", 201: "Created", 204: "No Content", 400: "Bad Request", 401: "Unauthorized", + 403: "Forbidden", 404: "Not Found", 405: "Method Not Allowed", 409: "Conflict", + 411: "Length Required", 413: "Payload Too Large", 414: "URI Too Long", + 415: "Unsupported Media Type", 429: "Too Many Requests", 431: "Request Header Fields Too Large", + 500: "Internal Server Error", 501: "Not Implemented", 502: "Bad Gateway", + 503: "Service Unavailable", 504: "Gateway Timeout", +} + +BODY_METHODS = ("POST", "PUT", "PATCH") +MAX_TARGET = 4096 +BODY_TOTAL_SECONDS = 30.0 + + +class _DeadlineReader(io.RawIOBase): + """Socket reader with a *total* deadline on top of the per-operation timeout. + + A per-``recv`` timeout alone lets a client that drips one byte every few seconds hold a + worker for as long as it likes. ``deadline`` (absolute, monotonic) caps the whole read. + ``nonblocking`` lets the handler ask "is anything buffered or waiting?" without blocking. + """ + + def __init__(self, sock: socket.socket, per_op: float): + self.sock = sock + self.per_op = per_op + self.deadline: Optional[float] = None + self.nonblocking = False + + def readable(self) -> bool: + return True + + def readinto(self, buf): + if self.nonblocking: + self.sock.setblocking(False) + try: + return self.sock.recv_into(buf) + except BlockingIOError: + return None + timeout = self.per_op + if self.deadline is not None: + left = self.deadline - time.monotonic() + if left <= 0: + raise TimeoutError("request read deadline exceeded") + timeout = min(timeout, left) + self.sock.settimeout(timeout) + return self.sock.recv_into(buf) + + +HEADER_TOTAL_SECONDS = 15.0 + + +def make_handler(app: Any, cfg: ServerConfig): + """Build a request-handler class bound to ``app`` (anything with ``handle(Request) -> Response``).""" + + class Handler(BaseHTTPRequestHandler): + protocol_version = "HTTP/1.1" + timeout = cfg.socket_timeout_seconds + server_version = "igv" + sys_version = "" + _served = 0 + + # -- plumbing ----------------------------------------------------------- + + def setup(self): + self.connection = self.request + self.connection.settimeout(self.timeout) + self._raw = _DeadlineReader(self.connection, self.timeout) + self.rfile = io.BufferedReader(self._raw, 65536) + self.wfile = self.connection.makefile("wb", 0) + + def _await_next_request(self) -> bool: + """Wait (politely, in short slices) for the next request to start. + + Returns False when the connection should end: peer closed, idle too long, or the + server is shutting down. Polling in slices is what lets a graceful shutdown + close idle keep-alive connections promptly on every platform. + """ + end = time.monotonic() + cfg.socket_timeout_seconds + self._raw.deadline = None + while True: + if self.server.stopping: + return False + self._raw.nonblocking = True + try: + have = self.rfile.peek(1) + except OSError: + return False + finally: + self._raw.nonblocking = False + if have: + return True + try: + readable, _, _ = select.select([self.connection], [], [], 0.25) + except (OSError, ValueError): + return False + if readable: + return True # data, or EOF (readline will then return b"") + if time.monotonic() > end: + return False + + def log_message(self, *args, **kwargs): # we log ourselves, structured + pass + + def version_string(self) -> str: + return "igv" + + def send_error(self, code, message=None, explain=None): + # Raised by the stdlib parser (bad request line, header too long, ...). + self.request_version = "HTTP/1.1" # never answer in HTTP/0.9 (no status line) + err = ApiError(message or _REASONS.get(code, "error"), status=code, + code="bad_request" if code < 500 else "server_error") + self._send(Response.json(code, err.to_body("-")), close=True) + + def handle_one_request(self): + self.request_version = "HTTP/1.1" # defaults, so error replies work before parse_request + self.command = None + self.requestline = "" + self.close_connection = True if self.close_connection else False + try: + if not self._await_next_request(): + self.close_connection = True + return + self._raw.deadline = time.monotonic() + HEADER_TOTAL_SECONDS + self.raw_requestline = self.rfile.readline(MAX_TARGET + 64) + if not self.raw_requestline: + self.close_connection = True + return + if len(self.raw_requestline) > MAX_TARGET + 63: + self.send_error(414) + return + if not self.parse_request(): + return + self.server.mark_busy(self.connection, True) + try: + self._serve() + finally: + self.server.mark_busy(self.connection, False) + self.wfile.flush() + except (socket.timeout, TimeoutError): + self.close_connection = True + except (ConnectionError, OSError): + self.close_connection = True + + # -- one request ------------------------------------------------------------ + + def _serve(self) -> None: + self._served += 1 + if self.server.stopping or self._served >= cfg.max_requests_per_connection: + self.close_connection = True + headers = [(k.lower(), v) for k, v in self.headers.items()] + if sum(len(k) + len(v) + 4 for k, v in headers) > cfg.max_header_bytes: + self.send_error(431, "headers too large") + return + if any(k == "transfer-encoding" for k, _ in headers): + self._send(Response.json(400, ApiError( + "Transfer-Encoding is not supported; send Content-Length", + code="unsupported_transfer_encoding", status=400).to_body("-")), close=True) + return + lengths = [v for k, v in headers if k == "content-length"] + body = b"" + if len(lengths) > 1 or (lengths and not lengths[0].strip().isdigit()): + self._send(Response.json(400, ApiError( + "invalid Content-Length", code="invalid_request", status=400).to_body("-")), close=True) + return + length = int(lengths[0]) if lengths else 0 + if self.command in BODY_METHODS and not lengths: + self._send(Response.json(411, ApiError( + "Content-Length is required", code="length_required", status=411).to_body("-")), close=True) + return + if length > cfg.max_body_bytes: + self._send(Response.json(413, ApiError( + f"body larger than {cfg.max_body_bytes} bytes", code="payload_too_large", + status=413).to_body("-")), close=True) + self._drain(length) + return + if length: + self._raw.deadline = time.monotonic() + BODY_TOTAL_SECONDS + body = self._read_body(length) + if body is None: + self.close_connection = True + return + req = Request(self.command, self.path, headers, body, self.client_address[0]) + try: + resp = app.handle(req) + except Exception: # backstop: App.handle should never raise + logging.getLogger("igv").exception("unhandled error in app.handle") + resp = Response.json(500, ApiError("internal error").to_body("-")) + self._send(resp, close=self.close_connection) + + def _drain(self, length: int) -> None: + """After refusing a body, read (and discard) a bounded amount of it, briefly. + + Closing a socket that still has unread request data makes the OS send a reset, + and the client can lose the response we just wrote; a short lingering read lets + an honest-but-oversize client actually see the 413. + """ + remaining = min(length, 4 * 1024 * 1024) + end = time.monotonic() + 2.0 + try: + while remaining > 0 and time.monotonic() < end: + data = self.rfile.read(min(remaining, 65536)) + if not data: + break + remaining -= len(data) + except OSError: + pass + + def _read_body(self, length: int) -> Optional[bytes]: + chunks = [] + remaining = length + end = time.monotonic() + BODY_TOTAL_SECONDS + while remaining: + if time.monotonic() > end: + return None + data = self.rfile.read(min(remaining, 65536)) + if not data: + return None + chunks.append(data) + remaining -= len(data) + return b"".join(chunks) + + def _send(self, resp: Response, close: bool = False) -> None: + if resp.stream is not None: + length = resp.length + else: + length = len(resp.body) + self.send_response_only(resp.status, _REASONS.get(resp.status, "Status")) + self.send_header("Content-Length", str(length)) + self.send_header("Cache-Control", "no-store") + self.send_header("X-Content-Type-Options", "nosniff") + for k, v in resp.headers.items(): + self.send_header(k, v) + if close or self.close_connection: + self.send_header("Connection", "close") + self.close_connection = True + self.end_headers() + if resp.stream is not None: + sent = 0 + for chunk in resp.stream: + self.wfile.write(chunk) + sent += len(chunk) + if sent != length: # the file changed under us: do not lie + self.close_connection = True + elif resp.body: + self.wfile.write(resp.body) + if (close or self.close_connection) and resp.status >= 400 and resp.stream is None: + self._linger() + + def _linger(self, max_bytes: int = 1 << 17, budget: float = 0.25) -> None: + """Soak up whatever the client is still sending before closing (see _drain).""" + end = time.monotonic() + budget + got = 0 + try: + self.connection.settimeout(0.05) + while got < max_bytes and time.monotonic() < end: + data = self.connection.recv(65536) + if not data: + break + got += len(data) + except OSError: + pass + + def _method(self): + # The stdlib dispatches on do_<METHOD>; route them all to _serve via + # handle_one_request. Unknown methods get a 501 from parse_request. + pass + + for m in ("GET", "POST", "PUT", "PATCH", "DELETE", "HEAD", "OPTIONS"): + setattr(Handler, f"do_{m}", Handler._method) + return Handler + + +class BoundedHTTPServer(socketserver.TCPServer): + allow_reuse_address = True + + def __init__(self, address, app: Any, cfg: ServerConfig): + self.cfg = cfg + self.stopping = False + self.request_queue_size = cfg.accept_queue + self._pool = ThreadPoolExecutor(max_workers=cfg.max_workers, thread_name_prefix="igv-http") + self._slots = threading.BoundedSemaphore(cfg.max_workers + cfg.accept_queue) + self._conn_lock = threading.Lock() + self._conns: Dict[socket.socket, bool] = {} # socket -> currently in a request + super().__init__(address, make_handler(app, cfg), bind_and_activate=True) + + # connection tracking, for shutdown -------------------------------------------------- + def mark_busy(self, sock: socket.socket, busy: bool) -> None: + with self._conn_lock: + if sock in self._conns: + self._conns[sock] = busy + + def process_request(self, request, client_address): + if not self._slots.acquire(blocking=False): + try: # overloaded: tell the client, cheaply, and drop the connection + request.settimeout(1.0) + body = b'{"error":{"code":"overloaded","message":"server is at capacity","request_id":"-"}}' + request.sendall(b"HTTP/1.1 503 Service Unavailable\r\nContent-Type: application/json\r\n" + b"Retry-After: 1\r\nConnection: close\r\nContent-Length: " + + str(len(body)).encode() + b"\r\n\r\n" + body) + request.settimeout(0.2) + request.recv(65536) # take the client's request off the wire: closing with + except OSError: # unread data would reset the connection under the 503 + pass + self.shutdown_request(request) + return + with self._conn_lock: + self._conns[request] = False + self._pool.submit(self._run, request, client_address) + + def _run(self, request, client_address) -> None: + try: + self.finish_request(request, client_address) + except Exception: + self.handle_error(request, client_address) + finally: + with self._conn_lock: + self._conns.pop(request, None) + self.shutdown_request(request) + self._slots.release() + + def handle_error(self, request, client_address): # keep stdlib's traceback print off stderr + logging.getLogger("igv").debug("connection error", exc_info=True) + + def stop(self, grace: float = 10.0) -> None: + """Stop accepting, let in-flight requests finish, drop idle connections.""" + self.stopping = True + self.shutdown() # ends serve_forever() + self.server_close() # closes the listening socket + deadline = time.monotonic() + grace + while time.monotonic() < deadline: + with self._conn_lock: + idle = [s for s, busy in self._conns.items() if not busy] + alive = bool(self._conns) + for s in idle: + try: + s.shutdown(socket.SHUT_RDWR) + except OSError: + pass + if not alive: + break + time.sleep(0.05) + self._pool.shutdown(wait=False, cancel_futures=True) diff --git a/isongraph-vector-server/isongraph_vector_server/limits.py b/isongraph-vector-server/isongraph_vector_server/limits.py new file mode 100644 index 0000000..71e23ea --- /dev/null +++ b/isongraph-vector-server/isongraph_vector_server/limits.py @@ -0,0 +1,180 @@ +"""Rate limiting, bounded concurrency and the graph read/write lock.""" + +from __future__ import annotations + +import threading +import time +from collections import deque +from contextlib import contextmanager +from typing import Callable, Deque, Dict, Optional + +from .errors import Overloaded, RateLimited + + +class TokenBucket: + """Classic token bucket. ``take(cost)`` returns 0.0 on success, else the + number of seconds until ``cost`` tokens will be available.""" + + def __init__(self, rate: float, burst: int, clock: Callable[[], float] = time.monotonic): + self.rate = float(rate) + self.burst = float(burst) + self._clock = clock + self._tokens = float(burst) + self._stamp = clock() + self._lock = threading.Lock() + + def take(self, cost: float = 1.0) -> float: + cost = min(cost, self.burst) # a request bigger than the bucket is charged the bucket + with self._lock: + now = self._clock() + self._tokens = min(self.burst, self._tokens + (now - self._stamp) * self.rate) + self._stamp = now + if self._tokens >= cost: + self._tokens -= cost + return 0.0 + return (cost - self._tokens) / self.rate + + def reconfigure(self, rate: float, burst: int) -> None: + with self._lock: + self.rate, self.burst = float(rate), float(burst) + self._tokens = min(self._tokens, self.burst) + + +class SlidingWindow: + """Count of events per key in the last ``window`` seconds (failed-auth throttle).""" + + def __init__(self, limit: int, window: float = 60.0, clock=time.monotonic, + max_keys: int = 10_000): + self.limit = limit + self.window = window + self._clock = clock + self._events: Dict[str, Deque[float]] = {} + self._lock = threading.Lock() + self._max_keys = max_keys + + def blocked(self, key: str) -> bool: + now = self._clock() + with self._lock: + q = self._events.get(key) + if not q: + return False + while q and now - q[0] > self.window: + q.popleft() + return len(q) >= self.limit + + def record(self, key: str) -> None: + now = self._clock() + with self._lock: + if len(self._events) >= self._max_keys and key not in self._events: + # Bounded memory under an address-spraying attack: drop the oldest key. + self._events.pop(next(iter(self._events))) + q = self._events.setdefault(key, deque()) + q.append(now) + while q and now - q[0] > self.window: + q.popleft() + + +class Gate: + """A counting semaphore that gives up after ``wait`` seconds instead of queueing forever.""" + + def __init__(self, size: int): + self.size = size + self._sem = threading.BoundedSemaphore(size) + self._lock = threading.Lock() + self.in_flight = 0 + + @contextmanager + def enter(self, wait: float, error: Optional[Exception] = None): + if not self._sem.acquire(timeout=wait): + raise error or Overloaded("server is at its concurrency limit; retry shortly", + headers={"Retry-After": "1"}) + with self._lock: + self.in_flight += 1 + try: + yield + finally: + with self._lock: + self.in_flight -= 1 + self._sem.release() + + +class LockTimeout(Exception): + pass + + +class RWLock: + """Readers-writer lock; writers are preferred so a stream of searches cannot + starve an upsert. Acquisition takes a timeout and raises :class:`LockTimeout`. + + Not re-entrant: a thread must not take it twice. + """ + + def __init__(self): + self._cond = threading.Condition(threading.Lock()) + self._readers = 0 + self._writer = False + self._writers_waiting = 0 + + def acquire_read(self, timeout: Optional[float] = None) -> None: + end = None if timeout is None else time.monotonic() + timeout + with self._cond: + while self._writer or self._writers_waiting: + if not self._wait(end): + raise LockTimeout() + self._readers += 1 + + def release_read(self) -> None: + with self._cond: + self._readers -= 1 + if self._readers == 0: + self._cond.notify_all() + + def acquire_write(self, timeout: Optional[float] = None) -> None: + end = None if timeout is None else time.monotonic() + timeout + with self._cond: + self._writers_waiting += 1 + try: + while self._writer or self._readers: + if not self._wait(end): + raise LockTimeout() + finally: + self._writers_waiting -= 1 + if self._writers_waiting == 0 and not self._writer: + self._cond.notify_all() + self._writer = True + + def release_write(self) -> None: + with self._cond: + self._writer = False + self._cond.notify_all() + + def _wait(self, end: Optional[float]) -> bool: + if end is None: + self._cond.wait() + return True + remaining = end - time.monotonic() + if remaining <= 0: + return False + self._cond.wait(remaining) + return True + + @contextmanager + def read(self, timeout: Optional[float] = None): + self.acquire_read(timeout) + try: + yield + finally: + self.release_read() + + @contextmanager + def write(self, timeout: Optional[float] = None): + self.acquire_write(timeout) + try: + yield + finally: + self.release_write() + + +def rate_limited(retry_after: float) -> RateLimited: + secs = max(1, int(retry_after + 0.999)) + return RateLimited("rate limit exceeded", headers={"Retry-After": str(secs)}) diff --git a/isongraph-vector-server/isongraph_vector_server/logs.py b/isongraph-vector-server/isongraph_vector_server/logs.py new file mode 100644 index 0000000..b3b3e0a --- /dev/null +++ b/isongraph-vector-server/isongraph_vector_server/logs.py @@ -0,0 +1,71 @@ +"""Structured (JSON-lines) logging that cannot carry secrets. + +Two layers: the request path never hands a header or body to the logger (it +logs method, route *template*, status, latency, tenant id and request id), +and a filter redacts anything shaped like an API key or bearer credential +from every record, including exception text, as a backstop. +""" + +from __future__ import annotations + +import json +import logging +import re +import sys +import time +from typing import Any + +_KEY = re.compile(r"igv_[a-z0-9-]{2,32}\.[0-9a-f]{16}\.[A-Za-z0-9_-]{20,}") +_BEARER = re.compile(r"(?i)(bearer|authorization[\"']?\s*[:=])\s*[^\s,;\"']+") + +LOGGER_NAME = "igv" + + +def redact(text: str) -> str: + text = _KEY.sub("igv_[REDACTED]", text) + return _BEARER.sub(lambda m: m.group(1) + " [REDACTED]", text) + + +class RedactingFilter(logging.Filter): + def filter(self, record: logging.LogRecord) -> bool: + record.msg = redact(str(record.getMessage())) + record.args = () + fields = getattr(record, "fields", None) + if fields: + record.fields = {k: redact(v) if isinstance(v, str) else v for k, v in fields.items()} + if record.exc_info and record.exc_info[1] is not None: + # Render the traceback ourselves so it passes through redaction too. + import traceback + record.exc_text = redact("".join(traceback.format_exception(*record.exc_info))) + record.exc_info = None + return True + + +class JsonFormatter(logging.Formatter): + def format(self, record: logging.LogRecord) -> str: + out = { + "ts": round(time.time(), 3), + "level": record.levelname.lower(), + "msg": record.getMessage(), + } + out.update(getattr(record, "fields", {}) or {}) + if record.exc_text: + out["exc"] = record.exc_text + return json.dumps(out, separators=(",", ":"), default=str) + + +def setup_logging(level: str = "INFO", stream: Any = None) -> logging.Logger: + logger = logging.getLogger(LOGGER_NAME) + for h in list(logger.handlers): + logger.removeHandler(h) + handler = logging.StreamHandler(stream or sys.stderr) + handler.setFormatter(JsonFormatter()) + handler.addFilter(RedactingFilter()) + logger.addHandler(handler) + logger.setLevel(getattr(logging, str(level).upper(), logging.INFO)) + logger.propagate = False + return logger + + +def log(level: int, msg: str, **fields: Any) -> None: + logging.getLogger(LOGGER_NAME).log(level, msg, extra={"fields": fields}) diff --git a/isongraph-vector-server/isongraph_vector_server/metrics.py b/isongraph-vector-server/isongraph_vector_server/metrics.py new file mode 100644 index 0000000..38ddab4 --- /dev/null +++ b/isongraph-vector-server/isongraph_vector_server/metrics.py @@ -0,0 +1,78 @@ +"""Aggregate metrics in Prometheus text format. + +Deliberately has no tenant or graph labels: the global ``/metrics`` endpoint +is protected by an operator token, but even so a per-tenant label would put +tenant ids (and by extension their activity) into whatever scrapes it. Per +tenant numbers are available to that tenant itself on ``GET /v1/usage``. +""" + +from __future__ import annotations + +import threading +from typing import Dict, Tuple + +BUCKETS = (0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0) + + +class Metrics: + def __init__(self): + self._lock = threading.Lock() + self._counters: Dict[Tuple[str, Tuple[Tuple[str, str], ...]], float] = {} + self._gauges: Dict[str, float] = {} + self._hist: Dict[str, list] = {} # route -> [bucket counts..., +Inf, sum, count] + + def inc(self, name: str, value: float = 1.0, **labels: str) -> None: + key = (name, tuple(sorted(labels.items()))) + with self._lock: + self._counters[key] = self._counters.get(key, 0.0) + value + + def gauge(self, name: str, value: float) -> None: + with self._lock: + self._gauges[name] = value + + def observe(self, route: str, seconds: float) -> None: + with self._lock: + h = self._hist.get(route) + if h is None: + h = self._hist[route] = [0] * (len(BUCKETS) + 1) + [0.0, 0] + for i, bound in enumerate(BUCKETS): + if seconds <= bound: + h[i] += 1 + h[len(BUCKETS)] += 1 + h[-2] += seconds + h[-1] += 1 + + def counter(self, name: str, **labels: str) -> float: + with self._lock: + return self._counters.get((name, tuple(sorted(labels.items()))), 0.0) + + def render(self) -> str: + lines = [] + with self._lock: + counters = dict(self._counters) + gauges = dict(self._gauges) + hist = {k: list(v) for k, v in self._hist.items()} + seen = set() + for (name, labels), value in sorted(counters.items()): + if name not in seen: + lines.append(f"# TYPE {name} counter") + seen.add(name) + lab = ",".join(f'{k}="{_esc(v)}"' for k, v in labels) + lines.append(f"{name}{{{lab}}} {value:g}" if lab else f"{name} {value:g}") + for name, value in sorted(gauges.items()): + lines.append(f"# TYPE {name} gauge") + lines.append(f"{name} {value:g}") + if hist: + lines.append("# TYPE igv_request_seconds histogram") + for route, h in sorted(hist.items()): + r = _esc(route) + for i, bound in enumerate(BUCKETS): + lines.append(f'igv_request_seconds_bucket{{route="{r}",le="{bound:g}"}} {h[i]}') + lines.append(f'igv_request_seconds_bucket{{route="{r}",le="+Inf"}} {h[len(BUCKETS)]}') + lines.append(f'igv_request_seconds_sum{{route="{r}"}} {h[-2]:g}') + lines.append(f'igv_request_seconds_count{{route="{r}"}} {h[-1]}') + return "\n".join(lines) + "\n" + + +def _esc(value: str) -> str: + return value.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n") diff --git a/isongraph-vector-server/isongraph_vector_server/names.py b/isongraph-vector-server/isongraph_vector_server/names.py new file mode 100644 index 0000000..db1527b --- /dev/null +++ b/isongraph-vector-server/isongraph_vector_server/names.py @@ -0,0 +1,104 @@ +"""Names, identifiers and the only code that turns a name into a path. + +Nothing a client sends is ever used as a path component directly. Tenant ids +come from the authenticated key (and were validated when the key was made), +graph names are matched against a whitelist regular expression, and the +directory is derived from the *validated* name with a fixed prefix. The +whitelist is lowercase-only on purpose: case-insensitive filesystems (Windows, +macOS) would otherwise map "Foo" and "foo" to one directory, i.e. let two +names alias each other. +""" + +from __future__ import annotations + +import os +import re +from pathlib import Path +from typing import Union + +TENANT_RE = re.compile(r"^[a-z0-9][a-z0-9-]{0,30}[a-z0-9]$") +GRAPH_RE = re.compile(r"^[a-z0-9][a-z0-9_-]{0,62}$") +# Node types, relationship types and property names. These become ISON block +# names / field names, which the format cannot quote, so keep them tame. +TYPE_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]{0,63}$") +PROP_RE = TYPE_RE +# String node ids. ASCII on purpose: ids appear in URLs, logs and ISON +# references. No whitespace, ':' or '|' (the library forbids them too). +ID_RE = re.compile(r"^[A-Za-z0-9_][A-Za-z0-9_.@+=~-]{0,127}$") +CANON_INT_RE = re.compile(r"^-?(0|[1-9][0-9]{0,15})$") +GENERATION_RE = re.compile(r"^[0-9]{6,12}$") +REQUEST_ID_RE = re.compile(r"^[A-Za-z0-9_.-]{8,64}$") + +MAX_INT_ID = 2 ** 53 - 1 + +# Files a snapshot generation may contain; the replication endpoints serve +# nothing else. +GENERATION_FILES = ("graph.isong", "graph.isong.embeddings.json", "MANIFEST.json") + + +def is_tenant_id(value: object) -> bool: + return isinstance(value, str) and TENANT_RE.match(value) is not None + + +def is_graph_name(value: object) -> bool: + return isinstance(value, str) and GRAPH_RE.match(value) is not None + + +def normalize_id(value: object): + """Return the canonical node id (int or str) or raise ``ValueError``. + + A string that spells a canonical integer ("5", "-12") *is* that integer: + the underlying library treats 5 and "5" as the same node (it refuses the + second as a twin), so the API says so up front instead of erroring later. + bool is not an id. Leading zeros keep a string a string ("007"). + """ + if isinstance(value, bool): + raise ValueError("id must be an integer or a string") + if isinstance(value, int): + if abs(value) > MAX_INT_ID: + raise ValueError("integer id out of range") + return value + if isinstance(value, str): + if CANON_INT_RE.match(value): + number = int(value) + if abs(number) > MAX_INT_ID: + raise ValueError("integer id out of range") + return number + if ID_RE.match(value): + return value + raise ValueError("id must match [A-Za-z0-9_][A-Za-z0-9_.@+=~-]{0,127}") + raise ValueError("id must be an integer or a string") + + +def graph_dirname(name: str) -> str: + if not is_graph_name(name): + raise ValueError("invalid graph name") + return "g-" + name + + +def contained(base: Path, candidate: Path) -> Path: + """Return ``candidate`` resolved, or raise if it is not inside ``base``. + + Defence in depth: the whitelist above already rules out separators and + dots, so this should never fire. It exists so that a future change which + loosens a regex fails closed instead of opening a traversal. + """ + base_r = Path(os.path.realpath(base)) + cand_r = Path(os.path.realpath(candidate)) + try: + cand_r.relative_to(base_r) + except ValueError: + raise PermissionError("path escapes its base directory") from None + return cand_r + + +def tenant_dir(data_dir: Union[str, Path], tenant_id: str) -> Path: + if not is_tenant_id(tenant_id): + raise ValueError("invalid tenant id") + base = Path(data_dir) / "tenants" + return contained(base, base / tenant_id) + + +def graph_dir(data_dir: Union[str, Path], tenant_id: str, name: str) -> Path: + base = tenant_dir(data_dir, tenant_id) / "graphs" + return contained(base, base / graph_dirname(name)) diff --git a/isongraph-vector-server/isongraph_vector_server/ops.py b/isongraph-vector-server/isongraph_vector_server/ops.py new file mode 100644 index 0000000..79f74be --- /dev/null +++ b/isongraph-vector-server/isongraph_vector_server/ops.py @@ -0,0 +1,891 @@ +"""Graph operations: everything a handler does to a graph, after validation. + +Rules every function here follows: + +* Encoder calls happen *before* a graph lock is taken (a slow embedding + endpoint must not block readers of the graph). +* A batch is validated completely, under the write lock, before the first + mutation: quota, dimension, endpoint existence and id-twin conflicts are + checked up front so a rejected batch changes nothing. +* Quota is reserved atomically on the tenant before applying and reconciled + after, so two concurrent writers (even into different graphs of one tenant) + cannot together exceed the limit. +* Null-valued properties are never stored (the ISON text format cannot tell + null from absent, so the service defines them to be the same). +""" + +from __future__ import annotations + +import base64 +import bisect +import importlib.util +import json +import math +import time +from contextlib import contextmanager +from dataclasses import dataclass +from typing import Any, Dict, Iterator, List, Optional, Sequence, Tuple + +from isongraph_vector import (UNKNOWN_MODELS, DimensionMismatchError, EmbeddingError, + NoEncoderError, SemanticGraph, graph_digest) +from ison_graph import (Direction, DuplicateEdgeError, DuplicateNodeError, EdgeNotFoundError, + GraphError, NodeNotFoundError) + +from . import names +from .config import Quotas +from .engine import Engine, GraphHandle, OpContext, TenantRuntime +from .errors import (BadRequest, Conflict, NotFound, QuotaExceeded, Unavailable, validation_error) +from .filters import compile_filter +from .limits import LockTimeout +from .storage import GraphMeta + +DEFAULT_EMBED_FIELDS = ["name", "description", "content", "text"] +_DIRECTIONS = {"out": Direction.OUT, "in": Direction.IN, "both": Direction.BOTH} + + +@dataclass +class OpEnv: + engine: Engine + rt: TenantRuntime + quotas: Quotas + ctx: OpContext + + @property + def encoder(self): + return self.engine.encoder + + +def busy() -> Unavailable: + return Unavailable("graph is busy; retry", code="busy", headers={"Retry-After": "1"}) + + +@contextmanager +def read_locked(env: OpEnv, handle: GraphHandle): + try: + handle.lock.acquire_read(env.ctx.lock_wait()) + except LockTimeout: + raise busy() from None + try: + if handle.deleted or handle.graph is None: + raise NotFound("graph not found") + yield handle.graph + finally: + handle.lock.release_read() + + +@contextmanager +def write_locked(env: OpEnv, handle: GraphHandle): + try: + handle.lock.acquire_write(env.ctx.lock_wait()) + except LockTimeout: + raise busy() from None + try: + if handle.deleted or handle.graph is None: + raise NotFound("graph not found") + yield handle.graph + finally: + handle.lock.release_write() + + +# -- serialization helpers --------------------------------------------------------------------- + + +def ref_json(ref: Tuple[str, Any]) -> Dict[str, Any]: + return {"type": ref[0], "id": ref[1]} + + +def live_props(props: Dict[str, Any]) -> Dict[str, Any]: + return {k: v for k, v in props.items() if v is not None} + + +def node_json(node: Any, include_props: bool = True) -> Dict[str, Any]: + out = {"type": node.type, "id": node.id} + if include_props: + out["properties"] = live_props(node.properties) + return out + + +def edge_json(edge: Any) -> Dict[str, Any]: + return {"rel": edge.rel_type, "source": ref_json(edge.source), "target": ref_json(edge.target), + "properties": live_props(edge.properties)} + + +def _score(value: float) -> float: + return float(value) if math.isfinite(value) else 0.0 + + +def _id_kind(value: Any) -> int: + return 0 if isinstance(value, int) else 1 + + +def node_sort_key(ref: Tuple[str, Any]): + return (ref[0], _id_kind(ref[1]), ref[1]) + + +def encode_cursor(key: Sequence[Any]) -> str: + raw = json.dumps(list(key), separators=(",", ":")).encode("utf-8") + return base64.urlsafe_b64encode(raw).decode("ascii").rstrip("=") + + +def decode_cursor(cursor: Optional[str], arity: int) -> Optional[tuple]: + """Cursors are plain keyset positions inside the caller's own data; a bad one is a 400.""" + if cursor is None: + return None + try: + raw = base64.urlsafe_b64decode(cursor + "=" * (-len(cursor) % 4)) + data = json.loads(raw) + if not isinstance(data, list): + raise ValueError + return tuple(data) + except (ValueError, TypeError): + raise BadRequest("invalid cursor", code="invalid_cursor") from None + + +def page_limit(env: OpEnv, raw: Optional[str], default: int = 100) -> int: + if raw is None: + return min(default, env.engine.cfg.max_page) + try: + n = int(raw) + except ValueError: + raise BadRequest("limit must be an integer", code="invalid_request") from None + if n < 1 or n > env.engine.cfg.max_page: + raise BadRequest(f"limit must be between 1 and {env.engine.cfg.max_page}", code="invalid_request") + return n + + +# -- graphs ------------------------------------------------------------------------------------- + + +def _numpy_available() -> bool: + return importlib.util.find_spec("numpy") is not None + + +def graph_summary(handle: GraphHandle, graph: SemanticGraph) -> Dict[str, Any]: + meta = handle.meta + store = graph.embedding_store + return { + "name": handle.name, + "status": "ok" if not handle.degraded else "degraded", + "directed": True, + "mode": meta.mode, + "dimension": store.dimension if store.dimension is not None else meta.dimension, + "embed_fields": list(meta.embed_fields), + "ann": meta.ann, + "nodes": graph.node_count(), + "edges": graph.edge_count(), + "embeddings": store.count(), + "generation": handle.generation, + "unsaved_changes": handle.dirty, + "saved_at": handle.last_save_ts, + } + + +def create_graph(env: OpEnv, body: Dict[str, Any]) -> Dict[str, Any]: + q = env.quotas + if body.get("directed", True) is not True: + raise BadRequest("undirected graphs are not supported by the service " + "(the ISON file format cannot record directedness); " + "use direction 'both' in traversals instead", + code="unsupported") + emb = body.get("embedding") or {} + mode = emb.get("mode", "server") + dimension = emb.get("dimension") + if mode == "server": + if dimension is not None: + raise validation_error("$.embedding.dimension", + "not allowed in server mode (it follows the configured encoder)") + if env.encoder is None: + raise BadRequest("this server has no embedding encoder configured; " + "create the graph with embedding.mode='client'", + code="encoder_unavailable") + try: + dimension = int(env.encoder.dimension) + except Exception: + raise Unavailable("the embedding service is not reachable", code="encoder_failed") from None + else: + if dimension is None: + raise validation_error("$.embedding.dimension", "required in client mode") + if dimension > q.max_dimension: + raise QuotaExceeded(f"vector dimension {dimension} exceeds this tenant's limit of {q.max_dimension}") + ann = body.get("ann") + if ann and not _numpy_available(): + raise BadRequest("ann='ivf' needs numpy on the server", code="unsupported") + meta = GraphMeta( + name=body["name"], mode=mode, dimension=dimension, + embed_fields=list(body.get("embed_fields") or DEFAULT_EMBED_FIELDS), + ann=ann, ann_nprobe=body.get("ann_nprobe")) + handle = env.engine.create_graph(env.rt, q, meta) + with read_locked(env, handle) as graph: + return graph_summary(handle, graph) + + +def list_graphs(env: OpEnv, cursor: Optional[str], limit: int) -> Dict[str, Any]: + with env.rt.glock: + handles = sorted(env.rt.graphs.values(), key=lambda h: h.name) + after = decode_cursor(cursor, 1) + items = [] + next_cursor = None + for h in handles: + if after is not None and h.name <= str(after[0]): + continue + if len(items) >= limit: + next_cursor = encode_cursor([items[-1]["name"]]) + break + if h.quarantined is not None: + items.append({"name": h.name, "status": "quarantined"}) + continue + try: + with read_locked(env, h) as graph: + items.append(graph_summary(h, graph)) + except NotFound: + continue + return {"items": items, "next_cursor": next_cursor} + + +def get_graph(env: OpEnv, handle: GraphHandle) -> Dict[str, Any]: + with read_locked(env, handle) as graph: + out = graph_summary(handle, graph) + # The library's graph_digest of the last saved generation: the value a + # replica verifies. (Not computed from live memory: the file format + # cannot tell null from absent, so the two can differ by that alone.) + out["snapshot_digest"] = handle.manifest.graph_digest if handle.manifest else None + return out + + +def usage(env: OpEnv) -> Dict[str, Any]: + rt, q = env.rt, env.quotas + with rt.glock: + n_graphs = len(rt.graphs) + return { + "quotas": { + "max_graphs": q.max_graphs, "max_nodes": q.max_nodes, "max_edges": q.max_edges, + "max_dimension": q.max_dimension, "max_batch_items": q.max_batch_items, + "max_top_k": q.max_top_k, "max_hops": q.max_hops, + "rate_per_second": q.rate_per_second, "rate_burst": q.rate_burst, + "max_inflight": q.max_inflight, + }, + "usage": {"graphs": n_graphs, "nodes": rt.used_nodes, "edges": rt.used_edges}, + "requests": {"total": rt.requests, "rate_limited": rt.rate_limited}, + } + + +def delete_graph(env: OpEnv, name: str) -> None: + env.engine.delete_graph(env.rt, name, env.ctx) + + +def save_graph(env: OpEnv, handle: GraphHandle) -> Dict[str, Any]: + written = env.engine.save_graph(handle, env.ctx, force=True) + return {"saved": bool(written), "generation": handle.generation, + "graph_digest": handle.manifest.graph_digest if handle.manifest else None} + + +def maybe_sync_save(env: OpEnv, handle: GraphHandle) -> None: + """``durability = sync``: the write is on disk before the client hears about it.""" + if env.engine.cfg.durability == "sync": + try: + env.engine.save_graph(handle, env.ctx) + except Unavailable: + raise + except Exception: + raise Unavailable("the change was applied in memory but could not be saved", + code="persist_failed") from None + + +# -- vectors and embedding ----------------------------------------------------------------------- + + +def _check_vector(env: OpEnv, handle: GraphHandle, vec: Sequence[float], path: str) -> None: + if len(vec) > env.quotas.max_dimension: + raise validation_error(path, f"vector longer than this tenant's limit of {env.quotas.max_dimension}") + if handle.meta.mode == "client" and handle.meta.dimension is not None and len(vec) != handle.meta.dimension: + raise BadRequest(f"{path}: expected {handle.meta.dimension} components, got {len(vec)}", + code="dimension_mismatch") + + +def _encode(env: OpEnv, texts: List[str]) -> List[List[float]]: + enc = env.encoder + if enc is None: + raise Unavailable("no embedding encoder is configured", code="encoder_unavailable") + env.ctx.check() + vectors = enc.encode_batch(texts) + if len(vectors) != len(texts): + raise Unavailable("the embedding service returned the wrong number of vectors", code="encoder_failed") + dim = None + for v in vectors: + if dim is None: + dim = len(v) + if len(v) != dim or any(not math.isfinite(x) for x in v): + raise Unavailable("the embedding service returned an unusable vector", code="encoder_failed") + if dim and dim > env.quotas.max_dimension: + raise QuotaExceeded(f"vector dimension {dim} exceeds this tenant's limit") + return [list(map(float, v)) for v in vectors] + + +def _canon_id(node_id: Any) -> str: + """Spelling-independent form of an id: 7 and "007" are one reference to the library.""" + if isinstance(node_id, str) and node_id.lstrip("-").isdigit(): + return str(int(node_id)) + return str(node_id) + + +def _twin_conflict(graph: SemanticGraph, node_type: str, node_id: Any) -> bool: + """The library refuses 7 and "007" as the same node (one reference spells both).""" + if isinstance(node_id, str) and node_id.lstrip("-").isdigit(): + return graph.has_node(node_type, int(node_id)) + if isinstance(node_id, int): + return graph.has_node(node_type, str(node_id)) + return False + + +def _clean_props(props: Dict[str, Any], quotas: Quotas, path: str) -> Dict[str, Any]: + if len(props) > quotas.max_properties: + raise validation_error(path, f"at most {quotas.max_properties} properties per node") + return live_props(props) + + +# -- node writes --------------------------------------------------------------------------------------- + + +def _batch_limit(env: OpEnv, n: int, what: str) -> None: + if n > env.quotas.max_batch_items: + raise BadRequest(f"at most {env.quotas.max_batch_items} {what} per request", + code="batch_too_large") + + +def upsert_nodes(env: OpEnv, handle: GraphHandle, items: List[Dict[str, Any]]) -> Dict[str, Any]: + """Create or fully replace nodes. Server mode embeds text; client mode takes vectors.""" + _batch_limit(env, len(items), "nodes") + meta, q = handle.meta, env.quotas + seen = set() + prepared: List[Dict[str, Any]] = [] + to_encode: List[Tuple[int, str]] = [] + graph0 = handle.graph + for i, it in enumerate(items): + key = (it["type"], _canon_id(it["id"])) + if key in seen: + raise validation_error(f"$.nodes[{i}]", "the same node appears twice in this batch") + seen.add(key) + props = _clean_props(it.get("properties") or {}, q, f"$.nodes[{i}].properties") + vec, text = it.get("vector"), it.get("embed_text") + if meta.mode == "client": + if vec is None and text is not None: + raise validation_error(f"$.nodes[{i}].embed_text", + "a client-mode graph takes 'vector', not text to embed") + if vec is not None: + _check_vector(env, handle, vec, f"$.nodes[{i}].vector") + else: + if vec is not None: + raise validation_error(f"$.nodes[{i}].vector", + "a server-mode graph embeds text itself; send 'embed_text' " + "(or embed fields in properties), or use a client-mode graph") + text = text if text is not None else graph0.embed_text_for(props) + if text: + to_encode.append((i, text)) + prepared.append({"type": it["type"], "id": it["id"], "props": props, "vector": vec, + "text": text or "", "server": meta.mode == "server"}) + + if to_encode: + for (i, _), vec in zip(to_encode, _encode(env, [t for _, t in to_encode])): + prepared[i]["vector"] = vec + + created = updated = embedded = 0 + with write_locked(env, handle) as graph: + store = graph.embedding_store + pinned = store.dimension + dims = {len(p["vector"]) for p in prepared if p["vector"] is not None} + if len(dims) > 1: + raise BadRequest("vectors in one batch have different lengths", code="dimension_mismatch") + if dims and pinned is not None and next(iter(dims)) != pinned: + raise Conflict(f"vectors have {next(iter(dims))} dimensions but this graph holds {pinned}; " + f"if the encoder changed, POST .../reembed", code="dimension_mismatch") + new = 0 + for p in prepared: + exists = graph.has_node(p["type"], p["id"]) + p["exists"] = exists + if not exists: + if _twin_conflict(graph, p["type"], p["id"]): + raise Conflict("a node whose id differs only by integer/string spelling already exists", + code="id_conflict") + new += 1 + env.rt.reserve(q, nodes=new) + applied_new = 0 + try: + for p in prepared: + ref = (p["type"], p["id"]) + if p["exists"]: + node = graph.get_node(*ref) + node.properties.clear() + node.properties.update(p["props"]) + updated += 1 + else: + node = graph.add_node(p["type"], p["id"]) + node.properties.update(p["props"]) + created += 1 + applied_new += 1 + if p["vector"] is not None: + store.add(ref, p["text"], vector=p["vector"]) + embedded += 1 + elif p["server"] and p["exists"]: + store.remove(ref) # new properties leave nothing to embed + finally: + if applied_new != new: + env.rt.adjust(nodes=applied_new - new) + handle.bump() + maybe_sync_save(env, handle) + return {"created": created, "updated": updated, "embedded": embedded} + + +def patch_node(env: OpEnv, handle: GraphHandle, node_type: str, node_id: Any, + body: Dict[str, Any]) -> Dict[str, Any]: + """Merge properties into an existing node (optimistic: the embed text is computed + outside the lock, and the write fails with 409 if the node changed meanwhile).""" + meta, q = handle.meta, env.quotas + set_props = _clean_props({k: v for k, v in (body.get("properties") or {}).items()}, q, "$.properties") + null_keys = {k for k, v in (body.get("properties") or {}).items() if v is None} + remove = set(body.get("remove") or []) | null_keys + vec, text = body.get("vector"), body.get("embed_text") + + with read_locked(env, handle) as graph: + try: + node = graph.get_node(node_type, node_id) + except GraphError: + raise NotFound("node not found") from None + before = dict(node.properties) + merged = {k: v for k, v in before.items() if k not in remove and v is not None} + merged.update(set_props) + if len(merged) > q.max_properties: + raise validation_error("$.properties", f"at most {q.max_properties} properties per node") + + new_vector: Optional[List[float]] = None + new_text = "" + touches_embedding = (text is not None or vec is not None + or any(f in set_props or f in remove for f in meta.embed_fields)) + if meta.mode == "client": + if vec is None and text is not None: + raise validation_error("$.embed_text", "a client-mode graph takes 'vector', not text to embed") + if vec is not None: + _check_vector(env, handle, vec, "$.vector") + new_vector, new_text = vec, text or "" + else: + if vec is not None: + raise validation_error("$.vector", "a server-mode graph embeds text itself") + if touches_embedding: + new_text = text if text is not None else handle.graph.embed_text_for(merged) + if new_text: + new_vector = _encode(env, [new_text])[0] + + with write_locked(env, handle) as graph: + try: + node = graph.get_node(node_type, node_id) + except GraphError: + raise NotFound("node not found") from None + if dict(node.properties) != before: + raise Conflict("the node changed while this update was being prepared; retry", + code="concurrent_update") + store = graph.embedding_store + if new_vector is not None and store.dimension is not None and len(new_vector) != store.dimension: + raise Conflict(f"vector has {len(new_vector)} dimensions but this graph holds {store.dimension}", + code="dimension_mismatch") + for k in remove: + node.properties.pop(k, None) + node.properties.update(set_props) + ref = (node_type, node_id) + if new_vector is not None: + store.add(ref, new_text, vector=new_vector) + elif meta.mode == "server" and touches_embedding: + store.remove(ref) + handle.bump() + out = node_json(node) + maybe_sync_save(env, handle) + return out + + +def delete_nodes(env: OpEnv, handle: GraphHandle, refs: List[Dict[str, Any]]) -> Dict[str, Any]: + _batch_limit(env, len(refs), "nodes") + deleted = missing = 0 + with write_locked(env, handle) as graph: + n0, e0 = graph.node_count(), graph.edge_count() + try: + for r in refs: + if graph.has_node(r["type"], r["id"]): + graph.remove_node(r["type"], r["id"]) + deleted += 1 + else: + missing += 1 + finally: + env.rt.adjust(nodes=graph.node_count() - n0, edges=graph.edge_count() - e0) + handle.bump() + maybe_sync_save(env, handle) + return {"deleted": deleted, "missing": missing} + + +def get_node(env: OpEnv, handle: GraphHandle, node_type: str, node_id: Any, + include_vector: bool = False) -> Dict[str, Any]: + with read_locked(env, handle) as graph: + try: + node = graph.get_node(node_type, node_id) + except GraphError: + raise NotFound("node not found") from None + out = node_json(node) + rec = graph.embedding_store.get((node_type, node_id)) + out["embedded"] = rec is not None + if rec is not None: + out["embedding"] = {"model": rec.model, "dimension": len(rec.vector), "text": rec.text} + if include_vector: + out["embedding"]["vector"] = rec.vector + return out + + +def _sorted_nodes(handle: GraphHandle, graph: SemanticGraph) -> List[tuple]: + with handle._cache_lock: + cached = handle._sorted_nodes + if cached is None or cached[0] != handle.version: + keys = sorted((node_sort_key(n.ref) for n in graph.nodes())) + cached = handle._sorted_nodes = (handle.version, keys) + return cached[1] + + +def list_nodes(env: OpEnv, handle: GraphHandle, node_type: Optional[str], cursor: Optional[str], + limit: int) -> Dict[str, Any]: + after = decode_cursor(cursor, 3) + with read_locked(env, handle) as graph: + keys = _sorted_nodes(handle, graph) + if node_type is not None: + start = bisect.bisect_left(keys, (node_type,)) + else: + start = 0 + if after is not None: + try: + start = max(start, bisect.bisect_right(keys, (after[0], int(after[1]), after[2]))) + except (TypeError, ValueError, IndexError): + raise BadRequest("invalid cursor", code="invalid_cursor") from None + items = [] + i = start + while i < len(keys) and len(items) < limit: + key = keys[i] + if node_type is not None and key[0] != node_type: + break + try: + items.append(node_json(graph.get_node(key[0], key[2]))) + except GraphError: + pass + i += 1 + more = i < len(keys) and (node_type is None or keys[i][0] == node_type) + next_cursor = encode_cursor(node_sort_key((items[-1]["type"], items[-1]["id"]))) \ + if (more and items) else None + return {"items": items, "next_cursor": next_cursor} + + +# -- edges --------------------------------------------------------------------------------------------------- + + +def upsert_edges(env: OpEnv, handle: GraphHandle, items: List[Dict[str, Any]]) -> Dict[str, Any]: + _batch_limit(env, len(items), "edges") + q = env.quotas + seen = set() + prepared = [] + for i, it in enumerate(items): + s = (it["source"]["type"], it["source"]["id"]) + t = (it["target"]["type"], it["target"]["id"]) + key = (it["rel"], s, t) + if key in seen: + raise validation_error(f"$.edges[{i}]", "the same edge appears twice in this batch") + seen.add(key) + props = _clean_props(it.get("properties") or {}, q, f"$.edges[{i}].properties") + prepared.append((it["rel"], s, t, props)) + created = updated = 0 + with write_locked(env, handle) as graph: + new = 0 + for i, (rel, s, t, _p) in enumerate(prepared): + if not graph.has_node(*s) or not graph.has_node(*t): + raise NotFound(f"edge {i}: source or target node does not exist") + if not graph.has_edge(rel, s, t): + new += 1 + env.rt.reserve(q, edges=new) + applied = 0 + try: + for rel, s, t, props in prepared: + if graph.has_edge(rel, s, t): + edge = graph.get_edge(rel, s, t) + edge.properties.clear() + edge.properties.update(props) + updated += 1 + else: + edge = graph.add_edge(rel, s, t) + edge.properties.update(props) + created += 1 + applied += 1 + except (DuplicateEdgeError, NodeNotFoundError) as exc: # pragma: no cover - pre-checked + raise Conflict(str(exc)[:120]) from None + finally: + if applied != new: + env.rt.adjust(edges=applied - new) + handle.bump() + maybe_sync_save(env, handle) + return {"created": created, "updated": updated} + + +def delete_edges(env: OpEnv, handle: GraphHandle, items: List[Dict[str, Any]]) -> Dict[str, Any]: + _batch_limit(env, len(items), "edges") + deleted = missing = 0 + with write_locked(env, handle) as graph: + e0 = graph.edge_count() + try: + for it in items: + s = (it["source"]["type"], it["source"]["id"]) + t = (it["target"]["type"], it["target"]["id"]) + try: + graph.remove_edge(it["rel"], s, t) + deleted += 1 + except EdgeNotFoundError: + missing += 1 + finally: + env.rt.adjust(edges=graph.edge_count() - e0) + handle.bump() + maybe_sync_save(env, handle) + return {"deleted": deleted, "missing": missing} + + +def _edge_key(e: Any) -> tuple: + return (e.rel_type, node_sort_key(e.source), node_sort_key(e.target)) + + +def list_edges(env: OpEnv, handle: GraphHandle, rel: Optional[str], cursor: Optional[str], + limit: int) -> Dict[str, Any]: + after = decode_cursor(cursor, 1) + with read_locked(env, handle) as graph: + with handle._cache_lock: + cached = handle._sorted_edges + if cached is None or cached[0] != handle.version: + edges = sorted(graph.edges(), key=_edge_key) + cached = handle._sorted_edges = (handle.version, edges, [_edge_key(e) for e in edges]) + _, edges, keys = cached + start = 0 + if rel is not None: + start = bisect.bisect_left(keys, (rel,)) + if after is not None: + try: + pos = int(after[0]) + except (TypeError, ValueError): + raise BadRequest("invalid cursor", code="invalid_cursor") from None + start = max(start, pos) + items = [] + i = start + while i < len(edges) and len(items) < limit: + if rel is not None and edges[i].rel_type != rel: + break + items.append(edge_json(edges[i])) + i += 1 + more = i < len(edges) and (rel is None or edges[i].rel_type == rel) + return {"items": items, "next_cursor": encode_cursor([i]) if more else None} + + +# -- search ---------------------------------------------------------------------------------------------------- + + +def _cap(env_value: int, got: Optional[int], default: int, path: str, what: str) -> int: + if got is None: + return min(default, env_value) + value = got + if value > env_value: + raise validation_error(path, f"must be <= {env_value} ({what} limit for this tenant)") + return value + + +def resolve_query(env: OpEnv, handle: GraphHandle, body: Dict[str, Any]) -> List[float]: + """Turn ``query`` (text) or ``vector`` into a query vector, encoding outside any lock.""" + text, vec = body.get("query"), body.get("vector") + if (text is None) == (vec is None): + raise BadRequest("send exactly one of 'query' (text) or 'vector'", code="invalid_request") + meta = handle.meta + if vec is not None: + if len(vec) > env.quotas.max_dimension: + raise validation_error("$.vector", "vector longer than this tenant's limit") + return vec + if meta.mode != "server": + raise BadRequest("this graph takes caller-supplied vectors; send 'vector', not 'query'", + code="text_query_unsupported") + if not text.strip(): + raise validation_error("$.query", "must not be empty") + enc = env.encoder + if enc is None: + raise Unavailable("no embedding encoder is configured", code="encoder_unavailable") + with read_locked(env, handle) as graph: + models = graph.embedding_store.models() + pinned = graph.embedding_store.dimension + current = getattr(enc, "model_name", None) + foreign = sorted(m for m in models if m not in UNKNOWN_MODELS and m != current) + if foreign: + raise Conflict("the stored vectors were made by a different embedding model than the one " + "the server now uses; POST .../reembed first", code="model_mismatch") + vector = _encode(env, [text])[0] + if pinned is not None and len(vector) != pinned: + raise Conflict("the stored vectors have a different dimension than the current encoder; " + "POST .../reembed first", code="dimension_mismatch") + return vector + + +def _search_errors(fn): + try: + return fn() + except DimensionMismatchError as exc: + raise BadRequest("query vector has the wrong number of dimensions for this graph", + code="dimension_mismatch") from None + except NoEncoderError: + raise BadRequest("this graph has no encoder", code="text_query_unsupported") from None + except EmbeddingError: + raise Unavailable("search failed", code="search_failed") from None + + +def _hit_json(graph: SemanticGraph, ref, score: float, include_props: bool, text: Optional[str] = None): + out = {"type": ref[0], "id": ref[1], "score": _score(score)} + if include_props: + try: + out["properties"] = live_props(graph.get_node(*ref).properties) + except GraphError: + out["properties"] = {} + if text is not None: + out["text"] = text + return out + + +def search(env: OpEnv, handle: GraphHandle, body: Dict[str, Any]) -> Dict[str, Any]: + q = env.quotas + top_k = _cap(q.max_top_k, body.get("top_k"), 10, "$.top_k", "top_k") + vec = resolve_query(env, handle, body) + pred, types = compile_filter(body.get("filter")) + env.ctx.check() + with read_locked(env, handle) as graph: + results = _search_errors(lambda: graph.similarity_search( + vec, top_k=top_k, node_type=body.get("node_type"), threshold=body.get("threshold", 0.0), + node_filter=pred, node_types=types, exact=body.get("exact", False))) + inc, txt = body.get("include_properties", True), body.get("include_text", False) + hits = [_hit_json(graph, r.node_ref, r.score, inc, r.text if txt else None) for r in results] + return {"results": hits, "count": len(hits)} + + +def similar(env: OpEnv, handle: GraphHandle, body: Dict[str, Any]) -> Dict[str, Any]: + top_k = _cap(env.quotas.max_top_k, body.get("top_k"), 10, "$.top_k", "top_k") + ref = (body["node"]["type"], body["node"]["id"]) + with read_locked(env, handle) as graph: + if not graph.has_node(*ref): + raise NotFound("node not found") + embedded = graph.embedding_store.get(ref) is not None + results = _search_errors(lambda: graph.similar_to_node( + ref, top_k=top_k, node_type=body.get("node_type"), threshold=body.get("threshold", 0.0))) + inc = body.get("include_properties", True) + hits = [_hit_json(graph, r.node_ref, r.score, inc) for r in results] + return {"results": hits, "count": len(hits), "node_embedded": embedded} + + +def _traversal_args(env: OpEnv, body: Dict[str, Any], *, max_hops: int, seeds: int, decay: float, + threshold: float) -> Dict[str, Any]: + q = env.quotas + pred, types = compile_filter(body.get("filter")) + return { + "rel_type": body.get("rel_type"), + "max_hops": _cap(q.max_hops, body.get("max_hops"), max_hops, "$.max_hops", "hop"), + "top_k_seeds": _cap(q.max_top_k, body.get("top_k_seeds"), seeds, "$.top_k_seeds", "seed"), + "direction": _DIRECTIONS[body.get("direction", "out")], + "decay": body.get("decay", decay), + "threshold": body.get("threshold", threshold), + "weight_field": body.get("weight_field"), + "node_filter": pred, + "node_types": types, + "strict_filter": body.get("strict_filter", False), + } + + +def multi_hop(env: OpEnv, handle: GraphHandle, body: Dict[str, Any]) -> Dict[str, Any]: + args = _traversal_args(env, body, max_hops=2, seeds=5, decay=0.8, threshold=0.3) + top = _cap(env.quotas.max_top_k, body.get("top_k_results"), 10, "$.top_k_results", "result") + vec = resolve_query(env, handle, body) + env.ctx.check() + with read_locked(env, handle) as graph: + results = _search_errors(lambda: graph.semantic_multi_hop( + vec, top_k_results=top, blend=body.get("blend", 0.0), **args)) + inc = body.get("include_properties", True) + out = [] + for r in results: + item = _hit_json(graph, r.node_ref, r.score, inc) + item["hops"] = r.hop_count + item["path"] = [ref_json(p) for p in r.path] + out.append(item) + return {"results": out, "count": len(out)} + + +def path(env: OpEnv, handle: GraphHandle, body: Dict[str, Any]) -> Dict[str, Any]: + args = _traversal_args(env, body, max_hops=5, seeds=3, decay=0.9, threshold=0.0) + args.pop("weight_field") # semantic_path scores by hop count only + target = (body["target"]["type"], body["target"]["id"]) + vec = resolve_query(env, handle, body) + env.ctx.check() + with read_locked(env, handle) as graph: + if not graph.has_node(*target): + raise NotFound("target node not found") + result = _search_errors(lambda: graph.semantic_path(vec, target, **args)) + if result is None: + return {"found": False, "result": None} + item = _hit_json(graph, result.node_ref, result.score, body.get("include_properties", True)) + item["hops"] = result.hop_count + item["path"] = [ref_json(p) for p in result.path] + return {"found": True, "result": item} + + +def subgraph(env: OpEnv, handle: GraphHandle, body: Dict[str, Any]) -> Dict[str, Any]: + args = _traversal_args(env, body, max_hops=2, seeds=5, decay=0.8, threshold=0.3) + top = _cap(env.quotas.max_top_k, body.get("top_k_results"), 10, "$.top_k_results", "result") + vec = resolve_query(env, handle, body) + env.ctx.check() + with read_locked(env, handle) as graph: + sub = _search_errors(lambda: graph.semantic_subgraph( + vec, top_k_results=top, blend=body.get("blend", 0.0), connect=body.get("connect", False), + connect_max_hops=body.get("connect_max_hops", 2), **args)) + try: + nodes = [node_json(n, body.get("include_properties", True)) for n in sub.nodes()] + edges = [edge_json(e) for e in sub.edges()] + finally: + sub.close() + return {"nodes": nodes, "edges": edges, "node_count": len(nodes), "edge_count": len(edges)} + + +# -- model lifecycle --------------------------------------------------------------------------------------- + + +def _server_mode_only(handle: GraphHandle) -> None: + if handle.meta.mode != "server": + raise BadRequest("embedding lifecycle only applies to server-mode graphs " + "(a client-mode graph's vectors belong to the caller)", code="unsupported") + + +def stale(env: OpEnv, handle: GraphHandle, include_unknown: bool, limit: int, offset: int) -> Dict[str, Any]: + _server_mode_only(handle) + if env.encoder is None: + raise Unavailable("no embedding encoder is configured", code="encoder_unavailable") + with read_locked(env, handle) as graph: + models = graph.embedding_store.models() + refs = graph.stale_embeddings(include_unknown=include_unknown) + page = refs[offset:offset + limit] + more = offset + limit < len(refs) + return { + "encoder_model": getattr(env.encoder, "model_name", None), + "stored_models": models, + "stale_count": len(refs), + "stale": [ref_json(r) for r in page], + "next_cursor": encode_cursor([offset + limit]) if more else None, + } + + +def reembed(env: OpEnv, handle: GraphHandle, include_unknown: bool) -> Dict[str, Any]: + """Re-encode stale vectors. Holds the graph's write lock for the duration + (it calls the encoder), so concurrent reads and writes of *this graph* wait.""" + _server_mode_only(handle) + if env.encoder is None: + raise Unavailable("no embedding encoder is configured", code="encoder_unavailable") + with write_locked(env, handle) as graph: + try: + count = graph.reembed_stale(include_unknown=include_unknown) + finally: + handle.bump() + maybe_sync_save(env, handle) + return {"reembedded": count} diff --git a/isongraph-vector-server/isongraph_vector_server/replication.py b/isongraph-vector-server/isongraph_vector_server/replication.py new file mode 100644 index 0000000..b07db74 --- /dev/null +++ b/isongraph-vector-server/isongraph_vector_server/replication.py @@ -0,0 +1,506 @@ +"""Snapshot-based read replicas. + +What a replica is: a second server process, started with ``role = "replica"``, +with its own data directory, that periodically *pulls* the primary's published +snapshot generations and serves them read-only. What it is not: a hot standby. +There is no write-ahead log, no streaming, no promotion and no failover. + +One sync cycle: + + 1. fetch the primary's index (every tenant, every graph's current + generation and the sha256 of its MANIFEST) + 2. for each tenant file / graph whose generation differs from the local one, + download MANIFEST, graph file and embeddings sidecar into a staging + directory + 3. verify, in order: the manifest against the index entry; each file's size + and sha256 against the manifest; the library's sidecar-vs-graph digest + check; the loaded graph's digest against the manifest's; the manifest's + digest against the index's + 4. only then install the generation, flip the replica's CURRENT and swap the + new in-memory graph in. Any failure leaves the replica serving what it had + ("refused"), and is counted and logged. + +Consistency, plainly: a replica serves a *saved generation*, not the primary's +live memory, so a read can be stale by up to +``primary autosave interval + replica sync interval + sync duration``. +Different graphs can be at different primary moments (generations are +per-graph; there is no cross-graph snapshot). If the last fully successful +sync is older than ``max_staleness_seconds`` the replica stops serving +(503 ``replica_stale``) instead of serving unboundedly old data. Revoking a +key or disabling a tenant on the primary reaches replicas on the next sync. +""" + +from __future__ import annotations + +import hashlib +import json +import logging +import os +import shutil +import threading +import time +import urllib.error +import urllib.request +import uuid +from pathlib import Path +from typing import Any, Dict, List, Optional + +from . import auth, names +from .config import ServerConfig +from .engine import Engine +from .errors import Unavailable +from .logs import log +from .metrics import Metrics +from .storage import (GRAPH_FILE, MANIFEST_FILE, SIDECAR_FILE, GraphMeta, GraphStorage, Manifest, + SnapshotError, atomic_write, new_semantic_graph, sha256_file) + +STATE_FILE = "REPLICA_STATE.json" + + +class ReplicationError(Exception): + """The primary could not be reached or answered badly (not a verification failure).""" + + +# -- the index (built by the primary, or read from a directory) ----------------------------------------- + + +def build_index(data_dir: str) -> Dict[str, Any]: + base = Path(data_dir) / "tenants" + tenants: List[Dict[str, Any]] = [] + if base.is_dir(): + for entry in sorted(os.scandir(base), key=lambda e: e.name): + if not (entry.is_dir(follow_symlinks=False) and names.is_tenant_id(entry.name)): + continue + tf = Path(entry.path) / "tenant.json" + try: + tsha, _ = sha256_file(tf) + except OSError: + continue + graphs = [] + gdir = Path(entry.path) / "graphs" + if gdir.is_dir(): + for g in sorted(os.scandir(gdir), key=lambda e: e.name): + if not (g.is_dir(follow_symlinks=False) and g.name.startswith("g-") + and names.is_graph_name(g.name[2:])): + continue + graphs.append(_graph_entry(Path(g.path), g.name[2:])) + tenants.append({"tenant_id": entry.name, "tenant_sha256": tsha, "graphs": graphs}) + return {"format": "igv-replication-index", "version": 1, "generated_at": time.time(), + "tenants": tenants} + + +def _graph_entry(path: Path, name: str) -> Dict[str, Any]: + st = GraphStorage(path) + entry: Dict[str, Any] = {"name": name, "generation": None, "manifest_sha256": None, + "graph_digest": None, "meta_sha256": None} + try: + entry["meta_sha256"], _ = sha256_file(path / "meta.json") + cur = st.read_current() + if cur is not None: + mpath = st.gen_path(cur) / MANIFEST_FILE + entry["manifest_sha256"], _ = sha256_file(mpath) + entry["graph_digest"] = json.loads(mpath.read_text(encoding="utf-8")).get("graph_digest") + entry["generation"] = cur + except (OSError, ValueError): + # Do NOT drop the entry: a replica must not mistake "unreadable right now" + # for "deleted on the primary". + entry["error"] = "unreadable" + return entry + + +# -- sources ----------------------------------------------------------------------------------------------- + + +class DirSource: + """Read the primary's data directory directly (shared volume, NFS, rsync target).""" + + def __init__(self, root: str): + self.root = Path(root) + + def index(self) -> Dict[str, Any]: + if not (self.root / "tenants").is_dir(): + raise ReplicationError("source directory has no tenants/") + return build_index(str(self.root)) + + def _copy(self, src: Path, dest: Path, max_bytes: int) -> None: + try: + if src.stat().st_size > max_bytes: + raise ReplicationError("file larger than the allowed maximum") + with open(src, "rb") as r, open(dest, "wb") as w: + shutil.copyfileobj(r, w, 1 << 20) + except OSError as exc: + raise ReplicationError(f"cannot read source file ({type(exc).__name__})") from None + + def fetch_tenant(self, tid: str, dest: Path, max_bytes: int) -> None: + self._copy(names.tenant_dir(self.root, tid) / "tenant.json", dest, max_bytes) + + def fetch_meta(self, tid: str, g: str, dest: Path, max_bytes: int) -> None: + self._copy(names.graph_dir(self.root, tid, g) / "meta.json", dest, max_bytes) + + def fetch_file(self, tid: str, g: str, gen: int, fname: str, dest: Path, max_bytes: int) -> None: + self._copy(names.graph_dir(self.root, tid, g) / f"gen-{gen:06d}" / fname, dest, max_bytes) + + +class _NoRedirect(urllib.request.HTTPRedirectHandler): + def redirect_request(self, *args, **kwargs): + return None + + +class HttpSource: + """Pull from a primary's ``/_repl/v1`` endpoints with the replication token.""" + + def __init__(self, base_url: str, token: str, timeout: float = 30.0): + self.base = base_url.rstrip("/") + self.token = token + self.timeout = timeout + self._opener = urllib.request.build_opener(_NoRedirect) + + def _open(self, path: str): + req = urllib.request.Request(self.base + path, headers={"Authorization": f"Bearer {self.token}"}) + try: + return self._opener.open(req, timeout=self.timeout) + except urllib.error.HTTPError as exc: + raise ReplicationError(f"primary answered HTTP {exc.code}") from None + except (urllib.error.URLError, OSError) as exc: + raise ReplicationError(f"primary unreachable ({type(exc).__name__})") from None + + def index(self) -> Dict[str, Any]: + with self._open("/_repl/v1/index") as resp: + data = resp.read(64 * 1024 * 1024) + try: + return json.loads(data) + except ValueError: + raise ReplicationError("primary sent an unreadable index") from None + + def _download(self, path: str, dest: Path, max_bytes: int) -> None: + got = 0 + with self._open(path) as resp, open(dest, "wb") as out: + declared = resp.headers.get("Content-Length") + if declared is not None and int(declared) > max_bytes: + raise ReplicationError("file larger than the allowed maximum") + while True: + chunk = resp.read(1 << 20) + if not chunk: + break + got += len(chunk) + if got > max_bytes: + raise ReplicationError("file larger than the allowed maximum") + out.write(chunk) + if declared is not None and int(declared) != got: + raise ReplicationError("truncated download") + + def fetch_tenant(self, tid: str, dest: Path, max_bytes: int) -> None: + self._download(f"/_repl/v1/tenant/{tid}", dest, max_bytes) + + def fetch_meta(self, tid: str, g: str, dest: Path, max_bytes: int) -> None: + self._download(f"/_repl/v1/meta/{tid}/{g}", dest, max_bytes) + + def fetch_file(self, tid: str, g: str, gen: int, fname: str, dest: Path, max_bytes: int) -> None: + self._download(f"/_repl/v1/file/{tid}/{g}/{gen:06d}/{fname}", dest, max_bytes) + + +def make_source(cfg: ServerConfig): + src = cfg.replica.source + if src.startswith("dir:"): + return DirSource(src[4:]) + if src.startswith(("http://", "https://")): + if not cfg.replication_token: + raise ValueError("IGV_REPLICATION_TOKEN is required to replicate from an HTTP primary") + return HttpSource(src, cfg.replication_token, cfg.replica.fetch_timeout_seconds) + raise ValueError("replica.source must be 'dir:<path>' or an http(s):// URL") + + +# -- the replica ----------------------------------------------------------------------------------------------- + + +class ReplicaSyncer: + def __init__(self, cfg: ServerConfig, engine: Engine, registry: auth.TenantRegistry, + metrics: Metrics, source=None): + self.cfg = cfg + self.engine = engine + self.registry = registry + self.metrics = metrics + self.source = source or make_source(cfg) + self.data_dir = Path(cfg.data_dir) + self.last_ok_wall: Optional[float] = None + self.last_error: Optional[str] = None + self.syncs_ok = 0 + self.syncs_failed = 0 + self.refused = 0 + self._lock = threading.Lock() + self._stop = threading.Event() + self._thread: Optional[threading.Thread] = None + self._load_state() + + # state that survives a restart, so staleness is measured honestly across one + def _load_state(self) -> None: + try: + data = json.loads((self.data_dir / STATE_FILE).read_text(encoding="utf-8")) + self.last_ok_wall = float(data["last_ok_wall"]) + except (OSError, ValueError, KeyError, TypeError): + self.last_ok_wall = None + + def _save_state(self) -> None: + try: + atomic_write(self.data_dir / STATE_FILE, + json.dumps({"last_ok_wall": self.last_ok_wall}).encode("utf-8")) + except OSError: + pass + + # -- serving gate ---------------------------------------------------------------------------------- + + def staleness_seconds(self) -> Optional[float]: + return None if self.last_ok_wall is None else max(0.0, time.time() - self.last_ok_wall) + + def is_ready(self) -> bool: + age = self.staleness_seconds() + return age is not None and age <= self.cfg.replica.max_staleness_seconds + + def check_fresh(self) -> None: + age = self.staleness_seconds() + if age is None: + raise Unavailable("replica has not completed a sync yet", code="replica_not_synced", + headers={"Retry-After": "5"}) + if age > self.cfg.replica.max_staleness_seconds: + raise Unavailable("replica data is older than its staleness bound; " + "use the primary or wait for the replica to resync", + code="replica_stale", headers={"Retry-After": "5"}) + + def metrics_snapshot(self) -> Dict[str, float]: + age = self.staleness_seconds() + return { + "igv_replica_staleness_seconds": -1 if age is None else age, + "igv_replica_syncs_ok": self.syncs_ok, + "igv_replica_syncs_failed": self.syncs_failed, + "igv_replica_refused_snapshots": self.refused, + } + + # -- one cycle ---------------------------------------------------------------------------------------- + + def sync_once(self) -> Dict[str, Any]: + report = {"updated": [], "refused": [], "errors": [], "removed": []} + with self._lock: + try: + index = self.source.index() + if index.get("format") != "igv-replication-index" or index.get("version") != 1: + raise ReplicationError("unrecognised index format") + tenants = index["tenants"] + if not isinstance(tenants, list): + raise ReplicationError("malformed index") + except (ReplicationError, KeyError, TypeError) as exc: + self.syncs_failed += 1 + self.last_error = str(exc)[:200] + self.metrics.inc("igv_replica_sync_failures_total") + log(logging.ERROR, "replica sync failed: cannot read the primary's index", + error=self.last_error) + report["errors"].append("index") + return report + + wanted_tenants = set() + clean = True + for t in tenants: + tid = t.get("tenant_id") if isinstance(t, dict) else None + if not names.is_tenant_id(tid): + continue + wanted_tenants.add(tid) + clean &= self._sync_tenant(tid, t, report) + clean &= self._remove_gone(wanted_tenants, tenants, report) + self.registry.rescan(force=True) + + if clean: + self.last_ok_wall = time.time() + now = self.last_ok_wall + for rt in list(self.engine.tenants.values()): + for h in list(rt.graphs.values()): + if h.graph is not None: + h.synced_at = now + self.syncs_ok += 1 + self.last_error = None + self._save_state() + self.metrics.inc("igv_replica_syncs_total") + else: + self.syncs_failed += 1 + return report + + def _fail(self, report, kind: str, tid: str, g: Optional[str], exc: Exception, refused: bool) -> None: + self.last_error = f"{tid}/{g}: {type(exc).__name__}" + key = "refused" if refused else "errors" + report[key].append(f"{tid}/{g}" if g else tid) + if refused: + self.refused += 1 + self.metrics.inc("igv_replica_refused_total") + else: + self.metrics.inc("igv_replica_sync_failures_total") + log(logging.ERROR, f"replica {kind}: keeping the previous version", tenant=tid, graph=g, + reason=str(exc)[:300]) + + def _sync_tenant(self, tid: str, entry: Dict[str, Any], report) -> bool: + ok = True + tdir = names.tenant_dir(self.data_dir, tid) + tfile = tdir / "tenant.json" + want = entry.get("tenant_sha256") + try: + have = sha256_file(tfile)[0] if tfile.is_file() else None + except OSError: + have = None + if want and have != want: + stage = self.data_dir / f".stage-{uuid.uuid4().hex[:12]}" + try: + stage.mkdir(parents=True, exist_ok=True) + tmp = stage / "tenant.json" + self.source.fetch_tenant(tid, tmp, 1 << 20) + if sha256_file(tmp)[0] != want: + raise SnapshotError("tenant file does not match the index checksum") + auth.parse_tenant_file(json.loads(tmp.read_text(encoding="utf-8")), tid) + (tdir / "graphs").mkdir(parents=True, exist_ok=True) + atomic_write(tfile, tmp.read_bytes()) + report["updated"].append(f"{tid}/tenant.json") + except (SnapshotError, ValueError) as exc: + self._fail(report, "refused a tenant file", tid, None, exc, refused=True) + ok = False + except ReplicationError as exc: + self._fail(report, "tenant fetch failed", tid, None, exc, refused=False) + ok = False + finally: + shutil.rmtree(stage, ignore_errors=True) + if not tfile.is_file(): + return False + rt = self.engine.runtime(tid, self.cfg.quotas) + gdir = tdir / "graphs" + wanted = set() + for g in entry.get("graphs", []): + name = g.get("name") if isinstance(g, dict) else None + if not names.is_graph_name(name): + continue + wanted.add(name) + if "error" in g: + ok = False + report["errors"].append(f"{tid}/{name}") + continue + ok &= self._sync_graph(rt, gdir, tid, name, g, report) + # graphs the primary no longer has + for name in list(rt.graphs): + if name not in wanted: + self.engine.drop_graph_local(rt, name) + report["removed"].append(f"{tid}/{name}") + return ok + + def _remove_gone(self, wanted_tenants, tenants, report) -> bool: + base = self.data_dir / "tenants" + if not base.is_dir(): + return True + for entry in os.scandir(base): + if entry.is_dir(follow_symlinks=False) and names.is_tenant_id(entry.name) \ + and entry.name not in wanted_tenants: + rt = self.engine.tenants.get(entry.name) + if rt is not None: + for name in list(rt.graphs): + self.engine.drop_graph_local(rt, name) + shutil.rmtree(entry.path, ignore_errors=True) + report["removed"].append(entry.name) + return True + + def _local_state(self, gpath: Path) -> Optional[Dict[str, Any]]: + try: + return json.loads((gpath / "REPLICA.json").read_text(encoding="utf-8")) + except (OSError, ValueError): + return None + + def _sync_graph(self, rt, gdir: Path, tid: str, name: str, g: Dict[str, Any], report) -> bool: + gpath = gdir / names.graph_dirname(name) + wanted = (g.get("generation"), g.get("manifest_sha256"), g.get("meta_sha256")) + state = self._local_state(gpath) + handle = rt.graphs.get(name) + if (state is not None and handle is not None and handle.graph is not None + and (state.get("generation"), state.get("manifest_sha256"), + state.get("meta_sha256")) == wanted): + return True + stage_root = gdir / f".tmp-sync-{uuid.uuid4().hex[:12]}" + try: + stage_root.mkdir(parents=True, exist_ok=True) + meta_path = stage_root / "meta.json" + self.source.fetch_meta(tid, name, meta_path, 1 << 20) + if sha256_file(meta_path)[0] != g.get("meta_sha256"): + raise SnapshotError("meta.json does not match the index checksum") + meta = GraphMeta.from_json(json.loads(meta_path.read_text(encoding="utf-8")), name) + + gen = g.get("generation") + if gen is None: + graph, manifest, stage_gen = new_semantic_graph(meta, self.engine.encoder), None, None + else: + if not isinstance(gen, int): + raise SnapshotError("bad generation in index") + stage_gen = stage_root / f"gen-{gen:06d}" + stage_gen.mkdir() + self.source.fetch_file(tid, name, gen, MANIFEST_FILE, stage_gen / MANIFEST_FILE, 1 << 20) + if sha256_file(stage_gen / MANIFEST_FILE)[0] != g.get("manifest_sha256"): + raise SnapshotError("MANIFEST does not match the index checksum") + m = Manifest.from_json(json.loads((stage_gen / MANIFEST_FILE).read_text(encoding="utf-8"))) + if m.generation != gen: + raise SnapshotError("manifest generation does not match the index") + if g.get("graph_digest") and m.graph_digest != g["graph_digest"]: + raise SnapshotError("manifest digest does not match the index") + cap = self.cfg.replica.max_file_bytes + for fname in (GRAPH_FILE, SIDECAR_FILE): + self.source.fetch_file(tid, name, gen, fname, stage_gen / fname, + min(cap, m.files[fname]["size"])) + storage_probe = GraphStorage(stage_root, self.cfg.keep_generations) + graph, manifest, caught = storage_probe.load_generation(stage_gen, meta, self.engine.encoder) + for w in caught: + log(logging.WARNING, "library warning while loading a replicated snapshot", + tenant=tid, graph=name, warning=w) + except (SnapshotError, ValueError, KeyError, TypeError) as exc: + shutil.rmtree(stage_root, ignore_errors=True) + self._fail(report, "refused a snapshot (verification failed)", tid, name, exc, refused=True) + return False + except ReplicationError as exc: + shutil.rmtree(stage_root, ignore_errors=True) + self._fail(report, "snapshot fetch failed", tid, name, exc, refused=False) + return False + except Exception as exc: # pragma: no cover - defensive + shutil.rmtree(stage_root, ignore_errors=True) + self._fail(report, "snapshot sync error", tid, name, exc, refused=False) + return False + + # Verified. Install it and swap it in. + try: + gpath.mkdir(parents=True, exist_ok=True) + atomic_write(gpath / "meta.json", meta_path.read_bytes()) + storage = GraphStorage(gpath, self.cfg.keep_generations) + if stage_gen is not None: + storage.install_generation(stage_gen, gen) + else: + try: + (gpath / "CURRENT").unlink() + except OSError: + pass + atomic_write(gpath / "REPLICA.json", json.dumps( + {"generation": gen, "manifest_sha256": g.get("manifest_sha256"), + "meta_sha256": g.get("meta_sha256")}).encode("utf-8")) + self.engine.swap_graph(rt, name, meta, storage, graph, manifest) + report["updated"].append(f"{tid}/{name}@{gen}") + self.metrics.inc("igv_replica_installs_total") + finally: + shutil.rmtree(stage_root, ignore_errors=True) + return True + + # -- background thread ------------------------------------------------------------------------------------ + + def start(self) -> None: + self._thread = threading.Thread(target=self._loop, name="igv-replica-sync", daemon=True) + self._thread.start() + + def _loop(self) -> None: + while not self._stop.is_set(): + try: + self.sync_once() + except Exception as exc: # pragma: no cover - defensive + self.syncs_failed += 1 + log(logging.ERROR, "replica sync crashed", error=str(exc)[:200]) + if self._stop.wait(self.cfg.replica.sync_interval_seconds): + break + + def stop(self) -> None: + self._stop.set() + if self._thread is not None: + self._thread.join(timeout=30) diff --git a/isongraph-vector-server/isongraph_vector_server/router.py b/isongraph-vector-server/isongraph_vector_server/router.py new file mode 100644 index 0000000..a072bb8 --- /dev/null +++ b/isongraph-vector-server/isongraph_vector_server/router.py @@ -0,0 +1,222 @@ +"""A stateless router that hash-shards tenants across backend nodes. + +It holds no data and no credentials. The tenant id is read from the API key +(``igv_<tenant>.<key-id>.<secret>``), mapped to a shard by rendezvous +(highest-random-weight) hashing, and the request is forwarded unchanged, +``Authorization`` header included. **The backend authenticates**: the router +only uses the tenant id as a routing hint, so forging a key that names another +tenant gets the request sent to that tenant's shard, where the key fails +verification. The router cannot widen anyone's access. + +What it does not do, deliberately and honestly: + +* No resharding. Rendezvous hashing means adding or removing a shard moves only + the tenants whose winner changed, but *moving their data is manual* (copy the + tenant directory to the new owner's data dir, or pin the tenant with + ``router.pins`` while you do). +* No failover. If a shard's primary is down, its tenants get 502/503. Reads can + fall back from a replica to the primary, never the other way round. +* No retries of writes, ever. Reads are retried once on the next candidate. +* No cross-tenant or cross-shard queries, joins or listings. +""" + +from __future__ import annotations + +import hashlib +import http.client +import logging +import secrets +import threading +import urllib.parse +from typing import Any, Dict, List, Optional, Tuple + +from . import names +from .auth import tenant_id_from_key +from .config import ServerConfig +from .errors import ApiError, BadGateway, NotFound, Unauthorized +from .httpd import Request, Response +from .logs import log +from .service import bearer_token, split_path + +READ_SUFFIXES = {"search", "multi_hop", "path", "subgraph", "similar"} +FORWARD_REQUEST_HEADERS = ("authorization", "content-type", "accept", "x-request-id") +FORWARD_RESPONSE_HEADERS = ("content-type", "retry-after", "x-request-id", "www-authenticate", "allow", + "x-snapshot-generation", "x-snapshot-age-seconds") +#: Backend answers that mean "this replica cannot serve you right now, ask the primary". +REPLICA_FALLBACK_CODES = (b"replica_stale", b"replica_not_synced", b"not_ready") + + +def rendezvous(tenant_id: str, shard_names: List[str]) -> str: + best, best_score = "", -1 + for name in shard_names: + score = int.from_bytes(hashlib.sha256(f"{name}\x00{tenant_id}".encode()).digest()[:8], "big") + if score > best_score: + best, best_score = name, score + return best + + +class _Backend: + def __init__(self, shard: str, kind: str, url: str, timeout: float): + parts = urllib.parse.urlsplit(url) + if parts.scheme not in ("http", "https") or not parts.hostname: + raise ValueError(f"bad backend URL for shard {shard}") + self.shard, self.kind, self.timeout = shard, kind, timeout + self.scheme, self.host = parts.scheme, parts.hostname + self.port = parts.port or (443 if parts.scheme == "https" else 80) + self.prefix = parts.path.rstrip("/") + self._idle: List[Any] = [] + self._idle_lock = threading.Lock() + self.max_idle = 64 + + def label(self) -> str: + return f"{self.shard}/{self.kind}" + + def _new(self): + cls = http.client.HTTPSConnection if self.scheme == "https" else http.client.HTTPConnection + return cls(self.host, self.port, timeout=self.timeout) + + def request(self, method: str, target: str, headers: Dict[str, str], body: bytes, + reuse: bool = False) -> Tuple[int, List[Tuple[str, str]], bytes]: + """One upstream round trip. + + ``reuse`` (reads only) draws from a pool of keep-alive connections; a pooled connection + that turns out to be dead is replaced by a fresh one and the read repeated. Writes always + use a fresh connection and are never repeated: after a failure the router cannot know + whether the backend applied the change. + """ + conn = None + if reuse: + with self._idle_lock: + conn = self._idle.pop() if self._idle else None + fresh = conn is None + if conn is None: + conn = self._new() + try: + conn.request(method, self.prefix + target, body=body or None, headers=headers) + resp = conn.getresponse() + data = resp.read() + except (OSError, http.client.HTTPException): + conn.close() + if not fresh: + return self.request(method, target, headers, body, reuse=False) + raise + if reuse and not resp.will_close: + with self._idle_lock: + if len(self._idle) < self.max_idle: + self._idle.append(conn) + conn = None + if conn is not None: + conn.close() + return resp.status, resp.getheaders(), data + + +class Router: + def __init__(self, cfg: ServerConfig): + self.cfg = cfg + rc = cfg.router + if not rc.shards: + raise ValueError("router needs at least one shard") + self.shard_names = [s["name"] for s in rc.shards] + self.primaries: Dict[str, _Backend] = {} + self.replicas: Dict[str, List[_Backend]] = {} + for s in rc.shards: + self.primaries[s["name"]] = _Backend(s["name"], "primary", s["primary"], rc.upstream_timeout_seconds) + self.replicas[s["name"]] = [_Backend(s["name"], "replica", u, rc.upstream_timeout_seconds) + for u in s.get("replicas", [])] + self._rr = 0 + self._rr_lock = threading.Lock() + + def shard_for(self, tenant_id: str) -> str: + pinned = self.cfg.router.pins.get(tenant_id) + return pinned if pinned else rendezvous(tenant_id, self.shard_names) + + def _next_replica(self, shard: str) -> Optional[_Backend]: + reps = self.replicas[shard] + if not reps: + return None + with self._rr_lock: + self._rr += 1 + return reps[self._rr % len(reps)] + + @staticmethod + def _is_read(method: str, segs: List[str]) -> bool: + if method == "GET": + return True + return (method == "POST" and len(segs) == 4 and segs[0] == "v1" and segs[1] == "graphs" + and segs[3] in READ_SUFFIXES) + + def handle(self, req: Request) -> Response: + rid = req.header("x-request-id") + if rid is None or not names.REQUEST_ID_RE.match(rid): + rid = secrets.token_hex(8) + try: + resp = self._route(req, rid) + except ApiError as exc: + resp = Response.json(exc.status, exc.to_body(rid), exc.headers) + except Exception: + logging.getLogger("igv").exception("router error", extra={"fields": {"request_id": rid}}) + resp = Response.json(500, ApiError("an internal error occurred").to_body(rid)) + resp.headers["X-Request-Id"] = rid + return resp + + def _route(self, req: Request, rid: str) -> Response: + segs, _ = split_path(req.target) + if segs == ["healthz"] and req.method == "GET": + return Response.json(200, {"status": "ok", "role": "router"}) + if not segs or segs[0] != "v1": + raise NotFound("no such resource") # /metrics, /_repl, ... are never proxied + token = bearer_token(req) + tid = tenant_id_from_key(token) if token else None + if tid is None: + raise Unauthorized("missing or invalid API key", + headers={"WWW-Authenticate": 'Bearer realm="igv"'}) + shard = self.shard_for(tid) + + read = self._is_read(req.method, segs) + headers = {"Content-Length": str(len(req.body)), "X-Request-Id": rid, + "X-Forwarded-For": req.peer, "Connection": "keep-alive" if read else "close"} + for name in FORWARD_REQUEST_HEADERS: + if name == "x-request-id": + continue + vals = req.header_all(name) + if len(vals) > 1 and name == "authorization": + # Ambiguous credentials: let the backend say 401 for the whole request. + headers["Authorization"] = "Bearer invalid" + elif vals: + headers[name.title()] = vals[0] + + forced_primary = (req.header("x-igv-consistency") or "").lower() == "primary" + candidates: List[_Backend] = [] + if read and not forced_primary and self.cfg.router.read_preference == "replica_preferred": + rep = self._next_replica(shard) + if rep is not None: + candidates.append(rep) + candidates.append(self.primaries[shard]) + + last_err: Optional[Exception] = None + for i, backend in enumerate(candidates): + last = i == len(candidates) - 1 + try: + status, hdrs, data = backend.request(req.method, req.target, headers, req.body, reuse=read) + except (OSError, http.client.HTTPException) as exc: + last_err = exc + log(logging.WARNING, "backend unreachable", shard=shard, kind=backend.kind, + error=type(exc).__name__) + if last or not read: + break + continue + if (not last and backend.kind == "replica" and status == 503 + and any(code in data for code in REPLICA_FALLBACK_CODES)): + continue + return self._relay(status, hdrs, data, backend) + raise BadGateway("the backend for this tenant is unavailable", code="backend_unavailable", + headers={"Retry-After": "2"}) + + def _relay(self, status: int, hdrs: List[Tuple[str, str]], data: bytes, backend: _Backend) -> Response: + out: Dict[str, str] = {} + for k, v in hdrs: + if k.lower() in FORWARD_RESPONSE_HEADERS: + out[k] = v + if self.cfg.router.expose_backend_header: + out["X-Igv-Backend"] = backend.label() + return Response(status, out, data) diff --git a/isongraph-vector-server/isongraph_vector_server/schemas.py b/isongraph-vector-server/isongraph_vector_server/schemas.py new file mode 100644 index 0000000..ea92ba6 --- /dev/null +++ b/isongraph-vector-server/isongraph_vector_server/schemas.py @@ -0,0 +1,215 @@ +"""Request body schemas (see ``validation.py`` for the dialect). + +Numeric caps here are hard ceilings; the per-tenant quotas (``max_top_k``, +``max_hops``, ``max_batch_items`` ...) are enforced by the handlers on top. +""" + +from __future__ import annotations + +from typing import Any, Dict + +NAME = {"kind": "graphname"} +TYPENAME = {"kind": "typename"} +PROPNAME = {"kind": "propname"} +ID = {"kind": "id"} +PROPS = {"kind": "props", "reserved": ["id"]} +EDGE_PROPS = {"kind": "props", "reserved": ["source", "target"]} +VECTOR = {"kind": "vector"} +EMBED_TEXT = {"kind": "text", "maxLength": 20_000} +QUERY_TEXT = {"kind": "text", "maxLength": 8_000} + +NODE_REF: Dict[str, Any] = { + "type": "object", + "required": ["type", "id"], + "properties": {"type": TYPENAME, "id": ID}, +} + +FILTER_OPS = ["eq", "ne", "lt", "lte", "gt", "gte", "in", "nin", "exists", "prefix", "contains"] + +CONDITION: Dict[str, Any] = { + "type": "object", + "required": ["field", "op"], + "properties": { + "field": PROPNAME, + "op": {"type": "string", "enum": FILTER_OPS}, + # any scalar, or (for in/nin) a short list of scalars; checked in filters.py + "value": {}, + }, +} + +FILTER: Dict[str, Any] = { + "type": "object", + "properties": { + "types": {"type": "array", "items": TYPENAME, "maxItems": 16}, + "where": {"type": "array", "items": CONDITION, "maxItems": 16}, + }, +} + +DIRECTION = {"type": "string", "enum": ["out", "in", "both"], "default": "out"} + +CREATE_GRAPH = { + "type": "object", + "required": ["name"], + "properties": { + "name": NAME, + "directed": {"type": "boolean", "default": True}, + "embedding": { + "type": "object", + "properties": { + "mode": {"type": "string", "enum": ["server", "client"], "default": "server"}, + "dimension": {"type": "integer", "minimum": 1, "maximum": 8192}, + }, + }, + "embed_fields": {"type": "array", "items": PROPNAME, "maxItems": 8, "minItems": 1}, + "ann": {"type": ["string", "null"], "enum": ["ivf", None]}, + "ann_nprobe": {"type": "integer", "minimum": 1, "maximum": 100_000}, + }, +} + +NODE_BODY = { + "type": "object", + "properties": { + "properties": PROPS, + "embed_text": EMBED_TEXT, + "vector": VECTOR, + }, +} + +NODE_UPSERT = { + "type": "object", + "required": ["type", "id"], + "properties": { + "type": TYPENAME, + "id": ID, + "properties": PROPS, + "embed_text": EMBED_TEXT, + "vector": VECTOR, + }, +} + +UPSERT_NODES = { + "type": "object", + "required": ["nodes"], + "properties": {"nodes": {"type": "array", "items": NODE_UPSERT, "minItems": 1, "maxItems": 10_000}}, +} + +PATCH_NODE = { + "type": "object", + "properties": { + "properties": PROPS, + "remove": {"type": "array", "items": PROPNAME, "maxItems": 64}, + "embed_text": EMBED_TEXT, + "vector": VECTOR, + }, +} + +DELETE_NODES = { + "type": "object", + "required": ["nodes"], + "properties": {"nodes": {"type": "array", "items": NODE_REF, "minItems": 1, "maxItems": 10_000}}, +} + +EDGE = { + "type": "object", + "required": ["rel", "source", "target"], + "properties": { + "rel": TYPENAME, + "source": NODE_REF, + "target": NODE_REF, + "properties": EDGE_PROPS, + }, +} + +EDGE_REF = { + "type": "object", + "required": ["rel", "source", "target"], + "properties": {"rel": TYPENAME, "source": NODE_REF, "target": NODE_REF}, +} + +UPSERT_EDGES = { + "type": "object", + "required": ["edges"], + "properties": {"edges": {"type": "array", "items": EDGE, "minItems": 1, "maxItems": 10_000}}, +} + +DELETE_EDGES = { + "type": "object", + "required": ["edges"], + "properties": {"edges": {"type": "array", "items": EDGE_REF, "minItems": 1, "maxItems": 10_000}}, +} + +_QUERY = { + "query": QUERY_TEXT, + "vector": VECTOR, +} + +SEARCH = { + "type": "object", + "properties": { + **_QUERY, + "top_k": {"type": "integer", "minimum": 1, "maximum": 10_000, "default": 10}, + "node_type": TYPENAME, + "threshold": {"type": "number", "minimum": -1.0, "maximum": 1.0, "default": 0.0}, + "filter": FILTER, + "exact": {"type": "boolean", "default": False}, + "include_properties": {"type": "boolean", "default": True}, + "include_text": {"type": "boolean", "default": False}, + }, +} + +_TRAVERSAL = { + **_QUERY, + "rel_type": TYPENAME, + "max_hops": {"type": "integer", "minimum": 0, "maximum": 50}, + "top_k_seeds": {"type": "integer", "minimum": 1, "maximum": 10_000}, + "direction": DIRECTION, + "decay": {"type": "number", "minimum": 0.0, "maximum": 1.0}, + "threshold": {"type": "number", "minimum": -1.0, "maximum": 1.0}, + "weight_field": PROPNAME, + "filter": FILTER, + "strict_filter": {"type": "boolean", "default": False}, + "include_properties": {"type": "boolean", "default": True}, +} + +MULTI_HOP = { + "type": "object", + "properties": { + **_TRAVERSAL, + "top_k_results": {"type": "integer", "minimum": 1, "maximum": 10_000}, + "blend": {"type": "number", "minimum": 0.0, "maximum": 1.0}, + }, +} + +PATH = { + "type": "object", + "required": ["target"], + "properties": {**_TRAVERSAL, "target": NODE_REF}, +} + +SUBGRAPH = { + "type": "object", + "properties": { + **_TRAVERSAL, + "top_k_results": {"type": "integer", "minimum": 1, "maximum": 10_000}, + "blend": {"type": "number", "minimum": 0.0, "maximum": 1.0}, + "connect": {"type": "boolean", "default": False}, + "connect_max_hops": {"type": "integer", "minimum": 1, "maximum": 3}, + }, +} + +SIMILAR = { + "type": "object", + "required": ["node"], + "properties": { + "node": NODE_REF, + "top_k": {"type": "integer", "minimum": 1, "maximum": 10_000, "default": 10}, + "node_type": TYPENAME, + "threshold": {"type": "number", "minimum": -1.0, "maximum": 1.0, "default": 0.0}, + "include_properties": {"type": "boolean", "default": True}, + }, +} + +REEMBED = { + "type": "object", + "properties": {"include_unknown": {"type": "boolean", "default": False}}, +} diff --git a/isongraph-vector-server/isongraph_vector_server/server.py b/isongraph-vector-server/isongraph_vector_server/server.py new file mode 100644 index 0000000..8c97098 --- /dev/null +++ b/isongraph-vector-server/isongraph_vector_server/server.py @@ -0,0 +1,124 @@ +"""Assemble and run a node (primary or replica) or a router.""" + +from __future__ import annotations + +import logging +import threading +from typing import Optional + +from isongraph_vector import EmbeddingEncoder + +from .auth import TenantRegistry +from .config import ServerConfig +from .encoders import BoundedEncoder, build_encoder +from .engine import Engine +from .httpd import BoundedHTTPServer +from .logs import log, setup_logging +from .metrics import Metrics +from .replication import ReplicaSyncer +from .router import Router +from .service import App + + +class Server: + """A storage node. ``start()`` returns once it is listening and ready.""" + + def __init__(self, cfg: ServerConfig, encoder: Optional[EmbeddingEncoder] = None, + replica_source=None, configure_logging: bool = True): + self.cfg = cfg.validate() + if configure_logging: + setup_logging(cfg.log_level) + self.metrics = Metrics() + if encoder is not None and not isinstance(encoder, BoundedEncoder): + encoder = BoundedEncoder(encoder, cfg.encoder.max_concurrency) + self.encoder = encoder if encoder is not None else build_encoder(cfg.encoder) + if self.encoder is not None: + # Learn an HTTP encoder's dimension now; an unreachable endpoint is logged, not fatal + # (graph creation and text requests then answer 503 encoder_failed until it is back). + try: + _ = self.encoder.dimension + except Exception as exc: + log(logging.WARNING, "embedding encoder not reachable at startup", error=type(exc).__name__) + self.engine = Engine(cfg, self.encoder, self.metrics) + self.registry = TenantRegistry(cfg.data_dir, cfg.quotas, cfg.registry_recheck_seconds) + self.replica: Optional[ReplicaSyncer] = None + self._replica_source = replica_source + self.app: Optional[App] = None + self.httpd: Optional[BoundedHTTPServer] = None + self._thread: Optional[threading.Thread] = None + + @property + def address(self): + return self.httpd.server_address[:2] + + @property + def url(self) -> str: + host, port = self.address + return f"http://{host}:{port}" + + def start(self) -> "Server": + self.engine.start() + self.registry.rescan(force=True) + if self.cfg.role == "replica": + self.replica = ReplicaSyncer(self.cfg, self.engine, self.registry, self.metrics, + source=self._replica_source) + self.app = App(self.cfg, self.engine, self.registry, self.metrics, self.replica) + self.httpd = BoundedHTTPServer((self.cfg.host, self.cfg.port), self.app, self.cfg) + self._thread = threading.Thread(target=self.httpd.serve_forever, kwargs={"poll_interval": 0.1}, + name="igv-accept", daemon=True) + self._thread.start() + if self.replica is not None and self.cfg.replica.autostart: + self.replica.start() + log(logging.INFO, "listening", role=self.cfg.role, host=self.address[0], port=self.address[1], + graphs_loaded=self.engine.startup_report["loaded"], + graphs_quarantined=self.engine.startup_report["quarantined"]) + return self + + def stop(self) -> None: + """Graceful shutdown: stop accepting, finish in-flight requests, final snapshot.""" + if self.replica is not None: + self.replica.stop() + if self.httpd is not None: + self.httpd.stop() + self.engine.shutdown() + log(logging.INFO, "stopped") + + def wait(self) -> None: + if self._thread is not None: + self._thread.join() + + +class RouterServer: + def __init__(self, cfg: ServerConfig, configure_logging: bool = True): + self.cfg = cfg.validate() + if configure_logging: + setup_logging(cfg.log_level) + self.router = Router(cfg) + self.httpd: Optional[BoundedHTTPServer] = None + self._thread: Optional[threading.Thread] = None + + @property + def address(self): + return self.httpd.server_address[:2] + + @property + def url(self) -> str: + host, port = self.address + return f"http://{host}:{port}" + + def start(self) -> "RouterServer": + self.httpd = BoundedHTTPServer((self.cfg.host, self.cfg.port), self.router, self.cfg) + self._thread = threading.Thread(target=self.httpd.serve_forever, kwargs={"poll_interval": 0.1}, + name="igv-router-accept", daemon=True) + self._thread.start() + log(logging.INFO, "router listening", host=self.address[0], port=self.address[1], + shards=len(self.cfg.router.shards)) + return self + + def stop(self) -> None: + if self.httpd is not None: + self.httpd.stop() + + def wait(self) -> None: + if self._thread is not None: + self._thread.join() diff --git a/isongraph-vector-server/isongraph_vector_server/service.py b/isongraph-vector-server/isongraph_vector_server/service.py new file mode 100644 index 0000000..36d5736 --- /dev/null +++ b/isongraph-vector-server/isongraph_vector_server/service.py @@ -0,0 +1,588 @@ +"""The request pipeline: routing, authentication, limits, validation, handlers. + +Order of operations for a tenant request (each step can end the request): + + 1. split and validate the path (decoded once, segment-wise; ``.``/``..``, + separators and control characters are rejected) + 2. match a route (404 / 405) + 3. authenticate: exactly one ``Authorization: Bearer <key>`` header; failures + are throttled per peer address; the tenant is whatever the *verified key* + says and nothing else - no header, query parameter or body field selects it + 4. role check (read-only key, read-only replica, replica too stale) + 5. per-tenant rate limit, then global and per-tenant concurrency gates + 6. query-string and JSON body validation against the route's schema + 7. the handler, which only ever sees the authenticated tenant's runtime +""" + +from __future__ import annotations + +import hashlib +import hmac +import json +import logging +import os +import secrets +import time +import urllib.parse +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Callable, Dict, List, Optional, Tuple + +from . import names, ops, schemas +from .auth import Principal, TenantRegistry +from .config import ServerConfig +from .engine import Engine, OpContext +from .errors import (ApiError, BadRequest, Forbidden, MethodNotAllowed, NotFound, Overloaded, + PayloadTooLarge, Unauthorized, Unavailable, UnsupportedMediaType, + validation_error) +from .httpd import Request, Response +from .limits import Gate, SlidingWindow, rate_limited +from .logs import log +from .metrics import Metrics +from .replication import build_index +from .names import GENERATION_FILES +from .validation import validate + +QUERY_MAX_PARAMS = 16 + + +# -- request parsing ---------------------------------------------------------------------------------- + + +def split_path(target: str) -> Tuple[List[str], Dict[str, str]]: + path, _, qs = target.partition("?") + if "#" in path or not path.startswith("/") or len(path) > 2048: + raise BadRequest("invalid request path", code="invalid_path") + segs: List[str] = [] + if path != "/": + for raw in path[1:].split("/"): + if raw == "": + raise NotFound("no such resource") + try: + seg = urllib.parse.unquote(raw, errors="strict") + except UnicodeDecodeError: + raise BadRequest("invalid request path", code="invalid_path") from None + if (seg in (".", "..") or "/" in seg or "\\" in seg + or any(ord(c) < 32 or ord(c) == 127 for c in seg) or len(seg) > 256): + raise BadRequest("invalid request path", code="invalid_path") + segs.append(seg) + query: Dict[str, str] = {} + if qs: + try: + pairs = urllib.parse.parse_qsl(qs, keep_blank_values=True, errors="strict", + max_num_fields=QUERY_MAX_PARAMS) + except (UnicodeDecodeError, ValueError): + raise BadRequest("invalid query string", code="invalid_query") from None + for k, v in pairs: + if k in query: + raise BadRequest(f"query parameter {k[:40]!r} given twice", code="invalid_query") + query[k] = v + return segs, query + + +def _no_dupes(pairs): + out = {} + for k, v in pairs: + if k in out: + raise ValueError("duplicate key") + out[k] = v + return out + + +def _reject_constant(name): + raise ValueError("non-finite number") + + +def parse_json_body(req: Request, optional: bool) -> Any: + if not req.body: + if optional: + return {} + raise BadRequest("a JSON body is required", code="invalid_json") + ctype = (req.header("content-type") or "").split(";")[0].strip().lower() + if ctype != "application/json": + raise UnsupportedMediaType("Content-Type must be application/json") + try: + text = req.body.decode("utf-8") + data = json.loads(text, object_pairs_hook=_no_dupes, parse_constant=_reject_constant) + except (UnicodeDecodeError, ValueError, RecursionError): + raise BadRequest("the body is not valid JSON (duplicate keys, NaN/Infinity and " + "invalid UTF-8 are rejected)", code="invalid_json") from None + return data + + +def _token_matches(presented: Optional[str], expected: Optional[str]) -> bool: + if not expected or presented is None: + return False + a = hashlib.sha256(presented.encode("utf-8", "replace")).digest() + b = hashlib.sha256(expected.encode("utf-8")).digest() + return hmac.compare_digest(a, b) + + +def bearer_token(req: Request) -> Optional[str]: + vals = req.header_all("authorization") + if len(vals) != 1: + return None + v = vals[0] + if len(v) < 8 or v[:7].lower() != "bearer ": + return None + return v[7:].strip() + + +# -- routes ---------------------------------------------------------------------------------------------- + + +@dataclass +class Route: + method: str + pattern: str + name: str + fn: Callable + auth: str = "tenant" # none | tenant | metrics | repl + write: bool = False + schema: Optional[dict] = None + query: Tuple[str, ...] = () + body_optional: bool = False + segs: List[str] = field(default_factory=list) + + def __post_init__(self): + self.segs = [s for s in self.pattern.split("/") if s] + + +@dataclass +class Call: + req: Request + principal: Optional[Principal] + env: Optional[ops.OpEnv] + params: Dict[str, str] + query: Dict[str, str] + body: Any + headers: Dict[str, str] = field(default_factory=dict) + request_id: str = "" + + +class App: + def __init__(self, cfg: ServerConfig, engine: Engine, registry: TenantRegistry, + metrics: Metrics, replica: Any = None): + self.cfg = cfg + self.engine = engine + self.registry = registry + self.metrics = metrics + self.replica = replica # ReplicaSyncer or None + self.global_gate = Gate(cfg.max_concurrent_requests) + self.auth_failures = SlidingWindow(cfg.failed_auth_per_minute) + self.routes: List[Route] = [] + self._build_routes() + + # ---- route table --------------------------------------------------------------------------------- + + def _build_routes(self) -> None: + R = self._add + R("GET", "/healthz", "healthz", self.h_healthz, auth="none") + R("GET", "/readyz", "readyz", self.h_readyz, auth="none") + R("GET", "/metrics", "metrics", self.h_metrics, auth="metrics") + R("GET", "/v1/usage", "usage", self.h_usage) + R("GET", "/v1/graphs", "graphs.list", self.h_graphs_list, query=("limit", "cursor")) + R("POST", "/v1/graphs", "graphs.create", self.h_graph_create, write=True, schema=schemas.CREATE_GRAPH) + g = "/v1/graphs/{g}" + R("GET", g, "graphs.get", self.h_graph_get) + R("DELETE", g, "graphs.delete", self.h_graph_delete, write=True) + R("POST", g + "/save", "graphs.save", self.h_graph_save, write=True, body_optional=True, + schema={"type": "object"}) + R("GET", g + "/nodes", "nodes.list", self.h_nodes_list, query=("type", "limit", "cursor")) + R("POST", g + "/nodes", "nodes.upsert", self.h_nodes_upsert, write=True, schema=schemas.UPSERT_NODES) + R("POST", g + "/nodes/delete", "nodes.delete_batch", self.h_nodes_delete, write=True, + schema=schemas.DELETE_NODES) + n = g + "/nodes/{t}/{id}" + R("GET", n, "node.get", self.h_node_get, query=("include_vector",)) + R("PUT", n, "node.put", self.h_node_put, write=True, schema=schemas.NODE_BODY) + R("PATCH", n, "node.patch", self.h_node_patch, write=True, schema=schemas.PATCH_NODE) + R("DELETE", n, "node.delete", self.h_node_delete, write=True) + R("GET", g + "/edges", "edges.list", self.h_edges_list, query=("rel", "limit", "cursor")) + R("POST", g + "/edges", "edges.upsert", self.h_edges_upsert, write=True, schema=schemas.UPSERT_EDGES) + R("POST", g + "/edges/delete", "edges.delete_batch", self.h_edges_delete, write=True, + schema=schemas.DELETE_EDGES) + R("POST", g + "/search", "search", self.h_search, schema=schemas.SEARCH) + R("POST", g + "/multi_hop", "multi_hop", self.h_multi_hop, schema=schemas.MULTI_HOP) + R("POST", g + "/path", "path", self.h_path, schema=schemas.PATH) + R("POST", g + "/subgraph", "subgraph", self.h_subgraph, schema=schemas.SUBGRAPH) + R("POST", g + "/similar", "similar", self.h_similar, schema=schemas.SIMILAR) + R("GET", g + "/stale", "stale", self.h_stale, query=("include_unknown", "limit", "cursor")) + R("POST", g + "/reembed", "reembed", self.h_reembed, write=True, schema=schemas.REEMBED, + body_optional=True) + if self.cfg.role == "primary": + R("GET", "/_repl/v1/index", "repl.index", self.h_repl_index, auth="repl") + R("GET", "/_repl/v1/tenant/{t}", "repl.tenant", self.h_repl_tenant, auth="repl") + R("GET", "/_repl/v1/meta/{t}/{g}", "repl.meta", self.h_repl_meta, auth="repl") + R("GET", "/_repl/v1/file/{t}/{g}/{gen}/{f}", "repl.file", self.h_repl_file, auth="repl") + + def _add(self, method, pattern, name, fn, **kw) -> None: + self.routes.append(Route(method, pattern, name, fn, **kw)) + + def _match(self, method: str, segs: List[str]): + allowed = [] + for route in self.routes: + if len(route.segs) != len(segs): + continue + params: Dict[str, str] = {} + ok = True + for pat, seg in zip(route.segs, segs): + if pat.startswith("{"): + params[pat[1:-1]] = seg + elif pat != seg: + ok = False + break + if not ok: + continue + if route.method == method: + return route, params, None + allowed.append(route.method) + return None, None, allowed + + # ---- entry point ------------------------------------------------------------------------------------- + + def handle(self, req: Request) -> Response: + t0 = time.monotonic() + rid = req.header("x-request-id") + if rid is None or not names.REQUEST_ID_RE.match(rid): + rid = secrets.token_hex(8) + info: Dict[str, Any] = {"route": "unmatched", "tenant": None} + try: + resp = self._dispatch(req, rid, info) + except ApiError as exc: + resp = Response.json(exc.status, exc.to_body(rid), exc.headers) + except Exception: + logging.getLogger("igv").exception("unhandled error", extra={"fields": {"request_id": rid}}) + resp = Response.json(500, ApiError("an internal error occurred").to_body(rid)) + resp.headers["X-Request-Id"] = rid + elapsed = time.monotonic() - t0 + self.metrics.inc("igv_requests_total", route=info["route"], status=f"{resp.status // 100}xx") + self.metrics.observe(info["route"], elapsed) + if resp.status >= 500 or self.cfg.log_level.upper() == "DEBUG" or \ + logging.getLogger("igv").isEnabledFor(logging.INFO): + log(logging.WARNING if resp.status >= 500 else logging.INFO, "request", + method=req.method, route=info["route"], status=resp.status, + ms=round(elapsed * 1000, 2), tenant=info["tenant"], request_id=rid) + return resp + + # ---- pipeline -------------------------------------------------------------------------------------------- + + def _client_key(self, req: Request) -> str: + if req.peer in self.cfg.trusted_proxies: + xff = req.header_all("x-forwarded-for") + if len(xff) == 1: + last = xff[0].split(",")[-1].strip() + if 0 < len(last) <= 64: + return last + return req.peer + + def _dispatch(self, req: Request, rid: str, info: Dict[str, Any]) -> Response: + segs, query = split_path(req.target) + route, params, allowed = self._match(req.method, segs) + if route is None: + if allowed: + raise MethodNotAllowed("method not allowed for this resource", + headers={"Allow": ", ".join(sorted(set(allowed)))}) + raise NotFound("no such resource") + info["route"] = route.name + + principal: Optional[Principal] = None + if route.auth == "tenant": + principal = self._authenticate(req) + info["tenant"] = principal.tenant_id + elif route.auth in ("metrics", "repl"): + expected = self.cfg.metrics_token if route.auth == "metrics" else self.cfg.replication_token + if not expected: + raise NotFound("no such resource") # not configured: the endpoint does not exist + client = self._client_key(req) + if self.auth_failures.blocked(client): + raise ApiError("too many failed authentication attempts", status=429, + code="rate_limited", headers={"Retry-After": "30"}) + if not _token_matches(bearer_token(req), expected): + self.auth_failures.record(client) + self.metrics.inc("igv_auth_failures_total") + raise self._unauth() + + if route.auth == "tenant" and not self.engine.ready: + raise Unavailable("server is starting", code="not_ready", headers={"Retry-After": "2"}) + + extra = set(query) - set(route.query) + if extra: + raise BadRequest(f"unknown query parameter {sorted(extra)[0][:40]!r}", code="invalid_query") + + if principal is None: + call = Call(req, None, None, params, query, None, request_id=rid) + return route.fn(call) + + cfg = self.cfg + if route.write: + if cfg.role == "replica": + raise Forbidden("this node is a read-only replica; send writes to the primary", + code="read_only_replica") + if principal.role != "rw": + raise Forbidden("this API key is read-only", code="read_only_key") + if cfg.role == "replica" and self.replica is not None: + self.replica.check_fresh() + + rt = self.engine.runtime(principal.tenant_id, principal.quotas) + rt.requests += 1 + wait = rt.bucket.take(1 + len(req.body) // 8192) + if wait: + rt.rate_limited += 1 + self.metrics.inc("igv_rate_limited_total") + raise rate_limited(wait) + + tenant_busy = Overloaded("this tenant has too many requests in flight", code="tenant_overloaded", + headers={"Retry-After": "1"}) + with self.global_gate.enter(cfg.queue_wait_seconds), \ + rt.gate.enter(cfg.queue_wait_seconds, tenant_busy): + body = None + if route.schema is not None: + raw = parse_json_body(req, route.body_optional) + body = validate(raw, route.schema) + ctx = OpContext(deadline=time.monotonic() + cfg.request_deadline_seconds, + lock_timeout=cfg.lock_timeout_seconds) + env = ops.OpEnv(self.engine, rt, principal.quotas, ctx) + call = Call(req, principal, env, params, query, body, request_id=rid) + return route.fn(call) + + def _unauth(self) -> Unauthorized: + return Unauthorized("missing or invalid API key", headers={"WWW-Authenticate": 'Bearer realm="igv"'}) + + def _authenticate(self, req: Request) -> Principal: + client = self._client_key(req) + if self.auth_failures.blocked(client): + self.metrics.inc("igv_auth_throttled_total") + raise ApiError("too many failed authentication attempts", status=429, code="rate_limited", + headers={"Retry-After": "30"}) + token = bearer_token(req) + principal = self.registry.authenticate(token) if token else None + if principal is None: + self.auth_failures.record(client) + self.metrics.inc("igv_auth_failures_total") + log(logging.WARNING, "authentication failed", peer=client) + raise self._unauth() + return principal + + # ---- response helpers ----------------------------------------------------------------------------------------- + + def _ok(self, call: Call, obj: Any, status: int = 200) -> Response: + data = json.dumps(obj, separators=(",", ":"), allow_nan=False).encode("utf-8") + if len(data) > self.cfg.max_response_bytes: + raise ApiError("the response would be too large; lower limit/top_k", status=413, + code="response_too_large") + headers = {"Content-Type": "application/json"} + headers.update(call.headers) + return Response(status, headers, data) + + def _handle(self, call: Call, name: str): + h = self.engine.get_handle(call.env.rt, name) + if self.cfg.role == "replica": + call.headers["X-Snapshot-Generation"] = str(h.generation) + if h.synced_at is not None: + call.headers["X-Snapshot-Age-Seconds"] = str(int(max(0, time.time() - h.synced_at))) + return h + + @staticmethod + def _graph_name(call: Call) -> str: + name = call.params["g"] + if not names.is_graph_name(name): + raise validation_error("path.graph", "invalid graph name") + return name + + @staticmethod + def _node_params(call: Call): + t, raw = call.params["t"], call.params["id"] + if not names.TYPE_RE.match(t): + raise validation_error("path.type", "invalid node type") + try: + return t, names.normalize_id(raw) + except ValueError as exc: + raise validation_error("path.id", str(exc)) from None + + @staticmethod + def _bool_q(call: Call, key: str) -> bool: + v = call.query.get(key) + if v is None: + return False + if v.lower() in ("1", "true"): + return True + if v.lower() in ("0", "false"): + return False + raise BadRequest(f"{key} must be true or false", code="invalid_query") + + # ---- unauthenticated ------------------------------------------------------------------------------------------- + + def h_healthz(self, call: Call) -> Response: + return Response.json(200, {"status": "ok"}) + + def h_readyz(self, call: Call) -> Response: + ready = self.engine.ready and (self.replica is None or self.replica.is_ready()) + return Response.json(200 if ready else 503, + {"status": "ready" if ready else "not_ready", "role": self.cfg.role}) + + def h_metrics(self, call: Call) -> Response: + self.metrics.gauge("igv_in_flight_requests", self.global_gate.in_flight) + self.metrics.gauge("igv_ready", 1 if self.engine.ready else 0) + if self.replica is not None: + for k, v in self.replica.metrics_snapshot().items(): + self.metrics.gauge(k, v) + return Response(200, {"Content-Type": "text/plain; version=0.0.4"}, + self.metrics.render().encode("utf-8")) + + # ---- tenant: graphs -------------------------------------------------------------------------------------------- + + def h_usage(self, call: Call) -> Response: + return self._ok(call, ops.usage(call.env)) + + def h_graphs_list(self, call: Call) -> Response: + limit = ops.page_limit(call.env, call.query.get("limit")) + return self._ok(call, ops.list_graphs(call.env, call.query.get("cursor"), limit)) + + def h_graph_create(self, call: Call) -> Response: + return self._ok(call, ops.create_graph(call.env, call.body), 201) + + def h_graph_get(self, call: Call) -> Response: + return self._ok(call, ops.get_graph(call.env, self._handle(call, self._graph_name(call)))) + + def h_graph_delete(self, call: Call) -> Response: + ops.delete_graph(call.env, self._graph_name(call)) + return self._ok(call, {"deleted": True}) + + def h_graph_save(self, call: Call) -> Response: + return self._ok(call, ops.save_graph(call.env, self._handle(call, self._graph_name(call)))) + + # ---- tenant: nodes ---------------------------------------------------------------------------------------------- + + def h_nodes_list(self, call: Call) -> Response: + t = call.query.get("type") + if t is not None and not names.TYPE_RE.match(t): + raise BadRequest("invalid node type", code="invalid_query") + limit = ops.page_limit(call.env, call.query.get("limit")) + return self._ok(call, ops.list_nodes(call.env, self._handle(call, self._graph_name(call)), + t, call.query.get("cursor"), limit)) + + def h_nodes_upsert(self, call: Call) -> Response: + h = self._handle(call, self._graph_name(call)) + return self._ok(call, ops.upsert_nodes(call.env, h, call.body["nodes"])) + + def h_nodes_delete(self, call: Call) -> Response: + h = self._handle(call, self._graph_name(call)) + return self._ok(call, ops.delete_nodes(call.env, h, call.body["nodes"])) + + def h_node_get(self, call: Call) -> Response: + t, i = self._node_params(call) + h = self._handle(call, self._graph_name(call)) + return self._ok(call, ops.get_node(call.env, h, t, i, self._bool_q(call, "include_vector"))) + + def h_node_put(self, call: Call) -> Response: + t, i = self._node_params(call) + h = self._handle(call, self._graph_name(call)) + item = {"type": t, "id": i, **call.body} + return self._ok(call, ops.upsert_nodes(call.env, h, [item])) + + def h_node_patch(self, call: Call) -> Response: + t, i = self._node_params(call) + h = self._handle(call, self._graph_name(call)) + return self._ok(call, ops.patch_node(call.env, h, t, i, call.body)) + + def h_node_delete(self, call: Call) -> Response: + t, i = self._node_params(call) + h = self._handle(call, self._graph_name(call)) + out = ops.delete_nodes(call.env, h, [{"type": t, "id": i}]) + if not out["deleted"]: + raise NotFound("node not found") + return self._ok(call, out) + + # ---- tenant: edges ---------------------------------------------------------------------------------------------- + + def h_edges_list(self, call: Call) -> Response: + rel = call.query.get("rel") + if rel is not None and not names.TYPE_RE.match(rel): + raise BadRequest("invalid relationship type", code="invalid_query") + limit = ops.page_limit(call.env, call.query.get("limit")) + return self._ok(call, ops.list_edges(call.env, self._handle(call, self._graph_name(call)), + rel, call.query.get("cursor"), limit)) + + def h_edges_upsert(self, call: Call) -> Response: + h = self._handle(call, self._graph_name(call)) + return self._ok(call, ops.upsert_edges(call.env, h, call.body["edges"])) + + def h_edges_delete(self, call: Call) -> Response: + h = self._handle(call, self._graph_name(call)) + return self._ok(call, ops.delete_edges(call.env, h, call.body["edges"])) + + # ---- tenant: retrieval ------------------------------------------------------------------------------------------- + + def h_search(self, call: Call) -> Response: + return self._ok(call, ops.search(call.env, self._handle(call, self._graph_name(call)), call.body)) + + def h_multi_hop(self, call: Call) -> Response: + return self._ok(call, ops.multi_hop(call.env, self._handle(call, self._graph_name(call)), call.body)) + + def h_path(self, call: Call) -> Response: + return self._ok(call, ops.path(call.env, self._handle(call, self._graph_name(call)), call.body)) + + def h_subgraph(self, call: Call) -> Response: + return self._ok(call, ops.subgraph(call.env, self._handle(call, self._graph_name(call)), call.body)) + + def h_similar(self, call: Call) -> Response: + return self._ok(call, ops.similar(call.env, self._handle(call, self._graph_name(call)), call.body)) + + def h_stale(self, call: Call) -> Response: + h = self._handle(call, self._graph_name(call)) + limit = ops.page_limit(call.env, call.query.get("limit")) + cur = ops.decode_cursor(call.query.get("cursor"), 1) + try: + offset = int(cur[0]) if cur else 0 + except (TypeError, ValueError): + raise BadRequest("invalid cursor", code="invalid_cursor") from None + return self._ok(call, ops.stale(call.env, h, self._bool_q(call, "include_unknown"), limit, max(0, offset))) + + def h_reembed(self, call: Call) -> Response: + h = self._handle(call, self._graph_name(call)) + return self._ok(call, ops.reembed(call.env, h, call.body.get("include_unknown", False))) + + # ---- replication source (primary only; separate credential) --------------------------------------------------- + + def _repl_ids(self, call: Call, with_graph: bool = True): + t = call.params["t"] + if not names.is_tenant_id(t): + raise NotFound("no such resource") + g = call.params.get("g") + if with_graph and (g is None or not names.is_graph_name(g)): + raise NotFound("no such resource") + return t, g + + def h_repl_index(self, call: Call) -> Response: + return self._ok(call, build_index(self.cfg.data_dir)) + + def _stream(self, path: Path) -> Response: + try: + size = path.stat().st_size + handle = open(path, "rb") + except OSError: + raise NotFound("no such resource") from None + + def chunks(): + with handle: + while True: + data = handle.read(1 << 20) + if not data: + return + yield data + return Response(200, {"Content-Type": "application/octet-stream"}, stream=chunks(), length=size) + + def h_repl_tenant(self, call: Call) -> Response: + t, _ = self._repl_ids(call, with_graph=False) + return self._stream(names.tenant_dir(self.cfg.data_dir, t) / "tenant.json") + + def h_repl_meta(self, call: Call) -> Response: + t, g = self._repl_ids(call) + return self._stream(names.graph_dir(self.cfg.data_dir, t, g) / "meta.json") + + def h_repl_file(self, call: Call) -> Response: + t, g = self._repl_ids(call) + gen, fname = call.params["gen"], call.params["f"] + if not names.GENERATION_RE.match(gen) or fname not in GENERATION_FILES: + raise NotFound("no such resource") + return self._stream(names.graph_dir(self.cfg.data_dir, t, g) / f"gen-{gen}" / fname) diff --git a/isongraph-vector-server/isongraph_vector_server/storage.py b/isongraph-vector-server/isongraph_vector_server/storage.py new file mode 100644 index 0000000..b4a61bb --- /dev/null +++ b/isongraph-vector-server/isongraph_vector_server/storage.py @@ -0,0 +1,545 @@ +"""Durable, verifiable snapshots of one graph. + +On-disk layout of a graph (``<data>/tenants/<tenant>/graphs/g-<name>/``):: + + meta.json immutable graph configuration (written once, atomically) + CURRENT text file: the generation number to serve + gen-000007/ + graph.isong the graph (ISON text) + graph.isong.embeddings.json the vectors, tagged with the graph's digest + MANIFEST.json generation, graph_digest, counts, sha256 + size of each file + gen-000006/ ... older generations kept for fallback and for replicas + +Why generations instead of overwriting ``graph.isong`` in place: the library +writes the graph file and the embeddings sidecar as two separate atomic +renames, so a crash between them leaves a *pair* that does not belong +together (the library detects this on load, by digest, but cannot repair it). +Here the pair is built completely in a temporary directory, the directory is +renamed into place, and only then is ``CURRENT`` atomically flipped. A crash +at any point leaves ``CURRENT`` naming a complete, verified generation. + +Verification on load, in order: every file's size and sha256 against the +manifest; the library's own sidecar-vs-graph digest check +(``on_mismatch="error"``); the sidecar's recorded digest against the +manifest's; and the digest of the loaded graph against the manifest's. If the +generation ``CURRENT`` names fails any of them, the newest older generation +that passes is served instead and the graph is flagged ``degraded``; if none +passes the graph is quarantined (not served, not deleted). +""" + +from __future__ import annotations + +import hashlib +import json +import os +import shutil +import threading +import time +import uuid +import warnings +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Callable, Dict, List, Optional, Tuple + +import hashlib as _hashlib + +from isongraph_vector import ( + GRAPH_DIGEST_BASIS, EmbeddingEncoder, EmbeddingError, NoEncoderError, SemanticGraph, + graph_digest, +) +from ison_graph import ISONGraph + +from . import names + +META_FORMAT = "igv-graph" +MANIFEST_FORMAT = "igv-snapshot" +FORMAT_VERSION = 1 +GRAPH_FILE = "graph.isong" +SIDECAR_FILE = "graph.isong.embeddings.json" +MANIFEST_FILE = "MANIFEST.json" + +#: Test seam for crash-safety tests: called with a point name as a save +#: progresses. Production code never sets it. (A child process in the test +#: suite installs a hook that announces the point and then sleeps, and the +#: parent SIGKILLs it there.) +fault_hook: Optional[Callable[[str], None]] = None + +_LOAD_WARN_LOCK = threading.Lock() + + +class SnapshotError(Exception): + """A snapshot could not be written, or failed verification.""" + + +class ClientModeEncoder(EmbeddingEncoder): + """Placeholder encoder for graphs whose vectors are supplied by the caller. + + Without it the library would lazily build a SentenceTransformer (and + download a model) the first time anything asked for an encoder. This one + refuses to encode, and stamps stored vectors with the model name + ``"provided"``, which the library treats as "unknown provenance". + """ + + model_name = "provided" + + @property + def dimension(self) -> int: + raise AttributeError("client-mode graphs have no encoder dimension") + + def encode(self, text: str): + raise NoEncoderError("this graph takes caller-supplied vectors; it has no encoder") + + def encode_batch(self, texts): + raise NoEncoderError("this graph takes caller-supplied vectors; it has no encoder") + + +def _fault(point: str) -> None: + hook = fault_hook + if hook is not None: + hook(point) + + +def sha256_file(path: Path) -> Tuple[str, int]: + h = hashlib.sha256() + size = 0 + with open(path, "rb") as handle: + while True: + chunk = handle.read(1 << 20) + if not chunk: + break + h.update(chunk) + size += len(chunk) + return h.hexdigest(), size + + +def fsync_dir(path: Path) -> None: + """Make a rename in ``path`` durable. A no-op where directories cannot be opened (Windows).""" + try: + fd = os.open(path, os.O_RDONLY) + except OSError: + return + try: + os.fsync(fd) + except OSError: + pass + finally: + os.close(fd) + + +def write_bytes_durably(path: Path, data: bytes) -> None: + with open(path, "wb") as handle: + handle.write(data) + handle.flush() + os.fsync(handle.fileno()) + + +def atomic_write(path: Path, data: bytes) -> None: + tmp = path.with_name(f".{path.name}.{os.getpid()}.{uuid.uuid4().hex[:8]}.tmp") + try: + write_bytes_durably(tmp, data) + if path.name == "CURRENT": + _fault("current_tmp_written") + os.replace(tmp, path) + except BaseException: + try: + tmp.unlink() + except OSError: + pass + raise + fsync_dir(path.parent) + + +@dataclass +class GraphMeta: + name: str + mode: str # "server" | "client" + dimension: Optional[int] # fixed for client mode; server mode follows the encoder + embed_fields: List[str] + ann: Optional[str] = None + ann_nprobe: Optional[int] = None + created_at: float = field(default_factory=time.time) + + def to_json(self) -> Dict[str, Any]: + return { + "format": META_FORMAT, "version": FORMAT_VERSION, "name": self.name, + "directed": True, "mode": self.mode, "dimension": self.dimension, + "embed_fields": self.embed_fields, "ann": self.ann, + "ann_nprobe": self.ann_nprobe, "created_at": self.created_at, + } + + @staticmethod + def from_json(data: Any, expect_name: str) -> "GraphMeta": + if (not isinstance(data, dict) or data.get("format") != META_FORMAT + or data.get("version") != FORMAT_VERSION): + raise SnapshotError("meta.json is not a graph meta document") + if data.get("name") != expect_name or not names.is_graph_name(expect_name): + raise SnapshotError("meta.json name does not match its directory") + if data.get("mode") not in ("server", "client"): + raise SnapshotError("meta.json has a bad mode") + fields = data.get("embed_fields") + if (not isinstance(fields, list) or not fields + or not all(isinstance(f, str) and names.PROP_RE.match(f) for f in fields)): + raise SnapshotError("meta.json has bad embed_fields") + return GraphMeta( + name=expect_name, mode=data["mode"], dimension=data.get("dimension"), + embed_fields=list(fields), ann=data.get("ann"), + ann_nprobe=data.get("ann_nprobe"), created_at=float(data.get("created_at", 0)), + ) + + +def new_semantic_graph(meta: GraphMeta, encoder: Optional[EmbeddingEncoder]) -> SemanticGraph: + """An empty in-memory graph configured the way the server always runs them. + + ``auto_embed`` is off: the server computes embeddings itself, outside the + graph lock, and hands the library finished vectors. + """ + enc = encoder if meta.mode == "server" else ClientModeEncoder() + return SemanticGraph( + name=meta.name, directed=True, embedding_db=":memory:", encoder=enc, + auto_embed=False, embed_fields=list(meta.embed_fields), + on_model_mismatch="ignore", ann=meta.ann, ann_nprobe=meta.ann_nprobe, + ) + + +@dataclass +class Manifest: + generation: int + graph_digest: str + node_count: int + edge_count: int + embedding_count: int + dimension: Optional[int] + files: Dict[str, Dict[str, Any]] + created_at: float + + def to_json(self) -> Dict[str, Any]: + return { + "format": MANIFEST_FORMAT, "version": FORMAT_VERSION, + "generation": self.generation, "graph_digest": self.graph_digest, + "node_count": self.node_count, "edge_count": self.edge_count, + "embedding_count": self.embedding_count, "dimension": self.dimension, + "files": self.files, "created_at": self.created_at, + } + + @staticmethod + def from_json(data: Any) -> "Manifest": + try: + if (data["format"] != MANIFEST_FORMAT or data["version"] != FORMAT_VERSION): + raise SnapshotError("unsupported manifest") + files = data["files"] + for fname in (GRAPH_FILE, SIDECAR_FILE): + entry = files[fname] + if not (isinstance(entry["sha256"], str) and len(entry["sha256"]) == 64 + and isinstance(entry["size"], int)): + raise SnapshotError("bad manifest file entry") + digest = data["graph_digest"] + if not (isinstance(digest, str) and digest.startswith("sha256:")): + raise SnapshotError("bad manifest digest") + return Manifest( + generation=int(data["generation"]), graph_digest=digest, + node_count=int(data["node_count"]), edge_count=int(data["edge_count"]), + embedding_count=int(data["embedding_count"]), dimension=data.get("dimension"), + files={k: files[k] for k in (GRAPH_FILE, SIDECAR_FILE)}, + created_at=float(data.get("created_at", 0)), + ) + except SnapshotError: + raise + except (KeyError, TypeError, ValueError) as exc: + raise SnapshotError(f"malformed manifest ({type(exc).__name__})") from None + + +def semantic_digest(graph: ISONGraph) -> str: + """Like ``graph_digest`` but ignoring null-valued properties. + + The ISON text format is columnar: a property one node has and another + lacks comes back from a save/load as an explicit ``null`` on the other. + Absent and null are therefore the same thing to this service (it never + stores or returns null properties). This digest is what "the file means + the same as memory" is checked with; the manifest still records the + library's own ``graph_digest`` of the file, which is what the sidecar + carries and what replicas verify. + """ + def ref_key(ref): + return [ref[0], 'i' if isinstance(ref[1], int) and not isinstance(ref[1], bool) + else 's', str(ref[1])] + + def dump(obj): + return json.dumps(obj, sort_keys=True, separators=(',', ':'), ensure_ascii=True, default=str) + + def live(props): + return {k: v for k, v in props.items() if v is not None} + + nodes = sorted(dump([ref_key(n.ref), live(n.properties)]) for n in graph.nodes()) + edges = sorted(dump([e.rel_type, ref_key(e.source), ref_key(e.target), live(e.properties)]) + for e in graph.edges()) + body = "|".join([str(graph.directed), chr(10).join(nodes), chr(10).join(edges)]) + return "sem:" + _hashlib.sha256(body.encode("utf-8")).hexdigest() + + +@dataclass +class Serialized: + """A consistent in-memory image of a graph, ready to be written without any lock.""" + graph_text: str + payload: Dict[str, Any] # the embeddings sidecar, minus the graph digest + semantic: str + node_count: int + edge_count: int + embedding_count: int + dimension: Optional[int] + + +def serialize(graph: SemanticGraph) -> Serialized: + """Capture graph + embeddings. The caller holds the graph's read lock. + + Only memory-to-memory work happens here (text rendering, a digest, the + vector export); the expensive parts - re-parsing, hashing the files, + JSON-encoding the vectors, fsync - run afterwards, outside the lock, in + ``GraphStorage.write_generation``. + """ + payload = graph.embedding_store.export_embeddings() + return Serialized( + graph_text=graph.to_ison(), payload=payload, semantic=semantic_digest(graph), + node_count=graph.node_count(), edge_count=graph.edge_count(), + embedding_count=payload["count"], dimension=payload.get("dimension"), + ) + + +class GraphStorage: + """All filesystem access for one graph directory.""" + + def __init__(self, directory: Path, keep_generations: int = 3): + self.dir = Path(directory) + self.keep = max(2, keep_generations) + + # -- small helpers ---------------------------------------------------------- + + def gen_path(self, generation: int) -> Path: + return self.dir / f"gen-{generation:06d}" + + def read_current(self) -> Optional[int]: + try: + text = (self.dir / "CURRENT").read_text(encoding="ascii").strip() + except (OSError, UnicodeDecodeError): + return None + return int(text) if names.GENERATION_RE.match(text) else None + + def generations(self) -> List[int]: + out = [] + try: + for entry in os.scandir(self.dir): + if entry.is_dir(follow_symlinks=False) and entry.name.startswith("gen-"): + num = entry.name[4:] + if names.GENERATION_RE.match(num) and (Path(entry.path) / MANIFEST_FILE).is_file(): + out.append(int(num)) + except FileNotFoundError: + pass + return sorted(out) + + def read_meta(self) -> GraphMeta: + try: + data = json.loads((self.dir / "meta.json").read_text(encoding="utf-8")) + except (OSError, ValueError) as exc: + raise SnapshotError(f"unreadable meta.json ({type(exc).__name__})") from None + return GraphMeta.from_json(data, self.dir.name[2:]) + + @staticmethod + def create(parent: Path, meta: GraphMeta, keep_generations: int = 3) -> "GraphStorage": + """Create the graph directory atomically (temp dir, then rename).""" + final = parent / names.graph_dirname(meta.name) + parent.mkdir(parents=True, exist_ok=True) + if final.exists(): + raise FileExistsError(meta.name) + tmp = parent / f".tmp-create-{uuid.uuid4().hex[:12]}" + tmp.mkdir() + try: + write_bytes_durably(tmp / "meta.json", json.dumps(meta.to_json(), indent=2).encode("utf-8")) + fsync_dir(tmp) + os.rename(tmp, final) + except BaseException: + shutil.rmtree(tmp, ignore_errors=True) + raise + fsync_dir(parent) + return GraphStorage(final, keep_generations) + + def remove(self) -> None: + """Delete the graph: rename out of the namespace first, then remove the files.""" + trash = self.dir.with_name(f".trash-{uuid.uuid4().hex[:12]}") + os.rename(self.dir, trash) + fsync_dir(trash.parent) + shutil.rmtree(trash, ignore_errors=True) + + # -- writing ------------------------------------------------------------------ + + def write_generation(self, image: Serialized) -> Manifest: + """Write ``image`` as the next generation and make it current. No locks needed.""" + existing = self.generations() + current = self.read_current() or 0 + generation = max([current] + existing) + 1 + tmp = self.dir / f".tmp-gen-{generation:06d}-{uuid.uuid4().hex[:8]}" + tmp.mkdir() + try: + # Re-read what we are about to publish and compare it with memory. + reread = ISONGraph.from_ison(image.graph_text, "graph") + if semantic_digest(reread) != image.semantic: + raise SnapshotError( + "the graph does not survive a save/load round trip unchanged; " + "refusing to publish a snapshot that differs from memory") + digest = graph_digest(reread) + payload = dict(image.payload) + payload["graph_digest"] = digest + payload["graph_digest_basis"] = GRAPH_DIGEST_BASIS + write_bytes_durably(tmp / GRAPH_FILE, image.graph_text.encode("utf-8")) + _fault("graph_written") + write_bytes_durably(tmp / SIDECAR_FILE, json.dumps(payload).encode("utf-8")) + files = {} + for fname in (GRAPH_FILE, SIDECAR_FILE): + file_sha, size = sha256_file(tmp / fname) + files[fname] = {"sha256": file_sha, "size": size} + manifest = Manifest( + generation=generation, graph_digest=digest, node_count=image.node_count, + edge_count=image.edge_count, embedding_count=image.embedding_count, + dimension=image.dimension, files=files, created_at=time.time()) + write_bytes_durably(tmp / MANIFEST_FILE, + json.dumps(manifest.to_json(), indent=2).encode("utf-8")) + fsync_dir(tmp) + _fault("files_written") + os.rename(tmp, self.gen_path(generation)) + fsync_dir(self.dir) + _fault("generation_renamed") + atomic_write(self.dir / "CURRENT", f"{generation:06d}\n".encode("ascii")) + _fault("current_flipped") + except BaseException: + shutil.rmtree(tmp, ignore_errors=True) + raise + self.prune() + return manifest + + def install_generation(self, staged: Path, generation: int) -> None: + """Adopt an already verified generation directory (replica sync).""" + target = self.gen_path(generation) + if target.exists(): + shutil.rmtree(target) + os.rename(staged, target) + fsync_dir(self.dir) + atomic_write(self.dir / "CURRENT", f"{generation:06d}\n".encode("ascii")) + self.prune(keep_extra={generation}) + + def prune(self, keep_extra=()) -> None: + gens = self.generations() + current = self.read_current() + keep = set(gens[-self.keep:]) | set(keep_extra) + if current is not None: + keep.add(current) + for g in gens: + if g not in keep: + shutil.rmtree(self.gen_path(g), ignore_errors=True) + + def clean_tmp(self) -> int: + """Remove leftovers of an interrupted save/create. Only the process that + owns the data directory may call this (it does at startup).""" + n = 0 + try: + for entry in os.scandir(self.dir): + if entry.name.startswith((".tmp-", ".trash-")) or ( + entry.name.startswith(".") and entry.name.endswith(".tmp")): + p = Path(entry.path) + if entry.is_dir(follow_symlinks=False): + shutil.rmtree(p, ignore_errors=True) + else: + try: + p.unlink() + except OSError: + continue + n += 1 + except FileNotFoundError: + pass + return n + + # -- reading ------------------------------------------------------------------ + + def verify_generation(self, gen_dir: Path) -> Manifest: + """Check manifest well-formedness and each file's size and sha256.""" + try: + manifest = Manifest.from_json(json.loads((gen_dir / MANIFEST_FILE).read_text(encoding="utf-8"))) + except (OSError, ValueError) as exc: + raise SnapshotError(f"unreadable manifest ({type(exc).__name__})") from None + for fname, entry in manifest.files.items(): + path = gen_dir / fname + try: + digest, size = sha256_file(path) + except OSError: + raise SnapshotError(f"{fname} is missing or unreadable") from None + if size != entry["size"] or digest != entry["sha256"]: + raise SnapshotError(f"{fname} does not match its manifest checksum") + return manifest + + def load_generation(self, gen_dir: Path, meta: GraphMeta, + encoder: Optional[EmbeddingEncoder]) -> Tuple[SemanticGraph, Manifest, List[str]]: + manifest = self.verify_generation(gen_dir) + enc = encoder if meta.mode == "server" else ClientModeEncoder() + caught: List[str] = [] + with _LOAD_WARN_LOCK, warnings.catch_warnings(record=True) as record: + warnings.simplefilter("always") + try: + graph = SemanticGraph.load( + gen_dir / GRAPH_FILE, encoder=enc, auto_embed=False, + embed_fields=list(meta.embed_fields), on_mismatch="error", + on_model_mismatch="ignore", ann=meta.ann, ann_nprobe=meta.ann_nprobe) + except (EmbeddingError, ValueError, OSError, KeyError, TypeError) as exc: + raise SnapshotError(f"{type(exc).__name__}: {str(exc)[:200]}") from None + caught = [str(w.message)[:300] for w in record] + graph.name = meta.name + try: + if graph_digest(graph) != manifest.graph_digest: + raise SnapshotError("loaded graph digest does not match the manifest") + recorded = graph.embedding_store.last_import_meta.get("graph_digest") + if recorded != manifest.graph_digest: + raise SnapshotError("embeddings sidecar is not tagged with the manifest's graph digest") + except BaseException: + graph.close() + raise + return graph, manifest, caught + + def load(self, meta: GraphMeta, encoder: Optional[EmbeddingEncoder]): + """Return ``(graph, manifest_or_None, degraded, warnings)``. + + See the module docstring for the fallback rules. A brand-new graph + (no generations, no CURRENT) loads as empty with ``manifest=None``. + """ + gens = self.generations() + current = self.read_current() + if not gens and current is None: + return new_semantic_graph(meta, encoder), None, False, [] + if current is not None and current in gens: + order = [current] + [g for g in reversed(gens) if g < current] + else: + order = list(reversed(gens)) + errors = [] + for gen in order: + try: + graph, manifest, caught = self.load_generation(self.gen_path(gen), meta, encoder) + return graph, manifest, gen != current, caught + except SnapshotError as exc: + errors.append(f"gen {gen}: {exc}") + raise SnapshotError("no generation passed verification: " + "; ".join(errors)) + + +def acquire_dir_lock(directory: Path): + """Exclusive, non-blocking lock so two server processes cannot share a data dir. + + Returns an open file object to keep alive for the life of the process. + """ + directory.mkdir(parents=True, exist_ok=True) + handle = open(directory / ".lock", "a+b") + try: + if os.name == "nt": + import msvcrt + handle.seek(0) + msvcrt.locking(handle.fileno(), msvcrt.LK_NBLCK, 1) + else: + import fcntl + fcntl.flock(handle.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB) + except OSError: + handle.close() + raise SnapshotError(f"{directory} is in use by another igv-server process") from None + return handle diff --git a/isongraph-vector-server/isongraph_vector_server/validation.py b/isongraph-vector-server/isongraph_vector_server/validation.py new file mode 100644 index 0000000..b4d10d4 --- /dev/null +++ b/isongraph-vector-server/isongraph_vector_server/validation.py @@ -0,0 +1,237 @@ +"""Explicit request validation: a small JSON-Schema-subset interpreter. + +Every endpoint that takes a body declares a schema (see ``schemas.py``); the +body is validated against it *before* any handler code runs. Objects reject +unknown keys by default, so a typo or a smuggled field is an error instead of +being silently ignored. Supported keywords: ``type`` (object, array, string, +integer, number, boolean, null, or a list of them), ``properties``, +``required``, ``additionalProperties`` (default False), ``items``, +``minItems``/``maxItems``, ``minLength``/``maxLength``, ``pattern``, +``enum``, ``minimum``/``maximum``, and ``kind`` for the domain types +(``id``, ``typename``, ``propname``, ``props``, ``vector``, ``text``). +""" + +from __future__ import annotations + +import math +import re +from typing import Any, Callable, Dict, List + +from . import names +from .errors import validation_error + +MAX_DEPTH = 8 +MAX_VECTOR = 8192 +MAX_STR = 1_000_000 + +# Unicode whitespace/format characters that the ISON text format strips or +# folds on a save/load round trip (found by fuzzing the round trip). A string +# containing them would not come back identical, so they are refused up front. +_ROUNDTRIP_UNSAFE = set("\u0085          " + "   

   ") + + +def _check_text(value: str, path: str) -> None: + for ch in value: + o = ord(ch) + if 0xD800 <= o <= 0xDFFF: + raise validation_error(path, "lone surrogate in string") + if ch in _ROUNDTRIP_UNSAFE: + raise validation_error(path, "string contains an unsupported whitespace character") + + +def _type_ok(value: Any, name: str) -> bool: + if name == "object": + return isinstance(value, dict) + if name == "array": + return isinstance(value, list) + if name == "string": + return isinstance(value, str) + if name == "boolean": + return isinstance(value, bool) + if name == "integer": + return isinstance(value, int) and not isinstance(value, bool) + if name == "number": + return isinstance(value, (int, float)) and not isinstance(value, bool) + if name == "null": + return value is None + raise AssertionError(f"unknown schema type {name}") + + +def _kind_id(value, path, schema): + try: + return names.normalize_id(value) + except ValueError as exc: + raise validation_error(path, str(exc)) from None + + +def _kind_typename(value, path, schema): + if not isinstance(value, str) or not names.TYPE_RE.match(value): + raise validation_error(path, "must match [A-Za-z_][A-Za-z0-9_]{0,63}") + return value + + +def _kind_graphname(value, path, schema): + if not isinstance(value, str) or not names.GRAPH_RE.match(value): + raise validation_error(path, "must match [a-z0-9][a-z0-9_-]{0,62}") + return value + + +def scalar_property_value(value: Any, path: str, max_chars: int = 20_000) -> Any: + """A property value: null, bool, int, finite float or string - nothing nested. + + Floats must round-trip through the ISON text format, which writes a float + with an exponent ("1e+20") as a string; those are refused rather than + silently changing type on the next restart. + """ + if value is None or isinstance(value, bool): + return value + if isinstance(value, int): + if abs(value) > names.MAX_INT_ID: + raise validation_error(path, "integer out of range (|n| <= 2^53-1)") + return value + if isinstance(value, float): + if not math.isfinite(value): + raise validation_error(path, "number must be finite") + if "e" in repr(value).lower(): + raise validation_error(path, "number too large or small to store exactly; send it as a string") + return value + if isinstance(value, str): + if len(value) > max_chars: + raise validation_error(path, f"string longer than {max_chars} characters") + _check_text(value, path) + return value + raise validation_error(path, "property values must be null, boolean, number or string") + + +def _kind_props(value, path, schema): + if not isinstance(value, dict): + raise validation_error(path, "must be an object") + limit = schema.get("maxProperties", 64) + if len(value) > limit: + raise validation_error(path, f"at most {limit} properties") + out = {} + for key, item in value.items(): + if not isinstance(key, str) or not names.PROP_RE.match(key): + raise validation_error(path, f"invalid property name {key[:40]!r}") + if key.startswith("_"): + raise validation_error(path, "property names may not start with '_' (reserved)") + if key in schema.get("reserved", ()): + raise validation_error(path, f"property name {key!r} is reserved (it is a column of the ISON format)") + out[key] = scalar_property_value(item, f"{path}.{key}", schema.get("maxChars", 20_000)) + return out + + +def _kind_vector(value, path, schema): + if not isinstance(value, list) or not value: + raise validation_error(path, "must be a non-empty array of numbers") + if len(value) > MAX_VECTOR: + raise validation_error(path, f"at most {MAX_VECTOR} components") + out: List[float] = [] + for i, item in enumerate(value): + if isinstance(item, bool) or not isinstance(item, (int, float)): + raise validation_error(f"{path}[{i}]", "must be a number") + f = float(item) + if not math.isfinite(f): + raise validation_error(f"{path}[{i}]", "must be finite") + out.append(f) + return out + + +def _kind_text(value, path, schema): + if not isinstance(value, str): + raise validation_error(path, "must be a string") + limit = schema.get("maxLength", 20_000) + if len(value) > limit: + raise validation_error(path, f"longer than {limit} characters") + _check_text(value, path) + return value + + +KINDS: Dict[str, Callable[[Any, str, dict], Any]] = { + "id": _kind_id, + "typename": _kind_typename, + "propname": _kind_typename, + "graphname": _kind_graphname, + "props": _kind_props, + "vector": _kind_vector, + "text": _kind_text, +} + + +def validate(value: Any, schema: Dict[str, Any], path: str = "$", depth: int = 0) -> Any: + """Validate ``value`` against ``schema``; return the normalised value. + + Raises ``BadRequest`` (code ``validation_failed``) naming the first + offending path. + """ + if depth > MAX_DEPTH: + raise validation_error(path, "nested too deeply") + + kind = schema.get("kind") + if kind is not None: + if value is None and schema.get("nullable"): + return None + return KINDS[kind](value, path, schema) + + types = schema.get("type") + if types is not None: + tlist = types if isinstance(types, list) else [types] + if not any(_type_ok(value, t) for t in tlist): + raise validation_error(path, "must be " + " or ".join(tlist)) + + if "enum" in schema and value not in schema["enum"]: + raise validation_error(path, "must be one of " + ", ".join(repr(e) for e in schema["enum"])) + + if isinstance(value, str): + if len(value) > schema.get("maxLength", MAX_STR): + raise validation_error(path, f"longer than {schema.get('maxLength', MAX_STR)} characters") + if len(value) < schema.get("minLength", 0): + raise validation_error(path, f"shorter than {schema['minLength']} characters") + if "pattern" in schema and not re.match(schema["pattern"], value): + raise validation_error(path, "does not match the required pattern") + _check_text(value, path) + + if isinstance(value, (int, float)) and not isinstance(value, bool): + if isinstance(value, float) and not math.isfinite(value): + raise validation_error(path, "must be finite") + if "minimum" in schema and value < schema["minimum"]: + raise validation_error(path, f"must be >= {schema['minimum']}") + if "maximum" in schema and value > schema["maximum"]: + raise validation_error(path, f"must be <= {schema['maximum']}") + + if isinstance(value, list): + if len(value) > schema.get("maxItems", 100_000): + raise validation_error(path, f"at most {schema.get('maxItems', 100_000)} items") + if len(value) < schema.get("minItems", 0): + raise validation_error(path, f"at least {schema['minItems']} items") + item_schema = schema.get("items") + if item_schema is not None: + return [validate(item, item_schema, f"{path}[{i}]", depth + 1) + for i, item in enumerate(value)] + return value + + if isinstance(value, dict) and ( + "properties" in schema or "required" in schema or "additionalProperties" in schema + or types == "object"): + props = schema.get("properties", {}) + extra = schema.get("additionalProperties", False) + out = {} + for key in schema.get("required", ()): + if key not in value: + raise validation_error(path, f"missing required field '{key}'") + for key, item in value.items(): + if key in props: + out[key] = validate(item, props[key], f"{path}.{key}", depth + 1) + elif extra is False: + raise validation_error(path, f"unknown field {str(key)[:40]!r}") + elif isinstance(extra, dict): + out[key] = validate(item, extra, f"{path}.{key}", depth + 1) + else: + out[key] = item + for key, sub in props.items(): + if key not in out and "default" in sub: + out[key] = sub["default"] + return out + + return value diff --git a/isongraph-vector-server/pyproject.toml b/isongraph-vector-server/pyproject.toml new file mode 100644 index 0000000..c8c58f1 --- /dev/null +++ b/isongraph-vector-server/pyproject.toml @@ -0,0 +1,40 @@ +[build-system] +requires = ["setuptools>=77.0", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "isongraph-vector-server" +version = "0.1.0" +description = "Multi-tenant HTTP service, tenant-sharding router and snapshot read replicas around isongraph-vector" +readme = "README.md" +license = "MIT" +authors = [{name = "Mahesh Vaikri"}] +requires-python = ">=3.10" +classifiers = [ + "Development Status :: 3 - Alpha", + "Intended Audience :: Developers", + "Programming Language :: Python :: 3", +] +# The only runtime dependency is the library itself (which brings ison-graph). +# The HTTP server, router, replication and CLI are standard library only. +dependencies = [ + "isongraph-vector>=1.0.0", +] + +[project.optional-dependencies] +# Vectorised search (numpy) and the opt-in IVF approximate index. +fast = ["numpy>=1.21.0"] +dev = ["pytest>=7.0.0", "numpy>=1.21.0"] + +[project.scripts] +igv-server = "isongraph_vector_server.cli:main" + +[tool.setuptools] +packages = ["isongraph_vector_server"] + +[tool.pytest.ini_options] +testpaths = ["tests"] +addopts = "-q --tb=short" +markers = [ + "slow: tests that take more than a few seconds (crash, load)", +] diff --git a/isongraph-vector-server/scripts/_bootstrap.py b/isongraph-vector-server/scripts/_bootstrap.py new file mode 100644 index 0000000..800e177 --- /dev/null +++ b/isongraph-vector-server/scripts/_bootstrap.py @@ -0,0 +1,20 @@ +"""Make `isongraph_vector` and `isongraph_vector_server` importable from a checkout +(same trick as the repo's root conftest.py: the library directory is `isongraph_vector-py`).""" + +import importlib.util +import sys +from pathlib import Path + +SERVER_ROOT = Path(__file__).resolve().parent.parent +REPO_ROOT = SERVER_ROOT.parent + +if str(SERVER_ROOT) not in sys.path: + sys.path.insert(0, str(SERVER_ROOT)) + +if "isongraph_vector" not in sys.modules and importlib.util.find_spec("isongraph_vector") is None: + pkg_dir = REPO_ROOT / "isongraph_vector-py" + spec = importlib.util.spec_from_file_location( + "isongraph_vector", pkg_dir / "__init__.py", submodule_search_locations=[str(pkg_dir)]) + module = importlib.util.module_from_spec(spec) + sys.modules["isongraph_vector"] = module + spec.loader.exec_module(module) diff --git a/isongraph-vector-server/scripts/load_test.py b/isongraph-vector-server/scripts/load_test.py new file mode 100644 index 0000000..48d8ede --- /dev/null +++ b/isongraph-vector-server/scripts/load_test.py @@ -0,0 +1,368 @@ +#!/usr/bin/env python3 +"""Throughput / latency of semantic search through the HTTP service, with N tenants. + +What it does + 1. creates N tenants (each with its own API key) and one graph per tenant holding M random + D-dimensional vectors (client-mode graphs: ingestion cost is not what is being measured) + 2. starts the server(s) -- as separate OS processes by default, so the load generator and the + server do not fight over one GIL + 3. runs C closed-loop client threads for a fixed duration, each over its own keep-alive + connection, each request a POST .../search for a random query vector against a random tenant + 4. prints a table and (with --out) writes JSON a benchmark wave can ingest + +Modes + --mode spawn (default) server in a child process + --mode inproc server in this process (client and server share the GIL: pessimistic) + --mode external --url and --keys-file point at a running deployment (keys file: JSON list of + {"key": ..., "graph": ...}); no setup is done + --shards K spawn K storage nodes plus a router in front (tenants placed by the router's + hash); shows the router's overhead and scale-out of independent tenants + +Examples + python scripts/load_test.py --tenants 8 --nodes 2000 --dim 64 --clients 16 --duration 10 + python scripts/load_test.py --shards 2 --tenants 16 --out results.json + python scripts/load_test.py --text # text queries: the server encodes (MockEncoder) + +Honest caveats: closed-loop load measures peak sustainable throughput, not latency under a fixed +arrival rate; the Python/stdlib server is GIL-bound, so throughput does not grow with --clients +past a handful, and does grow with --shards (separate processes). +""" + +from __future__ import annotations + +import argparse +import http.client +import json +import os +import platform +import random +import shutil +import socket +import statistics +import subprocess +import sys +import tempfile +import threading +import time +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import _bootstrap # noqa: E402,F401 + +from isongraph_vector_server import auth # noqa: E402 +from isongraph_vector_server.router import rendezvous # noqa: E402 + +HERE = Path(__file__).resolve().parent + + +def free_port() -> int: + s = socket.socket() + s.bind(("127.0.0.1", 0)) + port = s.getsockname()[1] + s.close() + return port + + +def request(host, port, method, path, key=None, body=None, conn=None, timeout=60): + own = conn is None + conn = conn or http.client.HTTPConnection(host, port, timeout=timeout) + headers = {"Content-Type": "application/json"} + if key: + headers["Authorization"] = f"Bearer {key}" + data = json.dumps(body).encode() if body is not None else None + try: + conn.request(method, path, body=data, headers=headers) + resp = conn.getresponse() + payload = resp.read() + return resp.status, (json.loads(payload) if payload else None) + finally: + if own: + conn.close() + + +def wait_ready(host, port, proc=None, timeout=60): + end = time.time() + timeout + while time.time() < end: + if proc is not None and proc.poll() is not None: + raise SystemExit("server exited: " + (proc.stderr.read().decode()[-1500:] if proc.stderr else "")) + try: + if request(host, port, "GET", "/readyz")[0] == 200: + return + except OSError: + time.sleep(0.1) + raise SystemExit("server did not become ready") + + +def start_node(data_dir: str, port: int, args) -> subprocess.Popen: + cfg = { + "data_dir": data_dir, "host": "127.0.0.1", "port": port, "log_level": "CRITICAL", + "autosave_interval_seconds": 0, "max_workers": max(32, args.clients + 8), + "accept_queue": 128, "max_concurrent_requests": max(32, args.clients + 8), + "failed_auth_per_minute": 10 ** 6, + "encoder": {"kind": "mock", "dimension": args.dim}, + "quotas": {"rate_per_second": 10 ** 6, "rate_burst": 10 ** 6, "max_inflight": args.clients + 8, + "max_nodes": args.nodes * 2, "max_dimension": max(args.dim, 16)}, + } + path = Path(data_dir).parent / f"cfg-{port}.json" + path.write_text(json.dumps(cfg)) + boot = ("import sys; sys.path.insert(0, %r); import _bootstrap; " + "from isongraph_vector_server.cli import main; sys.exit(main(['serve', '--config', %r]))" + % (str(HERE), str(path))) + return subprocess.Popen([sys.executable, "-c", boot], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE) + + +def start_router(shards, port: int, args) -> subprocess.Popen: + cfg = {"host": "127.0.0.1", "port": port, "log_level": "CRITICAL", "max_workers": max(32, args.clients + 8), + "accept_queue": 128, "router": {"shards": shards, "upstream_timeout_seconds": 60}} + path = Path(tempfile.gettempdir()) / f"igv-router-{port}.json" + path.write_text(json.dumps(cfg)) + boot = ("import sys; sys.path.insert(0, %r); import _bootstrap; " + "from isongraph_vector_server.cli import main; sys.exit(main(['router', '--config', %r]))" + % (str(HERE), str(path))) + return subprocess.Popen([sys.executable, "-c", boot], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE) + + +def populate(host, port, key, graph, args, rnd: random.Random): + s, r = request(host, port, "POST", "/v1/graphs", key, {"name": graph, "embedding": {"mode": "client", "dimension": args.dim}}) + assert s == 201, (s, r) + batch = 400 + for start in range(0, args.nodes, batch): + items = [{"type": "doc", "id": i, "properties": {"name": f"doc {i}", "bucket": i % 10}, + "vector": [rnd.gauss(0, 1) for _ in range(args.dim)]} + for i in range(start, min(args.nodes, start + batch))] + s, r = request(host, port, "POST", f"/v1/graphs/{graph}/nodes", key, {"nodes": items}) + assert s == 200, (s, r) + + +def _gen_proc(host, port, targets, args, begin_epoch): + return run_load(host, port, targets, args, begin_epoch) + + +def percentile(sorted_ms, p): + if not sorted_ms: + return float("nan") + k = min(len(sorted_ms) - 1, max(0, int(round(p / 100 * len(sorted_ms) + 0.5)) - 1)) + return sorted_ms[k] + + +def run_load(host, port, targets, args, begin_epoch=None): + rnd = random.Random(args.seed) + queries = [[rnd.gauss(0, 1) for _ in range(args.dim)] for _ in range(512)] + texts = [f"doc {rnd.randrange(args.nodes)}" for _ in range(512)] + stop_at = [0.0] + measure_start = [0.0] + results = [] + lock = threading.Lock() + started = threading.Barrier(args.clients + 1) + + def worker(wid): + r = random.Random(args.seed * 1000 + wid) + conn = http.client.HTTPConnection(host, port, timeout=60) + lat, errs, per_tenant = [], {}, {} + started.wait() + while time.perf_counter() < stop_at[0]: + t_idx = r.randrange(len(targets)) + key, graph = targets[t_idx]["key"], targets[t_idx]["graph"] + q = r.randrange(512) + body = ({"query": texts[q], "top_k": args.top_k, "threshold": -1.0} if args.text + else {"vector": queries[q], "top_k": args.top_k, "threshold": -1.0}) + t0 = time.perf_counter() + try: + conn.request("POST", f"/v1/graphs/{graph}/search", body=json.dumps(body).encode(), + headers={"Authorization": f"Bearer {key}", "Content-Type": "application/json"}) + resp = conn.getresponse() + resp.read() + status = resp.status + except (OSError, http.client.HTTPException): + status = -1 + conn.close() + conn = http.client.HTTPConnection(host, port, timeout=60) + now = time.perf_counter() + dt = (now - t0) * 1000 + if t0 < measure_start[0]: + continue # warm-up sample: not counted + if status == 200: + lat.append(dt) + per_tenant[t_idx] = per_tenant.get(t_idx, 0) + 1 + else: + errs[status] = errs.get(status, 0) + 1 + conn.close() + with lock: + results.append((lat, errs, per_tenant)) + + threads = [threading.Thread(target=worker, args=(w,)) for w in range(args.clients)] + [t.start() for t in threads] + if begin_epoch is None: + begin_epoch = time.time() + 0.3 + begin = time.perf_counter() + max(0.0, begin_epoch - time.time()) + measure_start[0] = begin + args.warmup + stop_at[0] = measure_start[0] + args.duration + started.wait() + [t.join() for t in threads] + return results + + +def summarize(results, args, wall): + all_lat, errs, per_tenant = [], {}, {} + for lat, e, pt in results: + all_lat += lat + for k, v in e.items(): + errs[k] = errs.get(k, 0) + v + for k, v in pt.items(): + per_tenant[k] = per_tenant.get(k, 0) + v + all_lat.sort() + n = len(all_lat) + counts = sorted(per_tenant.values()) or [0] + return { + "requests_ok": n, + "errors": {str(k): v for k, v in errs.items()}, + "throughput_rps": round(n / wall, 1), + "latency_ms": { + "mean": round(statistics.fmean(all_lat), 3) if n else None, + "p50": round(percentile(all_lat, 50), 3), "p95": round(percentile(all_lat, 95), 3), + "p99": round(percentile(all_lat, 99), 3), "max": round(all_lat[-1], 3) if n else None}, + "per_tenant_requests": {"min": counts[0], "max": counts[-1], "tenants_hit": len(per_tenant)}, + } + + +def main(): + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--tenants", type=int, default=8) + ap.add_argument("--nodes", type=int, default=2000, help="vectors per tenant graph") + ap.add_argument("--dim", type=int, default=64) + ap.add_argument("--top-k", type=int, default=10) + ap.add_argument("--clients", type=int, default=16) + ap.add_argument("--duration", type=float, default=10.0) + ap.add_argument("--warmup", type=float, default=2.0) + ap.add_argument("--mode", choices=["spawn", "inproc", "external"], default="spawn") + ap.add_argument("--shards", type=int, default=0, help="spawn K nodes + a router (spawn mode)") + ap.add_argument("--text", action="store_true", help="text queries (server encodes) instead of vectors") + ap.add_argument("--url", help="external mode: base URL") + ap.add_argument("--keys-file", help="external mode: JSON list of {key, graph}") + ap.add_argument("--gen-procs", type=int, default=1, + help="load-generator processes (clients are split across them); use >1 to keep the " + "generator's own GIL from being the bottleneck") + ap.add_argument("--seed", type=int, default=1) + ap.add_argument("--out") + args = ap.parse_args() + tmp = Path(tempfile.mkdtemp(prefix="igv-load-")) + procs, targets, server = [], [], None + host, port = "127.0.0.1", 0 + try: + if args.mode == "external": + u = args.url.replace("http://", "").split(":") + host, port = u[0], int(u[1]) + targets = json.loads(Path(args.keys_file).read_text()) + else: + rnd = random.Random(args.seed) + n_nodes = max(1, args.shards) + data_dirs = [str(tmp / f"node{i}") for i in range(n_nodes)] + names_ = [f"s{i}" for i in range(n_nodes)] + quotas = {"rate_per_second": 10 ** 6, "rate_burst": 10 ** 6, "max_inflight": args.clients + 8, + "max_nodes": args.nodes * 2, "max_dimension": max(args.dim, 16)} + tenant_owner = {} + for t in range(args.tenants): + tid = f"tenant-{t}" + owner = rendezvous(tid, names_) if args.shards else "s0" + idx = names_.index(owner) + auth.create_tenant(data_dirs[idx], tid, quotas) + targets.append({"tid": tid, "key": auth.add_key(data_dirs[idx], tid), "graph": "kb", "node": idx}) + tenant_owner[tid] = idx + if args.mode == "inproc": + from isongraph_vector_server.config import ServerConfig + from isongraph_vector_server.server import Server + assert not args.shards, "inproc supports a single node" + cfg = ServerConfig(data_dir=data_dirs[0], host="127.0.0.1", port=0, log_level="CRITICAL") + cfg.autosave_interval_seconds = 0 + cfg.max_workers = cfg.max_concurrent_requests = max(32, args.clients + 8) + cfg.accept_queue = 128 + cfg.failed_auth_per_minute = 10 ** 6 + cfg.encoder.kind, cfg.encoder.dimension = "mock", args.dim + cfg.quotas.rate_per_second = cfg.quotas.rate_burst = 10 ** 6 + cfg.quotas.max_inflight = args.clients + 8 + server = Server(cfg, configure_logging=False).start() + ports = [server.address[1]] + else: + ports = [free_port() for _ in range(n_nodes)] + for dd, p in zip(data_dirs, ports): + procs.append(start_node(dd, p, args)) + for p, proc in zip(ports, procs): + wait_ready(host, p, proc) + # populate each tenant directly on its node + print(f"populating {args.tenants} tenants x {args.nodes} vectors (dim {args.dim}) ...", flush=True) + for t in targets: + p = ports[t["node"]] + if args.text: + s, r = request(host, p, "POST", "/v1/graphs", t["key"], {"name": "kb"}) + assert s == 201, r + for start in range(0, args.nodes, 400): + items = [{"type": "doc", "id": i, "properties": {"name": f"doc {i}"}} + for i in range(start, min(args.nodes, start + 400))] + s, r = request(host, p, "POST", "/v1/graphs/kb/nodes", t["key"], {"nodes": items}) + assert s == 200, r + else: + populate(host, p, t["key"], "kb", args, rnd) + if args.shards: + port = free_port() + shards = [{"name": names_[i], "primary": f"http://127.0.0.1:{ports[i]}"} for i in range(n_nodes)] + procs.append(start_router(shards, port, args)) + for _ in range(100): + try: + if request(host, port, "GET", "/healthz")[0] == 200: + break + except OSError: + time.sleep(0.1) + else: + port = ports[0] + + print(f"load: {args.clients} clients, {len(targets)} tenants, {args.warmup}s warm-up + " + f"{args.duration}s measured, {'text' if args.text else 'vector'} queries, top_k={args.top_k}", flush=True) + if args.gen_procs > 1: + import copy + import multiprocessing as mp + per = copy.copy(args) + per.clients = max(1, -(-args.clients // args.gen_procs)) + jobs = [] + begin = time.time() + 3.0 + for i in range(args.gen_procs): + a = copy.copy(per) + a.seed = args.seed + i + jobs.append((host, port, targets, a, begin)) + with mp.get_context("spawn").Pool(args.gen_procs) as pool: + results = [x for part in pool.starmap(_gen_proc, jobs) for x in part] + else: + results = run_load(host, port, targets, args) + summary = summarize(results, args, args.duration) + report = { + "config": {k: v for k, v in vars(args).items() if k not in ("out", "keys_file")}, + "environment": {"python": platform.python_version(), "platform": platform.platform(), + "cpu_count": os.cpu_count(), "server_processes": max(1, args.shards)}, + "result": summary, + } + r = summary + print() + print(f" throughput {r['throughput_rps']:>10.1f} req/s (measured window only)") + lat = r["latency_ms"] + print(f" latency ms mean {lat['mean']} p50 {lat['p50']} p95 {lat['p95']} p99 {lat['p99']} max {lat['max']}") + print(f" ok / errors {r['requests_ok']} / {r['errors'] or 0}") + print(f" per-tenant min {r['per_tenant_requests']['min']} max {r['per_tenant_requests']['max']} " + f"({r['per_tenant_requests']['tenants_hit']} tenants hit)") + if args.out: + Path(args.out).write_text(json.dumps(report, indent=2)) + print(f" wrote {args.out}") + return 0 + finally: + if server is not None: + server.stop() + for p in procs: + p.terminate() + for p in procs: + try: + p.wait(timeout=10) + except subprocess.TimeoutExpired: + p.kill() + shutil.rmtree(tmp, ignore_errors=True) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/isongraph-vector-server/scripts/results/gp8_router1_16t.json b/isongraph-vector-server/scripts/results/gp8_router1_16t.json new file mode 100644 index 0000000..494fa47 --- /dev/null +++ b/isongraph-vector-server/scripts/results/gp8_router1_16t.json @@ -0,0 +1,40 @@ +{ + "config": { + "tenants": 16, + "nodes": 2000, + "dim": 64, + "top_k": 10, + "clients": 32, + "duration": 8.0, + "warmup": 2.0, + "mode": "spawn", + "shards": 1, + "text": false, + "url": null, + "gen_procs": 8, + "seed": 1 + }, + "environment": { + "python": "3.12.7", + "platform": "Windows-11-10.0.26300-SP0", + "cpu_count": 32, + "server_processes": 1 + }, + "result": { + "requests_ok": 14544, + "errors": {}, + "throughput_rps": 1818.0, + "latency_ms": { + "mean": 17.576, + "p50": 14.886, + "p95": 38.13, + "p99": 57.89, + "max": 102.969 + }, + "per_tenant_requests": { + "min": 846, + "max": 966, + "tenants_hit": 16 + } + } +} \ No newline at end of file diff --git a/isongraph-vector-server/scripts/results/gp8_router2_16t.json b/isongraph-vector-server/scripts/results/gp8_router2_16t.json new file mode 100644 index 0000000..b4f6bdd --- /dev/null +++ b/isongraph-vector-server/scripts/results/gp8_router2_16t.json @@ -0,0 +1,40 @@ +{ + "config": { + "tenants": 16, + "nodes": 2000, + "dim": 64, + "top_k": 10, + "clients": 32, + "duration": 8.0, + "warmup": 2.0, + "mode": "spawn", + "shards": 2, + "text": false, + "url": null, + "gen_procs": 8, + "seed": 1 + }, + "environment": { + "python": "3.12.7", + "platform": "Windows-11-10.0.26300-SP0", + "cpu_count": 32, + "server_processes": 2 + }, + "result": { + "requests_ok": 23814, + "errors": {}, + "throughput_rps": 2976.8, + "latency_ms": { + "mean": 10.737, + "p50": 8.388, + "p95": 28.762, + "p99": 42.065, + "max": 60.225 + }, + "per_tenant_requests": { + "min": 1415, + "max": 1543, + "tenants_hit": 16 + } + } +} \ No newline at end of file diff --git a/isongraph-vector-server/scripts/results/gp8_router4_16t.json b/isongraph-vector-server/scripts/results/gp8_router4_16t.json new file mode 100644 index 0000000..f9eca71 --- /dev/null +++ b/isongraph-vector-server/scripts/results/gp8_router4_16t.json @@ -0,0 +1,40 @@ +{ + "config": { + "tenants": 16, + "nodes": 2000, + "dim": 64, + "top_k": 10, + "clients": 32, + "duration": 8.0, + "warmup": 2.0, + "mode": "spawn", + "shards": 4, + "text": false, + "url": null, + "gen_procs": 8, + "seed": 1 + }, + "environment": { + "python": "3.12.7", + "platform": "Windows-11-10.0.26300-SP0", + "cpu_count": 32, + "server_processes": 4 + }, + "result": { + "requests_ok": 26663, + "errors": {}, + "throughput_rps": 3332.9, + "latency_ms": { + "mean": 9.592, + "p50": 9.157, + "p95": 15.431, + "p99": 19.07, + "max": 38.496 + }, + "per_tenant_requests": { + "min": 1606, + "max": 1756, + "tenants_hit": 16 + } + } +} \ No newline at end of file diff --git a/isongraph-vector-server/scripts/results/gp8_single_16t.json b/isongraph-vector-server/scripts/results/gp8_single_16t.json new file mode 100644 index 0000000..a356024 --- /dev/null +++ b/isongraph-vector-server/scripts/results/gp8_single_16t.json @@ -0,0 +1,40 @@ +{ + "config": { + "tenants": 16, + "nodes": 2000, + "dim": 64, + "top_k": 10, + "clients": 32, + "duration": 8.0, + "warmup": 2.0, + "mode": "spawn", + "shards": 0, + "text": false, + "url": null, + "gen_procs": 8, + "seed": 1 + }, + "environment": { + "python": "3.12.7", + "platform": "Windows-11-10.0.26300-SP0", + "cpu_count": 32, + "server_processes": 1 + }, + "result": { + "requests_ok": 16814, + "errors": {}, + "throughput_rps": 2101.8, + "latency_ms": { + "mean": 15.207, + "p50": 12.526, + "p95": 34.564, + "p99": 49.519, + "max": 76.345 + }, + "per_tenant_requests": { + "min": 987, + "max": 1115, + "tenants_hit": 16 + } + } +} \ No newline at end of file diff --git a/isongraph-vector-server/scripts/results/inproc_8t_2k.json b/isongraph-vector-server/scripts/results/inproc_8t_2k.json new file mode 100644 index 0000000..5d02ec6 --- /dev/null +++ b/isongraph-vector-server/scripts/results/inproc_8t_2k.json @@ -0,0 +1,39 @@ +{ + "config": { + "tenants": 8, + "nodes": 2000, + "dim": 64, + "top_k": 10, + "clients": 16, + "duration": 8.0, + "warmup": 2.0, + "mode": "inproc", + "shards": 0, + "text": false, + "url": null, + "seed": 1 + }, + "environment": { + "python": "3.12.7", + "platform": "Windows-11-10.0.26300-SP0", + "cpu_count": 32, + "server_processes": 1 + }, + "result": { + "requests_ok": 11175, + "errors": {}, + "throughput_rps": 1396.9, + "latency_ms": { + "mean": 11.444, + "p50": 10.131, + "p95": 22.711, + "p99": 29.8, + "max": 51.648 + }, + "per_tenant_requests": { + "min": 1325, + "max": 1452, + "tenants_hit": 8 + } + } +} \ No newline at end of file diff --git a/isongraph-vector-server/scripts/results/single_32t_2k.json b/isongraph-vector-server/scripts/results/single_32t_2k.json new file mode 100644 index 0000000..7211bff --- /dev/null +++ b/isongraph-vector-server/scripts/results/single_32t_2k.json @@ -0,0 +1,39 @@ +{ + "config": { + "tenants": 32, + "nodes": 2000, + "dim": 64, + "top_k": 10, + "clients": 16, + "duration": 8.0, + "warmup": 2.0, + "mode": "spawn", + "shards": 0, + "text": false, + "url": null, + "seed": 1 + }, + "environment": { + "python": "3.12.7", + "platform": "Windows-11-10.0.26300-SP0", + "cpu_count": 32, + "server_processes": 1 + }, + "result": { + "requests_ok": 15158, + "errors": {}, + "throughput_rps": 1894.8, + "latency_ms": { + "mean": 8.436, + "p50": 7.698, + "p95": 15.503, + "p99": 20.656, + "max": 42.725 + }, + "per_tenant_requests": { + "min": 434, + "max": 516, + "tenants_hit": 32 + } + } +} \ No newline at end of file diff --git a/isongraph-vector-server/scripts/results/single_4t_20k.json b/isongraph-vector-server/scripts/results/single_4t_20k.json new file mode 100644 index 0000000..4731b90 --- /dev/null +++ b/isongraph-vector-server/scripts/results/single_4t_20k.json @@ -0,0 +1,39 @@ +{ + "config": { + "tenants": 4, + "nodes": 20000, + "dim": 64, + "top_k": 10, + "clients": 16, + "duration": 8.0, + "warmup": 2.0, + "mode": "spawn", + "shards": 0, + "text": false, + "url": null, + "seed": 1 + }, + "environment": { + "python": "3.12.7", + "platform": "Windows-11-10.0.26300-SP0", + "cpu_count": 32, + "server_processes": 1 + }, + "result": { + "requests_ok": 11779, + "errors": {}, + "throughput_rps": 1472.4, + "latency_ms": { + "mean": 10.859, + "p50": 9.614, + "p95": 23.212, + "p99": 28.273, + "max": 38.366 + }, + "per_tenant_requests": { + "min": 2910, + "max": 2981, + "tenants_hit": 4 + } + } +} \ No newline at end of file diff --git a/isongraph-vector-server/scripts/results/single_8t_2k.json b/isongraph-vector-server/scripts/results/single_8t_2k.json new file mode 100644 index 0000000..4401c81 --- /dev/null +++ b/isongraph-vector-server/scripts/results/single_8t_2k.json @@ -0,0 +1,39 @@ +{ + "config": { + "tenants": 8, + "nodes": 2000, + "dim": 64, + "top_k": 10, + "clients": 16, + "duration": 8.0, + "warmup": 2.0, + "mode": "spawn", + "shards": 0, + "text": false, + "url": null, + "seed": 1 + }, + "environment": { + "python": "3.12.7", + "platform": "Windows-11-10.0.26300-SP0", + "cpu_count": 32, + "server_processes": 1 + }, + "result": { + "requests_ok": 16001, + "errors": {}, + "throughput_rps": 2000.1, + "latency_ms": { + "mean": 7.991, + "p50": 6.945, + "p95": 16.434, + "p99": 22.101, + "max": 38.002 + }, + "per_tenant_requests": { + "min": 1953, + "max": 2051, + "tenants_hit": 8 + } + } +} \ No newline at end of file diff --git a/isongraph-vector-server/scripts/results/single_8t_2k_c1.json b/isongraph-vector-server/scripts/results/single_8t_2k_c1.json new file mode 100644 index 0000000..4add970 --- /dev/null +++ b/isongraph-vector-server/scripts/results/single_8t_2k_c1.json @@ -0,0 +1,39 @@ +{ + "config": { + "tenants": 8, + "nodes": 2000, + "dim": 64, + "top_k": 10, + "clients": 1, + "duration": 8.0, + "warmup": 2.0, + "mode": "spawn", + "shards": 0, + "text": false, + "url": null, + "seed": 1 + }, + "environment": { + "python": "3.12.7", + "platform": "Windows-11-10.0.26300-SP0", + "cpu_count": 32, + "server_processes": 1 + }, + "result": { + "requests_ok": 12869, + "errors": {}, + "throughput_rps": 1608.6, + "latency_ms": { + "mean": 0.617, + "p50": 0.571, + "p95": 0.75, + "p99": 1.228, + "max": 25.434 + }, + "per_tenant_requests": { + "min": 1567, + "max": 1679, + "tenants_hit": 8 + } + } +} \ No newline at end of file diff --git a/isongraph-vector-server/scripts/results/single_8t_2k_c4.json b/isongraph-vector-server/scripts/results/single_8t_2k_c4.json new file mode 100644 index 0000000..57643e7 --- /dev/null +++ b/isongraph-vector-server/scripts/results/single_8t_2k_c4.json @@ -0,0 +1,39 @@ +{ + "config": { + "tenants": 8, + "nodes": 2000, + "dim": 64, + "top_k": 10, + "clients": 4, + "duration": 8.0, + "warmup": 2.0, + "mode": "spawn", + "shards": 0, + "text": false, + "url": null, + "seed": 1 + }, + "environment": { + "python": "3.12.7", + "platform": "Windows-11-10.0.26300-SP0", + "cpu_count": 32, + "server_processes": 1 + }, + "result": { + "requests_ok": 15588, + "errors": {}, + "throughput_rps": 1948.5, + "latency_ms": { + "mean": 2.047, + "p50": 1.9, + "p95": 3.337, + "p99": 4.95, + "max": 27.07 + }, + "per_tenant_requests": { + "min": 1883, + "max": 2026, + "tenants_hit": 8 + } + } +} \ No newline at end of file diff --git a/isongraph-vector-server/scripts/results/single_8t_2k_c64.json b/isongraph-vector-server/scripts/results/single_8t_2k_c64.json new file mode 100644 index 0000000..e7cd793 --- /dev/null +++ b/isongraph-vector-server/scripts/results/single_8t_2k_c64.json @@ -0,0 +1,39 @@ +{ + "config": { + "tenants": 8, + "nodes": 2000, + "dim": 64, + "top_k": 10, + "clients": 64, + "duration": 8.0, + "warmup": 2.0, + "mode": "spawn", + "shards": 0, + "text": false, + "url": null, + "seed": 1 + }, + "environment": { + "python": "3.12.7", + "platform": "Windows-11-10.0.26300-SP0", + "cpu_count": 32, + "server_processes": 1 + }, + "result": { + "requests_ok": 12993, + "errors": {}, + "throughput_rps": 1624.1, + "latency_ms": { + "mean": 39.342, + "p50": 31.808, + "p95": 99.15, + "p99": 125.209, + "max": 183.617 + }, + "per_tenant_requests": { + "min": 1594, + "max": 1654, + "tenants_hit": 8 + } + } +} \ No newline at end of file diff --git a/isongraph-vector-server/scripts/results/single_8t_2k_text.json b/isongraph-vector-server/scripts/results/single_8t_2k_text.json new file mode 100644 index 0000000..fe5d2c9 --- /dev/null +++ b/isongraph-vector-server/scripts/results/single_8t_2k_text.json @@ -0,0 +1,39 @@ +{ + "config": { + "tenants": 8, + "nodes": 2000, + "dim": 64, + "top_k": 10, + "clients": 16, + "duration": 8.0, + "warmup": 2.0, + "mode": "spawn", + "shards": 0, + "text": true, + "url": null, + "seed": 1 + }, + "environment": { + "python": "3.12.7", + "platform": "Windows-11-10.0.26300-SP0", + "cpu_count": 32, + "server_processes": 1 + }, + "result": { + "requests_ok": 15557, + "errors": {}, + "throughput_rps": 1944.6, + "latency_ms": { + "mean": 8.22, + "p50": 6.51, + "p95": 19.254, + "p99": 24.853, + "max": 36.622 + }, + "per_tenant_requests": { + "min": 1896, + "max": 1992, + "tenants_hit": 8 + } + } +} \ No newline at end of file diff --git a/isongraph-vector-server/tests/conftest.py b/isongraph-vector-server/tests/conftest.py new file mode 100644 index 0000000..c8d317a --- /dev/null +++ b/isongraph-vector-server/tests/conftest.py @@ -0,0 +1,207 @@ +"""Shared fixtures. Binds ``isongraph_vector`` from the checkout when it is not installed +(the library's directory name, ``isongraph_vector-py``, is not importable as-is).""" + +import importlib.util +import json +import sys +import http.client +from pathlib import Path + +import pytest + +HERE = Path(__file__).resolve().parent +SERVER_ROOT = HERE.parent +REPO_ROOT = SERVER_ROOT.parent + +if str(SERVER_ROOT) not in sys.path: + sys.path.insert(0, str(SERVER_ROOT)) + + +def _bind_library() -> None: + if "isongraph_vector" in sys.modules or importlib.util.find_spec("isongraph_vector") is not None: + return + pkg_dir = REPO_ROOT / "isongraph_vector-py" + spec = importlib.util.spec_from_file_location( + "isongraph_vector", pkg_dir / "__init__.py", submodule_search_locations=[str(pkg_dir)]) + module = importlib.util.module_from_spec(spec) + sys.modules["isongraph_vector"] = module + spec.loader.exec_module(module) + + +_bind_library() + +from isongraph_vector_server import auth # noqa: E402 +from isongraph_vector_server.config import ServerConfig # noqa: E402 +from isongraph_vector_server.server import RouterServer, Server # noqa: E402 + +DIM = 16 + + +class Client: + """Minimal HTTP client; one connection per call so tests are independent.""" + + def __init__(self, url: str, key: str = None): + self.host, self.port = url.replace("http://", "").split(":") + self.port = int(self.port) + self.key = key + + def request(self, method, path, body=None, key="__default__", headers=None, raw=None, timeout=20): + conn = http.client.HTTPConnection(self.host, self.port, timeout=timeout) + hdrs = {} + k = self.key if key == "__default__" else key + if k: + hdrs["Authorization"] = f"Bearer {k}" + data = None + if raw is not None: + data = raw + hdrs["Content-Type"] = "application/json" + elif body is not None: + data = json.dumps(body).encode() + hdrs["Content-Type"] = "application/json" + if headers: + hdrs.update(headers) + try: + conn.request(method, path, body=data, headers=hdrs) + resp = conn.getresponse() + payload = resp.read() + try: + parsed = json.loads(payload) if payload else None + except ValueError: + parsed = payload.decode("utf-8", "replace") + return resp.status, parsed, dict(resp.getheaders()) + finally: + conn.close() + + def get(self, path, **kw): + return self.request("GET", path, **kw) + + def post(self, path, body=None, **kw): + return self.request("POST", path, body, **kw) + + def put(self, path, body=None, **kw): + return self.request("PUT", path, body, **kw) + + def patch(self, path, body=None, **kw): + return self.request("PATCH", path, body, **kw) + + def delete(self, path, **kw): + return self.request("DELETE", path, **kw) + + +class Harness: + """A running node plus helpers to create tenants and clients.""" + + def __init__(self, tmp_path: Path, encoder=None, **overrides): + self.data_dir = str(tmp_path / "data") + cfg = ServerConfig(data_dir=self.data_dir, host="127.0.0.1", port=0, log_level="CRITICAL") + cfg.autosave_interval_seconds = 0 + cfg.registry_recheck_seconds = 0.0 + cfg.encoder.kind = "mock" + cfg.encoder.dimension = DIM + cfg.quotas.rate_per_second = 10_000 + cfg.quotas.rate_burst = 10_000 + cfg.quotas.max_inflight = 64 + cfg.max_concurrent_requests = 64 + cfg.failed_auth_per_minute = 10_000 + for k, v in overrides.items(): + if "." in k: + a, b = k.split(".") + setattr(getattr(cfg, a), b, v) + else: + setattr(cfg, k, v) + self.cfg = cfg + self.clients = [] + self._encoder = encoder + self.server = Server(cfg, encoder=encoder, configure_logging=False).start() + + def tenant(self, tid, role="rw", quotas=None): + try: + auth.create_tenant(self.data_dir, tid, quotas) + except FileExistsError: + pass + key = auth.add_key(self.data_dir, tid, role) + c = Client(self.server.url, key) + self.clients.append(c) + return c + + def restart(self, graceful=True, **overrides): + """Stop the node (gracefully: final snapshot) and start a new one on the same data dir.""" + if graceful: + self.server.stop() + else: + self.server.httpd.stop() + self.server.engine._stop.set() + self.server.engine._dir_lock.close() # as if the process had died: no final save + self.server.engine._dir_lock = None + for k, v in overrides.items(): + setattr(self.cfg, k, v) + self.server = Server(self.cfg, encoder=self._encoder, configure_logging=False).start() + for c in self.clients: + c.host, c.port = self.server.address[0], self.server.address[1] + return self.server + + def anon(self): + return Client(self.server.url, None) + + def stop(self): + self.server.stop() + + +@pytest.fixture +def harness(tmp_path): + made = [] + + def make(**overrides): + h = Harness(tmp_path / f"h{len(made)}", **overrides) + made.append(h) + return h + + yield make + for h in made: + try: + h.stop() + except Exception: + pass + + +@pytest.fixture +def replica_of(harness): + """Factory: a replica node pulling from ``source`` (a ``dir:`` path or a primary URL).""" + def make(source, **overrides): + opts = {"role": "replica", "replica.source": source, "replica.sync_interval_seconds": 3600, + "replica.max_staleness_seconds": 3600, "replica.autostart": False} + opts.update(overrides) + h = harness(**opts) + return h + return make + + +@pytest.fixture +def node(harness): + return harness() + + +@pytest.fixture +def alice(node): + return node.tenant("alice") + + +def create_graph(client, name="kb", **body): + status, resp, _ = client.post("/v1/graphs", {"name": name, **body}) + assert status == 201, (status, resp) + return resp + + +def seed(client, graph="kb"): + """A small graph: three people, two edges.""" + s, r, _ = client.post(f"/v1/graphs/{graph}/nodes", {"nodes": [ + {"type": "person", "id": 1, "properties": {"name": "Alice", "description": "builds payment APIs", "age": 31}}, + {"type": "person", "id": 2, "properties": {"name": "Bob", "description": "trains machine learning models", "age": 45}}, + {"type": "person", "id": "carol", "properties": {"name": "Carol", "description": "plans product roadmaps", "age": 38}}, + ]}) + assert s == 200, r + s, r, _ = client.post(f"/v1/graphs/{graph}/edges", {"edges": [ + {"rel": "KNOWS", "source": {"type": "person", "id": 1}, "target": {"type": "person", "id": 2}, "properties": {"weight": 0.9}}, + {"rel": "KNOWS", "source": {"type": "person", "id": 2}, "target": {"type": "person", "id": "carol"}}, + ]}) + assert s == 200, r diff --git a/isongraph-vector-server/tests/crash_child.py b/isongraph-vector-server/tests/crash_child.py new file mode 100644 index 0000000..93d3e50 --- /dev/null +++ b/isongraph-vector-server/tests/crash_child.py @@ -0,0 +1,72 @@ +"""Child process for the crash tests: saves a graph and gets killed by the parent. + +usage: + crash_child.py point <graphdir> <fault-point> publish gen 1 cleanly, then stall *inside* the + save of gen 2 at <fault-point> (parent SIGKILLs) + crash_child.py loop <graphdir> <nodes-per-round> keep mutating and saving forever +""" + +import sys +import time +from pathlib import Path + +HERE = Path(__file__).resolve().parent +sys.path.insert(0, str(HERE)) +import conftest # noqa: F401,E402 (binds isongraph_vector from the checkout) + +from isongraph_vector import MockEncoder # noqa: E402 +from isongraph_vector_server import storage # noqa: E402 +from isongraph_vector_server.storage import GraphMeta, GraphStorage, new_semantic_graph, serialize # noqa: E402 + +ENC = MockEncoder(16) + + +def fill(graph, start, stop): + for i in range(start, stop): + node = graph.add_node("n", i) + node.properties["name"] = f"node {i}" + graph.embedding_store.add(("n", i), f"node {i}", vector=ENC.encode(f"node {i}")) + + +def open_or_create(path: Path): + meta = GraphMeta(name=path.name[2:], mode="server", dimension=16, + embed_fields=["name"], ann=None, ann_nprobe=None) + if (path / "meta.json").exists(): + st = GraphStorage(path) + st.clean_tmp() + graph, manifest, degraded, _ = st.load(st.read_meta(), ENC) + assert not degraded + return st, graph + st = GraphStorage.create(path.parent, meta) + return st, new_semantic_graph(meta, ENC) + + +def main(): + mode, path = sys.argv[1], Path(sys.argv[2]) + st, graph = open_or_create(path) + if mode == "point": + point = sys.argv[3] + fill(graph, 0, 1500) + st.write_generation(serialize(graph)) # generation 1, clean + print("GEN1", flush=True) + fill(graph, 1500, 3000) + + def hook(name): + if name == point: + print("AT " + name, flush=True) + time.sleep(600) + storage.fault_hook = hook + st.write_generation(serialize(graph)) # stalls inside, parent kills us + else: + per_round = int(sys.argv[3]) + base = graph.node_count() + print("READY", flush=True) + while True: + fill(graph, base, base + per_round) + base += per_round + st.write_generation(serialize(graph)) + print(f"SAVED {base}", flush=True) + + +if __name__ == "__main__": + main() diff --git a/isongraph-vector-server/tests/test_api_basic.py b/isongraph-vector-server/tests/test_api_basic.py new file mode 100644 index 0000000..5e7a348 --- /dev/null +++ b/isongraph-vector-server/tests/test_api_basic.py @@ -0,0 +1,266 @@ +"""Happy-path API behaviour and the error model.""" + +import json + +import pytest + +from conftest import DIM, create_graph, seed + + +def test_health_and_ready_need_no_key(node): + anon = node.anon() + assert anon.get("/healthz")[:2] == (200, {"status": "ok"}) + s, body, _ = anon.get("/readyz") + assert s == 200 and body == {"status": "ready", "role": "primary"} + + +def test_graph_lifecycle(alice): + g = create_graph(alice, "kb") + assert g["mode"] == "server" and g["dimension"] == DIM and g["nodes"] == 0 + s, body, _ = alice.post("/v1/graphs", {"name": "kb"}) + assert s == 409 and body["error"]["code"] == "already_exists" + s, body, _ = alice.get("/v1/graphs") + assert [x["name"] for x in body["items"]] == ["kb"] + s, body, _ = alice.get("/v1/graphs/kb") + assert s == 200 and body["name"] == "kb" + assert alice.delete("/v1/graphs/kb")[0] == 200 + assert alice.get("/v1/graphs/kb")[0] == 404 + assert alice.delete("/v1/graphs/kb")[0] == 404 + + +def test_search_returns_best_match_first(alice): + create_graph(alice) + seed(alice) + s, r, _ = alice.post("/v1/graphs/kb/search", {"query": "trains machine learning models Bob", "top_k": 3, + "threshold": -1.0}) + assert s == 200, r + assert r["count"] == 3 + # MockEncoder is deterministic: the identical text has similarity 1.0. + s, r2, _ = alice.post("/v1/graphs/kb/search", {"query": "Bob trains machine learning models", "top_k": 1}) + assert s == 200 + s, r3, _ = alice.post("/v1/graphs/kb/search", {"query": "Bob trains machine learning models", "top_k": 3, + "threshold": -1.0}) + scores = [x["score"] for x in r3["results"]] + assert scores == sorted(scores, reverse=True) + + +def test_exact_text_scores_one(alice): + create_graph(alice) + seed(alice) + # The embed text for Bob is "Bob trains machine learning models" (name + description). + s, r, _ = alice.post("/v1/graphs/kb/search", {"query": "Bob trains machine learning models", "top_k": 1}) + top = r["results"][0] + assert top["id"] == 2 and top["score"] == pytest.approx(1.0, abs=1e-5) + assert top["properties"]["name"] == "Bob" + + +def test_vector_query_and_filters(alice): + create_graph(alice) + seed(alice) + s, r, _ = alice.post("/v1/graphs/kb/search", { + "query": "anything", "top_k": 10, "threshold": -1.0, + "filter": {"where": [{"field": "age", "op": "gte", "value": 40}]}}) + assert s == 200 + assert [x["id"] for x in r["results"]] == [2] + s, r, _ = alice.post("/v1/graphs/kb/search", { + "query": "anything", "top_k": 10, "threshold": -1.0, + "filter": {"types": ["person"], "where": [{"field": "name", "op": "in", "value": ["Alice", "Carol"]}]}}) + assert sorted(str(x["id"]) for x in r["results"]) == ["1", "carol"] + + +def test_similar_path_multihop_subgraph(alice): + create_graph(alice) + seed(alice) + s, r, _ = alice.post("/v1/graphs/kb/similar", {"node": {"type": "person", "id": 1}, "top_k": 5, + "threshold": -1.0}) + assert s == 200 and r["node_embedded"] and all(x["id"] != 1 for x in r["results"]) + + s, r, _ = alice.post("/v1/graphs/kb/multi_hop", { + "query": "Bob trains machine learning models", "rel_type": "KNOWS", "max_hops": 2, "threshold": 0.5}) + assert s == 200 + by_id = {str(x["id"]): x for x in r["results"]} + assert by_id["2"]["hops"] == 0 and by_id["carol"]["hops"] == 1 + assert by_id["carol"]["score"] == pytest.approx(by_id["2"]["score"] * 0.8, abs=1e-4) + + s, r, _ = alice.post("/v1/graphs/kb/path", { + "query": "Alice builds payment APIs", "target": {"type": "person", "id": "carol"}, "threshold": 0.5}) + assert s == 200 and r["found"] + assert [p["id"] for p in r["result"]["path"]] == [1, 2, "carol"] + + s, r, _ = alice.post("/v1/graphs/kb/subgraph", {"query": "Bob trains machine learning models", + "max_hops": 1, "threshold": 0.5}) + assert s == 200 and r["node_count"] == 2 and r["edge_count"] == 1 + + +def test_weight_field_changes_scores(alice): + create_graph(alice) + seed(alice) + q = {"query": "Alice builds payment APIs", "rel_type": "KNOWS", "max_hops": 1, "threshold": 0.5} + s, plain, _ = alice.post("/v1/graphs/kb/multi_hop", q) + s, weighted, _ = alice.post("/v1/graphs/kb/multi_hop", {**q, "weight_field": "weight"}) + p = {str(x["id"]): x["score"] for x in plain["results"]} + w = {str(x["id"]): x["score"] for x in weighted["results"]} + assert w["2"] == pytest.approx(p["2"] * 0.9, abs=1e-4) + + +def test_node_crud_and_patch_reembeds(alice): + create_graph(alice) + seed(alice) + s, r, _ = alice.get("/v1/graphs/kb/nodes/person/2") + assert s == 200 and r["embedded"] and r["properties"]["age"] == 45 + s, r, _ = alice.patch("/v1/graphs/kb/nodes/person/2", {"properties": {"description": "now writes poetry"}}) + assert s == 200 and r["properties"]["age"] == 45 + s, r, _ = alice.post("/v1/graphs/kb/search", {"query": "Bob now writes poetry", "top_k": 1}) + assert r["results"][0]["id"] == 2 and r["results"][0]["score"] == pytest.approx(1.0, abs=1e-5) + s, r, _ = alice.patch("/v1/graphs/kb/nodes/person/2", {"remove": ["age"]}) + assert "age" not in r["properties"] + assert alice.delete("/v1/graphs/kb/nodes/person/2")[0] == 200 + assert alice.get("/v1/graphs/kb/nodes/person/2")[0] == 404 + # the edge touching it went with it + s, r, _ = alice.get("/v1/graphs/kb/edges") + assert r["items"] == [] or all(e["source"]["id"] != 2 and e["target"]["id"] != 2 for e in r["items"]) + + +def test_put_replaces_properties(alice): + create_graph(alice) + alice.put("/v1/graphs/kb/nodes/doc/a", {"properties": {"name": "x", "extra": 1}}) + alice.put("/v1/graphs/kb/nodes/doc/a", {"properties": {"name": "y"}}) + s, r, _ = alice.get("/v1/graphs/kb/nodes/doc/a") + assert r["properties"] == {"name": "y"} + + +def test_client_mode_graph_takes_vectors(alice): + create_graph(alice, "vec", embedding={"mode": "client", "dimension": 3}) + s, r, _ = alice.post("/v1/graphs/vec/nodes", {"nodes": [ + {"type": "d", "id": 1, "vector": [1, 0, 0]}, {"type": "d", "id": 2, "vector": [0, 1, 0]}]}) + assert s == 200 and r["embedded"] == 2 + s, r, _ = alice.post("/v1/graphs/vec/search", {"vector": [0.9, 0.1, 0], "top_k": 1}) + assert r["results"][0]["id"] == 1 + # text is refused both ways + assert alice.post("/v1/graphs/vec/search", {"query": "x"})[1]["error"]["code"] == "text_query_unsupported" + assert alice.post("/v1/graphs/vec/nodes", {"nodes": [{"type": "d", "id": 3, "embed_text": "x"}]})[0] == 400 + # wrong length + s, r, _ = alice.post("/v1/graphs/vec/nodes", {"nodes": [{"type": "d", "id": 3, "vector": [1, 2]}]}) + assert s == 400 and r["error"]["code"] == "dimension_mismatch" + s, r, _ = alice.post("/v1/graphs/vec/search", {"vector": [1, 2]}) + assert s == 400 and r["error"]["code"] == "dimension_mismatch" + # server-mode graph refuses caller vectors + create_graph(alice, "txt") + s, r, _ = alice.post("/v1/graphs/txt/nodes", {"nodes": [{"type": "d", "id": 1, "vector": [1] * DIM}]}) + assert s == 400 + + +def test_pagination(alice): + create_graph(alice) + alice.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "n", "id": i, "properties": {"name": f"n{i}"}} + for i in range(25)]}) + seen, cursor = [], None + while True: + s, r, _ = alice.get("/v1/graphs/kb/nodes?limit=10" + (f"&cursor={cursor}" if cursor else "")) + assert s == 200 + seen += [x["id"] for x in r["items"]] + cursor = r["next_cursor"] + if not cursor: + break + assert seen == sorted(range(25)) + s, r, _ = alice.get("/v1/graphs/kb/nodes?cursor=%%%") + assert s == 400 and r["error"]["code"] == "invalid_cursor" + assert alice.get("/v1/graphs/kb/nodes?limit=0")[0] == 400 + assert alice.get("/v1/graphs/kb/nodes?limit=99999")[0] == 400 + # edges + alice.post("/v1/graphs/kb/edges", {"edges": [ + {"rel": "R", "source": {"type": "n", "id": i}, "target": {"type": "n", "id": i + 1}} for i in range(12)]}) + got, cursor = 0, None + while True: + s, r, _ = alice.get("/v1/graphs/kb/edges?limit=5" + (f"&cursor={cursor}" if cursor else "")) + got += len(r["items"]) + cursor = r["next_cursor"] + if not cursor: + break + assert got == 12 + + +def test_error_model_is_consistent_and_leaks_nothing(alice): + s, r, h = alice.get("/v1/graphs/nope") + assert s == 404 + assert set(r["error"]) >= {"code", "message", "request_id"} + assert r["error"]["request_id"] == h["X-Request-Id"] + for bad in ({"name": "UPPER"}, {"name": "../x"}, {}, {"name": "ok", "bogus": 1}): + s, r, _ = alice.post("/v1/graphs", bad) + assert s == 400 and r["error"]["code"] == "validation_failed", bad + text = json.dumps(r) + assert "Traceback" not in text and ".py" not in text + + +def test_validation_rejects_bad_bodies(alice): + create_graph(alice) + cases = [ + {"nodes": [{"type": "a:b", "id": 1}]}, # bad type + {"nodes": [{"type": "a", "id": "has space"}]}, + {"nodes": [{"type": "a", "id": 1.5}]}, + {"nodes": [{"type": "a", "id": True}]}, + {"nodes": [{"type": "a", "id": 1, "properties": {"x": [1]}}]}, + {"nodes": [{"type": "a", "id": 1, "properties": {"x": {"y": 1}}}]}, + {"nodes": [{"type": "a", "id": 1, "properties": {"id": 1}}]}, # reserved column + {"nodes": [{"type": "a", "id": 1, "properties": {"_x": 1}}]}, + {"nodes": [{"type": "a", "id": 1, "properties": {"x": 1e300}}]}, + {"nodes": [{"type": "a", "id": 1, "properties": {"x": "a b"}}]}, + {"nodes": [{"type": "a", "id": 1}, {"type": "a", "id": "1"}]}, # same node twice + {"nodes": []}, + {"nodes": [{"type": "a", "id": 1, "unknown": 1}]}, + ] + for body in cases: + s, r, _ = alice.post("/v1/graphs/kb/nodes", body) + assert s == 400, (body, s, r) + # duplicate JSON keys / NaN / bad JSON / wrong content-type + assert alice.request("POST", "/v1/graphs/kb/search", raw=b'{"query":"a","query":"b"}')[0] == 400 + assert alice.request("POST", "/v1/graphs/kb/search", raw=b'{"query":"a","vector":[NaN]}')[0] == 400 + assert alice.request("POST", "/v1/graphs/kb/search", raw=b'{not json')[0] == 400 + assert alice.request("POST", "/v1/graphs/kb/search", raw=b'\xff\xfe')[0] == 400 + s, r, _ = alice.request("POST", "/v1/graphs/kb/search", raw=b'{"query":"x"}', + headers={"Content-Type": "text/plain"}) + assert s == 415 + # unknown query parameter, wrong method + assert alice.get("/v1/graphs?bogus=1")[0] == 400 + s, _, h = alice.request("PUT", "/v1/graphs", {"name": "x"}) + assert s == 405 and "POST" in h["Allow"] + + +def test_ids_5_and_string_5_are_the_same_node(alice): + create_graph(alice) + alice.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "n", "id": 5, "properties": {"name": "five"}}]}) + s, r, _ = alice.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "n", "id": "5", "properties": {"name": "FIVE"}}]}) + assert s == 200 and r["updated"] == 1 + s, r, _ = alice.get("/v1/graphs/kb/nodes/n/5") + assert r["properties"]["name"] == "FIVE" + # "007" is a string id, but the library cannot hold both 7 and "007" + alice.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "n", "id": 7}]}) + s, r, _ = alice.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "n", "id": "007"}]}) + assert s == 409 and r["error"]["code"] == "id_conflict" + s, r, _ = alice.get("/v1/graphs/kb") + assert r["nodes"] == 2 # the rejected batch changed nothing + + +def test_edge_errors(alice): + create_graph(alice) + seed(alice) + s, r, _ = alice.post("/v1/graphs/kb/edges", {"edges": [ + {"rel": "X", "source": {"type": "person", "id": 1}, "target": {"type": "person", "id": 99}}]}) + assert s == 404 + # upsert existing edge updates properties + s, r, _ = alice.post("/v1/graphs/kb/edges", {"edges": [ + {"rel": "KNOWS", "source": {"type": "person", "id": 1}, "target": {"type": "person", "id": 2}, + "properties": {"weight": 0.1}}]}) + assert r == {"created": 0, "updated": 1} + s, r, _ = alice.post("/v1/graphs/kb/edges/delete", {"edges": [ + {"rel": "KNOWS", "source": {"type": "person", "id": 1}, "target": {"type": "person", "id": 2}}, + {"rel": "KNOWS", "source": {"type": "person", "id": 1}, "target": {"type": "person", "id": 2}}]}) + assert r == {"deleted": 1, "missing": 1} + + +def test_usage_endpoint(alice): + create_graph(alice) + seed(alice) + s, r, _ = alice.get("/v1/usage") + assert r["usage"] == {"graphs": 1, "nodes": 3, "edges": 2} + assert r["quotas"]["max_nodes"] == 100_000 diff --git a/isongraph-vector-server/tests/test_crash.py b/isongraph-vector-server/tests/test_crash.py new file mode 100644 index 0000000..109d845 --- /dev/null +++ b/isongraph-vector-server/tests/test_crash.py @@ -0,0 +1,253 @@ +"""Crash safety: SIGKILL a process in the middle of a save, then load what is on disk. + +Two kinds of test. The deterministic ones stall a child *inside* the save at a named +point (storage.fault_hook, a test seam) and kill it there; the stress ones kill a child +that is saving continuously at random moments. In both, the invariant is the same: +whatever is on disk must verify, with no repair step, and must be one whole generation. +""" + +import json +import os +import random +import signal +import subprocess +import sys +import time +from pathlib import Path + +import pytest + +from conftest import DIM, Client, SERVER_ROOT, create_graph, seed +from isongraph_vector import MockEncoder +from isongraph_vector_server.storage import GraphStorage + +CHILD = Path(__file__).resolve().parent / "crash_child.py" +pytestmark = pytest.mark.slow + + +def spawn(*args): + return subprocess.Popen([sys.executable, str(CHILD), *map(str, args)], stdout=subprocess.PIPE, + stderr=subprocess.PIPE, text=True, bufsize=1) + + +def read_until(proc, prefix, timeout=60): + end = time.time() + timeout + while time.time() < end: + line = proc.stdout.readline() + if not line: + if proc.poll() is not None: + raise AssertionError("child exited early: " + proc.stderr.read()[-2000:]) + continue + if line.startswith(prefix): + return line.strip() + raise AssertionError("timed out waiting for " + prefix) + + +def load(path): + st = GraphStorage(path) + graph, manifest, degraded, _ = st.load(st.read_meta(), MockEncoder(16)) + return st, graph, manifest, degraded + + +@pytest.mark.parametrize("point,expect_nodes,expect_gen", [ + ("graph_written", 1500, 1), # killed with a half-built temp dir + ("files_written", 1500, 1), # temp dir complete, not yet renamed + ("generation_renamed", 1500, 1), # gen-2 is on disk and complete, CURRENT still says 1 + ("current_tmp_written", 1500, 1), # CURRENT.tmp written, replace not done + ("current_flipped", 3000, 2), # CURRENT says 2 +]) +def test_kill_at_every_step_of_a_save(tmp_path, point, expect_nodes, expect_gen): + path = tmp_path / "g-kb" + proc = spawn("point", path, point) + try: + read_until(proc, "GEN1") + read_until(proc, "AT " + point) + proc.kill() + proc.wait(timeout=30) + finally: + if proc.poll() is None: + proc.kill() + # No repair step is run: load exactly what the dead process left behind. + st, graph, manifest, degraded = load(path) + assert not degraded, "CURRENT must always name a complete, verifying generation" + assert graph.node_count() == expect_nodes and manifest.generation == expect_gen + assert graph.embedding_store.count() == expect_nodes + assert st.verify_generation(st.gen_path(expect_gen)).graph_digest == manifest.graph_digest + graph.close() + # startup would now remove the debris; the next save still works and supersedes it + st.clean_tmp() + assert not [p for p in path.iterdir() if p.name.startswith(".tmp-")] + + +def test_next_save_after_a_crash_continues_cleanly(tmp_path): + path = tmp_path / "g-kb" + proc = spawn("point", path, "generation_renamed") + read_until(proc, "GEN1") + read_until(proc, "AT generation_renamed") + proc.kill() + proc.wait(timeout=30) + # gen-000002 exists on disk but CURRENT says 1; a new save must not collide with it + st, graph, manifest, degraded = load(path) + assert (path / "gen-000002").exists() and manifest.generation == 1 + from crash_child import fill + fill(graph, 5000, 5010) + from isongraph_vector_server.storage import serialize + m = st.write_generation(serialize(graph)) + assert m.generation == 3 + _, g2, m2, deg = load(path) + assert not deg and m2.generation == 3 and g2.node_count() == 1510 + + +def test_random_kills_during_continuous_saving(tmp_path): + path = tmp_path / "g-kb" + rnd = random.Random(7) + last_nodes = 0 + killed_mid_save = 0 + for round_no in range(10): + proc = spawn("loop", path, 400) + try: + read_until(proc, "READY") + read_until(proc, "SAVED") # at least one save lands + time.sleep(rnd.uniform(0.0, 0.35)) # then die at a random moment + finally: + proc.kill() + proc.wait(timeout=30) + had_tmp = any(p.name.startswith(".tmp-gen") for p in path.iterdir()) + killed_mid_save += had_tmp + st, graph, manifest, degraded = load(path) + assert not degraded, f"round {round_no}: CURRENT pointed at something that does not verify" + assert graph.node_count() == manifest.node_count + assert graph.node_count() % 400 == 0 and graph.node_count() >= last_nodes # whole generations only + assert graph.embedding_store.count() == graph.node_count() + last_nodes = graph.node_count() + graph.close() + assert last_nodes >= 400 * 10 or last_nodes > 0 + # (killed_mid_save is informational: it shows the kills did land inside saves) + print("kills that left a partial temp directory:", killed_mid_save) + + +def test_sigkill_of_a_real_server_loses_no_acknowledged_write_in_sync_mode(tmp_path): + from isongraph_vector_server import auth + data = tmp_path / "data" + cfg = tmp_path / "cfg.json" + cfg.write_text(json.dumps({ + "data_dir": str(data), "host": "127.0.0.1", "port": 0, "durability": "sync", "log_level": "CRITICAL", + "autosave_interval_seconds": 0, + "encoder": {"kind": "mock", "dimension": DIM}, + "quotas": {"rate_per_second": 100000, "rate_burst": 100000}})) + auth.create_tenant(str(data), "alice") + key = auth.add_key(str(data), "alice") + # port 0 is not discoverable from outside; pick a free one + import socket + s = socket.socket() + s.bind(("127.0.0.1", 0)) + port = s.getsockname()[1] + s.close() + cfg.write_text(cfg.read_text().replace('"port": 0', f'"port": {port}')) + env = dict(os.environ, PYTHONPATH=os.pathsep.join([str(SERVER_ROOT), str(Path(SERVER_ROOT).parent), + os.environ.get("PYTHONPATH", "")])) + boot = ("import sys; sys.path.insert(0, %r); import conftest; " + "from isongraph_vector_server.cli import main; sys.exit(main(['serve', '--config', %r]))" + % (str(Path(__file__).resolve().parent), str(cfg))) + + def start(): + p = subprocess.Popen([sys.executable, "-c", boot], env=env, stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE) + c = Client(f"http://127.0.0.1:{port}", key) + for _ in range(100): + try: + if c.get("/readyz", key=None)[0] == 200: + return p, c + except OSError: + time.sleep(0.1) + p.kill() + raise AssertionError("server did not start: " + p.stderr.read().decode()[-2000:]) + + proc, c = start() + try: + create_graph(c) + acked = [] + deadline = time.time() + 1.5 + i = 0 + while time.time() < deadline: + s, _, _ = c.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "n", "id": i, "properties": {"name": f"n{i}"}}]}) + if s == 200: + acked.append(i) + i += 1 + proc.kill() # SIGKILL / TerminateProcess: no shutdown hook runs + proc.wait(timeout=30) + assert len(acked) > 5 + proc, c = start() + have = {x["id"] for x in c.get("/v1/graphs/kb/nodes?limit=1000")[1]["items"]} + missing = [i for i in acked if i not in have] + assert not missing, f"acknowledged writes lost: {missing[:5]}" + assert c.get("/v1/graphs/kb")[1]["status"] == "ok" + finally: + proc.kill() + proc.wait(timeout=30) + + +def test_sigterm_style_shutdown_saves_unsaved_changes(tmp_path): + """The real CLI process, snapshot durability: a termination signal makes it stop accepting, + finish, write a final snapshot and exit 0 (SIGTERM on POSIX, CTRL_BREAK on Windows).""" + from isongraph_vector_server import auth + import socket + data = tmp_path / "data" + auth.create_tenant(str(data), "alice") + key = auth.add_key(str(data), "alice") + s = socket.socket() + s.bind(("127.0.0.1", 0)) + port = s.getsockname()[1] + s.close() + cfg = tmp_path / "cfg.json" + cfg.write_text(json.dumps({ + "data_dir": str(data), "host": "127.0.0.1", "port": port, "log_level": "CRITICAL", + "autosave_interval_seconds": 0, "encoder": {"kind": "mock", "dimension": DIM}})) + env = dict(os.environ) + boot = ("import sys; sys.path.insert(0, %r); import conftest; " + "from isongraph_vector_server.cli import main; sys.exit(main(['serve', '--config', %r]))" + % (str(Path(__file__).resolve().parent), str(cfg))) + flags = subprocess.CREATE_NEW_PROCESS_GROUP if os.name == "nt" else 0 + proc = subprocess.Popen([sys.executable, "-c", boot], env=env, stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, creationflags=flags) + c = Client(f"http://127.0.0.1:{port}", key) + try: + for _ in range(100): + try: + if c.get("/readyz", key=None)[0] == 200: + break + except OSError: + time.sleep(0.1) + create_graph(c) + seed(c) + assert c.get("/v1/graphs/kb")[1]["unsaved_changes"] is True + proc.send_signal(signal.CTRL_BREAK_EVENT if os.name == "nt" else signal.SIGTERM) + assert proc.wait(timeout=30) == 0 + finally: + if proc.poll() is None: + proc.kill() + st = GraphStorage(data / "tenants" / "alice" / "graphs" / "g-kb") + graph, manifest, degraded, _ = st.load(st.read_meta(), MockEncoder(16)) + assert not degraded and graph.node_count() == 3 and graph.edge_count() == 2 + + +def test_library_in_place_save_can_leave_a_mismatched_pair(tmp_path): + """Why generations exist: the library's own save() is two atomic renames, not one. Killed between + them, the graph file and its sidecar come from different saves. The library *detects* that on load + (digest check) but has no older copy to fall back to; the service's generations do.""" + from isongraph_vector import GraphEmbeddingMismatchError, SemanticGraph + from unittest import mock + enc = MockEncoder(8) + g = SemanticGraph(encoder=enc, auto_embed=False) + for i in range(5): + g.add_node("n", i) + g.embedding_store.add(("n", i), f"n{i}", vector=enc.encode(f"n{i}")) + path = tmp_path / "plain.isong" + g.save(path) + g.add_node("n", 99) + g.embedding_store.add(("n", 99), "n99", vector=enc.encode("n99")) + with mock.patch.object(type(g.embedding_store), "save_embeddings", side_effect=KeyboardInterrupt): + with pytest.raises(KeyboardInterrupt): + g.save(path) # "dies" after the graph file, before the sidecar + with pytest.raises(GraphEmbeddingMismatchError): + SemanticGraph.load(path, encoder=enc) # detected, but there is nothing to roll back to diff --git a/isongraph-vector-server/tests/test_isolation.py b/isongraph-vector-server/tests/test_isolation.py new file mode 100644 index 0000000..5a50ef3 --- /dev/null +++ b/isongraph-vector-server/tests/test_isolation.py @@ -0,0 +1,522 @@ +"""Adversarial tenant-isolation tests. + +The question every test asks: can tenant B (or an anonymous caller) read, +write, enumerate or *infer* anything about tenant A? Tenants share ids, graph +names and one process on purpose, so any bleed shows up as wrong content. +""" + +import json +import os +import threading +import time +from pathlib import Path + +import pytest + +from conftest import DIM, Client, create_graph, seed +from isongraph_vector_server import auth + +SECRET_A = "A-ONLY-SECRET-TEXT" + + +@pytest.fixture +def two(harness): + h = harness() + a, b = h.tenant("tenant-a"), h.tenant("tenant-b") + # identical graph name and identical node ids, different content + create_graph(a, "kb") + create_graph(b, "kb") + a.post("/v1/graphs/kb/nodes", {"nodes": [ + {"type": "doc", "id": 1, "properties": {"name": "alpha", "text": SECRET_A, "owner": "a"}}, + {"type": "doc", "id": 2, "properties": {"name": "beta", "text": "also A"}}]}) + b.post("/v1/graphs/kb/nodes", {"nodes": [ + {"type": "doc", "id": 1, "properties": {"name": "gamma", "text": "B content", "owner": "b"}}]}) + a.post("/v1/graphs/kb/edges", {"edges": [{"rel": "R", "source": {"type": "doc", "id": 1}, + "target": {"type": "doc", "id": 2}}]}) + return h, a, b + + +def tree(root): + return sorted(str(p.relative_to(root)) for p in Path(root).rglob("*")) + + +# ---- same ids / same names in two tenants --------------------------------------------------------------- + + +def test_same_graph_name_and_ids_are_separate_universes(two): + h, a, b = two + assert a.get("/v1/graphs/kb/nodes/doc/1")[1]["properties"]["owner"] == "a" + assert b.get("/v1/graphs/kb/nodes/doc/1")[1]["properties"]["owner"] == "b" + assert b.get("/v1/graphs/kb/nodes/doc/2")[0] == 404 # exists only for A + assert [x["id"] for x in b.get("/v1/graphs/kb/nodes")[1]["items"]] == [1] + assert b.get("/v1/graphs/kb/edges")[1]["items"] == [] + s, r, _ = b.post("/v1/graphs/kb/search", {"query": SECRET_A, "top_k": 50, "threshold": -1.0}) + assert SECRET_A not in json.dumps(r) and [x["id"] for x in r["results"]] == [1] + assert r["results"][0]["properties"]["owner"] == "b" + s, r, _ = b.post("/v1/graphs/kb/similar", {"node": {"type": "doc", "id": 1}, "top_k": 50, "threshold": -1.0}) + assert r["results"] == [] + s, r, _ = b.post("/v1/graphs/kb/multi_hop", {"query": "alpha", "threshold": -1.0, "max_hops": 3}) + assert all(x["properties"].get("owner") == "b" for x in r["results"]) + s, r, _ = b.post("/v1/graphs/kb/subgraph", {"query": "alpha", "threshold": -1.0, "max_hops": 3}) + assert SECRET_A not in json.dumps(r) + # B deleting "its" kb must not touch A's + assert b.delete("/v1/graphs/kb")[0] == 200 + assert a.get("/v1/graphs/kb")[1]["nodes"] == 2 + assert a.get("/v1/graphs/kb/nodes/doc/1")[1]["properties"]["text"] == SECRET_A + + +def test_writes_to_same_ids_do_not_cross(two): + h, a, b = two + b.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "doc", "id": 2, "properties": {"name": "B-two"}}]}) + b.patch("/v1/graphs/kb/nodes/doc/1", {"properties": {"name": "B-changed"}}) + assert a.get("/v1/graphs/kb/nodes/doc/2")[1]["properties"]["name"] == "beta" + assert a.get("/v1/graphs/kb/nodes/doc/1")[1]["properties"]["name"] == "alpha" + s, r, _ = b.post("/v1/graphs/kb/nodes/delete", {"nodes": [{"type": "doc", "id": 2}]}) + assert r["deleted"] == 1 + assert a.get("/v1/graphs/kb/nodes/doc/2")[0] == 200 + + +def test_unknown_to_me_looks_identical_whether_or_not_it_exists_for_someone_else(harness): + h = harness() + a, b = h.tenant("tenant-a"), h.tenant("tenant-b") + create_graph(a, "only-a") + s1, r1, _ = b.get("/v1/graphs/only-a") # exists, for A + s2, r2, _ = b.get("/v1/graphs/never-existed") # exists for nobody + assert (s1, r1["error"]["code"], r1["error"]["message"]) == (s2, r2["error"]["code"], r2["error"]["message"]) + for path, method, body in [("/v1/graphs/{}", "DELETE", None), ("/v1/graphs/{}/nodes", "GET", None), + ("/v1/graphs/{}/search", "POST", {"query": "x"}), + ("/v1/graphs/{}/save", "POST", {}), + ("/v1/graphs/{}/nodes/doc/1", "GET", None)]: + x = b.request(method, path.format("only-a"), body) + y = b.request(method, path.format("never-existed"), body) + assert x[0] == y[0] == 404 + assert x[1]["error"]["code"] == y[1]["error"]["code"] and x[1]["error"]["message"] == y[1]["error"]["message"] + assert a.get("/v1/graphs/only-a")[0] == 200 # and B's DELETE did nothing + + +def test_graph_listing_and_usage_are_own_only(two): + h, a, b = two + create_graph(a, "a-secret-graph-name") + names = [g["name"] for g in b.get("/v1/graphs")[1]["items"]] + assert names == ["kb"] + assert "a-secret-graph-name" not in json.dumps(b.get("/v1/graphs")[1]) + assert b.get("/v1/usage")[1]["usage"] == {"graphs": 1, "nodes": 1, "edges": 0} + assert a.get("/v1/usage")[1]["usage"]["graphs"] == 2 + + +# ---- batch endpoints --------------------------------------------------------------------------------------- + + +def test_edge_batch_cannot_reach_other_tenants_nodes(two): + h, a, b = two + # B names node doc/2 (exists only in A's kb) as an edge endpoint + s, r, _ = b.post("/v1/graphs/kb/edges", {"edges": [ + {"rel": "R", "source": {"type": "doc", "id": 1}, "target": {"type": "doc", "id": 2}}]}) + assert s == 404 + assert b.get("/v1/graphs/kb/edges")[1]["items"] == [] + assert a.get("/v1/graphs/kb/edges")[1]["items"][0]["properties"] == {} + # and the same edge delete removes nothing of A's + s, r, _ = b.post("/v1/graphs/kb/edges/delete", {"edges": [ + {"rel": "R", "source": {"type": "doc", "id": 1}, "target": {"type": "doc", "id": 2}}]}) + assert r == {"deleted": 0, "missing": 1} + assert len(a.get("/v1/graphs/kb/edges")[1]["items"]) == 1 + + +def test_node_batch_delete_reports_the_same_for_foreign_and_nonexistent(two): + h, a, b = two + s, r, _ = b.post("/v1/graphs/kb/nodes/delete", {"nodes": [{"type": "doc", "id": 2}, {"type": "doc", "id": 77}]}) + assert r == {"deleted": 0, "missing": 2} + assert a.get("/v1/graphs/kb")[1]["nodes"] == 2 + + +def test_upsert_batch_with_colliding_ids_only_touches_own_graph(two): + h, a, b = two + b.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "doc", "id": i, "properties": {"name": f"b{i}"}} + for i in range(1, 30)]}) + assert a.get("/v1/graphs/kb")[1]["nodes"] == 2 + assert a.get("/v1/graphs/kb/nodes/doc/1")[1]["properties"]["owner"] == "a" + + +def test_cursors_from_another_tenant_do_not_enumerate_it(two): + h, a, b = two + s, r, _ = a.get("/v1/graphs/kb/nodes?limit=1") + cursor_from_a = r["next_cursor"] + assert cursor_from_a + s, r, _ = b.get(f"/v1/graphs/kb/nodes?cursor={cursor_from_a}") + assert s == 200 and r["items"] == [] and SECRET_A not in json.dumps(r) + s, r, _ = b.get("/v1/graphs/kb/nodes?cursor=" + "AAAA") + assert s == 400 + + +# ---- credentials and headers ----------------------------------------------------------------------------------- + + +def test_no_header_selects_a_tenant(two): + h, a, b = two + tricks = { + "X-Tenant": "tenant-a", "X-Tenant-Id": "tenant-a", "X-Tenant-ID": "tenant-a", + "X-Forwarded-For": "127.0.0.1", "X-Real-IP": "127.0.0.1", "X-Original-URL": "/v1/graphs/kb", + "X-Forwarded-Host": "tenant-a.example", "Host": "tenant-a.example", "Forwarded": "for=tenant-a", + "X-Igv-Tenant": "tenant-a", "X-Auth-Tenant": "tenant-a", "X-HTTP-Method-Override": "DELETE", + "Cookie": "tenant=tenant-a; session=tenant-a", + } + for k, v in tricks.items(): + s, r, _ = b.get("/v1/graphs/kb/nodes/doc/2", headers={k: v}) + assert s == 404, (k, r) + s, r, _ = b.get("/v1/graphs/kb/nodes", headers={k: v}) + assert [x["id"] for x in r["items"]] == [1], k + # and the same tricks do not authenticate an anonymous caller as A + for k, v in tricks.items(): + assert h.anon().get("/v1/graphs", headers={k: v})[0] == 401 + + +def test_query_and_body_cannot_name_a_tenant(two): + h, a, b = two + for q in ("tenant=tenant-a", "tenant_id=tenant-a", "as=tenant-a", "api_key=" + a.key, "key=" + a.key, + "access_token=" + a.key): + assert b.get("/v1/graphs?" + q)[0] == 400, q + assert h.anon().get("/v1/graphs?" + q)[0] in (400, 401), q + s, r, _ = b.post("/v1/graphs/kb/search", {"query": "x", "tenant": "tenant-a"}) + assert s == 400 + s, r, _ = b.post("/v1/graphs", {"name": "zz", "tenant": "tenant-a"}) + assert s == 400 + assert a.get("/v1/graphs")[1]["items"][0]["name"] == "kb" + # the key in the URL does not authenticate either + assert h.anon().get("/v1/graphs", key=None)[0] == 401 + + +def test_key_forgery_variants_all_fail(two): + h, a, b = two + ta, ka, sa = a.key.split(".")[0], a.key.split(".")[1], a.key.split(".")[2] + tb, kb, sb = b.key.split(".")[0], b.key.split(".")[1], b.key.split(".")[2] + forged = [ + f"{ta}.{kb}.{sb}", # B's key id+secret under A's tenant prefix + f"{ta}.{ka}.{sb}", # A's key id, B's secret + f"{tb}.{ka}.{sa}", # A's key id+secret under B's prefix + f"{ta}.{ka}.{sa[:-1]}", # truncated + f"{ta}.{ka}.{sa}x", # extended + a.key.upper(), a.key.replace("igv_", "IGV_"), + a.key + "\x00", "Bearer " + a.key, a.key.replace(".", ","), a.key + "\t.", + f"igv_tenant-a.{'0' * 16}.{sa}", f"igv_nobody.{ka}.{sa}", "igv_", "", "null", "undefined", + "A" * 5000, "{}", "' OR '1'='1", + ] + for f in forged: + s, r, _ = h.anon().get("/v1/graphs", headers={"Authorization": f"Bearer {f}"}) + assert s == 401, repr(f[:60]) + assert r["error"]["code"] == "unauthorized" + for scheme in ("Basic", "Token", "Digest", "bearer-ish"): + assert h.anon().get("/v1/graphs", headers={"Authorization": f"{scheme} {a.key}"})[0] == 401 + # sanity: the real keys work, lowercase scheme too + assert a.get("/v1/graphs")[0] == 200 + assert h.anon().get("/v1/graphs", headers={"Authorization": f"bearer {a.key}"})[0] == 200 + + +def test_duplicate_authorization_headers_are_rejected(two): + import http.client + h, a, b = two + for first, second in ((a.key, b.key), (b.key, a.key), (a.key, "garbage")): + conn = http.client.HTTPConnection(a.host, a.port, timeout=10) + conn.putrequest("GET", "/v1/graphs/kb/nodes") + conn.putheader("Authorization", f"Bearer {first}") + conn.putheader("Authorization", f"Bearer {second}") + conn.endheaders() + resp = conn.getresponse() + resp.read() + assert resp.status == 401 + conn.close() + # comma-joined credentials are one malformed token, not two + s, _, _ = h.anon().get("/v1/graphs", headers={"Authorization": f"Bearer {a.key}, Bearer {b.key}"}) + assert s == 401 + + +def test_unknown_tenant_and_wrong_secret_are_indistinguishable(two): + h, a, b = two + sa = a.key.split(".")[2] + wrong_secret = a.key[:-3] + ("AAA" if not a.key.endswith("AAA") else "BBB") + unknown_tenant = f"igv_zzz-nobody.{a.key.split('.')[1]}.{sa}" + unknown_key_id = f"igv_tenant-a.{'f' * 16}.{sa}" + outs = [] + for k in (wrong_secret, unknown_tenant, unknown_key_id, "garbage"): + s, r, hd = h.anon().get("/v1/graphs", headers={"Authorization": f"Bearer {k}"}) + r["error"].pop("request_id") + outs.append((s, json.dumps(r, sort_keys=True), hd.get("WWW-Authenticate"))) + assert len(set(outs)) == 1 + + +def test_revoked_key_and_disabled_tenant_stop_working(harness): + h = harness() + c = h.tenant("tenant-a") + assert c.get("/v1/graphs")[0] == 200 + key_id = c.key.split(".")[1] + auth.revoke_key(h.data_dir, "tenant-a", key_id) + assert c.get("/v1/graphs")[0] == 401 + c2 = h.tenant("tenant-a") # a new key for the same tenant + assert c2.get("/v1/graphs")[0] == 200 + auth.set_disabled(h.data_dir, "tenant-a", True) + assert c2.get("/v1/graphs")[0] == 401 + auth.set_disabled(h.data_dir, "tenant-a", False) + assert c2.get("/v1/graphs")[0] == 200 + + +def test_key_created_after_the_server_started_works_at_once(harness): + """Regression: a freshly minted key used to be rejected until the registry's + recheck interval passed (observed failing before the fix).""" + h = harness(registry_recheck_seconds=60.0) + first = h.tenant("tenant-a") + assert first.get("/v1/graphs")[0] == 200 + second = h.tenant("tenant-a") # new key, tenant already cached + assert second.get("/v1/graphs")[0] == 200 + brand_new = h.tenant("tenant-c") # new tenant + assert brand_new.get("/v1/graphs")[0] == 200 + + +def test_read_only_key_cannot_write(harness): + h = harness() + rw = h.tenant("tenant-a") + create_graph(rw, "kb") + seed(rw) + ro = Client(h.server.url, auth.add_key(h.data_dir, "tenant-a", "ro")) + assert ro.post("/v1/graphs/kb/search", {"query": "x", "threshold": -1.0})[0] == 200 + assert ro.get("/v1/graphs/kb/nodes")[0] == 200 + for method, path, body in [("POST", "/v1/graphs", {"name": "n2"}), ("DELETE", "/v1/graphs/kb", None), + ("POST", "/v1/graphs/kb/nodes", {"nodes": [{"type": "a", "id": 1}]}), + ("PUT", "/v1/graphs/kb/nodes/a/1", {}), ("PATCH", "/v1/graphs/kb/nodes/person/1", {}), + ("DELETE", "/v1/graphs/kb/nodes/person/1", None), + ("POST", "/v1/graphs/kb/edges", {"edges": [{"rel": "R", "source": {"type": "a", "id": 1}, + "target": {"type": "a", "id": 1}}]}), + ("POST", "/v1/graphs/kb/save", {}), ("POST", "/v1/graphs/kb/reembed", {})]: + s, r, _ = ro.request(method, path, body) + assert s == 403 and r["error"]["code"] == "read_only_key", (method, path) + assert rw.get("/v1/graphs/kb")[1]["nodes"] == 3 + + +def test_tenant_key_forged_with_valid_shape_for_other_tenant_dir(two): + """A key is only valid for the tenant whose tenant.json holds its hash.""" + h, a, b = two + # copy B's hash record into A's file would be filesystem access; instead check the + # protocol-level fact: the same key id existing in two tenants authenticates only its own. + _, _, secret_b = b.key.split(".") + kid_b = b.key.split(".")[1] + assert h.anon().get("/v1/graphs", headers={"Authorization": f"Bearer igv_tenant-a.{kid_b}.{secret_b}"})[0] == 401 + + +# ---- names, paths, filesystem ----------------------------------------------------------------------------------- + + +CRAFTED = ["..", "../x", "..%2fx", "%2e%2e", "%2e%2e%2f", "a%2fb", "a/b", "a\\b", "a%5cb", "%00", "a%00b", + "CON", "NUL", "aux.", "a.", "a ", " a", "a%20", "a::$DATA", "..\\..", "....//", + "x" * 64, "x" * 300, "UPPER", "café", "a;b", "a:b", "a|b", "a*b", "a?b", "~", ".", "%252e%252e", + "%c0%ae%c0%ae", "%ef%bc%8e", "‮", "a\nb", "a%0ab", "%"] + + +def test_crafted_graph_names_cannot_escape_or_create_anything(two): + h, a, b = two + root = Path(h.data_dir).parent + before = tree(root) + for name in CRAFTED: + # create via body + s, r, _ = b.post("/v1/graphs", {"name": name.replace("%2f", "/").replace("%5c", "\\")}) + assert s in (400, 404), (name, s) + # address via path (the server decodes once, then validates) + for method in ("GET", "DELETE", "POST"): + body = {"query": "x"} if method == "POST" else None + from urllib.parse import quote + s, r, _ = b.request(method, f"/v1/graphs/{quote(name, safe='%')}" + + ("/search" if method == "POST" else ""), body) + assert s in (400, 404), (name, method, s) + assert tree(root) == before + assert a.get("/v1/graphs/kb")[0] == 200 and b.get("/v1/graphs/kb")[0] == 200 + + +def test_crafted_node_types_and_ids_in_path_and_body(two): + h, a, b = two + root = Path(h.data_dir).parent + before = tree(root) + for bad in ["..", "%2e%2e", "a%2fb", "a:b", "a|b", "a%00", "a%20b", "x" * 200, "%"]: + for path in (f"/v1/graphs/kb/nodes/{bad}/1", f"/v1/graphs/kb/nodes/doc/{bad}"): + for method in ("GET", "DELETE", "PUT", "PATCH"): + s, _, _ = b.request(method, path, {} if method in ("PUT", "PATCH") else None) + assert s in (400, 404), (path, method, s) + assert tree(root) == before + assert a.get("/v1/graphs/kb")[1]["nodes"] == 2 + + +def test_path_oddities_are_rejected(two): + h, a, b = two + import http.client + for target in ["/v1//graphs", "/v1/graphs/", "/v1/./graphs", "/v1/graphs/../graphs", + "/v1/graphs/%2e%2e/%2e%2e/etc", "/v1/graphs/kb/../../x", "/v1/graphs#frag", + "http://evil/v1/graphs", "*"]: + conn = http.client.HTTPConnection(b.host, b.port, timeout=10) + conn.putrequest("GET", target, skip_host=False) + conn.putheader("Authorization", f"Bearer {b.key}") + conn.endheaders() + resp = conn.getresponse() + data = resp.read() + assert resp.status in (400, 404, 405), (target, resp.status) + assert b"Traceback" not in data + conn.close() + + +def test_traversal_in_replication_routes(harness): + h = harness() + a = h.tenant("tenant-a") + # no replication token configured: the endpoints do not exist + for p in ("/_repl/v1/index", "/_repl/v1/tenant/tenant-a", "/_repl/v1/file/tenant-a/kb/000001/graph.isong"): + assert a.get(p)[0] == 404 + assert h.anon().get(p)[0] == 404 + + +def test_replication_endpoints_reject_tenant_keys_and_traversal(harness): + token = "r" * 40 + h = harness(replication_token=token) + a = h.tenant("tenant-a") + create_graph(a, "kb") + seed(a) + a.post("/v1/graphs/kb/save", {}) + assert a.get("/_repl/v1/index")[0] == 401 # a tenant key is not a replication token + assert h.anon().get("/_repl/v1/index")[0] == 401 + repl = Client(h.server.url, token) + s, idx, _ = repl.get("/_repl/v1/index") + assert s == 200 and idx["tenants"][0]["tenant_id"] == "tenant-a" + assert repl.get("/_repl/v1/file/tenant-a/kb/000001/graph.isong")[0] == 200 + # the replication token is not a tenant key + assert repl.get("/v1/graphs")[0] == 401 + for bad in ("/_repl/v1/file/tenant-a/kb/000001/..%2f..%2ftenant.json", "/_repl/v1/file/tenant-a/kb/000001/tenant.json", + "/_repl/v1/file/tenant-a/kb/../000001/graph.isong", "/_repl/v1/file/tenant-a/KB/000001/graph.isong", + "/_repl/v1/file/..%2f/kb/000001/graph.isong", "/_repl/v1/file/tenant-a/kb/1/graph.isong", + "/_repl/v1/file/tenant-a/kb/000001/MANIFEST.json.bak", "/_repl/v1/tenant/..", + "/_repl/v1/meta/tenant-a/..%2f..", "/_repl/v1/tenant/tenant-a%00"): + assert repl.get(bad)[0] in (400, 404), bad + + +# ---- metrics, errors, quotas: no side channels ------------------------------------------------------------------ + + +def test_metrics_endpoint_is_protected_and_aggregate(harness): + h = harness() + a = h.tenant("tenant-a") + assert a.get("/metrics")[0] == 404 # not configured: does not exist + token = "m" * 40 + h2 = harness(metrics_token=token) + t1 = h2.tenant("tenant-one") + create_graph(t1, "secretgraph") + assert t1.get("/metrics")[0] == 401 # a tenant key is not the metrics token + assert h2.anon().get("/metrics")[0] == 401 + s, text, _ = Client(h2.server.url, token).get("/metrics") + assert s == 200 and "igv_requests_total" in text + assert "tenant-one" not in text and "secretgraph" not in text + + +def test_quota_errors_report_only_own_usage(harness): + h = harness() + a = h.tenant("tenant-a") + b = h.tenant("tenant-b", quotas={"max_nodes": 5}) + create_graph(a, "kb") + a.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "n", "id": i} for i in range(100)]}) + create_graph(b, "kb") + s, r, _ = b.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "n", "id": i} for i in range(10)]}) + assert s == 403 and r["error"]["code"] == "quota_exceeded" + assert "100" not in r["error"]["message"] # A's usage never appears in B's error + assert "0 used" in r["error"]["message"] + assert b.get("/v1/graphs/kb")[1]["nodes"] == 0 # rejected batch applied nothing + + +def test_rate_limits_are_per_tenant(harness): + h = harness() + a = h.tenant("tenant-a", quotas={"rate_per_second": 0.5, "rate_burst": 3}) + b = h.tenant("tenant-b") + codes = [a.get("/v1/usage")[0] for _ in range(10)] + assert 429 in codes and codes[0] == 200 + assert all(b.get("/v1/usage")[0] == 200 for _ in range(20)) # B is unaffected + s, r, hd = a.get("/v1/usage") + assert s == 429 and "Retry-After" in hd and r["error"]["code"] == "rate_limited" + + +def test_failed_auth_throttle_blocks_the_peer_not_the_tenant(harness): + h = harness(failed_auth_per_minute=5) + a = h.tenant("tenant-a") + for _ in range(5): + assert h.anon().get("/v1/graphs", headers={"Authorization": "Bearer nope"})[0] == 401 + # the 6th bad attempt from this address is throttled ... + assert h.anon().get("/v1/graphs", headers={"Authorization": "Bearer nope"})[0] == 429 + # ... and so is a good key from the same address (that is the cost of throttling by address) + assert a.get("/v1/graphs")[0] == 429 + + +def test_logs_never_contain_keys(harness, caplog): + import logging + from isongraph_vector_server.logs import setup_logging + import io + stream = io.StringIO() + setup_logging("DEBUG", stream) + h = harness() + a = h.tenant("tenant-a") + create_graph(a, "kb") + a.post("/v1/graphs/kb/search", {"query": "x"}) + h.anon().get("/v1/graphs", headers={"Authorization": f"Bearer {a.key[:-4]}AAAA"}) + h.anon().get("/v1/graphs", headers={"Authorization": f"Bearer {a.key}x"}) + secret = a.key.split(".")[2] + out = stream.getvalue() + assert out and secret not in out and a.key not in out + assert secret[:12] not in out + from isongraph_vector_server.logs import redact + assert secret not in redact(f"oops {a.key} and Authorization: Bearer {a.key}") + + +def test_secret_is_stored_only_as_a_salted_hash(harness): + h = harness() + a = h.tenant("tenant-a") + secret = a.key.split(".")[2] + raw = (Path(h.data_dir) / "tenants" / "tenant-a" / "tenant.json").read_text() + assert secret not in raw and a.key not in raw + doc = json.loads(raw) + rec = doc["keys"][0] + assert rec["algo"] == "hmac-sha256" and len(rec["salt"]) == 32 and len(rec["hash"]) == 64 + b = h.tenant("tenant-a") # a second key gets a different salt + doc = json.loads((Path(h.data_dir) / "tenants" / "tenant-a" / "tenant.json").read_text()) + assert doc["keys"][0]["salt"] != doc["keys"][1]["salt"] + # describe_tenant (CLI) never prints salts or hashes + assert "hash" not in json.dumps(auth.describe_tenant(h.data_dir, "tenant-a")) + + +# ---- concurrency ---------------------------------------------------------------------------------------------------- + + +def test_concurrent_tenants_with_identical_ids_never_see_each_other(harness): + h = harness() + tenants = [h.tenant(f"tenant-{i}") for i in range(4)] + for i, c in enumerate(tenants): + create_graph(c, "kb") + errors = [] + + def work(i, c): + try: + marker = f"MARKER{i}" + for rnd in range(15): + s, r, _ = c.post("/v1/graphs/kb/nodes", {"nodes": [ + {"type": "doc", "id": n, "properties": {"name": f"{marker}-{n}", "owner": marker}} + for n in range(20)]}) + assert s == 200, r + s, r, _ = c.post("/v1/graphs/kb/search", {"query": f"{marker}-3", "top_k": 20, "threshold": -1.0}) + assert s == 200 + owners = {x["properties"]["owner"] for x in r["results"]} + assert owners == {marker}, owners + s, r, _ = c.get("/v1/graphs/kb/nodes/doc/5") + assert r["properties"]["owner"] == marker + except AssertionError as exc: + errors.append(exc) + + threads = [threading.Thread(target=work, args=(i, c)) for i, c in enumerate(tenants)] + [t.start() for t in threads] + [t.join() for t in threads] + assert not errors, errors[:3] + + +def test_operator_tokens_are_throttled_like_keys(harness): + h = harness(metrics_token="m" * 40, replication_token="r" * 40, failed_auth_per_minute=4) + anon = h.anon() + for path in ("/metrics", "/_repl/v1/index", "/metrics", "/_repl/v1/index"): + assert anon.get(path, headers={"Authorization": "Bearer wrong"})[0] == 401 + assert anon.get("/metrics", headers={"Authorization": "Bearer " + "m" * 40})[0] == 429 # peer is throttled + assert anon.get("/_repl/v1/index", headers={"Authorization": "Bearer " + "r" * 40})[0] == 429 diff --git a/isongraph-vector-server/tests/test_lifecycle.py b/isongraph-vector-server/tests/test_lifecycle.py new file mode 100644 index 0000000..f855f09 --- /dev/null +++ b/isongraph-vector-server/tests/test_lifecycle.py @@ -0,0 +1,331 @@ +"""Embedding model lifecycle, ANN opt-in, traversal options, CLI and config.""" + +import io +import json +import os + +import pytest + +from conftest import DIM, create_graph, seed +from isongraph_vector import MockEncoder +from isongraph_vector_server import auth +from isongraph_vector_server.cli import main +from isongraph_vector_server.config import ConfigError, load_config +from isongraph_vector_server.logs import setup_logging + + +class ModelB(MockEncoder): + """Same dimension, different model: vectors from the two are not comparable.""" + + def __init__(self, dim=DIM, name="model-b"): + super().__init__(dim) + self.model_name = name + + def encode(self, text): + return super().encode("b:" + text) + + def encode_batch(self, texts): + return [self.encode(t) for t in texts] + + +# ---- stale / reembed ---------------------------------------------------------------------------------------- + + +def test_changing_the_model_is_detected_and_repaired_by_reembed(harness): + h = harness() + c = h.tenant("alice") + create_graph(c) + seed(c) + c.post("/v1/graphs/kb/save", {}) + s, r, _ = c.get("/v1/graphs/kb/stale") + assert r["stale_count"] == 0 and r["stored_models"] == {f"mock-{DIM}": 3} + + h._encoder = ModelB() # operator swaps the embedding model + h.restart() + s, r, _ = c.get("/v1/graphs/kb/stale") + assert s == 200 and r["stale_count"] == 3 and r["encoder_model"] == "model-b" + assert len(r["stale"]) == 3 + # text queries would compare vectors from different spaces: refused, not silently wrong + s, r, _ = c.post("/v1/graphs/kb/search", {"query": "Bob", "threshold": -1.0}) + assert s == 409 and r["error"]["code"] == "model_mismatch" + # a vector query is the caller's responsibility and still works + assert c.post("/v1/graphs/kb/search", {"vector": [0.1] * DIM, "threshold": -1.0})[0] == 200 + + s, r, _ = c.post("/v1/graphs/kb/reembed", {}) + assert s == 200 and r["reembedded"] == 3 + assert c.get("/v1/graphs/kb/stale")[1]["stale_count"] == 0 + s, r, _ = c.post("/v1/graphs/kb/search", {"query": "Bob trains machine learning models", "top_k": 1}) + assert s == 200 and r["results"][0]["id"] == 2 + assert c.post("/v1/graphs/kb/reembed", {})[1]["reembedded"] == 0 + + +def test_changing_the_dimension_blocks_writes_until_reembed(harness): + h = harness() + c = h.tenant("alice") + create_graph(c) + seed(c) + h._encoder = ModelB(dim=32, name="model-32") + h.restart() + s, r, _ = c.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "person", "id": 9, "properties": {"name": "New"}}]}) + assert s == 409 and r["error"]["code"] == "dimension_mismatch" + assert c.get("/v1/graphs/kb")[1]["nodes"] == 3 # nothing half-applied + s, r, _ = c.post("/v1/graphs/kb/search", {"query": "x", "threshold": -1.0}) + assert s == 409 and r["error"]["code"] in ("model_mismatch", "dimension_mismatch") + assert c.post("/v1/graphs/kb/reembed", {})[1]["reembedded"] == 3 + assert c.get("/v1/graphs/kb")[1]["dimension"] == 32 + assert c.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "person", "id": 9, "properties": {"name": "New"}}]})[0] == 200 + assert c.post("/v1/graphs/kb/search", {"query": "New", "top_k": 1, "threshold": -1.0})[0] == 200 + + +def test_client_mode_graphs_have_no_model_lifecycle(harness): + h = harness() + c = h.tenant("alice") + create_graph(c, "v", embedding={"mode": "client", "dimension": 3}) + assert c.get("/v1/graphs/v/stale")[0] == 400 + assert c.post("/v1/graphs/v/reembed", {})[0] == 400 + + +def test_server_without_an_encoder_still_serves_client_mode_graphs(harness): + h = harness(**{"encoder.kind": "none"}) + c = h.tenant("alice") + s, r, _ = c.post("/v1/graphs", {"name": "kb"}) + assert s == 400 and r["error"]["code"] == "encoder_unavailable" + create_graph(c, "v", embedding={"mode": "client", "dimension": 2}) + c.post("/v1/graphs/v/nodes", {"nodes": [{"type": "d", "id": 1, "vector": [1, 0]}, {"type": "d", "id": 2, "vector": [0, 1]}]}) + assert c.post("/v1/graphs/v/search", {"vector": [1, 0.1], "top_k": 1})[1]["results"][0]["id"] == 1 + h.restart() + assert c.post("/v1/graphs/v/search", {"vector": [1, 0.1], "top_k": 1})[1]["results"][0]["id"] == 1 + + +# ---- traversal options ----------------------------------------------------------------------------------------------- + + +def chain(c): + create_graph(c) + c.post("/v1/graphs/kb/nodes", {"nodes": [ + {"type": "t", "id": i, "properties": {"name": f"node {i}", "grp": "x" if i % 2 == 0 else "y"}} for i in range(5)]}) + c.post("/v1/graphs/kb/edges", {"edges": [ + {"rel": "NEXT", "source": {"type": "t", "id": i}, "target": {"type": "t", "id": i + 1}} for i in range(4)]}) + + +def test_direction_and_strict_filter(harness): + h = harness() + c = h.tenant("alice") + chain(c) + q = {"query": "node 4", "rel_type": "NEXT", "max_hops": 4, "threshold": 0.9, "top_k_seeds": 1} + out = c.post("/v1/graphs/kb/multi_hop", q)[1]["results"] + assert [x["id"] for x in out] == [4] # edges point away from node 4 + out = c.post("/v1/graphs/kb/multi_hop", {**q, "direction": "in", "top_k_results": 10})[1]["results"] + assert sorted(x["id"] for x in out) == [0, 1, 2, 3, 4] + flt = {"where": [{"field": "grp", "op": "eq", "value": "x"}]} + out = c.post("/v1/graphs/kb/multi_hop", {**q, "direction": "in", "filter": flt, "top_k_results": 10})[1]["results"] + assert sorted(x["id"] for x in out) == [0, 2, 4] # only passing nodes returned ... + assert any(1 in [p["id"] for p in x["path"]] for x in out) # ... but paths may run through others + out = c.post("/v1/graphs/kb/multi_hop", {**q, "direction": "in", "filter": flt, "strict_filter": True, + "top_k_results": 10})[1]["results"] + assert [x["id"] for x in out] == [4] # strict: failing nodes are walls + s, r, _ = c.post("/v1/graphs/kb/path", {"query": "node 4", "target": {"type": "t", "id": 0}, "direction": "in", + "threshold": 0.9}) + assert r["found"] and [p["id"] for p in r["result"]["path"]] == [4, 3, 2, 1, 0] + s, r, _ = c.post("/v1/graphs/kb/path", {"query": "node 4", "target": {"type": "t", "id": 0}, "direction": "in", + "threshold": 0.9, "filter": flt, "strict_filter": True}) + assert r["found"] is False + assert c.post("/v1/graphs/kb/path", {"query": "x", "target": {"type": "t", "id": 99}})[0] == 404 + + +def test_filter_semantics(harness): + h = harness() + c = h.tenant("alice") + create_graph(c) + c.post("/v1/graphs/kb/nodes", {"nodes": [ + {"type": "t", "id": 1, "properties": {"name": "a b", "n": 5, "s": "apple", "flag": True}}, + {"type": "t", "id": 2, "properties": {"name": "a b", "n": 7.5, "s": "banana", "flag": False}}, + {"type": "t", "id": 3, "properties": {"name": "a b"}}]}) + + def ids(where): + s, r, _ = c.post("/v1/graphs/kb/search", {"query": "a b", "top_k": 10, "threshold": -1.0, + "filter": {"where": where}}) + assert s == 200, r + return sorted(x["id"] for x in r["results"]) + + assert ids([{"field": "n", "op": "gt", "value": 5}]) == [2] + assert ids([{"field": "n", "op": "gte", "value": 5}]) == [1, 2] + assert ids([{"field": "n", "op": "lt", "value": "x"}]) == [] # a number never orders against a string + assert ids([{"field": "flag", "op": "eq", "value": 1}]) == [] # True is not 1 + assert ids([{"field": "flag", "op": "eq", "value": True}]) == [1] + assert ids([{"field": "s", "op": "prefix", "value": "ba"}]) == [2] + assert ids([{"field": "s", "op": "contains", "value": "pp"}]) == [1] + assert ids([{"field": "s", "op": "in", "value": ["apple", "kiwi"]}]) == [1] + assert ids([{"field": "s", "op": "nin", "value": ["apple"]}]) == [2, 3] + assert ids([{"field": "s", "op": "ne", "value": "apple"}]) == [2, 3] # missing counts as not-equal + assert ids([{"field": "s", "op": "exists"}]) == [1, 2] + assert ids([{"field": "s", "op": "exists", "value": False}]) == [3] + assert ids([{"field": "n", "op": "gte", "value": 5}, {"field": "s", "op": "prefix", "value": "a"}]) == [1] + for bad in ([{"field": "n", "op": "gt", "value": [1]}], [{"field": "n", "op": "in", "value": 3}], + [{"field": "n", "op": "regex", "value": "x"}], [{"field": "n", "op": "gt"}], + [{"field": "n", "op": "in", "value": list(range(65))}], [{"field": "bad name", "op": "eq", "value": 1}], + [{"field": "s", "op": "prefix", "value": 3}]): + s, r, _ = c.post("/v1/graphs/kb/search", {"query": "a", "filter": {"where": bad}}) + assert s == 400, bad + # a filter cannot execute code: it is data + s, r, _ = c.post("/v1/graphs/kb/search", {"query": "a", "filter": {"where": [ + {"field": "s", "op": "eq", "value": "__import__('os').system('echo pwned')"}]}}) + assert s == 200 and r["results"] == [] + + +def test_undirected_graphs_are_refused_with_an_explanation(harness): + h = harness() + c = h.tenant("alice") + s, r, _ = c.post("/v1/graphs", {"name": "kb", "directed": False}) + assert s == 400 and r["error"]["code"] == "unsupported" and "direction" in r["error"]["message"] + + +def test_ann_is_opt_in_and_matches_exact_search(harness): + pytest.importorskip("numpy") + h = harness() + c = h.tenant("alice") + g = create_graph(c, "fast", ann="ivf", ann_nprobe=4) + assert g["ann"] == "ivf" + assert create_graph(c, "plain")["ann"] is None + for name in ("fast", "plain"): + c.post(f"/v1/graphs/{name}/nodes", {"nodes": [{"type": "d", "id": i, "properties": {"name": f"doc {i}"}} + for i in range(60)]}) + q = {"query": "doc 17", "top_k": 5, "threshold": -1.0} + assert c.post("/v1/graphs/fast/search", q)[1]["results"] == c.post("/v1/graphs/plain/search", q)[1]["results"] + assert c.post("/v1/graphs/fast/search", {**q, "exact": True})[0] == 200 + h.restart() # the setting persists; the index is rebuilt lazily + assert c.get("/v1/graphs/fast")[1]["ann"] == "ivf" + assert c.post("/v1/graphs/fast/search", q)[0] == 200 + + +def test_empty_graph_and_unembedded_nodes_are_handled(harness): + h = harness() + c = h.tenant("alice") + create_graph(c) + assert c.post("/v1/graphs/kb/search", {"query": "anything"})[1] == {"results": [], "count": 0} + assert c.post("/v1/graphs/kb/search", {"vector": [0.5] * 7})[0] == 200 # nothing pinned yet + c.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "t", "id": 1, "properties": {"color": "red"}}]}) # no embed field + s, r, _ = c.get("/v1/graphs/kb/nodes/t/1") + assert r["embedded"] is False + s, r, _ = c.post("/v1/graphs/kb/similar", {"node": {"type": "t", "id": 1}}) + assert s == 200 and r["node_embedded"] is False and r["results"] == [] + # PUT with an embedding, then PUT again with nothing to embed drops the stale vector + c.put("/v1/graphs/kb/nodes/t/2", {"properties": {"name": "has text"}}) + assert c.get("/v1/graphs/kb/nodes/t/2")[1]["embedded"] is True + c.put("/v1/graphs/kb/nodes/t/2", {"properties": {"color": "blue"}}) + assert c.get("/v1/graphs/kb/nodes/t/2")[1]["embedded"] is False + assert c.get("/v1/graphs/kb")[1]["embeddings"] == 0 + # explicit embed_text overrides the fields + c.put("/v1/graphs/kb/nodes/t/3", {"properties": {"name": "ignored"}, "embed_text": "custom text"}) + s, r, _ = c.post("/v1/graphs/kb/search", {"query": "custom text", "top_k": 1}) + assert r["results"][0]["id"] == 3 and r["results"][0]["score"] == pytest.approx(1.0, abs=1e-5) + + +# ---- CLI -------------------------------------------------------------------------------------------------------------- + + +def test_cli_tenant_and_key_management(tmp_path, capsys): + d = str(tmp_path / "data") + assert main(["tenant", "create", "acme", "--data-dir", d, "--quota", "max_nodes=50", "--quota", "rate_burst=5"]) == 0 + assert main(["tenant", "create", "acme", "--data-dir", d]) == 2 # already exists + assert main(["tenant", "create", "Bad_Name", "--data-dir", d]) == 2 + assert main(["tenant", "create", "x", "--data-dir", d]) == 2 # too short + assert main(["tenant", "create", "q", "--data-dir", d, "--quota", "bogus=1"]) == 2 + capsys.readouterr() + assert main(["key", "create", "acme", "--data-dir", d, "--role", "ro", "--label", "ci"]) == 0 + captured = capsys.readouterr() + token = captured.out.strip() + assert token.startswith("igv_acme.") and "shown once" in captured.err + assert token not in json.dumps(auth.describe_tenant(d, "acme")) + reg = auth.TenantRegistry(d, load_config().quotas, 0) + p = reg.authenticate(token) + assert p.tenant_id == "acme" and p.role == "ro" and p.quotas.max_nodes == 50 and p.quotas.rate_burst == 5 + key_id = token.split(".")[1] + assert main(["tenant", "list", "--data-dir", d]) == 0 + assert "acme\tactive\t1 key(s)" in capsys.readouterr().out + assert main(["key", "revoke", "acme", key_id, "--data-dir", d]) == 0 + assert reg.authenticate(token) is None + assert main(["key", "revoke", "acme", "0" * 16, "--data-dir", d]) == 2 + assert main(["tenant", "disable", "acme", "--data-dir", d]) == 0 + token2 = auth.add_key(d, "acme") + assert reg.authenticate(token2) is None + assert main(["tenant", "enable", "acme", "--data-dir", d]) == 0 + assert reg.authenticate(token2) is not None + assert main(["tenant", "set-quota", "acme", "--data-dir", d, "--quota", "max_nodes=7"]) == 0 + assert reg.authenticate(token2).quotas.max_nodes == 7 + assert main(["tenant", "show", "acme", "--data-dir", d]) == 0 + assert "hash" not in capsys.readouterr().out + + +def test_key_files_are_private_on_posix(tmp_path): + if os.name == "nt": + pytest.skip("POSIX permission bits") + d = str(tmp_path / "data") + auth.create_tenant(d, "acme") + mode = (tmp_path / "data" / "tenants" / "acme" / "tenant.json").stat().st_mode & 0o777 + assert mode == 0o600 + + +# ---- config and logs ------------------------------------------------------------------------------------------------------ + + +def test_config_file_env_overrides_and_validation(tmp_path): + f = tmp_path / "c.json" + f.write_text(json.dumps({"port": 9000, "data_dir": "/x", "quotas": {"max_nodes": 5}, + "encoder": {"kind": "mock", "dimension": 8}})) + cfg = load_config(str(f), env={"IGV_PORT": "9100", "IGV_METRICS_TOKEN": "m" * 30}) + assert (cfg.port, cfg.data_dir, cfg.quotas.max_nodes, cfg.encoder.dimension) == (9100, "/x", 5, 8) + assert cfg.metrics_token == "m" * 30 and "m" * 30 not in repr(cfg) # secrets never in repr + for bad in ({"metrics_token": "x" * 40}, {"replication_token": "x" * 40}, {"nope": 1}, {"quotas": {"nope": 1}}, + {"quotas": {"max_nodes": -1}}, {"role": "leader"}, {"durability": "maybe"}, + {"encoder": {"kind": "mock"}}, {"encoder": {"kind": "http", "url": "http://x"}}, + {"role": "replica"}, {"keep_generations": 1}, {"port": 99999}, {"max_workers": 0}, + {"router": {"shards": [{"name": "a"}]}}): + f.write_text(json.dumps(bad)) + with pytest.raises(ConfigError): + load_config(str(f), env={}) + with pytest.raises(ConfigError): + load_config(None, env={"IGV_METRICS_TOKEN": "short"}) + with pytest.raises(ConfigError): + load_config(None, env={"IGV_PORT": "abc"}) + + +def test_logs_are_json_lines_with_route_templates_and_no_payloads(harness): + stream = io.StringIO() + setup_logging("INFO", stream) + h = harness() + c = h.tenant("alice") + create_graph(c, "kb") + c.post("/v1/graphs/kb/search", {"query": "my very private query text"}) + c.get("/v1/graphs/kb/nodes/doc/some-private-id") + lines = [json.loads(line) for line in stream.getvalue().splitlines() if line.strip()] + reqs = [rec for rec in lines if rec["msg"] == "request"] + assert reqs and all({"ts", "level", "route", "status", "ms", "request_id"} <= set(r) for r in reqs) + assert {r["route"] for r in reqs} >= {"graphs.create", "search", "node.get"} + text = stream.getvalue() + assert "my very private query text" not in text and "some-private-id" not in text + assert all(r["tenant"] == "alice" for r in reqs) + + +def test_internal_errors_become_generic_500s_with_a_request_id(harness, monkeypatch): + h = harness() + c = h.tenant("alice") + create_graph(c) + from isongraph_vector_server import ops + + def boom(*a, **k): + raise RuntimeError("secret internal detail /srv/x.py") + + monkeypatch.setattr(ops, "list_nodes", boom) + s, r, hd = c.get("/v1/graphs/kb/nodes") + assert s == 500 and r["error"]["code"] == "internal_error" + assert "secret" not in json.dumps(r) and "/srv" not in json.dumps(r) and "RuntimeError" not in json.dumps(r) + assert r["error"]["request_id"] == hd["X-Request-Id"] + monkeypatch.undo() + assert c.get("/v1/graphs/kb/nodes")[0] == 200 + + +def test_the_shipped_example_config_is_valid(): + from pathlib import Path + cfg = load_config(str(Path(__file__).resolve().parent.parent / "config.example.json"), env={}) + assert cfg.encoder.kind == "http" and cfg.quotas.max_nodes == 100000 and cfg.role == "primary" diff --git a/isongraph-vector-server/tests/test_limits.py b/isongraph-vector-server/tests/test_limits.py new file mode 100644 index 0000000..f7f4798 --- /dev/null +++ b/isongraph-vector-server/tests/test_limits.py @@ -0,0 +1,532 @@ +"""Quotas, protocol limits, bounded concurrency, deadlines, and the locks underneath them.""" + +import json +import socket +import threading +import time + +import pytest + +from conftest import DIM, Client, create_graph, seed +from isongraph_vector import EmbeddingError, MockEncoder +from isongraph_vector_server.limits import LockTimeout, RWLock, SlidingWindow, TokenBucket + + +class SlowEncoder(MockEncoder): + def __init__(self, delay, dim=DIM): + super().__init__(dim) + self.delay = delay + + def encode(self, text): + time.sleep(self.delay) + return super().encode(text) + + def encode_batch(self, texts): + time.sleep(self.delay) + return [MockEncoder.encode(self, t) for t in texts] + + +class BrokenEncoder(MockEncoder): + def encode(self, text): + raise EmbeddingError("upstream https://secret.internal/embed returned 500: token=hunter2") + + def encode_batch(self, texts): + raise EmbeddingError("upstream https://secret.internal/embed returned 500: token=hunter2") + + +# ---- quotas --------------------------------------------------------------------------------------------------- + + +def nodes(start, n): + return {"nodes": [{"type": "n", "id": i, "properties": {"name": f"n{i}"}} for i in range(start, start + n)]} + + +def test_node_quota_is_exact_atomic_and_freed_by_deletes(harness): + h = harness() + c = h.tenant("alice", quotas={"max_nodes": 10}) + create_graph(c, "a") + create_graph(c, "b") + assert c.post("/v1/graphs/a/nodes", nodes(0, 6))[0] == 200 + s, r, _ = c.post("/v1/graphs/b/nodes", nodes(0, 5)) # 6 + 5 > 10: whole batch refused + assert s == 403 and r["error"]["code"] == "quota_exceeded" + assert c.get("/v1/graphs/b")[1]["nodes"] == 0 + assert c.post("/v1/graphs/b/nodes", nodes(0, 4))[0] == 200 # exactly at the limit + assert c.post("/v1/graphs/b/nodes", nodes(100, 1))[0] == 403 + # updating existing nodes costs nothing + assert c.post("/v1/graphs/a/nodes", nodes(0, 6))[0] == 200 + assert c.get("/v1/usage")[1]["usage"]["nodes"] == 10 + assert c.post("/v1/graphs/a/nodes/delete", {"nodes": [{"type": "n", "id": 0}, {"type": "n", "id": 1}]})[0] == 200 + assert c.get("/v1/usage")[1]["usage"]["nodes"] == 8 + assert c.post("/v1/graphs/b/nodes", nodes(200, 2))[0] == 200 + # deleting a whole graph frees its share + assert c.delete("/v1/graphs/a")[0] == 200 + assert c.get("/v1/usage")[1]["usage"]["nodes"] == 6 + + +def test_concurrent_writers_into_different_graphs_cannot_exceed_the_node_quota(harness): + h = harness() + c = h.tenant("alice", quotas={"max_nodes": 100}) + for g in ("a", "b", "c", "d"): + create_graph(c, g) + results = [] + + def writer(graph, base): + for k in range(12): + s, _, _ = c.post(f"/v1/graphs/{graph}/nodes", nodes(base + k * 5, 5)) + results.append(s) + + threads = [threading.Thread(target=writer, args=(g, i * 1000)) for i, g in enumerate("abcd")] + [t.start() for t in threads] + [t.join() for t in threads] + assert set(results) <= {200, 403} + total = sum(c.get(f"/v1/graphs/{g}")[1]["nodes"] for g in "abcd") + assert total <= 100 and total == c.get("/v1/usage")[1]["usage"]["nodes"] + assert total == 100 and results.count(200) == 20 # all 20 batches that fit were admitted + + +def test_edge_graph_dimension_and_batch_quotas(harness): + h = harness() + c = h.tenant("alice", quotas={"max_edges": 3, "max_graphs": 2, "max_dimension": 8, "max_batch_items": 5}) + s, r, _ = c.post("/v1/graphs", {"name": "kb"}) # encoder dimension 16 > 8 + assert s == 403 and "dimension" in r["error"]["message"] + create_graph(c, "vec", embedding={"mode": "client", "dimension": 8}) + assert c.post("/v1/graphs", {"name": "big", "embedding": {"mode": "client", "dimension": 9}})[0] == 403 + create_graph(c, "vec2", embedding={"mode": "client", "dimension": 4}) + assert c.post("/v1/graphs", {"name": "third", "embedding": {"mode": "client", "dimension": 4}})[0] == 403 # graphs + assert c.post("/v1/graphs/vec/nodes", {"nodes": [{"type": "n", "id": i} for i in range(6)]})[1]["error"]["code"] == "batch_too_large" + assert c.post("/v1/graphs/vec/nodes", {"nodes": [{"type": "n", "id": i} for i in range(5)]})[0] == 200 + assert c.post("/v1/graphs/vec/nodes", {"nodes": [{"type": "n", "id": 9, "vector": [0.1] * 9}]})[0] == 400 + ring = [{"rel": "R", "source": {"type": "n", "id": i}, "target": {"type": "n", "id": (i + 1) % 5}} for i in range(5)] + assert c.post("/v1/graphs/vec/edges", {"edges": ring[:3]})[0] == 200 + s, r, _ = c.post("/v1/graphs/vec/edges", {"edges": ring[3:]}) + assert s == 403 and r["error"]["code"] == "quota_exceeded" + assert c.get("/v1/usage")[1]["usage"]["edges"] == 3 + + +def test_query_parameter_caps_follow_the_tenants_quota(harness): + h = harness() + c = h.tenant("alice", quotas={"max_top_k": 5, "max_hops": 2}) + create_graph(c) + seed(c) + assert c.post("/v1/graphs/kb/search", {"query": "x", "top_k": 6})[0] == 400 + assert c.post("/v1/graphs/kb/search", {"query": "x", "top_k": 5})[0] == 200 + assert c.post("/v1/graphs/kb/multi_hop", {"query": "x", "max_hops": 3})[0] == 400 + assert c.post("/v1/graphs/kb/multi_hop", {"query": "x", "top_k_seeds": 6})[0] == 400 + assert c.post("/v1/graphs/kb/search", {"query": "x", "top_k": 10 ** 9})[0] == 400 + + +def test_oversized_vectors_and_property_counts_are_refused(harness): + h = harness() + c = h.tenant("alice", quotas={"max_properties": 3, "max_dimension": 32}) + create_graph(c) + s, r, _ = c.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "n", "id": 1, "properties": {f"p{i}": i for i in range(4)}}]}) + assert s == 400 + assert c.post("/v1/graphs/kb/search", {"vector": [0.1] * 33})[0] == 400 + assert c.post("/v1/graphs/kb/search", {"vector": [float("nan")]})[0] == 400 + + +def test_rejected_batches_leave_no_trace(harness): + h = harness() + c = h.tenant("alice") + create_graph(c) + seed(c) + c.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "person", "id": 7, "properties": {"name": "seven"}}]}) + before = c.get("/v1/graphs/kb")[1] + bad = {"nodes": [{"type": "person", "id": 50, "properties": {"name": "ok"}}, + {"type": "person", "id": "007"}]} # "007" collides with the existing 7: whole batch refused + s, r, _ = c.post("/v1/graphs/kb/nodes", bad) + assert s == 409 and c.get("/v1/graphs/kb/nodes/person/50")[0] == 404 + twins = {"nodes": [{"type": "person", "id": 60}, {"type": "person", "id": "60"}]} # same node twice, spelled two ways + assert c.post("/v1/graphs/kb/nodes", twins)[0] == 400 + after = c.get("/v1/graphs/kb")[1] + assert (after["nodes"], after["embeddings"]) == (before["nodes"], before["embeddings"]) + assert c.get("/v1/usage")[1]["usage"]["nodes"] == 4 + + +# ---- protocol limits --------------------------------------------------------------------------------------------- + + +def raw_exchange(node, data: bytes, read=True, timeout=5): + host, port = node.server.address + sock = socket.create_connection((host, port), timeout=timeout) + try: + sock.sendall(data) + if not read: + return sock + out = b"" + while True: + try: + chunk = sock.recv(65536) + except socket.timeout: + break + if not chunk: + break + out += chunk + return out + finally: + if read: + sock.close() + + +def status_of(raw: bytes) -> int: + return int(raw.split(b" ", 2)[1]) + + +def test_body_size_limit_and_framing_rules(harness): + h = harness(max_body_bytes=2000) + c = h.tenant("alice") + create_graph(c) + big = json.dumps({"query": "x" * 5000}).encode() + s, r, _ = c.post("/v1/graphs/kb/search", None, raw=big) + assert s == 413 and r["error"]["code"] == "payload_too_large" + auth = f"Authorization: Bearer {c.key}\r\n" + # chunked bodies are refused: one unambiguous framing + chunked = (f"POST /v1/graphs/kb/search HTTP/1.1\r\nHost: x\r\n{auth}Content-Type: application/json\r\n" + "Transfer-Encoding: chunked\r\n\r\n5\r\nhello\r\n0\r\n\r\n").encode() + assert status_of(raw_exchange(h, chunked)) == 400 + # POST without Content-Length + nolen = f"POST /v1/graphs/kb/search HTTP/1.1\r\nHost: x\r\n{auth}Content-Type: application/json\r\n\r\n".encode() + assert status_of(raw_exchange(h, nolen)) == 411 + # conflicting / malformed lengths (request-smuggling shapes) + two = (f"POST /v1/graphs/kb/search HTTP/1.1\r\nHost: x\r\n{auth}Content-Type: application/json\r\n" + "Content-Length: 2\r\nContent-Length: 40\r\n\r\n{}").encode() + assert status_of(raw_exchange(h, two)) == 400 + neg = (f"POST /v1/graphs/kb/search HTTP/1.1\r\nHost: x\r\n{auth}Content-Type: application/json\r\n" + "Content-Length: -5\r\n\r\n").encode() + assert status_of(raw_exchange(h, neg)) == 400 + both = (f"POST /v1/graphs/kb/search HTTP/1.1\r\nHost: x\r\n{auth}Content-Type: application/json\r\n" + "Content-Length: 2\r\nTransfer-Encoding: chunked\r\n\r\n{}").encode() + assert status_of(raw_exchange(h, both)) == 400 + # absurd request line / header block + assert status_of(raw_exchange(h, b"GET /" + b"a" * 70000 + b" HTTP/1.1\r\nHost: x\r\n\r\n")) in (400, 414) + hdr = f"GET /v1/graphs HTTP/1.1\r\nHost: x\r\n{auth}X-Big: {'a' * 60000}\r\n\r\n".encode() + assert status_of(raw_exchange(h, hdr)) in (400, 431) + # malformed request line / unknown method + assert status_of(raw_exchange(h, b"GARBAGE\r\n\r\n")) == 400 + assert status_of(raw_exchange(h, f"TRACE /v1/graphs HTTP/1.1\r\nHost: x\r\n{auth}\r\n".encode())) in (405, 501) + # the server is still fine afterwards + assert c.get("/v1/graphs")[0] == 200 + + +def test_slow_clients_cannot_hold_a_worker_forever(harness): + h = harness(socket_timeout_seconds=0.4, max_workers=2, accept_queue=2) + c = h.tenant("alice") + stalled = [raw_exchange(h, b"GET /v1/gra", read=False) for _ in range(2)] # half a request line, then silence + t0 = time.time() + time.sleep(0.9) + for s in stalled: + s.settimeout(2) + assert s.recv(10) == b"" # the server hung up on them + s.close() + assert c.get("/v1/graphs")[0] == 200 + # a slow-drip body is cut off too (headers promise 100 bytes, client sends 3 and stops) + auth = f"Authorization: Bearer {c.key}\r\n" + sock = raw_exchange(h, (f"POST /v1/graphs HTTP/1.1\r\nHost: x\r\n{auth}Content-Type: application/json\r\n" + "Content-Length: 100\r\n\r\n{\"n").encode(), read=False) + sock.settimeout(3) + data = b"" + try: + data = sock.recv(100) + except socket.timeout: + pass + sock.close() + assert data == b"" # closed without a response, worker freed + assert c.get("/v1/graphs")[0] == 200 + + +def test_connections_beyond_the_pool_get_an_immediate_503(harness): + h = harness(max_workers=2, accept_queue=1, socket_timeout_seconds=5) + c = h.tenant("alice") + holders = [raw_exchange(h, b"", read=False) for _ in range(3)] # 2 workers busy + 1 queued + time.sleep(0.3) + t0 = time.time() + out = raw_exchange(h, f"GET /v1/graphs HTTP/1.1\r\nHost: x\r\nAuthorization: Bearer {c.key}\r\n\r\n".encode(), + timeout=3) + assert status_of(out) == 503 and b"overloaded" in out and time.time() - t0 < 2 + for s in holders: + s.close() + time.sleep(0.3) + assert c.get("/v1/graphs")[0] == 200 + + +# ---- concurrency gates, deadlines, lock timeouts ----------------------------------------------------------------- + + +def test_tenant_inflight_cap_sheds_load_without_affecting_other_tenants(harness): + h = harness(encoder=SlowEncoder(0.6), queue_wait_seconds=0.05) + a = h.tenant("alice", quotas={"max_inflight": 1}) + b = h.tenant("bob") + create_graph(a) + create_graph(b) + out = {} + + def slow(): + out["slow"] = a.post("/v1/graphs/kb/search", {"query": "slow one"})[0] + + t = threading.Thread(target=slow) + t.start() + time.sleep(0.2) + s, r, hd = a.get("/v1/usage") + assert s == 503 and r["error"]["code"] == "tenant_overloaded" and hd["Retry-After"] == "1" + assert b.get("/v1/usage")[0] == 200 # another tenant is not shed + t.join() + assert out["slow"] == 200 + assert a.get("/v1/usage")[0] == 200 + + +def test_global_concurrency_cap_sheds_load(harness): + h = harness(encoder=SlowEncoder(0.6), queue_wait_seconds=0.05, max_concurrent_requests=1) + a, b = h.tenant("alice"), h.tenant("bob") + create_graph(a) + t = threading.Thread(target=lambda: a.post("/v1/graphs/kb/search", {"query": "slow"})) + t.start() + time.sleep(0.2) + s, r, _ = b.get("/v1/usage") + assert s == 503 and r["error"]["code"] == "overloaded" + t.join() + assert b.get("/v1/usage")[0] == 200 + + +def test_request_deadline_is_enforced_between_phases(harness): + h = harness(encoder=SlowEncoder(0.5), request_deadline_seconds=0.2) + c = h.tenant("alice") + s, r, _ = c.post("/v1/graphs", {"name": "kb"}) + assert s == 201 + s, r, _ = c.post("/v1/graphs/kb/search", {"query": "takes too long"}) + assert s == 504 and r["error"]["code"] == "deadline_exceeded" + s, r, _ = c.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "n", "id": 1, "properties": {"name": "x"}}]}) + assert s == 504 + assert c.get("/v1/graphs/kb")[1]["nodes"] == 0 # the late write did not happen + + +def test_a_busy_graph_answers_503_instead_of_queueing_forever(harness): + h = harness(lock_timeout_seconds=0.3) + c = h.tenant("alice") + create_graph(c) + seed(c) + handle = h.server.engine.tenants["alice"].graphs["kb"] + handle.lock.acquire_write(5) # someone is mid-write (or mid-reembed) + try: + t0 = time.time() + s, r, hd = c.post("/v1/graphs/kb/search", {"query": "x"}) + assert s == 503 and r["error"]["code"] == "busy" and hd["Retry-After"] == "1" + assert time.time() - t0 < 2 + assert c.post("/v1/graphs/kb/nodes", nodes(900, 1))[0] == 503 + assert c.get("/v1/graphs/kb/nodes/person/1")[0] == 503 + finally: + handle.lock.release_write() + assert c.post("/v1/graphs/kb/search", {"query": "x"})[0] == 200 + handle.lock.acquire_read(5) # a long reader blocks writers, not readers + try: + assert c.post("/v1/graphs/kb/search", {"query": "x"})[0] == 200 + assert c.post("/v1/graphs/kb/nodes", nodes(900, 1))[0] == 503 + finally: + handle.lock.release_read() + + +def test_encoder_failures_are_generic_to_the_client(harness): + h = harness(encoder=BrokenEncoder(DIM)) + c = h.tenant("alice") + create_graph(c) + for method, path, body in [("POST", "/v1/graphs/kb/search", {"query": "x"}), + ("POST", "/v1/graphs/kb/nodes", {"nodes": [{"type": "n", "id": 1, "properties": {"name": "x"}}]})]: + s, r, _ = c.request(method, path, body) + assert s == 503 and r["error"]["code"] == "encoder_failed" + assert "secret.internal" not in json.dumps(r) and "hunter2" not in json.dumps(r) + assert c.get("/v1/graphs/kb")[1]["nodes"] == 0 + + +def test_response_size_cap(harness): + h = harness(max_response_bytes=2000) + c = h.tenant("alice") + create_graph(c) + c.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "n", "id": i, "properties": {"name": "x" * 200}} for i in range(50)]}) + s, r, _ = c.get("/v1/graphs/kb/nodes?limit=100") + assert s == 413 and r["error"]["code"] == "response_too_large" + assert c.get("/v1/graphs/kb/nodes?limit=3")[0] == 200 + + +def test_mixed_concurrent_load_keeps_graph_store_and_quota_consistent(harness): + h = harness() + c = h.tenant("alice", quotas={"max_nodes": 100000}) + create_graph(c) + errors = [] + stop = time.time() + 3 + + def worker(wid): + n = 0 + while time.time() < stop: + try: + i = wid * 100000 + n + n += 1 + for s in ( + c.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "n", "id": i, "properties": {"name": f"w{wid} {i}"}}]})[0], + c.post("/v1/graphs/kb/search", {"query": f"w{wid}", "top_k": 5, "threshold": -1.0})[0], + c.patch(f"/v1/graphs/kb/nodes/n/{i}", {"properties": {"name": f"w{wid} patched {i}"}})[0], + c.post("/v1/graphs/kb/edges", {"edges": [{"rel": "R", "source": {"type": "n", "id": i}, + "target": {"type": "n", "id": i}}]})[0], + c.post("/v1/graphs/kb/multi_hop", {"query": f"w{wid}", "threshold": -1.0})[0], + ): + if s != 200: + errors.append(s) + if n % 3 == 0: + if c.delete(f"/v1/graphs/kb/nodes/n/{i}")[0] != 200: + errors.append("delete") + except Exception as exc: # noqa: BLE001 + errors.append(repr(exc)) + + def saver(): + while time.time() < stop: + if c.post("/v1/graphs/kb/save", {})[0] != 200: + errors.append("save") + time.sleep(0.2) + + threads = [threading.Thread(target=worker, args=(w,)) for w in range(6)] + [threading.Thread(target=saver)] + [t.start() for t in threads] + [t.join() for t in threads] + assert not errors, errors[:5] + g = c.get("/v1/graphs/kb")[1] + usage = c.get("/v1/usage")[1]["usage"] + assert g["nodes"] == g["embeddings"] == usage["nodes"] and g["edges"] == usage["edges"] + assert g["nodes"] > 50 + + +# ---- the building blocks --------------------------------------------------------------------------------------------- + + +def test_token_bucket_with_a_fake_clock(): + now = [0.0] + b = TokenBucket(rate=2, burst=3, clock=lambda: now[0]) + assert [b.take() for _ in range(3)] == [0.0, 0.0, 0.0] + wait = b.take() + assert wait == pytest.approx(0.5) + now[0] += 0.5 + assert b.take() == 0.0 and b.take() > 0 + now[0] += 100 + assert [b.take() for _ in range(3)] == [0.0, 0.0, 0.0] and b.take() > 0 # capped at burst + assert TokenBucket(1, 2, clock=lambda: 0.0).take(cost=50) == 0.0 # oversized cost is clamped to the burst + + +def test_sliding_window(): + now = [0.0] + w = SlidingWindow(limit=3, window=10, clock=lambda: now[0]) + for _ in range(3): + assert not w.blocked("ip") + w.record("ip") + assert w.blocked("ip") and not w.blocked("other") + now[0] = 11 + assert not w.blocked("ip") + + +def test_rwlock_semantics(): + lock = RWLock() + lock.acquire_read() + lock.acquire_read() # readers share + with pytest.raises(LockTimeout): + lock.acquire_write(0.05) # a writer waits for them + lock.release_read() + lock.release_read() + lock.acquire_write(0.05) + with pytest.raises(LockTimeout): + lock.acquire_read(0.05) + lock.release_write() + # writer preference: a waiting writer stops new readers (no writer starvation) + lock.acquire_read() + got = [] + w = threading.Thread(target=lambda: (lock.acquire_write(2), got.append("w"), lock.release_write())) + w.start() + time.sleep(0.1) + with pytest.raises(LockTimeout): + lock.acquire_read(0.1) # queued behind the waiting writer + lock.release_read() + w.join() + assert got == ["w"] + lock.acquire_read(0.1) + lock.release_read() + + +def test_rwlock_under_contention_never_overlaps_writers(): + lock = RWLock() + state = {"w": 0, "r": 0, "bad": 0} + guard = threading.Lock() + + def reader(): + for _ in range(300): + with lock.read(5): + with guard: + state["r"] += 1 + if state["w"]: + state["bad"] += 1 + with guard: + state["r"] -= 1 + + def writer(): + for _ in range(100): + with lock.write(5): + with guard: + state["w"] += 1 + if state["w"] > 1 or state["r"]: + state["bad"] += 1 + time.sleep(0.0005) + with guard: + state["w"] -= 1 + + ts = [threading.Thread(target=reader) for _ in range(6)] + [threading.Thread(target=writer) for _ in range(3)] + [t.start() for t in ts] + [t.join() for t in ts] + assert state["bad"] == 0 + + +def test_graceful_stop_is_prompt_even_with_idle_keepalive_connections(harness): + """Regression: stop() used to wait out the whole socket timeout (10 s) for every idle + keep-alive connection, because a thread blocked in recv() is not woken on all platforms.""" + import http.client + h = harness() + c = h.tenant("alice") + conns = [] + for _ in range(3): + conn = http.client.HTTPConnection(*h.server.address, timeout=5) + conn.request("GET", "/healthz") + conn.getresponse().read() + conns.append(conn) # left open and idle + t0 = time.time() + h.server.stop() + assert time.time() - t0 < 3 + h.restart() # fixture teardown wants a live server + + +def test_pipelined_requests_on_one_connection_are_both_answered(harness): + h = harness() + c = h.tenant("alice") + req = f"GET /v1/usage HTTP/1.1\r\nHost: x\r\nAuthorization: Bearer {c.key}\r\n\r\n".encode() + out = raw_exchange(h, req * 2 + b"GET /healthz HTTP/1.1\r\nHost: x\r\nConnection: close\r\n\r\n", timeout=3) + assert out.count(b"HTTP/1.1 200") == 3 + + +def test_a_client_dripping_bytes_is_cut_off_by_the_total_deadline(harness, monkeypatch): + from isongraph_vector_server import httpd + monkeypatch.setattr(httpd, "HEADER_TOTAL_SECONDS", 1.0) + h = harness(socket_timeout_seconds=5) # each byte arrives well inside the per-recv timeout + host, port = h.server.address + sock = socket.create_connection((host, port), timeout=5) + t0 = time.time() + closed = False + try: + for ch in b"GET /healthz HTTP/1.1\r\nHost: slow\r\n": + sock.sendall(bytes([ch])) + time.sleep(0.15) + sock.settimeout(0.01) + try: + if sock.recv(10) == b"": + closed = True + break + except socket.timeout: + pass + except OSError: + closed = True + break + except OSError: + closed = True + finally: + sock.close() + assert closed and time.time() - t0 < 4 diff --git a/isongraph-vector-server/tests/test_persistence.py b/isongraph-vector-server/tests/test_persistence.py new file mode 100644 index 0000000..ab29e94 --- /dev/null +++ b/isongraph-vector-server/tests/test_persistence.py @@ -0,0 +1,415 @@ +"""Snapshots, generations, verification on load, shutdown behaviour.""" + +import json +import os +import random +import shutil +import subprocess +import sys +import time +from pathlib import Path + +import pytest + +from conftest import DIM, Client, create_graph, seed +from isongraph_vector import SemanticGraph, graph_digest +from ison_graph import ISONGraph +from isongraph_vector_server import storage +from isongraph_vector_server.storage import GraphStorage, SnapshotError, semantic_digest + + +def gdir(h, tenant="alice", graph="kb"): + return Path(h.data_dir) / "tenants" / tenant / "graphs" / f"g-{graph}" + + +def searchable(c, text="Bob trains machine learning models"): + s, r, _ = c.post("/v1/graphs/kb/search", {"query": text, "top_k": 1}) + assert s == 200, r + return r["results"][0] + + +def test_state_survives_a_graceful_restart(harness): + h = harness() + c = h.tenant("alice") + create_graph(c) + seed(c) + h.restart() # graceful: final snapshot is written + g = c.get("/v1/graphs/kb")[1] + assert (g["nodes"], g["edges"], g["embeddings"]) == (3, 2, 3) + top = searchable(c) + assert top["id"] == 2 and top["score"] == pytest.approx(1.0, abs=1e-5) + assert c.get("/v1/graphs/kb/nodes/person/carol")[1]["properties"]["age"] == 38 + + +def test_unsaved_changes_are_lost_on_a_hard_stop_and_saved_ones_are_not(harness): + h = harness() + c = h.tenant("alice") + create_graph(c) + seed(c) + assert c.post("/v1/graphs/kb/save", {})[1]["saved"] + c.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "person", "id": 99, "properties": {"name": "late"}}]}) + assert c.get("/v1/graphs/kb")[1]["unsaved_changes"] is True + h.restart(graceful=False) + assert c.get("/v1/graphs/kb")[1]["nodes"] == 3 # documented loss window + assert c.get("/v1/graphs/kb/nodes/person/99")[0] == 404 + + +def test_sync_durability_survives_a_hard_stop(harness): + h = harness(durability="sync") + c = h.tenant("alice") + create_graph(c) + seed(c) + c.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "person", "id": 99, "properties": {"name": "durable"}}]}) + c.patch("/v1/graphs/kb/nodes/person/99", {"properties": {"name": "durable2"}}) + c.post("/v1/graphs/kb/nodes/delete", {"nodes": [{"type": "person", "id": 1}]}) + h.restart(graceful=False) + assert c.get("/v1/graphs/kb/nodes/person/99")[1]["properties"]["name"] == "durable2" + assert c.get("/v1/graphs/kb/nodes/person/1")[0] == 404 + assert c.get("/v1/graphs/kb")[1]["nodes"] == 3 + + +def test_autosave_writes_dirty_graphs(harness): + h = harness(autosave_interval_seconds=0.2) + c = h.tenant("alice") + create_graph(c) + seed(c) + deadline = time.time() + 5 + while time.time() < deadline and c.get("/v1/graphs/kb")[1]["unsaved_changes"]: + time.sleep(0.1) + g = c.get("/v1/graphs/kb")[1] + assert not g["unsaved_changes"] and g["generation"] >= 1 + h.restart(graceful=False) + assert c.get("/v1/graphs/kb")[1]["nodes"] == 3 + + +def test_snapshot_layout_manifest_and_digest(harness): + h = harness() + c = h.tenant("alice") + create_graph(c) + seed(c) + c.post("/v1/graphs/kb/save", {}) + d = gdir(h) + assert (d / "CURRENT").read_text().strip() == "000001" + gen = d / "gen-000001" + assert sorted(p.name for p in gen.iterdir()) == ["MANIFEST.json", "graph.isong", "graph.isong.embeddings.json"] + manifest = json.loads((gen / "MANIFEST.json").read_text()) + sidecar = json.loads((gen / "graph.isong.embeddings.json").read_text()) + assert sidecar["graph_digest"] == manifest["graph_digest"] + assert manifest["node_count"] == 3 and manifest["embedding_count"] == 3 + # the library, handed only the file pair, agrees (this is what the digest is for) + from conftest import REPO_ROOT # noqa: F401 + lib = SemanticGraph.load(gen / "graph.isong", encoder=None, auto_embed=False) + assert graph_digest(lib) == manifest["graph_digest"] + assert c.get("/v1/graphs/kb")[1]["snapshot_digest"] == manifest["graph_digest"] + # unchanged graph: save is a no-op unless forced; the endpoint forces + assert c.post("/v1/graphs/kb/save", {})[1]["generation"] == 2 + + +def test_generations_are_pruned(harness): + h = harness(keep_generations=3) + c = h.tenant("alice") + create_graph(c) + for i in range(6): + c.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "n", "id": i, "properties": {"name": f"n{i}"}}]}) + c.post("/v1/graphs/kb/save", {}) + gens = sorted(p.name for p in gdir(h).iterdir() if p.name.startswith("gen-")) + assert gens == ["gen-000004", "gen-000005", "gen-000006"] + + +# ---- verification on load ------------------------------------------------------------------------------------ + + +def two_generations(h): + c = h.tenant("alice") + create_graph(c) + seed(c) + c.post("/v1/graphs/kb/save", {}) # gen 1: 3 nodes + c.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "person", "id": 4, "properties": {"name": "Dan"}}]}) + c.post("/v1/graphs/kb/save", {}) # gen 2: 4 nodes + return c + + +def test_corrupt_current_generation_falls_back_and_is_flagged(harness): + h = harness() + c = two_generations(h) + h.server.stop() + victim = gdir(h) / "gen-000002" / "graph.isong" + victim.write_bytes(victim.read_bytes().replace(b"Dan", b"Eve")) # same length, wrong checksum + srv = h.restart() + g = c.get("/v1/graphs/kb")[1] + assert g["nodes"] == 3 and g["generation"] == 1 and g["status"] == "degraded" + assert srv.engine.startup_report["degraded"] == 1 + + +def test_sidecar_from_another_save_is_caught_even_with_matching_checksums(harness): + """Swap in an internally consistent but *different* sidecar and fix the manifest checksum: + the library's graph-digest check (and ours) must still refuse it.""" + h = harness() + c = two_generations(h) + h.server.stop() + g1, g2 = gdir(h) / "gen-000001", gdir(h) / "gen-000002" + foreign = (g1 / "graph.isong.embeddings.json").read_bytes() # belongs to the 3-node graph + (g2 / "graph.isong.embeddings.json").write_bytes(foreign) + manifest = json.loads((g2 / "MANIFEST.json").read_text()) + sha, size = storage.sha256_file(g2 / "graph.isong.embeddings.json") + manifest["files"]["graph.isong.embeddings.json"] = {"sha256": sha, "size": size} + (g2 / "MANIFEST.json").write_text(json.dumps(manifest)) + with pytest.raises(SnapshotError, match="sidecar|digest|belong"): + GraphStorage(gdir(h)).load_generation(g2, GraphStorage(gdir(h)).read_meta(), h.server.encoder) + h.restart() + assert c.get("/v1/graphs/kb")[1]["generation"] == 1 + + +def test_all_generations_corrupt_quarantines_only_that_graph(harness): + h = harness() + c = two_generations(h) + create_graph(c, "other") + c.post("/v1/graphs/other/save", {}) + h.server.stop() + for gen in ("gen-000001", "gen-000002"): + (gdir(h) / gen / "graph.isong").write_bytes(b"garbage") + h.restart() + s, r, _ = c.get("/v1/graphs/kb") + assert s == 503 and r["error"]["code"] == "graph_quarantined" + assert c.post("/v1/graphs/kb/search", {"query": "x"})[0] == 503 + assert c.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "a", "id": 1}]})[0] == 503 + assert c.get("/v1/graphs/other")[0] == 200 # neighbours unaffected + listing = {g["name"]: g["status"] for g in c.get("/v1/graphs")[1]["items"]} + assert listing["kb"] == "quarantined" and listing["other"] == "ok" + assert "garbage" not in json.dumps(c.get("/v1/graphs/kb")[1]) + # not deleted: the files are still there for an operator, and the tenant can drop it + assert (gdir(h) / "gen-000002").exists() + assert c.delete("/v1/graphs/kb")[0] == 200 + assert not gdir(h).exists() + + +def test_strict_startup_refuses_to_start(harness): + h = harness() + two_generations(h) + h.server.stop() + for gen in ("gen-000001", "gen-000002"): + (gdir(h) / gen / "MANIFEST.json").write_bytes(b"{}") + from isongraph_vector_server.server import Server + h.cfg.strict_startup = True + with pytest.raises(SnapshotError): + Server(h.cfg, configure_logging=False).start() + h.cfg.strict_startup = False + h.server = Server(h.cfg, configure_logging=False).start() # fixture teardown needs a live one + + +def test_leftover_temp_dirs_are_cleaned_on_start(harness): + h = harness() + c = h.tenant("alice") + create_graph(c) + seed(c) + c.post("/v1/graphs/kb/save", {}) + h.server.stop() + (gdir(h) / ".tmp-gen-000002-deadbeef").mkdir() + (gdir(h) / ".tmp-gen-000002-deadbeef" / "graph.isong").write_text("half written") + (gdir(h).parent / ".tmp-create-abc").mkdir() + h.restart() + assert not (gdir(h) / ".tmp-gen-000002-deadbeef").exists() + assert not (gdir(h).parent / ".tmp-create-abc").exists() + assert c.get("/v1/graphs/kb")[1]["nodes"] == 3 + + +def test_missing_current_pointer_uses_newest_verified_generation(harness): + h = harness() + c = two_generations(h) + h.server.stop() + (gdir(h) / "CURRENT").unlink() + h.restart() + assert c.get("/v1/graphs/kb")[1]["nodes"] == 4 + + +def test_two_processes_cannot_share_a_data_dir(harness): + h = harness() + from isongraph_vector_server.server import Server + cfg = h.cfg + with pytest.raises(SnapshotError, match="in use"): + Server(cfg, configure_logging=False).start() + + +def test_graceful_shutdown_finishes_in_flight_requests_and_saves(harness): + import threading + h = harness() + c = h.tenant("alice") + create_graph(c) + results = [] + + def writer(): + for i in range(30): + try: + s, _, _ = c.post("/v1/graphs/kb/nodes", + {"nodes": [{"type": "n", "id": i, "properties": {"name": f"n{i}"}}]}) + except OSError: # the server went away mid-request: not acknowledged + return + results.append((i, s)) + + t = threading.Thread(target=writer) + t.start() + time.sleep(0.15) + h.server.stop() + t.join() + acked = [i for i, s in results if s == 200] + assert acked, "the writer should have got some acknowledged writes in before shutdown" + h.restart() + # every write acknowledged before shutdown is present afterwards + have = {x["id"] for x in c.get("/v1/graphs/kb/nodes?limit=1000")[1]["items"]} + assert set(acked) <= have + + +# ---- the format's quirks ------------------------------------------------------------------------------------------- + + +def test_library_roundtrip_turns_missing_properties_into_null(): + """Characterisation of the library (not of this service): why `semantic_digest` exists.""" + g = ISONGraph() + g.add_node("t", 1, name="a") + g.add_node("t", 2) + back = ISONGraph.from_ison(g.to_ison(), "x") + assert back.get_node("t", 2).properties == {"name": None} + assert graph_digest(g) != graph_digest(back) + assert semantic_digest(g) == semantic_digest(back) + + +def test_heterogeneous_nodes_snapshot_and_reload_cleanly(harness): + h = harness() + c = h.tenant("alice") + create_graph(c) + c.post("/v1/graphs/kb/nodes", {"nodes": [ + {"type": "t", "id": 1, "properties": {"name": "a", "only_one": 1}}, + {"type": "t", "id": 2, "properties": {"name": "b"}}, + {"type": "t", "id": 3, "properties": {}}, + {"type": "u", "id": "x", "properties": {"other": 2.5, "flag": True}}]}) + c.post("/v1/graphs/kb/edges", {"edges": [ + {"rel": "R", "source": {"type": "t", "id": 1}, "target": {"type": "t", "id": 2}, "properties": {"w": 1}}, + {"rel": "R", "source": {"type": "t", "id": 2}, "target": {"type": "t", "id": 3}}]}) + before = {n["id"] if n["type"] == "t" else "x": n["properties"] for n in + c.get("/v1/graphs/kb/nodes?limit=100")[1]["items"]} + h.restart() + after = {n["id"] if n["type"] == "t" else "x": n["properties"] for n in + c.get("/v1/graphs/kb/nodes?limit=100")[1]["items"]} + assert before == after # nulls are never shown + edges = c.get("/v1/graphs/kb/edges")[1]["items"] + assert sorted(e["properties"].get("w", -1) for e in edges) == [-1, 1] + assert c.post("/v1/graphs/kb/save", {})[0] == 200 # re-saving a reloaded graph still verifies + + +def test_every_value_the_api_accepts_survives_a_snapshot_round_trip(): + """Fuzz the validator against the real ISON round trip: whatever validation lets + through must come back identical (this is how the unsafe characters were found).""" + from isongraph_vector_server import validation + from isongraph_vector_server.errors import ApiError + rnd = random.Random(1234) + alphabet = list("ab1 \"'\\\n\r\t#|:,;{}[]~$@/-.=_") + ["é", "日", "\U0001f600", "\x00", "\x01", + "\x7f", " ", "
", "\u0085", "", " ", + "​", "‎", "⁠", "\ud800"] + values = [0, 1, -1, 2 ** 53 - 1, 0.5, -0.0, 1.0, 100.0, 0.1, 1e15, 123456789.125, 1e-4, True, False, ""] + for _ in range(4000): + values.append("".join(rnd.choice(alphabet) for _ in range(rnd.randint(0, 6)))) + accepted = 0 + g = SemanticGraph(encoder=None, auto_embed=False) + i = 0 + for v in values: + try: + validation.scalar_property_value(v, "$") + except ApiError: + continue + accepted += 1 + i += 1 + node = g.add_node("t", i) + node.properties["v"] = v + node.properties["w"] = v + assert accepted > 2000 and i == accepted + img = storage.serialize(g) + back = ISONGraph.from_ison(img.graph_text, "x") + assert semantic_digest(back) == img.semantic + # explicit: strict equality of every value including type (1.0 stays a float, "1" stays a string) + for node in g.nodes(): + got = back.get_node("t", node.id).properties["v"] + want = node.properties["v"] + assert got == want and type(got) is type(want), (want, got) + + +def test_unsafe_unicode_whitespace_is_rejected_up_front(harness): + """Regression for a round-trip bug found by fuzzing: U+00A0, U+2028, U+0085 and friends + are stripped by the ISON text format, so the stored value would silently change.""" + h = harness() + c = h.tenant("alice") + create_graph(c) + for ch in (" ", "
", "\u0085", "", " "): + s, r, _ = c.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "t", "id": 1, "properties": {"v": f"a{ch}"}}]}) + assert s == 400, repr(ch) + assert c.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "t", "id": 1, "properties": {"v": "café 日\U0001f600"}}]})[0] == 200 + + +def test_save_refuses_to_publish_a_lossy_snapshot(harness): + """Defence in depth: if memory ever holds something the format cannot carry, the save fails + loudly and the previous generation stays current.""" + h = harness() + c = h.tenant("alice") + create_graph(c) + seed(c) + c.post("/v1/graphs/kb/save", {}) + handle = h.server.engine.tenants["alice"].graphs["kb"] + handle.graph.get_node("person", 1).properties["zzlast"] = "x " # bypasses validation; last column, so the format strips it + handle.bump() + s, r, _ = c.post("/v1/graphs/kb/save", {}) + assert s >= 500 and "round trip" not in json.dumps(r) # generic error, no internals + assert (gdir(h) / "CURRENT").read_text().strip() == "000001" + assert not [p for p in gdir(h).iterdir() if p.name.startswith(".tmp-gen")] + assert h.server.metrics.counter("igv_snapshot_failures_total") == 1 + + +def test_snapshot_does_not_block_readers_and_is_consistent_under_writes(harness): + import threading + h = harness() + c = h.tenant("alice") + create_graph(c) + stop = threading.Event() + errors = [] + + def writer(): + i = 0 + while not stop.is_set(): + s, r, _ = c.post("/v1/graphs/kb/nodes", {"nodes": [ + {"type": "n", "id": i * 2, "properties": {"name": f"a{i}"}}, + {"type": "n", "id": i * 2 + 1, "properties": {"name": f"b{i}"}}]}) + if s != 200: + errors.append(r) + i += 1 + + t = threading.Thread(target=writer) + t.start() + try: + for _ in range(8): + assert c.post("/v1/graphs/kb/save", {})[0] == 200 + time.sleep(0.02) + finally: + stop.set() + t.join() + assert not errors + h.server.stop() + # every generation on disk is internally consistent AND holds whole batches (even count of nodes) + st = GraphStorage(gdir(h)) + meta = st.read_meta() + for gen in st.generations(): + graph, manifest, _ = st.load_generation(st.gen_path(gen), meta, h.server.encoder) + assert graph.node_count() % 2 == 0 + assert graph.embedding_store.count() == graph.node_count() + graph.close() + h.restart() + + +def test_cli_verify_reports_problems(harness, tmp_path, capsys): + from isongraph_vector_server.cli import main + h = harness() + c = two_generations(h) + h.server.stop() + assert main(["verify", "--data-dir", h.data_dir]) == 0 + (gdir(h) / "gen-000002" / "graph.isong").write_bytes(b"x") + assert main(["verify", "--data-dir", h.data_dir]) == 1 + out = capsys.readouterr().out + assert "DEGRADED" in out + h.restart() diff --git a/isongraph-vector-server/tests/test_replication.py b/isongraph-vector-server/tests/test_replication.py new file mode 100644 index 0000000..43efbb6 --- /dev/null +++ b/isongraph-vector-server/tests/test_replication.py @@ -0,0 +1,309 @@ +"""Snapshot read replicas: what they serve, and when they refuse to.""" + +import json +import shutil +import time +from pathlib import Path + +import pytest + +from conftest import Client, create_graph, seed +from isongraph_vector_server import auth, storage + +TOKEN = "t" * 40 + + +def gen_dir(h, tenant, graph, gen): + return Path(h.data_dir) / "tenants" / tenant / "graphs" / f"g-{graph}" / f"gen-{gen:06d}" + + +@pytest.fixture +def primary(harness): + h = harness(replication_token=TOKEN) + a, b = h.tenant("tenant-a"), h.tenant("tenant-b") + create_graph(a, "kb") + seed(a) + create_graph(b, "kb") + b.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "person", "id": 1, "properties": {"name": "Zed"}}]}) + for c in (a, b): + assert c.post("/v1/graphs/kb/save", {})[0] == 200 + return h, a, b + + +def as_replica(replica_h, client): + return Client(replica_h.server.url, client.key) + + +def sync(replica_h): + return replica_h.server.replica.sync_once() + + +@pytest.fixture(params=["dir", "http"]) +def source(request, primary): + h, _, _ = primary + return f"dir:{h.data_dir}" if request.param == "dir" else h.server.url + + +def test_replica_serves_the_primarys_snapshot_read_only(primary, replica_of, source): + h, a, b = primary + r = replica_of(source, **({"replication_token": TOKEN} if source.startswith("http") else {})) + report = sync(r) + assert not report["errors"] and not report["refused"], report + ra, rb = as_replica(r, a), as_replica(r, b) + # same answers as the primary + q = {"query": "Bob trains machine learning models", "top_k": 3, "threshold": -1.0} + assert ra.post("/v1/graphs/kb/search", q)[1] == a.post("/v1/graphs/kb/search", q)[1] + s, g, hd = ra.get("/v1/graphs/kb") + assert g["nodes"] == 3 and g["snapshot_digest"] == a.get("/v1/graphs/kb")[1]["snapshot_digest"] + assert hd["X-Snapshot-Generation"] == "1" and int(hd["X-Snapshot-Age-Seconds"]) < 30 + # tenant isolation holds on the replica too + assert [x["id"] for x in rb.get("/v1/graphs/kb/nodes")[1]["items"]] == [1] + assert rb.get("/v1/graphs/kb/nodes/person/2")[0] == 404 + # writes of every kind are refused with a stable code + for method, path, body in [("POST", "/v1/graphs", {"name": "x"}), ("DELETE", "/v1/graphs/kb", None), + ("POST", "/v1/graphs/kb/nodes", {"nodes": [{"type": "a", "id": 1}]}), + ("PATCH", "/v1/graphs/kb/nodes/person/1", {}), ("POST", "/v1/graphs/kb/save", {}), + ("POST", "/v1/graphs/kb/reembed", {})]: + s, resp, _ = ra.request(method, path, body) + assert s == 403 and resp["error"]["code"] == "read_only_replica", (method, path) + # a replica does not itself serve replication + assert Client(r.server.url, TOKEN).get("/_repl/v1/index")[0] == 404 + + +def test_incremental_sync_only_moves_what_changed(primary, replica_of): + h, a, b = primary + r = replica_of(f"dir:{h.data_dir}") + first = sync(r) + assert sorted(first["updated"]) == ["tenant-a/kb@1", "tenant-a/tenant.json", "tenant-b/kb@1", "tenant-b/tenant.json"] + assert sync(r)["updated"] == [] # nothing changed + a.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "person", "id": 4, "properties": {"name": "Dan"}}]}) + a.post("/v1/graphs/kb/save", {}) + second = sync(r) + assert second["updated"] == ["tenant-a/kb@2"] + assert as_replica(r, a).get("/v1/graphs/kb")[1]["nodes"] == 4 + assert as_replica(r, b).get("/v1/graphs/kb")[1]["generation"] == 1 + + +def test_replica_lags_by_design_until_the_primary_saves_and_it_syncs(primary, replica_of): + h, a, b = primary + r = replica_of(f"dir:{h.data_dir}") + sync(r) + a.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "person", "id": 5, "properties": {"name": "Eve"}}]}) + ra = as_replica(r, a) + sync(r) + assert ra.get("/v1/graphs/kb/nodes/person/5")[0] == 404 # unsaved on the primary: invisible + a.post("/v1/graphs/kb/save", {}) + assert ra.get("/v1/graphs/kb/nodes/person/5")[0] == 404 # saved but not yet synced + sync(r) + assert ra.get("/v1/graphs/kb/nodes/person/5")[0] == 200 + + +# ---- refusing bad snapshots -------------------------------------------------------------------------------------- + + +def make_gen2(h, a): + a.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "person", "id": 4, "properties": {"name": "Dan"}}]}) + assert a.post("/v1/graphs/kb/save", {})[0] == 200 + return gen_dir(h, "tenant-a", "kb", 2) + + +def rewrite_manifest(g2, **file_updates): + m = json.loads((g2 / "MANIFEST.json").read_text()) + for fname in ("graph.isong", "graph.isong.embeddings.json"): + sha, size = storage.sha256_file(g2 / fname) + m["files"][fname] = {"sha256": sha, "size": size} + m.update(file_updates) + (g2 / "MANIFEST.json").write_text(json.dumps(m)) + + +def tamper_checksum(g2): + (g2 / "graph.isong").write_bytes((g2 / "graph.isong").read_bytes().replace(b"Dan", b"Eve")) + + +def tamper_graph_consistent_checksums(g2): + """The attacker fixes the manifest checksums too: only the digest binding catches it.""" + (g2 / "graph.isong").write_bytes((g2 / "graph.isong").read_bytes().replace(b"Dan", b"Eve")) + rewrite_manifest(g2) + + +def swap_in_older_sidecar(g2, h): + older = gen_dir(h, "tenant-a", "kb", 1) / "graph.isong.embeddings.json" + (g2 / "graph.isong.embeddings.json").write_bytes(older.read_bytes()) + rewrite_manifest(g2) + + +def lie_about_digest_in_manifest(g2): + m = json.loads((g2 / "MANIFEST.json").read_text()) + m["graph_digest"] = "sha256:" + "0" * 64 + (g2 / "MANIFEST.json").write_text(json.dumps(m)) + + +def truncate_sidecar(g2): + p = g2 / "graph.isong.embeddings.json" + p.write_bytes(p.read_bytes()[:-50]) + + +def strip_digest_from_sidecar(g2): + p = g2 / "graph.isong.embeddings.json" + payload = json.loads(p.read_text()) + payload.pop("graph_digest") + payload.pop("graph_digest_basis") + p.write_text(json.dumps(payload)) + rewrite_manifest(g2) + + +@pytest.mark.parametrize("tamper", [ + "tamper_checksum", "tamper_graph_consistent_checksums", "swap_in_older_sidecar", + "lie_about_digest_in_manifest", "truncate_sidecar", "strip_digest_from_sidecar"]) +def test_replica_refuses_a_snapshot_that_fails_verification(primary, replica_of, tamper): + h, a, b = primary + r = replica_of(f"dir:{h.data_dir}") + sync(r) + ra = as_replica(r, a) + before = ra.get("/v1/graphs/kb")[1] + last_ok = r.server.replica.last_ok_wall + g2 = make_gen2(h, a) + fn = globals()[tamper] + fn(g2, h) if tamper == "swap_in_older_sidecar" else fn(g2) + + report = sync(r) + assert report["refused"] == ["tenant-a/kb"], report + assert report["updated"] == [] + # still serving the last verified generation, byte for byte + now = ra.get("/v1/graphs/kb")[1] + assert (now["generation"], now["nodes"], now["snapshot_digest"]) == ( + before["generation"], before["nodes"], before["snapshot_digest"]) + assert ra.get("/v1/graphs/kb/nodes/person/4")[0] == 404 + assert r.server.replica.refused == 1 + assert "tenant-a/kb" in r.server.replica.last_error + assert r.server.replica.last_ok_wall == last_ok # a refused cycle is not a fresh one + assert r.server.metrics.counter("igv_replica_refused_total") == 1 + # nothing of the bad generation was installed + rgraph = Path(r.data_dir) / "tenants" / "tenant-a" / "graphs" / "g-kb" + assert not (rgraph / "gen-000002").exists() + assert (rgraph / "CURRENT").read_text().strip() == "000001" + assert not [p for p in rgraph.parent.iterdir() if p.name.startswith(".tmp")] + # other tenants still sync fine in the same cycle + b.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "person", "id": 9, "properties": {"name": "Ok"}}]}) + b.post("/v1/graphs/kb/save", {}) + report = sync(r) + assert report["refused"] == ["tenant-a/kb"] and report["updated"] == ["tenant-b/kb@2"] + # and once the primary publishes a good generation, the replica catches up + a.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "person", "id": 6, "properties": {"name": "Fay"}}]}) + a.post("/v1/graphs/kb/save", {}) + report = sync(r) + assert report["refused"] == [] and ra.get("/v1/graphs/kb/nodes/person/6")[0] == 200 + + +def test_replica_refuses_over_http_when_the_transport_corrupts_a_file(primary, replica_of): + h, a, b = primary + r = replica_of(h.server.url, replication_token=TOKEN) + sync(r) + g2 = make_gen2(h, a) + tamper_checksum(g2) + report = sync(r) + assert report["refused"] == ["tenant-a/kb"] + + +def test_staleness_bound_makes_the_replica_stop_serving(primary, replica_of, tmp_path): + h, a, b = primary + r = replica_of(f"dir:{h.data_dir}", **{"replica.max_staleness_seconds": 1.5}) + sync(r) + ra = as_replica(r, a) + assert ra.get("/v1/graphs/kb")[0] == 200 + assert r.anon().get("/readyz")[0] == 200 + # the primary disappears: syncs fail, the clock keeps running + shutil.move(h.data_dir + "/tenants", h.data_dir + "/tenants.gone") + try: + assert sync(r)["errors"] == ["index"] + time.sleep(1.7) + s, resp, hd = ra.get("/v1/graphs/kb") + assert s == 503 and resp["error"]["code"] == "replica_stale" and "Retry-After" in hd + assert ra.post("/v1/graphs/kb/search", {"query": "x"})[0] == 503 + assert r.anon().get("/readyz")[0] == 503 + assert r.anon().get("/healthz")[0] == 200 # alive, just not serving + finally: + shutil.move(h.data_dir + "/tenants.gone", h.data_dir + "/tenants") + assert not sync(r)["errors"] + assert ra.get("/v1/graphs/kb")[0] == 200 + assert r.anon().get("/readyz")[0] == 200 + + +def test_a_replica_that_never_synced_serves_nothing(primary, replica_of, tmp_path): + h, a, b = primary + r = replica_of(f"dir:{tmp_path / 'does-not-exist'}") + assert r.anon().get("/readyz")[0] == 503 + s, resp, _ = as_replica(r, a).get("/v1/graphs") + assert s == 401 or (s == 503 and resp["error"]["code"] == "replica_not_synced") + + +def test_replica_restart_keeps_serving_verified_data_and_remembers_staleness(primary, replica_of): + h, a, b = primary + r = replica_of(f"dir:{h.data_dir}", **{"replica.max_staleness_seconds": 3600}) + sync(r) + stamp = r.server.replica.last_ok_wall + shutil.move(h.data_dir + "/tenants", h.data_dir + "/tenants.gone") # primary unreachable during the restart + try: + r.restart() + assert r.server.replica.last_ok_wall == pytest.approx(stamp, abs=1e-3) + ra = as_replica(r, a) + assert ra.get("/v1/graphs/kb")[1]["nodes"] == 3 + finally: + shutil.move(h.data_dir + "/tenants.gone", h.data_dir + "/tenants") + + +def test_revoking_a_key_reaches_the_replica_on_the_next_sync(primary, replica_of): + h, a, b = primary + r = replica_of(f"dir:{h.data_dir}") + sync(r) + ra = as_replica(r, a) + assert ra.get("/v1/graphs/kb")[0] == 200 + auth.revoke_key(h.data_dir, "tenant-a", a.key.split(".")[1]) + assert a.get("/v1/graphs")[0] == 401 # primary: immediately + assert ra.get("/v1/graphs/kb")[0] == 200 # replica: documented lag until the next sync + sync(r) + assert ra.get("/v1/graphs/kb")[0] == 401 + + +def test_deletions_propagate(primary, replica_of): + h, a, b = primary + r = replica_of(f"dir:{h.data_dir}") + sync(r) + create_graph(a, "temp") + a.post("/v1/graphs/temp/save", {}) + sync(r) + ra = as_replica(r, a) + assert ra.get("/v1/graphs/temp")[0] == 200 + a.delete("/v1/graphs/temp") + report = sync(r) + assert report["removed"] == ["tenant-a/temp"] + assert ra.get("/v1/graphs/temp")[0] == 404 + assert not (Path(r.data_dir) / "tenants" / "tenant-a" / "graphs" / "g-temp").exists() + + +def test_an_unreadable_graph_on_the_primary_is_not_mistaken_for_a_deleted_one(primary, replica_of): + h, a, b = primary + r = replica_of(f"dir:{h.data_dir}") + sync(r) + (Path(h.data_dir) / "tenants" / "tenant-a" / "graphs" / "g-kb" / "meta.json").unlink() + report = sync(r) + assert report["removed"] == [] and "tenant-a/kb" in report["errors"] + assert as_replica(r, a).get("/v1/graphs/kb")[1]["nodes"] == 3 # replica keeps what it verified + + +def test_empty_never_saved_graph_replicates(primary, replica_of): + h, a, b = primary + create_graph(a, "fresh") # created, never saved: no generations + r = replica_of(f"dir:{h.data_dir}") + report = sync(r) + assert "tenant-a/fresh@None" in report["updated"] + s, g, _ = as_replica(r, a).get("/v1/graphs/fresh") + assert s == 200 and g["nodes"] == 0 + + +def test_http_source_requires_the_right_token(primary, replica_of): + h, a, b = primary + r = replica_of(h.server.url, replication_token="x" * 40) + report = sync(r) + assert report["errors"] == ["index"] + assert as_replica(r, a).get("/v1/graphs")[0] in (401, 503) diff --git a/isongraph-vector-server/tests/test_router.py b/isongraph-vector-server/tests/test_router.py new file mode 100644 index 0000000..ef2f588 --- /dev/null +++ b/isongraph-vector-server/tests/test_router.py @@ -0,0 +1,315 @@ +"""The stateless tenant-sharding router.""" + +import socket +import threading +import time +from collections import Counter +from pathlib import Path + +import pytest + +from conftest import Client, create_graph, seed +from isongraph_vector_server import auth +from isongraph_vector_server.config import ServerConfig +from isongraph_vector_server.router import Router, rendezvous +from isongraph_vector_server.server import RouterServer + + +def router_cfg(shards, **router): + cfg = ServerConfig(host="127.0.0.1", port=0, log_level="CRITICAL") + cfg.router.shards = shards + cfg.router.expose_backend_header = True + for k, v in router.items(): + setattr(cfg.router, k, v) + return cfg + + +@pytest.fixture +def cluster(harness): + """Two storage nodes behind a router. Tenants are created on the node that owns them.""" + n0, n1 = harness(), harness() + shards = [{"name": "s0", "primary": n0.server.url}, {"name": "s1", "primary": n1.server.url}] + rs = RouterServer(router_cfg(shards), configure_logging=False).start() + nodes = {"s0": n0, "s1": n1} + + class Cluster: + router = rs + node = nodes + + @staticmethod + def tenant(tid, role="rw"): + owner = rs.router.shard_for(tid) + key = auth.add_key(nodes[owner].data_dir, tid, role) if auth_exists(nodes[owner], tid) \ + else make_tenant(nodes[owner], tid, role) + return Client(rs.url, key), owner + + yield Cluster + rs.stop() + + +def auth_exists(node, tid): + return (Path(node.data_dir) / "tenants" / tid / "tenant.json").exists() + + +def make_tenant(node, tid, role): + auth.create_tenant(node.data_dir, tid) + return auth.add_key(node.data_dir, tid, role) + + +def find_tenants_on_each_shard(router): + found = {} + i = 0 + while len(found) < 2: + tid = f"tenant-{i}" + found.setdefault(router.router.shard_for(tid), tid) + i += 1 + return found + + +def test_rendezvous_is_deterministic_balanced_and_minimally_disruptive(): + shards = ["s0", "s1", "s2", "s3"] + tenants = [f"tenant-{i}" for i in range(2000)] + owner = {t: rendezvous(t, shards) for t in tenants} + assert owner == {t: rendezvous(t, list(reversed(shards))) for t in tenants} # order-independent + counts = Counter(owner.values()) + assert all(350 < n < 650 for n in counts.values()), counts # ~500 each + # drop one shard: only that shard's tenants move + fewer = {t: rendezvous(t, ["s0", "s1", "s3"]) for t in tenants} + moved = [t for t in tenants if owner[t] != fewer[t]] + assert all(owner[t] == "s2" for t in moved) and len(moved) == counts["s2"] + # add one: only tenants won by the new shard move + more = {t: rendezvous(t, shards + ["s4"]) for t in tenants} + assert all(more[t] == "s4" for t in tenants if owner[t] != more[t]) + + +def test_requests_reach_the_owning_shard_only(cluster): + on = find_tenants_on_each_shard(cluster.router) + clients = {} + for shard, tid in on.items(): + c, owner = cluster.tenant(tid) + assert owner == shard + create_graph(c) + seed(c) + clients[shard] = c + s, r, hd = c.get("/v1/graphs/kb") + assert s == 200 and hd["X-Igv-Backend"] == f"{shard}/primary" + # the data is on exactly one node, and the other node knows nothing of the tenant + for shard, tid in on.items(): + other = "s1" if shard == "s0" else "s0" + assert (Path(cluster.node[shard].data_dir) / "tenants" / tid).exists() or True + assert not (Path(cluster.node[other].data_dir) / "tenants" / tid).exists() + direct = Client(cluster.node[other].server.url, clients[shard].key) + assert direct.get("/v1/graphs")[0] == 401 + assert cluster.node[other].server.engine.tenants.get(tid) is None + + +def test_tenant_data_never_crosses_shards_or_tenants(cluster): + on = find_tenants_on_each_shard(cluster.router) + a, _ = cluster.tenant(on["s0"]) + b, _ = cluster.tenant(on["s1"]) + same_shard, owner = cluster.tenant(next(f"extra-{i}" for i in range(500) + if cluster.router.router.shard_for(f"extra-{i}") == "s0")) + for c in (a, b, same_shard): + create_graph(c) + a.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "d", "id": 1, "properties": {"name": "A-SECRET"}}]}) + b.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "d", "id": 1, "properties": {"name": "B-thing"}}]}) + assert "A-SECRET" not in str(b.get("/v1/graphs/kb/nodes")[1]) + assert "A-SECRET" not in str(same_shard.get("/v1/graphs/kb/nodes")[1]) # same node, other tenant + assert same_shard.get("/v1/graphs/kb/nodes/d/1")[0] == 404 + + +def test_a_key_that_names_another_tenant_is_routed_there_and_fails(cluster): + on = find_tenants_on_each_shard(cluster.router) + a, _ = cluster.tenant(on["s0"]) + b, _ = cluster.tenant(on["s1"]) + create_graph(a) + ta, ka, sa = a.key.split(".") + tb, kb, sb = b.key.split(".") + # B's credentials presented under A's tenant id: routed to A's shard, rejected there + for forged in (f"{ta}.{kb}.{sb}", f"{ta}.{ka}.{sb}"): + s, r, hd = Client(cluster.router.url, forged).get("/v1/graphs/kb") + assert s == 401, forged + # and A's under B's + s, r, hd = Client(cluster.router.url, f"{tb}.{ka}.{sa}").get("/v1/graphs/kb") + assert s == 401 + + +def test_router_rejects_malformed_credentials_without_asking_a_backend(cluster): + before = [n.server.metrics.counter("igv_requests_total", route="graphs.list", status="4xx") + for n in cluster.node.values()] + c = Client(cluster.router.url, None) + for hdrs in ({}, {"Authorization": "Bearer nope"}, {"Authorization": "Basic abc"}, {"Authorization": "Bearer "}): + s, r, _ = c.get("/v1/graphs", headers=hdrs) + assert s == 401 and r["error"]["code"] == "unauthorized" + after = [n.server.metrics.counter("igv_requests_total", route="graphs.list", status="4xx") + for n in cluster.node.values()] + assert before == after + + +def test_only_the_tenant_api_is_proxied(cluster): + tid = find_tenants_on_each_shard(cluster.router)["s0"] + c, _ = cluster.tenant(tid) + for path in ("/metrics", "/readyz", "/_repl/v1/index", "/", "/v2/graphs", "/admin"): + assert c.get(path)[0] == 404, path + assert Client(cluster.router.url, None).get("/healthz")[1]["role"] == "router" + + +def test_errors_and_headers_pass_through_unchanged(cluster): + c, _ = cluster.tenant("tenant-x") + create_graph(c) + s, r, hd = c.post("/v1/graphs", {"name": "kb"}) + assert s == 409 and r["error"]["code"] == "already_exists" + s, r, hd = c.post("/v1/graphs/kb/search", {"query": "x"}, headers={"X-Request-Id": "trace-abcdef123"}) + assert hd["X-Request-Id"] == "trace-abcdef123" + s, r, hd = c.request("PUT", "/v1/graphs", {"name": "x"}) + assert s == 405 and "Allow" in hd + + +def test_ambiguous_credentials_and_oversize_bodies(cluster): + import http.client + c, _ = cluster.tenant("tenant-x") + conn = http.client.HTTPConnection(c.host, c.port, timeout=10) + conn.putrequest("GET", "/v1/graphs") + conn.putheader("Authorization", f"Bearer {c.key}") + conn.putheader("Authorization", f"Bearer {c.key}") + conn.endheaders() + resp = conn.getresponse() + resp.read() + assert resp.status == 401 + s, r, _ = c.post("/v1/graphs", None, raw=b'{"name":"' + b"a" * 2_000_000 + b'"}') + assert s == 413 + + +def test_backend_down_means_502_for_its_tenants_only_and_no_failover(cluster): + on = find_tenants_on_each_shard(cluster.router) + a, _ = cluster.tenant(on["s0"]) + b, _ = cluster.tenant(on["s1"]) + create_graph(a) + create_graph(b) + cluster.node["s0"].server.httpd.stop() + s, r, hd = a.get("/v1/graphs") + assert s == 502 and r["error"]["code"] == "backend_unavailable" + assert b.get("/v1/graphs")[0] == 200 # the other shard is untouched + # and a write to the dead shard is not silently redirected anywhere + assert a.post("/v1/graphs", {"name": "zz"})[0] == 502 + assert not any(n.server.engine.tenants.get(on["s0"]) and "zz" in n.server.engine.tenants[on["s0"]].graphs + for n in [cluster.node["s1"]]) + + +class FlakyBackend: + """A listener that accepts, counts, and hangs up without answering.""" + + def __init__(self): + self.sock = socket.socket() + self.sock.bind(("127.0.0.1", 0)) + self.sock.listen(8) + self.count = 0 + self.url = f"http://127.0.0.1:{self.sock.getsockname()[1]}" + self._t = threading.Thread(target=self._run, daemon=True) + self._t.start() + + def _run(self): + while True: + try: + conn, _ = self.sock.accept() + except OSError: + return + self.count += 1 + try: + conn.recv(65536) + finally: + conn.close() + + def close(self): + self.sock.close() + + +def test_writes_are_never_retried_but_reads_fall_back_to_the_primary(harness): + primary = harness() + key = make_tenant(primary, "tenant-x", "rw") + flaky = FlakyBackend() + try: + cfg = router_cfg([{"name": "s0", "primary": primary.server.url, "replicas": [flaky.url]}], + read_preference="replica_preferred") + rs = RouterServer(cfg, configure_logging=False).start() + try: + c = Client(rs.url, key) + create_graph(c) # a write: straight to the primary + assert flaky.count == 0 + for _ in range(3): # reads: replica fails, primary answers + s, r, hd = c.get("/v1/graphs/kb") + assert s == 200 and hd["X-Igv-Backend"] == "s0/primary" + assert flaky.count == 3 + s, _, hd = c.post("/v1/graphs/kb/search", {"query": "x", "threshold": -1.0}) + assert s == 200 and hd["X-Igv-Backend"] == "s0/primary" and flaky.count == 4 + + # a primary that hangs up: the write is attempted exactly once + cfg2 = router_cfg([{"name": "s0", "primary": flaky.url}]) + rs2 = RouterServer(cfg2, configure_logging=False).start() + try: + before = flaky.count + s, r, _ = Client(rs2.url, key).post("/v1/graphs", {"name": "other"}) + assert s == 502 and flaky.count == before + 1 + finally: + rs2.stop() + finally: + rs.stop() + finally: + flaky.close() + + +def test_reads_go_to_a_replica_when_asked_and_primary_when_forced(harness, replica_of): + primary = harness(replication_token="t" * 40) + key = make_tenant(primary, "tenant-x", "rw") + direct = Client(primary.server.url, key) + create_graph(direct) + seed(direct) + direct.post("/v1/graphs/kb/save", {}) + replica = replica_of(f"dir:{primary.data_dir}") + replica.server.replica.sync_once() + cfg = router_cfg([{"name": "s0", "primary": primary.server.url, "replicas": [replica.server.url]}], + read_preference="replica_preferred") + rs = RouterServer(cfg, configure_logging=False).start() + try: + c = Client(rs.url, key) + s, r, hd = c.get("/v1/graphs/kb") + assert s == 200 and hd["X-Igv-Backend"] == "s0/replica" and hd["X-Snapshot-Generation"] == "1" + for path in ("search", "multi_hop", "similar"): + body = {"similar": {"node": {"type": "person", "id": 1}}}.get(path, {"query": "Alice", "threshold": -1.0}) + s, r, hd = c.post(f"/v1/graphs/kb/{path}", body) + assert s == 200 and hd["X-Igv-Backend"] == "s0/replica", path + # writes and the explicit opt-out go to the primary + s, r, hd = c.post("/v1/graphs/kb/nodes", {"nodes": [{"type": "person", "id": 7, "properties": {"name": "G"}}]}) + assert s == 200 and hd["X-Igv-Backend"] == "s0/primary" + s, r, hd = c.get("/v1/graphs/kb/nodes/person/7", headers={"X-Igv-Consistency": "primary"}) + assert s == 200 and hd["X-Igv-Backend"] == "s0/primary" + # read-your-writes is NOT promised on the replica path + assert c.get("/v1/graphs/kb/nodes/person/7")[0] == 404 + # a replica that has gone stale tells the router to ask the primary instead + replica.cfg.replica.max_staleness_seconds = 0.001 + time.sleep(0.05) + s, r, hd = c.get("/v1/graphs/kb/nodes/person/7") + assert s == 200 and hd["X-Igv-Backend"] == "s0/primary" + finally: + rs.stop() + + +def test_pins_override_the_hash(cluster, harness): + tid = find_tenants_on_each_shard(cluster.router)["s0"] + cfg = router_cfg([{"name": "s0", "primary": cluster.node["s0"].server.url}, + {"name": "s1", "primary": cluster.node["s1"].server.url}], pins={tid: "s1"}) + router = Router(cfg) + assert router.shard_for(tid) == "s1" and cluster.router.router.shard_for(tid) == "s0" + with pytest.raises(Exception): + router_cfg([{"name": "s0", "primary": "http://x"}], pins={"t": "nope"}).validate() + + +def test_cli_shard_of_matches_the_router(cluster, tmp_path, capsys): + import json + from isongraph_vector_server.cli import main + cfgfile = tmp_path / "r.json" + cfgfile.write_text(json.dumps({"router": {"shards": [{"name": "s0", "primary": "http://a:1"}, + {"name": "s1", "primary": "http://b:1"}]}})) + for tid in ("tenant-1", "tenant-2", "tenant-3", "acme", "globex"): + assert main(["shard-of", tid, "--config", str(cfgfile)]) == 0 + assert capsys.readouterr().out.strip() == rendezvous(tid, ["s0", "s1"]) diff --git a/isongraph-vector-ts/README.md b/isongraph-vector-ts/README.md index a46989e..411dde1 100644 --- a/isongraph-vector-ts/README.md +++ b/isongraph-vector-ts/README.md @@ -211,9 +211,63 @@ which is where a third of the query time used to go. Filtering by node type stays exact too: the type is a `vec0` metadata column, so it narrows the search rather than filtering its results. +## Approximate search (opt-in) + +Search is exact by default. For larger stores, `ann: 'ivf'` answers bounded top-k searches from an in-memory IVF-flat index: +the vectors are clustered into cells with seeded spherical k-means, and a query +scores only the `annNprobe` cells nearest to it, with the same cosine formula +as the exact scan. + +```typescript +import { SemanticGraph, EmbeddingStore } from 'isongraph-vector-ts'; + +const graph = new SemanticGraph({ encoder, ann: 'ivf' }); // also on EmbeddingStore +graph.similaritySearch(query, 10); // approximate once the store is large enough +graph.similaritySearch(query, 10, undefined, 0, undefined, undefined, true); // exact: true forces the full scan +graph.recallCheck(sampleQueries, 10); // recall@10 against the exact scan, with timings +graph.embeddingStore.annStats(); // cells, pending rows, memory, counters +``` + +Options (also `EmbeddingStore`'s second argument): `ann`, `annNlist` (default +about `4 * sqrt(N)`, between 16 and 65536), `annNprobe` (default 5% of the +cells; settable later as `store.annNprobe`), `annMinSize` (default 50000: below +it search stays exact) and `annSeed` (the same data and seed give the same +index). `annNprobe >= annNlist` is the exact scan. + +- **It is approximate.** A true neighbour in an unprobed cell is missed. Run + `recallCheck()` on queries from your own workload and raise `annNprobe` until + the recall holds; the figure depends on how your data clusters, and a number + measured on other data does not transfer. +- **Scores are exact.** The index chooses the candidates with float32 copies of + the vectors; the `topK` it returns are re-scored with the same float64 cosine + the exact scan uses, so a score never differs between the two paths. +- **Lazy, and never persisted.** The first search builds it (or call + `buildAnnIndex()` to choose the moment); a saved graph, its sidecar and the + graph digest are byte-for-byte what they would be without it. Reload a graph + with `loadGraph(path, { ann: 'ivf' })` and it is rebuilt on demand. +- **Kept current.** New and updated nodes are scored exactly until enough have + accumulated to be folded into cells; deletes never come back; the index + retrains by itself once churn passes half the trained size. Filters + (`nodeType`, `threshold`, `nodeFilter`) are applied to the probed cells, and + the probe widens (4x per step, ending in the exact scan) when too few + candidates survive, so a selective filter costs time, not results. +- **Costs memory.** A Float32Array copy of the vectors (4 bytes per component), + next to the stored ones, plus about 8 bytes per vector for the cell lists. +- **Faster to query, slow to build.** Without BLAS the exact scan of 384-d + vectors takes about 5 ms at 10k, 50 ms at 100k and 0.5 s at 1M; the index + answers in 0.2-3 ms at 10k-100k (recall@10 1.00 at the default probe) and + in 1.4-27 ms at 1M (recall 0.75 at `annNprobe` 1, 0.94 at the default 200), + on synthetic clustered data. Building it runs k-means in plain JavaScript + and takes about 8 s at 10k, 55 s at 100k and 8 min at 1M, so call + `buildAnnIndex()` at startup instead of letting the first search pay for it. + It is not worth it much below 10k vectors. `isongraph-vector-js/bench/ann.mjs` + in the repository reproduces the numbers (`npm run bench:ann`). +- The index is for the in-memory `EmbeddingStore` (and so for `SemanticGraph`); + `SqliteEmbeddingStore` has no `ann` option. + ## Embeddings from an HTTP endpoint -`HttpEncoder` is an `EmbeddingEncoder` that calls an OpenAI-compatible +`HttpEncoder` is an `AsyncEncoder` (and an `EmbeddingEncoder`) that calls an OpenAI-compatible `/v1/embeddings` endpoint or Ollama's `/api/embed`. It uses the platform `fetch` - no dependency, no `node:` import, so it lives in the main entry and runs on Node 18+ and in a browser. @@ -223,7 +277,7 @@ search queries alike. Point it at a service you are willing to send that data to (a local Ollama keeps it on your machine). ```typescript -import { HttpEncoder, EmbeddingStore } from 'isongraph-vector-ts'; +import { HttpEncoder, SemanticGraph, EmbeddingStore } from 'isongraph-vector-ts'; const encoder = new HttpEncoder({ url: 'https://api.openai.com/v1/embeddings', // or http://localhost:11434/api/embed @@ -232,34 +286,58 @@ const encoder = new HttpEncoder({ apiKey: process.env.OPENAI_API_KEY, // sent as a Bearer token }); -const texts = ['Alice is an engineer', 'Bob plans roadmaps']; -await encoder.prefetch([...texts, 'engineering']); // the network calls happen here +// A graph embeds its nodes over the network with the Async methods - no prefetch. +const graph = new SemanticGraph({ encoder }); +await graph.addNodeAsync('person', 1, { name: 'Alice is an engineer' }); +await graph.addNodeAsync('person', 2, { name: 'Bob plans roadmaps' }); +const hits = await graph.similaritySearchAsync('engineering', 1); +// The bare store has them too. const store = new EmbeddingStore(encoder); -store.addBatch([[['person', 1], texts[0]], [['person', 2], texts[1]]]); -store.similaritySearch('engineering', 1); // served from the prefetched vectors +await store.addBatchAsync([[['person', 1], 'Alice is an engineer'], [['person', 2], 'Bob plans roadmaps']]); +await store.similaritySearchAsync('engineering', 1); ``` Options: `url`, `model` (also the `modelName` recorded on every vector), `api` (`'openai'` default, or `'ollama'`), `apiKey`, `dimension` (otherwise learned from the first response; `NaN` until then), `batchSize` (64), `timeoutMs` (30000), `headers`, `version`, `maxRetries` (2), `retryDelayMs` -(500), `maxCacheEntries` (100000) and `fetch` (a replacement for the global). +(500), `maxCacheEntries` (100000), `concurrency` (1) and `fetch` (a replacement for +the global). 429 and 5xx answers, timeouts and network errors are retried a bounded number of times with backoff (a `Retry-After` header is honoured, up to 30 s); other errors throw an `HttpEncoderError` carrying the HTTP `status`. Errors name the endpoint's origin and path, never the API key or the query string. -**It is not synchronous, and `EmbeddingEncoder` is.** The stores call -`encode` / `encodeBatch` and expect a vector back, so `HttpEncoder` serves those -from a cache and throws an `HttpEncoderError` for a text it has not fetched. -Fetch first with `await encoder.prefetch(texts)` (or `encodeAsync` / -`encodeBatchAsync`, which also return the vectors). In practice that means -`SemanticGraph` auto-embedding does not work with it - the graph builds the -text itself and asks for the vector at once - so use it with the stores and -`autoEmbed: false`, prefetching the texts you pass, or embed with -`encodeBatchAsync` and hand the vectors in. A bounded cache (`maxCacheEntries`) -evicts the oldest vectors, so prefetch no more than you are about to use. +**It is asynchronous, and the Async methods are how you use it.** +`HttpEncoder` implements `AsyncEncoder` (`encodeAsync` / `encodeBatchAsync`), +and the store and graph have an `...Async` method wherever the encoder is +called: `addAsync`, `addBatchAsync`, `similaritySearchAsync` on the store, and +on the graph `addNodeAsync`, `addNodeWithEmbedAsync`, `updateNodeAsync`, +`embedNodeAsync`, `embedNodesAsync`, `embedAllNodesAsync`, `reembedStaleAsync`, +`similaritySearchAsync`, `semanticMultiHopAsync`, `semanticPathAsync`, +`semanticSubgraphAsync` and `recallCheckAsync`. Each awaits the network and +then does exactly what its synchronous namesake does, with the same model +metadata (`modelName` on every vector, `staleEmbeddings()`, `onModelMismatch`) +and the same key redaction. They also work with any synchronous encoder, and +`isAsyncEncoder(x)` tells the two kinds apart. Your own encoder can be +async-only: implement `AsyncEncoder` and pass it where an encoder goes. + +Failures are atomic. `addNodeAsync` embeds the text *before* it adds the node, +`updateNodeAsync` before it changes the properties, and `reembedStaleAsync` +encodes every stale text before it writes one, so a timeout or a 5xx leaves +the graph exactly as it was and the call can be retried. + +For bulk work, `batchSize` texts go in each request, `concurrency` (default 1: +one after another) requests run at once, and texts asked for at the same +moment share a single request. + +The synchronous interface is unchanged: `encode` / `encodeBatch` serve from a +cache and throw an `HttpEncoderError` for a text that has not been fetched, so +the prefetch workflow (`await encoder.prefetch(texts)`, then `addNode`, +`addBatch`, `similaritySearch`) still works. Every `...Async` call fills that +cache too. A bounded cache (`maxCacheEntries`) evicts the oldest vectors, so +prefetch no more than you are about to use. ## One format, six languages @@ -324,9 +402,9 @@ and `importEmbeddings(payload)` for the vectors, and ## Things that will bite you -- **Search is brute force**, sqlite-vec included: linear in the number of - nodes. This is built for an application's own knowledge graph, not as a - vector-database replacement. +- **Search is brute force** unless you opt into `ann: 'ivf'` (sqlite-vec is + exact too): linear in the number of nodes. This is built for an application's + own knowledge graph, not as a vector-database replacement. - **`updateNode` keeps the embedding current.** With `autoEmbed` on, changing an embed field re-embeds the node from its new properties, and the vector is dropped if nothing is left to embed. Updates to other fields, and graphs with @@ -337,12 +415,13 @@ and `importEmbeddings(payload)` for the vectors, and - **One embedding per node**, from one text field or a concatenation. No multi-vector nodes, no edge embeddings. - **`MockEncoder` has no semantic structure.** Use it to test plumbing, never - to judge relevance quality. For real vectors use `HttpEncoder` (below, - prefetch-then-search) or implement `EmbeddingEncoder` over the model of your - choice. + to judge relevance quality. For real vectors use `HttpEncoder` (below, with + the `...Async` methods) or implement `EmbeddingEncoder` / `AsyncEncoder` over + the model of your choice. - **`HttpEncoder` sends your texts to the endpoint you configure**, and its - `encode` only serves texts fetched beforehand - see *Embeddings from an HTTP - endpoint*. + synchronous `encode` only serves texts fetched beforehand - auto-embedding a + graph over the network is `addNodeAsync`, not `addNode`. See *Embeddings from + an HTTP endpoint*. - **Auto-embed renders typed properties.** Since ISONGraph 1.4.0 a property can be a number, a bool or null. Nulls are skipped and booleans render as `true` and `false`, so the same graph produces the same text in every port. @@ -353,8 +432,8 @@ and `importEmbeddings(payload)` for the vectors, and ## Tests ```bash -npm test # 194 tests, needs Node 24 for the SQLite half -npm run test:core # 160 tests, main and /node entries, Node 18 and up +npm test # 255 tests, needs Node 24 for the SQLite half +npm run test:core # 221 tests, main and /node entries, Node 18 and up npm run typecheck # tsc --noEmit, tests included npm run build # emits dist/, library only ``` diff --git a/isongraph-vector-ts/package.json b/isongraph-vector-ts/package.json index 0b0164f..9ef7fdd 100644 --- a/isongraph-vector-ts/package.json +++ b/isongraph-vector-ts/package.json @@ -13,7 +13,7 @@ "scripts": { "build": "tsc -p tsconfig.build.json", "test": "vitest run", - "test:core": "vitest run src/index.test.ts src/node.test.ts src/features.test.ts src/http-encoder.test.ts", + "test:core": "vitest run src/index.test.ts src/node.test.ts src/features.test.ts src/http-encoder.test.ts src/async-encoder.test.ts src/ann.test.ts", "prepublishOnly": "npm run build", "typecheck": "tsc --noEmit" }, diff --git a/isongraph-vector-ts/src/ann.test.ts b/isongraph-vector-ts/src/ann.test.ts new file mode 100644 index 0000000..bd51bf3 --- /dev/null +++ b/isongraph-vector-ts/src/ann.test.ts @@ -0,0 +1,519 @@ +/** + * The opt-in IVF approximate index. Mirrors the Python port's + * TestApproximateIndex / TestApproximateIndexInGraph: recall on clustered + * data, exactness when every cell is probed, upkeep under writes and deletes, + * filters, and that nothing about the index reaches a saved file. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import { mkdtempSync, rmSync, readFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { + EmbeddingStore, SemanticGraph, MockEncoder, EmbeddingError, graphDigest, + type NodeRef, +} from './index.js'; +import { saveGraph, loadGraph, embeddingsPathFor } from './node.js'; + +/** Recall@10 demanded on the clustered data below (4000 vectors, 40 clusters, 64 cells, nprobe 8). */ +const RECALL_FLOOR = 0.95; + +/** Small seeded generator, so a failure reproduces. */ +function rng(seed: number): () => number { + let a = seed >>> 0; + return () => { + a = (a + 0x6d2b79f5) >>> 0; + let t = a; + t = Math.imul(t ^ (t >>> 15), t | 1); + t ^= t + Math.imul(t ^ (t >>> 7), t | 61); + return ((t ^ (t >>> 14)) >>> 0) / 4294967296; + }; +} + +function gaussian(next: () => number): number { + return Math.sqrt(-2 * Math.log(1 - next())) * Math.cos(2 * Math.PI * next()); +} + +/** Clustered, roughly unit vectors and a function drawing queries from the same clusters. */ +function clustered(n: number, dim = 32, clusters = 40, seed = 1, noise = 1.0) { + const next = rng(seed); + const centers: number[][] = []; + for (let c = 0; c < clusters; c++) { + const v = Array.from({ length: dim }, () => gaussian(next)); + const norm = Math.hypot(...v); + centers.push(v.map(x => x / norm)); + } + const draw = (count: number): number[][] => + Array.from({ length: count }, () => { + const center = centers[Math.floor(next() * clusters)]; + return center.map(x => x + (noise * gaussian(next)) / Math.sqrt(dim)); + }); + return { data: draw(n), draw }; +} + +function annStore( + { n = 4000, dim = 32, seed = 1, types, ...options }: + { n?: number; dim?: number; seed?: number; types?: (i: number) => string } & Record<string, any> = {} +) { + const { data, draw } = clustered(n, dim, 40, seed); + const store = new EmbeddingStore(undefined, { + ann: 'ivf', annMinSize: 0, annNlist: 64, annNprobe: 8, ...options, + }); + store.addBatch(data.map((v, i) => [[types ? types(i) : 'doc', i] as NodeRef, `text ${i}`, v] as [NodeRef, string, number[]])); + return { store, data, draw }; +} + +const refs = (results: Array<{ nodeRef: NodeRef }>) => results.map(r => `${r.nodeRef[0]}:${r.nodeRef[1]}`); + +describe('approximate index: store', () => { + it('is off by default and exact search is untouched', () => { + const { store, data, draw } = annStore({ ann: null }); + expect(store.ann).toBeNull(); + expect(store.annStats().enabled).toBe(false); + expect(() => store.buildAnnIndex()).toThrow(/ann/); + + // identical (not merely close) to the same store with ann on and exact: true + const { store: other } = annStore(); + for (const q of draw(5)) { + const a = store.similaritySearch(q, 10); + const b = other.similaritySearch(q, 10, undefined, 0, true); + expect(a.map(r => [r.nodeRef, r.score])).toEqual(b.map(r => [r.nodeRef, r.score])); + } + expect(data.length).toBe(4000); + }); + + it('validates its arguments', () => { + expect(() => new EmbeddingStore(undefined, { ann: 'hnsw' as any })).toThrow(EmbeddingError); + expect(() => new EmbeddingStore(undefined, { annNprobe: 0 })).toThrow(/annNprobe/); + expect(() => new EmbeddingStore(undefined, { ann: 'ivf', annNlist: 0 })).toThrow(/annNlist/); + expect(() => new EmbeddingStore(undefined, { ann: 'ivf', annMinSize: -1 })).toThrow(/annMinSize/); + expect(() => new EmbeddingStore(undefined, { ann: 'ivf', annNlist: 1.5 })).toThrow(/annNlist/); + const store = new EmbeddingStore(undefined, { ann: 'ivf' }); + expect(() => { store.annNprobe = 0; }).toThrow(EmbeddingError); + store.annNprobe = 5; + expect(store.annNprobe).toBe(5); + store.annNprobe = null; + expect(store.annNprobe).toBeNull(); + }); + + it('reaches the recall floor on clustered data and reports it', () => { + const { store, draw } = annStore(); + const report = store.recallCheck(draw(60), 10); + expect(report.approximate).toBe(true); + expect(report.nlist).toBe(64); + expect(report.nprobe).toBe(8); + expect(report.queries).toBe(60); + expect(report.k).toBe(10); + expect(report.recall).toBeGreaterThanOrEqual(RECALL_FLOOR); + expect(report.minRecall).toBeLessThanOrEqual(report.recall); + expect(report.minRecall).toBeGreaterThanOrEqual(0); + expect(report.annMs).toBeGreaterThanOrEqual(0); + expect(report.exactMs).toBeGreaterThanOrEqual(0); + const stats = store.annStats(); + expect(stats.active).toBe(true); + expect(stats.nlist).toBe(64); + expect(stats.trainedN).toBe(4000); + expect(stats.indexBytes).toBeGreaterThan(0); + expect(stats.searches.approximate).toBeGreaterThanOrEqual(60); + }); + + it('returns the exact cosine score of each node, best first', () => { + const { store, draw } = annStore(); + for (const q of draw(10)) { + const truth = new Map(store.similaritySearch(q, 100, undefined, 0, true).map(r => [`${r.nodeRef}`, r.score])); + const got = store.similaritySearch(q, 10); + for (const r of got) if (truth.has(`${r.nodeRef}`)) expect(r.score).toBe(truth.get(`${r.nodeRef}`)); + const scores = got.map(r => r.score); + expect(scores).toEqual([...scores].sort((a, b) => b - a)); + expect(got.every(r => typeof r.text === 'string')).toBe(true); + } + expect(store.annStats().searches.approximate).toBe(10); // the index answered, not the scan + }); + + it('probing every cell is the exact scan', () => { + const { store, data, draw } = annStore({ annNprobe: 64 }); // nprobe == nlist + const reference = new EmbeddingStore(); + reference.addBatch(data.map((v, i) => [['doc', i] as NodeRef, `text ${i}`, v] as [NodeRef, string, number[]])); + for (const q of draw(8)) { + expect(store.similaritySearch(q, 10)).toEqual(reference.similaritySearch(q, 10)); + } + expect(store.recallCheck(draw(5)).approximate).toBe(false); + expect(store.annStats().searches.approximate).toBe(0); + }); + + it('stays exact under annMinSize, and the default minimum is 50000', () => { + const { store, draw } = annStore({ n: 500, annMinSize: 1000 }); + const q = draw(1)[0]; + expect(refs(store.similaritySearch(q, 10))).toEqual(refs(store.similaritySearch(q, 10, undefined, 0, true))); + expect(store.annStats().active).toBe(false); + expect(store.recallCheck([q]).approximate).toBe(false); + + const dflt = new EmbeddingStore(undefined, { ann: 'ivf' }); + expect(dflt.annStats().minSize).toBe(50000); + expect(dflt.annStats().mode).toBe('ivf'); + dflt.addBatch(clustered(2000).data.map((v, i) => [['doc', i] as NodeRef, '', v] as [NodeRef, string, number[]])); + dflt.similaritySearch(clustered(1).data[0], 5); + expect(dflt.annStats().active).toBe(false); + }); + + it('chooses nlist and nprobe automatically', () => { + const { store, draw } = annStore({ n: 2500, annNlist: undefined, annNprobe: undefined }); + store.similaritySearch(draw(1)[0], 5); + const stats = store.annStats(); + expect(stats.nlist).toBe(200); // 4 * sqrt(2500) + expect(stats.nprobe).toBe(10); // 5% of the cells + expect(stats.active).toBe(true); + + const mid = annStore({ n: 100, annNlist: undefined, annNprobe: undefined }); + mid.store.similaritySearch(mid.draw(1)[0], 5); + expect(mid.store.annStats().nlist).toBe(40); // 4 * sqrt(100) + expect(mid.store.annStats().nprobe).toBe(2); + + const tiny = annStore({ n: 9, annNlist: undefined, annNprobe: undefined }); + tiny.store.similaritySearch(tiny.draw(1)[0], 5); + expect(tiny.store.annStats().nlist).toBe(9); // at least 16, but never more cells than vectors + expect(tiny.store.annStats().nprobe).toBe(1); + }); + + it('topK null and exact: true scan everything; topK 0 is empty', () => { + const { store, draw } = annStore(); + const q = draw(1)[0]; + const before = store.annStats().searches.approximate; + expect(store.similaritySearch(q, null, undefined, -1).length).toBe(4000); + store.similaritySearch(q, 5, undefined, 0, true); + expect(store.annStats().searches.approximate).toBe(before); + expect(store.similaritySearch(q, 0)).toEqual([]); + }); + + it('a zero query falls back to the exact scan', () => { + const { store } = annStore(); + const res = store.similaritySearch(new Array(32).fill(0), 5, undefined, 0); + expect(res.map(r => r.score)).toEqual([0, 0, 0, 0, 0]); + expect(store.annStats().searches).toMatchObject({ approximate: 0, exact: 1 }); + }); + + it('shows writes immediately, and an update moves the row', () => { + const { store, draw } = annStore(); + store.buildAnnIndex(); + const fresh = draw(1)[0]; + store.add(['new', 1], 'fresh', fresh); + const top = store.similaritySearch(fresh, 3); + expect(top[0].nodeRef).toEqual(['new', 1]); + expect(top[0].score).toBeCloseTo(1, 6); + expect(store.annStats().pending).toBe(1); + + const target = draw(1)[0]; + store.add(['doc', 7], 'moved', target); + const hit = store.similaritySearch(target, 1)[0]; + expect(hit.nodeRef).toEqual(['doc', 7]); + expect(hit.score).toBeCloseTo(1, 6); + for (const q of [target, fresh]) { + const res = store.similaritySearch(q, 400, undefined, -1); + expect(new Set(refs(res)).size).toBe(res.length); // no stale second entry + const truth = new Map(store.similaritySearch(q, null, undefined, -1).map(r => [`${r.nodeRef}`, r.score])); + for (const r of res) expect(r.score).toBeCloseTo(truth.get(`${r.nodeRef}`)!, 9); + } + }); + + it('never returns a deleted node, follows compaction without retraining', () => { + const { store, draw } = annStore(); + store.buildAnnIndex(); + const gone = new Set<string>(); + for (let i = 0; i < 4000; i += 3) { // a third: forces compaction + store.remove(['doc', i]); + gone.add(`doc:${i}`); + } + const builds = store.annStats().searches.builds; + for (const q of draw(15)) { + expect(refs(store.similaritySearch(q, 10)).filter(r => gone.has(r))).toEqual([]); + // a tombstone would score 0, which beats any negative cosine + expect(refs(store.similaritySearch(q, 250, undefined, -1)).filter(r => gone.has(r))).toEqual([]); + } + expect(store.recallCheck(draw(40)).recall).toBeGreaterThanOrEqual(RECALL_FLOOR); + expect(store.annStats().searches.builds).toBe(builds); // followed, not retrained + expect(store.count()).toBe(4000 - gone.size); + }); + + it('folds pending rows into cells and retrains after heavy churn', () => { + const { store, draw } = annStore({ n: 3000 }); + store.buildAnnIndex(); + const builds = store.annStats().searches.builds; + const extra = draw(700); + store.addBatch(extra.map((v, i) => [['late', i] as NodeRef, '', v] as [NodeRef, string, number[]])); + expect(store.annStats().pending).toBe(700); + store.similaritySearch(extra[0], 5); // >= 512 pending: folded + let stats = store.annStats(); + expect(stats.pending).toBe(0); + expect(stats.searches.builds).toBe(builds); + + const more = draw(1200); + store.addBatch(more.map((v, i) => [['later', i] as NodeRef, '', v] as [NodeRef, string, number[]])); + store.similaritySearch(more[0], 5); // churn 1900 > 50% of 3000 + stats = store.annStats(); + expect(stats.searches.builds).toBe(builds + 1); + expect(stats.trainedN).toBe(4900); + expect(store.recallCheck(draw(30)).recall).toBeGreaterThanOrEqual(RECALL_FLOOR); + }); + + it('drops the index with a cleared store, and rebuilds for a new dimension', () => { + const { store, data } = annStore({ n: 1500, annNlist: 16, annNprobe: 4 }); + store.buildAnnIndex(); + store.clear(); + expect(store.similaritySearch(data[0], 5)).toEqual([]); + expect(store.annStats().active).toBe(false); + store.clear(true); + const small = clustered(1200, 8, 12, 3); + store.addBatch(small.data.map((v, i) => [['doc', i] as NodeRef, '', v] as [NodeRef, string, number[]])); + const res = store.similaritySearch(small.data[5], 5); + expect(res[0].nodeRef).toEqual(['doc', 5]); + expect(store.annStats().active).toBe(true); + expect(store.dimension).toBe(8); + }); + + it('applies nodeType, widening the probe for a rare type', () => { + const { store, draw } = annStore({ types: i => (i % 200 === 0 ? 'rare' : 'common'), annNprobe: 2 }); + const q = draw(1)[0]; + const common = store.similaritySearch(q, 10, 'common'); + expect(common).toHaveLength(10); + expect(new Set(common.map(r => r.nodeRef[0]))).toEqual(new Set(['common'])); + // 20 rare rows in the whole store and a topK of 15: two cells cannot hold them + const widenedBefore = store.annStats().searches.widened; + const rare = store.similaritySearch(q, 15, 'rare', -1); + expect(rare).toHaveLength(15); + expect(new Set(rare.map(r => r.nodeRef[0]))).toEqual(new Set(['rare'])); + const counts = store.annStats().searches; + expect(counts.widened > widenedBefore || counts.exact > 0).toBe(true); + expect(store.similaritySearch(q, 5, 'nope')).toEqual([]); + }); + + it('respects the threshold', () => { + const { store, draw } = annStore(); + const q = draw(1)[0]; + const best = store.similaritySearch(q, 1, undefined, 0, true)[0].score; + const res = store.similaritySearch(q, 10, undefined, best - 1e-3); + expect(res.every(r => r.score >= best - 1e-3)).toBe(true); + expect(store.similaritySearch(q, 10, undefined, 2)).toEqual([]); + const cut = store.similaritySearch(q, 10, undefined, 0, true)[4].score; + const got = store.similaritySearch(q, 10, undefined, cut); + const want = store.similaritySearch(q, 10, undefined, cut, true); + expect(new Set(refs(got))).toEqual(new Set(refs(want))); + // a threshold that plenty clear is applied inside the probed cells: no scan needed + const before = store.annStats().searches.approximate; + const floor = store.similaritySearch(q, 30, undefined, 0, true)[29].score; + const above = store.similaritySearch(q, 10, undefined, floor); + expect(above).toHaveLength(10); + expect(above.every(r => r.score >= floor)).toBe(true); + expect(store.annStats().searches.approximate).toBe(before + 1); + }); + + it('breaks ties in insertion order, like the exact scan', () => { + const store = new EmbeddingStore(undefined, { ann: 'ivf', annMinSize: 0, annNlist: 4, annNprobe: 2 }); + const same = [1, 0.5, 0.25, 0]; + store.addBatch([...Array.from({ length: 40 }, (_, i) => [['t', i] as NodeRef, '', same] as [NodeRef, string, number[]]), + [['x', 1], '', [0, 1, 0, 0]], [['x', 2], '', [0, 0, 1, 0]], [['x', 3], '', [0, 0, 0, 1]], [['x', 4], '', [-1, 0, 0, 0]]]); + expect(refs(store.similaritySearch(same, 5))).toEqual(['t:0', 't:1', 't:2', 't:3', 't:4']); + expect(store.annStats().searches.approximate).toBe(1); + }); + + it('recallCheck takes one text-free query, a list, or a typed array', () => { + const { store, data, draw } = annStore(); + const q = draw(3); + expect(store.recallCheck(q[0]).queries).toBe(1); // one flat vector + expect(store.recallCheck(q).queries).toBe(3); // a list of them + expect(store.recallCheck(new Float32Array(q[0])).queries).toBe(1); + const off = new EmbeddingStore(); + off.addBatch(data.slice(0, 100).map((v, i) => [['doc', i] as NodeRef, '', v] as [NodeRef, string, number[]])); + const r = off.recallCheck(q); + expect(r.recall).toBe(1); + expect(r.approximate).toBe(false); + expect(() => store.recallCheck(q, 0)).toThrow(/k/); + }); + + it('recallCheck takes text queries through the encoder', async () => { + const encoder = new MockEncoder(32); + const store = new EmbeddingStore(encoder, { ann: 'ivf', annMinSize: 0, annNlist: 16, annNprobe: 16 }); + store.addBatch(Array.from({ length: 300 }, (_, i) => [['doc', i] as NodeRef, `word${i} other${i % 7}`] as [NodeRef, string])); + expect(store.recallCheck('word5 other5').queries).toBe(1); + expect((await store.recallCheckAsync(['word5', 'word9'])).queries).toBe(2); + }); + + it('is deterministic for a seed', () => { + const run = (annSeed: number) => { + const { store, draw } = annStore({ annSeed }); + const out = draw(20).map(q => refs(store.similaritySearch(q, 10))); + expect(store.annStats().searches.approximate).toBe(20); + return out; + }; + expect(run(0)).toEqual(run(0)); + expect(run(7)).toEqual(run(7)); + }); + + it('survives an import and keeps every row findable', () => { + const { store, draw } = annStore({ n: 1200, annNlist: 16, annNprobe: 4 }); + store.buildAnnIndex(); + const q = draw(1)[0]; + const payload = JSON.parse(JSON.stringify(store.exportEmbeddings())); + payload.embeddings.push({ type: 'imp', id: 1, id_type: 'int', text: 'imported', vector: q }); + payload.count += 1; + store.importEmbeddings(payload); + expect(store.similaritySearch(q, 1)[0].nodeRef).toEqual(['imp', 1]); + const other = new EmbeddingStore(undefined, { ann: 'ivf', annMinSize: 0, annNlist: 16, annNprobe: 4 }); + other.similaritySearch(q, 1); // empty: nothing built, no error + other.importEmbeddings(payload, true); + expect(other.similaritySearch(q, 1)[0].nodeRef).toEqual(['imp', 1]); + expect(other.annStats().active).toBe(true); + }); + + it('never changes what is exported', () => { + const { store, data } = annStore({ n: 800, annNlist: 16, annNprobe: 4 }); + const plain = new EmbeddingStore(); + plain.addBatch(data.slice(0, 800).map((v, i) => [['doc', i] as NodeRef, `text ${i}`, v] as [NodeRef, string, number[]])); + store.buildAnnIndex(); + store.similaritySearch(data[0], 5); + expect(JSON.stringify(store.exportEmbeddings())).toBe(JSON.stringify(plain.exportEmbeddings())); + }); + + it('survives writes and searches interleaved (single thread, async)', async () => { + const { store, draw } = annStore({ n: 3000, annNlist: 32, annNprobe: 4 }); + const extra = draw(600); + const queries = draw(40); + await Promise.all([ + (async () => { + for (let i = 0; i < 100; i++) { + store.add(['w', i], '', extra[i % 600]); + if (i % 4 === 0) store.remove(['doc', i]); + if (i % 20 === 0) store.buildAnnIndex(); + await Promise.resolve(); + } + })(), + (async () => { + for (let i = 0; i < 120; i++) { + const res = await store.similaritySearchAsync(queries[i % 40], 8, undefined, -1); + expect(res).toHaveLength(8); + expect(new Set(refs(res)).size).toBe(8); + const scores = res.map(r => r.score); + expect(scores).toEqual([...scores].sort((a, b) => b - a)); + } + })(), + ]); + expect(store.recallCheck(queries.slice(0, 20)).recall).toBeGreaterThanOrEqual(RECALL_FLOOR); + }); +}); + +describe('approximate index: in a SemanticGraph', () => { + function graph(n = 3000, dim = 32, extra: Record<string, any> = {}) { + const { data, draw } = clustered(n, dim); + const g = new SemanticGraph({ + name: 'ann', encoder: new MockEncoder(dim), autoEmbed: false, + ann: 'ivf', annMinSize: 0, annNlist: 48, annNprobe: 6, ...extra, + }); + for (let i = 0; i < n; i++) g.addNode('item', i, { group: i % 50 }); + g.embeddingStore.addBatch(data.map((v, i) => [['item', i] as NodeRef, `item ${i}`, v] as [NodeRef, string, number[]])); + return { g, data, draw }; + } + + it('passes the options to its store, and recallCheck through', () => { + const { g, draw } = graph(); + const store = g.embeddingStore; + expect(store.ann).toBe('ivf'); + expect(store.annNprobe).toBe(6); + expect(g.similaritySearch(draw(1)[0], 5)).toHaveLength(5); + expect(store.annStats().active).toBe(true); + expect(g.recallCheck(draw(20)).recall).toBeGreaterThanOrEqual(RECALL_FLOOR); + expect(new SemanticGraph().embeddingStore.ann).toBeNull(); + }); + + it('exact: true skips the index, and similarToNode works', () => { + const { g, draw } = graph(); + const q = draw(1)[0]; + const before = g.embeddingStore.annStats().searches.approximate; + g.similaritySearch(q, 5, undefined, 0, undefined, undefined, true); + g.similaritySearch(q, 5, undefined, 0, () => true, undefined, true); + expect(g.embeddingStore.annStats().searches.approximate).toBe(before); + const near = g.similarToNode(['item', 3], 5); + expect(near).toHaveLength(5); + expect(refs(near)).not.toContain('item:3'); + }); + + it('a selective nodeFilter still gets topK results', () => { + const { g, draw } = graph(); + const q = draw(1)[0]; + const group7 = (n: any) => n.properties.group === 7; // 1 node in 50 + const res = g.similaritySearch(q, 10, undefined, 0, group7); + expect(res).toHaveLength(10); + expect(res.every(r => g.getNodeByRef(r.nodeRef).properties.group === 7)).toBe(true); + const want = g.similaritySearch(q, 10, undefined, 0, group7, undefined, true); + const overlap = refs(res).filter(r => refs(want).includes(r)).length; + expect(overlap).toBeGreaterThanOrEqual(7); + const scores = res.map(r => r.score); + expect(scores).toEqual([...scores].sort((a, b) => b - a)); + expect(g.embeddingStore.annStats().searches.approximate).toBeGreaterThanOrEqual(1); // probed cells, not a scan + // a filter nothing passes terminates with nothing + expect(g.similaritySearch(q, 10, undefined, 0, () => false)).toEqual([]); + // a lenient one agrees closely with the scan + const even = (n: any) => n.properties.group % 2 === 0; + const got = g.similaritySearch(q, 10, undefined, 0, even); + const ref = g.similaritySearch(q, 10, undefined, 0, even, undefined, true); + expect(refs(got).filter(r => refs(ref).includes(r)).length).toBeGreaterThanOrEqual(8); + // nodeTypes goes through the same gate + expect(g.similaritySearch(q, 5, undefined, 0, undefined, ['item'])).toHaveLength(5); + expect(g.similaritySearch(q, 5, undefined, 0, undefined, ['other'])).toEqual([]); + }); + + it('semanticMultiHop takes its seeds from the index, blend > 0 does not', () => { + const { g, draw } = graph(2000); + for (let i = 0; i < 1999; i++) g.addEdge('NEXT', ['item', i], ['item', i + 1]); + const q = draw(1)[0]; + const out = g.semanticMultiHop(q, { relType: 'NEXT', maxHops: 2, topKSeeds: 3, threshold: -1 }); + expect(out.length).toBeGreaterThan(0); + expect(out[0].hopCount).toBe(0); + const used = g.embeddingStore.annStats().searches.approximate; + expect(used).toBeGreaterThanOrEqual(1); + g.semanticMultiHop(q, { relType: 'NEXT', maxHops: 1, blend: 0.3, threshold: -1 }); + expect(g.embeddingStore.annStats().searches.approximate).toBe(used + 1); // the seed search only; scoreMap is a full scan + }); + + it('async text search honours exact', async () => { + const encoder = new MockEncoder(32); + const g = new SemanticGraph({ encoder, ann: 'ivf', annMinSize: 0, annNlist: 16, annNprobe: 4 }); + for (let i = 0; i < 400; i++) g.addNode('doc', i, { name: `entry number ${i} about topic ${i % 9}` }); + const before = g.embeddingStore.annStats().searches.approximate; + const hits = await g.similaritySearchAsync('entry number 12 about topic 3', 3); + expect(hits).toHaveLength(3); + const used = g.embeddingStore.annStats().searches.approximate; + expect(used).toBe(before + 1); + await g.similaritySearchAsync('entry number 12 about topic 3', 3, undefined, 0, undefined, undefined, true); + expect(g.embeddingStore.annStats().searches.approximate).toBe(used); + }); + + describe('files', () => { + const dirs: string[] = []; + afterEach(() => { while (dirs.length) rmSync(dirs.pop()!, { recursive: true, force: true }); }); + + it('loadGraph passes ann along and rebuilds lazily; nothing about it is saved', () => { + const dir = mkdtempSync(join(tmpdir(), 'isongraph-vector-ann-')); + dirs.push(dir); + const { g, draw } = graph(1500); + const path = join(dir, 'g.ison'); + saveGraph(g, path); + + const plain = new SemanticGraph({ name: 'ann', encoder: new MockEncoder(32), autoEmbed: false }); + for (let i = 0; i < 1500; i++) plain.addNode('item', i, { group: i % 50 }); + plain.embeddingStore.importEmbeddings(g.embeddingStore.exportEmbeddings()); + const plainPath = join(dir, 'plain.ison'); + saveGraph(plain, plainPath); + // same graph, same vectors: the ann graph's files are exactly the plain graph's + expect(readFileSync(path, 'utf8')).toBe(readFileSync(plainPath, 'utf8')); + expect(readFileSync(embeddingsPathFor(path), 'utf8')).toBe(readFileSync(embeddingsPathFor(plainPath), 'utf8')); + expect(readFileSync(embeddingsPathFor(path), 'utf8')).not.toMatch(/ivf|nlist|centroid/i); + expect(graphDigest(g)).toBe(graphDigest(plain)); + + const loaded = loadGraph(path, { + encoder: g.encoder!, ann: 'ivf', annMinSize: 0, annNlist: 32, annNprobe: 4, + }); + expect(loaded.embeddingStore.ann).toBe('ivf'); + expect(loaded.embeddingStore.annStats().active).toBe(false); // nothing persisted + expect(loaded.similaritySearch(draw(1)[0], 5)).toHaveLength(5); + expect(loaded.embeddingStore.annStats().active).toBe(true); + }); + }); +}); diff --git a/isongraph-vector-ts/src/async-encoder.test.ts b/isongraph-vector-ts/src/async-encoder.test.ts new file mode 100644 index 0000000..85c1eff --- /dev/null +++ b/isongraph-vector-ts/src/async-encoder.test.ts @@ -0,0 +1,520 @@ +/** + * The asynchronous encoder path: AsyncEncoder, the *Async methods on the store + * and the graph, and HttpEncoder driving auto-embedding with no prefetch. + * Everything runs against a local node:http server - never the real network. + */ + +import { describe, it, expect, afterEach, vi } from 'vitest'; +import { createServer } from 'node:http'; +import { + SemanticGraph, EmbeddingStore, MockEncoder, HttpEncoder, HttpEncoderError, EmbeddingError, + DimensionMismatchError, ModelMismatchError, NoEncoderError, Direction, isAsyncEncoder, + type AsyncEncoder, +} from './index.js'; + +const servers: any[] = []; +afterEach(async () => { + vi.restoreAllMocks(); + while (servers.length) { + const server = servers.pop(); + server.closeAllConnections(); + await new Promise<void>(resolve => server.close(() => resolve())); + } +}); + +interface Served { + base: string; + seen: any[]; + /** Most requests in flight at once. */ + peak: () => number; +} + +/** A server whose answers come from `respond(request, n)`; may delay with `wait` ms or `hang`. */ +async function serve(respond: (request: any, n: number) => any): Promise<Served> { + const seen: any[] = []; + let inflight = 0; + let peak = 0; + const server = createServer((req, res) => { + const chunks: Buffer[] = []; + req.on('data', (chunk: Buffer) => chunks.push(chunk)); + req.on('end', () => { + const raw = Buffer.concat(chunks).toString('utf8'); + const request = { + method: req.method, url: req.url, headers: req.headers, + body: raw ? JSON.parse(raw) : null, + }; + const n = seen.length; + seen.push(request); + const answer = respond(request, n); + if (answer.hang) return; + inflight++; + peak = Math.max(peak, inflight); + setTimeout(() => { + inflight--; + res.writeHead(answer.status ?? 200, { 'content-type': 'application/json', ...(answer.headers ?? {}) }); + res.end(typeof answer.body === 'string' ? answer.body : JSON.stringify(answer.body)); + }, answer.wait ?? 0); + }); + }); + servers.push(server); + await new Promise<void>(resolve => server.listen(0, '127.0.0.1', () => resolve())); + const { port } = server.address() as { port: number }; + return { base: `http://127.0.0.1:${port}`, seen, peak: () => peak }; +} + +/** Topic vectors: texts that mention a topic word point along its axis. */ +function vectorOf(text: string): number[] { + const t = text.toLowerCase(); + return [ + t.includes('engineer') ? 1 : 0, + t.includes('manager') ? 1 : 0, + t.includes('design') ? 1 : 0, + 0.1, + ]; +} + +const openaiAnswer = (request: any) => ({ + body: { + data: request.body.input.map((text: string, index: number) => ({ index, embedding: vectorOf(text) })).reverse(), + }, +}); +const ollamaAnswer = (request: any) => ({ body: { embeddings: request.body.input.map(vectorOf) } }); + +const fast = { retryDelayMs: 1 }; + +describe('AsyncEncoder', () => { + it('HttpEncoder is one, MockEncoder is not', () => { + const http = new HttpEncoder({ url: 'http://localhost:1/e', model: 'm' }); + expect(isAsyncEncoder(http)).toBe(true); + expect(isAsyncEncoder(new MockEncoder(8))).toBe(false); + expect(isAsyncEncoder(null)).toBe(false); + expect(isAsyncEncoder({ encodeAsync: () => 1 })).toBe(false); + }); + + const remote = (): AsyncEncoder => ({ + dimension: 4, + modelName: 'only-async', + version: 'v9', + encodeAsync: async (text: string) => vectorOf(text), + encodeBatchAsync: async (texts: string[]) => texts.map(vectorOf), + }); + + it('a pure async encoder works with the Async methods and stamps its model', async () => { + const store = new EmbeddingStore(remote()); + expect(store.encoder).not.toBeNull(); + await store.addAsync(['p', 1], 'Alice the engineer'); + await store.addBatchAsync([[['p', 2], 'Bob the manager'], [['p', 3], 'Carol the designer']]); + + expect(store.count()).toBe(3); + expect(store.models()).toEqual({ 'only-async': 3 }); + expect(store.version).toBe('v9'); + const hits = await store.similaritySearchAsync('engineering', 1); + expect(hits[0].nodeRef).toEqual(['p', 1]); + }); + + it('its synchronous face says to use the Async method', async () => { + const store = new EmbeddingStore(remote()); + expect(() => store.add(['p', 1], 'text')).toThrow(EmbeddingError); + expect(() => store.add(['p', 1], 'text')).toThrow(/Async/); + expect(() => store.similaritySearch('text')).toThrow(/Async/); + // vectors you already have never need the encoder + store.add(['p', 1], 'text', [1, 0, 0, 0]); + expect(store.similaritySearch([1, 0, 0, 0], 1).length).toBe(1); + }); + + it('works through a graph and reports stale vectors when the model changes', async () => { + const graph = new SemanticGraph({ encoder: remote() }); + await graph.addNodeAsync('person', 1, { name: 'Alice the engineer' }); + expect(graph.staleEmbeddings()).toEqual([]); + + graph.setEncoder({ ...remote(), modelName: 'newer' }); + expect(graph.staleEmbeddings()).toEqual([['person', 1]]); + expect(await graph.reembedStaleAsync()).toBe(1); + expect(graph.embeddingStore.models()).toEqual({ newer: 1 }); + }); + + it('async methods also work with a synchronous encoder', async () => { + const encoder = new MockEncoder(16); + const sync = new SemanticGraph({ encoder }); + sync.addNode('person', 1, { name: 'Alice' }); + const viaAsync = new SemanticGraph({ encoder }); + await viaAsync.addNodeAsync('person', 1, { name: 'Alice' }); + expect(viaAsync.getEmbedding(['person', 1])!.vector).toEqual(sync.getEmbedding(['person', 1])!.vector); + const hits = await viaAsync.similaritySearchAsync('Alice', 1); + expect(hits[0].score).toBeCloseTo(1, 6); + }); + + it('refuses a text query with no encoder', async () => { + const store = new EmbeddingStore(); + await expect(store.similaritySearchAsync('text')).rejects.toThrow(NoEncoderError); + await expect(store.addAsync(['p', 1], 'text')).rejects.toThrow(NoEncoderError); + store.add(['p', 1], 'x', [1, 0]); + expect(await store.similaritySearchAsync([1, 0], 1)).toHaveLength(1); + }); + + it('rejects an encoder that returns the wrong number of vectors', async () => { + const broken: AsyncEncoder = { + dimension: 4, encodeAsync: async () => [1, 0, 0, 0], encodeBatchAsync: async () => [[1, 0, 0, 0]], + }; + const store = new EmbeddingStore(broken); + await expect(store.addBatchAsync([[['p', 1], 'a'], [['p', 2], 'b']])).rejects.toThrow(/1 vectors for 2 texts/); + expect(store.count()).toBe(0); + }); +}); + +describe('SemanticGraph auto-embed over HTTP (openai shape)', () => { + it('addNodeAsync embeds with no prefetch; sync addNode still needs one', async () => { + const { base, seen } = await serve(openaiAnswer); + const encoder = new HttpEncoder({ url: `${base}/v1/embeddings`, model: 'remote-4', apiKey: 'sk-test-key' }); + const graph = new SemanticGraph({ encoder }); + + const node = await graph.addNodeAsync('person', 1, { name: 'Alice', description: 'an engineer' }); + expect(node.ref).toEqual(['person', 1]); + expect(seen).toHaveLength(1); + expect(seen[0].body.input).toEqual(['Alice an engineer']); + expect(seen[0].headers.authorization).toBe('Bearer sk-test-key'); + const record = graph.getEmbedding(['person', 1])!; + expect(record.vector).toEqual(vectorOf('Alice an engineer')); + expect(record.model).toBe('remote-4'); + expect(graph.embeddingStore.dimension).toBe(4); + + // the synchronous path is unchanged: it needs the text prefetched + expect(() => graph.addNode('person', 2, { name: 'Bob the manager' })).toThrow(HttpEncoderError); + await encoder.prefetch(['Bob the manager']); + graph.addNode('person', 3, { name: 'Bob the manager' }); + expect(graph.getEmbedding(['person', 3])).not.toBeNull(); + }); + + it('a node with nothing to embed makes no request', async () => { + const { base, seen } = await serve(openaiAnswer); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${base}/e`, model: 'm' }) }); + await graph.addNodeAsync('person', 1, { age: 30 }); + expect(seen).toHaveLength(0); + expect(graph.getEmbedding(['person', 1])).toBeNull(); + expect(graph.getNode('person', 1).properties.age).toBe(30); + }); + + it('autoEmbed off leaves the node unembedded; addNodeWithEmbedAsync embeds the given text', async () => { + const { base, seen } = await serve(openaiAnswer); + const encoder = new HttpEncoder({ url: `${base}/e`, model: 'm' }); + const off = new SemanticGraph({ encoder, autoEmbed: false }); + await off.addNodeAsync('person', 1, { name: 'Alice' }); + expect(seen).toHaveLength(0); + await off.addNodeWithEmbedAsync('person', 2, { name: 'Bob' }, 'a manager'); + expect(seen[0].body.input).toEqual(['a manager']); + expect(off.getEmbedding(['person', 2])!.text).toBe('a manager'); + await off.addNodeWithEmbedAsync('person', 3, { name: 'Carol the designer' }); // autoEmbed off -> no embed text, no vector + expect(off.getEmbedding(['person', 3])).toBeNull(); + }); + + it('updateNodeAsync re-embeds on an embed-field change and only then', async () => { + const { base, seen } = await serve(openaiAnswer); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${base}/e`, model: 'm' }) }); + await graph.addNodeAsync('person', 1, { name: 'Alice the engineer', age: 30 }); + expect(seen).toHaveLength(1); + + await graph.updateNodeAsync('person', 1, { age: 31 }); + expect(seen).toHaveLength(1); // no embed field touched + + await graph.updateNodeAsync('person', 1, { name: 'Alice the manager' }); + expect(seen).toHaveLength(2); + expect(graph.getEmbedding(['person', 1])!.text).toBe('Alice the manager'); + expect(graph.getEmbedding(['person', 1])!.vector).toEqual(vectorOf('manager')); + + await graph.updateNodeAsync('person', 1, { name: null }); // nothing left to embed + expect(graph.getEmbedding(['person', 1])).toBeNull(); + expect(seen).toHaveLength(2); + }); + + it('searches by text over the network and ranks by meaning', async () => { + const { base, seen } = await serve(openaiAnswer); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${base}/e`, model: 'm' }) }); + await graph.embedAllNodesAsync(); // empty graph: nothing to do + for (const [id, name] of [[1, 'Alice the engineer'], [2, 'Bob the manager'], [3, 'Carol the designer']] as const) { + await graph.addNodeAsync('person', id, { name }); + } + const before = seen.length; + const hits = await graph.similaritySearchAsync('who is the engineering lead', 2, undefined, 0); + expect(seen.length).toBe(before + 1); + expect(seen[before].body.input).toEqual(['who is the engineering lead']); + expect(hits[0].nodeRef).toEqual(['person', 1]); + + const filtered = await graph.similaritySearchAsync('engineering', 5, undefined, -1, n => n.id !== 1); + expect(filtered.every(r => r.nodeRef[1] !== 1)).toBe(true); + }); + + it('semanticMultiHopAsync, semanticPathAsync and semanticSubgraphAsync encode the query once', async () => { + const { base, seen } = await serve(openaiAnswer); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${base}/e`, model: 'm' }) }); + await graph.addNodeAsync('person', 1, { name: 'Alice the engineer' }); + await graph.addNodeAsync('person', 2, { name: 'Bob the manager' }); + await graph.addNodeAsync('person', 3, { name: 'Carol the designer' }); + graph.addEdge('KNOWS', ['person', 1], ['person', 2]); + graph.addEdge('KNOWS', ['person', 2], ['person', 3]); + + const mark = seen.length; + const hops = await graph.semanticMultiHopAsync('engineering lead', { relType: 'KNOWS', maxHops: 2, threshold: 0.5 }); + expect(seen.length).toBe(mark + 1); + expect(hops.map(r => r.nodeRef[1])).toContain(2); + + const path = await graph.semanticPathAsync('an engineer', ['person', 3], { relType: 'KNOWS', threshold: 0.5 }); + expect(path!.path.map(r => r[1])).toEqual([1, 2, 3]); + expect(seen.length).toBe(mark + 2); + + const sub = await graph.semanticSubgraphAsync('the engineers', { relType: 'KNOWS', maxHops: 1, threshold: 0.5, direction: Direction.OUT }); + expect(sub.nodeCount()).toBe(2); + expect(seen.length).toBe(mark + 3); + + // the sync versions still take a vector, no encoder involved + expect(graph.semanticMultiHop(vectorOf('engineer'), { relType: 'KNOWS', threshold: 0.5 }).length).toBeGreaterThan(0); + expect(seen.length).toBe(mark + 3); + }); + + it('similarToNode needs no encoder: a plain sync call keeps working', async () => { + const { base } = await serve(openaiAnswer); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${base}/e`, model: 'm' }) }); + await graph.addNodeAsync('person', 1, { name: 'Alice the engineer' }); + await graph.addNodeAsync('person', 2, { name: 'Bob the engineer' }); + expect(graph.similarToNode(['person', 1], 1)[0].nodeRef).toEqual(['person', 2]); + }); +}); + +describe('ollama shape', () => { + it('auto-embeds and searches', async () => { + const { base, seen } = await serve(ollamaAnswer); + const encoder = new HttpEncoder({ url: `${base}/api/embed`, model: 'nomic', api: 'ollama' }); + const graph = new SemanticGraph({ encoder }); + await graph.addNodeAsync('person', 1, { name: 'Alice the engineer' }); + await graph.addNodeAsync('person', 2, { name: 'Bob the manager' }); + expect(seen[0].url).toBe('/api/embed'); + expect(seen[0].body).toEqual({ model: 'nomic', input: ['Alice the engineer'] }); + const hits = await graph.similaritySearchAsync('engineer', 1); + expect(hits[0].nodeRef).toEqual(['person', 1]); + expect(graph.embeddingStore.models()).toEqual({ nomic: 2 }); + }); + + it('a vector of the wrong length is a DimensionMismatchError and changes nothing', async () => { + const { base } = await serve(ollamaAnswer); + const encoder = new HttpEncoder({ url: `${base}/api/embed`, model: 'nomic', api: 'ollama', dimension: 8 }); + const graph = new SemanticGraph({ encoder }); + await expect(graph.addNodeAsync('person', 1, { name: 'Alice' })).rejects.toThrow(DimensionMismatchError); + expect(graph.nodeCount()).toBe(0); + }); +}); + +describe('batching and concurrency', () => { + it('embedAllNodesAsync sends the texts in batchSize requests', async () => { + const { base, seen } = await serve(openaiAnswer); + const encoder = new HttpEncoder({ url: `${base}/e`, model: 'm', batchSize: 40 }); + const graph = new SemanticGraph({ encoder, autoEmbed: false }); + for (let i = 0; i < 100; i++) graph.addNode('doc', i, { name: `document ${i} engineer` }); + + expect(await graph.embedAllNodesAsync()).toBe(100); + expect(seen.map(r => r.body.input.length)).toEqual([40, 40, 20]); + expect(graph.embeddingStore.count()).toBe(100); + + // skipExisting: nothing is requested for nodes that already have a vector + expect(await graph.embedAllNodesAsync(undefined, true)).toBe(0); + expect(seen).toHaveLength(3); + }); + + it('concurrency runs chunk requests in parallel, up to the limit', async () => { + const { base, seen, peak } = await serve((request: any) => ({ ...openaiAnswer(request), wait: 40 })); + const encoder = new HttpEncoder({ url: `${base}/e`, model: 'm', batchSize: 1, concurrency: 3 }); + const vectors = await encoder.encodeBatchAsync(['a', 'b', 'c', 'd', 'e', 'f', 'g']); + expect(vectors).toEqual(['a', 'b', 'c', 'd', 'e', 'f', 'g'].map(vectorOf)); + expect(seen).toHaveLength(7); + expect(peak()).toBe(3); + + const serial = await serve((request: any) => ({ ...openaiAnswer(request), wait: 10 })); + await new HttpEncoder({ url: `${serial.base}/e`, model: 'm', batchSize: 1 }).encodeBatchAsync(['a', 'b', 'c']); + expect(serial.peak()).toBe(1); // default stays sequential + }); + + it('validates concurrency', () => { + expect(() => new HttpEncoder({ url: 'http://localhost:1/e', model: 'm', concurrency: 0 })).toThrow(/concurrency/); + expect(() => new HttpEncoder({ url: 'http://localhost:1/e', model: 'm', concurrency: 1.5 })).toThrow(/concurrency/); + }); + + it('identical texts requested at the same moment share one request', async () => { + const { base, seen } = await serve((request: any) => ({ ...openaiAnswer(request), wait: 30 })); + const encoder = new HttpEncoder({ url: `${base}/e`, model: 'm' }); + const results = await Promise.all([ + encoder.encodeAsync('same engineer'), encoder.encodeAsync('same engineer'), encoder.encodeAsync('same engineer'), + ]); + expect(seen).toHaveLength(1); + expect(results[0]).toEqual(vectorOf('same engineer')); + expect(results[1]).not.toBe(results[0]); // each caller owns its copy + }); + + it('many nodes added at once all land', async () => { + const { base, seen } = await serve((request: any) => ({ ...openaiAnswer(request), wait: 5 })); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${base}/e`, model: 'm' }) }); + await Promise.all(Array.from({ length: 25 }, (_, i) => graph.addNodeAsync('doc', i, { name: `doc ${i} designer` }))); + expect(graph.nodeCount()).toBe(25); + expect(graph.embeddingStore.count()).toBe(25); + expect(seen.length).toBeGreaterThan(0); + expect((await graph.similaritySearchAsync('design', 25, undefined, 0.5)).length).toBe(25); + }); + + it('a failed request in the middle of a batch rejects the whole call and stores nothing', async () => { + const { base } = await serve((request: any, n: number) => + n === 1 ? { status: 400, body: 'bad' } : openaiAnswer(request)); + const encoder = new HttpEncoder({ url: `${base}/e`, model: 'm', batchSize: 2, ...fast }); + const graph = new SemanticGraph({ encoder, autoEmbed: false }); + for (let i = 0; i < 6; i++) graph.addNode('doc', i, { name: `doc ${i}` }); + await expect(graph.embedAllNodesAsync()).rejects.toThrow(HttpEncoderError); + expect(graph.embeddingStore.count()).toBe(0); + }); +}); + +describe('stale detection and re-embedding over the network', () => { + it('reembedStaleAsync rewrites vectors from another model, in batches', async () => { + const { base, seen } = await serve(openaiAnswer); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${base}/e`, model: 'old', batchSize: 100 }) }); + for (let i = 0; i < 5; i++) await graph.addNodeAsync('doc', i, { name: `doc ${i} engineer` }); + expect(graph.staleEmbeddings()).toEqual([]); + + graph.setEncoder(new HttpEncoder({ url: `${base}/e`, model: 'new', batchSize: 100 })); + expect(graph.staleEmbeddings()).toHaveLength(5); + const mark = seen.length; + expect(await graph.reembedStaleAsync(false, 2)).toBe(5); + expect(seen.length - mark).toBe(3); // 2 + 2 + 1 texts + expect(graph.embeddingStore.models()).toEqual({ new: 5 }); + expect(graph.staleEmbeddings()).toEqual([]); + expect(seen[mark].body.model).toBe('new'); + }); + + it('a different dimension empties the store and re-embeds everything, learning it from the answer', async () => { + const wide = await serve((request: any) => ({ + body: { data: request.body.input.map((t: string, index: number) => ({ index, embedding: [...vectorOf(t), 0.5] })) }, + })); + const { base } = await serve(openaiAnswer); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${base}/e`, model: 'four' }) }); + await graph.addNodeAsync('doc', 1, { name: 'Alice the engineer' }); + expect(graph.embeddingStore.dimension).toBe(4); + + graph.setEncoder(new HttpEncoder({ url: `${wide.base}/e`, model: 'five' })); // dimension not known yet + expect(await graph.reembedStaleAsync()).toBe(1); + expect(graph.embeddingStore.dimension).toBe(5); + expect(graph.getEmbedding(['doc', 1])!.vector).toHaveLength(5); + }); + + it('a failing re-embed leaves the old vectors alone', async () => { + const { base } = await serve((request: any, n: number) => n === 0 ? openaiAnswer(request) : { status: 400, body: 'no' }); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${base}/e`, model: 'old', ...fast }) }); + await graph.addNodeAsync('doc', 1, { name: 'Alice the engineer' }); + graph.setEncoder(new HttpEncoder({ url: `${base}/e`, model: 'new', ...fast })); + await expect(graph.reembedStaleAsync()).rejects.toThrow(HttpEncoderError); + expect(graph.getEmbedding(['doc', 1])!.model).toBe('old'); + expect(graph.staleEmbeddings()).toHaveLength(1); + }); + + it('a text query from another model follows onModelMismatch', async () => { + const { base } = await serve(openaiAnswer); + const store = new EmbeddingStore(new HttpEncoder({ url: `${base}/e`, model: 'old' }), { onModelMismatch: 'error' }); + await store.addAsync(['p', 1], 'Alice the engineer'); + store.encoder = new HttpEncoder({ url: `${base}/e`, model: 'new' }); + await expect(store.similaritySearchAsync('engineer')).rejects.toThrow(ModelMismatchError); + expect(await store.similaritySearchAsync([1, 0, 0, 0.1], 1)).toHaveLength(1); // vectors are never checked + + store.onModelMismatch = 'warn'; + const warn = vi.spyOn(console, 'warn').mockImplementation(() => {}); + await store.similaritySearchAsync('engineer'); + await store.similaritySearchAsync('engineer'); + expect(warn).toHaveBeenCalledTimes(1); + }); +}); + +describe('failures from the network', () => { + it('a server error leaves the graph as it was', async () => { + const { base } = await serve(() => ({ status: 500, body: 'boom' })); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${base}/e`, model: 'm', maxRetries: 0 }) }); + const error: any = await graph.addNodeAsync('person', 1, { name: 'Alice' }).catch((e: any) => e); + expect(error).toBeInstanceOf(HttpEncoderError); + expect(error.status).toBe(500); + expect(graph.nodeCount()).toBe(0); + expect(graph.embeddingStore.count()).toBe(0); + // ... so the call can simply be retried + await expect(graph.addNodeAsync('person', 1, { name: 'Alice' })).rejects.toThrow(HttpEncoderError); + }); + + it('updateNodeAsync that cannot embed keeps the old properties and vector', async () => { + const { base } = await serve((request: any, n: number) => n === 0 ? openaiAnswer(request) : { status: 503, body: 'down' }); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${base}/e`, model: 'm', maxRetries: 0 }) }); + await graph.addNodeAsync('person', 1, { name: 'Alice the engineer' }); + await expect(graph.updateNodeAsync('person', 1, { name: 'Alice the manager' })).rejects.toThrow(HttpEncoderError); + expect(graph.getNode('person', 1).properties.name).toBe('Alice the engineer'); + expect(graph.getEmbedding(['person', 1])!.text).toBe('Alice the engineer'); + }); + + it('retries a 429 then succeeds', async () => { + const { base, seen } = await serve((request: any, n: number) => + n === 0 ? { status: 429, headers: { 'retry-after': '0' }, body: 'slow' } : openaiAnswer(request)); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${base}/e`, model: 'm', ...fast }) }); + await graph.addNodeAsync('person', 1, { name: 'Alice' }); + expect(seen).toHaveLength(2); + expect(graph.embeddingStore.count()).toBe(1); + }); + + it('a request that never completes times out, and the node is not added', async () => { + const { base, seen } = await serve(() => ({ hang: true })); + const graph = new SemanticGraph({ + encoder: new HttpEncoder({ url: `${base}/e`, model: 'm', timeoutMs: 50, maxRetries: 1, ...fast }), + }); + const error: any = await graph.addNodeAsync('person', 1, { name: 'Alice' }).catch((e: any) => e); + expect(error).toBeInstanceOf(HttpEncoderError); + expect(error.message).toContain('timed out'); + expect(seen).toHaveLength(2); + expect(graph.nodeCount()).toBe(0); + await expect(graph.similaritySearchAsync('text')).rejects.toThrow(/timed out/); + }); + + it('a refused connection is reported, not hung', async () => { + const { base } = await serve(openaiAnswer); + const dead = base.replace(/:\d+$/, ':1'); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${dead}/e`, model: 'm', maxRetries: 0 }) }); + await expect(graph.addNodeAsync('person', 1, { name: 'Alice' })).rejects.toThrow(/failed/); + }); + + it('the api key never appears in an error from any async path', async () => { + const secret = 'sk-do-not-leak-0123456789'; + // a server that echoes the credentials back, the worst case + const { base } = await serve((request: any) => ({ status: 401, body: `rejected ${request.headers.authorization}` })); + const encoder = new HttpEncoder({ + url: `${base}/e?api_key=query-secret`, model: 'm', apiKey: secret, headers: { 'X-Api-Token': 'header-secret' }, ...fast, + }); + const graph = new SemanticGraph({ encoder }); + const errors: any[] = []; + errors.push(await graph.addNodeAsync('person', 1, { name: 'Alice' }).catch((e: any) => e)); + errors.push(await graph.similaritySearchAsync('Alice').catch((e: any) => e)); + const manual = new SemanticGraph({ encoder, autoEmbed: false }); + manual.addNode('person', 1, { name: 'Alice' }); + errors.push(await manual.embedAllNodesAsync().catch((e: any) => e)); + errors.push(await graph.embeddingStore.addBatchAsync([[['p', 1], 'x']]).catch((e: any) => e)); + for (const error of errors) { + expect(error).toBeInstanceOf(Error); + for (const leaked of [secret, 'query-secret', 'header-secret']) { + expect(String(error.message)).not.toContain(leaked); + expect(String(error.stack)).not.toContain(leaked); + } + } + expect(errors.some(e => e instanceof HttpEncoderError && e.status === 401)).toBe(true); + expect(JSON.stringify(graph.encoder)).not.toContain(secret); + }); + + it('sends the key as a bearer token and lets custom headers win', async () => { + const { base, seen } = await serve(openaiAnswer); + const graph = new SemanticGraph({ encoder: new HttpEncoder({ url: `${base}/e`, model: 'm', apiKey: 'sk-abc' }) }); + await graph.addNodeAsync('person', 1, { name: 'Alice' }); + await graph.similaritySearchAsync('Alice'); + expect(seen.every(r => r.headers.authorization === 'Bearer sk-abc')).toBe(true); + + const custom = await serve(openaiAnswer); + const g2 = new SemanticGraph({ + encoder: new HttpEncoder({ url: `${custom.base}/e`, model: 'm', apiKey: 'ignored', headers: { Authorization: 'Token zzz' } }), + }); + await g2.addNodeAsync('person', 1, { name: 'Alice' }); + expect(custom.seen[0].headers.authorization).toBe('Token zzz'); + }); +}); diff --git a/isongraph-vector-ts/src/index.ts b/isongraph-vector-ts/src/index.ts index 2cfb7f2..48c5b37 100644 --- a/isongraph-vector-ts/src/index.ts +++ b/isongraph-vector-ts/src/index.ts @@ -128,6 +128,90 @@ export interface EmbeddingEncoder { encodeBatch(texts: string[]): number[][]; } +/** + * An encoder whose work is asynchronous - typically a network call. + * + * Same metadata as {@link EmbeddingEncoder} (`modelName` stamps every vector + * it writes and decides staleness, `version` goes to the sidecar). `dimension` + * may be `NaN` until the encoder has learned it from a first answer. + * + * Hand one to a store or graph and use the `...Async` methods + * (`addAsync`, `addNodeAsync`, `similaritySearchAsync`, ...). An encoder that + * is *also* an `EmbeddingEncoder` - `HttpEncoder` is both - keeps working with + * the synchronous methods too. The `...Async` methods accept either kind. + */ +export interface AsyncEncoder { + readonly dimension: number; + modelName?: string; + version?: string; + encodeAsync(text: string): Promise<number[]>; + encodeBatchAsync(texts: string[]): Promise<number[][]>; +} + +/** True for an object with `encodeAsync` and `encodeBatchAsync` methods. */ +export function isAsyncEncoder(value: unknown): value is AsyncEncoder { + const v = value as Partial<AsyncEncoder> | null | undefined; + return ( + v !== null && v !== undefined && typeof v === 'object' && + typeof v.encodeAsync === 'function' && typeof v.encodeBatchAsync === 'function' + ); +} + +/** + * The synchronous face of an encoder that only has an asynchronous one, so the + * stores (whose `encoder` is an EmbeddingEncoder) can hold it. Its metadata is + * read live from the wrapped encoder; its synchronous methods refuse, naming + * the asynchronous ones. + */ +class AsyncOnlyEncoder implements EmbeddingEncoder, AsyncEncoder { + readonly inner: AsyncEncoder; + + constructor(inner: AsyncEncoder) { + this.inner = inner; + } + + get dimension(): number { + return this.inner.dimension; + } + + get modelName(): string | undefined { + return this.inner.modelName; + } + + get version(): string | undefined { + return this.inner.version; + } + + encode(_text: string): number[] { + throw new EmbeddingError( + 'this encoder is asynchronous: use the Async methods (addAsync, addNodeAsync, ' + + 'similaritySearchAsync, ...) or pass the vector yourself' + ); + } + + encodeBatch(texts: string[]): number[][] { + return texts.map(text => this.encode(text)); + } + + encodeAsync(text: string): Promise<number[]> { + return this.inner.encodeAsync(text); + } + + encodeBatchAsync(texts: string[]): Promise<number[][]> { + return this.inner.encodeBatchAsync(texts); + } +} + +/** What a store or graph holds: sync encoders as they are, async-only ones wrapped. */ +function adoptEncoder(encoder: EmbeddingEncoder | AsyncEncoder | null | undefined): EmbeddingEncoder | null { + if (encoder === null || encoder === undefined) return null; + const sync = encoder as Partial<EmbeddingEncoder>; + if (typeof sync.encode === 'function' && typeof sync.encodeBatch === 'function') { + return encoder as EmbeddingEncoder; + } + return isAsyncEncoder(encoder) ? new AsyncOnlyEncoder(encoder) : (encoder as unknown as EmbeddingEncoder); +} + /** * Deterministic encoder with no dependencies, for tests and offline work. * @@ -289,6 +373,12 @@ export interface HttpEncoderOptions { retryDelayMs?: number; /** Vectors kept for the synchronous `encode`; oldest dropped first. Default 100000. */ maxCacheEntries?: number; + /** + * Requests in flight at once when one call needs several batches + * (`encodeBatchAsync` of more than `batchSize` texts). Default 1: batches go + * one after another. Raise it for a server that likes parallel work. + */ + concurrency?: number; /** Replacement for `globalThis.fetch` (a proxy-aware fetch, or a test double). */ fetch?: HttpEncoderFetch; } @@ -304,20 +394,29 @@ const ERROR_SNIPPET_CHARS = 200; * **Every text you embed is sent to the configured endpoint**, including the * text of every node a graph embeds and every search query. * - * `EmbeddingEncoder` is synchronous and a network call is not, so the encoder - * works in two steps: fetch the vectors with `await encoder.prefetch(texts)` - * (or `encodeAsync` / `encodeBatchAsync`, which fetch and return them), then - * use the synchronous `encode` / `encodeBatch` the stores call. Those serve - * from a cache and throw an HttpEncoderError for a text that was not fetched - * first. That makes `SemanticGraph` auto-embedding unusable with this encoder - * unless the node texts are prefetched; the plain stores (`addBatch`, - * `similaritySearch` with a prefetched query) are the intended fit. + * It is an {@link AsyncEncoder} (`encodeAsync` / `encodeBatchAsync`), and that + * is the way to use it: give it to a store or graph and call the `...Async` + * methods (`addNodeAsync`, `updateNodeAsync`, `embedAllNodesAsync`, + * `similaritySearchAsync`, ...), which await the network themselves. Identical + * texts requested at the same moment share one request. + * + * `EmbeddingEncoder` is synchronous and a network call is not, so the + * synchronous `encode` / `encodeBatch` serve from a cache: fetch first with + * `await encoder.prefetch(texts)` (every `...Async` call fills it too), and a + * text that was not fetched makes them throw an HttpEncoderError. That keeps + * the plain synchronous methods (`addNode`, `addBatch`, `similaritySearch`) + * usable for prefetched texts. * * @example * ```typescript * const encoder = new HttpEncoder({ * url: 'http://localhost:11434/api/embed', model: 'nomic-embed-text', api: 'ollama', * }); + * const graph = new SemanticGraph({ encoder }); + * await graph.addNodeAsync('person', 1, { name: 'Alice' }); // embeds over the network + * const hits = await graph.similaritySearchAsync('who is Alice'); + * + * // or, synchronously, after fetching the texts: * const texts = ['Alice is an engineer', 'Bob plans roadmaps']; * await encoder.prefetch([...texts, 'engineering']); * const store = new EmbeddingStore(encoder); @@ -341,7 +440,9 @@ export class HttpEncoder implements EmbeddingEncoder { private _maxRetries: number; private _retryDelayMs: number; private _maxCache: number; + private _concurrency: number; private _cache: Map<string, number[]> = new Map(); + private _inflight: Map<string, Promise<Map<string, number[]>>> = new Map(); private _fetch: HttpEncoderFetch; // Closures, not fields: they hold the credentials, so logging or // serialising the encoder cannot print them. @@ -395,6 +496,7 @@ export class HttpEncoder implements EmbeddingEncoder { this._maxRetries = Math.floor(nonNegative(options.maxRetries, 2, 'maxRetries')); this._retryDelayMs = nonNegative(options.retryDelayMs, 500, 'retryDelayMs'); this._maxCache = Math.floor(positive(options.maxCacheEntries, 100000, 'maxCacheEntries', false)); + this._concurrency = positive(options.concurrency, 1, 'concurrency', true); const fetchFn = options.fetch ?? (globalThis as { fetch?: HttpEncoderFetch }).fetch; if (typeof fetchFn !== 'function') { @@ -445,25 +547,64 @@ export class HttpEncoder implements EmbeddingEncoder { return (await this.encodeBatchAsync([text]))[0]; } - /** Fetch vectors for many texts, in order; only uncached texts hit the network. */ + /** + * Fetch vectors for many texts, in order. Only texts that are neither cached + * nor already being fetched hit the network, `batchSize` per request and + * `concurrency` requests at a time; the first failure rejects the call. + */ async encodeBatchAsync(texts: string[]): Promise<number[][]> { const found = new Map<string, number[]>(); - const missing = new Set<string>(); + const fresh: string[] = []; + const waits: Array<Promise<void>> = []; + const considered = new Set<string>(); for (const text of texts) { - if (found.has(text) || missing.has(text)) continue; + if (considered.has(text)) continue; + considered.add(text); const hit = this._cache.get(text); - if (hit) found.set(text, hit); - else missing.add(text); - } - const pending = [...missing]; - for (let start = 0; start < pending.length; start += this._batchSize) { - const chunk = pending.slice(start, start + this._batchSize); - const vectors = await this._request(chunk); - chunk.forEach((text, i) => { - found.set(text, vectors[i]); - this._remember(text, vectors[i]); - }); - } + if (hit) { + found.set(text, hit); + continue; + } + const shared = this._inflight.get(text); + if (shared) { + waits.push(shared.then(fetched => { found.set(text, fetched.get(text)!); })); + continue; + } + fresh.push(text); + } + + const chunks: string[][] = []; + for (let start = 0; start < fresh.length; start += this._batchSize) { + chunks.push(fresh.slice(start, start + this._batchSize)); + } + let next = 0; + let failed = false; + const worker = async (): Promise<void> => { + while (!failed && next < chunks.length) { + const chunk = chunks[next++]; + const job = this._request(chunk).then(vectors => { + const fetched = new Map<string, number[]>(); + chunk.forEach((text, i) => { + fetched.set(text, vectors[i]); + this._remember(text, vectors[i]); + }); + return fetched; + }); + for (const text of chunk) this._inflight.set(text, job); + const release = (): void => { + for (const text of chunk) if (this._inflight.get(text) === job) this._inflight.delete(text); + }; + job.then(release, release); + try { + for (const [text, vector] of await job) found.set(text, vector); + } catch (error) { + failed = true; + throw error; + } + } + }; + const workers = Array.from({ length: Math.min(this._concurrency, chunks.length) }, worker); + await Promise.all([...waits, ...workers]); return texts.map(text => found.get(text)!.slice()); } @@ -665,7 +806,7 @@ export interface ImportMeta { export type ModelMismatchMode = 'warn' | 'error' | 'ignore'; /** Options for EmbeddingStore beyond its encoder. */ -export interface EmbeddingStoreOptions { +export interface EmbeddingStoreOptions extends AnnOptions { /** * Free-form version of the encoder/model that made these vectors, recorded * in the sidecar. Falls back to the encoder's own `version`. @@ -717,8 +858,815 @@ export function selectTopK(results: SimilarityResult[], k: number): SimilarityRe return top; } +// ============================================================================= +// Approximate (IVF) search +// ============================================================================= + +/** Stores below this many vectors answer exactly even with `ann` on. */ +const ANN_MIN_SIZE = 50_000; +/** New or updated rows are scored exactly until this many (or this fraction of the trained size) are pending. */ +const ANN_FOLD_MIN = 512; +const ANN_FOLD_FRACTION = 0.02; +/** Retrain once writes + deletes since training exceed this fraction of the trained size. */ +const ANN_RETRAIN_FRACTION = 0.5; +/** + * k-means trains on up to this many sampled vectors per cell, for this many + * rounds. Python samples 40 per cell; with no BLAS here the build cost matters, + * and on 600k clustered vectors 24 per cell took 296 s against 390 s for 40 + * with the same recall at the default probe (0.978) and a little less at + * `annNprobe` 1 (0.93 against 0.96). + */ +const ANN_SAMPLE_PER_CELL = 24; +const ANN_KMEANS_ITERS = 8; +/** From this many cells on, nearest-cell lookups go through a coarse layer of about 3 x sqrt(nlist) super-cells. */ +const ANN_COARSE_MIN_CELLS = 256; +/** + * ... and look inside this many of the super-cells nearest the vector. Measured + * on 100k-300k clustered vectors: this finds the true nearest cell for 95-99% + * of them at about a third of the brute-force cost; looking inside only 4 + * left 2-20% in a different cell, which capped recall whatever `annNprobe`. + */ +const ANN_COARSE_PROBE = 24; +/** k-means rounds that place the super-cells over the centroids. */ +const ANN_COARSE_ROUNDS = 3; +/** Super-cells per sqrt(nlist): finer ones find the nearest cell more often for the same work. */ +const ANN_COARSE_FACTOR = 3; +/** float32 rows vs the float64 exact score differ by about 1e-7; candidates this close under the threshold are re-scored. */ +const ANN_THRESHOLD_SLACK = 1e-6; + +/** The approximate index kinds. Only IVF-flat exists. */ +export type AnnMode = 'ivf'; + +/** Approximate-search options shared by EmbeddingStore and SemanticGraph. */ +export interface AnnOptions { + /** + * `null`/`undefined` (default) keeps search exact, exactly as it always was. + * `'ivf'` answers bounded top-k searches (`topK` not null) from an in-memory + * IVF-flat index: vectors are clustered into `annNlist` cells and only the + * `annNprobe` cells nearest the query are scored, with the same cosine + * formula as the exact scan. It is *approximate* - a true neighbour in an + * unprobed cell is missed - so measure it with `recallCheck()`. The index is + * built lazily on the first search, never persisted (a saved graph and its + * sidecar are byte-for-byte what they would be without it), and kept + * current through writes and deletes. It keeps a Float32Array copy of the + * vectors (4 bytes per component) next to the stored ones. + */ + ann?: AnnMode | null; + /** Cells in the index. Default: about `4 * sqrt(N)` (at least 16, at most 65536) when it is built. */ + annNlist?: number; + /** Cells scanned per query. Default: about 5% of `annNlist`. `annNprobe >= annNlist` is the exact scan. */ + annNprobe?: number; + /** Below this many vectors the index is not used and search stays exact. Default 50000. */ + annMinSize?: number; + /** Seed for the k-means initialisation, so the same data gives the same index. Default 0. */ + annSeed?: number; +} + +/** What the approximate index looks like right now (see `annStats()`). */ +export interface AnnStats { + enabled: boolean; + mode: AnnMode | null; + /** An index exists and will answer. */ + active: boolean; + nlist: number | null; + nprobe: number | null; + /** Vectors the centroids were trained on. */ + trainedN: number | null; + /** Rows scored exactly on every query until they are folded into cells. */ + pending: number; + /** Writes + deletes since training. */ + churn: number; + buildSeconds: number | null; + /** Memory of the index itself (centroids, cell numbers, cell lists). */ + indexBytes: number; + /** Memory of the Float32Array copy of the vectors the index scores. */ + cacheBytes: number; + minSize: number; + /** Counts since the store was created. */ + searches: { approximate: number; exact: number; widened: number; builds: number }; +} + +/** Recall of the approximate index against the exact scan (see `recallCheck()`). */ +export interface RecallReport { + /** Mean over queries of |approximate top-k & exact top-k| / |exact top-k|. */ + recall: number; + /** The worst single query. */ + minRecall: number; + queries: number; + k: number; + nlist: number; + nprobe: number; + /** False when the index was not consulted (off, under `annMinSize`, or `annNprobe >= annNlist`); recall is then 1 by construction. */ + approximate: boolean; + /** Median milliseconds per approximate query / per exact query. */ + annMs: number; + exactMs: number; +} + +/** Banker's rounding, as Python's `round`, so the default cell counts agree across ports. */ +function roundHalfEven(x: number): number { + const floor = Math.floor(x); + const diff = x - floor; + if (diff < 0.5) return floor; + if (diff > 0.5) return floor + 1; + return floor % 2 === 0 ? floor : floor + 1; +} + +const nowMs = (): number => + typeof performance !== 'undefined' ? performance.now() : Date.now(); + +/** Small seeded generator (mulberry32): the same seed gives the same index. */ +class Rng { + private _a: number; + + constructor(seed: number) { + this._a = (Math.trunc(seed) ^ 0x9e3779b9) >>> 0; + } + + next(): number { + this._a = (this._a + 0x6d2b79f5) >>> 0; + let t = this._a; + t = Math.imul(t ^ (t >>> 15), t | 1); + t ^= t + Math.imul(t ^ (t >>> 7), t | 61); + return ((t ^ (t >>> 14)) >>> 0) / 4294967296; + } + + /** Integer in [0, n). */ + below(n: number): number { + return Math.floor(this.next() * n); + } +} + +/** Dot product of a[ao..ao+dim) and b[bo..bo+dim), four accumulators wide. */ +function dot2(a: Float32Array, ao: number, b: Float32Array, bo: number, dim: number): number { + let s0 = 0; + let s1 = 0; + let s2 = 0; + let s3 = 0; + let j = 0; + const lim = dim - 3; + for (; j < lim; j += 4) { + s0 += a[ao + j] * b[bo + j]; + s1 += a[ao + j + 1] * b[bo + j + 1]; + s2 += a[ao + j + 2] * b[bo + j + 2]; + s3 += a[ao + j + 3] * b[bo + j + 3]; + } + for (; j < dim; j++) s0 += a[ao + j] * b[bo + j]; + return s0 + s1 + s2 + s3; +} + +/** + * Nearest unit centroid to a vector, by dot product. + * + * With many cells a brute-force pass over all of them costs nlist x dim per + * vector, which is what makes k-means and assignment slow in a language with no + * BLAS. Past `ANN_COARSE_MIN_CELLS` the centroids are themselves grouped into + * about sqrt(nlist) super-cells and only the members of the few super-cells + * nearest the vector are compared: roughly 10x less work, and the cell it picks + * is the true nearest one all but rarely. A vector put in a near-miss cell is + * still found by any query that probes that cell. + */ +class CentroidSearch { + private readonly _nlist: number; + private readonly _dim: number; + private readonly _centroids: Float32Array; + private _coarse: Float32Array | null = null; + private _ncoarse = 0; + private _members = new Int32Array(0); + private _memberOffsets = new Int32Array(0); + private _coarseScore = new Float64Array(0); + private _probe: number; + private _pickIdx = new Int32Array(0); + private _pickScore = new Float64Array(0); + + constructor( + centroids: Float32Array, + nlist: number, + dim: number, + coarseFactor: number = ANN_COARSE_FACTOR, + probe: number = ANN_COARSE_PROBE + ) { + this._nlist = nlist; + this._dim = dim; + this._centroids = centroids; + this._probe = probe; + if (nlist >= ANN_COARSE_MIN_CELLS) this._buildCoarse(coarseFactor); + } + + private _buildCoarse(coarseFactor: number): void { + const nlist = this._nlist; + const dim = this._dim; + const centroids = this._centroids; + const ncoarse = Math.max(2, Math.round(coarseFactor * Math.sqrt(nlist))); + const coarse = new Float32Array(ncoarse * dim); + for (let i = 0; i < ncoarse; i++) { + const src = Math.floor((i * nlist) / ncoarse) * dim; + coarse.set(centroids.subarray(src, src + dim), i * dim); + } + const owner = new Int32Array(nlist); + const sums = new Float64Array(ncoarse * dim); + const counts = new Int32Array(ncoarse); + for (let round = 0; round < ANN_COARSE_ROUNDS; round++) { + for (let c = 0; c < nlist; c++) { + let best = 0; + let bestScore = -Infinity; + for (let k = 0; k < ncoarse; k++) { + const s = dot2(centroids, c * dim, coarse, k * dim, dim); + if (s > bestScore) { + bestScore = s; + best = k; + } + } + owner[c] = best; + } + if (round === ANN_COARSE_ROUNDS - 1) break; + sums.fill(0); + counts.fill(0); + for (let c = 0; c < nlist; c++) { + const k = owner[c]; + counts[k]++; + const base = k * dim; + const src = c * dim; + for (let j = 0; j < dim; j++) sums[base + j] += centroids[src + j]; + } + for (let k = 0; k < ncoarse; k++) { + if (counts[k] === 0) continue; + let len = 0; + for (let j = 0; j < dim; j++) len += sums[k * dim + j] * sums[k * dim + j]; + len = Math.sqrt(len); + if (len === 0) continue; + for (let j = 0; j < dim; j++) coarse[k * dim + j] = sums[k * dim + j] / len; + } + } + const memberOffsets = new Int32Array(ncoarse + 1); + for (let c = 0; c < nlist; c++) memberOffsets[owner[c] + 1]++; + for (let k = 0; k < ncoarse; k++) memberOffsets[k + 1] += memberOffsets[k]; + const fill = memberOffsets.slice(0, ncoarse); + const members = new Int32Array(nlist); + for (let c = 0; c < nlist; c++) members[fill[owner[c]]++] = c; + this._coarse = coarse; + this._ncoarse = ncoarse; + this._members = members; + this._memberOffsets = memberOffsets; + this._coarseScore = new Float64Array(ncoarse); + const probe = Math.min(this._probe, ncoarse); + this._pickIdx = new Int32Array(probe); + this._pickScore = new Float64Array(probe); + } + + /** Index of the centroid most similar to v[off..off+dim). */ + nearest(v: Float32Array, off: number): number { + const dim = this._dim; + const centroids = this._centroids; + let best = 0; + let bestScore = -Infinity; + const coarse = this._coarse; + if (coarse === null) { + for (let c = 0; c < this._nlist; c++) { + const s = dot2(centroids, c * dim, v, off, dim); + if (s > bestScore) { + bestScore = s; + best = c; + } + } + return best; + } + const pickIdx = this._pickIdx; + const pickScore = this._pickScore; + const probe = pickIdx.length; + let picked = 0; + for (let k = 0; k < this._ncoarse; k++) { + const s = dot2(coarse, k * dim, v, off, dim); + if (picked === probe && s <= pickScore[probe - 1]) continue; + let i = picked < probe ? picked++ : probe - 1; + while (i > 0 && pickScore[i - 1] < s) { + pickScore[i] = pickScore[i - 1]; + pickIdx[i] = pickIdx[i - 1]; + i--; + } + pickScore[i] = s; + pickIdx[i] = k; + } + const members = this._members; + const offsets = this._memberOffsets; + for (let p = 0; p < picked; p++) { + const k = pickIdx[p]; + for (let m = offsets[k]; m < offsets[k + 1]; m++) { + const c = members[m]; + const s = dot2(centroids, c * dim, v, off, dim); + if (s > bestScore) { + bestScore = s; + best = c; + } + } + } + return best; + } +} + +/** + * Float32Array snapshot of the store's vectors, kept in step with its writes. + * + * One contiguous matrix with spare row capacity, the vector norms, a type code + * per row and a liveness flag. A write patches it in place: an insert appends + * a row, an update overwrites one, a delete tombstones one (squeezed out once + * a quarter of the rows are dead). Rows stay in insertion order, which is the + * order the store's Map iterates, so ties break exactly as the exact scan's do. + * Built lazily by the first approximate search, never persisted. + */ +class ScanCache { + readonly dim: number; + matrix: Float32Array; + norms: Float64Array; + codes: Int32Array; + alive: Uint8Array; + /** Rows in use, tombstones included, and how many of those are tombstones. */ + n = 0; + dead = 0; + keys: Array<string | null> = []; + pos: Map<string, number> = new Map(); + codeOf: Map<string, number> = new Map(); + ivf: IvfIndex | null = null; + /** The query as float32, reused between searches. */ + query: Float32Array; + + constructor(dim: number, capacity: number) { + this.dim = dim; + this.matrix = new Float32Array(capacity * dim); + this.norms = new Float64Array(capacity); + this.codes = new Int32Array(capacity); + this.alive = new Uint8Array(capacity); + this.query = new Float32Array(dim); + } + + static build(records: Iterable<EmbeddingRecord>, count: number, dim: number): ScanCache { + const cache = new ScanCache(dim, count + Math.max(1024, count >> 4)); + for (const record of records) cache.put(record.id, record); + return cache; + } + + get capacity(): number { + return this.norms.length; + } + + private _grow(): void { + const capacity = this.capacity + Math.max(1024, this.capacity >> 2); + const matrix = new Float32Array(capacity * this.dim); + matrix.set(this.matrix.subarray(0, this.n * this.dim)); + const norms = new Float64Array(capacity); + norms.set(this.norms.subarray(0, this.n)); + const codes = new Int32Array(capacity); + codes.set(this.codes.subarray(0, this.n)); + const alive = new Uint8Array(capacity); + alive.set(this.alive.subarray(0, this.n)); + this.matrix = matrix; + this.norms = norms; + this.codes = codes; + this.alive = alive; + } + + /** Mirror one upserted record. */ + put(key: string, record: EmbeddingRecord): void { + const dim = this.dim; + let i = this.pos.get(key); + if (i === undefined) { + if (this.n >= this.capacity) this._grow(); + i = this.n++; + this.keys[i] = key; + this.pos.set(key, i); + const type = record.nodeRef[0]; + let code = this.codeOf.get(type); + if (code === undefined) { + code = this.codeOf.size; + this.codeOf.set(type, code); + } + this.codes[i] = code; + this.alive[i] = 1; + } + const vector = record.vector; + const base = i * dim; + let sum = 0; + for (let j = 0; j < dim; j++) { + const f = Math.fround(vector[j]); + this.matrix[base + j] = f; + sum += f * f; + } + this.norms[i] = Math.sqrt(sum); + if (this.ivf !== null) this.ivf.notePut(i, this.capacity); + } + + /** Mirror one deleted record. */ + drop(key: string): void { + const i = this.pos.get(key); + if (i === undefined) return; + this.pos.delete(key); + this.matrix.fill(0, i * this.dim, (i + 1) * this.dim); + this.norms[i] = 0; + this.alive[i] = 0; + this.keys[i] = null; + this.dead++; + if (this.ivf !== null) this.ivf.noteDrop(i); + while (this.n > 0 && this.alive[this.n - 1] === 0) { // trailing tombstones + this.n--; + this.keys.pop(); + this.dead--; + } + if (this.dead > 32 && this.dead * 4 > this.n) this._compact(); + } + + /** Squeeze the tombstones out, keeping the live rows in their order. */ + private _compact(): void { + const dim = this.dim; + const live = new Int32Array(this.n - this.dead); + let kept = 0; + for (let r = 0; r < this.n; r++) if (this.alive[r] === 1) live[kept++] = r; + if (this.ivf !== null) this.ivf.compact(live); + for (let w = 0; w < kept; w++) { + const r = live[w]; + if (r !== w) { + this.matrix.copyWithin(w * dim, r * dim, (r + 1) * dim); + this.norms[w] = this.norms[r]; + this.codes[w] = this.codes[r]; + const key = this.keys[r]!; + this.keys[w] = key; + this.pos.set(key, w); + } + } + this.alive.fill(1, 0, kept); + this.alive.fill(0, kept, this.n); + this.keys.length = kept; + this.n = kept; + this.dead = 0; + } + + bytes(): number { + return this.matrix.byteLength + this.norms.byteLength + this.codes.byteLength + this.alive.byteLength; + } +} + +/** + * Inverted-file ("IVF-flat") index over a {@link ScanCache}. + * + * Spherical k-means partitions the rows into `nlist` cells by direction, which + * is what cosine similarity compares. A query scores the `nprobe` cells whose + * centroids are nearest and only the rows inside them, with the same cosine + * formula as the exact scan. The vectors are not copied again: the index is a + * cell number per row (`assign`) plus the rows grouped by cell + * (`order`/`offsets`), about 8 bytes per vector plus the centroids. + * + * It follows its cache instead of being rebuilt on every write: + * - a **new** or **updated** row is *pending* (cell -1): every query scores + * the pending rows exactly, so a write is visible at once and costs the + * index nothing; + * - pending rows are folded into their nearest cell in one pass once there + * are enough of them (see `EmbeddingStore._annIndex`); + * - a **delete** clears the row's cell; stale entries in `order` are skipped + * at query time; + * - the cache's compaction renumbers rows, and `compact` carries the cells + * across without retraining; + * - `churn` counts writes and deletes since training; the store retrains when + * it passes a fraction of the trained size. + */ +class IvfIndex { + readonly nlist: number; + readonly dim: number; + readonly centroids: Float32Array; + assign: Int32Array; + order = new Int32Array(0); + offsets = new Int32Array(0); + pending: Set<number> = new Set(); + churn = 0; + trainedN = 0; + buildSeconds = 0; + private _search: CentroidSearch; + // Scratch kept between queries: a search allocates nothing proportional to the store. + private _chosen = new Int32Array(0); + private _chosenScore = new Float64Array(0); + candRows = new Int32Array(0); + candScores = new Float64Array(0); + + constructor(centroids: Float32Array, nlist: number, dim: number, assign: Int32Array) { + this.nlist = nlist; + this.dim = dim; + this.centroids = centroids; + this.assign = assign; + this._search = new CentroidSearch(centroids, nlist, dim); + } + + /** Train centroids on a sample, then give every live row its cell. */ + static build(cache: ScanCache, nlistWanted: number, seed: number): IvfIndex { + const started = nowMs(); + const { n, dim, matrix, norms, alive } = cache; + const live = new Int32Array(n - cache.dead); + let total = 0; + for (let r = 0; r < n; r++) if (alive[r] === 1) live[total++] = r; + const nlist = Math.max(1, Math.min(Math.floor(nlistWanted), total)); + const rng = new Rng(seed); + + // Sample up to 40 vectors per cell (a partial Fisher-Yates), as unit rows. + const sampleN = Math.min(total, ANN_SAMPLE_PER_CELL * nlist); + let sample: Int32Array; + if (sampleN === total) { + sample = live; + } else { + const pool = live.slice(); + for (let i = 0; i < sampleN; i++) { + const j = i + rng.below(total - i); + const t = pool[i]; + pool[i] = pool[j]; + pool[j] = t; + } + sample = pool.subarray(0, sampleN); + sample.sort(); + } + const x = new Float32Array(sampleN * dim); + for (let i = 0; i < sampleN; i++) { + const r = sample[i]; + const inv = norms[r] > 0 ? 1 / norms[r] : 0; + const src = r * dim; + const dst = i * dim; + for (let j = 0; j < dim; j++) x[dst + j] = matrix[src + j] * inv; + } + + // Initial centroids: distinct sample points. + const centroids = new Float32Array(nlist * dim); + const picks = new Int32Array(sampleN); + for (let i = 0; i < sampleN; i++) picks[i] = i; + for (let c = 0; c < nlist; c++) { + const j = c + rng.below(sampleN - c); + const t = picks[c]; + picks[c] = picks[j]; + picks[j] = t; + centroids.set(x.subarray(picks[c] * dim, (picks[c] + 1) * dim), c * dim); + } + + // Spherical k-means: assign to the most similar centroid, re-centre on the sum, re-normalise. + const cell = new Int32Array(sampleN); + const counts = new Int32Array(nlist); + const sums = new Float64Array(nlist * dim); + for (let iter = 0; iter < ANN_KMEANS_ITERS; iter++) { + const search = new CentroidSearch(centroids, nlist, dim); + counts.fill(0); + sums.fill(0); + for (let i = 0; i < sampleN; i++) { + const c = search.nearest(x, i * dim); + cell[i] = c; + counts[c]++; + const base = c * dim; + const src = i * dim; + for (let j = 0; j < dim; j++) sums[base + j] += x[src + j]; + } + for (let c = 0; c < nlist; c++) { + let len = 0; + if (counts[c] > 0) { + for (let j = 0; j < dim; j++) len += sums[c * dim + j] * sums[c * dim + j]; + len = Math.sqrt(len); + } + if (counts[c] === 0 || len === 0) { // a dead centroid: re-seed on a random point + const i = rng.below(sampleN); + centroids.set(x.subarray(i * dim, (i + 1) * dim), c * dim); + } else { + for (let j = 0; j < dim; j++) centroids[c * dim + j] = sums[c * dim + j] / len; + } + } + } + + const index = new IvfIndex(centroids, nlist, dim, new Int32Array(cache.capacity).fill(-1)); + index.trainedN = total; + for (let t = 0; t < total; t++) { + const r = live[t]; + index.assign[r] = index._search.nearest(matrix, r * dim); + } + index.relist(n); + index.buildSeconds = (nowMs() - started) / 1000; + return index; + } + + /** Regroup the rows by cell (a stable counting sort of `assign`). */ + relist(n: number): void { + const assign = this.assign; + const offsets = new Int32Array(this.nlist + 1); + let total = 0; + for (let r = 0; r < n; r++) { + const c = assign[r]; + if (c >= 0) { + offsets[c + 1]++; + total++; + } + } + for (let c = 0; c < this.nlist; c++) offsets[c + 1] += offsets[c]; + const fill = offsets.slice(0, this.nlist); + const order = new Int32Array(total); + for (let r = 0; r < n; r++) { + const c = assign[r]; + if (c >= 0) order[fill[c]++] = r; + } + this.order = order; + this.offsets = offsets; + } + + /** Give every pending row its nearest cell and regroup. */ + fold(cache: ScanCache): void { + if (this.pending.size > 0) { + for (const r of this.pending) { + this.assign[r] = this._search.nearest(cache.matrix, r * this.dim); + } + this.pending.clear(); + } + this.relist(cache.n); + } + + notePut(i: number, capacity: number): void { + if (i >= this.assign.length) { + const grown = new Int32Array(Math.max(capacity, i + 1)).fill(-1); + grown.set(this.assign); + this.assign = grown; + } + this.assign[i] = -1; + this.pending.add(i); + this.churn++; + } + + noteDrop(i: number): void { + if (i < this.assign.length) this.assign[i] = -1; + this.pending.delete(i); + this.churn++; + } + + /** Follow the cache's compaction (`live` = the old rows kept, in order). */ + compact(live: Int32Array): void { + const kept = live.length; + const moved = new Int32Array(kept); + for (let w = 0; w < kept; w++) moved[w] = this.assign[live[w]]; + this.assign.fill(-1); + this.assign.set(moved); + if (this.pending.size > 0) { + const old = Array.from(this.pending).sort((a, b) => a - b); + this.pending = new Set(old.map(r => lowerBound(live, r))); + } + this.relist(kept); + } + + bytes(): number { + return this.centroids.byteLength + this.assign.byteLength + this.order.byteLength + this.offsets.byteLength; + } + + /** + * Score the rows of the `lists` cells nearest `q`, plus every pending row, + * into `candRows`/`candScores`; returns how many were kept. `code` < 0 means + * any node type; rows scoring under `floor` are left out. + */ + gather(cache: ScanCache, q: Float32Array, qnorm: number, code: number, floor: number, lists: number): number { + const { nlist, dim, centroids } = this; + if (this._chosen.length < lists) { + this._chosen = new Int32Array(lists); + this._chosenScore = new Float64Array(lists); + } + const chosen = this._chosen; + let nChosen = 0; + if (lists >= nlist) { + for (let c = 0; c < nlist; c++) chosen[c] = c; + nChosen = nlist; + } else { + const chosenScore = this._chosenScore; + for (let c = 0; c < nlist; c++) { + const s = dot2(centroids, c * dim, q, 0, dim); + if (nChosen === lists && s <= chosenScore[lists - 1]) continue; + let i = nChosen < lists ? nChosen++ : lists - 1; + while (i > 0 && chosenScore[i - 1] < s) { + chosenScore[i] = chosenScore[i - 1]; + chosen[i] = chosen[i - 1]; + i--; + } + chosenScore[i] = s; + chosen[i] = c; + } + } + + const order = this.order; + const offsets = this.offsets; + let bound = this.pending.size; + for (let k = 0; k < nChosen; k++) bound += offsets[chosen[k] + 1] - offsets[chosen[k]]; + if (this.candRows.length < bound) { + const size = Math.max(bound, this.candRows.length * 2, 256); + this.candRows = new Int32Array(size); + this.candScores = new Float64Array(size); + } + const candRows = this.candRows; + const candScores = this.candScores; + const { matrix, norms, codes } = cache; + const assign = this.assign; + + let m = 0; + for (let k = 0; k < nChosen; k++) { + const c = chosen[k]; + for (let p = offsets[c], end = offsets[c + 1]; p < end; p++) { + const r = order[p]; + if (assign[r] < 0) continue; // deleted or updated since the last regroup + if (code >= 0 && codes[r] !== code) continue; + const denom = norms[r] * qnorm; + const s = denom > 0 ? dot2(matrix, r * dim, q, 0, dim) / denom : 0; + if (s < floor) continue; + candRows[m] = r; + candScores[m] = s; + m++; + } + } + for (const r of this.pending) { + if (code >= 0 && codes[r] !== code) continue; + const denom = norms[r] * qnorm; + const s = denom > 0 ? dot2(matrix, r * dim, q, 0, dim) / denom : 0; + if (s < floor) continue; + candRows[m] = r; + candScores[m] = s; + m++; + } + return m; + } + + /** The `k` best of the first `m` candidates: score descending, row ascending on ties. */ + best(m: number, k: number): Int32Array { + const rows = this.candRows; + const scores = this.candScores; + const size = Math.min(k, m); + // Min-heap of candidate slots whose root is the worst kept one. + const heap = new Int32Array(size); + const worse = (a: number, b: number): boolean => + scores[a] < scores[b] || (scores[a] === scores[b] && rows[a] > rows[b]); + let count = 0; + for (let i = 0; i < m; i++) { + if (count === size) { + const root = heap[0]; + if (!(scores[i] > scores[root] || (scores[i] === scores[root] && rows[i] < rows[root]))) continue; + heap[0] = i; + let at = 0; + for (;;) { + const l = 2 * at + 1; + if (l >= size) break; + const r = l + 1; + const child = r < size && worse(heap[r], heap[l]) ? r : l; + if (!worse(heap[child], heap[at])) break; + const t = heap[at]; + heap[at] = heap[child]; + heap[child] = t; + at = child; + } + } else { + let at = count++; + heap[at] = i; + while (at > 0) { + const parent = (at - 1) >> 1; + if (!worse(heap[at], heap[parent])) break; + const t = heap[at]; + heap[at] = heap[parent]; + heap[parent] = t; + at = parent; + } + } + } + // Drain worst-first into the back of the output. + const out = new Int32Array(count); + for (let o = count - 1; o >= 0; o--) { + out[o] = heap[0]; + const last = heap[o]; + if (o === 0) break; + heap[0] = last; + let at = 0; + for (;;) { + const l = 2 * at + 1; + if (l >= o) break; + const r = l + 1; + const child = r < o && worse(heap[r], heap[l]) ? r : l; + if (!worse(heap[child], heap[at])) break; + const t = heap[at]; + heap[at] = heap[child]; + heap[child] = t; + at = child; + } + } + return out; + } +} + +/** Index of the first element of sorted `values` that is >= `target`. */ +function lowerBound(values: Int32Array, target: number): number { + let lo = 0; + let hi = values.length; + while (lo < hi) { + const mid = (lo + hi) >> 1; + if (values[mid] < target) lo = mid + 1; + else hi = mid; + } + return lo; +} + /** * In-memory embedding storage with brute-force cosine similarity search. + * + * Search is exact by default. For large stores an opt-in approximate index + * (`ann: 'ivf'`, see {@link AnnOptions}) trades a little recall for much lower + * latency; this is built for an application's own knowledge graph, not as a + * vector-database replacement. */ export class EmbeddingStore { private _embeddings: Map<string, EmbeddingRecord> = new Map(); @@ -727,22 +1675,63 @@ export class EmbeddingStore { private _version: string | null; private _onModelMismatch: ModelMismatchMode = 'warn'; private _warnedMismatch: Set<string> = new Set(); + private _ann: AnnMode | null = null; + private _annNlist: number | null = null; + private _annNprobe: number | null = null; + private _annMinSize: number = ANN_MIN_SIZE; + private _annSeed = 0; + private _annCounts = { approximate: 0, exact: 0, widened: 0, builds: 0 }; + private _scan: ScanCache | null = null; /** Metadata of the most recent imported sidecar (empty before any). */ lastImportMeta: ImportMeta = {}; - constructor(encoder?: EmbeddingEncoder, options: EmbeddingStoreOptions = {}) { - this._encoder = encoder || null; + constructor(encoder?: EmbeddingEncoder | AsyncEncoder, options: EmbeddingStoreOptions = {}) { + this._encoder = adoptEncoder(encoder); this._version = options.version ?? null; this.onModelMismatch = options.onModelMismatch ?? 'warn'; + + const ann = options.ann ?? null; + if (ann !== null && ann !== 'ivf') { + throw new EmbeddingError(`ann must be null or 'ivf', got ${JSON.stringify(ann)}`); + } + const whole = (label: string, value: number | null | undefined, floor: number): number | null => { + if (value === undefined || value === null) return null; + if (typeof value !== 'number' || !Number.isInteger(value) || value < floor) { + throw new EmbeddingError(`${label} must be an integer >= ${floor}, got ${value}`); + } + return value; + }; + this._ann = ann; + this._annNlist = whole('annNlist', options.annNlist, 1); + this._annNprobe = whole('annNprobe', options.annNprobe, 1); + this._annMinSize = whole('annMinSize', options.annMinSize, 0) ?? ANN_MIN_SIZE; + this._annSeed = Math.trunc(Number(options.annSeed ?? 0)) || 0; + } + + /** The approximate index in use (`'ivf'`), or null when search is exact. */ + get ann(): AnnMode | null { + return this._ann; + } + + /** Cells scanned per query; null means "automatic" (about 5% of the cells). */ + get annNprobe(): number | null { + return this._annNprobe; + } + + set annNprobe(value: number | null) { + if (value !== null && (typeof value !== 'number' || !Number.isInteger(value) || value < 1)) { + throw new EmbeddingError(`annNprobe must be an integer >= 1, got ${value}`); + } + this._annNprobe = value; } get encoder(): EmbeddingEncoder | null { return this._encoder; } - set encoder(encoder: EmbeddingEncoder | null) { - this._encoder = encoder; + set encoder(encoder: EmbeddingEncoder | AsyncEncoder | null) { + this._encoder = adoptEncoder(encoder); } /** Vector length this store holds, or null while it is still empty. */ @@ -821,6 +1810,10 @@ export class EmbeddingStore { } else { vector = query; } + return this._checkQuery(vector); + } + + private _checkQuery(vector: number[]): number[] { if (vector.length === 0) { throw new DimensionMismatchError("query vector is empty"); } @@ -934,10 +1927,16 @@ export class EmbeddingStore { this._checkDimension(vector, "vector"); const key = this._nodeRefToKey(nodeRef); - this._embeddings.set(key, { id: key, nodeRef, vector, text, model: this._modelName() }); + this._set(key, { id: key, nodeRef, vector, text, model: this._modelName() }); return key; } + /** Every write goes through here, so the approximate index's cache stays in step. */ + private _set(key: string, record: EmbeddingRecord): void { + this._embeddings.set(key, record); + if (this._scan !== null) this._scan.put(key, record); + } + /** Add many embeddings, encoding their texts in one batched call. */ addBatch(entries: Array<[NodeRef, string] | [NodeRef, string, number[]]>): number { if (entries.length === 0) return 0; @@ -969,6 +1968,72 @@ export class EmbeddingStore { return entries.length; } + /** + * Encode texts with the store's encoder, awaiting it: an {@link AsyncEncoder} + * (such as HttpEncoder) over the network, any other encoder as `encodeBatch`. + * Nothing is stored. Rejects with NoEncoderError when there is no encoder. + */ + async encodeTextsAsync(texts: string[]): Promise<number[][]> { + const encoder = this._encoder; + if (!encoder) { + throw new NoEncoderError("No encoder provided and no vectors given"); + } + const vectors = isAsyncEncoder(encoder) + ? await encoder.encodeBatchAsync(texts) + : encoder.encodeBatch(texts); + if (vectors.length !== texts.length) { + throw new EmbeddingError(`encoder returned ${vectors.length} vectors for ${texts.length} texts`); + } + return vectors; + } + + /** + * Like {@link add}, but awaits the encoder, so it works with one that makes + * a network call. On a failure nothing is stored. + */ + async addAsync(nodeRef: NodeRef, text: string, vector?: number[]): Promise<string> { + if (vector === undefined) { + vector = (await this.encodeTextsAsync([text]))[0]; + } + return this.add(nodeRef, text, vector); + } + + /** + * Like {@link addBatch}, but awaits the encoder: the texts that need + * encoding go out in one `encodeBatchAsync` call (HttpEncoder splits it into + * `batchSize` requests). All entries are stored only once every vector is in. + */ + async addBatchAsync(entries: Array<[NodeRef, string] | [NodeRef, string, number[]]>): Promise<number> { + if (entries.length === 0) return 0; + const needsEncoding = entries + .map((e, i) => (e.length < 3 || e[2] === undefined ? i : -1)) + .filter(i => i >= 0); + + const vectorFor = new Map<number, number[]>(); + if (needsEncoding.length > 0) { + const encoded = await this.encodeTextsAsync(needsEncoding.map(i => entries[i][1])); + needsEncoding.forEach((slot, i) => vectorFor.set(slot, encoded[i])); + } + entries.forEach((entry, i) => { + this.add(entry[0], entry[1], vectorFor.get(i) ?? (entry[2] as number[])); + }); + return entries.length; + } + + /** + * The vector a query stands for: a vector as it is, a text through the + * encoder (awaited), with the same model-mismatch check as a synchronous + * text query. Used by the `...Async` searches. + */ + async resolveQueryAsync(query: string | number[]): Promise<number[]> { + if (typeof query !== 'string') return this._checkQuery(query); + if (!this._encoder) { + throw new NoEncoderError("No encoder provided for text query"); + } + this._checkModelCompat(); + return this._checkQuery((await this.encodeTextsAsync([query]))[0]); + } + get(nodeRef: NodeRef): EmbeddingRecord | null { const key = this._nodeRefToKey(nodeRef); return this._embeddings.get(key) || null; @@ -976,16 +2041,47 @@ export class EmbeddingStore { remove(nodeRef: NodeRef): boolean { const key = this._nodeRefToKey(nodeRef); - return this._embeddings.delete(key); + const removed = this._embeddings.delete(key); + if (removed && this._scan !== null) this._scan.drop(key); + return removed; } + /** + * Find the most similar nodes by embedding. + * + * Exact by default. With `ann: 'ivf'`, a bounded `topK` on a store of at + * least `annMinSize` vectors is answered from the approximate index (the + * scores are the same cosine values, but a true neighbour in an unprobed + * cell can be missing); `exact: true` forces the full scan, and has no effect + * on a store without `ann`. `topK: null` always scans everything. + */ similaritySearch( query: string | number[], topK: number | null = 10, nodeType?: string, - threshold: number = 0.0 + threshold: number = 0.0, + exact: boolean = false ): SimilarityResult[] { const queryVector = this._resolveQuery(query); + if (this._ann !== null) { + if (exact || topK === null || topK <= 0) { + this._annCounts.exact++; + } else { + const answer = this._annSearch(queryVector, topK, nodeType, threshold); + if (answer !== null) return answer; + this._annCounts.exact++; + } + } + return this._scanExact(queryVector, topK, nodeType, threshold); + } + + /** The brute-force scan every search used before the approximate index existed. */ + private _scanExact( + queryVector: number[], + topK: number | null, + nodeType: string | undefined, + threshold: number + ): SimilarityResult[] { const results: SimilarityResult[] = []; for (const record of this._embeddings.values()) { @@ -1001,6 +2097,325 @@ export class EmbeddingStore { return selectTopK(results, topK); } + /** {@link similaritySearch} with a text query encoded by awaiting the encoder. */ + async similaritySearchAsync( + query: string | number[], + topK: number | null = 10, + nodeType?: string, + threshold: number = 0.0, + exact: boolean = false + ): Promise<SimilarityResult[]> { + return this.similaritySearch(await this.resolveQueryAsync(query), topK, nodeType, threshold, exact); + } + + /** The vector a query stands for (a text goes through the synchronous encoder). */ + resolveQuery(query: string | number[]): number[] { + return this._resolveQuery(query); + } + + // -- approximate (IVF) search ------------------------------------------------- + + /** Cells for a store of `n` vectors: about 4 * sqrt(n), within [16, 65536]. */ + private static _annAutoNlist(n: number): number { + return Math.min(Math.max(roundHalfEven(4 * Math.sqrt(n)), 16), 65536); + } + + /** + * The cache holding the index, with the index built or maintained as the + * policy says; null means "answer exactly": `ann` is off, or the store holds + * fewer than `annMinSize` vectors. + * + * Policy: + * - built lazily by the first search (never persisted): k-means on a sample + * of up to 40 x nlist vectors, then every row assigned a cell; + * - new and updated rows are scored exactly until at least + * `max(512, 2%)` of the indexed rows are pending, then folded into cells; + * - deletes tombstone (nothing to do but skip them); + * - once writes + deletes since training exceed 50% of the trained size, + * the index is retrained from the current data, with an automatic `nlist` + * re-chosen for the new size; + * - a cleared store, a dimension change, or a store that shrinks below + * `annMinSize` drops the index with its cache. + */ + private _annIndex(): ScanCache | null { + if (this._ann === null || this._dimension === null) return null; + const live = this._embeddings.size; + if (live < Math.max(2, this._annMinSize)) { + this._scan = null; + return null; + } + let scan = this._scan; + if (scan === null) { + scan = this._scan = ScanCache.build(this._embeddings.values(), live, this._dimension); + } + let ivf = scan.ivf; + if (ivf !== null && ivf.churn > ANN_RETRAIN_FRACTION * ivf.trainedN) ivf = scan.ivf = null; + if (ivf === null) { + const nlist = this._annNlist ?? EmbeddingStore._annAutoNlist(live); + scan.ivf = IvfIndex.build(scan, nlist, this._annSeed); + this._annCounts.builds++; + } else if (ivf.pending.size >= Math.max(ANN_FOLD_MIN, ANN_FOLD_FRACTION * ivf.trainedN)) { + ivf.fold(scan); + } + return scan; + } + + private _annProbe(ivf: IvfIndex): number { + const wanted = this._annNprobe ?? Math.max(1, roundHalfEven(0.05 * ivf.nlist)); + return Math.min(wanted, ivf.nlist); + } + + /** + * Cells to probe at widening `scale`; null = widened enough to just scan. + * Scale 1 honours `annNprobe` as configured (`annNprobe >= nlist` is the + * exact scan); later steps multiply it by 4 and give up on the index once + * they would touch half the cells, where scanning everything costs about the + * same. + */ + private _annLists(ivf: IvfIndex, scale: number): number | null { + const lists = this._annProbe(ivf) * scale; + if (scale === 1) return lists >= ivf.nlist ? null : lists; + return lists * 2 >= ivf.nlist ? null : lists; + } + + /** Load the query into the cache's float32 buffer; null when it has no direction (all zeros). */ + private _annQuery(scan: ScanCache, queryVector: number[], nodeType: string | undefined): { qnorm: number; code: number } | null { + const q = scan.query; + let sum = 0; + for (let j = 0; j < scan.dim; j++) { + const f = Math.fround(queryVector[j]); + q[j] = f; + sum += f * f; + } + const qnorm = Math.sqrt(sum); + if (qnorm === 0) return null; + let code = -1; + if (nodeType) code = scan.codeOf.get(nodeType) ?? -2; // -2: no node of that type exists + return { qnorm, code }; + } + + /** + * Top-k from the index, or null to fall back to the exact scan. + * + * Filters (`nodeType`, `threshold`) apply to the probed rows. If fewer than + * `topK` survive, the probe is widened 4x and tried again, and past half the + * cells the exact scan answers instead - so a rare type or a high threshold + * costs time rather than silently returning a short list. + */ + private _annSearch( + queryVector: number[], + topK: number, + nodeType: string | undefined, + threshold: number + ): SimilarityResult[] | null { + const scan = this._annIndex(); + if (scan === null) return null; + const ivf = scan.ivf!; + const prepared = this._annQuery(scan, queryVector, nodeType); + if (prepared === null) return null; + if (prepared.code === -2) return []; + const floor = threshold - ANN_THRESHOLD_SLACK; + for (let scale = 1; ; scale *= 4) { + const lists = this._annLists(ivf, scale); + if (lists === null) return null; + const m = ivf.gather(scan, scan.query, prepared.qnorm, prepared.code, floor, lists); + if (m < topK) continue; + const best = ivf.best(m, topK); + const results: SimilarityResult[] = []; + for (let i = 0; i < best.length; i++) { + const record = this._embeddings.get(scan.keys[ivf.candRows[best[i]]]!)!; + results.push({ nodeRef: record.nodeRef, score: ivf.candScores[best[i]], text: record.text }); + } + this._annCounts.approximate++; + if (scale > 1) this._annCounts.widened++; + return this.annPolish(results, queryVector, threshold); + } + } + + /** + * Re-score approximate results with the exact float64 cosine the full scan + * uses (the index scores float32 copies), drop any that fall under + * `threshold` once exact, and order them best first. + */ + annPolish(results: SimilarityResult[], queryVector: number[], threshold: number): SimilarityResult[] { + const out: SimilarityResult[] = []; + for (const result of results) { + const record = this._embeddings.get(this._nodeRefToKey(result.nodeRef)); + if (!record) continue; + const score = this._cosineSimilarity(queryVector, record.vector); + if (score >= threshold) out.push({ nodeRef: result.nodeRef, score, text: result.text }); + } + return out.sort((a, b) => b.score - a.score); + } + + /** + * Candidates for a search that has to filter them itself (a graph's + * `nodeFilter`): every row of the probed cells that clears `threshold`, best + * first, lazily, with *approximate* (float32) scores - pass what you keep to + * {@link annPolish}. `scale` is the widening step (1, 4, 16, ...); null asks + * for the exact scan. `approximate` is false when the answer is exact - the + * caller has nothing left to widen to. + */ + annRanked( + queryVector: number[], + nodeType: string | undefined, + threshold: number, + scale: number | null + ): { ranked: Iterable<SimilarityResult>; approximate: boolean } { + if (scale !== null && this._ann !== null) { + const scan = this._annIndex(); + const ivf = scan === null ? null : scan.ivf; + const lists = ivf === null ? null : this._annLists(ivf, scale); + const prepared = scan === null || lists === null ? null : this._annQuery(scan, queryVector, nodeType); + if (scan !== null && ivf !== null && lists !== null && prepared !== null) { + if (prepared.code === -2) return { ranked: [], approximate: false }; + const m = ivf.gather(scan, scan.query, prepared.qnorm, prepared.code, threshold - ANN_THRESHOLD_SLACK, lists); + this._annCounts.approximate++; + if (scale > 1) this._annCounts.widened++; + const rows = ivf.candRows.slice(0, m); + const scores = ivf.candScores.slice(0, m); + const slots = new Int32Array(m); + for (let i = 0; i < m; i++) slots[i] = i; + slots.sort((a, b) => scores[b] - scores[a] || rows[a] - rows[b]); + const keys = scan.keys; + const embeddings = this._embeddings; + const ranked = (function* (): Generator<SimilarityResult> { + for (let i = 0; i < m; i++) { + const record = embeddings.get(keys[rows[slots[i]]]!)!; + yield { nodeRef: record.nodeRef, score: scores[slots[i]], text: record.text }; + } + })(); + return { ranked, approximate: true }; + } + } + return { ranked: this._scanExact(queryVector, null, nodeType, threshold), approximate: false }; + } + + /** + * Build the approximate index now instead of on the first search; returns + * {@link annStats}. `force` retrains even if an index exists. Does nothing + * (and says so in the stats) while the store is smaller than `annMinSize`. + * + * @throws EmbeddingError when the store was created without `ann` + */ + buildAnnIndex(force: boolean = false): AnnStats { + if (this._ann === null) { + throw new EmbeddingError("this store has no approximate index; create it with ann: 'ivf'"); + } + if (force && this._scan !== null) this._scan.ivf = null; + const scan = this._annIndex(); + if (scan !== null && scan.ivf !== null && scan.ivf.pending.size > 0) scan.ivf.fold(scan); + return this.annStats(); + } + + /** What the approximate index looks like right now. Never builds anything. */ + annStats(): AnnStats { + const scan = this._scan; + const ivf = scan === null ? null : scan.ivf; + return { + enabled: this._ann !== null, + mode: this._ann, + active: ivf !== null, + nlist: ivf === null ? null : ivf.nlist, + nprobe: ivf === null ? this._annNprobe : this._annProbe(ivf), + trainedN: ivf === null ? null : ivf.trainedN, + pending: ivf === null ? 0 : ivf.pending.size, + churn: ivf === null ? 0 : ivf.churn, + buildSeconds: ivf === null ? null : ivf.buildSeconds, + indexBytes: ivf === null ? 0 : ivf.bytes(), + cacheBytes: scan === null ? 0 : scan.bytes(), + minSize: this._annMinSize, + searches: { ...this._annCounts }, + }; + } + + /** + * Recall@k of the approximate index against the exact scan. + * + * Runs every query both ways and compares the *sets* of top-k nodes. + * `queries` is a text, a vector, or a list of either. Use real queries from + * your own workload: recall depends on how the data clusters, and a score + * measured on unrelated queries or data does not transfer. Raise `annNprobe` + * until the recall you need holds. + * + * Without an index in use (`ann` off, a store under `annMinSize`, or + * `annNprobe >= nlist`) both answers are the exact scan, recall is 1 and + * `approximate` is false. + */ + recallCheck( + queries: string | ArrayLike<number> | Array<string | ArrayLike<number>>, + k: number = 10, + nodeType?: string, + threshold: number = 0.0 + ): RecallReport { + if (!Number.isInteger(k) || k <= 0) throw new EmbeddingError(`k must be positive, got ${k}`); + const vectors = EmbeddingStore._queryList(queries).map(q => this._resolveQuery(q)); + return this._recall(vectors, k, nodeType, threshold); + } + + /** {@link recallCheck} with text queries encoded by awaiting the encoder. */ + async recallCheckAsync( + queries: string | ArrayLike<number> | Array<string | ArrayLike<number>>, + k: number = 10, + nodeType?: string, + threshold: number = 0.0 + ): Promise<RecallReport> { + if (!Number.isInteger(k) || k <= 0) throw new EmbeddingError(`k must be positive, got ${k}`); + const vectors: number[][] = []; + for (const query of EmbeddingStore._queryList(queries)) vectors.push(await this.resolveQueryAsync(query)); + return this._recall(vectors, k, nodeType, threshold); + } + + /** One query or many, as a list of texts and plain number arrays. */ + private static _queryList( + queries: string | ArrayLike<number> | Array<string | ArrayLike<number>> + ): Array<string | number[]> { + if (typeof queries === 'string') return [queries]; + const list = Array.from(queries as ArrayLike<unknown>); + if (list.length > 0 && typeof list[0] === 'number') return [list as number[]]; + return list.map(q => (typeof q === 'string' ? q : Array.from(q as ArrayLike<number>))); + } + + private _recall(vectors: number[][], k: number, nodeType: string | undefined, threshold: number): RecallReport { + const recalls: number[] = []; + const annTimes: number[] = []; + const exactTimes: number[] = []; + let used = false; + for (const vector of vectors) { + const before = this._annCounts.approximate; + let started = nowMs(); + const found = this.similaritySearch(vector, k, nodeType, threshold); + annTimes.push(nowMs() - started); + used = used || this._annCounts.approximate > before; + started = nowMs(); + const truth = this.similaritySearch(vector, k, nodeType, threshold, true); + exactTimes.push(nowMs() - started); + if (truth.length > 0) { + const wanted = new Set(truth.map(r => this._nodeRefToKey(r.nodeRef))); + const hits = found.filter(r => wanted.has(this._nodeRefToKey(r.nodeRef))).length; + recalls.push(hits / wanted.size); + } + } + const median = (values: number[]): number => { + if (values.length === 0) return 0; + const ordered = [...values].sort((a, b) => a - b); + const mid = ordered.length >> 1; + return ordered.length % 2 ? ordered[mid] : 0.5 * (ordered[mid - 1] + ordered[mid]); + }; + const ivf = this._scan === null ? null : this._scan.ivf; + return { + recall: recalls.length ? recalls.reduce((a, b) => a + b, 0) / recalls.length : 1, + minRecall: recalls.length ? Math.min(...recalls) : 1, + queries: vectors.length, + k, + nlist: ivf === null ? 0 : ivf.nlist, + nprobe: ivf === null ? this._annNprobe ?? 0 : this._annProbe(ivf), + approximate: used, + annMs: median(annTimes), + exactMs: median(exactTimes), + }; + } + /** * Cosine score for every embedded node, keyed by `type:id`. * @@ -1031,6 +2446,7 @@ export class EmbeddingStore { */ clear(resetDimension: boolean = false): void { this._embeddings.clear(); + this._scan = null; if (resetDimension) this._dimension = null; } @@ -1103,7 +2519,7 @@ export class EmbeddingStore { const ref = [item.type, id] as NodeRef; this._checkDimension(item.vector, `embedding for :${item.type}:${id}`); const key = this._nodeRefToKey(ref); - this._embeddings.set(key, { + this._set(key, { id: key, nodeRef: ref, vector: item.vector, @@ -1314,10 +2730,10 @@ export interface SemanticSearchResult { // Semantic Graph // ============================================================================= -export interface SemanticGraphOptions { +export interface SemanticGraphOptions extends AnnOptions { name?: string; directed?: boolean; - encoder?: EmbeddingEncoder; + encoder?: EmbeddingEncoder | AsyncEncoder; autoEmbed?: boolean; embedFields?: string[]; /** @@ -1427,10 +2843,15 @@ export class SemanticGraph extends ISONGraph { constructor(options: SemanticGraphOptions = {}) { super(options.name || "semantic_graph", options.directed ?? true); - this._encoder = options.encoder || null; + this._encoder = adoptEncoder(options.encoder); this._embeddingStore = new EmbeddingStore(this._encoder || undefined, { version: options.version, onModelMismatch: options.onModelMismatch, + ann: options.ann, + annNlist: options.annNlist, + annNprobe: options.annNprobe, + annMinSize: options.annMinSize, + annSeed: options.annSeed, }); this._autoEmbed = options.autoEmbed ?? true; this._embedFields = options.embedFields || ['name', 'description', 'content', 'text']; @@ -1444,9 +2865,9 @@ export class SemanticGraph extends ISONGraph { return this._encoder; } - setEncoder(encoder: EmbeddingEncoder): void { - this._encoder = encoder; - this._embeddingStore.encoder = encoder; + setEncoder(encoder: EmbeddingEncoder | AsyncEncoder): void { + this._encoder = adoptEncoder(encoder); + this._embeddingStore.encoder = this._encoder; } /** @@ -1510,6 +2931,45 @@ export class SemanticGraph extends ISONGraph { return node; } + /** + * Like {@link addNode}, but awaits the encoder, so auto-embedding works with + * one that makes a network call (an HttpEncoder needs no prefetch). + * + * The text is embedded *before* the node is added: if the encoder fails the + * graph is untouched and the call can simply be retried. + */ + async addNodeAsync( + nodeType: string, + nodeId: number | string, + properties: Record<string, any> = {} + ): Promise<Node> { + let text = ''; + let vector: number[] | null = null; + if (this._autoEmbed && this._encoder) { + text = this.embedTextFor(properties); + if (text) vector = (await this._embeddingStore.encodeTextsAsync([text]))[0]; + } + const node = super.addNode(nodeType, nodeId, properties); + if (vector !== null) this._embeddingStore.add(node.ref, text, vector); + return node; + } + + /** Like {@link addNodeWithEmbed}, awaiting the encoder (see {@link addNodeAsync}). */ + async addNodeWithEmbedAsync( + nodeType: string, + nodeId: number | string, + properties: Record<string, any> = {}, + embedText?: string + ): Promise<Node> { + if (embedText === undefined) { + return this.addNodeAsync(nodeType, nodeId, properties); + } + const vector = (await this._embeddingStore.encodeTextsAsync([embedText]))[0]; + const node = super.addNode(nodeType, nodeId, properties); + this._embeddingStore.add(node.ref, embedText, vector); + return node; + } + /** * Embed a node with given text. */ @@ -1517,6 +2977,18 @@ export class SemanticGraph extends ISONGraph { return this._embeddingStore.add(nodeRef, text); } + /** Like {@link embedNode}, awaiting the encoder. */ + embedNodeAsync(nodeRef: NodeRef, text: string): Promise<string> { + return this._embeddingStore.addAsync(nodeRef, text); + } + + /** Like {@link embedNodes}, awaiting the encoder; stores nothing if it fails. */ + async embedNodesAsync(items: Array<[NodeRef, string]>): Promise<number> { + const pairs = items.filter(([, text]) => Boolean(text)); + if (pairs.length === 0) return 0; + return this._embeddingStore.addBatchAsync(pairs); + } + /** Embed many nodes in a single batched encoder call. */ embedNodes(items: Array<[NodeRef, string]>): number { const pairs = items.filter(([, text]) => Boolean(text)); @@ -1558,6 +3030,41 @@ export class SemanticGraph extends ISONGraph { return node; } + /** + * Like {@link updateNode}, but awaits the encoder when a re-embed is needed. + * + * The new text is embedded before anything changes, so an encoder failure + * leaves the node's properties and vector exactly as they were. + */ + async updateNodeAsync( + nodeType: string, + nodeId: number | string, + properties: Record<string, any> + ): Promise<Node> { + const current = this.getNode(nodeType, nodeId); + const reembed = this._autoEmbed && this._embedFields.some(f => f in (properties ?? {})); + let text = ''; + let vector: number[] | null = null; + if (reembed) { + text = this.embedTextFor({ ...current.properties, ...properties }); + if (text && this._encoder) vector = (await this._embeddingStore.encodeTextsAsync([text]))[0]; + } + const node = super.updateNode(nodeType, nodeId, properties); + if (reembed) { + // Another update may have landed while the encoder was working. + const latest = this.embedTextFor(node.properties); + if (!latest) { + this._embeddingStore.remove(node.ref); + } else if (this._encoder) { + if (vector === null || latest !== text) { + vector = (await this._embeddingStore.encodeTextsAsync([latest]))[0]; + } + this._embeddingStore.add(node.ref, latest, vector); + } + } + return node; + } + /** * Remove node and its embedding. */ @@ -1610,21 +3117,43 @@ export class SemanticGraph extends ISONGraph { * With a gate the store cannot truncate first: the best k *unfiltered* * matches may all be filtered out, so every match is scored, filtered, and * only then cut to k. + * + * With the store's approximate index the candidates are the probed cells + * only. If fewer than k of them pass the gate, the probe is widened (4x each + * step) and the search retried, ending in the exact scan, so a selective + * filter costs time rather than results. */ private _seedSearch( query: string | number[], topK: number | null, nodeType: string | undefined, threshold: number, - gate: ((ref: NodeRef) => boolean) | null + gate: ((ref: NodeRef) => boolean) | null, + exact: boolean = false ): SimilarityResult[] { + const store = this._embeddingStore; if (gate === null) { - return this._embeddingStore.similaritySearch(query, topK, nodeType, threshold); + return store.similaritySearch(query, topK, nodeType, threshold, exact); + } + if (store.ann === null || exact || topK === null || topK <= 0) { + const results = store + .similaritySearch(query, null, nodeType, threshold) + .filter(r => gate(r.nodeRef)); + return topK === null ? results : results.slice(0, Math.max(topK, 0)); + } + const queryVector = store.resolveQuery(query); + for (let scale = 1; ; scale *= 4) { + const { ranked, approximate } = store.annRanked(queryVector, nodeType, threshold, scale); + const results: SimilarityResult[] = []; + for (const result of ranked) { + if (gate(result.nodeRef)) { + results.push(result); + if (results.length >= topK) break; + } + } + if (!approximate) return results; + if (results.length >= topK) return store.annPolish(results, queryVector, threshold); } - const results = this._embeddingStore - .similaritySearch(query, null, nodeType, threshold) - .filter(r => gate(r.nodeRef)); - return topK === null ? results : results.slice(0, Math.max(topK, 0)); } /** @@ -1636,6 +3165,8 @@ export class SemanticGraph extends ISONGraph { * the filter. * @param nodeTypes Keep only nodes whose type is in this list (a bare string * means one type). Combines with `nodeType` and `nodeFilter` by AND. + * @param exact Force the full scan even when the graph was created with an + * approximate index (`ann`); no effect otherwise. */ similaritySearch( query: string | number[], @@ -1643,10 +3174,45 @@ export class SemanticGraph extends ISONGraph { nodeType?: string, threshold: number = 0.0, nodeFilter?: (node: Node) => boolean, - nodeTypes?: string | string[] + nodeTypes?: string | string[], + exact: boolean = false ): SimilarityResult[] { const gate = this._nodeGate(nodeFilter, nodeTypes); - return this._seedSearch(query, topK, nodeType, threshold, gate); + return this._seedSearch(query, topK, nodeType, threshold, gate, exact); + } + + /** Recall@k of the approximate index against the exact scan (see {@link EmbeddingStore.recallCheck}). */ + recallCheck( + queries: string | ArrayLike<number> | Array<string | ArrayLike<number>>, + k: number = 10, + nodeType?: string, + threshold: number = 0.0 + ): RecallReport { + return this._embeddingStore.recallCheck(queries, k, nodeType, threshold); + } + + /** {@link recallCheck} with text queries encoded by awaiting the encoder. */ + recallCheckAsync( + queries: string | ArrayLike<number> | Array<string | ArrayLike<number>>, + k: number = 10, + nodeType?: string, + threshold: number = 0.0 + ): Promise<RecallReport> { + return this._embeddingStore.recallCheckAsync(queries, k, nodeType, threshold); + } + + /** {@link similaritySearch} with a text query encoded by awaiting the encoder. */ + async similaritySearchAsync( + query: string | number[], + topK: number | null = 10, + nodeType?: string, + threshold: number = 0.0, + nodeFilter?: (node: Node) => boolean, + nodeTypes?: string | string[], + exact: boolean = false + ): Promise<SimilarityResult[]> { + const vector = await this._embeddingStore.resolveQueryAsync(query); + return this.similaritySearch(vector, topK, nodeType, threshold, nodeFilter, nodeTypes, exact); } /** @@ -2106,6 +3672,31 @@ export class SemanticGraph extends ISONGraph { return null; } + /** {@link semanticMultiHop} with a text query encoded by awaiting the encoder (options form). */ + async semanticMultiHopAsync( + query: string | number[], + options: MultiHopOptions = {} + ): Promise<SemanticSearchResult[]> { + return this.semanticMultiHop(await this._embeddingStore.resolveQueryAsync(query), options); + } + + /** {@link semanticPath} with a text query encoded by awaiting the encoder (options form). */ + async semanticPathAsync( + query: string | number[], + targetRef: NodeRef, + options: SemanticPathOptions = {} + ): Promise<SemanticSearchResult | null> { + return this.semanticPath(await this._embeddingStore.resolveQueryAsync(query), targetRef, options); + } + + /** {@link semanticSubgraph} with a text query encoded by awaiting the encoder. */ + async semanticSubgraphAsync( + query: string | number[], + options: SubgraphOptions = {} + ): Promise<SemanticGraph> { + return this.semanticSubgraph(await this._embeddingStore.resolveQueryAsync(query), options); + } + /** * Extract the slice of the graph a query is actually about. * @@ -2200,6 +3791,22 @@ export class SemanticGraph extends ISONGraph { return this.embedNodes(batch); } + /** {@link embedAllNodes} awaiting the encoder: all the texts, then the stores. */ + async embedAllNodesAsync( + textFn?: (node: Node) => string, + skipExisting: boolean = false + ): Promise<number> { + const fn = textFn || ((node: Node): string => + this.embedTextFor(node.properties) || `${node.type}:${node.id}`); + const batch: Array<[NodeRef, string]> = []; + for (const node of this.nodes()) { + if (skipExisting && this._embeddingStore.get(node.ref)) continue; + const text = fn(node); + if (text) batch.push([node.ref, text]); + } + return this.embedNodesAsync(batch); + } + /** * Nodes whose stored vector was not made by the current encoder. * @@ -2258,6 +3865,45 @@ export class SemanticGraph extends ISONGraph { return count; } + /** + * {@link reembedStale} awaiting the encoder. + * + * Every stale text is encoded (`batchSize` per encoder call) before anything + * is written, so a failure partway leaves the store as it was. The new + * vectors' length decides whether the pinned dimension changes, which also + * works for an encoder that has not learned its dimension yet. + */ + async reembedStaleAsync(includeUnknown: boolean = false, batchSize: number = 256): Promise<number> { + const stale = this.staleEmbeddings(includeUnknown); + if (stale.length === 0) return 0; + + const store = this._embeddingStore; + const pairs: Array<[NodeRef, string]> = []; + for (const ref of stale) { + const record = store.get(ref); + if (!record) continue; + let text = record.text; + if (!text) { + const node = this._resolveNode(ref); + text = (node ? this.embedTextFor(node.properties) : '') || `${ref[0]}:${ref[1]}`; + } + pairs.push([ref, text]); + } + if (!this._encoder) throw new NoEncoderError("No encoder provided to re-embed with"); + + const step = Math.max(1, Math.floor(batchSize)); + const vectors: number[][] = []; + for (let start = 0; start < pairs.length; start += step) { + const chunk = pairs.slice(start, start + step); + vectors.push(...await store.encodeTextsAsync(chunk.map(([, text]) => text))); + } + if (vectors.length > 0 && store.dimension !== null && store.dimension !== vectors[0].length) { + store.clear(true); + } + pairs.forEach(([ref, text], i) => store.add(ref, text, vectors[i])); + return pairs.length; + } + /** * Build a SemanticGraph from a plain ISONGraph (for example one parsed with * ISONGraph.fromIson), copying its nodes and edges. diff --git a/scripts/js/gen_index_js.py b/scripts/js/gen_index_js.py new file mode 100644 index 0000000..bb08f22 --- /dev/null +++ b/scripts/js/gen_index_js.py @@ -0,0 +1,68 @@ +"""Generate isongraph-vector-js/src/index.js from isongraph-vector-ts/src/index.ts. + +Run from anywhere: python scripts/js/gen_index_js.py (needs npx + typescript +in isongraph-vector-ts). Pass --check to fail if the checked-in file differs. + +tsc emits the JavaScript; this script swaps the ison-graph import and applies +the handful of JS-specific wording tweaks the checked-in file already carries. +""" +import pathlib +import subprocess +import sys +import tempfile + +ROOT = pathlib.Path(__file__).resolve().parents[2] +TS = ROOT / 'isongraph-vector-ts' +OUT = ROOT / 'isongraph-vector-js' / 'src' / 'index.js' +SP = tempfile.mkdtemp(prefix='igv-gen-') + +subprocess.run( + ['npx', 'tsc', 'src/index.ts', '--outDir', SP + '/gen', '--target', 'ES2020', + '--module', 'ESNext', '--moduleResolution', 'node', '--skipLibCheck', + '--newLine', 'lf', '--strict'], + cwd=TS, check=True, shell=True) + +s = open(SP + '/gen/index.js', encoding='utf8', newline='').read().replace('\r\n', '\n') +s = s.replace("from 'ison-graph-ts'", "from 'ison-graph-js'") + +# header +start = s.index('/**') +end = s.index('*/', start) + 2 +header = '''/** + * ISONGraph Vector - Semantic Graph Extension for JavaScript + * + * Extends ISONGraph with embedding-based similarity search and semantic traversal. + * + * Generated from the TypeScript port's src/index.ts, so the two stay + * behaviourally identical. Edit that file rather than this one. + * + * @author Mahesh Vaikri + * @version 1.0.0 + */ +''' +s = header + '\n' + s[end:].lstrip('\n') + +def sub(old, new): + global s + assert old in s, old + s = s.replace(old, new, 1) + +sub('(Python/TypeScript/JavaScript/Rust/C++): a 32-bit FNV-1a', + '(Python/TypeScript/JavaScript/Rust/C++/C#): a 32-bit FNV-1a') +sub('scoring a perfect 1.0 against each other. TypeScript escaped', + 'scoring a perfect 1.0 against each other. JavaScript escaped') +sub('a change to any of the embedFields re-embeds the node', + 'a change to any of embedFields re-embeds the node') +sub('addNode, nothing is re-embedded while no encoder is set.', + 'addNode, re-embedding needs an encoder; without one the vector is kept.') +sub('this._embedFields.some(f => f in properties)', + 'this._embedFields.some(f => f in (properties ?? {}))') + +if '--check' in sys.argv: + current = open(OUT, encoding='utf8', newline='').read().replace('\r\n', '\n') + if current != s: + sys.exit('%s is out of date: run python scripts/js/gen_index_js.py' % OUT) + print('up to date:', OUT) +else: + open(OUT, 'w', encoding='utf8', newline='').write(s) + print('wrote', OUT, len(s.splitlines()), 'lines') diff --git a/scripts/js/strip_types.cjs b/scripts/js/strip_types.cjs new file mode 100644 index 0000000..1cf7750 --- /dev/null +++ b/scripts/js/strip_types.cjs @@ -0,0 +1,104 @@ +// Strip TypeScript types from a test file by deleting ranges, so the layout is kept. +// usage: node strip.cjs <typescript-module-dir> in.ts out.js +const path = require('path'); +const fs = require('fs'); +const ts = require(path.join(process.argv[2], 'typescript')); +const text = fs.readFileSync(process.argv[3], 'utf8').replace(/\r\n/g, '\n'); +const sf = ts.createSourceFile('x.ts', text, ts.ScriptTarget.ES2022, true, ts.ScriptKind.TS); +const cuts = []; +const cut = (a, b) => cuts.push([a, b]); + +function colonBefore(node) { + // start of the ':' token that precedes node.type + let i = node.type.getFullStart(); + while (i > 0 && text[i - 1] !== ':') i--; + return i - 1; +} + +function visit(node) { + switch (node.kind) { + case ts.SyntaxKind.InterfaceDeclaration: + case ts.SyntaxKind.TypeAliasDeclaration: + cut(node.getFullStart(), node.getEnd()); + return; + case ts.SyntaxKind.ImportDeclaration: { + const clause = node.importClause; + if (clause && clause.namedBindings && ts.isNamedImports(clause.namedBindings)) { + const els = clause.namedBindings.elements; + const keep = els.filter(e => !e.isTypeOnly && !(clause.isTypeOnly)); + if (keep.length === 0 && !clause.name) { cut(node.getFullStart(), node.getEnd()); return; } + if (keep.length !== els.length) { + // rebuild the specifier list in place, keeping the original layout of kept names + els.forEach((e, i) => { + if (!e.isTypeOnly) return; + let start = e.getFullStart(); + let end = e.getEnd(); + // eat the trailing comma and whitespace up to the newline + let j = end; + if (text[j] === ',') j++; + while (text[j] === ' ') j++; + if (text[j] === '\n') j++; + // include the leading newline+indent + let k = e.getStart(); + while (k > 0 && (text[k - 1] === ' ')) k--; + cut(k, j); + }); + } + } + break; + } + case ts.SyntaxKind.AsExpression: + case ts.SyntaxKind.SatisfiesExpression: + cut(node.expression.getEnd(), node.getEnd()); + break; + case ts.SyntaxKind.NonNullExpression: + cut(node.getEnd() - 1, node.getEnd()); + break; + case ts.SyntaxKind.ArrowFunction: + if (node.parameters.length === 1 && node.parameters[0].type && !node.typeParameters && + !node.parameters[0].initializer && !node.parameters[0].dotDotDotToken && !node.type) { + const prm = node.parameters[0]; + const open = text.lastIndexOf('(', prm.getStart()); + let close = prm.type.getEnd(); + while (text[close] !== ')') close++; + cut(open, open + 1); + cut(close, close + 1); + } + break; + case ts.SyntaxKind.Parameter: + if (node.questionToken) cut(node.questionToken.getStart(), node.questionToken.getEnd()); + break; + case ts.SyntaxKind.CallExpression: + case ts.SyntaxKind.NewExpression: + if (node.typeArguments && node.typeArguments.length) { + const a = node.expression.getEnd(); + cut(a, node.typeArguments.end + 1); + } + break; + default: + break; + } + if (node.type && !ts.isTypeAliasDeclaration(node) && node.kind !== ts.SyntaxKind.AsExpression && + node.kind !== ts.SyntaxKind.TypeAssertionExpression && node.kind !== ts.SyntaxKind.SatisfiesExpression && + node.kind !== ts.SyntaxKind.TypePredicate && !ts.isTypeNode(node)) { + cut(colonBefore(node), node.type.getEnd()); + } + if (node.typeParameters && node.typeParameters.length && !ts.isInterfaceDeclaration(node)) { + cut(node.typeParameters.pos - 1, node.typeParameters.end + 1); + } + ts.forEachChild(node, visit); +} +visit(sf); + +cuts.sort((a, b) => a[0] - b[0] || b[1] - a[1]); +const accepted = []; +let curEnd = -1; +for (const c of cuts) { + if (c[0] < curEnd) continue; // nested inside an accepted range + accepted.push(c); + curEnd = c[1]; +} +let out = text; +for (const [a, b] of accepted.reverse()) out = out.slice(0, a) + out.slice(b); +out = out.replace(/[ \t]+$/gm, '').replace(/\n{3,}/g, '\n\n'); +fs.writeFileSync(process.argv[4], out.replace(/\n/g, '\r\n'), 'utf8');