diff --git a/CLAUDE.md b/CLAUDE.md index 90f3d443..2c06fcd3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -53,7 +53,9 @@ before changing code. Commands, toolchain setup and environment gotchas are in `segment` → `vector` → `cache` → `build` → `version_sql`; `cache` → `external_ids`, `selection`; and `external_ids`, `selection`, `interrupts` → `traversal_sql`, `compute_sql` (siblings: neither depends - on the other); `compute_sql` → `spectral_sql`; `build`, `compute_sql` + on the other); `interrupts` → `build` (the build maps edge batches on + the compute pool, `interrupts::pool`); `compute_sql` → `spectral_sql`; + `build`, `compute_sql` → `community_sql`; `parallel_worker` → `cache`, `traversal_sql`; never the reverse. - Each module's `mod.rs` doc comment states its responsibility. Add a diff --git a/DESIGN.md b/DESIGN.md index 7b56cf6c..6d075fe7 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -430,7 +430,7 @@ M9 (0.5.0) moves loaded versions into dynamic shared memory so backends share on ### 5.7 Identity, lifecycle and visibility (M10 design) -*Designed 2026-10-23 for M10 (0.6.0), after 0.5.0's release; built so far: steps 1 (the spike), 2 (`IdMap`), 3 (`TextKeys`), 5 (ownership transfer), 6 (dropped roles) and 7 (quotas). The decisions are §14 items 41–49: items 44, 45 and 46 were decided by the owner on 2026-10-23, as recommended, and the rest are decided here. Claims about Postgres were read in the PG 15–18 sources (`tcop/utility.c`, `tcop/cmdtaglist.h`, `catalog/pg_shdepend.c`, `commands/event_trigger.c`, `utils/backend_progress.h`, `utils/adt/pgstatfuncs.c`, `utils/wait_event.h`, `utils/misc/guc_tables.c`); where a claim needed code to confirm it, step 1 of the work breakdown did so on PG 15, 16 and 18 (Spike results, below), and the text reflects what it found.* M10 has five items, which share little code. `text` node ids change the id map and everything that maps external ids. Ownership transfer, cleanup after a dropped role, and quotas change the catalog's lifecycle. Observability adds a per-backend table in shared memory beside the registry of §5.6. Each part states what Postgres allows an extension on PG 15–18 and where pg_warren stops. Nothing in M10 changes a kernel, the CSR layout, the segment format of existing id types, or the registry's format, so 0.6.0 needs no restart to keep sharing on PG 17 and 18 (§5.6). +*Designed 2026-10-23 for M10 (0.6.0), after 0.5.0's release; built so far: steps 1 (the spike), 2 (`IdMap`), 3 (`TextKeys`), 4 (text graphs end to end), 5 (ownership transfer), 6 (dropped roles) and 7 (quotas). The decisions are §14 items 41–49: items 44, 45 and 46 were decided by the owner on 2026-10-23, as recommended, and the rest are decided here. Claims about Postgres were read in the PG 15–18 sources (`tcop/utility.c`, `tcop/cmdtaglist.h`, `catalog/pg_shdepend.c`, `commands/event_trigger.c`, `utils/backend_progress.h`, `utils/adt/pgstatfuncs.c`, `utils/wait_event.h`, `utils/misc/guc_tables.c`); where a claim needed code to confirm it, step 1 of the work breakdown did so on PG 15, 16 and 18 (Spike results, below), and the text reflects what it found.* M10 has five items, which share little code. `text` node ids change the id map and everything that maps external ids. Ownership transfer, cleanup after a dropped role, and quotas change the catalog's lifecycle. Observability adds a per-backend table in shared memory beside the registry of §5.6. Each part states what Postgres allows an extension on PG 15–18 and where pg_warren stops. Nothing in M10 changes a kernel, the CSR layout, the segment format of existing id types, or the registry's format, so 0.6.0 needs no restart to keep sharing on PG 17 and 18 (§5.6). #### Text node ids (§14 items 41, 42) @@ -480,6 +480,14 @@ A new fuzz target, `decode_text_id_map`, covers the three kinds. **The shared cache.** A text version's region holds the three arrays in place. `u8`, `u64` and `u32` are `InPlace`: any bits are valid, and validation covers the rest. Neither the region format nor the registry format changes, since no existing array's layout moves, and a 0.5.0 backend never opens a text version because `resolve` refuses an id type it does not know. Admission counts `versions.bytes` alone, since no second direction is built (§5.5). Parallel labels are as for any graph. +*Built (step 4):* text graphs work end to end, with these decisions and departures from the text above. +- *Types.* `pg_warren`'s generic parameter is now the key storage, not the external id: `with_id_type!` yields `FixedKeys`, `FixedKeys`, `FixedKeys<[u8; 16]>` or `TextKeys`. `IdDatum` became four traits by concern: `source::IdDatum` (a datum to a key and back; `in_place` borrows a text from its tuple), `source::InPlaceId` (the fixed types' in-place layout, `unsafe`), `segment::StoredIds` (writing and reading an id map's segments) and `cache::CachedIds` (the region's arrays, their fill and views, and what a load builds beside the segments: the dense-id direction for a fixed type, nothing for text). A text *argument* is copied into an owned `Vec` rather than borrowed: an argument is one id, and borrowing it would tie every argument's lifetime to its datum for no measurable gain. Rows (the build's node and edge queries, `vector_from_query`, `gather`'s messages, `similarity`'s pairs) are read in place, through a `source::Row` that `for_each_tuple` lends for one call, so that nothing read from it outlives the memory context that holds what reading it detoasted; that context is reset after each batch of 10,000 rows and within a batch once it holds 64 MB (review of PR #60: long toasted ids were held uncharged until the batch ended). +- *Encoding check (as recorded under step 3): when a datum is built.* `IdDatum::datum` for text runs `pg_verifymbstr` on the id's bytes (22021 for bytes invalid in the database encoding), then builds the `text` (an id longer than a `text` can hold is 54000). A load checks only the `id_bytes` header's encoding (55000): scanning every id would make a load O(bytes) for ids the database had already validated as `text` when the build read them, while the datum check costs only the ids a call returns. +- *The build's intake, on the compute pool, for every id type.* The main thread copies each batch of edge rows into a `warren_core::graph::EdgeBatch` (relations, weights and endpoints; a text endpoint's bytes), charged to the build budget and sized to 1/32 of it, so that a small budget is left to the edge list: half of that share for the rows' fixed columns (at most 65,536 rows), the rest for their text ids. A batch is mapped when its rows are full or once the bytes its text ids hold (not their room) pass their share, and when it is emptied it frees text room past that share, which one long id may have grown (review of PR #60: comparing the batch's capacity with the whole share mapped every row of a text build below a budget of about 124 MiB, and of an `int8` build at budgets its rows filled exactly). `Workers::run` then maps the batch's endpoints on the pool (chunks of 4,096 rows, the cancellation flag checked per chunk, the main thread polling interrupts) through `IndexedKeys::index`: a `TextIndex` for text, built once per build, charged to the build budget and freed with the intake, and the id map itself for a fixed type, whose binary searches parallelise as well. `EdgeIntake::take` takes the mapped rows in order, so the edge list is exactly the one row-by-row intake makes (a property test, for both kinds of key). The pool moved from `compute_sql` to `interrupts::pool` so that `build` can use it, which adds `interrupts` → `build` to the feature DAG, and `compute_sqlstate` moved to `extension` beside `graph_sqlstate`. +- *Per-call bulk lookups: indexed past 65,536 ids, charged exactly.* `external_ids::DenseIds` answers a call's first 65,536 lookups by binary search (about 0.3 s at 10M ids). The next lookup, on a text id map, charges `TextIndex::bytes_for(id_map)` (two slots a node of the id space's target width: 80 MB at 10M nodes) to the call's `ComputeMemory` and only then builds the index, whose size is exactly that charge; every later lookup probes it. A call whose compute memory has no room for the index keeps the binary search for the rest of the call, which needs no memory, and does not try again (review of PR #60: it was refused with 53400 though the search would have answered); an index that fails to build is raised with its `GraphError`'s SQLSTATE (`graph_sqlstate`). `test/sql/text_ids_bulk.sql` crosses the threshold in `vector_from_query` and `similarity`, with room for the index and without. A build of the index costs about as much as 10^5 binary searches, so the threshold indexes only calls that would spend longer searching. Fixed-width ids always use the binary search of their pairs. `vector_from_query`, `gather`'s message query and `similarity`'s pairs map through it; a step filter, `node_filter` and the other point lookups use the binary search. +- *The region.* A text version's region holds arrays `IdBytes` (7), `IdOffsets` (8) and `IdOrder` (9) in its table of contents, its format still 1. Its fill checks the `id_bytes` header with `check_id_bytes_header` against the database encoding before the bytes are copied, and its views are `IdMap::from_text_segments`. +- *Tests.* `test/sql/text_ids.sql` runs every function that maps ids on a text graph and on the same graph with `bigint` ids, its IRIs zero-padded so that both orders give each node the same dense id, and compares the results; `a4_typed_retrieval_iris.sql` is Appendix A.4 written with IRIs, against recursive CTEs over the text keys and the fixture's truth; `shared_cache_text.sql` shares a text version across backends. `text_ids_toasted.sql` builds from ids of over 2 kB stored out of line, compressed and not; `text_ids_sql_ascii.sql` maps ids that are no encoding's UTF-8 in a SQL_ASCII database; `shared_cache_parallel_text.sql` runs a text graph's `PARALLEL SAFE` reads in a worker. `dump.py` dumps and restores a text graph, and restores one from a UTF8 database into a LATIN1 one, where reading it is refused (55000); `upgrade.py` builds one on every updated catalog. The soak's `--ids text` (§12) names node *n* `https://example.org/resource/`. + **Upgrade.** `graphs.id_type` has no `CHECK` (the code checks the type), so text ids change nothing in the catalog. Existing `int4`, `int8` and `uuid` graphs, their segments (format 1) and their regions are unchanged. `test/upgrade.py` checks that every release's graphs still read, refresh and compact after `ALTER EXTENSION pg_warren UPDATE`. The update script adds only what the other M10 items add. Older library builds cannot read a text graph, and a downgrade is not supported. #### Ownership transfer (§14 item 43) @@ -598,6 +606,7 @@ A slot counts since its backend started, and the totals since the table was crea - `vector::quota`: the quota, `set_vector_quota` and `vector_usage`. - `activity`: the activity table, `backends`, `progress` and `report_progress`. It is a leaf that `cache`, `build`, `version_sql`, `compute_sql`, `spectral_sql` and `community_sql` depend on. - `cache::introspection`: `cache()`. +- `interrupts::pool`: the compute pool, moved from `compute_sql` so that the build maps edge batches on it (step 4). The feature DAG gains no cycle. `CLAUDE.md`'s module list and DAG need the new modules once the code exists. @@ -1056,8 +1065,8 @@ For state that must be per-vertex and bytes-shaped with logic that neither SQL n ### 6.9 Parallelism - A Postgres backend is single-threaded and **no thread other than the main one may call any Postgres function** (palloc, elog, SPI, catalog access, …). Pure compute over extension-owned buffers (CSR, vectors, message buffers) is safe on worker threads. `warren-core` therefore takes plain slices and a thread count, and uses `rayon` with a dedicated pool sized by the `warren.threads` GUC (default: half the cores, min 1). The binding layer does all Postgres interaction before and after each kernel call — including, for `gather`, reading the message relation through SPI on the main thread before fan-out and emitting result rows after. -- *Implemented (M4, `compute_sql::settings`):* `warren.threads` (`USERSET`, 1–1024, default half of `available_parallelism`, at least 1) sizes one compute pool per backend, built on first use and rebuilt when the setting changes. The pool's threads are started with every signal blocked, so Postgres's handlers (which touch backend state non-atomically) only ever run on the main thread. The main-thread poll is `pg_warren::interrupts::PostgresInterrupts`, shared with traversal; a cancel marked while a kernel runs stops it with `query_canceled` (tested). `Workers::run` catches the unwind of the ERROR that `CHECK_FOR_INTERRUPTS` raises in the poll (rayon's scope does), waits for the job to see its cancellation flag and return, then resumes the unwind. That is sound only because nothing calls into Postgres between the catch and the resume; no Postgres call may be added to that window. The test marks the cancel from the main-thread poll, so no worker thread ever touches Postgres state. -- Interrupt handling: the main thread polls `CHECK_FOR_INTERRUPTS()` between kernel calls and sets a cancellation flag that worker loops check every ~64K vertices. **Implemented (M4):** `compute::Workers::run` runs a kernel as a job on the pool while the calling (main) thread waits, polling the `interrupts::Interrupts` hook (shared with traversal) every millisecond. When a poll says stop, or unwinds (pgrx raising the ERROR), a drop guard raises the job's `Cancellation` flag and the rayon scope waits for the job to stop before the error propagates. The scope catches that unwind (`halt_unwinding`), waits for the job, then resumes it; this is the one place pg_warren holds a caught Postgres ERROR, and it is sound only because nothing between the catch and the resume calls into Postgres. A panicking job marks itself finished from a drop guard, so the scope returns at once and re-raises the panic, which pgrx turns into an ERROR. Worker tasks check the flag at the start of each task of at most 4,096 rows (4,096 elements for `vector_map`, 65,536 for reductions); the push scatter checks it every 65,536 rows, as do `gather`'s sequential sorts (outbox, inbox). Use `pgrx::check_for_interrupts!()`. pgrx's `pg_sys` FFI is `#[pg_guard]`ed, so an ERROR raised inside it unwinds the Rust frames (destructors run, `RefCell` borrows release). Never call unguarded FFI. *A FATAL during a kernel (M9 step 5 review):* `pg_terminate_backend` raises a FATAL from the main thread's poll, which exits the process without unwinding, so nothing joins the pool while its threads may still read a shared version's segment, which Postgres unmaps as it exits (a crash, and a restart of the cluster). `Workers` records its running job, and `Workers::stop` cancels it and waits for it; the first pool a backend builds registers a `before_shmem_exit` callback (`compute_sql::settings`), which runs before Postgres unmaps anything and stops every pool of the backend still in use. `test/sql/shared_cache_exit_mid_kernel.sql` terminates a backend mid-kernel and checks that the cluster did not restart. +- *Implemented (M4, `compute_sql::settings`; `interrupts::pool` since M10 step 4):* `warren.threads` (`USERSET`, 1–1024, default half of `available_parallelism`, at least 1) sizes one compute pool per backend, built on first use and rebuilt when the setting changes. The pool's threads are started with every signal blocked, so Postgres's handlers (which touch backend state non-atomically) only ever run on the main thread. The main-thread poll is `pg_warren::interrupts::PostgresInterrupts`, shared with traversal; a cancel marked while a kernel runs stops it with `query_canceled` (tested). `Workers::run` catches the unwind of the ERROR that `CHECK_FOR_INTERRUPTS` raises in the poll (rayon's scope does), waits for the job to see its cancellation flag and return, then resumes the unwind. That is sound only because nothing calls into Postgres between the catch and the resume; no Postgres call may be added to that window. The test marks the cancel from the main-thread poll, so no worker thread ever touches Postgres state. +- Interrupt handling: the main thread polls `CHECK_FOR_INTERRUPTS()` between kernel calls and sets a cancellation flag that worker loops check every ~64K vertices. **Implemented (M4):** `compute::Workers::run` runs a kernel as a job on the pool while the calling (main) thread waits, polling the `interrupts::Interrupts` hook (shared with traversal) every millisecond. When a poll says stop, or unwinds (pgrx raising the ERROR), a drop guard raises the job's `Cancellation` flag and the rayon scope waits for the job to stop before the error propagates. The scope catches that unwind (`halt_unwinding`), waits for the job, then resumes it; this is the one place pg_warren holds a caught Postgres ERROR, and it is sound only because nothing between the catch and the resume calls into Postgres. A panicking job marks itself finished from a drop guard, so the scope returns at once and re-raises the panic, which pgrx turns into an ERROR. Worker tasks check the flag at the start of each task of at most 4,096 rows (4,096 elements for `vector_map`, 65,536 for reductions); the push scatter checks it every 65,536 rows, as do `gather`'s sequential sorts (outbox, inbox). Use `pgrx::check_for_interrupts!()`. pgrx's `pg_sys` FFI is `#[pg_guard]`ed, so an ERROR raised inside it unwinds the Rust frames (destructors run, `RefCell` borrows release). Never call unguarded FFI. *A FATAL during a kernel (M9 step 5 review):* `pg_terminate_backend` raises a FATAL from the main thread's poll, which exits the process without unwinding, so nothing joins the pool while its threads may still read a shared version's segment, which Postgres unmaps as it exits (a crash, and a restart of the cluster). `Workers` records its running job, and `Workers::stop` cancels it and waits for it; the first pool a backend builds registers a `before_shmem_exit` callback (`interrupts::pool`), which runs before Postgres unmaps anything and stops every pool of the backend still in use. `test/sql/shared_cache_exit_mid_kernel.sql` terminates a backend mid-kernel and checks that the cluster did not restart. - Errors are raised only through pgrx (`ErrorReport` with a standard SQLSTATE; detail and hint as needed), never by calling `pg_sys` error functions directly. Nothing in `pg_warren` uses `catch_unwind` around SPI; pgrx already installs the boundary at each `#[pg_extern]`. - Memory for vectors, message buffers and work buffers is allocated with the Rust allocator, not `palloc`, and accounted against a `warren.max_compute_memory` GUC; exceeding it raises an error rather than OOMing the server (*M4:* per call, §6.2). Graph-sized buffers are reserved with `try_reserve`, so allocator failure is also an error (`53200`), not an abort. - v1.1: a **compute background worker** that owns the DSM-resident graph and runs jobs submitted by backends via a shared queue, so compute does not compete with query backends and large jobs can outlive a client session. Results land in `warren.vectors`; `warren.job_status()` reports progress. @@ -1552,6 +1561,17 @@ A pooled backend's own memory is 4–12 MB with the shared cache against the ver *The soak with the shared cache* (`bench/results/soak/*-m9-shared.json`; full scale, PG 16 preloaded, the release build above; case a at the default cap, b-merged at `warren.shared_cache_memory = 8GB`, without its second version, library or community runs). Every query stage reads the shared version, and a backend's private memory falls by the version: case a's stages peak at 0.01–3.11 GB private against 1.22–4.31 GB in the 0.4.0 run, `pagerank` at 1.06 GB against 2.27 GB. The cold load into the shared cache takes 1.5–1.7 s for case a (2.4–3.0 s privately) and 7.4–33.3 s for b-merged's 6.0 GB (22.4–27.1 s privately; the first load reads the segments from disk). Times of stages that read a loaded version are within the run-to-run noise of the 0.4.0 run, except where 0.4.0 reloaded: on b-merged at the default `warren.max_cache_memory`, which no longer counts the version, `bfs` takes 0.89 s (25.1 s), `neighbours` 0.1 ms (19.2 s), `components` 7.6 s (28.6 s) and `pagerank` 16.5 s (38.4 s), as fast as 0.4.0 at an 8GB `warren.max_cache_memory`; `eigen`, refused there before (53400: the version and the solve over the two limits), now runs (135.7 s for 200 steps; it does not converge, as on case a). On case a, `louvain` completed (627.5 s, five levels, modularity 0.121) where the 0.4.0 run was cancelled at 382.7 s by the disk guard; `leiden` fails as in 0.4.0 (53400, compute memory). The `refresh` row is unchanged (a build is private): b-merged's misses at 4.42 GB, as in 0.4.0. +**Text node ids (M10 step 4, §5.7; `bench/results/soak/a-scale2*-m10*.json`).** Case a with IRIs as node ids (`bench/soak.py --ids text`: node *n* is `https://example.org/resource/`, 29 bytes and its digits) against the same case with `bigint` ids, *at half scale* (5M nodes, 50M edges), for the disk: the full-scale text run needs about 16 GB free (the 3.6 GB edge table, the recursive CTE's index, two or three 1.44 GB text versions and WAL, above `--disk-floor`'s 4 GB), and the machine had about 8 GB, shared with other work, so the full run is still to do. *Measured on* the soak's machine and settings (above), a release build of 5f905b4 from a private PG 16.15 tree, without the recursive CTE baseline or the community runs; the second version was skipped for disk, so `diff` and `distance` were not run. Each pair ran back to back under a load average of 2.0–4.4 from other agents' builds, so a time can move by 15–20% for noise alone (`pagerank`, which maps no ids, took 9.1 s against 6.7 s privately and 8.4 s against 7.2 s shared). Figures are text / `bigint`: +- *Map size:* the version is 719.5 / 560.6 MB, each within 300 bytes of `bench/soak.py`'s `segment_estimate` (§5.3). The text id map is 238.9 MB (the IRIs' bytes 178.9 MB, `id_offsets` 40 MB, `id_order` 20 MB), against 80 MB of pairs. Cached, 719.5 / 600.6 MB, since a text version builds no dense-id direction when it loads. At full scale: 1.44 / 1.12 GB. +- *Build:* `refresh` takes 56.4 / 41.8 s (58.6 / 40.1 s in the shared pair) and peaks at 0.92 / 0.79 GB private (VmHWM 2.08 / 1.95 GB); from the generator query, 61.1 / 46.4 s. Text costs about 35% more: the intake copies each row's two IRIs into its batch and maps them on the pool through the text index. +- *Cold load:* 1.03 / 0.77 s privately, 1.06 / 0.79 s into the shared cache. +- *Lookups:* `neighbours` of one IRI 0.1 / 0.1 ms (median of 20 calls), `most_similar` 1.1 / 1.0 ms, `similarity` of 100,000 pairs 1.31 / 0.41 s (past 65,536 lookups, so through a per-call index, which costs 40 MB at 5M nodes). +- *Traversal:* `bfs` to depth 6 (1.07M rows) 0.95 / 0.49 s, `reachable_set` (5.0M rows) 3.72 / 2.22 s. Every row's `text` datum is built and checked against the encoding, so row emission, already most of these calls' time (above), about doubles. The kernels that return no ids are unchanged: `components` 2.77 / 2.74 s, `eigen` for 200 steps 75.9 / 77.3 s, and the library within noise (`clustering` 27.0 / 28.8 s, betweenness 40.5 / 40.1 s, `minimum_spanning_forest` 22.5 / 22.0 s). +- *Peaks:* the stages that read the version peak about 0.12 GB higher privately, the difference in the versions' sizes (`pagerank` 1.28 / 1.16 GB, `eigen` 2.28 / 2.17 GB). The calls that return a row per node peak higher still: `components` 1.28 / 0.80 GB and `minimum_spanning_forest` 1.77 / 0.98 GB, for their rows of text ids. Not investigated. +- *Shared cache:* the shared version is 719.5 / 600.6 MB, and the private peaks fall by the version, as for `bigint` ids: `pagerank` 0.56 / 0.56 GB, `bfs` 0.12 / 0.12 GB, `components` 0.56 / 0.20 GB. + +Every §12 target that ran was met at this scale except `reachable_set`, which misses for both id types (12–13 / 22–23M edges/s). + **Parallel query (M9, §5.6 step 9, §14 item 33; `bench/parallel.py`, `bench/results/parallel/`).** A probe table of node pairs (half an edge's endpoints, half random), `SELECT count(*) FROM probe WHERE warren.has_edge('g1', src, dst)`, `sum(warren.degree('g1', src))` and `count(*)` over `LATERAL warren.neighbours('g1', src)`, under `EXPLAIN (ANALYZE, TIMING OFF)` in a session whose leader has already read the graph, with the graph reads `PARALLEL RESTRICTED` (0.4.0's label) and `PARALLEL SAFE` (set by `ALTER FUNCTION` on the same library, a release build of cf9dce1), on PG 16.15 preloaded (the shared cache) and without the preload (every backend and worker loads privately), on the soak's machine (4 vCPUs). With workers, the planner's parallel costs are set to 0 ("forced"); "planned" is the planner's own choice at its default costs and `max_parallel_workers_per_gather = 4`. Medians of three runs (two at full scale), ms: | Case a, probe rows, read | Restricted, any | Safe, 0 workers | Safe, 2 forced | Safe, 4 forced | Safe, planned | @@ -1793,7 +1813,7 @@ Renames are cheap before 1.0 and costly after. Also a written stability and depr 42. **How do text ids compare, and what is their SQL surface?** (M10 design; §5.7.) **Decided:** - *Comparison:* bytewise in the database encoding, which is `COLLATE "C"` order. Equality under every deterministic collation is already bytewise in Postgres. `create_graph` and `refresh` refuse an id column whose collation is nondeterministic (42804, hinting `COLLATE "C"` or `lower()`), because pg_warren could not match what SQL considers equal. - *Types:* `text` and `varchar` (recorded as `text`). `bpchar`, `name` and `citext` are refused, with the cast to write. - - *Encoding:* the database encoding is recorded in the `id_bytes` header, and a version restored into a database of another encoding is refused (55000). + - *Encoding:* the database encoding is recorded in the `id_bytes` header, and a version restored into a database of another encoding is refused (55000). An id's bytes are checked against the encoding when a `text` datum is built from them (22021). - *Length:* no limit beyond memory. - *SQL:* item 1's `anyelement`/`anyarray` dispatch, unchanged, with no overloads. String literals need `::text` (a lone untyped literal cannot resolve an `anyelement`; `ARRAY['…']` is `text[]`), a `varchar` argument is accepted where the result holds no ids, and the type witnesses are `NULL::text`. - *Upgrade:* existing graphs and their segments and regions are unchanged, and there is no catalog change for text ids. diff --git a/bench/results/soak/README.md b/bench/results/soak/README.md index bbfa7c27..89d4e54a 100644 --- a/bench/results/soak/README.md +++ b/bench/results/soak/README.md @@ -351,3 +351,26 @@ backend touched (about 1.1 GB) as well as the shared version. An earlier attempt at case a was discarded: a second copy of the harness had been started by mistake on the same machine, and one of the two was killed for memory during `louvain`. + +## Text node ids (M10 step 4, §5.7) + +`a-scale2-text-m10.json` and `a-scale2-m10.json` (private), then +`a-scale2-text-m10-shared.json` and `a-scale2-m10-shared.json` (shared +cache): case a at half scale (5M nodes, 50M edges) with IRIs and with +`bigint` ids, PostgreSQL 16.15 from a private tree, a release build of +5f905b4 (`pg_warren.so` sha256 92c54dcb…), on the same 4-vCPU Xeon, +harness 5f905b4, one after the other on 2026-10-23, under a load average +of 2.0–4.4 from other agents' builds on the machine: + + bench/soak.py a --ids text --label=-m10 --scale 2 --no-cte --no-eigen-steps \ + --eigen-max-iter 200 --skip communities --disk-floor 3221225472 \ + --pg-config /var/tmp/m10-pgtree/usr/lib/postgresql/16/bin/pg_config --parent /var/tmp/m10-soak \ + --extension-revision 5f905b4 --extension-profile release --threads 4 --statement-timeout 20min + bench/soak.py a --ids bigint --label=-m10 … # otherwise as above + bench/soak.py a --ids text --label=-m10-shared --shared-cache-memory 4GB \ + --skip library --skip second-version … # and --ids bigint + +Half scale, without the recursive CTE baseline, because the machine had +about 8 GB of disk free, shared with other work; the full-scale text run +needs about 16 GB (DESIGN.md §12). Both private runs skipped their second +version for the disk floor, so `diff` and `distance` were not run. diff --git a/bench/results/soak/a-scale2-m10-shared.json b/bench/results/soak/a-scale2-m10-shared.json new file mode 100644 index 00000000..752998b1 --- /dev/null +++ b/bench/results/soak/a-scale2-m10-shared.json @@ -0,0 +1,617 @@ +{ + "provenance": { + "argv": [ + "bench/soak.py", + "a", + "--scale", + "2", + "--no-cte", + "--no-eigen-steps", + "--eigen-max-iter", + "200", + "--skip", + "communities", + "--disk-floor", + "3221225472", + "--pg-config", + "/var/tmp/m10-pgtree/usr/lib/postgresql/16/bin/pg_config", + "--parent", + "/var/tmp/m10-soak", + "--results", + "/var/tmp/m10-soak/results", + "--extension-revision", + "5f905b4", + "--extension-profile", + "release", + "--threads", + "4", + "--statement-timeout", + "20min", + "--ids", + "bigint", + "--label=-m10-shared", + "--shared-cache-memory", + "4GB", + "--skip", + "library", + "--skip", + "second-version" + ], + "harness": { + "revision": "5f905b45bc076768657d5c383d1c5bea22965178", + "dirty": false + }, + "extension": { + "revision": "5f905b4", + "profile": "release", + "library": "/var/tmp/m10-pgtree/usr/lib/postgresql/16/lib/pg_warren.so", + "library_sha256": "92c54dcb438b65c6c804ccddf96e1292412d2edd776a29f684a8afdcd481b667" + } + }, + "case": "a", + "nodes": 5000000, + "edges": 50000000, + "weighted": false, + "ids": "bigint", + "relations": 1, + "store_merged": true, + "threads": 4, + "node_order": "id", + "eigen_max_iter": 200, + "statement_timeout": "20min", + "skip": [ + "communities", + "library", + "second-version" + ], + "planned_stages": [ + "cold_load_probe", + "default: bfs depth 6", + "default: components", + "default: eigen", + "default: pagerank", + "default: prewarm", + "default: reachable_set", + "refresh" + ], + "hardware": { + "cpus": 4, + "model": "Intel(R) Xeon(R) Processor", + "memory_bytes": 16872804352, + "kernel": "6.18.44-fc-v153", + "load_average": [ + 4.18408203125, + 3.51171875, + 3.21484375 + ] + }, + "started_at": "2026-10-23T19:55:39+00:00", + "stages": { + "table 0: create": { + "sql": "CREATE UNLOGGED TABLE soak_removed ()", + "seconds": [ + 0.0004 + ], + "median_seconds": 0.0004, + "rows": [], + "first_rows": [ + null + ], + "notices": [], + "warnings": [], + "rss_anon_before": 5570560, + "peak_rss_anon": 5570560, + "vm_hwm": 34435072, + "rss_anon_after": 5570560, + "rss_shmem_after": 14868480 + }, + "table 1: create": { + "sql": "CREATE UNLOGGED TABLE soak_edges (src int4 NOT NULL, dst int4 NOT NULL) WITH (autovacuum_enabled = false)", + "seconds": [ + 0.0003 + ], + "median_seconds": 0.0003, + "rows": [], + "first_rows": [ + null + ], + "notices": [], + "warnings": [], + "rss_anon_before": 5570560, + "peak_rss_anon": 5570560, + "vm_hwm": 34439168, + "rss_anon_after": 5570560, + "rss_shmem_after": 14872576 + }, + "table 2: insert": { + "sql": "INSERT INTO soak_edges SELECT 1 + floor(random() * 5000000)::int4 AS src, 1 + floor(random() * 5000000)::int4 AS dst FROM (SELECT setseed(0.25) OFFSET 0) seeded CROSS JOIN generate_series(1, 50000000)", + "seconds": [ + 16.9887 + ], + "median_seconds": 16.9887, + "rows": [], + "first_rows": [ + null + ], + "notices": [], + "warnings": [], + "rss_anon_before": 5570560, + "peak_rss_anon": 95416320, + "vm_hwm": 1254121472, + "rss_anon_after": 32468992, + "rss_shmem_after": 1095847936 + }, + "table 3: analyze": { + "sql": "ANALYZE soak_edges", + "seconds": [ + 0.2392 + ], + "median_seconds": 0.2392, + "rows": [], + "first_rows": [ + null + ], + "notices": [], + "warnings": [], + "rss_anon_before": 32468992, + "peak_rss_anon": 32473088, + "vm_hwm": 1191669760, + "rss_anon_after": 32473088, + "rss_shmem_after": 1095847936 + }, + "create_graph": { + "sql": "SELECT warren.create_graph('soak', $n$SELECT id FROM generate_series(1::bigint, 5000000) id$n$, $q$SELECT src::bigint, dst::bigint FROM soak_edges$q$, weighted := false, has_relations := false, store_merged := true, node_order := 'id')", + "seconds": [ + 0.0181 + ], + "median_seconds": 0.0181, + "rows": [ + "" + ], + "first_rows": [ + "" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 3268608, + "peak_rss_anon": 9691136, + "vm_hwm": 74063872, + "rss_anon_after": 9691136, + "rss_shmem_after": 10850304 + }, + "refresh": { + "sql": "SELECT warren.refresh('soak')", + "seconds": [ + 40.1496 + ], + "median_seconds": 40.1496, + "rows": [ + "1" + ], + "first_rows": [ + "1" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 9691136, + "peak_rss_anon": 790695936, + "vm_hwm": 1949499392, + "rss_anon_after": 48046080, + "rss_shmem_after": 1111080960 + }, + "refresh from the generator": { + "sql": "SELECT warren.refresh('soak_generated')", + "seconds": [ + 45.0779 + ], + "median_seconds": 45.0779, + "rows": [ + "2" + ], + "first_rows": [ + "2" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 9785344, + "peak_rss_anon": 827908096, + "vm_hwm": 1025052672, + "rss_anon_after": 48218112, + "rss_shmem_after": 621756416 + }, + "cold_load_probe": { + "sql": "SELECT warren.prewarm('soak')", + "seconds": [ + 0.786 + ], + "median_seconds": 0.786, + "rows": [ + "1" + ], + "first_rows": [ + "1" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 3268608, + "peak_rss_anon": 12300288, + "vm_hwm": 1228038144, + "rss_anon_after": 12300288, + "rss_shmem_after": 1200726016 + }, + "default: prewarm": { + "sql": "WITH before AS MATERIALIZED (SELECT (SELECT pg_stat_get_xact_tuples_returned('warren.segments'::regclass) + pg_stat_get_xact_tuples_fetched('warren.segments'::regclass) + sum(pg_stat_get_xact_tuples_fetched(indexrelid))::bigint FROM pg_index WHERE indrelid = 'warren.segments'::regclass AND 1 IS NOT NULL) AS reads), run AS MATERIALIZED (SELECT warren.prewarm('soak') AS result FROM before WHERE before.reads IS NOT NULL), after AS MATERIALIZED (SELECT (SELECT pg_stat_get_xact_tuples_returned('warren.segments'::regclass) + pg_stat_get_xact_tuples_fetched('warren.segments'::regclass) + sum(pg_stat_get_xact_tuples_fetched(indexrelid))::bigint FROM pg_index WHERE indrelid = 'warren.segments'::regclass AND run.result IS NOT NULL) AS reads FROM run) SELECT after.reads - before.reads, run.result FROM before, run, after", + "seconds": [ + 0.7615 + ], + "median_seconds": 0.7615, + "rows": [ + "1120|1" + ], + "first_rows": [ + "1120|1" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 3268608, + "peak_rss_anon": 12652544, + "vm_hwm": 1229266944, + "rss_anon_after": 12632064, + "rss_shmem_after": 1201664000 + }, + "default: bfs depth 6": { + "sql": "SELECT bench_seconds($soak$SELECT count(*) FROM warren.bfs('soak', ARRAY[1::bigint], 6)$soak$)", + "seconds": [ + 0.5171, + 0.5239, + 0.5627 + ], + "median_seconds": 0.5239, + "rows": [ + "0.562206" + ], + "first_rows": [ + "0.51602", + "0.523396", + "0.562206" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 12681216, + "peak_rss_anon": 119001088, + "vm_hwm": 1339478016, + "rss_anon_after": 7520256, + "rss_shmem_after": 1201799168, + "server_seconds": [ + 0.51602, + 0.523396, + 0.562206 + ], + "server_median_seconds": 0.523396 + }, + "default: bfs depth 6 count": { + "sql": "SELECT count(*) FROM warren.bfs('soak', ARRAY[1::bigint], 6)", + "seconds": [ + 0.5532 + ], + "median_seconds": 0.5532, + "rows": [ + "1068779" + ], + "first_rows": [ + "1068779" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 7520256, + "peak_rss_anon": 118280192, + "vm_hwm": 1338728448, + "rss_anon_after": 7446528, + "rss_shmem_after": 1201799168 + }, + "default: neighbours": { + "sql": "SELECT bench_seconds($soak$SELECT count(*) FROM warren.neighbours('soak', 1::bigint)$soak$)", + "seconds": [ + 0.0008, + 0.0003, + 0.0002, + 0.0002, + 0.0002, + 0.0003, + 0.0002, + 0.0002, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0002, + 0.0001, + 0.0001, + 0.0001, + 0.0001 + ], + "median_seconds": 0.0001, + "rows": [ + "5.6e-05" + ], + "first_rows": [ + "0.000456", + "9.9e-05", + "7.3e-05", + "6.6e-05", + "7e-05", + "0.000106", + "8.2e-05", + "6.5e-05", + "6e-05", + "5.9e-05", + "5.7e-05", + "5.8e-05", + "5.7e-05", + "6.3e-05", + "6e-05", + "5.7e-05", + "5.7e-05", + "5.5e-05", + "5.5e-05", + "5.6e-05" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 7446528, + "peak_rss_anon": 7475200, + "vm_hwm": 1228312576, + "rss_anon_after": 7475200, + "rss_shmem_after": 1201799168, + "server_seconds": [ + 0.000456, + 9.9e-05, + 7.3e-05, + 6.6e-05, + 7e-05, + 0.000106, + 8.2e-05, + 6.5e-05, + 6e-05, + 5.9e-05, + 5.7e-05, + 5.8e-05, + 5.7e-05, + 6.3e-05, + 6e-05, + 5.7e-05, + 5.7e-05, + 5.5e-05, + 5.5e-05, + 5.6e-05 + ], + "server_median_seconds": 6e-05 + }, + "default: reachable_set": { + "sql": "SELECT count(*) FROM warren.reachable_set('soak', ARRAY[1::bigint])", + "seconds": [ + 2.3008 + ], + "median_seconds": 2.3008, + "rows": [ + "4999752" + ], + "first_rows": [ + "4999752" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 7475200, + "peak_rss_anon": 252530688, + "vm_hwm": 1473183744, + "rss_anon_after": 38641664, + "rss_shmem_after": 1201799168 + }, + "default: reachable_set stats": { + "stats": [ + "{\"graph\": \"soak\", \"version\": 1, \"function\": \"reachable_set\", \"truncated\": false, \"nodes_visited\": 4999752, \"edges_traversed\": 11392720}" + ] + }, + "default: components": { + "sql": "SELECT count(*) FROM (SELECT component_id FROM warren.components('soak', NULL::bigint) GROUP BY 1) c", + "seconds": [ + 2.7275 + ], + "median_seconds": 2.7275, + "rows": [ + "1" + ], + "first_rows": [ + "1" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 38641664, + "peak_rss_anon": 204173312, + "vm_hwm": 1424777216, + "rss_anon_after": 26980352, + "rss_shmem_after": 1201815552 + }, + "default: pagerank": { + "sql": "WITH before AS MATERIALIZED (SELECT (SELECT pg_stat_get_xact_tuples_returned('warren.segments'::regclass) + pg_stat_get_xact_tuples_fetched('warren.segments'::regclass) + sum(pg_stat_get_xact_tuples_fetched(indexrelid))::bigint FROM pg_index WHERE indrelid = 'warren.segments'::regclass AND 1 IS NOT NULL) AS reads), run AS MATERIALIZED (SELECT warren.pagerank('soak', 'pr', tol := 0, max_iter := 20) AS result FROM before WHERE before.reads IS NOT NULL), after AS MATERIALIZED (SELECT (SELECT pg_stat_get_xact_tuples_returned('warren.segments'::regclass) + pg_stat_get_xact_tuples_fetched('warren.segments'::regclass) + sum(pg_stat_get_xact_tuples_fetched(indexrelid))::bigint FROM pg_index WHERE indrelid = 'warren.segments'::regclass AND run.result IS NOT NULL) AS reads FROM run) SELECT after.reads - before.reads, run.result FROM before, run, after", + "seconds": [ + 7.2307 + ], + "median_seconds": 7.2307, + "rows": [ + "0|20" + ], + "first_rows": [ + "0|20" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 26980352, + "peak_rss_anon": 559460352, + "vm_hwm": 1839964160, + "rss_anon_after": 68628480, + "rss_shmem_after": 1260625920 + }, + "default: pagerank from cold": { + "sql": "WITH before AS MATERIALIZED (SELECT (SELECT pg_stat_get_xact_tuples_returned('warren.segments'::regclass) + pg_stat_get_xact_tuples_fetched('warren.segments'::regclass) + sum(pg_stat_get_xact_tuples_fetched(indexrelid))::bigint FROM pg_index WHERE indrelid = 'warren.segments'::regclass AND 1 IS NOT NULL) AS reads), run AS MATERIALIZED (SELECT warren.pagerank('soak', 'pr', tol := 0, max_iter := 20) AS result FROM before WHERE before.reads IS NOT NULL), after AS MATERIALIZED (SELECT (SELECT pg_stat_get_xact_tuples_returned('warren.segments'::regclass) + pg_stat_get_xact_tuples_fetched('warren.segments'::regclass) + sum(pg_stat_get_xact_tuples_fetched(indexrelid))::bigint FROM pg_index WHERE indrelid = 'warren.segments'::regclass AND run.result IS NOT NULL) AS reads FROM run) SELECT after.reads - before.reads, run.result FROM before, run, after", + "seconds": [ + 7.504 + ], + "median_seconds": 7.504, + "rows": [ + "1120|20" + ], + "first_rows": [ + "1120|20" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 28631040, + "peak_rss_anon": 559484928, + "vm_hwm": 1882083328, + "rss_anon_after": 68648960, + "rss_shmem_after": 1302544384 + }, + "default: eigen": { + "sql": "SELECT count(*), max(residual) FROM warren.eigen('soak', 20, 'normalized_laplacian', 'smallest', tol := 1e-06, max_iter := 200)", + "seconds": [ + 73.9283 + ], + "median_seconds": 73.9283, + "rows": [ + "20|0.005383903658682934" + ], + "first_rows": [ + "20|0.005383903658682934" + ], + "notices": [], + "warnings": [ + "WARNING: eigen did not converge in 200 steps; the largest residual is 5.383903658682934e-3 (raise max_iter or tol)" + ], + "rss_anon_before": 68648960, + "peak_rss_anon": 1568911360, + "vm_hwm": 2891919360, + "rss_anon_after": 68681728, + "rss_shmem_after": 1302544384 + } + }, + "skipped": { + "second_version": "--skip second-version", + "library": "--skip library", + "communities": "--skip communities" + }, + "pg_warren": "0.6.0", + "postgres": "16.15 (Ubuntu 16.15-0ubuntu0.24.04.1)", + "edge_table_bytes": 1812856832, + "version_1": { + "seq": 1, + "bytes": 560624716, + "label": null, + "staged": false, + "current": true, + "vectors": 0, + "version": 1, + "built_at": "2026-10-23T19:55:56.507189+00:00", + "id_space": 5000000, + "built_lsn": "0/1CAE000", + "staleness": "00:01:25.781304", + "edge_count": 49999948, + "node_count": 5000000, + "label_vector": null, + "parent_graph": null, + "target_width": 4, + "dropped_edges": 0, + "compacted_from": null, + "parent_version": null, + "deduplicated_edges": 52 + }, + "cached_bytes": null, + "shared_bytes": 600625172, + "cache_memory_bytes": 4294967296, + "settings": { + "default": [ + "fsync=off", + "full_page_writes=off", + "jit=on", + "maintenance_work_mem=1GB", + "max_parallel_workers_per_gather=2", + "max_wal_size=1GB", + "shared_buffers=1GB", + "shared_preload_libraries=pg_warren", + "statement_timeout=20min", + "synchronous_commit=off", + "wal_level=minimal", + "warren.max_build_memory=4GB", + "warren.max_cache_memory=4GB", + "warren.max_compute_memory=4GB", + "warren.shared_cache_memory=4GB", + "warren.threads=4", + "work_mem=64MB" + ] + }, + "library_version": 1, + "finished_at": "2026-10-23T19:59:00+00:00", + "free_disk_after": 7025618944, + "load_average_after": [ + 3.91845703125, + 3.4794921875, + 3.23779296875 + ], + "measured": { + "cores": 4, + "bfs_seconds": 0.523396, + "components_seconds": 2.7275, + "pagerank_seconds": 7.2307, + "refresh_seconds": 40.1496, + "refresh_peak_bytes": 1949499392, + "cold_load_seconds": 0.7615, + "load_segment_reads": 1120, + "pagerank_segment_reads": 0, + "pagerank_loads": 0.0, + "pagerank_cold_segment_reads": 1120, + "pagerank_cold_seconds": 7.504, + "pagerank_cold_loads": 1.0, + "eigen_residual": 0.005383903658682934, + "eigen_converged": false, + "eigen_seconds_per_step": 0.36964149999999996, + "reachable_row_entries_per_second": 4951634.214186369, + "reachable_edges_per_second": 21731549.02642559, + "library": {}, + "communities": {} + }, + "verdicts": [ + { + "workload": "reachable_set, full component", + "target": "> 50M edges/s", + "measured": "22M edges/s", + "met": false + }, + { + "workload": "components()", + "target": "< 10 s", + "measured": "2.7 s", + "met": true + }, + { + "workload": "pagerank, 20 iterations", + "target": "< 30 s (8 threads)", + "measured": "7.2 s", + "met": true + }, + { + "workload": "refresh() incl. reverse CSR", + "target": "< 8 min, < 4 GiB RAM", + "measured": "40.1 s, peak 1.95 GB", + "met": true + }, + { + "workload": "cold load into backend", + "target": "< 15 s", + "measured": "0.8 s", + "met": true + } + ] +} diff --git a/bench/results/soak/a-scale2-m10.json b/bench/results/soak/a-scale2-m10.json new file mode 100644 index 00000000..b35c1dd3 --- /dev/null +++ b/bench/results/soak/a-scale2-m10.json @@ -0,0 +1,952 @@ +{ + "provenance": { + "argv": [ + "bench/soak.py", + "a", + "--scale", + "2", + "--no-cte", + "--no-eigen-steps", + "--eigen-max-iter", + "200", + "--skip", + "communities", + "--disk-floor", + "3221225472", + "--pg-config", + "/var/tmp/m10-pgtree/usr/lib/postgresql/16/bin/pg_config", + "--parent", + "/var/tmp/m10-soak", + "--results", + "/var/tmp/m10-soak/results", + "--extension-revision", + "5f905b4", + "--extension-profile", + "release", + "--threads", + "4", + "--statement-timeout", + "20min", + "--ids", + "bigint", + "--label=-m10" + ], + "harness": { + "revision": "5f905b45bc076768657d5c383d1c5bea22965178", + "dirty": false + }, + "extension": { + "revision": "5f905b4", + "profile": "release", + "library": "/var/tmp/m10-pgtree/usr/lib/postgresql/16/lib/pg_warren.so", + "library_sha256": "92c54dcb438b65c6c804ccddf96e1292412d2edd776a29f684a8afdcd481b667" + } + }, + "case": "a", + "nodes": 5000000, + "edges": 50000000, + "weighted": false, + "ids": "bigint", + "relations": 1, + "store_merged": true, + "threads": 4, + "node_order": "id", + "eigen_max_iter": 200, + "statement_timeout": "20min", + "skip": [ + "communities" + ], + "planned_stages": [ + "cold_load_probe", + "default: bfs depth 6", + "default: components", + "default: eigen", + "default: pagerank", + "default: prewarm", + "default: reachable_set", + "diff", + "library: betweenness_centrality", + "library: closeness_centrality", + "library: clustering", + "library: core_number", + "library: eigenvector_centrality", + "library: harmonic_centrality", + "library: hits", + "library: katz_centrality", + "library: minimum_spanning_forest", + "library: most_similar", + "library: similar_pairs", + "library: similarity", + "library: strong_components_vector", + "refresh" + ], + "hardware": { + "cpus": 4, + "model": "Intel(R) Xeon(R) Processor", + "memory_bytes": 16872804352, + "kernel": "6.18.44-fc-v153", + "load_average": [ + 3.57568359375, + 3.0859375, + 2.95751953125 + ] + }, + "started_at": "2026-10-23T19:45:30+00:00", + "stages": { + "table 0: create": { + "sql": "CREATE UNLOGGED TABLE soak_removed ()", + "seconds": [ + 0.0003 + ], + "median_seconds": 0.0003, + "rows": [], + "first_rows": [ + null + ], + "notices": [], + "warnings": [], + "rss_anon_before": 5554176, + "peak_rss_anon": 5554176, + "vm_hwm": 35196928, + "rss_anon_after": 5554176, + "rss_shmem_after": 15233024 + }, + "table 1: create": { + "sql": "CREATE UNLOGGED TABLE soak_edges (src int4 NOT NULL, dst int4 NOT NULL) WITH (autovacuum_enabled = false)", + "seconds": [ + 0.0003 + ], + "median_seconds": 0.0003, + "rows": [], + "first_rows": [ + null + ], + "notices": [], + "warnings": [], + "rss_anon_before": 5554176, + "peak_rss_anon": 5554176, + "vm_hwm": 35201024, + "rss_anon_after": 5554176, + "rss_shmem_after": 15237120 + }, + "table 2: insert": { + "sql": "INSERT INTO soak_edges SELECT 1 + floor(random() * 5000000)::int4 AS src, 1 + floor(random() * 5000000)::int4 AS dst FROM (SELECT setseed(0.25) OFFSET 0) seeded CROSS JOIN generate_series(1, 50000000)", + "seconds": [ + 17.3329 + ], + "median_seconds": 17.3329, + "rows": [], + "first_rows": [ + null + ], + "notices": [], + "warnings": [], + "rss_anon_before": 5554176, + "peak_rss_anon": 95412224, + "vm_hwm": 1254522880, + "rss_anon_after": 32464896, + "rss_shmem_after": 1095913472 + }, + "table 3: analyze": { + "sql": "ANALYZE soak_edges", + "seconds": [ + 0.2358 + ], + "median_seconds": 0.2358, + "rows": [], + "first_rows": [ + null + ], + "notices": [], + "warnings": [], + "rss_anon_before": 32464896, + "peak_rss_anon": 32468992, + "vm_hwm": 1192202240, + "rss_anon_after": 32468992, + "rss_shmem_after": 1095913472 + }, + "create_graph": { + "sql": "SELECT warren.create_graph('soak', $n$SELECT id FROM generate_series(1::bigint, 5000000) id$n$, $q$SELECT src::bigint, dst::bigint FROM soak_edges$q$, weighted := false, has_relations := false, store_merged := true, node_order := 'id')", + "seconds": [ + 0.0213 + ], + "median_seconds": 0.0213, + "rows": [ + "" + ], + "first_rows": [ + "" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 3198976, + "peak_rss_anon": 9691136, + "vm_hwm": 74174464, + "rss_anon_after": 9691136, + "rss_shmem_after": 10723328 + }, + "refresh": { + "sql": "SELECT warren.refresh('soak')", + "seconds": [ + 41.7892 + ], + "median_seconds": 41.7892, + "rows": [ + "1" + ], + "first_rows": [ + "1" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 9691136, + "peak_rss_anon": 790695936, + "vm_hwm": 1949818880, + "rss_anon_after": 48050176, + "rss_shmem_after": 1111195648 + }, + "refresh from the generator": { + "sql": "SELECT warren.refresh('soak_generated')", + "seconds": [ + 46.3726 + ], + "median_seconds": 46.3726, + "rows": [ + "2" + ], + "first_rows": [ + "2" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 9818112, + "peak_rss_anon": 827916288, + "vm_hwm": 1025548288, + "rss_anon_after": 48230400, + "rss_shmem_after": 622034944 + }, + "cold_load_probe": { + "sql": "SELECT warren.prewarm('soak')", + "seconds": [ + 0.7661 + ], + "median_seconds": 0.7661, + "rows": [ + "1" + ], + "first_rows": [ + "1" + ], + "notices": [ + "NOTICE: version 1 (600624716 bytes) cannot be kept within warren.max_cache_memory (65536 bytes, 0 of them held by transient vectors), so it is held only until the statement ends, the bytes past the cap charged to warren.max_compute_memory, and reloaded by each statement that reads it; raise warren.max_cache_memory" + ], + "warnings": [], + "rss_anon_before": 3211264, + "peak_rss_anon": 612896768, + "vm_hwm": 1230675968, + "rss_anon_after": 12267520, + "rss_shmem_after": 600281088 + }, + "default: prewarm": { + "sql": "WITH before AS MATERIALIZED (SELECT (SELECT pg_stat_get_xact_tuples_returned('warren.segments'::regclass) + pg_stat_get_xact_tuples_fetched('warren.segments'::regclass) + sum(pg_stat_get_xact_tuples_fetched(indexrelid))::bigint FROM pg_index WHERE indrelid = 'warren.segments'::regclass AND 1 IS NOT NULL) AS reads), run AS MATERIALIZED (SELECT warren.prewarm('soak') AS result FROM before WHERE before.reads IS NOT NULL), after AS MATERIALIZED (SELECT (SELECT pg_stat_get_xact_tuples_returned('warren.segments'::regclass) + pg_stat_get_xact_tuples_fetched('warren.segments'::regclass) + sum(pg_stat_get_xact_tuples_fetched(indexrelid))::bigint FROM pg_index WHERE indrelid = 'warren.segments'::regclass AND run.result IS NOT NULL) AS reads FROM run) SELECT after.reads - before.reads, run.result FROM before, run, after", + "seconds": [ + 0.7159 + ], + "median_seconds": 0.7159, + "rows": [ + "1120|1" + ], + "first_rows": [ + "1120|1" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 3207168, + "peak_rss_anon": 613216256, + "vm_hwm": 1231835136, + "rss_anon_after": 613191680, + "rss_shmem_after": 601137152 + }, + "default: bfs depth 6": { + "sql": "SELECT bench_seconds($soak$SELECT count(*) FROM warren.bfs('soak', ARRAY[1::bigint], 6)$soak$)", + "seconds": [ + 0.4901, + 0.5016, + 0.4901 + ], + "median_seconds": 0.4901, + "rows": [ + "0.489608" + ], + "first_rows": [ + "0.489317", + "0.50122", + "0.489608" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 613277696, + "peak_rss_anon": 719626240, + "vm_hwm": 1339404288, + "rss_anon_after": 608133120, + "rss_shmem_after": 601354240, + "server_seconds": [ + 0.489317, + 0.50122, + 0.489608 + ], + "server_median_seconds": 0.489608 + }, + "default: bfs depth 6 count": { + "sql": "SELECT count(*) FROM warren.bfs('soak', ARRAY[1::bigint], 6)", + "seconds": [ + 0.4935 + ], + "median_seconds": 0.4935, + "rows": [ + "1068779" + ], + "first_rows": [ + "1068779" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 608133120, + "peak_rss_anon": 718974976, + "vm_hwm": 1338753024, + "rss_anon_after": 608141312, + "rss_shmem_after": 601354240 + }, + "default: neighbours": { + "sql": "SELECT bench_seconds($soak$SELECT count(*) FROM warren.neighbours('soak', 1::bigint)$soak$)", + "seconds": [ + 0.0007, + 0.0003, + 0.0002, + 0.0001, + 0.0001, + 0.0002, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001 + ], + "median_seconds": 0.0001, + "rows": [ + "4.3e-05" + ], + "first_rows": [ + "0.000424", + "9.2e-05", + "6.3e-05", + "5.2e-05", + "5.3e-05", + "6.4e-05", + "5.1e-05", + "4.7e-05", + "4.7e-05", + "4.6e-05", + "4.5e-05", + "4.5e-05", + "4.7e-05", + "4.6e-05", + "4.5e-05", + "5e-05", + "4.6e-05", + "4.5e-05", + "4.5e-05", + "4.3e-05" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 608141312, + "peak_rss_anon": 608161792, + "vm_hwm": 1228386304, + "rss_anon_after": 608161792, + "rss_shmem_after": 601354240, + "server_seconds": [ + 0.000424, + 9.2e-05, + 6.3e-05, + 5.2e-05, + 5.3e-05, + 6.4e-05, + 5.1e-05, + 4.7e-05, + 4.7e-05, + 4.6e-05, + 4.5e-05, + 4.5e-05, + 4.7e-05, + 4.6e-05, + 4.5e-05, + 5e-05, + 4.6e-05, + 4.5e-05, + 4.5e-05, + 4.3e-05 + ], + "server_median_seconds": 4.7e-05 + }, + "default: reachable_set": { + "sql": "SELECT count(*) FROM warren.reachable_set('soak', ARRAY[1::bigint])", + "seconds": [ + 2.2152 + ], + "median_seconds": 2.2152, + "rows": [ + "4999752" + ], + "first_rows": [ + "4999752" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 608161792, + "peak_rss_anon": 850714624, + "vm_hwm": 1473138688, + "rss_anon_after": 639262720, + "rss_shmem_after": 601354240 + }, + "default: reachable_set stats": { + "stats": [ + "{\"graph\": \"soak\", \"version\": 1, \"function\": \"reachable_set\", \"truncated\": false, \"nodes_visited\": 4999752, \"edges_traversed\": 11392720}" + ] + }, + "default: components": { + "sql": "SELECT count(*) FROM (SELECT component_id FROM warren.components('soak', NULL::bigint) GROUP BY 1) c", + "seconds": [ + 2.7416 + ], + "median_seconds": 2.7416, + "rows": [ + "1" + ], + "first_rows": [ + "1" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 639266816, + "peak_rss_anon": 804798464, + "vm_hwm": 1424736256, + "rss_anon_after": 627605504, + "rss_shmem_after": 601370624 + }, + "default: pagerank": { + "sql": "WITH before AS MATERIALIZED (SELECT (SELECT pg_stat_get_xact_tuples_returned('warren.segments'::regclass) + pg_stat_get_xact_tuples_fetched('warren.segments'::regclass) + sum(pg_stat_get_xact_tuples_fetched(indexrelid))::bigint FROM pg_index WHERE indrelid = 'warren.segments'::regclass AND 1 IS NOT NULL) AS reads), run AS MATERIALIZED (SELECT warren.pagerank('soak', 'pr', tol := 0, max_iter := 20) AS result FROM before WHERE before.reads IS NOT NULL), after AS MATERIALIZED (SELECT (SELECT pg_stat_get_xact_tuples_returned('warren.segments'::regclass) + pg_stat_get_xact_tuples_fetched('warren.segments'::regclass) + sum(pg_stat_get_xact_tuples_fetched(indexrelid))::bigint FROM pg_index WHERE indrelid = 'warren.segments'::regclass AND run.result IS NOT NULL) AS reads FROM run) SELECT after.reads - before.reads, run.result FROM before, run, after", + "seconds": [ + 6.6971 + ], + "median_seconds": 6.6971, + "rows": [ + "0|20" + ], + "first_rows": [ + "0|20" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 627605504, + "peak_rss_anon": 1159450624, + "vm_hwm": 1840074752, + "rss_anon_after": 669237248, + "rss_shmem_after": 660250624 + }, + "default: pagerank from cold": { + "sql": "WITH before AS MATERIALIZED (SELECT (SELECT pg_stat_get_xact_tuples_returned('warren.segments'::regclass) + pg_stat_get_xact_tuples_fetched('warren.segments'::regclass) + sum(pg_stat_get_xact_tuples_fetched(indexrelid))::bigint FROM pg_index WHERE indrelid = 'warren.segments'::regclass AND 1 IS NOT NULL) AS reads), run AS MATERIALIZED (SELECT warren.pagerank('soak', 'pr', tol := 0, max_iter := 20) AS result FROM before WHERE before.reads IS NOT NULL), after AS MATERIALIZED (SELECT (SELECT pg_stat_get_xact_tuples_returned('warren.segments'::regclass) + pg_stat_get_xact_tuples_fetched('warren.segments'::regclass) + sum(pg_stat_get_xact_tuples_fetched(indexrelid))::bigint FROM pg_index WHERE indrelid = 'warren.segments'::regclass AND run.result IS NOT NULL) AS reads FROM run) SELECT after.reads - before.reads, run.result FROM before, run, after", + "seconds": [ + 7.7194 + ], + "median_seconds": 7.7194, + "rows": [ + "1120|20" + ], + "first_rows": [ + "1120|20" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 28614656, + "peak_rss_anon": 1159905280, + "vm_hwm": 1882501120, + "rss_anon_after": 669687808, + "rss_shmem_after": 702005248 + }, + "default: eigen": { + "sql": "SELECT count(*), max(residual) FROM warren.eigen('soak', 20, 'normalized_laplacian', 'smallest', tol := 1e-06, max_iter := 200)", + "seconds": [ + 77.2953 + ], + "median_seconds": 77.2953, + "rows": [ + "20|0.005383903658682934" + ], + "first_rows": [ + "20|0.005383903658682934" + ], + "notices": [], + "warnings": [ + "WARNING: eigen did not converge in 200 steps; the largest residual is 5.383903658682934e-3 (raise max_iter or tol)" + ], + "rss_anon_before": 669687808, + "peak_rss_anon": 2169950208, + "vm_hwm": 2892341248, + "rss_anon_after": 669720576, + "rss_shmem_after": 702005248 + }, + "library: eigenvector_centrality": { + "sql": "SELECT warren.eigenvector_centrality('soak', 'library', tol := 0, max_iter := 20)", + "seconds": [ + 6.986 + ], + "median_seconds": 6.986, + "rows": [ + "20" + ], + "first_rows": [ + "20" + ], + "notices": [], + "warnings": [ + "WARNING: eigenvector_centrality did not converge in 20 iterations" + ], + "rss_anon_before": 3207168, + "peak_rss_anon": 918523904, + "vm_hwm": 1598976000, + "rss_anon_after": 699924480, + "rss_shmem_after": 660746240 + }, + "library: katz_centrality": { + "sql": "SELECT warren.katz_centrality('soak', 'library', alpha := 0.01, tol := 0, max_iter := 20)", + "seconds": [ + 5.1214 + ], + "median_seconds": 5.1214, + "rows": [ + "20" + ], + "first_rows": [ + "20" + ], + "notices": [], + "warnings": [ + "WARNING: katz_centrality did not converge in 20 iterations" + ], + "rss_anon_before": 659935232, + "peak_rss_anon": 1001127936, + "vm_hwm": 1722806272, + "rss_anon_after": 693522432, + "rss_shmem_after": 702468096 + }, + "library: hits": { + "sql": "SELECT warren.hits('soak', 'library_hubs', 'library_authorities', tol := 0, max_iter := 20)", + "seconds": [ + 8.4529 + ], + "median_seconds": 8.4529, + "rows": [ + "20" + ], + "first_rows": [ + "20" + ], + "notices": [], + "warnings": [ + "WARNING: hits did not converge in 20 iterations" + ], + "rss_anon_before": 653520896, + "peak_rss_anon": 961150976, + "vm_hwm": 1766510592, + "rss_anon_after": 733548544, + "rss_shmem_after": 786132992 + }, + "library: clustering": { + "sql": "SELECT warren.clustering('soak', 'library')", + "seconds": [ + 28.7609 + ], + "median_seconds": 28.7609, + "rows": [ + "" + ], + "first_rows": [ + "" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 653545472, + "peak_rss_anon": 1493553152, + "vm_hwm": 2298908672, + "rss_anon_after": 693583872, + "rss_shmem_after": 827936768 + }, + "library: core_number": { + "sql": "SELECT warren.core_number('soak', 'library', persist := false)", + "seconds": [ + 9.9797 + ], + "median_seconds": 9.9797, + "rows": [ + "14" + ], + "first_rows": [ + "14" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 653582336, + "peak_rss_anon": 1573588992, + "vm_hwm": 2420633600, + "rss_anon_after": 653582336, + "rss_shmem_after": 827936768 + }, + "library: strong_components_vector": { + "sql": "SELECT warren.strong_components_vector('soak', 'library', persist := false)", + "seconds": [ + 8.4723 + ], + "median_seconds": 8.4723, + "rows": [ + "476" + ], + "first_rows": [ + "476" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 653582336, + "peak_rss_anon": 799535104, + "vm_hwm": 1646727168, + "rss_anon_after": 652673024, + "rss_shmem_after": 827936768 + }, + "library: minimum_spanning_forest": { + "sql": "SELECT count(*) FROM warren.minimum_spanning_forest('soak', NULL::bigint)", + "seconds": [ + 21.9616 + ], + "median_seconds": 21.9616, + "rows": [ + "4999999" + ], + "first_rows": [ + "4999999" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 652673024, + "peak_rss_anon": 978845696, + "vm_hwm": 1826123776, + "rss_anon_after": 667230208, + "rss_shmem_after": 827936768 + }, + "library: betweenness_centrality": { + "sql": "SELECT warren.betweenness_centrality('soak', 'library', k := 16, persist := false)", + "seconds": [ + 40.0763 + ], + "median_seconds": 40.0763, + "rows": [ + "16" + ], + "first_rows": [ + "16" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 667230208, + "peak_rss_anon": 1307258880, + "vm_hwm": 2154799104, + "rss_anon_after": 667234304, + "rss_shmem_after": 827936768 + }, + "library: closeness_centrality": { + "sql": "SELECT warren.closeness_centrality('soak', 'library', k := 16, persist := false)", + "seconds": [ + 3.6492 + ], + "median_seconds": 3.6492, + "rows": [ + "16" + ], + "first_rows": [ + "16" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 667234304, + "peak_rss_anon": 947253248, + "vm_hwm": 1794863104, + "rss_anon_after": 667262976, + "rss_shmem_after": 827936768 + }, + "library: harmonic_centrality": { + "sql": "SELECT warren.harmonic_centrality('soak', 'library', k := 16, persist := false)", + "seconds": [ + 3.6053 + ], + "median_seconds": 3.6053, + "rows": [ + "16" + ], + "first_rows": [ + "16" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 667262976, + "peak_rss_anon": 907276288, + "vm_hwm": 1754742784, + "rss_anon_after": 707280896, + "rss_shmem_after": 827936768 + }, + "library: most_similar": { + "sql": "SELECT count(*) FROM warren.most_similar('soak', 1::bigint)", + "seconds": [ + 0.001 + ], + "median_seconds": 0.001, + "rows": [ + "10" + ], + "first_rows": [ + "10" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 667279360, + "peak_rss_anon": 667283456, + "vm_hwm": 1515634688, + "rss_anon_after": 667283456, + "rss_shmem_after": 827936768 + }, + "library: similarity": { + "sql": "SELECT count(*) FROM warren.similarity('soak', $p$SELECT (1 + (i * 7919) % 5000000)::bigint, (1 + (i * 104729) % 5000000)::bigint FROM generate_series(1::bigint, 100000) i$p$, NULL::bigint)", + "seconds": [ + 0.4082 + ], + "median_seconds": 0.4082, + "rows": [ + "100000" + ], + "first_rows": [ + "100000" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 667283456, + "peak_rss_anon": 667295744, + "vm_hwm": 1515655168, + "rss_anon_after": 667295744, + "rss_shmem_after": 827944960 + }, + "library: similar_pairs": { + "sql": "SELECT count(*) FROM warren.similar_pairs('soak', NULL::bigint, nodes := 'library_mask')", + "seconds": [ + 5.1283 + ], + "median_seconds": 5.1283, + "rows": [ + "500000" + ], + "first_rows": [ + "500000" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 707305472, + "peak_rss_anon": 1813716992, + "vm_hwm": 2661732352, + "rss_anon_after": 716144640, + "rss_shmem_after": 827944960 + } + }, + "skipped": { + "second_version": "disk: 4.53 GB free, a second version of 560.62 MB would leave less than the floor", + "communities": "--skip communities" + }, + "pg_warren": "0.6.0", + "postgres": "16.15 (Ubuntu 16.15-0ubuntu0.24.04.1)", + "edge_table_bytes": 1812856832, + "version_1": { + "seq": 1, + "bytes": 560624716, + "label": null, + "staged": false, + "current": true, + "vectors": 0, + "version": 1, + "built_at": "2026-10-23T19:45:47.735039+00:00", + "id_space": 5000000, + "built_lsn": "0/1CAE000", + "staleness": "00:01:28.67013", + "edge_count": 49999948, + "node_count": 5000000, + "label_vector": null, + "parent_graph": null, + "target_width": 4, + "dropped_edges": 0, + "compacted_from": null, + "parent_version": null, + "deduplicated_edges": 52 + }, + "cached_bytes": 600624716, + "shared_bytes": 0, + "cache_memory_bytes": 4294967296, + "settings": { + "default": [ + "fsync=off", + "full_page_writes=off", + "jit=on", + "maintenance_work_mem=1GB", + "max_parallel_workers_per_gather=2", + "max_wal_size=1GB", + "shared_buffers=1GB", + "shared_preload_libraries=", + "statement_timeout=20min", + "synchronous_commit=off", + "wal_level=minimal", + "warren.max_build_memory=4GB", + "warren.max_cache_memory=4GB", + "warren.max_compute_memory=4GB", + "warren.shared_cache_memory=4GB", + "warren.threads=4", + "work_mem=64MB" + ] + }, + "library_version": 1, + "finished_at": "2026-10-23T19:51:20+00:00", + "free_disk_after": 7025860608, + "load_average_after": [ + 4.271484375, + 3.5869140625, + 3.181640625 + ], + "measured": { + "cores": 4, + "bfs_seconds": 0.489608, + "components_seconds": 2.7416, + "pagerank_seconds": 6.6971, + "refresh_seconds": 41.7892, + "refresh_peak_bytes": 1949818880, + "cold_load_seconds": 0.7159, + "load_segment_reads": 1120, + "pagerank_segment_reads": 0, + "pagerank_loads": 0.0, + "pagerank_cold_segment_reads": 1120, + "pagerank_cold_seconds": 7.7194, + "pagerank_cold_loads": 1.0, + "eigen_residual": 0.005383903658682934, + "eigen_converged": false, + "eigen_seconds_per_step": 0.3864765, + "reachable_row_entries_per_second": 5142975.803539184, + "reachable_edges_per_second": 22571301.914048396, + "cached_bytes": 600624716, + "id_space": 5000000, + "max_cache_memory": 4294967296, + "library": { + "eigenvector_centrality": { + "seconds": 6.986, + "peak_rss_anon": 918523904 + }, + "katz_centrality": { + "seconds": 5.1214, + "peak_rss_anon": 1001127936 + }, + "hits": { + "seconds": 8.4529, + "peak_rss_anon": 961150976 + }, + "clustering": { + "seconds": 28.7609, + "peak_rss_anon": 1493553152 + }, + "core_number": { + "seconds": 9.9797, + "peak_rss_anon": 1573588992 + }, + "strong_components_vector": { + "seconds": 8.4723, + "peak_rss_anon": 799535104 + }, + "minimum_spanning_forest": { + "seconds": 21.9616, + "peak_rss_anon": 978845696 + }, + "betweenness_centrality": { + "seconds": 40.0763, + "peak_rss_anon": 1307258880 + }, + "closeness_centrality": { + "seconds": 3.6492, + "peak_rss_anon": 947253248 + }, + "harmonic_centrality": { + "seconds": 3.6053, + "peak_rss_anon": 907276288 + }, + "most_similar": { + "seconds": 0.001, + "peak_rss_anon": 667283456 + }, + "similarity": { + "seconds": 0.4082, + "peak_rss_anon": 667295744 + }, + "similar_pairs": { + "seconds": 5.1283, + "peak_rss_anon": 1813716992 + } + }, + "communities": {} + }, + "verdicts": [ + { + "workload": "reachable_set, full component", + "target": "> 50M edges/s", + "measured": "23M edges/s", + "met": false + }, + { + "workload": "components()", + "target": "< 10 s", + "measured": "2.7 s", + "met": true + }, + { + "workload": "pagerank, 20 iterations", + "target": "< 30 s (8 threads)", + "measured": "6.7 s", + "met": true + }, + { + "workload": "refresh() incl. reverse CSR", + "target": "< 8 min, < 4 GiB RAM", + "measured": "41.8 s, peak 1.95 GB", + "met": true + }, + { + "workload": "cold load into backend", + "target": "< 15 s", + "measured": "0.7 s", + "met": true + }, + { + "workload": "memory: graph + pagerank scratch within max_cache_memory", + "target": "fits 4 GiB", + "measured": "600.62 MB + 440.00 MB scratch", + "met": true + } + ] +} diff --git a/bench/results/soak/a-scale2-text-m10-shared.json b/bench/results/soak/a-scale2-text-m10-shared.json new file mode 100644 index 00000000..0ac121df --- /dev/null +++ b/bench/results/soak/a-scale2-text-m10-shared.json @@ -0,0 +1,617 @@ +{ + "provenance": { + "argv": [ + "bench/soak.py", + "a", + "--scale", + "2", + "--no-cte", + "--no-eigen-steps", + "--eigen-max-iter", + "200", + "--skip", + "communities", + "--disk-floor", + "3221225472", + "--pg-config", + "/var/tmp/m10-pgtree/usr/lib/postgresql/16/bin/pg_config", + "--parent", + "/var/tmp/m10-soak", + "--results", + "/var/tmp/m10-soak/results", + "--extension-revision", + "5f905b4", + "--extension-profile", + "release", + "--threads", + "4", + "--statement-timeout", + "20min", + "--ids", + "text", + "--label=-m10-shared", + "--shared-cache-memory", + "4GB", + "--skip", + "library", + "--skip", + "second-version" + ], + "harness": { + "revision": "5f905b45bc076768657d5c383d1c5bea22965178", + "dirty": false + }, + "extension": { + "revision": "5f905b4", + "profile": "release", + "library": "/var/tmp/m10-pgtree/usr/lib/postgresql/16/lib/pg_warren.so", + "library_sha256": "92c54dcb438b65c6c804ccddf96e1292412d2edd776a29f684a8afdcd481b667" + } + }, + "case": "a", + "nodes": 5000000, + "edges": 50000000, + "weighted": false, + "ids": "text", + "relations": 1, + "store_merged": true, + "threads": 4, + "node_order": "id", + "eigen_max_iter": 200, + "statement_timeout": "20min", + "skip": [ + "communities", + "library", + "second-version" + ], + "planned_stages": [ + "cold_load_probe", + "default: bfs depth 6", + "default: components", + "default: eigen", + "default: pagerank", + "default: prewarm", + "default: reachable_set", + "refresh" + ], + "hardware": { + "cpus": 4, + "model": "Intel(R) Xeon(R) Processor", + "memory_bytes": 16872804352, + "kernel": "6.18.44-fc-v153", + "load_average": [ + 3.76806640625, + 3.50146484375, + 3.158203125 + ] + }, + "started_at": "2026-10-23T19:51:33+00:00", + "stages": { + "table 0: create": { + "sql": "CREATE UNLOGGED TABLE soak_removed ()", + "seconds": [ + 0.0004 + ], + "median_seconds": 0.0004, + "rows": [], + "first_rows": [ + null + ], + "notices": [], + "warnings": [], + "rss_anon_before": 5574656, + "peak_rss_anon": 5574656, + "vm_hwm": 34136064, + "rss_anon_after": 5574656, + "rss_shmem_after": 14868480 + }, + "table 1: create": { + "sql": "CREATE UNLOGGED TABLE soak_edges (src int4 NOT NULL, dst int4 NOT NULL) WITH (autovacuum_enabled = false)", + "seconds": [ + 0.0003 + ], + "median_seconds": 0.0003, + "rows": [], + "first_rows": [ + null + ], + "notices": [], + "warnings": [], + "rss_anon_before": 5574656, + "peak_rss_anon": 5574656, + "vm_hwm": 34140160, + "rss_anon_after": 5574656, + "rss_shmem_after": 14872576 + }, + "table 2: insert": { + "sql": "INSERT INTO soak_edges SELECT 1 + floor(random() * 5000000)::int4 AS src, 1 + floor(random() * 5000000)::int4 AS dst FROM (SELECT setseed(0.25) OFFSET 0) seeded CROSS JOIN generate_series(1, 50000000)", + "seconds": [ + 17.0652 + ], + "median_seconds": 17.0652, + "rows": [], + "first_rows": [ + null + ], + "notices": [], + "warnings": [], + "rss_anon_before": 5574656, + "peak_rss_anon": 95420416, + "vm_hwm": 1253851136, + "rss_anon_after": 32473088, + "rss_shmem_after": 1095847936 + }, + "table 3: analyze": { + "sql": "ANALYZE soak_edges", + "seconds": [ + 0.2045 + ], + "median_seconds": 0.2045, + "rows": [], + "first_rows": [ + null + ], + "notices": [], + "warnings": [], + "rss_anon_before": 32473088, + "peak_rss_anon": 32477184, + "vm_hwm": 1191505920, + "rss_anon_after": 32477184, + "rss_shmem_after": 1095847936 + }, + "create_graph": { + "sql": "SELECT warren.create_graph('soak', $n$SELECT 'https://example.org/resource/' || id FROM generate_series(1::bigint, 5000000) id$n$, $q$SELECT 'https://example.org/resource/' || src, 'https://example.org/resource/' || dst FROM soak_edges$q$, weighted := false, has_relations := false, store_merged := true, node_order := 'id')", + "seconds": [ + 0.0229 + ], + "median_seconds": 0.0229, + "rows": [ + "" + ], + "first_rows": [ + "" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 3272704, + "peak_rss_anon": 9744384, + "vm_hwm": 74334208, + "rss_anon_after": 9744384, + "rss_shmem_after": 10989568 + }, + "refresh": { + "sql": "SELECT warren.refresh('soak')", + "seconds": [ + 58.6037 + ], + "median_seconds": 58.6037, + "rows": [ + "1" + ], + "first_rows": [ + "1" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 9744384, + "peak_rss_anon": 919117824, + "vm_hwm": 2077986816, + "rss_anon_after": 44617728, + "rss_shmem_after": 1111375872 + }, + "refresh from the generator": { + "sql": "SELECT warren.refresh('soak_generated')", + "seconds": [ + 62.6968 + ], + "median_seconds": 62.6968, + "rows": [ + "2" + ], + "first_rows": [ + "2" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 9805824, + "peak_rss_anon": 990916608, + "vm_hwm": 1148588032, + "rss_anon_after": 44941312, + "rss_shmem_after": 783515648 + }, + "cold_load_probe": { + "sql": "SELECT warren.prewarm('soak')", + "seconds": [ + 1.0579 + ], + "median_seconds": 1.0579, + "rows": [ + "1" + ], + "first_rows": [ + "1" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 3272704, + "peak_rss_anon": 12304384, + "vm_hwm": 1511960576, + "rss_anon_after": 12304384, + "rss_shmem_after": 1484685312 + }, + "default: prewarm": { + "sql": "WITH before AS MATERIALIZED (SELECT (SELECT pg_stat_get_xact_tuples_returned('warren.segments'::regclass) + pg_stat_get_xact_tuples_fetched('warren.segments'::regclass) + sum(pg_stat_get_xact_tuples_fetched(indexrelid))::bigint FROM pg_index WHERE indrelid = 'warren.segments'::regclass AND 1 IS NOT NULL) AS reads), run AS MATERIALIZED (SELECT warren.prewarm('soak') AS result FROM before WHERE before.reads IS NOT NULL), after AS MATERIALIZED (SELECT (SELECT pg_stat_get_xact_tuples_returned('warren.segments'::regclass) + pg_stat_get_xact_tuples_fetched('warren.segments'::regclass) + sum(pg_stat_get_xact_tuples_fetched(indexrelid))::bigint FROM pg_index WHERE indrelid = 'warren.segments'::regclass AND run.result IS NOT NULL) AS reads FROM run) SELECT after.reads - before.reads, run.result FROM before, run, after", + "seconds": [ + 0.8825 + ], + "median_seconds": 0.8825, + "rows": [ + "1800|1" + ], + "first_rows": [ + "1800|1" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 3272704, + "peak_rss_anon": 12656640, + "vm_hwm": 1516056576, + "rss_anon_after": 12636160, + "rss_shmem_after": 1488490496 + }, + "default: bfs depth 6": { + "sql": "SELECT bench_seconds($soak$SELECT count(*) FROM warren.bfs('soak', ARRAY['https://example.org/resource/1'::text], 6)$soak$)", + "seconds": [ + 0.9837, + 1.0079, + 0.9919 + ], + "median_seconds": 0.9919, + "rows": [ + "0.991438" + ], + "first_rows": [ + "0.982498", + "1.007502", + "0.991438" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 12693504, + "peak_rss_anon": 115752960, + "vm_hwm": 1622712320, + "rss_anon_after": 7548928, + "rss_shmem_after": 1488637952, + "server_seconds": [ + 0.982498, + 1.007502, + 0.991438 + ], + "server_median_seconds": 0.991438 + }, + "default: bfs depth 6 count": { + "sql": "SELECT count(*) FROM warren.bfs('soak', ARRAY['https://example.org/resource/1'::text], 6)", + "seconds": [ + 1.0195 + ], + "median_seconds": 1.0195, + "rows": [ + "1068779" + ], + "first_rows": [ + "1068779" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 7548928, + "peak_rss_anon": 115761152, + "vm_hwm": 1622720512, + "rss_anon_after": 7557120, + "rss_shmem_after": 1488637952 + }, + "default: neighbours": { + "sql": "SELECT bench_seconds($soak$SELECT count(*) FROM warren.neighbours('soak', 'https://example.org/resource/1'::text)$soak$)", + "seconds": [ + 0.0008, + 0.0002, + 0.0002, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001 + ], + "median_seconds": 0.0001, + "rows": [ + "4.7e-05" + ], + "first_rows": [ + "0.000473", + "9.1e-05", + "6.1e-05", + "5.3e-05", + "5.6e-05", + "5.2e-05", + "5e-05", + "4.7e-05", + "5.2e-05", + "4.7e-05", + "4.5e-05", + "4.5e-05", + "4.7e-05", + "4.6e-05", + "4.6e-05", + "4.5e-05", + "4.6e-05", + "4.5e-05", + "4.5e-05", + "4.7e-05" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 7557120, + "peak_rss_anon": 7577600, + "vm_hwm": 1515061248, + "rss_anon_after": 7577600, + "rss_shmem_after": 1488637952, + "server_seconds": [ + 0.000473, + 9.1e-05, + 6.1e-05, + 5.3e-05, + 5.6e-05, + 5.2e-05, + 5e-05, + 4.7e-05, + 5.2e-05, + 4.7e-05, + 4.5e-05, + 4.5e-05, + 4.7e-05, + 4.6e-05, + 4.6e-05, + 4.5e-05, + 4.6e-05, + 4.5e-05, + 4.5e-05, + 4.7e-05 + ], + "server_median_seconds": 4.7e-05 + }, + "default: reachable_set": { + "sql": "SELECT count(*) FROM warren.reachable_set('soak', ARRAY['https://example.org/resource/1'::text])", + "seconds": [ + 4.0433 + ], + "median_seconds": 4.0433, + "rows": [ + "4999752" + ], + "first_rows": [ + "4999752" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 7577600, + "peak_rss_anon": 260956160, + "vm_hwm": 1768247296, + "rss_anon_after": 7581696, + "rss_shmem_after": 1488637952 + }, + "default: reachable_set stats": { + "stats": [ + "{\"graph\": \"soak\", \"version\": 1, \"function\": \"reachable_set\", \"truncated\": false, \"nodes_visited\": 4999752, \"edges_traversed\": 11392106}" + ] + }, + "default: components": { + "sql": "SELECT count(*) FROM (SELECT component_id FROM warren.components('soak', NULL::text) GROUP BY 1) c", + "seconds": [ + 3.4093 + ], + "median_seconds": 3.4093, + "rows": [ + "1" + ], + "first_rows": [ + "1" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 7581696, + "peak_rss_anon": 562900992, + "vm_hwm": 2070089728, + "rss_anon_after": 7585792, + "rss_shmem_after": 1488654336 + }, + "default: pagerank": { + "sql": "WITH before AS MATERIALIZED (SELECT (SELECT pg_stat_get_xact_tuples_returned('warren.segments'::regclass) + pg_stat_get_xact_tuples_fetched('warren.segments'::regclass) + sum(pg_stat_get_xact_tuples_fetched(indexrelid))::bigint FROM pg_index WHERE indrelid = 'warren.segments'::regclass AND 1 IS NOT NULL) AS reads), run AS MATERIALIZED (SELECT warren.pagerank('soak', 'pr', tol := 0, max_iter := 20) AS result FROM before WHERE before.reads IS NOT NULL), after AS MATERIALIZED (SELECT (SELECT pg_stat_get_xact_tuples_returned('warren.segments'::regclass) + pg_stat_get_xact_tuples_fetched('warren.segments'::regclass) + sum(pg_stat_get_xact_tuples_fetched(indexrelid))::bigint FROM pg_index WHERE indrelid = 'warren.segments'::regclass AND run.result IS NOT NULL) AS reads FROM run) SELECT after.reads - before.reads, run.result FROM before, run, after", + "seconds": [ + 8.3611 + ], + "median_seconds": 8.3611, + "rows": [ + "0|20" + ], + "first_rows": [ + "0|20" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 7585792, + "peak_rss_anon": 558510080, + "vm_hwm": 2128871424, + "rss_anon_after": 53723136, + "rss_shmem_after": 1547718656 + }, + "default: pagerank from cold": { + "sql": "WITH before AS MATERIALIZED (SELECT (SELECT pg_stat_get_xact_tuples_returned('warren.segments'::regclass) + pg_stat_get_xact_tuples_fetched('warren.segments'::regclass) + sum(pg_stat_get_xact_tuples_fetched(indexrelid))::bigint FROM pg_index WHERE indrelid = 'warren.segments'::regclass AND 1 IS NOT NULL) AS reads), run AS MATERIALIZED (SELECT warren.pagerank('soak', 'pr', tol := 0, max_iter := 20) AS result FROM before WHERE before.reads IS NOT NULL), after AS MATERIALIZED (SELECT (SELECT pg_stat_get_xact_tuples_returned('warren.segments'::regclass) + pg_stat_get_xact_tuples_fetched('warren.segments'::regclass) + sum(pg_stat_get_xact_tuples_fetched(indexrelid))::bigint FROM pg_index WHERE indrelid = 'warren.segments'::regclass AND run.result IS NOT NULL) AS reads FROM run) SELECT after.reads - before.reads, run.result FROM before, run, after", + "seconds": [ + 8.775 + ], + "median_seconds": 8.775, + "rows": [ + "1800|20" + ], + "first_rows": [ + "1800|20" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 13725696, + "peak_rss_anon": 559484928, + "vm_hwm": 2169094144, + "rss_anon_after": 51871744, + "rss_shmem_after": 1589538816 + }, + "default: eigen": { + "sql": "SELECT count(*), max(residual) FROM warren.eigen('soak', 20, 'normalized_laplacian', 'smallest', tol := 1e-06, max_iter := 200)", + "seconds": [ + 73.0481 + ], + "median_seconds": 73.0481, + "rows": [ + "20|0.005370926710008592" + ], + "first_rows": [ + "20|0.005370926710008592" + ], + "notices": [], + "warnings": [ + "WARNING: eigen did not converge in 200 steps; the largest residual is 5.370926710008592e-3 (raise max_iter or tol)" + ], + "rss_anon_before": 51871744, + "peak_rss_anon": 1557012480, + "vm_hwm": 3167268864, + "rss_anon_after": 56782848, + "rss_shmem_after": 1589555200 + } + }, + "skipped": { + "second_version": "--skip second-version", + "library": "--skip library", + "communities": "--skip communities" + }, + "pg_warren": "0.6.0", + "postgres": "16.15 (Ubuntu 16.15-0ubuntu0.24.04.1)", + "edge_table_bytes": 1812856832, + "version_1": { + "seq": 1, + "bytes": 719513652, + "label": null, + "staged": false, + "current": true, + "vectors": 0, + "version": 1, + "built_at": "2026-10-23T19:51:50.602156+00:00", + "id_space": 5000000, + "built_lsn": "0/1CAE000", + "staleness": "00:02:02.109328", + "edge_count": 49999948, + "node_count": 5000000, + "label_vector": null, + "parent_graph": null, + "target_width": 4, + "dropped_edges": 0, + "compacted_from": null, + "parent_version": null, + "deduplicated_edges": 52 + }, + "cached_bytes": null, + "shared_bytes": 719514196, + "cache_memory_bytes": 4294967296, + "settings": { + "default": [ + "fsync=off", + "full_page_writes=off", + "jit=on", + "maintenance_work_mem=1GB", + "max_parallel_workers_per_gather=2", + "max_wal_size=1GB", + "shared_buffers=1GB", + "shared_preload_libraries=pg_warren", + "statement_timeout=20min", + "synchronous_commit=off", + "wal_level=minimal", + "warren.max_build_memory=4GB", + "warren.max_cache_memory=4GB", + "warren.max_compute_memory=4GB", + "warren.shared_cache_memory=4GB", + "warren.threads=4", + "work_mem=64MB" + ] + }, + "library_version": 1, + "finished_at": "2026-10-23T19:55:37+00:00", + "free_disk_after": 7025643520, + "load_average_after": [ + 4.37451171875, + 3.53759765625, + 3.2216796875 + ], + "measured": { + "cores": 4, + "bfs_seconds": 0.991438, + "components_seconds": 3.4093, + "pagerank_seconds": 8.3611, + "refresh_seconds": 58.6037, + "refresh_peak_bytes": 2077986816, + "cold_load_seconds": 0.8825, + "load_segment_reads": 1800, + "pagerank_segment_reads": 0, + "pagerank_loads": 0.0, + "pagerank_cold_segment_reads": 1800, + "pagerank_cold_seconds": 8.775, + "pagerank_cold_loads": 1.0, + "eigen_residual": 0.005370926710008592, + "eigen_converged": false, + "eigen_seconds_per_step": 0.3652405, + "reachable_row_entries_per_second": 2817526.7726856773, + "reachable_edges_per_second": 12366123.710830262, + "library": {}, + "communities": {} + }, + "verdicts": [ + { + "workload": "reachable_set, full component", + "target": "> 50M edges/s", + "measured": "12M edges/s", + "met": false + }, + { + "workload": "components()", + "target": "< 10 s", + "measured": "3.4 s", + "met": true + }, + { + "workload": "pagerank, 20 iterations", + "target": "< 30 s (8 threads)", + "measured": "8.4 s", + "met": true + }, + { + "workload": "refresh() incl. reverse CSR", + "target": "< 8 min, < 4 GiB RAM", + "measured": "58.6 s, peak 2.08 GB", + "met": true + }, + { + "workload": "cold load into backend", + "target": "< 15 s", + "measured": "0.9 s", + "met": true + } + ] +} diff --git a/bench/results/soak/a-scale2-text-m10.json b/bench/results/soak/a-scale2-text-m10.json new file mode 100644 index 00000000..fbd18b72 --- /dev/null +++ b/bench/results/soak/a-scale2-text-m10.json @@ -0,0 +1,952 @@ +{ + "provenance": { + "argv": [ + "bench/soak.py", + "a", + "--scale", + "2", + "--no-cte", + "--no-eigen-steps", + "--eigen-max-iter", + "200", + "--skip", + "communities", + "--disk-floor", + "3221225472", + "--pg-config", + "/var/tmp/m10-pgtree/usr/lib/postgresql/16/bin/pg_config", + "--parent", + "/var/tmp/m10-soak", + "--results", + "/var/tmp/m10-soak/results", + "--extension-revision", + "5f905b4", + "--extension-profile", + "release", + "--threads", + "4", + "--statement-timeout", + "20min", + "--ids", + "text", + "--label=-m10" + ], + "harness": { + "revision": "5f905b45bc076768657d5c383d1c5bea22965178", + "dirty": false + }, + "extension": { + "revision": "5f905b4", + "profile": "release", + "library": "/var/tmp/m10-pgtree/usr/lib/postgresql/16/lib/pg_warren.so", + "library_sha256": "92c54dcb438b65c6c804ccddf96e1292412d2edd776a29f684a8afdcd481b667" + } + }, + "case": "a", + "nodes": 5000000, + "edges": 50000000, + "weighted": false, + "ids": "text", + "relations": 1, + "store_merged": true, + "threads": 4, + "node_order": "id", + "eigen_max_iter": 200, + "statement_timeout": "20min", + "skip": [ + "communities" + ], + "planned_stages": [ + "cold_load_probe", + "default: bfs depth 6", + "default: components", + "default: eigen", + "default: pagerank", + "default: prewarm", + "default: reachable_set", + "diff", + "library: betweenness_centrality", + "library: closeness_centrality", + "library: clustering", + "library: core_number", + "library: eigenvector_centrality", + "library: harmonic_centrality", + "library: hits", + "library: katz_centrality", + "library: minimum_spanning_forest", + "library: most_similar", + "library: similar_pairs", + "library: similarity", + "library: strong_components_vector", + "refresh" + ], + "hardware": { + "cpus": 4, + "model": "Intel(R) Xeon(R) Processor", + "memory_bytes": 16872804352, + "kernel": "6.18.44-fc-v153", + "load_average": [ + 2.0439453125, + 2.099609375, + 2.744140625 + ] + }, + "started_at": "2026-10-23T19:38:59+00:00", + "stages": { + "table 0: create": { + "sql": "CREATE UNLOGGED TABLE soak_removed ()", + "seconds": [ + 0.0005 + ], + "median_seconds": 0.0005, + "rows": [], + "first_rows": [ + null + ], + "notices": [], + "warnings": [], + "rss_anon_before": 5558272, + "peak_rss_anon": 5558272, + "vm_hwm": 35262464, + "rss_anon_after": 5558272, + "rss_shmem_after": 15233024 + }, + "table 1: create": { + "sql": "CREATE UNLOGGED TABLE soak_edges (src int4 NOT NULL, dst int4 NOT NULL) WITH (autovacuum_enabled = false)", + "seconds": [ + 0.0005 + ], + "median_seconds": 0.0005, + "rows": [], + "first_rows": [ + null + ], + "notices": [], + "warnings": [], + "rss_anon_before": 5558272, + "peak_rss_anon": 5558272, + "vm_hwm": 35266560, + "rss_anon_after": 5558272, + "rss_shmem_after": 15237120 + }, + "table 2: insert": { + "sql": "INSERT INTO soak_edges SELECT 1 + floor(random() * 5000000)::int4 AS src, 1 + floor(random() * 5000000)::int4 AS dst FROM (SELECT setseed(0.25) OFFSET 0) seeded CROSS JOIN generate_series(1, 50000000)", + "seconds": [ + 16.9763 + ], + "median_seconds": 16.9763, + "rows": [], + "first_rows": [ + null + ], + "notices": [], + "warnings": [], + "rss_anon_before": 5558272, + "peak_rss_anon": 95424512, + "vm_hwm": 1254735872, + "rss_anon_after": 32477184, + "rss_shmem_after": 1095913472 + }, + "table 3: analyze": { + "sql": "ANALYZE soak_edges", + "seconds": [ + 0.2312 + ], + "median_seconds": 0.2312, + "rows": [], + "first_rows": [ + null + ], + "notices": [], + "warnings": [], + "rss_anon_before": 32477184, + "peak_rss_anon": 32481280, + "vm_hwm": 1192239104, + "rss_anon_after": 32481280, + "rss_shmem_after": 1095913472 + }, + "create_graph": { + "sql": "SELECT warren.create_graph('soak', $n$SELECT 'https://example.org/resource/' || id FROM generate_series(1::bigint, 5000000) id$n$, $q$SELECT 'https://example.org/resource/' || src, 'https://example.org/resource/' || dst FROM soak_edges$q$, weighted := false, has_relations := false, store_merged := true, node_order := 'id')", + "seconds": [ + 0.019 + ], + "median_seconds": 0.019, + "rows": [ + "" + ], + "first_rows": [ + "" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 3198976, + "peak_rss_anon": 9744384, + "vm_hwm": 74354688, + "rss_anon_after": 9744384, + "rss_shmem_after": 10772480 + }, + "refresh": { + "sql": "SELECT warren.refresh('soak')", + "seconds": [ + 56.3619 + ], + "median_seconds": 56.3619, + "rows": [ + "1" + ], + "first_rows": [ + "1" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 9744384, + "peak_rss_anon": 919117824, + "vm_hwm": 2078314496, + "rss_anon_after": 44617728, + "rss_shmem_after": 1111515136 + }, + "refresh from the generator": { + "sql": "SELECT warren.refresh('soak_generated')", + "seconds": [ + 61.1308 + ], + "median_seconds": 61.1308, + "rows": [ + "2" + ], + "first_rows": [ + "2" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 9850880, + "peak_rss_anon": 990928896, + "vm_hwm": 1148993536, + "rss_anon_after": 44957696, + "rss_shmem_after": 783757312 + }, + "cold_load_probe": { + "sql": "SELECT warren.prewarm('soak')", + "seconds": [ + 1.0275 + ], + "median_seconds": 1.0275, + "rows": [ + "1" + ], + "first_rows": [ + "1" + ], + "notices": [ + "NOTICE: version 1 (719513652 bytes) cannot be kept within warren.max_cache_memory (65536 bytes, 0 of them held by transient vectors), so it is held only until the statement ends, the bytes past the cap charged to warren.max_compute_memory, and reloaded by each statement that reads it; raise warren.max_cache_memory" + ], + "warnings": [], + "rss_anon_before": 3211264, + "peak_rss_anon": 731783168, + "vm_hwm": 1514692608, + "rss_anon_after": 12267520, + "rss_shmem_after": 765435904 + }, + "default: prewarm": { + "sql": "WITH before AS MATERIALIZED (SELECT (SELECT pg_stat_get_xact_tuples_returned('warren.segments'::regclass) + pg_stat_get_xact_tuples_fetched('warren.segments'::regclass) + sum(pg_stat_get_xact_tuples_fetched(indexrelid))::bigint FROM pg_index WHERE indrelid = 'warren.segments'::regclass AND 1 IS NOT NULL) AS reads), run AS MATERIALIZED (SELECT warren.prewarm('soak') AS result FROM before WHERE before.reads IS NOT NULL), after AS MATERIALIZED (SELECT (SELECT pg_stat_get_xact_tuples_returned('warren.segments'::regclass) + pg_stat_get_xact_tuples_fetched('warren.segments'::regclass) + sum(pg_stat_get_xact_tuples_fetched(indexrelid))::bigint FROM pg_index WHERE indrelid = 'warren.segments'::regclass AND run.result IS NOT NULL) AS reads FROM run) SELECT after.reads - before.reads, run.result FROM before, run, after", + "seconds": [ + 0.8561 + ], + "median_seconds": 0.8561, + "rows": [ + "1800|1" + ], + "first_rows": [ + "1800|1" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 3207168, + "peak_rss_anon": 732102656, + "vm_hwm": 1518944256, + "rss_anon_after": 732078080, + "rss_shmem_after": 769253376 + }, + "default: bfs depth 6": { + "sql": "SELECT bench_seconds($soak$SELECT count(*) FROM warren.bfs('soak', ARRAY['https://example.org/resource/1'::text], 6)$soak$)", + "seconds": [ + 0.9486, + 0.9102, + 0.9602 + ], + "median_seconds": 0.9486, + "rows": [ + "0.959448" + ], + "first_rows": [ + "0.947536", + "0.909655", + "0.959448" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 732172288, + "peak_rss_anon": 834207744, + "vm_hwm": 1622233088, + "rss_anon_after": 729153536, + "rss_shmem_after": 769380352, + "server_seconds": [ + 0.947536, + 0.909655, + 0.959448 + ], + "server_median_seconds": 0.947536 + }, + "default: bfs depth 6 count": { + "sql": "SELECT count(*) FROM warren.bfs('soak', ARRAY['https://example.org/resource/1'::text], 6)", + "seconds": [ + 0.9007 + ], + "median_seconds": 0.9007, + "rows": [ + "1068779" + ], + "first_rows": [ + "1068779" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 729153536, + "peak_rss_anon": 834215936, + "vm_hwm": 1622245376, + "rss_anon_after": 729161728, + "rss_shmem_after": 769380352 + }, + "default: neighbours": { + "sql": "SELECT bench_seconds($soak$SELECT count(*) FROM warren.neighbours('soak', 'https://example.org/resource/1'::text)$soak$)", + "seconds": [ + 0.0008, + 0.0003, + 0.0002, + 0.0002, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0002, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001, + 0.0001 + ], + "median_seconds": 0.0001, + "rows": [ + "4.2e-05" + ], + "first_rows": [ + "0.000447", + "9.2e-05", + "6.4e-05", + "5.4e-05", + "5.6e-05", + "4.8e-05", + "4.5e-05", + "4.5e-05", + "4.5e-05", + "4.3e-05", + "4.3e-05", + "5.7e-05", + "4.6e-05", + "5e-05", + "4.5e-05", + "4.3e-05", + "4.2e-05", + "4.2e-05", + "4.2e-05", + "4.2e-05" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 729161728, + "peak_rss_anon": 729182208, + "vm_hwm": 1517608960, + "rss_anon_after": 729182208, + "rss_shmem_after": 769380352, + "server_seconds": [ + 0.000447, + 9.2e-05, + 6.4e-05, + 5.4e-05, + 5.6e-05, + 4.8e-05, + 4.5e-05, + 4.5e-05, + 4.5e-05, + 4.3e-05, + 4.3e-05, + 5.7e-05, + 4.6e-05, + 5e-05, + 4.5e-05, + 4.3e-05, + 4.2e-05, + 4.2e-05, + 4.2e-05, + 4.2e-05 + ], + "server_median_seconds": 4.5e-05 + }, + "default: reachable_set": { + "sql": "SELECT count(*) FROM warren.reachable_set('soak', ARRAY['https://example.org/resource/1'::text])", + "seconds": [ + 3.7237 + ], + "median_seconds": 3.7237, + "rows": [ + "4999752" + ], + "first_rows": [ + "4999752" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 729182208, + "peak_rss_anon": 980848640, + "vm_hwm": 1768964096, + "rss_anon_after": 726986752, + "rss_shmem_after": 769380352 + }, + "default: reachable_set stats": { + "stats": [ + "{\"graph\": \"soak\", \"version\": 1, \"function\": \"reachable_set\", \"truncated\": false, \"nodes_visited\": 4999752, \"edges_traversed\": 11392106}" + ] + }, + "default: components": { + "sql": "SELECT count(*) FROM (SELECT component_id FROM warren.components('soak', NULL::text) GROUP BY 1) c", + "seconds": [ + 2.7653 + ], + "median_seconds": 2.7653, + "rows": [ + "1" + ], + "first_rows": [ + "1" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 726990848, + "peak_rss_anon": 1282367488, + "vm_hwm": 2070441984, + "rss_anon_after": 724078592, + "rss_shmem_after": 769396736 + }, + "default: pagerank": { + "sql": "WITH before AS MATERIALIZED (SELECT (SELECT pg_stat_get_xact_tuples_returned('warren.segments'::regclass) + pg_stat_get_xact_tuples_fetched('warren.segments'::regclass) + sum(pg_stat_get_xact_tuples_fetched(indexrelid))::bigint FROM pg_index WHERE indrelid = 'warren.segments'::regclass AND 1 IS NOT NULL) AS reads), run AS MATERIALIZED (SELECT warren.pagerank('soak', 'pr', tol := 0, max_iter := 20) AS result FROM before WHERE before.reads IS NOT NULL), after AS MATERIALIZED (SELECT (SELECT pg_stat_get_xact_tuples_returned('warren.segments'::regclass) + pg_stat_get_xact_tuples_fetched('warren.segments'::regclass) + sum(pg_stat_get_xact_tuples_fetched(indexrelid))::bigint FROM pg_index WHERE indrelid = 'warren.segments'::regclass AND run.result IS NOT NULL) AS reads FROM run) SELECT after.reads - before.reads, run.result FROM before, run, after", + "seconds": [ + 9.116 + ], + "median_seconds": 9.116, + "rows": [ + "0|20" + ], + "first_rows": [ + "0|20" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 724078592, + "peak_rss_anon": 1278713856, + "vm_hwm": 2127269888, + "rss_anon_after": 771112960, + "rss_shmem_after": 828530688 + }, + "default: pagerank from cold": { + "sql": "WITH before AS MATERIALIZED (SELECT (SELECT pg_stat_get_xact_tuples_returned('warren.segments'::regclass) + pg_stat_get_xact_tuples_fetched('warren.segments'::regclass) + sum(pg_stat_get_xact_tuples_fetched(indexrelid))::bigint FROM pg_index WHERE indrelid = 'warren.segments'::regclass AND 1 IS NOT NULL) AS reads), run AS MATERIALIZED (SELECT warren.pagerank('soak', 'pr', tol := 0, max_iter := 20) AS result FROM before WHERE before.reads IS NOT NULL), after AS MATERIALIZED (SELECT (SELECT pg_stat_get_xact_tuples_returned('warren.segments'::regclass) + pg_stat_get_xact_tuples_fetched('warren.segments'::regclass) + sum(pg_stat_get_xact_tuples_fetched(indexrelid))::bigint FROM pg_index WHERE indrelid = 'warren.segments'::regclass AND run.result IS NOT NULL) AS reads FROM run) SELECT after.reads - before.reads, run.result FROM before, run, after", + "seconds": [ + 9.1688 + ], + "median_seconds": 9.1688, + "rows": [ + "1800|20" + ], + "first_rows": [ + "1800|20" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 11603968, + "peak_rss_anon": 1278418944, + "vm_hwm": 2169524224, + "rss_anon_after": 771350528, + "rss_shmem_after": 870387712 + }, + "default: eigen": { + "sql": "SELECT count(*), max(residual) FROM warren.eigen('soak', 20, 'normalized_laplacian', 'smallest', tol := 1e-06, max_iter := 200)", + "seconds": [ + 75.8788 + ], + "median_seconds": 75.8788, + "rows": [ + "20|0.005370926710008592" + ], + "first_rows": [ + "20|0.005370926710008592" + ], + "notices": [], + "warnings": [ + "WARNING: eigen did not converge in 200 steps; the largest residual is 5.370926710008592e-3 (raise max_iter or tol)" + ], + "rss_anon_before": 771350528, + "peak_rss_anon": 2276478976, + "vm_hwm": 3167629312, + "rss_anon_after": 776249344, + "rss_shmem_after": 870404096 + }, + "library: eigenvector_centrality": { + "sql": "SELECT warren.eigenvector_centrality('soak', 'library', tol := 0, max_iter := 20)", + "seconds": [ + 8.326 + ], + "median_seconds": 8.326, + "rows": [ + "20" + ], + "first_rows": [ + "20" + ], + "notices": [], + "warnings": [ + "WARNING: eigenvector_centrality did not converge in 20 iterations" + ], + "rss_anon_before": 3207168, + "peak_rss_anon": 1037402112, + "vm_hwm": 1886388224, + "rss_anon_after": 818798592, + "rss_shmem_after": 829349888 + }, + "library: katz_centrality": { + "sql": "SELECT warren.katz_centrality('soak', 'library', alpha := 0.01, tol := 0, max_iter := 20)", + "seconds": [ + 6.0818 + ], + "median_seconds": 6.0818, + "rows": [ + "20" + ], + "first_rows": [ + "20" + ], + "notices": [], + "warnings": [ + "WARNING: katz_centrality did not converge in 20 iterations" + ], + "rss_anon_before": 778809344, + "peak_rss_anon": 1120010240, + "vm_hwm": 2010664960, + "rss_anon_after": 812408832, + "rss_shmem_after": 871260160 + }, + "library: hits": { + "sql": "SELECT warren.hits('soak', 'library_hubs', 'library_authorities', tol := 0, max_iter := 20)", + "seconds": [ + 9.7759 + ], + "median_seconds": 9.7759, + "rows": [ + "20" + ], + "first_rows": [ + "20" + ], + "notices": [], + "warnings": [ + "WARNING: hits did not converge in 20 iterations" + ], + "rss_anon_before": 772407296, + "peak_rss_anon": 1080033280, + "vm_hwm": 2054127616, + "rss_anon_after": 852434944, + "rss_shmem_after": 954830848 + }, + "library: clustering": { + "sql": "SELECT warren.clustering('soak', 'library')", + "seconds": [ + 26.9576 + ], + "median_seconds": 26.9576, + "rows": [ + "" + ], + "first_rows": [ + "" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 772431872, + "peak_rss_anon": 1612439552, + "vm_hwm": 2586525696, + "rss_anon_after": 812470272, + "rss_shmem_after": 996761600 + }, + "library: core_number": { + "sql": "SELECT warren.core_number('soak', 'library', persist := false)", + "seconds": [ + 9.6788 + ], + "median_seconds": 9.6788, + "rows": [ + "14" + ], + "first_rows": [ + "14" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 772468736, + "peak_rss_anon": 1692475392, + "vm_hwm": 2708639744, + "rss_anon_after": 772468736, + "rss_shmem_after": 996761600 + }, + "library: strong_components_vector": { + "sql": "SELECT warren.strong_components_vector('soak', 'library', persist := false)", + "seconds": [ + 8.3175 + ], + "median_seconds": 8.3175, + "rows": [ + "476" + ], + "first_rows": [ + "476" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 772468736, + "peak_rss_anon": 918396928, + "vm_hwm": 1934692352, + "rss_anon_after": 771559424, + "rss_shmem_after": 996761600 + }, + "library: minimum_spanning_forest": { + "sql": "SELECT count(*) FROM warren.minimum_spanning_forest('soak', NULL::text)", + "seconds": [ + 22.4895 + ], + "median_seconds": 22.4895, + "rows": [ + "4999999" + ], + "first_rows": [ + "4999999" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 771559424, + "peak_rss_anon": 1773666304, + "vm_hwm": 2799271936, + "rss_anon_after": 777728000, + "rss_shmem_after": 996761600 + }, + "library: betweenness_centrality": { + "sql": "SELECT warren.betweenness_centrality('soak', 'library', k := 16, persist := false)", + "seconds": [ + 40.4677 + ], + "median_seconds": 40.4677, + "rows": [ + "16" + ], + "first_rows": [ + "16" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 777728000, + "peak_rss_anon": 1417756672, + "vm_hwm": 2434093056, + "rss_anon_after": 777732096, + "rss_shmem_after": 996761600 + }, + "library: closeness_centrality": { + "sql": "SELECT warren.closeness_centrality('soak', 'library', k := 16, persist := false)", + "seconds": [ + 3.6388 + ], + "median_seconds": 3.6388, + "rows": [ + "16" + ], + "first_rows": [ + "16" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 777732096, + "peak_rss_anon": 1057751040, + "vm_hwm": 2074148864, + "rss_anon_after": 777826304, + "rss_shmem_after": 996761600 + }, + "library: harmonic_centrality": { + "sql": "SELECT warren.harmonic_centrality('soak', 'library', k := 16, persist := false)", + "seconds": [ + 3.7307 + ], + "median_seconds": 3.7307, + "rows": [ + "16" + ], + "first_rows": [ + "16" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 777826304, + "peak_rss_anon": 1017839616, + "vm_hwm": 2034266112, + "rss_anon_after": 817831936, + "rss_shmem_after": 996769792 + }, + "library: most_similar": { + "sql": "SELECT count(*) FROM warren.most_similar('soak', 'https://example.org/resource/1'::text)", + "seconds": [ + 0.0011 + ], + "median_seconds": 0.0011, + "rows": [ + "10" + ], + "first_rows": [ + "10" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 777830400, + "peak_rss_anon": 777834496, + "vm_hwm": 1795121152, + "rss_anon_after": 777834496, + "rss_shmem_after": 996769792 + }, + "library: similarity": { + "sql": "SELECT count(*) FROM warren.similarity('soak', $p$SELECT 'https://example.org/resource/' || (1 + (i * 7919) % 5000000), 'https://example.org/resource/' || (1 + (i * 104729) % 5000000) FROM generate_series(1::bigint, 100000) i$p$, NULL::text)", + "seconds": [ + 1.3072 + ], + "median_seconds": 1.3072, + "rows": [ + "100000" + ], + "first_rows": [ + "100000" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 777834496, + "peak_rss_anon": 817860608, + "vm_hwm": 1834622976, + "rss_anon_after": 777859072, + "rss_shmem_after": 996827136 + }, + "library: similar_pairs": { + "sql": "SELECT count(*) FROM warren.similar_pairs('soak', NULL::text, nodes := 'library_mask')", + "seconds": [ + 5.5888 + ], + "median_seconds": 5.5888, + "rows": [ + "500000" + ], + "first_rows": [ + "500000" + ], + "notices": [], + "warnings": [], + "rss_anon_before": 817864704, + "peak_rss_anon": 1924263936, + "vm_hwm": 2941210624, + "rss_anon_after": 857718784, + "rss_shmem_after": 996827136 + } + }, + "skipped": { + "second_version": "disk: 4.37 GB free, a second version of 719.51 MB would leave less than the floor", + "communities": "--skip communities" + }, + "pg_warren": "0.6.0", + "postgres": "16.15 (Ubuntu 16.15-0ubuntu0.24.04.1)", + "edge_table_bytes": 1812856832, + "version_1": { + "seq": 1, + "bytes": 719513652, + "label": null, + "staged": false, + "current": true, + "vectors": 0, + "version": 1, + "built_at": "2026-10-23T19:39:16.325549+00:00", + "id_space": 5000000, + "built_lsn": "0/1CAE000", + "staleness": "00:01:58.257899", + "edge_count": 49999948, + "node_count": 5000000, + "label_vector": null, + "parent_graph": null, + "target_width": 4, + "dropped_edges": 0, + "compacted_from": null, + "parent_version": null, + "deduplicated_edges": 52 + }, + "cached_bytes": 719513652, + "shared_bytes": 0, + "cache_memory_bytes": 4294967296, + "settings": { + "default": [ + "fsync=off", + "full_page_writes=off", + "jit=on", + "maintenance_work_mem=1GB", + "max_parallel_workers_per_gather=2", + "max_wal_size=1GB", + "shared_buffers=1GB", + "shared_preload_libraries=", + "statement_timeout=20min", + "synchronous_commit=off", + "wal_level=minimal", + "warren.max_build_memory=4GB", + "warren.max_cache_memory=4GB", + "warren.max_compute_memory=4GB", + "warren.shared_cache_memory=4GB", + "warren.threads=4", + "work_mem=64MB" + ] + }, + "library_version": 1, + "finished_at": "2026-10-23T19:45:28+00:00", + "free_disk_after": 7026036736, + "load_average_after": [ + 3.57568359375, + 3.0859375, + 2.95751953125 + ], + "measured": { + "cores": 4, + "bfs_seconds": 0.947536, + "components_seconds": 2.7653, + "pagerank_seconds": 9.116, + "refresh_seconds": 56.3619, + "refresh_peak_bytes": 2078314496, + "cold_load_seconds": 0.8561, + "load_segment_reads": 1800, + "pagerank_segment_reads": 0, + "pagerank_loads": 0.0, + "pagerank_cold_segment_reads": 1800, + "pagerank_cold_seconds": 9.1688, + "pagerank_cold_loads": 1.0, + "eigen_residual": 0.005370926710008592, + "eigen_converged": false, + "eigen_seconds_per_step": 0.379394, + "reachable_row_entries_per_second": 3059351.1829631818, + "reachable_edges_per_second": 13427490.936434193, + "cached_bytes": 719513652, + "id_space": 5000000, + "max_cache_memory": 4294967296, + "library": { + "eigenvector_centrality": { + "seconds": 8.326, + "peak_rss_anon": 1037402112 + }, + "katz_centrality": { + "seconds": 6.0818, + "peak_rss_anon": 1120010240 + }, + "hits": { + "seconds": 9.7759, + "peak_rss_anon": 1080033280 + }, + "clustering": { + "seconds": 26.9576, + "peak_rss_anon": 1612439552 + }, + "core_number": { + "seconds": 9.6788, + "peak_rss_anon": 1692475392 + }, + "strong_components_vector": { + "seconds": 8.3175, + "peak_rss_anon": 918396928 + }, + "minimum_spanning_forest": { + "seconds": 22.4895, + "peak_rss_anon": 1773666304 + }, + "betweenness_centrality": { + "seconds": 40.4677, + "peak_rss_anon": 1417756672 + }, + "closeness_centrality": { + "seconds": 3.6388, + "peak_rss_anon": 1057751040 + }, + "harmonic_centrality": { + "seconds": 3.7307, + "peak_rss_anon": 1017839616 + }, + "most_similar": { + "seconds": 0.0011, + "peak_rss_anon": 777834496 + }, + "similarity": { + "seconds": 1.3072, + "peak_rss_anon": 817860608 + }, + "similar_pairs": { + "seconds": 5.5888, + "peak_rss_anon": 1924263936 + } + }, + "communities": {} + }, + "verdicts": [ + { + "workload": "reachable_set, full component", + "target": "> 50M edges/s", + "measured": "13M edges/s", + "met": false + }, + { + "workload": "components()", + "target": "< 10 s", + "measured": "2.8 s", + "met": true + }, + { + "workload": "pagerank, 20 iterations", + "target": "< 30 s (8 threads)", + "measured": "9.1 s", + "met": true + }, + { + "workload": "refresh() incl. reverse CSR", + "target": "< 8 min, < 4 GiB RAM", + "measured": "56.4 s, peak 2.08 GB", + "met": true + }, + { + "workload": "cold load into backend", + "target": "< 15 s", + "measured": "0.9 s", + "met": true + }, + { + "workload": "memory: graph + pagerank scratch within max_cache_memory", + "target": "fits 4 GiB", + "measured": "719.51 MB + 440.00 MB scratch", + "met": true + } + ] +} diff --git a/bench/soak.py b/bench/soak.py index 760c08dd..7f2174cc 100755 --- a/bench/soak.py +++ b/bench/soak.py @@ -39,7 +39,10 @@ their rows until VACUUM, after the run, so a run is cancelled if it would take the disk below `--disk-floor`; the edge table and the first version are dropped first. `--node-order locality` builds the graph with -dense ids in locality order (DESIGN.md §14 item 22).""" +dense ids in locality order (DESIGN.md §14 item 22). `--ids text` names +node n by the IRI `https://example.org/resource/` (DESIGN.md §5.7), +in the node query, the edge query and every call; its results are +`-text.json`.""" import argparse import datetime @@ -120,6 +123,7 @@ class Case: table: bool # read the edges from a table (else from the generator query) nodes: int = 10_000_000 edges: int = 100_000_000 + ids: str = "bigint" # the id type: bigint node numbers, or text IRIs (`--ids text`) def scaled(self, factor): return replace(self, nodes=self.nodes // factor, edges=self.edges // factor) @@ -131,6 +135,7 @@ def scaled(self, factor): "b-unmerged": Case("b-unmerged", weighted=True, relations=5, store_merged=False, table=False), } SEED = 0.25 +IRI = "https://example.org/resource/" # a text case's ids: this, then the node's number (DESIGN.md §5.7) # The deduplicated edges of every full-scale case with SEED (the parallel # edges the generator draws), recorded by the committed runs on PG 16: a # plan that evaluates the seeding subquery differently draws other edges. @@ -146,10 +151,21 @@ def generator_sql(case, seed): f"FROM (SELECT setseed({seed}) OFFSET 0) seeded CROSS JOIN generate_series(1, {case.edges})") +def external(ids, number): + """The external id of the node numbered by SQL expression `number`.""" + return f"'{IRI}' || {number}" if ids == "text" else f"{number}::bigint" + + +def node(ids, number): + """The external id of node `number`, as a typed SQL literal.""" + return f"'{IRI}{number}'::text" if ids == "text" else f"{number}::bigint" + + def edge_query(case, seed, table=None): - """The graph's edge query: endpoints as `bigint` (an `int8` id map), the - relation and weight of a multi-relation case derived from them.""" - columns = "src::bigint, dst::bigint" + """The graph's edge query: endpoints as `bigint` (an `int8` id map) or + as IRIs (a text id map), the relation and weight of a multi-relation + case derived from their numbers.""" + columns = f"{external(case.ids, 'src')}, {external(case.ids, 'dst')}" if case.relations > 1 or case.weighted: columns += f", 'r' || ((src # dst) % {case.relations}), (1 + (src + 3 * dst) % 9)::float8" if case.table if table is None else table: @@ -160,7 +176,8 @@ def edge_query(case, seed, table=None): def create_graph_sql(case, seed, name=GRAPH, table=None, node_order="id"): multi = case.relations > 1 - return (f"SELECT warren.create_graph('{name}', 'SELECT id FROM generate_series(1::bigint, {case.nodes}) id', " + nodes = f"SELECT {external(case.ids, 'id').removesuffix('::bigint')} FROM generate_series(1::bigint, {case.nodes}) id" + return (f"SELECT warren.create_graph('{name}', $n${nodes}$n$, " f"$q${edge_query(case, seed, table)}$q$, weighted := {str(case.weighted).lower()}, " f"has_relations := {str(multi).lower()}, store_merged := {str(case.store_merged).lower()}, " f"node_order := '{node_order}')") @@ -183,18 +200,20 @@ def _drop(graph, *names): return tuple(f"SELECT warren.drop_vector('{graph}', '{name}', NULL, true)" for name in names) -def library_calls(graph, nodes): +def library_calls(graph, nodes, ids="bigint"): """The library added in 0.4.0 (DESIGN.md §6.7, groups 1 and 2) on - `graph`, whose external ids are 1..`nodes`: the iterative ones for 20 + `graph`, whose nodes are numbered 1..`nodes` (`ids` names their id + type, as `Case.ids`): the iterative ones for 20 iterations, as `pagerank`, the sampled centralities from 16 sources. Left out: `label_propagation_majority` (tier 2, its votes counted in the executor: minutes an iteration and gigabytes of sort at this scale), and `topological_order` and `dag_longest_path` (the graph has cycles).""" g, out, drop = f"'{graph}'", "'library'", _drop(graph, "library") - pairs = (f"SELECT (1 + (i * 7919) % {nodes})::bigint, (1 + (i * 104729) % {nodes})::bigint " + pairs = (f"SELECT {external(ids, f'(1 + (i * 7919) % {nodes})')}, {external(ids, f'(1 + (i * 104729) % {nodes})')} " f"FROM generate_series(1::bigint, {PAIRS}) i") - mask = f"SELECT id, 1 FROM generate_series(1::bigint, {nodes}, 100) id" + mask = f"SELECT {external(ids, 'id').removesuffix('::bigint')}, 1 FROM generate_series(1::bigint, {nodes}, 100) id" + witness = f"NULL::{ids}" return ( Call("eigenvector_centrality", f"SELECT warren.eigenvector_centrality({g}, {out}, tol := 0, max_iter := 20)", cleanup=drop), @@ -206,16 +225,16 @@ def library_calls(graph, nodes): Call("core_number", f"SELECT warren.core_number({g}, {out}, persist := false)", cleanup=drop), Call("strong_components_vector", f"SELECT warren.strong_components_vector({g}, {out}, persist := false)", cleanup=drop), - Call("minimum_spanning_forest", f"SELECT count(*) FROM warren.minimum_spanning_forest({g}, NULL::bigint)"), + Call("minimum_spanning_forest", f"SELECT count(*) FROM warren.minimum_spanning_forest({g}, {witness})"), Call("betweenness_centrality", f"SELECT warren.betweenness_centrality({g}, {out}, k := {SOURCES}, persist := false)", cleanup=drop), Call("closeness_centrality", f"SELECT warren.closeness_centrality({g}, {out}, k := {SOURCES}, persist := false)", cleanup=drop), Call("harmonic_centrality", f"SELECT warren.harmonic_centrality({g}, {out}, k := {SOURCES}, persist := false)", cleanup=drop), - Call("most_similar", f"SELECT count(*) FROM warren.most_similar({g}, 1::bigint)"), - Call("similarity", f"SELECT count(*) FROM warren.similarity({g}, $p${pairs}$p$, NULL::bigint)"), - Call("similar_pairs", f"SELECT count(*) FROM warren.similar_pairs({g}, NULL::bigint, nodes := 'library_mask')", + Call("most_similar", f"SELECT count(*) FROM warren.most_similar({g}, {node(ids, 1)})"), + Call("similarity", f"SELECT count(*) FROM warren.similarity({g}, $p${pairs}$p$, {witness})"), + Call("similar_pairs", f"SELECT count(*) FROM warren.similar_pairs({g}, {witness}, nodes := 'library_mask')", setup=(f"SELECT warren.vector_from_query({g}, 'library_mask', $p${mask}$p$, persist := false)",), cleanup=_drop(graph, "library_mask")), ) @@ -236,10 +255,10 @@ def community_calls(graph): for name in ("louvain", "leiden")) -def community_stats_sql(graph, name): +def community_stats_sql(graph, name, ids="bigint"): """`levels|communities` of a community run's result.""" return (f"SELECT (SELECT count(*) {_levels(graph, name)}) || '|' || " - f"(SELECT count(DISTINCT value) FROM warren.vector_values('{graph}', '{name}', NULL::bigint))") + f"(SELECT count(DISTINCT value) FROM warren.vector_values('{graph}', '{name}', NULL::{ids}))") # The bytes of the catalog's segment and vector chunk tables, and the WAL @@ -272,12 +291,19 @@ def remove_sql(case): def segment_estimate(case): """The stored bytes of a version (DESIGN.md §5.3): per direction, each CSR's offsets and each copy of the edges' targets (u32) and weights; the - merged CSR is a second copy when stored with several relations.""" + merged CSR is a second copy when stored with several relations. The id + map is 16 bytes a node for `bigint` ids (pairs, and the dense-id + direction), and for IRIs their bytes, their offsets and their order.""" csrs = case.relations + (1 if case.relations > 1 and case.store_merged else 0) copies = 2 if case.relations > 1 and case.store_merged else 1 per_edge = 4 + (8 if case.weighted else 0) direction = 8 * (case.nodes + 1) * csrs + per_edge * case.edges * copies - return 2 * direction + 16 * case.nodes + case.nodes // 8 + if case.ids == "text": + digits = sum(d * (min(case.nodes, 10 ** d - 1) - 10 ** (d - 1) + 1) for d in range(1, len(str(case.nodes)) + 1)) + id_map = len(IRI) * case.nodes + digits + 8 * (case.nodes + 1) + 4 * case.nodes + else: + id_map = 16 * case.nodes + return 2 * direction + id_map + case.nodes // 8 def memory_status(text): @@ -440,8 +466,9 @@ class GeneratorDrift(RuntimeError): def check_edges(case, stats): """Fails unless a full-scale case's first version has the deduplicated - edges every committed run had (scaled cases have no recorded value).""" - if case != CASES[case.name]: + edges every committed run had (scaled cases have no recorded value), + which its edges have whatever their ids.""" + if replace(case, ids="bigint") != CASES[case.name]: return found, edges = stats["deduplicated_edges"], stats["edge_count"] if (found, edges) != (DEDUPLICATED_EDGES, case.edges - DEDUPLICATED_EDGES): @@ -638,7 +665,7 @@ def run(self, case): admin = self.cluster.database() admin.execute("DROP DATABASE IF EXISTS soak") admin.execute("CREATE DATABASE soak") - result = {"provenance": self.provenance, "case": case.name, "nodes": case.nodes, "edges": case.edges, "weighted": case.weighted, + result = {"provenance": self.provenance, "case": case.name, "nodes": case.nodes, "edges": case.edges, "weighted": case.weighted, "ids": case.ids, "relations": case.relations, "store_merged": case.store_merged, "threads": self.options.threads, "node_order": self.options.node_order, "eigen_max_iter": self.options.eigen_max_iter, "statement_timeout": self.options.statement_timeout, "skip": self.options.skip, "planned_stages": planned_stages(case, self.options), @@ -690,9 +717,9 @@ def _run(self, case, result): # The first pass's prewarm is then the cold load into the shared cache. s.run(f"SELECT warren.evict('{GRAPH}')") result["cache_memory_bytes"] = cap_bytes(self.options.cache_memory_kb) - self._queries(result, "default", cap=self.first_cap()) + self._queries(result, case, "default", cap=self.first_cap()) if needs_raised_pass(result, v1["id_space"]): - self._queries(result, "raised", cap=self.options.raised_cache_memory) + self._queries(result, case, "raised", cap=self.options.raised_cache_memory) if self.options.eigen and self.options.eigen_steps and case.name == "a": self._eigen_steps(case, result) if "second-version" in self.options.skip: @@ -746,7 +773,7 @@ def first_cap(self): kb = self.options.cache_memory_kb return None if kb is None else f"{kb}kB" - def _queries(self, result, label, cap): + def _queries(self, result, case, label, cap): """The read workloads in one backend, from a cold cache. A version that cannot fit the cap reloads on every statement, so latencies repeat less.""" reloads = label == "default" and (result["cached_bytes"] or 0) > cap_bytes(self.options.cache_memory_kb) @@ -761,13 +788,14 @@ def stage(name, sql, repeat=1, server=False): # After the SETs, and after a call loaded the library: before # that pg_settings does not list warren.*. result.setdefault("settings", {})[label] = s.run(SETTINGS_SQL)["rows"] - stage("bfs depth 6", f"SELECT count(*) FROM warren.bfs('{GRAPH}', ARRAY[1::bigint], 6)", 3, True) - stage("bfs depth 6 count", f"SELECT count(*) FROM warren.bfs('{GRAPH}', ARRAY[1::bigint], 6)") - stage("neighbours", f"SELECT count(*) FROM warren.neighbours('{GRAPH}', 1::bigint)", 3 if reloads else 20, True) - stage("reachable_set", f"SELECT count(*) FROM warren.reachable_set('{GRAPH}', ARRAY[1::bigint])") + first = node(case.ids, 1) + stage("bfs depth 6", f"SELECT count(*) FROM warren.bfs('{GRAPH}', ARRAY[{first}], 6)", 3, True) + stage("bfs depth 6 count", f"SELECT count(*) FROM warren.bfs('{GRAPH}', ARRAY[{first}], 6)") + stage("neighbours", f"SELECT count(*) FROM warren.neighbours('{GRAPH}', {first})", 3 if reloads else 20, True) + stage("reachable_set", f"SELECT count(*) FROM warren.reachable_set('{GRAPH}', ARRAY[{first}])") self.stage(result, f"{label}: reachable_set stats", {"stats": s.run("SELECT warren.last_stats()")["rows"]}) stage("components", f"SELECT count(*) FROM (SELECT component_id FROM warren.components('{GRAPH}', " - "NULL::bigint) GROUP BY 1) c") + f"NULL::{case.ids}) GROUP BY 1) c") stage("pagerank", pagerank_sql(GRAPH)) # From a cold cache: one statement, so it loads the version once # even when the version and its scratch exceed the cap (§5.5). @@ -788,7 +816,7 @@ def _library(self, result, case): if "library" not in self.options.skip: with self.session() as s: self._configure(s, self.first_cap()) - for call in library_calls(GRAPH, case.nodes): + for call in library_calls(GRAPH, case.nodes, case.ids): self._call(s, result, "library", call) if "communities" in self.options.skip: return @@ -799,7 +827,7 @@ def _library(self, result, case): stage = f"communities: {call.name}" self.stage(result, stage, self._guarded(s, call.sql)) if "errors" not in result["stages"][stage]: - self.stage(result, f"{stage} stats", {"rows": s.run(community_stats_sql(GRAPH, call.name))["rows"]}) + self.stage(result, f"{stage} stats", {"rows": s.run(community_stats_sql(GRAPH, call.name, case.ids))["rows"]}) after, free_after = s.run(DISK_SQL)["rows"][0].split("|"), self.free_disk() for sql in call.cleanup: s.run(sql) @@ -1034,6 +1062,8 @@ def main(argv): help="preload pg_warren and set warren.shared_cache_memory to this (e.g. 8GB); " "default: the server's default, private on PG 15 and 16") options.add_argument("--statement-timeout", help="statement_timeout of the read workloads, e.g. 15min") + options.add_argument("--ids", choices=["bigint", "text"], default="bigint", + help="the node ids: bigint numbers, or text IRIs (DESIGN.md §5.7)") options.add_argument("--node-order", choices=["id", "locality"], default="id", help="create_graph's node_order: how the first build orders dense ids") options.add_argument("--skip", action="append", default=[], choices=["second-version", "library", "communities"], @@ -1056,11 +1086,12 @@ def main(argv): settings=cluster_settings(args.shared_cache_memory)) as cluster: soak = Soak(cluster, args, found) for name in args.cases: - case = CASES[name].scaled(args.scale) if args.scale > 1 else CASES[name] + case = replace(CASES[name], ids=args.ids) + case = case.scaled(args.scale) if args.scale > 1 else case print(f"{name}: {case.nodes} nodes, {case.edges} edges, free disk {size(free_disk(args.parent))}", flush=True) result = soak.run(case) - suffix = f"-scale{args.scale}" if args.scale > 1 else "" + suffix = (f"-scale{args.scale}" if args.scale > 1 else "") + ("-text" if args.ids == "text" else "") out = args.results / f"{name}{suffix}{args.label}.json" out.write_text(json.dumps(result, indent=1) + "\n") for verdict in result["verdicts"]: diff --git a/bench/tests/test_soak.py b/bench/tests/test_soak.py index b5bfedfb..0aeb5ae7 100644 --- a/bench/tests/test_soak.py +++ b/bench/tests/test_soak.py @@ -305,7 +305,7 @@ def test_the_settings_are_recorded_after_each_pass_sets_them_and_loads_the_libra runner = soak.Soak(None, options, provenance={}) runner.session = lambda: FakeSession(log) result = {"stages": {}, "cached_bytes": GIB} - runner._queries(result, "raised", cap="8GB") + runner._queries(result, soak.CASES["a"], "raised", cap="8GB") snapshot = next(i for i, sql in enumerate(log) if "pg_settings" in sql) assert snapshot > max(i for i, sql in enumerate(log) if sql.startswith("SET ")) # pg_settings lists warren.* only once a call has loaded the library. @@ -330,7 +330,7 @@ def test_the_first_pass_runs_at_the_given_cap_and_counts_pagerank_against_it(): options.eigen = False runner = soak.Soak(None, options, provenance={}) runner.session = lambda: FakeSession(log) - runner._queries({"stages": {}, "cached_bytes": GIB}, "default", cap=runner.first_cap()) + runner._queries({"stages": {}, "cached_bytes": GIB}, soak.CASES["a"], "default", cap=runner.first_cap()) assert "SET warren.max_cache_memory = '418945kB'" in log assert any("warren.pagerank" in sql and "pg_stat_get_xact" in sql for sql in log) # Then once more from a cold cache, which must load the version once. @@ -549,3 +549,39 @@ def test_a_shared_version_needs_no_raised_pass(): assert not soak.needs_raised_pass({**over, "shared_bytes": 6 * GIB}, id_space) assert soak.needs_raised_pass({"cached_bytes": None, "cache_memory_bytes": 4 * GIB}, id_space) assert not soak.needs_raised_pass({"cached_bytes": GIB, "cache_memory_bytes": 4 * GIB}, id_space) + + +TEXT = "https://example.org/resource/" + + +def test_a_text_case_names_each_node_by_an_iri(): + case = soak.replace(soak.CASES["a"], ids="text") + assert soak.edge_query(case, seed=0.25) == ( + f"SELECT '{TEXT}' || src, '{TEXT}' || dst FROM soak_edges") + sql = soak.create_graph_sql(case, seed=0.25) + assert f"$n$SELECT '{TEXT}' || id FROM generate_series(1::bigint, 10000000) id$n$" in sql + assert soak.CASES["a"].ids == "bigint" + + +def test_a_text_case_estimates_its_id_map_from_its_iris(): + small = soak.replace(soak.CASES["a"], ids="text").scaled(1_000_000) # 10 nodes, 100 edges + # Offsets 8(N+1) and targets 4M per direction; then the IRIs' bytes + # (nine of one digit, one of two), their offsets 8(N+1), their u32 + # order 4N, and the tombstones N/8. + iris = 10 * len(TEXT) + 9 + 2 + assert soak.segment_estimate(small) == 2 * (8 * 11 + 4 * 100) + iris + 8 * 11 + 4 * 10 + 1 + + +def test_a_text_case_passes_text_ids_to_the_calls(): + calls = {call.name: call for call in soak.library_calls("soak", nodes=10, ids="text")} + assert "NULL::text" in calls["minimum_spanning_forest"].sql + assert f"'{TEXT}1'::text" in calls["most_similar"].sql + assert f"'{TEXT}' || (1 + (i * 7919) % 10)" in calls["similarity"].sql + assert f"SELECT '{TEXT}' || id, 1 FROM generate_series(1::bigint, 10, 100) id" in calls["similar_pairs"].setup[0] + assert "NULL::text" in soak.community_stats_sql("soak", "louvain", ids="text") + + +def test_a_full_text_case_checks_its_edges_as_its_bigint_case_does(): + case = soak.replace(soak.CASES["a"], ids="text") + with pytest.raises(soak.GeneratorDrift): + soak.check_edges(case, {"deduplicated_edges": 0, "edge_count": case.edges}) diff --git a/crates/pg_warren/src/build/error.rs b/crates/pg_warren/src/build/error.rs index c3dcb52e..3fc089a8 100644 --- a/crates/pg_warren/src/build/error.rs +++ b/crates/pg_warren/src/build/error.rs @@ -2,13 +2,14 @@ use pgrx::PgSqlErrorCode; use thiserror::Error; +use warren_core::compute::ComputeError; use warren_core::graph::GraphError; use crate::access::AccessError; use crate::access::SqlError; use crate::cache::LoadError; use crate::catalog::CatalogError; -use crate::extension::graph_sqlstate; +use crate::extension::{compute_sqlstate, graph_sqlstate}; use crate::segment::SegmentError; use crate::source::SourceError; @@ -36,6 +37,9 @@ pub(crate) enum BuildError { /// The graph structure could not be built, encoded or decoded. #[error(transparent)] Graph(#[from] GraphError), + /// Mapping edge endpoints on the compute pool failed or was cancelled. + #[error(transparent)] + Compute(#[from] ComputeError), } impl SqlError for BuildError { @@ -48,6 +52,7 @@ impl SqlError for BuildError { Self::Load(error) => error.sqlstate(), Self::Spi(_) => PgSqlErrorCode::ERRCODE_INTERNAL_ERROR, Self::Graph(error) => graph_sqlstate(error), + Self::Compute(error) => compute_sqlstate(error), } } } diff --git a/crates/pg_warren/src/build/pipeline.rs b/crates/pg_warren/src/build/pipeline.rs index cbfc0cbc..63f16fec 100644 --- a/crates/pg_warren/src/build/pipeline.rs +++ b/crates/pg_warren/src/build/pipeline.rs @@ -1,5 +1,6 @@ //! The build pipeline (DESIGN.md §5.4 steps 2–6), over the build's ports: -//! stable-id merge against the base version's id map, edge intake, the +//! stable-id merge against the base version's id map, edge intake (each +//! batch of rows mapped to dense ids on the compute pool, §5.7), the //! locality order of a first build (§14 item 22), CSR build, chunked //! segment write, version row and optional promotion. Graph logic is //! `warren_core::graph`'s; this module only sequences it, with @@ -7,18 +8,27 @@ //! budget: a build that would exceed it is refused with 53400 before its //! CSRs are built where the plan shows it, else where it reserves. +use warren_core::compute::Workers; use warren_core::graph::{ - BuildCharge, BuildMemory, BuildPlan, CsrLayout, EdgeIntake, EdgeList, ExternalId, FixedKeys, - GraphCsr, GraphError, IdMap, Membership, NodeOrder, Permutation, RelationDictionary, + BuildCharge, BuildMemory, BuildPlan, CsrLayout, EdgeBatch, EdgeIntake, EdgeList, GraphCsr, + GraphError, IdMap, IndexedKeys, Membership, NodeOrder, Permutation, RelationDictionary, RelationIndex, }; +use warren_core::interrupts::Interrupts; use super::error::BuildError; use super::source::{EdgeRow, EdgeSink, SourceReader}; use super::version::Catalog; use crate::cache::{load_identity, Identity}; use crate::catalog::{GraphDefinition, Lineage, NewVersion, VersionId, VersionStats}; -use crate::segment::{writer_memory, SegmentStore, SegmentWriter}; +use crate::segment::{writer_memory, SegmentStore, SegmentWriter, StoredIds}; + +/// Edge rows mapped to dense ids together, on the compute pool. +const BATCH_ROWS: usize = 65_536; + +/// The share of the build's budget a batch may hold before it is mapped +/// (1/32), so that a small budget is left to the edge list. +const BATCH_SHARE: u64 = 32; /// The name of the one relation of a graph without relations. const IMPLICIT_RELATION: &str = ""; @@ -45,12 +55,14 @@ pub(crate) struct Built { pub(crate) memory_peak: u64, } -/// Builds a new version of `graph` from its sources. Dense ids continue -/// from the graph's current version, if it has one. -pub(crate) fn build( +/// Builds a new version of `graph` from its sources, mapping edge +/// endpoints on `pool`. Dense ids continue from the graph's current +/// version, if it has one. +pub(crate) fn build( graph: &GraphDefinition, request: &RefreshRequest<'_>, source: &mut impl SourceReader, + pool: (&Workers, &mut impl Interrupts), segments: &mut impl SegmentStore, catalog: &mut impl Catalog, ) -> Result { @@ -65,8 +77,9 @@ pub(crate) fn build( }, }; let Identity { id_map: base, id_map_charge: base_charge, relations, relations_charge } = base; - let (mut id_map, _id_map_charge) = base.rebuild_within(source.node_ids(&memory)?)?; - drop((base, base_charge)); + // The base is consumed: a text id map's bytes and offsets are extended + // in place (DESIGN.md §5.7). + let (mut id_map, _id_map_charge) = base.rebuild_into(base_charge, source.node_ids(&memory)?)?; let layout = CsrLayout { build_reverse: graph.build_reverse, store_merged: graph.store_merged }; let mut relations = Relations { dictionary: relations, @@ -81,7 +94,7 @@ pub(crate) fn build( }; relations.check_offsets()?; let (mut edges, dropped_edges) = - edges(graph, &id_map, implicit, &mut relations, source, &memory)?; + edges(graph, &id_map, implicit, &mut relations, source, pool, &memory)?; if graph.node_order == NodeOrder::Locality && graph.current_version.is_none() { // A first build: its ids are provisional until ordered (§14 item 22). let permutation = Permutation::locality(&edges, id_map.id_space(), &memory)?; @@ -98,7 +111,7 @@ pub(crate) fn build( /// contraction made them. pub(crate) struct Contents { /// The id map, tombstones included. - pub(crate) id_map: IdMap>, + pub(crate) id_map: IdMap, /// The relation dictionary. pub(crate) relations: RelationDictionary, /// The edges, in dense ids. @@ -121,7 +134,7 @@ pub(crate) struct Derivation<'m> { /// its CSRs, then its identity segments and, for a contraction, its /// `membership`, into `segments`; records its statistics and bytes; and /// promotes it if asked. -pub(crate) fn write_contents( +pub(crate) fn write_contents( graph: &GraphDefinition, request: &RefreshRequest<'_>, contents: Contents, @@ -160,15 +173,18 @@ pub(crate) fn write_contents( } /// Reads the edge rows into an edge list charged to `memory`, interning new -/// relations. Returns it, without spare capacity, and the number of -/// dropped edges: rows with a NULL relation or weight the graph needs, or a -/// NULL, unknown or tombstoned endpoint. -fn edges( +/// relations, and mapping each batch of rows' endpoints on `pool` through +/// an index of `id_map` (a text id map's hash index, charged to `memory` +/// and freed with the intake). Returns the edges, without spare capacity, +/// and the number of dropped edges: rows with a NULL relation or weight +/// the graph needs, or a NULL, unknown or tombstoned endpoint. +fn edges( graph: &GraphDefinition, - id_map: &IdMap>, + id_map: &IdMap, implicit: Option, relations: &mut Relations, source: &mut impl SourceReader, + pool: (&Workers, &mut impl Interrupts), memory: &BuildMemory, ) -> Result<(EdgeList, u64), BuildError> { let intake = match graph.directed { @@ -176,8 +192,23 @@ fn edges( false => EdgeIntake::undirected(id_map, graph.weighted), }; let intake = intake.within(memory); - let mut collector = EdgeCollector { graph, implicit, relations, intake, incomplete_rows: 0 }; + let index = E::index(id_map, memory)?; + let size = BatchSize::within::(memory.limit()); + let (workers, interrupts) = pool; + let mut collector = EdgeCollector { + graph, + implicit, + relations, + intake, + batch: EdgeBatch::with_capacity(memory, size.rows, size.id_bytes)?, + size, + index: &index, + workers, + interrupts, + incomplete_rows: 0, + }; source.edges(&mut collector)?; + collector.flush()?; let EdgeCollector { intake, incomplete_rows, .. } = collector; let (edges, dropped_endpoints) = intake.finish(); Ok((edges, dropped_endpoints + incomplete_rows)) @@ -211,18 +242,67 @@ impl Relations { } } -/// The edge sink of a build: remaps rows through the edge intake. -struct EdgeCollector<'b, 'm, E> { +/// How far a build's batch of edge rows grows before it is mapped. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +struct BatchSize { + /// The rows the batch has room for. + rows: usize, + /// The bytes of text ids past which the batch is mapped before its + /// rows are full, and the room for them it keeps once mapped. + id_bytes: u64, +} + +impl BatchSize { + /// A batch's share of a budget of `limit` bytes: half of it for the + /// rows' fixed columns (at most [`BATCH_ROWS`] rows), the rest for + /// their text ids. + fn within(limit: u64) -> Self { + let share = limit / BATCH_SHARE; + let row = EdgeBatch::::ROW_BYTES; + let rows = usize::try_from(share / 2 / row).unwrap_or(BATCH_ROWS).clamp(1, BATCH_ROWS); + let id_bytes = share.saturating_sub(rows as u64 * row); + Self { rows, id_bytes } + } + + /// Whether `batch` is to be mapped now: its rows are full, or its text + /// ids are past their share. + fn is_full(&self, batch: &EdgeBatch) -> bool { + batch.len() >= self.rows || batch.len_bytes() > self.id_bytes + } +} + +/// The edge sink of a build: collects rows into a batch, maps each full +/// batch (of [`BATCH_ROWS`] rows, or a share of the budget) on the compute +/// pool, and takes it into the edge intake. +struct EdgeCollector<'b, 'm, E: IndexedKeys, I> { graph: &'b GraphDefinition, /// The one relation of a graph without relations. implicit: Option, relations: &'b mut Relations, - intake: EdgeIntake<'m, FixedKeys>, + intake: EdgeIntake<'m, E>, + batch: EdgeBatch, + size: BatchSize, + index: &'b E::Index<'m>, + workers: &'b Workers, + interrupts: &'b mut I, /// Rows dropped for a NULL relation or weight. incomplete_rows: u64, } -impl EdgeSink for EdgeCollector<'_, '_, E> { +impl EdgeCollector<'_, '_, E, I> { + /// Maps the batch's endpoints on the pool (the main thread polls for + /// interrupts meanwhile) and takes its rows into the intake. + fn flush(&mut self) -> Result<(), BuildError> { + if self.batch.is_empty() { + return Ok(()); + } + let (batch, index) = (&mut self.batch, self.index); + self.workers.run(self.interrupts, |cancellation| Ok(batch.map(index, cancellation)?))?; + Ok(self.intake.take(&mut self.batch)?) + } +} + +impl EdgeSink for EdgeCollector<'_, '_, E, I> { /// Reserves the edge list for `rows` up front, so that it is not grown /// geometrically past its final size. A reservation that fails (an /// overestimate, or one past the build's budget) is ignored: the list @@ -233,24 +313,28 @@ impl EdgeSink for EdgeCollector<'_, '_, E> { } } - fn row(&mut self, row: EdgeRow<'_, E>) -> Result<(), BuildError> { + fn row(&mut self, row: EdgeRow<'_, E::Key>) -> Result<(), BuildError> { let Some((relation, weight)) = relation_and_weight(self.graph, self.implicit, self.relations, &row)? else { self.incomplete_rows += 1; return Ok(()); }; - Ok(self.intake.push(relation, row.source.as_ref(), row.target.as_ref(), weight)?) + self.batch.push(relation, row.source, row.target, weight)?; + if self.size.is_full(&self.batch) { + self.flush()?; + } + Ok(()) } } /// The relation and weight of an edge row, or `None` if a field the graph /// needs is NULL. -fn relation_and_weight( +fn relation_and_weight( graph: &GraphDefinition, implicit: Option, relations: &mut Relations, - row: &EdgeRow<'_, E>, + row: &EdgeRow<'_, K>, ) -> Result, BuildError> { let relation = match (implicit, row.relation) { (Some(relation), _) => relation, @@ -276,8 +360,12 @@ mod tests { use crate::segment::{read_segment, MemorySegmentStore}; use pgrx::pg_sys; use warren_core::graph::{ - AnyCsr, EdgeIntake, FixedKeys, GraphCsr, GraphError, IdMap, RelationDictionary, Tombstones, + AnyCsr, EdgeIntake, FixedKeys, GraphCsr, GraphError, IdMap, RelationDictionary, TextIds, + TextKeys, Tombstones, }; + use warren_core::interrupts::Uninterrupted; + + use crate::cache::load_identity; fn graph() -> GraphDefinition { GraphDefinition { @@ -333,19 +421,46 @@ mod tests { }; let graph = GraphDefinition { current_version: self.catalog.current_version, ..graph.clone() }; - self.try_refresh(&graph, &request, source).unwrap() + self.try_refresh::, _>(&graph, &request, source).unwrap() } - fn try_refresh( + fn try_refresh( &mut self, graph: &GraphDefinition, request: &RefreshRequest<'_>, - source: &MemorySource, - ) -> Result { - build(graph, request, &mut source.clone(), &mut self.store, &mut self.catalog) + source: &MemorySource, + ) -> Result + where + MemorySource: SourceReader, + { + let workers = Workers::new(3).unwrap(); + let pool = (&workers, &mut Uninterrupted); + build(graph, request, &mut source.clone(), pool, &mut self.store, &mut self.catalog) .map(|built| built.version) } + /// [`World::refresh`] of a text graph. + fn refresh_text( + &mut self, + graph: &GraphDefinition, + source: &MemorySource>, + ) -> VersionId { + let request = RefreshRequest { + promote: true, + label: None, + chunk_size: 16, + memory_budget: u64::MAX, + }; + let graph = + GraphDefinition { current_version: self.catalog.current_version, ..graph.clone() }; + self.try_refresh::(&graph, &request, source).unwrap() + } + + fn text_id_map(&mut self, version: VersionId) -> IdMap { + let memory = BuildMemory::unlimited(); + load_identity::(&mut self.store, version, &memory).unwrap().id_map + } + fn segment(&mut self, version: VersionId, name: &str) -> Vec { read_segment(&mut self.store, version, name).unwrap() } @@ -684,7 +799,7 @@ mod tests { let source = MemorySource { nodes: (0..10).collect(), ..MemorySource::default() }; let request = RefreshRequest { promote: true, label: None, chunk_size: 16, memory_budget: 64 }; - let refused = world.try_refresh(&graph(), &request, &source); + let refused = world.try_refresh::, _>(&graph(), &request, &source); assert!(matches!( refused, Err(BuildError::Graph(GraphError::BuildMemoryLimit { limit: 64, .. })) @@ -705,7 +820,7 @@ mod tests { }; let request = RefreshRequest { promote: true, label: None, chunk_size: 16, memory_budget: 10_000 }; - let refused = world.try_refresh(&graph, &request, &source); + let refused = world.try_refresh::, _>(&graph, &request, &source); // Refused at the 56th relation, while reading: 57 CSRs (with the // merged one) × 2 directions × 11 offsets × 8 bytes, not after all 100. assert!( @@ -747,7 +862,7 @@ mod tests { chunk_size: 16, memory_budget: budget, }; - match world.try_refresh(&graph, &request, &source) { + match world.try_refresh::, _>(&graph, &request, &source) { Ok(version) => { built = Some((budget, world.catalog.version(version).stats)); break; @@ -786,7 +901,7 @@ mod tests { let source = one_relation_per_row(10, 30_000, "relation"); let request = RefreshRequest { promote: true, label: None, chunk_size: 16, memory_budget: 1 << 20 }; - let refused = world.try_refresh(&graph, &request, &source); + let refused = world.try_refresh::, _>(&graph, &request, &source); assert!( matches!(refused, Err(BuildError::Graph(GraphError::OffsetsOverBudget { .. }))), "{refused:?}" @@ -806,7 +921,7 @@ mod tests { let source = one_relation_per_row(1, 30_000, name); let request = RefreshRequest { promote: true, label: None, chunk_size: 16, memory_budget: 1 << 20 }; - let refused = world.try_refresh(&graph, &request, &source); + let refused = world.try_refresh::, _>(&graph, &request, &source); assert!( matches!( refused, @@ -832,4 +947,189 @@ mod tests { assert_eq!(world.catalog.version(version).stats.edge_count, 1); } } + + fn text(id: &str) -> Vec { + id.as_bytes().to_vec() + } + + fn text_edge(source: &str, target: &str) -> OwnedEdgeRow> { + (Some(text(source)), Some(text(target)), None, None) + } + + fn text_graph() -> GraphDefinition { + GraphDefinition { id_type: crate::source::IdType::Text, ..graph() } + } + + fn text_neighbours(csr: &AnyCsr, id_map: &IdMap, external: &str) -> Vec { + let dense = id_map.dense_id(external.as_bytes()).unwrap(); + let ids = csr.neighbours(dense).map(|d| id_map.external_id(d).unwrap().to_vec()); + ids.map(|id| String::from_utf8(id).unwrap()).collect() + } + + #[test] + fn a_text_graph_builds_its_three_id_segments_in_byte_order() { + let mut world = World::default(); + let source = MemorySource { + nodes: ["b", "a", "", "ab", "a"].map(text).to_vec(), + edges: vec![ + text_edge("a", "b"), + text_edge("", "ab"), + text_edge("ab", "missing"), + (None, Some(text("a")), None, None), + ], + ..MemorySource::default() + }; + let version = world.refresh_text(&text_graph(), &source); + for name in ["id_bytes", "id_offsets", "id_order"] { + assert!(world.has_segment(version, name), "{name}"); + } + assert!(!world.has_segment(version, "id_map")); + let id_map = world.text_id_map(version); + for (dense, id) in ["", "a", "ab", "b"].into_iter().enumerate() { + assert_eq!(id_map.dense_id(id.as_bytes()), Some(dense as u64), "{id:?}"); + } + let stats = world.catalog.version(version).stats; + assert_eq!((stats.node_count, stats.edge_count, stats.dropped_edges), (4, 2, 2)); + assert_eq!(world.catalog.version(version).bytes, world.stored_bytes(version)); + let forward = world.csr(version, "", "0"); + assert_eq!(text_neighbours(&forward, &id_map, "a"), ["b"]); + assert_eq!(text_neighbours(&forward, &id_map, ""), ["ab"]); + } + + #[test] + fn text_dense_ids_are_stable_across_builds() { + let mut world = World::default(); + let first = MemorySource { + nodes: ["x", "y", "z"].map(text).to_vec(), + edges: vec![text_edge("x", "y")], + ..MemorySource::default() + }; + let v1 = world.refresh_text(&text_graph(), &first); + let second = MemorySource { + nodes: ["w", "x", "z", "zz"].map(text).to_vec(), + edges: vec![text_edge("x", "y"), text_edge("zz", "w")], + ..MemorySource::default() + }; + let v2 = world.refresh_text(&text_graph(), &second); + let (before, after) = (world.text_id_map(v1), world.text_id_map(v2)); + for survivor in ["x", "z"] { + assert_eq!(after.dense_id(survivor.as_bytes()), before.dense_id(survivor.as_bytes())); + } + assert_eq!(after.dense_id(b"y"), None); + assert_eq!(after.external_id(1), Some(b"y".as_slice()), "a tombstone keeps its id"); + assert_eq!((after.dense_id(b"w"), after.dense_id(b"zz")), (Some(3), Some(4))); + let stats = world.catalog.version(v2).stats; + assert_eq!((stats.id_space, stats.edge_count, stats.dropped_edges), (5, 1, 1)); + } + + #[test] + fn a_text_graph_in_locality_order_renumbers_its_first_build() { + let mut world = World::default(); + let graph = GraphDefinition { node_order: NodeOrder::Locality, ..text_graph() }; + let path = [("a", "d"), ("d", "b"), ("b", "e"), ("e", "c")]; + let source = MemorySource { + nodes: ["a", "b", "c", "d", "e"].map(text).to_vec(), + edges: path.iter().map(|&(s, t)| text_edge(s, t)).collect(), + ..MemorySource::default() + }; + let version = world.refresh_text(&graph, &source); + let id_map = world.text_id_map(version); + let dense = |id: &str| id_map.dense_id(id.as_bytes()).unwrap() as i64; + for (source, target) in path { + assert_eq!((dense(source) - dense(target)).abs(), 1, "{source} - {target}"); + } + let forward = world.csr(version, "", "0"); + assert_eq!(text_neighbours(&forward, &id_map, "d"), ["b"]); + } + + /// How many times a batch sized for a budget of `limit` is mapped + /// while `rows` rows from `source` to `target` are pushed into it after + /// one row whose endpoints are `first`, the last partial batch + /// included; and the most bytes the batch held after a mapping. + fn flushes( + limit: u64, + id_map: &IdMap, + first: &E::Key, + endpoint: &E::Key, + rows: usize, + ) -> (usize, u64) { + let memory = BuildMemory::new(limit); + let size = BatchSize::within::(limit); + let relation = RelationDictionary::new().intern("r").unwrap(); + let mut intake = EdgeIntake::directed(id_map, false).within(&memory); + let mut batch = EdgeBatch::::with_capacity(&memory, size.rows, size.id_bytes).unwrap(); + let (mut flushes, mut held) = (0, 0); + for row in 0..=rows { + let key = if row == 0 { first } else { endpoint }; + batch.push(relation, Some(key), Some(key), 1.0).unwrap(); + if size.is_full(&batch) { + flushes += 1; + intake.take(&mut batch).unwrap(); + held = held.max(batch.bytes()); + } + } + (flushes + usize::from(!batch.is_empty()), held) + } + + /// A batch is mapped when its rows are full or its text ids pass their + /// share of the budget, never on every row: not for text ids at a + /// small budget, nor for a fixed type at a budget its rows fill + /// exactly, nor after one long text id outgrew the share. + #[test] + fn a_budget_limited_batch_is_mapped_once_it_is_full() { + const MB: u64 = 1 << 20; + let rows = 100_000; + let fixed = IdMap::>::new().rebuild(vec![7]).unwrap(); + for limit in [19 * MB, 38 * MB, 76 * MB, 64 * MB] { + let size = BatchSize::within::>(limit); + let (flushes, _) = flushes(limit, &fixed, &7, &7, rows); + assert_eq!(flushes, (rows + 1).div_ceil(size.rows), "int8 at {limit}"); + } + let iri = b"https://example.org/resource/1234567".as_slice(); + let long = vec![b'x'; 3 * MB as usize]; + let memory = BuildMemory::unlimited(); + let mut ids = TextIds::new(&memory); + for id in [iri, long.as_slice()] { + ids.push(id).unwrap(); + } + let text = IdMap::::new().rebuild_within(ids).unwrap().0; + for limit in [64 * MB, 4096 * MB] { + let size = BatchSize::within::(limit); + let (flushes, _) = flushes(limit, &text, iri, iri, rows); + assert!(flushes <= (rows + 1).div_ceil(size.rows) + 8, "text at {limit}: {flushes}"); + } + let (flushes, held) = flushes(64 * MB, &text, &long, iri, rows); + assert!(flushes <= 12, "text after a long id: {flushes}"); + assert!(held <= 64 * MB / BATCH_SHARE, "the long id's room is freed: {held}"); + } + + /// Rows are mapped on the pool a batch at a time: a build of more rows + /// than a batch takes every one, in order, for either kind of id. + #[test] + fn edge_rows_past_a_batch_are_all_taken() { + let rows = BATCH_ROWS as i64 + 100; + let mut world = World::default(); + let source = MemorySource { + nodes: (0..1000).collect(), + edges: (0..rows).map(|r| edge(r % 1000, (r * 3 + 1) % 1001)).collect(), + ..MemorySource::default() + }; + let version = world.refresh(&graph(), &source, true); + let stats = world.catalog.version(version).stats; + assert_eq!(stats.edge_count + stats.deduplicated_edges + stats.dropped_edges, rows as u64); + assert!(stats.dropped_edges > 0, "edges to 1000 are dropped"); + + let as_text = MemorySource { + nodes: source.nodes.iter().map(|n| text(&n.to_string())).collect(), + edges: source + .edges + .iter() + .map(|(s, t, _, _)| text_edge(&s.unwrap().to_string(), &t.unwrap().to_string())) + .collect(), + ..MemorySource::default() + }; + let mut text_world = World::default(); + let text_version = text_world.refresh_text(&text_graph(), &as_text); + assert_eq!(text_world.catalog.version(text_version).stats, stats); + } } diff --git a/crates/pg_warren/src/build/refresh.rs b/crates/pg_warren/src/build/refresh.rs index 794227b3..c1134ffd 100644 --- a/crates/pg_warren/src/build/refresh.rs +++ b/crates/pg_warren/src/build/refresh.rs @@ -11,6 +11,7 @@ use super::spi_catalog::SpiCatalog; use super::spi_source::SpiSource; use crate::access::{entry, require_owner}; use crate::catalog::{require_current_id_space, GraphDefinition, VersionId}; +use crate::interrupts::{workers, PostgresInterrupts}; use crate::segment::{SpiSegmentStore, CHUNK_SIZE}; use crate::source::{with_id_type, with_pinned_snapshot}; @@ -46,6 +47,7 @@ fn refresh_graph(name: &str, promote: bool, label: Option<&str>) -> Result Result<(), Error> { @@ -260,6 +263,45 @@ mod tests { Ok(()) } + /// DESIGN.md §5.7: a text graph's ids are its id map's three + /// segments, in bytewise order, and keep their dense ids across + /// refreshes; an id is any bytes, the empty one included. + #[pg_test] + fn text_graphs_build_with_stable_ids() -> Result<(), Error> { + Spi::run( + "CREATE TABLE t_nodes (id text); + CREATE TABLE t_edges (src varchar, dst text); + INSERT INTO t_nodes VALUES ('b'), (''), ('é'), ('B'); + INSERT INTO t_edges VALUES ('', 'b'), ('b', 'é'), ('é', 'unknown'), (NULL, 'b'); + SELECT warren.create_graph('t', 'SELECT id FROM t_nodes', + 'SELECT src, dst FROM t_edges')", + )?; + let first = refresh("SELECT warren.refresh('t')")?; + assert_eq!(version_row(first)?.as_deref(), Some("1|f|4|4|2|2|0|4|t|")); + let segments = Spi::get_one_with_args::( + "SELECT string_agg(DISTINCT segment, ' ' ORDER BY segment) FROM warren.segments + WHERE version_id = $1 AND segment LIKE 'id%'", + &[first.into()], + )?; + assert_eq!(segments.as_deref(), Some("id_bytes id_offsets id_order")); + let id_map = |version| { + let store = &mut SpiSegmentStore; + let tombstones = read_segment(store, version, "tombstones").unwrap_or_default(); + let memory = BuildMemory::unlimited(); + TextKeys::read_id_map(store, version, &tombstones, &memory).map(|(id_map, _)| id_map) + }; + let ids = id_map(first).map_err(|_| Error::InvalidPosition)?; + let order: Vec<_> = (0..4).filter_map(|dense| ids.external_id(dense)).collect(); + assert_eq!(order, [&b""[..], b"B", b"b", "é".as_bytes()]); + Spi::run("DELETE FROM t_nodes WHERE id = 'B'; INSERT INTO t_nodes VALUES ('a')")?; + let next = refresh("SELECT warren.refresh('t')")?; + let ids = id_map(next).map_err(|_| Error::InvalidPosition)?; + assert_eq!(ids.dense_id(b"b"), Some(2), "a kept id keeps its dense id"); + assert_eq!(ids.dense_id(b"a"), Some(4), "a new id is appended"); + assert_eq!(ids.dense_id(b"B"), None, "a dropped id is tombstoned"); + Ok(()) + } + #[pg_test] fn a_rolled_back_staged_refresh_leaves_no_rows() -> Result<(), Error> { ring()?; diff --git a/crates/pg_warren/src/build/source.rs b/crates/pg_warren/src/build/source.rs index 34b15d7e..fac5ce73 100644 --- a/crates/pg_warren/src/build/source.rs +++ b/crates/pg_warren/src/build/source.rs @@ -2,79 +2,89 @@ //! read under one snapshot (DESIGN.md §5.4 steps 1–3). Its SPI adapter is //! [`super::spi_source::SpiSource`]. -use warren_core::graph::{BuildBuffer, BuildMemory}; +use warren_core::graph::{BuildMemory, IdKeys}; use super::error::BuildError; -/// One row of an edge query. Fields the graph does not have are `None`. -#[derive(Debug, Clone, Copy, PartialEq)] -pub(crate) struct EdgeRow<'r, E> { +/// One row of an edge query, read in place. Fields the graph does not have +/// are `None`. +#[derive(Debug, PartialEq)] +pub(crate) struct EdgeRow<'r, K: ?Sized> { /// The external id of the source node. - pub(crate) source: Option, + pub(crate) source: Option<&'r K>, /// The external id of the target node. - pub(crate) target: Option, + pub(crate) target: Option<&'r K>, /// The relation name, if the graph has relations. pub(crate) relation: Option<&'r str>, /// The weight, if the graph is weighted. pub(crate) weight: Option, } -/// Reads a graph's node ids and edge rows. -pub(crate) trait SourceReader { - /// Every non-NULL external id the node query returns, in a buffer - /// charged to `memory`. - fn node_ids(&mut self, memory: &BuildMemory) -> Result, BuildError>; +/// Reads the node ids and edge rows of a graph whose ids `K` stores. +pub(crate) trait SourceReader { + /// Every non-NULL external id the node query returns, charged to + /// `memory`. + fn node_ids(&mut self, memory: &BuildMemory) -> Result; /// Passes every row of the edge query to `sink`, stopping at its first /// error, after telling it roughly how many rows to expect. - fn edges(&mut self, sink: &mut dyn EdgeSink) -> Result<(), BuildError>; + fn edges(&mut self, sink: &mut dyn EdgeSink) -> Result<(), BuildError>; } -/// Takes the rows of an edge query. -pub(crate) trait EdgeSink { +/// Takes the rows of an edge query, whose ids are `K`s. +pub(crate) trait EdgeSink { /// A hint, before the first row: about how many rows follow (for /// example the planner's estimate). It may be wrong either way. fn expect_rows(&mut self, rows: u64); /// Takes one row. - fn row(&mut self, row: EdgeRow<'_, E>) -> Result<(), BuildError>; + fn row(&mut self, row: EdgeRow<'_, K>) -> Result<(), BuildError>; } #[cfg(test)] #[allow(clippy::disallowed_methods)] // Test code may assign a transaction id. pub(super) mod tests { + use warren_core::graph::{FixedKeys, TextIds, TextKeys}; + use super::*; /// An edge row owned by a [`MemorySource`]. - pub(in crate::build) type OwnedEdgeRow = - (Option, Option, Option, Option); - - /// An in-memory [`SourceReader`]. - #[derive(Debug, Default, Clone)] - pub(in crate::build) struct MemorySource { - pub(in crate::build) nodes: Vec, - pub(in crate::build) edges: Vec, + pub(in crate::build) type OwnedEdgeRow = + (Option, Option, Option, Option); + + /// An in-memory [`SourceReader`], of `int8` ids or (`Vec`) text. + #[derive(Debug, Clone)] + pub(in crate::build) struct MemorySource { + pub(in crate::build) nodes: Vec, + pub(in crate::build) edges: Vec>, /// The row estimate to announce; the true count if `None`. pub(in crate::build) estimated_rows: Option, /// How many edge rows the sink has taken, shared by clones. pub(in crate::build) rows_read: std::rc::Rc>, } - impl SourceReader for MemorySource { - fn node_ids(&mut self, memory: &BuildMemory) -> Result, BuildError> { - let mut ids = BuildBuffer::new(memory); - for &id in &self.nodes { - ids.push(id)?; + impl Default for MemorySource { + fn default() -> Self { + Self { + nodes: Vec::new(), + edges: Vec::new(), + estimated_rows: None, + rows_read: std::rc::Rc::default(), } - Ok(ids) } + } - fn edges(&mut self, sink: &mut dyn EdgeSink) -> Result<(), BuildError> { + impl MemorySource { + fn each_edge( + &mut self, + sink: &mut dyn EdgeSink, + key: impl Fn(&T) -> &K, + ) -> Result<(), BuildError> { sink.expect_rows(self.estimated_rows.unwrap_or(self.edges.len() as u64)); for (source, target, relation, weight) in &self.edges { sink.row(EdgeRow { - source: *source, - target: *target, + source: source.as_ref().map(&key), + target: target.as_ref().map(&key), relation: relation.as_deref(), weight: *weight, })?; @@ -83,4 +93,35 @@ pub(super) mod tests { Ok(()) } } + + impl SourceReader> for MemorySource { + fn node_ids( + &mut self, + memory: &BuildMemory, + ) -> Result, BuildError> { + let mut ids = warren_core::graph::BuildBuffer::new(memory); + for &id in &self.nodes { + ids.push(id)?; + } + Ok(ids) + } + + fn edges(&mut self, sink: &mut dyn EdgeSink) -> Result<(), BuildError> { + self.each_edge(sink, |id| id) + } + } + + impl SourceReader for MemorySource> { + fn node_ids(&mut self, memory: &BuildMemory) -> Result { + let mut ids = TextIds::new(memory); + for id in &self.nodes { + ids.push(id)?; + } + Ok(ids) + } + + fn edges(&mut self, sink: &mut dyn EdgeSink<[u8]>) -> Result<(), BuildError> { + self.each_edge(sink, Vec::as_slice) + } + } } diff --git a/crates/pg_warren/src/build/spi_source.rs b/crates/pg_warren/src/build/spi_source.rs index 215a08c2..10bccadf 100644 --- a/crates/pg_warren/src/build/spi_source.rs +++ b/crates/pg_warren/src/build/spi_source.rs @@ -2,16 +2,17 @@ //! read-only cursors under the build's pinned snapshot, as the graph owner //! (DESIGN.md §5.4 step 1, §7). +use std::borrow::Borrow; + use pgrx::prelude::*; -use warren_core::graph::{BuildBuffer, BuildMemory}; +use warren_core::graph::BuildMemory; use super::error::BuildError; use super::source::{EdgeRow, EdgeSink, SourceReader}; use crate::access::{as_role, with_read_only, with_search_path}; use crate::catalog::GraphDefinition; use crate::source::{ - check_edge_columns, for_each_row, for_each_tuple, open_read_only, require_node_id_type, - text_in_place, IdDatum, + check_edge_columns, for_each_tuple, open_read_only, require_node_id_type, IdDatum, }; /// A graph's sources, read through SPI. @@ -27,7 +28,7 @@ impl<'g> SpiSource<'g> { } impl SourceReader for SpiSource<'_> { - fn node_ids(&mut self, memory: &BuildMemory) -> Result, BuildError> { + fn node_ids(&mut self, memory: &BuildMemory) -> Result { let graph = self.graph; as_role(graph.owner, || { with_search_path(&graph.search_path, || { @@ -36,10 +37,10 @@ impl SourceReader for SpiSource<'_> { let mut opened = open_read_only::(client, "node_query", &graph.node_query)?; require_node_id_type(&opened.columns, graph.id_type)?; - let mut ids = BuildBuffer::new(memory); - for_each_row::(&mut opened.cursor, |row| { - if let Some(id) = row.get::(1)?.map(E::from_sql) { - ids.push(id)?; + let mut ids = E::node_ids(memory); + for_each_tuple::(&mut opened.cursor, |row| { + if let Some(id) = row.id::(1)? { + E::push_node_id(&mut ids, id.borrow())?; } Ok(()) })?; @@ -50,7 +51,7 @@ impl SourceReader for SpiSource<'_> { }) } - fn edges(&mut self, sink: &mut dyn EdgeSink) -> Result<(), BuildError> { + fn edges(&mut self, sink: &mut dyn EdgeSink) -> Result<(), BuildError> { let graph = self.graph; let weight_column = if graph.has_relations { 4 } else { 3 }; as_role(graph.owner, || { @@ -67,11 +68,13 @@ impl SourceReader for SpiSource<'_> { )?; sink.expect_rows(opened.estimated_rows); for_each_tuple(&mut opened.cursor, |row| { + let source = row.id::(1)?; + let target = row.id::(2)?; sink.row(EdgeRow { - source: row.get::(1)?.map(E::from_sql), - target: row.get::(2)?.map(E::from_sql), + source: source.as_ref().map(Borrow::borrow), + target: target.as_ref().map(Borrow::borrow), relation: match graph.has_relations { - true => text_in_place(row, 3)?, + true => row.text(3)?, false => None, }, weight: match graph.weighted { @@ -94,6 +97,8 @@ mod tests { use pgrx::prelude::*; use pgrx::spi::Error; + use warren_core::graph::FixedKeys; + use super::SpiSource; use crate::build::error::BuildError; use crate::build::source::{EdgeRow, EdgeSink, SourceReader}; @@ -147,7 +152,7 @@ mod tests { let graph = GraphDefinition::find("leak").map_err(|_| Error::InvalidPosition)?; let mut probe = Probe { before: allocated(), first_batch: 0, peak: 0, rows: 0 }; with_pinned_snapshot(|| { - SourceReader::::edges(&mut SpiSource::new(&graph), &mut probe) + SourceReader::>::edges(&mut SpiSource::new(&graph), &mut probe) }) .map_err(|_| Error::InvalidPosition)?; assert_eq!(probe.rows, 400_000); diff --git a/crates/pg_warren/src/cache/bound.rs b/crates/pg_warren/src/cache/bound.rs index 51a1da14..fc5a398d 100644 --- a/crates/pg_warren/src/cache/bound.rs +++ b/crates/pg_warren/src/cache/bound.rs @@ -347,6 +347,7 @@ pub(crate) fn admit_transient(replaced: u64, added: u64) -> Result<(), CacheErro mod tests { use pgrx::prelude::*; use pgrx::spi::Error; + use warren_core::graph::FixedKeys; use crate::cache::end_statement; use crate::fixture::{failure, refresh, ring, segment_reads}; @@ -557,8 +558,8 @@ mod tests { statements(&["SELECT warren.prewarm('a')", "SELECT warren.prewarm('b')"])?; let c = Spi::get_one::("SELECT current_version FROM warren.graphs WHERE name = 'c'")? .ok_or(Error::InvalidPosition)?; - let expected = - super::super::open::expected_bytes::(c).map_err(|_| Error::InvalidPosition)?; + let expected = super::super::open::expected_bytes::>(c) + .map_err(|_| Error::InvalidPosition)?; assert!(expected > 90_000, "{expected}"); // a, the least recently used, makes room for c. super::make_room(expected); @@ -694,7 +695,8 @@ mod tests { assert_eq!((evict("a")?, evict("b")?), (Some(2), Some(0)), "a cold cache"); let resolved = super::super::resolve("a", None, "read").map_err(|_| Error::InvalidPosition)?; - let a = super::super::cached::(&resolved).map_err(|_| Error::InvalidPosition)?; + let a = super::super::cached::>(&resolved) + .map_err(|_| Error::InvalidPosition)?; let sum = "SELECT warren.vector_reduce('a', 'x', 'sum')"; Spi::run(sum)?; // a in use and its vector idle: ~94 kB + 16 kB, ~45 kB past the cap. @@ -777,7 +779,8 @@ mod tests { rings(&["a", "b"])?; let resolved = super::super::resolve("a", None, "read").map_err(|_| Error::InvalidPosition)?; - let held = super::super::cached::(&resolved).map_err(|_| Error::InvalidPosition)?; + let held = super::super::cached::>(&resolved) + .map_err(|_| Error::InvalidPosition)?; assert_eq!(evict("a")?, Some(1)); Spi::run( "SET LOCAL warren.max_cache_memory = '64kB'; diff --git a/crates/pg_warren/src/cache/load/fill.rs b/crates/pg_warren/src/cache/load/fill.rs index 5caa7c9d..e4949028 100644 --- a/crates/pg_warren/src/cache/load/fill.rs +++ b/crates/pg_warren/src/cache/load/fill.rs @@ -4,18 +4,20 @@ use std::mem::size_of; -use warren_core::graph::{DenseId, GraphError, IdPair, SEGMENT_HEADER_LEN}; +use warren_core::graph::{ + check_id_bytes_header, DenseId, GraphError, IdPair, SegmentKind, SEGMENT_HEADER_LEN, +}; use super::plan::{Placed, Plan}; -use super::LoadError; +use super::{CachedIds, LoadError}; use crate::cache::region::{typed, typed_mut, write_pair, Array, Entry, Unpadded}; use crate::catalog::VersionId; use crate::segment::{SegmentError, SegmentStore}; -use crate::source::IdDatum; +use crate::source::InPlaceId; /// Writes the header and every array of `plan` into `region`, which holds /// exactly its layout's bytes. -pub(in crate::cache) fn fill( +pub(in crate::cache) fn fill( segments: &mut impl SegmentStore, version: VersionId, plan: &Plan, @@ -29,31 +31,20 @@ pub(in crate::cache) fn fill( for placed in &plan.placed { let Some(entry) = layout.entry(placed.array) else { continue }; let into = &mut region[entry.range()]; - let count = match placed.array { - Array::IdPairs if placed.encoded_width != size_of::>() || !NATIVE_LE => { - let mut pairs = PairSink::::new(into, placed.encoded_width); - copy_payload(segments, version, placed, &mut |data| pairs.push(data))? - } - _ => { - let mut at = 0; - copy_payload(segments, version, placed, &mut |data| { - into[at..at + data.len()].copy_from_slice(data); - at += data.len(); - Ok(()) - })? - } - }; + let mut sink = E::sink(placed, into); + let count = copy_payload(segments, version, placed, plan.encoding, &mut *sink)?; + drop(sink); if placed.array == Array::Tombstones && count != plan.id_space { let expected = plan.id_space; return Err(GraphError::TombstonesLength { expected, found: count }.into()); } to_native(placed, into)?; } - by_dense::(region, layout.entry(Array::IdPairs), layout.entry(Array::ByDense)) + E::finish(region, layout) } /// Whether native layout is little-endian, the segments' byte order. -const NATIVE_LE: bool = cfg!(target_endian = "little"); +pub(super) const NATIVE_LE: bool = cfg!(target_endian = "little"); /// Puts a copied array of scalars in native byte order: nothing on a /// little-endian target. @@ -66,20 +57,22 @@ fn to_native(placed: &Placed, bytes: &mut [u8]) -> Result<(), LoadError> { return Ok(()); } match (placed.array, placed.encoded_width) { - (Array::IdPairs, _) => Ok(()), + (Array::IdPairs, _) | (_, 1) => Ok(()), (_, 4) => swap::(bytes, u32::from_le), _ => swap::(bytes, u64::from_le), } } /// Streams the payload of `placed`'s segment to `into`, after checking -/// its chunks are numbered `0..n` and its header (kind, element width, and -/// a payload of the planned length); returns the header's count. What -/// `into` returns stops the read. +/// its chunks are numbered `0..n` and its header (kind, element width, a +/// payload of the planned length, and for `id_bytes` the database +/// encoding, `encoding`); returns the header's count. What `into` returns +/// stops the read. fn copy_payload( segments: &mut impl SegmentStore, version: VersionId, placed: &Placed, + encoding: u8, into: &mut dyn FnMut(&[u8]) -> Result<(), LoadError>, ) -> Result { let segment = placed.segment.as_str(); @@ -101,8 +94,11 @@ fn copy_payload( header[have..have + take].copy_from_slice(&data[..take]); (have, data) = (have + take, &data[take..]); if have == SEGMENT_HEADER_LEN { - let width = placed.encoded_width; - count = Some(placed.kind.check_header(&header, width, placed.payload)?); + let (width, payload) = (placed.encoded_width, placed.payload); + count = Some(match placed.kind { + SegmentKind::IdBytes => check_id_bytes_header(&header, payload, encoding)?, + kind => kind.check_header(&header, width, payload)?, + }); } } received += data.len() as u64; @@ -133,7 +129,7 @@ fn copy_payload( /// Re-lays out encoded id-map pairs into `IdPair`s as they stream in: /// an `int4` pair is 12 bytes encoded and 16 in memory, and on a /// big-endian target every pair is reordered. -struct PairSink<'a, E> { +pub(super) struct PairSink<'a, E> { slots: &'a mut [u8], encoded_width: usize, next: usize, @@ -141,14 +137,14 @@ struct PairSink<'a, E> { pair: std::marker::PhantomData, } -impl<'a, E: IdDatum> PairSink<'a, E> { - fn new(slots: &'a mut [u8], encoded_width: usize) -> Self { +impl<'a, E: InPlaceId> PairSink<'a, E> { + pub(super) fn new(slots: &'a mut [u8], encoded_width: usize) -> Self { let carry = Vec::with_capacity(encoded_width); Self { slots, encoded_width, next: 0, carry, pair: std::marker::PhantomData } } /// Takes the next bytes of the payload. - fn push(&mut self, mut data: &[u8]) -> Result<(), LoadError> { + pub(super) fn push(&mut self, mut data: &[u8]) -> Result<(), LoadError> { let width = self.encoded_width; if !self.carry.is_empty() { let take = data.len().min(width - self.carry.len()); @@ -188,7 +184,7 @@ impl<'a, E: IdDatum> PairSink<'a, E> { /// Builds the id map's second direction, each dense id's external id, from /// its pairs: an error if a dense id is outside the id space (a repeated /// one leaves another unset, which `IdMap::from_segments` refuses). -fn by_dense( +pub(super) fn by_dense( region: &mut [u8], pairs: Option<&Entry>, by_dense: Option<&Entry>, diff --git a/crates/pg_warren/src/cache/load/id_map.rs b/crates/pg_warren/src/cache/load/id_map.rs new file mode 100644 index 00000000..042352f5 --- /dev/null +++ b/crates/pg_warren/src/cache/load/id_map.rs @@ -0,0 +1,146 @@ +//! A version's id map in its region (DESIGN.md §5.6, §5.7): which segments +//! a load places, how each is laid out, what a load builds beside them, +//! and the id map viewed in place. A fixed-width id type's pairs are +//! re-laid out as they stream in and its second direction is built at +//! load; a text id map's three arrays are its segments as they are. + +use std::mem::size_of; +use std::sync::Arc; + +use warren_core::graph::{ + DenseId, FixedKeys, IdMap, IdOrder, IdPair, SegmentBuffer, SegmentKind, TargetWidth, TextKeys, + Tombstones, +}; + +use super::fill::{by_dense, PairSink, NATIVE_LE}; +use super::plan::{Placed, Planner}; +use super::LoadError; +use crate::cache::region::{Array, Region, RegionError, RegionLayout}; +use crate::segment::{SegmentStore, StoredIds, ID_BYTES, ID_MAP, ID_OFFSETS, ID_ORDER}; +use crate::source::{IdDatum, InPlaceId}; + +/// Takes the bytes of a segment's payload as they stream in. +pub(crate) type PayloadSink<'a> = Box Result<(), LoadError> + 'a>; + +/// A key storage a loaded version's id map can have. +pub(crate) trait CachedIds: IdDatum + StoredIds { + /// Places the id map's segments; returns the id space they imply. + fn place(planner: &mut Planner<'_, impl SegmentStore>) -> Result; + + /// The region's arrays for `placed`, as (array, element width, + /// elements): an id-map segment's own and what a load builds beside it + /// over an id space of `id_space` nodes. + fn arrays(placed: &Placed, elements: usize, _id_space: usize) -> Vec<(Array, usize, usize)> { + vec![(placed.array, placed.encoded_width, elements)] + } + + /// Where `placed`'s payload goes as it streams in: into `into`, as it + /// is, unless the id type lays it out otherwise. + fn sink<'a>(_placed: &Placed, into: &'a mut [u8]) -> PayloadSink<'a> { + copied(into) + } + + /// Builds in `region` what a load builds beside the segments, once + /// every array is copied. + fn finish(_region: &mut [u8], _layout: &RegionLayout) -> Result<(), LoadError> { + Ok(()) + } + + /// The id map `region` holds, with the tombstones' words `words`. + fn view(region: &Arc, words: SegmentBuffer) -> Result, LoadError>; + + /// The bytes a load adds to a version's stored segments over an id + /// space of `id_space` nodes. + fn built_at_load(id_space: u64) -> u64; +} + +/// A sink copying the payload into `into`, from its start. +pub(super) fn copied(into: &mut [u8]) -> PayloadSink<'_> { + let mut at = 0; + Box::new(move |data: &[u8]| { + let end = at + data.len(); + let found = into.len() as u64; + into.get_mut(at..end) + .ok_or(RegionError::Short { needed: end as u64, found })? + .copy_from_slice(data); + at = end; + Ok(()) + }) +} + +impl CachedIds for FixedKeys { + fn place(planner: &mut Planner<'_, impl SegmentStore>) -> Result { + let pair = E::WIDTH + size_of::(); + Ok(planner.require(Array::IdPairs, ID_MAP, SegmentKind::IdMap, pair)? / pair as u64) + } + + fn arrays(placed: &Placed, elements: usize, id_space: usize) -> Vec<(Array, usize, usize)> { + match placed.array { + Array::IdPairs => vec![ + (Array::IdPairs, size_of::>(), elements), + (Array::ByDense, size_of::(), id_space), + ], + _ => vec![(placed.array, placed.encoded_width, elements)], + } + } + + fn sink<'a>(placed: &Placed, into: &'a mut [u8]) -> PayloadSink<'a> { + let relaid = placed.encoded_width != size_of::>() || !NATIVE_LE; + match placed.array { + Array::IdPairs if relaid => { + let mut pairs = PairSink::::new(into, placed.encoded_width); + Box::new(move |data: &[u8]| pairs.push(data)) + } + _ => copied(into), + } + } + + fn finish(region: &mut [u8], layout: &RegionLayout) -> Result<(), LoadError> { + by_dense::(region, layout.entry(Array::IdPairs), layout.entry(Array::ByDense)) + } + + fn view(region: &Arc, words: SegmentBuffer) -> Result, LoadError> { + let missing = RegionError::MissingArray; + let pairs = region.view::>(Array::IdPairs)?.ok_or(missing("id map"))?; + let by_dense = region.view::(Array::ByDense)?.ok_or(missing("id map by dense id"))?; + let tombstones = Tombstones::from_segment(pairs.len() as u64, words)?; + Ok(IdMap::from_segments(pairs, by_dense, tombstones)?) + } + + fn built_at_load(id_space: u64) -> u64 { + id_space.saturating_mul(size_of::() as u64) + } +} + +impl CachedIds for TextKeys { + fn place(planner: &mut Planner<'_, impl SegmentStore>) -> Result { + let word = size_of::(); + let offsets = + planner.require(Array::IdOffsets, ID_OFFSETS, SegmentKind::IdOffsets, word)?; + let id_space = (offsets / word as u64).saturating_sub(1); + planner.require(Array::IdBytes, ID_BYTES, SegmentKind::IdBytes, 1)?; + let order = match TargetWidth::for_id_space(id_space) { + TargetWidth::U32 => size_of::(), + TargetWidth::U64 => size_of::(), + }; + planner.require(Array::IdOrder, ID_ORDER, SegmentKind::IdOrder, order)?; + Ok(id_space) + } + + fn view(region: &Arc, words: SegmentBuffer) -> Result, LoadError> { + let missing = RegionError::MissingArray; + let offsets = region.view::(Array::IdOffsets)?.ok_or(missing("id offsets"))?; + let bytes = region.view::(Array::IdBytes)?.ok_or(missing("id bytes"))?; + let id_space = offsets.len().saturating_sub(1) as u64; + let order = match TargetWidth::for_id_space(id_space) { + TargetWidth::U32 => IdOrder::U32(region.view(Array::IdOrder)?.ok_or(missing("order"))?), + TargetWidth::U64 => IdOrder::U64(region.view(Array::IdOrder)?.ok_or(missing("order"))?), + }; + let tombstones = Tombstones::from_segment(id_space, words)?; + Ok(IdMap::from_text_segments(bytes, offsets, order, tombstones)?) + } + + fn built_at_load(_: u64) -> u64 { + 0 + } +} diff --git a/crates/pg_warren/src/cache/load/mod.rs b/crates/pg_warren/src/cache/load/mod.rs index ff9b66b3..975c23ef 100644 --- a/crates/pg_warren/src/cache/load/mod.rs +++ b/crates/pg_warren/src/cache/load/mod.rs @@ -6,11 +6,13 @@ //! 38). mod fill; +mod id_map; mod plan; mod vector; mod views; pub(super) use fill::fill; +pub(crate) use id_map::CachedIds; pub(super) use plan::plan; pub(super) use vector::{fill_vector, plan_vector, read_vector}; @@ -19,8 +21,7 @@ use std::sync::{Arc, Weak}; use pgrx::PgSqlErrorCode; use thiserror::Error; use warren_core::graph::{ - BuildCharge, BuildMemory, ExternalId, FixedKeys, GraphCsr, GraphError, IdMap, - RelationDictionary, + BuildCharge, BuildMemory, GraphCsr, GraphError, IdKeys, IdMap, RelationDictionary, }; use warren_core::version::{GraphVersion, VersionError}; @@ -29,9 +30,8 @@ use crate::access::SqlError; use crate::catalog::VersionId; use crate::extension::graph_sqlstate; use crate::segment::{ - read_segment_within, SegmentError, SegmentStore, ID_MAP, RELATION_DICTIONARY, TOMBSTONES, + read_segment_within, SegmentError, SegmentStore, StoredIds, RELATION_DICTIONARY, TOMBSTONES, }; -use crate::source::IdDatum; use crate::vector::VectorError; /// Why a version could not be loaded. @@ -70,7 +70,7 @@ impl SqlError for LoadError { #[derive(Debug, Clone, PartialEq)] pub(crate) struct LoadedVersion { /// Its id map, tombstones included. - pub(crate) id_map: IdMap>, + pub(crate) id_map: IdMap, /// Its relation dictionary. pub(crate) relations: RelationDictionary, /// Its CSRs. A loaded graph reports no deduplicated edges: that count @@ -78,9 +78,9 @@ pub(crate) struct LoadedVersion { pub(crate) graph: GraphCsr, } -impl LoadedVersion { +impl LoadedVersion { /// The version as `warren-core`'s view of it. - pub(crate) fn version(&self) -> Result>, VersionError> { + pub(crate) fn version(&self) -> Result, VersionError> { GraphVersion::new(&self.id_map, &self.relations, &self.graph) } @@ -100,7 +100,7 @@ impl LoadedVersion { /// memory it holds. pub(crate) struct Identity { /// The id map. - pub(crate) id_map: IdMap>, + pub(crate) id_map: IdMap, /// The charge for the id map. pub(crate) id_map_charge: BuildCharge, /// The relation dictionary. @@ -112,15 +112,14 @@ pub(crate) struct Identity { /// The id map and relation dictionary of `version`, without its CSRs: what /// a build needs of the version it starts from. Their segments, their /// decoding and what they hold are charged to `memory`. -pub(crate) fn load_identity( +pub(crate) fn load_identity( segments: &mut impl SegmentStore, version: VersionId, memory: &BuildMemory, ) -> Result, LoadError> { let (tombstones, _tombstones_charge) = read_segment_within(segments, version, TOMBSTONES, memory)?; - let (id_map, _id_map_charge) = read_segment_within(segments, version, ID_MAP, memory)?; - let (id_map, id_map_charge) = IdMap::decode_within(&id_map, &tombstones, memory)?; + let (id_map, id_map_charge) = E::read_id_map(segments, version, &tombstones, memory)?; let (encoded, _encoded_charge) = read_segment_within(segments, version, RELATION_DICTIONARY, memory)?; // Decoded first, then charged: it holds about twice the encoding, @@ -137,7 +136,7 @@ pub(crate) fn load_identity( /// there is one CSR per relation, a merged CSR without relations, or with /// several if the graph stores it (`store_merged`), and reverse CSRs for /// all or none. -pub(crate) fn load( +pub(crate) fn load( segments: &mut impl SegmentStore, version: VersionId, store_merged: bool, @@ -151,7 +150,7 @@ pub(crate) fn load( /// The version `memory` holds, filled by [`fill`], read through views of /// its region and validated as [`load`] validates it, and the region, /// which lives while any view of it does. -pub(super) fn read( +pub(super) fn read( memory: Box, version: VersionId, ) -> Result<(LoadedVersion, Weak), LoadError> { @@ -164,9 +163,10 @@ pub(super) fn read( mod tests { use super::*; use crate::segment::{write_version, MemorySegmentStore}; - use crate::source::IdDatum; + use crate::source::InPlaceId; use warren_core::graph::{ AnyCsr, CsrLayout, EdgeIntake, FixedKeys, GraphCsr, IdMap, RelationCsr, RelationDictionary, + TextIds, TextKeys, }; /// A version built in memory: ids 10, 20, 30, 40 (or as given) with @@ -185,7 +185,7 @@ mod tests { built_with([10i64, 20, 30, 40], relations, weighted, layout) } - fn built_with( + fn built_with( ids: [E; 4], relations: &[&str], weighted: bool, @@ -208,11 +208,11 @@ mod tests { Built { id_map, relations: dictionary, graph } } - fn stored(version: &Built) -> MemorySegmentStore { + fn stored(version: &Built) -> MemorySegmentStore { stored_in_chunks(version, 32) } - fn stored_in_chunks(version: &Built, chunk_size: usize) -> MemorySegmentStore { + fn stored_in_chunks(version: &Built, chunk_size: usize) -> MemorySegmentStore { let mut store = MemorySegmentStore::default(); let (id_map, relations, graph) = (&version.id_map, &version.relations, &version.graph); write_version(&mut store, 7, id_map, relations, graph, chunk_size).unwrap(); @@ -225,7 +225,7 @@ mod tests { fn loads_as_written(version: &Built) { let store_merged = version.graph.merged().is_some() || version.relations.len() < 2; - let loaded = load::(&mut stored(version), 7, store_merged).unwrap(); + let loaded = load::>(&mut stored(version), 7, store_merged).unwrap(); assert_eq!(loaded.id_map, version.id_map); assert_eq!(loaded.relations, version.relations); assert_eq!(loaded.graph, version.graph); @@ -250,9 +250,9 @@ mod tests { loads_as_written(&version); // What the cache holds, and so counts against its cap, is the // stored version's bytes less its merged CSR's. - let loaded = load::(&mut stored(&version), 7, false).unwrap(); + let loaded = load::>(&mut stored(&version), 7, false).unwrap(); let with_merged = built(&["cites", "extends", "funds"], true, build_reverse); - let with_merged = load::(&mut stored(&with_merged), 7, true).unwrap(); + let with_merged = load::>(&mut stored(&with_merged), 7, true).unwrap(); let merged = with_merged.graph.merged().map_or(0, |merged| merged.bytes()); assert!(merged > 0); assert_eq!(loaded.bytes() + merged, with_merged.bytes()); @@ -276,7 +276,7 @@ mod tests { store.chunks.retain(|(_, name, _), _| !name.contains(':')); let memory = BuildMemory::unlimited(); let Identity { id_map, id_map_charge, relations, relations_charge } = - load_identity::(&mut store, 7, &memory).unwrap(); + load_identity::>(&mut store, 7, &memory).unwrap(); assert_eq!(id_map, version.id_map); assert_eq!(relations, version.relations); assert_eq!(id_map_charge.bytes(), id_map.bytes()); @@ -286,7 +286,7 @@ mod tests { assert!(memory.peak() > 2 * id_map.bytes(), "{}", memory.peak()); let tight = BuildMemory::new(id_map.bytes()); assert!(matches!( - load_identity::(&mut store, 7, &tight), + load_identity::>(&mut store, 7, &tight), Err(LoadError::Graph(GraphError::BuildMemoryLimit { .. })) | Err(LoadError::Segment(SegmentError::Graph(GraphError::BuildMemoryLimit { .. }))) )); @@ -298,7 +298,7 @@ mod tests { let mut store = stored(&version); store.chunks.retain(|(_, name, _), _| name != "targets:0"); assert!(matches!( - load::(&mut store, 7, true), + load::>(&mut store, 7, true), Err(LoadError::Segment(SegmentError::SegmentMissing { version: 7, .. })) )); } @@ -308,10 +308,10 @@ mod tests { let mut store = stored(&built(&["a", "b"], false, true)); store.chunks.retain(|(_, name, _), _| !name.ends_with(":merged")); assert!(matches!( - load::(&mut store, 7, true), + load::>(&mut store, 7, true), Err(LoadError::Segment(SegmentError::SegmentMissing { version: 7, .. })) )); - assert!(load::(&mut store, 7, false).is_ok()); + assert!(load::>(&mut store, 7, false).is_ok()); } #[test] @@ -321,17 +321,17 @@ mod tests { let mut other_store = stored(&other); let rel_dict = other_store.chunks.remove(&(7, "rel_dict".to_owned(), 0)).unwrap(); store.chunks.insert((7, "rel_dict".to_owned(), 0), rel_dict); - assert!(load::(&mut store, 7, true).is_err()); + assert!(load::>(&mut store, 7, true).is_err()); } #[test] fn ids_of_every_type_load_as_written_whatever_the_chunk_size() { - fn check(ids: [E; 4]) { + fn check(ids: [E; 4]) { let layout = CsrLayout { build_reverse: true, store_merged: true }; let version = built_with(ids, &["a", "b"], true, layout); for chunk_size in [1, 5, 7, 12, 32, 4096] { let mut store = stored_in_chunks(&version, chunk_size); - let loaded = load::(&mut store, 7, true).unwrap(); + let loaded = load::>(&mut store, 7, true).unwrap(); assert_eq!(loaded.id_map, version.id_map, "chunks of {chunk_size}"); assert_eq!(loaded.graph, version.graph); } @@ -345,7 +345,7 @@ mod tests { fn every_array_of_a_loaded_version_is_aligned_in_one_region() { let layout = CsrLayout { build_reverse: true, store_merged: true }; let version = built_with([10i32, 20, 30, 40], &["a", "b", "c"], true, layout); - let loaded = load::(&mut stored(&version), 7, true).unwrap(); + let loaded = load::>(&mut stored(&version), 7, true).unwrap(); let mut starts = Vec::new(); let mut csrs: Vec<&RelationCsr> = loaded.graph.relations().iter().collect(); csrs.extend(loaded.graph.merged()); @@ -371,7 +371,7 @@ mod tests { let first = store.chunks.get_mut(&(7, "offsets:0".to_owned(), 0)).unwrap(); first[8] += 1; assert!(matches!( - load::(&mut store, 7, true), + load::>(&mut store, 7, true), Err(LoadError::Segment(SegmentError::Graph(GraphError::SegmentLength { .. }))) | Err(LoadError::Graph(GraphError::SegmentLength { .. })) )); @@ -383,7 +383,7 @@ mod tests { let targets = store.chunks.get(&(7, "targets:0".to_owned(), 0)).unwrap().clone(); store.chunks.insert((7, "offsets:0".to_owned(), 0), targets); assert!(matches!( - load::(&mut store, 7, true), + load::>(&mut store, 7, true), Err(LoadError::Segment(SegmentError::Graph(GraphError::SegmentKind { .. }))) | Err(LoadError::Graph(GraphError::SegmentKind { .. })) )); @@ -397,7 +397,7 @@ mod tests { // The first pair's dense id. id_map[16 + 8..16 + 16].copy_from_slice(&99u64.to_le_bytes()); assert!(matches!( - load::(&mut store, 7, true), + load::>(&mut store, 7, true), Err(LoadError::Graph(GraphError::IdMapNotPermutation)) )); } @@ -409,11 +409,103 @@ mod tests { let tombstones = store.chunks.get_mut(&(7, "tombstones".to_owned(), 0)).unwrap(); tombstones[8..16].copy_from_slice(&3u64.to_le_bytes()); assert!(matches!( - load::(&mut store, 7, true), + load::>(&mut store, 7, true), Err(LoadError::Segment(SegmentError::Graph(GraphError::TombstonesLength { expected: 4, found: 3 }))) | Err(LoadError::Graph(GraphError::TombstonesLength { expected: 4, found: 3 })) )); } + + /// A text version: ids `""`, `a`, `b` (tombstoned) and a long one, + /// with edges between the live ones, its segments in `chunk_size` + /// chunks. + fn text_version(chunk_size: usize) -> (IdMap, GraphCsr, MemorySegmentStore) { + let long = vec![b'x'; 300]; + let ids: [&[u8]; 4] = [b"", b"a", b"b", &long]; + let memory = BuildMemory::unlimited(); + let text = |ids: &[&[u8]]| { + let mut text = TextIds::new(&memory); + ids.iter().for_each(|id| text.push(id).unwrap()); + text + }; + let (first, _) = IdMap::new().rebuild_within(text(&ids)).unwrap(); + let (id_map, _) = first.rebuild_within(text(&[ids[0], ids[1], ids[3]])).unwrap(); + let mut dictionary = RelationDictionary::new(); + let relation = dictionary.intern("r").unwrap(); + let mut intake = EdgeIntake::directed(&id_map, false); + intake.push(relation, Some(ids[0]), Some(ids[3]), 1.0).unwrap(); + intake.push(relation, Some(ids[3]), Some(ids[1]), 1.0).unwrap(); + intake.push(relation, Some(ids[2]), Some(ids[1]), 1.0).unwrap(); + let (edges, dropped) = intake.finish(); + assert_eq!(dropped, 1); + let layout = CsrLayout { build_reverse: true, store_merged: true }; + let graph = GraphCsr::build_as(edges, 1, id_map.id_space(), layout).unwrap(); + let mut store = MemorySegmentStore::default(); + write_version(&mut store, 7, &id_map, &dictionary, &graph, chunk_size).unwrap(); + (id_map, graph, store) + } + + #[test] + fn a_text_version_loads_as_written_whatever_the_chunk_size() { + for chunk_size in [1, 5, 7, 12, 32, 4096] { + let (id_map, graph, mut store) = text_version(chunk_size); + let loaded = load::(&mut store, 7, true).unwrap(); + assert_eq!(loaded.id_map, id_map, "chunks of {chunk_size}"); + assert_eq!(loaded.graph, graph); + assert_eq!(loaded.id_map.dense_id(&[b'x'; 300]), Some(3)); + assert_eq!(loaded.id_map.dense_id(b"b"), None); + assert!(!store.chunks.keys().any(|(_, name, _)| name == "id_map")); + } + } + + #[test] + fn a_text_version_holds_its_segments_alone() { + let (id_map, graph, mut store) = text_version(4096); + let loaded = load::(&mut store, 7, true).unwrap(); + assert_eq!(loaded.id_map.bytes(), id_map.bytes()); + assert_eq!(::built_at_load(id_map.id_space()), 0); + assert!(loaded.bytes() >= id_map.bytes() + graph.bytes()); + } + + #[test] + fn a_text_version_of_another_encoding_is_refused() { + let (_, _, mut store) = text_version(4096); + store.encoding = 8; + assert!(matches!( + load::(&mut store, 7, true), + Err(LoadError::Segment(SegmentError::Graph(GraphError::IdEncoding { + expected: 8, + found: 6 + }))) + )); + let memory = BuildMemory::unlimited(); + assert!(matches!( + load_identity::(&mut store, 7, &memory), + Err(LoadError::Segment(SegmentError::Graph(GraphError::IdEncoding { .. }))) + )); + } + + #[test] + fn a_text_version_whose_order_is_not_sorted_is_refused() { + let (_, _, mut store) = text_version(4096); + let order = store.chunks.get_mut(&(7, "id_order".to_owned(), 0)).unwrap(); + // The first two dense ids in byte order, swapped. + let (first, second) = (order[16..20].to_vec(), order[20..24].to_vec()); + order[16..20].copy_from_slice(&second); + order[20..24].copy_from_slice(&first); + assert!(matches!( + load::(&mut store, 7, true), + Err(LoadError::Graph(GraphError::IdMapNotSorted)) + )); + } + + #[test] + fn a_text_version_whose_offsets_run_past_its_bytes_is_refused() { + let (_, _, mut store) = text_version(4096); + let offsets = store.chunks.get_mut(&(7, "id_offsets".to_owned(), 0)).unwrap(); + let last = offsets.len() - 8; + offsets[last..].copy_from_slice(&999u64.to_le_bytes()); + assert!(load::(&mut store, 7, true).is_err()); + } } diff --git a/crates/pg_warren/src/cache/load/plan.rs b/crates/pg_warren/src/cache/load/plan.rs index ba28c464..2e4047f2 100644 --- a/crates/pg_warren/src/cache/load/plan.rs +++ b/crates/pg_warren/src/cache/load/plan.rs @@ -5,20 +5,19 @@ use std::mem::size_of; use warren_core::graph::{ - DenseId, GraphError, IdPair, RelationDictionary, SegmentKind, TargetWidth, SEGMENT_HEADER_LEN, + GraphError, RelationDictionary, SegmentKind, TargetWidth, SEGMENT_HEADER_LEN, }; -use super::LoadError; +use super::{CachedIds, LoadError}; use crate::cache::region::{Array, RegionLayout}; use crate::catalog::VersionId; use crate::segment::{ - csr_segment, read_segment, CsrPart, CsrScope, Direction, SegmentError, SegmentStore, ID_MAP, + csr_segment, read_segment, CsrPart, CsrScope, Direction, SegmentError, SegmentStore, RELATION_DICTIONARY, TOMBSTONES, }; -use crate::source::IdDatum; /// A segment whose payload is copied into an array of the region. -pub(super) struct Placed { +pub(crate) struct Placed { /// The array it fills. pub(super) array: Array, /// The segment's name. @@ -41,6 +40,8 @@ pub(in crate::cache) struct Plan { pub(super) relations: Vec, /// The id space. pub(super) id_space: u64, + /// The database encoding, which an `id_bytes` header must record. + pub(super) encoding: u8, } impl Plan { @@ -50,21 +51,22 @@ impl Plan { } } -/// The plan of `version`'s region: the id map's pairs and second -/// direction, the tombstones and relation dictionary, then each +/// The plan of `version`'s region: the id map's arrays (a fixed type's +/// pairs and second direction, or a text id map's three), the tombstones +/// and relation dictionary, then each /// relation's CSRs and the merged CSR's, if stored (it must be where the /// graph stores it, `store_merged`, or has no relations), each forward /// and, if stored, reverse. -pub(in crate::cache) fn plan( +pub(in crate::cache) fn plan( segments: &mut impl SegmentStore, version: VersionId, store_merged: bool, ) -> Result { let relations = read_segment(segments, version, RELATION_DICTIONARY)?; let relation_count = RelationDictionary::decode(&relations)?.len(); + let encoding = segments.encoding(); let mut planner = Planner { segments, version, placed: Vec::new() }; - let pair = E::WIDTH + size_of::(); - let id_space = planner.require(Array::IdPairs, ID_MAP, SegmentKind::IdMap, pair)? / pair as u64; + let id_space = E::place(&mut planner)?; planner.require(Array::Tombstones, TOMBSTONES, SegmentKind::Tombstones, size_of::())?; let target = match TargetWidth::for_id_space(id_space) { TargetWidth::U32 => size_of::(), @@ -85,19 +87,12 @@ pub(in crate::cache) fn plan( let id_space_elements = usize::try_from(id_space).map_err(|_| GraphError::IdSpaceTooLarge)?; let mut arrays = Vec::with_capacity(placed.len() + 2); for placed in &placed { - let width = match placed.array { - Array::IdPairs => size_of::>(), - _ => placed.encoded_width, - }; let elements = elements(placed.payload, placed.encoded_width); - arrays.push((placed.array, width, elements)); - if placed.array == Array::IdPairs { - arrays.push((Array::ByDense, size_of::(), id_space_elements)); - } + arrays.extend(E::arrays(placed, elements, id_space_elements)); } arrays.push((Array::RelationDictionary, 1, relations.len())); let layout = RegionLayout::plan(version, &arrays)?; - Ok(Plan { layout, placed, relations, id_space }) + Ok(Plan { layout, placed, relations, id_space, encoding }) } /// The whole elements of `width` bytes in `payload` bytes: a payload that @@ -107,7 +102,7 @@ fn elements(payload: u64, width: usize) -> usize { } /// Finds the segments of a version and their sizes. -struct Planner<'s, S> { +pub(crate) struct Planner<'s, S> { segments: &'s mut S, version: VersionId, placed: Vec, @@ -115,7 +110,7 @@ struct Planner<'s, S> { impl Planner<'_, S> { /// Places segment `name`, which must exist; returns its payload bytes. - fn require( + pub(super) fn require( &mut self, array: Array, name: &str, diff --git a/crates/pg_warren/src/cache/load/views.rs b/crates/pg_warren/src/cache/load/views.rs index 18a7483d..20576520 100644 --- a/crates/pg_warren/src/cache/load/views.rs +++ b/crates/pg_warren/src/cache/load/views.rs @@ -3,24 +3,19 @@ use std::sync::Arc; -use warren_core::graph::{ - AnyCsr, Csr, GraphCsr, IdMap, IdPair, RelationCsr, RelationDictionary, Tombstones, -}; +use warren_core::graph::{AnyCsr, Csr, GraphCsr, RelationCsr, RelationDictionary}; -use super::{LoadError, LoadedVersion}; +use super::{CachedIds, LoadError, LoadedVersion}; use crate::cache::region::{Array, Region, RegionError}; use crate::segment::{CsrPart, CsrScope, Direction}; -use crate::source::IdDatum; /// The version `region` holds, checked as `load` checks a version: each /// array's invariants, then that they agree. -pub(super) fn loaded(region: &Arc) -> Result, LoadError> { +pub(super) fn loaded(region: &Arc) -> Result, LoadError> { let missing = RegionError::MissingArray; - let pairs = region.view::>(Array::IdPairs)?.ok_or(missing("id map"))?; - let by_dense = region.view::(Array::ByDense)?.ok_or(missing("id map by dense id"))?; let words = region.view::(Array::Tombstones)?.ok_or(missing("tombstones"))?; - let id_space = pairs.len() as u64; - let id_map = IdMap::from_segments(pairs, by_dense, Tombstones::from_segment(id_space, words)?)?; + let id_map = E::view(region, words)?; + let id_space = id_map.id_space(); let encoded = region.array_bytes(Array::RelationDictionary); let relations = RelationDictionary::decode(encoded.ok_or(missing("relation dictionary"))?)?; let mut csrs = Vec::with_capacity(relations.len()); diff --git a/crates/pg_warren/src/cache/mod.rs b/crates/pg_warren/src/cache/mod.rs index 70b9e9ef..ccc2cef6 100644 --- a/crates/pg_warren/src/cache/mod.rs +++ b/crates/pg_warren/src/cache/mod.rs @@ -32,7 +32,7 @@ mod warm; pub(crate) use bound::{admit_transient, compute_memory, define_memory_settings, trim}; pub(crate) use error::CacheError; -pub(crate) use load::{load, load_identity, Identity, LoadError, LoadedVersion}; +pub(crate) use load::{load, load_identity, CachedIds, Identity, LoadError, LoadedVersion}; pub(crate) use open::{ cached, cached_pair, cached_vector, cached_vector_hit, forget_vector, keep_vector, resolve, Resolved, diff --git a/crates/pg_warren/src/cache/open.rs b/crates/pg_warren/src/cache/open.rs index 0be11793..2f6af0ac 100644 --- a/crates/pg_warren/src/cache/open.rs +++ b/crates/pg_warren/src/cache/open.rs @@ -3,7 +3,6 @@ //! §5.5, §7). Every query function opens its version this way on every //! call. -use std::mem::size_of; use std::sync::Arc; use pgrx::pg_sys; @@ -15,13 +14,12 @@ use super::bound::{admit_load, make_room, unfit, unfit_notice}; use super::shared::{evict_vector, prune as prune_shared, shared, shared_hit}; use super::statement::{notice_due, pin_vector, pin_version, version_pinned}; use super::versions::{VersionCache, CACHE}; -use super::{installation, load, trim, CacheError, LoadedVersion}; +use super::{installation, load, trim, CacheError, CachedIds, LoadedVersion}; use crate::access::{as_catalog_owner, require_reader}; use crate::catalog::{require_version, GraphDefinition, VersionId}; use crate::parallel_worker::cache_notice_raised; use crate::read_only::{read_one, read_two}; use crate::segment::SpiSegmentStore; -use crate::source::IdDatum; use crate::vector::{ self, cached_vectors, forget, lookup_vector, remember, retain_transients, transient_versions, SpiVectorStore, VectorError, VectorId, VectorRecord, @@ -74,7 +72,7 @@ pub(crate) fn resolve( /// by trimming the cache to `warren.max_cache_memory`, evicting only what /// the statement has not read. A hit trims it too, so a lowered cap takes /// effect at the next lookup. -pub(crate) fn cached( +pub(crate) fn cached( resolved: &Resolved, ) -> Result>, CacheError> { if let Some(loaded) = CACHE.with_borrow_mut(|cache| cache.get::(resolved.version)) { @@ -90,9 +88,7 @@ pub(crate) fn cached( /// The version `resolved` names, loaded into this backend's cache, as /// [`cached`] describes. -fn load_private( - resolved: &Resolved, -) -> Result>, CacheError> { +fn load_private(resolved: &Resolved) -> Result>, CacheError> { let (graph_id, version) = (resolved.graph.graph_id, resolved.version); prune_dropped()?; let expected = expected_bytes::(version)?; @@ -125,7 +121,7 @@ fn load_private( /// loaded into the shared cache if it admits them, and the rest are /// refused together, before either is loaded into this backend's cache, /// if they would not fit beside what it holds. -pub(crate) fn cached_pair( +pub(crate) fn cached_pair( first: &Resolved, second: &Resolved, ) -> Result<[Arc>; 2], CacheError> { @@ -167,9 +163,7 @@ pub(crate) fn cached_pair( /// The loaded version `resolved` names if this backend's cache or the /// shared cache holds it, pinned until the statement ends; nothing is /// loaded. -fn hit( - resolved: &Resolved, -) -> Result>>, CacheError> { +fn hit(resolved: &Resolved) -> Result>>, CacheError> { if let Some(loaded) = CACHE.with_borrow_mut(|cache| cache.get::(resolved.version)) { pin_version(resolved.version); return Ok(Some(loaded)); @@ -264,9 +258,10 @@ pub(crate) fn evict_graph(graph_id: i32) -> usize { } /// About the bytes loading `version` adds to the cache: its stored -/// segments (`warren.versions.bytes`) and the id map's second direction, -/// which is built at load (an external id per dense id). -pub(super) fn expected_bytes(version: VersionId) -> Result { +/// segments (`warren.versions.bytes`) and what a load builds beside them: +/// a fixed type's second direction (an external id per dense id); nothing +/// for a text id map, which stores both directions (DESIGN.md §5.7). +pub(super) fn expected_bytes(version: VersionId) -> Result { let (stored, id_space) = as_catalog_owner(|| { Ok::<_, CacheError>(read_two::( "SELECT bytes, id_space FROM warren.versions WHERE version_id = $1", @@ -274,8 +269,8 @@ pub(super) fn expected_bytes(version: VersionId) -> Result| value.and_then(|value| u64::try_from(value).ok()); - let second_direction = count(id_space).unwrap_or(0).saturating_mul(size_of::() as u64); - Ok(count(stored).unwrap_or(0).saturating_add(second_direction)) + let built = E::built_at_load(count(id_space).unwrap_or(0)); + Ok(count(stored).unwrap_or(0).saturating_add(built)) } /// Which of `versions` still exist. @@ -301,6 +296,7 @@ fn existing_versions(versions: &[VersionId]) -> Result, CacheErro mod tests { use pgrx::prelude::*; use pgrx::spi::Error; + use warren_core::graph::FixedKeys; use std::sync::Arc; @@ -310,9 +306,9 @@ mod tests { }; /// Graph `name`'s current version, through the cache. - fn open(name: &str) -> Result>, Error> { + fn open(name: &str) -> Result>>, Error> { let resolved = resolve(name, None, "read").map_err(|_| Error::InvalidPosition)?; - cached::(&resolved).map_err(|_| Error::InvalidPosition) + cached::>(&resolved).map_err(|_| Error::InvalidPosition) } /// A version a call holds, evicted by the cap, is shared by later diff --git a/crates/pg_warren/src/cache/region/in_place.rs b/crates/pg_warren/src/cache/region/in_place.rs index a0cf28ac..6e96e3d2 100644 --- a/crates/pg_warren/src/cache/region/in_place.rs +++ b/crates/pg_warren/src/cache/region/in_place.rs @@ -5,7 +5,7 @@ use std::mem::{align_of, offset_of, size_of}; use warren_core::graph::IdPair; use super::RegionError; -use crate::source::IdDatum; +use crate::source::InPlaceId; /// An element a region holds in place. /// @@ -23,26 +23,30 @@ pub(crate) unsafe trait InPlace: Copy + Send + Sync + 'static {} pub(crate) unsafe trait Unpadded: InPlace {} // SAFETY: plain integers and floats: any bits are a value; alignment ≤ 8. +unsafe impl InPlace for u8 {} +// SAFETY: as for `u8`. unsafe impl InPlace for u32 {} -// SAFETY: as for `u32`. +// SAFETY: as for `u8`. unsafe impl InPlace for u64 {} -// SAFETY: as for `u32`. +// SAFETY: as for `u8`. unsafe impl InPlace for f32 {} -// SAFETY: as for `u32`. +// SAFETY: as for `u8`. unsafe impl InPlace for f64 {} -// SAFETY: `IdDatum`'s contract: any bits are an external id, aligned ≤ 8. -unsafe impl InPlace for E {} +// SAFETY: `InPlaceId`'s contract: any bits are an external id, aligned ≤ 8. +unsafe impl InPlace for E {} // SAFETY: a `#[repr(C)]` pair of an external id and a `u64`, both // `InPlace`; its padding, if any, is not part of its value. -unsafe impl InPlace for IdPair {} +unsafe impl InPlace for IdPair {} // SAFETY: integers and floats have no padding. +unsafe impl Unpadded for u8 {} +// SAFETY: as for `u8`. unsafe impl Unpadded for u32 {} // SAFETY: as for `u32`. unsafe impl Unpadded for u64 {} // SAFETY: as for `u32`. unsafe impl Unpadded for f64 {} -// SAFETY: `IdDatum`'s contract: an external id has no padding. -unsafe impl Unpadded for E {} +// SAFETY: `InPlaceId`'s contract: an external id has no padding. +unsafe impl Unpadded for E {} /// `bytes` as elements of `T`, which must be aligned for `T` and a whole /// number of them. @@ -66,7 +70,10 @@ pub(crate) fn typed_mut(bytes: &mut [u8]) -> Result<&mut [T], Regio /// Writes `pair` into `slot`, the bytes of one `IdPair`, field by /// field, so its padding stays initialized. -pub(crate) fn write_pair(slot: &mut [u8], pair: IdPair) -> Result<(), RegionError> { +pub(crate) fn write_pair( + slot: &mut [u8], + pair: IdPair, +) -> Result<(), RegionError> { let (start, dense) = (offset_of!(IdPair, external), offset_of!(IdPair, dense)); let end = slot.len(); if end != size_of::>() { diff --git a/crates/pg_warren/src/cache/region/layout.rs b/crates/pg_warren/src/cache/region/layout.rs index f7484dcb..c0409e77 100644 --- a/crates/pg_warren/src/cache/region/layout.rs +++ b/crates/pg_warren/src/cache/region/layout.rs @@ -43,6 +43,14 @@ pub(crate) enum Array { ByDense, /// The tombstones' words. Tombstones, + /// A text id map's ids, concatenated in dense-id order (DESIGN.md + /// §5.7). + IdBytes, + /// Where each dense id's id starts in `IdBytes`, and where the last + /// ends. + IdOffsets, + /// The dense ids sorted by text id. + IdOrder, /// The relation dictionary, as its encoded segment. RelationDictionary, /// One part of one CSR. @@ -244,6 +252,9 @@ fn encode_array(array: Array) -> ArrayKey { (5, direction, part, scope) } Array::Values => (6, 0, 0, 0), + Array::IdBytes => (7, 0, 0, 0), + Array::IdOffsets => (8, 0, 0, 0), + Array::IdOrder => (9, 0, 0, 0), } } @@ -255,6 +266,9 @@ fn decode_array(kind: u8, direction: u8, part: u8, scope: u64) -> Option (3, 0, 0, 0) => Some(Array::Tombstones), (4, 0, 0, 0) => Some(Array::RelationDictionary), (6, 0, 0, 0) => Some(Array::Values), + (7, 0, 0, 0) => Some(Array::IdBytes), + (8, 0, 0, 0) => Some(Array::IdOffsets), + (9, 0, 0, 0) => Some(Array::IdOrder), (5, ..) => { let direction = match direction { 0 => Direction::Forward, diff --git a/crates/pg_warren/src/cache/shared/dsm/mod.rs b/crates/pg_warren/src/cache/shared/dsm/mod.rs index 805a4740..10dc4da0 100644 --- a/crates/pg_warren/src/cache/shared/dsm/mod.rs +++ b/crates/pg_warren/src/cache/shared/dsm/mod.rs @@ -30,10 +30,9 @@ use super::session::{Mapped, Session, Unattached, Use}; use super::slot::{Handle, Key, Process, Slot, State, Ticket}; use super::unavailable::Unavailable; use super::{setting, Claimed, Found, SharedVersions}; -use crate::cache::load::{read, read_vector, LoadError, LoadedVersion}; +use crate::cache::load::{read, read_vector, CachedIds, LoadError, LoadedVersion}; use crate::cache::region::Region; use crate::catalog::VersionId; -use crate::source::IdDatum; use crate::vector::VectorRecord; use placement::{Located, Shared, SLOTS}; use segment::{Segment, SegmentRegion}; @@ -427,7 +426,7 @@ impl SharedVersions for Dsm { /// (`dsm_backend_shutdown`), so nothing here unmaps memory a call might /// still hold. A FATAL can exit while compute threads still read a /// mapping (from the main thread's poll, without unwinding); the - /// compute pool's own exit callback (`compute_sql::settings`) stops + /// compute pool's own exit callback (`interrupts::pool`) stops /// them first, so none reads after that detach. fn exit(&mut self) { with_backend(|backend| { @@ -528,7 +527,7 @@ impl Attached { /// `E`: built over the segment's region (and validated) the first time, /// then kept with the mapping, which it keeps in use while a call holds /// it. - pub(crate) fn version( + pub(crate) fn version( &self, version: VersionId, ) -> Result>, LoadError> { @@ -585,10 +584,7 @@ impl Loading { /// Reads the filled region as version `version`, validating it once, /// and keeps what it read for the published mapping. - pub(crate) fn validate( - &mut self, - version: VersionId, - ) -> Result<(), LoadError> { + pub(crate) fn validate(&mut self, version: VersionId) -> Result<(), LoadError> { let (loaded, region) = read::(Box::new(self.region), version)?; let loaded: Arc> = Arc::new(loaded); self.held = Held { loaded: Some(loaded), region }; diff --git a/crates/pg_warren/src/cache/shared/dsm/tests.rs b/crates/pg_warren/src/cache/shared/dsm/tests.rs index 7f512eae..ed116ac5 100644 --- a/crates/pg_warren/src/cache/shared/dsm/tests.rs +++ b/crates/pg_warren/src/cache/shared/dsm/tests.rs @@ -114,6 +114,7 @@ mod tests { use crate::segment::SpiSegmentStore; use crate::vector::{read_vector, SpiVectorStore, VectorRecord, VectorStore}; use warren_core::compute::AnyVector; + use warren_core::graph::FixedKeys; #[pg_test] fn the_shared_cache_is_on_by_default() -> Result<(), Error> { @@ -131,7 +132,7 @@ mod tests { let setting = Spi::get_one::("SHOW warren.shared_cache_memory")?; assert_eq!(setting.as_deref(), Some("0")); let (key, resolved) = graph("off", 50)?; - assert!(shared::(&resolved).map_err(|_| Error::InvalidPosition)?.is_none()); + assert!(shared::>(&resolved).map_err(|_| Error::InvalidPosition)?.is_none()); assert_eq!(slot(key), None); Ok(()) } @@ -140,7 +141,7 @@ mod tests { fn an_uncommitted_version_is_loaded_privately() -> Result<(), Error> { set_cap(c"64MB")?; let (key, resolved) = graph("uncommitted", 50)?; - assert!(shared::(&resolved).map_err(|_| Error::InvalidPosition)?.is_none()); + assert!(shared::>(&resolved).map_err(|_| Error::InvalidPosition)?.is_none()); Spi::run("SELECT warren.prewarm('uncommitted')")?; assert_eq!(slot(key), None); let evicted = Spi::get_one::("SELECT warren.evict('uncommitted')")?; @@ -152,12 +153,16 @@ mod tests { fn a_version_is_loaded_once_and_shared_by_later_statements() -> Result<(), Error> { set_cap(c"64MB")?; let (key, resolved) = graph("shared_once", 2000)?; - let read = - || through::(port()?, &resolved).ok().flatten().ok_or(Error::InvalidPosition); + let read = || { + through::>(port()?, &resolved) + .ok() + .flatten() + .ok_or(Error::InvalidPosition) + }; let first = read()?; assert_eq!(slot(key), Some((State::Ready, 1, 1)), "pinned and mapped by its loader"); - let private = - load::(&mut SpiSegmentStore, key.id, true).map_err(|_| Error::InvalidPosition)?; + let private = load::>(&mut SpiSegmentStore, key.id, true) + .map_err(|_| Error::InvalidPosition)?; assert_eq!((&first.id_map, &first.graph), (&private.id_map, &private.graph)); end_statement(); assert_eq!(slot(key), Some((State::Ready, 0, 1)), "mapped across statements"); @@ -174,8 +179,12 @@ mod tests { fn a_version_read_again_in_its_statement_skips_the_registry() -> Result<(), Error> { set_cap(c"64MB")?; let (key, resolved) = graph("fast", 100)?; - let read = - || through::(port()?, &resolved).ok().flatten().ok_or(Error::InvalidPosition); + let read = || { + through::>(port()?, &resolved) + .ok() + .flatten() + .ok_or(Error::InvalidPosition) + }; let first = read()?; let clocks = with_backend(|backend| backend.locked(|registry, _| registry.clocks())); let again = read()?; @@ -190,7 +199,8 @@ mod tests { fn a_version_larger_than_the_cap_is_loaded_privately() -> Result<(), Error> { set_cap(c"64kB")?; let (key, resolved) = graph("too_large", 20_000)?; - let read = through::(port()?, &resolved).map_err(|_| Error::InvalidPosition)?; + let read = + through::>(port()?, &resolved).map_err(|_| Error::InvalidPosition)?; assert!(read.is_none()); assert_eq!(slot(key), None, "a refusal changes nothing"); Ok(()) @@ -211,7 +221,8 @@ mod tests { }); assert!(matches!(claimed, Some(Claim::Load { .. }))); assert_eq!(slot(key), Some((State::Loading, 0, 0))); - let read = through::(port()?, &resolved).map_err(|_| Error::InvalidPosition)?; + let read = + through::>(port()?, &resolved).map_err(|_| Error::InvalidPosition)?; assert!(read.is_some(), "a waiter took the load over and loaded it"); assert_eq!(slot(key), Some((State::Ready, 1, 1))); drop(read); @@ -238,7 +249,7 @@ mod tests { // to this test's alone: the tests share the registry. set_cap(c"64MB")?; let (key, resolved) = graph("retired", 100)?; - drop(through::(port()?, &resolved).map_err(|_| Error::InvalidPosition)?); + drop(through::>(port()?, &resolved).map_err(|_| Error::InvalidPosition)?); end_statement(); end_message(); assert_eq!(slot(key), Some((State::Ready, 0, 1))); @@ -251,8 +262,12 @@ mod tests { fn a_draining_version_read_again_is_revived() -> Result<(), Error> { set_cap(c"64MB")?; let (key, resolved) = graph("revived", 100)?; - let read = - || through::(port()?, &resolved).ok().flatten().ok_or(Error::InvalidPosition); + let read = || { + through::>(port()?, &resolved) + .ok() + .flatten() + .ok_or(Error::InvalidPosition) + }; let first = read()?; end_statement(); assert_eq!(port()?.evict(&|evicted| *evicted == key), 1); @@ -270,7 +285,8 @@ mod tests { fn a_piece_of_a_version_held_past_it_keeps_the_mapping() -> Result<(), Error> { set_cap(c"64MB")?; let (key, resolved) = graph("piece", 100)?; - let read = through::(port()?, &resolved).map_err(|_| Error::InvalidPosition)?; + let read = + through::>(port()?, &resolved).map_err(|_| Error::InvalidPosition)?; let piece = read.ok_or(Error::InvalidPosition)?.id_map.clone(); end_statement(); assert_eq!(port()?.evict(&|evicted| *evicted == key), 1); @@ -287,7 +303,7 @@ mod tests { fn an_evicted_version_is_released_at_the_end_of_the_message() -> Result<(), Error> { set_cap(c"64MB")?; let (key, resolved) = graph("evicted", 100)?; - drop(through::(port()?, &resolved).map_err(|_| Error::InvalidPosition)?); + drop(through::>(port()?, &resolved).map_err(|_| Error::InvalidPosition)?); end_statement(); end_message(); assert_eq!(slot(key), Some((State::Ready, 0, 1)), "a ready version stays mapped"); @@ -302,7 +318,7 @@ mod tests { fn evict_by_an_owner_evicts_the_graphs_unpinned_shared_versions() -> Result<(), Error> { set_cap(c"64MB")?; let (key, resolved) = graph("evicted_by_owner", 100)?; - drop(through::(port()?, &resolved).map_err(|_| Error::InvalidPosition)?); + drop(through::>(port()?, &resolved).map_err(|_| Error::InvalidPosition)?); let evict = "SELECT warren.evict('evicted_by_owner')"; assert_eq!(Spi::get_one::(evict)?, Some(0), "its statement pins it"); assert_eq!(slot(key), Some((State::Ready, 1, 0)), "pinned, its mapping released"); @@ -316,7 +332,7 @@ mod tests { fn evict_by_another_role_only_releases_its_own_mappings() -> Result<(), Error> { set_cap(c"64MB")?; let (key, resolved) = graph("evicted_by_stranger", 100)?; - drop(through::(port()?, &resolved).map_err(|_| Error::InvalidPosition)?); + drop(through::>(port()?, &resolved).map_err(|_| Error::InvalidPosition)?); end_statement(); Spi::run("CREATE ROLE evict_stranger; SET ROLE evict_stranger")?; let evicted = Spi::get_one::("SELECT warren.evict('evicted_by_stranger')")?; @@ -330,7 +346,7 @@ mod tests { /// retention drops it: its key and the new version's resolution. fn superseded(name: &str) -> Result<(Key, Resolved), Error> { let (key, resolved) = graph(name, 100)?; - drop(through::(port()?, &resolved).map_err(|_| Error::InvalidPosition)?); + drop(through::>(port()?, &resolved).map_err(|_| Error::InvalidPosition)?); end_statement(); refresh(name, true)?; assert_eq!(slot(key), Some((State::Ready, 0, 1))); @@ -351,7 +367,8 @@ mod tests { fn a_shared_load_prunes_shared_versions_that_no_longer_exist() -> Result<(), Error> { set_cap(c"64MB")?; let (key, resolved) = superseded("pruned_shared")?; - let current = through::(port()?, &resolved).map_err(|_| Error::InvalidPosition)?; + let current = + through::>(port()?, &resolved).map_err(|_| Error::InvalidPosition)?; assert!(current.is_some()); assert_eq!(slot(key), None, "evicted, and released by this backend"); drop(current); @@ -376,9 +393,12 @@ mod tests { { set_cap(c"64MB")?; let [(old, old_read), (current, current_read)] = promoted("promoted")?; - drop(through::(port()?, &old_read).map_err(|_| Error::InvalidPosition)?); + drop(through::>(port()?, &old_read).map_err(|_| Error::InvalidPosition)?); end_statement(); - drop(through::(port()?, ¤t_read).map_err(|_| Error::InvalidPosition)?); + drop( + through::>(port()?, ¤t_read) + .map_err(|_| Error::InvalidPosition)?, + ); assert_eq!(slot(old), None, "evicted, and released by this backend"); assert_eq!(slot(current), Some((State::Ready, 1, 1))); forget(current) @@ -388,8 +408,12 @@ mod tests { fn loading_the_current_version_spares_the_versions_a_statement_pins() -> Result<(), Error> { set_cap(c"64MB")?; let [(old, old_read), (current, current_read)] = promoted("promoted_pinned")?; - let held = through::(port()?, &old_read).map_err(|_| Error::InvalidPosition)?; - drop(through::(port()?, ¤t_read).map_err(|_| Error::InvalidPosition)?); + let held = + through::>(port()?, &old_read).map_err(|_| Error::InvalidPosition)?; + drop( + through::>(port()?, ¤t_read) + .map_err(|_| Error::InvalidPosition)?, + ); assert_eq!(slot(old), Some((State::Ready, 1, 1)), "its statement pins it"); drop(held); forget(old)?; @@ -400,9 +424,12 @@ mod tests { fn loading_another_version_evicts_nothing() -> Result<(), Error> { set_cap(c"64MB")?; let [(old, old_read), (current, current_read)] = promoted("promoted_other")?; - drop(through::(port()?, ¤t_read).map_err(|_| Error::InvalidPosition)?); + drop( + through::>(port()?, ¤t_read) + .map_err(|_| Error::InvalidPosition)?, + ); end_statement(); - drop(through::(port()?, &old_read).map_err(|_| Error::InvalidPosition)?); + drop(through::>(port()?, &old_read).map_err(|_| Error::InvalidPosition)?); assert_eq!(slot(current), Some((State::Ready, 0, 1))); forget(old)?; forget(current) @@ -413,7 +440,8 @@ mod tests { set_cap(c"64MB")?; let (key, resolved) = graph("uncharged", 2000)?; Spi::run("SET LOCAL warren.max_cache_memory = '64kB'")?; - let shared = through::(port()?, &resolved).map_err(|_| Error::InvalidPosition)?; + let shared = + through::>(port()?, &resolved).map_err(|_| Error::InvalidPosition)?; assert!(shared.is_some(), "about 94 kB, larger than the private cap"); assert_eq!(compute_memory().in_use(), 0, "nothing of the shared version is charged"); let budget = compute_memory(); @@ -421,7 +449,7 @@ mod tests { assert_eq!(slot(key), Some((State::Ready, 1, 1)), "nor evicted by a budget's reclaim"); // The same version read privately (it is uncommitted) is charged // past the cap while the call holds it. - let private = cached::(&resolved).map_err(|_| Error::InvalidPosition)?; + let private = cached::>(&resolved).map_err(|_| Error::InvalidPosition)?; assert!(compute_memory().in_use() > 0, "a private version past the cap is charged"); drop((shared, private)); forget(key) @@ -543,7 +571,7 @@ mod tests { fn the_view_lists_shared_versions_and_vectors_to_their_graphs_readers() -> Result<(), Error> { set_cap(c"64MB")?; let (version, resolved) = graph("listed", 100)?; - drop(through::(port()?, &resolved).map_err(|_| Error::InvalidPosition)?); + drop(through::>(port()?, &resolved).map_err(|_| Error::InvalidPosition)?); Spi::run("SELECT warren.vector_const('listed', 'x', 1)")?; let bound = SpiVectorStore.bound(version.id).map_err(|_| Error::InvalidPosition)?; let record = bound.into_iter().next().ok_or(Error::InvalidPosition)?; diff --git a/crates/pg_warren/src/cache/shared/read.rs b/crates/pg_warren/src/cache/shared/read.rs index 8dd8899b..c0433527 100644 --- a/crates/pg_warren/src/cache/shared/read.rs +++ b/crates/pg_warren/src/cache/shared/read.rs @@ -21,7 +21,7 @@ use super::slot::{Key, Kind}; use super::SharedVersions; use crate::access::as_catalog_owner; use crate::cache::installation::installation; -use crate::cache::load::{fill, fill_vector, plan, plan_vector, LoadedVersion}; +use crate::cache::load::{fill, fill_vector, plan, plan_vector, CachedIds, LoadedVersion}; use crate::cache::open::Resolved; use crate::cache::statement::{pin_shared, retire_due, shared_notice_due}; use crate::cache::CacheError; @@ -29,7 +29,6 @@ use crate::catalog::VersionId; use crate::parallel_worker::cache_notice_raised; use crate::read_only::{latest_readable, read_one, read_one_latest}; use crate::segment::SpiSegmentStore; -use crate::source::IdDatum; use crate::vector::{SpiVectorStore, VectorId, VectorRecord}; use warren_core::compute::AnyVector; @@ -38,7 +37,7 @@ use warren_core::compute::AnyVector; /// unavailable, the version was created by this transaction (publishing it /// would share what no other backend can see, and what a rollback /// removes), or admission refused it. -pub(crate) fn shared( +pub(crate) fn shared( resolved: &Resolved, ) -> Result>>, CacheError> { match port_for(resolved)? { @@ -52,7 +51,7 @@ pub(crate) fn shared( /// where [`shared`] would load privately, and on a miss, which it leaves /// unclaimed: a two-version call takes what is shared first, so that its /// partner's load cannot evict it. -pub(crate) fn shared_hit( +pub(crate) fn shared_hit( resolved: &Resolved, ) -> Result>>, CacheError> { let Some(mut port) = port_for(resolved)? else { @@ -259,7 +258,7 @@ pub(super) fn evict_through(mut port: Dsm, which: &dyn Fn(&Key) -> bool) { /// The version `resolved` names through `port`, whatever transaction /// created it, or `None` if admission refused it. -pub(super) fn through( +pub(super) fn through( mut port: Dsm, resolved: &Resolved, ) -> Result>>, CacheError> { diff --git a/crates/pg_warren/src/cache/versions.rs b/crates/pg_warren/src/cache/versions.rs index 5b188e74..f110c6e0 100644 --- a/crates/pg_warren/src/cache/versions.rs +++ b/crates/pg_warren/src/cache/versions.rs @@ -9,7 +9,7 @@ use std::collections::HashMap; use std::sync::{Arc, Weak}; use std::time::Instant; -use warren_core::graph::ExternalId; +use warren_core::graph::IdKeys; use super::lru::Held; use super::LoadedVersion; @@ -57,10 +57,10 @@ pub(crate) struct VersionCache { } impl VersionCache { - /// Version `version`, if it is cached with external id type `E` (the + /// Version `version`, if it is cached with key storage `E` (the /// lookup marks it used) or was evicted while a call still holds it /// (the lookup caches it again). - pub(crate) fn get( + pub(crate) fn get( &mut self, version: VersionId, ) -> Option>> { @@ -87,7 +87,7 @@ impl VersionCache { /// Caches `loaded` as `version` of graph `graph_id`, replacing any /// entry for it, and returns it shared. - pub(crate) fn insert( + pub(crate) fn insert( &mut self, graph_id: i32, version: VersionId, @@ -99,7 +99,7 @@ impl VersionCache { } /// Caches `loaded` as `version` of graph `graph_id`. - fn cache( + fn cache( &mut self, graph_id: i32, version: VersionId, @@ -233,10 +233,10 @@ mod tests { fn a_loaded_version_is_cached_and_shared() { let mut store = store(); let mut cache = VersionCache::default(); - assert!(cache.get::(1).is_none()); - let loaded = cache.insert(10, 1, load::(&mut store, 1, true).unwrap()); + assert!(cache.get::>(1).is_none()); + let loaded = cache.insert(10, 1, load::>(&mut store, 1, true).unwrap()); assert_eq!(loaded.id_map.dense_id(&2), Some(1)); - let cached = cache.get::(1).unwrap(); + let cached = cache.get::>(1).unwrap(); assert!(Arc::ptr_eq(&loaded, &cached)); } @@ -244,10 +244,10 @@ mod tests { fn a_lookup_marks_a_version_used_and_held_reports_its_bytes() { let mut store = store(); let mut cache = VersionCache::default(); - let first = cache.insert(10, 1, load::(&mut store, 1, true).unwrap()); - cache.insert(10, 2, load::(&mut store, 2, true).unwrap()); + let first = cache.insert(10, 1, load::>(&mut store, 1, true).unwrap()); + cache.insert(10, 2, load::>(&mut store, 2, true).unwrap()); std::thread::sleep(std::time::Duration::from_millis(1)); - assert!(cache.get::(1).is_some()); + assert!(cache.get::>(1).is_some()); let held = cache.held(); assert_eq!(held.iter().map(|entry| entry.key).collect::>(), [1, 2]); assert_eq!(held[0].bytes, first.bytes()); @@ -264,27 +264,27 @@ mod tests { fn an_evicted_version_a_call_still_holds_is_cached_again_without_a_reload() { let mut store = store(); let mut cache = VersionCache::default(); - let held = cache.insert(10, 1, load::(&mut store, 1, true).unwrap()); + let held = cache.insert(10, 1, load::>(&mut store, 1, true).unwrap()); cache.remove(1); assert_eq!(cache.bytes(), 0, "the cache no longer counts it"); assert_eq!(cache.held_by_calls_bytes(), held.bytes(), "counted while held"); - let again = cache.get::(1).expect("the held copy"); + let again = cache.get::>(1).expect("the held copy"); assert!(Arc::ptr_eq(&held, &again)); assert_eq!(cached(&cache), [1], "cached again"); assert_eq!(cache.bytes(), held.bytes(), "and counted again"); cache.remove(1); drop((held, again)); assert_eq!(cache.held_by_calls_bytes(), 0); - assert!(cache.get::(1).is_none(), "gone once no call holds it"); + assert!(cache.get::>(1).is_none(), "gone once no call holds it"); } #[test] fn evicting_a_graph_except_some_versions_keeps_those() { let mut store = store(); let mut cache = VersionCache::default(); - cache.insert(10, 1, load::(&mut store, 1, true).unwrap()); - cache.insert(10, 2, load::(&mut store, 2, true).unwrap()); - cache.insert(11, 3, load::<[u8; 16]>(&mut store, 3, true).unwrap()); + cache.insert(10, 1, load::>(&mut store, 1, true).unwrap()); + cache.insert(10, 2, load::>(&mut store, 2, true).unwrap()); + cache.insert(11, 3, load::>(&mut store, 3, true).unwrap()); cache.evict_graph_except(10, |version| version == 2); assert_eq!(cached(&cache), [2, 3]); } @@ -296,12 +296,15 @@ mod tests { fn evicting_a_graph_or_pruning_keeps_what_calls_hold_counted() { let mut store = store(); let mut cache = VersionCache::default(); - let one = cache.insert(10, 1, load::(&mut store, 1, true).unwrap()); - cache.insert(10, 2, load::(&mut store, 2, true).unwrap()); + let one = cache.insert(10, 1, load::>(&mut store, 1, true).unwrap()); + cache.insert(10, 2, load::>(&mut store, 2, true).unwrap()); assert_eq!(cache.evict_graph(10), 2); assert_eq!((cache.bytes(), cache.held_by_calls_bytes()), (0, one.bytes())); - assert!(Arc::ptr_eq(&one, &cache.get::(1).unwrap()), "shared, not reloaded"); - assert!(cache.get::(2).is_none(), "no call held version 2"); + assert!( + Arc::ptr_eq(&one, &cache.get::>(1).unwrap()), + "shared, not reloaded" + ); + assert!(cache.get::>(2).is_none(), "no call held version 2"); cache.retain(&[]); assert_eq!((cache.bytes(), cache.held_by_calls_bytes()), (0, one.bytes()), "pruned, held"); drop(one); @@ -314,12 +317,12 @@ mod tests { fn a_prune_checks_held_versions_and_keeps_those_that_exist() { let mut store = store(); let mut cache = VersionCache::default(); - let held = cache.insert(10, 1, load::(&mut store, 1, true).unwrap()); + let held = cache.insert(10, 1, load::>(&mut store, 1, true).unwrap()); cache.remove(1); - cache.insert(10, 2, load::(&mut store, 2, true).unwrap()); + cache.insert(10, 2, load::>(&mut store, 2, true).unwrap()); assert_eq!(cache.known_versions(), [1, 2]); cache.retain(&cache.known_versions()); - let again = cache.get::(1).expect("the held copy, after the prune"); + let again = cache.get::>(1).expect("the held copy, after the prune"); assert!(Arc::ptr_eq(&held, &again)); assert_eq!(cached(&cache), [1, 2], "found again, so cached again"); cache.remove(1); @@ -330,17 +333,17 @@ mod tests { #[test] fn a_cached_version_is_found_only_with_its_id_type() { let mut cache = VersionCache::default(); - cache.insert(11, 3, load::<[u8; 16]>(&mut store(), 3, true).unwrap()); - assert!(cache.get::<[u8; 16]>(3).is_some()); - assert!(cache.get::(3).is_none()); + cache.insert(11, 3, load::>(&mut store(), 3, true).unwrap()); + assert!(cache.get::>(3).is_some()); + assert!(cache.get::>(3).is_none()); } #[test] fn versions_that_no_longer_exist_are_pruned() { let mut store = store(); let mut cache = VersionCache::default(); - cache.insert(10, 1, load::(&mut store, 1, true).unwrap()); - cache.insert(10, 2, load::(&mut store, 2, true).unwrap()); + cache.insert(10, 1, load::>(&mut store, 1, true).unwrap()); + cache.insert(10, 2, load::>(&mut store, 2, true).unwrap()); assert_eq!(cached(&cache), [1, 2]); cache.retain(&[2, 5]); assert_eq!(cached(&cache), [2]); @@ -350,9 +353,9 @@ mod tests { fn evicting_a_graph_drops_only_its_versions_and_spares_holders() { let mut store = store(); let mut cache = VersionCache::default(); - let held = cache.insert(10, 1, load::(&mut store, 1, true).unwrap()); - cache.insert(10, 2, load::(&mut store, 2, true).unwrap()); - cache.insert(11, 3, load::<[u8; 16]>(&mut store, 3, true).unwrap()); + let held = cache.insert(10, 1, load::>(&mut store, 1, true).unwrap()); + cache.insert(10, 2, load::>(&mut store, 2, true).unwrap()); + cache.insert(11, 3, load::>(&mut store, 3, true).unwrap()); assert_eq!(cache.evict_graph(10), 2); assert_eq!(cached(&cache), [3]); assert_eq!(held.id_map.node_count(), 3, "a scan holding a version keeps it"); diff --git a/crates/pg_warren/src/catalog/create_graph.rs b/crates/pg_warren/src/catalog/create_graph.rs index 1964dfe8..f0fdca1b 100644 --- a/crates/pg_warren/src/catalog/create_graph.rs +++ b/crates/pg_warren/src/catalog/create_graph.rs @@ -273,6 +273,40 @@ mod tests { Ok(()) } + /// DESIGN.md §5.7: `varchar` ids are `text` ids. + #[pg_test] + fn text_and_varchar_node_ids_are_text() -> Result<(), Error> { + Spi::run( + "CREATE TABLE cg_t (id text, v varchar(20)); + SELECT warren.create_graph('cgt', 'SELECT v FROM cg_t', 'SELECT id, v FROM cg_t')", + )?; + let id_type = + Spi::get_one::("SELECT id_type::text FROM warren.graphs WHERE name = 'cgt'")?; + assert_eq!(id_type.as_deref(), Some("text")); + Ok(()) + } + + #[pg_test(error = "node_query column 1 (node id) is of type character, which node ids cannot \ + have: its equality ignores trailing spaces; cast it to text (id::text)")] + fn bpchar_node_ids_are_refused() -> Result<(), Error> { + Spi::run( + "SELECT warren.create_graph('cg', 'SELECT ''a''::char(3)', 'SELECT NULL::text, NULL::text')", + ) + } + + #[pg_test(error = "edge_query column 2 (target id) has the nondeterministic collation \ + \"cg_ci\", whose equality node ids cannot follow (they compare as bytes): \ + write the column COLLATE \"C\" for exact ids, or normalise them in the \ + query (for example lower(id))")] + fn a_nondeterministic_collation_is_refused() -> Result<(), Error> { + Spi::run( + "CREATE COLLATION cg_ci (provider = icu, locale = 'und-u-ks-level2', + deterministic = false); + SELECT warren.create_graph('cg', 'SELECT ''a''::text', + 'SELECT ''a''::text, ''a''::text COLLATE cg_ci')", + ) + } + #[pg_test(error = "edge_query column 1 (source id) must be of type uuid, not bigint")] fn edge_endpoints_must_have_the_node_id_type() -> Result<(), Error> { tables()?; diff --git a/crates/pg_warren/src/catalog/version.rs b/crates/pg_warren/src/catalog/version.rs index e1971add..13b981da 100644 --- a/crates/pg_warren/src/catalog/version.rs +++ b/crates/pg_warren/src/catalog/version.rs @@ -3,7 +3,7 @@ //! catalog owner. use pgrx::prelude::*; -use warren_core::graph::{ExternalId, FixedKeys, GraphCsr, IdMap, TargetWidth}; +use warren_core::graph::{GraphCsr, IdKeys, IdMap, TargetWidth}; use super::retention::{retention, Kept, VersionSummary}; use super::{CatalogError, GraphDefinition}; @@ -35,8 +35,8 @@ pub(crate) struct VersionStats { impl VersionStats { /// The statistics of a version with this id map and these CSRs, and /// the given dropped and deduplicated edge counts. - pub(crate) fn measure( - id_map: &IdMap>, + pub(crate) fn measure( + id_map: &IdMap, graph: &GraphCsr, dropped_edges: u64, deduplicated_edges: u64, @@ -48,8 +48,8 @@ impl VersionStats { /// The statistics of a version with this id map, `edge_count` edges /// over its relations, and the given dropped and deduplicated edge /// counts. - pub(crate) fn new( - id_map: &IdMap>, + pub(crate) fn new( + id_map: &IdMap, edge_count: u64, dropped_edges: u64, deduplicated_edges: u64, diff --git a/crates/pg_warren/src/community_sql/contract.rs b/crates/pg_warren/src/community_sql/contract.rs index cf51725c..7e0048b0 100644 --- a/crates/pg_warren/src/community_sql/contract.rs +++ b/crates/pg_warren/src/community_sql/contract.rs @@ -16,7 +16,8 @@ use crate::catalog::{ level_graph_name, require_unreserved, CatalogError, GraphDefinition, GraphSource, Lineage, NewGraph, }; -use crate::compute_sql::{workers, Vectors}; +use crate::compute_sql::Vectors; +use crate::interrupts::workers; use crate::interrupts::PostgresInterrupts; use crate::segment::{SpiSegmentStore, CHUNK_SIZE}; use crate::source::{with_id_type, IdType}; diff --git a/crates/pg_warren/src/community_sql/contraction.rs b/crates/pg_warren/src/community_sql/contraction.rs index 9ca253f5..ecc19fdf 100644 --- a/crates/pg_warren/src/community_sql/contraction.rs +++ b/crates/pg_warren/src/community_sql/contraction.rs @@ -9,7 +9,7 @@ use warren_core::community::{contract, Contraction, Labels}; use warren_core::compute::{AnyVector, Workers}; use warren_core::graph::{ - AnyCsr, BuildCharge, BuildMemory, CsrLayout, ExternalId, GraphCsr, IdMap, RelationDictionary, + AnyCsr, BuildCharge, BuildMemory, CsrLayout, GraphCsr, IdKeys, IdMap, RelationDictionary, RelationIndex, }; use warren_core::interrupts::Interrupts; @@ -32,7 +32,7 @@ pub(crate) struct ContractRequest<'a> { /// Contracts `parent` by `labels` into a new version of `quotient` (a /// graph derived by contraction, its row locked), and returns its id. -pub(crate) fn contract_version( +pub(crate) fn contract_version( quotient: &GraphDefinition, parent: &LoadedVersion, labels: &AnyVector, @@ -121,7 +121,7 @@ mod tests { /// Nodes 10, 20, 30, 40 (30 tombstoned); edges 10 → 20 (`a`, 1.0), /// 20 → 40 (`b`, 2.0) and 40 → 10 (`b`, 4.0). - fn parent() -> LoadedVersion { + fn parent() -> LoadedVersion> { let id_map = IdMap::new() .rebuild(vec![10, 20, 30, 40]) .and_then(|m| m.rebuild(vec![10, 20, 40])) @@ -191,7 +191,7 @@ mod tests { .unwrap() } - fn loaded(&mut self, version: VersionId) -> LoadedVersion { + fn loaded(&mut self, version: VersionId) -> LoadedVersion> { crate::cache::load(&mut self.store, version, true).unwrap() } } diff --git a/crates/pg_warren/src/community_sql/error.rs b/crates/pg_warren/src/community_sql/error.rs index df5f2c91..13edccdb 100644 --- a/crates/pg_warren/src/community_sql/error.rs +++ b/crates/pg_warren/src/community_sql/error.rs @@ -10,7 +10,8 @@ use crate::access::{AccessError, SqlError}; use crate::build::BuildError; use crate::cache::{CacheError, LoadError}; use crate::catalog::{CatalogError, VersionId}; -use crate::compute_sql::{compute_sqlstate, ComputeSqlError}; +use crate::compute_sql::ComputeSqlError; +use crate::extension::compute_sqlstate; use crate::extension::graph_sqlstate; use crate::segment::SegmentError; diff --git a/crates/pg_warren/src/community_sql/local_moving.rs b/crates/pg_warren/src/community_sql/local_moving.rs index 959ff353..fea86e7a 100644 --- a/crates/pg_warren/src/community_sql/local_moving.rs +++ b/crates/pg_warren/src/community_sql/local_moving.rs @@ -11,7 +11,8 @@ use super::partition::{integers, undirected}; use super::CommunitySqlError; use crate::access::entry; use crate::cache::resolve; -use crate::compute_sql::{workers, Vectors}; +use crate::compute_sql::Vectors; +use crate::interrupts::workers; use crate::interrupts::PostgresInterrupts; use crate::source::with_id_type; diff --git a/crates/pg_warren/src/community_sql/partition.rs b/crates/pg_warren/src/community_sql/partition.rs index 4e546cc4..a1e8fa97 100644 --- a/crates/pg_warren/src/community_sql/partition.rs +++ b/crates/pg_warren/src/community_sql/partition.rs @@ -8,15 +8,16 @@ use warren_core::compute::MemoryCharge; use warren_core::graph::BuildMemory; use super::CommunitySqlError; -use crate::compute_sql::{workers, Vectors}; +use crate::cache::CachedIds; +use crate::compute_sql::Vectors; +use crate::interrupts::workers; use crate::interrupts::PostgresInterrupts; -use crate::source::IdDatum; /// The undirected weighted view of `vectors`' version: each edge weighs /// its weight if `weighted` (NULL: if the graph has weights), else 1; the /// two directions of a directed graph add up. A negative or NaN weight is /// refused, naming the edge by its external ids. -pub(super) fn undirected( +pub(super) fn undirected( vectors: &Vectors, weighted: Option, ) -> Result { @@ -31,7 +32,7 @@ pub(super) fn undirected( built.map_err(|error| match error { CommunityError::NegativeWeight { from, to, weight } => { let id_map = &vectors.loaded.id_map; - let external = |dense| id_map.external_id(dense).map(|id| format!("{id:?}")); + let external = |dense| id_map.external_id(dense).map(E::named); CommunitySqlError::NegativeWeight { from: external(from).unwrap_or_default(), to: external(to).unwrap_or_default(), @@ -46,7 +47,7 @@ pub(super) fn undirected( /// tombstoned node), refused as `contract` refuses labels that are not /// integers, with its charge to the call's budget. `what` names the /// argument, for the error. -pub(super) fn integers( +pub(super) fn integers( vectors: &mut Vectors, name: &str, what: &'static str, diff --git a/crates/pg_warren/src/community_sql/projection.rs b/crates/pg_warren/src/community_sql/projection.rs index cba1c037..b1db4661 100644 --- a/crates/pg_warren/src/community_sql/projection.rs +++ b/crates/pg_warren/src/community_sql/projection.rs @@ -6,15 +6,15 @@ use pgrx::prelude::*; use warren_core::community::{self, Aggregate}; use warren_core::compute::{ComputeMemory, MemoryCharge}; -use warren_core::graph::Membership; +use warren_core::graph::{FixedKeys, Membership}; use super::CommunitySqlError; use crate::access::{entry, is_reader}; -use crate::cache::{cached, compute_memory, resolve, Resolved}; +use crate::cache::{cached, compute_memory, resolve, CachedIds, Resolved}; use crate::catalog::{parent_of, GraphDefinition, GraphSource}; use crate::compute_sql::Vectors; use crate::segment::{read_segment, SpiSegmentStore, MEMBERSHIP}; -use crate::source::{with_id_type, IdDatum}; +use crate::source::with_id_type; /// `warren.project`: vector `y` of `out_graph`'s version `version` (NULL: /// its current one) as each community's `agg` (`sum`, `mean`, `max`, `min` @@ -93,7 +93,7 @@ struct Pair { /// within one compute budget. struct Opened { parent: Vectors, - quotient: Vectors, + quotient: Vectors>, membership: Membership, _membership_charge: MemoryCharge, memory: ComputeMemory, @@ -145,9 +145,9 @@ impl Pair { } /// Loads both versions and the membership, charged to one budget. - fn open(self) -> Result, CommunitySqlError> { + fn open(self) -> Result, CommunitySqlError> { let parent_loaded = cached::(&self.parent)?; - let quotient_loaded = cached::(&self.quotient)?; + let quotient_loaded = cached::>(&self.quotient)?; let memory = compute_memory(); let parent_nodes = parent_loaded.id_map.id_space(); // The segment as read, and the membership decoded from it. diff --git a/crates/pg_warren/src/compute_sql/centrality.rs b/crates/pg_warren/src/compute_sql/centrality.rs index 0596878f..ea3b55c7 100644 --- a/crates/pg_warren/src/compute_sql/centrality.rs +++ b/crates/pg_warren/src/compute_sql/centrality.rs @@ -12,10 +12,10 @@ use warren_core::compute::{ use pgrx::prelude::*; -use super::settings::workers; use super::vectors::Vectors; use super::{entry, selection, ComputeSqlError}; use crate::cache::resolve; +use crate::interrupts::workers; use crate::interrupts::PostgresInterrupts; use crate::selection::relations; use crate::source::with_id_type; diff --git a/crates/pg_warren/src/compute_sql/constructors.rs b/crates/pg_warren/src/compute_sql/constructors.rs index fec465de..0367736f 100644 --- a/crates/pg_warren/src/compute_sql/constructors.rs +++ b/crates/pg_warren/src/compute_sql/constructors.rs @@ -5,13 +5,13 @@ use pgrx::prelude::*; use warren_core::compute::{self, map, AnyExpr, AnyVector, Dtype, Expr, Vector}; use warren_core::graph::RelationSelection; -use super::settings::workers; use super::vectors::Vectors; use super::{entry, ComputeSqlError}; -use crate::cache::resolve; +use crate::cache::{resolve, CachedIds}; +use crate::interrupts::workers; use crate::interrupts::PostgresInterrupts; use crate::selection::{adjacency, direction, relations}; -use crate::source::{with_id_type, IdDatum}; +use crate::source::with_id_type; /// `value` as an expression of `dtype`: a `double precision` literal, cast /// as SQL casts it. @@ -55,7 +55,7 @@ fn vector_const( }) } -fn constant( +fn constant( mut vectors: Vectors, name: &str, expr: &AnyExpr, @@ -70,7 +70,7 @@ fn constant( const POLL_NODES: u64 = 1 << 16; /// `vector`, an `i64` vector, cast to `dtype` as SQL casts `bigint`. -fn cast( +fn cast( vectors: &Vectors, vector: AnyVector, dtype: Dtype, diff --git a/crates/pg_warren/src/compute_sql/error.rs b/crates/pg_warren/src/compute_sql/error.rs index bfebfb3e..bbf1d9b8 100644 --- a/crates/pg_warren/src/compute_sql/error.rs +++ b/crates/pg_warren/src/compute_sql/error.rs @@ -3,12 +3,13 @@ use pgrx::PgSqlErrorCode; use thiserror::Error; -use warren_core::compute::{ArithmeticError, ComputeError}; +use warren_core::compute::ComputeError; use warren_core::version::VersionError; use crate::access::{self, AccessError, SqlError}; use crate::cache::{CacheError, LoadError}; use crate::catalog::{CatalogError, VersionId}; +use crate::extension::compute_sqlstate; use crate::external_ids::ExternalIdError; use crate::selection::SelectionError; use crate::vector::VectorError; @@ -166,57 +167,6 @@ impl SqlError for ComputeSqlError { } } -/// The SQLSTATE of a kernel error: the one Postgres raises for the same -/// arithmetic, so the compiled path and the executor agree. -pub(crate) fn compute_sqlstate(error: &ComputeError) -> PgSqlErrorCode { - match error { - ComputeError::MemoryLimit { .. } => PgSqlErrorCode::ERRCODE_CONFIGURATION_LIMIT_EXCEEDED, - ComputeError::OutOfMemory { .. } => PgSqlErrorCode::ERRCODE_OUT_OF_MEMORY, - ComputeError::UnknownDtype(_) - | ComputeError::UnknownSemiring(_) - | ComputeError::UnknownBinaryOp(_) - | ComputeError::UnknownReduction(_) - | ComputeError::UnknownNorm(_) - | ComputeError::UnknownCombiner(_) - | ComputeError::UnknownSimilarity(_) - | ComputeError::ZeroWidth - | ComputeError::WidthMismatch { .. } - | ComputeError::Cycle { .. } - | ComputeError::NonPositiveWeight => PgSqlErrorCode::ERRCODE_INVALID_PARAMETER_VALUE, - ComputeError::DtypeMismatch { .. } - | ComputeError::OperandType { .. } - | ComputeError::CombinerType { .. } => PgSqlErrorCode::ERRCODE_DATATYPE_MISMATCH, - ComputeError::IdSpaceMismatch { .. } => { - PgSqlErrorCode::ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE - } - ComputeError::NullResult { .. } => PgSqlErrorCode::ERRCODE_NULL_VALUE_NOT_ALLOWED, - ComputeError::Arithmetic(error) => arithmetic_sqlstate(*error), - ComputeError::Cancelled => PgSqlErrorCode::ERRCODE_QUERY_CANCELED, - ComputeError::VectorBytes { .. } | ComputeError::VectorLength { .. } => { - PgSqlErrorCode::ERRCODE_DATA_CORRUPTED - } - _ => PgSqlErrorCode::ERRCODE_INTERNAL_ERROR, - } -} - -/// The SQLSTATE `float.c` and `int8.c` raise for an arithmetic error. -fn arithmetic_sqlstate(error: ArithmeticError) -> PgSqlErrorCode { - match error { - ArithmeticError::BigintOutOfRange - | ArithmeticError::FloatOverflow - | ArithmeticError::FloatUnderflow => PgSqlErrorCode::ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE, - ArithmeticError::DivisionByZero => PgSqlErrorCode::ERRCODE_DIVISION_BY_ZERO, - ArithmeticError::SqrtOfNegative - | ArithmeticError::ZeroToNegativePower - | ArithmeticError::ComplexPower => { - PgSqlErrorCode::ERRCODE_INVALID_ARGUMENT_FOR_POWER_FUNCTION - } - ArithmeticError::LogOfZero | ArithmeticError::LogOfNegative => { - PgSqlErrorCode::ERRCODE_INVALID_ARGUMENT_FOR_LOG - } - } -} - /// Runs the body of a compute function under the access guard (trusted /// `search_path`, errors raised with their SQLSTATE). pub(crate) fn entry(body: impl FnOnce(&access::Caller) -> Result) -> T { diff --git a/crates/pg_warren/src/compute_sql/from_query.rs b/crates/pg_warren/src/compute_sql/from_query.rs index e05a2bbc..c56b2d95 100644 --- a/crates/pg_warren/src/compute_sql/from_query.rs +++ b/crates/pg_warren/src/compute_sql/from_query.rs @@ -20,24 +20,23 @@ //! only if stored out of line, compressed or with a short header), so the //! read holds no Postgres memory per row. +use std::borrow::Borrow; use std::ffi::CStr; use pgrx::datum::UnboxDatum; use pgrx::prelude::*; -use pgrx::spi::SpiTupleTable; use pgrx::{Array, FromDatum, IntoDatum}; use warren_core::compute::{map, AnyVector, ComputeError, Dtype, MemoryCharge, Value, Vector}; use super::constructors::{dtype as parse_dtype, literal}; -use super::settings::workers; use super::vectors::Vectors; use super::{entry, ComputeSqlError}; use crate::access::{with_read_only, with_search_path, Caller}; -use crate::cache::resolve; -use crate::interrupts::PostgresInterrupts; +use crate::cache::{resolve, CachedIds}; +use crate::external_ids::DenseIds; +use crate::interrupts::{workers, PostgresInterrupts}; use crate::source::{ - describe, for_each_tuple, open_read_only, type_name, with_id_type, ColumnType, IdDatum, - SourceError, + describe, for_each_tuple, open_read_only, type_name, with_id_type, ColumnType, Row, SourceError, }; /// The SQL type a dtype's values are cast to. @@ -88,8 +87,8 @@ fn vector_from_query( /// Checks that a query's id column casts implicitly to the graph's id /// type `E`, so ids are never rounded. -pub(super) fn check_id_column(column: &ColumnType) -> Result<(), ComputeSqlError> { - let id_type = E::Sql::type_oid(); +pub(super) fn check_id_column(column: &ColumnType) -> Result<(), ComputeSqlError> { + let id_type = E::ID_TYPE.oid(); // SAFETY: can_coerce_type reads one type pair from the catalogs. let implicit = unsafe { pg_sys::can_coerce_type( @@ -109,28 +108,13 @@ pub(super) fn check_id_column(column: &ColumnType) -> Result<(), Com } /// The qualified SQL name of `E`'s type. -pub(super) fn id_type_name() -> String { +pub(super) fn id_type_name() -> String { // SAFETY: format_type_be_qualified returns a palloc'd NUL-terminated // string for any type oid. - let name = unsafe { CStr::from_ptr(pg_sys::format_type_be_qualified(E::Sql::type_oid())) }; + let name = unsafe { CStr::from_ptr(pg_sys::format_type_be_qualified(E::ID_TYPE.oid())) }; name.to_string_lossy().into_owned() } -/// The external id `id` as text, by its type's output function. -fn id_text(id: E) -> String { - let Some(datum) = id.to_sql().into_datum() else { - return String::new(); - }; - let (mut output, mut varlena) = (pg_sys::Oid::INVALID, false); - // SAFETY: the out-pointers point to live locals; the datum is of the - // type whose output function is called, which returns a palloc'd - // NUL-terminated string. - unsafe { - pg_sys::getTypeOutputInfo(E::Sql::type_oid(), &mut output, &mut varlena); - CStr::from_ptr(pg_sys::OidOutputFunctionCall(output, datum)).to_string_lossy().into_owned() - } -} - /// A query to read a vector from. struct Read<'a> { caller: &'a Caller, @@ -141,7 +125,7 @@ struct Read<'a> { impl Read<'_> { /// The vector of `T` the query's rows make. - fn vector(&self, vectors: &Vectors) -> Result + fn vector(&self, vectors: &Vectors) -> Result where AnyVector: From>, for<'a> T: Value + FromDatum + IntoDatum + UnboxDatum = T>, @@ -170,10 +154,12 @@ impl Read<'_> { ); let mut opened = open_read_only::(client, "query", &wrapped)?; let mut rows = Rows::::new(vectors, self.fill_value::(vectors)?)?; + let mut ids = DenseIds::new(&vectors.loaded.id_map, &vectors.memory); for_each_tuple(&mut opened.cursor, |row| match array { - true => rows.matrix_row(vectors, row), - false => rows.scalar_row(vectors, row), + true => rows.matrix_row(vectors, &mut ids, row), + false => rows.scalar_row(vectors, &mut ids, row), })?; + drop(ids); rows.finish(vectors) }) }) @@ -181,7 +167,10 @@ impl Read<'_> { } /// `fill` cast to `T` as SQL casts `double precision`. - fn fill_value(&self, vectors: &Vectors) -> Result { + fn fill_value( + &self, + vectors: &Vectors, + ) -> Result { let one = map( &*workers()?, &vectors.memory, @@ -208,7 +197,7 @@ where AnyVector: From>, for<'a> T: Value + FromDatum + IntoDatum + UnboxDatum = T>, { - fn new(vectors: &Vectors, fill: T) -> Result { + fn new(vectors: &Vectors, fill: T) -> Result { let id_space = vectors.id_space(); let charge = vectors.memory.charge(id_space)?; let mut seen = Vec::new(); @@ -220,26 +209,26 @@ where /// The dense id of the row's node, if it is a live node of the version /// and has had no row before. - fn node( + fn node( &mut self, - vectors: &Vectors, - row: &SpiTupleTable<'_>, + ids: &mut DenseIds<'_, E>, + row: &Row<'_>, ) -> Result, ComputeSqlError> { - let Some(id) = row.get::(1)?.map(E::from_sql) else { + let Some(id) = row.id::(1)? else { return Ok(None); }; - let Some(node) = vectors.loaded.id_map.dense_id(&id) else { + let Some(node) = ids.dense_id(id.borrow())? else { return Ok(None); }; let node = node as usize; if std::mem::replace(&mut self.seen[node], true) { - return Err(ComputeSqlError::DuplicateNode(id_text(id))); + return Err(ComputeSqlError::DuplicateNode(E::named(id.borrow()))); } Ok(Some(node)) } /// The vector, allocated at `width` on first use. - fn values( + fn values( &mut self, vectors: &Vectors, width: usize, @@ -263,12 +252,13 @@ where Ok(T::NULL.ok_or(ComputeError::NullResult { dtype: T::DTYPE })?) } - fn scalar_row( + fn scalar_row( &mut self, vectors: &Vectors, - row: &SpiTupleTable<'_>, + ids: &mut DenseIds<'_, E>, + row: &Row<'_>, ) -> Result<(), ComputeSqlError> { - let Some(node) = self.node(vectors, row)? else { + let Some(node) = self.node(ids, row)? else { return Ok(()); }; let value = match row.get::(2)? { @@ -279,12 +269,13 @@ where Ok(()) } - fn matrix_row( + fn matrix_row( &mut self, vectors: &Vectors, - row: &SpiTupleTable<'_>, + ids: &mut DenseIds<'_, E>, + row: &Row<'_>, ) -> Result<(), ComputeSqlError> { - let Some(node) = self.node(vectors, row)? else { + let Some(node) = self.node(ids, row)? else { return Ok(()); }; let elements = match row.get_datum_by_ordinal(2)? { @@ -319,7 +310,7 @@ where } /// The vector; of width 1 if the query returned no rows. - fn finish(self, vectors: &Vectors) -> Result { + fn finish(self, vectors: &Vectors) -> Result { Ok(match self.values { Some(values) => values.into(), None => Vector::filled(&vectors.memory, vectors.id_space(), 1, self.fill)?.into(), diff --git a/crates/pg_warren/src/compute_sql/gather/emit.rs b/crates/pg_warren/src/compute_sql/gather/emit.rs index b3665a1f..f90bbe55 100644 --- a/crates/pg_warren/src/compute_sql/gather/emit.rs +++ b/crates/pg_warren/src/compute_sql/gather/emit.rs @@ -12,9 +12,8 @@ use pgrx::prelude::*; use pgrx::{AnyElement, IntoDatum, PgSqlErrorCode}; use warren_core::compute::{Bytes, Gathered, Message, Received, RowAt}; -use crate::cache::LoadedVersion; +use crate::cache::{CachedIds, LoadedVersion}; use crate::external_ids::external; -use crate::source::IdDatum; /// A row of the result. pub(super) type Row = (AnyElement, AnyElement); @@ -24,7 +23,7 @@ pub(super) type Build = Box FnMut(::View<'a>, pg_sys::Oid) -> Option>; /// The rows of a gather, built one at a time. -pub(super) struct Stream { +pub(super) struct Stream { /// The loaded version, for external ids. pub(super) loaded: Arc>, /// The kernel's result. @@ -39,7 +38,7 @@ pub(super) struct Stream { pub(super) at: RowAt, } -impl Iterator for Stream { +impl Iterator for Stream { type Item = Row; fn next(&mut self) -> Option { diff --git a/crates/pg_warren/src/compute_sql/gather/mod.rs b/crates/pg_warren/src/compute_sql/gather/mod.rs index 16d4da33..815e9ad9 100644 --- a/crates/pg_warren/src/compute_sql/gather/mod.rs +++ b/crates/pg_warren/src/compute_sql/gather/mod.rs @@ -22,16 +22,16 @@ use warren_core::compute::{ use warren_core::graph::RelationSelection; use super::from_query::{check_id_column, id_type_name}; -use super::settings::workers; use super::vectors::Vectors; use super::{entry, ComputeSqlError}; use crate::access::{with_search_path, Caller}; -use crate::cache::{resolve, Resolved}; +use crate::cache::{resolve, CachedIds, Resolved}; use crate::external_ids::check_type; +use crate::interrupts::workers; use crate::interrupts::PostgresInterrupts; use crate::read_only::read_one; use crate::selection::{direction, relations}; -use crate::source::{describe, type_name, with_id_type, IdDatum, SourceError}; +use crate::source::{describe, type_name, with_id_type, SourceError}; pub(crate) use codec::TypeError; use codec::{Codec, ResultType}; use emit::{Build, Row, Stream}; @@ -140,7 +140,7 @@ fn result_oid(result: ResultType) -> Result { impl Arguments<'_> { /// Where the messages come from. - fn prepare( + fn prepare( &self, vectors: &mut Vectors, caller: &Caller, @@ -190,7 +190,7 @@ impl Arguments<'_> { } /// Gathers at id type `E`. - fn run( + fn run( &self, resolved: Resolved, caller: &Caller, @@ -272,7 +272,7 @@ macro_rules! sql_element { } sql_element!(f64, f32, bool); -impl Run<'_, E> { +impl Run<'_, E> { /// Fans `messages` out, combines them, and streams the result, whose /// single values `build` makes. fn stream( diff --git a/crates/pg_warren/src/compute_sql/gather/read.rs b/crates/pg_warren/src/compute_sql/gather/read.rs index c928bc66..30ea58a7 100644 --- a/crates/pg_warren/src/compute_sql/gather/read.rs +++ b/crates/pg_warren/src/compute_sql/gather/read.rs @@ -9,8 +9,9 @@ //! copy, a send function's state) is freed with its batch rather than //! held until the read ends. +use std::borrow::Borrow; + use pgrx::prelude::*; -use pgrx::spi::SpiTupleTable; use pgrx::{AnyElement, FromDatum, IntoDatum}; use warren_core::compute::{Bytes, Element, Messages}; use warren_core::graph::DenseId; @@ -18,10 +19,12 @@ use warren_core::graph::DenseId; use super::super::vectors::Vectors; use super::super::ComputeSqlError; use crate::access::{with_read_only, with_search_path, Caller}; -use crate::source::{for_each_tuple, open_read_only, IdDatum}; +use crate::cache::CachedIds; +use crate::external_ids::DenseIds; +use crate::source::{for_each_tuple, open_read_only, Row}; /// Every live node's value of `vector` (width 1), as a message. -pub(super) fn from_vector( +pub(super) fn from_vector( vectors: &Vectors, values: &[T], messages: &mut Messages, @@ -39,22 +42,24 @@ pub(super) fn from_vector( } /// Passes each row of `query` (columns `id`, `value`) whose id is a live -/// node of the version to `each`, with its dense id. -pub(super) fn from_query( +/// node of the version to `each`, with its dense id; ids are mapped +/// through [`DenseIds`], which indexes a text id map for a long query. +pub(super) fn from_query( vectors: &Vectors, caller: &Caller, query: &str, - mut each: impl FnMut(DenseId, &SpiTupleTable<'_>) -> Result<(), ComputeSqlError>, + mut each: impl FnMut(DenseId, &Row<'_>) -> Result<(), ComputeSqlError>, ) -> Result<(), ComputeSqlError> { with_search_path(caller.search_path(), || { with_read_only(|| { Spi::connect(|client| { let mut opened = open_read_only::(client, "messages", query)?; + let mut ids = DenseIds::new(&vectors.loaded.id_map, &vectors.memory); for_each_tuple(&mut opened.cursor, |row| { - let Some(id) = row.get::(1)?.map(E::from_sql) else { + let Some(id) = row.id::(1)? else { return Ok(()); }; - match vectors.loaded.id_map.dense_id(&id) { + match ids.dense_id(id.borrow())? { Some(node) => each(node, row), None => Ok(()), } @@ -68,7 +73,7 @@ pub(super) fn from_query( pub(super) fn element( messages: &mut Messages, node: DenseId, - row: &SpiTupleTable<'_>, + row: &Row<'_>, ) -> Result<(), ComputeSqlError> { if let Some(value) = row.get::(2)? { messages.push(node, value)?; @@ -81,7 +86,7 @@ pub(super) fn element( pub(super) fn payload( messages: &mut Messages, node: DenseId, - row: &SpiTupleTable<'_>, + row: &Row<'_>, ) -> Result<(), ComputeSqlError> { let Some(value) = row.get::(2)? else { return Ok(()); @@ -107,7 +112,7 @@ pub(super) fn binary( send: pg_sys::Oid, messages: &mut Messages, node: DenseId, - row: &SpiTupleTable<'_>, + row: &Row<'_>, ) -> Result<(), ComputeSqlError> { let Some(value) = row.get::(2)? else { return Ok(()); diff --git a/crates/pg_warren/src/compute_sql/gather/tests.rs b/crates/pg_warren/src/compute_sql/gather/tests.rs index 0f1358e4..a353d0e1 100644 --- a/crates/pg_warren/src/compute_sql/gather/tests.rs +++ b/crates/pg_warren/src/compute_sql/gather/tests.rs @@ -6,6 +6,7 @@ mod tests { use pgrx::spi::Error; use warren_core::compute::{Bytes, Messages}; + use warren_core::graph::FixedKeys; use crate::cache::resolve; use crate::compute_sql::gather::read; @@ -381,7 +382,7 @@ mod tests { unsafe { pg_sys::getTypeBinaryOutputInfo(pg_sys::INT4ARRAYOID, &mut send, &mut varlena) }; let (rows, first_batch, peak) = with_pinned_snapshot(|| { entry(|caller| { - let vectors = Vectors::::open(resolve("big", None, "read")?)?; + let vectors = Vectors::>::open(resolve("big", None, "read")?)?; let mut messages = Messages::::new(&vectors.memory)?; let (before, mut rows, mut first_batch, mut peak) = (allocated(), 0, 0, 0); read::from_query( diff --git a/crates/pg_warren/src/compute_sql/limits.rs b/crates/pg_warren/src/compute_sql/limits.rs index e33d0233..d3b375f2 100644 --- a/crates/pg_warren/src/compute_sql/limits.rs +++ b/crates/pg_warren/src/compute_sql/limits.rs @@ -10,8 +10,8 @@ mod tests { use pgrx::prelude::*; use pgrx::spi::Error; - use crate::compute_sql::settings::workers; use crate::fixture::{failure, refresh, ring}; + use crate::interrupts::workers; use crate::interrupts::PostgresInterrupts; use warren_core::interrupts::{Cancelled, Interrupts}; diff --git a/crates/pg_warren/src/compute_sql/map/mod.rs b/crates/pg_warren/src/compute_sql/map/mod.rs index 9140f32b..b954b413 100644 --- a/crates/pg_warren/src/compute_sql/map/mod.rs +++ b/crates/pg_warren/src/compute_sql/map/mod.rs @@ -18,13 +18,13 @@ use pgrx::JsonB; use serde_json::json; use warren_core::compute::{map, AnyVector}; -use super::settings::workers; use super::vectors::{bound_names, BoundName, Vectors}; use super::{entry, ComputeSqlError}; use crate::access::{with_read_only, with_search_path, Caller}; -use crate::cache::resolve; +use crate::cache::{resolve, CachedIds}; +use crate::interrupts::workers; use crate::interrupts::PostgresInterrupts; -use crate::source::{with_id_type, IdDatum}; +use crate::source::with_id_type; use parse::{analyse, Analysed, Column}; use translate::{translate, Compiled}; @@ -108,7 +108,7 @@ fn vector_map( }) } -fn mapped( +fn mapped( caller: &Caller, mut vectors: Vectors, y: &str, diff --git a/crates/pg_warren/src/compute_sql/mod.rs b/crates/pg_warren/src/compute_sql/mod.rs index 5035f561..0957ccd5 100644 --- a/crates/pg_warren/src/compute_sql/mod.rs +++ b/crates/pg_warren/src/compute_sql/mod.rs @@ -39,7 +39,6 @@ mod ownership; mod product; mod read; mod reduce; -mod settings; mod similarity; mod structure; mod summary; @@ -49,8 +48,7 @@ mod transient; mod vectors; pub(crate) use drop::drop_named; -pub(crate) use error::{compute_sqlstate, entry, ComputeSqlError}; -pub(crate) use settings::{define_compute_settings, workers}; +pub(crate) use error::{entry, ComputeSqlError}; pub(crate) use vectors::{vector_bytes, Vectors}; use warren_core::graph::{RelationIndex, RelationSelection}; diff --git a/crates/pg_warren/src/compute_sql/product.rs b/crates/pg_warren/src/compute_sql/product.rs index 618b2c0b..cd51ab3a 100644 --- a/crates/pg_warren/src/compute_sql/product.rs +++ b/crates/pg_warren/src/compute_sql/product.rs @@ -13,13 +13,13 @@ use warren_core::compute::{ }; use warren_core::graph::RelationSelection; -use super::settings::workers; use super::vectors::Vectors; use super::{entry, ComputeSqlError}; -use crate::cache::resolve; +use crate::cache::{resolve, CachedIds}; +use crate::interrupts::workers; use crate::interrupts::PostgresInterrupts; use crate::selection::relations; -use crate::source::{with_id_type, IdDatum}; +use crate::source::with_id_type; /// Which product a call computes. #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -47,12 +47,12 @@ struct Arguments<'a> { /// `vector` filled with `monoid`'s identity, at `vector`'s dtype and /// shape. -fn identity( +fn identity( vectors: &Vectors, like: &AnyVector, monoid: Monoid, ) -> Result { - fn filled( + fn filled( vectors: &Vectors, width: usize, monoid: Monoid, @@ -73,7 +73,7 @@ fn identity( } /// A copy of `vector`, charged to the call's budget, to write into. -fn copy( +fn copy( vectors: &Vectors, vector: &AnyVector, ) -> Result { @@ -89,7 +89,7 @@ fn copy( impl Arguments<'_> { /// Computes the product into `y`. - fn run( + fn run( &self, mut vectors: Vectors, shape: Shape, diff --git a/crates/pg_warren/src/compute_sql/read.rs b/crates/pg_warren/src/compute_sql/read.rs index 20b4ec61..2f37f19b 100644 --- a/crates/pg_warren/src/compute_sql/read.rs +++ b/crates/pg_warren/src/compute_sql/read.rs @@ -15,9 +15,9 @@ use warren_core::graph::DenseId; use super::vectors::Vectors; use super::{entry, ComputeSqlError}; -use crate::cache::{resolve, LoadedVersion}; +use crate::cache::{resolve, CachedIds, LoadedVersion}; use crate::external_ids::{check_type, external}; -use crate::source::{with_id_type, IdDatum}; +use crate::source::with_id_type; /// Element `index` of `vector` as a `double precision`. pub(super) fn element(vector: &AnyVector, index: usize) -> f64 { @@ -31,7 +31,9 @@ pub(super) fn element(vector: &AnyVector, index: usize) -> f64 { } /// The live nodes of `loaded` with their external ids, in dense id order. -fn live(loaded: Arc>) -> impl Iterator { +fn live( + loaded: Arc>, +) -> impl Iterator { (0..loaded.id_map.id_space()).filter_map(move |node| { match loaded.id_map.tombstones().contains(node) { true => None, @@ -42,7 +44,7 @@ fn live(loaded: Arc>) -> impl Iterator( +fn open( graph: &str, name: &str, version: Option, diff --git a/crates/pg_warren/src/compute_sql/reduce.rs b/crates/pg_warren/src/compute_sql/reduce.rs index 71a8e7cf..84727a81 100644 --- a/crates/pg_warren/src/compute_sql/reduce.rs +++ b/crates/pg_warren/src/compute_sql/reduce.rs @@ -5,11 +5,11 @@ use pgrx::prelude::*; use warren_core::compute::{norm_diff, reduce_live, Norm, Reduction}; use warren_core::version; -use super::settings::workers; use super::vectors::Vectors; use super::{entry, ComputeSqlError}; use crate::cache::resolve; use crate::catalog::compaction_between; +use crate::interrupts::workers; use crate::interrupts::PostgresInterrupts; use crate::source::with_id_type; diff --git a/crates/pg_warren/src/compute_sql/similarity.rs b/crates/pg_warren/src/compute_sql/similarity.rs index 3850c99b..7a21bbf3 100644 --- a/crates/pg_warren/src/compute_sql/similarity.rs +++ b/crates/pg_warren/src/compute_sql/similarity.rs @@ -6,25 +6,25 @@ //! Each returns rows of external ids, computed on the compute pool within //! the call's memory budget, and persists nothing. +use std::borrow::Borrow; use std::mem::size_of; use std::sync::Arc; use pgrx::prelude::*; -use pgrx::spi::SpiTupleTable; use pgrx::AnyElement; use warren_core::compute::{self, ComputeError, ComputeMemory, Mask, Neighbourhood, Similarity}; use warren_core::graph::DenseId; use super::from_query::{check_id_column, id_type_name}; -use super::settings::workers; use super::vectors::Vectors; use super::{entry, selection, ComputeSqlError}; use crate::access::{with_read_only, with_search_path, Caller}; -use crate::cache::{cached, compute_memory, resolve, LoadedVersion}; -use crate::external_ids::{check_type, element, external}; +use crate::cache::{cached, compute_memory, resolve, CachedIds, LoadedVersion}; +use crate::external_ids::{check_type, element, external, DenseIds}; +use crate::interrupts::workers; use crate::interrupts::PostgresInterrupts; use crate::selection::relations; -use crate::source::{describe, for_each_tuple, open_read_only, with_id_type, IdDatum, SourceError}; +use crate::source::{describe, for_each_tuple, open_read_only, with_id_type, Row, SourceError}; /// The similarity `measure` names. fn measure(measure: Option<&str>) -> Result { @@ -102,7 +102,7 @@ fn similarity( /// The pairs of live nodes `query`'s rows name, in its order, and their /// charge. -fn read_pairs( +fn read_pairs( caller: &Caller, query: &str, loaded: &LoadedVersion, @@ -128,7 +128,8 @@ fn read_pairs( format!("SELECT q.u::{id_type}, q.v::{id_type} FROM ({query}\n) AS q(u, v)"); let mut opened = open_read_only::(client, "pairs", &wrapped)?; let mut pairs = Pairs { pairs: Vec::new(), charge: memory.charge(0)?, memory }; - for_each_tuple(&mut opened.cursor, |row| pairs.push(loaded, row))?; + let mut ids = DenseIds::new(&loaded.id_map, memory); + for_each_tuple(&mut opened.cursor, |row| pairs.push(&mut ids, row))?; Ok((pairs.pairs, pairs.charge)) }) }) @@ -144,14 +145,16 @@ struct Pairs<'m> { impl Pairs<'_> { /// Adds the row's pair if both ids are live nodes. - fn push( + fn push( &mut self, - loaded: &LoadedVersion, - row: &SpiTupleTable<'_>, + ids: &mut DenseIds<'_, E>, + row: &Row<'_>, ) -> Result<(), ComputeSqlError> { - let dense = |ordinal| -> Result, ComputeSqlError> { - let id = row.get::(ordinal)?.map(E::from_sql); - Ok(id.and_then(|id| loaded.id_map.dense_id(&id))) + let mut dense = |ordinal| -> Result, ComputeSqlError> { + match row.id::(ordinal)? { + Some(id) => Ok(ids.dense_id(id.borrow())?), + None => Ok(None), + } }; let (Some(u), Some(v)) = (dense(1)?, dense(2)?) else { return Ok(()); @@ -203,7 +206,7 @@ fn most_similar( let rows: Vec<(AnyElement, f64)> = with_id_type!(resolved.graph.id_type, E => { let id = element::(graph, "node", node)?; let loaded = cached::(&resolved)?; - match loaded.id_map.dense_id(&id) { + match loaded.id_map.dense_id(id.borrow()) { None => Vec::new(), Some(dense) => { let selected = relations(&loaded.relations, rels.as_deref()); diff --git a/crates/pg_warren/src/compute_sql/structure.rs b/crates/pg_warren/src/compute_sql/structure.rs index 413ed2e5..e43060ec 100644 --- a/crates/pg_warren/src/compute_sql/structure.rs +++ b/crates/pg_warren/src/compute_sql/structure.rs @@ -11,14 +11,14 @@ use pgrx::prelude::*; use pgrx::AnyElement; use warren_core::compute; -use super::settings::workers; use super::vectors::Vectors; use super::{entry, selection, ComputeSqlError}; -use crate::cache::{cached, compute_memory, resolve, Resolved}; +use crate::cache::{cached, compute_memory, resolve, CachedIds, Resolved}; use crate::external_ids::{check_type, external}; +use crate::interrupts::workers; use crate::interrupts::PostgresInterrupts; use crate::selection::relations; -use crate::source::{with_id_type, IdDatum}; +use crate::source::with_id_type; /// `warren.core_number` (§6.7): vector `out` (`i64`) holding each node's /// core number in the undirected simple graph under the relations `rels` @@ -209,7 +209,7 @@ fn minimum_spanning_forest( } /// The forest's edges as rows of external ids. -fn forest_rows( +fn forest_rows( resolved: &Resolved, rels: Option<&[Option]>, weighted: Option, diff --git a/crates/pg_warren/src/compute_sql/summary.rs b/crates/pg_warren/src/compute_sql/summary.rs index 5962987c..4fd8d4d1 100644 --- a/crates/pg_warren/src/compute_sql/summary.rs +++ b/crates/pg_warren/src/compute_sql/summary.rs @@ -6,11 +6,11 @@ use pgrx::{AnyElement, JsonB}; use serde_json::{json, Value}; use warren_core::compute::{summarise, top_k}; -use super::settings::workers; use super::vectors::Vectors; use super::{entry, ComputeSqlError}; use crate::cache::resolve; use crate::external_ids::{check_type, external}; +use crate::interrupts::workers; use crate::interrupts::PostgresInterrupts; use crate::source::with_id_type; diff --git a/crates/pg_warren/src/compute_sql/vectors.rs b/crates/pg_warren/src/compute_sql/vectors.rs index a7c32b02..3583cc1b 100644 --- a/crates/pg_warren/src/compute_sql/vectors.rs +++ b/crates/pg_warren/src/compute_sql/vectors.rs @@ -18,11 +18,10 @@ use super::ComputeSqlError; use crate::access::require_creator_or_owner; use crate::cache::{ admit_transient, cached, cached_pair, cached_vector, cached_vector_hit, compute_memory, - forget_vector, keep_vector, shared_vector, trim, LoadedVersion, Resolved, + forget_vector, keep_vector, shared_vector, trim, CachedIds, LoadedVersion, Resolved, }; use crate::catalog::{GraphDefinition, VersionId}; use crate::segment::CHUNK_SIZE; -use crate::source::IdDatum; use crate::vector::{ admit_to_quota, drop_transient, keep_transient, transient, transients, write_vector, Binding, SpiVectorStore, VectorError, VectorId, VectorRecord, VectorStore, @@ -101,7 +100,7 @@ struct Read { vector: Arc, } -impl Vectors { +impl Vectors { /// Opens `resolved`'s version and lists its vectors, with a fresh call /// budget of `warren.max_compute_memory` (less what the statement holds /// past the cache's cap, the version included). @@ -333,6 +332,7 @@ mod tests { use pgrx::prelude::*; use pgrx::spi::Error; + use warren_core::graph::FixedKeys; use super::Vectors; use crate::cache::resolve; @@ -349,7 +349,8 @@ mod tests { // p (16,000 bytes) cannot be kept beside the transient vectors. Spi::run("SET LOCAL warren.max_cache_memory = '64kB'")?; let resolved = resolve("a", None, "read").map_err(|_| Error::InvalidPosition)?; - let mut vectors = Vectors::::open(resolved).map_err(|_| Error::InvalidPosition)?; + let mut vectors = + Vectors::>::open(resolved).map_err(|_| Error::InvalidPosition)?; let first = vectors.get("p").map_err(|_| Error::InvalidPosition)?; let second = vectors.get("p").map_err(|_| Error::InvalidPosition)?; assert!(Arc::ptr_eq(&first, &second), "p was loaded again"); diff --git a/crates/pg_warren/src/extension.rs b/crates/pg_warren/src/extension.rs index 6a0d6eb1..3371fa21 100644 --- a/crates/pg_warren/src/extension.rs +++ b/crates/pg_warren/src/extension.rs @@ -1,4 +1,5 @@ -//! Extension identity: the `warren` schema and `warren.version()`. +//! Extension identity: the `warren` schema and `warren.version()`, and the +//! SQLSTATEs of `warren-core`'s graph and compute errors. //! //! The schema is declared once, here, as `#[pg_schema] mod warren`; pgrx emits //! `CREATE SCHEMA IF NOT EXISTS warren` into the extension script, so the @@ -9,6 +10,7 @@ use pgrx::prelude::*; use crate::access::{entry, AccessError}; +use warren_core::compute::{ArithmeticError, ComputeError}; use warren_core::graph::GraphError; /// The `warren` SQL schema. Intentionally empty: it exists so pgrx creates @@ -29,10 +31,62 @@ pub(crate) fn graph_sqlstate(error: &GraphError) -> PgSqlErrorCode { | GraphError::IdSpaceTooLarge | GraphError::RelationNameTooLong => PgSqlErrorCode::ERRCODE_PROGRAM_LIMIT_EXCEEDED, GraphError::UnknownNodeOrder(_) => PgSqlErrorCode::ERRCODE_INVALID_PARAMETER_VALUE, + GraphError::IdEncoding { .. } => PgSqlErrorCode::ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE, _ => PgSqlErrorCode::ERRCODE_DATA_CORRUPTED, } } +/// The SQLSTATE of a kernel error: the one Postgres raises for the same +/// arithmetic, so the compiled path and the executor agree. +pub(crate) fn compute_sqlstate(error: &ComputeError) -> PgSqlErrorCode { + match error { + ComputeError::MemoryLimit { .. } => PgSqlErrorCode::ERRCODE_CONFIGURATION_LIMIT_EXCEEDED, + ComputeError::OutOfMemory { .. } => PgSqlErrorCode::ERRCODE_OUT_OF_MEMORY, + ComputeError::UnknownDtype(_) + | ComputeError::UnknownSemiring(_) + | ComputeError::UnknownBinaryOp(_) + | ComputeError::UnknownReduction(_) + | ComputeError::UnknownNorm(_) + | ComputeError::UnknownCombiner(_) + | ComputeError::UnknownSimilarity(_) + | ComputeError::ZeroWidth + | ComputeError::WidthMismatch { .. } + | ComputeError::Cycle { .. } + | ComputeError::NonPositiveWeight => PgSqlErrorCode::ERRCODE_INVALID_PARAMETER_VALUE, + ComputeError::DtypeMismatch { .. } + | ComputeError::OperandType { .. } + | ComputeError::CombinerType { .. } => PgSqlErrorCode::ERRCODE_DATATYPE_MISMATCH, + ComputeError::IdSpaceMismatch { .. } => { + PgSqlErrorCode::ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE + } + ComputeError::NullResult { .. } => PgSqlErrorCode::ERRCODE_NULL_VALUE_NOT_ALLOWED, + ComputeError::Arithmetic(error) => arithmetic_sqlstate(*error), + ComputeError::Cancelled => PgSqlErrorCode::ERRCODE_QUERY_CANCELED, + ComputeError::VectorBytes { .. } | ComputeError::VectorLength { .. } => { + PgSqlErrorCode::ERRCODE_DATA_CORRUPTED + } + _ => PgSqlErrorCode::ERRCODE_INTERNAL_ERROR, + } +} + +/// The SQLSTATE `float.c` and `int8.c` raise for an arithmetic error. +fn arithmetic_sqlstate(error: ArithmeticError) -> PgSqlErrorCode { + match error { + ArithmeticError::BigintOutOfRange + | ArithmeticError::FloatOverflow + | ArithmeticError::FloatUnderflow => PgSqlErrorCode::ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE, + ArithmeticError::DivisionByZero => PgSqlErrorCode::ERRCODE_DIVISION_BY_ZERO, + ArithmeticError::SqrtOfNegative + | ArithmeticError::ZeroToNegativePower + | ArithmeticError::ComplexPower => { + PgSqlErrorCode::ERRCODE_INVALID_ARGUMENT_FOR_POWER_FUNCTION + } + ArithmeticError::LogOfZero | ArithmeticError::LogOfNegative => { + PgSqlErrorCode::ERRCODE_INVALID_ARGUMENT_FOR_LOG + } + } +} + /// `warren.version()`: the version of the installed extension library. #[pg_extern(schema = "warren", stable, parallel_safe)] pub fn version() -> &'static str { diff --git a/crates/pg_warren/src/external_ids/error.rs b/crates/pg_warren/src/external_ids/error.rs index 1a6f9515..7323e6ce 100644 --- a/crates/pg_warren/src/external_ids/error.rs +++ b/crates/pg_warren/src/external_ids/error.rs @@ -1,11 +1,15 @@ -//! Why an id argument does not fit the graph. +//! Why an id argument does not fit the graph, or a call's ids could not +//! be looked up. use pgrx::PgSqlErrorCode; use thiserror::Error; +use warren_core::graph::GraphError; use crate::access::SqlError; +use crate::extension::graph_sqlstate; -/// Why an id argument does not fit the graph. +/// Why an id argument does not fit the graph, or a call's ids could not be +/// looked up. #[derive(Debug, Error)] pub(crate) enum ExternalIdError { /// An id argument that must be given was NULL. @@ -38,6 +42,9 @@ pub(crate) enum ExternalIdError { /// The argument's type. found: String, }, + /// A call's hash index of the graph's ids could not be built. + #[error(transparent)] + Index(#[from] GraphError), } impl SqlError for ExternalIdError { @@ -47,6 +54,7 @@ impl SqlError for ExternalIdError { Self::IdType { .. } | Self::NotAnArray { .. } => { PgSqlErrorCode::ERRCODE_DATATYPE_MISMATCH } + Self::Index(error) => graph_sqlstate(error), } } } diff --git a/crates/pg_warren/src/external_ids/lookup.rs b/crates/pg_warren/src/external_ids/lookup.rs new file mode 100644 index 00000000..bdf59823 --- /dev/null +++ b/crates/pg_warren/src/external_ids/lookup.rs @@ -0,0 +1,151 @@ +//! External ids mapped in bulk by one call (`vector_from_query`, `gather`'s +//! message queries, `similarity`'s pairs; DESIGN.md §5.7): through the id +//! map's own lookup until the call has mapped [`INDEXED_AFTER`] ids, then, +//! for text ids, whose lookup is a binary search of three cache misses a +//! probe, through a hash index of the version's live ids, built once and +//! charged to `warren.max_compute_memory` before it is built. A call whose +//! compute memory has no room for the index keeps the binary search. + +use warren_core::compute::{ComputeMemory, MemoryCharge}; +use warren_core::graph::{BuildMemory, DenseId, IdMap, IndexedKeys}; + +use super::ExternalIdError; +use crate::source::IdDatum; + +/// The ids a call maps before it builds a hash index: below this a binary +/// search per id costs less than building one (DESIGN.md §5.7). +pub(crate) const INDEXED_AFTER: usize = 65_536; + +/// Maps one call's external ids to dense ids. +pub(crate) struct DenseIds<'m, E: IdDatum> { + id_map: &'m IdMap, + memory: &'m ComputeMemory, + lookup: Lookup<'m, E>, +} + +/// How a call's ids are looked up. +enum Lookup<'m, E: IndexedKeys + 'm> { + /// By the id map's binary search, the ids mapped so far counted. + Searched(usize), + /// Through a hash index, with its charge. + Indexed { index: E::Index<'m>, _charge: MemoryCharge }, + /// By the binary search for the rest of the call: a fixed type, or an + /// index the call's compute memory has no room for. + Unindexed, +} + +impl<'m, E: IdDatum> DenseIds<'m, E> { + /// The lookups of one call into `id_map`, an index charged to `memory`. + pub(crate) fn new(id_map: &'m IdMap, memory: &'m ComputeMemory) -> Self { + Self { id_map, memory, lookup: Lookup::Searched(0) } + } + + /// The dense id of `key` if it is a live node's, as + /// [`IdMap::dense_id`] answers; past [`INDEXED_AFTER`] ids a text id + /// map's hash index answers, built on first use if it fits in the + /// call's compute memory. An error only if building the index fails. + pub(crate) fn dense_id(&mut self, key: &E::Key) -> Result, ExternalIdError> { + if let Lookup::Searched(mapped) = &mut self.lookup { + *mapped += 1; + if *mapped > INDEXED_AFTER { + self.lookup = self.indexed()?; + } + } + Ok(match &self.lookup { + Lookup::Indexed { index, .. } => E::probe(index, key), + Lookup::Searched(_) | Lookup::Unindexed => self.id_map.dense_id(key), + }) + } + + /// The lookup past the threshold: a text id map's hash index, if the + /// call's compute memory has room for it, else the binary search. + fn indexed(&self) -> Result, ExternalIdError> { + let Some(bytes) = E::lookup_index_bytes(self.id_map) else { + return Ok(Lookup::Unindexed); + }; + let Ok(charge) = self.memory.charge(bytes) else { + return Ok(Lookup::Unindexed); + }; + // Charged above, exactly: the index's own budget is unlimited. + let index = E::index(self.id_map, &BuildMemory::unlimited())?; + Ok(Lookup::Indexed { index, _charge: charge }) + } + + /// Whether lookups go through a hash index. + #[cfg(test)] + fn is_indexed(&self) -> bool { + matches!(self.lookup, Lookup::Indexed { .. }) + } + + /// Whether the call gave up on an index. + #[cfg(test)] + fn is_unindexed(&self) -> bool { + matches!(self.lookup, Lookup::Unindexed) + } +} + +#[cfg(test)] +#[allow(clippy::disallowed_methods)] // Test code may assign a transaction id. +mod tests { + use warren_core::graph::{FixedKeys, TextIds, TextIndex, TextKeys}; + + use super::*; + + fn text_map(count: usize) -> IdMap { + let mut ids = TextIds::new(&BuildMemory::unlimited()); + for n in 0..count { + ids.push(format!("node-{n}").as_bytes()).unwrap(); + } + let (first, _) = IdMap::new().rebuild_within(ids).unwrap(); + let mut kept = TextIds::new(&BuildMemory::unlimited()); + for n in (0..count).filter(|n| n % 10 != 3) { + kept.push(format!("node-{n}").as_bytes()).unwrap(); + } + first.rebuild_within(kept).unwrap().0 + } + + #[test] + fn a_call_past_the_threshold_indexes_text_ids_and_answers_as_the_id_map() { + let id_map = text_map(1000); + let memory = ComputeMemory::new(u64::MAX); + let mut ids = DenseIds::new(&id_map, &memory); + for round in 0..=(INDEXED_AFTER / 1000 + 1) { + for n in 0..1001 { + let key = format!("node-{n}"); + let expected = id_map.dense_id(key.as_bytes()); + assert_eq!(ids.dense_id(key.as_bytes()).unwrap(), expected, "{key}, {round}"); + } + } + assert!(ids.is_indexed()); + assert_eq!(memory.in_use(), TextIndex::bytes_for(&id_map)); + drop(ids); + assert_eq!(memory.in_use(), 0); + } + + /// A call whose index does not fit its compute memory keeps the + /// binary search, which needs none, and does not try again. + #[test] + fn an_index_that_does_not_fit_the_call_keeps_the_binary_search() { + let id_map = text_map(10); + let memory = ComputeMemory::new(TextIndex::bytes_for(&id_map) - 1); + let mut ids = DenseIds::new(&id_map, &memory); + for _ in 0..=INDEXED_AFTER + 10 { + assert_eq!(ids.dense_id(b"node-1").unwrap(), Some(1)); + assert_eq!(ids.dense_id(b"node-3").unwrap(), None, "tombstoned"); + } + assert!(!ids.is_indexed()); + assert!(ids.is_unindexed(), "it gave up on the index"); + assert_eq!(memory.in_use(), 0); + } + + #[test] + fn fixed_ids_keep_the_binary_search() { + let id_map = IdMap::>::new().rebuild(vec![5, 7]).unwrap(); + let memory = ComputeMemory::new(0); + let mut ids = DenseIds::new(&id_map, &memory); + for _ in 0..=INDEXED_AFTER { + assert_eq!(ids.dense_id(&7).unwrap(), Some(1)); + } + assert!(!ids.is_indexed()); + } +} diff --git a/crates/pg_warren/src/external_ids/mod.rs b/crates/pg_warren/src/external_ids/mod.rs index e1172bb7..a0ea46c7 100644 --- a/crates/pg_warren/src/external_ids/mod.rs +++ b/crates/pg_warren/src/external_ids/mod.rs @@ -4,17 +4,24 @@ //! (DESIGN.md §5.2, §9.3, §14 item 1). Used by every feature that takes or //! returns node ids (`traversal_sql`, `compute_sql`). //! +//! Calls that map many ids map them through `lookup::DenseIds`, which +//! indexes a text id map once a call has mapped enough of them. +//! //! An external id the version does not know, or knows only as tombstoned, //! is not a node of that version: it is skipped as a start or filter //! member, and a lookup of it finds nothing. NULL ids are skipped too. mod error; +mod lookup; pub(crate) use error::ExternalIdError; +pub(crate) use lookup::DenseIds; + +use std::borrow::Borrow; use pgrx::prelude::*; -use pgrx::{AnyArray, AnyElement, FromDatum, IntoDatum}; -use warren_core::graph::{DenseId, FixedKeys, IdMap}; +use pgrx::{AnyArray, AnyElement, FromDatum}; +use warren_core::graph::{DenseId, IdMap}; use crate::source::{type_name, IdDatum}; @@ -23,29 +30,37 @@ pub(crate) fn element( graph: &str, argument: &'static str, value: AnyElement, -) -> Result { +) -> Result { check_type::(graph, argument, value.oid(), false)?; - // SAFETY: the datum's type was checked above to be E's SQL type. - let sql = unsafe { E::Sql::from_polymorphic_datum(value.datum(), false, value.oid()) }; - sql.map(E::from_sql).ok_or(ExternalIdError::Null(argument)) + owned::(value).ok_or(ExternalIdError::Null(argument)) +} + +/// The id `value` holds, which is of a type `E` reads. +fn owned(value: AnyElement) -> Option { + // SAFETY: the caller checked that the datum's type is one E reads; it + // lives for the call. + unsafe { E::in_place(value.datum(), false) }.map(|id| E::owned(id.borrow())) } -/// [`element`], but an `int2` or `int4` value is also accepted for a -/// wider integer id type: for lookups whose result type does not depend -/// on the argument's, so `warren.degree('g', 42)` works on a `bigint` -/// graph. +/// [`element`], but for lookups whose result type does not depend on the +/// argument's, a narrower type is also accepted: an `int2` or `int4` value +/// for a wider integer id type, so `warren.degree('g', 42)` works on a +/// `bigint` graph, and a `varchar` for `text`. pub(crate) fn element_widened( graph: &str, argument: &'static str, value: AnyElement, -) -> Result { +) -> Result { + if value.oid() != E::ID_TYPE.oid() && E::ID_TYPE.reads(value.oid()) { + return owned::(value).ok_or(ExternalIdError::Null(argument)); + } // SAFETY: each read below is of the datum's own type, `value.oid()`. let narrow = unsafe { match value.oid() { pg_sys::INT2OID => { i16::from_polymorphic_datum(value.datum(), false, value.oid()).map(i64::from) } - pg_sys::INT4OID if value.oid() != E::Sql::type_oid() => { + pg_sys::INT4OID if value.oid() != E::ID_TYPE.oid() => { i32::from_polymorphic_datum(value.datum(), false, value.oid()).map(i64::from) } _ => None, @@ -53,7 +68,7 @@ pub(crate) fn element_widened( }; match narrow.and_then(E::from_integer) { Some(id) => Ok(id), - None => element(graph, argument, value), + None => element::(graph, argument, value), } } @@ -63,52 +78,42 @@ pub(crate) fn elements( graph: &str, argument: &'static str, array: AnyArray, -) -> Result, ExternalIdError> { +) -> Result, ExternalIdError> { check_type::(graph, argument, array.oid(), true)?; - Ok((&array) - .into_iter() - .flatten() - // SAFETY: the element type was checked above to be E's SQL type. - .filter_map(|value| unsafe { - E::Sql::from_polymorphic_datum(value.datum(), false, value.oid()) - }) - .map(E::from_sql) - .collect()) + Ok((&array).into_iter().flatten().filter_map(owned::).collect()) } /// The dense ids of the live nodes among `ids`, in order. -pub(crate) fn dense_ids(id_map: &IdMap>, ids: &[E]) -> Vec { - ids.iter().filter_map(|id| id_map.dense_id(id)).collect() +pub(crate) fn dense_ids(id_map: &IdMap, ids: &[E::Owned]) -> Vec { + ids.iter().filter_map(|id| id_map.dense_id(id.borrow())).collect() } /// The external id of `dense` as a SQL value of the graph's id type. -pub(crate) fn external( - id_map: &IdMap>, - dense: DenseId, -) -> Option { - let id = id_map.external_id(dense)?; - let datum = id.to_sql().into_datum()?; - // SAFETY: the datum was just made from E's SQL type, whose oid is - // E::Sql::type_oid(). - unsafe { AnyElement::from_polymorphic_datum(datum, false, E::Sql::type_oid()) } +pub(crate) fn external(id_map: &IdMap, dense: DenseId) -> Option { + let datum = E::datum(id_map.external_id(dense)?)?; + // SAFETY: the datum was just made of E's id type. + unsafe { AnyElement::from_polymorphic_datum(datum, false, E::ID_TYPE.oid()) } } /// The external ids of `path` as a SQL array of the graph's id type. -pub(crate) fn external_array( - id_map: &IdMap>, - path: &[DenseId], -) -> Option { - let ids: Vec = path - .iter() - .filter_map(|&dense| id_map.external_id(dense)) - .copied() - .map(E::to_sql) - .collect(); - let datum = ids.into_datum()?; - // SAFETY: get_array_type only reads the type cache. - let array_type = unsafe { pg_sys::get_array_type(E::Sql::type_oid()) }; - // SAFETY: the datum was just made as an array of E's SQL type. - unsafe { AnyArray::from_polymorphic_datum(datum, false, array_type) } +pub(crate) fn external_array(id_map: &IdMap, path: &[DenseId]) -> Option { + let mut ids: Vec = + path.iter().filter_map(|&dense| id_map.external_id(dense)).filter_map(E::datum).collect(); + let oid = E::ID_TYPE.oid(); + let (mut len, mut by_value, mut align) = (0, false, 0); + let count = i32::try_from(ids.len()).ok()?; + // SAFETY: get_typlenbyvalalign and get_array_type only read the type + // cache; construct_array copies `count` datums of type `oid`, laid out + // as it says, into a new array, raising an ERROR rather than return + // NULL. + let (array, array_type) = unsafe { + pg_sys::get_typlenbyvalalign(oid, &mut len, &mut by_value, &mut align); + let array = + pg_sys::construct_array(ids.as_mut_ptr(), count, oid, len.into(), by_value, align); + (array, pg_sys::get_array_type(oid)) + }; + // SAFETY: the datum was just made as an array of E's id type. + unsafe { AnyArray::from_polymorphic_datum(pg_sys::Datum::from(array), false, array_type) } } /// The element type of array type `oid`; an error naming `argument` if @@ -137,7 +142,7 @@ pub(crate) fn check_type( oid: pg_sys::Oid, array: bool, ) -> Result<(), ExternalIdError> { - let expected = E::Sql::type_oid(); + let expected = E::ID_TYPE.oid(); let found = match array { true => element_type(argument, oid, &type_name(expected))?, false => oid, diff --git a/crates/pg_warren/src/interrupts.rs b/crates/pg_warren/src/interrupts/mod.rs similarity index 91% rename from crates/pg_warren/src/interrupts.rs rename to crates/pg_warren/src/interrupts/mod.rs index 085854a7..0e0a89fa 100644 --- a/crates/pg_warren/src/interrupts.rs +++ b/crates/pg_warren/src/interrupts/mod.rs @@ -1,7 +1,11 @@ //! The Postgres adapter of `warren-core`'s `Interrupts` port: the poll //! traversals and compute kernels make on the main thread (DESIGN.md §6.9, -//! §9.3). Traversal and compute bindings both use it; it depends on no -//! other feature. +//! §9.3), and the compute pool whose jobs it polls for (`pool`). Builds, +//! traversal and compute bindings use it; it depends on no other feature. + +mod pool; + +pub(crate) use pool::{define_compute_settings, workers}; use warren_core::interrupts::{Cancelled, Interrupts}; diff --git a/crates/pg_warren/src/compute_sql/settings.rs b/crates/pg_warren/src/interrupts/pool.rs similarity index 98% rename from crates/pg_warren/src/compute_sql/settings.rs rename to crates/pg_warren/src/interrupts/pool.rs index c141e115..94a2c5b8 100644 --- a/crates/pg_warren/src/compute_sql/settings.rs +++ b/crates/pg_warren/src/interrupts/pool.rs @@ -43,7 +43,7 @@ pub(crate) fn define_compute_settings() { GucRegistry::define_int_guc( c"warren.threads", c"Threads a compute kernel (spmv, spmm, vector_map, reductions, gather, triangles, \ - eigen, distance) runs on.", + eigen, distance), and a refresh mapping edge endpoints, run on.", c"Defaults to half the cores. Compute threads never call into Postgres.", &THREADS, 1, diff --git a/crates/pg_warren/src/lib.rs b/crates/pg_warren/src/lib.rs index 0d8cfb3d..77eda6ce 100644 --- a/crates/pg_warren/src/lib.rs +++ b/crates/pg_warren/src/lib.rs @@ -37,7 +37,7 @@ pub extern "C-unwind" fn _PG_init() { cache::define_memory_settings(); cache::define_shared_cache(); traversal_sql::define_max_walk_paths(); - compute_sql::define_compute_settings(); + interrupts::define_compute_settings(); } /// Required by `cargo pgrx test`; must be visible at the crate root. diff --git a/crates/pg_warren/src/segment/chunk.rs b/crates/pg_warren/src/segment/chunk.rs index a329df8b..95c73861 100644 --- a/crates/pg_warren/src/segment/chunk.rs +++ b/crates/pg_warren/src/segment/chunk.rs @@ -37,6 +37,10 @@ pub(crate) trait SegmentStore { segment: &str, each: &mut EachChunk<'_, SegmentError>, ) -> Result<(), SegmentError>; + + /// The database encoding (`pg_database.encoding`), which a text id + /// map's `id_bytes` header records and must match (DESIGN.md §5.7). + fn encoding(&self) -> u8; } /// The chunk size of production builds: 8 MiB (DESIGN.md §5.1, §14 item 3). @@ -132,10 +136,21 @@ pub(crate) mod tests { assert_eq!(error.sqlstate(), PgSqlErrorCode::ERRCODE_OUT_OF_MEMORY); } - /// An in-memory [`SegmentStore`]. - #[derive(Debug, Default)] + /// An in-memory [`SegmentStore`], in a database of encoding + /// `encoding` (UTF8, 6, by default). + #[derive(Debug)] pub(crate) struct MemorySegmentStore { pub(crate) chunks: BTreeMap<(VersionId, String, i32), Vec>, + pub(crate) encoding: u8, + } + + /// `PG_UTF8`. + pub(crate) const UTF8: u8 = 6; + + impl Default for MemorySegmentStore { + fn default() -> Self { + Self { chunks: BTreeMap::new(), encoding: UTF8 } + } } impl SegmentStore for MemorySegmentStore { @@ -173,6 +188,10 @@ pub(crate) mod tests { .filter(|((v, s, _), _)| *v == version && s == segment) .try_for_each(|((_, _, chunk_no), data)| each(*chunk_no, data)) } + + fn encoding(&self) -> u8 { + self.encoding + } } fn chunk_sizes(store: &MemorySegmentStore, version: VersionId) -> Vec<(i32, usize)> { diff --git a/crates/pg_warren/src/segment/id_map.rs b/crates/pg_warren/src/segment/id_map.rs new file mode 100644 index 00000000..f8a7b715 --- /dev/null +++ b/crates/pg_warren/src/segment/id_map.rs @@ -0,0 +1,159 @@ +//! How a version's id map is stored (DESIGN.md §5.3, §5.7): a fixed-width +//! id type's as one `id_map` segment of pairs, a text id map's as its +//! `id_bytes` (recording the database encoding), `id_offsets` and +//! `id_order`. The tombstones are a segment of their own either way. + +use warren_core::graph::{ + BuildCharge, BuildMemory, ExternalId, FixedKeys, IdKeys, IdMap, TextIdPart, TextKeys, +}; + +use super::{ + read_segment_within, SegmentError, SegmentStore, SegmentWriter, ID_BYTES, ID_MAP, ID_OFFSETS, + ID_ORDER, +}; +use crate::catalog::VersionId; + +/// A key storage whose id map has segments. +pub(crate) trait StoredIds: IdKeys + Sized { + /// Writes the segments of `id_map`, without its tombstones. + fn write_id_map( + id_map: &IdMap, + writer: &mut SegmentWriter<'_, S>, + ) -> Result<(), SegmentError>; + + /// The id map of `version`, decoded and validated with `tombstones`, + /// its encoded tombstones; its segments, their decoding and what it + /// holds are charged to `memory`. Returns the charge for what it holds. + fn read_id_map( + segments: &mut impl SegmentStore, + version: VersionId, + tombstones: &[u8], + memory: &BuildMemory, + ) -> Result<(IdMap, BuildCharge), SegmentError>; +} + +impl StoredIds for FixedKeys { + fn write_id_map( + id_map: &IdMap, + writer: &mut SegmentWriter<'_, S>, + ) -> Result<(), SegmentError> { + writer.chunked_with_memory(ID_MAP, |chunk_size, memory, each| { + id_map.encode_chunked(chunk_size, memory, each) + }) + } + + fn read_id_map( + segments: &mut impl SegmentStore, + version: VersionId, + tombstones: &[u8], + memory: &BuildMemory, + ) -> Result<(IdMap, BuildCharge), SegmentError> { + let (id_map, _id_map_charge) = read_segment_within(segments, version, ID_MAP, memory)?; + Ok(IdMap::decode_within(&id_map, tombstones, memory)?) + } +} + +impl StoredIds for TextKeys { + fn write_id_map( + id_map: &IdMap, + writer: &mut SegmentWriter<'_, S>, + ) -> Result<(), SegmentError> { + let encoding = writer.encoding(); + for (name, part) in [ + (ID_BYTES, TextIdPart::IdBytes), + (ID_OFFSETS, TextIdPart::IdOffsets), + (ID_ORDER, TextIdPart::IdOrder), + ] { + writer.chunked_with_memory(name, |chunk_size, memory, each| { + id_map.encode_text_part(part, encoding, chunk_size, memory, each) + })?; + } + Ok(()) + } + + fn read_id_map( + segments: &mut impl SegmentStore, + version: VersionId, + tombstones: &[u8], + memory: &BuildMemory, + ) -> Result<(IdMap, BuildCharge), SegmentError> { + let encoding = segments.encoding(); + let (bytes, _bytes_charge) = read_segment_within(segments, version, ID_BYTES, memory)?; + let (offsets, _offsets_charge) = + read_segment_within(segments, version, ID_OFFSETS, memory)?; + let (order, _order_charge) = read_segment_within(segments, version, ID_ORDER, memory)?; + let decoded = + IdMap::decode_text_within(&bytes, &offsets, &order, tombstones, encoding, memory)?; + Ok(decoded) + } +} + +#[cfg(test)] +#[allow(clippy::disallowed_methods)] // Test code may assign a transaction id. +mod tests { + use warren_core::graph::{GraphError, TextIds}; + + use super::*; + use crate::segment::{MemorySegmentStore, TOMBSTONES}; + + fn text_map(ids: &[&[u8]]) -> IdMap { + let mut text = TextIds::new(&BuildMemory::unlimited()); + for id in ids { + text.push(id).unwrap(); + } + IdMap::new().rebuild_within(text).unwrap().0 + } + + fn written(id_map: &IdMap, store: &mut MemorySegmentStore) -> Vec { + let memory = BuildMemory::unlimited(); + let mut writer = SegmentWriter::new(store, 1, 7, &memory).unwrap(); + K::write_id_map(id_map, &mut writer).unwrap(); + let tombstones = id_map.tombstones().encode().unwrap(); + writer + .chunked_with_memory(TOMBSTONES, |_, _, each| { + each(&tombstones)?; + Ok(tombstones.len() as u64) + }) + .unwrap(); + tombstones + } + + #[test] + fn a_text_id_map_is_three_segments_that_read_back() { + let id_map = text_map(&[b"b", b"", b"https://example.org/a"]); + let mut store = MemorySegmentStore::default(); + let tombstones = written(&id_map, &mut store); + let names: std::collections::BTreeSet<_> = + store.chunks.keys().map(|(_, name, _)| name.as_str()).collect(); + assert_eq!(names, [ID_BYTES, ID_OFFSETS, ID_ORDER, TOMBSTONES].into()); + let memory = BuildMemory::unlimited(); + let (read, charge) = TextKeys::read_id_map(&mut store, 1, &tombstones, &memory).unwrap(); + assert_eq!(read, id_map); + assert_eq!(charge.bytes(), id_map.bytes()); + } + + #[test] + fn a_text_id_map_of_another_encoding_is_refused() { + let id_map = text_map(&[b"a"]); + let mut store = MemorySegmentStore { encoding: 8, ..Default::default() }; + let tombstones = written(&id_map, &mut store); + store.encoding = 6; + let memory = BuildMemory::unlimited(); + let error = TextKeys::read_id_map(&mut store, 1, &tombstones, &memory).unwrap_err(); + assert!( + matches!(error, SegmentError::Graph(GraphError::IdEncoding { expected: 6, found: 8 })), + "{error:?}" + ); + } + + #[test] + fn a_fixed_id_map_is_one_segment_that_reads_back() { + let id_map = IdMap::>::new().rebuild(vec![3, 1, 2]).unwrap(); + let mut store = MemorySegmentStore::default(); + let tombstones = written(&id_map, &mut store); + assert!(store.chunks.keys().any(|(_, name, _)| name == ID_MAP)); + let memory = BuildMemory::unlimited(); + let (read, _) = FixedKeys::read_id_map(&mut store, 1, &tombstones, &memory).unwrap(); + assert_eq!(read, id_map); + } +} diff --git a/crates/pg_warren/src/segment/layout.rs b/crates/pg_warren/src/segment/layout.rs index 44581273..4d3e9cd0 100644 --- a/crates/pg_warren/src/segment/layout.rs +++ b/crates/pg_warren/src/segment/layout.rs @@ -1,5 +1,5 @@ -//! The segment layout of a version (DESIGN.md §5.3): `id_map`, -//! `tombstones`, `rel_dict`, and the `offsets`/`targets`/`weights` of each +//! The segment layout of a version (DESIGN.md §5.3): its id map's segments +//! (`id_map`, or a text id map's three, §5.7), `tombstones`, `rel_dict`, and the `offsets`/`targets`/`weights` of each //! relation's forward and (if built) reverse CSR, plus the merged CSR's //! unless there is exactly one relation; and a contraction's `membership` //! (§6.6). `cache` reads the same layout. @@ -7,22 +7,22 @@ //! not the segment. use warren_core::graph::{ - AnyCsr, BuildCharge, BuildMemory, BuiltCsr, ChunkSink, CsrSink, ExternalId, FixedKeys, - GraphCsr, IdMap, Membership, RelationCsr, RelationDictionary, + AnyCsr, BuildCharge, BuildMemory, BuiltCsr, ChunkSink, CsrSink, GraphCsr, IdMap, Membership, + RelationCsr, RelationDictionary, }; use super::{ - csr_segment, write_segment, CsrPart, CsrScope, Direction, SegmentError, SegmentStore, ID_MAP, - MEMBERSHIP, RELATION_DICTIONARY, TOMBSTONES, + csr_segment, write_segment, CsrPart, CsrScope, Direction, SegmentError, SegmentStore, + StoredIds, MEMBERSHIP, RELATION_DICTIONARY, TOMBSTONES, }; use crate::catalog::VersionId; /// Writes every segment of a version in chunks of `chunk_size` bytes, and /// returns their total size. -pub(crate) fn write_version( +pub(crate) fn write_version( segments: &mut impl SegmentStore, version: VersionId, - id_map: &IdMap>, + id_map: &IdMap, relations: &RelationDictionary, graph: &GraphCsr, chunk_size: usize, @@ -83,14 +83,19 @@ impl<'s, S: SegmentStore> SegmentWriter<'s, S> { self.bytes } - /// The `id_map`, `tombstones` and `rel_dict` segments. - pub(crate) fn write_identity( + /// The database encoding of the store it writes to. + pub(crate) fn encoding(&self) -> u8 { + self.segments.encoding() + } + + /// The id map's segments, `tombstones` and `rel_dict`. + pub(crate) fn write_identity( &mut self, - id_map: &IdMap>, + id_map: &IdMap, relations: &RelationDictionary, ) -> Result<(), SegmentError> { let (chunk_size, memory) = (self.chunk_size, self.memory.clone()); - self.chunked(ID_MAP, |each| id_map.encode_chunked(chunk_size, &memory, each))?; + K::write_id_map(id_map, self)?; let tombstones = id_map.tombstones(); self.chunked(TOMBSTONES, |each| tombstones.encode_chunked(chunk_size, &memory, each))?; let _encoded_charge = memory.charge(relations.encoded_bytes())?; @@ -136,6 +141,21 @@ impl<'s, S: SegmentStore> SegmentWriter<'s, S> { Ok(()) } + /// [`Self::chunked`], passing `encode` the chunk size and the memory to + /// charge its chunk buffer to. + pub(crate) fn chunked_with_memory( + &mut self, + name: &str, + encode: impl FnOnce( + usize, + &BuildMemory, + &mut ChunkSink<'_, SegmentError>, + ) -> Result, + ) -> Result<(), SegmentError> { + let (chunk_size, memory) = (self.chunk_size, self.memory.clone()); + self.chunked(name, |each| encode(chunk_size, &memory, each)) + } + /// Stores the chunks `encode` produces as segment `name`, numbered from /// 0, and counts its bytes. fn chunked( diff --git a/crates/pg_warren/src/segment/mod.rs b/crates/pg_warren/src/segment/mod.rs index 2086b913..7fcad80c 100644 --- a/crates/pg_warren/src/segment/mod.rs +++ b/crates/pg_warren/src/segment/mod.rs @@ -8,6 +8,7 @@ mod chunk; mod error; +mod id_map; mod layout; mod name; mod spi; @@ -16,9 +17,11 @@ pub(crate) use chunk::{ read_segment, read_segment_within, write_segment, EachChunk, SegmentStore, CHUNK_SIZE, }; pub(crate) use error::SegmentError; +pub(crate) use id_map::StoredIds; pub(crate) use layout::{write_version, writer_memory, SegmentWriter}; pub(crate) use name::{ - csr_segment, CsrPart, CsrScope, Direction, ID_MAP, MEMBERSHIP, RELATION_DICTIONARY, TOMBSTONES, + csr_segment, CsrPart, CsrScope, Direction, ID_BYTES, ID_MAP, ID_OFFSETS, ID_ORDER, MEMBERSHIP, + RELATION_DICTIONARY, TOMBSTONES, }; pub(crate) use spi::{each_chunk, SpiSegmentStore}; diff --git a/crates/pg_warren/src/segment/name.rs b/crates/pg_warren/src/segment/name.rs index 2da31d12..c64c0d00 100644 --- a/crates/pg_warren/src/segment/name.rs +++ b/crates/pg_warren/src/segment/name.rs @@ -4,6 +4,12 @@ pub(crate) use warren_core::graph::{CsrDirection as Direction, CsrPart, CsrScope /// The `id_map` segment. pub(crate) const ID_MAP: &str = "id_map"; +/// The `id_bytes` segment of a text id map (DESIGN.md §5.7). +pub(crate) const ID_BYTES: &str = "id_bytes"; +/// The `id_offsets` segment of a text id map. +pub(crate) const ID_OFFSETS: &str = "id_offsets"; +/// The `id_order` segment of a text id map. +pub(crate) const ID_ORDER: &str = "id_order"; /// The `tombstones` segment. pub(crate) const TOMBSTONES: &str = "tombstones"; /// The `rel_dict` segment. diff --git a/crates/pg_warren/src/segment/spi.rs b/crates/pg_warren/src/segment/spi.rs index dd8225a4..efce6d9a 100644 --- a/crates/pg_warren/src/segment/spi.rs +++ b/crates/pg_warren/src/segment/spi.rs @@ -66,6 +66,12 @@ impl SegmentStore for SpiSegmentStore { ) }) } + + fn encoding(&self) -> u8 { + // SAFETY: reads the backend's database encoding, set at connection. + let encoding = unsafe { pg_sys::GetDatabaseEncoding() }; + u8::try_from(encoding).unwrap_or(u8::MAX) + } } /// Runs `query` (rows of chunk number and bytes, in chunk order) through a diff --git a/crates/pg_warren/src/source/columns.rs b/crates/pg_warren/src/source/columns.rs index e86d853d..e159dfb5 100644 --- a/crates/pg_warren/src/source/columns.rs +++ b/crates/pg_warren/src/source/columns.rs @@ -14,6 +14,10 @@ pub(crate) struct ColumnType { pub(crate) oid: pg_sys::Oid, /// The type's SQL name, as `format_type` prints it. pub(crate) name: String, + /// The name of the column's collation if it is nondeterministic + /// (equality that is not bytewise, DESIGN.md §5.7); `None` for a + /// deterministic one or a type without collation. + pub(crate) nondeterministic: Option, } /// Why a source query does not fit its graph. @@ -43,6 +47,43 @@ pub(crate) enum SourceError { /// The column's type. found: String, }, + /// An id column of a text-like type whose equality is not bytewise, or + /// that cannot hold every id. + #[error( + "{query} column {column} ({role}) is of type {found}, which node ids cannot have: \ + {why}; cast it to text ({cast})" + )] + IdTypeRefused { + /// `node_query` or `edge_query`. + query: &'static str, + /// The 1-based column number. + column: usize, + /// What the column holds. + role: &'static str, + /// The column's type. + found: String, + /// Why the type is refused. + why: &'static str, + /// The cast to write. + cast: &'static str, + }, + /// A text id column with a nondeterministic collation, whose equality + /// bytewise ids cannot follow (DESIGN.md §5.7). + #[error( + "{query} column {column} ({role}) has the nondeterministic collation \"{collation}\", \ + whose equality node ids cannot follow (they compare as bytes): write the column \ + COLLATE \"C\" for exact ids, or normalise them in the query (for example lower(id))" + )] + NondeterministicCollation { + /// `node_query` or `edge_query`. + query: &'static str, + /// The 1-based column number. + column: usize, + /// What the column holds. + role: &'static str, + /// The collation's name. + collation: String, + }, /// The query is not one SELECT (or VALUES, or TABLE). #[error("{query} must be a single SELECT (or VALUES, or TABLE)")] NotASelect { @@ -60,9 +101,10 @@ pub(crate) enum SourceError { impl SqlError for SourceError { fn sqlstate(&self) -> PgSqlErrorCode { match self { - Self::ColumnCount { .. } | Self::ColumnType { .. } => { - PgSqlErrorCode::ERRCODE_DATATYPE_MISMATCH - } + Self::ColumnCount { .. } + | Self::ColumnType { .. } + | Self::IdTypeRefused { .. } + | Self::NondeterministicCollation { .. } => PgSqlErrorCode::ERRCODE_DATATYPE_MISMATCH, Self::NulByte { .. } => PgSqlErrorCode::ERRCODE_INVALID_PARAMETER_VALUE, Self::NotASelect { .. } => PgSqlErrorCode::ERRCODE_SYNTAX_ERROR, } @@ -70,7 +112,8 @@ impl SqlError for SourceError { } /// The one column a node query must return: its node ids, whose type is the -/// graph's id type (`integer`, `bigint` or `uuid`). Returns that id type. +/// graph's id type (`integer`, `bigint`, `uuid` or `text`, which `varchar` +/// is read as), with a deterministic collation. Returns that id type. pub(crate) fn check_node_columns(columns: &[ColumnType]) -> Result { let [id] = columns else { return Err(SourceError::ColumnCount { @@ -79,8 +122,55 @@ pub(crate) fn check_node_columns(columns: &[ColumnType]) -> Result Result<(), SourceError> { + let unqualified = column.name.rsplit('.').next().unwrap_or_default(); + let (why, cast) = match column.oid { + pg_sys::BPCHAROID => ("its equality ignores trailing spaces", "id::text"), + pg_sys::NAMEOID => ("it is a system identifier type, cut at 63 bytes", "id::text"), + _ if unqualified == "citext" => { + ("its equality ignores case", "id::text for exact ids, or lower(id)::text") + } + _ => return Ok(()), + }; + Err(SourceError::IdTypeRefused { + query: at.query, + column: at.number, + role: at.role, + found: column.name.clone(), + why, + cast, + }) +} + +/// Refuses an id column with a nondeterministic collation. +fn require_deterministic(at: Column, column: &ColumnType) -> Result<(), SourceError> { + match &column.nondeterministic { + None => Ok(()), + Some(collation) => Err(SourceError::NondeterministicCollation { + query: at.query, + column: at.number, + role: at.role, + collation: collation.clone(), + }), + } } /// Checks that a node query returns the node ids of a graph whose id type @@ -92,15 +182,17 @@ pub(crate) fn require_node_id_type( let found = check_node_columns(columns)?; match columns { [id] if found != id_type => { - Err(mistyped("node_query", 1, "node id", id_type.sql_name(), id)) + let at = Column { query: "node_query", number: 1, role: "node id" }; + Err(mistyped(at, id_type.sql_name(), id)) } _ => Ok(()), } } /// The columns an edge query must return: source and target ids of the -/// graph's id type, then a `text` relation if the graph has relations, then -/// a `float8` weight if it is weighted. +/// graph's id type (for `text`, `text` or `varchar`, with a deterministic +/// collation), then a `text` relation if the graph has relations, then a +/// `float8` weight if it is weighted. pub(crate) fn check_edge_columns( columns: &[ColumnType], id_type: IdType, @@ -123,21 +215,33 @@ pub(crate) fn check_edge_columns( }); } for (number, (column, (role, (oid, name)))) in (1..).zip(columns.iter().zip(expected)) { - if column.oid != oid { - return Err(mistyped("edge_query", number, role, name, column)); + let at = Column { query: "edge_query", number, role }; + let endpoint = number <= 2; + if endpoint && id_type == IdType::Text { + refuse_text_like(at, column)?; + } + let fits = match endpoint { + true => id_type.reads(column.oid), + false => column.oid == oid, + }; + if !fits { + return Err(mistyped(at, name, column)); + } + if endpoint { + require_deterministic(at, column)?; } } Ok(()) } -fn mistyped( - query: &'static str, - column: usize, - role: &'static str, - expected: &'static str, - found: &ColumnType, -) -> SourceError { - SourceError::ColumnType { query, column, role, expected, found: found.name.clone() } +fn mistyped(at: Column, expected: &'static str, found: &ColumnType) -> SourceError { + SourceError::ColumnType { + query: at.query, + column: at.number, + role: at.role, + expected, + found: found.name.clone(), + } } #[cfg(test)] @@ -146,7 +250,19 @@ mod tests { use super::*; fn column(oid: pg_sys::Oid, name: &str) -> ColumnType { - ColumnType { oid, name: name.to_owned() } + ColumnType { oid, name: name.to_owned(), nondeterministic: None } + } + + fn text() -> ColumnType { + column(pg_sys::TEXTOID, "text") + } + + fn varchar() -> ColumnType { + column(pg_sys::VARCHAROID, "character varying") + } + + fn case_insensitive(column: ColumnType) -> ColumnType { + ColumnType { nondeterministic: Some("und-ci".to_owned()), ..column } } fn int8() -> ColumnType { @@ -170,16 +286,96 @@ mod tests { check_node_columns(&[int8(), int8()]), Err(SourceError::ColumnCount { query: "node_query", expected: "1", found: 2 }) ); + assert_eq!(check_node_columns(&[text()]), Ok(IdType::Text)); + assert_eq!(check_node_columns(&[varchar()]), Ok(IdType::Text)); assert_eq!( - check_node_columns(&[column(pg_sys::TEXTOID, "text")]), + check_node_columns(&[column(pg_sys::FLOAT8OID, "double precision")]), Err(SourceError::ColumnType { query: "node_query", column: 1, role: "node id", - expected: "integer, bigint or uuid", - found: "text".to_owned(), + expected: "integer, bigint, uuid or text", + found: "double precision".to_owned(), + }) + ); + } + + #[test] + fn text_like_types_whose_equality_is_not_bytewise_are_refused_with_their_cast() { + let bpchar = column(pg_sys::BPCHAROID, "character(8)"); + let name = column(pg_sys::NAMEOID, "name"); + let citext = column(pg_sys::Oid::from(90_000), "public.citext"); + for (refused, cast) in [(&bpchar, "id::text"), (&name, "id::text")] { + let error = check_node_columns(std::slice::from_ref(refused)).unwrap_err(); + assert!(matches!(error, SourceError::IdTypeRefused { cast: c, .. } if c == cast)); + } + let error = check_node_columns(std::slice::from_ref(&citext)).unwrap_err(); + assert!(error.to_string().contains("ignores case"), "{error}"); + assert!(error.to_string().contains("lower(id)::text"), "{error}"); + let error = check_edge_columns(&[text(), bpchar], IdType::Text, false, false).unwrap_err(); + assert!(matches!(error, SourceError::IdTypeRefused { column: 2, .. }), "{error:?}"); + assert_eq!(error.sqlstate(), PgSqlErrorCode::ERRCODE_DATATYPE_MISMATCH); + } + + /// A text-like endpoint of a graph whose ids are not text is of the + /// wrong type, with no hint to cast it to text. + #[test] + fn a_text_like_endpoint_of_a_bigint_graph_is_mistyped() { + let bpchar = column(pg_sys::BPCHAROID, "character(5)"); + assert_eq!( + check_edge_columns(&[int8(), bpchar], IdType::Int8, false, false), + Err(SourceError::ColumnType { + query: "edge_query", + column: 2, + role: "target id", + expected: "bigint", + found: "character(5)".to_owned(), + }) + ); + } + + #[test] + fn a_text_id_column_with_a_nondeterministic_collation_is_refused() { + let error = check_node_columns(&[case_insensitive(text())]).unwrap_err(); + assert_eq!( + error, + SourceError::NondeterministicCollation { + query: "node_query", + column: 1, + role: "node id", + collation: "und-ci".to_owned(), + } + ); + assert!(error.to_string().contains("COLLATE \"C\""), "{error}"); + assert_eq!(error.sqlstate(), PgSqlErrorCode::ERRCODE_DATATYPE_MISMATCH); + let edges = [text(), case_insensitive(varchar())]; + assert!(matches!( + check_edge_columns(&edges, IdType::Text, false, false), + Err(SourceError::NondeterministicCollation { query: "edge_query", column: 2, .. }) + )); + // A relation's collation is not an id's. + let relation = case_insensitive(text()); + assert_eq!( + check_edge_columns(&[text(), text(), relation], IdType::Text, true, false), + Ok(()) + ); + } + + #[test] + fn text_and_varchar_endpoints_mix_on_a_text_graph() { + assert_eq!(check_edge_columns(&[text(), varchar()], IdType::Text, false, false), Ok(())); + assert_eq!(require_node_id_type(&[varchar()], IdType::Text), Ok(())); + assert_eq!( + check_edge_columns(&[text(), int8()], IdType::Text, false, false), + Err(SourceError::ColumnType { + query: "edge_query", + column: 2, + role: "target id", + expected: "text", + found: "bigint".to_owned(), }) ); + assert!(check_edge_columns(&[varchar(), varchar()], IdType::Int8, false, false).is_err()); } #[test] diff --git a/crates/pg_warren/src/source/cursor.rs b/crates/pg_warren/src/source/cursor.rs index 7f1fd2da..add40aeb 100644 --- a/crates/pg_warren/src/source/cursor.rs +++ b/crates/pg_warren/src/source/cursor.rs @@ -7,14 +7,18 @@ use std::ffi::{CStr, CString}; use std::ptr; use pgrx::prelude::*; -use pgrx::spi::{SpiClient, SpiCursor, SpiHeapTupleData, SpiTupleTable}; +use pgrx::spi::{SpiClient, SpiCursor, SpiTupleTable}; use pgrx::{PgList, PgMemoryContexts, PgTupleDesc}; -use super::{ColumnType, SourceError}; +use super::{ColumnType, IdDatum, SourceError}; /// Rows fetched from a cursor at a time. const BATCH_ROWS: std::ffi::c_long = 10_000; +/// The bytes the rows of a batch may hold in their memory context (what +/// reading them detoasts) before it is reset, within the batch. +const RESET_BYTES: u64 = 64 << 20; + /// Runs `f` under one snapshot, taken after a command counter increment so /// that it sees every earlier write of this transaction, including the /// current statement's. Read-only cursors opened inside all use it. @@ -156,10 +160,32 @@ unsafe fn column_types(descriptor: pg_sys::TupleDesc) -> Vec { .map(|attribute| ColumnType { oid: attribute.atttypid, name: type_name(attribute.atttypid), + nondeterministic: nondeterministic(attribute.attcollation), }) .collect() } +/// The name of `collation` if it is a nondeterministic collation. +fn nondeterministic(collation: pg_sys::Oid) -> Option { + if collation == pg_sys::Oid::INVALID { + return None; + } + // SAFETY: a valid collation oid, from a tuple descriptor; both calls + // read the syscache and raise an ERROR for an unknown oid rather than + // return a bad value. get_collation_name returns NULL or a palloc'd + // NUL-terminated string. + unsafe { + if pg_sys::get_collation_isdeterministic(collation) { + return None; + } + let name = pg_sys::get_collation_name(collation); + Some(match name.is_null() { + true => collation.to_u32().to_string(), + false => CStr::from_ptr(name).to_string_lossy().into_owned(), + }) + } +} + /// A type's SQL name, as `format_type` prints it. pub(crate) fn type_name(oid: pg_sys::Oid) -> String { // SAFETY: format_type_be returns a palloc'd NUL-terminated string for @@ -168,50 +194,36 @@ pub(crate) fn type_name(oid: pg_sys::Oid) -> String { } /// Passes every remaining row of `cursor` to `each`, a batch at a time, -/// checking for interrupts between batches and freeing each batch after it. -pub(crate) fn for_each_row>( - cursor: &mut SpiCursor<'_>, - mut each: impl FnMut(&SpiHeapTupleData<'_>) -> Result<(), E>, -) -> Result<(), E> { - loop { - check_for_interrupts!(); - let batch = cursor.fetch(BATCH_ROWS)?; - if batch.is_empty() { - return Ok(()); - } - for row in batch { - each(&row)?; - } - // SAFETY: SPI_tuptable is the batch just consumed, which nothing - // refers to any more. - unsafe { pg_sys::SPI_freetuptable(pg_sys::SPI_tuptable) }; - } -} - -/// [`for_each_row`], passing each row as its batch's tuple table positioned -/// at it, so that [`text_in_place`] can read a text column without the copy -/// pgrx's typed getters make into a memory context that lives until SPI -/// finishes (one allocation per row, held for the whole read). Each batch -/// runs in a memory context reset after it, which holds whatever reading a -/// row detoasts. +/// checking for interrupts between batches and freeing each batch after +/// it. Each row is a [`Row`] of its batch's tuple table, so that its id +/// and text columns can be read in place, without the copy pgrx's typed +/// getters make into a memory context that lives until SPI finishes (one +/// allocation per row, held for the whole read). Rows are read in a memory +/// context of their own, which holds whatever reading them detoasts: it is +/// reset after each batch, and within one once it holds [`RESET_BYTES`]. pub(crate) fn for_each_tuple>( cursor: &mut SpiCursor<'_>, - mut each: impl FnMut(&SpiTupleTable<'_>) -> Result<(), E>, + mut each: impl FnMut(&Row<'_>) -> Result<(), E>, ) -> Result<(), E> { let mut rows = PgMemoryContexts::new("warren rows"); + let context = rows.value(); loop { check_for_interrupts!(); let mut batch = cursor.fetch(BATCH_ROWS)?; if batch.is_empty() { return Ok(()); } - // SAFETY: `rows` is a context this function owns; what is allocated - // in it while the batch is read is used only by `each`, which - // copies what it keeps, before the reset below. + // SAFETY: `rows` is a context this function owns. What is allocated + // in it while a row is read is reachable only through that `Row`, + // whose borrow ends with the call of `each` that it is passed to, + // before the context is reset, here or below. let read = unsafe { rows.switch_to(|_| { while batch.next().is_some() { - each(&batch)?; + each(&Row { table: &batch })?; + if pg_sys::MemoryContextMemAllocated(context, true) as u64 > RESET_BYTES { + pg_sys::MemoryContextReset(context); + } } Ok::<_, E>(()) }) @@ -225,18 +237,108 @@ pub(crate) fn for_each_tuple>( } } -/// The `text` in column `ordinal` of the row `table` is at, read in place -/// from the tuple (or, if stored out of line or compressed, detoasted into -/// the current memory context), without a copy. -pub(crate) fn text_in_place<'t>( - table: &'t SpiTupleTable<'_>, - ordinal: usize, -) -> Result, pgrx::spi::Error> { - let Some(datum) = table.get_datum_by_ordinal(ordinal)? else { - return Ok(None); - }; - // SAFETY: `datum` is the non-NULL `text` (the caller checked the - // column's type) of the tuple `table` is at, which lives until the tuple - // table is freed, so for as long as `table` is borrowed. - Ok(unsafe { <&str as FromDatum>::from_polymorphic_datum(datum, false, pg_sys::TEXTOID) }) +/// A row of a cursor, as [`for_each_tuple`] passes it: its batch's tuple +/// table positioned at it, for one call. What is read from it in place +/// borrows the row, so it cannot outlive the call, after which the memory +/// context that holds what reading it detoasted may be reset. Its typed +/// getters (through `Deref`) copy what they return. +pub(crate) struct Row<'r> { + table: &'r SpiTupleTable<'r>, +} + +impl<'r> Row<'r> { + /// The external id in column `ordinal` (of a type `E` reads), read in + /// place as [`Row::text`] reads a `text`. + pub(crate) fn id( + &self, + ordinal: usize, + ) -> Result>, pgrx::spi::Error> { + let Some(datum) = self.table.get_datum_by_ordinal(ordinal)? else { + return Ok(None); + }; + // SAFETY: `datum` is the non-NULL id (the caller checked the + // column's type) of the tuple the table is at, which lives until the + // tuple table is freed; detoasted, it lives in the rows' memory + // context, which is not reset while the row is borrowed. + Ok(unsafe { E::in_place(datum, false) }) + } + + /// The `text` in column `ordinal`, read in place from the tuple (or, if + /// stored out of line or compressed, detoasted into the rows' memory + /// context), without a copy. + pub(crate) fn text(&self, ordinal: usize) -> Result, pgrx::spi::Error> { + let Some(datum) = self.table.get_datum_by_ordinal(ordinal)? else { + return Ok(None); + }; + // SAFETY: as for `id`: the non-NULL `text` of the tuple, which lives + // while the row is borrowed. + Ok(unsafe { <&str as FromDatum>::from_polymorphic_datum(datum, false, pg_sys::TEXTOID) }) + } +} + +impl<'r> std::ops::Deref for Row<'r> { + type Target = SpiTupleTable<'r>; + + fn deref(&self) -> &SpiTupleTable<'r> { + self.table + } +} + +#[cfg(any(test, feature = "pg_test"))] +#[pgrx::pg_schema] +#[allow(clippy::disallowed_methods)] // Test code may assign a transaction id. +mod tests { + use pgrx::prelude::*; + + use super::*; + + /// Reading a cursor in a test failed. + #[derive(Debug)] + struct ReadError; + + impl From for ReadError { + fn from(_: SourceError) -> Self { + Self + } + } + + impl From for ReadError { + fn from(_: pgrx::spi::Error) -> Self { + Self + } + } + + /// Ids detoasted while a batch is read are freed as it is read, not + /// only once it ends: 100 rows of a compressed 1 MB text, all in one + /// batch, never hold more than the reset threshold and a row. + #[pg_test] + fn detoasted_rows_are_freed_within_a_batch() -> Result<(), pgrx::spi::Error> { + Spi::run( + "CREATE TABLE big_ids AS \ + SELECT repeat(md5(n::text), 32768) AS id FROM generate_series(1, 100) n", + )?; + let read = with_pinned_snapshot(|| { + Spi::connect(|client| { + let mut opened = + open_read_only::(client, "query", "SELECT id FROM big_ids")?; + let (mut rows, mut most) = (0, 0); + for_each_tuple::(&mut opened.cursor, |row| { + assert_eq!(row.text(1)?.map(str::len), Some(1 << 20)); + // SAFETY: reads the size of the current context. + let held = unsafe { + pg_sys::MemoryContextMemAllocated(pg_sys::CurrentMemoryContext, true) + }; + rows += 1; + most = most.max(held); + Ok(()) + })?; + Ok::<_, ReadError>((rows, most)) + }) + }); + assert!(read.is_ok(), "{read:?}"); + let (rows, most) = read.unwrap_or_default(); + assert_eq!(rows, 100); + assert!(most < (RESET_BYTES + (2 << 20)) as usize, "{most} bytes held"); + Ok(()) + } } diff --git a/crates/pg_warren/src/source/id_type.rs b/crates/pg_warren/src/source/id_type.rs index a4f915b8..51492329 100644 --- a/crates/pg_warren/src/source/id_type.rs +++ b/crates/pg_warren/src/source/id_type.rs @@ -1,8 +1,15 @@ //! The types a graph's node ids may have (`graphs.id_type`, DESIGN.md §5.1, -//! §5.2), and the external ids of `warren-core` as Postgres values. +//! §5.2, §5.7), and the external ids of `warren-core` as Postgres values. + +use std::borrow::Borrow; +use std::ffi::CStr; +use std::fmt::Debug; use pgrx::{pg_sys, FromDatum, IntoDatum, Uuid}; -use warren_core::graph::ExternalId; +use warren_core::graph::{ + BuildBuffer, BuildMemory, ExternalId, FixedKeys, GraphError, IdMap, IndexedKeys, TextIds, + TextIndex, TextKeys, +}; /// A graph's id type: the type of its node query's one column. #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -13,15 +20,19 @@ pub(crate) enum IdType { Int8, /// `uuid`, as `[u8; 16]` external ids. Uuid, + /// `text` (or `varchar`, read as `text`), as the bytes of each id in + /// the database encoding. + Text, } impl IdType { - /// The id type with this type oid, if it is one. + /// The id type with this type oid, if it is one: `varchar` is `text`. pub(crate) fn from_oid(oid: pg_sys::Oid) -> Option { match oid { pg_sys::INT4OID => Some(Self::Int4), pg_sys::INT8OID => Some(Self::Int8), pg_sys::UUIDOID => Some(Self::Uuid), + pg_sys::TEXTOID | pg_sys::VARCHAROID => Some(Self::Text), _ => None, } } @@ -32,6 +43,7 @@ impl IdType { Self::Int4 => pg_sys::INT4OID, Self::Int8 => pg_sys::INT8OID, Self::Uuid => pg_sys::UUIDOID, + Self::Text => pg_sys::TEXTOID, } } @@ -41,12 +53,70 @@ impl IdType { Self::Int4 => "integer", Self::Int8 => "bigint", Self::Uuid => "uuid", + Self::Text => "text", } } + + /// Whether a column of type `oid` holds ids of this type: its own + /// type, or `varchar` for `text`, which has the same representation. + pub(crate) fn reads(self, oid: pg_sys::Oid) -> bool { + Self::from_oid(oid) == Some(self) + } } -/// An external id type with its Postgres value type: what the adapter -/// reads from source rows and SQL arguments and returns to SQL. +/// The key storage of an id type with its Postgres values: how the adapter +/// reads ids from source rows and SQL arguments and returns them to SQL. +/// Implemented by [`FixedKeys`] of each [`InPlaceId`] and by [`TextKeys`]. +pub(crate) trait IdDatum: IndexedKeys + 'static { + /// The id type. + const ID_TYPE: IdType; + /// An external id held by value, as an argument holds it. + type Owned: Borrow + Clone + Debug + Send + Sync + 'static; + /// An external id read in place from a datum that lives for `'t`. + type InPlace<'t>: Borrow; + + /// The id a datum holds, read in place (a text id is detoasted, if it + /// must be, into the current memory context, else not copied), or + /// `None` for a NULL datum. + /// + /// # Safety + /// + /// `datum` is NULL or a datum of a type [`IdType::reads`] accepts for + /// [`Self::ID_TYPE`], which lives for `'t`. + unsafe fn in_place<'t>(datum: pg_sys::Datum, is_null: bool) -> Option>; + + /// The id `key`, by value. + fn owned(key: &Self::Key) -> Self::Owned; + + /// The id `key` as an error message names it: by its type's output + /// function, or for text its bytes clipped to [`NAMED_BYTES`] on a + /// character boundary and never checked against the encoding, so that + /// naming an id raises no error of its own. + fn named(key: &Self::Key) -> String; + + /// A datum of [`Self::ID_TYPE`] holding `key`. A text id is checked to + /// be valid in the database encoding first (DESIGN.md §5.7), so no + /// invalid text reaches SQL: a version's bytes are not checked when it + /// is loaded. + fn datum(key: &Self::Key) -> Option; + + /// The external id an integer of a narrower SQL type widens to, if + /// this is an integer id type it fits. + fn from_integer(value: i64) -> Option; + + /// No node ids, to take a node query's, charged to `memory`. + fn node_ids(memory: &BuildMemory) -> Self::Ids; + + /// Appends `key` to `ids`. + fn push_node_id(ids: &mut Self::Ids, key: &Self::Key) -> Result<(), GraphError>; + + /// The bytes of the hash index a call that maps many ids builds over + /// `id_map` (DESIGN.md §5.7), or `None` for an id type that maps them by + /// binary search alone. + fn lookup_index_bytes(id_map: &IdMap) -> Option; +} + +/// A fixed-width external id type with its Postgres value type. /// /// # Safety /// @@ -56,7 +126,9 @@ impl IdType { /// at most 8. Its segments are copied there unchanged on a little-endian /// target, so `Self::WIDTH` must be `size_of::()` and, on such a /// target, an id's encoding (`write_le`) must be its bytes in memory. -pub(crate) unsafe trait IdDatum: ExternalId + Send + Sync + 'static { +pub(crate) unsafe trait InPlaceId: ExternalId + Send + Sync + 'static { + /// The id type. + const ID_TYPE: IdType; /// The Rust type pgrx reads and writes the SQL value as. type Sql: FromDatum + IntoDatum; @@ -73,7 +145,8 @@ pub(crate) unsafe trait IdDatum: ExternalId + Send + Sync + 'static { // SAFETY: an integer: any bits are a value, no padding, alignment 4; // encoded little-endian in its size. -unsafe impl IdDatum for i32 { +unsafe impl InPlaceId for i32 { + const ID_TYPE: IdType = IdType::Int4; type Sql = i32; fn from_sql(value: i32) -> Self { @@ -91,7 +164,8 @@ unsafe impl IdDatum for i32 { // SAFETY: an integer: any bits are a value, no padding, alignment 8; // encoded little-endian in its size. -unsafe impl IdDatum for i64 { +unsafe impl InPlaceId for i64 { + const ID_TYPE: IdType = IdType::Int8; type Sql = i64; fn from_sql(value: i64) -> Self { @@ -109,7 +183,8 @@ unsafe impl IdDatum for i64 { // SAFETY: bytes: any bits are a value, no padding, alignment 1; // encoded as themselves. -unsafe impl IdDatum for [u8; 16] { +unsafe impl InPlaceId for [u8; 16] { + const ID_TYPE: IdType = IdType::Uuid; type Sql = Uuid; fn from_sql(value: Uuid) -> Self { @@ -125,22 +200,153 @@ unsafe impl IdDatum for [u8; 16] { } } +impl IdDatum for FixedKeys { + const ID_TYPE: IdType = E::ID_TYPE; + type Owned = E; + type InPlace<'t> = E; + + unsafe fn in_place<'t>(datum: pg_sys::Datum, is_null: bool) -> Option> { + // SAFETY: the caller guarantees a NULL datum or one of E's SQL type. + let sql = unsafe { E::Sql::from_polymorphic_datum(datum, is_null, E::ID_TYPE.oid()) }; + sql.map(E::from_sql) + } + + fn owned(key: &E) -> E { + *key + } + + fn named(key: &E) -> String { + let Some(datum) = Self::datum(key) else { + return String::new(); + }; + let (mut output, mut varlena) = (pg_sys::Oid::INVALID, false); + // SAFETY: the out-pointers point to live locals; the datum is of the + // type whose output function is called, which returns a palloc'd + // NUL-terminated string. + unsafe { + pg_sys::getTypeOutputInfo(E::ID_TYPE.oid(), &mut output, &mut varlena); + let text = pg_sys::OidOutputFunctionCall(output, datum); + CStr::from_ptr(text).to_string_lossy().into_owned() + } + } + + fn datum(key: &E) -> Option { + key.to_sql().into_datum() + } + + fn from_integer(value: i64) -> Option { + E::from_integer(value) + } + + fn node_ids(memory: &BuildMemory) -> BuildBuffer { + BuildBuffer::new(memory) + } + + fn push_node_id(ids: &mut BuildBuffer, key: &E) -> Result<(), GraphError> { + ids.push(*key) + } + + fn lookup_index_bytes(_: &IdMap) -> Option { + None + } +} + +/// The bytes of a text id an error message shows. +pub(crate) const NAMED_BYTES: usize = 256; + +/// The most bytes a `text` holds: Postgres's `MaxAllocSize` (1 GB less one +/// byte), less the varlena header. +const MAX_TEXT_BYTES: usize = 0x3fff_ffff - pg_sys::VARHDRSZ; + +impl IdDatum for TextKeys { + const ID_TYPE: IdType = IdType::Text; + type Owned = Vec; + type InPlace<'t> = &'t [u8]; + + unsafe fn in_place<'t>(datum: pg_sys::Datum, is_null: bool) -> Option> { + // SAFETY: the caller guarantees a NULL datum or a `text` or + // `varchar` one living for 't; both are varlenas, read as their + // bytes. + unsafe { <&[u8]>::from_polymorphic_datum(datum, is_null, pg_sys::TEXTOID) } + } + + fn owned(key: &[u8]) -> Vec { + key.to_vec() + } + + fn named(key: &[u8]) -> String { + if key.len() <= NAMED_BYTES { + return String::from_utf8_lossy(key).into_owned(); + } + // SAFETY: `key` holds more than NAMED_BYTES readable bytes, of + // which pg_mbcliplen reads at most NAMED_BYTES, stepping by the + // database encoding's character lengths without checking them (a + // text holds no NUL byte, at which it would stop), and returns a + // length within them. + let clipped = unsafe { + pg_sys::pg_mbcliplen(key.as_ptr().cast(), NAMED_BYTES as i32, NAMED_BYTES as i32) + }; + let clipped = usize::try_from(clipped).unwrap_or(0).min(NAMED_BYTES); + format!("{}…", String::from_utf8_lossy(&key[..clipped])) + } + + fn datum(key: &[u8]) -> Option { + let fits = i32::try_from(key.len()).ok().filter(|_| key.len() <= MAX_TEXT_BYTES); + let Some(len) = fits else { + pgrx::ereport!( + ERROR, + pgrx::PgSqlErrorCode::ERRCODE_PROGRAM_LIMIT_EXCEEDED, + format!("a text id of {} bytes is too long to return", key.len()) + ); + }; + let start = key.as_ptr().cast::(); + // SAFETY: `start` points to `len` readable bytes. Both functions + // are guarded and raise an ERROR rather than return a bad value: + // pg_verifymbstr, with `noError` false, raises 22021 for bytes + // invalid in the database encoding; the length fits a text. + unsafe { + pg_sys::pg_verifymbstr(start, len, false); + Some(pg_sys::Datum::from(pg_sys::cstring_to_text_with_len(start, len))) + } + } + + fn from_integer(_: i64) -> Option> { + None + } + + fn node_ids(memory: &BuildMemory) -> TextIds { + TextIds::new(memory) + } + + fn push_node_id(ids: &mut TextIds, key: &[u8]) -> Result<(), GraphError> { + ids.push(key) + } + + fn lookup_index_bytes(id_map: &IdMap) -> Option { + Some(TextIndex::bytes_for(id_map)) + } +} + /// Evaluates `$body` with `$E` the external id type of `$id_type` -/// (`i32`, `i64` or `[u8; 16]`): how the adapter dispatches a call on a +/// (`FixedKeys` of `i32`, `i64` or `[u8; 16]`, or `TextKeys`): how the adapter dispatches a call on a /// graph's id type to code generic over [`IdDatum`]. macro_rules! with_id_type { ($id_type:expr, $E:ident => $body:expr) => { match $id_type { $crate::source::IdType::Int4 => { - type $E = i32; + type $E = ::warren_core::graph::FixedKeys; $body } $crate::source::IdType::Int8 => { - type $E = i64; + type $E = ::warren_core::graph::FixedKeys; $body } $crate::source::IdType::Uuid => { - type $E = [u8; 16]; + type $E = ::warren_core::graph::FixedKeys<[u8; 16]>; + $body + } + $crate::source::IdType::Text => { + type $E = ::warren_core::graph::TextKeys; $body } } @@ -150,12 +356,12 @@ pub(crate) use with_id_type; #[cfg(test)] #[allow(clippy::disallowed_methods)] // Test code may assign a transaction id. -mod tests { +mod unit { use super::*; #[test] fn an_external_id_is_encoded_as_its_bytes_in_memory() { - fn check(id: E, in_memory: &[u8]) { + fn check(id: E, in_memory: &[u8]) { assert_eq!(E::WIDTH, std::mem::size_of::()); let mut encoded = Vec::new(); id.write_le(&mut encoded); @@ -176,10 +382,15 @@ mod tests { #[test] fn id_types_round_trip_through_their_oids() { - for id_type in [IdType::Int4, IdType::Int8, IdType::Uuid] { + for id_type in [IdType::Int4, IdType::Int8, IdType::Uuid, IdType::Text] { assert_eq!(IdType::from_oid(id_type.oid()), Some(id_type)); + assert!(id_type.reads(id_type.oid())); + } + assert_eq!(IdType::from_oid(pg_sys::VARCHAROID), Some(IdType::Text)); + assert!(IdType::Text.reads(pg_sys::VARCHAROID)); + for refused in [pg_sys::BPCHAROID, pg_sys::NAMEOID, pg_sys::INT2OID, pg_sys::BYTEAOID] { + assert_eq!(IdType::from_oid(refused), None); } - assert_eq!(IdType::from_oid(pg_sys::TEXTOID), None); } #[test] @@ -195,15 +406,55 @@ mod tests { assert_eq!(i32::from_integer(7), Some(7)); assert_eq!(i32::from_integer(i64::from(i32::MAX) + 1), None); assert_eq!(<[u8; 16]>::from_integer(1), None); + assert_eq!(::from_integer(1), None); } #[test] fn dispatch_picks_the_external_id_type() { - fn width(id_type: IdType) -> usize { - with_id_type!(id_type, E => ::WIDTH) + fn dispatched(id_type: IdType) -> IdType { + with_id_type!(id_type, E => ::ID_TYPE) } - assert_eq!(width(IdType::Int4), 4); - assert_eq!(width(IdType::Int8), 8); - assert_eq!(width(IdType::Uuid), 16); + for id_type in [IdType::Int4, IdType::Int8, IdType::Uuid, IdType::Text] { + assert_eq!(dispatched(id_type), id_type); + } + } +} + +#[cfg(any(test, feature = "pg_test"))] +#[pgrx::pg_schema] +mod tests { + use pgrx::prelude::*; + use warren_core::graph::{FixedKeys, TextKeys}; + + use super::{IdDatum, NAMED_BYTES}; + + /// An error names a text id clipped to a character boundary, and never + /// checks it, so naming it raises no error of its own. + #[pg_test] + fn a_text_id_is_named_clipped_and_unchecked() { + assert_eq!(TextKeys::named("a b".as_bytes()), "a b"); + let long = "é".repeat(1000); + assert_eq!(TextKeys::named(long.as_bytes()), format!("{}…", "é".repeat(NAMED_BYTES / 2))); + let odd = format!("x{long}"); + let named = TextKeys::named(odd.as_bytes()); + assert_eq!(named, format!("x{}…", "é".repeat((NAMED_BYTES - 1) / 2)), "on a boundary"); + let invalid = TextKeys::named(&[0xff; 1000]); + assert!(invalid.chars().count() <= NAMED_BYTES + 1, "{}", invalid.len()); + assert_eq!(FixedKeys::::named(&-7), "-7"); + } + + /// A text id is checked against the database encoding as it becomes a + /// datum (DESIGN.md §5.7); the test database is UTF8. + #[pg_test(error = "invalid byte sequence for encoding \"UTF8\": 0xff")] + fn a_text_id_invalid_in_the_database_encoding_is_refused() { + TextKeys::datum(&[b'a', 0xff]); + } + + #[pg_test] + fn a_text_id_becomes_a_text_datum() { + let datum = TextKeys::datum("é".as_bytes()); + // SAFETY: the datum was just made as a text. + let text = datum.and_then(|datum| unsafe { String::from_datum(datum, false) }); + assert_eq!(text.as_deref(), Some("é")); } } diff --git a/crates/pg_warren/src/source/mod.rs b/crates/pg_warren/src/source/mod.rs index 4429969e..6d4fa896 100644 --- a/crates/pg_warren/src/source/mod.rs +++ b/crates/pg_warren/src/source/mod.rs @@ -17,7 +17,6 @@ pub(crate) use columns::{ check_edge_columns, check_node_columns, require_node_id_type, ColumnType, SourceError, }; pub(crate) use cursor::{ - describe, for_each_row, for_each_tuple, open_read_only, source_columns, text_in_place, - type_name, with_pinned_snapshot, + describe, for_each_tuple, open_read_only, source_columns, type_name, with_pinned_snapshot, Row, }; -pub(crate) use id_type::{with_id_type, IdDatum, IdType}; +pub(crate) use id_type::{with_id_type, IdDatum, IdType, InPlaceId}; diff --git a/crates/pg_warren/src/spectral_sql/distance/methods.rs b/crates/pg_warren/src/spectral_sql/distance/methods.rs index 5ac272b9..57f358fa 100644 --- a/crates/pg_warren/src/spectral_sql/distance/methods.rs +++ b/crates/pg_warren/src/spectral_sql/distance/methods.rs @@ -14,12 +14,13 @@ use super::super::eigen::{admit, eigenpairs}; use super::super::matrix::OperatorArguments; use super::super::SpectralSqlError; use super::pair::Pair; -use crate::compute_sql::{workers, Vectors}; +use crate::cache::CachedIds; +use crate::compute_sql::Vectors; +use crate::interrupts::workers; use crate::interrupts::PostgresInterrupts; -use crate::source::IdDatum; /// `edit`: per relation (matched by name) and overall. -pub(super) fn edit(pair: &Pair) -> Result { +pub(super) fn edit(pair: &Pair) -> Result { let distance = pair.edit_distance()?; let relations: Vec = distance .relations @@ -41,7 +42,7 @@ fn change_json(change: &EdgeSetChange) -> Value { } /// `degree`: Wasserstein-1 and KS between the out-degree distributions. -pub(super) fn degree(pair: &Pair) -> Result { +pub(super) fn degree(pair: &Pair) -> Result { let (before, after) = (&pair.before, &pair.after); let pool = workers()?; let mut resources = @@ -63,7 +64,7 @@ pub(super) enum Eigenvalues<'a> { } /// `spectral`: the distance between the two eigenvalue sequences. -pub(super) fn spectral( +pub(super) fn spectral( pair: &mut Pair, operator: &OperatorArguments, eigenvalues: &Eigenvalues<'_>, @@ -87,7 +88,7 @@ pub(super) fn spectral( } /// One version's eigenvalues, and for a fresh solve whether it converged. -fn values( +fn values( vectors: &mut Vectors, operator: &OperatorArguments, eigenvalues: &Eigenvalues<'_>, @@ -116,7 +117,7 @@ fn values( /// `subspace`: principal angles between the spans of `_1..k` of /// each version. -pub(super) fn subspace( +pub(super) fn subspace( pair: &mut Pair, prefix: &str, k: usize, @@ -139,7 +140,7 @@ pub(super) fn subspace( /// `embedding`: row-wise distances between matrix `x` of the earlier /// version and matrix `y` of the later. -pub(super) fn embedding( +pub(super) fn embedding( pair: &mut Pair, x: &str, y: &str, diff --git a/crates/pg_warren/src/spectral_sql/distance/mod.rs b/crates/pg_warren/src/spectral_sql/distance/mod.rs index ff606011..d0482e89 100644 --- a/crates/pg_warren/src/spectral_sql/distance/mod.rs +++ b/crates/pg_warren/src/spectral_sql/distance/mod.rs @@ -17,8 +17,8 @@ use self::pair::Pair; use super::matrix::{solver_options, OperatorArguments}; use super::SpectralSqlError; use crate::access::entry; -use crate::cache::resolve; -use crate::source::{with_id_type, IdDatum}; +use crate::cache::{resolve, CachedIds}; +use crate::source::with_id_type; /// What a `distance` call compares, checked. enum Method<'a> { @@ -41,7 +41,7 @@ impl Method<'_> { } /// The distance between the versions of `pair`. - fn measure(&self, pair: &mut Pair) -> Result { + fn measure(&self, pair: &mut Pair) -> Result { match self { Self::Edit => methods::edit(pair), Self::Degree => methods::degree(pair), diff --git a/crates/pg_warren/src/spectral_sql/distance/pair.rs b/crates/pg_warren/src/spectral_sql/distance/pair.rs index 1d3ffce1..6032a445 100644 --- a/crates/pg_warren/src/spectral_sql/distance/pair.rs +++ b/crates/pg_warren/src/spectral_sql/distance/pair.rs @@ -12,11 +12,11 @@ use warren_core::spectral::{ use warren_core::version; use super::super::SpectralSqlError; -use crate::cache::Resolved; +use crate::cache::{CachedIds, Resolved}; use crate::catalog::compaction_between; -use crate::compute_sql::{workers, Vectors}; +use crate::compute_sql::Vectors; +use crate::interrupts::workers; use crate::interrupts::PostgresInterrupts; -use crate::source::IdDatum; /// The versions compared, `before` first. pub(super) struct Pair { @@ -27,7 +27,7 @@ pub(super) struct Pair { alignment: Option, } -impl Pair { +impl Pair { /// Opens `before` and `after`. Versions of one graph must not have a /// compaction between them, and their shared dense ids must name the /// same nodes; versions of two graphs are aligned by external id. diff --git a/crates/pg_warren/src/spectral_sql/eigen.rs b/crates/pg_warren/src/spectral_sql/eigen.rs index 4aaaa741..f83198ee 100644 --- a/crates/pg_warren/src/spectral_sql/eigen.rs +++ b/crates/pg_warren/src/spectral_sql/eigen.rs @@ -10,10 +10,11 @@ use super::caveats::caveats; use super::matrix::{solver_options, OperatorArguments}; use super::SpectralSqlError; use crate::access::entry; -use crate::cache::resolve; -use crate::compute_sql::{drop_named, vector_bytes, workers, Vectors}; +use crate::cache::{resolve, CachedIds}; +use crate::compute_sql::{drop_named, vector_bytes, Vectors}; +use crate::interrupts::workers; use crate::interrupts::PostgresInterrupts; -use crate::source::{with_id_type, IdDatum}; +use crate::source::with_id_type; /// One row of `eigen`'s result. type Row = (i32, f64, f64); @@ -25,7 +26,7 @@ type Row = (i32, f64, f64); /// whether transient eigenvectors fit `warren.max_cache_memory` (with the /// transient ones they replace or drop released), is checked before /// solving, so a refusal writes none. -fn solve( +fn solve( mut vectors: Vectors, operator: &OperatorArguments, options: &EigenOptions, @@ -130,7 +131,7 @@ fn eigen( } /// The eigenpairs of `operator` over `vectors`' version. -pub(super) fn eigenpairs( +pub(super) fn eigenpairs( vectors: &Vectors, operator: &OperatorArguments, options: &EigenOptions, @@ -146,7 +147,7 @@ pub(super) fn eigenpairs( /// Refuses `eigenpairs(vectors, operator, options)` if its working set /// cannot fit the compute memory, without solving. -pub(super) fn admit( +pub(super) fn admit( vectors: &Vectors, operator: &OperatorArguments, options: &EigenOptions, diff --git a/crates/pg_warren/src/spectral_sql/error.rs b/crates/pg_warren/src/spectral_sql/error.rs index 7bf2dc8d..c6bb0c47 100644 --- a/crates/pg_warren/src/spectral_sql/error.rs +++ b/crates/pg_warren/src/spectral_sql/error.rs @@ -8,7 +8,8 @@ use warren_core::version::VersionError; use crate::access::SqlError; use crate::cache::CacheError; use crate::catalog::{CatalogError, VersionId}; -use crate::compute_sql::{compute_sqlstate, ComputeSqlError}; +use crate::compute_sql::ComputeSqlError; +use crate::extension::compute_sqlstate; /// Why `eigen` or `distance` could not answer. #[derive(Debug, Error)] diff --git a/crates/pg_warren/src/traversal_sql/bfs.rs b/crates/pg_warren/src/traversal_sql/bfs.rs index d1916d5b..90e1696e 100644 --- a/crates/pg_warren/src/traversal_sql/bfs.rs +++ b/crates/pg_warren/src/traversal_sql/bfs.rs @@ -12,17 +12,17 @@ use std::sync::Arc; use pgrx::prelude::*; use pgrx::{AnyArray, AnyElement}; -use warren_core::graph::{FixedKeys, GraphCsr, GraphError, IdMap}; +use warren_core::graph::{GraphCsr, GraphError, IdMap}; use warren_core::traversal::{BfsCursor, Limits, NodeSet, TraversalError, Visit}; use super::args::bound; use super::stats::Recorder; use super::{entry, TraversalSqlError}; -use crate::cache::{cached, resolve, LoadedVersion}; +use crate::cache::{cached, resolve, CachedIds, LoadedVersion}; use crate::external_ids::{dense_ids, elements, external}; use crate::interrupts::PostgresInterrupts; use crate::selection::{direction, relations}; -use crate::source::{with_id_type, IdDatum}; +use crate::source::with_id_type; /// A row of `warren.bfs`. type BfsRow = (name!(node_id, AnyElement), name!(depth, i32), name!(via, Option)); @@ -138,13 +138,13 @@ struct Request<'a> { pull: bool, } -fn bfs_row(id_map: &IdMap>, visit: Visit) -> Option { +fn bfs_row(id_map: &IdMap, visit: Visit) -> Option { let via = visit.via.and_then(|via| external(id_map, via)); let depth = i32::try_from(visit.depth).unwrap_or(i32::MAX); Some((external(id_map, visit.node)?, depth, via)) } -fn node_row(id_map: &IdMap>, visit: Visit) -> Option { +fn node_row(id_map: &IdMap, visit: Visit) -> Option { external(id_map, visit.node) } @@ -161,19 +161,19 @@ impl Borrow for VersionGraph { struct LevelRows { loaded: Arc>, cursor: BfsCursor>, - row: fn(&IdMap>, Visit) -> Option, + row: fn(&IdMap, Visit) -> Option, /// The visits of the current level not yet returned, in reverse. level: Vec, more: bool, recorder: Recorder, } -impl LevelRows { +impl LevelRows { fn new( request: &Request<'_>, version: i64, loaded: Arc>, - row: fn(&IdMap>, Visit) -> Option, + row: fn(&IdMap, Visit) -> Option, ) -> Result { let direction = direction(request.direction)?; let relations = relations(&loaded.relations, request.rels.as_deref()); @@ -228,7 +228,7 @@ impl LevelRows { } } -impl Iterator for LevelRows { +impl Iterator for LevelRows { type Item = R; fn next(&mut self) -> Option { diff --git a/crates/pg_warren/src/traversal_sql/ids.rs b/crates/pg_warren/src/traversal_sql/ids.rs index e13db595..148b1b77 100644 --- a/crates/pg_warren/src/traversal_sql/ids.rs +++ b/crates/pg_warren/src/traversal_sql/ids.rs @@ -1,20 +1,21 @@ //! External ids spelled as text in traversal arguments (`warren.step` //! filters); the rest of id handling is in `external_ids`. +use std::borrow::Borrow; use std::ffi::CString; use pgrx::prelude::*; -use pgrx::FromDatum; use super::TraversalSqlError; -use crate::source::IdDatum; + +use crate::cache::CachedIds; /// The external id `text` spells, read with the id type's input function /// (which raises `invalid_text_representation` if it is not one). -pub(crate) fn parse(text: &str) -> Result { +pub(crate) fn parse(text: &str) -> Result { let text = CString::new(text) .map_err(|_| TraversalSqlError::Step("an id contains a NUL byte".to_owned()))?; - let oid = E::Sql::type_oid(); + let oid = E::ID_TYPE.oid(); let mut input = pg_sys::Oid::INVALID; let mut io_param = pg_sys::Oid::INVALID; // SAFETY: the out-pointers point to live locals; `text` is @@ -24,7 +25,8 @@ pub(crate) fn parse(text: &str) -> Result { pg_sys::getTypeInputInfo(oid, &mut input, &mut io_param); pg_sys::OidInputFunctionCall(input, text.as_ptr().cast_mut(), io_param, -1) }; - // SAFETY: the datum was just made by E's SQL type's input function. - let sql = unsafe { E::Sql::from_polymorphic_datum(datum, false, oid) }; - sql.map(E::from_sql).ok_or(TraversalSqlError::NullArgument("a filter id")) + // SAFETY: the datum was just made by the id type's input function, in + // the current memory context, which outlives this call. + let id = unsafe { E::in_place(datum, false) }; + id.map(|id| E::owned(id.borrow())).ok_or(TraversalSqlError::NullArgument("a filter id")) } diff --git a/crates/pg_warren/src/traversal_sql/neighbourhood.rs b/crates/pg_warren/src/traversal_sql/neighbourhood.rs index 98c5f5a4..95a553b0 100644 --- a/crates/pg_warren/src/traversal_sql/neighbourhood.rs +++ b/crates/pg_warren/src/traversal_sql/neighbourhood.rs @@ -5,6 +5,8 @@ //! shared version, or loading its own copy where the shared cache is //! unavailable. +use std::borrow::Borrow; + use pgrx::prelude::*; use pgrx::AnyElement; use warren_core::graph::DenseId; @@ -12,10 +14,10 @@ use warren_core::graph::Direction; use warren_core::traversal::Adjacency; use super::{entry, TraversalSqlError}; -use crate::cache::{cached, resolve, LoadedVersion, Resolved}; +use crate::cache::{cached, resolve, CachedIds, LoadedVersion, Resolved}; use crate::external_ids::{element, element_widened, external}; use crate::selection::{adjacency, direction}; -use crate::source::{with_id_type, IdDatum}; +use crate::source::with_id_type; /// `warren.neighbours`: the distinct nodes one edge from `node` in /// `direction` along `rels` (all if NULL), ascending by dense id. None if @@ -156,7 +158,7 @@ impl<'a> NodeQuery<'a> { /// Runs `f` on the loaded version, the adjacency the query selects and /// the node's dense id; `None` if the node is NULL or not a node of the /// version. - fn adjacent( + fn adjacent( &self, f: impl FnOnce(&LoadedVersion, &Adjacency<'_>, DenseId) -> T, ) -> Result, TraversalSqlError> { @@ -170,7 +172,7 @@ impl<'a> NodeQuery<'a> { let loaded = cached::(&self.resolved)?; let adjacency = adjacency(&loaded.graph, &loaded.relations, self.direction, self.rels.as_deref())?; - Ok(loaded.id_map.dense_id(&node).map(|node| f(&loaded, &adjacency, node))) + Ok(loaded.id_map.dense_id(node.borrow()).map(|node| f(&loaded, &adjacency, node))) } } diff --git a/crates/pg_warren/src/traversal_sql/paths.rs b/crates/pg_warren/src/traversal_sql/paths.rs index 3becbbfb..e6f29dc8 100644 --- a/crates/pg_warren/src/traversal_sql/paths.rs +++ b/crates/pg_warren/src/traversal_sql/paths.rs @@ -3,6 +3,8 @@ //! target that is not a node of the version reaches nothing, not even //! itself; a NULL one gives a NULL answer (no row for `dijkstra`). +use std::borrow::Borrow; + use pgrx::prelude::*; use pgrx::{AnyArray, AnyElement}; use warren_core::graph::{DenseId, Direction}; @@ -14,11 +16,11 @@ use warren_core::traversal::{ use super::args::bound; use super::stats::Recorder; use super::{entry, TraversalSqlError}; -use crate::cache::{cached, resolve, LoadedVersion, Resolved}; +use crate::cache::{cached, resolve, CachedIds, LoadedVersion, Resolved}; use crate::external_ids::{element, element_widened, external_array}; use crate::interrupts::PostgresInterrupts; use crate::selection::{adjacency, direction}; -use crate::source::{with_id_type, IdDatum, IdType}; +use crate::source::{with_id_type, IdType}; /// `warren.reachable`: whether `target` is within `max_depth` edges of /// `start` (unbounded if NULL) in `direction` along `rels`. @@ -196,7 +198,7 @@ impl Opened<'_> { /// Records stats under the function's name and runs `search` on the /// selected adjacency from start to target. `None` if an endpoint is /// NULL; `Some(None)` if one is not a node of the version. - fn run( + fn run( &self, search: impl FnOnce(&LoadedVersion, &Adjacency<'_>, DenseId, DenseId) -> Searched, ) -> Result>, TraversalSqlError> { @@ -212,7 +214,7 @@ impl Opened<'_> { let target = read(self.graph, "target", target)?; let loaded = cached::(&self.resolved)?; let (Some(start), Some(target)) = - (loaded.id_map.dense_id(&start), loaded.id_map.dense_id(&target)) + (loaded.id_map.dense_id(start.borrow()), loaded.id_map.dense_id(target.borrow())) else { return Ok(Some(None)); }; diff --git a/crates/pg_warren/src/traversal_sql/step.rs b/crates/pg_warren/src/traversal_sql/step.rs index c1deef97..a5563a70 100644 --- a/crates/pg_warren/src/traversal_sql/step.rs +++ b/crates/pg_warren/src/traversal_sql/step.rs @@ -9,6 +9,8 @@ //! its filter's external ids as `text`, in their id type's output format; //! `warren.walk` reads them back with the graph's id type's input function. +use std::borrow::Borrow; + use pgrx::heap_tuple::PgHeapTuple; use pgrx::prelude::*; use pgrx::{AllocatedByRust, AnyArray, Array}; @@ -16,10 +18,9 @@ use warren_core::traversal::{HopRange, NodeSet, Step}; use super::ids::parse; use super::{entry, TraversalSqlError}; -use crate::cache::LoadedVersion; +use crate::cache::{CachedIds, LoadedVersion}; use crate::external_ids::element_type; use crate::selection::{direction, relations}; -use crate::source::IdDatum; pgrx::extension_sql!( r#" @@ -152,7 +153,7 @@ fn texts(array: &AnyArray) -> Result>, TraversalSqlError> { } /// The steps of a walk, read into `warren-core` steps over `loaded`. -pub(crate) fn read_steps( +pub(crate) fn read_steps( loaded: &LoadedVersion, steps: &Array<'_, PgHeapTuple<'_, AllocatedByRust>>, ) -> Result, TraversalSqlError> { @@ -163,7 +164,7 @@ pub(crate) fn read_steps( .collect() } -fn read_step( +fn read_step( loaded: &LoadedVersion, step: &PgHeapTuple<'_, AllocatedByRust>, ) -> Result { @@ -198,12 +199,12 @@ fn read_step( } /// The live nodes among a step filter's external ids, as text. -fn filter_set( +fn filter_set( loaded: &LoadedVersion, members: &[Option], ) -> Result { let ids = - members.iter().flatten().map(|member| parse::(member)).collect::, _>>()?; - let dense = ids.iter().filter_map(|id| loaded.id_map.dense_id(id)); + members.iter().flatten().map(|member| parse::(member)).collect::, _>>()?; + let dense = ids.iter().filter_map(|id| loaded.id_map.dense_id(id.borrow())); Ok(NodeSet::from_dense_ids(loaded.id_map.id_space(), dense)?) } diff --git a/crates/pg_warren/src/traversal_sql/walk.rs b/crates/pg_warren/src/traversal_sql/walk.rs index 12addac2..a416e067 100644 --- a/crates/pg_warren/src/traversal_sql/walk.rs +++ b/crates/pg_warren/src/traversal_sql/walk.rs @@ -16,10 +16,10 @@ use super::stats::Recorder; use super::step::read_steps; use super::walk_rows::{max_walk_paths, DEFAULT_WALK_ROWS}; use super::{entry, TraversalSqlError}; -use crate::cache::{cached, resolve, LoadedVersion}; +use crate::cache::{cached, resolve, CachedIds, LoadedVersion}; use crate::external_ids::{dense_ids, elements, external, external_array}; use crate::interrupts::PostgresInterrupts; -use crate::source::{with_id_type, IdDatum}; +use crate::source::with_id_type; /// A row of `warren.walk`. type WalkRow = (Option, Option, Option>, Option); @@ -133,7 +133,7 @@ struct Walked<'w, E> { options: WalkOptions, } -impl Walked<'_, E> { +impl Walked<'_, E> { /// Runs the walk and returns its rows, recording its stats. fn rows( &self, diff --git a/crates/pg_warren/src/version_sql/compact.rs b/crates/pg_warren/src/version_sql/compact.rs index 1594db08..b3e87185 100644 --- a/crates/pg_warren/src/version_sql/compact.rs +++ b/crates/pg_warren/src/version_sql/compact.rs @@ -8,12 +8,12 @@ use warren_core::version::compact as renumber; use super::error::VersionOperationError; use crate::access::{entry, require_owner}; use crate::build::{max_build_memory, Catalog, SpiCatalog}; -use crate::cache::load; +use crate::cache::{load, CachedIds}; use crate::catalog::{ parse_node_order, version_record, GraphDefinition, NewVersion, VersionId, VersionStats, }; use crate::segment::{write_version, SegmentStore, SpiSegmentStore, CHUNK_SIZE}; -use crate::source::{with_id_type, IdDatum}; +use crate::source::with_id_type; use crate::vector::{move_vectors, moved_notice, SpiVectorStore, VectorStore}; /// `warren.compact`: renumbers graph `name`'s current version to remove its @@ -84,7 +84,7 @@ pub(crate) struct CompactionOrder<'a> { /// holds the same edges. The source's persisted vectors move to it, with /// their rows in the new dense ids. Returns the version and the vectors /// moved. -pub(crate) fn compact_version( +pub(crate) fn compact_version( graph: &GraphDefinition, source: VersionId, carried: &VersionStats, @@ -183,10 +183,16 @@ mod unit { let mut stores = Stores { segments: &mut store, vectors: &mut vectors, catalog: &mut catalog }; let ordering = CompactionOrder { order: NodeOrder::Id, memory: &BuildMemory::unlimited() }; - let compacted = - compact_version::(&graph(), source, &carried, &ordering, 16, &mut stores) - .unwrap() - .0; + let compacted = compact_version::>( + &graph(), + source, + &carried, + &ordering, + 16, + &mut stores, + ) + .unwrap() + .0; let row = catalog.version(compacted).clone(); assert_eq!(row.compacted_from, Some(source)); @@ -203,7 +209,7 @@ mod unit { ); assert!(!row.staged); assert_eq!(catalog.current_version, Some(compacted)); - let loaded = load::(&mut store, compacted, true).unwrap(); + let loaded = load::>(&mut store, compacted, true).unwrap(); assert_eq!(loaded.id_map.dense_id(&40), Some(2)); assert_eq!(loaded.id_map.tombstones().count(), 0); assert_eq!(loaded.graph.relations()[0].forward().neighbour_weights(2), Some(&[3.0][..])); diff --git a/crates/warren-core/src/graph/edge_batch/endpoints.rs b/crates/warren-core/src/graph/edge_batch/endpoints.rs new file mode 100644 index 00000000..d5897403 --- /dev/null +++ b/crates/warren-core/src/graph/edge_batch/endpoints.rs @@ -0,0 +1,138 @@ +//! A batch's endpoint column: each row's external id, or NULL, copied into +//! buffers charged to the build's memory. + +use std::mem::size_of; + +use crate::graph::build_memory::growth; +use crate::graph::{BuildBuffer, BuildMemory, ExternalId, GraphError}; + +/// One endpoint column of an [`EdgeBatch`](super::EdgeBatch). +pub trait Endpoints: Sync + Send { + /// An external id. + type Key: ?Sized; + + /// The bytes a row takes, beyond a text id's own. + const ROW_BYTES: u64; + + /// An empty column with room for `rows` rows (but not their text ids' + /// bytes), charged to `memory`, which keeps room for up to `id_bytes` + /// of text ids when it is cleared. + fn with_capacity(memory: &BuildMemory, rows: usize, id_bytes: u64) -> Result + where + Self: Sized; + + /// Appends a row's endpoint. + fn push(&mut self, key: Option<&Self::Key>) -> Result<(), GraphError>; + + /// Row `row`'s endpoint, if it is not NULL. + fn get(&self, row: usize) -> Option<&Self::Key>; + + /// Empties it, keeping its capacity, except text ids' room past the + /// bytes it was made to keep. + fn clear(&mut self); + + /// The bytes it holds, capacity included. + fn bytes(&self) -> u64; + + /// The bytes of the text ids it holds, capacity excluded: 0 for a + /// fixed type. + fn len_bytes(&self) -> u64; +} + +/// The endpoints of a fixed-width id type. +#[derive(Debug)] +pub struct FixedEndpoints(BuildBuffer>); + +impl Endpoints for FixedEndpoints { + type Key = E; + const ROW_BYTES: u64 = size_of::>() as u64; + + fn with_capacity(memory: &BuildMemory, rows: usize, _: u64) -> Result { + Ok(Self(BuildBuffer::with_capacity(memory, rows)?)) + } + + fn push(&mut self, key: Option<&E>) -> Result<(), GraphError> { + self.0.push(key.copied()) + } + + fn get(&self, row: usize) -> Option<&E> { + self.0.get(row)?.as_ref() + } + + fn clear(&mut self) { + self.0.truncate(0); + } + + fn bytes(&self) -> u64 { + self.0.bytes() + } + + fn len_bytes(&self) -> u64 { + 0 + } +} + +/// The endpoints of text ids: their bytes, where each row's ends, and +/// which rows are NULL. +#[derive(Debug)] +pub struct TextEndpoints { + bytes: BuildBuffer, + /// Where each row's bytes end; each starts where the previous ends. + ends: BuildBuffer, + nulls: BuildBuffer, + /// The room for text ids' bytes kept when it is cleared. + kept: u64, +} + +impl Endpoints for TextEndpoints { + type Key = [u8]; + const ROW_BYTES: u64 = (size_of::() + size_of::()) as u64; + + fn with_capacity(memory: &BuildMemory, rows: usize, id_bytes: u64) -> Result { + Ok(Self { + bytes: BuildBuffer::new(memory), + ends: BuildBuffer::with_capacity(memory, rows)?, + nulls: BuildBuffer::with_capacity(memory, rows)?, + kept: id_bytes, + }) + } + + fn push(&mut self, key: Option<&[u8]>) -> Result<(), GraphError> { + let id = key.unwrap_or_default(); + if self.bytes.capacity_left() < id.len() { + let room = growth(self.bytes.len(), self.bytes.memory(), 1).max(id.len()); + self.bytes.reserve_exact(room)?; + } + self.bytes.extend(id.iter().copied()); + self.ends.push(self.bytes.len())?; + self.nulls.push(key.is_none()) + } + + fn get(&self, row: usize) -> Option<&[u8]> { + if *self.nulls.get(row)? { + return None; + } + let start = match row { + 0 => 0, + _ => self.ends[row - 1], + }; + self.bytes.get(start..self.ends[row]) + } + + fn clear(&mut self) { + match self.bytes.bytes() > self.kept { + true => self.bytes.release(), + false => self.bytes.truncate(0), + } + self.ends.truncate(0); + self.nulls.truncate(0); + } + + fn bytes(&self) -> u64 { + self.bytes.bytes() + self.ends.bytes() + self.nulls.bytes() + } + + fn len_bytes(&self) -> u64 { + self.bytes.len() as u64 + } +} diff --git a/crates/warren-core/src/graph/edge_batch/mod.rs b/crates/warren-core/src/graph/edge_batch/mod.rs new file mode 100644 index 00000000..3a1333d0 --- /dev/null +++ b/crates/warren-core/src/graph/edge_batch/mod.rs @@ -0,0 +1,219 @@ +//! Edge rows taken a batch at a time (DESIGN.md §5.7): a build copies each +//! row's endpoints into a batch it owns, the compute pool maps the whole +//! batch's endpoints to dense ids through an index of the id map, and an +//! [`EdgeIntake`](super::EdgeIntake) then takes the rows in order. Mapping +//! is where the time goes for text ids, whose lookup costs three cache +//! misses, so it runs on every thread of the pool, never calling the +//! caller back. + +mod endpoints; + +pub use endpoints::{Endpoints, FixedEndpoints, TextEndpoints}; + +use std::mem::size_of; + +use rayon::prelude::*; + +use super::{ + BuildBuffer, BuildMemory, DenseId, ExternalId, FixedKeys, GraphError, IdKeys, IdMap, + RelationIndex, TextIndex, TextKeys, +}; +use crate::interrupts::{Cancellation, Cancelled}; + +/// Rows mapped between two checks of the cancellation flag, per thread. +const ROWS_PER_CHECK: usize = 4096; + +/// A key storage whose ids a batch can be mapped through: an index of its +/// live ids that many threads probe at once. +pub trait IndexedKeys: IdKeys + Sync + Send { + /// The index: the id map itself for a fixed type, whose lookup is one + /// binary search of its pairs; a [`TextIndex`] for text. + type Index<'m>: Sync + where + Self: 'm; + /// How a batch holds its endpoints. + type Endpoints: Endpoints; + + /// The index of `id_map`'s live ids, charged to `memory`. + fn index<'m>( + id_map: &'m IdMap, + memory: &BuildMemory, + ) -> Result, GraphError>; + + /// The dense id of a live node, as [`IdMap::dense_id`] answers. + fn probe(index: &Self::Index<'_>, key: &Self::Key) -> Option; +} + +impl IndexedKeys for FixedKeys { + type Index<'m> + = &'m IdMap + where + Self: 'm; + type Endpoints = FixedEndpoints; + + fn index<'m>(id_map: &'m IdMap, _: &BuildMemory) -> Result<&'m IdMap, GraphError> { + Ok(id_map) + } + + fn probe(index: &&IdMap, key: &E) -> Option { + index.dense_id(key) + } +} + +impl IndexedKeys for TextKeys { + type Index<'m> = TextIndex<'m>; + type Endpoints = TextEndpoints; + + fn index<'m>( + id_map: &'m IdMap, + memory: &BuildMemory, + ) -> Result, GraphError> { + TextIndex::new(id_map, memory) + } + + fn probe(index: &TextIndex<'_>, key: &[u8]) -> Option { + index.dense_id(key) + } +} + +/// The endpoints of a row as dense ids; `None` for a NULL or unknown one. +type Mapped = (Option, Option); + +/// Edge rows in a buffer the build owns, every column charged to one +/// build memory. +#[derive(Debug)] +pub struct EdgeBatch { + relations: BuildBuffer, + weights: BuildBuffer, + sources: K::Endpoints, + targets: K::Endpoints, + /// Each row's endpoints, as [`EdgeBatch::map`] mapped them if + /// `is_mapped`. + mapped: BuildBuffer, + /// Whether every row was mapped since the last was pushed. + is_mapped: bool, +} + +impl EdgeBatch { + /// The bytes a row takes, beyond its text ids' own. + pub const ROW_BYTES: u64 = (size_of::() + size_of::() + size_of::()) + as u64 + + 2 * K::Endpoints::ROW_BYTES; + + /// An empty batch with room for `rows` rows (but not their text ids' + /// bytes), charged to `memory`. Emptied, it keeps room for up to + /// `id_bytes` of text ids, half for each endpoint, and frees the rest. + pub fn with_capacity( + memory: &BuildMemory, + rows: usize, + id_bytes: u64, + ) -> Result { + Ok(Self { + relations: BuildBuffer::with_capacity(memory, rows)?, + weights: BuildBuffer::with_capacity(memory, rows)?, + sources: K::Endpoints::with_capacity(memory, rows, id_bytes / 2)?, + targets: K::Endpoints::with_capacity(memory, rows, id_bytes / 2)?, + mapped: BuildBuffer::with_capacity(memory, rows)?, + is_mapped: false, + }) + } + + /// Appends a row, copying its endpoints; a NULL (`None`) endpoint is + /// kept as one, to be dropped and counted by the intake. + pub fn push( + &mut self, + relation: RelationIndex, + source: Option<&K::Key>, + target: Option<&K::Key>, + weight: f64, + ) -> Result<(), GraphError> { + self.is_mapped = false; + self.sources.push(source)?; + self.targets.push(target)?; + self.relations.push(relation)?; + self.weights.push(weight)?; + self.mapped.push((None, None)) + } + + /// The rows it holds. + pub fn len(&self) -> usize { + self.relations.len() + } + + /// The bytes it holds, capacity included: what it charges its memory. + pub fn bytes(&self) -> u64 { + let columns = [self.relations.bytes(), self.weights.bytes(), self.mapped.bytes()]; + columns.iter().sum::() + self.sources.bytes() + self.targets.bytes() + } + + /// The bytes of the text ids it holds, capacity excluded: 0 for a + /// fixed type. + pub fn len_bytes(&self) -> u64 { + self.sources.len_bytes() + self.targets.len_bytes() + } + + /// Whether it holds no rows. + pub fn is_empty(&self) -> bool { + self.relations.is_empty() + } + + /// Maps every row's endpoints through `index`, in parallel on the + /// current `rayon` pool (a job on the compute pool), stopping with + /// [`Cancelled`] once `cancellation` is raised. + pub fn map( + &mut self, + index: &K::Index<'_>, + cancellation: &Cancellation, + ) -> Result<(), Cancelled> { + self.is_mapped = false; + let (sources, targets) = (&self.sources, &self.targets); + let probe = + |endpoints: &K::Endpoints, row| endpoints.get(row).and_then(|key| K::probe(index, key)); + self.mapped.par_chunks_mut(ROWS_PER_CHECK).enumerate().try_for_each(|(chunk, rows)| { + if cancellation.is_cancelled() { + return Err(Cancelled); + } + for (at, mapped) in rows.iter_mut().enumerate() { + let row = chunk * ROWS_PER_CHECK + at; + *mapped = (probe(sources, row), probe(targets, row)); + } + Ok(()) + })?; + self.is_mapped = true; + Ok(()) + } + + /// Row `row`'s relation, endpoints and weight: the endpoints as + /// [`EdgeBatch::map`] mapped them, or looked up in `id_map` if the + /// batch was not mapped since its last row. + pub(crate) fn row( + &self, + row: usize, + id_map: &IdMap, + ) -> (RelationIndex, Option, Option, f64) { + let (source, target) = match self.is_mapped { + true => self.mapped[row], + false => { + let dense = |endpoints: &K::Endpoints| { + endpoints.get(row).and_then(|key| id_map.dense_id(key)) + }; + (dense(&self.sources), dense(&self.targets)) + } + }; + (self.relations[row], source, target, self.weights[row]) + } + + /// Empties it, keeping its capacity, except text ids' room past what + /// it was made to keep. + pub(crate) fn clear(&mut self) { + self.relations.truncate(0); + self.weights.truncate(0); + self.sources.clear(); + self.targets.clear(); + self.mapped.truncate(0); + self.is_mapped = false; + } +} + +#[cfg(test)] +mod tests; diff --git a/crates/warren-core/src/graph/edge_batch/tests.rs b/crates/warren-core/src/graph/edge_batch/tests.rs new file mode 100644 index 00000000..e4beee7e --- /dev/null +++ b/crates/warren-core/src/graph/edge_batch/tests.rs @@ -0,0 +1,231 @@ +use proptest::prelude::*; + +use super::*; +use crate::graph::{EdgeIntake, EdgeList, FixedKeys, RelationDictionary, TextIds, TextKeys}; +use crate::interrupts::Cancellation; + +type Row = (Option, Option, f64); + +/// Whether each batch is mapped before it is taken: both, except under +/// Miri, where only the unmapped path runs. `EdgeBatch::map` runs on the +/// `rayon` pool, whose work stealing runs through crossbeam-epoch, which +/// Miri's Stacked Borrows model rejects (a known report against +/// crossbeam-epoch 0.9, not this crate, which has no unsafe here). +const MAPPED: &[bool] = if cfg!(miri) { &[false] } else { &[false, true] }; + +/// The text id map of `ids`, then a rebuild without `dropped`, which +/// tombstones it. +fn text_map(ids: &[&[u8]], dropped: &[u8]) -> IdMap { + let memory = BuildMemory::unlimited(); + let mut all = TextIds::new(&memory); + let mut kept = TextIds::new(&memory); + for id in ids { + all.push(id).unwrap(); + if *id != dropped { + kept.push(id).unwrap(); + } + } + let (first, _) = IdMap::new().rebuild_within(all).unwrap(); + first.rebuild_within(kept).unwrap().0 +} + +/// The edges of `rows` taken one at a time, and the dropped count. +fn row_by_row( + id_map: &IdMap, + directed: bool, + rows: &[Row<&K::Key>], +) -> (Vec<(DenseId, DenseId)>, Vec, u64) { + let relation = RelationDictionary::new().intern("r").unwrap(); + let mut intake = intake(id_map, directed); + for &(source, target, weight) in rows { + intake.push(relation, source, target, weight).unwrap(); + } + finished(intake) +} + +/// The edges of `rows` taken in batches of `size`, each mapped through +/// the index first if `map`, and the dropped count. +fn batched( + id_map: &IdMap, + directed: bool, + rows: &[Row<&K::Key>], + size: usize, + map: bool, +) -> (Vec<(DenseId, DenseId)>, Vec, u64) { + let memory = BuildMemory::unlimited(); + let relation = RelationDictionary::new().intern("r").unwrap(); + let index = K::index(id_map, &memory).unwrap(); + let mut intake = intake(id_map, directed); + let mut batch = EdgeBatch::::with_capacity(&memory, size, u64::MAX).unwrap(); + for chunk in rows.chunks(size.max(1)) { + for &(source, target, weight) in chunk { + batch.push(relation, source, target, weight).unwrap(); + } + if map { + batch.map(&index, &Cancellation::new()).unwrap(); + } + intake.take(&mut batch).unwrap(); + assert!(batch.is_empty()); + } + finished(intake) +} + +fn intake(id_map: &IdMap, directed: bool) -> EdgeIntake<'_, K> { + match directed { + true => EdgeIntake::directed(id_map, true), + false => EdgeIntake::undirected(id_map, true), + } +} + +fn finished(intake: EdgeIntake<'_, K>) -> (Vec<(DenseId, DenseId)>, Vec, u64) { + let (list, dropped): (EdgeList, u64) = intake.finish(); + let edges = list.edges().map(|(_, source, target)| (source, target)).collect(); + (edges, list.weights().unwrap_or_default().to_vec(), dropped) +} + +#[test] +fn a_mapped_batch_takes_the_edges_rows_would() { + let id_map = text_map(&[b"a", b"b", b"c", b"gone"], b"gone"); + let rows: Vec> = vec![ + (Some(b"a"), Some(b"b"), 1.0), + (Some(b"b"), Some(b"b"), 2.0), + (None, Some(b"c"), 3.0), + (Some(b"c"), Some(b"unknown"), 4.0), + (Some(b"gone"), Some(b"a"), 5.0), + (Some(b"c"), Some(b"a"), 6.0), + (Some(b""), Some(b"a"), 7.0), + ]; + for directed in [true, false] { + let expected = row_by_row(&id_map, directed, &rows); + assert_eq!(expected.2, 4); + for size in [1, 2, 7] { + for &map in MAPPED { + assert_eq!(batched(&id_map, directed, &rows, size, map), expected); + } + } + } +} + +#[test] +fn fixed_ids_are_batched_as_text_ids_are() { + let first = IdMap::>::new().rebuild(vec![10, 20, 30]).unwrap(); + let id_map = first.rebuild(vec![10, 30]).unwrap(); + let rows: Vec> = vec![ + (Some(&10), Some(&30), 1.0), + (Some(&20), Some(&30), 2.0), + (Some(&30), None, 3.0), + (Some(&30), Some(&10), 4.0), + ]; + let expected = row_by_row(&id_map, true, &rows); + assert_eq!(expected, (vec![(0, 2), (2, 0)], vec![1.0, 4.0], 2)); + for &map in MAPPED { + assert_eq!(batched(&id_map, true, &rows, 3, map), expected); + } +} + +// Not under Miri: the map runs on the pool (see `MAPPED`). +#[test] +#[cfg(not(miri))] +fn a_cancelled_map_stops() { + let id_map = text_map(&[b"a", b"b"], b""); + let memory = BuildMemory::unlimited(); + let index = TextKeys::index(&id_map, &memory).unwrap(); + let relation = RelationDictionary::new().intern("r").unwrap(); + let mut batch = EdgeBatch::::with_capacity(&memory, 0, u64::MAX).unwrap(); + batch.push(relation, Some(b"a"), Some(b"b"), 1.0).unwrap(); + let cancellation = Cancellation::new(); + cancellation.cancel(); + assert_eq!(batch.map(&index, &cancellation), Err(Cancelled)); +} + +#[test] +fn a_batch_is_charged_to_its_memory() { + let memory = BuildMemory::new(1 << 20); + let relation = RelationDictionary::new().intern("r").unwrap(); + let mut batch = EdgeBatch::::with_capacity(&memory, 0, u64::MAX).unwrap(); + batch.push(relation, Some(&[7; 1000]), None, 1.0).unwrap(); + assert!(memory.in_use() >= 1000, "{}", memory.in_use()); + assert_eq!(batch.bytes(), memory.in_use(), "it says what it holds"); + batch.clear(); + assert_eq!(batch.bytes(), memory.in_use(), "and keeps it once cleared"); + let mut small = EdgeBatch::::with_capacity(&BuildMemory::new(100), 0, 0).unwrap(); + assert!(small.push(relation, Some(&[7; 1000]), None, 1.0).is_err()); +} + +#[test] +fn a_cleared_batch_frees_text_room_past_what_it_keeps() { + let memory = BuildMemory::new(1 << 24); + let relation = RelationDictionary::new().intern("r").unwrap(); + let mut batch = EdgeBatch::::with_capacity(&memory, 4, 2048).unwrap(); + batch.push(relation, Some(b"abc"), None, 1.0).unwrap(); + batch.push(relation, Some(b"de"), Some(b"f"), 1.0).unwrap(); + assert_eq!(batch.len_bytes(), 6, "the ids' bytes, not their room"); + let kept = batch.bytes(); + batch.clear(); + assert_eq!((batch.len_bytes(), batch.bytes()), (0, kept), "room within what it keeps stays"); + batch.push(relation, Some(&[7; 5000]), None, 1.0).unwrap(); + batch.clear(); + assert_eq!(batch.bytes(), memory.in_use()); + assert!(batch.bytes() < kept, "a long id's room is freed: {}", batch.bytes()); + let mut fixed = EdgeBatch::>::with_capacity(&memory, 4, 0).unwrap(); + fixed.push(relation, Some(&1), Some(&2), 1.0).unwrap(); + assert_eq!(fixed.len_bytes(), 0); +} + +#[test] +fn a_batch_has_room_for_its_rows_up_front() { + let memory = BuildMemory::new(1 << 20); + let relation = RelationDictionary::new().intern("r").unwrap(); + let mut batch = EdgeBatch::>::with_capacity(&memory, 10, 0).unwrap(); + let row = EdgeBatch::>::ROW_BYTES; + assert_eq!(memory.in_use(), 10 * row); + for at in 0..10 { + batch.push(relation, Some(&at), Some(&at), 1.0).unwrap(); + } + assert_eq!(memory.in_use(), 10 * row, "its rows fit"); + let text = EdgeBatch::::with_capacity(&memory, 10, 0).unwrap(); + assert_eq!( + text.bytes(), + 10 * EdgeBatch::::ROW_BYTES, + "a text id's bytes grow as they come" + ); + let too_many = EdgeBatch::>::with_capacity(&BuildMemory::new(row * 9), 10, 0); + assert!(matches!(too_many, Err(GraphError::BuildMemoryLimit { .. }))); +} + +#[test] +fn the_text_index_is_as_large_as_it_says_before_it_is_built() { + let (a, b, c): (&[u8], &[u8], &[u8]) = (b"a", b"b", b"c"); + for ids in [&[][..], &[a][..], &[a, b, c][..]] { + let id_map = text_map(ids, b"b"); + let index = TextIndex::new(&id_map, &BuildMemory::unlimited()).unwrap(); + assert_eq!(TextIndex::bytes_for(&id_map), index.bytes()); + } +} + +proptest! { + #[test] + fn batches_of_any_size_take_what_rows_take( + ids in prop::collection::vec(prop::collection::vec(0u8..4, 0..3), 0..12), + rows in prop::collection::vec( + (prop::option::of(prop::collection::vec(0u8..4, 0..3)), + prop::option::of(prop::collection::vec(0u8..4, 0..3))), + 0..40, + ), + size in 1usize..9, + directed: bool, + ) { + let ids: Vec<&[u8]> = ids.iter().map(Vec::as_slice).collect(); + let dropped = ids.first().copied().unwrap_or_default(); + let id_map = text_map(&ids, dropped); + let rows: Vec> = rows + .iter() + .enumerate() + .map(|(at, (s, t))| (s.as_deref(), t.as_deref(), at as f64)) + .collect(); + let expected = row_by_row(&id_map, directed, &rows); + for &map in MAPPED { + prop_assert_eq!(batched(&id_map, directed, &rows, size, map), expected.clone()); + } + } +} diff --git a/crates/warren-core/src/graph/edge_list.rs b/crates/warren-core/src/graph/edge_list.rs index e5fbc4f4..aebc5510 100644 --- a/crates/warren-core/src/graph/edge_list.rs +++ b/crates/warren-core/src/graph/edge_list.rs @@ -6,8 +6,8 @@ use std::mem::size_of; use super::build_memory::growth; use super::{ - BuildBuffer, BuildMemory, DenseId, GraphError, IdKeys, IdMap, Permutation, RelationIndex, - Target, TargetWidth, + BuildBuffer, BuildMemory, DenseId, EdgeBatch, GraphError, IdKeys, IdMap, IndexedKeys, + Permutation, RelationIndex, Target, TargetWidth, }; /// Edges in dense ids, as parallel arrays, in the order they were pushed. @@ -301,6 +301,18 @@ impl<'m, K: IdKeys> EdgeIntake<'m, K> { ) -> Result<(), GraphError> { let source = source.and_then(|id| self.id_map.dense_id(id)); let target = target.and_then(|id| self.id_map.dense_id(id)); + self.push_dense(relation, source, target, weight) + } + + /// Takes one edge row with its endpoints already mapped to dense ids: + /// `None` for a NULL, unknown or tombstoned one, which drops it. + fn push_dense( + &mut self, + relation: RelationIndex, + source: Option, + target: Option, + weight: f64, + ) -> Result<(), GraphError> { let (Some(source), Some(target)) = (source, target) else { self.dropped_edges += 1; return Ok(()); @@ -320,6 +332,20 @@ impl<'m, K: IdKeys> EdgeIntake<'m, K> { } } +impl EdgeIntake<'_, K> { + /// Takes every row of `batch`, in order, as [`EdgeIntake::push`] would, + /// then empties it. Endpoints are as [`EdgeBatch::map`] mapped them, + /// or looked up here if it was not mapped since its last row. + pub fn take(&mut self, batch: &mut EdgeBatch) -> Result<(), GraphError> { + for row in 0..batch.len() { + let (relation, source, target, weight) = batch.row(row, self.id_map); + self.push_dense(relation, source, target, weight)?; + } + batch.clear(); + Ok(()) + } +} + #[cfg(test)] mod tests { use super::*; diff --git a/crates/warren-core/src/graph/error.rs b/crates/warren-core/src/graph/error.rs index 6af5f3cc..03266625 100644 --- a/crates/warren-core/src/graph/error.rs +++ b/crates/warren-core/src/graph/error.rs @@ -155,7 +155,10 @@ pub enum GraphError { }, /// A text id map's bytes must be in the database's encoding (DESIGN.md /// §5.7). - #[error("text ids are in encoding {found}, expected the database's {expected}")] + #[error( + "text ids are in encoding {found}, not the database's ({expected}): create the graph \ + again to read its ids in this encoding" + )] IdEncoding { /// The database's encoding. expected: u8, diff --git a/crates/warren-core/src/graph/id_map/text/index.rs b/crates/warren-core/src/graph/id_map/text/index.rs index a92c7a14..61167661 100644 --- a/crates/warren-core/src/graph/id_map/text/index.rs +++ b/crates/warren-core/src/graph/id_map/text/index.rs @@ -59,6 +59,18 @@ impl<'m> TextIndex<'m> { Ok(Self { id_map, slots, hasher, _charge: charge }) } + /// The heap bytes the index of `id_map` holds, known before it is + /// built: two slots a live node (at least one), at the target width + /// of its id space. + pub fn bytes_for(id_map: &IdMap) -> u64 { + let slots = id_map.node_count().saturating_mul(2).max(1); + let width = match TargetWidth::for_id_space(id_map.id_space()) { + TargetWidth::U32 => size_of::(), + TargetWidth::U64 => size_of::(), + }; + slots.saturating_mul(width as u64) + } + /// The dense id of a live node, or `None` if `id` is no live node's. pub fn dense_id(&self, id: &[u8]) -> Option { let hash = self.hasher.hash_one(id); diff --git a/crates/warren-core/src/graph/mod.rs b/crates/warren-core/src/graph/mod.rs index 12e7ef4c..8d19a700 100644 --- a/crates/warren-core/src/graph/mod.rs +++ b/crates/warren-core/src/graph/mod.rs @@ -13,6 +13,7 @@ pub(crate) mod build_memory; mod csr; mod direction; +mod edge_batch; pub(crate) mod edge_list; mod error; mod id_map; @@ -31,6 +32,7 @@ pub use csr::{ }; pub(crate) use csr::{AnyCsrSlices, CsrSlices}; pub use direction::Direction; +pub use edge_batch::{EdgeBatch, Endpoints, FixedEndpoints, IndexedKeys, TextEndpoints}; pub use edge_list::{EdgeIntake, EdgeList}; pub use error::GraphError; pub use id_map::{ diff --git a/sql/pg_warren--0.6.0.sql b/sql/pg_warren--0.6.0.sql index cd41123b..184f072f 100644 --- a/sql/pg_warren--0.6.0.sql +++ b/sql/pg_warren--0.6.0.sql @@ -7,10 +7,168 @@ The ordering of items is not stable, it is driven by a dependency graph. /* */ /* */ --- crates/pg_warren/src/extension.rs:16 +-- crates/pg_warren/src/extension.rs:18 CREATE SCHEMA IF NOT EXISTS warren; /* pg_warren::extension::warren */ /* */ +/* */ +-- crates/pg_warren/src/compute_sql/library/mod.rs:13 +-- requires: +-- extension::warren + +-- warren.components_vector (DESIGN.md §6.7): weakly connected components by +-- min-label propagation (min_second SpMV both ways, accumulating min) until +-- no label changes. Each component is labelled by its smallest dense id, +-- as warren.components labels it. Writes vector `out` (i64) and returns +-- the iterations run. +CREATE FUNCTION warren.components_vector(graph text, "out" text, rels text[] DEFAULT NULL, + version bigint DEFAULT NULL) +RETURNS int LANGUAGE plpgsql SET search_path = pg_catalog, pg_temp AS $$ +DECLARE + v bigint := coalesce(version, (warren.info(graph) ->> 'current_version')::bigint); + parts text[] := ARRAY['x', 'next']; + delta float8; + i int := 0; +BEGIN + IF starts_with("out", '_warren_') THEN + RAISE EXCEPTION 'vector names beginning with _warren_ are reserved for the library''s intermediates' + USING ERRCODE = 'reserved_name'; + END IF; + BEGIN + PERFORM warren.vector_dense_id(graph, '_warren_components_vector_x', 'i64', false, v); + LOOP + i := i + 1; + PERFORM warren.vector_map(graph, '_warren_components_vector_next', '_warren_components_vector_x', false, v); + PERFORM warren.spmv(graph, '_warren_components_vector_x', '_warren_components_vector_next', 'min_second', + rels := rels, accumulate := 'min', persist := false, version := v); + PERFORM warren.spmv(graph, '_warren_components_vector_x', '_warren_components_vector_next', 'min_second', + transpose := true, rels := rels, accumulate := 'min', persist := false, version := v); + delta := warren.vector_norm_diff(graph, '_warren_components_vector_next', '_warren_components_vector_x', + 'l1', v, v); + PERFORM warren.vector_map(graph, '_warren_components_vector_x', '_warren_components_vector_next', false, v); + EXIT WHEN delta = 0; + END LOOP; + PERFORM warren.vector_map(graph, "out", '_warren_components_vector_x', true, v); + EXCEPTION WHEN OTHERS OR query_canceled THEN + PERFORM warren.drop_vector(graph, '_warren_components_vector_' || part, v, true) FROM unnest(parts) part; + RAISE; + END; + PERFORM warren.drop_vector(graph, '_warren_components_vector_' || part, v, true) FROM unnest(parts) part; + RETURN i; +END $$; +/* */ + +/* */ +-- crates/pg_warren/src/compute_sql/library/mod.rs:34 +-- requires: +-- extension::warren + +-- warren.katz_centrality (DESIGN.md §6.7): Katz centrality by power +-- iteration, as NetworkX computes it. Centrality flows along edges, +-- weighted by edge weight (spmv's rule: weights if the graph has them, +-- unless weighted := false), attenuated by alpha, plus each node's beta: +-- the constant `beta`, or beta × vector `seed` (NetworkX's per-node beta). +-- From 0, each iteration takes alpha · Aᵀx + β; it stops when the L1 change +-- falls below (live nodes) × tol, or after max_iter iterations with a +-- WARNING (NetworkX raises). Alpha at or above 1 / the largest eigenvalue +-- diverges: once a value or the L2 norm is not finite the run is refused +-- (22003), as NetworkX raises, and writes nothing. With normalized, the +-- result is scaled to unit L2 norm. Writes vector `out` (f64) and returns +-- the iterations run. Constants enter expressions as constant vectors; a +-- constant vector is 0 at tombstoned rows, where vector_map evaluates too, +-- so a division by one is guarded. +CREATE FUNCTION warren.katz_centrality(graph text, "out" text, alpha float8 DEFAULT 0.1, + beta float8 DEFAULT 1.0, seed text DEFAULT NULL, + normalized boolean DEFAULT true, tol float8 DEFAULT 1e-6, + max_iter int DEFAULT 1000, rels text[] DEFAULT NULL, + weighted boolean DEFAULT NULL, version bigint DEFAULT NULL) +RETURNS int LANGUAGE plpgsql SET search_path = pg_catalog, pg_temp AS $$ +DECLARE + v bigint := coalesce(version, (warren.info(graph) ->> 'current_version')::bigint); + parts text[] := ARRAY['one', 'b', 'alpha', 'x', 'y', 'next', 'scale']; + nodes float8; + norm float8; + delta float8; + i int := 0; +BEGIN + IF "out" IS NULL OR alpha IS NULL OR beta IS NULL OR normalized IS NULL OR tol IS NULL OR max_iter IS NULL THEN + RAISE EXCEPTION '% must not be NULL', + CASE WHEN "out" IS NULL THEN 'out' WHEN alpha IS NULL THEN 'alpha' WHEN beta IS NULL THEN 'beta' + WHEN normalized IS NULL THEN 'normalized' WHEN tol IS NULL THEN 'tol' ELSE 'max_iter' END + USING ERRCODE = 'null_value_not_allowed'; + END IF; + IF starts_with("out", '_warren_') OR starts_with(seed, '_warren_') THEN + RAISE EXCEPTION 'vector names beginning with _warren_ are reserved for the library''s intermediates' + USING ERRCODE = 'reserved_name'; + END IF; + IF tol < 0 THEN + RAISE EXCEPTION 'tol must be at least 0, not %', tol USING ERRCODE = 'invalid_parameter_value'; + END IF; + IF max_iter < 1 THEN + RAISE EXCEPTION 'max_iter must be at least 1, not %', max_iter USING ERRCODE = 'invalid_parameter_value'; + END IF; + BEGIN + PERFORM warren.vector_const(graph, '_warren_katz_centrality_one', 1, persist := false, version := v); + nodes := warren.vector_reduce(graph, '_warren_katz_centrality_one', 'sum', v); + IF nodes = 0 THEN + RAISE EXCEPTION 'graph "%" has no live nodes', graph USING ERRCODE = 'invalid_parameter_value'; + END IF; + IF seed IS NULL THEN + PERFORM warren.vector_const(graph, '_warren_katz_centrality_b', beta, persist := false, version := v); + ELSE + -- vector_reduce names a missing seed vector (42704); vector_map would + -- call it a missing column. + PERFORM warren.vector_reduce(graph, seed, 'nnz', v); + PERFORM warren.vector_map(graph, '_warren_katz_centrality_b', format('%I::float8', seed), false, v); + PERFORM warren.vector_const(graph, '_warren_katz_centrality_scale', beta, persist := false, version := v); + PERFORM warren.vector_map(graph, '_warren_katz_centrality_b', + '_warren_katz_centrality_scale * _warren_katz_centrality_b', false, v); + END IF; + PERFORM warren.vector_const(graph, '_warren_katz_centrality_alpha', alpha, persist := false, version := v); + PERFORM warren.vector_const(graph, '_warren_katz_centrality_x', 0, persist := false, version := v); + WHILE i < max_iter LOOP + i := i + 1; + PERFORM warren.spmv(graph, '_warren_katz_centrality_x', '_warren_katz_centrality_y', + transpose := true, rels := rels, weighted := weighted, persist := false, version := v); + PERFORM warren.vector_map(graph, '_warren_katz_centrality_next', + '_warren_katz_centrality_alpha * _warren_katz_centrality_y + _warren_katz_centrality_b', false, v); + delta := warren.vector_norm_diff(graph, '_warren_katz_centrality_next', '_warren_katz_centrality_x', + 'l1', v, v); + IF NOT delta < 'Infinity' THEN + RAISE EXCEPTION 'katz_centrality diverged: its values are not finite after % iterations', i + USING ERRCODE = 'numeric_value_out_of_range', + HINT = 'alpha must be below 1 / the largest eigenvalue of the adjacency matrix.'; + END IF; + PERFORM warren.vector_map(graph, '_warren_katz_centrality_x', '_warren_katz_centrality_next', false, v); + EXIT WHEN delta < nodes * tol; + END LOOP; + IF NOT delta < nodes * tol THEN + RAISE WARNING 'katz_centrality did not converge in % iterations', i; + END IF; + IF normalized THEN + norm := warren.vector_reduce(graph, '_warren_katz_centrality_x', 'l2', v); + IF NOT norm < 'Infinity' THEN + RAISE EXCEPTION 'katz_centrality diverged: its L2 norm overflows after % iterations', i + USING ERRCODE = 'numeric_value_out_of_range', + HINT = 'alpha must be below 1 / the largest eigenvalue of the adjacency matrix.'; + END IF; + PERFORM warren.vector_const(graph, '_warren_katz_centrality_scale', CASE WHEN norm = 0 THEN 1 ELSE norm END, + persist := false, version := v); + PERFORM warren.vector_map(graph, "out", + 'CASE WHEN _warren_katz_centrality_scale <> 0 + THEN _warren_katz_centrality_x / _warren_katz_centrality_scale ELSE 0 END', true, v); + ELSE + PERFORM warren.vector_map(graph, "out", '_warren_katz_centrality_x', true, v); + END IF; + EXCEPTION WHEN OTHERS OR query_canceled THEN + PERFORM warren.drop_vector(graph, '_warren_katz_centrality_' || part, v, true) FROM unnest(parts) part; + RAISE; + END; + PERFORM warren.drop_vector(graph, '_warren_katz_centrality_' || part, v, true) FROM unnest(parts) part; + RETURN i; +END $$; +/* */ + /* */ -- crates/pg_warren/src/compute_sql/library/mod.rs:40 -- requires: @@ -142,212 +300,73 @@ END $$; /* */ /* */ --- crates/pg_warren/src/compute_sql/library/mod.rs:7 +-- crates/pg_warren/src/compute_sql/library/mod.rs:18 -- requires: -- extension::warren --- warren.pagerank (DESIGN.md §6.7): PageRank by power iteration, as --- NetworkX computes it. Rank flows along edges, split by out-weight (the --- edge count when unweighted); dangling nodes' rank, and the 1 - damping --- teleport, go to the personalisation: uniform, or proportional to vector --- `seed`. Stops when the L1 change falls below (live nodes) × tol, or after --- max_iter iterations. Writes vector `out` (f64, summing to 1) and returns --- the iterations run. Constants enter expressions as constant vectors, --- never as text, so no client setting (extra_float_digits) rounds them. -CREATE FUNCTION warren.pagerank(graph text, "out" text, damping float8 DEFAULT 0.85, - tol float8 DEFAULT 1e-6, max_iter int DEFAULT 100, - seed text DEFAULT NULL, rels text[] DEFAULT NULL, - weighted boolean DEFAULT NULL, version bigint DEFAULT NULL) +-- warren.label_propagation (DESIGN.md §6.7, A.1): min-label propagation. +-- Labels start as vector `seed` (default: each node's dense id); each +-- iteration every node takes the smallest label among itself and its +-- neighbours (both directions, over `rels`), until no label changes or +-- after max_iter iterations. With `mask`, only the nodes it selects (a +-- non-zero value, GraphBLAS semantics) send or take labels; the others +-- keep theirs. It is deterministic min-propagation, not majority-vote +-- label propagation. Writes vector `out` (f64) and returns the iterations +-- run. +CREATE FUNCTION warren.label_propagation(graph text, "out" text, seed text DEFAULT NULL, + max_iter int DEFAULT 100, mask text DEFAULT NULL, + rels text[] DEFAULT NULL, version bigint DEFAULT NULL) RETURNS int LANGUAGE plpgsql SET search_path = pg_catalog, pg_temp AS $$ DECLARE v bigint := coalesce(version, (warren.info(graph) ->> 'current_version')::bigint); - parts text[] := ARRAY['one', 'out', 'x', 'p', 'total', 'damping', 'teleport', 'msg', 'dangling_x', - 'dangling', 'y', 'next']; - nodes float8; - dangling float8; + parts text[] := ARRAY['x', 'msg', 'next']; + selected text; delta float8; i int := 0; BEGIN - IF starts_with("out", '_warren_') OR starts_with(seed, '_warren_') THEN + IF starts_with("out", '_warren_') OR starts_with(seed, '_warren_') OR starts_with(mask, '_warren_') THEN RAISE EXCEPTION 'vector names beginning with _warren_ are reserved for the library''s intermediates' USING ERRCODE = 'reserved_name'; END IF; BEGIN - -- Every live node's out-weight: the row sums of A. - PERFORM warren.vector_const(graph, '_warren_pagerank_one', 1, persist := false, version := v); - PERFORM warren.spmv(graph, '_warren_pagerank_one', '_warren_pagerank_out', 'plus_first', - rels := rels, weighted := weighted, persist := false, version := v); - nodes := warren.vector_reduce(graph, '_warren_pagerank_one', 'sum', v); - PERFORM warren.vector_const(graph, '_warren_pagerank_x', 1.0 / nodes, persist := false, version := v); IF seed IS NULL THEN - PERFORM warren.vector_map(graph, '_warren_pagerank_p', '_warren_pagerank_x', false, v); + PERFORM warren.vector_dense_id(graph, '_warren_label_propagation_x', 'f64', false, v); ELSE - -- Times the reciprocal: a constant vector is 0 at tombstoned rows, - -- where vector_map evaluates too, so dividing by one would fail. - PERFORM warren.vector_const(graph, '_warren_pagerank_total', 1.0 / warren.vector_reduce(graph, seed, 'sum', v), - persist := false, version := v); - PERFORM warren.vector_map(graph, '_warren_pagerank_p', format('%I * _warren_pagerank_total', seed), false, v); + PERFORM warren.vector_map(graph, '_warren_label_propagation_x', format('%I::float8', seed), false, v); + END IF; + IF mask IS NOT NULL THEN + selected := CASE warren.explain_map(graph, format('%I', mask), v) ->> 'dtype' + WHEN 'bool' THEN format('%I', mask) ELSE format('%I <> 0', mask) END; END IF; - PERFORM warren.vector_const(graph, '_warren_pagerank_damping', damping, persist := false, version := v); - PERFORM warren.vector_const(graph, '_warren_pagerank_teleport', 1 - damping, persist := false, version := v); WHILE i < max_iter LOOP i := i + 1; - PERFORM warren.vector_map(graph, '_warren_pagerank_msg', - 'CASE WHEN _warren_pagerank_out > 0 THEN _warren_pagerank_x / _warren_pagerank_out ELSE 0 END', false, v); - PERFORM warren.vector_map(graph, '_warren_pagerank_dangling_x', - 'CASE WHEN _warren_pagerank_out > 0 THEN 0 ELSE _warren_pagerank_x END', false, v); - dangling := warren.vector_reduce(graph, '_warren_pagerank_dangling_x', 'sum', v); - PERFORM warren.vector_const(graph, '_warren_pagerank_dangling', dangling, persist := false, version := v); - PERFORM warren.spmv(graph, '_warren_pagerank_msg', '_warren_pagerank_y', transpose := true, - rels := rels, weighted := weighted, persist := false, version := v); - PERFORM warren.vector_map(graph, '_warren_pagerank_next', - '_warren_pagerank_damping * (_warren_pagerank_y + _warren_pagerank_dangling * _warren_pagerank_p) - + _warren_pagerank_teleport * _warren_pagerank_p', false, v); - delta := warren.vector_norm_diff(graph, '_warren_pagerank_next', '_warren_pagerank_x', 'l1', v, v); - PERFORM warren.vector_map(graph, '_warren_pagerank_x', '_warren_pagerank_next', false, v); - EXIT WHEN delta < nodes * tol; + -- Unselected nodes send NULL (NaN), which min never picks over a label. + PERFORM warren.vector_map(graph, '_warren_label_propagation_msg', + CASE WHEN selected IS NULL THEN '_warren_label_propagation_x' + ELSE format('CASE WHEN %s THEN _warren_label_propagation_x END', selected) END, false, v); + PERFORM warren.vector_map(graph, '_warren_label_propagation_next', '_warren_label_propagation_x', false, v); + PERFORM warren.spmv(graph, '_warren_label_propagation_msg', '_warren_label_propagation_next', 'min_second', + rels := rels, mask := mask, accumulate := 'min', persist := false, version := v); + PERFORM warren.spmv(graph, '_warren_label_propagation_msg', '_warren_label_propagation_next', 'min_second', + transpose := true, rels := rels, mask := mask, accumulate := 'min', persist := false, + version := v); + delta := warren.vector_norm_diff(graph, '_warren_label_propagation_next', '_warren_label_propagation_x', + 'l1', v, v); + PERFORM warren.vector_map(graph, '_warren_label_propagation_x', '_warren_label_propagation_next', false, v); + EXIT WHEN delta = 0; END LOOP; - PERFORM warren.vector_map(graph, "out", '_warren_pagerank_x', true, v); + PERFORM warren.vector_map(graph, "out", '_warren_label_propagation_x', true, v); EXCEPTION WHEN OTHERS OR query_canceled THEN - PERFORM warren.drop_vector(graph, '_warren_pagerank_' || part, v, true) FROM unnest(parts) part; + PERFORM warren.drop_vector(graph, '_warren_label_propagation_' || part, v, true) FROM unnest(parts) part; RAISE; END; - PERFORM warren.drop_vector(graph, '_warren_pagerank_' || part, v, true) FROM unnest(parts) part; + PERFORM warren.drop_vector(graph, '_warren_label_propagation_' || part, v, true) FROM unnest(parts) part; RETURN i; END $$; /* */ /* */ --- crates/pg_warren/src/compute_sql/library/mod.rs:41 --- requires: --- extension::warren - --- warren.label_propagation_majority (DESIGN.md §6.7): majority-vote label --- propagation, synchronous and deterministic. Labels start as vector `seed` --- (default: each node's dense id); each iteration every node takes the most --- frequent label among its own and its neighbours' (direction ignored, each --- neighbour once, over `rels`; a self-loop counts the node's own label once --- more; weights are not read), the smallest on a tie, until no label --- changes, or after max_iter iterations with a WARNING (a synchronous --- update can oscillate). NetworkX's label propagation is asynchronous --- (random order and ties) or semi-synchronous (by colouring), so neither --- matches step for step. Tier 2: each iteration is one `gather` (mode, with --- include_self), which counts the labels in-engine, N rows through the --- executor. Writes vector `out` (f64) and returns the iterations run. -CREATE FUNCTION warren.label_propagation_majority(graph text, "out" text, seed text DEFAULT NULL, - max_iter int DEFAULT 100, rels text[] DEFAULT NULL, - version bigint DEFAULT NULL) -RETURNS int LANGUAGE plpgsql SET search_path = pg_catalog, pg_temp AS $$ -DECLARE - info jsonb := warren.info(graph); - v bigint := coalesce(version, (info ->> 'current_version')::bigint); - parts text[] := ARRAY['x', 'next']; - -- Each node's own label and its neighbours', counted in-engine; the most - -- votes win, then the smallest label. - votes text := format( - 'SELECT node_id, value - FROM warren.gather(%1$L, %2$L, NULL::%3$s, NULL::float8, ''mode'', ''both'', %4$L::text[], - version := %5$s, include_self := true)', - graph, '_warren_label_propagation_majority_x', info ->> 'id_type', rels, v); - nodes bigint; - delta float8; - i int := 0; -BEGIN - IF "out" IS NULL OR max_iter IS NULL THEN - RAISE EXCEPTION '% must not be NULL', CASE WHEN "out" IS NULL THEN 'out' ELSE 'max_iter' END - USING ERRCODE = 'null_value_not_allowed'; - END IF; - IF starts_with("out", '_warren_') OR starts_with(seed, '_warren_') THEN - RAISE EXCEPTION 'vector names beginning with _warren_ are reserved for the library''s intermediates' - USING ERRCODE = 'reserved_name'; - END IF; - IF max_iter < 1 THEN - RAISE EXCEPTION 'max_iter must be at least 1, not %', max_iter USING ERRCODE = 'invalid_parameter_value'; - END IF; - BEGIN - IF seed IS NULL THEN - PERFORM warren.vector_dense_id(graph, '_warren_label_propagation_majority_x', 'f64', false, v); - ELSE - -- vector_reduce names a missing seed vector (42704); vector_map would - -- call it a missing column. - PERFORM warren.vector_reduce(graph, seed, 'nnz', v); - PERFORM warren.vector_map(graph, '_warren_label_propagation_majority_x', format('%I::float8', seed), false, v); - END IF; - -- No live nodes, no labels: no iterations (NetworkX finds no communities). - nodes := (warren.vector_stats(graph, '_warren_label_propagation_majority_x', v) ->> 'count')::bigint; - WHILE i < max_iter AND nodes > 0 LOOP - i := i + 1; - PERFORM warren.vector_from_query(graph, '_warren_label_propagation_majority_next', votes, 'f64', - persist := false, version := v); - delta := warren.vector_norm_diff(graph, '_warren_label_propagation_majority_next', - '_warren_label_propagation_majority_x', 'l1', v, v); - PERFORM warren.vector_map(graph, '_warren_label_propagation_majority_x', - '_warren_label_propagation_majority_next', false, v); - EXIT WHEN delta = 0; - END LOOP; - IF delta <> 0 THEN - RAISE WARNING 'label_propagation_majority did not converge in % iterations', i; - END IF; - PERFORM warren.vector_map(graph, "out", '_warren_label_propagation_majority_x', true, v); - EXCEPTION WHEN OTHERS OR query_canceled THEN - PERFORM warren.drop_vector(graph, '_warren_label_propagation_majority_' || part, v, true) - FROM unnest(parts) part; - RAISE; - END; - PERFORM warren.drop_vector(graph, '_warren_label_propagation_majority_' || part, v, true) FROM unnest(parts) part; - RETURN i; -END $$; -/* */ - -/* */ --- crates/pg_warren/src/compute_sql/library/mod.rs:13 --- requires: --- extension::warren - --- warren.components_vector (DESIGN.md §6.7): weakly connected components by --- min-label propagation (min_second SpMV both ways, accumulating min) until --- no label changes. Each component is labelled by its smallest dense id, --- as warren.components labels it. Writes vector `out` (i64) and returns --- the iterations run. -CREATE FUNCTION warren.components_vector(graph text, "out" text, rels text[] DEFAULT NULL, - version bigint DEFAULT NULL) -RETURNS int LANGUAGE plpgsql SET search_path = pg_catalog, pg_temp AS $$ -DECLARE - v bigint := coalesce(version, (warren.info(graph) ->> 'current_version')::bigint); - parts text[] := ARRAY['x', 'next']; - delta float8; - i int := 0; -BEGIN - IF starts_with("out", '_warren_') THEN - RAISE EXCEPTION 'vector names beginning with _warren_ are reserved for the library''s intermediates' - USING ERRCODE = 'reserved_name'; - END IF; - BEGIN - PERFORM warren.vector_dense_id(graph, '_warren_components_vector_x', 'i64', false, v); - LOOP - i := i + 1; - PERFORM warren.vector_map(graph, '_warren_components_vector_next', '_warren_components_vector_x', false, v); - PERFORM warren.spmv(graph, '_warren_components_vector_x', '_warren_components_vector_next', 'min_second', - rels := rels, accumulate := 'min', persist := false, version := v); - PERFORM warren.spmv(graph, '_warren_components_vector_x', '_warren_components_vector_next', 'min_second', - transpose := true, rels := rels, accumulate := 'min', persist := false, version := v); - delta := warren.vector_norm_diff(graph, '_warren_components_vector_next', '_warren_components_vector_x', - 'l1', v, v); - PERFORM warren.vector_map(graph, '_warren_components_vector_x', '_warren_components_vector_next', false, v); - EXIT WHEN delta = 0; - END LOOP; - PERFORM warren.vector_map(graph, "out", '_warren_components_vector_x', true, v); - EXCEPTION WHEN OTHERS OR query_canceled THEN - PERFORM warren.drop_vector(graph, '_warren_components_vector_' || part, v, true) FROM unnest(parts) part; - RAISE; - END; - PERFORM warren.drop_vector(graph, '_warren_components_vector_' || part, v, true) FROM unnest(parts) part; - RETURN i; -END $$; -/* */ - -/* */ --- crates/pg_warren/src/compute_sql/library/mod.rs:24 +-- crates/pg_warren/src/compute_sql/library/mod.rs:24 -- requires: -- extension::warren @@ -408,48 +427,52 @@ END $$; /* */ /* */ --- crates/pg_warren/src/compute_sql/library/mod.rs:34 +-- crates/pg_warren/src/compute_sql/library/mod.rs:39 -- requires: -- extension::warren --- warren.katz_centrality (DESIGN.md §6.7): Katz centrality by power --- iteration, as NetworkX computes it. Centrality flows along edges, --- weighted by edge weight (spmv's rule: weights if the graph has them, --- unless weighted := false), attenuated by alpha, plus each node's beta: --- the constant `beta`, or beta × vector `seed` (NetworkX's per-node beta). --- From 0, each iteration takes alpha · Aᵀx + β; it stops when the L1 change --- falls below (live nodes) × tol, or after max_iter iterations with a --- WARNING (NetworkX raises). Alpha at or above 1 / the largest eigenvalue --- diverges: once a value or the L2 norm is not finite the run is refused --- (22003), as NetworkX raises, and writes nothing. With normalized, the --- result is scaled to unit L2 norm. Writes vector `out` (f64) and returns --- the iterations run. Constants enter expressions as constant vectors; a --- constant vector is 0 at tombstoned rows, where vector_map evaluates too, --- so a division by one is guarded. -CREATE FUNCTION warren.katz_centrality(graph text, "out" text, alpha float8 DEFAULT 0.1, - beta float8 DEFAULT 1.0, seed text DEFAULT NULL, - normalized boolean DEFAULT true, tol float8 DEFAULT 1e-6, - max_iter int DEFAULT 1000, rels text[] DEFAULT NULL, - weighted boolean DEFAULT NULL, version bigint DEFAULT NULL) +-- warren.hits (DESIGN.md §6.7): HITS hubs and authorities by power +-- iteration, as NetworkX computes them (its default, method := +-- 'power_iteration'). From hubs uniform at 1/N, each iteration takes the +-- authorities a = Aᵀh (a node pointed at by good hubs) and the hubs h = A·a +-- (a node pointing at good authorities), weighted by edge weight (spmv's +-- rule: weights if the graph has them, unless weighted := false), and +-- scales h to a largest value of 1; it stops when the L1 change of h falls +-- below tol (not scaled by N, as NetworkX), or after max_iter iterations +-- with a WARNING (NetworkX raises). With normalized, each result sums to +-- 1; otherwise each has a largest value of 1. A zero sum or maximum (no +-- edges) leaves the zeros as they are. Values too large for double +-- precision are refused (22003) and write nothing. Writes vectors `hubs` +-- and `authorities` (f64) and returns the iterations run. A constant +-- vector is 0 at tombstoned rows, where vector_map evaluates too, so a +-- division by one is guarded. +CREATE FUNCTION warren.hits(graph text, hubs text, authorities text, tol float8 DEFAULT 1e-8, + max_iter int DEFAULT 100, normalized boolean DEFAULT true, + rels text[] DEFAULT NULL, weighted boolean DEFAULT NULL, + version bigint DEFAULT NULL) RETURNS int LANGUAGE plpgsql SET search_path = pg_catalog, pg_temp AS $$ DECLARE v bigint := coalesce(version, (warren.info(graph) ->> 'current_version')::bigint); - parts text[] := ARRAY['one', 'b', 'alpha', 'x', 'y', 'next', 'scale']; + parts text[] := ARRAY['one', 'x', 'a', 'next', 'scale']; nodes float8; - norm float8; + scale float8; + hub_scale float8; delta float8; i int := 0; BEGIN - IF "out" IS NULL OR alpha IS NULL OR beta IS NULL OR normalized IS NULL OR tol IS NULL OR max_iter IS NULL THEN + IF hubs IS NULL OR authorities IS NULL OR normalized IS NULL OR tol IS NULL OR max_iter IS NULL THEN RAISE EXCEPTION '% must not be NULL', - CASE WHEN "out" IS NULL THEN 'out' WHEN alpha IS NULL THEN 'alpha' WHEN beta IS NULL THEN 'beta' + CASE WHEN hubs IS NULL THEN 'hubs' WHEN authorities IS NULL THEN 'authorities' WHEN normalized IS NULL THEN 'normalized' WHEN tol IS NULL THEN 'tol' ELSE 'max_iter' END USING ERRCODE = 'null_value_not_allowed'; END IF; - IF starts_with("out", '_warren_') OR starts_with(seed, '_warren_') THEN + IF starts_with(hubs, '_warren_') OR starts_with(authorities, '_warren_') THEN RAISE EXCEPTION 'vector names beginning with _warren_ are reserved for the library''s intermediates' USING ERRCODE = 'reserved_name'; END IF; + IF hubs = authorities THEN + RAISE EXCEPTION 'hubs and authorities must be different vectors' USING ERRCODE = 'invalid_parameter_value'; + END IF; IF tol < 0 THEN RAISE EXCEPTION 'tol must be at least 0, not %', tol USING ERRCODE = 'invalid_parameter_value'; END IF; @@ -457,63 +480,60 @@ BEGIN RAISE EXCEPTION 'max_iter must be at least 1, not %', max_iter USING ERRCODE = 'invalid_parameter_value'; END IF; BEGIN - PERFORM warren.vector_const(graph, '_warren_katz_centrality_one', 1, persist := false, version := v); - nodes := warren.vector_reduce(graph, '_warren_katz_centrality_one', 'sum', v); + PERFORM warren.vector_const(graph, '_warren_hits_one', 1, persist := false, version := v); + nodes := warren.vector_reduce(graph, '_warren_hits_one', 'sum', v); IF nodes = 0 THEN RAISE EXCEPTION 'graph "%" has no live nodes', graph USING ERRCODE = 'invalid_parameter_value'; END IF; - IF seed IS NULL THEN - PERFORM warren.vector_const(graph, '_warren_katz_centrality_b', beta, persist := false, version := v); - ELSE - -- vector_reduce names a missing seed vector (42704); vector_map would - -- call it a missing column. - PERFORM warren.vector_reduce(graph, seed, 'nnz', v); - PERFORM warren.vector_map(graph, '_warren_katz_centrality_b', format('%I::float8', seed), false, v); - PERFORM warren.vector_const(graph, '_warren_katz_centrality_scale', beta, persist := false, version := v); - PERFORM warren.vector_map(graph, '_warren_katz_centrality_b', - '_warren_katz_centrality_scale * _warren_katz_centrality_b', false, v); - END IF; - PERFORM warren.vector_const(graph, '_warren_katz_centrality_alpha', alpha, persist := false, version := v); - PERFORM warren.vector_const(graph, '_warren_katz_centrality_x', 0, persist := false, version := v); + PERFORM warren.vector_const(graph, '_warren_hits_x', 1.0 / nodes, persist := false, version := v); WHILE i < max_iter LOOP i := i + 1; - PERFORM warren.spmv(graph, '_warren_katz_centrality_x', '_warren_katz_centrality_y', - transpose := true, rels := rels, weighted := weighted, persist := false, version := v); - PERFORM warren.vector_map(graph, '_warren_katz_centrality_next', - '_warren_katz_centrality_alpha * _warren_katz_centrality_y + _warren_katz_centrality_b', false, v); - delta := warren.vector_norm_diff(graph, '_warren_katz_centrality_next', '_warren_katz_centrality_x', - 'l1', v, v); - IF NOT delta < 'Infinity' THEN - RAISE EXCEPTION 'katz_centrality diverged: its values are not finite after % iterations', i + PERFORM warren.spmv(graph, '_warren_hits_x', '_warren_hits_a', transpose := true, rels := rels, + weighted := weighted, persist := false, version := v); + PERFORM warren.spmv(graph, '_warren_hits_a', '_warren_hits_next', rels := rels, weighted := weighted, + persist := false, version := v); + scale := warren.vector_reduce(graph, '_warren_hits_next', 'max', v); + IF NOT abs(scale) < 'Infinity' THEN + RAISE EXCEPTION 'hits diverged: its values are not finite after % iterations', i USING ERRCODE = 'numeric_value_out_of_range', - HINT = 'alpha must be below 1 / the largest eigenvalue of the adjacency matrix.'; + HINT = 'The edge weights are too large: their products overflow double precision.'; END IF; - PERFORM warren.vector_map(graph, '_warren_katz_centrality_x', '_warren_katz_centrality_next', false, v); - EXIT WHEN delta < nodes * tol; + PERFORM warren.vector_const(graph, '_warren_hits_scale', CASE WHEN scale = 0 THEN 1 ELSE scale END, + persist := false, version := v); + PERFORM warren.vector_map(graph, '_warren_hits_next', + 'CASE WHEN _warren_hits_scale <> 0 THEN _warren_hits_next / _warren_hits_scale ELSE 0 END', false, v); + delta := warren.vector_norm_diff(graph, '_warren_hits_next', '_warren_hits_x', 'l1', v, v); + PERFORM warren.vector_map(graph, '_warren_hits_x', '_warren_hits_next', false, v); + EXIT WHEN delta < tol; END LOOP; - IF NOT delta < nodes * tol THEN - RAISE WARNING 'katz_centrality did not converge in % iterations', i; + IF NOT delta < tol THEN + RAISE WARNING 'hits did not converge in % iterations', i; END IF; IF normalized THEN - norm := warren.vector_reduce(graph, '_warren_katz_centrality_x', 'l2', v); - IF NOT norm < 'Infinity' THEN - RAISE EXCEPTION 'katz_centrality diverged: its L2 norm overflows after % iterations', i - USING ERRCODE = 'numeric_value_out_of_range', - HINT = 'alpha must be below 1 / the largest eigenvalue of the adjacency matrix.'; - END IF; - PERFORM warren.vector_const(graph, '_warren_katz_centrality_scale', CASE WHEN norm = 0 THEN 1 ELSE norm END, - persist := false, version := v); - PERFORM warren.vector_map(graph, "out", - 'CASE WHEN _warren_katz_centrality_scale <> 0 - THEN _warren_katz_centrality_x / _warren_katz_centrality_scale ELSE 0 END', true, v); + hub_scale := warren.vector_reduce(graph, '_warren_hits_x', 'sum', v); + scale := warren.vector_reduce(graph, '_warren_hits_a', 'sum', v); ELSE - PERFORM warren.vector_map(graph, "out", '_warren_katz_centrality_x', true, v); + hub_scale := 1; + scale := warren.vector_reduce(graph, '_warren_hits_a', 'max', v); + END IF; + IF NOT abs(hub_scale) < 'Infinity' OR NOT abs(scale) < 'Infinity' THEN + RAISE EXCEPTION 'hits diverged: its values are not finite after % iterations', i + USING ERRCODE = 'numeric_value_out_of_range', + HINT = 'The edge weights are too large: their products overflow double precision.'; END IF; + PERFORM warren.vector_const(graph, '_warren_hits_scale', CASE WHEN hub_scale = 0 THEN 1 ELSE hub_scale END, + persist := false, version := v); + PERFORM warren.vector_map(graph, hubs, + 'CASE WHEN _warren_hits_scale <> 0 THEN _warren_hits_x / _warren_hits_scale ELSE 0 END', true, v); + PERFORM warren.vector_const(graph, '_warren_hits_scale', CASE WHEN scale = 0 THEN 1 ELSE scale END, + persist := false, version := v); + PERFORM warren.vector_map(graph, authorities, + 'CASE WHEN _warren_hits_scale <> 0 THEN _warren_hits_a / _warren_hits_scale ELSE 0 END', true, v); EXCEPTION WHEN OTHERS OR query_canceled THEN - PERFORM warren.drop_vector(graph, '_warren_katz_centrality_' || part, v, true) FROM unnest(parts) part; + PERFORM warren.drop_vector(graph, '_warren_hits_' || part, v, true) FROM unnest(parts) part; RAISE; END; - PERFORM warren.drop_vector(graph, '_warren_katz_centrality_' || part, v, true) FROM unnest(parts) part; + PERFORM warren.drop_vector(graph, '_warren_hits_' || part, v, true) FROM unnest(parts) part; RETURN i; END $$; /* */ @@ -580,7 +600,83 @@ END $$; /* */ /* */ --- crates/pg_warren/src/traversal_sql/step.rs:24 +-- crates/pg_warren/src/compute_sql/library/mod.rs:7 +-- requires: +-- extension::warren + +-- warren.pagerank (DESIGN.md §6.7): PageRank by power iteration, as +-- NetworkX computes it. Rank flows along edges, split by out-weight (the +-- edge count when unweighted); dangling nodes' rank, and the 1 - damping +-- teleport, go to the personalisation: uniform, or proportional to vector +-- `seed`. Stops when the L1 change falls below (live nodes) × tol, or after +-- max_iter iterations. Writes vector `out` (f64, summing to 1) and returns +-- the iterations run. Constants enter expressions as constant vectors, +-- never as text, so no client setting (extra_float_digits) rounds them. +CREATE FUNCTION warren.pagerank(graph text, "out" text, damping float8 DEFAULT 0.85, + tol float8 DEFAULT 1e-6, max_iter int DEFAULT 100, + seed text DEFAULT NULL, rels text[] DEFAULT NULL, + weighted boolean DEFAULT NULL, version bigint DEFAULT NULL) +RETURNS int LANGUAGE plpgsql SET search_path = pg_catalog, pg_temp AS $$ +DECLARE + v bigint := coalesce(version, (warren.info(graph) ->> 'current_version')::bigint); + parts text[] := ARRAY['one', 'out', 'x', 'p', 'total', 'damping', 'teleport', 'msg', 'dangling_x', + 'dangling', 'y', 'next']; + nodes float8; + dangling float8; + delta float8; + i int := 0; +BEGIN + IF starts_with("out", '_warren_') OR starts_with(seed, '_warren_') THEN + RAISE EXCEPTION 'vector names beginning with _warren_ are reserved for the library''s intermediates' + USING ERRCODE = 'reserved_name'; + END IF; + BEGIN + -- Every live node's out-weight: the row sums of A. + PERFORM warren.vector_const(graph, '_warren_pagerank_one', 1, persist := false, version := v); + PERFORM warren.spmv(graph, '_warren_pagerank_one', '_warren_pagerank_out', 'plus_first', + rels := rels, weighted := weighted, persist := false, version := v); + nodes := warren.vector_reduce(graph, '_warren_pagerank_one', 'sum', v); + PERFORM warren.vector_const(graph, '_warren_pagerank_x', 1.0 / nodes, persist := false, version := v); + IF seed IS NULL THEN + PERFORM warren.vector_map(graph, '_warren_pagerank_p', '_warren_pagerank_x', false, v); + ELSE + -- Times the reciprocal: a constant vector is 0 at tombstoned rows, + -- where vector_map evaluates too, so dividing by one would fail. + PERFORM warren.vector_const(graph, '_warren_pagerank_total', 1.0 / warren.vector_reduce(graph, seed, 'sum', v), + persist := false, version := v); + PERFORM warren.vector_map(graph, '_warren_pagerank_p', format('%I * _warren_pagerank_total', seed), false, v); + END IF; + PERFORM warren.vector_const(graph, '_warren_pagerank_damping', damping, persist := false, version := v); + PERFORM warren.vector_const(graph, '_warren_pagerank_teleport', 1 - damping, persist := false, version := v); + WHILE i < max_iter LOOP + i := i + 1; + PERFORM warren.vector_map(graph, '_warren_pagerank_msg', + 'CASE WHEN _warren_pagerank_out > 0 THEN _warren_pagerank_x / _warren_pagerank_out ELSE 0 END', false, v); + PERFORM warren.vector_map(graph, '_warren_pagerank_dangling_x', + 'CASE WHEN _warren_pagerank_out > 0 THEN 0 ELSE _warren_pagerank_x END', false, v); + dangling := warren.vector_reduce(graph, '_warren_pagerank_dangling_x', 'sum', v); + PERFORM warren.vector_const(graph, '_warren_pagerank_dangling', dangling, persist := false, version := v); + PERFORM warren.spmv(graph, '_warren_pagerank_msg', '_warren_pagerank_y', transpose := true, + rels := rels, weighted := weighted, persist := false, version := v); + PERFORM warren.vector_map(graph, '_warren_pagerank_next', + '_warren_pagerank_damping * (_warren_pagerank_y + _warren_pagerank_dangling * _warren_pagerank_p) + + _warren_pagerank_teleport * _warren_pagerank_p', false, v); + delta := warren.vector_norm_diff(graph, '_warren_pagerank_next', '_warren_pagerank_x', 'l1', v, v); + PERFORM warren.vector_map(graph, '_warren_pagerank_x', '_warren_pagerank_next', false, v); + EXIT WHEN delta < nodes * tol; + END LOOP; + PERFORM warren.vector_map(graph, "out", '_warren_pagerank_x', true, v); + EXCEPTION WHEN OTHERS OR query_canceled THEN + PERFORM warren.drop_vector(graph, '_warren_pagerank_' || part, v, true) FROM unnest(parts) part; + RAISE; + END; + PERFORM warren.drop_vector(graph, '_warren_pagerank_' || part, v, true) FROM unnest(parts) part; + RETURN i; +END $$; +/* */ + +/* */ +-- crates/pg_warren/src/traversal_sql/step.rs:25 -- requires: -- extension::warren @@ -645,86 +741,51 @@ CREATE CAST (jsonb AS warren.step[]) WITH FUNCTION warren.steps(jsonb); /* */ /* */ --- crates/pg_warren/src/compute_sql/library/mod.rs:8 --- requires: --- extension::warren - --- warren.degree_vector (DESIGN.md §6.7): each node's degree in `direction` --- over `rels`, as vector `out` (f64); returns its vector id. -CREATE FUNCTION warren.degree_vector(graph text, "out" text, direction text DEFAULT 'out', - rels text[] DEFAULT NULL, version bigint DEFAULT NULL) -RETURNS bigint LANGUAGE plpgsql SET search_path = pg_catalog, pg_temp AS $$ -BEGIN - IF starts_with("out", '_warren_') THEN - RAISE EXCEPTION 'vector names beginning with _warren_ are reserved for the library''s intermediates' - USING ERRCODE = 'reserved_name'; - END IF; - RETURN warren.vector_degree(graph, "out", direction, rels, 'f64', true, version); -END $$; -/* */ - -/* */ --- crates/pg_warren/src/community_sql/library/mod.rs:9 +-- crates/pg_warren/src/community_sql/library/mod.rs:8 -- requires: -- extension::warren --- warren.leiden (DESIGN.md §6.7): Leiden community detection (Traag, --- Waltman and van Eck 2019) as a library over the primitives. Each level --- moves nodes between communities (warren.local_moving, from the partition --- the level below found) on the level's graph, refines each community --- into well-connected parts (warren.leiden_refine), lifts both partitions --- to `graph` (warren.lift), and contracts `graph` by the refined one --- (warren.contract) into the next level's graph; the next level starts --- from the unrefined partition, projected onto it (warren.project, min). --- As warren.louvain's, every level's graph is a quotient of `graph`, and --- only one exists at a time. Every community found is connected. As --- Traag et al.'s, it goes on while there is something to aggregate: it --- stops when a level's local moving leaves every node alone, when the --- refinement leaves every node alone (the next level's graph would be this --- one), or after `max_levels`; in the last two cases it returns the --- refined partition, so every exit returns a connected one. A level's --- local moving stops when a sweep raises modularity by no more than --- `tolerance`. That multilevel pass is one iteration; each later one --- starts its first level's moving from the partition the one before found, --- not from each node alone, as igraph's `n_iterations` and leidenalg's --- `optimise_partition` loop do. The run stops after `iterations`, or --- after an iteration whose partition is the one it started from: with the --- same start and seed, the next would find it again. A negative --- `iterations` runs until then, at most 100 iterations (a WARNING if it --- is still changing). Arguments and output as warren.louvain's; --- `randomness` is θ of the refinement. `levels_prefix` keeps every --- iteration's levels in turn (`_1`, … numbered on across --- iterations), so the last is `out`. The level's graph, a level graph of --- warren.contract's (`_warren_level_`), is dropped at the end of --- each iteration. -CREATE FUNCTION warren.leiden(graph text, "out" text, resolution float8 DEFAULT 1.0, - seed bigint DEFAULT 0, max_levels int DEFAULT 100, - tolerance float8 DEFAULT 1e-7, randomness float8 DEFAULT 0.01, - iterations int DEFAULT 2, weighted boolean DEFAULT NULL, levels_prefix text DEFAULT NULL, - version bigint DEFAULT NULL) +-- warren.louvain (DESIGN.md §6.7): Louvain community detection (Blondel +-- et al. 2008) as a library over the primitives. Each level moves nodes +-- between communities (warren.local_moving, from each node alone) on the +-- level's graph, lifts the partition found to `graph` (warren.lift), and +-- contracts `graph` by it (warren.contract) into the next level's graph, +-- whose nodes are the communities: contracting `graph` by the composed +-- partition gives the graph that contracting the level's graph would, so +-- every level's graph is a quotient of `graph`, one lift away, and only +-- one exists at a time. It stops when a level merges no node, when a +-- level raises modularity by no more than `tolerance` (NetworkX's +-- `threshold`), or after `max_levels`. The graph is undirected for this +-- (an edge and its reverse add up; edges weighted if `weighted`, NULL: if +-- the graph has weights). Communities are labelled by the smallest dense +-- id of their members: a level's nodes are keyed by those dense ids. Writes +-- vector `out` (i64, on version `version`, NULL: the current one), with +-- each level's partition as `_` if `levels_prefix` +-- is given, and returns the partition's modularity. The level's graph, a +-- level graph of warren.contract's (`_warren_level_`), and the +-- intermediate vectors are dropped at the end. +CREATE FUNCTION warren.louvain(graph text, "out" text, resolution float8 DEFAULT 1.0, + seed bigint DEFAULT 0, max_levels int DEFAULT 100, + tolerance float8 DEFAULT 1e-7, weighted boolean DEFAULT NULL, + levels_prefix text DEFAULT NULL, version bigint DEFAULT NULL) RETURNS float8 LANGUAGE plpgsql SET search_path = pg_catalog, pg_temp AS $$ DECLARE v bigint := coalesce(version, (warren.info(graph) ->> 'current_version')::bigint); - parts text[] := ARRAY['moved', 'refined', 'initial', 'keys', 'down', 'refined_down', 'prev', 'changed']; + parts text[] := ARRAY['single', 'labels', 'keys', 'down']; g text := graph; gv bigint := v; next_g text; next_v bigint; - level int; - iteration int := 0; - written int := 0; + level int := 0; nodes bigint; found bigint; - refined bigint; q float8; + q_next float8; BEGIN - IF "out" IS NULL OR resolution IS NULL OR seed IS NULL OR max_levels IS NULL OR tolerance IS NULL - OR randomness IS NULL OR iterations IS NULL THEN + IF "out" IS NULL OR resolution IS NULL OR seed IS NULL OR max_levels IS NULL OR tolerance IS NULL THEN RAISE EXCEPTION '% must not be NULL', CASE WHEN "out" IS NULL THEN 'out' WHEN resolution IS NULL THEN 'resolution' - WHEN seed IS NULL THEN 'seed' WHEN max_levels IS NULL THEN 'max_levels' - WHEN tolerance IS NULL THEN 'tolerance' WHEN randomness IS NULL THEN 'randomness' - ELSE 'iterations' END + WHEN seed IS NULL THEN 'seed' WHEN max_levels IS NULL THEN 'max_levels' ELSE 'tolerance' END USING ERRCODE = 'null_value_not_allowed'; END IF; IF starts_with("out", '_warren_') OR starts_with(levels_prefix, '_warren_') THEN @@ -734,193 +795,23 @@ BEGIN IF max_levels < 1 THEN RAISE EXCEPTION 'max_levels must be at least 1, not %', max_levels USING ERRCODE = 'invalid_parameter_value'; END IF; - IF iterations = 0 THEN - RAISE EXCEPTION 'iterations must be at least 1, or negative to run until the partition is stable, not 0' - USING ERRCODE = 'invalid_parameter_value'; - END IF; IF tolerance = 'NaN'::float8 OR tolerance < 0 THEN RAISE EXCEPTION 'tolerance must be a number of at least 0, not %', tolerance USING ERRCODE = 'invalid_parameter_value'; END IF; BEGIN + PERFORM warren.vector_dense_id(graph, '_warren_louvain_single', persist := false, version := v); + q := warren.modularity(graph, '_warren_louvain_single', resolution, weighted, v); LOOP - iteration := iteration + 1; - -- A later iteration's first level moves from the last partition. - IF iteration > 1 THEN - PERFORM warren.vector_map(graph, '_warren_leiden_prev', '_warren_leiden_down', false, v); - END IF; - level := 0; - refined := NULL; - LOOP - level := level + 1; - nodes := (SELECT (e ->> 'node_count')::bigint FROM jsonb_array_elements(warren.info(g) -> 'versions') e - WHERE (e ->> 'version')::bigint = gv); - IF level > 1 THEN - PERFORM warren.vector_from_query(g, '_warren_leiden_keys', - format('SELECT node_id, node_id FROM warren.id_map(%L, NULL::bigint, %s)', g, gv), 'i64', - persist := false, version := gv); - END IF; - found := warren.local_moving(g, '_warren_leiden_moved', - initial := CASE WHEN level > 1 THEN '_warren_leiden_initial' - WHEN iteration > 1 THEN '_warren_leiden_prev' END, - keys := CASE WHEN level > 1 THEN '_warren_leiden_keys' END, - resolution := resolution, seed := seed, tolerance := tolerance, - weighted := CASE WHEN level = 1 THEN weighted ELSE true END, - persist := false, version := gv); - -- This level's partition, on `graph`. - IF level = 1 THEN - PERFORM warren.vector_map(graph, '_warren_leiden_down', '_warren_leiden_moved', false, v); - ELSE - PERFORM warren.lift(g, '_warren_leiden_moved', graph, '_warren_leiden_down', - persist := false, version := gv); - END IF; - -- Every node alone: the partition is this level's nodes, the refined - -- partition of the level below, which is connected. - IF found < nodes THEN - refined := warren.leiden_refine(g, '_warren_leiden_moved', '_warren_leiden_refined', - keys := CASE WHEN level > 1 THEN '_warren_leiden_keys' END, - resolution := resolution, seed := seed, randomness := randomness, - weighted := CASE WHEN level = 1 THEN weighted ELSE true END, - persist := false, version := gv); - IF level = 1 THEN - PERFORM warren.vector_map(graph, '_warren_leiden_refined_down', '_warren_leiden_refined', false, v); - ELSE - PERFORM warren.lift(g, '_warren_leiden_refined', graph, '_warren_leiden_refined_down', - persist := false, version := gv); - END IF; - -- Stopping here, return the refined partition, which is connected, - -- as Traag et al.'s is; the moved one need not be. - IF level >= max_levels OR refined = nodes THEN - PERFORM warren.vector_map(graph, '_warren_leiden_down', '_warren_leiden_refined_down', false, v); - END IF; - END IF; - IF levels_prefix IS NOT NULL THEN - written := written + 1; - PERFORM warren.vector_map(graph, format('%s_%s', levels_prefix, written), '_warren_leiden_down', true, v); - END IF; - EXIT WHEN found = nodes OR level >= max_levels OR refined = nodes; - -- A level graph of its own, named after its version, a name no - -- graph can have had. - next_v := warren.contract(graph, '_warren_leiden_refined_down', NULL, - weighted := coalesce(weighted, true), version := v); - next_g := '_warren_level_' || next_v; - -- The next level starts from the unrefined partition: every member - -- of a refined community is in the same community, so min is it. - PERFORM warren.project(graph, '_warren_leiden_down', next_g, '_warren_leiden_initial', 'min', - persist := false, version := next_v); - IF g <> graph THEN - PERFORM warren.drop_vector(g, '_warren_leiden_' || part, gv, true) FROM unnest(parts) part; - PERFORM warren.drop_graph(g); - END IF; - g := next_g; - gv := next_v; - END LOOP; - IF g <> graph THEN - PERFORM warren.drop_vector(g, '_warren_leiden_' || part, gv, true) FROM unnest(parts) part; - PERFORM warren.drop_graph(g); - g := graph; - gv := v; - END IF; - EXIT WHEN iteration = iterations; - -- Labels are the least dense id of each community's members, so the - -- partitions are the same exactly when the labels are. - IF iteration > 1 THEN - PERFORM warren.vector_map(graph, '_warren_leiden_changed', - 'CASE WHEN _warren_leiden_down = _warren_leiden_prev THEN 0 ELSE 1 END', false, v); - EXIT WHEN warren.vector_reduce(graph, '_warren_leiden_changed', 'sum', v) = 0; - END IF; - IF iterations < 0 AND iteration = 100 THEN - RAISE WARNING 'leiden did not stabilise in 100 iterations'; - EXIT; - END IF; - END LOOP; - PERFORM warren.vector_map(graph, "out", '_warren_leiden_down', true, v); - q := warren.modularity(graph, "out", resolution, weighted, v); - EXCEPTION WHEN OTHERS OR query_canceled THEN - PERFORM warren.drop_vector(graph, '_warren_leiden_' || part, v, true) FROM unnest(parts) part; - RAISE; - END; - PERFORM warren.drop_vector(graph, '_warren_leiden_' || part, v, true) FROM unnest(parts) part; - IF g <> graph THEN - PERFORM warren.drop_vector(g, '_warren_leiden_' || part, gv, true) FROM unnest(parts) part; - PERFORM warren.drop_graph(g); - END IF; - RETURN q; -END $$; -/* */ - -/* */ --- crates/pg_warren/src/community_sql/library/mod.rs:8 --- requires: --- extension::warren - --- warren.louvain (DESIGN.md §6.7): Louvain community detection (Blondel --- et al. 2008) as a library over the primitives. Each level moves nodes --- between communities (warren.local_moving, from each node alone) on the --- level's graph, lifts the partition found to `graph` (warren.lift), and --- contracts `graph` by it (warren.contract) into the next level's graph, --- whose nodes are the communities: contracting `graph` by the composed --- partition gives the graph that contracting the level's graph would, so --- every level's graph is a quotient of `graph`, one lift away, and only --- one exists at a time. It stops when a level merges no node, when a --- level raises modularity by no more than `tolerance` (NetworkX's --- `threshold`), or after `max_levels`. The graph is undirected for this --- (an edge and its reverse add up; edges weighted if `weighted`, NULL: if --- the graph has weights). Communities are labelled by the smallest dense --- id of their members: a level's nodes are keyed by those dense ids. Writes --- vector `out` (i64, on version `version`, NULL: the current one), with --- each level's partition as `_` if `levels_prefix` --- is given, and returns the partition's modularity. The level's graph, a --- level graph of warren.contract's (`_warren_level_`), and the --- intermediate vectors are dropped at the end. -CREATE FUNCTION warren.louvain(graph text, "out" text, resolution float8 DEFAULT 1.0, - seed bigint DEFAULT 0, max_levels int DEFAULT 100, - tolerance float8 DEFAULT 1e-7, weighted boolean DEFAULT NULL, - levels_prefix text DEFAULT NULL, version bigint DEFAULT NULL) -RETURNS float8 LANGUAGE plpgsql SET search_path = pg_catalog, pg_temp AS $$ -DECLARE - v bigint := coalesce(version, (warren.info(graph) ->> 'current_version')::bigint); - parts text[] := ARRAY['single', 'labels', 'keys', 'down']; - g text := graph; - gv bigint := v; - next_g text; - next_v bigint; - level int := 0; - nodes bigint; - found bigint; - q float8; - q_next float8; -BEGIN - IF "out" IS NULL OR resolution IS NULL OR seed IS NULL OR max_levels IS NULL OR tolerance IS NULL THEN - RAISE EXCEPTION '% must not be NULL', - CASE WHEN "out" IS NULL THEN 'out' WHEN resolution IS NULL THEN 'resolution' - WHEN seed IS NULL THEN 'seed' WHEN max_levels IS NULL THEN 'max_levels' ELSE 'tolerance' END - USING ERRCODE = 'null_value_not_allowed'; - END IF; - IF starts_with("out", '_warren_') OR starts_with(levels_prefix, '_warren_') THEN - RAISE EXCEPTION 'vector names beginning with _warren_ are reserved for the library''s intermediates' - USING ERRCODE = 'reserved_name'; - END IF; - IF max_levels < 1 THEN - RAISE EXCEPTION 'max_levels must be at least 1, not %', max_levels USING ERRCODE = 'invalid_parameter_value'; - END IF; - IF tolerance = 'NaN'::float8 OR tolerance < 0 THEN - RAISE EXCEPTION 'tolerance must be a number of at least 0, not %', tolerance - USING ERRCODE = 'invalid_parameter_value'; - END IF; - BEGIN - PERFORM warren.vector_dense_id(graph, '_warren_louvain_single', persist := false, version := v); - q := warren.modularity(graph, '_warren_louvain_single', resolution, weighted, v); - LOOP - level := level + 1; - nodes := (SELECT (e ->> 'node_count')::bigint FROM jsonb_array_elements(warren.info(g) -> 'versions') e - WHERE (e ->> 'version')::bigint = gv); - IF level > 1 THEN - -- A level's nodes are the communities found below it, keyed by - -- the least dense id of their members in `graph`. - PERFORM warren.vector_from_query(g, '_warren_louvain_keys', - format('SELECT node_id, node_id FROM warren.id_map(%L, NULL::bigint, %s)', g, gv), 'i64', - persist := false, version := gv); + level := level + 1; + nodes := (SELECT (e ->> 'node_count')::bigint FROM jsonb_array_elements(warren.info(g) -> 'versions') e + WHERE (e ->> 'version')::bigint = gv); + IF level > 1 THEN + -- A level's nodes are the communities found below it, keyed by + -- the least dense id of their members in `graph`. + PERFORM warren.vector_from_query(g, '_warren_louvain_keys', + format('SELECT node_id, node_id FROM warren.id_map(%L, NULL::bigint, %s)', g, gv), 'i64', + persist := false, version := gv); END IF; found := warren.local_moving(g, '_warren_louvain_labels', keys := CASE WHEN level > 1 THEN '_warren_louvain_keys' END, @@ -970,346 +861,425 @@ END $$; /* */ /* */ --- crates/pg_warren/src/compute_sql/library/mod.rs:18 +-- crates/pg_warren/src/compute_sql/library/mod.rs:41 -- requires: -- extension::warren --- warren.label_propagation (DESIGN.md §6.7, A.1): min-label propagation. --- Labels start as vector `seed` (default: each node's dense id); each --- iteration every node takes the smallest label among itself and its --- neighbours (both directions, over `rels`), until no label changes or --- after max_iter iterations. With `mask`, only the nodes it selects (a --- non-zero value, GraphBLAS semantics) send or take labels; the others --- keep theirs. It is deterministic min-propagation, not majority-vote --- label propagation. Writes vector `out` (f64) and returns the iterations --- run. -CREATE FUNCTION warren.label_propagation(graph text, "out" text, seed text DEFAULT NULL, - max_iter int DEFAULT 100, mask text DEFAULT NULL, - rels text[] DEFAULT NULL, version bigint DEFAULT NULL) +-- warren.label_propagation_majority (DESIGN.md §6.7): majority-vote label +-- propagation, synchronous and deterministic. Labels start as vector `seed` +-- (default: each node's dense id); each iteration every node takes the most +-- frequent label among its own and its neighbours' (direction ignored, each +-- neighbour once, over `rels`; a self-loop counts the node's own label once +-- more; weights are not read), the smallest on a tie, until no label +-- changes, or after max_iter iterations with a WARNING (a synchronous +-- update can oscillate). NetworkX's label propagation is asynchronous +-- (random order and ties) or semi-synchronous (by colouring), so neither +-- matches step for step. Tier 2: each iteration is one `gather` (mode, with +-- include_self), which counts the labels in-engine, N rows through the +-- executor. Writes vector `out` (f64) and returns the iterations run. +CREATE FUNCTION warren.label_propagation_majority(graph text, "out" text, seed text DEFAULT NULL, + max_iter int DEFAULT 100, rels text[] DEFAULT NULL, + version bigint DEFAULT NULL) RETURNS int LANGUAGE plpgsql SET search_path = pg_catalog, pg_temp AS $$ DECLARE - v bigint := coalesce(version, (warren.info(graph) ->> 'current_version')::bigint); - parts text[] := ARRAY['x', 'msg', 'next']; - selected text; + info jsonb := warren.info(graph); + v bigint := coalesce(version, (info ->> 'current_version')::bigint); + parts text[] := ARRAY['x', 'next']; + -- Each node's own label and its neighbours', counted in-engine; the most + -- votes win, then the smallest label. + votes text := format( + 'SELECT node_id, value + FROM warren.gather(%1$L, %2$L, NULL::%3$s, NULL::float8, ''mode'', ''both'', %4$L::text[], + version := %5$s, include_self := true)', + graph, '_warren_label_propagation_majority_x', info ->> 'id_type', rels, v); + nodes bigint; delta float8; i int := 0; BEGIN - IF starts_with("out", '_warren_') OR starts_with(seed, '_warren_') OR starts_with(mask, '_warren_') THEN + IF "out" IS NULL OR max_iter IS NULL THEN + RAISE EXCEPTION '% must not be NULL', CASE WHEN "out" IS NULL THEN 'out' ELSE 'max_iter' END + USING ERRCODE = 'null_value_not_allowed'; + END IF; + IF starts_with("out", '_warren_') OR starts_with(seed, '_warren_') THEN RAISE EXCEPTION 'vector names beginning with _warren_ are reserved for the library''s intermediates' USING ERRCODE = 'reserved_name'; END IF; + IF max_iter < 1 THEN + RAISE EXCEPTION 'max_iter must be at least 1, not %', max_iter USING ERRCODE = 'invalid_parameter_value'; + END IF; BEGIN IF seed IS NULL THEN - PERFORM warren.vector_dense_id(graph, '_warren_label_propagation_x', 'f64', false, v); + PERFORM warren.vector_dense_id(graph, '_warren_label_propagation_majority_x', 'f64', false, v); ELSE - PERFORM warren.vector_map(graph, '_warren_label_propagation_x', format('%I::float8', seed), false, v); - END IF; - IF mask IS NOT NULL THEN - selected := CASE warren.explain_map(graph, format('%I', mask), v) ->> 'dtype' - WHEN 'bool' THEN format('%I', mask) ELSE format('%I <> 0', mask) END; + -- vector_reduce names a missing seed vector (42704); vector_map would + -- call it a missing column. + PERFORM warren.vector_reduce(graph, seed, 'nnz', v); + PERFORM warren.vector_map(graph, '_warren_label_propagation_majority_x', format('%I::float8', seed), false, v); END IF; - WHILE i < max_iter LOOP + -- No live nodes, no labels: no iterations (NetworkX finds no communities). + nodes := (warren.vector_stats(graph, '_warren_label_propagation_majority_x', v) ->> 'count')::bigint; + WHILE i < max_iter AND nodes > 0 LOOP i := i + 1; - -- Unselected nodes send NULL (NaN), which min never picks over a label. - PERFORM warren.vector_map(graph, '_warren_label_propagation_msg', - CASE WHEN selected IS NULL THEN '_warren_label_propagation_x' - ELSE format('CASE WHEN %s THEN _warren_label_propagation_x END', selected) END, false, v); - PERFORM warren.vector_map(graph, '_warren_label_propagation_next', '_warren_label_propagation_x', false, v); - PERFORM warren.spmv(graph, '_warren_label_propagation_msg', '_warren_label_propagation_next', 'min_second', - rels := rels, mask := mask, accumulate := 'min', persist := false, version := v); - PERFORM warren.spmv(graph, '_warren_label_propagation_msg', '_warren_label_propagation_next', 'min_second', - transpose := true, rels := rels, mask := mask, accumulate := 'min', persist := false, - version := v); - delta := warren.vector_norm_diff(graph, '_warren_label_propagation_next', '_warren_label_propagation_x', - 'l1', v, v); - PERFORM warren.vector_map(graph, '_warren_label_propagation_x', '_warren_label_propagation_next', false, v); + PERFORM warren.vector_from_query(graph, '_warren_label_propagation_majority_next', votes, 'f64', + persist := false, version := v); + delta := warren.vector_norm_diff(graph, '_warren_label_propagation_majority_next', + '_warren_label_propagation_majority_x', 'l1', v, v); + PERFORM warren.vector_map(graph, '_warren_label_propagation_majority_x', + '_warren_label_propagation_majority_next', false, v); EXIT WHEN delta = 0; END LOOP; - PERFORM warren.vector_map(graph, "out", '_warren_label_propagation_x', true, v); + IF delta <> 0 THEN + RAISE WARNING 'label_propagation_majority did not converge in % iterations', i; + END IF; + PERFORM warren.vector_map(graph, "out", '_warren_label_propagation_majority_x', true, v); EXCEPTION WHEN OTHERS OR query_canceled THEN - PERFORM warren.drop_vector(graph, '_warren_label_propagation_' || part, v, true) FROM unnest(parts) part; + PERFORM warren.drop_vector(graph, '_warren_label_propagation_majority_' || part, v, true) + FROM unnest(parts) part; RAISE; END; - PERFORM warren.drop_vector(graph, '_warren_label_propagation_' || part, v, true) FROM unnest(parts) part; + PERFORM warren.drop_vector(graph, '_warren_label_propagation_majority_' || part, v, true) FROM unnest(parts) part; RETURN i; END $$; /* */ /* */ --- crates/pg_warren/src/compute_sql/library/mod.rs:39 +-- crates/pg_warren/src/compute_sql/library/mod.rs:8 -- requires: -- extension::warren --- warren.hits (DESIGN.md §6.7): HITS hubs and authorities by power --- iteration, as NetworkX computes them (its default, method := --- 'power_iteration'). From hubs uniform at 1/N, each iteration takes the --- authorities a = Aᵀh (a node pointed at by good hubs) and the hubs h = A·a --- (a node pointing at good authorities), weighted by edge weight (spmv's --- rule: weights if the graph has them, unless weighted := false), and --- scales h to a largest value of 1; it stops when the L1 change of h falls --- below tol (not scaled by N, as NetworkX), or after max_iter iterations --- with a WARNING (NetworkX raises). With normalized, each result sums to --- 1; otherwise each has a largest value of 1. A zero sum or maximum (no --- edges) leaves the zeros as they are. Values too large for double --- precision are refused (22003) and write nothing. Writes vectors `hubs` --- and `authorities` (f64) and returns the iterations run. A constant --- vector is 0 at tombstoned rows, where vector_map evaluates too, so a --- division by one is guarded. -CREATE FUNCTION warren.hits(graph text, hubs text, authorities text, tol float8 DEFAULT 1e-8, - max_iter int DEFAULT 100, normalized boolean DEFAULT true, - rels text[] DEFAULT NULL, weighted boolean DEFAULT NULL, - version bigint DEFAULT NULL) -RETURNS int LANGUAGE plpgsql SET search_path = pg_catalog, pg_temp AS $$ +-- warren.degree_vector (DESIGN.md §6.7): each node's degree in `direction` +-- over `rels`, as vector `out` (f64); returns its vector id. +CREATE FUNCTION warren.degree_vector(graph text, "out" text, direction text DEFAULT 'out', + rels text[] DEFAULT NULL, version bigint DEFAULT NULL) +RETURNS bigint LANGUAGE plpgsql SET search_path = pg_catalog, pg_temp AS $$ +BEGIN + IF starts_with("out", '_warren_') THEN + RAISE EXCEPTION 'vector names beginning with _warren_ are reserved for the library''s intermediates' + USING ERRCODE = 'reserved_name'; + END IF; + RETURN warren.vector_degree(graph, "out", direction, rels, 'f64', true, version); +END $$; +/* */ + +/* */ +-- crates/pg_warren/src/community_sql/library/mod.rs:9 +-- requires: +-- extension::warren + +-- warren.leiden (DESIGN.md §6.7): Leiden community detection (Traag, +-- Waltman and van Eck 2019) as a library over the primitives. Each level +-- moves nodes between communities (warren.local_moving, from the partition +-- the level below found) on the level's graph, refines each community +-- into well-connected parts (warren.leiden_refine), lifts both partitions +-- to `graph` (warren.lift), and contracts `graph` by the refined one +-- (warren.contract) into the next level's graph; the next level starts +-- from the unrefined partition, projected onto it (warren.project, min). +-- As warren.louvain's, every level's graph is a quotient of `graph`, and +-- only one exists at a time. Every community found is connected. As +-- Traag et al.'s, it goes on while there is something to aggregate: it +-- stops when a level's local moving leaves every node alone, when the +-- refinement leaves every node alone (the next level's graph would be this +-- one), or after `max_levels`; in the last two cases it returns the +-- refined partition, so every exit returns a connected one. A level's +-- local moving stops when a sweep raises modularity by no more than +-- `tolerance`. That multilevel pass is one iteration; each later one +-- starts its first level's moving from the partition the one before found, +-- not from each node alone, as igraph's `n_iterations` and leidenalg's +-- `optimise_partition` loop do. The run stops after `iterations`, or +-- after an iteration whose partition is the one it started from: with the +-- same start and seed, the next would find it again. A negative +-- `iterations` runs until then, at most 100 iterations (a WARNING if it +-- is still changing). Arguments and output as warren.louvain's; +-- `randomness` is θ of the refinement. `levels_prefix` keeps every +-- iteration's levels in turn (`_1`, … numbered on across +-- iterations), so the last is `out`. The level's graph, a level graph of +-- warren.contract's (`_warren_level_`), is dropped at the end of +-- each iteration. +CREATE FUNCTION warren.leiden(graph text, "out" text, resolution float8 DEFAULT 1.0, + seed bigint DEFAULT 0, max_levels int DEFAULT 100, + tolerance float8 DEFAULT 1e-7, randomness float8 DEFAULT 0.01, + iterations int DEFAULT 2, weighted boolean DEFAULT NULL, levels_prefix text DEFAULT NULL, + version bigint DEFAULT NULL) +RETURNS float8 LANGUAGE plpgsql SET search_path = pg_catalog, pg_temp AS $$ DECLARE v bigint := coalesce(version, (warren.info(graph) ->> 'current_version')::bigint); - parts text[] := ARRAY['one', 'x', 'a', 'next', 'scale']; - nodes float8; - scale float8; - hub_scale float8; - delta float8; - i int := 0; + parts text[] := ARRAY['moved', 'refined', 'initial', 'keys', 'down', 'refined_down', 'prev', 'changed']; + g text := graph; + gv bigint := v; + next_g text; + next_v bigint; + level int; + iteration int := 0; + written int := 0; + nodes bigint; + found bigint; + refined bigint; + q float8; BEGIN - IF hubs IS NULL OR authorities IS NULL OR normalized IS NULL OR tol IS NULL OR max_iter IS NULL THEN + IF "out" IS NULL OR resolution IS NULL OR seed IS NULL OR max_levels IS NULL OR tolerance IS NULL + OR randomness IS NULL OR iterations IS NULL THEN RAISE EXCEPTION '% must not be NULL', - CASE WHEN hubs IS NULL THEN 'hubs' WHEN authorities IS NULL THEN 'authorities' - WHEN normalized IS NULL THEN 'normalized' WHEN tol IS NULL THEN 'tol' ELSE 'max_iter' END + CASE WHEN "out" IS NULL THEN 'out' WHEN resolution IS NULL THEN 'resolution' + WHEN seed IS NULL THEN 'seed' WHEN max_levels IS NULL THEN 'max_levels' + WHEN tolerance IS NULL THEN 'tolerance' WHEN randomness IS NULL THEN 'randomness' + ELSE 'iterations' END USING ERRCODE = 'null_value_not_allowed'; END IF; - IF starts_with(hubs, '_warren_') OR starts_with(authorities, '_warren_') THEN + IF starts_with("out", '_warren_') OR starts_with(levels_prefix, '_warren_') THEN RAISE EXCEPTION 'vector names beginning with _warren_ are reserved for the library''s intermediates' USING ERRCODE = 'reserved_name'; END IF; - IF hubs = authorities THEN - RAISE EXCEPTION 'hubs and authorities must be different vectors' USING ERRCODE = 'invalid_parameter_value'; + IF max_levels < 1 THEN + RAISE EXCEPTION 'max_levels must be at least 1, not %', max_levels USING ERRCODE = 'invalid_parameter_value'; END IF; - IF tol < 0 THEN - RAISE EXCEPTION 'tol must be at least 0, not %', tol USING ERRCODE = 'invalid_parameter_value'; + IF iterations = 0 THEN + RAISE EXCEPTION 'iterations must be at least 1, or negative to run until the partition is stable, not 0' + USING ERRCODE = 'invalid_parameter_value'; END IF; - IF max_iter < 1 THEN - RAISE EXCEPTION 'max_iter must be at least 1, not %', max_iter USING ERRCODE = 'invalid_parameter_value'; + IF tolerance = 'NaN'::float8 OR tolerance < 0 THEN + RAISE EXCEPTION 'tolerance must be a number of at least 0, not %', tolerance + USING ERRCODE = 'invalid_parameter_value'; END IF; BEGIN - PERFORM warren.vector_const(graph, '_warren_hits_one', 1, persist := false, version := v); - nodes := warren.vector_reduce(graph, '_warren_hits_one', 'sum', v); - IF nodes = 0 THEN - RAISE EXCEPTION 'graph "%" has no live nodes', graph USING ERRCODE = 'invalid_parameter_value'; - END IF; - PERFORM warren.vector_const(graph, '_warren_hits_x', 1.0 / nodes, persist := false, version := v); - WHILE i < max_iter LOOP - i := i + 1; - PERFORM warren.spmv(graph, '_warren_hits_x', '_warren_hits_a', transpose := true, rels := rels, - weighted := weighted, persist := false, version := v); - PERFORM warren.spmv(graph, '_warren_hits_a', '_warren_hits_next', rels := rels, weighted := weighted, - persist := false, version := v); - scale := warren.vector_reduce(graph, '_warren_hits_next', 'max', v); - IF NOT abs(scale) < 'Infinity' THEN - RAISE EXCEPTION 'hits diverged: its values are not finite after % iterations', i - USING ERRCODE = 'numeric_value_out_of_range', - HINT = 'The edge weights are too large: their products overflow double precision.'; + LOOP + iteration := iteration + 1; + -- A later iteration's first level moves from the last partition. + IF iteration > 1 THEN + PERFORM warren.vector_map(graph, '_warren_leiden_prev', '_warren_leiden_down', false, v); + END IF; + level := 0; + refined := NULL; + LOOP + level := level + 1; + nodes := (SELECT (e ->> 'node_count')::bigint FROM jsonb_array_elements(warren.info(g) -> 'versions') e + WHERE (e ->> 'version')::bigint = gv); + IF level > 1 THEN + PERFORM warren.vector_from_query(g, '_warren_leiden_keys', + format('SELECT node_id, node_id FROM warren.id_map(%L, NULL::bigint, %s)', g, gv), 'i64', + persist := false, version := gv); + END IF; + found := warren.local_moving(g, '_warren_leiden_moved', + initial := CASE WHEN level > 1 THEN '_warren_leiden_initial' + WHEN iteration > 1 THEN '_warren_leiden_prev' END, + keys := CASE WHEN level > 1 THEN '_warren_leiden_keys' END, + resolution := resolution, seed := seed, tolerance := tolerance, + weighted := CASE WHEN level = 1 THEN weighted ELSE true END, + persist := false, version := gv); + -- This level's partition, on `graph`. + IF level = 1 THEN + PERFORM warren.vector_map(graph, '_warren_leiden_down', '_warren_leiden_moved', false, v); + ELSE + PERFORM warren.lift(g, '_warren_leiden_moved', graph, '_warren_leiden_down', + persist := false, version := gv); + END IF; + -- Every node alone: the partition is this level's nodes, the refined + -- partition of the level below, which is connected. + IF found < nodes THEN + refined := warren.leiden_refine(g, '_warren_leiden_moved', '_warren_leiden_refined', + keys := CASE WHEN level > 1 THEN '_warren_leiden_keys' END, + resolution := resolution, seed := seed, randomness := randomness, + weighted := CASE WHEN level = 1 THEN weighted ELSE true END, + persist := false, version := gv); + IF level = 1 THEN + PERFORM warren.vector_map(graph, '_warren_leiden_refined_down', '_warren_leiden_refined', false, v); + ELSE + PERFORM warren.lift(g, '_warren_leiden_refined', graph, '_warren_leiden_refined_down', + persist := false, version := gv); + END IF; + -- Stopping here, return the refined partition, which is connected, + -- as Traag et al.'s is; the moved one need not be. + IF level >= max_levels OR refined = nodes THEN + PERFORM warren.vector_map(graph, '_warren_leiden_down', '_warren_leiden_refined_down', false, v); + END IF; + END IF; + IF levels_prefix IS NOT NULL THEN + written := written + 1; + PERFORM warren.vector_map(graph, format('%s_%s', levels_prefix, written), '_warren_leiden_down', true, v); + END IF; + EXIT WHEN found = nodes OR level >= max_levels OR refined = nodes; + -- A level graph of its own, named after its version, a name no + -- graph can have had. + next_v := warren.contract(graph, '_warren_leiden_refined_down', NULL, + weighted := coalesce(weighted, true), version := v); + next_g := '_warren_level_' || next_v; + -- The next level starts from the unrefined partition: every member + -- of a refined community is in the same community, so min is it. + PERFORM warren.project(graph, '_warren_leiden_down', next_g, '_warren_leiden_initial', 'min', + persist := false, version := next_v); + IF g <> graph THEN + PERFORM warren.drop_vector(g, '_warren_leiden_' || part, gv, true) FROM unnest(parts) part; + PERFORM warren.drop_graph(g); + END IF; + g := next_g; + gv := next_v; + END LOOP; + IF g <> graph THEN + PERFORM warren.drop_vector(g, '_warren_leiden_' || part, gv, true) FROM unnest(parts) part; + PERFORM warren.drop_graph(g); + g := graph; + gv := v; + END IF; + EXIT WHEN iteration = iterations; + -- Labels are the least dense id of each community's members, so the + -- partitions are the same exactly when the labels are. + IF iteration > 1 THEN + PERFORM warren.vector_map(graph, '_warren_leiden_changed', + 'CASE WHEN _warren_leiden_down = _warren_leiden_prev THEN 0 ELSE 1 END', false, v); + EXIT WHEN warren.vector_reduce(graph, '_warren_leiden_changed', 'sum', v) = 0; + END IF; + IF iterations < 0 AND iteration = 100 THEN + RAISE WARNING 'leiden did not stabilise in 100 iterations'; + EXIT; END IF; - PERFORM warren.vector_const(graph, '_warren_hits_scale', CASE WHEN scale = 0 THEN 1 ELSE scale END, - persist := false, version := v); - PERFORM warren.vector_map(graph, '_warren_hits_next', - 'CASE WHEN _warren_hits_scale <> 0 THEN _warren_hits_next / _warren_hits_scale ELSE 0 END', false, v); - delta := warren.vector_norm_diff(graph, '_warren_hits_next', '_warren_hits_x', 'l1', v, v); - PERFORM warren.vector_map(graph, '_warren_hits_x', '_warren_hits_next', false, v); - EXIT WHEN delta < tol; END LOOP; - IF NOT delta < tol THEN - RAISE WARNING 'hits did not converge in % iterations', i; - END IF; - IF normalized THEN - hub_scale := warren.vector_reduce(graph, '_warren_hits_x', 'sum', v); - scale := warren.vector_reduce(graph, '_warren_hits_a', 'sum', v); - ELSE - hub_scale := 1; - scale := warren.vector_reduce(graph, '_warren_hits_a', 'max', v); - END IF; - IF NOT abs(hub_scale) < 'Infinity' OR NOT abs(scale) < 'Infinity' THEN - RAISE EXCEPTION 'hits diverged: its values are not finite after % iterations', i - USING ERRCODE = 'numeric_value_out_of_range', - HINT = 'The edge weights are too large: their products overflow double precision.'; - END IF; - PERFORM warren.vector_const(graph, '_warren_hits_scale', CASE WHEN hub_scale = 0 THEN 1 ELSE hub_scale END, - persist := false, version := v); - PERFORM warren.vector_map(graph, hubs, - 'CASE WHEN _warren_hits_scale <> 0 THEN _warren_hits_x / _warren_hits_scale ELSE 0 END', true, v); - PERFORM warren.vector_const(graph, '_warren_hits_scale', CASE WHEN scale = 0 THEN 1 ELSE scale END, - persist := false, version := v); - PERFORM warren.vector_map(graph, authorities, - 'CASE WHEN _warren_hits_scale <> 0 THEN _warren_hits_a / _warren_hits_scale ELSE 0 END', true, v); + PERFORM warren.vector_map(graph, "out", '_warren_leiden_down', true, v); + q := warren.modularity(graph, "out", resolution, weighted, v); EXCEPTION WHEN OTHERS OR query_canceled THEN - PERFORM warren.drop_vector(graph, '_warren_hits_' || part, v, true) FROM unnest(parts) part; + PERFORM warren.drop_vector(graph, '_warren_leiden_' || part, v, true) FROM unnest(parts) part; RAISE; END; - PERFORM warren.drop_vector(graph, '_warren_hits_' || part, v, true) FROM unnest(parts) part; - RETURN i; + PERFORM warren.drop_vector(graph, '_warren_leiden_' || part, v, true) FROM unnest(parts) part; + IF g <> graph THEN + PERFORM warren.drop_vector(g, '_warren_leiden_' || part, gv, true) FROM unnest(parts) part; + PERFORM warren.drop_graph(g); + END IF; + RETURN q; END $$; /* */ /* */ --- crates/pg_warren/src/cache/shared/introspection.rs:136 --- pg_warren::cache::shared::introspection::shared_cache -CREATE FUNCTION warren."shared_cache"() RETURNS TABLE ( - "kind" TEXT, /* String */ - "graph" TEXT, /* Option < String > */ - "version" bigint, /* Option < i64 > */ - "vector" TEXT, /* Option < String > */ - "vector_id" bigint, /* Option < i64 > */ - "state" TEXT, /* Option < String > */ - "bytes" bigint, /* i64 */ - "pins" INT, /* i32 */ - "mappers" INT, /* i32 */ - "last_use" bigint, /* i64 */ - "cap" bigint /* Option < i64 > */ -) -STRICT VOLATILE PARALLEL SAFE -LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'shared_cache_wrapper'; -/* */ - -/* */ --- crates/pg_warren/src/catalog/dropped_roles/orphans.rs:34 --- pg_warren::catalog::dropped_roles::orphans::orphans -CREATE FUNCTION warren."orphans"() RETURNS TABLE ( - "kind" TEXT, /* String */ - "graph" TEXT, /* String */ - "role" oid /* pg_sys :: Oid */ -) -STRICT VOLATILE PARALLEL SAFE -LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'orphans_wrapper'; -/* */ - -/* */ --- crates/pg_warren/src/compute_sql/structure.rs:139 --- pg_warren::compute_sql::structure::dag_longest_path -CREATE FUNCTION warren."dag_longest_path"( +-- crates/pg_warren/src/compute_sql/product.rs:172 +-- pg_warren::compute_sql::product::spmv +CREATE FUNCTION warren."spmv"( "graph" TEXT, /* Option < & str > */ - "out" TEXT, /* Option < & str > */ + "x" TEXT, /* Option < & str > */ + "y" TEXT, /* Option < & str > */ + "semiring" TEXT DEFAULT 'plus_times', /* Option < & str > */ + "transpose" bool DEFAULT false, /* Option < bool > */ "rels" TEXT[] DEFAULT NULL, /* :: std :: option :: Option < Vec < Option < String > > > */ "weighted" bool DEFAULT NULL, /* Option < bool > */ + "mask" TEXT DEFAULT NULL, /* Option < & str > */ + "mask_complement" bool DEFAULT false, /* Option < bool > */ + "replace" bool DEFAULT false, /* Option < bool > */ + "accumulate" TEXT DEFAULT NULL, /* Option < & str > */ "persist" bool DEFAULT true, /* Option < bool > */ "version" bigint DEFAULT NULL /* Option < i64 > */ -) RETURNS double precision /* Option < f64 > */ +) RETURNS void VOLATILE PARALLEL UNSAFE LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'dag_longest_path_wrapper'; -/* */ - -/* */ --- crates/pg_warren/src/catalog/readers.rs:12 --- pg_warren::catalog::readers::grant - -CREATE FUNCTION warren."grant"(name text, role regrole) RETURNS void -STRICT VOLATILE LANGUAGE c AS 'MODULE_PATHNAME', 'grant_wrapper'; -/* */ - -/* */ --- crates/pg_warren/src/traversal_sql/components.rs:21 --- pg_warren::traversal_sql::components::components - -CREATE FUNCTION warren.components(graph text, id_type anyelement, version bigint DEFAULT NULL) -RETURNS TABLE (node_id anyelement, component_id bigint) -STABLE PARALLEL SAFE LANGUAGE c AS 'MODULE_PATHNAME', 'components_wrapper'; +AS 'MODULE_PATHNAME', 'spmv_wrapper'; /* */ /* */ --- crates/pg_warren/src/catalog/create_graph.rs:42 --- pg_warren::catalog::create_graph::create_graph -CREATE FUNCTION warren."create_graph"( +-- crates/pg_warren/src/catalog/drop.rs:13 +-- pg_warren::catalog::drop::drop_version +CREATE FUNCTION warren."drop_version"( "name" TEXT, /* & str */ - "node_query" TEXT, /* & str */ - "edge_query" TEXT, /* & str */ - "directed" bool DEFAULT true, /* bool */ - "weighted" bool DEFAULT false, /* bool */ - "has_relations" bool DEFAULT false, /* bool */ - "build_reverse" bool DEFAULT true, /* bool */ - "retain_versions" INT DEFAULT 2, /* i32 */ - "store_merged" bool DEFAULT true, /* bool */ - "node_order" TEXT DEFAULT 'id' /* & str */ + "version" bigint /* i64 */ ) RETURNS void STRICT LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'create_graph_wrapper'; -/* */ - -/* */ --- crates/pg_warren/src/vector/quota/sql.rs:85 --- pg_warren::vector::quota::sql::vector_usage - -CREATE FUNCTION warren."vector_usage"(name text) -RETURNS TABLE (role regrole, vectors bigint, bytes bigint, quota bigint) -STRICT STABLE PARALLEL SAFE LANGUAGE c AS 'MODULE_PATHNAME', 'vector_usage_wrapper'; +AS 'MODULE_PATHNAME', 'drop_version_wrapper'; /* */ /* */ --- crates/pg_warren/src/traversal_sql/paths.rs:114 --- pg_warren::traversal_sql::paths::dijkstra - -CREATE FUNCTION warren.dijkstra(graph text, start anyelement, target anyelement, - rels text[] DEFAULT NULL, direction text DEFAULT 'out', - version bigint DEFAULT NULL) -RETURNS TABLE (path anyarray, cost float8) -STABLE PARALLEL SAFE LANGUAGE c AS 'MODULE_PATHNAME', 'dijkstra_wrapper'; +-- crates/pg_warren/src/catalog/drop.rs:29 +-- pg_warren::catalog::drop::drop_graph +CREATE FUNCTION warren."drop_graph"( + "name" TEXT /* & str */ +) RETURNS void +STRICT +LANGUAGE c /* Rust */ +AS 'MODULE_PATHNAME', 'drop_graph_wrapper'; /* */ /* */ --- crates/pg_warren/src/compute_sql/constructors.rs:132 --- pg_warren::compute_sql::constructors::vector_dense_id -CREATE FUNCTION warren."vector_dense_id"( +-- crates/pg_warren/src/compute_sql/constructors.rs:38 +-- pg_warren::compute_sql::constructors::vector_const +CREATE FUNCTION warren."vector_const"( "graph" TEXT, /* Option < & str > */ "name" TEXT, /* Option < & str > */ - "dtype" TEXT DEFAULT 'i64', /* Option < & str > */ + "value" double precision, /* Option < f64 > */ + "dtype" TEXT DEFAULT 'f64', /* Option < & str > */ "persist" bool DEFAULT true, /* Option < bool > */ "version" bigint DEFAULT NULL /* Option < i64 > */ ) RETURNS bigint /* Option < i64 > */ VOLATILE PARALLEL UNSAFE LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'vector_dense_id_wrapper'; +AS 'MODULE_PATHNAME', 'vector_const_wrapper'; /* */ /* */ --- crates/pg_warren/src/traversal_sql/paths.rs:25 --- pg_warren::traversal_sql::paths::reachable +-- crates/pg_warren/src/traversal_sql/walk.rs:38 +-- pg_warren::traversal_sql::walk::walk -CREATE FUNCTION warren.reachable(graph text, start anyelement, target anyelement, - max_depth int DEFAULT NULL, rels text[] DEFAULT NULL, - direction text DEFAULT 'out', version bigint DEFAULT NULL) -RETURNS boolean -STABLE PARALLEL SAFE LANGUAGE c AS 'MODULE_PATHNAME', 'reachable_wrapper'; +CREATE FUNCTION warren.walk(graph text, start anyarray, steps warren.step[], + mode text DEFAULT 'ends', distinct_ends boolean DEFAULT true, + max_results bigint DEFAULT NULL, reverse boolean DEFAULT false, + version bigint DEFAULT NULL) +RETURNS TABLE (end_id anyelement, path anyarray, rels text[], count bigint) +STABLE PARALLEL SAFE LANGUAGE c AS 'MODULE_PATHNAME', 'walk_wrapper'; /* */ /* */ --- crates/pg_warren/src/compute_sql/summary.rs:87 --- pg_warren::compute_sql::summary::vector_stats -CREATE FUNCTION warren."vector_stats"( - "graph" TEXT, /* Option < & str > */ - "name" TEXT, /* Option < & str > */ - "version" bigint DEFAULT NULL /* Option < i64 > */ +-- crates/pg_warren/src/version_sql/diff.rs:23 +-- pg_warren::version_sql::diff::diff +CREATE FUNCTION warren."diff"( + "name" TEXT, /* & str */ + "v1" bigint, /* i64 */ + "v2" bigint /* i64 */ ) RETURNS jsonb /* JsonB */ -STABLE PARALLEL RESTRICTED +STRICT STABLE PARALLEL RESTRICTED LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'vector_stats_wrapper'; +AS 'MODULE_PATHNAME', 'diff_wrapper'; /* */ /* */ --- crates/pg_warren/src/extension.rs:37 --- pg_warren::extension::version -CREATE FUNCTION warren."version"() RETURNS TEXT /* & '_ str */ -STRICT STABLE PARALLEL SAFE +-- crates/pg_warren/src/traversal_sql/step.rs:94 +-- pg_warren::traversal_sql::step::s + +CREATE FUNCTION warren.s(rels text[], direction text DEFAULT 'out', filter "any" DEFAULT NULL, + min_hops int DEFAULT 1, max_hops int DEFAULT NULL) +RETURNS warren.step +STABLE PARALLEL SAFE LANGUAGE c AS 'MODULE_PATHNAME', 's_wrapper'; +/* */ + +/* */ +-- crates/pg_warren/src/version_sql/compact.rs:25 +-- pg_warren::version_sql::compact::compact +CREATE FUNCTION warren."compact"( + "name" TEXT, /* Option < & str > */ + "node_order" TEXT DEFAULT NULL /* Option < & str > */ +) RETURNS bigint /* i64 */ + LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'version_wrapper'; +AS 'MODULE_PATHNAME', 'compact_wrapper'; /* */ /* */ --- crates/pg_warren/src/compute_sql/summary.rs:20 --- pg_warren::compute_sql::summary::vector_topk +-- crates/pg_warren/src/traversal_sql/components.rs:21 +-- pg_warren::traversal_sql::components::components -CREATE FUNCTION warren.vector_topk(graph text, name text, k int, id_type anyelement, - descending bool DEFAULT true, version bigint DEFAULT NULL) -RETURNS TABLE (node_id anyelement, value float8) -STABLE PARALLEL RESTRICTED LANGUAGE c AS 'MODULE_PATHNAME', 'vector_topk_wrapper'; +CREATE FUNCTION warren.components(graph text, id_type anyelement, version bigint DEFAULT NULL) +RETURNS TABLE (node_id anyelement, component_id bigint) +STABLE PARALLEL SAFE LANGUAGE c AS 'MODULE_PATHNAME', 'components_wrapper'; +/* */ + +/* */ +-- crates/pg_warren/src/compute_sql/from_query.rs:58 +-- pg_warren::compute_sql::from_query::vector_from_query +CREATE FUNCTION warren."vector_from_query"( + "graph" TEXT, /* Option < & str > */ + "name" TEXT, /* Option < & str > */ + "query" TEXT, /* Option < & str > */ + "dtype" TEXT DEFAULT 'f64', /* Option < & str > */ + "fill" double precision DEFAULT 0, /* Option < f64 > */ + "persist" bool DEFAULT true, /* Option < bool > */ + "version" bigint DEFAULT NULL /* Option < i64 > */ +) RETURNS bigint /* Option < i64 > */ +VOLATILE PARALLEL UNSAFE +LANGUAGE c /* Rust */ +AS 'MODULE_PATHNAME', 'vector_from_query_wrapper'; /* */ /* */ @@ -1326,33 +1296,31 @@ AS 'MODULE_PATHNAME', 'explain_map_wrapper'; /* */ /* */ --- crates/pg_warren/src/build/refresh.rs:20 --- pg_warren::build::refresh::refresh -CREATE FUNCTION warren."refresh"( - "name" TEXT, /* & str */ - "promote" bool DEFAULT true, /* bool */ - "label" TEXT DEFAULT NULL /* Option < & str > */ -) RETURNS bigint /* i64 */ +-- crates/pg_warren/src/traversal_sql/paths.rs:116 +-- pg_warren::traversal_sql::paths::dijkstra -LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'refresh_wrapper'; +CREATE FUNCTION warren.dijkstra(graph text, start anyelement, target anyelement, + rels text[] DEFAULT NULL, direction text DEFAULT 'out', + version bigint DEFAULT NULL) +RETURNS TABLE (path anyarray, cost float8) +STABLE PARALLEL SAFE LANGUAGE c AS 'MODULE_PATHNAME', 'dijkstra_wrapper'; /* */ /* */ --- crates/pg_warren/src/compute_sql/from_query.rs:57 --- pg_warren::compute_sql::from_query::vector_from_query -CREATE FUNCTION warren."vector_from_query"( - "graph" TEXT, /* Option < & str > */ - "name" TEXT, /* Option < & str > */ - "query" TEXT, /* Option < & str > */ - "dtype" TEXT DEFAULT 'f64', /* Option < & str > */ - "fill" double precision DEFAULT 0, /* Option < f64 > */ - "persist" bool DEFAULT true, /* Option < bool > */ - "version" bigint DEFAULT NULL /* Option < i64 > */ -) RETURNS bigint /* Option < i64 > */ -VOLATILE PARALLEL UNSAFE +-- crates/pg_warren/src/catalog/readers.rs:12 +-- pg_warren::catalog::readers::grant + +CREATE FUNCTION warren."grant"(name text, role regrole) RETURNS void +STRICT VOLATILE LANGUAGE c AS 'MODULE_PATHNAME', 'grant_wrapper'; +/* */ + +/* */ +-- crates/pg_warren/src/catalog/dropped_roles/orphans.rs:70 +-- pg_warren::catalog::dropped_roles::orphans::drop_orphans +CREATE FUNCTION warren."drop_orphans"() RETURNS bigint /* i64 */ +STRICT LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'vector_from_query_wrapper'; +AS 'MODULE_PATHNAME', 'drop_orphans_wrapper'; /* */ /* */ @@ -1370,97 +1338,107 @@ AS 'MODULE_PATHNAME', 'vector_reduce_wrapper'; /* */ /* */ --- crates/pg_warren/src/compute_sql/read.rs:132 --- pg_warren::compute_sql::read::id_map - -CREATE FUNCTION warren.id_map(graph text, id_type anyelement, version bigint DEFAULT NULL) -RETURNS TABLE (node_id anyelement, dense_id bigint, live boolean) -STABLE PARALLEL SAFE LANGUAGE c AS 'MODULE_PATHNAME', 'id_map_wrapper'; +-- crates/pg_warren/src/spectral_sql/eigen.rs:92 +-- pg_warren::spectral_sql::eigen::eigen +CREATE FUNCTION warren."eigen"( + "graph" TEXT, /* Option < & str > */ + "k" INT, /* Option < i32 > */ + "matrix" TEXT DEFAULT 'adjacency', /* Option < & str > */ + "which" TEXT DEFAULT 'largest', /* Option < & str > */ + "rels" TEXT[] DEFAULT NULL, /* :: std :: option :: Option < Vec < Option < String > > > */ + "weighted" bool DEFAULT NULL, /* Option < bool > */ + "symmetrize" bool DEFAULT true, /* Option < bool > */ + "tol" double precision DEFAULT 1e-6, /* Option < f64 > */ + "max_iter" INT DEFAULT NULL, /* Option < i32 > */ + "vectors_prefix" TEXT DEFAULT NULL, /* Option < & str > */ + "version" bigint DEFAULT NULL, /* Option < i64 > */ + "dtype" TEXT DEFAULT 'f32', /* Option < & str > */ + "persist" bool DEFAULT true /* Option < bool > */ +) RETURNS TABLE ( + "i" INT, /* i32 */ + "eigenvalue" double precision, /* f64 */ + "residual" double precision /* f64 */ +) +VOLATILE PARALLEL UNSAFE +LANGUAGE c /* Rust */ +AS 'MODULE_PATHNAME', 'eigen_wrapper'; /* */ /* */ --- crates/pg_warren/src/traversal_sql/neighbourhood.rs:52 --- pg_warren::traversal_sql::neighbourhood::degree - -CREATE FUNCTION warren.degree(graph text, node anyelement, direction text DEFAULT 'out', - rels text[] DEFAULT NULL, version bigint DEFAULT NULL) -RETURNS bigint -STABLE PARALLEL SAFE LANGUAGE c AS 'MODULE_PATHNAME', 'degree_wrapper'; +-- crates/pg_warren/src/cache/shared/introspection.rs:136 +-- pg_warren::cache::shared::introspection::shared_cache +CREATE FUNCTION warren."shared_cache"() RETURNS TABLE ( + "kind" TEXT, /* String */ + "graph" TEXT, /* Option < String > */ + "version" bigint, /* Option < i64 > */ + "vector" TEXT, /* Option < String > */ + "vector_id" bigint, /* Option < i64 > */ + "state" TEXT, /* Option < String > */ + "bytes" bigint, /* i64 */ + "pins" INT, /* i32 */ + "mappers" INT, /* i32 */ + "last_use" bigint, /* i64 */ + "cap" bigint /* Option < i64 > */ +) +STRICT VOLATILE PARALLEL SAFE +LANGUAGE c /* Rust */ +AS 'MODULE_PATHNAME', 'shared_cache_wrapper'; /* */ /* */ --- crates/pg_warren/src/compute_sql/constructors.rs:163 --- pg_warren::compute_sql::constructors::triangles -CREATE FUNCTION warren."triangles"( +-- crates/pg_warren/src/compute_sql/constructors.rs:93 +-- pg_warren::compute_sql::constructors::vector_degree +CREATE FUNCTION warren."vector_degree"( "graph" TEXT, /* Option < & str > */ - "out" TEXT, /* Option < & str > */ + "name" TEXT, /* Option < & str > */ + "direction" TEXT DEFAULT 'out', /* Option < & str > */ "rels" TEXT[] DEFAULT NULL, /* :: std :: option :: Option < Vec < Option < String > > > */ + "dtype" TEXT DEFAULT 'f64', /* Option < & str > */ "persist" bool DEFAULT true, /* Option < bool > */ "version" bigint DEFAULT NULL /* Option < i64 > */ ) RETURNS bigint /* Option < i64 > */ VOLATILE PARALLEL UNSAFE LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'triangles_wrapper'; +AS 'MODULE_PATHNAME', 'vector_degree_wrapper'; /* */ /* */ --- crates/pg_warren/src/spectral_sql/distance/mod.rs:96 --- pg_warren::spectral_sql::distance::distance -CREATE FUNCTION warren."distance"( - "graph" TEXT, /* Option < & str > */ - "v1" bigint, /* Option < i64 > */ - "v2" bigint, /* Option < i64 > */ - "method" TEXT, /* Option < & str > */ - "vectors_prefix" TEXT DEFAULT NULL, /* Option < & str > */ - "k" INT DEFAULT NULL, /* Option < i32 > */ - "x" TEXT DEFAULT NULL, /* Option < & str > */ - "y" TEXT DEFAULT NULL, /* Option < & str > */ - "compared_rows" TEXT DEFAULT 'shared', /* Option < & str > */ - "matrix" TEXT DEFAULT NULL, /* Option < & str > */ - "which" TEXT DEFAULT 'largest', /* Option < & str > */ - "rels" TEXT[] DEFAULT NULL, /* :: std :: option :: Option < Vec < Option < String > > > */ - "weighted" bool DEFAULT NULL, /* Option < bool > */ - "symmetrize" bool DEFAULT true, /* Option < bool > */ - "tol" double precision DEFAULT 1e-6, /* Option < f64 > */ - "max_iter" INT DEFAULT NULL, /* Option < i32 > */ - "dtype" TEXT DEFAULT 'f32', /* Option < & str > */ - "graph2" TEXT DEFAULT NULL /* Option < & str > */ -) RETURNS jsonb /* JsonB */ -STABLE PARALLEL RESTRICTED -LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'distance_wrapper'; +-- crates/pg_warren/src/compute_sql/read.rs:134 +-- pg_warren::compute_sql::read::id_map + +CREATE FUNCTION warren.id_map(graph text, id_type anyelement, version bigint DEFAULT NULL) +RETURNS TABLE (node_id anyelement, dense_id bigint, live boolean) +STABLE PARALLEL SAFE LANGUAGE c AS 'MODULE_PATHNAME', 'id_map_wrapper'; /* */ /* */ --- crates/pg_warren/src/community_sql/projection.rs:60 --- pg_warren::community_sql::projection::lift -CREATE FUNCTION warren."lift"( - "out_graph" TEXT, /* Option < & str > */ - "y" TEXT, /* Option < & str > */ - "graph" TEXT, /* Option < & str > */ - "x" TEXT, /* Option < & str > */ - "persist" bool DEFAULT true, /* Option < bool > */ - "version" bigint DEFAULT NULL /* Option < i64 > */ -) RETURNS void -VOLATILE PARALLEL UNSAFE -LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'lift_wrapper'; +-- crates/pg_warren/src/compute_sql/structure.rs:178 +-- pg_warren::compute_sql::structure::minimum_spanning_forest + +CREATE FUNCTION warren.minimum_spanning_forest(graph text, id_type anyelement, rels text[] DEFAULT NULL, + weighted bool DEFAULT NULL, version bigint DEFAULT NULL) +RETURNS TABLE (source anyelement, target anyelement, weight float8) +STABLE PARALLEL RESTRICTED LANGUAGE c AS 'MODULE_PATHNAME', 'minimum_spanning_forest_wrapper'; /* */ /* */ --- crates/pg_warren/src/compute_sql/reduce.rs:47 --- pg_warren::compute_sql::reduce::vector_norm_diff -CREATE FUNCTION warren."vector_norm_diff"( +-- crates/pg_warren/src/community_sql/local_moving.rs:97 +-- pg_warren::community_sql::local_moving::leiden_refine +CREATE FUNCTION warren."leiden_refine"( "graph" TEXT, /* Option < & str > */ - "a" TEXT, /* Option < & str > */ - "b" TEXT, /* Option < & str > */ - "norm" TEXT DEFAULT 'l2', /* Option < & str > */ - "a_version" bigint DEFAULT NULL, /* Option < i64 > */ - "b_version" bigint DEFAULT NULL /* Option < i64 > */ -) RETURNS double precision /* f64 */ -STABLE PARALLEL RESTRICTED + "communities" TEXT, /* Option < & str > */ + "out" TEXT, /* Option < & str > */ + "keys" TEXT DEFAULT NULL, /* Option < & str > */ + "resolution" double precision DEFAULT 1.0, /* Option < f64 > */ + "seed" bigint DEFAULT 0, /* Option < i64 > */ + "randomness" double precision DEFAULT 0.01, /* Option < f64 > */ + "weighted" bool DEFAULT NULL, /* Option < bool > */ + "persist" bool DEFAULT true, /* Option < bool > */ + "version" bigint DEFAULT NULL /* Option < i64 > */ +) RETURNS bigint /* Option < i64 > */ +VOLATILE PARALLEL UNSAFE LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'vector_norm_diff_wrapper'; +AS 'MODULE_PATHNAME', 'leiden_refine_wrapper'; /* */ /* */ @@ -1477,57 +1455,55 @@ VOLATILE PARALLEL UNSAFE LANGUAGE c AS 'MODULE_PATHNAME', 'gather_wrapper'; /* */ /* */ --- crates/pg_warren/src/community_sql/projection.rs:26 --- pg_warren::community_sql::projection::project -CREATE FUNCTION warren."project"( +-- crates/pg_warren/src/catalog/dropped_roles/drop_owned.rs:16 +-- pg_warren::catalog::dropped_roles::drop_owned::drop_owned + +CREATE FUNCTION warren."drop_owned"(role regrole) RETURNS bigint +STRICT VOLATILE LANGUAGE c AS 'MODULE_PATHNAME', 'drop_owned_wrapper'; +/* */ + +/* */ +-- crates/pg_warren/src/compute_sql/summary.rs:87 +-- pg_warren::compute_sql::summary::vector_stats +CREATE FUNCTION warren."vector_stats"( "graph" TEXT, /* Option < & str > */ - "x" TEXT, /* Option < & str > */ - "out_graph" TEXT, /* Option < & str > */ - "y" TEXT, /* Option < & str > */ - "agg" TEXT DEFAULT 'sum', /* Option < & str > */ - "persist" bool DEFAULT true, /* Option < bool > */ + "name" TEXT, /* Option < & str > */ "version" bigint DEFAULT NULL /* Option < i64 > */ -) RETURNS void -VOLATILE PARALLEL UNSAFE +) RETURNS jsonb /* JsonB */ +STABLE PARALLEL RESTRICTED LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'project_wrapper'; +AS 'MODULE_PATHNAME', 'vector_stats_wrapper'; /* */ /* */ --- crates/pg_warren/src/compute_sql/structure.rs:178 --- pg_warren::compute_sql::structure::minimum_spanning_forest +-- crates/pg_warren/src/catalog/owner/mod.rs:27 +-- pg_warren::catalog::owner::set_owner -CREATE FUNCTION warren.minimum_spanning_forest(graph text, id_type anyelement, rels text[] DEFAULT NULL, - weighted bool DEFAULT NULL, version bigint DEFAULT NULL) -RETURNS TABLE (source anyelement, target anyelement, weight float8) -STABLE PARALLEL RESTRICTED LANGUAGE c AS 'MODULE_PATHNAME', 'minimum_spanning_forest_wrapper'; +CREATE FUNCTION warren."set_owner"(name text, owner regrole) RETURNS void +STRICT VOLATILE LANGUAGE c AS 'MODULE_PATHNAME', 'set_owner_wrapper'; /* */ /* */ --- crates/pg_warren/src/traversal_sql/bfs.rs:87 --- pg_warren::traversal_sql::bfs::reachable_set +-- crates/pg_warren/src/vector/quota/sql.rs:97 +-- pg_warren::vector::quota::sql::vector_usage -CREATE FUNCTION warren.reachable_set(graph text, start anyarray, max_depth int DEFAULT NULL, - rels text[] DEFAULT NULL, direction text DEFAULT 'out', - version bigint DEFAULT NULL) -RETURNS SETOF anyelement -STABLE PARALLEL SAFE LANGUAGE c AS 'MODULE_PATHNAME', 'reachable_set_wrapper'; +CREATE FUNCTION warren."vector_usage"(name text) +RETURNS TABLE (role regrole, vectors bigint, bytes bigint, quota bigint) +STRICT STABLE PARALLEL SAFE LANGUAGE c AS 'MODULE_PATHNAME', 'vector_usage_wrapper'; /* */ /* */ --- crates/pg_warren/src/version_sql/compact.rs:25 --- pg_warren::version_sql::compact::compact -CREATE FUNCTION warren."compact"( - "name" TEXT, /* Option < & str > */ - "node_order" TEXT DEFAULT NULL /* Option < & str > */ -) RETURNS bigint /* i64 */ +-- crates/pg_warren/src/compute_sql/read.rs:100 +-- pg_warren::compute_sql::read::matrix_rows -LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'compact_wrapper'; +CREATE FUNCTION warren.matrix_rows(graph text, name text, id_type anyelement, + version bigint DEFAULT NULL) +RETURNS TABLE (node_id anyelement, "row" float8[]) +STABLE PARALLEL RESTRICTED LANGUAGE c AS 'MODULE_PATHNAME', 'matrix_rows_wrapper'; /* */ /* */ --- crates/pg_warren/src/compute_sql/similarity.rs:241 +-- crates/pg_warren/src/compute_sql/similarity.rs:247 -- pg_warren::compute_sql::similarity::similar_pairs CREATE FUNCTION warren.similar_pairs(graph text, id_type anyelement, k bigint DEFAULT 10, @@ -1537,30 +1513,6 @@ RETURNS TABLE (source anyelement, target anyelement, score float8) STABLE PARALLEL RESTRICTED LANGUAGE c AS 'MODULE_PATHNAME', 'similar_pairs_wrapper'; /* */ -/* */ --- crates/pg_warren/src/traversal_sql/walk.rs:38 --- pg_warren::traversal_sql::walk::walk - -CREATE FUNCTION warren.walk(graph text, start anyarray, steps warren.step[], - mode text DEFAULT 'ends', distinct_ends boolean DEFAULT true, - max_results bigint DEFAULT NULL, reverse boolean DEFAULT false, - version bigint DEFAULT NULL) -RETURNS TABLE (end_id anyelement, path anyarray, rels text[], count bigint) -STABLE PARALLEL SAFE LANGUAGE c AS 'MODULE_PATHNAME', 'walk_wrapper'; -/* */ - -/* */ --- crates/pg_warren/src/catalog/promote.rs:14 --- pg_warren::catalog::promote::promote -CREATE FUNCTION warren."promote"( - "name" TEXT, /* & str */ - "version" bigint /* i64 */ -) RETURNS void -STRICT -LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'promote_wrapper'; -/* */ - /* */ -- crates/pg_warren/src/compute_sql/centrality.rs:125 -- pg_warren::compute_sql::centrality::closeness_centrality @@ -1583,88 +1535,61 @@ AS 'MODULE_PATHNAME', 'closeness_centrality_wrapper'; /* */ /* */ --- crates/pg_warren/src/compute_sql/structure.rs:63 --- pg_warren::compute_sql::structure::strong_components_vector -CREATE FUNCTION warren."strong_components_vector"( - "graph" TEXT, /* Option < & str > */ - "out" TEXT, /* Option < & str > */ - "rels" TEXT[] DEFAULT NULL, /* :: std :: option :: Option < Vec < Option < String > > > */ - "persist" bool DEFAULT true, /* Option < bool > */ - "version" bigint DEFAULT NULL /* Option < i64 > */ -) RETURNS bigint /* Option < i64 > */ -VOLATILE PARALLEL UNSAFE -LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'strong_components_vector_wrapper'; -/* */ +-- crates/pg_warren/src/traversal_sql/neighbourhood.rs:25 +-- pg_warren::traversal_sql::neighbourhood::neighbours -/* */ --- crates/pg_warren/src/community_sql/contract.rs:39 --- pg_warren::community_sql::contract::contract -CREATE FUNCTION warren."contract"( - "graph" TEXT, /* Option < & str > */ - "label_vector" TEXT, /* Option < & str > */ - "out_graph" TEXT, /* Option < & str > */ - "weighted" bool DEFAULT true, /* Option < bool > */ - "drop_self_loops" bool DEFAULT false, /* Option < bool > */ - "promote" bool DEFAULT true, /* Option < bool > */ - "version" bigint DEFAULT NULL /* Option < i64 > */ -) RETURNS bigint /* Option < i64 > */ -VOLATILE PARALLEL UNSAFE -LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'contract_wrapper'; +CREATE FUNCTION warren.neighbours(graph text, node anyelement, direction text DEFAULT 'out', + rels text[] DEFAULT NULL, version bigint DEFAULT NULL) +RETURNS SETOF anyelement +STABLE PARALLEL SAFE LANGUAGE c AS 'MODULE_PATHNAME', 'neighbours_wrapper'; /* */ /* */ --- crates/pg_warren/src/catalog/owner/reassign.rs:19 --- pg_warren::catalog::owner::reassign::reassign_owned +-- crates/pg_warren/src/compute_sql/similarity.rs:51 +-- pg_warren::compute_sql::similarity::similarity -CREATE FUNCTION warren."reassign_owned"("old" regrole, "new" regrole) RETURNS bigint -STRICT VOLATILE LANGUAGE c AS 'MODULE_PATHNAME', 'reassign_owned_wrapper'; +CREATE FUNCTION warren.similarity(graph text, pairs text, id_type anyelement, + measure text DEFAULT 'jaccard', rels text[] DEFAULT NULL, + version bigint DEFAULT NULL) +RETURNS TABLE (source anyelement, target anyelement, score float8) +STABLE PARALLEL RESTRICTED LANGUAGE c AS 'MODULE_PATHNAME', 'similarity_wrapper'; /* */ /* */ --- crates/pg_warren/src/compute_sql/constructors.rs:93 --- pg_warren::compute_sql::constructors::vector_degree -CREATE FUNCTION warren."vector_degree"( +-- crates/pg_warren/src/compute_sql/structure.rs:139 +-- pg_warren::compute_sql::structure::dag_longest_path +CREATE FUNCTION warren."dag_longest_path"( "graph" TEXT, /* Option < & str > */ - "name" TEXT, /* Option < & str > */ - "direction" TEXT DEFAULT 'out', /* Option < & str > */ + "out" TEXT, /* Option < & str > */ "rels" TEXT[] DEFAULT NULL, /* :: std :: option :: Option < Vec < Option < String > > > */ - "dtype" TEXT DEFAULT 'f64', /* Option < & str > */ + "weighted" bool DEFAULT NULL, /* Option < bool > */ "persist" bool DEFAULT true, /* Option < bool > */ "version" bigint DEFAULT NULL /* Option < i64 > */ -) RETURNS bigint /* Option < i64 > */ +) RETURNS double precision /* Option < f64 > */ VOLATILE PARALLEL UNSAFE LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'vector_degree_wrapper'; +AS 'MODULE_PATHNAME', 'dag_longest_path_wrapper'; /* */ /* */ --- crates/pg_warren/src/traversal_sql/stats.rs:30 --- pg_warren::traversal_sql::stats::last_stats -CREATE FUNCTION warren."last_stats"() RETURNS jsonb /* Option < JsonB > */ -STRICT VOLATILE PARALLEL RESTRICTED +-- crates/pg_warren/src/community_sql/local_moving.rs:30 +-- pg_warren::community_sql::local_moving::local_moving +CREATE FUNCTION warren."local_moving"( + "graph" TEXT, /* Option < & str > */ + "out" TEXT, /* Option < & str > */ + "initial" TEXT DEFAULT NULL, /* Option < & str > */ + "keys" TEXT DEFAULT NULL, /* Option < & str > */ + "resolution" double precision DEFAULT 1.0, /* Option < f64 > */ + "seed" bigint DEFAULT 0, /* Option < i64 > */ + "max_sweeps" INT DEFAULT 100, /* Option < i32 > */ + "tolerance" double precision DEFAULT 1e-7, /* Option < f64 > */ + "weighted" bool DEFAULT NULL, /* Option < bool > */ + "persist" bool DEFAULT true, /* Option < bool > */ + "version" bigint DEFAULT NULL /* Option < i64 > */ +) RETURNS bigint /* Option < i64 > */ +VOLATILE PARALLEL UNSAFE LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'last_stats_wrapper'; -/* */ - -/* */ --- crates/pg_warren/src/catalog/dropped_roles/event_trigger.rs:21 --- pg_warren::catalog::dropped_roles::event_trigger::on_drop_owned - -CREATE FUNCTION warren."on_drop_owned"() RETURNS event_trigger -VOLATILE LANGUAGE c AS 'MODULE_PATHNAME', 'on_drop_owned_wrapper'; -/* */ - -/* */ --- crates/pg_warren/src/compute_sql/similarity.rs:48 --- pg_warren::compute_sql::similarity::similarity - -CREATE FUNCTION warren.similarity(graph text, pairs text, id_type anyelement, - measure text DEFAULT 'jaccard', rels text[] DEFAULT NULL, - version bigint DEFAULT NULL) -RETURNS TABLE (source anyelement, target anyelement, score float8) -STABLE PARALLEL RESTRICTED LANGUAGE c AS 'MODULE_PATHNAME', 'similarity_wrapper'; +AS 'MODULE_PATHNAME', 'local_moving_wrapper'; /* */ /* */ @@ -1690,83 +1615,114 @@ AS 'MODULE_PATHNAME', 'betweenness_centrality_wrapper'; /* */ /* */ --- crates/pg_warren/src/compute_sql/read.rs:98 --- pg_warren::compute_sql::read::matrix_rows - -CREATE FUNCTION warren.matrix_rows(graph text, name text, id_type anyelement, - version bigint DEFAULT NULL) -RETURNS TABLE (node_id anyelement, "row" float8[]) -STABLE PARALLEL RESTRICTED LANGUAGE c AS 'MODULE_PATHNAME', 'matrix_rows_wrapper'; +-- crates/pg_warren/src/compute_sql/reduce.rs:47 +-- pg_warren::compute_sql::reduce::vector_norm_diff +CREATE FUNCTION warren."vector_norm_diff"( + "graph" TEXT, /* Option < & str > */ + "a" TEXT, /* Option < & str > */ + "b" TEXT, /* Option < & str > */ + "norm" TEXT DEFAULT 'l2', /* Option < & str > */ + "a_version" bigint DEFAULT NULL, /* Option < i64 > */ + "b_version" bigint DEFAULT NULL /* Option < i64 > */ +) RETURNS double precision /* f64 */ +STABLE PARALLEL RESTRICTED +LANGUAGE c /* Rust */ +AS 'MODULE_PATHNAME', 'vector_norm_diff_wrapper'; /* */ /* */ --- crates/pg_warren/src/compute_sql/structure.rs:103 --- pg_warren::compute_sql::structure::topological_order -CREATE FUNCTION warren."topological_order"( - "graph" TEXT, /* Option < & str > */ - "out" TEXT, /* Option < & str > */ - "rels" TEXT[] DEFAULT NULL, /* :: std :: option :: Option < Vec < Option < String > > > */ - "persist" bool DEFAULT true, /* Option < bool > */ - "version" bigint DEFAULT NULL /* Option < i64 > */ -) RETURNS bigint /* Option < i64 > */ -VOLATILE PARALLEL UNSAFE -LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'topological_order_wrapper'; +-- crates/pg_warren/src/traversal_sql/neighbourhood.rs:83 +-- pg_warren::traversal_sql::neighbourhood::has_edge + +CREATE FUNCTION warren.has_edge(graph text, src anyelement, dst anyelement, + rel text DEFAULT NULL, version bigint DEFAULT NULL) +RETURNS boolean +STABLE PARALLEL SAFE LANGUAGE c AS 'MODULE_PATHNAME', 'has_edge_wrapper'; /* */ /* */ --- crates/pg_warren/src/compute_sql/centrality.rs:157 --- pg_warren::compute_sql::centrality::harmonic_centrality -CREATE FUNCTION warren."harmonic_centrality"( +-- crates/pg_warren/src/community_sql/contract.rs:40 +-- pg_warren::community_sql::contract::contract +CREATE FUNCTION warren."contract"( "graph" TEXT, /* Option < & str > */ - "out" TEXT, /* Option < & str > */ - "k" bigint DEFAULT NULL, /* Option < i64 > */ - "seed" bigint DEFAULT 0, /* Option < i64 > */ - "directed" bool DEFAULT true, /* Option < bool > */ - "sources" TEXT DEFAULT NULL, /* Option < & str > */ - "rels" TEXT[] DEFAULT NULL, /* :: std :: option :: Option < Vec < Option < String > > > */ - "weighted" bool DEFAULT NULL, /* Option < bool > */ - "persist" bool DEFAULT true, /* Option < bool > */ + "label_vector" TEXT, /* Option < & str > */ + "out_graph" TEXT, /* Option < & str > */ + "weighted" bool DEFAULT true, /* Option < bool > */ + "drop_self_loops" bool DEFAULT false, /* Option < bool > */ + "promote" bool DEFAULT true, /* Option < bool > */ "version" bigint DEFAULT NULL /* Option < i64 > */ ) RETURNS bigint /* Option < i64 > */ VOLATILE PARALLEL UNSAFE LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'harmonic_centrality_wrapper'; +AS 'MODULE_PATHNAME', 'contract_wrapper'; /* */ /* */ --- crates/pg_warren/src/catalog/dropped_roles/drop_owned.rs:15 --- pg_warren::catalog::dropped_roles::drop_owned::drop_owned - -CREATE FUNCTION warren."drop_owned"(role regrole) RETURNS bigint -STRICT VOLATILE LANGUAGE c AS 'MODULE_PATHNAME', 'drop_owned_wrapper'; +-- crates/pg_warren/src/catalog/info.rs:16 +-- pg_warren::catalog::info::info +CREATE FUNCTION warren."info"( + "name" TEXT /* & str */ +) RETURNS jsonb /* JsonB */ +STRICT STABLE PARALLEL SAFE +LANGUAGE c /* Rust */ +AS 'MODULE_PATHNAME', 'info_wrapper'; /* */ /* */ --- crates/pg_warren/src/spectral_sql/eigen.rs:91 --- pg_warren::spectral_sql::eigen::eigen -CREATE FUNCTION warren."eigen"( +-- crates/pg_warren/src/spectral_sql/distance/mod.rs:96 +-- pg_warren::spectral_sql::distance::distance +CREATE FUNCTION warren."distance"( "graph" TEXT, /* Option < & str > */ - "k" INT, /* Option < i32 > */ - "matrix" TEXT DEFAULT 'adjacency', /* Option < & str > */ + "v1" bigint, /* Option < i64 > */ + "v2" bigint, /* Option < i64 > */ + "method" TEXT, /* Option < & str > */ + "vectors_prefix" TEXT DEFAULT NULL, /* Option < & str > */ + "k" INT DEFAULT NULL, /* Option < i32 > */ + "x" TEXT DEFAULT NULL, /* Option < & str > */ + "y" TEXT DEFAULT NULL, /* Option < & str > */ + "compared_rows" TEXT DEFAULT 'shared', /* Option < & str > */ + "matrix" TEXT DEFAULT NULL, /* Option < & str > */ "which" TEXT DEFAULT 'largest', /* Option < & str > */ "rels" TEXT[] DEFAULT NULL, /* :: std :: option :: Option < Vec < Option < String > > > */ "weighted" bool DEFAULT NULL, /* Option < bool > */ "symmetrize" bool DEFAULT true, /* Option < bool > */ "tol" double precision DEFAULT 1e-6, /* Option < f64 > */ "max_iter" INT DEFAULT NULL, /* Option < i32 > */ - "vectors_prefix" TEXT DEFAULT NULL, /* Option < & str > */ - "version" bigint DEFAULT NULL, /* Option < i64 > */ "dtype" TEXT DEFAULT 'f32', /* Option < & str > */ - "persist" bool DEFAULT true /* Option < bool > */ -) RETURNS TABLE ( - "i" INT, /* i32 */ - "eigenvalue" double precision, /* f64 */ - "residual" double precision /* f64 */ -) -VOLATILE PARALLEL UNSAFE + "graph2" TEXT DEFAULT NULL /* Option < & str > */ +) RETURNS jsonb /* JsonB */ +STABLE PARALLEL RESTRICTED LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'eigen_wrapper'; +AS 'MODULE_PATHNAME', 'distance_wrapper'; +/* */ + +/* */ +-- crates/pg_warren/src/compute_sql/summary.rs:20 +-- pg_warren::compute_sql::summary::vector_topk + +CREATE FUNCTION warren.vector_topk(graph text, name text, k int, id_type anyelement, + descending bool DEFAULT true, version bigint DEFAULT NULL) +RETURNS TABLE (node_id anyelement, value float8) +STABLE PARALLEL RESTRICTED LANGUAGE c AS 'MODULE_PATHNAME', 'vector_topk_wrapper'; +/* */ + +/* */ +-- crates/pg_warren/src/catalog/readers.rs:25 +-- pg_warren::catalog::readers::revoke + +CREATE FUNCTION warren."revoke"(name text, role regrole) RETURNS void +STRICT VOLATILE LANGUAGE c AS 'MODULE_PATHNAME', 'revoke_wrapper'; +/* */ + +/* */ +-- crates/pg_warren/src/traversal_sql/paths.rs:71 +-- pg_warren::traversal_sql::paths::shortest_path + +CREATE FUNCTION warren.shortest_path(graph text, start anyelement, target anyelement, + max_depth int DEFAULT NULL, rels text[] DEFAULT NULL, + direction text DEFAULT 'out', version bigint DEFAULT NULL) +RETURNS anyarray +STABLE PARALLEL SAFE LANGUAGE c AS 'MODULE_PATHNAME', 'shortest_path_wrapper'; /* */ /* */ @@ -1782,62 +1738,65 @@ STABLE PARALLEL SAFE LANGUAGE c AS 'MODULE_PATHNAME', 'bfs_wrapper'; /* */ /* */ --- crates/pg_warren/src/community_sql/local_moving.rs:96 --- pg_warren::community_sql::local_moving::leiden_refine -CREATE FUNCTION warren."leiden_refine"( +-- crates/pg_warren/src/community_sql/projection.rs:26 +-- pg_warren::community_sql::projection::project +CREATE FUNCTION warren."project"( "graph" TEXT, /* Option < & str > */ - "communities" TEXT, /* Option < & str > */ - "out" TEXT, /* Option < & str > */ - "keys" TEXT DEFAULT NULL, /* Option < & str > */ - "resolution" double precision DEFAULT 1.0, /* Option < f64 > */ - "seed" bigint DEFAULT 0, /* Option < i64 > */ - "randomness" double precision DEFAULT 0.01, /* Option < f64 > */ - "weighted" bool DEFAULT NULL, /* Option < bool > */ + "x" TEXT, /* Option < & str > */ + "out_graph" TEXT, /* Option < & str > */ + "y" TEXT, /* Option < & str > */ + "agg" TEXT DEFAULT 'sum', /* Option < & str > */ "persist" bool DEFAULT true, /* Option < bool > */ "version" bigint DEFAULT NULL /* Option < i64 > */ -) RETURNS bigint /* Option < i64 > */ +) RETURNS void VOLATILE PARALLEL UNSAFE LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'leiden_refine_wrapper'; +AS 'MODULE_PATHNAME', 'project_wrapper'; /* */ /* */ --- crates/pg_warren/src/catalog/drop.rs:13 --- pg_warren::catalog::drop::drop_version -CREATE FUNCTION warren."drop_version"( +-- crates/pg_warren/src/catalog/create_graph.rs:42 +-- pg_warren::catalog::create_graph::create_graph +CREATE FUNCTION warren."create_graph"( "name" TEXT, /* & str */ - "version" bigint /* i64 */ + "node_query" TEXT, /* & str */ + "edge_query" TEXT, /* & str */ + "directed" bool DEFAULT true, /* bool */ + "weighted" bool DEFAULT false, /* bool */ + "has_relations" bool DEFAULT false, /* bool */ + "build_reverse" bool DEFAULT true, /* bool */ + "retain_versions" INT DEFAULT 2, /* i32 */ + "store_merged" bool DEFAULT true, /* bool */ + "node_order" TEXT DEFAULT 'id' /* & str */ ) RETURNS void STRICT LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'drop_version_wrapper'; +AS 'MODULE_PATHNAME', 'create_graph_wrapper'; /* */ /* */ --- crates/pg_warren/src/compute_sql/product.rs:172 --- pg_warren::compute_sql::product::spmv -CREATE FUNCTION warren."spmv"( - "graph" TEXT, /* Option < & str > */ - "x" TEXT, /* Option < & str > */ - "y" TEXT, /* Option < & str > */ - "semiring" TEXT DEFAULT 'plus_times', /* Option < & str > */ - "transpose" bool DEFAULT false, /* Option < bool > */ - "rels" TEXT[] DEFAULT NULL, /* :: std :: option :: Option < Vec < Option < String > > > */ - "weighted" bool DEFAULT NULL, /* Option < bool > */ - "mask" TEXT DEFAULT NULL, /* Option < & str > */ - "mask_complement" bool DEFAULT false, /* Option < bool > */ - "replace" bool DEFAULT false, /* Option < bool > */ - "accumulate" TEXT DEFAULT NULL, /* Option < & str > */ - "persist" bool DEFAULT true, /* Option < bool > */ - "version" bigint DEFAULT NULL /* Option < i64 > */ -) RETURNS void -VOLATILE PARALLEL UNSAFE +-- crates/pg_warren/src/catalog/dropped_roles/orphans.rs:34 +-- pg_warren::catalog::dropped_roles::orphans::orphans +CREATE FUNCTION warren."orphans"() RETURNS TABLE ( + "kind" TEXT, /* String */ + "graph" TEXT, /* String */ + "role" oid /* pg_sys :: Oid */ +) +STRICT VOLATILE PARALLEL SAFE LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'spmv_wrapper'; +AS 'MODULE_PATHNAME', 'orphans_wrapper'; +/* */ + +/* */ +-- crates/pg_warren/src/vector/quota/sql.rs:22 +-- pg_warren::vector::quota::sql::set_vector_quota + +CREATE FUNCTION warren."set_vector_quota"(name text, bytes bigint, role regrole DEFAULT NULL) +RETURNS void VOLATILE LANGUAGE c AS 'MODULE_PATHNAME', 'set_vector_quota_wrapper'; /* */ /* */ --- crates/pg_warren/src/compute_sql/read.rs:62 +-- crates/pg_warren/src/compute_sql/read.rs:64 -- pg_warren::compute_sql::read::vector_values CREATE FUNCTION warren.vector_values(graph text, name text, id_type anyelement, @@ -1847,58 +1806,56 @@ STABLE PARALLEL RESTRICTED LANGUAGE c AS 'MODULE_PATHNAME', 'vector_values_wrapp /* */ /* */ --- crates/pg_warren/src/catalog/readers.rs:25 --- pg_warren::catalog::readers::revoke - -CREATE FUNCTION warren."revoke"(name text, role regrole) RETURNS void -STRICT VOLATILE LANGUAGE c AS 'MODULE_PATHNAME', 'revoke_wrapper'; +-- crates/pg_warren/src/catalog/promote.rs:14 +-- pg_warren::catalog::promote::promote +CREATE FUNCTION warren."promote"( + "name" TEXT, /* & str */ + "version" bigint /* i64 */ +) RETURNS void +STRICT +LANGUAGE c /* Rust */ +AS 'MODULE_PATHNAME', 'promote_wrapper'; /* */ /* */ --- crates/pg_warren/src/cache/warm.rs:37 --- pg_warren::cache::warm::evict -CREATE FUNCTION warren."evict"( - "name" TEXT /* & str */ +-- crates/pg_warren/src/cache/warm.rs:18 +-- pg_warren::cache::warm::prewarm +CREATE FUNCTION warren."prewarm"( + "name" TEXT, /* & str */ + "version" bigint DEFAULT NULL /* Option < i64 > */ ) RETURNS bigint /* i64 */ -STRICT VOLATILE PARALLEL RESTRICTED +VOLATILE PARALLEL RESTRICTED LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'evict_wrapper'; +AS 'MODULE_PATHNAME', 'prewarm_wrapper'; /* */ /* */ --- crates/pg_warren/src/traversal_sql/step.rs:93 --- pg_warren::traversal_sql::step::s +-- crates/pg_warren/src/catalog/dropped_roles/event_trigger.rs:21 +-- pg_warren::catalog::dropped_roles::event_trigger::on_drop_owned -CREATE FUNCTION warren.s(rels text[], direction text DEFAULT 'out', filter "any" DEFAULT NULL, - min_hops int DEFAULT 1, max_hops int DEFAULT NULL) -RETURNS warren.step -STABLE PARALLEL SAFE LANGUAGE c AS 'MODULE_PATHNAME', 's_wrapper'; +CREATE FUNCTION warren."on_drop_owned"() RETURNS event_trigger +VOLATILE LANGUAGE c AS 'MODULE_PATHNAME', 'on_drop_owned_wrapper'; /* */ /* */ --- crates/pg_warren/src/community_sql/modularity.rs:26 --- pg_warren::community_sql::modularity::modularity -CREATE FUNCTION warren."modularity"( - "graph" TEXT, /* Option < & str > */ - "labels" TEXT, /* Option < & str > */ - "resolution" double precision DEFAULT 1.0, /* Option < f64 > */ - "weighted" bool DEFAULT NULL, /* Option < bool > */ - "version" bigint DEFAULT NULL /* Option < i64 > */ -) RETURNS double precision /* Option < f64 > */ -STABLE PARALLEL RESTRICTED -LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'modularity_wrapper'; +-- crates/pg_warren/src/traversal_sql/bfs.rs:87 +-- pg_warren::traversal_sql::bfs::reachable_set + +CREATE FUNCTION warren.reachable_set(graph text, start anyarray, max_depth int DEFAULT NULL, + rels text[] DEFAULT NULL, direction text DEFAULT 'out', + version bigint DEFAULT NULL) +RETURNS SETOF anyelement +STABLE PARALLEL SAFE LANGUAGE c AS 'MODULE_PATHNAME', 'reachable_set_wrapper'; /* */ /* */ --- crates/pg_warren/src/traversal_sql/paths.rs:69 --- pg_warren::traversal_sql::paths::shortest_path +-- crates/pg_warren/src/traversal_sql/neighbourhood.rs:54 +-- pg_warren::traversal_sql::neighbourhood::degree -CREATE FUNCTION warren.shortest_path(graph text, start anyelement, target anyelement, - max_depth int DEFAULT NULL, rels text[] DEFAULT NULL, - direction text DEFAULT 'out', version bigint DEFAULT NULL) -RETURNS anyarray -STABLE PARALLEL SAFE LANGUAGE c AS 'MODULE_PATHNAME', 'shortest_path_wrapper'; +CREATE FUNCTION warren.degree(graph text, node anyelement, direction text DEFAULT 'out', + rels text[] DEFAULT NULL, version bigint DEFAULT NULL) +RETURNS bigint +STABLE PARALLEL SAFE LANGUAGE c AS 'MODULE_PATHNAME', 'degree_wrapper'; /* */ /* */ @@ -1925,21 +1882,70 @@ AS 'MODULE_PATHNAME', 'spmm_wrapper'; /* */ /* */ --- crates/pg_warren/src/catalog/owner/mod.rs:27 --- pg_warren::catalog::owner::set_owner +-- crates/pg_warren/src/compute_sql/constructors.rs:163 +-- pg_warren::compute_sql::constructors::triangles +CREATE FUNCTION warren."triangles"( + "graph" TEXT, /* Option < & str > */ + "out" TEXT, /* Option < & str > */ + "rels" TEXT[] DEFAULT NULL, /* :: std :: option :: Option < Vec < Option < String > > > */ + "persist" bool DEFAULT true, /* Option < bool > */ + "version" bigint DEFAULT NULL /* Option < i64 > */ +) RETURNS bigint /* Option < i64 > */ +VOLATILE PARALLEL UNSAFE +LANGUAGE c /* Rust */ +AS 'MODULE_PATHNAME', 'triangles_wrapper'; +/* */ -CREATE FUNCTION warren."set_owner"(name text, owner regrole) RETURNS void -STRICT VOLATILE LANGUAGE c AS 'MODULE_PATHNAME', 'set_owner_wrapper'; +/* */ +-- crates/pg_warren/src/community_sql/modularity.rs:26 +-- pg_warren::community_sql::modularity::modularity +CREATE FUNCTION warren."modularity"( + "graph" TEXT, /* Option < & str > */ + "labels" TEXT, /* Option < & str > */ + "resolution" double precision DEFAULT 1.0, /* Option < f64 > */ + "weighted" bool DEFAULT NULL, /* Option < bool > */ + "version" bigint DEFAULT NULL /* Option < i64 > */ +) RETURNS double precision /* Option < f64 > */ +STABLE PARALLEL RESTRICTED +LANGUAGE c /* Rust */ +AS 'MODULE_PATHNAME', 'modularity_wrapper'; /* */ /* */ --- crates/pg_warren/src/traversal_sql/neighbourhood.rs:23 --- pg_warren::traversal_sql::neighbourhood::neighbours +-- crates/pg_warren/src/compute_sql/similarity.rs:185 +-- pg_warren::compute_sql::similarity::most_similar -CREATE FUNCTION warren.neighbours(graph text, node anyelement, direction text DEFAULT 'out', - rels text[] DEFAULT NULL, version bigint DEFAULT NULL) -RETURNS SETOF anyelement -STABLE PARALLEL SAFE LANGUAGE c AS 'MODULE_PATHNAME', 'neighbours_wrapper'; +CREATE FUNCTION warren.most_similar(graph text, node anyelement, k bigint DEFAULT 10, + measure text DEFAULT 'jaccard', rels text[] DEFAULT NULL, + version bigint DEFAULT NULL) +RETURNS TABLE (node_id anyelement, score float8) +STABLE PARALLEL RESTRICTED LANGUAGE c AS 'MODULE_PATHNAME', 'most_similar_wrapper'; +/* */ + +/* */ +-- crates/pg_warren/src/cache/warm.rs:37 +-- pg_warren::cache::warm::evict +CREATE FUNCTION warren."evict"( + "name" TEXT /* & str */ +) RETURNS bigint /* i64 */ +STRICT VOLATILE PARALLEL RESTRICTED +LANGUAGE c /* Rust */ +AS 'MODULE_PATHNAME', 'evict_wrapper'; +/* */ + +/* */ +-- crates/pg_warren/src/compute_sql/structure.rs:103 +-- pg_warren::compute_sql::structure::topological_order +CREATE FUNCTION warren."topological_order"( + "graph" TEXT, /* Option < & str > */ + "out" TEXT, /* Option < & str > */ + "rels" TEXT[] DEFAULT NULL, /* :: std :: option :: Option < Vec < Option < String > > > */ + "persist" bool DEFAULT true, /* Option < bool > */ + "version" bigint DEFAULT NULL /* Option < i64 > */ +) RETURNS bigint /* Option < i64 > */ +VOLATILE PARALLEL UNSAFE +LANGUAGE c /* Rust */ +AS 'MODULE_PATHNAME', 'topological_order_wrapper'; /* */ /* */ @@ -1958,154 +1964,148 @@ AS 'MODULE_PATHNAME', 'vector_map_wrapper'; /* */ /* */ --- crates/pg_warren/src/compute_sql/drop.rs:16 --- pg_warren::compute_sql::drop::drop_vector -CREATE FUNCTION warren."drop_vector"( +-- crates/pg_warren/src/compute_sql/structure.rs:28 +-- pg_warren::compute_sql::structure::core_number +CREATE FUNCTION warren."core_number"( "graph" TEXT, /* Option < & str > */ - "name" TEXT, /* Option < & str > */ - "version" bigint DEFAULT NULL, /* Option < i64 > */ - "if_exists" bool DEFAULT false /* Option < bool > */ -) RETURNS void + "out" TEXT, /* Option < & str > */ + "rels" TEXT[] DEFAULT NULL, /* :: std :: option :: Option < Vec < Option < String > > > */ + "persist" bool DEFAULT true, /* Option < bool > */ + "version" bigint DEFAULT NULL /* Option < i64 > */ +) RETURNS bigint /* Option < i64 > */ VOLATILE PARALLEL UNSAFE LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'drop_vector_wrapper'; -/* */ - -/* */ --- crates/pg_warren/src/catalog/info.rs:16 --- pg_warren::catalog::info::info -CREATE FUNCTION warren."info"( - "name" TEXT /* & str */ -) RETURNS jsonb /* JsonB */ -STRICT STABLE PARALLEL SAFE -LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'info_wrapper'; +AS 'MODULE_PATHNAME', 'core_number_wrapper'; /* */ /* */ --- crates/pg_warren/src/version_sql/diff.rs:23 --- pg_warren::version_sql::diff::diff -CREATE FUNCTION warren."diff"( - "name" TEXT, /* & str */ - "v1" bigint, /* i64 */ - "v2" bigint /* i64 */ -) RETURNS jsonb /* JsonB */ -STRICT STABLE PARALLEL RESTRICTED +-- crates/pg_warren/src/traversal_sql/stats.rs:30 +-- pg_warren::traversal_sql::stats::last_stats +CREATE FUNCTION warren."last_stats"() RETURNS jsonb /* Option < JsonB > */ +STRICT VOLATILE PARALLEL RESTRICTED LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'diff_wrapper'; +AS 'MODULE_PATHNAME', 'last_stats_wrapper'; /* */ /* */ --- crates/pg_warren/src/catalog/drop.rs:29 --- pg_warren::catalog::drop::drop_graph -CREATE FUNCTION warren."drop_graph"( - "name" TEXT /* & str */ -) RETURNS void -STRICT +-- crates/pg_warren/src/compute_sql/structure.rs:63 +-- pg_warren::compute_sql::structure::strong_components_vector +CREATE FUNCTION warren."strong_components_vector"( + "graph" TEXT, /* Option < & str > */ + "out" TEXT, /* Option < & str > */ + "rels" TEXT[] DEFAULT NULL, /* :: std :: option :: Option < Vec < Option < String > > > */ + "persist" bool DEFAULT true, /* Option < bool > */ + "version" bigint DEFAULT NULL /* Option < i64 > */ +) RETURNS bigint /* Option < i64 > */ +VOLATILE PARALLEL UNSAFE LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'drop_graph_wrapper'; +AS 'MODULE_PATHNAME', 'strong_components_vector_wrapper'; /* */ /* */ --- crates/pg_warren/src/catalog/dropped_roles/orphans.rs:70 --- pg_warren::catalog::dropped_roles::orphans::drop_orphans -CREATE FUNCTION warren."drop_orphans"() RETURNS bigint /* i64 */ -STRICT +-- crates/pg_warren/src/compute_sql/centrality.rs:157 +-- pg_warren::compute_sql::centrality::harmonic_centrality +CREATE FUNCTION warren."harmonic_centrality"( + "graph" TEXT, /* Option < & str > */ + "out" TEXT, /* Option < & str > */ + "k" bigint DEFAULT NULL, /* Option < i64 > */ + "seed" bigint DEFAULT 0, /* Option < i64 > */ + "directed" bool DEFAULT true, /* Option < bool > */ + "sources" TEXT DEFAULT NULL, /* Option < & str > */ + "rels" TEXT[] DEFAULT NULL, /* :: std :: option :: Option < Vec < Option < String > > > */ + "weighted" bool DEFAULT NULL, /* Option < bool > */ + "persist" bool DEFAULT true, /* Option < bool > */ + "version" bigint DEFAULT NULL /* Option < i64 > */ +) RETURNS bigint /* Option < i64 > */ +VOLATILE PARALLEL UNSAFE LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'drop_orphans_wrapper'; +AS 'MODULE_PATHNAME', 'harmonic_centrality_wrapper'; /* */ /* */ --- crates/pg_warren/src/compute_sql/similarity.rs:179 --- pg_warren::compute_sql::similarity::most_similar +-- crates/pg_warren/src/traversal_sql/paths.rs:27 +-- pg_warren::traversal_sql::paths::reachable -CREATE FUNCTION warren.most_similar(graph text, node anyelement, k bigint DEFAULT 10, - measure text DEFAULT 'jaccard', rels text[] DEFAULT NULL, - version bigint DEFAULT NULL) -RETURNS TABLE (node_id anyelement, score float8) -STABLE PARALLEL RESTRICTED LANGUAGE c AS 'MODULE_PATHNAME', 'most_similar_wrapper'; +CREATE FUNCTION warren.reachable(graph text, start anyelement, target anyelement, + max_depth int DEFAULT NULL, rels text[] DEFAULT NULL, + direction text DEFAULT 'out', version bigint DEFAULT NULL) +RETURNS boolean +STABLE PARALLEL SAFE LANGUAGE c AS 'MODULE_PATHNAME', 'reachable_wrapper'; /* */ /* */ --- crates/pg_warren/src/cache/warm.rs:18 --- pg_warren::cache::warm::prewarm -CREATE FUNCTION warren."prewarm"( +-- crates/pg_warren/src/build/refresh.rs:21 +-- pg_warren::build::refresh::refresh +CREATE FUNCTION warren."refresh"( "name" TEXT, /* & str */ - "version" bigint DEFAULT NULL /* Option < i64 > */ + "promote" bool DEFAULT true, /* bool */ + "label" TEXT DEFAULT NULL /* Option < & str > */ ) RETURNS bigint /* i64 */ -VOLATILE PARALLEL RESTRICTED + LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'prewarm_wrapper'; +AS 'MODULE_PATHNAME', 'refresh_wrapper'; /* */ /* */ --- crates/pg_warren/src/compute_sql/structure.rs:28 --- pg_warren::compute_sql::structure::core_number -CREATE FUNCTION warren."core_number"( +-- crates/pg_warren/src/compute_sql/constructors.rs:132 +-- pg_warren::compute_sql::constructors::vector_dense_id +CREATE FUNCTION warren."vector_dense_id"( "graph" TEXT, /* Option < & str > */ - "out" TEXT, /* Option < & str > */ - "rels" TEXT[] DEFAULT NULL, /* :: std :: option :: Option < Vec < Option < String > > > */ + "name" TEXT, /* Option < & str > */ + "dtype" TEXT DEFAULT 'i64', /* Option < & str > */ "persist" bool DEFAULT true, /* Option < bool > */ "version" bigint DEFAULT NULL /* Option < i64 > */ ) RETURNS bigint /* Option < i64 > */ VOLATILE PARALLEL UNSAFE LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'core_number_wrapper'; -/* */ - -/* */ --- crates/pg_warren/src/vector/quota/sql.rs:18 --- pg_warren::vector::quota::sql::set_vector_quota - -CREATE FUNCTION warren."set_vector_quota"(name text, bytes bigint, role regrole DEFAULT NULL) -RETURNS void VOLATILE LANGUAGE c AS 'MODULE_PATHNAME', 'set_vector_quota_wrapper'; +AS 'MODULE_PATHNAME', 'vector_dense_id_wrapper'; /* */ /* */ --- crates/pg_warren/src/traversal_sql/neighbourhood.rs:81 --- pg_warren::traversal_sql::neighbourhood::has_edge - -CREATE FUNCTION warren.has_edge(graph text, src anyelement, dst anyelement, - rel text DEFAULT NULL, version bigint DEFAULT NULL) -RETURNS boolean -STABLE PARALLEL SAFE LANGUAGE c AS 'MODULE_PATHNAME', 'has_edge_wrapper'; +-- crates/pg_warren/src/extension.rs:91 +-- pg_warren::extension::version +CREATE FUNCTION warren."version"() RETURNS TEXT /* & '_ str */ +STRICT STABLE PARALLEL SAFE +LANGUAGE c /* Rust */ +AS 'MODULE_PATHNAME', 'version_wrapper'; /* */ /* */ --- crates/pg_warren/src/compute_sql/constructors.rs:38 --- pg_warren::compute_sql::constructors::vector_const -CREATE FUNCTION warren."vector_const"( +-- crates/pg_warren/src/compute_sql/drop.rs:16 +-- pg_warren::compute_sql::drop::drop_vector +CREATE FUNCTION warren."drop_vector"( "graph" TEXT, /* Option < & str > */ "name" TEXT, /* Option < & str > */ - "value" double precision, /* Option < f64 > */ - "dtype" TEXT DEFAULT 'f64', /* Option < & str > */ - "persist" bool DEFAULT true, /* Option < bool > */ - "version" bigint DEFAULT NULL /* Option < i64 > */ -) RETURNS bigint /* Option < i64 > */ + "version" bigint DEFAULT NULL, /* Option < i64 > */ + "if_exists" bool DEFAULT false /* Option < bool > */ +) RETURNS void VOLATILE PARALLEL UNSAFE LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'vector_const_wrapper'; +AS 'MODULE_PATHNAME', 'drop_vector_wrapper'; /* */ /* */ --- crates/pg_warren/src/community_sql/local_moving.rs:29 --- pg_warren::community_sql::local_moving::local_moving -CREATE FUNCTION warren."local_moving"( +-- crates/pg_warren/src/catalog/owner/reassign.rs:22 +-- pg_warren::catalog::owner::reassign::reassign_owned + +CREATE FUNCTION warren."reassign_owned"("old" regrole, "new" regrole) RETURNS bigint +STRICT VOLATILE LANGUAGE c AS 'MODULE_PATHNAME', 'reassign_owned_wrapper'; +/* */ + +/* */ +-- crates/pg_warren/src/community_sql/projection.rs:60 +-- pg_warren::community_sql::projection::lift +CREATE FUNCTION warren."lift"( + "out_graph" TEXT, /* Option < & str > */ + "y" TEXT, /* Option < & str > */ "graph" TEXT, /* Option < & str > */ - "out" TEXT, /* Option < & str > */ - "initial" TEXT DEFAULT NULL, /* Option < & str > */ - "keys" TEXT DEFAULT NULL, /* Option < & str > */ - "resolution" double precision DEFAULT 1.0, /* Option < f64 > */ - "seed" bigint DEFAULT 0, /* Option < i64 > */ - "max_sweeps" INT DEFAULT 100, /* Option < i32 > */ - "tolerance" double precision DEFAULT 1e-7, /* Option < f64 > */ - "weighted" bool DEFAULT NULL, /* Option < bool > */ + "x" TEXT, /* Option < & str > */ "persist" bool DEFAULT true, /* Option < bool > */ "version" bigint DEFAULT NULL /* Option < i64 > */ -) RETURNS bigint /* Option < i64 > */ +) RETURNS void VOLATILE PARALLEL UNSAFE LANGUAGE c /* Rust */ -AS 'MODULE_PATHNAME', 'local_moving_wrapper'; +AS 'MODULE_PATHNAME', 'lift_wrapper'; /* */ /* */ diff --git a/test/dump.py b/test/dump.py index 8524b27f..d5fd8d65 100755 --- a/test/dump.py +++ b/test/dump.py @@ -3,7 +3,8 @@ --no-install), builds graphs in a scratch cluster, dumps them with pg_dump (plain and custom formats), restores each dump into a fresh database and checks that pg_warren answers as before. Also checks a trusted install's -installer dump (DESIGN.md §7). Holds /tmp/pgrx-test.lock for the whole +installer dump (DESIGN.md §7), and that a text graph restored into a +database of another encoding is refused (DESIGN.md §5.7). Holds /tmp/pgrx-test.lock for the whole run.""" import argparse @@ -31,12 +32,13 @@ def main(argv): extension.install(args.pg_config) with Cluster(args.pg_config, args.runas) as cluster: Installer.trusted().prepare(cluster) - failures = dump.check(cluster, directory) + dump.check_trusted(cluster, directory) + failures = (dump.check(cluster, directory) + dump.check_trusted(cluster, directory) + + dump.check_cross_encoding(cluster, directory)) finally: shutil.rmtree(directory, ignore_errors=True) for failure in failures: print(failure) - print(f"dump and restore (plain, custom, trusted): {'FAILED' if failures else 'ok'}") + print(f"dump and restore (plain, custom, trusted, cross-encoding): {'FAILED' if failures else 'ok'}") return 1 if failures else 0 diff --git a/test/harness/dump.py b/test/harness/dump.py index b9c20b74..227301f2 100644 --- a/test/harness/dump.py +++ b/test/harness/dump.py @@ -39,7 +39,11 @@ FROM public.nodes; CREATE TABLE public.small_nodes AS SELECT generate_series(1, 3) AS id; CREATE TABLE public.small_edges AS SELECT 1 AS src, 2 AS dst; - GRANT SELECT ON public.nodes, public.edges, public.small_nodes, public.small_edges TO dump_owner; + CREATE TABLE public.iri_nodes AS SELECT 'https://example.org/' || id AS id FROM public.nodes; + INSERT INTO public.iri_nodes VALUES (''), ('é'); + CREATE TABLE public.iri_edges AS SELECT id::varchar AS src, 'é' AS dst FROM public.iri_nodes; + GRANT SELECT ON public.nodes, public.edges, public.small_nodes, public.small_edges, + public.iri_nodes, public.iri_edges TO dump_owner; SET ROLE dump_owner; SELECT warren.create_graph('g', 'SELECT id FROM public.nodes', 'SELECT src, dst, rel, w FROM public.edges', @@ -49,12 +53,18 @@ SELECT warren.refresh('g'); SELECT warren.vector_const('g', 'one', 1); SELECT warren.refresh('small'); + SELECT warren.create_graph('iri', 'SELECT id FROM public.iri_nodes', + 'SELECT src, dst FROM public.iri_edges'); + SELECT warren.refresh('iri'); + SELECT warren.vector_from_query('iri', 'len', 'SELECT id, length(id) FROM public.iri_nodes'); SELECT warren.grant('g', 'dump_reader'); SELECT warren.set_vector_quota('g', 1000000); SELECT warren.set_vector_quota('g', 2000000, 'dump_reader'); RESET ROLE; DELETE FROM public.nodes WHERE id = 8; SELECT warren.refresh('g'); + DELETE FROM public.iri_nodes WHERE id = ''; + SELECT warren.refresh('iri'); SELECT warren.compact('g'); SELECT warren.vector_degree('g', 'deg'); SET ROLE dump_reader; @@ -92,6 +102,14 @@ FROM warren.versions v JOIN warren.graphs g USING (graph_id), warren.neighbours(g.name, 1, 'both', version := v.version_id) n WHERE g.name = 'small' ORDER BY 1, 2""", + "text_ids": """ + SELECT v.version_id, m.node_id, m.dense_id, m.live, + ARRAY(SELECT warren.neighbours(g.name, m.node_id, 'in', version := v.version_id) n ORDER BY n), + (SELECT x.value FROM warren.vectors w, warren.vector_values(g.name, w.name, NULL::text, w.version_id) x + WHERE w.graph_id = g.graph_id AND w.version_id = v.version_id AND x.node_id = m.node_id) + FROM warren.versions v JOIN warren.graphs g USING (graph_id), + warren.id_map(g.name, NULL::text, v.version_id) m + WHERE g.name = 'iri' AND m.live ORDER BY 1, 3""", "lifted": "SELECT * FROM public.lifted() ORDER BY 1", "quotas": """ SELECT g.name, u.role::text, u.vectors, u.bytes, coalesce(u.quota::text, '') @@ -299,3 +317,59 @@ def check_trusted(cluster, directory): if rebuilt != expected: failures.append(f"trusted: the rebuilt graph answers {rebuilt}, not {expected}") return failures + + +# A text graph over ids that convert to any encoding, read back in the +# restored database: refused there if that database's encoding is not the +# one its segments record (DESIGN.md §5.7, §14 item 42). +ENCODED_SEED = """ + CREATE TABLE public.ascii_nodes AS SELECT 'node-' || n AS id FROM generate_series(1, 4) n; + CREATE TABLE public.ascii_edges AS SELECT 'node-1' AS src, 'node-2' AS dst; + SELECT warren.create_graph('ascii', 'SELECT id FROM public.ascii_nodes', + 'SELECT src, dst FROM public.ascii_edges'); + SELECT warren.refresh('ascii'); +""" +ENCODING_REFUSED = """ + DO $$ + DECLARE + message text; + BEGIN + PERFORM warren.degree('ascii', 'node-1'::text); + RAISE EXCEPTION 'a UTF8 text graph was read in a LATIN1 database'; + EXCEPTION WHEN object_not_in_prerequisite_state THEN + GET STACKED DIAGNOSTICS message = MESSAGE_TEXT; + IF message NOT LIKE '%encoding%' THEN + RAISE EXCEPTION 'refused for another reason: %', message; + END IF; + END $$; +""" + + +def check_cross_encoding(cluster, directory): + """Checks that a text graph dumped from a UTF8 database and restored + into a LATIN1 one is refused there (55000), and still read in a UTF8 + one; returns the failures.""" + admin = cluster.database() + admin.execute("CREATE DATABASE dump_utf8 ENCODING 'UTF8' TEMPLATE template0 LOCALE 'C'") + source = cluster.database("dump_utf8") + source.execute("CREATE EXTENSION pg_warren") + source.execute(ENCODED_SEED) + path = directory / "encoded.custom" + error = _run(dump_command(cluster, "dump_utf8", path, "-Fc")) + if error: + return [f"cross-encoding dump: {error}"] + failures = [] + for name, encoding in (("dump_latin1", "LATIN1"), ("dump_utf8_again", "UTF8")): + admin.execute(f"CREATE DATABASE {name} ENCODING '{encoding}' TEMPLATE template0 LOCALE 'C'") + error = _run(restore_command(cluster, name, path, True)) + if error: + failures.append(f"{encoding} restore: {error}") + continue + restored = cluster.database(name) + if encoding == "LATIN1": + error = _run(restored.command("-c", ENCODING_REFUSED)) + if error: + failures.append(f"LATIN1: {error}") + elif (found := restored.query("SELECT warren.degree('ascii', 'node-1'::text)")) != [["1"]]: + failures.append(f"UTF8: the restored text graph gives degree {found}, not 1") + return failures diff --git a/test/harness/upgrade.py b/test/harness/upgrade.py index 4222d2c0..ebc32510 100644 --- a/test/harness/upgrade.py +++ b/test/harness/upgrade.py @@ -325,7 +325,29 @@ def after_update(upgraded): expected = "upgrade:id upgrade_local:locality upgrade_new:id upgrade_typed:locality" if orders != [[expected]]: problems.append(f"node orders after compacting and creating are {orders}") - return problems + contraction_problems(upgraded) + quota_problems(upgraded) + return (problems + contraction_problems(upgraded) + text_problems(upgraded) + + quota_problems(upgraded)) + + +def text_problems(upgraded): + """Builds a graph with text node ids (0.6.0, DESIGN.md §5.7), which no + release before it could, on the updated catalog, and refreshes it.""" + upgraded.execute( + "CREATE TABLE upgrade_iris AS SELECT 'https://example.org/' || id AS id FROM upgrade_nodes;" + " SELECT warren.create_graph('upgrade_text', 'SELECT id FROM upgrade_iris'," + " $q$SELECT 'https://example.org/' || src, ('https://example.org/' || dst)::varchar" + " FROM upgrade_edges$q$);" + " SELECT warren.refresh('upgrade_text');" + " DELETE FROM upgrade_iris WHERE id LIKE '%/1';" + " SELECT warren.refresh('upgrade_text')") + found = upgraded.query( + "SELECT (SELECT id_type::text FROM warren.graphs WHERE name = 'upgrade_text') || ':'" + " || (SELECT string_agg(n, ',' ORDER BY n)" + " FROM warren.neighbours('upgrade_text', 'https://example.org/2'::text, 'both') n)") + expected = "text:https://example.org/3" + if found != [[expected]]: + return [f"a text graph after the update gives {found}, not {expected}"] + return [] # A quota set on a graph from before the update (0.6.0, DESIGN.md §5.7): diff --git a/test/sql/a4_typed_retrieval_iris.sql b/test/sql/a4_typed_retrieval_iris.sql new file mode 100644 index 00000000..8a169ffc --- /dev/null +++ b/test/sql/a4_typed_retrieval_iris.sql @@ -0,0 +1,85 @@ +-- Appendix A.4, typed multi-hop retrieval, on a research knowledge graph +-- keyed by IRIs (DESIGN.md §5.7, §9.2): the UK institutions reachable from +-- a root paper through 1–3 citation hops, then authorship by an active +-- person, then affiliation, as warren.walk finds them and as a chain of +-- recursive CTEs over the text keys does, against the fixture's truth. +-- fixture: citation --papers 20000 --people 6000 --institutions 400 --seed 1 + +CREATE FUNCTION iri(kind text, id bigint) RETURNS text IMMUTABLE LANGUAGE sql +RETURN 'https://example.org/' || kind || '/' || id; + +CREATE TABLE kg_nodes AS +SELECT iri('paper', id) AS id FROM papers +UNION ALL SELECT iri('person', id) FROM people +UNION ALL SELECT iri('institution', id) FROM institutions; +CREATE TABLE kg_edges AS +SELECT iri('paper', citing) AS src, iri('paper', cited) AS dst, 'cites' AS rel FROM citations +UNION ALL SELECT iri('paper', paper_id), iri('person', person_id), 'authored_by' FROM authorship +UNION ALL SELECT iri('person', person_id), iri('institution', institution_id), 'affiliated_with' FROM affiliations; +CREATE INDEX ON kg_edges (src, rel); +CREATE TABLE active_people AS SELECT iri('person', id) AS id FROM people WHERE active; +CREATE TABLE uk AS SELECT iri('institution', id) AS id FROM institutions WHERE country = 'GB'; +SELECT iri('paper', value::text::bigint) AS root FROM truth WHERE key = 'root' \gset + +SELECT warren.create_graph('kg', 'SELECT id FROM kg_nodes', 'SELECT src, dst, rel FROM kg_edges', + has_relations := true); +SELECT warren.refresh('kg'); + +CREATE TABLE steps AS +SELECT ARRAY[warren.s(ARRAY['cites'], 'out', min_hops := 1, max_hops := 3), + warren.s(ARRAY['authored_by'], 'out', ARRAY(SELECT id FROM active_people)), + warren.s(ARRAY['affiliated_with'], 'out', ARRAY(SELECT id FROM uk))] AS filtered, + ARRAY[warren.s(ARRAY['cites'], 'out', min_hops := 1, max_hops := 3), + warren.s(ARRAY['authored_by'], 'out'), + warren.s(ARRAY['affiliated_with'], 'out', ARRAY(SELECT id FROM uk))] AS unfiltered; + +CREATE TABLE walked AS +SELECT end_id FROM warren.walk('kg', ARRAY[:'root'], (SELECT filtered FROM steps)); +SELECT (warren.last_stats() ->> 'nodes_visited')::bigint AS visited_filtered \gset + +-- The oracle: one recursive CTE per step, over the text keys. +CREATE TABLE oracle AS +WITH RECURSIVE near (id, depth) AS ( + SELECT :'root'::text, 0 + UNION + SELECT e.dst, n.depth + 1 FROM near n JOIN kg_edges e ON e.src = n.id AND e.rel = 'cites' WHERE n.depth < 3), +authors AS ( + SELECT DISTINCT e.dst AS id + FROM (SELECT DISTINCT id FROM near WHERE depth >= 1) p + JOIN kg_edges e ON e.src = p.id AND e.rel = 'authored_by' JOIN active_people a ON a.id = e.dst) +SELECT DISTINCT e.dst AS id +FROM authors a JOIN kg_edges e ON e.src = a.id AND e.rel = 'affiliated_with' JOIN uk ON uk.id = e.dst; + +SELECT assert.that((SELECT count(*) FROM walked) > 5, 'the walk reaches several institutions'); +SELECT assert.equal(ARRAY(SELECT end_id FROM walked ORDER BY 1), ARRAY(SELECT id FROM oracle ORDER BY 1), + 'walk ends equal the recursive CTEs over text keys'); +SELECT assert.equal(ARRAY(SELECT end_id FROM walked ORDER BY 1), + ARRAY(SELECT iri('institution', institution_id) FROM truth_walk_ends ORDER BY 1), + 'walk ends equal the fixture''s truth'); +SELECT assert.equal((SELECT count FROM warren.walk('kg', ARRAY[:'root'], (SELECT filtered FROM steps), 'count')), + (SELECT count(*) FROM walked), 'count is the cardinality of ends'); + +-- Paths: at most max_results, each along edges of the right relations, +-- through active authors, ending at a UK institution. +CREATE TABLE paths AS +SELECT end_id, path, rels FROM warren.walk('kg', ARRAY[:'root'], (SELECT filtered FROM steps), 'paths', + distinct_ends := false, max_results := 1000); +SELECT assert.that((SELECT count(*) FROM paths) BETWEEN 1 AND 1000, 'paths are bounded by max_results'); +SELECT assert.equal((SELECT count(*) FROM paths p, generate_subscripts(p.rels, 1) i + WHERE NOT EXISTS (SELECT FROM kg_edges e WHERE e.src = p.path[i] AND e.dst = p.path[i + 1] + AND e.rel = p.rels[i])), + 0::bigint, 'every hop of every path is an edge of its relation'); +SELECT assert.equal((SELECT count(*) FROM paths p + WHERE p.path[array_length(p.path, 1) - 1] NOT IN (SELECT id FROM active_people) + OR p.end_id NOT IN (SELECT id FROM uk) OR p.end_id <> p.path[array_length(p.path, 1)]), + 0::bigint, 'every path passes the filters'); + +-- Filters prune while walking: without the second step's, more is visited. +SELECT count(*) FROM warren.walk('kg', ARRAY[:'root'], (SELECT unfiltered FROM steps)); +SELECT assert.that((warren.last_stats() ->> 'nodes_visited')::bigint > :visited_filtered, + 'dropping the authorship filter visits more nodes'); + +-- Reversed from the ends, the walks lead back to the root. +SELECT assert.that(:'root' IN (SELECT end_id FROM warren.walk('kg', ARRAY(SELECT end_id FROM walked), + (SELECT filtered FROM steps), reverse := true)), + 'the reversed walk reports the root among its starts'); diff --git a/test/sql/shared_cache_parallel_text.sql b/test/sql/shared_cache_parallel_text.sql new file mode 100644 index 00000000..2ec6a9f2 --- /dev/null +++ b/test/sql/shared_cache_parallel_text.sql @@ -0,0 +1,52 @@ +-- A text graph's PARALLEL SAFE reads in a parallel worker (DESIGN.md §5.6, +-- §5.7, §14 item 33): the worker reads the text id map as the leader does, +-- from the shared version or its own private copy, and returns the rows +-- the leader would, ids and all. +\ir include/shared_cache.sql +CREATE TABLE nodes AS SELECT 'https://example.org/node/' || n AS id, n FROM generate_series(1, 20011) n; +CREATE TABLE edges AS SELECT a.id AS src, b.id AS dst FROM nodes a JOIN nodes b ON b.n = a.n % 20011 + 1; +SELECT warren.create_graph('g', 'SELECT id FROM nodes', 'SELECT src, dst FROM edges'); +SELECT warren.refresh('g'); +CREATE TABLE pairs AS SELECT src, dst FROM edges; +ANALYZE pairs; +-- The setting that runs a query's plan in a worker; PG 16 renamed it. +SELECT CASE WHEN current_setting('server_version_num')::int >= 160000 + THEN 'debug_parallel_query' ELSE 'force_parallel_mode' END AS force \gset + +-- The workers a statement's plan launched. +CREATE FUNCTION workers_launched(statement text) RETURNS bigint LANGUAGE plpgsql AS $$ +DECLARE + plan json; +BEGIN + EXECUTE 'EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, FORMAT JSON) ' || statement INTO plan; + RETURN (SELECT coalesce(sum((m[1])::bigint), 0) + FROM regexp_matches(plan::text, '"Workers Launched": (\d+)', 'g') AS m); +END +$$; + +SELECT dblink_connect('b', :'conninfo'); + +CREATE TABLE reads (query text); +INSERT INTO reads VALUES + ($$SELECT string_agg(n, ',' ORDER BY n) FROM warren.neighbours('g', 'https://example.org/node/7'::text, 'both') n$$), + ($$SELECT warren.degree('g', 'https://example.org/node/7'::text, 'both')::text$$), + ($$SELECT warren.has_edge('g', 'https://example.org/node/7'::text, 'https://example.org/node/8'::text)::text$$), + ($$SELECT count(*)::text FROM warren.id_map('g', NULL::text) WHERE live$$), + ($$SELECT warren.shortest_path('g', 'https://example.org/node/1'::text, 'https://example.org/node/6'::text)::text$$), + ($$SELECT string_agg(b::text, ',' ORDER BY b::text) FROM warren.bfs('g', ARRAY['https://example.org/node/1'], 4) b$$), + ($$SELECT count(*)::text FROM warren.reachable_set('g', ARRAY['https://example.org/node/1'], 1000)$$), + ($$SELECT count(DISTINCT component_id)::text FROM warren.components('g', NULL::text)$$), + ($$SELECT count(*)::text FROM pairs WHERE warren.has_edge('g', src, dst)$$); +SELECT assert.equal(in_worker, in_leader, query) +FROM reads, + LATERAL dblink('b', format($$RESET %s; %s$$, :'force', query)) AS l(in_leader text), + LATERAL dblink('b', format($$SET %s = on; %s$$, :'force', query)) AS w(in_worker text); +SELECT assert.equal(launched, 1::bigint, 'each read ran in a worker: ' || query) +FROM reads, + LATERAL dblink('b', format($$SET %s = on; SELECT workers_launched(%L)$$, :'force', query)) + AS t(launched bigint); +SELECT assert.equal(in_leader, '20011', 'every pair is an edge') +FROM dblink('b', format($$RESET %s; SELECT count(*)::text FROM pairs WHERE warren.has_edge('g', src, dst)$$, :'force')) + AS t(in_leader text); + +SELECT dblink_disconnect('b'); diff --git a/test/sql/shared_cache_text.sql b/test/sql/shared_cache_text.sql new file mode 100644 index 00000000..338f0bb4 --- /dev/null +++ b/test/sql/shared_cache_text.sql @@ -0,0 +1,42 @@ +-- DESIGN.md §5.6, §5.7: a text version in the shared cache. Backends read +-- one text graph's version through the shared region (its id bytes, +-- offsets and order, mapped in place), answer as the backend that built +-- it does, map more ids than a call maps by binary search (so through the +-- call's own index), and read a refresh's new ids at their next statement. +-- Where the shared cache is off or unavailable, nothing is shared and the +-- answers are the same. +\ir include/shared_cache.sql +CREATE FUNCTION iri(n bigint) RETURNS text IMMUTABLE LANGUAGE sql RETURN 'https://example.org/node/' || n; +CREATE TABLE nodes AS SELECT iri(n) AS id FROM generate_series(1, 70001) n; +CREATE TABLE edges AS SELECT iri(n) AS src, iri(n % 70001 + 1) AS dst FROM generate_series(1, 70001) n; +SELECT warren.create_graph('t', 'SELECT id FROM nodes', 'SELECT src, dst FROM edges'); +SELECT warren.refresh('t') AS v1 \gset +SELECT CASE WHEN :'shared' THEN 1 ELSE 0 END::bigint AS one \gset +SELECT dblink_connect(session, :'conninfo') FROM unnest(ARRAY['a', 'b']) AS session; + +SELECT assert.equal(n, iri(2), 'a and b read a text id') +FROM unnest(ARRAY['a', 'b']) AS session, + LATERAL dblink(session, $$SELECT warren.neighbours('t', 'https://example.org/node/1'::text)$$) AS t(n text); +SELECT assert.equal(shared_segments(:v1), :one, 'the text version is shared'); +SELECT assert.equal(c, 70001::bigint, 'every id reads back through the shared id map') +FROM dblink('a', $$SELECT count(*) FROM warren.id_map('t', NULL::text) m JOIN nodes n ON n.id = m.node_id + WHERE m.live$$) AS t(c bigint); + +-- A vector from a query of every id: past the ids a call maps by binary +-- search, so through an index of the shared id map. +SELECT * FROM dblink('b', $$SELECT warren.vector_from_query('t', 'n', 'SELECT id, length(id)::float8 FROM nodes', + persist := false)$$) AS t(version bigint); +SELECT assert.equal(s, (SELECT sum(length(id))::float8 FROM nodes), 'each id''s value lands on its node') +FROM dblink('b', $$SELECT sum(x.value) FROM warren.vector_values('t', 'n', NULL::text) x + JOIN nodes n ON n.id = x.node_id AND length(n.id) = x.value$$) AS t(s float8); + +-- A refresh adds an id: the next statement of each backend reads it. +INSERT INTO nodes VALUES ('é'); +INSERT INTO edges VALUES (iri(1), 'é'); +SELECT warren.refresh('t') AS v2 \gset +SELECT assert.equal(ns, ARRAY[iri(2), 'é'], 'a and b read the new id') +FROM unnest(ARRAY['a', 'b']) AS session, + LATERAL dblink(session, $$SELECT array_agg(n ORDER BY n) FROM warren.neighbours('t', 'https://example.org/node/1'::text) n$$) + AS t(ns text[]); +SELECT assert.equal(shared_segments(:v2), :one, 'the refreshed text version is shared'); +SELECT dblink_disconnect(session) FROM unnest(ARRAY['a', 'b']) AS session; diff --git a/test/sql/text_ids.sql b/test/sql/text_ids.sql new file mode 100644 index 00000000..3d93b98a --- /dev/null +++ b/test/sql/text_ids.sql @@ -0,0 +1,232 @@ +-- Text node ids end to end (DESIGN.md §5.7): every function that maps ids, +-- on a text graph and on the same graph with bigint ids. The text id of +-- node n is an IRI ending in n zero-padded to six digits, so the text +-- graph's bytewise order is the bigint graph's numeric order, both give +-- each node the same dense id, and every result must be the same once its +-- ids are read back as numbers. +-- fixture: random --nodes 300 --edges 1500 --directed --seed 11 + +CREATE FUNCTION t(n bigint) RETURNS text IMMUTABLE LANGUAGE sql +RETURN 'https://example.org/node/' || lpad(n::text, 6, '0'); +CREATE FUNCTION n(id text) RETURNS bigint IMMUTABLE LANGUAGE sql RETURN right(id, 6)::bigint; +CREATE FUNCTION n(ids text[]) RETURNS bigint[] IMMUTABLE STRICT LANGUAGE sql +RETURN ARRAY(SELECT n(id) FROM unnest(ids) WITH ORDINALITY u(id, at) ORDER BY at); + +CREATE TABLE rel_edges AS +SELECT src, dst, CASE WHEN (src + dst) % 3 = 0 THEN 'b' ELSE 'a' END AS rel, + ((src * 7 + dst) % 5 + 1)::float8 AS w +FROM edges; +CREATE TABLE text_nodes AS SELECT t(id) AS id FROM nodes; +CREATE TABLE text_edges AS SELECT t(src) AS src, t(dst)::varchar AS dst, rel, w FROM rel_edges; + +SELECT warren.create_graph('gi', 'SELECT id FROM nodes', 'SELECT src, dst, rel, w FROM rel_edges', + weighted := true, has_relations := true); +SELECT warren.create_graph('gt', 'SELECT id FROM text_nodes', 'SELECT src, dst, rel, w FROM text_edges', + weighted := true, has_relations := true); +SELECT warren.create_graph('ui', 'SELECT id FROM nodes', 'SELECT src, dst, w FROM rel_edges', + directed := false, weighted := true); +SELECT warren.create_graph('ut', 'SELECT id FROM text_nodes', 'SELECT src, dst, w FROM text_edges', + directed := false, weighted := true); +SELECT warren.refresh(g) FROM unnest(ARRAY['gi', 'gt', 'ui', 'ut']) g; + +SELECT assert.equal(warren.info('gt') ->> 'id_type', 'text', 'a varchar edge column is read as text'); + +-- Whether two queries return the same rows, in any order. +CREATE FUNCTION same(a text, b text) RETURNS boolean LANGUAGE plpgsql AS $$ +DECLARE + differing bigint; + rows bigint; +BEGIN + EXECUTE format('SELECT count(*) FROM (((%1$s) EXCEPT ALL (%2$s)) UNION ALL ((%2$s) EXCEPT ALL (%1$s))) d', + a, b) INTO differing; + EXECUTE format('SELECT count(*) FROM (%s) r', a) INTO rows; + IF rows = 0 THEN + RAISE EXCEPTION 'no rows to compare: %', a; + END IF; + RETURN differing = 0; +END $$; + +-- Identity and adjacency. +SELECT assert.that(same($$SELECT node_id, dense_id, live FROM warren.id_map('gi', NULL::bigint)$$, + $$SELECT n(node_id), dense_id, live FROM warren.id_map('gt', NULL::text)$$), + 'the id maps give each node the same dense id'); +SELECT assert.that(same($$SELECT id, warren.neighbours('gi', id) FROM nodes WHERE id % 7 = 0$$, + $$SELECT id, n(warren.neighbours('gt', t(id))) FROM nodes WHERE id % 7 = 0$$), + 'neighbours'); +SELECT assert.that(same($$SELECT id, warren.degree('gi', id, 'both', ARRAY['a']) FROM nodes$$, + $$SELECT id, warren.degree('gt', t(id), 'both', ARRAY['a']) FROM nodes$$), + 'degree'); +SELECT assert.that(same($$SELECT src, dst, warren.has_edge('gi', src, dst + 1, 'b') FROM rel_edges$$, + $$SELECT src, dst, warren.has_edge('gt', t(src), t(dst + 1), 'b') FROM rel_edges$$), + 'has_edge'); + +-- Traversal. +SELECT assert.that(same($$SELECT node_id, depth, via FROM warren.bfs('gi', ARRAY[1, 2]::bigint[], 3)$$, + $$SELECT n(node_id), depth, n(via) FROM warren.bfs('gt', ARRAY[t(1), t(2)], 3)$$), + 'bfs'); +SELECT assert.that(same($$SELECT node_id, depth FROM warren.bfs('gi', ARRAY[1]::bigint[], 4, + node_filter := ARRAY[1, 3, 5, 7, 9]::bigint[])$$, + $$SELECT n(node_id), depth FROM warren.bfs('gt', ARRAY[t(1)], 4, + node_filter := ARRAY[t(1), t(3), t(5), t(7), t(9)])$$), + 'bfs with a node filter'); +SELECT assert.that(same($$SELECT warren.reachable_set('gi', ARRAY[5]::bigint[], 2, direction := 'in')$$, + $$SELECT n(warren.reachable_set('gt', ARRAY[t(5)], 2, direction := 'in'))$$), + 'reachable_set'); +SELECT assert.that(same($$SELECT id, warren.reachable('gi', 0::bigint, id, 3) FROM nodes$$, + $$SELECT id, warren.reachable('gt', t(0), t(id), 3) FROM nodes$$), + 'reachable'); +SELECT assert.that(same($$SELECT id, warren.shortest_path('gi', 0::bigint, id) FROM nodes$$, + $$SELECT id, n(warren.shortest_path('gt', t(0), t(id))) FROM nodes$$), + 'shortest_path'); +SELECT assert.that(same($$SELECT id, d.path, d.cost FROM nodes, warren.dijkstra('gi', 3::bigint, id) d$$, + $$SELECT id, n(d.path), d.cost FROM nodes, warren.dijkstra('gt', t(3), t(id)) d$$), + 'dijkstra'); +SELECT assert.that(same( + $$SELECT end_id, path, rels, count FROM warren.walk('gi', ARRAY[0, 1]::bigint[], + ARRAY[warren.s(ARRAY['a']), warren.s(NULL, 'both', ARRAY(SELECT id FROM nodes WHERE id % 2 = 0))], 'paths')$$, + $$SELECT n(end_id), n(path), rels, count FROM warren.walk('gt', ARRAY[t(0), t(1)], + ARRAY[warren.s(ARRAY['a']), warren.s(NULL, 'both', ARRAY(SELECT t(id) FROM nodes WHERE id % 2 = 0))], 'paths')$$), + 'walk with a step filter'); +SELECT assert.that(same($$SELECT source, target, weight FROM warren.minimum_spanning_forest('ui', NULL::bigint)$$, + $$SELECT n(source), n(target), weight FROM warren.minimum_spanning_forest('ut', NULL::text)$$), + 'minimum_spanning_forest'); + +-- Vectors keyed by external id. +SELECT warren.vector_from_query('gi', 'x', 'SELECT id, (id % 17)::float8 FROM nodes'); +SELECT warren.vector_from_query('gt', 'x', 'SELECT t(id), (id % 17)::float8 FROM nodes'); +SELECT warren.vector_from_query('gi', 'm', 'SELECT id, ARRAY[id % 3, id % 5]::float8[] FROM nodes'); +SELECT warren.vector_from_query('gt', 'm', 'SELECT t(id)::varchar, ARRAY[id % 3, id % 5]::float8[] FROM nodes'); +SELECT assert.that(same($$SELECT * FROM warren.vector_values('gi', 'x', NULL::bigint)$$, + $$SELECT n(node_id), value FROM warren.vector_values('gt', 'x', NULL::text)$$), + 'vector_from_query and vector_values'); +SELECT assert.that(same($$SELECT * FROM warren.matrix_rows('gi', 'm', NULL::bigint)$$, + $$SELECT n(node_id), "row" FROM warren.matrix_rows('gt', 'm', NULL::text)$$), + 'a matrix from a query with varchar ids, and matrix_rows'); +SELECT warren.pagerank(g, 'pr', seed := 'x') FROM unnest(ARRAY['gi', 'gt']) g; +SELECT assert.that(same($$SELECT * FROM warren.vector_topk('gi', 'pr', 20, NULL::bigint)$$, + $$SELECT n(node_id), value FROM warren.vector_topk('gt', 'pr', 20, NULL::text)$$), + 'pagerank with a seed vector, and vector_topk'); +SELECT warren.sssp_vector('gi', 'sssp', 4::bigint); +SELECT warren.sssp_vector('gt', 'sssp', t(4)); +SELECT assert.that(same($$SELECT * FROM warren.vector_values('gi', 'sssp', NULL::bigint)$$, + $$SELECT n(node_id), value FROM warren.vector_values('gt', 'sssp', NULL::text)$$), + 'sssp_vector'); +SELECT warren.vector_from_query(g, 'few', q) +FROM (VALUES ('gi', 'SELECT id, 1 FROM nodes WHERE id < 40'), ('gt', 'SELECT t(id), 1 FROM nodes WHERE id < 40')) v(g, q); +SELECT warren.betweenness_centrality(g, 'bc', sources := 'few') FROM unnest(ARRAY['gi', 'gt']) g; +SELECT assert.that(same($$SELECT * FROM warren.vector_values('gi', 'bc', NULL::bigint)$$, + $$SELECT n(node_id), value FROM warren.vector_values('gt', 'bc', NULL::text)$$), + 'betweenness_centrality from a vector of sources'); +SELECT assert.that(same($$SELECT * FROM warren.components('ui', NULL::bigint)$$, + $$SELECT n(node_id), component_id FROM warren.components('ut', NULL::text)$$), + 'components'); +SELECT warren.strong_components_vector(g, 'scc') FROM unnest(ARRAY['gi', 'gt']) g; +SELECT assert.that(same($$SELECT * FROM warren.vector_values('gi', 'scc', NULL::bigint)$$, + $$SELECT n(node_id), value FROM warren.vector_values('gt', 'scc', NULL::text)$$), + 'strong_components_vector'); + +-- Messages and similarity, read by external id. +CREATE TABLE msgs AS SELECT id, k, 'm' || id || '.' || k AS body FROM nodes, generate_series(1, 2) k WHERE id % 3 > 0; +SELECT assert.that(same( + $$SELECT node_id, value FROM warren.gather('gi', 'SELECT id, body FROM msgs ORDER BY k', NULL::bigint, NULL::text[])$$, + $$SELECT n(node_id), value FROM warren.gather('gt', 'SELECT t(id), body FROM msgs ORDER BY k', NULL::text, NULL::text[])$$), + 'gather'); +SELECT assert.that(same( + $$SELECT * FROM warren.similarity('gi', 'SELECT src, dst FROM rel_edges', NULL::bigint)$$, + $$SELECT n(source), n(target), score FROM warren.similarity('gt', 'SELECT src, dst FROM text_edges', NULL::text)$$), + 'similarity of pairs from a query, one column varchar'); +SELECT assert.that(same($$SELECT * FROM warren.most_similar('gi', 9::bigint, 5)$$, + $$SELECT n(node_id), score FROM warren.most_similar('gt', t(9), 5)$$), + 'most_similar'); +SELECT assert.that(same($$SELECT * FROM warren.similar_pairs('gi', NULL::bigint, 3, nodes := 'few')$$, + $$SELECT n(source), n(target), score FROM warren.similar_pairs('gt', NULL::text, 3, nodes := 'few')$$), + 'similar_pairs over a vector of nodes'); + +-- Communities: a text graph contracts into a bigint quotient. +SELECT warren.louvain(g, 'louvain') FROM unnest(ARRAY['ui', 'ut']) g; +SELECT assert.that(same($$SELECT * FROM warren.vector_values('ui', 'louvain', NULL::bigint)$$, + $$SELECT n(node_id), value FROM warren.vector_values('ut', 'louvain', NULL::text)$$), + 'louvain'); +SELECT assert.equal(warren.modularity('ut', 'louvain'), warren.modularity('ui', 'louvain'), 'modularity'); +SELECT warren.leiden(g, 'leiden', seed := 3) FROM unnest(ARRAY['ui', 'ut']) g; +SELECT assert.that(same($$SELECT * FROM warren.vector_values('ui', 'leiden', NULL::bigint)$$, + $$SELECT n(node_id), value FROM warren.vector_values('ut', 'leiden', NULL::text)$$), + 'leiden'); +SELECT warren.vector_from_query(g, 'x', q) +FROM (VALUES ('ui', 'SELECT id, id::float8 FROM nodes'), ('ut', 'SELECT t(id), id::float8 FROM nodes')) v(g, q); +SELECT warren.contract(g, 'louvain', q) FROM (VALUES ('ui', 'qi'), ('ut', 'qt')) v(g, q); +SELECT assert.equal(warren.info('qt') ->> 'id_type', 'bigint', 'the quotient of a text graph has bigint ids'); +SELECT warren.project(g, 'x', q, 'sum') FROM (VALUES ('ui', 'qi'), ('ut', 'qt')) v(g, q); +SELECT assert.that(same($$SELECT * FROM warren.vector_values('qi', 'sum', NULL::bigint)$$, + $$SELECT * FROM warren.vector_values('qt', 'sum', NULL::bigint)$$), + 'project'); +SELECT warren.lift(q, 'sum', g, 'lifted') FROM (VALUES ('ui', 'qi'), ('ut', 'qt')) v(g, q); +SELECT assert.that(same($$SELECT * FROM warren.vector_values('ui', 'lifted', NULL::bigint)$$, + $$SELECT n(node_id), value FROM warren.vector_values('ut', 'lifted', NULL::text)$$), + 'lift'); + +-- Versions: a refresh with new and dropped ids, the diff, a compaction. +SELECT current_version AS i1 FROM warren.graphs WHERE name = 'ui' \gset +SELECT current_version AS t1 FROM warren.graphs WHERE name = 'ut' \gset +DELETE FROM nodes WHERE id % 10 = 0; +INSERT INTO nodes SELECT id FROM generate_series(1000, 1010) id; +INSERT INTO rel_edges SELECT id, id - 999, 'a', 1 FROM generate_series(1000, 1010) id; +TRUNCATE text_nodes, text_edges; +INSERT INTO text_nodes SELECT t(id) FROM nodes; +INSERT INTO text_edges SELECT t(src), t(dst), rel, w FROM rel_edges; +SELECT warren.refresh('ui') AS i2 \gset +SELECT warren.refresh('ut') AS t2 \gset +SELECT assert.equal(warren.diff('ut', :t1, :t2) - 'from' - 'to', warren.diff('ui', :i1, :i2) - 'from' - 'to', 'diff'); +SELECT assert.that(same($$SELECT node_id, dense_id, live FROM warren.id_map('ui', NULL::bigint)$$, + $$SELECT n(node_id), dense_id, live FROM warren.id_map('ut', NULL::text)$$), + 'a refresh keeps dense ids and appends new ids alike'); +SELECT assert.equal((SELECT array_agg(warren.distance('ut', :t1, :t2, m) - 'from' - 'to') FROM unnest(ARRAY['edit', 'degree']) m), + (SELECT array_agg(warren.distance('ui', :i1, :i2, m) - 'from' - 'to') FROM unnest(ARRAY['edit', 'degree']) m), + 'distance between versions'); +SELECT warren.compact(g) FROM unnest(ARRAY['ui', 'ut']) g; +SELECT assert.that(same($$SELECT node_id, dense_id, live FROM warren.id_map('ui', NULL::bigint)$$, + $$SELECT n(node_id), dense_id, live FROM warren.id_map('ut', NULL::text)$$), + 'a compaction renumbers alike'); +SELECT assert.that(same($$SELECT id, warren.neighbours('ui', id) FROM nodes$$, + $$SELECT id, n(warren.neighbours('ut', t(id))) FROM nodes$$), + 'neighbours after the compaction'); + +-- Ids are bytes, compared bytewise: an empty id and ids differing only in +-- case are distinct nodes; a literal is typed as text. +CREATE TABLE odd_nodes (id text); +INSERT INTO odd_nodes VALUES (''), ('a'), ('A'), ('é'), ('a b'); +SELECT warren.create_graph('odd', 'SELECT id FROM odd_nodes', $$SELECT '', 'é' UNION ALL SELECT 'a', 'A'$$); +SELECT warren.refresh('odd'); +SELECT assert.equal((SELECT string_agg(node_id || '=' || dense_id, ',' ORDER BY dense_id) + FROM warren.id_map('odd', NULL::text)), + '=0,A=1,a=2,a b=3,é=4', 'dense ids follow bytewise order'); +SELECT assert.equal(ARRAY(SELECT warren.neighbours('odd', 'a'::text)), ARRAY['A'], 'neighbours of a text literal'); +SELECT assert.equal(ARRAY(SELECT warren.neighbours('odd', ''::text)), ARRAY['é'], 'the empty id is a node'); +SELECT assert.equal(warren.degree('odd', 'a'::varchar), 1::bigint, 'a varchar argument where no id is returned'); +SELECT assert.raises($$SELECT warren.neighbours('odd', 'a'::varchar)$$, 'a varchar argument typing returned ids', '42804'); +SELECT assert.raises($$SELECT warren.neighbours('odd', 'a')$$, 'an untyped literal names no id type', '42804'); +SELECT assert.raises($$SELECT warren.neighbours('odd', 1::bigint)$$, 'a bigint id on a text graph', '42804'); +SELECT assert.raises($$SELECT warren.id_map('odd', NULL::varchar)$$, 'a varchar witness is not text', '42804'); + +-- Refused sources. +SELECT assert.raises($$SELECT warren.create_graph('bad', 'SELECT id::bpchar FROM odd_nodes', 'SELECT NULL::text, NULL::text')$$, + 'bpchar ids', '42804'); +SELECT assert.raises($$SELECT warren.create_graph('bad', 'SELECT id::name FROM odd_nodes', 'SELECT NULL::text, NULL::text')$$, + 'name ids', '42804'); +SELECT assert.raises($$SELECT warren.create_graph('bad', 'SELECT id FROM odd_nodes', 'SELECT id, id::bigint FROM nodes')$$, + 'edges of another id type', '42804'); + +-- A nondeterministic collation would make equal ids of different bytes. +SELECT EXISTS (SELECT FROM pg_collation WHERE collprovider = 'i') AS icu \gset +\if :icu +CREATE COLLATION case_insensitive (provider = icu, locale = 'und-u-ks-level2', deterministic = false); +SELECT assert.raises($$SELECT warren.create_graph('bad', 'SELECT id COLLATE case_insensitive FROM odd_nodes', + 'SELECT NULL::text, NULL::text')$$, + 'a node id of a nondeterministic collation', '42804'); +SELECT assert.raises($$SELECT warren.create_graph('bad', 'SELECT id FROM odd_nodes', + 'SELECT id COLLATE case_insensitive, id FROM odd_nodes')$$, + 'an edge endpoint of a nondeterministic collation', '42804'); +SELECT warren.create_graph('folded', 'SELECT lower(id COLLATE case_insensitive) COLLATE "C" FROM odd_nodes', + 'SELECT NULL::text, NULL::text'); +SELECT assert.that(warren.refresh('folded') IS NOT NULL, 'a deterministic collation over folded ids builds'); +\endif diff --git a/test/sql/text_ids_bulk.sql b/test/sql/text_ids_bulk.sql new file mode 100644 index 00000000..3b0ad8aa --- /dev/null +++ b/test/sql/text_ids_bulk.sql @@ -0,0 +1,40 @@ +-- One call's text ids past 65,536 lookups (DESIGN.md §5.7): the call then +-- looks them up through a hash index charged to warren.max_compute_memory, +-- or, when that has no room for the index, by the binary search it used +-- before; either way it answers as it did below the threshold. 50,000 +-- nodes: the index takes 8 bytes a node (400 kB), a vector from a query 9. + +CREATE TABLE iris AS +SELECT n, 'https://example.org/node/' || n AS id FROM generate_series(1, 50000) n; +SELECT warren.create_graph('bulk', 'SELECT id FROM iris', + $q$SELECT a.id, b.id FROM iris a JOIN iris b ON b.n = a.n % 50000 + 1$q$); +SELECT warren.refresh('bulk'); + +-- 70,000 lookups: every node once, and 20,000 ids of no node. +CREATE VIEW bulk_rows AS +SELECT id, n::float8 AS value FROM iris +UNION ALL SELECT 'https://example.org/none/' || n, 1 FROM generate_series(1, 20000) n; + +SELECT warren.vector_from_query('bulk', 'indexed', 'SELECT id, value FROM bulk_rows'); +SELECT assert.equal(warren.vector_reduce('bulk', 'indexed', 'sum'), 50000::float8 * 50001 / 2, + 'past the threshold, through the index'); + +SET warren.max_compute_memory = '600kB'; +SELECT warren.vector_from_query('bulk', 'searched', 'SELECT id, value FROM bulk_rows'); +SELECT assert.equal(warren.vector_reduce('bulk', 'searched', 'sum'), 50000::float8 * 50001 / 2, + 'without room for the index, by the binary search'); +RESET warren.max_compute_memory; + +-- similarity's pairs: 20,000 of a node and its successor, then 25,000 +-- of unknown ids, which are not kept, so that the call passes 65,536 +-- lookups with every pair it keeps already held (512 kB, once grown). +CREATE VIEW bulk_pairs AS +SELECT a.id AS u, b.id AS v FROM iris a JOIN iris b ON b.n = a.n + 1 WHERE a.n <= 20000 +UNION ALL SELECT 'https://example.org/none/' || n, 'https://example.org/none/' || n +FROM generate_series(1, 25000) n; +SELECT assert.equal((SELECT count(*) FROM warren.similarity('bulk', 'SELECT u, v FROM bulk_pairs', NULL::text)), + 20000::bigint, 'similarity past the threshold scores every pair of nodes'); +SET warren.max_compute_memory = '800kB'; +SELECT assert.equal((SELECT count(*) FROM warren.similarity('bulk', 'SELECT u, v FROM bulk_pairs', NULL::text)), + 20000::bigint, 'and without room for the index'); +RESET warren.max_compute_memory; diff --git a/test/sql/text_ids_sql_ascii.sql b/test/sql/text_ids_sql_ascii.sql new file mode 100644 index 00000000..a14193a8 --- /dev/null +++ b/test/sql/text_ids_sql_ascii.sql @@ -0,0 +1,37 @@ +-- Text ids in a SQL_ASCII database (DESIGN.md §5.7, §14 item 42), where +-- any byte but NUL is valid text: ids that are not UTF-8 are mapped, +-- stored and returned as their bytes, unchanged. +CREATE EXTENSION IF NOT EXISTS dblink; +SELECT format('dbname=text_ids_ascii user=%s host=%s port=%s', session_user, + split_part(current_setting('unix_socket_directories'), ',', 1), + current_setting('port')) AS conninfo \gset +CREATE DATABASE text_ids_ascii ENCODING 'SQL_ASCII' TEMPLATE template0 LOCALE 'C'; +SELECT dblink_connect('ascii', :'conninfo'); +SELECT dblink_exec('ascii', $$ + CREATE EXTENSION pg_warren; + -- café in Latin-1, two bytes that are no encoding's, and an ASCII id. + CREATE TABLE ids AS SELECT * FROM (VALUES (1, E'caf\xe9'), (2, E'\xff\xfe'), (3, 'plain')) v(n, id); + CREATE TABLE links AS SELECT a.id AS src, b.id AS dst FROM ids a JOIN ids b ON b.n = a.n % 3 + 1; + DO $do$ BEGIN + PERFORM warren.create_graph('g', 'SELECT id FROM ids', 'SELECT src, dst FROM links'); + PERFORM warren.refresh('g'); + PERFORM warren.vector_from_query('g', 'n', 'SELECT id, n FROM ids'); + END $do$ +$$); + +-- Each id's bytes in hex, so that they cross the connection unchanged. +SELECT assert.equal(found, 'fffe', 'the out-neighbour of café is the id of no encoding') +FROM dblink('ascii', $$SELECT string_agg(encode(convert_to(m, 'SQL_ASCII'), 'hex'), ',') + FROM warren.neighbours('g', E'caf\xe9'::text, 'out') m$$) AS t(found text); +SELECT assert.equal(found, '636166e9,706c61696e,fffe', 'id_map returns every id''s bytes, in byte order') +FROM dblink('ascii', $$SELECT string_agg(encode(convert_to(node_id, 'SQL_ASCII'), 'hex'), ',' ORDER BY dense_id) + FROM warren.id_map('g', NULL::text)$$) AS t(found text); +SELECT assert.equal(found, '706c61696e:3,fffe:2,636166e9:1', 'vectors are read by the same ids') +FROM dblink('ascii', $$SELECT string_agg(encode(convert_to(node_id, 'SQL_ASCII'), 'hex') || ':' || value, ',' + ORDER BY value DESC) + FROM warren.vector_values('g', 'n', NULL::text)$$) AS t(found text); +SELECT assert.equal(found, '1', 'a lookup by the bytes finds the node') +FROM dblink('ascii', $$SELECT warren.degree('g', E'\xff\xfe'::text, 'in')::text$$) AS t(found text); + +SELECT dblink_disconnect('ascii'); +DROP DATABASE text_ids_ascii; diff --git a/test/sql/text_ids_toasted.sql b/test/sql/text_ids_toasted.sql new file mode 100644 index 00000000..47896c3e --- /dev/null +++ b/test/sql/text_ids_toasted.sql @@ -0,0 +1,47 @@ +-- Text ids stored out of line (DESIGN.md §5.7): a build reads ids of over +-- 2 kB from the node and the edge query, compressed and not, detoasting +-- each in the rows' memory context, and maps them as any other id. + +-- Node n's id: 3,000 bytes, compressible (EXTENDED) or not (EXTERNAL, +-- random bytes in hex). +CREATE TABLE compressed (n int, id text); +INSERT INTO compressed SELECT n, repeat(md5(n::text), 94) FROM generate_series(1, 300) n; +CREATE TABLE external (n int, id text); +ALTER TABLE external ALTER COLUMN id SET STORAGE EXTERNAL; +INSERT INTO external +SELECT n, string_agg(md5(n::text || ':' || k), '' ORDER BY k) FROM generate_series(1, 300) n, + generate_series(1, 94) k GROUP BY n; + +SELECT assert.that((SELECT bool_and(length(id) > 2048) FROM compressed) + AND (SELECT bool_and(length(id) > 2048) FROM external), 'every id is over 2 kB'); +SELECT assert.that(pg_column_compression(id) IS NOT NULL, 'compressed ids are compressed') +FROM compressed LIMIT 1; +SELECT assert.that(pg_relation_size(reltoastrelid) > 0, 'external ids are stored out of line') +FROM pg_class WHERE relname = 'external'; + +-- A path 1 → 2 → … → 300 over each table's ids. +CREATE FUNCTION path_query(ids text) RETURNS text LANGUAGE sql +RETURN format('SELECT a.id, b.id FROM %1$I a JOIN %1$I b ON b.n = a.n + 1', ids); +SELECT warren.create_graph(t, format('SELECT id FROM %I', t), path_query(t)) +FROM unnest(ARRAY['compressed', 'external']) t; +SELECT warren.refresh(t) FROM unnest(ARRAY['compressed', 'external']) t; + +SELECT assert.equal((warren.info('compressed') #>> '{versions,0,edge_count}')::bigint, 299::bigint, 'compressed: every edge'); +SELECT assert.equal((warren.info('external') #>> '{versions,0,edge_count}')::bigint, 299::bigint, 'external: every edge'); +SELECT assert.equal( + (SELECT array_agg(n ORDER BY n) FROM warren.neighbours('compressed', (SELECT id FROM compressed WHERE n = 7), 'both') x + JOIN compressed c ON c.id = x), + ARRAY[6, 8], 'compressed ids map as any other'); +SELECT assert.equal( + (SELECT array_agg(n ORDER BY n) FROM warren.neighbours('external', (SELECT id FROM external WHERE n = 7), 'both') x + JOIN external e ON e.id = x), + ARRAY[6, 8], 'external ids map as any other'); + +-- An error names a long id clipped to 256 bytes. +DO $$ +BEGIN + PERFORM warren.vector_from_query('external', 'twice', 'SELECT id, 1 FROM external UNION ALL SELECT id, 2 FROM external WHERE n = 5'); + RAISE EXCEPTION 'a node with two rows was not refused'; +EXCEPTION WHEN cardinality_violation THEN + PERFORM assert.that(length(SQLERRM) < 320 AND SQLERRM LIKE '%…', format('the id is clipped: %s bytes', octet_length(SQLERRM))); +END $$;