diff --git a/CHANGELOG.md b/CHANGELOG.md index 5561679..f701db7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -42,6 +42,27 @@ it, fails with 40001 instead of deleting the new creator's vector. `ALTER EXTENSION pg_warren UPDATE` creates the five functions. +- **Quotas on readers' vectors** (DESIGN.md §5.7, §14 items 46 and 47; + docs/semantics.md, "Quotas on readers' vectors"). A graph's owner may + bound the persisted vectors each other role stores on the graph, counted + as their elements' bytes across all versions: + `warren.set_vector_quota(name, bytes, role DEFAULT NULL)` sets the + graph's default (`info`'s new `reader_vector_quota`) or a role's own + quota, and `warren.vector_usage(name)` shows each role's vectors, bytes + and quota (a reader sees its own row). A persisted write that would take + the writer past its quota, and above what it stored, is refused with + 53400; replacing a vector with one no larger always works. Members of + the owner role are exempt, and transient vectors are not counted. One + role's writes to one graph take a transaction-scoped lock on (graph, + role), shown in `pg_locks` as `advisory` with `objsubid` 22353. + `reassign_owned` counts the vectors it moves against the new role's + quotas. `drop_owned` and the `DROP OWNED` trigger remove a role's quotas + only on graphs whose owner the caller is a member of; `orphans()` and + `drop_orphans()` handle those of dropped roles. A quota bounds a role's + vectors on other owners' graphs, not the catalog's total storage. `ALTER EXTENSION pg_warren UPDATE` adds + the column, the table `warren.vector_quotas` (dumped with the + extension), an index on `warren.vectors (graph_id, created_by)` and the + two functions; existing graphs have no quota. ### Fixed diff --git a/CLAUDE.md b/CLAUDE.md index 638a783..90f3d44 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -37,14 +37,16 @@ before changing code. Commands, toolchain setup and environment gotchas are in `version/` (diff, compaction across versions). - `pg_warren/src/`: `extension` (schema + `warren.version()`, SQL errors), `access/` (identity switches, privilege checks), `source/` - (node/edge query columns and read-only cursors), `catalog/`, + (node/edge query columns and read-only cursors), `catalog/` (and + `quota`, the lock and decision shared by vector writes and ownership + moves), `segment/` (chunked persistence), `cache/` (loader; `shared`, the shared cache across backends, and `shared::introspection`, `warren.shared_cache()`), `build/`, `version_sql/` (compact, diff), `interrupts` (the main-thread poll), `read_only` (single-row SPI reads that assign no transaction id), `parallel_worker` (what a parallel worker records and raises), `external_ids/` (id arguments ↔ dense ids), `selection/` (direction and relations), `traversal_sql/`, `vector/` (persistence and cache of named - vectors), `compute_sql/`, `spectral_sql/` (eigen, distance), + vectors; `quota`, the bound on readers' vectors), `compute_sql/`, `spectral_sql/` (eigen, distance), `community_sql/` (contract, project, lift, modularity, local moving, and the Louvain/Leiden library). Feature dependencies form a DAG: `access`, `source` → `catalog` → @@ -132,8 +134,10 @@ before changing code. Commands, toolchain setup and environment gotchas are in with `Spi::connect_mut` + `update`. Clippy's `disallowed-methods` (`clippy.toml`) enforces this; test modules allow it on the `mod` item. - **Parallel labels** (DESIGN.md §9.3): writers and anything that may persist - are `PARALLEL UNSAFE`; anything that reads a graph or vector is `PARALLEL - RESTRICTED`; only functions that read no graph are `PARALLEL SAFE`. + are `PARALLEL UNSAFE`; anything that reads a vector, or a graph in a way + §9.3 does not list as safe, is `PARALLEL RESTRICTED`; graph reads that use + no vector, catalog-only reads and functions that read no graph are + `PARALLEL SAFE`. `parallel_labels`'s test declares every function's label; add each new function there (and its read to `READS` if it is not unsafe). - **Every SQL entry point uses the guard.** The body of every diff --git a/DESIGN.md b/DESIGN.md index f554025..7b56cf6 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -169,8 +169,10 @@ CREATE TABLE warren.graphs ( retain_versions int NOT NULL DEFAULT 2 CHECK (retain_versions >= 1), -- promoted versions kept current_version bigint, -- one of this graph's versions (checked by trigger, below); NULL until first promote store_merged boolean NOT NULL DEFAULT true, -- store the merged CSR of several relations (M7, §5.3) - node_order text NOT NULL DEFAULT 'id' CHECK (node_order IN ('id', 'locality')) + node_order text NOT NULL DEFAULT 'id' CHECK (node_order IN ('id', 'locality')), -- how dense ids are ordered at the first build and at compact (0.4.0, §5.2, §14 item 22) + reader_vector_quota bigint CHECK (reader_vector_quota >= 0) + -- bytes of persisted vectors each role but the owner may store; NULL: none (0.6.0, §5.7) ); CREATE TABLE warren.versions ( @@ -220,6 +222,14 @@ CREATE TABLE warren.vector_chunks ( data bytea NOT NULL, PRIMARY KEY (vector_id, chunk_no) ); +CREATE INDEX vectors_created_by ON warren.vectors (graph_id, created_by); -- a quota's sum (0.6.0, §5.7) + +CREATE TABLE warren.vector_quotas ( -- per-role overrides of graphs.reader_vector_quota (0.6.0, §5.7) + graph_id int NOT NULL REFERENCES warren.graphs ON DELETE CASCADE, + role regrole NOT NULL, + bytes bigint NOT NULL CHECK (bytes >= 0), + PRIMARY KEY (graph_id, role) +); ``` Chunks are fixed-size (8 MiB, `CHUNK_SIZE` in `pg_warren/src/build/chunk.rs`; 1 KiB in `pg_test` builds so that small test graphs span several chunks), numbered from 0 per segment, so a multi-GB graph is many TOASTed rows. Reading a segment concatenates its chunks in order and rejects a gap. Because all of this is ordinary tables, it is WAL-logged, backed up, replicated to standbys, and — crucially for §8 — **transactional**: a version built inside a transaction that then rolls back simply disappears. @@ -231,7 +241,7 @@ The bootstrap SQL registers every `warren.*` table and sequence with `pg_extensi - *Direct DML.* A direct `DELETE FROM warren.versions` now waits on the graph's row (`FOR SHARE`), so it can deadlock with a `promote` or `drop_version` of the same graph. Lock the graph's row first (`SELECT … FROM warren.graphs WHERE graph_id = … FOR UPDATE`), or use `drop_version`. - *Locking is the triggers' own.* The update trigger reads the version `FOR KEY SHARE`, as a key's check would, so a concurrent delete waits and then fails. The delete trigger reads the graph's row `FOR SHARE`, so it sees the latest committed `current_version`, not the snapshot's. A promotion committed after a REPEATABLE READ transaction's snapshot then makes its delete fail with `serialization_failure`, and under READ COMMITTED with 23503. An earlier version read the graph from the snapshot, and a REPEATABLE READ delete removed a version promoted after the snapshot (found in the M7 security review; harness test `current_version_race`). -`label_vector` is NULL except on a contraction's versions (0.4.0, §6.6); it is protected without closing a cycle again (`bootstrap.rs` has a test that fails on any cycle): a trigger clears it when its vector is deleted. A restore needs no flags, but it does need the roles that `owner`, `readers` and `created_by` name, since `regrole` values are dumped by name (`pg_dumpall --roles-only` first, on another cluster). `pg_dump` still warns `there are circular foreign-key constraints on this table: versions`, for the self-references, and hints at `--disable-triggers`. Ignore that hint: the self-references need no order, and disabling triggers would skip the checks above. `test/dump.py` (CI, every major) fails on any other warning from the dumps or the restores. It builds graphs with promoted, compacted and staged versions and the vectors of two roles. It dumps them in plain and custom formats and restores each through `psql` and `pg_restore`, then compares `info`, `bfs` and `neighbours` on every version and every vector's values, and builds a new version and vector on the restored catalog. A non-superuser's dump is in §7. +`label_vector` is NULL except on a contraction's versions (0.4.0, §6.6); it is protected without closing a cycle again (`bootstrap.rs` has a test that fails on any cycle): a trigger clears it when its vector is deleted. A restore needs no flags, but it does need the roles that `owner`, `readers`, `created_by` and `vector_quotas.role` name, since `regrole` values are dumped by name (`pg_dumpall --roles-only` first, on another cluster). `pg_dump` still warns `there are circular foreign-key constraints on this table: versions`, for the self-references, and hints at `--disable-triggers`. Ignore that hint: the self-references need no order, and disabling triggers would skip the checks above. `test/dump.py` (CI, every major) fails on any other warning from the dumps or the restores. It builds graphs with promoted, compacted and staged versions and the vectors of two roles. It dumps them in plain and custom formats and restores each through `psql` and `pg_restore`, then compares `info`, `bfs` and `neighbours` on every version and every vector's values, and builds a new version and vector on the restored catalog. A non-superuser's dump is in §7. The bootstrap SQL also runs `REVOKE ALL … FROM PUBLIC` on every `warren.*` table and sequence. No role except the catalog owner (the role that owns them, §7) can read or write the catalog, segments or vectors directly; all access goes through `warren.*` functions. This is what makes the access check in §7 meaningful: segment and vector bytes are reachable only through functions that perform it. The schema itself grants `USAGE` to `PUBLIC`, so any role can call `warren.*` functions (each checks access itself). @@ -420,7 +430,7 @@ M9 (0.5.0) moves loaded versions into dynamic shared memory so backends share on ### 5.7 Identity, lifecycle and visibility (M10 design) -*Designed 2026-10-23 for M10 (0.6.0), after 0.5.0's release; built so far: steps 1 (the spike), 2 (`IdMap`), 3 (`TextKeys`), 5 (ownership transfer) and 6 (dropped roles). The decisions are §14 items 41–49: items 44, 45 and 46 were decided by the owner on 2026-10-23, as recommended, and the rest are decided here. Claims about Postgres were read in the PG 15–18 sources (`tcop/utility.c`, `tcop/cmdtaglist.h`, `catalog/pg_shdepend.c`, `commands/event_trigger.c`, `utils/backend_progress.h`, `utils/adt/pgstatfuncs.c`, `utils/wait_event.h`, `utils/misc/guc_tables.c`); where a claim needed code to confirm it, step 1 of the work breakdown did so on PG 15, 16 and 18 (Spike results, below), and the text reflects what it found.* M10 has five items, which share little code. `text` node ids change the id map and everything that maps external ids. Ownership transfer, cleanup after a dropped role, and quotas change the catalog's lifecycle. Observability adds a per-backend table in shared memory beside the registry of §5.6. Each part states what Postgres allows an extension on PG 15–18 and where pg_warren stops. Nothing in M10 changes a kernel, the CSR layout, the segment format of existing id types, or the registry's format, so 0.6.0 needs no restart to keep sharing on PG 17 and 18 (§5.6). +*Designed 2026-10-23 for M10 (0.6.0), after 0.5.0's release; built so far: steps 1 (the spike), 2 (`IdMap`), 3 (`TextKeys`), 5 (ownership transfer), 6 (dropped roles) and 7 (quotas). The decisions are §14 items 41–49: items 44, 45 and 46 were decided by the owner on 2026-10-23, as recommended, and the rest are decided here. Claims about Postgres were read in the PG 15–18 sources (`tcop/utility.c`, `tcop/cmdtaglist.h`, `catalog/pg_shdepend.c`, `commands/event_trigger.c`, `utils/backend_progress.h`, `utils/adt/pgstatfuncs.c`, `utils/wait_event.h`, `utils/misc/guc_tables.c`); where a claim needed code to confirm it, step 1 of the work breakdown did so on PG 15, 16 and 18 (Spike results, below), and the text reflects what it found.* M10 has five items, which share little code. `text` node ids change the id map and everything that maps external ids. Ownership transfer, cleanup after a dropped role, and quotas change the catalog's lifecycle. Observability adds a per-backend table in shared memory beside the registry of §5.6. Each part states what Postgres allows an extension on PG 15–18 and where pg_warren stops. Nothing in M10 changes a kernel, the CSR layout, the segment format of existing id types, or the registry's format, so 0.6.0 needs no restart to keep sharing on PG 17 and 18 (§5.6). #### Text node ids (§14 items 41, 42) @@ -498,8 +508,8 @@ A new fuzz target, `decode_text_id_map`, covers the three kinds. **What works, and is built.** 1. *SQL counterparts of the two per-database commands.* They work everywhere, trusted installs included, with no trigger, in each database as the commands are run: - - `warren.reassign_owned(old regrole, new regrole) RETURNS bigint` gives every graph `old` owns to `new`, with `set_owner`'s rules except its source check (as `REASSIGN OWNED` checks none; the `search_path` `NOTICE` stays). It also sets `created_by = new` on every vector `old` created; a vector's bytes and id never change, and its creator is ownership data read at each write, not cached (§5.5). It requires the privileges of both roles, as `REASSIGN OWNED` does, and returns the number of graphs moved. - - `warren.drop_owned(role regrole) RETURNS bigint` drops every graph `role` owns (as `drop_graph` does: versions, segments and everyone's vectors), deletes the persisted vectors it created on other graphs, removes it from every `readers` list, and drops its quota rows. It requires the privileges of the role, as `DROP OWNED` does, and returns the number of graphs dropped. The writes to other owners' graphs run as the catalog owner, as all catalog writes do (§7). `DROP OWNED` revokes a role's privileges on other owners' objects in the same way. Shared copies of what it deletes are evicted by the prune (§5.6), as after `drop_graph`. + - `warren.reassign_owned(old regrole, new regrole) RETURNS bigint` gives every graph `old` owns to `new`, with `set_owner`'s rules except its source check (as `REASSIGN OWNED` checks none; the `search_path` `NOTICE` stays). It also sets `created_by = new` on every vector `old` created; a vector's bytes and id never change, and its creator is ownership data read at each write, not cached (§5.5). The move counts against `new`'s quotas on readers' vectors as a write by `new` would, and is refused (53400) if it takes `new` past one (PR #59 review; "Quotas on readers' vectors" below). It requires the privileges of both roles, as `REASSIGN OWNED` does, and returns the number of graphs moved. + - `warren.drop_owned(role regrole) RETURNS bigint` drops every graph `role` owns (as `drop_graph` does: versions, segments and everyone's vectors), deletes the persisted vectors it created on other graphs, removes it from every `readers` list, and drops its quota rows on graphs whose owner the caller is a member of (every one, for a caller that may drop orphans, item 45b). It requires the privileges of the role, as `DROP OWNED` does, and returns the number of graphs dropped. The writes to other owners' graphs run as the catalog owner, as all catalog writes do (§7). `DROP OWNED` revokes a role's privileges on other owners' objects in the same way. Shared copies of what it deletes are evicted by the prune (§5.6), as after `drop_graph`. 2. *The optional event trigger* (item 44). A superuser opts in, in each database that has pg_warren, with `CREATE EVENT TRIGGER warren_drop_owned ON ddl_command_start WHEN TAG IN ('DROP OWNED') EXECUTE FUNCTION warren.on_drop_owned();`. The function is part of the extension (C, `RETURNS event_trigger`, its body under `access::entry`), declared with `#[pg_extern(sql = …)]` over a raw `fcinfo` (Spike results). Postgres does not stop a direct call, so it first checks that it was called as an event trigger (39P03, `event_trigger_protocol_violated`, otherwise, as Postgres's own event-trigger functions refuse). It runs as the role that ran `DROP OWNED`, so it reads and writes the catalog as the catalog owner, as `drop_owned` does. It reads the roles from the parse tree (a role that does not exist is skipped, and `DROP OWNED` then raises its own error) and, as item 44 decides, refuses with 2BP01 (`dependent_objects_still_exist`, as `DROP ROLE` itself refuses) while a named role owns graphs, naming them, `warren.reassign_owned` and `warren.drop_owned`; otherwise it does `drop_owned`'s other work. No extension script creates the trigger (§7), so a trusted install works without it. While it exists it depends on the extension's function, so `DROP EXTENSION` needs `CASCADE`, which drops it too; the installer can do that, since Postgres does not check ownership of what `CASCADE` drops. `pg_dump` dumps the trigger, so a non-superuser's restore fails on that one statement, which `test/dump.py` expects. 3. *A sweep for what neither saw:* a `DROP ROLE` without `DROP OWNED`, or one run before the trigger existed. `warren.orphans() RETURNS TABLE (kind text, graph text, role oid)` lists graphs whose owner no longer exists, reader entries and quota rows that name dropped roles, and vectors whose creator was dropped. `warren.drop_orphans() RETURNS bigint` drops the orphaned graphs and prunes the rest. Item 45 decides who may call it, and who may adopt an orphan with `set_owner`. @@ -511,7 +521,7 @@ A new fuzz target, `decode_text_id_map`, covers the three kinds. - *The trigger.* `on_drop_owned` reads `fcinfo->context` and refuses unless it is an `EventTriggerData` (39P03); it does nothing for a parse tree other than a `DropOwnedStmt`, so a trigger created without `WHEN TAG` does no harm. *Addition:* `DROP OWNED` checks the caller's privileges of each role only after `ddl_command_start`, so the trigger skips a role the caller lacks the privileges of, as it skips one that does not exist: `DROP OWNED` then raises its own error, and the trigger reveals no graph names to that caller and writes nothing. Each role is taken once. *The refusal reads without a lock* (a `DROP OWNED` that waited on graph rows would stall `ddl_command_start`): a `set_owner` or `reassign_owned` to the role that is uncommitted when the trigger reads, and commits after, is not seen, and the graph is orphaned at `DROP ROLE` for `orphans()` and `drop_orphans()` to find. The refusal names, in the statement's order, each role that owns graphs and its graphs, then both functions: `cannot drop objects owned by role "a" while it owns graphs "g", "h"; give them a new owner with warren.reassign_owned(old, new) or drop them with warren.drop_owned(role) first`. The function's `EXECUTE` is not revoked: Postgres checks no privilege when an event trigger fires, and a direct call is refused anyway. - *The sweep.* `orphans()` gives one `vector` row per graph and dropped creator (the signature has no vector column), sorted by kind, graph and role, and is `PARALLEL SAFE` (it reads no graph). *Privacy (conservative):* a superuser and members of the extension's owner see every row; any other caller sees the `reader` and `vector` rows of graphs it owns, and no orphaned graph, so the sweep reveals no other role's graph names. `drop_orphans()` returns the number of graphs dropped, as `drop_owned` does. - *Item 45b.* `access::may_drop_orphans` asks whether the caller could act as `pg_extension.extowner`, as `DROP EXTENSION` needs: on PG 16+ it has its privileges or may `SET ROLE` to it (`has_privs_of_role` or `member_can_set_role`; a membership granted `WITH INHERIT FALSE, SET FALSE` is not enough, PR #57 review), on PG 15 membership (`is_member_of_role`); a superuser always may; `drop_graph` of an orphan and `drop_orphans` require it (`drop_orphans` with 42501 naming the extension's owner; `drop_graph` keeps its 42704, "the owner of graph … no longer exists", for anyone else). `set_owner` on an orphan, after the lock as before it, requires a superuser (`access::require_adopter`; 42501 `only a superuser may give graph "g", whose owner no longer exists, a new owner`); the new owner's source check and `NOTICE` are as for any transfer. -- *Quotas.* `warren.vector_quotas` comes with step 7, so `clean_up_after` and `drop_orphans` each mark where its rows are deleted, and `orphans()`'s query where its `quota` rows are listed. +- *Quotas.* `warren.vector_quotas` came with step 7: `clean_up_after` deletes the role's rows (so `drop_owned` and the trigger do) on the graphs whose owner the caller is a member of (`pg_has_role(caller, owner, 'MEMBER')`), or on every graph if the caller may drop orphans (item 45b). *PR #59 review:* deleting every row let a reader shed a quota another owner set on it by running `drop_owned` on itself and writing again; a row on another owner's graph now stays, and once the role is dropped `orphans()` lists it. `orphans()` lists the rows that name dropped roles as kind `quota` (sorted before `reader`), and `drop_orphans` removes them in a fourth pass, locked as the others are. `reassign_owned` leaves `old`'s rows, as it leaves its reader entries, and counts the vectors it moves against `new`'s quotas (below). - *Tests.* `#[pg_test]`s, each seen to fail first: `drop_owned` (what it drops and keeps; the privileges, with a `NOINHERIT` member; a dropped role), `reassign_owned` (graphs and vectors moved, no source check, reader entries kept; both privileges; a role that owns nothing), the trigger through a real `CREATE EVENT TRIGGER` (a direct call; the refusal after `REASSIGN OWNED`, naming both roles' graphs, changing nothing; the cleanup after `reassign_owned`, and for a role that owns no graph; a caller without the privileges and a missing role left to `DROP OWNED`; other commands passing through), the sweep (rows, privacy, `drop_orphans`, item 45b), and `drop_graph` and `set_owner` on orphans; the refusal's message is a plain `#[test]`. `test/sql`: `drop_owned` (the standard procedure without the trigger orphans the graphs, the sweep, the functions without the trigger, then with it: item 44's refusal, `reassign_owned` then `DROP OWNED` and `DROP ROLE`, a reader's cleanup, a stranger left to `DROP OWNED`, and `DROP EXTENSION` needing `CASCADE`), and `trusted_install` (non-superusers reassign and drop a leaving role's graphs; the installer drops an orphan, may not adopt one, and cannot create the trigger; with a superuser's trigger, its `DROP EXTENSION` needs `CASCADE`). `test/dump.py` dumps and restores the trigger, and expects a trusted install's installer's restore to fail at its `CREATE EVENT TRIGGER` and `ALTER EVENT TRIGGER … OWNER TO` and nowhere else. #### Quotas on readers' vectors (§14 items 46, 47) @@ -519,12 +529,20 @@ A new fuzz target, `decode_text_id_map`, covers the three kinds. Today a reader can persist vectors without bound, in storage the catalog owner holds (§7). A quota bounds what each role other than the owner stores on a graph. - *What is counted:* the elements' bytes of every persisted vector the role wrote (`created_by`) on the graph, across all versions: `length × width × size(dtype)`. That is what the vector holds when loaded, not its TOAST footprint; it is predictable, and it is what `info` and the cache report. Transient vectors are not counted: they are memory, which `warren.max_cache_memory` bounds per backend (§6.2). Members of the owner role are exempt, since the owner pays for its own storage. - *Where it is set* (item 46, decided): `graphs.reader_vector_quota bigint` (bytes; NULL for none) is the default for every role, and `warren.vector_quotas (graph_id, role regrole, bytes)` holds per-role overrides. `warren.set_vector_quota(name, bytes bigint, role regrole DEFAULT NULL)` sets either; it is for the owner, a NULL role sets the graph's default, and NULL bytes removes the quota. `warren.vector_usage(name) RETURNS TABLE (role regrole, vectors bigint, bytes bigint, quota bigint)` shows the owner every role, and a reader its own row. `info` gains `reader_vector_quota`. The new table is registered with `pg_extension_config_dump` and revoked from `PUBLIC` (§5.1), references only `graphs`, so it closes no foreign-key cycle, and is deleted with its graph. -- *Enforcement* (item 47, decided). Before a persisted write, wherever a quota applies (every vector-writing function with `persist := true`, and the library's results): - 1. Take a transaction-scoped heavyweight lock on (graph, role). It is an advisory-type lock tag with pg_warren's own `objsubid`, so it cannot collide with applications' advisory locks (whose `objsubid` is 1 or 2), and it serialises one role's writes to one graph until they commit. It is taken on the leader: persisting writers are `PARALLEL UNSAFE` (§9.3), and a lock a parallel worker takes is released when the worker exits. +- *Enforcement* (item 47, decided). Before every persisted write by a role that is not exempt (every vector-writing function with `persist := true`, and the library's results), quota or none: + 1. Take a transaction-scoped heavyweight lock on (graph, role), and only then read the quota that applies. It is an advisory-type lock tag with pg_warren's own `objsubid`, so it cannot collide with applications' advisory locks (whose `objsubid` is 1 or 2), and it serialises one role's writes to one graph until they commit. It is taken on the leader: persisting writers are `PARALLEL UNSAFE` (§9.3), and a lock a parallel worker takes is released when the worker exits. 2. Sum the role's usage under the latest committed state (`read_only::read_one_latest`, as the prune reads, §5.6), so a REPEATABLE READ snapshot cannot miss a write committed after it. Add what the transaction itself wrote, which it sees, and subtract the vector being replaced if it is the role's own. 3. Refuse with 53400 only if usage after the write would exceed the quota *and* exceed usage before it. So replacing a vector with one no larger always works, and a quota lowered below what is in use blocks only growth. The message names the sizes and the remedies: drop vectors, use `persist := false`, or ask the owner. - An index on `warren.vectors (graph_id, created_by)` keeps the sum to one index scan. Not counted against a quota: `compact`'s move (the owner's operation, and vectors only shrink), `drop_*` and retention. + An index on `warren.vectors (graph_id, created_by)` keeps the sum to one index scan. Not counted against a quota: `compact`'s move (the owner's operation, and vectors only shrink), `drop_*` and retention. `reassign_owned`'s move *is* counted, against the role the vectors move to (below). +- *Scope.* A quota bounds what a role stores on *another owner's* graph, and only that: it is not a bound on the catalog's total storage. Any role that may create a graph stores what it likes on its own graphs, exempt, including the quotient `contract` makes for it, which it owns. A quotient does not inherit its parent's `reader_vector_quota` (or its per-role rows): its owner sets its own. A bound per database and role across graphs is item 46c, deferred (46d). +- *Implemented (step 7).* `catalog::quota`, the ledger: the decision (`admits`, a plain `#[test]`), the lock, the quota and the usage, shared by `vector::quota` and `reassign_owned` (the catalog cannot depend on `vector`); `vector::quota`: `admit` (enforcement), `set_vector_quota` and `vector_usage`; `compute_sql::Vectors::save` calls `admit` before every persisted write, so every vector-writing function and the library's results (which persist through `vector_map`) are counted. The decisions the text above left open: + - *Removing a role's quota.* `set_vector_quota(name, NULL, role)` deletes the role's row, after which the graph's default applies to it; `set_vector_quota(name, NULL)` removes the default. To lift a default for one role, give it a large quota of its own. Both quotas are `CHECK`ed non-negative (22023 from the function, 23514 from a direct write). `set_vector_quota` takes the graph's row lock and re-checks ownership after it, as every management function does (`GraphDefinition::lock`). + - *The order of a counted write.* The exemption (the caller a member of the owner role, `is_member_of_role`, as every owner check) is checked first; then the lock is taken, for every write that is not exempt; then the role's quota is read under the latest snapshot, and only if one applies is usage read, again under the latest snapshot, which also sees the transaction's own writes of earlier commands (a call writing several vectors, such as `eigen`, counts each against the next), then decided. The replaced vector is subtracted only if the writer wrote it (the sum filters on `created_by`). A refused write writes nothing: the check runs before the old vector is deleted. *PR #59 review:* the lock was first taken only where a quota applied, after reading it, so a quota set while a write under none was uncommitted let the next writer miss that write; every non-exempt write now locks first. *The error when a replacement races:* the check runs before the replacement's own (the delete by id and creator, PR #57 review), so a write whose replaced vector was moved to another role meanwhile counts it as not its own and may be refused with 53400 where it would otherwise fail with 40001. That is conservative, never a bypass: either way nothing is written. + - *`reassign_owned` (PR #59 review).* Moving `created_by` from `old` to `new` was uncounted, so a reader under a quota could `reassign_owned` its vectors to a group it is a member of, which reset its own usage to 0, and write again, without bound. Now, after the `UPDATE` that moves the vectors (returning their bytes per graph), for each graph where neither `new` nor the caller is a member of the owner (after the graphs moved) and the caller may not drop orphans: take the (graph, `new`) lock, read `new`'s quota, and if one applies read its usage under the latest snapshot, which sees the move, and refuse (53400, `CatalogError::QuotaExceeded`, naming both roles, the graph, the bytes moved and the quota) if the move takes `new` past its quota and above what it stored. A refusal rolls back the whole call. Taking the lock after the `UPDATE` is safe: the decision reads usage only once the lock is held. + - *The lock.* `LockAcquire` on an advisory tag (`MyDatabaseId`, graph id, role oid, `objsubid` 0x5751 = 22353) in `ExclusiveLock`, transaction-scoped, as `pg_advisory_xact_lock` takes its locks. It waits as heavyweight locks do (deadlock detection, `lock_timeout`, cancel). + - *`vector_usage`.* Its rows are the roles that have persisted vectors or a quota row on the graph, and, for a reader, only its own (listed even with no vectors); `quota` is the one that applies, NULL for none and for members of the owner role (`pg_has_role(role, owner, 'MEMBER')`). It is `PARALLEL SAFE`, like `info`: it reads catalog rows and no vector. + - *Tests.* Plain `#[test]`s of the decision; `#[pg_test]`s of refusal (its message), the transaction's own writes and one call's earlier writes counted, replacement no larger, a vector counting against its writer, a quota lowered below use, the owner's exemption (a member included), overrides and their removal, `vector_usage`'s privacy, `set_vector_quota`'s checks, the lock in `pg_locks`, and the bytes per dtype; `access::guard`'s test runs eve's writes under a quota. `test/sql/vector_quota_race.sql`: a second session of one role waits on the first's lock (`blocked_by_me.sql`) and is refused once the first commits, a REPEATABLE READ writer whose snapshot predates another session's committed write is refused, and both succeed once the quota allows; *PR #59 review,* each seen to fail first: a writer behind one that wrote under no quota waits and counts that write against a quota committed meanwhile, and a `reassign_owned` behind an uncommitted write of the role it moves vectors to waits and is refused. `#[pg_test]`s, also seen to fail first: `reassign_owned` refused past the new role's quota (and allowed to a member of the owner, and when the new role is exempt), a role's `drop_owned` of itself and its `DROP OWNED` under the trigger keeping its quota on another owner's graph while the graph's owner's `drop_owned` removes it, and `set_vector_quota` refusing `'-'` (22023) and a role oid that names no role (42704) for a quota. `test/dump.py` dumps and restores a default and a per-role quota (and `vector_usage`); `test/upgrade.py` checks that the update leaves every graph with no default quota and that a quota set on an upgraded graph refuses a reader's write. - *Rejected:* a running counter in the quota row, because every deletion path (`drop_vector`, replacement, the cascades from `drop_version`, retention and `drop_graph`, `compact`'s move) would have to keep it, and cascades run no pg_warren code. Also rejected: a `SUSET` setting per role. A setting follows the session, not `current_user`, so `SET ROLE` would escape it, and hosted installs have no superuser to set it. #### Observability (§14 items 48, 49) @@ -659,7 +677,7 @@ Each step is a pull request that leaves `main` green. 4. *`pg_warren`: text graphs end to end:* `IdType::Text`; the collation, type and encoding checks; the build's intake; the region's arrays and the loader; `external_ids`; every SQL function. Also the `text_ids` tests, and the soak at 10M nodes with synthetic IRIs (`bench/soak.py --ids text`), recorded in §12. 5. *Ownership transfer:* `set_owner`, ownership re-checked after the lock, and `set_owner_race`. **Done** (see "Ownership transfer", *Implemented*). 6. *Dropped roles:* `drop_owned`, `reassign_owned`, `orphans`, `drop_orphans` and `on_drop_owned`, as items 44 and 45 decide; `docs/semantics.md`'s procedure; the trusted-install and dump tests. **Done** (see "When an owner role is dropped", *Implemented*). -7. *Quotas:* the schema (`graphs.reader_vector_quota`, `warren.vector_quotas`, the index), `set_vector_quota`, `vector_usage` and enforcement, as item 46 decides; the race tests. +7. *Quotas:* the schema (`graphs.reader_vector_quota`, `warren.vector_quotas`, the index), `set_vector_quota`, `vector_usage` and enforcement, as item 46 decides; the race tests. **Done** (see "Quotas on readers' vectors", *Implemented*). 8. *The activity table and the cache counters:* `warren.cache()`, `warren.backends()` and the wait event. 9. *Progress:* the `warren-core` port, reporting from every call in the table above, `warren.progress()` and `warren.report_progress()`, and the library calling it; the cost measured. 10. *Release 0.6.0:* the install script, `sql/pg_warren--0.5.0--0.6.0.sql`, `sql/released.sha256`, and `test/upgrade.py` from every release. @@ -1301,7 +1319,7 @@ Decisions made in M3 (`pg_warren::traversal_sql`, the SQL binding): ### 9.3 Function reference -All take `graph text` first, external IDs for nodes, and `version bigint DEFAULT NULL`. All are `STABLE`. **Parallel labels (decided: M3, PR #4 review; M5 fix; M9, §14 item 33):** every function that writes or may persist what it computes (`create_graph`, `refresh`, `promote`, `compact`, `drop_*`, `grant`, `revoke`, the vector constructors and kernels — `vector_const`, `vector_from_query`, `vector_degree`, `vector_dense_id`, `vector_map`, `spmv`, `spmm`, `gather`, `triangles` — and the library functions) is `PARALLEL UNSAFE`. The graph reads that use no vector are `PARALLEL SAFE` (M9, §14 item 33, §5.6 step 9): `neighbours`, `degree`, `has_edge`, `id_map`, `info` and the traversals `reachable`, `shortest_path`, `dijkstra`, `bfs`, `reachable_set`, `components` and `walk`. A parallel worker maps the shared version, or, wherever the version is read privately (the shared cache unavailable or off, admission refusing the version, a version the current transaction created: §5.6, "Parallel query"), loads its own copy within `warren.max_cache_memory` and `warren.max_compute_memory`, so a parallel plan can hold one per worker plus the leader's; a traversal it runs records nothing, so `last_stats()` reports only the leader's calls; and it raises none of the cache's `NOTICE`s, which the leader raises where it reads the version itself. Every other function that reads a graph or a vector is `PARALLEL RESTRICTED`: `diff`, `distance`, `prewarm`, `evict`, `last_stats`, the vector reads `vector_values`, `matrix_rows`, `vector_reduce`, `vector_norm_diff`, `vector_stats`, `vector_topk`, `modularity`, `explain_map`, and the library's set-returning functions `minimum_spanning_forest`, `similarity`, `most_similar` and `similar_pairs`: a worker cannot see transient vectors (§6.2), and `prewarm`, `evict` and `last_stats` are about the leader's session. `version`, `s`, `steps` and `shared_cache` (M9), which read no graph, are `PARALLEL SAFE`. A restricted function still runs in the leader while a parallel plan executes, which is parallel mode, so the read path assigns no transaction id and writes nothing: its single-row catalog reads go through `read_only::read_one` (a `Spi::connect` select, which never calls pgrx's `mark_mutable` and so never assigns a transaction id; it makes no promise about the snapshot, which is SPI's as for any read), never pgrx's `Spi::get_one*`, which runs the query through `SpiClient::update` and so calls `GetCurrentTransactionId()` ("cannot assign XIDs during a parallel operation"). `parallel_labels`'s tests check every function's label against a declared table. Under `debug_parallel_query = on`, they call every non-unsafe function once (and read a transient vector), with the arguments of its `READS` entry (one call per function, `distance` one per method it tests), both as `PERFORM` in a subtransaction with no transaction id and as an SPI `SELECT` in a transaction that holds one, and check that it raises no error. Both run without a row limit, under which (as with pgrx's `Spi::get_one`, a cursor's fetch, or any caller that passes a count) Postgres runs a plan without parallel mode or workers: there a `RESTRICTED` call runs in the leader in parallel mode and a `SAFE` one in a worker, but these calls do not observe which; other arguments and code paths of the same function are not covered. One call (`vector_reduce`) runs in the InitPlan of a cost-induced parallel scan. Separately, every `PARALLEL SAFE` graph read is shown to run in a worker (`Workers Launched: 1`) and return the leader's rows, a parallel scan calls `has_edge` and `degree` per row, and a worker's traversal leaves `last_stats()` alone. Clippy's `disallowed-methods` refuses pgrx's `Spi::get_*`, `run*` and `explain*` outside tests and the writer paths that allow them explicitly. +All take `graph text` first, external IDs for nodes, and `version bigint DEFAULT NULL`. All are `STABLE`. **Parallel labels (decided: M3, PR #4 review; M5 fix; M9, §14 item 33):** every function that writes or may persist what it computes (`create_graph`, `refresh`, `promote`, `compact`, `drop_*`, `grant`, `revoke`, the vector constructors and kernels — `vector_const`, `vector_from_query`, `vector_degree`, `vector_dense_id`, `vector_map`, `spmv`, `spmm`, `gather`, `triangles` — and the library functions) is `PARALLEL UNSAFE`. The graph reads that use no vector are `PARALLEL SAFE` (M9, §14 item 33, §5.6 step 9): `neighbours`, `degree`, `has_edge`, `id_map`, `info` and the traversals `reachable`, `shortest_path`, `dijkstra`, `bfs`, `reachable_set`, `components` and `walk`. A parallel worker maps the shared version, or, wherever the version is read privately (the shared cache unavailable or off, admission refusing the version, a version the current transaction created: §5.6, "Parallel query"), loads its own copy within `warren.max_cache_memory` and `warren.max_compute_memory`, so a parallel plan can hold one per worker plus the leader's; a traversal it runs records nothing, so `last_stats()` reports only the leader's calls; and it raises none of the cache's `NOTICE`s, which the leader raises where it reads the version itself. Every other function that reads a graph or a vector is `PARALLEL RESTRICTED`: `diff`, `distance`, `prewarm`, `evict`, `last_stats`, the vector reads `vector_values`, `matrix_rows`, `vector_reduce`, `vector_norm_diff`, `vector_stats`, `vector_topk`, `modularity`, `explain_map`, and the library's set-returning functions `minimum_spanning_forest`, `similarity`, `most_similar` and `similar_pairs`: a worker cannot see transient vectors (§6.2), and `prewarm`, `evict` and `last_stats` are about the leader's session. `version`, `s`, `steps` and `shared_cache` (M9), which read no graph, are `PARALLEL SAFE`, as are `orphans` and `vector_usage` (M10), which read only catalog rows, as `info` does; `set_vector_quota` writes, so it is `PARALLEL UNSAFE`. A restricted function still runs in the leader while a parallel plan executes, which is parallel mode, so the read path assigns no transaction id and writes nothing: its single-row catalog reads go through `read_only::read_one` (a `Spi::connect` select, which never calls pgrx's `mark_mutable` and so never assigns a transaction id; it makes no promise about the snapshot, which is SPI's as for any read), never pgrx's `Spi::get_one*`, which runs the query through `SpiClient::update` and so calls `GetCurrentTransactionId()` ("cannot assign XIDs during a parallel operation"). `parallel_labels`'s tests check every function's label against a declared table. Under `debug_parallel_query = on`, they call every non-unsafe function once (and read a transient vector), with the arguments of its `READS` entry (one call per function, `distance` one per method it tests), both as `PERFORM` in a subtransaction with no transaction id and as an SPI `SELECT` in a transaction that holds one, and check that it raises no error. Both run without a row limit, under which (as with pgrx's `Spi::get_one`, a cursor's fetch, or any caller that passes a count) Postgres runs a plan without parallel mode or workers: there a `RESTRICTED` call runs in the leader in parallel mode and a `SAFE` one in a worker, but these calls do not observe which; other arguments and code paths of the same function are not covered. One call (`vector_reduce`) runs in the InitPlan of a cost-induced parallel scan. Separately, every `PARALLEL SAFE` graph read is shown to run in a worker (`Workers Launched: 1`) and return the leader's rows, a parallel scan calls `has_edge` and `degree` per row, and a worker's traversal leaves `last_stats()` alone. Clippy's `disallowed-methods` refuses pgrx's `Spi::get_*`, `run*` and `explain*` outside tests and the writer paths that allow them explicitly. ```sql warren.bfs(graph text, start anyarray, max_depth int, @@ -1341,10 +1359,13 @@ warren.set_owner(name, owner regrole) RETURNS void; -- §5.7, §14 item 43 (0. warren.reassign_owned(old regrole, new regrole) RETURNS bigint; warren.drop_owned(role regrole) RETURNS bigint; -- §5.7 (0.6.0) warren.orphans() RETURNS TABLE (kind text, graph text, role oid); warren.drop_orphans() RETURNS bigint; -- §5.7, §14 item 45 (0.6.0) warren.on_drop_owned() RETURNS event_trigger; -- §5.7, §14 item 44: for a superuser's CREATE EVENT TRIGGER (0.6.0) +warren.set_vector_quota(name, bytes bigint, role regrole DEFAULT NULL) RETURNS void; -- §5.7, §14 item 46 (0.6.0) +warren.vector_usage(name) RETURNS TABLE (role regrole, vectors bigint, bytes bigint, quota bigint); -- §5.7 (0.6.0) -- promote, drop_version, drop_graph RETURN void; promote applies retention (§7) warren.info(name) RETURNS jsonb; -- current version, stats, staleness (now() - built_at), bytes, versions list; -- owners and readers may call it. Keys: name, owner, readers, id_type, search_path, the flags, - -- retain_versions, store_merged, node_order, current_version, staleness, bytes, versions[{version, seq, staged, + -- retain_versions, store_merged, node_order, current_version, reader_vector_quota (0.6.0), + -- staleness, bytes, versions[{version, seq, staged, -- current, label, built_at, built_lsn, staleness, node_count, id_space, edge_count, -- dropped_edges, deduplicated_edges, target_width, bytes, compacted_from, vectors}] warren.last_stats() RETURNS jsonb; -- nodes visited, edges traversed, path taken, for the session's last call (Appendix A.6) @@ -1784,7 +1805,7 @@ Renames are cheap before 1.0 and costly after. Also a written stability and depr - *Locking and caches:* it takes the graph's row lock, which now re-reads `owner`, and every management function re-checks ownership after the lock. Nothing is evicted, since no cache or shared key carries the owner. 44. **What does `DROP OWNED` do to a role's graphs when pg_warren's event trigger is installed?** (Decided 2026-10-23 by the owner, as recommended; M10, §5.7.) Postgres's documented way to drop a role that owns objects is `REASSIGN OWNED BY old TO new; DROP OWNED BY old;` in each database, then `DROP ROLE`. Of these, only `DROP OWNED` fires event triggers. `DROP ROLE` is a global object, and `REASSIGN OWNED`, though per database, is dispatched as a global-object command, so pg_warren cannot see it. So `REASSIGN OWNED` never moves a graph, and whatever the trigger does at `DROP OWNED` happens to graphs the administrator may have believed reassigned. M10 also adds `warren.reassign_owned(old, new)` and `warren.drop_owned(role)`, which work with or without the trigger. Options: - - **a. Refuse.** The trigger fails the `DROP OWNED` (2BP01, `dependent_objects_still_exist`, as `DROP ROLE` refuses) while a role it names owns graphs, naming the graphs and the two functions. Where the role owns none, it removes the role from readers lists, drops its quota rows and deletes the vectors it created on others' graphs, as `DROP OWNED` revokes privileges and drops owned objects. An administrator following the standard procedure is stopped, not surprised. Cost: one extra call for whoever really meant to drop the graphs. + - **a. Refuse.** The trigger fails the `DROP OWNED` (2BP01, `dependent_objects_still_exist`, as `DROP ROLE` refuses) while a role it names owns graphs, naming the graphs and the two functions. Where the role owns none, it removes the role from readers lists, drops its quota rows (on graphs whose owner the caller is a member of, as `drop_owned` does: PR #59 review) and deletes the vectors it created on others' graphs, as `DROP OWNED` revokes privileges and drops owned objects. An administrator following the standard procedure is stopped, not surprised. Cost: one extra call for whoever really meant to drop the graphs. - **b. Drop.** The trigger drops the role's graphs, as `DROP OWNED` drops everything else the role owns. This is consistent with Postgres, but the standard procedure then drops graphs the administrator meant to reassign, since `REASSIGN OWNED` never moved them. Their versions, vectors and history are lost silently; only the source tables remain to rebuild from. - **c. No trigger at all**, only the functions and the sweep. Nothing in the standard procedure touches graphs: they are orphaned at `DROP ROLE` (and exposed to OID reuse) until someone runs `warren.drop_orphans()`. @@ -1804,10 +1825,10 @@ Renames are cheap before 1.0 and costly after. Also a written stability and depr - **c. Per database and role, across graphs, set by an administrator** (a superuser, or whoever item 45 trusts). This protects the disk from a reader of many graphs. But a graph's owner cannot shape their own readers' room, and it is of no use where no administrator exists. - **d. a in M10, and c later** if M11's deployments ask for it. - **Recommendation: d.** **Decided: d** (the owner, 2026-10-23). + **Recommendation: d.** **Decided: d** (the owner, 2026-10-23). *Notes (PR #59 review):* under a, a quota bounds a role's vectors on another owner's graph only. It is not a bound on total catalog storage: `create_graph`, and the quotient `contract` makes and the caller owns, give any role storage it is exempt on, which c would bound (deferred). A quotient does not inherit its parent's `reader_vector_quota`. `reassign_owned`'s move counts against the new role's quota, and `drop_owned` removes a role's quotas only on graphs whose owner the caller is a member of. 47. **How is a vector quota enforced?** (M10 design; §5.7.) **Decided:** - - *Lock:* a persisted write that a quota applies to first takes a transaction-scoped lock on (graph, role), on the leader: an advisory-type lock tag with pg_warren's own `objsubid`, so it cannot collide with applications' advisory locks. + - *Lock:* every persisted write by a role that is not exempt first takes a transaction-scoped lock on (graph, role), on the leader, before it reads the quota (PR #59 review), as does `reassign_owned` on (graph, new role) for the vectors it moves: an advisory-type lock tag with pg_warren's own `objsubid`, so it cannot collide with applications' advisory locks. - *Usage:* it then sums the role's usage under the latest committed state, as the prune reads it, plus its own transaction's writes, less the vector it replaces. - *Refusal:* 53400, only if usage would rise past the quota. Replacing a vector with one no larger always works. - *Index:* `warren.vectors (graph_id, created_by)`. diff --git a/README.md b/README.md index 2e29aaa..0fdd772 100644 --- a/README.md +++ b/README.md @@ -290,10 +290,10 @@ See [CHANGELOG.md](CHANGELOG.md). This is pre-1.0 software: the SQL surface may still change between minor versions, with an upgrade script each time. Not there yet: the rest of the road to 1.0 (DESIGN.md §13) is text ids, -quotas and observability (M10; ownership transfer, `warren.set_owner`, and -cleanup after a dropped role, `warren.reassign_owned`, `warren.drop_owned`, -the optional `DROP OWNED` trigger and `warren.drop_orphans`, are built for -0.6.0); +and observability (M10; ownership transfer, `warren.set_owner`, cleanup +after a dropped role, `warren.reassign_owned`, `warren.drop_owned`, the +optional `DROP OWNED` trigger and `warren.drop_orphans`, and quotas on +readers' vectors, `warren.set_vector_quota`, are built for 0.6.0); benchmarks and robustness on real graphs and deployments, and packages (M11); and a reviewed, frozen SQL surface (M12, 1.0.0). Incremental maintenance, general sparse matrix algebra (contraction's `PᵀAP` is the diff --git a/crates/pg_warren/src/access/guard.rs b/crates/pg_warren/src/access/guard.rs index 952a179..91f2a61 100644 --- a/crates/pg_warren/src/access/guard.rs +++ b/crates/pg_warren/src/access/guard.rs @@ -300,6 +300,7 @@ mod tests { ("revoke", "warren.revoke('victim', 'eve')"), ("s", "warren.s(NULL)"), ("set_owner", "warren.set_owner('victim', 'eve')"), + ("set_vector_quota", "warren.set_vector_quota('victim', 1000000, 'eve')"), ("shared_cache", "warren.shared_cache()"), ("shortest_path", "warren.shortest_path('victim', 1::bigint, 2::bigint)"), ("similar_pairs", "warren.similar_pairs('victim', NULL::bigint)"), @@ -320,6 +321,7 @@ mod tests { ("vector_reduce", "warren.vector_reduce('victim', 'c', 'sum')"), ("vector_stats", "warren.vector_stats('victim', 'c')"), ("vector_topk", "warren.vector_topk('victim', 'c', 3, NULL::bigint)"), + ("vector_usage", "warren.vector_usage('victim')"), ("vector_values", "warren.vector_values('victim', 'c', NULL::bigint)"), ("version", "warren.version()"), ("walk", "warren.walk('victim', ARRAY[1::bigint], ARRAY[warren.s(NULL)])"), @@ -369,10 +371,12 @@ mod tests { // Label vector `c` and quotient `victim_q`, which eve reads, so // that contract, project and lift run all the way as eve; and // `eve_minion`, whose privileges eve has, so that drop_owned and - // reassign_owned run all the way too (it owns nothing). + // reassign_owned run all the way too (it owns nothing); and a + // quota on readers' vectors, so that eve's writes are counted. Spi::run( "CREATE ROLE eve_minion; GRANT eve_minion TO eve; SELECT warren.grant('victim', 'eve'); + SELECT warren.set_vector_quota('victim', 1000000); SELECT warren.vector_const('victim', 'c', 1); SELECT warren.contract('victim', 'c', 'victim_q'); SELECT warren.vector_const('victim_q', 'c', 1); diff --git a/crates/pg_warren/src/catalog/bootstrap.rs b/crates/pg_warren/src/catalog/bootstrap.rs index 6dcb3db..ddddbe0 100644 --- a/crates/pg_warren/src/catalog/bootstrap.rs +++ b/crates/pg_warren/src/catalog/bootstrap.rs @@ -1,6 +1,7 @@ //! The catalog tables, created by the extension script (DESIGN.md §5.1): -//! `warren.graphs`, `versions`, `segments`, `vectors` and `vector_chunks`, -//! revoked from `PUBLIC` and registered with `pg_extension_config_dump`. +//! `warren.graphs`, `versions`, `segments`, `vectors`, `vector_chunks` +//! and `vector_quotas`, revoked from `PUBLIC` and registered with +//! `pg_extension_config_dump`. // Emitted at the end of the script (`finalize`), after `CREATE SCHEMA warren`. // Functions reach these tables only at run time, through SPI. @@ -25,7 +26,8 @@ CREATE TABLE warren.graphs ( current_version bigint, store_merged boolean NOT NULL DEFAULT true, node_order text NOT NULL DEFAULT 'id' - CHECK (node_order IN ('id'::pg_catalog.text, 'locality'::pg_catalog.text)) + CHECK (node_order IN ('id'::pg_catalog.text, 'locality'::pg_catalog.text)), + reader_vector_quota bigint CHECK (reader_vector_quota >= 0) ); CREATE TABLE warren.versions ( @@ -79,6 +81,17 @@ CREATE TABLE warren.vector_chunks ( ); -- IEEE floats rarely compress, so skip pglz's attempt on every write. ALTER TABLE warren.vector_chunks ALTER COLUMN data SET STORAGE EXTERNAL; +-- A quota sums one role's vectors on one graph (DESIGN.md §5.7). +CREATE INDEX vectors_created_by ON warren.vectors (graph_id, created_by); + +-- Per-role quotas on readers' persisted vectors, overriding +-- graphs.reader_vector_quota (DESIGN.md §5.7, §14 item 46). +CREATE TABLE warren.vector_quotas ( + graph_id int NOT NULL REFERENCES warren.graphs ON DELETE CASCADE, + role regrole NOT NULL, + bytes bigint NOT NULL CHECK (bytes >= 0), + PRIMARY KEY (graph_id, role) +); -- graphs.current_version is not a foreign key: with versions.graph_id it -- would make a cycle, and pg_dump cannot order the data of tables in a @@ -164,6 +177,7 @@ SELECT pg_catalog.pg_extension_config_dump('warren.segments', ''); SELECT pg_catalog.pg_extension_config_dump('warren.vectors', ''); SELECT pg_catalog.pg_extension_config_dump('warren.vectors_vector_id_seq', ''); SELECT pg_catalog.pg_extension_config_dump('warren.vector_chunks', ''); +SELECT pg_catalog.pg_extension_config_dump('warren.vector_quotas', ''); "#, name = "catalog", finalize, @@ -231,6 +245,37 @@ mod tests { Ok(()) } + /// A quota sums one role's vectors on one graph (DESIGN.md §5.7). + #[pg_test] + fn vectors_are_indexed_by_graph_and_creator() -> Result<(), Error> { + let index = Spi::get_one::( + "SELECT pg_get_indexdef('warren.vectors_created_by'::regclass)", + )?; + assert_eq!( + index.as_deref(), + Some("CREATE INDEX vectors_created_by ON warren.vectors USING btree (graph_id, created_by)") + ); + Ok(()) + } + + #[pg_test] + fn quotas_are_bytes_not_negative() -> Result<(), Error> { + ring("quota_check", 3, 2)?; + assert_eq!( + sqlstate("UPDATE warren.graphs SET reader_vector_quota = -1")?.as_deref(), + Some("23514") + ); + assert_eq!( + sqlstate( + "INSERT INTO warren.vector_quotas SELECT graph_id, 'quota_check_owner', -1 + FROM warren.graphs" + )? + .as_deref(), + Some("23514") + ); + Ok(()) + } + #[pg_test] fn versions_are_indexed_by_parent_and_label_vector() -> Result<(), Error> { let indexes = Spi::get_one::( @@ -293,7 +338,7 @@ mod tests { build_reverse boolean not null, weighted boolean not null, \ has_relations boolean not null, retain_versions integer not null, \ current_version bigint, store_merged boolean not null, \ - node_order text not null" + node_order text not null, reader_vector_quota bigint" ) ); Ok(()) @@ -338,6 +383,10 @@ mod tests { columns("vector_chunks")?.as_deref(), Some("vector_id bigint not null, chunk_no integer not null, data bytea not null") ); + assert_eq!( + columns("vector_quotas")?.as_deref(), + Some("graph_id integer not null, role regrole not null, bytes bigint not null") + ); Ok(()) } @@ -374,7 +423,8 @@ mod tests { dumped.as_deref(), Some( "warren.graphs, warren.graphs_graph_id_seq, warren.segments, \ - warren.vector_chunks, warren.vectors, warren.vectors_vector_id_seq, \ + warren.vector_chunks, warren.vector_quotas, warren.vectors, \ + warren.vectors_vector_id_seq, \ warren.versions, warren.versions_version_id_seq" ) ); diff --git a/crates/pg_warren/src/catalog/dropped_roles/drop_owned.rs b/crates/pg_warren/src/catalog/dropped_roles/drop_owned.rs index d746c05..a4ad849 100644 --- a/crates/pg_warren/src/catalog/dropped_roles/drop_owned.rs +++ b/crates/pg_warren/src/catalog/dropped_roles/drop_owned.rs @@ -4,14 +4,15 @@ use pgrx::prelude::*; use super::super::CatalogError; -use super::clean_up_after; +use super::{clean_up_after, Cleaner}; use crate::access::{as_catalog_owner, entry, refuse_bootstrap_superuser, require_privileges_of}; /// `warren.drop_owned(role regrole)`: drops every graph `role` owns (with /// its versions and everyone's vectors, as `drop_graph` does), deletes the /// vectors it wrote on other graphs and removes it from every graph's -/// readers. The caller needs the privileges of `role`. Returns the number -/// of graphs dropped. +/// readers, and its quotas on readers' vectors on graphs whose owner the +/// caller is a member of. The caller needs the privileges of `role`. +/// Returns the number of graphs dropped. #[pg_extern( schema = "warren", sql = r#" @@ -26,6 +27,7 @@ fn drop_owned(role: pg_sys::Oid) -> i64 { fn drop_everything_of(role: pg_sys::Oid) -> Result { require_privileges_of(role, "drop its graphs and vectors")?; refuse_bootstrap_superuser(role, "drop")?; + let by = Cleaner::current()?; let dropped = as_catalog_owner(|| { Spi::connect_mut(|client| { // Deleting a graph's row takes its lock, waiting for a running @@ -38,7 +40,7 @@ fn drop_everything_of(role: pg_sys::Oid) -> Result { &[role.into()], )? .len(); - clean_up_after(client, role)?; + clean_up_after(client, role, by)?; Ok::<_, CatalogError>(dropped) }) })?; @@ -52,14 +54,15 @@ mod tests { use pgrx::prelude::*; use pgrx::spi::Error; - use crate::fixture::{failure, failure_as, neighbouring_graphs, ownership}; + use crate::fixture::{failure, failure_as, neighbouring_graphs, ownership, quotas}; #[pg_test] - fn drop_owned_drops_the_roles_graphs_its_vectors_and_its_reader_entries() -> Result<(), Error> { + fn drop_owned_drops_the_roles_graphs_vectors_reader_entries_and_quotas() -> Result<(), Error> { neighbouring_graphs()?; let dropped = Spi::get_one::("SELECT warren.drop_owned('dr_owner')")?; assert_eq!(dropped, Some(1)); assert_eq!(ownership()?.as_deref(), Some("dk:dk_owner:{} / dk.theirs:dk_owner")); + assert_eq!(quotas()?.as_deref(), Some("")); Ok(()) } @@ -116,4 +119,27 @@ mod tests { ); Ok(()) } + + /// The security review of PR #59: a reader that drops what it owns + /// keeps the quota another graph's owner set on it; the sweep removes + /// it once the role is dropped. + #[pg_test] + fn a_role_that_drops_what_it_owns_keeps_its_quotas_on_others_graphs() -> Result<(), Error> { + neighbouring_graphs()?; + assert_eq!(failure_as("dr_owner", "warren.drop_owned('dr_owner')")?, None); + assert_eq!(quotas()?.as_deref(), Some("dk:dr_owner:4096")); + assert_eq!(ownership()?.as_deref(), Some("dk:dk_owner:{} / dk.theirs:dk_owner")); + Ok(()) + } + + /// The owner of the graph a quota is on may remove it with the role's + /// other entries. + #[pg_test] + fn the_graphs_owner_removes_a_roles_quota_with_drop_owned() -> Result<(), Error> { + neighbouring_graphs()?; + Spi::run("GRANT dr_owner TO dk_owner")?; + assert_eq!(failure_as("dk_owner", "warren.drop_owned('dr_owner')")?, None); + assert_eq!(quotas()?.as_deref(), Some("")); + Ok(()) + } } diff --git a/crates/pg_warren/src/catalog/dropped_roles/event_trigger.rs b/crates/pg_warren/src/catalog/dropped_roles/event_trigger.rs index aa34bd9..be399ae 100644 --- a/crates/pg_warren/src/catalog/dropped_roles/event_trigger.rs +++ b/crates/pg_warren/src/catalog/dropped_roles/event_trigger.rs @@ -7,7 +7,7 @@ use pgrx::prelude::*; use pgrx::PgList; use super::super::CatalogError; -use super::clean_up_after; +use super::{clean_up_after, Cleaner}; use crate::access::{as_catalog_owner, entry, has_privileges_of, role_name}; use crate::read_only::read_one; @@ -77,10 +77,11 @@ fn refuse_or_clean_up(roles: &[pg_sys::Oid]) -> Result<(), CatalogError> { if let Some(message) = refusal(&owned) { return Err(CatalogError::OwnsGraphs(message)); } + let by = Cleaner::current()?; as_catalog_owner(|| { Spi::connect_mut(|client| { for &role in roles { - clean_up_after(client, role)?; + clean_up_after(client, role, by)?; } Ok::<_, CatalogError>(()) }) @@ -131,7 +132,7 @@ mod tests { use pgrx::prelude::*; use pgrx::spi::Error; - use crate::fixture::{failure, neighbouring_graphs, ownership, statement_failure_as}; + use crate::fixture::{failure, neighbouring_graphs, ownership, quotas, statement_failure_as}; /// The trigger as `docs/semantics.md` tells a superuser to create it. fn install() -> Result<(), Error> { @@ -187,7 +188,8 @@ mod tests { } #[pg_test] - fn after_reassign_owned_drop_owned_removes_the_roles_reader_entries() -> Result<(), Error> { + fn after_reassign_owned_drop_owned_removes_the_roles_reader_entries_and_quotas( + ) -> Result<(), Error> { neighbouring_graphs()?; install()?; Spi::run( @@ -199,6 +201,7 @@ mod tests { ownership()?.as_deref(), Some("dk:dk_owner:{} dr:dr_heir:{} / dk.mine:dr_heir dk.theirs:dk_owner") ); + assert_eq!(quotas()?.as_deref(), Some("")); Ok(()) } @@ -219,6 +222,25 @@ mod tests { Ok(()) } + /// The security review of PR #59: a role's own `DROP OWNED` leaves the + /// quota a graph's owner set on it. + #[pg_test] + fn a_roles_own_drop_owned_keeps_its_quotas() -> Result<(), Error> { + neighbouring_graphs()?; + install()?; + Spi::run( + "CREATE ROLE dk_reader; SELECT warren.grant('dk', 'dk_reader'); + SELECT warren.set_vector_quota('dk', 0, 'dk_reader')", + )?; + assert_eq!(statement_failure_as("dk_reader", "DROP OWNED BY dk_reader")?, None); + assert_eq!(quotas()?.as_deref(), Some("dk:dk_reader:0 dk:dr_owner:4096")); + assert_eq!( + ownership()?.as_deref(), + Some("dk:dk_owner:{dr_owner} dr:dr_owner:{} / dk.mine:dr_owner dk.theirs:dk_owner") + ); + Ok(()) + } + /// `DROP OWNED` checks privileges after `ddl_command_start`, so the /// trigger leaves a role the caller lacks the privileges of, unread, to /// `DROP OWNED`'s own refusal; and a role that does not exist to its diff --git a/crates/pg_warren/src/catalog/dropped_roles/mod.rs b/crates/pg_warren/src/catalog/dropped_roles/mod.rs index 682fef3..da785f5 100644 --- a/crates/pg_warren/src/catalog/dropped_roles/mod.rs +++ b/crates/pg_warren/src/catalog/dropped_roles/mod.rs @@ -14,11 +14,38 @@ mod orphans; use pgrx::prelude::*; use pgrx::spi::SpiClient; +use crate::access::{may_drop_orphans, AccessError}; + +/// Who is cleaning up after a role, for which of its quotas go. +#[derive(Debug, Clone, Copy)] +struct Cleaner { + /// The calling role. + caller: pg_sys::Oid, + /// Whether it may drop orphans (DESIGN.md §14 item 45). + may_drop_orphans: bool, +} + +impl Cleaner { + /// The calling role; before any identity switch. + fn current() -> Result { + // SAFETY: reads the backend's current user id: the caller. + let caller = unsafe { pg_sys::GetUserId() }; + Ok(Self { caller, may_drop_orphans: may_drop_orphans()? }) + } +} + /// Removes what `DROP OWNED` removes of `role` besides what it owns: the /// vectors it wrote on other roles' graphs (those on its own graphs go -/// with them) and its entries in every graph's readers. `client` must run -/// as the catalog owner. -fn clean_up_after(client: &mut SpiClient<'_>, role: pg_sys::Oid) -> Result<(), pgrx::spi::Error> { +/// with them), its entries in every graph's readers and its quotas on +/// readers' vectors where `by` may set them (a member of the graph's +/// owner) or may drop orphans: a role cannot shed a quota another owner +/// set on it, which then waits for `warren.drop_orphans` once the role is +/// dropped (DESIGN.md §5.7). `client` must run as the catalog owner. +fn clean_up_after( + client: &mut SpiClient<'_>, + role: pg_sys::Oid, + by: Cleaner, +) -> Result<(), pgrx::spi::Error> { client.update("DELETE FROM warren.vectors WHERE created_by::oid = $1", None, &[role.into()])?; client.update( "UPDATE warren.graphs SET readers = ARRAY( @@ -28,7 +55,12 @@ fn clean_up_after(client: &mut SpiClient<'_>, role: pg_sys::Oid) -> Result<(), p None, &[role.into()], )?; - // Quotas (work breakdown step 7) add the role's rows of - // `warren.vector_quotas` here (DESIGN.md §5.7, `drop_owned`). + client.update( + "DELETE FROM warren.vector_quotas q USING warren.graphs g + WHERE q.role::oid = $1 AND g.graph_id = q.graph_id + AND ($3 OR pg_catalog.pg_has_role($2::oid, g.owner::oid, 'MEMBER'))", + None, + &[role.into(), by.caller.into(), by.may_drop_orphans.into()], + )?; Ok(()) } diff --git a/crates/pg_warren/src/catalog/dropped_roles/orphans.rs b/crates/pg_warren/src/catalog/dropped_roles/orphans.rs index 240c530..21f1259 100644 --- a/crates/pg_warren/src/catalog/dropped_roles/orphans.rs +++ b/crates/pg_warren/src/catalog/dropped_roles/orphans.rs @@ -8,13 +8,16 @@ use super::super::CatalogError; use crate::access::{as_catalog_owner, entry, is_owner, may_drop_orphans, require_orphan_dropper}; /// Every reference to a dropped role, as `(kind, graph, role, owner)`: -/// graphs whose owner was dropped, reader entries, and (once per graph and -/// role) vectors whose creator was. Quotas (work breakdown step 7) add -/// their rows of `warren.vector_quotas` here (DESIGN.md §5.7). +/// graphs whose owner was dropped, quotas on readers' vectors, reader +/// entries, and (once per graph and role) vectors whose creator was. const ORPHANS: &str = " SELECT 'graph', g.name, g.owner::oid, g.owner::oid FROM warren.graphs g WHERE NOT EXISTS (SELECT FROM pg_catalog.pg_roles r WHERE r.oid = g.owner::oid) UNION ALL + SELECT 'quota', g.name, q.role::oid, g.owner::oid + FROM warren.vector_quotas q JOIN warren.graphs g USING (graph_id) + WHERE NOT EXISTS (SELECT FROM pg_catalog.pg_roles r WHERE r.oid = q.role::oid) + UNION ALL SELECT DISTINCT 'reader', g.name, x.reader::oid, g.owner::oid FROM warren.graphs g, pg_catalog.unnest(g.readers) AS x (reader) WHERE NOT EXISTS (SELECT FROM pg_catalog.pg_roles r WHERE r.oid = x.reader::oid) @@ -24,8 +27,8 @@ const ORPHANS: &str = " WHERE NOT EXISTS (SELECT FROM pg_catalog.pg_roles r WHERE r.oid = v.created_by::oid) ORDER BY 1, 2, 3"; -/// `warren.orphans()`: what names a dropped role (`kind` `graph`, `reader` -/// or `vector`), with the graph and the dropped role's oid. A superuser +/// `warren.orphans()`: what names a dropped role (`kind` `graph`, +/// `quota`, `reader` or `vector`), with the graph and the dropped role's oid. A superuser /// and members of the extension's owner see every row; others see those /// of graphs they own. `PARALLEL SAFE`: it reads no graph. #[pg_extern(schema = "warren", volatile, parallel_safe)] @@ -60,8 +63,8 @@ fn visible_orphans() -> Result, CatalogError> } /// `warren.drop_orphans()`: drops every graph whose owner was dropped (as -/// `drop_graph` does), removes reader entries that name dropped roles and -/// deletes the vectors whose creator was dropped. For a superuser or a +/// `drop_graph` does), removes reader entries and quotas that name dropped +/// roles and deletes the vectors whose creator was dropped. For a superuser or a /// member of the extension's owner (DESIGN.md §14 item 45). Returns the /// number of graphs dropped. #[pg_extern(schema = "warren")] @@ -79,7 +82,7 @@ macro_rules! dropped { /// Each pass of the sweep: a query that locks the rows naming a dropped /// role and returns their keys, then the statement that changes those rows /// (`$1`, their keys), checking each again. -const PASSES: [(&str, &str); 3] = [ +const PASSES: [(&str, &str); 4] = [ ( concat!( "SELECT graph_id::int8 FROM warren.graphs g WHERE ", @@ -121,6 +124,17 @@ const PASSES: [(&str, &str); 3] = [ dropped!("v.created_by") ), ), + ( + concat!( + "SELECT graph_id::int8 FROM warren.vector_quotas q WHERE ", + dropped!("q.role"), + " FOR UPDATE" + ), + concat!( + "DELETE FROM warren.vector_quotas q WHERE graph_id = ANY ($1) AND ", + dropped!("q.role") + ), + ), ]; fn sweep() -> Result { @@ -130,8 +144,6 @@ fn sweep() -> Result { // row would re-check it (EvalPlanQual) under its first snapshot, which // misses a role created by the transaction it waited for, and so would // drop a graph adopted for that role, or prune a reader just granted. - // Quotas (work breakdown step 7) add a pass for the rows of - // `warren.vector_quotas` that name dropped roles. let dropped = as_catalog_owner(|| { Spi::connect_mut(|client| { let mut changed = Vec::with_capacity(PASSES.len()); @@ -158,7 +170,7 @@ mod tests { use pgrx::prelude::*; use pgrx::spi::Error; - use crate::fixture::{failure_as, neighbouring_graphs, ownership, rows}; + use crate::fixture::{failure_as, neighbouring_graphs, ownership, quotas, rows}; /// [`neighbouring_graphs`], then a bare `DROP ROLE dr_owner`; returns /// its OID. @@ -173,11 +185,12 @@ mod tests { const LISTED: &str = "SELECT kind || ':' || graph || ':' || role::text FROM warren.orphans()"; #[pg_test] - fn orphans_lists_graphs_reader_entries_and_vectors_of_dropped_roles() -> Result<(), Error> { + fn orphans_lists_graphs_quotas_reader_entries_and_vectors_of_dropped_roles() -> Result<(), Error> + { let gone = orphaned()?; assert_eq!( rows(LISTED)?, - Some(format!("graph:dr:{gone} reader:dk:{gone} vector:dk:{gone}")) + Some(format!("graph:dr:{gone} quota:dk:{gone} reader:dk:{gone} vector:dk:{gone}")) ); Ok(()) } @@ -193,7 +206,7 @@ mod tests { Spi::run("SET ROLE dk_stranger")?; let strangers = rows(LISTED)?; Spi::run("RESET ROLE")?; - assert_eq!(owners, Some(format!("reader:dk:{gone} vector:dk:{gone}"))); + assert_eq!(owners, Some(format!("quota:dk:{gone} reader:dk:{gone} vector:dk:{gone}"))); assert_eq!(strangers.as_deref(), Some("")); Ok(()) } @@ -203,6 +216,7 @@ mod tests { orphaned()?; assert_eq!(Spi::get_one::("SELECT warren.drop_orphans()")?, Some(1)); assert_eq!(ownership()?.as_deref(), Some("dk:dk_owner:{} / dk.theirs:dk_owner")); + assert_eq!(quotas()?.as_deref(), Some("")); assert_eq!(rows(LISTED)?.as_deref(), Some("")); Ok(()) } @@ -226,7 +240,10 @@ mod tests { Spi::run("SET ROLE dr_member")?; let listed = rows(LISTED)?; Spi::run("RESET ROLE")?; - assert_eq!(listed, Some(format!("graph:dr:{gone} reader:dk:{gone} vector:dk:{gone}"))); + assert_eq!( + listed, + Some(format!("graph:dr:{gone} quota:dk:{gone} reader:dk:{gone} vector:dk:{gone}")) + ); assert_eq!(failure_as("dr_member", "warren.drop_orphans()")?, None); assert_eq!(ownership()?.as_deref(), Some("dk:dk_owner:{} / dk.theirs:dk_owner")); Ok(()) @@ -263,7 +280,10 @@ mod tests { let setter = rows(LISTED)?; Spi::run("RESET ROLE")?; assert_eq!(weak.as_deref(), Some("")); - assert_eq!(setter, Some(format!("graph:dr:{gone} reader:dk:{gone} vector:dk:{gone}"))); + assert_eq!( + setter, + Some(format!("graph:dr:{gone} quota:dk:{gone} reader:dk:{gone} vector:dk:{gone}")) + ); assert_eq!(failure_as("dr_setter", "warren.drop_orphans()")?, None); Ok(()) } diff --git a/crates/pg_warren/src/catalog/error.rs b/crates/pg_warren/src/catalog/error.rs index 8dda4bd..942dd96 100644 --- a/crates/pg_warren/src/catalog/error.rs +++ b/crates/pg_warren/src/catalog/error.rs @@ -104,6 +104,29 @@ pub(crate) enum CatalogError { /// installed (DESIGN.md §14 item 44a); the message names them. #[error("{0}")] OwnsGraphs(String), + /// `reassign_owned` would take what the new role stores on a graph + /// past its quota on readers' vectors there (DESIGN.md §5.7). + #[error( + "reassigning the persisted vectors of role \"{from}\" on graph \"{graph}\" ({moved} bytes) \ + would take those of role \"{to}\" to {after} bytes, past its quota of {quota} bytes (it \ + stores {before}); drop some of them first, or ask the graph's owner for a larger quota" + )] + QuotaExceeded { + /// The role whose vectors move. + from: String, + /// The role they move to. + to: String, + /// The graph. + graph: String, + /// The bytes that move. + moved: u64, + /// `to`'s quota on the graph. + quota: u64, + /// What `to` stores there before the move. + before: u64, + /// And after it. + after: u64, + }, /// An SPI call failed. #[error(transparent)] Spi(#[from] pgrx::spi::Error), @@ -134,6 +157,7 @@ impl SqlError for CatalogError { PgSqlErrorCode::ERRCODE_E_R_I_E_EVENT_TRIGGER_PROTOCOL_VIOLATED } Self::OwnsGraphs(_) => PgSqlErrorCode::ERRCODE_DEPENDENT_OBJECTS_STILL_EXIST, + Self::QuotaExceeded { .. } => PgSqlErrorCode::ERRCODE_CONFIGURATION_LIMIT_EXCEEDED, Self::Spi(_) => PgSqlErrorCode::ERRCODE_INTERNAL_ERROR, } } diff --git a/crates/pg_warren/src/catalog/info.rs b/crates/pg_warren/src/catalog/info.rs index f902c2a..709233f 100644 --- a/crates/pg_warren/src/catalog/info.rs +++ b/crates/pg_warren/src/catalog/info.rs @@ -9,7 +9,8 @@ use super::{CatalogError, GraphDefinition}; use crate::access::{as_catalog_owner, entry, is_reader, require_reader}; use crate::read_only::read_one; -/// `warren.info`: graph `name`'s definition, current version, staleness +/// `warren.info`: graph `name`'s definition (with its default quota on +/// readers' vectors, `reader_vector_quota`), current version, staleness /// (`now() - built_at` of the current version), total bytes, and every /// version with its statistics, oldest first. #[pg_extern(schema = "warren", stable, parallel_safe)] @@ -28,6 +29,7 @@ fn describe(name: &str) -> Result { 'has_relations', g.has_relations, 'build_reverse', g.build_reverse, 'store_merged', g.store_merged, 'node_order', g.node_order, 'retain_versions', g.retain_versions, 'current_version', g.current_version, + 'reader_vector_quota', g.reader_vector_quota, 'staleness', (SELECT (now() - c.built_at)::text FROM warren.versions c WHERE c.version_id = g.current_version AND c.graph_id = g.graph_id), @@ -133,6 +135,9 @@ mod tests { assert_eq!(info("{retain_versions}")?.as_deref(), Some("2")); assert_eq!(info("{store_merged}")?.as_deref(), Some("true")); assert_eq!(info("{node_order}")?.as_deref(), Some("\"id\"")); + assert_eq!(info("{reader_vector_quota}")?.as_deref(), Some("null")); + Spi::run("SELECT warren.set_vector_quota('i', 4096)")?; + assert_eq!(info("{reader_vector_quota}")?.as_deref(), Some("4096")); assert_eq!(info("{current_version}")?, Some(promoted.to_string())); assert_eq!(info("{versions,1,version}")?, Some(staged.to_string())); assert_eq!(info("{versions,0,current}")?.as_deref(), Some("true")); diff --git a/crates/pg_warren/src/catalog/mod.rs b/crates/pg_warren/src/catalog/mod.rs index c69ed64..40f0ca1 100644 --- a/crates/pg_warren/src/catalog/mod.rs +++ b/crates/pg_warren/src/catalog/mod.rs @@ -15,6 +15,7 @@ mod graph; mod info; mod owner; mod promote; +pub(crate) mod quota; mod readers; mod retention; mod version; diff --git a/crates/pg_warren/src/catalog/owner/reassign.rs b/crates/pg_warren/src/catalog/owner/reassign.rs index 0dcf8f1..2c4b89a 100644 --- a/crates/pg_warren/src/catalog/owner/reassign.rs +++ b/crates/pg_warren/src/catalog/owner/reassign.rs @@ -6,15 +6,18 @@ use pgrx::prelude::*; use super::super::create_graph::{writable_notice, writable_schemas}; use super::super::graph::NEW_OWNER; +use super::super::quota::{admits, lock, quota_of, stored, vector_bytes, Usage}; use super::super::CatalogError; use crate::access::{ - as_catalog_owner, entry, refuse_bootstrap_superuser, require_can_receive, require_privileges_of, + as_catalog_owner, entry, may_drop_orphans, refuse_bootstrap_superuser, require_can_receive, + require_privileges_of, role_name, }; /// `warren.reassign_owned(old regrole, new regrole)`: gives every graph /// `old` owns to `new`, with `set_owner`'s rules except its source check /// (`REASSIGN OWNED` checks none), and makes `new` the creator of every -/// vector `old` wrote. The caller needs the privileges of `old` and must +/// vector `old` wrote, within `new`'s quotas on readers' vectors. The +/// caller needs the privileges of `old` and must /// be able to `SET ROLE` to `new`. Returns the number of graphs moved. #[pg_extern( schema = "warren", @@ -27,6 +30,61 @@ fn reassign_owned(old: pg_sys::Oid, new: pg_sys::Oid) -> i64 { entry(|_| reassign(old, new)) } +/// Makes `$2` the creator of every vector `$1` wrote, and returns, per +/// graph, the bytes moved where they count against `$2`'s quota: unless +/// `$2` or the caller `$3` is a member of the graph's owner (after the +/// graphs moved), or `$4` (the caller may drop orphans). +const MOVE_VECTORS: &str = concat!( + "WITH moved AS ( + UPDATE warren.vectors v SET created_by = $2::oid::regrole WHERE v.created_by::oid = $1 + RETURNING v.graph_id, ", + vector_bytes!(), + " AS bytes) + SELECT g.graph_id, g.name, sum(m.bytes)::int8 + FROM moved m JOIN warren.graphs g ON g.graph_id = m.graph_id + WHERE NOT $4 + AND NOT pg_catalog.pg_has_role($2::oid, g.owner::oid, 'MEMBER') + AND NOT pg_catalog.pg_has_role($3::oid, g.owner::oid, 'MEMBER') + GROUP BY g.graph_id, g.name + ORDER BY g.graph_id" +); + +/// Vectors moved to another role on one graph whose bytes count against +/// that role's quota there. +struct CountedMove { + /// The graph. + graph_id: i32, + /// Its name. + graph: String, + /// The bytes moved. + bytes: u64, +} + +/// Refuses the move with [`CatalogError::QuotaExceeded`] if it takes what +/// `new` stores on the graph past its quota there (DESIGN.md §5.7): with +/// the (graph, `new`) lock held, as a write by `new` would be, and the +/// move already in what the latest snapshot sees. +fn admit_move(counted: &CountedMove, from: &str, new: pg_sys::Oid) -> Result<(), CatalogError> { + lock(counted.graph_id, new); + let Some(quota) = quota_of(counted.graph_id, new)? else { + return Ok(()); + }; + let (stored, _) = stored(counted.graph_id, new, None)?; + let usage = Usage::of(stored.saturating_sub(counted.bytes), 0, counted.bytes); + if admits(false, Some(quota), usage) { + return Ok(()); + } + Err(CatalogError::QuotaExceeded { + from: from.to_owned(), + to: role_name(new).unwrap_or_default(), + graph: counted.graph.clone(), + moved: counted.bytes, + quota, + before: usage.before, + after: usage.after, + }) +} + /// What a graph's search_path NOTICE needs: its name, whether it is built /// from queries, and its `search_path`. type Moved = (String, bool, String); @@ -38,7 +96,10 @@ fn reassign(old: pg_sys::Oid, new: pg_sys::Oid) -> Result { // One UPDATE takes each row's lock, waiting for a running refresh or // set_owner, and re-checks `owner` once it holds it (EvalPlanQual), as // `GraphDefinition::lock` re-checks ownership (DESIGN.md §5.7). - let moved = as_catalog_owner(|| { + // SAFETY: reads the backend's current user id: the caller. + let caller = unsafe { pg_sys::GetUserId() }; + let exempt = may_drop_orphans()?; + let (moved, counted) = as_catalog_owner(|| { Spi::connect_mut(|client| { let graphs = client .update( @@ -57,14 +118,28 @@ fn reassign(old: pg_sys::Oid, new: pg_sys::Oid) -> Result { )) }) .collect::, _>>()?; - client.update( - "UPDATE warren.vectors SET created_by = $2::oid::regrole WHERE created_by::oid = $1", - None, - &[old.into(), new.into()], - )?; - Ok::<_, CatalogError>(graphs) + let counted = client + .update( + MOVE_VECTORS, + None, + &[old.into(), new.into(), caller.into(), exempt.into()], + )? + .map(|row| { + Ok::(CountedMove { + graph_id: row.get(1)?.unwrap_or_default(), + graph: row.get(2)?.unwrap_or_default(), + bytes: row + .get::(3)? + .map_or(0, |b| u64::try_from(b).unwrap_or_default()), + }) + }) + .collect::, _>>()?; + Ok::<_, CatalogError>((graphs, counted)) }) })?; + for counted in &counted { + admit_move(counted, &from, new)?; + } for (graph, built_from_queries, search_path) in &moved { if !built_from_queries { continue; @@ -83,7 +158,9 @@ mod tests { use pgrx::prelude::*; use pgrx::spi::Error; - use crate::fixture::{failure, failure_as, neighbouring_graphs, ownership}; + use crate::fixture::{ + failure, failure_as, neighbouring_graphs, ownership, refresh, ring, rows, + }; /// `REASSIGN OWNED` checks no source, so neither does this: `dr_heir` /// cannot read `dr`'s tables. Reader entries stay, as privileges do. @@ -166,4 +243,65 @@ mod tests { ); Ok(()) } + + /// Graph `rq`, a ring of 100 nodes (an `f64` vector is 800 bytes) with + /// a default quota of 1000 bytes, where `rq_mallory`, a member of + /// `rq_group`, is a reader. + fn quota_and_a_group() -> Result<(), Error> { + ring("rq", 100, 2)?; + refresh("rq", true)?; + Spi::run( + "CREATE ROLE rq_group; CREATE ROLE rq_mallory; GRANT rq_group TO rq_mallory; + SELECT warren.grant('rq', 'rq_mallory'); + SELECT warren.set_vector_quota('rq', 1000)", + ) + } + + fn vectors() -> Result, Error> { + rows("SELECT name || ':' || created_by::text FROM warren.vectors ORDER BY name") + } + + /// The security review of PR #59: moving vectors to another role must + /// not take that role past its quota, or a reader could empty its own + /// usage and write again without bound. + #[pg_test] + fn reassign_owned_cannot_move_vectors_past_the_new_roles_quota() -> Result<(), Error> { + quota_and_a_group()?; + let write = |name: &str| format!("warren.vector_const('rq', '{name}', 1)"); + let reassign = "warren.reassign_owned('rq_mallory', 'rq_group')"; + assert_eq!(failure_as("rq_mallory", &write("a"))?, None); + assert_eq!(failure_as("rq_mallory", reassign)?, None, "rq_group has room for a"); + assert_eq!(failure_as("rq_mallory", &write("b"))?, None); + assert_eq!( + failure_as("rq_mallory", reassign)?.as_deref(), + Some( + "53400: reassigning the persisted vectors of role \"rq_mallory\" on graph \"rq\" \ + (800 bytes) would take those of role \"rq_group\" to 1600 bytes, past its quota \ + of 1000 bytes (it stores 800); drop some of them first, or ask the graph's owner \ + for a larger quota" + ) + ); + assert_eq!(vectors()?.as_deref(), Some("a:rq_group b:rq_mallory"), "nothing moved"); + // The graph's owner may move them: it could set the quota anyway. + Spi::run("GRANT rq_mallory, rq_group TO rq_owner")?; + assert_eq!(failure_as("rq_owner", reassign)?, None); + assert_eq!(vectors()?.as_deref(), Some("a:rq_group b:rq_group")); + Ok(()) + } + + /// A move to a member of the graph's owner role is exempt, as its + /// writes are. + #[pg_test] + fn reassign_owned_to_the_owner_is_not_counted() -> Result<(), Error> { + quota_and_a_group()?; + Spi::run("GRANT rq_owner TO rq_group")?; + assert_eq!(failure_as("rq_mallory", "warren.vector_const('rq', 'a', 1)")?, None); + assert_eq!(failure_as("rq_mallory", "warren.vector_const('rq', 'b', 1, 'f32')")?, None); + assert_eq!( + failure_as("rq_mallory", "warren.reassign_owned('rq_mallory', 'rq_group')")?, + None + ); + assert_eq!(vectors()?.as_deref(), Some("a:rq_group b:rq_group")); + Ok(()) + } } diff --git a/crates/pg_warren/src/catalog/quota/decision.rs b/crates/pg_warren/src/catalog/quota/decision.rs new file mode 100644 index 0000000..401b86b --- /dev/null +++ b/crates/pg_warren/src/catalog/quota/decision.rs @@ -0,0 +1,68 @@ +//! The decision on a write or a move of persisted vectors (DESIGN.md §14 +//! item 47): admitted unless it takes a role's usage past its quota and +//! above what it was. + +/// What a role stores on a graph before and after a write, in bytes. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct Usage { + /// Before the write. + pub(crate) before: u64, + /// After it. + pub(crate) after: u64, +} + +impl Usage { + /// Usage after writing `written` bytes in place of `replaced` bytes of + /// the role's own (0 if the write replaces none), from `stored` bytes. + pub(crate) fn of(stored: u64, replaced: u64, written: u64) -> Self { + Self { before: stored, after: stored.saturating_sub(replaced).saturating_add(written) } + } +} + +/// Whether a write of `usage` is admitted under `quota` (bytes; `None` +/// for none), the writer `exempt` if a member of the owner role: unless it +/// takes usage past the quota and above what it was before, so replacing a +/// vector with one no larger always is, and a quota lowered below what is +/// in use blocks only growth. +pub(crate) fn admits(exempt: bool, quota: Option, usage: Usage) -> bool { + exempt || quota.is_none_or(|quota| usage.after <= quota || usage.after <= usage.before) +} + +#[cfg(test)] +mod tests { + use super::{admits, Usage}; + + #[test] + fn a_write_within_the_quota_is_admitted() { + assert!(admits(false, Some(1000), Usage::of(200, 0, 800))); + assert!(!admits(false, Some(1000), Usage::of(200, 0, 801))); + } + + #[test] + fn without_a_quota_or_when_exempt_every_write_is_admitted() { + assert!(admits(false, None, Usage::of(u64::MAX, 0, u64::MAX))); + assert!(admits(true, Some(0), Usage::of(0, 0, 1))); + } + + #[test] + fn replacing_a_vector_with_one_no_larger_is_admitted_past_the_quota() { + assert!(admits(false, Some(100), Usage::of(800, 800, 800))); + assert!(admits(false, Some(100), Usage::of(800, 800, 400))); + assert!(!admits(false, Some(100), Usage::of(800, 400, 800))); + } + + #[test] + fn a_quota_lowered_below_use_blocks_only_growth() { + assert!(!admits(false, Some(500), Usage::of(800, 0, 1))); + assert!(admits(false, Some(500), Usage::of(800, 100, 100))); + assert!(!admits(false, Some(0), Usage::of(0, 0, 1))); + assert!(admits(false, Some(0), Usage::of(0, 0, 0))); + } + + #[test] + fn usage_after_a_write_replaces_only_what_it_replaces() { + assert_eq!(Usage::of(800, 300, 100), Usage { before: 800, after: 600 }); + assert_eq!(Usage::of(10, 20, 5), Usage { before: 10, after: 5 }); + assert_eq!(Usage::of(u64::MAX, 0, 1), Usage { before: u64::MAX, after: u64::MAX }); + } +} diff --git a/crates/pg_warren/src/catalog/quota/ledger.rs b/crates/pg_warren/src/catalog/quota/ledger.rs new file mode 100644 index 0000000..b57762c --- /dev/null +++ b/crates/pg_warren/src/catalog/quota/ledger.rs @@ -0,0 +1,95 @@ +//! The quota ledger in the catalog: the quota of a role on a graph, what +//! it stores there, and the (graph, role) lock (DESIGN.md §14 item 47). + +use pgrx::prelude::*; + +use super::super::CatalogError; +use crate::access::as_catalog_owner; +use crate::read_only::read_one_latest; + +/// pg_warren's `objsubid` (`locktag_field4`) on its advisory-type locks. +/// Applications' advisory locks have 1 (one `bigint` key) or 2 (two `int` +/// keys), so the quota's lock never collides with theirs; `pg_locks` shows +/// it as `advisory` with this `objsubid`. +pub(crate) const QUOTA_LOCK_OBJSUBID: u16 = 0x5751; + +/// The quota of role `$2` on graph `$1`: its own row, else the graph's +/// default (NULL for none). +const QUOTA: &str = "SELECT coalesce( + (SELECT q.bytes FROM warren.vector_quotas q WHERE q.graph_id = $1 AND q.role::oid = $2), + (SELECT g.reader_vector_quota FROM warren.graphs g WHERE g.graph_id = $1))"; + +/// The bytes of the persisted vectors role `$2` wrote on graph `$1`, and of +/// those among them that are vector `$3` (the one replaced, if any), as an +/// array of two: one scan of `vectors (graph_id, created_by)`. +const STORED: &str = concat!( + "SELECT ARRAY[coalesce(sum(", + vector_bytes!(), + "), 0), coalesce(sum(", + vector_bytes!(), + ") FILTER (WHERE v.vector_id = $3), 0)]::int8[] + FROM warren.vectors v WHERE v.graph_id = $1 AND v.created_by::oid = $2" +); + +/// The quota of `role` on graph `graph_id`, as committed now. +pub(crate) fn quota_of(graph_id: i32, role: pg_sys::Oid) -> Result, CatalogError> { + let quota = as_catalog_owner(|| { + Ok::<_, CatalogError>(read_one_latest::(QUOTA, &[graph_id.into(), role.into()])?) + })?; + Ok(quota.map(|bytes| u64::try_from(bytes).unwrap_or_default())) +} + +/// What `role` stores on graph `graph_id`, and what of it is vector +/// `replacing`, in bytes: under the latest snapshot, which sees what is +/// committed now (whatever the isolation level) and this transaction's own +/// earlier writes. +pub(crate) fn stored( + graph_id: i32, + role: pg_sys::Oid, + replacing: Option, +) -> Result<(u64, u64), CatalogError> { + let sums = as_catalog_owner(|| { + Ok::<_, CatalogError>(read_one_latest::>( + STORED, + &[graph_id.into(), role.into(), replacing.into()], + )?) + })? + .unwrap_or_default(); + let bytes = |at: usize| sums.get(at).map_or(0, |&sum| u64::try_from(sum).unwrap_or_default()); + Ok((bytes(0), bytes(1))) +} + +/// Takes the transaction-scoped lock on (graph `graph_id`, `role`), +/// waiting for a transaction that holds it to end; take it before reading +/// the role's quota or usage. Only on the leader: persisting writers and +/// `reassign_owned` are `PARALLEL UNSAFE`, and a lock a parallel worker +/// took would be released when the worker exits (DESIGN.md §5.7, Spike +/// results). +pub(crate) fn lock(graph_id: i32, role: pg_sys::Oid) { + // SAFETY: reads the backend's database id, on the main thread. + let database = unsafe { pg_sys::MyDatabaseId }; + let tag = pg_sys::LOCKTAG { + locktag_field1: database.to_u32(), + locktag_field2: graph_id.cast_unsigned(), + locktag_field3: role.to_u32(), + locktag_field4: QUOTA_LOCK_OBJSUBID, + locktag_type: LOCKTAG_ADVISORY, + locktag_lockmethodid: USER_LOCKMETHOD, + }; + // SAFETY: a heavyweight lock request on the main thread, held until + // the transaction ends (not a session lock); waiting, it raises an + // ERROR on deadlock, `lock_timeout` or cancel, which pgrx's guarded + // FFI unwinds. With `dontWait` false it always returns having the lock. + unsafe { + pg_sys::LockAcquire(&tag, EXCLUSIVE_LOCK, false, false); + } +} + +/// `LOCKTAG_ADVISORY`, as `LOCKTAG.locktag_type` holds it. +const LOCKTAG_ADVISORY: u8 = pg_sys::LockTagType::LOCKTAG_ADVISORY as u8; + +/// `USER_LOCKMETHOD`, the lock method of advisory locks. +const USER_LOCKMETHOD: u8 = pg_sys::USER_LOCKMETHOD as u8; + +/// `ExclusiveLock`, the mode `pg_advisory_xact_lock` takes. +const EXCLUSIVE_LOCK: pg_sys::LOCKMODE = pg_sys::ExclusiveLock as pg_sys::LOCKMODE; diff --git a/crates/pg_warren/src/catalog/quota/mod.rs b/crates/pg_warren/src/catalog/quota/mod.rs new file mode 100644 index 0000000..478b4a0 --- /dev/null +++ b/crates/pg_warren/src/catalog/quota/mod.rs @@ -0,0 +1,22 @@ +//! The ledger of quotas on readers' vectors (DESIGN.md §5.7, §14 items 46 +//! and 47): the quota that applies to a role on a graph, what the role +//! stores there, the (graph, role) lock under which both are read, and the +//! decision. `vector::quota` admits writes with it and +//! `reassign_owned` moves of vectors to another role. + +/// The bytes of persisted vector `v`, a row of `warren.vectors`, in SQL: +/// what it holds when loaded, as `compute_sql::vector_bytes` counts it. +macro_rules! vector_bytes { + () => { + "v.length * v.width * CASE v.dtype WHEN 'f32' THEN 4 WHEN 'bool' THEN 1 ELSE 8 END" + }; +} +pub(crate) use vector_bytes; + +mod decision; +mod ledger; + +pub(crate) use decision::{admits, Usage}; +#[cfg(any(test, feature = "pg_test"))] +pub(crate) use ledger::QUOTA_LOCK_OBJSUBID; +pub(crate) use ledger::{lock, quota_of, stored}; diff --git a/crates/pg_warren/src/compute_sql/vectors.rs b/crates/pg_warren/src/compute_sql/vectors.rs index a1840f1..a7c32b0 100644 --- a/crates/pg_warren/src/compute_sql/vectors.rs +++ b/crates/pg_warren/src/compute_sql/vectors.rs @@ -24,8 +24,8 @@ use crate::catalog::{GraphDefinition, VersionId}; use crate::segment::CHUNK_SIZE; use crate::source::IdDatum; use crate::vector::{ - drop_transient, keep_transient, transient, transients, write_vector, Binding, SpiVectorStore, - VectorError, VectorId, VectorRecord, VectorStore, + admit_to_quota, drop_transient, keep_transient, transient, transients, write_vector, Binding, + SpiVectorStore, VectorError, VectorId, VectorRecord, VectorStore, }; /// The budget cached vectors are charged to while cached: unlimited, since @@ -266,8 +266,10 @@ impl Vectors { /// any vector of that name of either kind; a persisted one replaced /// must have been written by the caller, or the caller must own the /// graph. A transient one is refused if it would take this backend's - /// transient vectors past `warren.max_cache_memory`; either kind is - /// kept in the cache only as far as the cap allows. + /// transient vectors past `warren.max_cache_memory`, a persisted one if + /// it would take the caller's persisted vectors on the graph past its + /// quota (DESIGN.md §5.7); either kind is kept in the cache only as far + /// as the cap allows. pub(crate) fn save( &mut self, name: &str, @@ -277,7 +279,11 @@ impl Vectors { self.check_writable(name, persist)?; let graph_id = self.graph.graph_id; let replacing = self.record(name).map(VectorRecord::checked); - if !persist { + if persist { + let written = vector_bytes(vector.dtype(), vector.width(), vector.id_space()); + let replaced = replacing.map(|old| old.vector); + admit_to_quota(&self.graph, name, replaced, written).map_err(VectorError::from)?; + } else { let replaced = transient(graph_id, self.version, name).map_or(0, |old| old.bytes()); admit_transient(replaced, vector.bytes())?; } diff --git a/crates/pg_warren/src/fixture.rs b/crates/pg_warren/src/fixture.rs index c291b2d..7d236be 100644 --- a/crates/pg_warren/src/fixture.rs +++ b/crates/pg_warren/src/fixture.rs @@ -309,9 +309,9 @@ pub(crate) fn statement_failure_as(role: &str, statement: &str) -> Result