Soak at full scale: text vs bigint ids, 10M nodes / 100M edges (M10)

DESIGN.md

@@ -1561,17 +1561,49 @@ A pooled backend's own memory is 4–12 MB with the shared cache against the ver
 
 *The soak with the shared cache* (`bench/results/soak/*-m9-shared.json`; full scale, PG 16 preloaded, the release build above; case a at the default cap, b-merged at `warren.shared_cache_memory = 8GB`, without its second version, library or community runs). Every query stage reads the shared version, and a backend's private memory falls by the version: case a's stages peak at 0.01–3.11 GB private against 1.22–4.31 GB in the 0.4.0 run, `pagerank` at 1.06 GB against 2.27 GB. The cold load into the shared cache takes 1.5–1.7 s for case a (2.4–3.0 s privately) and 7.4–33.3 s for b-merged's 6.0 GB (22.4–27.1 s privately; the first load reads the segments from disk). Times of stages that read a loaded version are within the run-to-run noise of the 0.4.0 run, except where 0.4.0 reloaded: on b-merged at the default `warren.max_cache_memory`, which no longer counts the version, `bfs` takes 0.89 s (25.1 s), `neighbours` 0.1 ms (19.2 s), `components` 7.6 s (28.6 s) and `pagerank` 16.5 s (38.4 s), as fast as 0.4.0 at an 8GB `warren.max_cache_memory`; `eigen`, refused there before (53400: the version and the solve over the two limits), now runs (135.7 s for 200 steps; it does not converge, as on case a). On case a, `louvain` completed (627.5 s, five levels, modularity 0.121) where the 0.4.0 run was cancelled at 382.7 s by the disk guard; `leiden` fails as in 0.4.0 (53400, compute memory). The `refresh` row is unchanged (a build is private): b-merged's misses at 4.42 GB, as in 0.4.0.
 
-**Text node ids (M10 step 4, §5.7; `bench/results/soak/a-scale2*-m10*.json`).** Case a with IRIs as node ids (`bench/soak.py --ids text`: node *n* is `https://example.org/resource/<n>`, 29 bytes and its digits) against the same case with `bigint` ids, *at half scale* (5M nodes, 50M edges), for the disk: the full-scale text run needs about 16 GB free (the 3.6 GB edge table, the recursive CTE's index, two or three 1.44 GB text versions and WAL, above `--disk-floor`'s 4 GB), and the machine had about 8 GB, shared with other work, so the full run is still to do. *Measured on* the soak's machine and settings (above), a release build of 5f905b4 from a private PG 16.15 tree, without the recursive CTE baseline or the community runs; the second version was skipped for disk, so `diff` and `distance` were not run. Each pair ran back to back under a load average of 2.0–4.4 from other agents' builds, so a time can move by 15–20% for noise alone (`pagerank`, which maps no ids, took 9.1 s against 6.7 s privately and 8.4 s against 7.2 s shared). Figures are text / `bigint`:
+**Text node ids (M10 step 4, §5.7; `bench/results/soak/a-scale2*-m10*.json`).** Case a with IRIs as node ids (`bench/soak.py --ids text`: node *n* is `https://example.org/resource/<n>`, 29 bytes and its digits) against the same case with `bigint` ids, first *at half scale* (5M nodes, 50M edges), for the disk (the full-scale run needs about 16 GB free: the 3.6 GB edge table, the recursive CTE's index, two or three 1.44 GB text versions and WAL, above `--disk-floor`; the machine then had about 8 GB), then at full scale (below). *Measured on* the soak's machine and settings (above), a release build of 5f905b4 from a private PG 16.15 tree, without the recursive CTE baseline or the community runs; the second version was skipped for disk, so `diff` and `distance` were not run. Each pair ran back to back under a load average of 2.0–4.4 from other agents' builds, so a time can move by 15–20% for noise alone (`pagerank`, which maps no ids, took 9.1 s against 6.7 s privately and 8.4 s against 7.2 s shared). Figures are text / `bigint`:
 - *Map size:* the version is 719.5 / 560.6 MB, each within 300 bytes of `bench/soak.py`'s `segment_estimate` (§5.3). The text id map is 238.9 MB (the IRIs' bytes 178.9 MB, `id_offsets` 40 MB, `id_order` 20 MB), against 80 MB of pairs. Cached, 719.5 / 600.6 MB, since a text version builds no dense-id direction when it loads. At full scale: 1.44 / 1.12 GB.
 - *Build:* `refresh` takes 56.4 / 41.8 s (58.6 / 40.1 s in the shared pair) and peaks at 0.92 / 0.79 GB private (VmHWM 2.08 / 1.95 GB); from the generator query, 61.1 / 46.4 s. Text costs about 35% more: the intake copies each row's two IRIs into its batch and maps them on the pool through the text index.
 - *Cold load:* 1.03 / 0.77 s privately, 1.06 / 0.79 s into the shared cache.
 - *Lookups:* `neighbours` of one IRI 0.1 / 0.1 ms (median of 20 calls), `most_similar` 1.1 / 1.0 ms, `similarity` of 100,000 pairs 1.31 / 0.41 s (past 65,536 lookups, so through a per-call index, which costs 40 MB at 5M nodes).
 - *Traversal:* `bfs` to depth 6 (1.07M rows) 0.95 / 0.49 s, `reachable_set` (5.0M rows) 3.72 / 2.22 s. Every row's `text` datum is built and checked against the encoding, so row emission, already most of these calls' time (above), about doubles. The kernels that return no ids are unchanged: `components` 2.77 / 2.74 s, `eigen` for 200 steps 75.9 / 77.3 s, and the library within noise (`clustering` 27.0 / 28.8 s, betweenness 40.5 / 40.1 s, `minimum_spanning_forest` 22.5 / 22.0 s).
-- *Peaks:* the stages that read the version peak about 0.12 GB higher privately, the difference in the versions' sizes (`pagerank` 1.28 / 1.16 GB, `eigen` 2.28 / 2.17 GB). The calls that return a row per node peak higher still: `components` 1.28 / 0.80 GB and `minimum_spanning_forest` 1.77 / 0.98 GB, for their rows of text ids. Not investigated.
+- *Peaks:* the stages that read the version peak about 0.12 GB higher privately, the difference in the versions' sizes (`pagerank` 1.28 / 1.16 GB, `eigen` 2.28 / 2.17 GB). The calls that return a row per node peak higher still: `components` 1.28 / 0.80 GB and `minimum_spanning_forest` 1.77 / 0.98 GB, for their rows of text ids (explained at full scale, below).
 - *Shared cache:* the shared version is 719.5 / 600.6 MB, and the private peaks fall by the version, as for `bigint` ids: `pagerank` 0.56 / 0.56 GB, `bfs` 0.12 / 0.12 GB, `components` 0.56 / 0.20 GB.
 
 Every §12 target that ran was met at this scale except `reachable_set`, which misses for both id types (12–13 / 22–23M edges/s).
 
+*At full scale* (10M nodes, 100M edges; `bench/results/soak/a-text-m10.json` and `a-m10.json`): case a with every stage, the recursive CTE baseline, the second version (`diff`, `distance`), the library and both community detections included, a release build of 7934941 on the soak's machine and settings, text then `bigint` back to back with no other agent building (load average 1.0–1.5, the soak's own). Neither run came within `--disk-floor` (3 GiB, of about 20 GB free). Figures are text / `bigint`; peaks are private RssAnon:
+
+| Stage | Text | `bigint` | Text / `bigint` |
+|---|---|---|---|
+| Version (stored / cached) | 1.44 / 1.44 GB | 1.12 / 1.20 GB | 1.28 / 1.20 |
+| `refresh` from the table | 110.2 s, 1.80 GB (VmHWM 2.96) | 79.1 s, 1.55 GB (VmHWM 2.71) | 1.39 time |
+| `refresh` from the generator | 132.6 s, 1.87 GB | 99.1 s, 1.57 GB | 1.34 |
+| Second `refresh` (0.1% fewer edges) | 115.6 s | 88.3 s | 1.31 |
+| Cold load | 1.67 s (3,520 segment reads) | 1.27 s (2,208) | 1.32 |
+| `neighbours` of one id | 0.06 ms | 0.06 ms | 1 |
+| `bfs` depth 6 (1.47M rows) | 1.42 s, 1.58 GB | 0.66 s, 1.34 GB | 2.16 |
+| Recursive CTE, depth 6 | 7.33 s | 6.89 s | `bfs` 5.2× / 10.5× faster |
+| `reachable_set` (10.0M rows) | 7.49 s, 13.4M edges/s | 5.27 s, 19.0M edges/s | 1.42 |
+| `components` | 10.58 s, 2.49 GB | 5.19 s, 1.54 GB | 2.04 |
+| `pagerank`, 20 iterations (warm / cold) | 15.9 / 17.1 s, 2.50 GB | 12.8 / 14.4 s, 2.27 GB | 1.25 |
+| `eigen`, 200 steps | 127.4 s, 4.55 GB | 114.4 s, 4.31 GB | 1.11 |
+| `diff` / `distance` edit / degree | 21.0 / 0.83 / 0.50 s | 18.7 / 0.69 / 0.36 s | 1.12 |
+| `eigenvector_centrality` / `katz_centrality` / `hits`, 20 iterations | 15.2 / 11.2 / 17.5 s | 12.4 / 8.0 / 14.2 s | 1.2–1.4 |
+| `clustering` / `core_number` / `strong_components_vector` | 44.5 / 20.2 / 16.5 s | 41.8 / 19.0 / 16.3 s | 1.0–1.06 |
+| `minimum_spanning_forest` (10.0M rows) | 51.9 s, 3.55 GB | 48.2 s, 1.92 GB | 1.08 time, 1.85 peak |
+| betweenness / closeness / harmonic, `k := 16` | 58.9 / 5.5 / 5.4 s | 56.5 / 5.4 / 5.3 s | 1.0–1.04 |
+| `most_similar` / `similarity` of 100,000 pairs / `similar_pairs` | 1 ms / 2.15 s / 10.4 s | 1 ms / 0.47 s / 9.7 s | `similarity` 4.6 |
+| `louvain` | 544.6 s, 6.66 GB, modularity 0.1209, 5 levels, 16 communities | 528.2 s, 6.42 GB, 0.1209, 5 levels, 17 communities | 1.03 |
+| `leiden` | 811.8 s, 8.43 GB, modularity 0.1871, 16 levels, 16 communities | 803.7 s, 8.10 GB, 0.1864, 16 levels, 15 communities | 1.01 |
+
+- *Size and build.* The text version is 28% larger, its id map the IRIs' bytes plus offsets and order; cached, 20% larger, since it builds no dense-id direction when it loads. The build costs 31–39% more (the half-scale 35%) and peaks 0.25 GB higher, the larger id map it holds. Every stage that reads the version peaks about 0.24 GB higher, the difference in the versions' cached sizes.
+- *Where text costs.* Calls that emit a row per reached node pay for each id's `text` datum (built and checked against the encoding): `bfs` and `components` take twice as long, `reachable_set` 42% longer. `similarity`'s 100,000 pairs pass 65,536 lookups, so it builds its per-call index (80 MB at 10M nodes, the 8 bytes a node of §5.7) and maps each pair through it, 1.7 s more. Kernels that map no ids are within 12% (`pagerank`, `eigen`) or unchanged (the library, the community detections, which return a modularity).
+- *Why `components` and `minimum_spanning_forest` peak higher.* Both build all their rows before they return the first: a `Vec` of `(AnyElement, …)` whose text datums are each `palloc`ed (`cstring_to_text_with_len`) in the set-returning function's multi-call memory context, which lives until the call ends. An IRI here is 33–36 bytes and its 4-byte varlena header, so AllocSet rounds it to a 64-byte chunk plus its 8-byte header: 72 bytes an id. Measured past each call's starting RssAnon, `components` holds 1,031 / 314 MB (717 MB more for 10M ids: 72 bytes each) and `minimum_spanning_forest` 2,030 / 646 MB (1,384 MB more for 20.0M ids: 69 bytes each); at half scale 78 and 68 bytes. With `bigint` ids the datum is by value and only the `Vec` is held (24 and 40 bytes a row). *The fix* (not built here): return the rows lazily, a `TableIterator` over the kernel's result (the labels, 4 bytes a node, or the forest's edges) and the loaded version, which makes each row's external ids in `next()`. pgrx calls `next()` after the first row in the caller's memory context, which for a function in `FROM` is the per-tuple context that `ExecMakeTableFunctionResult` resets before each call, and the row is then copied into the tuplestore (bounded by `work_mem`, then spilled). The datums would then live one row each, and no `Vec` of rows would be held: here the text peaks would fall by about 1.0 and 1.8 GB (the datums and the `Vec`) and the `bigint` ones by about 0.24 and 0.40 GB (the `Vec`), which leaves the kernel's result (4 bytes a node, or 16 bytes a forest edge). Other functions that build every row up front before they return one could be made lazy the same way.
+- *Disk.* The community detections wrote, text / `bigint`: `louvain` 4.31 / 4.64 GB of segments and 6.14 / 6.14 GB of WAL, `leiden` 9.51 / 13.10 GB and 15.09 / 15.06 GB (the harness drops the edge table and the first version first), within the disk.
+
+At full scale every §12 target was met for both id types except `reachable_set` (13 / 19M edges/s, against 50M) and `bfs` against the recursive CTE (5.2 / 10.5×, against 20×); `components` (10.6 s) also misses its 10 s target with text ids only, which the lazy rows above would not change (it is time, not memory: the datums are built either way).
+
 **Parallel query (M9, §5.6 step 9, §14 item 33; `bench/parallel.py`, `bench/results/parallel/`).** A probe table of node pairs (half an edge's endpoints, half random), `SELECT count(*) FROM probe WHERE warren.has_edge('g1', src, dst)`, `sum(warren.degree('g1', src))` and `count(*)` over `LATERAL warren.neighbours('g1', src)`, under `EXPLAIN (ANALYZE, TIMING OFF)` in a session whose leader has already read the graph, with the graph reads `PARALLEL RESTRICTED` (0.4.0's label) and `PARALLEL SAFE` (set by `ALTER FUNCTION` on the same library, a release build of cf9dce1), on PG 16.15 preloaded (the shared cache) and without the preload (every backend and worker loads privately), on the soak's machine (4 vCPUs). With workers, the planner's parallel costs are set to 0 ("forced"); "planned" is the planner's own choice at its default costs and `max_parallel_workers_per_gather = 4`. Medians of three runs (two at full scale), ms:
 
 | Case a, probe rows, read | Restricted, any | Safe, 0 workers | Safe, 2 forced | Safe, 4 forced | Safe, planned |

bench/results/soak/README.md

@@ -374,3 +374,21 @@ Half scale, without the recursive CTE baseline, because the machine had
 about 8 GB of disk free, shared with other work; the full-scale text run
 needs about 16 GB (DESIGN.md §12). Both private runs skipped their second
 version for the disk floor, so `diff` and `distance` were not run.
+
+## Text node ids at full scale (M10 step 4, §5.7)
+
+`a-text-m10.json` then `a-m10.json`: case a at full scale (10M nodes,
+100M edges) with IRIs and with `bigint` ids, every stage included (the
+recursive CTE baseline, the second version with `diff` and `distance`,
+the library and both community detections). PostgreSQL 16.15 from a
+private tree, a release build of 7934941 (`pg_warren.so` sha256
+a935ebf8…), harness 7934941 (clean), the same 4-vCPU Xeon, one after the
+other on 2026-10-23 with no other agent building: load average 1.0–1.5
+at each run's start and end, the soak's own. About 20 GB of disk was
+free at the start, and neither run came within the floor.
+
+    bench/soak.py a --ids text --label=-m10 --no-eigen-steps --eigen-max-iter 200 \
+        --disk-floor 3221225472 \
+        --pg-config /var/tmp/m10-pgtree/usr/lib/postgresql/16/bin/pg_config --parent /var/tmp/m10-soak \
+        --extension-revision 7934941 --extension-profile release --threads 4 --statement-timeout 20min
+    bench/soak.py a --ids bigint --label=-m10 …   # otherwise as above