diff --git a/DESIGN.md b/DESIGN.md index 7cc38bd7..01a1fba1 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -1892,7 +1892,7 @@ Test it by installing as a non-superuser in CI and running the suite. **Make the - Not yet run: the large tier. *Deferred (owner, 2026-10-25):* the large tier is deferred to the end of M11, not dropped; keepset's fetch holds a dataset in memory, so the large datasets don't download on this machine. They are performance runs, not on the critical path. - A head-to-head with igraph and Neo4j (with and without GDS) on the same machine, on graphs from the same suite, published with its caveats. *Done (M11):* `suite.py head-to-head` (engines behind a port, the runner tested against a fake engine), run on the medium tier and four tiny graphs; results, caveats and where pg_warren loses in §12 *Head-to-head* and `bench/results/corpus/head_to_head/`. The large tier's head-to-head waits on the large tier. - Robustness: - - a soak of several days behind a connection pooler, watching for memory growth. *The harness and CI are done; the multi-day run is not* (`test/pooler_soak.py`, `test/harness/pooler_soak/`; CI on PG 16, preloaded, 2 min, about 2.5 min with setup). PgBouncer in transaction and in session pooling at once, each a login role in the graphs' owner role with 48 clients over 8 server connections (a pool hands out its last released connection first, so a few backends see most clients); the clients mix traversals on a `bigint` and a `text` graph, `last_stats()` in the traversal's transaction, transient vectors on a version named by id, persisted vectors written, read and dropped, prepared statements (SQL `PREPARE` in session pooling, the protocol's unnamed statement in transaction pooling), introspection and calls cut short by `statement_timeout`; writers edit the source and refresh, stage then promote or drop, and compact, so versions retire under the readers; a chaos thread cancels calls and makes clients disconnect idle in a transaction or in a call. Sampled: each pooled backend's `smaps_rollup`, its memory contexts' total (`pg_log_backend_memory_contexts`, read back from the log) and its `backends()` private and transient bytes; the shared cache, the DSM segments the cluster maps, the activity and progress rows, the versions, segments, vectors and chunks and the catalog tables' sizes; and the poolers' `SHOW` counters. A series *grew* when its robust slope (Theil–Sen) over the steady window, after the warm-up, and over each half of it, adds more than its tolerance (an absolute amount plus a fraction of its level; it does not scale with time, so a longer run is stricter); a backend's warm-up starts at its first sample, and each pool is also judged by the median of its backends' slopes, since pools replace the server connections clients broke. It also fails on an SQLSTATE an operation does not allow, on anything left once clients and then poolers stop (activity and progress rows, pins, staged versions, orphans, retention, the DSM segments, and, after `evict`, the shared bytes), on session state that session pooling loses or leaks, and on trouble in the server log; it writes `samples.csv` as it goes and `summary.json`. It probes what each mode keeps of a session: transaction pooling loses a `SET`, a transient vector, a SQL prepared statement and `last_stats()` from one transaction to the next and leaks them to other clients, and session pooling carries them and, through `DISCARD ALL`, leaks nothing (docs/limits.md, "Behind a connection pooler"). *Runs:* the CI mode on PG 16 and 18 (debug builds, 2 min); and 2.5 h on PG 16 (debug, 64 clients per pool, 0.2 s think time, a 30-minute warm-up): 7.0M statements and transactions, every failure an allowed one, and over the 7,140 s steady window the median pooled backend's anonymous memory grew 0.26 MB/h (session) and 0.40 MB/h (transaction) at 15.5 MB, its memory contexts and pg_warren's private and transient bytes were flat, and the shared cache, DSM segments, versions (5), segments (45), vectors and the catalog's sizes were flat or fell. *Found and fixed:* `DISCARD ALL` left a backend's transient vectors and `last_stats()` (§6.2), so in session pooling the next client could read them and they stayed until the backend exited (about 10 MB per session backend in the first runs); and a persisted vector dropped on another backend stayed in the cache of the backend that wrote it whenever the shared cache served that backend's reads, since only a private load pruned (§5.5), so a transaction-pooled backend gained about 8 MB a minute, up to `warren.max_cache_memory`. The harness's own clients first left a persisted vector behind when a cancel kept a persist from its drop or a compaction moved the vector to the compacted version (§7), each keeping its version from retention; an application that drops a vector by the version it wrote has the same hazard. *Left:* the multi-day run. A GitHub-hosted job stops after 6 hours, so it needs a machine of our own: a self-hosted runner, or the owner's machine, running `--hours 48` (or longer) on a release build in a private tree with `--no-lock`, its `summary.json` and the slopes recorded here. Chained 6-hour scheduled jobs on hosted runners would not be one run: each starts a new cluster and new backends, so no backend lives long enough to show a slow leak; + - a soak of several days behind a connection pooler, watching for memory growth. *The harness and CI are done; the multi-day run is not* (`test/pooler_soak.py`, `test/harness/pooler_soak/`; CI on PG 16, preloaded, 2 min, about 2.5 min with setup). PgBouncer in transaction and in session pooling at once, each a login role in the graphs' owner role with 48 clients over 8 server connections (a pool hands out its last released connection first, so a few backends see most clients); the clients mix traversals on a `bigint` and a `text` graph, `last_stats()` in the traversal's transaction, transient vectors on a version named by id, persisted vectors written, read and dropped, prepared statements (SQL `PREPARE` in session pooling, the protocol's unnamed statement in transaction pooling), introspection and calls cut short by `statement_timeout`; writers edit the source and refresh, stage then promote or drop, and compact, so versions retire under the readers; a chaos thread cancels calls and makes clients disconnect idle in a transaction or in a call. Sampled: each pooled backend's `smaps_rollup`, its memory contexts' total (`pg_log_backend_memory_contexts`, read back from the log) and its `backends()` private and transient bytes; the shared cache, the DSM segments the cluster maps, the activity rows, the progress rows and those that outlived their call (a row whose command began before its backend last went idle), the versions, segments, vectors and chunks and the catalog tables' sizes; and the poolers' `SHOW` counters. A series *grew* when its robust slope (Theil–Sen) over the steady window, after the warm-up, and over each half of it, adds more than its tolerance (an absolute amount plus a fraction of its level; it does not scale with time, so a longer run is stricter); a backend's warm-up starts at its first sample, and each pool is also judged by the median of its backends' slopes, since pools replace the server connections clients broke. Before the clock starts, every server connection of each pool, pinned by an open transaction, runs every kind of call the clients make three times (`warm.py`): a backend's anonymous memory is a high-water mark that rises in steps the first few times it runs each kind of call (a first `refresh` about 4 MB, the next ones less) and is then flat, and only one client in 48 writes, so without it a backend first served the writer well into the steady window and its warm-up read as growth (CI runs 40372279107, a session backend, 6.0 MB against 5.2 allowed; in 6 local runs (PG 16 and 18) the most a backend grew was 3.5–6.8 MB, in 7 runs after warming 0.1–1.1 MB). A backend the pool opens later, in place of one a client broke, starts cold and keeps its own warm-up. The progress rows are reported, not judged: they count the calls in flight as each sample is taken, about 5, a gauge of the load whose slope over 18 samples exceeded a bound of 2 in up to one run in twenty with no trend (0.1–5% across 11 local runs, by shuffling each run's samples; CI run 40390992808); a progress row that outlives its call is what would leak, and none is allowed, in a sample or once the clients stop. It also fails on an SQLSTATE an operation does not allow, on anything left once clients and then poolers stop (a progress row that outlived its call, activity and progress rows, pins, staged versions, orphans, retention, the DSM segments, and, after `evict`, the shared bytes), on session state that session pooling loses or leaks, and on trouble in the server log; it writes `samples.csv` as it goes and `summary.json`. It probes what each mode keeps of a session: transaction pooling loses a `SET`, a transient vector, a SQL prepared statement and `last_stats()` from one transaction to the next and leaks them to other clients, and session pooling carries them and, through `DISCARD ALL`, leaks nothing (docs/limits.md, "Behind a connection pooler"). *Runs:* the CI mode on PG 16 and 18 (debug builds, 2 min); and 2.5 h on PG 16 (debug, 64 clients per pool, 0.2 s think time, a 30-minute warm-up): 7.0M statements and transactions, every failure an allowed one, and over the 7,140 s steady window the median pooled backend's anonymous memory grew 0.26 MB/h (session) and 0.40 MB/h (transaction) at 15.5 MB, its memory contexts and pg_warren's private and transient bytes were flat, and the shared cache, DSM segments, versions (5), segments (45), vectors and the catalog's sizes were flat or fell. *Found and fixed:* `DISCARD ALL` left a backend's transient vectors and `last_stats()` (§6.2), so in session pooling the next client could read them and they stayed until the backend exited (about 10 MB per session backend in the first runs); and a persisted vector dropped on another backend stayed in the cache of the backend that wrote it whenever the shared cache served that backend's reads, since only a private load pruned (§5.5), so a transaction-pooled backend gained about 8 MB a minute, up to `warren.max_cache_memory`. The harness's own clients first left a persisted vector behind when a cancel kept a persist from its drop or a compaction moved the vector to the compacted version (§7), each keeping its version from retention; an application that drops a vector by the version it wrote has the same hazard. *Left:* the multi-day run. A GitHub-hosted job stops after 6 hours, so it needs a machine of our own: a self-hosted runner, or the owner's machine, running `--hours 48` (or longer) on a release build in a private tree with `--no-lock`, its `summary.json` and the slopes recorded here. Chained 6-hour scheduled jobs on hosted runners would not be one run: each starts a new cluster and new backends, so no backend lives long enough to show a slow leak; - many backends refreshing and querying one graph at once. *Done* (`test/concurrent_refresh.py`, CI on PG 16, 20 s preloaded and 20 s at the default configuration): 8 readers (`reachable_set`, `bfs`, `neighbours`, `dijkstra`, `walk`, `degree`, on the current version or one named by id; `spmv` and `pagerank` persisted, read back and dropped; `cache()`, `shared_cache()`, `backends()`, `progress()`), 3 refreshers (source edits, `refresh` alone and in the edit's transaction, rolled back, staged then promoted or dropped, `compact`; one with a `lock_timeout`) and a backend that writes and drops persisted vectors. The source moves through numbered generations, each a deterministic edit of the last, and the graph's queries add a marker node that names the generation, so every read is checked against an oracle of the generation its version was built from, read in the same statement. Checked: every read matches some complete version; every failure is a documented one, by SQLSTATE and message (a version retired after its id was read, 42704; a lock timeout, 55P03; a staged version superseded by a compaction, 55000; a vector moved by a compaction meanwhile, 42704 or 40001); afterwards no backend left in `backends()`, nothing pinned or mapped in the shared cache and, after `evict`, no more shared bytes than at the start, a fresh backend's private memory 0, the current version the one the last committed promotion made, no staged version, vector or partial segment left, every remaining version matching its generation, retention back to `retain_versions` after a refresh, and nothing in the server log but those SQLSTATEs. *Found and fixed:* a vector written to a version retention deleted meanwhile failed with a raw foreign-key violation (now 42704, §7); `compact` failed with `data_corrupted` when a vector it was moving was dropped meanwhile (now skipped, §5.2); and, preloaded, a read that revived a shared version another backend was evicting failed with an internal error, "cannot pin a segment that is already pinned" (the segment is now unpinned under the registry's lock, §5.6). 240 s preloaded and 45 s in each mode then ran clean; - physical streaming replicas, which §5.1 claims and nothing tested. *Done* (`test/replicas.py`, CI on PG 16, preloaded and at the default configuration, about 25 s; also run on PG 18): a primary, a hot standby made by `pg_basebackup -R` and a cascading standby of the standby. On the primary: graphs with `bigint` and `text` ids, weighted, several relations, directed and undirected; staged, promoted, dropped and compacted versions; persisted, dropped and moved vectors (`pagerank`, `spmv`, `eigen`, a matrix), Louvain, a contraction and a projection; an owner, a reader and a quota. On each standby, once it has replayed the primary's WAL (replay LSN at least the primary's flush LSN): every read gives the same rows as on the primary (traversals, `walk`, `dijkstra`, `components`, vector and matrix reads, `diff`, `distance`, `modularity`, similarity, `info`, and the compute that persists nothing: transient vectors from every function that has `persist`, `gather`, `eigen` without stored eigenvectors), a reader reads and a stranger is refused (42501); the server's own `cache()`, `shared_cache()`, `backends()`, `progress()`, `report_progress`, `prewarm`, `evict` and `last_stats` work; two backends share a version where the cluster shares, and `evict` leaves nothing pinned; every function that writes (a test fails on any function of the SQL surface that is neither read nor refused) is refused with 25006 and pg_warren's message. A version retired on the primary (retention after a refresh, then `VACUUM`) while one standby session holds it in an open REPEATABLE READ transaction and another reads it in a running statement: each read is the old version's answer or cancelled by the recovery conflict (40001), with and without `hot_standby_feedback`; once replayed, a new statement does not find it (42704) and reads the new one as the primary does. Then the standby is promoted: its graphs and vectors read as the primary's, it refreshes and persists a vector, and the cascade follows it. The server logs hold only refusals, the stranger's 42501, the 42704, recovery conflicts (40001, 25P02) and walreceivers ended at promotion or shutdown (57P01). *Found:* every write on a standby failed with an internal error (XX000, "cannot assign TransactionIds during recovery"); now 25006 (§5.1, *Standbys*). §5.1 now says what a standby can and cannot do; - crash recovery during `refresh`. *Done* (`test/crash_recovery.py`, CI on PG 16, preloaded and at the default configuration, about 75 s): a refresh of a 200K-node, 2M-edge graph is stopped waiting to read its edges (its edge query takes a shared advisory lock the test holds), while it writes its segments (the segments table has grown past 1 MB; where another backend's progress shows, its phase is checked to be `building` or `writing`) and after it returned, before its commit; each by `pg_terminate_backend`, by SIGKILL of the backend (the postmaster reinitialises shared memory, the registry with it) and by an immediate shutdown and restart (WAL recovery). No test-only hook was needed. A version is built in its refresh's transaction (§5.1), so a staged version left by a crash is rolled back with it, and nothing needs cleaning up. Checked after each: the catalog (versions, segments by version and name, vectors and their chunks, the current version) and the graph's answers exactly as before; the interrupted version not recorded; no row of the stopped backend in `backends()` or `progress()`, and nothing pinned; the shared cache loads the current version again and another backend reads it from there; the server log holds only what the interruption explains (57P01, 57P02, 57P03, the killed backend's signal 9); and after each way, a refresh succeeds and builds the source. Found nothing; diff --git a/test/harness/pooler_soak/samples.py b/test/harness/pooler_soak/samples.py index 643fcea4..7f5654df 100644 --- a/test/harness/pooler_soak/samples.py +++ b/test/harness/pooler_soak/samples.py @@ -4,7 +4,8 @@ context totals `pg_log_backend_memory_contexts` writes to the server log), pg_warren's own accounting (`warren.backends()`'s private, transient and mapped bytes per backend; `warren.shared_cache()`), the -activity and progress rows, the DSM segments the cluster's processes map, +activity and progress rows (and the progress rows that outlived their +call: `OUTLIVED_PROGRESS_ROWS`), the DSM segments the cluster's processes map, the catalog's rows and table sizes, and each pooler's `SHOW POOLS` and `SHOW STATS`. @@ -25,9 +26,19 @@ "coalesce(b.mapped_bytes, 0) FROM pg_stat_activity a " "LEFT JOIN warren.backends() b ON b.pid = a.pid AND b.kind = 'backend' " "WHERE a.backend_type = 'client backend' AND a.usename = ANY ('{{{roles}}}')") +# A progress row is a call in progress: its frames end with the call, or +# with the statement or transaction around it (DESIGN.md §5.7). One that +# outlived its call is on a backend that has since gone idle: its command +# began before the backend's state last changed, to a state other than +# running a statement. A call that begins after `pg_stat_activity`'s +# snapshot shows the backend idle began after that change, so a row the +# two reads see apart is not counted. +OUTLIVED_PROGRESS_ROWS = ("SELECT count(*) FROM warren.progress() p JOIN pg_stat_activity a USING (pid) " + "WHERE a.state <> 'active' AND p.command_start < a.state_change") CLUSTER_METRICS = { "activity_rows": "SELECT count(*) FROM warren.backends() WHERE kind = 'backend'", "progress_rows": "SELECT count(*) FROM warren.progress()", + "outlived_progress_rows": OUTLIVED_PROGRESS_ROWS, "shared_bytes": "SELECT coalesce(sum(bytes), 0) FROM warren.shared_cache() WHERE kind = 'cluster'", "shared_entries": "SELECT count(*) FROM warren.shared_cache() WHERE kind IN ('version', 'vector')", "shared_pinned": "SELECT count(*) FROM warren.shared_cache() WHERE kind IN ('version', 'vector') AND pins > 0", diff --git a/test/harness/pooler_soak/settled.py b/test/harness/pooler_soak/settled.py index 969ad902..f665aa1d 100644 --- a/test/harness/pooler_soak/settled.py +++ b/test/harness/pooler_soak/settled.py @@ -3,9 +3,11 @@ exit should have freed, and nothing in the catalog that leaked. `after_clients`, with the poolers still up and their server backends -idle: the session pool's backends hold no transient vector (its -`DISCARD ALL` reset each one when its client left); the transaction -pool's are reported (no reset runs between transactions). `after_poolers`, +idle: no progress row outlived its call (a call a client abandoned may +still run, until its backend sees the client gone); the session pool's +backends hold no transient vector (its `DISCARD ALL` reset each one when +its client left); the transaction pool's are reported (no reset runs +between transactions). `after_poolers`, once the poolers have closed their server connections: no activity row of a backend that has gone, no progress row, nothing pinned or mapped in the shared cache, no staged version, no segment or chunk without its @@ -18,15 +20,16 @@ import time from .fixture import GRAPHS +from .samples import OUTLIVED_PROGRESS_ROWS from .workload import PERSISTED_NAMES_PER_CLIENT SETTLE_SECONDS = 15 RETAIN = {"g": 3, "t": 2} -def until(admin, sql, seconds=SETTLE_SECONDS): - """Polls `sql`, a count, until it is 0 or `seconds` pass; its last value.""" - deadline = time.monotonic() + seconds +def until(admin, sql): + """Polls `sql`, a count, until it is 0 or `SETTLE_SECONDS` pass; its last value.""" + deadline = time.monotonic() + SETTLE_SECONDS while (value := admin.run(sql).value()) != "0" and time.monotonic() < deadline: time.sleep(0.2) return value @@ -35,6 +38,8 @@ def until(admin, sql, seconds=SETTLE_SECONDS): def after_clients(admin, roles): """Failures, and a report of the transient bytes the transaction pool's backends keep.""" failures, report = [], {} + if (left := until(admin, OUTLIVED_PROGRESS_ROWS)) != "0": + failures.append(f"{left} progress rows outlived their call once the clients stopped") for mode, role in roles.items(): sql = (f"SELECT coalesce(sum(b.transient_bytes), 0) FROM warren.backends() b JOIN pg_stat_activity a " f"USING (pid) WHERE b.kind = 'backend' AND a.usename = '{role}'") diff --git a/test/harness/pooler_soak/verdict.py b/test/harness/pooler_soak/verdict.py index 9f47469f..72801430 100644 --- a/test/harness/pooler_soak/verdict.py +++ b/test/harness/pooler_soak/verdict.py @@ -11,12 +11,15 @@ replaces its backends. A backend that a pool started later has a warm-up of its own, as long as the run's from its first sample. Per cluster (`CLUSTER_TOLERANCES`): the shared -cache's bytes and entries, the DSM segments mapped, the activity and -progress rows, the versions (promoted and staged), segments, vectors -and chunks, and the catalog tables' sizes. The tolerances leave room for -what varies without leaking: version sizes as the source changes, -staged versions in flight, the dead tuples autovacuum has not yet -reclaimed. Other series (RSS, PSS, the pools' counters) are reported.""" +cache's bytes and entries, the DSM segments mapped, the activity rows, +the progress rows that outlived their call (none allowed), the versions +(promoted and staged), segments, vectors and chunks, and the catalog +tables' sizes. The tolerances leave room for what varies without +leaking: version sizes as the source changes, staged versions in +flight, the dead tuples autovacuum has not yet reclaimed. Other series +(RSS, PSS, the pools' counters, and all progress rows, which count the +calls in flight as a sample is taken, a gauge of the load) are +reported.""" import statistics from collections import defaultdict @@ -37,7 +40,7 @@ "dsm_segments": Tolerance(4, 0.25), "dsm_bytes": Tolerance(16 * MIB, 0.25), "activity_rows": Tolerance(4, 0), - "progress_rows": Tolerance(2, 0), + "outlived_progress_rows": Tolerance(0, 0), "promoted_versions": Tolerance(2, 0), "staged_versions": Tolerance(2, 0), "segments": Tolerance(8, 0.25), diff --git a/test/harness/pooler_soak/warm.py b/test/harness/pooler_soak/warm.py new file mode 100644 index 00000000..5da095ab --- /dev/null +++ b/test/harness/pooler_soak/warm.py @@ -0,0 +1,74 @@ +"""Warming each pool's server backends before the pooler soak's clock +starts (DESIGN.md §13 M11), so that the steady state begins at every +backend's working set. + +A backend's anonymous memory is a high-water mark: glibc keeps what a +call's peak took, so it rises in steps the first few times the backend +runs each kind of call (a refresh about 4 MB, then less) and then stays +flat. Only one client in a pool writes, and the pooler decides which +backend serves it (session pooling keeps it on one backend for its +connection, transaction pooling hands out its last released connection +first), so without this a backend may first serve a write well into the +steady window, and its warm-up would look like growth. `warm` opens as +many sessions as the pool has server connections, each pinned to its own +backend by an open transaction, and runs every kind of call on each, +`PASSES` times, before the transaction commits.""" + +import random + +from .fixture import edit_sql +from .workload import READS, traversal_sql + +PASSES = 3 + + +def calls(rng, nodes, name): + """One pass of every kind of call the soak's clients make, in one + transaction; `name` is the persisted vector it writes and drops.""" + graph_reads = [traversal_sql(graph, kind, rng.randint(1, nodes), rng.randint(1, nodes)) + for graph in ("g", "t") for kind in READS] + current = "(SELECT (warren.info('g') ->> 'current_version')::bigint)" + return [edit_sql(rng, nodes), "SELECT warren.refresh('g')", "SELECT warren.refresh('t')", + "SELECT warren.drop_version('g', warren.refresh('g', promote := false))", "SELECT warren.compact('g')", + *graph_reads, + f"SELECT warren.pagerank('g', '{name}', max_iter := 10, version := {current})", + f"SELECT warren.vector_reduce('g', '{name}', 'sum', {current})", + f"SELECT warren.drop_vector('g', '{name}', {current})", + f"SELECT warren.vector_const('g', '{name}_x', 1, persist := false, version := {current})", + f"SELECT warren.spmv('g', '{name}_x', '{name}_y', weighted := true, persist := false, " + f"version := {current})", + *(f"SELECT warren.drop_vector('g', '{name}_{v}', {current})" for v in "xy"), + "SELECT (SELECT count(*) FROM warren.cache()), (SELECT count(*) FROM warren.backends()), " + "(SELECT count(*) FROM warren.progress()), (SELECT count(*) FROM warren.shared_cache())"] + + +def warm(connect, pool_size, nodes, seed): + """Runs every kind of call `PASSES` times on each of a pool's + `pool_size` server backends, through sessions `connect()` opens; the + pids of the backends warmed.""" + rng = random.Random(f"{seed}-warm") + sessions = [] + try: + pids = set() + for _ in range(pool_size): + sessions.append(connect()) + _run(sessions[-1], "BEGIN") + pids.add(_run(sessions[-1], "SELECT pg_backend_pid()").value()) + if len(pids) != pool_size: + raise RuntimeError(f"warming: {len(pids)} server backends for {pool_size} sessions") + for number, session in enumerate(sessions): + for _ in range(PASSES): + for sql in calls(rng, nodes, f"warm{number}"): + _run(session, sql) + _run(session, "COMMIT") + return pids + finally: + for session in sessions: + session.close() + + +def _run(session, sql): + done = session.run(sql) + if not done.ok: + raise RuntimeError(f"warming: {sql[:200]}: {done.sqlstate} {done.message}") + return done diff --git a/test/pooler_soak.py b/test/pooler_soak.py index 76a9e250..3880831d 100644 --- a/test/pooler_soak.py +++ b/test/pooler_soak.py @@ -6,7 +6,9 @@ mode (default: transaction and session at once, each its own login role and pool of --pool-size server connections for --clients client connections). It first probes what each pool mode keeps of a client's -session state (harness/pooler_soak/probes.py), then runs the clients' +session state (harness/pooler_soak/probes.py), warms every server +connection of each pool with every kind of call +(harness/pooler_soak/warm.py), then runs the clients' mix of reads, compute, persisted vectors, introspection, timeouts, cancels, disconnects in transactions and writes that retire versions (harness/pooler_soak/workload.py) for --minutes or --hours, sampling @@ -35,12 +37,13 @@ import threading import time from contextlib import ExitStack, nullcontext +from functools import partial from pathlib import Path from harness import extension, server_log, shared_cache from harness.cluster import Cluster from harness.pooler import MODES, PGBOUNCER, Pooler -from harness.pooler_soak import probes, report, settled, verdict +from harness.pooler_soak import probes, report, settled, verdict, warm from harness.pooler_soak.fixture import DATABASE, GRAPHS, role, setup_sql from harness.pooler_soak.samples import Sampler from harness.pooler_soak.workload import Chaos, Tally, Workload @@ -147,6 +150,8 @@ def soak(args, out): roles[mode]) for i, mode in enumerate(args.modes)} sampler = Sampler(cluster, admin, pools, args.context_every) dsm_before = sampler.dsm_count() + for pooler, role_name in pools.values(): + warm.warm(partial(Session, pooler, DATABASE, user=role_name), args.pool_size, args.nodes, args.seed) rows = run_clients(args, cluster, pools, sampler, tally, out) failures, kept = settled.after_clients(admin, roles) failures += settled.after_poolers(admin, roles, args.clients * len(args.modes), dsm_before, diff --git a/test/tests/test_pooler_soak_settled.py b/test/tests/test_pooler_soak_settled.py new file mode 100644 index 00000000..eb92e50f --- /dev/null +++ b/test/tests/test_pooler_soak_settled.py @@ -0,0 +1,23 @@ +"""What the pooler soak checks once its clients have stopped.""" + +from harness.pooler_soak.samples import OUTLIVED_PROGRESS_ROWS +from harness.pooler_soak.settled import after_clients +from harness.session import Outcome + + +class Answers: + """Answers each statement with the value of the first key it contains, else 0.""" + + def __init__(self, values): + self.values = values + + def run(self, sql): + value = next((value for key, value in self.values.items() if key in sql), "0") + return Outcome([[value]], "00000", "") + + +def test_a_progress_row_left_once_the_clients_have_stopped_fails(monkeypatch): + monkeypatch.setattr("harness.pooler_soak.settled.SETTLE_SECONDS", 0) + failures, _ = after_clients(Answers({OUTLIVED_PROGRESS_ROWS: "2"}), {"transaction": "soak_transaction"}) + assert failures == ["2 progress rows outlived their call once the clients stopped"] + assert after_clients(Answers({}), {"transaction": "soak_transaction"}) == ([], {"transaction": 0}) diff --git a/test/tests/test_pooler_soak_verdict.py b/test/tests/test_pooler_soak_verdict.py index 7fcd0b8f..0e26964e 100644 --- a/test/tests/test_pooler_soak_verdict.py +++ b/test/tests/test_pooler_soak_verdict.py @@ -67,3 +67,22 @@ def test_a_backend_born_late_has_its_own_warm_up(): judged = {(j.metric, j.subject): j.growth for j in judge_rows(rows, warmup=60)} assert judged[("anonymous", "transaction/9")].points == 34, "its first 60 s are its warm-up" assert not judged[("anonymous", "transaction/9")].grew + + +def test_calls_in_flight_are_reported_not_judged(): + """`progress_rows` counts the calls running as each sample is taken: a + gauge of the load, around 5 calls in the CI soak, whose robust slope + over 18 samples exceeded a bound of 2 in up to one run in twenty + with no trend at all (CI run 40390992808: 2.6 over 85 s at a level of + 5, both halves over 2).""" + counts = [2, 5, 3, 6, 2, 4, 5, 4, 6, 3, 6, 5, 7, 4, 6, 5, 9, 8] + rows = [(30 + 5 * i, "progress_rows", "cluster", count) for i, count in enumerate(counts)] + assert [j.metric for j in judge_rows(rows, warmup=30)] == [] + + +def test_progress_rows_that_outlive_their_call_grow_from_the_first(): + left = [(t, "outlived_progress_rows", "cluster", t // 100) for t in range(0, 600, 10)] + none = [(t, "outlived_progress_rows", "cluster", 0) for t in range(0, 600, 10)] + assert ("outlived_progress_rows", "cluster") in names(judge_rows(left, warmup=60)) + assert names(judge_rows(none, warmup=60)) == set() + assert CLUSTER_TOLERANCES["outlived_progress_rows"].allowed(0) == 0 diff --git a/test/tests/test_pooler_soak_warm.py b/test/tests/test_pooler_soak_warm.py new file mode 100644 index 00000000..7a60d925 --- /dev/null +++ b/test/tests/test_pooler_soak_warm.py @@ -0,0 +1,50 @@ +"""How the pooler soak warms each pool's server backends before its clock starts.""" + +import itertools + +import pytest + +from harness.pooler_soak.warm import PASSES, warm +from harness.session import Outcome + + +class Backend: + """A pooled session pinned to one server backend by its open transaction.""" + + pids = itertools.count(100) + + def __init__(self, distinct=True): + self.pid = str(next(self.pids)) if distinct else "99" + self.ran, self.closed = [], False + + def run(self, sql): + self.ran.append(sql) + value = self.pid if sql == "SELECT pg_backend_pid()" else "7" + return Outcome([[value]], "00000", "") + + def close(self): + self.closed = True + + +def test_every_server_backend_runs_every_kind_of_call_in_its_own_transaction(): + backends = [] + + def connect(): + backends.append(Backend()) + return backends[-1] + + warmed = warm(connect, 3, nodes=100, seed=1) + assert warmed == {b.pid for b in backends} and len(backends) == 3 + for backend in backends: + assert backend.ran[:2] == ["BEGIN", "SELECT pg_backend_pid()"] and backend.ran[-1] == "COMMIT" + calls = " ".join(backend.ran) + for call in ("warren.refresh('g')", "warren.refresh('t')", "promote := false", "warren.drop_version", + "warren.compact('g')", "warren.pagerank", "warren.spmv", "warren.progress()"): + assert calls.count(call) == PASSES, call + assert calls.count("warren.bfs('g'") == calls.count("warren.bfs('t'") == PASSES + assert backend.closed + + +def test_a_pool_that_does_not_give_each_session_its_own_backend_is_refused(): + with pytest.raises(RuntimeError, match="2 server backends for 3 sessions"): + warm(iter([Backend(), Backend(distinct=False), Backend(distinct=False)]).__next__, 3, nodes=100, seed=1)