Skip to content

Pooler soak: warm backends first; judge progress rows that outlive their call - #91

Merged
wfaithfull merged 1 commit into
mainfrom
soak-flake
Oct 26, 2026
Merged

wfaithfull merged 1 commit into
mainfrom
soak-flake

Conversation

@wfaithfull

Copy link
Copy Markdown
Contributor

The CI pooler soak failed twice today on diffs that couldn't cause it: run 40372279107 on #89 and run 40390992808 on #90, which changes only docs. Neither failure is a leak in pg_warren. Both come from how the soak measures, so this PR changes the harness only; no Rust changes.

progress_rows (#90's failure). A row in warren.progress() is a call that is still running:

  • its operation frame is an RAII guard that is cleared even when an ERROR unwinds the call;
  • its command frame is cleared by reset callbacks on the message and transaction contexts;
  • the slot is freed when the backend exits.

Checked on a live cluster:

  • after a timeout, a success, or a cancel, the row is gone at once;
  • a backend whose client was killed keeps its row only while it is still running the call.

So the count measures load (about 5 calls in flight per sample), and judging its trend flagged random noise. Shuffling each run's samples in time to remove any trend, the old bound still flagged growth in 0.1–5% of shuffles across 11 runs.

  • The soak now judges outlived_progress_rows with zero allowed: rows on a backend that is not active, whose command_start is before its state_change.
  • progress_rows is still sampled but no longer judged.
  • The settle check also fails if a row outlives its call after the clients stop.

One backend's anonymous memory (#89's first run). Anonymous memory is a high-water mark. It steps up the first few times a backend runs each kind of write: about 4 MB, then 1–2 MB, then flat. The single writer lands wherever the pooler sends it, so a backend often got its first writes inside the 85 s window.

  • Measured directly, reads grow 0.01–0.13 KiB per call, and 400 writes on one backend stay flat.
  • warm.py now holds each pool's 8 server connections and runs every kind of call on each, three times, before the clock starts.
  • In 6 local runs of the old code, every run had a near miss. In 7 runs with the warming, the most any starting backend grew was 0.1–1.1 MB.
  • The allowance, and the rule that a single backend can fail on its own, are unchanged: a leak in the write path would show only on the writer's backend.

What's left: a backend the pool opens mid-run, after chaos breaks a connection, still starts cold. In the warmed runs such backends grew at most 2.1 MB against an allowance of about 4.5 MB.

Docs: DESIGN §13's soak paragraph is updated with the above.

Tests:

  • New tests in test/tests/test_pooler_soak_{verdict,settled,warm}.py, each seen failing first.
  • The harness pytest suite passes (230 tests).
  • 7 local soak runs on PG 16 all pass.

Generated by Claude Code

…ogress rows

The pooler soak failed now and then on unrelated changes (CI runs
40372279107 and 40390992808). Neither was a leak:

- A backend's anonymous memory is a high-water mark: it rises in steps
  the first few times the backend runs each kind of call (a first
  refresh about 4 MB) and then stays flat. Only one client in 48 writes,
  and the pooler decides which backend serves it, so a backend often
  first served a write inside the 85 s steady window. Every local run
  had one to three backends grow 3.5-6.8 MB that way. warm.py now pins
  each pool's server connections with open transactions and runs every
  kind of call on each of them three times before the clock starts.
  After that, the most any backend grew in 7 runs was 0.1-1.1 MB.
- progress_rows counts the calls in flight when a sample is taken. It
  measures load, not a leak, and with no trend its slope went over 2
  in up to 1 run in 20. A frame clears when its call, statement or
  transaction ends (checked after a cancel, a statement timeout in a
  transaction, idle in a transaction, and a client killed in a call).
  So progress rows are now reported only. What is judged is a progress
  row that outlived its call, meaning one whose command began before
  its backend last went idle. None is allowed in any sample, nor once
  the clients stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
@wfaithfull
wfaithfull merged commit 6771e70 into main Oct 26, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants