Repository navigation
Pooler soak: warm backends first; judge progress rows that outlive their call - #91
Merged
Merged
Conversation
…ogress rows The pooler soak failed now and then on unrelated changes (CI runs 40372279107 and 40390992808). Neither was a leak: - A backend's anonymous memory is a high-water mark: it rises in steps the first few times the backend runs each kind of call (a first refresh about 4 MB) and then stays flat. Only one client in 48 writes, and the pooler decides which backend serves it, so a backend often first served a write inside the 85 s steady window. Every local run had one to three backends grow 3.5-6.8 MB that way. warm.py now pins each pool's server connections with open transactions and runs every kind of call on each of them three times before the clock starts. After that, the most any backend grew in 7 runs was 0.1-1.1 MB. - progress_rows counts the calls in flight when a sample is taken. It measures load, not a leak, and with no trend its slope went over 2 in up to 1 run in 20. A frame clears when its call, statement or transaction ends (checked after a cancel, a statement timeout in a transaction, idle in a transaction, and a client killed in a call). So progress rows are now reported only. What is judged is a progress row that outlived its call, meaning one whose command began before its backend last went idle. None is allowed in any sample, nor once the clients stop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The CI pooler soak failed twice today on diffs that couldn't cause it: run 40372279107 on #89 and run 40390992808 on #90, which changes only docs. Neither failure is a leak in pg_warren. Both come from how the soak measures, so this PR changes the harness only; no Rust changes.
progress_rows(#90's failure). A row inwarren.progress()is a call that is still running:Checked on a live cluster:
So the count measures load (about 5 calls in flight per sample), and judging its trend flagged random noise. Shuffling each run's samples in time to remove any trend, the old bound still flagged growth in 0.1–5% of shuffles across 11 runs.
outlived_progress_rowswith zero allowed: rows on a backend that is notactive, whosecommand_startis before itsstate_change.progress_rowsis still sampled but no longer judged.One backend's anonymous memory (#89's first run). Anonymous memory is a high-water mark. It steps up the first few times a backend runs each kind of write: about 4 MB, then 1–2 MB, then flat. The single writer lands wherever the pooler sends it, so a backend often got its first writes inside the 85 s window.
warm.pynow holds each pool's 8 server connections and runs every kind of call on each, three times, before the clock starts.What's left: a backend the pool opens mid-run, after chaos breaks a connection, still starts cold. In the warmed runs such backends grew at most 2.1 MB against an allowance of about 4.5 MB.
Docs: DESIGN §13's soak paragraph is updated with the above.
Tests:
test/tests/test_pooler_soak_{verdict,settled,warm}.py, each seen failing first.Generated by Claude Code