Skip to content

Crash recovery test: count the source by sorting; report a failed count - #92

Merged
wfaithfull merged 1 commit into
mainfrom
crash-recovery-flake
Oct 26, 2026
Merged

wfaithfull merged 1 commit into
mainfrom
crash-recovery-flake

Conversation

@wfaithfull

Copy link
Copy Markdown
Contributor

This fixes the crash-recovery test, which failed on #90 (a docs-only PR) in CI run 40406996012. It reported FAIL a refresh after terminate, with the source has []. The refresh itself had worked.

Cause: the test checks the result by counting the source's distinct edges, SELECT DISTINCT src, dst FROM public.edges over 2M rows. That count sometimes ran for minutes inside Postgres and hit the 120 s statement timeout. The test read the failed query as an empty source.

  • When the count is slow, Postgres runs it as a parallel hash aggregate. It estimates 40,000 groups, because the tables are never analysed (autovacuum is off), against 200,000 real ones. It then spills into about 172,000 batches.
  • It isn't pg_warren. The same query on plain clusters, with no extension and nothing preloaded, with a 15 s timeout:
    • PG 16: 1 run in 60 was slow.
    • PG 18: 6 in 60 were slow.
    • PG 18 under load: 21 in 80 were slow.
    • PG 16 with enable_hashagg = off: none in 100, the longest 0.4 s.
  • Caught live, the slow backend was CPU-bound in the aggregate. Nothing was waiting on a lock, nothing was left of the terminated backend, and pg_warren's threads were idle.
  • The version numbers in CI (2 → 34 → 100) are sequence values skipping ahead after recovery, as usual.

Changes:

  • count_source turns off hash aggregation for the count, so Postgres sorts instead. The check is unchanged.
  • A failed count is now reported with its SQLSTATE and message, not as an empty source.
  • Three pytest tests use a fake session. The two new checks were seen failing first, the timeout one with exactly CI's message.
  • DESIGN §13's crash-recovery entry gets one clause on why the count sorts.

Runs:

  • Before the fix, 1 of 3 preloaded runs failed on PG 16.
  • After it, 15 of 15 preloaded runs passed on PG 16, while 21 of 80 runs of the old query were slow on PG 18 alongside them.
  • One more full default run, preloaded and then not, also passed.
  • No Rust changes.

Generated by Claude Code

"a refresh after terminate" failed in CI run 40406996012 (PG 16,
preload) with "the source has []": the refresh was fine, but the test's
own oracle query, a parallel DISTINCT over the 2M unanalysed edge rows,
ran into the session's 120 s statement_timeout (57014), and the check
compared against the failed outcome's empty rows.

The slow query is Postgres's, not pg_warren's: now and then the final
HashAggregate above Gather spills into some 10^5 batches (EXPLAIN
ANALYZE: Batches 172393, 140-170 s for 200000 groups). It reproduces in
a cluster without pg_warren on PG 16 (1 in 60 runs) and PG 18 (6 in 60;
21 in 80 under load). With enable_hashagg off the same count sorts and
took at most 0.4 s in 100 runs.

The source is now counted with hash aggregation off, and a failed count
is reported with its SQLSTATE and message instead of as an empty source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
@wfaithfull
wfaithfull merged commit 85505a9 into main Oct 26, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants