Skip to content

bug: sanction checks scan every player_sanctions chunk - #421

Merged
lukepolo merged 1 commit into
mainfrom
perf/player-sanctions-drop-hypertable
Sep 19, 2026
Merged

lukepolo merged 1 commit into
mainfrom
perf/player-sanctions-drop-hypertable

Conversation

@lukepolo

Copy link
Copy Markdown
Contributor

player_sanctions is partitioned on created_at but only ever queried by player_steam_id, so chunk exclusion never prunes and each is_muted / is_gagged / is_banned call sequentially scans every chunk. There is no retention or compression policy on the other side of the trade, and the partitioning also forced the primary key from (id) to (id, created_at).

Converts it back to a plain table, restoring PRIMARY KEY (id).

player_sanctions is partitioned on created_at but only ever queried by
player_steam_id, so chunk exclusion never prunes and each is_muted /
is_gagged / is_banned call sequentially scans every chunk. There is no
retention or compression policy on the other side of the trade, and the
partitioning also forced the primary key from (id) to (id, created_at).

Converts it back to a plain table, restoring PRIMARY KEY (id).
@lukepolo

Copy link
Copy Markdown
Contributor Author

Correcting the premise here after profiling prod directly.

The 34-164x speedup I measured came from a sample database with 86 chunks on player_sanctions. Prod has 4 chunks, 10 rows, 40 kB, and the sanction checks already run at 4 buffers / 0.067 ms per call — the index is being used.

Those checks are still 4.5% of prod DB time, but purely from being called 380 million times in 31 hours, and this migration does not reduce call count. So it will not measurably help prod.

Still worth merging as correctness cleanup — a 10-row moderation table has no business being a hypertable, and it protects whichever instances have let chunks accumulate. Just shouldn't be merged expecting a prod win.

@lukepolo

Copy link
Copy Markdown
Contributor Author

Retracting my previous comment — it was wrong, and this PR is a significant prod win.

I dismissed it because prod's buffer counts looked fine (4 buffers/call). That was the wrong signal. The cost isn't data volume, it's that chunk exclusion can never prune: the table is partitioned on created_at but every sanction check filters on player_steam_id, so each call runs a ChunkAppend over a sequential scan of every chunk. Tiny chunks mean few buffers, which is exactly why the buffer count hid it.

Reproduced at prod's exact shape (PG 17.11, 10 sanction rows, 4 chunks, 477 players, index on player_steam_id present):

Custom Scan (ChunkAppend) on player_sanctions
  Chunks excluded during startup: 0
  ->  Seq Scan on _hyper_2_15_chunk
  ->  Seq Scan on _hyper_2_16_chunk
  ->  Seq Scan on _hyper_2_17_chunk
  ->  Seq Scan on _hyper_2_18_chunk
Execution Time: 0.066 ms

That 0.066 ms matches prod's measured 0.067 ms per call almost exactly, so the repro is faithful.

Same workload, hypertable vs plain:

hypertable (4 chunks) plain speedup
3 checks × 477 players 47.92 ms 2.81 ms 17.1x
1 check × 1 player 0.157 ms 0.047 ms 3.3x

The sanction checks are 11.2 hours of DB CPU over a 31-hour window on prod, so at 17x on the multi-player path this is worth roughly 8% of total database CPU. The commit message had it right the first time.

@lukepolo
lukepolo merged commit 03e7c15 into main Sep 19, 2026
2 checks passed
@lukepolo
lukepolo deleted the perf/player-sanctions-drop-hypertable branch September 19, 2026 20:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant