Skip to content

Collect per-relation and per-database vacuum statistics - #415

Open
Alena0704 wants to merge 3 commits into
OPENGPDB_STABLEfrom
vacuum-statistics
Open

Collect per-relation and per-database vacuum statistics#415
Alena0704 wants to merge 3 commits into
OPENGPDB_STABLEfrom
vacuum-statistics

Conversation

@Alena0704

Copy link
Copy Markdown
Contributor

Teach lazy vacuum to accumulate a set of counters for every vacuumed heap relation and index, and deliver them to the statistics collector with a new PGSTAT_MTYPE_VACSTATS message:

  • tuples_deleted and dead_tuples (dead but not yet removable);
  • the page counterparts: pages_deleted (heap truncation / index page deletion) and dead_pages;
  • pages_frozen and pages_all_visible set by vacuum, which show whether vacuum is actually freezing tuples and keeping the visibility map up to date;
  • wraparound_vacuum_count: runs that the freeze table age escalated to a full-table scan, which cannot skip pages through the visibility map. This release has no autovacuum for user tables, so a manual VACUUM is the only thing that ever advances relfrozenxid, and such an escalation is otherwise invisible. Note the freeze limits are clamped to their minimum values on a young cluster, making every relation match them; those runs are deliberately not reported as wraparound-driven ones;
  • total_time of the vacuum, in microseconds.

The rev_all_frozen_pages/rev_all_visible_pages counters track how quickly the work of vacuum is undone: pages that lost their all-frozen/all-visible status. They are counted when ordinary DML clears the all-visible bit and are delivered with the regular relation statistics (PgStat_TableCounts), so they live directly in the relation/database entries of the collector rather than in the vacuum counters structure. rev_all_frozen_pages always stays zero here because the visibility map has no all-frozen bit in this release.

The counters are aggregated per relation (tables and indexes) and per database in the statistics collector.
They are exposed by the new vacuum_stats contrib extension, so no system catalog changes are needed: pg_stat_vacuum_tables, pg_stat_vacuum_indexes and pg_stat_vacuum_database show the local node, and the gp_stat_vacuum_* counterparts collect the master and all segments.

@Alena0704
Alena0704 force-pushed the vacuum-statistics branch 5 times, most recently from 0612af9 to ebb78d2 Compare August 26, 2026 08:12
@Alena0704
Alena0704 marked this pull request as ready for review August 26, 2026 09:20
@leborchuk
leborchuk requested a lite review from Copilot August 26, 2026 13:46

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds end-to-end support for collecting per-relation (heap + index) and per-database vacuum counters in the stats collector via a new PGSTAT_MTYPE_VACSTATS message, and exposes them through a new vacuum_stats contrib extension (including Greenplum-wide gp_stat_vacuum_* views).

Changes:

  • Extend pgstat infrastructure to carry and accumulate new vacuum/run counters (including “revoked all-visible” page counters).
  • Teach lazy vacuum and heap DML paths to populate and report the new counters.
  • Add the contrib/vacuum_stats extension plus regression/TAP/isolation tests and wire them into CI.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/test/isolation/specs/vacuum-extending-in-repetable-read.spec New isolation test exercising dead-tuples vs deleted-tuples behavior under REPEATABLE READ snapshot.
src/test/isolation/isolation_schedule Adds the new isolation test to the schedule (but with a spelling typo in the test name).
src/test/isolation/expected/vacuum-extending-in-repetable-read.out Expected output for the new isolation test (currently appears inconsistent/malformed).
src/include/pgstat.h Adds PGSTAT_MTYPE_VACSTATS, PgStat_VacuumStats, revocation counters, and new reporting macros/API.
src/backend/postmaster/pgstat.c Implements VACSTATS message send/receive and accumulation into rel/db entries; initializes/reset paths for new counters.
src/backend/commands/vacuumlazy.c Tracks and reports new per-vacuum counters for heap and indexes; adds elapsed-time measurement and page counters.
src/backend/access/heap/heapam.c Increments “revoked all-visible” counters when heap DML clears visibility-map all-visible bits.
contrib/vacuum_stats/vacuum_stats.control New extension control file.
contrib/vacuum_stats/vacuum_stats.c C accessor functions for rel/db vacuum counters (and revocation counters).
contrib/vacuum_stats/vacuum_stats--1.0.sql Defines SQL functions and views (pg_stat_vacuum_*, gp_stat_vacuum_*) and grants.
contrib/vacuum_stats/sql/vacuum_stats.sql pg_regress smoke test for the extension and views.
contrib/vacuum_stats/expected/vacuum_stats.out Expected output for pg_regress smoke test.
contrib/vacuum_stats/t/001_vacuum_statistics.pl TAP test covering table/index/db scenarios plus restart/crash-reset behavior.
contrib/vacuum_stats/README.md Documentation for motivation, counters, tests, views, and usage.
contrib/vacuum_stats/Makefile Build + installcheck + TAP targets for the new extension.
contrib/vacuum_stats/.gitignore Ignores generated test directories.
contrib/Makefile Adds vacuum_stats to contrib build.
.github/workflows/build-gpdb.yml Adds isolation installcheck job and runs contrib/vacuum_stats installcheck + TAP in CI.
Suppressed comments (2)

src/test/isolation/expected/vacuum-extending-in-repetable-read.out:42

  • This stats row conflicts with the preceding wait condition (dead_tuples=100). After the first VACUUM while the REPEATABLE READ snapshot is held, dead_tuples should be 100 and tuples_deleted should still be 0; the expected output currently shows the opposite and also appears to omit the pages_frozen column value. Please regenerate/fix the expected output for this step.
relname        tuples_deleted dead_tuples    pages_frozen   

test_vacuum_stat_isolation0              100            0              

src/test/isolation/expected/vacuum-extending-in-repetable-read.out:62

  • The final stats print also looks malformed (relname concatenated with the first numeric value and missing the pages_frozen column value). If the intent is to show the state after the second VACUUM, the row should have separate columns for relname, tuples_deleted, dead_tuples, and pages_frozen. Regenerate the expected output so it matches the query output exactly.
relname        tuples_deleted dead_tuples    pages_frozen   

test_vacuum_stat_isolation100            100            0              

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/test/isolation/expected/vacuum-extending-in-repetable-read.out Outdated
Comment thread src/test/isolation/isolation_schedule Outdated
Teach lazy vacuum to accumulate a set of counters for every vacuumed
heap relation and index, and deliver them to the statistics collector
with a new PGSTAT_MTYPE_VACSTATS message:

- tuples_deleted and dead_tuples (dead but not yet removable);
- the page counterparts: pages_deleted (heap truncation / index page
  deletion) and dead_pages;
- pages_frozen and pages_all_visible set by vacuum, which show whether
  vacuum is actually freezing tuples and keeping the visibility map up
  to date;
- wraparound_vacuum_count: runs that the freeze table age escalated to a
  full-table scan, which cannot skip pages through the visibility map.
  This release has no autovacuum for user tables, so a manual VACUUM is
  the only thing that ever advances relfrozenxid, and such an escalation
  is otherwise invisible.  Note the freeze limits are clamped to their
  minimum values on a young cluster, making every relation match them;
  those runs are deliberately not reported as wraparound-driven ones;
- total_time of the vacuum, in microseconds.

The rev_all_frozen_pages/rev_all_visible_pages counters track how
quickly the work of vacuum is undone: pages that lost their
all-frozen/all-visible status.  They are counted when ordinary DML
clears the all-visible bit and are delivered with the regular relation
statistics (PgStat_TableCounts), so they live directly in the
relation/database entries of the collector rather than in the vacuum
counters structure.  rev_all_frozen_pages always stays zero here
because the visibility map has no all-frozen bit in this release.

The counters are aggregated per relation (tables and indexes) and per
database in the statistics collector.
They are exposed by the new vacuum_stats contrib extension, so no
system catalog changes are needed: pg_stat_vacuum_tables,
pg_stat_vacuum_indexes and pg_stat_vacuum_database show the local
node, and the gp_stat_vacuum_* counterparts collect the master and all
segments.
Rename the isolation test from "repetable" to "repeatable" (the typo
came from the upstream patch) and update the schedule and the README.

Drop relname from the columns the test prints.  The isolationtester
formats every field with printf("%-15s"), which pads but never
truncates, so the 26-character relation name ran into the next column
and made the expected file look malformed; the relation is already
pinned by the WHERE clause, so printing it added nothing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants