Skip to content

feat: drill and restore compare every table's row count with the backup exactly (2.11.0) - #64

Merged
supertj merged 4 commits into
mainfrom
feature/exact-row-counts
Nov 7, 2026
Merged

supertj merged 4 commits into
mainfrom
feature/exact-row-counts

Conversation

@supertj

@supertj supertj commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

What

Drill and restore now compare every table's row count with the backup, exactly. One row more or less in any table fails the check row counts match the backup.

Before this, the manifest's row counts were n_live_tup planner estimates taken on a separate connection before pg_dump ran, so the drill could only check that tables which had rows didn't come back empty. Losing 10 of 10,000 rows passed.

How

  • The backup's count comes from the dump itself: pg_restore --data-only --use-list=<TABLE DATA entries only> --file=-, counting the lines of each COPY section (countRowsInDump / copyRowCounter in restore-engine.ts). The dump was taken by pg_dump in one consistent snapshot, so its row counts are exactly what the backup holds.
  • Only TABLE DATA entries are emitted, so text elsewhere in the dump (for example a sequence name containing a newline and COPY ... FROM stdin; inside a setval(...) literal) can't be mistaken for a data section.
  • The counter streams: it never holds a whole data line, only the first few characters. COPY headers may span lines inside quoted names and are read whole, up to 1 MB (a valid 1,600-column header can pass 200k characters; the shared limit in dump-sql.ts is raised to match).
  • Counts are keyed by schema and name (tableKey), not a dot-joined string, so public."a.b" and "public.a".b stay separate on both sides.
  • Nothing extra runs against the source database: no count(*) there, no extra connection, no dependency on exporting a snapshot through a pooler. The cost is one more read of the dump during the drill, run concurrently with the restored-side counts. If verification fails partway, the reader is cancelled and awaited before verifyRestored returns, so no pg_restore outlives the drill.
  • Existing snapshots get the check too, since it needs nothing new in the manifest.
  • Compared: ordinary tables, including partition leaves. Not compared, because the dump has no rows of their own or only part of them: materialized views (their data is a REFRESH), partitioned parents, and tables owned by an extension (CREATE EXTENSION adds its own rows, and the dump only holds the rows past the extension's filter). Those keep the existing non-empty check.
  • Restored-side counts for ordinary tables use count(*) from only: a legacy inheritance parent's plain count(*) includes its children's rows, while the dump holds the parent's own rows.
  • If the dump can't be counted (pg_restore fails, a temp dir can't be made, a COPY header it can't parse, output cut off mid-section) or a table has no data section in the dump, the check fails. It never falls back to the estimate, and counting never throws.
  • restore runs the same check through the shared verifyRestored.

Tests

  • test/copy-row-counter.test.js: counting across every chunk split, escaped \. values, column-less tables, multi-line quoted headers, huge rows without buffering, and refusing to count output it can't read; rowCountsCheck cases.
  • test/drill-row-counts.test.js (Docker): real dump with newlines, tabs and backslashes in values, legacy inheritance, partitions, a matview, a column-less table, public."a.b" next to "public.a".b, a sequence name carrying a fake COPY line, and a 1,600-column table. A full drill passes with every count matching; deleting rows and adding one to an inheritance child fail with the tables named. Cancelling returns null; a failed verification leaves no pg_restore running.
  • Full suite: 452 tests, 452 pass, 0 fail (npm test on 4cd60b9).
  • Reproducible drill (examples/reproducible-drill/run.sh --with-login) rerun with this build: row counts match the backup — all 12 tables match the backup exactly (1,476,250 rows) in both drill and restore; README updated from that run.

No preview environment for the CLI; verified by the tests and the reproducible drill above.

Version 2.11.0.

…up exactly (2.11.0)

The manifest's row counts are n_live_tup estimates taken before pg_dump, so
the drill could only check that tables with rows didn't come back empty.
The backup's exact count now comes from the dump itself: pg_restore
--data-only over the archive, counting each COPY section. Nothing extra
runs against the source database. One row more or less fails the check;
if the dump can't be counted, the check fails rather than falling back to
the estimate.
…tries, allow headers up to 1 MB

Dot-joined keys made public."a.b" and "public.a".b one table, so a row
missing from one could pass. setval and large-object SQL can hold a line
that starts with COPY inside a quoted name; pg_restore now writes only the
TABLE DATA entries for counting. A valid 1600-column COPY header can pass
100,000 characters.
@supertj
supertj merged commit 2343b0f into main Nov 7, 2026
1 check passed
@supertj
supertj deleted the feature/exact-row-counts branch November 7, 2026 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant