Repository navigation
feat: drill and restore compare every table's row count with the backup exactly (2.11.0) - #64
Merged
Merged
Conversation
…up exactly (2.11.0) The manifest's row counts are n_live_tup estimates taken before pg_dump, so the drill could only check that tables with rows didn't come back empty. The backup's exact count now comes from the dump itself: pg_restore --data-only over the archive, counting each COPY section. Nothing extra runs against the source database. One row more or less fails the check; if the dump can't be counted, the check fails rather than falling back to the estimate.
…tries, allow headers up to 1 MB Dot-joined keys made public."a.b" and "public.a".b one table, so a row missing from one could pass. setval and large-object SQL can hold a line that starts with COPY inside a quoted name; pg_restore now writes only the TABLE DATA entries for counting. A valid 1600-column COPY header can pass 100,000 characters.
… tables among the exclusions
This was referenced Nov 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Drill and restore now compare every table's row count with the backup, exactly. One row more or less in any table fails the check
row counts match the backup.Before this, the manifest's row counts were
n_live_tupplanner estimates taken on a separate connection beforepg_dumpran, so the drill could only check that tables which had rows didn't come back empty. Losing 10 of 10,000 rows passed.How
pg_restore --data-only --use-list=<TABLE DATA entries only> --file=-, counting the lines of eachCOPYsection (countRowsInDump/copyRowCounterinrestore-engine.ts). The dump was taken bypg_dumpin one consistent snapshot, so its row counts are exactly what the backup holds.TABLE DATAentries are emitted, so text elsewhere in the dump (for example a sequence name containing a newline andCOPY ... FROM stdin;inside asetval(...)literal) can't be mistaken for a data section.COPYheaders may span lines inside quoted names and are read whole, up to 1 MB (a valid 1,600-column header can pass 200k characters; the shared limit indump-sql.tsis raised to match).tableKey), not a dot-joined string, sopublic."a.b"and"public.a".bstay separate on both sides.count(*)there, no extra connection, no dependency on exporting a snapshot through a pooler. The cost is one more read of the dump during the drill, run concurrently with the restored-side counts. If verification fails partway, the reader is cancelled and awaited beforeverifyRestoredreturns, so nopg_restoreoutlives the drill.REFRESH), partitioned parents, and tables owned by an extension (CREATE EXTENSIONadds its own rows, and the dump only holds the rows past the extension's filter). Those keep the existing non-empty check.count(*) from only: a legacy inheritance parent's plaincount(*)includes its children's rows, while the dump holds the parent's own rows.COPYheader it can't parse, output cut off mid-section) or a table has no data section in the dump, the check fails. It never falls back to the estimate, and counting never throws.restoreruns the same check through the sharedverifyRestored.Tests
test/copy-row-counter.test.js: counting across every chunk split, escaped\.values, column-less tables, multi-line quoted headers, huge rows without buffering, and refusing to count output it can't read;rowCountsCheckcases.test/drill-row-counts.test.js(Docker): real dump with newlines, tabs and backslashes in values, legacy inheritance, partitions, a matview, a column-less table,public."a.b"next to"public.a".b, a sequence name carrying a fakeCOPYline, and a 1,600-column table. A full drill passes with every count matching; deleting rows and adding one to an inheritance child fail with the tables named. Cancelling returns null; a failed verification leaves nopg_restorerunning.npm teston 4cd60b9).examples/reproducible-drill/run.sh --with-login) rerun with this build:row counts match the backup — all 12 tables match the backup exactly (1,476,250 rows)in both drill and restore; README updated from that run.No preview environment for the CLI; verified by the tests and the reproducible drill above.
Version 2.11.0.