Repository navigation
Backup/restore that actually restores, with a round-trip test - #10
Merged
Merged
Conversation
Tested on a compose stack (2026.07.2-rc1), the documented procedure did not work at any step: - `docker compose exec postgres /scripts/backup.sh` fails: the scripts are mounted into forail-task at /etc/forail/, not into postgres. - backup.sh piped pg_dump into gzip without pipefail, so a failed dump still produced a "backup" and exit 0. - restore.sh psql-loaded a plain dump into the live database: 1,868 "already exists" errors, nothing replaced (data written after the backup survived), "Restore complete." and exit 0. Worse, the dump's setval() calls did run, moving sequences back below existing ids, so the next inserts failed with duplicate keys and forail-init could no longer start. - Nothing recorded which FORAIL_SECRET_KEY a backup belongs to; restored under another key, every credential is unreadable. backup.sh: pg_dump custom format, written under a temporary name and renamed only when complete, pipefail, mode 0600, plus a .meta file with the Forail version, last migration and a fingerprint of the secret key. restore.sh: refuses without --yes, refuses while other sessions are connected (forail-web/forail-task must be stopped; it runs in a one-off `docker compose run --rm --no-deps forail-task` container), refuses on a secret-key mismatch unless told otherwise. It restores into a new database and swaps it in by renaming, so a failure changes nothing and the previous database stays available until dropped. pg_restore --clean is not an option: it cannot drop the partitioned event tables. Old .sql.gz backups still restore. scripts/test-backup-restore.sh runs the round trip against a running stack: marker data and an encrypted credential, backup, a later write, restore refused while running, restore, then checks the marker is back, the later write is gone, the secret decrypts and new writes succeed. It passes on a fresh 2026.07.2-rc1 stack. Also verified by hand: backup on 2026.07.2-rc1, upgrade to an image built from current backend develop work; forail-init applied 0211 and 0212, all data and job history present, credentials decrypt.
The README told operators to run ./scripts/backup.sh and restore.sh on the host, where neither pg_dump nor the database credentials are. Give the commands that actually work -- backup in forail-task, restore in a one-off container with the writers stopped -- and say plainly that the database backup is useless without FORAIL_SECRET_KEY. The full procedure, upgrade and rollback are in the Administrator Handbook.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
B5. The documented backup command failed; restore over a live DB replaced nothing, exited 0 and rolled sequences back. New scripts (custom-format dump, secret-key fingerprint, restore into a new DB and swap) and
scripts/test-backup-restore.sh, passing on a 2026.07.2-rc1 stack. Upgrade rc1 → current backend verified by hand. Docs not yet updated.