Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,15 @@ jobs:
shellcheck scripts/verify-wedding-backup-denial.sh scripts/verify-wedding-backup-denial-pod.sh scripts/tests/test-verify-wedding-backup-denial.sh
bash scripts/tests/test-verify-wedding-backup-denial.sh

# The stale shared copy of the Wedding catalogue is removed on this proof's
# word, and that cannot be undone. These cases keep a missing object,
# different bytes, a live writer or an unreviewed bucket from passing as
# covered, and pin the pod script to commands that only read.
- name: 💒 Validate Wedding shared backup coverage proof
run: |
shellcheck scripts/verify-wedding-shared-backup-coverage.sh scripts/verify-wedding-shared-backup-coverage-pod.sh scripts/tests/test-verify-wedding-shared-backup-coverage.sh
bash scripts/tests/test-verify-wedding-shared-backup-coverage.sh

- name: 💾 Validate two-stage PVC retirement
# A merge-group artifact is speculative, but a PVC deletion is not: its
# deletionTimestamp survives queue eviction and main cannot undo it.
Expand Down
94 changes: 94 additions & 0 deletions .github/workflows/verify-wedding-shared-backup-coverage.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
name: Verify Wedding Shared Backup Coverage

# Prove that the dedicated Wedding backup bucket covers everything the stale copy
# of the catalogue in the shared backup bucket still holds (#4481). Removing
# that copy is irreversible, so the decision needs this evidence first.
#
# DORMANT. There is no schedule and no pull_request trigger, and the operator
# must type an exact confirmation phrase, so an accidental dispatch does nothing.
#
# MAIN ONLY. `workflow_dispatch` accepts any branch, and this job holds the
# production kubeconfig. The prod environment already refuses non-main refs; the
# explicit check below fails fast and names the reason.
#
# READ ONLY. The proof lists both buckets and reads the objects it has to hash.
# It never writes to or deletes from either bucket.
on:
workflow_dispatch:
inputs:
confirm:
description: "Type exactly: verify-wedding-shared-backup-coverage"
required: true
type: string

permissions: {}

# The proof reads production state and two backup credentials. Serialize with
# every other production mutation so a deploy cannot change the wiring mid-run.
concurrency:
group: prod-deploy
cancel-in-progress: false
queue: max

jobs:
coverage:
name: 💒 Prove the dedicated bucket covers the shared Wedding catalogue
runs-on: ubuntu-latest
timeout-minutes: 90
environment: prod
permissions:
contents: read # checkout repository
steps:
# Inputs reach bash through env, never `${{ }}` inside run, so a crafted
# value cannot inject shell.
- name: 🛑 Require main and explicit confirmation
shell: bash
env:
CONFIRM: ${{ inputs.confirm }}
DISPATCH_REF: ${{ github.ref }}
run: |
set -euo pipefail
if [[ "${DISPATCH_REF}" != 'refs/heads/main' ]]; then
printf 'Refusing to run from %s: dispatch this workflow from main.\n' "${DISPATCH_REF}" >&2
exit 1
fi
if [[ "${CONFIRM}" != 'verify-wedding-shared-backup-coverage' ]]; then
printf 'Refusing to run: the confirm input must be exactly "verify-wedding-shared-backup-coverage".\n' >&2
printf 'The proof starts pods beside two production backup credentials. Nothing has been touched.\n' >&2
exit 1
fi
printf 'Confirmation accepted.\n'

- name: 📑 Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
ref: ${{ github.sha }}

- name: ⚙️ Setup Go
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
with:
go-version-file: go.mod
cache: false

- name: 🔑 Restore production kubeconfig
shell: bash
env:
KUBE_CONFIG: ${{ secrets.KUBE_CONFIG }}
run: |
set -euo pipefail
test -n "${KUBE_CONFIG}"
umask 077
mkdir -p ~/.kube
printf '%s' "${KUBE_CONFIG}" >~/.kube/config
chmod 600 ~/.kube/config

- name: 🎯 Select stable production API endpoint
shell: bash
env:
HCLOUD_TOKEN: ${{ secrets.HCLOUD_TOKEN }}
run: ./scripts/use-prod-stable-api-endpoint.sh

- name: 💒 Compare the shared catalogue with the dedicated one
shell: bash
run: ./scripts/verify-wedding-shared-backup-coverage.sh --confirm
11 changes: 10 additions & 1 deletion docs/dr/velero-cnpg.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,16 @@ The `Mirror Wedding Backup Catalogue` and `Verify Wedding Backup Denial`
workflows both need the shared credential inside `wedding-app`, so neither can
run any more; #4482 re-homes the denial proof and removes the finished cutover
tooling, and #4481 removes the predecessor catalogue that still sits in the
shared bucket. `Verify Wedding Backup Cutover`
shared bucket. That copy is no longer written to or pruned, and every holder of
the shared credential can read it. Before it is removed, dispatch `Verify
Wedding Shared Backup Coverage` on `main` with
`confirm=verify-wedding-shared-backup-coverage`. It only reads: a pod beside the
shared credential in `umami` and one beside the dedicated credential in
`wedding-app` each list their side, and the verdict is `COVERED` only when every
object in the shared copy is in the dedicated bucket with matching content, or
was provably removed there by the 30-day retention. Remove the shared copy only
after a `COVERED` run, and run the proof again afterwards: it then reports
`EMPTY`. `Verify Wedding Backup Cutover`
is a main-only, protected production dispatch: confirm
`verify-wedding-backup-cutover` to request a fresh online primary backup. It
refuses a shared archive, unhealthy database, changed database identity, or
Expand Down
Loading
Loading