Repository navigation
Re-home the Wedding backup denial proof and remove cutover tooling that needs the retired shared store #4482
Description
🤖 Generated by the Agentic Engineer
Evidence
#3253 retires the Wedding tenant's access to the shared backup credential: the wedding-db ObjectStore and the wedding-db-backup-r2 Secret leave the wedding-app namespace. Three operator tools were written for the cutover and assume both still exist there:
Verify Wedding Backup Denial(scripts/verify-wedding-backup-denial.shand its pod script) mounts the shared credential next to the dedicated one, as the positive control and to check that a refused write left no trace. It passed once before retirement (run 37234932425) and cannot pass again in its current form.docs/dr/runbook.mdScenario 7 still wants this proof after every rotation of the dedicated token.Mirror Wedding Backup Catalogue(workflow, two scripts, a Go evaluator, about 2,300 lines with tests) copied the shared catalogue into the dedicated bucket. The copy and the catch-up are complete (Mirror the shared Wedding backup catalogue into the dedicated bucket and prove parity #3767, Switch the Wedding database to the dedicated backup store and prove new backups land there #3768).Recover Wedding DB Incident(workflow and script, about 1,100 lines with tests) is pinned to the object identities of the 2026-09-09 incident and to the shared store. Its last successful run was on 2026-09-13.
All three now refuse before doing any work. That is safe, but CI still runs their fixture suites on every pull request, and the runbook names a proof nobody can run.
Expected outcome
A dedicated-token rotation can again be followed by an observed denial, without the tenant namespace ever holding the shared credential, and the repository no longer carries one-shot tooling that cannot run.
Acceptance criteria
- The denial proof runs again with the shared credential kept out of the
wedding-appnamespace (for example from a platform-owned namespace that already holds it, using a run-owned copy of the lower-privilege dedicated credential), keeping its positive controls and its no-trace check. - A protected
mainrun of the re-homed proof reports the denial. - The catalogue mirror and the incident recovery tooling are removed, with their CI steps, or a recorded reason keeps them.
- The tenant's own database manifest names the dedicated store, so the production patch only pins the archive identity and a stale reference to the retired store is gone.
-
docs/dr/runbook.md,docs/dr/velero-cnpg.mdanddocs/dr/restore-drill.mddescribe only tooling that can run.
Delivered: #4495 removed the completed one-shot tooling and re-homed the denial verifier; #4503 merged through the protected queue and completed the tenant ownership handoff. Its production deployment succeeded. Read-only post-deployment verification finds the tenant source verified and reconciliation healthy, the redundant store override absent, and the same database identity/configuration with all three instances ready and archive/backup conditions true. Tenant-selected backup storage and Platform-owned persistence protections are preserved.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Projects
- Status
Activity
chore: agent-claim #4482 nonce=62f4b3760117596a922549b7d5288bd1
devantler commentedon Oct 5, 2026
#4495 is validated and reviewed at
a1e663faf16e179110afe92c20ee2f5c0ed8be0band has entered the protected merge queue. Exact-head CI succeeded, including the actual pinned-client denial regression. The formal current-head review is clean under the verified provider-limit fallback.The tenant declaration fix in devantler-tech/wedding-app#374 has merged and published v1.15.15 from
447b6dae3d89c64a627a78e06ff9c104f1e8e23a. Its signed source is applied and live readback is healthy with 3/3 database instances, continuous archiving and a successful last backup. The published source manifest names the dedicated store; the tenant issue is closed with its deployment receipt.This parent stays open. Remaining delivery is the protected
maindenial receipt and removal of Platform's redundant store override while preserving the archive identity, followed by readback. Queue entry and CI are not substitutes for those receipts.chore: agent-claim #4482 nonce=512d3ae60f07ede58d9bd0b6d1f9bf49
chore: agent-claim #4482 nonce=72df0c75184f8c94f4408cbcee2d9c79
chore: agent-claim #4482 nonce=82b8de98b5bc9e662c287371df1edf65
chore: agent-claim #4482 nonce=4c2c58072eff98b2d9a84a1649c6abb5
devantler commentedon Oct 6, 2026
The denial proof is dispatched, so this issue is no longer blocked.
Dispatched from
mainon 2026-10-06 at 16:32Z: run 37496522511. It is waiting in theprod-deployconcurrency group behind the merge queue's deploys. The Slack ask of 2026-10-05 is withdrawn: the maintainer's direction, given in an interactive session on 2026-10-06, is that dispatching a reviewed, main-only proof is the engineer's own step.Next: record the denial receipt here, then remove Platform's redundant store override while preserving the archive identity, and read the result back.
devantler commentedon Oct 6, 2026
Denial receipt: DENIAL OBSERVED
Run 37496522511, dispatched from
mainat06a15022, reported:This is the first run of the re-homed proof since #4495 merged, so the runbook's post-rotation proof is runnable again.
Remaining on this issue: remove Platform's redundant store override while preserving the archive identity, then read the result back.
devantler commentedon Oct 7, 2026
Closing: every acceptance criterion is met
Verified 2026-10-07 ~12:16Z.
main.mainrun reports the denial — run 37496522511 (workflow_dispatchonmainat06a15022, conclusionsuccess) reportedDENIAL OBSERVED.main, the production patch for the database now only disables pruning, sets synchronous replication and pins the archive identity; it no longer names a backup store;ContinuousArchivingSuccess.Removing the old copy from the shared bucket is tracked separately in #4481 and is not part of this issue.