Decommission idle/broken prod client tenants [redacted]/[redacted]/[redacted] (Phase-0 EFS cleanup)

[saadqbal]

Context: Follow-up from the Phase-0 EFS cost investigation (backend#1161). While chasing a metronomic ~65 GB/day read on the tracebloc-clients-prod EFS (fs-06b3faf51675ff9f9, ~$62/mo throughput), we found three long-idle/broken client tenants on the prod client EKS cluster (tracebloc-clients-prod):

• [redacted] (164d) — tracebloc-jobs-manager in CrashLoopBackOff, ~4,017 restarts in 7 days, both containers dying at auth (Unable to log in with provided credentials), re-pulling jobs-manager:prod + pods-monitor:prod on every restart.
• [redacted] (228d) — jobs-manager idle (Received 0 messages).
• [redacted] (224d) — jobs-manager idle. (Both also logging Unknown database 'metadata'.)

Action taken (2026-07-27): kubectl delete namespace [redacted] [redacted] [redacted] on the prod client cluster. All three terminated cleanly.

Data safety: all StorageClasses ({[redacted],[redacted],[redacted]}-awsefs) and PVs are reclaimPolicy: Retain. The 9 PVs (mysql 2Gi, client-logs 10Gi, client-pvc 50Gi ×3) are now Released — no EFS data deleted; access points + contents remain on fs-06b3faf51675ff9f9 and are recoverable.

Follow-ups:

• Confirm the ~65 GB/day EFS read drops after decommission (verifying now; if it doesn't, the reader is elsewhere).
• If these tenants are permanently retired: clean up the 9 Released PVs and their retained EFS access points (they still cost EFS storage) — otherwise leave for recovery.
• [redacted] had invalid backend credentials — if it's ever re-provisioned, fix the client secret first.

Cross-ref: backend#1161, epic #1145.