fix(egress-ledger): reconnect redis indefinitely so egress-gateway self-heals

[rschlaefli]

Problem

codeapi-egress-gateway pods run but stay 0/1 (readiness never passes), which keeps the app-codeapi ArgoCD app Degraded. Observed in prd: two pods 0/1 for days, /ready returning 503, with the only log line being Egress gateway readiness failed and an empty {} error (the real cause is swallowed by the logger).

Root cause

redisConnection() in service/src/egress-ledger.ts caches a singleton ioredis client and configured it with:

const retryStrategy = times => {
  if (times > 5) return null;   // give up permanently after 5 attempts
  return 2000;
};

When retryStrategy returns null, ioredis moves the connection to the terminal end state and stops reconnecting for the lifetime of the process. If redis is briefly unreachable while the client is first created — e.g. a transient network/NetworkPolicy blip during a deploy/cutover — the singleton exhausts its 5 retries (~10s) and gives up for good.

From then on:

• /ready → pingEgressLedger() → redisConnection().ping() hits the dead client and rejects with Connection is closed (the empty {} in the logs) → readiness 503 forever.
• /live returns 200 unconditionally → liveness never fails → the pod is never restarted, so the dead singleton is never recreated. The pod is wedged until a manual restart.

Confirmation

Diagnosed against the running prd pod:

• A fresh ioredis client built with the exact production options (enableReadyCheck: true, maxRetriesPerRequest: 1, TLS) inside the same pod returns PING_OK PONG against the managed redis endpoint → redis is reachable; auth/TLS/network are fine.
• The same-namespace BullMQ worker (service-worker) connects cleanly with zero redis errors → not a reachability problem.
• The only difference is the long-lived singleton that had already given up. A kubectl rollout restart of the deployment immediately brought the pod to 1/1 and app-codeapi back to Healthy, confirming the diagnosis.

Fix

Use a capped-backoff strategy that retries indefinitely and never returns null, so the client self-heals once redis is reachable again:

const retryStrategy = times => Math.min(times * 500, 2000);

This is the identical pattern already used by service/src/file-server.ts and service/src/tool-call-server.ts in this repo, so it aligns the egress-ledger client with its siblings rather than introducing a new convention.

Individual readiness pings still fail fast during an actual outage (via maxRetriesPerRequest: 1), which is correct — readiness should reflect real state — but the connection now recovers on its own instead of staying dead.

Scope / related

service/src/queue.ts has a similar bounded retryStrategy, but the BullMQ producer/worker pods crash-loop and restart on a dead connection (they surface it through liveness), so they self-heal a different way. Left out of scope for this focused fix.

Verification

• prd already recovered via kubectl rollout restart deployment/codeapi-egress-gateway (1/1, endpoint populated, app-codeapi Healthy). This change prevents the wedge from recurring on the next transient redis interruption.
• No behavior change when redis is healthy (first connect succeeds on attempt 1).

Manual check before merge: build the egress-gateway image from this branch and confirm /ready recovers on its own after a simulated brief redis outage (block, then unblock) without a pod restart.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.