Skip to content

Commit 1616cda

Browse files
ericallamTrigger.dev RepoOps
authored andcommitted
fix(webapp): pick a random address for the webhooks Redis seed on each reconnect
Webhook ingest recovers within seconds when the webhooks Redis cluster fails over, even when the cluster client has to reconnect through the configuration endpoint. It now resolves that endpoint itself and tries its addresses at random, instead of repeatedly connecting to the failed node that DNS still lists. Mono-RevId: d6d8a10f05888db57b7dffaea24679bebc1cc81a
1 parent ff2aa23 commit 1616cda

1 file changed

Lines changed: 20 additions & 2 deletions

File tree

‎apps/webapp/app/v3/webhookEngine.server.ts‎

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import {
55
} from "@internal/webhook-engine";
66
import type { WebhookDeliverTaskErrorType, WebhookRedisOptions } from "@internal/webhook-engine";
77
import { tryCatch } from "@trigger.dev/core/utils";
8+
import { resolve4 } from "node:dns/promises";
89
import { z } from "zod";
910
import { prisma, webhookPartitionPrisma, webhookPrisma } from "~/db.server";
1011
import { env } from "~/env.server";
@@ -27,6 +28,24 @@ import { meter, tracer } from "./tracer.server";
2728

2829
export const webhookEngine = singleton("WebhookEngine", createWebhookEngine);
2930

31+
/**
32+
* Resolves the seed host to one of its addresses at random on every connect and reconnect. After a
33+
* failover, ElastiCache's configuration endpoint keeps listing the old primary for minutes, and the
34+
* system resolver always ranks the address in the client's own subnet first, so a cluster client
35+
* that has to reconnect through the seed can keep landing on the dead node. TLS still verifies the
36+
* seed host (`servername`). A name that doesn't resolve, such as an IP or localhost, is used as is.
37+
*/
38+
function randomSeedAddress(
39+
address: string,
40+
callback: (err: Error | null, address: string) => void
41+
) {
42+
resolve4(address).then(
43+
(addresses) =>
44+
callback(null, addresses[Math.floor(Math.random() * addresses.length)] ?? address),
45+
() => callback(null, address)
46+
);
47+
}
48+
3049
/**
3150
* The webhooks Redis cluster, seeded from one node; ioredis discovers the rest. A primary that stops
3251
* answering without closing its sockets (an ElastiCache failover reboots the old one) never replies
@@ -50,8 +69,7 @@ function webhookRedisConnection(): WebhookRedisOptions {
5069
redisOptions,
5170
clusterOptions: {
5271
keyPrefix: "webhook:",
53-
dnsLookup: (address: string, callback: (err: Error | null, address: string) => void) =>
54-
callback(null, address),
72+
dnsLookup: randomSeedAddress,
5573
slotsRefreshInterval: 2_000,
5674
slotsRefreshTimeout: 2_000,
5775
},

0 commit comments

Comments
 (0)