From fce112f928f48ae98214a095731fd736044d0434 Mon Sep 17 00:00:00 2001 From: Benedikt Franke Date: Mon, 31 Aug 2026 10:05:21 +0200 Subject: [PATCH 1/9] feat(optional-modifications): add resolve-upstreams patch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit nginx resolves upstream names once at startup and caches the result for the life of a worker. When a container is recreated onto a different address, nginx keeps using the old one and every request fails until nginx is restarted. Ingest fails silently in that state: relay serves /api/store/, /api// and /api/0/relays/, so events are dropped while the UI keeps answering. Ship this as an opt-in patch rather than a change to nginx.conf, per https://github.com/getsentry/self-hosted/issues/3894#issuecomment-2606270599 — the caching is intentional and the suggested remedy was to apply it locally. The resolver address is rendered at container start from the container's own /etc/resolv.conf via the nginx image entrypoint, so the patch does not hardcode 127.0.0.11 and stays engine-agnostic. nginx.conf keeps its existing mount target and pulls the generated snippet in with an include, so a custom nginx.conf is neither overwritten nor made unwritable. Measured cost with 32 workers over 70s: 14 A queries total, no AAAA — one per upstream per valid=10s window, independent of worker count and traffic, because the shared zone means a single worker performs the lookup. Verified recovery after a forced address change (filler container holding the old address): patched recovers at t=8s, unpatched still 502 at t=30s. The README records the tradeoffs, notably that resolve replaces a loud startup failure on an unresolvable upstream with a running container serving 502s. --- .../patches/resolve-upstreams/README.md | 114 ++++++++++++++++++ .../docker-compose.yml.patch | 20 +++ .../nginx-resolver.conf.template.patch | 4 + .../resolve-upstreams/nginx.conf.patch | 23 ++++ 4 files changed, 161 insertions(+) create mode 100644 optional-modifications/patches/resolve-upstreams/README.md create mode 100644 optional-modifications/patches/resolve-upstreams/docker-compose.yml.patch create mode 100644 optional-modifications/patches/resolve-upstreams/nginx-resolver.conf.template.patch create mode 100644 optional-modifications/patches/resolve-upstreams/nginx.conf.patch diff --git a/optional-modifications/patches/resolve-upstreams/README.md b/optional-modifications/patches/resolve-upstreams/README.md new file mode 100644 index 00000000000..31d30b2dc4e --- /dev/null +++ b/optional-modifications/patches/resolve-upstreams/README.md @@ -0,0 +1,114 @@ +# resolve-upstreams + +Make nginx re-resolve the `relay` and `web` container addresses instead of caching them for the +life of a worker process. + +## The problem + +nginx resolves the names in an `upstream` block once, at startup, and keeps the result forever. If +a container is recreated and comes back on a different address, nginx keeps sending traffic to the +old one. Every request fails until nginx itself is restarted. + +The UI failing is obvious. Ingest failing is not: `relay` serves the `/api/store/`, `/api//` +and `/api/0/relays/` routes, so events are dropped while the UI keeps working normally and nothing +in the stack reports an error. + +`docker compose restart relay` alone does not trigger this — the address is usually reused. It +takes a recreate (`docker compose up -d --force-recreate`, an image upgrade, a host reboot with a +different container start order, or anything else that lets another container claim the old +address) for the addresses to actually move. + +## What the patch changes + +- Adds `resolve` to both `upstream` servers, so nginx re-resolves the name on a timer. +- Adds `zone 512k`, which `resolve` requires — it needs the upstream group in shared memory. +- Adds a `resolver` directive pointing at the container engine's own DNS, rendered at container + start from `nginx-resolver.conf.template` by the nginx image entrypoint + (`NGINX_ENTRYPOINT_LOCAL_RESOLVERS=1`). The address differs per engine: `127.0.0.11` on Docker, + the network gateway on Podman. Reading it from the container's `/etc/resolv.conf` keeps the patch + engine-agnostic. + +`nginx.conf` stays mounted at `/etc/nginx/nginx.conf`, exactly where it is without the patch. The +generated snippet lands in `conf.d/` and is pulled in with an `include`, so a custom `nginx.conf` +is neither overwritten nor made unwritable. + +## How to apply + +From the repository root, then run `./install.sh`: + +```bash +patch -p0 < optional-modifications/patches/resolve-upstreams/nginx-resolver.conf.template.patch +patch -p0 < optional-modifications/patches/resolve-upstreams/nginx.conf.patch +patch -p0 < optional-modifications/patches/resolve-upstreams/docker-compose.yml.patch +``` + +The first patch creates a new file. All three belong together: `nginx.conf` will not start without +the rendered snippet, and the snippet is only rendered if the `docker-compose.yml` changes +(the environment variable and the template bind mount) are in place. + +## Cost + +Measured on Docker 28.5.2, `nginx:1.31.4-alpine`, 32 worker processes, with a logging DNS server +in front of the engine resolver, over 70 seconds: + +``` +7 query[A] web +7 query[A] relay +``` + +14 queries, no AAAA. That is ~6 per minute per upstream — one per `valid=10s` window, and only one +query regardless of how many workers exist or how much traffic flows, because the shared zone means +a single worker performs the lookup. `ipv6=off` keeps nginx from also asking for AAAA records that +a default Compose network cannot route. + +`valid=10s` is the recovery time. Raise it to cut the (already small) query volume further; the +cost is a correspondingly longer outage after an address change. + +## Verified behaviour + +Recovery, with a filler container claiming the old address so it cannot be reused: + +``` +web 172.31.7.2 -> 172.31.7.4 +patched: t2s=502 t4s=502 t6s=502 t8s=200 t10s=200 t12s=200 +unpatched: t5s=502 t10s=502 t15s=502 t20s=502 t25s=502 t30s=502 (never recovers) +``` + +## Tradeoffs + +**A missing upstream no longer stops nginx from starting.** Without the patch, a name that does not +resolve is fatal and loud: + +``` +nginx: [emerg] host not found in upstream "relay:3000" in /etc/nginx/nginx.conf:69 +``` + +With `resolve`, nginx starts, logs an error, and serves 502s until the name resolves. That is the +point — it is what lets nginx come up before `relay` and recover on its own — but it also means a +typo in an upstream name, or a resolver that is unreachable, produces a healthy-looking container +serving errors instead of a container that visibly refuses to start. The nginx healthcheck does not +catch this: it requests `/`, and `curl` without `-f` exits 0 on a 502. + +**The resolver address must be discoverable.** If the nginx container's `/etc/resolv.conf` has no +`nameserver` line, the rendered directive has no address and nginx refuses to start +(`[emerg] no name servers defined`). Note that an unpatched install is also broken in that +situation — the static upstreams fail with `host not found in upstream` — so this is a different +error message rather than a newly broken case. + +**Do not bind-mount over `/etc/nginx/conf.d`.** The entrypoint renders the snippet there; if that +directory is read-only, it aborts before nginx starts. + +## Engines + +Verified on Docker. The Podman path is engine-agnostic by construction (the address comes from +`/etc/resolv.conf` rather than being hardcoded) but has not been tested — if you run Podman, +confirm the rendered `/etc/nginx/conf.d/nginx-resolver.conf` names your aardvark resolver: + +```bash +docker compose exec nginx cat /etc/nginx/conf.d/nginx-resolver.conf +``` + +## Background + +- https://github.com/getsentry/self-hosted/issues/3894 +- https://github.com/getsentry/self-hosted/pull/4492 diff --git a/optional-modifications/patches/resolve-upstreams/docker-compose.yml.patch b/optional-modifications/patches/resolve-upstreams/docker-compose.yml.patch new file mode 100644 index 00000000000..8903db75460 --- /dev/null +++ b/optional-modifications/patches/resolve-upstreams/docker-compose.yml.patch @@ -0,0 +1,20 @@ +--- docker-compose.yml 2026-08-31 09:58:28.827732343 +0200 ++++ docker-compose.resolve-upstreams.yml 2026-08-31 09:59:13.197843697 +0200 +@@ -601,11 +601,17 @@ + ports: + - "$SENTRY_BIND:80/tcp" + image: "nginx:1.31.4-alpine" ++ environment: ++ NGINX_ENTRYPOINT_LOCAL_RESOLVERS: 1 + volumes: + - type: bind + read_only: true + source: ./nginx.conf + target: /etc/nginx/nginx.conf ++ - type: bind ++ read_only: true ++ source: ./nginx-resolver.conf.template ++ target: /etc/nginx/templates/nginx-resolver.conf.template + - sentry-nginx-cache:/var/cache/nginx + - sentry-nginx-www:/var/www + healthcheck: diff --git a/optional-modifications/patches/resolve-upstreams/nginx-resolver.conf.template.patch b/optional-modifications/patches/resolve-upstreams/nginx-resolver.conf.template.patch new file mode 100644 index 00000000000..0a079a4ec59 --- /dev/null +++ b/optional-modifications/patches/resolve-upstreams/nginx-resolver.conf.template.patch @@ -0,0 +1,4 @@ +--- /dev/null 2026-08-27 11:01:02.392055778 +0200 ++++ nginx-resolver.conf.template 2026-08-31 09:59:17.962753633 +0200 +@@ -0,0 +1 @@ ++resolver ${NGINX_LOCAL_RESOLVERS} ipv6=off valid=10s; diff --git a/optional-modifications/patches/resolve-upstreams/nginx.conf.patch b/optional-modifications/patches/resolve-upstreams/nginx.conf.patch new file mode 100644 index 00000000000..546de69bc4a --- /dev/null +++ b/optional-modifications/patches/resolve-upstreams/nginx.conf.patch @@ -0,0 +1,23 @@ +--- nginx.conf 2026-08-31 09:58:28.827732343 +0200 ++++ nginx.resolve-upstreams.conf 2026-08-31 09:59:13.197674745 +0200 +@@ -65,13 +65,18 @@ + proxy_read_timeout 90s; + proxy_send_timeout 5s; + ++ # See optional-modifications/patches/resolve-upstreams/README.md ++ include /etc/nginx/conf.d/nginx-resolver.conf; ++ + upstream relay { +- server relay:3000; ++ zone relay 512k; ++ server relay:3000 resolve; + keepalive 2; + } + + upstream sentry { +- server web:9000; ++ zone sentry 512k; ++ server web:9000 resolve; + keepalive 2; + } + From 002ebd6593f041672c8ce6c76050f0a597522b53 Mon Sep 17 00:00:00 2001 From: Benedikt Franke Date: Mon, 31 Aug 2026 10:16:01 +0200 Subject: [PATCH 2/9] docs(resolve-upstreams): trim the README The first version explained the mechanism at a length that only repeated what the patch files show. Keep the problem, the apply order and the tradeoffs a reader cannot derive from the diff. --- .../patches/resolve-upstreams/README.md | 109 ++++-------------- 1 file changed, 21 insertions(+), 88 deletions(-) diff --git a/optional-modifications/patches/resolve-upstreams/README.md b/optional-modifications/patches/resolve-upstreams/README.md index 31d30b2dc4e..ec18352a9e6 100644 --- a/optional-modifications/patches/resolve-upstreams/README.md +++ b/optional-modifications/patches/resolve-upstreams/README.md @@ -1,38 +1,17 @@ # resolve-upstreams -Make nginx re-resolve the `relay` and `web` container addresses instead of caching them for the -life of a worker process. +nginx resolves the names in an `upstream` block once, at startup, and keeps the result for the life +of a worker. When a container is recreated onto a different address, nginx keeps using the old one +and every request fails until nginx is restarted. -## The problem +The UI failing is obvious; ingest failing is not. `relay` serves `/api/store/`, `/api//` and +`/api/0/relays/`, so events are dropped while the UI keeps answering and nothing reports an error. -nginx resolves the names in an `upstream` block once, at startup, and keeps the result forever. If -a container is recreated and comes back on a different address, nginx keeps sending traffic to the -old one. Every request fails until nginx itself is restarted. +This patch adds `resolve` to both upstreams, the `zone` that `resolve` requires, and a `resolver` +directive rendered at container start from the container's own `/etc/resolv.conf` — the engine DNS +address is `127.0.0.11` on Docker but the network gateway on Podman, so it is not hardcoded. -The UI failing is obvious. Ingest failing is not: `relay` serves the `/api/store/`, `/api//` -and `/api/0/relays/` routes, so events are dropped while the UI keeps working normally and nothing -in the stack reports an error. - -`docker compose restart relay` alone does not trigger this — the address is usually reused. It -takes a recreate (`docker compose up -d --force-recreate`, an image upgrade, a host reboot with a -different container start order, or anything else that lets another container claim the old -address) for the addresses to actually move. - -## What the patch changes - -- Adds `resolve` to both `upstream` servers, so nginx re-resolves the name on a timer. -- Adds `zone 512k`, which `resolve` requires — it needs the upstream group in shared memory. -- Adds a `resolver` directive pointing at the container engine's own DNS, rendered at container - start from `nginx-resolver.conf.template` by the nginx image entrypoint - (`NGINX_ENTRYPOINT_LOCAL_RESOLVERS=1`). The address differs per engine: `127.0.0.11` on Docker, - the network gateway on Podman. Reading it from the container's `/etc/resolv.conf` keeps the patch - engine-agnostic. - -`nginx.conf` stays mounted at `/etc/nginx/nginx.conf`, exactly where it is without the patch. The -generated snippet lands in `conf.d/` and is pulled in with an `include`, so a custom `nginx.conf` -is neither overwritten nor made unwritable. - -## How to apply +## Apply From the repository root, then run `./install.sh`: @@ -42,71 +21,25 @@ patch -p0 < optional-modifications/patches/resolve-upstreams/nginx.conf.patch patch -p0 < optional-modifications/patches/resolve-upstreams/docker-compose.yml.patch ``` -The first patch creates a new file. All three belong together: `nginx.conf` will not start without -the rendered snippet, and the snippet is only rendered if the `docker-compose.yml` changes -(the environment variable and the template bind mount) are in place. - -## Cost - -Measured on Docker 28.5.2, `nginx:1.31.4-alpine`, 32 worker processes, with a logging DNS server -in front of the engine resolver, over 70 seconds: - -``` -7 query[A] web -7 query[A] relay -``` - -14 queries, no AAAA. That is ~6 per minute per upstream — one per `valid=10s` window, and only one -query regardless of how many workers exist or how much traffic flows, because the shared zone means -a single worker performs the lookup. `ipv6=off` keeps nginx from also asking for AAAA records that -a default Compose network cannot route. - -`valid=10s` is the recovery time. Raise it to cut the (already small) query volume further; the -cost is a correspondingly longer outage after an address change. - -## Verified behaviour - -Recovery, with a filler container claiming the old address so it cannot be reused: - -``` -web 172.31.7.2 -> 172.31.7.4 -patched: t2s=502 t4s=502 t6s=502 t8s=200 t10s=200 t12s=200 -unpatched: t5s=502 t10s=502 t15s=502 t20s=502 t25s=502 t30s=502 (never recovers) -``` +All three belong together: `nginx.conf` will not start without the rendered snippet, and the snippet +is only rendered with the `docker-compose.yml` changes in place. ## Tradeoffs -**A missing upstream no longer stops nginx from starting.** Without the patch, a name that does not -resolve is fatal and loud: - -``` -nginx: [emerg] host not found in upstream "relay:3000" in /etc/nginx/nginx.conf:69 -``` - -With `resolve`, nginx starts, logs an error, and serves 502s until the name resolves. That is the -point — it is what lets nginx come up before `relay` and recover on its own — but it also means a -typo in an upstream name, or a resolver that is unreachable, produces a healthy-looking container -serving errors instead of a container that visibly refuses to start. The nginx healthcheck does not -catch this: it requests `/`, and `curl` without `-f` exits 0 on a 502. +`valid=10s` is the recovery time. Measured cost with 32 workers: 6 queries per minute per upstream, +independent of worker count and traffic, because the shared zone means one worker does the lookup. -**The resolver address must be discoverable.** If the nginx container's `/etc/resolv.conf` has no -`nameserver` line, the rendered directive has no address and nginx refuses to start -(`[emerg] no name servers defined`). Note that an unpatched install is also broken in that -situation — the static upstreams fail with `host not found in upstream` — so this is a different -error message rather than a newly broken case. +An unresolvable upstream no longer stops nginx from starting. That is what lets nginx come up before +`relay`, but a typo in an upstream name or an unreachable resolver now yields a running container +serving 502s instead of `[emerg] host not found in upstream`. The healthcheck does not catch it — +it requests `/`, and `curl` without `-f` exits 0 on a 502. -**Do not bind-mount over `/etc/nginx/conf.d`.** The entrypoint renders the snippet there; if that -directory is read-only, it aborts before nginx starts. +If the nginx container's `/etc/resolv.conf` has no `nameserver`, nginx refuses to start. An +unpatched install is also broken in that case, with a different message. -## Engines +Do not bind-mount over `/etc/nginx/conf.d`; the snippet is rendered there. -Verified on Docker. The Podman path is engine-agnostic by construction (the address comes from -`/etc/resolv.conf` rather than being hardcoded) but has not been tested — if you run Podman, -confirm the rendered `/etc/nginx/conf.d/nginx-resolver.conf` names your aardvark resolver: - -```bash -docker compose exec nginx cat /etc/nginx/conf.d/nginx-resolver.conf -``` +Verified on Docker. The Podman path is engine-agnostic by construction but untested. ## Background From 990435493201e9fa331c5322f74f84fbbc2ce61d Mon Sep 17 00:00:00 2001 From: Benedikt Franke Date: Mon, 31 Aug 2026 10:22:39 +0200 Subject: [PATCH 3/9] fix(resolve-upstreams): anchor the compose patch away from the image tag The hunk carried image: "nginx:1.31.4-alpine" as a context line, so every dependabot bump broke it. That tag moved four times in the last twelve months, which gave the patch a shelf life of about one quarter. Insert the mount and the environment key after the volumes block instead. The tag is no longer in any hunk's context, and the patch applies cleanly through a simulated bump to 1.32.0. --- .../resolve-upstreams/docker-compose.yml.patch | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/optional-modifications/patches/resolve-upstreams/docker-compose.yml.patch b/optional-modifications/patches/resolve-upstreams/docker-compose.yml.patch index 8903db75460..3a2e15d0e44 100644 --- a/optional-modifications/patches/resolve-upstreams/docker-compose.yml.patch +++ b/optional-modifications/patches/resolve-upstreams/docker-compose.yml.patch @@ -1,13 +1,6 @@ --- docker-compose.yml 2026-08-31 09:58:28.827732343 +0200 -+++ docker-compose.resolve-upstreams.yml 2026-08-31 09:59:13.197843697 +0200 -@@ -601,11 +601,17 @@ - ports: - - "$SENTRY_BIND:80/tcp" - image: "nginx:1.31.4-alpine" -+ environment: -+ NGINX_ENTRYPOINT_LOCAL_RESOLVERS: 1 - volumes: - - type: bind ++++ docker-compose.resolve-upstreams.yml 2026-08-31 10:21:46.281494103 +0200 +@@ -606,8 +606,14 @@ read_only: true source: ./nginx.conf target: /etc/nginx/nginx.conf @@ -17,4 +10,8 @@ + target: /etc/nginx/templates/nginx-resolver.conf.template - sentry-nginx-cache:/var/cache/nginx - sentry-nginx-www:/var/www ++ environment: ++ NGINX_ENTRYPOINT_LOCAL_RESOLVERS: 1 healthcheck: + <<: *healthcheck_defaults + test: From bf4b93f92a2646de7a3a05e1331914eb88e643d9 Mon Sep 17 00:00:00 2001 From: Benedikt Franke Date: Mon, 31 Aug 2026 10:29:12 +0200 Subject: [PATCH 4/9] docs(resolve-upstreams): make a failed patch leave a working install The three patches are interdependent, and applying them as separate commands in the previous order put the dangerous state one upstream change away: nginx.conf gained an include for a snippet that docker-compose.yml was never patched to render, so nginx refused to start. Reproduced by adding a volume to the nginx service, which is enough to reject the compose hunk. Chain them with && and patch nginx.conf last, so a rejected hunk stops the sequence before anything depends on it. Verified both directions: a failing compose patch now leaves the tree untouched, and the compose and template patches without the nginx.conf one boot fine. --- .../patches/resolve-upstreams/README.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/optional-modifications/patches/resolve-upstreams/README.md b/optional-modifications/patches/resolve-upstreams/README.md index ec18352a9e6..3696387b660 100644 --- a/optional-modifications/patches/resolve-upstreams/README.md +++ b/optional-modifications/patches/resolve-upstreams/README.md @@ -16,13 +16,16 @@ address is `127.0.0.11` on Docker but the network gateway on Podman, so it is no From the repository root, then run `./install.sh`: ```bash -patch -p0 < optional-modifications/patches/resolve-upstreams/nginx-resolver.conf.template.patch +patch -p0 < optional-modifications/patches/resolve-upstreams/docker-compose.yml.patch && \ +patch -p0 < optional-modifications/patches/resolve-upstreams/nginx-resolver.conf.template.patch && \ patch -p0 < optional-modifications/patches/resolve-upstreams/nginx.conf.patch -patch -p0 < optional-modifications/patches/resolve-upstreams/docker-compose.yml.patch ``` -All three belong together: `nginx.conf` will not start without the rendered snippet, and the snippet -is only rendered with the `docker-compose.yml` changes in place. +Keep the order and the `&&`. All three belong together — `nginx.conf` will not start without the +rendered snippet, and the snippet is only rendered with the `docker-compose.yml` changes in place — +so `nginx.conf` is patched last, after the parts it depends on have succeeded. If one of these files +has moved upstream and its patch no longer applies, stopping there leaves a working install rather +than an `nginx.conf` that requires a snippet nothing renders. ## Tradeoffs From 649c6e7e8e38703779e380834d7f3de59cc45eb0 Mon Sep 17 00:00:00 2001 From: Benedikt Franke Date: Mon, 31 Aug 2026 10:30:56 +0200 Subject: [PATCH 5/9] docs(resolve-upstreams): cut the README to instructions Keep the commands and the constraints; drop the prose that explained mechanics the patch files already show. --- .../patches/resolve-upstreams/README.md | 45 ++++++------------- 1 file changed, 14 insertions(+), 31 deletions(-) diff --git a/optional-modifications/patches/resolve-upstreams/README.md b/optional-modifications/patches/resolve-upstreams/README.md index 3696387b660..665e5c28476 100644 --- a/optional-modifications/patches/resolve-upstreams/README.md +++ b/optional-modifications/patches/resolve-upstreams/README.md @@ -1,19 +1,12 @@ # resolve-upstreams -nginx resolves the names in an `upstream` block once, at startup, and keeps the result for the life -of a worker. When a container is recreated onto a different address, nginx keeps using the old one -and every request fails until nginx is restarted. - -The UI failing is obvious; ingest failing is not. `relay` serves `/api/store/`, `/api//` and -`/api/0/relays/`, so events are dropped while the UI keeps answering and nothing reports an error. - -This patch adds `resolve` to both upstreams, the `zone` that `resolve` requires, and a `resolver` -directive rendered at container start from the container's own `/etc/resolv.conf` — the engine DNS -address is `127.0.0.11` on Docker but the network gateway on Podman, so it is not hardcoded. +Make nginx re-resolve `relay` and `web` instead of caching their addresses until it restarts. +Without this, a recreated container that comes back on a different address 502s every request — +silently for ingest, since `relay` serves `/api/store/`, `/api//` and `/api/0/relays/`. ## Apply -From the repository root, then run `./install.sh`: +From the repository root, keeping the order and the `&&`, then run `./install.sh`: ```bash patch -p0 < optional-modifications/patches/resolve-upstreams/docker-compose.yml.patch && \ @@ -21,28 +14,18 @@ patch -p0 < optional-modifications/patches/resolve-upstreams/nginx-resolver.conf patch -p0 < optional-modifications/patches/resolve-upstreams/nginx.conf.patch ``` -Keep the order and the `&&`. All three belong together — `nginx.conf` will not start without the -rendered snippet, and the snippet is only rendered with the `docker-compose.yml` changes in place — -so `nginx.conf` is patched last, after the parts it depends on have succeeded. If one of these files -has moved upstream and its patch no longer applies, stopping there leaves a working install rather -than an `nginx.conf` that requires a snippet nothing renders. - -## Tradeoffs - -`valid=10s` is the recovery time. Measured cost with 32 workers: 6 queries per minute per upstream, -independent of worker count and traffic, because the shared zone means one worker does the lookup. - -An unresolvable upstream no longer stops nginx from starting. That is what lets nginx come up before -`relay`, but a typo in an upstream name or an unreachable resolver now yields a running container -serving 502s instead of `[emerg] host not found in upstream`. The healthcheck does not catch it — -it requests `/`, and `curl` without `-f` exits 0 on a 502. - -If the nginx container's `/etc/resolv.conf` has no `nameserver`, nginx refuses to start. An -unpatched install is also broken in that case, with a different message. +A rejected hunk stops the sequence before anything depends on it. -Do not bind-mount over `/etc/nginx/conf.d`; the snippet is rendered there. +## Notes -Verified on Docker. The Podman path is engine-agnostic by construction but untested. +- `valid=10s` is the recovery time. Cost: 6 DNS queries per minute per upstream, regardless of + worker count and traffic. +- An unresolvable upstream no longer stops nginx from starting; it serves 502s instead. The + healthcheck does not catch this — it requests `/`, and `curl` without `-f` exits 0 on a 502. +- nginx will not start if its `/etc/resolv.conf` has no `nameserver`. An unpatched install is also + broken in that case, with a different message. +- Do not bind-mount over `/etc/nginx/conf.d`; the resolver snippet is rendered there. +- Verified on Docker. The Podman path is engine-agnostic by construction but untested. ## Background From ab501e662ed8090f1b19f70d5f15012aa25c8fca Mon Sep 17 00:00:00 2001 From: Benedikt Franke Date: Mon, 31 Aug 2026 10:31:24 +0200 Subject: [PATCH 6/9] docs(resolve-upstreams): one sentence per line Short sentences, one per line, so edits produce single-line diffs. --- .../patches/resolve-upstreams/README.md | 33 ++++++++++++------- 1 file changed, 21 insertions(+), 12 deletions(-) diff --git a/optional-modifications/patches/resolve-upstreams/README.md b/optional-modifications/patches/resolve-upstreams/README.md index 665e5c28476..d87faf338e7 100644 --- a/optional-modifications/patches/resolve-upstreams/README.md +++ b/optional-modifications/patches/resolve-upstreams/README.md @@ -1,12 +1,16 @@ # resolve-upstreams -Make nginx re-resolve `relay` and `web` instead of caching their addresses until it restarts. -Without this, a recreated container that comes back on a different address 502s every request — -silently for ingest, since `relay` serves `/api/store/`, `/api//` and `/api/0/relays/`. +nginx caches the addresses of `relay` and `web` until it restarts. +A recreated container can come back on a different address. +Every request then fails with 502. +Ingest fails silently, because `relay` serves `/api/store/`, `/api//` and `/api/0/relays/`. +This patch makes nginx re-resolve both names instead. ## Apply -From the repository root, keeping the order and the `&&`, then run `./install.sh`: +Run from the repository root. +Keep the order and the `&&`. +Then run `./install.sh`. ```bash patch -p0 < optional-modifications/patches/resolve-upstreams/docker-compose.yml.patch && \ @@ -18,14 +22,19 @@ A rejected hunk stops the sequence before anything depends on it. ## Notes -- `valid=10s` is the recovery time. Cost: 6 DNS queries per minute per upstream, regardless of - worker count and traffic. -- An unresolvable upstream no longer stops nginx from starting; it serves 502s instead. The - healthcheck does not catch this — it requests `/`, and `curl` without `-f` exits 0 on a 502. -- nginx will not start if its `/etc/resolv.conf` has no `nameserver`. An unpatched install is also - broken in that case, with a different message. -- Do not bind-mount over `/etc/nginx/conf.d`; the resolver snippet is rendered there. -- Verified on Docker. The Podman path is engine-agnostic by construction but untested. +- `valid=10s` is the recovery time. + It costs 6 DNS queries per minute per upstream. + Worker count and traffic do not change that. +- An unresolvable upstream no longer stops nginx from starting. + It serves 502s instead. + The healthcheck does not catch this. + It requests `/`, and `curl` without `-f` exits 0 on a 502. +- nginx will not start if its `/etc/resolv.conf` has no `nameserver`. + An unpatched install is also broken in that case, with a different message. +- Do not bind-mount over `/etc/nginx/conf.d`. + The resolver snippet is rendered there. +- Verified on Docker. + The Podman path is engine-agnostic by construction, but untested. ## Background From 5a3dc204dbafed1f3164dff7cfa15c406826ed90 Mon Sep 17 00:00:00 2001 From: Benedikt Franke Date: Mon, 31 Aug 2026 10:31:57 +0200 Subject: [PATCH 7/9] docs(resolve-upstreams): use active voice Name the actor in each sentence and drop the hedged phrasing. --- .../patches/resolve-upstreams/README.md | 31 +++++++++---------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/optional-modifications/patches/resolve-upstreams/README.md b/optional-modifications/patches/resolve-upstreams/README.md index d87faf338e7..2a4da9c934f 100644 --- a/optional-modifications/patches/resolve-upstreams/README.md +++ b/optional-modifications/patches/resolve-upstreams/README.md @@ -1,15 +1,14 @@ # resolve-upstreams nginx caches the addresses of `relay` and `web` until it restarts. -A recreated container can come back on a different address. -Every request then fails with 502. -Ingest fails silently, because `relay` serves `/api/store/`, `/api//` and `/api/0/relays/`. -This patch makes nginx re-resolve both names instead. +A recreated container can return on a different address. +nginx keeps using the old one and answers every request with 502. +Ingest breaks silently, because `relay` serves `/api/store/`, `/api//` and `/api/0/relays/`. +This patch makes nginx re-resolve both names. ## Apply -Run from the repository root. -Keep the order and the `&&`. +Run these from the repository root, keeping the order and the `&&`. Then run `./install.sh`. ```bash @@ -22,19 +21,19 @@ A rejected hunk stops the sequence before anything depends on it. ## Notes -- `valid=10s` is the recovery time. +- `valid=10s` sets the recovery time. It costs 6 DNS queries per minute per upstream. - Worker count and traffic do not change that. + Worker count and traffic do not affect that. - An unresolvable upstream no longer stops nginx from starting. - It serves 502s instead. - The healthcheck does not catch this. - It requests `/`, and `curl` without `-f` exits 0 on a 502. -- nginx will not start if its `/etc/resolv.conf` has no `nameserver`. - An unpatched install is also broken in that case, with a different message. + nginx serves 502s instead. + The healthcheck misses this, because it requests `/` and `curl` without `-f` exits 0 on a 502. +- nginx refuses to start if its `/etc/resolv.conf` lists no `nameserver`. + An unpatched install also breaks in that case, with a different message. - Do not bind-mount over `/etc/nginx/conf.d`. - The resolver snippet is rendered there. -- Verified on Docker. - The Podman path is engine-agnostic by construction, but untested. + The entrypoint renders the resolver snippet there. +- Tested on Docker. + The patch reads the resolver address from the container, so it should suit Podman, but nobody has + tested that. ## Background From 4b835a0756941ea76fb868279e448b08e5caeda4 Mon Sep 17 00:00:00 2001 From: Benedikt Franke Date: Mon, 31 Aug 2026 11:31:33 +0200 Subject: [PATCH 8/9] ref(resolve-upstreams): inline the pointer to the README One directive, one trailing clause, one line saved. --- .../patches/resolve-upstreams/nginx.conf.patch | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/optional-modifications/patches/resolve-upstreams/nginx.conf.patch b/optional-modifications/patches/resolve-upstreams/nginx.conf.patch index 546de69bc4a..11899408270 100644 --- a/optional-modifications/patches/resolve-upstreams/nginx.conf.patch +++ b/optional-modifications/patches/resolve-upstreams/nginx.conf.patch @@ -1,11 +1,10 @@ ---- nginx.conf 2026-08-31 09:58:28.827732343 +0200 -+++ nginx.resolve-upstreams.conf 2026-08-31 09:59:13.197674745 +0200 -@@ -65,13 +65,18 @@ +--- nginx.conf ++++ nginx.resolve-upstreams.conf +@@ -65,13 +65,17 @@ proxy_read_timeout 90s; proxy_send_timeout 5s; -+ # See optional-modifications/patches/resolve-upstreams/README.md -+ include /etc/nginx/conf.d/nginx-resolver.conf; ++ include /etc/nginx/conf.d/nginx-resolver.conf; # see optional-modifications/patches/resolve-upstreams + upstream relay { - server relay:3000; From b234f399652d1954efc6b6cca0610b78e88a6efa Mon Sep 17 00:00:00 2001 From: Benedikt Franke Date: Mon, 31 Aug 2026 12:50:53 +0200 Subject: [PATCH 9/9] docs(nginx): scope the patch against the merged depends_on policy State that restart: true already covers Compose operations, so a reader can tell whether the patch applies to them, and link the prior attempts. --- .../patches/resolve-upstreams/README.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/optional-modifications/patches/resolve-upstreams/README.md b/optional-modifications/patches/resolve-upstreams/README.md index 2a4da9c934f..b7ac453cfac 100644 --- a/optional-modifications/patches/resolve-upstreams/README.md +++ b/optional-modifications/patches/resolve-upstreams/README.md @@ -6,6 +6,9 @@ nginx keeps using the old one and answers every request with 502. Ingest breaks silently, because `relay` serves `/api/store/`, `/api//` and `/api/0/relays/`. This patch makes nginx re-resolve both names. +`restart: true` on nginx's `depends_on` already restarts nginx after a Compose operation recreates `web` or `relay`. +Apply this patch only if you also need to survive address changes that no Compose operation causes, such as a crash restart or a Docker daemon restart. + ## Apply Run these from the repository root, keeping the order and the `&&`. @@ -37,5 +40,7 @@ A rejected hunk stops the sequence before anything depends on it. ## Background -- https://github.com/getsentry/self-hosted/issues/3894 -- https://github.com/getsentry/self-hosted/pull/4492 +- [Report of nginx routing to a stale relay address](https://github.com/getsentry/self-hosted/issues/3894) +- [The `depends_on` policy that covers Compose operations](https://github.com/getsentry/self-hosted/pull/3914) +- [First attempt at re-resolution, with the discussion that led to this patch](https://github.com/getsentry/self-hosted/pull/4295) +- [Second attempt, which changed the default configuration](https://github.com/getsentry/self-hosted/pull/4492)