From a5b668716b27bdf222d619766e147bc5e2550d4f Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Thu, 3 Sep 2026 22:52:43 +0200 Subject: [PATCH 1/5] infra(api): carry the frontend deploy's three edges back to the API MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit api/cloudbuild.yaml invented the candidate rollout; app/cloudbuild.yaml (#11207) then improved it and the improvements never came back. - `:latest` waits on `promote`, not on `build-image`. The tag used to reach the registry before the candidate was deployed, let alone smoked, so a failed smoke still left `:latest` on the image that failed it. - The smoke re-asserts the `candidate` tag after its probes as well as before. The tag is shared across builds; a concurrent one moving it mid-smoke would otherwise let this build promote a revision it only believed it probed. - The probes stop piping into `grep -q`, which exits at the first match and SIGPIPEs curl — the form only ever passed because `-ceu` carries no `pipefail`. They go through the same `expect` helper app/cloudbuild.yaml uses; `/health` keeps a bare variant, because reaching it without the origin header is what its gate exemption has to prove. The expanded script (Cloud Build `$$` resolved) passes `bash -n`, and the helpers were exercised against a local server for the happy path, a missing needle and a 404. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01UEScQMZFvxxNNyNJYryfa3 --- CHANGELOG.md | 16 +++++++++++ api/cloudbuild.yaml | 67 +++++++++++++++++++++++++++++++++++++-------- 2 files changed, 71 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0475c20e96..9de06d8d4b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -222,6 +222,22 @@ aggregate instead: an italic *Catalog* line at the end of the version section an ### Changed +- **The API deploy gets the three edges the frontend deploy already had** — the candidate + rollout `api/cloudbuild.yaml` invented was then improved in `app/cloudbuild.yaml` (#11207) + and the improvements never came back. Three of them do now. `:latest` waits on `promote` + instead of on `build-image`: it used to reach the registry before the candidate had been + deployed, let alone smoked, so a build whose smoke failed still left `:latest` naming the + image that failed it. The smoke re-asserts the `candidate` tag AFTER its probes as well as + before — the tag is shared across builds, so a concurrent one could move it mid-smoke and + this build would promote a revision it only believed it had probed; a competing build only + ever tags its own revision, so seeing ours at both ends proves every probe in between hit + it. And the probes stop piping into `grep -q`, which exits at the first match and SIGPIPEs + curl — the form only ever passed because `-ceu` carries no `pipefail`, so a later hardening + pass adding it would have turned every deploy red. They go through the same `expect` helper + `app/cloudbuild.yaml` uses, which fetches to a file and names the probe and the missing + needle when it fails; `/health` keeps a bare variant because reaching it without the origin + header is exactly what its gate exemption has to prove. (#PRNUM) + - **The frontend deploys through a candidate revision instead of straight onto live traffic** — `app/cloudbuild.yaml` now follows the same candidate-rollout pattern as `api/cloudbuild.yaml`: deploy with `--no-traffic --tag=candidate diff --git a/api/cloudbuild.yaml b/api/cloudbuild.yaml index 0e4042b8e7..80af0afe5a 100644 --- a/api/cloudbuild.yaml +++ b/api/cloudbuild.yaml @@ -40,14 +40,6 @@ steps: id: "push-image" waitFor: ["build-image"] - # Push latest tag - - name: "gcr.io/cloud-builders/docker" - args: - - "push" - - "europe-west4-docker.pkg.dev/$PROJECT_ID/anyplot/${_SERVICE_NAME}:latest" - id: "push-latest" - waitFor: ["build-image"] - # Deploy to Cloud Run - name: "gcr.io/cloud-builders/gcloud" args: @@ -146,9 +138,31 @@ steps: ORIGIN_SECRET=$$(gcloud secrets versions access latest --secret=ORIGIN_SECRET 2>/dev/null || true) HDR=() if [ -n "$$ORIGIN_SECRET" ]; then HDR=(-H "X-Origin-Secret: $$ORIGIN_SECRET"); fi + # `expect ` sends the origin header; + # `expect_bare` is the same probe without it. Both fetch to a file rather + # than piping into `grep -q`: `grep -q` exits at the first match and + # SIGPIPEs curl, so the pipe form only ever passed because `-ceu` carries + # no `pipefail` — a later hardening pass that adds it would red every + # deploy. Writing the body out also lets each failure name the probe and + # what was expected, which is the whole value of a log someone reads + # while a deploy is blocked. Same helper as app/cloudbuild.yaml. + _check() { + grep -qF "$$2" body.out || { echo "$$3 ($$1 is missing: $$2)"; exit 1; } + echo "OK: $$1" + } + expect() { + curl -fsS $$RETRY "$${HDR[@]}" -o body.out "$$URL$$1" \ + || { echo "candidate did not serve $$1"; exit 1; } + _check "$$1" "$$2" "$$3" + } + expect_bare() { + curl -fsS $$RETRY -o body.out "$$URL$$1" \ + || { echo "candidate did not serve $$1 without the origin header"; exit 1; } + _check "$$1" "$$2" "$$3" + } # /health stays bare: it is exempt from the gate, and that is what makes # it the probe that always reaches a cold candidate. - curl -fsS $$RETRY "$$URL/health" | grep -q '"healthy"' + expect_bare "/health" '"healthy"' "the candidate's /health did not report healthy" # …and that the secret this BUILD can read is the one the SERVICE was # given. /health reports the verdict for the request it was asked with # (never the value), so a rotation applied to only one of the two shows @@ -169,9 +183,11 @@ steps: # unreachable (optional_db), so they prove the app serves but not the # database. /plots/filter takes require_db — it is the probe that fails # when the Cloud SQL connection is broken. - curl -fsS $$RETRY "$${HDR[@]}" "$$URL/libraries" | grep -q '"libraries"' - curl -fsS $$RETRY "$${HDR[@]}" "$$URL/languages" | grep -q '"languages"' - curl -fsS $$RETRY "$${HDR[@]}" "$$URL/plots/filter" >/dev/null + expect "/libraries" '"libraries"' "the candidate did not serve the library list" + expect "/languages" '"languages"' "the candidate did not serve the language list" + curl -fsS $$RETRY "$${HDR[@]}" -o /dev/null "$$URL/plots/filter" \ + || { echo "/plots/filter failed — the candidate cannot reach Cloud SQL"; exit 1; } + echo "OK: /plots/filter" # Fail-closed admin gate. 401 is the answer with ADMIN_TOKEN present and # no header sent; a 503 here would mean the secret never arrived, which # is exactly the misconfiguration worth failing the build over. With the @@ -179,6 +195,17 @@ steps: # says "admin gate" about something that never reached it. code=$$(curl -s $$RETRY "$${HDR[@]}" -o /dev/null -w '%{http_code}' "$$URL/debug/status") test "$$code" = "401" || { echo "admin gate expected 401, got $$code"; exit 1; } + # Re-assert the tag AFTER the probes. `candidate` is a shared tag, so a + # concurrent build could move it between the check above and the last + # probe, and this build would then have smoked someone else's revision + # while promoting its own. A competing build only ever tags its OWN + # revision and never ours back, so seeing our revision at both ends means + # every probe in between hit it. (app/cloudbuild.yaml, #11207, gained this + # while the API — the service with the DB connection and the admin gate — + # still only checked once.) + REV_AFTER=$$(gcloud run services describe ${_SERVICE_NAME} --region=${_REGION} --platform=managed --format=json \ + | python3 -c "import json,sys; t=json.load(sys.stdin)['status']['traffic']; c=next(x for x in t if x.get('tag')=='candidate'); print(c['revisionName'])") + test "$$REV_AFTER" = "$$REV" || { echo "candidate tag moved to $$REV_AFTER mid-smoke (started on $$REV) — not promoting"; exit 1; } echo "smoke OK" id: "smoke" waitFor: ["deploy"] @@ -198,6 +225,22 @@ steps: id: "promote" waitFor: ["smoke"] + # `:latest` moves only after THIS build promoted, so it can no longer name an + # image that was never rolled out. It used to wait on `build-image`, which put + # the tag on the registry before the candidate had been deployed, let alone + # smoked: a build whose smoke failed still left `:latest` pointing at the image + # that failed it. Same fix and same wording as app/cloudbuild.yaml (#11207). + # It is not a guarantee across concurrent builds — two overlapping deploys race + # for the tag and the later push wins whichever revision serves. That is why the + # deploy step pulls `:$BUILD_ID` and never `:latest`; the tag is a convenience + # for humans, not an input to the rollout. + - name: "gcr.io/cloud-builders/docker" + args: + - "push" + - "europe-west4-docker.pkg.dev/$PROJECT_ID/anyplot/${_SERVICE_NAME}:latest" + id: "push-latest" + waitFor: ["promote"] + # Report deployed URL - name: "gcr.io/cloud-builders/gcloud" args: From 6a67a7f45acea972995b5c05c2023b4c8f1a6be1 Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Thu, 3 Sep 2026 22:53:17 +0200 Subject: [PATCH 2/5] docs(changelog): PR reference Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01UEScQMZFvxxNNyNJYryfa3 --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9de06d8d4b..152bd2dafa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -236,7 +236,7 @@ aggregate instead: an italic *Catalog* line at the end of the version section an pass adding it would have turned every deploy red. They go through the same `expect` helper `app/cloudbuild.yaml` uses, which fetches to a file and names the probe and the missing needle when it fails; `/health` keeps a bare variant because reaching it without the origin - header is exactly what its gate exemption has to prove. (#PRNUM) + header is exactly what its gate exemption has to prove. (#11212) - **The frontend deploys through a candidate revision instead of straight onto live traffic** — `app/cloudbuild.yaml` now follows the same candidate-rollout pattern as From 914b5b91b6afa99a6b870c01c1538344d33540c4 Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Thu, 3 Sep 2026 23:41:06 +0200 Subject: [PATCH 3/5] docs(api): say what the second tag read detects, not what it proves MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Copilot, twice and correctly: status.traffic is control-plane state. Both reads can name this revision while tag-URL propagation still sends a probe to the previous candidate — the residual app/cloudbuild.yaml already documents at its own smoke step. The check stays; the claim is now "the tag was never reassigned while the probes ran", with the residual and its mild worst case spelled out. Also: the frontend entry's parenthetical said the API pushes :latest alongside the deploy, which this PR is what changes. Put in the past tense and pointed at the entry above it. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01UEScQMZFvxxNNyNJYryfa3 --- CHANGELOG.md | 12 ++++++++---- api/cloudbuild.yaml | 19 +++++++++++++++---- 2 files changed, 23 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 152bd2dafa..002522bb16 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -230,8 +230,11 @@ aggregate instead: an italic *Catalog* line at the end of the version section an image that failed it. The smoke re-asserts the `candidate` tag AFTER its probes as well as before — the tag is shared across builds, so a concurrent one could move it mid-smoke and this build would promote a revision it only believed it had probed; a competing build only - ever tags its own revision, so seeing ours at both ends proves every probe in between hit - it. And the probes stop piping into `grep -q`, which exits at the first match and SIGPIPEs + ever tags its own revision, so ours at both ends means the tag was never reassigned while + the probes ran. That detects an observed reassignment rather than proving where a probe + landed: `status.traffic` is control-plane state, and tag-URL propagation can lag it, which + is the residual `app/cloudbuild.yaml` already documents at its own smoke step. And the + probes stop piping into `grep -q`, which exits at the first match and SIGPIPEs curl — the form only ever passed because `-ceu` carries no `pipefail`, so a later hardening pass adding it would have turned every deploy red. They go through the same `expect` helper `app/cloudbuild.yaml` uses, which fetches to a file and names the probe and the missing @@ -242,8 +245,9 @@ aggregate instead: an italic *Catalog* line at the end of the version section an traffic** — `app/cloudbuild.yaml` now follows the same candidate-rollout pattern as `api/cloudbuild.yaml`: deploy with `--no-traffic --tag=candidate --revision-suffix=b$BUILD_ID`, smoke the candidate on its tag URL, then `update-traffic` - to exactly that revision (the chains are not identical — this one pushes `:latest` only - after the promotion, where the API still pushes it alongside the deploy). The service + to exactly that revision. (The chains were not identical at the time — this one pushed + `:latest` only after the promotion, where the API still pushed it alongside the deploy; + the API caught up in the entry above.) The service carries the whole crawler path in `app/nginx.conf` — the `$is_bot` map, the `location =` bypasses, the `@seo_proxy` upstream — and that is the file whose breakage served every bot an HTTP 502 for four weeks in 2026 while humans, Plausible and CI all saw a healthy diff --git a/api/cloudbuild.yaml b/api/cloudbuild.yaml index 80af0afe5a..8e99937601 100644 --- a/api/cloudbuild.yaml +++ b/api/cloudbuild.yaml @@ -199,10 +199,21 @@ steps: # concurrent build could move it between the check above and the last # probe, and this build would then have smoked someone else's revision # while promoting its own. A competing build only ever tags its OWN - # revision and never ours back, so seeing our revision at both ends means - # every probe in between hit it. (app/cloudbuild.yaml, #11207, gained this - # while the API — the service with the DB connection and the admin gate — - # still only checked once.) + # revision and never ours back, so our revision at both ends means the + # tag was never reassigned while the probes ran. + # + # What that is and is not: `status.traffic` is CONTROL-PLANE state, so + # this detects an observed reassignment, it does not prove a given probe + # reached this revision. Tag-URL propagation can lag a reassignment, so a + # probe can still land on the previous candidate — the same residual + # app/cloudbuild.yaml documents at its own smoke step, and closing it + # would need a build-unique tag or a build id the service serves in its + # own response. The worst case stays mild: it needs the PREVIOUS + # candidate to have passed every probe too, so the failure is "promoted a + # revision we only believed we smoked", not "shipped a known-broken one". + # (app/cloudbuild.yaml, #11207, gained this second read while the API — + # the service with the DB connection and the admin gate — still checked + # once.) REV_AFTER=$$(gcloud run services describe ${_SERVICE_NAME} --region=${_REGION} --platform=managed --format=json \ | python3 -c "import json,sys; t=json.load(sys.stdin)['status']['traffic']; c=next(x for x in t if x.get('tag')=='candidate'); print(c['revisionName'])") test "$$REV_AFTER" = "$$REV" || { echo "candidate tag moved to $$REV_AFTER mid-smoke (started on $$REV) — not promoting"; exit 1; } From c73da7c8eff03b2089556f701a4c6dc14249a21a Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Fri, 4 Sep 2026 00:01:19 +0200 Subject: [PATCH 4/5] docs(changelog): move this PR's entry into a fragment MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The fragment convention landed on main in #11215 while this PR was open: a PR writes changelog.d/.md and leaves CHANGELOG.md alone. The one CHANGELOG.md edit that stays is not this PR's own entry but the correction of a PREVIOUS one — #11207's frontend bullet still says the API pushes :latest alongside the deploy, which is exactly what this PR changes. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01UEScQMZFvxxNNyNJYryfa3 --- CHANGELOG.md | 5 +++-- changelog.d/api-deploy-edges.md | 20 ++++++++++++++++++++ 2 files changed, 23 insertions(+), 2 deletions(-) create mode 100644 changelog.d/api-deploy-edges.md diff --git a/CHANGELOG.md b/CHANGELOG.md index f2d50a7ce7..6248130453 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -290,8 +290,9 @@ aggregate instead: an italic *Catalog* line at the end of the version section an traffic** — `app/cloudbuild.yaml` now follows the same candidate-rollout pattern as `api/cloudbuild.yaml`: deploy with `--no-traffic --tag=candidate --revision-suffix=b$BUILD_ID`, smoke the candidate on its tag URL, then `update-traffic` - to exactly that revision (the chains are not identical — this one pushes `:latest` only - after the promotion, where the API still pushes it alongside the deploy). The service + to exactly that revision. (The chains were not identical at the time — this one pushed + `:latest` only after the promotion, where the API still pushed it alongside the deploy; + the API caught up in a later entry of this same section.) The service carries the whole crawler path in `app/nginx.conf` — the `$is_bot` map, the `location =` bypasses, the `@seo_proxy` upstream — and that is the file whose breakage served every bot an HTTP 502 for four weeks in 2026 while humans, Plausible and CI all saw a healthy diff --git a/changelog.d/api-deploy-edges.md b/changelog.d/api-deploy-edges.md new file mode 100644 index 0000000000..11c442c29e --- /dev/null +++ b/changelog.d/api-deploy-edges.md @@ -0,0 +1,20 @@ +### Changed + +- **The API deploy gets the three edges the frontend deploy already had** — the candidate + rollout `api/cloudbuild.yaml` invented was then improved in `app/cloudbuild.yaml` (#11207) + and the improvements never came back. Three of them do now. `:latest` waits on `promote` + instead of on `build-image`: it used to reach the registry before the candidate had been + deployed, let alone smoked, so a build whose smoke failed still left `:latest` naming the + image that failed it. The smoke re-asserts the `candidate` tag AFTER its probes as well as + before — the tag is shared across builds, so a concurrent one could move it mid-smoke and + this build would promote a revision it only believed it had probed; a competing build only + ever tags its own revision, so ours at both ends means the tag was never reassigned while + the probes ran. That detects an observed reassignment rather than proving where a probe + landed: `status.traffic` is control-plane state, and tag-URL propagation can lag it, which + is the residual `app/cloudbuild.yaml` already documents at its own smoke step. And the + probes stop piping into `grep -q`, which exits at the first match and SIGPIPEs + curl — the form only ever passed because `-ceu` carries no `pipefail`, so a later hardening + pass adding it would have turned every deploy red. They go through the same `expect` helper + `app/cloudbuild.yaml` uses, which fetches to a file and names the probe and the missing + needle when it fails; `/health` keeps a bare variant because reaching it without the origin + header is exactly what its gate exemption has to prove. (#11212) From 96f936db0f4af3f22df5348897717f3eab449369 Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Fri, 4 Sep 2026 00:01:44 +0200 Subject: [PATCH 5/5] =?UTF-8?q?docs(changelog):=20leave=20the=20previous?= =?UTF-8?q?=20entry=20alone=20=E2=80=94=20the=20gate=20refuses=20it?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measured, not assumed: with the correction in place, `tools.changelog check --base origin/main` answers "CHANGELOG.md [Unreleased] gained a bullet". The fragment gate compares bullet SETS, so editing an existing bullet is indistinguishable from adding one. So #11207's frontend entry keeps its now-stale parenthetical ("the API still pushes :latest alongside the deploy") for the moment. It needs either a skip-changelog-labelled touch or a pass at the release cut, where [Unreleased] is edited by hand anyway. Reported rather than forced. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01UEScQMZFvxxNNyNJYryfa3 --- CHANGELOG.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6248130453..f2d50a7ce7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -290,9 +290,8 @@ aggregate instead: an italic *Catalog* line at the end of the version section an traffic** — `app/cloudbuild.yaml` now follows the same candidate-rollout pattern as `api/cloudbuild.yaml`: deploy with `--no-traffic --tag=candidate --revision-suffix=b$BUILD_ID`, smoke the candidate on its tag URL, then `update-traffic` - to exactly that revision. (The chains were not identical at the time — this one pushed - `:latest` only after the promotion, where the API still pushed it alongside the deploy; - the API caught up in a later entry of this same section.) The service + to exactly that revision (the chains are not identical — this one pushes `:latest` only + after the promotion, where the API still pushes it alongside the deploy). The service carries the whole crawler path in `app/nginx.conf` — the `$is_bot` map, the `location =` bypasses, the `@seo_proxy` upstream — and that is the file whose breakage served every bot an HTTP 502 for four weeks in 2026 while humans, Plausible and CI all saw a healthy