fix(GEN-3647): restore api.gen.pro docs routing after CloudFront exit

cloudflare/DEPLOY-no-worker.md

@@ -0,0 +1,55 @@
+# Alternative fix (NO Worker): Cloudflare Origin Rules + URL Rewrite
+
+Use this if you prefer not to run a Worker. It reproduces the CloudFront path-split
+with native Cloudflare rules. Slightly more moving parts than the Worker, but no code.
+
+> Goal is identical: `api.gen.pro/v1`,`/up`,… → Rails; everything else → GitHub Pages
+> `poweredbygen.github.io/api-docs/*` (with the `/api-docs` base-path prepend).
+
+## Prereq DNS
+Add an **unproxied** record so Rails is reachable as a distinct origin:
+- `origin-api.gen.pro` → `5.161.246.2`, **DNS only (grey cloud)**.
+
+## Step 1 — Send API paths to the Rails origin (Origin Rule)
+Rules → **Origin Rules** → Create rule:
+- **When incoming requests match:**
+  `(http.host eq "api.gen.pro" and (starts_with(http.request.uri.path, "/v1") or http.request.uri.path eq "/up" or starts_with(http.request.uri.path, "/users") or starts_with(http.request.uri.path, "/rails") or starts_with(http.request.uri.path, "/cable") or starts_with(http.request.uri.path, "/auth")))`
+- **Then — Override:** DNS record / Host header → `origin-api.gen.pro`
+  (keep the visitor Host `api.gen.pro` so Rails routing/cookies/CORS are unchanged).
+
+## Step 2 — Send everything else to GitHub Pages (Origin Rule)
+Origin Rules → Create rule (place AFTER step 1 so API wins):
+- **When:** `(http.host eq "api.gen.pro")`  *(catch-all; step 1's rule, ordered first, has already peeled off API paths)*
+- **Then — Override:**
+  - **Host header** → `poweredbygen.github.io`
+  - **(SNI)** → `poweredbygen.github.io`
+
+## Step 3 — Prepend `/api-docs` to the docs path (URL Rewrite / Transform Rule)
+Rules → **Transform Rules → Rewrite URL** → Create rule:
+- **When:** same catch-all as step 2 (api.gen.pro, non-API). To be safe, exclude API:
+  `(http.host eq "api.gen.pro" and not starts_with(http.request.uri.path, "/v1") and http.request.uri.path ne "/up" and not starts_with(http.request.uri.path, "/users") and not starts_with(http.request.uri.path, "/rails") and not starts_with(http.request.uri.path, "/cable") and not starts_with(http.request.uri.path, "/auth"))`
+- **Then — Rewrite to… Path → Dynamic:**
+  `concat("/api-docs", http.request.uri.path)`
+  This turns `/_astro/x.css` → `/api-docs/_astro/x.css` and `/llms.txt` → `/api-docs/llms.txt`, matching how Pages serves the project.
+
+## Step 4 — SSL mode
+Ensure the zone (or a Configuration Rule for api.gen.pro) uses **Full** SSL so CF
+talks HTTPS to both origins. GitHub Pages presents a valid cert; the Rails origin
+presents a self-signed/CF-Origin cert (Full, not Full-Strict, for that origin —
+or install a CF Origin Certificate on the box and use Full-Strict).
+
+## Caveat vs the Worker
+GitHub Pages issues **trailing-slash 301 redirects** whose `Location` points at
+`poweredbygen.github.io/api-docs/...`. The Worker rewrites those back to
+`api.gen.pro/...`; native rules do **not**. If you see the Pages hostname leak in a
+redirect, add a **Response Header Transform** rewriting `Location`, or just use the
+Worker (`api-gen-pro-router.worker.js`) which handles it in one artifact.
+
+## Verify (same as Worker path)
+```bash
+for p in / /llms.txt /llms-full.txt /openapi.yaml /reference/agents/; do
+  curl -s -o /dev/null -w "$p -> %{http_code}\n" "https://api.gen.pro$p"; done   # all 200
+curl -s -o /dev/null -w "/up -> %{http_code}\n"  https://api.gen.pro/up                  # 200
+curl -s -o /dev/null -w "/v1 -> %{http_code}\n"  https://api.gen.pro/v1/templates/projects # 200
+curl -s -o /dev/null -w "/agents -> %{http_code}\n" https://api.gen.pro/v1/agents         # 401
+```

cloudflare/DEPLOY.md

@@ -0,0 +1,53 @@
+# Fix: api.gen.pro docs site 404 (post-AWS-migration routing gap)
+
+## What broke
+`api.gen.pro` serves **two** things off one hostname:
+- API paths (`/v1`, `/up`, ...) → Rails (`gen-backend-v2`, now on Hetzner `5.161.246.2`)
+- Docs (`/`, `/llms.txt`, `/openapi.yaml`, `/guides/*`, `/reference/*`, `/cards/*`) → Astro Starlight on **GitHub Pages** (`poweredbygen.github.io/api-docs/`)
+
+A **CloudFront** distribution used to split these paths and rewrite the Pages origin
+path to `/api-docs`. The AWS migration removed CloudFront and pointed `api.gen.pro`
+**entirely at Rails**. Rails has no docs routes → every docs URL 404s.
+
+The API itself was never down. Docs content is healthy at `poweredbygen.github.io/api-docs/`.
+
+## The fix (Cloudflare Worker — no AWS, no redeploy)
+`api-gen-pro-router.worker.js` restores the split at the CF edge.
+
+### Steps (Cloudflare dashboard, zone `gen.pro`)
+1. **Create an unproxied origin record for Rails** so the Worker can reach the
+   backend without looping through `api.gen.pro/*`:
+   - DNS → add `origin-api` → `5.161.246.2`, **Proxy status: DNS only (grey cloud)**.
+   - (The box presents the `api.gen.pro` cert / a CF Origin cert; `fetch()` to
+     `https://origin-api.gen.pro` from the Worker resolves to the same box.)
+2. **Workers & Pages → Create Worker** → paste `api-gen-pro-router.worker.js` → Deploy.
+3. **Workers Routes** → add route `api.gen.pro/*` → select this Worker.
+4. Verify (below). Total time: a few minutes.
+
+### If you prefer NOT to add a Worker
+Equivalent with **Origin Rules + a Redirect/Transform Rule** is possible but messier
+because the `/api-docs` base-path rewrite + asset paths need a URL Rewrite rule.
+The Worker is the clean single-artifact fix.
+
+## Verify after deploy
+```bash
+# Docs must be 200 again:
+for p in / /llms.txt /llms-full.txt /openapi.yaml /reference/agents/; do
+  curl -s -o /dev/null -w "$p -> %{http_code}\n" "https://api.gen.pro$p"
+done
+# API must still work:
+curl -s -o /dev/null -w "/up -> %{http_code}\n"  https://api.gen.pro/up                 # 200
+curl -s -o /dev/null -w "/v1 -> %{http_code}\n"  https://api.gen.pro/v1/templates/projects # 200
+curl -s -o /dev/null -w "/v1/agents -> %{http_code}\n" https://api.gen.pro/v1/agents      # 401
+```
+
+## Permanent hardening (separate follow-up)
+The underlying fragility is `astro.config` has `site: 'https://api.gen.pro'` with **no
+`base:`** and **no `CNAME`** — it only ever worked because something rewrote the origin
+path. Long-term, either:
+- give the docs their **own subdomain** (`docs.gen.pro`) with a Pages custom-domain
+  CNAME (cleanest — removes the dual-purpose host entirely), or
+- set Astro `base: '/api-docs'` so published asset paths match the Pages path
+  (lets you drop the prepend logic).
+
+Both are larger changes; the Worker unblocks production now.

cloudflare/api-gen-pro-router.worker.js

@@ -0,0 +1,109 @@
+/**
+ * api.gen.pro edge router (Cloudflare Worker)
+ *
+ * WHY THIS EXISTS
+ * ---------------
+ * `api.gen.pro` is a dual-purpose hostname:
+ *   - API paths (/v1, /up, ...)  -> Rails backend (gen-backend-v2, Phusion Passenger)
+ *   - everything else (docs)     -> Astro Starlight docs site on GitHub Pages
+ *                                   (poweredbygen.github.io/api-docs/*)
+ *
+ * Before the AWS migration, a CloudFront distribution split these paths and
+ * rewrote the GitHub Pages origin path to `/api-docs`. When CloudFront was torn
+ * down, api.gen.pro was pointed entirely at Rails, so every docs URL started
+ * 404ing (Rails has no route for /llms.txt, /, /guides/*, etc.).
+ *
+ * This Worker restores that split at the Cloudflare edge — no AWS, no redeploy
+ * of the docs, no change to Rails.
+ *
+ * BIND THIS WORKER TO:  api.gen.pro/*   (Workers Routes)
+ *
+ * IMPORTANT NOTES
+ * ---------------
+ * 1. Rails origin is reached by IP with Host: api.gen.pro because the public
+ *    cert is terminated at the CF edge and the origin uses a self-signed/CF
+ *    Origin cert. Fetching the proxied hostname directly from the Worker would
+ *    loop back through this same route. We therefore target the origin via a
+ *    Cloudflare "Origin Rule" / a dedicated unproxied hostname instead. See
+ *    RAILS_ORIGIN below.
+ * 2. The published docs HTML uses ROOT-RELATIVE asset paths (/_astro/...), but
+ *    GitHub Pages only serves them under /api-docs/_astro/... . So we prepend
+ *    /api-docs to ALL non-API paths (including /_astro and /assets), which is
+ *    exactly what CloudFront used to do.
+ */
+
+// API path prefixes that must go to Rails. Everything else -> docs.
+const API_PREFIXES = [
+  '/v1',
+  '/up',
+  '/users',
+  '/rails',
+  '/cable',
+  '/admin',
+  '/sidekiq',
+  '/auth',
+];
+
+// Rails origin. Uses the existing UNPROXIED (grey-cloud) backend hostname
+// `origin-app1.gen.pro` (-> 5.161.246.2) so the Worker reaches Rails directly
+// without re-entering the api.gen.pro/* route. (`api.gen.pro` itself is a proxied
+// CNAME to this same host.)
+const RAILS_ORIGIN = 'https://origin-app1.gen.pro';
+
+// GitHub Pages docs origin + project base path.
+const DOCS_ORIGIN = 'https://poweredbygen.github.io';
+const DOCS_BASE = '/api-docs';
+
+function isApiPath(pathname) {
+  return API_PREFIXES.some(
+    (p) => pathname === p || pathname.startsWith(p + '/') || pathname.startsWith(p + '?'),
+  );
+}
+
+export default {
+  async fetch(request) {
+    const url = new URL(request.url);
+
+    // --- API traffic -> Rails (unchanged behavior) ---
+    if (isApiPath(url.pathname)) {
+      const originUrl = new URL(RAILS_ORIGIN);
+      originUrl.pathname = url.pathname;
+      originUrl.search = url.search;
+
+      const apiReq = new Request(originUrl.toString(), request);
+      // Preserve the public Host so Rails routing/cookies/CORS behave identically.
+      apiReq.headers.set('Host', 'api.gen.pro');
+      apiReq.headers.set('X-Forwarded-Host', 'api.gen.pro');
+      apiReq.headers.set('X-Forwarded-Proto', 'https');
+      return fetch(apiReq);
+    }
+
+    // --- Everything else -> GitHub Pages docs, with /api-docs prepended ---
+    const docsUrl = new URL(DOCS_ORIGIN);
+    docsUrl.pathname = DOCS_BASE + url.pathname; // /_astro/x -> /api-docs/_astro/x
+    docsUrl.search = url.search;
+
+    const docsReq = new Request(docsUrl.toString(), {
+      method: request.method,
+      headers: request.headers,
+      redirect: 'manual',
+    });
+
+    const resp = await fetch(docsReq);
+
+    // Rewrite GitHub Pages redirects (e.g. trailing-slash 301s point at
+    // poweredbygen.github.io/api-docs/...) back to api.gen.pro/... so the user
+    // never sees the Pages hostname or the /api-docs prefix.
+    if (resp.status >= 300 && resp.status < 400 && resp.headers.has('location')) {
+      const loc = resp.headers.get('location');
+      const rewritten = loc
+        .replace(DOCS_ORIGIN + DOCS_BASE, 'https://api.gen.pro')
+        .replace(DOCS_BASE + '/', '/');
+      const headers = new Headers(resp.headers);
+      headers.set('location', rewritten);
+      return new Response(resp.body, { status: resp.status, headers });
+    }
+
+    return resp;
+  },
+};