From 14fcbea4d4b4dea05a9d0711288482c93b961c64 Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 15:48:53 -0700 Subject: [PATCH 01/17] docs(specs): PR-side deploy verification design Co-Authored-By: Claude Fable 5.1 --- ...9-03-pr-side-deploy-verification-design.md | 203 ++++++++++++++++++ 1 file changed, 203 insertions(+) create mode 100644 docs/superpowers/specs/2026-09-03-pr-side-deploy-verification-design.md diff --git a/docs/superpowers/specs/2026-09-03-pr-side-deploy-verification-design.md b/docs/superpowers/specs/2026-09-03-pr-side-deploy-verification-design.md new file mode 100644 index 000000000..b26b2b0de --- /dev/null +++ b/docs/superpowers/specs/2026-09-03-pr-side-deploy-verification-design.md @@ -0,0 +1,203 @@ +# PR-side deploy verification — design + +**Date:** 2026-09-03 +**Status:** approved + +## Problem + +Landing #963 took nine fix PRs (#967, #970, #973, #974, #981, #982, #983, +#984, #987). Every one of them repaired a step that only the push-to-main +`deploy` job executes: promoting to Vercel, verifying the promoted Website +against production, building and smoking the cockpit redirect artifact. +PR CI cannot reach those steps, so each failure was discovered after merge, +one at a time, with production stalled in between. The Website did not +promote for two days. + +Two failure classes account for all nine: + +1. **Remote-target assumptions.** Tests and scripts that pass against local + servers but not against a deployed origin: local fixture ports, a + `localhost` route abort, a spec that never loaded under CJS. +2. **Platform behaviour.** Things only a real Vercel deployment exhibits: + team scope on `promote`, deployment protection on unaliased artifacts, the + CDN's consecutive-slash collapse, a missing preview-environment variable. + +## Goal + +Run the same verification the deploy job runs, on pull requests, against +real Vercel preview deployments, with full runtime parity so the Website's +embedded runtime handshake is exercised too. + +## Non-goals + +- Fork pull requests. Both lanes need repository secrets and skip on forks. +- Deleting stale preview deployments when a PR closes. +- Replacing the post-promotion verification on main. It stays; the PR lanes + are additional. + +## Facts the design rests on + +- The Website Vercel project is git-linked, but its automatic previews are + built by Vercel with the preview environment and embed the production + examples. The examples' `frame-ancestors` policy and the runtime bridge's + parent-origin allowlist contain only production origins plus the single + preview origin the deploy job registers when it assembles the examples. + The allowlist validator rejects wildcards by design. A git-integration + preview therefore cannot host a working runtime frame. Measured against a + main-equivalent artifact whose origin had rotated out of the allowlist: + 86 passed, 1 failed, 20 did not run (the workspace-shell file is serial). +- Any Vercel deployment, including a preview, can be aliased to an arbitrary + unused `.vercel.app` name through the CLI or the aliases API. Aliases + inherit deployment protection. +- Deployment protection answers every path on a protected deployment with + `302 -> vercel.com/sso-api`. Protection Bypass for Automation secrets are + issued per project. The Website and cockpit projects have one each; the + examples project has none. +- Playwright's `extraHTTPHeaders` is global to the browser context. A page + that embeds an iframe from a second protected project needs a second + secret sent only to that origin. +- `resolveRuntimeUrl` in `libs/cockpit-shell` reads + `NEXT_PUBLIC_COCKPIT_RUNTIME_BASE_URL` at build time, defaulting to + `https://examples.threadplane.ai`. +- `scripts/assemble-examples.ts` reads `RUNTIME_PARENT_PREVIEW_ORIGINS` and + bakes those origins into the examples' CSP and bridge allowlist. +- Deploy-job step timings from run 33777374419: Website build + deploy + 2m05s, examples assembly 8m00s, examples deploy 18s, Website suite 1m13s, + cockpit build 1m04s, cockpit deploy 13s, cockpit smoke 27s. + +## Design + +### Deterministic aliases + +Each lane run derives two alias names from the event: + +| event | Website alias | examples alias | +| --- | --- | --- | +| `pull_request` #N | `threadplane-pr-N-cacheplane.vercel.app` | `threadplane-examples-pr-N-cacheplane.vercel.app` | +| `merge_group` | `threadplane-mq--cacheplane.vercel.app` | `threadplane-examples-mq--cacheplane.vercel.app` | + +Because both names are known before anything is built, the examples can be +assembled with the Website alias in their parent-origin policy and the +Website can be built with the examples alias as its runtime base, in either +order. A later push to the same PR re-points both aliases at the new +deployments. + +### Job: `website-preview-e2e` ("Website — e2e (deployed preview)") + +- `needs: ci-scope`; runs when `github.event_name == 'push'` is false and + the `website_e2e` scope is true, and only for same-repo pull requests or + merge-group candidates. +- Steps: + 1. `npm ci`, install Chromium. + 2. Assemble the examples with `RUNTIME_PARENT_PREVIEW_ORIGINS` set to + `https://`. + 3. In `deploy/examples`: write `.vercel/project.json` for the examples + project, `vercel pull --environment=preview`, + `vercel deploy --prebuilt --yes` (no `--prod`), capture the URL, + `vercel alias set `. + 4. At the repo root: write `.vercel/project.json` for the Website project, + `vercel pull --environment=preview`, `vercel build` (no `--prod`) with + `NEXT_PUBLIC_COCKPIT_RUNTIME_BASE_URL=https://` and + `GROWTH_FORM_POLICY=growth_v1`, `vercel deploy --prebuilt --archive=tgz + --skip-domain --yes` (no `--prod`), capture the URL, + `vercel alias set `. + 5. Guard: fail with a provisioning message if either + `VERCEL_AUTOMATION_BYPASS_SECRET` or + `VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET` is empty. + 6. `BASE_URL=https:// npx nx e2e website --skip-nx-cache` + with both bypass secrets and + `RUNTIME_BYPASS_ORIGIN=https://` in the environment. +- Every `vercel` call passes `--token` and, for `alias`, `--scope` (alias + takes bare URLs, like `promote`). + +### Playwright config: per-origin bypass + +`createWebsitePlaywrightConfig` keeps `extraHTTPHeaders` for the Website +secret. When `VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET` and +`RUNTIME_BYPASS_ORIGIN` are both set, the config also registers a +`globalSetup` script and points `use.storageState` at a file the script +writes. The script requests +`${RUNTIME_BYPASS_ORIGIN}/?x-vercel-protection-bypass=&x-vercel-set-bypass-cookie=true` +with a Playwright request context, which makes Vercel answer with the +`_vercel_jwt` bypass cookie scoped to that origin, and saves the resulting +cookies as storage state. Every browser context then carries the examples +cookie, so the runtime iframe and its subresources load without any spec +importing a custom fixture. The examples secret travels only in that one +setup request; the Website secret keeps travelling only as the global +header. Nothing changes for local runs, production smoke, or the +post-promotion run, where the two variables are unset. + +The existing config unit test file gains cases for: `globalSetup` and +`storageState` are set only when both variables are set; the storage-state +path lives under the scratch output directory, never in the repo; the +examples secret never appears in `extraHTTPHeaders`. + +### Job: `cockpit-preview-smoke` ("Cockpit — immutable preview smoke") + +- `needs: ci-scope`; same event and same-repo guards; runs when the + `cockpit_deploy_smoke` scope is true. +- Steps: `npm ci`; write `.vercel/project.json` for the cockpit project; + `vercel pull --environment=preview`; `vercel build --local-config + vercel.cockpit.json` with `COCKPIT_WEBSITE_ORIGIN=https://threadplane.ai` + (no `--prod`); `vercel deploy --prebuilt --archive=tgz --skip-domain --yes + --env COCKPIT_WEBSITE_ORIGIN=https://threadplane.ai` (no `--prod`); guard + on `VERCEL_COCKPIT_AUTOMATION_BYPASS_SECRET`; run + `apps/cockpit/scripts/deploy-smoke.ts --mode preview --retries 20 + --retry-delay-ms 5000` against the captured URL. +- No alias is needed; the smoke takes the bare deployment URL. +- The token-free `cockpit-deploy-smoke` dry-run job stays unchanged so forks + keep a check. + +### Required gate + +Both jobs are added to `required-pr-checks.needs`, to its `RESULT_*` / +`SCOPE_*` env, and to the `require_scoped` table under the `website_e2e` +and `cockpit_deploy_smoke` scope keys respectively. On forks the jobs are +skipped, which `require_scoped` already tolerates. + +### Provisioning + +- Generate Protection Bypass for Automation on the `threadplane-examples` + Vercel project (`PATCH /v1/projects/{id}/protection-bypass` with + `{"generate":{}}`) and store the value as the repository secret + `VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET`. +- Secrets are resolved when a workflow run is created; a run that started + before the secret existed must be re-run. + +### Workflow guards (`scripts/ci-workflow.spec.mjs`) + +- Both new jobs exist, are scope-gated, and carry the same-repo guard. +- Neither preview deploy passes `--prod`; neither job contains + `vercel promote`. +- Both alias names are derived from the event and used consistently in the + assembly env, the build env, and the `alias set` calls. +- Each bypass secret is wired only to its own project's step. +- The required gate needs and aggregates both jobs. + +## Error handling + +- Missing secret: fail immediately with a message naming the secret and the + Vercel project, before any deployment is created. +- Alias collision or alias failure: fail the job; the deployment URL is + printed so the failure can be inspected. +- Vercel deploy failure: the step fails on the CLI's exit code; nothing is + aliased. +- Suite or smoke failure: reported as usual; the aliases keep pointing at + the deployments under test for inspection. + +## Testing + +- Unit: Playwright config cases above; workflow guard cases above. +- Integration: open a PR that touches only `apps/website/e2e` and confirm + the Website lane reaches Ready in the runtime handshake tests; open a PR + that touches only `apps/cockpit` and confirm the cockpit lane runs the + 399-probe smoke. Then, as a mutation check, temporarily reintroduce the + `localhost:4300` route abort from #983 on a branch and confirm the Website + lane goes red. + +## Cost + +Roughly 12 to 14 minutes of wall clock per PR run of the Website lane, +dominated by examples assembly, in parallel with existing lanes. About 3 +minutes for the cockpit lane. Three Vercel deployments per Website-lane run, +one per cockpit-lane run. From cabafc3625cfe44112f27a2d811e01da5b2bb3dd Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 16:29:24 -0700 Subject: [PATCH 02/17] docs(plans): PR-side deploy verification implementation plan Co-Authored-By: Claude Fable 5.1 --- .../2026-09-03-pr-side-deploy-verification.md | 952 ++++++++++++++++++ 1 file changed, 952 insertions(+) create mode 100644 docs/superpowers/plans/2026-09-03-pr-side-deploy-verification.md diff --git a/docs/superpowers/plans/2026-09-03-pr-side-deploy-verification.md b/docs/superpowers/plans/2026-09-03-pr-side-deploy-verification.md new file mode 100644 index 000000000..65878add1 --- /dev/null +++ b/docs/superpowers/plans/2026-09-03-pr-side-deploy-verification.md @@ -0,0 +1,952 @@ +# PR-side deploy verification Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Run the deploy job's Website and cockpit verification on pull requests against real Vercel preview deployments, with a working runtime iframe, so deploy-only failures surface before merge. + +**Architecture:** Two new scope-gated CI jobs. `website-preview-e2e` assembles the examples with a deterministic Website alias in their parent-origin policy, deploys them as an examples preview under a deterministic alias, builds and deploys the Website as a preview pointed at that examples alias, and runs the ordinary Playwright suite against it. A Playwright global setup seeds the examples origin's protection-bypass cookie as storage state so the iframe loads. `cockpit-preview-smoke` deploys a throwaway cockpit preview and runs the 399-probe smoke against it. Both join the required gate; workflow-guard tests pin the shape. + +**Tech Stack:** GitHub Actions, Vercel CLI (`pull`/`build`/`deploy`/`alias`), Playwright (`globalSetup`, `storageState`, `request`), vitest, `node:test` workflow guards. + +**Spec:** `docs/superpowers/specs/2026-09-03-pr-side-deploy-verification-design.md` + +**Conventions the engineer must know:** + +- Website unit tests run from the repo root with `npx nx test website`, or targeted with `cd apps/website && npx vitest run `. Do not pass `--root`; it breaks config paths. Running vitest from `apps/website` falsely fails `cockpit-retirement.spec.ts` (cwd-relative paths); ignore that file when running targeted, and confirm with `npx nx test website` before committing. +- Workflow guards run with `node --test --test-reporter=tap scripts/ci-workflow.spec.mjs`. +- Commit messages end with `Co-Authored-By: Claude Fable 5.1 `. +- Never `echo` a secret. Pipe values into `gh secret set`. +- `apps/website/test-results/` is gitignored; generated storage state goes there. + +--- + +## File structure + +| File | Responsibility | +| --- | --- | +| `apps/website/e2e/runtime-bypass-setup.ts` (new) | Playwright `globalSetup`: obtain the examples origin's `_vercel_jwt` bypass cookie and write storage state. Exports `buildRuntimeBypassUrl` and `RUNTIME_BYPASS_STORAGE_STATE`. | +| `apps/website/playwright.config.ts` (modify) | Register the setup and storage state only when `RUNTIME_BYPASS_ORIGIN` and `VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET` are both set. | +| `apps/website/src/playwright-config.spec.ts` (modify) | Unit tests for the conditional wiring. | +| `apps/website/src/runtime-bypass-setup.spec.ts` (new) | Unit tests for the URL builder and the early return. | +| `.github/workflows/ci.yml` (modify) | Add `cockpit-preview-smoke` and `website-preview-e2e` jobs; wire both into `required-pr-checks`. | +| `scripts/ci-workflow.spec.mjs` (modify) | Guards for both jobs and the gate wiring. | + +--- + +### Task 1: Runtime bypass global setup (URL builder + early return) + +**Files:** +- Create: `apps/website/e2e/runtime-bypass-setup.ts` +- Create: `apps/website/src/runtime-bypass-setup.spec.ts` + +- [ ] **Step 1: Write the failing tests** + +Create `apps/website/src/runtime-bypass-setup.spec.ts`: + +```ts +import { describe, expect, it } from 'vitest'; +import runtimeBypassSetup, { + RUNTIME_BYPASS_STORAGE_STATE, + buildRuntimeBypassUrl, + seedRuntimeBypass, +} from '../e2e/runtime-bypass-setup'; + +describe('runtime bypass setup', () => { + it('asks Vercel for the bypass cookie at the runtime origin root', () => { + expect( + buildRuntimeBypassUrl( + 'https://threadplane-examples-pr-7-cacheplane.vercel.app', + 'examples-secret' + ) + ).toBe( + 'https://threadplane-examples-pr-7-cacheplane.vercel.app/?x-vercel-protection-bypass=examples-secret&x-vercel-set-bypass-cookie=true' + ); + }); + + it('rejects a runtime origin that is not a bare https origin', () => { + for (const origin of [ + 'http://threadplane-examples-pr-7-cacheplane.vercel.app', + 'https://threadplane-examples-pr-7-cacheplane.vercel.app/langgraph', + 'https://user:pw@threadplane-examples-pr-7-cacheplane.vercel.app', + ]) { + expect(() => buildRuntimeBypassUrl(origin, 'examples-secret')).toThrow( + /bare https origin/ + ); + } + }); + + it('does nothing when the origin or the secret is unset', async () => { + await expect(seedRuntimeBypass({})).resolves.toBe('skipped'); + await expect( + seedRuntimeBypass({ RUNTIME_BYPASS_ORIGIN: 'https://x.vercel.app' }) + ).resolves.toBe('skipped'); + await expect( + seedRuntimeBypass({ VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET: 's' }) + ).resolves.toBe('skipped'); + }); + + it('ignores the config object Playwright hands to a global setup', async () => { + // Playwright calls the default export with its FullConfig. Reading the + // environment from that argument would silently skip the seeding. + const saved = { ...process.env }; + delete process.env['RUNTIME_BYPASS_ORIGIN']; + delete process.env['VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET']; + try { + await expect( + runtimeBypassSetup({ + RUNTIME_BYPASS_ORIGIN: 'https://x.vercel.app', + VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET: 's', + } as never) + ).resolves.toBe('skipped'); + } finally { + process.env = saved; + } + }); + + it('writes storage state under the gitignored test-results directory', () => { + expect(RUNTIME_BYPASS_STORAGE_STATE).toMatch( + /apps\/website\/test-results\/runtime-bypass-storage-state\.json$/ + ); + }); +}); +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `cd apps/website && npx vitest run runtime-bypass-setup` +Expected: FAIL with `Failed to resolve import "../e2e/runtime-bypass-setup"`. + +- [ ] **Step 3: Write the setup module** + +Create `apps/website/e2e/runtime-bypass-setup.ts`: + +```ts +import { request } from '@playwright/test'; +import { mkdirSync, writeFileSync } from 'node:fs'; +import { dirname, resolve } from 'node:path'; + +/** + * Playwright globalSetup for runs whose Website preview embeds a runtime + * from a second protected Vercel project. Deployment protection answers + * every path on that origin with 302 -> vercel.com/sso-api, and Playwright's + * extraHTTPHeaders is global, so the Website project's secret would reach + * the runtime origin and be rejected. Vercel issues a per-origin `_vercel_jwt` + * bypass cookie when a request carries the owning project's secret together + * with `x-vercel-set-bypass-cookie=true`; this setup obtains that cookie once + * and stores it as storage state, so every browser context carries it and + * the runtime iframe and its subresources load. The examples secret travels + * only in this one request. + */ +export const RUNTIME_BYPASS_STORAGE_STATE = resolve( + __dirname, + '..', + 'test-results', + 'runtime-bypass-storage-state.json' +); + +type SetupEnvironment = Readonly>; + +export function buildRuntimeBypassUrl(origin: string, secret: string): string { + let parsed: URL; + try { + parsed = new URL(origin); + } catch { + throw new Error(`RUNTIME_BYPASS_ORIGIN must be a bare https origin, received ${origin}`); + } + if ( + parsed.protocol !== 'https:' || + parsed.username || + parsed.password || + parsed.pathname !== '/' || + parsed.search || + parsed.hash + ) { + throw new Error(`RUNTIME_BYPASS_ORIGIN must be a bare https origin, received ${origin}`); + } + const url = new URL('/', parsed.origin); + url.searchParams.set('x-vercel-protection-bypass', secret); + url.searchParams.set('x-vercel-set-bypass-cookie', 'true'); + return url.toString(); +} + +export async function seedRuntimeBypass( + environment: SetupEnvironment +): Promise<'skipped' | 'seeded'> { + const origin = environment['RUNTIME_BYPASS_ORIGIN']; + const secret = environment['VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET']; + if (!origin || !secret) return 'skipped'; + + const context = await request.newContext(); + try { + const response = await context.get(buildRuntimeBypassUrl(origin, secret), { + maxRedirects: 0, + }); + const state = await context.storageState(); + const seeded = state.cookies.some((cookie) => cookie.name === '_vercel_jwt'); + if (!seeded) { + throw new Error( + `Runtime bypass setup: ${origin} answered ${response.status()} without a _vercel_jwt cookie. Check VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET against the Vercel project that owns that origin.` + ); + } + mkdirSync(dirname(RUNTIME_BYPASS_STORAGE_STATE), { recursive: true }); + writeFileSync(RUNTIME_BYPASS_STORAGE_STATE, JSON.stringify(state)); + return 'seeded'; + } finally { + await context.dispose(); + } +} + +// Playwright calls globalSetup with its FullConfig as the only argument. +// Read the environment from the process, never from that argument. +export default async function runtimeBypassSetup(): Promise<'skipped' | 'seeded'> { + return seedRuntimeBypass(process.env); +} +``` + +- [ ] **Step 4: Run the tests to verify they pass** + +Run: `cd apps/website && npx vitest run runtime-bypass-setup` +Expected: `Tests 5 passed (5)`. + +- [ ] **Step 5: Commit** + +```bash +git add apps/website/e2e/runtime-bypass-setup.ts apps/website/src/runtime-bypass-setup.spec.ts +git commit -m "feat(website): playwright global setup that seeds the runtime origin bypass cookie + +Co-Authored-By: Claude Fable 5.1 " +``` + +--- + +### Task 2: Wire the setup into the Playwright config conditionally + +**Files:** +- Modify: `apps/website/playwright.config.ts` +- Modify: `apps/website/src/playwright-config.spec.ts` + +- [ ] **Step 1: Write the failing tests** + +Append inside the `describe('Website Playwright configuration', ...)` block in `apps/website/src/playwright-config.spec.ts`, after the test named `'holds the runtime frame by its session params rather than the local host'`: + +```ts + it('seeds the runtime origin bypass only when both the origin and the examples secret are set', () => { + const base = createWebsitePlaywrightConfig({ + BASE_URL: 'https://threadplane-pr-7-cacheplane.vercel.app', + VERCEL_AUTOMATION_BYPASS_SECRET: 'website-secret', + }); + expect(base.globalSetup).toBeUndefined(); + expect(base.use?.storageState).toBeUndefined(); + + const originOnly = createWebsitePlaywrightConfig({ + BASE_URL: 'https://threadplane-pr-7-cacheplane.vercel.app', + VERCEL_AUTOMATION_BYPASS_SECRET: 'website-secret', + RUNTIME_BYPASS_ORIGIN: 'https://threadplane-examples-pr-7-cacheplane.vercel.app', + }); + expect(originOnly.globalSetup).toBeUndefined(); + + const both = createWebsitePlaywrightConfig({ + BASE_URL: 'https://threadplane-pr-7-cacheplane.vercel.app', + VERCEL_AUTOMATION_BYPASS_SECRET: 'website-secret', + VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET: 'examples-secret', + RUNTIME_BYPASS_ORIGIN: 'https://threadplane-examples-pr-7-cacheplane.vercel.app', + }); + expect(both.globalSetup).toMatch(/runtime-bypass-setup\.ts$/); + expect(both.use?.storageState).toMatch( + /apps\/website\/test-results\/runtime-bypass-storage-state\.json$/ + ); + // The examples secret must never ride the global header, which reaches + // the Website origin on every request. + expect(JSON.stringify(both.use?.extraHTTPHeaders)).not.toContain('examples-secret'); + expect(both.use?.extraHTTPHeaders).toEqual({ + 'x-vercel-protection-bypass': 'website-secret', + 'x-vercel-set-bypass-cookie': 'true', + }); + }); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cd apps/website && npx vitest run playwright-config` +Expected: 1 failed: `expected undefined to match /runtime-bypass-setup\.ts$/`. + +- [ ] **Step 3: Implement the conditional wiring** + +In `apps/website/playwright.config.ts`: + +Add after the existing import line: + +```ts +import { resolve } from 'node:path'; +import { RUNTIME_BYPASS_STORAGE_STATE } from './e2e/runtime-bypass-setup'; +``` + +Add after `const reuseExistingServer = ...;`: + +```ts + // A PR preview embeds its runtime from a second protected Vercel project. + // extraHTTPHeaders is global, so that origin needs its own bypass, seeded + // once as a cookie by e2e/runtime-bypass-setup.ts. Both variables must be + // present; the deploy job and local runs set neither. + const runtimeBypass = Boolean( + environment['RUNTIME_BYPASS_ORIGIN'] && + environment['VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET'] + ); +``` + +Inside `defineConfig({ ... })`, add directly after the `testIgnore,` line: + +```ts + globalSetup: runtimeBypass + ? resolve(__dirname, 'e2e', 'runtime-bypass-setup.ts') + : undefined, +``` + +Inside the `use: { ... }` object, add directly after the `trace: 'off',` / `video: 'off',` lines: + +```ts + ...(runtimeBypass ? { storageState: RUNTIME_BYPASS_STORAGE_STATE } : {}), +``` + +- [ ] **Step 4: Run the config tests to verify they pass** + +Run: `cd apps/website && npx vitest run playwright-config runtime-bypass-setup` +Expected: `Tests 16 passed (16)`. + +- [ ] **Step 5: Confirm the ordinary suite still collects and lint is clean** + +Run: `npx playwright test --config apps/website/playwright.config.ts --list | tail -1` +Expected: a line ending in `Total: 107 tests in 12 files` (count may differ by a few; there must be no error). + +Run: `npx nx lint website --skip-nx-cache 2>&1 | grep -E "problems|error" | tail -1` +Expected: `0 errors`. + +Run: `npx nx test website --skip-nx-cache 2>&1 | tail -3` +Expected: `Successfully ran target test for project website`. + +- [ ] **Step 6: Commit** + +```bash +git add apps/website/playwright.config.ts apps/website/src/playwright-config.spec.ts +git commit -m "feat(website): opt-in runtime bypass storage state for protected previews + +Co-Authored-By: Claude Fable 5.1 " +``` + +--- + +### Task 3: `cockpit-preview-smoke` job + +**Files:** +- Modify: `.github/workflows/ci.yml` (insert after the `cockpit-deploy-smoke` job, before `examples-chat-smoke`) +- Modify: `scripts/ci-workflow.spec.mjs` + +- [ ] **Step 1: Write the failing guard test** + +Append inside the `describe('CI workflow', ...)` block in `scripts/ci-workflow.spec.mjs`, after the test named `'verifies every protected immutable preview with its own automation bypass'`: + +```js + it('smokes a throwaway cockpit preview on same-repo PRs and queue candidates', async () => { + const workflow = await readWorkflow(); + const job = readJobBlock(workflow, 'cockpit-preview-smoke'); + const ifBlock = readJobFieldBlock(job, 'if'); + + assert.deepEqual(readJobNeeds(job), ['ci-scope']); + assert.match(ifBlock, /github\.event_name != 'push'/); + assert.match(ifBlock, /needs\.ci-scope\.outputs\.cockpit_deploy_smoke == 'true'/); + assert.match( + ifBlock, + /github\.event_name == 'merge_group' \|\| github\.event\.pull_request\.head\.repo\.full_name == github\.repository/ + ); + + const prepare = readNamedStep(job, 'Prepare cockpit Vercel project (preview)'); + const build = readNamedStep(job, 'Build cockpit redirect service (preview)'); + const deploy = readNamedStep(job, 'Deploy throwaway cockpit preview'); + const smoke = readNamedStep(job, 'Exhaustively verify the cockpit preview'); + + assert.match(prepare, /"projectName":"threadplane-cockpit"/); + assert.match(prepare, /vercel pull --yes --environment=preview/); + assert.match(build, /vercel build --local-config vercel\.cockpit\.json/); + assert.doesNotMatch(build, /--prod/); + assert.match(deploy, /id:\s*deploy_cockpit_preview/); + assert.match(deploy, /vercel deploy --prebuilt --archive=tgz --skip-domain --yes/); + assert.doesNotMatch(deploy, /--prod/); + assert.match(deploy, /--env COCKPIT_WEBSITE_ORIGIN=https:\/\/threadplane\.ai/); + assert.match( + smoke, + /VERCEL_AUTOMATION_BYPASS_SECRET:\s*\$\{\{ secrets\.VERCEL_COCKPIT_AUTOMATION_BYPASS_SECRET \}\}/ + ); + assert.match(smoke, /-z "\$\{VERCEL_AUTOMATION_BYPASS_SECRET\}"/); + assert.match( + smoke, + /--url "\$\{\{ steps\.deploy_cockpit_preview\.outputs\.deployment_url \}\}"[\s\S]*--mode preview/ + ); + assert.doesNotMatch(job, /vercel promote/); + }); +``` + +- [ ] **Step 2: Run the guard to verify it fails** + +Run: `node --test --test-reporter=tap scripts/ci-workflow.spec.mjs 2>&1 | grep -E "^not ok|^# (pass|fail)"` +Expected: `not ok ... smokes a throwaway cockpit preview` and `# fail 1`. + +- [ ] **Step 3: Add the job to `ci.yml`** + +Insert the following block directly after the `cockpit-deploy-smoke` job (after its last `run:` line) and before ` examples-chat-smoke:`: + +```yaml + cockpit-preview-smoke: + name: Cockpit — immutable preview smoke + timeout-minutes: 20 + needs: ci-scope + # PR-side twin of the deploy job's cockpit verification. Deploys a + # throwaway preview of the redirect service and runs the exhaustive smoke + # against it, so platform behaviour (deployment protection, the CDN's + # slash collapse, route ordering) is exercised before merge. Needs + # repository secrets, so same-repo PRs and merge-queue candidates only; + # the token-free dry-run job above still covers forks. + if: >- + github.event_name != 'push' && + needs.ci-scope.outputs.cockpit_deploy_smoke == 'true' && + (github.event_name == 'merge_group' || github.event.pull_request.head.repo.full_name == github.repository) + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + - uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 + with: + node-version: 22 + cache: npm + - run: npm ci + - name: Prepare cockpit Vercel project (preview) + run: | + mkdir -p .vercel + cat > .vercel/project.json <> "$GITHUB_OUTPUT" + - name: Exhaustively verify the cockpit preview + run: | + if [ -z "${VERCEL_AUTOMATION_BYPASS_SECRET}" ]; then + echo "::error::VERCEL_COCKPIT_AUTOMATION_BYPASS_SECRET is unset — the protected cockpit preview cannot be verified. Enable 'Protection Bypass for Automation' on the Vercel threadplane-cockpit project and store the value as this repository secret." + exit 1 + fi + npx tsx apps/cockpit/scripts/deploy-smoke.ts --url "${{ steps.deploy_cockpit_preview.outputs.deployment_url }}" --mode preview --retries 20 --retry-delay-ms 5000 + env: + VERCEL_AUTOMATION_BYPASS_SECRET: ${{ secrets.VERCEL_COCKPIT_AUTOMATION_BYPASS_SECRET }} +``` + +- [ ] **Step 4: Run the guards and validate YAML** + +Run: `node --test --test-reporter=tap scripts/ci-workflow.spec.mjs 2>&1 | grep -E "^not ok|^# (pass|fail)"` +Expected: `# fail 0`. + +Run: `python3 -c "import yaml;yaml.safe_load(open('.github/workflows/ci.yml'));print('yaml ok')"` +Expected: `yaml ok`. + +- [ ] **Step 5: Commit** + +```bash +git add .github/workflows/ci.yml scripts/ci-workflow.spec.mjs +git commit -m "ci: smoke a throwaway cockpit preview on pull requests + +Co-Authored-By: Claude Fable 5.1 " +``` + +--- + +### Task 4: `website-preview-e2e` job + +**Files:** +- Modify: `.github/workflows/ci.yml` (insert after the `website-e2e` job, before `required-pr-checks`) +- Modify: `scripts/ci-workflow.spec.mjs` + +- [ ] **Step 1: Write the failing guard test** + +Append inside the `describe('CI workflow', ...)` block in `scripts/ci-workflow.spec.mjs`, after the cockpit-preview test from Task 3: + +```js + it('runs the Website suite against a deterministic aliased preview with a matching examples preview', async () => { + const workflow = await readWorkflow(); + const job = readJobBlock(workflow, 'website-preview-e2e'); + const ifBlock = readJobFieldBlock(job, 'if'); + + assert.deepEqual(readJobNeeds(job), ['ci-scope']); + assert.match(ifBlock, /github\.event_name != 'push'/); + assert.match(ifBlock, /needs\.ci-scope\.outputs\.website_e2e == 'true'/); + assert.match( + ifBlock, + /github\.event_name == 'merge_group' \|\| github\.event\.pull_request\.head\.repo\.full_name == github\.repository/ + ); + + const aliases = readNamedStep(job, 'Derive deterministic preview aliases'); + const guard = readNamedStep(job, 'Require preview bypass secrets'); + const assemble = readNamedStep(job, 'Build and assemble Angular examples for the preview'); + const examples = readNamedStep(job, 'Deploy examples preview and alias it'); + const website = readNamedStep(job, 'Build, deploy, and alias the Website preview'); + const suite = readNamedStep(job, 'Run the Website suite against the aliased preview'); + + assert.match(aliases, /id:\s*aliases/); + assert.match(aliases, /key="pr-\$\{\{ github\.event\.pull_request\.number \}\}"/); + assert.match(aliases, /key="mq-\$\(echo "\$\{\{ github\.event\.merge_group\.head_sha \}\}" \| cut -c1-8\)"/); + assert.match(aliases, /website=threadplane-\$\{key\}-cacheplane\.vercel\.app/); + assert.match(aliases, /examples=threadplane-examples-\$\{key\}-cacheplane\.vercel\.app/); + + assert.match(guard, /-z "\$\{VERCEL_AUTOMATION_BYPASS_SECRET\}"/); + assert.match(guard, /-z "\$\{VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET\}"/); + assert.match(guard, /VERCEL_AUTOMATION_BYPASS_SECRET:\s*\$\{\{ secrets\.VERCEL_AUTOMATION_BYPASS_SECRET \}\}/); + assert.match(guard, /VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET:\s*\$\{\{ secrets\.VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET \}\}/); + + // The examples are assembled with the Website alias in their policy and + // the Website is built with the examples alias as its runtime base. + assert.match( + assemble, + /RUNTIME_PARENT_PREVIEW_ORIGINS:\s*https:\/\/\$\{\{ steps\.aliases\.outputs\.website \}\}/ + ); + assert.match(examples, /working-directory:\s*deploy\/examples/); + assert.match(examples, /"projectName":"threadplane-examples"/); + assert.match(examples, /vercel pull --yes --environment=preview/); + assert.match(examples, /vercel deploy --prebuilt --yes/); + assert.doesNotMatch(examples, /--prod/); + assert.match( + examples, + /vercel alias set "\$url" "\$\{\{ steps\.aliases\.outputs\.examples \}\}" --scope=\$\{\{ secrets\.VERCEL_ORG_ID \}\}/ + ); + + assert.match(website, /"projectName":"threadplane"/); + assert.match(website, /vercel pull --yes --environment=preview/); + assert.match( + website, + /NEXT_PUBLIC_COCKPIT_RUNTIME_BASE_URL:\s*https:\/\/\$\{\{ steps\.aliases\.outputs\.examples \}\}/ + ); + assert.match(website, /GROWTH_FORM_POLICY:\s*growth_v1/); + assert.match(website, /vercel build --token/); + assert.match(website, /vercel deploy --prebuilt --archive=tgz --skip-domain --yes/); + assert.doesNotMatch(website, /--prod/); + assert.match( + website, + /vercel alias set "\$url" "\$\{\{ steps\.aliases\.outputs\.website \}\}" --scope=\$\{\{ secrets\.VERCEL_ORG_ID \}\}/ + ); + + assert.match(suite, /BASE_URL:\s*https:\/\/\$\{\{ steps\.aliases\.outputs\.website \}\}/); + assert.match(suite, /RUNTIME_BYPASS_ORIGIN:\s*https:\/\/\$\{\{ steps\.aliases\.outputs\.examples \}\}/); + assert.match(suite, /VERCEL_AUTOMATION_BYPASS_SECRET:\s*\$\{\{ secrets\.VERCEL_AUTOMATION_BYPASS_SECRET \}\}/); + assert.match(suite, /VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET:\s*\$\{\{ secrets\.VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET \}\}/); + assert.match(suite, /npx nx e2e website --skip-nx-cache/); + assert.doesNotMatch(job, /vercel promote/); + }); +``` + +- [ ] **Step 2: Run the guard to verify it fails** + +Run: `node --test --test-reporter=tap scripts/ci-workflow.spec.mjs 2>&1 | grep -E "^not ok|^# (pass|fail)"` +Expected: `not ok ... deterministic aliased preview` and `# fail 1`. + +- [ ] **Step 3: Add the job to `ci.yml`** + +Insert directly after the `website-e2e` job (after its last step, the `Production-smoke spec must load` step's `run:` block) and before ` required-pr-checks:`: + +```yaml + website-preview-e2e: + name: Website — e2e (deployed preview) + timeout-minutes: 40 + needs: ci-scope + # PR-side twin of the deploy job's post-promotion verification. The + # ordinary suite runs against a real, protected Vercel preview of the + # Website whose runtime iframe loads from a matching examples preview, so + # remote-target assumptions and platform behaviour surface before merge. + # Two deterministic aliases break the ordering problem: the examples are + # assembled with the Website alias in their parent-origin policy, and the + # Website is built with the examples alias as its runtime base. Needs + # repository secrets, so same-repo PRs and merge-queue candidates only. + if: >- + github.event_name != 'push' && + needs.ci-scope.outputs.website_e2e == 'true' && + (github.event_name == 'merge_group' || github.event.pull_request.head.repo.full_name == github.repository) + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + - uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 + with: + node-version: 22 + cache: npm + - run: npm ci + - name: Cache Playwright browsers + uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + with: + path: ~/.cache/ms-playwright + key: playwright-${{ runner.os }}-${{ hashFiles('package-lock.json') }} + restore-keys: | + playwright-${{ runner.os }}- + - run: npx playwright install --with-deps chromium + - name: Derive deterministic preview aliases + id: aliases + run: | + if [ "${{ github.event_name }}" = "merge_group" ]; then + key="mq-$(echo "${{ github.event.merge_group.head_sha }}" | cut -c1-8)" + else + key="pr-${{ github.event.pull_request.number }}" + fi + echo "website=threadplane-${key}-cacheplane.vercel.app" >> "$GITHUB_OUTPUT" + echo "examples=threadplane-examples-${key}-cacheplane.vercel.app" >> "$GITHUB_OUTPUT" + - name: Require preview bypass secrets + # Both previews sit behind deployment protection, and bypass secrets + # are issued per Vercel project. Fail before creating anything. + run: | + if [ -z "${VERCEL_AUTOMATION_BYPASS_SECRET}" ]; then + echo "::error::VERCEL_AUTOMATION_BYPASS_SECRET is unset — the protected Website preview cannot be verified. Enable 'Protection Bypass for Automation' on the Vercel threadplane project and store the value as this repository secret." + exit 1 + fi + if [ -z "${VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET}" ]; then + echo "::error::VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET is unset — the runtime iframe from the protected examples preview cannot load. Enable 'Protection Bypass for Automation' on the Vercel threadplane-examples project and store the value as this repository secret." + exit 1 + fi + env: + VERCEL_AUTOMATION_BYPASS_SECRET: ${{ secrets.VERCEL_AUTOMATION_BYPASS_SECRET }} + VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET: ${{ secrets.VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET }} + - name: Build and assemble Angular examples for the preview + run: npx tsx scripts/assemble-examples.ts + env: + RUNTIME_PARENT_PREVIEW_ORIGINS: https://${{ steps.aliases.outputs.website }} + - name: Deploy examples preview and alias it + working-directory: deploy/examples + run: | + mkdir -p .vercel + cat > .vercel/project.json < .vercel/project.json <&1 | grep -E "^not ok|^# (pass|fail)"` +Expected: `# fail 0`. + +Run: `python3 -c "import yaml;yaml.safe_load(open('.github/workflows/ci.yml'));print('yaml ok')"` +Expected: `yaml ok`. + +- [ ] **Step 5: Commit** + +```bash +git add .github/workflows/ci.yml scripts/ci-workflow.spec.mjs +git commit -m "ci: run the Website suite against an aliased preview with a matching examples preview + +Co-Authored-By: Claude Fable 5.1 " +``` + +--- + +### Task 5: Wire both jobs into the required gate + +**Files:** +- Modify: `.github/workflows/ci.yml` (`required-pr-checks` job) +- Modify: `scripts/ci-workflow.spec.mjs` + +- [ ] **Step 1: Write the failing guard test** + +Append inside the `describe('CI workflow', ...)` block in `scripts/ci-workflow.spec.mjs`, after the Task 4 test: + +```js + it('requires both PR-side preview verifications through the scoped gate', async () => { + const required = await readRequiredPrChecksJob(); + const needs = readJobNeeds(required); + + assert.ok(needs.includes('website-preview-e2e')); + assert.ok(needs.includes('cockpit-preview-smoke')); + assert.match( + required, + /RESULT_WEBSITE_PREVIEW_E2E:\s*\$\{\{ needs\.website-preview-e2e\.result \}\}/ + ); + assert.match( + required, + /RESULT_COCKPIT_PREVIEW_SMOKE:\s*\$\{\{ needs\.cockpit-preview-smoke\.result \}\}/ + ); + assert.match( + required, + /require_scoped "website_e2e" "Website — e2e \(deployed preview\)" "\$RESULT_WEBSITE_PREVIEW_E2E" "\$SCOPE_WEBSITE_E2E"/ + ); + assert.match( + required, + /require_scoped "cockpit_deploy_smoke" "Cockpit — immutable preview smoke" "\$RESULT_COCKPIT_PREVIEW_SMOKE" "\$SCOPE_COCKPIT_DEPLOY_SMOKE"/ + ); + }); +``` + +Note on forks: both jobs skip there, `require_scoped` treats `skipped` as acceptable, so the gate stays green on forks without these lanes. + +- [ ] **Step 2: Run the guard to verify it fails** + +Run: `node --test --test-reporter=tap scripts/ci-workflow.spec.mjs 2>&1 | grep -E "^not ok|^# (pass|fail)"` +Expected: `not ok ... requires both PR-side preview verifications` and `# fail 1`. + +- [ ] **Step 3: Edit the `required-pr-checks` job** + +In the `needs:` list, add after ` - website-e2e`: + +```yaml + - website-preview-e2e + - cockpit-preview-smoke +``` + +In the step `env:` block, add after `RESULT_WEBSITE_E2E: ${{ needs.website-e2e.result }}`: + +```yaml + RESULT_WEBSITE_PREVIEW_E2E: ${{ needs.website-preview-e2e.result }} + RESULT_COCKPIT_PREVIEW_SMOKE: ${{ needs.cockpit-preview-smoke.result }} +``` + +In the `run:` script, add after the line `require_scoped "website_e2e" "Website — e2e" "$RESULT_WEBSITE_E2E" "$SCOPE_WEBSITE_E2E"`: + +```bash + require_scoped "website_e2e" "Website — e2e (deployed preview)" "$RESULT_WEBSITE_PREVIEW_E2E" "$SCOPE_WEBSITE_E2E" + require_scoped "cockpit_deploy_smoke" "Cockpit — immutable preview smoke" "$RESULT_COCKPIT_PREVIEW_SMOKE" "$SCOPE_COCKPIT_DEPLOY_SMOKE" +``` + +- [ ] **Step 4: Run the guards and validate YAML** + +Run: `node --test --test-reporter=tap scripts/ci-workflow.spec.mjs 2>&1 | grep -E "^not ok|^# (pass|fail)"` +Expected: `# fail 0`. + +Run: `python3 -c "import yaml;yaml.safe_load(open('.github/workflows/ci.yml'));print('yaml ok')"` +Expected: `yaml ok`. + +- [ ] **Step 5: Commit** + +```bash +git add .github/workflows/ci.yml scripts/ci-workflow.spec.mjs +git commit -m "ci: gate merges on the PR-side preview verifications + +Co-Authored-By: Claude Fable 5.1 " +``` + +--- + +### Task 6: Provision the examples bypass secret + +**Files:** none in the repo. External state: Vercel project `threadplane-examples`, GitHub repository secret. Run Steps 1–4 in one shell session so `$tok` and `$out` carry over. + +The token lives in the primary checkout's root `.env` as `VERCEL_API_TOKEN`. Team `cacheplane` is `team_RWMT2bzjj1nkSXI3N3arQ6CP`; the examples project is `prj_ZDFXcUa8iK3DI4i3S4dMbfFvxo78`. + +- [ ] **Step 1: Confirm the project has no bypass yet** + +```bash +tok=$(grep -E "^VERCEL_API_TOKEN=" /Users/blove/repos/angular-agent-framework/.env | cut -d= -f2- | tr -d '"'"'"' ') +curl -s -H "Authorization: Bearer $tok" "https://api.vercel.com/v9/projects/prj_ZDFXcUa8iK3DI4i3S4dMbfFvxo78?teamId=team_RWMT2bzjj1nkSXI3N3arQ6CP" | python3 -c 'import sys,json;p=json.load(sys.stdin);print("bypass entries:", len(p.get("protectionBypass") or {}))' +``` + +Expected: `bypass entries: 0`. If it prints 1 or more, skip Step 2 and read the existing key in Step 3 instead of the generated one. + +- [ ] **Step 2: Generate the bypass (value never printed)** + +```bash +umask 077 +out=/private/tmp/claude-501/examples-bypass.json +curl -s -X PATCH -H "Authorization: Bearer $tok" -H "Content-Type: application/json" "https://api.vercel.com/v1/projects/prj_ZDFXcUa8iK3DI4i3S4dMbfFvxo78/protection-bypass?teamId=team_RWMT2bzjj1nkSXI3N3arQ6CP" -d '{"generate":{}}' > "$out" +python3 -c 'import sys,json;d=json.load(open(sys.argv[1]));print("entries now:", [(k[:4]+"…", v.get("scope")) for k,v in (d.get("protectionBypass") or {}).items()])' "$out" +``` + +Expected: `entries now: [('xxxx…', 'automation-bypass')]`. + +- [ ] **Step 3: Store it as the repository secret** + +```bash +python3 -c 'import sys,json;d=json.load(open(sys.argv[1]));print(next(k for k,v in d["protectionBypass"].items() if v.get("scope")=="automation-bypass"),end="")' "$out" | gh secret set VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET --repo cacheplane/angular-agent-framework +gh secret list | grep VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET +rm -f "$out" +``` + +Expected: a line starting `VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET` with today's timestamp, then the scratch file removed. + +- [ ] **Step 4: Prove the bypass opens an examples deployment** + +```bash +# GitHub secrets cannot be read back; re-read the key from Vercel for the probe. +key=$(curl -s -H "Authorization: Bearer $tok" "https://api.vercel.com/v9/projects/prj_ZDFXcUa8iK3DI4i3S4dMbfFvxo78?teamId=team_RWMT2bzjj1nkSXI3N3arQ6CP" | python3 -c 'import sys,json;p=json.load(sys.stdin);print(next(k for k,v in p["protectionBypass"].items() if v.get("scope")=="automation-bypass"),end="")') +u=$(curl -s -H "Authorization: Bearer $tok" "https://api.vercel.com/v6/deployments?projectId=prj_ZDFXcUa8iK3DI4i3S4dMbfFvxo78&teamId=team_RWMT2bzjj1nkSXI3N3arQ6CP&limit=1" | python3 -c 'import sys,json;print(json.load(sys.stdin)["deployments"][0]["url"])') +curl -s -o /dev/null -w "without: %{http_code}\n" "https://$u/langgraph/streaming/" +curl -s -o /dev/null -H "x-vercel-protection-bypass: $key" -w "with bypass: %{http_code}\n" "https://$u/langgraph/streaming/" +``` + +Expected: `without: 302` then `with bypass: 200`. + +--- + +### Task 7: Open the PR and verify both lanes live + +**Files:** none new. This task exercises the lanes end to end. + +- [ ] **Step 1: Push and open the PR** + +```bash +git push -u origin blove/pr-side-deploy-verification +gh pr create --base main --title "ci: PR-side deploy verification against real Vercel previews" --body-file - <<'MD' +## Why + +Landing #963 took nine fix PRs, every one repairing a step only the push-only deploy job executes. See `docs/superpowers/specs/2026-09-03-pr-side-deploy-verification-design.md`. + +## What + +- `website-preview-e2e`: assembles the examples with a deterministic Website alias in their policy, deploys them as an examples preview under a deterministic alias, builds and deploys the Website as a preview pointed at that alias, and runs the ordinary suite against it. A Playwright global setup seeds the examples origin's bypass cookie so the runtime iframe reaches Ready. +- `cockpit-preview-smoke`: deploys a throwaway cockpit preview and runs the 399-probe smoke against it. +- Both join the required gate under their existing scope keys. Same-repo PRs and merge-queue candidates only; the token-free dry-run stays for forks. + +## Provisioning + +`VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET` was generated on the threadplane-examples project and stored before this PR was opened, so this PR's own run exercises both lanes. + +🤖 Generated with [Claude Code](https://claude.com/claude-code) +MD +``` + +- [ ] **Step 2: Watch the two new lanes on this PR** + +The PR touches `.github/workflows/ci.yml`, which the scope classifier treats as a global CI change, so every scope is on and both lanes run. + +```bash +gh pr checks --watch --fail-fast 2>&1 | grep -E "deployed preview|immutable preview smoke|CI — required" +``` + +Expected: `Website — e2e (deployed preview)` pass, `Cockpit — immutable preview smoke` pass, `CI — required` pass. + +- [ ] **Step 3: Confirm the runtime handshake actually ran** + +Open the Website lane's log and confirm the serial workspace-shell tests ran rather than skipping: + +```bash +run=$(gh pr checks --json name,link --jq '.[]|select(.name=="Website — e2e (deployed preview)")|.link' | grep -oE 'runs/[0-9]+' | cut -d/ -f2) +gh run view "$run" --log 2>/dev/null | sed 's/\x1b\[[0-9;]*m//g' | grep -E "workspace-shell.spec.ts:146|passed|failed|did not run" | tail -5 +``` + +Expected: the line for `workspace-shell.spec.ts:146 › moves Docs to Run to Code to API to Docs without replacing the runtime frame` shows `✓`, a `passed` total, and no `did not run`. + +- [ ] **Step 4: Confirm the aliases resolve** + +```bash +n=$(gh pr view --json number --jq .number) +curl -s -o /dev/null -w "website alias: %{http_code}\n" "https://threadplane-pr-${n}-cacheplane.vercel.app/" +curl -s -o /dev/null -w "examples alias: %{http_code}\n" "https://threadplane-examples-pr-${n}-cacheplane.vercel.app/langgraph/streaming/" +``` + +Expected: both `302` (protected, which proves the aliases exist and point at deployments). + +- [ ] **Step 5: Mutation check — reintroduce the #983 bug on a throwaway commit** + +```bash +git checkout -b blove/mutation-localhost-abort +python3 - <<'PY' +from pathlib import Path +p = Path('apps/website/e2e/workspace-shell.spec.ts'); s = p.read_text() +old = """ await page.route( + (url) => url.searchParams.has('cockpit_cap'), + (route) => route.abort() + ); +""" +assert old in s +p.write_text(s.replace(old, " await page.route('http://localhost:4300/**', (request) => request.abort());\n", 1)) +PY +git commit -am "test: MUTATION — do not merge" +git push -u origin blove/mutation-localhost-abort +gh pr create --base main --draft --title "MUTATION CHECK — do not merge" --body "Verifies website-preview-e2e goes red on a host-bound route abort. Close without merging." +gh pr checks --watch 2>&1 | grep -E "deployed preview" +``` + +Expected: `Website — e2e (deployed preview)` **fail**, because the reduced-motion loader is gone on a real preview. + +Then close the mutation PR and delete its branch: + +```bash +gh pr close --delete-branch +git checkout blove/pr-side-deploy-verification +``` + +- [ ] **Step 6: Merge** + +Once the real PR is green and any AI review comments are addressed: + +```bash +gh pr merge --squash +``` + +--- + +### Task 8: Record the lanes for the next engineer + +**Files:** +- Modify: `CONTRIBUTING.md` (add a short subsection near the existing CI notes) + +- [ ] **Step 1: Add the subsection** + +Find the CI section of `CONTRIBUTING.md` (search for `CI — required`) and add after it: + +```markdown +### PR-side deploy verification + +Two lanes run the deploy job's verification on pull requests against real +Vercel previews, so deploy-only failures surface before merge: + +- **Website — e2e (deployed preview)** builds and deploys the Website and + the examples as previews under deterministic aliases + (`threadplane-pr--cacheplane.vercel.app` and + `threadplane-examples-pr--cacheplane.vercel.app`) and runs the ordinary + suite against the Website alias. The runtime iframe loads because the + examples are assembled with the Website alias in their parent-origin + policy and Playwright seeds the examples origin's bypass cookie + (`apps/website/e2e/runtime-bypass-setup.ts`). +- **Cockpit — immutable preview smoke** deploys a throwaway cockpit preview + and runs the exhaustive redirect smoke against it. + +Both need repository secrets and therefore skip on fork PRs. Each Vercel +project has its own Protection Bypass for Automation secret: +`VERCEL_AUTOMATION_BYPASS_SECRET` (Website), +`VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET`, and +`VERCEL_COCKPIT_AUTOMATION_BYPASS_SECRET`. A secret added while a run is in +flight does not reach that run; re-run after provisioning. +``` + +- [ ] **Step 2: Commit** + +```bash +git add CONTRIBUTING.md +git commit -m "docs(contributing): describe the PR-side deploy verification lanes + +Co-Authored-By: Claude Fable 5.1 " +git push +``` + +This can land in the same PR as Tasks 1–5 if done before Task 7, or as a one-line follow-up. From 4c81e2b2e10ea7bdc7c7dac8b8d92603b3e2d44e Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 16:37:02 -0700 Subject: [PATCH 03/17] feat(website): playwright global setup that seeds the runtime origin bypass cookie Co-Authored-By: Claude Fable 5.1 --- apps/website/e2e/runtime-bypass-setup.ts | 88 +++++++++++++++++++ apps/website/src/runtime-bypass-setup.spec.ts | 65 ++++++++++++++ 2 files changed, 153 insertions(+) create mode 100644 apps/website/e2e/runtime-bypass-setup.ts create mode 100644 apps/website/src/runtime-bypass-setup.spec.ts diff --git a/apps/website/e2e/runtime-bypass-setup.ts b/apps/website/e2e/runtime-bypass-setup.ts new file mode 100644 index 000000000..7ad4fded8 --- /dev/null +++ b/apps/website/e2e/runtime-bypass-setup.ts @@ -0,0 +1,88 @@ +import { request } from '@playwright/test'; +import { mkdirSync, writeFileSync } from 'node:fs'; +import { dirname, resolve } from 'node:path'; + +/** + * Playwright globalSetup for runs whose Website preview embeds a runtime + * from a second protected Vercel project. Deployment protection answers + * every path on that origin with 302 -> vercel.com/sso-api, and Playwright's + * extraHTTPHeaders is global, so the Website project's secret would reach + * the runtime origin and be rejected. Vercel issues a per-origin `_vercel_jwt` + * bypass cookie when a request carries the owning project's secret together + * with `x-vercel-set-bypass-cookie=true`; this setup obtains that cookie once + * and stores it as storage state, so every browser context carries it and + * the runtime iframe and its subresources load. The examples secret travels + * only in this one request. + */ +export const RUNTIME_BYPASS_STORAGE_STATE = resolve( + __dirname, + '..', + 'test-results', + 'runtime-bypass-storage-state.json' +); + +type SetupEnvironment = Readonly>; + +export function buildRuntimeBypassUrl(origin: string, secret: string): string { + let parsed: URL; + try { + parsed = new URL(origin); + } catch { + throw new Error( + `RUNTIME_BYPASS_ORIGIN must be a bare https origin, received ${origin}` + ); + } + if ( + parsed.protocol !== 'https:' || + parsed.username || + parsed.password || + parsed.pathname !== '/' || + parsed.search || + parsed.hash + ) { + throw new Error( + `RUNTIME_BYPASS_ORIGIN must be a bare https origin, received ${origin}` + ); + } + const url = new URL('/', parsed.origin); + url.searchParams.set('x-vercel-protection-bypass', secret); + url.searchParams.set('x-vercel-set-bypass-cookie', 'true'); + return url.toString(); +} + +export async function seedRuntimeBypass( + environment: SetupEnvironment +): Promise<'skipped' | 'seeded'> { + const origin = environment['RUNTIME_BYPASS_ORIGIN']; + const secret = environment['VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET']; + if (!origin || !secret) return 'skipped'; + + const context = await request.newContext(); + try { + const response = await context.get(buildRuntimeBypassUrl(origin, secret), { + maxRedirects: 0, + }); + const state = await context.storageState(); + const seeded = state.cookies.some( + (cookie) => cookie.name === '_vercel_jwt' + ); + if (!seeded) { + throw new Error( + `Runtime bypass setup: ${origin} answered ${response.status()} without a _vercel_jwt cookie. Check VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET against the Vercel project that owns that origin.` + ); + } + mkdirSync(dirname(RUNTIME_BYPASS_STORAGE_STATE), { recursive: true }); + writeFileSync(RUNTIME_BYPASS_STORAGE_STATE, JSON.stringify(state)); + return 'seeded'; + } finally { + await context.dispose(); + } +} + +// Playwright calls globalSetup with its FullConfig as the only argument. +// Read the environment from the process, never from that argument. +export default async function runtimeBypassSetup(): Promise< + 'skipped' | 'seeded' +> { + return seedRuntimeBypass(process.env); +} diff --git a/apps/website/src/runtime-bypass-setup.spec.ts b/apps/website/src/runtime-bypass-setup.spec.ts new file mode 100644 index 000000000..987454b05 --- /dev/null +++ b/apps/website/src/runtime-bypass-setup.spec.ts @@ -0,0 +1,65 @@ +import { describe, expect, it } from 'vitest'; +import runtimeBypassSetup, { + RUNTIME_BYPASS_STORAGE_STATE, + buildRuntimeBypassUrl, + seedRuntimeBypass, +} from '../e2e/runtime-bypass-setup'; + +describe('runtime bypass setup', () => { + it('asks Vercel for the bypass cookie at the runtime origin root', () => { + expect( + buildRuntimeBypassUrl( + 'https://threadplane-examples-pr-7-cacheplane.vercel.app', + 'examples-secret' + ) + ).toBe( + 'https://threadplane-examples-pr-7-cacheplane.vercel.app/?x-vercel-protection-bypass=examples-secret&x-vercel-set-bypass-cookie=true' + ); + }); + + it('rejects a runtime origin that is not a bare https origin', () => { + for (const origin of [ + 'http://threadplane-examples-pr-7-cacheplane.vercel.app', + 'https://threadplane-examples-pr-7-cacheplane.vercel.app/langgraph', + 'https://user:pw@threadplane-examples-pr-7-cacheplane.vercel.app', + ]) { + expect(() => buildRuntimeBypassUrl(origin, 'examples-secret')).toThrow( + /bare https origin/ + ); + } + }); + + it('does nothing when the origin or the secret is unset', async () => { + await expect(seedRuntimeBypass({})).resolves.toBe('skipped'); + await expect( + seedRuntimeBypass({ RUNTIME_BYPASS_ORIGIN: 'https://x.vercel.app' }) + ).resolves.toBe('skipped'); + await expect( + seedRuntimeBypass({ VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET: 's' }) + ).resolves.toBe('skipped'); + }); + + it('ignores the config object Playwright hands to a global setup', async () => { + // Playwright calls the default export with its FullConfig. Reading the + // environment from that argument would silently skip the seeding. + const saved = { ...process.env }; + delete process.env['RUNTIME_BYPASS_ORIGIN']; + delete process.env['VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET']; + try { + await expect( + runtimeBypassSetup({ + RUNTIME_BYPASS_ORIGIN: 'https://x.vercel.app', + VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET: 's', + } as never) + ).resolves.toBe('skipped'); + } finally { + process.env = saved; + } + }); + + it('writes storage state under the gitignored test-results directory', () => { + expect(RUNTIME_BYPASS_STORAGE_STATE).toMatch( + /apps\/website\/test-results\/runtime-bypass-storage-state\.json$/ + ); + }); +}); From 673257079bb7c556125720a673cd1efda04c87dd Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 16:42:11 -0700 Subject: [PATCH 04/17] docs(plans): keep the runtime bypass storage state out of uploaded artifact dirs Co-Authored-By: Claude Fable 5.1 --- .../plans/2026-09-03-pr-side-deploy-verification.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/plans/2026-09-03-pr-side-deploy-verification.md b/docs/superpowers/plans/2026-09-03-pr-side-deploy-verification.md index 65878add1..0300128d4 100644 --- a/docs/superpowers/plans/2026-09-03-pr-side-deploy-verification.md +++ b/docs/superpowers/plans/2026-09-03-pr-side-deploy-verification.md @@ -16,7 +16,7 @@ - Workflow guards run with `node --test --test-reporter=tap scripts/ci-workflow.spec.mjs`. - Commit messages end with `Co-Authored-By: Claude Fable 5.1 `. - Never `echo` a secret. Pipe values into `gh secret set`. -- `apps/website/test-results/` is gitignored; generated storage state goes there. +- `dist/` is gitignored and never uploaded as a CI artifact; generated storage state goes to `dist/apps/website/e2e-runtime-bypass/storage-state.json`. (`apps/website/test-results/` is uploaded on failure by other lanes, so the bypass cookie must not live there.) --- @@ -105,7 +105,7 @@ describe('runtime bypass setup', () => { it('writes storage state under the gitignored test-results directory', () => { expect(RUNTIME_BYPASS_STORAGE_STATE).toMatch( - /apps\/website\/test-results\/runtime-bypass-storage-state\.json$/ + /dist\/apps\/website\/e2e-runtime-bypass\/storage-state\.json$/ ); }); }); @@ -253,7 +253,7 @@ Append inside the `describe('Website Playwright configuration', ...)` block in ` }); expect(both.globalSetup).toMatch(/runtime-bypass-setup\.ts$/); expect(both.use?.storageState).toMatch( - /apps\/website\/test-results\/runtime-bypass-storage-state\.json$/ + /dist\/apps\/website\/e2e-runtime-bypass\/storage-state\.json$/ ); // The examples secret must never ride the global header, which reaches // the Website origin on every request. From a6830a9ad0965890f6aa7348bfc4aa9a81f3ec95 Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 16:44:37 -0700 Subject: [PATCH 05/17] =?UTF-8?q?fix(website):=20harden=20the=20runtime=20?= =?UTF-8?q?bypass=20setup=20=E2=80=94=20redact,=20relocate=20state,=20Same?= =?UTF-8?q?Site=3DNone,=20loopback=20tests?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- apps/website/e2e/runtime-bypass-setup.ts | 61 ++++++--- apps/website/src/runtime-bypass-setup.spec.ts | 117 +++++++++++++++++- 2 files changed, 162 insertions(+), 16 deletions(-) diff --git a/apps/website/e2e/runtime-bypass-setup.ts b/apps/website/e2e/runtime-bypass-setup.ts index 7ad4fded8..042e8f74c 100644 --- a/apps/website/e2e/runtime-bypass-setup.ts +++ b/apps/website/e2e/runtime-bypass-setup.ts @@ -14,37 +14,50 @@ import { dirname, resolve } from 'node:path'; * the runtime iframe and its subresources load. The examples secret travels * only in this one request. */ +// Kept outside apps/website/test-results/ on purpose: other CI lanes upload +// that directory as a failure artifact, and this file holds a bearer cookie. export const RUNTIME_BYPASS_STORAGE_STATE = resolve( __dirname, '..', - 'test-results', - 'runtime-bypass-storage-state.json' + '..', + '..', + 'dist', + 'apps', + 'website', + 'e2e-runtime-bypass', + 'storage-state.json' ); type SetupEnvironment = Readonly>; +const LOOPBACK_HOSTNAMES = new Set(['localhost', '127.0.0.1', '[::1]']); + +function bareOriginError(origin: string): Error { + return new Error( + `RUNTIME_BYPASS_ORIGIN must be a bare https origin, received ${origin}` + ); +} + export function buildRuntimeBypassUrl(origin: string, secret: string): string { let parsed: URL; try { parsed = new URL(origin); } catch { - throw new Error( - `RUNTIME_BYPASS_ORIGIN must be a bare https origin, received ${origin}` - ); + throw bareOriginError(origin); } + const isLoopbackHttp = + parsed.protocol === 'http:' && LOOPBACK_HOSTNAMES.has(parsed.hostname); if ( - parsed.protocol !== 'https:' || + (parsed.protocol !== 'https:' && !isLoopbackHttp) || parsed.username || parsed.password || parsed.pathname !== '/' || parsed.search || parsed.hash ) { - throw new Error( - `RUNTIME_BYPASS_ORIGIN must be a bare https origin, received ${origin}` - ); + throw bareOriginError(origin); } - const url = new URL('/', parsed.origin); + const url = new URL(parsed.origin); url.searchParams.set('x-vercel-protection-bypass', secret); url.searchParams.set('x-vercel-set-bypass-cookie', 'true'); return url.toString(); @@ -59,9 +72,18 @@ export async function seedRuntimeBypass( const context = await request.newContext(); try { - const response = await context.get(buildRuntimeBypassUrl(origin, secret), { - maxRedirects: 0, - }); + let response; + try { + // maxRedirects: 0 returns the 3xx instead of following it, and + // Playwright stores Set-Cookie from that response before deciding + // about redirects, which is what captures the cookie. + response = await context.get(buildRuntimeBypassUrl(origin, secret), { + maxRedirects: 0, + }); + } catch (error) { + const message = (error as Error).message.split(secret).join('***'); + throw new Error(`Runtime bypass setup: ${message}`); + } const state = await context.storageState(); const seeded = state.cookies.some( (cookie) => cookie.name === '_vercel_jwt' @@ -71,8 +93,19 @@ export async function seedRuntimeBypass( `Runtime bypass setup: ${origin} answered ${response.status()} without a _vercel_jwt cookie. Check VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET against the Vercel project that owns that origin.` ); } + // The runtime iframe is a cross-site navigation from the Website preview, + // so the cookie must be SameSite=None (which requires Secure) or the + // browser will never attach it to that navigation. + const crossSiteState = { + ...state, + cookies: state.cookies.map((cookie) => ({ + ...cookie, + sameSite: 'None' as const, + secure: true, + })), + }; mkdirSync(dirname(RUNTIME_BYPASS_STORAGE_STATE), { recursive: true }); - writeFileSync(RUNTIME_BYPASS_STORAGE_STATE, JSON.stringify(state)); + writeFileSync(RUNTIME_BYPASS_STORAGE_STATE, JSON.stringify(crossSiteState)); return 'seeded'; } finally { await context.dispose(); diff --git a/apps/website/src/runtime-bypass-setup.spec.ts b/apps/website/src/runtime-bypass-setup.spec.ts index 987454b05..f683a700c 100644 --- a/apps/website/src/runtime-bypass-setup.spec.ts +++ b/apps/website/src/runtime-bypass-setup.spec.ts @@ -1,3 +1,6 @@ +import { createServer } from 'node:http'; +import type { AddressInfo } from 'node:net'; +import { existsSync, readFileSync, rmSync } from 'node:fs'; import { describe, expect, it } from 'vitest'; import runtimeBypassSetup, { RUNTIME_BYPASS_STORAGE_STATE, @@ -57,9 +60,119 @@ describe('runtime bypass setup', () => { } }); - it('writes storage state under the gitignored test-results directory', () => { + it('writes storage state under the gitignored dist directory', () => { expect(RUNTIME_BYPASS_STORAGE_STATE).toMatch( - /apps\/website\/test-results\/runtime-bypass-storage-state\.json$/ + /dist\/apps\/website\/e2e-runtime-bypass\/storage-state\.json$/ ); }); + + it('redacts the secret from a transport failure message', async () => { + // Bind to a loopback port then close it immediately so the follow-up + // request reliably fails fast (ECONNREFUSED) without a real listener. + const probe = createServer(); + await new Promise((res) => probe.listen(0, '127.0.0.1', res)); + const port = (probe.address() as AddressInfo).port; + await new Promise((res) => probe.close(() => res())); + + const secret = 'redact-me-please'; + await expect( + seedRuntimeBypass({ + RUNTIME_BYPASS_ORIGIN: `http://127.0.0.1:${port}`, + VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET: secret, + }) + ).rejects.toSatisfy((error: unknown) => { + const message = (error as Error).message; + return !message.includes(secret) && message.includes('***'); + }); + }); + + it('allows http only for loopback hosts, rejecting it elsewhere', () => { + for (const origin of [ + 'http://localhost:4000', + 'http://127.0.0.1:4000', + 'http://[::1]:4000', + ]) { + expect(() => buildRuntimeBypassUrl(origin, 'secret')).not.toThrow(); + } + expect(() => + buildRuntimeBypassUrl( + 'http://threadplane-examples-pr-7-cacheplane.vercel.app', + 'secret' + ) + ).toThrow(/bare https origin/); + }); + + it('seeds a normalized, cross-site cookie from a loopback bypass response', async () => { + rmSync(RUNTIME_BYPASS_STORAGE_STATE, { force: true }); + let requestPath: string | undefined; + const server = createServer((req, res) => { + requestPath = req.url; + res.statusCode = 302; + res.setHeader('Location', '/'); + res.setHeader( + 'Set-Cookie', + '_vercel_jwt=abc123; Path=/; HttpOnly; SameSite=Lax' + ); + res.end(); + }); + await new Promise((res) => server.listen(0, '127.0.0.1', res)); + const port = (server.address() as AddressInfo).port; + const secret = 'loopback-secret'; + + try { + await expect( + seedRuntimeBypass({ + RUNTIME_BYPASS_ORIGIN: `http://127.0.0.1:${port}`, + VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET: secret, + }) + ).resolves.toBe('seeded'); + + expect(requestPath).toBe( + `/?x-vercel-protection-bypass=${secret}&x-vercel-set-bypass-cookie=true` + ); + + expect(existsSync(RUNTIME_BYPASS_STORAGE_STATE)).toBe(true); + const state = JSON.parse( + readFileSync(RUNTIME_BYPASS_STORAGE_STATE, 'utf8') + ); + const cookie = state.cookies.find( + (c: { name: string }) => c.name === '_vercel_jwt' + ); + expect(cookie).toMatchObject({ + value: 'abc123', + sameSite: 'None', + secure: true, + }); + } finally { + await new Promise((res) => server.close(() => res())); + rmSync(RUNTIME_BYPASS_STORAGE_STATE, { force: true }); + } + }); + + it('rejects with the status code when the loopback response sets no cookie', async () => { + const server = createServer((_req, res) => { + res.statusCode = 302; + res.setHeader('Location', '/'); + res.end(); + }); + await new Promise((res) => server.listen(0, '127.0.0.1', res)); + const port = (server.address() as AddressInfo).port; + + try { + await expect( + seedRuntimeBypass({ + RUNTIME_BYPASS_ORIGIN: `http://127.0.0.1:${port}`, + VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET: 'no-cookie-secret', + }) + ).rejects.toThrow(/without a _vercel_jwt cookie/); + await expect( + seedRuntimeBypass({ + RUNTIME_BYPASS_ORIGIN: `http://127.0.0.1:${port}`, + VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET: 'no-cookie-secret', + }) + ).rejects.toThrow(/302/); + } finally { + await new Promise((res) => server.close(() => res())); + } + }); }); From cee0db824382eb972ee442aac525184fcb1886ba Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 16:48:54 -0700 Subject: [PATCH 06/17] feat(website): opt-in runtime bypass storage state for protected previews Co-Authored-By: Claude Fable 5.1 --- apps/website/playwright.config.ts | 15 +++++++++ apps/website/src/playwright-config.spec.ts | 38 ++++++++++++++++++++++ 2 files changed, 53 insertions(+) diff --git a/apps/website/playwright.config.ts b/apps/website/playwright.config.ts index da4036680..e0234b060 100644 --- a/apps/website/playwright.config.ts +++ b/apps/website/playwright.config.ts @@ -1,4 +1,6 @@ import { defineConfig, devices } from '@playwright/test'; +import { resolve } from 'node:path'; +import { RUNTIME_BYPASS_STORAGE_STATE } from './e2e/runtime-bypass-setup'; type WebsitePlaywrightEnvironment = Readonly< Record @@ -22,6 +24,15 @@ export const createWebsitePlaywrightConfig = ( const reuseExistingServer = environment['PLAYWRIGHT_REUSE_EXISTING_SERVER'] === 'true'; + // A PR preview embeds its runtime from a second protected Vercel project. + // extraHTTPHeaders is global, so that origin needs its own bypass, seeded + // once as a cookie by e2e/runtime-bypass-setup.ts. Both variables must be + // present; the deploy job and local runs set neither. + const runtimeBypass = Boolean( + environment['RUNTIME_BYPASS_ORIGIN'] && + environment['VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET'] + ); + // The public-copy gate crawls every sitemap route. Against a prebuilt // production server that is seconds; against `next dev` each route compiles // on demand, which is far too slow to belong in the ordinary suite. It runs @@ -59,6 +70,9 @@ export const createWebsitePlaywrightConfig = ( ? '**/custom-runtime-bfcache.spec.ts' : undefined, testIgnore, + globalSetup: runtimeBypass + ? resolve(__dirname, 'e2e', 'runtime-bypass-setup.ts') + : undefined, fullyParallel: true, // Match the cockpit configs: 2 retries on CI to absorb transient Next.js // dev-server startup flake; 0 locally for fast feedback. @@ -83,6 +97,7 @@ export const createWebsitePlaywrightConfig = ( // artifacts disabled so request headers and page state are never retained. trace: 'off', video: 'off', + ...(runtimeBypass ? { storageState: RUNTIME_BYPASS_STORAGE_STATE } : {}), }, // Declare chromium as the only browser project. This suppresses the // misleading "missing system dependencies" warning for webkit/firefox. diff --git a/apps/website/src/playwright-config.spec.ts b/apps/website/src/playwright-config.spec.ts index 8c4317335..e69154cfb 100644 --- a/apps/website/src/playwright-config.spec.ts +++ b/apps/website/src/playwright-config.spec.ts @@ -120,6 +120,44 @@ describe('Website Playwright configuration', () => { expect(shell).toContain("url.searchParams.has('cockpit_cap')"); }); + it('seeds the runtime origin bypass only when both the origin and the examples secret are set', () => { + const base = createWebsitePlaywrightConfig({ + BASE_URL: 'https://threadplane-pr-7-cacheplane.vercel.app', + VERCEL_AUTOMATION_BYPASS_SECRET: 'website-secret', + }); + expect(base.globalSetup).toBeUndefined(); + expect(base.use?.storageState).toBeUndefined(); + + const originOnly = createWebsitePlaywrightConfig({ + BASE_URL: 'https://threadplane-pr-7-cacheplane.vercel.app', + VERCEL_AUTOMATION_BYPASS_SECRET: 'website-secret', + RUNTIME_BYPASS_ORIGIN: + 'https://threadplane-examples-pr-7-cacheplane.vercel.app', + }); + expect(originOnly.globalSetup).toBeUndefined(); + + const both = createWebsitePlaywrightConfig({ + BASE_URL: 'https://threadplane-pr-7-cacheplane.vercel.app', + VERCEL_AUTOMATION_BYPASS_SECRET: 'website-secret', + VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET: 'examples-secret', + RUNTIME_BYPASS_ORIGIN: + 'https://threadplane-examples-pr-7-cacheplane.vercel.app', + }); + expect(both.globalSetup).toMatch(/runtime-bypass-setup\.ts$/); + expect(both.use?.storageState).toMatch( + /dist\/apps\/website\/e2e-runtime-bypass\/storage-state\.json$/ + ); + // The examples secret must never ride the global header, which reaches + // the Website origin on every request. + expect(JSON.stringify(both.use?.extraHTTPHeaders)).not.toContain( + 'examples-secret' + ); + expect(both.use?.extraHTTPHeaders).toEqual({ + 'x-vercel-protection-bypass': 'website-secret', + 'x-vercel-set-bypass-cookie': 'true', + }); + }); + it('starts Website, all migrated runtime apps under custom-runtime E2E, and the fixture', () => { const config = createWebsitePlaywrightConfig({}); From 1d4455c83d56e12596f254796a79b42b8c94d8b7 Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 16:57:17 -0700 Subject: [PATCH 07/17] ci: smoke a throwaway cockpit preview on pull requests Co-Authored-By: Claude Fable 5.1 --- .github/workflows/ci.yml | 50 ++++++++++++++++++++++++++++++++++++ scripts/ci-workflow.spec.mjs | 38 +++++++++++++++++++++++++++ 2 files changed, 88 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8c3820a84..af2df6bfa 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -354,6 +354,56 @@ jobs: - run: npm ci - run: npx tsx apps/cockpit/scripts/deploy-smoke.ts --url https://cockpit.threadplane.ai --mode preview --dry-run + cockpit-preview-smoke: + name: Cockpit — immutable preview smoke + timeout-minutes: 20 + needs: ci-scope + # PR-side twin of the deploy job's cockpit verification. Deploys a + # throwaway preview of the redirect service and runs the exhaustive smoke + # against it, so platform behaviour (deployment protection, the CDN's + # slash collapse, route ordering) is exercised before merge. Needs + # repository secrets, so same-repo PRs and merge-queue candidates only; + # the token-free dry-run job above still covers forks. + if: >- + github.event_name != 'push' && + needs.ci-scope.outputs.cockpit_deploy_smoke == 'true' && + (github.event_name == 'merge_group' || github.event.pull_request.head.repo.full_name == github.repository) + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + - uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 + with: + node-version: 22 + cache: npm + - run: npm ci + - name: Prepare cockpit Vercel project (preview) + run: | + mkdir -p .vercel + cat > .vercel/project.json <> "$GITHUB_OUTPUT" + - name: Exhaustively verify the cockpit preview + run: | + if [ -z "${VERCEL_AUTOMATION_BYPASS_SECRET}" ]; then + echo "::error::VERCEL_COCKPIT_AUTOMATION_BYPASS_SECRET is unset — the protected cockpit preview cannot be verified. Enable 'Protection Bypass for Automation' on the Vercel threadplane-cockpit project and store the value as this repository secret." + exit 1 + fi + npx tsx apps/cockpit/scripts/deploy-smoke.ts --url "${{ steps.deploy_cockpit_preview.outputs.deployment_url }}" --mode preview --retries 20 --retry-delay-ms 5000 + env: + VERCEL_AUTOMATION_BYPASS_SECRET: ${{ secrets.VERCEL_COCKPIT_AUTOMATION_BYPASS_SECRET }} + examples-chat-smoke: name: examples/chat — python smoke needs: ci-scope diff --git a/scripts/ci-workflow.spec.mjs b/scripts/ci-workflow.spec.mjs index c2c18c810..9a1bc10fb 100644 --- a/scripts/ci-workflow.spec.mjs +++ b/scripts/ci-workflow.spec.mjs @@ -591,6 +591,44 @@ describe('CI workflow', () => { ); }); + it('smokes a throwaway cockpit preview on same-repo PRs and queue candidates', async () => { + const workflow = await readWorkflow(); + const job = readJobBlock(workflow, 'cockpit-preview-smoke'); + const ifBlock = readJobFieldBlock(job, 'if'); + + assert.deepEqual(readJobNeeds(job), ['ci-scope']); + assert.match(ifBlock, /github\.event_name != 'push'/); + assert.match(ifBlock, /needs\.ci-scope\.outputs\.cockpit_deploy_smoke == 'true'/); + assert.match( + ifBlock, + /github\.event_name == 'merge_group' \|\| github\.event\.pull_request\.head\.repo\.full_name == github\.repository/ + ); + + const prepare = readNamedStep(job, 'Prepare cockpit Vercel project (preview)'); + const build = readNamedStep(job, 'Build cockpit redirect service (preview)'); + const deploy = readNamedStep(job, 'Deploy throwaway cockpit preview'); + const smoke = readNamedStep(job, 'Exhaustively verify the cockpit preview'); + + assert.match(prepare, /"projectName":"threadplane-cockpit"/); + assert.match(prepare, /vercel pull --yes --environment=preview/); + assert.match(build, /vercel build --local-config vercel\.cockpit\.json/); + assert.doesNotMatch(build, /--prod/); + assert.match(deploy, /id:\s*deploy_cockpit_preview/); + assert.match(deploy, /vercel deploy --prebuilt --archive=tgz --skip-domain --yes/); + assert.doesNotMatch(deploy, /--prod/); + assert.match(deploy, /--env COCKPIT_WEBSITE_ORIGIN=https:\/\/threadplane\.ai/); + assert.match( + smoke, + /VERCEL_AUTOMATION_BYPASS_SECRET:\s*\$\{\{ secrets\.VERCEL_COCKPIT_AUTOMATION_BYPASS_SECRET \}\}/ + ); + assert.match(smoke, /-z "\$\{VERCEL_AUTOMATION_BYPASS_SECRET\}"/); + assert.match( + smoke, + /--url "\$\{\{ steps\.deploy_cockpit_preview\.outputs\.deployment_url \}\}"[\s\S]*--mode preview/ + ); + assert.doesNotMatch(job, /vercel promote/); + }); + it('gates Cockpit deployment on the production Website smoke even for Cockpit-only changes', async () => { const deployJob = await readDeployJob(); const websiteOrCockpit = From 8a209a78434eca85f591f0dacf601e5acf761b03 Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 16:57:44 -0700 Subject: [PATCH 08/17] docs(specs): record the mixed-credential probe result for the runtime origin Co-Authored-By: Claude Fable 5.1 --- .../specs/2026-09-03-pr-side-deploy-verification-design.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/superpowers/specs/2026-09-03-pr-side-deploy-verification-design.md b/docs/superpowers/specs/2026-09-03-pr-side-deploy-verification-design.md index b26b2b0de..85ba6546c 100644 --- a/docs/superpowers/specs/2026-09-03-pr-side-deploy-verification-design.md +++ b/docs/superpowers/specs/2026-09-03-pr-side-deploy-verification-design.md @@ -56,6 +56,12 @@ embedded runtime handshake is exercised too. - Playwright's `extraHTTPHeaders` is global to the browser context. A page that embeds an iframe from a second protected project needs a second secret sent only to that origin. +- Probed 2026-09-03 against a protected cockpit deployment: a request that + carries a valid `_vercel_jwt` cookie for that origin **and** an + `x-vercel-protection-bypass` header holding a different project's secret is + served (308 from the app); the wrong header alone is refused (302 to SSO). + So the Website secret riding the global header does not break the cookie + seeded for the examples origin. - `resolveRuntimeUrl` in `libs/cockpit-shell` reads `NEXT_PUBLIC_COCKPIT_RUNTIME_BASE_URL` at build time, defaulting to `https://examples.threadplane.ai`. From 3f5a979b407225778e39fef3e6109b7320e2617c Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 17:03:07 -0700 Subject: [PATCH 09/17] ci(cockpit-preview-smoke): fail on an empty deploy URL and remove the throwaway preview Co-Authored-By: Claude Fable 5.1 --- .github/workflows/ci.yml | 15 ++++++++++++++- scripts/ci-workflow.spec.mjs | 10 ++++++++++ 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index af2df6bfa..3ef415b38 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -379,7 +379,7 @@ jobs: - name: Prepare cockpit Vercel project (preview) run: | mkdir -p .vercel - cat > .vercel/project.json < .vercel/project.json <<'EOF' {"projectId":"${{ secrets.VERCEL_COCKPIT_PROJECT_ID }}","orgId":"${{ secrets.VERCEL_ORG_ID }}","projectName":"threadplane-cockpit"} EOF npx vercel pull --yes --environment=preview --token=${{ secrets.VERCEL_TOKEN }} @@ -392,7 +392,12 @@ jobs: - name: Deploy throwaway cockpit preview id: deploy_cockpit_preview run: | + set -euo pipefail url=$(npx vercel deploy --prebuilt --archive=tgz --skip-domain --yes --env COCKPIT_WEBSITE_ORIGIN=https://threadplane.ai --token=${{ secrets.VERCEL_TOKEN }} | tail -n 1) + if [ -z "$url" ]; then + echo "::error::vercel deploy produced no deployment URL for the throwaway cockpit preview." + exit 1 + fi echo "deployment_url=$url" >> "$GITHUB_OUTPUT" - name: Exhaustively verify the cockpit preview run: | @@ -403,6 +408,14 @@ jobs: npx tsx apps/cockpit/scripts/deploy-smoke.ts --url "${{ steps.deploy_cockpit_preview.outputs.deployment_url }}" --mode preview --retries 20 --retry-delay-ms 5000 env: VERCEL_AUTOMATION_BYPASS_SECRET: ${{ secrets.VERCEL_COCKPIT_AUTOMATION_BYPASS_SECRET }} + - name: Remove the throwaway cockpit preview + # The artifact exists only to be smoked. `--safe` refuses to remove a + # deployment that carries an alias, so this can never touch the + # production artifact. `remove` takes a bare URL, so, like `promote`, + # it needs the team scope spelled out. + if: always() && steps.deploy_cockpit_preview.outputs.deployment_url != '' + continue-on-error: true + run: npx vercel remove "${{ steps.deploy_cockpit_preview.outputs.deployment_url }}" --safe --yes --scope=${{ secrets.VERCEL_ORG_ID }} --token=${{ secrets.VERCEL_TOKEN }} examples-chat-smoke: name: examples/chat — python smoke diff --git a/scripts/ci-workflow.spec.mjs b/scripts/ci-workflow.spec.mjs index 9a1bc10fb..8d125d4f1 100644 --- a/scripts/ci-workflow.spec.mjs +++ b/scripts/ci-workflow.spec.mjs @@ -627,6 +627,16 @@ describe('CI workflow', () => { /--url "\$\{\{ steps\.deploy_cockpit_preview\.outputs\.deployment_url \}\}"[\s\S]*--mode preview/ ); assert.doesNotMatch(job, /vercel promote/); + assert.match(deploy, /set -euo pipefail/); + assert.match(deploy, /if \[ -z "\$url" \]/); + assert.match(smoke, /--retries 20 --retry-delay-ms 5000/); + const cleanup = readNamedStep(job, 'Remove the throwaway cockpit preview'); + assert.match(cleanup, /if:\s*always\(\) && steps\.deploy_cockpit_preview\.outputs\.deployment_url != ''/); + assert.match(cleanup, /continue-on-error:\s*true/); + assert.match( + cleanup, + /vercel remove "\$\{\{ steps\.deploy_cockpit_preview\.outputs\.deployment_url \}\}" --safe --yes --scope=\$\{\{ secrets\.VERCEL_ORG_ID \}\}/ + ); }); it('gates Cockpit deployment on the production Website smoke even for Cockpit-only changes', async () => { From d374030eeabee18a2515e30e442998b8d0eec3b7 Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 17:07:15 -0700 Subject: [PATCH 10/17] ci: run the Website suite against an aliased preview with a matching examples preview Co-Authored-By: Claude Fable 5.1 --- .github/workflows/ci.yml | 107 +++++++++++++++++++++++++++++++++++ scripts/ci-workflow.spec.mjs | 75 ++++++++++++++++++++++++ 2 files changed, 182 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3ef415b38..ac74d3be3 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -744,6 +744,113 @@ jobs: npx playwright test apps/website/e2e/platform-production-smoke.spec.ts \ --config apps/website/playwright.config.ts --list + website-preview-e2e: + name: Website — e2e (deployed preview) + timeout-minutes: 40 + needs: ci-scope + # PR-side twin of the deploy job's post-promotion verification. The + # ordinary suite runs against a real, protected Vercel preview of the + # Website whose runtime iframe loads from a matching examples preview, so + # remote-target assumptions and platform behaviour surface before merge. + # Two deterministic aliases break the ordering problem: the examples are + # assembled with the Website alias in their parent-origin policy, and the + # Website is built with the examples alias as its runtime base. A later + # push re-points both aliases; the deployments behind them are kept for + # inspection. Needs repository secrets, so same-repo PRs and merge-queue + # candidates only. + if: >- + github.event_name != 'push' && + needs.ci-scope.outputs.website_e2e == 'true' && + (github.event_name == 'merge_group' || github.event.pull_request.head.repo.full_name == github.repository) + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + - uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 + with: + node-version: 22 + cache: npm + - run: npm ci + - name: Cache Playwright browsers + uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + with: + path: ~/.cache/ms-playwright + key: playwright-${{ runner.os }}-${{ hashFiles('package-lock.json') }} + restore-keys: | + playwright-${{ runner.os }}- + - run: npx playwright install --with-deps chromium + - name: Derive deterministic preview aliases + id: aliases + run: | + if [ "${{ github.event_name }}" = "merge_group" ]; then + key="mq-$(echo "${{ github.event.merge_group.head_sha }}" | cut -c1-8)" + else + key="pr-${{ github.event.pull_request.number }}" + fi + echo "website=threadplane-${key}-cacheplane.vercel.app" >> "$GITHUB_OUTPUT" + echo "examples=threadplane-examples-${key}-cacheplane.vercel.app" >> "$GITHUB_OUTPUT" + - name: Require preview bypass secrets + # Both previews sit behind deployment protection, and bypass secrets + # are issued per Vercel project. Fail before creating anything. + run: | + if [ -z "${VERCEL_AUTOMATION_BYPASS_SECRET}" ]; then + echo "::error::VERCEL_AUTOMATION_BYPASS_SECRET is unset — the protected Website preview cannot be verified. Enable 'Protection Bypass for Automation' on the Vercel threadplane project and store the value as this repository secret." + exit 1 + fi + if [ -z "${VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET}" ]; then + echo "::error::VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET is unset — the runtime iframe from the protected examples preview cannot load. Enable 'Protection Bypass for Automation' on the Vercel threadplane-examples project and store the value as this repository secret." + exit 1 + fi + env: + VERCEL_AUTOMATION_BYPASS_SECRET: ${{ secrets.VERCEL_AUTOMATION_BYPASS_SECRET }} + VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET: ${{ secrets.VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET }} + - name: Build and assemble Angular examples for the preview + run: npx tsx scripts/assemble-examples.ts + env: + RUNTIME_PARENT_PREVIEW_ORIGINS: https://${{ steps.aliases.outputs.website }} + - name: Deploy examples preview and alias it + working-directory: deploy/examples + run: | + set -euo pipefail + mkdir -p .vercel + cat > .vercel/project.json <<'EOF' + {"projectId":"${{ secrets.VERCEL_EXAMPLES_PROJECT_ID }}","orgId":"${{ secrets.VERCEL_ORG_ID }}","projectName":"threadplane-examples"} + EOF + npx vercel pull --yes --environment=preview --token=${{ secrets.VERCEL_TOKEN }} + url=$(npx vercel deploy --prebuilt --yes --token=${{ secrets.VERCEL_TOKEN }} | tail -n 1) + if [ -z "$url" ]; then + echo "::error::vercel deploy produced no deployment URL for the examples preview." + exit 1 + fi + echo "examples deployment: $url" + npx vercel alias set "$url" "${{ steps.aliases.outputs.examples }}" --scope=${{ secrets.VERCEL_ORG_ID }} --token=${{ secrets.VERCEL_TOKEN }} + - name: Build, deploy, and alias the Website preview + env: + GROWTH_FORM_POLICY: growth_v1 + NEXT_PUBLIC_COCKPIT_RUNTIME_BASE_URL: https://${{ steps.aliases.outputs.examples }} + run: | + set -euo pipefail + mkdir -p .vercel + cat > .vercel/project.json <<'EOF' + {"projectId":"${{ secrets.VERCEL_WEBSITE_PROJECT_ID }}","orgId":"${{ secrets.VERCEL_ORG_ID }}","projectName":"threadplane"} + EOF + npx vercel pull --yes --environment=preview --token=${{ secrets.VERCEL_TOKEN }} + rm -rf .vercel/output + npx vercel build --token=${{ secrets.VERCEL_TOKEN }} + url=$(npx vercel deploy --prebuilt --archive=tgz --skip-domain --yes --token=${{ secrets.VERCEL_TOKEN }} | tail -n 1) + if [ -z "$url" ]; then + echo "::error::vercel deploy produced no deployment URL for the Website preview." + exit 1 + fi + echo "website deployment: $url" + npx vercel alias set "$url" "${{ steps.aliases.outputs.website }}" --scope=${{ secrets.VERCEL_ORG_ID }} --token=${{ secrets.VERCEL_TOKEN }} + - name: Run the Website suite against the aliased preview + run: npx nx e2e website --skip-nx-cache + env: + BASE_URL: https://${{ steps.aliases.outputs.website }} + RUNTIME_BYPASS_ORIGIN: https://${{ steps.aliases.outputs.examples }} + VERCEL_AUTOMATION_BYPASS_SECRET: ${{ secrets.VERCEL_AUTOMATION_BYPASS_SECRET }} + VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET: ${{ secrets.VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET }} + required-pr-checks: name: CI — required needs: diff --git a/scripts/ci-workflow.spec.mjs b/scripts/ci-workflow.spec.mjs index 8d125d4f1..d86c2c081 100644 --- a/scripts/ci-workflow.spec.mjs +++ b/scripts/ci-workflow.spec.mjs @@ -639,6 +639,81 @@ describe('CI workflow', () => { ); }); + it('runs the Website suite against a deterministic aliased preview with a matching examples preview', async () => { + const workflow = await readWorkflow(); + const job = readJobBlock(workflow, 'website-preview-e2e'); + const ifBlock = readJobFieldBlock(job, 'if'); + + assert.deepEqual(readJobNeeds(job), ['ci-scope']); + assert.match(ifBlock, /github\.event_name != 'push'/); + assert.match(ifBlock, /needs\.ci-scope\.outputs\.website_e2e == 'true'/); + assert.match( + ifBlock, + /github\.event_name == 'merge_group' \|\| github\.event\.pull_request\.head\.repo\.full_name == github\.repository/ + ); + + const aliases = readNamedStep(job, 'Derive deterministic preview aliases'); + const guard = readNamedStep(job, 'Require preview bypass secrets'); + const assemble = readNamedStep(job, 'Build and assemble Angular examples for the preview'); + const examples = readNamedStep(job, 'Deploy examples preview and alias it'); + const website = readNamedStep(job, 'Build, deploy, and alias the Website preview'); + const suite = readNamedStep(job, 'Run the Website suite against the aliased preview'); + + assert.match(aliases, /id:\s*aliases/); + assert.match(aliases, /key="pr-\$\{\{ github\.event\.pull_request\.number \}\}"/); + assert.match(aliases, /key="mq-\$\(echo "\$\{\{ github\.event\.merge_group\.head_sha \}\}" \| cut -c1-8\)"/); + assert.match(aliases, /website=threadplane-\$\{key\}-cacheplane\.vercel\.app/); + assert.match(aliases, /examples=threadplane-examples-\$\{key\}-cacheplane\.vercel\.app/); + + assert.match(guard, /-z "\$\{VERCEL_AUTOMATION_BYPASS_SECRET\}"/); + assert.match(guard, /-z "\$\{VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET\}"/); + assert.match(guard, /VERCEL_AUTOMATION_BYPASS_SECRET:\s*\$\{\{ secrets\.VERCEL_AUTOMATION_BYPASS_SECRET \}\}/); + assert.match(guard, /VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET:\s*\$\{\{ secrets\.VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET \}\}/); + + // The examples are assembled with the Website alias in their policy and + // the Website is built with the examples alias as its runtime base. + assert.match( + assemble, + /RUNTIME_PARENT_PREVIEW_ORIGINS:\s*https:\/\/\$\{\{ steps\.aliases\.outputs\.website \}\}/ + ); + assert.match(examples, /working-directory:\s*deploy\/examples/); + assert.match(examples, /"projectName":"threadplane-examples"/); + assert.match(examples, /vercel pull --yes --environment=preview/); + assert.match(examples, /vercel deploy --prebuilt --yes/); + assert.doesNotMatch(examples, /--prod/); + assert.match(examples, /set -euo pipefail/); + assert.match(examples, /if \[ -z "\$url" \]/); + assert.match( + examples, + /vercel alias set "\$url" "\$\{\{ steps\.aliases\.outputs\.examples \}\}" --scope=\$\{\{ secrets\.VERCEL_ORG_ID \}\}/ + ); + + assert.match(website, /"projectName":"threadplane"/); + assert.match(website, /vercel pull --yes --environment=preview/); + assert.match( + website, + /NEXT_PUBLIC_COCKPIT_RUNTIME_BASE_URL:\s*https:\/\/\$\{\{ steps\.aliases\.outputs\.examples \}\}/ + ); + assert.match(website, /GROWTH_FORM_POLICY:\s*growth_v1/); + assert.match(website, /vercel build --token/); + assert.match(website, /vercel deploy --prebuilt --archive=tgz --skip-domain --yes/); + assert.doesNotMatch(website, /--prod/); + assert.match(website, /set -euo pipefail/); + assert.match(website, /if \[ -z "\$url" \]/); + assert.match( + website, + /vercel alias set "\$url" "\$\{\{ steps\.aliases\.outputs\.website \}\}" --scope=\$\{\{ secrets\.VERCEL_ORG_ID \}\}/ + ); + + assert.match(suite, /BASE_URL:\s*https:\/\/\$\{\{ steps\.aliases\.outputs\.website \}\}/); + assert.match(suite, /RUNTIME_BYPASS_ORIGIN:\s*https:\/\/\$\{\{ steps\.aliases\.outputs\.examples \}\}/); + assert.match(suite, /VERCEL_AUTOMATION_BYPASS_SECRET:\s*\$\{\{ secrets\.VERCEL_AUTOMATION_BYPASS_SECRET \}\}/); + assert.match(suite, /VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET:\s*\$\{\{ secrets\.VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET \}\}/); + assert.match(suite, /npx nx e2e website --skip-nx-cache/); + assert.doesNotMatch(job, /vercel promote/); + assert.doesNotMatch(job, /vercel remove/); + }); + it('gates Cockpit deployment on the production Website smoke even for Cockpit-only changes', async () => { const deployJob = await readDeployJob(); const websiteOrCockpit = From 2f27e77e41106a3314b1622875c5815307d71f43 Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 17:17:41 -0700 Subject: [PATCH 11/17] ci: drop --skip-domain from preview deploys, validate preview URLs, request a SameSite=None bypass cookie Co-Authored-By: Claude Fable 5.1 --- .github/workflows/ci.yml | 14 ++++---------- apps/website/e2e/runtime-bypass-setup.ts | 4 ++-- apps/website/src/runtime-bypass-setup.spec.ts | 4 ++-- scripts/ci-workflow.spec.mjs | 10 ++++++---- 4 files changed, 14 insertions(+), 18 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ac74d3be3..be191c944 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -393,7 +393,7 @@ jobs: id: deploy_cockpit_preview run: | set -euo pipefail - url=$(npx vercel deploy --prebuilt --archive=tgz --skip-domain --yes --env COCKPIT_WEBSITE_ORIGIN=https://threadplane.ai --token=${{ secrets.VERCEL_TOKEN }} | tail -n 1) + url=$(npx vercel deploy --prebuilt --archive=tgz --yes --env COCKPIT_WEBSITE_ORIGIN=https://threadplane.ai --token=${{ secrets.VERCEL_TOKEN }} | tail -n 1) if [ -z "$url" ]; then echo "::error::vercel deploy produced no deployment URL for the throwaway cockpit preview." exit 1 @@ -817,10 +817,7 @@ jobs: EOF npx vercel pull --yes --environment=preview --token=${{ secrets.VERCEL_TOKEN }} url=$(npx vercel deploy --prebuilt --yes --token=${{ secrets.VERCEL_TOKEN }} | tail -n 1) - if [ -z "$url" ]; then - echo "::error::vercel deploy produced no deployment URL for the examples preview." - exit 1 - fi + url=$(node -e 'const parsed = new URL(process.argv[1]); if (parsed.protocol !== "https:" || parsed.username || parsed.password || parsed.pathname !== "/" || parsed.search || parsed.hash) throw new Error("Vercel returned a non-origin examples preview URL"); process.stdout.write(parsed.origin)' "$url") echo "examples deployment: $url" npx vercel alias set "$url" "${{ steps.aliases.outputs.examples }}" --scope=${{ secrets.VERCEL_ORG_ID }} --token=${{ secrets.VERCEL_TOKEN }} - name: Build, deploy, and alias the Website preview @@ -836,11 +833,8 @@ jobs: npx vercel pull --yes --environment=preview --token=${{ secrets.VERCEL_TOKEN }} rm -rf .vercel/output npx vercel build --token=${{ secrets.VERCEL_TOKEN }} - url=$(npx vercel deploy --prebuilt --archive=tgz --skip-domain --yes --token=${{ secrets.VERCEL_TOKEN }} | tail -n 1) - if [ -z "$url" ]; then - echo "::error::vercel deploy produced no deployment URL for the Website preview." - exit 1 - fi + url=$(npx vercel deploy --prebuilt --archive=tgz --yes --token=${{ secrets.VERCEL_TOKEN }} | tail -n 1) + url=$(node -e 'const parsed = new URL(process.argv[1]); if (parsed.protocol !== "https:" || parsed.username || parsed.password || parsed.pathname !== "/" || parsed.search || parsed.hash) throw new Error("Vercel returned a non-origin Website preview URL"); process.stdout.write(parsed.origin)' "$url") echo "website deployment: $url" npx vercel alias set "$url" "${{ steps.aliases.outputs.website }}" --scope=${{ secrets.VERCEL_ORG_ID }} --token=${{ secrets.VERCEL_TOKEN }} - name: Run the Website suite against the aliased preview diff --git a/apps/website/e2e/runtime-bypass-setup.ts b/apps/website/e2e/runtime-bypass-setup.ts index 042e8f74c..5acb21ebc 100644 --- a/apps/website/e2e/runtime-bypass-setup.ts +++ b/apps/website/e2e/runtime-bypass-setup.ts @@ -9,7 +9,7 @@ import { dirname, resolve } from 'node:path'; * extraHTTPHeaders is global, so the Website project's secret would reach * the runtime origin and be rejected. Vercel issues a per-origin `_vercel_jwt` * bypass cookie when a request carries the owning project's secret together - * with `x-vercel-set-bypass-cookie=true`; this setup obtains that cookie once + * with `x-vercel-set-bypass-cookie=samesitenone`; this setup obtains that cookie once * and stores it as storage state, so every browser context carries it and * the runtime iframe and its subresources load. The examples secret travels * only in this one request. @@ -59,7 +59,7 @@ export function buildRuntimeBypassUrl(origin: string, secret: string): string { } const url = new URL(parsed.origin); url.searchParams.set('x-vercel-protection-bypass', secret); - url.searchParams.set('x-vercel-set-bypass-cookie', 'true'); + url.searchParams.set('x-vercel-set-bypass-cookie', 'samesitenone'); return url.toString(); } diff --git a/apps/website/src/runtime-bypass-setup.spec.ts b/apps/website/src/runtime-bypass-setup.spec.ts index f683a700c..ca5f13965 100644 --- a/apps/website/src/runtime-bypass-setup.spec.ts +++ b/apps/website/src/runtime-bypass-setup.spec.ts @@ -16,7 +16,7 @@ describe('runtime bypass setup', () => { 'examples-secret' ) ).toBe( - 'https://threadplane-examples-pr-7-cacheplane.vercel.app/?x-vercel-protection-bypass=examples-secret&x-vercel-set-bypass-cookie=true' + 'https://threadplane-examples-pr-7-cacheplane.vercel.app/?x-vercel-protection-bypass=examples-secret&x-vercel-set-bypass-cookie=samesitenone' ); }); @@ -128,7 +128,7 @@ describe('runtime bypass setup', () => { ).resolves.toBe('seeded'); expect(requestPath).toBe( - `/?x-vercel-protection-bypass=${secret}&x-vercel-set-bypass-cookie=true` + `/?x-vercel-protection-bypass=${secret}&x-vercel-set-bypass-cookie=samesitenone` ); expect(existsSync(RUNTIME_BYPASS_STORAGE_STATE)).toBe(true); diff --git a/scripts/ci-workflow.spec.mjs b/scripts/ci-workflow.spec.mjs index d86c2c081..8fa3ab12a 100644 --- a/scripts/ci-workflow.spec.mjs +++ b/scripts/ci-workflow.spec.mjs @@ -614,7 +614,8 @@ describe('CI workflow', () => { assert.match(build, /vercel build --local-config vercel\.cockpit\.json/); assert.doesNotMatch(build, /--prod/); assert.match(deploy, /id:\s*deploy_cockpit_preview/); - assert.match(deploy, /vercel deploy --prebuilt --archive=tgz --skip-domain --yes/); + assert.match(deploy, /vercel deploy --prebuilt --archive=tgz --yes/); + assert.doesNotMatch(deploy, /--skip-domain/); assert.doesNotMatch(deploy, /--prod/); assert.match(deploy, /--env COCKPIT_WEBSITE_ORIGIN=https:\/\/threadplane\.ai/); assert.match( @@ -682,7 +683,7 @@ describe('CI workflow', () => { assert.match(examples, /vercel deploy --prebuilt --yes/); assert.doesNotMatch(examples, /--prod/); assert.match(examples, /set -euo pipefail/); - assert.match(examples, /if \[ -z "\$url" \]/); + assert.match(examples, /new URL\(process\.argv\[1\]\)/); assert.match( examples, /vercel alias set "\$url" "\$\{\{ steps\.aliases\.outputs\.examples \}\}" --scope=\$\{\{ secrets\.VERCEL_ORG_ID \}\}/ @@ -696,10 +697,11 @@ describe('CI workflow', () => { ); assert.match(website, /GROWTH_FORM_POLICY:\s*growth_v1/); assert.match(website, /vercel build --token/); - assert.match(website, /vercel deploy --prebuilt --archive=tgz --skip-domain --yes/); + assert.match(website, /vercel deploy --prebuilt --archive=tgz --yes/); + assert.doesNotMatch(website, /--skip-domain/); assert.doesNotMatch(website, /--prod/); assert.match(website, /set -euo pipefail/); - assert.match(website, /if \[ -z "\$url" \]/); + assert.match(website, /new URL\(process\.argv\[1\]\)/); assert.match( website, /vercel alias set "\$url" "\$\{\{ steps\.aliases\.outputs\.website \}\}" --scope=\$\{\{ secrets\.VERCEL_ORG_ID \}\}/ From 7c5bcbcf50eb856a1033368bb8ad7fdb3418ac2b Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 17:22:30 -0700 Subject: [PATCH 12/17] docs(specs): gate the preview lanes only when they were eligible to run Co-Authored-By: Claude Fable 5.1 --- ...2026-09-03-pr-side-deploy-verification-design.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/docs/superpowers/specs/2026-09-03-pr-side-deploy-verification-design.md b/docs/superpowers/specs/2026-09-03-pr-side-deploy-verification-design.md index 85ba6546c..75fad7599 100644 --- a/docs/superpowers/specs/2026-09-03-pr-side-deploy-verification-design.md +++ b/docs/superpowers/specs/2026-09-03-pr-side-deploy-verification-design.md @@ -156,10 +156,15 @@ examples secret never appears in `extraHTTPHeaders`. ### Required gate -Both jobs are added to `required-pr-checks.needs`, to its `RESULT_*` / -`SCOPE_*` env, and to the `require_scoped` table under the `website_e2e` -and `cockpit_deploy_smoke` scope keys respectively. On forks the jobs are -skipped, which `require_scoped` already tolerates. +Both jobs are added to `required-pr-checks.needs` and to its `RESULT_*` env. +`require_scoped` demands `success` for any in-scope job, and the scope keys +are computed from changed files alone, so on a fork PR the lanes can be in +scope yet legitimately skipped. A wrapper `require_preview` therefore +applies the scoped rule only when `PREVIEW_LANES_ELIGIBLE` (the same +same-repo-or-merge-queue expression the jobs use) is true, and otherwise +treats the lane as unselected, where only a real failure or cancellation is +an error. The wrapper is used for the `website_e2e` and +`cockpit_deploy_smoke` entries. ### Provisioning From 11d5901620ae3f3e49084a261a3d59cfbeabdd9e Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 17:24:09 -0700 Subject: [PATCH 13/17] ci: gate merges on the PR-side preview verifications when they were eligible to run Co-Authored-By: Claude Fable 5.1 --- .github/workflows/ci.yml | 27 +++++++++++++++++++++++++++ scripts/ci-workflow.spec.mjs | 31 +++++++++++++++++++++++++++++++ 2 files changed, 58 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index be191c944..6e9d685f3 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -861,6 +861,8 @@ jobs: - examples-ag-ui-e2e - cockpit-e2e-summary - website-e2e + - website-preview-e2e + - cockpit-preview-smoke - posthog-sync-plan - scripts-tests - growth-lifecycle @@ -886,6 +888,8 @@ jobs: RESULT_EXAMPLES_AG_UI_E2E: ${{ needs.examples-ag-ui-e2e.result }} RESULT_COCKPIT_E2E: ${{ needs.cockpit-e2e-summary.result }} RESULT_WEBSITE_E2E: ${{ needs.website-e2e.result }} + RESULT_WEBSITE_PREVIEW_E2E: ${{ needs.website-preview-e2e.result }} + RESULT_COCKPIT_PREVIEW_SMOKE: ${{ needs.cockpit-preview-smoke.result }} RESULT_POSTHOG: ${{ needs.posthog-sync-plan.result }} RESULT_SCRIPTS_TESTS: ${{ needs.scripts-tests.result }} RESULT_GROWTH_LIFECYCLE: ${{ needs.growth-lifecycle.result }} @@ -904,6 +908,11 @@ jobs: SCOPE_POSTHOG: ${{ needs.ci-scope.outputs.posthog }} SCOPE_SCRIPTS_TESTS: ${{ needs.ci-scope.outputs.scripts_tests }} SCOPE_GROWTH_LIFECYCLE: ${{ needs.ci-scope.outputs.growth_lifecycle }} + # The preview lanes need repository secrets, so they skip on fork + # PRs. Their scope keys are computed from changed files alone, so on + # a fork they can be in scope yet legitimately skipped. This mirrors + # the lanes' own `if` so the gate only demands them when they could run. + PREVIEW_LANES_ELIGIBLE: ${{ github.event_name == 'merge_group' || github.event.pull_request.head.repo.full_name == github.repository }} run: | set -euo pipefail @@ -939,6 +948,22 @@ jobs: fi } + # Scoped check for the PR-side preview lanes: enforced only when the + # lanes were eligible to run; otherwise treated as unselected, where + # only a real failure or cancellation is an error. + require_preview() { + local scope_key="$1" + local label="$2" + local result="$3" + local scoped="$4" + + if [[ "$PREVIEW_LANES_ELIGIBLE" == "true" ]]; then + require_scoped "$scope_key" "$label" "$result" "$scoped" + else + require_scoped "$scope_key" "$label" "$result" "false" + fi + } + require_always "CI scope" "$RESULT_CI_SCOPE" require_scoped "library" "Library — lint / test / build" "$RESULT_LIBRARY" "$SCOPE_LIBRARY" require_scoped \ @@ -960,6 +985,8 @@ jobs: "$SCOPE_EXAMPLES_AG_UI" require_scoped "cockpit_e2e" "Cockpit — e2e" "$RESULT_COCKPIT_E2E" "$SCOPE_COCKPIT_E2E" require_scoped "website_e2e" "Website — e2e" "$RESULT_WEBSITE_E2E" "$SCOPE_WEBSITE_E2E" + require_preview "website_e2e" "Website — e2e (deployed preview)" "$RESULT_WEBSITE_PREVIEW_E2E" "$SCOPE_WEBSITE_E2E" + require_preview "cockpit_deploy_smoke" "Cockpit — immutable preview smoke" "$RESULT_COCKPIT_PREVIEW_SMOKE" "$SCOPE_COCKPIT_DEPLOY_SMOKE" require_scoped "posthog" "PostHog — dashboards-as-code drift check" "$RESULT_POSTHOG" "$SCOPE_POSTHOG" require_scoped "scripts_tests" "Scripts — generator / proxy vitest suites" "$RESULT_SCRIPTS_TESTS" "$SCOPE_SCRIPTS_TESTS" require_scoped "growth_lifecycle" "Growth lifecycle — Node 22" "$RESULT_GROWTH_LIFECYCLE" "$SCOPE_GROWTH_LIFECYCLE" diff --git a/scripts/ci-workflow.spec.mjs b/scripts/ci-workflow.spec.mjs index 8fa3ab12a..a08a3f6dd 100644 --- a/scripts/ci-workflow.spec.mjs +++ b/scripts/ci-workflow.spec.mjs @@ -716,6 +716,35 @@ describe('CI workflow', () => { assert.doesNotMatch(job, /vercel remove/); }); + it('requires both PR-side preview verifications through the scoped gate', async () => { + const required = await readRequiredPrChecksJob(); + const needs = readJobNeeds(required); + + assert.ok(needs.includes('website-preview-e2e')); + assert.ok(needs.includes('cockpit-preview-smoke')); + assert.match( + required, + /RESULT_WEBSITE_PREVIEW_E2E:\s*\$\{\{ needs\.website-preview-e2e\.result \}\}/ + ); + assert.match( + required, + /RESULT_COCKPIT_PREVIEW_SMOKE:\s*\$\{\{ needs\.cockpit-preview-smoke\.result \}\}/ + ); + assert.match( + required, + /PREVIEW_LANES_ELIGIBLE:\s*\$\{\{ github\.event_name == 'merge_group' \|\| github\.event\.pull_request\.head\.repo\.full_name == github\.repository \}\}/ + ); + assert.match(required, /require_preview\(\) \{/); + assert.match( + required, + /require_preview "website_e2e" "Website — e2e \(deployed preview\)" "\$RESULT_WEBSITE_PREVIEW_E2E" "\$SCOPE_WEBSITE_E2E"/ + ); + assert.match( + required, + /require_preview "cockpit_deploy_smoke" "Cockpit — immutable preview smoke" "\$RESULT_COCKPIT_PREVIEW_SMOKE" "\$SCOPE_COCKPIT_DEPLOY_SMOKE"/ + ); + }); + it('gates Cockpit deployment on the production Website smoke even for Cockpit-only changes', async () => { const deployJob = await readDeployJob(); const websiteOrCockpit = @@ -1121,6 +1150,8 @@ describe('CI workflow', () => { 'examples-ag-ui-e2e', 'cockpit-e2e-summary', 'website-e2e', + 'website-preview-e2e', + 'cockpit-preview-smoke', 'posthog-sync-plan', 'scripts-tests', 'growth-lifecycle', From ac97bd8ad16256c9c49836b8b5e1696978d46bb0 Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 17:30:50 -0700 Subject: [PATCH 14/17] ci: name a skipped-but-eligible preview lane; document the PR-side lanes Co-Authored-By: Claude Fable 5.1 --- .github/workflows/ci.yml | 6 +++++ CONTRIBUTING.md | 26 +++++++++++++++++++ .../2026-09-03-pr-side-deploy-verification.md | 16 ++++++++---- scripts/ci-workflow.spec.mjs | 1 + 4 files changed, 44 insertions(+), 5 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6e9d685f3..7eb4ada21 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -957,6 +957,12 @@ jobs: local result="$3" local scoped="$4" + if [[ "$PREVIEW_LANES_ELIGIBLE" == "true" && "$scoped" == "true" && "$result" == "skipped" ]]; then + echo "::error::${label} was eligible and in scope ${scope_key} but was skipped — its job-level if no longer matches the gate's PREVIEW_LANES_ELIGIBLE expression." + failed=1 + return + fi + if [[ "$PREVIEW_LANES_ELIGIBLE" == "true" ]]; then require_scoped "$scope_key" "$label" "$result" "$scoped" else diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 52cd556bf..4ebe26fd0 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -195,6 +195,32 @@ branch protection and has no rulesets, so the unprivileged `GET /repos/{owner}/{repo}/rules/branches/main` endpoint returns `[]` and is not an alternative. +### PR-side deploy verification + +Two lanes run the deploy job's verification on pull requests against real +Vercel previews, so deploy-only failures surface before merge: + +- **Website — e2e (deployed preview)** builds and deploys the Website and + the examples as previews under deterministic aliases + (`threadplane-pr--cacheplane.vercel.app` and + `threadplane-examples-pr--cacheplane.vercel.app`; `mq-` for + merge-queue candidates) and runs the ordinary suite against the Website + alias. The runtime iframe loads because the examples are assembled with + the Website alias in their parent-origin policy and Playwright seeds the + examples origin's bypass cookie (`apps/website/e2e/runtime-bypass-setup.ts`). + A later push re-points both aliases; the deployments behind them are kept. +- **Cockpit — immutable preview smoke** deploys a throwaway cockpit preview, + runs the exhaustive redirect smoke against it, and removes it. + +Both need repository secrets and therefore skip on fork PRs; the required +gate only demands them when they were eligible to run. Each Vercel project +has its own Protection Bypass for Automation secret: +`VERCEL_AUTOMATION_BYPASS_SECRET` (Website), +`VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET`, and +`VERCEL_COCKPIT_AUTOMATION_BYPASS_SECRET`. A secret added while a run is in +flight does not reach that run; re-run after provisioning. Never pass +`--skip-domain` to a preview deploy; Vercel requires it to accompany `--prod`. + ## Code review Every PR gets a genuine advisory AI code review diff --git a/docs/superpowers/plans/2026-09-03-pr-side-deploy-verification.md b/docs/superpowers/plans/2026-09-03-pr-side-deploy-verification.md index 0300128d4..0dd9e6d57 100644 --- a/docs/superpowers/plans/2026-09-03-pr-side-deploy-verification.md +++ b/docs/superpowers/plans/2026-09-03-pr-side-deploy-verification.md @@ -699,16 +699,16 @@ Append inside the `describe('CI workflow', ...)` block in `scripts/ci-workflow.s ); assert.match( required, - /require_scoped "website_e2e" "Website — e2e \(deployed preview\)" "\$RESULT_WEBSITE_PREVIEW_E2E" "\$SCOPE_WEBSITE_E2E"/ + /require_preview "website_e2e" "Website — e2e \(deployed preview\)" "\$RESULT_WEBSITE_PREVIEW_E2E" "\$SCOPE_WEBSITE_E2E"/ ); assert.match( required, - /require_scoped "cockpit_deploy_smoke" "Cockpit — immutable preview smoke" "\$RESULT_COCKPIT_PREVIEW_SMOKE" "\$SCOPE_COCKPIT_DEPLOY_SMOKE"/ + /require_preview "cockpit_deploy_smoke" "Cockpit — immutable preview smoke" "\$RESULT_COCKPIT_PREVIEW_SMOKE" "\$SCOPE_COCKPIT_DEPLOY_SMOKE"/ ); }); ``` -Note on forks: both jobs skip there, `require_scoped` treats `skipped` as acceptable, so the gate stays green on forks without these lanes. +Note on forks: `require_scoped` demands `success` for any in-scope job, and the scope keys are file-derived, so on a fork PR the lanes can be in scope yet legitimately skipped. The gate therefore uses a `require_preview` wrapper that applies the scoped rule only when `PREVIEW_LANES_ELIGIBLE` (the lanes' own same-repo-or-merge-queue clause) is true; see the implementation in `ci.yml`. - [ ] **Step 2: Run the guard to verify it fails** @@ -731,11 +731,17 @@ In the step `env:` block, add after `RESULT_WEBSITE_E2E: ${{ needs.website-e2e.r RESULT_COCKPIT_PREVIEW_SMOKE: ${{ needs.cockpit-preview-smoke.result }} ``` +Also add, after the last `SCOPE_*` line: + +```yaml + PREVIEW_LANES_ELIGIBLE: ${{ github.event_name == 'merge_group' || github.event.pull_request.head.repo.full_name == github.repository }} +``` + In the `run:` script, add after the line `require_scoped "website_e2e" "Website — e2e" "$RESULT_WEBSITE_E2E" "$SCOPE_WEBSITE_E2E"`: ```bash - require_scoped "website_e2e" "Website — e2e (deployed preview)" "$RESULT_WEBSITE_PREVIEW_E2E" "$SCOPE_WEBSITE_E2E" - require_scoped "cockpit_deploy_smoke" "Cockpit — immutable preview smoke" "$RESULT_COCKPIT_PREVIEW_SMOKE" "$SCOPE_COCKPIT_DEPLOY_SMOKE" + require_preview "website_e2e" "Website — e2e (deployed preview)" "$RESULT_WEBSITE_PREVIEW_E2E" "$SCOPE_WEBSITE_E2E" + require_preview "cockpit_deploy_smoke" "Cockpit — immutable preview smoke" "$RESULT_COCKPIT_PREVIEW_SMOKE" "$SCOPE_COCKPIT_DEPLOY_SMOKE" ``` - [ ] **Step 4: Run the guards and validate YAML** diff --git a/scripts/ci-workflow.spec.mjs b/scripts/ci-workflow.spec.mjs index a08a3f6dd..48308f1b8 100644 --- a/scripts/ci-workflow.spec.mjs +++ b/scripts/ci-workflow.spec.mjs @@ -743,6 +743,7 @@ describe('CI workflow', () => { required, /require_preview "cockpit_deploy_smoke" "Cockpit — immutable preview smoke" "\$RESULT_COCKPIT_PREVIEW_SMOKE" "\$SCOPE_COCKPIT_DEPLOY_SMOKE"/ ); + assert.match(required, /was eligible and in scope \$\{scope_key\} but was skipped/); }); it('gates Cockpit deployment on the production Website smoke even for Cockpit-only changes', async () => { From 0194e1193eeda3a9946a0b22ef9847ed2287110d Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 17:41:02 -0700 Subject: [PATCH 15/17] ci: validate the cockpit preview URL before removal; keep the bypass cookie header off the runtime origin; cover the 307 bypass answer Co-Authored-By: Claude Fable 5.1 --- .github/workflows/ci.yml | 9 ++-- apps/website/playwright.config.ts | 10 ++++- apps/website/src/playwright-config.spec.ts | 3 +- apps/website/src/runtime-bypass-setup.spec.ts | 41 +++++++++++++++++++ scripts/ci-workflow.spec.mjs | 4 +- 5 files changed, 60 insertions(+), 7 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 7eb4ada21..6b260bb73 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -394,10 +394,7 @@ jobs: run: | set -euo pipefail url=$(npx vercel deploy --prebuilt --archive=tgz --yes --env COCKPIT_WEBSITE_ORIGIN=https://threadplane.ai --token=${{ secrets.VERCEL_TOKEN }} | tail -n 1) - if [ -z "$url" ]; then - echo "::error::vercel deploy produced no deployment URL for the throwaway cockpit preview." - exit 1 - fi + url=$(node -e 'const parsed = new URL(process.argv[1]); if (parsed.protocol !== "https:" || parsed.username || parsed.password || parsed.pathname !== "/" || parsed.search || parsed.hash) throw new Error("Vercel returned a non-origin cockpit preview URL"); process.stdout.write(parsed.origin)' "$url") echo "deployment_url=$url" >> "$GITHUB_OUTPUT" - name: Exhaustively verify the cockpit preview run: | @@ -409,6 +406,10 @@ jobs: env: VERCEL_AUTOMATION_BYPASS_SECRET: ${{ secrets.VERCEL_COCKPIT_AUTOMATION_BYPASS_SECRET }} - name: Remove the throwaway cockpit preview + # Removal runs on every outcome, including a failed smoke: the smoke + # log carries the full probe-by-probe report, and keeping a failed + # artifact would only accumulate protected deployments nobody can + # open without the bypass. # The artifact exists only to be smoked. `--safe` refuses to remove a # deployment that carries an alias, so this can never touch the # production artifact. `remove` takes a bare URL, so, like `promote`, diff --git a/apps/website/playwright.config.ts b/apps/website/playwright.config.ts index e0234b060..aefd7d30b 100644 --- a/apps/website/playwright.config.ts +++ b/apps/website/playwright.config.ts @@ -89,7 +89,15 @@ export const createWebsitePlaywrightConfig = ( extraHTTPHeaders: { 'x-vercel-protection-bypass': environment['VERCEL_AUTOMATION_BYPASS_SECRET'], - 'x-vercel-set-bypass-cookie': 'true', + // extraHTTPHeaders is global, so with a runtime bypass in play + // this header would also reach the examples origin, asking it + // to set a cookie for a secret that belongs to the Website + // project. The bypass header alone still authorizes every + // Website request; only skip the cookie request when a + // runtime bypass is active. + ...(runtimeBypass + ? {} + : { 'x-vercel-set-bypass-cookie': 'true' }), }, } : {}), diff --git a/apps/website/src/playwright-config.spec.ts b/apps/website/src/playwright-config.spec.ts index e69154cfb..8e55e3308 100644 --- a/apps/website/src/playwright-config.spec.ts +++ b/apps/website/src/playwright-config.spec.ts @@ -152,9 +152,10 @@ describe('Website Playwright configuration', () => { expect(JSON.stringify(both.use?.extraHTTPHeaders)).not.toContain( 'examples-secret' ); + // The set-cookie header is dropped so the runtime origin is never asked + // to issue a cookie for a wrong-project secret. expect(both.use?.extraHTTPHeaders).toEqual({ 'x-vercel-protection-bypass': 'website-secret', - 'x-vercel-set-bypass-cookie': 'true', }); }); diff --git a/apps/website/src/runtime-bypass-setup.spec.ts b/apps/website/src/runtime-bypass-setup.spec.ts index ca5f13965..09ce066be 100644 --- a/apps/website/src/runtime-bypass-setup.spec.ts +++ b/apps/website/src/runtime-bypass-setup.spec.ts @@ -149,6 +149,47 @@ describe('runtime bypass setup', () => { } }); + it('seeds a normalized cookie from a loopback 307 bypass response', async () => { + rmSync(RUNTIME_BYPASS_STORAGE_STATE, { force: true }); + const server = createServer((_req, res) => { + res.statusCode = 307; + res.setHeader('Location', '/'); + res.setHeader( + 'Set-Cookie', + '_vercel_jwt=abc307; Path=/; HttpOnly; SameSite=Lax' + ); + res.end(); + }); + await new Promise((res) => server.listen(0, '127.0.0.1', res)); + const port = (server.address() as AddressInfo).port; + const secret = 'loopback-secret-307'; + + try { + await expect( + seedRuntimeBypass({ + RUNTIME_BYPASS_ORIGIN: `http://127.0.0.1:${port}`, + VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET: secret, + }) + ).resolves.toBe('seeded'); + + expect(existsSync(RUNTIME_BYPASS_STORAGE_STATE)).toBe(true); + const state = JSON.parse( + readFileSync(RUNTIME_BYPASS_STORAGE_STATE, 'utf8') + ); + const cookie = state.cookies.find( + (c: { name: string }) => c.name === '_vercel_jwt' + ); + expect(cookie).toMatchObject({ + value: 'abc307', + sameSite: 'None', + secure: true, + }); + } finally { + await new Promise((res) => server.close(() => res())); + rmSync(RUNTIME_BYPASS_STORAGE_STATE, { force: true }); + } + }); + it('rejects with the status code when the loopback response sets no cookie', async () => { const server = createServer((_req, res) => { res.statusCode = 302; diff --git a/scripts/ci-workflow.spec.mjs b/scripts/ci-workflow.spec.mjs index 48308f1b8..7b83beb43 100644 --- a/scripts/ci-workflow.spec.mjs +++ b/scripts/ci-workflow.spec.mjs @@ -629,11 +629,13 @@ describe('CI workflow', () => { ); assert.doesNotMatch(job, /vercel promote/); assert.match(deploy, /set -euo pipefail/); - assert.match(deploy, /if \[ -z "\$url" \]/); + assert.match(deploy, /new URL\(process\.argv\[1\]\)/); assert.match(smoke, /--retries 20 --retry-delay-ms 5000/); const cleanup = readNamedStep(job, 'Remove the throwaway cockpit preview'); assert.match(cleanup, /if:\s*always\(\) && steps\.deploy_cockpit_preview\.outputs\.deployment_url != ''/); assert.match(cleanup, /continue-on-error:\s*true/); + const rawCleanup = readNamedStep(workflow, 'Remove the throwaway cockpit preview'); + assert.match(rawCleanup, /Removal runs on every outcome/); assert.match( cleanup, /vercel remove "\$\{\{ steps\.deploy_cockpit_preview\.outputs\.deployment_url \}\}" --safe --yes --scope=\$\{\{ secrets\.VERCEL_ORG_ID \}\}/ From 01a310955142e1341b083da6f5ccab7f7c67819e Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 17:41:12 -0700 Subject: [PATCH 16/17] =?UTF-8?q?docs(specs):=20match=20the=20implementati?= =?UTF-8?q?on=20=E2=80=94=20no=20--skip-domain=20on=20previews,=20samesite?= =?UTF-8?q?none,=20cockpit=20cleanup?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- ...9-03-pr-side-deploy-verification-design.md | 27 ++++++++++++------- 1 file changed, 18 insertions(+), 9 deletions(-) diff --git a/docs/superpowers/specs/2026-09-03-pr-side-deploy-verification-design.md b/docs/superpowers/specs/2026-09-03-pr-side-deploy-verification-design.md index 75fad7599..3b1a88bf3 100644 --- a/docs/superpowers/specs/2026-09-03-pr-side-deploy-verification-design.md +++ b/docs/superpowers/specs/2026-09-03-pr-side-deploy-verification-design.md @@ -105,8 +105,9 @@ deployments. `vercel pull --environment=preview`, `vercel build` (no `--prod`) with `NEXT_PUBLIC_COCKPIT_RUNTIME_BASE_URL=https://` and `GROWTH_FORM_POLICY=growth_v1`, `vercel deploy --prebuilt --archive=tgz - --skip-domain --yes` (no `--prod`), capture the URL, - `vercel alias set `. + --yes` (no `--prod`, and no `--skip-domain`: Vercel requires that flag + to accompany `--prod`), validate the captured URL is a bare https + origin, `vercel alias set `. 5. Guard: fail with a provisioning message if either `VERCEL_AUTOMATION_BYPASS_SECRET` or `VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET` is empty. @@ -123,10 +124,14 @@ secret. When `VERCEL_EXAMPLES_AUTOMATION_BYPASS_SECRET` and `RUNTIME_BYPASS_ORIGIN` are both set, the config also registers a `globalSetup` script and points `use.storageState` at a file the script writes. The script requests -`${RUNTIME_BYPASS_ORIGIN}/?x-vercel-protection-bypass=&x-vercel-set-bypass-cookie=true` -with a Playwright request context, which makes Vercel answer with the -`_vercel_jwt` bypass cookie scoped to that origin, and saves the resulting -cookies as storage state. Every browser context then carries the examples +`${RUNTIME_BYPASS_ORIGIN}/?x-vercel-protection-bypass=&x-vercel-set-bypass-cookie=samesitenone` +with a Playwright request context (redirects not followed), which makes +Vercel answer with the `_vercel_jwt` bypass cookie scoped to that origin; +the cookies are normalized to `SameSite=None; Secure` (the iframe is +cross-site) and saved as storage state. In this mode the global header +carries only `x-vercel-protection-bypass` for the Website secret and no +`x-vercel-set-bypass-cookie`, so the runtime origin is never asked to issue +a cookie for a wrong-project secret. Every browser context then carries the examples cookie, so the runtime iframe and its subresources load without any spec importing a custom fixture. The examples secret travels only in that one setup request; the Website secret keeps travelling only as the global @@ -145,11 +150,15 @@ examples secret never appears in `extraHTTPHeaders`. - Steps: `npm ci`; write `.vercel/project.json` for the cockpit project; `vercel pull --environment=preview`; `vercel build --local-config vercel.cockpit.json` with `COCKPIT_WEBSITE_ORIGIN=https://threadplane.ai` - (no `--prod`); `vercel deploy --prebuilt --archive=tgz --skip-domain --yes - --env COCKPIT_WEBSITE_ORIGIN=https://threadplane.ai` (no `--prod`); guard + (no `--prod`); `vercel deploy --prebuilt --archive=tgz --yes + --env COCKPIT_WEBSITE_ORIGIN=https://threadplane.ai` (no `--prod`, no + `--skip-domain`); validate the captured URL is a bare https origin; guard on `VERCEL_COCKPIT_AUTOMATION_BYPASS_SECRET`; run `apps/cockpit/scripts/deploy-smoke.ts --mode preview --retries 20 - --retry-delay-ms 5000` against the captured URL. + --retry-delay-ms 5000` against the captured URL; finally + `vercel remove --safe --yes --scope=` on every outcome, so + throwaway artifacts do not accumulate (`--safe` never touches an aliased, + i.e. production, deployment). - No alias is needed; the smoke takes the bare deployment URL. - The token-free `cockpit-deploy-smoke` dry-run job stays unchanged so forks keep a check. From d014700ca41a8f0d331a914a620196523d12e23f Mon Sep 17 00:00:00 2001 From: Brian Love Date: Thu, 3 Sep 2026 17:41:53 -0700 Subject: [PATCH 17/17] =?UTF-8?q?test:=20MUTATION=20CHECK=20=E2=80=94=20re?= =?UTF-8?q?introduce=20the=20host-bound=20route=20abort=20(do=20not=20merg?= =?UTF-8?q?e)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- apps/website/e2e/workspace-shell.spec.ts | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/apps/website/e2e/workspace-shell.spec.ts b/apps/website/e2e/workspace-shell.spec.ts index 4728b70f0..e1d53af12 100644 --- a/apps/website/e2e/workspace-shell.spec.ts +++ b/apps/website/e2e/workspace-shell.spec.ts @@ -779,10 +779,7 @@ test.describe('workspace shell', () => { // params Run mode stamps on every runtime URL rather than by host: against // the deployed site the frame loads from the production runtime origin, // and a `localhost:4300` route lets it reach ready before the assertion. - await page.route( - (url) => url.searchParams.has('cockpit_cap'), - (route) => route.abort() - ); + await page.route('http://localhost:4300/**', (request) => request.abort()); await page.setViewportSize({ width: 390, height: 844 }); await page.goto(`${streamingDocsPath}?mode=run`); await page.getByRole('button', { name: 'Open navigation' }).click();