Move bench.vortex.dev into this repository #113

Move vortex-data/benchmarks-website (bench.vortex.dev) into this repository, under bench/. After the move, this repository has the two web apps that are not built from the Vortex source: the marketing site and bench. The vortex monorepo keeps docs and explore.

The move gives bench the same supply-chain controls as the marketing site: Renovate with a 7-day cooldown (#132), dependency review, a secret scan and SHA-pinned actions. It also puts both Next.js apps in one place.

Stages

• 1. Import bench as it is: Import benchmarks-website under bench/ #112.
• 2. Move bench to Bun and Biome, in a Bun workspace. Make sure that the testcontainers suite still runs.
• 3. Put both apps on one design system. Done in Apply the new Vortex design system #120, without a shared package.
• 4. Move bench to Next.js 16. Check the Data Cache and revalidateTag behavior. Both apps run Next.js 16.4.1.
• 5. Serve both sites from one Next.js app and one Vercel project. Moved to its own issue: Serve vortex.dev and bench.vortex.dev from one app and one Vercel project #157.

Cutover steps for stage 1

Do these steps in this order.

• Freeze benchmarks-website. Close its Renovate onboarding PR (chore: Configure Renovate benchmarks-website#41). Move its open PRs (Fix missing timing summary penalties and dataset coverage benchmarks-website#32 to Update dependency resend to v6.12.2 #38) here with git format-patch and git am --directory=bench.
• Create the GitHub environments bench-preview and bench-production. bench-production accepts deployments from main only.
• Add the secret VERCEL_TOKEN to both environments. Use a token that is limited to the benchmarks-website project. Create it at vercel.com/account/tokens. (vercel tokens create does not work with a vercel login session.)
• Copy these repository variables from benchmarks-website: VERCEL_ORG_ID, GH_BENCH_SCHEMA_ROLE_ARN, RDS_BENCH_REGION, RDS_BENCH_INSTANCE_ENDPOINT and RDS_BENCH_DB_NAME. Set BENCH_SITE_BASE_URL to https://bench.vortex.dev.
• In Vercel, change the Root Directory of the benchmarks-website project from web/ to bench/web. After this change, benchmarks-website cannot deploy.
• Set the variable BENCH_VERCEL_PROJECT_ID. This starts bench deploys from this repository.
• Check a bench preview deploy from Import benchmarks-website under bench/ #112. Done on 2026-10-28: the preview builds and deploys, /api/health reports schema version 3, and the home page shows the same data as production.
• Merge Import benchmarks-website under bench/ #112. The merge deploys bench to production. Done on 2026-10-28: bench.vortex.dev serves f7e0ef5.
• Update the AWS trust policy of GitHubBenchmarkSchemaRole. Done on 2026-10-28. Before the change, the role trusted vortex@develop, vortex@ct/bench-v4 and benchmarks-website@develop. CloudTrail for the previous 90 days showed that only benchmarks-website used the role; the monorepo uses only the ingest role, which did not change. The role now trusts only vortex.dev@main:

  aws iam update-assume-role-policy --role-name GitHubBenchmarkSchemaRole --policy-document "$(bash bench/infra/provision.sh --print-trust schema)"

• Run gh workflow run bench-schema-deploy.yml -f dry_run=true. It must report no drift. Done on 2026-10-28: 10 applied, 0 pending, 0 orphaned.
• In the main ruleset, make "Bench: format, lint, build, test" a required check. Done on 2026-10-28. Let Renovate automerge bench updates #119 lets Renovate automerge bench updates again, except the Vercel CLI pin.
• In the main ruleset, replace the required Preview deployment with the required status check Vercel. Done on 2026-10-28. On the bench-only Update dependency vitest to v4.1.11 [SECURITY] #118, all required checks pass and the PR can merge.
• Point the README and description of benchmarks-website to vortex-data/vortex.dev/tree/main/bench. Then archive it.
• Merge chore: remove the Cloudflare Pages deploys and update bench paths vortex#10913. It updates the bench paths in comments in the monorepo.

Open work ported from benchmarks-website

Connor's open PRs, ported with his authorship and sign-off. Merge #126 before #127, which is stacked on it. The originals and Renovate's vortex-data/benchmarks-website#41 were closed on 2026-10-28 with pointers to the ports. When the ports merge, archive the repository.

• Fix missing timing summary penalties and dataset coverage #123: ports Fix missing timing summary penalties and dataset coverage benchmarks-website#32 (summary penalties and coverage). It fixes empty datasets issue? benchmarks-website#25.
• Bound database pool waits and attach Vercel cleanup #124: ports Bound database pool waits and attach Vercel cleanup benchmarks-website#33 (database pool waits and Vercel cleanup).
• Keep bundle failures shared until an explicit retry #125: ports Keep bundle failures shared until an explicit retry benchmarks-website#36 (shared retry for bundle failures).
• Test the migration runner and separate drift from operational failures #126: ports Test the migration runner and separate drift from operational failures benchmarks-website#37 (migration runner tests, and drift versus operational failures).
• Deploy the tested web commit and verify its public read paths #127: ports Deploy the tested web commit and verify its public read paths benchmarks-website#38 (deploy verification).

Decisions

• The import is one squashed commit. The main ruleset allows only squash merges, so the merge removes the history anyway. Also, 10 of the 39 bench commits fail the DCO check, and only their authors can fix them. The history stays in the archived repository.
• CI runs only for the app that changed. The changed-paths action finds the apps that a change touches. The jobs use if:, not on.paths. A required check that never starts blocks the merge, but a skipped job counts as passed.
• The Vercel CLI has its own npm lockfile. It is in .github/vercel-cli, and overrides there patch the advisories in dependencies that the CLI pins (ci(bench): lock the Vercel CLI and patch its advisories #137). Before ci(bench): lock the Vercel CLI and patch its advisories #137, the workflow installed the CLI without a lockfile. Stage 5 removes the CLI, because the Vercel git integration will deploy bench.
• The marketing site does not build for bench-only commits. vercel.json has an Ignored Build Step. When Vercel skips a build, it sets the Vercel status to success, but it does not create a GitHub deployment. The main ruleset requires a Preview deployment, so it could block bench-only PRs. A required Vercel status check gives the same protection without this problem.

Advisories

• When Import benchmarks-website under bench/ #112 merged, bench used next 15.5.28 and undici 7.30.1. Both were inside the cooldown, so bench/web/pnpm-workspace.yaml exempted exactly these versions (approved on 2026-10-28).
• bench now uses Next.js 16.4.1 and Bun. On 2026-10-29, bun audit reported no advisories for either app.
• Remove the minimumReleaseAgeExclude list from bench/web/pnpm-workspace.yaml after 2026-10-29 14:18 UTC. By then every version in it is past the window. The file went away when bench moved to Bun.
• CLAUDE.md records how each advisory is handled, including those in the Vercel CLI.

Follow-ups from review of #112

These are in the bench code as it was imported. They are not caused by the move.

• bench/web/lib/cache-fill.ts holds one pool connection for the advisory lock while the fill queries the same pool. With BENCH_DB_POOL_MAX=1, this deadlocked. Fixed in fix(bench): never size the database pool below 2 #146: the pool is never smaller than 2.
• bench/infra/provision.sh skipped the public ingress rule when the security group had any rule that started at port 5432. Fixed in fix(bench): match the exact ingress rule before skipping it in provision.sh #142: it checks the protocol, both ports and 0.0.0.0/0 before it skips.

AWS hardening

A review of the bench AWS resources on 2026-10-28 found these items. #121 records the settings in bench/infra/provision.sh and bench/infra/README.md.

• Turn on deletion protection and storage autoscaling (to 100 GiB) for vortex-bench-prod.
• Take a pg_dump of the database to an operator's computer before the changes.
• Do a point-in-time restore test and write bench/infra/RESTORE.md. The restore took 10 minutes. The ledger and the row counts were correct.
• Upgrade Postgres from 16.4 to 16.15 and apply the pending OS update. The engine upgrade stopped connections for 33 seconds, and the OS update for about 3 minutes.
• Turn on automatic minor version upgrades. They apply in the maintenance window, Saturday 09:58 UTC.
• Replicate the automated backups to us-west-2, with 14 days of point-in-time recovery.
• Add CloudWatch alarms for free storage, CPU and connections, and an RDS event subscription for failures and low storage. They send to the incident.io source "AWS CloudWatch (vortex)". A test alert opened and resolved.
• Delete the unused RDS Proxy vortex-bench-proxy and its IAM role. It had no client connections from 2026-05-31 to 2026-10-28, and it cost $21.60 a month.
• Log every Postgres connection and disconnection to CloudWatch Logs, with 30-day retention.
• Deactivate the access key of the IAM user bench-web-cli. The user has AdministratorAccess, and the key was last used on 2026-06-19.
• Remove the v3 leftovers: the security group benchmark-website-sg, which allowed SSH from the internet, and the Elastic IP 98.94.159.55.
• Narrow the trust policy of GitHubBenchmarkIngestRole to develop, as provision.sh declares. It also trusted the deleted branch ct/bench-v4.
• Delete the IAM role GithubVortexCiBenchmarkRole. It trusted every ref of vortex-data/vortex, and IAM, CloudTrail and a code search found no use of it.
• Turn on an account CloudTrail trail, GuardDuty in us-east-1 with RDS login monitoring, and IAM Access Analyzer.
• Merge Document bench backups, alarms and restores, and remove the RDS Proxy #121.
• Merge Log bench Postgres connections to CloudWatch #122. It records the logging settings in provision.sh and the README.
• Take a second pg_dump to an operator's computer on 2026-10-30, before the password rotation below.
• Rotate the password of the database role bench_read (2026-10-30). Vercel stores it as a sensitive variable, which nobody can read back, in both projects and both environments. The role got a SCRAM hash, so the plain password did not reach the server or its logs. A production deploy and /api/health confirmed the new password.
• Delete the IAM user bench-web-cli. Done on 2026-11-02: its inactive access key (last used on 2026-06-19) was deleted, AdministratorAccess was detached, and the user was deleted. It had no console password, inline policies or groups.
• Delete the snapshot vortex-bench-prod-pre-16-15-20261028. Done on 2026-11-02, with the two manual snapshots from 2026-06-16. The automated backups cover any point in the last 35 days, and the us-west-2 replica keeps 14 days.
• Add retries to the database connection in scripts/post-ingest.py in vortex-data/vortex. Four ingest jobs failed during the two restarts on 2026-10-28. Fixed in fix(bench): retry the post-ingest database write across RDS restarts vortex#11018: the script retries for about 4 minutes.
• Keep the us-west-2 backup replica (decided on 2026-11-02). Its estimated cost was $9 a month; the first days (2026-10-28 to 2026-11-01) cost $0.37, that is $0.16 of backup storage in us-west-2 and $0.21 of transfer from us-east-1. Confirm the real cost on the November bill.
• Buy a 1-year reserved instance for db.r7g.large (decided and bought on 2026-11-02): PostgreSQL, single-AZ, us-east-1, all upfront, $1,529, ID vortex-bench-prod-r7g-large-1y. On demand costs about $2,094 a year. The reservation is only a billing discount, so the instance did not change. The class stays db.r7g.large, because its 16 GiB of memory keeps bench's working set in cache.
• Decide on encryption at rest: not now (decided on 2026-11-02). RDS can encrypt only a new instance, so the change needs a snapshot copy, a restore and an instance swap. The database holds public benchmark data, so the swap is not worth its risk.

Repository security, 2026-10-30

• Turn off Dependabot security updates. Renovate already makes these updates, and Dependabot PRs could not pass the bun.lock snapshot check.
• Revoke an unused deployment-protection bypass secret on the vortex Vercel project.

🤖 Generated with Claude Code