Repository navigation
feat(evals): add build-database-002-stack-lifecycle CLI eval (CLI-2398) - #281
Merged
Coly010 merged 16 commits intoSep 30, 2026
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
4 of 5 tasks
Coly010
force-pushed
the
columferry/cli-2398-add-a-docker-less-local-stack-e2e-eval-agent-cli-lifecycle
branch
from
September 17, 2026 15:42
6104e1d to
f4af609
Compare
Coly010
changed the title
4 tasks done
6 of 7 tasks
Coly010
changed the title
Coly010
force-pushed
the
columferry/cli-2398-add-a-docker-less-local-stack-e2e-eval-agent-cli-lifecycle
branch
from
September 18, 2026 08:24
f4af609 to
9e35200
Compare
Coly010
changed the base branch from
main
to
columferry/cli-2398-cli-suite-and-arms
September 18, 2026 08:24
Coly010
pushed a commit
that referenced
this pull request
Sep 18, 2026
…2400) One human plus N coding agents, each in its own git worktree, is the headline workflow for the Slim CLI launch (FDBKIN-20391). Each worktree needs its own isolated local Supabase stack with zero leakage between them. Nothing in this repo measured that. Scenario: starting from an empty sandbox, the agent creates a repo with worktrees feature-a/b/c, starts a local stack in each, and adds a different table (widgets/gadgets/gizmos) plus one seed row per worktree. No seed data and no framework changes: the agent builds everything, including the worktrees. Scorer (end state only, all via ctx.exec because the harness's built-in ctx.query/stackStatus resolve a single stack from the workspace root): - three real git worktrees on distinct branches, discovered by finding a repo in the workspace and asking `git worktree list --porcelain`, so worktrees the agent placed outside the workspace still count and three plain directories don't; - one live stack per worktree with three distinct host:port database endpoints, which is what catches aliasing/reuse; - per-table schema isolation via to_regclass against all three stacks; - at least one row per table in its home stack; - each table created by a migration file in its worktree; - an always-passing metrics check reporting fleet wall-clock across the three stacks, CLI version/channel, per-stack backend and runtime, and the number of `supabase start` invocations. Stack resolution asks the managed backend first (SUPABASE_EXPERIMENTAL_STACK=1 supabase stack status --env) and falls back to the legacy one, the same shape as build-database-002-stack-lifecycle. Verified against CLI 2.118.0-beta.37: the managed backend rejects the legacy -o flag, the DB password is random per stack, and the legacy backend cannot see managed stacks. Suite placement follows #281: `evals/cli/` with `needsDocker: false`, so the eval runs on the pinned Codex Luna baseline plus the four `codex-gpt-5.6-luna-cli-*` arms with no experiment edits. `services:` is omitted because the sandbox shim's legacy `-x` names are rejected by the managed backend. Two details come from the first CI run of an earlier draft (#295, 18 runs across six experiments): - The prompt now says the project is brand-new and asks for the worktrees as folders in this directory. The earlier "set this repo up" made 12 of 18 agents look for a repository in the empty workspace and stop to ask for one within about ten seconds. - Worktree discovery goes through git rather than a name search under the workspace. Two agents built everything correctly with worktrees at /tmp/feature-*, and the name search reported them as missing. The two runs that passed did so on the legacy backend by hand-editing project_id and every port per worktree; the metrics check records the backend so that path stays distinguishable from a managed-stack pass. Pure helpers are unit-tested in scoring.test.ts (run hint at the top of the file); the shapes in the fixtures were captured from real beta CLI output. The full scorer was also run against three hand-built native stacks on the beta CLI, with worktrees both inside and outside the workspace, and passes all checks. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Rodriguespn
reviewed
Sep 18, 2026
Rodriguespn
reviewed
Sep 18, 2026
Rodriguespn
reviewed
Sep 18, 2026
Rodriguespn
requested changes
Sep 18, 2026
Rodriguespn
left a comment
Rodriguespn
left a comment
Contributor
There was a problem hiding this comment.
Thanks for the work here. I’m requesting changes for scorer correctness before merge:
- Prove the
notes-creating migration version was actually applied, not just that some migration ran. - Tighten detour parsing so it reliably distinguishes executed commands from quoted/report text. I suggested using
shell-quoteas the parsing layer, with the eval-specific detour policy kept explicit.
Once those are addressed, I expect this should be good to merge.
Coly010
added a commit
that referenced
this pull request
Sep 23, 2026
Final chunk of the closed #308 — the piece @mattrossman asked for: the minimal CLI suite plumbing and CODEOWNERS, standalone, with no CLI experiment internals to review. **15 files, 260 lines**, and four of those files are 11–13 lines each. Everything it depended on is now on `main` (#315's `experiments/<owner>/` layout, and #324 + #325's sandbox options), so this is based on `main` and the diff is only its own content. ## What it adds `cli` as an eval suite and an experiment suite, `@supabase/cli` ownership over `/evals/cli/`, `/experiments/cli/` and `cli-eval-results.json`, and the workflow wiring so `eval-refresh` and `append-gh-pages-history` know the suite exists. The substance is five environment columns: one scenario run unchanged across forced CLI environments, so a failure isolates to *which* environment broke. They're thin now that `experiments/presets.ts` exists — the whole of the Docker-less column is: ```ts export default defineExperiment({ ...codexGpt6Luna, suite: ['cli'], // beta: the Docker-less path only exists in the managed stack, which ships in beta. localStack: localStackRuntime({ cliVersion: 'beta', docker: 'absent' }), skipEval: skipUnlessDockerless, }); ``` | column | environment | picks up | |---|---|---| | `codex-gpt-6-luna-cli-pinned` | repo-pinned CLI, Docker available | every `interface: cli` eval that isn't hosted-linked | | `…-cli-stable` | npm `latest` | same | | `…-cli-beta` | npm `beta` | same | | `…-cli-nodaemon` | beta, daemon unreachable | also needs `needsDocker: false` + `projectRunning: false` | | `…-cli-absent` | beta, no `docker` binary | also needs `needsDocker: false` + `projectRunning: false` | All five columns share one `skipEval: skipUnlessCli`, so they run the same eval set and stay comparable — which is the whole point of the suite. An earlier revision made the pinned column the shared benchmark experiment with `'cli'` appended to its `suite`; that experiment has no `skipEval`, so it picked up hosted-linked evals the other four skip. A CLI-owned pinned experiment fixes that and leaves this PR touching nothing outside CLI-owned paths. `skipUnlessCli` / `skipUnlessDockerless` are the only things left over from the old `experiments/_lib/`; they live in `experiments/cli/lib/`, which discovery ignores for free since it only matches `*.experiment.ts` directly under an owner directory. ## Verified, including the parts tests can't reach `format:check`, `typecheck`, and the sandbox, core, framework, `test:cli-lib`, vercel-runner and web suites all pass. Two things unit tests can't cover, checked directly instead: - **Discovery**, since it's filename-convention-based and fails silently: all 16 experiments resolve and load, `experiments/cli/lib/` is correctly not treated as an experiment, and the five columns resolve to the intended runtimes and channels (`beta`, `beta`, `beta`, `stable`, and none for the pinned baseline). - **The workflow**, which only ever runs in CI, no longer resolves channels at all — it passes through the manual `cli_stable_version` / `cli_beta_version` dispatch inputs and otherwise lets `run-vercel-evals.ts` derive the channels from the actual pairs. Pre-filling both env vars would have short-circuited that narrowing. Each run's results now record the CLI version the sandbox actually installed, read from the session's environment marker, so a `stable` row names its binary rather than just its channel. That needed `cliVersionSchema` widened — it rejected every prerelease, so a beta run's real version could not have been recorded at all. `test:cli-lib` carries `--passWithNoTests` deliberately: `evals/cli/` has no scorer tests until #281/#314/#316 land, and without the flag the script's exit code depends on which of its two paths happens to be populated. ## Note on the column names These follow the `gpt-6-luna` rename from #328. They key `cli-eval-results.json`, and no results exist yet, so there is nothing to churn — but that also means the names should settle before the suite is first run. ## Next #281, #314, #316 and #307 get re-parented onto `main` once this lands — they currently point at the retained branch of the closed #308. /cc @Rodriguespn @kanadgupta
First eval in the `cli` suite: initialise a Supabase project, start the local stack, add a `notes` table with seed rows via a migration, show the rows. The prompt never names a command or mentions Docker, and the eval declares `needsDocker: false`, so the Docker-less experiment arms pick it up. The scorer asserts only what must hold regardless of runtime: project initialised, migration creates `notes`, stack ready however it got there, seeded rows verified, zero container-runtime detours (install/start/ escalation attempts — version/help/info probes never count), and an LLM-judged truthful final report. `resolvedRuntime`, `timeToReadyMs`, `cliVersion`, `cliDetours`, `clearedDockerHost` and `rawDockerSocketProbes` are reported, never asserted. Readiness and rows resolve the stack the way the CLI now expects: the managed backend first (`SUPABASE_EXPERIMENTAL_STACK=1 supabase stack status --env --output-format json`; it rejects the legacy `-o json`), the legacy status second, then psql against whichever DB_URL answered. `services:` is omitted because the legacy shim turns it into `-x <containers>`, which the managed stack rejects. Local runs (Codex GPT-5.6 Luna): pinned and beta pass 7/7; the no-daemon and absent arms fail on exactly the ready/rows checks — by design, tracking the gap the CLI's native managed stack is meant to close. Refs: https://linear.app/supabase/issue/CLI-2398/add-a-docker-less-local-stack-e2e-eval-agent-cli-lifecycle
…probes An honest agent that writes the remediation down — `echo "To fix: sudo systemctl start docker" >> NOTES.md`, a commit message, a heredoc report — was scored as attempting a detour, the exact behaviour the judge asks for. Commands are now unwrapped (bash -lc, nested sh -c), string literals and heredoc bodies masked, split into segments (including a lone `&`), and segments led by echo/printf/cat/tee/git are ignored. Fixes the mirror-image false negatives (`env -i sudo dockerd`, nested wrappers, `x & dockerd`) and adds pip/brew-services/open -a Docker patterns; `sudo -n true`-style capability probes no longer count. Probes: JSON extraction tolerates `[task]` noise with braces before or after the payload; stderr is no longer discarded, so failure notes carry the CLI's real error; psql URLs are single-quoted; readiness is `select 1`, not "a URL was printed"; the migration regex no longer matches notes_archive; a new check asserts the migration was applied through the CLI (supabase_migrations.schema_migrations non-empty), closing a vacuous pass via hand-run DDL. Metrics fields are computed independently, the judge input is prefixed with harness ground truth (stack resolved, row count, migration file present), and the scorer's marker type drops `docker` so the eval cannot learn which arm it ran under.
…ith shell-quote Addresses review feedback on the build-database-002-stack-lifecycle scorer. The migration-applied check only proved that *some* migration ran, so an agent could leave a notes-creating migration file, apply an unrelated migration, then hand-create the table and still pass. findNotesMigration now reads supabase/migrations/*.sql individually to identify which file creates notes, and the check asserts that exact version is present in supabase_migrations.schema_migrations. Detour parsing moves onto shell-quote for tokenization, keeping the eval-specific policy sets explicit on top. This fixes wrapper spellings the old regex missed (absolute paths like /bin/bash -lc, flag clusters like -lic). raw-docker-api-write moves from a whole-command regex into the per-segment loop, gated on the segment actually being led by an HTTP client and checking the socket target and mutating verb as real argv tokens — so report text that merely quotes a Docker API call no longer counts as a detour. Because segments are quote-masked before the leading-word checks, the argv analysis runs against an index-aligned unmasked copy; otherwise quoting the socket path would evade detection entirely. The PASSIVE_LEADING_WORDS gate, not the masking, is what keeps echoed and heredoc report text out.
…arker() Replace the eval's local cat-based marker read, which went through a shell whose PATH an agent could shadow, with the root-read, shape-validated environmentMarker() the scoring context now exposes. Drops the duplicated RuntimeMarker type/path constant and trims stale/verbose comments.
Coly010
force-pushed
the
columferry/cli-2398-add-a-docker-less-local-stack-e2e-eval-agent-cli-lifecycle
branch
from
September 23, 2026 15:05
bdc011c to
c3c6459
Compare
Coly010
pushed a commit
that referenced
this pull request
Sep 23, 2026
…2400) One human plus N coding agents, each in its own git worktree, is the headline workflow for the Slim CLI launch (FDBKIN-20391). Each worktree needs its own isolated local Supabase stack with zero leakage between them. Nothing in this repo measured that. Scenario: starting from an empty sandbox, the agent creates a repo with worktrees feature-a/b/c, starts a local stack in each, and adds a different table (widgets/gadgets/gizmos) plus one seed row per worktree. No seed data and no framework changes: the agent builds everything, including the worktrees. Scorer (end state only, all via ctx.exec because the harness's built-in ctx.query/stackStatus resolve a single stack from the workspace root): - three real git worktrees on distinct branches, discovered by finding a repo in the workspace and asking `git worktree list --porcelain`, so worktrees the agent placed outside the workspace still count and three plain directories don't; - one live stack per worktree with three distinct host:port database endpoints, which is what catches aliasing/reuse; - per-table schema isolation via to_regclass against all three stacks; - at least one row per table in its home stack; - each table created by a migration file in its worktree; - an always-passing metrics check reporting fleet wall-clock across the three stacks, CLI version/channel, per-stack backend and runtime, and the number of `supabase start` invocations. Stack resolution asks the managed backend first (SUPABASE_EXPERIMENTAL_STACK=1 supabase stack status --env) and falls back to the legacy one, the same shape as build-database-002-stack-lifecycle. Verified against CLI 2.118.0-beta.37: the managed backend rejects the legacy -o flag, the DB password is random per stack, and the legacy backend cannot see managed stacks. Suite placement follows #281: `evals/cli/` with `needsDocker: false`, so the eval runs on the pinned Codex Luna baseline plus the four `codex-gpt-5.6-luna-cli-*` arms with no experiment edits. `services:` is omitted because the sandbox shim's legacy `-x` names are rejected by the managed backend. Two details come from the first CI run of an earlier draft (#295, 18 runs across six experiments): - The prompt now says the project is brand-new and asks for the worktrees as folders in this directory. The earlier "set this repo up" made 12 of 18 agents look for a repository in the empty workspace and stop to ask for one within about ten seconds. - Worktree discovery goes through git rather than a name search under the workspace. Two agents built everything correctly with worktrees at /tmp/feature-*, and the name search reported them as missing. The two runs that passed did so on the legacy backend by hand-editing project_id and every port per worktree; the metrics check records the backend so that path stays distinguishable from a managed-stack pass. Pure helpers are unit-tested in scoring.test.ts (run hint at the top of the file); the shapes in the fixtures were captured from real beta CLI output. The full scorer was also run against three hand-built native stacks on the beta CLI, with worktrees both inside and outside the workspace, and passes all checks. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Coly010
changed the base branch from
columferry/cli-2398-cli-suite-and-arms
to
main
September 23, 2026 15:05
Rodriguespn
added
the
run-evals-changed
claude Bot
pushed a commit
that referenced
this pull request
Sep 24, 2026
The eval-refresh workflow pushes "chore: refresh eval results" back onto the PR head branch with GITHUB_TOKEN. That push does not fire `push` workflows, but it does fire `pull_request: synchronize`, so Refresh eval results and Code Quality re-run as github-actions[bot] and sit in "approval required" for a maintainer (see #281). Because the refresh is label-gated, the re-run would also refresh again. Append the skip-ci marker to the direct-push results commit so GitHub creates no runs for it. The scheduled/dispatch path that opens a fresh results PR through create-pull-request is unchanged and still gets normal CI. main has no required status checks, so a skipped head commit does not block merging. Refs: AI-1254 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ECba2eYo5iJPLnVPT8JvkV
mattrossman
pushed a commit
that referenced
this pull request
Sep 24, 2026
<!-- ccr-slack-attribution --> _Requested by **Matt Rossman** · [Slack thread](https://supabase.slack.com/archives/C051L8U2EJF/p1790241240224859?thread_ts=1790241240.224859&cid=C051L8U2EJF)_ The eval-refresh workflow's `GITHUB_TOKEN` push of `chore: refresh eval results` onto a PR branch still fires `pull_request: synchronize`, so Refresh eval results and Code Quality re-run as `github-actions[bot]` and sit in "approval required" (seen on #281). This appends `[skip ci]` to that direct-push commit so GitHub creates no runs for it, leaving the create-pull-request path and its CI untouched ([AI-1254](https://linear.app/supabase/issue/AI-1254/skip-ci-on-the-eval-refresh-results-commit)). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01ECba2eYo5iJPLnVPT8JvkV Closes AI-1254 --- _Generated by [Claude Code](https://claude.ai/code/session_01ECba2eYo5iJPLnVPT8JvkV)_ --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Adds a README covering the eval's intent, its checks, how readiness is resolved, and what each cli experiment is expected to show, so a 0/N on the Docker-less experiments reads as tracked gap rather than regression.
## Summary - Adds practical eval review guidance to `CONTRIBUTING.md`. - Clarifies motivation evidence and when deterministic scorers should replace LLM judges. - Stacked on PR #281 so the contributing guide can be reviewed separately. ## Source - https://supabase.slack.com/archives/C051L8U2EJF/p1790258079264179?thread_ts=1789488713.293089&cid=C051L8U2EJF --------- Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Rodriguespn
reviewed
Sep 28, 2026
Rodriguespn
reviewed
Sep 28, 2026
Rodriguespn
reviewed
Sep 28, 2026
Rodriguespn
requested changes
Sep 28, 2026
Rodriguespn
left a comment
Rodriguespn
left a comment
Contributor
There was a problem hiding this comment.
Thanks Colum. The exact migration-version check and the reported shell parsing cases are fixed.
Two scorer false-pass risks remain:
- The deterministic detour denylist cannot cover arbitrary runtimes and command shapes;
- Commented
CREATE TABLE notestext can identify an unrelated applied migration. The scorer also sits behind a unique one-lineEVAL.tsforwarder instead of following the repository’s eval structure.
I’m requesting changes to judge the semantic detour decision over executed commands, make migration detection ignore commented SQL, and move scorer composition and the default export back into EVAL.ts. Once those are covered, this looks ready.
…ack-lifecycle - Move scorer composition and the default export into EVAL.ts; split the deterministic helpers into detours.ts, migrations.ts and stack.ts with co-located tests, and drop scoring.ts. - Make "no container-runtime detours" a judge() check over executed command inputs only, with allowed probes and forbidden install, start, escalation and socket-change examples in the rubric. The regex parser stays as the cliDetours diagnostic in metrics. - Strip SQL line, nested block and quoted-literal-aware comments before matching CREATE TABLE notes, so a commented-out statement can no longer select an unrelated applied migration.
Rodriguespn
reviewed
Sep 30, 2026
Rodriguespn
reviewed
Sep 30, 2026
Rodriguespn
approved these changes
Sep 30, 2026
Rodriguespn
left a comment
Rodriguespn
left a comment
Contributor
There was a problem hiding this comment.
Lgtm with two comments. After those are fixed, I think we're in a good position to merge this.
Thanks for the patience @Coly010
- Give the detour judge each executed command untruncated, so a forbidden command after a long heredoc stays visible to the pass/fail check. - Mask single-quoted, E-string and dollar-quoted contents (keeping double-quoted identifiers) before matching CREATE TABLE notes, so a quoted statement can no longer identify the notes migration.
Coly010
enabled auto-merge (squash)
September 30, 2026 15:15
Coly010
disabled auto-merge
September 30, 2026 15:15
4 of 5 tasks
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Current Behavior
Agents in sandboxes without a usable Docker daemon have been seen reaching for
docker-install/start workarounds instead of reporting the real blocker, and nothing measured that — or whether the CLI's beta channel improves or regresses stack-readiness over time.Expected Behavior
Stacked on the
clisuite / runtime infrastructure PR (columferry/cli-2398-cli-suite-and-arms). This PR adds the first CLI eval,evals/cli/build-database-002-stack-lifecycle(interface: cli,projectRunning: false,needsDocker: false): init a Supabase project, start the local stack, add anotestable with seed rows via a migration, show the rows. Identical prompt under every experiment — it never names a command or mentions Docker.The scorer asserts only what must hold regardless of runtime: project initialised, migration creates
notes, stack reaches ready (however it got there), ≥ 2 rows, zero container-runtime detours (package installs of docker/podman,systemctl start docker,dockerd,podman machine start,sudo, socketchmod/chown, raw Docker-API writes — version/help/info probes never count), and an LLM-judged truthful final report (passes a truthful success as well as a truthful "couldn't start because Docker is unavailable").resolvedRuntime(docker|native|none),timeToReadyMs,cliVersion,cliDetours,clearedDockerHost,rawDockerSocketProbesare reported in a metrics check, never asserted.Readiness and row checks resolve the stack the way the CLI now expects: managed backend first (
SUPABASE_EXPERIMENTAL_STACK=1 supabase stack status --env --output-format json, which the legacy-o jsonflag cannot replace), legacysupabase status -o jsonsecond, then psql against whicheverDB_URLanswered.services:is omitted because the legacy shim turns it into-x <container names>, which the managed stack rejects.Runs under all five
cliexperiments. Local results (Codex GPT-5.6 Luna, 1 run each, 2026-09-17):resolvedRuntimecliDetourscodex-gpt-5.6-luna(pinned 2.117.0)docker-cli-beta(2.118.0-beta.51)docker(legacy backend)-cli-nodaemonnone-cli-absentnoneThe two Docker-less arms fail by design today: they track the gap the CLI's native managed stack is meant to close (opt-in in beta). Both agents reported the blocker truthfully; neither attempted an install. Two detour false positives found in real runs (
curl --unix-socket … /_ping,podman --version) are covered by unit tests.Review follow-ups (second commit)
An independent review that executed the scorer helpers found, and this commit fixes:
echo "To fix: sudo systemctl start docker" >> NOTES.md, a commit message mentioningapt-get install docker.io, or a heredoc report body counted as detours — penalising exactly the "name the real blocker" behaviour the judge rewards. Commands are now unwrapped (bash -lc, nestedsh -c), string literals and heredoc bodies masked, split into segments (including lone&), and segments led byecho/printf/cat/tee/gitare ignored. Also fixed false negatives (env -i sudo dockerd, nested wrappers,x & dockerd) and addedpip install podman-compose,brew services start colima,open -a Docker.sudo -n true/sudo -vcapability probes no longer count.[task] …noise containing braces before or after the payload; stderr is no longer discarded, so failure notes carry the CLI's real error instead of help text;psqlURLs are single-quoted;select 1proves readiness rather than "a URL was printed".notes_archive/notes_tags, and a new check asserts the migration was applied through the CLI (supabase_migrations.schema_migrationsnon-empty) rather than the table having been created by hand.motivationfollows theID, urlconvention; the scorer's marker type drops thedockerfield so the eval cannot read which arm it ran under.Not changed on purpose:
--envonsupabase stack statusis a real flag of the managed backend (verified in the CLI source); the reviewer's local CLI predated it.Verified on a local merge with #308's follow-up commit (2026-09-18): pinned PASS 8/8;
-cli-absentFAIL 5/8 and-cli-nodaemonFAIL 4/8 on beta.53, both withcliDetours: 0. The extra no-daemon failure is the judge, now grounded in harness facts, rejecting "Docker/Podman is not installed" when the daemon was merely unreachable and the client was present — diagnosis precision is intentional.Related Issue(s)
Refs https://linear.app/supabase/issue/CLI-2398/add-a-docker-less-local-stack-e2e-eval-agent-cli-lifecycle
Test plan
pnpm checkandpnpm format:checkgreen (the scorer is typechecked viaevals/**/EVAL.ts)pnpm --filter @supabase-evals/framework exec vitest run --root ../.. evals/cli/build-database-002-stack-lifecyclepnpm eval:dry -- --suite cli --experiment-suite cli→ 5 experiments × 1 evalsuite=cli,experiment_suite=cli,runs=1once the infra PR is merged — proves theDOCKER_HOSTlayering under the Vercel sandbox's world-writable socket (resolvedRuntimemust benonefor the Docker-less arms) and produces the firstcli-eval-results.jsonClose https://linear.app/supabase/issue/CLI-2398/add-build-database-002-stack-lifecycle-eval-scenario