diff --git a/README.md b/README.md index 910da43..daf1ab7 100644 --- a/README.md +++ b/README.md @@ -54,18 +54,13 @@ lest runs list - **Steps:** `run` a script, send an `http` request, `assert` conditions, drive a `browser`, call another `flow`, or group steps in order or in `parallel`. -- **Browsers:** declarative actions or a Playwright script, saved sign-in - sessions, screenshots at failure, and recorded videos with a visible cursor. -- **Services:** start a dev server or mock before the steps, wait until it - is ready, and stop it after. -- **Demos:** record a flow and get a cut video with waits removed, popups in - place, chapters from your beat labels, and a still per beat. The demo stays - a test, so it fails when the product drifts. - **Expressions:** one language, [CEL](https://github.com/google/cel-spec), for conditions, assertions and outputs, and `${{ expr }}` inside strings. Values reach scripts as environment variables, never as text spliced into shell code. - **Retries and polling:** `retry: {attempts, delay, until}` on any step. +- **Services:** start a dev server or mock before the steps, wait until it + is ready, and stop it after. - **Environments and inputs:** named variable sets and per-run choices. - **Secrets:** referenced by name, resolved from the OS keyring, environment variables or any password-manager command, and redacted from every output @@ -77,13 +72,136 @@ lest runs list `lest cleanup `. - **Reports:** a JSON report per run (`lest schema report` describes it), JUnit XML, and `lest bundle` evidence zips with checksums. -- **CI:** `lest affected --base origin/main` lists the flows a change touches; - webhook notifications (JSON or Slack-compatible) after each run. -- **Optional AI:** off by default. With a provider of your choice, failed - steps get a plain-words explanation next to their deterministic headline, - and Open in agent hands the failure to your own agent CLI. Nothing changes - in the layout either way. +An API flow from the examples signs in, adds a plant, polls until its sensor +pairs, and removes the plant again: + +```yaml + - id: add_plant + http: + method: POST + url: "${{ vars.base_url }}/api/plants" + headers: { Authorization: "Bearer ${{ steps.sign_in.outputs.token }}" } + json: { name: Basil, room: Kitchen, moisture: 55 } + expect: self.status == 201 + outputs: { id: self.body.id } + cleanup: + run: curl -fsS -X DELETE -H "Authorization: Bearer $TOKEN" "$base_url/api/plants/$ID" + env: + ID: "${{ self.outputs.id }}" + TOKEN: "${{ steps.sign_in.outputs.token }}" + + - id: sensor_online + http: + url: "${{ vars.base_url }}/api/plants/${{ steps.add_plant.outputs.id }}/status" + headers: { Authorization: "Bearer ${{ steps.sign_in.outputs.token }}" } + outputs: { sensor: self.body.sensor } + retry: { attempts: 6, delay: 300ms, until: self.outputs.sensor == 'online' } +``` + +![A passed run of the Plants API flow, with the polling step open and its cleanup listed](docs/images/flow-passed.png) + +## Failures that explain themselves + +A failed step leads with a headline that says what was expected and what +happened instead, taken from the step itself: the HTTP status and body, the +assertion's values, or for a browser step the page it was on and the text it +showed. Browser steps keep a screenshot at failure, and the CLI prints the +command that reruns from the failed step. + +```console +$ lest run wrong-password +▶ Wrong password (fails on purpose) (local) · run 20261007T144538Z-4uhm + starting service sprout + ✗ Sign in through the API 1ms HTTP 401 from POST http://127.0.0.1:4173/api/login: wrong email or password + ✗ Sign in on the login page 15.5s actions[3] expectText: expected h1 to contain "Good morning", but it never appeared on http://127.0.0.1:4173/login ("Sign in · Sprout"); the page shows "Wrong email or password" + rerun from the failure: lest run wrong-password --from page_login +failed wrong-password 15.7s 20261007T144538Z-4uhm +``` + +![The same failure in the UI, with the headline, actions and the screenshot at failure](docs/images/flow-failed.png) + +## Browsers + +Browser steps use Playwright: declarative actions for short checks, or a +script with helpers for a visible cursor, paced typing and holds. Sessions +save a sign-in once and reuse it in later flows. With Watch browser on, the +UI shows the page live while the step runs. + +![A recording in progress, with the live browser view showing a consent popup](docs/images/flow-running-live.png) + +## Demos + +A demo is a flow that records its browser step. Lest cuts the recording to +the moments the script marks, removes the waits, places popups over the page +that opened them, and writes chapters and a still for each beat. Because the +demo is still a flow, it fails when the product drifts instead of going +stale. + +```yaml +demo: + title: Water by the forecast + beats: + - { marker: plants, label: The dashboard shows every plant } + - { marker: consent, label: SkyCast asks for permission } + - { marker: connected, label: The forecast appears } +``` + +```js +await phase("consent", async () => { + const consent = await popup(() => ui.click("#connect")); + beat("consent"); + await ui.dwell(900); + await ui.click("#allow"); + await consent.waitForEvent("close"); +}); +``` + +The cut from the example, 11 seconds from a 14 second take: + +![The cut demo: the dashboard, the consent popup, and the forecast](docs/images/demo.gif) + +![The demo page, with the cut, its story beats and the latest recording](docs/images/demo.png) + +## The UI + +`lest ui` serves the UI on a loopback port for this project; `lest ui --app` +opens it in its own window. Everything the CLI does is there: browse and +filter flows, run them with an environment and inputs, follow a run live, +cancel it, and read past runs. The CLI and UI share the same reports. + +![The flows page, grouped by folder with each flow's last result](docs/images/flows.png) + +![Past runs, newest first, filterable by flow and status](docs/images/runs.png) + +Light and dark themes follow the system or a setting. + +![A failed run in the dark theme](docs/images/flow-failed-dark.png) + +## Optional AI + +AI is off by default, and the UI looks the same without it. With a provider +of your choice (an agent CLI you already use, or any command that reads a +prompt and prints text), a failed step gets a plain-words explanation in +place of its headline, with the headline kept underneath, and Open in agent +starts your agent CLI in the project with the failure's context. Each +failure is explained once and cached, under a daily limit. + +![A failed step explained by a model, with Open in agent in the step's menu](docs/images/flow-failed-ai.png) + +![AI settings: provider, command, agent command and daily limit](docs/images/settings-ai.png) + +From the CLI, `lest explain ` prints the same explanation (or the +headline when AI is off), and `lest context ` prints everything needed to +pick up a failure, ready to paste anywhere. + +## CI + +`lest run` exits non-zero on failure and can write JUnit XML. +`lest affected --base origin/main` lists the flows whose files, called flows, +scripts or `affects` globs a change touches, so a pipeline runs only those. +Webhook notifications (JSON or Slack-compatible) go out after each run, and +`lest bundle` zips a run's report and artifacts with checksums for evidence. ## Examples diff --git a/crates/lest/src/args.rs b/crates/lest/src/args.rs index 45ab24d..cfaed12 100644 --- a/crates/lest/src/args.rs +++ b/crates/lest/src/args.rs @@ -140,8 +140,12 @@ pub struct UiArgs { #[arg(long, default_value_t = 0)] pub port: u16, /// Print the address instead of opening a browser. - #[arg(long)] + #[arg(long, conflicts_with = "app")] pub no_browser: bool, + /// Open in its own window (a Chromium-family browser's app mode) instead + /// of a browser tab. + #[arg(long)] + pub app: bool, } #[derive(Debug, Args)] diff --git a/crates/lest/src/main.rs b/crates/lest/src/main.rs index 23c46c6..cdba827 100644 --- a/crates/lest/src/main.rs +++ b/crates/lest/src/main.rs @@ -966,6 +966,56 @@ fn cmd_config(ctx: &Ctx, command: ConfigCommand) -> Result { } } +/// Opens the UI as a standalone window: a Chromium-family browser in app +/// mode, with its own profile so it is a separate window and app. +fn open_app_window(url: &str, profile: &Path) -> bool { + let app = format!("--app={url}"); + let data = format!("--user-data-dir={}", profile.display()); + if cfg!(target_os = "macos") { + let home = std::env::var_os("HOME").map(PathBuf::from).unwrap_or_default(); + for name in ["Google Chrome", "Chromium", "Microsoft Edge", "Brave Browser"] { + let bundle = format!("{name}.app"); + if Path::new("/Applications").join(&bundle).is_dir() || home.join("Applications").join(&bundle).is_dir() { + return std::process::Command::new("open") + .args(["-na", name, "--args", &app, &data]) + .status() + .is_ok_and(|s| s.success()); + } + } + false + } else { + let Some(browser) = ["google-chrome", "chromium", "chromium-browser", "microsoft-edge", "brave-browser"] + .into_iter() + .find(|b| which::which(b).is_ok()) + else { + return false; + }; + let mut cmd = std::process::Command::new(browser); + cmd.args([&app, &data]).stdout(std::process::Stdio::null()).stderr(std::process::Stdio::null()); + // Its own process group, so Ctrl-C in this terminal does not close + // the window (or other projects' windows sharing the profile). + #[cfg(unix)] + { + use std::os::unix::process::CommandExt; + cmd.process_group(0); + } + let Ok(mut child) = cmd.spawn() else { return false }; + // A browser that cannot start (no display, a locked profile) exits + // at once with an error; one that hands the window to a running + // browser exits at once with success. + let deadline = std::time::Instant::now() + std::time::Duration::from_millis(1500); + while std::time::Instant::now() < deadline { + match child.try_wait() { + Ok(Some(status)) => return status.success(), + Ok(None) => std::thread::sleep(std::time::Duration::from_millis(50)), + Err(_) => return false, + } + } + std::thread::spawn(move || child.wait()); + true + } +} + async fn cmd_ui(ctx: &Ctx, a: UiArgs) -> Result { let roots = ctx.roots.clone(); let project = ctx.project.clone(); @@ -981,7 +1031,14 @@ async fn cmd_ui(ctx: &Ctx, a: UiArgs) -> Result { let url = server.url(); eprintln!("lest ui: {url}"); eprintln!("project: {}", ctx.project.root.display()); - if !a.no_browser { + if a.app && open_app_window(&url, &ctx.roots.data_dir.join("ui-window")) { + eprintln!("opened in its own window"); + } else if !a.no_browser { + if a.app { + eprintln!( + "could not open an app window (no Chromium-family browser, or it failed to start); opening a browser tab" + ); + } let opener = if cfg!(target_os = "macos") { "open" } else { "xdg-open" }; if std::process::Command::new(opener).arg(&url).status().is_err() { eprintln!("open the address above in a browser"); diff --git a/docs/design.md b/docs/design.md index a42a857..3bd99a0 100644 --- a/docs/design.md +++ b/docs/design.md @@ -14,8 +14,8 @@ each. Behavior is specified in the reference docs (start with 1. **A run is deterministic and reproducible.** Nothing a model says can change a run's outcome. AI sits around runs, never inside them. 2. **One run model.** A suite is a flow whose steps call other flows. A demo - is a flow that records a browser step. Every surface (CLI, desktop, web - UI, notifications) consumes one event stream and one report format. + is a flow that records a browser step. Every surface (CLI, UI, + notifications) consumes one event stream and one report format. 3. **The core owns everything; the shells are thin.** The CLI and the UI call the same library. No behavior exists only in one shell. 4. **Values are data, never source text.** Template values reach a shell @@ -37,7 +37,7 @@ crates/lest-core flow spec, expressions, validation, runner, reports, store, secrets, tool profiles, browser bridge, media, AI provider, notifications crates/lest-server local HTTP API + server-sent events + embedded UI -crates/lest the `lest` binary: CLI, `lest ui`, optional desktop shell +crates/lest the `lest` binary: CLI and `lest ui` ui/ React + TypeScript UI (served by lest-server) harness/ the Node browser harness the runner launches examples/ a self-contained sample app and flows exercising every feature @@ -46,28 +46,30 @@ examples/ a self-contained sample app and flows exercising every feat ### Server-first UI The UI is a web app served by `lest ui` on `127.0.0.1` and talks to the core -over HTTP and server-sent events. The desktop app is a Tauri window that -loads the same URL. +over HTTP and server-sent events. `lest ui --app` opens it as its own window +(a Chromium-family browser in app mode, with a separate profile). -Why not Tauri IPC: +Why not a desktop webview with IPC: - **One transport.** IPC commands plus hand-written TypeScript mirrors drift - from the Rust types. A small HTTP API with JSON types generated from the - Rust structs has one source of truth. + from the Rust types. One small HTTP API serves the UI, scripts and tests. - **The real app is testable.** Playwright can drive the actual UI in CI on Linux (WebKit webviews expose no automation protocol), so screenshots in docs and PRs come from the shipped UI, not from a separate fixture harness. - **Files are URLs.** Artifacts, screenshots and videos are served with range requests instead of base64 strings over IPC, so a 200 MB recording plays without loading into memory. -- **The desktop shell is optional.** `cargo install` without the `desktop` - feature still gives a full UI in the browser; Linux servers and CI need no - webview libraries. +- **No webview to build.** One binary with the UI embedded, on every + platform, with no webview libraries; CI builds and tests it on Linux. A + native shell can wrap the same URL later if a window manager integration + ever needs it; `--app` covers a standalone window today. The server binds loopback only, requires a per-launch random token (passed in -the URL once, then held in an HttpOnly SameSite=Strict cookie), checks the -`Host` header against loopback names to block DNS rebinding, and sets a strict -Content-Security-Policy. +the URL once, then held in an HttpOnly SameSite=Strict cookie named for the +port), checks the `Host` header against loopback names to block DNS +rebinding, accepts changes only from its own origin, serves run files in a +sandbox, and sets a strict Content-Security-Policy. Holding the token is +equivalent to running commands as the user ([ai.md](ai.md), Trust). ### Runner @@ -228,9 +230,10 @@ implemented the same way. ### State and paths Config in the user config dir (`lest/config.yml`), run data in the user state -dir (`lest/runs///report.json` plus `artifacts/`), with a retention -cap and `lest data purge`. Projects are directories with a `lest.yaml`; the UI -opens projects instead of falling back to a global tests directory. +dir (`lest/runs///report.json` plus `artifacts/`), removed with +`lest data prune --keep ` or `lest data purge`. Projects are directories +with a `lest.yaml`; the UI opens projects instead of falling back to a global +tests directory. ## Not built, on purpose diff --git a/docs/images/demo.gif b/docs/images/demo.gif new file mode 100644 index 0000000..fd42fc2 Binary files /dev/null and b/docs/images/demo.gif differ diff --git a/docs/images/demo.png b/docs/images/demo.png index 5b54139..8b75bca 100644 Binary files a/docs/images/demo.png and b/docs/images/demo.png differ diff --git a/docs/images/flow-failed-ai.png b/docs/images/flow-failed-ai.png index 684b7e7..a36970d 100644 Binary files a/docs/images/flow-failed-ai.png and b/docs/images/flow-failed-ai.png differ diff --git a/docs/images/flow-failed-dark.png b/docs/images/flow-failed-dark.png index 3173313..4fb23da 100644 Binary files a/docs/images/flow-failed-dark.png and b/docs/images/flow-failed-dark.png differ diff --git a/docs/images/flow-failed.png b/docs/images/flow-failed.png index 79a43b6..991408b 100644 Binary files a/docs/images/flow-failed.png and b/docs/images/flow-failed.png differ diff --git a/docs/images/flow-passed.png b/docs/images/flow-passed.png index f38a565..d4fa1f2 100644 Binary files a/docs/images/flow-passed.png and b/docs/images/flow-passed.png differ diff --git a/docs/images/flow-running-live.png b/docs/images/flow-running-live.png index a197678..f63b110 100644 Binary files a/docs/images/flow-running-live.png and b/docs/images/flow-running-live.png differ diff --git a/docs/images/flows.png b/docs/images/flows.png index 7d4c49b..b49a43d 100644 Binary files a/docs/images/flows.png and b/docs/images/flows.png differ diff --git a/docs/images/runs.png b/docs/images/runs.png index d2de884..0cfc3c9 100644 Binary files a/docs/images/runs.png and b/docs/images/runs.png differ diff --git a/docs/images/settings-ai.png b/docs/images/settings-ai.png index ac6227c..c4b4d31 100644 Binary files a/docs/images/settings-ai.png and b/docs/images/settings-ai.png differ diff --git a/docs/images/settings.png b/docs/images/settings.png index 1f9cba3..e432c18 100644 Binary files a/docs/images/settings.png and b/docs/images/settings.png differ diff --git a/docs/ui.md b/docs/ui.md index b4e9754..a3497ff 100644 --- a/docs/ui.md +++ b/docs/ui.md @@ -4,6 +4,12 @@ same flows, runs the same runner and shows the same reports as the CLI. The source is in `ui/` (see [ui/README.md](../ui/README.md) for development). +`lest ui` opens the address in your default browser; `--no-browser` only +prints it. `lest ui --app` opens it in its own window instead: a +Chromium-family browser (Chrome, Chromium, Edge or Brave) in app mode, with a +profile of its own under the data directory. Without one, it falls back to a +browser tab. + The left rail holds Flows, Runs, Demos and Settings, and shows the project name. It collapses to icons; the choice is remembered in the browser. diff --git a/ui/scripts/screenshots.mjs b/ui/scripts/screenshots.mjs index 9a7bdc2..4f99597 100644 --- a/ui/scripts/screenshots.mjs +++ b/ui/scripts/screenshots.mjs @@ -167,6 +167,12 @@ async function capture() { await apage.click('[aria-label="AI provider"] [role="radio"]:has-text("Custom command")'); await apage.fill("#ai-command", aiCommand); await apage.waitForSelector('.settings-status:has-text("Using")'); + // A custom command has no agent of its own; Open in agent needs one. + await apage.fill("#ai-agent", "claude {prompt}"); + for (let i = 0; (await api("/settings")).ai.agent !== "claude {prompt}"; i++) { + if (i > 50) throw new Error("the agent command was not saved"); + await apage.waitForTimeout(200); + } await apage.waitForSelector('.settings-status .saved:has-text("Saved")'); await shot(apage, "settings-ai.png"); diff --git a/ui/src/components/RunView.tsx b/ui/src/components/RunView.tsx index 7dab63c..459680b 100644 --- a/ui/src/components/RunView.tsx +++ b/ui/src/components/RunView.tsx @@ -583,7 +583,7 @@ function FailedCard({ s, ctx }: { s: StepState; ctx: Ctx }) { const stored = !ctx.live && !!run.report; const explainState = useExplanation(ai?.status ?? null, stored ? run.runId : null, s.id); const slot = explainSlot(headline, explainState); - const handoff = useHandoff(ai?.status ?? null, stored ? run.runId : null, s.id); + const handoff = useHandoff(ai?.status ?? null, ai?.ai.agent ?? "", stored ? run.runId : null, s.id); const [launch, setLaunch] = useState<{ error: string; command: string | null } | null>(null); const openAgent = async () => { try { @@ -702,7 +702,7 @@ function useExplanation(status: AiStatus | null, runId: string | null, stepId: s } /** The agent handoff for a finished run, fetched when AI may be on. */ -function useHandoff(status: AiStatus | null, runId: string | null, stepId: string): Handoff | null { +function useHandoff(status: AiStatus | null, agent: string, runId: string | null, stepId: string): Handoff | null { const [handoff, setHandoff] = useState(null); const resolved = status?.resolved?.label ?? ""; useEffect(() => { @@ -711,6 +711,8 @@ function useHandoff(status: AiStatus | null, runId: string | null, stepId: strin return; } let alive = true; + // The previous command must not be copied while the new one loads. + setHandoff(null); api.handoff(runId, stepId).then( (h) => alive && setHandoff(h), () => alive && setHandoff(null), @@ -718,7 +720,7 @@ function useHandoff(status: AiStatus | null, runId: string | null, stepId: strin return () => { alive = false; }; - }, [status?.configured, resolved, runId, stepId]); + }, [status?.configured, resolved, agent, runId, stepId]); return handoff; } diff --git a/ui/src/lib/ai.ts b/ui/src/lib/ai.ts index 5a63e08..4c363a7 100644 --- a/ui/src/lib/ai.ts +++ b/ui/src/lib/ai.ts @@ -43,8 +43,7 @@ export function useAiSettings(): SettingsResponse | null { return s; } -/** Saves the AI settings. The agent command has no field in the UI, so the - * one in the settings file is kept. */ +/** Saves the AI settings and shares the result with every view. */ export async function saveAi(ai: AiConfig): Promise { const s = await api.saveAi(ai); settings = s; diff --git a/ui/src/pages/Settings.tsx b/ui/src/pages/Settings.tsx index 080ba6f..c2211f6 100644 --- a/ui/src/pages/Settings.tsx +++ b/ui/src/pages/Settings.tsx @@ -99,10 +99,12 @@ function AiForm({ settings }: { settings: SettingsResponse }) { const [saved, setSaved] = useState<"" | "saving" | "saved">(""); const [saveError, setSaveError] = useState(null); const timer = useRef | undefined>(undefined); + const pending = useRef(null); const latest = useRef(ai); latest.current = ai; const save = async (next: AiConfig) => { + pending.current = null; setSaved("saving"); try { await saveAi(clean(next)); @@ -117,10 +119,19 @@ function AiForm({ settings }: { settings: SettingsResponse }) { const next = { ...latest.current, ...patch }; setAi(next); clearTimeout(timer.current); - if (debounce) timer.current = setTimeout(() => void save(next), TEXT_SAVE_DELAY_MS); - else void save(next); + if (debounce) { + pending.current = next; + timer.current = setTimeout(() => void save(next), TEXT_SAVE_DELAY_MS); + } else void save(next); }; - useEffect(() => () => clearTimeout(timer.current), []); + // Navigating to another page saves an edit still waiting for its pause. + useEffect( + () => () => { + clearTimeout(timer.current); + if (pending.current) void saveAi(clean(pending.current)).catch(() => {}); + }, + [], + ); useEffect(() => { if (saved !== "saved") return; const t = setTimeout(() => setSaved(""), 2000);