Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
144 changes: 131 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,18 +54,13 @@ lest runs list
- **Steps:** `run` a script, send an `http` request, `assert` conditions,
drive a `browser`, call another `flow`, or group steps in order or in
`parallel`.
- **Browsers:** declarative actions or a Playwright script, saved sign-in
sessions, screenshots at failure, and recorded videos with a visible cursor.
- **Services:** start a dev server or mock before the steps, wait until it
is ready, and stop it after.
- **Demos:** record a flow and get a cut video with waits removed, popups in
place, chapters from your beat labels, and a still per beat. The demo stays
a test, so it fails when the product drifts.
- **Expressions:** one language, [CEL](https://github.com/google/cel-spec),
for conditions, assertions and outputs, and `${{ expr }}` inside strings.
Values reach scripts as environment variables, never as text spliced into
shell code.
- **Retries and polling:** `retry: {attempts, delay, until}` on any step.
- **Services:** start a dev server or mock before the steps, wait until it
is ready, and stop it after.
- **Environments and inputs:** named variable sets and per-run choices.
- **Secrets:** referenced by name, resolved from the OS keyring, environment
variables or any password-manager command, and redacted from every output
Expand All @@ -77,13 +72,136 @@ lest runs list
`lest cleanup <run>`.
- **Reports:** a JSON report per run (`lest schema report` describes it),
JUnit XML, and `lest bundle` evidence zips with checksums.
- **CI:** `lest affected --base origin/main` lists the flows a change touches;
webhook notifications (JSON or Slack-compatible) after each run.

- **Optional AI:** off by default. With a provider of your choice, failed
steps get a plain-words explanation next to their deterministic headline,
and Open in agent hands the failure to your own agent CLI. Nothing changes
in the layout either way.
An API flow from the examples signs in, adds a plant, polls until its sensor
pairs, and removes the plant again:

```yaml
- id: add_plant
http:
method: POST
url: "${{ vars.base_url }}/api/plants"
headers: { Authorization: "Bearer ${{ steps.sign_in.outputs.token }}" }
json: { name: Basil, room: Kitchen, moisture: 55 }
expect: self.status == 201
outputs: { id: self.body.id }
cleanup:
run: curl -fsS -X DELETE -H "Authorization: Bearer $TOKEN" "$base_url/api/plants/$ID"
env:
ID: "${{ self.outputs.id }}"
TOKEN: "${{ steps.sign_in.outputs.token }}"

- id: sensor_online
http:
url: "${{ vars.base_url }}/api/plants/${{ steps.add_plant.outputs.id }}/status"
headers: { Authorization: "Bearer ${{ steps.sign_in.outputs.token }}" }
outputs: { sensor: self.body.sensor }
retry: { attempts: 6, delay: 300ms, until: self.outputs.sensor == 'online' }
```

![A passed run of the Plants API flow, with the polling step open and its cleanup listed](docs/images/flow-passed.png)

## Failures that explain themselves

A failed step leads with a headline that says what was expected and what
happened instead, taken from the step itself: the HTTP status and body, the
assertion's values, or for a browser step the page it was on and the text it
showed. Browser steps keep a screenshot at failure, and the CLI prints the
command that reruns from the failed step.

```console
$ lest run wrong-password
▶ Wrong password (fails on purpose) (local) · run 20261007T144538Z-4uhm
starting service sprout
✗ Sign in through the API 1ms HTTP 401 from POST http://127.0.0.1:4173/api/login: wrong email or password
✗ Sign in on the login page 15.5s actions[3] expectText: expected h1 to contain "Good morning", but it never appeared on http://127.0.0.1:4173/login ("Sign in · Sprout"); the page shows "Wrong email or password"
rerun from the failure: lest run wrong-password --from page_login
failed wrong-password 15.7s 20261007T144538Z-4uhm
```

![The same failure in the UI, with the headline, actions and the screenshot at failure](docs/images/flow-failed.png)

## Browsers

Browser steps use Playwright: declarative actions for short checks, or a
script with helpers for a visible cursor, paced typing and holds. Sessions
save a sign-in once and reuse it in later flows. With Watch browser on, the
UI shows the page live while the step runs.

![A recording in progress, with the live browser view showing a consent popup](docs/images/flow-running-live.png)

## Demos

A demo is a flow that records its browser step. Lest cuts the recording to
the moments the script marks, removes the waits, places popups over the page
that opened them, and writes chapters and a still for each beat. Because the
demo is still a flow, it fails when the product drifts instead of going
stale.

```yaml
demo:
title: Water by the forecast
beats:
- { marker: plants, label: The dashboard shows every plant }
- { marker: consent, label: SkyCast asks for permission }
- { marker: connected, label: The forecast appears }
```

```js
await phase("consent", async () => {
const consent = await popup(() => ui.click("#connect"));
beat("consent");
await ui.dwell(900);
await ui.click("#allow");
await consent.waitForEvent("close");
});
```

The cut from the example, 11 seconds from a 14 second take:

![The cut demo: the dashboard, the consent popup, and the forecast](docs/images/demo.gif)

![The demo page, with the cut, its story beats and the latest recording](docs/images/demo.png)

## The UI

`lest ui` serves the UI on a loopback port for this project; `lest ui --app`
opens it in its own window. Everything the CLI does is there: browse and
filter flows, run them with an environment and inputs, follow a run live,
cancel it, and read past runs. The CLI and UI share the same reports.

![The flows page, grouped by folder with each flow's last result](docs/images/flows.png)

![Past runs, newest first, filterable by flow and status](docs/images/runs.png)

Light and dark themes follow the system or a setting.

![A failed run in the dark theme](docs/images/flow-failed-dark.png)

## Optional AI

AI is off by default, and the UI looks the same without it. With a provider
of your choice (an agent CLI you already use, or any command that reads a
prompt and prints text), a failed step gets a plain-words explanation in
place of its headline, with the headline kept underneath, and Open in agent
starts your agent CLI in the project with the failure's context. Each
failure is explained once and cached, under a daily limit.

![A failed step explained by a model, with Open in agent in the step's menu](docs/images/flow-failed-ai.png)

![AI settings: provider, command, agent command and daily limit](docs/images/settings-ai.png)

From the CLI, `lest explain <run>` prints the same explanation (or the
headline when AI is off), and `lest context <run>` prints everything needed to
pick up a failure, ready to paste anywhere.

## CI

`lest run` exits non-zero on failure and can write JUnit XML.
`lest affected --base origin/main` lists the flows whose files, called flows,
scripts or `affects` globs a change touches, so a pipeline runs only those.
Webhook notifications (JSON or Slack-compatible) go out after each run, and
`lest bundle` zips a run's report and artifacts with checksums for evidence.

## Examples

Expand Down
6 changes: 5 additions & 1 deletion crates/lest/src/args.rs
Original file line number Diff line number Diff line change
Expand Up @@ -140,8 +140,12 @@ pub struct UiArgs {
#[arg(long, default_value_t = 0)]
pub port: u16,
/// Print the address instead of opening a browser.
#[arg(long)]
#[arg(long, conflicts_with = "app")]
pub no_browser: bool,
/// Open in its own window (a Chromium-family browser's app mode) instead
/// of a browser tab.
#[arg(long)]
pub app: bool,
}

#[derive(Debug, Args)]
Expand Down
59 changes: 58 additions & 1 deletion crates/lest/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -966,6 +966,56 @@ fn cmd_config(ctx: &Ctx, command: ConfigCommand) -> Result<u8> {
}
}

/// Opens the UI as a standalone window: a Chromium-family browser in app
/// mode, with its own profile so it is a separate window and app.
fn open_app_window(url: &str, profile: &Path) -> bool {
let app = format!("--app={url}");
let data = format!("--user-data-dir={}", profile.display());
if cfg!(target_os = "macos") {
let home = std::env::var_os("HOME").map(PathBuf::from).unwrap_or_default();
for name in ["Google Chrome", "Chromium", "Microsoft Edge", "Brave Browser"] {
let bundle = format!("{name}.app");
if Path::new("/Applications").join(&bundle).is_dir() || home.join("Applications").join(&bundle).is_dir() {
return std::process::Command::new("open")
.args(["-na", name, "--args", &app, &data])
.status()
.is_ok_and(|s| s.success());
}
}
false
} else {
let Some(browser) = ["google-chrome", "chromium", "chromium-browser", "microsoft-edge", "brave-browser"]
.into_iter()
.find(|b| which::which(b).is_ok())
else {
return false;
};
let mut cmd = std::process::Command::new(browser);
cmd.args([&app, &data]).stdout(std::process::Stdio::null()).stderr(std::process::Stdio::null());
// Its own process group, so Ctrl-C in this terminal does not close
// the window (or other projects' windows sharing the profile).
#[cfg(unix)]
{
use std::os::unix::process::CommandExt;
cmd.process_group(0);
}
let Ok(mut child) = cmd.spawn() else { return false };
// A browser that cannot start (no display, a locked profile) exits
// at once with an error; one that hands the window to a running
// browser exits at once with success.
let deadline = std::time::Instant::now() + std::time::Duration::from_millis(1500);
while std::time::Instant::now() < deadline {
match child.try_wait() {
Ok(Some(status)) => return status.success(),
Ok(None) => std::thread::sleep(std::time::Duration::from_millis(50)),
Err(_) => return false,
}
}
std::thread::spawn(move || child.wait());
true
}
}

async fn cmd_ui(ctx: &Ctx, a: UiArgs) -> Result<u8> {
let roots = ctx.roots.clone();
let project = ctx.project.clone();
Expand All @@ -981,7 +1031,14 @@ async fn cmd_ui(ctx: &Ctx, a: UiArgs) -> Result<u8> {
let url = server.url();
eprintln!("lest ui: {url}");
eprintln!("project: {}", ctx.project.root.display());
if !a.no_browser {
if a.app && open_app_window(&url, &ctx.roots.data_dir.join("ui-window")) {
eprintln!("opened in its own window");
} else if !a.no_browser {
if a.app {
eprintln!(
"could not open an app window (no Chromium-family browser, or it failed to start); opening a browser tab"
);
}
let opener = if cfg!(target_os = "macos") { "open" } else { "xdg-open" };
if std::process::Command::new(opener).arg(&url).status().is_err() {
eprintln!("open the address above in a browser");
Expand Down
37 changes: 20 additions & 17 deletions docs/design.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ each. Behavior is specified in the reference docs (start with
1. **A run is deterministic and reproducible.** Nothing a model says can
change a run's outcome. AI sits around runs, never inside them.
2. **One run model.** A suite is a flow whose steps call other flows. A demo
is a flow that records a browser step. Every surface (CLI, desktop, web
UI, notifications) consumes one event stream and one report format.
is a flow that records a browser step. Every surface (CLI, UI,
notifications) consumes one event stream and one report format.
3. **The core owns everything; the shells are thin.** The CLI and the UI
call the same library. No behavior exists only in one shell.
4. **Values are data, never source text.** Template values reach a shell
Expand All @@ -37,7 +37,7 @@ crates/lest-core flow spec, expressions, validation, runner, reports,
store, secrets, tool profiles, browser bridge, media,
AI provider, notifications
crates/lest-server local HTTP API + server-sent events + embedded UI
crates/lest the `lest` binary: CLI, `lest ui`, optional desktop shell
crates/lest the `lest` binary: CLI and `lest ui`
ui/ React + TypeScript UI (served by lest-server)
harness/ the Node browser harness the runner launches
examples/ a self-contained sample app and flows exercising every feature
Expand All @@ -46,28 +46,30 @@ examples/ a self-contained sample app and flows exercising every feat
### Server-first UI

The UI is a web app served by `lest ui` on `127.0.0.1` and talks to the core
over HTTP and server-sent events. The desktop app is a Tauri window that
loads the same URL.
over HTTP and server-sent events. `lest ui --app` opens it as its own window
(a Chromium-family browser in app mode, with a separate profile).

Why not Tauri IPC:
Why not a desktop webview with IPC:

- **One transport.** IPC commands plus hand-written TypeScript mirrors drift
from the Rust types. A small HTTP API with JSON types generated from the
Rust structs has one source of truth.
from the Rust types. One small HTTP API serves the UI, scripts and tests.
- **The real app is testable.** Playwright can drive the actual UI in CI on
Linux (WebKit webviews expose no automation protocol), so screenshots in
docs and PRs come from the shipped UI, not from a separate fixture harness.
- **Files are URLs.** Artifacts, screenshots and videos are served with range
requests instead of base64 strings over IPC, so a 200 MB recording plays
without loading into memory.
- **The desktop shell is optional.** `cargo install` without the `desktop`
feature still gives a full UI in the browser; Linux servers and CI need no
webview libraries.
- **No webview to build.** One binary with the UI embedded, on every
platform, with no webview libraries; CI builds and tests it on Linux. A
native shell can wrap the same URL later if a window manager integration
ever needs it; `--app` covers a standalone window today.

The server binds loopback only, requires a per-launch random token (passed in
the URL once, then held in an HttpOnly SameSite=Strict cookie), checks the
`Host` header against loopback names to block DNS rebinding, and sets a strict
Content-Security-Policy.
the URL once, then held in an HttpOnly SameSite=Strict cookie named for the
port), checks the `Host` header against loopback names to block DNS
rebinding, accepts changes only from its own origin, serves run files in a
sandbox, and sets a strict Content-Security-Policy. Holding the token is
equivalent to running commands as the user ([ai.md](ai.md), Trust).

### Runner

Expand Down Expand Up @@ -228,9 +230,10 @@ implemented the same way.
### State and paths

Config in the user config dir (`lest/config.yml`), run data in the user state
dir (`lest/runs/<flow>/<run>/report.json` plus `artifacts/`), with a retention
cap and `lest data purge`. Projects are directories with a `lest.yaml`; the UI
opens projects instead of falling back to a global tests directory.
dir (`lest/runs/<flow>/<run>/report.json` plus `artifacts/`), removed with
`lest data prune --keep <n>` or `lest data purge`. Projects are directories
with a `lest.yaml`; the UI opens projects instead of falling back to a global
tests directory.

## Not built, on purpose

Expand Down
Binary file added docs/images/demo.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/images/demo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/images/flow-failed-ai.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/images/flow-failed-dark.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/images/flow-failed.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/images/flow-passed.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/images/flow-running-live.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/images/flows.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/images/runs.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/images/settings-ai.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/images/settings.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 6 additions & 0 deletions docs/ui.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@
same flows, runs the same runner and shows the same reports as the CLI. The
source is in `ui/` (see [ui/README.md](../ui/README.md) for development).

`lest ui` opens the address in your default browser; `--no-browser` only
prints it. `lest ui --app` opens it in its own window instead: a
Chromium-family browser (Chrome, Chromium, Edge or Brave) in app mode, with a
profile of its own under the data directory. Without one, it falls back to a
browser tab.

The left rail holds Flows, Runs, Demos and Settings, and shows the project
name. It collapses to icons; the choice is remembered in the browser.

Expand Down
6 changes: 6 additions & 0 deletions ui/scripts/screenshots.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,12 @@ async function capture() {
await apage.click('[aria-label="AI provider"] [role="radio"]:has-text("Custom command")');
await apage.fill("#ai-command", aiCommand);
await apage.waitForSelector('.settings-status:has-text("Using")');
// A custom command has no agent of its own; Open in agent needs one.
await apage.fill("#ai-agent", "claude {prompt}");
for (let i = 0; (await api("/settings")).ai.agent !== "claude {prompt}"; i++) {
if (i > 50) throw new Error("the agent command was not saved");
await apage.waitForTimeout(200);
}
await apage.waitForSelector('.settings-status .saved:has-text("Saved")');
await shot(apage, "settings-ai.png");

Expand Down
8 changes: 5 additions & 3 deletions ui/src/components/RunView.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -583,7 +583,7 @@ function FailedCard({ s, ctx }: { s: StepState; ctx: Ctx }) {
const stored = !ctx.live && !!run.report;
const explainState = useExplanation(ai?.status ?? null, stored ? run.runId : null, s.id);
const slot = explainSlot(headline, explainState);
const handoff = useHandoff(ai?.status ?? null, stored ? run.runId : null, s.id);
const handoff = useHandoff(ai?.status ?? null, ai?.ai.agent ?? "", stored ? run.runId : null, s.id);
const [launch, setLaunch] = useState<{ error: string; command: string | null } | null>(null);
const openAgent = async () => {
try {
Expand Down Expand Up @@ -702,7 +702,7 @@ function useExplanation(status: AiStatus | null, runId: string | null, stepId: s
}

/** The agent handoff for a finished run, fetched when AI may be on. */
function useHandoff(status: AiStatus | null, runId: string | null, stepId: string): Handoff | null {
function useHandoff(status: AiStatus | null, agent: string, runId: string | null, stepId: string): Handoff | null {
const [handoff, setHandoff] = useState<Handoff | null>(null);
const resolved = status?.resolved?.label ?? "";
useEffect(() => {
Expand All @@ -711,14 +711,16 @@ function useHandoff(status: AiStatus | null, runId: string | null, stepId: strin
return;
}
let alive = true;
// The previous command must not be copied while the new one loads.
setHandoff(null);
api.handoff(runId, stepId).then(
(h) => alive && setHandoff(h),
() => alive && setHandoff(null),
);
return () => {
alive = false;
};
}, [status?.configured, resolved, runId, stepId]);
}, [status?.configured, resolved, agent, runId, stepId]);
return handoff;
}

Expand Down
3 changes: 1 addition & 2 deletions ui/src/lib/ai.ts
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,7 @@ export function useAiSettings(): SettingsResponse | null {
return s;
}

/** Saves the AI settings. The agent command has no field in the UI, so the
* one in the settings file is kept. */
/** Saves the AI settings and shares the result with every view. */
export async function saveAi(ai: AiConfig): Promise<SettingsResponse> {
const s = await api.saveAi(ai);
settings = s;
Expand Down
Loading
Loading