From aae2e2fbccb816a90ce650dac9181df874e16b77 Mon Sep 17 00:00:00 2001 From: Patrick Bertsch Date: Sun, 25 Oct 2026 19:21:34 -0600 Subject: [PATCH 1/8] feat(lint): report unresolved steps, selectors and placeholders; unknown steps fail probe lint probe lint now checks the references a run would resolve, without a device or a run. Recipe calls and use statements resolve the way --dry-run resolves them, with the recipes folder from the config. Every step is walked, nested steps included, for selectors that can never match and for placeholders that no example column, recipe parameter, stored value or runtime value defines. A step or recipe call that resolves to nothing, and a selector that can never match, fail the command. A placeholder no value defines, and a use file that does not exist, are warnings. Environment variables are not checked. The recipe loading that --dry-run used moves into loadFileRecipes, so lint and dry-run share it. --- CHANGELOG.md | 2 + internal/cli/lint.go | 42 ++- internal/cli/lint_references_test.go | 26 ++ internal/mcp/server.go | 2 +- internal/runner/lint.go | 294 ++++++++++++++++++ internal/runner/lint_test.go | 185 +++++++++++ internal/runner/runner.go | 38 ++- .../src/content/docs/tools/cli-reference.md | 15 +- 8 files changed, 579 insertions(+), 25 deletions(-) create mode 100644 internal/cli/lint_references_test.go create mode 100644 internal/runner/lint.go create mode 100644 internal/runner/lint_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 32c2bf4a..641a56c6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added - **MCP: `start_tests`, `test_status`, `test_result` and `cancel_tests` run tests in the background.** `run_tests` blocks until `probe test` exits, so a multi-device run (`--parallel`, `composite_devices`, several devices) could outlast the MCP client's tool timeout and be reported as failed while it kept running. `start_tests` takes the same arguments, returns a `job_id` at once and runs the command in its own process group; `test_status` returns the state, elapsed time and the last lines of output; `test_result` returns the retained output (the last 1 MB) and the exit code; `cancel_tests` sends SIGTERM to the run and kills it after 5 seconds. Jobs live in the `probe-mcp` process and are killed when it exits. The hosted-upload flags are refused here exactly as in `run_tests`. `run_tests` is unchanged. The MCP server now has 40 tools. +- **`probe lint` checks the references a run would resolve.** Lint reports each step and recipe call that matches no built-in step and no recipe in the file, its `use` statements or the recipes folder (the check `probe test --dry-run` makes), each selector that can never match, and each `` placeholder that no example column, recipe parameter or stored value defines. It runs without a device and gives the line of each finding. - **Recordings show the element each step acted on.** A step that taps, double taps, long presses, types, clears, toggles, or swipes or scrolls with a selector records its selector, and its bounds and tap point when the element is on screen as the step starts. The viewer shows them under "Element". Text typed into a field is never recorded, and a failed lookup is an advisory warning that does not change the step. - **`probe test --record` writes a step-by-step recording of each test.** Each test attempt includes source files and lines, screenshots before and after steps, widget trees after steps, and the app's `dart:io` HTTP exchanges with credential redaction, plus an offline viewer. Bundles stay under `reports_folder/recordings` or `--record-dir`; nothing uploads, and recording warnings do not change test results or exit codes. Native SDK and WebView traffic and screenshot redaction rules are not yet recorded; screenshots are not redacted and widget trees are not captured before steps. Keep recordings out of version control because they can contain screens and request or response bodies. - **Recording manifests name the platform and OS version.** `manifest.json` in each recording bundle now carries the run's `platform` (`ios` or `android`) and `os_version`, the same values the report shows, instead of leaving both empty. @@ -23,6 +24,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **The MCP tools keep to the local tool surface.** `run_tests` and `run_script` refuse the upload flags (`--upload-report`, `--upload-local`, `--upload-dry-run`, and any other `--upload-` flag) with the message "hosted upload is not available through MCP". The MCP tool text no longer describes the upload flags. Use the CLI from a terminal for uploads. - **Building from source needs Go 1.26.10 or newer (was 1.26.1).** The `go` line in `go.mod`, `studio/go.mod`, `tools/probe-convert/go.mod` and `go.work` is now 1.26.10, and the install docs say so. With the default `GOTOOLCHAIN=auto`, an older Go 1.26 downloads 1.26.10 on its own; with `GOTOOLCHAIN=local` the build stops with a version error until Go is upgraded. - **Older evidence files removed.** The disclosure guard also scans `docs/evidence`. +- **`probe lint` fails on references a run cannot resolve.** A step or recipe call that resolves to nothing, or a selector that can never match, now makes the command exit with status 1, where it used to pass. A placeholder no value defines and a `use` file that does not exist are warnings and do not change the exit status. A repository whose CI runs `probe lint` may see new failures on files that already fail at run time; `probe test --dry-run` reports the same steps. Environment variables (`${NAME}`) are not checked. ### Docs - **Recording evidence write-ups no longer name private apps.** The prose and commands in the evidence notes use neutral labels and placeholders. The raw run output that the notes cite is unchanged. diff --git a/internal/cli/lint.go b/internal/cli/lint.go index 9874d7a8..14b01dd3 100644 --- a/internal/cli/lint.go +++ b/internal/cli/lint.go @@ -17,6 +17,10 @@ var lintCmd = &cobra.Command{ Example: ` probe lint tests/ probe lint tests/login.probe`, RunE: func(cmd *cobra.Command, args []string) error { + cfg, err := loadConfig(cmd) + if err != nil { + return fmt.Errorf("loading config: %w", err) + } searchPaths := args if len(searchPaths) == 0 { searchPaths = []string{"tests"} @@ -42,15 +46,39 @@ var lintCmd = &cobra.Command{ continue } - // Lint checks - warnings := lintProgram(prog) - if len(warnings) > 0 { - fmt.Printf(" \033[33m⚠\033[0m %s\n", f) - for _, w := range warnings { - fmt.Printf(" %s\n", w) + // Lint checks: a reference a run could not resolve is an error and fails the command; a placeholder that + // no value defines is a warning. + refs, err := runner.LintReferences(cfg, f) + if err != nil { + fmt.Printf(" \033[31m✗\033[0m %s — %s\n", f, err) + errs++ + continue + } + var problems, warnings []string + for _, ref := range refs { + line := fmt.Sprintf("line %d: %s", ref.Line, ref.Message) + if ref.Warning { + warnings = append(warnings, line) + } else { + problems = append(problems, line) } - } else { + } + warnings = append(lintProgram(prog), warnings...) + if len(problems) == 0 && len(warnings) == 0 { statusOK(os.Stdout, "%s (%d test(s))", f, len(prog.Tests)) + continue + } + if len(problems) > 0 { + errs++ + fmt.Printf(" \033[31m✗\033[0m %s\n", f) + } else { + fmt.Printf(" \033[33m⚠\033[0m %s\n", f) + } + for _, p := range problems { + fmt.Printf(" %s\n", p) + } + for _, w := range warnings { + fmt.Printf(" %s\n", w) } } diff --git a/internal/cli/lint_references_test.go b/internal/cli/lint_references_test.go new file mode 100644 index 00000000..9a6fe2a4 --- /dev/null +++ b/internal/cli/lint_references_test.go @@ -0,0 +1,26 @@ +package cli + +import ( + "os" + "path/filepath" + "testing" +) + +// probe lint fails on a reference a run cannot resolve, and only warns about a placeholder no value defines. +func TestLintCommandFailsOnUnknownStepsButNotOnWarnings(t *testing.T) { + dir := t.TempDir() + warn := filepath.Join(dir, "warn.probe") + if err := os.WriteFile(warn, []byte("test \"x\"\n type \"\" into the \"Email\" field\n"), 0o644); err != nil { + t.Fatal(err) + } + bad := filepath.Join(dir, "bad.probe") + if err := os.WriteFile(bad, []byte("test \"x\"\n frobnicate the cart\n"), 0o644); err != nil { + t.Fatal(err) + } + if err := lintCmd.RunE(lintCmd, []string{warn}); err != nil { + t.Errorf("a placeholder warning must not fail lint: %v", err) + } + if err := lintCmd.RunE(lintCmd, []string{bad}); err == nil { + t.Error("an unknown step must fail lint") + } +} diff --git a/internal/mcp/server.go b/internal/mcp/server.go index 949beb87..aca46283 100644 --- a/internal/mcp/server.go +++ b/internal/mcp/server.go @@ -427,7 +427,7 @@ Takes the same arguments as run_tests (see its description for the flags). Then }, { Name: "lint", - Description: "Validate .probe files; warns on count and zero-state text checks, never fails.", + Description: "Validate .probe files without running them. Fails on a step or recipe call that resolves to nothing and on a selector that can never match; warns on unresolved placeholders, unknown use files and count or zero-state text checks.", InputSchema: mcpSchema{ Type: "object", Properties: map[string]mcpProp{ diff --git a/internal/runner/lint.go b/internal/runner/lint.go new file mode 100644 index 00000000..cb443447 --- /dev/null +++ b/internal/runner/lint.go @@ -0,0 +1,294 @@ +package runner + +import ( + "context" + "fmt" + "maps" + "os" + "path/filepath" + "reflect" + "regexp" + "sort" + "strings" + + "github.com/alphawavesystems/flutter-probe/internal/config" + "github.com/alphawavesystems/flutter-probe/internal/parser" +) + +// LintFinding is one problem LintReferences found in a file. An error is a reference a run cannot satisfy as written: a +// step or recipe call that resolves to nothing, or a selector that can never match. A warning is a placeholder that no +// example column, recipe parameter or stored value defines (a run leaves it in the text as written), or a use that names +// no file (a run only needs that file when a call uses one of its recipes). +type LintFinding struct { + Line int + Warning bool + Message string +} + +// placeholderPattern matches a placeholder in a step's text. Names are identifiers, with dots for names such as +// response.status. generators do not match it as a name, and are skipped by name below. +var placeholderPattern = regexp.MustCompile(`<([A-Za-z_][A-Za-z0-9_.]*)>`) + +// runtimeVars are placeholder names the executor sets itself while it runs a step. +var runtimeVars = []string{"clipboard", "response.status", "response.body"} + +var ( + stepType = reflect.TypeFor[parser.Step]() + stepSliceType = reflect.TypeFor[[]parser.Step]() + selectorType = reflect.TypeFor[parser.Selector]() +) + +// LintReferences checks path for references that a run could not resolve, without running any step. Recipe calls and +// use statements resolve the way --dry-run resolves them, with the recipes folder from cfg. A file that does not parse +// returns the parse error. +func LintReferences(cfg *config.Config, path string) ([]LintFinding, error) { + src, err := os.ReadFile(path) + if err != nil { + return nil, err + } + prog, err := parser.ParseFile(string(src)) + if err != nil { + return nil, err + } + ctx := context.Background() + r := New(cfg, nil, nil, RunOptions{DryRun: true, Files: []string{path}}) + if err := r.loadRecipes(ctx); err != nil { + return nil, err + } + r.baseRecipes = make(map[string]parser.RecipeDef, len(r.recipes)) + maps.Copy(r.baseRecipes, r.recipes) + if err := r.loadFileRecipes(ctx, path, prog); err != nil { + return nil, err + } + + var findings []LintFinding + for _, u := range prog.Uses { + target := filepath.Join(filepath.Dir(path), u.Path) + if _, err := os.Stat(target); os.IsNotExist(err) { + // A missing file is a warning: a run only fails when a call needs one of its recipes, and that call is + // reported as an unknown step. + findings = append(findings, LintFinding{Line: u.Line, Warning: true, Message: fmt.Sprintf("use %q: no file at %s", u.Path, target)}) + } + } + + // The file's own bodies are checked. The recipes it can call only add the names they define, because a recipe + // parameter or a stored value from one of them stays set for the rest of the run. + var own [][]parser.Step + for _, t := range prog.Tests { + own = append(own, t.Body) + } + for _, h := range prog.Hooks { + own = append(own, h.Body) + } + for _, rec := range prog.Recipes { + own = append(own, rec.Body) + } + for _, c := range prog.CompositeTests { + own = append(own, c.Body) + } + + defined := map[string]bool{} + for _, name := range runtimeVars { + defined[name] = true + } + for _, t := range prog.Tests { + if t.Examples == nil { + continue + } + headers := t.Examples.Headers + if t.Examples.Source != "" { + if csv, err := loadCSVExamples(filepath.Dir(path), t.Examples.Source); err == nil { + headers = csv.Headers + } + } + for _, h := range headers { + defined[h] = true + } + } + for _, rec := range r.recipes { + for _, p := range rec.Params { + defined[p] = true + } + walkSteps(rec.Body, func(s parser.Step) { markStored(s, defined) }) + } + for _, body := range own { + walkSteps(body, func(s parser.Step) { markStored(s, defined) }) + } + + for _, body := range own { + walkSteps(body, func(s parser.Step) { + line := stepLine(s) + if rc, ok := s.(parser.RecipeCall); ok { + if _, _, _, found := resolveCall(r.recipes, rc); !found { + msg := fmt.Sprintf("unknown step %q: not a built-in step and no recipe with that name is defined (check the use statements and the recipes folder)", rc.Name) + if strings.HasPrefix(rc.Name, "use ") || rc.Name == "use" { + // The parser reads use only at the top of a file; inside a test it is an ordinary call. + msg = "a use statement must come before the first test; inside a test it is not read" + } + findings = append(findings, LintFinding{Line: rc.Line, Message: msg}) + } + } + texts, sels := scanStep(s) + for _, sel := range sels { + if problem := selectorProblem(sel); problem != "" { + findings = append(findings, LintFinding{Line: line, Message: problem}) + } + } + for _, text := range texts { + for _, m := range placeholderPattern.FindAllStringSubmatch(text, -1) { + name := m[1] + if defined[name] || strings.HasPrefix(name, "random.") { + continue + } + findings = append(findings, LintFinding{Line: line, Warning: true, Message: fmt.Sprintf("<%s> is not defined: no example column, recipe parameter or stored value has that name", name)}) + } + } + }) + } + return dedupeFindings(findings), nil +} + +// markStored records the names a step stores, which a placeholder may use later in the run. +func markStored(s parser.Step, defined map[string]bool) { + if a, ok := s.(parser.ActionStep); ok && (a.Verb == parser.VerbStore || a.Verb == parser.VerbReadWithAI) { + defined[a.Name] = true + } +} + +// selectorProblem names why a selector can never match, or returns "" when it can. A plain text selector is not checked: +// an empty one is how a step with no target is written, and some steps take no target at all. +func selectorProblem(sel parser.Selector) string { + switch sel.Kind { + case parser.SelectorID: + if sel.Text == "" || sel.Text == "#" { + return "a #id selector needs a name after the #" + } + case parser.SelectorOrdinal: + if sel.Ordinal < 1 || sel.Text == "" { + return "an ordinal selector needs a number and text, such as 2nd \"Buy\"" + } + case parser.SelectorPositional: + if sel.Container == "" { + return "the container after \"in\" is empty" + } + case parser.SelectorRelational: + if sel.Anchor == "" { + return fmt.Sprintf("the %s selector has no anchor text after it", sel.Relation) + } + } + return "" +} + +// stepLine is the 1-based line a step starts on, or 0 when the step does not record one. +func stepLine(s parser.Step) int { + if n, ok := s.(parser.Node); ok { + return n.GetLine() + } + return 0 +} + +// walkSteps calls visit for every step in steps and, depth first, for the steps nested in each of them. +func walkSteps(steps []parser.Step, visit func(parser.Step)) { + for _, s := range steps { + if s == nil { + continue + } + visit(s) + walkSteps(nestedSteps(s), visit) + } +} + +// nestedSteps returns the steps held in a step's fields: a conditional's branches, a loop's or retry's body, a timed +// block's inner step and the step a device section wraps. +func nestedSteps(s parser.Step) []parser.Step { + var out []parser.Step + v := reflect.ValueOf(s) + for v.Kind() == reflect.Pointer || v.Kind() == reflect.Interface { + if v.IsNil() { + return nil + } + v = v.Elem() + } + if v.Kind() != reflect.Struct { + return nil + } + for i := 0; i < v.NumField(); i++ { + f := v.Field(i) + if !f.CanInterface() { + continue + } + switch f.Type() { + case stepSliceType: + for j := 0; j < f.Len(); j++ { + if st, ok := f.Index(j).Interface().(parser.Step); ok { + out = append(out, st) + } + } + case stepType: + if !f.IsNil() { + if st, ok := f.Interface().(parser.Step); ok { + out = append(out, st) + } + } + } + } + return out +} + +// scanStep collects the text values and the selectors in a step's own fields. Nested steps are left out, because +// walkSteps visits them on their own. +func scanStep(s parser.Step) ([]string, []parser.Selector) { + var texts []string + var sels []parser.Selector + if rc, ok := s.(parser.RecipeCall); ok { + // The name of a call holds markers in the places of its quoted arguments; they are not variables. The + // arguments can hold placeholders of their own. + scanValue(reflect.ValueOf(rc.Args), &texts, &sels) + scanValue(reflect.ValueOf(rc.NumArgs), &texts, &sels) + return texts, sels + } + scanValue(reflect.ValueOf(s), &texts, &sels) + return texts, sels +} + +func scanValue(v reflect.Value, texts *[]string, sels *[]parser.Selector) { + switch v.Kind() { + case reflect.String: + *texts = append(*texts, v.String()) + case reflect.Interface, reflect.Pointer: + if !v.IsNil() { + scanValue(v.Elem(), texts, sels) + } + case reflect.Slice, reflect.Array: + for i := 0; i < v.Len(); i++ { + scanValue(v.Index(i), texts, sels) + } + case reflect.Struct: + if v.Type() == selectorType && v.CanInterface() { + *sels = append(*sels, v.Interface().(parser.Selector)) + } + for i := 0; i < v.NumField(); i++ { + f := v.Field(i) + if f.Type() == stepType || f.Type() == stepSliceType { + continue + } + scanValue(f, texts, sels) + } + } +} + +// dedupeFindings drops repeated findings and orders the rest by line, keeping the order of the file within a line. +func dedupeFindings(findings []LintFinding) []LintFinding { + seen := map[string]bool{} + var out []LintFinding + for _, f := range findings { + key := fmt.Sprintf("%d|%t|%s", f.Line, f.Warning, f.Message) + if seen[key] { + continue + } + seen[key] = true + out = append(out, f) + } + sort.SliceStable(out, func(i, j int) bool { return out[i].Line < out[j].Line }) + return out +} diff --git a/internal/runner/lint_test.go b/internal/runner/lint_test.go new file mode 100644 index 00000000..6a5a1e77 --- /dev/null +++ b/internal/runner/lint_test.go @@ -0,0 +1,185 @@ +package runner + +import ( + "os" + "path/filepath" + "strings" + "testing" + + "github.com/alphawavesystems/flutter-probe/internal/config" + "github.com/alphawavesystems/flutter-probe/internal/parser" +) + +// lintFiles writes files into a fresh directory, then lints the file called main and returns its findings. +func lintFiles(t *testing.T, files map[string]string, main string) []LintFinding { + t.Helper() + dir := t.TempDir() + for name, src := range files { + if err := os.WriteFile(filepath.Join(dir, name), []byte(src), 0o644); err != nil { + t.Fatal(err) + } + } + findings, err := LintReferences(&config.Config{}, filepath.Join(dir, main)) + if err != nil { + t.Fatalf("LintReferences: %v", err) + } + return findings +} + +// lintSource lints one file called flows.probe. +func lintSource(t *testing.T, src string) []LintFinding { + t.Helper() + return lintFiles(t, map[string]string{"flows.probe": src}, "flows.probe") +} + +// requireFinding fails unless findings holds one finding on line with the given severity whose message contains substr. +func requireFinding(t *testing.T, findings []LintFinding, line int, warning bool, substr string) { + t.Helper() + for _, f := range findings { + if f.Line == line && f.Warning == warning && strings.Contains(f.Message, substr) { + return + } + } + t.Errorf("no finding on line %d (warning=%t) containing %q; got %+v", line, warning, substr, findings) +} + +func requireNoFindings(t *testing.T, findings []LintFinding) { + t.Helper() + if len(findings) != 0 { + t.Errorf("expected no findings, got %+v", findings) + } +} + +func TestLintReferencesFlagsUnknownSteps(t *testing.T) { + findings := lintSource(t, "test \"checkout\"\n tap \"Buy\"\n frobnicate the cart\n") + requireFinding(t, findings, 3, false, "unknown step") + if len(findings) != 1 { + t.Errorf("expected only the unknown step, got %+v", findings) + } +} + +// A use statement inside a test is an ordinary call to a recipe named use, which no run can resolve. +func TestLintReferencesExplainsAUseInsideATest(t *testing.T) { + findings := lintSource(t, "test \"nav\"\n use \"lib.probe\"\n tap \"Go\"\n") + requireFinding(t, findings, 2, false, "must come before the first test") +} + +func TestLintReferencesAcceptsRecipesDefinedInTheFile(t *testing.T) { + src := "recipe \"sign in\" (email)\n type \"\" into the \"Email\" field\n\ntest \"login\"\n sign in \"a@b.c\"\n" + requireNoFindings(t, lintSource(t, src)) +} + +func TestLintReferencesAcceptsRecipesFromTheRecipesFolder(t *testing.T) { + recipes := t.TempDir() + if err := os.WriteFile(filepath.Join(recipes, "shared.probe"), []byte("recipe \"open menu\"\n tap \"Menu\"\n"), 0o644); err != nil { + t.Fatal(err) + } + dir := t.TempDir() + path := filepath.Join(dir, "flows.probe") + if err := os.WriteFile(path, []byte("test \"nav\"\n open menu\n"), 0o644); err != nil { + t.Fatal(err) + } + findings, err := LintReferences(&config.Config{Recipes: recipes}, path) + if err != nil { + t.Fatal(err) + } + requireNoFindings(t, findings) +} + +func TestLintReferencesWarnsAboutUseOfAMissingFile(t *testing.T) { + findings := lintSource(t, "use \"missing.probe\"\n\ntest \"x\"\n tap \"Go\"\n") + requireFinding(t, findings, 1, true, "no file at") +} + +// A missing use file fails only the calls that need its recipes: they are errors, and the use is just a warning. +func TestLintReferencesErrorsOnACallThatNeedsAMissingUse(t *testing.T) { + findings := lintSource(t, "use \"missing.probe\"\n\ntest \"x\"\n open menu\n") + requireFinding(t, findings, 1, true, "no file at") + requireFinding(t, findings, 4, false, "unknown step") +} + +func TestLintReferencesAcceptsRecipesFromAUseStatement(t *testing.T) { + files := map[string]string{ + "lib.probe": "recipe \"open menu\"\n tap \"Menu\"\n", + "flows.probe": "use \"lib.probe\"\n\ntest \"nav\"\n open menu\n", + } + requireNoFindings(t, lintFiles(t, files, "flows.probe")) +} + +func TestLintReferencesWarnsAboutAPlaceholderNothingDefines(t *testing.T) { + findings := lintSource(t, "test \"login\"\n type \"\" into the \"Email\" field\n") + requireFinding(t, findings, 2, true, " is not defined") + if len(findings) != 1 { + t.Errorf("expected only the placeholder warning, got %+v", findings) + } +} + +func TestLintReferencesAcceptsPlaceholdersThatAValueDefines(t *testing.T) { + cases := map[string]string{ + "example column": "test \"login\"\n type \"\" into the \"Email\" field\n\nwith examples:\n email\n \"a@b.c\"\n", + "stored value": "test \"login\"\n store \"abc\" as token\n type \"\" into the \"Field\"\n", + "recipe param": "recipe \"sign in\" (email)\n type \"\" into the \"Email\" field\n\ntest \"login\"\n sign in \"a@b.c\"\n", + "runtime value": "test \"status\"\n see \"\"\n", + "random value": "test \"gen\"\n see \"\"\n", + } + for name, src := range cases { + t.Run(name, func(t *testing.T) { + requireNoFindings(t, lintSource(t, src)) + }) + } +} + +func TestLintReferencesFlagsASelectorThatCanNeverMatch(t *testing.T) { + cases := []struct { + name string + step string + want string + }{ + {"ordinal without text", "tap 2nd", "an ordinal selector needs a number and text"}, + {"relational without anchor", "tap \"Save\" below", "the below selector has no anchor text"}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + findings := lintSource(t, "test \"x\"\n "+c.step+"\n") + requireFinding(t, findings, 2, false, c.want) + }) + } +} + +func TestLintReferencesAcceptsWorkingSelectors(t *testing.T) { + src := "test \"x\"\n tap 2nd \"Buy\"\n tap \"Save\" below \"Cancel\"\n tap \"Buy\" in \"Cart\"\n tap #buy_button\n" + requireNoFindings(t, lintSource(t, src)) +} + +// walkSteps reaches the steps nested in conditionals, loops, retries and timed blocks, not only the top level. +func TestWalkStepsVisitsNestedBodies(t *testing.T) { + inner := parser.ActionStep{Verb: parser.VerbStore, Text: "x", Name: "v", Line: 3} + steps := []parser.Step{ + parser.ConditionalStep{Then: []parser.Step{inner}, Line: 1}, + parser.LoopStep{Body: []parser.Step{parser.WaitStep{Line: 4}}, Line: 2}, + } + var lines []int + walkSteps(steps, func(s parser.Step) { lines = append(lines, stepLine(s)) }) + want := []int{1, 3, 2, 4} + if len(lines) != len(want) { + t.Fatalf("visited lines %v, want %v", lines, want) + } + for i := range want { + if lines[i] != want[i] { + t.Errorf("visited lines %v, want %v", lines, want) + break + } + } +} + +func TestLintReferencesReportsAParseErrorWithoutFindings(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "flows.probe") + if err := os.WriteFile(path, []byte("test \"unterminated\n tap \"broken\n"), 0o644); err != nil { + t.Fatal(err) + } + findings, err := LintReferences(&config.Config{}, path) + if err == nil { + t.Fatalf("expected the parse error, got findings %+v", findings) + } +} diff --git a/internal/runner/runner.go b/internal/runner/runner.go index ee0033f2..8d6405ae 100644 --- a/internal/runner/runner.go +++ b/internal/runner/runner.go @@ -181,20 +181,11 @@ func (r *Runner) Run(ctx context.Context) ([]TestResult, error) { return results, nil } -func (r *Runner) runFile(ctx context.Context, path string) ([]TestResult, error) { - src, err := os.ReadFile(path) - if err != nil { - return nil, fmt.Errorf("read %s: %w", path, err) - } - - prog, err := parser.ParseFile(string(src)) - if err != nil { - return nil, fmt.Errorf("parse %s: %w", path, err) - } - - // --dry-run resolves each file on its own, like running that file alone: the - // recipes of other files in the same run must not make an unresolved call look - // fine (a runtime run of this one file could not see them). +// loadFileRecipes makes the recipes that path can call available to its tests: those of the recipes folder, those of +// its `use` statements (followed in turn), and its own. --dry-run resolves a file on its own, like running that file +// alone: the recipes of other files in the same run must not make an unresolved call look fine (a runtime run of this +// one file could not see them). +func (r *Runner) loadFileRecipes(ctx context.Context, path string, prog *parser.Program) error { if r.opts.DryRun && r.baseRecipes != nil { r.recipes = make(map[string]parser.RecipeDef, len(r.baseRecipes)) for k, v := range r.baseRecipes { @@ -211,7 +202,7 @@ func (r *Runner) runFile(ctx context.Context, path string) ([]TestResult, error) } for _, u := range prog.Uses { if err := r.loadUse(ctx, path, u, seen); err != nil { - return nil, err + return err } } @@ -219,6 +210,23 @@ func (r *Runner) runFile(ctx context.Context, path string) ([]TestResult, error) for _, rec := range prog.Recipes { r.recipes[rec.Name] = rec } + return nil +} + +func (r *Runner) runFile(ctx context.Context, path string) ([]TestResult, error) { + src, err := os.ReadFile(path) + if err != nil { + return nil, fmt.Errorf("read %s: %w", path, err) + } + + prog, err := parser.ParseFile(string(src)) + if err != nil { + return nil, fmt.Errorf("parse %s: %w", path, err) + } + + if err := r.loadFileRecipes(ctx, path, prog); err != nil { + return nil, err + } // Filter tests by tag tests := r.filterTests(prog.Tests) diff --git a/website/src/content/docs/tools/cli-reference.md b/website/src/content/docs/tools/cli-reference.md index a1423a44..58b59e7e 100644 --- a/website/src/content/docs/tools/cli-reference.md +++ b/website/src/content/docs/tools/cli-reference.md @@ -11,7 +11,7 @@ description: Complete reference for all probe CLI commands and flags. | `probe validate` | Check `probe.yaml` and the `probe_version` pin without connecting to a device. Exits non-zero when a check fails; `--format json` prints the report. Run it in CI before `probe test` | | `probe doctor` | Check that `probe.yaml`, the `tests/` folder, the Flutter SDK, the platform tools (`adb`, `xcrun`) and the visible devices are ready for `probe test`. Says how to fix what is missing and changes nothing. Exits non-zero only when `probe.yaml` or `tests/` is missing. `--format json` prints the report | | `probe test [path]` | Run `.probe` test files | -| `probe lint [path]` | Validate `.probe` files for syntax errors; warns, never fails, on count and zero-state text checks that a substring match passes by accident | +| `probe lint [path]` | Validate `.probe` files without running them: syntax errors, steps and recipe calls that resolve to nothing, and selectors that can never match fail the command; unresolved placeholders and count or zero-state text checks warn | | `probe perf trend` / `probe perf compare` | Show a measured number over past runs, or compare two baseline files. See [Performance testing](/advanced/performance/) | | `probe device list` | List connected devices/simulators | | `probe device view` | Interactive device view over a local WebSocket (preview) | @@ -152,7 +152,18 @@ probe lint tests/ probe lint tests/smoke/login.probe ``` -Lint also warns, and never fails, on a `see` or `wait until ... appears` whose text is a count or a zero state (`see "0 items"`, `wait until "No items" appears`) and has no `matching` pattern, because a substring match also passes on "10 items". Anchor the pattern for an exact check: `see "0 items" matching "^0 items$"`. A `matching` pattern anchored at both ends (`^...$`) silences the warning; any other pattern is still a substring match and is still flagged. Warnings do not change the exit status. +Lint checks the references a run would resolve, without running a step, and fails the command (exit status 1) for an error: + +- A step or recipe call that matches no built-in step and no recipe in the file, its `use` statements or the recipes folder. This is the check `probe test --dry-run` makes. A `use` statement inside a test is reported too, because a file reads `use` only before its first test. +- A selector that can never match: an ordinal selector without a number and text, an `#id` without a name, a `below`, `above`, `left of` or `right of` selector without anchor text, or an `in` with an empty container. + +It warns, without failing, when: + +- A `` placeholder is not an example column, a recipe parameter, a value stored with `store` or `read ... with ai`, one of the runtime values `clipboard`, `response.status` or `response.body`, or a `` generator. A run leaves such a placeholder in the text as written. +- A `use` names a file that does not exist. A run only needs that file when a call uses one of its recipes, and such a call is reported as an unknown step. +- A `see` or `wait until ... appears` whose text is a count or a zero state (`see "0 items"`, `wait until "No items" appears`) and has no `matching` pattern, because a substring match also passes on "10 items". Anchor the pattern for an exact check: `see "0 items" matching "^0 items$"`. A `matching` pattern anchored at both ends (`^...$`) silences the warning; any other pattern is still a substring match and is still flagged. + +Environment variables (`${NAME}`) are not checked, because lint does not know the environment that will run the tests. ## probe record From d66c544c73834f918c30a63c71b0d75ae2e0d059 Mon Sep 17 00:00:00 2001 From: Patrick Bertsch Date: Sun, 25 Oct 2026 19:53:32 -0600 Subject: [PATCH 2/8] fix(tests): the smoke navigation step uses built-in steps; use paths resolve from each file The step `go back to main menu` is read as the built-in `go back` plus a recipe call named `main menu`, so the test failed at run time. probe test --dry-run and probe lint both reported it. The step now inlines the three steps of the recipe it meant: go back, wait 1 seconds, see "Home". Five test files named tests/recipes/common.probe from the project root. A use path resolves relative to the file that contains it, so those paths now name the file from its own location. A follow-up will decide whether a phrase like this should be able to name a recipe. --- tests/cart.probe | 2 +- tests/login.probe | 2 +- tests/recipes/login.probe | 2 +- tests/regression/visual.probe | 2 +- tests/smoke/navigation.probe | 6 ++++-- 5 files changed, 8 insertions(+), 6 deletions(-) diff --git a/tests/cart.probe b/tests/cart.probe index 27898d44..cc696cb3 100644 --- a/tests/cart.probe +++ b/tests/cart.probe @@ -1,5 +1,5 @@ # Code activation flow tests -use "tests/recipes/common.probe" +use "recipes/common.probe" before each test open and wait for the app diff --git a/tests/login.probe b/tests/login.probe index 27e66075..229f93b9 100644 --- a/tests/login.probe +++ b/tests/login.probe @@ -1,5 +1,5 @@ # Activation code entry tests -use "tests/recipes/common.probe" +use "recipes/common.probe" before each test open and wait for the app diff --git a/tests/recipes/login.probe b/tests/recipes/login.probe index c77a8000..b1fa7346 100644 --- a/tests/recipes/login.probe +++ b/tests/recipes/login.probe @@ -1,4 +1,4 @@ -use "tests/recipes/common.probe" +use "common.probe" recipe "activate code" (d1, d2, d3, d4, d5) navigate to keypad diff --git a/tests/regression/visual.probe b/tests/regression/visual.probe index b5de09ef..228566df 100644 --- a/tests/regression/visual.probe +++ b/tests/regression/visual.probe @@ -1,6 +1,6 @@ # Visual regression test suite # Each test captures a screenshot and compares against a baseline. -use "tests/recipes/common.probe" +use "../recipes/common.probe" before each test open and wait for the app diff --git a/tests/smoke/navigation.probe b/tests/smoke/navigation.probe index d6265f6e..ad76286a 100644 --- a/tests/smoke/navigation.probe +++ b/tests/smoke/navigation.probe @@ -1,5 +1,5 @@ # Smoke test: core navigation paths -use "tests/recipes/common.probe" +use "../recipes/common.probe" before each test open and wait for the app @@ -22,7 +22,9 @@ test "navigate to keypad and back" @smoke navigate to keypad see "Activar" - go back to main menu + go back + wait 1 seconds + see "Home" test "navigate to lector and back" @smoke From c45d0d1395d633fdfc288750414739c9208109c2 Mon Sep 17 00:00:00 2001 From: Patrick Bertsch Date: Sun, 25 Oct 2026 19:53:35 -0600 Subject: [PATCH 3/8] fix(probe-convert): Robot recipes and their calls agree in lower case The lexer writes every call word in lower case, and the runner matches a recipe name exactly. A recipe header with capital letters can therefore never be called, and the Robot converter wrote its keyword recipes in the Robot spelling (`Login As User`), so no converted call matched. The converter now writes keyword recipes in lower case, and a call to a keyword the file defines uses the same spelling, whatever case the body wrote. The checkout golden is regenerated. A new test checks that the converted example defines and calls its recipes in lower case. A follow-up will normalise recipe headers in the runner, so capitalised headers become callable. --- CHANGELOG.md | 1 + tools/probe-convert/convert/robot/robot.go | 45 ++++++++++++++++++- .../probe-convert/convert/robot/robot_test.go | 25 ++++++++++- .../testdata/golden/robot/checkout.probe | 14 +++--- 4 files changed, 75 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 641a56c6..a3906dfb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,6 +36,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **VS Code: the extension compiles again with TypeScript 7 and @types/node 26.** The tsconfig names the Node types, which TypeScript 7 no longer loads by default. Without it, `npm test` failed to compile after those two upgrades. - **CI: the Dependabot auto-merge job adds approved pull requests to the merge queue again.** It passed `--squash` with `--auto`, so the queue ignored the request, nothing was queued, and the run still reported success. The job now passes no merge method and logs a warning for each pull request it cannot queue. - **CI: the Studio Go tests and the frontend and VS Code test suites run on every change to `studio/` or `vscode/`.** They ran in no workflow before, and only the browser suite covered Studio. The frontend suites run on Node 26. +- **probe-convert writes Robot keyword recipes in lower case, and its calls match them.** The recipe kept its Robot spelling (`Login As User`), while the lexer writes every call word in lower case, so no converted call matched its recipe. The checkout golden is regenerated, and a test checks the names. A recipe name that keeps capital letters still cannot be called from a test; that is tracked separately. - **CI: the CLI test package runs in seconds, not minutes.** The hosted-upload failure test waited out the 5-minute retry window; it now uses a short window and checks that the failure is still only a warning. - **`probe test --device` finds the Android application ID for a camelCase `project.app`.** When the bundle ID as written is not installed on an Android device and its snake_case form is (`com.example.myApp` → `com.example.my_app`), the single-device run uses the snake_case ID, as `--devices` and parallel runs already did. Before, the restart relaunched the ID as written and failed. A run keeps the ID as written when that ID is installed, and when neither form is, so a project that runs today is unchanged. `--devices` and parallel runs still convert without this check. - **Composite runs write the platform and OS version into each recording manifest.** `probe test --composite-device` now passes each device's platform and OS version to its recorder, as single-device runs already do, so `manifest.json` is no longer empty for those devices. Devices the CLI cannot read a platform for (a WiFi composite device) still write empty values. The recording ships with `probe test --record`, which is not released yet. diff --git a/tools/probe-convert/convert/robot/robot.go b/tools/probe-convert/convert/robot/robot.go index 2075d0a6..dde7d07f 100644 --- a/tools/probe-convert/convert/robot/robot.go +++ b/tools/probe-convert/convert/robot/robot.go @@ -40,6 +40,7 @@ func convertRobot(src, path string) (string, []convert.Warning) { w := convert.NewProbeWriter() var warnings []convert.Warning lines := strings.Split(src, "\n") + keywords := collectKeywordNames(lines) w.Comment("Converted from Robot Framework") w.BlankLine() @@ -127,7 +128,7 @@ func convertRobot(src, path string) (string, []convert.Warning) { if currentSection == sectionTestCases { w.Test(name) } else { - w.Recipe(name, nil) + w.Recipe(strings.ToLower(name), nil) } continue } @@ -194,7 +195,12 @@ func convertRobot(src, path string) (string, []convert.Warning) { } else { // Unknown keyword — emit as recipe call or TODO. if isLikelyKeywordCall(keyword) { - call := substituteVars(keyword) + call := strings.ToLower(substituteVars(keyword)) + if name, ok := keywords[robotKeywordKey(call)]; ok { + // The lexer writes a call in lower case, and probe matches recipe names case-sensitively, so the + // call uses the lower-case spelling the recipe is written under. + call = name + } if len(args) > 0 { call += " with " + strings.Join(quoteArgs(args), " and ") } @@ -214,6 +220,41 @@ func convertRobot(src, path string) (string, []convert.Warning) { return w.String(), warnings } +// robotKeywordKey is how Robot Framework matches keyword names: case, spaces and underscores do not count. +func robotKeywordKey(name string) string { + return strings.ToLower(strings.NewReplacer(" ", "", "_", "").Replace(strings.TrimSpace(name))) +} + +// collectKeywordNames returns the keywords the file defines, keyed by robotKeywordKey, each mapped to the name its recipe +// is written under. A call writes the keyword as its body spells it, so it resolves through this map. The sections are +// read the way convertRobot reads them, so a header the converter ignores does not end the keywords section. +func collectKeywordNames(lines []string) map[string]string { + names := map[string]string{} + inKeywords := false + for _, rawLine := range lines { + trimmed := strings.TrimSpace(rawLine) + if trimmed == "" || strings.HasPrefix(trimmed, "#") { + continue + } + if strings.HasPrefix(trimmed, "***") { + upper := strings.ToUpper(trimmed) + switch { + case strings.Contains(upper, "SETTING"), strings.Contains(upper, "VARIABLE"), + strings.Contains(upper, "TEST CASE"), strings.Contains(upper, "TASK"): + inKeywords = false + case strings.Contains(upper, "KEYWORD"): + inKeywords = true + } + continue + } + if inKeywords && rawLine[0] != ' ' && rawLine[0] != '\t' { + name := substituteVars(trimmed) + names[robotKeywordKey(name)] = strings.ToLower(name) + } + } + return names +} + // splitRobotCells splits a Robot Framework line by 2+ spaces or tab. func splitRobotCells(line string) []string { // Robot uses 2+ spaces or \t as cell separators. diff --git a/tools/probe-convert/convert/robot/robot_test.go b/tools/probe-convert/convert/robot/robot_test.go index 914b3beb..7bb869aa 100644 --- a/tools/probe-convert/convert/robot/robot_test.go +++ b/tools/probe-convert/convert/robot/robot_test.go @@ -1,6 +1,7 @@ package robot import ( + "os" "strings" "testing" ) @@ -45,7 +46,7 @@ Login With Credentials t.Fatalf("convert: %v", err) } - assertContains(t, result.ProbeCode, `recipe "Login With Credentials"`) + assertContains(t, result.ProbeCode, `recipe "login with credentials"`) assertContains(t, result.ProbeCode, "tap on #emailField") assertContains(t, result.ProbeCode, "") assertContains(t, result.ProbeCode, "") @@ -165,6 +166,28 @@ XPath Fallback assertContains(t, result.ProbeCode, "LinearLayout") } +// The lexer writes every call in lower case, and probe matches a call only to a recipe with exactly that name. So the +// converter writes recipes in lower case too, and the calls in the converted checkout example name them that way. +func TestConvert_RecipesAndCallsAgreeInCase(t *testing.T) { + src, err := os.ReadFile("../../examples/robot/checkout.robot") + if err != nil { + t.Fatal(err) + } + result, err := New().Convert(src, "checkout.robot") + if err != nil { + t.Fatalf("convert: %v", err) + } + assertContains(t, result.ProbeCode, `recipe "login as user"`) + assertContains(t, result.ProbeCode, `recipe "add item to cart"`) + assertContains(t, result.ProbeCode, `login as user with "user@test.com" and "pass123"`) + assertContains(t, result.ProbeCode, `add item to cart with "Running Shoes"`) + for _, stale := range []string{`recipe "Login As User"`, `Login As User with`, `Add Item To Cart with`} { + if strings.Contains(result.ProbeCode, stale) { + t.Errorf("output still writes %q, which no call can match\n%s", stale, result.ProbeCode) + } + } +} + func assertContains(t *testing.T, haystack, needle string) { t.Helper() if !strings.Contains(haystack, needle) { diff --git a/tools/probe-convert/testdata/golden/robot/checkout.probe b/tools/probe-convert/testdata/golden/robot/checkout.probe index 03724d63..51bca221 100644 --- a/tools/probe-convert/testdata/golden/robot/checkout.probe +++ b/tools/probe-convert/testdata/golden/robot/checkout.probe @@ -7,7 +7,7 @@ after each close the app -recipe "Login As User" +recipe "login as user" # arguments: email, password tap on "Sign In" type "" into #emailField @@ -15,7 +15,7 @@ recipe "Login As User" tap on #loginButton wait until "Dashboard" appears -recipe "Add Item To Cart" +recipe "add item to cart" # arguments: item_name tap on "Shop" tap on "" @@ -24,15 +24,15 @@ recipe "Add Item To Cart" test "Add Product To Cart" @cart @smoke - Login As User with "user@test.com" and "pass123" - Add Item To Cart with "Running Shoes" + login as user with "user@test.com" and "pass123" + add item to cart with "Running Shoes" see "Cart (1)" take a screenshot test "Remove Product From Cart" @cart - Login As User with "user@test.com" and "pass123" - Add Item To Cart with "Running Shoes" + login as user with "user@test.com" and "pass123" + add item to cart with "Running Shoes" tap on "Cart" long press on #cartItem1 tap on "Remove" @@ -40,7 +40,7 @@ test "Remove Product From Cart" test "Check Product Visibility" @ui - Login As User with "user@test.com" and "pass123" + login as user with "user@test.com" and "pass123" tap on "Shop" wait until #productList appears see #featuredBanner From d89a10790df2c5558ac6493f41969ca8aa5c0d1a Mon Sep 17 00:00:00 2001 From: Patrick Bertsch Date: Sun, 25 Oct 2026 19:34:18 -0600 Subject: [PATCH 4/8] feat(lint): explain a go back that is followed by words on its line go back takes no arguments, so the words after it on the same line are read as a separate step. Lint reported those words as an unknown step, which does not say why. A call on the line of a go back step now gets the reason in its message. --- internal/runner/lint.go | 13 +++++++++++-- internal/runner/lint_test.go | 10 ++++++++++ 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/internal/runner/lint.go b/internal/runner/lint.go index cb443447..993a463c 100644 --- a/internal/runner/lint.go +++ b/internal/runner/lint.go @@ -88,6 +88,8 @@ func LintReferences(cfg *config.Config, path string) ([]LintFinding, error) { } defined := map[string]bool{} + // Lines where a go back step starts. A call on such a line was read apart from the step (go back takes no arguments). + goBackLines := map[int]bool{} for _, name := range runtimeVars { defined[name] = true } @@ -112,7 +114,12 @@ func LintReferences(cfg *config.Config, path string) ([]LintFinding, error) { walkSteps(rec.Body, func(s parser.Step) { markStored(s, defined) }) } for _, body := range own { - walkSteps(body, func(s parser.Step) { markStored(s, defined) }) + walkSteps(body, func(s parser.Step) { + markStored(s, defined) + if a, ok := s.(parser.ActionStep); ok && a.Verb == parser.VerbGoBack { + goBackLines[a.Line] = true + } + }) } for _, body := range own { @@ -121,7 +128,9 @@ func LintReferences(cfg *config.Config, path string) ([]LintFinding, error) { if rc, ok := s.(parser.RecipeCall); ok { if _, _, _, found := resolveCall(r.recipes, rc); !found { msg := fmt.Sprintf("unknown step %q: not a built-in step and no recipe with that name is defined (check the use statements and the recipes folder)", rc.Name) - if strings.HasPrefix(rc.Name, "use ") || rc.Name == "use" { + if goBackLines[rc.Line] { + msg = fmt.Sprintf("go back takes no arguments: the words after it on line %d are read as a separate step %q", rc.Line, rc.Name) + } else if strings.HasPrefix(rc.Name, "use ") || rc.Name == "use" { // The parser reads use only at the top of a file; inside a test it is an ordinary call. msg = "a use statement must come before the first test; inside a test it is not read" } diff --git a/internal/runner/lint_test.go b/internal/runner/lint_test.go index 6a5a1e77..fdc187a7 100644 --- a/internal/runner/lint_test.go +++ b/internal/runner/lint_test.go @@ -64,6 +64,16 @@ func TestLintReferencesExplainsAUseInsideATest(t *testing.T) { requireFinding(t, findings, 2, false, "must come before the first test") } +// go back takes no arguments, so the words after it on its line are read as a separate step. The message says so, and +// the call is not reported as an unknown step on its own. +func TestLintReferencesExplainsWordsAfterGoBack(t *testing.T) { + findings := lintSource(t, "test \"nav\"\n go back to main menu\n") + requireFinding(t, findings, 2, false, "go back takes no arguments") + if len(findings) != 1 { + t.Errorf("expected only the go back explanation, got %+v", findings) + } +} + func TestLintReferencesAcceptsRecipesDefinedInTheFile(t *testing.T) { src := "recipe \"sign in\" (email)\n type \"\" into the \"Email\" field\n\ntest \"login\"\n sign in \"a@b.c\"\n" requireNoFindings(t, lintSource(t, src)) From c7bc2e12c9512cb7c8f022d4126ca5a072612c08 Mon Sep 17 00:00:00 2001 From: Patrick Bertsch Date: Sun, 25 Oct 2026 19:34:22 -0600 Subject: [PATCH 5/8] docs: go back takes no arguments; the editor's lint scope; the Studio diagram names the parser The grammar and the syntax page say that go back takes no arguments, and that words after it on the same line are a separate step. The grammar's production is unchanged; only its comment is new. The Studio page now says what the editor's lint marks (parse errors and empty tests), and that probe lint also checks references, selectors and placeholders. Its architecture diagram listed an internal/lint package that does not exist; the Go backend imports internal/parser. The generated ProbeScript dictionary is regenerated, because its source hash covers the grammar page. --- studio/frontend/src/probescript/dictionary.generated.ts | 2 +- website/src/content/docs/probescript/grammar.md | 2 +- website/src/content/docs/probescript/syntax.md | 2 +- website/src/content/docs/tools/cli-reference.md | 2 +- website/src/content/docs/tools/studio.md | 4 ++-- 5 files changed, 6 insertions(+), 6 deletions(-) diff --git a/studio/frontend/src/probescript/dictionary.generated.ts b/studio/frontend/src/probescript/dictionary.generated.ts index a0a5c1c0..e9b36d20 100644 --- a/studio/frontend/src/probescript/dictionary.generated.ts +++ b/studio/frontend/src/probescript/dictionary.generated.ts @@ -2,7 +2,7 @@ // Do not edit by hand. After a change under website/src/content/docs/probescript/, run // `npm run gen:probescript` in studio/frontend. -export const SOURCE_SHA256 = "780a6ca36ec1c85cd6412d7d657766c71b4829ce4339f209995dc8d9212be7fe"; +export const SOURCE_SHA256 = "4270861a86aee40bd8fb75fcbd86a92e53cd2519760dc2d69a2e3d8cd39526f8"; export const COMMANDS = [ { diff --git a/website/src/content/docs/probescript/grammar.md b/website/src/content/docs/probescript/grammar.md index 7009ebd2..035011be 100644 --- a/website/src/content/docs/probescript/grammar.md +++ b/website/src/content/docs/probescript/grammar.md @@ -759,7 +759,7 @@ close-step = "close" , fillers , [ "app" | WORD | STRING ] ; restart-step = "restart" , fillers , [ "app" ] ; kill-step = "kill" , fillers , [ "app" ] ; clear-data-step = "clear app data" ; -go-back-step = "go back" ; +go-back-step = "go back" ; (* takes no arguments: words after it on the same line are a separate step *) press-enter-step = "press enter" ; (* the keyboard action key on the focused text field *) shake-step = "shake" ; pause-step = "pause" ; diff --git a/website/src/content/docs/probescript/syntax.md b/website/src/content/docs/probescript/syntax.md index 4c669f08..908d4435 100644 --- a/website/src/content/docs/probescript/syntax.md +++ b/website/src/content/docs/probescript/syntax.md @@ -251,7 +251,7 @@ compare screenshot "baseline" # compare against visual regression baseline dump tree # dump widget tree for debugging save logs # save app logs press enter # keyboard action key on the focused field (Maestro pressKey: Enter) -go back # pop the current route (no-op + warning at the root; use "close the app" to exit) +go back # pop the current route (no-op + warning at the root; use "close the app" to exit); takes no arguments, so words after it on the same line are a separate step rotate landscape # rotate device shake # simulate device shake gesture log "checkpoint reached" # print to test output diff --git a/website/src/content/docs/tools/cli-reference.md b/website/src/content/docs/tools/cli-reference.md index 58b59e7e..98728626 100644 --- a/website/src/content/docs/tools/cli-reference.md +++ b/website/src/content/docs/tools/cli-reference.md @@ -154,7 +154,7 @@ probe lint tests/smoke/login.probe Lint checks the references a run would resolve, without running a step, and fails the command (exit status 1) for an error: -- A step or recipe call that matches no built-in step and no recipe in the file, its `use` statements or the recipes folder. This is the check `probe test --dry-run` makes. A `use` statement inside a test is reported too, because a file reads `use` only before its first test. +- A step or recipe call that matches no built-in step and no recipe in the file, its `use` statements or the recipes folder. This is the check `probe test --dry-run` makes. A `use` statement inside a test is reported too, because a file reads `use` only before its first test. A `go back` followed by words on its line is reported with that reason, because `go back` takes no arguments. - A selector that can never match: an ordinal selector without a number and text, an `#id` without a name, a `below`, `above`, `left of` or `right of` selector without anchor text, or an `in` with an empty container. It warns, without failing, when: diff --git a/website/src/content/docs/tools/studio.md b/website/src/content/docs/tools/studio.md index 21c045ad..e3342481 100644 --- a/website/src/content/docs/tools/studio.md +++ b/website/src/content/docs/tools/studio.md @@ -9,7 +9,7 @@ FlutterProbe Studio is a standalone desktop application for designing, recording ## What you get -- **Monaco editor** with ProbeScript syntax highlighting, live lint markers, and the same TextMate grammar that powers the VS Code extension. Completions, hover help and snippets cover every command in the [ProbeScript dictionary](/probescript/dictionary/) and every reserved word in the [grammar](/probescript/grammar/); they are generated from those pages, so the editor and the docs agree. +- **Monaco editor** with ProbeScript syntax highlighting, live lint markers (parse errors and empty tests; `probe lint` also checks step and recipe references, selectors and placeholders), and the same TextMate grammar that powers the VS Code extension. Completions, hover help and snippets cover every command in the [ProbeScript dictionary](/probescript/dictionary/) and every reserved word in the [grammar](/probescript/grammar/); they are generated from those pages, so the editor and the docs agree. - **Workspace folder picker** — point Studio at any project root with a `tests/` folder. Choice persists across launches. - **Device picker** that lists booted simulators, online emulators, and physical iOS/Android devices side by side, with a `physical` tag and a clear hint when nothing is connectable. - **Physical device support** — the same `internal/device` code the `probe` CLI uses: `iproxy` tunneling for physical iOS over USB, `adb forward` for physical Android (same path as emulators), and mDNS auto-discovery (`_flutterprobe._tcp`) for WiFi-connected devices. See [Physical devices](#physical-devices) below. @@ -51,7 +51,7 @@ Studio's own UI is tested two ways. `make studio-e2e-web` runs the real frontend │ ┌─────────▼─────────────────────────────┐ │ │ │ Go backend (App struct) │ │ │ │ imports internal/runner │ │ -│ │ imports internal/lint │ │ +│ │ imports internal/parser │ │ │ │ imports internal/probelink │ │ │ │ imports internal/device │ │ │ └─────────┬─────────────────────────────┘ │ From da3511d3d43b05e227889322fc040f5392022a06 Mon Sep 17 00:00:00 2001 From: Patrick Bertsch Date: Mon, 26 Oct 2026 17:06:49 -0600 Subject: [PATCH 6/8] docs: show probe lint output in the changelog Add an example of an error and of a warning from probe lint to the changelog entry, with the exit status for each. --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a3906dfb..a4b53ed2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,7 +24,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **The MCP tools keep to the local tool surface.** `run_tests` and `run_script` refuse the upload flags (`--upload-report`, `--upload-local`, `--upload-dry-run`, and any other `--upload-` flag) with the message "hosted upload is not available through MCP". The MCP tool text no longer describes the upload flags. Use the CLI from a terminal for uploads. - **Building from source needs Go 1.26.10 or newer (was 1.26.1).** The `go` line in `go.mod`, `studio/go.mod`, `tools/probe-convert/go.mod` and `go.work` is now 1.26.10, and the install docs say so. With the default `GOTOOLCHAIN=auto`, an older Go 1.26 downloads 1.26.10 on its own; with `GOTOOLCHAIN=local` the build stops with a version error until Go is upgraded. - **Older evidence files removed.** The disclosure guard also scans `docs/evidence`. -- **`probe lint` fails on references a run cannot resolve.** A step or recipe call that resolves to nothing, or a selector that can never match, now makes the command exit with status 1, where it used to pass. A placeholder no value defines and a `use` file that does not exist are warnings and do not change the exit status. A repository whose CI runs `probe lint` may see new failures on files that already fail at run time; `probe test --dry-run` reports the same steps. Environment variables (`${NAME}`) are not checked. +- **`probe lint` fails on references a run cannot resolve.** A step or recipe call that resolves to nothing, or a selector that can never match, now makes the command exit with status 1, where it used to pass. A placeholder no value defines and a `use` file that does not exist are warnings and do not change the exit status. A repository whose CI runs `probe lint` may see new failures on files that already fail at run time; `probe test --dry-run` reports the same steps. Environment variables (`${NAME}`) are not checked. For example, a step `call send reset code` that names no recipe prints `line 7: unknown step "send reset code": not a built-in step and no recipe with that name is defined (check the use statements and the recipes folder)` and makes the command exit with status 1. A file whose only findings are an undefined placeholder, such as `line 5: is not defined: no example column, recipe parameter or stored value has that name`, or a missing `use` file, prints them with a warning mark and exits with status 0. ### Docs - **Recording evidence write-ups no longer name private apps.** The prose and commands in the evidence notes use neutral labels and placeholders. The raw run output that the notes cite is unchanged. From f74e26173d8bb7e37c6fd0ea625c7f70cea18667 Mon Sep 17 00:00:00 2001 From: Patrick Bertsch Date: Mon, 26 Oct 2026 17:41:50 -0600 Subject: [PATCH 7/8] test(lint): cover selector limits, read with ai variables, hook and composite bodies, and the report Adds tests for the checks that had none: an ordinal selector without a position or text, an id selector without a name, an empty in container, a placeholder defined by read with ai, unknown steps in hook bodies and in composite device blocks, the count and zero-state checks, the order of errors and warnings, and the failure mark with the error count. A temporary mutation check removed the composite walk, the read with ai definition and the zero-state rule; each removal fails the new tests. --- internal/cli/lint_references_test.go | 121 +++++++++++++++++++++++++++ internal/runner/lint_test.go | 56 +++++++++++++ 2 files changed, 177 insertions(+) diff --git a/internal/cli/lint_references_test.go b/internal/cli/lint_references_test.go index 9a6fe2a4..e0dd3c8e 100644 --- a/internal/cli/lint_references_test.go +++ b/internal/cli/lint_references_test.go @@ -1,9 +1,14 @@ package cli import ( + "bytes" + "io" "os" "path/filepath" + "strings" "testing" + + "github.com/alphawavesystems/flutter-probe/internal/parser" ) // probe lint fails on a reference a run cannot resolve, and only warns about a placeholder no value defines. @@ -24,3 +29,119 @@ func TestLintCommandFailsOnUnknownStepsButNotOnWarnings(t *testing.T) { t.Error("an unknown step must fail lint") } } + +// lintOutput runs probe lint on paths and returns what the command printed to standard output, and its error. The report +// is written with fmt.Print, so the test swaps os.Stdout for a pipe while the command runs. +func lintOutput(t *testing.T, paths ...string) (string, error) { + t.Helper() + r, w, err := os.Pipe() + if err != nil { + t.Fatal(err) + } + saved := os.Stdout + os.Stdout = w + captured := make(chan string) + go func() { + var b bytes.Buffer + _, _ = io.Copy(&b, r) + captured <- b.String() + }() + runErr := lintCmd.RunE(lintCmd, paths) + os.Stdout = saved + w.Close() + out := <-captured + r.Close() + return out, runErr +} + +// A file's errors print before its warnings, a file with errors gets the failure mark, and the command counts the files +// that have errors. +func TestLintCommandPrintsErrorsBeforeWarningsAndCountsFiles(t *testing.T) { + dir := t.TempDir() + mixed := filepath.Join(dir, "mixed.probe") + if err := os.WriteFile(mixed, []byte("test \"x\"\n type \"\" into the \"Email\" field\n frobnicate the cart\n"), 0o644); err != nil { + t.Fatal(err) + } + bad := filepath.Join(dir, "bad.probe") + if err := os.WriteFile(bad, []byte("test \"x\"\n frobnicate the cart\n"), 0o644); err != nil { + t.Fatal(err) + } + out, err := lintOutput(t, mixed, bad) + if err == nil || err.Error() != "2 file(s) have errors" { + t.Fatalf("err = %v, want \"2 file(s) have errors\"", err) + } + if !strings.Contains(out, "✗") { + t.Errorf("a file with errors needs the failure mark:\n%s", out) + } + errAt := strings.Index(out, "line 3: unknown step") + warnAt := strings.Index(out, "line 2: is not defined") + if errAt < 0 || warnAt < 0 || errAt > warnAt { + t.Errorf("errors must print before warnings:\n%s", out) + } +} + +// A warning alone does not fail the command, and the run ends with All files valid. A zero-state text check with no +// anchor is reported through the same path. +func TestLintCommandWarnsOnAZeroStateCheckAndStillPasses(t *testing.T) { + dir := t.TempDir() + zero := filepath.Join(dir, "zero.probe") + if err := os.WriteFile(zero, []byte("test \"x\"\n see \"No items\"\n"), 0o644); err != nil { + t.Fatal(err) + } + out, err := lintOutput(t, zero) + if err != nil { + t.Fatalf("a warning must not fail lint: %v", err) + } + if !strings.Contains(out, "⚠") || !strings.Contains(out, "matches any longer text") { + t.Errorf("want the warning mark and the zero-state warning:\n%s", out) + } + if !strings.Contains(out, "All files valid.") { + t.Errorf("a run with only warnings ends with All files valid.:\n%s", out) + } +} + +// lintProgram warns on a count or zero-state text check that a substring match passes by accident. It does not warn on an +// anchored check, a negated check, or a sentence that only starts with a zero-state word. +func TestLintProgramFlagsUnanchoredCountAndZeroStateChecks(t *testing.T) { + cases := []struct { + name, src string + warn bool + }{ + {"zero state", "test \"x\"\n see \"No items\"\n", true}, + {"count", "test \"x\"\n see \"3 items\"\n", true}, + {"anchored zero state", "test \"x\"\n see \"No items\" matching \"^No items$\"\n", false}, + {"negated count", "test \"x\"\n don't see \"3 items\"\n", false}, + {"longer sentence", "test \"x\"\n see \"Nothing is wrong with this page\"\n", false}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + prog, err := parser.ParseFile(c.src) + if err != nil { + t.Fatal(err) + } + warns := lintProgram(prog) + if got := len(warns) > 0; got != c.warn { + t.Errorf("lintProgram warned = %t, want %t: %q", got, c.warn, warns) + } + }) + } +} + +// A composite test's device blocks are linted: a bad step in one fails probe lint, and a valid block passes. +func TestLintCommandChecksCompositeDeviceBlocks(t *testing.T) { + dir := t.TempDir() + good := filepath.Join(dir, "good.probe") + if err := os.WriteFile(good, []byte("composite test \"login\"\n A:\n tap \"Login\"\n"), 0o644); err != nil { + t.Fatal(err) + } + bad := filepath.Join(dir, "bad.probe") + if err := os.WriteFile(bad, []byte("composite test \"login\"\n A:\n bogus thing\n"), 0o644); err != nil { + t.Fatal(err) + } + if _, err := lintOutput(t, good); err != nil { + t.Errorf("a valid composite test must pass lint: %v", err) + } + if _, err := lintOutput(t, bad); err == nil { + t.Error("an unknown step in a device block must fail lint") + } +} diff --git a/internal/runner/lint_test.go b/internal/runner/lint_test.go index fdc187a7..6cefbb7f 100644 --- a/internal/runner/lint_test.go +++ b/internal/runner/lint_test.go @@ -182,6 +182,62 @@ func TestWalkStepsVisitsNestedBodies(t *testing.T) { } } +// An ordinal selector needs a position of at least 1 and the text it counts. Checked on the selector directly, so the rule +// holds whatever syntax reaches it. +func TestSelectorProblemRejectsAnOrdinalWithoutPositionOrText(t *testing.T) { + bad := map[string]parser.Selector{ + "position zero": {Kind: parser.SelectorOrdinal, Text: "Buy", Ordinal: 0}, + "no text": {Kind: parser.SelectorOrdinal, Text: "", Ordinal: 2}, + } + for name, sel := range bad { + t.Run(name, func(t *testing.T) { + if selectorProblem(sel) == "" { + t.Errorf("ordinal selector %+v: want a problem", sel) + } + }) + } + if problem := selectorProblem(parser.Selector{Kind: parser.SelectorOrdinal, Text: "Buy", Ordinal: 2}); problem != "" { + t.Errorf("a working ordinal selector: unexpected problem %q", problem) + } +} + +// An #id selector needs a name after the #. A # that is not followed by a letter or underscore starts a comment in the +// lexer, so a bare # never reaches this check from a file; it is checked on the selector directly. +func TestSelectorProblemRejectsAnIDWithoutAName(t *testing.T) { + for _, text := range []string{"", "#"} { + if selectorProblem(parser.Selector{Kind: parser.SelectorID, Text: text}) == "" { + t.Errorf("id selector with text %q: want a problem", text) + } + } + if problem := selectorProblem(parser.Selector{Kind: parser.SelectorID, Text: "#buy_button"}); problem != "" { + t.Errorf("a named id selector: unexpected problem %q", problem) + } +} + +func TestLintReferencesFlagsAnEmptyContainer(t *testing.T) { + findings := lintSource(t, "test \"x\"\n tap \"Buy\" in \"\"\n") + requireFinding(t, findings, 2, false, "the container after \"in\" is empty") +} + +// read ... with ai into names a variable, so a placeholder that uses it is defined. +func TestLintReferencesAcceptsAPlaceholderFromReadWithAI(t *testing.T) { + src := "test \"x\"\n read \"the order total\" with ai into total\n type \"\" into the \"Amount\" field\n" + requireNoFindings(t, lintSource(t, src)) +} + +// Hook bodies are checked like test bodies. The dry run checks only test bodies, so lint is what reports a bad step in a +// hook before a run does. +func TestLintReferencesChecksHookBodies(t *testing.T) { + src := "before each test\n bogus thing\n\ntest \"x\"\n tap \"Go\"\n" + requireFinding(t, lintSource(t, src), 2, false, "unknown step") +} + +// The device blocks of a composite test are checked too, and a valid block passes. The dry run skips composite tests. +func TestLintReferencesChecksCompositeDeviceBlocks(t *testing.T) { + requireNoFindings(t, lintSource(t, "composite test \"login\"\n A:\n tap \"Login\"\n")) + requireFinding(t, lintSource(t, "composite test \"login\"\n A:\n bogus thing\n"), 3, false, "unknown step") +} + func TestLintReferencesReportsAParseErrorWithoutFindings(t *testing.T) { dir := t.TempDir() path := filepath.Join(dir, "flows.probe") From 919dc65a261081f59ecf86c5f3f837d1f9d48baf Mon Sep 17 00:00:00 2001 From: Patrick Bertsch Date: Mon, 26 Oct 2026 17:41:55 -0600 Subject: [PATCH 8/8] docs(lint): state the hook and composite scope, the known limits and the report example Lint checks hook bodies and composite device blocks, which the dry run skips. The changelog example is the command's output for two sample files, with the colour codes left out. The CLI reference lists the four known limits of the references check, and the MCP lint description says what lint checks beyond the dry run. --- CHANGELOG.md | 21 +++++++++++++++++-- internal/mcp/server.go | 2 +- .../src/content/docs/tools/cli-reference.md | 9 ++++++++ 3 files changed, 29 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a4b53ed2..15bc84c7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,7 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added - **MCP: `start_tests`, `test_status`, `test_result` and `cancel_tests` run tests in the background.** `run_tests` blocks until `probe test` exits, so a multi-device run (`--parallel`, `composite_devices`, several devices) could outlast the MCP client's tool timeout and be reported as failed while it kept running. `start_tests` takes the same arguments, returns a `job_id` at once and runs the command in its own process group; `test_status` returns the state, elapsed time and the last lines of output; `test_result` returns the retained output (the last 1 MB) and the exit code; `cancel_tests` sends SIGTERM to the run and kills it after 5 seconds. Jobs live in the `probe-mcp` process and are killed when it exits. The hosted-upload flags are refused here exactly as in `run_tests`. `run_tests` is unchanged. The MCP server now has 40 tools. -- **`probe lint` checks the references a run would resolve.** Lint reports each step and recipe call that matches no built-in step and no recipe in the file, its `use` statements or the recipes folder (the check `probe test --dry-run` makes), each selector that can never match, and each `` placeholder that no example column, recipe parameter or stored value defines. It runs without a device and gives the line of each finding. +- **`probe lint` checks the references a run would resolve.** Lint reports each step and recipe call that matches no built-in step and no recipe in the file, its `use` statements or the recipes folder (the check `probe test --dry-run` makes), each selector that can never match, and each `` placeholder that no example column, recipe parameter or stored value defines. It runs without a device, gives the line of each finding, and checks hook bodies and the device blocks of composite tests too. - **Recordings show the element each step acted on.** A step that taps, double taps, long presses, types, clears, toggles, or swipes or scrolls with a selector records its selector, and its bounds and tap point when the element is on screen as the step starts. The viewer shows them under "Element". Text typed into a field is never recorded, and a failed lookup is an advisory warning that does not change the step. - **`probe test --record` writes a step-by-step recording of each test.** Each test attempt includes source files and lines, screenshots before and after steps, widget trees after steps, and the app's `dart:io` HTTP exchanges with credential redaction, plus an offline viewer. Bundles stay under `reports_folder/recordings` or `--record-dir`; nothing uploads, and recording warnings do not change test results or exit codes. Native SDK and WebView traffic and screenshot redaction rules are not yet recorded; screenshots are not redacted and widget trees are not captured before steps. Keep recordings out of version control because they can contain screens and request or response bodies. - **Recording manifests name the platform and OS version.** `manifest.json` in each recording bundle now carries the run's `platform` (`ios` or `android`) and `os_version`, the same values the report shows, instead of leaving both empty. @@ -24,7 +24,24 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **The MCP tools keep to the local tool surface.** `run_tests` and `run_script` refuse the upload flags (`--upload-report`, `--upload-local`, `--upload-dry-run`, and any other `--upload-` flag) with the message "hosted upload is not available through MCP". The MCP tool text no longer describes the upload flags. Use the CLI from a terminal for uploads. - **Building from source needs Go 1.26.10 or newer (was 1.26.1).** The `go` line in `go.mod`, `studio/go.mod`, `tools/probe-convert/go.mod` and `go.work` is now 1.26.10, and the install docs say so. With the default `GOTOOLCHAIN=auto`, an older Go 1.26 downloads 1.26.10 on its own; with `GOTOOLCHAIN=local` the build stops with a version error until Go is upgraded. - **Older evidence files removed.** The disclosure guard also scans `docs/evidence`. -- **`probe lint` fails on references a run cannot resolve.** A step or recipe call that resolves to nothing, or a selector that can never match, now makes the command exit with status 1, where it used to pass. A placeholder no value defines and a `use` file that does not exist are warnings and do not change the exit status. A repository whose CI runs `probe lint` may see new failures on files that already fail at run time; `probe test --dry-run` reports the same steps. Environment variables (`${NAME}`) are not checked. For example, a step `call send reset code` that names no recipe prints `line 7: unknown step "send reset code": not a built-in step and no recipe with that name is defined (check the use statements and the recipes folder)` and makes the command exit with status 1. A file whose only findings are an undefined placeholder, such as `line 5: is not defined: no example column, recipe parameter or stored value has that name`, or a missing `use` file, prints them with a warning mark and exits with status 0. +- **`probe lint` fails on references a run cannot resolve.** A step or recipe call that resolves to nothing, or a selector that can never match, now makes the command exit with status 1, where it used to pass. A placeholder no value defines and a `use` file that does not exist are warnings and do not change the exit status. Lint also checks hook bodies and the device blocks of composite tests. `probe test --dry-run` checks only test bodies, so a bad step in those places fails at run time today, and a repository whose CI runs `probe lint` may see new failures there. Environment variables (`${NAME}`) are not checked, and the other limits are listed in the CLI reference. Within a file, errors print before warnings. Example output for a file with an error (colour codes omitted; the last line goes to stderr): + + ```text + ✗ x.probe + line 7: unknown step "send reset code": not a built-in step and no recipe with that name is defined (check the use statements and the recipes folder) + line 8: is not defined: no example column, recipe parameter or stored value has that name + + Error: 1 file(s) have errors + ``` + + A file with only warnings, such as a placeholder no value defines or a missing `use` file, prints them under a warning mark and still exits with status 0. The run then ends with `All files valid.`: + + ```text + ⚠ y.probe + line 3: is not defined: no example column, recipe parameter or stored value has that name + + All files valid. + ``` ### Docs - **Recording evidence write-ups no longer name private apps.** The prose and commands in the evidence notes use neutral labels and placeholders. The raw run output that the notes cite is unchanged. diff --git a/internal/mcp/server.go b/internal/mcp/server.go index aca46283..bbd3939b 100644 --- a/internal/mcp/server.go +++ b/internal/mcp/server.go @@ -427,7 +427,7 @@ Takes the same arguments as run_tests (see its description for the flags). Then }, { Name: "lint", - Description: "Validate .probe files without running them. Fails on a step or recipe call that resolves to nothing and on a selector that can never match; warns on unresolved placeholders, unknown use files and count or zero-state text checks.", + Description: "Validate .probe files without running them. Fails on a step or recipe call that resolves to nothing and on a selector that can never match; warns on unresolved placeholders, unknown use files and count or zero-state text checks. It also checks hook and composite test bodies, which `probe test --dry-run` does not.", InputSchema: mcpSchema{ Type: "object", Properties: map[string]mcpProp{ diff --git a/website/src/content/docs/tools/cli-reference.md b/website/src/content/docs/tools/cli-reference.md index 98728626..b71dcba1 100644 --- a/website/src/content/docs/tools/cli-reference.md +++ b/website/src/content/docs/tools/cli-reference.md @@ -163,8 +163,17 @@ It warns, without failing, when: - A `use` names a file that does not exist. A run only needs that file when a call uses one of its recipes, and such a call is reported as an unknown step. - A `see` or `wait until ... appears` whose text is a count or a zero state (`see "0 items"`, `wait until "No items" appears`) and has no `matching` pattern, because a substring match also passes on "10 items". Anchor the pattern for an exact check: `see "0 items" matching "^0 items$"`. A `matching` pattern anchored at both ends (`^...$`) silences the warning; any other pattern is still a substring match and is still flagged. +Lint also checks the bodies of hooks (`before each test`, `before all` and their `after` forms) and the device blocks (`A:`, `B:`) of composite tests. `probe test --dry-run` checks only test bodies, so a bad step in a hook or a composite test fails at run time. A CI job that runs `probe lint` on composite files may see new errors there. + Environment variables (`${NAME}`) are not checked, because lint does not know the environment that will run the tests. +Known limits. A follow-up will address these: + +- Placeholders, recipe parameters and stored names are collected across the whole file and every recipe it can call, not per test, so a name defined in one test counts for the others. +- The body of a recipe that comes from a `use` file is not checked. That file's recipes only define their names here. +- A `use` line inside a used file is not checked for a missing file. Only the `use` lines of the file given to lint are. +- A `#` followed by a space starts a comment, as it does elsewhere in ProbeScript, so `tap #` reads as `tap` with no target. Lint does not report it. + ## probe record Record user interactions and generate ProbeScript.