diff --git a/CHANGELOG.md b/CHANGELOG.md index 32c2bf4a..15bc84c7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added - **MCP: `start_tests`, `test_status`, `test_result` and `cancel_tests` run tests in the background.** `run_tests` blocks until `probe test` exits, so a multi-device run (`--parallel`, `composite_devices`, several devices) could outlast the MCP client's tool timeout and be reported as failed while it kept running. `start_tests` takes the same arguments, returns a `job_id` at once and runs the command in its own process group; `test_status` returns the state, elapsed time and the last lines of output; `test_result` returns the retained output (the last 1 MB) and the exit code; `cancel_tests` sends SIGTERM to the run and kills it after 5 seconds. Jobs live in the `probe-mcp` process and are killed when it exits. The hosted-upload flags are refused here exactly as in `run_tests`. `run_tests` is unchanged. The MCP server now has 40 tools. +- **`probe lint` checks the references a run would resolve.** Lint reports each step and recipe call that matches no built-in step and no recipe in the file, its `use` statements or the recipes folder (the check `probe test --dry-run` makes), each selector that can never match, and each `` placeholder that no example column, recipe parameter or stored value defines. It runs without a device, gives the line of each finding, and checks hook bodies and the device blocks of composite tests too. - **Recordings show the element each step acted on.** A step that taps, double taps, long presses, types, clears, toggles, or swipes or scrolls with a selector records its selector, and its bounds and tap point when the element is on screen as the step starts. The viewer shows them under "Element". Text typed into a field is never recorded, and a failed lookup is an advisory warning that does not change the step. - **`probe test --record` writes a step-by-step recording of each test.** Each test attempt includes source files and lines, screenshots before and after steps, widget trees after steps, and the app's `dart:io` HTTP exchanges with credential redaction, plus an offline viewer. Bundles stay under `reports_folder/recordings` or `--record-dir`; nothing uploads, and recording warnings do not change test results or exit codes. Native SDK and WebView traffic and screenshot redaction rules are not yet recorded; screenshots are not redacted and widget trees are not captured before steps. Keep recordings out of version control because they can contain screens and request or response bodies. - **Recording manifests name the platform and OS version.** `manifest.json` in each recording bundle now carries the run's `platform` (`ios` or `android`) and `os_version`, the same values the report shows, instead of leaving both empty. @@ -23,6 +24,24 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **The MCP tools keep to the local tool surface.** `run_tests` and `run_script` refuse the upload flags (`--upload-report`, `--upload-local`, `--upload-dry-run`, and any other `--upload-` flag) with the message "hosted upload is not available through MCP". The MCP tool text no longer describes the upload flags. Use the CLI from a terminal for uploads. - **Building from source needs Go 1.26.10 or newer (was 1.26.1).** The `go` line in `go.mod`, `studio/go.mod`, `tools/probe-convert/go.mod` and `go.work` is now 1.26.10, and the install docs say so. With the default `GOTOOLCHAIN=auto`, an older Go 1.26 downloads 1.26.10 on its own; with `GOTOOLCHAIN=local` the build stops with a version error until Go is upgraded. - **Older evidence files removed.** The disclosure guard also scans `docs/evidence`. +- **`probe lint` fails on references a run cannot resolve.** A step or recipe call that resolves to nothing, or a selector that can never match, now makes the command exit with status 1, where it used to pass. A placeholder no value defines and a `use` file that does not exist are warnings and do not change the exit status. Lint also checks hook bodies and the device blocks of composite tests. `probe test --dry-run` checks only test bodies, so a bad step in those places fails at run time today, and a repository whose CI runs `probe lint` may see new failures there. Environment variables (`${NAME}`) are not checked, and the other limits are listed in the CLI reference. Within a file, errors print before warnings. Example output for a file with an error (colour codes omitted; the last line goes to stderr): + + ```text + ✗ x.probe + line 7: unknown step "send reset code": not a built-in step and no recipe with that name is defined (check the use statements and the recipes folder) + line 8: is not defined: no example column, recipe parameter or stored value has that name + + Error: 1 file(s) have errors + ``` + + A file with only warnings, such as a placeholder no value defines or a missing `use` file, prints them under a warning mark and still exits with status 0. The run then ends with `All files valid.`: + + ```text + ⚠ y.probe + line 3: is not defined: no example column, recipe parameter or stored value has that name + + All files valid. + ``` ### Docs - **Recording evidence write-ups no longer name private apps.** The prose and commands in the evidence notes use neutral labels and placeholders. The raw run output that the notes cite is unchanged. @@ -34,6 +53,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **VS Code: the extension compiles again with TypeScript 7 and @types/node 26.** The tsconfig names the Node types, which TypeScript 7 no longer loads by default. Without it, `npm test` failed to compile after those two upgrades. - **CI: the Dependabot auto-merge job adds approved pull requests to the merge queue again.** It passed `--squash` with `--auto`, so the queue ignored the request, nothing was queued, and the run still reported success. The job now passes no merge method and logs a warning for each pull request it cannot queue. - **CI: the Studio Go tests and the frontend and VS Code test suites run on every change to `studio/` or `vscode/`.** They ran in no workflow before, and only the browser suite covered Studio. The frontend suites run on Node 26. +- **probe-convert writes Robot keyword recipes in lower case, and its calls match them.** The recipe kept its Robot spelling (`Login As User`), while the lexer writes every call word in lower case, so no converted call matched its recipe. The checkout golden is regenerated, and a test checks the names. A recipe name that keeps capital letters still cannot be called from a test; that is tracked separately. - **CI: the CLI test package runs in seconds, not minutes.** The hosted-upload failure test waited out the 5-minute retry window; it now uses a short window and checks that the failure is still only a warning. - **`probe test --device` finds the Android application ID for a camelCase `project.app`.** When the bundle ID as written is not installed on an Android device and its snake_case form is (`com.example.myApp` → `com.example.my_app`), the single-device run uses the snake_case ID, as `--devices` and parallel runs already did. Before, the restart relaunched the ID as written and failed. A run keeps the ID as written when that ID is installed, and when neither form is, so a project that runs today is unchanged. `--devices` and parallel runs still convert without this check. - **Composite runs write the platform and OS version into each recording manifest.** `probe test --composite-device` now passes each device's platform and OS version to its recorder, as single-device runs already do, so `manifest.json` is no longer empty for those devices. Devices the CLI cannot read a platform for (a WiFi composite device) still write empty values. The recording ships with `probe test --record`, which is not released yet. diff --git a/internal/cli/lint.go b/internal/cli/lint.go index 9874d7a8..14b01dd3 100644 --- a/internal/cli/lint.go +++ b/internal/cli/lint.go @@ -17,6 +17,10 @@ var lintCmd = &cobra.Command{ Example: ` probe lint tests/ probe lint tests/login.probe`, RunE: func(cmd *cobra.Command, args []string) error { + cfg, err := loadConfig(cmd) + if err != nil { + return fmt.Errorf("loading config: %w", err) + } searchPaths := args if len(searchPaths) == 0 { searchPaths = []string{"tests"} @@ -42,15 +46,39 @@ var lintCmd = &cobra.Command{ continue } - // Lint checks - warnings := lintProgram(prog) - if len(warnings) > 0 { - fmt.Printf(" \033[33m⚠\033[0m %s\n", f) - for _, w := range warnings { - fmt.Printf(" %s\n", w) + // Lint checks: a reference a run could not resolve is an error and fails the command; a placeholder that + // no value defines is a warning. + refs, err := runner.LintReferences(cfg, f) + if err != nil { + fmt.Printf(" \033[31m✗\033[0m %s — %s\n", f, err) + errs++ + continue + } + var problems, warnings []string + for _, ref := range refs { + line := fmt.Sprintf("line %d: %s", ref.Line, ref.Message) + if ref.Warning { + warnings = append(warnings, line) + } else { + problems = append(problems, line) } - } else { + } + warnings = append(lintProgram(prog), warnings...) + if len(problems) == 0 && len(warnings) == 0 { statusOK(os.Stdout, "%s (%d test(s))", f, len(prog.Tests)) + continue + } + if len(problems) > 0 { + errs++ + fmt.Printf(" \033[31m✗\033[0m %s\n", f) + } else { + fmt.Printf(" \033[33m⚠\033[0m %s\n", f) + } + for _, p := range problems { + fmt.Printf(" %s\n", p) + } + for _, w := range warnings { + fmt.Printf(" %s\n", w) } } diff --git a/internal/cli/lint_references_test.go b/internal/cli/lint_references_test.go new file mode 100644 index 00000000..e0dd3c8e --- /dev/null +++ b/internal/cli/lint_references_test.go @@ -0,0 +1,147 @@ +package cli + +import ( + "bytes" + "io" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/alphawavesystems/flutter-probe/internal/parser" +) + +// probe lint fails on a reference a run cannot resolve, and only warns about a placeholder no value defines. +func TestLintCommandFailsOnUnknownStepsButNotOnWarnings(t *testing.T) { + dir := t.TempDir() + warn := filepath.Join(dir, "warn.probe") + if err := os.WriteFile(warn, []byte("test \"x\"\n type \"\" into the \"Email\" field\n"), 0o644); err != nil { + t.Fatal(err) + } + bad := filepath.Join(dir, "bad.probe") + if err := os.WriteFile(bad, []byte("test \"x\"\n frobnicate the cart\n"), 0o644); err != nil { + t.Fatal(err) + } + if err := lintCmd.RunE(lintCmd, []string{warn}); err != nil { + t.Errorf("a placeholder warning must not fail lint: %v", err) + } + if err := lintCmd.RunE(lintCmd, []string{bad}); err == nil { + t.Error("an unknown step must fail lint") + } +} + +// lintOutput runs probe lint on paths and returns what the command printed to standard output, and its error. The report +// is written with fmt.Print, so the test swaps os.Stdout for a pipe while the command runs. +func lintOutput(t *testing.T, paths ...string) (string, error) { + t.Helper() + r, w, err := os.Pipe() + if err != nil { + t.Fatal(err) + } + saved := os.Stdout + os.Stdout = w + captured := make(chan string) + go func() { + var b bytes.Buffer + _, _ = io.Copy(&b, r) + captured <- b.String() + }() + runErr := lintCmd.RunE(lintCmd, paths) + os.Stdout = saved + w.Close() + out := <-captured + r.Close() + return out, runErr +} + +// A file's errors print before its warnings, a file with errors gets the failure mark, and the command counts the files +// that have errors. +func TestLintCommandPrintsErrorsBeforeWarningsAndCountsFiles(t *testing.T) { + dir := t.TempDir() + mixed := filepath.Join(dir, "mixed.probe") + if err := os.WriteFile(mixed, []byte("test \"x\"\n type \"\" into the \"Email\" field\n frobnicate the cart\n"), 0o644); err != nil { + t.Fatal(err) + } + bad := filepath.Join(dir, "bad.probe") + if err := os.WriteFile(bad, []byte("test \"x\"\n frobnicate the cart\n"), 0o644); err != nil { + t.Fatal(err) + } + out, err := lintOutput(t, mixed, bad) + if err == nil || err.Error() != "2 file(s) have errors" { + t.Fatalf("err = %v, want \"2 file(s) have errors\"", err) + } + if !strings.Contains(out, "✗") { + t.Errorf("a file with errors needs the failure mark:\n%s", out) + } + errAt := strings.Index(out, "line 3: unknown step") + warnAt := strings.Index(out, "line 2: is not defined") + if errAt < 0 || warnAt < 0 || errAt > warnAt { + t.Errorf("errors must print before warnings:\n%s", out) + } +} + +// A warning alone does not fail the command, and the run ends with All files valid. A zero-state text check with no +// anchor is reported through the same path. +func TestLintCommandWarnsOnAZeroStateCheckAndStillPasses(t *testing.T) { + dir := t.TempDir() + zero := filepath.Join(dir, "zero.probe") + if err := os.WriteFile(zero, []byte("test \"x\"\n see \"No items\"\n"), 0o644); err != nil { + t.Fatal(err) + } + out, err := lintOutput(t, zero) + if err != nil { + t.Fatalf("a warning must not fail lint: %v", err) + } + if !strings.Contains(out, "⚠") || !strings.Contains(out, "matches any longer text") { + t.Errorf("want the warning mark and the zero-state warning:\n%s", out) + } + if !strings.Contains(out, "All files valid.") { + t.Errorf("a run with only warnings ends with All files valid.:\n%s", out) + } +} + +// lintProgram warns on a count or zero-state text check that a substring match passes by accident. It does not warn on an +// anchored check, a negated check, or a sentence that only starts with a zero-state word. +func TestLintProgramFlagsUnanchoredCountAndZeroStateChecks(t *testing.T) { + cases := []struct { + name, src string + warn bool + }{ + {"zero state", "test \"x\"\n see \"No items\"\n", true}, + {"count", "test \"x\"\n see \"3 items\"\n", true}, + {"anchored zero state", "test \"x\"\n see \"No items\" matching \"^No items$\"\n", false}, + {"negated count", "test \"x\"\n don't see \"3 items\"\n", false}, + {"longer sentence", "test \"x\"\n see \"Nothing is wrong with this page\"\n", false}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + prog, err := parser.ParseFile(c.src) + if err != nil { + t.Fatal(err) + } + warns := lintProgram(prog) + if got := len(warns) > 0; got != c.warn { + t.Errorf("lintProgram warned = %t, want %t: %q", got, c.warn, warns) + } + }) + } +} + +// A composite test's device blocks are linted: a bad step in one fails probe lint, and a valid block passes. +func TestLintCommandChecksCompositeDeviceBlocks(t *testing.T) { + dir := t.TempDir() + good := filepath.Join(dir, "good.probe") + if err := os.WriteFile(good, []byte("composite test \"login\"\n A:\n tap \"Login\"\n"), 0o644); err != nil { + t.Fatal(err) + } + bad := filepath.Join(dir, "bad.probe") + if err := os.WriteFile(bad, []byte("composite test \"login\"\n A:\n bogus thing\n"), 0o644); err != nil { + t.Fatal(err) + } + if _, err := lintOutput(t, good); err != nil { + t.Errorf("a valid composite test must pass lint: %v", err) + } + if _, err := lintOutput(t, bad); err == nil { + t.Error("an unknown step in a device block must fail lint") + } +} diff --git a/internal/mcp/server.go b/internal/mcp/server.go index 949beb87..bbd3939b 100644 --- a/internal/mcp/server.go +++ b/internal/mcp/server.go @@ -427,7 +427,7 @@ Takes the same arguments as run_tests (see its description for the flags). Then }, { Name: "lint", - Description: "Validate .probe files; warns on count and zero-state text checks, never fails.", + Description: "Validate .probe files without running them. Fails on a step or recipe call that resolves to nothing and on a selector that can never match; warns on unresolved placeholders, unknown use files and count or zero-state text checks. It also checks hook and composite test bodies, which `probe test --dry-run` does not.", InputSchema: mcpSchema{ Type: "object", Properties: map[string]mcpProp{ diff --git a/internal/runner/lint.go b/internal/runner/lint.go new file mode 100644 index 00000000..993a463c --- /dev/null +++ b/internal/runner/lint.go @@ -0,0 +1,303 @@ +package runner + +import ( + "context" + "fmt" + "maps" + "os" + "path/filepath" + "reflect" + "regexp" + "sort" + "strings" + + "github.com/alphawavesystems/flutter-probe/internal/config" + "github.com/alphawavesystems/flutter-probe/internal/parser" +) + +// LintFinding is one problem LintReferences found in a file. An error is a reference a run cannot satisfy as written: a +// step or recipe call that resolves to nothing, or a selector that can never match. A warning is a placeholder that no +// example column, recipe parameter or stored value defines (a run leaves it in the text as written), or a use that names +// no file (a run only needs that file when a call uses one of its recipes). +type LintFinding struct { + Line int + Warning bool + Message string +} + +// placeholderPattern matches a placeholder in a step's text. Names are identifiers, with dots for names such as +// response.status. generators do not match it as a name, and are skipped by name below. +var placeholderPattern = regexp.MustCompile(`<([A-Za-z_][A-Za-z0-9_.]*)>`) + +// runtimeVars are placeholder names the executor sets itself while it runs a step. +var runtimeVars = []string{"clipboard", "response.status", "response.body"} + +var ( + stepType = reflect.TypeFor[parser.Step]() + stepSliceType = reflect.TypeFor[[]parser.Step]() + selectorType = reflect.TypeFor[parser.Selector]() +) + +// LintReferences checks path for references that a run could not resolve, without running any step. Recipe calls and +// use statements resolve the way --dry-run resolves them, with the recipes folder from cfg. A file that does not parse +// returns the parse error. +func LintReferences(cfg *config.Config, path string) ([]LintFinding, error) { + src, err := os.ReadFile(path) + if err != nil { + return nil, err + } + prog, err := parser.ParseFile(string(src)) + if err != nil { + return nil, err + } + ctx := context.Background() + r := New(cfg, nil, nil, RunOptions{DryRun: true, Files: []string{path}}) + if err := r.loadRecipes(ctx); err != nil { + return nil, err + } + r.baseRecipes = make(map[string]parser.RecipeDef, len(r.recipes)) + maps.Copy(r.baseRecipes, r.recipes) + if err := r.loadFileRecipes(ctx, path, prog); err != nil { + return nil, err + } + + var findings []LintFinding + for _, u := range prog.Uses { + target := filepath.Join(filepath.Dir(path), u.Path) + if _, err := os.Stat(target); os.IsNotExist(err) { + // A missing file is a warning: a run only fails when a call needs one of its recipes, and that call is + // reported as an unknown step. + findings = append(findings, LintFinding{Line: u.Line, Warning: true, Message: fmt.Sprintf("use %q: no file at %s", u.Path, target)}) + } + } + + // The file's own bodies are checked. The recipes it can call only add the names they define, because a recipe + // parameter or a stored value from one of them stays set for the rest of the run. + var own [][]parser.Step + for _, t := range prog.Tests { + own = append(own, t.Body) + } + for _, h := range prog.Hooks { + own = append(own, h.Body) + } + for _, rec := range prog.Recipes { + own = append(own, rec.Body) + } + for _, c := range prog.CompositeTests { + own = append(own, c.Body) + } + + defined := map[string]bool{} + // Lines where a go back step starts. A call on such a line was read apart from the step (go back takes no arguments). + goBackLines := map[int]bool{} + for _, name := range runtimeVars { + defined[name] = true + } + for _, t := range prog.Tests { + if t.Examples == nil { + continue + } + headers := t.Examples.Headers + if t.Examples.Source != "" { + if csv, err := loadCSVExamples(filepath.Dir(path), t.Examples.Source); err == nil { + headers = csv.Headers + } + } + for _, h := range headers { + defined[h] = true + } + } + for _, rec := range r.recipes { + for _, p := range rec.Params { + defined[p] = true + } + walkSteps(rec.Body, func(s parser.Step) { markStored(s, defined) }) + } + for _, body := range own { + walkSteps(body, func(s parser.Step) { + markStored(s, defined) + if a, ok := s.(parser.ActionStep); ok && a.Verb == parser.VerbGoBack { + goBackLines[a.Line] = true + } + }) + } + + for _, body := range own { + walkSteps(body, func(s parser.Step) { + line := stepLine(s) + if rc, ok := s.(parser.RecipeCall); ok { + if _, _, _, found := resolveCall(r.recipes, rc); !found { + msg := fmt.Sprintf("unknown step %q: not a built-in step and no recipe with that name is defined (check the use statements and the recipes folder)", rc.Name) + if goBackLines[rc.Line] { + msg = fmt.Sprintf("go back takes no arguments: the words after it on line %d are read as a separate step %q", rc.Line, rc.Name) + } else if strings.HasPrefix(rc.Name, "use ") || rc.Name == "use" { + // The parser reads use only at the top of a file; inside a test it is an ordinary call. + msg = "a use statement must come before the first test; inside a test it is not read" + } + findings = append(findings, LintFinding{Line: rc.Line, Message: msg}) + } + } + texts, sels := scanStep(s) + for _, sel := range sels { + if problem := selectorProblem(sel); problem != "" { + findings = append(findings, LintFinding{Line: line, Message: problem}) + } + } + for _, text := range texts { + for _, m := range placeholderPattern.FindAllStringSubmatch(text, -1) { + name := m[1] + if defined[name] || strings.HasPrefix(name, "random.") { + continue + } + findings = append(findings, LintFinding{Line: line, Warning: true, Message: fmt.Sprintf("<%s> is not defined: no example column, recipe parameter or stored value has that name", name)}) + } + } + }) + } + return dedupeFindings(findings), nil +} + +// markStored records the names a step stores, which a placeholder may use later in the run. +func markStored(s parser.Step, defined map[string]bool) { + if a, ok := s.(parser.ActionStep); ok && (a.Verb == parser.VerbStore || a.Verb == parser.VerbReadWithAI) { + defined[a.Name] = true + } +} + +// selectorProblem names why a selector can never match, or returns "" when it can. A plain text selector is not checked: +// an empty one is how a step with no target is written, and some steps take no target at all. +func selectorProblem(sel parser.Selector) string { + switch sel.Kind { + case parser.SelectorID: + if sel.Text == "" || sel.Text == "#" { + return "a #id selector needs a name after the #" + } + case parser.SelectorOrdinal: + if sel.Ordinal < 1 || sel.Text == "" { + return "an ordinal selector needs a number and text, such as 2nd \"Buy\"" + } + case parser.SelectorPositional: + if sel.Container == "" { + return "the container after \"in\" is empty" + } + case parser.SelectorRelational: + if sel.Anchor == "" { + return fmt.Sprintf("the %s selector has no anchor text after it", sel.Relation) + } + } + return "" +} + +// stepLine is the 1-based line a step starts on, or 0 when the step does not record one. +func stepLine(s parser.Step) int { + if n, ok := s.(parser.Node); ok { + return n.GetLine() + } + return 0 +} + +// walkSteps calls visit for every step in steps and, depth first, for the steps nested in each of them. +func walkSteps(steps []parser.Step, visit func(parser.Step)) { + for _, s := range steps { + if s == nil { + continue + } + visit(s) + walkSteps(nestedSteps(s), visit) + } +} + +// nestedSteps returns the steps held in a step's fields: a conditional's branches, a loop's or retry's body, a timed +// block's inner step and the step a device section wraps. +func nestedSteps(s parser.Step) []parser.Step { + var out []parser.Step + v := reflect.ValueOf(s) + for v.Kind() == reflect.Pointer || v.Kind() == reflect.Interface { + if v.IsNil() { + return nil + } + v = v.Elem() + } + if v.Kind() != reflect.Struct { + return nil + } + for i := 0; i < v.NumField(); i++ { + f := v.Field(i) + if !f.CanInterface() { + continue + } + switch f.Type() { + case stepSliceType: + for j := 0; j < f.Len(); j++ { + if st, ok := f.Index(j).Interface().(parser.Step); ok { + out = append(out, st) + } + } + case stepType: + if !f.IsNil() { + if st, ok := f.Interface().(parser.Step); ok { + out = append(out, st) + } + } + } + } + return out +} + +// scanStep collects the text values and the selectors in a step's own fields. Nested steps are left out, because +// walkSteps visits them on their own. +func scanStep(s parser.Step) ([]string, []parser.Selector) { + var texts []string + var sels []parser.Selector + if rc, ok := s.(parser.RecipeCall); ok { + // The name of a call holds markers in the places of its quoted arguments; they are not variables. The + // arguments can hold placeholders of their own. + scanValue(reflect.ValueOf(rc.Args), &texts, &sels) + scanValue(reflect.ValueOf(rc.NumArgs), &texts, &sels) + return texts, sels + } + scanValue(reflect.ValueOf(s), &texts, &sels) + return texts, sels +} + +func scanValue(v reflect.Value, texts *[]string, sels *[]parser.Selector) { + switch v.Kind() { + case reflect.String: + *texts = append(*texts, v.String()) + case reflect.Interface, reflect.Pointer: + if !v.IsNil() { + scanValue(v.Elem(), texts, sels) + } + case reflect.Slice, reflect.Array: + for i := 0; i < v.Len(); i++ { + scanValue(v.Index(i), texts, sels) + } + case reflect.Struct: + if v.Type() == selectorType && v.CanInterface() { + *sels = append(*sels, v.Interface().(parser.Selector)) + } + for i := 0; i < v.NumField(); i++ { + f := v.Field(i) + if f.Type() == stepType || f.Type() == stepSliceType { + continue + } + scanValue(f, texts, sels) + } + } +} + +// dedupeFindings drops repeated findings and orders the rest by line, keeping the order of the file within a line. +func dedupeFindings(findings []LintFinding) []LintFinding { + seen := map[string]bool{} + var out []LintFinding + for _, f := range findings { + key := fmt.Sprintf("%d|%t|%s", f.Line, f.Warning, f.Message) + if seen[key] { + continue + } + seen[key] = true + out = append(out, f) + } + sort.SliceStable(out, func(i, j int) bool { return out[i].Line < out[j].Line }) + return out +} diff --git a/internal/runner/lint_test.go b/internal/runner/lint_test.go new file mode 100644 index 00000000..6cefbb7f --- /dev/null +++ b/internal/runner/lint_test.go @@ -0,0 +1,251 @@ +package runner + +import ( + "os" + "path/filepath" + "strings" + "testing" + + "github.com/alphawavesystems/flutter-probe/internal/config" + "github.com/alphawavesystems/flutter-probe/internal/parser" +) + +// lintFiles writes files into a fresh directory, then lints the file called main and returns its findings. +func lintFiles(t *testing.T, files map[string]string, main string) []LintFinding { + t.Helper() + dir := t.TempDir() + for name, src := range files { + if err := os.WriteFile(filepath.Join(dir, name), []byte(src), 0o644); err != nil { + t.Fatal(err) + } + } + findings, err := LintReferences(&config.Config{}, filepath.Join(dir, main)) + if err != nil { + t.Fatalf("LintReferences: %v", err) + } + return findings +} + +// lintSource lints one file called flows.probe. +func lintSource(t *testing.T, src string) []LintFinding { + t.Helper() + return lintFiles(t, map[string]string{"flows.probe": src}, "flows.probe") +} + +// requireFinding fails unless findings holds one finding on line with the given severity whose message contains substr. +func requireFinding(t *testing.T, findings []LintFinding, line int, warning bool, substr string) { + t.Helper() + for _, f := range findings { + if f.Line == line && f.Warning == warning && strings.Contains(f.Message, substr) { + return + } + } + t.Errorf("no finding on line %d (warning=%t) containing %q; got %+v", line, warning, substr, findings) +} + +func requireNoFindings(t *testing.T, findings []LintFinding) { + t.Helper() + if len(findings) != 0 { + t.Errorf("expected no findings, got %+v", findings) + } +} + +func TestLintReferencesFlagsUnknownSteps(t *testing.T) { + findings := lintSource(t, "test \"checkout\"\n tap \"Buy\"\n frobnicate the cart\n") + requireFinding(t, findings, 3, false, "unknown step") + if len(findings) != 1 { + t.Errorf("expected only the unknown step, got %+v", findings) + } +} + +// A use statement inside a test is an ordinary call to a recipe named use, which no run can resolve. +func TestLintReferencesExplainsAUseInsideATest(t *testing.T) { + findings := lintSource(t, "test \"nav\"\n use \"lib.probe\"\n tap \"Go\"\n") + requireFinding(t, findings, 2, false, "must come before the first test") +} + +// go back takes no arguments, so the words after it on its line are read as a separate step. The message says so, and +// the call is not reported as an unknown step on its own. +func TestLintReferencesExplainsWordsAfterGoBack(t *testing.T) { + findings := lintSource(t, "test \"nav\"\n go back to main menu\n") + requireFinding(t, findings, 2, false, "go back takes no arguments") + if len(findings) != 1 { + t.Errorf("expected only the go back explanation, got %+v", findings) + } +} + +func TestLintReferencesAcceptsRecipesDefinedInTheFile(t *testing.T) { + src := "recipe \"sign in\" (email)\n type \"\" into the \"Email\" field\n\ntest \"login\"\n sign in \"a@b.c\"\n" + requireNoFindings(t, lintSource(t, src)) +} + +func TestLintReferencesAcceptsRecipesFromTheRecipesFolder(t *testing.T) { + recipes := t.TempDir() + if err := os.WriteFile(filepath.Join(recipes, "shared.probe"), []byte("recipe \"open menu\"\n tap \"Menu\"\n"), 0o644); err != nil { + t.Fatal(err) + } + dir := t.TempDir() + path := filepath.Join(dir, "flows.probe") + if err := os.WriteFile(path, []byte("test \"nav\"\n open menu\n"), 0o644); err != nil { + t.Fatal(err) + } + findings, err := LintReferences(&config.Config{Recipes: recipes}, path) + if err != nil { + t.Fatal(err) + } + requireNoFindings(t, findings) +} + +func TestLintReferencesWarnsAboutUseOfAMissingFile(t *testing.T) { + findings := lintSource(t, "use \"missing.probe\"\n\ntest \"x\"\n tap \"Go\"\n") + requireFinding(t, findings, 1, true, "no file at") +} + +// A missing use file fails only the calls that need its recipes: they are errors, and the use is just a warning. +func TestLintReferencesErrorsOnACallThatNeedsAMissingUse(t *testing.T) { + findings := lintSource(t, "use \"missing.probe\"\n\ntest \"x\"\n open menu\n") + requireFinding(t, findings, 1, true, "no file at") + requireFinding(t, findings, 4, false, "unknown step") +} + +func TestLintReferencesAcceptsRecipesFromAUseStatement(t *testing.T) { + files := map[string]string{ + "lib.probe": "recipe \"open menu\"\n tap \"Menu\"\n", + "flows.probe": "use \"lib.probe\"\n\ntest \"nav\"\n open menu\n", + } + requireNoFindings(t, lintFiles(t, files, "flows.probe")) +} + +func TestLintReferencesWarnsAboutAPlaceholderNothingDefines(t *testing.T) { + findings := lintSource(t, "test \"login\"\n type \"\" into the \"Email\" field\n") + requireFinding(t, findings, 2, true, " is not defined") + if len(findings) != 1 { + t.Errorf("expected only the placeholder warning, got %+v", findings) + } +} + +func TestLintReferencesAcceptsPlaceholdersThatAValueDefines(t *testing.T) { + cases := map[string]string{ + "example column": "test \"login\"\n type \"\" into the \"Email\" field\n\nwith examples:\n email\n \"a@b.c\"\n", + "stored value": "test \"login\"\n store \"abc\" as token\n type \"\" into the \"Field\"\n", + "recipe param": "recipe \"sign in\" (email)\n type \"\" into the \"Email\" field\n\ntest \"login\"\n sign in \"a@b.c\"\n", + "runtime value": "test \"status\"\n see \"\"\n", + "random value": "test \"gen\"\n see \"\"\n", + } + for name, src := range cases { + t.Run(name, func(t *testing.T) { + requireNoFindings(t, lintSource(t, src)) + }) + } +} + +func TestLintReferencesFlagsASelectorThatCanNeverMatch(t *testing.T) { + cases := []struct { + name string + step string + want string + }{ + {"ordinal without text", "tap 2nd", "an ordinal selector needs a number and text"}, + {"relational without anchor", "tap \"Save\" below", "the below selector has no anchor text"}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + findings := lintSource(t, "test \"x\"\n "+c.step+"\n") + requireFinding(t, findings, 2, false, c.want) + }) + } +} + +func TestLintReferencesAcceptsWorkingSelectors(t *testing.T) { + src := "test \"x\"\n tap 2nd \"Buy\"\n tap \"Save\" below \"Cancel\"\n tap \"Buy\" in \"Cart\"\n tap #buy_button\n" + requireNoFindings(t, lintSource(t, src)) +} + +// walkSteps reaches the steps nested in conditionals, loops, retries and timed blocks, not only the top level. +func TestWalkStepsVisitsNestedBodies(t *testing.T) { + inner := parser.ActionStep{Verb: parser.VerbStore, Text: "x", Name: "v", Line: 3} + steps := []parser.Step{ + parser.ConditionalStep{Then: []parser.Step{inner}, Line: 1}, + parser.LoopStep{Body: []parser.Step{parser.WaitStep{Line: 4}}, Line: 2}, + } + var lines []int + walkSteps(steps, func(s parser.Step) { lines = append(lines, stepLine(s)) }) + want := []int{1, 3, 2, 4} + if len(lines) != len(want) { + t.Fatalf("visited lines %v, want %v", lines, want) + } + for i := range want { + if lines[i] != want[i] { + t.Errorf("visited lines %v, want %v", lines, want) + break + } + } +} + +// An ordinal selector needs a position of at least 1 and the text it counts. Checked on the selector directly, so the rule +// holds whatever syntax reaches it. +func TestSelectorProblemRejectsAnOrdinalWithoutPositionOrText(t *testing.T) { + bad := map[string]parser.Selector{ + "position zero": {Kind: parser.SelectorOrdinal, Text: "Buy", Ordinal: 0}, + "no text": {Kind: parser.SelectorOrdinal, Text: "", Ordinal: 2}, + } + for name, sel := range bad { + t.Run(name, func(t *testing.T) { + if selectorProblem(sel) == "" { + t.Errorf("ordinal selector %+v: want a problem", sel) + } + }) + } + if problem := selectorProblem(parser.Selector{Kind: parser.SelectorOrdinal, Text: "Buy", Ordinal: 2}); problem != "" { + t.Errorf("a working ordinal selector: unexpected problem %q", problem) + } +} + +// An #id selector needs a name after the #. A # that is not followed by a letter or underscore starts a comment in the +// lexer, so a bare # never reaches this check from a file; it is checked on the selector directly. +func TestSelectorProblemRejectsAnIDWithoutAName(t *testing.T) { + for _, text := range []string{"", "#"} { + if selectorProblem(parser.Selector{Kind: parser.SelectorID, Text: text}) == "" { + t.Errorf("id selector with text %q: want a problem", text) + } + } + if problem := selectorProblem(parser.Selector{Kind: parser.SelectorID, Text: "#buy_button"}); problem != "" { + t.Errorf("a named id selector: unexpected problem %q", problem) + } +} + +func TestLintReferencesFlagsAnEmptyContainer(t *testing.T) { + findings := lintSource(t, "test \"x\"\n tap \"Buy\" in \"\"\n") + requireFinding(t, findings, 2, false, "the container after \"in\" is empty") +} + +// read ... with ai into names a variable, so a placeholder that uses it is defined. +func TestLintReferencesAcceptsAPlaceholderFromReadWithAI(t *testing.T) { + src := "test \"x\"\n read \"the order total\" with ai into total\n type \"\" into the \"Amount\" field\n" + requireNoFindings(t, lintSource(t, src)) +} + +// Hook bodies are checked like test bodies. The dry run checks only test bodies, so lint is what reports a bad step in a +// hook before a run does. +func TestLintReferencesChecksHookBodies(t *testing.T) { + src := "before each test\n bogus thing\n\ntest \"x\"\n tap \"Go\"\n" + requireFinding(t, lintSource(t, src), 2, false, "unknown step") +} + +// The device blocks of a composite test are checked too, and a valid block passes. The dry run skips composite tests. +func TestLintReferencesChecksCompositeDeviceBlocks(t *testing.T) { + requireNoFindings(t, lintSource(t, "composite test \"login\"\n A:\n tap \"Login\"\n")) + requireFinding(t, lintSource(t, "composite test \"login\"\n A:\n bogus thing\n"), 3, false, "unknown step") +} + +func TestLintReferencesReportsAParseErrorWithoutFindings(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "flows.probe") + if err := os.WriteFile(path, []byte("test \"unterminated\n tap \"broken\n"), 0o644); err != nil { + t.Fatal(err) + } + findings, err := LintReferences(&config.Config{}, path) + if err == nil { + t.Fatalf("expected the parse error, got findings %+v", findings) + } +} diff --git a/internal/runner/runner.go b/internal/runner/runner.go index ee0033f2..8d6405ae 100644 --- a/internal/runner/runner.go +++ b/internal/runner/runner.go @@ -181,20 +181,11 @@ func (r *Runner) Run(ctx context.Context) ([]TestResult, error) { return results, nil } -func (r *Runner) runFile(ctx context.Context, path string) ([]TestResult, error) { - src, err := os.ReadFile(path) - if err != nil { - return nil, fmt.Errorf("read %s: %w", path, err) - } - - prog, err := parser.ParseFile(string(src)) - if err != nil { - return nil, fmt.Errorf("parse %s: %w", path, err) - } - - // --dry-run resolves each file on its own, like running that file alone: the - // recipes of other files in the same run must not make an unresolved call look - // fine (a runtime run of this one file could not see them). +// loadFileRecipes makes the recipes that path can call available to its tests: those of the recipes folder, those of +// its `use` statements (followed in turn), and its own. --dry-run resolves a file on its own, like running that file +// alone: the recipes of other files in the same run must not make an unresolved call look fine (a runtime run of this +// one file could not see them). +func (r *Runner) loadFileRecipes(ctx context.Context, path string, prog *parser.Program) error { if r.opts.DryRun && r.baseRecipes != nil { r.recipes = make(map[string]parser.RecipeDef, len(r.baseRecipes)) for k, v := range r.baseRecipes { @@ -211,7 +202,7 @@ func (r *Runner) runFile(ctx context.Context, path string) ([]TestResult, error) } for _, u := range prog.Uses { if err := r.loadUse(ctx, path, u, seen); err != nil { - return nil, err + return err } } @@ -219,6 +210,23 @@ func (r *Runner) runFile(ctx context.Context, path string) ([]TestResult, error) for _, rec := range prog.Recipes { r.recipes[rec.Name] = rec } + return nil +} + +func (r *Runner) runFile(ctx context.Context, path string) ([]TestResult, error) { + src, err := os.ReadFile(path) + if err != nil { + return nil, fmt.Errorf("read %s: %w", path, err) + } + + prog, err := parser.ParseFile(string(src)) + if err != nil { + return nil, fmt.Errorf("parse %s: %w", path, err) + } + + if err := r.loadFileRecipes(ctx, path, prog); err != nil { + return nil, err + } // Filter tests by tag tests := r.filterTests(prog.Tests) diff --git a/studio/frontend/src/probescript/dictionary.generated.ts b/studio/frontend/src/probescript/dictionary.generated.ts index a0a5c1c0..e9b36d20 100644 --- a/studio/frontend/src/probescript/dictionary.generated.ts +++ b/studio/frontend/src/probescript/dictionary.generated.ts @@ -2,7 +2,7 @@ // Do not edit by hand. After a change under website/src/content/docs/probescript/, run // `npm run gen:probescript` in studio/frontend. -export const SOURCE_SHA256 = "780a6ca36ec1c85cd6412d7d657766c71b4829ce4339f209995dc8d9212be7fe"; +export const SOURCE_SHA256 = "4270861a86aee40bd8fb75fcbd86a92e53cd2519760dc2d69a2e3d8cd39526f8"; export const COMMANDS = [ { diff --git a/tests/cart.probe b/tests/cart.probe index 27898d44..cc696cb3 100644 --- a/tests/cart.probe +++ b/tests/cart.probe @@ -1,5 +1,5 @@ # Code activation flow tests -use "tests/recipes/common.probe" +use "recipes/common.probe" before each test open and wait for the app diff --git a/tests/login.probe b/tests/login.probe index 27e66075..229f93b9 100644 --- a/tests/login.probe +++ b/tests/login.probe @@ -1,5 +1,5 @@ # Activation code entry tests -use "tests/recipes/common.probe" +use "recipes/common.probe" before each test open and wait for the app diff --git a/tests/recipes/login.probe b/tests/recipes/login.probe index c77a8000..b1fa7346 100644 --- a/tests/recipes/login.probe +++ b/tests/recipes/login.probe @@ -1,4 +1,4 @@ -use "tests/recipes/common.probe" +use "common.probe" recipe "activate code" (d1, d2, d3, d4, d5) navigate to keypad diff --git a/tests/regression/visual.probe b/tests/regression/visual.probe index b5de09ef..228566df 100644 --- a/tests/regression/visual.probe +++ b/tests/regression/visual.probe @@ -1,6 +1,6 @@ # Visual regression test suite # Each test captures a screenshot and compares against a baseline. -use "tests/recipes/common.probe" +use "../recipes/common.probe" before each test open and wait for the app diff --git a/tests/smoke/navigation.probe b/tests/smoke/navigation.probe index d6265f6e..ad76286a 100644 --- a/tests/smoke/navigation.probe +++ b/tests/smoke/navigation.probe @@ -1,5 +1,5 @@ # Smoke test: core navigation paths -use "tests/recipes/common.probe" +use "../recipes/common.probe" before each test open and wait for the app @@ -22,7 +22,9 @@ test "navigate to keypad and back" @smoke navigate to keypad see "Activar" - go back to main menu + go back + wait 1 seconds + see "Home" test "navigate to lector and back" @smoke diff --git a/tools/probe-convert/convert/robot/robot.go b/tools/probe-convert/convert/robot/robot.go index 2075d0a6..dde7d07f 100644 --- a/tools/probe-convert/convert/robot/robot.go +++ b/tools/probe-convert/convert/robot/robot.go @@ -40,6 +40,7 @@ func convertRobot(src, path string) (string, []convert.Warning) { w := convert.NewProbeWriter() var warnings []convert.Warning lines := strings.Split(src, "\n") + keywords := collectKeywordNames(lines) w.Comment("Converted from Robot Framework") w.BlankLine() @@ -127,7 +128,7 @@ func convertRobot(src, path string) (string, []convert.Warning) { if currentSection == sectionTestCases { w.Test(name) } else { - w.Recipe(name, nil) + w.Recipe(strings.ToLower(name), nil) } continue } @@ -194,7 +195,12 @@ func convertRobot(src, path string) (string, []convert.Warning) { } else { // Unknown keyword — emit as recipe call or TODO. if isLikelyKeywordCall(keyword) { - call := substituteVars(keyword) + call := strings.ToLower(substituteVars(keyword)) + if name, ok := keywords[robotKeywordKey(call)]; ok { + // The lexer writes a call in lower case, and probe matches recipe names case-sensitively, so the + // call uses the lower-case spelling the recipe is written under. + call = name + } if len(args) > 0 { call += " with " + strings.Join(quoteArgs(args), " and ") } @@ -214,6 +220,41 @@ func convertRobot(src, path string) (string, []convert.Warning) { return w.String(), warnings } +// robotKeywordKey is how Robot Framework matches keyword names: case, spaces and underscores do not count. +func robotKeywordKey(name string) string { + return strings.ToLower(strings.NewReplacer(" ", "", "_", "").Replace(strings.TrimSpace(name))) +} + +// collectKeywordNames returns the keywords the file defines, keyed by robotKeywordKey, each mapped to the name its recipe +// is written under. A call writes the keyword as its body spells it, so it resolves through this map. The sections are +// read the way convertRobot reads them, so a header the converter ignores does not end the keywords section. +func collectKeywordNames(lines []string) map[string]string { + names := map[string]string{} + inKeywords := false + for _, rawLine := range lines { + trimmed := strings.TrimSpace(rawLine) + if trimmed == "" || strings.HasPrefix(trimmed, "#") { + continue + } + if strings.HasPrefix(trimmed, "***") { + upper := strings.ToUpper(trimmed) + switch { + case strings.Contains(upper, "SETTING"), strings.Contains(upper, "VARIABLE"), + strings.Contains(upper, "TEST CASE"), strings.Contains(upper, "TASK"): + inKeywords = false + case strings.Contains(upper, "KEYWORD"): + inKeywords = true + } + continue + } + if inKeywords && rawLine[0] != ' ' && rawLine[0] != '\t' { + name := substituteVars(trimmed) + names[robotKeywordKey(name)] = strings.ToLower(name) + } + } + return names +} + // splitRobotCells splits a Robot Framework line by 2+ spaces or tab. func splitRobotCells(line string) []string { // Robot uses 2+ spaces or \t as cell separators. diff --git a/tools/probe-convert/convert/robot/robot_test.go b/tools/probe-convert/convert/robot/robot_test.go index 914b3beb..7bb869aa 100644 --- a/tools/probe-convert/convert/robot/robot_test.go +++ b/tools/probe-convert/convert/robot/robot_test.go @@ -1,6 +1,7 @@ package robot import ( + "os" "strings" "testing" ) @@ -45,7 +46,7 @@ Login With Credentials t.Fatalf("convert: %v", err) } - assertContains(t, result.ProbeCode, `recipe "Login With Credentials"`) + assertContains(t, result.ProbeCode, `recipe "login with credentials"`) assertContains(t, result.ProbeCode, "tap on #emailField") assertContains(t, result.ProbeCode, "") assertContains(t, result.ProbeCode, "") @@ -165,6 +166,28 @@ XPath Fallback assertContains(t, result.ProbeCode, "LinearLayout") } +// The lexer writes every call in lower case, and probe matches a call only to a recipe with exactly that name. So the +// converter writes recipes in lower case too, and the calls in the converted checkout example name them that way. +func TestConvert_RecipesAndCallsAgreeInCase(t *testing.T) { + src, err := os.ReadFile("../../examples/robot/checkout.robot") + if err != nil { + t.Fatal(err) + } + result, err := New().Convert(src, "checkout.robot") + if err != nil { + t.Fatalf("convert: %v", err) + } + assertContains(t, result.ProbeCode, `recipe "login as user"`) + assertContains(t, result.ProbeCode, `recipe "add item to cart"`) + assertContains(t, result.ProbeCode, `login as user with "user@test.com" and "pass123"`) + assertContains(t, result.ProbeCode, `add item to cart with "Running Shoes"`) + for _, stale := range []string{`recipe "Login As User"`, `Login As User with`, `Add Item To Cart with`} { + if strings.Contains(result.ProbeCode, stale) { + t.Errorf("output still writes %q, which no call can match\n%s", stale, result.ProbeCode) + } + } +} + func assertContains(t *testing.T, haystack, needle string) { t.Helper() if !strings.Contains(haystack, needle) { diff --git a/tools/probe-convert/testdata/golden/robot/checkout.probe b/tools/probe-convert/testdata/golden/robot/checkout.probe index 03724d63..51bca221 100644 --- a/tools/probe-convert/testdata/golden/robot/checkout.probe +++ b/tools/probe-convert/testdata/golden/robot/checkout.probe @@ -7,7 +7,7 @@ after each close the app -recipe "Login As User" +recipe "login as user" # arguments: email, password tap on "Sign In" type "" into #emailField @@ -15,7 +15,7 @@ recipe "Login As User" tap on #loginButton wait until "Dashboard" appears -recipe "Add Item To Cart" +recipe "add item to cart" # arguments: item_name tap on "Shop" tap on "" @@ -24,15 +24,15 @@ recipe "Add Item To Cart" test "Add Product To Cart" @cart @smoke - Login As User with "user@test.com" and "pass123" - Add Item To Cart with "Running Shoes" + login as user with "user@test.com" and "pass123" + add item to cart with "Running Shoes" see "Cart (1)" take a screenshot test "Remove Product From Cart" @cart - Login As User with "user@test.com" and "pass123" - Add Item To Cart with "Running Shoes" + login as user with "user@test.com" and "pass123" + add item to cart with "Running Shoes" tap on "Cart" long press on #cartItem1 tap on "Remove" @@ -40,7 +40,7 @@ test "Remove Product From Cart" test "Check Product Visibility" @ui - Login As User with "user@test.com" and "pass123" + login as user with "user@test.com" and "pass123" tap on "Shop" wait until #productList appears see #featuredBanner diff --git a/website/src/content/docs/probescript/grammar.md b/website/src/content/docs/probescript/grammar.md index 7009ebd2..035011be 100644 --- a/website/src/content/docs/probescript/grammar.md +++ b/website/src/content/docs/probescript/grammar.md @@ -759,7 +759,7 @@ close-step = "close" , fillers , [ "app" | WORD | STRING ] ; restart-step = "restart" , fillers , [ "app" ] ; kill-step = "kill" , fillers , [ "app" ] ; clear-data-step = "clear app data" ; -go-back-step = "go back" ; +go-back-step = "go back" ; (* takes no arguments: words after it on the same line are a separate step *) press-enter-step = "press enter" ; (* the keyboard action key on the focused text field *) shake-step = "shake" ; pause-step = "pause" ; diff --git a/website/src/content/docs/probescript/syntax.md b/website/src/content/docs/probescript/syntax.md index 4c669f08..908d4435 100644 --- a/website/src/content/docs/probescript/syntax.md +++ b/website/src/content/docs/probescript/syntax.md @@ -251,7 +251,7 @@ compare screenshot "baseline" # compare against visual regression baseline dump tree # dump widget tree for debugging save logs # save app logs press enter # keyboard action key on the focused field (Maestro pressKey: Enter) -go back # pop the current route (no-op + warning at the root; use "close the app" to exit) +go back # pop the current route (no-op + warning at the root; use "close the app" to exit); takes no arguments, so words after it on the same line are a separate step rotate landscape # rotate device shake # simulate device shake gesture log "checkpoint reached" # print to test output diff --git a/website/src/content/docs/tools/cli-reference.md b/website/src/content/docs/tools/cli-reference.md index a1423a44..b71dcba1 100644 --- a/website/src/content/docs/tools/cli-reference.md +++ b/website/src/content/docs/tools/cli-reference.md @@ -11,7 +11,7 @@ description: Complete reference for all probe CLI commands and flags. | `probe validate` | Check `probe.yaml` and the `probe_version` pin without connecting to a device. Exits non-zero when a check fails; `--format json` prints the report. Run it in CI before `probe test` | | `probe doctor` | Check that `probe.yaml`, the `tests/` folder, the Flutter SDK, the platform tools (`adb`, `xcrun`) and the visible devices are ready for `probe test`. Says how to fix what is missing and changes nothing. Exits non-zero only when `probe.yaml` or `tests/` is missing. `--format json` prints the report | | `probe test [path]` | Run `.probe` test files | -| `probe lint [path]` | Validate `.probe` files for syntax errors; warns, never fails, on count and zero-state text checks that a substring match passes by accident | +| `probe lint [path]` | Validate `.probe` files without running them: syntax errors, steps and recipe calls that resolve to nothing, and selectors that can never match fail the command; unresolved placeholders and count or zero-state text checks warn | | `probe perf trend` / `probe perf compare` | Show a measured number over past runs, or compare two baseline files. See [Performance testing](/advanced/performance/) | | `probe device list` | List connected devices/simulators | | `probe device view` | Interactive device view over a local WebSocket (preview) | @@ -152,7 +152,27 @@ probe lint tests/ probe lint tests/smoke/login.probe ``` -Lint also warns, and never fails, on a `see` or `wait until ... appears` whose text is a count or a zero state (`see "0 items"`, `wait until "No items" appears`) and has no `matching` pattern, because a substring match also passes on "10 items". Anchor the pattern for an exact check: `see "0 items" matching "^0 items$"`. A `matching` pattern anchored at both ends (`^...$`) silences the warning; any other pattern is still a substring match and is still flagged. Warnings do not change the exit status. +Lint checks the references a run would resolve, without running a step, and fails the command (exit status 1) for an error: + +- A step or recipe call that matches no built-in step and no recipe in the file, its `use` statements or the recipes folder. This is the check `probe test --dry-run` makes. A `use` statement inside a test is reported too, because a file reads `use` only before its first test. A `go back` followed by words on its line is reported with that reason, because `go back` takes no arguments. +- A selector that can never match: an ordinal selector without a number and text, an `#id` without a name, a `below`, `above`, `left of` or `right of` selector without anchor text, or an `in` with an empty container. + +It warns, without failing, when: + +- A `` placeholder is not an example column, a recipe parameter, a value stored with `store` or `read ... with ai`, one of the runtime values `clipboard`, `response.status` or `response.body`, or a `` generator. A run leaves such a placeholder in the text as written. +- A `use` names a file that does not exist. A run only needs that file when a call uses one of its recipes, and such a call is reported as an unknown step. +- A `see` or `wait until ... appears` whose text is a count or a zero state (`see "0 items"`, `wait until "No items" appears`) and has no `matching` pattern, because a substring match also passes on "10 items". Anchor the pattern for an exact check: `see "0 items" matching "^0 items$"`. A `matching` pattern anchored at both ends (`^...$`) silences the warning; any other pattern is still a substring match and is still flagged. + +Lint also checks the bodies of hooks (`before each test`, `before all` and their `after` forms) and the device blocks (`A:`, `B:`) of composite tests. `probe test --dry-run` checks only test bodies, so a bad step in a hook or a composite test fails at run time. A CI job that runs `probe lint` on composite files may see new errors there. + +Environment variables (`${NAME}`) are not checked, because lint does not know the environment that will run the tests. + +Known limits. A follow-up will address these: + +- Placeholders, recipe parameters and stored names are collected across the whole file and every recipe it can call, not per test, so a name defined in one test counts for the others. +- The body of a recipe that comes from a `use` file is not checked. That file's recipes only define their names here. +- A `use` line inside a used file is not checked for a missing file. Only the `use` lines of the file given to lint are. +- A `#` followed by a space starts a comment, as it does elsewhere in ProbeScript, so `tap #` reads as `tap` with no target. Lint does not report it. ## probe record diff --git a/website/src/content/docs/tools/studio.md b/website/src/content/docs/tools/studio.md index 21c045ad..e3342481 100644 --- a/website/src/content/docs/tools/studio.md +++ b/website/src/content/docs/tools/studio.md @@ -9,7 +9,7 @@ FlutterProbe Studio is a standalone desktop application for designing, recording ## What you get -- **Monaco editor** with ProbeScript syntax highlighting, live lint markers, and the same TextMate grammar that powers the VS Code extension. Completions, hover help and snippets cover every command in the [ProbeScript dictionary](/probescript/dictionary/) and every reserved word in the [grammar](/probescript/grammar/); they are generated from those pages, so the editor and the docs agree. +- **Monaco editor** with ProbeScript syntax highlighting, live lint markers (parse errors and empty tests; `probe lint` also checks step and recipe references, selectors and placeholders), and the same TextMate grammar that powers the VS Code extension. Completions, hover help and snippets cover every command in the [ProbeScript dictionary](/probescript/dictionary/) and every reserved word in the [grammar](/probescript/grammar/); they are generated from those pages, so the editor and the docs agree. - **Workspace folder picker** — point Studio at any project root with a `tests/` folder. Choice persists across launches. - **Device picker** that lists booted simulators, online emulators, and physical iOS/Android devices side by side, with a `physical` tag and a clear hint when nothing is connectable. - **Physical device support** — the same `internal/device` code the `probe` CLI uses: `iproxy` tunneling for physical iOS over USB, `adb forward` for physical Android (same path as emulators), and mDNS auto-discovery (`_flutterprobe._tcp`) for WiFi-connected devices. See [Physical devices](#physical-devices) below. @@ -51,7 +51,7 @@ Studio's own UI is tested two ways. `make studio-e2e-web` runs the real frontend │ ┌─────────▼─────────────────────────────┐ │ │ │ Go backend (App struct) │ │ │ │ imports internal/runner │ │ -│ │ imports internal/lint │ │ +│ │ imports internal/parser │ │ │ │ imports internal/probelink │ │ │ │ imports internal/device │ │ │ └─────────┬─────────────────────────────┘ │