feat(lint): check the references a run would resolve

CHANGELOG.md

@@ -8,6 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ### Added
 - **MCP: `start_tests`, `test_status`, `test_result` and `cancel_tests` run tests in the background.** `run_tests` blocks until `probe test` exits, so a multi-device run (`--parallel`, `composite_devices`, several devices) could outlast the MCP client's tool timeout and be reported as failed while it kept running. `start_tests` takes the same arguments, returns a `job_id` at once and runs the command in its own process group; `test_status` returns the state, elapsed time and the last lines of output; `test_result` returns the retained output (the last 1 MB) and the exit code; `cancel_tests` sends SIGTERM to the run and kills it after 5 seconds. Jobs live in the `probe-mcp` process and are killed when it exits. The hosted-upload flags are refused here exactly as in `run_tests`. `run_tests` is unchanged. The MCP server now has 40 tools.
+- **`probe lint` checks the references a run would resolve.** Lint reports each step and recipe call that matches no built-in step and no recipe in the file, its `use` statements or the recipes folder (the check `probe test --dry-run` makes), each selector that can never match, and each `<name>` placeholder that no example column, recipe parameter or stored value defines. It runs without a device, gives the line of each finding, and checks hook bodies and the device blocks of composite tests too.
 - **Recordings show the element each step acted on.** A step that taps, double taps, long presses, types, clears, toggles, or swipes or scrolls with a selector records its selector, and its bounds and tap point when the element is on screen as the step starts. The viewer shows them under "Element". Text typed into a field is never recorded, and a failed lookup is an advisory warning that does not change the step.
 - **`probe test --record` writes a step-by-step recording of each test.** Each test attempt includes source files and lines, screenshots before and after steps, widget trees after steps, and the app's `dart:io` HTTP exchanges with credential redaction, plus an offline viewer. Bundles stay under `reports_folder/recordings` or `--record-dir`; nothing uploads, and recording warnings do not change test results or exit codes. Native SDK and WebView traffic and screenshot redaction rules are not yet recorded; screenshots are not redacted and widget trees are not captured before steps. Keep recordings out of version control because they can contain screens and request or response bodies.
 - **Recording manifests name the platform and OS version.** `manifest.json` in each recording bundle now carries the run's `platform` (`ios` or `android`) and `os_version`, the same values the report shows, instead of leaving both empty.
@@ -23,6 +24,24 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 - **The MCP tools keep to the local tool surface.** `run_tests` and `run_script` refuse the upload flags (`--upload-report`, `--upload-local`, `--upload-dry-run`, and any other `--upload-` flag) with the message "hosted upload is not available through MCP". The MCP tool text no longer describes the upload flags. Use the CLI from a terminal for uploads.
 - **Building from source needs Go 1.26.10 or newer (was 1.26.1).** The `go` line in `go.mod`, `studio/go.mod`, `tools/probe-convert/go.mod` and `go.work` is now 1.26.10, and the install docs say so. With the default `GOTOOLCHAIN=auto`, an older Go 1.26 downloads 1.26.10 on its own; with `GOTOOLCHAIN=local` the build stops with a version error until Go is upgraded.
 - **Older evidence files removed.** The disclosure guard also scans `docs/evidence`.
+- **`probe lint` fails on references a run cannot resolve.** A step or recipe call that resolves to nothing, or a selector that can never match, now makes the command exit with status 1, where it used to pass. A placeholder no value defines and a `use` file that does not exist are warnings and do not change the exit status. Lint also checks hook bodies and the device blocks of composite tests. `probe test --dry-run` checks only test bodies, so a bad step in those places fails at run time today, and a repository whose CI runs `probe lint` may see new failures there. Environment variables (`${NAME}`) are not checked, and the other limits are listed in the CLI reference. Within a file, errors print before warnings. Example output for a file with an error (colour codes omitted; the last line goes to stderr):
+
+  ```text
+    ✗  x.probe
+       line 7: unknown step "send reset code": not a built-in step and no recipe with that name is defined (check the use statements and the recipes folder)
+       line 8: <code> is not defined: no example column, recipe parameter or stored value has that name
+
+  Error: 1 file(s) have errors
+  ```
+
+  A file with only warnings, such as a placeholder no value defines or a missing `use` file, prints them under a warning mark and still exits with status 0. The run then ends with `All files valid.`:
+
+  ```text
+    ⚠  y.probe
+       line 3: <code> is not defined: no example column, recipe parameter or stored value has that name
+
+    All files valid.
+  ```
 
 ### Docs
 - **Recording evidence write-ups no longer name private apps.** The prose and commands in the evidence notes use neutral labels and placeholders. The raw run output that the notes cite is unchanged.
@@ -34,6 +53,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 - **VS Code: the extension compiles again with TypeScript 7 and @types/node 26.** The tsconfig names the Node types, which TypeScript 7 no longer loads by default. Without it, `npm test` failed to compile after those two upgrades.
 - **CI: the Dependabot auto-merge job adds approved pull requests to the merge queue again.** It passed `--squash` with `--auto`, so the queue ignored the request, nothing was queued, and the run still reported success. The job now passes no merge method and logs a warning for each pull request it cannot queue.
 - **CI: the Studio Go tests and the frontend and VS Code test suites run on every change to `studio/` or `vscode/`.** They ran in no workflow before, and only the browser suite covered Studio. The frontend suites run on Node 26.
+- **probe-convert writes Robot keyword recipes in lower case, and its calls match them.** The recipe kept its Robot spelling (`Login As User`), while the lexer writes every call word in lower case, so no converted call matched its recipe. The checkout golden is regenerated, and a test checks the names. A recipe name that keeps capital letters still cannot be called from a test; that is tracked separately.
 - **CI: the CLI test package runs in seconds, not minutes.** The hosted-upload failure test waited out the 5-minute retry window; it now uses a short window and checks that the failure is still only a warning.
 - **`probe test --device` finds the Android application ID for a camelCase `project.app`.** When the bundle ID as written is not installed on an Android device and its snake_case form is (`com.example.myApp` → `com.example.my_app`), the single-device run uses the snake_case ID, as `--devices` and parallel runs already did. Before, the restart relaunched the ID as written and failed. A run keeps the ID as written when that ID is installed, and when neither form is, so a project that runs today is unchanged. `--devices` and parallel runs still convert without this check.
 - **Composite runs write the platform and OS version into each recording manifest.** `probe test --composite-device` now passes each device's platform and OS version to its recorder, as single-device runs already do, so `manifest.json` is no longer empty for those devices. Devices the CLI cannot read a platform for (a WiFi composite device) still write empty values. The recording ships with `probe test --record`, which is not released yet.

internal/cli/lint.go

@@ -17,6 +17,10 @@ var lintCmd = &cobra.Command{
 	Example: `  probe lint tests/
   probe lint tests/login.probe`,
 	RunE: func(cmd *cobra.Command, args []string) error {
+		cfg, err := loadConfig(cmd)
+		if err != nil {
+			return fmt.Errorf("loading config: %w", err)
+		}
 		searchPaths := args
 		if len(searchPaths) == 0 {
 			searchPaths = []string{"tests"}
@@ -42,15 +46,39 @@ var lintCmd = &cobra.Command{
 				continue
 			}
 
-			// Lint checks
-			warnings := lintProgram(prog)
-			if len(warnings) > 0 {
-				fmt.Printf("  \033[33m⚠\033[0m  %s\n", f)
-				for _, w := range warnings {
-					fmt.Printf("     %s\n", w)
+			// Lint checks: a reference a run could not resolve is an error and fails the command; a placeholder that
+			// no value defines is a warning.
+			refs, err := runner.LintReferences(cfg, f)
+			if err != nil {
+				fmt.Printf("  \033[31m✗\033[0m  %s — %s\n", f, err)
+				errs++
+				continue
+			}
+			var problems, warnings []string
+			for _, ref := range refs {
+				line := fmt.Sprintf("line %d: %s", ref.Line, ref.Message)
+				if ref.Warning {
+					warnings = append(warnings, line)
+				} else {
+					problems = append(problems, line)
 				}
-			} else {
+			}
+			warnings = append(lintProgram(prog), warnings...)
+			if len(problems) == 0 && len(warnings) == 0 {
 				statusOK(os.Stdout, "%s  (%d test(s))", f, len(prog.Tests))
+				continue
+			}
+			if len(problems) > 0 {
+				errs++
+				fmt.Printf("  \033[31m✗\033[0m  %s\n", f)
+			} else {
+				fmt.Printf("  \033[33m⚠\033[0m  %s\n", f)
+			}
+			for _, p := range problems {
+				fmt.Printf("     %s\n", p)
+			}
+			for _, w := range warnings {
+				fmt.Printf("     %s\n", w)
 			}
 		}
 

internal/cli/lint_references_test.go

@@ -0,0 +1,147 @@
+package cli
+
+import (
+	"bytes"
+	"io"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"github.com/alphawavesystems/flutter-probe/internal/parser"
+)
+
+// probe lint fails on a reference a run cannot resolve, and only warns about a placeholder no value defines.
+func TestLintCommandFailsOnUnknownStepsButNotOnWarnings(t *testing.T) {
+	dir := t.TempDir()
+	warn := filepath.Join(dir, "warn.probe")
+	if err := os.WriteFile(warn, []byte("test \"x\"\n  type \"<email>\" into the \"Email\" field\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	bad := filepath.Join(dir, "bad.probe")
+	if err := os.WriteFile(bad, []byte("test \"x\"\n  frobnicate the cart\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	if err := lintCmd.RunE(lintCmd, []string{warn}); err != nil {
+		t.Errorf("a placeholder warning must not fail lint: %v", err)
+	}
+	if err := lintCmd.RunE(lintCmd, []string{bad}); err == nil {
+		t.Error("an unknown step must fail lint")
+	}
+}
+
+// lintOutput runs probe lint on paths and returns what the command printed to standard output, and its error. The report
+// is written with fmt.Print, so the test swaps os.Stdout for a pipe while the command runs.
+func lintOutput(t *testing.T, paths ...string) (string, error) {
+	t.Helper()
+	r, w, err := os.Pipe()
+	if err != nil {
+		t.Fatal(err)
+	}
+	saved := os.Stdout
+	os.Stdout = w
+	captured := make(chan string)
+	go func() {
+		var b bytes.Buffer
+		_, _ = io.Copy(&b, r)
+		captured <- b.String()
+	}()
+	runErr := lintCmd.RunE(lintCmd, paths)
+	os.Stdout = saved
+	w.Close()
+	out := <-captured
+	r.Close()
+	return out, runErr
+}
+
+// A file's errors print before its warnings, a file with errors gets the failure mark, and the command counts the files
+// that have errors.
+func TestLintCommandPrintsErrorsBeforeWarningsAndCountsFiles(t *testing.T) {
+	dir := t.TempDir()
+	mixed := filepath.Join(dir, "mixed.probe")
+	if err := os.WriteFile(mixed, []byte("test \"x\"\n  type \"<email>\" into the \"Email\" field\n  frobnicate the cart\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	bad := filepath.Join(dir, "bad.probe")
+	if err := os.WriteFile(bad, []byte("test \"x\"\n  frobnicate the cart\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	out, err := lintOutput(t, mixed, bad)
+	if err == nil || err.Error() != "2 file(s) have errors" {
+		t.Fatalf("err = %v, want \"2 file(s) have errors\"", err)
+	}
+	if !strings.Contains(out, "✗") {
+		t.Errorf("a file with errors needs the failure mark:\n%s", out)
+	}
+	errAt := strings.Index(out, "line 3: unknown step")
+	warnAt := strings.Index(out, "line 2: <email> is not defined")
+	if errAt < 0 || warnAt < 0 || errAt > warnAt {
+		t.Errorf("errors must print before warnings:\n%s", out)
+	}
+}
+
+// A warning alone does not fail the command, and the run ends with All files valid. A zero-state text check with no
+// anchor is reported through the same path.
+func TestLintCommandWarnsOnAZeroStateCheckAndStillPasses(t *testing.T) {
+	dir := t.TempDir()
+	zero := filepath.Join(dir, "zero.probe")
+	if err := os.WriteFile(zero, []byte("test \"x\"\n  see \"No items\"\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	out, err := lintOutput(t, zero)
+	if err != nil {
+		t.Fatalf("a warning must not fail lint: %v", err)
+	}
+	if !strings.Contains(out, "⚠") || !strings.Contains(out, "matches any longer text") {
+		t.Errorf("want the warning mark and the zero-state warning:\n%s", out)
+	}
+	if !strings.Contains(out, "All files valid.") {
+		t.Errorf("a run with only warnings ends with All files valid.:\n%s", out)
+	}
+}
+
+// lintProgram warns on a count or zero-state text check that a substring match passes by accident. It does not warn on an
+// anchored check, a negated check, or a sentence that only starts with a zero-state word.
+func TestLintProgramFlagsUnanchoredCountAndZeroStateChecks(t *testing.T) {
+	cases := []struct {
+		name, src string
+		warn      bool
+	}{
+		{"zero state", "test \"x\"\n  see \"No items\"\n", true},
+		{"count", "test \"x\"\n  see \"3 items\"\n", true},
+		{"anchored zero state", "test \"x\"\n  see \"No items\" matching \"^No items$\"\n", false},
+		{"negated count", "test \"x\"\n  don't see \"3 items\"\n", false},
+		{"longer sentence", "test \"x\"\n  see \"Nothing is wrong with this page\"\n", false},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			prog, err := parser.ParseFile(c.src)
+			if err != nil {
+				t.Fatal(err)
+			}
+			warns := lintProgram(prog)
+			if got := len(warns) > 0; got != c.warn {
+				t.Errorf("lintProgram warned = %t, want %t: %q", got, c.warn, warns)
+			}
+		})
+	}
+}
+
+// A composite test's device blocks are linted: a bad step in one fails probe lint, and a valid block passes.
+func TestLintCommandChecksCompositeDeviceBlocks(t *testing.T) {
+	dir := t.TempDir()
+	good := filepath.Join(dir, "good.probe")
+	if err := os.WriteFile(good, []byte("composite test \"login\"\n  A:\n    tap \"Login\"\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	bad := filepath.Join(dir, "bad.probe")
+	if err := os.WriteFile(bad, []byte("composite test \"login\"\n  A:\n    bogus thing\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := lintOutput(t, good); err != nil {
+		t.Errorf("a valid composite test must pass lint: %v", err)
+	}
+	if _, err := lintOutput(t, bad); err == nil {
+		t.Error("an unknown step in a device block must fail lint")
+	}
+}

internal/mcp/server.go

@@ -427,7 +427,7 @@ Takes the same arguments as run_tests (see its description for the flags). Then
 	},
 	{
 		Name:        "lint",
-		Description: "Validate .probe files; warns on count and zero-state text checks, never fails.",
+		Description: "Validate .probe files without running them. Fails on a step or recipe call that resolves to nothing and on a selector that can never match; warns on unresolved placeholders, unknown use files and count or zero-state text checks. It also checks hook and composite test bodies, which `probe test --dry-run` does not.",
 		InputSchema: mcpSchema{
 			Type: "object",
 			Properties: map[string]mcpProp{