cli: terminal UI polish (capability detection, grouped help, error blocks, progress bar)

CHANGELOG.md

@@ -5,6 +5,7 @@
 Bug fixes from a review of 0.1.3, and live progress on a terminal. No new commands; the behaviour changes are marked.
 
 - **Live progress on a terminal.** A spinner with a label and elapsed time covers every wait over about 300 ms (`scrape`, `scrape -`, `batch submit`/`wait`, and any other API call), and `scrape`, `scrape -` and `batch wait` end with a `✓`/`✗` summary line (page, status, credits, duration) on stderr. API errors keep their `ERR::` code and hint in a short block. Everything is on stderr, only when stderr is a terminal, and off with `--json`, `-q`, `NO_COLOR`, `TERM=dumb` or `CI`: stdout and the plain output are unchanged. New dependency: `golang.org/x/term`, for the terminal width.
+- **A polished terminal UI.** On a terminal only; piped, redirected, in CI, under `NO_COLOR`/`CLICOLOR=0`/`TERM=dumb`, and with `--json` or `-q`, every byte is what it was. `spicrawl --help` groups the commands by purpose (extract data, set up, monitor, reference), colours headings and the names you type, mutes defaults and API routes, wraps long descriptions under their own column and ends with the docs link; `spicrawl x --help` gets the same layout. Colour adapts to the terminal: the 16-colour palette (which follows your theme) by default, a 256-colour or 24-bit accent where `TERM`/`COLORTERM` say it is there, and a deeper accent on a light background (`COLORFGBG`). Outside a UTF-8 locale the glyphs fall back to ASCII (`+ x ! >` and a `| / - \` spinner). The error block mutes the `ERR::` code so the sentence leads, accents the hint, aligns its keys, wraps long text under its own column and makes URLs clickable (OSC 8) in iTerm2, WezTerm, kitty, Windows Terminal, VS Code, GNOME Terminal and other terminals that have them (`FORCE_HYPERLINK=0/1` overrides). The spinner measures the terminal on every frame, so it follows a resize and never wraps. New in `internal/ui`, not yet used by a command: a determinate progress bar with a count and an ETA (`Spinner.Progress(done, total)`).
 - **Retries are safe.** A connection that drops after the request was sent now exits 12 ("may have run and been billed"), not 10 ("nothing was sent, safe to retry"). A gateway 5xx without an API problem document is retried only for GET/HEAD, and is no longer mistaken for the target's status under `--original-status`. A 301/302/303 on a POST is an error naming the Location instead of a silent GET. `Retry-After` waits are capped at 60 s and announced on stderr.
 - **Ctrl-C exits 130** with "interrupted", including while `scrape -` waits on stdin. A mistyped subcommand (`spicrawl batch wiat`) is a usage error with a suggestion, not help and exit 0.
 - **JSON mode keeps stderr clean** (**behaviour change**). With `--json`, or whenever stdout is not a terminal, stderr is now the problem document of a failure and nothing else; it is empty on success. Warnings and status lines ("wrote N bytes", "more results: --cursor ...", the `--engine` deprecation notice, the `ws://` warning) used to print there as prose, sometimes before the problem document, so `cmd --json 2>err.json; jq . err.json` failed. They still print in human mode. The API's `warnings` field is still in the JSON on stdout.

DESIGN.md

@@ -28,6 +28,14 @@ never invent a field the API does not accept (the API rejects unknown fields).
    stdout and its one-line message ("no sessions") on stderr, so
    `spicrawl sessions list | wc -l` is 0 and `-q` silences the message. JSON
    mode prints the empty collection on stdout.
+   The same gate (plus `CLICOLOR=0`) covers the grouped, coloured `--help`,
+   which is written to stdout and so tests stdout; piped, the help is cobra's own.
+   Inside the gate the UI adapts to what the terminal can do (`ui.Caps`): colour
+   is a semantic role (accent, success, warn, danger, muted) resolved to the
+   16-colour palette, the 256-colour cube or 24-bit colour; glyphs fall back to
+   ASCII outside a UTF-8 locale; hyperlinks are OSC 8 only on terminals known to
+   have them. Commands never write escape codes themselves, they go through
+   `ui.UI` (or `internal/cmd/view.go`), so a new surface gets all of that.
 2. **JSON when not a terminal.** `--json` forces it. Human mode prints tables
    or the document itself.
 3. **Never prompts without a TTY.** Anything interactive has a flag form.
@@ -116,7 +124,7 @@ internal/api        HTTP client, Problem, DoWithRetry
 internal/config     key/base resolution, config file
 internal/exitcode   exit codes
 internal/output     Printer (JSON/human/JSONL, stderr info and warnings in human mode only)
-internal/ui         stderr spinner, summary line, error block, colour (off unless a terminal)
+internal/ui         terminal output: spinner and progress bar, summary line, error block, colour roles, glyphs, width-aware text, capability detection (off unless a terminal)
 internal/cmd        one file per command group; each registers in init()
 ```
 

README.md

@@ -205,7 +205,9 @@ The key is passed as an argument, so other users on the machine can see it in th
 
 When stderr is a terminal, a wait longer than about 300 ms shows a spinner with what the CLI is doing and how long it has taken (`⠹ Scraping example.com/pricing (rendering JavaScript) · 4.2s`, `⠸ Batch job 3/10 done · 12.1s`). It is cleared before the result prints, and `scrape`, `scrape -` and `batch wait` end with a `✓` or `✗` line on stderr: page, target status, credits charged, duration. API errors keep their `ERR::` code and hint in a short block.
 
-All of it is on stderr and stdout is byte-for-byte unchanged. It switches itself off, and the CLI prints exactly what it printed before, when stderr is not a terminal (a pipe, a file, an agent or CI log), with `--json` or `-q/--quiet`, with `NO_COLOR` set, with `TERM=dumb`, or when `CI` is set.
+On a terminal `spicrawl --help` lists the commands grouped by purpose, with examples and the docs link; piped, the help is the plain text it always was. Colour follows your terminal: your theme's own palette by default, richer where the terminal advertises it, ASCII symbols outside a UTF-8 locale.
+
+All of it is on stderr and stdout is byte-for-byte unchanged. It switches itself off, and the CLI prints exactly what it printed before, when stderr is not a terminal (a pipe, a file, an agent or CI log), with `--json` or `-q/--quiet`, with `NO_COLOR` set (or `CLICOLOR=0`), with `TERM=dumb`, or when `CI` is set.
 
 ### How to convert a web page to Markdown from the command line
 

internal/cmd/help.go

@@ -0,0 +1,311 @@
+package cmd
+
+// The --help a person sees on a terminal. Cobra's own help is plain and
+// alphabetical; here, when stdout is a live terminal (ui.Live: a TTY, and no
+// NO_COLOR, TERM=dumb or CI), the root's commands are grouped by what they are
+// for, headings and the names you type are coloured, and long descriptions
+// wrap under their own column. Everywhere else, piped, redirected, in CI, for
+// an agent, the help is cobra's, byte for byte.
+//
+// It is installed with SetHelpFunc from this init, so no command file changes:
+// every command's help (spicrawl x --help, spicrawl help x, a bare group) goes
+// through the root's help func.
+
+import (
+	"fmt"
+	"io"
+	"regexp"
+	"strings"
+
+	"github.com/spf13/cobra"
+	"github.com/spf13/pflag"
+
+	"github.com/OfficialSpicrawl/cli/internal/ui"
+)
+
+const helpDocsURL = "https://docs.spicrawl.com"
+
+// helpMaxWidth caps the line length: a 250-column terminal does not get 250
+// columns of prose.
+const helpMaxWidth = 100
+
+func init() {
+	plain := rootCmd.HelpFunc() // cobra's own, kept for everything that is not a live terminal
+	rootCmd.SetHelpFunc(func(c *cobra.Command, args []string) {
+		if u := helpUI(c.OutOrStdout()); u.On() {
+			io.WriteString(c.OutOrStdout(), helpRender(u, c))
+			return
+		}
+		plain(c, args)
+	})
+}
+
+// helpUI is the live terminal for a help written to w: on only when w is a
+// terminal and nothing asked for plain output: NO_COLOR, TERM=dumb, CI (see
+// ui.Live), and --json or --quiet, which promise no escape codes at all. Tests
+// force it.
+func helpUI(w io.Writer) *ui.UI {
+	switch {
+	case flagJSON || flagQuiet:
+		return nil
+	case viewForce:
+		return ui.Force(w)
+	}
+	return ui.New(w)
+}
+
+// helpGroups orders the root's commands. A command in none of them lands in a
+// last "More commands" group, so a new command shows up without touching this.
+var helpGroups = []struct {
+	title string
+	names []string
+}{
+	{"Extract data", []string{"scrape", "batch", "sessions", "browser"}},
+	{"Set up", []string{"login", "logout", "auth", "init", "mcp", "skill", "config"}},
+	{"Monitor", []string{"status", "logs", "usage"}},
+	{"Reference", []string{"docs", "schema", "exit-codes", "version", "completion", "help"}},
+}
+
+// helpRootExamples are the first things to try. Every line parses (see
+// TestHelpRootExamplesAreRealCommands), so they cannot rot.
+const helpRootExamples = `  # Save your API key once
+  spicrawl login
+
+  # A page as clean markdown
+  spicrawl scrape https://example.com --format markdown
+
+  # A JavaScript-rendered page, waiting for the data to appear
+  spicrawl scrape https://example.com/pricing --render --wait-for '.plans' --format markdown
+
+  # Many URLs as one job, followed until it finishes
+  spicrawl batch submit urls.txt --render --wait
+
+  # Is everything reachable, and does my key work?
+  spicrawl status`
+
+// apiRoute matches the "(GET /v1/requests)" some Short texts end with: useful
+// to a script author, noise to a reader, so the screen mutes it.
+var apiRoute = regexp.MustCompile(`\s*\(((?:GET|POST|PUT|PATCH|DELETE) /v1/[^)]*)\)\s*$`)
+
+func helpRender(u *ui.UI, c *cobra.Command) string {
+	width := helpMaxWidth
+	if cols := u.Cols(); cols > 0 {
+		width = min(cols-1, helpMaxWidth) // one short of the row: see ui.Fail
+	}
+	var b strings.Builder
+	section := func(title string) { b.WriteString("\n" + u.Bold(strings.ToUpper(title)) + "\n") }
+
+	if text := strings.TrimRight(firstNonEmpty(c.Long, c.Short), " \n"); text != "" {
+		b.WriteString(text + "\n")
+	}
+
+	section("Usage")
+	switch {
+	case c.HasAvailableSubCommands():
+		fmt.Fprintf(&b, "  %s %s\n", c.CommandPath(), u.Dim("[command]"))
+	case c.Runnable():
+		b.WriteString("  " + c.UseLine() + "\n")
+	}
+	if len(c.Aliases) > 0 {
+		fmt.Fprintf(&b, "  %s %s\n", u.Dim("aliases:"), strings.Join(c.Aliases, ", "))
+	}
+
+	var subs []*cobra.Command
+	for _, s := range c.Commands() {
+		if s.IsAvailableCommand() || s.Name() == "help" {
+			subs = append(subs, s)
+		}
+	}
+	pad := 0 // one name column for every group, so they line up
+	for _, s := range subs {
+		pad = max(pad, ui.Width(s.Name()))
+	}
+	if c.Parent() == nil {
+		placed := map[string]bool{}
+		for _, g := range helpGroups {
+			var in []*cobra.Command
+			for _, n := range g.names {
+				for _, s := range subs {
+					if s.Name() == n {
+						in = append(in, s)
+						placed[n] = true
+					}
+				}
+			}
+			helpCommands(&b, u, section, g.title, in, pad, width)
+		}
+		var rest []*cobra.Command
+		for _, s := range subs {
+			if !placed[s.Name()] {
+				rest = append(rest, s)
+			}
+		}
+		helpCommands(&b, u, section, "More commands", rest, pad, width)
+	} else {
+		helpCommands(&b, u, section, "Commands", subs, pad, width)
+	}
+
+	ex := c.Example
+	if c.Parent() == nil {
+		ex = helpRootExamples
+	}
+	if ex != "" {
+		section("Examples")
+		for _, l := range strings.Split(strings.TrimRight(ex, "\n"), "\n") {
+			if strings.HasPrefix(strings.TrimSpace(l), "#") {
+				l = u.Dim(l)
+			}
+			b.WriteString(l + "\n")
+		}
+	}
+
+	if s := helpFlags(u, c.LocalFlags(), width); s != "" {
+		section("Flags")
+		b.WriteString(s)
+	}
+	if s := helpFlags(u, c.InheritedFlags(), width); s != "" {
+		section("Global flags")
+		b.WriteString(s)
+	}
+
+	if c.Parent() == nil {
+		section("Learn more")
+		b.WriteString("  " + u.Dim("Docs") + "          " + u.Link(helpDocsURL, helpDocsURL) + "\n")
+		b.WriteString("  " + u.Dim("Exit codes") + "    spicrawl exit-codes\n")
+	}
+	if c.HasAvailableSubCommands() {
+		fmt.Fprintf(&b, "\nUse %s for more information about a command.\n",
+			u.Accent(fmt.Sprintf("%q", c.CommandPath()+" [command] --help")))
+	}
+	return b.String()
+}
+
+// helpCommands writes one titled list of commands: names in the accent colour,
+// descriptions wrapped under their own column.
+func helpCommands(b *strings.Builder, u *ui.UI, section func(string), title string, cmds []*cobra.Command, pad, width int) {
+	if len(cmds) == 0 {
+		return
+	}
+	section(title)
+	for _, s := range cmds {
+		short, route := s.Short, ""
+		if m := apiRoute.FindStringSubmatchIndex(short); m != nil {
+			short, route = short[:m[0]], short[m[2]:m[3]]
+		}
+		room := width - 2 - pad - 2
+		lines := ui.Wrap(short, room)
+		if route != "" {
+			r := "(" + route + ")"
+			if last := len(lines) - 1; ui.Width(lines[last])+1+ui.Width(r) <= room {
+				lines[last] += " " + u.Dim(r)
+			} else {
+				lines = append(lines, u.Dim(r))
+			}
+		}
+		for i, l := range lines {
+			name := strings.Repeat(" ", pad)
+			if i == 0 {
+				name = u.Accent(s.Name()) + strings.Repeat(" ", pad-ui.Width(s.Name()))
+			}
+			b.WriteString("  " + name + "  " + l + "\n")
+		}
+	}
+}
+
+// helpFlags lays out a flag set as a table: "-q, --quiet", a muted value type,
+// then the description wrapped under its own column with a muted default.
+func helpFlags(u *ui.UI, fs *pflag.FlagSet, width int) string {
+	type row struct{ plain, left, desc string }
+	var rows []row
+	left := 0
+	fs.VisitAll(func(f *pflag.Flag) {
+		if f.Hidden {
+			return
+		}
+		name, usage := pflag.UnquoteUsage(f)
+		plain, styled := "    --"+f.Name, "    "+u.Accent("--"+f.Name)
+		if f.Shorthand != "" {
+			plain, styled = "-"+f.Shorthand+", --"+f.Name, u.Accent("-"+f.Shorthand)+", "+u.Accent("--"+f.Name)
+		}
+		if name != "" {
+			plain += " " + name
+			styled += " " + u.Dim(name)
+		}
+		if !zeroDefault(f) {
+			d := f.DefValue
+			if f.Value.Type() == "string" {
+				d = fmt.Sprintf("%q", d)
+			}
+			// Joined with a private-use rune, which Wrap does not break at and
+			// measures as one cell; swapped back for a space below.
+			usage += " " + strings.ReplaceAll("(default "+d+")", " ", "\ue000")
+		}
+		rows = append(rows, row{plain, styled, usage})
+		left = max(left, ui.Width(plain))
+	})
+	if len(rows) == 0 {
+		return ""
+	}
+	// A very long flag name must not squeeze every description into a sliver.
+	left = min(left, 34)
+	room := width - 2 - left - 2
+	var b strings.Builder
+	for _, r := range rows {
+		if room < 24 || ui.Width(r.plain) > left { // narrow terminal or long name: description below
+			b.WriteString("  " + r.left + "\n")
+			for _, l := range ui.Wrap(r.desc, max(width-8, 20)) {
+				b.WriteString("        " + strings.ReplaceAll(l, "\ue000", " ") + "\n")
+			}
+			continue
+		}
+		for i, l := range ui.Wrap(r.desc, room) {
+			l = strings.ReplaceAll(l, "\ue000", " ")
+			lead := strings.Repeat(" ", left)
+			if i == 0 {
+				lead = r.left + strings.Repeat(" ", left-ui.Width(r.plain))
+			}
+			b.WriteString("  " + lead + "  " + helpMuteDefault(u, l) + "\n")
+		}
+	}
+	return b.String()
+}
+
+// helpMuteDefault mutes a trailing "(default x)", which is never split across
+// lines (see helpFlags).
+func helpMuteDefault(u *ui.UI, l string) string {
+	if i := strings.LastIndex(l, "(default "); i >= 0 && strings.HasSuffix(l, ")") {
+		return l[:i] + u.Dim(l[i:])
+	}
+	return l
+}
+
+// zeroDefault mirrors pflag: a flag whose default is its type's zero value
+// shows no "(default ...)".
+func zeroDefault(f *pflag.Flag) bool {
+	switch f.Value.Type() {
+	case "bool":
+		return f.DefValue == "false"
+	case "duration":
+		return f.DefValue == "0" || f.DefValue == "0s"
+	case "int", "int8", "int32", "int64", "uint", "uint8", "uint16", "uint32", "uint64", "count", "float32", "float64":
+		return f.DefValue == "0"
+	case "string":
+		return f.DefValue == ""
+	case "stringSlice", "stringArray", "intSlice":
+		return f.DefValue == "[]"
+	}
+	switch f.Value.String() {
+	case "false", "<nil>", "", "0":
+		return true
+	}
+	return false
+}
+
+func firstNonEmpty(ss ...string) string {
+	for _, s := range ss {
+		if s != "" {
+			return s
+		}
+	}
+	return ""
+}