Fix bugs found in a review of 0.1.3

CHANGELOG.md

@@ -1,5 +1,20 @@
 # Changelog
 
+## Unreleased
+
+Bug fixes from a review of 0.1.3. No new commands; two behaviour changes are marked.
+
+- **Retries are safe.** A connection that drops after the request was sent now exits 12 ("may have run and been billed"), not 10 ("nothing was sent, safe to retry"). A gateway 5xx without an API problem document is retried only for GET/HEAD, and is no longer mistaken for the target's status under `--original-status`. A 301/302/303 on a POST is an error naming the Location instead of a silent GET. `Retry-After` waits are capped at 60 s and announced on stderr.
+- **Ctrl-C exits 130** with "interrupted", including while `scrape -` waits on stdin. A mistyped subcommand (`spicrawl batch wiat`) is a usage error with a suggestion, not help and exit 0.
+- **Keys:** a trailing CR/LF or space on the key (flag, env or file) is trimmed. The config file is written via a private temp file, short keys are fully masked, and commands that get the key and base URL from flags work without `$HOME`.
+- **scrape:** `--body null` is a usage error, not a panic (also `batch submit`). Big integers survive `--json` (**key order now follows the API**, as `--jsonl` already did). Stdout write errors stop the run instead of exiting 0. `-o` writes atomically and **a target error page no longer overwrites the file**. `-o -` means stdout. Screenshot labels never overwrite each other. `--body` merges deeply. A BOM on stdin is stripped. Human output no longer HTML-escapes `<>&`. The multi-URL summary counts correctly. `--impersonate` help says it is on by default.
+- **batch:** `wait` and `submit --wait` exit 4 when the job failed or was cancelled. Polling survives transient 5xx and network errors. `results -o` no longer empties an existing file on failure.
+- **browser:** a server without the cloud browser says so (exit 8) instead of `NOT_FOUND`. `$(spicrawl browser url -q)` yields the bare URL.
+- **usage summary** shows the enforced monthly allowance. **status** fails when the key check cannot reach the API.
+- **mcp install / init** back up the client config to `.bak`, write through symlinks, refuse to replace a customised `spicrawl` entry without `--force`, and accept VS Code JSONC/BOM `mcp.json`.
+- **Docs:** the embedded skill, `openapi.yaml` and schemas are synced with the API (cache hits are billed, `impersonate` defaults to true, batch honours `--format`).
+- **install.sh** accepts `SPICRAWL_VERSION=0.1.3` as well as `v0.1.3` and explains a failed download. The npm launcher forwards Ctrl-C to the binary.
+
 ## 0.1.3 — 2026-10-05
 
 - The repository and Go module moved to `github.com/OfficialSpicrawl/cli`. Install from source with `go install github.com/OfficialSpicrawl/cli/cmd/spicrawl@latest`.

DESIGN.md

@@ -129,7 +129,7 @@ spicrawl batch list [--status S] [--limit N]
 spicrawl batch get <id>
 spicrawl batch results <id> [--status S] [--all] [-o FILE]   JSONL
 spicrawl batch content <id> <seq> [-o FILE]                   one item's document
-spicrawl batch wait <id> [--max-wait 30m]                     exit 11 when it gives up
+spicrawl batch wait <id> [--max-wait 30m]                     exit 11 when it gives up, 4 if failed/cancelled
 spicrawl batch cancel <id> [--yes]                           --yes required without a TTY
 spicrawl batch retry|close <id>
 spicrawl batch append <id> <file|->

README.md

@@ -1,6 +1,6 @@
 # Spicrawl CLI: scrape websites to Markdown and JSON from the terminal
 
-The Spicrawl CLI is a command-line web scraper that turns any URL into Markdown, HTML, text or JSON, batch-scrapes up to 10,000 URLs per job, and connects Claude Code, Cursor, Codex and VS Code to Spicrawl over MCP, with JSON output and 13 stable exit codes for shell scripts, CI and AI agents.
+The Spicrawl CLI is a command-line web scraper that turns any URL into Markdown, HTML, text or JSON, batch-scrapes up to 10,000 URLs per job, and connects Claude Code, Cursor, Codex and VS Code to Spicrawl over MCP, with JSON output and 14 stable exit codes for shell scripts, CI and AI agents.
 
 [![npm version](https://img.shields.io/npm/v/@spicrawl/cli.svg)](https://www.npmjs.com/package/@spicrawl/cli)
 [![npm downloads](https://img.shields.io/npm/dm/@spicrawl/cli.svg)](https://www.npmjs.com/package/@spicrawl/cli)
@@ -50,7 +50,7 @@ The result is one JSON object with the page under `content`:
 - **Parallel scrapes from stdin.** `spicrawl scrape -` reads one URL per line and runs 4 requests at once by default (`--concurrency`), with every scrape flag available.
 - **Structured data without a model.** `--extract` (CSS selectors), `--autoparse` (JSON-LD, OpenGraph) and `--links`.
 - **One-command agent setup.** `spicrawl init` configures 4 clients (Claude Code, Cursor, VS Code and Codex) for the hosted MCP server and installs the agent skill.
-- **13 stable exit codes (0 to 12).** Each maps to an error class an agent can branch on; codes are never renumbered, only added.
+- **14 stable exit codes (0 to 12, and 130).** Each maps to an error class an agent can branch on; codes are never renumbered, only added.
 - **Offline request schemas.** `spicrawl schema` prints 4 JSON Schemas (`scrape`, `batch`, `batch-item`, `session`) from inside the binary.
 - **6 platforms.** Linux, macOS and Windows on x64 and arm64, built with GoReleaser.
 
@@ -71,7 +71,7 @@ npx @spicrawl/cli --help       # or run it without installing
 curl -fsSL https://raw.githubusercontent.com/OfficialSpicrawl/cli/main/install.sh | sh
 ```
 
-It downloads the archive for your OS and CPU from the latest GitHub release, checks its SHA-256 against `checksums.txt`, and installs `spicrawl` into `~/.local/bin` (`SPICRAWL_INSTALL_DIR` changes it; `SPICRAWL_VERSION=v0.1.3` pins a release). It never prompts and never uses `sudo`.
+It downloads the archive for your OS and CPU from the latest GitHub release, checks its SHA-256 against `checksums.txt`, and installs `spicrawl` into `~/.local/bin` (`SPICRAWL_INSTALL_DIR` changes it; `SPICRAWL_VERSION=v0.1.3` or `0.1.3` pins a release). It never prompts and never uses `sudo`.
 
 ### GitHub releases (prebuilt archives)
 
@@ -224,12 +224,12 @@ Screenshots are saved next to the `-o` file as `<name>.<label>.<format>`, or int
 
 ```sh
 id=$(spicrawl batch submit urls.txt --render --name nightly --json | jq -r .id)
-spicrawl batch wait "$id" --max-wait 10m    # exits 11 if the job is still running
+spicrawl batch wait "$id" --max-wait 10m    # exits 11 if still running, 4 if it failed or was cancelled
 spicrawl batch results "$id" --all -o results.jsonl
 spicrawl batch retry "$id"                  # re-run the failed items
 ```
 
-Batch items currently return raw page HTML: the worker applies `--render`, `--proxy` and `--block`, and accepts but does not yet apply other scrape flags such as `--format markdown`. For Markdown or extraction per URL, use `spicrawl scrape - --concurrency 8 < urls.txt`. Results stay readable until the job's `results_expire_at` (72 hours after submission by default). More in the [batch guide](https://docs.spicrawl.com/cli/batch).
+Each batch item is one fetch or render, returned as `html` (the default), `markdown`, `text` or `json` with `--format`. Screenshots, extraction (`--extract`, `--links`), actions, sessions, cache controls and PDF output are refused with a 400 at submit, before anything is queued or charged; use `spicrawl scrape - --concurrency 8 < urls.txt` for those. `batch wait` and `batch submit --wait` print the final job either way, and exit 4 when it ended `failed` or `cancelled`, so a script can tell it from `completed`; partial results stay readable with `batch results`. Results stay readable until the job's `results_expire_at` (72 hours after submission by default). More in the [batch guide](https://docs.spicrawl.com/cli/batch).
 
 ### How to keep cookies and logins across scrapes
 
@@ -266,7 +266,7 @@ All four call the same Spicrawl API with the same API key and the same credits.
 
 - If you want an agent to use Spicrawl with the least setup, run `spicrawl init`: it writes the MCP config and the skill for you.
 - If an agent already has a shell, it can call `spicrawl` directly instead of MCP: output is JSON when piped and the exit code says what went wrong.
-- If you need Markdown for a list of URLs, use `spicrawl scrape -` rather than `spicrawl batch`, because batch items return raw HTML today.
+- If you need Markdown for a list of URLs, use `spicrawl batch submit urls.txt --format markdown` for a server-side job, or `spicrawl scrape -` for one JSON line per URL as it finishes.
 
 ## Built for scripts and AI agents
 
@@ -278,7 +278,7 @@ All four call the same Spicrawl API with the same API key and the same credits.
 
 ### Exit codes
 
-There are 13 exit codes, 0 to 12. They are stable: never renumbered, only added. `spicrawl exit-codes --json` prints them as `[{code, name, meaning}]`.
+There are 14 exit codes: 0 to 12, and 130. They are stable: never renumbered, only added. `spicrawl exit-codes --json` prints them as `[{code, name, meaning}]`.
 
 | Code | Name | Meaning | What an agent should do |
 |---|---|---|---|
@@ -294,7 +294,8 @@ There are 13 exit codes, 0 to 12. They are stable: never renumbered, only added.
 | 9 | `session` | `ERR::SESSION::*` | Create a new session; if the session is busy, wait and retry. |
 | 10 | `network` | the Spicrawl API could not be reached (DNS, refused, TLS); nothing was sent | Safe to retry after a pause; check the network and `--base-url`. |
 | 11 | `pending` | a `--wait` gave up before the job finished | The job is still running: resume with `spicrawl batch wait <id>`. |
-| 12 | `timeout` | the request was sent but `--timeout` expired; it may have run and been billed | Check `spicrawl logs` before retrying. |
+| 12 | `timeout` | the request was sent but `--timeout` expired or the connection dropped; it may have run and been billed | Check `spicrawl logs` before retrying. |
+| 130 | `interrupted` | Ctrl-C | Nothing to do; a request already sent may still have run. |
 
 ## Configuration
 
@@ -315,7 +316,7 @@ Settings resolve in this order: command-line flag, environment variable, config
 
 ## Beta limitations and coming soon
 
-Spicrawl is in beta. Coming soon: AI extraction (`--ai`, `--ai-schema`), the cloud browser (`spicrawl browser url`), the managed proxy pool (`--premium-proxy`, `--country`, `--sticky-key`), stealth mode (`--stealth`), the `spicrawl.com/install.sh` installer and Homebrew. Today, batch items return raw HTML, and there is no whole-site crawl or sitemap command: collect URLs with `--links` and scrape or batch the list.
+Spicrawl is in beta. Coming soon: AI extraction (`--ai`, `--ai-schema`), the cloud browser (`spicrawl browser url`), the managed proxy pool (`--premium-proxy`, `--country`, `--sticky-key`), stealth mode (`--stealth`), the `spicrawl.com/install.sh` installer and Homebrew. Today there is no whole-site crawl or sitemap command: collect URLs with `--links` and scrape or batch the list.
 
 ## FAQ
 

install.sh

@@ -2,11 +2,11 @@
 #
 # Install the Spicrawl CLI (the spicrawl command).
 #
-#   curl -fsSL https://spicrawl.com/install.sh | sh
+#   curl -fsSL https://raw.githubusercontent.com/OfficialSpicrawl/cli/main/install.sh | sh
 #
 # Environment:
 #   SPICRAWL_INSTALL_DIR   where to put the binary (default: ~/.local/bin)
-#   SPICRAWL_VERSION       a release tag such as v1.2.3 (default: latest)
+#   SPICRAWL_VERSION       a release such as v1.2.3 or 1.2.3 (default: latest)
 #   SPICRAWL_REPO          GitHub owner/repo hosting the releases
 #   SPICRAWL_DOWNLOAD_URL  base URL holding the archives and checksums.txt
 #                        (a mirror; overrides SPICRAWL_REPO and SPICRAWL_VERSION)
@@ -51,6 +51,12 @@ if [ "$os" = darwin ] && [ "$arch" = amd64 ]; then
   fi
 fi
 
+# Release tags carry a "v"; accept the bare version too.
+case "$SPICRAWL_VERSION" in
+  latest | v*) ;;
+  *) SPICRAWL_VERSION="v${SPICRAWL_VERSION}" ;;
+esac
+
 archive="spicrawl_${os}_${arch}.tar.gz"
 if [ -n "${SPICRAWL_DOWNLOAD_URL:-}" ]; then
   base="${SPICRAWL_DOWNLOAD_URL%/}"
@@ -82,15 +88,23 @@ fi
 
 command -v tar >/dev/null 2>&1 || die "need tar"
 
+# download URL FILE: on failure say which URL and what to check.
+download() {
+  fetch "$1" "$2" || die "download failed: $1
+  The server was unreachable or has no such file (HTTP 404 means this release or platform does not exist).
+  Check your network, SPICRAWL_VERSION (a published tag such as v0.1.3), SPICRAWL_REPO and SPICRAWL_DOWNLOAD_URL,
+  or pick a release from https://github.com/${SPICRAWL_REPO}/releases"
+}
+
 # --- download and verify ----------------------------------------------------
 
 tmp=$(mktemp -d 2>/dev/null || mktemp -d -t spicrawl)
 trap 'rm -rf "$tmp"' EXIT
 trap 'exit 130' INT TERM
 
 say "downloading ${base}/${archive}"
-fetch "${base}/${archive}" "${tmp}/${archive}" || die "download failed: ${base}/${archive}"
-fetch "${base}/checksums.txt" "${tmp}/checksums.txt" || die "download failed: ${base}/checksums.txt"
+download "${base}/${archive}" "${tmp}/${archive}"
+download "${base}/checksums.txt" "${tmp}/checksums.txt"
 
 want=$(awk -v f="$archive" '$2 == f || $2 == "*" f { print $1; exit }' "${tmp}/checksums.txt")
 [ -n "$want" ] || die "${archive} is not listed in checksums.txt"

internal/api/client.go

@@ -30,16 +30,44 @@ type Client struct {
 	BaseURL string
 	APIKey  string
 	HTTP    *http.Client
+	// Notify, when set, receives progress lines (DoWithRetry waiting between
+	// attempts). The caller decides where they go and whether --quiet mutes them.
+	Notify func(format string, a ...any)
 }
 
 func New(baseURL, apiKey string, timeout time.Duration) *Client {
 	return &Client{
 		BaseURL: strings.TrimRight(baseURL, "/"),
 		APIKey:  apiKey,
-		HTTP:    &http.Client{Timeout: timeout},
+		HTTP:    &http.Client{Timeout: timeout, CheckRedirect: noMethodChangingRedirect},
 	}
 }
 
+// noMethodChangingRedirect follows redirects that keep the method (307/308,
+// or any redirect of a GET) and hands back the 3xx for the rest. Following a
+// 301/302/303 would silently turn a POST into a GET: the call would "succeed"
+// without the request ever being made. Do reports the 3xx as a *RedirectError.
+func noMethodChangingRedirect(req *http.Request, via []*http.Request) error {
+	if len(via) >= 10 {
+		return errors.New("stopped after 10 redirects")
+	}
+	if req.Method != via[0].Method {
+		return http.ErrUseLastResponse
+	}
+	return nil
+}
+
+// RedirectError means the API answered with a redirect the CLI will not
+// follow. It is almost always a wrong --base-url (http instead of https).
+type RedirectError struct {
+	Status   int
+	Location string
+}
+
+func (e *RedirectError) Error() string {
+	return fmt.Sprintf("the API answered %d and redirects to %s; the CLI does not follow redirects that would change POST to GET: use that URL as --base-url / SPICRAWL_BASE_URL", e.Status, e.Location)
+}
+
 // Response is a completed call that the API answered with a 2xx.
 type Response struct {
 	Status int
@@ -76,6 +104,17 @@ type NetworkError struct{ Err error }
 func (e *NetworkError) Error() string { return "cannot reach the Spicrawl API: " + e.Err.Error() }
 func (e *NetworkError) Unwrap() error { return e.Err }
 
+// ConnectionLostError means the connection failed after the request was
+// written (reset, EOF, broken body). The server may have run the request,
+// and billed it, so blindly retrying can pay twice. It exits like a timeout.
+type ConnectionLostError struct{ Err error }
+
+func (e *ConnectionLostError) Error() string {
+	return "lost the connection to the Spicrawl API after the request was sent; the request may have run and been billed. " +
+		"Check `spicrawl logs` before retrying (" + e.Err.Error() + ")"
+}
+func (e *ConnectionLostError) Unwrap() error { return e.Err }
+
 // TimeoutError means the request was sent but no complete answer arrived
 // before the client-side timeout (--timeout). The server may have run the
 // request, and billed it, so blindly retrying can pay twice.
@@ -100,14 +139,18 @@ func isTimeout(err error) bool {
 }
 
 // transportError classifies a failure of the HTTP round trip. sent reports
-// whether the request headers were written to the connection: a timeout
-// before that point is a connection failure (nothing reached the API), a
-// timeout after it may have left the request running server-side.
+// whether the request headers were written to the connection. Before that
+// point (dial, DNS, refused, TLS) nothing reached the API: a NetworkError. After
+// it the server may have the request and may have run it: a timeout is a
+// TimeoutError, any other failure a ConnectionLostError. Neither is safe to retry.
 func transportError(err error, sent bool) error {
-	if sent && isTimeout(err) {
+	switch {
+	case !sent:
+		return &NetworkError{Err: err}
+	case isTimeout(err):
 		return &TimeoutError{Err: err}
 	}
-	return &NetworkError{Err: err}
+	return &ConnectionLostError{Err: err}
 }
 
 // Do sends the request. A non-2xx answer is returned as *Problem.
@@ -160,14 +203,22 @@ func (c *Client) Do(ctx context.Context, req Request) (*Response, error) {
 	defer resp.Body.Close()
 	b, err := io.ReadAll(resp.Body)
 	if err != nil {
-		if ctx.Err() == nil && isTimeout(err) {
-			return nil, &TimeoutError{Err: fmt.Errorf("read response: %w", err)}
+		if ctx.Err() != nil {
+			return nil, ctx.Err()
 		}
-		return nil, &NetworkError{Err: fmt.Errorf("read response: %w", err)}
+		// The request was sent: only the answer is missing.
+		return nil, transportError(fmt.Errorf("read response: %w", err), true)
 	}
 	if resp.StatusCode < 200 || resp.StatusCode > 299 {
 		if req.AllowNonProblem && !strings.HasPrefix(resp.Header.Get("Content-Type"), "application/problem+json") {
-			return &Response{Status: resp.StatusCode, Header: resp.Header, Body: b}, nil
+			// A target's status is relayed with X-Target-Status. A 3xx or 5xx without
+			// it is the API's own redirect or a gateway page in front of it.
+			if resp.Header.Get("X-Target-Status") != "" || (resp.StatusCode >= 400 && resp.StatusCode < 500) {
+				return &Response{Status: resp.StatusCode, Header: resp.Header, Body: b}, nil
+			}
+		}
+		if loc := resp.Header.Get("Location"); resp.StatusCode/100 == 3 && loc != "" {
+			return nil, &RedirectError{Status: resp.StatusCode, Location: loc}
 		}
 		return nil, parseProblem(resp, b)
 	}
@@ -222,7 +273,10 @@ func parseProblem(resp *http.Response, b []byte) *Problem {
 			Title:  http.StatusText(resp.StatusCode),
 			Detail: strings.TrimSpace(truncate(string(b), 300)),
 		}
-		p.Retryable = resp.StatusCode >= 500
+		// A gateway page says nothing about whether the API ran the request, so
+		// only a read is safe to resend; a POST may pay twice.
+		p.Retryable = resp.StatusCode >= 500 && resp.Request != nil &&
+			(resp.Request.Method == http.MethodGet || resp.Request.Method == http.MethodHead)
 	}
 	if p.Status == 0 {
 		p.Status = resp.StatusCode
@@ -245,8 +299,12 @@ func AsProblem(err error) (*Problem, bool) {
 	return p, ok
 }
 
+// maxRetryWait caps one wait between attempts: a Retry-After of an hour must
+// not hang the CLI.
+var maxRetryWait = time.Minute
+
 // DoWithRetry retries only what the server marked retryable, honouring
-// Retry-After, with exponential backoff capped at 30 s. attempts <= 1 means
+// Retry-After (capped at maxRetryWait), with exponential backoff capped at 30 s. attempts <= 1 means
 // no retry. Transport failures are never retried here; in particular a
 // *TimeoutError is returned at once, because the request may already have
 // run and been billed (a POST retried after a timeout can pay twice).
@@ -269,6 +327,12 @@ func (c *Client) DoWithRetry(ctx context.Context, req Request, attempts int) (*R
 		if p.RetryAfterSeconds > 0 {
 			wait = time.Duration(p.RetryAfterSeconds) * time.Second
 		}
+		if wait > maxRetryWait {
+			wait = maxRetryWait
+		}
+		if c.Notify != nil {
+			c.Notify("retrying in %s (attempt %d of %d): %s", wait, i+1, attempts, p.Code)
+		}
 		select {
 		case <-ctx.Done():
 			return nil, ctx.Err()