diff --git a/CHANGELOG.md b/CHANGELOG.md index 53924d4..fefba3d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,20 @@ # Changelog +## Unreleased + +Bug fixes from a review of 0.1.3. No new commands; two behaviour changes are marked. + +- **Retries are safe.** A connection that drops after the request was sent now exits 12 ("may have run and been billed"), not 10 ("nothing was sent, safe to retry"). A gateway 5xx without an API problem document is retried only for GET/HEAD, and is no longer mistaken for the target's status under `--original-status`. A 301/302/303 on a POST is an error naming the Location instead of a silent GET. `Retry-After` waits are capped at 60 s and announced on stderr. +- **Ctrl-C exits 130** with "interrupted", including while `scrape -` waits on stdin. A mistyped subcommand (`spicrawl batch wiat`) is a usage error with a suggestion, not help and exit 0. +- **Keys:** a trailing CR/LF or space on the key (flag, env or file) is trimmed. The config file is written via a private temp file, short keys are fully masked, and commands that get the key and base URL from flags work without `$HOME`. +- **scrape:** `--body null` is a usage error, not a panic (also `batch submit`). Big integers survive `--json` (**key order now follows the API**, as `--jsonl` already did). Stdout write errors stop the run instead of exiting 0. `-o` writes atomically and **a target error page no longer overwrites the file**. `-o -` means stdout. Screenshot labels never overwrite each other. `--body` merges deeply. A BOM on stdin is stripped. Human output no longer HTML-escapes `<>&`. The multi-URL summary counts correctly. `--impersonate` help says it is on by default. +- **batch:** `wait` and `submit --wait` exit 4 when the job failed or was cancelled. Polling survives transient 5xx and network errors. `results -o` no longer empties an existing file on failure. +- **browser:** a server without the cloud browser says so (exit 8) instead of `NOT_FOUND`. `$(spicrawl browser url -q)` yields the bare URL. +- **usage summary** shows the enforced monthly allowance. **status** fails when the key check cannot reach the API. +- **mcp install / init** back up the client config to `.bak`, write through symlinks, refuse to replace a customised `spicrawl` entry without `--force`, and accept VS Code JSONC/BOM `mcp.json`. +- **Docs:** the embedded skill, `openapi.yaml` and schemas are synced with the API (cache hits are billed, `impersonate` defaults to true, batch honours `--format`). +- **install.sh** accepts `SPICRAWL_VERSION=0.1.3` as well as `v0.1.3` and explains a failed download. The npm launcher forwards Ctrl-C to the binary. + ## 0.1.3 — 2026-10-05 - The repository and Go module moved to `github.com/OfficialSpicrawl/cli`. Install from source with `go install github.com/OfficialSpicrawl/cli/cmd/spicrawl@latest`. diff --git a/DESIGN.md b/DESIGN.md index 0167139..669f814 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -129,7 +129,7 @@ spicrawl batch list [--status S] [--limit N] spicrawl batch get spicrawl batch results [--status S] [--all] [-o FILE] JSONL spicrawl batch content [-o FILE] one item's document -spicrawl batch wait [--max-wait 30m] exit 11 when it gives up +spicrawl batch wait [--max-wait 30m] exit 11 when it gives up, 4 if failed/cancelled spicrawl batch cancel [--yes] --yes required without a TTY spicrawl batch retry|close spicrawl batch append diff --git a/README.md b/README.md index 0271aa4..7f84a46 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Spicrawl CLI: scrape websites to Markdown and JSON from the terminal -The Spicrawl CLI is a command-line web scraper that turns any URL into Markdown, HTML, text or JSON, batch-scrapes up to 10,000 URLs per job, and connects Claude Code, Cursor, Codex and VS Code to Spicrawl over MCP, with JSON output and 13 stable exit codes for shell scripts, CI and AI agents. +The Spicrawl CLI is a command-line web scraper that turns any URL into Markdown, HTML, text or JSON, batch-scrapes up to 10,000 URLs per job, and connects Claude Code, Cursor, Codex and VS Code to Spicrawl over MCP, with JSON output and 14 stable exit codes for shell scripts, CI and AI agents. [![npm version](https://img.shields.io/npm/v/@spicrawl/cli.svg)](https://www.npmjs.com/package/@spicrawl/cli) [![npm downloads](https://img.shields.io/npm/dm/@spicrawl/cli.svg)](https://www.npmjs.com/package/@spicrawl/cli) @@ -50,7 +50,7 @@ The result is one JSON object with the page under `content`: - **Parallel scrapes from stdin.** `spicrawl scrape -` reads one URL per line and runs 4 requests at once by default (`--concurrency`), with every scrape flag available. - **Structured data without a model.** `--extract` (CSS selectors), `--autoparse` (JSON-LD, OpenGraph) and `--links`. - **One-command agent setup.** `spicrawl init` configures 4 clients (Claude Code, Cursor, VS Code and Codex) for the hosted MCP server and installs the agent skill. -- **13 stable exit codes (0 to 12).** Each maps to an error class an agent can branch on; codes are never renumbered, only added. +- **14 stable exit codes (0 to 12, and 130).** Each maps to an error class an agent can branch on; codes are never renumbered, only added. - **Offline request schemas.** `spicrawl schema` prints 4 JSON Schemas (`scrape`, `batch`, `batch-item`, `session`) from inside the binary. - **6 platforms.** Linux, macOS and Windows on x64 and arm64, built with GoReleaser. @@ -71,7 +71,7 @@ npx @spicrawl/cli --help # or run it without installing curl -fsSL https://raw.githubusercontent.com/OfficialSpicrawl/cli/main/install.sh | sh ``` -It downloads the archive for your OS and CPU from the latest GitHub release, checks its SHA-256 against `checksums.txt`, and installs `spicrawl` into `~/.local/bin` (`SPICRAWL_INSTALL_DIR` changes it; `SPICRAWL_VERSION=v0.1.3` pins a release). It never prompts and never uses `sudo`. +It downloads the archive for your OS and CPU from the latest GitHub release, checks its SHA-256 against `checksums.txt`, and installs `spicrawl` into `~/.local/bin` (`SPICRAWL_INSTALL_DIR` changes it; `SPICRAWL_VERSION=v0.1.3` or `0.1.3` pins a release). It never prompts and never uses `sudo`. ### GitHub releases (prebuilt archives) @@ -224,12 +224,12 @@ Screenshots are saved next to the `-o` file as `.