Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion sensors/network.md
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,7 @@ kill_switch_file: /etc/openctem/STOP
| `checks.allow` | Job types: `scan`, `validate`, `retest`, `collect`, `refresh_content` |
| `allow_custom_templates`, `allow_interactsh` | Platform-supplied custom templates (off unless `true`); out-of-band callbacks (from sensor v0.38.4 they follow the platform's job unless `allow_interactsh: false`; before, off unless `true`) |
| `rate.max_rps`, `rate.max_job_seconds` | Request rate ceiling; longest job run |
| `rate.max_host_rps`, `rate.max_host_inflight` | The most requests per second (1 to 200) and requests in flight (1 to 50) toward **one target host**: the ceiling no job passes. See [How hard a scan works one host](tool-settings.md#how-hard-a-scan-works-one-host) |
| `rate.max_host_rps`, `rate.max_host_inflight` | The operator's ceiling for **one target host**: requests per second (1 to 200) and requests in flight (1 to 50). One of the limits of a job's per-host budget, of which the lowest wins: no job and no sensor level is above it. See [How hard a scan works one host](tool-settings.md#how-hard-a-scan-works-one-host) |
| `kill_switch`, `kill_switch_file` | Stop every job (while the file exists) |
| `http.user_agent`, `http.user_agents`, `http.allow_insecure_tls`, `http.headers` | A User-Agent for every tool, or a rotation list of up to 200 (one per target host and run; set one of the two); `false` refuses tools that skip TLS verification; up to 4 headers that mark this sensor's scan traffic (below) |
| `managed.accept` | `false`: ignore any narrowing policy the platform manages for this sensor. Default `true` |
Expand Down
2 changes: 1 addition & 1 deletion sensors/settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ removed name is set, and its error names the replacement:
| <span id="SSL_CERT_DIR"></span>`SSL_CERT_DIR` | path | | | | System trust store directory override. |
| <span id="SENSOR_STATE_DIR"></span>`SENSOR_STATE_DIR` | path | | `/var/lib/openctem/state` | | Local state: the paired identity and signing key (identity/), the renewed API key and the tool cost history. Mount a persistent volume. |
| <span id="SENSOR_KEY_AUTORENEW"></span>`SENSOR_KEY_AUTORENEW` | bool | | `auto` | | API key auto-renewal: true, false, or unset (on when the state directory persists). |
| <span id="SENSOR_PACING_LEVEL"></span>`SENSOR_PACING_LEVEL` | enum | | `standard` | | What one target host gets from a job that carries no signed budget: `gentle` (5 requests per second, 2 in flight), `standard` (20, 8) or `fast` (100, 25). The local policy's `rate.max_host_rps` and `rate.max_host_inflight` cap it. An unknown value is reported at start and the sensor runs at `gentle`. See [How hard a scan works one host](tool-settings.md#how-hard-a-scan-works-one-host). |
| <span id="SENSOR_PACING_LEVEL"></span>`SENSOR_PACING_LEVEL` | enum | | `standard` | | The sensor's pacing level, the most one target host gets from a job here: `gentle` (5 requests per second, 2 in flight), `standard` (20, 8) or `fast` (100, 25). A job's signed budget and the local policy's `rate.max_host_rps` and `rate.max_host_inflight` can only lower it; no job raises it. An unknown value is reported at start and the sensor runs at `gentle`. See [How hard a scan works one host](tool-settings.md#how-hard-a-scan-works-one-host). |
| <span id="SENSOR_MAX_JOBS"></span>`SENSOR_MAX_JOBS` | int | | | | Cap on concurrent jobs, 1-100. Unset: the slots follow CPU, memory and tool costs. |
| <span id="SENSOR_JOB_ID"></span>`SENSOR_JOB_ID` | string | | | | Run the one platform command with this id and exit (or `-job`; a Kubernetes Job, see [Kubernetes](deploy-kubernetes.md)). It is not in the settings registry, so **Setup & health** lists it as unknown (`config.env_unknown`); that warning is harmless here. |
| <span id="SENSOR_DRAIN_GRACE"></span>`SENSOR_DRAIN_GRACE` | duration | | `30s` | | How long running jobs may finish on shutdown. |
Expand Down
77 changes: 62 additions & 15 deletions sensors/tool-settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,31 +159,78 @@ Related: [Tools and capabilities](../scanning/tools.md),

## How hard a scan works one host

A scan never sends one target host more than its **per-host budget**: 20 requests per second and 8
requests in flight by default (the Standard level). The budget is counted per host, whatever the scheme,
port or path of the scan's targets, and whatever tool runs.
The tools that send requests to a scan's targets run under a **per-host budget**: 20 requests per second
and 8 requests in flight toward one host by default (the Standard level). The budget is counted per host,
whatever the scheme, port, path or spelling of the scan's targets.

| Level | Requests per second per host | In flight per host |
|---|---|---|
| Gentle | 5 | 2 |
| Standard (default) | 20 | 8 |
| Fast | 100 | 25 |

- The sensor's operator picks the level with [`SENSOR_PACING_LEVEL`](settings.md#SENSOR_PACING_LEVEL) and
sets a ceiling no job passes with `rate.max_host_rps` and `rate.max_host_inflight` in the
[local policy](network.md).
- A rate or concurrency setting above the budget is lowered to it. A lower setting stays.
- With the tool sandbox on (the default), the sensor enforces the budget outside the tools, for nuclei,
httpx, katana and path discovery. With `SENSOR_SANDBOX=off` the tools' own flags enforce it, and a run
over several hosts gets one host's budget in all.
### The lowest limit wins

**When a target pushes back.** A host that answers 429 or 503 is slowed down (its rate is halved, its
A job's budget is, for each of the two numbers, the **lowest** of:

1. the sensor's level, which the sensor's operator picks with
[`SENSOR_PACING_LEVEL`](settings.md#SENSOR_PACING_LEVEL);
2. the budget the platform signs into the job, when it sends one;
3. the operator's ceiling, `rate.max_host_rps` and `rate.max_host_inflight` in the
[local policy](network.md);
4. the built-in ceiling of 200 requests per second and 50 in flight.

A job can only lower what a host gets. A job signed for Fast on a Standard sensor runs at Standard; a job
signed for Gentle on a Fast sensor runs at Gentle. To scan your own hosts faster, raise the level on the
sensor. A job that asks for more is not refused: it runs at the lower limit, and its result says which
limit set each number and what was asked for.

A rate or concurrency setting above the budget is lowered to it, and the run's result and log say what was
asked and what applies. A lower setting stays.

### Which tools

| Tools | Paced by |
|---|---|
| nuclei (scans, re-verification and retests), httpx, katana, path discovery, screenshots, service detection | The per-host budget |
| Port scan (naabu) | Packets per second of the budget's level (100, 300 or 1000), not the request budget |
| DNS (dnsx), passive discovery (subfinder), lookups (asn, rdap), code scans (trivy, semgrep, opengrep, codeql, betterleaks) | They send nothing to the target hosts |

### What enforces it

**With the tool sandbox on (the default)**, every tool connection passes through the sensor, which holds
each host to the budget outside the tools. Names that resolve to one address share that address's budget.
The sensor holds:

- every new connection, and the number of connections open to a host;
- every request over plain HTTP;
- every request over HTTPS, including on connections the tool keeps open, for the scan tools the sensor
runs (HTTP/1.1, and HTTP/2 from a tool that sends each request on its own). The sensor counts them from
the outside and does not read the traffic.

It does **not** count requests that a client packs together on one HTTPS connection (a browser that loads
a page over HTTP/2, as the screenshot tool does), nor how many are in flight on such a connection: it
cannot tell them apart without reading them. For those, the tools' own rate and concurrency flags, which
the sensor derives from the same budget, are what holds the budget.

**With `SENSOR_SANDBOX=off`**, nothing is in the path. The tools' own flags are the only enforcement, a
run gets one host's budget in all whatever its target list, and nothing reacts to what the hosts answer.
The run's result says that its pacing is limited.

### When a target pushes back

A host that answers the tool's requests with 429 or 503 is slowed down (its rate is halved, its
`Retry-After` is waited out) and sped up again gradually when it answers normally. A host that answers
nothing but push-back is paused, and the run reports it as a coverage gap `throttled` with counts; the run
completes as partial and keeps everything it found. The scanner never changes its address or its
identification to get past a limit.

To see such answers on HTTPS without reading the tools' traffic, the sensor sends its own small request
(`GET /`) to each target origin every 10 seconds. It verifies the origin's certificate; an origin with a
certificate that does not verify, or a target given without `http://` or `https://`, is paced but not
watched this way.
The sensor cannot see such answers on ordinary HTTPS. With the sandbox on, it therefore sends its own
small request (`GET /`) to each target origin every 10 seconds, the same way out as the tool. The request
is always the same and carries only the sensor's own `User-Agent`: nothing from the scan's settings is on
it. A 429, a 503 or a block page there halves the host's rate. On its own it does nothing more: it never
pauses a host and never stops a scan, so a target cannot end its own scan by answering that one request
differently. A host is paused when the tool's own connections fail as well.

This request verifies the origin's certificate. An origin with a certificate that does not verify, or a
target given without `http://` or `https://`, is paced but not watched this way.
Loading