From 72840fa38f1e4c7a19c1a84f4365e584b4bba518 Mon Sep 17 00:00:00 2001 From: Nguyen Manh <18032641+0xmanhnv@users.noreply.github.com> Date: Tue, 17 Nov 2026 08:39:54 +0000 Subject: [PATCH 1/2] docs(sensors): per-host budget, SENSOR_PACING_LEVEL and the local policy host ceiling Signed-off-by: Nguyen Manh <18032641+0xmanhnv@users.noreply.github.com> --- sensors/network.md | 1 + sensors/settings.md | 1 + sensors/tool-settings.md | 31 +++++++++++++++++++++++++++++++ 3 files changed, 33 insertions(+) diff --git a/sensors/network.md b/sensors/network.md index d3dfd7c..13693fe 100644 --- a/sensors/network.md +++ b/sensors/network.md @@ -234,6 +234,7 @@ kill_switch_file: /etc/openctem/STOP | `checks.allow` | Job types: `scan`, `validate`, `retest`, `collect`, `refresh_content` | | `allow_custom_templates`, `allow_interactsh` | Platform-supplied custom templates (off unless `true`); out-of-band callbacks (from sensor v0.38.4 they follow the platform's job unless `allow_interactsh: false`; before, off unless `true`) | | `rate.max_rps`, `rate.max_job_seconds` | Request rate ceiling; longest job run | +| `rate.max_host_rps`, `rate.max_host_inflight` | The most requests per second (1 to 200) and requests in flight (1 to 50) toward **one target host**: the ceiling no job passes. See [How hard a scan works one host](tool-settings.md#how-hard-a-scan-works-one-host) | | `kill_switch`, `kill_switch_file` | Stop every job (while the file exists) | | `http.user_agent`, `http.user_agents`, `http.allow_insecure_tls`, `http.headers` | A User-Agent for every tool, or a rotation list of up to 200 (one per target host and run; set one of the two); `false` refuses tools that skip TLS verification; up to 4 headers that mark this sensor's scan traffic (below) | | `managed.accept` | `false`: ignore any narrowing policy the platform manages for this sensor. Default `true` | diff --git a/sensors/settings.md b/sensors/settings.md index 138aab5..498129a 100644 --- a/sensors/settings.md +++ b/sensors/settings.md @@ -65,6 +65,7 @@ removed name is set, and its error names the replacement: | `SSL_CERT_DIR` | path | | | | System trust store directory override. | | `SENSOR_STATE_DIR` | path | | `/var/lib/openctem/state` | | Local state: the paired identity and signing key (identity/), the renewed API key and the tool cost history. Mount a persistent volume. | | `SENSOR_KEY_AUTORENEW` | bool | | `auto` | | API key auto-renewal: true, false, or unset (on when the state directory persists). | +| `SENSOR_PACING_LEVEL` | enum | | `standard` | | What one target host gets from a job that carries no signed budget: `gentle` (5 requests per second, 2 in flight), `standard` (20, 8) or `fast` (100, 25). The local policy's `rate.max_host_rps` and `rate.max_host_inflight` cap it. An unknown value is reported at start and the sensor runs at `gentle`. See [How hard a scan works one host](tool-settings.md#how-hard-a-scan-works-one-host). | | `SENSOR_MAX_JOBS` | int | | | | Cap on concurrent jobs, 1-100. Unset: the slots follow CPU, memory and tool costs. | | `SENSOR_JOB_ID` | string | | | | Run the one platform command with this id and exit (or `-job`; a Kubernetes Job, see [Kubernetes](deploy-kubernetes.md)). It is not in the settings registry, so **Setup & health** lists it as unknown (`config.env_unknown`); that warning is harmless here. | | `SENSOR_DRAIN_GRACE` | duration | | `30s` | | How long running jobs may finish on shutdown. | diff --git a/sensors/tool-settings.md b/sensors/tool-settings.md index 9b8c766..6697e1e 100644 --- a/sensors/tool-settings.md +++ b/sensors/tool-settings.md @@ -156,3 +156,34 @@ These options are the ones the current sensor release ({{ site.data.versions.sen Related: [Tools and capabilities](../scanning/tools.md), [Sensor-local policy](network.md#sensor-local-policy), [Settings reference](settings.md). + +## How hard a scan works one host + +A scan never sends one target host more than its **per-host budget**: 20 requests per second and 8 +requests in flight by default (the Standard level). The budget is counted per host, whatever the scheme, +port or path of the scan's targets, and whatever tool runs. + +| Level | Requests per second per host | In flight per host | +|---|---|---| +| Gentle | 5 | 2 | +| Standard (default) | 20 | 8 | +| Fast | 100 | 25 | + +- The sensor's operator picks the level with [`SENSOR_PACING_LEVEL`](settings.md#SENSOR_PACING_LEVEL) and + sets a ceiling no job passes with `rate.max_host_rps` and `rate.max_host_inflight` in the + [local policy](network.md). +- A rate or concurrency setting above the budget is lowered to it. A lower setting stays. +- With the tool sandbox on (the default), the sensor enforces the budget outside the tools, for nuclei, + httpx, katana and path discovery. With `SENSOR_SANDBOX=off` the tools' own flags enforce it, and a run + over several hosts gets one host's budget in all. + +**When a target pushes back.** A host that answers 429 or 503 is slowed down (its rate is halved, its +`Retry-After` is waited out) and sped up again gradually when it answers normally. A host that answers +nothing but push-back is paused, and the run reports it as a coverage gap `throttled` with counts; the run +completes as partial and keeps everything it found. The scanner never changes its address or its +identification to get past a limit. + +To see such answers on HTTPS without reading the tools' traffic, the sensor sends its own small request +(`GET /`) to each target origin every 10 seconds. It verifies the origin's certificate; an origin with a +certificate that does not verify, or a target given without `http://` or `https://`, is paced but not +watched this way. From b711ea587578db6c7ed8adf261d92303578855a1 Mon Sep 17 00:00:00 2001 From: Nguyen Manh <18032641+0xmanhnv@users.noreply.github.com> Date: Tue, 17 Nov 2026 12:54:22 +0000 Subject: [PATCH 2/2] docs(sensors): the lowest limit wins, which tools run under the per-host budget, what holds it and what does not - A job can only lower what a host gets: the budget is the lowest of the sensor level, the signed job budget, the operator ceiling and the built-in ceiling. - The tools under the budget, the port scan (packets per second) and the tools that send nothing to the targets. - What the sensor holds outside the tools, what only the flags of the tools hold, and what an unsandboxed run loses. - The canary request is fixed and only slows a host. Signed-off-by: Nguyen Manh <18032641+0xmanhnv@users.noreply.github.com> --- sensors/network.md | 2 +- sensors/settings.md | 2 +- sensors/tool-settings.md | 77 ++++++++++++++++++++++++++++++++-------- 3 files changed, 64 insertions(+), 17 deletions(-) diff --git a/sensors/network.md b/sensors/network.md index 13693fe..faf5c10 100644 --- a/sensors/network.md +++ b/sensors/network.md @@ -234,7 +234,7 @@ kill_switch_file: /etc/openctem/STOP | `checks.allow` | Job types: `scan`, `validate`, `retest`, `collect`, `refresh_content` | | `allow_custom_templates`, `allow_interactsh` | Platform-supplied custom templates (off unless `true`); out-of-band callbacks (from sensor v0.38.4 they follow the platform's job unless `allow_interactsh: false`; before, off unless `true`) | | `rate.max_rps`, `rate.max_job_seconds` | Request rate ceiling; longest job run | -| `rate.max_host_rps`, `rate.max_host_inflight` | The most requests per second (1 to 200) and requests in flight (1 to 50) toward **one target host**: the ceiling no job passes. See [How hard a scan works one host](tool-settings.md#how-hard-a-scan-works-one-host) | +| `rate.max_host_rps`, `rate.max_host_inflight` | The operator's ceiling for **one target host**: requests per second (1 to 200) and requests in flight (1 to 50). One of the limits of a job's per-host budget, of which the lowest wins: no job and no sensor level is above it. See [How hard a scan works one host](tool-settings.md#how-hard-a-scan-works-one-host) | | `kill_switch`, `kill_switch_file` | Stop every job (while the file exists) | | `http.user_agent`, `http.user_agents`, `http.allow_insecure_tls`, `http.headers` | A User-Agent for every tool, or a rotation list of up to 200 (one per target host and run; set one of the two); `false` refuses tools that skip TLS verification; up to 4 headers that mark this sensor's scan traffic (below) | | `managed.accept` | `false`: ignore any narrowing policy the platform manages for this sensor. Default `true` | diff --git a/sensors/settings.md b/sensors/settings.md index 498129a..0067efe 100644 --- a/sensors/settings.md +++ b/sensors/settings.md @@ -65,7 +65,7 @@ removed name is set, and its error names the replacement: | `SSL_CERT_DIR` | path | | | | System trust store directory override. | | `SENSOR_STATE_DIR` | path | | `/var/lib/openctem/state` | | Local state: the paired identity and signing key (identity/), the renewed API key and the tool cost history. Mount a persistent volume. | | `SENSOR_KEY_AUTORENEW` | bool | | `auto` | | API key auto-renewal: true, false, or unset (on when the state directory persists). | -| `SENSOR_PACING_LEVEL` | enum | | `standard` | | What one target host gets from a job that carries no signed budget: `gentle` (5 requests per second, 2 in flight), `standard` (20, 8) or `fast` (100, 25). The local policy's `rate.max_host_rps` and `rate.max_host_inflight` cap it. An unknown value is reported at start and the sensor runs at `gentle`. See [How hard a scan works one host](tool-settings.md#how-hard-a-scan-works-one-host). | +| `SENSOR_PACING_LEVEL` | enum | | `standard` | | The sensor's pacing level, the most one target host gets from a job here: `gentle` (5 requests per second, 2 in flight), `standard` (20, 8) or `fast` (100, 25). A job's signed budget and the local policy's `rate.max_host_rps` and `rate.max_host_inflight` can only lower it; no job raises it. An unknown value is reported at start and the sensor runs at `gentle`. See [How hard a scan works one host](tool-settings.md#how-hard-a-scan-works-one-host). | | `SENSOR_MAX_JOBS` | int | | | | Cap on concurrent jobs, 1-100. Unset: the slots follow CPU, memory and tool costs. | | `SENSOR_JOB_ID` | string | | | | Run the one platform command with this id and exit (or `-job`; a Kubernetes Job, see [Kubernetes](deploy-kubernetes.md)). It is not in the settings registry, so **Setup & health** lists it as unknown (`config.env_unknown`); that warning is harmless here. | | `SENSOR_DRAIN_GRACE` | duration | | `30s` | | How long running jobs may finish on shutdown. | diff --git a/sensors/tool-settings.md b/sensors/tool-settings.md index 6697e1e..eb6bc25 100644 --- a/sensors/tool-settings.md +++ b/sensors/tool-settings.md @@ -159,9 +159,9 @@ Related: [Tools and capabilities](../scanning/tools.md), ## How hard a scan works one host -A scan never sends one target host more than its **per-host budget**: 20 requests per second and 8 -requests in flight by default (the Standard level). The budget is counted per host, whatever the scheme, -port or path of the scan's targets, and whatever tool runs. +The tools that send requests to a scan's targets run under a **per-host budget**: 20 requests per second +and 8 requests in flight toward one host by default (the Standard level). The budget is counted per host, +whatever the scheme, port, path or spelling of the scan's targets. | Level | Requests per second per host | In flight per host | |---|---|---| @@ -169,21 +169,68 @@ port or path of the scan's targets, and whatever tool runs. | Standard (default) | 20 | 8 | | Fast | 100 | 25 | -- The sensor's operator picks the level with [`SENSOR_PACING_LEVEL`](settings.md#SENSOR_PACING_LEVEL) and - sets a ceiling no job passes with `rate.max_host_rps` and `rate.max_host_inflight` in the - [local policy](network.md). -- A rate or concurrency setting above the budget is lowered to it. A lower setting stays. -- With the tool sandbox on (the default), the sensor enforces the budget outside the tools, for nuclei, - httpx, katana and path discovery. With `SENSOR_SANDBOX=off` the tools' own flags enforce it, and a run - over several hosts gets one host's budget in all. +### The lowest limit wins -**When a target pushes back.** A host that answers 429 or 503 is slowed down (its rate is halved, its +A job's budget is, for each of the two numbers, the **lowest** of: + +1. the sensor's level, which the sensor's operator picks with + [`SENSOR_PACING_LEVEL`](settings.md#SENSOR_PACING_LEVEL); +2. the budget the platform signs into the job, when it sends one; +3. the operator's ceiling, `rate.max_host_rps` and `rate.max_host_inflight` in the + [local policy](network.md); +4. the built-in ceiling of 200 requests per second and 50 in flight. + +A job can only lower what a host gets. A job signed for Fast on a Standard sensor runs at Standard; a job +signed for Gentle on a Fast sensor runs at Gentle. To scan your own hosts faster, raise the level on the +sensor. A job that asks for more is not refused: it runs at the lower limit, and its result says which +limit set each number and what was asked for. + +A rate or concurrency setting above the budget is lowered to it, and the run's result and log say what was +asked and what applies. A lower setting stays. + +### Which tools + +| Tools | Paced by | +|---|---| +| nuclei (scans, re-verification and retests), httpx, katana, path discovery, screenshots, service detection | The per-host budget | +| Port scan (naabu) | Packets per second of the budget's level (100, 300 or 1000), not the request budget | +| DNS (dnsx), passive discovery (subfinder), lookups (asn, rdap), code scans (trivy, semgrep, opengrep, codeql, betterleaks) | They send nothing to the target hosts | + +### What enforces it + +**With the tool sandbox on (the default)**, every tool connection passes through the sensor, which holds +each host to the budget outside the tools. Names that resolve to one address share that address's budget. +The sensor holds: + +- every new connection, and the number of connections open to a host; +- every request over plain HTTP; +- every request over HTTPS, including on connections the tool keeps open, for the scan tools the sensor + runs (HTTP/1.1, and HTTP/2 from a tool that sends each request on its own). The sensor counts them from + the outside and does not read the traffic. + +It does **not** count requests that a client packs together on one HTTPS connection (a browser that loads +a page over HTTP/2, as the screenshot tool does), nor how many are in flight on such a connection: it +cannot tell them apart without reading them. For those, the tools' own rate and concurrency flags, which +the sensor derives from the same budget, are what holds the budget. + +**With `SENSOR_SANDBOX=off`**, nothing is in the path. The tools' own flags are the only enforcement, a +run gets one host's budget in all whatever its target list, and nothing reacts to what the hosts answer. +The run's result says that its pacing is limited. + +### When a target pushes back + +A host that answers the tool's requests with 429 or 503 is slowed down (its rate is halved, its `Retry-After` is waited out) and sped up again gradually when it answers normally. A host that answers nothing but push-back is paused, and the run reports it as a coverage gap `throttled` with counts; the run completes as partial and keeps everything it found. The scanner never changes its address or its identification to get past a limit. -To see such answers on HTTPS without reading the tools' traffic, the sensor sends its own small request -(`GET /`) to each target origin every 10 seconds. It verifies the origin's certificate; an origin with a -certificate that does not verify, or a target given without `http://` or `https://`, is paced but not -watched this way. +The sensor cannot see such answers on ordinary HTTPS. With the sandbox on, it therefore sends its own +small request (`GET /`) to each target origin every 10 seconds, the same way out as the tool. The request +is always the same and carries only the sensor's own `User-Agent`: nothing from the scan's settings is on +it. A 429, a 503 or a block page there halves the host's rate. On its own it does nothing more: it never +pauses a host and never stops a scan, so a target cannot end its own scan by answering that one request +differently. A host is paused when the tool's own connections fail as well. + +This request verifies the origin's certificate. An origin with a certificate that does not verify, or a +target given without `http://` or `https://`, is paced but not watched this way.