Repository navigation
feat: every scan job runs under the per-host budget (nuclei, httpx, katana, path discovery) - #408
Merged
Merged
Conversation
…s what a tool sends to one host nuclei, httpx, katana and path discovery derive their flags from the budget of the job (Standard: 20 requests per second and 8 in flight per target host) and hand it to the tool host, whose forwarder holds each host to it. A rate or concurrency a person chose above the budget is lowered to it. Without a forwarder a tool whose rate flag is global gets one host budget for the whole run. SENSOR_PACING_LEVEL sets the level of jobs without a signed budget; the local policy rate.max_host_rps and rate.max_host_inflight are the ceiling no job passes. The sensor reports pacing.budget@1. go.mod pins the head of sdk-go feat/host-budget (sdk-go #509). Signed-off-by: Nguyen Manh <18032641+0xmanhnv@users.noreply.github.com>
Contributor
Security gatePASS: security-sensitive, acknowledged, no open HIGH. Why this is security-sensitive
Diff reviewNo high-risk pattern on the added lines. Code scanningNo new high or critical alert. Security checks on the head commit
AcknowledgeA maintainer reviews the change and adds the Path list: |
0xmanhnv
marked this pull request as ready for review
November 17, 2026 09:34
…get, a setting above it is recorded, httpx counts what a target costs Security review of the per-host budget: - internal/toolrun/traffic.go lists every compiled-in tool by what it sends to its targets. The runner refuses to start a tool that sends requests to its targets without a budget, and a test fails when a tool is missing from the list or contradicts its manifest. - Under the budget now: nuclei re-verification and retests, webshot and svcdetect, besides nuclei, httpx, katana and webguess. The port scan is paced in packets per second by the level of the budget; DNS, passive lookups and code scans send nothing to the targets. - A rate or concurrency a person set above the budget is recorded with what was asked and what applies (target_rate.budget.clamped) and logged, on every path: the tool host, the tool child and the in-process run. - httpx -rl counts targets, and a target costs its host two requests with the favicon probe on, and fourteen connections more with JARM where httpx dials itself: the rate is the budget divided by that cost. Measured with the real binary: at -rl 20 one host received 42 requests a second. - Without a forwarder a run gets one budget of one host, whatever its target list, also for an httpx sweep. - A job can only lower the budget (sdk-go): the tests and the docs say a signed Fast on a Standard sensor stays Standard. - internal/scanners/budget_real_test.go runs nuclei, httpx and katana as built against a local HTTPS server with keep-alive and HTTP/2 offered and asserts what the server counts. - docs/REFERENCE.md: the tool list, what is enforced where and what is not, the canary. Signed-off-by: Nguyen Manh <18032641+0xmanhnv@users.noreply.github.com>
Signed-off-by: Nguyen Manh <18032641+0xmanhnv@users.noreply.github.com>
For CI while this pull request is a draft. Before it leaves draft it is re-pinned to the merged commit on sdk-go main. Signed-off-by: Nguyen Manh <18032641+0xmanhnv@users.noreply.github.com>
0xmanhnv
marked this pull request as draft
November 17, 2026 13:11
0xmanhnv
marked this pull request as ready for review
November 17, 2026 13:29
…ir command's budget, the port scan is started with the budget The sensor reported pacing.budget@1 whenever it ran commands, also with no job-signing key pinned: it then ran every job at its own level while the platform believed a signed budget was read. The capability is now reported only when the sensor verifies job signatures. A retest ran under the sensor's own budget whatever its command's signed budget said. It now runs under its command's budget, as a re-verification does; the lowest limit still wins. The tool host refuses to start a tool that connects to its targets without a budget, so the port scan is started with the scan's budget and as a port scan (it stays paced in packets per second). The runner refuses any other tool that asks for that pacing. A rate or concurrency of zero never reaches nuclei as a flag. Signed-off-by: Nguyen Manh <18032641+0xmanhnv@users.noreply.github.com>
Signed-off-by: Nguyen Manh <18032641+0xmanhnv@users.noreply.github.com>
0xmanhnv
marked this pull request as draft
November 17, 2026 16:20
0xmanhnv
marked this pull request as ready for review
November 17, 2026 16:47
…-go to the head of its review-fix branch (0a69d86) The runner refuses a tool of network egress-proxy (any public host, its targets included) without a per-host budget, and trivy, semgrep, codeql, rdap and asn are started with the scan's budget. A scan handed no budget runs under adapt.FallbackBudget. REFERENCE and the changelog entry say which tools are held and what the host_busy gap means. Signed-off-by: Nguyen Manh <18032641+0xmanhnv@users.noreply.github.com>
0xmanhnv
marked this pull request as draft
November 17, 2026 19:21
- go.mod pins the sdk-go review branch head e100eb8 (still a branch commit: this pull request stays a draft until sdk-go #510 merges). - docs/REFERENCE.md and the changelog fragment: the sensor-side gap reason is now slots_full; the target keeps its tool's outcome, so no follow-up scans the host again at once; a tunnel closed while its answer may still have been arriving also leaves the gap; the vendor traffic of the egress-proxy tools (asn, rdap, trivy, semgrep, codeql) gets the per-host budget (two connections per host on Gentle); a vendor host that is only a DNS alias of a target is not matched as one. Signed-off-by: Nguyen Manh <18032641+0xmanhnv@users.noreply.github.com>
The SDK no longer closes a connection whose answer it cannot be sure ended (its first request, or HTTP/2) for a waiting connection until it has been quiet for 30 seconds; only such a close leaves a slots_full gap, so ordinary keep-alive reuse leaves none. docs/REFERENCE.md says so. Signed-off-by: Nguyen Manh <18032641+0xmanhnv@users.noreply.github.com>
…nnection is a gap - go.mod pins sdk-go ee3f722 (the follow-up pull request's head) until it merges. - docs/REFERENCE.md: a connection is held 30 s only on its first request, 1 s (5 s when the tool offered HTTP/2) once the host has answered on it; only an HTTPS connection closed on its first request is a slots_full gap; a waiting connection whose tool gave up stops at once; a target that trickles bytes keeps its slot (bounded, visible gap). - toolrun.Budget: the fallback is also bounded by the operator's per-host ceiling. Signed-off-by: Nguyen Manh <18032641+0xmanhnv@users.noreply.github.com>
…wording An HTTP/2-offering connection is closed for a waiting one after the same 30 seconds of quiet as an unsure one, not after 5 seconds. Signed-off-by: Nguyen Manh <18032641+0xmanhnv@users.noreply.github.com>
Signed-off-by: Nguyen Manh <18032641+0xmanhnv@users.noreply.github.com>
0xmanhnv
marked this pull request as ready for review
November 17, 2026 23:41
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
sdk-go #509, #510 and #511 are merged on
main.go.modpins sdk-gov0.51.1-0.20261117233817-f8315b68975c, the #511 merge commit onmain(its tree is identical to the #511 branch head d2bdb4c this PR pinned before). The pin change isgo.modandgo.sumonly.Why
A scan must not flood one target. #404 lowered the default rate, but one target host could still get far more than the level says:
rate_limita person set (150 in a scan's or the organization's settings) went to one host unchanged;The trust rule: the lowest limit wins
The per-host budget of a job (sdk-go #509) is, for each number, the lowest of the sensor's level (
SENSOR_PACING_LEVEL, default Standard: 20 requests per second and 8 in flight per target host), the job's signed budget, the operator's ceiling (rate.max_host_rps,rate.max_host_inflight) and the built-in ceiling (200 and 50). A job can only lower it. To run faster on hosts you own, raise the level on the sensor. The result says which limit set each number (budget.won_by).Which tools (code and docs now agree; a test keeps them so)
internal/toolrun/traffic.golists every compiled-in tool by what it sends to its targets.TestEveryToolHasATrafficKindfails when a tool is missing or contradicts its manifest's network class, and the runner refuses to start a tool that sends requests to its targets without a budget (TestToolThatSendsToTargetsIsNotStartedWithoutABudget). The SDK's tool host now refuses the same for any tool whose manifest saysnetwork: targets(sdk-go #509), so tools the operator installs in an adapter directory are covered too: they get the scan's budget through the SDK's compat scanner.nuclei-validate), httpx, katana, webguess, webshot, svcdetectnetwork: targets-rate100, 300, 1000); a chosenratereplaces it underrate.max_rps. It is started with the budget and as a port scan (RunOptions.PortProbes); the runner refuses any other tool that asks for thatinternal/lookup), trivy, semgrep, codeqlnetwork: egress-proxy(any public host, the targets included)What is enforced where
SENSOR_SANDBOX=offRetry-Afteron one name slows the address and never pauses or refuses the other names there. A connection the slots refuse makes its host a coverage gapGET /(the sensor's own User-Agent, nothing of the job's identification), on the forwarder's dial path; it slows, it never pauses a host or stops a scan by itself; two rate-limit answers to it in a row are reported as a coverage gapthrottledmarkedobserved, which stops nothingwon_by,clamped, back-offs, breaker openings with the evidence that opened thempacing_limited: no_forwarderFlags under the budget (Standard):
-rate-limit20 for each host worked at once (at most 8, never above the ceilings),-c 8,-bshosts up to 25,-mhe 15,-timeout 10,-retries 1,-ss template-sprayover several hosts-rate-limit20, never above the budget's rate (5 at Gentle)-rlcounts targets: a list of one host-rl 10 -threads 8with the favicon probe on (two requests a target); several hosts their budgets in all, at most the sweep-rl 50 -threads 25-rl 20 -c 8for each host crawled at oncerateper origin, at most 20rate, at most 20 for each host probed at onceA setting above the budget is no longer lowered silently (finding 9): a
rate_limit,concurrency,threadsorratea person set (settings layers, the scanner config, the scan's own request) above the budget is recorded inmetadata.target_rate.budget.clampedwith what was asked and what applies, and the job's log has a line for it, on every path (tool host, tool child, in-process).Second round of fixes (this revision)
pacing.budget@1also when it verifies no job signatures, and then ignored the signed budgetadvertiseHostBudgetinmain.go: the capability is reported only when a job-signing key or root is pinned (Kit.VerifiesJobs)TestHostBudgetCapabilityNeedsVerifiedJobsValidatingCommandExecutor.retestBudget: the command's budget, which the kit's retest command passes on the context; never above the sensor's ownTestRetestRunsUnderItsCommandsBudget(a signed Gentle retest:-rate-limit 5)network: targetstool without a budgetPortProbes; the runner's guard checks bothTestToolThatSendsToTargetsIsNotStartedWithoutABudgetScanner.pacedLimitsTestPacedLimitsNeverPassAZeroRateEverything about shared addresses, connection slots, the observed throttle and adapter tools is in sdk-go #509.
Third round of fixes (this revision)
A further security review of sdk-go #509 found that a tool declaring
network: egress-proxyskipped the budget guard. The SDK side (and the other findings: open connections bounded by the slots, a slot refusal reported as the sensor-side gapslots_fullinstead ofrate_floor, the fallback budget, vendor hosts that are targets) is sdk-go #510. Here:toolrun.requireBudget) let a tool of networkegress-proxystart without a budgetTestEgressProxyToolIsNotStartedWithoutABudget(Run, RunRaw, RunTask; listed and unlisted)egress-proxy) were started without a budget, which the SDK now refusestoolrun.Budget(opts))toolrun.Budgetfalls back toadapt.FallbackBudget(the lowest of the sensor's level, Standard and the local policy's per-host ceiling)Sixth round (this revision)
sdk-go #511 changed the SDK only; this pull request re-pins
go.modto its head (ee3f722) and updates the docs:slots_fullgap; keep-alive reuse leaves none, also for tools that offer HTTP/2 (webshot, Go-based tools);slots_fullgap;toolrun.Budgetnow names the operator's per-host ceiling in the fallback.Fifth round
A follow-up on sdk-go #510 changed the SDK only; this pull request re-pins
go.modto the new sdk-go #510 head (4f41626) and updatesdocs/REFERENCE.md:slots_fullgap, so ordinary keep-alive reuse (sure idle connections, closed at once) makes no gap and no partial run.Fourth round
A fourth review of sdk-go #510 changed the SDK only; this pull request re-pins
go.modto the new sdk-go #510 head (e100eb8) and updates the docs:slots_full(the platform useshost_busyfor a task waiting on a host another task holds);slots_fullgap no longer makes the targetunfinished: the target keeps its tool's outcome and the gap alone marks the host not covered, so the platform does not re-scan the host at once in a follow-up command;slots_fullgap;rate.max_host_rpsandrate.max_host_inflight;docs/REFERENCE.mdstates the cost for asn, rdap, trivy, semgrep and codeql: their own services (registries, rule and pack hosts, RDAP and ASN servers) get the per-host budget, so on Gentle two connections per host, and long parallel downloads may seeslots_full; and that a vendor host that is only a DNS alias of a target is not matched as a target.No sensor code referred to the old reason name, so nothing else changes here.
Measured with the real tools
internal/scanners/budget_real_test.goruns nuclei v3.11.1, httpx v1.12.0 and katana v1.8.0 with the command lines the sensor builds, against a server in the test process on 127.0.0.1 that speaks HTTPS, keeps connections alive, offers HTTP/2 and counts every request and connection. Build box, in a slot.-rl 30 -threads 24)The tools refill their rate once a second, so the stated tolerance for "any 1 s" is one refill (40); the forwarder's is a bucket of one token.
What this test found: httpx's
-rlcounts targets, not requests. With the default probes one target is two requests (the page and the favicon), and where httpx dials itself a JARM fingerprint is fourteen connections more. At-rl 20one host received 42 requests a second (300 requests for 150 URLs in 7.2 s). The rate is now the budget divided by what a target costs (httpx.Scanner.requestsPerTarget).Scan time
-rl 20was the only bound-rl 50 -threads 25(100 requests/s with the favicon)-rl 10 -threads 8(20 requests/s)rate_limitof 150 on one hostSENSOR_PACING_LEVEL=fastgives 100/s on hosts you ownSENSOR_SANDBOX=off, nuclei over 8 hostsSENSOR_SANDBOX=off, httpx with JARM on, a list of one hostThreat model
TestBudgetBoundsChosenValues: an organization asking for 5000/200 runs at 20/8; a signed Fast, or a signed budget above every ceiling, on a Standard sensor runs 20/8).targets,egress-proxy, andvendorwith a target among its configured vendor hosts).pacing.budget@1is reported only by a sensor that verifies job signatures.tool.yamlfiles are unchanged.Residual, honestly
ratereplaces the level's.SENSOR_SANDBOX=off: flags only, no reaction to what the targets answer.pacing.budget@1.NUCLEI_REAL_BIN,HTTPX_REAL_BIN,KATANA_REAL_BIN); CI skips them, the numbers above are from the build box.egress-proxytools is held to the target budget (see Fourth round); a vendor host that is only a DNS alias of a target is not matched as one.Tests
internal/toolrun/traffic_test.go,tool_traffic_test.go: the tool registry and the start guard (requests, port probes, nothing).host_budget_capability_test.go: the capability with and without verified jobs.internal/executor/retest_test.go: the retest's budget.internal/scanners/nuclei/pacing_test.go: the command line by budget, host count and confinement; chosen values bounded and recorded; the re-verification rate under the budget.internal/recon/httpx/budget_args_test.go,katana/,naabu/: flags by budget; the cost of a target; clamps.internal/recon/politeness_test.go,internal/scanners/recon_limits_test.go: the budget over one and many hosts, with and without a forwarder.internal/scanners/budget_real_test.go: the measurements above.Checks
Final revision, against sdk-go
mainf8315b6 (#511 merged), build box:go mod verifyexit 0;go build ./...exit 0;go vet ./...exit 0;golangci-lint run ./...(v2.14.0) 0 issues;go test -count=1 ./...(the whole suite) exit 0.Sixth revision, against sdk-go #511 head ee3f722 (see the pin), build box:
go build ./...exit 0;go vet ./...exit 0;golangci-lint run ./...(v2.14.0) 0 issues;go test -count=1 ./...(the whole suite) exit 0;gofmt -llists only the four files named below that this pull request does not touch.Fifth revision, against sdk-go #510 head 4f41626 (see the pin), build box:
go mod tidyexit 0;go build ./...exit 0;go vet ./...exit 0;golangci-lint run ./...(v2.14.0) exit 0;go test -count=1 ./...(the whole suite) exit 0.Fourth revision, against sdk-go #510 head e100eb8 (see the pin):
go mod tidyexit 0;go build ./...exit 0;go vet ./...exit 0;golangci-lint run ./...(v2.14.0) 0 issues;go test -race -count=1on the root package, toolrun, lookup, scanners, recon and executor exit 0;go test -count=1 ./...(the whole suite) exit 0.gofmt -llists four files this pull request does not touch (tool_detect.go,policy_cmd_test.go,tools_cmd_betterleaks_codeql_test.go,internal/scanners/nuclei/validate_real_test.go).Third revision, against sdk-go #510 head (see the pin):
go mod tidyexit 0;go build ./...exit 0;go vet ./...exit 0;go test -race -count=1on the root package, toolrun, recon, scanners, executor, webguess, webshot, svcdetect and lookup exit 1 on the box under load: two job-file privacy tests (TestListsArePrivateToTheirJobin recon/dnsx,TestPathReportIsPrivateToItsJobin scanners/internal/report) failed once; both packages then passed three times in a row alone, at this pin and at the previous one, and neither touches this change;go test -count=1 ./...(the whole suite) exit 0;golangci-lint run ./...(v2.14.0) exit 0, 0 issues.Second revision, against sdk-go aa27239:
go mod verifyexit 0;go build ./...exit 0;go build -tags platform ./...exit 0;go vet ./...exit 0;go test -race -count=1on the root package, toolrun, recon, scanners, executor, webguess, webshot and svcdetect exit 0;go test -count=1 ./...(the whole suite) exit 0;golangci-lint run ./...(v2.14.0) exit 0, 0 issues; the vocabulary check exit 0. The confined nuclei tests (RealNucleiKeepsToWebScope,RealNucleiThroughThePool, nuclei v3.11.1, in the container image CI uses): 2 passed. The real-tool measurements above were not run again for this revision; the flags the sensor builds are unchanged.deps-on-mainis red whilego.modpins the sdk-go branch: that is the draft state described at the top.Docs
docs/REFERENCE.md("Per-host budget": the tool list with adapter tools, what is enforced where, names that share an address, the capability and signed jobs, the canary's observed throttle),docs/LOCAL_POLICY.md,docs/RUN.md,docs/sensor-policy.example.yaml, changelog fragment.Public docs (docs
sensors/tool-settings.md, "How hard a scan works one host") need a follow-up for this revision; no docs pull request is opened here. The sentences to change:Retry-Afteron one of them slows that address for a while and never pauses or refuses the other names there."slots_full: the sensor's own limit, not the host; the scan of that host is not repeated at once, and the host's findings are not held for it. A connection that is closed for another one while its first answer may still be arriving also makes its host such a gap;"egress-proxy| The per-host budget: such a tool may reach any public host, so every host it reaches is held to the budget".throttled(marked as observed) and carries on at the lower rate."slots_full."