AppSec: synchronous reverse_dns postoverflow enrichment stalls in-band responses via unbuffered pipeline backpressure (bouncer fail-open bursts)

[roybj]

Summary

With the crowdsecurity/rdns postoverflow installed, a slow PTR lookup for an alert's source IP freezes AppSec in-band inspection for the whole engine. Remediation-component requests then exceed APPSEC_PROCESS_TIMEOUT and fail open (APPSEC_FAILURE_ACTION=passthrough), i.e. requests are served uninspected - on our busiest host 76-1,413 timeouts/hour (~0.5% of inspections, bursts to 480/min).

Environment: crowdsec v1.7.8 (rpm), AlmaLinux 8.10, central LAPI, appsec acquisition on 127.0.0.1:7422 with crowdsec-openresty-bouncer v1.2.1 in-band, default routines: 1, ~600 alerts/h (WAF + log scenarios), OS resolver = ISP shared resolver, no local cache, default resolv.conf timeouts (5sx2).

Mechanism (all refs v1.7.8):

1. crowdsecurity/rdns runs in the postoverflow parse inside runOutput -> parser.reverse_dns (pkg/parser/enrich_dns.go:19) -> net.LookupAddr - synchronous, unbounded by any context/timeout of its own. Hostile alert IPs frequently sit in lame/slow PTR zones -> 0.2-10 s lookups.
2. While it blocks, the event train jams: overflow chan <- buckets <- parsers <- acquisition output (all unbuffered).
3. The AppSec runner shares that output channel (runner.outChan = out, pkg/acquisition/modules/appsec/run.go:137) and blocks in handleRequest on r.outChan <- evt (appsec_runner.go:336/341/420/426). For in-band matches this send happens before the response is written to the remediation component (appsec_runner.go:472).
4. InChan is also unbuffered (appsec/config.go:169), so every handler blocks; all in-flight inspections exceed the client's read budget -> fail-open burst. Burst size = request rate x PTR stall length.

Evidence

• Goroutine dump captured while a probe to :7422 was hanging (437 ms) - runOutput blocked in the lookup:

goroutine 157 [IO wait]:
internal/poll.runtime_pollWait(...)
...
net.dnsPacketRoundTrip(...)
net.(*Resolver).exchange(...)
net.(*Resolver).tryOneName(...)
net.(*Resolver).goLookupPTR(...)
net.LookupAddr(...)
github.com/crowdsecurity/crowdsec/pkg/parser.reverse_dns(...)
	github.com/crowdsecurity/crowdsec/pkg/parser/enrich_dns.go:19
github.com/crowdsecurity/crowdsec/pkg/parser.(*RuntimeStatic).Apply(...)
	github.com/crowdsecurity/crowdsec/pkg/parser/runtime.go:167
[... postoverflow parse inside main.runOutput ...]

• The inband parsing histogram is blind to it: 1.04 M inspections, mean 0.47 ms, only 9 ever >250 ms - while the same period logged tens of thousands of >1000 ms client read-timeouts. The stall sits between accept and the timed span.
• Every clustered burst-second across a 90-min window (22/22) had an alert landing at LAPI +2...+12 s later - the stalled-lookup-then-flush signature.
• Burst-adjacent alert source IPs re-resolved hours later (warm cache = lower bound): 10/26 still >200 ms, incl. 1,056 ms and a SERVFAIL at 487 ms.
• Removal experiment: cscli postoverflows remove crowdsecurity/rdns --force + restart dropped the host from 76-1,413/h to 57 in the first full hour (lowest on record) and 15-74/h over the next nine hours, while an untouched sibling host (same stack, same LAPI) ran high (127 that hour). Probe watcher: 0 stalls >=350 ms in 6,038 probes post-change.
• routines: 4 does not mitigate (measured null): all runners share the same jammed output channel.

Related: cmd/crowdsec/output.go already acknowledges this failure class - "Under high load, LAPI may take between 1 and 2 seconds to process ~100 alerts, which slows down everything including the WAF" - and made PushAlerts async. Parser-stage enrichment has the same coupling but no such fix. Symptom also matches crowdsecurity/cs-openresty-bouncer#75 (closed unresolved: high-traffic timeouts with 0.2 ms engine latency).

Suggested fixes (any one breaks the coupling): bound the enrichment (LookupAddr with a short-timeout context), make enrichment async/cached, buffer the overflow/acquisition channels, or give the AppSec runner a dedicated buffered emission queue so in-band responses never share fate with the alert pipeline.

Side note: in this deployment the reverse_dns value did not appear in any of 1,834 alert payloads at LAPI - if that's expected for appsec/scenario alerts, the enrichment cost bought nothing here.

Workaround for affected users: remove crowdsecurity/rdns (requires --force; owned by crowdsecurity/linux / whitelist-good-actors). seo-bots-whitelist degrades gracefully to its IP-list expression.

[github-actions]

@roybj: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[github-actions]

@roybj: There are no 'kind' label on this issue. You need a 'kind' label to start the triage process.

• /kind feature
• /kind enhancement
• /kind refactoring
• /kind bug
• /kind packaging

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[Meganitrospeed]

/kind bug

[Meganitrospeed]

@roybj seems only you can assign it, we found this issue too

[roybj]

/kind bug

[roybj]

@Meganitrospeed thanks for the confirmation, and sorry for the delay. Label is set now. The governance bot only accepts /kind from the issue author or a collaborator, which is why yours did not take.

Status update against v1.8.0 (released today), since it changed this exact code path.

What improved. reverse_dns no longer calls net.LookupAddr directly. Since 1f5e9c8 (#4268, 2026-07-01) it goes through the new pkg/dnscache:

• lookups are bounded at lookupTimeout = 3 * time.Second
• results are cached in a 16384-entry LRU, positive TTL 1h, negative TTL 5m, tunable via dnscache.Configure
• concurrent lookups for the same key coalesce via singleflight

That genuinely shrinks the worst case: an unbounded lookup governed by resolv.conf defaults (5s x 2 attempts on our hosts) is now capped at 3s, and a repeat offender costs one query per TTL instead of one per alert.

What did not change, i.e. the defect this issue is about. The coupling is intact at v1.8.0:

• the postoverflow parse still runs synchronously in the output loop: parser.Parse(postOverflowCTX, ...) at cmd/crowdsec/output.go:134
• the pipeline channels are still unbuffered: inEvents / logLines at cmd/crowdsec/crowdsec.go:177-178
• the AppSec runner still shares that channel (w.AppsecRuntime.OutChan = out, pkg/acquisition/modules/appsec/run.go:143) and InChan is still unbuffered (config.go:237)
• emission is still an unguarded blocking send: w.OutChan <- evt at pkg/appsec/emit.go:80
• and on an in-band match it still happens before the answer reaches the remediation component: handleInBandInterrupt is called at appsec_runner.go:479 and reaches that send via emitMatch (:365), while the response is only written at :488

So on a cache miss the engine still parks behind a single PTR query and the in-band verdict is still gated behind it. The bound moves the stall ceiling from ~10s to 3s, which is still 3x the remediation component's default APPSEC_PROCESS_TIMEOUT of 1000ms, so fail-open bursts persist, just shorter and shallower.

The cache also helps least in the case that hurts. The stalls come from alert source IPs, which are near-unique across an attack wave, and the lame/SERVFAIL zones that produce the slow lookups sit on the 5m negative TTL, so they re-stall on a 5-minute cycle rather than being absorbed.

All of the suggested fixes in the original report still apply. The cheapest one that actually breaks the coupling is to stop letting the in-band response share fate with the alert pipeline: give the runner a buffered emission queue, or emit after the response has been written.

@Meganitrospeed until something lands, the workaround holds: cscli postoverflows remove crowdsecurity/rdns --force plus a restart. The --force is required because rdns is owned by both crowdsecurity/linux and crowdsecurity/whitelist-good-actors, so a plain remove is a refusing no-op. We applied it to 28 AppSec hosts on 2026-08-11; it has stayed removed and effective for three weeks, including across nightly cscli hub upgrade runs. seo-bots-whitelist degrades gracefully to its IP-list expression.

For transparency: we are still on v1.7.8 in production, so the v1.8.0 notes above are from reading the tag, not from running it.

[ostsjoe]

Third-party confirmation, with a more severe outcome than the two reports above: on our deployment this does not just fail open, it runs to OOMKill.

Environment: crowdsec v1.7.8 (docker, EKS), embedded LAPI + AppSec in one pod, appsec acquisition on 0.0.0.0:7422, crowdsec-spoa-bouncer v0.3.0 in-band via HAProxy SPOE (so a different remediation component to both reports above), crowdsecurity/rdns installed, SQLite with use_wal: true, GOMEMLIMIT=4GiB inside a 6GiB container limit. ~112 AppSec req/s inspected, ~30–150 alerts/min.

The escalation

Two OOMKills in 75 minutes on 2026-09-03, routines: 1 at the time:

12:03Z -> 12:35Z  (32 min)   goroutines   8,482 -> 247,912   RSS 230MB -> 6.02GB   SIGKILL
12:38Z -> 13:17Z  (39 min)   goroutines   3,939 -> 258,321   RSS 243MB -> 5.96GB   SIGKILL

An earlier instance on 2026-08-30 self-recovered rather than dying — 6 minutes, 5,133 -> 40,502 goroutines, RSS 483MB -> 1.16GB — so duration is not bounded by anything except the memory limit. Both shapes are the same mechanism, just stopped at different points.

Signature matches #4600 exactly

• Accumulation ran at ~5,800/min against ~5,800/min inbound — precisely the arrival rate, i.e. nothing completing.
• CPU flat at 45–190m throughout. Blocked on a channel, not spinning. cs_regexp_cache_size 0.
• cs_appsec_reqs stayed steady while cs_appsec_inband_parsing_time_seconds.count collapsed from ~6,000 to ~30 per 20s bucket. Those two counters sit on opposite sides of the block — the request counter increments in the handler before InChan, the parsing histogram only after the runner finishes in-band rules. That divergence is a clean fingerprint if others want to check whether they are hitting this.
• On release, 46,194 parses completed in a single 20s bucket at 0.155ms each. Queued, not slow — which also confirms the histogram is blind to the stall, as you noted.
• LAPI's own HTTP server answered heartbeat and allowlist requests every minute in under 1ms right through it. Only the inspection path stalls.

Why it reaches OOM rather than just timing out

The memory is goroutine stacks, not heap: go_memstats_stack_sys_bytes went from a ~5MB baseline to 2.03GB and 2.12GB in the two events. Those stacks belong to live goroutines parked on the channel send, so GC cannot reclaim them — GOMEMLIMIT accounts for them but can only respond by collecting harder, which achieves nothing, and then the kernel wins. Worth flagging for anyone who assumes GOMEMLIMIT bounds this: it does not.

Partial mitigation data (adds to the routines: 4 null result)

We can confirm your measured null for the appsec routines knob alone, but the picture changes if the stages downstream of the runner are widened too. We set:

# acquis.d/appsec.yaml
routines: 4
# config.yaml.local
crowdsec_service:
  parser_routines: 4
  buckets_routines: 2

output_routines deliberately left at 1 (SQLite is single-writer).

Result over 5 days: the stalls still occur, but they became self-limiting instead of unbounded.

before:  30–40 min,  ~250,000 goroutines,  6.0GB,   OOMKill
after:    1–12 min,     54,162 goroutines,  2.37GB,  self-recovers

Nine separate stalls in one 2.5h window post-change, longest ~11 minutes, all drained on their own. Zero restarts in 5 days.

That is consistent with your diagnosis rather than contradicting it: more parser/bucket routines add in-flight capacity between the runner and the jam in runOutput, which absorbs a longer PTR stall before backpressure reaches the AppSec handler. It buys headroom, it does not decouple anything, and it has obvious diminishing returns. Offering it as a stopgap for anyone who cannot drop rdns, not as a fix.

Note on v1.8.0

Agreed with your read that pkg/dnscache does not close this. In our traffic the cache would help less than average: stall sources are attack IPs, near-unique per wave, and the lame/SERVFAIL zones that produce the slow lookups land on the 5m negative TTL, so they re-stall on that cycle.

We have not yet removed crowdsecurity/rdns — holding to see whether 1.8.1 addresses the coupling, since our current state is contained. Happy to report back with before/after numbers if we do pull it, as a second data point alongside your 28 hosts.

Of the suggested fixes, emitting after the in-band response is written looks like the smallest change that actually breaks the coupling for the AppSec path specifically, independent of whatever enrichment happens to be slow.