Repository navigation
daemon: rule windows no longer hold every event forever (0.13.23) - #286
Merged
Merged
Conversation
After 0.13.22, dev2's daemon still grew ~280 MB an hour (heap 473 MB live after a full GC, 4.5 h after restart). A sampling heap profile put the live allocations in the per-line event path: the rule engine. Each rule kept a window per source address (or per path, or global) holding the full ThreatEvent of every match (URL, user agent, details). Old entries were dropped only when a NEW event arrived for the same key, and cleanup() removed only windows that were already empty, so every address that matched a rule once and never returned kept its events for the life of the daemon. dev2 sees hundreds of thousands of addresses a day. - Windows keep a timestamp and source IP per entry, never the event. - cleanup() expires each window by its own rule's length and forgets empty windows once their cooldown has passed; it runs every minute, not five. - At most 1000 entries (or the threshold, if higher) per window; a detection past the cap reports "1000+ events". - Pruning is a moving head index instead of filtering the array on every event, so a busy aggregate window is no longer O(n) per event. Tests: 10,000 one-off addresses are all forgotten once the window passes (the old cleanup kept them all); thresholds and cooldowns behave as before; the cap and its label. The existing rule suites pass unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ThreatCrush Security Scan18 finding(s) HIGH/CRITICAL: 1 | MEDIUM: 9 | LOW: 8
Snippets are redacted; ThreatCrush never prints matched credential material. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
daemon: rule windows no longer hold every event forever (0.13.23)
After 0.13.22, dev2's daemon still grew ~280 MB an hour (heap 473 MB live
after a full GC, 4.5 h after restart). A sampling heap profile put the live
allocations in the per-line event path: the rule engine.
Each rule kept a window per source address (or per path, or global) holding
the full ThreatEvent of every match (URL, user agent, details). Old entries
were dropped only when a NEW event arrived for the same key, and cleanup()
removed only windows that were already empty, so every address that matched
a rule once and never returned kept its events for the life of the daemon.
dev2 sees hundreds of thousands of addresses a day.
windows once their cooldown has passed; it runs every minute, not five.
past the cap reports "1000+ events".
event, so a busy aggregate window is no longer O(n) per event.
Tests: 10,000 one-off addresses are all forgotten once the window passes (the
old cleanup kept them all); thresholds and cooldowns behave as before; the
cap and its label. The existing rule suites pass unchanged.
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com