feat(evmrpc): bound eth_getLogs peak memory with a matched-log cap#3759
feat(evmrpc): bound eth_getLogs peak memory with a matched-log cap#3759amir-deris wants to merge 4 commits into
Conversation
Thread a limit param through the ReceiptStore.FilterLogs interface so log queries abort with ErrTooManyLogs once matches exceed the cap instead of materializing an unbounded result set. Enforce it in both the litt range path (errgroup-cancelled fan-out) and the block-by-block fallback (cooperative early-abort), bounding peak memory to O(maxLog). Rename applyOpenEndedLogLimit to applyOpenEndedBlockWindow to reflect that it windows the block range; the log cap is now enforced separately. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
An organization admin can view or raise the cap at claude.ai/admin-settings/claude-code. The cap resets at the start of the next billing period.
Once the cap resets or is raised, push a new commit or reopen this pull request to trigger a review.
PR SummaryMedium Risk Overview The cap is threaded through Breaking: clients that previously received a partial but successful response when matches exceeded the limit must narrow filters or ranges and handle the new error. Reviewed by Cursor Bugbot for commit 000b292. Bugbot is set up for automated code reviews on this repo. Configure here. |
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 76f591d. Configure here.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3759 +/- ##
==========================================
- Coverage 59.93% 59.03% -0.90%
==========================================
Files 2288 2201 -87
Lines 189868 180049 -9819
==========================================
- Hits 113793 106300 -7493
+ Misses 65942 64413 -1529
+ Partials 10133 9336 -797
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
![seidroid[bot]](https://avatars.githubusercontent.com/in/2745135?s=60&v=4){kind=small}
There was a problem hiding this comment.
Solid, well-tested change that bounds eth_getLogs peak memory by pushing a matched-log cap into ReceiptStore.FilterLogs; the implementation and call-site updates are correct. Main concern is an undocumented behavior change: the cap is sourced from MaxLogNoBlock (documented as open-ended-only) but now also errors on previously-uncapped bounded queries, and the config doc is now stale.
Findings: 0 blocking | 5 non-blocking | 2 posted inline
Blockers
- None at the file/PR level.
Non-blocking
- Config semantics drift:
MaxLogNoBlock(mapstructuremax_log_no_block) is documented in evmrpc/config/config.go:103 and the config template (~line 699) as "max number of logs returned if block range is open-ended." This PR now uses it as a hard error threshold for ALL queries (bounded and open-ended). Update those doc comments to reflect that it now (a) applies to bounded queries and (b) causes an ErrTooManyLogs error rather than truncating returned logs. Consider whether a distinct, clearly-named general cap setting is warranted rather than overloading the open-ended one. - Breaking-change coverage: the PR description's "breaking" section calls out the open-ended truncation→error switch but not that bounded eth_getLogs requests matching more than MaxLogNoBlock (default 10000) logs — previously uncapped and successful — now return ErrTooManyLogs. This should be surfaced in the changelog/release notes so client integrations can adjust.
- Cursor's second-opinion review file (cursor-review.md) was empty — that pass produced no output. REVIEW_GUIDELINES.md was also empty, so no repo-specific standards were applied.
- 2 suggestion(s)/nit(s) flagged inline on specific lines.
| // both bounded and open-ended queries. Exceeding it is an error (not | ||
| // silent truncation), so peak memory stays bounded to the cap. A value of 0 | ||
| // disables the cap. | ||
| limit := f.filterConfig.maxLog |
There was a problem hiding this comment.
[suggestion] limit is sourced from MaxLogNoBlock, whose config doc reads "max number of logs returned if block range is open-ended." On main, bounded queries were uncapped on both the range path (tryFilterLogsRange passed no limit) and the block-by-block path (limit only applied when applyOpenEndedLogLimit). Applying limit unconditionally here silently repurposes an open-ended-only setting into a hard error threshold for bounded queries: a bounded eth_getLogs matching more than MaxLogNoBlock (default 10000) logs now returns ErrTooManyLogs where it previously succeeded. If this broader cap is intended, update the max_log_no_block doc comment (config.go:103 and the config template) and the changelog; otherwise consider preserving the open-ended-only semantics or introducing a distinct general-cap setting.
| break | ||
| } | ||
| before := len(localLogs) | ||
| f.GetLogsForBlockPooled(block, crit, &localLogs) |
There was a problem hiding this comment.
[nit] Nit: the cap is checked only between blocks, so GetLogsForBlockPooled materializes a full block's matching logs before collected is incremented. Peak matched-log memory is therefore limit + (workers × single-block logs), not strictly limit. This is inherent (at least one block must be materialized) and matches the PR's documented "cap plus at most one in-flight block per worker" — memory stays bounded — so no change needed, just flagging that the bound is cap + overshoot rather than the cap alone.
There was a problem hiding this comment.
Pushes a matched-log cap into the receipt store to bound eth_getLogs peak memory; the concurrency/cancellation logic is sound, but the config documentation claims 0 disables the cap (it's coerced to the default) and the range path counts synthetic logs that later get normalized away, so eth_getLogs can spuriously return ErrTooManyLogs.
Findings: 0 blocking | 6 non-blocking | 3 posted inline
Blockers
- None at the file/PR level.
Non-blocking
- Cursor's second-opinion review file (cursor-review.md) is empty — that pass produced no output.
- The PR description flags a breaking behavior change (open-ended queries over maxLog now return ErrTooManyLogs instead of silently truncating), and the author notes it 'must be called out in the changelog/docs', but the diff contains no changelog/doc update. Add one before merge.
- Nit: the drain comment in GetLogsByFilters (filter.go:922-923) says producer goroutines could be 'left blocked on a full buffer', but fetchBlocksByCrit fully buffers
res(size = range) and closes it before returning, so producers never block and the channel is already closed when drained — the loop is harmless cleanup but the stated rationale is inaccurate. - 3 suggestion(s)/nit(s) flagged inline on specific lines.
|
|
||
| // max number of logs returned if block range is open-ended | ||
| // max number of logs a single eth_getLogs query may match before it errors, | ||
| // for both bounded and open-ended block ranges (0 disables the cap) |
There was a problem hiding this comment.
[suggestion] This doc says (0 disables the cap), but NewFilterAPI coerces any maxLog <= 0 to DefaultMaxLogLimit (10000) at filter.go:287-288, so an operator setting max_log_no_block: 0 gets the default cap, not an uncapped query. Either document that 0 falls back to the default, or skip the coercion when the value is explicitly 0 to actually allow disabling. (Flagged by Codex P2.)
| // maxLog caps the number of matching logs a single query may return, for | ||
| // both bounded and open-ended queries. Exceeding it is an error (not | ||
| // silent truncation), so peak memory stays bounded to the cap. A value of 0 | ||
| // disables the cap. |
There was a problem hiding this comment.
[nit] Same inaccuracy as the config doc: "A value of 0 disables the cap" isn't reachable here — limit comes from f.filterConfig.maxLog, which NewFilterAPI has already forced to a positive DefaultMaxLogLimit when the configured value was <= 0. At this call site limit is always > 0.
| sdkCtx := f.ctxProvider(int64(toBlock)) | ||
|
|
||
| logs, err := store.FilterLogs(sdkCtx, fromBlock, toBlock, crit) | ||
| logs, err := store.FilterLogs(sdkCtx, fromBlock, toBlock, crit, limit) |
There was a problem hiding this comment.
[suggestion] The range path applies the cap against the store's pre-normalization count. filterLogsByTags counts every indexed matching log (including synthetic/Cosmos receipts), but normalizeRangeQueryLogs (called just below) rebuilds only from EVM-visible txs — for the eth_* namespace includeSyntheticReceipts=false, so synthetic logs are dropped afterward. Result: eth_getLogs can return ErrTooManyLogs even when the actual RPC result is below maxLog. The block-by-block fallback counts correctly (its collectLogs already excludes synthetic), so the two paths are inconsistent. Consider capping against the normalized count, or documenting that the range-path cap is a conservative over-count. (Codex P1.)
There was a problem hiding this comment.
A well-structured change that pushes a matched-log cap into both eth_getLogs query paths to bound peak memory, with consistent error-on-overflow semantics and good test coverage for both paths. No correctness blockers; the main gaps are documentation of a config field that cannot actually disable the cap and a broader client-visible breaking change that lacks a changelog entry.
Findings: 0 blocking | 4 non-blocking | 1 posted inline
Blockers
- None at the file/PR level.
Non-blocking
- Breaking change is under-documented and broader than the PR description states. The description frames the break as only "open-ended queries now error instead of silently truncating." But because
limit := f.filterConfig.maxLogis now passed unconditionally to both the litt range path and the block-by-block fallback, bounded queries (both fromBlock and toBlock set) are now capped too — previously they had no matched-log cap and would return a large result set successfully. A bounded query matching more thanmaxLog(default 10000) logs within the block window will now returnErrTooManyLogswhere it previously succeeded. Add a CHANGELOG.md entry and call out the bounded-query behavior change explicitly (Codex raised the missing-changelog point). - cursor-review.md was empty — the Cursor second-opinion pass produced no output, so only Codex's findings (both incorporated here) were available to merge.
- Minor: in
GetLogsByFiltersthe post-wg.Wait()drain loopfor range blocks {}is effectively unnecessary and its comment ("producer goroutines are not left blocked on a full buffer") is inaccurate —fetchBlocksByCritsizes the channel buffer to the full block range (make(chan ..., end-begin+1)), so producers can never block on a full buffer. Harmless, but the rationale should be corrected or the loop dropped. - 1 suggestion(s)/nit(s) flagged inline on specific lines.
|
|
||
| // max number of logs returned if block range is open-ended | ||
| // max number of logs a single eth_getLogs query may match before it errors, | ||
| // for both bounded and open-ended block ranges (0 disables the cap) |
There was a problem hiding this comment.
[suggestion] This comment says 0 disables the cap, but that is not achievable via config: NewFilterAPI replaces any maxLog <= 0 with DefaultMaxLogLimit (10000) at evmrpc/filter.go:287-289, and the default for MaxLogNoBlock is already 10000 (config.go:237). So operators cannot disable the cap by setting max_log_no_block = 0 as documented — it silently becomes 10000. Either drop the "(0 disables the cap)" claim here (and mirror it in the limit comment at filter.go:33-34) or actually honor 0 by not defaulting it in NewFilterAPI. (Raised by Codex.)
There was a problem hiding this comment.
This PR pushes a matched-log cap into the receipt store to bound eth_getLogs peak memory, returning ErrTooManyLogs instead of silently truncating. The implementation is sound, well-documented, and well-tested; no blockers. The main non-blocking concern (also raised by Codex) is that the range path enforces the cap on the raw store count before normalization, which can produce a false ErrTooManyLogs for queries whose EVM-visible result is within the limit.
Findings: 0 blocking | 6 non-blocking | 3 posted inline
Blockers
- None at the file/PR level.
Non-blocking
- Behavior change worth emphasizing: on main, bounded block-range queries had no matched-log cap and would return arbitrarily large result sets successfully; they now error once the match count exceeds max_log_no_block (default 10000). The PR description only calls out the open-ended truncation→error change as breaking, but the new cap on bounded queries is an additional behavior change. It is captured in the CHANGELOG, but any client-facing docs should also state that wide bounded queries returning >max_log_no_block logs will now fail.
- Test coverage: the end-to-end RPC test (TestFilterGetLogsMatchedLogCap) uses a blockHash query, which skips the range path and exercises only the block-by-block fallback. The litt range path's cap is covered at the store level (TestLittIdxFilterLogsLimit) but there is no end-to-end test exercising tryFilterLogsRange + normalizeRangeQueryLogs + the cap together through the JSON-RPC handler.
- Cursor's second-opinion review file (cursor-review.md) was empty — that pass produced no output.
- 3 suggestion(s)/nit(s) flagged inline on specific lines.
| sdkCtx := f.ctxProvider(int64(toBlock)) | ||
|
|
||
| logs, err := store.FilterLogs(sdkCtx, fromBlock, toBlock, crit) | ||
| logs, err := store.FilterLogs(sdkCtx, fromBlock, toBlock, crit, limit) |
There was a problem hiding this comment.
[suggestion] The cap is enforced by the store on the raw tag-index match count, which includes non-EVM-visible / synthetic logs, before normalizeRangeQueryLogs filters the result down to the EVM-visible set. Since normalized <= raw, a query whose EVM-visible result is within limit can still trip ErrTooManyLogs when the raw match count exceeds it — a false positive that the block-by-block path (which counts already-normalized logs via GetLogsForBlockPooled) does not have. Consider whether the range path should cap on the EVM-visible count for consistency, or document that the cap is intentionally a conservative pre-normalization bound. (Matches Codex's finding.)
|
|
||
| // Re-check the cap against the normalized (EVM-visible) count, which the | ||
| // store's tag-index count does not match. | ||
| if limit > 0 && int64(len(normalized)) > limit { |
There was a problem hiding this comment.
[nit] This post-normalization re-check appears to be dead code: it only fires when len(normalized) > limit, but the store already aborts with ErrTooManyLogs whenever the raw count exceeds limit, and normalized <= raw. So whenever the store returns successfully (raw <= limit), normalized <= limit too and this branch can never trigger. It's harmless as defense-in-depth, but it cannot recover the false-positive case described above — it can only tighten the cap further, never loosen it. A brief comment clarifying it's defensive would help, or it could be removed.
| var limit int64 | ||
| if applyOpenEndedLogLimit { | ||
| limit = f.filterConfig.maxLog | ||
| // Drain the blocks channel so its producer goroutines are not left blocked |
There was a problem hiding this comment.
[nit] The rationale in this comment is slightly inaccurate. fetchBlocksByCrit sizes the channel buffer to the full range (end-begin+1) and blocks on errChan until every producer goroutine has finished and res is closed, so by the time blocks is returned here the producers have already completed — they are never "left blocked on a full buffer." The drain loop is still fine (it just releases buffered ResultBlocks for GC and terminates on the closed channel), but the comment's justification doesn't hold. Consider rewording to reflect that it's freeing buffered blocks rather than unblocking producers.
1 participant
eth_getLogscould materialize an unbounded result set: a wide block rangeor loose filter that matched millions of logs would allocate them all before
returning, spiking peak memory. The block-range and open-ended windowing caps
limited how many blocks were scanned, but nothing capped how many matching
logs were held in memory at once.
This PR pushes a matched-log cap all the way into the receipt store so a query
aborts instead of accumulating past the limit:
ReceiptStore.FilterLogstakes a newlimit int64parameter.
limit > 0caps matches;limit <= 0disables the cap(preserves prior behavior for internal callers, benches, and the simulator).
receipt.ErrTooManyLogs(+NewTooManyLogsError(limit))is returned when a query exceeds the cap, so overflow is a clear error rather
than silent truncation.
filterLogsByTags) — the parallel fan-out tracks arunning matched count; the worker that trips the cap returns
ErrTooManyLogs,cancelling the errgroup so already-scheduled blocks bail before reading.
Peak memory is bounded to the cap plus at most one in-flight block per worker.
GetLogsByFilters) — an atomic counter drivescooperative early-abort: workers stop materializing blocks and the fan-out
stops queueing batches once the cap is exceeded, then the overflow is reported
after
wg.Wait. The blocks channel is drained so producer goroutines aren'tleft blocked.
applyOpenEndedLogLimit→applyOpenEndedBlockWindowto reflectthat it windows the block range; the matched-log cap is now enforced
separately by the caller.
The cap value comes from the existing
maxLogfilter config, so no new configsurface is introduced.
On
main, open-ended queries (missingfromBlockortoBlock) that matchedmore than
maxLoglogs were silently truncated viamergeSortedLogs(..., limit)— callers received a partial result with noindication that logs were dropped. This change returns
ErrTooManyLogsinstead of truncating.This matches the ticket's suggested direction (fail loudly rather than return
incomplete data), but it is a breaking API change: clients that previously got
a truncated-but-successful response will now receive an error and must narrow
their block range or filter criteria. This must be called out in the
changelog/docs.
Testing performed to validate your change
TestLittIdxFilterLogsLimit: verifies FilterLogs returns all logs at orbelow the limit, returns
ErrTooManyLogswhen exceeded, and treatslimit <= 0as uncapped.FilterLogscall sites (tests, benches, simulator, thepebble backend stub, and the
fakeReceiptStoreinwatermark_manager_test)to the new signature.
parallel-order) passes unchanged with the added parameter.