evmonly: parallelize OCC validation and merge behind the serial acceptance barrier

[bdchatham]

Describe your changes and provide context

On giga-testnet-2, occ_validate + occ_merge are ~34% of main-loop wall time and ran entirely on the calling goroutine, while conflicts are ~0.85 per ~1,800-tx block. Nearly all of that time is bookkeeping for transactions that are accepted as they stand. This PR keeps acceptance strictly in block order (the same frontier, same rerun/incarnation/fallback rules) but moves the work behind it onto the existing occWorkerPool.

Validation (validateBlockSTM):

• The write index (stateAccessIndex) is now sharded by address (64 contiguous address ranges) and records a txIndexSpan{first,last} per key instead of a single index. All speculative results are indexed up front in parallel, each worker filling the shards it owns.
• Conflict checks go from conflictsWithAfter(key, sourcePrefix) to conflictsWithin(key, sourcePrefix, txIndex): a write invalidates a result only if it sits in [sourcePrefix, txIndex). With every result indexed, this is the same question the serial frontier asked, because when the frontier is at i every write below i is an accepted one. A rerun's writes are added; the previous incarnation's stay. Both the span and the stale entries can only produce false positives (an unnecessary rerun, which then validates trivially against its exact prefix), never a missed conflict.
• acceptValidatedPrefix runs a parallel pass over [nextToValidate, n): workers evaluate stmFrontierAccepts (error, gas, read/write conflicts against the precomputed cumulative gas) and lower a shared stop to the first index that would not be accepted. Everything before stop is folded into the prefix with applyRange (each worker applies the shards it owns; per-address state lives entirely in one shard, so per-shard order equals block order). The serial frontier then handles only the transaction at stop — recording conflicts, scheduling the rerun, or surfacing the gas/execution error exactly as before.
• Each incarnation records the shards it touched as a bitmask (occTxExecution.shards, computed on the worker that produced it), so index/apply workers skip whole results that hold nothing in their shards instead of scanning every result's maps.
• Dependency chains that would make every parallel pass accept ~nothing switch to a serial stretch that doubles each time it happens (serialUntil), so worst-case blocks don't pay pool wake-ups per transaction. Blocks under 64 pending results skip the pool entirely.

Merge (mergeOCCResults):

• blockSTMState is sharded the same way; changeSetIntoParallel computes each shard's changeset (with its own baseAccounts row cache, so each account is still read once) on the pool into pooled per-shard fragments and concatenates shards in order, which is canonical address order — output is byte-identical to the serial ChangeSetInto. A prefix under occMinParallelMergeKeys (256) keys is merged serially straight into the pooled changeset. The tx/receipt loop (cumulative gas, log indexes) stays serial and unchanged.
• The previous prefetchBaseAccounts pass is subsumed: base rows are now read inside the per-shard merge.
• A closed pool during merge falls back to sequential execution like the other errOCCWorkerPoolClosed sites.

What does not change: transaction order, cumulative gas / block gas-limit / overflow checks, occMaxTxIncarnations, only-earliest-invalid-rerun, sequential fallback reasons, final changeset semantics. Conflict counts are still deterministic (a pure function of the block); they can be slightly higher than before due to span false positives.

Known shape: a hot contract's storage lands in one shard, so that shard's merge/apply runs on one worker. BenchmarkExecuteTransferBlock gained an erc20_single_contract case (1,000 transfers on one ERC-20) to keep that measured: 19.9 → 18.6 ms/op vs giga-1. Whole-block local benchmarks (8 cores, incl. parse — parse is ~50% of it): conflict_free 15.1 → 12.9 ms/op, hot_recipient and same_sender_nonce_chain unchanged within noise, allocations flat. The rollout check is occ_validate + occ_merge share of the main loop on testnet-2 and executed tx/s.

Testing performed to validate your change

• New TestExecutorOCCLargeMixedBlockMatchesSequential: ~330-tx block (independent transfers, same-sender nonce chains, hot recipient, a late reader of the hot balance that must be rerun after the parallel pass) — asserts Txs, Receipts, GasUsed and the full ChangeSet equal the sequential executor's, and that a rerun actually happened.
• New TestParallelMergeResolvesEveryTouchedAccountOnce (replaces the prefetch tests): parallel merge output equals serial, canonical order, one row read per account.
• New occ_shards_test.go: conflictsWithin lower/upper bounds and span over-approximation, occShardOf monotonic and covering, occShardsOwnedBy partitioning, cumulativeGasFrom overflow truncation, firstUnacceptedResult returning the lowest rejection under concurrency (error, gas limit, conflict, source-prefix), touchedShards, and the serial-merge guard against a closed pool.
• Existing conflict-matrix / source-prefix / gas / incarnation tests updated only for the txIndex parameter.
• go test -race ./giga/evmonly/... and ./sei-tendermint/internal/evmonlyapp/... green; make fmtcheck clean; golangci-lint run ./giga/evmonly/... 0 issues (v2.13.2, Go 1.27.1).

Link to Devin session: https://app.devin.ai/sessions/ff612badcded4aa5914ea408dbb41888
Open in Devin Desktop: https://app.devin.ai/desktop/session/ff612badcded4aa5914ea408dbb41888?variant=devin
Requested by: @bdchatham

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[cursor]

PR Summary

High Risk
Changes core optimistic parallel block execution, conflict detection, and final changeset assembly; correctness relies on staying equivalent to sequential acceptance, with span indexing allowing extra reruns but not missed conflicts.

Overview
Block-STM OCC still accepts transactions strictly in block order, but the heavy bookkeeping behind that frontier now runs on the existing worker pool instead of entirely on the calling goroutine.

Validation indexes all speculative writes up front into a 64-way address-sharded write index that tracks first/last tx index spans per key. Conflict checks move from “any write at or after sourcePrefix” to conflictsWithin(sourcePrefix, txIndex), matching the serial frontier’s notion of which prior writes matter. acceptValidatedPrefix parallel-scans pending results for the first one the frontier would reject, bulk-applies the accepted run into a sharded prefix, and leaves a single transaction for the serial frontier to rerun or error on. serialBackoff lengthens serial stretches when parallel passes accept too little; small blocks skip the pool.

Merge shards blockSTMState the same way, replaces account prefetch with changeSetIntoParallel (per-shard base reads + fragment concat in canonical order), and falls back to sequential execution if the pool is closed. Incarnations record touched shard bitmasks so workers skip irrelevant results.

Docs, shard/unit tests, a large mixed-block OCC vs sequential parity test, and an ERC-20 single-contract load benchmark case exercise the new paths.

^{Reviewed by Cursor Bugbot for commit 592e38f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	✅ passed	✅ passed	✅ passed	Sep 19, 2026, 7:08 PM

[codecov]

Codecov Report

❌ Patch coverage is 92.51969% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.69%. Comparing base (7e3ca24) to head (592e38f).
⚠️ Report is 41 commits behind head on giga-1.

Files with missing lines	Patch %	Lines
giga/evmonly/occ.go	91.86%	10 Missing ⚠️
giga/evmonly/occ_shards.go	93.12%	9 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           giga-1    #4261       +/-   ##
===========================================
+ Coverage   65.55%   84.69%   +19.14%     
===========================================
  Files        2081       27     -2054     
  Lines      157460     4358   -153102     
===========================================
- Hits       103222     3691    -99531     
+ Misses      54097      667    -53430     
+ Partials      141        0      -141     

Flag	Coverage Δ
sei-chain	?
sei-chain-pr	84.69% <92.51%> (?)
sei-db	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
giga/evmonly/occ_shards.go	93.12% <93.12%> (ø)
giga/evmonly/occ.go	90.97% <91.86%> (+1.78%)	⬆️

... and 2062 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[seidroid]

The parallel validation/merge redesign is correct as far as I can verify: the [sourcePrefix, txIndex) span check is equivalent to the old accepted-only index (since sourcePrefix is only ever 0 or txIndex), firstUnacceptedResult genuinely returns the minimum stop index, and shard ownership is a disjoint partition with no races. The findings are performance regressions for small and storage-heavy blocks, plus a test-coverage gap around the new helpers.

Findings: 0 blocking | 4 non-blocking | 2 posted inline

Blockers

• None at the file/PR level.

Non-blocking

• [suggestion] Sharding by addr[0] gives no parallelism for the case the merge is most likely to be slow on: a block dominated by one hot contract's storage. All of that contract's storage entries land in a single shard, so changeSetIntoParallel degenerates to serial for the storage portion (which also carries the sort and the per-slot source.GetState reads) while still paying the fan-out and fragment allocations. The benchmark in the description (BenchmarkExecuteTransferBlock) is the opposite shape — many distinct EOAs — so it doesn't exercise this. Worth benchmarking a contract-heavy block before rollout; if it regresses, storage could be sub-sharded by slot range within an address, which preserves the canonical-order-by-concatenation property.
• [suggestion] None of the new helpers in occ_shards.go have a direct unit test: conflictsWithin/writtenWithin bounds, occShardOf monotonicity across shard boundaries, cumulativeGasFrom's overflow truncation, firstUnacceptedResult returning the true minimum under concurrency, and the serialUntil backoff are all covered only transitively by TestExecutorOCCLargeMixedBlockMatchesSequential. Notably, the three existing conflict tests were updated by passing math.MaxInt as txIndex, which disables the upper bound this PR introduces — so the new hi semantics have no unit-level coverage at all. A small table test asserting that a write at an index >= txIndex does not conflict, and that one in [sourcePrefix, txIndex) does, would pin the central semantic change of the PR.
• 2 suggestion(s)/nit(s) flagged inline on specific lines.

[bdchatham]

@seidroid review

[seidroid]

The parallel validation/merge redesign holds up: the [sourcePrefix, txIndex) span check is exact for the only two sourcePrefix values that occur, firstUnacceptedResult provably returns the minimum stop index, shard ownership is a disjoint partition with block-order preserved per shard, and the previous round's small-block and fragment-allocation findings are genuinely fixed. Two non-blocking items: a readability/structure point in the new backoff loop and a no-op line that voids one test assertion.

Findings: 0 blocking | 2 non-blocking | 2 posted inline

Blockers

• None at the file/PR level.

Non-blocking

• None at the file/PR level.
• 2 suggestion(s)/nit(s) flagged inline on specific lines.