perf(drive-abci): don't create GroveDB checkpoints while replaying history - #4570
perf(drive-abci): don't create GroveDB checkpoints while replaying history#4570PastaPastaPasta wants to merge 1 commit into
Conversation
…story Protocol version 11 turns checkpoints on, with a policy of one every 600 seconds of chain time keeping the last 3. Mainnet blocks are about 2.5 minutes apart, so that is a checkpoint every four blocks — which at replay speed is roughly 37 a second, each one a RocksDB checkpoint over the whole database plus a copy of the platform state, and all but the last three deleted again immediately. Measured at 15.1 ms a block from height 318,704, against about 7 ms for everything else in a block put together. A finished mainnet sync had four checkpoint directories left on disk out of some 26,000 created. Checkpoints are restore points for a running node, so skip them for blocks more than ten minutes old and write the first real one on reaching the tip. They live outside the tree, so no app hash changes.
|
🕓 Ready for review — 8 ahead in queue (commit 302f7ef) |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## v4.2-dev #4570 +/- ##
============================================
- Coverage 87.68% 87.22% -0.46%
============================================
Files 2778 2786 +8
Lines 359190 361994 +2804
============================================
+ Hits 314960 315767 +807
- Misses 44230 46227 +1997
🚀 New features to boost your workflow:
|
|
Warning Review limit reachedNext included review available in 56 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (3)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
2 participants
Issue being fixed or feature implemented
A node replaying mainnet history creates and destroys tens of thousands of GroveDB checkpoints.
Protocol version 11 (mainnet height 318,704) turns checkpoints on:
with
frequency_seconds: 600, num_checkpoints: 3. The interval is in chain time, and mainnet blocks are ~2.5 minutes apart, so that is a checkpoint every four blocks. At replay speed that is roughly 37 a second, each one a RocksDB checkpoint over the whole database — hard-linking every SST, flushing and copying the WAL — plus a copy of the platform state, and all but the last three deleted again immediately.Measured with per-block phase timing, replaying mainnet:
against roughly 7,000 µs for everything else in a block put together. A finished mainnet sync left four checkpoint directories on disk out of some 26,000 created:
The last three are four blocks apart, which is the cadence.
It shows up as an I/O stall rather than CPU: during the affected range drive-abci sat at 0.38 of one core with the system 47–73% idle and the disk at 2,000–4,000 tps.
What was done?
Skip checkpoint creation for blocks more than ten minutes old. Checkpoints are restore points for a running node; a node catching up has no use for restore points into blocks it is about to replace, and it writes its first real checkpoint on reaching the tip.
The age test is a new
utils::is_historical_block, with unit tests — the predicate is the testable part, since a historical replay makes every block historical by construction.Checkpoints live outside the tree, so no app hash changes.
How Has This Been Tested?
Same 3,000-block window from mainnet height 331,648, two local peers, back to back:
fb_checkpoint−60.8% per block. The 47.6 blocks/s reproduces what a full sync does over that range.
Full mainnet replay, genesis to 424,981, with this change on top of an otherwise identical build: 4,684.9 s → 3,032.9 s, 35% of the whole sync. Every committed app hash matched a reference sync across all 424,971 heights.
cargo test -p drive-abci --lib should_checkpoint utils::replay— 11 passed.One existing test needed updating:
test_first_block_should_always_checkpointbuilt its context withblock_time_ms = 1_000_000, which is 1970 and now reads as history. It uses a current timestamp, and a companion test covers the new behaviour for an old block.Breaking Changes
None for a node that is running. Once it reaches the tip it checkpoints exactly as it does today.
A node still catching up keeps no checkpoints, where before it kept three covering the last thirty minutes of chain time. Nothing during sync depends on them: a node that is interrupted picks up where it left off and carries on by itself, with no operator action, the same as it does today.
Checklist:
For repository code-owners and collaborators only
🤖 Generated with Claude Code