Skip to content

perf(drive-abci): stop rewriting the whole platform state every block - #4571

Open
PastaPastaPasta wants to merge 1 commit into
perf/checkpoint-skip-during-replayfrom
perf/platform-state-writes
Open

perf(drive-abci): stop rewriting the whole platform state every block#4571
PastaPastaPasta wants to merge 1 commit into
perf/checkpoint-skip-during-replayfrom
perf/platform-state-writes

Conversation

@PastaPastaPasta

@PastaPastaPasta PastaPastaPasta commented Sep 1, 2026

Copy link
Copy Markdown
Member

Issue being fixed or feature implemented

The saved platform state is written to GroveDB aux storage on every block, and on mainnet it is 1.28 MB — almost all of it masternode lists, validator sets and the chain-lock and instant-lock quorum sets.

PlatformSerializable::serialize_to_bytes also did self.clone() before converting to the saving form, so a block paid two full deep copies of some 4,000 masternodes plus the validator sets, then serialized 1.28 MB, then wrote it.

Measured replaying mainnet, per block: 1.08 ms serializing, 0.23 ms for the extra clone, and 1.28 MB into the block's transaction — 545 GB of writes over a full sync.

Stacked on #4570, which introduces the utils::is_historical_block predicate this uses.

What was done?

Three things.

Serialize from a borrowed state. A TryFrom<&PlatformState> for the saving form clones each field once instead of cloning the whole state first. A test asserts the bytes are byte-identical to the owned path.

Rewrite the full record only when it changed. The heavy fields carry a dirty flag, set by the accessors that can change them. While replaying history the full record is written only when the flag is set; a small companion record under a second aux key carries the per-block fields — block info, quorum hashes, protocol versions — every block. Both are written in the block's transaction, so a reader never sees them disagree, and a database without the companion record reads exactly as before.

Once the node is at the tip the full record is written every block again, so an up-to-date node always has a complete record on disk and an older drive-abci can still read it. Skipping is confined to a node that is catching up, where the remedy for any format trouble is the resync it is already doing.

Don't take a mutable borrow when nothing moved. Core reports the same quorums on most blocks and an empty masternode diff often, but update_quorum_info and update_state_masternode_list reached for _mut() accessors regardless — and the borrow alone marks the state dirty. Both now decide read-only whether anything actually changed first.

How Has This Been Tested?

Full mainnet replay, genesis to 424,981. Every committed app hash matched a reference sync across all 424,971 heights, and the final app hash matched exactly.

Crash recovery specifically: two mid-sync restarts, at heights 40,200 and 80,188. Both resumed at the right height from the companion record and continued with no app-hash mismatch — which requires the heavy fields to be correct too, since they feed the app hash through masternode identity updates and rewards.

cargo test -p drive-abci --lib — 2,776 passed, including the new byte-identity test.

Breaking Changes

None for normal operation, an interrupted sync included. A node that stops mid-sync restarts and carries on by itself with no operator action: the companion record restores the height, and the heavy fields are unchanged since the full record was written. Two mid-sync restarts are part of the testing above.

The one case that needs care is a deliberate downgrade. A node interrupted mid-initial-sync leaves a full record that lags the database, and an older drive-abci reading that database panics in the Info handler, which compares the saved state's app hash against GroveDB's root hash — confirmed by running a pre-change binary against a database this one wrote. That is why the skip is confined to historical blocks: a node that has caught up has a complete record on disk and can always be rolled back. A node caught mid-sync would need to finish syncing on the new build, or resync on the old one.

Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have added or updated relevant unit/integration/functional/e2e tests
  • I have made corresponding changes to the documentation

For repository code-owners and collaborators only

  • I have assigned this pull request to a milestone

🤖 Generated with Claude Code

The saved platform state is 1.28 MB on mainnet, almost all of it masternode lists, validator sets and quorum sets, and it was serialized and written to GroveDB aux storage on every block. Serialization also cloned the entire state first, so a block paid two full copies of some 4,000 masternodes.

Serialization now builds the saving form from a borrowed state, and a test asserts the bytes are identical to the owned path. The heavy fields carry a dirty flag set by the accessors that can change them, and while replaying history the full record is rewritten only when it is set, with a small companion record holding the per-block fields written every block; both land in the block's transaction, and a database without the companion record reads exactly as before. Once at the tip the full record is written every block again, so an up-to-date node always has a complete record on disk. The two Core-driven update paths now decide read-only whether anything actually moved before taking a mutable borrow, because Core reports the same masternodes and quorums on most blocks and the borrow alone would force the rewrite.
@coderabbitai

coderabbitai Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Team

Run ID: b35837f3-97b3-42ea-b8d4-7a7422dde3ff

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@thepastaclaw

thepastaclaw commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

🕓 Ready for review — 9 ahead in queue (commit a319483)
Queue position: 10/58 · 2 reviews active
ETA: start ~14:34 UTC · complete ~15:28 UTC (median 54m across 30 recent reviews; 2 slots)
Queued 2d 10h ago · Last checked: 2026-09-03 10:50 UTC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants