Repository navigation
An append keeps the page's validation memo, so a read after it stops hashing the page - #633
Merged
Conversation
…hashing the page A read of a page checks its bytes against a memo of one hash per 4 KiB page. The memo is keyed by (write_sequence, checksum), an append changes both, and the write also dropped the memo outright, so the first read after every append started from nothing: it hashed the whole payload and then every page of it again. For a page filled by appends and read after each one -- a log that is followed while it is written -- the cost of a read grew with the page. An append leaves every byte before the old end untouched, so the hash of each page that was full at the old end is still that page's hash. The write now takes the memo out before it, and puts back one that keeps those hashes and recomputes only the page that held the old end and anything past it. A write that is not a pure append still drops the memo, as before, and a memo that described anything other than the page just before the write is not carried. Measured on the door under callgrind, driven by the shared-store harness: the per-write cost of the read path grew 2.5x between 300 and 1,200 writes, and its own cost inside the replica read was 81% of the whole growth. With the cold validation skipped (a measuring plant, never shippable) a 1,200-write run executed 40.5% fewer instructions. Tests: the carried memo equals a cold build after every append of widths that straddle the 4 KiB boundary; an overwrite drops it and a write past the end with a hole carries it; and a byte damaged in a page whose hash was carried is still not served. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This was referenced Nov 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A read of a page checks its bytes against a memo of one hash per 4 KiB page. The memo is keyed by
(write_sequence, checksum), an append changes both, and the write also dropped the memo outright, so the first read after every append started from nothing: it hashed the whole payload and then every page of it again. For a page filled by appends and read after each one (a log that is followed while it is written), the cost of a read grew with the page.An append leaves every byte before the old end untouched, so each page that was full at the old end keeps its hash.
write_chunknow takes the memo out before the write and puts back one that keeps those hashes and recomputes only the page that held the old end and anything past it. A write that is not a pure append still drops the memo, exactly as before, and a memo that described anything other than the page just before the write is not carried.Measured
Door under callgrind (journal mode, the shared-store harness driving it, instructions per write):
At 1,200 writes the run executes 30.5% fewer instructions (522.3M to 362.9M); the ceiling is 40.5%. Before the change, the replica read's own cost was 81% of the per-write growth (54.9K to 177K instructions per write, hashing inlined).
What is left of the growth is a separate term:
invalidate_chunk_cache_boundedcollects every cached read range of the page on each write (about 34K per write at 1,200 writes). Not part of this change.Tests
Planted defects, each caught: keeping the stale partial-page hash fails all three tests; carrying on a non-append write fails the overwrite test.
Checks run locally
The four named guards pass;
cargo build --all-targets --lockedpasses with-D warnings, with and without--all-features; the library suite is 692 passed, 1 failed. The one failure,the_published_mutation_copy_figures_are_the_ones_this_tree_produces, fails with identical numbers on unmodified main (measured 2557 / 799,992, README 2485 / 768,312) and CI does not run it. The network consensus jobs I left to CI.cargo fmt --checkreports many files, none of them new here; CI has no format step, so I did not reformat.🤖 Generated with Claude Code