Repository navigation
The metadata change log is a deque, so a full log drops a record without moving the rest - #631
Merged
Merged
Conversation
…out moving the rest MatrixMetaService keeps its change log as a Vec bounded at 1,024 records. Once it has filled it is always at the bound, and every append drops the oldest record with drain(0..1), which shifts every record the log keeps. That is paid on every put, for ever. A profile of the journal door at 8,000 puts of 1 KiB put 27.7% of its instructions in append_metadata_change and 33% in memmove, and the per-put cost was 181,052 instructions against 124,903 at 1,000 puts: 45% more, from a term that was constant per put once the log was full. The object change feed had exactly this defect and was cured by holding it as a deque; this is the second place it was found. The log is a VecDeque now, with the trim helper that took a Vec replaced by one that takes a deque (nothing else used it). Behaviour is unchanged: the bound, the order and what a snapshot carries are the same, and a snapshot still holds a Vec at its edge. The declaration is pinned by a test that fails if the field goes back to a Vec; it excludes comments and asserts the declaration exists once, so it cannot pass checking nothing. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was wrong
MatrixMetaServicekeeps its change log as aVec<MetadataChange>bounded at 1,024 records. Once the log has filled it is always at the bound, and every append drops the oldest record withdrain(0..1), which shifts every record the log keeps. That is paid on every put, for ever, so it is invisible in a short profile and dominates a long one.Callgrind of the journal door (three replicas, 1 KiB puts) found it by scale rather than by size: per-put instructions were not flat.
At 8,000 puts
append_metadata_changewas 27.7% of all instructions (~50,000 per put),memmove33%, and theMetadataChangedrop 27.6%. The object change feed had exactly this defect and was cured by holding it as a deque; this is the second place it was found.The change
The log is a
VecDeque<MetadataChange>. Dropping the oldest record moves nothing. The trim helper that took aVecis replaced by one that takes a deque (nothing else used it). Behaviour is unchanged: the bound, the order and what a snapshot carries are the same, and a snapshot still holds aVecat its edge (converted on export and install). About 15 call sites change mechanically (first/lastbecomefront/back,pushbecomespush_back).Measured
Callgrind, journal door, three replicas, 1 KiB puts, instructions per put:
Across an 8x larger store the per-put cost now rises 5% instead of 45%.
Tests
the_metadata_change_log_is_a_deque_because_it_drops_one_record_per_append: pins the field declaration as a deque. Comments are excluded and the declaration count is asserted, so it cannot pass checking nothing. Planted check: with its expected type changed toVec, it fails.the_published_mutation_copy_figures_are_the_ones_this_tree_produces(README 2485 vs tree 2557), fails identically on unmodifiedmainand is untouched here.🤖 Generated with Claude Code