Repository navigation
A batch put into the SSD tier moves its keys in the disk order by key, not by walking it - #456
Merged
Conversation
…, not by walking it Staging a batch for the SSD tier removed the batch's keys from the disk order with `retain`, which visits every entry in the order, and then pushed them back at the end. One batch therefore cost as much as the number of entries on disk, however few keys it staged. This is the same walk that `invalidate_keys_locked` used to do before #454, on the write path. Each staged key now leaves the order by its own handle and is pushed back as before, so the keys that were not in the batch keep their places and the batch follows them in the order it was staged. The batch's keys are unique (`staged_ssd_positions` collapses repeats), so this removes exactly what the walk removed. The set switch at eight keys is gone with the walk it served. Tests: the order after a batch is compared with the order the walk would have left, at batch sizes either side of eight, for rewritten keys, and for a mix of rewritten and new keys; the test passes on the old code as well as the new, which is what makes it an oracle. A source guard pins that the batch path does not walk the order again. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Staging a batch for the SSD tier removed the batch's keys from the disk order with
retain, which visits every entry in the order, and then pushed them back at the end. One batch therefore cost as much as the number of entries on disk, however few keys it staged. This is the same walk thatinvalidate_keys_lockedused to do before #454, here on the write path. Each staged key now leaves the order by its own handle and is pushed back as before. The batch's keys are unique (staged_ssd_positionscollapses repeats), so this removes exactly what the walk removed, and the set switch at eight keys goes with the walk it served.Measured (SSD-only cache,
put_batchof 50 new keys, best of 6, debug build, so read the shape and not the microseconds):The walk grows with the disk tier (about 5x for 16x the entries); the per-key removal does not (the three figures on this side are not monotonic, so treat them as flat within noise).
Scope, stated plainly: single
puts do not reach this path in a way that costs anything. Oneputof an existing key and oneputof a new key were both flat on main (86-105 us at 1,000 to 16,000 entries) and flat on this branch, so this helps callers that write throughput_batch, not callers that write one key at a time.Tests (
src/tests/put_disk_order.rs): the order after a batch is compared with the order the walk would have left (the keys not in the batch keep their places, then the batch in the order it was staged), at batch sizes 1, 2, 7, 8, 9, 40 for rewritten keys and for a mix of 5 rewritten and 4 new keys; the order and the index hold the same keys. That test passes on main as well as on this branch, which is what makes the old result the oracle. A source guard pins that the batch path does not walk the order again; it fails on main and passes here. Two planted defects are caught: the walk restored (caught by the guard) and no staged key removed (caught by both tests).Checks:
cargo test --lib: 863 pass, 0 fail.cargo fmt --all --check: no differences.🤖 Generated with Claude Code