Skip to content

A whole-object read of an appended object reads the chunk once per run of extents - #638

Open
matrixarkai wants to merge 1 commit into
mainfrom
perf/an-object-read-asks-the-chunk-once-per-contiguous-run
Open

matrixarkai wants to merge 1 commit into
mainfrom
perf/an-object-read-asks-the-chunk-once-per-contiguous-run

Conversation

@matrixarkai

Copy link
Copy Markdown
Owner

What

A whole-object read of an object built by appends now reads the chunk once per
run of nearby extents instead of once per extent.

Why

An object built by appends has one extent per append (the existing regression
test says so on purpose), and read_extents made a full chunk-range read for
each: admission, the page map, the head stats, and a replica quorum vote that
hashes what it returns. A 400 KB write-ahead log made of a few thousand appends
read at about 15 MB/s through the object server, against 85-135 MB/s for an
object of the same size written in one request, and the rate did not improve
with more concurrent clients.

How

In the multi-extent path of read_extents_with_stats_bounded_by_chunks,
consecutive extents of one chunk are batched while the next one starts within
4 KiB of the previous one's end and the span stays within 1 MiB. A batch is read
with one chunk-range read and the extents are sliced out of it.

  • Only when the caller set no payload budget and no chunk-count budget, since a
    span reads the bytes between two extents.
  • A span that cannot be read (for example the bytes between two extents belong
    to a neighbour that has since been discarded) falls back to one read per
    extent, so a read of an object never depends on its neighbour's bytes.
  • No durable byte changes; this is a reader-side change.

Measured

Release object server, same populate workload either side, A/B in both orders
(main, branch, branch, main):

before after
whole-object GET, one client 89 / 88 qps 360 / 306 qps
whole-object GET throughput about 20 MB/s 70-82 MB/s
sync write p50, async write p50, replica read p50 unchanged unchanged

A second run with two object servers driven at once reproduced it (35 / 44 qps
before, 159-163 qps after). At the store level in a debug build, 4000 extents read
in 31 ms against 67 ms in one order and 43 ms against 74 ms in the reversed order.
200 interleaved-append extents are served by 11 chunk reads (204 with the span
disabled).

Concurrency 4 and 16 still gain only about 1.3x over one client: the object
server holds one lock around the store for the whole GET, and the store's read
cache is updated by every read, so a read-write lock is not available. That is a
separate change and is not attempted here.

Tests

  • a_whole_object_read_of_interleaved_appends_reads_the_chunk_once_per_span:
    the bytes read equal the bytes appended, and the number of chunk reads is far
    below the number of extents. Fails with the span disabled (204 reads for 200
    extents).
  • a_span_read_does_not_depend_on_the_bytes_between_its_extents: deleting the
    interleaved neighbour leaves the object readable, whole and by range.
  • bench_whole_object_read_of_appended_object (ignored): prices the read at 250,
    1000 and 4000 extents.
  • Library suite: 696 pass. One failure,
    the_published_mutation_copy_figures_are_the_ones_this_tree_produces, also
    fails on the current main (the README figures are out of date) and is not
    touched by this change.

🤖 Generated with Claude Code

…n of extents

An object built by appends has one extent per append, and a whole-object read
paid a full chunk-range read for each of them: admission, the page map, the
head stats and a replica vote that hashes what it returns. A 400 KB write-ahead
log made of a few thousand appends therefore read at about 15 MB/s, against
85-135 MB/s for an object of the same size written in one request.

Consecutive extents of one chunk that sit close together are now read as one
span and sliced apart, so the fixed price is paid once per run instead of once
per extent. Two objects appended in turn share a chunk and interleave, so the
extents of one are separated by the other's; a span reads across up to 4 KiB
between two extents and at most 1 MiB in all.

Only taken when the caller set no payload and no chunk-count budget, because a
span reads the bytes between two extents and a budget is a promise about what a
read touches. A span that cannot be read falls back to one read per extent, so a
read of an object never depends on the bytes of its neighbour. No durable byte
changes.

Measured, release door, same workload either side, A/B in both orders:
  whole-object GET, one client   89 / 88 qps  ->  360 / 306 qps
  whole-object GET throughput    ~20 MB/s     ->  70-82 MB/s
  write latency and replica read latency unchanged
At the store level in a debug build, 4000 extents read in 31 ms against 67 ms
and 43 ms against 74 ms in the reversed order. 200 interleaved-append extents are
served by 11 chunk reads (204 with the span disabled).

Tests: the bytes read equal the bytes appended, including after the interleaved
neighbour is deleted; the read-count assertion fails with the span disabled; an
ignored bench prices the read at three object sizes.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants