Skip to content

A leader queues its append packets before its own fsync - #637

Merged
matrixarkai merged 1 commit into
mainfrom
perf/a-leader-queues-its-append-packets-before-its-fsync
Nov 14, 2026
Merged

matrixarkai merged 1 commit into
mainfrom
perf/a-leader-queues-its-append-packets-before-its-fsync

Conversation

@matrixarkai

Copy link
Copy Markdown
Owner

What

The metadata and chunk consensus loops queued a Ready's packets for the followers only after process_ready had synced the journal, so a leader's replication waited behind its own fsync. A leader's append and heartbeat packets do not depend on its own copy being durable (matrixraft counts the leader's entry toward a quorum only after it reports the Ready persisted), and matrixraft can hand those packets over before the sync through an opt-in sink (DurablePeer::set_early_packet_sink).

This installs that sink at the start of each loop. Every packet that does depend on durability (all non-leader packets, every acknowledgment) still comes back from process_ready and is queued after it, as before.

The two places that queue packets now share one helper, queue_packets, so both paths follow a single rule: each packet goes to the queue of the peer it names, a full queue is counted as rejected and never waited on, and a packet for a peer with no queue is dropped.

Measured

This repository's network_chunk_consensus fixture: three real chunkserver processes, raft over TCP, a real WAL fsync on every replica. A scratch test that is not part of this change timed 200 sequential replicated 256-byte writes through the leader. Eight interleaved paired rounds, alternating which variant went first, each variant launching its own chunkserver (the two binaries were checked to differ before any run: the sink build calls set_early_packet_sink and the baseline does not).

median per write range
without the sink 15.2 ms 12.7 to 18.4 ms
with the sink 10.0 ms 9.3 to 11.5 ms

About 1.5x. The sink was faster in all 8 paired rounds, and the two ranges do not overlap.

Caveats: a debug build; one disk shared with other work that was under I/O pressure; sequential writes from a single client; one fixture. On a faster disk the fsync is a smaller share of a write and the gain would shrink.

Verified

  • A unit test for queue_packets: a packet reaches the peer it names, a full queue is counted as rejected and never waited on, a packet for an unknown peer is dropped. Two planted faults each turn it red: no longer counting a full queue, and dropping every packet.
  • 223 cluster library tests pass, and star_repaird's test passes.
  • With the sink build verified in place (checked by the string it contains), run one test at a time: network_metadata_consensus passes in full (0 failures), network_chunk_consensus passes in full (0 failures), and chunk_membership_controller passes (3).
  • cargo check --all-targets with warnings denied is clean for the default features and for --all-features.

A flaky box, stated plainly

Running those same two files with the default parallelism on this shared machine fails a few tests per run, and which tests changes from run to run. The cause every time is binding 127.0.0.1:PORT failed: Address already in use: the fixtures reserve ports and release them before the node binds, and on a busy machine something else takes the port in between. The unchanged baseline fails just as often (18 failures over six runs, against 21 with the sink, all with that cause). So I only count the one-test-at-a-time runs as evidence, and CI here will tell whether it is clean on a quiet runner.

🤖 Generated with Claude Code

The metadata and chunk consensus loops handed a Ready's packets to the
peer queues only after process_ready had synced the journal, so a
leader's replication to its followers waited behind its own fsync.

A leader's append and heartbeat packets do not depend on its own copy
being durable: matrixraft counts the leader's entry toward a quorum only
after it reports the Ready persisted. matrixraft can hand those packets
over before the sync through an opt-in sink, so install one at the start
of each loop, and let the packets that do depend on durability keep
returning from process_ready and queueing after it.

The two places that queue packets now share one helper, queue_packets,
so the packets a Ready returns after it is durable and the packets a
leader hands over before it is durable follow a single rule: each goes
to the queue of the peer it names, a full queue is counted and never
waited on, and a packet for a peer with no queue is dropped.

Measured through this repository's three-process chunk group, 200
sequential 256-byte replicated writes, 8 interleaved rounds: median
15.2 ms without the sink and 10.0 ms with it, faster in all 8 paired
rounds. A debug build on a shared disk, one fixture.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@matrixarkai
matrixarkai merged commit 43df3b3 into main Nov 14, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants