Repository navigation
A leader queues its append packets before its own fsync - #637
Merged
matrixarkai merged 1 commit intoNov 14, 2026
Merged
Conversation
The metadata and chunk consensus loops handed a Ready's packets to the peer queues only after process_ready had synced the journal, so a leader's replication to its followers waited behind its own fsync. A leader's append and heartbeat packets do not depend on its own copy being durable: matrixraft counts the leader's entry toward a quorum only after it reports the Ready persisted. matrixraft can hand those packets over before the sync through an opt-in sink, so install one at the start of each loop, and let the packets that do depend on durability keep returning from process_ready and queueing after it. The two places that queue packets now share one helper, queue_packets, so the packets a Ready returns after it is durable and the packets a leader hands over before it is durable follow a single rule: each goes to the queue of the peer it names, a full queue is counted and never waited on, and a packet for a peer with no queue is dropped. Measured through this repository's three-process chunk group, 200 sequential 256-byte replicated writes, 8 interleaved rounds: median 15.2 ms without the sink and 10.0 ms with it, faster in all 8 paired rounds. A debug build on a shared disk, one fixture. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The metadata and chunk consensus loops queued a Ready's packets for the followers only after
process_readyhad synced the journal, so a leader's replication waited behind its ownfsync. A leader's append and heartbeat packets do not depend on its own copy being durable (matrixraft counts the leader's entry toward a quorum only after it reports the Ready persisted), and matrixraft can hand those packets over before the sync through an opt-in sink (DurablePeer::set_early_packet_sink).This installs that sink at the start of each loop. Every packet that does depend on durability (all non-leader packets, every acknowledgment) still comes back from
process_readyand is queued after it, as before.The two places that queue packets now share one helper,
queue_packets, so both paths follow a single rule: each packet goes to the queue of the peer it names, a full queue is counted as rejected and never waited on, and a packet for a peer with no queue is dropped.Measured
This repository's
network_chunk_consensusfixture: three realchunkserverprocesses, raft over TCP, a real WALfsyncon every replica. A scratch test that is not part of this change timed 200 sequential replicated 256-byte writes through the leader. Eight interleaved paired rounds, alternating which variant went first, each variant launching its ownchunkserver(the two binaries were checked to differ before any run: the sink build callsset_early_packet_sinkand the baseline does not).About 1.5x. The sink was faster in all 8 paired rounds, and the two ranges do not overlap.
Caveats: a debug build; one disk shared with other work that was under I/O pressure; sequential writes from a single client; one fixture. On a faster disk the fsync is a smaller share of a write and the gain would shrink.
Verified
queue_packets: a packet reaches the peer it names, a full queue is counted as rejected and never waited on, a packet for an unknown peer is dropped. Two planted faults each turn it red: no longer counting a full queue, and dropping every packet.star_repaird's test passes.network_metadata_consensuspasses in full (0 failures),network_chunk_consensuspasses in full (0 failures), andchunk_membership_controllerpasses (3).cargo check --all-targetswith warnings denied is clean for the default features and for--all-features.A flaky box, stated plainly
Running those same two files with the default parallelism on this shared machine fails a few tests per run, and which tests changes from run to run. The cause every time is
binding 127.0.0.1:PORT failed: Address already in use: the fixtures reserve ports and release them before the node binds, and on a busy machine something else takes the port in between. The unchanged baseline fails just as often (18 failures over six runs, against 21 with the sink, all with that cause). So I only count the one-test-at-a-time runs as evidence, and CI here will tell whether it is clean on a quiet runner.🤖 Generated with Claude Code