CASSANDRA-21654: Coordinator node experiences read failures on ungraceful shutdown of replica - #5138
Open
isaacreath wants to merge 1 commit into
Open
CASSANDRA-21654: Coordinator node experiences read failures on ungraceful shutdown of replica#5138isaacreath wants to merge 1 commit into
isaacreath wants to merge 1 commit into
Conversation
…eful shutdown of replica (draft)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Currently, when a node has crashed but has not yet been evicted from Gossip, coordinator nodes can continue selecting that node as a replica for read and write requests. When this happens, messages will begin to pile up on the
OutboundConnectionuntilpendingCapacityInBytesis reached for the connection. Once this has been reached, some requests to that connection start to fail due to the connection becoming overloaded. At high throughput, this surfaces as a read failure to clients even when other nodes in the cluster are healthy and could have serviced the request.This PR changes the behavior on read requests when a replica connection reaches the maximum pending bytes. If a table has speculative retry enabled and the coordinator node has exhausted the send queue for that replica's
OutboundConnection, we will perform an immediate speculation to another replica in the cluster so long as we have not exhausted the global pool for outbound requests from the node.If the table has configured speculative retry as
NONE, or the client is usingEACH_QUORUMorALLfor reads, we will not speculate to ensure that we retain the existing non-speculative behavior. Additionally, we guard this new behavior behind a feature flag,read_fallback_on_overloaded_connection, to ensure that users can safely roll out and test the new behavior in their fleets.