Repository navigation
fix(aio): pin the instance for cluster-endpoint connections in aurora_connection_tracker - #1286
Closed
AhmadMasry wants to merge 1 commit into
Closed
AhmadMasry wants to merge 1 commit into
AhmadMasry wants to merge 1 commit into
Conversation
…_connection_tracker The connect-time live-probe pin used current_host_info, which for a cluster-endpoint connection is the cluster endpoint. The first execute compared it with the topology writer (an instance endpoint), saw a writer change, and invalidate_all closed the connection about to run its statement. Remember the instance a cluster-endpoint connection landed on and pin that, as sync does by always taking the writer from topology. The departed-host failover path resolves a cluster endpoint to the same instance, so idle connections to a demoted writer are still closed when topology lags. Fixes aws#1276
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #1276.
With the async API and
aurora_connection_tracker(part of the default plugin chain), every connection made through a cluster endpoint was closed by the tracker on its first statement, and the app sawFailoverSuccessError.Cause:
_pin_current_writerpins the writer from topology, then its live role probe replaces the pin withplugin_service.current_host_infowhen the connection is a writer. For a cluster-endpoint connection that is the cluster endpoint. On the first execute,_invalidate_writer_changecompares the pin with the topology writer (an instance endpoint), sees a writer change, andinvalidate_allcloses every connection keyed under the cluster endpoint, including the one about to execute.Sync doesn't hit this:
AuroraConnectionTrackerPluginalways takes_current_writerfrom topology (Utils.get_writer(all_hosts)), so it compares instance with instance.Changes in
aio/aurora_connection_tracker.py:_fill_instance_aliasalready identifies the instance a cluster-endpoint connection landed on. The plugin now keeps it as_connected_instance._invalidate_departed_hostresolves a cluster-endpointpre_failover_hostto_connected_instance. Without this, the fix above would stop the departed-host path from firing for cluster-endpoint connections when topology lags after a failover, because the pin (an instance) would no longer match the departed host (the cluster endpoint). After a move,_connected_instancefollows the connection to its new instance.Testing:
tests/unit/test_aio_aurora_connection_tracker.py:test_cluster_endpoint_connection_survives_its_first_execute: the pin is the instance and the first execute doesn't close the connection. Failed before the fix.test_failover_invalidates_departed_instance_of_cluster_endpoint_connection: with lagged topology, a failover from a cluster-endpoint connection still closes idle connections to the demoted writer.test_reader_endpoint_failover_does_not_invalidate_the_writer: a reader-endpoint connection failing over between readers doesn't close writer connections.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.