Skip to content

fix(aio): pin the instance for cluster-endpoint connections in aurora_connection_tracker - #1286

Closed
AhmadMasry wants to merge 1 commit into
aws:mainfrom
AhmadMasry:fix/aio-tracker-cluster-endpoint-pin
Closed

AhmadMasry wants to merge 1 commit into
aws:mainfrom
AhmadMasry:fix/aio-tracker-cluster-endpoint-pin

Conversation

@AhmadMasry

Copy link
Copy Markdown
Contributor

Description

Fixes #1276.

With the async API and aurora_connection_tracker (part of the default plugin chain), every connection made through a cluster endpoint was closed by the tracker on its first statement, and the app saw FailoverSuccessError.

Cause: _pin_current_writer pins the writer from topology, then its live role probe replaces the pin with plugin_service.current_host_info when the connection is a writer. For a cluster-endpoint connection that is the cluster endpoint. On the first execute, _invalidate_writer_change compares the pin with the topology writer (an instance endpoint), sees a writer change, and invalidate_all closes every connection keyed under the cluster endpoint, including the one about to execute.

Sync doesn't hit this: AuroraConnectionTrackerPlugin always takes _current_writer from topology (Utils.get_writer(all_hosts)), so it compares instance with instance.

Changes in aio/aurora_connection_tracker.py:

  • _fill_instance_alias already identifies the instance a cluster-endpoint connection landed on. The plugin now keeps it as _connected_instance.
  • The live-probe pin uses that instance when the connection's host isn't an instance endpoint. If it couldn't be identified (for example a custom endpoint), the topology pin is kept. This matches sync, where the writer is always a topology instance.
  • _invalidate_departed_host resolves a cluster-endpoint pre_failover_host to _connected_instance. Without this, the fix above would stop the departed-host path from firing for cluster-endpoint connections when topology lags after a failover, because the pin (an instance) would no longer match the departed host (the cluster endpoint). After a move, _connected_instance follows the connection to its new instance.

Testing:

  • New unit tests in tests/unit/test_aio_aurora_connection_tracker.py:
    • test_cluster_endpoint_connection_survives_its_first_execute: the pin is the instance and the first execute doesn't close the connection. Failed before the fix.
    • test_failover_invalidates_departed_instance_of_cluster_endpoint_connection: with lagged topology, a failover from a cluster-endpoint connection still closes idle connections to the demoted writer.
    • test_reader_endpoint_failover_does_not_invalidate_the_writer: a reader-endpoint connection failing over between readers doesn't close writer connections.
  • Full unit suite: 2,394 passed. flake8, isort and mypy are clean on the changed files.
  • Not yet run against a live Aurora cluster.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…_connection_tracker

The connect-time live-probe pin used current_host_info, which for a
cluster-endpoint connection is the cluster endpoint. The first execute
compared it with the topology writer (an instance endpoint), saw a writer
change, and invalidate_all closed the connection about to run its
statement.

Remember the instance a cluster-endpoint connection landed on and pin
that, as sync does by always taking the writer from topology. The
departed-host failover path resolves a cluster endpoint to the same
instance, so idle connections to a demoted writer are still closed when
topology lags.

Fixes aws#1276
@AhmadMasry

Copy link
Copy Markdown
Contributor Author

Superseded by #1290, which combines this fix with the related async monitoring fixes (#1284, #1276, #1287, #1288) in one PR.

@AhmadMasry AhmadMasry closed this Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[aio] aurora_connection_tracker closes its own connection on the first statement of every cluster-endpoint connection

1 participant