fix(aio): async monitoring fixes (#1284, #1276, #1287, #1288)

[AhmadMasry]

Description

Fixes four async monitoring bugs in one PR, one commit per issue. They touch the same area (topology monitor, connection tracker, host monitoring), and #1288's fix relies on #1284's: it gives host_monitoring_v2 a topology provider, which without #1284 would add one topology monitor per connection.

Fixes #1284
Fixes #1276
Fixes #1287
Fixes #1288

Supersedes #1285, #1286 and #1289.

#1284: share the async topology monitor per cluster (4c05386)

Every async connect builds its own host list provider, and each provider started its own AsyncClusterTopologyMonitor with its own monitoring connection, so a pool of N connections held 2N server connections. Sync shares one monitor per cluster id through monitor_service.run_if_absent.

• aio/host_list_provider.py: module-level registry keyed by (provider class, cluster id, event loop); _get_or_create_monitor() reuses the running monitor and drops entries whose task has ended. The loop is in the key because an asyncio task only runs on its own loop; the provider class keeps Aurora, Multi-AZ and Global monitors apart.
• aio/cluster_topology_monitor.py: the monitor publishes topology to a weak set of subscribed providers.
• Lifecycle matches sync: closing a connection doesn't stop the cluster's monitor; stop() stops it (like stop_monitor), and release_resources_async() stops it at shutdown.

#1276: aurora_connection_tracker closed cluster-endpoint connections on their first statement (4b0ce7e)

The connect-time live-probe pin used current_host_info, which for a cluster-endpoint connection is the cluster endpoint. The first execute compared it with the topology writer (an instance), saw a writer change, and invalidate_all closed the connection about to execute. Sync always takes the writer from topology, so it compares instance with instance.

• The plugin remembers the instance a cluster-endpoint connection landed on (already identified by _fill_instance_alias) and pins that.
• _invalidate_departed_host resolves a cluster-endpoint pre_failover_host to the same instance, so idle connections to a demoted writer are still closed when topology lags after a failover.

#1287: host_monitoring_v2 monitors per event loop (c612461)

_cleanup_idle_monitors() ran on every monitored statement and stopped any monitor that wasn't is_usable(), which was False whenever the monitor ran on a different loop from the caller's. With one event loop per thread, loops stopped each other's monitors on every statement, recreating monitors and monitoring connections and leaving in-flight statements unmonitored.

• The registry is keyed by (monitor key, event loop); is_usable() is False only when the monitor is stopped, its task has ended, or its loop is closed; the registry is guarded by a lock, like sync's monitor service.
• An app with N event loops holds N monitoring connections per monitored host. One per loop is required because a monitor aborts connections that belong to its loop.

#1288: host_monitoring_v2 without a topology plugin on a cluster endpoint (26fa22c)

_build_host_list_provider only chose a topology provider for failover-type plugins. With host_monitoring_v2 alone on an Aurora cluster endpoint, identify_connection returned None and the first statement on every connection raised AwsWrapperError; the plugin had already cached the cluster endpoint, so later statements monitored the cluster DNS name.

• aio/wrapper.py: host_monitoring and host_monitoring_v2 get a topology provider when the database dialect is a TopologyAwareDatabaseDialect (Aurora, Multi-AZ, Global), as sync does for those dialects. Plain PostgreSQL/MySQL/RDS keep the static provider and pay nothing extra.
• aio/host_monitoring_plugin.py: _get_monitoring_host_info only caches the monitoring host once it is known. Sync already behaves this way (it overwrites the cached value with identify_connection's None).

Testing

• New unit tests (each failed before its fix, except the guard tests noted):
  • test_aio_shared_topology_monitor.py (Async: every connection opens its own topology monitor connection (pool of N holds 2N connections) #1284): 10 providers for one cluster share one monitor and one monitoring connection; topology reaches every provider; different clusters get separate monitors; the monitor outlives its connections and is reused; stop() stops it.
  • test_aio_aurora_connection_tracker.py ([aio] aurora_connection_tracker closes its own connection on the first statement of every cluster-endpoint connection #1276): test_cluster_endpoint_connection_survives_its_first_execute, test_failover_invalidates_departed_instance_of_cluster_endpoint_connection, and the guard test_reader_endpoint_failover_does_not_invalidate_the_writer.
  • test_aio_host_monitoring_plugin.py ([aio] host_monitoring_v2: event loops stop each other's monitors, recreating them on every statement #1287, [aio] host_monitoring_v2 without a topology plugin fails the first statement on cluster-endpoint connections #1288): test_event_loops_do_not_stop_each_others_monitors, the guard test_monitor_whose_loop_closed_is_replaced, and test_failed_instance_identification_is_not_cached.
  • test_aio_host_list_provider_autodetect.py ([aio] host_monitoring_v2 without a topology plugin fails the first statement on cluster-endpoint connections #1288): host monitoring gets a topology provider on Aurora PG/MySQL dialects and stays static on RDS/plain PG or with no dialect.
• Reproductions from the issues, without a database:
  • Async: every connection opens its own topology monitor connection (pool of N holds 2N connections) #1284: 10 connections went from 10 topology monitors / 10 monitoring connections to 1 / 1.
  • [aio] aurora_connection_tracker closes its own connection on the first statement of every cluster-endpoint connection #1276: the pin is the instance and the first statement no longer closes the connection.
  • [aio] host_monitoring_v2: event loops stop each other's monitors, recreating them on every statement #1287: 40 statements on 2 loops went from 40 monitors / 40 monitoring connections to 2 / 2, with the in-flight statement's monitor still running.
  • [aio] host_monitoring_v2 without a topology plugin fails the first statement on cluster-endpoint connections #1288: a cluster-endpoint connection with host_monitoring_v2 alone now monitors its instance.
• Full unit suite: 2,404 passed. flake8, isort and mypy are clean on the changed files.
• Not yet run against a live Aurora cluster.

Separate gap, not changed here: async blue/green doesn't stop the topology monitor on switchover, while sync does (blue_green_plugin.py:908).

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.