Skip to content

fix(aio): keep host_monitoring_v2 monitors per event loop - #1289

Closed
AhmadMasry wants to merge 1 commit into
aws:mainfrom
AhmadMasry:fix/aio-efm-per-loop-monitors
Closed

AhmadMasry wants to merge 1 commit into
aws:mainfrom
AhmadMasry:fix/aio-efm-per-loop-monitors

Conversation

@AhmadMasry

Copy link
Copy Markdown
Contributor

Description

Fixes #1287.

With the async API, host_monitoring_v2 monitors were stopped and recreated on every statement when an application runs more than one asyncio event loop (for example one per worker thread):

  • every monitored statement runs _cleanup_idle_monitors(), which stops monitors that aren't is_usable();
  • AsyncHostMonitorV2.is_usable() returned False whenever the monitor ran on a different loop from the caller's;
  • the registry key didn't include the loop, so loops shared and fought over one entry, and the cleanup step stopped other loops' monitors whatever their key.

Each recreation opened a new monitoring connection, and statements still running on the other loop lost failure detection.

Changes in aio/host_monitoring_plugin.py:

  • The registry is keyed by (monitor key, event loop). Each loop gets one monitor per host and detection settings and keeps reusing it. One per loop is needed because the monitor is an asyncio task and aborts connections that belong to its loop.
  • is_usable() is False only when the monitor is stopped, its task has finished, or its loop is closed. Idle expiry is unchanged, and stopping another loop's idle monitor already goes through cancel_task_threadsafe.
  • The module registry is guarded by a lock, as the sync monitor service guards its registry with an RLock.

Compared with sync, which shares one monitor per settings and host across all threads, an application with N event loops now holds N monitoring connections per monitored host, instead of a new one per statement.

Testing:

  • New tests in tests/unit/test_aio_host_monitoring_plugin.py:
    • test_event_loops_do_not_stop_each_others_monitors: statements alternating between a loop on another thread and the test's loop keep exactly two monitors, none stopped. Failed before the fix (one monitor left).
    • test_monitor_whose_loop_closed_is_replaced: a monitor whose loop has closed is still stopped and replaced.
  • The reproduction in [aio] host_monitoring_v2: event loops stop each other's monitors, recreating them on every statement #1287 (40 statements on 2 loops, 1 host) went from 40 monitors and 40 monitoring connections to 2 and 2, with the in-flight statement's monitor still running. Same with different detection settings per loop.
  • Full unit suite: 2,393 passed. flake8, isort and mypy are clean on the changed files.
  • Not yet run against a live Aurora cluster.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Every monitored statement ran _cleanup_idle_monitors, which stopped any
monitor that wasn't is_usable(), and is_usable() was False whenever the
monitor ran on a different loop from the caller's. With one event loop
per thread, each loop stopped the other loops' monitors on every
statement: monitors and their monitoring connections were recreated
continuously, and in-flight statements on the other loop lost failure
detection.

Key the registry by monitor key and event loop, so each loop reuses its
own monitor, and treat a monitor as unusable only when it is stopped, its
task has finished, or its loop is closed. Guard the registry with a lock,
as sync's monitor service does.

Fixes aws#1287
@AhmadMasry

Copy link
Copy Markdown
Contributor Author

Superseded by #1290, which combines this fix with the related async monitoring fixes (#1284, #1276, #1287, #1288) in one PR.

@AhmadMasry AhmadMasry closed this Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[aio] host_monitoring_v2: event loops stop each other's monitors, recreating them on every statement

1 participant