Skip to content

Stop the level collector without waiting for the next tick - #191

Draft
Areson wants to merge 1 commit into
Areson/parallel-monitor-stopfrom
Areson/level-collector-stop
Draft

Areson wants to merge 1 commit into
Areson/parallel-monitor-stopfrom
Areson/level-collector-stop

Conversation

@Areson

@Areson Areson commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Stack: PR 5 of 5 (top). Stacked on #190. Review only this PR's commit, 66c753e.

Why

The level collector checks for stop only after each 1 s ticker tick. So a monitor stop waits up to 1 s, 0.5 s on average. When monitors are removed one reload at a time, that adds about 0.5 s per removed monitor.

What

The level collector's Run loop waits for either the next tick or stop, not just the tick.

How

  • Run still checks for stop after a tick, because select picks at random when both are ready.
  • Dropped tick counting from Count collection faults per monitor #187 is unchanged.
  • A collection in progress still finishes before the collector stops. The engine max runtime bounds it.
  • Existing tests drive Run by sending one value per tick on a buffered stopChan, then closing it. Once Run gets a value, it keeps that one-per-tick behavior, so those tests see the close on the next tick as before.

Risk

Medium. Behavior change: a stop can now drop a batch that was collected but not yet sent to the sinks. The old code had the same race, but the wait for the next tick usually hid it. That's fine when a monitor stops because it lost its source. It can lose the last interval of metrics on a planned restart.

Testing

A local benchmark ran 400 monitors in one process against 4 MySQL 8.0 servers, with the whole stack applied. In a lock-loss storm, 200 of the 400 monitors lost their source at once, then got it back.

  • Every removed monitor stopped at once.
  • The removed monitors came back within 0.4 s of each other, against a 76 s spread before.
  • No monitor sent metrics after it was removed, against about 150 before.
  • 400 reloads ran, and none failed.
  • Stops went from up to 1 s to about 50 µs.

Bigger picture

Top of a stack of 5 fixes found by the same benchmark. Merge the stack bottom-up, starting with #187.

🤖 Generated with Claude Code

@Areson
Areson added this pull request to stack #192 October 21, 2026 21:52
@Areson
Areson force-pushed the Areson/level-collector-stop branch 2 times, most recently from 6f91e04 to b6e3bc9 Compare October 22, 2026 14:54
@Areson
Areson force-pushed the Areson/level-collector-stop branch from b6e3bc9 to 5a1b113 Compare October 22, 2026 15:51
@Areson
Areson force-pushed the Areson/level-collector-stop branch from 5a1b113 to ac29448 Compare October 22, 2026 16:01
@Areson
Areson force-pushed the Areson/level-collector-stop branch from ac29448 to ebf6b26 Compare October 22, 2026 17:28
The level collector checked for stop only after each ticker tick, so
stopping a monitor waited up to one tick (1s), 0.5s on average. With
removals processed one reload at a time, that wait added up to about 0.5s
per removed monitor.

Run now waits for either the next tick or stop. It still checks for stop
after a tick, because select picks randomly when both are ready. Dropped
tick counting is unchanged. A collection in progress still finishes
before the collector stops; it's bounded by the engine max runtime.

Tests drive Run by sending values on a buffered stopChan, one per tick,
then closing it. Once Run receives a value, it keeps that one-per-tick
behavior, so those tests see the close on the next tick as before.

With the 1s tick, a stop that took 0.3s to 0.9s depending on when in the
tick it arrived now takes well under a millisecond in the new test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Areson
Areson force-pushed the Areson/level-collector-stop branch from ebf6b26 to 66c753e Compare October 22, 2026 19:32

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant