Repository navigation
Conversation
4561117 to
94dcd0e
Compare
3e84730 to
05bee02
Compare
fc1ff2b to
7d3c4e5
Compare
Each prepared plan leaked one heartbeat.BlipReader goroutine when repl.lag was collected at more than one level, which is common because levels inherit domains. Lag.Prepare calls prepareBlip for every such level, and after the first call prepareBlip returned a nil cleanup func, overwriting the one that stops the reader. Engine.Stop therefore never stopped it. The leaked reader kept querying a closed database handle and wrote errors into the status of the next monitor with the same ID. - prepareBlip returns the same cleanup func for every level. - Lag.Prepare stops its reader if it fails after starting it. - Engine.Prepare calls the cleanup funcs of collectors it discards on error. - BlipReader waits return when Stop is called, so it exits promptly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TestPrepareBlipAtMultipleLevelsReturnsReaderCleanup counted readers relative to a baseline taken at the start of the test. TestCollectWithAllSources runs earlier and never stopped its blip reader, which starts two goroutine hops after Prepare returns. Under load that reader started after the baseline was taken, so the test saw one extra reader and failed. - Earlier repl.lag tests stop the readers they start. - Tests that count readers wait for zero running readers first, then expect exactly the readers they start. - Waits allow 5s, more than heartbeat.ReadTimeout, for a reader to exit after an in-flight query. The engine test in monitor counted readers the same way, so it gets the same change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
7d3c4e5 to
e663e58
Compare
zyin-squareup
left a comment
There was a problem hiding this comment.
🤖 This review was generated by an AI agent on behalf of @zyin-squareup.
Summary: Reviewed only ee8d0c8 and e663e58. The fix is correct. prepareBlip now returns c.stopReader for every level. Before, each later level returned nil, which overwrote the cleanup in Lag.Prepare. That overwrite always happened, because configured is never assigned, so every level reaches prepareBlip. Now Engine.stopCollectors gets a real cleanup func, and the leak is closed. The other error-path cleanups are consistent with the Collector.Prepare contract in collector.go:22-27 ("called once when the collector is destroyed"):
- the deferred stop on named
errinLag.Prepare - the deferred cleanup of discarded collectors in
Engine.Prepare - the cleanup of the failing collector
Calling BlipReader.Stop twice is safe, because it checks stopChan and doneChan before closing. repl.lag is the only in-tree collector that returns a cleanup func, so the new engine error-path cleanup cannot affect other domains. The early return from wait() only changes behaviour on Stop; otherwise the reader keeps its old time.Sleep timing.
Blocking: none.
Non-blocking: one inline note on heartbeat/reader.go: a single stale status write can still happen after Stop when a query is in flight.
Validation:
go build ./...andgo veton theheartbeat,metrics/repl.lag,monitorandtestpackages: pass.go test -run TestPrepareBlip ./metrics/repl.lag/: pass.- The two new
monitorengine tests need the MySQL fixture, which was not available locally. CIbuildis green. - I could not run revert checks in my sandbox. By reading the assertions, reverting
prepareBliptoreturn nil, nilwould always failTestPrepareBlipAtMultipleLevelsReturnsReaderCleanupat its nil-cleanupt.Fatal. Removing the deferred stop would failTestPrepareBlipErrorStopsReaderwith 1 reader running.
Why
Each prepared plan leaks one heartbeat reader goroutine (
heartbeat.BlipReader) whenrepl.lagis collected at more than one level. That's common, because levels inherit domains.Lag.PreparecallsprepareBlipfor every level. After the first call,prepareBlipreturns a nil cleanup func, which overwrites the one that stops the reader. SoEngine.Stopnever stops it. The leaked reader keeps querying a closed database handle and writes errors into the status of the next monitor with the same ID.What
prepareBlipreturns the same cleanup func for every level.Lag.Preparestops its reader if it fails after it starts the reader.Engine.Preparecalls the cleanup funcs of collectors it discards on error.BlipReaderwaits return onStop, so the reader exits promptly.How
test/goroutine.goadds a small helper that counts goroutines, so tests can assert that a stop leaves none behind.Risk
Low. The change only affects cleanup on stop and on prepare errors.
Testing
A local benchmark ran 8 monitors in one process and restarted all of them in 10 waves.
Bigger picture
Part of a stack of 5 fixes found by the same benchmark. Monitor restarts are frequent when a host loses and regains its source; that's the case PRs 4 and 5 speed up.
🤖 Generated with Claude Code