Repository navigation
Accept retry options on built-in sinks - #194
Conversation
15dad2b to
bbcc87f
Compare
sink.Make parsed buffer-size, send-timeout, and send-retry-wait into RetryArgs but then passed the full options map to the specific sink constructor, which rejects unknown options. Any monitor that set a retry option failed to build its sink. Pass the specific sink, and the HTTP client factory, a copy of the options without the retry keys. Defaults are unchanged. Also fix the retry sink docs, which showed a separate "retry" sink section that Make rejects as unregistered. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reject buffer-size outside 1..10000 (MAX_RETRY_BUFFER_SIZE) and send-timeout or send-retry-wait that isn't greater than zero. A negative send-timeout made Retry buffer without ever sending, a zero silently meant the default, and a huge buffer-size could allocate a huge buffer slice. Parse and range errors now name the sink and option. Clone the options and delete each retry key where it's parsed, so a new retry option can't be missed when stripping. Document the valid values, link the retry options from each built-in sink page, and note the removed sinks.retry section in the release notes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bbcc87f to
db29086
Compare
Cap send-timeout at 1m (MAX_RETRY_SEND_TIMEOUT). The level collector sends to a monitor's sinks one at a time without its own timeout, so a long send-timeout stalled the monitor's other sinks in an outage. Stop the wait between sends when send-timeout expires. time.Sleep ignored the timeout, so a long send-retry-wait overran it. Move the range checks into RetryArgs.Validate, which the factory calls and NewRetry enforces with a panic like its other argument checks, so custom sinks that build RetryArgs get the same bounds. Parse the retry options in one loop that deletes each key and wraps each error in one place, and derive test boundaries from the constants. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Remove the Validate panic from NewRetry. NewRetry is exported, so panicking on values that v2.1.0 accepted would break Go callers in a minor release. NewRetry handles every value as before; the factory still validates sink options, and RetryArgs and NewRetry now say that callers building RetryArgs from user input should call Validate. Parse the retry options in a fixed order instead of map order, so the same invalid options always return the same error. Fix the Retry comment: "noop" doesn't use Retry either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zyin-squareup
left a comment
There was a problem hiding this comment.
🤖 Review findings - created by zyin-squareup's code review agent - for /pull/194, 7c8d3be.
Verdict: ship — clean, approve.
This PR makes the built-in sinks accept the retry options buffer-size, send-timeout, and send-retry-wait. The factory now parses them in a fixed order, removes them from a copy of the options map, and checks their ranges. It also stops the send-retry-wait wait when send-timeout expires. The riskiest piece was the new early return in Retry.Send; tracing it showed no regression and no lost metrics. I found no defects.
Assumptions & unknowns
- Reviewed head
7c8d3bebf7876f0105d51d60a031bb39942bbd93against a freshly fetchedorigin/main. The PR is 4 commits ahead and 0 behind. - I couldn't run local validation because this review session's permissions don't allow
gocommands. All conclusions below come from reading the source, plus CI.
The one thing that could have broken, verified
I checked the new interruptible wait in Retry.Send (retry.go#L214-L222) by reading the source:
- On
ctx2.Done()it returnsnil. That's the same as the existing check at the top of the loop (#L206-L210). - The pending item is still on the stack, because
popremoves only an item that was sent. So the next cycle retries it, and nothing is dropped or counted twice. - The deferred calls still reset
sendingand cancelctx2. No lock is held during the wait, and the timer is stopped on the early-return path. - Base slept without checking the timeout, then sent with an already-expired context, which logged a spurious
SINK_SEND_ERROR. Head skips that send. - The production caller passes
context.Background()(monitor/level_collector.go), so onlysend-timeoutcancelsctx2.
Verified correct
- No config that loaded before now fails. At base, all four constructors reject unknown keys with
invalid option, so any retry option already failed. That includes the0and negative values base would otherwise have accepted. - Retry keys don't collide with any sink's own option names. The caller's map isn't changed, because
maps.Clonecopies it. Nil options (a sink listed without options inmonitor/loader.go) still work, because cloning, reading, and deleting are all safe on a nil map. logandnoopstill return before the retry options are parsed (factory.go#L102-L107). Custom registered sinks go through their own factory and aren't affected.- Defaults are unchanged. An unset option stays zero,
Validateaccepts zero, andNewRetryreplaces zero with the default. Users can no longer set a zero value explicitly.NewRetrystill doesn't validate its arguments, so existing Go callers keep working. - datadog is still built as
NewDelta(NewRetry(...)), and bothHTTPClient.MakeForSinkand the sink constructors receive the stripped options. - Tests:
TestFactoryMake_RetryOptionsenters throughsink.Makefor all four sinks. By reading the code, it would fail on base withinvalid option.TestRetry_SendTimeoutBoundsRetryWaitwould fail with the old unconditional sleep. I couldn't confirm either by running them.
- Docs match the code: the valid ranges, defaults, the statement that a monitor sends to its sinks one at a time, and the removal of the
sinks.retrysection. - CI: build, DCO, Semgrep, and zizmor pass.
- Not applicable: security, migrations, wire compatibility.
This review was generated by Claude Code (claude-opus-5-5). Local tests were not run: this review session's permissions don't allow go commands, so I relied on CI and reading the source.
sink.Makeparses the retry optionsbuffer-size,send-timeout, andsend-retry-waitintoRetryArgs, then passes the full options map to the specific sink constructor. The chronosphere, signalfx, datadog, and prom-pushgateway sinks all returninvalid option: <key>for keys they don't know, so a monitor that sets any retry option can't build its sink.This PR changes
Maketo parse the three options in one loop, in a fixed order, so the same invalid options always return the same error. The specific sink andHTTPClient.MakeForSinkget a clone of the options with the retry keys deleted. The caller's map isn't modified, and defaults stay the same.It also validates the retry options:
buffer-sizemust be from 1 to 10000 (MAX_RETRY_BUFFER_SIZE). The upper bound stops a typo from allocating a huge buffer slice. 10000 is about 2.8 hours of batches at a 1s interval.send-timeoutmust be greater than zero and at most 1m (MAX_RETRY_SEND_TIMEOUT). The level collector sends to a monitor's sinks one at a time with no timeout of its own, so a longsend-timeoutwould stall the monitor's other sinks during an outage. A negativesend-timeoutmadeRetrybuffer without ever sending.send-retry-waitmust be greater than zero. A zero for either duration silently meant the default.Parse and range errors name the sink and option, for example
invalid datadog sink option send-timeout: .... Because a built-in sink used to reject any retry option, no config that loaded before now fails.The upper bounds and the negative checks live in the new exported
RetryArgs.Validate, which the factory calls.NewRetrydoesn't call it, so existing Go callers that pass values outside these ranges keep working as before. Custom sinks that buildRetryArgsfrom user input should callValidatefirst.Retry.Sendnow stops waitingsend-retry-waitbetween sends whensend-timeoutexpires. Previously asend-retry-waitlonger thansend-timeoutlet a send cycle run past the timeout by a whole wait.The retry sink docs and the config quick reference showed a separate
sinks.retrysection, whichMakerejects as an unregistered sink. Both now say to set the retry options on each built-in sink. The retry docs list the three options and their valid values, and each built-in sink page links to them.Testing
TestFactoryMake_RetryOptionsbuilds each built-in sink through the factory with all three retry options set. It checks that:RetrywrapperMakeForSinkgets only the sink-specific optionssink-retry-droppedcounterOn
mainit fails for all four sinks withinvalid option.TestFactoryMake_RetryDefaultschecks that the defaults don't change when no retry options are set.TestFactoryMake_InvalidRetryOptionschecks that non-numeric, zero, negative, and too-large values, and bad durations, are rejected with an error that names the sink and option. It also checks that the maximum values are accepted, and that with several invalid options the error always names the first in parse order.TestRetryArgsValidatechecks the ranges and thatNewRetrystill accepts out-of-range values without panicking.TestRetry_SendTimeoutBoundsRetryWaitchecks that a 1msend-retry-waitdoesn't holdSendpast a 200mssend-timeout. Without the fix, it fails.go vet ./...,go test ./..., andgo test -race ./sink/...pass.🤖 Generated with Claude Code