feat(consumers): Auto-restart on single-partition lag

[sentry-junior]

Automatically resolve single-partition consumer lag by failing the pod healthcheck so Kubernetes restarts the member and Kafka rebalances.

Why

We saw one assigned partition fall far behind peers while offsets still moved slowly. Kafka does not rebalance in that case. GLOBAL subscription schedulers wait on the slow partition and shed stale work.

A hard "no commits at all" check is not enough. The observed failure was throughput collapse, not a hung poll loop.

What

Option-gated watchdog in the Snuba rust healthcheck (--health-check-file required):

1. Hard stall (consumer.partition_stall_timeout_secs > 0): partition has in-flight work and no commit progress past the timeout.
2. Relative slowdown (default on when the timeout is set): over one timeout-sized window, a partition's commit rate is below consumer.partition_slow_ratio of the median sibling rate on this assignment while still receiving work. Default ratio is 0.25. Set ratio to 0 for hard-stall only.

Either failure stops touching the health file. The k8s liveness probe restarts the pod and Kafka reassigns the partitions.

Enabling the stall timeout auto-selects the Snuba healthcheck even if --health-check is still arroyo.

Naming / docs

• Renamed experimental_healthcheck → consumer.commit_progress_healthcheck (consumer-level commit/idle progress).
• Kept experimental_healthcheck as a legacy alias.
• Documented strategy implementations and progress modes in docs/source/architecture/consumer.rst (Healthchecks).

Enable

Via sentry-options (no deploy flag change if health-check file is already set):

• consumer.partition_stall_timeout_secs: start around 300
• consumer.partition_slow_ratio: default 0.25 (optional)
• consumer.commit_progress_healthcheck: optional consumer-level mode

Follow-up outside this PR: rename the key in getsentry/sentry-options-automator (experimental_healthcheck: false → consumer.commit_progress_healthcheck: false). Safe either way while the alias exists.

Limits

• Relative check needs ≥2 active partitions on the same pod. Single-partition members only get the hard-stall check.
• Quiet assignments (median sibling rate under 50 offsets/s) skip relative slowdown to avoid false positives.
• Restart helps when the owning member is the problem. If the partition is inherently hot/data-path broken, lag can follow the reassignment.
• Python consumers still use the arroyo poll-only healthcheck.

Test plan

• cargo test --lib strategies::healthcheck (9 tests, including legacy alias + relative slowdown)
• Enable options in a non-prod region on a multi-partition rust consumer with health-check file
• Confirm healthy steady state still touches /tmp/health.txt
• Confirm a collapsed partition fails liveness and lag moves after restart

Requested by PDPM.

--

View Junior Session in Sentry

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 9abdb3a. Configure here.}