fix(bot): retry transient Telegram server errors instead of dropping responses - #196
Open
hcsum wants to merge 1 commit into
Open
fix(bot): retry transient Telegram server errors instead of dropping responses#196hcsum wants to merge 1 commit into
hcsum wants to merge 1 commit into
Conversation
…responses Telegram occasionally answers a Bot API call with a transient gateway error (`Call to 'sendMessage' failed! (502: Bad Gateway)`) instead of a proper response. Only `429` was treated as retryable, so a `502` propagated out of `bot.api` and: - `ResponseStreamer` / `ToolCallStreamer` marked the stream permanently broken, so the live message stopped updating for the rest of the run — typing simply disappeared; - the assistant response finalizer dropped the finished reply entirely, and logged `CRITICAL: Stopping event processing`, which never described what actually happened (event processing keeps running). Treat `500/502/503/504` as retryable with an exponential backoff capped at 8s, alongside the existing `429` handling that honours `retry_after`. The retry stays in the single existing boundary — the global `bot.api.config` middleware — so every API call is covered without stacking retries on top of the callers' own loops. The misleading log line is corrected.
hcsum
marked this pull request as ready for review
July 29, 2026 08:39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What happened
I hit this while using the bot. Telegram rate-limited me, and a few seconds later the same calls started coming back as
502 Bad Gateway. Every live message froze: the typing indicator disappeared, the streamed reply stopped updating, and nothing recovered afterwards.From my logs, the whole thing takes 9 seconds:
The
429at the start is handled correctly. It gets retried after 5 seconds, using theretry_afterTelegram sends back. The502six seconds later gets no retry at all, and four streams (the reply, the thinking stream, the tool-call stream, and one final edit) are marked broken within 2.6 seconds. Once a stream is broken it stays broken for the rest of the run, so the message never updates again even though Telegram recovers a moment later.Cause
getTelegramRetryAfterMs()insrc/utils/telegram-rate-limit-retry.tsonly classifies429as retryable. Everything else returnsnull, which the globalbot.api.configmiddleware insrc/bot/index.tstreats as "give up and rethrow". A502from Telegram's gateway is transient and usually succeeds on the next attempt, but it currently propagates on the first failure.The same 502s also hit
sendChatAction(that is the disappearing typing indicator) andfinalizeAssistantResponse, where a failed send drops the finished reply entirely.Fix
Treat
500/502/503/504as retryable, with an exponential backoff of 1s, 2s, 4s, 8s capped at 8s. The429handling is unchanged and still honoursretry_after.The change is in the shared helper, so the existing global
bot.api.configmiddleware picks it up for every API call.maxRetriesstays at 5. Callers are not modified, so a persistent outage still marks the stream broken as before.One log line in
event-subscription-service.tssaidCRITICAL: Stopping event processing due to error, which does not describe what the code does: the catch block clears state, marks the session idle, and event processing continues. Changed it to say the response was dropped.Network-level failures (
HttpError: Network request for 'sendChatAction' failed!) show up in the same logs and are also not retried, but they carry noerror_codeand need different handling, so they are out of scope here.Tests
Five new cases in
tests/utils/telegram-rate-limit-retry.test.tscovering 5xx detection, the backoff curve and cap, retrying after a502, and giving up aftermaxRetries.Full suite passes (1194 tests) and
npm run lintis clean.npm run typecheckreports three pre-existingdate_timeerrors insrc/bot/render/from grammY typings; they reproduce on a cleanmain.