Skip to content

fix(bot): retry transient Telegram server errors instead of dropping responses - #196

Open
hcsum wants to merge 1 commit into
grinev:mainfrom
hcsum:fix/telegram-transient-server-errors
Open

fix(bot): retry transient Telegram server errors instead of dropping responses#196
hcsum wants to merge 1 commit into
grinev:mainfrom
hcsum:fix/telegram-transient-server-errors

Conversation

@hcsum

@hcsum hcsum commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

What happened

I hit this while using the bot. Telegram rate-limited me, and a few seconds later the same calls started coming back as 502 Bad Gateway. Every live message froze: the typing indicator disappeared, the streamed reply stopped updating, and nothing recovered afterwards.

From my logs, the whole thing takes 9 seconds:

[2026-07-25T01:14:52.788Z] [WARN] [ToolCallStreamer] Stream sync rate-limited, retrying in 5000ms:
  session=ses_0692c1fa..., reason=throttle_elapsed
  GrammyError: Call to 'sendMessage' failed! (429: Too Many Requests: retry after 5)

[2026-07-25T01:14:54.707Z] [WARN] [ResponseStreamer] Stream sync rate-limited, retrying in 5000ms:
  message=thinking:msg_f96d6dab..., reason=throttle_elapsed
  GrammyError: Call to 'sendMessage' failed! (429: Too Many Requests: retry after 5)

[2026-07-25T01:14:58.897Z] [ERROR] [ResponseStreamer] Stream marked as broken:
  message=msg_f96d6dab..., reason=throttle_elapsed,
  error=Call to 'sendMessage' failed! (502: Bad Gateway)

[2026-07-25T01:14:58.902Z] [ERROR] [ToolCallStreamer] Stream marked as broken:
  reason=throttle_elapsed, error=Call to 'sendMessage' failed! (502: Bad Gateway)

[2026-07-25T01:14:59.762Z] [ERROR] [ResponseStreamer] Stream marked as broken:
  message=msg_f96d6ad5..., reason=complete,
  error=Call to 'editMessageText' failed! (502: Bad Gateway)

[2026-07-25T01:15:01.516Z] [ERROR] [ResponseStreamer] Stream marked as broken:
  message=thinking:msg_f96d6dab..., reason=throttle_elapsed,
  error=Call to 'sendMessage' failed! (502: Bad Gateway)

The 429 at the start is handled correctly. It gets retried after 5 seconds, using the retry_after Telegram sends back. The 502 six seconds later gets no retry at all, and four streams (the reply, the thinking stream, the tool-call stream, and one final edit) are marked broken within 2.6 seconds. Once a stream is broken it stays broken for the rest of the run, so the message never updates again even though Telegram recovers a moment later.

Cause

getTelegramRetryAfterMs() in src/utils/telegram-rate-limit-retry.ts only classifies 429 as retryable. Everything else returns null, which the global bot.api.config middleware in src/bot/index.ts treats as "give up and rethrow". A 502 from Telegram's gateway is transient and usually succeeds on the next attempt, but it currently propagates on the first failure.

The same 502s also hit sendChatAction (that is the disappearing typing indicator) and finalizeAssistantResponse, where a failed send drops the finished reply entirely.

Fix

Treat 500/502/503/504 as retryable, with an exponential backoff of 1s, 2s, 4s, 8s capped at 8s. The 429 handling is unchanged and still honours retry_after.

The change is in the shared helper, so the existing global bot.api.config middleware picks it up for every API call. maxRetries stays at 5. Callers are not modified, so a persistent outage still marks the stream broken as before.

One log line in event-subscription-service.ts said CRITICAL: Stopping event processing due to error, which does not describe what the code does: the catch block clears state, marks the session idle, and event processing continues. Changed it to say the response was dropped.

Network-level failures (HttpError: Network request for 'sendChatAction' failed!) show up in the same logs and are also not retried, but they carry no error_code and need different handling, so they are out of scope here.

Tests

Five new cases in tests/utils/telegram-rate-limit-retry.test.ts covering 5xx detection, the backoff curve and cap, retrying after a 502, and giving up after maxRetries.

Full suite passes (1194 tests) and npm run lint is clean. npm run typecheck reports three pre-existing date_time errors in src/bot/render/ from grammY typings; they reproduce on a clean main.

…responses

Telegram occasionally answers a Bot API call with a transient gateway error
(`Call to 'sendMessage' failed! (502: Bad Gateway)`) instead of a proper
response. Only `429` was treated as retryable, so a `502` propagated out of
`bot.api` and:

- `ResponseStreamer` / `ToolCallStreamer` marked the stream permanently broken,
  so the live message stopped updating for the rest of the run — typing simply
  disappeared;
- the assistant response finalizer dropped the finished reply entirely, and
  logged `CRITICAL: Stopping event processing`, which never described what
  actually happened (event processing keeps running).

Treat `500/502/503/504` as retryable with an exponential backoff capped at 8s,
alongside the existing `429` handling that honours `retry_after`. The retry
stays in the single existing boundary — the global `bot.api.config` middleware —
so every API call is covered without stacking retries on top of the callers'
own loops. The misleading log line is corrected.
@hcsum
hcsum marked this pull request as ready for review July 29, 2026 08:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant