fix(webhooks): stop admission-refusal retry loops and per-retry log writes by waleedlatif1 · Pull Request #8870 · simstudioai/sim

waleedlatif1 · 2026-10-10T01:26:21Z

Summary

Stops webhook senders from looping on refused deliveries, stops each retry from writing a new execution log, and stops pollers from hammering failing or blocked sources.

Root causes (from the webhook and poller logs)

A usage-limit refusal answered every provider with 402. Telegram resends a non-2xx update until it is acknowledged or 24h pass, so one over-limit bot looped all day. Each attempt ran the full gate and wrote a new execution row, a trace archive blob, and a workspace_files row after being refused.
Pollers ran the full gate per item for over-limit owners. They marked each item seen even when it was refused (lost), counted failures toward the 100-failure auto-disable, and refetched failing sources (including 429 FLOOD_WAIT_*) every minute.
A deleted path hit by Slack logged three warnings per request and was retried by Slack.

Admission codes

Refusals that hold until a person acts now carry stable codes next to WORKFLOW_NOT_DEPLOYED: USAGE_LIMIT_EXCEEDED and ACCOUNT_SUSPENDED.
An unreadable usage ledger (it fails closed as "exceeded") stays untagged and retryable, and so do reservation headroom and concurrency denials.

Ack-and-drop opt-in (acknowledgeAdmissionRejections on the provider handler)

An opted-in provider gets an empty 200 for a coded refusal, with an ignored outcome. Transient refusals still fail so the sender retries.
Telegram: opted in. It retries non-2xx until it gives up and keeps updates up to 24h (Bot API setWebhook).
Slack (custom and native): opted in. It retries 3 times within ~6 min, and failures count toward disabling the app's subscription at 95% over 60 min (Events API).
WhatsApp, Teams, Stripe and generic keep the error status. Meta retries up to 7 days with decreasing frequency, Graph backs off for up to 4h, Stripe retries for 3 days, and generic callers may read the 402. So an upgrade can still recover those events. GitHub does not redeliver.
A dropped target (acknowledged refusal or missing block) never masks another fan-out target's retryable failure.

Blocked-run log dedupe

Unattended webhook and poll refusals record at most one execution error row per workflow and gate per 15 minutes (Redis SET NX EX; fails open).
A caller-supplied logging session is always completed.

Polling

A coded refusal of a polled event stops the batch in every poller (RSS, Gmail, Outlook, IMAP, Drive, Sheets, Calendar, HubSpot). The poller throws a shared PollAdmissionRefusedError out of the item's idempotency callback.
While nothing in the batch has completed, the poller returns skipped without advancing its cursor or counting a failure, so the refused items stay pending. Once an item has completed, the refusal is an ordinary item failure, so completed work is saved exactly as before and never replays. RSS, which tracks items by GUID, always records the completed ones and leaves the rest unseen.
The refusal also records the workspace in Redis for 5 minutes. The orchestrator then skips those workspaces in one MGET: no fetch, no failure count, no state change. Healthy payers cost no billing reads, and billing-disabled deployments skip this.
RSS source fetch failures back off 2^(n-1) min (jittered, capped at 1h), measured from the poll's start. The wait is longer when the feed's Retry-After/FLOOD_WAIT_<n>, counted from its answer, asks for more.
A feed 4xx logs once at warn. Item-processing failures never back off.
Other pollers' source failures are unchanged.
Poll state keys are system-managed, so deploy change detection ignores them.

Deleted path

The POST 404 is unchanged but now carries x-slack-no-retry: 1, sent unconditionally, so it reveals nothing the 404 doesn't.
The duplicate warnings drop to debug, leaving the route handler's one line.

Telegram secret token

setWebhook registers a per-webhook secret_token, and deliveries without it get 401.
Back-compat: rows deployed before this have no stored secret and stay accepted until their next deploy.
A new registration reuses the active deployment's secret for the same bot, so cutover deliveries verify. Stored {{VAR}} bot tokens are resolved before comparing, which also stops retiring an old version from deleting the active bot's webhook.

Behavior changes

Telegram and Slack events that arrive while the owner is over limit or suspended are dropped instead of retried.
Slack slash commands in that state get a blank 200 instead of a Slack error.
A failing RSS feed now auto-disables after about 4 days of backed-off polling instead of about 100 minutes.
Polling resumes within 5 minutes of a limit being raised.

What to watch after deploy

Volume of 402 "Client error response" on /api/webhooks/trigger/* should collapse for Telegram and Slack paths. Generic webhooks keep their 402.
"Uploading file to execution storage" after "Attributed usage gate blocked actor" should drop to at most one per workflow per 15 min.
Poller ERROR lines for over-limit owners ("Failed to process webhook for item", "Error processing item") should stop; expect one info "Skipping polls for N workspaces refused by admission" per tick.
Rate-limited RSS feeds (429, FLOOD_WAIT) should log one warn per backed-off attempt instead of errors every minute.
Slack traffic to deleted paths or removed trigger blocks should drop by its retries, with one warn line per request.
Telegram: watch for "Rejected Telegram webhook request with invalid secret token" warnings after redeploys (expected 0).

Type of Change

Bug fix

Testing

Regression tests, each shown failing on pre-fix code:
- processor.test.ts: ack seam, transient and headroom refusals still fail, polled raw code, refusal recording.
- preprocessing.test.ts: codes, usage_unavailable, throttle per gate, supplied session.
- route.test.ts: 404 no-retry header, mixed fan-out (refusal vs. failure, missing block vs. failure).
- slack-dispatch.test.ts: mixed fan-out.
- rss.test.ts: refused item unseen, Retry-After.
- polling/utils.test.ts: backoff anchored at poll start, Retry-After.
- polling/orchestrator.test.ts: refusal skip, backoff skip, billing-disabled.
- polling/google-calendar.test.ts: a refused first event stops the batch with no cursor advance or failure count; a refusal after a completed event saves the cursor as before.
- telegram.test.ts: env-var token secret reuse and delete guard.
blocked-run-log.integration.ts against real Redis: one winner of 25 concurrent claims, per-gate keys, expiry, fail-open.
Locally: the affected Vitest suites (145 files) and type-check passed. check:realtime-prune passes on its own. CI runs the full check:audits (including check:unused-exports) and integration suites.

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing (new tests pass the test-audit authoring gate)
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

vercel · 2026-10-10T01:26:28Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Oct 10, 2026 2:47am UTC

cubic-dev-ai

All reported issues were addressed across 33 files

_{Turn on auto-fix | Re-trigger cubic}

greptile-apps · 2026-10-10T01:33:49Z

Confidence Score: 5/5

[High impact] The PR appears safe to merge; no new blocking issue was found.

Summary

The PR acknowledges blocked Telegram and Slack deliveries, limits repeated refusal logs, pauses blocked polling, adds RSS backoff, and checks Telegram delivery secrets.

Telegram and Slack drop lasting admission refusals instead of retrying them.
Repeated admission refusals create at most one blocked-run log per workflow and gate every 15 minutes.
Pollers leave admission-refused events pending and skip recently refused workspaces.
RSS feeds that fail get longer waits between fetch attempts.
Telegram webhook deliveries can be checked with a registered secret token.
Shared idempotency test mocks replace repeated mock setups without changing behavior.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Webhook delivery] --> B[Check admission]
  B --> C{Stable refusal?}
  C -->|No| D[Run workflow or return retryable error]
  C -->|Yes| E{Telegram or Slack?}
  E -->|Yes| F[Acknowledge and drop]
  E -->|No| G[Return refusal]
  H[Polled event] --> B
  C -->|Polled refusal| I[Record workspace pause]
  I --> J[Preserve pending work]
  I --> K[Skip workspace for five minutes]

_{Reviews (7) · Last reviewed commit: "chore(testing): stub IdempotencyService...." · Reviewed by Greptile}

waleedlatif1 · 2026-10-10T01:41:40Z

@greptile

waleedlatif1 · 2026-10-10T01:41:41Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-10T01:41:46Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 34 files

_{Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic}

waleedlatif1 · 2026-10-10T01:58:36Z

@greptile

waleedlatif1 · 2026-10-10T01:58:37Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-10T01:58:43Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 34 files

_{Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic}

waleedlatif1 · 2026-10-10T02:08:07Z

@greptile

waleedlatif1 · 2026-10-10T02:08:08Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-10T02:08:12Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 34 files

_{Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic}

…locked-run logs Usage-limit, suspended-account, and missing-billing-account refusals now carry stable codes, so surfaces can tell a refusal that holds until a person acts from a transient one. Unattended surfaces can opt into throttleErrorLogs: each refusal of a workflow by the same gate records at most one execution error row per 15-minute window (Redis SET NX, in-process LRU without Redis, fail-open on Redis errors). A sender resending a refused delivery no longer writes an execution row, trace archive, and workspace_files row per attempt. A caller-supplied logging session is always completed.

…egram and Slack Telegram resends a non-2xx update until it succeeds or 24 hours pass, and Slack disables an app's event subscription once most deliveries fail, so answering a usage-limit refusal with 402 only loops. Providers opt in with acknowledgeAdmissionRejections and get an empty 200 with an ignored outcome; transient refusals (rate limit, concurrency, reservation outage) still fail so the sender retries. Generic webhooks keep the 402. Polling receives the raw refusal with its code. Webhook preprocessing throttles its error rows.

…sources The poll orchestrator checks each workspace's payer once per tick and skips its webhooks while the payer is over its usage limit: nothing is fetched, marked seen, or counted as a failure, so items deliver once the payer is back under the limit and triggers are no longer auto-disabled over billing state. A webhook with consecutive poll failures waits 2^(n-1) minutes (capped at an hour) before the next fetch, and a source's Retry-After or FLOOD_WAIT_<n> is persisted and honored. RSS logs a source's 4xx once at warn, and an admission refusal mid-poll leaves the remaining items unseen.

…og them once A POST to a path with no webhook keeps its 404 but now carries x-slack-no-retry, sent unconditionally so it reveals nothing the 404 does not. The processor and route lines for an unknown path drop to debug, leaving the route handler's single client-error line.

Register a generated secret_token with setWebhook, store it as providerConfig.secretToken, and reject deliveries whose X-Telegram-Bot-Api-Secret-Token header does not match (401). Webhooks registered before this change have no stored secret and stay accepted until their next deploy registers one. A new registration reuses the active deployment's secret for the same bot so deliveries keep verifying during cutover. Drops the stale empty User-Agent warning: the proxy exempts webhook trigger routes from the empty-UA block.

An unreadable usage ledger fails closed as exceeded; it said nothing about the payer, yet it was tagged USAGE_LIMIT_EXCEEDED and acknowledged-and-dropped for Telegram and Slack. It now stays untagged and retryable. Reservation headroom denials clear as in-flight runs settle, so they leave the deterministic set too. The blocked-run log claim drops its in-process fallback: usage refusals only happen on hosted billing deployments, which run Redis, and without Redis every refusal records its row. A Redis failure logs at debug.

Active rows store the bot token as authored, often a {{VAR}} reference, while subscription calls receive it resolved, so the comparison never matched: every deploy minted a fresh secret (a candidate that never activated left the bot rejected by the active row), and retiring an old version could delete the webhook the active version still used. Stored tokens are now resolved against the background webhook env before comparing.

…source failures The per-tick payer pre-check read billing attribution and the usage ledger for every polled workspace, including healthy idle ones. A deterministic admission refusal of a polled event now records the workspace in Redis for five minutes, and the orchestrator skips only those workspaces in one MGET; healthy payers cost no billing reads, and billing-disabled deployments skip the mechanism. Backoff now follows source fetch failures only, tracked in providerConfig and stamped from the failed poll's start, so transient item-processing refusals never back a webhook off. Poll state keys are system-managed so deploy change detection ignores them.

…backoff Every poller's outer catch now records a source failure, so a failing Gmail, Outlook, IMAP, Drive, Sheets, Calendar, or HubSpot source backs off like RSS instead of only RSS. markWebhookSuccess clears the backoff in its existing reset write, the window uses the shared jittered backoff, and the orchestrator asks a boolean isPollBackedOff. Smaller cleanups: one isDroppedDispatch predicate for the Slack and path fan-outs, explicit precedence for a polled refusal's code, a typed RSS refusal error instead of a flag, Telegram resolves only the stored bot token, and PollOutcome lives with the polling types.

Poll outcome docs describe what a skipped poll actually does, source failures keep logging the full error object as the pollers did before, the backoff table pins the clock past the poll start so it proves the window is anchored there, and the RSS rate-limit test drives a Retry-After header.

… refusal Only RSS stopped at a refused item; Gmail, Outlook, and IMAP advanced their cursors past refused emails, and every poller counted the refusals toward auto-disable. A shared PollAdmissionRefusedError now leaves the idempotency callback, stops the batch, and returns skipped before any cursor update or failure count; items that already ran replay as idempotent no-ops. Source backoff goes back to RSS only, where the rate-limited feed was: the other pollers' fetch helpers do not carry status or Retry-After, so routing their failures through it would back off on a guess. The block-missing 404 also tells Slack not to redeliver.

…an-out failure A poller stops its batch on a deterministic refusal only while nothing in the batch has completed; once an item has run, the refusal is an ordinary item failure, so the poller saves its completed work exactly as before and no completed event can replay after the idempotency window. In a multi-target delivery a missing block's no-retry 404 no longer stands in for a target that failed and needs the sender to retry. A source's Retry-After is counted from its answer rather than the poll's start. The RSS backoff keys are cleared by RSS's own state write, so other pollers' success path is unchanged, and two fields nothing reads are dropped.

A workspace without a billing account fails inside payer resolution and takes the retryable attribution-error path; the branch that tagged BILLING_ACCOUNT_REQUIRED only ran for an attribution with no actor, which system attribution never produces. The branch goes back to its staging form and the deterministic set keeps the usage limit and suspended accounts.

…lling utils mock Two gates that fail without a code (a ban lookup error and a usage lookup error) no longer share one throttle claim, so neither hides the other's row. The polling utils module gets one central mock in @sim/testing, replacing the partial importOriginal mocks and the hand-rolled factory in the table trigger test.

waleedlatif1 · 2026-10-10T02:24:14Z

@greptile

waleedlatif1 · 2026-10-10T02:24:15Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-10T02:24:20Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 37 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Turn on auto-fix | Re-trigger cubic}

…its token with Subscription creation resolves the incoming bot token with the deployer's env, while cleanup resolves with the background env; the active-row matcher now uses the same env as its caller, so a bot referenced through a personal variable still reuses the active secret and is not deleted from under the active deployment. Tests: the idempotency service gets one central mock in @sim/testing, used by every test that mocked it locally, and the new tests import single factory and mock files instead of the @sim/testing barrel.

waleedlatif1 · 2026-10-10T02:40:08Z

@greptile

waleedlatif1 · 2026-10-10T02:40:09Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-10T02:40:15Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 40 files

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Turn on auto-fix | Re-trigger cubic}

…n the central mock

waleedlatif1 · 2026-10-10T02:47:53Z

@greptile

waleedlatif1 · 2026-10-10T02:47:55Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-10T02:48:00Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 40 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Turn on auto-fix | Re-trigger cubic}

cubic-dev-ai Bot reviewed Oct 10, 2026

View reviewed changes

greptile-apps Bot reviewed Oct 10, 2026

View reviewed changes

Comment thread apps/sim/lib/webhooks/processor.ts Outdated

Comment thread apps/sim/lib/webhooks/polling/utils.ts

Comment thread apps/sim/lib/webhooks/polling/google-drive.ts Outdated

Comment thread apps/sim/lib/execution/preprocessing.test.ts

vercel Bot temporarily deployed to Preview October 10, 2026 01:40 Inactive

cubic-dev-ai Bot reviewed Oct 10, 2026

View reviewed changes

Comment thread apps/sim/lib/webhooks/polling/utils.ts Outdated

greptile-apps Bot reviewed Oct 10, 2026

View reviewed changes

Comment thread apps/sim/lib/webhooks/polling/hubspot.ts Outdated

Comment thread apps/sim/lib/webhooks/processor.ts

waleedlatif1 force-pushed the fix/webhook-usage-gate-admission branch from 94db255 to 5b2a564 Compare October 10, 2026 01:58

vercel Bot temporarily deployed to Preview October 10, 2026 01:58 Inactive

cubic-dev-ai Bot reviewed Oct 10, 2026

View reviewed changes

Comment thread apps/sim/lib/webhooks/polling/imap.ts

Comment thread apps/sim/lib/webhooks/polling/google-calendar.ts

greptile-apps Bot reviewed Oct 10, 2026

View reviewed changes

Comment thread apps/sim/lib/execution/preprocessing.ts Outdated

vercel Bot temporarily deployed to Preview October 10, 2026 02:07 Inactive

cubic-dev-ai Bot reviewed Oct 10, 2026

View reviewed changes

Comment thread apps/sim/lib/execution/preprocessing.ts Outdated

greptile-apps Bot reviewed Oct 10, 2026

View reviewed changes

Comment thread apps/sim/lib/webhooks/polling/google-calendar.test.ts Outdated

waleedlatif1 added 7 commits October 9, 2026 19:23

refactor(webhooks): declare the Telegram admission opt-in on its handler

1ec25a8

waleedlatif1 added 8 commits October 9, 2026 19:23

waleedlatif1 force-pushed the fix/webhook-usage-gate-admission branch from 410b107 to 3a141dd Compare October 10, 2026 02:24

vercel Bot temporarily deployed to Preview October 10, 2026 02:24 Inactive

cubic-dev-ai Bot reviewed Oct 10, 2026

View reviewed changes

greptile-apps Bot reviewed Oct 10, 2026

View reviewed changes

Comment thread apps/sim/lib/webhooks/providers/telegram.ts Outdated

Comment thread apps/sim/lib/webhooks/polling/google-calendar.test.ts Outdated

Comment thread apps/sim/lib/webhooks/polling/google-calendar.test.ts Outdated

vercel Bot temporarily deployed to Preview October 10, 2026 02:40 Inactive

cubic-dev-ai Bot reviewed Oct 10, 2026

View reviewed changes

Comment thread packages/testing/src/mocks/idempotency-service.mock.ts

chore(testing): stub IdempotencyService.createWebhookIdempotencyKey i…

6080fb0

…n the central mock

vercel Bot temporarily deployed to Preview October 10, 2026 02:47 Inactive

cubic-dev-ai Bot reviewed Oct 10, 2026

View reviewed changes

waleedlatif1 merged commit b8989cc into staging Oct 10, 2026
48 checks passed

waleedlatif1 deleted the fix/webhook-usage-gate-admission branch October 10, 2026 03:27

waleedlatif1 mentioned this pull request Oct 10, 2026

fix(logs): close the log-once coverage gaps from the cross-PR review #8881

Open

6 tasks done

Conversation

waleedlatif1 commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Diagram

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 10, 2026

Uh oh!

waleedlatif1 commented Oct 10, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 10, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 10, 2026

Uh oh!

waleedlatif1 commented Oct 10, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 10, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 10, 2026

Uh oh!

waleedlatif1 commented Oct 10, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 10, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 10, 2026

Uh oh!

waleedlatif1 commented Oct 10, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 10, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

waleedlatif1 commented Oct 10, 2026 •

edited

Loading

vercel Bot commented Oct 10, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

greptile-apps Bot commented Oct 10, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading