Repository navigation
fix: dev-mailbox oncall 409 redelivery loop behind EventQueueAgeAlarm - #91
Merged
alexeygrigorev merged 3 commits intoNov 20, 2026
Conversation
A redelivered event restores a completed condition/paths step from run history as a whole, so its children never re-entered the steps context under their nested ids. dev-mailbox's verify-delivery then read route.else.post-webhook as missing and failed every redelivery even though the webhook leg had been delivered on the first attempt (runs 91acff66, 9143bacc, 08674c39 on 18-20 Nov). Re-enter the recorded branch (no predicate re-evaluation): completed children restore their own records, unfinished ones run again, like any unfinished top-level step on a redelivery.
…ivery Every dev-mailbox failure on 18-20 Nov had the same shape: Hub's oncall ingest stored the first attempt but the response timed out; each queue redelivery then resent the format-email output recovered from run history (DynamoDB map, key order lost), so Hub answered 409 delivery_id_conflict for the same X-Delivery-Id forever. The run looped through five 1800s redeliveries into the DLQ, and every redelivery tripped the 900s EventQueueAgeAlarm (all four alarms of 19-20 Nov). verify-delivery now treats an ingest 409 as already delivered (the key is the message id's hash, so Hub holds this email). A retry policy (4 attempts, 120s) re-enqueues a failed leg as a fresh message instead of waiting out the visibility timeout, so a transient Hub outage no longer ages the queue past the alarm threshold.
The deploy and the dispatchable inspection now report the event-queue alarms and the client-check alarm the operator topic pages on: current state, verbatim transitions over three days, oldest-message age peaks, dead-letter depth and ClientCheckFailed counts. An alarm email only says a threshold was crossed; these lines prove recovery and non-recurrence. Read-only, within the existing inspection grants (pinned by a test).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the root of the Dapier runtime alerts from 19–20 Nov 2026 (Hub intake 7ca6d265).
Root cause (one loop behind 3 of the 4 alert types)
Every dev-mailbox failure on 18–20 Nov (runs
91acff66,9143bacc,08674c39) had the same shape:verify-deliveryfailed the run.routecondition was restored from history as a whole, soroute.else.post-webhookwas missing fromstepsand read as failed. This is an engine bug.delivery_id_conflictfor the sameX-Delivery-Id, every time.EventQueueAgeAlarmfires when the oldest message reaches 900s. All four alarm datapoints (911/903/905/907s at 19 Nov 00:55 and 18:40, 20 Nov 10:22 and 11:52) are about 15 min after one of these messages was first sent, or re-received for the 4th time (SQS moves the message to the back after 3 receives, which resets its age).Changes
retry: {attempts: 4, backoff_seconds: 120}so a transient Hub outage retries as a fresh message instead of a 1800s redelivery that trips the queue-age alarm.scripts/inspect_alarms.py, run by deploy and by the dispatchable inspection. It prints alarm states and transitions, queue-age peaks, DLQ depth and ClientCheckFailed counts, read-only within the existing grants (pinned by a test).Tests
Targeted locally:
test_branch_recovery.py(new),test_dev_mailbox_workflow.py,test_backup.py,test_grok_quota_fallback.py,test_retry_policy.py, engine/worker/logic/paths/replay suites: all green. Full suite runs in CI.After merge, the dev-mailbox YAML is published live via
dapier workflows save+publish, and the failed runs are replayed.