Skip to content

ci(dlq): dispatchable event DLQ drain (deletes only named events) - #92

Merged
alexeygrigorev merged 1 commit into
mainfrom
fix/dlq-drain-7ca6d265
Nov 20, 2026
Merged

alexeygrigorev merged 1 commit into
mainfrom
fix/dlq-drain-7ca6d265

Conversation

@alexeygrigorev

Copy link
Copy Markdown
Member

Follow-up to #91 (Hub intake 7ca6d265): drains the event DLQ that the dev-mailbox 409 loop filled.

  • EventDeadLetterAlarm has been in ALARM since 13 Nov, so it sends no new pages. Live inspection from deploy run 44740195169: DLQ depth 4 on 20 Nov 12:06.
  • The 3 dev-mailbox events were delivered, and their runs have been replayed green. The DLQ copies are stale.
  • You can't redrive from CI: the EventQueue's DenyUntrustedPublishers boundary admits only Dapier roles and the inbound SNS topic. That boundary is intentionally left untouched.

Adds:

  • scripts/drain_dlq.py: lists every message (event id = the run id's suffix, plus connector, event, subject, receives). --delete <ids> removes only the named events.
  • .github/workflows/dlq-drain.yml: workflow_dispatch with a delete input (empty = list only).
  • infra/dlq-drain-access.yaml: its own stack, deployed by deploy.yml. Grants sqs:ReceiveMessage/DeleteMessage/GetQueueAttributes on dapier-DeadLetterQueue-* only.
  • tests/test_drain_dlq.py: listing, targeted delete, grants scope.

Each dev-mailbox 409 loop of 18-20 Nov ended with the event in the DLQ,
and EventDeadLetterAlarm has sat in ALARM since 13 Nov (4 messages now),
so it pages nobody about new dead letters. The events themselves were
delivered and their runs replayed green; the DLQ copies are stale.

The DLQ cannot be redriven to the event queue from CI (the queue policy's
DenyUntrustedPublishers boundary), so the drain lists every message with
its event id - the suffix of the run id - and deletes only the event ids
an operator names after replaying them. Grants live in their own stack,
scoped to the dapier DLQ.
@alexeygrigorev
alexeygrigorev merged commit 25e3b42 into main Nov 20, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant