Skip to content

fix(mailbox): wake Developer webhook only for principal-ping and user-facing outages - #95

Merged
alexeygrigorev merged 1 commit into
mainfrom
fix/dev-mailbox-wake-classifier
Nov 21, 2026
Merged

alexeygrigorev merged 1 commit into
mainfrom
fix/dev-mailbox-wake-classifier

Conversation

@alexeygrigorev

Copy link
Copy Markdown
Member

Problem

The dev-mailbox workflow woke the Grok Bot Developer webhook routine ("Dapier/dev mailbox wake") for every mail to dev@dtcdev.click; the routine just ran ~/bin/oncall-send on Hetzner. That was 12% of all Grok Bot usage over the last 7 days.

Change

workflows/dev-mailbox.yaml now classifies each mail (new classify-email code step) into two buckets:

  • Wake the Developer webhook: subjects containing [principal-ping] (Hub pings), and alerts that explicitly say prod is down for users — narrow explicit markers in subject/body ("prod is down", "production is down", "site is down", "down for users", [user-facing], [outage], …).
  • Routine (default): infra/CMP workers/release follower/poller, deadman FAIL/RECOVERED, telegram relay heartbeat, disk WARN, GitHub Actions failures, digests, dapier failure notices, ordinary mail → the direct oncall ingest (https://hub.dtcdev.click/webhooks/oncall → Hub2 SQLite queue → ~/bin/oncall-send) only. decide-grok-route answers route: none, so neither the webhook nor the sticky Hub session runs.

The direct oncall ingest stays the primary delivery for every event (same channel oncall-send already delivers through). If the ingest fails (and is not the 409 already-delivered duplicate), decide-grok-route falls back to route: webhook regardless of classification and verify-delivery still fails the run — so Dapier retries the event, and once retries are exhausted the failure notice reaches dev@ (operator inbox). Never silently dropped.

Wake events keep the existing GrokBot-quota divert to the Dev sticky Hub session. Tofu/orin mailboxes are unchanged (their per-mail webhook wakes are their purpose).

Tests

  • classifier buckets: principal-ping wake, user-facing-outage wake (6 phrasings), 13 routine subjects (all real alert formats from live dev@ traffic) stay quiet
  • routing: routine → none; ingest failure → webhook fallback; 409 duplicate → not a failure; fallback bypasses the quota divert; 401 rotation only on wake routes
  • verifier: new step ids, none route needs only the ingest, fallback does not mask the ingest failure
  • full-chain tests through the real engine: routine alert → oncall only; ingest 503 → webhook fallback + loud failure; principal-ping → quota divert / webhook as before

make test: 5009 passed, 48 skipped.

…-facing outages

dev-mailbox woke the Grok Bot Developer webhook for every mail to
dev@dtcdev.click; routine alerts were 12% of all Grok Bot usage in 7
days. A new classify-email step now buckets each mail:

- wake: [principal-ping] subjects and alerts that explicitly say prod
  is down for users (narrow explicit markers in subject or body)
- routine: everything else - infra/CMP workers/release follower/poller,
  deadman FAIL/RECOVERED, telegram relay heartbeat, disk WARN, GitHub
  Actions failures, digests, dapier failure notices - rides the direct
  oncall ingest alone (route: none skips both Hub legs)

The oncall ingest stays the primary delivery for every event. When it
fails (and is not the 409 already-delivered duplicate), decide-grok-route
falls back to waking the Developer webhook regardless of classification,
and verify-delivery still fails the run, so Dapier retries the event and
the failure notice reaches dev@ - never silently dropped. Wake events
keep the existing quota divert to the Dev sticky Hub session; tofu/orin
mailboxes are unchanged.
@alexeygrigorev
alexeygrigorev merged commit b03ed80 into main Nov 21, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant