Repository navigation
fix(mailbox): wake Developer webhook only for principal-ping and user-facing outages - #95
Merged
Merged
Conversation
…-facing outages dev-mailbox woke the Grok Bot Developer webhook for every mail to dev@dtcdev.click; routine alerts were 12% of all Grok Bot usage in 7 days. A new classify-email step now buckets each mail: - wake: [principal-ping] subjects and alerts that explicitly say prod is down for users (narrow explicit markers in subject or body) - routine: everything else - infra/CMP workers/release follower/poller, deadman FAIL/RECOVERED, telegram relay heartbeat, disk WARN, GitHub Actions failures, digests, dapier failure notices - rides the direct oncall ingest alone (route: none skips both Hub legs) The oncall ingest stays the primary delivery for every event. When it fails (and is not the 409 already-delivered duplicate), decide-grok-route falls back to waking the Developer webhook regardless of classification, and verify-delivery still fails the run, so Dapier retries the event and the failure notice reaches dev@ - never silently dropped. Wake events keep the existing quota divert to the Dev sticky Hub session; tofu/orin mailboxes are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The dev-mailbox workflow woke the Grok Bot Developer webhook routine ("Dapier/dev mailbox wake") for every mail to dev@dtcdev.click; the routine just ran
~/bin/oncall-sendon Hetzner. That was 12% of all Grok Bot usage over the last 7 days.Change
workflows/dev-mailbox.yamlnow classifies each mail (newclassify-emailcode step) into two buckets:[principal-ping](Hub pings), and alerts that explicitly say prod is down for users — narrow explicit markers in subject/body ("prod is down", "production is down", "site is down", "down for users",[user-facing],[outage], …).https://hub.dtcdev.click/webhooks/oncall→ Hub2 SQLite queue →~/bin/oncall-send) only.decide-grok-routeanswersroute: none, so neither the webhook nor the sticky Hub session runs.The direct oncall ingest stays the primary delivery for every event (same channel
oncall-sendalready delivers through). If the ingest fails (and is not the 409 already-delivered duplicate),decide-grok-routefalls back toroute: webhookregardless of classification andverify-deliverystill fails the run — so Dapier retries the event, and once retries are exhausted the failure notice reaches dev@ (operator inbox). Never silently dropped.Wake events keep the existing GrokBot-quota divert to the Dev sticky Hub session. Tofu/orin mailboxes are unchanged (their per-mail webhook wakes are their purpose).
Tests
noneroute needs only the ingest, fallback does not mask the ingest failuremake test: 5009 passed, 48 skipped.