Repository navigation
Alert in Discord when the Actions collector goes silent in Loki - #6567
Merged
Merged
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 72bdf573f5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Contributor
Add a staging alert rule that fires when Loki receives no lines from the GitHub Actions queue collector for ten minutes. It posts to the alarms-staging-internal Discord channel through a new contact point. apply-alerting.sh now also pushes provisioning/alerting-hosted/<env>/: contact points first, then rule groups. The local stack does not mount that directory. The contact point gets its webhook URL from SSM, through the staging apply workflow, so the URL is not in this repository. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QZtQde6na3jLiKVjAavLPz
apply-alerting.sh found an unset ${VAR} only when it reached the group
that used it. A missing DISCORD_ALARMS_WEBHOOK therefore failed after
apply.sh had already pushed dashboards and data sources, and after any
earlier alert files.
apply-alerting.sh now reads every ${VAR} in the files it will push for
the env, and names all unset ones before it pushes anything. With
--preflight it only makes that check. apply.sh runs the preflight with
its other prerequisite checks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QZtQde6na3jLiKVjAavLPz
backspace
force-pushed
the
cs-13575-actions-collector-loki-silence-alert
branch
from
October 8, 2026 08:25
d4cc8ca to
dcd2a42
Compare
jurgenwerk
approved these changes
Oct 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
Merge after cardstack/infra#753 is applied, including its SSM parameter and grant. This PR's staging apply workflow reads
/staging/boxel-grafana/discord_alarms_webhook, and fails with AccessDenied before that.Background and Goal
The GitHub Actions Queue dashboard was empty from 2026-10-04 until a redeploy on 2026-10-06: the collector's log router had stopped sending to Loki, and nothing reported it. This PR adds a staging alert that fires when Loki receives no collector lines for ten minutes, and posts it to the
alarms-staging-internalDiscord channel. cardstack/infra#753 has the log-router fix.Where to start
provisioning/alerting-hosted/staging/rules/actions-collector-group.json: the rule. No lines gives "no data", not zero, sonoDataStateisAlerting. It routes to Discord throughnotification_settings.receiver; the default notification policy does not change.scripts/apply-alerting.sh: now pushesprovisioning/alerting-hosted/<env>/too, contact points first (created or replaced by uid), then rule groups.Key decisions and non-obvious mechanics
alerting-hosted/<env>/is for what exists in one hosted environment only. The collector runs in staging only, so this rule would always fire in production. docker-compose does not mount the directory, so local Grafana never sees a rule whose contact point it lacks.${DISCORD_ALARMS_WEBHOOK}, which the staging workflow reads from SSM. On an error, the script prints only the server's message, not the request body.DISCORD_ALARMS_WEBHOOKfails the apply.🤖 Generated with Claude Code
https://claude.ai/code/session_01QZtQde6na3jLiKVjAavLPz