Repository navigation
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Problem
specfuse monitor run files one issue per new fingerprint, with no upper bound. When something splits one failure into many fingerprints, the run files all of them. #3592 is the case that showed it: 339 issues in one hour against one repo, 336 for one component.
#3592 fixes the cause in that case (per-message Azure tokens left in the signature). The next normalization gap, a new provider, or a real incident across many endpoints will cause the same flood. Nothing in the run limits it today.
Why the existing counters don't help
throttledin the run summary counts findings whose issue already exists and that got no GitHub write this run (the comment throttle). It says nothing about how many new issues the run creates, so it reported 0 in the error-logs: Service Bus TrackingId survives fingerprint normalization; one issue per message (339 issues in one run) #3592 run.- Filing more than about 100 issues in an hour trips GitHub's secondary rate limit on content creation.
gh issue createthen exits 1, and the filing error leaves the environment's watermark where it was, so the next run re-scans the same window and tries to file the same issues again. In the error-logs: Service Bus TrackingId survives fingerprint normalization; one issue per message (339 issues in one run) #3592 run, earlier errored runs had already stretched the window back four weeks, which made the flood larger.
Proposed behavior
- A per-component cap on issues created in one run, with a default (for example 10) and an override in
monitoring.yml. - Findings past the cap are not filed one by one. They roll into one summary issue for that component and run, listing each fingerprint with its title, occurrence count and first-seen time. The summary itself should be deduplicated, so the next run updates it instead of opening another.
- The run summary reports the overflow (
created 10, rolled up 326), and the run exits non-zero, or marks the component as degraded, so an operator sees it. - The watermark: rolled-up findings count as recorded, so the watermark advances and the next run doesn't refile them. A filing error still holds the watermark, as it does today.
Open questions
- Should the cap be per component, per environment, or per run? Per component keeps one noisy component from hiding findings in the others.
- When a rolled-up fingerprint shows up again in a later run that is under the cap, does it get its own issue then, or stay in the summary until someone acts on it?
- Should hitting the cap also back off, i.e. stop before GitHub rate-limits the run?
Feature-sized: new config surface, a new issue shape, and a change to watermark semantics.
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Activity