Skip to content

monitor: cap new issues per component per run, and roll the overflow into one summary issue #3593

Open

Description

@clabonte

Problem

specfuse monitor run files one issue per new fingerprint, with no upper bound. When something splits one failure into many fingerprints, the run files all of them. #3592 is the case that showed it: 339 issues in one hour against one repo, 336 for one component.

#3592 fixes the cause in that case (per-message Azure tokens left in the signature). The next normalization gap, a new provider, or a real incident across many endpoints will cause the same flood. Nothing in the run limits it today.

Why the existing counters don't help

Proposed behavior

  1. A per-component cap on issues created in one run, with a default (for example 10) and an override in monitoring.yml.
  2. Findings past the cap are not filed one by one. They roll into one summary issue for that component and run, listing each fingerprint with its title, occurrence count and first-seen time. The summary itself should be deduplicated, so the next run updates it instead of opening another.
  3. The run summary reports the overflow (created 10, rolled up 326), and the run exits non-zero, or marks the component as degraded, so an operator sees it.
  4. The watermark: rolled-up findings count as recorded, so the watermark advances and the next run doesn't refile them. A filing error still holds the watermark, as it does today.

Open questions

  • Should the cap be per component, per environment, or per run? Per component keeps one noisy component from hiding findings in the others.
  • When a rolled-up fingerprint shows up again in a later run that is under the cap, does it get its own issue then, or stay in the summary until someone acts on it?
  • Should hitting the cap also back off, i.e. stop before GitHub rate-limits the run?

Feature-sized: new config surface, a new issue shape, and a change to watermark semantics.

Related: #3592, #3579.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions