Skip to content

fix(reporting): issue-event retention prune re-escalates still-open occurrences every 7 days #1885

Open

Description

@bokelley

Summary

The issue-event retention prune deletes a delivered escalated event while its occurrence is still open or acknowledged. The escalation sweep decides whether an occurrence has already escalated by checking for that event, so its next run records escalated again: same event_id, delivered as a new event. This repeats every retention period (7 days by default) for as long as the occurrence stays open.

Found while reviewing the TypeScript port (adcontextprotocol/adcp-client#3654). The TypeScript outbox doubles as its occurrence ledger, so there the same prune lost occurrences outright. Python keeps occurrence state in reporting_issue_lifecycle, so it doesn't lose them, but it does re-escalate them.

Where (next @ 74bbe6f, the #1875 merge)

  • packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/issue_events.py
    • sweep_escalations treats an occurrence as escalated only while the event row exists: NOT EXISTS (SELECT 1 FROM reporting_issue_events e WHERE e.issue_id = l.issue_id AND e.kind = 'escalated'), then ON CONFLICT (issue_id, kind) DO NOTHING.
    • prune deletes every delivered event older than older_than, whatever state its occurrence is in: WHERE delivered_at IS NOT NULL AND delivered_at < %s.
  • src/adcp/reporting/ledger/store.py (InMemoryReportingLedgerStore) does the same: sweep_issue_escalations builds escalated = {e.issue_id ... if e.kind == "escalated"}, and prune_issue_events deletes by delivered_at alone.
  • ReportingIssueEventDispatcher runs prune_issue_events(older_than=timedelta(days=7)) on every turn.

Reproduction

  1. A mismatch opens, escalates at opened_at + consumer_mismatch_escalation, and escalated is delivered. The occurrence stays open (or acknowledged): the buyer never supersedes the statement.
  2. Seven days after that delivery, the dispatcher's prune deletes the escalated row. It also deletes the opened row, which is harmless here because the occurrence lives in reporting_issue_lifecycle.
  3. The next worker turn's sweep finds the occurrence open, past the boundary, with no escalated row. It inserts escalated again with the same event_id (issue_event_id(issue_id, "escalated")), and the dispatcher delivers it.
  4. Steps 2 and 3 repeat every 7 days.

An observer that deduplicates on event_id only drops the repeat if it keeps its dedupe set for longer than the retention period. Otherwise the seller gets paged again for an escalation it already handled.

Suggested fix

Either:

  • prune only events of occurrences that are retired (resolved or waived) and whose every event was delivered, deleting the occurrence's events together; or
  • record "escalated" on the lifecycle row (for example an escalated_at column) and have the sweep check that instead of the event row.

The TypeScript side took the first option in adcontextprotocol/adcp-client#3654: prune whole resolved, fully delivered occurrences, and never an open one.

Add a test: an occurrence that escalates, stays open past the retention, and is swept again records no second escalated.

Activity

  1. bokelley commented on Nov 9, 2026

    @bokelley
    ContributorAuthor

    Related: #1886. Period retirement never resolves a retired period's open occurrences, so with this prune behaviour they re-escalate every retention period forever, even after their period is gone. Fixing one without the other leaves the loop open for retired periods.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions