Make webhooks the primary ingestion path and retire IMAP safely #299

Context

Requested by @ecarreras after reviewing whether webhook ingestion is ready to become the global default.

Webhook event coverage is already broad, but the fallback IMAP path can still produce a different identity, action, or authorization decision for the same GitHub event. In the audited window for the original two canary repositories:

• 82 emails were correctly deduplicated against a webhook-created job;
• 9 distinct work_allowed jobs were created only by IMAP even though webhook had ignored the event or could not correlate it;
• on RuntimeError: Caught handled exception, but response already started. #292, assignment produced webhook job #7860 and IMAP job #7861; closing the issue later produced IMAP job #7902 as open_issue + work_allowed, while the issues:closed webhook was correctly ignored;
• the same close/reopen misclassification was observed on OperationalError: database is locked #285, OperationalError: database is locked #287, gisce/erp#29463, and gisce/erp#29476;
• the merge notification for PR refactor: add explicit dashboard read model #284 was also ingested later by IMAP with the wrong action.

This is therefore not just a transport switch. We need one transport-neutral event and authorization contract, evidence that the fallback behaves identically, and a reversible rollout.

Goal

Make GitHub webhooks the authoritative ingestion path for all enabled repositories, keep IMAP temporarily as a deduplicated safety net, and retire the IMAP reader only after two clean observation windows.

The transition must not introduce duplicate jobs, dropped actionable events, reopened work from close/merge notifications, or broader authorization.

Non-goals

• Fix the independent OpenClaw session-pruning race or every source of blocked jobs.
• Replace the queue or persistence architecture.
• Replay historical outside_canary deliveries that may already have been processed by IMAP.

Invariants

1. The same GitHub event must produce the same canonical identity, action, and work intent regardless of transport or arrival order.
2. IMAP may cover a missing webhook delivery, but it must never upgrade authorization or reopen completed work.
3. A close or merge notification must never become open_issue + work_allowed.
4. Ambiguous or failed classification must fail conservatively to review_only/archive, never retain work_allowed by accident.
5. Rollout and rollback must be observable, auditable, and possible without losing queued work.

Current state

• Support bot assignment webhook events and cross-source coalescing for the covered assignment format — fix: support bot assignment webhook events #264.
• Apply deterministic actionability guards to comment/review webhook paths and handle merge events conservatively — fix: enforce actionable GitHub event dispatch #272.
• Make webhook intent-classifier failures conservative — fix: make webhook intent fallback conservative #258.
• Expand webhookCanaryRepos from two to five repositories:
  • gisce/github-agent-bridge
  • gisce/erp
  • gisce/keepupd
  • gisce/oorq
  • gisce/poweremail-modules
• Complete a full clean observation window for the expanded canary.

Delivery plan

1. Define the cross-source event contract and regression matrix

• Define a normalized event envelope shared by webhook and IMAP: repository, thread, subject type, event type/action, actor, timestamp, canonical event identity, and provenance.
• Add real/redacted fixtures for assigned, closed, merged, review request, issue comment, review comment, and PR review events from both transports.
• Add parity tests proving that both sources produce the same action, work intent, work key, and canonical identity.
• Cover both arrival orders: webhook-first and email-first.

Suggested first slice: tests and contract only, so later implementation PRs have a hard compatibility target.

2. Fix IMAP event resolution and canonical identity

• Resolve Message-ID values such as issue_event/<id> to the actual GitHub event instead of inferring intent from subject/footer text.
• Normalize assigned, closed, merged, review request, and related events through the same mapping used by webhook ingestion.
• Generate a transport-independent canonical event identity so the same event coalesces into one job.
• Preserve provenance from both receipts for audit/debugging without creating a second job.
• Prove that close/merge notifications cannot reopen work even when IMAP arrives first or GitHub enrichment is temporarily unavailable.

3. Make authorization transport-neutral

• Move the actionability decision behind one shared guard used by webhook and IMAP before any LLM classification.
• Preserve the agreed positive signals: explicit bot mention, bot assignment, or a follow-up/review on a PR authored by the bot.
• Decide and document whether authorship of an issue (not a PR) authorizes follow-up work; default to no authorization until explicitly agreed.
• Ensure reviewer/subscriber status, notification recipient, X-GitHub-Reason, and email footer text do not authorize work by themselves.
• Ensure the LLM cannot upgrade an event that failed the deterministic guard.
• Add a regression for the divergent #260 comment: webhook and IMAP must reach the same decision.

4. Repair the coverage gate and expose mismatches

• Include email:* and other currently uncorrelated keys in coverage calculations instead of excluding the problematic population.
• Report actionable IMAP-only events, webhook-only actionable events, split canonical identities, and action/intent mismatches by repository and event type.
• Keep enough decision trace to explain why each receipt was correlated, ignored, downgraded, or dispatched.
• Alert on missing webhook deliveries and on any unexplained actionable IMAP-only event.
• Make the promotion gate fail when any unexplained cross-source mismatch exists.

5. Observe and progressively widen the canary

• Add keepupd, oorq, and poweremail-modules to the original github-agent-bridge + erp canary set.
• Observe at least seven complete days with IMAP still enabled as fallback.
• Investigate and classify every actionable IMAP-only event and every split job during that window.
• Require zero duplicate jobs, zero action/intent mismatches, and zero unexplained actionable IMAP-only events before global promotion.
• Add further repositories only in small batches after the previous batch is clean; do not use primary as the mechanism to bypass the canary allowlist.

6. Close the organization-hook operational gaps

• Migrate from the shared legacy webhook secret to GITHUB_AGENT_BRIDGE_WEBHOOK_SECRETS_BY_OWNER.
• Give the operational identity the minimum permission needed to inspect the organization hook and its deliveries; currently it lacks admin:org_hook.
• Run and record an audited ping/delivery check.
• Verify subscribed events, TLS validation, GitHub retries, payload limits, secret rotation, and delivery retention.
• Alert when expected deliveries stop arriving or repeated deliveries fail.
• Document the operator runbook and rollback procedure.

7. Promote webhooks and retire IMAP

• Set WEBHOOK_MODE=primary and WEBHOOK_PRIMARY_ACK=true only after the promotion gate passes.
• Keep IMAP active temporarily as a deduplicated fallback and observe a second stable window.
• Verify that IMAP fallback does not create actionable work absent a confirmed webhook delivery gap.
• Disable the IMAP reader timer.
• Monitor after cutover and keep a tested rollback path for the agreed period.
• Remove obsolete IMAP-only code/configuration in a later cleanup PR, not in the cutover itself.

Suggested PR-sized slices

1. Parity contract and fixtures — normalized envelope plus cross-source regression matrix.
2. IMAP resolver and canonical identity — issue_event/<id> resolution, action mapping, and deduplication.
3. Shared authorization guard — one deterministic policy for both transports.
4. Coverage gate and decision trace — include email:*, expose mismatches, and alert.
5. Operational hardening — per-owner secrets, hook inspection/ping, monitoring, and runbook.
6. Primary rollout — configuration change, second observation window, and IMAP timer shutdown.

Slices 2 and 3 depend on the contract in slice 1. Slice 4 must land before promotion. Operational hardening can proceed in parallel with slices 1–4. The final rollout is deliberately separate and should not be bundled with code changes.

Acceptance criteria

• Replaying every event in the parity matrix through webhook and IMAP yields the same canonical identity, action, work intent, and work key.
• No close or merge fixture can produce open_issue + work_allowed.
• The coverage gate includes uncorrelated email receipts and reports zero unexplained actionable IMAP-only events.
• The expanded canary completes seven consecutive clean days with zero split jobs and zero authorization mismatches.
• Organization-hook delivery inspection, ping, secret rotation, retries, TLS, and missing-delivery alerts are verified and documented.
• primary mode completes a second stable observation window with IMAP still deduplicated.
• The IMAP reader is disabled without losing or duplicating actionable work, and rollback has been tested/documented.