Skip to content

fix(monitor): normalize per-occurrence Azure tokens out of failure signatures (closes #3592) - #3594

Merged
clabonte merged 1 commit into
mainfrom
fix/issue-3592-azure-message-normalization
Nov 17, 2026
Merged

clabonte merged 1 commit into
mainfrom
fix/issue-3592-azure-message-normalization

Conversation

@clabonte

Copy link
Copy Markdown
Contributor

Root cause

Both Azure providers build a failure signature from the exception message with GUIDs, timestamps and numbers replaced, but several per-occurrence tokens survived, so every occurrence of one failure got its own fingerprint and issue (339 issues in one Backend run):

  • the GUID pattern was \b-anchored and _ is a word character, so the Service Bus TrackingId:<guid>_B4 GUID never matched; its _B<n> partition suffix varied too;
  • SystemTracker carries subscription + partition, so MessageLockLost split per subscription;
  • Azure Storage errors echo an HTTP Date: header;
  • Functions host wrapper exceptions (Grpc.Core.RpcException, Rpc.RpcException) embed the inner exception's whole stack, so each call path was its own signature.

Fix

Normalization moves to a shared private module, specfuse/monitor/providers/_azure_normalize.py (same pattern as _azure_auth.py), so the two providers can't drift. It strips embedded stack frames, replaces TrackingId/SystemTracker/Reference/LockToken values, matches GUIDs on hex boundaries, and replaces HTTP dates, on top of the existing timestamp/number rules.

Replaying the 339 real signatures through the new rules: 338 distinct → 95; the lock-lost family (224 issues) → 3.

Side effect: existing findings whose signature changes are filed once more under the new signature.

The per-run creation cap that would have contained the flood is filed separately as #3593 (feature-sized).

Tests

  • tests/test_azure_message_normalization.py — for both error-logs and dlq adapters: two occurrences of lock-lost, wrapper-with-stack, and blob-lease failures each share a fingerprint; distinct failures (including prose with "at the gateway") stay apart; a GUID followed by _B4, ), . or nothing is normalized. 8 subtests fail on unchanged code.

Verification

  • scripts/smoke-test.sh: full suite OK, smoke test OK.

Closes #3592.

🤖 Generated with Claude Code

…gnatures

Closes #3592. Both Azure providers built a failure signature from the
exception message with GUIDs, timestamps and numbers replaced, but several
per-occurrence tokens survived, so every occurrence of one failure got its
own fingerprint and its own issue (339 issues in one Backend run):

- the GUID pattern was `\b`-anchored, and `_` is a word character, so the
  Service Bus `TrackingId:<guid>_B4` GUID never matched; its `_B<n>`
  partition suffix varied too;
- `SystemTracker` carries the subscription and partition, so MessageLockLost
  split per subscription;
- Azure Storage errors echo an HTTP `Date:` header (`Fri, 14 Nov 2026 ...`);
- the Functions host wrapper exceptions (`Grpc.Core.RpcException`,
  `Rpc.RpcException`) embed the inner exception's whole stack, so each call
  path was its own signature.

The normalization moves to a shared private module,
`specfuse/monitor/providers/_azure_normalize.py` (the `_azure_auth.py`
pattern), so the two providers cannot drift again. It strips embedded stack
frames, replaces TrackingId/SystemTracker/Reference/LockToken values, matches
GUIDs on hex boundaries, and replaces HTTP dates, on top of the existing
timestamp and number rules. Replaying the 339 real signatures through the new
rules leaves 95 distinct; the lock-lost family (224 issues) becomes 3.

Existing findings whose signature changes are filed once more under the new
signature.

Tests:
- tests/test_azure_message_normalization.py — for both the error-logs and
  dlq adapters: two occurrences of a lock-lost, a wrapper-with-stack, and a
  blob-lease failure each share a fingerprint; distinct failures (including
  prose containing "at the gateway") stay apart; a GUID followed by `_B4`,
  `)`, `.` or nothing is normalized.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@clabonte
clabonte merged commit f032266 into main Nov 17, 2026
7 checks passed
@clabonte
clabonte deleted the fix/issue-3592-azure-message-normalization branch November 17, 2026 02:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

error-logs: Service Bus TrackingId survives fingerprint normalization; one issue per message (339 issues in one run)

1 participant