Repository navigation
fix(monitor): normalize per-occurrence Azure tokens out of failure signatures (closes #3592) - #3594
Merged
Merged
Conversation
…gnatures Closes #3592. Both Azure providers built a failure signature from the exception message with GUIDs, timestamps and numbers replaced, but several per-occurrence tokens survived, so every occurrence of one failure got its own fingerprint and its own issue (339 issues in one Backend run): - the GUID pattern was `\b`-anchored, and `_` is a word character, so the Service Bus `TrackingId:<guid>_B4` GUID never matched; its `_B<n>` partition suffix varied too; - `SystemTracker` carries the subscription and partition, so MessageLockLost split per subscription; - Azure Storage errors echo an HTTP `Date:` header (`Fri, 14 Nov 2026 ...`); - the Functions host wrapper exceptions (`Grpc.Core.RpcException`, `Rpc.RpcException`) embed the inner exception's whole stack, so each call path was its own signature. The normalization moves to a shared private module, `specfuse/monitor/providers/_azure_normalize.py` (the `_azure_auth.py` pattern), so the two providers cannot drift again. It strips embedded stack frames, replaces TrackingId/SystemTracker/Reference/LockToken values, matches GUIDs on hex boundaries, and replaces HTTP dates, on top of the existing timestamp and number rules. Replaying the 339 real signatures through the new rules leaves 95 distinct; the lock-lost family (224 issues) becomes 3. Existing findings whose signature changes are filed once more under the new signature. Tests: - tests/test_azure_message_normalization.py — for both the error-logs and dlq adapters: two occurrences of a lock-lost, a wrapper-with-stack, and a blob-lease failure each share a fingerprint; distinct failures (including prose containing "at the gateway") stay apart; a GUID followed by `_B4`, `)`, `.` or nothing is normalized. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
Both Azure providers build a failure signature from the exception message with GUIDs, timestamps and numbers replaced, but several per-occurrence tokens survived, so every occurrence of one failure got its own fingerprint and issue (339 issues in one Backend run):
\b-anchored and_is a word character, so the Service BusTrackingId:<guid>_B4GUID never matched; its_B<n>partition suffix varied too;SystemTrackercarries subscription + partition, so MessageLockLost split per subscription;Date:header;Grpc.Core.RpcException,Rpc.RpcException) embed the inner exception's whole stack, so each call path was its own signature.Fix
Normalization moves to a shared private module,
specfuse/monitor/providers/_azure_normalize.py(same pattern as_azure_auth.py), so the two providers can't drift. It strips embedded stack frames, replacesTrackingId/SystemTracker/Reference/LockTokenvalues, matches GUIDs on hex boundaries, and replaces HTTP dates, on top of the existing timestamp/number rules.Replaying the 339 real signatures through the new rules: 338 distinct → 95; the lock-lost family (224 issues) → 3.
Side effect: existing findings whose signature changes are filed once more under the new signature.
The per-run creation cap that would have contained the flood is filed separately as #3593 (feature-sized).
Tests
tests/test_azure_message_normalization.py— for both error-logs and dlq adapters: two occurrences of lock-lost, wrapper-with-stack, and blob-lease failures each share a fingerprint; distinct failures (including prose with "at the gateway") stay apart; a GUID followed by_B4,),.or nothing is normalized. 8 subtests fail on unchanged code.Verification
scripts/smoke-test.sh: full suite OK, smoke test OK.Closes #3592.
🤖 Generated with Claude Code