Repository navigation
Description
Summary
error-logs fingerprints a Service Bus exception from its message, and the message normalization leaves the per-message Service Bus TrackingId partly intact. Every occurrence of the same failure therefore gets a new fingerprint and its own issue. One specfuse monitor run (specfuse-loop 0.33.0) against RestoManagerApp/Backend opened 339 issues in an hour (#1420 to #1758), and 336 of them were for one component.
Observed
Same failure class and same code location, distinct fingerprints. These are the Failure signature lines from three issues created in the same run:
Azure.Messaging.ServiceBus.ServiceBusException:The lock supplied is invalid. ... Reference:<guid>, TrackingId:4311145c-fcf8-<num>-9bdb-95c15255c504_B4, SystemTracker:sbns-main-prd:topic:sbt-application~<num>|on-document-extracted-...
Azure.Messaging.ServiceBus.ServiceBusException:The lock supplied is invalid. ... Reference:<guid>, TrackingId:<num>-8e81-474e-acd6-337165f0c89b_B8, SystemTracker:sbns-main-prd:topic:sbt-application~<num>|on-trial-milestone-record-...
Azure.Messaging.ServiceBus.ServiceBusException:The lock supplied is invalid. ... Reference:<guid>, TrackingId:68128c21-0bb5-44ea-<num>-77822ab6f35b_B5, SystemTracker:sbns-main-prd:topic:sbt-application~<num>|on-trial-milestone-rec...
Reference:<guid>is normalized. TheTrackingIdGUID is not, because it is followed by_B4/_B8._is a word character, so a\b-anchored GUID pattern does not match. Only the all-digit groups get replaced with<num>, which leaves the rest unique per message.SystemTrackercarries the subscription name. That one is arguably a legitimate dimension, but it is truncated at a fixed length, so the fingerprint also depends on where the cut lands.- The same exception wrapped by the Functions host (
Grpc.Core.RpcException/Rpc.RpcException, whoseDetailembeds the full Service Bus message plus stack) inherits the same uniqueness.
Breakdown of the 336 backend-worker issues from that run:
| Title (class in method) | Issues |
|---|---|
ServiceBusException in AmqpReceiver+<RenewMessageLockInternalAsync> |
100 |
Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException |
94 |
Grpc.Core.RpcException in ServiceBusFunction+<ProcessMessageAsync> |
43 |
Grpc.Core.RpcException in ServiceBusFunction+<CarryOutAsync> |
41 |
Each of these issues reports Occurrences: 1. This predates 0.33.0: issues #1360 to #1389 (2026-11-07) show the same pattern.
Secondary effects in the same run
- Filing hit GitHub's content-creation limits. Two
gh issue createcalls exited 1 (the hourly core quota was barely used, so this is the secondary rate limit). The filing errors then causeenvironment production: watermark not advanced, so the next cycle re-scans the same window and keeps filing. - The window reached back to 2026-10-18. First-seen times go back four weeks, so this run scanned well past one hourly window. I have not established why the window was that long.
Expected
- Message normalization replaces a GUID regardless of the character that follows it (
_B4,),.), and treats known per-message Azure tokens (TrackingId,Reference,Timestamp,LockToken) as volatile. - One fingerprint per failure class and location for this case: four issues, not 336.
- A run that would create more than N new issues in one component stops at a cap, or rolls them into one summary issue, instead of filing them all. The existing
throttledcounter reported 0 here.
Repro
error-logs on any Azure Functions Service Bus consumer that sees MessageLockLost. The TrackingId:<guid>_B<n> shape is standard in Service Bus error messages.
Environment
- specfuse-loop 0.33.0, specfuse 0.17.0, run locally (
specfuse monitor run) - Consumer: RestoManagerApp/Backend,
.specfuse/monitoring.yml,backend-workererror-logs check
Activity