Skip to content

error-logs: Service Bus TrackingId survives fingerprint normalization; one issue per message (339 issues in one run) #3592

Closed

Description

@clabonte

Summary

error-logs fingerprints a Service Bus exception from its message, and the message normalization leaves the per-message Service Bus TrackingId partly intact. Every occurrence of the same failure therefore gets a new fingerprint and its own issue. One specfuse monitor run (specfuse-loop 0.33.0) against RestoManagerApp/Backend opened 339 issues in an hour (#1420 to #1758), and 336 of them were for one component.

Observed

Same failure class and same code location, distinct fingerprints. These are the Failure signature lines from three issues created in the same run:

Azure.Messaging.ServiceBus.ServiceBusException:The lock supplied is invalid. ... Reference:<guid>, TrackingId:4311145c-fcf8-<num>-9bdb-95c15255c504_B4, SystemTracker:sbns-main-prd:topic:sbt-application~<num>|on-document-extracted-...
Azure.Messaging.ServiceBus.ServiceBusException:The lock supplied is invalid. ... Reference:<guid>, TrackingId:<num>-8e81-474e-acd6-337165f0c89b_B8, SystemTracker:sbns-main-prd:topic:sbt-application~<num>|on-trial-milestone-record-...
Azure.Messaging.ServiceBus.ServiceBusException:The lock supplied is invalid. ... Reference:<guid>, TrackingId:68128c21-0bb5-44ea-<num>-77822ab6f35b_B5, SystemTracker:sbns-main-prd:topic:sbt-application~<num>|on-trial-milestone-rec...
  • Reference:<guid> is normalized. The TrackingId GUID is not, because it is followed by _B4/_B8. _ is a word character, so a \b-anchored GUID pattern does not match. Only the all-digit groups get replaced with <num>, which leaves the rest unique per message.
  • SystemTracker carries the subscription name. That one is arguably a legitimate dimension, but it is truncated at a fixed length, so the fingerprint also depends on where the cut lands.
  • The same exception wrapped by the Functions host (Grpc.Core.RpcException / Rpc.RpcException, whose Detail embeds the full Service Bus message plus stack) inherits the same uniqueness.

Breakdown of the 336 backend-worker issues from that run:

Title (class in method) Issues
ServiceBusException in AmqpReceiver+<RenewMessageLockInternalAsync> 100
Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException 94
Grpc.Core.RpcException in ServiceBusFunction+<ProcessMessageAsync> 43
Grpc.Core.RpcException in ServiceBusFunction+<CarryOutAsync> 41

Each of these issues reports Occurrences: 1. This predates 0.33.0: issues #1360 to #1389 (2026-11-07) show the same pattern.

Secondary effects in the same run

  1. Filing hit GitHub's content-creation limits. Two gh issue create calls exited 1 (the hourly core quota was barely used, so this is the secondary rate limit). The filing errors then cause environment production: watermark not advanced, so the next cycle re-scans the same window and keeps filing.
  2. The window reached back to 2026-10-18. First-seen times go back four weeks, so this run scanned well past one hourly window. I have not established why the window was that long.

Expected

  • Message normalization replaces a GUID regardless of the character that follows it (_B4, ), .), and treats known per-message Azure tokens (TrackingId, Reference, Timestamp, LockToken) as volatile.
  • One fingerprint per failure class and location for this case: four issues, not 336.
  • A run that would create more than N new issues in one component stops at a cap, or rolls them into one summary issue, instead of filing them all. The existing throttled counter reported 0 here.

Repro

error-logs on any Azure Functions Service Bus consumer that sees MessageLockLost. The TrackingId:<guid>_B<n> shape is standard in Service Bus error messages.

Environment

  • specfuse-loop 0.33.0, specfuse 0.17.0, run locally (specfuse monitor run)
  • Consumer: RestoManagerApp/Backend, .specfuse/monitoring.yml, backend-worker error-logs check
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions