perf(spanner): optimize built-in metrics hot path and harden concurrency - #18329
Open
olavloite wants to merge 1 commit into
Open
perf(spanner): optimize built-in metrics hot path and harden concurrency#18329olavloite wants to merge 1 commit into
olavloite wants to merge 1 commit into
Conversation
Optimize hot-path execution in the Spanner client's built-in metrics
subsystem and resolve subtle regex boundary and concurrency issues:
- Correctness & Regex Parsing:
- Add negative lookbehind `(?<![a-zA-Z0-9_-])` to GFE and AFE timing
regexes to prevent false-positive matches (e.g. `safe; dur=55`).
- Guard regex evaluations in `extract_front_end_latencies` with fast string
containment checks (`"gfet4t7" in text`, `"afe" in text`).
- Extract helper `_extract_metric_latency` and flatten header detection.
- Caching & Allocations:
- Add bounded LRU cache (`maxsize=128`) for resource path parsing,
yielding a ~9x speedup matching the Java client's caching strategy.
- Add bounded LRU cache (`maxsize=64`) for RPC method name formatting.
- Introduce `_ObservableDict` to achieve zero-allocation steady-state
OpenTelemetry attribute caching with O(1) invalidation on mutation.
- Consolidate RPC attempt preparation in `MetricsInterceptor._prepare_attempt`.
- Concurrency & Resource Safety:
- Extract `_BaseAsyncResponseWrapper` and synchronize `_metrics_recorded`
via `threading.Lock` across `cancel()`, `__del__()`, and `_record_metrics()`.
- Safely close unawaited initial metadata coroutines on cancellation.
- Guarantee ContextVar token reset in `MetricsCapture.__exit__` via `finally`.
- Harden `_safe_decode_utf8` to return `""` when passed `None`.
Contributor
There was a problem hiding this comment.
Code Review
This pull request optimizes and hardens the Spanner metrics collection system. Key changes include caching resource path parsing and method name formatting using lru_cache, introducing thread-safe locking and cancellation handling in the response wrappers, caching OpenTelemetry attributes with an _ObservableDict to invalidate the cache on modifications, and refactoring metadata parsing to safely handle various formats. Extensive unit tests have been added to cover these optimizations and edge cases. There are no review comments to address, and I have no feedback to provide.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Optimize hot-path execution in the Spanner client's built-in metrics subsystem and resolve subtle regex boundary and concurrency issues:
Correctness & Regex Parsing:
(?<![a-zA-Z0-9_-])to GFE and AFE timing regexes to prevent false-positive matches (e.g.safe; dur=55).extract_front_end_latencieswith fast string containment checks ("gfet4t7" in text,"afe" in text)._extract_metric_latencyand flatten header detection.Caching & Allocations:
maxsize=128) for resource path parsing, yielding a ~9x speedup matching the Java client's caching strategy.maxsize=64) for RPC method name formatting._ObservableDictto achieve zero-allocation steady-state OpenTelemetry attribute caching with O(1) invalidation on mutation.MetricsInterceptor._prepare_attempt.Concurrency & Resource Safety:
_BaseAsyncResponseWrapperand synchronize_metrics_recordedviathreading.Lockacrosscancel(),__del__(), and_record_metrics().MetricsCapture.__exit__viafinally._safe_decode_utf8to return""when passedNone.