Repository navigation
Conversation
Benchmarking against the OpenTelemetry spec's "Performance Implications" section (DRIVERS-3620) showed that the Java driver paid almost the same cost under a 1% sampler as with every span recorded. A CPU profile placed the cost in three places: building the tag set on span stop (chained KeyValues.and() merges, run on every command regardless of sampling), decoding the just-encoded BSON command to read its name, and scope handling inside the Micrometer/OpenTelemetry bridge. This change addresses the first two: - MongodbObservationContext gains a `recording` flag. After the observation starts, MicrometerTracer asks TracingSpanRecordingProbe whether the tracing SDK is recording the span and clears the flag if not. The probe resolves the micrometer-tracing span type reflectively, since driver-core depends only on micrometer-observation; when that module is absent every span is treated as recording, as before. - DefaultMongodbObservationConvention returns no high-cardinality key values for a non-recording span, and builds all key-value sets from a single array instead of repeated KeyValues.and() merges. Low-cardinality values are still produced for non-recording spans because metrics handlers on the same registry consume them. - CommandMessage exposes getCommandName(); TracingManager uses it and decodes the command only for getMore, to read the cursor id. Measured on the spec's benchmark (5 interleaved repetitions, medians): added CPU per operation roughly halved in every configuration, and the api-only cost is within 1% of tracing off on all four tasks. Span counts and tag contents are unchanged.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reduces the Java driver's OpenTelemetry tracing overhead, following the Implementation Guidelines in mongodb/specifications#1986 (DRIVERS-3620). Draft: needs a JAVA ticket and a decision on the reflective probe before review proper.
Why
Benchmarking on the spec's method showed Java paying almost the same cost with a 1% sampler as with every span recorded. A JFR profile under
sdk-ratioput 34% of CPU samples on tracing frames, in this order:DefaultMongodbObservationConventionfor all low- and high-cardinality key values unconditionally inObservation.stop(), and the convention chainedKeyValues.and(), which merges and re-sorts per call. This ran on every command whether or not the sampler kept the span.TracingManager.createTracingSpandecoded the just-encoded BSON command into aByteBufBsonDocumentonly to read its first key.What changes
MongodbObservationContext: newrecordingflag (defaulttrue).MicrometerTracer: afterobservation.start(), asksTracingSpanRecordingProbewhether the SDK is recording the span; clears the flag if not.TracingSpanRecordingProbe(new, internal):driver-coredepends only onmicrometer-observation, so themicrometer-tracingspan type is resolved reflectively once via method handles (TracingObservationHandler.TracingContext.getSpan(),Span.isNoop()). Micrometer's tracing handler stores the span in the context under that class key duringonStart, insidestart(); the OTel bridge'sisNoop()is!isRecording(). Whenmicrometer-tracingis absent or no span is attached, every span is treated as recording (previous behaviour).DefaultMongodbObservationConvention: high-cardinality key values are skipped for a non-recording span; all three builders fill oneKeyValue[]and callKeyValues.ofonce. Low-cardinality values are still produced for non-recording spans because metrics handlers on the same registry consume them.CommandMessage.getCommandName();TracingManageruses it and decodes the command only forgetMore(cursor id).No public API change beyond two accessors on
MongodbObservationContext. Tag names and values are unchanged.Verification
DefaultMongodbObservationConventionTest(6) and aCommandMessageTestcase: pass.MicrometerProseTestsync and reactive: pass.MicrometerTracingTestsync and reactive at the pinned spec revision: pass except the two tests needingconfigureFailPoint, which the local server lacked.sdk-always, 0.02/opsdk-ratio).Measured effect (same host and harness, 5 interleaved reps, medians; loss vs
off/ added CPU µs per op excl. GC)Added CPU per op roughly halved in every configuration;
api-onlyis within 1% ofoffon all four tasks. Thesdk-ratio/sdk-alwaysgap stays small because the remaining cost is span creation and scope handling in the SDK, which bears on the spec PR's gating diagnostic.Harness: companion draft "DRIVERS-3620: OpenTelemetry overhead benchmark suite".
Review notes
optionalImplementationonmicrometer-tracing, which is heavier for one boolean.