Repository navigation
Telemetry: log the SDK's startup warnings and a failed start - #906
Merged
Merged
Conversation
Telemetry::new ran while no tracing subscriber was installed, so the OpenTelemetry SDK's warnings about the OTEL_* values it replaces with a default went nowhere: an OTEL_TRACES_SAMPLER_ARG that is no number silently sampled every trace. It is now built under the console log, as the configuration is loaded (telemetry::with_console_log). A start that failed after the subscriber was installed returned from main as Err: printed to stderr only, with telemetry.shutdown() skipped. main now logs it with tracing::error! and flushes the telemetry before it returns the error, so the log export carries it. Fixes #559 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012e6DJaXfLrW5Fz4BhRngM8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #559
What was wrong
mainbuiltTelemetry::newwhile notracingsubscriber was installed: the temporary one ofConfig::loadwas gone and the process's did not exist yet. The OpenTelemetry SDK reads itsOTEL_*variables there and warns throughtracing, so the warnings went nowhere. Helm'stelemetry.samplerArg: "10%", or anyOTEL_TRACES_SAMPLER_ARGthat is no number, silently sampled every trace.mainasErr: printed to stderr only, never logged throughtracing, andtelemetry.shutdown()was skipped. Withtelemetry.logson, the log backend never showed why an instance was crash-looping.The change
telemetry::with_console_logruns a closure with the console log alone as the thread's subscriber.mainloads the configuration under it, as before, and now buildsTelemetry::newunder it too.maindid after installing the subscriber moved intorun(config).mainlogs its error withtracing::error!(forge-server failed to start, with the error's whole chain inerror), flushes the telemetry, and then returns the error: the exit status and the line on stderr are as before.docs/changes/2.2.0/559.fixed.md. The issue was inBacklog; since the fix ships in the next release, I moved it to the2.2.0milestone.No admin API changes, so there is no web UI screen to add.
Verification
New test
crates/server/tests/startup_logs.rsruns the realforge-serverbinary with traces and logs on,OTEL_TRACES_SAMPLER=parentbased_traceidratio,OTEL_TRACES_SAMPLER_ARG=10%, a mock OTLP/HTTP receiver, and a database URL on a closed port. It needs no Docker. It asserts that:WARNnamingOTEL_TRACES_SAMPLER_ARGand10%is on stdout;ERRORline on stdout namingconnect database, and stderr still carries it;forge-server failed to startwith severity ERROR and the cause inerror. The batch delays are set to 10 minutes, so only the shutdown's flush can have sent it.The test fails against the old
main.rs(no warning on stdout) and passes with the fix.Run locally:
cargo fmt --all -- --check: cleancargo xtask clippy(-D warnings): cleancargo xtask doc-check: cleancargo xtask changelog --check: cleancargo test -p forge-serveragainst Docker (Postgres and versitygw throughmirror.gcr.io): 714 passed, 0 failed, across the unit tests and all 44 integration binariesNot run locally: the other crates' tests,
cargo xtask e2e, the web suites and the*-e2escripts. The change touches onlycrates/server; CI covers the rest.Not verified: a failure of the shutdown flush itself on the error path (an unreachable collector) is not tested; it logs a warning as on a normal stop.
Generated by Claude Code