Skip to content

Telemetry: log the SDK's startup warnings and a failed start - #906

Merged
schmitch merged 1 commit into
mainfrom
claude/559-telemetry-startup-logs
Nov 17, 2026
Merged

schmitch merged 1 commit into
mainfrom
claude/559-telemetry-startup-logs

Conversation

@schmitch

Copy link
Copy Markdown
Member

Fixes #559

What was wrong

  • main built Telemetry::new while no tracing subscriber was installed: the temporary one of Config::load was gone and the process's did not exist yet. The OpenTelemetry SDK reads its OTEL_* variables there and warns through tracing, so the warnings went nowhere. Helm's telemetry.samplerArg: "10%", or any OTEL_TRACES_SAMPLER_ARG that is no number, silently sampled every trace.
  • A start that failed after the subscriber was installed (database, migrations, bucket, …) returned from main as Err: printed to stderr only, never logged through tracing, and telemetry.shutdown() was skipped. With telemetry.logs on, the log backend never showed why an instance was crash-looping.

The change

  • telemetry::with_console_log runs a closure with the console log alone as the thread's subscriber. main loads the configuration under it, as before, and now builds Telemetry::new under it too.
  • What main did after installing the subscriber moved into run(config). main logs its error with tracing::error! (forge-server failed to start, with the error's whole chain in error), flushes the telemetry, and then returns the error: the exit status and the line on stderr are as before.
  • Changelog fragment docs/changes/2.2.0/559.fixed.md. The issue was in Backlog; since the fix ships in the next release, I moved it to the 2.2.0 milestone.

No admin API changes, so there is no web UI screen to add.

Verification

New test crates/server/tests/startup_logs.rs runs the real forge-server binary with traces and logs on, OTEL_TRACES_SAMPLER=parentbased_traceidratio, OTEL_TRACES_SAMPLER_ARG=10%, a mock OTLP/HTTP receiver, and a database URL on a closed port. It needs no Docker. It asserts that:

  • the SDK's WARN naming OTEL_TRACES_SAMPLER_ARG and 10% is on stdout;
  • the failed start is an ERROR line on stdout naming connect database, and stderr still carries it;
  • the receiver got the log record forge-server failed to start with severity ERROR and the cause in error. The batch delays are set to 10 minutes, so only the shutdown's flush can have sent it.

The test fails against the old main.rs (no warning on stdout) and passes with the fix.

Run locally:

  • cargo fmt --all -- --check: clean
  • cargo xtask clippy (-D warnings): clean
  • cargo xtask doc-check: clean
  • cargo xtask changelog --check: clean
  • cargo test -p forge-server against Docker (Postgres and versitygw through mirror.gcr.io): 714 passed, 0 failed, across the unit tests and all 44 integration binaries

Not run locally: the other crates' tests, cargo xtask e2e, the web suites and the *-e2e scripts. The change touches only crates/server; CI covers the rest.

Not verified: a failure of the shutdown flush itself on the error path (an unreachable collector) is not tested; it logs a warning as on a normal stop.


Generated by Claude Code

Telemetry::new ran while no tracing subscriber was installed, so the
OpenTelemetry SDK's warnings about the OTEL_* values it replaces with a
default went nowhere: an OTEL_TRACES_SAMPLER_ARG that is no number
silently sampled every trace. It is now built under the console log, as
the configuration is loaded (telemetry::with_console_log).

A start that failed after the subscriber was installed returned from
main as Err: printed to stderr only, with telemetry.shutdown() skipped.
main now logs it with tracing::error! and flushes the telemetry before
it returns the error, so the log export carries it.

Fixes #559

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012e6DJaXfLrW5Fz4BhRngM8
@schmitch
schmitch added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit 3c3b8b8 Nov 17, 2026
19 checks passed
@schmitch
schmitch deleted the claude/559-telemetry-startup-logs branch November 17, 2026 10:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Telemetry: the SDK's startup warnings and a fatal startup error never reach the logs

2 participants