Skip to content

Telemetry: export to the OTLP endpoint directly, never through the outbound proxy - #900

Queued
schmitch wants to merge 2 commits into
mainfrom
claude/558-otlp-exporter-no-proxy
Queued

schmitch wants to merge 2 commits into
mainfrom
claude/558-otlp-exporter-no-proxy

Conversation

@schmitch

Copy link
Copy Markdown
Member

Fixes #558

The problem

opentelemetry-otlp builds its HTTP client with reqwest::blocking::Client::builder(), which takes the proxy from HTTP_PROXY / HTTPS_PROXY / ALL_PROXY. With an outbound proxy set for the proxy repositories and no NO_PROXY entry for the Collector, every export went to that proxy, also for the default http://localhost:4318: the exports failed, and to a plain-HTTP endpoint the proxy was sent the payload and OTEL_EXPORTER_OTLP_HEADERS in the clear.

The change

  • crates/server/src/telemetry.rs: the metric, span and log exporters (the three Forge builds) each get a client built with .no_proxy(), through WithHttpConfig::with_http_client, which is how opentelemetry-otlp 0.33.1 (the version in Cargo.lock) takes a custom client. no_proxy() also turns off the platform's proxy settings (reqwest's system-proxy).
  • A client passed in replaces the exporter's own, which carried the export timeout, so direct_client resolves the timeout the same way the exporter does: the signal's OTEL_EXPORTER_OTLP_{METRICS,TRACES,LOGS}_TIMEOUT, then OTEL_EXPORTER_OTLP_TIMEOUT, then 10 s. The client is built on a thread of its own, as the exporter builds its default one, since a blocking client cannot be built on an async worker.
  • crates/server/Cargo.toml: the server names reqwest's blocking feature, which it now uses itself (it was already on through opentelemetry-http; Cargo.lock is unchanged).
  • docs/deployment.md: Telemetry says the export connects directly and ignores the proxy variables, and what to do when a backend is reachable through the proxy alone; the Outbound proxy note says the variables route the upstream traffic only.
  • docs/changes/2.2.0/558.fixed.md: the changelog fragment. The issue was in Backlog; it is fixed ahead of its milestone, so the fragment is in the next release's directory and the issue is moved to the 2.2.0 milestone.

Things to decide in review

  • Behaviour change. An installation that exported straight to a backend reachable only through the outbound proxy no longer reaches it; it exports to a Collector, which forwards through the proxy. The issue prefers .no_proxy() over exempting loopback alone, and the guide and the fragment say this. There is no setting to turn the proxy back on for the export.
  • fixed, not security. The headers reached the operator's own egress proxy, and only for a plain-HTTP endpoint (an https:// endpoint is tunnelled with CONNECT). I filed it under Fixed; rename the fragment to 558.security.md if you judge otherwise.

No admin API changes, so there is no web UI screen.

Verification

  • crates/server/tests/otlp_export.rs (the export as main sets it up) now runs with HTTP_PROXY, HTTPS_PROXY and ALL_PROXY (both cases) pointing at a second local listener standing in for the proxy, NO_PROXY unset and OTEL_EXPORTER_OTLP_HEADERS=api-key=s3cret. It asserts that the proxy was sent nothing and that metrics, traces and logs reach the OTLP receiver with the header. It passes with the change, and fails with telemetry.rs reverted to main.
  • cargo fmt --all --check, cargo clippy --workspace --all-targets --all-features -- -D warnings, cargo xtask changelog --check, cargo xtask docs --check, cargo xtask unused and cargo xtask doc-check: all clean.
  • cargo test -p forge-server --lib --bins --test telemetry --test otlp_export: 291 unit tests, 6 telemetry tests and the export test pass (Docker through mirror.gcr.io).

Not verified locally:

  • The rest of forge-server's integration test binaries, the other crates' tests and the e2e suites. A run of all of the server's binaries (test-parallel.sh -p forge-server) ran out of disk while linking in the session's container, so that is left to CI.
  • A real egress proxy and a real Collector: the test uses local stand-ins for both, over plain HTTP. An https:// OTLP endpoint was not exercised.

Generated by Claude Code

…tbound proxy

opentelemetry-otlp builds its HTTP client from the environment's proxy
settings, so with HTTP_PROXY / HTTPS_PROXY / ALL_PROXY set for the proxy
repositories and no NO_PROXY entry for the Collector, every export went to
the outbound proxy: the exports failed, and to a plain-HTTP endpoint the
proxy was sent the payload and OTEL_EXPORTER_OTLP_HEADERS in the clear.

Give the metric, span and log exporters a client built with no_proxy(),
with the timeout the exporter resolves for its default client, and say in
the deployment guide that the proxy variables do not route the export.

Fixes #558

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012e6DJbiZTV7AXpNjVtgYrt
@schmitch
schmitch added this pull request to the merge queue Nov 17, 2026
Any commits made after this event will not be merged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Telemetry: the OTLP exporter goes through HTTP_PROXY / ALL_PROXY, also for a Collector on localhost

2 participants