feat(observability): operator-controlled sampling for supervisor spans #5362

User Story

I use OpenShell with supervisor trace export enabled. I need to sample supervisor spans the way I sample other services, so traces give me latency and error detail without exporting every connection and request.

Problem Statement

Supervisors sample with the SDK default parentbased_always_on. The gateway honors OTEL_TRACES_SAMPLER, but supervisors don't receive it, and [openshell.gateway.otlp] has no sampling setting. Two mechanisms stand in for sampling:

• The deny-span limiter from feat(ocsf): correlate audit events with sampled OpenTelemetry traces #4789, a per-destination and process-wide budget.
• In feat(observability): correlate HTTP enforcement with traces and OCSF #5214, request spans that a workload's unsampled traceparent cannot suppress.

Impact / Why This Matters

Span volume scales with workload traffic, and operators have no setting to control it. That invites using traces for questions metrics should answer, such as request rates. It also lets one sandbox drive the load on the trace pipeline. Traces should serve latency and error investigation; OCSF is the complete record.

Proposed Design

• Add a sampler setting to the gateway's OTLP trace configuration, and deliver it to supervisors with their collector settings.
• Keep the sampling decision local. A workload traceparent must not force sampling, so a remote parent's sampled flag does not decide, unlike a plain parent-based sampler.
• Revisit feat(ocsf): correlate audit events with sampled OpenTelemetry traces #4789's deny-span limiter once sampling exists.
• Document collector tail sampling for keeping error and deny traces.

Suggested UX (if applicable)

[openshell.gateway.otlp.traces]
endpoint = "http://collector:4317"
sample_ratio = 0.1

Acceptance Criteria

• Operators can set a ratio for supervisor spans, and the supervisor applies it.
• A workload traceparent cannot raise the supervisor's sampling rate.
• Records still join through metadata.correlation_uid when their span isn't sampled.
• The docs describe how supervisor sampling interacts with trace.uid and with collector tail sampling.

Alternatives Considered

• Pass OTEL_TRACES_SAMPLER through to supervisors. Supervisor settings come only from [openshell.gateway.otlp], so platform tooling cannot change them.
• Rely on rate limiters. They are ad hoc samplers that only act on denials.

Agent Investigation

• Supervisors build their tracer in crates/openshell-otel/src/setup.rs with no sampler, so the SDK default applies.
• docs/how-it-works/gateways/configuration.mdx states that OTEL_* tuning variables reach the gateway but no supervisor.
• feat(observability): correlate HTTP enforcement with traces and OCSF #5214's unsampled_http_traceparent_cannot_suppress_local_span_export test asserts that an unsampled workload traceparent does not suppress export.