feat(observability): supervisor decision and latency metrics #5364

User Story

I operate OpenShell gateways and sandboxes. I want alerts on supervisor latency, and I want to see which latency bucket is growing and whether request or denial rates rose with it. Today I'd have to reconstruct that from traces or OCSF records.

Problem Statement

The gateway serves Prometheus metrics, but supervisors export none. There is no rate of egress decisions by outcome, and no latency distribution for policy evaluation, request inspection, or upstream connect.

Impact / Why This Matters

Operators cannot alert on a denial storm or on supervisor-added latency. The workaround is keeping every trace or counting OCSF records in the log pipeline, which pushes rate questions onto traces and SIEMs and still gives no latency distribution.

Proposed Design

• Counters for egress decisions, labeled by outcome, protocol, and policy.
• Histograms for policy evaluation, supervisor-added time on inspected requests, and upstream connect.
• Exemplars linking histogram buckets to sampled trace IDs, so an alert leads to a representative trace.
• Bounded labels: no destination host labels.
• Open question: how metrics leave a supervisor. Options include OTLP metrics push to the collector, aggregation through the gateway's Prometheus endpoint, or a per-supervisor scrape target.

Acceptance Criteria

• Operators can query decision rates by outcome and policy, and latency histograms for the stages above.
• Histogram buckets carry exemplars when a sampled span is active.
• Label cardinality stays bounded under any workload behavior.
• The docs list each metric alongside the gateway metrics page.

Alternatives Considered

• Derive rates from OCSF records in the log pipeline. This works, but it pushes counting onto every SIEM and gives no latency data.
• Derive rates and latency from unsampled traces. Traces are for latency and error investigation; exporting all of them to count requests is costly and depends on sampling being off.

Agent Investigation

• Gateway metrics and their histogram conventions are documented in docs/observability/gateway-metrics.mdx.
• No crate under crates/openshell-supervisor* records metrics.
• Related: feat(observability): correlate HTTP enforcement with traces and OCSF #5214 and feat(ocsf): correlate audit events with sampled OpenTelemetry traces #4789 add request and deny spans; these metrics answer rate questions so those spans can be sampled (feat(observability): operator-controlled sampling for supervisor spans #5362).