Summary

Two observability gaps: nothing measures what a sandbox actually consumes (CPU seconds, RSS) — so no noisy-neighbor detection, no usage-based billing beyond wall-clock seconds, no overcommit data — and claim latency is exported only as counter totals (averages), hiding tails. Three proposals, independently landable.

Current state

• /metrics (hand-rendered, server/metrics.go): pool/claim gauges, lifecycle counters, *_seconds_total for average latency only.
• The usage journal bills wall-clock seconds by lifecycle interval folding; it never sees actual resource consumption.
• Per-claim VMM pids are known to cocoon; nothing reads their /proc stats.
• No trace correlation between an SDK call and the node's logs/journal.

Proposal

1. Per-sandbox resource stats

Source: the VMM process (/proc/<pid>: utime+stime, VmRSS) — works for CH and FC alike, no guest cooperation, no cgroup dependency.

• GET /v1/sandboxes/{id}/stats (sandbox token) and GET /v1/stats (root, all claims) — sampled on demand, {cpu_seconds, rss_bytes, state}.
• Per-tenant aggregates on /metrics (sandboxd_tenant_cpu_seconds_total, sandboxd_tenant_rss_bytes), sampled by the existing telemetry tick. Deliberately NO per-sandbox Prometheus series — unbounded cardinality; the operator index + stats endpoint covers spot checks.
• Billing: release/reap/hibernate usage events gain cpu_seconds/max_rss — read once at teardown, off the claim path. Cumulative across hibernate cycles: accumulate into the claim record at each hibernate (the VMM dies there; the counter must survive it). A reap whose VMM is already gone records the last accumulated value — honest, not fabricated.

2. Claim-path histograms

Replace averages-only with fixed-bucket histograms per tier (warm/clone/cold) and per phase (queue → provision → probe), hand-rendered like the rest of metrics.go. Buckets chosen from the numbers of record (0.5ms/5/50/100/250/500/1000). The phase timestamps largely exist for the *_seconds_total counters; this changes the export shape, not the instrumentation.

3. Trace propagation (cheap, no OTel dependency)

Accept traceparent on the resource-creating endpoints; echo the trace id into the structured claim log line and the usage event. Correlation happens in the operator's collector. Full OTLP export is rejected for now: an SDK dependency + config surface for something a log pipeline already does; revisit when someone actually runs a tracing backend against this.

Hot-path cost

Claim path: zero new syscalls (phase timestamps already taken; histogram observe is arithmetic). Stats: on-demand reads + one /proc read per claim per telemetry tick for tenant aggregates (bounded by live claims, off the data plane). Teardown: one /proc read before destroy. State the measured delta in the PR per the perf discipline.

Acceptance evidence

• Unit: stat parsing, hibernate-cycle accumulation, reap-after-death fallback.
• e2e: spin CPU in a sandbox → stats shows monotonic cpu_seconds; release event carries plausible cpu_seconds; /metrics shows tenant gauges and tier histograms; rpcbench p50/p99 unchanged vs baseline in the same run.

Estimated size: ~200-250 lines + tests; proposals land separately (1 is the substance, 2 and 3 are small).