Repository navigation
feat(server): export supervisor session placement metrics #4688
Description
User Story
I run multi-replica OpenShell gateways with PostgreSQL. I need to see whether consistent-hash placement (#3661) is working: how often replicas redirect supervisors, whether supervisors end up on the replica the ring picked, and how many replicas each gateway thinks are in the ring.
Problem Statement
The only sign of placement today is the supervisor session: redirecting to ring-preferred replica log line. /metrics has nothing for redirects, for supervisors that end up on the wrong replica, or for ring size. So operators can't tell when replicas disagree about the ring, or when redirect dials fail and fall back to the gateway Service. The peer relay still gets traffic to the owner either way, at the cost of an extra hop on every relay.
Impact / Why This Matters
Operators running several replicas can't confirm placement works, or alert when it stops converging. Today a replica that drops out of its peers' rings, or a redirect target nobody can dial, only shows up indirectly, as more route="peer" relay attempts.
Proposed Design
Three new series next to the existing multi-replica metrics in gateway_metrics.rs:
openshell_server_supervisor_session_redirects_total{reason}: aSessionRedirectthe replica actually sent.connectwhen a new connection is redirected,shutdownwhen a stopping replica hands off an open session. A shutdown redirect that couldn't be queued doesn't count.openshell_server_supervisor_session_off_ring_admissions_total: a redirected supervisor admitted by a replica whose own ring names a different owner. That covers rings that disagree and failed redirect dials that fall back to the gateway Service and land on the wrong replica. Handoffs during a rollout count too, since the receiving replica's ring still names the stopping owner until its next refresh, so the docs say to alert on a sustained rate outside rollouts.openshell_server_gateway_ring_members: replicas in this replica's ring, updated every time the ring changes. Only replicas that join the ring export it. On SQLite or without a peer endpoint there's no ring, so the gauge stays absent instead of reading0.
Labels stay bounded: no sandbox or replica IDs.
Acceptance Criteria
- The three series appear on
/metricsas described, documented indocs/observability/gateway-metrics.mdx, with tests for the shutdown count, the off-ring rule, and the ring gauge on refresh, shutdown, and stale-ring paths.
Alternatives Considered
- A new
SupervisorHellofield that marks hellos sent after a failed redirect dial would count fallbacks exactly. It means a protocol change, so we'll only do it if the off-ring count turns out too noisy. - Not counting a replica that is next in line for the owner would hide rollout handoffs. But on two replicas the only non-owner is always next in line, so the counter would never move on the most common HA setup.
- Labeling redirects with the target replica would show where sessions go, but replica IDs are unbounded, and the scrape target already names the replica.
Part of #4223.
Activity