Skip to content

feat(server): export supervisor session placement metrics #4688

Open

Description

@EmilienM

User Story

I run multi-replica OpenShell gateways with PostgreSQL. I need to see whether consistent-hash placement (#3661) is working: how often replicas redirect supervisors, whether supervisors end up on the replica the ring picked, and how many replicas each gateway thinks are in the ring.

Problem Statement

The only sign of placement today is the supervisor session: redirecting to ring-preferred replica log line. /metrics has nothing for redirects, for supervisors that end up on the wrong replica, or for ring size. So operators can't tell when replicas disagree about the ring, or when redirect dials fail and fall back to the gateway Service. The peer relay still gets traffic to the owner either way, at the cost of an extra hop on every relay.

Impact / Why This Matters

Operators running several replicas can't confirm placement works, or alert when it stops converging. Today a replica that drops out of its peers' rings, or a redirect target nobody can dial, only shows up indirectly, as more route="peer" relay attempts.

Proposed Design

Three new series next to the existing multi-replica metrics in gateway_metrics.rs:

  • openshell_server_supervisor_session_redirects_total{reason}: a SessionRedirect the replica actually sent. connect when a new connection is redirected, shutdown when a stopping replica hands off an open session. A shutdown redirect that couldn't be queued doesn't count.
  • openshell_server_supervisor_session_off_ring_admissions_total: a redirected supervisor admitted by a replica whose own ring names a different owner. That covers rings that disagree and failed redirect dials that fall back to the gateway Service and land on the wrong replica. Handoffs during a rollout count too, since the receiving replica's ring still names the stopping owner until its next refresh, so the docs say to alert on a sustained rate outside rollouts.
  • openshell_server_gateway_ring_members: replicas in this replica's ring, updated every time the ring changes. Only replicas that join the ring export it. On SQLite or without a peer endpoint there's no ring, so the gauge stays absent instead of reading 0.

Labels stay bounded: no sandbox or replica IDs.

Acceptance Criteria

  • The three series appear on /metrics as described, documented in docs/observability/gateway-metrics.mdx, with tests for the shutdown count, the off-ring rule, and the ring gauge on refresh, shutdown, and stale-ring paths.

Alternatives Considered

  • A new SupervisorHello field that marks hellos sent after a failed redirect dial would count fallbacks exactly. It means a protocol change, so we'll only do it if the off-ring count turns out too noisy.
  • Not counting a replica that is next in line for the owner would hide rollout handoffs. But on two replicas the only non-owner is always next in line, so the counter would never move on the most common HA setup.
  • Labeling redirects with the target replica would show where sessions go, but replica IDs are unbounded, and the scrape target already names the replica.

Part of #4223.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:triage-neededOpened without agent diagnostics and needs triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions