solana/rpc: record endpoint rate-limit headroom per method

CHANGELOG.md

@@ -29,6 +29,9 @@ All notable changes to this project will be documented in this file.
 - SDK
   - The Go telemetry SDK no longer reports a transaction the program rejected as a success. Finalization only means the cluster agreed on the transaction: a rejected instruction finalizes too, carrying the rejection in `err`, which the executor never read. It now returns a `*telemetry.ProgramError` holding the ledger's error and the program's log output, and leads the message with the program's own explanation so a caller that just prints the error still gets the reason. This is the check the serviceability executor already made. (malbeclabs/infra#1703, #4152)
   - A samples-account-full or missing-account rejection that reaches execution now returns the same `ErrSamplesAccountFull` / `ErrAccountNotFound` the equivalent preflight rejection does, via the new `ProgramError.CustomErrorCode()`. Preflight catches nearly all of these, but a write that simulated cleanly and then failed against the bank it landed on reported its code only through the finalized transaction, so a caller's account-full handling worked on one side of preflight and not the other. (malbeclabs/infra#1703, #4152)
+  - Rate-limit headroom reported by a ledger RPC endpoint is now recorded per JSON-RPC method: new `doublezero_solana_rpc_ratelimit_method_limit` and `doublezero_solana_rpc_ratelimit_method_remaining` gauges, fed by a passive transport wrapper on every retrying client `tools/solana/pkg/rpc` builds. A 429 was previously opaque: solana-go keeps only a code on its error types (`*HTTPError` is `{Code int, err error}`, `*RPCError` is `{Code, Message, Data}`) and discards the `http.Response` inside `CallForInto`, so the `X-Ratelimit-*` headers naming the cap were gone before any caller saw the error — a lake indexer activity was rate-limited on `getTransaction` for 4.5 hours with no way to learn the number it was exceeding, and the endpoint returns no such headers on a 200 either. Limits are enforced per method, per source IP, over a rolling window, so the numbers are attributed to the method they apply to; a global request-rate graph does not show this. Costs nothing where unused: the request body is inspected only once a response header proves there is something to record, it is read through `GetBody` so the body the transport sends is untouched, and an endpoint reporting no headers produces no series. Two further signals make a refusal attributable to a hop: `doublezero_solana_rpc_ratelimited_total` labels each rate-limited response by the shape that carried it, since an HTTP 429 status is what an edge limiter returns while a limiter at or behind the origin commonly answers 200 with the refusal inside the JSON-RPC envelope — only the latter reaches a caller as an `*RPCError`, and no HTTP-level metric shows it; and `doublezero_solana_rpc_responses_total` counts responses by the serving node the endpoint names, which answers whether load is concentrated on one node given limits are enforced per source IP. Neither is recoverable after the fact, which is why the 4.5h rate limit could not be attributed to the edge or the ledger node. (malbeclabs/lake#753, #4161)
+- Collector
+  - A failed internet-latency submission now retries from the first unwritten sample rather than restarting at the beginning of the flushed partition, and only the unwritten remainder is requeued — the same fix the device telemetry submitter got in #4145. A partition larger than one transaction is written in batches, so any failure part-way through left the earlier batches onchain while the retry and the next tick re-sent them, appending those samples a second time and skewing the latency they feed. Reachable today from an RPC timeout mid-partition; surfacing program rejections adds another way in. (malbeclabs/infra#1703, #4152)
 - Device Telemetry
   - A submission the telemetry program rejects onchain no longer burns the tick's remaining attempts: the agent logs the rejection with the program's explanation at Error and moves on, counting `submitter_program_error` on the errors counter. Before this, the init half of the init→write path could not be seen to fail — it skips preflight, so the rejection only showed up on the finalized transaction, which the SDK read as success — and the agent looped init→write→`account not found` every few seconds with nothing in the log naming the cause. Observed on chi-dn-dzd4, where the device's `metrics_publisher` had been set to a key the agent did not hold. Samples are requeued as with any other failure, so the next tick retries once the cause is fixed. An init the program rejects because the account already exists is excepted: that leaves the write with what it needed, so the write now runs either way and only a write that still finds nothing there reports the init failure as the reason. (malbeclabs/infra#1703, #4152)
   - A ledger RPC outage no longer stops TWAMP probing on the device telemetry agent: the pinger caches the last known epoch and refreshes it off the probe path, instead of fetching it inline and skipping the tick on failure. Probing stops when no epoch has ever been fetched, when the cached one exceeds the new `-max-epoch-staleness` (default 10h, clamped to what the sample buffer holds at `-probe-interval`), or when the cached epoch's projected end has passed. Samples taken against a cached epoch are written to that epoch's account, so a query scoped to a later epoch will not return them — the projected-end bound is what keeps that from spanning a rollover. The refresh cadence follows `-probe-interval` and can be set with the new `-epoch-refresh-interval`. (#4143)

tools/solana/pkg/jsonrpc/metrics.go

@@ -21,4 +21,17 @@ var (
 		Name: "doublezero_solana_rpc_retries_exhausted_total",
 		Help: "Number of Solana JSON-RPC requests that failed after exhausting all retry attempts, by method.",
 	}, []string{"method"})
+
+	// rateLimitedTotal counts rate-limited responses by the shape that carried the
+	// limit. That shape is the diagnostic: an edge/CDN limiter answers with an HTTP
+	// 429 status, while a limiter at or behind the origin commonly answers 200 with
+	// the refusal inside the JSON-RPC envelope. Only the second reaches this package
+	// as an *RPCError, and only the first is visible to the transport layer, so the
+	// split between these two label values says which hop refused the call — the
+	// question a provider will ask first, and one we could not answer for a 4.5h
+	// getTransaction rate limit because nothing recorded it.
+	rateLimitedTotal = promauto.NewCounterVec(prometheus.CounterOpts{
+		Name: "doublezero_solana_rpc_ratelimited_total",
+		Help: "Rate-limited JSON-RPC responses, by method and the error shape that carried the limit (http_status or jsonrpc_error).",
+	}, []string{"method", "carrier"})
 )

tools/solana/pkg/jsonrpc/retry.go

@@ -126,6 +126,9 @@ func doRetry(ctx context.Context, opt RetryOptions, label string, idempotent boo
 		}
 
 		lastErr = f(ctx)
+		if carrier := rateLimitCarrier(lastErr); carrier != "" {
+			rateLimitedTotal.WithLabelValues(label, carrier).Inc()
+		}
 		if lastErr == nil || !opt.IsRetryableFunc(lastErr) {
 			return lastErr
 		}
@@ -211,6 +214,32 @@ func isRetryableRPCCode(code int) bool {
 	return false
 }
 
+// rateLimitCarrier reports which error shape carried a rate limit, or "" if err is
+// not one. The distinction is the point: an *HTTPError means the refusal arrived as
+// an HTTP 429 status, which an edge or CDN limiter produces and the transport layer
+// can also see; an *RPCError means it arrived inside a JSON-RPC envelope on an
+// otherwise-successful response, which is what a limiter at or behind the origin
+// typically does and which no HTTP-level metric will ever show.
+//
+// -32429 is included because a provider fronting Agave mints it to mirror HTTP 429
+// inside an envelope (see isRetryableRPCCode); a bare positive 429 in an *RPCError
+// is a different origin from the same provider's -32429, which is precisely the
+// kind of thing worth being able to tell apart after the fact.
+func rateLimitCarrier(err error) string {
+	if err == nil {
+		return ""
+	}
+	var httpErr *jsonrpc.HTTPError
+	if errors.As(err, &httpErr) && httpErr.Code == http.StatusTooManyRequests {
+		return "http_status"
+	}
+	var rpcErr *jsonrpc.RPCError
+	if errors.As(err, &rpcErr) && (rpcErr.Code == http.StatusTooManyRequests || rpcErr.Code == -32429) {
+		return "jsonrpc_error"
+	}
+	return ""
+}
+
 func isRetryableJSONRPC(err error) bool {
 	if err == nil {
 		return false

tools/solana/pkg/jsonrpc/retry_test.go

@@ -420,3 +420,32 @@ func fastRetryOpt(max int) *RetryOptions {
 		MaxBackoff:  2 * time.Millisecond,
 	}
 }
+
+// TestRateLimitCarrier distinguishes which hop refused the call. An HTTP 429 status
+// is edge-shaped and also visible to the transport; a 429 inside a JSON-RPC envelope
+// on an otherwise-successful response is origin-shaped and invisible to every
+// HTTP-level metric. Being unable to tell these apart is what left a 4.5h
+// getTransaction rate limit unattributable.
+func TestRateLimitCarrier(t *testing.T) {
+	tests := []struct {
+		name string
+		err  error
+		want string
+	}{
+		{"nil", nil, ""},
+		{"http 429", &jsonrpc.HTTPError{Code: 429}, "http_status"},
+		{"http 503 is not a rate limit", &jsonrpc.HTTPError{Code: 503}, ""},
+		{"envelope 429", &jsonrpc.RPCError{Code: 429, Message: "Too many requests for a specific RPC call"}, "jsonrpc_error"},
+		{"envelope -32429", &jsonrpc.RPCError{Code: -32429}, "jsonrpc_error"},
+		{"envelope -32005 is not a rate limit", &jsonrpc.RPCError{Code: -32005}, ""},
+		{"wrapped envelope 429", fmt.Errorf("get transaction: %w", &jsonrpc.RPCError{Code: 429}), "jsonrpc_error"},
+		{"unrelated", errors.New("connection reset by peer"), ""},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			if got := rateLimitCarrier(tt.err); got != tt.want {
+				t.Errorf("rateLimitCarrier() = %q, want %q", got, tt.want)
+			}
+		})
+	}
+}

tools/solana/pkg/rpc/ratelimit.go

@@ -0,0 +1,136 @@
+package rpc
+
+import (
+	"encoding/json"
+	"io"
+	"net/http"
+	"strconv"
+
+	"github.com/prometheus/client_golang/prometheus"
+	"github.com/prometheus/client_golang/prometheus/promauto"
+)
+
+// Rate-limit headroom as reported by the endpoint, by JSON-RPC method. Method
+// cardinality is bounded by the Solana JSON-RPC surface, so it is safe as a label.
+//
+// These exist because a 429 is otherwise opaque. solana-go's error types carry
+// only a code — *HTTPError is {Code int, err error} and *RPCError is
+// {Code, Message, Data} — and the http.Response is discarded inside CallForInto,
+// so by the time a rate limit reaches a caller the headers that say what the cap
+// actually was are gone. A caller could see "Too many requests for a specific RPC
+// call" for hours without ever learning the number it was exceeding.
+//
+// Rate limits are enforced per method, per source IP, over a rolling window, so
+// `remaining` approaching zero on one method is the signal that matters — a global
+// request-rate graph will not show it, and neither will a graph of another method.
+var (
+	rateLimitMethodLimit = promauto.NewGaugeVec(prometheus.GaugeOpts{
+		Name: "doublezero_solana_rpc_ratelimit_method_limit",
+		Help: "Endpoint-reported request cap for a JSON-RPC method, per rolling rate-limit window.",
+	}, []string{"method"})
+
+	rateLimitMethodRemaining = promauto.NewGaugeVec(prometheus.GaugeOpts{
+		Name: "doublezero_solana_rpc_ratelimit_method_remaining",
+		Help: "Endpoint-reported requests left for a JSON-RPC method in the current rate-limit window.",
+	}, []string{"method"})
+
+	// responsesTotal counts responses by the serving node the endpoint names in
+	// X-Ratelimit-Node / X-RPC-Node, or "unknown" when it names none.
+	//
+	// Rate limits are enforced per source IP, so when a provider asks whether load is
+	// concentrated on one of their nodes — and whether spreading source IPs would help
+	// — this is the answer. Node identity is not on the error path, so it cannot be
+	// recovered after a refusal; recording it per response is what makes it available
+	// at all. Cardinality is the endpoint's node count, which is small.
+	responsesTotal = promauto.NewCounterVec(prometheus.CounterOpts{
+		Name: "doublezero_solana_rpc_responses_total",
+		Help: "JSON-RPC responses received, by the serving node the endpoint reports.",
+	}, []string{"node"})
+)
+
+const (
+	headerMethodLimit     = "X-Ratelimit-Method-Limit"
+	headerMethodRemaining = "X-Ratelimit-Method-Remaining"
+	headerRPCNode         = "X-RPC-Node"
+
+	// maxObservedBody bounds how much of a request body is read to recover the
+	// method name. JSON-RPC requests put "method" near the front, and this only
+	// runs on responses that already carry rate-limit headers.
+	maxObservedBody = 512
+)
+
+// rateLimitObserver records rate-limit headers into metrics as responses pass
+// through. It never alters the request or response and never fails a call: an
+// endpoint that reports nothing simply produces no series.
+type rateLimitObserver struct{ inner http.RoundTripper }
+
+func (o *rateLimitObserver) RoundTrip(req *http.Request) (*http.Response, error) {
+	resp, err := o.inner.RoundTrip(req)
+	if err != nil || resp == nil {
+		return resp, err
+	}
+
+	node := resp.Header.Get(headerRPCNode)
+	if node == "" {
+		node = "unknown"
+	}
+	responsesTotal.WithLabelValues(node).Inc()
+
+	// Cheap exit first. Most endpoints report nothing, and this must not add
+	// per-request work on the hot path when there is nothing to record — so the
+	// request body is only inspected once a header proves it is worth it.
+	limit := resp.Header.Get(headerMethodLimit)
+	remaining := resp.Header.Get(headerMethodRemaining)
+	if limit == "" && remaining == "" {
+		return resp, nil
+	}
+
+	method := requestMethod(req)
+	if method == "" {
+		return resp, nil
+	}
+	if v, convErr := strconv.ParseFloat(limit, 64); convErr == nil {
+		rateLimitMethodLimit.WithLabelValues(method).Set(v)
+	}
+	if v, convErr := strconv.ParseFloat(remaining, 64); convErr == nil {
+		rateLimitMethodRemaining.WithLabelValues(method).Set(v)
+	}
+	return resp, nil
+}
+
+// requestMethod recovers the JSON-RPC method from a request body, or "batch" for
+// a batch request whose members may differ. It reads through GetBody so the body
+// the transport is sending is left untouched; a request without GetBody (not
+// produced by the clients in this package) yields "" and is skipped.
+func requestMethod(req *http.Request) string {
+	if req.GetBody == nil {
+		return ""
+	}
+	body, err := req.GetBody()
+	if err != nil {
+		return ""
+	}
+	defer body.Close()
+
+	buf, err := io.ReadAll(io.LimitReader(body, maxObservedBody))
+	if err != nil || len(buf) == 0 {
+		return ""
+	}
+	for _, b := range buf {
+		switch b {
+		case ' ', '\t', '\r', '\n':
+			continue
+		case '[':
+			return "batch"
+		}
+		break
+	}
+	// Decode into just the method field; a truncated body fails cleanly to "".
+	var probe struct {
+		Method string `json:"method"`
+	}
+	if json.Unmarshal(buf, &probe) != nil {
+		return ""
+	}
+	return probe.Method
+}