Harmonized metrics for enabled APIs and event processing counters across all services

[pbleser-oc]

Is your feature request related to a problem? Please describe.

With #1312, we have recently made made some changes to a few services to be able to control by configuration whether they should process

• inbound requests via HTTP,
• inbound events via NATS,
• inbound requests via gRPC.

The purpose there, eventually, is to be able to scale instances separately, since the load distribution on those APIs can differ significantly. For example, maybe a single instance is sufficient to process inbound events, where as six instances are needed to process inbound HTTP requests.

In order to do that, one has to be able to monitor the number of running instances of each service that have

• HTTP processing enabled,
• gRPC processing enabled,
• events processing enabled.

Eventually, that would also serve as input for autoscaling, but also for alerting (e.g. in Grafana or Prometheus) when the number of instances for any of those goes below a given threshold.

Describe the solution you'd like

One way to implement it is how it was added to the graph service with #3293, or commit e436532:

• define a Prometheus gauge for events being enabled
• define a Prometheus gauge for HTTP being enabled (or gRPC, depending on which APIs are being served by that service)
• define a Prometheus CounterVec to count the events that have been processed, with labels for the event type and, if possible, the result of each event processing

services/xyz/pkg/metrics/metrics.go

type Metrics struct {
	EventsEnabled       prometheus.Gauge
	HttpEnabled         prometheus.Gauge
	EventsProcessed     *prometheus.CounterVec
}

const (
	ResultSuccess     = "success"
	ResultFailure     = "failure"
)

const (
	LabelResult    = "result"
	LabelEvent     = "event"
)

func New(registerer prometheus.Registerer, logger *log.Logger) (*Metrics, error) {
	m := &Metrics{
		EventsEnabled: prometheus.NewGauge(prometheus.GaugeOpts{
			Namespace: Namespace,
			Subsystem: Subsystem,
			Name:      "events_enabled",
			Help:      "Whether this instance consumes events (1) or not (0)",
		}),
		HttpEnabled: prometheus.NewGauge(prometheus.GaugeOpts{
			Namespace: Namespace,
			Subsystem: Subsystem,
			Name:      "http_enabled",
			Help:      "Whether this instance processes HTTP API calls (1) or not (0)",
		}),
		EventsProcessed: prometheus.NewCounterVec(prometheus.CounterOpts{
			Namespace: Namespace,
			Subsystem: Subsystem,
			Name:      "events",
			Help:      "Number of consumed events",
		}, []string{LabelEvent, LabelResult}),
// ...

The "Enabled" gauges can simply be set to 1 or 0 in the function that constructs and wires the components of that service:

services/xyz/pkg/command/server.go

if !cfg.HTTP.Disabled {
    mtrcs.HttpEnabled.Set(1)
    // ...
} else {
    mtrcs.HttpEnabled.Set(0)
    logger.Info().Str("transport", "http").Msg("HTTP server is disabled")
}

And the counter can be incremented in the function that consumes the inbound events and dispatches them to other functions or remote calls, probably along with some sort of error result, which allows for setting a value for a "result" label accordingly:

services/xyz/pkg/service/events/service.go

for loop := true; loop; {
	select {
	case e := <-evChannel:
		switch ev := e.Event.(type) {
		case events.UserSignedIn:
			name := "UserSignedIn"
			if err := doSomethingWithThis(...); err != nil {
				m.EventsProcessed.WithLabelValues(name, metrics.ResultFailure).Inc()
			} else {
				m.EventsProcessed.WithLabelValues(name, metrics.ResultSuccess).Inc()
			}
// ...

This would apply to the services listed in #1312, namely

• search
• activitylog
• eventhistory
• policies
• userlog
• graph
• frontend

If we decide to use the strategy as recommended above, then we don't need to update policies and graph, as those two have already been implemented that way.

We should first discuss with @butonic whether that would fit his requirements for observability and scaling.

[butonic]

Yes, I think we should have counters to measure what is going on. Otherwise we will only be guessing which service is processing events / requests. Maybe we can make these metrics counters more reusable by adding the necessary parameters into New(registerer prometheus.Registerer, logger *log.Logger) and move the code into a shared package?

[pbleser-oc]

Maybe we can make these metrics counters more reusable by adding the necessary parameters into New(registerer prometheus.Registerer, logger *log.Logger) and move the code into a shared package?

We already had a conversation about having more unified and reusable code for the metrics, but there is no issue for that yet.

Since 8821053 we have a pkg/metrics/metrics.go with a few reusable functions to register metrics in a cleaner way, as well as for constructing the build_info metric.

We could start adding constants for label names and label values there, as well as somewhat reusable functions to construct the metrics.