feat(webapp): make the snapshot store metrics and the append alarm observable locally

The engine metrics never reached the local Prometheus. The collector had no
metrics pipeline, so the scrape target that its own configuration describes as
carrying every OpenTelemetry metric returned nothing, and Prometheus evaluated
no rules because it loaded none.

The collector now exports metrics on the port the compose file already
publishes, Prometheus loads rule files, and three rules ship with the repo. The
first pages on a sustained append failure, which is the one that matters: each
failure burns an attempt on the run it touches, so runs exhaust their retry
budget from an infrastructure fault rather than anything the task did.

Set INTERNAL_OTEL_METRIC_EXPORTER_ENABLED and INTERNAL_OTEL_METRIC_EXPORTER_URL
to see the metrics; both are already in .env.example.

Author: d-cs

docker/config/alerts/snapshot-store.yml

@@ -0,0 +1,48 @@
+# Alerting rules for the execution-snapshot store.
+#
+# These are paging rules, not dashboard panels. A sustained append failure burns an attempt number
+# on every state transition, so runs exhaust their retry budget from an infrastructure fault rather
+# than from anything the task did. That is silent to the customer until their run fails for good.
+groups:
+  - name: snapshot-store
+    interval: 30s
+    rules:
+      - alert: SnapshotStoreAppendFailing
+        # Any sustained failure at all, not a rate threshold: one failed append per minute still
+        # burns one attempt per minute on the runs it touches.
+        expr: sum(rate(run_engine_snapshot_store_append_failed_total[5m])) > 0
+        for: 10m
+        labels:
+          severity: page
+        annotations:
+          summary: "Execution-snapshot appends to Redis are failing"
+          description: >-
+            Appends have failed continuously for 10 minutes. Each failure burns an attempt number on
+            the run it touches, so runs will exhaust their retry budget from infrastructure rather
+            than task failure. Turn the snapshotStoreMode flag down to off, which takes effect
+            without a deploy, then investigate the Redis endpoint.
+
+      - alert: SnapshotStoreQuarantineFailing
+        # A quarantine that cannot be applied leaves a run's Redis head permanently behind Postgres.
+        expr: sum(rate(run_engine_snapshot_store_quarantine_total{outcome="failed"}[10m])) > 0
+        for: 15m
+        labels:
+          severity: page
+        annotations:
+          summary: "Execution-snapshot keyspaces cannot be quarantined"
+          description: >-
+            A divergent Redis head could not be dropped, so that run keeps a head that is behind
+            Postgres. Do not move the dial to redis-read while this is firing: a frozen head is
+            served as if it were current.
+
+      - alert: SnapshotStoreSweepNotCompleting
+        # The sweep is the only reaper for orphaned keyspaces. Its absence is silent by nature.
+        expr: sum(increase(run_engine_snapshot_store_sweep_pass_total{outcome="completed"}[24h])) == 0
+        for: 1h
+        labels:
+          severity: ticket
+        annotations:
+          summary: "No execution-snapshot sweep pass completed in 24 hours"
+          description: >-
+            The sweep runs every 6 hours by default, so 24 hours with no completed pass means the
+            job is not running, is failing, or is losing its lock every pass.

docker/config/prometheus.yml

@@ -5,6 +5,11 @@ global:
   scrape_interval: 15s
   evaluation_interval: 15s
 
+# Paging rules live in the repo so they are reviewable and can be checked locally with
+# `promtool check rules`. Mounted read-only by docker-compose.extras.yml.
+rule_files:
+  - /etc/prometheus/alerts/*.yml
+
 scrape_configs:
   # Scrape OpenTelemetry Collector's Prometheus exporter
   # This includes all OTel metrics (batch queue, fair queue, etc.)

docker/docker-compose.extras.yml

@@ -95,6 +95,7 @@ services:
     restart: always
     volumes:
       - ./config/prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - ./config/alerts:/etc/prometheus/alerts:ro
       - prometheus-data:/prometheus
     ports:
       - "${PROMETHEUS_HOST_PORT:-9090}:9090"

docker/otel-collector-config.yaml

@@ -12,6 +12,15 @@ processors:
 exporters:
   logging:
     verbosity: normal
+  # Serves the collected metrics on :8889/metrics, which docker-compose.extras.yml publishes and
+  # config/prometheus.yml already scrapes. Without this pipeline that scrape target returns nothing,
+  # so no OTel metric is visible locally.
+  prometheus:
+    endpoint: 0.0.0.0:8889
+    # Keeps the metric names as OpenTelemetry emitted them, so a rule written against
+    # run_engine_snapshot_store_* matches without a namespace guess.
+    resource_to_telemetry_conversion:
+      enabled: false
   otlphttp:
     endpoint: "http://host.docker.internal:3030/otel"
     compression: none
@@ -26,3 +35,7 @@ service:
       receivers: [otlp]
       processors: [batch]
       exporters: [otlphttp]
+    metrics:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [prometheus]