Skip to content

Monitoring, alerting, and on-call runbook #133

Description

@truthixify

Tier: L (5-7 days) | Type: docs

Context. guides/ops/self-hosted-deployment.mdx (Wave 7) walks getting contracts up. Nothing today tells the operator what to watch, when to page, or how to respond. Teams running Wraith in production need a full runbook: healthy metric baselines, ready-to-import dashboards, alert rules, incident playbooks, and a tabletop exercise. This is more than a page; it is a bundle of ops artifacts, hence the larger tier.

Scope.

  • New guides/ops/monitoring-and-on-call.mdx.
  • Key metrics: RPC latency, announcement-lag, indexer backlog, watcher event-drop rate, contract error rate.
  • Grafana dashboard JSON in guides/ops/dashboards/ (one for RPC, one for indexer, one for contracts).
  • Prometheus alerting rules file with rationale per rule.
  • Six named incident playbooks: RPC outage, Horizon backpressure, indexer stall, contract mispublish, watcher drop-spike, key-rotation incident.
  • Sev matrix mapping incident types to response SLA, cross-linked with the auditor guide's severity ladder.
  • Tabletop exercise script for a quarterly drill.

Acceptance.

  • Reviewer imports each dashboard into a Grafana instance without error
  • Alert rules validated with promtool check rules
  • Reviewer walks one playbook end-to-end on a staging deployment
  • Renders in mint dev
  • Passes Compile docs snippets CI
  • Cross-linked with self-hosted-deployment.mdx and reference/auditor-guide.mdx
  • Nav entry under Operations

Files. guides/ops/monitoring-and-on-call.mdx (new), guides/ops/dashboards/ (new), guides/ops/alerts/ (new), docs.json.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Stellar WaveIssues in the Stellar wave programdocsDocumentationdripsFunded via Drips Networkhelp wantedExtra attention is neededopsstellar-waveAuto-created for Wave 8wave-8Auto-created for Wave 8writing

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions