Repository navigation
Conversation
This commit tests the preview end to end against both environments, under the scoped `observability-preview` Grafana account. The preview must show two dashboards in one per-PR folder, nested inside `Previews`, with one link for each environment. Do not merge this commit. The titles say "preview smoke test". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T6hzc2RPYWnjG8xPbmYhf4
Contributor
Grafana previewPreview deployed for 2 dashboards in staging and production.
Dashboards:
Previews are torn down automatically when this PR is closed or merged. (Run: https://github.com/cardstack/boxel/actions/runs/39978134274) |
Contributor
Observability diff (vs staging)Show diffdiff --git a/tmp/remote-canon.DpaZPa/dashboards/boxel-status/job-queue.json b/tmp/committed-canon.7J8D3v/dashboards/boxel-status/job-queue.json
index 76875ee..08397de 100644
--- a/tmp/remote-canon.DpaZPa/dashboards/boxel-status/job-queue.json
+++ b/tmp/committed-canon.7J8D3v/dashboards/boxel-status/job-queue.json
@@ -119,7 +119,7 @@
"refId": "A"
}
],
- "title": "Pending",
+ "title": "Pending (preview smoke test)",
"type": "stat"
},
{
diff --git a/tmp/remote-canon.DpaZPa/dashboards/boxel-status/overview.json b/tmp/committed-canon.7J8D3v/dashboards/boxel-status/overview.json
index e77b1ad..b32c1da 100644
--- a/tmp/remote-canon.DpaZPa/dashboards/boxel-status/overview.json
+++ b/tmp/committed-canon.7J8D3v/dashboards/boxel-status/overview.json
@@ -49,6 +49,7 @@
"mode": "html"
},
"pluginVersion": "12.4.3",
+ "title": "",
"transparent": true,
"type": "text"
},
@@ -467,6 +468,7 @@
"refId": "A"
}
],
+ "title": "",
"transformations": [
{
"id": "labelsToFields",
@@ -1293,136 +1295,72 @@
"type": "cloudwatch",
"uid": "cef5x9o3yzawwf"
},
- "description": "ECS CPU and memory utilisation for `boxel-prerender-server-${env}` on cluster `${env}` — averages over the panel refresh window. Task counts (Run / Need) appear in the stat row below this bargauge. Locally these show 'No data' — CloudWatch is staging/production only.",
+ "description": "ECS task count for `boxel-realm-server-${env}` — `Run` is RunningTaskCount, `Need` is DesiredTaskCount (both from ECS/ContainerInsights, Maximum over the panel refresh window). Run < Need means the service is under-scaled or tasks are crashing.",
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
- "decimals": 1,
- "links": [
+ "decimals": 2,
+ "mappings": [
{
- "targetBlank": false,
- "title": "Open Prerender Server dashboard",
- "url": "/d/boxel-svc-prerender-server"
+ "options": {
+ "pattern": "^0\\.0*(\\d+)$",
+ "result": {
+ "color": "red",
+ "text": "0 / $1"
+ }
+ },
+ "type": "regex"
+ },
+ {
+ "options": {
+ "pattern": "^(\\d+)\\.0*(\\d+)$",
+ "result": {
+ "color": "green",
+ "text": "$1 / $2"
+ }
+ },
+ "type": "regex"
}
],
- "mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
- "color": "transparent"
- }
- ]
- },
- "unit": "short"
- },
- "overrides": [
- {
- "matcher": {
- "id": "byFrameRefID",
- "options": "A"
- },
- "properties": [
- {
- "id": "displayName",
- "value": "CPU"
- },
- {
- "id": "unit",
- "value": "percent"
+ "color": "red"
},
{
- "id": "thresholds",
- "value": {
- "mode": "absolute",
- "steps": [
- {
- "color": "transparent"
- },
- {
- "color": "green",
- "value": 0
- },
- {
- "color": "yellow",
- "value": 70
- },
- {
- "color": "red",
- "value": 90
- }
- ]
- }
+ "color": "green",
+ "value": 1
}
]
},
- {
- "matcher": {
- "id": "byFrameRefID",
- "options": "B"
- },
- "properties": [
- {
- "id": "displayName",
- "value": "Mem"
- },
- {
- "id": "unit",
- "value": "percent"
- },
- {
- "id": "thresholds",
- "value": {
- "mode": "absolute",
- "steps": [
- {
- "color": "transparent"
- },
- {
- "color": "green",
- "value": 0
- },
- {
- "color": "yellow",
- "value": 70
- },
- {
- "color": "red",
- "value": 90
- }
- ]
- }
- }
- ]
- }
- ]
+ "unit": "none"
+ },
+ "overrides": []
},
"gridPos": {
- "h": 4,
+ "h": 3,
"w": 4,
- "x": 4,
- "y": 19
+ "x": 0,
+ "y": 23
},
- "id": 14,
+ "id": 23,
"options": {
- "displayMode": "gradient",
- "maxVizHeight": 300,
- "minVizHeight": 10,
- "minVizWidth": 0,
- "namePlacement": "auto",
+ "colorMode": "value",
+ "graphMode": "none",
+ "justifyMode": "center",
"orientation": "horizontal",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
- "fields": "",
+ "fields": "e1",
"values": false
},
- "showUnfilled": true,
- "sizing": "auto",
- "valueMode": "color"
+ "textMode": "value",
+ "wideLayout": true
},
"pluginVersion": "12.4.3",
"targets": [
@@ -1433,16 +1371,18 @@
},
"dimensions": {
"ClusterName": "${env}",
- "ServiceName": "boxel-prerender-server-${env}"
+ "ServiceName": "boxel-realm-server-${env}"
},
+ "hide": true,
+ "id": "a",
"metricEditorMode": 0,
- "metricName": "CPUUtilization",
+ "metricName": "RunningTaskCount",
"metricQueryType": 0,
- "namespace": "AWS/ECS",
+ "namespace": "ECS/ContainerInsights",
"period": "",
"refId": "A",
"region": "default",
- "statistic": "Average"
+ "statistic": "Maximum"
},
{
"datasource": {
@@ -1451,27 +1391,60 @@
},
"dimensions": {
"ClusterName": "${env}",
- "ServiceName": "boxel-prerender-server-${env}"
+ "ServiceName": "boxel-realm-server-${env}"
},
+ "hide": true,
+ "id": "b",
"metricEditorMode": 0,
- "metricName": "MemoryUtilization",
+ "metricName": "DesiredTaskCount",
"metricQueryType": 0,
- "namespace": "AWS/ECS",
+ "namespace": "ECS/ContainerInsights",
"period": "",
"refId": "B",
"region": "default",
- "statistic": "Average"
+ "statistic": "Maximum"
+ },
+ {
+ "datasource": {
+ "type": "cloudwatch",
+ "uid": "cef5x9o3yzawwf"
+ },
+ "dimensions": {},
+ "expression": "a + b/100",
+ "id": "e1",
+ "metricEditorMode": 1,
+ "metricName": "",
+ "metricQueryType": 0,
+ "namespace": "",
+ "period": "",
+ "refId": "e1",
+ "region": "default",
+ "statistic": ""
}
],
- "title": "Prerender",
- "type": "bargauge"
+ "title": "Tasks",
+ "transformations": [
+ {
+ "id": "convertFieldType",
+ "options": {
+ "conversions": [
+ {
+ "destinationType": "string",
+ "targetField": "Value"
+ }
+ ],
+ "fields": {}
+ }
+ }
+ ],
+ "type": "stat"
},
{
"datasource": {
"type": "cloudwatch",
"uid": "cef5x9o3yzawwf"
},
- "description": "ECS CPU and memory utilisation for `boxel-prerender-manager-${env}` on cluster `${env}` — averages over the panel refresh window. Task counts (Run / Need) appear in the stat row below this bargauge. Locally these show 'No data' — CloudWatch is staging/production only.",
+ "description": "ECS CPU and memory utilisation for `boxel-prerender-server-${env}` on cluster `${env}` — averages over the panel refresh window. Task counts (Run / Need) appear in the stat row below this bargauge. Locally these show 'No data' — CloudWatch is staging/production only.",
"fieldConfig": {
"defaults": {
"color": {
@@ -1481,8 +1454,8 @@
"links": [
{
"targetBlank": false,
- "title": "Open Prerender Manager dashboard",
- "url": "/d/boxel-svc-prerender-manager"
+ "title": "Open Prerender Server dashboard",
+ "url": "/d/boxel-svc-prerender-server"
}
],
"mappings": [],
@@ -1580,10 +1553,10 @@
"gridPos": {
"h": 4,
"w": 4,
- "x": 8,
+ "x": 4,
"y": 19
},
- "id": 15,
+ "id": 14,
"options": {
"displayMode": "gradient",
"maxVizHeight": 300,
@@ -1611,7 +1584,7 @@
},
"dimensions": {
"ClusterName": "${env}",
- "ServiceName": "boxel-prerender-manager-${env}"
+ "ServiceName": "boxel-prerender-server-${env}"
},
"metricEditorMode": 0,
"metricName": "CPUUtilization",
@@ -1629,7 +1602,7 @@
},
"dimensions": {
"ClusterName": "${env}",
- "ServiceName": "boxel-prerender-manager-${env}"
+ "ServiceName": "boxel-prerender-server-${env}"
},
"metricEditorMode": 0,
"metricName": "MemoryUtilization",
@@ -1641,7 +1614,7 @@
"statistic": "Average"
}
],
- "title": "Prerender Mgr",
+ "title": "Prerender",
"type": "bargauge"
},
{
@@ -1649,136 +1622,72 @@
"type": "cloudwatch",
"uid": "cef5x9o3yzawwf"
},
- "description": "ECS CPU and memory utilisation for `boxel-worker-${env}` on cluster `${env}` — averages over the panel refresh window. Task counts (Run / Need) appear in the stat row below this bargauge. Locally these show 'No data' — CloudWatch is staging/production only.",
+ "description": "ECS task count for `boxel-prerender-server-${env}` — `Run` is RunningTaskCount, `Need` is DesiredTaskCount (both from ECS/ContainerInsights, Maximum over the panel refresh window). Run < Need means the service is under-scaled or tasks are crashing.",
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
- "decimals": 1,
- "links": [
+ "decimals": 2,
+ "mappings": [
{
- "targetBlank": false,
- "title": "Open Worker dashboard",
- "url": "/d/boxel-svc-worker"
+ "options": {
+ "pattern": "^0\\.0*(\\d+)$",
+ "result": {
+ "color": "red",
+ "text": "0 / $1"
+ }
+ },
+ "type": "regex"
+ },
+ {
+ "options": {
+ "pattern": "^(\\d+)\\.0*(\\d+)$",
+ "result": {
+ "color": "green",
+ "text": "$1 / $2"
+ }
+ },
+ "type": "regex"
}
],
- "mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
- "color": "transparent"
+ "color": "red"
+ },
+ {
+ "color": "green",
+ "value": 1
}
]
},
- "unit": "short"
+ "unit": "none"
},
- "overrides": [
- {
- "matcher": {
- "id": "byFrameRefID",
- "options": "A"
- },
- "properties": [
- {
- "id": "displayName",
- "value": "CPU"
- },
- {
- "id": "unit",
- "value": "percent"
- },
- {
- "id": "thresholds",
- "value": {
- "mode": "absolute",
- "steps": [
- {
- "color": "transparent"
- },
- {
- "color": "green",
- "value": 0
- },
- {
- "color": "yellow",
- "value": 70
- },
- {
- "color": "red",
- "value": 90
- }
- ]
- }
- }
- ]
- },
- {
- "matcher": {
- "id": "byFrameRefID",
- "options": "B"
- },
- "properties": [
- {
- "id": "displayName",
- "value": "Mem"
- },
- {
- "id": "unit",
- "value": "percent"
- },
- {
- "id": "thresholds",
- "value": {
- "mode": "absolute",
- "steps": [
- {
- "color": "transparent"
- },
- {
- "color": "green",
- "value": 0
- },
- {
- "color": "yellow",
- "value": 70
- },
- {
- "color": "red",
- "value": 90
- }
- ]
- }
- }
- ]
- }
- ]
+ "overrides": []
},
"gridPos": {
- "h": 4,
+ "h": 3,
"w": 4,
- "x": 12,
- "y": 19
+ "x": 4,
+ "y": 23
},
- "id": 16,
+ "id": 24,
"options": {
- "displayMode": "gradient",
- "maxVizHeight": 300,
- "minVizHeight": 10,
- "minVizWidth": 0,
- "namePlacement": "auto",
+ "colorMode": "value",
+ "graphMode": "none",
+ "justifyMode": "center",
"orientation": "horizontal",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
- "fields": "",
+ "fields": "e1",
"values": false
},
- "showUnfilled": true,
- "sizing": "auto",
- "valueMode": "color"
+ "textMode": "value",
+ "wideLayout": true
},
"pluginVersion": "12.4.3",
"targets": [
@@ -1789,16 +1698,18 @@
},
"dimensions": {
"ClusterName": "${env}",
- "ServiceName": "boxel-worker-${env}"
+ "ServiceName": "boxel-prerender-server-${env}"
},
+ "hide": true,
+ "id": "a",
"metricEditorMode": 0,
- "metricName": "CPUUtilization",
+ "metricName": "RunningTaskCount",
"metricQueryType": 0,
- "namespace": "AWS/ECS",
+ "namespace": "ECS/ContainerInsights",
"period": "",
"refId": "A",
"region": "default",
- "statistic": "Average"
+ "statistic": "Maximum"
},
{
"datasource": {
@@ -1807,27 +1718,60 @@
},
"dimensions": {
"ClusterName": "${env}",
- "ServiceName": "boxel-worker-${env}"
+ "ServiceName": "boxel-prerender-server-${env}"
},
+ "hide": true,
+ "id": "b",
"metricEditorMode": 0,
- "metricName": "MemoryUtilization",
+ "metricName": "DesiredTaskCount",
"metricQueryType": 0,
- "namespace": "AWS/ECS",
+ "namespace": "ECS/ContainerInsights",
"period": "",
"refId": "B",
"region": "default",
- "statistic": "Average"
+ "statistic": "Maximum"
+ },
+ {
+ "datasource": {
+ "type": "cloudwatch",
+ "uid": "cef5x9o3yzawwf"
+ },
+ "dimensions": {},
+ "expression": "a + b/100",
+ "id": "e1",
+ "metricEditorMode": 1,
+ "metricName": "",
+ "metricQueryType": 0,
+ "namespace": "",
+ "period": "",
+ "refId": "e1",
+ "region": "default",
+ "statistic": ""
}
],
- "title": "Worker",
- "type": "bargauge"
+ "title": "Tasks",
+ "transformations": [
+ {
+ "id": "convertFieldType",
+ "options": {
+ "conversions": [
+ {
+ "destinationType": "string",
+ "targetField": "Value"
+ }
+ ],
+ "fields": {}
+ }
+ }
+ ],
+ "type": "stat"
},
{
"datasource": {
- "type": "prometheus",
- "uid": "bes7ustjf8w74b"
+ "type": "cloudwatch",
+ "uid": "cef5x9o3yzawwf"
},
- "description": "Synapse process metrics scraped from /\\_synapse/metrics via the synapse-prometheus datasource (local Prometheus locally; AMP in staging/production). CPU is rate over 5m as a percentage of one core; Mem is process_resident_memory_bytes.",
+ "description": "ECS CPU and memory utilisation for `boxel-prerender-manager-${env}` on cluster `${env}` — averages over the panel refresh window. Task counts (Run / Need) appear in the stat row below this bargauge. Locally these show 'No data' — CloudWatch is staging/production only.",
"fieldConfig": {
"defaults": {
"color": {
@@ -1837,8 +1781,8 @@
"links": [
{
"targetBlank": false,
- "title": "Open Synapse dashboard",
- "url": "/d/000000012"
+ "title": "Open Prerender Manager dashboard",
+ "url": "/d/boxel-svc-prerender-manager"
}
],
"mappings": [],
@@ -1904,7 +1848,7 @@
},
{
"id": "unit",
- "value": "decbytes"
+ "value": "percent"
},
{
"id": "thresholds",
@@ -1917,34 +1861,29 @@
{
"color": "green",
"value": 0
+ },
+ {
+ "color": "yellow",
+ "value": 70
+ },
+ {
+ "color": "red",
+ "value": 90
}
]
}
- },
- {
- "id": "max",
- "value": 8589934592
}
]
}
]
},
"gridPos": {
- "h": 7,
+ "h": 4,
"w": 4,
- "x": 16,
+ "x": 8,
"y": 19
},
- "id": 17,
- "links": [
- {
- "icon": "external link",
- "targetBlank": false,
- "title": "Open Synapse dashboard",
- "type": "link",
- "url": "/d/000000012"
- }
- ],
+ "id": 15,
"options": {
"displayMode": "gradient",
"maxVizHeight": 300,
@@ -1961,110 +1900,197 @@
},
"showUnfilled": true,
"sizing": "auto",
- "text": {
- "titleSize": 22,
- "valueSize": 21
- },
"valueMode": "color"
},
"pluginVersion": "12.4.3",
"targets": [
{
"datasource": {
- "type": "prometheus",
- "uid": "bes7ustjf8w74b"
+ "type": "cloudwatch",
+ "uid": "cef5x9o3yzawwf"
},
- "expr": "rate(process_cpu_seconds_total{job=\"synapse\"}[5m]) * 100",
- "instant": true,
- "legendFormat": "CPU",
- "refId": "A"
+ "dimensions": {
+ "ClusterName": "${env}",
+ "ServiceName": "boxel-prerender-manager-${env}"
+ },
+ "metricEditorMode": 0,
+ "metricName": "CPUUtilization",
+ "metricQueryType": 0,
+ "namespace": "AWS/ECS",
+ "period": "",
+ "refId": "A",
+ "region": "default",
+ "statistic": "Average"
},
{
"datasource": {
- "type": "prometheus",
- "uid": "bes7ustjf8w74b"
+ "type": "cloudwatch",
+ "uid": "cef5x9o3yzawwf"
},
- "expr": "process_resident_memory_bytes{job=\"synapse\"}",
- "instant": true,
- "legendFormat": "Mem",
- "refId": "B"
+ "dimensions": {
+ "ClusterName": "${env}",
+ "ServiceName": "boxel-prerender-manager-${env}"
+ },
+ "metricEditorMode": 0,
+ "metricName": "MemoryUtilization",
+ "metricQueryType": 0,
+ "namespace": "AWS/ECS",
+ "period": "",
+ "refId": "B",
+ "region": "default",
+ "statistic": "Average"
}
],
- "title": "Synapse Process",
+ "title": "Prerender Mgr",
"type": "bargauge"
},
{
"datasource": {
- "type": "grafana-postgresql-datasource",
- "uid": "cef5v5sl9k7i8f"
+ "type": "cloudwatch",
+ "uid": "cef5x9o3yzawwf"
},
- "description": "Boxel application database transaction rate (xact_commit + xact_rollback) averaged since the postmaster started. Not a rolling window — recent spikes will be smoothed. Drill into the DB-specific dashboard for detail.",
+ "description": "ECS task count for `boxel-prerender-manager-${env}` — `Run` is RunningTaskCount, `Need` is DesiredTaskCount (both from ECS/ContainerInsights, Maximum over the panel refresh window). Run < Need means the service is under-scaled or tasks are crashing.",
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
- "decimals": 0,
- "links": [
+ "decimals": 2,
+ "mappings": [
{
- "targetBlank": false,
- "title": "Open Database dashboard",
- "url": "/d/boxeldatabase1"
+ "options": {
+ "pattern": "^0\\.0*(\\d+)$",
+ "result": {
+ "color": "red",
+ "text": "0 / $1"
+ }
+ },
+ "type": "regex"
+ },
+ {
+ "options": {
+ "pattern": "^(\\d+)\\.0*(\\d+)$",
+ "result": {
+ "color": "green",
+ "text": "$1 / $2"
+ }
+ },
+ "type": "regex"
}
],
- "mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
- "color": "transparent"
+ "color": "red"
},
{
"color": "green",
- "value": 0
+ "value": 1
}
]
},
- "unit": "short"
+ "unit": "none"
},
"overrides": []
},
"gridPos": {
- "h": 7,
+ "h": 3,
"w": 4,
- "x": 20,
- "y": 19
+ "x": 8,
+ "y": 23
},
- "id": 18,
+ "id": 25,
"options": {
- "colorMode": "background_solid",
+ "colorMode": "value",
"graphMode": "none",
- "justifyMode": "auto",
- "orientation": "vertical",
+ "justifyMode": "center",
+ "orientation": "horizontal",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
- "fields": "",
+ "fields": "e1",
"values": false
},
- "textMode": "value_and_name"
+ "textMode": "value",
+ "wideLayout": true
},
"pluginVersion": "12.4.3",
"targets": [
{
"datasource": {
- "type": "grafana-postgresql-datasource",
- "uid": "cef5v5sl9k7i8f"
+ "type": "cloudwatch",
+ "uid": "cef5x9o3yzawwf"
},
- "editorMode": "code",
- "format": "table",
- "rawQuery": true,
- "rawSql": "SELECT 60.0 * (xact_commit + xact_rollback) / NULLIF(EXTRACT(EPOCH FROM (NOW() - pg_postmaster_start_time())), 0) AS \"txn/min\" FROM pg_stat_database WHERE datname = 'boxel';",
- "refId": "A"
+ "dimensions": {
+ "ClusterName": "${env}",
+ "ServiceName": "boxel-prerender-manager-${env}"
+ },
+ "hide": true,
+ "id": "a",
+ "metricEditorMode": 0,
+ "metricName": "RunningTaskCount",
+ "metricQueryType": 0,
+ "namespace": "ECS/ContainerInsights",
+ "period": "",
+ "refId": "A",
+ "region": "default",
+ "statistic": "Maximum"
+ },
+ {
+ "datasource": {
+ "type": "cloudwatch",
+ "uid": "cef5x9o3yzawwf"
+ },
+ "dimensions": {
+ "ClusterName": "${env}",
+ "ServiceName": "boxel-prerender-manager-${env}"
+ },
+ "hide": true,
+ "id": "b",
+ "metricEditorMode": 0,
+ "metricName": "DesiredTaskCount",
+ "metricQueryType": 0,
+ "namespace": "ECS/ContainerInsights",
+ "period": "",
+ "refId": "B",
+ "region": "default",
+ "statistic": "Maximum"
+ },
+ {
+ "datasource": {
+ "type": "cloudwatch",
+ "uid": "cef5x9o3yzawwf"
+ },
+ "dimensions": {},
+ "expression": "a + b/100",
+ "id": "e1",
+ "metricEditorMode": 1,
+ "metricName": "",
+ "metricQueryType": 0,
+ "namespace": "",
+ "period": "",
+ "refId": "e1",
+ "region": "default",
+ "statistic": ""
+ }
+ ],
+ "title": "Tasks",
+ "transformations": [
+ {
+ "id": "convertFieldType",
+ "options": {
+ "conversions": [
+ {
+ "destinationType": "string",
+ "targetField": "Value"
+ }
+ ],
+ "fields": {}
+ }
}
],
- "title": "Postgres DB",
"type": "stat"
},
{
@@ -2072,72 +2098,136 @@
"type": "cloudwatch",
"uid": "cef5x9o3yzawwf"
},
- "description": "ECS task count for `boxel-realm-server-${env}` — `Run` is RunningTaskCount, `Need` is DesiredTaskCount (both from ECS/ContainerInsights, Maximum over the panel refresh window). Run < Need means the service is under-scaled or tasks are crashing.",
+ "description": "ECS CPU and memory utilisation for `boxel-worker-${env}` on cluster `${env}` — averages over the panel refresh window. Task counts (Run / Need) appear in the stat row below this bargauge. Locally these show 'No data' — CloudWatch is staging/production only.",
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
- "decimals": 2,
- "mappings": [
- {
- "options": {
- "pattern": "^0\\.0*(\\d+)$",
- "result": {
- "color": "red",
- "text": "0 / $1"
- }
- },
- "type": "regex"
- },
+ "decimals": 1,
+ "links": [
{
- "options": {
- "pattern": "^(\\d+)\\.0*(\\d+)$",
- "result": {
- "color": "green",
- "text": "$1 / $2"
- }
- },
- "type": "regex"
+ "targetBlank": false,
+ "title": "Open Worker dashboard",
+ "url": "/d/boxel-svc-worker"
}
],
+ "mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
- "color": "red"
+ "color": "transparent"
+ }
+ ]
+ },
+ "unit": "short"
+ },
+ "overrides": [
+ {
+ "matcher": {
+ "id": "byFrameRefID",
+ "options": "A"
+ },
+ "properties": [
+ {
+ "id": "displayName",
+ "value": "CPU"
},
{
- "color": "green",
- "value": 1
+ "id": "unit",
+ "value": "percent"
+ },
+ {
+ "id": "thresholds",
+ "value": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "transparent"
+ },
+ {
+ "color": "green",
+ "value": 0
+ },
+ {
+ "color": "yellow",
+ "value": 70
+ },
+ {
+ "color": "red",
+ "value": 90
+ }
+ ]
+ }
}
]
},
- "unit": "none"
- },
- "overrides": []
+ {
+ "matcher": {
+ "id": "byFrameRefID",
+ "options": "B"
+ },
+ "properties": [
+ {
+ "id": "displayName",
+ "value": "Mem"
+ },
+ {
+ "id": "unit",
+ "value": "percent"
+ },
+ {
+ "id": "thresholds",
+ "value": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "transparent"
+ },
+ {
+ "color": "green",
+ "value": 0
+ },
+ {
+ "color": "yellow",
+ "value": 70
+ },
+ {
+ "color": "red",
+ "value": 90
+ }
+ ]
+ }
+ }
+ ]
+ }
+ ]
},
"gridPos": {
- "h": 3,
+ "h": 4,
"w": 4,
- "x": 0,
- "y": 23
+ "x": 12,
+ "y": 19
},
- "id": 23,
+ "id": 16,
"options": {
- "colorMode": "value",
- "graphMode": "none",
- "justifyMode": "center",
+ "displayMode": "gradient",
+ "maxVizHeight": 300,
+ "minVizHeight": 10,
+ "minVizWidth": 0,
+ "namePlacement": "auto",
"orientation": "horizontal",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
- "fields": "e1",
+ "fields": "",
"values": false
- },
- "textMode": "value",
- "wideLayout": true
+ },
+ "showUnfilled": true,
+ "sizing": "auto",
+ "valueMode": "color"
},
"pluginVersion": "12.4.3",
"targets": [
@@ -2148,18 +2238,16 @@
},
"dimensions": {
"ClusterName": "${env}",
- "ServiceName": "boxel-realm-server-${env}"
+ "ServiceName": "boxel-worker-${env}"
},
- "hide": true,
- "id": "a",
"metricEditorMode": 0,
- "metricName": "RunningTaskCount",
+ "metricName": "CPUUtilization",
"metricQueryType": 0,
- "namespace": "ECS/ContainerInsights",
+ "namespace": "AWS/ECS",
"period": "",
"refId": "A",
"region": "default",
- "statistic": "Maximum"
+ "statistic": "Average"
},
{
"datasource": {
@@ -2168,60 +2256,27 @@
},
"dimensions": {
"ClusterName": "${env}",
- "ServiceName": "boxel-realm-server-${env}"
+ "ServiceName": "boxel-worker-${env}"
},
- "hide": true,
- "id": "b",
"metricEditorMode": 0,
- "metricName": "DesiredTaskCount",
+ "metricName": "MemoryUtilization",
"metricQueryType": 0,
- "namespace": "ECS/ContainerInsights",
+ "namespace": "AWS/ECS",
"period": "",
"refId": "B",
"region": "default",
- "statistic": "Maximum"
- },
- {
- "datasource": {
- "type": "cloudwatch",
- "uid": "cef5x9o3yzawwf"
- },
- "dimensions": {},
- "expression": "a + b/100",
- "id": "e1",
- "metricEditorMode": 1,
- "metricName": "",
- "metricQueryType": 0,
- "namespace": "",
- "period": "",
- "refId": "e1",
- "region": "default",
- "statistic": ""
- }
- ],
- "title": "Tasks",
- "transformations": [
- {
- "id": "convertFieldType",
- "options": {
- "conversions": [
- {
- "destinationType": "string",
- "targetField": "Value"
- }
- ],
- "fields": {}
- }
+ "statistic": "Average"
}
],
- "type": "stat"
+ "title": "Worker",
+ "type": "bargauge"
},
{
"datasource": {
"type": "cloudwatch",
"uid": "cef5x9o3yzawwf"
},
- "description": "ECS task count for `boxel-prerender-server-${env}` — `Run` is RunningTaskCount, `Need` is DesiredTaskCount (both from ECS/ContainerInsights, Maximum over the panel refresh window). Run < Need means the service is under-scaled or tasks are crashing.",
+ "description": "ECS task count for `boxel-worker-${env}` — `Run` is RunningTaskCount, `Need` is DesiredTaskCount (both from ECS/ContainerInsights, Maximum over the panel refresh window). Run < Need means the service is under-scaled or tasks are crashing.",
"fieldConfig": {
"defaults": {
"color": {
@@ -2269,10 +2324,10 @@
"gridPos": {
"h": 3,
"w": 4,
- "x": 4,
+ "x": 12,
"y": 23
},
- "id": 24,
+ "id": 26,
"options": {
"colorMode": "value",
"graphMode": "none",
@@ -2297,7 +2352,7 @@
},
"dimensions": {
"ClusterName": "${env}",
- "ServiceName": "boxel-prerender-server-${env}"
+ "ServiceName": "boxel-worker-${env}"
},
"hide": true,
"id": "a",
@@ -2317,7 +2372,7 @@
},
"dimensions": {
"ClusterName": "${env}",
- "ServiceName": "boxel-prerender-server-${env}"
+ "ServiceName": "boxel-worker-${env}"
},
"hide": true,
"id": "b",
@@ -2367,302 +2422,278 @@
},
{
"datasource": {
- "type": "cloudwatch",
- "uid": "cef5x9o3yzawwf"
+ "type": "prometheus",
+ "uid": "bes7ustjf8w74b"
},
- "description": "ECS task count for `boxel-prerender-manager-${env}` — `Run` is RunningTaskCount, `Need` is DesiredTaskCount (both from ECS/ContainerInsights, Maximum over the panel refresh window). Run < Need means the service is under-scaled or tasks are crashing.",
+ "description": "Synapse process metrics scraped from /\\_synapse/metrics via the synapse-prometheus datasource (local Prometheus locally; AMP in staging/production). CPU is rate over 5m as a percentage of one core; Mem is process_resident_memory_bytes.",
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
- "decimals": 2,
- "mappings": [
- {
- "options": {
- "pattern": "^0\\.0*(\\d+)$",
- "result": {
- "color": "red",
- "text": "0 / $1"
- }
- },
- "type": "regex"
- },
+ "decimals": 1,
+ "links": [
{
- "options": {
- "pattern": "^(\\d+)\\.0*(\\d+)$",
- "result": {
- "color": "green",
- "text": "$1 / $2"
- }
- },
- "type": "regex"
+ "targetBlank": false,
+ "title": "Open Synapse dashboard",
+ "url": "/d/000000012"
}
],
+ "mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
- "color": "red"
+ "color": "transparent"
+ }
+ ]
+ },
+ "unit": "short"
+ },
+ "overrides": [
+ {
+ "matcher": {
+ "id": "byFrameRefID",
+ "options": "A"
+ },
+ "properties": [
+ {
+ "id": "displayName",
+ "value": "CPU"
},
{
- "color": "green",
- "value": 1
+ "id": "unit",
+ "value": "percent"
+ },
+ {
+ "id": "thresholds",
+ "value": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "transparent"
+ },
+ {
+ "color": "green",
+ "value": 0
+ },
+ {
+ "color": "yellow",
+ "value": 70
+ },
+ {
+ "color": "red",
+ "value": 90
+ }
+ ]
+ }
}
]
},
- "unit": "none"
- },
- "overrides": []
+ {
+ "matcher": {
+ "id": "byFrameRefID",
+ "options": "B"
+ },
+ "properties": [
+ {
+ "id": "displayName",
+ "value": "Mem"
+ },
+ {
+ "id": "unit",
+ "value": "decbytes"
+ },
+ {
+ "id": "thresholds",
+ "value": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "transparent"
+ },
+ {
+ "color": "green",
+ "value": 0
+ }
+ ]
+ }
+ },
+ {
+ "id": "max",
+ "value": 8589934592
+ }
+ ]
+ }
+ ]
},
"gridPos": {
- "h": 3,
+ "h": 7,
"w": 4,
- "x": 8,
- "y": 23
+ "x": 16,
+ "y": 19
},
- "id": 25,
+ "id": 17,
+ "links": [
+ {
+ "icon": "external link",
+ "targetBlank": false,
+ "title": "Open Synapse dashboard",
+ "type": "link",
+ "url": "/d/000000012"
+ }
+ ],
"options": {
- "colorMode": "value",
- "graphMode": "none",
- "justifyMode": "center",
+ "displayMode": "gradient",
+ "maxVizHeight": 300,
+ "minVizHeight": 10,
+ "minVizWidth": 0,
+ "namePlacement": "auto",
"orientation": "horizontal",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
- "fields": "e1",
+ "fields": "",
"values": false
},
- "textMode": "value",
- "wideLayout": true
- },
- "pluginVersion": "12.4.3",
- "targets": [
- {
- "datasource": {
- "type": "cloudwatch",
- "uid": "cef5x9o3yzawwf"
- },
- "dimensions": {
- "ClusterName": "${env}",
- "ServiceName": "boxel-prerender-manager-${env}"
- },
- "hide": true,
- "id": "a",
- "metricEditorMode": 0,
- "metricName": "RunningTaskCount",
- "metricQueryType": 0,
- "namespace": "ECS/ContainerInsights",
- "period": "",
- "refId": "A",
- "region": "default",
- "statistic": "Maximum"
- },
- {
- "datasource": {
- "type": "cloudwatch",
- "uid": "cef5x9o3yzawwf"
- },
- "dimensions": {
- "ClusterName": "${env}",
- "ServiceName": "boxel-prerender-manager-${env}"
- },
- "hide": true,
- "id": "b",
- "metricEditorMode": 0,
- "metricName": "DesiredTaskCount",
- "metricQueryType": 0,
- "namespace": "ECS/ContainerInsights",
- "period": "",
- "refId": "B",
- "region": "default",
- "statistic": "Maximum"
- },
+ "showUnfilled": true,
+ "sizing": "auto",
+ "text": {
+ "titleSize": 22,
+ "valueSize": 21
+ },
+ "valueMode": "color"
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
{
"datasource": {
- "type": "cloudwatch",
- "uid": "cef5x9o3yzawwf"
+ "type": "prometheus",
+ "uid": "bes7ustjf8w74b"
},
- "dimensions": {},
- "expression": "a + b/100",
- "id": "e1",
- "metricEditorMode": 1,
- "metricName": "",
- "metricQueryType": 0,
- "namespace": "",
- "period": "",
- "refId": "e1",
- "region": "default",
- "statistic": ""
- }
- ],
- "title": "Tasks",
- "transformations": [
+ "expr": "rate(process_cpu_seconds_total{job=\"synapse\"}[5m]) * 100",
+ "instant": true,
+ "legendFormat": "CPU",
+ "refId": "A"
+ },
{
- "id": "convertFieldType",
- "options": {
- "conversions": [
- {
- "destinationType": "string",
- "targetField": "Value"
- }
- ],
- "fields": {}
- }
+ "datasource": {
+ "type": "prometheus",
+ "uid": "bes7ustjf8w74b"
+ },
+ "expr": "process_resident_memory_bytes{job=\"synapse\"}",
+ "instant": true,
+ "legendFormat": "Mem",
+ "refId": "B"
}
],
- "type": "stat"
+ "title": "Synapse Process",
+ "type": "bargauge"
},
{
"datasource": {
- "type": "cloudwatch",
- "uid": "cef5x9o3yzawwf"
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
},
- "description": "ECS task count for `boxel-worker-${env}` — `Run` is RunningTaskCount, `Need` is DesiredTaskCount (both from ECS/ContainerInsights, Maximum over the panel refresh window). Run < Need means the service is under-scaled or tasks are crashing.",
+ "description": "Boxel application database transaction rate (xact_commit + xact_rollback) averaged since the postmaster started. Not a rolling window — recent spikes will be smoothed. Drill into the DB-specific dashboard for detail.",
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
- "decimals": 2,
- "mappings": [
- {
- "options": {
- "pattern": "^0\\.0*(\\d+)$",
- "result": {
- "color": "red",
- "text": "0 / $1"
- }
- },
- "type": "regex"
- },
+ "decimals": 0,
+ "links": [
{
- "options": {
- "pattern": "^(\\d+)\\.0*(\\d+)$",
- "result": {
- "color": "green",
- "text": "$1 / $2"
- }
- },
- "type": "regex"
+ "targetBlank": false,
+ "title": "Open Database dashboard",
+ "url": "/d/boxeldatabase1"
}
],
+ "mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
- "color": "red"
+ "color": "transparent"
},
{
"color": "green",
- "value": 1
+ "value": 0
}
]
},
- "unit": "none"
+ "unit": "short"
},
"overrides": []
},
"gridPos": {
- "h": 3,
+ "h": 7,
"w": 4,
- "x": 12,
- "y": 23
+ "x": 20,
+ "y": 19
},
- "id": 26,
+ "id": 18,
"options": {
- "colorMode": "value",
+ "colorMode": "background_solid",
"graphMode": "none",
- "justifyMode": "center",
- "orientation": "horizontal",
+ "justifyMode": "auto",
+ "orientation": "vertical",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
- "fields": "e1",
+ "fields": "",
"values": false
},
- "textMode": "value",
- "wideLayout": true
+ "textMode": "value_and_name"
},
"pluginVersion": "12.4.3",
"targets": [
{
"datasource": {
- "type": "cloudwatch",
- "uid": "cef5x9o3yzawwf"
- },
- "dimensions": {
- "ClusterName": "${env}",
- "ServiceName": "boxel-worker-${env}"
- },
- "hide": true,
- "id": "a",
- "metricEditorMode": 0,
- "metricName": "RunningTaskCount",
- "metricQueryType": 0,
- "namespace": "ECS/ContainerInsights",
- "period": "",
- "refId": "A",
- "region": "default",
- "statistic": "Maximum"
- },
- {
- "datasource": {
- "type": "cloudwatch",
- "uid": "cef5x9o3yzawwf"
- },
- "dimensions": {
- "ClusterName": "${env}",
- "ServiceName": "boxel-worker-${env}"
- },
- "hide": true,
- "id": "b",
- "metricEditorMode": 0,
- "metricName": "DesiredTaskCount",
- "metricQueryType": 0,
- "namespace": "ECS/ContainerInsights",
- "period": "",
- "refId": "B",
- "region": "default",
- "statistic": "Maximum"
- },
- {
- "datasource": {
- "type": "cloudwatch",
- "uid": "cef5x9o3yzawwf"
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
},
- "dimensions": {},
- "expression": "a + b/100",
- "id": "e1",
- "metricEditorMode": 1,
- "metricName": "",
- "metricQueryType": 0,
- "namespace": "",
- "period": "",
- "refId": "e1",
- "region": "default",
- "statistic": ""
- }
- ],
- "title": "Tasks",
- "transformations": [
- {
- "id": "convertFieldType",
- "options": {
- "conversions": [
- {
- "destinationType": "string",
- "targetField": "Value"
- }
- ],
- "fields": {}
- }
+ "editorMode": "code",
+ "format": "table",
+ "rawQuery": true,
+ "rawSql": "SELECT 60.0 * (xact_commit + xact_rollback) / NULLIF(EXTRACT(EPOCH FROM (NOW() - pg_postmaster_start_time())), 0) AS \"txn/min\" FROM pg_stat_database WHERE datname = 'boxel';",
+ "refId": "A"
}
],
+ "title": "Postgres DB",
"type": "stat"
},
+ {
+ "description": "All firing and pending Grafana alert rules. Drill into a rule for details and silences. Replaces the Worker Status alertlist (which was a single-panel dashboard).",
+ "gridPos": {
+ "h": 8,
+ "w": 24,
+ "x": 0,
+ "y": 36
+ },
+ "id": 6,
+ "options": {
+ "alertInstanceLabelFilter": "",
+ "alertName": "",
+ "dashboardAlerts": false,
+ "groupBy": [],
+ "groupMode": "default",
+ "maxItems": 30,
+ "sortOrder": 1,
+ "stateFilter": {
+ "error": true,
+ "firing": true,
+ "noData": true,
+ "normal": false,
+ "pending": true
+ },
+ "viewMode": "list"
+ },
+ "title": "Active Alerts (preview smoke test)",
+ "type": "alertlist"
+ },
{
"datasource": {
"type": "grafana-postgresql-datasource",
@@ -2698,8 +2729,7 @@
}
]
}
- },
- "overrides": []
+ }
},
"gridPos": {
"h": 4,
@@ -2766,8 +2796,7 @@
}
]
}
- },
- "overrides": []
+ }
},
"gridPos": {
"h": 4,
@@ -2843,8 +2872,7 @@
]
},
"unit": "s"
- },
- "overrides": []
+ }
},
"gridPos": {
"h": 4,
@@ -2915,8 +2943,7 @@
}
]
}
- },
- "overrides": []
+ }
},
"gridPos": {
"h": 4,
@@ -3119,35 +3146,6 @@
"title": "Indexing throughput",
"type": "timeseries"
},
- {
- "description": "All firing and pending Grafana alert rules. Drill into a rule for details and silences. Replaces the Worker Status alertlist (which was a single-panel dashboard).",
- "gridPos": {
- "h": 8,
- "w": 24,
- "x": 0,
- "y": 36
- },
- "id": 6,
- "options": {
- "alertInstanceLabelFilter": "",
- "alertName": "",
- "dashboardAlerts": false,
- "groupBy": [],
- "groupMode": "default",
- "maxItems": 30,
- "sortOrder": 1,
- "stateFilter": {
- "error": true,
- "firing": true,
- "noData": true,
- "normal": false,
- "pending": true
- },
- "viewMode": "list"
- },
- "title": "Active Alerts",
- "type": "alertlist"
- },
{
"gridPos": {
"h": 8,
@@ -3180,7 +3178,7 @@
{
"hide": 2,
"name": "env",
- "query": "staging",
+ "query": "__ENV__",
"skipUrlSync": true,
"type": "constant"
},
(Run: https://github.com/cardstack/boxel/actions/runs/39978134450) |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Do not merge this PR. It exercises the observability PR preview end to end now that #6172 has merged and the scoped Grafana account is applied in both environments. Close it after the check.
#6172 itself changed no dashboard JSON, so its own runs never produced a preview. This changes two panel titles so one is produced for real.
The change
Two panel titles, one line each:
boxel-status/overview.jsonActive Alerts (preview smoke test)boxel-status/job-queue.jsonPending (preview smoke test)Both dashboards live in the same folder, so the preview should produce one per-PR folder holding two dashboards. The titles name the test, so the change cannot be merged by mistake.
What to check
This is the first production preview, so the production half is what's new. Staging has already been exercised end to end by hand.
staginglink and aproductionlink, and both open a dashboard carrying the new panel title.Previewsin both Grafanas — this is what the scoped credential depends on, and the thing most worth eyeballing.GRAFANA_SECRET. Don't click one on the production preview; it performs the production mutation.Previewsfolder survives.Background
The credential this runs under is
observability-preview: Viewer, plus Admin on thePreviewsfolder and nothing else. It cannot overwrite or delete a canonical dashboard — measured at 403 in both environments. It can still read every dashboard, and so still readGRAFANA_SECRET; CS-14263 closes that.Infra: cardstack/infra#747, applied in both environments.
🤖 Generated with Claude Code
https://claude.ai/code/session_01T6hzc2RPYWnjG8xPbmYhf4