[backspace]

Depends on cardstack/infra#747. Apply that first — this branch points the preview workflow at a role that does not exist yet, so its credential step fails until then.

What this changes

The PR preview pushed the changed dashboards to the staging Grafana only. It now pushes them to the production Grafana also, and the PR gets one comment rather than one for each environment:

Grafana preview

Preview deployed for 2 dashboards in staging and production.

• Folder: Preview – PR Type updates #123: Boxel Status — staging · production

Dashboards:

• [PR Type updates #123] Job Queue — staging · production
• [PR Type updates #123] Overview — staging · production

The credential question

This workflow runs on pull_request. GitHub loads it, and every script it calls, from the pull request itself. A contributor controls the code that runs after the job assumes the role. Whatever that role can read, a pull request can read.

So the preview uses a dedicated role, boxel-observability-preview, granting three parameters per environment:

/<env>/grafana/grafanactl_token        authenticate grafanactl
/<env>/boxel-grafana/realm_server_url  fill the realm_server template variable
/<env>/boxel/GRAFANA_SECRET            make the operator buttons work

Not boxel-observability-apply, which also grants the Grafana database credentials.

GRAFANA_SECRET is in the list deliberately. A preview whose operator buttons return 401 cannot show a reviewer whether a button change works, and the token is already readable by every Grafana viewer: apply.sh substitutes it into six canonical dashboards as a constant template variable, and hide=2 hides it from the variable bar rather than from the dashboard JSON. Withholding it from CI protects nothing that is not already exposed. The durable fix is to stop embedding it and serve operator calls through a Grafana datasource proxy; this grant should be dropped once that lands.

Caution for reviewers: preview buttons act on the real environment. A button clicked on a production preview performs the production mutation. Prefer the staging preview for exercising one.

A preview still cannot address a canonical dashboard: render-preview.sh rewrites every UID and then fails closed if any manifest lacks the pr<n>- prefix.

Tests

• actionlint, prettier --check and shellcheck pass.
• The comment script was run against stub github, context and core objects over four conditions: both environments, production failed, no dashboard changed, and both failed.
• render-preview.sh against a real five-dashboard change substitutes the supplied operator token and realm-server URL, and emits pr999- UIDs for every dashboard and the folder.
• Omitting the token now fails the render with exit 1, rather than silently producing a dead preview.
• Removing the prefix from prefix_uid fails the render with exit 1, so the UID assertion is not vacuous.

Still open

This branch is behind main. I have not merged main in, because that merge touches files carrying uncommitted local work.

🤖 Generated with Claude Code

https://claude.ai/code/session_01T6hzc2RPYWnjG8xPbmYhf4

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-17T09:22:38.726531Z	a57d7b1	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a57d7b1649

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[github-actions]

Host Test Results

    1 files  ±0      1 suites  ±0   1h 41m 0s ⏱️ - 11m 47s
5 905 tests +5  5 894 ✅ +5  11 💤 ±0  0 ❌ ±0 
5 920 runs  +5  5 909 ✅ +5  11 💤 ±0  0 ❌ ±0 

Results for commit dbf918e. ± Comparison against earlier commit e11a2cd.

Realm Server Test Results

    1 files  ±0    362 suites  ±0   1h 56m 4s ⏱️ - 11m 1s
6 024 tests ±0  6 024 ✅ ±0  0 💤 ±0  0 ❌ ±0 
6 087 runs  ±0  6 087 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit dbf918e. ± Comparison against earlier commit e11a2cd.

[habdelra]

[Claude Code 🤖]

This review went after the trust boundary the workflow_run split rests on, the teardown paths, and the load-bearing comments the two new workflows introduce. I traced the overlay step's effect on the working tree, the resolve-the-PR lookup across merged / closed-unmerged, and the pr<n>- assertion against the rendered tree; I did not exercise either workflow end to end, and the dashboard manifests themselves are unreviewed.

The split is the right shape and the pr<n>- assertion holds — it runs over every JSON in the rendered tree, not just the changed dashboards, so a manifest added under a new resources/ subdirectory also fails closed. Three asks below; none block, but two are comments that assert more than their mechanism delivers, which is the kind of thing the next author builds on.

On your question about GRAFANA_SECRET. Substitute a dummy for the production leg. The reason isn't secret exposure — canonical production dashboards already carry the real value in a constant template variable, so anyone with production Grafana access can read it today. It's that a preview dashboard is a PR-authored artifact whose operator buttons reindex realms, complete jobs, grant realm permissions and add credit against production, with nothing in the panel saying so; the confirm dialogs read identically to the canonical dashboard's. A reviewer opening a preview to check a layout change is one click from a production mutation. A dummy still renders every panel and still runs the customCode path — the button just comes back 401, which is the visible, safe outcome. Staging keeps the real secret, so an operator button added in a PR can still be exercised for real, one environment over. Mechanically this is a branch in ci-apply-preview.sh: skip the /production/boxel/GRAFANA_SECRET read and export a placeholder when ENV_NAME=production, which also drops one SSM path from what the production role needs.

Recommendations:

1. The overlay leaves the pull request's version of everything outside .github and packages/observability on disk; the header claims only resources is taken. A --head-ref option on render-preview.sh removes the class — see the thread on the Overlay step.
2. A pull request closed without merging looks like it resolves to found=false, leaving the production preview for the daily sweep — see the thread on the Resolve step; the README's "deleted when the PR closes" wants softening either way.
3. The sweep's non-main guard can't stop the threat its comment names, because workflow_dispatch runs the chosen ref's copy of the guard — see the thread on that step.

Adjacent, out of scope: prefix_uid's right-truncation at 40 characters can map two distinct canonical UIDs onto one preview UID, silently dropping a dashboard from a preview that changes both. Pre-existing, and the new assertion doesn't see it because the truncated UID still carries the prefix. Worth a collision check whenever someone next touches that function.

---

Generated by Claude Code

[richardhjtan]

[Claude Code 🤖] Went after the trust boundary from the other side than the review above: not what the overlay leaves outside the two restored trees, but what a pull request can leave inside them. Also swept the production job for the usual workflow_run holes. No end-to-end run.

One finding below; the rest of what I pushed on holds. Nothing after the overlay resolves uses: ./ — install-grafanactl runs while only default-branch code is on disk, and the two later actions are SHA-pinned externals. ci-apply-preview.sh and its callees are bash, aws and grafanactl only, invoked by explicit relative path: no mise, no pnpm, no node, so the PR-controlled repo root (package.json, .mise.toml, lockfile) is never read. Every PR-derived value reaches a run: block through env: rather than ${{ }} interpolation, so there is no template-injection path through the branch name or base ref.

On approving: I am not adding one, and the reason is the open thread above rather than anything I found. A preview dashboard is a PR-authored artifact that currently receives the real production GRAFANA_SECRET, so its operator buttons reindex realms and grant permissions against production behind confirm dialogs identical to the canonical dashboard's. That wants a decision before this merges, and it is yours rather than mine.

[richardhjtan]

[Claude Code 🤖] Re-review of e25f927 only: what the narrower-role design really takes away from the pull request, and where the earlier threads now stand. I didn't do an end-to-end run.

One blocking finding. The preview role's Grafana token isn't scoped to previews, so it can still reach canonical production dashboards and GRAFANA_SECRET. Details are in the inline comment on the env: block in observability-preview.yml.

Earlier threads:

• Overlay leaves PR-added files inside .github / packages/observability: resolved (the workflow is gone). I replied in that thread.
• Overlay leaves the PR's repo root outside the restored trees: resolved (same reason).
• commits/{sha}/pulls misses closed-without-merge PRs: resolved. Cleanup now reads the PR number from the pull_request payload, so the README's "both previews are deleted when the PR closes" holds.
• Sweep's "refuse non-main ref" guard justification: still open. The comment still says it stops a ref "whose copy of this workflow was rewritten", and the diff didn't touch that comment.
• Operator buttons on a preview acting against production: resolved as far as the preview dashboards go (placeholder secret, 401). The secret's other route in is the new finding.

[richardhjtan]

[Claude Code 🤖] Re-review of the operator-secret change: what the README and the PR description now promise next to what the preview actually carries. I did not do an end-to-end run.

Two blocking findings. The README still describes inert previews, and the deferred fix has no ticket. The grafanactl-token thread on the role's env: block is resolved: the false "worst case" claim is gone, and since main needs no approval to merge, the PR doesn't add reach.

1. packages/observability/README.md still says the role grants two parameters and that preview buttons return 401. See the inline comment. Regression, blocking.
2. The description defers the durable fix: stop embedding GRAFANA_SECRET, proxy operator calls through Grafana, then drop this grant. That needs a Linear ticket linked from the description before merge (AGENTS.md, PR Instructions). Blocking.

[backspace]

[Claude Code 🤖] Correcting the record on the Codex thread (#discussion_r4035160101) — GitHub will not accept a reply there because the anchor is outdated, and my two replies in it each describe a design that was since replaced.

What shipped is neither of them. The exfiltration path Codex identified is closed by narrowing what the credential reaches, not by fencing the code:

• The preview assumes a dedicated IAM role granting three SSM parameters. The Grafana one is a dedicated service account, observability-preview, which is Viewer plus Admin on a Previews folder and nothing else. Per-PR folders nest inside that parent.
• Measured in both environments with that credential alone: writing into the canonical folder returns 403, writing into General returns 403, and a full apply/cleanup cycle succeeds entirely inside Previews with the canonical dashboards untouched.

Two corrections to what I wrote in that thread.

I said the role would not grant GRAFANA_SECRET. It does. A preview whose operator buttons return 401 cannot show a reviewer whether a button change works, and the token is readable by any Grafana viewer regardless — apply.sh substitutes it into six canonical dashboards as a constant variable, where hide=2 hides it from the variable bar but not from the JSON model or the API.

I also said the worst case was "write pr<n>- dashboards". That was wrong, for a reason the Codex thread never reached: the Grafana token then in use was the Admin one, which no folder scope constrained, and which could read GRAFANA_SECRET straight back out of the canonical dashboards whatever the SSM grant said. A later review caught it.

Residual, stated plainly: a pull request can still read every dashboard, and so can still read GRAFANA_SECRET. It cannot overwrite or delete a canonical dashboard — that was the irreversible half, and it is closed. CS-14263 closes the rest by moving operator calls behind a Grafana datasource proxy, after which the secret leaves dashboard bodies and this grant can be dropped.