[lukemelia]

Background and Goal

We deploy Boxel on AWS, but realm servers are meant to federate (#6795, ADR 004). People outside our organization will host their own realm servers, which work with ours, and they may not use AWS. They run the same code we do, so the runtime must not depend on one provider. ADR 003 sets the rules that make running a realm server elsewhere a matter of writing adapters and changing configuration. This PR also brings the runtime into line with those rules.

Builds on the ADR scaffolding from #6755, now in main. It is numbered 003 because #6758 holds 002 and #6795 holds 004.

Where to start

• docs/adr/003-reach-cloud-services-through-adapters.md: §1 holds the rules. It ends with a Remediation section that lists no gaps.
• eslint/cloud-provider-sdk-imports.cjs: the lint rule's settings, the packages it covers, and its allowlist of adapter modules. The rule itself is @cardstack/boxel/no-cloud-provider-sdk.
• packages/realm-server/lib/s3-client.ts and lib/instance-id.ts: the two new seams.

Key decisions

• Scope is runtime code only. Infrastructure-as-code, deploy configuration (the scripts/start-*.sh the deployed tasks run, ember-cli-deploy-s3/cloudfront), CI workflows and ops runbooks are provider-specific by nature and are left out.
• Proprietary services go behind capability-named interfaces, as SecretsKeyProvider and MediaCacheAdapter already do. Each interface has a local adapter for dev and CI.
• Open protocols are the boundary, and the S3 API counts as one. Every S3 client now comes from lib/s3-client.ts. S3_ENDPOINT and S3_FORCE_PATH_STYLE point it at R2, MinIO or GCS interop. Unset, it reaches AWS as before.
• Loading a provider SDK is confined by a lint rule. @cardstack/boxel/no-cloud-provider-sdk reports static imports, re-exports, dynamic import(), require() and import x = require(), which no-restricted-imports can't all see. It covers every runtime package, including base, bxl and boxel-ui. Four modules are allowlisted: the KMS provider, the S3 media-cache adapter, the prerender artifact sink, and s3-client.ts. ai-bot and host have root: true configs and boxel-ui has a flat config, so each declares the rule again.
• The health check stays on /, and the load balancer's user agent moves into configuration. healthCheck answers the prefixes in HEALTH_CHECK_USER_AGENTS, and the deployed start scripts set it to ELB-HealthChecker. The ALB probe is unchanged, so this needs no infra change. A deployed server with the setting unset logs a warning at boot. A dedicated health path was rejected because it would force the app and the infra to change in a set order (ADR §3).
• Instance identity no longer comes from ECS. getInstanceId() is a base name plus a per-boot suffix. The base name is BOXEL_INSTANCE_ID if set, otherwise the hostname, and the suffix is always added, because the orphan cleanup closes every open reservation under a worker id, so ids must not repeat across replicas or restarts. Worker ids use it. The worker manager passes its whole id to its workers in BOXEL_INHERITED_INSTANCE_ID, so the worker id in job_reservations and in the [worker <id> …] log prefix still match.
• A new X-Boxel-Instance-Id response header carries a 12-hex digest of the instance id, replacing X-ECS-Container-Metadata-URI-v4. It tells replicas apart and changes on restart without publishing the hostname, which on Fargate spells out the task's private IP.
• The indexing dashboard is opt-in. The worker manager serves it only under --indexingDashboard, which the development worker task passes, instead of on the absence of ECS metadata. Inferring development from the realm URLs would count a self-hosted deployment on private addresses as local.

Operational changes after deploy

• Deployed worker ids change from <ECS task id>-<runtime id>-pid-N to <hostname>-<suffix>-pid-N. On Fargate the hostname is the task's private-IP DNS name. The Loki worker_id label follows automatically, because the Fluent Bit filter matches any non-space id. job_reservations rows written before the deploy keep the old form. The Grafana job panels label new ids by suffix and pid, and any other id in full.
• The load harness reads X-Boxel-Instance-Id. Against a deployment that predates this PR, it sees no replica ids, so it compares builds only.

Tests

New pure tests cover s3ClientConfig, healthCheck, getInstanceId (in fresh processes) and instanceIdHeader, and the lint rule has its own tests in eslint-plugin-boxel. They pass along with the existing media-cache adapter, artifact-sink and load-harness tests, run directly under QUnit. No suite that boots a realm-server was run locally. Per-client S3 endpoints are left to a follow-up.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ABJLVRj5d5YngGbpKqi4MP

[github-actions]

Preview deployments

• Host staging preview

• Host production preview

• Boxel UI staging preview

Host Test Results

    1 files  ± 0      1 suites  ±0   2h 11m 37s ⏱️ + 41m 24s
5 809 tests +23  5 798 ✅ +23  11 💤 ±0  0 ❌ ±0 
5 824 runs  +23  5 813 ✅ +23  11 💤 ±0  0 ❌ ±0 

Results for commit 0bc2d97. ± Comparison against earlier commit b538115.

Realm Server Test Results

    1 files  ± 0    357 suites  +3   2h 7m 14s ⏱️ + 6m 4s
5 947 tests +26  5 947 ✅ +26  0 💤 ±0  0 ❌ ±0 
6 010 runs  +26  6 010 ✅ +26  0 💤 ±0  0 ❌ ±0 

Results for commit 0bc2d97. ± Comparison against earlier commit b538115.

[lukemelia]

[Claude Code 🤖] This review checked the runtime changes against ADR 003's own rules. It also traced where the new instance and worker ids flow: Grafana, logs, job_reservations and response headers. It did not re-review the ADR's prose.

No blocking issues. Two small regressions are worth fixing in this PR, and one header choice is yours to decide.

1. Grafana shows the same label for every deployed worker. Ten panels in boxel-status/indexing.json, job-queue.json and prerender-html.json (under packages/observability/grafanactl/resources/dashboards/) map worker_id through ^(.{6}).*$ to View logs ($1). Six characters of an ECS task id told workers apart. Six characters of ip-10-…-<suffix>-pid-N are ip-10- for every worker in the VPC. The links still work because they key on the job id, but you can no longer tell workers apart in the table. ^.*-([0-9a-f]{8}-pid-\d+)$ → View logs ($1) shows the per-boot suffix and pid; old-format ids won't match, so Grafana shows them in full. Regression, non-blocking.
2. bot-runner never runs ESLint in CI. See the comment on RUNTIME_PACKAGE_FILES. Regression against §1.2, small.
3. The task's private IP is now on every public response. See the comment on instanceIdHeader. This is a decision for you, non-blocking.
4. ADR 003 links to ADR 004, which isn't on main yet. Merge the federation ADR first, or the link 404s until it lands.

[lukemelia]

[Claude Code 🤖] The rule covers packages/bot-runner, but CI never runs ESLint there. That package's lint script is only ember-tsc --noEmit, so an SDK import in bot-runner would pass CI, and §1.2's "a lint rule MUST enforce this" doesn't hold for it. Running ESLint directly on a probe import in bot-runner does report the error, so only the CI script is missing.

To fix it, add "lint:js": "eslint . --report-unused-disable-directives --cache", and have lint run both steps through concurrently, as billing does. eslint . in bot-runner then reports 6 no-useless-escape errors, all on one string literal in tests/bot-runner-test.ts, and --fix clears them.

Regression against this PR's own ADR, non-blocking but cheap to fix here.

[lukemelia]

[Claude Code 🤖] Fixed in b538115028. bot-runner now has lint:js, and lint runs it alongside lint:types, so CI enforces the SDK rule there. The escaped quotes in tests/bot-runner-test.ts are cleaned up.

[lukemelia]

[Claude Code 🤖] On Fargate the hostname is the task's private-IP DNS name (ip-10-…), so this header gives every client the replica's internal address. The header it replaces carried a container id and a link-local URL, neither of which is routable.

The load harness only needs a value that differs per process, so a digest works just as well, and logs and job_reservations keep the readable id:

ctxt.set(
  INSTANCE_ID_HEADER,
  createHash('sha256').update(getInstanceId()).digest('hex').slice(0, 12),
);

If you're comfortable publishing internal IPs, keep it as is. Your decision, non-blocking. Whichever you choose, the README's description of the header follows it.

[lukemelia]

[Claude Code 🤖] Keeping the hostname in the header, by the author's decision. The private IP is acceptable to expose here.

[lukemelia]

[Claude Code 🤖] Reconsidered: the header now carries a digest of the instance id instead of the hostname, in a7e910d7c3. See the reply on the later thread about this header.

[lukemelia]

[Claude Code 🤖] Review item 1 is fixed in 73b4cb20cc. The worker-id label in the ten indexing, job-queue and prerender-html panels now shows <suffix>-pid-N. Ids in the old format don't match the pattern and appear in full.

[github-actions]

Observability diff (vs staging)

Show diff

diff --git a/tmp/remote-canon.ZxA2SB/dashboards/boxel-status/indexing.json b/tmp/committed-canon.S2UBpv/dashboards/boxel-status/indexing.json
index 4ef70a7..e73e87a 100644
--- a/tmp/remote-canon.ZxA2SB/dashboards/boxel-status/indexing.json
+++ b/tmp/committed-canon.S2UBpv/dashboards/boxel-status/indexing.json
@@ -80,6 +80,10 @@
           "uid": "cef5v5sl9k7i8f"
         },
         "description": "System-wide operator action: queue a full reindex across every realm. The button disables itself while a `full-reindex` orchestration job is already pending or running. Per-realm reindex moved to the Realms dashboard. Click POSTs with `Authorization: Bearer ${grafana_secret}` (substituted from SSM at apply time, CS-10929).",
+        "fieldConfig": {
+          "defaults": {},
+          "overrides": []
+        },
         "gridPos": {
           "h": 8,
           "w": 24,
@@ -836,13 +840,23 @@
                   "value": [
                     {
                       "options": {
-                        "pattern": "^(.{6}).*$",
+                        "pattern": "^.*-([0-9a-f]{8}-pid-\\d+)$",
                         "result": {
                           "index": 0,
                           "text": "View logs ($1)"
                         }
                       },
                       "type": "regex"
+                    },
+                    {
+                      "options": {
+                        "pattern": "^(.+)$",
+                        "result": {
+                          "index": 1,
+                          "text": "View logs ($1)"
+                        }
+                      },
+                      "type": "regex"
                     }
                   ]
                 }
@@ -1283,13 +1297,23 @@
                   "value": [
                     {
                       "options": {
-                        "pattern": "^(.{6}).*$",
+                        "pattern": "^.*-([0-9a-f]{8}-pid-\\d+)$",
                         "result": {
                           "index": 0,
                           "text": "View logs ($1)"
                         }
                       },
                       "type": "regex"
+                    },
+                    {
+                      "options": {
+                        "pattern": "^(.+)$",
+                        "result": {
+                          "index": 1,
+                          "text": "View logs ($1)"
+                        }
+                      },
+                      "type": "regex"
                     }
                   ]
                 }
@@ -1408,13 +1432,23 @@
                   "value": [
                     {
                       "options": {
-                        "pattern": "^(.{6}).*$",
+                        "pattern": "^.*-([0-9a-f]{8}-pid-\\d+)$",
                         "result": {
                           "index": 0,
                           "text": "View logs ($1)"
                         }
                       },
                       "type": "regex"
+                    },
+                    {
+                      "options": {
+                        "pattern": "^(.+)$",
+                        "result": {
+                          "index": 1,
+                          "text": "View logs ($1)"
+                        }
+                      },
+                      "type": "regex"
                     }
                   ]
                 }
@@ -1545,13 +1579,23 @@
                   "value": [
                     {
                       "options": {
-                        "pattern": "^(.{6}).*$",
+                        "pattern": "^.*-([0-9a-f]{8}-pid-\\d+)$",
                         "result": {
                           "index": 0,
                           "text": "View logs ($1)"
                         }
                       },
                       "type": "regex"
+                    },
+                    {
+                      "options": {
+                        "pattern": "^(.+)$",
+                        "result": {
+                          "index": 1,
+                          "text": "View logs ($1)"
+                        }
+                      },
+                      "type": "regex"
                     }
                   ]
                 }
diff --git a/tmp/remote-canon.ZxA2SB/dashboards/boxel-status/job-queue.json b/tmp/committed-canon.S2UBpv/dashboards/boxel-status/job-queue.json
index 60676d5..76875ee 100644
--- a/tmp/remote-canon.ZxA2SB/dashboards/boxel-status/job-queue.json
+++ b/tmp/committed-canon.S2UBpv/dashboards/boxel-status/job-queue.json
@@ -847,13 +847,23 @@
                   "value": [
                     {
                       "options": {
-                        "pattern": "^(.{6}).*$",
+                        "pattern": "^.*-([0-9a-f]{8}-pid-\\d+)$",
                         "result": {
                           "index": 0,
                           "text": "View logs ($1)"
                         }
                       },
                       "type": "regex"
+                    },
+                    {
+                      "options": {
+                        "pattern": "^(.+)$",
+                        "result": {
+                          "index": 1,
+                          "text": "View logs ($1)"
+                        }
+                      },
+                      "type": "regex"
                     }
                   ]
                 }
@@ -1032,13 +1042,23 @@
                   "value": [
                     {
                       "options": {
-                        "pattern": "^(.{6}).*$",
+                        "pattern": "^.*-([0-9a-f]{8}-pid-\\d+)$",
                         "result": {
                           "index": 0,
                           "text": "View logs ($1)"
                         }
                       },
                       "type": "regex"
+                    },
+                    {
+                      "options": {
+                        "pattern": "^(.+)$",
+                        "result": {
+                          "index": 1,
+                          "text": "View logs ($1)"
+                        }
+                      },
+                      "type": "regex"
                     }
                   ]
                 }
diff --git a/tmp/remote-canon.ZxA2SB/dashboards/boxel-status/prerender-html.json b/tmp/committed-canon.S2UBpv/dashboards/boxel-status/prerender-html.json
index 213deeb..f07fb9b 100644
--- a/tmp/remote-canon.ZxA2SB/dashboards/boxel-status/prerender-html.json
+++ b/tmp/committed-canon.S2UBpv/dashboards/boxel-status/prerender-html.json
@@ -947,13 +947,23 @@
                   "value": [
                     {
                       "options": {
-                        "pattern": "^(.{6}).*$",
+                        "pattern": "^.*-([0-9a-f]{8}-pid-\\d+)$",
                         "result": {
                           "index": 0,
                           "text": "View logs ($1)"
                         }
                       },
                       "type": "regex"
+                    },
+                    {
+                      "options": {
+                        "pattern": "^(.+)$",
+                        "result": {
+                          "index": 1,
+                          "text": "View logs ($1)"
+                        }
+                      },
+                      "type": "regex"
                     }
                   ]
                 }
@@ -1407,13 +1417,23 @@
                   "value": [
                     {
                       "options": {
-                        "pattern": "^(.{6}).*$",
+                        "pattern": "^.*-([0-9a-f]{8}-pid-\\d+)$",
                         "result": {
                           "index": 0,
                           "text": "View logs ($1)"
                         }
                       },
                       "type": "regex"
+                    },
+                    {
+                      "options": {
+                        "pattern": "^(.+)$",
+                        "result": {
+                          "index": 1,
+                          "text": "View logs ($1)"
+                        }
+                      },
+                      "type": "regex"
                     }
                   ]
                 }
@@ -1532,13 +1552,23 @@
                   "value": [
                     {
                       "options": {
-                        "pattern": "^(.{6}).*$",
+                        "pattern": "^.*-([0-9a-f]{8}-pid-\\d+)$",
                         "result": {
                           "index": 0,
                           "text": "View logs ($1)"
                         }
                       },
                       "type": "regex"
+                    },
+                    {
+                      "options": {
+                        "pattern": "^(.+)$",
+                        "result": {
+                          "index": 1,
+                          "text": "View logs ($1)"
+                        }
+                      },
+                      "type": "regex"
                     }
                   ]
                 }
@@ -1669,13 +1699,23 @@
                   "value": [
                     {
                       "options": {
-                        "pattern": "^(.{6}).*$",
+                        "pattern": "^.*-([0-9a-f]{8}-pid-\\d+)$",
                         "result": {
                           "index": 0,
                           "text": "View logs ($1)"
                         }
                       },
                       "type": "regex"
+                    },
+                    {
+                      "options": {
+                        "pattern": "^(.+)$",
+                        "result": {
+                          "index": 1,
+                          "text": "View logs ($1)"
+                        }
+                      },
+                      "type": "regex"
                     }
                   ]
                 }

(Run: https://github.com/cardstack/boxel/actions/runs/39482747213)

[habdelra]

A BOXEL_INSTANCE_ID override is used verbatim with no per-boot suffix, so the uniqueness that worker ids and job_reservations rely on is lost the moment a deployment sets it.

An operator sets BOXEL_INSTANCE_ID in the task definition or Deployment env, which is identical for every replica and survives restarts. Two worker-manager replicas then produce worker ids <id>-pid-<n>, and containerised pids are usually identical across replicas, so both have <id>-pid-27. When one replica's worker exits, finalizeOrphanedReservations(workerId) runs UPDATE job_reservations ... WHERE worker_id = $1 AND completed_at IS NULL and closes the other replica's live reservations, so their jobs get claimed twice. A restarted instance also reuses the earlier boot's id, the exact case the comment says the suffix guards against. Append the per-boot suffix to the override too, or require the override to be unique per replica and per boot.

---

Generated by Claude Code

[lukemelia]

[Claude Code 🤖] Fixed in a7e910d7c3. BOXEL_INSTANCE_ID now replaces only the hostname, and the per-boot suffix is always added, so replicas that share the setting, and a restarted instance, each get their own id. The worker manager hands its whole id to its workers in BOXEL_INHERITED_INSTANCE_ID, which a worker takes as it is, so the suffix isn't added twice.

[habdelra]

X-Boxel-Instance-Id puts the container hostname on every public response, and on Fargate or EC2 that hostname spells out the task's private VPC IP.

Any anonymous GET to the realm server, prerender or worker manager gets back X-Boxel-Instance-Id: ip-10-0-1-23.ec2.internal-9f3c2a1b, which discloses internal addressing; on Kubernetes it discloses the pod name. The ECS header it replaces only showed a link-local metadata URI. Send an opaque id instead (for example a hash of hostname plus suffix, or just the random suffix), or send the header only to authenticated or operator callers.

---

Generated by Claude Code

[lukemelia]

[Claude Code 🤖] Fixed in a7e910d7c3. X-Boxel-Instance-Id carries a 12-hex SHA-256 digest of the instance id (getPublicInstanceId). It still tells replicas apart and changes on restart, and the hostname stays out of public responses. Logs and job_reservations keep the readable id.

[habdelra]

With HEALTH_CHECK_USER_AGENTS unset, the from-memory health check is turned off without any warning.

A deployment that doesn't start through start-production.sh or start-staging.sh never sets the variable. That covers a self-hosted realm server, the federation case this ADR is for, and any ECS task whose command bypasses those scripts. Its load-balancer probe on / then falls through to serveHostApp and serveFromRealm, which is slower and can time out or 5xx under load or during boot. The task then gets marked unhealthy and recycled, and nothing logs that the probe is going unanswered. Log at boot when the variable is unset on a deployed server URL, or keep a provider-neutral path.

---

Generated by Claude Code

[lukemelia]

[Claude Code 🤖] Fixed in 0d1d70fdc5. A server whose own URL is deployed and has no HEALTH_CHECK_USER_AGENTS logs a warning at boot that points to docs/realm-server-health-signals.md. A dedicated health-check path is the option ADR 003 §3 rejected, so the setting stays.

[habdelra]

Gating the indexing dashboard on isDeployedServerURL treats any private IP as local, so a deployed worker manager whose realms map to private addresses turns the dev-only dashboard on.

A self-hosted deployment maps realms to http://10.0.4.12:4201/... or http://192.168.1.5/.... isLocalHostname returns true for non-public IPs, so isIndexingDashboardEnabled() returns true and /_indexing-dashboard, which exposes pending jobs, realm URLs and args, is served in production. Under the old ECS check a deployed manager never served it. A dev machine reached by LAN IP or a custom host gets the opposite flip.

---

Generated by Claude Code

[lukemelia]

[Claude Code 🤖] Fixed in 87fe3039b5. The worker manager serves /_indexing-dashboard only under --indexingDashboard, which only the development worker task passes, so it no longer depends on how the realm URLs look.

[habdelra]

The new value-mapping regex only matches ids that end in an 8-hex suffix followed by -pid-N, so overridden ids and pre-deploy ids lose their 'View logs' label.

Under ^.*-([0-9a-f]{8}-pid-\d+)$, a worker id built from BOXEL_INSTANCE_ID (replica-7-pid-42) or an old job_reservations row (a1de...-3236013547-pid-42, where the runtime id is 10 digits) doesn't match, so the mapping is skipped and the cell shows the raw id instead of 'View logs (...)'. The old ^(.{6}).*$ matched every id. The same pattern was copied into job-queue.json and prerender-html.json.

---

Generated by Claude Code

[lukemelia]

[Claude Code 🤖] Fixed in 978de53b44. Each of the ten mappings now has a second, catch-all ^(.+)$ → View logs ($1) after the suffix-and-pid one. Ids from the current format, which now include ids built from BOXEL_INSTANCE_ID, get the short label, and any other id gets the label in full.

[habdelra]

RUNTIME_PACKAGE_FILES leaves out runtime packages that ADR §1.1 governs: packages/base, packages/bxl and packages/boxel-ui.

ADR §1.1 covers 'code that runs in a deployed service or in the browser', and §1.2 says a lint rule MUST enforce the SDK confinement. A import { S3Client } from '@aws-sdk/client-s3' added to packages/base (card code that the realm server loads and the browser runs) or to packages/bxl (which the realm server evaluates) still lints clean, because those packages inherit the root config but don't match any of the listed globs.

---

Generated by Claude Code

[lukemelia]

[Claude Code 🤖] Fixed in 0bc2d97632. The rule covers base and bxl through the root config, and boxel-ui through its flat config. ADR 003's scope names all three.

[habdelra]

no-restricted-imports only checks static import and export declarations, so require() and dynamic import() of a provider SDK get past the rule the ADR says MUST enforce the boundary.

const { SQSClient } = await import('@aws-sdk/client-sqs'), or require('aws-sdk') in a .cjs runtime file under packages/realm-server, lints clean. Pair this rule with no-restricted-modules, or a no-restricted-syntax selector on ImportExpression and require calls, using the same patterns.

---

Generated by Claude Code

[lukemelia]

[Claude Code 🤖] Fixed in 0bc2d97632. @cardstack/boxel/no-cloud-provider-sdk replaces no-restricted-imports. It reports static imports, re-exports, dynamic import(), require() and require.resolve() (whatever require is bound to, so a createRequire result counts), and import x = require(). A probe with a dynamic import and a require is reported in all ten covered packages.

[habdelra]

S3_ENDPOINT and S3_FORCE_PATH_STYLE are process-global, while region can be set per client, so the media cache and the prerender artifact sink can't point at different stores.

An operator who puts the media cache on R2 (region 'auto', endpoint R2) and keeps prerender artifacts on AWS S3 can set MEDIA_CACHE_REGION and PRERENDER_ARTIFACTS_REGION separately, but setting S3_ENDPOINT sends both clients to R2. The artifact sink then writes to a bucket that doesn't exist there. Accept endpoint and path-style in createS3Client's options (for example MEDIA_CACHE_S3_ENDPOINT) the same way region is accepted, with S3_ENDPOINT as the default.

---

Generated by Claude Code

[lukemelia]

[Claude Code 🤖] A follow-up, not this PR: per-client endpoint and path-style settings, falling back to S3_ENDPOINT and S3_FORCE_PATH_STYLE, are tracked separately. ADR 003 §1.3 holds as it stands, because the endpoint does come from configuration.

[habdelra]

healthCheck re-reads, splits, trims and filters HEALTH_CHECK_USER_AGENTS on every GET to / and /_standby, which are hot routes.

Every host-app page load and every standby probe allocates a new array and runs the string processing before falling through, work that could happen once. Parse the list at module load, or memoize it on the raw env string, which still lets tests change it.

---

Generated by Claude Code

[lukemelia]

[Claude Code 🤖] Fixed in 0d1d70fdc5. healthCheckUserAgents() keeps the parsed list until the raw setting changes, so a request only compares one string, and tests can still change the setting.