feat(e2e-ui): local workbench identity and cluster gitops

distributed_cli/skills/distributed-graphql/SKILL.md

@@ -185,6 +185,13 @@ distributed scaffold my-service --query-api --read-models --store sqlite
 Emits typed causal handlers, a service-derived GraphQL engine, `src/query/`
 permissions, the `graphql` feature, and repository/token-key wiring.
 
+## Client / replica drift (dogfood)
+
+If unit tests pass but the live UI 500s with schema/surface errors: the API
+pod is still compiling or clients/`js/dist` are stale. Rebuild and wait —
+don't redesign roles. Bare `__typename` (role fingerprint) ≠ application
+surface clients.
+
 ## Reference
 
 - Framework docs: `README § GraphQL query service` in the Distributed repo

distributed_cli/skills/distributed-usage/SKILL.md

@@ -309,6 +309,14 @@ Rules the fixture demonstrates:
 
 Run the full app: `cd tests/e2e-ui && make`. Suite: `make test`.
 
+### e2e-ui cluster-dev dogfood (agents)
+
+Namespace = hops `--name`. Fix the live pods, not a host `make run`. After
+source changes: `make gen-client`, restart, **wait until API logs `listening`
+and Vite is ready**, then curl the FQDNs. Schema/pure errors usually mean
+stale build, not a roles redesign. Full checklist: hops skill
+`references/local-workbench.md` § Dogfood.
+
 ## Manifest entrypoint
 
 Every service should export `read_model_catalog()` registering its read

tests/e2e-ui/.gitignore

@@ -14,3 +14,4 @@ e2e/.auth
 playwright-report
 test-results
 blob-report
+gitops/cluster/e2e-identity/grants.yaml

tests/e2e-ui/README.md

@@ -4,6 +4,42 @@ A copyable Distributed service and SvelteKit UI demonstrating one modeled
 projection from aggregate transition to server read model, generated GraphQL
 client, optimistic replica update, and causal confirmation.
 
+## Option A — hops local workbench (preferred for cluster-shaped DX)
+
+One-time: `hops local start` (control plane). Then:
+
+```bash
+cd tests/e2e-ui
+# apps (workbench)
+hops local up ./gitops/envs/local --name e2e
+hops local status
+hops local open
+# when done
+hops local down --name e2e
+```
+
+Optional **platform XRs** under `gitops/cluster/` (AuthStack + PSQLStack):
+
+```bash
+hops config install <meta>/xrs/stacks/k8s/psql
+hops config install <meta>/xrs/stacks/k8s/auth
+kubectl apply -f gitops/cluster/stacks/psql.yaml
+kubectl apply -f gitops/cluster/stacks/auth.yaml
+```
+
+Optional cloud providers (non-secret YAML only):
+
+```bash
+hops local aws --gitops ./gitops/cluster
+hops local github --gitops ./gitops/cluster
+```
+
+Charts: `api/.gitops/deploy`, `ui/.gitops/deploy`.  
+App Applications: `gitops/envs/local/` (legacy: `gitops/env/local/`).  
+Control plane: `gitops/cluster/` (`stacks/`, `configurations/`, …).
+
+## Option B — compose + host processes
+
 ```bash
 cd tests/e2e-ui
 make up

tests/e2e-ui/api/.gitops/deploy/Chart.yaml

@@ -0,0 +1,6 @@
+apiVersion: v2
+name: e2e-ui-api
+description: Local workbench dogfood chart for e2e-ui API (cluster-dev / package postures)
+type: application
+version: 0.1.0
+appVersion: "0.1.0"

tests/e2e-ui/api/.gitops/deploy/README.md

@@ -0,0 +1,13 @@
+# e2e-ui API chart (workbench)
+
+Renders:
+
+- **Deployment / Service** (`appRuntime: cluster-dev` \| `package`)
+- **Optional PSQLCluster** when `database.enabled: true`
+
+OIDC project / demo humans / web app live on the **UI** chart
+(`ui/.gitops/deploy` `identity.*`). This chart only consumes residual OIDC
+settings (`OIDC_ISSUER`, `OIDC_AUDIENCE`, shared `e2e-ui-oidc` secret keys) so
+Bearer tokens map engine roles.
+
+Platform AuthStack + ProviderConfig stay under `gitops/cluster/`.

tests/e2e-ui/api/.gitops/deploy/templates/_helpers.tpl

@@ -0,0 +1,9 @@
+{{- define "e2e-ui-api.name" -}}
+{{- default .Chart.Name .Values.name | trunc 63 | trimSuffix "-" -}}
+{{- end -}}
+
+{{- define "e2e-ui-api.labels" -}}
+app.kubernetes.io/name: {{ include "e2e-ui-api.name" . }}
+app.kubernetes.io/instance: {{ .Release.Name }}
+app.kubernetes.io/managed-by: {{ .Release.Service }}
+{{- end -}}

tests/e2e-ui/api/.gitops/deploy/templates/deployment-cluster-dev.yaml

@@ -0,0 +1,87 @@
+{{- if or (eq .Values.appRuntime "cluster-dev") (and .Values.local (eq .Values.appRuntime "cluster-dev")) }}
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: {{ include "e2e-ui-api.name" . }}
+  labels:
+    {{- include "e2e-ui-api.labels" . | nindent 4 }}
+    hops.ops.com.ai/runtime: cluster-dev
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: {{ include "e2e-ui-api.name" . }}
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: {{ include "e2e-ui-api.name" . }}
+        hops.ops.com.ai/runtime: cluster-dev
+    spec:
+      containers:
+        - name: api
+          image: {{ .Values.clusterDev.image | quote }}
+          workingDir: {{ .Values.clusterDev.workingDir | quote }}
+          {{- if eq .Values.sourceDelivery.mode "sync" }}
+          command:
+            - /bin/bash
+            - -c
+            - |
+              set -e
+              echo "waiting for source sync at {{ .Values.sourceDelivery.mountPath }}..."
+              waited=0
+              while [ ! -f {{ .Values.sourceDelivery.mountPath }}/.hops-synced ] \
+                && [ ! -f {{ .Values.clusterDev.workingDir }}/Cargo.toml ]; do
+                sleep 2
+                waited=$((waited + 2))
+                if [ $((waited % 30)) -eq 0 ]; then
+                  echo "still waiting for hops source sync (${waited}s) — is tar watcher running?"
+                fi
+                if [ "$waited" -ge 600 ]; then
+                  echo "ERROR: source sync timeout after ${waited}s (no .hops-synced / Cargo.toml)"
+                  exit 1
+                fi
+              done
+              echo "source present; starting dev process"
+              command -v cargo
+              rustc --version
+              cd {{ .Values.clusterDev.workingDir }}
+              cargo run -p e2e-runner --bin e2e-ui
+          {{- else }}
+          command:
+            {{- toYaml .Values.clusterDev.command | nindent 12 }}
+          {{- end }}
+          ports:
+            - containerPort: {{ .Values.service.targetPort }}
+              name: http
+          env:
+            {{- range $k, $v := .Values.env }}
+            - name: {{ $k }}
+              value: {{ $v | quote }}
+            {{- end }}
+            {{- range $k, $ref := .Values.secretEnv }}
+            - name: {{ $k }}
+              valueFrom:
+                secretKeyRef:
+                  name: {{ $ref.secretName | quote }}
+                  key: {{ $ref.key | quote }}
+            {{- end }}
+            # Keep cargo on PATH even if something mangles the environment.
+            - name: PATH
+              value: "/usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
+          volumeMounts:
+            - name: source
+              mountPath: {{ .Values.sourceDelivery.mountPath }}
+            - name: cargo-target
+              mountPath: {{ .Values.clusterDev.workingDir }}/target
+      volumes:
+        - name: source
+          {{- if eq .Values.sourceDelivery.mode "hostPath" }}
+          hostPath:
+            path: {{ .Values.sourceDelivery.hostPath | quote }}
+            type: Directory
+          {{- else }}
+          emptyDir: {}
+          {{- end }}
+        - name: cargo-target
+          emptyDir: {}
+{{- end }}

tests/e2e-ui/api/.gitops/deploy/templates/deployment-package.yaml

@@ -0,0 +1,31 @@
+{{- /* Package Deployment: production/CI image posture. Omitted for local/cluster-dev. */ -}}
+{{- if and (not .Values.local) (ne .Values.appRuntime "cluster-dev") (ne .Values.appRuntime "host") }}
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: {{ include "e2e-ui-api.name" . }}
+  labels:
+    {{- include "e2e-ui-api.labels" . | nindent 4 }}
+    hops.ops.com.ai/runtime: package
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: {{ include "e2e-ui-api.name" . }}
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: {{ include "e2e-ui-api.name" . }}
+        hops.ops.com.ai/runtime: package
+    spec:
+      containers:
+        - name: api
+          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
+          imagePullPolicy: {{ .Values.image.pullPolicy }}
+          ports:
+            - containerPort: {{ .Values.service.targetPort }}
+              name: http
+          env:
+            - name: BIND
+              value: {{ .Values.env.BIND | quote }}
+{{- end }}

tests/e2e-ui/api/.gitops/deploy/templates/psqlcluster.yaml

@@ -0,0 +1,15 @@
+{{- if and .Values.database.enabled (or .Values.local (eq .Values.appRuntime "cluster-dev") (eq .Values.appRuntime "host")) }}
+# Local-capable PSQLCluster XR. Storage class defaults to values.database.storage.class
+# (local-path for k3s/dory; override for other provisioners).
+apiVersion: hops.ops.com.ai/v1alpha1
+kind: PSQLCluster
+metadata:
+  name: {{ .Values.database.name }}
+  labels:
+    {{- include "e2e-ui-api.labels" . | nindent 4 }}
+spec:
+  clusterName: {{ .Values.database.clusterName }}
+  storage:
+    size: {{ .Values.database.storage.size }}
+    class: {{ .Values.database.storage.class }}
+{{- end }}

tests/e2e-ui/api/.gitops/deploy/templates/service.yaml

@@ -0,0 +1,13 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: {{ include "e2e-ui-api.name" . }}
+  labels:
+    {{- include "e2e-ui-api.labels" . | nindent 4 }}
+spec:
+  selector:
+    app.kubernetes.io/name: {{ include "e2e-ui-api.name" . }}
+  ports:
+    - name: http
+      port: {{ .Values.service.port }}
+      targetPort: {{ .Values.service.targetPort }}

tests/e2e-ui/api/.gitops/deploy/values.yaml

@@ -0,0 +1,67 @@
+# Local workbench values for e2e-ui API.
+# Storage class for PSQLCluster: override per local CP (e.g. local-path, hostpath, psql).
+
+local: false
+appRuntime: package  # host | cluster-dev | package
+
+name: e2e-ui-api
+image:
+  repository: ghcr.io/hops-ops/e2e-ui-api
+  tag: latest
+  pullPolicy: IfNotPresent
+
+# Source tree is the *distributed* monorepo root; API Cargo workspace is tests/e2e-ui.
+clusterDev:
+  image: rust:1.94-bookworm
+  # bash -c keeps Docker ENV (PATH includes /usr/local/cargo/bin). -lc can drop it.
+  command:
+    - /bin/bash
+    - -c
+    - |
+      set -e
+      cd /workspace/tests/e2e-ui
+      cargo run -p e2e-runner --bin e2e-ui
+  workingDir: /workspace/tests/e2e-ui
+  sourcePath: /workspace
+
+service:
+  port: 8791
+  targetPort: 8791
+
+env:
+  BIND: "0.0.0.0:8791"
+  # Offline-friendly default until PSQL platform pack is installed.
+  DATABASE_URL: "sqlite:/tmp/e2e-ui.db?mode=rwc"
+  # OidcBearer — requires issuer **and** audience (project id). If either is empty
+  # the API falls back to DevHeaders and live user surfaces misbehave.
+  OIDC_ISSUER: "http://zitadel-zitadel.auth.svc.cluster.local:8080"
+  # Residual after Zitadel Project Ready (external-name / status.atProvider.id).
+  # Override from Application values (gitops/envs/local/api.yaml) for local dogfood.
+  OIDC_AUDIENCE: ""
+  OIDC_JWKS_URI: "http://zitadel-zitadel.auth.svc.cluster.local:8080/oauth/v2/keys"
+  ZITADEL_PROJECT_ID: ""
+
+# Residual secrets (same e2e-ui-oidc object as the UI chart, in app namespace).
+# OIDC app + humans live on the UI chart (identity.*); API only consumes tokens.
+secretEnv:
+  OIDC_CLIENT_ID:
+    secretName: e2e-ui-oidc
+    key: OIDC_CLIENT_ID
+  ZITADEL_SERVICE_USER_TOKEN:
+    secretName: e2e-ui-oidc
+    key: ZITADEL_SERVICE_USER_TOKEN
+
+database:
+  # Dogfood local CP often lacks PSQLCluster CRD; disable by default for up path.
+  # Enable when PSQL pack is installed: --set database.enabled=true via Application values.
+  enabled: false
+  name: e2e-ui-api-psql
+  clusterName: local
+  storage:
+    size: 2Gi
+    class: local-path
+
+sourceDelivery:
+  mode: none  # none | hostPath | sync
+  hostPath: ""
+  mountPath: /workspace

tests/e2e-ui/gitops/README.md

@@ -0,0 +1,26 @@
+# e2e-ui local gitops
+
+```text
+gitops/
+  cluster/          # fixture CP tree (real meta repos: put this at meta root)
+  envs/
+    local/          # app Applications → worktree namespaces
+  env/local/        # deprecated alias of envs/local
+```
+
+```bash
+# Bootstrap + cluster gitops apply/watch (stays in foreground = gitops cluster)
+hops local start --backend dory --gitops ./gitops/cluster
+
+# Or cluster watch alone if CP already started
+# hops local gitops cluster ./gitops/cluster
+
+# Per-worktree apps (watches by default) — separate terminal
+hops local gitops worktree ./gitops/envs/local --name dogfood
+
+# One-shot (CI / scripts): add --once
+# hops local gitops cluster ./gitops/cluster --once
+```
+
+On multi-project metas: put `gitops/cluster` at the **meta root**; projects keep
+`.gitops/deploy` charts; env Application YAMLs point at those charts.