From 9b9276352791321a1cf5e0157f7a580dc07b7f8f Mon Sep 17 00:00:00 2001 From: Dimitri Krattiger Date: Tue, 22 Sep 2026 08:57:04 -0600 Subject: [PATCH 1/6] deploy: enable unsupervised-main agents to do finder prod-testing (in-pod build, scoped SA) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bundle to give panopticon task agents the finder prod-testing capability: - finder-repro-rbac.yaml: dedicated finder-repro namespace + panopticon-repro SA (pods CRUD/exec/log) + ResourceQuota/LimitRange ceiling. - unsupervised-main/Dockerfile.finder-builder + publish.finder-builder.yml: pre-baked Harbor builder image (deps installed, source overlaid per build) reproducing build.finder.yml; published with existing HARBOR_USERNAME/PASSWORD. - build-finder-in-pod.sh: agent-run in-pod build (no DinD) — overlay current src, pyinstaller, copy binary out; pulls via unsupervised-regcred. - README: design, apply sequence, rebuild cadence. Still pending (credential-handling, blocked by the auto-mode classifier — need operator OK): image-layer.head.dockerfile (kubectl + kubeconfig-wiring wrapper) and apply.sh (regenerates PROD_REPRO_KUBECONFIG_B64 scoped to finder-repro). Co-Authored-By: Claude Opus 4.8 --- .../unsupervised-main-prod-testing/README.md | 59 +++++++++++ .../build-finder-in-pod.sh | 90 +++++++++++++++++ .../finder-repro-rbac.yaml | 98 +++++++++++++++++++ .../Dockerfile.finder-builder | 44 +++++++++ .../publish.finder-builder.yml | 70 +++++++++++++ 5 files changed, 361 insertions(+) create mode 100644 deploy/unsupervised-main-prod-testing/README.md create mode 100644 deploy/unsupervised-main-prod-testing/build-finder-in-pod.sh create mode 100644 deploy/unsupervised-main-prod-testing/finder-repro-rbac.yaml create mode 100644 deploy/unsupervised-main-prod-testing/unsupervised-main/Dockerfile.finder-builder create mode 100644 deploy/unsupervised-main-prod-testing/unsupervised-main/publish.finder-builder.yml diff --git a/deploy/unsupervised-main-prod-testing/README.md b/deploy/unsupervised-main-prod-testing/README.md new file mode 100644 index 00000000..4c6bd5f2 --- /dev/null +++ b/deploy/unsupervised-main-prod-testing/README.md @@ -0,0 +1,59 @@ +# Enable panopticon agents to do finder prod-testing + +Give unsupervised-main task agents the ability to **provision test pods, generate the finder +executable, and profile results** in prod — the workflow that's currently done by hand. + +Approach (operator-chosen): **build the finder binary in a k8s pod (no Docker-in-Docker)**, from a +**pre-baked Harbor builder image**, with the repro ServiceAccount **scoped to a dedicated namespace**. + +## What already exists +- Repo env-file injects `PROD_REPRO_KUBECONFIG_B64` (SA `panopticon-repro`, embedded token — no + `awscli` needed) and `PROD_READONLY_KUBECONFIG_B64`. +- The `panopticon-repro` SA already has pods create/delete/exec/log (in `default`). +- Harbor push creds already exist in CI (`vars.HARBOR_USERNAME` + `secrets.HARBOR_PASSWORD`); the + in-cluster pull secret `unsupervised-regcred` already exists. + +## What this bundle adds + +| File | Goes to | Purpose | +|---|---|---| +| `finder-repro-rbac.yaml` | prod cluster (`kubectl apply`) | Dedicated `finder-repro` ns + SA + Role (pods CRUD/exec/log) + ResourceQuota/LimitRange ceiling | +| `unsupervised-main/Dockerfile.finder-builder` | unsupervised-main (PR) | Pre-baked builder image: heavy deps (Rust ext + py deps) installed, source overlaid per build | +| `unsupervised-main/publish.finder-builder.yml` | unsupervised-main (PR) | CI job that builds & pushes `harbor…/images/finder-builder` (reuses existing Harbor creds) | +| `build-finder-in-pod.sh` | panopticon image layer (`/usr/local/bin`) | Agent-run: overlay current src into a builder pod, `pyinstaller`, copy binary out | +| `image-layer.head.dockerfile` | panopticon `$CONFIG/layers/` | **⚠ pending — see below.** kubectl + auto-wiring wrapper | +| `apply.sh` | operator runs once | **⚠ pending — see below.** wire config + scope the SA | + +## ⚠ Two credential-handling files still to write (need your OK) +The auto-mode classifier blocked writing files that **decode your prod kubeconfig secret**, which is +correct — they touch a credential. They are: +1. **`image-layer.head.dockerfile`** — installs `kubectl` and a wrapper that lazily materializes + `~/.kube/config` from `$PROD_REPRO_KUBECONFIG_B64` on first use (works under `bash -c`; no host-hook + change; the existing `neutralize-claude-hooks.sh` stays as-is). +2. **`apply.sh`** — one-shot operator script: assemble the layer file (head + `build-finder-in-pod.sh` + as a heredoc) into `$CONFIG/layers/unsupervised-main.dockerfile`; `PATCH /repos/unsupervised-main` + with `image_layer_file`; **regenerate `PROD_REPRO_KUBECONFIG_B64`** to point at the `finder-repro` + namespace (backing up the old value first); rebuild the composed image. + +Approve those and I'll write them. + +## Apply sequence (once everything's written) +1. **PR** `Dockerfile.finder-builder` + `publish.finder-builder.yml` into unsupervised-main; run the CI + job once → `harbor…/images/finder-builder:latest` exists. +2. `kubectl --context apply -f finder-repro-rbac.yaml` (creates `finder-repro`). +3. Copy the pull secret into the new ns (no new account): + `kubectl --context -n default get secret unsupervised-regcred -o yaml \ + | sed 's/namespace: default/namespace: finder-repro/' \ + | kubectl --context -n finder-repro apply -f -` +4. `./apply.sh` (wires the layer + repo config; **rewrites the kubeconfig secret to finder-repro** — + the one prod-credential mutation, done with a backup). +5. Smoke test: `build-finder-in-pod.sh` produces a binary; provision a repro pod in `finder-repro`; + profile via `kubectl exec` (passive cgroup `cpu.stat`). + +## Profiling +No extra agent-side capability: it's `kubectl exec` into the test pod (cgroup `cpu.stat` + +table-completion progress; py-spy in-pod). See the finder-load perf notes. + +## Rebuild cadence +The pre-baked image only needs rebuilding when the **heavy deps** change (requirements.txt / Rust ext / +python-utils) — ordinary `fc.py` edits are overlaid at build-in-pod time, so day-to-day this is free. diff --git a/deploy/unsupervised-main-prod-testing/build-finder-in-pod.sh b/deploy/unsupervised-main-prod-testing/build-finder-in-pod.sh new file mode 100644 index 00000000..1684d1b6 --- /dev/null +++ b/deploy/unsupervised-main-prod-testing/build-finder-in-pod.sh @@ -0,0 +1,90 @@ +#!/usr/bin/env bash +# Build the finder PyInstaller binary in a k8s pod (NO Docker-in-Docker). +# +# Baked into the panopticon repo image layer at /usr/local/bin/build-finder-in-pod.sh so +# agents can run it. Uses the pre-baked Harbor builder image (deps already installed); it +# only overlays the agent's CURRENT finder src and runs pyinstaller (~2 min), then copies +# the binary out. kubectl is auto-wired to the finder-repro namespace by the image-layer +# wrapper; the builder pod pulls via the unsupervised-regcred secret (copied into finder-repro). +# +# Usage: build-finder-in-pod.sh [--src DIR] [--out FILE] [--ns NS] [--image IMG] +# [--name POD] [--rebuild-rust] [--keep] +set -euo pipefail + +SRC="subrepos/finder" +OUT="./finder" +NS="finder-repro" +IMAGE="${FINDER_BUILDER_IMAGE:-harbor.unsupervised.com/images/finder-builder:latest}" +POD="" +REBUILD_RUST=0 +KEEP=0 + +while [ $# -gt 0 ]; do + case "$1" in + --src) SRC="$2"; shift 2;; + --out) OUT="$2"; shift 2;; + --ns) NS="$2"; shift 2;; + --image) IMAGE="$2"; shift 2;; + --name) POD="$2"; shift 2;; + --rebuild-rust) REBUILD_RUST=1; shift;; + --keep) KEEP=1; shift;; + *) echo "unknown arg: $1" >&2; exit 2;; + esac +done + +[ -d "$SRC/src" ] || { echo "no finder src at $SRC/src (run from the repo root, or pass --src)" >&2; exit 2; } +# Pod name is deterministic from the src content so concurrent agents don't collide, without +# needing $RANDOM (unavailable in some restricted shells): hash the tree listing. +if [ -z "$POD" ]; then + h="$(find "$SRC/src" -type f -printf '%P %s\n' 2>/dev/null | cksum | cut -d' ' -f1)" + POD="finder-build-${h}" +fi + +cleanup() { [ "$KEEP" = 1 ] || kubectl -n "$NS" delete pod "$POD" --ignore-not-found --wait=false >/dev/null 2>&1 || true; } +trap cleanup EXIT + +echo "== launching builder pod $POD (image=$IMAGE, ns=$NS) ==" +kubectl -n "$NS" delete pod "$POD" --ignore-not-found --wait=true >/dev/null 2>&1 || true +kubectl -n "$NS" apply -f - <&2; exit 3; } + +echo "== syncing current finder src into the pod (overlay editable install) ==" +tar -C "$SRC/src" -czf - . | kubectl -n "$NS" exec -i "$POD" -- \ + bash -c 'rm -rf /build/subrepos/finder/src && mkdir -p /build/subrepos/finder/src && tar -C /build/subrepos/finder/src -xzf -' + +if [ "$REBUILD_RUST" = 1 ]; then + echo "== (--rebuild-rust) syncing pybfinder + rebuilding the Rust ext ==" + tar -C "$SRC/pybfinder" -czf - . | kubectl -n "$NS" exec -i "$POD" -- \ + bash -c 'rm -rf /build/subrepos/finder/pybfinder && mkdir -p /build/subrepos/finder/pybfinder && tar -C /build/subrepos/finder/pybfinder -xzf -' + kubectl -n "$NS" exec "$POD" -- bash -lc 'cd /build/subrepos/finder/pybfinder && maturin build -r && pip install --force-reinstall target/wheels/*.whl' +fi + +echo "== running pyinstaller in-pod ==" +kubectl -n "$NS" exec "$POD" -- bash -lc 'cd /build/subrepos/finder && sh pyinstaller.sh' + +echo "== copying binary out -> $OUT ==" +kubectl -n "$NS" cp "$POD:/build/subrepos/finder/dist/finder" "$OUT" +chmod +x "$OUT" +echo "== done: $OUT ($(wc -c < "$OUT") bytes). NOTE: this is a linux/amd64 binary — run it in a pod, not on the agent host. ==" diff --git a/deploy/unsupervised-main-prod-testing/finder-repro-rbac.yaml b/deploy/unsupervised-main-prod-testing/finder-repro-rbac.yaml new file mode 100644 index 00000000..7de68b00 --- /dev/null +++ b/deploy/unsupervised-main-prod-testing/finder-repro-rbac.yaml @@ -0,0 +1,98 @@ +# Scoped RBAC for panopticon agents doing finder prod-testing. +# Applies to the prod EKS cluster. Replaces the current arrangement where the +# `panopticon-repro` ServiceAccount lives in `default` with pod create/delete/exec/log. +# +# Decision (operator-approved): give agents a DEDICATED namespace with a resource +# ceiling, not `default`, so an agent can't spawn an unbounded 256Gi run or interfere +# with real workloads. The verbs match exactly what the finder-pod-testing workflow +# needs and nothing more. +# +# Apply: kubectl --context apply -f finder-repro-rbac.yaml +# The ResourceQuota / LimitRange numbers are ceilings — tune to the real test budget. +apiVersion: v1 +kind: Namespace +metadata: + name: finder-repro + labels: + app.kubernetes.io/managed-by: panopticon + panopticon.unsupervised.com/purpose: finder-prod-testing +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: panopticon-repro + namespace: finder-repro +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: panopticon-repro + namespace: finder-repro +rules: + - apiGroups: [""] + resources: ["pods", "pods/log"] + verbs: ["get", "list", "watch"] + - apiGroups: [""] + resources: ["pods"] + verbs: ["create", "delete"] + - apiGroups: [""] + resources: ["pods/exec"] + verbs: ["create"] + # cp needs to read pod status/attach; keep it minimal. + - apiGroups: [""] + resources: ["pods/status"] + verbs: ["get"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: panopticon-repro + namespace: finder-repro +subjects: + - kind: ServiceAccount + name: panopticon-repro + namespace: finder-repro +roleRef: + kind: Role + name: panopticon-repro + apiGroup: rbac.authorization.k8s.io +--- +# Ceiling on the whole namespace so agents can't run away. Finder repro pods are +# legitimately large (200Gi+), so these are generous but bounded. +apiVersion: v1 +kind: ResourceQuota +metadata: + name: finder-repro-quota + namespace: finder-repro +spec: + hard: + pods: "8" + requests.cpu: "128" + limits.cpu: "128" + requests.memory: 600Gi + limits.memory: 600Gi + requests.ephemeral-storage: 1000Gi + limits.ephemeral-storage: 1000Gi +--- +# Cap any single pod so one pod can't consume the entire quota, and give pods that +# omit limits a small safe default rather than unbounded. +apiVersion: v1 +kind: LimitRange +metadata: + name: finder-repro-limits + namespace: finder-repro +spec: + limits: + - type: Container + max: + cpu: "64" + memory: 300Gi + ephemeral-storage: 500Gi + default: + cpu: "2" + memory: 8Gi + ephemeral-storage: 20Gi + defaultRequest: + cpu: "1" + memory: 4Gi + ephemeral-storage: 10Gi diff --git a/deploy/unsupervised-main-prod-testing/unsupervised-main/Dockerfile.finder-builder b/deploy/unsupervised-main-prod-testing/unsupervised-main/Dockerfile.finder-builder new file mode 100644 index 00000000..eb29e89b --- /dev/null +++ b/deploy/unsupervised-main-prod-testing/unsupervised-main/Dockerfile.finder-builder @@ -0,0 +1,44 @@ +# Pre-baked finder BUILDER image — heavy deps installed once, source overlaid per build. +# +# Destined for the unsupervised-main repo (e.g. subrepos/finder/Dockerfile.finder-builder), +# published to Harbor by the companion CI job (publish.finder-builder.yml). Kept here in the +# panopticon bundle as the reviewable source; copy into unsupervised-main via a normal PR. +# +# Reproduces the CANONICAL build (.github/workflows/build.finder.yml: Py3.10 + Rust + maturin +# + editable installs), but STOPS before `pyinstaller.sh`. Finder is installed editable, so at +# build-in-pod time the agent overlays the current src at the known path and runs pyinstaller +# (~2 min) instead of rebuilding the ~heavy deps + Rust ext. +# +# Build context = repo root (needs both subrepos/finder and subrepos/python-utils, since +# finder's requirements.txt has `-e ../python-utils`). Build with submodules checked out: +# docker build -f subrepos/finder/Dockerfile.finder-builder -t /images/finder-builder: . +FROM python:3.10-bookworm + +ENV DEBIAN_FRONTEND=noninteractive +RUN apt-get update \ + && apt-get install --yes --no-install-recommends \ + build-essential libpq-dev curl git ca-certificates pkg-config \ + && rm -rf /var/lib/apt/lists/* + +# Rust toolchain (pybfinder is a maturin/Rust extension). Pin to match CI's stable. +RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable +ENV PATH="/root/.cargo/bin:${PATH}" + +RUN pip install --no-cache-dir maturin pyinstaller + +# Bring in both subrepos at the SAME relative layout finder expects (`-e ../python-utils`). +WORKDIR /build +COPY subrepos/python-utils /build/subrepos/python-utils +COPY subrepos/finder /build/subrepos/finder + +WORKDIR /build/subrepos/finder +# Install python deps + editable python-utils + editable finder (from requirements.txt), +# then build & install the Rust extension. This is the expensive part we pre-bake. +RUN pip install --no-cache-dir -r requirements.txt \ + && (cd pybfinder && maturin build -r && pip install target/wheels/*.whl) + +# NOTE: no `pyinstaller.sh` here. The editable finder lives at /build/subrepos/finder/src; +# build-finder-in-pod.sh overlays the agent's current src there and runs pyinstaller. +# A marker the build script checks for, so a stale/wrong image fails loudly: +RUN echo "finder-builder: deps baked; overlay src at /build/subrepos/finder/src then run pyinstaller.sh" \ + > /build/FINDER_BUILDER_READY diff --git a/deploy/unsupervised-main-prod-testing/unsupervised-main/publish.finder-builder.yml b/deploy/unsupervised-main-prod-testing/unsupervised-main/publish.finder-builder.yml new file mode 100644 index 00000000..cbfa07a3 --- /dev/null +++ b/deploy/unsupervised-main-prod-testing/unsupervised-main/publish.finder-builder.yml @@ -0,0 +1,70 @@ +# Publish the pre-baked finder BUILDER image to Harbor. +# Destined for unsupervised-main/.github/workflows/publish.finder-builder.yml (copy via PR). +# +# Reuses the SAME Harbor credentials CI already uses for unsupervised-main / unsupervised-base +# (vars.HARBOR_USERNAME + secrets.HARBOR_PASSWORD) — no new account. In-cluster PULL reuses the +# existing `unsupervised-regcred` secret (copied into the finder-repro namespace), so nothing new +# on the pull side either. +# +# Rebuild cadence: manual (workflow_dispatch) + whenever the heavy deps change. Rebuild is only +# needed when requirements.txt / the Rust ext / python-utils change — NOT for ordinary fc.py edits +# (those are overlaid at build-in-pod time), so this stays cheap. +name: Publish Finder Builder Image +on: + workflow_dispatch: + inputs: + tag: + description: "Image tag (default: short SHA)" + required: false + default: "" + push: + branches: [staging] + paths: + - "subrepos/finder/requirements.txt" + - "subrepos/finder/pybfinder/**" + - "subrepos/finder/bfinder/**" + - "subrepos/finder/Cargo.*" + - "subrepos/finder/Dockerfile.finder-builder" + - "subrepos/python-utils/**" + +concurrency: + group: publish-finder-builder-${{ github.ref }} + cancel-in-progress: true + +jobs: + publish: + runs-on: ubuntu-latest + steps: + - name: Checkout (with submodules) + uses: actions/checkout@v7 + with: + submodules: true + + - name: Resolve tag + id: tag + run: | + t="${{ inputs.tag }}" + [ -n "$t" ] || t="$(git rev-parse --short HEAD)" + echo "tag=$t" >> "$GITHUB_OUTPUT" + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Log in to Harbor + uses: docker/login-action@v3 + with: + registry: harbor.unsupervised.com + username: ${{ vars.HARBOR_USERNAME }} + password: ${{ secrets.HARBOR_PASSWORD }} + + - name: Build & push finder-builder + uses: docker/build-push-action@v6 + with: + context: . + file: subrepos/finder/Dockerfile.finder-builder + push: true + tags: | + harbor.unsupervised.com/images/finder-builder:${{ steps.tag.outputs.tag }} + harbor.unsupervised.com/images/finder-builder:latest + cache-from: type=registry,ref=harbor.unsupervised.com/images/finder-builder:buildcache + cache-to: type=registry,ref=harbor.unsupervised.com/images/finder-builder:buildcache,mode=max From 58551d6eee96372076d41d18b919ae0bf1ec7fd0 Mon Sep 17 00:00:00 2001 From: Dimitri Krattiger Date: Tue, 22 Sep 2026 09:11:29 -0600 Subject: [PATCH 2/6] deploy: add data-access (scoped IRSA), turn-handoff gate, credential files - finder-repro-rbac.yaml: add finder-test run-SA (IRSA -> read-only prod S3); panopticon-repro stays the control-plane SA. - iam-finder-repro-readonly.json: scoped IAM role templates (trust = EKS OIDC + finder-repro:finder-test; permissions = read-only s3 on unsupervised-prod-internal/internal/*). - image-layer.head.dockerfile + apply.sh: the credential-handling files (kubectl wrapper that materializes the kubeconfig; apply.sh wires config and rewrites PROD_REPRO_KUBECONFIG_B64 scoped to finder-repro, test-before-overwrite + backup). Layer build smoke-tested. - repro-pod.template.yaml: finder test pod running as finder-test. - prod-testing-gate.md: operator turn-handoff approval rule for agents. Co-Authored-By: Claude Opus 4.8 --- .../unsupervised-main-prod-testing/README.md | 91 ++++++------ .../unsupervised-main-prod-testing/apply.sh | 130 ++++++++++++++++++ .../finder-repro-rbac.yaml | 18 +++ .../iam-finder-repro-readonly.json | 51 +++++++ .../image-layer.head.dockerfile | 35 +++++ .../prod-testing-gate.md | 35 +++++ .../repro-pod.template.yaml | 44 ++++++ 7 files changed, 356 insertions(+), 48 deletions(-) create mode 100644 deploy/unsupervised-main-prod-testing/apply.sh create mode 100644 deploy/unsupervised-main-prod-testing/iam-finder-repro-readonly.json create mode 100644 deploy/unsupervised-main-prod-testing/image-layer.head.dockerfile create mode 100644 deploy/unsupervised-main-prod-testing/prod-testing-gate.md create mode 100644 deploy/unsupervised-main-prod-testing/repro-pod.template.yaml diff --git a/deploy/unsupervised-main-prod-testing/README.md b/deploy/unsupervised-main-prod-testing/README.md index 4c6bd5f2..026ffb6c 100644 --- a/deploy/unsupervised-main-prod-testing/README.md +++ b/deploy/unsupervised-main-prod-testing/README.md @@ -1,59 +1,54 @@ # Enable panopticon agents to do finder prod-testing Give unsupervised-main task agents the ability to **provision test pods, generate the finder -executable, and profile results** in prod — the workflow that's currently done by hand. +executable, and profile results** against production data — the workflow currently done by hand. -Approach (operator-chosen): **build the finder binary in a k8s pod (no Docker-in-Docker)**, from a -**pre-baked Harbor builder image**, with the repro ServiceAccount **scoped to a dedicated namespace**. +Design (operator-chosen): **in-pod finder build (no Docker-in-Docker)** from a **pre-baked Harbor +builder image**; repro ServiceAccount **scoped to a dedicated `finder-repro` namespace**; prod +**data read-only via a scoped IRSA SA**; every prod run gated by an **operator turn-handoff**, with +the namespace **ResourceQuota as the enforced backstop**. -## What already exists -- Repo env-file injects `PROD_REPRO_KUBECONFIG_B64` (SA `panopticon-repro`, embedded token — no - `awscli` needed) and `PROD_READONLY_KUBECONFIG_B64`. -- The `panopticon-repro` SA already has pods create/delete/exec/log (in `default`). -- Harbor push creds already exist in CI (`vars.HARBOR_USERNAME` + `secrets.HARBOR_PASSWORD`); the - in-cluster pull secret `unsupervised-regcred` already exists. +## Two identities (keep them straight) +- **Control** — `panopticon-repro` SA: creates/execs pods (RBAC). No data access. +- **Run** — `finder-test` SA: the SA the test *pods* run as; IRSA -> read-only prod S3. Data + access rides here, not on kubectl. -## What this bundle adds +## Files | File | Goes to | Purpose | |---|---|---| -| `finder-repro-rbac.yaml` | prod cluster (`kubectl apply`) | Dedicated `finder-repro` ns + SA + Role (pods CRUD/exec/log) + ResourceQuota/LimitRange ceiling | -| `unsupervised-main/Dockerfile.finder-builder` | unsupervised-main (PR) | Pre-baked builder image: heavy deps (Rust ext + py deps) installed, source overlaid per build | -| `unsupervised-main/publish.finder-builder.yml` | unsupervised-main (PR) | CI job that builds & pushes `harbor…/images/finder-builder` (reuses existing Harbor creds) | -| `build-finder-in-pod.sh` | panopticon image layer (`/usr/local/bin`) | Agent-run: overlay current src into a builder pod, `pyinstaller`, copy binary out | -| `image-layer.head.dockerfile` | panopticon `$CONFIG/layers/` | **⚠ pending — see below.** kubectl + auto-wiring wrapper | -| `apply.sh` | operator runs once | **⚠ pending — see below.** wire config + scope the SA | - -## ⚠ Two credential-handling files still to write (need your OK) -The auto-mode classifier blocked writing files that **decode your prod kubeconfig secret**, which is -correct — they touch a credential. They are: -1. **`image-layer.head.dockerfile`** — installs `kubectl` and a wrapper that lazily materializes - `~/.kube/config` from `$PROD_REPRO_KUBECONFIG_B64` on first use (works under `bash -c`; no host-hook - change; the existing `neutralize-claude-hooks.sh` stays as-is). -2. **`apply.sh`** — one-shot operator script: assemble the layer file (head + `build-finder-in-pod.sh` - as a heredoc) into `$CONFIG/layers/unsupervised-main.dockerfile`; `PATCH /repos/unsupervised-main` - with `image_layer_file`; **regenerate `PROD_REPRO_KUBECONFIG_B64`** to point at the `finder-repro` - namespace (backing up the old value first); rebuild the composed image. - -Approve those and I'll write them. - -## Apply sequence (once everything's written) -1. **PR** `Dockerfile.finder-builder` + `publish.finder-builder.yml` into unsupervised-main; run the CI - job once → `harbor…/images/finder-builder:latest` exists. -2. `kubectl --context apply -f finder-repro-rbac.yaml` (creates `finder-repro`). -3. Copy the pull secret into the new ns (no new account): - `kubectl --context -n default get secret unsupervised-regcred -o yaml \ - | sed 's/namespace: default/namespace: finder-repro/' \ - | kubectl --context -n finder-repro apply -f -` -4. `./apply.sh` (wires the layer + repo config; **rewrites the kubeconfig secret to finder-repro** — - the one prod-credential mutation, done with a backup). -5. Smoke test: `build-finder-in-pod.sh` produces a binary; provision a repro pod in `finder-repro`; - profile via `kubectl exec` (passive cgroup `cpu.stat`). - -## Profiling -No extra agent-side capability: it's `kubectl exec` into the test pod (cgroup `cpu.stat` + -table-completion progress; py-spy in-pod). See the finder-load perf notes. +| `finder-repro-rbac.yaml` | prod cluster (`kubectl apply`) | `finder-repro` ns + `panopticon-repro` (control) + `finder-test` (run) SAs + Role + ResourceQuota/LimitRange | +| `iam-finder-repro-readonly.json` | **AWS (operator creates)** | IAM role templates: trust (EKS OIDC + `finder-repro:finder-test`) + read-only s3 on `unsupervised-prod-internal/internal/*` | +| `unsupervised-main/Dockerfile.finder-builder` | unsupervised-main (PR) | Pre-baked builder image (deps installed, source overlaid per build) | +| `unsupervised-main/publish.finder-builder.yml` | unsupervised-main (PR) | CI publish job (reuses existing `HARBOR_USERNAME`/`HARBOR_PASSWORD`) | +| `image-layer.head.dockerfile` | `$CONFIG/layers/` (via apply.sh) | kubectl + auto-wiring wrapper (materializes kubeconfig from the injected env var) | +| `build-finder-in-pod.sh` | image layer `/usr/local/bin` | Agent-run in-pod build | +| `repro-pod.template.yaml` | reference | A finder test pod running as `finder-test` (IRSA S3) | +| `prod-testing-gate.md` | unsupervised-main AGENTS.md | The turn-handoff approval rule agents must follow | +| `apply.sh` | operator runs | `config` (wire layer+repo) / `scope-sa` (prod RBAC + rewrite kubeconfig secret) | + +## Guardrail = policy + backstop +- **Policy (turn-handoff, `prod-testing-gate.md`):** before any pod in `finder-repro`, the agent + ends its turn with a proposal (change, pod size, **which exports it reads**, what it measures); + you approve in the dashboard. Trust-based, per-run, full context. +- **Backstop (enforced by the API server):** `ResourceQuota` caps the namespace (8 pods / 128 CPU + / 600Gi / 1000Gi scratch), `LimitRange` caps any one pod (64 CPU / 300Gi / 500Gi). IRSA is + read-only, one bucket. So worst case, even if the policy is ignored, the blast radius is bounded. + +## Apply sequence +1. **PR** `Dockerfile.finder-builder` + `publish.finder-builder.yml` into unsupervised-main; run the + CI job once -> `harbor…/images/finder-builder:latest` exists. +2. **AWS (you):** create IAM role `prod-finder-repro-readonly` from `iam-finder-repro-readonly.json`; + uncomment the `finder-test` SA's `role-arn` annotation in `finder-repro-rbac.yaml`. +3. `apply.sh config` — assemble+install the image layer, PATCH repo config, force image rebuild. +4. `kubectl --context apply -f finder-repro-rbac.yaml`; copy the pull secret into the ns: + `kubectl -n default get secret unsupervised-regcred -o yaml | sed 's/namespace: default/namespace: finder-repro/' | kubectl -n finder-repro apply -f -` + (or just run `apply.sh scope-sa`, which does the apply + pull-secret copy + kubeconfig rewrite). +5. `apply.sh scope-sa` — mint a `finder-repro`-scoped kubeconfig, **test it (probe pod) before + overwriting**, back up the old value, rewrite `PROD_REPRO_KUBECONFIG_B64`. +6. Add `prod-testing-gate.md` to unsupervised-main's AGENTS.md. +7. Smoke test: `build-finder-in-pod.sh` -> binary; provision a `repro-pod.template.yaml` pod; profile. ## Rebuild cadence -The pre-baked image only needs rebuilding when the **heavy deps** change (requirements.txt / Rust ext / -python-utils) — ordinary `fc.py` edits are overlaid at build-in-pod time, so day-to-day this is free. +The pre-baked image only rebuilds when the **heavy deps** change (requirements.txt / Rust ext / +python-utils). Ordinary `fc.py` edits are overlaid at build-in-pod time — day-to-day this is free. diff --git a/deploy/unsupervised-main-prod-testing/apply.sh b/deploy/unsupervised-main-prod-testing/apply.sh new file mode 100644 index 00000000..3d516c29 --- /dev/null +++ b/deploy/unsupervised-main-prod-testing/apply.sh @@ -0,0 +1,130 @@ +#!/usr/bin/env bash +# One-shot operator setup to enable unsupervised-main agents' finder prod-testing. +# +# apply.sh config # wire the image layer + repo config (idempotent, non-prod) +# apply.sh scope-sa # prod: create finder-repro RBAC, mint a scoped kubeconfig, +# # and REWRITE the PROD_REPRO_KUBECONFIG_B64 secret (backed up first) +# apply.sh all # config then scope-sa +# +# scope-sa is the only step that mutates prod + a credential. It TESTS the new kubeconfig +# (creates+deletes a probe pod in finder-repro) and only overwrites the secret if the test +# passes; the old secret value is backed up to a timestamped file first. +set -euo pipefail + +HERE="$(cd "$(dirname "$0")" && pwd)" +PANOPTICON_CONFIG="${PANOPTICON_CONFIG:-$HOME/.config/panopticon}" +LAYERS_DIR="$PANOPTICON_CONFIG/layers" +SECRETS_DIR="$PANOPTICON_CONFIG/secrets" +ENV_FILE="$SECRETS_DIR/unsupervised_main.env" +SVC="${PANOPTICON_SERVICE_URL:-http://localhost:8000}" +PROD_CTX="${PROD_CTX:-arn:aws:eks:us-east-1:037004398141:cluster/prod}" +NS="finder-repro" +SA="panopticon-repro" +LAYER_NAME="unsupervised-main.dockerfile" + +log() { printf '\n== %s ==\n' "$*"; } + +config() { + log "assembling image layer -> $LAYERS_DIR/$LAYER_NAME" + mkdir -p "$LAYERS_DIR" + { + cat "$HERE/image-layer.head.dockerfile" + printf '\n# --- inlined build-finder-in-pod.sh (build context is Dockerfile-only, so no COPY) ---\n' + printf "RUN cat > /usr/local/bin/build-finder-in-pod.sh <<'FINDERBUILD' && chmod 0755 /usr/local/bin/build-finder-in-pod.sh\n" + cat "$HERE/build-finder-in-pod.sh" + printf '\nFINDERBUILD\n' + } > "$LAYERS_DIR/$LAYER_NAME" + + log "PATCH repo config: image_layer_file=$LAYER_NAME" + curl -fsS -X PATCH "$SVC/repos/unsupervised-main" \ + -H 'Content-Type: application/json' \ + -d "{\"image_layer_file\": \"$LAYER_NAME\"}" >/dev/null + echo " ok" + + log "forcing image rebuild (drop cached composed tags; runner rebuilds on next spawn)" + for wf in github-peer-reviewed github-self-reviewed; do + docker rmi "panopticon-$wf-unsupervised-main" 2>/dev/null || true + done + echo " dropped; a fresh task spawn will compose base -> workflow -> repo layer." +} + +scope_sa() { + log "applying scoped RBAC to prod ($NS)" + kubectl --context "$PROD_CTX" apply -f "$HERE/finder-repro-rbac.yaml" + + log "copying pull secret unsupervised-regcred into $NS" + kubectl --context "$PROD_CTX" -n default get secret unsupervised-regcred -o yaml \ + | sed -e 's/^ namespace: default/ namespace: '"$NS"'/' \ + -e '/resourceVersion:/d' -e '/uid:/d' -e '/creationTimestamp:/d' \ + | kubectl --context "$PROD_CTX" -n "$NS" apply -f - + + log "minting a long-lived token secret for $NS/$SA" + kubectl --context "$PROD_CTX" -n "$NS" apply -f - </dev/null || true)" + [ -n "$tok" ] && break; sleep 1 + done + [ -n "$tok" ] || { echo "token secret never populated" >&2; exit 3; } + + log "building the scoped kubeconfig" + server="$(kubectl --context "$PROD_CTX" config view --minify -o jsonpath='{.clusters[0].cluster.server}')" + ca="$(kubectl --context "$PROD_CTX" -n "$NS" get secret "${SA}-token" -o jsonpath='{.data.ca\.crt}')" + token="$(printf '%s' "$tok" | base64 -d)" + newkc="$(mktemp)"; trap 'rm -f "$newkc"' RETURN + cat > "$newkc" </dev/null + kubectl --kubeconfig "$newkc" -n "$NS" run rbac-probe --image=public.ecr.aws/docker/library/busybox:latest \ + --restart=Never --command -- sh -c 'exit 0' >/dev/null + kubectl --kubeconfig "$newkc" -n "$NS" delete pod rbac-probe --wait=false >/dev/null 2>&1 || true + echo " new kubeconfig works in $NS." + + log "backing up + rewriting PROD_REPRO_KUBECONFIG_B64 in $ENV_FILE" + cp -p "$ENV_FILE" "$ENV_FILE.bak-$(date -u +%Y%m%dT%H%M%SZ)" + newval="$(base64 < "$newkc" | tr -d '\n')" + # replace just that one line (values may contain '/', so use a python rewrite, not sed) + ENVFILE="$ENV_FILE" NEWVAL="$newval" python3 - <<'PY' +import os +p=os.environ["ENVFILE"]; nv=os.environ["NEWVAL"] +lines=open(p).read().splitlines(); out=[]; done=False +for ln in lines: + if ln.startswith("PROD_REPRO_KUBECONFIG_B64="): + out.append(f"PROD_REPRO_KUBECONFIG_B64={nv}"); done=True + else: out.append(ln) +assert done, "PROD_REPRO_KUBECONFIG_B64 line not found" +open(p,"w").write("\n".join(out)+"\n") +print(" rewrote PROD_REPRO_KUBECONFIG_B64 (backup kept)") +PY + echo " done. New task spawns will use the finder-repro-scoped credential." +} + +case "${1:-all}" in + config) config ;; + scope-sa) scope_sa ;; + all) config; scope_sa ;; + *) echo "usage: apply.sh [config|scope-sa|all]" >&2; exit 2 ;; +esac diff --git a/deploy/unsupervised-main-prod-testing/finder-repro-rbac.yaml b/deploy/unsupervised-main-prod-testing/finder-repro-rbac.yaml index 7de68b00..c1643b8b 100644 --- a/deploy/unsupervised-main-prod-testing/finder-repro-rbac.yaml +++ b/deploy/unsupervised-main-prod-testing/finder-repro-rbac.yaml @@ -23,6 +23,24 @@ metadata: name: panopticon-repro namespace: finder-repro --- +# Run identity for finder TEST PODS (separate from the control SA above). The pods an +# agent provisions set `serviceAccountName: finder-test` so they can READ prod S3 exports +# via IRSA. Data access rides on this SA, NOT on kubectl / the control credential. +# +# ACTIVATION (operator, AWS-side — see iam-finder-repro-readonly.json): +# 1. Create IAM role `prod-finder-repro-readonly` (trust = EKS OIDC + sub +# system:serviceaccount:finder-repro:finder-test; permissions = read-only s3 on +# unsupervised-prod-internal/internal/*). +# 2. Uncomment the annotation below with that role's ARN and re-apply. +# Until then this SA exists but has no data access (small/synthetic tests still work). +apiVersion: v1 +kind: ServiceAccount +metadata: + name: finder-test + namespace: finder-repro + # annotations: + # eks.amazonaws.com/role-arn: arn:aws:iam::037004398141:role/prod-finder-repro-readonly +--- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: diff --git a/deploy/unsupervised-main-prod-testing/iam-finder-repro-readonly.json b/deploy/unsupervised-main-prod-testing/iam-finder-repro-readonly.json new file mode 100644 index 00000000..12b198dc --- /dev/null +++ b/deploy/unsupervised-main-prod-testing/iam-finder-repro-readonly.json @@ -0,0 +1,51 @@ +{ + "_README": [ + "Templates for the IAM role that lets finder TEST PODS read prod S3 exports via IRSA.", + "Operator creates this AWS-side (agents cannot). Read-only, one bucket, scoped to the", + "finder-repro:finder-test ServiceAccount. Then uncomment the role-arn annotation on the", + "finder-test SA in finder-repro-rbac.yaml and re-apply.", + "", + " ROLE=prod-finder-repro-readonly", + " aws iam create-role --role-name $ROLE --assume-role-policy-document file://", + " aws iam put-role-policy --role-name $ROLE --policy-name s3-read --policy-document file://", + "", + "Scope the permission_policy Resource to specific export prefixes if you want to expose", + "only curated staged exports rather than the whole internal/ prefix." + ], + "trust_policy": { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "Federated": "arn:aws:iam::037004398141:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/5182D36C43F8988418E19551325D9F69" + }, + "Action": "sts:AssumeRoleWithWebIdentity", + "Condition": { + "StringEquals": { + "oidc.eks.us-east-1.amazonaws.com/id/5182D36C43F8988418E19551325D9F69:sub": "system:serviceaccount:finder-repro:finder-test", + "oidc.eks.us-east-1.amazonaws.com/id/5182D36C43F8988418E19551325D9F69:aud": "sts.amazonaws.com" + } + } + } + ] + }, + "permission_policy": { + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "ListExports", + "Effect": "Allow", + "Action": ["s3:ListBucket"], + "Resource": "arn:aws:s3:::unsupervised-prod-internal", + "Condition": { "StringLike": { "s3:prefix": ["internal/*"] } } + }, + { + "Sid": "ReadExports", + "Effect": "Allow", + "Action": ["s3:GetObject"], + "Resource": "arn:aws:s3:::unsupervised-prod-internal/internal/*" + } + ] + } +} diff --git a/deploy/unsupervised-main-prod-testing/image-layer.head.dockerfile b/deploy/unsupervised-main-prod-testing/image-layer.head.dockerfile new file mode 100644 index 00000000..11ca94d6 --- /dev/null +++ b/deploy/unsupervised-main-prod-testing/image-layer.head.dockerfile @@ -0,0 +1,35 @@ +# Repo image layer (ADR 0005) for unsupervised-main — finder prod-testing tooling. +# +# This is a Dockerfile *fragment*, not a full Dockerfile: sessionservice/images.py composes +# `FROM \n\n` into one image, and the build context is a temp dir +# holding ONLY the generated Dockerfile — so there is NO COPY. Extra files are inlined via RUN. +# apply.sh appends build-finder-in-pod.sh (as a heredoc) after this head and installs the result +# to $PANOPTICON_CONFIG/layers/unsupervised-main.dockerfile. +# +# The base ends on `USER root` and ships curl + ca-certificates, so kubectl installs cleanly; +# the base entrypoint drops back to the `panopticon` user at runtime. + +USER root + +# --- kubectl (real binary), arch-matched to the container --- +ARG KUBECTL_VERSION=v1.30.5 +RUN arch="$(dpkg --print-architecture)" \ + && curl -fsSLo /usr/local/bin/kubectl.real \ + "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/${arch}/kubectl" \ + && chmod 0755 /usr/local/bin/kubectl.real \ + && /usr/local/bin/kubectl.real version --client=true --output=yaml >/dev/null + +# --- kubectl wrapper: lazily materialize ~/.kube/config from the injected +# PROD_REPRO_KUBECONFIG_B64 env var on first use. Works under `bash -c` +# (Claude's Bash tool) — no login-shell / profile.d dependency. The env var is +# injected by the runner from the repo env-file; it is never baked into the image. --- +RUN cat > /usr/local/bin/kubectl <<'WRAP' && chmod 0755 /usr/local/bin/kubectl +#!/usr/bin/env bash +set -euo pipefail +cfg="${KUBECONFIG:-$HOME/.kube/config}" +if [ ! -s "$cfg" ] && [ -n "${PROD_REPRO_KUBECONFIG_B64:-}" ]; then + mkdir -p "$(dirname "$cfg")" + ( umask 077; printf '%s' "$PROD_REPRO_KUBECONFIG_B64" | base64 -d > "$cfg" ) +fi +exec /usr/local/bin/kubectl.real "$@" +WRAP diff --git a/deploy/unsupervised-main-prod-testing/prod-testing-gate.md b/deploy/unsupervised-main-prod-testing/prod-testing-gate.md new file mode 100644 index 00000000..52507433 --- /dev/null +++ b/deploy/unsupervised-main-prod-testing/prod-testing-gate.md @@ -0,0 +1,35 @@ +# Prod-testing approval gate (agent instruction) + +> Add this to unsupervised-main's agent instructions (AGENTS.md / CLAUDE.md) so any task +> agent working the repo sees it. It is the *policy* half of the guardrail; the +> `finder-repro` ResourceQuota/LimitRange is the *technical* backstop. + +You have the ability to build a modified finder binary and run it in a prod test pod +(`build-finder-in-pod.sh`, `kubectl` scoped to the `finder-repro` namespace). This reads +and runs against **production data**. Treat it as a privileged action. + +## When to use it +Only when a change's correctness or performance **can only be shown empirically** and a unit +test can't — e.g. reproducing a data-scale bug (the >2 GB string overflow shape) or measuring +a load/finding perf number. Prefer a cheap **synthetic input** that isolates the mechanism over +pulling a full prod export whenever that suffices. + +## MUST: pause for operator approval before any prod pod +Before you `kubectl run`/`apply` **any** pod in `finder-repro`, **end your turn and hand it to +the operator** with a concrete proposal: +- what change / which binary, +- pod size (cpu + memory) and expected wall-clock, +- **which prod exports it will read** (the S3 URIs / run ids), +- what it measures and the pass/fail criterion. + +Do not create the pod until the operator advances the turn back to you approving it. A denial +means don't run it. This is the same review surface as a plan review. + +## Bounds you operate within (enforced regardless) +- Namespace `finder-repro` only; ResourceQuota caps totals (8 pods / 128 CPU / 600Gi / 1000Gi + scratch) and LimitRange caps any single pod (64 CPU / 300Gi / 500Gi). The API server rejects + anything over. +- Data access is **read-only** on `unsupervised-prod-internal` via the `finder-test` SA. You + cannot write prod data or re-export from the warehouse — if a test needs fresh/other data, + ask the operator to stage the export; do not attempt to generate it yourself. +- Always delete your test pods when done. diff --git a/deploy/unsupervised-main-prod-testing/repro-pod.template.yaml b/deploy/unsupervised-main-prod-testing/repro-pod.template.yaml new file mode 100644 index 00000000..04e3909f --- /dev/null +++ b/deploy/unsupervised-main-prod-testing/repro-pod.template.yaml @@ -0,0 +1,44 @@ +# Template: a finder repro/test pod an agent provisions in finder-repro. +# Runs as `finder-test` (IRSA -> read-only prod S3 exports). The agent kubectl-cp's the +# binary it built (build-finder-in-pod.sh) into the pod, then execs it against a config +# that points at real S3 export URIs. Size within the ResourceQuota ceiling. +# +# The agent fills: , memory/cpu, and the finder config (feature_ids/file_locations +# pointing at s3://unsupervised-prod-internal/internal/... exports the operator staged). +apiVersion: v1 +kind: Pod +metadata: + name: # e.g. finder-repro- + namespace: finder-repro + labels: + app.kubernetes.io/managed-by: panopticon + purpose: finder-repro +spec: + restartPolicy: Never + serviceAccountName: finder-test # IRSA: read-only prod S3 exports + imagePullSecrets: + - name: unsupervised-regcred # Harbor pull (copied into finder-repro) + containers: + - name: finder + # A base image just to hold/exec the cp'd binary; the binary is self-contained + # (PyInstaller). Any image with a shell works; reuse the prod image for parity. + image: harbor.unsupervised.com/images/unsupervised-main:latest + command: ["sleep", "36000"] + env: + - { name: FINDER_INPUT_DATA_MEMORY_BUDGET_GB, value: "200" } + - { name: TMPDIR, value: "/scratch" } + - { name: FINDER_WORKERS, value: "52" } + resources: + # Stay under the LimitRange max (64 CPU / 300Gi mem / 500Gi ephemeral). + requests: { cpu: "16", memory: 200Gi } + limits: { cpu: "32", memory: 256Gi } + volumeMounts: + - { name: scratch, mountPath: /scratch } + volumes: + - name: scratch + emptyDir: { sizeLimit: 400Gi } +# Agent workflow, once Ready: +# kubectl -n finder-repro cp ./finder :/scratch/finder +# kubectl -n finder-repro cp ./cfg.json :/scratch/cfg.json +# kubectl -n finder-repro exec -- bash -lc 'chmod +x /scratch/finder && /scratch/finder -i /scratch/cfg.json' +# # profile passively: kubectl -n finder-repro exec -- cat /sys/fs/cgroup/cpu.stat From 0c38ce02dc1e271b7cbef73382c843282dd059c8 Mon Sep 17 00:00:00 2001 From: Dimitri Krattiger Date: Tue, 22 Sep 2026 09:37:42 -0600 Subject: [PATCH 3/6] deploy: rework for broad-role interim (default ns, no IAM) Neither the operator nor I have IAM-admin on prod (SSO grants only Unsupervised-Engineer, which can't iam:CreateRole), so the scoped finder-repro namespace is deferred. Interim uses the broad prod-unsupervised-main role by running test pods in default as unsupervised-unsupervised -- zero cluster/credential/IAM changes (panopticon-repro already has pod-create in default; its kubeconfig already points there; unsupervised-unsupervised + unsupervised-regcred already exist). - build-finder-in-pod.sh / repro-pod.template.yaml: target default / unsupervised-unsupervised. - apply.sh: config is the interim default (layer + repo PATCH only); scope-sa gated as phase-2. - prod-testing-gate.md: default ns; the turn-handoff is now the ONLY guardrail (no quota) so approval is emphasized as mandatory. - finder-repro-rbac.yaml + iam-finder-repro-readonly.json: banner-marked PHASE 2 (needs IAM-admin). - README: two-phase framing. Trade-off accepted by operator: broad role + no compute quota in the interim; tighten to the scoped read-only role + quota once an IAM-admin can create it. Co-Authored-By: Claude Opus 4.8 --- .../unsupervised-main-prod-testing/README.md | 17 +++++-- .../unsupervised-main-prod-testing/apply.sh | 27 +++++----- .../build-finder-in-pod.sh | 4 +- .../finder-repro-rbac.yaml | 6 +++ .../prod-testing-gate.md | 49 +++++++++---------- .../repro-pod.template.yaml | 31 +++++++----- 6 files changed, 79 insertions(+), 55 deletions(-) diff --git a/deploy/unsupervised-main-prod-testing/README.md b/deploy/unsupervised-main-prod-testing/README.md index 026ffb6c..14eb2333 100644 --- a/deploy/unsupervised-main-prod-testing/README.md +++ b/deploy/unsupervised-main-prod-testing/README.md @@ -3,10 +3,19 @@ Give unsupervised-main task agents the ability to **provision test pods, generate the finder executable, and profile results** against production data — the workflow currently done by hand. -Design (operator-chosen): **in-pod finder build (no Docker-in-Docker)** from a **pre-baked Harbor -builder image**; repro ServiceAccount **scoped to a dedicated `finder-repro` namespace**; prod -**data read-only via a scoped IRSA SA**; every prod run gated by an **operator turn-handoff**, with -the namespace **ResourceQuota as the enforced backstop**. +Two phases: +- **Interim (now, broad role):** test pods run in **`default`** as `unsupervised-unsupervised` (the + existing `prod-unsupervised-main` IRSA role) to read prod exports. **No cluster/credential/IAM + changes** — everything needed already exists in `default`. Trade-off: agents get the broad + (read+write) role and there is **no ResourceQuota** — the operator turn-handoff gate is the only + guardrail. Files marked "phase 2" below are NOT used here. +- **Phase 2 (later, needs IAM-admin):** scoped `finder-repro` namespace + read-only IRSA SA + + ResourceQuota. Blocked until an IAM-admin creates the scoped role (the `Unsupervised-Engineer` SSO + role can't `iam:CreateRole`). `finder-repro-rbac.yaml` + `iam-finder-repro-readonly.json` + `apply.sh + scope-sa` are the phase-2 path, kept ready. + +Common to both: **in-pod finder build (no Docker-in-Docker)** from a pre-baked Harbor builder image, +and every prod run gated by an **operator turn-handoff**. ## Two identities (keep them straight) - **Control** — `panopticon-repro` SA: creates/execs pods (RBAC). No data access. diff --git a/deploy/unsupervised-main-prod-testing/apply.sh b/deploy/unsupervised-main-prod-testing/apply.sh index 3d516c29..857ecc96 100644 --- a/deploy/unsupervised-main-prod-testing/apply.sh +++ b/deploy/unsupervised-main-prod-testing/apply.sh @@ -1,14 +1,15 @@ #!/usr/bin/env bash # One-shot operator setup to enable unsupervised-main agents' finder prod-testing. # -# apply.sh config # wire the image layer + repo config (idempotent, non-prod) -# apply.sh scope-sa # prod: create finder-repro RBAC, mint a scoped kubeconfig, -# # and REWRITE the PROD_REPRO_KUBECONFIG_B64 secret (backed up first) -# apply.sh all # config then scope-sa -# -# scope-sa is the only step that mutates prod + a credential. It TESTS the new kubeconfig -# (creates+deletes a probe pod in finder-repro) and only overwrites the secret if the test -# passes; the old secret value is backed up to a timestamped file first. +# apply.sh config # INTERIM (default): wire the image layer + repo config only. +# # No cluster/credential changes — agents run test pods in `default` +# # as unsupervised-unsupervised (broad prod-unsupervised-main IRSA), +# # and panopticon-repro's existing kubeconfig already grants pod-create +# # in `default`. This is all that's needed for the broad-role interim. +# apply.sh scope-sa # PHASE 2 (needs an IAM-admin first): create the finder-repro RBAC, +# # mint a finder-repro-scoped kubeconfig, and REWRITE +# # PROD_REPRO_KUBECONFIG_B64 (backed up first). Requires the scoped IAM +# # role + finder-test SA annotation to exist — do NOT run until then. set -euo pipefail HERE="$(cd "$(dirname "$0")" && pwd)" @@ -122,9 +123,11 @@ PY echo " done. New task spawns will use the finder-repro-scoped credential." } -case "${1:-all}" in +case "${1:-config}" in config) config ;; - scope-sa) scope_sa ;; - all) config; scope_sa ;; - *) echo "usage: apply.sh [config|scope-sa|all]" >&2; exit 2 ;; + scope-sa) + echo "PHASE 2: only run scope-sa after an IAM-admin has created prod-finder-repro-readonly" + echo "and you've uncommented the finder-test SA role-arn. Ctrl-C now if that's not done." >&2 + sleep 5; scope_sa ;; + *) echo "usage: apply.sh [config|scope-sa] (config = broad-role interim; scope-sa = phase 2)" >&2; exit 2 ;; esac diff --git a/deploy/unsupervised-main-prod-testing/build-finder-in-pod.sh b/deploy/unsupervised-main-prod-testing/build-finder-in-pod.sh index 1684d1b6..29eaea34 100644 --- a/deploy/unsupervised-main-prod-testing/build-finder-in-pod.sh +++ b/deploy/unsupervised-main-prod-testing/build-finder-in-pod.sh @@ -13,7 +13,9 @@ set -euo pipefail SRC="subrepos/finder" OUT="./finder" -NS="finder-repro" +# Interim: pods run in `default` (broad prod-unsupervised-main IRSA via unsupervised-unsupervised). +# Phase 2 (once an IAM-admin provisions the scoped role): switch to NS=finder-repro. +NS="default" IMAGE="${FINDER_BUILDER_IMAGE:-harbor.unsupervised.com/images/finder-builder:latest}" POD="" REBUILD_RUST=0 diff --git a/deploy/unsupervised-main-prod-testing/finder-repro-rbac.yaml b/deploy/unsupervised-main-prod-testing/finder-repro-rbac.yaml index c1643b8b..e2800be3 100644 --- a/deploy/unsupervised-main-prod-testing/finder-repro-rbac.yaml +++ b/deploy/unsupervised-main-prod-testing/finder-repro-rbac.yaml @@ -1,3 +1,9 @@ +# ============================================================================ +# PHASE 2 — NOT USED IN THE BROAD-ROLE INTERIM. Do NOT apply this until an +# IAM-admin has created the prod-finder-repro-readonly role (iam-finder-repro- +# readonly.json) and you've uncommented the finder-test SA's role-arn. Until +# then, agents run test pods in `default` as unsupervised-unsupervised. +# ============================================================================ # Scoped RBAC for panopticon agents doing finder prod-testing. # Applies to the prod EKS cluster. Replaces the current arrangement where the # `panopticon-repro` ServiceAccount lives in `default` with pod create/delete/exec/log. diff --git a/deploy/unsupervised-main-prod-testing/prod-testing-gate.md b/deploy/unsupervised-main-prod-testing/prod-testing-gate.md index 52507433..db48a5a7 100644 --- a/deploy/unsupervised-main-prod-testing/prod-testing-gate.md +++ b/deploy/unsupervised-main-prod-testing/prod-testing-gate.md @@ -1,35 +1,34 @@ # Prod-testing approval gate (agent instruction) -> Add this to unsupervised-main's agent instructions (AGENTS.md / CLAUDE.md) so any task -> agent working the repo sees it. It is the *policy* half of the guardrail; the -> `finder-repro` ResourceQuota/LimitRange is the *technical* backstop. +> Add to unsupervised-main's agent instructions (AGENTS.md / CLAUDE.md). In the interim +> (broad role, pods in `default`) this turn-handoff is the **only** guardrail — there is no +> namespace ResourceQuota to fall back on — so it is not optional. -You have the ability to build a modified finder binary and run it in a prod test pod -(`build-finder-in-pod.sh`, `kubectl` scoped to the `finder-repro` namespace). This reads -and runs against **production data**. Treat it as a privileged action. +You can build a modified finder binary and run it in a prod test pod (`build-finder-in-pod.sh`, +`kubectl`). Interim: these pods run in the **`default` namespace** as `unsupervised-unsupervised`, +which has **broad production access** (read+write via the main-app IRSA role) and runs **alongside +real prod workloads with no resource quota**. Treat this as a high-privilege, high-blast-radius +action. ## When to use it -Only when a change's correctness or performance **can only be shown empirically** and a unit -test can't — e.g. reproducing a data-scale bug (the >2 GB string overflow shape) or measuring -a load/finding perf number. Prefer a cheap **synthetic input** that isolates the mechanism over -pulling a full prod export whenever that suffices. +Only when a change's correctness or performance **can only be shown empirically** and a unit test +can't — a data-scale bug repro or a load/finding perf number. Strongly prefer a cheap **synthetic +input** that isolates the mechanism (no prod data, no big pod) whenever it suffices. -## MUST: pause for operator approval before any prod pod -Before you `kubectl run`/`apply` **any** pod in `finder-repro`, **end your turn and hand it to -the operator** with a concrete proposal: +## MUST: pause for operator approval before ANY prod pod +Before you `kubectl run`/`apply` any pod, **end your turn and hand it to the operator** with: - what change / which binary, -- pod size (cpu + memory) and expected wall-clock, -- **which prod exports it will read** (the S3 URIs / run ids), +- **pod size (cpu + memory) and expected wall-clock** — there is no quota clamp, so this is your + only bound; keep it as small as the test allows, +- **which prod exports it will read** (S3 URIs / run ids), - what it measures and the pass/fail criterion. -Do not create the pod until the operator advances the turn back to you approving it. A denial -means don't run it. This is the same review surface as a plan review. +Do not create the pod until the operator advances the turn back approving it. A denial means don't +run it. -## Bounds you operate within (enforced regardless) -- Namespace `finder-repro` only; ResourceQuota caps totals (8 pods / 128 CPU / 600Gi / 1000Gi - scratch) and LimitRange caps any single pod (64 CPU / 300Gi / 500Gi). The API server rejects - anything over. -- Data access is **read-only** on `unsupervised-prod-internal` via the `finder-test` SA. You - cannot write prod data or re-export from the warehouse — if a test needs fresh/other data, - ask the operator to stage the export; do not attempt to generate it yourself. -- Always delete your test pods when done. +## Rules while running +- Namespace `default` only; do not touch, delete, or exec into pods you did not create. +- Label every pod you create `app.kubernetes.io/managed-by=panopticon` so it's identifiable. +- **Always delete your test pods the moment you're done** — nothing else will reclaim them. +- Read-only intent: you are validating a change, not mutating prod. Do not write to prod buckets + or trigger re-exports; if a test needs fresh/other data, ask the operator to stage the export. diff --git a/deploy/unsupervised-main-prod-testing/repro-pod.template.yaml b/deploy/unsupervised-main-prod-testing/repro-pod.template.yaml index 04e3909f..4f24bfc9 100644 --- a/deploy/unsupervised-main-prod-testing/repro-pod.template.yaml +++ b/deploy/unsupervised-main-prod-testing/repro-pod.template.yaml @@ -1,7 +1,11 @@ -# Template: a finder repro/test pod an agent provisions in finder-repro. -# Runs as `finder-test` (IRSA -> read-only prod S3 exports). The agent kubectl-cp's the -# binary it built (build-finder-in-pod.sh) into the pod, then execs it against a config -# that points at real S3 export URIs. Size within the ResourceQuota ceiling. +# Template: a finder repro/test pod an agent provisions. +# +# INTERIM (broad role): runs in `default` as `unsupervised-unsupervised` — the existing +# prod-unsupervised-main IRSA role, so it can read prod S3 exports. No new cluster objects. +# PHASE 2 (scoped, needs IAM-admin): namespace: finder-repro, serviceAccountName: finder-test. +# +# NOTE: `default` has no ResourceQuota, so nothing but the operator turn-handoff gate bounds +# this in the interim. Keep pod sizes explicit (below) and ALWAYS delete pods when done. # # The agent fills: , memory/cpu, and the finder config (feature_ids/file_locations # pointing at s3://unsupervised-prod-internal/internal/... exports the operator staged). @@ -9,19 +13,19 @@ apiVersion: v1 kind: Pod metadata: name: # e.g. finder-repro- - namespace: finder-repro + namespace: default labels: app.kubernetes.io/managed-by: panopticon purpose: finder-repro spec: restartPolicy: Never - serviceAccountName: finder-test # IRSA: read-only prod S3 exports + serviceAccountName: unsupervised-unsupervised # broad IRSA -> prod S3 exports (interim) imagePullSecrets: - - name: unsupervised-regcred # Harbor pull (copied into finder-repro) + - name: unsupervised-regcred # Harbor pull (already in default) containers: - name: finder # A base image just to hold/exec the cp'd binary; the binary is self-contained - # (PyInstaller). Any image with a shell works; reuse the prod image for parity. + # (PyInstaller). Reuse the prod image for parity. image: harbor.unsupervised.com/images/unsupervised-main:latest command: ["sleep", "36000"] env: @@ -29,7 +33,7 @@ spec: - { name: TMPDIR, value: "/scratch" } - { name: FINDER_WORKERS, value: "52" } resources: - # Stay under the LimitRange max (64 CPU / 300Gi mem / 500Gi ephemeral). + # Keep these explicit — in `default` there is no quota to clamp a runaway pod. requests: { cpu: "16", memory: 200Gi } limits: { cpu: "32", memory: 256Gi } volumeMounts: @@ -38,7 +42,8 @@ spec: - name: scratch emptyDir: { sizeLimit: 400Gi } # Agent workflow, once Ready: -# kubectl -n finder-repro cp ./finder :/scratch/finder -# kubectl -n finder-repro cp ./cfg.json :/scratch/cfg.json -# kubectl -n finder-repro exec -- bash -lc 'chmod +x /scratch/finder && /scratch/finder -i /scratch/cfg.json' -# # profile passively: kubectl -n finder-repro exec -- cat /sys/fs/cgroup/cpu.stat +# kubectl -n default cp ./finder :/scratch/finder +# kubectl -n default cp ./cfg.json :/scratch/cfg.json +# kubectl -n default exec -- bash -lc 'chmod +x /scratch/finder && /scratch/finder -i /scratch/cfg.json' +# # profile passively: kubectl -n default exec -- cat /sys/fs/cgroup/cpu.stat +# kubectl -n default delete pod # ALWAYS clean up From 08f12d340a9fd5c04c4f57df04e6d679f4bd7432 Mon Sep 17 00:00:00 2001 From: Dimitri Krattiger Date: Tue, 22 Sep 2026 11:17:01 -0600 Subject: [PATCH 4/6] deploy: builder image carries no finder/Rust source (overlay + auto-recompile) Addresses the proprietary-source concern: the pre-baked builder image now bakes ONLY the toolchain + third-party deps + the compiled bfinder wheel + a RUST_SRC_HASH marker -- no finder Python source and no Rust source persist in Harbor (python-utils remains as an installed dep, unavoidable for pyinstaller). Strictly less than the prod image already ships. build-finder-in-pod.sh overlays the agent's current finder source at build time and recompiles the bfinder wheel ONLY when the overlaid Rust source hash differs from RUST_SRC_HASH -- so a Python-only change uses the fast baked wheel, and a Rust change is never tested against a stale wheel. --force-rust overrides. The overlaid source is transient (deleted with the pod). Co-Authored-By: Claude Opus 4.8 --- .../build-finder-in-pod.sh | 60 ++++++++++--------- .../Dockerfile.finder-builder | 52 +++++++++------- 2 files changed, 62 insertions(+), 50 deletions(-) diff --git a/deploy/unsupervised-main-prod-testing/build-finder-in-pod.sh b/deploy/unsupervised-main-prod-testing/build-finder-in-pod.sh index 29eaea34..61b3f1d3 100644 --- a/deploy/unsupervised-main-prod-testing/build-finder-in-pod.sh +++ b/deploy/unsupervised-main-prod-testing/build-finder-in-pod.sh @@ -1,24 +1,25 @@ #!/usr/bin/env bash # Build the finder PyInstaller binary in a k8s pod (NO Docker-in-Docker). # -# Baked into the panopticon repo image layer at /usr/local/bin/build-finder-in-pod.sh so -# agents can run it. Uses the pre-baked Harbor builder image (deps already installed); it -# only overlays the agent's CURRENT finder src and runs pyinstaller (~2 min), then copies -# the binary out. kubectl is auto-wired to the finder-repro namespace by the image-layer -# wrapper; the builder pod pulls via the unsupervised-regcred secret (copied into finder-repro). +# Baked into the panopticon repo image layer at /usr/local/bin. Uses the pre-baked Harbor builder +# image, which carries ONLY toolchain + deps + the compiled bfinder wheel (no finder/Rust source). +# This script overlays the agent's CURRENT finder source, recompiles the Rust wheel *iff* the Rust +# source changed (auto-detected via RUST_SRC_HASH — a Python-only change never recompiles; a Rust +# change is never tested stale), installs finder editable, and runs pyinstaller. kubectl is +# auto-wired to `default` (interim) by the image-layer wrapper; the builder pod pulls via +# unsupervised-regcred. # # Usage: build-finder-in-pod.sh [--src DIR] [--out FILE] [--ns NS] [--image IMG] -# [--name POD] [--rebuild-rust] [--keep] +# [--name POD] [--force-rust] [--keep] set -euo pipefail SRC="subrepos/finder" OUT="./finder" -# Interim: pods run in `default` (broad prod-unsupervised-main IRSA via unsupervised-unsupervised). -# Phase 2 (once an IAM-admin provisions the scoped role): switch to NS=finder-repro. +# Interim: default ns (broad prod-unsupervised-main IRSA). Phase 2: NS=finder-repro. NS="default" IMAGE="${FINDER_BUILDER_IMAGE:-harbor.unsupervised.com/images/finder-builder:latest}" POD="" -REBUILD_RUST=0 +FORCE_RUST=0 KEEP=0 while [ $# -gt 0 ]; do @@ -28,15 +29,13 @@ while [ $# -gt 0 ]; do --ns) NS="$2"; shift 2;; --image) IMAGE="$2"; shift 2;; --name) POD="$2"; shift 2;; - --rebuild-rust) REBUILD_RUST=1; shift;; + --force-rust) FORCE_RUST=1; shift;; # recompile the wheel even if the hash matches --keep) KEEP=1; shift;; *) echo "unknown arg: $1" >&2; exit 2;; esac done [ -d "$SRC/src" ] || { echo "no finder src at $SRC/src (run from the repo root, or pass --src)" >&2; exit 2; } -# Pod name is deterministic from the src content so concurrent agents don't collide, without -# needing $RANDOM (unavailable in some restricted shells): hash the tree listing. if [ -z "$POD" ]; then h="$(find "$SRC/src" -type f -printf '%P %s\n' 2>/dev/null | cksum | cut -d' ' -f1)" POD="finder-build-${h}" @@ -67,26 +66,33 @@ YAML echo "== waiting for Ready ==" kubectl -n "$NS" wait --for=condition=Ready "pod/$POD" --timeout=300s - -# Sanity: right image? kubectl -n "$NS" exec "$POD" -- test -f /build/FINDER_BUILDER_READY \ || { echo "pod is not a finder-builder image (missing /build/FINDER_BUILDER_READY)" >&2; exit 3; } -echo "== syncing current finder src into the pod (overlay editable install) ==" -tar -C "$SRC/src" -czf - . | kubectl -n "$NS" exec -i "$POD" -- \ - bash -c 'rm -rf /build/subrepos/finder/src && mkdir -p /build/subrepos/finder/src && tar -C /build/subrepos/finder/src -xzf -' - -if [ "$REBUILD_RUST" = 1 ]; then - echo "== (--rebuild-rust) syncing pybfinder + rebuilding the Rust ext ==" - tar -C "$SRC/pybfinder" -czf - . | kubectl -n "$NS" exec -i "$POD" -- \ - bash -c 'rm -rf /build/subrepos/finder/pybfinder && mkdir -p /build/subrepos/finder/pybfinder && tar -C /build/subrepos/finder/pybfinder -xzf -' - kubectl -n "$NS" exec "$POD" -- bash -lc 'cd /build/subrepos/finder/pybfinder && maturin build -r && pip install --force-reinstall target/wheels/*.whl' -fi +echo "== overlaying current finder source into the pod (transient — deleted with the pod) ==" +tar -C "$SRC" --exclude=venv --exclude=.venv --exclude=target --exclude=dist \ + --exclude=__pycache__ --exclude=.git -czf - . \ + | kubectl -n "$NS" exec -i "$POD" -- bash -c 'mkdir -p /build/subrepos/finder && tar -C /build/subrepos/finder -xzf -' -echo "== running pyinstaller in-pod ==" -kubectl -n "$NS" exec "$POD" -- bash -lc 'cd /build/subrepos/finder && sh pyinstaller.sh' +echo "== recompiling the bfinder wheel only if the Rust source changed ==" +kubectl -n "$NS" exec "$POD" -- bash -lc " + set -euo pipefail + cd /build/subrepos/finder + rust_hash() { find Cargo.toml Cargo.lock bfinder pybfinder -type f | sort | xargs sha256sum | sha256sum | cut -d' ' -f1; } + baked=\$(cat /build/RUST_SRC_HASH) + now=\$(rust_hash) + if [ '${FORCE_RUST}' = '1' ] || [ \"\$now\" != \"\$baked\" ]; then + echo ' Rust source changed (or --force-rust) -> recompiling wheel' + (cd pybfinder && maturin build -r && pip install --force-reinstall --no-deps target/wheels/*.whl) + else + echo ' Rust unchanged -> using baked wheel (fast path)' + fi + echo '== installing finder (editable) from overlaid src + pyinstaller ==' + pip install -e . --no-deps + sh pyinstaller.sh +" echo "== copying binary out -> $OUT ==" kubectl -n "$NS" cp "$POD:/build/subrepos/finder/dist/finder" "$OUT" chmod +x "$OUT" -echo "== done: $OUT ($(wc -c < "$OUT") bytes). NOTE: this is a linux/amd64 binary — run it in a pod, not on the agent host. ==" +echo "== done: $OUT ($(wc -c < "$OUT") bytes). linux/amd64 binary — run it in a pod, not on the agent host. ==" diff --git a/deploy/unsupervised-main-prod-testing/unsupervised-main/Dockerfile.finder-builder b/deploy/unsupervised-main-prod-testing/unsupervised-main/Dockerfile.finder-builder index eb29e89b..66e78f24 100644 --- a/deploy/unsupervised-main-prod-testing/unsupervised-main/Dockerfile.finder-builder +++ b/deploy/unsupervised-main-prod-testing/unsupervised-main/Dockerfile.finder-builder @@ -1,17 +1,18 @@ -# Pre-baked finder BUILDER image — heavy deps installed once, source overlaid per build. +# Pre-baked finder BUILDER image — toolchain + deps + compiled bfinder wheel ONLY. # -# Destined for the unsupervised-main repo (e.g. subrepos/finder/Dockerfile.finder-builder), -# published to Harbor by the companion CI job (publish.finder-builder.yml). Kept here in the -# panopticon bundle as the reviewable source; copy into unsupervised-main via a normal PR. +# Deliberately carries NO finder Python source and NO Rust source: the agent overlays the +# CURRENT finder source at build-in-pod time and runs pyinstaller. The Rust `bfinder`/`pybfinder` +# source is used here only to compile the wheel, then deleted — so nothing proprietary beyond the +# already-shipped compiled artifacts persists in Harbor. (python-utils remains as an installed +# runtime dep, unavoidable — pyinstaller bundles it.) # -# Reproduces the CANONICAL build (.github/workflows/build.finder.yml: Py3.10 + Rust + maturin -# + editable installs), but STOPS before `pyinstaller.sh`. Finder is installed editable, so at -# build-in-pod time the agent overlays the current src at the known path and runs pyinstaller -# (~2 min) instead of rebuilding the ~heavy deps + Rust ext. +# We record a hash of the Rust source (RUST_SRC_HASH). build-finder-in-pod.sh compares the +# overlaid source against it and recompiles the wheel iff the change touches the Rust — so a +# Python-only change uses the fast baked wheel, a Rust change is never tested stale. # -# Build context = repo root (needs both subrepos/finder and subrepos/python-utils, since -# finder's requirements.txt has `-e ../python-utils`). Build with submodules checked out: -# docker build -f subrepos/finder/Dockerfile.finder-builder -t /images/finder-builder: . +# Destined for unsupervised-main (e.g. .github/docker/finder-builder.Dockerfile), published to +# Harbor by publish.finder-builder.yml. Build context = repo root (needs subrepos/finder + +# subrepos/python-utils), with submodules checked out. FROM python:3.10-bookworm ENV DEBIAN_FRONTEND=noninteractive @@ -20,25 +21,30 @@ RUN apt-get update \ build-essential libpq-dev curl git ca-certificates pkg-config \ && rm -rf /var/lib/apt/lists/* -# Rust toolchain (pybfinder is a maturin/Rust extension). Pin to match CI's stable. RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable ENV PATH="/root/.cargo/bin:${PATH}" - RUN pip install --no-cache-dir maturin pyinstaller -# Bring in both subrepos at the SAME relative layout finder expects (`-e ../python-utils`). WORKDIR /build +# python-utils (finder's editable dep) — stays as an installed package. COPY subrepos/python-utils /build/subrepos/python-utils -COPY subrepos/finder /build/subrepos/finder +# Only the Rust source + requirements from finder — NOT the finder Python src. +COPY subrepos/finder/Cargo.toml subrepos/finder/Cargo.lock /build/subrepos/finder/ +COPY subrepos/finder/bfinder /build/subrepos/finder/bfinder +COPY subrepos/finder/pybfinder /build/subrepos/finder/pybfinder +COPY subrepos/finder/requirements.txt /build/subrepos/finder/requirements.txt WORKDIR /build/subrepos/finder -# Install python deps + editable python-utils + editable finder (from requirements.txt), -# then build & install the Rust extension. This is the expensive part we pre-bake. -RUN pip install --no-cache-dir -r requirements.txt \ - && (cd pybfinder && maturin build -r && pip install target/wheels/*.whl) +# Install third-party deps + editable python-utils, but NOT finder (`-e .`) — no finder src here. +RUN grep -v '^-e \.$' requirements.txt > requirements-nofinder.txt \ + && pip install --no-cache-dir -r requirements-nofinder.txt +# Compile + install the bfinder wheel from the Rust source. +RUN cd pybfinder && maturin build -r && pip install --no-cache-dir target/wheels/*.whl -# NOTE: no `pyinstaller.sh` here. The editable finder lives at /build/subrepos/finder/src; -# build-finder-in-pod.sh overlays the agent's current src there and runs pyinstaller. -# A marker the build script checks for, so a stale/wrong image fails loudly: -RUN echo "finder-builder: deps baked; overlay src at /build/subrepos/finder/src then run pyinstaller.sh" \ +# Record a content+path hash of the Rust source, then DELETE the Rust source (keep only the +# installed compiled wheel). Same hash recipe as build-finder-in-pod.sh's rust_hash(). +RUN find Cargo.toml Cargo.lock bfinder pybfinder -type f | sort | xargs sha256sum | sha256sum \ + | cut -d' ' -f1 > /build/RUST_SRC_HASH \ + && rm -rf bfinder pybfinder target Cargo.toml Cargo.lock requirements.txt requirements-nofinder.txt \ + && echo "finder-builder: toolchain + deps + bfinder wheel baked; overlay finder src at build time." \ > /build/FINDER_BUILDER_READY From 6fe7104bc158b1300c63c75e1296fae1312f4a97 Mon Sep 17 00:00:00 2001 From: Dimitri Krattiger Date: Tue, 22 Sep 2026 12:38:36 -0600 Subject: [PATCH 5/6] deploy: sync CI fixes into bundle (maturin -o wheel path; Dockerfile-specific .dockerignore) The published builder image required two CI fixes, now reflected here: - Dockerfile.finder-builder + build-finder-in-pod.sh: maturin build -r -o /tmp/wheels (the wheel lands in the Cargo *workspace* target, not pybfinder/target, so the old target/wheels/*.whl glob missed it). - finder-builder.Dockerfile.dockerignore: BuildKit uses this instead of the repo-root .dockerignore (which excludes subrepos/), so the build context includes subrepos/finder + subrepos/python-utils. finder-builder:latest is now published to Harbor (native amd64, no finder/Rust source). Co-Authored-By: Claude Opus 4.8 --- .../build-finder-in-pod.sh | 2 +- .../unsupervised-main/Dockerfile.finder-builder | 2 +- .../finder-builder.Dockerfile.dockerignore | 13 +++++++++++++ 3 files changed, 15 insertions(+), 2 deletions(-) create mode 100644 deploy/unsupervised-main-prod-testing/unsupervised-main/finder-builder.Dockerfile.dockerignore diff --git a/deploy/unsupervised-main-prod-testing/build-finder-in-pod.sh b/deploy/unsupervised-main-prod-testing/build-finder-in-pod.sh index 61b3f1d3..62041935 100644 --- a/deploy/unsupervised-main-prod-testing/build-finder-in-pod.sh +++ b/deploy/unsupervised-main-prod-testing/build-finder-in-pod.sh @@ -83,7 +83,7 @@ kubectl -n "$NS" exec "$POD" -- bash -lc " now=\$(rust_hash) if [ '${FORCE_RUST}' = '1' ] || [ \"\$now\" != \"\$baked\" ]; then echo ' Rust source changed (or --force-rust) -> recompiling wheel' - (cd pybfinder && maturin build -r && pip install --force-reinstall --no-deps target/wheels/*.whl) + (cd pybfinder && maturin build -r -o /tmp/wheels && pip install --force-reinstall --no-deps /tmp/wheels/*.whl) else echo ' Rust unchanged -> using baked wheel (fast path)' fi diff --git a/deploy/unsupervised-main-prod-testing/unsupervised-main/Dockerfile.finder-builder b/deploy/unsupervised-main-prod-testing/unsupervised-main/Dockerfile.finder-builder index 66e78f24..53992014 100644 --- a/deploy/unsupervised-main-prod-testing/unsupervised-main/Dockerfile.finder-builder +++ b/deploy/unsupervised-main-prod-testing/unsupervised-main/Dockerfile.finder-builder @@ -39,7 +39,7 @@ WORKDIR /build/subrepos/finder RUN grep -v '^-e \.$' requirements.txt > requirements-nofinder.txt \ && pip install --no-cache-dir -r requirements-nofinder.txt # Compile + install the bfinder wheel from the Rust source. -RUN cd pybfinder && maturin build -r && pip install --no-cache-dir target/wheels/*.whl +RUN cd pybfinder && maturin build -r -o /tmp/wheels && pip install --no-cache-dir /tmp/wheels/*.whl # Record a content+path hash of the Rust source, then DELETE the Rust source (keep only the # installed compiled wheel). Same hash recipe as build-finder-in-pod.sh's rust_hash(). diff --git a/deploy/unsupervised-main-prod-testing/unsupervised-main/finder-builder.Dockerfile.dockerignore b/deploy/unsupervised-main-prod-testing/unsupervised-main/finder-builder.Dockerfile.dockerignore new file mode 100644 index 00000000..cc56c437 --- /dev/null +++ b/deploy/unsupervised-main-prod-testing/unsupervised-main/finder-builder.Dockerfile.dockerignore @@ -0,0 +1,13 @@ +# Dockerfile-specific ignore for the finder-builder build. +# BuildKit uses `.dockerignore` (this file) INSTEAD of the repo-root .dockerignore, +# which excludes `subrepos/`. This build needs subrepos/finder + subrepos/python-utils in context, +# so we do NOT exclude them here — just trim heavy/irrelevant junk. +.git +**/.git +**/.venv +**/venv +**/target +**/node_modules +**/__pycache__ +**/*.pyc +**/dist From 6c2e55230e1797c9b968b1f9dc23314adde51332 Mon Sep 17 00:00:00 2001 From: Dimitri Krattiger Date: Thu, 24 Sep 2026 14:41:15 -0600 Subject: [PATCH 6/6] chore: add a DeepWork review rule for scope discipline MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PRs keep arriving with more in them than the task required — helpers with one call site, guards for states that can't occur, drive-by reformatting, comments restating the next line. Each is individually defensible, which is why they accumulate, and the cost lands on whoever reviews: attention is spent per line whether or not the line needed to exist. `unsupervised-main` already runs DeepWork Reviews (`.deepreview` files under app/ and test/, instructions under .deepwork/review/). This repo had none, so nothing was watching for it here. One rule, `all_changed_files`: the match is only a tripwire, and the reviewer gets the whole changeset — "is this diff bigger than the task required" is a question about the whole, not about any one file. Deliberately a single rule rather than several: each spawns its own sub-agent with real overhead, and scope creep is one judgement. The rule only ever argues for removing things. Correctness, coverage, and completeness belong to human review and to other rules; a rule that could argue in both directions would just relitigate the whole PR. Co-Authored-By: Claude Opus 5 (1M context) --- .deepreview | 84 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 84 insertions(+) create mode 100644 .deepreview diff --git a/.deepreview b/.deepreview new file mode 100644 index 00000000..ca6bc6cc --- /dev/null +++ b/.deepreview @@ -0,0 +1,84 @@ +# DeepWork review: scope discipline. +# +# Catches the failure mode where a PR does what was asked AND a pile of things +# that weren't — unrequested abstractions, defensive branches for impossible +# states, drive-by refactors, narration comments. Each one is individually +# defensible, which is why they accumulate; the cost lands on the reviewer, who +# pays attention per line regardless of whether the line needed to exist. +# +# `all_changed_files` (tripwire): the match is only the trigger. Any source +# change hands the reviewer the entire changeset, which is the only way to judge +# proportionality — "is this diff bigger than the task required" is a question +# about the whole, not about any one file. +# +# Deliberately ONE rule: each rule spawns its own sub-agent with real overhead, +# and scope creep is a single judgement, not several. + +scope-discipline: + description: | + Flag additions the task did not require: unrequested abstractions and + helpers, error handling for states that cannot occur, unrelated cleanup + bundled into an unrelated change, comments that restate the code, and + redundant tests. Proportionality of the diff to the stated task. + match: + include: + - "**/*.py" + - "**/*.md" + exclude: + - ".venv/**" + - "**/migrations/versions/**" + review: + strategy: all_changed_files + instructions: | + Judge this changeset against one question: **is everything here required + by the task, and is the diff proportionate to it?** + + Infer the task from the PR title, the branch name, and the commit + messages. If you genuinely cannot tell what was asked, say so and stop — + do not guess a narrower task and then flag everything outside it. + + Flag each of these, with `file:line` and a one-line reason: + + 1. **Unrequested abstraction** — a helper, wrapper, base class, or + indirection introduced for a single call site. A one-shot operation + usually does not need a helper. + 2. **Speculative generality** — parameters, config flags, or extension + points with exactly one caller and no stated requirement. Designing for + a hypothetical future requirement. + 3. **Impossible-state handling** — try/except, null guards, or fallbacks + for conditions the surrounding code or a framework guarantee already + rules out. Validate at system boundaries (user input, external APIs), + not between two functions in the same module. + 4. **Drive-by changes** — renames, reformatting, import reordering, or + refactors unrelated to the task. A bug fix does not need surrounding + cleanup. These are the most expensive kind of noise: they inflate the + diff while hiding the real change inside it. + 5. **Narration comments** — comments that restate what the next line does, + explain why the change is correct, or address the reviewer rather than + the next reader. A comment should state a constraint the code cannot + show. Anything that stops being true once the PR merges is noise. + 6. **Redundant tests** — a new test whose failure conditions are already + covered by an existing one. More tests is not automatically better; + each one is code to maintain. + 7. **Disproportionate documentation** — a multi-section document, summary + file, or expanded README where the change warranted a sentence. + + For each finding, state the concrete recommendation: **delete**, **split + into a separate PR**, or **keep** with the justification that makes it + required. Prefer "delete" — the default answer for something the task did + not ask for is that it should not be in this PR. + + Then give a single overall verdict: + + - **PROPORTIONATE** — the diff matches the task. + - **BLOATED** — name the two or three changes that would shrink it most, + and estimate the lines they would remove. + + Two things to get right, because getting them wrong makes this rule worse + than useless: + + - **Do not flag work the task actually required**, even where it is large. + A big diff is not itself a finding; a big diff for a small task is. + - **Do not flag missing work.** Other rules and human review cover + correctness, coverage, and completeness. This rule only ever argues for + removing things, never for adding them.