diff --git a/.deepreview b/.deepreview new file mode 100644 index 00000000..ca6bc6cc --- /dev/null +++ b/.deepreview @@ -0,0 +1,84 @@ +# DeepWork review: scope discipline. +# +# Catches the failure mode where a PR does what was asked AND a pile of things +# that weren't — unrequested abstractions, defensive branches for impossible +# states, drive-by refactors, narration comments. Each one is individually +# defensible, which is why they accumulate; the cost lands on the reviewer, who +# pays attention per line regardless of whether the line needed to exist. +# +# `all_changed_files` (tripwire): the match is only the trigger. Any source +# change hands the reviewer the entire changeset, which is the only way to judge +# proportionality — "is this diff bigger than the task required" is a question +# about the whole, not about any one file. +# +# Deliberately ONE rule: each rule spawns its own sub-agent with real overhead, +# and scope creep is a single judgement, not several. + +scope-discipline: + description: | + Flag additions the task did not require: unrequested abstractions and + helpers, error handling for states that cannot occur, unrelated cleanup + bundled into an unrelated change, comments that restate the code, and + redundant tests. Proportionality of the diff to the stated task. + match: + include: + - "**/*.py" + - "**/*.md" + exclude: + - ".venv/**" + - "**/migrations/versions/**" + review: + strategy: all_changed_files + instructions: | + Judge this changeset against one question: **is everything here required + by the task, and is the diff proportionate to it?** + + Infer the task from the PR title, the branch name, and the commit + messages. If you genuinely cannot tell what was asked, say so and stop — + do not guess a narrower task and then flag everything outside it. + + Flag each of these, with `file:line` and a one-line reason: + + 1. **Unrequested abstraction** — a helper, wrapper, base class, or + indirection introduced for a single call site. A one-shot operation + usually does not need a helper. + 2. **Speculative generality** — parameters, config flags, or extension + points with exactly one caller and no stated requirement. Designing for + a hypothetical future requirement. + 3. **Impossible-state handling** — try/except, null guards, or fallbacks + for conditions the surrounding code or a framework guarantee already + rules out. Validate at system boundaries (user input, external APIs), + not between two functions in the same module. + 4. **Drive-by changes** — renames, reformatting, import reordering, or + refactors unrelated to the task. A bug fix does not need surrounding + cleanup. These are the most expensive kind of noise: they inflate the + diff while hiding the real change inside it. + 5. **Narration comments** — comments that restate what the next line does, + explain why the change is correct, or address the reviewer rather than + the next reader. A comment should state a constraint the code cannot + show. Anything that stops being true once the PR merges is noise. + 6. **Redundant tests** — a new test whose failure conditions are already + covered by an existing one. More tests is not automatically better; + each one is code to maintain. + 7. **Disproportionate documentation** — a multi-section document, summary + file, or expanded README where the change warranted a sentence. + + For each finding, state the concrete recommendation: **delete**, **split + into a separate PR**, or **keep** with the justification that makes it + required. Prefer "delete" — the default answer for something the task did + not ask for is that it should not be in this PR. + + Then give a single overall verdict: + + - **PROPORTIONATE** — the diff matches the task. + - **BLOATED** — name the two or three changes that would shrink it most, + and estimate the lines they would remove. + + Two things to get right, because getting them wrong makes this rule worse + than useless: + + - **Do not flag work the task actually required**, even where it is large. + A big diff is not itself a finding; a big diff for a small task is. + - **Do not flag missing work.** Other rules and human review cover + correctness, coverage, and completeness. This rule only ever argues for + removing things, never for adding them. diff --git a/deploy/unsupervised-main-prod-testing/README.md b/deploy/unsupervised-main-prod-testing/README.md new file mode 100644 index 00000000..14eb2333 --- /dev/null +++ b/deploy/unsupervised-main-prod-testing/README.md @@ -0,0 +1,63 @@ +# Enable panopticon agents to do finder prod-testing + +Give unsupervised-main task agents the ability to **provision test pods, generate the finder +executable, and profile results** against production data — the workflow currently done by hand. + +Two phases: +- **Interim (now, broad role):** test pods run in **`default`** as `unsupervised-unsupervised` (the + existing `prod-unsupervised-main` IRSA role) to read prod exports. **No cluster/credential/IAM + changes** — everything needed already exists in `default`. Trade-off: agents get the broad + (read+write) role and there is **no ResourceQuota** — the operator turn-handoff gate is the only + guardrail. Files marked "phase 2" below are NOT used here. +- **Phase 2 (later, needs IAM-admin):** scoped `finder-repro` namespace + read-only IRSA SA + + ResourceQuota. Blocked until an IAM-admin creates the scoped role (the `Unsupervised-Engineer` SSO + role can't `iam:CreateRole`). `finder-repro-rbac.yaml` + `iam-finder-repro-readonly.json` + `apply.sh + scope-sa` are the phase-2 path, kept ready. + +Common to both: **in-pod finder build (no Docker-in-Docker)** from a pre-baked Harbor builder image, +and every prod run gated by an **operator turn-handoff**. + +## Two identities (keep them straight) +- **Control** — `panopticon-repro` SA: creates/execs pods (RBAC). No data access. +- **Run** — `finder-test` SA: the SA the test *pods* run as; IRSA -> read-only prod S3. Data + access rides here, not on kubectl. + +## Files + +| File | Goes to | Purpose | +|---|---|---| +| `finder-repro-rbac.yaml` | prod cluster (`kubectl apply`) | `finder-repro` ns + `panopticon-repro` (control) + `finder-test` (run) SAs + Role + ResourceQuota/LimitRange | +| `iam-finder-repro-readonly.json` | **AWS (operator creates)** | IAM role templates: trust (EKS OIDC + `finder-repro:finder-test`) + read-only s3 on `unsupervised-prod-internal/internal/*` | +| `unsupervised-main/Dockerfile.finder-builder` | unsupervised-main (PR) | Pre-baked builder image (deps installed, source overlaid per build) | +| `unsupervised-main/publish.finder-builder.yml` | unsupervised-main (PR) | CI publish job (reuses existing `HARBOR_USERNAME`/`HARBOR_PASSWORD`) | +| `image-layer.head.dockerfile` | `$CONFIG/layers/` (via apply.sh) | kubectl + auto-wiring wrapper (materializes kubeconfig from the injected env var) | +| `build-finder-in-pod.sh` | image layer `/usr/local/bin` | Agent-run in-pod build | +| `repro-pod.template.yaml` | reference | A finder test pod running as `finder-test` (IRSA S3) | +| `prod-testing-gate.md` | unsupervised-main AGENTS.md | The turn-handoff approval rule agents must follow | +| `apply.sh` | operator runs | `config` (wire layer+repo) / `scope-sa` (prod RBAC + rewrite kubeconfig secret) | + +## Guardrail = policy + backstop +- **Policy (turn-handoff, `prod-testing-gate.md`):** before any pod in `finder-repro`, the agent + ends its turn with a proposal (change, pod size, **which exports it reads**, what it measures); + you approve in the dashboard. Trust-based, per-run, full context. +- **Backstop (enforced by the API server):** `ResourceQuota` caps the namespace (8 pods / 128 CPU + / 600Gi / 1000Gi scratch), `LimitRange` caps any one pod (64 CPU / 300Gi / 500Gi). IRSA is + read-only, one bucket. So worst case, even if the policy is ignored, the blast radius is bounded. + +## Apply sequence +1. **PR** `Dockerfile.finder-builder` + `publish.finder-builder.yml` into unsupervised-main; run the + CI job once -> `harbor…/images/finder-builder:latest` exists. +2. **AWS (you):** create IAM role `prod-finder-repro-readonly` from `iam-finder-repro-readonly.json`; + uncomment the `finder-test` SA's `role-arn` annotation in `finder-repro-rbac.yaml`. +3. `apply.sh config` — assemble+install the image layer, PATCH repo config, force image rebuild. +4. `kubectl --context apply -f finder-repro-rbac.yaml`; copy the pull secret into the ns: + `kubectl -n default get secret unsupervised-regcred -o yaml | sed 's/namespace: default/namespace: finder-repro/' | kubectl -n finder-repro apply -f -` + (or just run `apply.sh scope-sa`, which does the apply + pull-secret copy + kubeconfig rewrite). +5. `apply.sh scope-sa` — mint a `finder-repro`-scoped kubeconfig, **test it (probe pod) before + overwriting**, back up the old value, rewrite `PROD_REPRO_KUBECONFIG_B64`. +6. Add `prod-testing-gate.md` to unsupervised-main's AGENTS.md. +7. Smoke test: `build-finder-in-pod.sh` -> binary; provision a `repro-pod.template.yaml` pod; profile. + +## Rebuild cadence +The pre-baked image only rebuilds when the **heavy deps** change (requirements.txt / Rust ext / +python-utils). Ordinary `fc.py` edits are overlaid at build-in-pod time — day-to-day this is free. diff --git a/deploy/unsupervised-main-prod-testing/apply.sh b/deploy/unsupervised-main-prod-testing/apply.sh new file mode 100644 index 00000000..857ecc96 --- /dev/null +++ b/deploy/unsupervised-main-prod-testing/apply.sh @@ -0,0 +1,133 @@ +#!/usr/bin/env bash +# One-shot operator setup to enable unsupervised-main agents' finder prod-testing. +# +# apply.sh config # INTERIM (default): wire the image layer + repo config only. +# # No cluster/credential changes — agents run test pods in `default` +# # as unsupervised-unsupervised (broad prod-unsupervised-main IRSA), +# # and panopticon-repro's existing kubeconfig already grants pod-create +# # in `default`. This is all that's needed for the broad-role interim. +# apply.sh scope-sa # PHASE 2 (needs an IAM-admin first): create the finder-repro RBAC, +# # mint a finder-repro-scoped kubeconfig, and REWRITE +# # PROD_REPRO_KUBECONFIG_B64 (backed up first). Requires the scoped IAM +# # role + finder-test SA annotation to exist — do NOT run until then. +set -euo pipefail + +HERE="$(cd "$(dirname "$0")" && pwd)" +PANOPTICON_CONFIG="${PANOPTICON_CONFIG:-$HOME/.config/panopticon}" +LAYERS_DIR="$PANOPTICON_CONFIG/layers" +SECRETS_DIR="$PANOPTICON_CONFIG/secrets" +ENV_FILE="$SECRETS_DIR/unsupervised_main.env" +SVC="${PANOPTICON_SERVICE_URL:-http://localhost:8000}" +PROD_CTX="${PROD_CTX:-arn:aws:eks:us-east-1:037004398141:cluster/prod}" +NS="finder-repro" +SA="panopticon-repro" +LAYER_NAME="unsupervised-main.dockerfile" + +log() { printf '\n== %s ==\n' "$*"; } + +config() { + log "assembling image layer -> $LAYERS_DIR/$LAYER_NAME" + mkdir -p "$LAYERS_DIR" + { + cat "$HERE/image-layer.head.dockerfile" + printf '\n# --- inlined build-finder-in-pod.sh (build context is Dockerfile-only, so no COPY) ---\n' + printf "RUN cat > /usr/local/bin/build-finder-in-pod.sh <<'FINDERBUILD' && chmod 0755 /usr/local/bin/build-finder-in-pod.sh\n" + cat "$HERE/build-finder-in-pod.sh" + printf '\nFINDERBUILD\n' + } > "$LAYERS_DIR/$LAYER_NAME" + + log "PATCH repo config: image_layer_file=$LAYER_NAME" + curl -fsS -X PATCH "$SVC/repos/unsupervised-main" \ + -H 'Content-Type: application/json' \ + -d "{\"image_layer_file\": \"$LAYER_NAME\"}" >/dev/null + echo " ok" + + log "forcing image rebuild (drop cached composed tags; runner rebuilds on next spawn)" + for wf in github-peer-reviewed github-self-reviewed; do + docker rmi "panopticon-$wf-unsupervised-main" 2>/dev/null || true + done + echo " dropped; a fresh task spawn will compose base -> workflow -> repo layer." +} + +scope_sa() { + log "applying scoped RBAC to prod ($NS)" + kubectl --context "$PROD_CTX" apply -f "$HERE/finder-repro-rbac.yaml" + + log "copying pull secret unsupervised-regcred into $NS" + kubectl --context "$PROD_CTX" -n default get secret unsupervised-regcred -o yaml \ + | sed -e 's/^ namespace: default/ namespace: '"$NS"'/' \ + -e '/resourceVersion:/d' -e '/uid:/d' -e '/creationTimestamp:/d' \ + | kubectl --context "$PROD_CTX" -n "$NS" apply -f - + + log "minting a long-lived token secret for $NS/$SA" + kubectl --context "$PROD_CTX" -n "$NS" apply -f - </dev/null || true)" + [ -n "$tok" ] && break; sleep 1 + done + [ -n "$tok" ] || { echo "token secret never populated" >&2; exit 3; } + + log "building the scoped kubeconfig" + server="$(kubectl --context "$PROD_CTX" config view --minify -o jsonpath='{.clusters[0].cluster.server}')" + ca="$(kubectl --context "$PROD_CTX" -n "$NS" get secret "${SA}-token" -o jsonpath='{.data.ca\.crt}')" + token="$(printf '%s' "$tok" | base64 -d)" + newkc="$(mktemp)"; trap 'rm -f "$newkc"' RETURN + cat > "$newkc" </dev/null + kubectl --kubeconfig "$newkc" -n "$NS" run rbac-probe --image=public.ecr.aws/docker/library/busybox:latest \ + --restart=Never --command -- sh -c 'exit 0' >/dev/null + kubectl --kubeconfig "$newkc" -n "$NS" delete pod rbac-probe --wait=false >/dev/null 2>&1 || true + echo " new kubeconfig works in $NS." + + log "backing up + rewriting PROD_REPRO_KUBECONFIG_B64 in $ENV_FILE" + cp -p "$ENV_FILE" "$ENV_FILE.bak-$(date -u +%Y%m%dT%H%M%SZ)" + newval="$(base64 < "$newkc" | tr -d '\n')" + # replace just that one line (values may contain '/', so use a python rewrite, not sed) + ENVFILE="$ENV_FILE" NEWVAL="$newval" python3 - <<'PY' +import os +p=os.environ["ENVFILE"]; nv=os.environ["NEWVAL"] +lines=open(p).read().splitlines(); out=[]; done=False +for ln in lines: + if ln.startswith("PROD_REPRO_KUBECONFIG_B64="): + out.append(f"PROD_REPRO_KUBECONFIG_B64={nv}"); done=True + else: out.append(ln) +assert done, "PROD_REPRO_KUBECONFIG_B64 line not found" +open(p,"w").write("\n".join(out)+"\n") +print(" rewrote PROD_REPRO_KUBECONFIG_B64 (backup kept)") +PY + echo " done. New task spawns will use the finder-repro-scoped credential." +} + +case "${1:-config}" in + config) config ;; + scope-sa) + echo "PHASE 2: only run scope-sa after an IAM-admin has created prod-finder-repro-readonly" + echo "and you've uncommented the finder-test SA role-arn. Ctrl-C now if that's not done." >&2 + sleep 5; scope_sa ;; + *) echo "usage: apply.sh [config|scope-sa] (config = broad-role interim; scope-sa = phase 2)" >&2; exit 2 ;; +esac diff --git a/deploy/unsupervised-main-prod-testing/build-finder-in-pod.sh b/deploy/unsupervised-main-prod-testing/build-finder-in-pod.sh new file mode 100644 index 00000000..62041935 --- /dev/null +++ b/deploy/unsupervised-main-prod-testing/build-finder-in-pod.sh @@ -0,0 +1,98 @@ +#!/usr/bin/env bash +# Build the finder PyInstaller binary in a k8s pod (NO Docker-in-Docker). +# +# Baked into the panopticon repo image layer at /usr/local/bin. Uses the pre-baked Harbor builder +# image, which carries ONLY toolchain + deps + the compiled bfinder wheel (no finder/Rust source). +# This script overlays the agent's CURRENT finder source, recompiles the Rust wheel *iff* the Rust +# source changed (auto-detected via RUST_SRC_HASH — a Python-only change never recompiles; a Rust +# change is never tested stale), installs finder editable, and runs pyinstaller. kubectl is +# auto-wired to `default` (interim) by the image-layer wrapper; the builder pod pulls via +# unsupervised-regcred. +# +# Usage: build-finder-in-pod.sh [--src DIR] [--out FILE] [--ns NS] [--image IMG] +# [--name POD] [--force-rust] [--keep] +set -euo pipefail + +SRC="subrepos/finder" +OUT="./finder" +# Interim: default ns (broad prod-unsupervised-main IRSA). Phase 2: NS=finder-repro. +NS="default" +IMAGE="${FINDER_BUILDER_IMAGE:-harbor.unsupervised.com/images/finder-builder:latest}" +POD="" +FORCE_RUST=0 +KEEP=0 + +while [ $# -gt 0 ]; do + case "$1" in + --src) SRC="$2"; shift 2;; + --out) OUT="$2"; shift 2;; + --ns) NS="$2"; shift 2;; + --image) IMAGE="$2"; shift 2;; + --name) POD="$2"; shift 2;; + --force-rust) FORCE_RUST=1; shift;; # recompile the wheel even if the hash matches + --keep) KEEP=1; shift;; + *) echo "unknown arg: $1" >&2; exit 2;; + esac +done + +[ -d "$SRC/src" ] || { echo "no finder src at $SRC/src (run from the repo root, or pass --src)" >&2; exit 2; } +if [ -z "$POD" ]; then + h="$(find "$SRC/src" -type f -printf '%P %s\n' 2>/dev/null | cksum | cut -d' ' -f1)" + POD="finder-build-${h}" +fi + +cleanup() { [ "$KEEP" = 1 ] || kubectl -n "$NS" delete pod "$POD" --ignore-not-found --wait=false >/dev/null 2>&1 || true; } +trap cleanup EXIT + +echo "== launching builder pod $POD (image=$IMAGE, ns=$NS) ==" +kubectl -n "$NS" delete pod "$POD" --ignore-not-found --wait=true >/dev/null 2>&1 || true +kubectl -n "$NS" apply -f - <&2; exit 3; } + +echo "== overlaying current finder source into the pod (transient — deleted with the pod) ==" +tar -C "$SRC" --exclude=venv --exclude=.venv --exclude=target --exclude=dist \ + --exclude=__pycache__ --exclude=.git -czf - . \ + | kubectl -n "$NS" exec -i "$POD" -- bash -c 'mkdir -p /build/subrepos/finder && tar -C /build/subrepos/finder -xzf -' + +echo "== recompiling the bfinder wheel only if the Rust source changed ==" +kubectl -n "$NS" exec "$POD" -- bash -lc " + set -euo pipefail + cd /build/subrepos/finder + rust_hash() { find Cargo.toml Cargo.lock bfinder pybfinder -type f | sort | xargs sha256sum | sha256sum | cut -d' ' -f1; } + baked=\$(cat /build/RUST_SRC_HASH) + now=\$(rust_hash) + if [ '${FORCE_RUST}' = '1' ] || [ \"\$now\" != \"\$baked\" ]; then + echo ' Rust source changed (or --force-rust) -> recompiling wheel' + (cd pybfinder && maturin build -r -o /tmp/wheels && pip install --force-reinstall --no-deps /tmp/wheels/*.whl) + else + echo ' Rust unchanged -> using baked wheel (fast path)' + fi + echo '== installing finder (editable) from overlaid src + pyinstaller ==' + pip install -e . --no-deps + sh pyinstaller.sh +" + +echo "== copying binary out -> $OUT ==" +kubectl -n "$NS" cp "$POD:/build/subrepos/finder/dist/finder" "$OUT" +chmod +x "$OUT" +echo "== done: $OUT ($(wc -c < "$OUT") bytes). linux/amd64 binary — run it in a pod, not on the agent host. ==" diff --git a/deploy/unsupervised-main-prod-testing/finder-repro-rbac.yaml b/deploy/unsupervised-main-prod-testing/finder-repro-rbac.yaml new file mode 100644 index 00000000..e2800be3 --- /dev/null +++ b/deploy/unsupervised-main-prod-testing/finder-repro-rbac.yaml @@ -0,0 +1,122 @@ +# ============================================================================ +# PHASE 2 — NOT USED IN THE BROAD-ROLE INTERIM. Do NOT apply this until an +# IAM-admin has created the prod-finder-repro-readonly role (iam-finder-repro- +# readonly.json) and you've uncommented the finder-test SA's role-arn. Until +# then, agents run test pods in `default` as unsupervised-unsupervised. +# ============================================================================ +# Scoped RBAC for panopticon agents doing finder prod-testing. +# Applies to the prod EKS cluster. Replaces the current arrangement where the +# `panopticon-repro` ServiceAccount lives in `default` with pod create/delete/exec/log. +# +# Decision (operator-approved): give agents a DEDICATED namespace with a resource +# ceiling, not `default`, so an agent can't spawn an unbounded 256Gi run or interfere +# with real workloads. The verbs match exactly what the finder-pod-testing workflow +# needs and nothing more. +# +# Apply: kubectl --context apply -f finder-repro-rbac.yaml +# The ResourceQuota / LimitRange numbers are ceilings — tune to the real test budget. +apiVersion: v1 +kind: Namespace +metadata: + name: finder-repro + labels: + app.kubernetes.io/managed-by: panopticon + panopticon.unsupervised.com/purpose: finder-prod-testing +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: panopticon-repro + namespace: finder-repro +--- +# Run identity for finder TEST PODS (separate from the control SA above). The pods an +# agent provisions set `serviceAccountName: finder-test` so they can READ prod S3 exports +# via IRSA. Data access rides on this SA, NOT on kubectl / the control credential. +# +# ACTIVATION (operator, AWS-side — see iam-finder-repro-readonly.json): +# 1. Create IAM role `prod-finder-repro-readonly` (trust = EKS OIDC + sub +# system:serviceaccount:finder-repro:finder-test; permissions = read-only s3 on +# unsupervised-prod-internal/internal/*). +# 2. Uncomment the annotation below with that role's ARN and re-apply. +# Until then this SA exists but has no data access (small/synthetic tests still work). +apiVersion: v1 +kind: ServiceAccount +metadata: + name: finder-test + namespace: finder-repro + # annotations: + # eks.amazonaws.com/role-arn: arn:aws:iam::037004398141:role/prod-finder-repro-readonly +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: panopticon-repro + namespace: finder-repro +rules: + - apiGroups: [""] + resources: ["pods", "pods/log"] + verbs: ["get", "list", "watch"] + - apiGroups: [""] + resources: ["pods"] + verbs: ["create", "delete"] + - apiGroups: [""] + resources: ["pods/exec"] + verbs: ["create"] + # cp needs to read pod status/attach; keep it minimal. + - apiGroups: [""] + resources: ["pods/status"] + verbs: ["get"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: panopticon-repro + namespace: finder-repro +subjects: + - kind: ServiceAccount + name: panopticon-repro + namespace: finder-repro +roleRef: + kind: Role + name: panopticon-repro + apiGroup: rbac.authorization.k8s.io +--- +# Ceiling on the whole namespace so agents can't run away. Finder repro pods are +# legitimately large (200Gi+), so these are generous but bounded. +apiVersion: v1 +kind: ResourceQuota +metadata: + name: finder-repro-quota + namespace: finder-repro +spec: + hard: + pods: "8" + requests.cpu: "128" + limits.cpu: "128" + requests.memory: 600Gi + limits.memory: 600Gi + requests.ephemeral-storage: 1000Gi + limits.ephemeral-storage: 1000Gi +--- +# Cap any single pod so one pod can't consume the entire quota, and give pods that +# omit limits a small safe default rather than unbounded. +apiVersion: v1 +kind: LimitRange +metadata: + name: finder-repro-limits + namespace: finder-repro +spec: + limits: + - type: Container + max: + cpu: "64" + memory: 300Gi + ephemeral-storage: 500Gi + default: + cpu: "2" + memory: 8Gi + ephemeral-storage: 20Gi + defaultRequest: + cpu: "1" + memory: 4Gi + ephemeral-storage: 10Gi diff --git a/deploy/unsupervised-main-prod-testing/iam-finder-repro-readonly.json b/deploy/unsupervised-main-prod-testing/iam-finder-repro-readonly.json new file mode 100644 index 00000000..12b198dc --- /dev/null +++ b/deploy/unsupervised-main-prod-testing/iam-finder-repro-readonly.json @@ -0,0 +1,51 @@ +{ + "_README": [ + "Templates for the IAM role that lets finder TEST PODS read prod S3 exports via IRSA.", + "Operator creates this AWS-side (agents cannot). Read-only, one bucket, scoped to the", + "finder-repro:finder-test ServiceAccount. Then uncomment the role-arn annotation on the", + "finder-test SA in finder-repro-rbac.yaml and re-apply.", + "", + " ROLE=prod-finder-repro-readonly", + " aws iam create-role --role-name $ROLE --assume-role-policy-document file://", + " aws iam put-role-policy --role-name $ROLE --policy-name s3-read --policy-document file://", + "", + "Scope the permission_policy Resource to specific export prefixes if you want to expose", + "only curated staged exports rather than the whole internal/ prefix." + ], + "trust_policy": { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "Federated": "arn:aws:iam::037004398141:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/5182D36C43F8988418E19551325D9F69" + }, + "Action": "sts:AssumeRoleWithWebIdentity", + "Condition": { + "StringEquals": { + "oidc.eks.us-east-1.amazonaws.com/id/5182D36C43F8988418E19551325D9F69:sub": "system:serviceaccount:finder-repro:finder-test", + "oidc.eks.us-east-1.amazonaws.com/id/5182D36C43F8988418E19551325D9F69:aud": "sts.amazonaws.com" + } + } + } + ] + }, + "permission_policy": { + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "ListExports", + "Effect": "Allow", + "Action": ["s3:ListBucket"], + "Resource": "arn:aws:s3:::unsupervised-prod-internal", + "Condition": { "StringLike": { "s3:prefix": ["internal/*"] } } + }, + { + "Sid": "ReadExports", + "Effect": "Allow", + "Action": ["s3:GetObject"], + "Resource": "arn:aws:s3:::unsupervised-prod-internal/internal/*" + } + ] + } +} diff --git a/deploy/unsupervised-main-prod-testing/image-layer.head.dockerfile b/deploy/unsupervised-main-prod-testing/image-layer.head.dockerfile new file mode 100644 index 00000000..11ca94d6 --- /dev/null +++ b/deploy/unsupervised-main-prod-testing/image-layer.head.dockerfile @@ -0,0 +1,35 @@ +# Repo image layer (ADR 0005) for unsupervised-main — finder prod-testing tooling. +# +# This is a Dockerfile *fragment*, not a full Dockerfile: sessionservice/images.py composes +# `FROM \n\n` into one image, and the build context is a temp dir +# holding ONLY the generated Dockerfile — so there is NO COPY. Extra files are inlined via RUN. +# apply.sh appends build-finder-in-pod.sh (as a heredoc) after this head and installs the result +# to $PANOPTICON_CONFIG/layers/unsupervised-main.dockerfile. +# +# The base ends on `USER root` and ships curl + ca-certificates, so kubectl installs cleanly; +# the base entrypoint drops back to the `panopticon` user at runtime. + +USER root + +# --- kubectl (real binary), arch-matched to the container --- +ARG KUBECTL_VERSION=v1.30.5 +RUN arch="$(dpkg --print-architecture)" \ + && curl -fsSLo /usr/local/bin/kubectl.real \ + "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/${arch}/kubectl" \ + && chmod 0755 /usr/local/bin/kubectl.real \ + && /usr/local/bin/kubectl.real version --client=true --output=yaml >/dev/null + +# --- kubectl wrapper: lazily materialize ~/.kube/config from the injected +# PROD_REPRO_KUBECONFIG_B64 env var on first use. Works under `bash -c` +# (Claude's Bash tool) — no login-shell / profile.d dependency. The env var is +# injected by the runner from the repo env-file; it is never baked into the image. --- +RUN cat > /usr/local/bin/kubectl <<'WRAP' && chmod 0755 /usr/local/bin/kubectl +#!/usr/bin/env bash +set -euo pipefail +cfg="${KUBECONFIG:-$HOME/.kube/config}" +if [ ! -s "$cfg" ] && [ -n "${PROD_REPRO_KUBECONFIG_B64:-}" ]; then + mkdir -p "$(dirname "$cfg")" + ( umask 077; printf '%s' "$PROD_REPRO_KUBECONFIG_B64" | base64 -d > "$cfg" ) +fi +exec /usr/local/bin/kubectl.real "$@" +WRAP diff --git a/deploy/unsupervised-main-prod-testing/prod-testing-gate.md b/deploy/unsupervised-main-prod-testing/prod-testing-gate.md new file mode 100644 index 00000000..db48a5a7 --- /dev/null +++ b/deploy/unsupervised-main-prod-testing/prod-testing-gate.md @@ -0,0 +1,34 @@ +# Prod-testing approval gate (agent instruction) + +> Add to unsupervised-main's agent instructions (AGENTS.md / CLAUDE.md). In the interim +> (broad role, pods in `default`) this turn-handoff is the **only** guardrail — there is no +> namespace ResourceQuota to fall back on — so it is not optional. + +You can build a modified finder binary and run it in a prod test pod (`build-finder-in-pod.sh`, +`kubectl`). Interim: these pods run in the **`default` namespace** as `unsupervised-unsupervised`, +which has **broad production access** (read+write via the main-app IRSA role) and runs **alongside +real prod workloads with no resource quota**. Treat this as a high-privilege, high-blast-radius +action. + +## When to use it +Only when a change's correctness or performance **can only be shown empirically** and a unit test +can't — a data-scale bug repro or a load/finding perf number. Strongly prefer a cheap **synthetic +input** that isolates the mechanism (no prod data, no big pod) whenever it suffices. + +## MUST: pause for operator approval before ANY prod pod +Before you `kubectl run`/`apply` any pod, **end your turn and hand it to the operator** with: +- what change / which binary, +- **pod size (cpu + memory) and expected wall-clock** — there is no quota clamp, so this is your + only bound; keep it as small as the test allows, +- **which prod exports it will read** (S3 URIs / run ids), +- what it measures and the pass/fail criterion. + +Do not create the pod until the operator advances the turn back approving it. A denial means don't +run it. + +## Rules while running +- Namespace `default` only; do not touch, delete, or exec into pods you did not create. +- Label every pod you create `app.kubernetes.io/managed-by=panopticon` so it's identifiable. +- **Always delete your test pods the moment you're done** — nothing else will reclaim them. +- Read-only intent: you are validating a change, not mutating prod. Do not write to prod buckets + or trigger re-exports; if a test needs fresh/other data, ask the operator to stage the export. diff --git a/deploy/unsupervised-main-prod-testing/repro-pod.template.yaml b/deploy/unsupervised-main-prod-testing/repro-pod.template.yaml new file mode 100644 index 00000000..4f24bfc9 --- /dev/null +++ b/deploy/unsupervised-main-prod-testing/repro-pod.template.yaml @@ -0,0 +1,49 @@ +# Template: a finder repro/test pod an agent provisions. +# +# INTERIM (broad role): runs in `default` as `unsupervised-unsupervised` — the existing +# prod-unsupervised-main IRSA role, so it can read prod S3 exports. No new cluster objects. +# PHASE 2 (scoped, needs IAM-admin): namespace: finder-repro, serviceAccountName: finder-test. +# +# NOTE: `default` has no ResourceQuota, so nothing but the operator turn-handoff gate bounds +# this in the interim. Keep pod sizes explicit (below) and ALWAYS delete pods when done. +# +# The agent fills: , memory/cpu, and the finder config (feature_ids/file_locations +# pointing at s3://unsupervised-prod-internal/internal/... exports the operator staged). +apiVersion: v1 +kind: Pod +metadata: + name: # e.g. finder-repro- + namespace: default + labels: + app.kubernetes.io/managed-by: panopticon + purpose: finder-repro +spec: + restartPolicy: Never + serviceAccountName: unsupervised-unsupervised # broad IRSA -> prod S3 exports (interim) + imagePullSecrets: + - name: unsupervised-regcred # Harbor pull (already in default) + containers: + - name: finder + # A base image just to hold/exec the cp'd binary; the binary is self-contained + # (PyInstaller). Reuse the prod image for parity. + image: harbor.unsupervised.com/images/unsupervised-main:latest + command: ["sleep", "36000"] + env: + - { name: FINDER_INPUT_DATA_MEMORY_BUDGET_GB, value: "200" } + - { name: TMPDIR, value: "/scratch" } + - { name: FINDER_WORKERS, value: "52" } + resources: + # Keep these explicit — in `default` there is no quota to clamp a runaway pod. + requests: { cpu: "16", memory: 200Gi } + limits: { cpu: "32", memory: 256Gi } + volumeMounts: + - { name: scratch, mountPath: /scratch } + volumes: + - name: scratch + emptyDir: { sizeLimit: 400Gi } +# Agent workflow, once Ready: +# kubectl -n default cp ./finder :/scratch/finder +# kubectl -n default cp ./cfg.json :/scratch/cfg.json +# kubectl -n default exec -- bash -lc 'chmod +x /scratch/finder && /scratch/finder -i /scratch/cfg.json' +# # profile passively: kubectl -n default exec -- cat /sys/fs/cgroup/cpu.stat +# kubectl -n default delete pod # ALWAYS clean up diff --git a/deploy/unsupervised-main-prod-testing/unsupervised-main/Dockerfile.finder-builder b/deploy/unsupervised-main-prod-testing/unsupervised-main/Dockerfile.finder-builder new file mode 100644 index 00000000..53992014 --- /dev/null +++ b/deploy/unsupervised-main-prod-testing/unsupervised-main/Dockerfile.finder-builder @@ -0,0 +1,50 @@ +# Pre-baked finder BUILDER image — toolchain + deps + compiled bfinder wheel ONLY. +# +# Deliberately carries NO finder Python source and NO Rust source: the agent overlays the +# CURRENT finder source at build-in-pod time and runs pyinstaller. The Rust `bfinder`/`pybfinder` +# source is used here only to compile the wheel, then deleted — so nothing proprietary beyond the +# already-shipped compiled artifacts persists in Harbor. (python-utils remains as an installed +# runtime dep, unavoidable — pyinstaller bundles it.) +# +# We record a hash of the Rust source (RUST_SRC_HASH). build-finder-in-pod.sh compares the +# overlaid source against it and recompiles the wheel iff the change touches the Rust — so a +# Python-only change uses the fast baked wheel, a Rust change is never tested stale. +# +# Destined for unsupervised-main (e.g. .github/docker/finder-builder.Dockerfile), published to +# Harbor by publish.finder-builder.yml. Build context = repo root (needs subrepos/finder + +# subrepos/python-utils), with submodules checked out. +FROM python:3.10-bookworm + +ENV DEBIAN_FRONTEND=noninteractive +RUN apt-get update \ + && apt-get install --yes --no-install-recommends \ + build-essential libpq-dev curl git ca-certificates pkg-config \ + && rm -rf /var/lib/apt/lists/* + +RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable +ENV PATH="/root/.cargo/bin:${PATH}" +RUN pip install --no-cache-dir maturin pyinstaller + +WORKDIR /build +# python-utils (finder's editable dep) — stays as an installed package. +COPY subrepos/python-utils /build/subrepos/python-utils +# Only the Rust source + requirements from finder — NOT the finder Python src. +COPY subrepos/finder/Cargo.toml subrepos/finder/Cargo.lock /build/subrepos/finder/ +COPY subrepos/finder/bfinder /build/subrepos/finder/bfinder +COPY subrepos/finder/pybfinder /build/subrepos/finder/pybfinder +COPY subrepos/finder/requirements.txt /build/subrepos/finder/requirements.txt + +WORKDIR /build/subrepos/finder +# Install third-party deps + editable python-utils, but NOT finder (`-e .`) — no finder src here. +RUN grep -v '^-e \.$' requirements.txt > requirements-nofinder.txt \ + && pip install --no-cache-dir -r requirements-nofinder.txt +# Compile + install the bfinder wheel from the Rust source. +RUN cd pybfinder && maturin build -r -o /tmp/wheels && pip install --no-cache-dir /tmp/wheels/*.whl + +# Record a content+path hash of the Rust source, then DELETE the Rust source (keep only the +# installed compiled wheel). Same hash recipe as build-finder-in-pod.sh's rust_hash(). +RUN find Cargo.toml Cargo.lock bfinder pybfinder -type f | sort | xargs sha256sum | sha256sum \ + | cut -d' ' -f1 > /build/RUST_SRC_HASH \ + && rm -rf bfinder pybfinder target Cargo.toml Cargo.lock requirements.txt requirements-nofinder.txt \ + && echo "finder-builder: toolchain + deps + bfinder wheel baked; overlay finder src at build time." \ + > /build/FINDER_BUILDER_READY diff --git a/deploy/unsupervised-main-prod-testing/unsupervised-main/finder-builder.Dockerfile.dockerignore b/deploy/unsupervised-main-prod-testing/unsupervised-main/finder-builder.Dockerfile.dockerignore new file mode 100644 index 00000000..cc56c437 --- /dev/null +++ b/deploy/unsupervised-main-prod-testing/unsupervised-main/finder-builder.Dockerfile.dockerignore @@ -0,0 +1,13 @@ +# Dockerfile-specific ignore for the finder-builder build. +# BuildKit uses `.dockerignore` (this file) INSTEAD of the repo-root .dockerignore, +# which excludes `subrepos/`. This build needs subrepos/finder + subrepos/python-utils in context, +# so we do NOT exclude them here — just trim heavy/irrelevant junk. +.git +**/.git +**/.venv +**/venv +**/target +**/node_modules +**/__pycache__ +**/*.pyc +**/dist diff --git a/deploy/unsupervised-main-prod-testing/unsupervised-main/publish.finder-builder.yml b/deploy/unsupervised-main-prod-testing/unsupervised-main/publish.finder-builder.yml new file mode 100644 index 00000000..cbfa07a3 --- /dev/null +++ b/deploy/unsupervised-main-prod-testing/unsupervised-main/publish.finder-builder.yml @@ -0,0 +1,70 @@ +# Publish the pre-baked finder BUILDER image to Harbor. +# Destined for unsupervised-main/.github/workflows/publish.finder-builder.yml (copy via PR). +# +# Reuses the SAME Harbor credentials CI already uses for unsupervised-main / unsupervised-base +# (vars.HARBOR_USERNAME + secrets.HARBOR_PASSWORD) — no new account. In-cluster PULL reuses the +# existing `unsupervised-regcred` secret (copied into the finder-repro namespace), so nothing new +# on the pull side either. +# +# Rebuild cadence: manual (workflow_dispatch) + whenever the heavy deps change. Rebuild is only +# needed when requirements.txt / the Rust ext / python-utils change — NOT for ordinary fc.py edits +# (those are overlaid at build-in-pod time), so this stays cheap. +name: Publish Finder Builder Image +on: + workflow_dispatch: + inputs: + tag: + description: "Image tag (default: short SHA)" + required: false + default: "" + push: + branches: [staging] + paths: + - "subrepos/finder/requirements.txt" + - "subrepos/finder/pybfinder/**" + - "subrepos/finder/bfinder/**" + - "subrepos/finder/Cargo.*" + - "subrepos/finder/Dockerfile.finder-builder" + - "subrepos/python-utils/**" + +concurrency: + group: publish-finder-builder-${{ github.ref }} + cancel-in-progress: true + +jobs: + publish: + runs-on: ubuntu-latest + steps: + - name: Checkout (with submodules) + uses: actions/checkout@v7 + with: + submodules: true + + - name: Resolve tag + id: tag + run: | + t="${{ inputs.tag }}" + [ -n "$t" ] || t="$(git rev-parse --short HEAD)" + echo "tag=$t" >> "$GITHUB_OUTPUT" + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Log in to Harbor + uses: docker/login-action@v3 + with: + registry: harbor.unsupervised.com + username: ${{ vars.HARBOR_USERNAME }} + password: ${{ secrets.HARBOR_PASSWORD }} + + - name: Build & push finder-builder + uses: docker/build-push-action@v6 + with: + context: . + file: subrepos/finder/Dockerfile.finder-builder + push: true + tags: | + harbor.unsupervised.com/images/finder-builder:${{ steps.tag.outputs.tag }} + harbor.unsupervised.com/images/finder-builder:latest + cache-from: type=registry,ref=harbor.unsupervised.com/images/finder-builder:buildcache + cache-to: type=registry,ref=harbor.unsupervised.com/images/finder-builder:buildcache,mode=max