diff --git a/README.md b/README.md index aa1d84f..83eec56 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,7 @@ whole repository does. | [`failproofai-policy-publish`](skills/failproofai-policy-publish/) | The publishing companion to policy authoring - take tested policies, build an installable pack, publish its release assets to GitHub with `failproofai publish`, preview it, and verify the consumer path with `failproofai policies add /`. Cloud fleet rollout remains part of `fp-cloud-cli`. | Maintained here. Not synced from anywhere - edit in this repo. | | [`fp-cloud-cli`](skills/fp-cloud-cli/) | Operate FailproofAI Cloud with `fp`: inspect telemetry, evals and usage; triage issues and audits; manage keys, users, orgs, and settings; publish Cloud policy versions; deploy them to fleet machines; observe enforcement; promote or roll back. Global options go **before** the command: `fp --json sessions`, not `fp sessions --json`. | Synced from `FailproofAI/failproofai` → `fp-cloud-cli/skill/`. Do **not** hand-edit here. | | [`failproofai-sdk`](skills/failproofai-sdk/) | Make an AI agent report what it did - plan which points in the agent loop to record, write the instrumentation with the `failproofai_sdk` Python module, thread session/agent identity through it, and verify the events actually land. For an agent loop that is **not** one of the 12 supported CLIs. | Synced from `FailproofAI/failproofai` → `sdk/python/skill/`. Do **not** hand-edit here. | -| [`agenteye-evaluator`](skills/agenteye-evaluator/) | Put automatic quality scores on an agent's production runs - decide which dimensions are worth scoring from real sessions, scaffold the scoring service with the `agenteye-evaluator` Python SDK, score with rules or an LLM judge, test it against a captured session, deploy it and confirm scores land. | Synced from `FailproofAI/agenteye` → `evaluator-sdk/skill/` (private). Do **not** hand-edit here. | +| [`failproofai-eval-brainstorm`](skills/failproofai-eval-brainstorm/) | Work out **what is worth measuring** about an agent's production runs, from the sessions it actually produced - scan the population, confirm the signal is really in the telemetry (a measurement over a payload key nobody emits does not fail, it scores every session identically and looks like it works), check it separates good runs from bad, and converge on two to four proposals. Each one ends in the plain-English prompt that authors it. It stops there: composing, backtesting and deploying the evaluation is the dashboard's eval authoring page. | Synced from `FailproofAI/agenteye` → `agent/skills/failproofai-eval-brainstorm/` (private). Do **not** hand-edit here. | ## Naming and compatibility @@ -42,7 +42,7 @@ preserved. Do not mechanically replace `AgentEye` everywhere. |---|---|---| | `fp-cloud-cli` | `skills/fp-cloud-cli/` | `agenteye-cli` | | `failproofai-sdk` | `skills/failproofai-sdk/` | `agenteye-python-sdk` | -| `agenteye-evaluator` | `skills/agenteye-evaluator/` | **unchanged** because the current distribution and import remain `agenteye-evaluator` and `agenteye_evaluator` | +| `failproofai-eval-brainstorm` | `skills/failproofai-eval-brainstorm/` | `agenteye-evaluator` - **retired, not renamed.** That skill built a v1 "server-push" evaluator service, which no longer exists. Scoring is now Evaluator v2: hosted evaluations authored in the dashboard, or a worker built on `failproofai-sdk`. What survives is the half only a human could do - deciding what to measure - and that is this skill. The `agenteye-evaluator` distribution and `agenteye_evaluator` module names are untouched by this. | The rename does not alter the `fp` or `failproofai` binaries, SDK behavior, Cloud API, saved credentials, or policy format. It only changes which skill identifier new installations use. @@ -61,6 +61,15 @@ Legacy wire and package literals such as `X-AgentEye-*`, `ae_session`, selected `AGENTEYE_*` variables, `agenteye-evaluator`, and `agenteye_evaluator` remain documented where they are still part of the running system. +An installation of the retired `agenteye-evaluator` skill keeps working but is no longer +served from here, and what it teaches - scaffolding a v1 push evaluator - no longer +resolves to anything you can deploy. Migrate it once: + +```bash +npx skills remove agenteye-evaluator +npx skills add FailproofAI/skills --skill failproofai-eval-brainstorm -a claude-code +``` + ## Install Using the [`skills`](https://skills.sh) CLI (`vercel-labs/skills`). It auto-detects @@ -182,10 +191,9 @@ skills/ ← this repo │ ├── SKILL.md │ ├── references/ ← events · install · integration │ └── agents/openai.yaml - └── agenteye-evaluator/ ← mirror · name unchanged upstream + └── failproofai-eval-brainstorm/ ← mirror · what to measure, not how to score it ├── SKILL.md - ├── references/ ← brainstorm · scaffold · sdk-api · session-data - └── agents/openai.yaml + └── references/ ← patterns · writing-the-prompt · cli ``` Each skill folder is **self-contained**: its references/scripts/assets live inside diff --git a/skills/agenteye-evaluator/SKILL.md b/skills/agenteye-evaluator/SKILL.md deleted file mode 100644 index 54b372c..0000000 --- a/skills/agenteye-evaluator/SKILL.md +++ /dev/null @@ -1,332 +0,0 @@ ---- -name: agenteye-evaluator -description: |- - The way to put automatic quality scores on an AI agent's production runs — both deciding what is worth measuring and building the service that measures it. Reach for it even on vague phrasing like "I want evals" or "how do I know if my agent is any good?" - - Trigger when the user wants to: - • decide what to score — plan which dimensions to track, grounded in what real sessions show; may stop at a written plan without building anything; - • build or change an evaluator — scaffold the scoring service, add a dimension, score with rules or an LLM judge, test it against a real captured session, deploy it and confirm scores land. - - Served by the `agenteye-evaluator` Python SDK, with the `fp` CLI supplying real session data to design against. - - NOT for reading eval results that already exist or checking whether quality dropped (that's `fp-cloud-cli` — `fp evals`), instrumenting an agent with the FailproofAI SDK (that's `failproofai-sdk`), or alerting on scores. ---- - -# AgentEye Evaluator - -An evaluator is **a small HTTP service you own**. When an agent session finishes, -the AgentEye server POSTs the whole transcript to it and you return scores. -Nothing is uploaded to AgentEye; there is no registry and no plugin system. You -run the service, AgentEye calls it. - -``` -agent run ends (agent_end event) - → AgentEye server POSTs the full transcript to YOUR service - → you return {"scores": {"helpfulness": 0.9, ...}} - → scores land in the evaluations table → visible in the dashboard and `fp evals` -``` - -The SDK part of this is small — a decorator and two models. **The hard part is -deciding what to score**, and only the user knows that. So most of this skill is -about getting to a good answer with them before any code exists. - -**Two modes.** *Plan mode* decides **what** to evaluate: work through the user's -real logs with them, converge on 2-4 dimensions, and stop at a written **eval -plan** — no code. It's a complete, valid endpoint — see `references/brainstorm.md`. -*Build mode* builds or changes the evaluator that does the scoring. Sections 1-5 are -the design loop both modes share; 6-12 are the build, and can be seeded by a plan -that brainstorm mode produced. - -## 1. Work in the repo that holds the evaluator - -The evaluator gets built into an image and deployed as a long-running service, so -it needs a real home — not a scratch file. Before writing anything, figure out -where it lives: - -1. **Look for one** in or under the working directory: something importing - `agenteye_evaluator`, or a function decorated `@app.evaluator`. - `grep -rl "agenteye_evaluator" .` is usually enough. -2. **If you don't find one, ask** — don't assume it doesn't exist. Evaluators - often live in their own repo, separate from the agent being scored. "Do you - already have an evaluator service somewhere, or should I set one up here?" -3. **Only then scaffold.** `references/scaffold.md` has the project template and, - importantly, the install ladder — the SDK is **not** on public PyPI, so - `pip install agenteye-evaluator` is the one thing you must not blindly run. - -If they already have an evaluator, you're adding a dimension to working code: -read it first and match what's there rather than rewriting it. - -## 2. Interview before you write - -*This is the compressed design loop the build path uses. When the goal is only to -decide what to measure — no code yet — use the fuller, collaborative procedure in -`references/brainstorm.md` and stop at its plan.* - -Resist jumping to code. A skilled guess at what to measure is still a guess, and -an evaluator that scores the wrong thing is worse than none — it produces a -dashboard people learn to ignore. - -Ask a few questions that actually change the answer (pick what fits; this isn't a -form to march through): - -- **What does this agent do, and for whom?** A support bot, a coding agent, and a - research agent fail in completely different ways. -- **Describe a run that went well. Now one that went badly.** This is the - highest-yield question by far — the gap between those two stories *is* the eval. -- **If you could see one number per run, what would it be?** -- **What would make you roll back a release?** Surfaces the thing they actually - care about, which is often not the thing they named first. - -**Listen for the failure in their words, not for metric names.** "It made stuff -up" → factuality. "It kept calling the same tool over and over" → tool -efficiency. "It gave up and told the user to contact support" → deflection. If -they open with a metric name — "we want accuracy" — ask what an inaccurate run -looks like, because accuracy means six different things and you can't compute a -word. - -## 3. Ground the interview in real sessions - -The interview tells you what they *intend*; the transcripts tell you what -*happens*. You need both, and they usually disagree. - -Pull real sessions — `references/session-data.md` covers how, and it matters more -than it looks: the obvious way to build a session fixture is subtly wrong, and -there's a purpose-built endpoint that gives you byte-identical bytes to what your -evaluator will actually receive. - -Read 3-5 sessions end to end, including at least one the user calls bad. You're -answering one question per candidate dimension: - -> **Is the signal actually in the transcript?** - -This kills more dimensions than anything else. "Did the customer come back next -week" is a great metric and is not in the events. "Was the answer correct" may -need a ground truth that doesn't exist. What *is* there: what tools ran and in -what order, what the model said, what errored, how long it took, how it ended. - -## 4. Reconcile, then propose 2-4 dimensions - -Come back with the two halves joined: *"You said you care about X. Your sessions -show Y. So I'd score these three things."* - -Aim for **2-4 dimensions**. Each one is a number a human reads on a dashboard; -past four they stop reading and the eval stops changing decisions. Every proposal -must pass two tests: - -- **Computable** from `req.events` alone. -- **Discriminating** — it separates the good session from the bad one the user - showed you. A dimension that scores 0.9 on both teaches nothing. Check this - against the real sessions *before* proposing, not after. - -Score keys are arbitrary strings; the platform stores and trends whatever you -send. That's freedom, but it means nothing downstream will correct a bad choice — -so **get explicit sign-off on the dimension names before writing code**. Renaming -later splits the history: old sessions keep the old key and the trend breaks. - -## 5. The `EvalRequest` contract - -What lands in your function (full detail in `references/sdk-api.md`): - -| field | notes | -|---|---| -| `events` | `list[AgentEvent]`, chronological. The transcript. This is the one you score from. | -| `session_id` / `agent_id` / `environment` | Identity. Useful for routing logic or per-env thresholds. | -| `started_at` | First event's timestamp. | -| `ended_at` | **`None` is normal — don't rely on it.** | - -Two things that surprise people: - -- **`ended_at` is the `agent_end` timestamp, or `None`.** It is *not* "when the - session stopped". Sessions that never emit `agent_end` get evaluated anyway, by - an inactivity scanner, and arrive with `ended_at: None`. If you compute - duration as `ended_at - started_at`, you crash on a real and common case — use - the last event's `ts` instead. -- **`payload` is the entire raw event JSON, flattened** — not a nested sub-object. - So it's `e.payload.get("tool_name")`, not `e.payload["input"]["tool_name"]`, and - `payload["type"]` duplicates `event_type`. The per-event-type keys are tabulated - in `references/session-data.md`. - -## 6. Write the scorer — rules before judges - -Start with the cheapest thing that separates the good session from the bad one. -Counting and ratios over `req.events` are free, instant, deterministic, and -testable — and for tool-loops, error rates, deflection, and "did it ever finish", -they're genuinely as good as anything an LLM will tell you. - -`deploy/examples/evaluator/evaluator.py` in the AgentEye repo is the reference -implementation; mirror its shape: - -```python -import os -from agenteye_evaluator import EvalRequest, EvalResponse, Evaluator - -app = Evaluator(token=os.environ.get("EVALUATOR_TOKEN")) - -@app.config -def config(): - # Backstop for sessions that never emit agent_end. Without this, they're - # never scored. A plain dict is fine — EvaluatorConfig isn't required. - return {"inactivity_timeout_secs": 1800} - -@app.evaluator -def evaluate(req: EvalRequest) -> EvalResponse: - tool_uses = [e for e in req.events if e.event_type == "tool_use"] - distinct = {e.payload.get("tool_name") for e in tool_uses} - score = len(distinct) / len(tool_uses) if tool_uses else 1.0 - return EvalResponse( - scores={"tool_efficiency": round(score, 2)}, - reasoning={"tool_efficiency": f"{len(distinct)} distinct of {len(tool_uses)} calls."}, - summary=f"{len(req.events)} events, {len(tool_uses)} tool calls.", - ) -``` - -**Always return `reasoning` alongside `scores`.** A bare number sends whoever -reads the dashboard back to the raw transcript to find out why; one sentence per -dimension is what makes the score actionable. It's optional in the API and -mandatory in practice. - -## 7. Test against a real session - -The decorators return the function unchanged, so it stays directly callable — no -HTTP needed for the interesting tests: - -```python -req = EvalRequest.model_validate_json(open("fixtures/session-abc.json").read()) -assert evaluate(req).scores["tool_efficiency"] == 1.0 -``` - -Use `TestClient(app.app)` only when you're testing the wire layer (auth, status -codes). For scoring logic, call the function. - -**Know how your fixture lies.** A captured session is real data, which makes it -tempting to trust completely. Two gaps to keep in mind: a session whose payload -failed to parse arrives as `payload: null` and will 422 your evaluator, but the -CLI coerces that to `{}` so it can never show up in a CLI-built fixture. And a -CLI-reconstructed fixture is an approximation of the real body — the export -endpoint isn't (`references/session-data.md`). - -Test the empty session (`events: []`) and the `ended_at: None` session. Both are -real, both are common, and both are where evaluators crash. - -## 8. LLM-as-judge: the 30-second wall - -For anything subjective — was the answer correct, was the tone right, did it -follow policy — rules run out and you want a model to read the transcript. That's -a good instinct, but a synchronous judge collides with the dispatcher's limits: - -| limit | value | why it bites | -|---|---|---| -| request timeout | **30s** | The server gives up on your POST. | -| concurrency | **8** (2 workers × 4 claim batch) | The whole deployment's budget, not per-agent. | -| retries | **5×**, exponential backoff | A 5xx or timeout is retried — including yours. | - -**Return `JobPending` once the judge isn't reliably under 30s — switch at p99 -above ~15-18s**, not at 29s. The failure when you don't is worse than a slow -score: - -1. At 30s the server cancels its side. **Your judge keeps running and you still - pay for it.** Nobody reads the result. -2. The timeout is transient, so it's **retried 5×** — the same judge runs five - times, five times the cost, and the session still ends with zero scores. -3. That call held 1 of only 8 slots for 30s. Do this at volume and you - head-of-line-block scoring for every agent in the deployment. - -Throughput ceiling is roughly 8 ÷ your latency — about 1150 sessions/hour at 25s -each. If they're scoring more than that, sync is off the table regardless. - -## 9. Going async: `JobPending` + `@app.job_lookup` - -Return the job immediately, do the work elsewhere, answer polls: - -```python -@app.evaluator -def evaluate(req): - job_id = enqueue(req) # hand off to a worker/queue - return JobPending(job_id=job_id, next_poll_secs=15) - -@app.job_lookup -def lookup(job_id): - row = store.get(job_id) # O(1) — see below - if row is None or row.running: - return JobPending(job_id=job_id) # "still working" - return EvalResponse(scores=row.scores) -``` - -Two traps: - -- **The poll GET shares the same 30s timeout.** `@app.job_lookup` must be a cheap - lookup that never blocks on the judge. If it waits for the result, you've - rebuilt the synchronous problem with extra steps. -- **In-process job state breaks under replicas.** The poll can land on a - different pod than the POST did, and the deploy example scales `replicas`. - Job state belongs somewhere shared — Redis, a table, anything both pods see. - -Without `@app.job_lookup` registered, polls get a 404 and the evaluation never -completes. Only register it if you return `JobPending`. - -## 10. Errors: what's retried, what's terminal - -This is the SDK's sharpest edge and it isn't in the README. **Raising an -exception is not how you report a failed evaluation.** An exception becomes a -generic HTTP 500, the server reads 5xx as transient, and it retries — five times, -re-billing your judge each time, before giving up with no scores. - -To fail *terminally*, return a raw dict — there's no typed model for it, and -`EvalResponse`/`JobPending` can't express it: - -```python -return {"status": "error", "error": "model service unavailable"} # non-empty str, required -``` - -| you return / do | server sees | outcome | -|---|---|---| -| `EvalResponse(...)` | `done` | scores stored | -| `JobPending(job_id=...)` | `pending` | polled until done | -| `{"status": "error", "error": "…"}` | `error` | **terminal**, recorded, not retried | -| `raise SomeError(...)` | 500 | **retried 5×**, then terminal with no scores | - -So: retry-worthy blips (a 429 from your model provider) — let them raise. -Permanent failures (unparseable transcript, missing config) — return the error -dict. Anything over 25 MiB gets a 413 before your code runs; that's terminal too. - -## 11. Deploy: two env vars on the server - -There is nothing to upload. AgentEye starts calling your evaluator when the -**server** has: - -```bash -EVALUATOR_ENDPOINT=http://evaluator:9000 # unset => the whole pipeline is a no-op -EVALUATOR_TOKEN= # must be byte-identical on both sides -``` - -Then restart the server. Notes worth knowing before you debug: - -- **Unset `EVALUATOR_ENDPOINT` means silence, not error.** No warning, no scores. - It's the first thing to check when nothing lands. -- **A token mismatch fails fast, not loudly** — 401 is a 4xx, so it's terminal - and never retried. -- **One evaluator per deployment.** No per-agent routing. If different agents need - different scoring, branch on `req.agent_id` inside your evaluator. -- `references/scaffold.md` has the container setup. Don't copy the Dockerfile from - `deploy/examples/evaluator/` — it installs the SDK from monorepo source, which - only works inside the AgentEye repo. - -## 12. Confirm it worked - -Scores landing is the only proof. After deploying, finish a session and check: - -```bash -fp --json evals --session-id # your scores, or status=error/timeout -fp --json evals --aggregate --since 24h -``` - -If a session shows `status: error` or `timeout`, the `error` field carries your -message (the one from the error dict — exceptions never reach it). Reading -existing scores from here on is the `fp-cloud-cli` skill's job. - -**Debug order when nothing appears:** is `EVALUATOR_ENDPOINT` set on the server → -did the session emit `agent_end` (or does `@app.config` return -`inactivity_timeout_secs`) → does the server reach your service → do the tokens -match. - - diff --git a/skills/agenteye-evaluator/agents/openai.yaml b/skills/agenteye-evaluator/agents/openai.yaml deleted file mode 100644 index 2a2ba06..0000000 --- a/skills/agenteye-evaluator/agents/openai.yaml +++ /dev/null @@ -1,9 +0,0 @@ -# Codex skill configuration (optional). See https://developers.openai.com/codex/skills -# -# Codex reads SKILL.md's `name`/`description` the same way Claude Code does. -# This file only tunes Codex-specific behavior. - -# Let Codex auto-select this skill when a task matches the description -# (set to false to require explicit `$agenteye-evaluator` invocation). -policy: - allow_implicit_invocation: true diff --git a/skills/agenteye-evaluator/references/brainstorm.md b/skills/agenteye-evaluator/references/brainstorm.md deleted file mode 100644 index b69eb5f..0000000 --- a/skills/agenteye-evaluator/references/brainstorm.md +++ /dev/null @@ -1,338 +0,0 @@ -# Brainstorm mode — deciding what to evaluate (plan only) - -The deliverable of this mode is **a decision, not code**. You research the user's -real logs *with them* and stop at a written eval plan. Do not scaffold a project, -do not write `evaluator.py`, do not touch a deploy. When they're ready to build, -that's the build loop (SKILL.md §6–12) — seeded by the plan you produced here. - -Two things make a plan worth trusting, and they run through every step: - -- **It's collaborative.** You present, they react, you refine — turn by turn, not - one big reveal at the end. The user is the only source of what "good" means; the - logs are the only source of what actually happens. You're joining the two. -- **It's data-backed.** Every observation, every candidate dimension, every - rejection is grounded in a number you actually queried — not a hunch from - skimming a couple of sessions. Run the `fp` CLI *frequently*. Before you - propose anything, scan the whole population; quantify each claim; and put the - numbers in front of the user. The goal is the most evidence-grounded plan their - telemetry can support. - -The command mechanics live in [`session-data.md`](session-data.md) — how to pull a -session, the full event vocabulary, and the CLI gotchas. This file is the *method*; -that file is the *tooling*. The palette near the end is the short list of what to -run and when. - -## Step 0 — Frame it, and say where it stops - -Open by setting the contract out loud: *"We'll read your real logs together and come -out with a plan of what's worth scoring. I won't write evaluator code in this mode — -when the plan's right, you decide whether to build it."* That one sentence keeps the -session honest and stops it drifting into a scaffold. - -Then get just enough of the human half to read the logs against — borrow §2's -highest-yield prompt (**one good-run story, one bad-run story**) and *who* the agent -serves. Keep it short. The transcripts do the heavy lifting from here; you're only -learning what to look *for*. - -## Step 1a — Scan the whole population first (data before sampling) - -Before you open a single session, quantify the landscape so your corpus is *chosen -from evidence*, not guessed. Run, at minimum: - -- **Facets — what even exists.** `fp --json list agents`, `list envs`, - `list error_types`, `list score_filters`. This is the vocabulary of their - deployment: which agents run, which environments, what errors occur, and which - score keys (if any) already exist. -- **Where it hurts, and how often.** `fp --json errors --aggregate --since 7d` - for the error hotspots; `fp --json sessions --status error,timeout --since 7d - --all --limit 1000` for the count and identity of failed runs (compare against a - healthy window to get a rate, not just a number). -- **What's already scored.** `fp --json evals --aggregate --since 7d` → - `score_stats` tells you which dimensions are tracked today and how they're - distributed — so you don't re-propose something they already have, and you can spot - a score that's uniformly high (measuring nothing) or already regressing. -- **Anything the flags can't express — drop to SQL.** `fp --json query run - --sql "…"` runs over the exposed tables `events`, `evaluations`, `agent_sessions` - (`fp --json query schema` prints the columns first — check it before - guessing one). This is how you get the shape of the whole dataset: - - ```bash - # event-type histogram across the population - fp --json query run --sql \ - "SELECT event_type, count() c FROM events GROUP BY event_type ORDER BY c DESC" - - # the biggest / longest sessions — your outlier candidates - fp --json query run --sql \ - "SELECT session_id, count() c FROM events GROUP BY session_id ORDER BY c DESC LIMIT 20" - ``` - - Once `query schema` confirms the columns your deployment promotes (e.g. tool, - duration, or token fields), the same pattern gives you duration distributions, - token distributions, and per-tool call counts — the raw material for the tool-loop, - cost, and latency dimensions you'll weigh later. - -Report the numbers back to the user as you go. This scan tells you what's common, -what's rare, and which sessions are *actually* the worst — and it directly seeds -Steps 3–5, where those same queries become discrimination evidence. - -## Step 1b — Assemble the corpus from the scan - -Now pull the sessions you'll read closely — **5–8**, chosen from Step 1a, not at -random: - -- **≥2 the user calls bad** and **≥2 they call good** — the good/bad gap *is* the - eval, so you need both piles. -- **1–2 typical** middle-of-the-road runs (a dimension that only fires on extremes - isn't much use). -- **1 outlier** the queries surfaced — the longest, the most tool-looping, the one - with the most errors. - -Pull mechanics are in [`session-data.md`](session-data.md): the export endpoint gives -a byte-identical fixture, or `fp --json events --full --session-id --order -asc --all --limit 1000`. Skim on the payload-free feed first (its one-line server -`summary` per event is ideal for reading a session's *shape*), then `--full` only on -the ones you're reading line by line. **Confirm the corpus with the user** before you -dive in — "these five, including the two you flagged; right set?" - -> **No logs yet?** If the agent isn't emitting sessions, stop — you can't ground a -> plan in transcripts that don't exist. Say so plainly, and either point them at the -> `failproofai-sdk` skill to instrument first, or brainstorm *provisionally* from -> the good/bad interview alone and stamp the plan **"unverified against real logs."** -> Don't dress a hunch up as evidence. - -## Step 2 — Read each session against a fixed catalog - -The systematic core. Read every session in the corpus against the *same* checklist, -so sessions become comparable and patterns fall out of the columns. Record -**observations, not judgments** — "searched the same query 5×", not "was inefficient". - -| Row | What to capture | Where it lives | -|---|---|---| -| **Shape** | ordered event-type skeleton, event count, wall-clock span (first→last `ts`) | the event stream; note `ended_at` may be absent | -| **Goal vs outcome** | did it do what it set out to? | `agent_start.goal` vs `agent_end.outcome`/`summary` (or the absence of `agent_end`) | -| **Tool usage / loops** | tools used, call count, distinct vs repeated, longest identical-repeat run, failures, slow calls, orphan calls | `tool_use.tool_name`/`input`, `tool_result.error`/`duration_ms`; pair by `tool_call_id` | -| **Model behavior** | models used, stop reasons (watch `length`/refusal), token totals + peaks, round-trips | `model_response.stop_reason`/`input_tokens`/`output_tokens`, count of `model_request` | -| **Errors & control flow** | error events; hook allow/deny/block; did a human have to step in, and for how long | `error.*`, `hook_completed.outcome`, `human_wait`/`human_input.duration_ms`, `human_interrupt` | -| **Sub-agents** | fan-out, and did the children finish | `agent_start` with `parent_id`; matching `agent_end` | -| **How it ended** | clean / never ended / errored / interrupted | `agent_end` vs none vs `error` vs `human_interrupt` | -| **Plain-words read** | one line: what actually went well or badly here | your judgment, *after* the rows above | - -The event-type → payload-field vocabulary is tabulated in -[`session-data.md`](session-data.md#event-vocabulary) — don't guess field names, and -remember conversation text lives in `model_request.messages` / `model_response.content` -(there's no `user_message` event type). - -## Step 3 — Diff good vs bad, then quantify the separation - -Lay the good catalogs beside the bad ones and find the rows that **move**. A candidate -dimension is born wherever an observable quantity separates the piles. - -Then don't stop at the sample — **confirm it holds across the population.** The metric -that looked decisive in two sessions may wash out over two hundred. Where the metric -is expressible in SQL, compute it per session and compare the cohorts: - -```bash -# does "how it ended" actually track the user's good/bad call? -fp --json query run --sql \ - "SELECT session_id, countIf(event_type='agent_end') AS ended, count() AS events - FROM events GROUP BY session_id ORDER BY ended ASC, events DESC LIMIT 30" -``` - -Rows that read the same across both piles are dead ends — **note them so you don't -revisit them**, and so you can show the user you checked. - -## Step 4 — Turn observations into candidate dimensions - -For each row that moves, phrase a measurable quantity **with a direction**. Worked -mappings: - -| What you saw | Candidate dimension | -|---|---| -| bad runs repeat the same search 4–6× | `tool_efficiency` — distinct ÷ total tool calls | -| bad runs never emit `agent_end` | completion / did-it-finish | -| bad runs end by telling the user to contact support | deflection rate | -| some runs burn 5× the tokens for the same goal | a token/cost **magnitude** metric | -| a human had to interrupt | human-intervention rate | -| "was the final answer correct / on-policy?" | correctness / policy — **flag the ground-truth risk now** | - -## Step 5 — Two filters, each verified with a query - -Run every candidate through both gates. These are §3–4's tests, applied *per -candidate* and *backed by data* rather than eyeballed. - -**Filter A — is the signal in the transcript, and actually populated?** Name the exact -event types and payload fields that feed the dimension. Then *prove the field is there -in their data* — the schema allowing a field doesn't mean their instrumentation emits -it: - -```bash -fp --json events --full --session-id --all --limit 1000 \ - | jq '[.events[] | select(.event_type=="tool_use") | .payload.tool_name] | unique' -``` - -If you can't name the source, cut it — or park it as **"needs instrumentation"** and -point at the `failproofai-sdk` skill. Classic casualties: "did the customer come -back next week", "was the user satisfied", anything needing external ground truth. - -**Filter B — does it discriminate, measurably?** Don't hand-wave on two sessions. -Compute the candidate across the good vs bad cohorts (SQL over the population wherever -the metric is expressible) and show the actual gap in the distributions. A candidate -that reads ~the same on both is cut, or demoted to **"monitor-only, not a quality -signal."** - -Keep a running **"rejected & why"** list as you go, each entry carrying the -query/number that killed it. It's a deliverable, not waste — it's how the user knows -you looked and why the plan is *this* and not something else. - -## Step 6 — Classify each survivor by shape and method - -So the plan is build-ready without being a build, tag every surviving dimension two -ways: - -- **Shape** (how the platform stores and renders it): - - **rate** — a 0–1 fraction, shown as a bar. The default for a quality signal. - - **magnitude metric** — a physical quantity with a unit (cost $, latency ms, - tokens). Use when the thing is a *count with a unit*, not a fraction. - - **label** — a category tag (a bucket, not a number). - - Rule of thumb: a quality *fraction* → rate; a *quantity with a unit* → magnitude; a - *bucket* → label. -- **Method:** **rule** (counting / ratios / thresholds over events — deterministic, - free, instant) vs **judge** (an LLM reads `messages`/`content` — subjective, costs - latency and money). **Default to rule.** Reach for a judge only when the signal - lives in free text and no count approximates it. Choosing a judge is a *flag for - build mode* — it's what triggers the async / 30-second concerns in SKILL.md §8–9 — - not something to solve here. - -## Step 7 — Converge with the user (all the way through) - -Collaboration isn't a final step — it runs the whole time. But it peaks here: present -the surviving candidates as a short slate, **each carrying the numbers you queried**. -Per candidate: - -- the one-line definition and its direction, -- the real session it's grounded in (cite the id and what you saw), -- its **prevalence** — how often the failure shows up across the population (e.g. - "in 34% of last week's error sessions"), -- the **measured good-vs-bad gap**. - -Then ask for reactions — keep / cut / rename / merge — and the question that catches -what you missed: *"is there a failure you've seen that none of these catch?"* Their -answer sends you back to the transcripts, maybe to pull one more session to confirm -or refute. This is present → react → refine, turn by turn. - -Two limits carried over from §4, and worth holding firm on: - -- **Converge on 2–4 dimensions.** Past four, nobody reads the dashboard and the eval - stops changing decisions. -- **Get explicit sign-off on the exact score-key names.** Names are permanent in - practice — renaming later splits the history and breaks the trend. Agree the strings - now, in the plan. - -## Step 8 — Deliver the plan, then hand the user the wheel - -Write up the plan (template below) and present it. Then **offer the next step — don't -pick it for them.** Three explicit exits: - -1. **Save it** — write the plan to a markdown file (suggest `eval-plan.md` at the repo - root or `docs/`). **Ask first, and ask where** — plan mode may be running before - the evaluator repo even exists, and the file can carry paraphrased customer data, so - never write it unprompted. Cite session ids and *paraphrased* observations; never - paste raw transcript. -2. **Build it** — switch to build mode (SKILL.md §6–12), seeded by this plan: each - dimension rule-first, tested against a real captured session, judges async. -3. **Keep brainstorming** — back to the logs: pull more sessions, refine, rename, or - merge dimensions, re-check discrimination. - -That's the whole point of the mode — the user leaves with a data-backed decision and -chooses what to do with it. - -## CLI command palette - -The go-to commands, by what you're trying to learn. Full mechanics, the event -vocabulary, and every gotcha are in [`session-data.md`](session-data.md); this is the -quick index. - -| To learn… | Run | -|---|---| -| what exists (agents, envs, errors, score keys) | `fp --json list agents` / `list envs` / `list error_types` / `list score_filters` | -| where it hurts, how often | `fp --json errors --aggregate --since 7d` · `fp --json sessions --status error,timeout --since 7d --all --limit 1000` | -| what's already scored / regressing | `fp --json evals --aggregate --since 7d` (→ `score_stats`) · `fp --json evals --score :..0.5 --since 7d --all --limit 200` | -| a session's shape, then its content | `fp --json events --session-id --order asc --all --limit 1000` → add `--full` | -| anything the flags can't (duration, tokens, loops, histograms) | `fp --json query run --sql "…"` over `events`/`evaluations`/`agent_sessions` · `query schema` for columns | - -**Run them *properly* — the gotchas that make a command lie:** - -- **Globals go before the command:** `fp --json events …`, never `fp - events --json` (exit 2). -- **`--all` is capped by `--limit` (default 50).** A bare `--all` returns 50 rows and - looks complete. Always `--all --limit 1000`; bigger needs cursor paging. -- **`--since` is a closed enum:** `all` `15m` `1h` `6h` `24h` `7d`. Anything else is a - usage error — use `--from`/`--to` with RFC3339 instead. -- **`evals`/`errors` filters are single-valued** (a repeated flag = last wins); - `sessions`/`events` take CSV and repeats. -- **Keep `--full` bound to one `--session-id`** — it's slow at scale. -- Needs `events:read`; `evals` also needs `evaluations:read`. - -## The eval-plan artifact - -The written deliverable. Structure it so every dimension traces back to a number, and -so the "rejected" list shows the work. - -```markdown -# Eval plan — -_Status: proposed · not yet built · _ - -## What this agent does -1–2 lines: the agent, its users, and what "good" means (from the interview). - -## Population scan (the data behind the plan) -The queries that framed this, with their numbers: session volume + status split, -error-type breakdown, existing score distributions, and any SQL histograms -(duration / tokens / tool loops). Every dimension below traces back to something here. - -## Sessions reviewed -| session_id | user's verdict | one-line shape | how it ended | -|---|---|---|---| -| run-8842 | bad | search ×5, no answer | no agent_end (never finished) | -| run-9001 | good | 2 tools, answered | clean agent_end | -_This corpus grounds every dimension below._ - -## Proposed dimensions (N = 2–4) -### · · -- **Definition:** what the number means, and which direction is good. -- **Why it matters:** the failure it catches, in the user's words. -- **Grounded in:** session — . -- **Signal source:** event types + payload fields (e.g. `tool_use.tool_name` + - `.input`; pair `tool_result` via `tool_call_id`). -- **Data backing:** the query/queries run + what they returned — prevalence across the - population (e.g. "34% of error sessions") and the field-populated check. -- **Discrimination evidence:** measured across cohorts — good ≈ ; bad ≈ - (from ``) — the gap. -- **Method:** rule (count/ratio) or judge (reads ); if judge, note the - async / 30s implication for build. -- **Open questions / risks:** ground-truth gaps · is the field actually populated in - their data · thresholds still to pick · non-rate shape to confirm against the SDK - contract. - -## Considered but rejected -| candidate | why rejected (with the number) | -|---|---| -| customer-returned-next-week | not in the transcript — no such event | -| answer correctness | needs ground truth we don't have; parked for a judge + labeled set | -| helpfulness (generic) | didn't discriminate — good and bad both ≈ 0.8 across 40 sessions | - -## Open decisions for the user -- Final score-key names (permanent — renaming splits history). -- Any failure mode above that none of these catch? - -## Next step (your choice) -Save this · build it (→ build mode, rule-first, tested against session ) · keep -brainstorming. Nothing here is built yet. -``` - -One caveat to carry into every dimension's **open questions**: the SDK types a score as -a plain number (`references/sdk-api.md`). A plan may legitimately call for a -**magnitude** or **label** shape — but *how* to emit a non-rate shape is a build-mode -detail. Name the intended shape here and confirm the representation against the SDK -contract when you build; don't assert it in the plan. diff --git a/skills/agenteye-evaluator/references/scaffold.md b/skills/agenteye-evaluator/references/scaffold.md deleted file mode 100644 index 540a37e..0000000 --- a/skills/agenteye-evaluator/references/scaffold.md +++ /dev/null @@ -1,270 +0,0 @@ -# Scaffolding an evaluator project - -Templates for standing up a new evaluator service. Adapt them — they're a -starting point, not a spec. If the user already has an evaluator, don't scaffold: -read their code and match it. - -## Contents - -- [Installing the SDK — the ladder](#installing-the-sdk--the-ladder) -- [Project layout](#project-layout) -- [`evaluator.py`](#evaluatorpy) -- [`tests/test_evaluator.py`](#teststest_evaluatorpy) -- [`pyproject.toml`](#pyprojecttoml) -- [`Dockerfile`](#dockerfile) -- [Running it](#running-it) -- [Wiring it to AgentEye](#wiring-it-to-agenteye) - -## Installing the SDK — the ladder - -**`pip install agenteye-evaluator` from public PyPI is not the install path.** -The package is published only as a private release artifact — there is no PyPI -publish step. Worse, the name is unclaimed on public PyPI, so an unqualified -install could pull a stranger's package. Work down this ladder and stop at the -first rung that applies: - -1. **Inside the AgentEye monorepo** (there's an `evaluator-sdk/` directory): - ```bash - pip install ./evaluator-sdk - ``` - Tracks the SDK on the current branch. This is what the in-repo examples do. - -2. **From the private release** — wheels are attached to GitHub Releases on - `agenteye-enterprise/releases`, tagged `evaluator-sdk/v`: - ```bash - gh release download evaluator-sdk/v \ - --repo agenteye-enterprise/releases --pattern '*.whl' - pip install ./agenteye_evaluator-*.whl - ``` - Needs `gh auth login` and access to that private repo. - -3. **Neither works** → stop and tell the user to ask their Failproof AI contact - for the wheel. Don't improvise an install; a wrong package here is a supply-chain - problem, not a typo. - -`uvicorn` is not an SDK dependency — install it alongside: `pip install 'uvicorn[standard]'`. - -## Project layout - -``` -my-evaluator/ -├── evaluator.py # the service -├── tests/ -│ └── test_evaluator.py -├── fixtures/ # real sessions pulled from AgentEye (see session-data.md) -│ └── run-001.json -├── pyproject.toml -├── Dockerfile -└── .env.example # EVALUATOR_TOKEN=... (never commit the real one) -``` - -Fixtures are real production transcripts. Before committing them, check with the -user whether that's OK — they can contain customer data. If in doubt, keep -`fixtures/` out of git and have each developer pull their own. - -## `evaluator.py` - -Starter with one deterministic dimension. Replace it with the dimensions the user -actually signed off on — this is scaffolding, not a recommendation of what to score. - -```python -"""Evaluator service for . Scores each finished session.""" -from __future__ import annotations - -import os - -from agenteye_evaluator import EvalRequest, EvalResponse, Evaluator - -app = Evaluator(token=os.environ.get("EVALUATOR_TOKEN")) - - -@app.config -def config(): - # Sessions that never emit `agent_end` are only ever scored if we advertise - # this: it tells the server how long to wait before evaluating an idle - # session. Without it, a crashed run is silently never scored. - return {"inactivity_timeout_secs": 1800} - - -@app.evaluator -def evaluate(req: EvalRequest) -> EvalResponse: - events = req.events - tool_uses = [e for e in events if e.event_type == "tool_use"] - errors = sum(1 for e in events if e.event_type == "error") - - # Repeating an identical call is the signature of a stuck agent: score the - # ratio of distinct calls to total. - if tool_uses: - distinct = {(e.payload.get("tool_name"), str(e.payload.get("input"))) for e in tool_uses} - efficiency = round(len(distinct) / len(tool_uses), 2) - why = f"{len(distinct)} distinct of {len(tool_uses)} tool call(s)." - else: - efficiency = 1.0 - why = "no tool calls — nothing to repeat." - - return EvalResponse( - scores={"tool_efficiency": efficiency}, - reasoning={"tool_efficiency": why}, - summary=f"{len(events)} event(s), {len(tool_uses)} tool call(s), {errors} error(s).", - ) -``` - -## `tests/test_evaluator.py` - -The decorators return the function unchanged, so test the scoring logic by -calling it directly — no HTTP, no client. - -```python -import pytest -from agenteye_evaluator import EvalRequest - -from evaluator import evaluate - - -def make_req(events, ended_at=None): - return EvalRequest.model_validate({ - "schema_version": "1", - "session_id": "t", "agent_id": "a", "environment": "test", - "started_at": "2026-01-01T00:00:00Z", "ended_at": ended_at, - "events": events, - }) - - -def event(i, event_type, **payload): - return {"id": i, "ts": f"2026-01-01T00:00:{i:02d}Z", - "event_type": event_type, "payload": {"type": event_type, **payload}} - - -def test_repeated_tool_calls_score_low(): - events = [event(i, "tool_use", tool_name="search", input={"q": "x"}) for i in range(4)] - assert evaluate(make_req(events)).scores["tool_efficiency"] == 0.25 - - -def test_distinct_tool_calls_score_high(): - events = [event(i, "tool_use", tool_name=f"t{i}", input={"q": i}) for i in range(4)] - assert evaluate(make_req(events)).scores["tool_efficiency"] == 1.0 - - -def test_empty_session_does_not_crash(): - assert evaluate(make_req([])).scores["tool_efficiency"] == 1.0 - - -def test_session_without_agent_end(): - # ended_at is None for every session the inactivity scanner picks up — - # the common case, not an edge case. - assert evaluate(make_req([event(1, "tool_use", tool_name="s")], ended_at=None)).scores - - -@pytest.mark.skipif(not __import__("pathlib").Path("fixtures/run-001.json").exists(), - reason="no fixture pulled yet") -def test_real_session(): - req = EvalRequest.model_validate_json(open("fixtures/run-001.json").read()) - scores = evaluate(req).scores - assert 0.0 <= scores["tool_efficiency"] <= 1.0 -``` - -Use `TestClient(app.app)` only for wire-level concerns (auth, status codes): - -```python -from fastapi.testclient import TestClient -from evaluator import app - -def test_health_is_open(): - assert TestClient(app.app).get("/health").status_code == 200 -``` - -`TestClient` needs starlette's HTTP client, which the SDK doesn't pull in and -which **changed name across versions** — older starlette wants `httpx`, current -starlette wants `httpx2`, and importing `TestClient` without the right one raises -a `RuntimeError` naming the package it wants. Install whichever it asks for. This -is a good reason to keep wire-level tests to a minimum: the scoring tests above -need neither. - -## `pyproject.toml` - -```toml -[project] -name = "my-evaluator" -version = "0.1.0" -requires-python = ">=3.10" -dependencies = [ - "agenteye-evaluator", # installed via the ladder above, not from public PyPI - "uvicorn[standard]>=0.30", -] - -[project.optional-dependencies] -# Testing the scoring function needs nothing but pytest. TestClient additionally -# needs starlette's HTTP client — see the note under the TestClient snippet above. -dev = ["pytest>=8"] - -# evaluator.py sits at the project root, so pytest needs the root on sys.path to -# import it from tests/. Without this, `from evaluator import evaluate` fails -# with ModuleNotFoundError. -[tool.pytest.ini_options] -pythonpath = ["."] -``` - -Pin `agenteye-evaluator` however your install path dictates — a wheel path, a -private index, or a vendored copy. Leaving it as a bare public dependency is the -one thing to avoid. - -## `Dockerfile` - -**Don't copy the one from `deploy/examples/evaluator/`** — it does -`COPY evaluator-sdk /app/evaluator-sdk`, which only works inside the AgentEye -monorepo. Bring the wheel in instead: - -```dockerfile -FROM python:3.12-slim - -WORKDIR /app - -# Ship the wheel alongside the build context (rung 2 of the install ladder). -COPY wheels/ /tmp/wheels/ -RUN pip install --no-cache-dir /tmp/wheels/agenteye_evaluator-*.whl \ - && pip install --no-cache-dir 'uvicorn[standard]>=0.30' \ - && rm -rf /tmp/wheels - -COPY evaluator.py /app/ - -# Non-root so the image satisfies a Kubernetes runAsNonRoot policy. -RUN useradd -u 10001 -m app -USER 10001 - -EXPOSE 9000 -CMD ["uvicorn", "evaluator:app", "--host", "0.0.0.0", "--port", "9000"] -``` - -## Running it - -```bash -EVALUATOR_TOKEN=dev-secret uvicorn evaluator:app --host 0.0.0.0 --port 9000 - -curl -s localhost:9000/health # open, no auth -curl -s -H "Authorization: Bearer dev-secret" localhost:9000/config -curl -s -H "Authorization: Bearer dev-secret" -H 'Content-Type: application/json' \ - --data @fixtures/run-001.json localhost:9000/evaluate # replay a real session -``` - -That last call is the highest-value check you can run: a real transcript through -the real wire path, before anything is deployed. - -## Wiring it to AgentEye - -Nothing is uploaded. The **AgentEye server** needs two env vars and a restart: - -```bash -EVALUATOR_ENDPOINT=http://evaluator:9000 -EVALUATOR_TOKEN=dev-secret # byte-identical to the evaluator's -``` - -Then finish a session and confirm the scores landed: - -```bash -fp --json evals --session-id -``` - -If nothing appears, work down: is `EVALUATOR_ENDPOINT` set on the server (unset is -a silent no-op) → did the session emit `agent_end`, or does `@app.config` return -`inactivity_timeout_secs` → can the server reach your host/port → do the tokens -match (a mismatch is a 401, which is terminal and never retried). diff --git a/skills/agenteye-evaluator/references/sdk-api.md b/skills/agenteye-evaluator/references/sdk-api.md deleted file mode 100644 index f220440..0000000 --- a/skills/agenteye-evaluator/references/sdk-api.md +++ /dev/null @@ -1,240 +0,0 @@ -# `agenteye-evaluator` API reference - -The SKILL.md body has the workflow and the design loop; this is the flag-level -contract. Read it when you need an exact signature, field, or status shape. - -## Contents - -- [Exports](#exports) -- [`Evaluator`](#evaluator) -- [The three decorators](#the-three-decorators) -- [Models](#models) -- [Return shapes and coercion](#return-shapes-and-coercion) -- [HTTP routes](#http-routes) -- [Wire format](#wire-format) -- [Server-side dispatch: what calls you, and how](#server-side-dispatch-what-calls-you-and-how) -- [Server env vars](#server-env-vars) -- [Logging](#logging) - -## Exports - -Everything public comes from the top-level package. Anything under -`agenteye_evaluator._models` / `._server` is private and may move. - -```python -from agenteye_evaluator import ( - AgentEvent, EvalRequest, EvalResponse, EvaluatorConfig, Evaluator, JobPending, __version__, -) -``` - -Package name `agenteye-evaluator`, import name `agenteye_evaluator`. Requires -Python ≥ 3.10. Depends on `fastapi`, `pydantic>=2`, `structlog`. **`uvicorn` is -not a dependency** — install it yourself to serve the app. - -## `Evaluator` - -```python -Evaluator(token: str | None = None, *, title: str = "AgentEye Evaluator") -``` - -- `token` — the shared secret. Compared with `hmac.compare_digest`. **`token=None` - disables auth entirely**, which is fine locally and a hole in production. -- `title` — keyword-only; FastAPI app title, cosmetic. - -Attributes: `.app` is the underlying `FastAPI` instance (use it for -`TestClient(app.app)`). The `Evaluator` itself is an ASGI callable, so -`uvicorn evaluator:app` works directly. - -## The three decorators - -Each returns the function **unchanged**, so decorated functions stay directly -callable — that's what makes unit tests cheap. Each accepts a **sync or async** -function. Each raises `ValueError` if registered twice. - -| decorator | serves | required? | -|---|---|---| -| `@app.evaluator` | `POST /evaluate` | Yes — this is the evaluator. | -| `@app.job_lookup` | `GET /evaluate/{job_id}` | Only if you ever return `JobPending`. Absent → polls get **404**. | -| `@app.config` | `GET /config` | No, but see `inactivity_timeout_secs` below. | - -```python -EvalReturn = Union[EvalResponse, JobPending, dict] -EvaluatorFn = Callable[[EvalRequest], Union[EvalReturn, Awaitable[EvalReturn]]] -JobLookupFn = Callable[[str], Union[EvalReturn, Awaitable[EvalReturn]]] -ConfigFn = Callable[[], Union[EvaluatorConfig, dict, Awaitable[...]]] -``` - -## Models - -All models set `extra="ignore"`, so unknown keys are dropped rather than -rejected. That's why a CLI event dict validates straight into `AgentEvent`. - -```python -class AgentEvent(BaseModel): - id: int - ts: datetime - event_type: str - payload: dict[str, Any] = Field(default_factory=dict) - -class EvalRequest(BaseModel): - schema_version: str - session_id: str - agent_id: str - environment: str - started_at: datetime - ended_at: datetime | None = None - events: list[AgentEvent] = Field(default_factory=list) - -class EvalResponse(BaseModel): - scores: dict[str, float] | None = None - reasoning: dict[str, str] | None = None - summary: str | None = None - -class JobPending(BaseModel): - job_id: str - next_poll_secs: int | None = None - -class EvaluatorConfig(BaseModel): - inactivity_timeout_secs: int | None = None - default_poll_interval_secs: int = 10 -``` - -Field notes that bite: - -- **`ended_at` is the `agent_end` event's timestamp, or `None`.** Not "when the - session stopped". Sessions evaluated by the inactivity scanner never had an - `agent_end`, so they arrive `None`. Deriving duration from it crashes on real data. -- **`payload` is the whole event JSON flattened** — event-specific fields sit at - the top level, and `payload["type"]` duplicates `event_type`. Keys per event - type are in `session-data.md`. -- **`payload` typed as `dict` rejects an explicit `null`.** The default only - applies to a *missing* key. A server-side unparseable payload serializes as - `null` → 422 → terminal. Rare, and not something your code can prevent. -- `scores` keys are arbitrary; the platform trends whatever you send. -- **`summary` is truncated at 8192 bytes** server-side (`last_error` at 2048). -- Serialization uses `exclude_none`, so unset fields are omitted, not `null`. - -## Return shapes and coercion - -Your function may return one of exactly three things. Anything else is a -`TypeError` → HTTP 500. - -| return | wire `status` | terminal? | -|---|---|---| -| `EvalResponse(...)` | `done` | yes — scores stored | -| `JobPending(job_id=...)` | `pending` | no — server polls | -| `dict` with `status` ∈ `{done, pending, error}` | as given | `error` is terminal | - -**The `error` status has no typed model.** To fail terminally you must return a -raw dict, and `error` must be a **non-empty `str`**: - -```python -return {"status": "error", "error": "model service unavailable"} -``` - -Coercion edges, all pinned by tests: - -- `dict` without `status` → 500. -- `{"status": "pending"}` without `job_id` → 500 (at the SDK, before the wire). -- `{"status": "error"}` with no message, or a non-str message → 500. -- `done`/`pending` dicts drop unknown keys (`extra="ignore"`); the `error` path - **preserves** extras (`{"status": "error", **data}`). Asymmetric on purpose. - -**Raising is not reporting.** An exception becomes a generic 500 with the body -`"evaluator raised an internal error"` — your exception text never reaches the -server (`from None`), and the server treats 5xx as *transient* and retries. - -## HTTP routes - -| route | auth | purpose | -|---|---|---| -| `GET /health` | **open even when a token is set** | liveness | -| `POST /evaluate` | bearer | the evaluation | -| `GET /evaluate/{job_id}` | bearer | poll an async job | -| `GET /config` | bearer | advertise timeouts/cadence | - -- Bearer scheme match is case-insensitive (`bearer xyz` is accepted, per RFC 6750). -- **Request body cap `MAX_BODY_BYTES` = 25 MiB** — checked against `Content-Length` - *before* the body is read. Over → 413 → 4xx → terminal, not retried. -- `GET /config` with no `@app.config` registered still returns - `{"default_poll_interval_secs": 10}` — the SDK always advertises a cadence. -- Validation failures return 422 and **deliberately do not echo the payload** - (a transcript would leak into logs); likewise 500s never echo exception text, - and the token never appears in any log field. Tests assert all three. - -## Wire format - -Request (`POST /evaluate`): - -```json -{ - "schema_version": "1", - "session_id": "run-001", - "agent_id": "support-bot", - "environment": "prod", - "started_at": "2026-01-01T00:00:00Z", - "ended_at": null, - "events": [ - {"id": 1, "ts": "2026-01-01T00:00:00Z", "event_type": "agent_start", - "payload": {"type": "agent_start", "goal": "help the user", "session_id": "run-001"}} - ] -} -``` - -Response, done / pending / error: - -```json -{"status": "done", "scores": {"helpfulness": 0.9}, "reasoning": {"helpfulness": "..."}, "summary": "..."} -{"status": "pending", "job_id": "abc-123", "next_poll_secs": 15} -{"status": "error", "error": "model service unavailable"} -``` - -## Server-side dispatch: what calls you, and how - -Worth knowing because it explains every timeout and duplicate you'll see. - -1. **Enqueue.** An `agent_end` event enqueues one job, `ON CONFLICT (org_id, - session_id) DO NOTHING` — **one in-flight job per session**, so concurrent - `agent_end`s don't double-score. -2. **Fallback enqueue.** A scanner (every 60s) enqueues idle sessions — **only if - your `GET /config` returns `inactivity_timeout_secs`**. Values ≤ 0 are dropped. - Config is re-fetched every `EVALUATOR_CONFIG_REFRESH_SECS` (default 300). -3. **Claim.** `EVALUATOR_WORKERS` (2) × `EVALUATOR_CLAIM_BATCH` (4) → - **8 concurrent calls** against your endpoint, deployment-wide. -4. **Dispatch.** `POST /evaluate`, `Authorization: Bearer`, `User-Agent: - agenteye-server/`, timeout `EVALUATOR_REQUEST_TIMEOUT_MS` (**30s**). -5. **Classify.** `done` → terminal. `pending` + `job_id` → poll (`job_id` may be - omitted on a *poll* response, but not on the POST — that's a protocol - violation and terminal). `error` → terminal. Unknown/missing `status` → - terminal error. **5xx / 429 / transport → transient, retried** with backoff - (base 2s, cap 1800s) up to `EVALUATOR_MAX_ATTEMPTS` (5). **4xx → terminal**, so - a token mismatch fails immediately rather than retrying. -6. **Poll.** `GET /evaluate/{job_id}`, **same 30s timeout**. Cadence precedence, - each clamped to [1s, 3600s]: response `next_poll_secs` → `/config`'s - `default_poll_interval_secs` → `EVALUATOR_POLLING_INTERVAL_SECS` (10). Wallclock - cap `EVALUATOR_MAX_POLL_DURATION_SECS` (3600) → recorded as `timeout`. -7. **Land.** Terminal results are written to ClickHouse `agenteye.evaluations` - with `status` ∈ `done | error | timeout`. Multiple evaluations accumulate per - session as a timeline. - -## Server env vars - -Set on the **AgentEye server**, not on your evaluator. - -| var | default | notes | -|---|---|---| -| `EVALUATOR_ENDPOINT` | — | **Unset → the whole pipeline is a silent no-op.** | -| `EVALUATOR_TOKEN` | — | Must match your `Evaluator(token=...)` byte for byte. | -| `EVALUATOR_REQUEST_TIMEOUT_MS` | 30000 | Applies to POST **and** poll GET. | -| `EVALUATOR_WORKERS` | 2 | × claim batch = concurrency. | -| `EVALUATOR_CLAIM_BATCH` | 4 | | -| `EVALUATOR_MAX_ATTEMPTS` | 5 | Retries on transient failures. | -| `EVALUATOR_POLLING_INTERVAL_SECS` | 10 | Lowest-precedence cadence. | -| `EVALUATOR_MAX_POLL_DURATION_SECS` | 3600 | Then `timeout`. | -| `EVALUATOR_CONFIG_REFRESH_SECS` | 300 | How often `/config` is re-read. | - -## Logging - -The SDK logs via `structlog`. Notable events: `/config` responses tag -`source="user"` vs `source="default"` so you can tell whether your `@app.config` -was actually picked up. The bearer token is never logged. diff --git a/skills/agenteye-evaluator/references/session-data.md b/skills/agenteye-evaluator/references/session-data.md deleted file mode 100644 index b84fad0..0000000 --- a/skills/agenteye-evaluator/references/session-data.md +++ /dev/null @@ -1,198 +0,0 @@ -# Getting real session data - -Two jobs: **find sessions worth looking at**, and **turn one into a fixture** you -can replay through your evaluator. Both go through the `fp` CLI, which -needs a logged-in session (`fp login`) and a dashboard URL -(`AGENTEYE_DASHBOARD_URL` or `--base-url`). - -## Contents - -- [Getting a fixture — the export endpoint](#getting-a-fixture--the-export-endpoint) -- [Fallback — reconstructing from the CLI](#fallback--reconstructing-from-the-cli) -- [Finding sessions worth reading](#finding-sessions-worth-reading) -- [Reading a session](#reading-a-session) -- [Event vocabulary](#event-vocabulary) -- [CLI gotchas](#cli-gotchas) - -## Getting a fixture — the export endpoint - -There's an endpoint whose entire purpose is this: `GET /api/sessions/{id}/export` -returns **byte-identical bytes to what your evaluator receives**, because it's -generated by the same builder the dispatcher uses. The route deliberately -forwards raw bytes without re-serializing, so key order and number precision -survive. - -The CLI has no `export` command, but the endpoint takes the same `ae_session` -cookie your CLI login already stored (it's permission-gated on `events:read`), so -`curl` reaches it with no new credential: - -```bash -BASE="${AGENTEYE_DASHBOARD_URL:?set your dashboard URL}" -TOKEN=$(python3 -c "import json,os,pathlib; \ - p=pathlib.Path(os.environ.get('AGENTEYE_HOME') or (pathlib.Path.home()/'.agenteye'))/'cli.json'; \ - print(json.load(open(p))['session_token'])") - -mkdir -p fixtures -curl -sSf -b "ae_session=$TOKEN" "$BASE/api/sessions/run-001/export" -o fixtures/run-001.json -``` - -Multi-org logins need the active tenant too — add `-H "X-AgentEye-Org: "` -(`fp --json orgs current` tells you the slug). If it 401s, the session -expired: `fp login`. If it 403s, the login lacks `events:read`. - -Then it round-trips straight into the model, which is the whole point: - -```python -from agenteye_evaluator import EvalRequest -req = EvalRequest.model_validate_json(open("fixtures/run-001.json").read()) -``` - -**Prefer this over anything below.** It's the only source that can't drift from -what production sends. - -## Fallback — reconstructing from the CLI - -If `curl` isn't available or the endpoint is unreachable, you can approximate the -body from the events feed. Be honest with the user that it's an approximation. - -**Build it from `events` alone — never from `fp sessions`.** The sessions -feed looks like the right source and isn't: - -- It's anchored on `agent_start`, so a session without one returns **zero rows** - even though the real pipeline still evaluates it. -- Its `started_at` is the min over `agent_start` rows only, not over all events. -- Adding `--since` re-scopes `started_at` to the window, silently corrupting it. -- It has **no `ended_at` field at all** — and `last_event_at` is *not* `ended_at`. - Mapping one onto the other fabricates an `ended_at` for precisely the sessions - that don't have one. - -The events feed carries `session_id`, `agent_id`, and `environment` on every row, -so with `--order asc` everything you need is derivable: - -```bash -fp --json events --full --session-id run-001 --order asc --all --limit 1000 \ - > /tmp/events.json -``` - -```python -import json -from agenteye_evaluator import EvalRequest - -rows = json.load(open("/tmp/events.json"))["events"] -if not rows: - raise SystemExit("no events — wrong session id, or the window excluded them") - -last_end = max((e["ts"] for e in rows if e["event_type"] == "agent_end"), default=None) - -req = EvalRequest.model_validate({ - "schema_version": "1", - "session_id": rows[0]["session_id"], - "agent_id": rows[0]["agent_id"], - "environment": rows[0]["environment"], - "started_at": rows[0]["ts"], # --order asc => first row is min(ts) - "ended_at": last_end, # None when the session never ended cleanly - "events": rows, # extra CLI-only keys are dropped by extra="ignore" -}) -``` - -`events` needs no massaging: `AgentEvent` ignores unknown fields, so the CLI's -extra columns (`summary`, `is_error`, `output_tokens`, …) fall away and -`id`/`ts`/`event_type`/`payload` remain. - -Where this still differs from production: a payload the server can't parse is -sent as `payload: null` (which **422s your evaluator**), but the CLI coerces it to -`{}`, so a reconstructed fixture can never contain that case. - -## Finding sessions worth reading - -You want a good one and a bad one. Discover valid filter values before filtering — -guessing an env or agent id wastes a round trip: - -```bash -fp --json list agents # valid agent ids -fp --json list envs # valid environments -fp --json list score_filters # score keys that already exist -fp --json list error_types -``` - -| goal | command | -|---|---| -| failed runs | `fp --json sessions --status error,timeout --since 7d --all --limit 1000` | -| where it hurts | `fp --json errors --aggregate --since 7d` | -| recent runs | `fp --json sessions --since 24h --all --limit 200` | -| already-scored bad runs | `fp --json evals --score helpfulness:..0.5 --since 7d --all --limit 200` | -| which score regressed | `fp --json evals --aggregate --since 7d` → `score_stats` | - -There is **no latency or duration filter** on any command, and `sessions` has no -`--score` flag (score-based discovery goes through `evals`, then take -`session_id`). For anything else — duration, payload predicates, custom -ordering — drop to SQL: - -```bash -fp --json query run --sql "SELECT session_id, count() c FROM events GROUP BY session_id ORDER BY c DESC LIMIT 10" -``` - -Exposed tables: `events`, `evaluations`, `agent_sessions`. `fp --json query schema` -shows the layout. - -## Reading a session - -```bash -fp --json events --session-id run-001 --order asc --all --limit 1000 # timeline (no payload) -fp --json events --full --session-id run-001 --order asc --all --limit 1000 # with payload -``` - -The default feed is payload-free and carries a server-computed one-line `summary` -per event — ideal for skimming a transcript's shape. Add `--full` when you need -the actual content. Keep `--full` bounded to one `--session-id`; it's slow at scale. - -## Event vocabulary - -`payload` is the **entire event JSON flattened** — these are top-level keys, not -nested under anything. `payload["type"]` duplicates `event_type`. Every payload -also carries `timestamp`, `session_id`, `agent_id`, `environment`, plus whatever -the SDK caller passed as extra fields. - -| `event_type` | payload keys you'd score from | -|---|---| -| `agent_start` | `goal`, `parent_id` | -| `agent_end` | `outcome`, `summary` | -| `tool_use` | `tool_name`, `tool_call_id`, `input` | -| `tool_result` | `tool_name`, `tool_call_id`, `output`, `error`, `duration_ms` | -| `model_request` | `model`, `messages`, `system`, `tools` | -| `model_response` | `model`, `stop_reason`, `input_tokens`, `output_tokens`, `content`, `role` | -| `error` | `error_type`, `message`, `traceback` | -| `hook_triggered` | `hook_name`, `hook_id`, `trigger_event`, `input` | -| `hook_completed` | `hook_name`, `hook_id`, `outcome`, `output`, `error`, `duration_ms` | -| `human_wait` | `input_id`, `prompt`, `options`, `reason` | -| `human_input` | `input_id`, `response`, `duration_ms` | -| `human_pause` | `reason`, `user_id` | -| `human_interrupt` | `reason`, `user_id`, `at_step` | - -Correlate a `tool_use` with its `tool_result` via `tool_call_id`. Conversation -text lives in `model_request.messages` and `model_response.content` — there is no -`user_message` / `agent_message` event type in the Python SDK's schema, so don't -assume one exists just because an example scores it. - -**Verify against the user's own data before scoring a key.** Instrumentation -varies; a field the schema allows may never be populated in their deployment: - -```bash -fp --json events --full --session-id run-001 --all --limit 1000 \ - | jq '[.events[] | select(.event_type=="tool_use") | .payload.tool_name] | unique' -``` - -## CLI gotchas - -- **Globals go before the command**: `fp --json events`, never - `fp events --json` (exit 2). -- **`--all` is capped by `--limit`, which defaults to 50.** A bare `--all` returns - 50 rows with `next_cursor: null`, looking complete. Pass `--all --limit 1000`. - Server-side the limit clamps at 1000 — a bigger session needs cursor paging. -- **`--since` is a closed enum**: `all`, `15m`, `1h`, `6h`, `24h`, `7d`. Anything - else is a usage error; use `--from`/`--to` with full RFC3339 (`2026-05-01T00:00:00Z`). -- **Exit codes**: 0 ok · 2 usage · 3 unreachable · 4 not signed in (`fp login`, - which you can't complete for them — it needs an emailed code) · 5 missing - permission · 6 not found. -- `sessions`/`events` filters accept repeated flags and CSV; `evals`/`errors` - filters are single-valued (a repeated flag doesn't accumulate — last wins). diff --git a/skills/failproofai-eval-brainstorm/SKILL.md b/skills/failproofai-eval-brainstorm/SKILL.md new file mode 100644 index 0000000..5c2c5c8 --- /dev/null +++ b/skills/failproofai-eval-brainstorm/SKILL.md @@ -0,0 +1,261 @@ +--- +name: failproofai-eval-brainstorm +description: |- + Works out WHAT is worth measuring about an AI agent's production runs: reads the user's real sessions and returns 2-4 specific, writable evaluation proposals, each with the prompt that authors it. Reach for it on vague phrasing — "I want evals", "what should I be measuring?", "how do I know if my agent is any good?" + + Trigger when the user wants to: + • decide what to score — which dimensions are worth tracking, grounded in their own sessions; + • find blind spots — what keeps going wrong that nothing measures; + • sanity-check an idea — is it worth writing, can it be written against their data, do they have it already. + + Stops at the proposal; never writes evaluator code. + + NOT for authoring or deploying the evaluation itself (the eval authoring page does that, from this prompt), reading scores that already exist or spotting a regression (`fp-cloud-cli`), making an agent emit events at all (`failproofai-sdk`), or turning a recurring behaviour into enforcement (`failproofai-policy-author`). +--- + +# Finding what to evaluate + +An evaluation runs over a finished session and returns one of three things: a score +(a 0–1 fraction), a metric (a quantity with a unit), or an assertion (pass/fail). +You describe it in plain English and the authoring page writes it, tests it against +real sessions, and deploys it. + +So the code is not the scarce thing. **Knowing which number is worth having is.** +That is this skill's entire job: + + scan the population → confirm the signal exists → diff good against bad + → 2-4 candidates → the prompt that authors each one + +## Where this stops + +You produce **proposals**, not evaluations. Each proposal ends in a prompt the user +can paste into eval authoring, which handles composing, backtesting and deploying. +Say this out loud at the start: + +> "We'll read your real sessions and come out with a short list of what's worth +> measuring. I won't write the evaluator — when the list is right, you decide which +> ones to author." + +That sentence keeps the session honest. Without it this drifts into authoring, and +authoring without a grounded answer to *what* is how people end up with four +evaluations nobody looks at. + +## 1. Learn what "good" means — briefly + +The logs say what happened. Only the user says what *should* have. Get two things +and move on: + +- **one good-run story and one bad-run story** — the highest-yield question there is; +- **who the agent serves**, and what it is on the hook for. + +Keep it short. You are only learning what to look *for*; the sessions do the rest. + +## 2. Scan the whole population before you sample + +Choose your corpus from evidence, not from whichever session is on screen. Establish, +with actual queries: + +- **What exists** — which agents run, in which environments, what error types occur. +- **Where it hurts** — error hotspots, and the rate (compare a window against a + baseline; a raw count is not a rate). +- **The shape of the data** — event-type histogram, session length distribution, + the biggest and longest sessions. + +Report the numbers back as you go. This scan is not preamble; it is what makes every +later claim checkable, and it directly seeds the discrimination work in step 6. + +Mechanics differ by where you are running — see `references/dashboard.md` (assistant) +or `references/cli.md` (local, with `fp`). + +## 3. Check what is already measured + +Two different questions, and you need both: + +- **What is defined** — the evaluations that exist, enabled or not. +- **What has results** — the score keys actually landing, and their distributions. + +They are not the same. An evaluation deployed yesterday has a definition and no +results, and if you only look at results you will propose it again. Re-proposing +something the user already has is the fastest way to lose their trust in the slate. + +While you are here, look at the distributions: a score that is uniformly high across +every session is measuring nothing useful, and that is worth saying. + +## 4. Confirm the signal is really there + +**This is the step that separates a proposal from a wish**, and it is the one people +skip. + +`event.payload` is free-form. An evaluation that reads a key the user's agents never +emit does not fail — it reads nothing on every session, scores them all identically, +and looks exactly like a working evaluation. Nobody notices for a month. + +So before proposing anything, check the payload profile: for each event type, which +top-level payload keys actually appear, in what fraction of events, with what types +and value sets. Then, for every candidate: + +- **Name the exact event types and payload fields it reads.** +- **Confirm each one is present**, and say at what rate. A key on 4% of events is not + a foundation. + +If the signal is not in the telemetry, do not propose it. Say so plainly and name what +instrumentation would be needed — that is a real answer, and it points at +`failproofai-sdk`. Classic casualties: "was the user satisfied", "did the customer come +back", anything needing ground truth that was never recorded. + +## 5. Read the sessions that disagree + +Now pull sessions — **5 to 8**, chosen from the scan, not at random: + +- **≥2 the user calls bad** and **≥2 they call good**. The gap between those piles + *is* the evaluation; you need both. +- **1-2 ordinary** runs — a measurement that only fires on extremes is not much use. +- **1 outlier** the queries surfaced: the longest, the loopiest, the most errors. + +Confirm the corpus before you dig in: *"these six, including the two you flagged — +right set?"* + +Read every one against the **same** checklist, so they become comparable and patterns +fall out of the columns. Record observations, not judgments — "searched the same query +five times", not "was inefficient". + +| Row | What to capture | +|---|---| +| **Shape** | ordered event-type skeleton, event count, wall-clock span | +| **Goal vs outcome** | what it set out to do, against how it ended | +| **Tool usage** | tools used, call count, distinct vs repeated, longest identical-repeat run, failures | +| **Model behaviour** | stop reasons, round-trips, token peaks | +| **Errors & control flow** | error events, hook allow/deny, whether a human had to step in | +| **Sub-agents** | fan-out, and whether the children finished | +| **How it ended** | cleanly / never / errored / interrupted | +| **Plain-words read** | one line on what actually went well or badly — *after* the rows above | + +## 6. Two gates, each carrying a number + +Every candidate passes both, or it is cut and the cut is reported. + +**Gate A — is it there?** Step 4, applied per candidate. Named fields, confirmed +present, presence rate stated. + +**Gate B — does it discriminate?** Compute the candidate across the good and bad +cohorts and state the gap. Two sessions are an anecdote; the metric that looked +decisive in a pair often washes out over two hundred, and SQL over the population is +how you find that out before the user does. + +A candidate that reads the same on both sides is cut, or demoted to "worth watching, +not a quality signal". + +> **Low variance is not a defect.** An evaluation asked for "fraction of tool calls +> that errored" returning 0.00 on healthy sessions is returning the *right answer*. +> Never rewrite a working measurement because the number does not move — that is how +> people talk themselves out of their best evaluations. + +**Keep a running "considered and rejected" list**, each entry carrying the number that +killed it, and **put it in your answer** — it is a required part of the output, not a +note to yourself. It is how the user knows you looked, and why the slate is this one +and not something else. + +Without it a slate is unfalsifiable: three confident proposals read exactly the same +whether you tested ten candidates or thought of three. **Name at least one thing you +cut and the number that cut it**, every time. "I checked X; good runs 0.81, bad runs +0.79 across 40 sessions, so it is not a quality signal" is worth more to the reader +than a fourth proposal. + +## 7. Converge — 2 to 4, named once + +Present the survivors as a short slate and ask for reactions: keep, cut, rename, merge. +Then the question that catches what you missed: *"is there a failure you've seen that +none of these would catch?"* Their answer sends you back to the sessions. + +Two limits to hold firm on: + +- **2 to 4 proposals.** Past four nobody reads the dashboard and the evaluation stops + changing decisions. +- **Get explicit sign-off on each key name.** Keys are permanent in practice — + renaming splits the history and breaks every trend built on it. Agree the exact + strings now. + +## 8. Hand over the prompt — and the link, in the same breath + +The deliverable for each survivor is **the prompt**, written so the authoring page +composes cleanly from it on the first try. That has its own craft — the grammar that +picks the result type, how specific to be, which field names to name outright. See +`references/writing-the-prompt.md`. + +**Then build the link for every proposal, before you write your answer.** As the +dashboard assistant that is `build_eval_authoring_link`, once per survivor. From a +machine with `fp` there is no such tool — build it yourself, as `references/cli.md` +shows. Either way the link goes in the proposal. A proposal without its link is not +finished — the whole point is that the operator can act on it in one click, and a +prompt they have to copy, navigate to authoring, and paste is most of the friction +this exists to remove. + +Three things not to do, because each one reads as helpful and lands as a dead end: + +- **Do not offer to generate the links.** "Shall I generate the authoring links?" is + a round trip that buys nothing — you already know which proposals you are making. +- **Do not ask which ones they want first.** Build a link for each; they pick by + clicking. Choosing is the cheap part for them and the expensive part for you. +- **Do not write the words "link" or "here are the links" without a link.** Describing + a link you did not build is worse than omitting it: it reads as done. + +The rule in one line: **if you named a proposal, you built its link.** + +## The proposal format + +Each proposal, in full: + +> **`tool_retry_loop`** · score · for `checkout-bot` in production +> +> **Measures:** fraction of tool calls that repeat the previous call's tool and +> arguments exactly. 0 is healthy, 1 is a pure retry loop. +> +> **Why you:** 34% of your error sessions last week show four or more identical +> retries; your clean sessions show none. Nothing raises an error, so no alert fires — +> a standing score is the only way to see this move. +> +> **Reads:** `tool_use.tool_name` (100% of `tool_use` events), `tool_use.input` (98%). +> +> **Grounded in:** session `run-8842` — `search_orders` called five times with +> byte-identical arguments, then the session ended with no answer. +> +> **Prompt:** "Fraction of a session's tool calls that repeat the previous call's tool +> name and input exactly. Read `tool_use.tool_name` and `tool_use.input`. 0 when every +> call is distinct, 1 when every call after the first is a repeat." +> +> **Author it:** [/acme/eval-authoring/new?intent=…](#) ← the link from step 8 + +**Every field above is required, including the last one.** A proposal that stops at +the prompt has handed the operator homework instead of a decision. + +Then **the rejected list** — a short table of what you considered and the number that +killed each one — and the open question about what none of them catch. An answer with +proposals but no rejected list is incomplete, however good the proposals are. + +## When the honest answer is "nothing yet" + +Two cases, and both are real answers — say them plainly rather than manufacturing a +slate: + +- **No telemetry.** Nothing to ground a proposal in. Point at `failproofai-sdk` to + instrument first, or brainstorm provisionally from the good/bad stories alone and + stamp it **"unverified against real sessions"**. Never dress a hunch as evidence. +- **Nothing separates.** You looked, and no observable quantity tracks the user's + good/bad call. Report what you checked and what it returned. That is a finding about + their fleet, and it is more useful than four generic dimensions. + +## Reference files + +| File | Read it for | +|---|---| +| `references/patterns.md` | observation → candidate catalog, and the result-kind mapping | +| `references/writing-the-prompt.md` | writing an intent prompt that composes cleanly first try | +| `references/cli.md` | running this locally with `fp` — commands, and the gotchas that make one lie | +| `references/dashboard.md` | running this as the dashboard assistant — which tool answers what | + +The method is one text; only the grounding mechanics differ, so each surface +carries the reference it can actually use and not the other. Working from a +machine with `fp`, you have `cli.md` and no `dashboard.md`. Working as the +dashboard assistant, the reverse — there is no shell there, so a command palette +would only be something to recite at the user. **Whichever one you have is the +one for you; the missing file is not an error and not worth mentioning.** diff --git a/skills/failproofai-eval-brainstorm/references/cli.md b/skills/failproofai-eval-brainstorm/references/cli.md new file mode 100644 index 0000000..9800eac --- /dev/null +++ b/skills/failproofai-eval-brainstorm/references/cli.md @@ -0,0 +1,115 @@ +# Running this locally with `fp` + +For a coding agent on the user's machine, with the `fp` CLI pointed at their +organisation. Needs `events:read`; the `evals` command also needs `evaluations:read`. + +## The palette + +| To learn… | Run | +|---|---| +| what exists | `fp --json list agents` · `list envs` · `list error_types` · `list score_filters` | +| where it hurts | `fp --json errors --aggregate --since 7d` | +| which runs failed | `fp --json sessions --status error,timeout --since 7d --all --limit 1000` | +| what is already scored | `fp --json evals --aggregate --since 7d` → `score_stats` | +| which scores are low | `fp --json evals --score :..0.5 --since 7d --all --limit 200` | +| a session's shape | `fp --json events --session-id --order asc --all --limit 1000` | +| a session's content | the same, plus `--full` | +| anything the flags cannot express | `fp --json query run --sql "…"` — check `fp --json query schema` for columns first | + +SQL runs over `events`, `evaluations` and `agent_sessions`. Two queries carry most of +the method: + +```bash +# event-type histogram across the population +fp --json query run --sql \ + "SELECT event_type, count() c FROM events GROUP BY event_type ORDER BY c DESC" + +# the candidate per session, so the good and bad cohorts can be compared +fp --json query run --sql \ + "SELECT session_id, count() c, countIf(event_type='error') errs + FROM events GROUP BY session_id ORDER BY errs DESC LIMIT 50" +``` + +**Reading a payload key in SQL.** The store is ClickHouse and `payload` is a `String` +holding JSON, so Postgres spellings are syntax errors, not empty results: +`payload->>'key'` and `::float` both fail outright. Use the JSON functions, and note +that the column is `event_type` (not `type`) and the timestamp is `ts`: + +```bash +fp --json query run --sql \ + "SELECT agent_id, + count() ends, + round(avg(JSONExtractBool(payload,'resolved')), 3) pct_resolved, + round(avg(JSONHas(payload,'sentiment_score')), 3) sentiment_present + FROM events WHERE event_type='agent_end' GROUP BY agent_id ORDER BY ends DESC" +``` + +`JSONExtractString` / `Float` / `Bool` read a value; `JSONHas` reads presence. That last +column is the useful trick: it answers Gate A and Gate B in one query, putting the rate a +key is present at next to the number it produces. + +## Gate A without a payload profile + +The dashboard assistant has a profile of the organisation's payload keys. **The CLI has +no equivalent command**, so confirming a field exists is on you — and skipping it is how +you propose a measurement over a key nobody emits. + +Derive it from the data instead. Over one session: + +```bash +fp --json events --full --session-id --all --limit 1000 \ + | jq '[.events[] | select(.event_type=="tool_use") | .payload | keys[]] | unique' +``` + +Across the population, sample several sessions and intersect — a key present in one run +and absent from the next five is not a foundation. Say the presence rate you actually +observed, and say how you observed it; "seen in 6 of 8 sessions I checked" is an honest +claim, "present" is not. + +## The gotchas that make a command lie + +- **Globals go before the command.** `fp --json events …`, never `fp events --json` — + the latter exits 2. +- **`--all` is capped by `--limit`, which defaults to 50.** A bare `--all` returns 50 + rows and looks complete. Always `--all --limit 1000`; past that, page with `--cursor`. +- **`--since` is a closed set** — `all`, `15m`, `1h`, `6h`, `24h`, `7d`. Anything else is + a usage error; use `--from` / `--to` with RFC3339 for a custom range. +- **`evals` and `errors` filters are single-valued** — a repeated flag means last-wins. + `sessions` and `events` take CSV and repeats. +- **Keep `--full` bound to one `--session-id`.** It is slow at scale, and you do not need + payloads to read a session's shape. + +## Building the authoring link + +There is no `build_eval_authoring_link` here — that tool belongs to the dashboard +assistant. Build the link yourself; it is three pieces, and `fp` has two of them: + +```bash +ORG=$(fp --json whoami | jq -r '.active_org // empty') +BASE=${FP_DASHBOARD_URL:-https://app.befailproof.ai} +PROMPT='Fraction of tool calls … Use the evaluation key `tool_retry_rate`.' + +[ -n "$ORG" ] && echo "$BASE/$ORG/eval-authoring/new?intent=$(jq -rn --arg p "$PROMPT" '$p|@uri')" +``` + +**If `active_org` comes back empty, do not build the link.** Under an API key with no +`--org`, `whoami` reports `null` — and an instance-scoped key then resolves server-side +to the *default* org, so a link built from a guess would open authoring against the wrong +tenant. Hand over the prompt instead and say why: *"pass `--org ` and I will build +the link."* + +`@uri` matters: the prompt carries backticks, quotes and dashes, and the page decodes +exactly what you encode. One link per proposal, same rules as the method — build it, do +not offer to. + +## The local deliverable + +Same slate, written down. Offer to save it as `eval-plan.md` — **ask first, and ask +where**: the file can carry paraphrased customer data, and this may be running before any +repo for it exists. Cite session ids and paraphrase what you saw; never paste raw +transcript into a file. + +Each proposal still ends in the prompt and the link built above, which opens the +dashboard's eval authoring page to compose, backtest and deploy. The link is the handoff; +the prompt beside it is what the user can read before they click, and what they paste +if the link could not be built. diff --git a/skills/failproofai-eval-brainstorm/references/patterns.md b/skills/failproofai-eval-brainstorm/references/patterns.md new file mode 100644 index 0000000..d4ebc98 --- /dev/null +++ b/skills/failproofai-eval-brainstorm/references/patterns.md @@ -0,0 +1,88 @@ +# Candidate patterns + +Worked mappings from what you *saw* to what you can *measure*. Use them to name a +candidate fast, then put it through both gates — a pattern from this table is still +a guess until its fields are confirmed present and its gap is computed. + +## Observation → candidate + +| What you saw in the sessions | Candidate measurement | +|---|---| +| bad runs repeat the same call four to six times | repeat rate — calls identical to the previous call, over total calls | +| bad runs never reach an end event | completion — did the run finish | +| a tool is called and never once succeeds | per-tool success rate, scoped to that tool | +| the agent calls a tool with arguments its schema forbids | invalid-argument rate (points at a wrong tool description) | +| some runs burn many times the work for the same goal | volume — calls, model round-trips, or events per run | +| a human had to step in | intervention count, or the wait time before they did | +| the run ends by telling the user to contact someone else | deflection rate | +| a sub-agent is spawned and its result never comes back | orphaned-handoff rate | +| the run says it did something its own results contradict | **needs a judge** — see below | +| errors cluster in one environment or one build | any of the above, scoped by a condition | + +## Picking the result kind + +Let the phrasing decide it. These map one-to-one, and getting it right first time is +most of what makes a prompt compose cleanly: + +| The measurement is… | Kind | Phrase it as | +|---|---|---| +| a fraction, a rate, a ratio — bounded 0 to 1 | **score** | "Fraction of…", "Ratio of… to…" | +| a count or a quantity with a unit — unbounded | **metric** | "How many…", "Number of…", "Total… in the session" | +| a yes/no fact about the run | **assertion** | "Did the agent…?", "Did every… get a…?" | + +Default to **score** for anything you want to watch a trend on. A rate is comparable +across sessions of different sizes; a raw count is not, and a count that rises because +runs got longer looks like a regression that is not there. + +## Rules before judges + +Two ways to compute anything here: + +- a **rule** — counting, ratios, thresholds over the event stream. Deterministic, free, + instant, and it explains itself. +- a **judge** — a model reads the conversation text and decides. Subjective, costs + latency and money, and needs its own prompt kept honest over time. + +**Default to the rule.** Reach for a judge only when the signal genuinely lives in free +text and no count approximates it — "was the final answer correct", "did it follow the +policy it was given". When you do propose one, say so in the proposal: it changes the +cost and the review the user is signing up for. + +Many things that *feel* like judge territory have a rule hiding in them. "Did it give +up?" is usually "did the run end without reaching an end event". "Was it confused?" is +often "how many times did it call the same tool with the same arguments". Look for the +count before reaching for the model. + +## Generic versus theirs + +There is a floor of measurements that work on any agent because they read the core +event stream rather than anyone's custom payload: error rate, completion, tool success, +calls per run, round-trips per run, hook allow/deny, human waits. + +**These are safe and they are also undifferentiated.** They are worth proposing when +the user has nothing at all — something measured beats nothing — but they are not why +anyone needs a brainstorm. The proposals that earn the session are the ones reading the +keys only *this* user's agents emit: their order id, their tenant, their retrieval +score, their stage name, their model version. + +So spend the payload profile well. Scan it for keys that look like a **status**, a +**stage**, a **verdict**, a **score**, a **version**, a **channel**, or a **cost** — those +are where the user's own semantics live, and an evaluation over one of them measures +something nobody else could have proposed. + +## Conditions — when a measurement should not run everywhere + +An evaluation can carry a condition that decides which sessions it runs on at all. +Reach for one when: + +- the measurement only makes sense for one agent, one environment, or one build; +- the population is mixed and an average over all of it means nothing (a retrieval + score across sessions that never retrieve anything); +- you are scoping to a stage — only sessions that got as far as checkout. + +Two cautions. A condition that excludes everything produces an evaluation that never +runs and reports nothing — it looks deployed and is inert. And a condition reading a key +that is not present excludes every session for that reason, which is the same failure +wearing a different hat. Both are caught by the authoring page's backtest, but proposing +them wastes the user's round trip, so check the condition's fields in the payload profile +exactly as you check the measurement's. diff --git a/skills/failproofai-eval-brainstorm/references/writing-the-prompt.md b/skills/failproofai-eval-brainstorm/references/writing-the-prompt.md new file mode 100644 index 0000000..516bf04 --- /dev/null +++ b/skills/failproofai-eval-brainstorm/references/writing-the-prompt.md @@ -0,0 +1,88 @@ +# Writing the prompt + +The prompt is the deliverable. Everything before it was research; this is the thing the +user actually carries away, and its quality decides whether their first attempt at +authoring works or wastes a round trip. + +**Write it to stand alone.** The authoring page sees your prompt and a profile of the +organisation's event payloads. It does not see this conversation, the sessions you read, +or the reasoning that got you here. Anything load-bearing has to be *in the prompt*. + +## Four properties of a prompt that composes first try + +**1. One measurement, phrased so the kind is obvious.** "Fraction of…" gets a score, +"How many…" a metric, "Did the agent…?" an assertion. A prompt asking for two things at +once gets a muddle of both. + +**2. Name the fields outright.** Say which event types and payload keys to read, in the +exact spelling the profile showed. This is the single highest-value sentence in the +prompt — without it every field access is a guess, and a guessed key reads nothing, +scores every session the same, and looks like it works. + +**3. Anchor the ends.** Say what the lowest value means and what the highest means. +"0 when every call is distinct, 1 when every call after the first is a repeat" removes +the ambiguity that otherwise gets resolved by coin flip — and it is what makes the +number readable on a dashboard six weeks later. + +**4. Carry the scope in the prose.** There is no separate scope field. If the +measurement is only meaningful for one agent, one environment, or one stage, write that +into the sentence — "only for sessions from the `checkout-bot` agent" — and it becomes +the evaluation's condition. + +**5. State the key you agreed.** There is no separate key field either, and the link +carries nothing but this prose. Leave the key out and the page names the evaluation +itself — reasonably, but not what the user just signed off on in step 7, and keys are +permanent in practice. One clause fixes it: + +> Use the evaluation key `tool_retry_rate`. + +The same goes for the result kind, but by omission rather than statement: the page +infers score / metric / assertion from property 1's phrasing, so get the phrasing right +and do not try to declare the kind outright. + +## Specific beats broad + +Broad prompts make the model waffle, produce vague code, and can time out outright. +"Cost and general efficiency stuff" is not a measurement. "Total model requests in the +session" is. + +If a proposal genuinely covers two things, it is two proposals — and if that pushes you +past four, one of them was not worth it. + +## Before and after + +| Weak | Why | Strong | +|---|---|---| +| "Measure tool efficiency" | names no quantity, no direction, no field | "Fraction of a session's tool calls that repeat the previous call's tool name and input exactly. Read `tool_use.tool_name` and `tool_use.input`. 0 when every call is distinct, 1 when every call after the first is a repeat." | +| "Is the agent doing a good job?" | not measurable; no source | "Did the session reach an `agent_end` event with a successful outcome?" | +| "Track retrieval quality" | plausible, but is the field even there? | "Average of `retrieval.score` across the session's `retrieval` events, only for sessions that have at least one. 0 is no match, 1 is an exact match." | +| "Count errors and latency" | two measurements | split: "Number of `error` events in the session" · "Total wall-clock seconds from first to last event" | + +## What happens next, and what it tells you + +The authoring page composes the prompt into an evaluation, then backtests it against +real sessions before anything is deployed. What it reports back is feedback on the +*prompt*, and each outcome points at a specific fix: + +| The page says | What it means about the prompt | Fix | +|---|---|---| +| it ran clean, scores vary | the prompt worked | nothing — review and deploy | +| it reads keys no session has | you named a field that is not in their data | back to the payload profile; you skipped gate A | +| its result cannot depend on the session | you described a constant, not a property of the run | rephrase around something that varies per session | +| the condition skipped every session | your scope sentence excluded everything | widen the scope, or check the condition's own fields exist | +| it raised an error | usually ambiguity that produced bad code | cut it to one measurement, name the fields, try again | +| every session scored the same | **usually fine** | leave it — see below | + +That last row is the one people get wrong. An evaluation asked for "fraction of tool +calls that returned an error" scoring 0.00 on healthy sessions is returning the correct +answer. Low variance is not a defect, and rewriting a working measurement until the +number moves is how a good evaluation gets turned into a bad one. + +## Names are permanent + +The key is what every trend, filter and dashboard is built on, and renaming it splits +the history. Get the user to say the exact string out loud before they author it. + +Short, lowercase, underscore-separated, and naming the thing measured rather than the +verdict: `tool_retry_loop`, `completion_rate`, `retrieval_score`. Not `quality`, not +`eval_1`, not anything carrying this week's number in it. diff --git a/skills/failproofai-policy-author/SKILL.md b/skills/failproofai-policy-author/SKILL.md index 533026e..35837db 100644 --- a/skills/failproofai-policy-author/SKILL.md +++ b/skills/failproofai-policy-author/SKILL.md @@ -12,7 +12,7 @@ description: |- Served by the `failproofai` CLI. - NOT for publishing a finished policy pack to GitHub (`failproofai-policy-publish`); nor Cloud-managed policy versions, fleet rollout, telemetry, or org operations (`fp-cloud-cli`), evaluator scoring (`agenteye-evaluator`), or repo invariants that belong in tests. + NOT for publishing a finished policy pack to GitHub (`failproofai-policy-publish`); nor Cloud-managed policy versions, fleet rollout, telemetry, or org operations (`fp-cloud-cli`), what to evaluate (`failproofai-eval-brainstorm`), or repo invariants that belong in tests. --- # failproofai Policies diff --git a/skills/failproofai-sdk/SKILL.md b/skills/failproofai-sdk/SKILL.md index bc1cd52..adb6657 100644 --- a/skills/failproofai-sdk/SKILL.md +++ b/skills/failproofai-sdk/SKILL.md @@ -10,7 +10,7 @@ description: |- Served by the `failproofai_sdk` Python SDK, inside the user's own agent. - NOT for reading telemetry that already landed or operating a deployment (that's `fp-cloud-cli`), or building the evaluator service that scores runs (that's `agenteye-evaluator`). + NOT for reading telemetry that already landed or operating a deployment (that's `fp-cloud-cli`), or deciding what is worth evaluating (that's `failproofai-eval-brainstorm`). --- # Failproof AI Python SDK diff --git a/skills/failproofai/SKILL.md b/skills/failproofai/SKILL.md index fe903d7..16c0016 100644 --- a/skills/failproofai/SKILL.md +++ b/skills/failproofai/SKILL.md @@ -10,7 +10,7 @@ description: |- • find the right surface — audits, sessions, policies, keys and orgs, fleet deploys, self-hosting; • fix a live machine — a stopped daemon, a session that never lands, a dead hook. - It can stand alone. When focused sibling skills are installed, route policy authoring to `failproofai-policy-author`, pack publishing to `failproofai-policy-publish`, Cloud and fleet work to `fp-cloud-cli`, scoring to `agenteye-evaluator`, and instrumentation to `failproofai-sdk`. + It can stand alone. When focused sibling skills are installed, route policy authoring to `failproofai-policy-author`, pack publishing to `failproofai-policy-publish`, Cloud and fleet work to `fp-cloud-cli`, eval planning to `failproofai-eval-brainstorm`, and instrumentation to `failproofai-sdk`. --- # FailproofAI @@ -66,7 +66,7 @@ reach, and an agent that meets one of these needs to know it is the same product | `AGENTEYE_HOME`, `~/.agenteye/events` | the **local daemon's** legacy SDK spool, which it still watches | | `AGENTEYE_KEY` (collector ingest), `AGENTEYE_API_KEY` (dashboard admin) | ingest credentials. `FP_API_KEY` was named deliberately *not* to collide — never tell anyone to reuse either | | `ghcr.io/agenteye-enterprise`, k8s namespace `agenteye`, ClickHouse `agenteye.events` | self-hosted infrastructure | -| dist `agenteye-evaluator`, module `agenteye_evaluator`, UA `agenteye-server/` | the evaluator package — and the one sibling skill that keeps its name | +| dist `agenteye-evaluator`, module `agenteye_evaluator`, UA `agenteye-server/` | the RETIRED v1 evaluator package. Scoring is Evaluator v2 now: author a hosted evaluation in the dashboard, or run a worker on `failproofai-sdk` | | `incidents:read`/`:write`/`:ack`, `alerts:ack`, the `INCIDENT_ID` positional on `issues show` | retired grants and arguments the server still parses | **The env prefix follows the binary.** `fp` reads `FP_HOME`, `FP_JSON`, `FP_TOKEN`, @@ -96,10 +96,12 @@ fork wrong wastes the most time of anything in this product. ## Route first -Read this before doing any work. Three of the specialists — `fp-cloud-cli`, `failproofai-sdk` -and `agenteye-evaluator` — are mirrors, synced from a private repo and marked do-not-hand-edit; -duplicating or patching them here is a maintenance bug. Two of the three were renamed with the -product; the evaluator was not, and `agenteye-evaluator` is its real current name. +Read this before doing any work. Two of the specialists — `fp-cloud-cli` and `failproofai-sdk` +— are mirrors, synced from a private repo and marked do-not-hand-edit; duplicating or patching +them here is a maintenance bug. `failproofai-eval-brainstorm` is a mirror too, and answers the +question the retired `agenteye-evaluator` skill used to: **what is worth measuring**. It stops at +the proposal — the dashboard's eval authoring page composes, backtests and deploys the +evaluation itself, so nothing here scaffolds an evaluator service any more. | The request is | Go to | |---|---| @@ -107,7 +109,7 @@ product; the evaluator was not, and `agenteye-evaluator` is its real current nam | Publish tested policies as a GitHub pack others can install | **`failproofai-policy-publish`** | | Publish a Cloud policy version, deploy it to fleet machines, observe, enforce, or roll back | **`fp-cloud-cli`** | | Query FailproofAI Cloud — browse sessions, events, errors, evals; triage issues/alerts; manage keys, users, roles, settings | **`fp-cloud-cli`** (the cloud CLI skill) | -| Decide what to score, or build/extend an evaluator service | **`agenteye-evaluator`** | +| Decide what to score — which dimensions are worth tracking, grounded in real sessions | **`failproofai-eval-brainstorm`** | | Instrument an agent that is **not** one of the 12 supported CLIs — a Python/LangChain/custom loop | **`failproofai-sdk`** | | Every surface at once — product architecture, commands, terminology, setup, and skill selection | **stay here** | | Anything local-machine: install, connect, daemon, backfill, flush, capture paths, upgrade, uninstall | **stay here** | diff --git a/skills/failproofai/references/literals.md b/skills/failproofai/references/literals.md index 3080c3c..d2ff917 100644 --- a/skills/failproofai/references/literals.md +++ b/skills/failproofai/references/literals.md @@ -61,7 +61,7 @@ one for the other, in either direction. | dist `agenteye-evaluator` | the package index name. `pip install` fails | | module `agenteye_evaluator` | every `import` in every evaluator service anyone has written | | user-agent `agenteye-server/` | how the evaluator identifies itself to the service it calls | -| the skill `agenteye-evaluator` | see *Skill names* below — this is the one sibling that keeps its old name | +| the skill `agenteye-evaluator` | **retired.** Deciding what to measure is now `failproofai-eval-brainstorm`; see *Skill names* below | The evaluator is the clean case: it was not renamed, so there is nothing here to modernise and no migration to describe. Cross-reference it by its real name. @@ -117,14 +117,14 @@ the wrong half installs nothing. |---|---|---| | `fp-cloud-cli` | `agenteye-cli` | mirror, synced from upstream | | `failproofai-sdk` | `agenteye-python-sdk` | mirror. The module genuinely renamed to `failproofai_sdk` | -| `agenteye-evaluator` | — | **not renamed upstream.** Keep this name when cross-referencing it | +| `failproofai-eval-brainstorm` | `agenteye-evaluator` | **retired, not renamed** — the v1 evaluator service it built no longer exists. The distribution `agenteye-evaluator` and module `agenteye_evaluator` keep their names | | `failproofai` | — | maintained in this repo | | `failproofai-policy-author` | — | maintained in this repo | | `failproofai-policy-publish` | — | new | | `failproofai` | — | the complete umbrella skill | **The three mirrors are marked "do not hand-edit."** Never edit files under -`skills/fp-cloud-cli/`, `skills/failproofai-sdk/` or `skills/agenteye-evaluator/`. If one +`skills/fp-cloud-cli/`, `skills/failproofai-sdk/` or `skills/failproofai-eval-brainstorm/`. If one of them contradicts this page, the fix belongs upstream — and the directory name on disk may lag the shipped skill name, which is not a discrepancy to correct locally. diff --git a/skills/failproofai/references/product-verticals.md b/skills/failproofai/references/product-verticals.md index 0df6ebc..8b593c1 100644 --- a/skills/failproofai/references/product-verticals.md +++ b/skills/failproofai/references/product-verticals.md @@ -13,7 +13,7 @@ a machine that was never meant to have it. |---|---|---|---| | Observe | hook activity + transcripts on disk, dashboard at `127.0.0.1:8020` | `events` `sessions` `errors` | `fp-cloud-cli` | | Enforce | packs you install + custom/convention policies, hooks in 12 harnesses | `policies` `fleet` `guardrails`, the backtest in the dashboard | `failproofai-policy-author`, `failproofai-policy-publish`, `fp-cloud-cli` | -| Evaluate | — nothing | `evals` + an evaluator service you host | `agenteye-evaluator` | +| Evaluate | — nothing | `evals` + hosted evaluations authored in the dashboard | `failproofai-eval-brainstorm` | | Audit | `failproofai audit`, offline, no account | `audits` → findings → `issues` | `failproofai`, `fp-cloud-cli` | | Manage | — nothing | `orgs` `keys` `users` `query` `alerts` `settings` `usage` | `fp-cloud-cli` | @@ -274,11 +274,15 @@ list and aggregate honour every filter. The hard part is deciding what to score, and only the user knows that. The SDK part is small. -Route: **`agenteye-evaluator`** — designing the dimensions, scaffolding the service, rules vs -LLM judge, testing against a real captured session, deploying it and confirming scores land. -That skill keeps its `agenteye` name because the package genuinely was not renamed: dist -`agenteye-evaluator`, module `agenteye_evaluator`, user-agent `agenteye-server/`. Do -not "correct" it. +Route: **`failproofai-eval-brainstorm`** — scanning the population, confirming the signal is +really in the telemetry, checking it separates good runs from bad, and converging on two to +four proposals, each ending in the prompt that authors it. It stops there: the dashboard's +eval authoring page composes the evaluation, backtests it against real sessions and deploys it. + +It replaces the retired `agenteye-evaluator` skill, which also scaffolded the v1 evaluator +service the server POSTed transcripts to — that service no longer exists. The *package* names +were never renamed and are still correct where they appear: dist `agenteye-evaluator`, module +`agenteye_evaluator`, user-agent `agenteye-server/`. --- diff --git a/skills/failproofai/references/sessions.md b/skills/failproofai/references/sessions.md index dd1fef5..bcd4a49 100644 --- a/skills/failproofai/references/sessions.md +++ b/skills/failproofai/references/sessions.md @@ -7,7 +7,7 @@ dashboard-only, which `fp` reaches, and which only the HTTP API reaches.** Those sets differ, and none contains the others. Anchors are into the docs tree (`docs/sessions/*.mdx`, `docs/reference/*.mdx`, -`docs/reference/openapi.json`) and the shipped `fp-cloud-cli` / `agenteye-evaluator` skills. +`docs/reference/openapi.json`) and the shipped `fp-cloud-cli` / `failproofai-eval-brainstorm` skills. The rename has landed: this file writes **`fp`** (`uv tool install fp-cloud-cli`), and that is what to resolve first — `command -v fp agenteye`. `agenteye` is the legacy fallback, a separate package that is still installable; the read commands below behave the same there, but diff --git a/skills/failproofai/references/skill-directory.md b/skills/failproofai/references/skill-directory.md index 8c6a247..42fb167 100644 --- a/skills/failproofai/references/skill-directory.md +++ b/skills/failproofai/references/skill-directory.md @@ -9,7 +9,7 @@ on **what the user is trying to change**, not on which nouns they said. ├── failproofai-policy-publish ─► publish a GitHub pack others can install ├── fp-cloud-cli ───────────────► query and administer FailproofAI Cloud ├── failproofai-sdk ────────────► instrument an agent that is not one of the 12 CLIs - └── agenteye-evaluator ─────────► decide what to score, build the scoring service + └── failproofai-eval-brainstorm ► decide what is worth scoring, grounded in real sessions ## Install any of them @@ -25,16 +25,21 @@ with that command. ## Three of these are mirrors `fp-cloud-cli` and `failproofai-sdk` are synced from `FailproofAI/failproofai`; -`agenteye-evaluator` is synced from the private `FailproofAI/agenteye` repository. All three +`failproofai-eval-brainstorm` is synced from the private `FailproofAI/agenteye` repository. All three are marked **do-not-hand-edit.** Patching them here is a maintenance bug: the next sync silently reverts your change, and in the meantime two copies of the same claim disagree. Fix upstream, or carry the correction in a skill that is maintained here — `failproofai`, `failproofai-policy-author`, and `failproofai-policy-publish`. -Two of the three were renamed with the product; the evaluator was not. **`agenteye-evaluator` -is its real current name** — do not "fix" it. The renamed mirror folders now match their -shipped skill names: `skills/fp-cloud-cli/` and `skills/failproofai-sdk/`. +The mirror folders match their shipped skill names: `skills/fp-cloud-cli/`, +`skills/failproofai-sdk/` and `skills/failproofai-eval-brainstorm/`. + +`agenteye-evaluator` was **retired, not renamed.** It taught you to build a v1 "server-push" +evaluator service, and that service no longer exists — scoring is Evaluator v2, either a +hosted evaluation authored in the dashboard or a worker built on `failproofai-sdk`. The half +of it worth keeping was never the code: it was deciding what to measure, which is now +`failproofai-eval-brainstorm`. ## The six @@ -98,7 +103,7 @@ describe; when another skill says "hand off to the cloud CLI", this is the desti | | | |---|---| | **Owns** | making an agent that is **not** one of the 12 supported CLIs report what it did: planning which points in the loop to record, threading session and agent identity, emitting tool/model/hook/human events, and proving the `.jsonl` files land | -| **Refuses** | reading telemetry that already arrived or operating a deployment (`fp-cloud-cli`), and building the evaluator that scores runs (`agenteye-evaluator`) | +| **Refuses** | reading telemetry that already arrived or operating a deployment (`fp-cloud-cli`), and deciding what is worth evaluating (`failproofai-eval-brainstorm`) | | **Route to it when** | the agent is a Python loop, a LangChain/LangGraph/CrewAI/LlamaIndex/Pydantic AI app, or anything custom — there are no hooks to install because there is no harness | | **Install** | `npx skills add FailproofAI/skills --skill failproofai-sdk -a claude-code` | | **Maintained** | **mirror — do not hand-edit** | @@ -107,18 +112,23 @@ Was `agenteye-python-sdk`, and unlike the wire literals **the module genuinely r `failproofai_sdk`. The SDK's job ends at the file it writes; a separate collector ships it, which is why "my events never appear" splits between this skill and `failproofai`. -### `agenteye-evaluator` — decide what to score, build the scorer +### `failproofai-eval-brainstorm` — decide what is worth scoring | | | |---|---| -| **Owns** | both halves of evaluation-you-own: choosing 2–4 dimensions worth measuring against real sessions (a plan is a valid end state, with no code), and building the HTTP service the server POSTs finished transcripts to | -| **Refuses** | reading eval results that already exist or checking whether quality dropped (`fp evals`, via `fp-cloud-cli`), instrumenting an agent, and alerting on scores | -| **Route to it when** | the user says "I want evals" or "how do I know if my agent is any good?" | -| **Install** | `npx skills add FailproofAI/skills --skill agenteye-evaluator -a claude-code` | +| **Owns** | working out WHAT to measure from the sessions an agent actually produced: scan the population, confirm the signal is really in the telemetry, check it separates good runs from bad, converge on two to four proposals, and write the prompt that authors each one | +| **Refuses** | authoring or deploying the evaluation itself (the dashboard's eval authoring page composes, backtests and deploys it from that prompt), reading eval results that already exist or checking whether quality dropped (`fp evals`, via `fp-cloud-cli`), instrumenting an agent (`failproofai-sdk`), and alerting on scores | +| **Route to it when** | the user says "I want evals", "what should I be measuring?", or "how do I know if my agent is any good?" | +| **Install** | `npx skills add FailproofAI/skills --skill failproofai-eval-brainstorm -a claude-code` | | **Maintained** | **mirror — do not hand-edit** | -The one skill that keeps the old name, because the package did: distribution -`agenteye-evaluator`, module `agenteye_evaluator`, user-agent `agenteye-server/`. +Replaces the retired `agenteye-evaluator`, which also scaffolded the v1 evaluator service the +server POSTed transcripts to. That service is gone; the deciding half is not, and it is the +half only someone looking at real sessions can do. + +The trap it exists to prevent: `event.payload` is free-form, so an evaluation reading a key +nobody emits **does not fail**. It reads nothing on every session, scores them identically, +and looks like it is working. ## Naming, so cross-references resolve diff --git a/skills/fp-cloud-cli/SKILL.md b/skills/fp-cloud-cli/SKILL.md index 106414a..ef9d1c3 100644 --- a/skills/fp-cloud-cli/SKILL.md +++ b/skills/fp-cloud-cli/SKILL.md @@ -10,7 +10,7 @@ description: |- Served by the `fp` CLI against FailproofAI Cloud. - NOT for publishing reusable GitHub policy packs (`failproofai-policy-publish`), evaluator scoring (`agenteye-evaluator`), instrumenting an app (`failproofai-sdk`), or debugging the local collector/daemon. + NOT for publishing reusable GitHub policy packs (`failproofai-policy-publish`), deciding what to evaluate (`failproofai-eval-brainstorm`), instrumenting an app (`failproofai-sdk`), or debugging the local collector/daemon. --- # FailproofAI Cloud CLI