Replace the retired evaluator skill with eval brainstorming

[chhhee10]

Why

agenteye-evaluator taught two things: deciding what is worth measuring, and scaffolding the v1 "server-push" evaluator service the server POSTed finished transcripts to.

That service no longer exists. Scoring is Evaluator v2 — a hosted evaluation authored in the dashboard, or a worker built on failproofai-sdk. So the skill has been walking people through building something they cannot deploy, against a private SDK artifact they cannot obtain, and its upstream source folder was deleted. It is already declared retired in the enterprise docs; this repo was still serving it.

The deciding half did not retire with it. It is the half only someone looking at real sessions can do, and it is the half people actually stall on.

What changes

New: skills/failproofai-eval-brainstorm/

Scan the population → check what is already defined → confirm the signal is really in the telemetry → read 5–8 sessions against one fixed checklist → two gates each carrying a number → converge on 2–4 proposals. Each ends in the plain-English prompt that authors it, and it stops there: composing, backtesting and deploying belong to the dashboard's eval authoring page.

The trap it exists to prevent, and the reason the "confirm the signal" step is not optional:

Event payloads are free-form, so an evaluation reading a key nobody emits does not fail. It reads nothing on every session, scores them all identically, and looks like it is working. Nobody notices for a month.

Two more rules worth naming, because both are ways a good slate goes wrong:

• Low variance is not a defect. "Fraction of tool calls that errored" scoring 0.00 on healthy sessions is the right answer. Rewriting a working measurement until the number moves is how people talk themselves out of their best evaluations.
• The rejected list is a deliverable. Each cut carries the number that killed it, so the user can see why the shortlist is this one.

Removed: skills/agenteye-evaluator/ — with a migration block in the README, the same shape as the agenteye-cli → fp-cloud-cli rename. An existing installation keeps working; it is just no longer served, and what it teaches no longer resolves to anything deployable.

Re-pointed: every sibling that routed to the old name. The three NOT for lines that must partition the space (fp-cloud-cli, failproofai-sdk, failproofai-policy-author), the umbrella skill's routing table, and its literals / skill-directory / sessions / product-verticals references. A stale routing table is worse than a stale doc — an agent follows it to a skill that is not there.

What is deliberately left alone

Mentions of the agenteye-evaluator distribution and the agenteye_evaluator module. Those names were never renamed and are still correct wherever they appear in the legacy-literals tables; only the skill retired. The README's naming table now says exactly that, so the next reader does not "fix" one into the other.

Source of truth

Mirrored from the private FailproofAI/agenteye repo (agent/skills/failproofai-eval-brainstorm/), where the dashboard assistant reads the same folder at runtime — companion PR: FailproofAI/agenteye#826. Marked do-not-hand-edit here, like the other mirrors.

references/dashboard.md exists upstream and is not mirrored: it describes the in-dashboard assistant's tools, and a reader with fp on their own machine has references/cli.md instead. The method is one text; only the grounding mechanics differ, and the SKILL.md says so rather than referencing a file that is not there.

Synced since this PR opened — 9731c49

Opened 2026-09-18; four fixes landed upstream after that, each found by running the skill for real rather than reading it. SKILL.md, cli.md and writing-the-prompt.md are now byte-identical to agenteye (checked, not eyeballed); patterns.md was already current.

• The rejected list is required in the answer, not kept as a note to self — a live run proposed three evaluations and cut nothing visibly, so there was no telling ten tested candidates from three guessed ones.
• Step 8 builds a link for every proposal, in the same reply, instead of offering to. And it no longer names build_eval_authoring_link as the only way: that is a dashboard tool, and from fp it does not exist — while the same step forbade writing "link" without one. That was a contradiction a CLI agent could only resolve by skipping the link or inventing one.
• cli.md shows how to read a payload key in SQL. The store is ClickHouse and payload is a String of JSON, so payload->>'key' is a syntax error, not an empty result. Every worked example read promoted columns only, and Gate B is exactly a payload-key query.
• cli.md builds the link from fp — .active_org from fp --json whoami (both auth modes, read from fp_cli/commands/auth_cmds.py), base from FP_DASHBOARD_URL or app.befailproof.ai, prompt through jq @uri (round-trip checked). It refuses when active_org is null: an instance-scoped key then resolves to the default org, so a guessed link would open the wrong tenant.
• writing-the-prompt.md: state the agreed key in the prose. The link carries the intent and nothing else, so an omitted key is one the page invents.

Checks

python3 scripts/validate-skills.py → 6 skills checked · 0 errors · 0 warnings.

Every description is under the 1024-char truncation limit and under the 1000-char warning line — including failproofai's and failproofai-policy-author's, which my edits pushed up and I trimmed back. That cap matters here more than most: what truncation eats first is the trailing NOT for … scope, so an over-long description does not fail, it misfires.

fp commands and flags in references/cli.md were verified against the current CLI source rather than carried over on trust — the --since enum, query run / query schema, evals --aggregate's score_stats, and the --all stops-at-50 footgun.

🤖 Generated with Claude Code