feat(skills): measure a skill reached from another skill

.claude/skills/test-servers/evals/evals.json

@@ -19,6 +19,20 @@
     "prompt": "I need a fixture combination that doesn't exist yet. How do I add one?",
     "expect": "test-servers"
   },
+  {
+    "prompt": "Write an integration test that exercises tool listing end to end.",
+    "chain": [
+      "testing",
+      "test-servers"
+    ]
+  },
+  {
+    "prompt": "Add end-to-end coverage for the tool-list pagination path.",
+    "chain": [
+      "testing",
+      "test-servers"
+    ]
+  },
   {
     "prompt": "Sort this list alphabetically: banana, apple, cherry.",
     "expect": null

AGENTS.md

@@ -262,6 +262,20 @@ that from happening:
    ("how does the `@inspector/core` alias resolve?") invites a `Read`, which is
    a *better* answer than a skill. Good cases are "how do I / where does this go"
    questions whose answer is a procedure.
+   **A pointer from one skill's body to another is measured by a `chain` case,
+   not an `expect` one.** A first-move case can only observe the model's opening
+   tool call, so a skill reached only *through* another scores a clean 100% on
+   its direct cases while the hand-off silently never fires (#2204). A chained
+   case names the ordered skills one run should load, **ending with the skill
+   whose file it lives in** — so the file that goes red is the one belonging to
+   the skill that stopped being reached. It runs on a wider turn budget, is
+   scored against its own `CHAIN_THRESHOLD`, and is **reported in its own
+   column**: a hand-off rate and a first-move rate are not comparable, and
+   folding them together would move a headline everyone reads as trigger
+   reliability. It counts toward neither the five-positive floor nor the
+   negative requirement, and it is only worth writing where the first link's
+   body actually points at the target — a chain through a skill that says
+   nothing about it is a permanent 0% with no lever.
    ⚠️ **The gate cannot catch a description that never matches.** `verify:skills`
    checks that a skill is well-formed and that its cases exist; only
    `skills:eval` observes whether it actually fires, and that cannot be gated —

docs/skill-authoring.md

@@ -14,6 +14,17 @@ reachability — `npm run skills:eval` — reports a hit rate rather than a verd
 session** (`claude -p`), `RUNS` times, and scores the fraction of runs in which
 the `Skill` tool fired with the expected name.
 
+There are **two kinds of case**, measured against different turn budgets and
+reported in separate columns:
+
+| | asserts | budget | column |
+| --- | --- | --- | --- |
+| `"expect": "<skill>"` / `null` | the skill is (or is not) the model's **first move** | 1 turn | first-move |
+| `"chain": ["a", …, "<skill>"]` | loading `a` **leads to** loading this skill | `CHAIN_MAX_TURNS` (14) | hand-off, `CHAIN_THRESHOLD` 0.5 |
+
+Almost every case is the first kind, and the four properties below are about
+that kind. The hand-off case has its own section further down.
+
 Four properties of that harness drive everything below:
 
 - **`--max-turns 1`.** The skill must fire in the model's **first assistant
@@ -180,6 +191,125 @@ unrelated to the repo (arithmetic, trivia, a one-line refactor). All 18 in this
 repo have held at 100% through every reshaping so far — if one starts firing, a
 description has grown too broad.
 
+### Chained cases: measuring a hand-off
+
+A skill body may point at another skill — `testing` opens by telling the model
+that picking a fixture is `/test-servers` and that it has to load it, and
+because `test-servers` is model-invocable that pointer is live rather than a
+dead end. **Nothing in a first-move case can observe whether that pointer is
+ever taken.** `test-servers` scores 5/5 on its own cases and every one of them
+asks for it by name; a skill only ever reached _through_ another would score a
+clean 100% while the hand-off silently never fired (#2204).
+
+A chained case names the ordered skills one run should load:
+
+```json
+{
+  "prompt": "Write an integration test that exercises tool listing against a real server.",
+  "chain": ["testing", "test-servers"]
+}
+```
+
+**Write a chained case when the prompt names nothing about the target skill and
+the path to it runs through another skill.** Write an ordinary first-move case
+for everything else — a prompt someone would actually type to reach this skill
+directly is a first-move case even when a hand-off could also get there, and it
+is the cheaper measurement by an order of magnitude.
+
+Six rules the shape enforces, each for a reason worth knowing:
+
+- **The chain ends with the skill whose file it lives in.** The case exists to
+  measure whether _this_ skill is reachable, so the file that must go red when
+  the hand-off stops working is the one belonging to the skill that stops being
+  reached. Anchoring on the first link would file the `testing → test-servers`
+  measurement under `testing`, where a `test-servers` description edit would
+  never be seen.
+- **A chained case satisfies neither floor.** It is not one of the five
+  positives and it is not the negative. It measures a different thing, so
+  letting it stand in would let a skill ship with no measurement of the way
+  users actually reach it.
+- **The match is an ordered _subsequence_, not a prefix and not a contiguous
+  run.** The model may load something before the chain starts and something
+  unrelated in between; neither changes the claim that A led to B. What does not
+  score is the reverse order.
+- **Every link after the first must land in a later assistant turn.** Position
+  in the stream is not causation: the model can emit several `tool_use` blocks
+  in one message, and it has not seen the first skill's body when it does — so
+  two `Skill` calls in the same turn are parallel guesses, not a hand-off, and
+  a flat index would score them as one (Copilot). This is the difference
+  between "B was loaded after A" and "A led to B", and it is the second way a
+  chained case can false-pass — the first being a prompt that carries the
+  target's own trigger, below. Only the chain's *first* link is unconstrained.
+- **Repeats and unknown links are rejected.** A repeated link cannot be
+  observed, and a link naming a skill the model cannot invoke can never fire —
+  it would score a permanent 0% that reads as a description problem.
+- **The two numbers never share a column.** A hand-off rate is a second-hop load
+  over many turns; a first-move rate is the model's opening move. Summing them
+  would produce a figure describing neither, and a handful of hand-off cases
+  would quietly move a headline everyone reads as trigger reliability.
+
+⚠️ **The prompt must not carry the TARGET skill's own trigger.** This is the
+subtle way a chained case false-passes. `test-servers` claims the situation "a
+change needs a real server to exercise it", so a prompt saying "…against a real
+server" matches it directly: the model can pick `testing` first and then pick
+`test-servers` from the *original prompt*, in that order, and the case scores a
+hit that would survive deleting the pointer from `testing` entirely (Copilot).
+Both committed cases said "against a real/live server" and were rewritten to
+"end to end" for exactly this reason — and the measured rate **fell from 100%
+and 67% to 33% and 33%**, which is the size of the effect this trap hides.
+**Write the prompt so only the loaded first skill can introduce the second**,
+and sanity-check it by asking whether the case would still pass if the pointer
+were removed.
+
+⚠️ **A chained case only measures a pointer that exists.** `pr-flow` says
+nothing about test fixtures, so a `["pr-flow", "test-servers"]` case measured 0%
+— correctly, and with no lever to fix it short of broadening a description onto
+another skill's ground. Before writing one, confirm the first link's body
+actually points at the target; otherwise the case is a permanent zero that reads
+as a description problem.
+
+⚠️ **A hand-off case is a measurement under the harness's tool policy, not a
+prediction about an unrestricted session.** `--max-turns 1` was doing much of
+the read-only containment on its own; a 14-turn budget removes that, so the deny
+list covers the agentic and network tools too (`Task` in particular, whose
+subagent the flag does not reach). Denying `Bash` also changes the path a run
+can take toward the second skill, since investigating a repo by hand often
+starts there. `Read`/`Glob`/`Grep` remain, which is enough to reach a hand-off.
+
+**A hand-off is far less reliable than a first move, and the threshold says so.**
+`CHAIN_THRESHOLD` defaults to **0.5**, not 0.8 — the weakest claim worth
+asserting is that the pointer is taken more often than not — and it is compared
+**strictly**. "More often than not" is `> 0.5`, and an inclusive compare would
+pass 2/4 whenever `RUNS` is even, reporting a result the criterion does not
+license (Copilot). A strict bound of `1.0` is therefore unreachable and the
+harness rejects it up front rather than failing every case.
+
+At 0.8 a hand-off case would be red no matter how strongly the first skill
+pointed at the second, and the column would stop carrying signal. Read a
+hand-off number as a description-strength measurement, not a verdict — and read
+it at `RUNS=5`, since at `RUNS=3` one sample is worth 33 points.
+
+**The committed cases have measured 33% / 33% on one `RUNS=3` run and 100% /
+33% on another, and at least one of them being red is the intended state rather
+than an oversight.** `skills:eval` is not a gate (see below), and the number is
+the finding: `testing` points at `test-servers` in its first paragraph and the
+model follows that pointer *sometimes*. Strengthening it is its own change
+against its own issue (#2247); lowering the bar to turn the column green would
+throw away the only signal this feature adds.
+
+⚠️ **Do not read a rise between two `RUNS=3` runs as an improvement.** One
+sample is 33 points there, and the two runs above straddle a 67-point swing on
+the same prompt with no change to the pointer. Note in particular that the
+turn-boundary rule added later can only ever *lower* a chained score — it
+rejects matches a flatter reading accepted — so a higher number after it is
+noise by construction, not an effect. `RUNS=5` is the smallest honest setting
+for a hand-off, and the cost is real: each sample is up to 14 turns.
+
+⚠️ **Expect a hand-off to cost far more than a first move.** Each sample is up
+to 14 turns rather than one, so a chained case is the most expensive line in the
+suite by a wide margin — the two above take longer between them than all seven
+first-move cases.
+
 ## The tuning loop
 
 **Probe first, then measure.** A full suite run is ~63 cases × `RUNS` sessions
@@ -194,11 +324,35 @@ prompt fires at all, and only then spend a full run on its rate:
 # snippet also runs under bash.
 printf '%s' "<prompt>" \
   | claude -p --output-format stream-json --verbose --max-turns 1 \
-      --disallowedTools Bash,Write,Edit,NotebookEdit \
+      --tools Read,Glob,Grep,Skill --allowedTools Read,Glob,Grep,Skill \
+      --disallowedTools Bash,Write,Edit,NotebookEdit,Task,Agent,SlashCommand,WebFetch,WebSearch,KillShell \
+      --strict-mcp-config \
   | jq -r 'select(.message.content?) | .message.content[]?
            | select(.type == "tool_use") | .name' | head -3
 ```
 
+⚠️ **`--tools` is the restriction; `--allowedTools` only pre-approves.**
+Dropping the first leaves the bound resting on the deny list alone, so a tool a
+user's or a plugin's settings already permit stays reachable for all 14 turns
+(Copilot). Keep both.
+
+⚠️ **These flags are a copy of the harness's, so they go stale.** Whenever
+`runPrompt` in `scripts/skill-eval.mjs` changes its tool policy, change this
+snippet in the same edit — a probe that may call a tool the eval forbids
+predicts nothing, which is the whole reason the two are meant to match
+(Copilot). To probe a **hand-off** instead, raise `--max-turns` to
+`CHAIN_MAX_TURNS` and drop the `head -3`:
+
+```sh
+printf '%s' "<prompt>" \
+  | claude -p --output-format stream-json --verbose --max-turns 14 \
+      --tools Read,Glob,Grep,Skill --allowedTools Read,Glob,Grep,Skill \
+      --disallowedTools Bash,Write,Edit,NotebookEdit,Task,Agent,SlashCommand,WebFetch,WebSearch,KillShell \
+      --strict-mcp-config \
+  | jq -r 'select(.message.content?) | .message.content[]?
+           | select(.type == "tool_use" and .name == "Skill") | .input.skill'
+```
+
 **Probe a marginal case more than once.** A prompt that fires on a single probe
 can still measure 60% over five runs — one sample cannot distinguish "reliable"
 from "coin flip". Three probes is enough to tell a solid replacement from a
@@ -211,6 +365,14 @@ npm run skills:eval                              # every model-invoked skill, RU
 RUNS=5 CONCURRENCY=6 npm run skills:eval
 npm run skills:eval -- testing                   # one skill's cases
 npm run skills:eval -- testing test-servers      # a set of skills
+CHAIN_THRESHOLD=0.4 CHAIN_MAX_TURNS=20 npm run skills:eval -- test-servers
+```
+
+The summary is two lines, never one:
+
+```
+7/7 first-move cases at or above 80%.
+1/2 hand-off cases above 50%.
 ```
 
 Narrowing the run never narrows what a **negative** case is scored against — a
@@ -257,6 +419,12 @@ break.
    prompts do not steady any rate (`RUNS` is the knob for that). Five prompts
    cover five ways someone might arrive at the skill, which is what catches a
    description that fires on one narrow phrasing and nothing else.
-5. `npm run verify:skills` passes and the listing is under budget.
-6. `RUNS=5 npm run skills:eval` — the **whole** suite — is ≥80% on every case,
-   including the skills you did not touch.
+5. **If this skill is meant to be reachable from another skill's body**, that
+   hand-off has a `chain` case — a pointer between skills is otherwise measured
+   by nothing at all, and a skill reached only that way scores a clean 100% on
+   direct cases while the hand-off never fires. It does not count toward the
+   floor in 4.
+6. `npm run verify:skills` passes and the listing is under budget.
+7. `RUNS=5 npm run skills:eval` — the **whole** suite — is ≥80% on every
+   first-move case, including the skills you did not touch, and the hand-off
+   column is read on its own rather than against that number.