Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions bundles/rf-agentskills/skills/rf-test-quality/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
---
name: rf-test-quality
description: "Use after changing an RF test, to prove it fails when its code breaks (mutants)."
license: Apache-2.0
compatibility: "Requires Python 3.10+ with robotframework>=7 in the project environment for the bundled mutation_ledger script (stdlib only otherwise)."
metadata:
author: manykarim
version: "1.0.0"
---

# Robot Framework Test Quality Skill

## When to use

Load this skill after you add or change a test, before you call it done. A test that passes against broken code proves nothing.

## The rule

Every new or changed test is seen failing once. Break the thing it checks, run it, and confirm it goes red. Then restore the code and confirm it goes green.

## Mutant checks in a suite

Copy `assets/mutants.resource` from this skill into the project's `resources/` folder, import it, and wrap the real check in `Mutant Must Fail`, feeding it the value the test produced and a mutated expectation:

```robotframework
*** Settings ***
Resource ../resources/mutants.resource

*** Test Cases ***
Status Check Catches A Wrong Code
${total}= Evaluate 2 + 3
Should Be Equal As Integers ${total} 5
${killed}= Mutant Must Fail T1 flip-total
... Should Be Equal As Integers ${total} 6
Should Be True ${killed}
```

Only an assertion failure counts as a kill. A mutant that fails because of a typo, a missing keyword, a missing variable or a wrong argument count is recorded as a survivor: fix the mutant call.

Each call writes one row to `mutation-ledger.jsonl` in the output directory. `Mutation Kill Rate` returns `{killed, total, rate, survivors}`. A rerun of the same test and operator replaces its earlier row, so the rate uses the latest result per mutant. Quote the rate for each changed test. A survivor means the check is too weak: fix the check, not the mutant.

## Reading the ledger

```bash
uv run python "${CLAUDE_SKILL_DIR}/scripts/mutation_ledger.py" rate results/mutation-ledger.jsonl
uv run python "${CLAUDE_SKILL_DIR}/scripts/mutation_ledger.py" selftest
uv run python "${CLAUDE_SKILL_DIR}/scripts/mutation_ledger.py" rate output/mutation-ledger.jsonl
```

The script path is relative to this skill's directory. Not using uv? Run the same commands with `.venv/bin/python` or `poetry run python` instead of `uv run python`; see rf-setup.

An empty or missing ledger reports a rate of 0.0, so a run with no mutants never looks like a full kill.

## Companion Skills

| Need | Skill |
|------|-------|
| Write tests, suites, user keywords, resources and variables in Robot Framework syntax | `rf-language` |
| Install Robot Framework or a library, fix the environment | `rf-setup` |
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
*** Settings ***
Documentation Mutant checks: a new or changed test must fail when its code or data is broken.
Library ../scripts/mutation_ledger.py

*** Variables ***
${MUTATION_LOG} ${OUTPUT_DIR}${/}mutation-ledger.jsonl

*** Keywords ***
Mutant Must Fail
[Documentation] Runs ``keyword`` with a mutated input or expectation; it must fail
... on its assertion. A typo or a missing keyword is not a kill.
... Writes one ledger row and returns True when the mutant was killed.
... A survivor is a finding about the check, never a pass.
[Arguments] ${test_id} ${operator} ${keyword} @{args}
${status} ${message}= Run Keyword And Ignore Error ${keyword} @{args}
${killed}= Is Kill ${{$status == 'PASS'}} ${message}
Record Mutation ${MUTATION_LOG} ${test_id} ${operator} ${killed}
RETURN ${killed}

Mutation Kill Rate
[Documentation] Returns ``{killed, total, rate, survivors}`` read from the ledger.
${r}= Kill Rate ${MUTATION_LOG}
RETURN ${r}
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
#!/usr/bin/env python3
# /// script
# requires-python = ">=3.10"
# dependencies = ["robotframework>=7"]
# ///
"""Kill ledger for hand-made mutants. Stdlib only.

A check is worth keeping only when it fails once its code or data is broken.
Each mutant run appends one JSON line; the kill rate is read back from the file.

uv run python scripts/mutation_ledger.py rate LEDGER.jsonl # {killed, total, rate, survivors}
uv run python scripts/mutation_ledger.py selftest

Also importable as a Robot Framework library (see assets/mutants.resource).
"""
import json
import os
import re
import sys
import tempfile

# Failures that mean the mutant never ran its check (typo, bad call), not a kill.
_NOT_A_KILL = re.compile(
r"No keyword with name|Variable '.*' not found|expected \d+(?: to \d+)? arguments?, got"
r"|Importing .* failed|Resource file .* does not exist"
)


def is_kill(passed, message):
"""True only when the mutated check failed on its assertion."""
return not passed and not _NOT_A_KILL.search(str(message))


def record_mutation(path, test_id, operator, killed):
"""Append one row. ``killed`` is True when the mutated run failed."""
row = {"test": str(test_id), "operator": str(operator), "killed": bool(killed)}
with open(path, "a", encoding="utf-8") as fh:
fh.write(json.dumps(row) + "\n")
return row


def kill_rate(path):
"""Summarise the ledger. A missing or empty ledger has rate 0.0, never 1.0."""
rows = []
if os.path.exists(path):
with open(path, encoding="utf-8") as fh:
rows = [json.loads(line) for line in fh if line.strip()]
# The ledger is append-only: a rerun of a mutant replaces its earlier row.
rows = list({(r["test"], r["operator"]): r for r in rows}.values())
killed = sum(1 for r in rows if r["killed"])
survivors = [f'{r["test"]}:{r["operator"]}' for r in rows if not r["killed"]]
total = len(rows)
return {"killed": killed, "total": total,
"rate": round(killed / total, 3) if total else 0.0,
"survivors": survivors}


def _selftest():
with tempfile.TemporaryDirectory() as d:
p = os.path.join(d, "kills.jsonl")
assert kill_rate(p)["rate"] == 0.0
for op, killed in (("a", False), ("a", True), ("b", True), ("c", True), ("d", False)):
record_mutation(p, "T1", op, killed)
r = kill_rate(p)
assert r == {"killed": 3, "total": 4, "rate": 0.75,
"survivors": ["T1:d"]}, r
assert not is_kill(False, "No keyword with name 'X' found.")
print(json.dumps({"selftest": "ok"}))


def main(argv):
if argv[:1] == ["selftest"]:
_selftest()
return 0
if len(argv) == 2 and argv[0] == "rate":
print(json.dumps(kill_rate(argv[1])))
return 0
print(__doc__, file=sys.stderr)
return 2


if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
id: narrow-test-quality-mutant-01
skill: rf-test-quality
tier: narrow
description: |
Add a mutant check for an existing equality test and run it. Gated by a
Mutant Must Fail call in the suite and a mutation ledger row written by
the run.
prompt: |
tests/example.robot checks that Build Greeting returns "Hello, World".
Prove the check fails when the expected value is wrong: add a mutant
check with a wrong value, run the suite into results/, and report the
kill rate.
model: claude-haiku-5-5
max_turns: 14
timeout_seconds: 300
allowed_tools:
- Read
- Write
- Edit
- Bash
- Glob
- Grep
- Skill
grader_checks:
- type: file_contains
name: suite_uses_mutant_check
path: tests/example.robot
regex: 'Mutant Must Fail\s+\S+\s+\S+\s+Should Be Equal\s+\$\{greeting\}'
- type: robot_pass
name: suite_still_passes
path: tests/example.robot
- type: file_contains
name: ledger_row_written
path: results/mutation-ledger.jsonl
regex: '"killed": true'
fixture: sut-minimal
primary_metric: file_contains
28 changes: 28 additions & 0 deletions plugins/rf-core/evals/triggers/rf-test-quality.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Trigger query set for rf-test-quality (format: eval/triggers/README.md).
# Tune descriptions on the train split only; never edit validation queries
# to make a description pass (rotate them only when the skill's intent changes).
skill: rf-test-quality
model: claude-haiku-5-5
runs: 3
threshold: 0.5
queries:
# should trigger (train)
- {id: tq-t01, split: train, should_trigger: true, query: "I added a Robot Framework test, how do I prove it fails when the code is broken?"}
- {id: tq-t02, split: train, should_trigger: true, query: "Add mutant checks to my Robot suite and report the kill rate"}
- {id: tq-t03, split: train, should_trigger: true, note: "indirect", query: "my robot test always passes even when the endpoint is wrong, is the check too weak?"}
- {id: tq-t04, split: train, should_trigger: true, note: "edit existing file", query: "Edit tests/login.robot so each new check is wrapped in Mutant Must Fail with a wrong expected value"}
- {id: tq-t05, split: train, should_trigger: true, note: "indirect", query: "Before I merge, I want proof my new Robot keywords actually catch regressions"}
# should trigger (validation)
- {id: tq-v01, split: validation, should_trigger: true, query: "Show that the new status code check in tests/api.robot goes red on a wrong value"}
- {id: tq-v02, split: validation, should_trigger: true, query: "Which of my changed Robot tests survive a mutated expectation?"}
- {id: tq-v03, split: validation, should_trigger: true, query: "Read mutation-ledger.jsonl from my Robot run and tell me the kill rate and survivors"}
# should NOT trigger (train)
- {id: tq-n01, split: train, should_trigger: false, note: "sibling rf-setup", query: "Install robotframework-browser into our uv project"}
- {id: tq-n02, split: train, should_trigger: false, note: "sibling rf-language", query: "Add a suite setup in an __init__.robot file"}
- {id: tq-n03, split: train, should_trigger: false, note: "sibling rf-requests", query: "Write a Robot test that POSTs a user with RequestsLibrary and checks the 201"}
- {id: tq-n04, split: train, should_trigger: false, note: "non-RF look-alike", query: "Write a pytest test that checks my Flask health endpoint returns 200"}
- {id: tq-n05, split: train, should_trigger: false, note: "sibling rf-browser", query: "Click the login button and wait for the dashboard in a Browser library test"}
# should NOT trigger (validation)
- {id: tq-nv01, split: validation, should_trigger: false, note: "non-RF look-alike", query: "Run mutmut on my Python package"}
- {id: tq-nv02, split: validation, should_trigger: false, note: "sibling rf-language", query: "Rename a keyword across my Robot suites and fix the callers"}
- {id: tq-nv03, split: validation, should_trigger: false, note: "non-RF look-alike", query: "Set up Stryker mutation testing for my TypeScript project"}
59 changes: 59 additions & 0 deletions plugins/rf-core/skills/rf-test-quality/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
---
name: rf-test-quality
description: "Use after changing an RF test, to prove it fails when its code breaks (mutants)."
license: Apache-2.0
compatibility: "Requires Python 3.10+ with robotframework>=7 in the project environment for the bundled mutation_ledger script (stdlib only otherwise)."
metadata:
author: manykarim
version: "1.0.0"
---

# Robot Framework Test Quality Skill

## When to use

Load this skill after you add or change a test, before you call it done. A test that passes against broken code proves nothing.

## The rule

Every new or changed test is seen failing once. Break the thing it checks, run it, and confirm it goes red. Then restore the code and confirm it goes green.

## Mutant checks in a suite

Copy `assets/mutants.resource` from this skill into the project's `resources/` folder, import it, and wrap the real check in `Mutant Must Fail`, feeding it the value the test produced and a mutated expectation:

```robotframework
*** Settings ***
Resource ../resources/mutants.resource

*** Test Cases ***
Status Check Catches A Wrong Code
${total}= Evaluate 2 + 3
Should Be Equal As Integers ${total} 5
${killed}= Mutant Must Fail T1 flip-total
... Should Be Equal As Integers ${total} 6
Should Be True ${killed}
```

Only an assertion failure counts as a kill. A mutant that fails because of a typo, a missing keyword, a missing variable or a wrong argument count is recorded as a survivor: fix the mutant call.

Each call writes one row to `mutation-ledger.jsonl` in the output directory. `Mutation Kill Rate` returns `{killed, total, rate, survivors}`. A rerun of the same test and operator replaces its earlier row, so the rate uses the latest result per mutant. Quote the rate for each changed test. A survivor means the check is too weak: fix the check, not the mutant.

## Reading the ledger

```bash
uv run python "${CLAUDE_SKILL_DIR}/scripts/mutation_ledger.py" rate results/mutation-ledger.jsonl
uv run python "${CLAUDE_SKILL_DIR}/scripts/mutation_ledger.py" selftest
uv run python "${CLAUDE_SKILL_DIR}/scripts/mutation_ledger.py" rate output/mutation-ledger.jsonl
```

The script path is relative to this skill's directory. Not using uv? Run the same commands with `.venv/bin/python` or `poetry run python` instead of `uv run python`; see rf-setup.

An empty or missing ledger reports a rate of 0.0, so a run with no mutants never looks like a full kill.

## Companion Skills

| Need | Skill |
|------|-------|
| Write tests, suites, user keywords, resources and variables in Robot Framework syntax | `rf-language` |
| Install Robot Framework or a library, fix the environment | `rf-setup` |
23 changes: 23 additions & 0 deletions plugins/rf-core/skills/rf-test-quality/assets/mutants.resource
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
*** Settings ***
Documentation Mutant checks: a new or changed test must fail when its code or data is broken.
Library ../scripts/mutation_ledger.py

*** Variables ***
${MUTATION_LOG} ${OUTPUT_DIR}${/}mutation-ledger.jsonl

*** Keywords ***
Mutant Must Fail
[Documentation] Runs ``keyword`` with a mutated input or expectation; it must fail
... on its assertion. A typo or a missing keyword is not a kill.
... Writes one ledger row and returns True when the mutant was killed.
... A survivor is a finding about the check, never a pass.
[Arguments] ${test_id} ${operator} ${keyword} @{args}
${status} ${message}= Run Keyword And Ignore Error ${keyword} @{args}
${killed}= Is Kill ${{$status == 'PASS'}} ${message}
Record Mutation ${MUTATION_LOG} ${test_id} ${operator} ${killed}
RETURN ${killed}

Mutation Kill Rate
[Documentation] Returns ``{killed, total, rate, survivors}`` read from the ledger.
${r}= Kill Rate ${MUTATION_LOG}
RETURN ${r}
Loading
Loading