Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions bundles/rf-agentskills/skills/rf-run-digest/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
---
name: rf-run-digest
description: "Use after an RF run: read results with robotcode results, not output.xml."
license: Apache-2.0
compatibility: "Requires robotcode[all]>=2.8 (or robotcode[runner]) in the project environment for the results commands. No bundled scripts."
metadata:
author: manykarim
version: "1.0.0"
---

# Robot Framework Run Digest Skill

## When to use

Load this skill after a `robot` or `pabot` run, before you look at its result. output.xml and log.html are large; reading or parsing them wastes time and hides the few failures that matter.

## The rule

Read results with `robotcode results`, not output.xml. Never write your own output.xml parser.

## Read a run

Without `-o`, each command finds the result file from the project's robot.toml profile (output.xml or output.json). Pass `-o <file or directory>` after the subcommand to name another one. `--format json` goes before `results`.

```bash
uv run robotcode results summary # headline status and counts
uv run robotcode results summary --failed # plus each failing test with its message
uv run robotcode results show --failed # failing tests, one line each
uv run robotcode results stats --by suite --by tag # counts per suite and tag
uv run robotcode results log -bl "<longname>" --level FAIL # keyword tree of one test
uv run robotcode results diff before.xml after.xml --only new-failures
```

Not using uv? Run the same commands with `.venv/bin/robotcode` or `poetry run robotcode`; see rf-setup.

Start with `summary --failed`. Open `log` for one named test only. Quote the summary counts and the failure messages when you report a run.

`--search TEXT` or `--search-regex PATTERN` narrows any command to tests whose names, messages, tags or log text match; `-i`/`-e` filter by tag and `-s`/`-t` by suite or test name. Filters run before counting, so `stats --by tag --search Timeout` counts only the matching tests.

`results` has no failure grouping. To group failures that differ only by ids or hosts, read `uv run robotcode --format json results summary --failed` and group the `failed[].message` values yourself. `results diff` always exits 0; read `newFailures` from its JSON to gate a build.

## Companion Skills

| Need | Skill |
|------|-------|
| Install Robot Framework or a library, fix the environment | `rf-setup` |
| Write tests, suites, user keywords, resources and variables in Robot Framework syntax | `rf-language` |
59 changes: 59 additions & 0 deletions bundles/rf-agentskills/skills/rf-test-quality/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
---
name: rf-test-quality
description: "Use after changing an RF test, to prove it fails when its code breaks (mutants)."
license: Apache-2.0
compatibility: "Requires Python 3.10+ with robotframework>=7 in the project environment for the bundled mutation_ledger script (stdlib only otherwise)."
metadata:
author: manykarim
version: "1.0.0"
---

# Robot Framework Test Quality Skill

## When to use

Load this skill after you add or change a test, before you call it done. A test that passes against broken code proves nothing.

## The rule

Every new or changed test is seen failing once. Break the thing it checks, run it, and confirm it goes red. Then restore the code and confirm it goes green.

## Mutant checks in a suite

Copy `assets/mutants.resource` from this skill into the project's `resources/` folder, import it, and wrap the real check in `Mutant Must Fail`, feeding it the value the test produced and a mutated expectation:

```robotframework
*** Settings ***
Resource ../resources/mutants.resource

*** Test Cases ***
Status Check Catches A Wrong Code
${total}= Evaluate 2 + 3
Should Be Equal As Integers ${total} 5
${killed}= Mutant Must Fail T1 flip-total
... Should Be Equal As Integers ${total} 6
Should Be True ${killed}
```

Only an assertion failure counts as a kill. A mutant that fails because of a typo, a missing keyword, a missing variable or a wrong argument count is recorded as a survivor: fix the mutant call.

Each call writes one row to `mutation-ledger.jsonl` in the output directory. `Mutation Kill Rate` returns `{killed, total, rate, survivors}`. A rerun of the same test and operator replaces its earlier row, so the rate uses the latest result per mutant. Quote the rate for each changed test. A survivor means the check is too weak: fix the check, not the mutant.

## Reading the ledger

```bash
uv run python "${CLAUDE_SKILL_DIR}/scripts/mutation_ledger.py" rate results/mutation-ledger.jsonl
uv run python "${CLAUDE_SKILL_DIR}/scripts/mutation_ledger.py" selftest
uv run python "${CLAUDE_SKILL_DIR}/scripts/mutation_ledger.py" rate output/mutation-ledger.jsonl
```

The script path is relative to this skill's directory. Not using uv? Run the same commands with `.venv/bin/python` or `poetry run python` instead of `uv run python`; see rf-setup.

An empty or missing ledger reports a rate of 0.0, so a run with no mutants never looks like a full kill.

## Companion Skills

| Need | Skill |
|------|-------|
| Write tests, suites, user keywords, resources and variables in Robot Framework syntax | `rf-language` |
| Install Robot Framework or a library, fix the environment | `rf-setup` |
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
*** Settings ***
Documentation Mutant checks: a new or changed test must fail when its code or data is broken.
Library ../scripts/mutation_ledger.py

*** Variables ***
${MUTATION_LOG} ${OUTPUT_DIR}${/}mutation-ledger.jsonl

*** Keywords ***
Mutant Must Fail
[Documentation] Runs ``keyword`` with a mutated input or expectation; it must fail
... on its assertion. A typo or a missing keyword is not a kill.
... Writes one ledger row and returns True when the mutant was killed.
... A survivor is a finding about the check, never a pass.
[Arguments] ${test_id} ${operator} ${keyword} @{args}
${status} ${message}= Run Keyword And Ignore Error ${keyword} @{args}
${killed}= Is Kill ${{$status == 'PASS'}} ${message}
Record Mutation ${MUTATION_LOG} ${test_id} ${operator} ${killed}
RETURN ${killed}

Mutation Kill Rate
[Documentation] Returns ``{killed, total, rate, survivors}`` read from the ledger.
${r}= Kill Rate ${MUTATION_LOG}
RETURN ${r}
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
#!/usr/bin/env python3
# /// script
# requires-python = ">=3.10"
# dependencies = ["robotframework>=7"]
# ///
"""Kill ledger for hand-made mutants. Stdlib only.

A check is worth keeping only when it fails once its code or data is broken.
Each mutant run appends one JSON line; the kill rate is read back from the file.

uv run python scripts/mutation_ledger.py rate LEDGER.jsonl # {killed, total, rate, survivors}
uv run python scripts/mutation_ledger.py selftest

Also importable as a Robot Framework library (see assets/mutants.resource).
"""
import json
import os
import re
import sys
import tempfile

# Failures that mean the mutant never ran its check (typo, bad call), not a kill.
_NOT_A_KILL = re.compile(
r"No keyword with name|Variable '.*' not found|expected \d+(?: to \d+)? arguments?, got"
r"|Importing .* failed|Resource file .* does not exist"
)


def is_kill(passed, message):
"""True only when the mutated check failed on its assertion."""
return not passed and not _NOT_A_KILL.search(str(message))


def record_mutation(path, test_id, operator, killed):
"""Append one row. ``killed`` is True when the mutated run failed."""
row = {"test": str(test_id), "operator": str(operator), "killed": bool(killed)}
with open(path, "a", encoding="utf-8") as fh:
fh.write(json.dumps(row) + "\n")
return row


def kill_rate(path):
"""Summarise the ledger. A missing or empty ledger has rate 0.0, never 1.0."""
rows = []
if os.path.exists(path):
with open(path, encoding="utf-8") as fh:
rows = [json.loads(line) for line in fh if line.strip()]
# The ledger is append-only: a rerun of a mutant replaces its earlier row.
rows = list({(r["test"], r["operator"]): r for r in rows}.values())
killed = sum(1 for r in rows if r["killed"])
survivors = [f'{r["test"]}:{r["operator"]}' for r in rows if not r["killed"]]
total = len(rows)
return {"killed": killed, "total": total,
"rate": round(killed / total, 3) if total else 0.0,
"survivors": survivors}


def _selftest():
with tempfile.TemporaryDirectory() as d:
p = os.path.join(d, "kills.jsonl")
assert kill_rate(p)["rate"] == 0.0
for op, killed in (("a", False), ("a", True), ("b", True), ("c", True), ("d", False)):
record_mutation(p, "T1", op, killed)
r = kill_rate(p)
assert r == {"killed": 3, "total": 4, "rate": 0.75,
"survivors": ["T1:d"]}, r
assert not is_kill(False, "No keyword with name 'X' found.")
print(json.dumps({"selftest": "ok"}))


def main(argv):
if argv[:1] == ["selftest"]:
_selftest()
return 0
if len(argv) == 2 and argv[0] == "rate":
print(json.dumps(kill_rate(argv[1])))
return 0
print(__doc__, file=sys.stderr)
return 2


if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
28 changes: 28 additions & 0 deletions plugins/rf-core/evals/tasks/narrow/narrow-run-digest-01.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
id: narrow-run-digest-01
skill: rf-run-digest
tier: narrow
description: |
Run a suite and report from `robotcode results summary --failed`. Gated by
a results.txt that holds the command's output.
prompt: |
Run tests/example.robot into results/, save the output of the robotcode
results summary --failed command to results/results.txt, then report the
pass and fail counts from it.
model: claude-haiku-5-5
max_turns: 14
timeout_seconds: 300
allowed_tools:
- Read
- Write
- Edit
- Bash
- Glob
- Grep
- Skill
grader_checks:
- type: file_contains
name: results_saved
path: results/results.txt
regex: '_Total:_ \d+'
fixture: sut-minimal
primary_metric: file_contains
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
id: narrow-test-quality-mutant-01
skill: rf-test-quality
tier: narrow
description: |
Add a mutant check for an existing equality test and run it. Gated by a
Mutant Must Fail call in the suite and a mutation ledger row written by
the run.
prompt: |
tests/example.robot checks that Build Greeting returns "Hello, World".
Prove the check fails when the expected value is wrong: add a mutant
check with a wrong value, run the suite into results/, and report the
kill rate.
model: claude-haiku-5-5
max_turns: 14
timeout_seconds: 300
allowed_tools:
- Read
- Write
- Edit
- Bash
- Glob
- Grep
- Skill
grader_checks:
- type: file_contains
name: suite_uses_mutant_check
path: tests/example.robot
regex: 'Mutant Must Fail\s+\S+\s+\S+\s+Should Be Equal\s+\$\{greeting\}'
- type: robot_pass
name: suite_still_passes
path: tests/example.robot
- type: file_contains
name: ledger_row_written
path: results/mutation-ledger.jsonl
regex: '"killed": true'
fixture: sut-minimal
primary_metric: file_contains
28 changes: 28 additions & 0 deletions plugins/rf-core/evals/triggers/rf-run-digest.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Trigger query set for rf-run-digest (format: eval/triggers/README.md).
# Tune descriptions on the train split only; never edit validation queries
# to make a description pass (rotate them only when the skill's intent changes).
skill: rf-run-digest
model: claude-haiku-5-5
runs: 3
threshold: 0.5
queries:
# should trigger (train)
- {id: rd-t01, split: train, should_trigger: true, query: "My robot run just finished with 80 failures, what went wrong?"}
- {id: rd-t02, split: train, should_trigger: true, query: "Summarize results/output.xml from the last Robot Framework run"}
- {id: rd-t03, split: train, should_trigger: true, note: "indirect", query: "log.html is huge, how do I see which robot tests failed and why?"}
- {id: rd-t04, split: train, should_trigger: true, note: "edit existing file", query: "Edit run_tests.sh so it prints the failed tests from output.xml after every robot call"}
- {id: rd-t05, split: train, should_trigger: true, note: "indirect", query: "The nightly robot job went red again, I just need the short story of what broke"}
# should trigger (validation)
- {id: rd-v01, split: validation, should_trigger: true, query: "Group the failures in this Robot output.xml by error message"}
- {id: rd-v02, split: validation, should_trigger: true, query: "Compare two Robot Framework runs and tell me which tests changed status"}
- {id: rd-v03, split: validation, should_trigger: true, query: "Show the failing keyword path for one test in my Robot output.xml"}
# should NOT trigger (train)
- {id: rd-n01, split: train, should_trigger: false, note: "sibling rf-setup", query: "Install robotframework-requests into our uv project"}
- {id: rd-n02, split: train, should_trigger: false, note: "sibling rf-language", query: "Add a suite teardown in an __init__.robot file"}
- {id: rd-n03, split: train, should_trigger: false, note: "sibling rf-requests", query: "Write a Robot test that GETs /health with RequestsLibrary and checks the 200"}
- {id: rd-n04, split: train, should_trigger: false, note: "non-RF look-alike", query: "Summarize the failures in this pytest run output"}
- {id: rd-n05, split: train, should_trigger: false, note: "sibling rf-browser", query: "Click the login button and wait for the dashboard in a Browser library test"}
# should NOT trigger (validation)
- {id: rd-nv01, split: validation, should_trigger: false, note: "non-RF look-alike", query: "Parse this JUnit XML report from pytest"}
- {id: rd-nv02, split: validation, should_trigger: false, note: "sibling rf-language", query: "Rename a keyword across my Robot suites and fix the callers"}
- {id: rd-nv03, split: validation, should_trigger: false, note: "non-RF look-alike", query: "Why did my GitHub Actions build fail on the npm install step?"}
28 changes: 28 additions & 0 deletions plugins/rf-core/evals/triggers/rf-test-quality.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Trigger query set for rf-test-quality (format: eval/triggers/README.md).
# Tune descriptions on the train split only; never edit validation queries
# to make a description pass (rotate them only when the skill's intent changes).
skill: rf-test-quality
model: claude-haiku-5-5
runs: 3
threshold: 0.5
queries:
# should trigger (train)
- {id: tq-t01, split: train, should_trigger: true, query: "I added a Robot Framework test, how do I prove it fails when the code is broken?"}
- {id: tq-t02, split: train, should_trigger: true, query: "Add mutant checks to my Robot suite and report the kill rate"}
- {id: tq-t03, split: train, should_trigger: true, note: "indirect", query: "my robot test always passes even when the endpoint is wrong, is the check too weak?"}
- {id: tq-t04, split: train, should_trigger: true, note: "edit existing file", query: "Edit tests/login.robot so each new check is wrapped in Mutant Must Fail with a wrong expected value"}
- {id: tq-t05, split: train, should_trigger: true, note: "indirect", query: "Before I merge, I want proof my new Robot keywords actually catch regressions"}
# should trigger (validation)
- {id: tq-v01, split: validation, should_trigger: true, query: "Show that the new status code check in tests/api.robot goes red on a wrong value"}
- {id: tq-v02, split: validation, should_trigger: true, query: "Which of my changed Robot tests survive a mutated expectation?"}
- {id: tq-v03, split: validation, should_trigger: true, query: "Read mutation-ledger.jsonl from my Robot run and tell me the kill rate and survivors"}
# should NOT trigger (train)
- {id: tq-n01, split: train, should_trigger: false, note: "sibling rf-setup", query: "Install robotframework-browser into our uv project"}
- {id: tq-n02, split: train, should_trigger: false, note: "sibling rf-language", query: "Add a suite setup in an __init__.robot file"}
- {id: tq-n03, split: train, should_trigger: false, note: "sibling rf-requests", query: "Write a Robot test that POSTs a user with RequestsLibrary and checks the 201"}
- {id: tq-n04, split: train, should_trigger: false, note: "non-RF look-alike", query: "Write a pytest test that checks my Flask health endpoint returns 200"}
- {id: tq-n05, split: train, should_trigger: false, note: "sibling rf-browser", query: "Click the login button and wait for the dashboard in a Browser library test"}
# should NOT trigger (validation)
- {id: tq-nv01, split: validation, should_trigger: false, note: "non-RF look-alike", query: "Run mutmut on my Python package"}
- {id: tq-nv02, split: validation, should_trigger: false, note: "sibling rf-language", query: "Rename a keyword across my Robot suites and fix the callers"}
- {id: tq-nv03, split: validation, should_trigger: false, note: "non-RF look-alike", query: "Set up Stryker mutation testing for my TypeScript project"}
47 changes: 47 additions & 0 deletions plugins/rf-core/skills/rf-run-digest/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
---
name: rf-run-digest
description: "Use after an RF run: read results with robotcode results, not output.xml."
license: Apache-2.0
compatibility: "Requires robotcode[all]>=2.8 (or robotcode[runner]) in the project environment for the results commands. No bundled scripts."
metadata:
author: manykarim
version: "1.0.0"
---

# Robot Framework Run Digest Skill

## When to use

Load this skill after a `robot` or `pabot` run, before you look at its result. output.xml and log.html are large; reading or parsing them wastes time and hides the few failures that matter.

## The rule

Read results with `robotcode results`, not output.xml. Never write your own output.xml parser.

## Read a run

Without `-o`, each command finds the result file from the project's robot.toml profile (output.xml or output.json). Pass `-o <file or directory>` after the subcommand to name another one. `--format json` goes before `results`.

```bash
uv run robotcode results summary # headline status and counts
uv run robotcode results summary --failed # plus each failing test with its message
uv run robotcode results show --failed # failing tests, one line each
uv run robotcode results stats --by suite --by tag # counts per suite and tag
uv run robotcode results log -bl "<longname>" --level FAIL # keyword tree of one test
uv run robotcode results diff before.xml after.xml --only new-failures
```

Not using uv? Run the same commands with `.venv/bin/robotcode` or `poetry run robotcode`; see rf-setup.

Start with `summary --failed`. Open `log` for one named test only. Quote the summary counts and the failure messages when you report a run.

`--search TEXT` or `--search-regex PATTERN` narrows any command to tests whose names, messages, tags or log text match; `-i`/`-e` filter by tag and `-s`/`-t` by suite or test name. Filters run before counting, so `stats --by tag --search Timeout` counts only the matching tests.

`results` has no failure grouping. To group failures that differ only by ids or hosts, read `uv run robotcode --format json results summary --failed` and group the `failed[].message` values yourself. `results diff` always exits 0; read `newFailures` from its JSON to gate a build.

## Companion Skills

| Need | Skill |
|------|-------|
| Install Robot Framework or a library, fix the environment | `rf-setup` |
| Write tests, suites, user keywords, resources and variables in Robot Framework syntax | `rf-language` |
Loading
Loading