Skip to content

feat(rf-core): add rf-test-quality mutant checks - #11

Open
ai94iq wants to merge 5 commits into
MarketSquare:mainfrom
ai94iq:feat/test-quality-mutants
Open

ai94iq wants to merge 5 commits into
MarketSquare:mainfrom
ai94iq:feat/test-quality-mutants

Conversation

@ai94iq

@ai94iq ai94iq commented Nov 15, 2026 •

Copy link
Copy Markdown

Adds the rf-test-quality skill. It proves a new or changed test fails when its check is broken.

Verified: plugin tests and trigger set tests, 35 passed. A live robot run of the resource scored a real failure as a kill and a typo as a survivor (rate 0.5). Catalog, governance, plugin references and docs voice checks pass.

What this changes

You wrap a mutated check in Mutant Must Fail. Only an assertion failure counts as a kill. A typo or a missing keyword counts as a survivor. The ledger keeps the latest row per mutant and reports the kill rate.

Review checklist

Tick each item, or write "n/a: " after it. Reviewers don't approve a PR
with an unticked item that has no reason (see CONTRIBUTING.md, "Review").

  • Purpose. Each new or changed skill and subagent says what it is for and when to use it, in its description.
  • Supported versions. catalog.yaml lists the libraries and version ranges the content was checked against, and the Robot Framework range. n/a: no new library, catalog.yaml is unchanged
  • Examples. .robot examples pass robot --dryrun; Python examples run.
  • Security-relevant changes. Hooks, scripts, MCP configuration, manifests, subagent tools and skill allowed-tools changes are listed below; the security gate passes, and any new baseline entry has a reason. n/a: scanners not installed locally, and the CI security job waits for maintainer approval on fork PRs
  • Tool permissions. Every subagent declares an explicit tools list; no allowed-tools entry grants unrestricted shell access. n/a: no subagent or allowed-tools change
  • Documentation voice. Text follows the writing rules in CONTRIBUTING.md and passes the docs voice check.

Security-relevant changes

  • scripts/mutation_ledger.py (new): stdlib only, appends one JSON line per mutant to a ledger file in the output dir. No network, no shell.

Adds a skill, a Mutant Must Fail resource and a stdlib kill ledger so a
new or changed test is shown to fail when its check is broken. An empty
ledger reports rate 0.0, never a full kill.

Verified: pytest plugins/rf-core/tests/test_test_quality_skill.py, 4 passed;
with the ledger broken, 3 failed. build-catalog --check, governance,
validate-skills, plugin references and docs voice pass.
@ai94iq
ai94iq requested a review from manykarim as a code owner November 15, 2026 07:47
@ai94iq ai94iq changed the title feat: add rf-test-quality mutant checks skill feat(rf-core): add rf-test-quality mutant checks Nov 15, 2026
Mutant Must Fail now counts a kill only on an assertion failure; a
missing keyword, missing variable or wrong argument count is a survivor.
kill_rate keeps the latest ledger row per test and operator. SKILL.md
mutates the value the test checks, imports the resource through
${CLAUDE_SKILL_DIR} and runs scripts with uv run. Trigger set grows to
8 positive and 8 negative queries with the sibling and look-alike
quotas; the narrow task targets tests/example.robot and adds robot_pass.
Tests use 3-of-4 data and assert the exact rate.

Verified: uv run --frozen python -m pytest -q
plugins/rf-core/tests/test_test_quality_skill.py
tests/eval/test_trigger_sets.py: red 6 failed, 29 passed (../red.log),
green 35 passed (../green.log). selftest ok. Plugin and bundle copies
identical (cmp).
@ai94iq

ai94iq commented Nov 15, 2026

Copy link
Copy Markdown
Author

Merge chain: #11, then #12, then #13, then #14, then #15, then #16.

All six merge into main cleanly in this order (dry run on 2026-11-15).

Apart from #13 and #15, the PRs do not depend on each other.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant