Repository navigation
Conversation
Adds a skill, a Mutant Must Fail resource and a stdlib kill ledger so a new or changed test is shown to fail when its check is broken. An empty ledger reports rate 0.0, never a full kill. Verified: pytest plugins/rf-core/tests/test_test_quality_skill.py, 4 passed; with the ledger broken, 3 failed. build-catalog --check, governance, validate-skills, plugin references and docs voice pass.
Mutant Must Fail now counts a kill only on an assertion failure; a
missing keyword, missing variable or wrong argument count is a survivor.
kill_rate keeps the latest ledger row per test and operator. SKILL.md
mutates the value the test checks, imports the resource through
${CLAUDE_SKILL_DIR} and runs scripts with uv run. Trigger set grows to
8 positive and 8 negative queries with the sibling and look-alike
quotas; the narrow task targets tests/example.robot and adds robot_pass.
Tests use 3-of-4 data and assert the exact rate.
Verified: uv run --frozen python -m pytest -q
plugins/rf-core/tests/test_test_quality_skill.py
tests/eval/test_trigger_sets.py: red 6 failed, 29 passed (../red.log),
green 35 passed (../green.log). selftest ok. Plugin and bundle copies
identical (cmp).
This was referenced Nov 15, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the rf-test-quality skill. It proves a new or changed test fails when its check is broken.
Verified: plugin tests and trigger set tests, 35 passed. A live robot run of the resource scored a real failure as a kill and a typo as a survivor (rate 0.5). Catalog, governance, plugin references and docs voice checks pass.
What this changes
You wrap a mutated check in
Mutant Must Fail. Only an assertion failure counts as a kill. A typo or a missing keyword counts as a survivor. The ledger keeps the latest row per mutant and reports the kill rate.Review checklist
Tick each item, or write "n/a: " after it. Reviewers don't approve a PR
with an unticked item that has no reason (see CONTRIBUTING.md, "Review").
catalog.yamllists the libraries and version ranges the content was checked against, and the Robot Framework range. n/a: no new library,catalog.yamlis unchanged.robotexamples passrobot --dryrun; Python examples run.toolsand skillallowed-toolschanges are listed below; the security gate passes, and any new baseline entry has a reason. n/a: scanners not installed locally, and the CI security job waits for maintainer approval on fork PRstoolslist; noallowed-toolsentry grants unrestricted shell access. n/a: no subagent orallowed-toolschangeSecurity-relevant changes
scripts/mutation_ledger.py(new): stdlib only, appends one JSON line per mutant to a ledger file in the output dir. No network, no shell.