Skip to content

feat(rf-core): add rf-run-digest skill - #12

Open
ai94iq wants to merge 17 commits into
MarketSquare:mainfrom
ai94iq:feat/digest-runs
Open

ai94iq wants to merge 17 commits into
MarketSquare:mainfrom
ai94iq:feat/digest-runs

Conversation

@ai94iq

@ai94iq ai94iq commented Nov 15, 2026 •

Copy link
Copy Markdown

Adds the rf-run-digest skill. After a run, the agent reads a short digest instead of output.xml or log.html.

Verified: plugin tests, 42 passed. The digest selftest passes on bare Python. The digest of a real Browser run matches its log. Catalog, governance, plugin references and docs voice checks pass.

What this changes

The digest gives the summary and groups failures by message. It reads Browser, SeleniumLibrary, AppiumLibrary, MaestroLibrary and RequestsLibrary failures. A listener writes digest.txt when the run ends. Its opt-in shrink argument rewrites output.xml without the keywords of passed tests.

Review checklist

Tick each item, or write "n/a: " after it. Reviewers don't approve a PR
with an unticked item that has no reason (see CONTRIBUTING.md, "Review").

  • Purpose. Each new or changed skill and subagent says what it is for and when to use it, in its description.
  • Supported versions. catalog.yaml lists the libraries and version ranges the content was checked against, and the Robot Framework range. n/a: no new library, catalog.yaml is unchanged
  • Examples. .robot examples pass robot --dryrun; Python examples run.
  • Security-relevant changes. Hooks, scripts, MCP configuration, manifests, subagent tools and skill allowed-tools changes are listed below; the security gate passes, and any new baseline entry has a reason. n/a: scanners not installed locally, and the CI security job waits for maintainer approval on fork PRs
  • Tool permissions. Every subagent declares an explicit tools list; no allowed-tools entry grants unrestricted shell access. n/a: no subagent, and the skill sets no allowed-tools
  • Documentation voice. Text follows the writing rules in CONTRIBUTING.md and passes the docs voice check.

Security-relevant changes

  • scripts/run_digest.py (new): stdlib only. Reads output.xml. For shrink only, runs python -m robot.rebot with an argument list, never a shell. No network.
  • scripts/digest_listener.py (new): Robot listener, writes digest.txt to the output dir. With shrink, rewrites output.xml in place. No network.
  • scripts/digest_selftest.py (new): selftest on built-in samples.

Adds a skill, a Mutant Must Fail resource and a stdlib kill ledger so a
new or changed test is shown to fail when its check is broken. An empty
ledger reports rate 0.0, never a full kill.

Verified: pytest plugins/rf-core/tests/test_test_quality_skill.py, 4 passed;
with the ledger broken, 3 failed. build-catalog --check, governance,
validate-skills, plugin references and docs voice pass.
Add the rf-core skill rf-run-digest: read a short digest of a Robot
Framework run (summary, failures grouped by signature) before output.xml.
Ships digest_listener.py and run_digest.py (stdlib only), a pytest that
runs the selftest and checks summary and grouped failures on a tiny
output.xml, a trigger query set and one narrow eval task. Bundle
regenerated with build-catalog.py.

Verified: rf-core pytest 332 passed; the new summary test failed with
run_digest.py broken; build-catalog --check, check-governance,
validate-skills --channel all, check-plugin-references,
check-docs-voice and check_content_rules all exit 0.
The digest listener rewrote output.xml in place with rebot --removekeywords
PASSED on every run, undocumented. It now does so only when given the
shrink listener argument (digest_listener.py:shrink). SKILL.md documents
the argument and what it removes. Both skill copies are identical.

Verified: pytest -q plugins/rf-core/tests/test_run_digest_skill.py
(log pytest-blocker1.log in the session scratchpad): 4 passed, 1 failed.
The failing test is the body-redaction test fixed in the next commit; the
two listener tests failed before this change.
The failures view printed the last request body (sent:) and response
body (body:) of each failed test, so tokens and passwords landed in
digest.txt. Bodies are no longer printed; the request line and status
stay. Both skill copies are identical.

Verified: pytest -q plugins/rf-core/tests/test_run_digest_skill.py
(log pytest-blocker2.log in the session scratchpad): 5 passed. The body
test failed before this change with the token in the output.
failures and trace now print a screenshot: line for failed tests, taken
from the last <img src> the test logged (Browser, SeleniumLibrary,
AppiumLibrary, MaestroLibrary). norm() drops the Playwright "Call log:"
tail, cuts absolute file paths to their basename and replaces 32+ hex
session ids with <id> before grouping. SKILL.md names the libraries.
The bundles/rf-agentskills copies match the plugin files (cmp).

Verified (log: scratchpad/digest-pytest.log, written this run):
- pytest plugins/rf-core/tests/test_run_digest_skill.py: 6 new tests
  red before the change, then 12 passed
- run_digest.py selftest: ok
- failures on a real Browser run: 1 TimeoutError group with its
  FAILURE_SCREENSHOT_1.png line. The Appium fixture is built from the
  library's message format, not a live run.
@ai94iq
ai94iq requested a review from manykarim as a code owner November 15, 2026 07:47
@ai94iq ai94iq changed the title Feat/digest runs feat(rf-core): add rf-run-digest skill Nov 15, 2026
Mutant Must Fail now counts a kill only on an assertion failure; a
missing keyword, missing variable or wrong argument count is a survivor.
kill_rate keeps the latest ledger row per test and operator. SKILL.md
mutates the value the test checks, imports the resource through
${CLAUDE_SKILL_DIR} and runs scripts with uv run. Trigger set grows to
8 positive and 8 negative queries with the sibling and look-alike
quotas; the narrow task targets tests/example.robot and adds robot_pass.
Tests use 3-of-4 data and assert the exact rate.

Verified: uv run --frozen python -m pytest -q
plugins/rf-core/tests/test_test_quality_skill.py
tests/eval/test_trigger_sets.py: red 6 failed, 29 passed (../red.log),
green 35 passed (../green.log). selftest ok. Plugin and bundle copies
identical (cmp).
SKILL.md calls the scripts through uv run and ${CLAUDE_SKILL_DIR}.
run_digest.py drops the fingerprint, body capture and unwrap helpers,
uses argparse, and moves the selftest to digest_selftest.py. The
listener reads DIGEST_MAX_PER_GROUP inside a try. Trigger set meets
the 8/8 and sibling quotas; the narrow task targets tests/example.robot
and adds a robot_pass grader.

Verified (logs in scratchpad mksq/digest-logs): red.txt 3 failed,
11 passed before the fix; green.txt 42 passed for
test_run_digest_skill.py and tests/eval/test_trigger_sets.py; selftest
ok; failures output on cs2369-r1 output.xml matches before.txt apart
from the removed fp tag.
@ai94iq

ai94iq commented Nov 15, 2026

Copy link
Copy Markdown
Author

Merge chain: #11, then #12, then #13, then #14, then #15, then #16.

All six merge into main cleanly in this order (dry run on 2026-11-15).

Apart from #13 and #15, the PRs do not depend on each other.

ai94iq and others added 10 commits November 15, 2026 18:47
…ndled parser

The skill now points to robotcode results summary/show/stats/log/diff
instead of its own output.xml parser and listener. Removes the scripts,
their tests and the shrink option; updates the bundle copy, the command
test map, the trigger set and the eval task.
Stack on rf-test-quality so the two merge one after the other: union of the
skill lists and counts in the installer tests, robotcode declared in the
skill compatibility field, skill version kept at the plugin version.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant