diff --git a/openspec/changes/ai-contributor-docs/.openspec.yaml b/openspec/changes/ai-contributor-docs/.openspec.yaml new file mode 100644 index 0000000..90080a9 --- /dev/null +++ b/openspec/changes/ai-contributor-docs/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-11-16 diff --git a/openspec/changes/ai-contributor-docs/design.md b/openspec/changes/ai-contributor-docs/design.md new file mode 100644 index 0000000..2540315 --- /dev/null +++ b/openspec/changes/ai-contributor-docs/design.md @@ -0,0 +1,88 @@ +## Context + +See proposal.md (Why). What this change builds on: + +- `CONTRIBUTING.md` (173 lines) already has a table of PR checks with their commands. The same commands run in `.github/workflows/ci.yml`: `build-catalog.py --check`, `check-governance.py`, `check_content_rules.py`, `validate-skills.py`, `check-plugin-references.py`, `check-drift.sh`, `check-docs-voice.py`, and `pytest tests/ --ignore=tests/eval` plus each plugin's tests. The harness tests run with `uv sync --all-extras && uv run pytest tests/eval/` (`ci.yml:122`). +- `scripts/check-docs-voice.py` checks the top-level `*.md` files, so new top-level documents are covered without a change. +- `.gitignore` doesn't ignore `CLAUDE.md` or `AGENTS.md`, and neither file exists in the repository. +- `openspec/config.yaml` sets only `schema: spec-driven`. +- RobotCode's `AI_POLICY.md` (robotcodedev/robotcode, main) is the model for the policy. RobotCode's "Working with AI Agents" guide and its chat plugin already teach agents to use the CLI, and our skills hand CLI usage to that plugin. +- The CLI guide (https://robotcode.io/guides/cli/) has these pages: + - Discovering Tests (`discover`) + - Analyzing Code (`analyze code`) + - Analyzing Results (`results`) + - REPL (`repl`) + - Debugging (`robot-debug`) + - Browsing Documentation (`doc`) + - Wrapper (`--wrapper`) + - the `robot.toml` page for `robot`/`run`, and Troubleshooting for `config`/`profiles` + +## Goals / Non-Goals + +**Goals:** +- One short text that a coding agent can follow to make a PR that passes CI the first time. +- A policy that says what AI-assisted contributions must meet, adapted to a repository whose output is itself agent content. +- RobotCode as the shared base for Robot Framework capabilities, written down where contributors and OpenSpec proposals see it. +- Checks that fail when the guide and CI drift apart, so the guide doesn't need a reviewer to keep it true. + +**Non-Goals:** +- Changing the repository's own tooling to follow the RobotCode-first rule: that is `robotcode-first-tooling`. +- Changing review rules: that is `lean-governance`. This change links to `GOVERNANCE.md` and doesn't restate the rules. +- Detecting AI-written PRs automatically. Disclosure relies on the contributor. +- Removing stale documents: that is `repo-cleanup`. + +## Decisions + +### D1. Two documents: a policy and an operational guide + +`AI_POLICY.md` is for people. It explains what is expected and why, and follows the structure of RobotCode's policy (human responsibility, disclosure, licensing, communication, PRs, low-context submissions, enforcement), with a section for what is specific to this repository. `AGENTS.md` is for agents. It is a list of facts and commands, kept under about 120 lines, that links to the policy instead of repeating it. + +Alternative: one file. It would mix rules for people with command lists for agents, and the agent part would grow into a second CONTRIBUTING.md. + +### D2. `CLAUDE.md` imports `AGENTS.md` + +`CLAUDE.md` contains `@AGENTS.md` and nothing else. Claude Code expands the import. Codex, Copilot, Cursor and OpenCode read `AGENTS.md` directly. One text serves every agent we support. + +Alternatives: +- A symlink. Plugins avoid symlinks, and Windows checkouts handle them badly. +- A copy. It would drift. + +### D3. The RobotCode-first rule and its capability map live in `AGENTS.md`; the policy states the rule + +The rule is stated in full in both documents: it is a policy, and agents need it in context. The capability map (need → `robotcode` command → guide page link) appears only in `AGENTS.md`, because only agents act on it. It links the guide pages rather than copying their options, so it stays correct when RobotCode adds flags. + +The rule says what "re-implement" means in practice: our own code that parses Robot Framework files to resolve keywords, runs `robot` and parses `output.xml` for results, or builds libdoc output to look up keywords, when a `robotcode` command gives the same answer. Custom code that calls `robotcode` and adds project-specific logic is fine. + +### D4. The guide is checked against CI, not reviewed + +`tests/test_contributor_guidance.py`: +- **Section parsing.** It reads the "Checks" section of `AGENTS.md` (one command per bullet, in backticks) and asserts that each command string occurs in `.github/workflows/ci.yml`, ignoring whitespace differences. +- **Paths.** It asserts that every `scripts/` named anywhere in `AGENTS.md` exists. +- **`CLAUDE.md`.** It asserts the file's only non-blank line is `@AGENTS.md`. +- **Templates.** It asserts the PR template has an "AI assistance" heading and the RobotCode checklist item, and that the issue template mentions disclosure. + +When `lean-governance` or `repo-cleanup` removes a script or a CI step, this test makes them update `AGENTS.md` in the same PR. + +### D5. OpenSpec context and rules + +`openspec/config.yaml` gets a `context` block of about 20 lines: +- what the repository ships; +- the top-level layout; +- the RobotCode-first rule in two sentences; +- "write for people in the Robot Framework Voice; see CONTRIBUTING.md". + +It gets two rules, one for `proposal` and one for `tasks`, as listed in the proposal. The context is plain facts, so it doesn't go stale faster than `AGENTS.md`; the test in D4 doesn't check it, because OpenSpec reads it only when a change is drafted. + +### D6. CONTRIBUTING.md keeps its required sections and drops duplicates + +`programme-documentation` requires a "Versions" section in `CONTRIBUTING.md`, so it stays. +- The "Review" section shrinks to two sentences and a link to `GOVERNANCE.md`, which `lean-governance` rewrites. +- The layout block gains `legacy/` (the one-shot `rf-agentskills` forwarder) and `openspec/`, and says that `bundles/rf-agentskills/` is generated from the plugins listed in `plugins/rf-agentskills/catalog.yaml`. +- A new first paragraph links `AGENTS.md` for agents and `AI_POLICY.md` for anyone using AI tools. + +## Risks / Trade-offs + +- [The guide drifts on things the test can't see, such as the layout or the OpenSpec steps] → It's kept short, and `repo-cleanup` removes the material that changes most often. A stale line costs a failed CI run, not a wrong release. +- [The RobotCode-first rule blocks a contribution where `robotcode` is close but not good enough] → The rule allows a custom script that names the upstream issue it bridges, and maintainers decide. +- [Disclosure can't be enforced] → The template asks for it every time, and the policy lets maintainers close low-context PRs. As at RobotCode, the aim is honest contributors, not detection. +- [`@AGENTS.md` import syntax changes in Claude Code] → The test pins the one-line form; a change shows up as a failing test, not a silent gap. diff --git a/openspec/changes/ai-contributor-docs/proposal.md b/openspec/changes/ai-contributor-docs/proposal.md new file mode 100644 index 0000000..c5f25ae --- /dev/null +++ b/openspec/changes/ai-contributor-docs/proposal.md @@ -0,0 +1,68 @@ +## Why + +Much of the work on this repository is done with coding agents, and the content it ships (skills, hooks, subagents, scripts) is read and run by other people's agents. Yet nothing tells a contributor, human or AI, how AI-assisted work is accepted, which local checks to run, which files are generated, or which tool to build on for Robot Framework capabilities. `openspec/config.yaml` gives proposals no project context either. The result is AI-written changes that re-implement what RobotCode already does, hand-edit generated files, or fail CI on checks nobody ran (PR #20 failed a repository check that was never run locally). + +## What Changes + +- **`AI_POLICY.md`** (new), adapted from RobotCode's AI policy: + - from RobotCode's policy: a human owns, understands and tests every change; AI use is disclosed in the PR or issue, with an optional `AI-Assisted-By:` commit trailer; the contributor confirms the right to submit under Apache-2.0; low-context automated submissions are declined; maintainers decide; + - specific to this repository: + - skill, hook, subagent and script content is security-relevant, because other people's agents read and run it; + - every keyword a skill teaches must resolve (`scripts/check-skill-keywords.py`); + - generated files are never edited by hand: name the command that regenerates them; + - paid eval runs and re-baselines are dispatched by maintainers; + - eval tasks and fixtures read like real user requests and real projects; + - the **RobotCode-first rule** (below). +- **The RobotCode-first rule.** + - The `robotcode` CLI (https://robotcode.io/guides/cli/) is the core of the Robot Framework AI toolstack. + - AI solutions in this repository use it to run, discover, analyse, inspect results, run keywords step by step in the REPL, look up keywords and debug, instead of new custom implementations. + - Gaps go upstream to RobotCode, the central place for such improvements. + - Custom scripts (hooks, skill helpers, eval grading) are fine when they don't re-implement those capabilities. + + The rule comes with a capability map: need → command → guide page. Enforcing it in the repository's own tooling is the separate change `robotcode-first-tooling`. +- **`AGENTS.md`** (new): a short operational guide for coding agents. It covers: + - the layout; + - the exact local checks, the same commands CI runs; + - generated files and how to regenerate them; + - files never to commit; + - the OpenSpec workflow; + - links to `AI_POLICY.md` and the RobotCode-first rule. + + **`CLAUDE.md`** (new) contains only `@AGENTS.md`, so Claude Code loads the same text. +- **PR template.** Add an AI-disclosure section and a checklist item: "Robot Framework capabilities go through `robotcode`; gaps are reported upstream". The issue template gets a disclosure note. +- **`openspec/config.yaml`.** + - Add `context` with the project summary, the layout, the RobotCode-first rule and the docs voice. + - Add `rules`: + - a proposal that touches Robot Framework capabilities says which `robotcode` command it uses, or why none fits; + - every task has a verify clause. +- **`CONTRIBUTING.md`.** + - Fix the layout: add `legacy/` and `openspec/`, and say that `bundles/` is generated from `plugins/rf-agentskills/catalog.yaml`. + - Replace the review-group list with a link to `GOVERNANCE.md`, which the change `lean-governance` rewrites. + - Link `AGENTS.md` and `AI_POLICY.md`. The README links them too. +- **Automated checks.** A repository test asserts: + - `AGENTS.md` and `AI_POLICY.md` exist; + - every script `AGENTS.md` names exists, and every check command in it appears in `.github/workflows/ci.yml`; + - `CLAUDE.md` imports `AGENTS.md`; + - the PR template has the disclosure section and the RobotCode item. + + `scripts/check-docs-voice.py` already covers top-level `*.md` files, so the new documents get the voice check. + +## Capabilities + +### New Capabilities +- `contributor-guidance`: the AI contribution policy, the RobotCode-first rule, the agent guide, the PR template items, the OpenSpec project context, and the checks that keep them consistent with CI. + +### Modified Capabilities + +None. `programme-documentation` (added by the unarchived change `launch-plugin-programme`) keeps its requirements. This change adds its own documents next to them and doesn't move the "Versions" section, which that capability requires in `CONTRIBUTING.md`. + +## Impact + +- **New files:** `AI_POLICY.md`, `AGENTS.md`, `CLAUDE.md`, and a test `tests/test_contributor_guidance.py`. +- **Changed files:** `.github/pull_request_template.md`, `.github/ISSUE_TEMPLATE/plugin-feedback.yml`, `openspec/config.yaml`, `CONTRIBUTING.md` and `README.md`. +- **No change to:** code, plugins, the installer, the release or the evals. +- **Related changes:** + - `repo-cleanup` removes the stale material `AGENTS.md` would otherwise have to warn about. + - `lean-governance` owns the review rules this change links to. + - `robotcode-first-tooling` applies the RobotCode-first rule to the repository's own scripts. +- **Contributors:** a PR with substantial AI help needs a disclosure line. Small fixes don't. diff --git a/openspec/changes/ai-contributor-docs/specs/contributor-guidance/spec.md b/openspec/changes/ai-contributor-docs/specs/contributor-guidance/spec.md new file mode 100644 index 0000000..01554b4 --- /dev/null +++ b/openspec/changes/ai-contributor-docs/specs/contributor-guidance/spec.md @@ -0,0 +1,76 @@ +## ADDED Requirements + +### Requirement: AI contribution policy + +The repository SHALL contain `AI_POLICY.md`, linked from `README.md`, `CONTRIBUTING.md` and `AGENTS.md`. It SHALL state that: +- AI tools and coding agents are welcome, and the human who submits a contribution is responsible for it: they understand it, have reviewed all generated output, have run the documented checks, and handle review feedback themselves; +- a pull request, issue or comment made with substantial AI help says so where reviewers see it, with an optional `AI-Assisted-By: ` commit trailer; grammar fixes, formatting and ordinary editor completions need no disclosure; +- by submitting, the contributor confirms the right to submit the content under the Apache-2.0 licence, and that generated content with unclear origin or copied third-party material is disclosed or left out; +- skills, hooks, subagents and scripts are read and run by other people's agents, so changes to them are security-relevant and go through the security gate; +- every keyword a skill teaches resolves against the library it names, as `scripts/check-skill-keywords.py` checks; +- generated files (`bundles/`, `docs/catalog.md`, `.claude-plugin/marketplace.json`, `plugins/*/variants/`, eval baselines) are never edited by hand, and a change to them names the command that regenerated them; +- paid live eval runs and re-baselines are dispatched by maintainers; +- eval tasks and fixtures read like real user requests and real projects, and don't hint at the grading; +- low-context, fully automated or unverified submissions may be closed, and maintainers decide how the policy applies. + +#### Scenario: Disclosure asked for +- **WHEN** a contributor opens a PR whose code an agent wrote +- **THEN** the PR template asks for the disclosure, and `AI_POLICY.md` says what it confirms + +#### Scenario: Generated file edited by hand +- **WHEN** a contributor asks an agent that follows `AGENTS.md` to change a skill that the bundle copies +- **THEN** the guide tells it to edit `plugins//` and run `python scripts/build-catalog.py`, not to edit `bundles/` + +### Requirement: RobotCode is the base of the Robot Framework AI toolstack + +`AI_POLICY.md` and `AGENTS.md` SHALL state the RobotCode-first rule: +- the `robotcode` CLI (https://robotcode.io/guides/cli/) is the core of the Robot Framework AI toolstack; +- AI solutions in this repository use it for running tests, discovery, static analysis, result analysis, running keywords step by step in the REPL, keyword and library documentation, and debugging, in preference to new custom implementations of those capabilities; +- a capability RobotCode lacks is reported or contributed upstream to RobotCode, the central place for such improvements, rather than worked around here; +- custom scripts (hook glue, skill helpers, eval grading, project conventions) are welcome when they call `robotcode` for these capabilities instead of re-implementing them. + +`AGENTS.md` SHALL contain a capability map that names, for each capability, the `robotcode` command and the guide page that documents it: `robot`/`run`, `discover`, `analyze code`, `results`, `repl`, `robot-debug`, `doc`, `--wrapper`, and `config`/`profiles`. + +#### Scenario: Agent needs keyword documentation +- **WHEN** an agent working on a skill needs the arguments of a library keyword +- **THEN** `AGENTS.md` points it to `robotcode doc keyword`, not to a custom libdoc script + +#### Scenario: Missing capability +- **WHEN** a contributor finds that `robotcode` can't do something a skill needs +- **THEN** the policy tells them to open an issue or PR at RobotCode, and a custom script for it states in a comment which upstream issue it covers + +### Requirement: Agent guide matches CI + +The repository SHALL contain `AGENTS.md` with the layout, the local checks, the generated files with their regenerate commands, the files never to commit, and the OpenSpec workflow. `CLAUDE.md` SHALL consist of an import of `AGENTS.md`, so both agents read one text. A repository test SHALL fail when: +- `AGENTS.md`, `AI_POLICY.md` or `CLAUDE.md` is missing; +- a `scripts/…` path named in `AGENTS.md` doesn't exist; +- a check command listed in the "Checks" section of `AGENTS.md` doesn't appear in `.github/workflows/ci.yml`; +- `CLAUDE.md` doesn't import `AGENTS.md`. + +`AGENTS.md`, `AI_POLICY.md` and `CLAUDE.md` SHALL pass the docs voice check. + +#### Scenario: Script renamed +- **WHEN** a PR renames `scripts/check-drift.sh` and doesn't update `AGENTS.md` +- **THEN** the repository test fails and names the missing path + +#### Scenario: Check dropped from CI +- **WHEN** a PR removes `python scripts/check-docs-voice.py` from `ci.yml` but `AGENTS.md` still lists it +- **THEN** the repository test fails and names the command + +### Requirement: PR template asks about AI use and RobotCode + +`.github/pull_request_template.md` SHALL have an "AI assistance" section that asks for the disclosure or "none", and a checklist item stating that Robot Framework capabilities go through `robotcode` and that gaps are reported upstream. The issue template SHALL ask reporters to disclose AI-generated reports. A repository test SHALL check that both are present. + +#### Scenario: Template item removed +- **WHEN** a PR deletes the RobotCode item from the PR template +- **THEN** the repository test fails + +### Requirement: OpenSpec proposals get project context + +`openspec/config.yaml` SHALL set `context` with the project summary (marketplace plugins, the installer, the eval harness, security scanning), the layout, the RobotCode-first rule and the docs voice. It SHALL also set `rules`: +- a proposal that touches a Robot Framework capability names the `robotcode` command it uses, or says why none fits; +- every task has a verify clause. + +#### Scenario: Proposal drafted by an agent +- **WHEN** an agent runs `openspec instructions proposal` for a new change +- **THEN** the returned context contains the RobotCode-first rule and the proposal rule diff --git a/openspec/changes/ai-contributor-docs/tasks.md b/openspec/changes/ai-contributor-docs/tasks.md new file mode 100644 index 0000000..bd00bd5 --- /dev/null +++ b/openspec/changes/ai-contributor-docs/tasks.md @@ -0,0 +1,31 @@ +## 1. Policy (D1, D3) + +- [ ] 1.1 Write `AI_POLICY.md`, following the sections of RobotCode's policy (human responsibility, disclosure, licensing and right to submit under Apache-2.0, issues and comments, pull requests, low-context submissions, enforcement), plus a section "In this repository" with the security, keyword, generated-file, eval-dispatch and realistic-eval rules, and the RobotCode-first rule; verify `python scripts/check-docs-voice.py` passes and every rule in the spec's "AI contribution policy" requirement appears in it. + +## 2. Agent guide (D1–D3) + +- [ ] 2.1 Write `AGENTS.md` (at most about 120 lines) with these sections: Layout, Checks (one backticked command per bullet, copied from `ci.yml`), Generated files (each with its regenerate command), Never commit, OpenSpec workflow, RobotCode first (the rule plus the capability map, linking each guide page under https://robotcode.io/guides/cli/), and Policy (link). Verify every command in Checks runs locally on a clean checkout after `uv sync --all-extras`. +- [ ] 2.2 Add `CLAUDE.md` containing only `@AGENTS.md`; verify Claude Code shows the `AGENTS.md` content in `/memory` when started at the repository root. + +## 3. Templates and OpenSpec (D5) + +- [ ] 3.1 Add an "AI assistance" section to `.github/pull_request_template.md` (disclosure, or "none") and the checklist item "Robot Framework capabilities go through `robotcode`; gaps are reported upstream"; add a disclosure note to `.github/ISSUE_TEMPLATE/plugin-feedback.yml`. Verify the issue form still renders: `python -c "import yaml,sys; yaml.safe_load(open(sys.argv[1]))" .github/ISSUE_TEMPLATE/plugin-feedback.yml`. +- [ ] 3.2 Add `context` and `rules` (proposal: name the `robotcode` command or say why none fits; tasks: every task has a verify clause) to `openspec/config.yaml`; verify `openspec instructions proposal --change ai-contributor-docs --json` returns them. + +## 4. Contributor docs (D6) + +- [ ] 4.1 In `CONTRIBUTING.md`, add the opening links to `AGENTS.md` and `AI_POLICY.md`, replace the review-group list with a link to `GOVERNANCE.md`, and fix the layout block (`legacy/`, `openspec/`, how `bundles/` is generated). Keep "Versions". Verify `python scripts/check-docs-voice.py` passes and `tests/test_repository_references.py` passes. +- [ ] 4.2 Link `AGENTS.md` and `AI_POLICY.md` from `README.md` (contributing section); verify the docs voice check passes. + +## 5. Checks (D4) + +- [ ] 5.1 Add `tests/test_contributor_guidance.py` for the "Agent guide matches CI" and "PR template asks about AI use and RobotCode" requirements; verify it passes, and fails when (a) a `scripts/` path in `AGENTS.md` is renamed, (b) a Checks command is missing from `ci.yml`, (c) `CLAUDE.md` no longer imports `AGENTS.md`, (d) the RobotCode item is removed from the PR template (try each by temporary edit, then revert). +- [ ] 5.2 Run `python -m pytest tests/ --ignore=tests/eval`, `python scripts/build-catalog.py --check`, `bash scripts/check-drift.sh` and `python scripts/check-docs-voice.py`; verify all pass. + +## 6. PR + +- [ ] 6.1 Open a PR against `main` (ask before posting) with an AI-assistance disclosure in its description; verify the required checks pass. + +## 7. Initial check + +- [ ] 7.1 After merge, check the new policy and agent guide in use: one full CI run on `main`, one local run of the checks in `AGENTS.md` or CONTRIBUTING, and the next contributor PR. Write down anything that broke or got harder. If a decision in `design.md` turns out wrong or risky, propose the change to the maintainer and update `design.md`. Verify that the findings are recorded in this change. diff --git a/openspec/changes/lean-governance/.openspec.yaml b/openspec/changes/lean-governance/.openspec.yaml new file mode 100644 index 0000000..90080a9 --- /dev/null +++ b/openspec/changes/lean-governance/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-11-16 diff --git a/openspec/changes/lean-governance/design.md b/openspec/changes/lean-governance/design.md new file mode 100644 index 0000000..5ec2f93 --- /dev/null +++ b/openspec/changes/lean-governance/design.md @@ -0,0 +1,150 @@ +## Context + +See proposal.md (Why). The state on 2026-11-16 that shapes the approach: + +- **Ruleset `main` (id 26891587, active)** has these rules: + - deletion and non-fast-forward; + - a `pull_request` rule with `required_approving_review_count: 0` and `require_code_owner_review: false`; + - 12 required checks, including `required-reviews` and `Security gate result`. + + Its only bypass actor is the Admin repository role in `pull_request` mode. No classic branch protection. +- **Six people have the Admin role**: emanlove, Hi-Fi, aaltat, Noordsestern, Simakvokka and manykarim. Ten more have Read. The teams `robotframework-plugins-reviewers` and `-security` return 404, and `REVIEW_TEAMS_TOKEN` isn't set, so `required-reviews.yml` reads empty team lists. +- **Repository settings:** + - `allow_auto_merge` is false; + - secret scanning, push protection and Dependabot security updates are disabled; + - there is no `.github/dependabot.yml`. +- **The `required-reviews` machinery** is `required-reviews.yml` (133 lines), `review-relay.yml`, `scripts/required-reviews.py` (155 lines), `.github/governance.yaml` (teams, 26 security paths, `tier_grace` until 2027-01-31) and the test cases and fixtures in `tests/test_governance.py`. +- **`check-governance.py`** has two parts: + - CODEOWNERS ↔ `catalog.yaml` agreement; + - verified evidence, with grace by date. + + It runs in `ci.yml` (Lint job) and in `scripts/check-template.sh`. +- **External contributors:** #11–#18 come from one external contributor. Two of them add community plugins (`rf-mind`, `rf-maestro`). + +## Goals / Non-Goals + +**Goals:** +- GitHub enforces review natively; no workflow computes approvals. +- A maintainer PR needs no second human. An external PR needs exactly one. +- Nothing in governance fails because the calendar moved. + +**Non-Goals:** +- Changing the security gate's scanners, rules or baselines themselves. +- Changing the lifecycle rules (deprecation, removal, demotion) or the maintainer response targets in GOVERNANCE.md. +- Dependabot version updates and auto-merging bot PRs. There are no bot PRs today; that can be added when there are (see Open Questions). +- Creating any GitHub team. The maintainers are the repository roles. + +## Decisions + +### D1 Maintainers are a repository role, not a team + +The maintainers are the people with the Maintain or Admin role. Rulesets can name roles as bypass actors directly, and CODEOWNERS needs owners with write access anyway. That leaves no team to create, no token to read members, and no list to keep in sync. + +`CODEOWNERS` becomes a single default line, `* @manykarim`, plus any other maintainer who wants review requests. It names people because CODEOWNERS can't name a role. + +*Alternative: a `robotframework-plugins-maintainers` team.* It's tidier once there are several active maintainers, but it needs an org owner to create it. Kept as an option for the maintainer step (task 4.1); CODEOWNERS can switch to it in one line. + +### D2 Two rulesets, so checks can't be bypassed while review can + +A ruleset's bypass list applies to all of its rules. One ruleset therefore can't let maintainers skip the approval while still forcing them through the checks. The split: + +| Ruleset | Rules | Bypass | +|---|---|---| +| `main-checks` | required status checks (today's list minus `required-reviews`), no deletion, no force push | none | +| `main-review` | pull request: 1 approval, code owner review, dismiss stale approvals | Maintain and Admin roles, `pull_request` mode | + +In `pull_request` mode, a maintainer can merge their own green PR with "bypass rules" on the PR page, but can't push to `main` directly. Turning on repository auto-merge lets a maintainer arm "merge when ready", so nobody has to come back to press the button. + +*Alternative: keep `required_approving_review_count: 0` and rely on the security gate alone.* That would let an external PR merge with no human looking at it. One approval for outsiders is the minimum that still stops a hostile plugin from landing on green checks. + +### D3 Remove `required-reviews`, don't slim it + +Once D2 is in place, the check has nothing left to decide: the plugin-owner, Foundation and security groups all go away. The workflows, the script, `governance.yaml` and the fixtures are deleted. + +Order matters. `required-reviews` must leave the required checks (task 4.2) before the workflow is deleted, otherwise every open PR waits forever for a status nobody sets. + +### D4 Tier evidence only; no grace, no dates + +`check-governance.py` keeps the tier evidence: trigger sets, narrow tasks and baselines for every skill, and a bundle being verified only if all its members are. It drops: +- the maintainer count; +- Foundation membership; +- `tier_grace`, `--today` and `--team-members`; +- the CODEOWNERS ↔ `catalog.yaml` agreement, which D1 makes meaningless because plugin maintainers no longer appear in CODEOWNERS. + +The security gate and the keyword check, which `verified` now names, are already required checks on every PR, so the tier check doesn't run them again. It only has to show that they exist for the plugin, which they do for every plugin. + +`catalog.yaml` keeps `maintainers` (at least one; the catalogue check already refuses an active plugin with nobody listed), because users and the lifecycle rules need a named person. + +### D5 Baseline expiry stays, but warns ahead + +Expiry is the one date-driven failure that is deliberate: an accepted finding gets looked at again within 12 months. To keep it from surprising an unrelated PR, `compare-findings.py` gets a `--warn-days 30` window: +- in PR runs, an entry inside the window is a warning in the summary; +- with `--fail-on-warn`, which the scheduled `security-gate.yml` run on `main` passes, it is a failure. + +A maintainer therefore sees the nightly failure up to 30 days before any PR would. + +### D6 Turn on GitHub's free protections + +For a public repository, secret scanning, push protection and Dependabot security updates cost nothing and need no review. Push protection catches a leaked key before it reaches a PR, earlier than the PR gate's Gitleaks step, which stays. Dependabot security updates only open PRs when an advisory hits a locked dependency. Those PRs go through the same checks and a maintainer arms auto-merge. + +### D7 The PR template asks only what checks can't answer + +The template has three parts: +- **What this changes** (free text); +- **Checked by CI**: a plain list with no checkboxes. It covers the security gate, skill validation and links, `.robot` dry runs, keyword check, catalogue and docs voice; +- **For the reviewer**: "Anything the checks can't see, such as a behaviour change, a new tool permission, or a judgement call?" + +Change B adds an AI disclosure line under the free text. Both changes edit only their own sections, so whichever is applied second rebases cleanly. + +### D8 Reconcile `launch-plugin-programme` here + +The launch change can't be archived with 15 open tasks, and several of them are about the review model this change replaces. Evidence gathered on 2026-11-16: + +| Launch task | State | Evidence | Action | +|---|---|---|---| +| 1.3 tags | done | `v1.1.0`, `v1.2.0`, `v3.0.0`, `rf-agentskills-v0.3.0`…`v0.8.0` (incl. rc) exist in the new repo | tick, after the commit comparison in task 1.1 | +| 5.4 security gate | done | `security-gate.yml` runs on `pull_request` (forks included, #11–#18), nightly cron; `Security gate result` is a required check | tick | +| 5.5 deep scan | built, not verified | workflow exists; the last 3 push runs on `main` failed | carry over as task 5.1 here | +| 5.6 live evals not on forks | done | `skill-evaluation.yml` treats a fork PR as having no credentials (comment at line 175; "Evaluation not run (no credentials)" job) | tick | +| 6.1 two teams | superseded | replaced by D1/D2 | mark superseded | +| 6.4 template and protection | done, then changed here | PR template exists; ruleset `main` requires the checks | tick, noting that D2 and D7 replace both | +| 8.1 adoption data | built, not verified | `adoption-collect.yml` exists; no run and no `adoption-data` branch yet | carry over | +| 9.2, 9.3, 9.4 docs and template | done | README, CONTRIBUTING, GOVERNANCE and SECURITY exist and pass the voice check; `Template plugin passes the PR checks` is a required check | tick | +| 10.1 trusted publishing | done | `pypi` and `testpypi` environments; `robotframework-plugins` 2026.11.0.dev1 on PyPI from `release.yml` | tick | +| 10.2–10.4 releases and switch-over | open | no `--v*` tags; `rf-agentskills` on PyPI is 0.8.0; the old repo isn't archived | carry over | +| 10.5 archive | — | done by task 1.3 here | tick when archived | + +The carried-over tasks aren't about governance. They move here only so the launch change can close; their wording doesn't change. + +## Risks / Trade-offs + +- **[A maintainer merges a harmful change of their own on green checks]** → The checks (house rules, scanners, Gitleaks, push protection) are the same for everyone. The bypass is logged on the PR. With one active maintainer, a second human approval was never actually happening anyway. +- **[An external PR gets one approval from a maintainer who skims it]** → The security gate must pass too. The template's "for the reviewer" question points at what to read by hand, and the `security-deep-scan` label stays available. +- **[Plugin maintainers from outside lose their say over changes to their plugin]** → They never had an enforceable say, because GitHub doesn't count approvals from people without write access. CONTRIBUTING asks authors to mention a plugin's maintainers in the PR, and maintainers can be given the Write role later. +- **[Removing the `required-reviews` status before the rulesets exist]** → task 4 applies the rulesets first, and task 3 deletes the workflow after that. +- **[Dependabot PRs add noise]** → Security updates only, no version updates. + +## Migration Plan + +1. Archive `launch-plugin-programme` after reconciling it (task 1), so the deltas in this change have main specs to apply to. +2. Prepare the code and doc changes (tasks 2–3) on a branch and open the PR. `required-reviews` runs base-branch code, so it keeps waiting for the security group on this PR (it touches workflows); that is expected. +3. Maintainer step (task 4), before step 2's PR merges: + - create `main-checks` and `main-review`; + - delete the old `main` ruleset; + - turn on auto-merge, secret scanning, push protection and Dependabot security updates. +4. Merge the PR. The workflows are gone and the rulesets enforce review. +5. Dry run (task 5): a maintainer PR merges with no approval; a test PR from a fork can't merge until one approval; a failing check blocks both. + +**Rollback:** restore the old ruleset from its JSON (saved in task 4.1) and revert the merge commit. + +## Confirmed decisions (maintainer, 2026-11-16) + +Each decision stands for now. After implementation, the check in task 6.1 confirms it. If that check, or anything found along the way, shows a problem or a risk, the decision can change again. In that case, update this design before going on. + +- **D1:** maintainers are the Maintain and Admin repository roles, with `* @manykarim` in CODEOWNERS. No team. +- **D5:** baseline expiry is the one check that may fail because of a date. The nightly scan warns 30 days ahead. +- **D6:** turn on secret scanning, push protection, Dependabot security updates and repository auto-merge. Auto-merge for bot PRs stays open (see below). + +## Open Questions + +- Should Dependabot version updates, and auto-merge for their patch bumps, be added later? That would be its own small change once version drift becomes a chore; it doesn't affect this design. diff --git a/openspec/changes/lean-governance/proposal.md b/openspec/changes/lean-governance/proposal.md new file mode 100644 index 0000000..a1af3b0 --- /dev/null +++ b/openspec/changes/lean-governance/proposal.md @@ -0,0 +1,64 @@ +## Why + +The review model from `launch-plugin-programme` was designed for a Foundation programme with several reviewers. In practice the repository has one active maintainer and no review teams, and the model already gets in the way: +- The Foundation reviewers and security reviewers teams don't exist, or can't be read: the team API returns 404, and the `REVIEW_TEAMS_TOKEN` secret the check needs isn't set. A PR that needs either group can never pass `required-reviews`, so every maintainer PR that touches a plugin is merged with the admin bypass. +- External contributors are already here: eight open PRs (#11–#18), two of which add community plugins. Under the current rules a new plugin's only "owner" is its author, whose approval doesn't count, so those PRs can't pass without a bypass either. +- `check-governance.py` fails every PR on 2027-01-31, when the `tier_grace` entry for "second maintainer, Foundation maintainer" runs out. + +The maintainer's direction: review and approval should be as lean as needed, and most checks should be automated, with little to no human intervention. + +## What Changes + +- **One maintainers group.** The repository's people with the Maintain or Admin role are the maintainers. The Foundation reviewers team, the security reviewers team and the `foundation`/`security` groups go away. +- **Native enforcement.** Two GitHub rulesets on `main` replace the custom review check: + - **checks**: the required status checks, no bypass for anyone; + - **review**: one approval from a code owner, which maintainers may bypass when they merge through a PR. + + So a maintainer's own PR merges once all automated checks pass, and an external PR needs one maintainer approval. Repository auto-merge is enabled, so a PR can be set to merge itself when it goes green. +- **BREAKING (contributors):** `.github/workflows/required-reviews.yml`, `.github/workflows/review-relay.yml`, `scripts/required-reviews.py`, its tests and fixtures, and the `required-reviews` status are removed. `.github/governance.yaml` is removed; `CODEOWNERS` gets a default owner (the maintainers) and drops the per-plugin team lines. +- **Security by automation, not a second team.** Security-relevant paths are guarded by the security gate, which is already a required check: house content rules, scanners, Gitleaks and the reviewed baseline. GitHub secret scanning, push protection and Dependabot security updates, all disabled today, are turned on. The `security-deep-scan` label stays. +- **Tiers from evidence, not head counts.** `verified` requires the automated evidence only: trigger sets, eval tasks, baselines on the current models, a passing eval gate, a passing security gate and keyword check. The "two maintainers, one from the Foundation team" rule and `tier_grace` are dropped. No check fails because a date has passed, except the security baseline's expiry, which the nightly scan now flags 30 days ahead. +- **A short PR template.** The six human checkboxes become a list of what the checks cover plus two questions the checks can't answer. Change B (`ai-contributor-docs`) adds the AI disclosure line to the same template. +- **Docs.** GOVERNANCE.md, the CONTRIBUTING "Review" section, the README "Trust tiers" section and SECURITY.md describe the lean model. +- **Reconcile `launch-plugin-programme`** so it can be archived. Tick the tasks that are done, with evidence, mark 6.1 (create two teams) as superseded, and carry the release and switch-over tasks that remain (8.1, 10.2–10.4) into this change. + +## Capabilities + +### New Capabilities + +None. + +### Modified Capabilities + +- `plugin-trust-tiers`: + - verified criteria lose the maintainer counts and team membership; + - review becomes one maintainer approval for external PRs, with maintainers merging their own PRs on green checks; + - the PR template lists the automated checks; + - tier results don't depend on the date. +- `content-security-scanning`: + - security-relevant content no longer needs a security team approval; + - baseline changes need no security team; + - the nightly scan fails 30 days before a baseline entry expires; + - repository secret scanning and push protection are on. + +Both capabilities are added by the unarchived change `launch-plugin-programme`; this change modifies its requirements. `launch-plugin-programme` is archived first (task 1), so the deltas here apply to the synced main specs. + +## Impact + +- **Workflows and scripts:** + - removed: `required-reviews.yml`, `review-relay.yml`, `scripts/required-reviews.py`; + - changed: `scripts/check-governance.py` (tier evidence only; no `--today`, no `tier_grace`), `scripts/compare-findings.py` (30-day warning; nightly mode fails on it), `security-gate.yml` (nightly mode flag), `scripts/check-template.sh`. +- **Config:** + - removed: `.github/governance.yaml`; + - changed: `.github/CODEOWNERS`, `.github/pull_request_template.md`. +- **Tests:** `tests/test_governance.py` (the required-review cases are removed, the tier cases kept), `tests/fixtures/required-reviews/` (removed), and `tests/test_compare_findings.py`. +- **GitHub settings (maintainer step, asked before applying):** + - ruleset `main` (id 26891587) is split into "checks" and "review"; + - `required-reviews` is removed from the required checks; + - auto-merge, secret scanning, push protection and Dependabot security updates are enabled; + - optionally a `robotframework-plugins-maintainers` team. +- **Docs:** GOVERNANCE.md, CONTRIBUTING.md, README.md ("Trust tiers"), SECURITY.md, RELEASING.md (nothing about review there today; checked). +- **Open PRs #11–#18:** after the switch they need one maintainer approval and green checks, with no `required-reviews` status. +- **Related changes:** + - `ai-contributor-docs` edits the same PR template and CONTRIBUTING.md. Apply this change first, or rebase B's template edit on it. + - `repo-cleanup` doesn't touch these files. diff --git a/openspec/changes/lean-governance/specs/content-security-scanning/spec.md b/openspec/changes/lean-governance/specs/content-security-scanning/spec.md new file mode 100644 index 0000000..7420643 --- /dev/null +++ b/openspec/changes/lean-governance/specs/content-security-scanning/spec.md @@ -0,0 +1,35 @@ +## MODIFIED Requirements + +### Requirement: Every content type is in scope + +The security checks SHALL cover these content types in every plugin: `SKILL.md` files, skill references and assets, subagent definitions in every agent format (Markdown and the Codex TOML variant), hook configuration and hook scripts, helper scripts, MCP server configuration, and plugin and marketplace manifests. Hooks, scripts, MCP configuration, subagent `tools` lists, skill `allowed-tools` and manifests SHALL count as security-relevant content. A change to them SHALL pass the security gate like any other change; it SHALL NOT need an approval from a separate security team. + +#### Scenario: A subagent is scanned +- **WHEN** a PR changes only `plugins//agents/.md` +- **THEN** the PR gate scans that file for prompt injection, credential access and data exfiltration patterns + +### Requirement: The gate fails on new findings only + +Each plugin SHALL have a reviewed baseline of accepted findings, at `security/baselines/.yaml`. Each entry SHALL record the scanner, the rule, the file, a reason and an expiry date no more than 12 months away. The gate SHALL fail on any HIGH or CRITICAL finding that is not in the baseline, and on any baseline entry that has expired. It SHALL report MEDIUM and lower findings without failing. The PR gate SHALL list entries that expire within 30 days as warnings. The scheduled scan of `main` SHALL fail on them, so that a maintainer renews or removes an entry before it blocks a PR. + +#### Scenario: Known false positive does not fail +- **WHEN** a skill's reference warns against `sudo pip install` and SkillSpector flags it, and the finding is in the baseline with a reason +- **THEN** the gate passes and lists the finding as accepted + +#### Scenario: Expired baseline entry +- **WHEN** a baseline entry's expiry date has passed +- **THEN** the gate fails until the entry is renewed with a new review or removed + +#### Scenario: Expiry is flagged ahead +- **WHEN** a baseline entry expires in 20 days +- **THEN** the PR gate passes with a warning naming the entry, and the scheduled scan of `main` fails naming it + +## ADDED Requirements + +### Requirement: Repository-level secret protection + +The repository SHALL have GitHub secret scanning and push protection enabled, so that a push containing a recognised secret is blocked before it lands. Dependabot security updates SHALL be enabled for the repository's dependency manifests. + +#### Scenario: Secret in a push +- **WHEN** someone pushes a commit that contains a recognised API key to any branch of the repository +- **THEN** GitHub blocks the push and names the secret type diff --git a/openspec/changes/lean-governance/specs/plugin-trust-tiers/spec.md b/openspec/changes/lean-governance/specs/plugin-trust-tiers/spec.md new file mode 100644 index 0000000..f8952b1 --- /dev/null +++ b/openspec/changes/lean-governance/specs/plugin-trust-tiers/spec.md @@ -0,0 +1,65 @@ +## MODIFIED Requirements + +### Requirement: Verified tier entry criteria + +A plugin SHALL have tier `verified` only when it meets every community criterion and also: +- ships trigger query sets for each skill and narrow eval tasks in the `rf-skill-eval` harness; +- has committed eval baselines measured on the current models, and passes the eval gate; +- passes the security gate and the library-skill keyword check. + +The tier check SHALL decide from this evidence alone. It SHALL NOT require a number of maintainers or membership of a team. A bundle is `verified` when every plugin it bundles is. + +#### Scenario: Verified without evals is refused +- **WHEN** a PR sets `tier: verified` for a plugin that has no trigger sets +- **THEN** the tier check fails and names the missing evidence + +#### Scenario: One maintainer is enough +- **WHEN** a `verified` plugin has one maintainer and all the evidence above +- **THEN** the tier check passes + +### Requirement: Review follows a published checklist + +Every PR SHALL use a PR template that lists the automated checks the PR must pass and asks the author for what the checks can't show: what the change is for, and anything a reviewer should look at by hand. The template SHALL NOT ask authors or reviewers to tick items that a required check already verifies. + +#### Scenario: New plugin PR shows the template +- **WHEN** a contributor opens a PR that adds a plugin +- **THEN** the PR description lists the required checks and the two questions, and has no checkbox for a property a required check enforces + +## ADDED Requirements + +### Requirement: Review is one maintainer approval, and checks do the rest + +The maintainers of the repository are the people with the Maintain or Admin role on it. `main` SHALL be protected so that: +- every required status check passes before a PR merges, with no bypass for anyone; +- a PR needs one approval from a code owner, and `CODEOWNERS` SHALL name the maintainers as the default owner of every path; +- a maintainer MAY merge their own PR without an approval once every required check passes, through the PR (never by a direct push). + +A plugin's maintainers listed in `catalog.yaml` are the people who answer for it (see `plugin-lifecycle`). Their approval SHALL NOT be required, because GitHub only counts approvals from people with write access. + +#### Scenario: Maintainer PR merges on green checks +- **WHEN** a maintainer opens a PR that edits `plugins/rf-core/hooks/hooks.json` and every required check passes +- **THEN** the maintainer can merge it without another person's approval + +#### Scenario: External PR needs one maintainer +- **WHEN** a contributor without write access opens a PR that adds `plugins/rf-mind/`, and every required check passes +- **THEN** the PR can't merge until one maintainer approves it + +#### Scenario: Failing check blocks everyone +- **WHEN** the security gate fails on a maintainer's PR +- **THEN** the PR can't be merged, by the maintainer or anyone else + +### Requirement: No governance check fails because of the date + +The ownership and tier checks SHALL give the same result for the same repository tree on any date. Exceptions with an end date SHALL NOT exist in the governance configuration. (Security baseline expiry is a separate, deliberate re-review; see `content-security-scanning`.) + +#### Scenario: Same tree, later date +- **WHEN** the tier check runs on the same commit on 2026-11-16 and on 2027-02-01 +- **THEN** both runs give the same result + +## REMOVED Requirements + +### Requirement: Every change is reviewed by an owner of the plugin + +**Reason**: It needs a Foundation reviewers team and a security reviewers team, which don't exist. A custom `required-reviews` check enforces it, and no PR touching a plugin has passed that check without the admin bypass. It also asks for approvals from plugin maintainers who have no write access, which GitHub can't count. + +**Migration**: Replaced by "Review is one maintainer approval, and checks do the rest". The `required-reviews` status, its workflows and `.github/governance.yaml` are removed. The ruleset requires one code-owner approval instead, and maintainers bypass it for their own PRs. Security-relevant paths are guarded by the security gate (`content-security-scanning`). diff --git a/openspec/changes/lean-governance/tasks.md b/openspec/changes/lean-governance/tasks.md new file mode 100644 index 0000000..b700056 --- /dev/null +++ b/openspec/changes/lean-governance/tasks.md @@ -0,0 +1,62 @@ +## 1. Close `launch-plugin-programme` (D8) + +- [ ] 1.1 For launch task 1.3, compare `git rev-parse ^{commit}` in the new repository with `manykarim/robotframework-agentskills` for every listed tag; verify they all match, then tick 1.3 with the result. +- [ ] 1.2 In `openspec/changes/launch-plugin-programme/tasks.md`: + - tick 5.4, 5.6, 6.4, 9.2, 9.3, 9.4 and 10.1, each with its evidence from design D8; + - mark 6.1 "superseded by `lean-governance`"; + - mark 5.5, 8.1 and 10.2–10.4 "carried over to `lean-governance`". + + Verify that `openspec list` shows no unticked task in the change other than 10.5. +- [ ] 1.3 Archive `launch-plugin-programme` with its specs synced (`/opsx:archive`). Verify that `openspec validate --specs --strict` passes and that `openspec validate lean-governance` no longer reports "target spec does not exist". + +## 2. Checks and scripts (D4, D5) + +- [ ] 2.1 Reduce `scripts/check-governance.py` to the tier evidence: drop the maintainer count, Foundation membership, `tier_grace`, `--today`, `--team-members` and the CODEOWNERS agreement. Update `tests/test_governance.py`: keep "Verified without evals is refused", and add "One maintainer is enough" and "Same tree, later date" (run with the clock patched to 2027-02-01). Verify that `pytest tests/test_governance.py` passes and `python scripts/check-governance.py` prints no warning. +- [ ] 2.2 Add `--warn-days` (default 30) and `--fail-on-warn` to `scripts/compare-findings.py`, and pass `--fail-on-warn` from `security-gate.yml` only on `schedule` and `workflow_dispatch` runs. Add tests in `tests/test_compare_findings.py` for "Expiry is flagged ahead": a 20-day entry gives a warning and exit 0, and exit 1 with `--fail-on-warn`. Verify that `pytest tests/test_compare_findings.py` passes. +- [ ] 2.3 Delete `scripts/required-reviews.py`, the required-review tests in `tests/test_governance.py`, `tests/fixtures/required-reviews/` and `.github/governance.yaml`. Remove `.github/governance.yaml` and `scripts/required-reviews.py` from every list that names them (CODEOWNERS, `check_content_rules.py` or `security-scan.sh` path lists, if any). Verify that `grep -rn "required-reviews\|governance.yaml" --exclude-dir=openspec .` only finds the workflow files deleted in 3.1. +- [ ] 2.4 Replace `.github/CODEOWNERS` with a default owner line for the maintainers (D1), and update `scripts/check-template.sh` and `templates/plugin/` if they mention CODEOWNERS lines per plugin. Verify that `bash scripts/check-template.sh` passes. + +## 3. Workflows, template and docs (D3, D7) + +- [ ] 3.1 Delete `.github/workflows/required-reviews.yml` and `.github/workflows/review-relay.yml`, and update the ci.yml comment at the `check-governance.py` step. Verify that `actionlint` (or `gh workflow list` after merge) shows neither workflow. +- [ ] 3.2 Rewrite `.github/pull_request_template.md` as in D7: "What this changes", "Checked by CI" with no checkboxes, and "For the reviewer". Leave a marked spot for the AI disclosure line from `ai-contributor-docs`. Verify that the template names only checks that exist in `ci.yml` and `security-gate.yml`. +- [ ] 3.3 Rewrite GOVERNANCE.md: + - Tiers: evidence only. + - Review: the two rulesets, one approval for external PRs, maintainers merge on green. + - Keep the commitments and the lifecycle as they are. + + Then update: + - the CONTRIBUTING.md "Review" section and step 4 of "Adding a plugin", which drops the CODEOWNERS line and mentions the plugin's maintainers in the PR instead; + - the README "Trust tiers" section; + - the SECURITY.md sentence about security reviewers. + + Verify that `python scripts/check-docs-voice.py` passes, and that `grep -rn "Foundation reviewer\|security reviewer\|required-reviews" README.md CONTRIBUTING.md GOVERNANCE.md SECURITY.md` finds nothing. +- [ ] 3.4 Run `pytest tests --ignore=tests/eval`, `python scripts/build-catalog.py --check` and `bash scripts/check-template.sh`; verify that all of them pass. + +## 4. GitHub settings (maintainer step; ask before each change) + +- [ ] 4.1 Save the current ruleset with `gh api repos/MarketSquare/robotframework-plugins/rulesets/26891587 > ruleset-main-before.json` (scratch, not committed). No team (D1, confirmed 2026-11-16). Verify that the JSON file is saved. +- [ ] 4.2 With the maintainer's approval, create the two rulesets from D2 and delete ruleset 26891587: + - `main-checks`: today's required checks minus `required-reviews`, no deletion, no force push, no bypass; + - `main-review`: 1 approval, code owner review, dismiss stale approvals, bypass for the Maintain and Admin roles in `pull_request` mode. + + Verify with `gh api .../rulesets` that both rulesets are active and that `required-reviews` is no longer a required check. +- [ ] 4.3 Approved on 2026-11-16 (D6); confirm the timing with the maintainer, then enable repository auto-merge, secret scanning, push protection and Dependabot security updates. Verify that `gh api repos/MarketSquare/robotframework-plugins --jq '.allow_auto_merge, .security_and_analysis'` shows all of them enabled. + +## 5. Dry run and carried-over launch tasks + +- [ ] 5.1 Merge the PR with tasks 2–3 through the maintainer bypass. Then verify: + - a maintainer test PR (a docs typo) merges with no approval once its checks pass; + - a test PR from a fork, or from an account without write access, shows "1 approving review required"; + - a test PR with a failing check can't be merged, even with the bypass. + + Close both test PRs without merging. +- [ ] 5.2 (Launch 5.5) Find out why the `security-deep-scan.yml` push runs on `main` fail, fix it, and verify with a labelled test PR that the comment appears and that an unlabelled PR doesn't run the job. +- [ ] 5.3 (Launch 8.1) After the next weekly `adoption-collect.yml` run, verify that the `adoption-data` branch has daily view and clone data; switch to a fine-grained token if `GITHUB_TOKEN` is refused. +- [ ] 5.4 (Launch 10.2) Release `rf-core`, `rf-web`, `rf-api`, `rf-mobile` and `rf-desktop` at 1.0.0, the `rf-agentskills` 4.0.0 bundle and the first CalVer release of `robotframework-plugins`. Verify installs from the marketplace on Claude Code, Copilot CLI and Codex, and `uvx robotframework-plugins install --agent claude-code --yes`, in isolated homes. +- [ ] 5.5 (Launch 10.3) Build and publish the forwarding `rf-agentskills` 0.9.0 from `legacy/rf-agentskills/`, then delete that folder. Verify that `uvx rf-agentskills install --agent claude-code --yes` prints the notice and installs. +- [ ] 5.6 (Launch 10.4) Push the final README notice to `manykarim/robotframework-agentskills` and archive it. Verify that a marketplace add pinned to `v3.0.0` still installs and that the README shows the notice. + +## 6. Initial check + +- [ ] 6.1 After tasks 1–5, review each confirmed decision against what happened. Did any PR get stuck? Did the maintainer bypass work? Did the scanners or push protection block real work? Did the 30-day warning fire? Write down the findings. If one shows a problem or a risk, propose the changed decision to the maintainer and update `design.md` before archiving. Verify that the findings are recorded in this change. diff --git a/openspec/changes/repo-cleanup/.openspec.yaml b/openspec/changes/repo-cleanup/.openspec.yaml new file mode 100644 index 0000000..90080a9 --- /dev/null +++ b/openspec/changes/repo-cleanup/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-11-16 diff --git a/openspec/changes/repo-cleanup/design.md b/openspec/changes/repo-cleanup/design.md new file mode 100644 index 0000000..bed603f --- /dev/null +++ b/openspec/changes/repo-cleanup/design.md @@ -0,0 +1,133 @@ +## Context + +See proposal.md (Why). Facts that shape the approach, checked at `56547f0`: + +- **Required checks** on `main` include "Validate plugin structure", "Check generated variants and plugin layout", "Lint", "Template plugin passes the PR checks" and the six "Test (…)" jobs. "Build Claude Code plugin", "Build portable skills packages" and "Validate SKILL.md frontmatter (agentskills.io)" are not required. +- **Duplicate checks:** + - `ci.yml` runs `scripts/validate-skills.py --channel all` in `validate-plugin` (`:213`) and again in `validate-skills` (`:287`); + - it runs `scripts/build-agent-variants.py --check` in `validate-plugin` (`:211`) and again through `scripts/check-drift.sh:107`. +- **`build-claude-plugin`** (`ci.yml:298-343`): + - its inline Python repeats checks that `validate-skills.py` and `check-drift.sh` already make, except one: the ban on a plugin `.mcp.json` or `servers/`; + - its tarball `rf-agentskills-claude-plugin.tar.gz` has no consumer, because `release.yml` builds its own archives. +- **`build-portable-skills`** (`ci.yml:345-395`) builds Codex and Copilot symlink tarballs. `release.yml` ships only `-skills-.tar.gz` and the skill zips. +- **Double export in the release:** `release.yml:83-85` runs `export-skills.py` and then `build-skill-zips.py --plugin`, which exports again because it doesn't get `--skills`. +- **The `latest` branch:** + - `update-latest.yml` force-pushes `latest` to `main` after every green CI run; + - `release.yml:124-129` moves `stable` and `latest` to each release tag; + - RELEASING.md:30 documents only the release behaviour. +- **The harness stages the merged plugin content as `plugins/rf-agentskills`** (`application/variant.py:29`, `claude_code_runner.py:375`). This is the bundle's shape, built by `staging.merge_plugins`, not a pre-split leftover, so it stays. +- **The plugin MCP path:** + - `claude_code_runner.py:261-268` and `:452-470` merge a staged plugin's `.mcp.json` into the treatment arm, and `:867-871` checks the baseline for leaks of those servers; + - no plugin ships one, and `skill-catalog` "The bundle ships no MCP server" forbids it. +- **`RF_SKILL_EVAL_LOG_LEVEL` stays:** `configure_logging` reads it from the environment (`config.py:66`). Only the `Settings` field that mirrors it is unused. `CLAUDE_MODEL_DEFAULT` has no reader; task YAMLs and `domain/task.py` set the model. +- **Async tests:** there are none in `tests/`, `installer/` or `plugins/*/tests`. `pytest-asyncio` is installed only because both `pyproject.toml` files set `asyncio_mode = "auto"`. +- **The Docker harness** (`scripts/docker-test-harness.sh`, `docs/installer/docker/`) calls the `rf-agentskills` command and `installer/src/rf_agentskills/`. Neither has existed since the rename (`installer/pyproject.toml:69`). Nothing in CI runs it. +- **`tests/test_repository_references.py`** exempts all of `docs/` from the old-name check. Of the documents that would stay outside `docs/archive/`, only ADR-003 (`:65,70`) names the old repository. +- **`scripts/check-docs-voice.py:64`** checks `docs/**/*.md` except `docs/reports/`. + +## Goals / Non-Goals + +**Goals:** +- Every file in the repository is either used, or an archived record that says it is one. +- CI runs each check once, and the required check names don't change. + +**Non-Goals:** +- **The harness ports/service layering** (`application/ports.py`, `evaluation_service.py`). It is a refactor, not a removal, and `robotcode-first-tooling` touches the same code. +- **Keyword resolution and the libdoc specs.** These belong to `robotcode-first-tooling`. +- **Contributor docs and governance:** + - rewriting CONTRIBUTING, README structure or GOVERNANCE belongs to `ai-contributor-docs` and `lean-governance`; + - reconciling `launch-plugin-programme` belongs to `lean-governance`. +- **Things kept on purpose:** + - `bundles/rf-agentskills`: an active marketplace entry. It is a copy because agents don't install dependencies that an update adds. + - `legacy/rf-agentskills`: deleted after 0.9.0 is on PyPI. PyPI is still at 0.8.0, and that release is a manual step. + - The installer's old-marketplace and `rf-agentskills-files` migration: users of 0.x and 3.0.0 still upgrade through it. + - The `rf-tools` cleanup (`cli.py:925-990`): removing it would change behaviour for upgrading users, so it gets its own deprecation. + - The `rf-agentskills/installed.json` record names: renaming them would need a migration. + +## Decisions + +### D1. Archive, don't delete, the dated documents + +Plans, reports, proposals and raw logs move to `docs/archive/` with `git mv`. They keep their names and are flattened only where two would clash. `docs/archive/README.md` holds one line per file: date, what it was, and what replaced it. + +- The archive is excluded from `check-docs-voice.py` (like `docs/reports/`), because nobody edits historical text. +- `tests/test_repository_references.py` exempts `docs/archive/` and `docs/ci/architecture/adr/` instead of `docs/`. ADRs are immutable records; ADR-006 gets one status line saying the Parquet writer was removed. +- Source comments that link to moved files are repointed: + - `installer/pyproject.toml:7`; + - `installer/src/robotframework_plugins/__init__.py:7`, `adapters/codex.py:3`, `adapters/claude_code.py` and `transforms.py:721`; + - the installer test docstrings; + - `plugins/rf-core/hooks/README.md` (the bundle copy follows through `build-catalog.py`); + - `eval/fixtures/README.md`, `plugins/rf-core/evals/tasks/README.md` and the ADRs. + +Files that move: `marketplace-implementation-plan.md`, `skill-architecture-review.md`, `e2e-test-results.md`, `RF_AGENTSKILLS_ISSUE_REPORT.md`, `issues/*`, `installer/proposal.md`, `installer/docker-test-harness-proposal.md`, `plugin/hooks-fix-proposal.md`, `ci/implementation-plan.md`, `ci/rf-agentskills-eval-implementation-plan.md` and `ci/claude-code-extension-quality-evaluation.md`. `ci/architecture/ddd-design.md` ("Draft (v1)") moves too, and `docs/ci/usage.md` stops linking to it. + +Alternative: delete them, since git keeps the history. That's rejected because the installer CHANGELOG and the source comments cite them as the record of why a fix was made, and a link to a commit is harder to follow. + +### D2. Delete the Docker harness + +It has been broken since the rename and nobody noticed, so nothing depends on it. The installer tests (fake agent CLIs, ubuntu and windows matrix) cover the installer logic. + +The harness did one thing nothing else does: it installed into the real agent CLIs. That coverage is lost. If it's wanted back, the right form is a CI job with a current image, built from git history. `installer/CHANGELOG.md` keeps its historical entry. + +Alternative: fix it. That's rejected for now: it needs seven agent CLIs installed in an image, and it would stay outside CI. + +### D3. Harness removals are limited to code with no caller + +A symbol is removed when nothing in `src/`, `tests/`, `scripts/`, `plugins/` or `.github/` imports or calls it, apart from its own unit test: +- `domain/events.py`, `domain/skill.py` and `infrastructure/persistence/parquet_writer.py`; +- `ConfigError`, `PersistenceError` and `VERDICT_STATUSES`; +- `Settings.claude_model_default` and `Settings.rf_skill_eval_log_level`; +- `ClaudeCodeRunner._extra_mcp_servers`, its call in `_mcp_servers` and the plugin-server leak check; +- `CLAUDE_MODEL_DEFAULT` in `.env.example` and `docs/ci/usage.md`. + +The `config_builder.py` docstring stops mentioning `rf-tools`. The test `test_extra_mcp_servers_returns_substituted_servers` goes with its subject. The CLI commands, the run records and the scoring stay as they are. + +### D4. Dependencies follow imports + +- From the harness: drop `scipy`, `numpy` and `httpx` (no import) and `polars` (only the deleted writer), plus their mypy overrides. +- Drop the `validation` extra: it duplicates the dev group, and no document refers to it (the hooks README line was removed in `delegate-robotcode-skills`). +- Drop `pytest-asyncio` and `asyncio_mode = "auto"` from both `pyproject.toml` files and from the `pip install` lines in `ci.yml` (`:65`, `:169`, `:187`), since there are no async tests. +- Drop the sdist `exclude` list, because `include` already limits the sdist to five entries. +- Remove `skills` from the ruff excludes, and rewrite the `tests/test_*.py` comment to say the shared suite is linted by the `lint` job's own command. +- Regenerate `uv.lock` with `uv lock`. + +Alternative: keep `rf-mcp[all]` lighter. Out of scope, because the eval tasks use its library set. + +### D5. One run per check, same required names + +- In `validate-plugin`, the `build-agent-variants.py --check` and `validate-skills.py` steps go. `check-drift` and `validate-skills` run them. +- `build-claude-plugin` goes. Its MCP and `servers/` ban moves into `scripts/check-drift.sh` as a "no plugin MCP server" section, so the required "Check generated variants and plugin layout" check keeps enforcing it. +- In `build-portable-skills`, the Codex and Copilot steps and their uploads go. The standalone tarball and the skill zips stay, because they are what a release ships and CI proves that they build. +- In `release.yml:85`, `build-skill-zips.py` gets `--skills skills`, so it reuses the export from `:83`. + +Alternative: merge `validate-skills` into `validate-plugin`. That's rejected because it would drop a job name that branch protection or contributors may watch, for no gain. + +### D6. `release.yml` owns `latest` + +Delete `update-latest.yml`. After this change, `latest` moves only on a release, as RELEASING.md:30 already says. + +Users who pinned the marketplace to `latest` stop getting unreleased `main` commits. That is the documented behaviour, and the mismatch with it was the bug. README, RELEASING and the installer README never tell users to pin `latest` (checked with `grep -n latest`). + +### D7. Main specs: only what no other active change covers + +`launch-plugin-programme` already removes or rewrites these sync-era requirements: +- library-skill-structure "Restructured skills are distributed"; +- platynui-skill "Skill is integrated into distribution channels"; +- setup-skill "Setup skill is distributed without drift"; +- skill-content-verification; +- library-skill-install-guidance; +- skill-metadata-conformance "Sync propagates skills…"; +- skill-catalog "Distribution tooling rejects orphaned copies" and "Retirements are released as a breaking content change" (the lockstep version). + +This change modifies only the requirements that would still be wrong after that archive (see `specs/`). `skill-eval-harness` "Tiered CI execution" keeps its `skills/rf-selenium/` example: `openrouter-ci-eval-tier` modifies that requirement, so this change doesn't touch it, and that change's archive should fix the example. + +## Risks / Trade-offs + +- [Archiving `launch-plugin-programme` before or after this change collides on the same capability] → The deltas touch disjoint requirements, so either archive order works. Task 6.2 runs `openspec validate --all`. +- [A deleted script or workflow was used outside the repository] → Before deleting, grep the tree and the open PRs. The release notes of the next installer version name the removed scripts. +- [Removing `pytest-asyncio` breaks a future async test] → The test would fail loudly on collection; re-add the plugin then. +- [`uv lock` moves other pins] → Plain `uv lock` keeps the existing pins and only drops packages nothing needs. If the lock diff shows any other version change, investigate it before merging. + +## Migration Plan + +One PR, no release needed, since no shipped artifact changes. Roll back by reverting the PR. Everything removed stays in git history. diff --git a/openspec/changes/repo-cleanup/proposal.md b/openspec/changes/repo-cleanup/proposal.md new file mode 100644 index 0000000..6cd67c4 --- /dev/null +++ b/openspec/changes/repo-cleanup/proposal.md @@ -0,0 +1,51 @@ +## Why + +The repository grew out of the single-plugin `rf-agentskills` repository, and the move left files that nothing runs, dependencies nothing imports, and a few tools that no longer work. New contributors, human or AI, can't tell which parts matter, and the stale parts give wrong instructions. This change removes what is unused and fixes what is broken or conflicting, with no change for users of the plugins or the installer. + +## What Changes + +- **Root config:** delete `.mcp.json` (registers `claude-flow` and an `rf-mcp` command nothing reads). +- **Eval harness dead code:** remove modules, errors, constants and settings that have no caller, and the treatment-arm path that merges a plugin `.mcp.json` (no plugin ships one, and the catalogue forbids it). +- **Dependencies:** drop `scipy`, `numpy`, `httpx` and `polars` from the harness, the duplicate `validation` extra, and stale `pyproject.toml` entries (the sdist exclude list, the `skills` ruff exclude, the "legacy plugin tests" comment). Regenerate `uv.lock`. +- **CI and release:** + - delete `update-latest.yml`, which force-pushes `latest` to `main` while `release.yml` points it at release tags; + - delete the unused `.github/actions/setup-eval`; + - run each check once (`validate-skills.py` and the agent-variants check run twice today); + - stop building the Codex and Copilot symlink tarballs that no release ships; + - stop exporting the skills twice in the release. +- **Scripts:** delete `scripts/validate-marketplace.py` (covered by `build-catalog.py --check` and the marketplace tests) and the broken Docker test harness (`scripts/docker-test-harness.sh`, `docs/installer/docker*`). Fix old names in `build-skill-zips.py`, `build-description-variant.py` and `scripts/templates/opencode-plugin.js`. +- **Docs:** + - move dated plans, one-off reports, proposals and raw logs to `docs/archive/`; + - narrow the old-repository-name exemption in `tests/test_repository_references.py` from all of `docs/` to the archive and the ADRs; + - fix the factual errors in the current guides (`docs/ci/usage.md`, `local-testing.md`, `faq.md`, `plugins/rf-core/hooks/README.md`, the README "Prerequisites" line); + - repoint source comments that link to moved files. +- **Specs:** update main-spec requirements that still require the root `skills/` tree, the sync script or plugin MCP servers. Requirements already covered by the unarchived `launch-plugin-programme` change are left to it. + +Kept on purpose (see design.md): +- the `bundles/rf-agentskills` bundle; +- `legacy/rf-agentskills` until the forwarder release 0.9.0 is on PyPI; +- the installer's migration code for the old marketplace; +- the install-record names. + +## Capabilities + +### New Capabilities + +None. + +### Modified Capabilities + +- `skill-eval-harness`: the treatment arm provisions skills, hooks and subagents; plugin MCP servers are no longer part of it. +- `skill-metadata-conformance`: references use `plugins//skills/rf-/` paths, and the release-package scenario names only the packages a release ships. +- `skill-catalog`: the generator-skill ban lists the channels that exist today. +- `python-library-skill`, `language-skill`, `platynui-skill`, `setup-skill`, `library-skill-structure`: skill locations are `plugins//skills/…`, and distribution goes through the plugin, the bundle copy and the installer assets instead of a sync script. + +## Impact + +- **Code:** `src/rf_skill_eval/` (deletions only; no CLI command changes), `tests/eval/` (tests of removed code go), `tests/test_repository_references.py`. +- **Config:** `pyproject.toml`, `uv.lock`, `.env.example`, `.mcp.json` (deleted). +- **CI:** `.github/workflows/ci.yml`, `release.yml`, `update-latest.yml` (deleted), `.github/actions/setup-eval/` (deleted). The required check names stay the same. +- **Scripts:** `scripts/validate-marketplace.py`, `scripts/docker-test-harness.sh` (deleted); three scripts get comment and name fixes; the regenerated OpenCode variant. +- **Docs:** about 15 files move to `docs/archive/`; `docs/installer/docker*` is deleted; current guides are corrected. +- **Users:** none. Plugins, the bundle, the installer and the release artifacts stay the same, except the unreleased CI-only Codex and Copilot tarballs. +- **Related changes:** `ai-contributor-docs` rewrites the contributor docs (this change only fixes factual errors). `lean-governance` reconciles and archives `launch-plugin-programme`, which fixes the remaining sync-era requirements. diff --git a/openspec/changes/repo-cleanup/specs/language-skill/spec.md b/openspec/changes/repo-cleanup/specs/language-skill/spec.md new file mode 100644 index 0000000..decd3da --- /dev/null +++ b/openspec/changes/repo-cleanup/specs/language-skill/spec.md @@ -0,0 +1,32 @@ +## MODIFIED Requirements + +### Requirement: Skill exists with conformant frontmatter and layout + +The repository SHALL provide the skill at `plugins/rf-core/skills/rf-language/` with: +- a `SKILL.md` whose frontmatter `name` is `rf-language` (equal to the directory name), with `description`, `license`, `compatibility` and `metadata` (`author`, `version`) in the form required for every shipped skill; +- a `references/` directory; +- an `assets/examples/` directory; +- a `scripts/` directory containing `rf_conventions.py`. + +The `compatibility` text SHALL be at most 500 characters and SHALL state that Python and `robotframework>=7` in the project environment are needed for the bundled script, that the guidance is version-gated, and that `robotcode[all]>=2.8` is required in the project environment (it also provides Robocop). The skill SHALL contain only regular files (no symlinks). + +#### Scenario: Directory and frontmatter +- **WHEN** `plugins/rf-core/skills/rf-language/SKILL.md` is parsed +- **THEN** `name` is `rf-language`, `license`, `compatibility` (≤ 500 characters) and `metadata` (`author`, `version`) are present, `scripts/rf_conventions.py` exists, and the skill validator used for all skills reports no violation + +### Requirement: Skill is distributed to every channel without drift + +The skill SHALL ship in the `rf-core` plugin, be copied unchanged into `bundles/rf-agentskills/` by the catalogue build (with its own `scripts/rf_conventions.py`), and be included in the installer assets. `scripts/build-catalog.py --check`, the drift check and the skill validator SHALL pass. In addition, the context-injection hook's skill list SHALL name `rf-language`; the subagents `rf-test-architect`, `rf-debug-expert` and `rf-migration-guide` SHALL direct the agent to `rf-language` (and, for migration, to its `references/migration.md` and the `rf_conventions` report); the `rf-core` README SHALL list the skill. + +#### Scenario: Channels populated +- **WHEN** `scripts/build-catalog.py --check` runs +- **THEN** the bundle copy exists with `name: rf-language` and `scripts/check-drift.sh` exits 0 + +#### Scenario: Installer ships the skill +- **WHEN** the installer is built and installs for Claude Code into a temporary project +- **THEN** `.claude/skills/rf-language/SKILL.md` exists and is recorded in the install manifest + +#### Scenario: Hook and subagents reference the skill +- **WHEN** a prompt mentioning a `.robot` or `.resource` file triggers the context-injection hook +- **THEN** the injected text lists `rf-language` +- **AND** `rf-test-architect.md`, `rf-debug-expert.md` and `rf-migration-guide.md` name `rf-language` diff --git a/openspec/changes/repo-cleanup/specs/library-skill-structure/spec.md b/openspec/changes/repo-cleanup/specs/library-skill-structure/spec.md new file mode 100644 index 0000000..13380c1 --- /dev/null +++ b/openspec/changes/repo-cleanup/specs/library-skill-structure/spec.md @@ -0,0 +1,28 @@ +## MODIFIED Requirements + +### Requirement: Library skills follow the standard SKILL.md skeleton + +Each library skill's `SKILL.md` (`plugins/rf-web/skills/rf-{browser,selenium}/SKILL.md`, `plugins/rf-mobile/skills/rf-appium/SKILL.md` and `plugins/rf-api/skills/rf-{requests,restinstance}/SKILL.md`) SHALL contain these level-2 sections in this order: +1. a short orientation (at most ~5 lines, stating what the library is for and when to prefer a sibling skill); +2. `When to use` (the trigger-terms and sibling-boundary block defined by the `skill-triggering` capability), which is the first level-2 section; +3. `Installation` (the short block defined by the `library-skill-install-guidance` capability); +4. `Import and defaults`; +5. `Which keyword for which situation`; +6. `Agent workflow`; +7. `Gotchas`; +8. `When to read the references`; +9. `Companion Skills`. + +Optional sections (for example a locator strategy or a short pattern) MAY appear between `Which keyword for which situation` and `Gotchas`. + +#### Scenario: Required sections present and ordered +- **WHEN** the structure test parses the level-2 headings of each of the five library `SKILL.md` files +- **THEN** each file contains every required section heading, and the required headings appear in the order given above + +#### Scenario: When to use leads the body +- **WHEN** the level-2 headings of a library `SKILL.md` are parsed +- **THEN** `When to use` is the first level-2 heading and appears within the first 20 body lines, and no other section precedes `Installation` + +#### Scenario: Frontmatter unchanged by this capability +- **WHEN** a library `SKILL.md` is restructured +- **THEN** its frontmatter `name` and `description` stay the same (descriptions are owned by the `skill-triggering` capability) diff --git a/openspec/changes/repo-cleanup/specs/platynui-skill/spec.md b/openspec/changes/repo-cleanup/specs/platynui-skill/spec.md new file mode 100644 index 0000000..19fd27c --- /dev/null +++ b/openspec/changes/repo-cleanup/specs/platynui-skill/spec.md @@ -0,0 +1,14 @@ +## MODIFIED Requirements + +### Requirement: PlatynUI library skill exists and follows the house structure + +The repository SHALL provide a Robot Framework PlatynUI library skill at `plugins/rf-desktop/skills/rf-platynui/`, structured like the other library skills: a `SKILL.md` with valid frontmatter (`name: rf-platynui` matching its directory, and a `description`), a `references/` directory of deep-dive documents, and an `assets/examples/` directory of `.robot` examples. + +#### Scenario: Skill directory and frontmatter +- **WHEN** the repository is inspected +- **THEN** `plugins/rf-desktop/skills/rf-platynui/SKILL.md` exists with frontmatter `name: rf-platynui` and a non-empty `description` +- **AND** `references/` and `assets/examples/` subdirectories exist with at least one file each + +#### Scenario: Marketplace validation passes +- **WHEN** the marketplace/SKILL.md validation test suite runs +- **THEN** the skill passes the same frontmatter/structure checks applied to every other skill diff --git a/openspec/changes/repo-cleanup/specs/python-library-skill/spec.md b/openspec/changes/repo-cleanup/specs/python-library-skill/spec.md new file mode 100644 index 0000000..62741e5 --- /dev/null +++ b/openspec/changes/repo-cleanup/specs/python-library-skill/spec.md @@ -0,0 +1,26 @@ +## MODIFIED Requirements + +### Requirement: Python library skill exists and follows the house structure + +The repository SHALL provide a skill at `plugins/rf-core/skills/rf-python-library/` with a `SKILL.md` whose frontmatter has `name: rf-python-library`, a `description`, a `compatibility` field that names Python ≥ 3.10 and `robotframework>=7`, a `references/` directory, a `scripts/` directory and an `assets/examples/` directory. `SKILL.md` SHALL be at most 300 lines and SHALL contain, in this order, the sections: a quick reference with version gate, a default library template, decision tables, placement and import, agent workflow, Gotchas, when to read the references, and Companion Skills. + +#### Scenario: Directory, frontmatter and size +- **WHEN** the repository is inspected +- **THEN** `plugins/rf-core/skills/rf-python-library/SKILL.md` exists with `name: rf-python-library`, a non-empty `description` and a `compatibility` value that mentions Python and `robotframework>=7` +- **AND** the file has at most 300 lines and all required sections in the stated order + +#### Scenario: Spec validator passes +- **WHEN** the skill metadata validator and the marketplace validation tests run +- **THEN** the new skill passes the same checks as every other shipped skill + +### Requirement: Distributed without drift + +The skill SHALL ship in the `rf-core` plugin, be copied unchanged into `bundles/rf-agentskills/` by the catalogue build, be included in the installer assets, be named in the `UserPromptSubmit` hook's skill list, and be listed in the `rf-core` README. `scripts/build-catalog.py --check` and `scripts/check-drift.sh` SHALL pass, and the script SHALL be a regular file (no symlink) in every channel. + +#### Scenario: All channels in sync +- **WHEN** `scripts/build-catalog.py --check` and then `scripts/check-drift.sh` run +- **THEN** both pass, and the plugin and the bundle each contain `rf-python-library/scripts/check_library.py` as a regular file + +#### Scenario: Hook mentions the skill +- **WHEN** the context-injection hook runs on the prompt "my robot framework library says it contains no keywords" +- **THEN** the injected text names `rf-python-library` diff --git a/openspec/changes/repo-cleanup/specs/setup-skill/spec.md b/openspec/changes/repo-cleanup/specs/setup-skill/spec.md new file mode 100644 index 0000000..4e45cba --- /dev/null +++ b/openspec/changes/repo-cleanup/specs/setup-skill/spec.md @@ -0,0 +1,14 @@ +## MODIFIED Requirements + +### Requirement: Setup skill exists and follows the house structure + +The repository SHALL provide a skill at `plugins/rf-core/skills/rf-setup/` with a `SKILL.md` whose frontmatter has `name: rf-setup` (matching its directory) and a non-empty `description`, a `references/` directory and an `assets/examples/` directory. + +#### Scenario: Skill directory and frontmatter +- **WHEN** the repository is inspected +- **THEN** `plugins/rf-core/skills/rf-setup/SKILL.md` exists with frontmatter `name: rf-setup` and a description that mentions installing Robot Framework and uv, venv/pip and poetry +- **AND** `references/` and `assets/examples/` exist and each contain at least one file + +#### Scenario: Marketplace validation passes +- **WHEN** the marketplace/SKILL.md validation test suite runs +- **THEN** the skill passes the same frontmatter and structure checks applied to every other skill diff --git a/openspec/changes/repo-cleanup/specs/skill-catalog/spec.md b/openspec/changes/repo-cleanup/specs/skill-catalog/spec.md new file mode 100644 index 0000000..7cc99c3 --- /dev/null +++ b/openspec/changes/repo-cleanup/specs/skill-catalog/spec.md @@ -0,0 +1,13 @@ +## MODIFIED Requirements + +### Requirement: Generator skills are not shipped in any remaining channel + +The content SHALL NOT contain the skills `rf-keyword-builder`, `rf-testcase-builder` or `rf-resource-architect`, nor the scripts `keyword_builder.py`, `testcase_builder.py` or `resource_architect.py`, in any channel: the plugins under `plugins/` (the source of every marketplace install), the `bundles/rf-agentskills` copy, the installer's bundled assets, or the release tarballs built from them. + +#### Scenario: No generator skill directories in any channel +- **WHEN** `plugins/*/skills/`, `bundles/rf-agentskills/skills/` and a freshly built installer `_assets/` tree are listed +- **THEN** none contains a skill whose `name:` or directory is `rf-keyword-builder`, `keyword-builder`, `rf-testcase-builder`, `testcase-builder`, `rf-resource-architect` or `resource-architect` + +#### Scenario: No generator scripts shipped +- **WHEN** the plugin and bundle `scripts/` directories, the skill `scripts/` directories and the installer `_assets/` tree are listed +- **THEN** none contains `keyword_builder.py`, `testcase_builder.py` or `resource_architect.py` diff --git a/openspec/changes/repo-cleanup/specs/skill-eval-harness/spec.md b/openspec/changes/repo-cleanup/specs/skill-eval-harness/spec.md new file mode 100644 index 0000000..098b7cb --- /dev/null +++ b/openspec/changes/repo-cleanup/specs/skill-eval-harness/spec.md @@ -0,0 +1,21 @@ +## MODIFIED Requirements + +### Requirement: Paired treatment and baseline arms + +The harness SHALL support two arms for every task. `treatment` provisions the plugin content as shipped: the skills, hooks and subagents of the repository's content plugins, merged the way the `rf-agentskills` bundle ships them. `baseline` provisions none of these. Apart from that, both arms SHALL use the same fixture, prompt, model, allowed tools, turn and time limits, and MCP servers (the per-task servers, such as `rf-mcp`). The baseline arm SHALL NOT be able to discover any of the repository's skills, whether user-level, project-level (`.claude/skills/`) or plugin-provided. + +#### Scenario: Baseline run has no skills available +- **WHEN** a task runs in the `baseline` arm +- **THEN** the session's isolated config dir and workspace contain no skill, hook or subagent from the repository's plugins +- **AND** the transcript contains no successful load of one of the repository's skills + +#### Scenario: Arms differ only by the plugin +- **WHEN** the same task runs in `treatment` and `baseline` +- **THEN** both runs use identical prompt, model, `max_turns`, `timeout_seconds`, allowed tools, fixture contents and MCP configuration +- **AND** the run records store the arm name + +#### Scenario: Selecting arms on the command line +- **WHEN** a batch is started with `--arms treatment,baseline` +- **THEN** every selected task runs in both arms +- **WHEN** a batch is started without `--arms` +- **THEN** only the `treatment` arm runs diff --git a/openspec/changes/repo-cleanup/specs/skill-metadata-conformance/spec.md b/openspec/changes/repo-cleanup/specs/skill-metadata-conformance/spec.md new file mode 100644 index 0000000..d1dbd30 --- /dev/null +++ b/openspec/changes/repo-cleanup/specs/skill-metadata-conformance/spec.md @@ -0,0 +1,13 @@ +## MODIFIED Requirements + +### Requirement: Repository references use the canonical identifiers + +Subagent definitions, hook-injected context, eval task `skill:` fields, CI and release packaging, and user documentation SHALL refer to skills only by their `rf-` identifier or their `plugins//skills/rf-/` path. The legacy long directory names (`robotframework-*-skill`, `robotframework-results`, `robotframework-libdoc-*`) and the legacy short plugin names MUST NOT be used as skill identifiers. + +#### Scenario: No legacy identifier remains +- **WHEN** the repository's skills, plugin agents and scripts, hooks, eval tasks, workflows, tests and README are searched for legacy skill identifiers +- **THEN** no match is found, apart from the changelogs and the installer's legacy-name list used for migration + +#### Scenario: Release packages use the new directory names +- **WHEN** the release workflow builds a plugin's standalone skills tarball and its Claude Desktop skill archives +- **THEN** every packaged skill directory is named `rf-` and its name matches its `SKILL.md` diff --git a/openspec/changes/repo-cleanup/tasks.md b/openspec/changes/repo-cleanup/tasks.md new file mode 100644 index 0000000..b255028 --- /dev/null +++ b/openspec/changes/repo-cleanup/tasks.md @@ -0,0 +1,89 @@ +## 1. Root config and dependencies (D4) + +- [ ] 1.1 Delete `.mcp.json`. Verify: `grep -rn "claude-flow" --exclude-dir=openspec --exclude-dir=.git .` finds only `.gitignore` (whose `.claude-flow/` and `.swarm/` lines stay, since they keep local tool files out). +- [ ] 1.2 Edit `pyproject.toml`: + - remove `scipy`, `numpy`, `httpx`, `polars` and their mypy overrides; + - remove the `validation` extra and the sdist `exclude` list; + - remove `skills` from the ruff excludes, and rewrite the `tests/test_*.py` comment; + - remove `pytest-asyncio` and `asyncio_mode` here and in `installer/pyproject.toml`; + - fix the classifier comment that says the harness grades "rf-agentskills". + + Run `uv lock`. Verify: the `uv.lock` diff only removes packages; `uv sync --all-packages` succeeds; `uv run pytest -q` (tests/eval) passes; `uv run ruff check src tests/eval` and `uv run mypy` pass. + +## 2. Eval harness dead code (D3) + +- [ ] 2.1 Delete: + - `domain/events.py`, `domain/skill.py` and `infrastructure/persistence/parquet_writer.py`; + - `ConfigError`, `PersistenceError` and `VERDICT_STATUSES`; + - `Settings.claude_model_default` and `Settings.rf_skill_eval_log_level`. + + Verify: `grep -rn "RunStarted\|parquet_writer\|ConfigError\|PersistenceError\|VERDICT_STATUSES\|claude_model_default" src tests scripts .github` finds nothing; `uv run mypy` and `uv run pytest -q` pass. +- [ ] 2.2 Remove the plugin MCP merge: + - in `claude_code_runner.py`: `_extra_mcp_servers`, its use in `_mcp_servers`, and the plugin-server leak check; + - its test in `tests/eval/test_runner_workspace.py`; + - the `rf-tools` wording in `infrastructure/mcp/config_builder.py`. + + Verify: `uv run pytest -q tests/eval/test_runner_workspace.py tests/eval/test_providers.py` passes, and a dry-run batch (`uv run rf-skill-eval run --help` plus the existing runner tests) still writes the per-task `rf-mcp` server. +- [ ] 2.3 Remove `CLAUDE_MODEL_DEFAULT` from `.env.example` and `docs/ci/usage.md` (keep `RF_SKILL_EVAL_LOG_LEVEL`). Verify: `grep -rn CLAUDE_MODEL_DEFAULT --exclude-dir=openspec .` finds nothing. + +## 3. CI and release (D5, D6) + +- [ ] 3.1 Delete `.github/workflows/update-latest.yml` and `.github/actions/setup-eval/`. Verify: `grep -rn "setup-eval\|update-latest" .github scripts tests README.md RELEASING.md CONTRIBUTING.md` finds nothing. +- [ ] 3.2 Add a "no plugin MCP server" section to `scripts/check-drift.sh`: no `plugins/*/.mcp.json`, `plugins/*/servers`, or the same under `bundles/*`. Add a test in `tests/test_drift_detection.py` that creates `plugins/x/.mcp.json` in a temporary copy and expects drift. Verify: `bash scripts/check-drift.sh` passes on the tree, and the new test passes. +- [ ] 3.3 In `ci.yml`: + - remove the variants and `validate-skills.py` steps from `validate-plugin`; + - remove the `build-claude-plugin` job; + - remove the Codex and Copilot steps and uploads from `build-portable-skills`; + - remove `pytest-asyncio` from the `pip install` lines and their comments. + + Verify: on the PR, the required checks have the same names and pass, and `grep -n "rf-agentskills-claude-plugin\|codex/skills\|github/skills" .github/workflows/ci.yml` finds nothing. +- [ ] 3.4 In `release.yml`, pass `--skills skills` to `build-skill-zips.py`. Verify: locally, `python scripts/export-skills.py rf-web skills && python scripts/build-skill-zips.py --plugin rf-web --skills skills --out ` writes `rf-browser.zip` and `rf-selenium.zip` (no test covers this script today), and the `build-portable-skills` CI job, which runs the same pair, passes on the PR. + +## 4. Scripts (D2) + +- [ ] 4.1 Delete `scripts/validate-marketplace.py`, `scripts/docker-test-harness.sh`, `docs/installer/docker.md` and `docs/installer/docker/`. Verify: `grep -rn "validate-marketplace\|docker-test-harness\|installer/docker" --exclude-dir=openspec --exclude-dir=archive .` finds only `installer/CHANGELOG.md` history. +- [ ] 4.2 Fix the old names: + - the `build-skill-zips.py` docstring (`robotframework-plugins install`, a real plugin in the usage example); + - the `build-description-variant.py` docstring (it stages `plugins/`, not `skills/`); + - the header of `scripts/templates/opencode-plugin.js`. + + Run `python scripts/build-agent-variants.py`. Verify: `python scripts/build-agent-variants.py --check` and `bash scripts/check-drift.sh` pass. + +## 5. Docs (D1) + +- [ ] 5.1 `git mv` the files listed in D1 to `docs/archive/`, and write `docs/archive/README.md` (one line each: date, what it was, what replaced it). Verify: `ls docs` shows only `archive/`, `catalog.md`, `ci/` and `installer/` if anything remains there. +- [ ] 5.2 Exclude `docs/archive/` in `scripts/check-docs-voice.py`. In `tests/test_repository_references.py`, replace the `docs/` prefix with `docs/archive/` and `docs/ci/architecture/adr/`. Add the ADR-006 status line. Verify: `python scripts/check-docs-voice.py` and `pytest tests/test_repository_references.py tests/test_docs_voice.py` pass. +- [ ] 5.3 Repoint links to moved files: + - in the installer sources, tests and `pyproject.toml`; + - in `plugins/rf-core/hooks/README.md`, `eval/fixtures/README.md` and `plugins/rf-core/evals/tasks/README.md`; + - in the ADRs, and in `docs/ci/usage.md` (`ddd-design.md`). + + Run `python scripts/build-catalog.py` to refresh the bundle copy of the hooks README. Verify: a link check (`grep -rnoE "docs/[A-Za-z0-9_./-]+\.(md|txt)"` over the tree, each target exists) finds no dead link, and `python scripts/build-catalog.py --check` passes. +- [ ] 5.4 Fix the factual errors in current docs: + - `docs/ci/usage.md:289,338`, `docs/ci/local-testing.md:25-30,267` and `docs/ci/faq.md:322-336` (the dead `skills/README.md` link); + - `plugins/rf-core/hooks/README.md:381` (`plugins/rf-core/tests/test_hook_scripts.py`); + - the docstrings of `tests/test_library_skill_structure.py:4-7` and `tests/_shell.py:34`; + - the README "Prerequisites" line that calls the plugins "rf-agentskills". + + Verify: `grep -rn "sync-skills\|plugins/rf-agentskills/tests\|skills/README.md" README.md docs/ci plugins tests --include=*.md --include=*.py` finds only the drift test's assertion that `sync-skills.sh` doesn't exist; `python scripts/check-docs-voice.py` passes. + +## 6. Specs and checks + +- [ ] 6.1 Run the full local checks: + - `uv run pytest -q`; + - `python -m pytest tests --ignore=tests/eval`; + - `python -m pytest plugins/rf-core/tests plugins/rf-desktop/tests`; + - `python scripts/build-catalog.py --check`; + - `bash scripts/check-drift.sh`; + - `python scripts/check-docs-voice.py`; + - `python scripts/validate-skills.py --channel all`; + - `uv run ruff check src tests/eval`; + - `uv run mypy`. + + Verify: all pass, apart from environment-only skips that also skip on `main`. +- [ ] 6.2 Run `openspec validate repo-cleanup` and `openspec validate --all`. Verify: both pass. +- [ ] 6.3 Open a PR against `main` (ask before posting). Verify: the required checks pass. + +## 7. Initial check + +- [ ] 7.1 After merge, check the removals and moves in use: one full CI run on `main`, one local run of the checks in `AGENTS.md` or CONTRIBUTING, and the next contributor PR. Write down anything that broke or got harder. If a decision in `design.md` turns out wrong or risky, propose the change to the maintainer and update `design.md`. Verify that the findings are recorded in this change. diff --git a/openspec/changes/robotcode-first-tooling/.openspec.yaml b/openspec/changes/robotcode-first-tooling/.openspec.yaml new file mode 100644 index 0000000..90080a9 --- /dev/null +++ b/openspec/changes/robotcode-first-tooling/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-11-16 diff --git a/openspec/changes/robotcode-first-tooling/design.md b/openspec/changes/robotcode-first-tooling/design.md new file mode 100644 index 0000000..77219a7 --- /dev/null +++ b/openspec/changes/robotcode-first-tooling/design.md @@ -0,0 +1,133 @@ +## Context + +See proposal.md (Why). What the change builds on, checked on robotcode 2.8.0 with Robot Framework 7.5.1 in the repository's `.venv` (Linux, no display): + +- **The model to copy.** `plugins/rf-core/scripts/validate_robot.mjs` (tier 0) and `validate_robot_project.mjs` run `robotcode --format json analyze code` from the project root and classify diagnostics by `code`. Tier 1 runs Robocop and classifies by rule id (`DEPR08` = `Run Keyword If`/`Unless`). Only tier 3, the opt-in dry run, still starts `python -m robot --dryrun`. +- **`robotcode analyze code` covers keyword resolution.** On a suite importing String, Collections, AppiumLibrary 3.2.1 and PlatynUI.BareMetal 0.12.0.dev330: + - valid calls of all four libraries (`Open Application`, `Pointer Click`, …) give no diagnostic, without a device or desktop; + - `Convert To Uper Case` and `Not A Real Keyword` give `KeywordNotFound` (severity 1); + - `Convert To Upper Case a b c` gives `DataError` "expected 1 argument, got 3"; + - AppiumLibrary's `Element Should Be Visible` gives `DeprecatedKeyword` (severity 4, a hint); + - `Run Keyword If` gives nothing; Robocop 8.2.10 `--select "DEPR*"` reports it; + - a missing library gives `DataError` "Importing test library 'NoSuchLib' failed: ModuleNotFoundError …"; + - the JSON is `{"diagnostics": {file: [{range, message, severity, code, source}]}, "summary": {files, errors, warnings, infos, hints}}`; one file takes about 0.8 s. +- **`robotcode results` covers result reading.** `robotcode --format json results summary -o out/output.xml` gives `{"status", "counts": {"total", "passed", "failed", "skipped", "notRun"}, …}`; `results show` adds a `tests` list with `name`, `fullName`, `status`, `message`, `source`, `lineno`. The output path is given with `-o`, not as an argument. +- **`robotcode doc keywords ` with `--format json`** lists `name`, `args`, `doc`, `short_doc`, `tags`, `source`, `lineno` per keyword. There is no `deprecated` field; deprecation shows only in `doc` (`**DEPRECATED!!** …`). +- **Where the repo re-implements this today:** + - keyword resolution: `scripts/check-skill-keywords.py` (`get_model`, `LibraryDocumentation`, own matching with embedded arguments), `src/rf_skill_eval/scoring/keyword_resolution.py` (the same against libdoc JSON specs); + - result parsing: `grader/robot_runner.py` `_parse_output_xml`, used by `deterministic.py` and `scoring/custom/language.py`; `scoring/custom/python_library.py` `_statuses`; `scripts/desktop-live-check.py` `result_of`; + - running: the same four files start `robot` or `python -m robot`; `language.py` starts pinned versions with `uvx --from robotframework==X robot` (task on `sut-rf71`); + - deprecation: `deterministic.py` `_DEPRECATED_PATTERNS` (three regexes). +- **The spec fallback.** PR #20 moves the AppiumLibrary and PlatynUI libdoc specs to `eval/graders/specs/` and adds `spec_dir`. CI's eval jobs run `uv sync --all-packages`, which installs the dev group with both libraries and `robotcode[all]`, so the specs are no longer needed there. +- **What is custom value, not re-implementation:** + - `check_library.py`: library-author defects (rejected or leaked keywords, listener exposure, `str` in unions, TEST-scope state, broad `except`). They need the library object in-process; `robotcode doc lib` gives only the listing. + - `rf_conventions.py`: separator, naming, BDD, template and layout conventions, from the parser. + - `scoring/custom/platynui.py`: PlatynUI-specific checks (unsafe expressions) on the parsed model. + - `min_calls` counting: RobotCode has no call graph. + +## Goals / Non-Goals + +**Goals:** +- No tooling resolves keywords, parses results or starts `robot` itself where RobotCode does it. +- One guard, in CI, that needs no human judgement per PR: a reasoned allowlist decides. +- Every gap found becomes a drafted RobotCode issue. + +**Non-Goals:** +- Changing what the skills teach users; they already teach RobotCode. +- Rewriting `check_library.py` or `rf_conventions.py`. +- Replacing Robocop: it is part of `robotcode[all]` and RobotCode uses it. +- Scanning `installer/` or `tests/`: the installer doesn't touch Robot Framework, and tests may build fixtures however they need. + +## Decisions + +### D1. Skill keyword checker: materialise the snippets, let RobotCode resolve + +For each library skill, `check-skill-keywords.py` writes a temporary project: +- one `.robot` file per fenced block or example file, with a generated `*** Settings ***` importing the skill's target library and the standard libraries the checker allows today; +- a fragment without a section header becomes the body of a generated test case; +- the `*** Keywords ***` sections of the whole skill go into one `skill.resource` that every file imports, so a keyword defined in one block resolves in another, as today. + +It runs `robotcode --format json analyze code` once on the directory, maps each diagnostic back through a line table to the source file and line, and keeps only `KeywordNotFound`, `DeprecatedKeyword` and argument `DataError`. An import failure of the target library marks the skill `skipped (library not installed)`. Close-match suggestions use `difflib` over the names from `robotcode doc keywords --format json`. `Run Keyword If`/`Unless` comes from `robocop check --select DEPR08` on the same files. The `Wait For Condition` vocabulary check and the allowlist stay as they are. + +Before switching, a parity run compares the new checker with the old on the current skills and on the test fixtures. Differences are explained in the PR; new true findings (for example argument errors the old checker never saw) are fixed or allowlisted. + +Alternative: keep the extractor and matcher and only take keyword names from `robotcode doc keywords --format json` instead of `LibraryDocumentation`. It removes the libdoc import but keeps our own matching (embedded arguments, normalisation) and gains no argument checks. Fallback if the parity run shows that materialised snippets give noise that can't be suppressed (for example duplicate keyword definitions across blocks). + +### D2. `keywords_resolve`: analyse, then count + +The check runs `robotcode --format json analyze code ` in the workspace. An error diagnostic (`KeywordNotFound`, argument `DataError`, or an import failure of a library listed in `required_libraries`) fails it. For `min_calls`, the existing call-graph walk stays (allowlisted `get_model`), but a call counts as a library call when its name is in the keyword set of an imported non-standard library, read with `robotcode --format json doc keywords ` and cached per library. The `specs` and `spec_dir` params, `_libdoc()` and `eval/graders/specs/` go; `validate-tasks` rejects the params. + +Alternative: keep the specs as a fallback for libraries that don't install. Rejected: every library in the tasks installs on CI, and a fallback that never runs rots, as the spec path did. + +### D3. Grader runs: one helper over `robotcode robot` and `robotcode results` + +A single helper replaces `robot_runner.run_robot`, `_parse_output_xml` and `python_library._statuses`. It runs `robotcode robot --outputdir [--dryrun] [args] ` and then `robotcode --format json results show -o /output.xml`. It returns counts and per-test status and message. `robotcode` is found next to `sys.executable` first, then on PATH, as `_robot_binary()` does today. + +For pinned Robot Framework versions (`language.py`, task on `sut-rf71`), the helper runs `uvx --from robotframework==X --with "robotcode[runner]" robotcode robot …` if robotcode 2.8's runner supports that version (to verify for 7.1). If not, the `uvx … robot` call stays, allowlisted with the reason, and results are still read with `robotcode results` from the harness environment. + +### D4. `no_deprecated_keywords` + +The check reads the `DeprecatedKeyword` diagnostics from the same `analyze code` call and runs Robocop's hard-deprecation rules (`DEPR03/04/07/08/09/10/11`, the set the hooks already use). The regex list goes. This can flag more than before: library-deprecated keywords, `[Return]`, `Force Tags`. That is intended and is one reason for the re-baseline. + +### D5. The guard + +`scripts/check-robotcode-first.py` scans `.py`, `.mjs` and `.js` files under the tooling paths with line regexes for the patterns in the spec. It skips comment lines. It matches imports and calls, not file names, so prose that mentions `output.xml` (the hint in `maybe_inject_rf_context.mjs`) is not a finding. `scripts/robotcode-first-allowlist.toml` holds `[[allow]] file, pattern, reason`, mirroring `skill-keywords-allowlist.toml`; an entry that matches nothing fails. + +It runs as a step in `ci.yml`'s lint job and as `tests/test_robotcode_first.py`, so the check is automated and needs no reviewer. + +Allowlist after this change (expected): + +| File | Pattern | Reason | +|---|---|---| +| `plugins/rf-core/skills/rf-python-library/scripts/check_library.py` | `LibraryDocumentation` | library-author defect analysis needs the in-process library object | +| `plugins/rf-core/skills/rf-language/scripts/rf_conventions.py` | `get_model`, `ModelVisitor` | convention detection; RobotCode has none | +| `src/rf_skill_eval/scoring/keyword_resolution.py` | `get_model` | `min_calls` call-graph counting | +| `src/rf_skill_eval/scoring/custom/platynui.py` | `get_model` | PlatynUI-specific usage checks | +| `scripts/check-skill-keywords.py` | `get_model` | only if the D1 fallback is taken | +| `src/rf_skill_eval/scoring/custom/language.py` | `uvx … robot` | only if D3's pinned-version runner doesn't work | + +Regexes over source can be fooled. The guard is a tripwire for honest contributors and agents, not a sandbox, so a reviewer of a PR that touches the allowlist sees why each exception exists. + +### D6. Order with other work + +1. PR #20 merges (it adds `eval/graders/specs/` and `spec_dir`, which D2 deletes). +2. `fix-desktop-eval-turn-overhead` is archived, so its `min_calls` requirement is in the main spec. This change then words "spec-covered libraries" as "imported non-standard libraries" in a follow-up delta, or the requirement is reworded during its archive. +3. If `repo-cleanup` lands first, `scripts/validate-marketplace.py` is gone; otherwise it gets a temporary allowlist entry. + +## RobotCode gaps found (drafts to file after review) + +Tested on robotcode 2.8.0 with Robot Framework 7.5.1. None blocks this change; each makes our code simpler once fixed. + +1. **Import failures have no own diagnostic code.** A failed library import is reported as `DataError`, the same code as an argument-count error. Tools that must tell "the environment is missing a library" from "the code is wrong" have to parse the message text (`Importing test library '…' failed`). The hook (`validate_robot.mjs`) and D1/D2 all need this distinction. Proposal: a distinct code such as `ImportFailed` / `LibraryImportFailed`. +2. **`doc keywords --format json` has no `deprecated` flag.** Deprecation is only in the free-text `doc`, written differently per library (`*DEPRECATED*`, `**DEPRECATED!!**`). Proposal: a boolean `deprecated` (Robot Framework's `KeywordDoc.deprecated`) in the JSON. +3. **The `DeprecatedKeyword` message for AppiumLibrary reads "is deprecated: !!."** It comes from AppiumLibrary's `**DEPRECATED!!** Use Expect Element instead` doc. Check against plain `robot` first; if Robot Framework's own warning is equally garbled, report it to AppiumLibrary instead. +4. **`KeywordNotFound` has no suggestions.** Robot Framework's runtime error offers "Did you mean …"; the static diagnostic doesn't. Optional; we compute suggestions from `doc keywords` meanwhile. + +`robotcode/issues#689` (reading libdoc spec files) is moot after D2: our need for spec files goes away. Whether to close it is the maintainer's call. + +## Risks / Trade-offs + +- [Materialised snippets produce noise, such as duplicate keywords across blocks or fragments that only make sense in prose] → Parity run before switching; D1 fallback keeps our matcher with RobotCode's keyword names. +- [The skill checker gets slower: one `analyze code` per skill plus `doc keywords` per library] → One directory run per skill (about 1 s per file measured; the analysis cache is off in a fresh temp dir). Acceptable for a CI job; measure in the parity run. +- [A RobotCode release changes JSON fields or codes] → The JSON fields used are pinned in tests with recorded output; `robotcode[all]` stays pinned to a range in the dev group. +- [More checks find more issues, so eval scores move] → Expected. The re-baseline after merge records them; the PR states which checks became stricter. +- [The guard's regexes miss an indirect use (an alias import, `subprocess` with a variable)] → Accepted: it's a tripwire, and AGENTS.md (from `ai-contributor-docs`) states the rule for humans and agents. + +## Migration Plan + +1. Land the guard first with an allowlist covering every current use, so CI is green and new uses are blocked from day one. +2. Replace the uses one area at a time (grader helper, `keywords_resolve`, skill checker, hook tier 3, desktop live check), deleting allowlist entries as they go; the stale-entry rule forces this. +3. Re-baseline GLM in CI after the eval-facing parts merge. + +Rollback: each step is its own commit; reverting one restores its allowlist entries. + +## Confirmed decisions (maintainer, 2026-11-16) + +Each decision stands for now. After implementation, task 8.1 checks it. If that check, or anything found along the way, shows a problem or a risk, the decision can change again. In that case, update this design before going on. + +- **Stricter checks:** argument-count errors from `analyze code` and the Robocop deprecation rules are wanted, even though they flag more than today and move eval scores. +- **Windows PC:** robotcode may be installed in the Windows desktop venv for `scripts/desktop-live-check.py` (task 5.2). + +## Open Questions + +- None open. The Windows question is settled above. diff --git a/openspec/changes/robotcode-first-tooling/proposal.md b/openspec/changes/robotcode-first-tooling/proposal.md new file mode 100644 index 0000000..e61518a --- /dev/null +++ b/openspec/changes/robotcode-first-tooling/proposal.md @@ -0,0 +1,40 @@ +## Why + +RobotCode's CLI (https://robotcode.io/guides/cli/) is the base of the Robot Framework AI toolstack: it runs suites with the project's `robot.toml` and profiles, discovers tests, analyses code, reads results, browses keyword documentation, and offers a REPL and a terminal debugger. Our skills already tell agents to use it ("Keywords: `robotcode doc keyword`, not memory; results: `robotcode results`, not output.xml", `plugins/rf-core/scripts/maybe_inject_rf_context.mjs:62`), and the validation hooks call `robotcode analyze code`. The repository's own tooling does the opposite in several places: + +- `scripts/check-skill-keywords.py` (617 lines) and the harness's `scoring/keyword_resolution.py` (344 lines) resolve keywords themselves with `robot.api.get_model` and `LibraryDocumentation`. The harness also needs committed libdoc spec files for AppiumLibrary and PlatynUI because it assumed they "cannot run on the CI runner". Both libraries install and load on Linux CI; `robotcode analyze code` resolves their keywords without a device or desktop. +- The grader runs plain `robot` and parses `output.xml` with ElementTree in four modules, where `robotcode robot` and `robotcode results --format json` give the same counts and failure messages. +- The deprecated-keyword check is a three-pattern regex. + +Each re-implementation is code we maintain, that drifts from how Robot Framework and RobotCode behave, and that improves only this repository. Gaps we find should be fixed in RobotCode so every AI solution for Robot Framework gets them. + +## What Changes + +- **Keyword checks use `robotcode analyze code`.** `check-skill-keywords.py` writes the skills' Robot Framework snippets into a temporary project and runs `robotcode --format json analyze code` on it; it maps `KeywordNotFound`, `DeprecatedKeyword` and argument `DataError` diagnostics back to the Markdown file and line. What stays custom: Markdown extraction, the allowlist, the `Wait For Condition` vocabulary check, and close-match suggestions (from `robotcode doc keywords --format json`). Legacy `Run Keyword If`/`Unless` comes from Robocop's `DEPR08` rule, which the hooks already use. +- **The eval harness drops libdoc spec files.** `keywords_resolve` resolves through `robotcode analyze code` against the installed libraries; `min_calls` counting stays custom but takes library keyword names from `robotcode doc keywords --format json`. The `spec_dir`/`specs` task params and `eval/graders/specs/` (added by PR #20) are removed. `no_deprecated_keywords` uses the `DeprecatedKeyword` diagnostic plus Robocop's deprecation rules instead of its regex list. +- **The grader runs and reads through robotcode.** `robot_pass`, `robot_dryrun`, the rf-language and rf-python-library graders and `scripts/desktop-live-check.py` run `robotcode robot` and read `robotcode results summary/show --format json`; the ElementTree parsers are deleted. +- **The opt-in per-file dry-run hook runs `robotcode robot --dryrun`**, so `robot.toml` and profiles apply as in the other hook tiers. +- **A guard keeps it that way.** `scripts/check-robotcode-first.py` fails CI when tooling under `scripts/`, `src/`, `plugins/*/scripts`, `plugins/*/skills/*/scripts` or `plugins/*/hooks` uses a Robot Framework API or command that RobotCode covers (libdoc, result parsing, suite building, running `robot`/`rebot`/`libdoc` directly) outside an allowlist. Each allowlist entry names the file, the pattern and why RobotCode doesn't cover it; an entry that matches nothing fails. +- **Gaps go upstream.** The design lists the RobotCode gaps found (for example, failed library imports share the generic `DataError` code with argument errors). Each becomes a drafted robotcode issue that the maintainer reviews and files under robotcode's AI policy. + +Not changed: `check_library.py` (rf-python-library) and `rf_conventions.py` (rf-language) stay. Their findings (library-author defects, project style conventions) are not RobotCode capabilities; they get allowlist entries with reasons. + +## Capabilities + +### New Capabilities + +- `robotcode-first-tooling`: repository tooling uses the RobotCode CLI for core Robot Framework capabilities, an automated guard with a reasoned allowlist enforces it, and capability gaps are reported to RobotCode instead of worked around. + +### Modified Capabilities + +- `skill-content-verification`: the deterministic keyword checker resolves through `robotcode analyze code` and reports argument-count errors too. +- `skill-eval-harness`: grader runs and result reading go through `robotcode robot` and `robotcode results`; static keyword resolution uses the installed libraries, not libdoc spec files. +- `rf-validation-hooks`: the opt-in per-file dry run uses `robotcode robot --dryrun`. + +## Impact + +- **Code:** `scripts/check-skill-keywords.py`, `scripts/desktop-live-check.py`, new `scripts/check-robotcode-first.py` and `scripts/robotcode-first-allowlist.toml`; `src/rf_skill_eval/grader/robot_runner.py`, `scoring/deterministic.py`, `scoring/keyword_resolution.py`, `scoring/custom/language.py`, `scoring/custom/python_library.py`; `plugins/rf-core/scripts/validate_robot.mjs` (tier 3). +- **Eval data:** the five PlatynUI and Appium task YAMLs lose `spec_dir`/`specs`; `eval/graders/specs/` is deleted. Task hashes change, so a GLM re-baseline is needed (maintainer-dispatched). +- **CI:** a new step in `ci.yml` runs the guard; the keyword job needs `robotcode` (already in the dev group). +- **Order:** PR #20 (`fix/realistic-eval-tasks`) merges first. `fix-desktop-eval-turn-overhead` (its `min_calls` requirement still mentions "spec-covered libraries") is archived first or reconciled in this change. The `ai-contributor-docs` change states the rule; this change enforces it for tooling. +- **Users:** only the opt-in dry-run hook changes behaviour, and only for projects without robotcode, which is already a documented prerequisite (`robotcode-prerequisite`). diff --git a/openspec/changes/robotcode-first-tooling/specs/rf-validation-hooks/spec.md b/openspec/changes/robotcode-first-tooling/specs/rf-validation-hooks/spec.md new file mode 100644 index 0000000..a73aa99 --- /dev/null +++ b/openspec/changes/robotcode-first-tooling/specs/rf-validation-hooks/spec.md @@ -0,0 +1,17 @@ +## MODIFIED Requirements + +### Requirement: Opt-in per-file dry run + +When `RF_AGENTSKILLS_FILE_DRYRUN` is truthy, per-file validation of a `.robot` suite file (not `.resource`, not `__init__.robot`) SHALL additionally run `robotcode robot --dryrun` on that file from the project root, with all outputs disabled, so the project's `robot.toml` and profiles apply. It SHALL report `[ ERROR ]` lines and missing-keyword or argument failures as non-blocking additional context, capped like the deprecation feedback. It is off by default. When `robotcode` is not available, the dry run SHALL be skipped silently, like the other tiers. + +#### Scenario: Unknown keyword surfaced when enabled +- **WHEN** the flag is set and the agent writes a test that calls a keyword that no import provides +- **THEN** the hook adds the dry-run "No keyword with name" message as additional context and exits 0 unless another tier found a blocking error + +#### Scenario: Disabled by default +- **WHEN** the flag is not set +- **THEN** no `robotcode` or `robot` process is started for the dry run + +#### Scenario: Profile settings apply +- **WHEN** the flag is set and the project's `robot.toml` adds a `python-path` entry that a suite's library import needs +- **THEN** the dry run resolves that import and reports no import error diff --git a/openspec/changes/robotcode-first-tooling/specs/robotcode-first-tooling/spec.md b/openspec/changes/robotcode-first-tooling/specs/robotcode-first-tooling/spec.md new file mode 100644 index 0000000..2017a7c --- /dev/null +++ b/openspec/changes/robotcode-first-tooling/specs/robotcode-first-tooling/spec.md @@ -0,0 +1,45 @@ +## Purpose + +Keep the repository's own tooling on the RobotCode CLI for core Robot Framework capabilities (running, discovery, analysis, results, keyword documentation), enforce it with an automated guard, and send capability gaps upstream to RobotCode instead of re-implementing them here. + +## ADDED Requirements + +### Requirement: Tooling uses RobotCode for core Robot Framework capabilities + +Repository tooling (files under `scripts/`, `src/`, `plugins/*/scripts/`, `plugins/*/skills/*/scripts/` and `plugins/*/hooks/`) SHALL use the `robotcode` CLI to run suites (`robotcode robot`), discover suites and tests (`robotcode discover`), analyse Robot Framework code (`robotcode analyze code`), read run results (`robotcode results`) and read library or keyword documentation (`robotcode doc`). It SHALL NOT re-implement these capabilities with Robot Framework's Python API or by starting `robot`, `rebot` or `libdoc` directly, unless an allowlist entry covers the use. Custom code SHALL remain allowed for what RobotCode does not provide, such as Markdown extraction, hook integration, project-convention detection, library-author checks and eval grading logic. + +#### Scenario: Results read through RobotCode +- **WHEN** a grader needs the pass, fail and skip counts or failure messages of a finished run +- **THEN** it reads them from `robotcode results summary` or `robotcode results show` with `--format json`, not by parsing `output.xml` + +#### Scenario: Custom value stays custom +- **WHEN** a skill script detects project naming conventions with the Robot Framework parser +- **THEN** the use is allowed through an allowlist entry whose reason states that RobotCode has no such capability + +### Requirement: Automated guard with a reasoned allowlist + +The repository SHALL provide a deterministic check that scans the tooling paths for direct uses of Robot Framework APIs and commands that RobotCode covers: `robot.libdoc`/`robot.libdocpkg`/`LibraryDocumentation`, `ExecutionResult` and XML parsing of result files (`xml.etree`/`ElementTree`), `TestSuiteBuilder`, `get_model`/`ModelVisitor`, and process calls that start `robot`, `rebot` or `libdoc` (including `-m robot`). It SHALL exit non-zero for any use that no allowlist entry covers, naming the file, line and pattern. Each allowlist entry SHALL name the file, the pattern and a reason. An entry that matches nothing SHALL fail the check. The check SHALL run in CI on every pull request and in the pytest suite, and SHALL need no human review to pass or fail. + +#### Scenario: New re-implementation fails +- **WHEN** a pull request adds `from robot.api import ExecutionResult` to a file under `src/` that no allowlist entry covers +- **THEN** the check exits non-zero and names the file, the line and the pattern + +#### Scenario: Allowlisted use passes +- **WHEN** `plugins/rf-core/skills/rf-python-library/scripts/check_library.py` uses `LibraryDocumentation` and an allowlist entry for that file and pattern gives the reason +- **THEN** the check passes for that use + +#### Scenario: Stale entry fails +- **WHEN** an allowlist entry names a file or pattern that no longer occurs +- **THEN** the check fails and names the stale entry + +#### Scenario: Robocop is not flagged +- **WHEN** a hook runs `robocop check` +- **THEN** the check does not report it, because Robocop is part of `robotcode[all]` and RobotCode does not replace it + +### Requirement: RobotCode gaps are reported upstream + +When repository tooling needs a Robot Framework capability that RobotCode lacks or gets wrong, the change that meets the need SHALL record the gap in its design with the RobotCode version tested and a minimal reproduction, and SHALL draft an issue for `robotcodedev/robotcode`. A maintainer SHALL review each draft before it is filed, and the filed issue SHALL carry the AI disclosure that RobotCode's AI policy asks for. A local workaround SHALL be allowlisted with a reason that links the issue, so it is removed when RobotCode closes the gap. + +#### Scenario: Workaround linked to its issue +- **WHEN** tooling keeps a workaround for a RobotCode gap +- **THEN** its allowlist entry's reason contains the URL of the RobotCode issue diff --git a/openspec/changes/robotcode-first-tooling/specs/skill-content-verification/spec.md b/openspec/changes/robotcode-first-tooling/specs/skill-content-verification/spec.md new file mode 100644 index 0000000..410bd00 --- /dev/null +++ b/openspec/changes/robotcode-first-tooling/specs/skill-content-verification/spec.md @@ -0,0 +1,30 @@ +## MODIFIED Requirements + +### Requirement: Deterministic keyword checker gates CI + +The repository SHALL provide a deterministic checker that extracts the Robot Framework content of library skills, analyses it with `robotcode analyze code` against the skill's target library, the standard libraries and the user keywords defined in the same skill, and reports keyword calls that are unknown, deprecated or called with the wrong number of arguments. Legacy `Run Keyword If`/`Unless` calls SHALL be reported from Robocop's deprecation rule. It SHALL exit non-zero when any call is unknown, deprecated, has an argument error or is a legacy `Run Keyword If`/`Unless`, unless an allowlist entry covers the call. Each allowlist entry SHALL name the skill, the file, the keyword and a reason. Findings SHALL name the Markdown or example file and line the call comes from, not a generated file. The checker SHALL run in a CI job and in the pytest suite. When a target library cannot be imported, the checker SHALL report that skill as skipped and SHALL NOT pass it silently. + +#### Scenario: Clean repository passes +- **WHEN** the checker runs on the repository with all target libraries installed +- **THEN** it exits 0 and reports each checked skill with its file count + +#### Scenario: Injected defect fails +- **WHEN** a fixture skill contains a call to a deprecated or nonexistent keyword that no allowlist entry covers +- **THEN** the checker exits non-zero and names the skill, file, line and keyword, with close-match suggestions when available + +#### Scenario: Wrong argument count fails +- **WHEN** a fixture skill calls `Convert To Upper Case` with three arguments +- **THEN** the checker exits non-zero and names the skill, file, line and keyword with RobotCode's argument message + +#### Scenario: Allowlisted exception passes +- **WHEN** a call is unknown but matches an allowlist entry for that skill and file +- **THEN** the checker does not fail on it + +#### Scenario: Stale allowlist entry reported +- **WHEN** an allowlist entry matches no call in the current content +- **THEN** the checker fails and names the stale entry + +#### Scenario: Missing library is not a silent pass +- **WHEN** a target library cannot be imported +- **THEN** the checker's output marks that skill as `skipped (library not installed)` +- **AND** in CI, where all target libraries are installed, a skipped skill fails the job diff --git a/openspec/changes/robotcode-first-tooling/specs/skill-eval-harness/spec.md b/openspec/changes/robotcode-first-tooling/specs/skill-eval-harness/spec.md new file mode 100644 index 0000000..e3ddd9d --- /dev/null +++ b/openspec/changes/robotcode-first-tooling/specs/skill-eval-harness/spec.md @@ -0,0 +1,29 @@ +## ADDED Requirements + +### Requirement: Grader runs and reads results through RobotCode + +Grader checks that run a produced suite (`robot_pass`, `robot_dryrun` and the custom graders that run hidden suites) SHALL start the run with `robotcode robot` from the workspace and SHALL read pass, fail and skip counts and failure messages from `robotcode results` with `--format json`. A check whose run needs a Robot Framework version other than the harness's own MAY start that version through `uvx`, with `robotcode` in the same environment where its runner supports that version; otherwise the use SHALL be allowlisted with a reason. When `robotcode` is not installed in the grader environment, the check SHALL be `skipped`, never `passed`. + +#### Scenario: Counts from RobotCode +- **WHEN** a produced suite has one passing, one failing and one skipped test and a `robot_pass` check runs on it +- **THEN** the verdict details report `tests=3 passed=1 failed=1` and the check fails + +#### Scenario: RobotCode missing +- **WHEN** `robotcode` is not on the grader's PATH or next to its interpreter +- **THEN** `robot_pass` and `robot_dryrun` return `skipped` with a reason naming `robotcode` + +### Requirement: Static keyword resolution uses the installed libraries + +The `keywords_resolve` check SHALL resolve the produced suite's keyword calls with `robotcode analyze code` against the libraries installed in the grader environment, and SHALL NOT read libdoc spec files. A `KeywordNotFound` or argument error in the suite, or a failed import of a library the task requires, SHALL fail the check. For `min_calls`, a library call SHALL be a call to a keyword of an imported library that is not a Robot Framework standard library, with the library's keyword names taken from `robotcode doc keywords --format json`. Task YAMLs SHALL NOT have `spec_dir` or `specs` params, and the repository SHALL NOT keep libdoc spec files for grading. + +#### Scenario: PlatynUI resolves without a desktop +- **WHEN** the grader environment has `robotframework-PlatynUI` installed, runs on Linux without a display, and the produced suite imports `PlatynUI.BareMetal` and calls `Pointer Click` +- **THEN** the check resolves the call and counts it as one library call + +#### Scenario: Misspelled keyword fails +- **WHEN** the produced suite calls `Convert To Uper Case` +- **THEN** the check fails and its details name the keyword and the `KeywordNotFound` diagnostic + +#### Scenario: Spec params rejected +- **WHEN** a task YAML gives `keywords_resolve` a `specs` param +- **THEN** task validation fails and names the param diff --git a/openspec/changes/robotcode-first-tooling/tasks.md b/openspec/changes/robotcode-first-tooling/tasks.md new file mode 100644 index 0000000..f794c6f --- /dev/null +++ b/openspec/changes/robotcode-first-tooling/tasks.md @@ -0,0 +1,46 @@ +## 0. Preconditions + +- [ ] 0.1 PR #20 (`fix/realistic-eval-tasks`) is merged and `fix-desktop-eval-turn-overhead` is archived; verify `eval/graders/specs/` and the `min_calls` requirement are on `main`. + +## 1. Guard first (D5) + +- [ ] 1.1 Add `scripts/check-robotcode-first.py` and `scripts/robotcode-first-allowlist.toml` with an entry, and its reason, for every current use; verify it exits 0 on `main` and that tests cover "New re-implementation fails", "Allowlisted use passes", "Stale entry fails" and "Robocop is not flagged". +- [ ] 1.2 Run it in `ci.yml` (lint job) and as `tests/test_robotcode_first.py`; verify the CI step fails on a throwaway commit that adds `from robot.api import ExecutionResult` under `src/`. + +## 2. Grader runs and results (D3) + +- [ ] 2.1 Replace `grader/robot_runner.py` (`run_robot`, `_parse_output_xml`) and `python_library._statuses` with one helper over `robotcode robot` and `robotcode --format json results show -o …`; switch `deterministic._run_robot_check`, `custom/language.py` and `custom/python_library.py` to it; verify the "Counts from RobotCode" and "RobotCode missing" scenarios in `tests/eval` and that the existing grader tests pass unchanged in meaning. +- [ ] 2.2 Check whether `uvx --from robotframework==7.1 --with "robotcode[runner]" robotcode robot` runs the `sut-rf71` grader suite; use it if so, else keep the `uvx … robot` call with an allowlist entry; record the result in this design. +- [ ] 2.3 Remove the allowlist entries the stale-entry rule now reports; verify the guard passes. + +## 3. Static keyword resolution (D2, D4) + +- [ ] 3.1 Rewrite `keywords_resolve` on `robotcode analyze code` plus `robotcode doc keywords --format json` for `min_calls`; delete `_libdoc()` and spec loading; verify "PlatynUI resolves without a desktop" and "Misspelled keyword fails" on Linux without a display, and the existing `min_calls` tests. +- [ ] 3.2 Remove `spec_dir`/`specs` from the five PlatynUI and Appium task YAMLs, delete `eval/graders/specs/`, and make `validate-tasks` reject the params; verify "Spec params rejected" and `validate-tasks` for all plugins. +- [ ] 3.3 Base `no_deprecated_keywords` on `DeprecatedKeyword` diagnostics and Robocop's hard-deprecation rules; verify tests for a library-deprecated keyword, `Run Keyword If` and `[Return]`. +- [ ] 3.4 Reword the `min_calls` requirement in `openspec/specs/skill-eval-harness/spec.md` from "spec-covered libraries" to "imported non-standard libraries" (delta in this change); verify `openspec validate robotcode-first-tooling`. + +## 4. Skill keyword checker (D1) + +- [ ] 4.1 Build the materialising checker behind a flag and run a parity comparison with the current checker on all library skills and on the test fixtures; write the differences and their causes into this design. +- [ ] 4.2 If parity holds, make it the default and delete `LibraryDocumentation`/matcher code; else take the D1 fallback (RobotCode keyword names, own matcher). Verify the skill-content-verification scenarios, including "Wrong argument count fails", and that the CI keyword job passes. +- [ ] 4.3 Fix or allowlist the new true findings the stricter check reports; verify `scripts/check-skill-keywords.py --require-all` exits 0. + +## 5. Hook and manual tools + +- [ ] 5.1 Switch `validate_robot.mjs` tier 3 to `robotcode robot --dryrun` from the project root; verify the rf-validation-hooks scenarios, including "Profile settings apply", in `plugins/rf-core/tests`, and the variant check (`build-agent-variants.py --check`). +- [ ] 5.2 Install `robotcode` in the Windows desktop venv (approved on 2026-11-16) and switch `scripts/desktop-live-check.py` to the D3 helper. Verify with one live run on the PC. If the install causes problems, keep the script allowlisted with that reason and tell the maintainer. + +## 6. Upstream gaps + +- [ ] 6.1 Draft the RobotCode issues listed in the design (import-failure code, `deprecated` in doc JSON, the AppiumLibrary deprecation message after checking plain `robot`, suggestions), each with version, minimal reproduction and AI disclosure, in a scratch file; the maintainer reviews and files them. Link the filed issues from the matching allowlist reasons. + +## 7. Re-baseline and PR + +- [ ] 7.1 Run `pytest tests`, `pytest tests/eval`, `pytest plugins/*/tests`, `ruff`, `mypy`, the guard and `validate-tasks`; verify all pass. +- [ ] 7.2 Open the PR (ask before posting); verify the required checks pass. +- [ ] 7.3 After merge, a maintainer dispatches the GLM re-baseline (OpenRouter `z-ai/glm-5.3-flash`, all tiers, both arms) and commits the baselines; optionally a local Haiku 5.5 run on the Claude Code subscription confirms the PlatynUI and Appium tasks. Verify the eval gate reports no `rebaseline-needed`. + +## 8. Initial check + +- [ ] 8.1 After the re-baseline in 7.3, review the confirmed decisions against the results. How many new findings do the stricter checks produce, and how many of them are true? How did eval scores and turns move per task? Did the Windows live check behave the same as before? Write down the findings. If a stricter rule mostly produces noise, or the switch causes another problem, propose the changed decision to the maintainer and update `design.md`. Verify that the findings are recorded in this change.