feat(security): add security-model prepare / verify / update skills - #1156
Merged
Conversation
The security family routes inbound reports against a project's security model, but had no skills for producing or maintaining that model. Three skills close the loop: - security-model-prepare — draft a first model in draft-first mode and land it, with its AGENTS.md -> SECURITY.md discoverability chain, as one reviewable PR per repository. Consent on the private list before the first repository write. - security-model-verify — pre-flight an existing model per repository: discoverability (the only hard gate) and completeness against the minimum bar. Mechanical gaps get a PR, substantive ones a private mail, never a public issue. - security-model-update — read the decision history back into the model. Produces known-non-finding entries and a model-gap list, gated by a blocking regression check: a proposal that would close a report the project historically fixed does not ship, and narrowing is the only permitted resolution. The rubric for what a model contains is referenced by URL from alpha-omega-security/threat-model rather than vendored, so the framework holds no divergent copy of a specification it does not own. Ships the model-PR helper (create-or-append AGENTS.md / SECURITY.md scaffold, pure merge core unit-tested), eval suites for all three skills, STRIDE family F in the security threat model, and the adopter config the skills read from <project-config>/security-model.md. Generated-by: Claude Code (Opus 5)
The hand-written argv list and a few long string expressions did not match ruff's formatting, which the workspace ruff-format check enforces across every member. Generated-by: Claude Code (Opus 5)
Member
Author
|
ported from the security scans. |
6 tasks
potiuk
added a commit
that referenced
this pull request
Sep 8, 2026
* fix(docs): reconcile documentation with the state of the tree Closes a batch of doc-vs-reality inconsistencies, most of them filed as issues after an earlier audit pass. Reported (#882-#888, #935): - labels-and-capabilities: the family:* section documented 4 of the 10 canonical families and conflated them with three repo-area labels that are rejected in frontmatter; "multi-capability tools are rare, gmail is the only one" sat above a table of ten. A stray blank line was also splitting the tool-capability table in two. - adapters/registry: the mail-source and mail-archive rows omitted adapters that declare those contracts. Fossil is NOT misclassified — it is deliberately both a forge bridge and a VCS backend; the real bug was the opposite, tools/vcs omitting Fossil from its backend table. - adapters/add-a-harness: detection table missing Kiro; stale harness values for agent-guard, permission-audit, sandbox-lint, spec-loop. - setup/personal-use-unadopted-repo and AGENTS.md: the per-skill override chain listed an organization layer that does not exist. Organizations supply config-key defaults, not skill overrides. - setup/mixed-adoption-teams: the symlink recipe linked through the relay dir, producing four hops instead of two. - RFC-AI-0002 and RFC-AI-0004: status said Draft while the body described shipped, enforced behaviour. RFC-AI-0004 also said "four shifts" above a four-item list in a document that says six everywhere else; the two missing gap bullets are added rather than the numeral flipped. - spec-loop/organization-adapters: #935's bullet was already fixed by #994; the rest of the Known gaps section was stale the same way. Found while verifying, not separately reported: - Skill counts: security 12->15 and the modes glance table (Triage 34->35, Drafting 9->11) went stale in #1156. repo-health 6->7, the CONTRIBUTING tree's four prefix counts, and one of three skill counts in contributor-growth were already drifting. - spec-loop/adapters: Bitbucket write coverage gained PR-comment creation. - spec-loop/project-agnosticism: the non-ASF smoke suite is 15 cases over 6 steps across 5 skills, not 6 cases over 2; and the catalogue produces 3 asf-coupling warnings, not 0. Generated-by: Claude Code (Opus 5) * fix(skill-and-tool-validator): skip dot-directories under skills/ The eval-coverage check globbed skills/.mypy_cache, .pytest_cache, .ruff_cache and .claude as if each were a skill, so a dirty working tree produced four phantom "no eval suite" warnings. Those directories are gitignored, so they exist only locally — exactly when a bogus warning is hardest to act on, because it names a path the reader cannot find in the repo. One enumeration already had the guard; the other four did not. Adds it to all of them, with a regression test. Generated-by: Claude Code (Opus 5)
This was referenced Sep 8, 2026
potiuk
added a commit
that referenced
this pull request
Sep 8, 2026
The marker sat at a1cff44, 17 commits behind main. Bumping it alone would claim those commits are described by the specs, so the drift is closed first. Specs updated for what actually shipped: - meta-and-quality-tooling: skill-evals errors, rather than passing, when a case's CLI produced no gradeable output (#1161). - security-reporting: the tracker dashboard projects the current partial bucket to its end-of-bucket value, splitting RATE series (accumulate from zero) from LEVEL series (carry over), and deliberately not projecting mean-based signals (#1158). - project-agnosticism: <PROJECT> and <project> are two placeholders holding different values, and the lint carries both spellings plus spaced variants (#1154). - adapters: the forwarder relay's contact_handle defaults to an org-level shared inbox rather than a named individual (#1135). The multi-hop coordinator case is designed in RFC-AI-0008 and unimplemented. - issue-management-family: the family's eval suites, and the note that --cli runs belong outside a credential-denying sandbox (#1145). Commits needing no spec change: #1152, #1143 and #1156 updated their own specs in-commit; #1149, #1147, #1151 are behaviour-preserving bug fixes; #1155 and #1141 are CI and dependency chores; #1159's spec edits landed with it; #1144 removes hardcoded literals that no spec asserted. One genuine gap recorded rather than papered over: no spec covers marketplace distribution or the dev-version stamping rule from #1160, which is load-bearing because `claude plugin update` compares version strings, not commit SHAs. Logged in adoption-and-setup as wanting its own spec. Generated-by: Claude Code (Opus 5)
4 tasks
potiuk
added a commit
that referenced
this pull request
Sep 8, 2026
…1163) Closes the gap #1162 recorded rather than papered over: the plugin marketplace surface had no spec at all. `marketplace-distribution.md` covers the two manifest families (the vendor-neutral Agent Plugins 1.0 root manifest with its closed ten-field schema, and the five client-specific ones), the ten Claude-Code-only per-family plugins, and the generator/CI gate that keeps them in step with `family:` frontmatter. The load-bearing part is the versioning rule. The marketplace is served from `main`, so adopters install dev versions as the normal case, and `claude plugin update` compares version strings rather than commit SHAs — a frozen `.devN` suffix leaves an adopter told they are "already at the latest version" indefinitely, recoverable only by a full uninstall and reinstall. Recorded with the reason, along with the fact that nothing checks the stamp is fresh, which is the one failure mode that is silent in exactly the wrong direction. Status is `experimental`, not `stable`: only the Claude Code manifests have been exercised against a live install. The rest conform to their vendors' published documentation and nothing more, which the Known gaps now say. While registering it: ten specs were absent from both index files (`overview.md` and `README.md`) — among them `security-model-preparation` from #1156 and every per-family spec. All ten are now listed in both, so the indexes are complete. Also corrects `docs/setup/marketplaces.md`, which said 41 of 70 skill descriptions carry placeholder syntax; recounted from live frontmatter it is 45 of 74. Generated-by: Claude Code (Opus 5)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
but had no skills for producing or maintaining that model. Three skills close the
loop: draft a first model and land it, pre-flight an existing one, and read the
accrued triage decisions back into it.
alpha-omega-security/threat-model
rather than vendored. A second copy of a specification is a second specification;
what the framework owns is the workflow around it — consent, the discoverability
chain, PR mechanics, confidentiality scrubbing, and the feedback loop.
vocabulary is gone, replaced by
<governance-body>/<upstream>/<project-config>, and the ASF-specific license header andapache/repo prefixin the PR helper are now flags.
security-model-prepareAGENTS.md→SECURITY.mdchain as one PR per repository. Consent on the private list before the first repository write.security-model-verifysecurity-model-updateThe parts worth reviewing closely
security-model-updateis fenced deliberately. Known non-findings sit firstin the model's disposition precedence and are fed to automated triagers verbatim as
a negative prompt, so a loose entry suppresses a whole class of reports ahead of
every other safeguard. The rules that follow from that:
BY-DESIGN: property-disclaimedclose feeds the section. NoOUT-OF-MODEL:*route is ever promoted — those sit below it in the precedenceorder, so relabelling one lifts it above the checks that decided it.
a match condition. An unreproduced report stays open pending a reproducer.
historically fixed report, and one that would now close does not ship. Narrowing
is the only permitted resolution — never widening.
Why mail and not a public issue for substantive findings: many projects have no
usable public tracker; a public list of a project's threat-model gaps is an
inventory a hostile researcher would mine; and maintainers who read mail may never
see a tracker notification. PRs are the exception — they need a repo write anyway
and carry no inventory.
Type of change
docs/security/security-model-preparation.md, modes,capabilities, mode-economics, marketplaces)
projects/_template/security-model.md)tools/spec-loop/specs/security-model-preparation.md;helper script + tests under
skills/security-model-verify/Test plan
prekpasses on every changed file (doctoc, SPDX, markdownlint, typos,lychee, check-placeholders, symlink-lint, check-family-plugins,
skill-and-tool-validate, spec-validate)
skill-and-tool-validateclean — no violations, and the eval-coverageadvisory is cleared for all three skills
25 new unit tests on the PR helper's create-or-append merge core
--cli "claude -p"): verify 11/11,prepare 9/9, update 15/15 — 0 failed, 0 manual, 0 errored
vendor-neutrality-score --in-placeregenerated: 74/74 skills, 0vendor-coupled
Caveat on the eval numbers: that was a self-eval — the authoring model grading
its own fixtures. Per
AGENTS.mdthat is a smoke test for the cheap failure class,not independent evidence. A cross-model pass is worth running before merge.
RFC-AI-0004 compliance
gh pr create --web;security-model-updateis read-only on the trackergit/ghcheck-placeholderspasses; the ASF-specific headerand repo prefix are now flags with neutral defaults
repository write on a project you do not maintain
before send
Notes for reviewers
Three things I'd flag rather than bury:
capability:authoringwas "skills thatauthor or maintain other skills"; it now also covers the project's own security
model. The alternative was mislabelling these as
fixorresolve. Happy toadd a new capability instead if you'd prefer the old definition held.
docs/security/threat-model.mdisrelease-blocking and says each new Drafting skill ships with its own row, so
leaving "the twelve skills" stale would have been a defect. It is a
security-review artefact and wants a real read, not my say-so.
tools/spec-loop/.last-syncis 12 commits behindmain— pre-existing, notfrom this PR. I added a spec for this change but deliberately did not bump
.last-sync, since that would falsely claim those 12 commits are spec-synced.Worth a companion sync PR.
Not included: eval coverage for the composition steps (PR body, mail body). Those
are prose and would need structural fixtures.
🤖 Generated with Claude Code
https://claude.ai/code/session_01So3JRGXrbqSGrohtZuHWKg