Skip to content

feat(security): add security-model prepare / verify / update skills - #1156

Merged
potiuk merged 2 commits into
apache:mainfrom
potiuk:security-model-skills
Sep 8, 2026
Merged

feat(security): add security-model prepare / verify / update skills#1156
potiuk merged 2 commits into
apache:mainfrom
potiuk:security-model-skills

Conversation

@potiuk

@potiuk potiuk commented Sep 8, 2026

Copy link
Copy Markdown
Member

Summary

  • The security family routes inbound reports against a project's security model,
    but had no skills for producing or maintaining that model. Three skills close the
    loop: draft a first model and land it, pre-flight an existing one, and read the
    accrued triage decisions back into it.
  • The rubric for what a model contains is referenced by URL from
    alpha-omega-security/threat-model
    rather than vendored. A second copy of a specification is a second specification;
    what the framework owns is the workflow around it — consent, the discoverability
    chain, PR mechanics, confidentiality scrubbing, and the feedback loop.
  • Generalised from the ASF security team's own model-preparation workflow: PMC
    vocabulary is gone, replaced by <governance-body> / <upstream> /
    <project-config>, and the ASF-specific license header and apache/ repo prefix
    in the PR helper are now flags.
Skill Mode What it does
security-model-prepare Drafting Draft-first production of a first model; lands it plus the AGENTS.mdSECURITY.md chain as one PR per repository. Consent on the private list before the first repository write.
security-model-verify Triage Per-repository pre-flight. Discoverability is the only hard gate; completeness is graded, and every gap is a proposal. Mechanical gaps get a PR, substantive ones a private mail.
security-model-update Drafting Reads closed trackers, reporter threads and canned responses back into the model — known-non-finding entries plus a model-gap list.

The parts worth reviewing closely

security-model-update is fenced deliberately. Known non-findings sit first
in the model's disposition precedence and are fed to automated triagers verbatim as
a negative prompt, so a loose entry suppresses a whole class of reports ahead of
every other safeguard. The rules that follow from that:

  • Only a recurring BY-DESIGN: property-disclaimed close feeds the section. No
    OUT-OF-MODEL:* route is ever promoted — those sit below it in the precedence
    order, so relabelling one lifts it above the checks that decided it.
  • Report quality (no reproducer, reachability not demonstrated) is forbidden as
    a match condition. An unreproduced report stays open pending a reproducer.
  • The discharging claim must exist in the model and cover the entry's component.
  • The regression check is blocking: the proposed model is re-run against every
    historically fixed report, and one that would now close does not ship. Narrowing
    is the only permitted resolution — never widening.

Why mail and not a public issue for substantive findings: many projects have no
usable public tracker; a public list of a project's threat-model gaps is an
inventory a hostile researcher would mine; and maintainers who read mail may never
see a tracker notification. PRs are the exception — they need a repo write anyway
and carry no inventory.

Type of change

  • Skill change — eval fixtures included (35 cases, 7 steps)
  • Cross-cutting (security threat model: STRIDE family F, mitigations M.30–M.36)
  • Documentation (docs/security/security-model-preparation.md, modes,
    capabilities, mode-economics, marketplaces)
  • Project template (projects/_template/security-model.md)
  • Other: new spec tools/spec-loop/specs/security-model-preparation.md;
    helper script + tests under skills/security-model-verify/

Test plan

  • prek passes on every changed file (doctoc, SPDX, markdownlint, typos,
    lychee, check-placeholders, symlink-lint, check-family-plugins,
    skill-and-tool-validate, spec-validate)
  • skill-and-tool-validate clean — no violations, and the eval-coverage
    advisory is cleared for all three skills
  • Workspace checks: ruff 29/29 members, mypy 26/26, pytest 32/32 — including
    25 new unit tests on the PR helper's create-or-append merge core
  • Eval suites pass 35/35 (--cli "claude -p"): verify 11/11,
    prepare 9/9, update 15/15 — 0 failed, 0 manual, 0 errored
  • vendor-neutrality-score --in-place regenerated: 74/74 skills, 0
    vendor-coupled

Caveat on the eval numbers: that was a self-eval — the authoring model grading
its own fixtures. Per AGENTS.md that is a smoke test for the cheap failure class,
not independent evidence. A cross-model pass is worth running before merge.

RFC-AI-0004 compliance

  • HITL — every external write is shown and approved first; PRs open via
    gh pr create --web; security-model-update is read-only on the tracker
  • Sandbox — the helper is stdlib-only and reaches only git / gh
  • Vendor neutralitycheck-placeholders passes; the ASF-specific header
    and repo prefix are now flags with neutral defaults
  • Write-access discipline — drafts only; consent gate before the first
    repository write on a project you do not maintain
  • Privacy LLM — the update loop's scrub step runs before display, not
    before send

Notes for reviewers

Three things I'd flag rather than bury:

  1. I widened one taxonomy definition. capability:authoring was "skills that
    author or maintain other skills"; it now also covers the project's own security
    model. The alternative was mislabelling these as fix or resolve. Happy to
    add a new capability instead if you'd prefer the old definition held.
  2. STRIDE family F was scope I added. docs/security/threat-model.md is
    release-blocking and says each new Drafting skill ships with its own row, so
    leaving "the twelve skills" stale would have been a defect. It is a
    security-review artefact and wants a real read, not my say-so.
  3. tools/spec-loop/.last-sync is 12 commits behind main — pre-existing, not
    from this PR. I added a spec for this change but deliberately did not bump
    .last-sync, since that would falsely claim those 12 commits are spec-synced.
    Worth a companion sync PR.

Not included: eval coverage for the composition steps (PR body, mail body). Those
are prose and would need structural fixtures.

🤖 Generated with Claude Code

https://claude.ai/code/session_01So3JRGXrbqSGrohtZuHWKg

The security family routes inbound reports against a project's security
model, but had no skills for producing or maintaining that model. Three
skills close the loop:

- security-model-prepare — draft a first model in draft-first mode and
  land it, with its AGENTS.md -> SECURITY.md discoverability chain, as
  one reviewable PR per repository. Consent on the private list before
  the first repository write.
- security-model-verify — pre-flight an existing model per repository:
  discoverability (the only hard gate) and completeness against the
  minimum bar. Mechanical gaps get a PR, substantive ones a private
  mail, never a public issue.
- security-model-update — read the decision history back into the model.
  Produces known-non-finding entries and a model-gap list, gated by a
  blocking regression check: a proposal that would close a report the
  project historically fixed does not ship, and narrowing is the only
  permitted resolution.

The rubric for what a model contains is referenced by URL from
alpha-omega-security/threat-model rather than vendored, so the framework
holds no divergent copy of a specification it does not own.

Ships the model-PR helper (create-or-append AGENTS.md / SECURITY.md
scaffold, pure merge core unit-tested), eval suites for all three
skills, STRIDE family F in the security threat model, and the adopter
config the skills read from <project-config>/security-model.md.

Generated-by: Claude Code (Opus 5)
The hand-written argv list and a few long string expressions did not
match ruff's formatting, which the workspace ruff-format check enforces
across every member.

Generated-by: Claude Code (Opus 5)
@potiuk

potiuk commented Sep 8, 2026

Copy link
Copy Markdown
Member Author

ported from the security scans.

@potiuk
potiuk merged commit 183a163 into apache:main Sep 8, 2026
11 checks passed
potiuk added a commit that referenced this pull request Sep 8, 2026
* fix(docs): reconcile documentation with the state of the tree

Closes a batch of doc-vs-reality inconsistencies, most of them filed as
issues after an earlier audit pass.

Reported (#882-#888, #935):
- labels-and-capabilities: the family:* section documented 4 of the 10
  canonical families and conflated them with three repo-area labels that
  are rejected in frontmatter; "multi-capability tools are rare, gmail is
  the only one" sat above a table of ten. A stray blank line was also
  splitting the tool-capability table in two.
- adapters/registry: the mail-source and mail-archive rows omitted
  adapters that declare those contracts. Fossil is NOT misclassified —
  it is deliberately both a forge bridge and a VCS backend; the real bug
  was the opposite, tools/vcs omitting Fossil from its backend table.
- adapters/add-a-harness: detection table missing Kiro; stale harness
  values for agent-guard, permission-audit, sandbox-lint, spec-loop.
- setup/personal-use-unadopted-repo and AGENTS.md: the per-skill override
  chain listed an organization layer that does not exist. Organizations
  supply config-key defaults, not skill overrides.
- setup/mixed-adoption-teams: the symlink recipe linked through the relay
  dir, producing four hops instead of two.
- RFC-AI-0002 and RFC-AI-0004: status said Draft while the body described
  shipped, enforced behaviour. RFC-AI-0004 also said "four shifts" above a
  four-item list in a document that says six everywhere else; the two
  missing gap bullets are added rather than the numeral flipped.
- spec-loop/organization-adapters: #935's bullet was already fixed by
  #994; the rest of the Known gaps section was stale the same way.

Found while verifying, not separately reported:
- Skill counts: security 12->15 and the modes glance table (Triage 34->35,
  Drafting 9->11) went stale in #1156. repo-health 6->7, the CONTRIBUTING
  tree's four prefix counts, and one of three skill counts in
  contributor-growth were already drifting.
- spec-loop/adapters: Bitbucket write coverage gained PR-comment creation.
- spec-loop/project-agnosticism: the non-ASF smoke suite is 15 cases over
  6 steps across 5 skills, not 6 cases over 2; and the catalogue produces
  3 asf-coupling warnings, not 0.

Generated-by: Claude Code (Opus 5)

* fix(skill-and-tool-validator): skip dot-directories under skills/

The eval-coverage check globbed skills/.mypy_cache, .pytest_cache,
.ruff_cache and .claude as if each were a skill, so a dirty working tree
produced four phantom "no eval suite" warnings. Those directories are
gitignored, so they exist only locally — exactly when a bogus warning is
hardest to act on, because it names a path the reader cannot find in the
repo.

One enumeration already had the guard; the other four did not. Adds it to
all of them, with a regression test.

Generated-by: Claude Code (Opus 5)
potiuk added a commit that referenced this pull request Sep 8, 2026
The marker sat at a1cff44, 17 commits behind main. Bumping it alone
would claim those commits are described by the specs, so the drift is
closed first.

Specs updated for what actually shipped:

- meta-and-quality-tooling: skill-evals errors, rather than passing, when
  a case's CLI produced no gradeable output (#1161).
- security-reporting: the tracker dashboard projects the current partial
  bucket to its end-of-bucket value, splitting RATE series (accumulate
  from zero) from LEVEL series (carry over), and deliberately not
  projecting mean-based signals (#1158).
- project-agnosticism: <PROJECT> and <project> are two placeholders
  holding different values, and the lint carries both spellings plus
  spaced variants (#1154).
- adapters: the forwarder relay's contact_handle defaults to an
  org-level shared inbox rather than a named individual (#1135). The
  multi-hop coordinator case is designed in RFC-AI-0008 and unimplemented.
- issue-management-family: the family's eval suites, and the note that
  --cli runs belong outside a credential-denying sandbox (#1145).

Commits needing no spec change: #1152, #1143 and #1156 updated their own
specs in-commit; #1149, #1147, #1151 are behaviour-preserving bug fixes;
#1155 and #1141 are CI and dependency chores; #1159's spec edits landed
with it; #1144 removes hardcoded literals that no spec asserted.

One genuine gap recorded rather than papered over: no spec covers
marketplace distribution or the dev-version stamping rule from #1160,
which is load-bearing because `claude plugin update` compares version
strings, not commit SHAs. Logged in adoption-and-setup as wanting its own
spec.

Generated-by: Claude Code (Opus 5)
potiuk added a commit that referenced this pull request Sep 8, 2026
…1163)

Closes the gap #1162 recorded rather than papered over: the plugin
marketplace surface had no spec at all.

`marketplace-distribution.md` covers the two manifest families (the
vendor-neutral Agent Plugins 1.0 root manifest with its closed ten-field
schema, and the five client-specific ones), the ten Claude-Code-only
per-family plugins, and the generator/CI gate that keeps them in step
with `family:` frontmatter.

The load-bearing part is the versioning rule. The marketplace is served
from `main`, so adopters install dev versions as the normal case, and
`claude plugin update` compares version strings rather than commit SHAs
— a frozen `.devN` suffix leaves an adopter told they are "already at the
latest version" indefinitely, recoverable only by a full uninstall and
reinstall. Recorded with the reason, along with the fact that nothing
checks the stamp is fresh, which is the one failure mode that is silent
in exactly the wrong direction.

Status is `experimental`, not `stable`: only the Claude Code manifests
have been exercised against a live install. The rest conform to their
vendors' published documentation and nothing more, which the Known gaps
now say.

While registering it: ten specs were absent from both index files
(`overview.md` and `README.md`) — among them `security-model-preparation`
from #1156 and every per-family spec. All ten are now listed in both, so
the indexes are complete.

Also corrects `docs/setup/marketplaces.md`, which said 41 of 70 skill
descriptions carry placeholder syntax; recounted from live frontmatter it
is 45 of 74.

Generated-by: Claude Code (Opus 5)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant