fix(docs): reconcile documentation with the state of the tree - #1159
Merged
Conversation
Closes a batch of doc-vs-reality inconsistencies, most of them filed as issues after an earlier audit pass. Reported (apache#882-apache#888, apache#935): - labels-and-capabilities: the family:* section documented 4 of the 10 canonical families and conflated them with three repo-area labels that are rejected in frontmatter; "multi-capability tools are rare, gmail is the only one" sat above a table of ten. A stray blank line was also splitting the tool-capability table in two. - adapters/registry: the mail-source and mail-archive rows omitted adapters that declare those contracts. Fossil is NOT misclassified — it is deliberately both a forge bridge and a VCS backend; the real bug was the opposite, tools/vcs omitting Fossil from its backend table. - adapters/add-a-harness: detection table missing Kiro; stale harness values for agent-guard, permission-audit, sandbox-lint, spec-loop. - setup/personal-use-unadopted-repo and AGENTS.md: the per-skill override chain listed an organization layer that does not exist. Organizations supply config-key defaults, not skill overrides. - setup/mixed-adoption-teams: the symlink recipe linked through the relay dir, producing four hops instead of two. - RFC-AI-0002 and RFC-AI-0004: status said Draft while the body described shipped, enforced behaviour. RFC-AI-0004 also said "four shifts" above a four-item list in a document that says six everywhere else; the two missing gap bullets are added rather than the numeral flipped. - spec-loop/organization-adapters: apache#935's bullet was already fixed by apache#994; the rest of the Known gaps section was stale the same way. Found while verifying, not separately reported: - Skill counts: security 12->15 and the modes glance table (Triage 34->35, Drafting 9->11) went stale in apache#1156. repo-health 6->7, the CONTRIBUTING tree's four prefix counts, and one of three skill counts in contributor-growth were already drifting. - spec-loop/adapters: Bitbucket write coverage gained PR-comment creation. - spec-loop/project-agnosticism: the non-ASF smoke suite is 15 cases over 6 steps across 5 skills, not 6 cases over 2; and the catalogue produces 3 asf-coupling warnings, not 0. Generated-by: Claude Code (Opus 5)
The eval-coverage check globbed skills/.mypy_cache, .pytest_cache, .ruff_cache and .claude as if each were a skill, so a dirty working tree produced four phantom "no eval suite" warnings. Those directories are gitignored, so they exist only locally — exactly when a bogus warning is hardest to act on, because it names a path the reader cannot find in the repo. One enumeration already had the guard; the other four did not. Adds it to all of them, with a regression test. Generated-by: Claude Code (Opus 5)
This was referenced Sep 8, 2026
Closed
Closed
potiuk
added a commit
that referenced
this pull request
Sep 8, 2026
The marker sat at a1cff44, 17 commits behind main. Bumping it alone would claim those commits are described by the specs, so the drift is closed first. Specs updated for what actually shipped: - meta-and-quality-tooling: skill-evals errors, rather than passing, when a case's CLI produced no gradeable output (#1161). - security-reporting: the tracker dashboard projects the current partial bucket to its end-of-bucket value, splitting RATE series (accumulate from zero) from LEVEL series (carry over), and deliberately not projecting mean-based signals (#1158). - project-agnosticism: <PROJECT> and <project> are two placeholders holding different values, and the lint carries both spellings plus spaced variants (#1154). - adapters: the forwarder relay's contact_handle defaults to an org-level shared inbox rather than a named individual (#1135). The multi-hop coordinator case is designed in RFC-AI-0008 and unimplemented. - issue-management-family: the family's eval suites, and the note that --cli runs belong outside a credential-denying sandbox (#1145). Commits needing no spec change: #1152, #1143 and #1156 updated their own specs in-commit; #1149, #1147, #1151 are behaviour-preserving bug fixes; #1155 and #1141 are CI and dependency chores; #1159's spec edits landed with it; #1144 removes hardcoded literals that no spec asserted. One genuine gap recorded rather than papered over: no spec covers marketplace distribution or the dev-version stamping rule from #1160, which is load-bearing because `claude plugin update` compares version strings, not commit SHAs. Logged in adoption-and-setup as wanting its own spec. Generated-by: Claude Code (Opus 5)
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
(docs/labels-and-capabilities.md: family:* section incomplete, self-contradicting statement #882–docs/rfcs/RFC-AI-0004.md: status says "Draft", text says implementation operationalises every principle #888, Update organization-adapters.md: the organizations/ structural check has shipped #935), plus the same defect class found while verifying them.
out to be wrong or already fixed — those are called out below rather than
edited to match.
skills/as ifthey were skills.
Reported issues
family:*section documented 4 of the 10 canonical families, and conflated them with three repo-area labels (family:tools/ci/docs) that the validator rejects in frontmatter. Separately, "multi-capability tools are rare —tools/gmailis the only one today" sat directly above a table containing ten of them, and gmail declares three contracts, not the two the intro claimed. A stray blank line was also splitting the tool-capability table in two, so everything belowcontract:mail-archiverendered headerless.organizationlayer that does not exist.AGENTS.mdhad the identical defect — now fixed too, with an explicit sentence: organizations supply config-key defaults, not skill overrides.mail-source(so weremaildirandponymail). But Fossil is not misclassified —tools/fossildeclarescontract:tracker + contract:source-control,tools/vcshas a completeFossilBackendwith tests, and it is already listed under Forge/tracker. The real Fossil bug is the inverse:tools/vcs/README.mdandlabels-and-capabilities.mdboth omitted the backend. Fixed that instead.agent-guard,spec-loopboth missing Kiro).Draftwhile each body described shipped, enforced behaviour. Set toImplemented, notStable— each RFC's own promotion gate requires public review plus pilot validation, neither of which has happened. RFC-AI-0004 also said "the four shifts" above a four-item list in a document that says six everywhere else; the two missing bullets (Principles 5 and 6) are added rather than the numeral flipped.Found while verifying
Skill-count drift, some of it mine:
README.mdsecurity12→15and thedocs/modes.mdglance table(Triage
34→35, Drafting9→11) went stale in feat(security): add security-model prepare / verify / update skills #1156 — I added theskills and the mode-section rows but never the summary counts. This was
the source of the validator's two
modes-docwarnings.README.mdrepo-health6→7, theCONTRIBUTING.mdtree'sfour prefix counts (all wrong), and one of three skill counts in
docs/contributor-growth/README.md.Note the modes glance column tracks
mode:frontmatter, not section rows —Mentoring declares 7 with 10 rows and Pairing declares 3 with 4, and both are
correct, because some skills are deliberately cross-listed. Only Triage and
Drafting were genuinely wrong.
Stale spec claims:
specs/adapters.mdsaid Bitbucket writes are "limited to Cloudissue-comment creation"; feat(bitbucket): add guarded cloud PR comment create #1142 added PR-comment creation.
specs/project-agnosticism.mddescribed the non-ASF smoke suite as"6 cases across 2 steps" driving one skill — it is 15 cases across 6 steps
driving five — and claimed the catalogue produces 0 asf-coupling
warnings when it produces 3.
Tool fix
skill-and-tool-validatorenumeratedskills/.mypy_cache,.pytest_cache,.ruff_cacheand.claudeas skills, so a dirty working tree produced fourphantom "no eval suite" warnings naming paths the reader cannot find in the
repo. One of the five enumerations already had the dot-dir guard; the other
four now do, with a regression test.
Validator advisory warnings: 36 → 30.
Test plan
prek run --all-filespasses (exit 0)skill-and-tool-validate— 0 violations; the 4 eval-coverage and 2modes-doc warnings are gone
spec-validatecleansymlink-lint,check-family-plugins,check-workspace-members,check-placeholdersall cleanfamily:/mode:frontmatter, not fromskill-name prefixes
Notes for reviewers
A correction to #1156's test plan. That PR's body claims the three
security-model-*eval suites passed 35/35. That number is void. Sandboxedclaude -preturnsNot logged inand exits 0; the runner's documentedfallback wraps non-JSON stdout as
{"raw_output": …}, and theintersection-only comparator then passes any case whose
expected.jsondeclares no
raw_outputkey. Nothing was graded.Re-run outside the sandbox, the real result is 28/35 — 7 failures:
security-model-preparestep-4-provenance/case-4-hedge-tag-attemptsecurity-model-verifystep-a/case-6-injection-claims-preapproved,step-b/case-4-bare-headingssecurity-model-updatestep-2/case-2-single-occurrence,step-2/case-3-report-quality-condition,step-2/case-5-claim-covers-wrong-component,step-3/case-4-two-offenders-with-a-widening-suggestionThose skills are already on
mainand are not touched by this PR. Follow-upwill diagnose each failure as either an over-specified fixture or a genuine
gap in the skill prose, and fix accordingly.
Related, deliberately not in this PR: the harness passes a case when the
CLI emits no JSON at all.
--exactguards it, the default does not. Making"no JSON extracted" an ERROR by default would change results across all ~70
suites, which wants its own PR and a full validation run.
tools/spec-loop/.last-syncremains 13 commits behindmain; this PRcorrects two stale specs but does not bump it.
🤖 Generated with Claude Code
https://claude.ai/code/session_01So3JRGXrbqSGrohtZuHWKg