Skip to content

fix(docs): reconcile documentation with the state of the tree - #1159

Merged
potiuk merged 2 commits into
apache:mainfrom
potiuk:docs-consistency-sweep
Sep 8, 2026
Merged

fix(docs): reconcile documentation with the state of the tree#1159
potiuk merged 2 commits into
apache:mainfrom
potiuk:docs-consistency-sweep

Conversation

@potiuk

@potiuk potiuk commented Sep 8, 2026

Copy link
Copy Markdown
Member

Summary

Reported issues

Issue What was actually wrong
#882 The family:* section documented 4 of the 10 canonical families, and conflated them with three repo-area labels (family:tools/ci/docs) that the validator rejects in frontmatter. Separately, "multi-capability tools are rare — tools/gmail is the only one today" sat directly above a table containing ten of them, and gmail declares three contracts, not the two the intro claimed. A stray blank line was also splitting the tool-capability table in two, so everything below contract:mail-archive rendered headerless.
#883 The per-skill override chain listed an organization layer that does not exist. AGENTS.md had the identical defect — now fixed too, with an explicit sentence: organizations supply config-key defaults, not skill overrides.
#884 The symlink recipe iterated the relay dir, producing four hops where the enforced rule is two.
#885 Half disproved. Gmail was indeed missing from mail-source (so were maildir and ponymail). But Fossil is not misclassified — tools/fossil declares contract:tracker + contract:source-control, tools/vcs has a complete FossilBackend with tests, and it is already listed under Forge/tracker. The real Fossil bug is the inverse: tools/vcs/README.md and labels-and-capabilities.md both omitted the backend. Fixed that instead.
#886 All three claims confirmed, plus two more stale rows in the same table (agent-guard, spec-loop both missing Kiro).
#887, #888 Status said Draft while each body described shipped, enforced behaviour. Set to Implemented, not Stable — each RFC's own promotion gate requires public review plus pilot validation, neither of which has happened. RFC-AI-0004 also said "the four shifts" above a four-item list in a document that says six everywhere else; the two missing bullets (Principles 5 and 6) are added rather than the numeral flipped.
#935 Already fixed by #994. Verified that, then fixed the rest of the section, which was stale the same way — all three Known-gaps bullets described work that has since shipped.

Found while verifying

Skill-count drift, some of it mine:

  • README.md security 1215 and the docs/modes.md glance table
    (Triage 3435, Drafting 911) went stale in feat(security): add security-model prepare / verify / update skills #1156 — I added the
    skills and the mode-section rows but never the summary counts. This was
    the source of the validator's two modes-doc warnings.
  • Pre-existing: README.md repo-health 67, the CONTRIBUTING.md tree's
    four prefix counts (all wrong), and one of three skill counts in
    docs/contributor-growth/README.md.

Note the modes glance column tracks mode: frontmatter, not section rows —
Mentoring declares 7 with 10 rows and Pairing declares 3 with 4, and both are
correct, because some skills are deliberately cross-listed. Only Triage and
Drafting were genuinely wrong.

Stale spec claims:

  • specs/adapters.md said Bitbucket writes are "limited to Cloud
    issue-comment creation"; feat(bitbucket): add guarded cloud PR comment create #1142 added PR-comment creation.
  • specs/project-agnosticism.md described the non-ASF smoke suite as
    "6 cases across 2 steps" driving one skill — it is 15 cases across 6 steps
    driving five — and claimed the catalogue produces 0 asf-coupling
    warnings when it produces 3.

Tool fix

skill-and-tool-validator enumerated skills/.mypy_cache, .pytest_cache,
.ruff_cache and .claude as skills, so a dirty working tree produced four
phantom "no eval suite" warnings naming paths the reader cannot find in the
repo. One of the five enumerations already had the dot-dir guard; the other
four now do, with a regression test.

Validator advisory warnings: 36 → 30.

Test plan

  • prek run --all-files passes (exit 0)
  • skill-and-tool-validate — 0 violations; the 4 eval-coverage and 2
    modes-doc warnings are gone
  • spec-validate clean
  • Workspace: ruff 29/29, ruff-format 29/29, mypy 26/26, pytest 32/32
  • symlink-lint, check-family-plugins, check-workspace-members,
    check-placeholders all clean
  • Counts re-derived from live family:/mode: frontmatter, not from
    skill-name prefixes

Notes for reviewers

A correction to #1156's test plan. That PR's body claims the three
security-model-* eval suites passed 35/35. That number is void. Sandboxed
claude -p returns Not logged in and exits 0; the runner's documented
fallback wraps non-JSON stdout as {"raw_output": …}, and the
intersection-only comparator then passes any case whose expected.json
declares no raw_output key. Nothing was graded.

Re-run outside the sandbox, the real result is 28/35 — 7 failures:

Suite Result Failing cases
security-model-prepare 8/9 step-4-provenance/case-4-hedge-tag-attempt
security-model-verify 9/11 step-a/case-6-injection-claims-preapproved, step-b/case-4-bare-headings
security-model-update 11/15 step-2/case-2-single-occurrence, step-2/case-3-report-quality-condition, step-2/case-5-claim-covers-wrong-component, step-3/case-4-two-offenders-with-a-widening-suggestion

Those skills are already on main and are not touched by this PR. Follow-up
will diagnose each failure as either an over-specified fixture or a genuine
gap in the skill prose, and fix accordingly.

Related, deliberately not in this PR: the harness passes a case when the
CLI emits no JSON at all. --exact guards it, the default does not. Making
"no JSON extracted" an ERROR by default would change results across all ~70
suites, which wants its own PR and a full validation run.

tools/spec-loop/.last-sync remains 13 commits behind main; this PR
corrects two stale specs but does not bump it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01So3JRGXrbqSGrohtZuHWKg

Closes a batch of doc-vs-reality inconsistencies, most of them filed as
issues after an earlier audit pass.

Reported (apache#882-apache#888, apache#935):
- labels-and-capabilities: the family:* section documented 4 of the 10
  canonical families and conflated them with three repo-area labels that
  are rejected in frontmatter; "multi-capability tools are rare, gmail is
  the only one" sat above a table of ten. A stray blank line was also
  splitting the tool-capability table in two.
- adapters/registry: the mail-source and mail-archive rows omitted
  adapters that declare those contracts. Fossil is NOT misclassified —
  it is deliberately both a forge bridge and a VCS backend; the real bug
  was the opposite, tools/vcs omitting Fossil from its backend table.
- adapters/add-a-harness: detection table missing Kiro; stale harness
  values for agent-guard, permission-audit, sandbox-lint, spec-loop.
- setup/personal-use-unadopted-repo and AGENTS.md: the per-skill override
  chain listed an organization layer that does not exist. Organizations
  supply config-key defaults, not skill overrides.
- setup/mixed-adoption-teams: the symlink recipe linked through the relay
  dir, producing four hops instead of two.
- RFC-AI-0002 and RFC-AI-0004: status said Draft while the body described
  shipped, enforced behaviour. RFC-AI-0004 also said "four shifts" above a
  four-item list in a document that says six everywhere else; the two
  missing gap bullets are added rather than the numeral flipped.
- spec-loop/organization-adapters: apache#935's bullet was already fixed by
  apache#994; the rest of the Known gaps section was stale the same way.

Found while verifying, not separately reported:
- Skill counts: security 12->15 and the modes glance table (Triage 34->35,
  Drafting 9->11) went stale in apache#1156. repo-health 6->7, the CONTRIBUTING
  tree's four prefix counts, and one of three skill counts in
  contributor-growth were already drifting.
- spec-loop/adapters: Bitbucket write coverage gained PR-comment creation.
- spec-loop/project-agnosticism: the non-ASF smoke suite is 15 cases over
  6 steps across 5 skills, not 6 cases over 2; and the catalogue produces
  3 asf-coupling warnings, not 0.

Generated-by: Claude Code (Opus 5)
The eval-coverage check globbed skills/.mypy_cache, .pytest_cache,
.ruff_cache and .claude as if each were a skill, so a dirty working tree
produced four phantom "no eval suite" warnings. Those directories are
gitignored, so they exist only locally — exactly when a bogus warning is
hardest to act on, because it names a path the reader cannot find in the
repo.

One enumeration already had the guard; the other four did not. Adds it to
all of them, with a regression test.

Generated-by: Claude Code (Opus 5)
@potiuk
potiuk merged commit 3d9b2c9 into apache:main Sep 8, 2026
10 checks passed
potiuk added a commit that referenced this pull request Sep 8, 2026
The marker sat at a1cff44, 17 commits behind main. Bumping it alone
would claim those commits are described by the specs, so the drift is
closed first.

Specs updated for what actually shipped:

- meta-and-quality-tooling: skill-evals errors, rather than passing, when
  a case's CLI produced no gradeable output (#1161).
- security-reporting: the tracker dashboard projects the current partial
  bucket to its end-of-bucket value, splitting RATE series (accumulate
  from zero) from LEVEL series (carry over), and deliberately not
  projecting mean-based signals (#1158).
- project-agnosticism: <PROJECT> and <project> are two placeholders
  holding different values, and the lint carries both spellings plus
  spaced variants (#1154).
- adapters: the forwarder relay's contact_handle defaults to an
  org-level shared inbox rather than a named individual (#1135). The
  multi-hop coordinator case is designed in RFC-AI-0008 and unimplemented.
- issue-management-family: the family's eval suites, and the note that
  --cli runs belong outside a credential-denying sandbox (#1145).

Commits needing no spec change: #1152, #1143 and #1156 updated their own
specs in-commit; #1149, #1147, #1151 are behaviour-preserving bug fixes;
#1155 and #1141 are CI and dependency chores; #1159's spec edits landed
with it; #1144 removes hardcoded literals that no spec asserted.

One genuine gap recorded rather than papered over: no spec covers
marketplace distribution or the dev-version stamping rule from #1160,
which is load-bearing because `claude plugin update` compares version
strings, not commit SHAs. Logged in adoption-and-setup as wanting its own
spec.

Generated-by: Claude Code (Opus 5)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant