Skip to content

Add review-doc for writing review of docs.page pages - #558

Open
BenInvertase wants to merge 20 commits into
mainfrom
add-check-docs-skill
Open

BenInvertase wants to merge 20 commits into
mainfrom
add-check-docs-skill

Conversation

@BenInvertase

@BenInvertase BenInvertase commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds review-doc, a writing review for one docs.page page.

Technical writing is a profession. What makes a page easy to scan, act on, and trust is not common knowledge — mixed voice, vague links, padding, and unexplained jargon slip through even when the facts are right. Agents do not pick that up on their own. This skill is a utility that holds them to a best-practice bar: 16 writing checks, a results table after every run, then fix everything, decide one by one, or mute a check.

Say use review-doc on docs/quickstart.mdx (or “review doc”). Usage: skills/review-doc/README.md.

Eval results

We ran the same gold pages with the skill and without it. Fixtures: skills/review-doc/evals/.

Without the skill, agents often catch obvious wording issues. With it, they also follow the review (table, skip product idioms, group overlapping edits, mute, accept/reject).

Case With skill Without skill
Find planted writing issues (review only) 8/8 5/8
Leave valid docs.page idioms alone 9/9 8/9
Fix planted issues, keep <Info> 6/6 6/6
Group overlapping findings, do not edit 4/4 4/4
Mute a check so it does not come back 4/4 2/4
Accept one finding, reject another 4/4 3/4
Total 35/35 (100%) 28/35 (80%)

Each of the 88 writing rules also has a tiny gold page (hit, and skip when the rule says skip). 172/172 passed.

Scope

  • app/ (hosted site, MCP, Ask AI)
  • packages/cli/
  • packages/mdx-bundler/
  • docs/ (product documentation)
  • Repo / CI / other

Type of change

  • Bug fix
  • New feature
  • Documentation
  • Refactor / chore

Test plan

  • bun run check passes locally
  • use review-doc on docs/<page>.mdx prints results after every run
  • fix all / review each / mute check <id> match the README
  • With-skill vs without-skill evals on the gold fixtures (above)

Notes for reviewers

Code is skills/review-doc/. Live review logs are gitignored; logs/example.json is the schema.

@railway-app

railway-app Bot commented Sep 8, 2026

Copy link
Copy Markdown

🚅 Deployed to the docs.page-pr-558 environment in docs.page

2 services not affected by this PR
  • docs.page
  • Redis

@BenInvertase BenInvertase changed the title Add the check-docs agent skill Add check-docs for reviewing docs.page pages Sep 8, 2026
Do not require a job title or invent an audience. Title and description
are enough, and recommended edits may only reuse words already on the page.
Skip first-mention expansion for interface names the same way PDF and
HTTP already skip, so short how-tos are not forced to spell them out.
Six gold pages cover planted issues, skip-if idioms, fix-all, conflict
grouping, mute, and review-each, with scripted assertions in evals.json.
One hit (and skip, when Skip-if exists) per writing scan. build_workers.py
regenerates workers.json and fails if a check heading is missing.
Iteration runs, benchmarks, and the static viewer stay on disk and should
not ship with the skill.
Keep the writing review distinct from the docs check CLI (links, assets,
render). Publishers can say review-doc or review doc.
@BenInvertase BenInvertase changed the title Add check-docs for reviewing docs.page pages Add review-doc for writing review of docs.page pages Sep 8, 2026
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012j4ivvniwRPsyYqfYSqrfp
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012j4ivvniwRPsyYqfYSqrfp
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012j4ivvniwRPsyYqfYSqrfp
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012j4ivvniwRPsyYqfYSqrfp
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012j4ivvniwRPsyYqfYSqrfp
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012j4ivvniwRPsyYqfYSqrfp
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012j4ivvniwRPsyYqfYSqrfp
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ BenInvertase
❌ claude
You have signed the CLA already but the status is still pending? Let us recheck it.

These two documentation fixes move to a separate PR at the author's request,
so this branch stays focused on the review-doc skill. Reverts
5c838e2 and
0984081.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012j4ivvniwRPsyYqfYSqrfp

@alexdukeinvertase alexdukeinvertase left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

*Test

  • Ran review-doc locally on the React Native Firebase getting-started page (~430 lines)
  • Looked at the results table, then a full suggested rewrite side by side with the original

Learning

  • I felt the checklist/table was good — easy to see which areas failed
  • Long list of individual fixes below it was overwhelming.
  • Side by side, a lot of the edits were tiny, so I had to look hard to spot what changed. Not wrong; the DX of using the output is the hard part
  • I would think users would just hit “fix all” after a bit, because so many fixes are small (e.g. e.g. → “for example”), rather than accept/reject one by one.

Recommendations

  • **Skill description ** add a short blurb on why choose this over other writing skills — e.g. you get a checklist and a results table first, then you decide what to change, instead of the agent quietly rewriting the page.
  • Let me approve a kind of change (style vs structural) rather than only tone-1, tone-2, … — that matches how I’d actually want to work when most edits are small

Group the violation list under each failing check and apply after the results print so the publisher can review a real diff. review each remains the opt-in pause.
Auto-applying a new heading id breaks inbound #slug links. Keep those findings for accept, and still apply sentence case when the id is unchanged.
Default apply still writes body edits and slug-stable sentence case; a slug-changing heading stays until accept. Review each leaves the same page untouched.
@claude
claude Bot marked this pull request as ready for review September 11, 2026 13:48
@claude
claude Bot requested a review from Ehesp September 11, 2026 13:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants