Stop the claim verifier contradicting a PR against its own live page

[CamSoper]

Proposed changes

On #21720 the pre-merge review stamped three correct edits contradicted (a glow-up refreshing an example terraform-provider pin from 0.10.0 to 1.4.0, which is the latest release). Two of the three verdicts cited the live published copy of the page under review as their source. That page still shows the old value, so the check is a tautology: any PR that changes a number "contradicts" it. A human had to dispute the findings before the update lane ran gh release list and conceded.

It isn't a one-off. Across the 117 v3 review records in the ledger, 66 verdicts are contradicted; 9 cite the reviewed page's own live URL and 9 more rest on other pulumi.com pages alone. The verifier prompt already forbids both, but only the generated-from-data rule was enforced in code.

Source-discipline gates (verify-claims.py)

• A contradicted/mismatch whose only source is the reviewed page's own live URL, a contradicted whose only sources are editorial pulumi.com pages, and a contradicted citing only the reviewed file's repo path each get one re-check in the pass1 (gh) lane, told exactly what the first pass leaned on and why it isn't evidence.
• An independently sourced answer stands, whichever way it goes. Otherwise the verdict is downgraded to unverifiable, stamped source_discipline_gate, keeps the original reasoning, and asks the composer for an author question instead of a 🚨.
• A re-check rather than a blunt downgrade, because "the PR changed what the live page says" is a useful change detector. It just isn't a judge.
• Exempt: generated reference pages (the prompt's existing carve-out), and a claim that is about the cited URL (the pass2 dead-link case). Inert in --dry-run and without a repo root.

Routing

• A version claim with a non-URL source_hint now routes pass1 instead of pass3. claim-extraction.md tells the extractor to put the package name in source_hint, and the router treated any bare name as "named external source, go web search". So a pin for a Pulumi package without "pulumi" in its name (terraform-provider) got web-searched, where the top hit is pulumi.com's own page.

Prose kept in sync

• Verifier prompt and fact-check.md now say which rules are harness-enforced. fact-check.md was missing the self-reference rule entirely, and its worked-example count was stale.
• claim-extraction.md: worked example 14 for a Pulumi.yaml packages: block (the version binds to the package, parameters names the wrapped provider), and the version row prefers the pulumi/<repo> hint form.

Verification

make test-review-pipeline passes, including 61 new tests built on the real ledger verdicts from #21720, #21552, #21602, #21394, and #21509.

Live replay of the #21720 claims (real model calls, same inputs):

	master	this branch
version claims with hint terraform-provider	all pass3 (web search)	all pass1 (gh)
contradicted citing the page's own live URL	3 of 5	0 of 5
re-check observed live	n/a	yes: self-reference caught, re-verified against gh release list, verified

Fork CI smoke (CamSoper#246, the exact #21720 diff, one run): all three 1.4.0 claims routed pass1 and came back verified against gh release list -R pulumi/pulumi-terraform-provider; zero contradicted; validator clean. The routing fix kept the claims off web search, so the re-check hop itself didn't fire in CI -- its live evidence is the local replay above.

Known residual: when the extracted claim text misdescribes the page ("pins the random provider to 1.4.0"), the verifier can still return mismatch against the page itself. That's an extraction error, and worked example 14 is the fix for it, but it's prompt-level and the replay can't prove it.

Also worth knowing: the pulumi/<repo> hint form yields entity keys like version/pulumi-pulumi-gcp (vs version/pulumi-gcp for the bare form). That quirk predates this PR -- worked example 4 already recommends the pulumi/<repo> form -- but more claims will take that shape now, so claims-index continuity shifts for them. Not touching entity_key.py here.

reverify-claims.py calls process_claim, so the nightly lane gets the re-check too. Its own own-corpus demotion still runs afterward.

Related issues (optional)

Follow-ups filed: #21732 (give the verifier the PR body's rationale), #21733 (27% of verified verdicts cite only the page under review). Companion: #21736 (persist the claim records this trace couldn't find).

[pulumi-bot]

Your site preview for commit d15a885 is ready! 🎉

http://www-testing-pulumi-docs-origin-pr-21737-d15a885d.s3-website.us-west-2.amazonaws.com

[github-actions]

Reviewer's guide v2 — not for the author

Tip

This is the reviewer's guide. Work through the ⚠️ checklist below, then approve — approving asserts only that the ⚠️ items looked right to you. Machine-verified this run: links, shortcodes, page metadata, and every claim marked verified (receipts on the evidence page). Code samples are read, not compiled.

PR author: your to-do list is the other review comment, "Author action guide" — nothing on this card is yours.

Approval needed from: @pulumi/docs-tools — any member's approval satisfies the merge gate.

Note

What this PR changes:

• verify-claims.py gains a source-discipline gate: a contradicted/mismatch whose only source is the live published copy of the reviewed page gets one re-check in the pass1 lane, and is downgraded to unverifiable if that re-check can't settle it.
• The same gate covers two neighboring shapes — contradicted resting only on other editorial pulumi.com pages (same-site-only) and contradicted citing only the reviewed file's own repo path (own-file-only).
• route_claim now sends a version claim whose source_hint is a bare package name to pass1 instead of pass3, since that hint names a package rather than an external authority.
• claim-extraction.md adds worked example 14 (which package a version: key in a nested packages: block belongs to) and asks for pulumi/<repo>-form hints.
• fact-check.md adds a fifth source-discipline rule describing the self-reference gate and corrects the worked-example count from 12 to 14.
• New tests (test_verify_claims_self_reference.py, plus routing cases) drive the gate from real verdict records captured in testdata/self-reference-verdicts.json.

Wrongness that would matter here is a gate that swallows a real regression — silently downgrading a contradicted that was actually correct — or one that fires on evidence which is genuinely independent. The changed Python and its tests were read together, the reference-doc prose was checked against the code it describes, and link/shortcode/metadata checks ran over the changed files; no factual claims about the world were extracted from this diff.

Review confidence:

Dimension	Level	Notes
mechanics	HIGH
facts	MEDIUM	Not yours to check — the diff asserts nothing about the outside world; the gate's behavior was read against its tests, not executed here (the pipeline test job runs them).

⚠️ Check these before approving

ID	Where	Finding
F1	.claude/commands/docs-review/scripts/verify-claims.py L422	The new version carve-out routes every version claim with a non-URL hint to the gh/local-reads lane, including hints that name a genuinely external authority — the test suite pins Node.js to that lane. Nothing is lost mechanically (that lane can escalate back out to web search, and the escalation path is exercised elsewhere in the suite), so this is a cost-and-latency call, not a correctness one: confirm you're happy paying an extra hop on third-party version pins to stop Pulumi package pins being judged against pulumi.com's own page. ✋ Accepted as-is by CamSoper on 2026-09-18. Accepting as-is — it's the right trade. Version claims carrying a bare non-URL hint are rare (~60 of ~4,000 claims in the index, most with no hint at all), pass1 escalates to web search when gh can't close it, and the failure this replaces was a false blocker needing a human dispute to clear. An extra hop on the occasional Node.js pin is cheap next to that; if PR #21736's claim artifacts show it costing real money, a narrower rule is easy to add with data in hand.

Editorial stances introduced by this PR

Superlative, ranking, or comparative language the diff adds. No verdict — a page's own framing isn't fact-checkable — but confirm each is a stance the docs should take, and that no agent-written rewrite introduced it unasked.

• L181 .claude/commands/docs-review/references/fact-check.md — "The canonical worked-example set — composite/split, implicit comparison, quantitative, negative, the third-party-attribution flip, and the hard…" — positioning (found by regex)

✅ What you can rubber-stamp

• Facts: no factual claims found in the changed lines.
• Mechanics: no internal links added.
• Style: 0 advisory suggestion(s) left with the author; never blocking.

💡 Pre-existing issues in touched files: 0 — details on the evidence page.

📎 Full evidence: verification trail, investigation log, review history.

_{Review v2 · updated 2026-09-18T19:24:26Z · head commit d15a885}

---

For the reviewer: the ⚠️ items above are the minutes that matter — the receipts for everything machine-verified are on the evidence page. The author's open items live on their own card (the comment headed "Author action guide"); while any are open, a Waiting on the author list above tracks them, and merge stays blocked until they're answered. If something here seems off, comment on the PR — @claude <your point> #update-review re-adjudicates with your input.

[github-actions]

Author action guide v2 — nothing blocks merge

Note

Nothing here blocks merge — no open items need an answer from you. A human reviewer still approves the merge.

This PR changes the docs-review tooling itself — it stops the claim verifier from treating the live published copy of a page (or another pulumi.com page) as proof that the page is wrong, and adds tests for the new behavior; the review checked the changed logic, its tests, and the reference docs that describe it, and found nothing that needs an answer from you.

🚨 Fix or disagree

Nothing to fix — this section is empty.

❓ Questions for you

No open questions for you.

📎 Full evidence: verification trail, investigation log, review history.

_{Review v2 · updated 2026-09-18T19:24:26Z · head commit d15a885}

---

How to answer

Every 🚨 and ❓ item above needs one of these before merge:

1. Fix it — push the change. If your push lines up with the flagged lines, this card shows a 🔄 banner within a minute and then refreshes itself. No banner? Comment:

   @claude I pushed a fix for F1 #update-review
   

2. Disagree with it — say which item and why; the review re-checks with your input:

   @claude F2: <your reasoning — e.g. where the figure comes from> #update-review
   

   Either way your answer counts: the review marks the item resolved, or keeps it with a 🛡️ note for your human reviewer to weigh — it stops blocking merge in both cases.

3. Accept it as-is — you own the PR; a one-line reason is a valid answer, and your reviewer sees it beside the finding:

   @claude F2: accepting as-is — <your reason> #update-review
   

   To accept every open item at once: @claude accepting all open items — <reason> #update-review.

The #update-review hashtag matters — it routes your reply to this review, and (besides pushing a fix) it is the only reply that unblocks merge. A bare @claude gets you ad-hoc help and leaves this card — and the merge block — untouched.

Please don't edit, hide, or delete this comment — it is the review's record. Full mechanics: CONTRIBUTING.md §AI-assisted contributions.

[CamSoper]

@claude F1: accepting as-is -- it's the right trade. Version claims that carry a bare non-URL hint are rare (version is ~60 of ~4,000 claims in the index, and most carry no hint at all), pass1 escalates to web search when gh can't close it, and the failure this replaces was a false blocker that needed a human dispute to clear. An extra hop on the occasional Node.js pin is cheap next to that. If PR #21736's claim artifacts later show it costing real money, a narrower rule is easy to add with data in hand. #update-review

[github-actions]

🤖 Review updated on @CamSoper's request.