Skip to content

Commit 4111195

Browse files
j15zclaude
j15z
and
claude
committed
feat(skills): add the audit-comparisons skill
Runs the comparison-page fact check end to end with no arguments: preflight that web fetching and the browser both work, one read-only checker per profile, then a fresh fixer that sees only that profile's flags and decides for itself which are real. Encodes what three review rounds taught us. Long-form value and detail are never shortened, because answer engines read them and a previous run cut 456 KB of sourced text. Absence of evidence is written as "not documented", never as absence of the feature. Pages that defeat fetching are retried in a browser and then reported as unverifiable rather than guessed. Dates refresh only for pages actually read. Nothing is written to the repo except edits to the data files. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 2918918 commit 4111195

3 files changed

Lines changed: 316 additions & 0 deletions

File tree

‎.claude/skills/audit-comparisons/SKILL.md‎

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
---
2+
name: audit-comparisons
3+
description: Re-verify every claim on the comparison pages against its own citations, fix what is wrong, refresh the check dates, and report what could not be verified. Runs end to end with no input. Use when comparison data may be stale, before publishing comparison pages, or when asked to fact-check, re-verify, or refresh /comparisons.
4+
---
5+
6+
# Audit comparisons
7+
8+
Re-checks `apps/sim/lib/compare/data/` — 20 competitor profiles plus Sim's own — against the
9+
sources each claim already cites. Wrong or unsupported wording gets corrected, check dates move
10+
to today for sources that were actually read, and anything unreadable is reported instead of
11+
being quietly asserted or quietly dropped.
12+
13+
Run it with no arguments. It needs one thing from the user, once, before any work starts: a
14+
working browser. Everything after that is automatic.
15+
16+
## What this protects
17+
18+
These pages exist to be read by answer engines and LLMs. The long-form text IS the product.
19+
Earlier audits destroyed value by "tidying" — they cut 456 KB of sourced detail, dropped 250
20+
`detail` fields, and reshaped the schema. That is the single worst outcome of a run.
21+
22+
1. **Never shorten `value` or `detail`.** They are long on purpose. `shortValue` (3-10 words) is
23+
the only short field; it is a compression of `value` that introduces no new claim. If a
24+
correction makes a claim narrower, the text stays long — it gets *more* precise, not smaller.
25+
A run that reduces total bytes of `apps/sim/lib/compare/data/competitors/` without a
26+
correction to justify each cut has failed.
27+
2. **Never change shape.** No new fields, no renamed keys, no new files, no edits outside
28+
`apps/sim/lib/compare/data/`. The audit updates language and dates in files that already exist.
29+
3. **Absence of evidence is not evidence of absence.** If a capability cannot be confirmed, say
30+
it is not documented and set `confidence: 'unknown'`. Never write that a product lacks a
31+
feature because the docs could not be found.
32+
4. **Cite only pages actually read in this run,** and only where the page really contains the
33+
claim. A 200 response is not support.
34+
5. **Preserve qualifications.** Plan gating, report scope and NDA conditions, preview/beta
35+
labels, region limits, self-hosted-vs-cloud splits. Losing a qualification is a factual error.
36+
6. **No artifacts.** No reports, notes, manifests or scratch files in the repo. Findings travel
37+
in agent messages. The tree ends PR-ready.
38+
39+
## Step 0 — Preflight (before spawning anything)
40+
41+
Fail here, loudly, rather than halfway through 21 profiles.
42+
43+
1. `git status --porcelain` in the repo. If `apps/sim/lib/compare/data/` already has uncommitted
44+
edits, tell the user and stop — the audit needs a clean baseline to diff against.
45+
2. **Static fetch:** WebFetch `https://docs.sim.ai/platform/enterprise/scim`. Expect real prose.
46+
3. **Client-rendered fetch:** WebFetch `https://trust.sim.ai/`. This one is expected to come back
47+
as a bare title — that is the failure mode the browser exists for, not a problem.
48+
4. **Browser:** navigate to `https://trust.sim.ai/`, wait ~3s for the client render, then read the
49+
page text. It must list the compliance certifications. If the browser pane is unavailable, the
50+
navigation is refused, or the text stays empty:
51+
52+
> Stop. Tell the user the audit needs browser access, name what failed, and ask them to enable
53+
> it. Do not start the checkers — without the browser fallback every JavaScript-rendered
54+
> pricing page and trust centre in the set becomes "unverifiable", which is a useless run.
55+
56+
Only when all four pass, continue. Say so in one line and keep going — do not ask permission.
57+
58+
## Step 1 — Units
59+
60+
One unit per profile file: every `apps/sim/lib/compare/data/competitors/*.ts` plus
61+
`apps/sim/lib/compare/data/sim.ts`. A unit is roughly 85 claims and 50-90 URLs, which is the
62+
largest chunk one agent can check without losing precision. Never batch two profiles into one
63+
checker.
64+
65+
`sim.ts` is cited by every competitor page, so audit it too — its claims are load-bearing
66+
everywhere.
67+
68+
## Step 2 — Check (bounded, read-only)
69+
70+
Spawn checkers in batches of 5. Each gets the verbatim prompt in
71+
`agents/checker.md` with `{{PROFILE_PATH}}` replaced. The checker reads one file, opens every
72+
cited URL, and returns two things: a per-URL verdict list, and flags for claims that are wrong,
73+
unsupported, over-stated, or unverifiable. It edits nothing.
74+
75+
## Step 3 — Fix (fresh eyes, flags only)
76+
77+
For each profile that came back with flags, spawn a *fresh* agent with the verbatim prompt in
78+
`agents/fixer.md`. It receives only the file path and that profile's flags plus per-URL verdicts —
79+
no checker reasoning, no conversation history, nothing about the other profiles. It re-opens the
80+
cited page itself, decides whether it agrees, and only then edits.
81+
82+
The split is the point: the checker is free to be suspicious because it cannot change anything,
83+
and the fixer is a genuine second opinion because it never sees the argument, only the claim.
84+
Roughly a quarter of flags are false positives — usually a retrieval that got truncated or a
85+
second attached source that already supported the claim — and the fixer is what catches them.
86+
87+
## Step 4 — Dates
88+
89+
The page's "verified as of" line is derived from the citation dates in the data
90+
(`getLatestVerifiedDate`, and the page review date from the oldest citation), so there is no
91+
separate date to maintain. The fixer sets `asOf` to **today** for every source it or the checker
92+
actually read and confirmed supports its claim, and leaves `asOf` untouched for any source that
93+
came back unverifiable. That way a refreshed date always means "a machine read this page today
94+
and the claim still stood".
95+
96+
## Step 5 — Verify the tree
97+
98+
After every fixer returns:
99+
100+
```bash
101+
cd apps/sim && bun run type-check # expect 0 errors
102+
cd apps/sim && bunx vitest run "app/(landing)/comparisons" "lib/compare"
103+
bunx biome check $(git diff --name-only | grep -E '\.tsx?$' | tr '\n' ' ')
104+
git status --porcelain # MUST contain no '??' lines
105+
git diff --stat # MUST touch only lib/compare/data
106+
```
107+
108+
Then compare size: `cat apps/sim/lib/compare/data/competitors/*.ts | wc -c` against the same
109+
command at `git stash`-free HEAD. A large drop means someone truncated long-form text — find it
110+
and restore it before reporting success.
111+
112+
## Step 6 — Report
113+
114+
Print to the conversation only. No files.
115+
116+
- One line per profile: claims checked, URLs read, flags raised, fixes applied, flags rejected.
117+
- **Unverifiable list** — every page that could not be read after both WebFetch and the browser,
118+
and every cell whose claim now rests on one. This is the most important section; surface it
119+
even when it is long.
120+
- Anything a fixer disagreed with, with its reason.
121+
- Confirmation that the tree has no new files and no changes outside `lib/compare/data`.
122+
123+
## Known-hard sources
124+
125+
Lessons from previous runs — these are traps, not edge cases:
126+
127+
- **Trust centres and status pages** (`trust.sim.ai` and most vendor equivalents) are client
128+
rendered. WebFetch returns a bare title. Always browser.
129+
- **Pricing pages with interval toggles** (make.com, pipedream.com) hide the other interval's
130+
prices behind a control. Read them in the browser and operate the toggle.
131+
- **Plan matrices with tick marks** (Zapier SSO/SCIM) list every plan name in text while the
132+
ticks restrict the feature to two of them. Text extraction reads the names and loses the
133+
restriction. Browser, and look at the marks.
134+
- **Docs that carry two generations of guidance** (CrewAI's agent docs kept old Docker examples
135+
beside an explicit deprecation notice). Prefer the explicit deprecation or "current" section,
136+
and never "correct" a claim to match a stale example.
137+
- **OpenAI help-centre and product pages** return 403 to automated browsers. Expect
138+
unverifiable; say so rather than guessing.
139+
- **A flag may be wrong because the fact already cites a second source** that supports it. Check
140+
every attached source before agreeing with a flag.

‎.claude/skills/audit-comparisons/agents/checker.md‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
# Checker prompt
2+
3+
Spawn one per profile. Replace `{{PROFILE_PATH}}`. Pass verbatim.
4+
5+
---
6+
7+
You are fact-checking one competitor profile that backs Sim's public comparison pages. You are
8+
read-only: you will not edit anything, and a separate reviewer decides what to do with your
9+
findings.
10+
11+
File: `{{PROFILE_PATH}}` (repo root: the current working directory)
12+
13+
## What to check
14+
15+
Read the whole file. Every one of these is a claim that must be supported by the sources attached
16+
to it:
17+
18+
- each fact in `facts.*`: its `value`, its `detail`, and its `shortValue`
19+
- each entry in `standoutFeatures` and `limitations` (`title`, `description`, `shortDescription`)
20+
- `oneLiner`
21+
- any prose fields present: `leadAnswer`, `betterThanAnswer`, `sectionIntros`
22+
23+
A compound claim needs support for every clause. "SOC 2 Type II and ISO 27001, report under NDA"
24+
is three checks, not one.
25+
26+
## How to read a source
27+
28+
1. **WebFetch** the URL and ask a question specific to the claim.
29+
2. If that returns an error, a 403, an empty body, a bare title, or an obvious JavaScript shell,
30+
**use the browser**: navigate to the URL, wait ~3 seconds for the client render, then read the
31+
page text. If the page hides what you need behind a control — a monthly/annual pricing toggle,
32+
an expander, a tab — operate the control and read again.
33+
3. Only if both fail is the URL `unreachable`.
34+
35+
Text extraction silently loses meaning. A plan matrix lists every plan name in text while tick
36+
marks restrict the feature to two of them; a pricing page shows one interval at a time. When a
37+
claim depends on a table, a tick, a badge or a toggle, look at it in the browser before judging.
38+
39+
Beware pages carrying two generations of guidance — an old code example left beside an explicit
40+
deprecation notice. The explicit "this was removed / current behaviour" section wins. Never
41+
propose a correction that matches the stale half.
42+
43+
## Verdicts
44+
45+
For every URL: `supported` (you read it and it contains the claim), `contradicted`, `absent` (you
46+
read it, the claim is not there), or `unreachable` (you could not read it, after the browser).
47+
48+
## What to flag
49+
50+
Flag only concrete defects:
51+
52+
- the source contradicts the wording
53+
- the source does not contain the claim at all
54+
- the wording overstates: asserts as general something the page gates to a plan, a region, a
55+
preview, or a specific product tier
56+
- a material qualification present in the source is missing from the wording
57+
- the claim's only support is an unreachable page
58+
- `shortValue` or `shortDescription` states something `value`/`description` does not
59+
60+
**Do not flag:** style, tone, wanting stronger certainty, hypothetical objections, or wording you
61+
would have phrased differently. A fact carrying an explicit uncertainty label is correct, not
62+
defective.
63+
64+
**Before flagging, check the fact's other sources.** Facts often cite two or three pages, and the
65+
claim you cannot find in the first is frequently stated plainly in the second. A flag that
66+
ignores an attached source wastes the reviewer's time.
67+
68+
## Proposed corrections
69+
70+
Where you propose replacement wording, it must be **at least as long and as specific** as what it
71+
replaces. These pages exist so LLMs and answer engines can read comprehensive detail. Never
72+
propose shortening, summarizing, or deleting `value` or `detail`. Corrections make a claim more
73+
precise, not smaller. If a capability turns out to be narrower than stated, the text says exactly
74+
how it is narrow — with the plan, the scope and the condition spelled out.
75+
76+
Never propose asserting that a product lacks a feature because you could not find evidence of it.
77+
The correct form is that it is not publicly documented, with `confidence: 'unknown'`.
78+
79+
## Rules
80+
81+
- Edit nothing. Create no files, anywhere, for any reason. Keep notes in your own context.
82+
- Do not run git, tests, or formatters.
83+
84+
## Report back
85+
86+
Plain text, compact, in your final message:
87+
88+
1. `CHECKED: <n> claims, <n> URLs — supported <n>, contradicted <n>, absent <n>, unreachable <n>`
89+
2. `FLAGS:` a numbered list. For each: the object path (e.g. `facts.security.compliance.value`),
90+
the current text verbatim, the URL, a **verbatim quote** from the page that justifies the flag
91+
(or "page unreachable: <what you tried>"), and your proposed wording.
92+
3. `URL VERDICTS:` one line per URL — `<url> — <verdict>` — so dates can be refreshed. This list
93+
must cover every URL in the file.
94+
4. `UNREACHABLE:` every URL that defeated both WebFetch and the browser, with what you tried.
95+
96+
If nothing is wrong, say so. Zero flags is a good outcome, not a failed review.

‎.claude/skills/audit-comparisons/agents/fixer.md‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
# Fixer prompt
2+
3+
Spawn fresh, one per flagged profile. Replace `{{PROFILE_PATH}}`, `{{FLAGS}}` and
4+
`{{URL_VERDICTS}}`. Pass verbatim. Never include the checker's reasoning, the conversation
5+
history, or any other profile's findings — the independence is what makes this step worth doing.
6+
7+
---
8+
9+
You are reviewing proposed corrections to one data file behind Sim's public comparison pages.
10+
Someone else flagged these claims. You do not know who, and you do not have their reasoning. Your
11+
job is to decide, from the sources themselves, which flags are real — and to apply only those.
12+
13+
File: `{{PROFILE_PATH}}` (repo root: the current working directory)
14+
15+
## Flags
16+
17+
{{FLAGS}}
18+
19+
## URL verdicts from the check pass
20+
21+
{{URL_VERDICTS}}
22+
23+
## For each flag
24+
25+
1. Open the cited page yourself. **WebFetch** first; if it returns an error, a 403, an empty body,
26+
a bare title, or a JavaScript shell, **use the browser** — navigate, wait ~3 seconds for the
27+
render, operate any toggle or tab hiding the answer, and read the text.
28+
2. Read the fact's *other* sources too. A flag is wrong whenever another attached source already
29+
supports the wording.
30+
3. Decide:
31+
- **agree** — the page contradicts the text, omits the claim, or the text loses a
32+
qualification the page states. Fix it.
33+
- **disagree** — the wording is supported, or the flag is a style preference, or it asks for
34+
certainty the data honestly does not claim. Change nothing.
35+
- **unverifiable** — you could not read the page either. Do not guess. Set the fact's
36+
`confidence` to `'unknown'` only if nothing else supports it, leave the `asOf` date alone,
37+
and report it.
38+
39+
Expect to disagree with some. Retrievals get truncated and pages hide text behind controls;
40+
roughly a quarter of flags do not survive a second look. Record a verbatim quote for every
41+
decision, in both directions.
42+
43+
## How to edit
44+
45+
- **Never shorten `value` or `detail`.** They are long deliberately: answer engines and LLMs read
46+
them. A correction makes a claim more precise, never smaller. If a capability is narrower than
47+
stated, spell out exactly how — the plan it needs, the scope it covers, the condition attached.
48+
Deleting a `detail` field is a defect, not a cleanup.
49+
- `shortValue` / `shortDescription` stay 3-10 words and remain a pure compression of the long
50+
text. If your fix changes what the long text claims, re-compress it — never let the short form
51+
assert something the long form does not.
52+
- Keep the file's conventions: a yes/no capability starts `Yes: ` or `No: `; single quotes,
53+
trailing commas, 2-space indent; valid TypeScript.
54+
- Never assert a product lacks a feature because evidence was not found. The honest form is that
55+
it is not publicly documented, with `confidence: 'unknown'`.
56+
- Do not change the shape: no new fields, no renamed keys, no reordering, no new files anywhere.
57+
- Touch nothing in the file except the flagged claims and the dates below.
58+
59+
## Dates
60+
61+
Today is the date given to you by the environment. For every URL marked `supported` in the
62+
verdict list above, and every URL you confirmed yourself, set that source's `asOf` to today —
63+
including sources whose wording needed no change. A refreshed date means "read today, claim still
64+
stands".
65+
66+
Leave `asOf` exactly as it is for any URL that was `unreachable` or that you could not read. Never
67+
refresh a date for a page nobody opened.
68+
69+
## Rules
70+
71+
- Create no files, anywhere, for any reason.
72+
- Do not run git, tests, or formatters.
73+
74+
## Report back
75+
76+
1. `DISPOSITIONS:` one line per flag — `<n>: agreed | disagreed | unverifiable — <one sentence>`
77+
plus the verbatim quote you relied on.
78+
2. `EDITS:` the object paths you changed.
79+
3. `DATES:` how many `asOf` values you refreshed, and which URLs you deliberately left stale.
80+
4. `UNVERIFIABLE:` every claim now resting on a page nobody could read.

0 commit comments

Comments
 (0)