Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
140 changes: 140 additions & 0 deletions .claude/skills/audit-comparisons/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
---
name: audit-comparisons
description: Re-verify every claim on the comparison pages against its own citations, fix what is wrong, refresh the check dates, and report what could not be verified. Runs end to end with no input. Use when comparison data may be stale, before publishing comparison pages, or when asked to fact-check, re-verify, or refresh /comparisons.
---

# Audit comparisons

Re-checks `apps/sim/lib/compare/data/` — 20 competitor profiles plus Sim's own — against the
sources each claim already cites. Wrong or unsupported wording gets corrected, check dates move
to today for sources that were actually read, and anything unreadable is reported instead of
being quietly asserted or quietly dropped.

Run it with no arguments. It needs one thing from the user, once, before any work starts: a
working browser. Everything after that is automatic.

## What this protects

These pages exist to be read by answer engines and LLMs. The long-form text IS the product.
Earlier audits destroyed value by "tidying" — they cut 456 KB of sourced detail, dropped 250
`detail` fields, and reshaped the schema. That is the single worst outcome of a run.

1. **Never shorten `value` or `detail`.** They are long on purpose. `shortValue` (3-10 words) is
the only short field; it is a compression of `value` that introduces no new claim. If a
correction makes a claim narrower, the text stays long — it gets *more* precise, not smaller.
A run that reduces total bytes of `apps/sim/lib/compare/data/competitors/` without a
correction to justify each cut has failed.
2. **Never change shape.** No new fields, no renamed keys, no new files, no edits outside
`apps/sim/lib/compare/data/`. The audit updates language and dates in files that already exist.
3. **Absence of evidence is not evidence of absence.** If a capability cannot be confirmed, say
it is not documented and set `confidence: 'unknown'`. Never write that a product lacks a
feature because the docs could not be found.
4. **Cite only pages actually read in this run,** and only where the page really contains the
claim. A 200 response is not support.
5. **Preserve qualifications.** Plan gating, report scope and NDA conditions, preview/beta
labels, region limits, self-hosted-vs-cloud splits. Losing a qualification is a factual error.
6. **No artifacts.** No reports, notes, manifests or scratch files in the repo. Findings travel
in agent messages. The tree ends PR-ready.

## Step 0 — Preflight (before spawning anything)

Fail here, loudly, rather than halfway through 21 profiles.

1. `git status --porcelain` in the repo. If `apps/sim/lib/compare/data/` already has uncommitted
edits, tell the user and stop — the audit needs a clean baseline to diff against.
2. **Static fetch:** WebFetch `https://docs.sim.ai/platform/enterprise/scim`. Expect real prose.
3. **Client-rendered fetch:** WebFetch `https://trust.sim.ai/`. This one is expected to come back
as a bare title — that is the failure mode the browser exists for, not a problem.
4. **Browser:** navigate to `https://trust.sim.ai/`, wait ~3s for the client render, then read the
page text. It must list the compliance certifications. If the browser pane is unavailable, the
navigation is refused, or the text stays empty:

> Stop. Tell the user the audit needs browser access, name what failed, and ask them to enable
> it. Do not start the checkers — without the browser fallback every JavaScript-rendered
> pricing page and trust centre in the set becomes "unverifiable", which is a useless run.

Only when all four pass, continue. Say so in one line and keep going — do not ask permission.

## Step 1 — Units

One unit per profile file: every `apps/sim/lib/compare/data/competitors/*.ts` plus
`apps/sim/lib/compare/data/sim.ts`. A unit is roughly 85 claims and 50-90 URLs, which is the
largest chunk one agent can check without losing precision. Never batch two profiles into one
checker.

`sim.ts` is cited by every competitor page, so audit it too — its claims are load-bearing
everywhere.

## Step 2 — Check (bounded, read-only)

Spawn checkers in batches of 5. Each gets the verbatim prompt in
`agents/checker.md` with `{{PROFILE_PATH}}` replaced. The checker reads one file, opens every
cited URL, and returns two things: a per-URL verdict list, and flags for claims that are wrong,
unsupported, over-stated, or unverifiable. It edits nothing.

## Step 3 — Fix (fresh eyes, flags only)

For each profile that came back with flags, spawn a *fresh* agent with the verbatim prompt in
`agents/fixer.md`. It receives only the file path and that profile's flags plus per-URL verdicts —
no checker reasoning, no conversation history, nothing about the other profiles. It re-opens the
cited page itself, decides whether it agrees, and only then edits.

The split is the point: the checker is free to be suspicious because it cannot change anything,
and the fixer is a genuine second opinion because it never sees the argument, only the claim.
Roughly a quarter of flags are false positives — usually a retrieval that got truncated or a
second attached source that already supported the claim — and the fixer is what catches them.

## Step 4 — Dates

The page's "verified as of" line is derived from the citation dates in the data
(`getLatestVerifiedDate`, and the page review date from the oldest citation), so there is no
separate date to maintain. The fixer sets `asOf` to **today** for every source it or the checker
actually read and confirmed supports its claim, and leaves `asOf` untouched for any source that
came back unverifiable. That way a refreshed date always means "a machine read this page today
and the claim still stood".

## Step 5 — Verify the tree

After every fixer returns:

```bash
cd apps/sim && bun run type-check # expect 0 errors
cd apps/sim && bunx vitest run "app/(landing)/comparisons" "lib/compare"
bunx biome check $(git diff --name-only | grep -E '\.tsx?$' | tr '\n' ' ')
git status --porcelain # MUST contain no '??' lines
git diff --stat # MUST touch only lib/compare/data
```

Then compare size: `cat apps/sim/lib/compare/data/competitors/*.ts | wc -c` against the same
command at `git stash`-free HEAD. A large drop means someone truncated long-form text — find it
and restore it before reporting success.

## Step 6 — Report

Print to the conversation only. No files.

- One line per profile: claims checked, URLs read, flags raised, fixes applied, flags rejected.
- **Unverifiable list** — every page that could not be read after both WebFetch and the browser,
and every cell whose claim now rests on one. This is the most important section; surface it
even when it is long.
- Anything a fixer disagreed with, with its reason.
- Confirmation that the tree has no new files and no changes outside `lib/compare/data`.

## Known-hard sources

Lessons from previous runs — these are traps, not edge cases:

- **Trust centres and status pages** (`trust.sim.ai` and most vendor equivalents) are client
rendered. WebFetch returns a bare title. Always browser.
- **Pricing pages with interval toggles** (make.com, pipedream.com) hide the other interval's
prices behind a control. Read them in the browser and operate the toggle.
- **Plan matrices with tick marks** (Zapier SSO/SCIM) list every plan name in text while the
ticks restrict the feature to two of them. Text extraction reads the names and loses the
restriction. Browser, and look at the marks.
- **Docs that carry two generations of guidance** (CrewAI's agent docs kept old Docker examples
beside an explicit deprecation notice). Prefer the explicit deprecation or "current" section,
and never "correct" a claim to match a stale example.
- **OpenAI help-centre and product pages** return 403 to automated browsers. Expect
unverifiable; say so rather than guessing.
- **A flag may be wrong because the fact already cites a second source** that supports it. Check
every attached source before agreeing with a flag.
96 changes: 96 additions & 0 deletions .claude/skills/audit-comparisons/agents/checker.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
# Checker prompt

Spawn one per profile. Replace `{{PROFILE_PATH}}`. Pass verbatim.

---

You are fact-checking one competitor profile that backs Sim's public comparison pages. You are
read-only: you will not edit anything, and a separate reviewer decides what to do with your
findings.

File: `{{PROFILE_PATH}}` (repo root: the current working directory)

## What to check

Read the whole file. Every one of these is a claim that must be supported by the sources attached
to it:

- each fact in `facts.*`: its `value`, its `detail`, and its `shortValue`
- each entry in `standoutFeatures` and `limitations` (`title`, `description`, `shortDescription`)
- `oneLiner`
- any prose fields present: `leadAnswer`, `betterThanAnswer`, `sectionIntros`

A compound claim needs support for every clause. "SOC 2 Type II and ISO 27001, report under NDA"
is three checks, not one.

## How to read a source

1. **WebFetch** the URL and ask a question specific to the claim.
2. If that returns an error, a 403, an empty body, a bare title, or an obvious JavaScript shell,
**use the browser**: navigate to the URL, wait ~3 seconds for the client render, then read the
page text. If the page hides what you need behind a control — a monthly/annual pricing toggle,
an expander, a tab — operate the control and read again.
3. Only if both fail is the URL `unreachable`.

Text extraction silently loses meaning. A plan matrix lists every plan name in text while tick
marks restrict the feature to two of them; a pricing page shows one interval at a time. When a
claim depends on a table, a tick, a badge or a toggle, look at it in the browser before judging.

Beware pages carrying two generations of guidance — an old code example left beside an explicit
deprecation notice. The explicit "this was removed / current behaviour" section wins. Never
propose a correction that matches the stale half.

## Verdicts

For every URL: `supported` (you read it and it contains the claim), `contradicted`, `absent` (you
read it, the claim is not there), or `unreachable` (you could not read it, after the browser).

## What to flag

Flag only concrete defects:

- the source contradicts the wording
- the source does not contain the claim at all
- the wording overstates: asserts as general something the page gates to a plan, a region, a
preview, or a specific product tier
- a material qualification present in the source is missing from the wording
- the claim's only support is an unreachable page
- `shortValue` or `shortDescription` states something `value`/`description` does not

**Do not flag:** style, tone, wanting stronger certainty, hypothetical objections, or wording you
would have phrased differently. A fact carrying an explicit uncertainty label is correct, not
defective.

**Before flagging, check the fact's other sources.** Facts often cite two or three pages, and the
claim you cannot find in the first is frequently stated plainly in the second. A flag that
ignores an attached source wastes the reviewer's time.

## Proposed corrections

Where you propose replacement wording, it must be **at least as long and as specific** as what it
replaces. These pages exist so LLMs and answer engines can read comprehensive detail. Never
propose shortening, summarizing, or deleting `value` or `detail`. Corrections make a claim more
precise, not smaller. If a capability turns out to be narrower than stated, the text says exactly
how it is narrow — with the plan, the scope and the condition spelled out.

Never propose asserting that a product lacks a feature because you could not find evidence of it.
The correct form is that it is not publicly documented, with `confidence: 'unknown'`.

## Rules

- Edit nothing. Create no files, anywhere, for any reason. Keep notes in your own context.
- Do not run git, tests, or formatters.

## Report back

Plain text, compact, in your final message:

1. `CHECKED: <n> claims, <n> URLs — supported <n>, contradicted <n>, absent <n>, unreachable <n>`
2. `FLAGS:` a numbered list. For each: the object path (e.g. `facts.security.compliance.value`),
the current text verbatim, the URL, a **verbatim quote** from the page that justifies the flag
(or "page unreachable: <what you tried>"), and your proposed wording.
3. `URL VERDICTS:` one line per URL — `<url> — <verdict>` — so dates can be refreshed. This list
must cover every URL in the file.
4. `UNREACHABLE:` every URL that defeated both WebFetch and the browser, with what you tried.

If nothing is wrong, say so. Zero flags is a good outcome, not a failed review.
80 changes: 80 additions & 0 deletions .claude/skills/audit-comparisons/agents/fixer.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# Fixer prompt

Spawn fresh, one per flagged profile. Replace `{{PROFILE_PATH}}`, `{{FLAGS}}` and
`{{URL_VERDICTS}}`. Pass verbatim. Never include the checker's reasoning, the conversation
history, or any other profile's findings — the independence is what makes this step worth doing.

---

You are reviewing proposed corrections to one data file behind Sim's public comparison pages.
Someone else flagged these claims. You do not know who, and you do not have their reasoning. Your
job is to decide, from the sources themselves, which flags are real — and to apply only those.

File: `{{PROFILE_PATH}}` (repo root: the current working directory)

## Flags

{{FLAGS}}

## URL verdicts from the check pass

{{URL_VERDICTS}}

## For each flag

1. Open the cited page yourself. **WebFetch** first; if it returns an error, a 403, an empty body,
a bare title, or a JavaScript shell, **use the browser** — navigate, wait ~3 seconds for the
render, operate any toggle or tab hiding the answer, and read the text.
2. Read the fact's *other* sources too. A flag is wrong whenever another attached source already
supports the wording.
3. Decide:
- **agree** — the page contradicts the text, omits the claim, or the text loses a
qualification the page states. Fix it.
- **disagree** — the wording is supported, or the flag is a style preference, or it asks for
certainty the data honestly does not claim. Change nothing.
- **unverifiable** — you could not read the page either. Do not guess. Set the fact's
`confidence` to `'unknown'` only if nothing else supports it, leave the `asOf` date alone,
and report it.

Expect to disagree with some. Retrievals get truncated and pages hide text behind controls;
roughly a quarter of flags do not survive a second look. Record a verbatim quote for every
decision, in both directions.

## How to edit

- **Never shorten `value` or `detail`.** They are long deliberately: answer engines and LLMs read
them. A correction makes a claim more precise, never smaller. If a capability is narrower than
stated, spell out exactly how — the plan it needs, the scope it covers, the condition attached.
Deleting a `detail` field is a defect, not a cleanup.
- `shortValue` / `shortDescription` stay 3-10 words and remain a pure compression of the long
text. If your fix changes what the long text claims, re-compress it — never let the short form
assert something the long form does not.
- Keep the file's conventions: a yes/no capability starts `Yes: ` or `No: `; single quotes,
trailing commas, 2-space indent; valid TypeScript.
- Never assert a product lacks a feature because evidence was not found. The honest form is that
it is not publicly documented, with `confidence: 'unknown'`.
- Do not change the shape: no new fields, no renamed keys, no reordering, no new files anywhere.
- Touch nothing in the file except the flagged claims and the dates below.

## Dates

Today is the date given to you by the environment. For every URL marked `supported` in the
verdict list above, and every URL you confirmed yourself, set that source's `asOf` to today —
including sources whose wording needed no change. A refreshed date means "read today, claim still
stands".

Leave `asOf` exactly as it is for any URL that was `unreachable` or that you could not read. Never
refresh a date for a page nobody opened.

## Rules

- Create no files, anywhere, for any reason.
- Do not run git, tests, or formatters.

## Report back

1. `DISPOSITIONS:` one line per flag — `<n>: agreed | disagreed | unverifiable — <one sentence>`
plus the verbatim quote you relied on.
2. `EDITS:` the object paths you changed.
3. `DATES:` how many `asOf` values you refreshed, and which URLs you deliberately left stale.
4. `UNVERIFIABLE:` every claim now resting on a page nobody could read.
4 changes: 2 additions & 2 deletions apps/sim/app/(landing)/comparisons/comparison-sections.ts
Original file line number Diff line number Diff line change
Expand Up @@ -87,14 +87,14 @@ export const COMPARISON_SECTIONS: ComparisonSectionDef[] = [
group: 'security',
title: 'Security & compliance',
rows: [
{ key: 'soc2', label: 'SOC 2' },
{ key: 'compliance', label: 'Compliance' },
{ key: 'dataResidency', label: 'Data residency' },
{ key: 'rbac', label: 'Role-based access control' },
{ key: 'auditLogging', label: 'Audit logging' },
{ key: 'additionalCompliance', label: 'Additional compliance' },
{ key: 'modelAndToolGovernance', label: 'Model & tool governance' },
{ key: 'credentialGovernance', label: 'Credential governance' },
{ key: 'sso', label: 'Single sign-on (SSO)' },
{ key: 'scim', label: 'SCIM provisioning' },
{ key: 'sessionPolicy', label: 'Custom session policy' },
{ key: 'thirdPartyVetting', label: 'Vetted first-party integrations' },
{ key: 'piiRedaction', label: 'PII redaction' },
Expand Down
2 changes: 1 addition & 1 deletion apps/sim/app/(landing)/comparisons/utils.ts
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ export function buildComparisonFaqs(competitor: CompetitorProfile): ComparisonFa
},
{
question: `Is Sim more secure than ${name}?`,
answer: `Security is a like-for-like comparison, not a one-line verdict. Sim: ${summarizeFact(simProfile.facts.security.additionalCompliance.value)} ${name}: ${summarizeFact(facts.security.additionalCompliance.value)} Check the Security & compliance rows above for the full breakdown, including SSO, audit logging, and data residency.`,
answer: `Security is a like-for-like comparison, not a one-line verdict. Sim: ${summarizeFact(simProfile.facts.security.compliance.value)} ${name}: ${summarizeFact(facts.security.compliance.value)} Check the Security & compliance rows above for the full breakdown, including SSO, audit logging, and data residency.`,
},
{
question: `Which has stronger AI agent capabilities, Sim or ${name}?`,
Expand Down
Loading
Loading