Cloudflare managed robots.txt disallows ClaudeBot/GPTBot/CCBot — roadmap 1.5 is silently failing

[eaitbrahim]

Summary

The served robots.txt disallows the major AI crawlers — including ClaudeBot and GPTBot. This directly contradicts roadmap item 1.5, which requires the opposite:

| 1.5 | llms.txt at root summarizing the site for AI crawlers; ensure GPTBot/ClaudeBot/PerplexityBot not blocked in robots.txt | none | File fetchable; AI crawlers appear in CF analytics |

— docs/seo/IMPLEMENTATION-ROADMAP.md:14

It is failing silently: nothing in this repo is wrong, so no build, check, or review would ever catch it.

Evidence

public/robots.txt and dist/robots.txt are both correct and identical (4 lines):

User-agent: *
Allow: /

Sitemap: https://keeltrading.com/sitemap-index.xml

But curl https://keeltrading.com/robots.txt returns 65 lines. Cloudflare prepends a # BEGIN Cloudflare Managed content section at the edge, containing:

User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /

User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CloudflareBrowserRenderingCrawler
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: meta-externalagent
Disallow: /

Nine AI/LLM user-agents blocked, plus an ai-train=no content signal. None of it is in the repo — it is injected by a zone-level Cloudflare dashboard setting, so it cannot be overridden from the origin.

Impact

• GEO work is undercut at the root: ChatGPT, Claude, CCBot-derived corpora and Google's AI surfaces are told not to fetch the site.
• llms.txt ships and serves 200, but the crawlers it exists for are disallowed from reading the pages it points at — half of 1.5 is done and the other half negates it.
• Google-Extended: Disallow opts out of Gemini/AI Overviews grounding. (Normal Google Search indexing is unaffected — search=yes and User-agent: * Allow: / still stand.)
• CloudflareBrowserRenderingCrawler: Disallow can also affect link-preview/unfurl rendering, which matters for the outreach in docs/seo/LINK-ACQUISITION.md.

Root cause

Cloudflare's managed robots.txt / AI-crawler-blocking feature is enabled on the keeltrading.com zone. This is a dashboard toggle, not code — it was very likely on by default rather than chosen.

Fix

Manual, dashboard-only (cannot be done from this repo):

1. Cloudflare dashboard → the keeltrading.com zone
2. Find the AI crawler control setting — the one that injects managed robots.txt content (Cloudflare has shipped this under AI Crawl Control, previously AI Audit / "Block AI bots"; the label has moved across releases, so look for the setting that mentions managed robots.txt or content signals)
3. Turn off the managed AI-bot block for the zone
4. Decide the Content Signals policy deliberately. ai-train=no is a defensible stance for an open-source project; ai-input / search should stay permissive so answer engines can cite the site.

Falsifiability

curl -s https://keeltrading.com/robots.txt | grep -A1 -E '^User-agent: (ClaudeBot|GPTBot|Google-Extended)'

Passes when no AI crawler is followed by Disallow: /. Confirm afterwards that AI crawlers appear in Cloudflare analytics, per 1.5's own check.

Regression guard

Because this failed silently from outside the repo, a live-site check is being added so it cannot regress unnoticed. See linked PR.